Intercycle and Workflow Layers

These two layers, described in the HWRF System Overview page, handle the high-level automation. The Workflow Layer divides each HWRF forecast cycle into a set of distinct batch job. In each batch job, the Workflow Layer sets up the batch system for an HWRF script, runs that script, and then submits the next job or jobs in the workflow. The Intercycle Layer glues together multiple forecast cycles, handling dependencies between each cycle's batch jobs.

While it is possible to discuss the two layers independently, they tend to be implemented together in one package. In this page, we discuss the available implementations and point the user to more information.

HWRF Rocoto Workflow

Rocoto is a NOAA-originated public open source workflow management system. Most organizations that run the 2015 HWRF use Rocoto to do it. The HWRF Rocoto system is found in the rocoto/ subdirectory of HWRF, which contains a number of XML files. Those files each describe one job in the workflow, one set of dependencies, or details for running in a specific supercomputer. This workflow management system is capable of running a single cycle or a set of cycles for the same storm. It handles inter-cycle dependencies as efficiently as possible, submitting the next cycle's jobs as soon as their dependencies are met. There are options for throttling usage of various resources, and limiting the number of cycles that are active at any given time.

There is extensive documentation on running the Rocoto-based workflow here:

• HWRF Rocoto Workflow

HWRF ecFlow Workflow

The ecFlow system is an ECMWF-developed workflow system used by NCEP Central Operations (NCO) to run the operational HWRF. It is far less flexible than the Rocoto system, and is designed specifically to run the operational configuration in real-time. With manual tweaking, one can run retrospective cases, and this is done as part of the yearly test before HWRF is upgraded.

At present, it is not possible for users to use this system. However, once the NCEP Environmental Modeling Center is given permission to use ecFlow on WCOSS, we plan on setting up a more flexible system.

The ecFlow-based system is in the following locations:

• ecf/HWRF_2015_ECDEF.def — job dependencies and definitions of various jobs
• *.ecf files in ecf/ — job cards for submission to bsub on the WCOSS operational machine
• jobs/JHWRF.in, ush/hwrf_make_jobs.py — used to make the so-called J-Job scripts
• jobs/JHWRF_* — files generated by the hwrf_make_jobs script

For more details on this system:

• NCO SPA Guide

HWRF Script Wrappers

This is not really a Workflow Layer, but is instead a method to avoid having a Workflow Layer. It is a set of scripts that makes it easy to directly call the scripts. This is ideal for a user who just wants to run a single forecast cycle. It is also a good last resort if ecFlow and Rocoto are unavailable.

The wrappers are discussed in the HWRF Users Guide:

• http://www.dtcenter.org/HurrWRF/users/docs/

The HHS and kick_scripts

This is the old system the Rocoto-based workflow replaces. This system no longer exists, but is discussed here for historical purposes. It had a number of advantages over the other workflow systems described above, but a few key disadvantages that made it unusable for the current system.

The kick_scripts-based Workflow Layer consisted of three parts:

• emcsub — a wrapper around the local system job submission program (bsub, qsub, msub, ...) This implemented a system-independent way of describing job resource requirements. The emcsub would translate the requirements to the local system job card syntax, and submit the job.
• kick_scripts — a bunch of POSIX sh programs, one per batch job, each of which connects indirectly to the ex-scripts (see scripts). Each batch job runs one or more ex-scripts, and then submits the next kick_script in the chain.
• kick_scripts/hwrf_driver.sh — The driver script. This script submits the first kick_script based on configuration options set in the driver.

Its usage was simple:

cd kick_scripts
./hwrf_driver.sh 12L 2005092318 HISTORY

Running the hwrf_driver.sh submitted the first job. That job then ran its piece of the workflow, and submitted the second job. The second job did its work and submitted the third job. The third submitted the fourth, the forth submitted the fifth, and so on. This method has advantages over the Rocoto and ecFlow workflow management methods:

• Jobs can be automatically generated on the fly. With Rocoto and ecFlow, all possible jobs have to be described in advance.
• No CRON nor server process is necessary since there is no central workflow manager.

However, these kick_scripts only work on a single cycle at a time. To run multiple cycles, one must use the HHS. The HHS simply did exactly what a human user would do. It ran the hwrf_driver.sh for the first cycle, waited for the 12hr forecast to be ready, and then submitted the next cycle. Usage was simple:

cd kick_scripts
cp /path/to/hwrf_hhs_driver.sh .
./hwrf_hhs_driver.sh 12L 2005 HISTORY

The HHS driver submitted an automatically-resubmitting batch job that ran and monitored the HWRF.

The HHS had superior monitoring and safeguards, compared to the other workflows we use now. In particular:

• HHS could detect other workflows attempting to run in the same place
• HHS did archive verification
• HHS stopped submitting any jobs if any unexpected failure was encountered (this is both good and bad)
• HHS did not require CRON nor a server (it is an automatically-resubmitting batch job)
• HHS had disk space and archive space monitoring capabilities, and would not submit new cycles if specified limits were met.

If those workflow management systems had so many advantages, why did we abandon them?

• If a batch job submits the next batch job, several things are not possible:
  • A batch job cannot have multiple job dependencies.
  • A batch job cannot have a time or data dependency.
  • Resubmissions of failed jobs.
• The method of having jobs submit jobs was unreliable since these submissions often failed:
  • Compute nodes don't have the ability to submit jobs on all clusters.
  • Even if compute nodes can submit jobs, they are generally less reliable at doing so than the front-end nodes.
  • If the batch system goes down for, say, two hours, you will hit your wallclock limit before you can submit the next job.
  • If a job is "lost" (submitted, queued, but never runs) then the entire system breaks.
• Combining the Workflow Layer and Intercycle Layer into one workflow management system has advantages.
  • Dependencies can be more specific.
  • More advanced throttling options are available.

The other advantages of the HHS and kick_scripts can eventually be added to the Rocoto-based system. However, the disadvantages of having jobs submit jobs was the key to the inevitable demise of the HHS+kick_scripts system.