|
HWRF
trunk@4391
|
Runs regrib operations on many input times, sending output to an hwrf.gribtask.GRIBTask. More...
Runs regrib operations on many input times, sending output to an hwrf.gribtask.GRIBTask.
This class keeps track of a set of named regribbing operations and target grids. Each operation may connect to another by name, allowing the output of one operation to be the input to one or more other operations. The RegribMany.make can then produce the output of an operation, on the condition that its inputs from earlier operations are ready.
The RegribMany is not intended to be used alone. Instead, the GRIBTask class in hwrf.gribtask should be used to execute RegribMany tasks. You can run multiple GRIBTasks at the same time, on the same RegribMany outputs, and the GRIBTasks will automatically work together via file locking and a database to produce the outputs in parallel, as fast as possible.
Here is a simple example of the use of a RegribMany object:
Now lets step through that and explain what it does.
First, we create the RegribMany object. We're actually doing this in a sloppy way: it will search the $PATH for cnvgrib, wgrib, and other programs. It is better to specify the exact location. That can be done by passing arguments to the RegribMany constructor. See the RegribBase instructor for a list.
This creates a pair of GRIB1Selectors, operations recognized by RegribMany. A GRIB1Selector takes a Task, in this case mypost, and calls its "products" function to get a Product as input to the next step in a regribbing operation. The domain=d01 is passed into products, as is the time=X for each time X the RegribMany processes.
This instructs the RegribMany to add an operation to its dictionary. The operation is called "cgrid", and consists of a GRIBGridCompute created by the clatlon function, defined later in this module. The clatlon takes our getd02, and creates a 30x30 degree, 600x600 gridpoint, 0.05 degree grid, centered on that domain at each time. If domain d02 moves, this clatlon output will be a moving domain.
This adds another operation to the RegribMany's dictionary, this time a grib output operation called cGRIB1. The r.grid('cgrid') looks up the cgrid operation we just inserted, connecting that to the cGRIB1. Recall that getd01 and getd02 are also operations, and produce a GRIB1 file. When we multiply that by a grid, it creates a new operation that will use copygb to output a new GRIB1 file on that grid. Hence, we have created two temporary files:
When we add those two together, it pastes the second one on top of the first via a copygb merge operation. That result is then stored as cGRIB1.
This adds a third operation to the RegribMany, called cGRIB2. The r.GRIB('cGRIB1') looks up the operation that creates cGRIB1, connecting that to the new cGRIB2. When we multiply by the special constant GRIB2, it creates a new operation to convert that file to GRIB2.
This sets the delivery location for the cGRIB2 outputs. It will create files with names like:
Without that command, the output will go to automatically-generated locations somewhere in intercom, which is likely not what you want.
This creates a GRIBTask that will execute the regribber we just made.
The GRIBTask is configured to be able to run our RegribMany for all forecast times starting at the analysis time (conf.cycle) and running to 126 hours afterwards (126*3600), every six hours (6*3600). The GRIBTask will set the time= keyword argument to mypost.products, allowing each RegribMany operation to run on the correct input time.
This tells the gribber to run to completion, generating our output files.
Public Member Functions | |
| def | __init__ (self, _copy=None, workprefix=None, kwargs) |
| Creats a new RegribMany. More... | |
| def | __enter__ (self) |
| Has no effect. More... | |
| def | __exit__ (self, etype, evalue, traceback) |
| Has no effect. More... | |
| def | has_name (self, name) |
| Returns True if an operation by the given name exists. More... | |
| def | reset (self) |
| Erases all cached results. More... | |
| def | copy (self) |
| Returns a copy of self with no cached results. More... | |
| def | GRIB (self, name) |
| Returns a GRIB1Fetcher object that will grab the GRIB1 data with the given name. More... | |
| def | nonGRIBOps (self) |
| Iterates over all non-GRIBOp operations yielding tuples containing the name and operation. More... | |
| def | GRIBOps (self) |
| Iterates over all GRIBOp operations yielding tuples containing the name and operation. More... | |
| def | is_grid (self, name) |
| Returns True if the name corresponds to a grid. More... | |
| def | grid (self, name) |
| Returns a GRIBGridFetcher object that will grab the grid structure with the given name. More... | |
| def | subset (self, name) |
| Returns a GRIBSubsetterFetcher object that will grab the GRIB subsetter with the given name. More... | |
| def | add (self, name, arg, kwargs) |
| Adds the given generator "arg" under the specified name (in "name") to this object. More... | |
| def | to_com |
| Requests delivery to the com directory in file "location" of operation "name". More... | |
| def | to_intercom |
| Requests delivery to the intercom directory in file "location" of operation "name". More... | |
| def | deliveries |
| Iterates over all deliveries yielding a six element tuple. More... | |
| def | has_deliveries (self) |
| Returns True if there are any requested deliveries and False otherwise. More... | |
| def | has_data (self, name, kwargs) |
| Returns True if there is a cached result for the specified operation. More... | |
| def | is_ready (self, name, kwargs) |
| Determines if the specified operation is ready. More... | |
| def | input_valid (self, name, kwargs) |
| Determines if it is possible to run the specified operation with the given keyword arguments. More... | |
| def | make (self, name, kwargs) |
| Executes the operation previously stored in self.add(), passing it the given keyword arguments, returning the result. More... | |
| def | names (self) |
| Iterates over the names provided to self.add(name,...) in the order they were added. More... | |
 Public Member Functions inherited from hwrf.regrib.RegribBase | |
| def | __init__ |
| Creates a RegribBase. More... | |
| def | getcopygb (self) |
| Returns the copygb ImmutableRunner. | |
| def | getcnvgrib_g12 (self) |
| Returns the cnvgrib -p32 -g12 command, which should be an ImmutableRunner. | |
| def | getcnvgrib_g21 (self) |
| Returns the cnvgrib -g21 command, which should be an ImmutableRunner. More... | |
| def | getwgrib (self) |
| Returns the wgrib command, which should be an ImmutableRunner. More... | |
| def | getlog (self) |
| Returns the logger.Logging object to use for logging messages, or None if no logger was provided to the constructor. More... | |
| def | setlog (self, val) |
| Sets the logger.Logging object to use for logging messages. More... | |
| def | getprodargs (self) |
| Returns the array of arguments to send to the products iterators. More... | |
| def | getprodkwargs (self) |
| Returns a dict of keyword arguments to send to the products iterators. More... | |
| def | setsync_frequently (self, flag) |
| Sets the sync_frequently flag. More... | |
| def | getsync_frequently (self) |
| Returns the value of the sync_frequently flag. More... | |
Additional Inherited Members | |
| Properties inherited from hwrf.regrib.RegribBase | |
| sync_frequently | |
| Boolean flag: should we call sync frequently? More... | |
| logger | |
| A logging.Logger for log messages. More... | |
| prodargs | |
| A list of additional positional arguments to send to Task.products. More... | |
| prodkwargs | |
| A dict of additional keyword arguments to send to Task.products. More... | |
| wgrib | |
| A produtil.prog.ImmutableRunner for running wgrib, or None if unavailable. More... | |
| copygb | |
| A produtil.prog.ImmutableRunner for running copygb, or None if unavailable. More... | |
| cnvgrib_g21 | |
| A produtil.prog.ImmutableRunner for running cnvgrib to convert from GRIB2 to GRIB1, or None if unavailable. More... | |
| cnvgrib_g12 | |
| A produtil.prog.ImmutableRunner for running cnvgrib to convert from GRIB1 to GRIB2, or None if unavailable. More... | |
| def hwrf.regrib.RegribMany.__init__ | ( | self, | |
_copy = None, |
|||
workprefix = None, |
|||
| kwargs | |||
| ) |
Creats a new RegribMany.
| workprefix | The workprefix, if specified, sets the location in which to run. The _copy is used by the self.copy() for copying. |
| _copy,kwargs | All other arguments are passed to the superclass RegribBase.__init__ constructor . |
| def hwrf.regrib.RegribMany.__enter__ | ( | self | ) |
Has no effect.
This is for forward compatibility to a later thread-safe implementation.
| def hwrf.regrib.RegribMany.__exit__ | ( | self, | |
| etype, | |||
| evalue, | |||
| traceback | |||
| ) |
Has no effect.
This is for forward compatibility to a later thread-safe implementation.
| etype,evalue,traceback | Exception information. |
| def hwrf.regrib.RegribMany.add | ( | self, | |
| name, | |||
| arg, | |||
| kwargs | |||
| ) |
Adds the given generator "arg" under the specified name (in "name") to this object.
| name | the operation name to query |
| arg | A object of a subclass of GRIBBase |
| kwargs | Stored for later passing to function calls to arg |
Definition at line 548 of file regrib.py.
Referenced by produtil.fileop.FileWaiter.add().
| def hwrf.regrib.RegribMany.copy | ( | self | ) |
Returns a copy of self with no cached results.
Definition at line 510 of file regrib.py.
Referenced by produtil.mpiprog.MPIRanksSPMD.__add__(), hwrf.storminfo.StormInfo.__add__(), produtil.mpiprog.MPIRank.__add__(), produtil.mpiprog.MPIRank.__getitem__(), produtil.prog.ImmutableRunner.runner(), hwrf.wrf.WRFSimulation.swcorner_dynamic(), and hwrf.wrf.WRFSimulation.swcorner_dynamic_multistorm().
| def hwrf.regrib.RegribMany.deliveries | ( | self, | |
name = None |
|||
| ) |
Iterates over all deliveries yielding a six element tuple.
| name | The name of the operation of interest. The tuple contents:
|
| def hwrf.regrib.RegribMany.GRIB | ( | self, | |
| name | |||
| ) |
Returns a GRIB1Fetcher object that will grab the GRIB1 data with the given name.
| name | the operation name to query |
| def hwrf.regrib.RegribMany.GRIBOps | ( | self | ) |
| def hwrf.regrib.RegribMany.grid | ( | self, | |
| name | |||
| ) |
Returns a GRIBGridFetcher object that will grab the grid structure with the given name.
| name | the operation name to query |
| def hwrf.regrib.RegribMany.has_data | ( | self, | |
| name, | |||
| kwargs | |||
| ) |
Returns True if there is a cached result for the specified operation.
| kwargs | The keyword arguments are passed on to the get_data routine for the caching object (usually an hwrf.gribtask.GRIBTask). |
| name | The operation name. |
| def hwrf.regrib.RegribMany.has_deliveries | ( | self | ) |
Returns True if there are any requested deliveries and False otherwise.
| def hwrf.regrib.RegribMany.has_name | ( | self, | |
| name | |||
| ) |
| def hwrf.regrib.RegribMany.input_valid | ( | self, | |
| name, | |||
| kwargs | |||
| ) |
Determines if it is possible to run the specified operation with the given keyword arguments.
| name | Obtains the operation by the specified name. |
| kwargs | The input_valid function is called on the generator for the named operation, passing these keyword arguments. |
| def hwrf.regrib.RegribMany.is_grid | ( | self, | |
| name | |||
| ) |
| def hwrf.regrib.RegribMany.is_ready | ( | self, | |
| name, | |||
| kwargs | |||
| ) |
Determines if the specified operation is ready.
| name | Obtains the operation by the specified name. |
| kwargs | The is_ready funciton is called on the named operation's generator, passing the given keyword arguments to it. |
Definition at line 641 of file regrib.py.
Referenced by hwrf.regrib.RegribMany.has_data().
| def hwrf.regrib.RegribMany.make | ( | self, | |
| name, | |||
| kwargs | |||
| ) |
Executes the operation previously stored in self.add(), passing it the given keyword arguments, returning the result.
If a cache is available (usually a GRIBTask), stores the result in that cache.
| name | the name of the operation of interest |
| kwargs | Passed to the make function in the generator for the named operation |
| def hwrf.regrib.RegribMany.names | ( | self | ) |
| def hwrf.regrib.RegribMany.nonGRIBOps | ( | self | ) |
| def hwrf.regrib.RegribMany.reset | ( | self | ) |
| def hwrf.regrib.RegribMany.subset | ( | self, | |
| name | |||
| ) |
Returns a GRIBSubsetterFetcher object that will grab the GRIB subsetter with the given name.
| name | the operation name to query |
| def hwrf.regrib.RegribMany.to_com | ( | self, | |
| location, | |||
| name, | |||
category = None, |
|||
prodname = None, |
|||
keep = True |
|||
| ) |
Requests delivery to the com directory in file "location" of operation "name".
| category,prodname | The category and prodname are used to create the FileProduct object. |
| keep | Sent to the final deliver_file operation. It should be left as True. |
| name | the operation's name |
| location | the destination, which will be sent through hwrf.config.HWRFConfig.conftimestrinterp() by the hwrf.gribtask.GRIBTask |
| def hwrf.regrib.RegribMany.to_intercom | ( | self, | |
| location, | |||
| name, | |||
category = None, |
|||
prodname = None, |
|||
keep = True |
|||
| ) |
Requests delivery to the intercom directory in file "location" of operation "name".
| category,prodname | The category and prodname are used to create the FileProduct object. |
| keep | Sent to the final deliver_file operation. It should be left as True. |
| name | the operation's name |
| location | the destination, which will be sent through hwrf.config.HWRFConfig.conftimestrinterp() by the hwrf.gribtask.GRIBTask |
1.8.10
