|
HWRF
trunk@4391
|
An hwrf.hwrftask.HWRFTask that performs regribbing operations. More...
An hwrf.hwrftask.HWRFTask that performs regribbing operations.
An HWRFTask that runs regribbing for a list of input/output times. This class keeps track of many different grids and grib files, each of which is given a name. The regribbing operations are specified in an hwrf.regrib.RegribMany object. It is possible for multiple jobs to run the same GRIBTask at the same time: the task uses lock files and a produtil.datastore.Datastore to communicate between them.
Definition at line 46 of file gribtask.py.
Public Member Functions | |
| def | __init__ (self, dstore, conf, section, regribmany, start, end, step, atime=None, kwargs) |
| An hwrf.hwrftask.HWRFTask that performs regribbing operations. More... | |
| def | make_time_array (self) |
| Create a time-indexed array of None. More... | |
| def | make_time_array_list (self) |
| Create an array of times to be processed. More... | |
| def | get_data (self, name, time=None, update=False, kwargs) |
| Returns intermediate results. More... | |
| def | set_data (self, name, data, time=None, logger=None, kwargs) |
| Stores intermediate results. More... | |
| def | make_products (self) |
| Creates Product objects. More... | |
| def | products |
| Iterates over products: More... | |
| def | uncomplete (self) |
| Marks all tasks as unstarted and products as unavailable. More... | |
| def | call_completed_callbacks (self) |
| Calls the Product.call_callback function for all completed and delivered products. More... | |
| def | unrun (self) |
| Deletes all output products, and marks this task and all subtasks as incomplete. More... | |
| def | run_helper |
| Internal function that underlies run() and runpart() More... | |
| def | run (self, kwargs) |
| Performs all regribbing, returning when complete. More... | |
| def | runpart (self, kwargs) |
| Performs a small amount of work and returns. More... | |
 Public Member Functions inherited from hwrf.hwrftask.HWRFTask | |
| def | __init__ (self, dstore, conf, section, taskname=None, workdir=None, outdir=None, storminfo=UNSPECIFIED, taskvars=UNSPECIFIED, kwargs) |
| Creates an HWRFTask. More... | |
| def | isfakestorm (self) |
| def | ismultistorm (self) |
| def | get_workdir (self) |
| Returns the directory the class should work in, as set by the "workdir" metadata value. More... | |
| def | set_workdir (self, val) |
| Sets the directory the class should work in. More... | |
| def | get_outdir (self) |
| Gets the directory that should receive output data. More... | |
| def | set_outdir (self, val) |
| Sets the directory that should receive output data. More... | |
| def | realtime (self) |
| Is this job a real-time forecast job? More... | |
| def | redirect (self) |
| Should subprograms' outputs be redirected to separate files? | |
| def | scrub (self) |
| Should temporary files be deleted as soon as they are not needed? | |
| def | tvset (self, opt, val) |
| Sets a taskvar option's value. More... | |
| def | tvdel (self, opt) |
| Deletes an object-local value set by tvset. More... | |
| def | tvget (self, opt) |
| Gets a taskvar's value. More... | |
| def | tvhave |
| Is a taskvar set? More... | |
| def | taskvars (self) |
| The dict of object-local values used for string substitution. More... | |
| def | confint |
| Alias for self.conf.getint for section self.section. More... | |
| def | confstr |
| Alias for self.conf.getstr for section self.section. More... | |
| def | conffloat |
| Alias for self.conf.getfloat for section self.section. More... | |
| def | confbool |
| Alias for self.conf.getbool for section self.section. More... | |
| def | confget |
| Alias for self.conf.get for section self.section. More... | |
| def | confitems |
| Alias for self.conf.items for section self.section. More... | |
| def | confstrinterp (self, string, section=None, kwargs) |
| Alias for self.icstr for backward compatibility. More... | |
| def | conftimestrinterp (self, string, ftime, atime=None, section=None, kwargs) |
| Alias for self.timestr for backward comaptibility. More... | |
| def | confraw |
| Get a raw configuration value before string expansion. More... | |
| def | icstr (self, string, section=None, kwargs) |
| Expands a string in the given conf section. More... | |
| def | timestr (self, string, ftime, atime=None, section=None, kwargs) |
| Expands a string in the given conf section, including time vars. More... | |
| def | getdir |
| Alias for hwrf.config.HWRFConfig.get() for the "dir" section. More... | |
| def | getexe |
| Alias for hwrf.config.HWRFConfig.get() for the "exe" section. More... | |
| def | getconf (self) |
| Returns this HWRFTask's hwrf.config.HWRFConfig object. More... | |
| def | getsection (self) |
| Returns this HWRFTask's section name in the HWRFConfig. More... | |
| def | log |
| Obtain a logging domain. More... | |
| def | inputiter (self) |
| Iterates over all inputs required by this task. More... | |
| Public Member Functions inherited from produtil.datastore.Task | |
| def | __init__ (self, dstore, taskname, logger=None, kwargs) |
| Task constructor. More... | |
| def | jlogfile (self) |
| returns the jlogfile logger. More... | |
| def | postmsg (self, message, args, kwargs) |
| same as produtil.log.jlogger.info() More... | |
| def | setstate (self, val) |
| Sets the state of this job. More... | |
| def | getstate (self) |
| Returns the job state. More... | |
| def | strstate (self) |
| A string representation of the job state. More... | |
| def | gettaskname (self) |
| Returns the task name part of the database ID, which is the same as the prodname. More... | |
| def | products (self, args, kwargs) |
| Iterate over the products this task produces. More... | |
| def | log (self) |
| Returns the logger object for this task. More... | |
| def | clean (self) |
| Cleans up any unneeded data used by this task. More... | |
| def | unrun (self) |
| Undoes the effect of run(). More... | |
| def | run (self) |
| Performs the work this Task should do and generates all products. More... | |
| def | is_completed (self) |
| Is this task complete? More... | |
| def | completed (self) |
| Read-only property: is this task completed? Same as is_completed() More... | |
| def | runpart (self) |
| Run some of this task's work, deliver some products. More... | |
| Public Member Functions inherited from produtil.datastore.Datum | |
| def | __init__ (self, dstore, prodname, category, meta=None, cache=30, location=None, kwargs) |
| Datum constructor. More... | |
| def | __enter__ (self) |
| Acquires this object's thread lock. More... | |
| def | __exit__ (self, etype, evalue, traceback) |
| Releases this object's thread lock. More... | |
| def | validate (self) |
| Validates this object's Datastore, prodname and category. More... | |
| def | getid (self) |
| Returns the database ID of this datum. More... | |
| def | getdatastore (self) |
| Returns the datastore of this datum. More... | |
| def | transaction (self) |
| Creates, but does not lock, a Transaction for this datum's datastore. More... | |
| def | getprodtype (self) |
| Returns the product type of this Datum. More... | |
| def | getprodname (self) |
| Returns the product name part of the database ID. More... | |
| def | getcategory (self) |
| Returns the product category part of the database ID. More... | |
| def | getlocation (self) |
| Returns the "location" field of this Datum's database entry. More... | |
| def | setlocation (self, v) |
| Sets the "location" field of this Datum's database entry. More... | |
| def | __hash__ (self) |
| Integer hash function. More... | |
| def | __str__ (self) |
| Human-readable description of this Datum. More... | |
| def | __repr__ (self) |
| Python code-like description of this Datum. More... | |
| def | __cmp__ (self, other) |
| Compares two Datums' prodnames and categories. More... | |
| def | set_loc_avail (self, loc, avail) |
| Sets the location and availability of this Datum in a single transaction. More... | |
| def | update (self) |
| Discards all cached metadata and refreshes it from the Datastore. More... | |
| def | __getitem__ (self, k) |
| Returns the value of the specified metadata key or raises KeyError. More... | |
| def | meta |
| Return the value of a metadata key. More... | |
| def | get |
| Alias for self.meta() Returns the value of the specified metadata key or returns default if it is unset. More... | |
| def | __setitem__ (self, k, v) |
| Sets the value of the specified metadata key. More... | |
| def | __delitem__ (self, k) |
| Deletes the specified metadata key, which must not be "available" or "location". More... | |
| def | __contains__ (self, k) |
| Determines if a metadata key is set. More... | |
| def | iteritems (self) |
| Iterates over all metadata (key,value) pairs for this Datum, including "available" and "location". More... | |
Public Attributes | |
| start | |
| the start time of the simulation | |
| end | |
| the end time of the simulation | |
| timestep | |
| the frequency at which GRIB files should be created | |
| atime | |
| the "atime" argument to send to Product.product() functions | |
| workerdesc | |
| None (unused) | |
| state | |
| Public Attributes inherited from hwrf.hwrftask.HWRFTask | |
| storminfo | |
| The hwrf.storminfo.StormInfo describing the vitals information for the storm processed by this HWRFTask. More... | |
Additional Inherited Members | |
| Properties inherited from hwrf.hwrftask.HWRFTask | |
| workdir | |
| The directory in which this task should be run. More... | |
| outdir | |
| The directory in which this task should deliver its final output. More... | |
| conf | |
| This HWRFTask's hwrf.config.HWRFConfig object. More... | |
| section | |
| The confsection in self.section for this HWRFTask (read-only) More... | |
| Properties inherited from produtil.datastore.Task | |
| state | |
| Read-write property: the job state. More... | |
| taskname | |
| Read-only property: the name of this task. More... | |
| Properties inherited from produtil.datastore.Datum | |
| prodname = property(getprodname,None,None,) | |
| Read-only property, an alias for getprodname(): the product name part of the database ID. More... | |
| category = property(getcategory,None,None,) | |
| Read-only property, an alias for getcategory(), the category name part of the database ID. More... | |
| prodtype | |
| Read-only property, an alias for getprodtype(), the product type. More... | |
| did | |
| Read-only property, an alias for getid(). More... | |
| dstore | |
| Read-only property, an alias for getdatastore(), the Datastore in which this Datum resides. More... | |
| location | |
| Read-write property, an alias for getlocation() and setlocation(). More... | |
| def hwrf.gribtask.GRIBTask.__init__ | ( | self, | |
| dstore, | |||
| conf, | |||
| section, | |||
| regribmany, | |||
| start, | |||
| end, | |||
| step, | |||
atime = None, |
|||
| kwargs | |||
| ) |
An hwrf.hwrftask.HWRFTask that performs regribbing operations.
An HWRFTask that runs regribbing for a list of input/output times. This class keeps track of many different grids and grib files, each of which is given a name. The regribbing operations are specified in an hwrf.regrib.RegribMany object. It is possible for multiple jobs to run the same GRIBTask at the same time: the task uses lock files and a produtil.datastore.Datastore to communicate between them. Creates a new GRIBTask:
| dstore | the produtil.datastore.Datastore to use |
| conf | the HWRFConfig to use for configuration options. This conf is passed down to the RegribMany during regribbing operations. |
| section | the config section to use. |
| regribmany | the hwrf.regrib.RegribMany that describes the regribbing operations |
| start,end,step | the start and end times and timestep for the list of times to regrib. |
| atime | the analysis time of the simulation. This is needed by some of the regribbing operations. Note that this analysis time may differ from the conf.cycle, especially for data assimilation first guess jobs. |
| kwargs | passed to the HWRFTask constructor |
Definition at line 57 of file gribtask.py.
| def hwrf.gribtask.GRIBTask.call_completed_callbacks | ( | self | ) |
Calls the Product.call_callback function for all completed and delivered products.
Definition at line 397 of file gribtask.py.
Referenced by hwrf.tracker.TrackerTask.run().
| def hwrf.gribtask.GRIBTask.get_data | ( | self, | |
| name, | |||
time = None, |
|||
update = False, |
|||
| kwargs | |||
| ) |
Returns intermediate results.
Used by RegribMany to retrieve intermediate results. The extra update flag is not used by RegribMany, but rather used by GRIBTask.run() to force a recheck of the database to see if another GRIBTask has completed a regribbing operation by sending update=True.
| name | the name of the result type |
| time | the time at which the result is requested |
| update | if True, call update() on the result |
| kwargs | unused |
Definition at line 128 of file gribtask.py.
Referenced by hwrf.gribtask.GRIBTask.run_helper().
| def hwrf.gribtask.GRIBTask.make_products | ( | self | ) |
Creates Product objects.
Called automatically from run(), this creates Product objects for all output products. Note that this is potentially an expensive function if the gribtask has a large number of products.
Definition at line 196 of file gribtask.py.
| def hwrf.gribtask.GRIBTask.make_time_array | ( | self | ) |
Create a time-indexed array of None.
Creates a new hwrf.numerics.TimeArray that maps from list of times to be processed to None.
Definition at line 113 of file gribtask.py.
| def hwrf.gribtask.GRIBTask.make_time_array_list | ( | self | ) |
Create an array of times to be processed.
Returns a new hwrf.numerics.TimeArray for the list of times to be processed. Unlike make_time_array, this TimeArray will assume any times that don't have data assigned have an empty list().
Definition at line 119 of file gribtask.py.
| def hwrf.gribtask.GRIBTask.products | ( | self, | |
name = None, |
|||
time = None, |
|||
yieldtime = False |
|||
| ) |
Iterates over products:
Loops over all products that meet the specifications, yielding each one in turn.
| name | the name of the products (same name as in the RegribMany). |
| time | the forecast time of the product |
| yieldtime | if True, instead of iterating over products, this will iterate over tuples containing the time and product. |
Definition at line 274 of file gribtask.py.
Referenced by hwrf.wrf.ExternalWRFTask.__init__(), hwrf.tracker.TrackerTask.call_completed_callbacks(), hwrf.wps.WPSTask.deliver_products(), hwrf.relocate.RelocationTask.deliver_products(), hwrf.wps.WPSTask.undeliver_products(), hwrf.prep.PrepHybrid.unrun(), hwrf.post.PostManyWRF.unrun(), hwrf.wrf.ExternalWRFTask.unrun(), and hwrf.wrf.ExternalWRFTask.wrf_check().
| def hwrf.gribtask.GRIBTask.run | ( | self, | |
| kwargs | |||
| ) |
Performs all regribbing, returning when complete.
Runs all regribbing. Does not return until all regribbing is complete, or a fatal error happens. It is safe to run this in multiple threads at the same time. Through file locking and database usage, the jobs will work together.
| kwargs | keyword arguments passed to run_helper() |
Definition at line 617 of file gribtask.py.
Referenced by produtil.datastore.Task.runpart().
| def hwrf.gribtask.GRIBTask.run_helper | ( | self, | |
one = False, |
|||
worker = None, |
|||
raiseall = False, |
|||
now = False |
|||
| ) |
Internal function that underlies run() and runpart()
Called from run() and runpart() to perform the actual work.
| one | True for runpart(), False for run(). If True, exits after one product is processed. |
| worker | obsolete and ignored |
| raiseall | If true, nearly all exceptions are raised. |
| now | If true, the function will not sleep or wait. |
Definition at line 443 of file gribtask.py.
Referenced by hwrf.gribtask.GRIBTask.run(), and hwrf.gribtask.GRIBTask.runpart().
| def hwrf.gribtask.GRIBTask.runpart | ( | self, | |
| kwargs | |||
| ) |
Performs a small amount of work and returns.
| kwargs | keyword arguments passed to run_helper() |
Definition at line 627 of file gribtask.py.
| def hwrf.gribtask.GRIBTask.set_data | ( | self, | |
| name, | |||
| data, | |||
time = None, |
|||
logger = None, |
|||
| kwargs | |||
| ) |
Stores intermediate results.
Used by RegribMany to store intermediate results. Will also deliver any output products.
| name | the name of the result being set |
| data | the data, any implementation-defined object. If this is a product that is delivered, it is sent to the frominfo= argument of Product.deliver() |
| time | the time at which the result is set |
| logger | a logging.Logger to use for logging messages |
| kwargs | ignored |
Definition at line 156 of file gribtask.py.
| def hwrf.gribtask.GRIBTask.uncomplete | ( | self | ) |
Marks all tasks as unstarted and products as unavailable.
Marks this task and all subtasks as incomplete, ensuring that the next call to run or runpart will attempt to produce any products that are not delivered. All products that are "available" but are not on disk are marked as unavailable.
Definition at line 363 of file gribtask.py.
Referenced by hwrf.post.PostManyWRF.unrun().
| def hwrf.gribtask.GRIBTask.unrun | ( | self | ) |
Deletes all output products, and marks this task and all subtasks as incomplete.
Definition at line 430 of file gribtask.py.
1.8.10
