|
HWRF
trunk@4391
|
Stores products and tasks in an sqlite3 database file. More...
Stores products and tasks in an sqlite3 database file.
This module maintains an sqlite3 database file that stores information about Products and Tasks. A Product is a file or group of files created by some Task. Both Product and Task classes derive from Datum, which is the base class of anything that can be stored in the Datastore.
Classes | |
| class | CallbackExceptions |
| Exception raised when a Product class encounters exceptions while calling its callback functions in Product.call_callbacks. More... | |
| class | Datastore |
| Stores information about Datum objects in a database. More... | |
| class | Datum |
| Superclass of anything that can be stored in a Datastore. More... | |
| class | DatumException |
| Superclass of all exceptions local to produtil.datastore. More... | |
| class | DatumLockHeld |
| Raised when a LockDatum is held by another Worker. More... | |
| class | FakeException |
| This is a fake exception used to get a stack trace. More... | |
| class | FileProduct |
| A subclass of Product that represents file delivery. More... | |
| class | InvalidID |
| Raised when a Datum or subclass receives a prodname or category name that is invalid. More... | |
| class | InvalidOperation |
| Raised when an invalid Datum operation is requested, such as delivering an UpstreamProduct. More... | |
| class | Product |
| A piece of data produced by a Task. More... | |
| class | Task |
| Represents a process or actor that makes a Product. More... | |
| class | Transaction |
| Datastore transaction support. More... | |
| class | UnknownLocation |
| Raised when delivering data, but no location is provided. More... | |
| class | UpstreamFile |
| Represents a Product created by an external workflow. More... | |
Functions | |
| def | wait_for_products |
| Waits for products to be available and performs an action on them. More... | |
Variables | |
| list | __all__ |
| Symbols exported by "from produtil.datastore import *". More... | |
| tuple | _has_dcolon = re.compile('\A.*::.*\Z') |
| Regular expression to detect a database ID with a double colon in it. More... | |
| tuple | _has_dstar = re.compile('\A.*\*\*.*\Z') |
| Regular expression to detect a database ID with a double asterisk in it. More... | |
| string | TASK_CATEGORY = '**task**' |
| Special product category used for Tasks. More... | |
| int | FAILED = -10 |
| Constant used for Task.state to indicate a run was attempted but failed. More... | |
| int | UNSTARTED = 0 |
| Constant used for Task.state to indicate no attempt was made to run. More... | |
| int | RUNNING = 10 |
| Constant used for Task.state to indicate the task is presently running. More... | |
| int | PARTIAL = 20 |
| Constant used for Task.state to indicate the task was attempted but exited prematurely. More... | |
| int | COMPLETED = 30 |
| Constant used for Task.state to indicate the task completed successfully. More... | |
| def produtil.datastore.wait_for_products | ( | plist, | |
| logger, | |||
renamer = None, |
|||
action = None, |
|||
renamer_args = None, |
|||
action_args = None, |
|||
sleeptime = 20, |
|||
maxtime = 1800 |
|||
| ) |
Waits for products to be available and performs an action on them.
Waits for a specified list of products to be available, and performs some action on each product when it becomes available. Sleeps sleeptime seconds between checks. Returns the number of products that were found before the maxtime was reached.
| plist | A Product or a list of Product objects. |
| logger | A logging.Logger object in which to log messages. |
| renamer | Optional: a function or callable object that provides a new name for each product. This is passed the product, the logger and the contents of *renamer_args. Default: os.path.basename(p.location) |
| action | Optional: an action to perform on each product. This is passed the product, the output of renamer, the logger and the contents of *action_args. Default: perform no action. |
| renamer_args | Optional: arguments to renamer. |
| action_args | Optional: arguments to action. |
| sleeptime | - after checking availability of all products, if at least one is unavailable, the code will sleep for this much time before rechecking. Will be overridden by 0.01 if it is set to something lower than that. Default: 20 |
| maxtime | - maximum amount of time to spend in this routine before giving up. |
Definition at line 979 of file datastore.py.
Referenced by hwrf.finalmergetask.FinalMergeTask.copy_inputs(), hwrf.relocate.RelocationTask.copy_inputs(), hwrf.gsi.GSIBase.copy_wrf_inout(), hwrf.gsi.FGATGSI.copy_wrf_inout(), hwrf.relocate.RelocationTask.create_atcf(), produtil.datastore.UpstreamFile.deliver(), and hwrf.gsi.GSIBase.grab_wrf_enkf().
| produtil.datastore.__all__ |
Symbols exported by "from produtil.datastore import *".
Definition at line 15 of file datastore.py.
| produtil.datastore._has_dcolon = re.compile('\A.*::.*\Z') |
Regular expression to detect a database ID with a double colon in it.
Definition at line 87 of file datastore.py.
| produtil.datastore._has_dstar = re.compile('\A.*\*\*.*\Z') |
Regular expression to detect a database ID with a double asterisk in it.
Definition at line 91 of file datastore.py.
| produtil.datastore.COMPLETED = 30 |
Constant used for Task.state to indicate the task completed successfully.
Definition at line 130 of file datastore.py.
| produtil.datastore.FAILED = -10 |
Constant used for Task.state to indicate a run was attempted but failed.
Definition at line 101 of file datastore.py.
| produtil.datastore.PARTIAL = 20 |
Constant used for Task.state to indicate the task was attempted but exited prematurely.
Practically speaking, there is no way to tell the difference between RUNNING and PARTIAL since the job cannot ensure that it resets the state before unexpectedly exiting.
Definition at line 122 of file datastore.py.
| produtil.datastore.RUNNING = 10 |
Constant used for Task.state to indicate the task is presently running.
Definition at line 113 of file datastore.py.
| produtil.datastore.TASK_CATEGORY = '**task**' |
Special product category used for Tasks.
Definition at line 95 of file datastore.py.
| produtil.datastore.UNSTARTED = 0 |
Constant used for Task.state to indicate no attempt was made to run.
Definition at line 107 of file datastore.py.
1.8.10
