hwrf.copywrf.WRFCopyTask Class Reference

wrf file delivery task More...

Detailed Description

wrf file delivery task

This is a Task that copies WRF input and output files from the WRF run directory to the COM directory.

Definition at line 67 of file copywrf.py.

Inheritance diagram for hwrf.copywrf.WRFCopyTask:

Public Member Functions
def	__init__ (self, dstore, conf, section, wrftask, out_prefix, kwargs)
	WRFCopyTask constructor. More...
def	ncks_path (self)
	Returns the path to ncks. More...
def	compression_copier
	creates and returns a compression_copier for deliver_file More...
def	decompression_copier (self, src)
	returns a decompression copier for deliver_file More...
def	comfile
	get the full path to a file More...
def	d_initial
	deliver a file generated before WRF starts More...
def	d_final
	deliver a file generated at the end of the WRF run More...
def	d_wrfprod
	deliver a file generated during the WRF simulation such as a history or restart file More...
def	run
	watch for files to show up, delivering them when they do More...
def	unrun (self)
	delete delivered files More...
def	runpart
	deliver one output file and return. More...
def	run_helper
	internal implementation of run() and runpart() More...
Public Member Functions inherited from hwrf.hwrftask.HWRFTask
def	__init__ (self, dstore, conf, section, taskname=None, workdir=None, outdir=None, storminfo=UNSPECIFIED, taskvars=UNSPECIFIED, kwargs)
	Creates an HWRFTask. More...
def	isfakestorm (self)
def	ismultistorm (self)
def	get_workdir (self)
	Returns the directory the class should work in, as set by the "workdir" metadata value. More...
def	set_workdir (self, val)
	Sets the directory the class should work in. More...
def	get_outdir (self)
	Gets the directory that should receive output data. More...
def	set_outdir (self, val)
	Sets the directory that should receive output data. More...
def	realtime (self)
	Is this job a real-time forecast job? More...
def	redirect (self)
	Should subprograms' outputs be redirected to separate files?
def	scrub (self)
	Should temporary files be deleted as soon as they are not needed?
def	tvset (self, opt, val)
	Sets a taskvar option's value. More...
def	tvdel (self, opt)
	Deletes an object-local value set by tvset. More...
def	tvget (self, opt)
	Gets a taskvar's value. More...
def	tvhave
	Is a taskvar set? More...
def	taskvars (self)
	The dict of object-local values used for string substitution. More...
def	confint
	Alias for self.conf.getint for section self.section. More...
def	confstr
	Alias for self.conf.getstr for section self.section. More...
def	conffloat
	Alias for self.conf.getfloat for section self.section. More...
def	confbool
	Alias for self.conf.getbool for section self.section. More...
def	confget
	Alias for self.conf.get for section self.section. More...
def	confitems
	Alias for self.conf.items for section self.section. More...
def	confstrinterp (self, string, section=None, kwargs)
	Alias for self.icstr for backward compatibility. More...
def	conftimestrinterp (self, string, ftime, atime=None, section=None, kwargs)
	Alias for self.timestr for backward comaptibility. More...
def	confraw
	Get a raw configuration value before string expansion. More...
def	icstr (self, string, section=None, kwargs)
	Expands a string in the given conf section. More...
def	timestr (self, string, ftime, atime=None, section=None, kwargs)
	Expands a string in the given conf section, including time vars. More...
def	getdir
	Alias for hwrf.config.HWRFConfig.get() for the "dir" section. More...
def	getexe
	Alias for hwrf.config.HWRFConfig.get() for the "exe" section. More...
def	getconf (self)
	Returns this HWRFTask's hwrf.config.HWRFConfig object. More...
def	getsection (self)
	Returns this HWRFTask's section name in the HWRFConfig. More...
def	log
	Obtain a logging domain. More...
def	inputiter (self)
	Iterates over all inputs required by this task. More...
Public Member Functions inherited from produtil.datastore.Task
def	__init__ (self, dstore, taskname, logger=None, kwargs)
	Task constructor. More...
def	jlogfile (self)
	returns the jlogfile logger. More...
def	postmsg (self, message, args, kwargs)
	same as produtil.log.jlogger.info() More...
def	setstate (self, val)
	Sets the state of this job. More...
def	getstate (self)
	Returns the job state. More...
def	strstate (self)
	A string representation of the job state. More...
def	gettaskname (self)
	Returns the task name part of the database ID, which is the same as the prodname. More...
def	products (self, args, kwargs)
	Iterate over the products this task produces. More...
def	log (self)
	Returns the logger object for this task. More...
def	clean (self)
	Cleans up any unneeded data used by this task. More...
def	unrun (self)
	Undoes the effect of run(). More...
def	run (self)
	Performs the work this Task should do and generates all products. More...
def	is_completed (self)
	Is this task complete? More...
def	completed (self)
	Read-only property: is this task completed? Same as is_completed() More...
def	runpart (self)
	Run some of this task's work, deliver some products. More...
Public Member Functions inherited from produtil.datastore.Datum
def	__init__ (self, dstore, prodname, category, meta=None, cache=30, location=None, kwargs)
	Datum constructor. More...
def	__enter__ (self)
	Acquires this object's thread lock. More...
def	__exit__ (self, etype, evalue, traceback)
	Releases this object's thread lock. More...
def	validate (self)
	Validates this object's Datastore, prodname and category. More...
def	getid (self)
	Returns the database ID of this datum. More...
def	getdatastore (self)
	Returns the datastore of this datum. More...
def	transaction (self)
	Creates, but does not lock, a Transaction for this datum's datastore. More...
def	getprodtype (self)
	Returns the product type of this Datum. More...
def	getprodname (self)
	Returns the product name part of the database ID. More...
def	getcategory (self)
	Returns the product category part of the database ID. More...
def	getlocation (self)
	Returns the "location" field of this Datum's database entry. More...
def	setlocation (self, v)
	Sets the "location" field of this Datum's database entry. More...
def	__hash__ (self)
	Integer hash function. More...
def	__str__ (self)
	Human-readable description of this Datum. More...
def	__repr__ (self)
	Python code-like description of this Datum. More...
def	__cmp__ (self, other)
	Compares two Datums' prodnames and categories. More...
def	set_loc_avail (self, loc, avail)
	Sets the location and availability of this Datum in a single transaction. More...
def	update (self)
	Discards all cached metadata and refreshes it from the Datastore. More...
def	__getitem__ (self, k)
	Returns the value of the specified metadata key or raises KeyError. More...
def	meta
	Return the value of a metadata key. More...
def	get
	Alias for self.meta() Returns the value of the specified metadata key or returns default if it is unset. More...
def	__setitem__ (self, k, v)
	Sets the value of the specified metadata key. More...
def	__delitem__ (self, k)
	Deletes the specified metadata key, which must not be "available" or "location". More...
def	__contains__ (self, k)
	Determines if a metadata key is set. More...
def	iteritems (self)
	Iterates over all metadata (key,value) pairs for this Datum, including "available" and "location". More...

Public Attributes
	out_prefix
	Prefix to prepend to output filenames after the com path. More...
	state
Public Attributes inherited from hwrf.hwrftask.HWRFTask
	storminfo
	The hwrf.storminfo.StormInfo describing the vitals information for the storm processed by this HWRFTask. More...

Protected Member Functions
def	deliver_group
	deliver files to COM More...

Additional Inherited Members
Properties inherited from hwrf.hwrftask.HWRFTask
	workdir
	The directory in which this task should be run. More...
	outdir
	The directory in which this task should deliver its final output. More...
	conf
	This HWRFTask's hwrf.config.HWRFConfig object. More...
	section
	The confsection in self.section for this HWRFTask (read-only) More...
Properties inherited from produtil.datastore.Task
	state
	Read-write property: the job state. More...
	taskname
	Read-only property: the name of this task. More...
Properties inherited from produtil.datastore.Datum
	prodname = property(getprodname,None,None,)
	Read-only property, an alias for getprodname(): the product name part of the database ID. More...
	category = property(getcategory,None,None,)
	Read-only property, an alias for getcategory(), the category name part of the database ID. More...
	prodtype
	Read-only property, an alias for getprodtype(), the product type. More...
	did
	Read-only property, an alias for getid(). More...
	dstore
	Read-only property, an alias for getdatastore(), the Datastore in which this Datum resides. More...
	location
	Read-write property, an alias for getlocation() and setlocation(). More...

Constructor & Destructor Documentation

def hwrf.copywrf.WRFCopyTask.__init__	(		self,
			dstore,
			conf,
			section,
			wrftask,
			out_prefix,
			kwargs
	)

WRFCopyTask constructor.

Constructor for the WRFCopyTask

Parameters

dstore	the produtil.datastore.Datastore
conf	the hwrf.config.HWRFConfig
section	the section to use in conf
wrftask	the task that runs WRF. This should be an hwrf.wrf.ExternalWRFTask, or a subclass of hwrf.fcsttask.WRFTaskBase
out_prefix	output file prefix, a string suitable for passing into hwrf.config.HWRFConfig.strinterp()
kwargs	passed to the superclass constructor

Definition at line 71 of file copywrf.py.

Member Function Documentation

def hwrf.copywrf.WRFCopyTask.comfile	(		self,
			orig,
			destname = None
	)

get the full path to a file

Generates a full path to the delivery location of the specified source file. Returns the full path and the basename in a tuple.

Returns

a tuple (path,basename) where the path is the full path to the file, and the basename is os.path.basename(path)

Parameters

orig	the original filename
destname	optional: the desired destination name format which will be sent through hwrf.hwrftask.HWRFTask.confstrinterp() to generate the final destination filename

Definition at line 192 of file copywrf.py.

Referenced by hwrf.multistorm.WRFCopyTaskMS.comfile(), hwrf.copywrf.WRFCopyTask.d_final(), and hwrf.copywrf.WRFCopyTask.d_wrfprod().

def hwrf.copywrf.WRFCopyTask.compression_copier	(		self,
			src,
			vsubset = None
	)

creates and returns a compression_copier for deliver_file

Returns the object that should be sent as the "copier" argument to produtil.fileop.deliver_file() to copy the given source file. This is either None, or a function that calls ncks to compress NetCDF files. If a vsubset argument is present, the file is subsetted, retaining only the variables vsubset (a comma separated list).

Parameters

src	the source file
vsubset	unused, but may one day be used to subset the file

Returns

None if the source file is not NetCDF. If it is NetCDF, then a copy(s,t,x) function is returned, suitable for passing to the copy argument of produtil.fileop.deliver_file()

Definition at line 135 of file copywrf.py.

Referenced by hwrf.copywrf.WRFCopyTask.deliver_group().

def hwrf.copywrf.WRFCopyTask.d_final	(		self,
			inprod,
			check = None,
			destname = None
	)

deliver a file generated at the end of the WRF run

Requests delivery of a file created by WRF that is not complete until the WRF exits. Examples of this are the wrfdiag, hifreq and patcf files. These files will be delivered when the underlying WRF Task has a state of COMPLETED. The optional "check" argument enables calling the potentially expensive "check" subroutine on the upstream product every time it is considered for delivery. If the input is a Product, and check=False, then only the "available" subroutine is called, which will not be updated unless another Task marks the product as available. The default is check=False for Products and check=True for filenames.

Parameters

inprod	the input produtil.datastore.Product
check	do we need to call the product's check() function? This is needed if it is a produtil.datastore.UpstreamFile
destname	optional: the destname argument to comfile() which will be used to generate the delivered filename in the COM directory.

Definition at line 247 of file copywrf.py.

def hwrf.copywrf.WRFCopyTask.d_initial	(		self,
			inprod,
			check = None,
			destname = None
	)

deliver a file generated before WRF starts

Requests delivery of a file that is created before the wrf.exe invocation. The "inprod" may be a Product or a filename relative to the WRF run directory. The optional "check" argument enables calling the potentially expensive "check" subroutine on the upstream product every time it is considered for delivery. If the input is a Product, and check=False, then only the "available" subroutine is called, which will not be updated unless another Task marks the product as available. The default is check=False for Products and check=True for filenames.

Parameters

inprod	the input produtil.datastore.Product
check	do we need to call the product's check() function? This is needed if it is a produtil.datastore.UpstreamFile
destname	optional: the destname argument to comfile() which will be used to generate the delivered filename in the COM directory.

Definition at line 227 of file copywrf.py.

def hwrf.copywrf.WRFCopyTask.d_wrfprod	(		self,
			product,
			check = False,
			destname = None
	)

deliver a file generated during the WRF simulation such as a history or restart file

Requests delivery of a WRF I/O subsystem output file. The "product" argument must be a Product object. The optional argument "check" enables calling the potentially expensive "check" subroutine on the product every time it is considered for delivery. If check=False, then only the "available" subroutine is called, which will not be updated unless another Task marks the product as available.

Parameters

product	the produtil.datastore.Product
check	do we need to call the product's check() function? This is needed if it is a produtil.datastore.UpstreamFile
destname	optional: the destname argument to comfile() which will be used to generate the delivered filename in the COM directory.

Definition at line 298 of file copywrf.py.

def hwrf.copywrf.WRFCopyTask.decompression_copier	(		self,
			src
	)

returns a decompression copier for deliver_file

Returns an object that has the reverse effect of self.compression_copier. This will uncompress files that compression_copier copier would compress. NetCDF files will all be converted to 64-bit indexing NetCDF 3 files.

Parameters

src	the source file

Returns

None if the source file is not NetCDF. If it is NetCDF, then a copy(s,t,x) function is returned, suitable for passing to the copy argument of produtil.fileop.deliver_file()

Definition at line 166 of file copywrf.py.

def hwrf.copywrf.WRFCopyTask.deliver_group (   self,   group,   check_all = False  )

protected

deliver files to COM

This is an internal implementation function. Do not call it directly. Takes a list of tuples containing an upstream product, a downstream product to deliver, and a boolean telling whether to check() the upstream product. Delivers available products. Returns a tuple containing two booleans: the first is True iff something was delivered, and the second is true iff something is left in the group that has not been delivered yet. The check_all argument can be used to force a check on all products by setting check_all=True.

Parameters

group	either self._initial, self._wrfprod or self._final
check_all	if True, run product.check() on all products

Definition at line 340 of file copywrf.py.

Referenced by hwrf.copywrf.WRFCopyTask.run_helper().

def hwrf.copywrf.WRFCopyTask.ncks_path

(

self

)

Returns the path to ncks.

Returns the path to the ncks program, used to convert between NetCDF 3 and compressed NetCDF 4 file formats. Returns None if ncks cannot be found. This function will only search for ncks once, and will cache the result. Set self._ncks_path=False to force a recheck.

Definition at line 118 of file copywrf.py.

Referenced by hwrf.copywrf.WRFCopyTask.compression_copier(), and hwrf.copywrf.WRFCopyTask.decompression_copier().

def hwrf.copywrf.WRFCopyTask.run	(		self,
			check_all = False
	)

watch for files to show up, delivering them when they do

Keeps watching for WRF files to show up, copying them when they do. This is just a simple wrapper around self.runpart, and does not return until runpart sets the state to something other than RUNNING.

Parameters

check_all

if True, all non-delivered products have product.check() called on them

Definition at line 324 of file copywrf.py.

Referenced by produtil.datastore.Task.runpart().

def hwrf.copywrf.WRFCopyTask.run_helper	(		self,
			runpart,
			check_all = False
	)

internal implementation of run() and runpart()

This is the internal implementation of run and runpart. It delivers files, and returns False if all files are delivered.

Parameters

runpart	If runpart=True, run_helper() will return immediately after delivering one file.
check_all	Optional. If True, forces a call to check() on all undelivered products, even if those products are not checked by default.

Definition at line 417 of file copywrf.py.

Referenced by hwrf.copywrf.WRFCopyTask.run(), hwrf.gribtask.GRIBTask.run(), hwrf.copywrf.WRFCopyTask.runpart(), and hwrf.gribtask.GRIBTask.runpart().

def hwrf.copywrf.WRFCopyTask.runpart	(		self,
			check_all = False
	)

deliver one output file and return.

Delivers one output file and returns. Sets the state to COMPLETED if all files are delivered.

Parameters

check_all

Optional. If True, forces a call to check() on all undelivered products, even if those products are not checked by default.

Definition at line 409 of file copywrf.py.

def hwrf.copywrf.WRFCopyTask.unrun

(

self

)

delete delivered files

Calls the undeliver function on all products, deleting them from the destination. Product objects' undeliver() functions are called to achieve this.

Definition at line 400 of file copywrf.py.

Member Data Documentation

hwrf.copywrf.WRFCopyTask.out_prefix

Prefix to prepend to output filenames after the com path.

This string is sent through the self.confstrinterp, so it can contain {...} escape sequences.

Definition at line 87 of file copywrf.py.

Referenced by hwrf.copywrf.WRFCopyTask.comfile(), and hwrf.multistorm.WRFCopyTaskMS.comfile().

---

The documentation for this class was generated from the following file:

• /mnt/pan2/projects/hwrfv3/Samuel.Trahan/trunk/ush/hwrf/copywrf.py