Provides the location of a file in an archive, on disk or on a remote server via sftp or ftp. More...

Detailed Description

Provides the location of a file in an archive, on disk or on a remote server via sftp or ftp.

This class is a collection of functions that know how to provide the location of a file in either an archive or a filesystem. It does not know how to actually obtain the file. This serves as the underlying "where is that file" implementation of InputSource. All of this is driven by a section in an hwrf.config.HWRFConfig object.

For example, suppose one set up this configuration file:

 [wcoss_fcst_nco]
 # WCOSS: Input locations for the production HWRF
 gfs               = /com/gfs/prod/gfs.{aYMD}/
 gdas1             = /com/gfs/prod/gdas.{aYMD}/
 gfs_sf            = gfs.t{aHH}z.sf{fahr:02d}
 gfs_sfcanl        = gfs.t{aHH}z.sfcanl
 gdas1_bufr        = gdas1.t{aHH}z.{obstype}.tm00.bufr_d

In this example, "gfs" is a dataset, while "gfs_sfcanl" is an item in the dataset. The DataCatalog.locate() function can find the location of a gfs_sf file given the inputs required for string expansion by hwrf.config.HWRFConfig.timestrinterp(). In this case, only the analysis time is required for the "{aYMD}" in the dataset location and "{aHH}" in the gfs_sfcanl filename.

 dc=DataCatalog(conf,"wcoss_fcst_nco","2015091800")
 sfcanl=dc.locate("gfs","gfs_sfcanl")
 print sfcanl

That code would print "/com/gfs/prod/gfs.20150818/gfs.t00z.sfcanl" which is the operational output path of the GFS surface analysis file for the analysis time in question.

Suppose we wanted the spectral forecast file, "gfs_sf" instead, for forecast hour 54. That also requires the forecast time ("ftime") in order to fill in the "{fahr:02d}" in the filename with the number 54.

 dc=DataCatalog(conf,"wcoss_fcst_nco","2015091800")
 sf48a=dc.locate("gfs","gfs_sf",ftime="2015092006")
 sf48b=dc.locate("gfs","gfs_sf",ftime=48*3600)
 print sf48a
 print sf48b

That code would print "/com/gfs/prod/gfs.20150818/gfs.t00z.sf54" twice. Note that you can specify the forecast time as an absolute time, or as a number of seconds relative to the analysis time and achieve the same effect either way.

If we want the bufr file, we have to provide one more piece of information: the observation type, to fill in "{obstype}".

 dc=DataCatalog(conf,"wcoss_fcst_nco","2015091800")
 gpm=dc.locate("gdas1","gdas1_bufr",obstype="gpm")
 print gpm

which prints "/com/gfs/prod/gdas.20150918/gdas1.t00z.gpm.tm00.bufr_d"

Definition at line 109 of file input.py.

Inheritance diagram for hwrf.input.DataCatalog:

Public Member Functions
def	__init__ (self, conf, section, anltime)
	DataCatalog constructor. More...

def	__repr__ (self)
	A string representation of this DataCatalog.

def	rt_updated (self)
	Is this dataset updated in real-time? More...

def	parse (self, string, atime=None, ftime=None, logger=None, dates=None, kwargs)
	Internal function that performs string interpolation. More...

def	locate (self, dataset, item, atime=None, ftime=None, logger=None, dates=None, kwargs)
	Find the location of a requested piece of data. More...

Public Attributes
	conf
	The configuration object, an hwrf.config.HWRFConfig or subclass. More...

	section
	The section used for dataset and item locations in conf. More...

	anltime
	The default analysis time for parse() and locate() if none is specified. More...

Constructor & Destructor Documentation

def hwrf.input.DataCatalog.__init__	(	self,
		conf,
		section,
		anltime
	)

DataCatalog constructor.

Parameters

conf	the configuration object, an hwrf.config.HWRFConfig
section	the section that provides location information
anltime	the default analysis time

Definition at line 171 of file input.py.

Member Function Documentation

def hwrf.input.DataCatalog.locate	(	self,
		dataset,
		item,
		atime = `None`,
		ftime = `None`,
		logger = `None`,
		dates = `None`,
		kwargs
	)

Find the location of a requested piece of data.

Locates the specified item for the specified dataset, at the given analysis time ("atime") and forecast time ("ftime"). If the requested data is known to not exist, returns None. This should be overridden by subclasses. The present implementation just does this: {dataset}/{item} expanding dataset and item with self.parse. Any kwargs are passed along: this allows such things as ensemble ID, or switching between GRIB1 or GRIB2 via a keyword argument.

Parameters

dataset	The name of the dataset.
item	The name of the item in the dataset.
atime	Optional: the analysis time. The default is self.anltime.
ftime	Optional: the forecast time which can be anything accepted by hwrf.numerics.to_datetime_rel() relative to the analysis time.
logger	Optional: a logging.Logger for log messages. If this is provided, several steps along the way of finding the data location are logged.
dates	Optional: dates for which this datasource is valid. This is passed to in_date_range() for validation. This is used to implement the InputSource date ranges.
kwargs	Additional keyword arguments are passed by parse() to the hwrf.config.HWRFConfig.timestrinterp() for string replacement.

Returns: The path to the requested data or None if it is not found.

Definition at line 259 of file input.py.

Referenced by hwrf.input.DataCatalog.parse().

def hwrf.input.DataCatalog.parse	(	self,
		string,
		atime = `None`,
		ftime = `None`,
		logger = `None`,
		dates = `None`,
		kwargs
	)

Internal function that performs string interpolation.

This is an internal implementation function that you should not call directly. It performs string interpolation using the underlying conf object. This acts exactly like the expansions done in the hwrf.conf file: {stuff} is expanded to the contents of the "stuff" variable. Expansions are done in the section specified in the constructor. In addition, various a* and f* variables are expanded based on the analysis time ("atime") and forecast time ("ftime"). See hwrf.config.HWRFConfig.timestrinterp() for details.

Parameters

string	the string being expanded
atime	Optional: the analysis time. The default is self.anltime
ftime	Optional: the forecast time.
logger	Optional: a logging.Logger for log messages
dates	Optional: dates for which this datasource is valid. This is passed to in_date_range() for validation. This is used to implement the InputSource date ranges.
kwargs	Additional keyword arguments are passed to the hwrf.config.HWRFConfig.timestrinterp() for string replacement.

Returns: The return value from string interpolation or None if nothing was found.

Definition at line 210 of file input.py.

Referenced by hwrf.input.DataCatalog.locate(), and hwrf.input.DataCatalog.rt_updated().

def hwrf.input.DataCatalog.rt_updated ( self )

Is this dataset updated in real-time?

Returns: True if this dataset is updated in real-time, False otherwise. By default, this will return True if conf[section,"rt_updated"] is set to "yes" or False otherwise.

Definition at line 199 of file input.py.