produtil.fileop Namespace Reference

This module provides a set of utility functions to do filesystem operations. More...

Detailed Description

This module provides a set of utility functions to do filesystem operations.

It replaces or improves upon several os, stat, and sys module functions by working around Python bugs, providing an API layer that allows forward compatibility to future Python versions, and adding logging capabilities.

Classes
class	CannotFindExe
	Thrown when find_exe cannot find an executable in the path or directory list. More...
class	CannotLinkMulti
	This exception is raised when the caller tries to create multiple symlinks in a single target, but the target is not a directory. More...
class	DeliveryFailed
	This exception is raised when a file cannot be delivered. More...
class	FileOpError
	This is the superclass of several exceptions relating to multi-file operations in produtil.fileop. More...
class	FileOpErrors
	This exception is raised when an operation that processes multiple files catches more than one exception. More...
class	FileWaiter
	A class that waits for files to meet some requirements. More...
class	FindExeInvalidExeName
	Thrown when find_exe is given an executable name that contains a directory path. More...
class	InvalidExecutable
	Thrown when a find_exe fails. More...
class	RelativePathError
	Raised when a relative path is given, but an absolute path is expected. More...
class	UnexpectedAbsolutePath
	This exception indicates that the renamer function sent to make_symlinks_in returned an absolute path. More...
class	VerificationFailed
	This exception is raised when a copy of a file has different content than the original. More...
class	WrongSymlink
	Raised when os.symlink makes a symlink to a target other than the one that was requested. More...

Functions
def	realcwd ()
	Returns the current working directory, expanding any symbolic links. More...
def	chdir
	Changes to the specified directory. More...
def	touch
	Open the file for append and set mtime and atime. More...
def	netcdfver (filename)
	What is the NetCDF version of this file? More...
def	gribver (filename)
	What is the GRIB version of this file? More...
def	makedirs
	Make a directory tree, working around filesystem bugs. More...
def	remove_file
	Deletes the specified file. More...
def	rmall (args, kwargs)
	Deletes the specified list of files. More...
def	lstat_stat
	Runs lstat and stat on a file as efficiently as possible. More...
def	isnonempty (filename)
	Returns True if the filename refers to an existent file that is non-empty, and False otherwise. More...
def	deliver_file
	This moves or copies the file "infile" to "outfile" in a unit operation; outfile will never be seen in an incomplete state. More...
def	find_exe
	Searches the $PATH or a specified iterable of directory names to find an executable file with the given name. More...
def	symlink_read_test
	Opens the specified file for reading and attempts to read data to it. More...
def	make_symlinks_in
	Creates symbolic links from a set of source files to a target directory. More...
def	make_symlink
	Creates a symbolic link "target" that points to "source". More...
def	replace_symlink
	Do not call this routine directly: you want make_symlink instead. More...
def	ln_sf
def	unblock
	Attempts to modify the given stream to be non-blocking. More...
def	call_fcntrl
	Internal function that implements unblock() More...
def	fortlink
	This is a convenience routine that makes many symbolic links to fort.N files for various integers N using make_symlink. More...
def	fortcopy (forts, basedir=None, logger=None, only_log_errors=False, kwargs)
	A convenience function for copying files to local fort.N files for various integers N using deliver_file(...,keep=True). More...
def	norm_expand_path
	Normalizes path and expand home directories. More...
def	norm_abs_path
	Return relative path. More...
def	check_last_lines
	Checks the last few bytes of a file to see if the specified search string is present. More...
def	check_file
	Determines whether the specified file exists, and meets additional requirements. More...
def	wait_for_files
	Waits for files to meet requirements. More...

Variables
list	__all__
	List of symbols exported by "from produtil.fileop import *". More...
tuple	module_logger = logging.getLogger('produtil.fileop')
	sync_frequently = False

Function Documentation

def produtil.fileop.call_fcntrl	(		stream,
			on,
			off,
			logger = None
	)

Internal function that implements unblock()

Parameters

stream	the stream to modify
on	flags to turn on
off	flags to turn off
logger	a logging.Logger for messages

Returns

True on success, False otherwise.

Definition at line 804 of file fileop.py.

Referenced by produtil.fileop.unblock().

def produtil.fileop.chdir	(		path,
			logger = None
	)

Changes to the specified directory.

Please use produtil.cd.NamedDir instead.

This is generally not a good idea since you will not cd back if an unhandled exception is raised. It is better to use the produtil.cd module, which provides ways to enter a directory in a "with" block and optionally delete it afterwards. Such functionality could also be implemented via a try...finally block.

Parameters

path	the path to cd to
logger	a logging.Logger for log messages

Definition at line 145 of file fileop.py.

Referenced by scripts.exhwrf_gsi_post.main(), and scripts.exhwrf_input.main().

def produtil.fileop.check_file	(		filename,
			min_size = None,
			min_mtime_age = None,
			min_atime_age = None,
			min_ctime_age = None,
			logger = None
	)

Determines whether the specified file exists, and meets additional requirements.

Parameters

filename	The file to analyze.
min_size	If present, the file must be at least this many bytes.
min_mtime_age	If specified, the file must have been modified more than this many seconds in the past.
min_atime_age	if specified, the file atime must be at least this many seconds old. The meaning of atime varies, but usually means the last access time.
min_ctime_age	If specified, the file ctime must be at least this many seconds old. The meaning of ctime varies between platforms and file types, but usually means the file creation or inode change time. See stat(2) for details.
logger	a logging.Logger for log messages.

Note

This routine can also be used on directories, but one should avoid the min_size option when doing that.

Returns

True if requirements are met, False otherwise.

Definition at line 982 of file fileop.py.

Referenced by produtil.datastore.UpstreamFile.check(), produtil.fileop.FileWaiter.check(), and produtil.fileop.check_last_lines().

def produtil.fileop.check_last_lines	(		filename,
			searchstr,
			lastbytes = 10000,
			logger = None
	)

Checks the last few bytes of a file to see if the specified search string is present.

Returns True if the string is present or False if the file existed but the string was not present. Will raise an exception if the file is non-existent or cannot be read.

Parameters

filename	The file to search (a string).
searchstr	The string to search for. Must not contain end-of-line chars.
lastbytes	The number of bytes at the end of the file to check. Can be larger than the file size.
logger	A logging.Logger for log messages.

Returns

True if the file contains the given string, False otherwise

Definition at line 948 of file fileop.py.

def produtil.fileop.deliver_file	(		infile,
			outfile,
			keep = True,
			verify = False,
			blocksize = 1048576,
			tempprefix = None,
			permmask = 002,
			removefailed = True,
			logger = None,
			preserve_perms = True,
			preserve_times = True,
			preserve_group = None,
			copy_acl = None,
			moveok = True,
			force = True,
			copier = None
	)

This moves or copies the file "infile" to "outfile" in a unit operation; outfile will never be seen in an incomplete state.

If the caller specifies keep=False (default is True) and moveok=True, and the source and destination are on the same filesystem then the delivery is done with a simple move. Otherwise a copy is done to a temporary file on the same filesystem as the target. If verification is requested (verify=True) then the temporary file is verified by filecmp.cmp, before moving the temporary file to the final location.

When requested, and when possible, the permissions and ownership are preserved. Both copy_acl and preserve_group have defaults set by the produtil.cluster module. If the cluster uses access control lists for data restriction classes, then copy_acl will be set to True, otherwise it is false. If group quotas are enabled, preserve_group is False, otherwise it is True.

Note

The original file is never deleted, but it may be moved to the target if keep=False. If a copy is done instead, the original file is still present.

Parameters

infile	the origin file
outfile	the destination file or its parent directory
keep	If False, the original file is no longer needed. If False and moveok=True, the file might be delivered by a "mv" operation, avoiding any data duplication (no "cp").
verify	If a "cp" is done, reopen the target and source and verify they are the same. Note that providing a copier will break the verification functionality if the copier changes the contents of the destination file (such as a copier that compresses).
blocksize	block size during copy operations
tempprefix	Prefix for temporary files during copy operations. Do not include directory paths in the tempprefix.
permmask	Permission bits to remove Default: world write (002)
removefailed	If True, delete temporary files if the delivery fails
logger	the logging.Logger for log messages.
preserve_perms	If True, copy the old file's permissions to the new file
preserve_times	If True, copy the old file's timestamps to the new file
preserve_group	If True, copy the old file's group ID to the new file
copy_acl	If True, copy the access control lists from one file to another
moveok	If True, delivery by "mv" is allowed. Must also set keep=False.
force	If False, delivery will be aborted (raise TargetFileExists) if the target file already exists.
copier	If present, this function or callable object is used to copy data from the source file to the temporary file before moving it to the target. The copier is called as copier(infile,temp_file_name,temp_file_object) Where the temp_file_name is the name of the destination file and the temp_file_object is an object that can be used to write to the file. The copier should NOT close the temp_file_object.

Definition at line 359 of file fileop.py.

Referenced by hwrf.multistorm.RealInit.add_wrfinput(), produtil.datastore.FileProduct.deliver(), hwrf.regrib.GRIB2Product.deliver(), hwrf.regrib.GRIB1Product.deliver(), scripts.exhwrf_output.deliver_multi(), hwrf.bufrprep.Bufrprep.deliver_products(), hwrf.gsi.GSIBase.deliver_products(), hwrf.fcsttask.AnalysisCycle.deliver_products(), produtil.fileop.fortcopy(), produtil.fileop.isnonempty(), hwrf.post.link_post_fix(), scripts.exhwrf_ensda.main(), hwrf.post.EGRIB1Product.make(), hwrf.regrib.UpstreamGRIB1.make(), hwrf.regrib.GRIB2Product.make(), hwrf.regrib.GRIB1Product.make(), hwrf.post.PostOneWRF.make_control(), hwrf.gsi.GSIBase.make_diag_files(), produtil.fileop.make_symlinks_in(), hwrf.ww3.WW3Init.run(), and hwrf.bufrprep.Bufrprep.tdrdump().

def produtil.fileop.find_exe	(		name,
			dirlist = None,
			raise_missing = True
	)

Searches the $PATH or a specified iterable of directory names to find an executable file with the given name.

Returns the exectuable's location. If the executable cannot be found, and raise_missing=True, raises CannotFindExe, otherwise returns None. Raises FindExeInvalidExeName if "name" is not the same as its os.path.basename.

Parameters

name	The name of the executable to find.
dirlist	The list of directories to search, or None to search $PATH
raise_missing	If True, the CannotFindExe exception is raised for executables that cannot be found. If False, return None in that situation.

Definition at line 573 of file fileop.py.

Referenced by hwrf.nhc_products.NHCProducts.deliver_wrfdiag(), produtil.mpi_impl.mpirun_lsf.mpirunner(), produtil.mpi_impl.mpiexec.mpirunner(), produtil.mpi_impl.mpiexec_mpt.mpirunner(), produtil.mpi_impl.impi.mpirunner(), produtil.mpi_impl.lsf_cray_intel.mpirunner2(), produtil.mpi_impl.srun.mpirunner_impl(), hwrf.copywrf.WRFCopyTask.ncks_path(), hwrf.ww3.WW3Post.ncks_path(), hwrf.relocate.Stage1.relocate_storm(), hwrf.launcher.HWRFLauncher.sanity_check_executables(), and hwrf.gsi.GSIBase.wrfout_copier().

def produtil.fileop.fortcopy	(		forts,
			basedir = None,
			logger = None,
			only_log_errors = False,
			kwargs
	)

A convenience function for copying files to local fort.N files for various integers N using deliver_file(...,keep=True).

It works similarly to fortlink. The force= argument tells fortcopy to overwrite existing files. Otherwise, an exception will be raised if the destination file already exists. The optional basedir argument is the parent directory of the fort.N.

Call like this:

fortcopy({ 15:"/usr/local/share/file1",
           23:"./file2"})

And you will create files:

./fort.15 (copied from /usr/local/share/file1)
./fort.23 (copied from ./file2)

All other keyword arguments are sent to deliver_file.

Parameters

forts	Mapping from Fortran unit number to copy target.
basedir	Where to put the files instead of the current directory.
logger	A logging.Logger for log messages.
only_log_errors	Only log failed operations instead of logging everything.
kwargs	All other keyword arguments are passed to deliver_file()

Definition at line 868 of file fileop.py.

Referenced by hwrf.finalmergetask.FinalMergeTask.run_ext(), and hwrf.relocate.RelocationTask.run_ext().

def produtil.fileop.fortlink	(		forts,
			force = False,
			basedir = None,
			logger = None
	)

This is a convenience routine that makes many symbolic links to fort.N files for various integers N using make_symlink.

It works similarly to fortcopy. The optional basedir is the relative directory. The optional force argument is passed on to make_symlink and has the usual meaning: replace existing files.

Call like this:

fortlink({ 15:"/usr/local/share/file1",
           23:"./file2"})

And you will create these symbolic links:

./fort.15 -> /usr/local/share/file1
./fort.23 -> ./file2

as with other symlink routines in this module, set force=True to remove target fort.N files if they already exist.

Parameters

forts	Mapping from Fortran unit number to link target.
force	Remove target files if they exist.
basedir	Where to make the links instead of the current directory.
logger	A logging.Logger for log messages.

Definition at line 834 of file fileop.py.

Referenced by hwrf.tracker.TrackerTask.run(), hwrf.finalmergetask.FinalMergeTask.run_ext(), and hwrf.relocate.RelocationTask.run_ext().

def produtil.fileop.gribver

(

filename

)

What is the GRIB version of this file?

Returns the GRIB file version: 1 or 2. If the file is not a GRIB file, or if the answer is indeterminate, returns None. Only the first GRIB record is tested.

Parameters

filename

the path to the file to test

Definition at line 201 of file fileop.py.

Referenced by hwrf.wps.Ungrib.get_grib().

def produtil.fileop.isnonempty

(

filename

)

Returns True if the filename refers to an existent file that is non-empty, and False otherwise.

Parameters

filename

The file to test.

Definition at line 333 of file fileop.py.

Referenced by hwrf.tracker.GRIB1VintTave.action_vinttave(), hwrf.multistorm.RealInit.add_wrfinput(), hwrf.relocate.Stage3.anl_4x(), hwrf.relocate.Stage3.anl_cs_10m(), hwrf.wps.WPSTask.check_geogrid(), hwrf.wps.Ungrib.check_outfiles(), hwrf.post.check_post(), scripts.exhwrf_check_init.check_relocate(), hwrf.regrib.GRIB2Product.deliver(), hwrf.regrib.GRIB1Product.deliver(), hwrf_alerts.email_afos_to_sdm(), hwrf.wps.Ungrib.get_grib(), hwrf.gsi.GSIBase.grab_gfs_enkf(), hwrf.bufrprep.Bufrprep.grab_obstype_section(), hwrf.gsi.GSIBase.grab_obstype_section(), hwrf.bufrprep.Bufrprep.grab_prepbufr(), hwrf.gsi.GSIBase.grab_prepbufr(), produtil.fileop.gribver(), hwrf.wps.WPSTask.link_geogrid(), hwrf.launcher.multistorm_parse_args(), hwrf.launcher.parse_launch_args(), hwrf.relocate.Stage3.pert_ct_gfs(), hwrf.relocate.Stage3.pert_ct_weak(), hwrf.prelaunch.prelaunch_rsmc(), hwrf.gsipost.GSIPost.run(), hwrf.ww3.WW3Init.run(), hwrf.relocate.Stage1.split_wrf(), hwrf.launcher.HWRFLauncher.timeless_sanity_check(), hwrf.gribtask.GRIBTask.uncomplete(), and hwrf.wrf.ExternalWRFTask.update_state().

def produtil.fileop.lstat_stat	(		filename,
			raise_nonexist = False
	)

Runs lstat and stat on a file as efficiently as possible.

Returns (lstat(filename),stat(filename)) where each is None if it fails due to non-existence. Does this in as few filesystem metadata operations as possible. Will raise an exception if the stat fails for any reason other than non-existence of a file, or if the file or linked file is non-existent and raise_nonexist=True.

Parameters

filename	The file to test.
raise_nonexist	Should we raise an exception if the file does not exist?

Returns

a tuple (L,S) where L is the lstat return value, and S is the stat return value. Each will be None if the file or link target do not exist.

Definition at line 307 of file fileop.py.

Referenced by produtil.fileop.deliver_file(), produtil.fileop.isnonempty(), and produtil.fileop.make_symlinks_in().

def produtil.fileop.make_symlink	(		source,
			target,
			force = False,
			logger = None,
			max_tries = 20
	)

Creates a symbolic link "target" that points to "source".

If the target already exists and is NOT a directory, then the file will be replaced. The replacement is done in a unit operation so that the target will always exist (unless the operation fails).

Parameters

source	The file to link to.
target	The name of the link.
force	If True, and target exists, delete it first.
logger	a logging.Logger for log messages.

Definition at line 677 of file fileop.py.

Referenced by produtil.fileop.fortlink(), hwrf.gsi.GSIBase.grab_fix_parm(), hwrf.gsi.GSIBase.grab_gfs_enkf(), hwrf.gsi.GSIBase.grab_wrf_enkf(), hwrf.hycom.HYCOMIniter.link_coupled_inputs(), hwrf.wps.WPSTask.link_geogrid(), hwrf.tracker.TrackerTask.link_grib(), produtil.fileop.make_symlinks_in(), and hwrf.gsipost.GSIPost.run().

def produtil.fileop.make_symlinks_in	(		sources,
			targetdir,
			force = False,
			renamer = None,
			logger = None,
			copy = False
	)

Creates symbolic links from a set of source files to a target directory.

If "force" is True, then any existing files will first be deleted.

The "renamer" can be a function that generates paths of the symlinks, relative to targetdir, for each symlink in "sources". If the return value from "renamer" is an absolute path, an exception will be thrown. If the return value is None, then no link will be made.

Example: make_symlinks_in(['/path/to/a','/path/to/b'],'.', renamer=lambda s: os.path.basename(s)+'.linkified')

will create a.linkified, linked to /path/to/a, and b.linkified, linked to /path/to/b in directory "."

Parameters

sources	The list of files to link to.
targetdir	The directory in which to place the links.
force	Remove existing files if needed.
renamer	Function to generate link names.
logger	A logging.Logger for log messages.
copy	If True, files are copied instead of linked.

Definition at line 620 of file fileop.py.

Referenced by produtil.fileop.symlink_read_test().

def produtil.fileop.makedirs	(		filename,
			numtries = 10,
			logger = None
	)

Make a directory tree, working around filesystem bugs.

This makedirs implementation works around a common bug: if two processes try to recursively make a directory tree simultaneously, makedirs can fail when two processes make the same path component at the same time. This implementation automatically retries in that situation.

Parameters

filename	the directory path
numtries	the number of times to retry
logger	a logging.Logger for log messages

Definition at line 224 of file fileop.py.

Referenced by produtil.locking.LockFile.acquire_impl(), hwrf.mpipomtc.POMInit.deliver_products(), hwrf.bufrprep.Bufrprep.deliver_products(), hwrf.finalmergetask.FinalMergeTask.deliver_products(), hwrf.relocate.RelocationTask.deliver_products(), hwrf.gsi.GSIBase.deliver_products(), hwrf.fcsttask.RealNMM.deliver_products(), hwrf.fcsttask.WRFAnl.deliver_products(), hwrf.fcsttask.AnalysisCycle.deliver_products(), hwrf.launcher.HWRFLauncher.gen_vitals(), hwrf.launcher.launch(), hwrf.hycom.HYCOMIniter.link_coupled_inputs(), scripts.exhwrf_output.main(), scripts.exhwrf_para_archive.main_disk(), hwrf.config.HWRFConfig.makedirs(), hwrf.finalmergetask.FinalMergeTask.run(), hwrf.wps.Geogrid.run(), hwrf.relocate.Stage1.run(), hwrf.wps.Ungrib.run(), hwrf.wps.Metgrid.run(), hwrf.relocate.Stage2.run(), hwrf.relocate.Stage3.run(), hwrf.relocate.Merge.run(), hwrf.gribtask.GRIBTask.run_helper(), hwrf.prep.PrepHybrid.run_ipiece(), setup_hurricane.StormCurses.setup(), hwrf.input.tempopen(), hwrf.post.PostManyWRF.wrftask(), and hwrf.relocate.RelocationInfo.write_info().

def produtil.fileop.netcdfver

(

filename

)

What is the NetCDF version of this file?

Returns one of three strings based on the NetCDF version of the given file, or returns None if the file is not NetCDF:

• "CDF1" = NetCDF classic format
• "CDF2" = NetCDF 64-bit offset format
• "HDF5" = HDF5 file, and hence possibly a NetCDF4 file.
• None = Not NetCDF and not HDF5

  Parameters

  filename

  the name of the file to test

Definition at line 177 of file fileop.py.

Referenced by hwrf.copywrf.WRFCopyTask.compression_copier(), hwrf.copywrf.WRFCopyTask.decompression_copier(), hwrf.gsi.GSIBase.grab_wrf_enkf(), and hwrf.relocate.Stage1.relocate_storm().

def produtil.fileop.norm_abs_path	(		rel,
			fromdir = None
	)

Return relative path.

This routine generates relative file paths (using os.path.relpath) that are relative to the specified "from" directory fromdir. The fromdir will be first sent through norm_expand_path to eliminate system-specific weirdness, such as a/./b, a/../b, ~username and so on. This will raise RelativePathError if the resulting path is not absolute.

Parameters

rel	the path
fromdir	the directory from which we want the relative path

Definition at line 934 of file fileop.py.

def produtil.fileop.norm_expand_path	(		path = None,
			fullnorm = False
	)

Normalizes path and expand home directories.

Calls os.path.normpath and os.path.expanduser on its argument, or on os.getcwd() if no argument is supplied (or if path=None). This removes extraneous a/./b, a/../b, expands ~username and ~, and other system-specific expansions. See the Python documentation of normpath and expanduser for details. Will also call realpath and normcase if fullnorm=True. Raises RelativePathError if the resulting path is not absolute.

Parameters

path	the path to expand
fullnorm	If True, call os.path.normcase() and os.path.realpath() normapth and expanduser.

Definition at line 910 of file fileop.py.

Referenced by hwrf_scrub.Deleter.add(), and produtil.fileop.norm_abs_path().

def produtil.fileop.realcwd

(

)

Returns the current working directory, expanding any symbolic links.

Definition at line 140 of file fileop.py.

def produtil.fileop.remove_file	(		filename,
			info = True,
			logger = None
	)

Deletes the specified file.

Does nothing if the filename is None, is the empty string or already does not exist. Otherwise, the file is deleted.

Parameters

filename	The file to delete.
info	Optional: indicates that warnings about a file already not existing should be sent to the logger at INFO level (info=True) instead of WARNING (info=False).
logger	the logging.Logger for messages

Definition at line 251 of file fileop.py.

Referenced by hwrf.copywrf.WRFCopyTask.compression_copier(), hwrf.copywrf.WRFCopyTask.decompression_copier(), hwrf.nhc_products.NHCProducts.deliver_wrfdiag(), hwrf.relocate.RelocationTask.make_analysis_namelist(), hwrf.post.PostOneWRF.make_control(), hwrf.relocate.RelocationTask.make_ghost_namelist(), hwrf.relocate.Stage1.relocate_storm(), produtil.fileop.rmall(), hwrf.gsipost.GSIPost.run(), hwrf.ww3.WW3Init.run(), hwrf.wps.Geogrid.run(), hwrf.relocate.Merge.run(), hwrf.finalmergetask.FinalMergeTask.run_ext(), hwrf.relocate.RelocationTask.run_ext(), produtil.datastore.FileProduct.undeliver(), hwrf.regrib.GRIB2Product.undeliver(), hwrf.regrib.GRIB1Product.undeliver(), hwrf.coupling.CouplingStatus.unset(), hwrf.gsi.unset_gsistatus(), and hwrf.gsi.GSIBase.wrfout_copier().

def produtil.fileop.replace_symlink	(		source,
			target,
			logger = None,
			max_tries = 20
	)

Do not call this routine directly: you want make_symlink instead.

This routine creates a new symbolic link and renames that link to "target." That always replaces target with a symbolic link to source, even if target did not already exist.

Parameters

source	the file to link from
target	the file to link to
logger	a logging.Logger for messages

Definition at line 723 of file fileop.py.

Referenced by produtil.fileop.make_symlink().

def produtil.fileop.rmall	(		args,
			kwargs
	)

Deletes the specified list of files.

Deletes files listed in "args". Each one is passed to remove_file. Exceptions that derive from EnvironmentError are collected, and will be raised at the end, thus allowing removal of later files to continue if earlier ones failed. If only one file causes an exception, that exception will be raised, otherwise FileOpErrors will be raised

Parameters

args	The files to delete.
kwargs	Keyword arguments passed to remove_file().

Definition at line 276 of file fileop.py.

def produtil.fileop.symlink_read_test	(		filename,
			readsize = 40,
			logger = None
	)

Opens the specified file for reading and attempts to read data to it.

Logs the process. Will NOT raise any I/O or system errors; they are ignored. This is a workaround for a bug in Cray: symlinks to recently created files cannot be read by the compute node unless the batch node reads from them first (or unless you wait a while).

Definition at line 604 of file fileop.py.

Referenced by produtil.fileop.make_symlink(), and produtil.fileop.replace_symlink().

def produtil.fileop.touch	(		filename,
			times = None
	)

Open the file for append and set mtime and atime.

Opens the specified file in append mode, but writes nothing. Sets the access and modification times.

Parameters

filename	the string filename
times	A 2-tuple of numbers, of the form (atime, mtime). These are UNIX epoch times (seconds since 1970 began in UTC).

Definition at line 164 of file fileop.py.

Referenced by hwrf.launcher.multistorm_parse_args().

def produtil.fileop.unblock	(		stream,
			logger = None
	)

Attempts to modify the given stream to be non-blocking.

This only works with streams that have an underlying POSIX fileno, such as those from open.

Will re-raise any exception received, other than AttributeError and EnvironmentError. Hence, I/O errors and attempts to make a non-fileno stream non-blocking will produce a False return value, while anything else will raise an exception.

Parameters

stream	the stream to unblock
logger	a logging.Logger for log messages

Returns

True on success, False otherwise.

Definition at line 789 of file fileop.py.

def produtil.fileop.wait_for_files	(		flist,
			logger = None,
			maxwait = 1800,
			sleeptime = 20,
			min_size = 1,
			min_mtime_age = 30,
			min_atime_age = None,
			min_ctime_age = None,
			min_fraction = 1.0,
			log_each_file = True
	)

Waits for files to meet requirements.

This is a simple wrapper around the FileWaiter class for convenience. It is equivalent to creating a FileWaiter with the provided arguments, and calling its checkfiles routine.

Parameters

flist	the file or list of files to wait for. This is simply sent into self.add.
logger	a logging.Logger for messages
maxwait	maximum seconds to wait
sleeptime	sleep time in seconds between checks
min_size	minimum file size
min_mtime_age	minimum modification time age,
min_atime_age	minimum access time age.
min_ctime_age	time since last file status change (see stat(2))
min_fraction	the minimum fraction of the provided files that must match the above requirements in order for FileWaiter.wait to return True. Default is 1.0, which means all of them.
log_each_file	log messages about each file checked

Definition at line 1189 of file fileop.py.

Referenced by produtil.fileop.FileWaiter.checkfiles().

Variable Documentation

produtil.fileop.__all__

Initial value:

= ['FileOpError','FileOpErrors','CannotLinkMulti',
         'UnexpectedAbsolutePath','InvalidExecutable',
         'FindExeInvalidExeName','CannotFindExe','RelativePathError',
         'DeliveryFailed','VerificationFailed','realcwd','chdir',
         'makedirs','remove_file','rmall','lstat_stat','isnonempty',
         'check_file','deliver_file','make_symlinks_in','find_exe',
         'make_symlink','replace_symlink','unblock','fortcopy',
         'norm_expand_path','norm_abs_path','check_last_lines',
         'wait_for_files','FileWaiter','call_fcntrl','gribver',
         'netcdfver','touch']

List of symbols exported by "from produtil.fileop import *".

Definition at line 9 of file fileop.py.