produtil Namespace Reference

Platform-independent weather and ocean forecasting utility package. More...

Detailed Description

Platform-independent weather and ocean forecasting utility package.

The produtil package is a general production weather and ocean forecasting utility package. It implements a number of classes and functions needed to implement a reliable, cross-platform weather or ocean forecasting system. This package is entirely model-independent: nothing in it is specific to, or reliant on, the HWRF model.

Note that before you use anything in this module, you must first call the produtil.setup.setup() function, and that function should only be called once per process. Generally this is done at the top of the main program.

File and Product Manipulation

There are a number of file and directory manipulation routines in the produtil package. In many cases, these replace Python standard library routines that either have known bugs or lack logging functionality. If a function exists in produtil and the Python standard library, it is best to use the produtil version to avoid Python's bugs.

• produtil.fileop — Many simple routines to manipulate files and directories. Works around many Python bugs and adds logging to file manipulation routines.
• produtil.acl — A wrapper around libacl. This is used by the produtil.fileop.deliver_file() to copy access control lists (ACLs)
• produtil.cd — Two classes to implement safe cd-in-cd-out blocks using the Python "with" construct. Also implements temporary directories and deletion of pre-existing directories if requested.
• produtil.locking — File locking that works around Lustre, GPFS and Panasas bugs.
• produtil.retry — Retry operations.
• produtil.rstprod — Handle NOAA restricted data requirements.
• produtil.dbn_alert — Trigger DBNet alerts.
• produtil.datastore — A database and product tracking.
• produtil.atparse — A simple text preparser.

Program Execution

The produtil package has flexible, shell-like syntaxes for specifying program execution, including complex MPI execution with multiple executables. Most critically, this package works around a bug in Python's subprocess module, which forgets to close pipes after a fork() call, causing deadlocks in multi-stage pipelines. That bug renders Python's subprocess module worthless for complex pipelines. The produtil.run does not suffer from that problem.

• produtil.run — shell-like syntax for running programs, including a cross-platform way of requesting MPI and OpenMP program execution.
• produtil.prog, produtil.mpiprog — Object tree that underlies the produtil.run implementation.
• produtil.mpi_impl — Contains one module for each MPI implementation supported by produtil.
• produtil.pipeline — Launches and monitors processes for produtil.run.

You should never need to access the mpi_impl or pipeline modules directly, and you should only need the prog and mpiprog modules for type checking. (For example, is my argument a produtil.prog.ImmutableRunner?) In nearly all cases, you can use the produtil.run functions to access the full functionality of all of the program execution modules.

Other Utilities

• produtil.setup — Contains the produtil.setup.setup() function, which initializes the entire produtil package.
• produtil.log — Initialization of the logging module. Sets up logging to match what is required in the NCEP production environment. This is highly configurable (as is the Python logging module).
• produtil.sigsafety — raises an exception on fatal signals, instead of causing an immediate uncontrolled exit. This is connected to the produtil.locking module to work around bugs in Lustre, Panasas and GPFS file locking.
• produtil.rusage — monitor and limit process resource usage
• produtil.batchsystem — Query information about the batch system and current batch job.
• produtil.cluster — Query information about the cluster.

Namespaces
	acl
	Manipulates Access Control Lists (ACL)
	atparse
	ATParser is a text parser that replaces strings with variables and function output.
	batchsystem
	Provides information about the batch system.
	cd
	Change directory, handle temporary directories.
	cluster
	Provides information about the cluster on which this job is running.
	datastore
	Stores products and tasks in an sqlite3 database file.
	dbnalert
	This module runs the NCO dbn_alert program, or logs dbn_alert messages if run with dbn alerts disabled.
	fileop
	This module provides a set of utility functions to do filesystem operations.
	listing
	Contains the Listing class, which emulates "ls -l".
	locking
	Handles file locking using Python "with" blocks.
	log
	Configures logging.
	mpi_impl
	Converts a group of MPI ranks to a runnable command.
	mpiprog
	Object structure for describing MPI programs.
	pipeline
	Internal module that launches and monitors processes.
	prog
	Implements the produtil.run: provides the object tree for representing shell commands.
	retry
	Contains retry_io() which automates retrying operations.
	rstprod
	Handles data restriction classes.
	run
	A shell-like syntax for running serial, MPI and OpenMP programs.
	rusage
	This module allows querying resource usage and limits, as well as setting resource limits.
	setup
	Contains setup(), which initializes the produtil package.
	sigsafety
	Sets up signal handlers to ensure a clean exit.
	tempdir
	This module is an alias for produtil.cd, for backward compatibility.
	workpool
	Contains the WorkPool class, which maintains pools of threads that perform small tasks.