A shell-like syntax for running serial, MPI and OpenMP programs. More...

Detailed Description

A shell-like syntax for running serial, MPI and OpenMP programs.

This module implements a shell-like syntax for launching MPI and non-MPI programs from Python. It recognizes three types of executables: mpi, "small serial" (safe for running on a batch node) and "big serial" (which should be run via aprun if applicable). There is no difference between "small serial" and "big serial" programs except on certain architectures (like Cray) where the job script runs on a heavily-loaded batch node and has compute nodes assigned for running other programs.

Program Types

There are three types of programs: mpi, serial and "big non-MPI." A "big" executable is one that is either OpenMP, or is a serial program that cannot safely be run on heavily loaded batch nodes. On Cray architecture machines, the job script runs on a heavily-populated "batch" node, with some compute nodes assigned for "large" programs. In such environments, the "big" executables are run on compute nodes and the small ones on the batch node.

mpi('exename') = an executable "exename" that calls MPI_Init and MPI_Finalize exactly once each, in that order.
exe('exename') = a small non-MPI program safe to run on a batch node
bigexe('exename') = a big non-MPI program that must be run on a compute node it may or may not use other forms of parallelism

You can also make reusable aliases to avoid having to call those functions over and over (more on that later). Examples:

Python: wrf=mpi('./wrf.exe')
Python: lsl=alias(exe('/bin/ls')['-l'].env(LANG='C',LS_COLORS='never'))

Those can then be reused later on as if the code is pasted in, similar to a shell alias.

Serial Execution Syntax

Select your serial programs by exe('name') for small serial programs and bigexe('name') for big serial programs. The return value of those functions can then be used with a shell-like syntax to specify redirection and piping. Example:

shell version: ls -l / | wc -l
Python version: run(exe('ls')['-l','/'] | exe('wc')['-l'])

Redirection syntax similar to the shell (< > and << operators):

1 run( ( exe('myprogram')['arg1','arg2','...'] < 'infile' ) > 'outfile')

Note the extra set of parentheses: you cannot do "exe('prog') < infile

outfile" because of the order of precedence of Python operators

Append also works:

1 run(exe('myprogram')['arg1','arg2','...'] >> 'appendfile')

You can also send strings as input with <<

1 run(exe('myprogram')['arg1','arg2','...'] << 'some input string')

One difference from shells is that < and << always modify the beginning of the pipeline:

shell: cat < infile | wc -l
Python #1: ( exe('cat') < 'infile' ) | exe('wc')['-l']
Python #2: exe('cat') | ( exe('wc')['-l'] < 'infile' )

Note that the last second one, equivalent to cat|wc -l<infile, would NOT work in a shell since you would be giving wc -l two inputs.

Parallel Execution Syntax

Use mpi('exename') to select your executable, use [] to set arguments, use multiplication to set the number of ranks and use addition to combine different executables together into a multiple program multiple data (MPMD) MPI program.

Run ten copies of ls -l:

1 run(mpirun(mpiserial(('ls')['-l'])*10))

Run HyCOM coupled HWRF: one wm3c.exe, 30 hycom.exe and 204 wrf.exe:

1 run(mpirun(mpi('wm3c.exe') + mpi('hycom.exe')*30 + mpi('wrf.exe')*204))

You can set environment variables, pipe MPI output and handle redirection using the mpirun() function, which converts MPI programs into an bigexe()-style object (Runner):

Shell version:

1 result=$( mpirun -n 30 hostname | sort -u | wc -l )

Python version:

1 result=runstr( mpirun(mpi('hostname')*30) | exe['sort']['-u'] | exe['wc']['-l'] )

Aliases

If you find yourself frequently needing the same command, or you need to store a command for multiple uses, then then you should define an alias. Let's say you want "long output" format Japanese language "ls" output:

1 exe('ls')['-l','/path/to/dir'].env(LANG='JP')

but you find yourself running that on many different directories. Then you may want to make an alias:

1 jplsl=alias(exe('ls')['-l'].env(LANG='JP'))

The return value jplsl can be treated as an exe()-like return value since it was from exe() originally, but any new arguments will be appended to the original set:

1 run(jplsl['/path/to/dir'])

Note that if we did this:

 badlsl=exe('ls')['-l'].env(LANG='JP')  # Bad! No alias!
 run(badlsl['/'])  # will list /
 run(badlsl['/home'])  # will list / and /home
 run(badlsl['/usr/bin']) # will list / /home and /usr/bin
 
 goodlsl=alias(exe('ls')['-l'].env(LANG='JP')
 run(goodlsl['/'])  # will list /
 run(goodlsl['/home'])  # will list /home
 run(goodlsl['/usr/bin']) # will list /usr/bin

Then the run(badlsl['/home']) would list /home AND / which is NOT what we want. Why does it do that? It is because badlsl is not an alias — it is a regular output from exe(), so every time we call its [] operator, we add an argument to the original command. When we call alias() it returns a copy-on-write version (goodlsl), where every call to [] creates a new object.

Note that alias() also works with pipelines, but most operations will only modify the last the command in the pipeline (or the first, for operations that change stdin).

Classes
class	ExitStatusException
	Raised to indicate that a program generated an invalid return code. More...

class	InvalidRunArgument
	Raised to indicate that an invalid argument was sent into one of the run module functions. More...

Functions
def	alias (arg)
	Attempts to generate an unmodifiable "copy on write" version of the argument. More...

def	batchexe (name, kwargs)
	Returns a prog.ImmutableRunner object that represents a small serial program that can be safely run on a busy batch node. More...

def	exe (name, kwargs)
	Returns a prog.ImmutableRunner object that represents a large serial program that must be run on a compute node. More...

def	bigexe (name, kwargs)
	Alias for exe() for backward compatibility. More...

def	mpirun (arg, kwargs)
	Converts an MPI program specification into a runnable shell program suitable for run(), runstr() or checkrun(). More...

def	make_pipeline (arg, capture, kwargs)
	This internal implementation function generates a prog.PopenCommand object for the specified input, which may be a prog.Runner or mpiprog.MPIRanksBase. More...

def	runbg (arg, capture=False, kwargs)
	Not implemented: background execution. More...

def	waitprocs
	Not implemented: background process monitoring. More...

def	runsync
	Runs the "sync" command as an exe(). More...

def	run (arg, logger=None, sleeptime=None, kwargs)
	Executes the specified program and attempts to return its exit status. More...

def	checkrun (arg, logger=None, kwargs)
	This is a simple wrapper round run that raises ExitStatusException if the program exit status is non-zero. More...

def	openmp
	Sets the number of OpenMP threads for the specified program. More...

def	runstr (arg, logger=None, kwargs)
	Executes the specified program or pipeline, capturing its stdout and returning that as a string. More...

def	mpi (arg, kwargs)
	Returns an MPIRank object that represents the specified MPI executable. More...

def	mpiserial (arg, kwargs)
	Generates an mpiprog.MPISerial object that represents an MPI rank that executes a serial (non-MPI) program. More...

Variables
list	__all__
	List of symbols exported by "from produtil.run import *". More...

tuple	module_logger = logging.getLogger('produtil.run')
	Default logger used by some functions if no logger is given.

Function Documentation

def produtil.run.alias ( arg )

Attempts to generate an unmodifiable "copy on write" version of the argument.

The returned copy will generate a modifiable duplicate of itself if you attempt to change it.

Returns: a produtil.prog.ImmutableRunner

Parameters

arg	a produtil.prog.Runner or produtil.prog.ImmutableRunner

Definition at line 220 of file run.py.

def produtil.run.batchexe	(	name,
		kwargs
	)

Returns a prog.ImmutableRunner object that represents a small serial program that can be safely run on a busy batch node.

Parameters

name	the executable name or path
kwargs	passed to produtil.prog.Runner.__init__

Returns: a new produtil.prog.ImmutableRunner

Definition at line 234 of file run.py.

def produtil.run.bigexe	(	name,
		kwargs
	)

Alias for exe() for backward compatibility.

Use exe() instead.

Definition at line 254 of file run.py.

Referenced by hwrf.finalmergetask.FinalMergeTask.run_ext(), and hwrf.relocate.RelocationTask.run_ext().

def produtil.run.checkrun	(	arg,
		logger = `None`,
		kwargs
	)

This is a simple wrapper round run that raises ExitStatusException if the program exit status is non-zero.

Parameters

arg	the produtil.prog.Runner to execute (output of exe(), bigexe() or mpirun()
logger	a logging.Logger to log messages
kwargs	The optional run=[] argument can provide a different list of acceptable exit statuses.

Definition at line 398 of file run.py.

Referenced by hwrf.regrib.action_grbindex(), hwrf.relocate.Merge.blend_gsi(), hwrf.finalmergetask.FinalMergeTask.convert_wrfinput2bin(), hwrf.finalmergetask.FinalMergeTask.finalmerge2netcdf(), hwrf.relocate.Merge.inter_2to1ges(), hwrf.relocate.Merge.inter_3to2(), hwrf.bufrprep.Bufrprep.prep_prepbufr(), hwrf.ensda.CycleTDRCheck.readensdatrigger(), hwrf.bufrprep.Bufrprep.readtdrstmid(), hwrf.bufrprep.Bufrprep.readtdrtime(), hwrf.relocate.Stage1.relocate_storm(), hwrf.relocate.Stage2.relocate_storm(), hwrf.relocate.Merge.relocate_storm(), hwrf.wps.Geogrid.run(), hwrf.wps.Ungrib.run(), hwrf.wps.Metgrid.run(), hwrf.finalmergetask.FinalMergeTask.run_ext(), hwrf.relocate.RelocationTask.run_ext(), hwrf.relocate.Stage3.update_3dvar(), and hwrf.relocate.Merge.update_3dvar().

def produtil.run.exe	(	name,
		kwargs
	)

Returns a prog.ImmutableRunner object that represents a large serial program that must be run on a compute node.

Note: This function does NOT search $PATH on Cray. That ensures the $PATH will be expanded on the compute node instead. Use produtil.fileop.find_exe() if you want to explicitly search the PATH before execution.

Parameters

name	the executable name or path
kwargs	passed to produtil.prog.Runner.__init__

Returns: a new produtil.prog.ImmutableRunner

Definition at line 242 of file run.py.

Referenced by produtil.run.bigexe(), hwrf.relocate.Merge.blend_gsi(), hwrf.finalmergetask.FinalMergeTask.convert_wrfinput2bin(), hwrf.finalmergetask.FinalMergeTask.finalmerge2netcdf(), hwrf.relocate.Merge.inter_2to1ges(), hwrf.relocate.Merge.inter_3to2(), hwrf.bufrprep.Bufrprep.prep_prepbufr(), hwrf.ensda.CycleTDRCheck.readensdatrigger(), hwrf.bufrprep.Bufrprep.readtdrstmid(), hwrf.bufrprep.Bufrprep.readtdrtime(), hwrf.relocate.Stage1.relocate_storm(), hwrf.relocate.Stage2.relocate_storm(), hwrf.relocate.Merge.relocate_storm(), hwrf.wps.Ungrib.run(), hwrf.finalmergetask.FinalMergeTask.run_exe(), hwrf.relocate.Stage3.update_3dvar(), and hwrf.relocate.Merge.update_3dvar().

def produtil.run.make_pipeline	(	arg,
		capture,
		kwargs
	)

This internal implementation function generates a prog.PopenCommand object for the specified input, which may be a prog.Runner or mpiprog.MPIRanksBase.

Parameters

arg	the produtil.prog.Runner to convert. This is the output of exe(), bigexe() or mpirun()
capture	if True, capture the stdout into a string
kwargs	additional keyword arguments, same as for mpirun()

Definition at line 276 of file run.py.

Referenced by produtil.run.run(), produtil.run.runbg(), and produtil.run.runstr().

def produtil.run.mpi	(	arg,
		kwargs
	)

Returns an MPIRank object that represents the specified MPI executable.

Parameters

arg	the MPI program to run
kwargs	logger=L for a logging.Logger to log messages

Definition at line 465 of file run.py.

Referenced by hwrf.wps.Geogrid.run(), and hwrf.wps.Metgrid.run().

def produtil.run.mpirun	(	arg,
		kwargs
	)

Converts an MPI program specification into a runnable shell program suitable for run(), runstr() or checkrun().

Options for kwargs:

allranks=True — to run on all available MPI ranks. This cannot be used if a specific number of ranks (other than 1) was requested in the arg.
logger=L — a logging.Logger for log messages
Other platform-specific arguments. See produtil.mpi_impl for details.

Parameters

arg	the mpiprog.MPIRanksBase describing the MPI program to run. This is the output of the mpi() or mpiserial() function.
kwargs	additional arguments to control output.

Returns: a prog.Runner object for the specified mpiprog.MPIRanksBase object.

Definition at line 258 of file run.py.

Referenced by hwrf.wps.Geogrid.run(), and hwrf.wps.Metgrid.run().

def produtil.run.mpiserial	(	arg,
		kwargs
	)

Generates an mpiprog.MPISerial object that represents an MPI rank that executes a serial (non-MPI) program.

The given value MUST be from bigexe() or exe(), NOT from mpi().

Parameters

arg	the MPI program to run
kwargs	logger=L for a logging.Logger to log messages

Definition at line 472 of file run.py.

def produtil.run.openmp	(	arg,
		threads = `None`
	)

Sets the number of OpenMP threads for the specified program.

Warning: Generally, when using MPI with OpenMP, the batch system must be configured correctly to handle this or unexpected errors will result.

Parameters

arg	The "arg" argument must be from mpiserial, mpi, exe or bigexe.
threads	The optional "threads" argument is an integer number of threads. If it is not specified, the maximum possible number of threads will be used. Note that using threads=None with mpirun(...,allranks=True) will generally not work unless the batch system has already configured the environment correctly for an MPI+OpenMP task with default maximum threads and ranks.

Returns: see run()

Definition at line 415 of file run.py.

Referenced by hwrf.finalmergetask.FinalMergeTask.run_ext(), and hwrf.relocate.RelocationTask.run_ext().

def produtil.run.run	(	arg,
		logger = `None`,
		sleeptime = `None`,
		kwargs
	)

Executes the specified program and attempts to return its exit status.

In the case of a pipeline, the highest exit status seen is returned. For MPI programs, exit statuses are unreliable and generally implementation-dependent, but it is usually safe to assume that a program that runs MPI_Finalize() and exits normally will return 0, and anything that runs MPI_Abort(MPI_COMM_WORLD) will return non-zero. Programs that exit due to a signal will return statuses >255 and can be interpreted with WTERMSIG, WIFSIGNALLED, etc.

Parameters

arg	the produtil.prog.Runner to execute (output of exe(), bigexe() or mpirun()
logger	a logging.Logger to log messages
sleeptime	time to sleep between checks of child process
kwargs	ignored

Definition at line 376 of file run.py.

Referenced by produtil.run.checkrun(), and hwrf.finalmergetask.FinalMergeTask.run_exe().

def produtil.run.runbg	(	arg,
		capture = `False`,
		kwargs
	)

Not implemented: background execution.

Runs the specified process in the background. Specify capture=True to capture the command's output. Returns a produtil.prog.PopenCommand. Call poll() to determine process completion, and use the stdout_data property to get the output after completion, if capture=True was specified.

Bug:: produtil.run.runbg() is not implemented

Warning: this is not implemented

Parameters

arg	the produtil.prog.Runner to execute (output of exe(), bigexe() or mpirun()
capture	if True, capture output
kwargs	same as for mpirun()

Definition at line 303 of file run.py.

def produtil.run.runstr	(	arg,
		logger = `None`,
		kwargs
	)

Executes the specified program or pipeline, capturing its stdout and returning that as a string.

If the exit status is non-zero, then NonZeroExit is thrown.

Example:

1 runstr(exe('false'),ret=(1))

succeeds if "false" returns 1, and raises ExitStatusError otherwise.

Parameters

arg	The "arg" argument must be from mpiserial, mpi, exe or bigexe.
logger	a logging.Logger for logging messages
kwargs	You can specify an optional list or tuple "ret" that contains an alternative list of valid return codes. All return codes are zero or positive: negative values represent signal-terminated programs (ie.: SIGTERM produces -15, SIGKILL produces -9, etc.)

Definition at line 434 of file run.py.

def produtil.run.runsync ( logger = None )

Runs the "sync" command as an exe().

Definition at line 372 of file run.py.

Referenced by hwrf_expt.init_module().

def produtil.run.waitprocs	(	procs,
		logger = `None`,
		timeout = `None`,
		usleep = `1000`
	)

Not implemented: background process monitoring.

Waits for one or more backgrounded processes to complete. Logs to the specified logger while doing so. If a timeout is specified, returns False after the given time if some processes have not returned. The usleep argument is the number of microseconds to sleep between checks (can be a fraction). The first argument, procs specifies the processes to check. It must be a produtil.prog.Pipeline (return value from runbg) or an iterable (list or tuple) of such.

Bug:: produtil.run.waitprocs() is untested

Warning: This is not tested and probably does not work.

Parameters

procs	the processes to watch
logger	the logging.Logger for log messages
timeout	how long to wait before giving up
usleep	sleep time between checks

Definition at line 324 of file run.py.

Variable Documentation

produtil.run.__all__

Initial value:

 = ['alias','exe','run','runstr','mpi','mpiserial','mpirun',
          'runbg','prog','mpiprog','mpiimpl','waitprocs','runsync',
          'InvalidRunArgument','ExitStatusException','checkrun',
          'batchexe','bigexe']

List of symbols exported by "from produtil.run import *".

Definition at line 166 of file run.py.

Detailed Description

Program Types

Serial Execution Syntax

Parallel Execution Syntax

Aliases

Classes

Functions

Variables

Function Documentation

Variable Documentation