Represents a single stage of a pipeline to execute. More...

Detailed Description

Represents a single stage of a pipeline to execute.

This is a linked list class used to store information about a program or pipeline of programs to be run. It has the capability of converting itself to a Pipeline object (run(Runner)), or converting itself to a POSIX sh command (Runner.to_shell()). Note that some commands cannot be represented in POSIX sh, such as commands with non-ASCII characters or commands that have Python streams as their stdout or stdin. Those commands can still be run with a Pipeline, but trying to convert them to a POSIX sh command will throw NotValidPosixSh or a subclass thereof.

Definition at line 299 of file prog.py.

Inheritance diagram for produtil.prog.Runner:

Public Member Functions
def	__init__ (self, args, kwargs)
	Creates a new Runner. More...

def	getthreads (self)
	Returns the number of threads requested by this program. More...

def	setthreads (self, nthreads)
	Sets the number of threads requested by this program. More...

def	delthreads (self)
	Removes the request for threads. More...

def	first (self)
	Returns the first Runner in this pipeline. More...

def	prerun (self, arg)
	Adds a function or callable object to be called before running the program. More...

def	__getitem__ (self, args)
	Add one or more arguments to the executable. More...

def	__str__ (self)
	Alias for repr()

def	__repr__ (self)
	Attempts to produce valid Python code to represent this Runnable. More...

def	__eq__ (self, other)
	Returns True if the other object is a Runner that is equal to this one, and False otherwise. More...

def	isplainexe (self)
	Returns true if this is simply an executable with arguments (no redirection, no prerun objects, no environment modification, no piping), and False otherwise. More...

def	cd (self, dirpath)
	Requests that this process run in the specified directory. More...

def	__lt__ (self, stdin)
	Connects the given object to stdin, via inp(stdin,string=False). More...

def	__gt__ (self, stdout)
	Connects the given object to stdout, truncating it if it is a file. More...

def	__lshift__ (self, stdin)
	Sends the specified string into stdin. More...

def	__rshift__ (self, stdout)
	Appends stdout to the specified file. More...

def	__pos__ (self)
	Sends stderr to stdout. More...

def	__ge__ (self, outerr)
	Redirects stderr and stdout to the specified file, truncating it. More...

def	__or__ (self, other)
	Pipes this Runner to the other Runner. More...

def	argins (self, index, arg)
	Inserts the specified argument before the given index. More...

def	args (self)
	Iterates over the executable and arguments of this command.

def	copy
	Returns a deep copy of this object, almost. More...

def	copyenv (self)
	Instructs this command to duplicate the parent process environment (the default). More...

def	clearenv (self)
	Instructs this command to start with an empty environment except for certain critical variables without which most programs cannot run. More...

def	env (self, kwargs)
	Sets environment variables for this Runner. More...

def	to_shell (self)
	Returns a string that expresses this object as a POSIX sh shell command if possible, or raises a subclass of NotValidPosixSh if not. More...

def	runner (self)
	Returns self if self is modifiable, otherwise returns a modifiable copy of self. More...

def	pipeto (self, other)
	Specifies that this Runner will send its stdout to the other runner's stdin. More...

def	inp
	Specifies that the first Runner in this pipeline takes input from the given file or string specified by stdin. More...

def	out
	Specifies that this process sends output from its stdout stream to the given file or stream. More...

def	err2out (self)
	Sends stderr to stdout. More...

def	err
	Specifies that this process sends output from its stderr stream to the given file or stream. More...

Constructor & Destructor Documentation

def produtil.prog.Runner.__init__	(	self,
		args,
		kwargs
	)

Creates a new Runner.

The only non-keyword argument can be one of three things:

A Runner to copy. Every aspect of the Runner that can be copied will be. Note that if a stream-like object is connected to stdin, stdout or stderr, it will NOT be copied.
A list of strings. This will be used as the command path, and arguments.

Many options can be set via keyword arguments:

clearenv=True - the environment should be cleared before running this command. Any arguments set by the env= keyword or the .env(...) member function ignore this. Also, PATH, USER, LOGNAME and HOME are retained since most programs cannot run without them.
env=dict(var=value,...) - a dict of environment variables to set before running the Runner. Does NOT affect this parent's process, only the child process.
in=filename - a file to send to stdin.
instr=str - a string to send to stdin
out=filename - a file to connect to stdout. Will truncate the file.
outa=filename - same as "out=filename," but appends to the file.
err2out - redirects stderr to stdout
err=filename - a file to connect to stderr. Will truncate the file.
erra=filename - same as "err=filename," but appends to the file.
prerun=[obj,anotherobj,...] - sent to self.prerun, this is a list of functions or callable objects to run before executing the process. The objects are not called until execution is requested via self._gen.
Parameters

args the arguments to the program

kwargs other settings (see constructor description).

Definition at line 311 of file prog.py.

Member Function Documentation

def produtil.prog.Runner.__eq__	(	self,
		other
	)

Returns True if the other object is a Runner that is equal to this one, and False otherwise.

Parameters

other the object to compare

Definition at line 509 of file prog.py.

def produtil.prog.Runner.__ge__	(	self,
		outerr
	)

Redirects stderr and stdout to the specified file, truncating it.

Same as err2out().out(filename,append=False)

Parameters

outerr the stdout and stderr file

Returns: self

Definition at line 566 of file prog.py.

def produtil.prog.Runner.__getitem__	(	self,
		args
	)

Add one or more arguments to the executable.

Can ONLY accept strings, ints, floats or iterables (tuple, list). Strings, ints and floats are sent to _stringify_args, and the result is added to the end of the list of arguments to the command to run. For iterables (tuple, list), adds all elements to the list of arguments, passing each through _stringify_args.

Parameters

args	one or more arguments to add

Returns: self

Definition at line 453 of file prog.py.

def produtil.prog.Runner.__gt__	(	self,
		stdout
	)

Connects the given object to stdout, truncating it if it is a file.

Same as out(stdout,append=False).

Parameters

stdout the stdout object

Returns: self

Definition at line 544 of file prog.py.

def produtil.prog.Runner.__lshift__	(	self,
		stdin
	)

Sends the specified string into stdin.

Same as inp(stdin,string=True).

Parameters

stdin the stdin file

Returns: self

Definition at line 550 of file prog.py.

def produtil.prog.Runner.__lt__	(	self,
		stdin
	)

Connects the given object to stdin, via inp(stdin,string=False).

Parameters

stdin the stdin object

Returns: self

Definition at line 539 of file prog.py.

def produtil.prog.Runner.__or__	(	self,
		other
	)

Pipes this Runner to the other Runner.

Same as pipeto(other).

Returns: other

Parameters

other the other runner to pipe into

Definition at line 572 of file prog.py.

def produtil.prog.Runner.__pos__ ( self )

Sends stderr to stdout.

Same as err2out().

Returns: self

Definition at line 562 of file prog.py.

def produtil.prog.Runner.__repr__ ( self )

Attempts to produce valid Python code to represent this Runnable.

Generally, that can be done, unless an input string is too long, or a stream is connected to a Python object. In those cases, human-readable representations are given, which are not exactly Python code.

Definition at line 473 of file prog.py.

Referenced by produtil.prog.Runner.__str__().

def produtil.prog.Runner.__rshift__	(	self,
		stdout
	)

Appends stdout to the specified file.

Same as out(stdout,append=True).

Parameters

stdout the stdout file

Returns: self

Definition at line 556 of file prog.py.

def produtil.prog.Runner.argins	(	self,
		index,
		arg
	)

Inserts the specified argument before the given index.

This function is intended for internal use only. It is used to implement threading on Cray, where arguments relating to threading have to be added after the Runner is generated.

Warning: It is generally not safe to call this function outside the produtil.mpi_impl subpackage since its modules may generate completely different commands than you asked in order to execute your requested programs.

Parameters

arg	a string argument to add
index	the index to insert before

Note: Index 0 is the executable, while later indices are arguments.

Definition at line 578 of file prog.py.

def produtil.prog.Runner.cd	(	self,
		dirpath
	)

Requests that this process run in the specified directory.

The directory must already exist before the program starts.

Parameters

dirpath the directory to cd into, which must already exist.

Returns: self

Definition at line 532 of file prog.py.

Referenced by produtil.prog.Runner.__init__().

def produtil.prog.Runner.clearenv ( self )

Instructs this command to start with an empty environment except for certain critical variables without which most programs cannot run.

(Retains PATH, USER, LOGNAME and HOME.)

Returns: self

Definition at line 636 of file prog.py.

Referenced by produtil.prog.Runner.__init__().

def produtil.prog.Runner.copy	(	self,
		typeobj = `None`
	)

Returns a deep copy of this object, almost.

If stdin, stdout or stderr are connected to streams instead of files or strings, then the streams are not copied. Instead, the exact same stream objects are connected to the same unit in the new Runner.

Parameters

typeobj the type of the new object or None for Runner. Do not set this unless you know what you're doing.

Returns: the new object

Definition at line 602 of file prog.py.

Referenced by produtil.prog.ImmutableRunner.runner().

def produtil.prog.Runner.copyenv ( self )

Instructs this command to duplicate the parent process environment (the default).

Returns: self

Definition at line 629 of file prog.py.

def produtil.prog.Runner.delthreads ( self )

Removes the request for threads.

Definition at line 406 of file prog.py.

def produtil.prog.Runner.env	(	self,
		kwargs
	)

Sets environment variables for this Runner.

The variables should be specified as keyword arguments.

Parameters

kwargs varname=value arguments

Returns: self

Definition at line 668 of file prog.py.

def produtil.prog.Runner.err	(	self,
		stderr,
		append = `False`
	)

Specifies that this process sends output from its stderr stream to the given file or stream.

The stderr object must be a string filename, or a stream. If append=False, and the stderr is a filename, the file will be truncated, if append=True then it is appended. Raises MultipleStderr if the stderr location is already specified.

Parameters

stderr	the stderr output file
append	if True, append to the file otherwise truncate

Returns: self

Definition at line 811 of file prog.py.

Referenced by produtil.prog.Runner.__init__().

def produtil.prog.Runner.err2out ( self )

Sends stderr to stdout.

Returns: self

Definition at line 802 of file prog.py.

Referenced by produtil.prog.Runner.__ge__(), produtil.prog.Runner.__init__(), and produtil.prog.Runner.__pos__().

def produtil.prog.Runner.first ( self )

Returns the first Runner in this pipeline.

Definition at line 413 of file prog.py.

def produtil.prog.Runner.getthreads ( self )

Returns the number of threads requested by this program.

Definition at line 399 of file prog.py.

def produtil.prog.Runner.inp	(	self,
		stdin,
		string = `False`
	)

Specifies that the first Runner in this pipeline takes input from the given file or string specified by stdin.

If string=True, then stdin is converted to a string via str(), otherwise it must be a filename or a stream. Raises MultipleStdin if the stdin source is already specified.

Parameters

stdin	the input file or string
string	if True, stdin is a string. Otherwise, it is a file.

Returns: self

Definition at line 753 of file prog.py.

Referenced by produtil.prog.Runner.__lshift__(), produtil.prog.Runner.__lt__(), and produtil.prog.Runner.pipeto().

def produtil.prog.Runner.isplainexe ( self )

Returns true if this is simply an executable with arguments (no redirection, no prerun objects, no environment modification, no piping), and False otherwise.

Definition at line 524 of file prog.py.

def produtil.prog.Runner.out	(	self,
		stdout,
		append = `False`
	)

Specifies that this process sends output from its stdout stream to the given file or stream.

The stdout object must be a string filename, or a stream. If append=False, and the stdout is a filename, the file will be truncated, if append=True then it is appended. Raises MultipleStdout if the stdout location is already specified

Parameters

stdout	the stdout file
append	if True, append to the file, otherwise truncate

Returns: self

Definition at line 780 of file prog.py.

Referenced by produtil.prog.Runner.__ge__(), produtil.prog.Runner.__gt__(), and produtil.prog.Runner.__rshift__().

def produtil.prog.Runner.pipeto	(	self,
		other
	)

Specifies that this Runner will send its stdout to the other runner's stdin.

This will raise MultipleStdout if this Runner's stdout target is already specified, or MultipleStdin if the other's stdin is already specified.

Parameters

other the runner to pipe into

Returns: other

Definition at line 721 of file prog.py.

Referenced by produtil.prog.Runner.__or__().

def produtil.prog.Runner.prerun	(	self,
		arg
	)

Adds a function or callable object to be called before running the program.

The callables should be very fast operations, and are executed by self._gen when creating the Pipeline. They take, as an argument, the Runner and an optional "logger" keyword argument that is either None, or a logging.Logger to use to log messages.

Parameters

arg	a callable object that takes self as an argument, and an optional keyword argument "logger" with a logging.Logger for log messages

Definition at line 419 of file prog.py.

Referenced by produtil.prog.Runner.__init__(), and produtil.prog.Runner.err().

def produtil.prog.Runner.runner ( self )

Returns self if self is modifiable, otherwise returns a modifiable copy of self.

This is intended to be used to implement unmodifiable subclasses of Runner

Returns: self

Definition at line 715 of file prog.py.

Referenced by produtil.prog.ImmutableRunner.argins(), produtil.prog.ImmutableRunner.pipeto(), and produtil.prog.ImmutableRunner.runner().

def produtil.prog.Runner.setthreads	(	self,
		nthreads
	)

Sets the number of threads requested by this program.

Definition at line 402 of file prog.py.

def produtil.prog.Runner.to_shell ( self )

Returns a string that expresses this object as a POSIX sh shell command if possible, or raises a subclass of NotValidPosixSh if not.

Definition at line 679 of file prog.py.

The documentation for this class was generated from the following file:

/mnt/pan2/projects/hwrfv3/Samuel.Trahan/trunk/ush/produtil/prog.py

args	the arguments to the program
kwargs	other settings (see constructor description).

Detailed Description

Public Member Functions

Constructor & Destructor Documentation

Member Function Documentation