Ingredients

One of the most fragile parts of pipeline definition is ensuring that the inputs to a recipe are correct. Broadly, there are two failure modes here which should be avoided. The first is perhaps the more obvious: it is always desireable to check that the inputs actually make sense before attempting to use them to operate on some piece of data – particularly when that data may be scientifically valuable. By checking inputs when the recipe is started, awkward errors during the recipe run may be avoided.

The second failure mode concerns the source of the inputs. As we have seen, recipe inputs may be provided on the command line, read from a configuration file, calculated by another recipe as part of the pipeline run, etc. It is important that these inputs are presented to the recipe code in a consistent way: if, for example, a float is required, a float should be provided, not a string read from the command line.

All LOFAR recipes define a series of inputs, as described in the recipe design section. These inputs are ultimately derived from the Field class, which provides validation and type-conversion (where appropriate) of the inputs. A number of pre-defined fields are available, covering many basic use cases (see below); the recipe author is also encouraged to defined their own as necessary.

All recipes ultimately derive from RecipeIngredients (see the last section). This provides a number of standard fields, which are present in all recipes, as well as ensuring that additional, per-recipe fields are handled appropriately, including type-checking on recipe instantiation. Within the recipe environment, the contents of fields are available as self.inputs, an instance of LOFARingredient.

Pre-defined fields

class lofarpipe.support.lofaringredient.Field(*opts, **attrs)

Fields provided validation and type checking of input/output.

Unlimited user-defined fields are possible; they should all derive from this class.

coerce(value)

Try to convert value into the appropriate type for this sort of field. Results should be checked with is_valid().

Parameters:value – value to be coerced Return type:coerced value

generate_option(name)

Generated an optparse option.

Parameters:name – Destination Return type:optparse.Option

is_valid(value)

Check whether value is a valid value for this field.

This must be defined in subclasses.

Parameters:value – value to be checked Return type:bool

class lofarpipe.support.lofaringredient.StringField(*opts, **attrs)

A Field which accepts any string as its value.

class lofarpipe.support.lofaringredient.IntField(*opts, **attrs)

A Field which accepts any int as its value.

class lofarpipe.support.lofaringredient.FloatField(*opts, **attrs)

A Field which accepts any float as its value.

class lofarpipe.support.lofaringredient.FileField(*opts, **attrs)

A Field which accepts the name of an extant file.

class lofarpipe.support.lofaringredient.ExecField(*opts, **attrs)

A Field which accepts the name of an executable file.

class lofarpipe.support.lofaringredient.DirectoryField(*opts, **attrs)

A Field which accepts the name of an extant directory.

class lofarpipe.support.lofaringredient.BoolField(*opts, **attrs)

A Field which accepts a bool.

class lofarpipe.support.lofaringredient.ListField(*opts, **attrs)

A Field which accepts a non-string iterable (ie, list or tuple).

class lofarpipe.support.lofaringredient.DictField(*opts, **attrs)

A Field which accepts a dict.

class lofarpipe.support.lofaringredient.FileList(*opts, **attrs)

A Field which accepts a list of extant filenames.

Infrastructure

class lofarpipe.support.lofaringredient.RecipeIngredients

All LOFAR recipes ultimately inherit from this. It provides the basic ingredient structure, as well as the default fields which are available in every recipe.

Recipe inputs

Parameters:

• args – (ListField) Args (ListField)
• config – (FileField) Configuration file (FileField)
• dry_run – (BoolField) Dry run (BoolField)
• job_name – (StringField) Job name (StringField)
• runtime_directory – (FileField) Runtime directory on the head node (FileField)
• start_time – (StringField) [Expert use] Pipeline start time (StringField)
• task_files – (FileList) Task definition file (FileList)
• working_directory – (StringField) Working directory on the compute nodes (StringField)

Recipe outputs (job.results[parameter])

None.

class lofarpipe.support.lofaringredient.RecipeIngredientsMeta(name, bases, ns)

This metaclass ensures that the appropriate instances of Field are available in the inputs of every LOFAR recipe.

class lofarpipe.support.lofaringredient.LOFARingredient(fields)

LOFARingredient provides dict-like access to a group of instances of Field. If a field is defined which does not have a value set, but which does have a default, that is returned.