Recipes

All LOFAR pipeline recipes are ultimately derived from lofarpipe.cuisine.WSRTrecipe.WSRTrecipe. However, all the functionality it provides is encapsulated and enhanced by the BaseRecipe class, and it is from this that all pipeline recipes should be derived. This class also includes the “ingredients” system, which controls recipe inputs and outputs.

A number of “mix-in” classes may be added to BaseRecipe to provide additional functionality, such as the ability to dispatch jobs to remote hosts (RemoteCommandRecipeMixIn). Recipe authors may mix-in whatever functionality is required to achieve their aims.

The control class provides a recipe with a little extra functionality to help it act as an overall pipeline. This can include interfacing with an external control system, for example, or keeping track of the pipeline progress (StatefulRecipe).

The relationship between all these classes is illustrated below.

(deprecated) lofarpipe.support.control.control lofarpipe.support.lofarrecipe.LOFARrecipe

class lofarpipe.support.baserecipe.BaseRecipe

Provides standard boiler-plate used in the various LOFAR pipeline recipes.

go()

This is where the work of the recipe gets done. Subclasses should define their own go() method, but remember to call this one to perform necessary initialisation.

run_task(configblock, datafiles=[], **kwargs)

A task is a combination of a recipe and a set of parameters. Tasks can be prefedined in the task file set in the pipeline configuration (default: tasks.cfg).

Here, we load a task configuration and execute it. This is a “shorthand” version of lofarpipe.cuisine.WSRTrecipe.WSRTrecipe.cook_recipe().

class lofarpipe.support.stateful.StatefulRecipe

Enables recipes to save and restore state.

This is used exactly as BaseRecipe, but will write a statefile in the job directory, recording the current state of the pipeline after each recipe completes. If the pipeline is interrupted, it can automatically resume where it left off.

To reset the pipeline and start from the beginning again, just remove the statefile.

class lofarpipe.support.control.control

Basic pipeline control framework.

Define a pipeline by subclassing and provding a body for the pipeline_logic().

This class provides little, but can be specialised to eg provide a MAC/SAS interface etc.

pipeline_logic()

Define pipeline logic here in subclasses

class lofarpipe.support.remotecommand.RemoteCommandRecipeMixIn

Mix-in for recipes to dispatch jobs using the remote command mechanism.

See the distribution section for details.

lofarpipe.support.deprecated.ipython.IPythonRecipeMixIn

The use of IPython within the pipeline framework is deprecated.

_schedule_jobs(jobs, max_per_node=None)

Schedule a series of compute jobs. Blocks until completion.

Parameters:

• jobs – iterable of ComputeJob to be scheduled
• max_per_node (integer or none) – maximum number of simultaneous jobs on any given node

Return type:

dict mapping integer job id to ComputeJob