DPPP

*Master Side of the recipe *

class lofarpipe.recipes.master.dppp.dppp

Runs DPPP (either NDPPP or – in the unlikely event it’s required – IDPPP) on a number of MeasurementSets. This is used for compressing and/or flagging data

1. Load input data files
2. Load parmdb and sourcedb
3. Call the node side of the recipe
4. Parse logfile for fully flagged baselines
5. Create mapfile with successful noderecipe runs

Command line arguments

1. A mapfile describing the data to be processed.
2. Mapfile with target output locations <if procided input and output mapfiles are validated>

Recipe inputs

Parameters:

• clobber – (BoolField) If True, pre-existing output files will be removed before processing starts. If False, the pipeline will abort if files already exist with the appropriate output filenames (BoolField)
• data_end_time – (StringField) End time to be passed to DPPP; used to pad data (StringField)
• data_start_time – (StringField) Start time to be passed to DPPP; used to pad data (StringField)
• demix_always – (ListField) List of sources that must always be demixed (used by demixing step only) (ListField)
• demix_if_needed – (ListField) List of sources that will only be demixed if needed, based on some heuristics (used by demixing step only) (ListField)
• executable – (ExecField) The full path to the relevant DPPP executable (ExecField)
• mapfile – (StringField) Name of the output mapfile containing the names of the MS-files produced by the DPPP recipe (StringField)
• nproc – (IntField) Maximum number of simultaneous processes per output node (IntField)
• nthreads – (IntField) Number of threads per (N)DPPP process (IntField)
• parmdb_mapfile – (StringField) Path to mapfile containing the parmdb files (used by demixing step only) (StringField)
• parset – (FileField) The full path to a DPPP configuration parset. The msin and msout keys will be added by this recipe (FileField)
• sourcedb_mapfile – (StringField) Path to mapfile containing the sourcedb files (used by demixing step only) (StringField)
• suffix – (StringField) Added to the input filename to generate the output filename (StringField)
• working_directory – (StringField) Working directory used on output nodes. Results will be written here (StringField)

Recipe outputs (job.results[parameter])

Parameters:

• fullyflagged – (ListField) A list of all baselines which were completely flagged in any of the input MeasurementSets (ListField)
• mapfile – (FileField) The full path to a mapfile describing the processed data (FileField)

*node Side of the recipe *

class lofarpipe.recipes.nodes.dppp.dppp(job_id, host, port)

Call ndppp with a parset augmented with locally calculate parameters:

1. preparations. set nthreads, Validate input, clean workspace
2. Perform house keeping, test if work is already done
3. Update the parset with locally calculate information
4. Add ms names to the parset, start/end times if availabe, etc.
5. Add demixing parameters to the parset
6. Run ndppp

run(infile, outfile, parmdb, sourcedb, parsetfile, executable, environment, demix_always, demix_if_needed, start_time, end_time, nthreads, clobber)

This function contains all the needed functionality