imager_prepare

*Master Side of the recipe *

class lofarpipe.recipes.master.imager_prepare.imager_prepare

Prepare phase master:

1. Validate input
2. Create mapfiles with input for work to be perform on the individual nodes based on the structured input mapfile. The input mapfile contains a list of measurement sets. Each node computes a single subband group but needs this for all timeslices.
3. Call the node scripts with correct input
4. validate performance Only output the measurement nodes that finished succesfull

Command Line arguments:

The only command line argument is the a to a mapfile containing “all” the measurement sets needed for creating the sky images. First ordered on timeslice then on subband group and finaly on index in the frequency range.

Arguments:

Recipe inputs

Parameters:

• asciistat_executable – (ExecField) full path to the ascii stat executable (ExecField)
• mapfile – (StringField) Full path of mapfile; contains a list of the successfully generated and concatenated sub-band groups (StringField)
• msselect_executable – (ExecField) The full path to the msselect executable (ExecField)
• ndppp_exec – (ExecField) The full path to the ndppp executable (ExecField)
• parset – (FileField) The full path to a prepare parset (FileField)
• processed_ms_dir – (StringField) Path to directory for processed measurment sets (StringField)
• raw_ms_per_image_mapfile – (StringField) Path to mapfile containing the raw ms for each producedimage (StringField)
• rficonsole_executable – (ExecField) The full path to the rficonsole executable (ExecField)
• slices_mapfile – (StringField) Path to mapfile containing the produced subband groups (StringField)
• slices_per_image – (IntField) The number of (time) slices for each output image (IntField)
• statplot_executable – (ExecField) The full path to the statplot executable (ExecField)
• subbands_per_image – (IntField) The number of subbands to be collected in each output image (IntField)
• target_mapfile – (StringField) Contains the node and path to target files, defines the number of nodes the script will start on. (StringField)
• working_directory – (StringField) Working directory used by the nodes: local data (StringField)

Recipe outputs (job.results[parameter])

Parameters:

• mapfile – (FileField) path to a mapfile Which contains a list of thesuccessfully generated and concatenated measurement set (FileField)
• raw_ms_per_image_mapfile – (FileField) Path to mapfile containing the raw ms for each producedimage (FileField)
• slices_mapfile – (FileField) Path to mapfile containing the produced subband groups (FileField)

_create_input_map_for_sbgroup(slices_per_image, n_subband_groups, subbands_per_image, idx_sb_group, input_mapfile)

Creates an input mapfile: This is a subset of the complete input_mapfile based on the subband details suplied: The input_mapfile is structured: First all subbands for a complete timeslice and the the next timeslice. The result value contains all the information needed for a single subbandgroup to be computed on a single compute node

_validate_input_map(input_map, output_map, slices_per_image, subbands_per_image)

Return 1 if the inputs supplied are incorrect, the number if inputs and output does not match. Return 0 if correct. The number of inputs is correct iff. len(input_map) == len(output_map) * slices_per_image * subbands_per_image

go()

Entry point for recipe: Called by the pipeline framework

*Node Side of the recipe*

class lofarpipe.recipes.nodes.imager_prepare.imager_prepare(job_id, host, port)

Steps perform on the node:

0. Create directories and assure that they are empty.
1. Collect the Measurement Sets (MSs): copy to the current node.
2. Start dppp: Combines the data from subgroups into single timeslice.
3. Flag rfi.
4. Add addImagingColumns to the casa ms.
5. Concatenate the time slice measurment sets, to a single virtual ms.
6. Filter bad stations. Find station with repeated bad measurement and remove these completely from the dataset.

Members:

_copy_input_files(processed_ms_dir, input_map)

Collect all the measurement sets in a single directory: The measurement sets are located on different nodes on the cluster. This function collects all the file in the input map in the processed_ms_dir Return value is a set of missing files

_run_dppp(working_dir, time_slice_dir_path, slices_per_image, input_map, subbands_per_image, collected_ms_dir_name, parset, ndppp)

Run NDPPP: Create dir for grouped measurements, assure clean workspace Call with log for cplus and catch segfaults. Pparameters are supplied in parset

_concat_timeslices(group_measurements_collected, output_file_path)

Msconcat to combine the time slices in a single ms: It is a virtual ms, a ms with symbolic links to actual data is created!

_run_rficonsole(rficonsole_executable, time_slice_dir, time_slices)

_run_rficonsole runs the rficonsole application on the supplied timeslices in time_slices.

_filter_bad_stations(group_measurements_collected, asciistat_executable, statplot_executable, msselect_executable)

A Collection of scripts for finding and filtering of bad stations:

1. First a number of statistics with regards to the spread of the data is collected using the asciistat_executable.
2. Secondly these statistics are consumed by the statplot_executable which produces a set of bad stations.
3. In the final step the bad stations are removed from the dataset using ms select

REF: http://www.lofar.org/wiki/lib/exe/fetch.php?media=msss:pandeymartinez-week9-v1p2.pdf

run(environment, parset, working_dir, processed_ms_dir, ndppp_executable, output_measurement_set, time_slices_per_image, subbands_per_group, raw_ms_mapfile, asciistat_executable, statplot_executable, msselect_executable, rficonsole_executable)

Entry point for the node recipe