gainoutliercorrection

*Master Side of the recipe *

class lofarpipe.recipes.master.gainoutliercorrection.gainoutliercorrection

Recipe to correct outliers in the gain solutions of an parmdb, using the program parmexportcal The main purpose of this program is to strip off the time axis information from a instrument model (a.k.a ParmDB) -or- a minimal implementation of the edit_parmdb program. Search all gains for outliers and swap these for the median

1. Validate input
2. load mapfiles, validate if a target output location is provided
3. Call node side of the recipe
4. validate performance, return corrected files

Command line arguments

1. A mapfile describing the data to be processed.
2. A mapfile with target location <mapfiles are validated if present>

Recipe inputs

Parameters:

• executable – (StringField) Full path to the parmexportcal executable, not settings this results in edit_parmdb behaviour (StringField)
• mapfile – (StringField) Full path of mapfile to produce; it will contain a list of the generated instrument-model files (StringField)
• sigma – (FloatField) Clip at sigma * median: activates ‘edit_parmdb’ functionality (FloatField)
• suffix – (StringField) Suffix of the table name of the instrument model (StringField)
• working_directory – (StringField) Working directory used on output nodes. Results will be written here. (StringField)

Recipe outputs (job.results[parameter])

Parameters:mapfile – (FileField) mapfile with corrected parmdbs (FileField)

*Node Side of the recipe *

class lofarpipe.recipes.nodes.gainoutliercorrection.gainoutliercorrection(job_id, host, port)

Perform a gain outlier correction on the provided parmdb. The functionality is based on the edit_parmdb script of John Swinbank.

Outliers in the gain are swapped with the median. resulting gains are written back to the supplied ms:

1. Select correction correction method
2. Call parmexportcal for gain correction
3. use gainoutliercorrect from Swinbank Step are summarized in the functions of this recipe

_filter_stations_parmdb(infile, outfile, sigma)

Performs a gain outlier correction of the infile parmdb with the corrected parmdb written to outfile. Outliers in the gain with a distance of median of sigma times std are replaced with the mean. The last value of the complex array is skipped (John Swinbank: “I found it [the last value] was bad when I hacked together some code to do this”)

_read_polarisation_data_and_type_from_db(parmdb, station)

Read the polarisation data and type from the db.

_convert_data_to_ComplexArray(data, type_pair)

Performs a conversion of a 2d array to a 1d complex valued array. with real/imag values or with amplitude phase values

_swap_outliers_with_median(polarization_data, type_pair, sigma)

Perform the actual find and replace of the outliers Calculation are perform with complex arithmatic therefore the 2d arrays are converted to complex value array of 1 d

_write_corrected_data(parmdb, station, polarization_data, corected_data)

Use pyparmdb to write (now corrected) data to the parmdb