Standard Imaging Pipeline recipes¶
On this page the three toplevel recipes of the LOFAR Automatic Imaging Pipeline for MSSS type observations. The Calibrator pipeline creates an instrument model based on a calibration observation. The instrument model, the calibration solution, is applied to the actual measurements in the target pipeline. These Measurement sets are then used by the imaging pipeline to produce, sky images and a list of sources found in this image. Each of these steps will get more details in each of the chapters
Calibrator Pipeline¶
- class msss_calibrator_pipeline.msss_calibrator_pipeline¶
The calibrator pipeline can be used to determine the instrument database (parmdb) from the observation of a known “calibrator” source. It creates an instrument model of the current LOFAR instrument (As sum of instrumental properties and Ionospere disturbances TODOW). The output of this toplevel pipeline recipe is this instrument model. Which can be used in a later target pipeline calibrate target data.
This pipeline will perform the following operations:
- Preparations, Parse and validate input and set local variables
- Create database (files), A sourcedb with A-Team sources, a vds file describing the nodes, a parmdb for calibration solutions
- DPPP. flagging, using standard parset Demix the relevant A-team sources), using the A-team sourcedb.
- Run BBS to calibrate the calibrator source(s), again using standard parset, and the sourcedb made earlier
- Perform gain correction on the created instrument table
- Create output for consumption by the LOFAR framework
Per subband-group, the following output products will be delivered:
- An parmdb with instrument calibration solution to be applied to a target measurement set in the target pipeline
Recipes of the calibrator pipeline (step)
Target Pipeline¶
- class msss_target_pipeline.msss_target_pipeline¶
The target pipeline can be used to calibrate the observation of a “target” source using an instrument database that was previously determined using the calibrator_pipeline.
This pipeline will perform the following operations:
- Prepare phase, collect data from parset and input mapfiles
- Copy the instrument files to the correct node, create new file with succesfull copied mss.
- Create database needed for performing work: Vds, descibing data on the nodes sourcedb, For skymodel (A-team) parmdb for outputtting solutions
- Run NDPPP to demix the A-Team sources
- Run bss using the instrument file from the target observation, to correct for instrumental effects
- Second dppp run for flaging NaN’s in the MS.
- Create feedback file for further processing by the LOFAR framework (MAC)
Per subband-group, the following output products will be delivered:
- A new MS with a DATA column containing calibrated data
Recipes of the target pipeline (step)
Imager Pipeline¶
- class msss_imager_pipeline.msss_imager_pipeline¶
The Automatic MSSS imager pipeline is used to generate MSSS images and find sources in the generated images. Generated images and lists of found sources are complemented with meta data and thus ready for consumption by the Long Term Storage (LTA)
subband groups The imager_pipeline is able to generate images on the frequency range of LOFAR in parallel. Combining the frequency subbands together in so called subbandgroups. Each subband group will result in an image and sourcelist, (typically 8, because ten subband groups are combined).
Time Slices MSSS images are compiled from a number of so-called (time) slices. Each slice comprises a short (approx. 10 min) observation of a field (an area on the sky) containing typically 80 subbands. The number of slices will be different for LBA observations (typically 9) and HBA observations (typically 2), due to differences in sensitivity.
Each image will be compiled on a different cluster node to balance the processing load. The input- and output- files and locations are determined by the scheduler and specified in the parset-file.
This pipeline performs the following operations:
- Prepare Phase. Copy the preprocessed MS’s from the different compute nodes to the nodes where the images will be compiled (the prepare phase) Combine the subbands in subband groups, concattenate the timeslice in a single large measurement set and perform flagging, RFI and bad station exclusion.
- Create db. Generate a local sky model (LSM) from the global sky model (GSM) for the sources that are in the field-of-view (FoV). The LSM is stored as sourcedb. In step 3 calibration of the measurement sets is performed on these sources and in step 4 to create a mask for the awimager. The calibration solution will be placed in an instrument table/db also created in this step.
- BBS. Calibrate the measurement set with the sourcedb from the gsm. In later iterations sourced found in the created images will be added to this list. Resulting in a selfcalibration cycle.
- Awimager. The combined measurement sets are now imaged. The imaging is performed using a mask: The sources in the sourcedb are used to create an casa image masking known sources. Together with the measurement set an image is created.
- Sourcefinding. The images created in step 4 are fed to pyBDSM to find and describe sources. In multiple itterations substracting the found sources, all sources are collectedin a sourcelist. Step I. The sources found in step 5 are fed back into step 2. This allows the Measurement sets to be calibrated with sources currently found in the image. This loop will continue until convergence (3 times for the time being).
- Finalize. Meta data with regards to the input, computations performed and results are collected an added to the casa image. The images created are converted from casa to HDF5 and copied to the correct output location.
- Export meta data: An outputfile with meta data is generated ready for consumption by the LTA and/or the LOFAR framework.
Per subband-group, the following output products will be delivered:
- An image
- A source list
- (Calibration solutions and corrected visibilities)
Recipes of the Imager Pipeline (step)
aditional recipes
