copier

*Master Side of the recipe *

class lofarpipe.recipes.master.copier.copier

The copier recipe is used to copy paths provided in the source mapfile to the same node as ‘matched’ list provided in the target mapfile. The primairy use is to collect data on computation nodes, which nodes is sometime only specified in the mapfiles. There are Two operations performed by this script 1. COPY the source path to the parent directory of the path provided in the target mapfilem eg: Copy instrument tables next to the measurement sets on which they will applied. To Use this operation set target_dir to “” or do not specify it 2. COLLECT information from source nodes to a central path on different nodes specified in the target mapfile. eg Copy instrument tables from the node they are produces to the same node as the measurement sets privided in the target file BUT place them all in the same dir. Provide a target_dir for this operation: all paths not starting with /, not absolute will be placed in a dir with this name relative to the working dir.

Arguments

A mapfile describing the data to be processed.

Recipe inputs

Parameters:

• allow_rename – (BoolField) Allow renaming of basename at target location (BoolField)
• mapfile – (StringField) full path to mapfile containing copied paths (StringField)
• mapfile_source – (StringField) Full path of mapfile of node:path pairs of source dataset (StringField)
• mapfile_target – (StringField) Full path of mapfile of node:path pairs of target location (StringField)
• mapfiles_dir – (StringField) Path of directory, shared by all nodes, which will be used to write mapfile for master-node communication, (StringField)

Recipe outputs (job.results[parameter])

Parameters:mapfile_target_copied – (StringField) Path to mapfile containing all the succesfull copiedtarget files (StringField)

* Node Side of the recipe *

class lofarpipe.recipes.nodes.copier.copier(job_id, host, port)

Node script for copying files between nodes. See master script for full public interface