How the pipeline detects new data

The pipeline starts from looking for observation number 1, unless another number has been specified via the -from or -list options.

The various -loop options determine what the criterion is for concluding that the observation it is waiting for has indeed arrived.

-loop wait

If you use this option, the pipeline monitors the size of the file that is is expecting. For example, if it has just reduced observation number 41, it waits for observation number 42 to appear on disk and watches it size growing as it is being written out by the data handling system. If the file does not grow in size for a certain amount of time it concluded that readout is complete and proceeds with reducing it. Obviously this method is not very robust if the pipeline is operating or network-mounted disks or with acquisition systems that are prone to stalling during readout. However it may be the only option for online data reduction with some data handling systems. This option should be used with SCUBA and IRCAM.

-loop flag

This option instructs the pipeline to monitor not the data file itself, but a ``flag'' file whose appearance indicates readout completion. Typically this is a zero-length file written by the data acquisition after the data file writing is done. This is most robust in architectures where there is no chance of a data file being written without a flag file or vice versa. This option should be used with UFTI, the WFS and MICHELLE.

-loop inf

Under this option, the pipeline reduces data assuming it is available and keeps going one observation at a time until no more data is to be found (or infinity, whichever comes first!), at which point it terminates. It overrides the -to option. This is suitable for offline data reduction of any instrument and is the default option if none is specified.

-loop list

The specified data frames (and/or range of observations) are assumed to be available, are reduced and then the pipeline exits. This option is implied by usage of the -list option, or usage of the -from and -to options in the same invocation. It is unlikely that a user will need to explicitly specify it.