ORAC-DR for IRIS2

Background

ORAC-DR is an online Data Reduction Pipeline, delivered as part of the ORAC project for the United Kingdom Infrared Telescope (UKIRT) by the Joint Astronomy Centre (JAC) and the UK Astronomy Technology Centre (ATC). Though originally intended primarily as a replacement for the existing, but somewhat inflexible, UKIRT data reduction packages CGS4DR and IRCAMDR, it has proved to be sufficiently flexible that it is now also routinely used for the real-time analysis of data from the SCUBA submillimetre camera at the James Clerk Maxwell Telescope (JCMT).

ORAC-DR is intended to run with a minimum of observer interaction; whenever new data is received from the instrument, it will be processed in accordance with one of a number of pre-defined "recipes". These recipes can be quite basic (for the purposes of "quick-look" reduction) or more sophisticated (e.g., publication-quality wide-field mosaicing using on-the-fly flatfields and fully-registered images). The pipeline software itself is written in Perl, and runs on either Unix or Linux platforms. The recipes themselves call up a number of Starlink "monoliths" (e.g., routines from CCDPACK and KAPPA). A useful summary of what ORAC-DR is (and isn't) from the observers point of view is given in this document by Andy Adamson. The pipeline and recipes are relatively robust, in the sense that it can operate on broken or incomplete sequences of observations (e.g., if one or more images from a mosaic sequence are lost, the software can skip them and work with the data it does have).

Since quite a lot of time and effort has gone into developing ORAC-DR for use with the UFTI camera at UKIRT (which uses an identical array to that intended for IRIS2), and since many potential IRIS2 users from the PATT side will already have had some experience with it, it makes a lot of sense to consider using as much of it as possible for IRIS2. This requires mainly that we give some thought to the standard observing modes to be used with IRIS2 (so that pre-existing recipes can be used or adapted), and that the image headers record certain key parameters (e.g., whether the observation is a dark, object, or sky frame; what the telescope offsets are from the base position; etc.). While the pipeline can be run manually (process a subset of images using a recipe different from the default), it is most efficient when all the information it needs is already in the data. Further details about what ORAC-DR does are available in these (local) copies of SUN 230: ORAC-DR: Overview and General Introduction and ORAC-DR - imaging data reduction. I strongly urge you to read this before going on to the next section. More extensive documentation describing the requirements for ORAC-DR are available from the ORAC-DR documentation index. The purpose of this document is to focus attention on the work that needs to be done to implement ORAC-DR at the AAO for use with IRIS2.

Observing Modes and Recipes

To a certain degree, the data reduction recipes dictate the way in which the observations are made in the first place. While this may seem restrictive, it is worth bearing in mind that much of what we do in astronomy is very repetitive, and follows a "recipe" anyway. It is worth considering the various standard modes of observing with UFTI and with IRCAM at UKIRT, since matching recipes are already available for these. However, we are by no means obligated to adopt these. In any case, what works well on the summit of Mauna Kea at 4200 metres may not work so well for Siding Spring at 1164 metres!

Let's take a look at the stock observing modes used with UFTI:

ARRAY_TESTS - calculates read-noise and dark current from a series of bias and dark frames.
NIGHT_LOG - scans the image headers and produces a text log file of the night's observations.
REDUCE_DARK - masks bad pixels, and files this frame as a dark in the index file.
SKY_FLAT - takes any 5 "jittered" frames and median-combines them to produce a flatfield, and files this in the index file.
SKY_FLAT_MASKED - same as above, but any objects detected are masked and not used in the median processing, thus giving improved flatness in crowded fields or with bright stars.
BRIGHT_POINT_SOURCE - mosaics a 5-point jitter observation of a point source ("bright" here usually means a photometric standard star), but requires a separate flatfield first be created with SKY_FLAT(_MASKED).
BRIGHT_POINT_SOURCE_APHOT - as above, but also performs aperture photometry of the brightest source in each frame, as well as in the final mosaic.
JITTERn_SELF_FLAT - mosaics an n-point jitter observation, where n=3, 5, or 9. The flatfield is created by median-combining all n frames, with detected objects masked out.
JITTERn_SELF_FLAT_APHOT - as above, but then also performs aperture photometry of the brightest source in each frame, as well as in the final mosaic.
JITTERn_SELF_FLAT_NO_MASK - as above, but object masking is not used when making the flatfield, in order to speed up execution.
JITTERn_SELF_FLAT_BASIC - as above, but object masking in the flatfield, as well as proper image registration and resampling are skipped in order to further speed up execution (telescope offsets are assumed to be accurate, and used for the mosaicing - usually good for a "quick look").
QUADRANT_JITTER - mosaics a 4-point jitter in which the object of interest is placed in the centre of each quadrant of the array in turn. The flatfield is created by masking out the quadrant in which the object is known to be, and then median-combining the three "good" pixel values in the other images. Useful for objects which are slightly extended, but no bigger than one quadrant of the array.
EXTENDED_nxn - takes an nxn grid of overlapping images (where n=3 or 5) and mosaics them together. Offset sky frames are taken alternately with object frames. Each object frames is sky-subtracted using the average of the bracketing sky frames. A new flatfield is created for each row of the mosaic from the n+1 sky frames interspersed with that row.
EXTENDED_nxn_BASIC - as above, but uses telescope offsets in place of accurate image registration and resampling, in order to speed up execution.

Other recipes exist for imaging sources moving at non-siderial rates (comets, asteroids, satellites) and for observations at thermal wavelengths with chopping, but the above gives a fair overview of the most commonly used observing modes and recipes.

Required Header Information

ORAC-DR will operate (within reason) on data having only a minimum of information contained in the header. However, optimum performance and minimum user interaction requires the presence of at least two keywords in the image header. The first of these is RECIPE, set to one of the available recipes. If this is not present, then it can be specified on the command line when ORAC-DR is run on a specific group of files (or if RECIPE is defined, its value is overridden by that on the command line). The RECIPE is normally set by the observer when preparing their observation sequence (called an Exec at UKIRT, akin to the OFFSET_RUN files used at the AAT). Here is an example Exec used with UFTI at UKIRT:

CONFIG K_60                # configure UFTI using config K_60
DRRECIPE REDUCE_DARK       # use this recipe, must be upper case!
DARK                       # take a dark frame
DRRECIPE JITTER9_SELF_FLAT # use this recipe, must be upper case!
STARTGROUP                 # reduce following as a new dataset
DO 1 JITTER_9_20AS         # do system 9-point jitter exec once
SET DARK                   # blank off UFTI - important!
DRRECIPE QUICK_LOOK        # set to display-only as default

The concept of a "group" is also important to ORAC-DR. Every observation is considered to be part of a group. If the observation is a dark frame, then it is in a group by itself. A sequence of observations which are all part of a jitter pattern, or a mosaic, are considered to be part of the same group, and the group number will be the observation number of the first frame in that sequence. Fully-reduced mosaics have filenames of the form g<date>_<group_number>_mos. The command STARTGROUP above indicates that the next observation (let's say observation 140) should have the keywords GRPNUM and OBSNUM set to 140, and the keyword GRPMEM set to 1. The following observation (141) will also have GRPNUM set to 140, but OBSNUM will be 141, and GRPMEM will be 2 (and so on, until the next occurrence of STARTGROUP). The other header keywords required as a minimum are: GRPMAX, the maximum number of group members (this can be 0 if not known a priori); OBSTYPE, which describes the type of observation (BIAS, DARK, ARC, FLAT, OBJECT, SKY); and STANDARD, whether or not the target is a recognised photometric or spectroscopic standard (ignored unless OBSTYPE is "OBJECT"). A full rundown of the required and optional keywords is contained in orac016, available from the ORAC-DR documentation index.

Issues

Our file naming convention, directory structure, file format, and FITS keyword names may not match those of the JAC. For example, raw UFTI frames have names of the form ufti_data/19990330/raw/raw/f19990330_00042.fits, while raw IRCAM3/TUFTI frames have names of the form ircam_data/20000130/rodir/ro000130_95.sdf). But since ORAC-DR can handle these two instruments equally well, then we should be able to accommodate whatever conventions are (or have already been) adopted for IRIS2.
(Need to write a "oracdr_iris2" startup script and a _IRIS2_HELLO_ primitive, based on those for UFTI, then modify the various ORAC Perl modules to include IRIS2. JAC have offered to help with this, partly to aid in keeping a unified source tree released by them to Starlink.)
There are a number of ways in which ORAC-DR can be told when a new file has finished being written, and is ready to be processed; which is most appropriate for us?
(Since IRIS2 will write the FITS file to a dummy file, then rename it when complete, the most efficient way is to use the "-loop flag" option, with the file itself as the flag file)
How will guiding and offsetting be performed with IRIS2?
(The original plan for an internal guider with IRIS2 had to be dropped. Instead, the existing guide probe arrangement will be used. Although guiding can be commanded on and off, there is no feedback from the system to indicate whether the guide star has been reacquired. Also, telescope offsets beyond a certain amount will be outside the range of travel of the guide probe, and so no guiding will be possible. At the 0.45" pixel scale (f/8), and with the AAT's excellent tracking capability, this may not be much of a problem, but at 0.24" (f/15) or 0.10" (f/36) it may well be.)
To what degree should we let the existing UKIRT ORAC-DR recipes and observing modes dictate our own?

Last Updated: 8 March 2000 by Stuart Ryder