Data Reduction

After the Starlink project came to an end in mid-2005, the Joint Astronomy Centre (link is external) (JAC) in Hawaii, which like the AAO has a large investment in data reduction pipelines built around Starlink infrastructure, took on the role of maintaining the Starlink Software Collection. The JAC has released a number of improved versions since 2005, the latest being "Hikianalia" (link is external), which is available for 64-bit Linux machines, as well as Macs running OS X 10.6 and 10.7.

This version of the Starlink/ORAC-DR software, like its predecessor, includes the _ADD_AUTO_ASTROMETRY_ primitive. Currently, all IRIS2 images are corrected for astrometric distortion (due to the camera optics) before being aligned and combined into a mosaic. The mosaic's World Coordinate System (WCS) which converts pixel coordinates to Right Ascension and Declination has typically only been good to ~5" or so, depending on when and where the telescope pointing was last checked. Both the IRIS2 recipes JITTER_SELF_FLAT and CHOP_SKY_JITTER now call this _ADD_AUTO_ASTROMETRY_ primitive, which uses SExtractor (link is external) to detect objects in the mosaic, downloads 2MASS Point Source Catalogue (link is external) objects from the CDS (link is external) server, then correlates them and applies a new astrometric solution using a 4 coefficient model (link is external) (we thank Brad Cavanagh at JAC for implementing these facilities). However, in addition, this version includes a new recipe which can provide estimates of a number of image statistics e.g. FWHM of point sources, JHKKs photometric zero-points, extinction and sky brightness. This utilises photometry from the 2MASS catalogue and the near-IR broadband colour transformation equations for IRIS2 (Mk2 array) determined by Stuart Ryder (details in the February 2007 AAO Newsletter). For now, these features can only be invoked by using the JITTER_SELF_FLAT_PS recipe.

New-and-improved recipes available

As announced in the February 2005 AAO Newsletter, there have been some recent significant improvements to some of the imaging recipes, as well as the bad pixel masks, in use at the AAT for on-line data reduction. Improvements include cross-talk correction, linearity correction, bad pixel masking, astrometric distortion correction, and pixel over-sampling. 

V2 of these improvements were released on 25 August 2006, and includes automatic selection of an appropriate bad pixel mask, depending on whether the data was taken using the original (Mk1) science-grade array (Mk1: May 2002 - May 2005), the engineering-grade array (Eng: Jun 2005 - April 2006), or the new science-grade array (Mk2: May 2006 onwards).

From the outset, it was intended that IRIS2 users would have access to a data reduction pipeline that would give them a fair indication, in quasi-real time of the data quality and astronomical content of their data. After considering various options, and taking into consideration the (limited) internally-available resources, the decision was made to adopt ORAC-DR for on-line data reduction. ORAC-DR is a generic data reduction pipeline created at the Joint Astronomy Centre in Hawaii, originally for use with various UKIRT and JCMT instruments. IRIS2 is the first external instrument to be supported by ORAC-DR, and we are grateful to the JAC for making it (and some software effort) available for this purpose.

The advantages of using ORAC-DR include:

  • the availability of recipes for both infrared imaging and spectroscopy. IRIS2 has an identical array to UFTI at UKIRT.
  • the likelihood that UK observers will already have used ORAC-DR at UKIRT.
  • the relative ease with which recipes can be read, understood, and "tweaked" as necessary.
  • the similarity of the computing infrastructure at JAC and AAO (Solaris/Linux workstations with Starlinksoftware installed).
  • the near publication-quality results of some of the more sophisticated recipes.
  • its "fire and forget" nature - once started, ORAC-DR requires next to no user interaction or intervention. It automatically senses new incoming data, processes it in accordance with the recipe (specified either in the FITS header, or over-ridden from the command line), displays the results so far, then waits for the next frame.

For a complete description of what ORAC-DR does, and how it works, you should consult the ORAC-DR WWW pages. Here we give only instructions on how to get it started and some troubleshooting hints. You should also review the IRIS2 observing guide for information on how the various recipes are employed. Please direct any queries or comments you may have about the use of ORAC-DR with IRIS2 to Paul Dobbie rather than contacting JAC directly.

On-line processing at the AAT

The latest version of ORAC-DR runs on aatlxa, a PC in the control room running Centos 5.4 Linux. Log in to this machine as

iris2red (and enter the iris2red password from the blue folder on the shelf above aatlxa)

oracdr_iris2

oracdr -from 1 -loop wait &

then sit back and enjoy! Note that if iris2 has been set-up to run on aatvme14 (as opposed to aatvme10) then before starting oracdr you will need to as type:

setenv ORAC_DATA_IN /data_vme14/aatobs/iris2_data/YYMMDD

where YYMMDD is the current date. To re-process data from a different night, follow the steps below for off-line processing (except the source /applic/... step). If you wish to re-use dome flats from a previous night, follow these instructions.

Reduced data is written to /iris2_reduce/iris2red/<yymmdd>/. If this disk fills up, you should feel free to purge old data directories from previous IRIS2 runs. Do not execute any other Starlink tasks from this window, or it may cause ORAC-DR to fail, or give odd error messages.

Off-line processing on your AAT visitor account

To run ORAC-DR using your visitor account at the AAT, open up a terminal and type

starlink

Then type

oracdr_iris2 ut 

where ut is the UT date on which your data was taken, in the format YYYYMMDD. 

Then, define the IN and OUT folders

setenv ORAC_DATA_IN full directory path where data files are

setenv ORAC_DATA_OUT full directory path where output will go

oracdr command-line options

ORAC-DR will open up a GUI to report status, warnings, and errors, as well as any GAIA image displays (one for the most recent processed frame, and another for completed mosaics) or GWM plotting windows required.

ORAC-DR can be stopped and re-started easily. Simply click "Exit ORAC-DR" in the GUI. It can then be re-started with the oracdr command, with various command-line options. For instance, to re-start from observation number 57 while still taking data, type
oracdr -from 57 -loop wait & ).

You can use gaia to view the reduced images. Simply type

gaia &

 Reprocessing tricks

It is not uncommon at the AAT for exposure sequences to be curtailed by weather, or cloud may interfere with a few exposures from your sequence. Or you may decide you want to process the best seeing images from a sequence seperately. Can you do that? Yes you can - but because ORAC-DR is primarily driven by the content in the file headers, you need to update the raw data files files. There are three ways you can do this - modify the FITS headers directly using a Perl script; copy the raw FITS files to a new location, edit the headers and reprocess these files; or make ORAC-DR create NDF copies of the raw data and edit those.

Modify the raw FITS files in situ

There is now a Perl script orac_fits_fix.pl which can do all this for you.

Usage: orac_fits_fix [-n NOFFSETS] [-r RECIPE] FITS_file1 ... FITS_filen
(if filenames are specified, the number of files should be multiples of NOFFSETS) 
Given a set of files, it will recreate the GRPNUM, GRPMEM etc. keywords so that they can be sequentially processed by recipes such as JITTER_SELF_FLAT, etc. You can also change the NOFFSETS keyword (by default it sets NOFFSETS to the number of files provided on the command line), which determines the number of images that must be acquired before a new flatfield and reduced mosaic is created.

For example, 
orac_fits_fix file[1-9] will fix headers to process as a group of 9 files in a mosaic;
orac_fits_fix -n 4 file[1-8] will fix headers to process as a group of 8 files, making a new flatfield and mosaic after every 4 files;
orac_fits_fix -r JITTER_SELF_FLAT_KEEPBAD file[1-8] will fix headers to make a mosaic of 8 files, with recipe JITTER_SELF_FLAT_KEEPBAD

Copying the raw data and editing the FITS Headers (e.g. in IRAF)

  1. Create a new directory for the files to edit, and copy them. For example suppose we had 15 H-band images taken on 09APR2003 as runs 94-108, with only runs 100-106 having data free from clouds:
    mkdir /iris2_raw/iris2red/030409
    cp /data_vme14/aatobs/iris2_data/030409/09apr010[0-6].fits /iris2_raw/iris2red/030409/
  2. Tell ORAC-DR where to find these files:
    setenv ORAC_DATA_IN /iris2_raw/iris2red/030409
  3. Edit the file headers. You can use whatever FITS header editor you prefer. Suppose you prefer to use IRAF - the hedit task in the images package can edit FITS headers. Start an xgterm, go to the raw data directory and start editing:
    xgterm &
    Then in the new xgterm:
    cl
    cd /iris2_raw/iris2red/030409
    images
    hedit 09apr*.fits field=grpnum value=100 update+ verify-

    will change all the 09apr files to be in group 100 (instead of the group 94 they were before).
    hedit 09apr0100.fits field=grpmem value=1 update+ verify-
    hedit 09apr0101.fits field=grpmem value=2 update+ verify-
    ....
    hedit 09apr0106.fits field=grpmem value=7 update+ verify-

    will update the files 100-106 to be members 1-7 of group 100 (instead of the members 7-13 of group 94 they were before), and
    hedit 09apr*.fits field=noffsets value=7 update+ verify-
    will tell ORAC-DR's JITTER_SELF_FLAT recipe (the one originally specified when the data was taken) to make a flat and mosaic from 7 images, instead of the 15 originally specified. This example will not cover all cases - in some cases if you have complicated interleavings of objects and sky you won't be able to do pipeline reduction of partial data sets. JITTER_SELF_FLAT (especially with random dithering) however is pretty robust so long as you have more than 3 decent frames.
    Other FITS header entries you might need to edit are:
    RECIPE 
    OBSTYPE
     ('OBJECT' or 'SKY' etc.)
    RUN <Run_number>
    OBSNUM <Run_number>
  4. Now reprocess the data (ensuring that you have not invoked CONVERT)
    oracdr -skip -list 100:105,106 &
    The -skip option will be necessary if your run numbers are not contiguous (since otherwise ORAC-DR will sit waiting for the next run, even if it's not there). Your newly processed data will appear in the same directory as usual.
  5. Now if you have redefined ORAC_DATA_IN, make ORAC-DR go back to looking for data in the usual place:
    oracdr_iris2

Editing the FITS headers as NDF files.

  1. Since ORAC-DR only ever operates on NDF copies of the raw data, you can make ORAC-DR create NDF copies of the raw data in the $ORAC_DATA_OUT directory using the QUICK_LOOK recipe:

    oracdr -skip -list 100:105,106 QUICK_LOOK &

    This leaves NDF copies of all the input images in $ORAC_DATA_OUT. You can now edit the fits headers of these files with the kappa fitsmod (link is external) task, but you must not do this from the window in which ORAC-DR is being run. Open a new terminal window, and type:

    cd /iris2_reduce/iris2red/030409 (or wherever $ORAC_DATA_OUT corresponds to)

    kappa
    fitsmod 09apr0100 edit=U keyword=GRPNUM position=! value=100 comment=\$C
     

    etc. Scripts are available if you need to change the group membership, or number of offsets for a large number of files.

  2. The guidelines for the changes you make are the same as item 3 above, as are the ORAC-DR commands at item 4 to actually redo the processing.

A full description of the various suffices attached to processed versions of the data is available from the ORAC-DR home page (link is external) for imaging (link is external) and spectroscopic (link is external) data. For imaging reduction, you will probably be most interested in the mosaicing output, which have filenames of the form

giyyyymmdd_n_mos_a.sdf

where yyyymmdd is the UT date of the file, n is the first frame number of the group of images which go into making up the mosaic pattern, and a is the sub-mosaic number. Thus, for a 9 point jitter on 28 July 2002 UT beginning with frame 21, a reduced sub-mosaic will be created after frame 29 has been obtained, and will be called gi20020728_21_mos_0.sdf, as well as a "grand mosaic" gi20020728_21_mos.sdf. If the 9 point jitter pattern is then repeated immediately (i.e. the value of npatt in the Observing Sequence is >1), then a new sub-mosaic gi20020728_21_mos_1.sdf will be created after frame 38, and then added to the existing grand mosaic gi20020728_21_mos.sdf. Note that the effective exposure time of any sub-mosaic is the same as each of the component frames, i.e., if each frame consists of 10 coadds of 5 seconds each, then the counts in each sub-mosaic are normalised to 5 seconds. However, the grand mosaic formed from adding 3 sub-mosaics will be normalised to the equivalent of 15 seconds exposure time.

For spectroscopy reduction, the raw images (_mraw) have their known bad pixels masked (_bp), then their variance array is updated for read-noise and gain (_rnv, _pov). Spectra taken with the Sapphire_240 grism are flipped laterally (_reo), so that wavelength increases with pixel number. They are then flatfielded (_ff) if requested, and an approximate wavelength scale applied (_wce). (Object  - Sky) pairs are co-added to make the group giyyyymmdd_nn.sdf files. A cross-cut in the spatial (y) axis is produced (_ypr), and an optimal extraction (_oes) is performed about the strongest positive and negative peaks found there. The negative beam is inverted, cross-correlated with the positive beam, then shifted before co-adding (_sp). The co-added beams are then normalised to a 1 second integration time (_nsp), before division by the rectified standard star spectrum (_dbs). Lastly, the spectrum is flux-calibrated based on the known magnitude and spectral type of the standard (_fc), then lightly smoothed and negative pixels clipped (_fcs) to assist with autoscaling of the display.

Using your own recipes and primitives

ORAC-DR will always search the directory defined by the environment variable ORAC_RECIPE_DIR for a given recipe, or else use the default version. Similarly, it will search for primitives within ORAC_PRIMITIVE_DIR, or else use the default version. Thus, you may place your own modified recipes in ~iris2red/recipes/, modified primitives in ~iris2red/primitives/, and then type

setenv ORAC_RECIPE_DIR ~iris2red/recipes/

setenv ORAC_PRIMITIVE_DIR ~iris2red/primitives/

(don't put these in ~iris2red/.login, as subsequent observers may unwittingly end up using your modified recipes/primitives)

oracdr <command line arguments> 

Typing oracdr_iris2 however will undo the above definitions, so you will need to re-define them each night.

Flux Calibration

The end product of the IRIS2 ORAC-DR spectroscopy routines is a flux-calibrated spectrum in units of W/m2/µm. While the relative flux scale along the spectrum should be good to 5% or so, note that IRIS2 does not produce absolute spectrophotometry. First, there are no spectrophotometric standards in the IR, with fluxes tabulated every 20 Å or so as in the optical. Rather, we assume the telluric standard to have a blackbody shape throughout the near-IR, which is a reasonably good approximation in the absence of any surrounding hot dust. Secondly, while both the target and the standard star are observed at a similar airmass, we have to assume that slit losses (dependent on seeing, centering and guiding) are similar. So the absolute flux scale is probably only good to 30% or so.

ORAC-DR is reporting that it cannot find a suitable calibration file

  • FLAT: IRIS2 flat-field calibration files need to have the same filter (and for spectroscopy, the same readout method) as the observations being calibrated.
  • DARK: IRIS2 dark calibration files need to have the same exposure time (as recorded in the FITS keyword EXPOSED) and number of cycles as the observations being calibrated.
  • READNOISE: A readnoise value needs to be calculated at the beginning of each night, from the Array_Tests sequence.
  • STANDARD: A standard star calibration observation needs to be taken in the same filter, and have the same number of axes, as the observations being calibrated.

The methods of remedying these problems depend on which calibration information is missing:

  • FLAT, DARK: Reduce a suitable flat-field or dark frame taken later in the night (see below for information on reducing observations out of turn), then re-reduce the observations in question.
  • READNOISE: Reduce an AAO_Array_Tests sequence, then re-reduce the observations.
  • STANDARD: Reduce a suitable standard star observation, then re-reduce the observations in question.

It is possible to force ORAC-DR to re-use calibration files from a previous night. For instance, the spectroscopic flats are reproducible enough that only one set should be required per run. To re-use flats from 26 Nov 2004 for 28 Nov 2004:
cd /iris2_reduce/iris2red/041128
cp ../041126/flat_* .
cp ../041126/index.flat .

ORAC-DR is 'hanging' - not reducing data, not responding - or keeps crashing on start-up.

While it's possible that ORAC-DR has truly hung, it's much more likely that ORAC-DR is doing CPU-intensive calculations. These typically come at the end of sequences, when ORAC-DR is creating a final mosaic. If you wish to see what ORAC-DR is doing during these moments, run up ORAC-DR with the -verbose option (in addition to other tags). Check the "Warnings" and "Errors" sections of the ORAC-DR GUI to see if there are any messages there about missing files, recipes or primitives (the font colours can be hard to see).

If you wish to run ORAC-DR in a less CPU-intensive mode, try running with the _BASIC recipes. These recipes are specially designed to perform less intensive calculations, specifically in the mosaicing steps where registration and image detection and matching are done. However, with the advent of fast PCs running Linux, ORAC-DR is able to keep up with the data rate from most programs using the full-fledged recipes.

If it won't start up at all, chances are there are some rogue ORAC-DR/Starlink processes still running in the background. The simplest way to eliminate these is with the all-powerful oracdr_nuke command.

If ORAC-DR hangs soon after startup at the "Orac says: Pre-starting mandatory monoliths..." stage, then check that your /etc/hosts file contains the following line:

127.0.0.1 aaa.bbb.ccc localhost localhost.localdomain

where aaa.bbb.ccc is the full domain name of your computer, e.g. "lapsdr.aao.gov.au". Without this line, ORAC-DR cannot connect to Gaia.

JITTER_SELF_FLAT is too slow, I want to run JITTER_SELF_FLAT_BASIC

To re-reduce a set of observations, simply stop ORAC-DR (see above), then restart by adding the recipe name to the command-line options. For example, if you wish to reduce observations 71 to 79 with JITTER_SELF_FLAT_BASIC instead of JITTER_SELF_FLAT, type:

oracdr -list 71:79 JITTER_SELF_FLAT_BASIC

(similarly for any other recipe with a _BASIC variant). This command will tell ORAC-DR to only reduce observations 71 through 79 (inclusive) with recipeJITTER_SELF_FLAT_BASIC. Please note that all observations you specify will be reduced with the given recipe. If you mistakenly included a dark frame in the above example (suppose you typed 70:79), it would not get reduced with the REDUCE_DARK recipe. Generally this is not a problem, as groups will be properly reduced based on header keywords.

Also, you do not necessarily need to stop ORAC-DR. It is possible to start a parallel ORAC-DR session by typing the first startup command in a new xterm, then running ORAC-DR as described above. This will allow for real-time data reduction to continue, but also allow you to run different recipes on pre-existing data. This option will probably be slower than running a single instance of ORAC-DR, as both sessions have to use the same CPU. On a dual-processor machine (like aatssx) this may not be a problem.

I aborted an observation and now ORAC-DR is waiting for the file to appear (or else ORAC-DR fell over and needs to be restarted)

Unfortunately, after aborting an IRIS2 run, the run number does not get reset, and there will be a missing file number. Simply stop ORAC-DR and restart it. For example, if you aborted run 143 and the next available file is 144, type

oracdr -from 144 -loop wait &

This will start ORAC-DR on observation 144 and continue as normal.

GAIA displays an image that is skewed

This problem was noticed during an IRIS2 commissioning run, and no solution is apparent. The data file is fine, GAIA simply has unknown problems in displaying it.

ORAC-DR complains, or aborts when converting files to FITS

The usual error message is something like:

Err:!! No input files found matching the supplied specification. 
Err:!  FITS2NDF: Error converting a FITS file into an NDF. 
Err:!! SAI__ERROR: Error 
Err:Error creating symlink from ORAC_DATA_OUT to /data/sse/2/sdr/odi// 
Err: 
Exiting...

Chances are, the problem is actually that the wrong UT date for the files was given at startup. Re-issue the oracdr_iris2 command with the right UT date.

Alternatively, you may get a warning like:

#335 Err: !! Command interpreter invoked by a call to "system" to execute an external
#335 Err: !     command returned an error status of 256.
#335 Err: !  Command was: $CONVERT_DIR/convertndf from  'TEXT' '/home/sdr/odo/'
#335 Err: !     'map_dist' '.txt' ' ' '/home/sdr/odo/t6054.TEMP_1.NDF_1'
#335 Err: !  NDF_EXIST: Unable to associate an NDF structurewith the 'MAP1' parameter.
#335 Err: !! Aborted attempt to associate an existing NDF structure with the 'MAP1'
#335 Err: !     parameter.
#335 Err: !  Error creating a new CmpMap.
#335 Err: !! PAR__ABORT: Parameter request aborted
#335 Err: Error in obeyw to monolith atools_mon (task=astcmpmap): 146703171
#335 Err: Recipe completed with error status = 146703171
#335 Err: Continuing but this may cause problems during group processing

but it may continue to process data, or crash altogether. This can happen if you have executed other Starlink tasks (e.g. from the convert or kappapackages) from the terminal window where you started ORAC-DR. You will need to retstart ORAC-DR from a fresh login.

ORAC-DR isn't doing what I expected, since I messed up one of the FITS keywords. How do I rectify this?

The solution is outlined in the Correcting Headers (link is external) section of SUN/232 (link is external). Since ORAC-DR only ever operates on the NDF copies of the data, the solution is to pass all the affected images first through ORAC-DR using just the QUICK_LOOK recipe. This leaves NDF copies of all the input images in ORAC_DATA_OUT, where their headers can now be corrected using the kappa fitsmod task. Some of the more important keywords can be modified directly as described above.

These pages contain information on the functionality of the IRIS2 Infrared Imager and Spectrograph. Pages maintained by Chris Lidman (chris.lidman@aao.gov.au).