[Skip to Content]

AAO

General Links
Professional Links
Australian Human Resources Institute Diversity Awards 2013 Finalist

Cookbook for AAOmega data reduction with 2dfdr


AAOmega data is processed using a revised version of the 2dfdr software package. 2dfdr currently has modes which reduce data for a number of instruments including AAOmega with both the 2df and Spiral IFU top ends, 6dF, FMOS and the older 2dF and Spiral. 2dfdr is under rapid development at the moment in preparation for the HERMES and SAMI instruments and at this time, the full 2dfdr manual, available from the AAOmega manuals page, is out of date. This CookBook is aimed at providing a basic introduction to the reduction process. An updated manual will be available shortly. In the meantime, please contact your support astronomer or the AAOmega team aaomega_instsci@aao.gov.au with any questions about the software or problems with data reduction.

Download the software


2dfdr is rapidly evolving as more is learned about AAOmega data. The user should check the ftp site below to ensure they have the latest version of the software before starting a campaign of data reduction.

The software is available as a set of binary executables available for linux and mac operating systems.

Download the latest tar file from the FTP site ftp://ftp.aao.gov.au/pub/2df/

The file will have a name something like: 2dfdr-linux-5.33.tgz

Unpack the tar file and extract the software to your chosen software directory:

tar -xvzf 2dfdr-linux-5.33.tgz

Running 2dfdr for AAOmega


To run 2dfdr, add the full directory pathname of 2dfdr_install/bin to your PATH (ensuring that you do not have the environmental variables DRCONTROL_DIR or DRCONTROL_ENV already set). To do this:

    tcsh
    set path = ($path /path/to/software/2dfdr_install/bin)
    bash
    export PATH=/path/to/software/2dfdr_install/bin:$PATH

2dfdr should be run in a separate working directory for each set of observations with a particular field plate. A meaningful directory structure for your observing run can save a lot of heartache later on. An example directory structure might be:

    Observing95june05/
        night1/
            field1/
            field1b/
            field2/
            field3/
        night2/
            field1/
            field4/

Note that due to the way the flat and arc frames are used, each independent observation (i.e. with a different configuration of the fibres on the field plate) will require a new directory, even if all you have done is tweak the positions of fibre on a previously observed configuration. Once the AAOmega slit wheel is moved with a change of field plate, a new set of flats and arcs are required for the reduction. Data from multiple repeats of the same field, or for fields that contain some repeat observations can be automatically combined, but this is done after the full reduction of data for each field.

Data from the blue and red arms can be reduced in the same directory, but this is often not easy to work with and so most users create separate ccd_1 and ccd_2 sub directories with blue and red data, respectively.

If you have bias or dark calibration files, these need to be reduced in a separate directory (the reduced, combined output is then copied into the working directory of choice) i.e.

    Observing95june05/
                bias/
		dark/

Move to your working directory of choice and then the software can be started with the command

    drcontrol

This brings up the Front Page window from which you can select from pre-described formats or, by selecting the option at the bottom of the window, the full range of .idx files provided (include those aaomega###.idx where ### is the grating name (e.g. 385R, 580V or 1700D etc.) with which the data were taken).

2dfdr Front Page GUI

If you already know which .idx file you wish to use you can start with the command:

    drcontrol ###.idx

The .idx files are all stored at

    /path/to/software/2dfdr_install/share/2dfdr/*.idx

Users, if required, can make a copy of these instrument (.idx) files in the local directory and modify them to set their own reduction preferences. Not all grating configurations currently have corresponding .idx files.

These commands bring up the main 2dfdr window shown here:

2dfdr Main GUI

Starting a reduction

The basic files needed to reduce AAOmega data are:

Additional frames of various types may be needed, depending on the nature of your data.

Automatic reduction depends on the use of a file naming convention in which the name has a root that is the same for all files. The root name is followed by a four-digit integer run number. Raw data from the AAT conforms to this convention with names of the form 13apr10001.fits (for CCD_1), 13apr20001.fits (for CCD_2). Data from the archive also conforms to the convention though the names are changed to run0001.fts, etc.

In the main window you will see your files, their class and their reduction status.

In the image above we see that the first file, Run 20, is file 02aug10020.fits (which is run 20 for ccd1 from 2nd August). The file is a Multi-Fibre Fibre Flat Field (class MFFFF) frame. The file is Not Reduced and so the status is Not Reduced.

If we hit the Plot button to the right of the file information, we can see the 2D image shown here. This is useful to check that everything looks okay. Note that the full CCD is 2kx4k and so many of the displays you will see during reduction are heavily aliased and will often show strange artifacts which are simply not in the data.

Raw Flat Frame

Reduce the data

The user should now be able to simply hit the Start Auto Reduction button, in the bottom left corner, to reduce the data in the current working directory.

The process runs as follows:

When the process is complete your working directory will contain a set of *red.fits files which are the reduced data, and the combined data will be in the dateccd_combined.fits file. The formats of these multi-extension files are described in detail in the 2dfdr file format page.

If your data overlap in wavelength it is possible to splice them together using the "Splice Red & Blue" option under the Commands menu.

.idx file format


The data reduction process can be tweaked and personalised using the .idx files. These set predefined values for many of the reduction operations that can be activated. The basic .idx file, aaomega.idx, is designed to give quick reduction for real-time analysis and to introduce new users to the reduction system. Additional files are provided for each grating, aaomega####.idx, which provide some grating specific modifications such as the spectral distortion model. Results from data processed using this set of .idx file will provide quality real-time first-look reductions. When performing science reductions, the experienced user can use one of the grating specific .idx files as a template and create their own private .idx files which will implement the best combination of options for their data. Note that resolution settings; wavelength range; type of target and target intensity all modify the optimal reduction strategy for AAOmega data.

The file format for the user-specified .idx file is as follows, with an example given below.

The format of these override commands is:

DRC_OVERRIDE_PAR STRING VALUE REDUCE


where STRING and VALUE are drawn from the options in the tables below.

# My example idx file name
# Load the basic defaults
DRC_INCLUDE aaomega.idx

# Differences from the defaults
#DRC_OVERRIDE_PAR DISTORTION 4.0E-9 REDUCE
#DRC_OVERRIDE_PAR DISTX 975 REDUCE
DRC_OVERRIDE_PAR DISTX 1010 REDUCE
DRC_OVERRIDE_PAR ARCFITORDER 4 REDUCE

One then places the .idx file either in the 2dfdr install directory (where the aaomega.idx file is located) or in the working directory, and runs the software with:

drcontrol myidxfile.idx &

Data reduction options


The various options which can be set when reducing 2dfdr data can be found on a series of tab pages in the 2dfdr main window. The options are broken up into a number of sections, with the goal of keeping related parameters in the same section.

Data
The plotting window
General
Extract
Calib
Sky
Combine
Plots

An overview of the various parameters is given below. This page is intended as a guide, rather than a detailed manual describing each reduction step. Default values for high quality reductions of blue and red low-resolution , low intensity data are shown. Note, these are not the settings found in the .idx files shipped with AAOmega, the pre-packaged files are intended for quick-look reduction to quickly provide the user with data. The more experienced user can use modified .idx files.

Data tab


This is shown in the main 2dfdr window as seen in detail above.

The plotting windows

A user generated plot will produce a window such as that shown below (for a reduced and calibrated ARC frame). This window MUST be cleared, by pressing the 'q' key to quit the window, before new images can be plotted.

Pressing '?' brings up a list of options, shown in the second figure below.

Note that the '<' and '>' keys allow the user to step through plotting reduced spectra, once a first spectrum has been plotted by pressing 'i' over a region of the reduce image.

2dfdr plotting window

Things to note in this reduced low resolution blue arc frame are:
  1. The spectral curvature has been removed completely, arc lines are perfectly vertical, with no ragged edges (those not sitting in a line are unused fibres).
  2. The blue ccd bad columns are clearly visible (exhibiting the opposite spectral curvature since they are contours of constant x-pixel number and not constant wavelength). Check that these bad columns do not damage key spectral features, e.g. the sky lines needed for sky subtraction etc.
  3. In this reduction, there are some unused pixels to the far right of the spectrum. For this reduction, the central wavelength selected for the output data is not well matched to the observed central wavelength. and so a small amount of information has been discarded at the blue end of these spectra in this example.

2dfdr plotting help window

General tab


The General tab covers preprocessing options which are applied prior to extraction of the spectra.

General tab

Parameter Possible value Description .idx keyword Blue Defaults (low-res) Red Defaults
(low-res)
Fit overscan FIT
Median
All CCDs require an overscan correction. FITOVERSCAN FIT FIT
Polynomial Order of Overscan fit Integer The blue AAOmega CCD has a strong gradient in the first ~100 pixels and requires a high order fit. Full 2D bias correction may reduce the order of this fit. OVERSCANORDER 9 9
Subtract Bias Frame True
False
Bias frame subtraction can significantly improve the data quality for the blue camera. The user must ensure an adequate number of bias frames have been observed (30-50 seems to work well) and reduced in a separate directory and the DARKcombined.fits copied..
USEBIASIM True False
Subtract Dark Frame True
False
Dark frame subtraction can also significantly improve the data quality for the blue camera. The user must ensure an adequate number of dark frames have been observed (20 seems to work well) and reduced in a separate directory and the DARKcombined.fits copied.
USEDARKIM True False
Divide Image by Long Slit Flat Field True
False
This option is available but still requires specific additional calibration frames (defocused flat field frames)
USEFLATIM False False
Divide Spectra by Fibre Flat Field True
False
Fibre-to-fibre relative response is corrected for using a fibre flat field frame. USEFFLAT True True
Use Laplacian edge detection CR rejection NO
YES
"OBJECT ONLY"
A quality cosmic ray rejection algorithm following that of van Dokkum 2001 PASP 113 1420 LACOSMIC "OBJECT ONLY" "OBJECT ONLY"
Nsigma for Laplacian CR rejection Real The sigma clipping level for the CR rejection LACOSMICSIG 5.0 5.0
Use Running Median True False USERUNMEDIAN False False
Use Local Average Flat True
False
See note below LAF_FLAG False False
L.A.F Smoothing Parameter Integer The smoothing length if using the local average option for the flat fielding LAF_PAR 10 N.A.
Process Overscan When Reducing BIAS Frames True
False
BIASOVSCAN False False

Note on the local average flat field option: In order to make a fibre flat field free from the signature of the quartz-halogen lamp (complete with sharp dichroic repose features) used to make it, the following steps are taken:

  1. the fibre profiles are extracted from the flat field frame
  2. the flat field spectra are wavelength calibrated onto a common wavelength solution and normalized
  3. an average spectrum is created to represent the lamp response function
  4. the lamp response function is then projected back onto the raw pixel spectra (i.e. wavelength is removed), creating a lamp response function for each fibre
  5. the flat field spectra are divided by 2D lamp response function to remove the features of the quartz-halogen lamp
Unfortunately, for the 6700A dichroic mirror, a low order field angle dependent signature is observed in the flat field due to a light leak in the mirror. This can be successfully removed using a local average estimate for lamp response function, averaging over 10-20 adjacent fibres rather than the full CCD. While this is inherently noisier than using the full CCD, it has fewer systematics for the 6700A dichroic data in the blue arm.

Extract tab


The extraction tab deals with parameters related to the extraction of the fibre traces from the raw 2D CCD frame

Extract tab


Parameter Possible value Description .idx file string Blue Defaults (low-res) Red Defaults
(low-res)
Method TRAM
NEWTRAM
GAUSS
OPTEX
SMCOPTEX
2dfdr as a number of options for extracting the fibre profiles.
TRAM is a quick look extraction.
GAUSS performs a TRAM extraction but using Gaussian summation and pixel weighting to suppress additional read-noise
OPTEX performs an `optimal extraction'. This is required for SPIRAL data, but provides no benefit over GAUSS for MOS data at this time.
SMCOPTEX is Scott Croom's `optimal extraction'.
EXTR_OPERATION GAUSS GAUSS
Optimal extraction scattered light model POLYN
BSPLINE
USER
For the OPTEX extraction, the model fit to the background pedestal fit can be selected as a free parameter. OPTEX_SLMODEL BSPLINE BSPLINE
Optimal Extraction Number of Scattered Light Parameters 0 to 40 For the OPTEX extraction, the order of the background pedestal fit can be selected as a free parameter. OPTEXTR_NSLPARS 1 1
Rotate/Shift to Match YES
NO
FLAT ONLY
Once the tramline map has been traced, the map can be adjusted to match subtle variation in the data. Usually this is done only for the flat field. It should be used if the AAOmega slit was moved during observation, for example between taking twilight flats and science data during a night. MATCH FLAT ONLY FLAT ONLY
Distortion coefficient real For some AAOmega grating and wavelength combinations, the default camera distortion model does not perform well in matching the tram line maps for a flat field. These parameters can be adjusted to improve the fit. It is very unusual to have to adjust these values. DISTORTION 4.0E-9 4.125E-9
X centre of distortion real DISTX 975 1010
Y centre of distortion real DISTY 2048.5 2048.5
Scattered Light Subtraction None
1DFIT
2DFILT
Like all spectrographs, AAOmega suffers some level of scattered light. This can often be subtracted out of images via a low order model fit.
1DFIT - this option fits to blank spaces between fibres along a column of data and creates a low order 2D fit to the scattered light
2DFILT - this option resembles unsharp masking of the full 2D frame. It can work well for low signal data, but will be a very poor approximation for high light levels.
SCATSUB 1DFIT 1DFIT
Subtract Scattered light from Offset Sky Frames True
False
Sometimes it is not necessary to subtract the scattered light from all frames. SUBSKY TRUE TRUE

Calib tab

AAOmega data is usually calibrated using CuAr +FeAr hollow cathode arc lamps and Helium+Neon bulbs. ThAr lamps are available for higher resolutions.

However, due to the feed angle of the light from the calibration system, there can be a slight misalignment (at the 10th of a pixel level) between the calibration arc and the science spectra. This can result in poor sky subtraction (p-Cygni like residual sky line profiles). Hence, for observing modes where OH night sky lines are visible in the data, secondary calibration can be performed fitting to these lines.

Calibrate tab

Parameter Possible value Description .idx file string Blue Defaults (low-res) Red Defaults
(low-res)
Polynomial order for arc fit Integer The order of the wavelength fit depends on the grating choice, wavelength range and available arc lines to fit to.
ARCFITORDER 4 4
Use whale shark matching in arc fit TRUE
FALSE
Use a 2D fitting algorithm on arc data rather than 1D
USE_WSM_ALGOL FALSE FALSE
Wavelength calibrate from sky lines TRUE
FALSE
Due to the way the arc light illuminates AAOmega, a small (<0.1pixel) shift is introduced in the wavelength calibration depending on the field plate position of fibres. This can degrade the effectiveness of the sky subtraction. A correction to place all spectra on a common wavelength solution can therefore be calculated from the OH night sky lines, but only when there are lines visible on the CCD SKYSCRUNCH TRUE TRUE
Polynomial order of sky fitting Integer In the blue, usual only the 5577A O2 line is available to fit to, hence a 1 order fit. In the red, many more lines are usually available. SKYFITORDER 1 4
Calibrate Flux TRUE
FALSE
Not yet implemented CALIBFLUX FALSE FALSE
Flux calibration table String Not available CALIB_TABFILE N.A. N.A.

Sky tab


The Sky tab covers options associated with the sky subtraction methods available.
For sky subtraction, a high signal-to-noise sky spectrum is created by combining the dedicated sky fibres allocated to blank sky positions in each configuration. A number of options govern how the sky spectrum is created and subtracted from the data.

Sky tab

Parameter Possible value Description .idx file string Blue Defaults (low-res) Red Defaults
(low-res)
Throughput Calibrate TRUE
FALSE
The amount of sky to subtract from a given science fibre must usually be determined from the data itself. For data with very little sky (e.g. a short standard star exposures) one may wish to dispense with this correction and skip sky subtraction is it will have little effect on the final data set.
THRUPUT TRUE TRUE
Throughput Calibration Method OFFSKY
SKYLINE(KGB)
SKYFLUX(COR)
SKYFLUX(MED)
The normalization correction is calculated either from an offset-sky/Twilight-flat (usually at high resolution in the blue) or from the night sky emission lines which one is trying to remove. The different SKY methods are detailed in the 2dfdr manual, and outlined below*. TPMETH SKYFLUX(COR) SKYFLUX(MED)
Subtract Sky TRUE
FALSE
Should the system attempt sky subtraction. For some data sets this is not necessary or even possible. For Nod-and-Shuffle data, for example, this step would be redundant. SKYSUB TRUE TRUE
Sky Fibre Combination Operation MEDIAN
MEAN
with 20-30 sky spectra to combine to make an average sky, one could either perform a clipped MEAN, or use a simple MEDIAN. We find the median is usually more robust in practice, with little real loss in signal-to-noise SKYCOMBINE MEDIAN MEDIAN
Use iterative sky subtraction TRUE
FALSE
Shift/scale/smooth sky spectrum to minimise sky residuals ITERSKY FALSE TRUE
Correct for telluric absorption TRUE
FALSE
Correct for telluric absorption, by dividing data by a mean object spectrum in the regions containing atmospheric absorption bands TELCOR FALSE TRUE
Correct Min S/N for telluric absorption TRUE
FALSE
Only spectra with S/N greater than this are included in the mean spectrum for telluric correction TELSN 3.0 3.0
Use PCA after Normal Sky Subtract TRUE
FALSE
Undertake Principle Component Analysis of spectrum components after sky subtraction to remove sky residuals (not recommended for stars or fieldsof targets very close in velocity). PCASKY TRUE TRUE
Number of PCA eigenspectra 0-30 Number of eigenspectra used in PCA analysis PCANUM 10 10
Min wavelength for PCA sky subtraction Real Minimum wavelength of data range over which PCA sky subtraction is applied PCALMIN 5550.0 5700.0
Max wavelength for PCA sky subtraction Real Maximum wavelength of data range over which PCA sky subtraction is applied PCALMAX 5600.0 9000.0

Notes: * SKY methods. The three skyline throughput calibration methods are variations on a theme of the same basic principle. In order to determine how much sky to subtract off from each fibres, the system calibrates the relative sky flux by looking at the flux in night sky emission lines (usually OH, but also some O2, Na and other species), unless the OFFSKY method has been selected, in which case a twilight-flat or Offset-Sky frame must be provided. In all cases, the SKYLINE/FLUX calculates an multiplicative scaling for the sky based on the integrated flux in the sky lines. SKYFLUX(COR) performs a more robust fit to the core of the skyline profiles, which is more robust against PSF variations across the field. SKYFLUX(MED) performs the same task as SKYFLUX(COR) but for red data, which typically has more strong unblended lines to measure than the blue; a median scaling is derived, which is more robust against CCD defects and cosmic rays. SKYLINE(KGB) is an extended implementation (designed and implemented by Karl Glazebrook) which attempts to track PSF variations across the CCD due to the camera optics, minimizing the sky residual globally. This method worked well for 2dF data, but is not recommended for the current AAOmega implementation.

Combine tab


Data from multiple observations of the same fields, and also data from multiple observations of separate fibre configurations (usually with some overlap in the targets, which is being performed to increase a sub-set of exposures times) is routinely performed by 2dfdr.

Typically the data from each camera (red and blue) is combined separately before the spectra are spliced into one continuous spectrum.
Combining of reduced files occurs in 'Auto Reduction' mode when all local object frames have been processed. It can also be done manually using the Combine Reduced Runs item in the 2dfdr Commands menu.

The 2dfdr combine algorithm combines data based on either object name or object location. That is, fibres having the same name (or location) are added and normalised to produce the output. This is to include all objects, whether they are contained within every frame or only a sub-set of the frames. The combine has the following features: NOTE: Currently the combined file exposure time is NOT set properly. Exposure time is given in only one place for a file, the value of the .fits header keyword 'EXPOSED'. This exposure time applies to all fibres within the file. When fibres are combined, this keyword is copied from the first frame - no attempt is made at setting it properly.


Combine tab
Parameter Possible value Description .idx file string Blue Defaults (low-res) Red Defaults
(low-res)
Combine Reduced Data TRUE
FALSE
This option will automatically combine the reduced spectra at the end of a automatic reduction run (that has been started using the START button). A frame named dateccd_combined.fits will be generated in the working directory.
AUTO_COMB TRUE TRUE
Adjust Continuum Levels TRUE
FALSE
This parameter performs a cosmetic adjustment of the spectra, allowing for additive offsets. * COMB_ADJUST TRUE TRUE
Flux Weight NONE
FRAMES
OBJECT
When combining data, a flux weight may be applied to each spectrum. One weight can be applied for the whole frames (best when individual fibres have low signal) or a weight may be determined on an object-by-object basis (best when spectra have high signal to noise). FLUXWT FRAMES FRAMES
Rejection Threshold 1-100 A spurion* sigma-clipping rejection is applied, with the clipping threshold set to this value. CSIGREJ 5.0 5.0
Systematic Uncertainty 1-100 An offset factor allows for systematic uncertainties in the data during clipping. CSYST 0.1 0.1
Smoothing Scale 1-100 To determine weights between spectra, the spectra are smoothed to remove local defects. CSMOOTH 101 101
Combine A/B spectra TRUE
FALSE
For Cross-beam-switched data, with two fibres allocated to a target for observations in the A and B positions, this option will invert the B spectrum and add it to the A spectrum, then set the B spectrum position to zero in the output image. COMBAB TRUE TRUE
Combine Matching Name TRUE
FALSE
If TRUE combine matching names, if FALSE match positions COMBNAME TRUE TRUE
Arm re-scrunched in splicing RED
BLUE
In order to join the red and blue spectra, data must be placed on a common wavelength scale. It is usual to rebin the red data, to preserve the higher resolution in the blue. SCRUNCHARM RED RED
Coadd spliced spectra in overlap region TRUE
FALSE
Combine the spectral pixels in the overlap region. SPLICECOADD TRUE TRUE
Splicing mid-point Angstroms A mid-point about which to join the spectra. SPLICEMIDPOINT 5700.0 5700.0
Include RWSS in extension TRUE
FALSE
Include an extension containing spectra reduced without sky subtraction so that these can be used to calculate the weights when splicing INC_RWSS TRUE TRUE

Notes:
* The "Adjust Continuum Levels" option was first introduced for the original 2dF galaxy redshift survey data. This cosmetic adjustment was found to improve treatment of scattered light in the original 2dF spectrographs, and provided cosmetic improvements for spectra which enhanced the redshift success rate. It should be used with caution when spectrophotometric integrity is of paramount importance.
* The spurions is the quantum particle of erroneous information. Most cosmic rays are caused by spurions.

Plots tab


Plots tab
Parameter Possible value Description .idx file string Blue Defaults (low-res) Red Defaults
(low-res)
Scaling TRUE
FALSE
Auto scale figures at the 95% of pixels level, as opossed to min/max.
AUTO True True
Plot Type GREY
COLOUR
CONTOUR
CVMAG
A number of different plot styles are supported. PLOTTYPE Grey Grey
Pixels per bin Rebin the spectrum before plotting. NBIN 1 1
Remove Sky Residual TRUE
FALSE
FIXSKY FALSE FALSE
Plot Tram Map YES
NO
FLAT ONLY
The user will sometimes want to check that the tramline tracing for the flat field has worked correctly. Usually one does this only for the flat field, and usual only for a few test data sets from any given observing run. PLTMAP FLAT ONLY FLAT ONLY
Plot Combined Sky TRUE
FALSE
Diagnostic plot. SKYPLOT FALSE FALSE
Plot Telluric Correction TRUE
FALSE
Diagnostic plot. TELPLOT FALSE FALSE
Plot Throughput map TRUE
FALSE
Diagnostic plot. THPLOT FALSE FALSE
Plot averaged flat field TRUE
FALSE
Diagnostic plot. AFFPLOT FALSE FALSE
Plot overscan bias level YES
NO
FLAT ONLY
Enable plotting of the overscan bias level BIASPLOT NO NO
Plot background scattered light TRUE
FALSE
Diagnostic plot. BGDPLOT FALSE FALSE
Plot optimal extraction fits TRUE
FALSE
Diagnostic plot. OPTPLOT FALSE FALSE
Plot PCA sky subtraction diagnostics TRUE
FALSE
Diagnostic plot. PCAPLOT FALSE FALSE

RnD tab


This tab collects options currently under research and development and are generally not required by users and so are not described in detail here.

Plots tab

Sarah Brough (sb@aao.gov.au)