Data Reduction

Cookbook for data reduction with 2dfdr

AAOmega data is reduced using the 2dfdr software package. This CookBook is aimed at providing a basic introduction to the reduction process. An updated manual will be available shortly. In the meantime, please contact your support astronomer or the AAOmega team aaomega_instsci@aao.gov.au with any questions about the software or problems with data reduction.

Installing 2dfdr

This is described in the general 2dfdr webpages.

Starting a reduction

Move to your working directory of choice and the software can be started with the command

>    drcontrol

This brings up the Front Page window from which you can select from pre-described formats or, by selecting the option at the bottom of the window, the full range of .idx files provided (include those aaomega###.idx where ### is the grating name (e.g. 385R, 580V or 1700D etc.) with which the data were taken).

2dfdr Front Page GUI

If you already know which .idx file you wish to use you can start with the command:

    drcontrol ###.idx

The .idx files are all stored at

    /path/to/software/2dfdr_install/share/2dfdr/*.idx

Users, if required, can make a copy of these instrument (.idx) files in the local directory and modify them to set their own reduction preferences. Not all grating configurations currently have corresponding .idx files.

These commands bring up the main 2dfdr window shown here:

2dfdr Main GUI

 The basic files needed to reduce AAOmega data are:

  • a multi-fibre flat field exposure (class MFFFF) These exposures are made with a quartz lamp that provides a uniform spectrum. They are used to calibrate the detector, and to find the centre and profile of each spectrum.
  • an arc exposure (class MFARC) These exposures are made with lamps having various known strong lines. They are used to calibrate the central wavelength and dispersion.
  • one or more science frames (class MFOBJECT)

Additional frames of various types may be needed, depending on the nature of your data.

Automatic reduction depends on the use of a file naming convention in which the name has a root that is the same for all files. The root name is followed by a four-digit integer run number. Raw data from the AAT conforms to this convention with names of the form 13apr10001.fits (for CCD_1), 13apr20001.fits (for CCD_2). Data from the archive also conforms to the convention though the names are changed to run0001.fts, etc.

In the main window you will see your files, their class and their reduction status.

In the image above we see that the first file, Run 20, is file 02aug10020.fits (which is run 20 for ccd1 from 2nd August). The file is a Multi-Fibre Fibre Flat Field (class MFFFF) frame. The file is Not Reduced and so the status is Not Reduced.

If we hit the Plot button to the right of the file information, we can see the 2D image shown here. This is useful to check that everything looks okay. Note that the full CCD is 2kx4k and so many of the displays you will see during reduction are heavily aliased and will often show strange artifacts which are simply not in the data.

Raw Flat Frame

Reduce the data

The user should now be able to simply hit the Start Auto Reduction button, in the bottom left corner, to reduce the data in the current working directory.

The process runs as follows:

  • Reduce any and all multi-fibre flat field frames
  • Reduce any and all arc frames
  • Re-reduce the flat field frames using the accurate wavelength solution obtained from the arc frame reduction to compute a better average illumination correction
  • Reduce any and all science frames
  • Combine the science frames

When the process is complete your working directory will contain a set of *red.fits files which are the reduced data, and the combined data will be in the dateccd_combined.fits file.

If your data overlap in wavelength it is possible to splice them together using the "Splice Red & Blue" option under the Commands menu.

.idx file format

The data reduction process can be tweaked and personalised using the .idx files. These set predefined values for many of the reduction operations that can be activated. The basic .idx file, aaomega.idx, is designed to give quick reduction for real-time analysis and to introduce new users to the reduction system. Additional files are provided for each grating, aaomega####.idx, which provide some grating specific modifications such as the spectral distortion model. Results from data processed using this set of .idx file will provide quality real-time first-look reductions. When performing science reductions, the experienced user can use one of the grating specific .idx files as a template and create their own private .idx files which will implement the best combination of options for their data. Note that resolution settings; wavelength range; type of target and target intensity all modify the optimal reduction strategy for AAOmega data.

The file format for the user-specified .idx file is as follows, with an example given below.

  • The comment character is '#'.

  • The default AAOmega values are loaded from the aaomega.idx file provided with 2dfdr, via the DRC_INCLUDE command.

  • Each parameter to be overridden then has a DRC_OVERRIDE_PAR command giving the parameter name and the new value.

  • The final command REDUCE is required as well. Note that some parameters have been commented out (left as the default values loaded from the aaomega.idx file) via '#' in the example below.

The format of these override commands is:

DRC_OVERRIDE_PAR STRING VALUE REDUCE

where STRING and VALUE are drawn from the options in the tables below.

# My example idx file name
# Load the basic defaults
DRC_INCLUDE aaomega.idx


# Differences from the defaults
#DRC_OVERRIDE_PAR DISTORTION 4.0E-9 REDUCE
#DRC_OVERRIDE_PAR DISTX 975 REDUCE
DRC_OVERRIDE_PAR DISTX 1010 REDUCE
DRC_OVERRIDE_PAR ARCFITORDER 4 REDUCE

One then places the .idx file either in the 2dfdr install directory (where the aaomega.idx file is located) or in the working directory, and runs the software with:

drcontrol myidxfile.idx &


Data reduction options

The various options which can be set when reducing 2dfdr data can be found on a series of tab pages in the 2dfdr main window. The options are broken up into a number of sections, with the goal of keeping related parameters in the same section.

An overview of the various parameters is given below. This page is intended as a guide, rather than a detailed manual describing each reduction step. Default values for high quality reductions of blue and red low-resolution , low intensity data are shown. Note, these are not the settings found in the .idx files shipped with AAOmega, the pre-packaged files are intended for quick-look reduction to quickly provide the user with data. The more experienced user can use modified .idx files.

Data tab

This is shown in the main 2dfdr window as seen in detail above.

The plotting windows

A user generated plot will produce a window such as that shown below (for a reduced and calibrated ARC frame). This window MUST be cleared, by pressing the 'q' key to quit the window, before new images can be plotted.

Pressing '?' brings up a list of options, shown in the second figure below.

Note that the '<' and '>' keys allow the user to step through plotting reduced spectra, once a first spectrum has been plotted by pressing 'i' over a region of the reduce image.

2dfdr plotting window

Things to note in this reduced low resolution blue arc frame are:

  1. The spectral curvature has been removed completely, arc lines are perfectly vertical, with no ragged edges (those not sitting in a line are unused fibres).
  2. The blue ccd bad columns are clearly visible (exhibiting the opposite spectral curvature since they are contours of constant x-pixel number and not constant wavelength). Check that these bad columns do not damage key spectral features, e.g. the sky lines needed for sky subtraction etc.
  3. In this reduction, there are some unused pixels to the far right of the spectrum. For this reduction, the central wavelength selected for the output data is not well matched to the observed central wavelength. and so a small amount of information has been discarded at the blue end of these spectra in this example.

2dfdr plotting help window

General tab

The General tab covers preprocessing options which are applied prior to extraction of the spectra.

General tab

 
Parameter Possible value Description .idx keyword Blue Defaults (low-res) Red Defaults
(low-res)
Fit overscan FIT
Median
All CCDs require an overscan correction. FITOVERSCAN FIT FIT
Polynomial Order of Overscan fit Integer The blue AAOmega CCD has a strong gradient in the first ~100 pixels and requires a high order fit. Full 2D bias correction may reduce the order of this fit. OVERSCANORDER 9 9
Subtract Bias Frame True
False
Bias frame subtraction can significantly improve the data quality for the blue camera. The user must ensure an adequate number of bias frames have been observed (30-50 seems to work well) and reduced in a separate directory and the DARKcombined.fits copied.. USEBIASIM True False
Subtract Dark Frame True
False
Dark frame subtraction can also significantly improve the data quality for the blue camera. The user must ensure an adequate number of dark frames have been observed (20 seems to work well) and reduced in a separate directory and the DARKcombined.fits copied. USEDARKIM True False
Divide Image by Long Slit Flat Field True
False
This option is available but still requires specific additional calibration frames (defocused flat field frames) USEFLATIM False False
Divide Spectra by Fibre Flat Field True
False
Fibre-to-fibre relative response is corrected for using a fibre flat field frame. USEFFLAT True True
Use Laplacian edge detection CR rejection NO
YES
"OBJECT ONLY"
A quality cosmic ray rejection algorithm following that of van Dokkum 2001 PASP 113 1420 LACOSMIC "OBJECT ONLY" "OBJECT ONLY"
Nsigma for Laplacian CR rejection Real The sigma clipping level for the CR rejection LACOSMICSIG 5.0 5.0
Use Running Median True False   USERUNMEDIAN False False
Use Local Average Flat True
False
See note below LAF_FLAG False False
L.A.F Smoothing Parameter Integer The smoothing length if using the local average option for the flat fielding LAF_PAR 10 N.A.
Process Overscan When Reducing BIAS Frames True
False
  BIASOVSCAN False False

Note on the local average flat field option: In order to make a fibre flat field free from the signature of the quartz-halogen lamp (complete with sharp dichroic repose features) used to make it, the following steps are taken:

  1. the fibre profiles are extracted from the flat field frame
  2. the flat field spectra are wavelength calibrated onto a common wavelength solution and normalized
  3. an average spectrum is created to represent the lamp response function
  4. the lamp response function is then projected back onto the raw pixel spectra (i.e. wavelength is removed), creating a lamp response function for each fibre
  5. the flat field spectra are divided by 2D lamp response function to remove the features of the quartz-halogen lamp

Unfortunately, for the 6700A dichroic mirror, a low order field angle dependent signature is observed in the flat field due to a light leak in the mirror. This can be successfully removed using a local average estimate for lamp response function, averaging over 10-20 adjacent fibres rather than the full CCD. While this is inherently noisier than using the full CCD, it has fewer systematics for the 6700A dichroic data in the blue arm.

Extract tab

The extraction tab deals with parameters related to the extraction of the fibre traces from the raw 2D CCD frame

Extract tab

Parameter Possible value Description .idx file string Blue Defaults (low-res) Red Defaults
(low-res)
Method TRAM
NEWTRAM
GAUSS
OPTEX
SMCOPTEX
2dfdr as a number of options for extracting the fibre profiles.
TRAM is a quick look extraction.
GAUSS performs a TRAM extraction but using Gaussian summation and pixel weighting to suppress additional read-noise
OPTEX performs an `optimal extraction'. This is required for SPIRAL data, but provides no benefit over GAUSS for MOS data at this time.
SMCOPTEX is Scott Croom's `optimal extraction'.
EXTR_OPERATION GAUSS GAUSS
Optimal extraction scattered light model POLYN
BSPLINE
USER
For the OPTEX extraction, the model fit to the background pedestal fit can be selected as a free parameter. OPTEX_SLMODEL BSPLINE BSPLINE
Optimal Extraction Number of Scattered Light Parameters 0 to 40 For the OPTEX extraction, the order of the background pedestal fit can be selected as a free parameter. OPTEXTR_NSLPARS 1 1
Rotate/Shift to Match YES
NO
FLAT ONLY
Once the tramline map has been traced, the map can be adjusted to match subtle variation in the data. Usually this is done only for the flat field. It should be used if the AAOmega slit was moved during observation, for example between taking twilight flats and science data during a night. MATCH FLAT ONLY FLAT ONLY
Distortion coefficient real For some AAOmega grating and wavelength combinations, the default camera distortion model does not perform well in matching the tram line maps for a flat field. These parameters can be adjusted to improve the fit. It is very unusual to have to adjust these values. DISTORTION 4.0E-9 4.125E-9
X centre of distortion real DISTX 975 1010
Y centre of distortion real DISTY 2048.5 2048.5
Scattered Light Subtraction None
1DFIT
2DFILT
Like all spectrographs, AAOmega suffers some level of scattered light. This can often be subtracted out of images via a low order model fit.
1DFIT - this option fits to blank spaces between fibres along a column of data and creates a low order 2D fit to the scattered light
2DFILT - this option resembles unsharp masking of the full 2D frame. It can work well for low signal data, but will be a very poor approximation for high light levels.
SCATSUB 1DFIT 1DFIT
Subtract Scattered light from Offset Sky Frames True
False
Sometimes it is not necessary to subtract the scattered light from all frames. SUBSKY TRUE TRUE

Calib tab

AAOmega data is usually calibrated using CuAr +FeAr hollow cathode arc lamps and Helium+Neon bulbs. ThAr lamps are available for higher resolutions.

However, due to the feed angle of the light from the calibration system, there can be a slight misalignment (at the 10th of a pixel level) between the calibration arc and the science spectra. This can result in poor sky subtraction (p-Cygni like residual sky line profiles). Hence, for observing modes where OH night sky lines are visible in the data, secondary calibration can be performed fitting to these lines.

Calibrate tab

Parameter Possible value Description .idx file string Blue Defaults (low-res) Red Defaults
(low-res)
Polynomial order for arc fit Integer The order of the wavelength fit depends on the grating choice, wavelength range and available arc lines to fit to. ARCFITORDER 4 4
Use whale shark matching in arc fit TRUE
FALSE
Use a 2D fitting algorithm on arc data rather than 1D USE_WSM_ALGOL FALSE FALSE
Wavelength calibrate from sky lines TRUE
FALSE
Due to the way the arc light illuminates AAOmega, a small (<0.1pixel) shift is introduced in the wavelength calibration depending on the field plate position of fibres. This can degrade the effectiveness of the sky subtraction. A correction to place all spectra on a common wavelength solution can therefore be calculated from the OH night sky lines, but only when there are lines visible on the CCD SKYSCRUNCH TRUE TRUE
Polynomial order of sky fitting Integer In the blue, usual only the 5577A O2 line is available to fit to, hence a 1 order fit. In the red, many more lines are usually available. SKYFITORDER 1 4
Calibrate Flux TRUE
FALSE
Not yet implemented CALIBFLUX FALSE FALSE
Flux calibration table String Not available CALIB_TABFILE N.A. N.A.

Sky tab

The Sky tab covers options associated with the sky subtraction methods available.
For sky subtraction, a high signal-to-noise sky spectrum is created by combining the dedicated sky fibres allocated to blank sky positions in each configuration. A number of options govern how the sky spectrum is created and subtracted from the data.

Sky tab

Parameter Possible value Description .idx file string Blue Defaults (low-res) Red Defaults
(low-res)
Throughput Calibrate TRUE
FALSE
The amount of sky to subtract from a given science fibre must usually be determined from the data itself. For data with very little sky (e.g. a short standard star exposures) one may wish to dispense with this correction and skip sky subtraction is it will have little effect on the final data set. THRUPUT TRUE TRUE
Throughput Calibration Method OFFSKY
SKYLINE(KGB)
SKYFLUX(COR)
SKYFLUX(MED)
The normalization correction is calculated either from an offset-sky/Twilight-flat (usually at high resolution in the blue) or from the night sky emission lines which one is trying to remove. The different SKY methods are detailed in the 2dfdr manual, and outlined below*. TPMETH SKYFLUX(COR) SKYFLUX(MED)
Subtract Sky TRUE
FALSE
Should the system attempt sky subtraction. For some data sets this is not necessary or even possible. For Nod-and-Shuffle data, for example, this step would be redundant. SKYSUB TRUE TRUE
Sky Fibre Combination Operation MEDIAN
MEAN
with 20-30 sky spectra to combine to make an average sky, one could either perform a clipped MEAN, or use a simple MEDIAN. We find the median is usually more robust in practice, with little real loss in signal-to-noise SKYCOMBINE MEDIAN MEDIAN
Use iterative sky subtraction TRUE
FALSE
Shift/scale/smooth sky spectrum to minimise sky residuals ITERSKY FALSE TRUE
Correct for telluric absorption TRUE
FALSE
Correct for telluric absorption, by dividing data by a mean object spectrum in the regions containing atmospheric absorption bands TELCOR FALSE TRUE
Correct Min S/N for telluric absorption TRUE
FALSE
Only spectra with S/N greater than this are included in the mean spectrum for telluric correction TELSN 3.0 3.0
Use PCA after Normal Sky Subtract TRUE
FALSE
Undertake Principle Component Analysis of spectrum components after sky subtraction to remove sky residuals (not recommended for stars or fieldsof targets very close in velocity). PCASKY TRUE TRUE
Number of PCA eigenspectra 0-30 Number of eigenspectra used in PCA analysis PCANUM 10 10
Min wavelength for PCA sky subtraction Real Minimum wavelength of data range over which PCA sky subtraction is applied PCALMIN 5550.0 5700.0
Max wavelength for PCA sky subtraction Real Maximum wavelength of data range over which PCA sky subtraction is applied PCALMAX 5600.0 9000.0

Notes: * SKY methods. The three skyline throughput calibration methods are variations on a theme of the same basic principle. In order to determine how much sky to subtract off from each fibres, the system calibrates the relative sky flux by looking at the flux in night sky emission lines (usually OH, but also some O2, Na and other species), unless the OFFSKY method has been selected, in which case a twilight-flat or Offset-Sky frame must be provided. In all cases, the SKYLINE/FLUX calculates an multiplicative scaling for the sky based on the integrated flux in the sky lines. SKYFLUX(COR) performs a more robust fit to the core of the skyline profiles, which is more robust against PSF variations across the field. SKYFLUX(MED) performs the same task as SKYFLUX(COR) but for red data, which typically has more strong unblended lines to measure than the blue; a median scaling is derived, which is more robust against CCD defects and cosmic rays. SKYLINE(KGB) is an extended implementation (designed and implemented by Karl Glazebrook) which attempts to track PSF variations across the CCD due to the camera optics, minimizing the sky residual globally. This method worked well for 2dF data, but is not recommended for the current AAOmega implementation.

Combine tab

Data from multiple observations of the same fields, and also data from multiple observations of separate fibre configurations (usually with some overlap in the targets, which is being performed to increase a sub-set of exposures times) is routinely performed by 2dfdr.

Typically the data from each camera (red and blue) is combined separately before the spectra are spliced into one continuous spectrum.
Combining of reduced files occurs in 'Auto Reduction' mode when all local object frames have been processed. It can also be done manually using the Combine Reduced Runs item in the 2dfdr Commands menu.

The 2dfdr combine algorithm combines data based on either object name or object location. That is, fibres having the same name (or location) are added and normalised to produce the output. This is to include all objects, whether they are contained within every frame or only a sub-set of the frames. The combine has the following features:

  • Multiple configurations of the same field can be combined together when objects are in common. Note that this can result in more spectra than the instrument can produce in one exposure.

  • Only fibre types 'S' (sky) and 'P' (program) fibres are combined. This includes cases in which a fibre has been disabled part way through a field observation, so only good data is combined. Other fibres such as unused and parked fibres have all values set to zero.

  • The first spectra will be all those from the first frame in the combine including unused/parked and sky fibres. Any additional spectra will be only sky and program spectra from objects in subsequent frames and not present in the first frame. if the data combined are all from the same configuration there will be no difference in the fibre count.

  • All the fibre table extension information is properly propoagated. Additional fibres are numbered beginning from the last fibre of the first frame. So for AAOmega, the first 400 fibres will be from the first frame, and fibre 401 and beyond will be additional fibres from subsequent frames (if any).

  • Variances are handled properly.

NOTE: Currently the combined file exposure time is NOT set properly. Exposure time is given in only one place for a file, the value of the .fits header keyword 'EXPOSED'. This exposure time applies to all fibres within the file. When fibres are combined, this keyword is copied from the first frame - no attempt is made at setting it properly.

Combine tab

Parameter Possible value Description .idx file string Blue Defaults (low-res) Red Defaults
(low-res)
Combine Reduced Data TRUE
FALSE
This option will automatically combine the reduced spectra at the end of a automatic reduction run (that has been started using the START button). A frame named dateccd_combined.fits will be generated in the working directory. AUTO_COMB TRUE TRUE
Adjust Continuum Levels TRUE
FALSE
This parameter performs a cosmetic adjustment of the spectra, allowing for additive offsets. * COMB_ADJUST TRUE TRUE
Flux Weight NONE
FRAMES
OBJECT
When combining data, a flux weight may be applied to each spectrum. One weight can be applied for the whole frames (best when individual fibres have low signal) or a weight may be determined on an object-by-object basis (best when spectra have high signal to noise). FLUXWT FRAMES FRAMES
Rejection Threshold 1-100 A spurion* sigma-clipping rejection is applied, with the clipping threshold set to this value. CSIGREJ 5.0 5.0
Systematic Uncertainty 1-100 An offset factor allows for systematic uncertainties in the data during clipping. CSYST 0.1 0.1
Smoothing Scale 1-100 To determine weights between spectra, the spectra are smoothed to remove local defects. CSMOOTH 101 101
Combine A/B spectra TRUE
FALSE
For Cross-beam-switched data, with two fibres allocated to a target for observations in the A and B positions, this option will invert the B spectrum and add it to the A spectrum, then set the B spectrum position to zero in the output image. COMBAB TRUE TRUE
Combine Matching Name TRUE
FALSE
If TRUE combine matching names, if FALSE match positions COMBNAME TRUE TRUE
Arm re-scrunched in splicing RED
BLUE
In order to join the red and blue spectra, data must be placed on a common wavelength scale. It is usual to rebin the red data, to preserve the higher resolution in the blue. SCRUNCHARM RED RED
Coadd spliced spectra in overlap region TRUE
FALSE
Combine the spectral pixels in the overlap region. SPLICECOADD TRUE TRUE
Splicing mid-point Angstroms A mid-point about which to join the spectra. SPLICEMIDPOINT 5700.0 5700.0
Include RWSS in extension TRUE
FALSE
Include an extension containing spectra reduced without sky subtraction so that these can be used to calculate the weights when splicing INC_RWSS TRUE TRUE

Notes:
* The "Adjust Continuum Levels" option was first introduced for the original 2dF galaxy redshift survey data. This cosmetic adjustment was found to improve treatment of scattered light in the original 2dF spectrographs, and provided cosmetic improvements for spectra which enhanced the redshift success rate. It should be used with caution when spectrophotometric integrity is of paramount importance.
* The spurions is the quantum particle of erroneous information. Most cosmic rays are caused by spurions.

Plots tab

Plots tab

Parameter Possible value Description .idx file string Blue Defaults (low-res) Red Defaults
(low-res)
Scaling TRUE
FALSE
Auto scale figures at the 95% of pixels level, as opossed to min/max. AUTO True True
Plot Type GREY
COLOUR
CONTOUR
CVMAG
A number of different plot styles are supported. PLOTTYPE Grey Grey
Pixels per bin   Rebin the spectrum before plotting. NBIN 1 1
Remove Sky Residual TRUE
FALSE
  FIXSKY FALSE FALSE
Plot Tram Map YES
NO
FLAT ONLY
The user will sometimes want to check that the tramline tracing for the flat field has worked correctly. Usually one does this only for the flat field, and usual only for a few test data sets from any given observing run. PLTMAP FLAT ONLY FLAT ONLY
Plot Combined Sky TRUE
FALSE
Diagnostic plot. SKYPLOT FALSE FALSE
Plot Telluric Correction TRUE
FALSE
Diagnostic plot. TELPLOT FALSE FALSE
Plot Throughput map TRUE
FALSE
Diagnostic plot. THPLOT FALSE FALSE
Plot averaged flat field TRUE
FALSE
Diagnostic plot. AFFPLOT FALSE FALSE
Plot overscan bias level YES
NO
FLAT ONLY
Enable plotting of the overscan bias level BIASPLOT NO NO
Plot background scattered light TRUE
FALSE
Diagnostic plot. BGDPLOT FALSE FALSE
Plot optimal extraction fits TRUE
FALSE
Diagnostic plot. OPTPLOT FALSE FALSE
Plot PCA sky subtraction diagnostics TRUE
FALSE
Diagnostic plot. PCAPLOT FALSE FALSE

RnD tab

This tab collects options currently under research and development and are generally not required by users and so are not described in detail here.

2dfdr RnD tab