Reducing data with 2dfdr-v4

The version 4 release of 2dfdr represents a major modification to the 2dfdr software system. The goal of the modification is sustainability of the software, much effort having been put into removing dependencies on the STARLINK software suite. Most of the basic underlying code and data reduction options remain the same as in the previous release.

There has been significant modification to the .idx file format, used to allow the user to preset reduction option flags. This overhaul is intended to make it easier for users to access the full power of the 2dfdr package without a detailed knowledge of the underlying workings of the software.

This page is intended as a guide to the changes to the 2dfdr GUI and to the new .idx file format. It is not intended as a replacement for the 2dfdr user manual.

Download, and installation+setup instruction are outlined in the new user cookbook found at:

http://www2.aao.gov.au/twiki/bin/view/Main/CookBook2dfdr

The output file data format, which remains largely unchanged from previous versions, is discussed in the Guide to the AAOmega File Format

MacOS release

With 2dfdrV4 there is also, for the first time, a 2dfdr release for the Mac. The operation of 2dfdr should be identical to that on the Linux platform. The software has been tested on archival AAOmega data, but users should report residual issues with this new platform to the instrument scientist (Sarah Brough sb@aao.gov.au).

Changes to the PGPLOT routines

The older starlink implementation of the PGPLOT libraries and Tcl/Tk button widgets have been replaced with a new implementation. The changes to the plotting window are discussed below. The most significant modification most users will encounter is that each time an image is displayed on the screen, the user will need to quit this window, by pressing the 'q' key before a new image can be displayed.

Information on improving failed wavelength solutions

From time to time the 2dfdr software may fail to provide the correct wavelength solution. Information on correcting this can be found at: http://www.aao.gov.au/AAO/2df/aaomega/aaomega_wavelength.html

New .idx file format

2dfdr is provided with a number of .idx files. These set predefined values for many of the reduction operations than can be activated. The basic .idx file, aaomega.idx, is designed to give quick reduction for real-time analysis and to introduce new users to the reduction system. Additional files are provided for each grating, aaomega####.idx, which provide some grating specific modifications such as the spectral distortion model. Results from data processed using this set of .idx file will not be optimal, but should provide quality real-time first-look reductions. When performing science reductions, the experienced user should therefore use one of the grating specific .idx files as a template and create their own private .idx files which will implement the best combination of options for the data at hand. Note that resolution settings; wavelength range; type of target and target intensity all modify the optimal reduction strategy for AAOmega data, with sky subtraction methods usually being the most common modification required.

The file format for the user-specified .idx file is as follows, with example is given below.

The comment character is '#'.
The default AAOmega values are loaded from the aaomega.idx file provided with 2dfdr, via the DRC_INCLUDE command.
Each parameter to be overridden then has a DRC_OVERRIDE_PAR command giving the parameter name and the new value.
The final command REDUCE is required as well. Note that some parameters have been commented out (left as the default values loaded from the aaomega.idx file) via '#' in the example below.

The format of these override commands is:

DRC_OVERRIDE_PAR STRING VALUE REDUCE

where STRING and VALUE are drawn from the options in the tables below.

# My example idx file name

# Load the basic defaults

DRC_INCLUDE aaomega.idx

# Differences from the defaults
#DRC_OVERRIDE_PAR DISTORTION 4.0E-9 REDUCE
#DRC_OVERRIDE_PAR DISTX 975 REDUCE
DRC_OVERRIDE_PAR DISTX 1010 REDUCE
DRC_OVERRIDE_PAR ARCFITORDER 4 REDUCE

One then places the .idx file either in the 2dfdr install directory (where the aaomega.idx file is located) or in the working directory, and runs the software with the usual call to

drcontrol myidxfile.idx &

Data reduction options

Data reduction options tabs

The various options which can be set when reducing 2dfdr data can be found on a series of tab pages in the 2dfdr main window. The options are broken up into a number of sections, with the goal of keeping related parameters in the same section.

Data
The plotting window
General
Extract
Calib
Sky
Combine
Plots

An overview of the various parameters is given blow. This page is intended as a guide, rather than a detailed manual describing each reduction step. Default values for high quality reductions of blue and red low-res , low intensity data are shown. Note, these are not the settings found in the .idx files shipped with AAOmega, the pre-packaged files are intended for quick-look reduction to quickly provide the novice user with example data. The more experienced user should use modified .idx file for publication quality spectra. A selection of examples will be made available.

Data tab

The 2dfdr GUI has changed little for its v4 incarnation.

The basic file organization dialog has not changed in the new release with the exception that the multiple DREXEC task tabs have been removed. This removal is part of work to make 2dfdr properly multi-threaded in the future.

The plotting windows

The format of the plotting windows has changed with the removal of the STARLINK PGPLOT implementation. A user generated plot will produce a window such as that shown below (for a reduced and calibrated ARC frame). This window MUST be cleared, by pressing the 'q' key to quit the window, before new images can be plotted.

Pressing '?' brings up a list of options, shown in the second figure below.

Note that the '<' and '>' keys now allow the user to step through plotting reduced spectra, once a first spectrum has been plotted by pressing 'x' over a region of the reduce image.

Things to note in this reduced low resolution blue arc frame are:

The spectral curvature has been removed completely, arc lines are perfectly vertical, with no ragged edges.
The blue ccd bad columns are clearly visible (exhibiting the opposite spectral curvature since the are contours of constant x-pixel number and not constant wavelength). Check that these bad columns do not damage key spectral features, e.g. the sky lines needed for sky subtraction etc.
In this reduction, there are some unused pixels to the far right of the spectrum. For this reduction, the central wavelength selected for the output data is not well matched to the observed central wavelength. and so a small amount of information has been discarded at the blue end of these spectra in this example.

General tab

The General tab covers preprocessing options which are applied prior to extraction of the spectra.

Parameter	Possible value	Description	.idx keyword	Blue Defaults (low-res)	Red Defaults (low-res)
Fit overscan	FIT Median	All CCDs require an overscan correction.	FITOVERSCAN	FIT	FIT
Order of overscan fit	Integer	The blue AAOmega CCD has a strong gradient in the first ~100 pixels and requires a high order fit. Full 2D bias correction may reduce the order of this fit.	OVERSCANORDER	9	9
Subtract Bias Frame	True False	Bias frame subtraction can significantly improve the data quality for the blue camera. The user must ensure an adequate number of bias frames have been observed (30-50 seems to work well).	USEBIASIM	True	False
Divide Image by Long Slit Flat Field	True False	This option is available but still requires specific additional calibration frames (defocused flat field frames)	USEFLATIM	False	False
Divide Spectra by Fibre Flat Field	True False	Fibre-to-fibre relative response is corrected for using a fibre flat field frame.	USEFFLAT	True	True
Use Laplacian edge detection CR rejection	NO YES "OBJECT ONLY"	A quality cosmic ray rejection algorithm following that of van Dokkum 2001 PASP 113 1420	LACOSMIC	"OBJECT ONLY"	"OBJECT ONLY"
Nsigma for LAplacian CR rejection	Real	The sigma clipping level for the CR rejection	LACOSMICSIG	5.0	5.0
Plot overscan, Extraction and Background	On Off	Enable plotting of some diagnostic information	PLTFITS	Off	Off
Use local average flat	True False	See note below	LAF_FLAG	False	False
L.A.F smoothing	Integer	The smoothing length if using the local average option for the flat fielding	LAF_PAR	10	N.A.

Note on the local average flat field option: In order to make a fibre flat field free from the signature of the quartz-halogen lamp (complete with sharp dichroic repose features) used to make it, the following steps are taken:

the fibre profiles are extracted from the flat field frame
the flat field spectra are wavelength calibrated onto a common wavelength solution and normalized
an average spectrum is created to represent the lamp response function
the lamp response function is then projected back onto the raw pixel spectra (i.e. wavelength is removed), creating a lamp response function for each fibre
the flat field spectra are divided by 2D lamp response function to remove the features of the quartz-halogen lamp

Unfortunately, for the 6700A dichoric mirror, a low order field angle dependent signature is observed in the flat field due to a light leak in the mirror. This can be successfully removed using a local average estimate for lamp response function, averaging over 10-20 adjacent fibres rather than the full CCD. While this is inherently noisier than using the full CCD, it has fewer systematics for the 6700A dichroic data in the blue arm.

Extract tab

The extraction tab deals with parameters related to the extraction of the fibre traces from the raw 2D CCD frame

Parameter	Possible value	Description	.idx file string	Blue Defaults (low-res)	Red Defaults (low-res)
Method	TRAM GAUSS FIT OPTEX	2dfdr as a number of options for extracting the fibre profiles. TRAM is a quick look extraction. GAUSS performs a TRAM extraction but using Gaussian summation and pixel weighting to suppress additional read-noise FIT is not yet reliably implemented for AAOmega. OPTEX performs an `optimal extraction'. This is required for SPIRAL data, but provides no benefit over GAUSS for MOS data at this time.	EXTR_OPERATION	GAUSS	GAUSS
NSLPARS	0 to 3	For the OPTEX extraction, the order of the background pedestal fit can be selected as a free parameter.	OPTEXTR_NSLPARS	1	1
Plot Tram Map	YES NO FLAT ONLY	The user will some times want to check that the tramline tracing for the flat field has worked correctly. Usually one does this only for the flat field, and usual only for a few test data sets from any given observing run.	PLTMAP	FLAT ONLY	FLAT ONLY
Rotate/Shift to Match	YES NO FLAT ONLY	Once the tramline map has been has been traced, the map can be adjusted to match subtle variation in the data. Usual this is done only for the flat field. It should be used if the AAOmega slit was moved during observation, for example between taking twilight flats and science data during a night.	MATCH	FLAT ONLY	FLAT ONLY
Dist. coeff.	real	For some AAOmega grating and wavelength combinations, the default camera distortion model does not perform well in matching the tram line maps for a flat field. These parameters can be adjusted to improve the fit. It is very unusual to have to adjust these values.	DISTORTION	4.0E-9	4.125E-9
X center of dist.	real		DISTX	975	1010
Y center of dist.	real		DISTY	2048.5	2048.5
Scattered Light	None 1DFIT 2DFILT	Like all specctrographs, AAOmega suffers some level of scattered light. This can often be subtracted out of images via a low order model fit. 1DFIT - this option fits to blank spaces between fibres along a cloumn of data and creates a low order 2D fit to the scattered light 2DFILT - this option resembles unsharp masking of the full 2D frame. It can work well for low signal data, but will be a very poor approximation for high light levels.	SCATSUB	1DFIT	1DFIT
Subtract Scattered light from Offset Sky Frames	True False	Some times it is not neccesary to subtract the scattered light from all frames.	SUBSKY	TRUE	TRUE

Calib tab

AAOmega data is usually calibrated using CuAr +FeAr hollow cathode arc lamps and Helium+Neon bulbs. ThAr lamps are available for higher resolutions.

However, due to the feed angle of the light from the calibration system, there can be a slight misalignment (at the 10th of a pixel level) between the calibration arc and the science spectra. This can result in poor sky subtraction (p-Cygni like residual sky line profiles). Hence, for observing modes where OH night sky lines are visible in the data, and secondary calibration can be performed, fitting to these lines.

Parameter	Possible value	Description	.idx file string	Blue Defaults (low-res)	Red Defaults (low-res)
Polynomial order for arc fit	Integer	The order of the wavelength fit depends on the grating choice, wavelength range and available arc lines to fit to.	ARCFITORDER	4	4
Wavelength calibrate from sky lines	TRUE FALSE	Due to the way the arc light illuminates AAOmega, a small (<0.1pixel) shift is introduced in the wavelength calibration depending on the field plate position of fibres. This can degrade the effectiveness of the sky subtraction. A correction to place all spectra on a common wavelength solution can therefore be calculated from the OH night sky lines, but only when there are line visible on the CCD	SKYSCRUNCH	TRUE	TRUE
Polynomial order for sky fitting	Integer	In the blue, usual only the 5577A O2 line is available to fit to, hence a 1 order fit. In the red, many more lines are usual available.	SKYFITORDER	1	4
Flux calibrate	TRUE FALSE	Not yet implemented	CALIBFLUX	FALSE	FALSE
Flux calibration table	String	Not available	CALIB_TABFILE	N.A.	N.A.

Sky tab

The Sky tab covers options associated with the sky subtraction methods available.

For sky subtraction, an high signal-to-noise sky spectrum is created by combining the dedicated sky fibres allocated to blank sky positions in each configurations. A number of options govern how the sky spectrum is created and subtracted from the data.

Parameter	Possible value	Description	.idx file string	Blue Defaults (low-res)	Red Defaults (low-res)
Throughput Calibrate	TRUE FALSE	The amount of sky to subtract from a given science fibre must usually be determined from the data it's self. For data with very little sky (e.g. a short standard star exposures) one may wish to dispense with this correction on skip sky subtraction is it will have little effect on the final data set.	THRUPUT	TRUE	TRUE
Subract Sky	TRUE FALSE	Should the system attempt sky subtraction. For some data sets this is not necessary or even possible. For Nod-and-Shuffle data, for example, this step would be redundant.	SKYSUB	TRUE	TRUE
Plot Combined Sky	TRUE FALSE	Diagnostic plot.	SKYPLOT	FALSE	FALSE
Plot Throughput map	TRUE FALSE	Diagnostic plot.	THPLOT	FALSE	FALSE
Throughput Calibration Method	OFFSKY SKYLINE SKYLINE(KGB) SKYLINE(COR) SKYLINE(MED)	The normalization correction is calculated either from an offset-sky/Twilight-flat (usually at high resolution in the blue) or from the night sky emission lines which one is trying to remove. The different SKYLINE methods are detailed in the 2df dr manual, and outlined below*.	TPMETH	SKYLINE(COR)	SKYLINE(MED)
Sky Fibre Combination Operation	MEDIAN MEAN	with 20-30 sky spectra to combine to make an average sky, one could either perform a clipped MEAN, or use a simple MEDIAN. We find the median is usually more robust in practice, with little real lose in signal-to-noise	SKYCOMBINE	MEDIAN	MEDIAN
Use optimal sky subtraction	TRUE FALSE	The major error in the sky subtraction is small variation in the PSF shape between fibres. This option performs a rather slow and CPU intensive iterative minimization in order to provide a reduced residual in the sky subtraction. The routine iterates over small (sub pixel) shifts in the sky spectrum, variations in the line width and adjustments to the throughput calibration.	OPTSKY	FALSE	TRUE

Notes: * SKYLINE methods. The four skyline throughput calibration methods are variations on a theme of the same basic principle. In order to determine how much sky to subtract off from each fibres, the system calibrates the relative sky flux by looking at the flux in night sky emission lines (usually OH, but also some O2, Na and other species), unless the OFFSKY method has been selected, in which case a twilight-flat or Offset-Sky frame must be provided. In all case, the SKYLINE calculates an multiplicative scaling for the sky based on the integrated flux in the sky lines. The initial implementation, SKYLINE, was the first 2dfdr version to use this method, but is often not very robust. SKYLINE(COR) performs a more robust fit to the core of the skyline profiles, which is more robust against PSF variations across the field. SKYLINE(MED) performs the same task as SKYLINE(COR), but for red data, which typically has more strong unblended lines to measures than the blue, a median scaling is derived, which is more robust against CCD defects and cosmic rays. SKYLINE(KGB) is a extended implementation (designed and implemented by Karl Glazebrook) which attempts to track PSF variations across the CCD due to the camera optics, minimizing the sky residual globally. This method worked well for 2dF data, but does not usual provide significant gains for the current AAOmega implementation.

Combine tab

Data from multiple observations of the same fields, and also data from multiple observations of separate fibre configurations (usually with some overlap in the targets, which is being performed to increase a sub-set of exposures times) is routinely performed by 2dfdr.

Typically the data from each camera (red and blue) is combined separately before the spectra are spliced into one continuous spectrum.

Parameter	Possible value	Description	.idx file string	Blue Defaults (low-res)	Red Defaults (low-res)
Combine Reduced Data	TRUE FALSE	This options will automatically combine the reduced spectra at the end of a automatic reduction run (that has been started using the START button). A frame named combined_frames.fits will be generated in the working directory.	AUTO_COMB	TRUE	TRUE
Adjust Continuum Levels	TRUE FALSE	This parameter performs a cosmetic adjustment of the spectra, allowing for additive offsets. *	COMB_ADJUST	TRUE	TRUE
Flux Weight	NONE FRAMES OBJECT	When combining data, a flux weight may be applied to each spectrum. One weight can be applied for the whole frames (best when individual fibres have low signal) or a weight may be determined on an object-by-object basis (best when spectra have high signal to noise).	FLUXWT	FRAMES	FRAMES
Rejection Threshold	1-100	A spurion* sigma-clipping rejection is applied, with the clipping threshold set to this value.	CSIGREJ	5.0	5.0
Systematic Uncertainty	1-100	An offset factor allows for systematic uncertainties in the data during clipping.	CSYST	0.1	0.1
Smoothing Scale	1-100	To determine weightings between spectra, the spectra are smoothed to remove local defects.	CSMOOTH	101	101
Combine A/B spectra	TRUE FALSE	For Cross-beam-switched data, with two fibres allocated to a target for observations in the A and B positions, this option will invert the B spectrum and add it to the A spectrum, then set the B spectrum position to zero in the output image.	COMBAB	TRUE	TRUE
Arm re-scrunched in splicing	RED BLUE	In order to join the red and blue spectra, data must be places on a common wavelength scale. It is usual to rebin the red data, to preserve the higher resolution in the blue.	SCRUNCHARM	RED	RED
Coadd spliced spectra in overlap region	TRUE FALSE	Combine the spectral pixels in the overlap region.	SPLICECOADD	TRUE	TRUE
Splicing mid-point	Angstroms	A mid point about which to join the spectra.	SPLICEMIDPOINT	5700.0	5700.0

Notes:
* The "Adjust Continuum Levels" option was first introduced for the original 2dF galaxy redshift survey data. This cosmetic adjustment was found to improve treatment of scattered light in the original 2dF spectrographs, and provided cosmetic improvements for spectra which enhanced the redshift success rate. It should be used with caution when spectrophotometric integrity is of paramount importance.
* The spurions is the quantum particle of erroneous information. Most cosmic rays are caused by spurions.

Plots tab

Few people use the 2dfdr Postscript plot generator, and so these plots are simplistic but can be useful.

Parameter	Possible value	Description	.idx file string	Blue Defaults (low-res)	Red Defaults (low-res)
Scaling	TRUE FALSE	Auto scale figures at the 95% of pixels level, as opossed to min/max.	AUTO	True	True
Plot Type	GREY COLOUR CONTOUR CVMAG	A number of different plot styles are supported. This feature has received little interest over the years and so has not been maintained at an optimal level.	PLOTTYPE	Grey	Grey
Pixels per bin		Rebin th spectrum before plotting.	NBIN	1	1
Remove sky residual	TRUE FALSE	Suppress plotting of sky regions.	FIXSKY	False	False

Sarah Brough sb@aao.gov.au