[Skip to Content]

AAO

General Links
Professional Links
Australian Human Resources Institute Diversity Awards 2013 Finalist

2dfdr File Format

The 2dfdr data reduction package uses FITS format for input, output and internal manipulation of files. FITS (Flexible Image Transport System) is the nearly universally accepted file format for astronomical data endorsed by the IAU. An overview of FITS with links to reference documents is available at the NASA FITS webpages.

When analysing data from an observing run, one needs to map the combined spectra returned from the reduction task back to individual objects from the input catalogue. All of the relevant information is contained within the combined output file(s).

This page documents the format of FITS files used and written by the 2dfdr data reduction software. First the various file types are explained, and then the internal file extensions are discussed.

Contents:


Summary of the 2dfdr Output File Format


The table below gives a summary of the 2dfdr output file content, for either the individual frame *red.fits files or a _combined.fits file. The file is a standard Multi-Extension FITS file (FITS MEF).

IRAF Format Contents
Primary image .fits[0] The primary extension in the .fits file is a WxN image containing the number of pixels in each spectrum and N is the number of spectra represented. This is 400 for AAOmega data (392 science fibres and 8 guide fibres). Unused science fibres and Sky spectra, are included in the output file along with the guide fibre spectra, even though the spectra contain no information. In the case where multiple sets of AAOmega datasets, which contained a subset of common objects, have been combined, the format is a little more complex.
First Extension .fits[1] The variance extension is also a WxN array identical in size to the primary extension. Each member contains the variance for the corresponding element in the primary extension.
Second Extension .fits[2] FITS binary table, with N rows, one for each fibre. Each Row contains information for each object such as RA, Dec, 2dF Pivot number and more
Other Extensions ... The files contain several other extensions which are generally only used when deeper analysis of the data is required. They are not necessarily in order and are accessed by name.
For example: Sky spectrum
Stored as a FITS binary table, with W rows.
Each row contains the one element of the 'typical' sky spectrum used in the data reduction ('typical' because for a combined frame it is not obvious how the final sky spectrum for each fibre should be represented here).
Note: The variance information is correctly propagated, the sky spectrum is not presented here for this purpose.

File Types

Introduction

The 2dfdr software creates, reads and writes several file types. The possible types are listed here.

A file type can be distinguished by its filename and the type suffix. An example is the AAOmega file 31jan10083red.fits. The red suffix indicates it is a reduced file. Other suffixes are im for image files, ex for extracted files and tlm for tramline map files. Files without such a suffix are normally raw files.

Filenames have four digits immediately preceding the type suffix. These four digits are the exposure sequence number, and usually begin with 0001 each evening.

The digit preceding the sequence number in the example indicates which the AAOmega spectrograph was used, 1 for blue and 2 for red.

raw

These files are produced by the instrument. They are always read-only to 2dfdr, i.e. they are never modified by the reduction software.

im(age)

This is the raw file that has

Image files have names that are formed by suffixing im to the sequence number of the corresponding raw file. This gives names like 31jan10083im.fits.

ex(tracted)

An ex(tracted) file has had intensity information extracted from the image file. This is done for each fibre used in the exposure. The spectrum of each fibre is given producing a Wavelength by fibre Number (WxN) array.

Extracted files have names that are formed by suffixing ex to the sequence number of the corresponding raw file. This gives names like 31jan10083ex.fits.

red(uced)

This is the final reduced file. It contains a Wavelength by fibre Number (WxN) array. It is produced by applying observation type (fibre flat, arc, science, etc.) specific algorithms to the ex(tracted) file.

Reduced files have names that are formed by suffixing red to the sequence number of the corresponding raw file. This gives names like 31jan10083red.fits.

tlm

This is the tramline map file. It is normally produced in the same step as the ex(tracted) file. It provides the centre fibre positions on the image in pixel units in its PRIMARY array. It may contain a SIGMAPRF HDU array of the same size containing the sigma of the Gaussian profile.

Tramline map files have names that are formed by suffixing "tlm" to the sequence number of the corresponding raw file. This gives names like 31jan10083tlm.fits.

combined reduced

This is the result of combining two or more red(uced) files.

2dfdr uses the name combined_frames.fits.

spliced reduced

This is the result of splicing a pair of AAOmega red and blue reduced files.

2dfdr uses the name spliced.fits.

combined BIAS

Sometimes called the "master" bias file, this is the result of combining two or more reduced bias files.

2dfdr uses the name BIAScombined.fits.

combined LFLAT

This is the result of combining two or more reduced long-slit flat files.

2dfdr uses the name LFLATcombined.fits.

diagnostic

2dfdr produces various files used to diagnose particular operations on the data.

File Parts

Introduction

Each FITS format file contains one or more Header Data Units (HDU). HDUs outside the primary HDU are known as "extensions". There are three types of HDU: image, ASCII table or binary table. 2dfdr uses only image HDUs with a binary table HDU used to hold the fibre information.

The FITS standard requires each HDU to have a header (keyword) section and data section. Each header section contains any number of keyword name, value and comment 80-character records. The purpose of the keyword section is to provide information about the connected HDU data section, and the FITS format specifies a minimum set of keywords that each HDU must have (SIMPLE, BITPIX, NAXIS, NAXISn where n is 1 to array rank, and END).

For 2dfdr purposes keywords in the primary IMAGE HDU provide a description of the observation. All other HDUs (outside the primary) in a 2dfdr file contain only those keywords mandated in the FITS standard.

Exception to all info in primary header rule: fibre table.* <<-- FILL IN HERE!

2dfdr output files have a history section (within the primary header) which describes the processing steps used in producing the file. This is contained in FITS keywords with the name "HISTORY".

What follows are descriptions of the individual HDUs used in 2dfdr output files. Whether a particular HDU appears in a 2dfdr output file mostly depends on the file type. Each section below has a Where Used: attempting to document the correspondence.

All 2dfdr output files have at least the first three HDUs (marked 0, 1 and 2 below). The IMAGE, VARIANCE and FIBRE TABLE also ALWAYS appear in this order.

[0] Primary

A 2-dimension image array holding the raw CCD image (raw files), processed CCD image (im files), or spectral data (ex and red files). Images are the same size as the CCD. Raw images are slightly larger as they contain the overscan bias region. Spectral data is dimensioned wavelength (number of pixels in spectral direction) by the number of fibres. The data type is 16-bit integer for raw files produced by the instruments, and 32-bit IEEE floating point for all data files produced by 2dfdr.

Where Used: Every file has this HDU. It is written and read by 2dfdr.

[1] VARIANCE

A 2-dimension image array holding the expected image data variance. The array is identical in size to the primary array, and each member contains the variance for the corresponding primary array member. The data type is always 32-bit IEEE floating point.

Values are initially derived from the image data along with values determined from photon statistics, and the detector read noise and gain. The variance is then propagated and adjusted through subsequent steps of the processing.

Where Used: All 2dfdr output files have this HDU. Raw files do not have this HDU. It is written and read by 2dfdr.

[2] FIBRE TABLE

A 2-dimension binary table containing a row for each fibre. Each row describes how the corresponding fibre was used in the observation. The table columns are character data (e.g. the name of the astronomical object the fibre was observing), integer data (e.g. the fibre number) and floating point data (e.g. the object right ascension and declination). Descriptions for all columns are below.

The table appears in raw instrument-produced files where it is created during the observation when it is filled with information output by the "configure" program. The table is then copied from the raw file to all 2dfdr output files.

There are two types of fibre tables that are identified by their names, either "FIBRES" or "FIBRES_IFU". FIBRE_IFU tables are produced only by the AAOmega IFU instrument and they have slightly different columns to other instruments.

Where Used: Every file has this HDU. It is READ ONLY to 2dfdr with one exception. The exception can occur when reduced files using different configurations are combined. In this case rows are added to the fibre table.

Fibre Table Columns


The fibre binary table lists, for each fibre, the columns listed in the table below:
Column Name Type Description
1 NAME String Object name from the configure .fld file
2 RA Real Right Ascension from the configure .fld file
3 DEC Real Declination from the configure .fld file
4 X Integer 2dF field plate X co-ordinate (in microns)
5 Y Integer 2dF field plate Y co-ordinate (in microns)
6 XERR Integer Reported error in X in final fibre placement
7 YERR Integer Reported error in Y in final fibre placement
8 THETA Real Angle of fibre on field plate
9 TYPE Character Fibre type: F-guide, N-broken, dead or no fibre, P-program (science), S - Sky, U-unallocated or unused
10 PIVOT Integer 2dF fibre pivot number
11 MAGNITUDE Real Object magnitude from the configure .fld file
12 PID Integer Program ID from the configure .fld file
13 COMMENT String Comment from the configure .fld file
14 RETRACTOR Integer 2dF retractor number
15 WLEN Real Wavelength from the configure .fld file. This column was added around 2005
16 EXPOSURE Integer This column may appear in some combined output files where the combined datasets contained a subset of common objects and therefore exposure times differ for different objects. The column gives the exposure time in seconds for the fibre.

An Important Note on 2dF Fibre-Pivot Number and 2dfdr Fibre Number


There are two very important, and very different, numbers which one must understand in order to recover the information on which object each fibre was allocated: Fibre slit position AND 2dF Fibre-Pivot position. For the most part there is a one-to-one correspondence between these numbers. Usually the fibre at AAOmega slit position 1 (bottom of the CCD image) will map directly to 2dF Pivot position 1, and 400 will map to 400 (note, 400 is a guide fibre and so maps to a blank space at the top of the CCD image). However, during manufacture or repair of each of the AAOmega slit units, it is sometimes possible for the order of fibres in each of the AAOmega slits to fall out of synchronization with the 2dF Pivot numbering. It is not practical to mechanically alter either position so each of the two fibre numbers (slit position and Pivot position) are propagated in the fibre table.

In the primary image (and also the variance array, stored in the first extension) the fibre at the bottom of the image, which is the fibre at slit position 1, corresponds to the first row in the fibre table (the second fits extension). The table contains a column entry, PIVOT, which gives the 2dF pivot position for this fibre. This is the fibre number seen by the configure software. The very top fibre in a CCD image corresponds to the very last entry in the fibre table (which will be an AAOmega guide fibre in the case of a single AAOmega data set). There is ALWAYS a one-to-one correspondence between each spectrum position in the image and the fibre table. There is typically a one-to-one correspondence between slit position and 2dF pivot position but with a number of known mismatches and discontinuities which are tracked via the PIVOT column of the fibre table.

Examples for Accessing the FITS Fibre Table


This list is not exhaustive.

With configure


One can save a list file (file menu -> ..list) which contains the allocated 2dF Fibre-Pivot number for each allocated fibre. Note, this is the Pivot number for 2dF NOT the fibre number in the reduced 2D spectra file.

2dfinfo


The 2dfinfo procedure comes packaged with 2dfdr. It can be used to recover information on the fibre from the .fits file. The syntax for the command is:

2dfinfo file.fits <option>

If the <option> is omitted then the list of options is given. To recover the fibre table information one would use:

2dfinfo file.fits fibres

IRAF


The IRAF/STSDAS package TABLES has a number of routines designed for manipulating tables. A simple example might be:

IRAF> tdump combined_frame.fits[2] > output.txt

This would create a complete listing of the fibre binary table information and pipe it to an ascii text file. Formatting the output can be achieved with:

IRAF> tprint combined_frame.fits[2] columns="NAME,RA,DEC" > output.txt

IDL


For users of IDL, the NASA IDL astronomy library has some excellent FITS data access routines.

Starting from a combined FITS frame, combined_frame.fits, one might use the following code extracts to manipulate AAOmega data. Note, there are cleverer (and quicker) ways to perform the operations below with the NASA astrolib tasks, the code here is given as a simple example.

file='combined_frame.fits'

;; Read in the spectral image, store the header information
spec=mrdfits(dir+file_comb,0,header0)

;; And the variance array
spec_var=mrdfits(dir+file_comb,1)

;; Make a wavelength vector, note the use of CRPIX1, which is often not expected by many users.
;; If missed, the wavelength solution will tend to be wrong by half a CCD width
crpix=fxpar(header0,'crpix1')-1.0 ; The -1.0 is needed as IDL is ZERO indexed
crval=fxpar(header0,'crval1')
cdelt=fxpar(header0,'cdelt1')
wave=((findgen(n_elements(spec[*,0]))-crpix)*cdelt)+crval

;; Read in the object identification information
fxbopen,unit,file,2
fxbreadm,unit $
,['name','ra','dec','x','y','xerr','yerr','theta','type','pivot','magnitude']$
,id,ra,dec,x,y,xerr,yerr,theta,type,pivot,mag<
fxbclose,unit

;; And read a copy of the sky spectrum subtracted from the data.
;; Note, for a combined frame, this is the sky spectrum from the first file in the list of combined frames.
;; It is a good representative sky spectrum, but should be used with caution for the combined spectral data.
fxbopen,unit,file,7
fxbreadm,unit,['SKY'],sky
fxbclose,unit

Axis Information

Axis information represents the abscissa and ordinate for the image or spectra contained in the file.

The abscissa information represents either the pixel number (image) or the wavelength at the centre of each pixel in Angstroms (spectra). The ordinate information represents either pixel number (image) or fibre number (spectra).

This is NOT a HDU. Instead the information is held in the FITS standard header keyword values CRVALn, CDELTn and CRPIXn, where n is either 1 (abscissa) or 2 (ordinate). FITS keywords CTYPEn and CUNITn complete the description by holding the axis label and units, respectively. These same keywords are used by external FITS viewers (See fv, ds9, etc.) to describe the axis so 2dfdr output files are correctly handled by these viewers.

Within the 2dfdr code, axis information is generally held in a one-dimension vector of 32-bit IEEE floating point values. The vector values are constrained to always being linear since only 3 keywords are used. The transformation from FITS keyword values to vector, and vice versa, is hidden in the TDFIO_AXIS_READ and TDFIO_AXIS_WRITE routines, respectively.

History: The internal vector format comes from when 2dfdr used Starlink's Extensible N-Dimensional Data Format (NDF). The NDF "CENTRE" axis component was used. This was a vector holding the pixel centre coordinates, with a separate vector for each axis. The NDF axis attributes of LABEL and UNITS were passed to the FITS keyword string values CTYPEn and CUNITn, respectively.

N.B. The functionality of this HDU is nearly identical to the WAVELA Extension, and many of the same issues apply. One of the two should/could (probably) be eliminated.

Where Used: Every file has this information. Within 2dfdr, axis information is used when plotting files (read only). The abscissa values (wavelength) are set during scrunching and splicing. They are also used to judge wavelengths of interest when matching arc peaks and known wavelength intensities (read only).

WAVELA

WAVELA is a 2-dimension binary table identical in size to the primary spectral and variance arrays. Each element holds the wavelength in nanometres of the corresponding spectral datum. The data type is 32-bit IEEE floating point. The first dimension is in the spectral direction, whilst the second is in the fibre. This extension is written and read by 2dfdr.

Once established the values in this array NEVER change.

This information is derived original derived by 2dfdr from keyword values SPECID, GRATPMM, GRATANGR, GRATANGL, CAMANGL and ORDER. The one exception is the original 2df instrument which employs a ray tracing algorithm.

Where Used: All types of output ex(tracted) files and arc red(uced) files have an WAVELA HDU. ??? Why ??? It is specifically removed from flat and science red(uced) files.

N.B. The ability to provide a separate wavelength for each pixel is NOT used. All values in a single column are identical, and the relationship between row values is constrained to be linear. The one except is the original 2df spectrograph where column differences are found. This format is being retained for future development.

N.B. This extension is nearly identical in function to the Axis Information, and many of the same issues apply. One of the two should/could be (probably) eliminated.

SHIFTS

SHIFTS is a 2-dimension binary table holding the polynomial coefficients used to rebin (aka scrunch) data onto the calibrated wavelength scale.

SHIFTS(FIBNO,COEFF) where

The data type is 32-bit IEEE floating point. Notice this is the file storage type, but normally it is used internally as 64-bit IEEE floating point (DOUBLE PRECISION). This is required by the FIG_REBIN routine (originally from FIGARO).

The name "SHIFTS" is a misnomer since the values represent a polynomial and not a simple shift. It is thought the original algorithm was a shift and this concept was expanded without changing the name.

Where Used: The WAVELA HDU is read and written by 2dfdr. It is created (written) in two situations

  1. During the spectra processing for red(uced) arc frames (see reduce_arc.f). These values are used to rebin fibre flat and science frames.
  2. During spectra processing for red(uced) sky frames (see reduce_sky.f).These values are used to rebin science frames.

THPUT

A 1-dimension binary table holding the fibre throughput. The vector has one element for each fibre. Each element contains a multiplicative factor to account for differences in fibre throughput. The data type is 32-bit IEEE floating point.

The fibre throughput is optionally computed when an object spectra is being reduced using sky fibres in the observation. The results are placed in the object frame's THPUT HDU, and also used to calibrate the object data. This is done when the 'SKYLINE' (or its variants) or 'SKYFLUX' (or its variants) throughput calculation method is chosen.

Fibre throughput can also be computed during the reduction of a sky frame. The values are placed in the sky frame's THPUT HDU, and can later be used to calibrate object frames. This is done when the 'OFFSKY' throughput calculation method is chosen when reducing the object frame.

Object spectra is scaled for fibre differences by dividing by the throughput.

Where Used: This extension is written and read by 2dfdr.

SKY

A 1-dimension binary table holding the combined and normalised sky spectrum. The vector has one element for each wavelength. The data type is 32-bit IEEE floating point.

The sky spectrum is optionally computed when an object spectra is being reduced. When it is, the sky spectrum is subtracted from each object spectrum, and the sky spectrum values are stored in this extension. Each entry contains the 'typical' sky spectrum used in the data reduction; 'typical' because for a combined frame it is not obvious how the final sky spectrum for each fibre should be represented.

The values in the SKY extension are used during splicing of AAOmega blue and red spectra for what purpose???.

Note: The variance information is correctly propagated, the sky spectrum is not presented here for this purpose.

Where Used: This extension is written and read by 2dfdr.

TELCOR

A 1-dimension binary table holding the telluric absorption correction used. The vector has one element for each wavelength. The data type is 32-bit IEEE floating point.

Where Used: This extension is ONLY written by 2dfdr.

SIGMAPRF

A 2-dimension binary array which appears ONLY in the tramline map file. The data type is 32-bit IEEE floating point. Under the assumption that the tramline PSF is a Gaussian function, the sigma value is the estimate of the sigma of the Gaussian. The array has a sigma for each fibre and each spectra pixel. This is currently (Nov2010) being explored by the astronomers with various specific experimental data. Once further confidence is obtained in the derived values they will be implemented instead of the fixed value in the various coding sections including optimal extraction.

Where Used: This HDU is currently ONLY written by 2dfdr.

DELTA

N.B. This HDU is obsolete. The Fortran code to produce this HDU has NOT been converted from NDF to FITS. FIT extraction does NOT work because in also has not been converted to FITS. See bug report

History: A 2-dimension binary table identical in size to the IMAGE array holding ???. The data type is 32-bit IEEE floating point. This and the SIGMA2 HDU were used for FIT spectra extraction. Both extensions are created when the tramline map is made. The arrays hold ??? sigma and delta of the Gaussian fit to spectra profiles ??? <<-- FILL IN HERE! They are used during the spectra extraction but again only when FIT extraction is requested. This extension is written and read by 2dfdr.

Where Used: No recent 2dfdr files have this HDU.

NDF_CLASS

A 1 x 1 binary table. Its value was part of the implementation of OO (object-orientated) Fortran. Its use is deprecated.

A 10-character string describing the file type. Known types are

MFFFF Fibre flat field image, raw and im(age)
MFSFFF Fibre flat field spectra, ex(tracted) and red(uced)
MFARC Arc image, raw and im(age)
MFSARC Arc spectra, ex(tracted) and red(uced)
MFOBJECT Science image, raw and im(age)
MFSOBJECT Science spectra, ex(tracted), red(uced), combined and spliced
MFFLX Flux calibration image, raw and im(age)
MFSFLX Flux calibration spectra, ex(tracted) and red(uced)
BIAS Bias frame image, raw and red(uced)
LFLAT Long-slit flat image, raw and red(uced)

Where Used: All 2dfdr output files contain this HDU.

REDUCED

A 1 x 1 binary table indicating the file was reduced. If present it contains the single logical value "true". Its use is deprecated.

Where Used: All red(uced) 2dfdr output files contain this HDU.




Sarah Brough (sb_at_aao.gov.au)
Ron Heald (rwh_at_aao.gov.au)