Learning from FITS: Limitations in use in modern astronomical research

The Flexible Image Transport System (FITS) standard has been a great boon to astronomy, allowing observatories, scientists and the public to exchange astronomical information easily. The FITS standard, however, is showing its age. Developed in the late 1970s, the FITS authors made a number of implementation choices that, while common at the time, are now seen to limit its utility with modern data. The authors of the FITS standard could not anticipate the challenges which we are facing today in astronomical computing. Difficulties we now face include, but are not limited to, addressing the need to handle an expanded range of specialized data product types (data models), being more conducive to the networked exchange and storage of data, handling very large datasets, and capturing significantly more complex metadata and data relationships. There are members of the community today who find some or all of these limitations unworkable, and have decided to move ahead with storing data in other formats. If this fragmentation continues, we risk abandoning the advantages of broad interoperability, and ready archivability, that the FITS format provides for astronomy. In this paper we detail some selected important problems which exist within the FITS standard today. These problems may provide insight into deeper underlying issues which reside in the format and we provide a discussion of some lessons learned. It is not our intention here to prescribe specific remedies to these issues; rather, it is to call attention of the FITS and greater astronomical computing communities to these problems in the hope that it will spur action to address them.

[1]  Tim Jenness,et al.  New Features in AST - a WCS Management and Manipulation Library , 2012, 1210.5483.

[2]  Michael McCann,et al.  Oceanographic Data Provenance Tracking with the Shore Side Data System , 2008, IPAW.

[3]  Simon Miles Automatically Adapting Source Code to Document Provenance , 2010, IPAW.

[4]  Albert J. Fleig,et al.  Provenance Tracking in an Earth Science Data Processing System , 2008, IPAW.

[5]  D. Wells,et al.  Fits: a flexible image transport system , 1981 .

[6]  Mark Taylor,et al.  HDX Data Model: FITS, NDF and XML Implementation , 2003 .

[7]  David Naylor,et al.  Testing results and current status of FTS-2, an imaging Fourier transform spectrometer for SCUBA-2 , 2010, Astronomical Telescopes + Instrumentation.

[8]  Eric Bertin,et al.  Automatic Astrometric and Photometric Calibration with SCAMP , 2006 .

[9]  R. H. Harten,et al.  The FITS Tape Formats: Flexible Image Transport Systems , 1980, Photonics West - Lasers and Applications in Science and Engineering.

[10]  Jennifer L. West,et al.  Using the medical image processing package, ImageJ, for astronomy , 2006 .

[11]  Luc Moreau,et al.  The Open Provenance Model: An Overview , 2008, IPAW.

[12]  Mark R. Calabretta,et al.  Representations of world coordinates in FITS , 2002, astro-ph/0207407.

[13]  R. H. Harten,et al.  An Extension of FITS for Groups of Small Arrays of Data , 1981 .

[14]  Jean-Mathias Griessmeier,et al.  LOFAR and HDF5: Toward a New Radio Data Standard , 2010 .

[15]  Eric W. Greisen FITS: A Remarkable Achievement in Information Exchange , 2003 .

[16]  David Berry,et al.  Developments in the Starlink Software Collection , 2009 .

[17]  Stephan Ott,et al.  The Herschel Data Processing System - HIPE and Pipelines - Up and Running Since the Start of the Mission , 2010, 1011.1209.

[18]  Clive G. Page,et al.  Definition of the Flexible Image Transport System (FITS), version 3.0 , 2010 .

[19]  T. Jenness,et al.  Advantages of Extensible Self-described Data Formats: Lessons Learned from NDF , 2014 .

[20]  Jessica D. Mink,et al.  Astronomical data formats: What we have and how we got here , 2015, Astron. Comput..

[21]  P. A. R. Ade,et al.  SCUBA-2: the 10 000 pixel bolometer camera on the James Clerk Maxwell Telescope , 2013, 1301.3650.

[22]  E. W. Greisen,et al.  Representations of spectral coordinates in FITS , 2005 .

[23]  S. Markoff,et al.  LOFAR - low frequency array , 2006 .

[24]  D. G. Barnes Realtime, Object-oriented Reduction of Parkes Multibeam Data using AIPS++ , 1998 .

[25]  Mireille Louys,et al.  Units in the VO , 2014 .

[26]  R. Manchester,et al.  psrchive and psrfits: An Open Approach to Radio Pulsar Data Storage and Analysis , 2004, Publications of the Astronomical Society of Australia.

[27]  Edward J. Shaya,et al.  Specifics on a XML Data Format for Scientific Data , 2001 .

[28]  Malcolm Currie Standard data formats , 1988 .

[29]  Malcolm J. Currie,et al.  CONVERT -- A Format-conversion Package , 2013 .

[30]  A. S. Fruchter,et al.  DrizzlePac: Managing Multi-component WCS Solutions for HST Data , 2013 .

[31]  John D. Monnier,et al.  Software tools for optical interferometry , 2006, SPIE Astronomical Telescopes + Instrumentation.

[32]  Vyacheslav V. Kitaeff,et al.  The impact of JPEG2000 lossy compression on the scientific quality of radio astronomy imagery , 2014 .

[33]  P. Dowler CAOM-2.0: The Inevitable Evolution of a Data Model , 2012 .

[34]  A. R. Whitney,et al.  The Murchison Widefield Array: The Square Kilometre Array Precursor at Low Radio Frequencies , 2012, Publications of the Astronomical Society of Australia.

[35]  Tim Cornwell,et al.  SKA Exascale Software Challenges , 2010 .

[36]  G. van Diepen,et al.  Casacore Table Data System and its use in the MeasurementSet , 2015, Astron. Comput..

[37]  Dirk Petry,et al.  Analysing ALMA data with CASA , 2012, 1201.3454.

[38]  Arcot Rajasekar,et al.  iRODS: A Distributed Data Management Cyberinfrastructure for Observatories , 2007 .

[39]  David R. DeBoer,et al.  Australian SKA Pathfinder: A High-Dynamic Range Wide-Field of View Survey Telescope , 2009, Proceedings of the IEEE.

[40]  R. H. Harten,et al.  The FITS tables extension , 1988 .

[41]  Michael Droettboom,et al.  ASDF: A new data format for astronomy , 2015 .

[42]  Edward J. Shaya,et al.  Converting FITS into XML: Methods and Advantages , 2001 .

[43]  Robert Mann,et al.  Astronomical Data Analysis Software and Systems XXI , 2012 .

[44]  Tim Jenness,et al.  Observatory/data centre partnerships and the VO-centric archive: The JCMT Science Archive experience , 2014, Astron. Comput..

[45]  Paul T. Groth,et al.  Pipeline-centric provenance model , 2009, WORKS '09.

[46]  Brian Thomas,et al.  Significant problems in FITS limit its use in modern astronomical research , 2014 .

[47]  R. O. Redman,et al.  Implementing a Common Database Architecture at the CADC using CAOM-2 , 2013 .

[48]  B. Schlesinger,et al.  Definition of the Flexible Image Transport System (FITS) , 2001 .

[49]  Robert Walter Garwood SDFITS: A Standard for Storage and Interchange of Single Dish Data , 2000 .

[50]  Rob Seaman,et al.  Astronomical Tiled Image Compression: How & Why , 2006 .

[51]  E. Greisen,et al.  Representations of celestial coordinates in FITS , 2002, astro-ph/0207413.

[52]  G. A. Renting,et al.  Status of LOFAR Data in HDF5 Format , 2012 .

[53]  Donald C. Wells,et al.  Speculations on the Future of FITS , 1997 .

[54]  Keith Shortridge,et al.  Learning from 25 years of the extensible N-Dimensional Data Format , 2014, Astron. Comput..

[55]  Malcolm J. Currie Data-format conversion , 1997 .

[56]  R. F. Warren-Smith,et al.  The STARLINK software collection , 1992 .

[57]  R. H. Harten,et al.  Generalized extensions and blocking factors for FITS , 1988 .

[58]  R. F. Warren-Smith,et al.  World Coordinate Systems as Objects , 1998 .

[59]  R. L. White,et al.  Lossless Astronomical Image Compression and the Effects of Noise , 2009, 0903.2140.

[60]  Russ R. Laher,et al.  More flexibility in representing geometric distortion in astronomical images , 2012, Other Conferences.

[61]  Tim Jenness Reimplementing the Hierarchical Data System using HDF5 , 2015 .

[62]  Francisco G. Valdes The IRAF/NOAO Spectral World Coordinate Systems , 1993 .

[63]  D. C. Price,et al.  HDFITS: Porting the FITS data model to HDF5 , 2015, Astron. Comput..

[64]  David Taubman,et al.  Astronomical imagery: Considerations for a contemporary approach with JPEG2000 , 2014 .

[65]  Yogesh L. Simmhan,et al.  The Open Provenance Model core specification (v1.1) , 2011, Future Gener. Comput. Syst..

[66]  Mark R. Calabretta,et al.  Representations of distortions in FITS world coordinate systems , 2004 .

[67]  Prasanth H. Nair,et al.  Astropy: A community Python package for astronomy , 2013, 1307.6212.

[68]  A. Belloche,et al.  APECS : the Atacama pathfinder experiment control system , 2006 .