The LOFAR Transients Pipeline

Current and future astronomical survey facilities provide a remarkably rich opportunity for transient astronomy, combining unprecedented fields of view with high sensitivity and the ability to access previously unexplored wavelength regimes. This is particularly true of LOFAR, a recently-commissioned, low-frequency radio interferometer, based in the Netherlands and with stations across Europe. The identification of and response to transients is one of LOFAR's key science goals. However, the large data volumes which LOFAR produces, combined with the scientific requirement for rapid response, make automation essential. To support this, we have developed the LOFAR Transients Pipeline, or TraP. The TraP ingests multi-frequency image data from LOFAR or other instruments and searches it for transients and variables, providing automatic alerts of significant detections and populating a lightcurve database for further analysis by astronomers. Here, we discuss the scientific goals of the TraP and how it has been designed to meet them. We describe its implementation, including both the algorithms adopted to maximize performance as well as the development methodology used to ensure it is robust and reliable, particularly in the presence of artefacts typical of radio astronomy imaging. Finally, we report on a series of tests of the pipeline carried out using simulated LOFAR observations with a known population of transients.

[1]  Tara Murphy,et al.  VAST - a real-time pipeline for detecting radio transients and variables on the Australian SKA Pathfinder (ASKAP) telescope , 2012, 1201.3130.

[2]  Timothy M. Brown,et al.  Las Cumbres Observatory Global Telescope , 2009 .

[3]  Steve Croft,et al.  THE ALLEN TELESCOPE ARRAY Pi GHz SKY SURVEY II. DAILY AND MONTHLY MONITORING FOR TRANSIENTS AND VARIABILITY IN THE BOÖTES FIELD , 2011, 1107.1517.

[4]  University of Sydney,et al.  The Phoenix Deep Survey: The 1.4 GHz Microjansky Catalog , 2002, astro-ph/0211068.

[5]  G. Amdhal,et al.  Validity of the single processor approach to achieving large scale computing capabilities , 1967, AFIPS '67 (Spring).

[6]  Robert J. Brunner,et al.  XID: Cross-Association of ROSAT/Bright Source Catalog X-Ray Sources with USNO A-2 Optical Point Sources , 2000 .

[7]  Steve Croft,et al.  ASGARD: A LARGE SURVEY FOR SLOW GALACTIC RADIO TRANSIENTS. I. OVERVIEW AND FIRST RESULTS , 2012, 1211.1042.

[8]  Rob Seaman,et al.  Sky Event Reporting Metadata Version 2.0 , 2011 .

[9]  L.H.A. Scheers Transient and variable radio sources in the LOFAR sky: an architecture for a detection framework , 2011 .

[10]  K. Grainge,et al.  Automated rapid follow-up of Swift gamma-ray burst alerts at 15 GHz with the AMI Large Array , 2012, 1211.3115.

[11]  A. Bridle,et al.  Variability of extragalactic sources at 2.7 GHz. III - The nature of the variations in different source classes , 1977 .

[12]  Walter A. Hendricks,et al.  The Sampling Distribution of the Coefficient of Variation , 1936 .

[13]  Alle-Jan van der Veen,et al.  Self-Calibration for the LOFAR Radio Astronomical Array , 2007, IEEE Transactions on Signal Processing.

[14]  Christopher R. Genovese,et al.  A New Source Detection Algorithm Using the False-Discovery Rate , 2001, astro-ph/0110570.

[15]  S. Markoff,et al.  The LOFAR Multifrequency Snapshot Sky Survey (MSSS) - I. Survey description and first results , 2015 .

[16]  Jerry C. Whitaker,et al.  The Electronics Handbook , 2005 .

[17]  Ramesh Narayan,et al.  The physics of pulsar scintillation , 1992, Philosophical Transactions of the Royal Society of London. Series A: Physical and Engineering Sciences.

[18]  R. Lande,et al.  On Comparing Coefficients of Variation , 1977 .

[19]  Halton Arp,et al.  A Westerbork 1415 MHz survey of background radio sources. II - Optical identifications with deep IIIA-J plates , 1977 .

[20]  B. Stappers,et al.  The LOFAR Known Pulsar Data Pipeline , 2010 .

[21]  R. Fisher Dispersion on a sphere , 1953, Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences.

[22]  E. Greisen,et al.  The NRAO VLA Sky Survey , 1996 .

[23]  Bart Scheers,et al.  The LOFAR Transients Key Project , 2006 .

[24]  Stefan J. Wijnholds,et al.  AARTFAAC: Towards a 24x7, All-sky Monitor for LOFAR , 2012 .

[25]  B. J. Rickett,et al.  Radio propagation through the turbulent interstellar plasma. , 1990 .

[26]  B. Bolch,et al.  More on Unbiased Estimation of the Standard Deviation , 1968 .

[27]  Larry Denneau,et al.  The Pan-STARRS Moving Object Processing System , 2013, 1302.7281.

[28]  P. C. Gregory,et al.  The Nature of the First Cygnus X-3 Radio Outburst , 1972 .

[29]  Baltimore,et al.  VARIABLE AND TRANSIENT RADIO SOURCES IN THE FIRST SURVEY , 2011, 1107.5901.

[30]  Shami Chatterjee,et al.  A 22-yr southern sky survey for transient and variable radio sources using the Molonglo Observatory Synthesis Telescope , 2010, 1011.0003.

[31]  B. M. Gaensler,et al.  Long-term Monitoring of Molonglo Calibrators , 1999, Publications of the Astronomical Society of Australia.

[32]  Peter Michelson,et al.  The Large Area Telescope , 2007 .

[33]  J. Swinbank The LOFAR Transients Pipeline , 2010 .

[34]  R. A. M. J. Wijers,et al.  Probing the bright radio flare and afterglow of GRB 130427A with the Arcminute Microkelvin Imager , 2014, 1403.2217.

[35]  J. Swinbank Standards and systems for transient response , 2009 .

[36]  Brian E. Granger,et al.  IPython: A System for Interactive Scientific Computing , 2007, Computing in Science & Engineering.

[37]  Prasanth H. Nair,et al.  Astropy: A community Python package for astronomy , 2013, 1307.6212.

[38]  E. O. Ofek,et al.  SENSITIVE SEARCH FOR RADIO VARIABLES AND TRANSIENTS IN THE EXTENDED CHANDRA DEEP FIELD SOUTH , 2013, 1303.6282.

[39]  F. E. A Relational Model of Data Large Shared Data Banks , 2000 .

[40]  H. Rottgering,et al.  The Westerbork Northern Sky Survey (WENSS) I. A 570 square degree Mini-Survey around the North Ecliptic Pole ? , 1997 .

[41]  J. G. Jernigan,et al.  First Results from the All-Sky Monitor on the Rossi X-Ray Timing Explorer , 1996, astro-ph/9608109.

[42]  Y. Lyubarsky A model for fast extragalactic radio bursts , 2014, 1401.6674.

[43]  R. P. Fender,et al.  Radio Transients: An antediluvian review , 2011 .

[44]  Martin L. Kersten,et al.  MonetDB/SQL Meets SkyServer: the Challenges of a Scientific Database , 2007, 19th International Conference on Scientific and Statistical Database Management (SSDBM 2007).

[45]  Observatoire de la Cote d'Azur,et al.  Progress with the LOFAR Imaging Pipeline , 2009 .

[46]  J. Chiang,et al.  THE LARGE AREA TELESCOPE ON THE FERMI GAMMA-RAY SPACE TELESCOPE MISSION , 2009, 0902.1089.

[47]  F. A. Jenet,et al.  Using the Intensity Modulation Index to Test Pulsar Radio Emission Models , 2003, astro-ph/0306031.

[48]  Karen Randall,et al.  The Allen Telescope Array: The First Widefield, Panchromatic, Snapshot Radio Camera for Radio Astronomy and SETI , 2009, Proceedings of the IEEE.

[49]  A. J. van der Horst,et al.  New methods to constrain the radio transient rate: results from a survey of four fields with LOFAR. , 2014, Monthly notices of the Royal Astronomical Society.

[50]  James J. Condon,et al.  ERRORS IN ELLIPTICAL GAUSSIAN FITS , 1997 .

[51]  Gianni Bernardi,et al.  The First Deep WSRT 150~MHz Full Polarization Observations , 2009 .

[52]  Martin L. Kersten,et al.  MonetDB: Two Decades of Research in Column-oriented Database Architectures , 2012, IEEE Data Eng. Bull..

[53]  R. S. Booth,et al.  An Overview of the MeerKAT Project , 2012 .

[54]  S. R. Kulkarni,et al.  An outburst of relativistic particles from the soft γ-ray repeater SGR1900+14 , 1999, Nature.

[55]  E. Bertin,et al.  SExtractor: Software for source extraction , 1996 .

[56]  John Swinbank Comet: A VOEvent broker , 2014 .

[57]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[58]  A. Noutsos,et al.  The LOFAR pilot surveys for pulsars and fast radio transients , 2014, 1408.0411.

[59]  Ernest E. Croner,et al.  The Palomar Transient Factory: System Overview, Performance, and First Results , 2009, 0906.5350.

[60]  Sergey E. Koposov,et al.  Q3C, Quad Tree Cube -- The new Sky-indexing Concept for Huge Astronomical Catalogues and its Realization for Main Astronomical Queries (Cone Search and Xmatch) in Open Source Database PostgreSQL , 2006 .

[61]  A. T. McKay,et al.  Distribution of the Coefficient of Variation and the Extended “T” Distribution , 1932 .

[62]  J. Conway,et al.  LOFAR: Recent Imaging Results and Future Prospects , 2011, 1106.3195.

[63]  Ben Stappers,et al.  LOFAR Transients and the Radio Sky Monitor , 2008 .

[64]  Amir Sagiv,et al.  Collective Processes in Relativistic Plasma and Their Implications for Gamma-Ray Burst Afterglows , 2002, astro-ph/0202337.

[65]  Alexander S. Szalay,et al.  The Zones Algorithm for Finding Points-Near-a-Point or Cross-Matching Spatial Datasets , 2007, ArXiv.

[66]  S. Markoff,et al.  LOFAR - low frequency array , 2006 .

[67]  R. Perley,et al.  The VLA Low-Frequency Sky Survey , 2005, 0706.1191.

[68]  Steve Croft,et al.  THE ALLEN TELESCOPE ARRAY TWENTY-CENTIMETER SURVEY-A 700-SQUARE-DEGREE, MULTI-EPOCH RADIO DATA SET. II. INDIVIDUAL EPOCH TRANSIENT STATISTICS , 2011 .

[69]  A. R. Whitney,et al.  A survey for transients and variables with the Murchison Widefield Array 32-tile prototype at 154 MHz , 2013, 1311.2989.

[70]  Oxford,et al.  Exploring the Optical Transient Sky with the Palomar Transient Factory , 2009, 0906.5355.

[71]  Jonathan I. Katz,et al.  LOW FREQUENCY RADIO PULSES FROM GAMMA-RAY BURSTS? , 2000 .

[72]  J. N. Spreeuw Search and detection of low frequency radio transients , 2004 .

[73]  Kirk M. Wolter,et al.  An Investigation of Some Estimators of Variance for Systematic Sampling , 1984 .

[74]  B. Bolch,et al.  The Teacher's Corner: More on Unbiased Estimation of the Standard Deviation , 1968 .

[75]  宮川 雄大,et al.  日本天文学会 早川幸男基金による渡航報告書 VI Microquasar Workshop-Microquasars and beyond- , 2007 .

[76]  A. Noutsos,et al.  Observing pulsars and fast transients with LOFAR , 2011, 1104.1577.

[77]  J. Högbom,et al.  APERTURE SYNTHESIS WITH A NON-REGULAR DISTRIBUTION OF INTERFEROMETER BASELINES. Commentary , 1974 .

[78]  Nolan Li,et al.  Batch is back: CasJobs, serving multi-TB data on the Web , 2005, IEEE International Conference on Web Services (ICWS'05).

[79]  D. Frail,et al.  The radio afterglow from the γ-ray burst of 8 May 1997 , 1997, Nature.

[80]  W. A. Dent Quasi-Stellar Sources: Variation in the Radio Emission of 3C 273 , 1965, Science.

[81]  R. J. Nijboer,et al.  LOFAR imaging capabilities and system sensitivity , 2013, 1308.4267.

[82]  Eduardo Serrano,et al.  LSST: From Science Drivers to Reference Design and Anticipated Data Products , 2008, The Astrophysical Journal.

[83]  John E. Freund,et al.  Outline of Basic Statistics: Dictionary and Formulas , 2010 .

[84]  R. Lupton,et al.  A Method for Optimal Image Subtraction , 1997, astro-ph/9712287.

[85]  Werner Hürlimann,et al.  A uniform approximation to the sampling distribution of the coefficient of variation , 1995 .

[86]  E. O. Ofek,et al.  A REVISED VIEW OF THE TRANSIENT RADIO SKY , 2011, 1111.0007.

[87]  P. Williams,et al.  MILLISECOND IMAGING OF RADIO TRANSIENTS WITH THE POCKET CORRELATOR , 2011, 1106.4876.

[88]  Jorge J. Moré,et al.  The Levenberg-Marquardt algo-rithm: Implementation and theory , 1977 .

[89]  H. Laan,et al.  A Model for Variable Extragalactic Radio Sources , 1966, Nature.

[90]  Will Saunders,et al.  On the likelihood ratio for source identification. , 1992 .

[91]  Andrew Hopkins,et al.  Compact continuum source finding for next generation radio surveys , 2012, 1202.4500.

[92]  C. R. Robinson,et al.  The Burst and Transient Source Experiment Earth Occultation Technique , 2002 .

[93]  Peter Z. Kunszt,et al.  Indexing the Sphere with the Hierarchical Triangular Mesh , 2007, ArXiv.

[94]  D. Dragomir,et al.  Las Cumbres Observatory Global Telescope Network , 2013, 1305.2437.

[95]  Roland Diehl,et al.  THE FERMI GAMMA-RAY BURST MONITOR , 2009, 0908.0450.

[96]  Eric W. Greisen,et al.  AIPS, the VLA, and the VLBA , 2003 .

[97]  John D. Hunter,et al.  Matplotlib: A 2D Graphics Environment , 2007, Computing in Science & Engineering.

[98]  G. J. Fishman,et al.  BATSE - The burst and transient source experiment on the Gamma Ray Observatory , 1992 .

[99]  A. J. Drake,et al.  FIRST RESULTS FROM THE CATALINA REAL-TIME TRANSIENT SURVEY , 2008, 0809.1394.

[100]  David W. Stockburger,et al.  Sampling Distribution , 2011, International Encyclopedia of Statistical Science.

[101]  A. R. Whitney,et al.  The Murchison Widefield Array: The Square Kilometre Array Precursor at Low Radio Frequencies , 2012, Publications of the Astronomical Society of Australia.

[102]  J. Curran,et al.  VAST: An ASKAP Survey for Variables and Slow Transients , 2012, Publications of the Astronomical Society of Australia.

[103]  E. F. Codd,et al.  A relational model of data for large shared data banks , 1970, CACM.

[104]  E. O. Ofek,et al.  An Extremely Luminous Panchromatic Outburst from the Nucleus of a Distant Galaxy , 2011, Science.

[105]  Matthew J. Graham,et al.  Responding to the event deluge , 2012, Other Conferences.

[106]  Alan A. Wells,et al.  The Swift Gamma-Ray Burst Mission , 2004, astro-ph/0405233.

[107]  William Kent,et al.  ASlMPLE GUIDE TO FIVE NORMAL FORMS IN RELATIONAL DATABASE THEORY , 2000 .