Digging Deeper: Algorithms for Computationally-Limited Searches in Astronomy

Astronomy, like most other fields, is being deluged by exponentially growing streams of ever more complex data. While these massive data streams bring a great discovery potential, their full scientific exploitation poses many challenges, due to both data volumes and data complexity. Moreover, the need to discover and characterize interesting, faint signals in such data streams quickly and robustly, in order to deploy costly follow-up resources that are often necessary for the full scientific returns, makes the challenges even sharper. Examples in astronomy include transient events and variable sources found in digital synoptic sky surveys, gravitational wave signals, faint radio transients, pulsars, and other types of variable sources in the next generation of panoramic radio surveys, etc. Similar situations arise in the context of space science and planetary exploration, environmental monitoring, security, etc. In most cases, rapid discovery and characterization of interesting signals is highly computationally limited. The goal of this study was to define a number of interesting, often mission-critical challenges of this nature in the broader context of time-domain astronomy, but with an eye on their applicability elsewhere.

[1]  Ciro Donalek,et al.  A comparison of period finding algorithms , 2013, 1307.2209.

[2]  David R. Thompson,et al.  Semi‐supervised Eigenbasis novelty detection , 2013, Stat. Anal. Data Min..

[3]  J. Curran,et al.  VAST: An ASKAP Survey for Variables and Slow Transients , 2012, Publications of the Astronomical Society of Australia.

[4]  Pavlos Protopapas,et al.  An Information Theoretic Algorithm for Finding Periodicities in Stellar Light Curves , 2012, IEEE Transactions on Signal Processing.

[5]  Ciro Donalek,et al.  Flashes in a star stream: Automated classification of astronomical transient events , 2012, 2012 IEEE 8th International Conference on E-Science.

[6]  S. Djorgovski,et al.  Data challenges of time domain astronomy , 2012, Distributed and Parallel Databases.

[7]  Ciro Donalek,et al.  Connecting the time domain community with the Virtual Astronomical Observatory , 2012, Other Conferences.

[8]  C. Donalek,et al.  CLaSPS: A NEW METHODOLOGY FOR KNOWLEDGE EXTRACTION FROM COMPLEX ASTRONOMICAL DATA SETS , 2012, 1206.2919.

[9]  David R. Thompson Machine Learning for Exploring Data Streams, with Lessons from the Very Long Baseline Array , 2012 .

[10]  LIMITS ON THE EVENT RATES OF FAST RADIO TRANSIENTS FROM THE V-FASTR EXPERIMENT , 2012, 1205.5840.

[11]  S. Djorgovski,et al.  Sky Surveys , 2012, 1203.5111.

[12]  S. G. Djorgovski,et al.  DAta Mining and Exploration (DAME): New Tools for Knowledge Discovery in Astronomy , 2012 .

[13]  S. G. Djorgovski,et al.  Constructing Concept Schemes From Astronomical Telegrams Via Natural Language Clustering , 2012 .

[14]  Saudi Arabia,et al.  Classification by Boosting Differences in Input Vectors , 2012 .

[15]  Umaa Rebbapragada,et al.  VAST Memo Offline and Online Classification of Simulated VAST Transients , 2012 .

[16]  S. Djorgovski,et al.  Towards an Automated Classification of Transient Events in Synoptic Sky Surveys , 2011, 1110.4655.

[17]  David R. Thompson,et al.  Semi-Supervised Novelty Detection with Adaptive Eigenbases, and Application to Radio Transients , 2011 .

[18]  Massimo Brescia,et al.  Extracting Knowledge From Massive Astronomical Data Sets , 2011 .

[19]  A. A. Mahabal,et al.  Exploring the Time Domain with Synoptic Sky Surveys , 2011, Proceedings of the International Astronomical Union.

[20]  S. G. Djorgovski,et al.  Real-Time Classification of Transient Events in Synoptic Sky Surveys , 2011, Proceedings of the International Astronomical Union.

[21]  S. Djorgovski,et al.  The Catalina Real-time Transient Survey , 2011, Proceedings of the International Astronomical Union.

[22]  Rob Seaman,et al.  The VAO Transient Facility , 2011 .

[23]  S. G. Djorgovski,et al.  Discovery, classification, and scientific exploration of transient events from the Catalina Real-time Transient Survey , 2011, 1111.0313.

[24]  David R. Thompson,et al.  DETECTION OF FAST RADIO TRANSIENTS WITH MULTIPLE STATIONS: A CASE STUDY USING THE VERY LONG BASELINE ARRAY , 2011, 1104.4900.

[25]  Joshua S. Bloom,et al.  Data Mining and Machine-Learning in Time-Domain Discovery & Classification , 2011, 1104.3142.

[26]  A. A. Mahabal,et al.  The Catalina Real-Time Transient Survey (CRTS) , 2011, 1102.5004.

[27]  A. Mahabal,et al.  SkyDiscovery: Humans and Machines Working Together , 2011 .

[28]  S. Djorgovski,et al.  Extracting meaning from astronomical telegrams , 2011 .

[29]  Wei-Yin Loh,et al.  Classification and regression trees , 2011, WIREs Data Mining Knowl. Discov..

[30]  Ciro Donalek,et al.  Exploring the Variable Sky with the Catalina Real-Time Transient Survey , 2010 .

[31]  Alain Biem,et al.  A streaming approach to radio astronomy imaging , 2010, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing.

[32]  Ciro Donalek,et al.  Mixing Bayesian Techniques for Effective Real-time Classification of Astronomical Transients , 2010 .

[33]  Ciro Donalek,et al.  Classification of Optical Transients: Experiences from PQ and CRTS Surveys , 2010 .

[34]  Marie desJardins,et al.  Confidence-Based Feature Acquisition to Minimize Training and Test Costs , 2010, SDM.

[35]  A. J. Drake,et al.  FIRST RESULTS FROM THE CATALINA REAL-TIME TRANSIENT SURVEY , 2008, 0809.1394.

[36]  H. Jaap van den Herik,et al.  The ROC isometrics approach to construct reliable classifiers , 2009, Intell. Data Anal..

[37]  C. Donalek,et al.  New Approaches to Object Classification in Synoptic Sky Surveys , 2008 .

[38]  C. Donalek,et al.  Towards Real-time Classification of Astronomical Transients , 2008, 0810.4527.

[39]  S. G. Djorgovski,et al.  Automated probabilistic classification of transients and variables , 2008, 0802.3199.

[40]  J. G. Doyle,et al.  Periodic Bursts of Coherent Radio Emission from an Ultracool Dwarf , 2007, 0705.2054.

[41]  S. G. Djorgovski,et al.  The Palomar-Quest digital synoptic sky survey , 2007, 0801.3005.

[42]  Caltech,et al.  Object detection in multi-epoch data , 2006, astro-ph/0612707.

[43]  Chris H. Q. Ding,et al.  Supernova Recognition Using Support Vector Machines , 2006, 2006 5th International Conference on Machine Learning and Applications (ICMLA'06).

[44]  S. Bailey,et al.  How to Find More Supernovae with Less Work: Object Classification Techniques for Difference Imaging , 2006, 0705.0493.

[45]  E. Glikman,et al.  Some Pattern Recognition Challenges in Data-Intensive Astronomy , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[46]  R. N. Manchester,et al.  Transient radio bursts from rotating neutron stars , 2005, Nature.

[47]  B. Krishnan,et al.  Improved Stack-Slide Searches for Gravitational-Wave Pulsars , 2005, gr-qc/0505082.

[48]  T. Lazio,et al.  A powerful bursting radio source towards the Galactic Centre , 2005, Nature.

[49]  Jeffrey D. Scargle,et al.  An algorithm for optimal partitioning of data on an interval , 2003, IEEE Signal Processing Letters.

[50]  Nick Kaiser,et al.  Pan-STARRS: a wide-field optical survey telescope array , 2004, SPIE Astronomical Telescopes + Instrumentation.

[51]  Qiang Yang,et al.  Decision trees with minimal costs , 2004, ICML.

[52]  J. Anthony Tyson,et al.  Survey and Other Telescope Technologies and Discoveries , 2002 .

[53]  J. Anthony Tyson,et al.  Large Synoptic Survey Telescope: Overview , 2002, SPIE Astronomical Telescopes + Instrumentation.

[54]  Robert J. Brunner,et al.  Exploration of parameter spaces in a virtual observatory , 2001, SPIE Optics + Photonics.

[55]  D. Frail,et al.  Discovery of Radio Emission from the Brown Dwarf LP944-20 , 2001, astro-ph/0102301.

[56]  Asu,et al.  Exploration of Large Digital Sky Surveys , 2000, astro-ph/0012489.

[57]  B. Paczyński Monitoring All Sky for Variability , 2000, astro-ph/0005284.

[58]  R Edelson,et al.  The Discrete Correlation Function: a New Method for Analyzing Unevenly Sampled Variability Data , 1988 .

[59]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .