A Compact Methodology to Understand, Evaluate, and Predict the Performance of Automatic Target Recognition

This paper offers a compacted mechanism to carry out the performance evaluation work for an automatic target recognition (ATR) system: (a) a standard description of the ATR system's output is suggested, a quantity to indicate the operating condition is presented based on the principle of feature extraction in pattern recognition, and a series of indexes to assess the output in different aspects are developed with the application of statistics; (b) performance of the ATR system is interpreted by a quality factor based on knowledge of engineering mathematics; (c) through a novel utility called “context-probability” estimation proposed based on probability, performance prediction for an ATR system is realized. The simulation result shows that the performance of an ATR system can be accounted for and forecasted by the above-mentioned measures. Compared to existing technologies, the novel method can offer more objective performance conclusions for an ATR system. These conclusions may be helpful in knowing the practical capability of the tested ATR system. At the same time, the generalization performance of the proposed method is good.

[1]  Julius T. Tou,et al.  Pattern Recognition Principles , 1974 .

[2]  Bir Bhanu,et al.  Automatic Target Recognition: State of the Art Survey , 1986, IEEE Transactions on Aerospace and Electronic Systems.

[3]  Firooz A. Sadjadi,et al.  Automatic Target Recognition Algorithm Performance Evaluation: The Bottleneck In The Development Life Cycle , 1989, Defense, Security, and Sensing.

[4]  Vincent J. Velten,et al.  Image characterization for automatic target recognition algorithm evaluations , 1990, Defense, Security, and Sensing.

[5]  Hatem N. Nasr,et al.  Automatic evaluation and adaptation of automatic target recogition systems , 1990, Defense, Security, and Sensing.

[6]  M. W. Roth.  Survey of neural network technology for automatic target recognition , 1990, IEEE Trans. Neural Networks.

[7]  D. Pei,et al.  A combinatorial approach toward DNA recognition , 1991, Science.

[8]  Christine M. Netishen,et al.  Performance of a High-Resolution Polarimetric SAR Automatic Target Recognition System , 1993 .

[9]  B. Bhanu,et al.  Image understanding research for automatic target recognition , 1993, IEEE Aerospace and Electronic Systems Magazine.

[10]  Effects of Polarization and Resolution on the Performance of a SAR Automatic Target Recognition System , 1995 .

[11]  Edmund G. Zelnio,et al.  Characterization of ATR performance evaluation , 1996, Defense, Security, and Sensing.

[12]  Edmund G. Zelnio,et al.  Extensibility and other model-based ATR evaluation concepts , 1997, Defense, Security, and Sensing.

[13]  Mark S. Schmalz.  Automated analysis and prediction of accuracy and performance in ATR algorithms: II. Experimental results and system performance analysis , 1997, Defense, Security, and Sensing.

[14]  Gerhard X. Ritter,et al.  Performance evaluation of data compression transforms for underwater imaging and object recognition , 1997, Oceans '97. MTS/IEEE Conference Proceedings.

[15]  Mark S. Schmalz.  Automated analysis and prediction of accuracy and performance in ATR algorithms: I. Requirements, theory, and software implementation , 1997, Defense, Security, and Sensing.

[16]  B. D. Guenther,et al.  Aided and automatic target recognition based upon sensory inputs from image forming systems , 1997 .

[17]  Timothy D. Ross,et al.  Evaluation of SAR ATR algorithm performance sensitivity to MSTAR extended operating conditions , 1998, Defense, Security, and Sensing.

[18]  Michael I. Miller,et al.  Hilbert-Schmidt Lower Bounds for Estimators on Matrix Lie Groups for ATR , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[19]  Vince E. Diehl,et al.  Upper bound calculations of ATR performance for ladar sensors , 1998, Defense, Security, and Sensing.

[20]  Larry L. Horowitz,et al.  Fundamental SAR ATR performance predictions for design trade-offs: 1D HRR versus 2D SAR versus 3D SAR , 1999, Defense, Security, and Sensing.

[21]  Randolph L. Moses,et al.  ATR performance prediction using attributed scattering features , 1999, Defense, Security, and Sensing.

[22]  Bir Bhanu,et al.  Bounding SAR ATR performance based on model similarity , 1999, Defense, Security, and Sensing.

[23]  Jeffrey M. Hausdorff,et al.  Physionet: Components of a New Research Resource for Complex Physiologic Signals". Circu-lation Vol , 2000 .

[24]  Debashis Kushary,et al.  Bootstrap Methods and Their Application , 2000, Technometrics.

[25]  Dan E. Dudgeon,et al.  ATR Performance Modeling and Estimation , 2000, Digit. Signal Process..

[26]  L. Álvarez-Vallina,et al.  Pharmacologic suppression of target cell recognition by engineered T cells expressing chimeric T-cell receptors , 2000, Cancer Gene Therapy.

[27]  Timothy D. Ross.  Confidence intervals for ATR performance metrics , 2001, SPIE Defense + Commercial Sensing.

[28]  Jeffrey H. Shapiro,et al.  Analytic performance bounds on SAR-image target recognition using physics-based signatures , 2001, SPIE Defense + Commercial Sensing.

[29]  John M. Irvine.  Evaluation of ATR algorithms employing motion imagery , 2001, Proceedings 30th Applied Imagery Pattern Recognition Workshop (AIPR 2001). Analysis and Understanding of Time Varying Imagery.

[30]  Qun Zhao,et al.  Support vector machines for SAR automatic target recognition , 2001 .

[31]  John C. Mossing,et al.  Open source tools for ATR development and performance evaluation , 2002, SPIE Defense + Commercial Sensing.

[32]  Kannan Ramchandran,et al.  Information-Theoretic Bounds on Target Recognition Performance Based on Degraded Image Data , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[33]  David Blacknell.  A comparison of SAR ATR performance with information theoretic predictions , 2003, SPIE Defense + Commercial Sensing.

[34]  Bir Bhanu,et al.  Performance modeling of vote-based object recognition , 2003, SPIE Defense + Commercial Sensing.

[35]  Mark R. Stevens,et al.  A scoring, truthing, and registration toolkit for evaluation of target detection and tracking , 2004, SPIE Defense + Commercial Sensing.

[36]  Vincent L. Myers,et al.  Information theoretic bounds of ATR algorithm performance for sidescan sonar target classification , 2005, SPIE Defense + Commercial Sensing.

[37]  Steven C. Gustafson,et al.  Receiver operating characteristic and confidence error metrics for assessing the performance of automatic target recognition systems , 2005 .

[38]  Erik Blasch,et al.  Fidelity score for ATR performance modeling , 2005, SPIE Defense + Commercial Sensing.

[39]  Mark R. Stevens,et al.  An Image Metric-Based ATR Performance Prediction Testbed , 2006, 35th IEEE Applied Imagery and Pattern Recognition Workshop (AIPR'06).

[40]  Fredrik Gustafsson,et al.  Ground Target Recognition Using Rectangle Estimation , 2006, IEEE Transactions on Image Processing.

[41]  Mark R. Stevens,et al.  Evaluation testbed for ATD performance prediction (ETAPP) , 2006, SPIE Defense + Commercial Sensing.

[42]  Gábor Lugosi,et al.  Prediction, learning, and games , 2006 .

[43]  Magnús Snorrason,et al.  Evaluation testbed for ATD performance prediction (ETAPP) , 2007, SPIE Defense + Commercial Sensing.

[44]  Erik Blasch,et al.  Score-based SAR ATR performance model with operating condition dependencies , 2007, SPIE Defense + Commercial Sensing.

[45]  Bahram Javidi,et al.  Physics of Automatic Target Recognition , 2007 .

[46]  A. Leon-Garcia.  Probability, statistics, and random processes for electrical engineering , 2008 .

[47]  María Dolores Ugarte,et al.  Probability and Statistics with R , 2008 .

[48]  Xiang Li,et al.  Performance evaluation for automatic target recognition based on cloud theory , 2008, 2008 International Conference on Radar.

[49]  Tomaž Ambrožič,et al.  Use of Automatic Target Recognition System for the Displacement Measurements in a Small Diameter Tunnel Ahead of the Face of the Motorway Tunnel During Excavation , 2008, Sensors.

[50]  Jordi J. Mallorqui,et al.  Assessment of Polarimetric SAR Interferometry for Improving Ship Classification based on Simulated Data , 2008, Sensors.

[51]  Boris Kovalerchuk,et al.  Modeling ATR processes to predict their performance by using invariance, robustness and self-refusal approach , 2009, 2009 12th International Conference on Information Fusion.

[52]  Yael Edan,et al.  An Objective Function to Evaluate Performance of Human–Robot Collaboration in Target Recognition Tasks , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[53]  Tatsuya Kawahara,et al.  Computer Assisted Language Learning system based on dynamic question generation and error prediction for automatic speech recognition , 2009, Speech Commun..

[54]  D. Blacknell.  Shadow-based SAR ATR performance prediction , 2009, Defense + Commercial Sensing.

[55]  Yimin Liu,et al.  Extended Target Recognition in Cognitive Radar Networks , 2010, Sensors.

[56]  Chein-I Chang,et al.  Multiparameter Receiver Operating Characteristic Analysis for Signal Detection and Classification , 2010, IEEE Sensors Journal.

[57]  J. Steitz,et al.  Poly(A) Tail Recognition by a Viral RNA Element Through Assembly of a Triple Helix , 2010, Science.

[58]  Hoi-Shun Lui,et al.  Performance evaluation of subsurface target recognition based on ultrawideband short-pulse excitation , 2010, 2010 IEEE Antennas and Propagation Society International Symposium.

[59]  Cheng Xiao,et al.  Automatic Target Recognition of SAR Images Based on Global Scattering Center Model , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[60]  Jianlin Zhao,et al.  [EEG signal classification based on EMD and SVM]. , 2011, Sheng wu yi xue gong cheng xue za zhi = Journal of biomedical engineering = Shengwu yixue gongchengxue zazhi.

[61]  Alice J. O'Toole,et al.  Demographic effects on estimates of automatic face recognition performance , 2011, Face and Gesture 2011.

[62]  Gui Gao,et al.  An Improved Scheme for Target Discrimination in High-Resolution SAR Images , 2011, IEEE Transactions on Geoscience and Remote Sensing.

[63]  Liu Jin-jiang,et al.  The Detection and Recognition of Electrocardiogram's Waveform Based on Sparse Decomposition and Neural Network , 2011 .

[64]  Luo Lai-bang,et al.  A Infrared Target Recognition Method Based on Classifier Combination , 2012 .

[65]  Kenneth W. Bauer,et al.  Automatic Target Recognition Classification System Evaluation Methodology , 2012 .

[66]  Alice J. O'Toole,et al.  Demographic effects on estimates of automatic face recognition performance , 2012, Image Vis. Comput..

[67]  Kenneth W. Bauer,et al.  The Evaluation of Competing Classifiers , 2012 .

[68]  Jingchang Huang,et al.  Seismic Target Classification Using a Wavelet Packet Manifold in Unattended Ground Sensors Systems , 2013, Sensors.

[69]  José Luis Lázaro,et al.  Computational Burden Resulting from Image Recognition of High Resolution Radar Sensors , 2013, Sensors.

[70]  Jean-Jacques Laurin,et al.  Study of Microwave Tomography Measurement Setup Configurations for Breast Cancer Detection Based on Breast Compression , 2013 .

[71]  Jing Zhu,et al.  Sensor Reliability Evaluation Scheme for Target Classification Using Belief Function Theory , 2013, Sensors.

[72]  Weidong Yang,et al.  ATR algorithm performance evaluation based on the simulation image and real image , 2013, Other Conferences.

[73]  Naveen Kumar,et al.  An Overview of Automatic Target Recognition Systems for Underwater Mine Classification , 2016 .