The Deputy Under Secretary of Defense for Science and Technology (DUSD/S&T), as part of their ongoing ATR Program, has sponsored an effort to develop and demonstrate methods for evaluating ATR algorithms that utilize multiple data sources, i.e., fusion-based ATR. The AFRL COMPASE Center has formed a strong ATR evaluation team and this paper presents results from this program, focusing on the human-in-the-loop, i.e. assisted image exploitation. Reliance on Automated Target Recognition (ATR) technology is essential to the future success of Intelligence, Surveillance, and Reconnaissance (ISR) missions. Often, ATR technology is designed to aid the analyst, but the final decision rests with the human. Traditionally, evaluation of ATR systems has focused mainly on the performance of the algorithm. Assessing the benefits of ATR assistance for the user raises interesting methodological challenges. We will review the critical issues associated with evaluations of human-in-the-loop ATR systems and present a methodology for conducting these evaluations. Experimental design issues addressed in this discussion include training, learning effects, and human factors issues. The evaluation process becomes increasingly complex when data fusion is introduced. Even in the absence of ATR assistance, the simultaneous exploitation of multiple frames of co-registered imagery is not well understood. We will explore how the methodology developed for exploitation of a single source of data can be extended to the fusion setting.
[1]
John F. Hamilton,et al.
A Free Response Approach To The Measurement And Characterization Of Radiographic Observer Performance
,
1977,
Other Conferences.
[2]
A Kirlik.
Modeling Strategic Behavior in Human-Automation Interaction: Why an "Aid" Can (and Should) Go Unused
,
1993,
Human factors.
[3]
Tim J. Patterson,et al.
Multisensor image cueing (MUSIC)
,
2002,
SPIE Defense + Commercial Sensing.
[4]
D P Chakraborty,et al.
Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data.
,
1989,
Medical physics.
[5]
G. G. Kuperman.
Human system interface (HSI) issues in assisted target recognition (ASTR)
,
1997,
Proceedings of the IEEE 1997 National Aerospace and Electronics Conference. NAECON 1997.
[6]
John M. Irvine.
Evaluating assisted target recognition performance: an assessment of DARPA's SAIP system
,
1999,
Defense, Security, and Sensing.
[7]
N Moray,et al.
Trust, control strategies and allocation of function in human-machine systems.
,
1992,
Ergonomics.
[8]
Irving Biederman,et al.
An Account of Object Identification Confusions
,
1997
.
[9]
Barbara L. O'Kane,et al.
Recognition of human activities using handheld thermal systems
,
1998,
Defense, Security, and Sensing.