Metrics for Accessing Heterogeneous Data: Is There Any Hope? (Panel)

Government and industry are investing substantial resources in new technologies for accessing heterogeneous information sources, including text-based corpora, structured data, imagery, geo-spatial data, audio, video, and more. Managers of programs that fund relevant research face a difficult problem: they are required to justify investment in certain technologies and approaches versus alternate ones. These program managers recognize a need for good evaluation criteria, but there is little consensus on which criteria to use. Metrics are required to help evaluate the contribution of alternate techniques ‘to satisfying the ultimate goal: putting useful information into thehands of users. A number of metrics have bein proposed, including the following: l Relevance-based measures, including precision, the percentage of retrieved information nodes that are deemed relevant, and recall, the percentage of all relevant nodes in the entire information space that are actually retrieved. l Utility-based measures, which measure the “value-added” provided by information returned by a search. Variants include informativeness (Tague-Sutcliffe), which measures the extent to which an actual search trail corresponds to some ideal answer trail, and the overall value of the information retrieved. l Cost-based measures, including user interaction time and monetury cost. l User satisfaction