Evaluation of an architecture for intelligent query and exploration of time-oriented clinical data

OBJECTIVE Evaluate KNAVE-II, a knowledge-based framework for visualization, interpretation, and exploration of longitudinal clinical data, clinical concepts and patterns. KNAVE-II mediates queries to a distributed temporal-abstraction architecture (IDAN), which uses a knowledge-based problem-solving method specializing in on-the-fly computation of clinical queries. METHODS A two-phase, balanced cross-over study to compare efficiency and satisfaction of a group of clinicians when answering queries of variable complexity about time-oriented clinical data, typical for oncology protocols, using KNAVE-II, versus standard methods: both paper charts and a popular electronic spreadsheet (ESS) in Phase I; an ESS in Phase II. The measurements included the time required to answer and the correctness of answer for each query and each complexity category, and for all queries, assessed versus a predetermined gold standard set by a domain expert. User satisfaction was assessed by the Standard Usability Score (SUS) tool-specific questionnaire and by a "Usability of Tool Comparison" comparative questionnaire developed for this study. RESULTS In both evaluations, subjects answered higher-complexity queries significantly faster using KNAVE-II than when using paper charts or an ESS up to a mean of 255 s difference per query versus the ESS for hard queries (p=0.0003) in the second evaluation. Average correctness scores when using KNAVE-II versus paper charts, in the first phase, and the ESS, in the second phase, were significantly higher over all queries. In the second evaluation, 91.6% (110/120) of all of the questions asked within queries of all levels produced correct answers using KNAVE-II, opposed to only 57.5% (69/120) using the ESS (p<0.0001). User satisfaction with KNAVE-II was significantly superior compared to using either a paper chart or the ESS (p=0.006). Clinicians ranked KNAVE-II superior to both paper and the ESS. CONCLUSIONS An evaluation of the functionality and usability of KNAVE-II and its supporting knowledge-based temporal-mediation architecture has produced highly encouraging results regarding saving of physician time, enhancement of accuracy of clinical assessment, and user satisfaction.

[1]  N. Lavrac,et al.  Intelligent Data Analysis in Medicine and Pharmacology , 1997 .

[2]  Yuval Shahar,et al.  A framework for a distributed, hybrid, multiple-ontology clinical-guideline library, and automated guideline-support tools , 2004, J. Biomed. Informatics.

[3]  Tiziana Catarci,et al.  Formalizing visual interaction with historical databases , 2002, Inf. Syst..

[4]  Catherine Plaisant,et al.  The challenge of information visualization evaluation , 2004, AVI.

[5]  Yuval Shahar,et al.  Applying Temporal Abstraction in Medical Information Systems , 2003 .

[6]  Yuval Shahar,et al.  Probabilistic Abstraction of Multiple Longitudinal Electronic Medical Records , 2005, AIME.

[7]  Yuval Shahar,et al.  Intelligent Querying and Exploration of Multiple Time-Oriented Medical Records , 2007, MedInfo.

[8]  Tiziana Catarci,et al.  An Ontology Based Visual Tool for Query Formulation Support , 2004, OTM Workshops.

[9]  Yuval Shahar,et al.  Model-based visualization of temporal abstractions , 1998, Proceedings. Fifth International Workshop on Temporal Representation and Reasoning (Cat. No.98EX157).

[10]  Yuval Shahar,et al.  An active database architecture for knowledge-based incremental abstraction of complex concepts from continuously arriving time-oriented raw data , 2007, Journal of Intelligent Information Systems.

[11]  Mary Czerwinski,et al.  Empirical evaluation of information visualizations: an introduction , 2000, Int. J. Hum. Comput. Stud..

[12]  Jonathan C. Roberts,et al.  The Craft of Information Visualization , 2008 .

[13]  Luca Chittaro,et al.  Visualizing queries on databases of temporal histories: new metaphors and their evaluation , 2003, Data Knowl. Eng..

[14]  Yuval Shahar,et al.  Semiautomated Acquisition of Clinical Temporal-abstraction Knowledge , 1998 .

[15]  Luca Chittaro,et al.  Abstraction on clinical data sequences: an object-oriented data model and a query language based on the event calculus , 1999, Artif. Intell. Medicine.

[16]  B Jones,et al.  Modelling and design of cross-over trials. , 1996, Statistics in medicine.

[17]  Luca Chittaro,et al.  Information visualization and its application to medicine , 2001, Artif. Intell. Medicine.

[18]  Silvia Miksch,et al.  Abstraction and Representation of Repeated Patterns in High-Frequency Data , 2000 .

[19]  Silvia Miksch,et al.  Utilizing temporal data abstraction for data validation and therapy planning for artificially ventilated newborn infants , 1996, Artif. Intell. Medicine.

[20]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[21]  Yuval Shahar,et al.  A framework for distributed mediation of temporal-abstraction queries to clinical databases , 2005, Artif. Intell. Medicine.

[22]  Yuval Shahar,et al.  DEGEL: A Hybrid, Multiple-Ontology Framework for Specification and Retrieval of Clinical Guidelines , 2003, AIME.

[23]  Herbert A. Simon,et al.  Why a Diagram is (Sometimes) Worth Ten Thousand Words , 1987, Cogn. Sci..

[24]  Yuval Shahar,et al.  Evaluation of KNAVE-II: a Tool for Intelligent Query and Exploration of Patient Data , 2004, MedInfo.

[25]  Edward R. Tufte,et al.  Envisioning Information , 1990 .

[26]  B. Jones,et al.  A review of uniform cross-over designs , 2008 .

[27]  Yuval Shahar,et al.  Knowledge-based temporal abstraction in clinical domains , 1996, Artif. Intell. Medicine.

[28]  Yuval Shahar,et al.  A Framework for Knowledge-Based Temporal Abstraction , 1997, Artif. Intell..

[29]  Yuval Shahar,et al.  Incremental application of knowledge to continuously arriving time-oriented data , 2007, Journal of Intelligent Information Systems.

[30]  Edward Tufte,et al.  Visual Explanations , 1997 .

[31]  Yuval Shahar,et al.  Intelligent visualization and exploration of time-oriented clinical data , 1999, Proceedings of the 32nd Annual Hawaii International Conference on Systems Sciences. 1999. HICSS-32. Abstracts and CD-ROM of Full Papers.

[32]  Catherine Plaisant,et al.  SpaceTree: supporting exploration in large node link tree, design evolution and empirical evaluation , 2002, IEEE Symposium on Information Visualization, 2002. INFOVIS 2002..

[33]  Catherine Garbay,et al.  Multi‐level temporal abstraction for medical scenario construction , 2005 .

[34]  Ben Shneiderman,et al.  Dynamic Query Tools for Time Series Data Sets: Timebox Widgets for Interactive Exploration , 2004, Inf. Vis..

[35]  Ben Shneiderman,et al.  LifeLines: using visualization to enhance navigation and analysis of patient records , 1998, AMIA.

[36]  Yuval Shahar,et al.  CAPSUL: A constraint-based specification of repeating patterns in time-oriented data , 2001, Annals of Mathematics and Artificial Intelligence.

[37]  Michael Spenke,et al.  Visualization and interactive analysis of blood parameters with InfoZoom , 2001, Artif. Intell. Medicine.

[38]  Yuval Shahar,et al.  Original Investigation: Semi-automated Entry of Clinical Temporal-abstraction Knowledge , 1999, J. Am. Medical Informatics Assoc..

[39]  Yuval Shahar,et al.  Distributed, intelligent, interactive visualization and exploration of time-oriented clinical data and their abstractions , 2006, Artif. Intell. Medicine.

[40]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[41]  B. Thomas,et al.  Usability Evaluation In Industry , 1996 .

[42]  Yuval Shahar,et al.  A Framework for Intelligent Visualization of Multiple Time-Oriented Medical Records , 2005, AMIA.

[43]  Luca Chittaro,et al.  Data mining on temporal data: a visual approach and its clinical application to hemodialysis , 2003, J. Vis. Lang. Comput..

[44]  Göran Falkman.  Information visualisation in clinical Odontology: multidimensional analysis and interactive data exploration , 2001, Artif. Intell. Medicine.

[45]  Yuval Shahar,et al.  Runtime application of Hybrid-Asbru clinical guidelines , 2007, J. Biomed. Informatics.