m-TSNE: A Framework for Visualizing High-Dimensional Multivariate Time Series

Multivariate time series (MTS) have become increasingly common in healthcare domains where human vital signs and laboratory results are collected for predictive diagnosis. Recently, there have been increasing efforts to visualize healthcare MTS data based on star charts or parallel coordinates. However, such techniques might not be ideal for visualizing a large MTS dataset, since it is difficult to obtain insights or interpretations due to the inherent high dimensionality of MTS. In this paper, we propose 'm-TSNE': a simple and novel framework to visualize high-dimensional MTS data by projecting them into a low-dimensional (2-D or 3-D) space while capturing the underlying data properties. Our framework is easy to use and provides interpretable insights for healthcare professionals to understand MTS data. We evaluate our visualization framework on two real-world datasets and demonstrate that the results of our m-TSNE show patterns that are easy to understand while the other methods' visualization may have limitations in interpretability.

[1]  Edward J. Coyle,et al.  An energy efficient hierarchical clustering algorithm for wireless sensor networks , 2003, IEEE INFOCOM 2003. Twenty-second Annual Joint Conference of the IEEE Computer and Communications Societies (IEEE Cat. No.03CH37428).

[2]  Andrew Hayen,et al.  The Influence of Graphic Display Format on the Interpretations of Quantitative Risk Information among Adults with Lower Education and Literacy , 2012, Medical decision making : an international journal of the Society for Medical Decision Making.

[3]  Maaike van den Haak,et al.  Evaluating consumer health information websites: The importance of collecting observational, user-driven data , 2010, 2010 IEEE International Professional Comunication Conference.

[4]  P. Boutros,et al.  Current and Evolving Methods to Visualize Biological Data in Cancer Research. , 2016, Journal of the National Cancer Institute.

[5]  Georgy Kopanitsa,et al.  Development, implementation and evaluation of an information model for archetype based user responsive medical data visualization , 2015, J. Biomed. Informatics.

[6]  B. J. Fogg,et al.  A behavior model for persuasive design , 2009, Persuasive '09.

[7]  Emden R. Gansner,et al.  Graphviz - Open Source Graph Drawing Tools , 2001, GD.

[8]  P. Blackman Actual causes of death in the United States. , 1994, JAMA.

[9]  Julie O. Culver,et al.  Health literacy, numeracy, and interpretation of graphical breast cancer risk estimates. , 2011, Patient education and counseling.

[10]  Joan Sargeant,et al.  How do surgeons decide to refer patients for adjuvant cancer treatment? Protocol for a qualitative study , 2012, Implementation Science.

[11]  Tejal K. Gandhi,et al.  Design and implementation of an application and associated services to support interdisciplinary medication reconciliation efforts at an integrated healthcare delivery network. , 2006, Journal of the American Medical Informatics Association : JAMIA.

[12]  Sudhansu Chokroverty,et al.  Is There a Clinical Role For Smartphone Sleep Apps? Comparison of Sleep Cycle Detection by a Smartphone Application to Polysomnography. , 2015, Journal of clinical sleep medicine : JCSM : official publication of the American Academy of Sleep Medicine.

[13]  Martin G. Everett,et al.  A Graph-theoretic perspective on centrality , 2006, Soc. Networks.

[14]  Joon Lee,et al.  Personalized Mortality Prediction Driven by Electronic Medical Data and a Patient Similarity Metric , 2015, PloS one.

[15]  Edward Rolf Tufte,et al.  The visual display of quantitative information , 1985 .

[16]  Ben Shneiderman,et al.  LifeFlow: visualizing an overview of event sequences , 2011, CHI.

[17]  Minsuk Kahng,et al.  TimeStitch: Interactive multi-focus cohort discovery and comparison , 2015, 2015 IEEE Conference on Visual Analytics Science and Technology (VAST).

[18]  A. Asuncion,et al.  UCI Machine Learning Repository, University of California, Irvine, School of Information and Computer Sciences , 2007 .

[19]  Catherine Plaisant,et al.  The challenge of information visualization evaluation , 2004, AVI.

[20]  André Kushniruk,et al.  Defining a Framework for Health Information Technology Evaluation , 2011, ITCH.

[21]  Stefan Biffl,et al.  PlanningLines: novel glyphs for representing temporal uncertainties and their evaluation , 2005, Ninth International Conference on Information Visualisation (IV'05).

[22]  Joshua J. Neumiller,et al.  Nurse identified hospital to home medication discrepancies: implications for improving transitional care. , 2010, Geriatric nursing.

[23]  Valerio Pascucci,et al.  Robust on-line computation of Reeb graphs: simplicity and speed , 2007, ACM Trans. Graph..

[24]  Sheldon P Stone,et al.  Community-acquired pneumonia , 1998, The Lancet.

[25]  Jiaquan Xu,et al.  Deaths: final data for 2010. , 2013, National vital statistics reports : from the Centers for Disease Control and Prevention, National Center for Health Statistics, National Vital Statistics System.

[26]  Christi J. van Asperen,et al.  The effectiveness of a graphical presentation in addition to a frequency format in the context of familial breast cancer risk communication: a multicenter controlled trial , 2013, BMC Medical Informatics and Decision Making.

[27]  Jeffrey Heer,et al.  A tour through the visualization zoo , 2010, ACM Queue.

[28]  Marie desJardins,et al.  Visualizing Multivariate Time Series Data to Detect Specific Medical Conditions , 2008, AMIA.

[29]  A. Sheikh,et al.  Evaluating eHealth Interventions: The Need for Continuous Systemic Evaluation , 2009, PLoS medicine.

[30]  Matthew D. Cooper,et al.  ActiviTree: Interactive Visual Exploration of Sequences in Event-Based Data Using Graph Similarity , 2009, IEEE Transactions on Visualization and Computer Graphics.

[31]  Silvia Miksch,et al.  Challenges of Time-oriented Data in Visual Analytics for Healthcare , 2012 .

[32]  Ben Shneiderman,et al.  Temporal Event Sequence Simplification , 2013, IEEE Transactions on Visualization and Computer Graphics.

[33]  Wolfgang Jank,et al.  Representing Unevenly-Spaced Time Series Data for Visualization and Interactive Exploration , 2005, INTERACT.

[34]  E. Law,et al.  Internet-delivered cognitive-behavioral treatment for adolescents with chronic pain and their parents: a randomized controlled multicenter trial , 2016, Pain.

[35]  V. Garovic,et al.  Beyond Bar and Line Graphs: Time for a New Data Presentation Paradigm , 2015, PLoS biology.

[36]  Amar K. Das,et al.  Social network analysis of physician interactions: the effect of institutional boundaries on breast cancer care. , 2011, AMIA ... Annual Symposium proceedings. AMIA Symposium.

[37]  Tony Cornford,et al.  Building a house on shifting sand: methodological considerations when evaluating the implementation and adoption of national electronic health record systems , 2012, BMC Health Services Research.

[38]  Hans-Peter Kriegel,et al.  A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise , 1996, KDD.

[39]  John M. Chambers,et al.  Graphical Methods for Data Analysis , 1983 .

[40]  Tao Wang,et al.  Social Networkers' Attitudes Toward Direct-to-Consumer Personal Genome Testing , 2009, The American journal of bioethics : AJOB.

[41]  Marcus A. Badgeley,et al.  EHDViz: clinical dashboard development using open-source technologies , 2016, BMJ Open.

[42]  Fei Wang,et al.  A methodology for interactive mining and visual analysis of clinical event patterns using electronic health record data , 2014, J. Biomed. Informatics.

[43]  Herbert H. Clark,et al.  Navigating joint projects with dialogue , 2003, Cogn. Sci..

[44]  Geoffrey E. Hinton,et al.  Stochastic Neighbor Embedding , 2002, NIPS.

[45]  Yi Pan,et al.  Integration of breast cancer gene signatures based on graph centrality , 2011, BMC Systems Biology.

[46]  Yarden Livnat,et al.  Evaluation of an Electronic Module for Reconciling Medications in Home Health Plans of Care , 2016, Applied Clinical Informatics.

[47]  Chandrajit L. Bajaj,et al.  Time-varying contour topology , 2006, IEEE Transactions on Visualization and Computer Graphics.

[48]  Cyrus Shahabi,et al.  A PCA-based similarity measure for multivariate time series , 2004, MMDB '04.

[49]  Terri R Fried,et al.  Shared Decision Making--Finding the Sweet Spot. , 2016, The New England journal of medicine.

[50]  J. Gilsdorf,et al.  Community-acquired pneumonia in children. , 1987, Seminars in respiratory infections.

[51]  付伶俐 打磨Using Language,倡导新理念 , 2014 .

[52]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[53]  Lianne Jeffs,et al.  Defining quality outcomes for complex-care patients transitioning across the continuum using a structured panel process , 2013, BMJ quality & safety.

[54]  W. Krzanowski Between-Groups Comparison of Principal Components , 1979 .

[55]  Jimeng Sun,et al.  Predicting Patient's Trajectory of Physiological Data using Temporal Trends in Similar Patients: A System for Near-Term Prognostics. , 2010, AMIA ... Annual Symposium proceedings. AMIA Symposium.

[56]  Jeffrey Heer,et al.  A Tour through the Visualization Zoo , 2010 .

[57]  N. Schork,et al.  Effect of direct-to-consumer genomewide profiling to assess disease risk. , 2011, The New England journal of medicine.

[58]  N. Keating,et al.  Factors Affecting Influential Discussions Among Physicians: A Social Network Analysis of a Primary Care Practice , 2007, Journal of General Internal Medicine.

[59]  Ben Shneiderman,et al.  LifeLines: using visualization to enhance navigation and analysis of patient records , 1998, AMIA.

[60]  Debprakash Patnaik,et al.  Spatio-temporal Structure of US Critical Care Transfer Network , 2011, AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science.

[61]  Using medication reconciliation to prevent errors. , 2006, Sentinel event alert.

[62]  Michael Thompson,et al.  A visual analytics approach to understanding care process variation and conformance , 2015, VAHC '15.

[63]  D. Snider,et al.  Indicators for chronic disease surveillance. , 2004, MMWR. Recommendations and reports : Morbidity and mortality weekly report. Recommendations and reports.

[64]  HeerJeffrey,et al.  D3 Data-Driven Documents , 2011 .

[65]  J. G. Snodgrass,et al.  A standardized set of 260 pictures: norms for name agreement, image agreement, familiarity, and visual complexity. , 1980, Journal of experimental psychology. Human learning and memory.

[66]  Danielle M Stitt,et al.  Medication discrepancies identified at time of hospital discharge in a geriatric population. , 2011, The American journal of geriatric pharmacotherapy.

[67]  Olga C Damman,et al.  Making comparative performance information more comprehensible: an experimental evaluation of the impact of formats on consumer understanding , 2015, BMJ Quality & Safety.

[68]  Fei Wang,et al.  Frequence: interactive mining and visualization of temporal frequent event sequences , 2014, IUI.

[69]  Fei Wang,et al.  Mining and exploring care pathways from electronic medical records with visual analytics , 2015, J. Biomed. Informatics.

[70]  Gil Alterovitz,et al.  SMART precision cancer medicine: a FHIR-based app to provide genomic information at the point of care , 2016, J. Am. Medical Informatics Assoc..

[71]  Céline McKeown,et al.  Designing for Situation Awareness: An Approach to User-Centered Design , 2013 .

[72]  Satoru Kawai,et al.  An Algorithm for Drawing General Undirected Graphs , 1989, Inf. Process. Lett..

[73]  Hans J. W. Spoelder,et al.  Visualization of time-dependent data with feature tracking and event detection , 2001, The Visual Computer.

[74]  Carlos S. Frenk,et al.  Galaxy formation through hierarchical clustering , 1991 .

[75]  David Gotz,et al.  DecisionFlow: Visual Analytics for High-Dimensional Temporal Event Sequence Data , 2014, IEEE Transactions on Visualization and Computer Graphics.

[76]  E. Rackow Rehospitalizations among patients in the Medicare fee-for-service program. , 2009, The New England journal of medicine.

[77]  Takenobu Tokunaga,et al.  Hierarchical Bayesian Clustering for Automatic Text Classification , 1995, IJCAI.

[78]  Mingming Jia,et al.  COSMIC (the Catalogue of Somatic Mutations in Cancer): a resource to investigate acquired mutations in human cancer , 2009, Nucleic Acids Res..

[79]  I. Dubchak,et al.  Visualizing genomes: techniques and challenges , 2010, Nature Methods.

[80]  James D. Hollan,et al.  Distributed cognition: toward a new foundation for human-computer interaction research , 2000, TCHI.

[81]  Stephanie K. Mueller,et al.  Hospital-based medication reconciliation practices: a systematic review. , 2012, Archives of internal medicine.

[82]  Penny Rheingans,et al.  Illustration-inspired techniques for visualizing time-varying data , 2005, VIS 05. IEEE Visualization, 2005..

[83]  Adam Perer Visualizations to Support Patient-Clinician Communication of Care Plans , 2013 .

[84]  Pascal Su,et al.  Direct-to-Consumer Genetic Testing: A Comprehensive View , 2013, The Yale journal of biology and medicine.

[85]  Julia Muennich Cowell,et al.  Population Health , 2016, The Journal of school nursing : the official publication of the National Association of School Nurses.

[86]  Misha Pavel,et al.  Advancing the Science of mHealth , 2012, Journal of health communication.

[87]  Meinard Müller,et al.  Information retrieval for music and motion , 2007 .

[88]  Rebecca Cohen A brief introduction to the new Healthy People 2020 initiative. , 2013, Beginnings.

[89]  David Gotz,et al.  Data-driven exploration of care plans for patients , 2013, CHI Extended Abstracts.

[90]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[91]  Ben Shneiderman,et al.  Cohort Comparison of Event Sequences with Balanced Integration of Visual Analytics and Statistics , 2015, IUI.

[92]  Jean Scholtz,et al.  Visual-Analytics Evaluation , 2009, IEEE Computer Graphics and Applications.

[93]  J. Bishop,et al.  Things and Places : How the Mind Connects with the World , 2010 .

[94]  Ben Shneiderman,et al.  Aligning temporal data by sentinel events: discovering patterns in electronic health records , 2008, CHI.

[95]  Jesus J. Caban,et al.  Visual analytics in healthcare - opportunities and research challenges , 2015, J. Am. Medical Informatics Assoc..

[96]  Silvia Miksch,et al.  Connecting time-oriented data and information to a coherent interactive visualization , 2004, CHI.

[97]  E. Jensen,et al.  The Performance of Fertility Awareness-based Method Apps Marketed to Avoid Pregnancy , 2016, The Journal of the American Board of Family Medicine.

[98]  D. Yach,et al.  The global burden of chronic diseases: overcoming impediments to prevention and control. , 2004, JAMA.

[99]  John B. Bell,et al.  Analyzing and Tracking Burning Structures in Lean Premixed Hydrogen Flames , 2010, IEEE Transactions on Visualization and Computer Graphics.

[100]  David S. Pieczkiewicz,et al.  Design and Evaluation of a Web-Based Interactive Visualization System for Lung Transplant Home Monitoring Data , 2007, AMIA.

[101]  N. Gale,et al.  Using the framework method for the analysis of qualitative data in multi-disciplinary health research , 2013, BMC Medical Research Methodology.

[102]  M. Mugavero,et al.  A Qualitative Study on Health Numeracy and Patient–Provider Communication of Laboratory Numbers in Older African Americans with HIV , 2016, The Journal of the Association of Nurses in AIDS Care : JANAC.

[103]  Jimeng Sun,et al.  Visual cluster analysis in support of clinical decision intelligence. , 2011, AMIA ... Annual Symposium proceedings. AMIA Symposium.

[104]  Vivian West,et al.  Innovative information visualization of electronic health record data: a systematic review , 2014, J. Am. Medical Informatics Assoc..

[105]  Heidrun Schumann,et al.  Visualization of Time-Oriented Data , 2011, Human-Computer Interaction Series.

[106]  George Karypis,et al.  Evaluation of hierarchical clustering algorithms for document datasets , 2002, CIKM '02.

[107]  Nadir Weibel,et al.  ChronoViz: a system for supporting navigation of time-coded data , 2011, CHI Extended Abstracts.

[108]  Ben Shneiderman,et al.  Querying event sequences by exact match or similarity search: Design and empirical evaluation , 2012, Interact. Comput..

[109]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[110]  Seon-Hee Yim,et al.  Reflections on the US FDA's Warning on Direct-to-Consumer Genetic Testing , 2014, Genomics & informatics.

[111]  Julie O. Culver,et al.  Genetic Cancer Risk Assessment and Counseling: Recommendations of the National Society of Genetic Counselors , 2004, Journal of Genetic Counseling.

[112]  David S. Pieczkiewicz,et al.  Evaluating the decision accuracy and speed of clinical data visualizations , 2010, J. Am. Medical Informatics Assoc..

[113]  H. Edelsbrunner,et al.  Efficient algorithms for agglomerative hierarchical clustering methods , 1984 .

[114]  Beverly L. Harrison,et al.  Timelines: An Interactive System for the Collection and Visualization of Temporal Data , 1994 .

[115]  Jarke J. van Wijk,et al.  Cluster and Calendar Based Visualization of Time Series Data , 1999, INFOVIS.

[116]  Silvia Miksch,et al.  CareVis: Integrated visualization of computerized protocols and temporal patient data , 2006, Artif. Intell. Medicine.

[117]  Ben Shneiderman,et al.  Finding comparable temporal categorical records: A similarity measure with an interactive visualization , 2009, 2009 IEEE Symposium on Visual Analytics Science and Technology.

[118]  Ben Shneiderman,et al.  Temporal Summaries: Supporting Temporal Categorical Searching, Aggregation and Comparison , 2009, IEEE Transactions on Visualization and Computer Graphics.

[119]  Viju Raghupathi,et al.  Big data analytics in healthcare: promise and potential , 2014, Health Information Science and Systems.

[120]  H. H. Clark Coordinating with each other in a material world , 2005 .

[121]  Franziska Matzer,et al.  Biopsychosocial Health Care Needs at the Emergency Room: Challenge of Complexity , 2012, PloS one.

[122]  Niklas Elmqvist,et al.  Graphical Perception of Multiple Time Series , 2010, IEEE Transactions on Visualization and Computer Graphics.

[123]  M S Waterman,et al.  Identification of common molecular subsequences. , 1981, Journal of molecular biology.

[124]  Krist Wongsuphasawat,et al.  Outflow : Visualizing Patient Flow by Symptoms and Outcome , 2011 .

[125]  John T. Stasko,et al.  Distributed Cognition as a Theoretical Framework for Information Visualization , 2008, IEEE Transactions on Visualization and Computer Graphics.

[126]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[127]  Nan Cao,et al.  GapFlow : Visualizing Gaps in Care for Medical Treatment Plans , 2013 .

[128]  K. Lohr,et al.  The efficacy of behavioral interventions to modify dietary fat and fruit and vegetable intake: a review of the evidence. , 2002, Preventive medicine.

[129]  Christopher De Sa,et al.  Incremental Knowledge Base Construction Using DeepDive , 2015, The VLDB Journal.

[130]  C. Theobald,et al.  Reducing hospital readmission rates: current strategies and future directions. , 2014, Annual review of medicine.

[131]  Wei Chen,et al.  RankExplorer: Visualization of Ranking Changes in Large Time Series Data , 2012, IEEE Transactions on Visualization and Computer Graphics.

[132]  Vikas Kumar,et al.  Understanding variations in pediatric asthma care processes in the emergency department using visual analytics , 2015, J. Am. Medical Informatics Assoc..

[133]  Susan N. Friel,et al.  Making sense of graphs: Critical factors influencing comprehension and instructional implications. , 2001 .

[134]  Marylyn D. Ritchie,et al.  Synthesis-View: visualization and interpretation of SNP association results for multi-cohort, multi-phenotype data and meta-analysis , 2010, BioData Mining.

[135]  Daniëlle R. M. Timmermans,et al.  Communicating cancer treatment information using the Web: utilizing the patient’s perspective in website development , 2014, BMC Medical Informatics and Decision Making.

[136]  A. Sheikh,et al.  Undertaking sociotechnical evaluations of health information technologies. , 2014, Informatics in primary care.

[137]  Jessica S. Ancker,et al.  The Triangle Model for evaluating the effect of health information technology on healthcare quality and safety , 2012, J. Am. Medical Informatics Assoc..

[138]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[139]  J. Frankovich,et al.  Evidence-based medicine in the EMR era. , 2011, The New England journal of medicine.

[140]  Y. Lokhnygina,et al.  Inpatient medication reconciliation at admission and discharge: A retrospective cohort study of age and other risk factors for medication discrepancies. , 2010, The American journal of geriatric pharmacotherapy.

[141]  Susan R Snyder,et al.  A Real-World Approach to a Value-Based Decision-Making Framework for Genetic Testing , 2015 .

[142]  Gail Javitt,et al.  The current landscape for direct-to-consumer genetic testing: legal, ethical, and policy issues. , 2008, Annual review of genomics and human genetics.

[143]  M. V. D. Panne,et al.  Topological manipulation of isosurfaces , 2004 .

[144]  Kenneth D. Mandl,et al.  SMART on FHIR: a standards-based, interoperable apps platform for electronic health records , 2016, J. Am. Medical Informatics Assoc..

[145]  Dong Hyun Jeong,et al.  Using Personality Factors to Predict Interface Learning Performance , 2010, 2010 43rd Hawaii International Conference on System Sciences.

[146]  Rebecca Randell,et al.  Using realist reviews to understand how health IT works, for whom, and in what circumstances , 2015, J. Am. Medical Informatics Assoc..

[147]  Karen Holtzblatt,et al.  Rapid Contextual Design: A How-To Guide to Key Techniques for User-Centered Design , 2004, UBIQ.

[148]  Vladimir I. Levenshtein,et al.  Binary codes capable of correcting deletions, insertions, and reversals , 1965 .

[149]  Niklas Elmqvist,et al.  Patterns for visualization evaluation , 2012, BELIV '12.

[150]  E. McFadden,et al.  Toxicity and response criteria of the Eastern Cooperative Oncology Group , 1982, American journal of clinical oncology.

[151]  David Borland,et al.  Visualization of EHR and Health Related Data for Information Discovery , 2017 .

[152]  C Popow,et al.  Support for Fast Comprehension of ICU Data: Visualization using Metaphor Graphics , 2001, Methods of Information in Medicine.

[153]  Jurriaan P Oudhoff,et al.  The Effect of Different Graphical and Numerical Likelihood Formats on Perception of Likelihood and Choice , 2015, Medical decision making : an international journal of the Society for Medical Decision Making.

[154]  Valerio Pascucci,et al.  Interactive exploration of large-scale time-varying data using dynamic tracking graphs , 2012, IEEE Symposium on Large Data Analysis and Visualization (LDAV).

[155]  David Tresner-Kirsch,et al.  High Prevalence of Medication Discrepancies Between Home Health Referrals and Centers for Medicare and Medicaid Services Home Health Certification and Plan of Care and Their Potential to Affect Safety of Vulnerable Elderly Adults , 2016, Journal of the American Geriatrics Society.

[156]  Alex M. Fichtenholtz,et al.  Development and validation of a clinical cancer genomic profiling test based on massively parallel DNA sequencing , 2013, Nature Biotechnology.

[157]  Brian D. Fisher,et al.  Pair Analytics: Capturing Reasoning Processes in Collaborative Visual Analytics , 2011, 2011 44th Hawaii International Conference on System Sciences.

[158]  Mark P. J. van der Loo,et al.  The stringdist Package for Approximate String Matching , 2014, R J..

[159]  Brian D. Fisher,et al.  Field experiment methodology for pair analytics , 2014, BELIV.

[160]  Ben Shneiderman,et al.  Towards event sequence representation, reasoning and visualization for EHR data , 2012, IHI '12.

[161]  Ellen Uiters,et al.  Functional, interactive and critical health literacy: Varying relationships with control over care and number of GP visits. , 2015, Patient education and counseling.

[162]  Samantha Fawkner,et al.  Why healthcare professionals should know a little about infographics , 2016, British Journal of Sports Medicine.

[163]  F. Aldrich,et al.  Graphicacy - the fourth 'R'? , 2000 .

[164]  Ian T. Jolliffe,et al.  Principal Component Analysis , 2002, International Encyclopedia of Statistical Science.

[165]  M. Saeed Multiparameter Intelligent Monitoring in Intensive Care II ( MIMIC-II ) : A public-access intensive care unit database , 2011 .

[166]  Alexander Baldwin,et al.  The Quality and Accuracy of Mobile Apps to Prevent Driving After Drinking Alcohol , 2016, JMIR mHealth and uHealth.

[167]  Samantha Thomas,et al.  The highest utilizers of care: individualized care plans to coordinate care, improve healthcare service utilization, and reduce costs at an academic tertiary care center. , 2015, Journal of hospital medicine.

[168]  Brian D. Fisher,et al.  Visual Analytics as a Translational Cognitive Science , 2011, Top. Cogn. Sci..

[169]  Dorin Comaniciu,et al.  Mean Shift: A Robust Approach Toward Feature Space Analysis , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[170]  Deborah Silver,et al.  Visualizing features and tracking their evolution , 1994, Computer.

[171]  Bum Chul Kwon,et al.  Peekquence : Visual Analytics for Event Sequence Data , 2016 .

[172]  Meinard Müller,et al.  Dynamic Time Warping , 2008 .

[173]  Joanne Greenhalgh,et al.  What’s in a mechanism? Development of a key concept in realist evaluation , 2015, Implementation Science.

[174]  Kristin A. Cook,et al.  Illuminating the Path: The Research and Development Agenda for Visual Analytics , 2005 .

[175]  Ben Shneiderman,et al.  Strategies for evaluating information visualization tools: multi-dimensional in-depth long-term case studies , 2006, BELIV '06.

[176]  John Wilbanks,et al.  First, design for data sharing , 2016, Nature Biotechnology.

[177]  James T Enns,et al.  Multiple-object tracking is based on scene, not retinal, coordinates. , 2005, Journal of experimental psychology. Human perception and performance.

[178]  E J Topol,et al.  Influence of individual differences in disease perception on consumer response to direct‐to‐consumer genomic testing , 2015, Clinical genetics.

[179]  Wei-Ying Ma,et al.  Hierarchical clustering of WWW image search results using visual, textual and link information , 2004, MULTIMEDIA '04.

[180]  P. Ubel,et al.  The impact of the format of graphical presentation on health-related knowledge and treatment choices. , 2008, Patient education and counseling.

[181]  Kensaku Kawamoto,et al.  Evaluating Implementation Fidelity in Health Information Technology Interventions , 2007, AMIA.

[182]  Lior Rokach,et al.  Data Mining And Knowledge Discovery Handbook , 2005 .

[183]  I. Lipkus Numeric, Verbal, and Visual Formats of Conveying Health Risks: Suggested Best Practices and Future Recommendations , 2007, Medical decision making : an international journal of the Society for Medical Decision Making.

[184]  Emma Saxon Beyond bar charts , 2015, BMC Biology.

[185]  Monika Richter,et al.  Cognition In The Wild , 2016 .

[186]  Gábor Csárdi,et al.  The igraph software package for complex network research , 2006 .

[187]  Edward M. Reingold,et al.  Graph drawing by force‐directed placement , 1991, Softw. Pract. Exp..

[188]  Cydney B Nielsen,et al.  Visualization: A Mind-Machine Interface for Discovery. , 2016, Trends in genetics : TIG.

[189]  Gerald M. Karam,et al.  Visualization using timelines , 1994, ISSTA '94.

[190]  J. Gore,et al.  Relevance of graph literacy in the development of patient-centered communication tools. , 2016, Patient education and counseling.

[191]  Anuradha Lakshminarayana,et al.  The clinical trial landscape in oncology and connectivity of somatic mutational profiles to targeted therapies , 2016, Human Genomics.

[192]  Alfred Inselberg,et al.  The plane with parallel coordinates , 1985, The Visual Computer.

[193]  Ben Shneiderman,et al.  Twinlist: Novel User Interface Designs for Medication Reconciliation , 2013, AMIA.

[194]  Diwakar Gupta,et al.  Appointment scheduling in health care: Challenges and opportunities , 2008 .

[195]  Michael Burch,et al.  Timeline trees: visualizing sequences of transactions in information hierarchies , 2008, AVI '08.

[196]  Yuval Shahar,et al.  AsbruView: Capturing Complex, Time-Oriented Plans - Beyond Flow Charts , 2002, Diagrammatic Representation and Reasoning.

[197]  P. Holme,et al.  Exploring the assortativity-clustering space of a network's degree sequence. , 2006, Physical review. E, Statistical, nonlinear, and soft matter physics.

[198]  Michael G. Kahn,et al.  The visual display of temporal information , 1991, Artif. Intell. Medicine.

[199]  Ben Shneiderman,et al.  A Visual Interface for Multivariate Temporal Data: Finding Patterns of Events across Multiple Histories , 2006, 2006 IEEE Symposium On Visual Analytics Science And Technology.

[200]  M. Petticrew,et al.  Developing and evaluating complex interventions: the new Medical Research Council guidance , 2008, BMJ : British Medical Journal.

[201]  Jing Li,et al.  Accelerate Healthcare Data Analytics: An Agile Practice to Perform Collaborative and Reproducible Analyses , 2016, MIE.

[202]  John B. Bell,et al.  Interactive Exploration and Analysis of Large-Scale Simulations Using Topology-Based Data Segmentation , 2011, IEEE Transactions on Visualization and Computer Graphics.

[203]  P. Varkey,et al.  Multidisciplinary approach to inpatient medication reconciliation in an academic setting. , 2007, American journal of health-system pharmacy : AJHP : official journal of the American Society of Health-System Pharmacists.

[204]  Alastair Baker,et al.  Crossing the Quality Chasm: A New Health System for the 21st Century , 2001, BMJ : British Medical Journal.

[205]  K. Shojania,et al.  Medication Reconciliation During Transitions of Care as a Patient Safety Strategy , 2013, Annals of Internal Medicine.

[206]  Melonie P. Heron Deaths: leading causes for 2010. , 2013, National vital statistics reports : from the Centers for Disease Control and Prevention, National Center for Health Statistics, National Vital Statistics System.

[207]  Ben Shneiderman,et al.  LifeLines: visualizing personal histories , 1996, CHI.

[208]  Jessica S. Ancker,et al.  The Practice of Informatics: Design Features of Graphs in Health Risk Communication: A Systematic Review , 2006, J. Am. Medical Informatics Assoc..

[209]  Kendra Schwartz,et al.  Patterns and correlates of patient referral to surgeons for treatment of breast cancer. , 2007, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.

[210]  Heather Skirton,et al.  Direct-to-consumer genomic testing: systematic review of the literature on user perspectives , 2012, European Journal of Human Genetics.

[211]  V. Mor,et al.  The revolving door of rehospitalization from skilled nursing facilities. , 2010, Health affairs.

[212]  J. Peter Kincaid,et al.  Electronic authoring and delivery of technical information , 1988 .

[213]  Elizabeth M. Borycki,et al.  Exploring the Contextual and Human Factors of Electronic Medication Reconciliation Research: A Scoping Review , 2013, CSHI.

[214]  Elske Ammenwerth,et al.  Evaluation of health information systems - problems and challenges , 2003, Int. J. Medical Informatics.

[215]  Juliana Chen,et al.  The Most Popular Smartphone Apps for Weight Loss: A Quality Assessment , 2015, JMIR mHealth and uHealth.

[216]  Marko Savic,et al.  Correction: From Static to Interactive: Transforming Data Visualization to Improve Transparency , 2016, PLoS biology.

[217]  Matthew D. Cooper,et al.  Everyday Life Discoveries: Mining and Visualizing Activity Patterns in Social Science Diary Data , 2007, 2007 11th International Conference Information Visualization (IV '07).

[218]  Ben Shneiderman,et al.  Novel user interface design for medication reconciliation: an evaluation of Twinlist , 2015, J. Am. Medical Informatics Assoc..

[219]  H. Begleiter,et al.  Event related potentials during object recognition tasks , 1995, Brain Research Bulletin.

[220]  Colleen M McBride,et al.  Consumers' views of direct-to-consumer genetic information. , 2010, Annual review of genomics and human genetics.

[221]  David J. Stone,et al.  "Big data" in the intensive care unit. Closing the data loop. , 2013, American journal of respiratory and critical care medicine.

[222]  Elske Ammenwerth,et al.  Can evaluation studies benefit from triangulation? A case study , 2003, Int. J. Medical Informatics.

[223]  Philip T. Kortum,et al.  Determining what individual SUS scores mean: adding an adjective rating scale , 2009 .

[224]  Tim Oates,et al.  Visualizing Variable-Length Time Series Motifs , 2012, SDM.

[225]  David Borland,et al.  Data-Driven Healthcare: Challenges and Opportunities for Interactive Visualization , 2016, IEEE Computer Graphics and Applications.

[226]  Ben Shneiderman,et al.  Interactive Information Visualization to Explore and Query Electronic Health Records , 2013, Found. Trends Hum. Comput. Interact..

[227]  F. Hawkins Human factors in aviation. , 1979, Journal of psychosomatic research.

[228]  M. Sheelagh T. Carpendale,et al.  Empirical Studies in Information Visualization: Seven Scenarios , 2012, IEEE Transactions on Visualization and Computer Graphics.

[229]  Lidewij Henneman,et al.  Presenting health risk information in different formats: the effect on participants' cognitive and emotional evaluation and decisions. , 2008, Patient education and counseling.