Guidelines for Development and Evaluation of Usage Data Analytics Tools for Human-Machine Interactions with Industrial Manufacturing Systems

We present the lessons learned during the development and evaluation process for UX-sensors, a visual data analytics tool for inspecting logged usage data from flexible manufacturing systems (FMS). Based on the experiences during a collaborative development process with practitioners from one FMS supplier company, we propose guidelines to support other developers of visual data analytics tools for usage data logging in context of complex industrial systems. For instance, involving stakeholders with different roles can help to identify user requirements and generate valuable development ideas. Tool developers should confirm early access to real usage data from customers' systems and familiarize themselves with the log data structure. We argue that combining expert evaluations with field study methods can provide a more diverse set of usability issues to address. For future research, we encourage studies on insights emerging from usage data analytics and their impact on the viewpoints of the supplier and customer.

[1]  Jakob Nielsen,et al.  Chapter 4 – The Usability Engineering Lifecycle , 1993 .

[2]  Tobias Isenberg,et al.  A Systematic Review on the Practice of Evaluating Visualization , 2013, IEEE Transactions on Visualization and Computer Graphics.

[3]  Tamara Munzner,et al.  On Regulatory and Organizational Constraints in Visualization Design and Evaluation , 2016, BELIV '16.

[4]  Hoda A. ElMaraghy,et al.  Flexible and reconfigurable manufacturing systems paradigms , 2005 .

[5]  Ben Shneiderman,et al.  Strategies for evaluating information visualization tools: multi-dimensional in-depth long-term case studies , 2006, BELIV '06.

[6]  Melanie Tory,et al.  Evaluating Visualizations: Do Expert Reviews Work? , 2005, IEEE Computer Graphics and Applications.

[7]  M. Patton.  Qualitative research and evaluation methods , 1980 .

[8]  Sherry Koshman,et al.  Information Visualization: Human-Centered Issues and Perspectives , 2009, J. Assoc. Inf. Sci. Technol..

[9]  Ben Shneiderman,et al.  Creativity Support Tools: Report From a U.S. National Science Foundation Sponsored Workshop , 2006, Int. J. Hum. Comput. Interact..

[10]  Clemens Holzmann,et al.  Interaction visualization and analysis in automation industry , 2015, MUM.

[11]  Oliver Sträter.  Human and Automation: System Design and Research Issues: Thomas B. Sheridan (Ed.); Wiley, New York, 2002 , 2003, Reliab. Eng. Syst. Saf..

[12]  Andreas Butz,et al.  Information visualization evaluation in large companies: Challenges, experiences and recommendations , 2011, Inf. Vis..

[13]  Ben Shneiderman,et al.  LifeFlow: visualizing an overview of event sequences , 2011, CHI.

[14]  Thomas B. Sheridan,et al.  Humans and Automation: System Design and Research Issues , 2002 .

[15]  Chris North,et al.  Information Visualization , 2008, Lecture Notes in Computer Science.

[16]  Tamara Munzner,et al.  A Nested Model for Visualization Design and Validation , 2009, IEEE Transactions on Visualization and Computer Graphics.

[17]  M. Sheelagh T. Carpendale,et al.  Empirical Studies in Information Visualization: Seven Scenarios , 2012, IEEE Transactions on Visualization and Computer Graphics.

[18]  James R. Lewis,et al.  IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use , 1995, Int. J. Hum. Comput. Interact..

[19]  Ben Shneiderman,et al.  Integrating statistics and visualization: case studies of gaining clarity during exploratory data analysis , 2008, CHI.

[20]  Jakob Nielsen,et al.  Heuristic Evaluation of Prototypes (individual) , 2022 .

[21]  David Gotz,et al.  DecisionFlow: Visual Analytics for High-Dimensional Temporal Event Sequence Data , 2014, IEEE Transactions on Visualization and Computer Graphics.

[22]  Michael John,et al.  Data cracker: developing a visual game analytic tool for analyzing online gameplay , 2011, CHI.

[23]  David Gotz,et al.  Progressive Visual Analytics: User-Driven Visual Exploration of In-Progress Analytics , 2014, IEEE Transactions on Visualization and Computer Graphics.

[24]  Jeffrey Heer,et al.  Interactive analysis of big data , 2012, XRDS.

[25]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[26]  Camilla Forsell,et al.  An heuristic set for evaluation in information visualization , 2010, AVI.

[27]  Tamara Munzner,et al.  Design Study Methodology: Reflections from the Trenches and the Stacks , 2012, IEEE Transactions on Visualization and Computer Graphics.

[28]  Chris North,et al.  An Insight-Based Longitudinal Study of Visual Analytics , 2006, IEEE Transactions on Visualization and Computer Graphics.

[29]  J. Manyika.  Big data: The next frontier for innovation, competition, and productivity , 2011 .

[30]  Celine Latulipe,et al.  Creativity factor evaluation: towards a standardized survey metric for creativity support , 2009, C&C '09.

[31]  Clemens Holzmann,et al.  Logging and visualization of touch interactions on teach pendants , 2014, MobileHCI '14.

[32]  M. Sheelagh T. Carpendale,et al.  Evaluating Information Visualizations , 2008, Information Visualization.

[33]  Catherine Plaisant,et al.  The challenge of information visualization evaluation , 2004, AVI.