Extracting and analysing design process data from log files of ICT supported co-creative sessions

The observation of designers' behaviour in collaborative design activities and the analysis of protocols improved the understanding of how novel ideas emerge, what occurs among designers and, indirectly, what methods have a good impact on the outcomes. Yet, protocol analysis requires recording the design sessions, often in a simulated environment, thus introducing a bias in the observation. Moreover, the analysis takes up to 1000 times the duration of the observed design session. These limitations definitely hinder the scalability of this practice to large experiments in real operational environments. This paper investigates the possibility to use the data collected in log files, automatically recorded during collaborative design sessions assisted by an ICT design support tool, as a means to extract relevant information about the design process and ultimately to infer insights about co-designers' cognition during the session. In this perspective, the paper proposes a set of metrics tailored to an Augmented Reality-based collaborative design tool. The study has been carried about by processing the data collected in 5 real case studies conducted in three different design companies.

[1]  Jeff W T Kan,et al.  Using the FBS Ontology to Capture Semantic Design Information in Design Protocol Studies , 2007 .

[2]  Henrik Bellhäuser,et al.  Identifying individual differences using log-file analysis: Distributed learning as mediator between conscientiousness and exam grades , 2018, Learning and Individual Differences.

[3]  James E. Pitkow,et al.  Characterizing Browsing Behaviors on the World-Wide Web , 1995 .

[4]  Samuel Greiff,et al.  Computer-generated log-file analyses as a window into students' minds? A showcase study based on the PISA 2012 assessment of problem solving , 2015, Comput. Educ..

[5]  Peter Filzmoser,et al.  Dynamic log file analysis: An unsupervised cluster evolution approach for anomaly detection , 2018, Comput. Secur..

[6]  Christophe Kolski,et al.  Study on generic tangible objects used to collaborate remotely on RFID tabletops , 2018, Journal on Multimodal User Interfaces.

[7]  Deborah A. Fields,et al.  Connecting play: understanding multimodal participation in virtual worlds , 2012, ICMI '12.

[8]  Dinesh N. Patil,et al.  Web Browser Analysis for Detecting User Activities , 2019 .

[9]  Andreas Breiter,et al.  Analyzing mobile application usage: generating log files from mobile screen recordings , 2018, MobileHCI.

[10]  Cédric Masclet,et al.  CODING SCHEMES FOR THE ANALYSIS OF ICT SUPPORTED CO-CREATIVE DESIGN SESSIONS , 2018 .

[11]  Andreas Breiter,et al.  Monitoring User Patterns in School Information Systems Using Logfile Analysis , 2012, ITEM.

[12]  James E. Pitkow,et al.  Characterizing Browsing Strategies in the World-Wide Web , 1995, Comput. Networks ISDN Syst..

[13]  Margherita Peruzzini,et al.  How to analyse the workers' experience in integrated product-process design , 2018, J. Ind. Inf. Integr..

[14]  M. W. van Someren,et al.  The think aloud method: a practical approach to modelling cognitive processes , 1994 .

[15]  Federico Rotini,et al.  An OTSM-TRIZ Based Framework Towards the Computer-Aided Identification of Cognitive Processes in Design Protocols , 2015 .

[16]  Ghang Lee,et al.  Avoiding design errors: a case study of redesigning an architectural studio , 2003 .

[17]  Yvonne Kammerer,et al.  How children navigate a multiperspective hypermedia environment: The role of spatial working memory capacity , 2016, Comput. Hum. Behav..

[18]  Maria Bannert,et al.  Short- and long-term effects of students' self-directed metacognitive prompts on navigation behavior and learning performance , 2015, Comput. Hum. Behav..

[19]  Gaetano Cascini,et al.  Sources of creativity stimulation for designing the next generation of technical systems: correlations with R&D designers’ performance , 2018, Research in Engineering Design.

[20]  Hongming Zhu,et al.  Extending Mobile App Analytics for Usability Test Logging , 2017, INTERACT.

[21]  Ben J Hicks,et al.  The characterisation of engineering activity through email communication and content dynamics, for support of engineering project management , 2017, Design Science.

[22]  Lucienne Blessing,et al.  DRM, a Design Research Methodology , 2009 .

[23]  Hsinchun Chen,et al.  Browsing in hypertext: a cognitive study , 1992, IEEE Trans. Syst. Man Cybern..

[24]  Rahinah Ibrahim,et al.  Impacts of VR 3D sketching on novice designers’ spatial cognition in collaborative conceptual architectural design , 2011 .

[25]  Young-Jin Lee Using Self-Organizing Map and Clustering to Investigate Problem-Solving Patterns in the Massive Open Online Course: An Exploratory Study , 2019 .

[26]  Federico Rotini,et al.  EXPERIENCING PROTOCOL ANALYSIS OF COMPUTER-GUIDED DESIGN TASKS , 2012 .