Comparison of paper- and tool-based participatory design approaches: a case study with PDotCapturer

In the current Participatory Design (PD) practice, paper-based approaches are commonly applied. However, the use of software tools can potentially enhance the PD process and outcome, given their flexibility for prototype presentation, data storage, retrieval and analysis. Driven by this assumption, we developed the specific-purpose PD online tool PDotCapturer. To evaluate the effectiveness of PDotCapturer, as compared with a paper-based approach, we conducted a within-subjects study where 33 participants were asked to provide feedback on e-learning prototypes, using either paper or PDotCapturer in one session and vice-versa in another session. The evaluation criteria were quantity and quality of the participants' comments. For qualitative data analysis, we applied the refined coding scheme CAt+. Results show that using PDotCapturer partly led to feedback of higher quality, but the number of comments was higher for the paper-based approach. Our work contributes to the HCI community by augmenting the repertoire of the PD toolbox.

[1]  Matthias Heintz,et al.  Paper or Pixel? Comparing Paper- and Tool-Based Participatory Design Approaches , 2015, INTERACT.

[2]  Denis Gillet,et al.  Innovations in STEM education: the Go-Lab federation of online labs , 2014, Smart Learning Environments.

[3]  Thomas G. Dietterich,et al.  Toward harnessing user feedback for machine learning , 2007, IUI '07.

[4]  Jeannette Kindred,et al.  "He Will Crush You Like an Academic Ninja!": Exploring Teacher Ratings on Ratemyprofessors.com , 2006, J. Comput. Mediat. Commun..

[5]  Simone Diniz Junqueira Barbosa,et al.  UISKEI: a sketch-based prototyping tool for defining and evaluating user interface behavior , 2012, AVI.

[6]  David McMenemy,et al.  A classification scheme for content analyses of YouTube video comments , 2013, J. Documentation.

[7]  Kevin A Hallgren,et al.  Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. , 2012, Tutorials in quantitative methods for psychology.

[8]  Brian P. Bailey,et al.  Are informal tools better?: comparing DEMAIS, pencil and paper, and authorware for early multimedia design , 2003, CHI '03.

[9]  Mark W. Newman,et al.  DENIM: An Informal Web Site Design Tool Inspired by Observations of Practice , 2003, Hum. Comput. Interact..

[10]  Adrian Holzer,et al.  Pdot: participatory design online tool , 2014, CHI Extended Abstracts.

[11]  Saskia Brand-Gruwel,et al.  An approach to participatory instructional design in secondary education: an exploratory study , 2010 .

[12]  Orit Shaer,et al.  Reality-based interaction: a framework for post-WIMP interfaces , 2008, CHI.

[13]  Katherine Crewe,et al.  The Quality of Participatory Design: The Effects of Citizen Input on the Design of the Boston Southwest Corridor , 2001 .

[14]  Christopher D. Hundhausen,et al.  The design and experimental evaluation of a tool to support the construction and wizard-of-oz testing of low fidelity prototypes , 2008, 2008 IEEE Symposium on Visual Languages and Human-Centric Computing.

[15]  Carman Neustaedter,et al.  "LINC-ing" the family: the participatory design of an inkable family calendar , 2006, CHI.

[16]  Jakob Nielsen,et al.  Heuristic Evaluation of Prototypes (individual) , 2022 .

[17]  Andy Dearden.  A Tool to Support Collaboration in Electronic Paper Prototyping , 2005 .

[18]  Hock-Hai Teo,et al.  An empirical study of the effects of interactivity on web user attitude , 2003, Int. J. Hum. Comput. Stud..

[19]  Douglas G. Altman,et al.  Practical statistics for medical research , 1990 .

[20]  Elizabeth Foss,et al.  FACIT PD: a framework for analysis and creation of intergenerational techniques for participatory design , 2013, CHI.

[21]  Jacob Cohen,et al.  The Equivalence of Weighted Kappa and the Intraclass Correlation Coefficient as Measures of Reliability , 1973 .

[22]  Elizabeth Foss,et al.  Layered elaboration: a new technique for co-design with children , 2010, CHI.

[23]  J. Miller,et al.  A Comparison of Tool-Based and Paper-Based Software Inspection , 1998, Empirical Software Engineering.

[24]  J. Fleiss.  Statistical methods for rates and proportions , 1974 .

[25]  Elizabeth Foss,et al.  DisCo: a co-design online tool for asynchronous distributed child and adult design partners , 2012, IDC '12.

[26]  Elizabeth B.-N. Sanders,et al.  A framework for organizing the tools and techniques of participatory design , 2010, PDC '10.

[27]  W. Grove.  Statistical Methods for Rates and Proportions, 2nd ed , 1981 .