Critter: Augmenting Creative Work with Dynamic Checklists, Automated Quality Assurance, and Contextual Reviewer Feedback

Checklists and guidelines have played an increasingly important role in complex tasks ranging from the cockpit to the operating theater. Their role in creative tasks like design is less explored. In a needfinding study with expert web designers, we identified designers' challenges in adhering to a checklist of design guidelines. We built Critter, which addressed these challenges with three components: Dynamic Checklists that progressively disclose guideline complexity with a self-pruning hierarchical view, AutoQA to automate common quality assurance checks, and guideline-specific feedback provided by a reviewer to highlight mistakes as they appear. In an observational study, we found that the more engaged a designer was with Critter, the fewer mistakes they made in following design guidelines. Designers rated the AutoQA and contextual feedback experience highly, and provided feedback on the tradeoffs of the hierarchical Dynamic Checklists. We additionally found that a majority of designers rated the AutoQA experience as excellent and felt that it increased the quality of their work. Finally, we discuss broader implications for supporting complex creative tasks.

[1]  Sidney L. Smith,et al.  Application of Guidelines for Designing User Interface Software , 1985 .

[3]  Larry Ambrose,et al.  The power of feedback. , 2002, Healthcare executive.

[4]  T. R. Viswanathan,et al.  Integrated circuit testing for quality assurance in manufacturing: history, current status, and future trends , 1997 .

[5]  Elizabeth Fayram Fostering Reflection and Providing Feedback: Helping Others Learn From Experience , 2004 .

[6]  Scott R. Klemmer,et al.  Early and Repeated Exposure to Examples Improves Creative Work , 2012, CogSci.

[7]  P. Pronovost,et al.  The checklist--a tool for error management and performance improvement. , 2006, Journal of critical care.

[8]  David R. Schwartz,et al.  The use of guidelines in interface design , 1991, CHI.

[9]  Maneesh Agrawala,et al.  Interactive furniture layout using interior design guidelines , 2011, SIGGRAPH 2011.

[10]  Daniel Boorman,et al.  TODAY'S ELECTRONIC CHECKLISTS REDUCE LIKELIHOOD OF CREW ERRORS AND HELP PREVENT MISHAPS , 2001 .

[11]  Barry G. Silverman,et al.  Expert critics: operationalizing the judgement/decision-making literature as a theory of “bugs” and repair strategies , 1991 .

[12]  Gerhard Fischer,et al.  Embedding critics in design environments , 1993, The Knowledge Engineering Review.

[13]  D. Schoen,et al.  The Reflective Practitioner: How Professionals Think in Action , 1985 .

[14]  M. Hewson,et al.  Giving feedback in medical education: verification of recommended techniques. , 1998, Journal of general internal medicine.

[15]  Dominique L. Scapin,et al.  Organizing human factors knowledge for the evaluation and design of interfaces , 1990, Int. J. Hum. Comput. Interact..

[16]  Jonas Löwgren,et al.  Supporting the Use of Guidelines and Style Guides in Professional User Interface Design , 1993, Interact. Comput..

[17]  Jun Gong,et al.  GUIDELINES FOR HANDHELD MOBILE DEVICE INTERFACE DESIGN , 2004 .

[18]  Elizabeth D. Mynatt,et al.  Recognizing creative needs in user interface design , 2002, Creativity & Cognition.

[19]  Justin Cheng,et al.  Peer and self assessment in massive online classes , 2013, ACM Trans. Comput. Hum. Interact..

[20]  Scott Henninger,et al.  A framework for developing experience-based usability guidelines , 1995, Symposium on Designing Interactive Systems.

[21]  Scott R. Klemmer,et al.  Interactive Guidance Techniques for Improving Creative Feedback , 2018, CHI.

[22]  Earl L. Wiener,et al.  Human factors of flight-deck checklists: The normal checklist , 1990 .

[23]  Vera Dammann,et al.  Quality Assurance in Analytical Chemistry: Applications in Environmental, Food and Materials Analysis, Biotechnology, and Medical Engineering , 2006 .

[24]  Huhn Kim,et al.  Effective organization of design guidelines reflecting designer's design strategies , 2010 .

[25]  J. Bergs,et al.  Systematic review and meta‐analysis of the effect of the World Health Organization surgical safety checklist on postoperative complications , 2014, The British journal of surgery.

[26]  Michael S. Bernstein,et al.  Expert crowdsourcing with flash teams , 2014, UIST.

[27]  Mark Warschauer,et al.  Automated writing evaluation: defining the classroom research agenda , 2006 .

[28]  R. Helmreich On error management: lessons from aviation , 2000, BMJ : British Medical Journal.

[29]  Mary Shaw,et al.  Writing good software engineering research papers: minitutorial , 2003, ICSE 2003.

[30]  R. J. Bogumil,et al.  The reflective practitioner: How professionals think in action , 1985, Proceedings of the IEEE.

[31]  Krzysztof Z. Gajos,et al.  Human computation tasks with global constraints , 2012, CHI.

[32]  Michael Scriven,et al.  THE LOGIC AND METHODOLOGY OF CHECKLISTS , 2005 .

[33]  W. Berry,et al.  A Surgical Safety Checklist to Reduce Morbidity and Mortality in a Global Population , 2009, The New England journal of medicine.

[34]  Mary Shaw,et al.  Writing good software engineering research papers , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[35]  Björn Hartmann,et al.  Almost an Expert: The Effects of Rubrics and Expertise on Perceived Value of Crowdsourced Design Critiques , 2016, CSCW.

[36]  Scott R. Klemmer,et al.  Shepherding the crowd yields better work , 2012, CSCW.

[37]  Michael S. Bernstein,et al.  No Workflow Can Ever Be Enough , 2017, Proc. ACM Hum. Comput. Interact..

[38]  Adam Marcus,et al.  Argonaut: Macrotask Crowdsourcing for Complex Data Processing , 2015, Proc. VLDB Endow..

[39]  I. Seidman Interviewing as qualitative research : a guide for researchersin education and the social sciences , 1991 .

[40]  Jose L. Galvan,et al.  Writing Literature Reviews , 2017 .

[41]  Tathagat Varma,et al.  Automated software testing: introduction, management and performance , 2000, SOEN.

[42]  Jakob Nielsen,et al.  Assessing the usability of a user interface standard , 1991, CHI.

[43]  Murray Hill,et al.  Lint, a C Program Checker , 1978 .

[44]  Myung Hwan Yun,et al.  Affective Evaluation of Vehicle Interior Craftsmanship: Systematic Checklists for Touch/Feel Quality of Surface-Covering Material , 2004 .

[45]  C. Marlin Brown,et al.  Human-Computer Interface Design Guidelines , 1998 .

[46]  Panayiotis Zaphiris,et al.  Research-derived web design guidelines for older people , 2005, Assets '05.