Challenges of Assessing Collaborative Problem Solving

An assessment of Collaborative Problem Solving (CPS) proficiency was developed by an expert group for the PISA 2015 international evaluation of student skills and knowledge. The assessment framework defined CPS skills by crossing three major CPS competencies with four problem solving processes that were adopted from PISA 2012 Complex Problem Solving to form a matrix of 12 specific skills. The three CPS competencies are (1) establishing and maintaining shared understanding, (2) taking appropriate action, and (3) establishing and maintaining team organization. For the assessment, computer-based agents provide the means to assess students by varying group composition and discourse across multiple collaborative situations within a short period of time. Student proficiency is then measured by the extent to which students respond to requests and initiate actions or communications to advance the group goals. This chapter identifies considerations and challenges in the design of a collaborative problem solving assessment for large-scale testing.

[1]  Gary Klein,et al.  Naturalistic Decision Making , 2008, Hum. Factors.

[2]  Yigal Rosen,et al.  Comparability of Conflict Opportunities in Human-to-Human and Human-to-Agent Online Collaborative Problem Solving , 2014, Technol. Knowl. Learn..

[3]  Kyle Lewis,et al.  Transactive Memory Systems, Learning, and Learning Transfer , 2005, Organ. Sci..

[4]  G. T. Jackson,et al.  Motivation and performance in a game-based intelligent tutoring system , 2013 .

[5]  Yigal Rosen,et al.  Using a Collaborative Database to Enhance Students’ Knowledge Construction , 2009 .

[6]  Daniel L. Schwartz,et al.  The Emergence of Abstract Representations in Dyad Problem Solving , 1995 .

[7]  Jung-Lung Hsu,et al.  The Effects of Communicative Genres on Intra-Group Conflict in Virtual Student Teams , 2009, Int. J. Distance Educ. Technol..

[8]  Joachim Funke,et al.  Negative affective environments improve complex solving performance , 2010 .

[9]  E. Schegloff,et al.  A simplest systematics for the organization of turn-taking for conversation , 1974 .

[10]  John Yen,et al.  NDM-Based Cognitive Agents for Supporting Decision-Making Teams , 2010, Hum. Comput. Interact..

[11]  David Williamson Shaffer,et al.  Games, Learning, and Society: Models of Situated Action , 2012 .

[12]  Sabine Sonnentag,et al.  The relationship between high performance and knowledge about how to master cooperation situations , 2002 .

[13]  Stephen M. Fiore,et al.  Process mapping and shared cognition: Teamwork and the development of shared problem models. , 2004 .

[14]  Nikol Rummel,et al.  Are two heads always better than one? Differential effects of collaboration on students’ computer-supported learning in mathematics , 2011, Int. J. Comput. Support. Collab. Learn..

[15]  Peter W. Foltz,et al.  Assessing Collaborative Problem Solving Through Automated Technologies , 2014 .

[16]  E. Aronson,et al.  The Jigsaw Classroom: Building Cooperation in the Classroom , 1997 .

[17]  Arthur C. Graesser,et al.  Inducing and Tracking Confusion with Contradictions during Complex Learning , 2013, Int. J. Artif. Intell. Educ..

[18]  Paul A. Kirschner,et al.  Identifying the pitfalls for social interaction in computer-supported collaborative learning environments: a review of the research , 2003, Comput. Hum. Behav..

[19]  Suchitra Krishnan-Sarin,et al.  Training the transdisciplinary scientist: a general framework applied to tobacco use behavior. , 2003, Nicotine & tobacco research : official journal of the Society for Research on Nicotine and Tobacco.

[20]  Eduardo Salas,et al.  Team Performance Assessment and Measurement: Theory, Methods, and Applications. Series in Applied Psychology. , 1997 .

[21]  A. Graesser,et al.  Domain-general problem solving skills and education in the 21st century , 2014 .

[22]  Arthur C. Graesser,et al.  The Cambridge Handbook of Psycholinguistics: Computational Modeling of Discourse and Conversation , 2012 .

[23]  G. Stasser,et al.  Hidden Profiles: A Brief History , 2003 .

[24]  W. Lewis Johnson,et al.  Tactical Language and Culture Training Systems: Using Artificial Intelligence to Teach Foreign Languages and Cultures , 2008, AAAI.

[25]  Arthur C. Graesser,et al.  Is it an Agent, or Just a Program?: A Taxonomy for Autonomous Agents , 1996, ATAL.

[26]  H. H. Clark,et al.  Referring as a collaborative process , 1986, Cognition.

[27]  Arthur C. Graesser,et al.  Operation ARIES!: A Serious Game for Teaching Scientific Inquiry , 2011, Serious Games and Edutainment Applications.

[28]  Renée E. DeRouin,et al.  Uncovering Workplace Interpersonal Skills: A Review, Framework, and Research Agenda , 2008 .

[29]  Arthur C. Graesser,et al.  Learning by Communicating in Natural Language With Conversational Agents , 2014 .

[30]  A. Hollingshead,et al.  Transactive Memory and Performance in Work Groups: Specificity, Communication, Ability Differences, and Work Allocation , 2008 .

[31]  Carolyn Penstein Rosé,et al.  Analyzing collaborative learning processes automatically: Exploiting the advances of computational linguistics in computer-supported collaborative learning , 2008, Int. J. Comput. Support. Collab. Learn..

[32]  Timothy O’Connor,et al.  The emergence of group cognition , 2010 .

[33]  Eva L. Baker,et al.  Computer-Based Feedback for Computer-Based Collaborative Problem Solving , 2010 .

[34]  Carolyn Prince,et al.  An Overview of Team Performance Measurement , 1997 .

[35]  Herbert H. Clark,et al.  Grounding in communication , 1991, Perspectives on socially shared cognition.

[36]  Stephen M. Fiore,et al.  Toward an Understanding of Macrocognition in Teams: Predicting Processes in Complex Collaborative Contexts , 2010, Hum. Factors.

[37]  E. Salas,et al.  Taking stock of naturalistic decision making , 2001 .

[38]  E. Salas,et al.  Reflections on shared cognition , 2001 .

[39]  Nancy J. Cooke,et al.  On Teams, Teamwork, and Team Performance: Discoveries and Developments , 2008, Hum. Factors.

[40]  Stephen M. Fiore,et al.  Processes in complex team problem-solving: Parsing and defining the theoretical problem space , 2008 .

[41]  Keith K. Millis,et al.  Operation ARA: A computerized learning game that teaches critical thinking and scientific reasoning , 2012 .

[42]  Patrick Griffin,et al.  The Changing Role of Education and Schools , 2012 .

[43]  Susan R. Fussell,et al.  Conversational Argumentation in Decision Making: Chinese and U.S. Participants in Face-to-Face and Instant-Messaging Interactions , 2007 .

[44]  Peter W. Foltz,et al.  Automated Communication Analysis of Teams , 2008 .

[45]  Ronald E. Rice,et al.  Unusual Routines: Organizational (Non)Sensemaking , 2008 .

[46]  Eduardo Salas,et al.  Complex collaborative problem-solving processes in mission control. , 2014, Aviation, space, and environmental medicine.

[47]  Emiel Krahmer,et al.  Generating Multimodal References , 2007 .

[48]  Baruch B. Schwarz,et al.  Argumentation and Explanation in Conceptual Change: Indications From Protocol Analyses of Peer-to-Peer Dialog , 2009, Cogn. Sci..

[49]  Joachim Funke,et al.  Complex problem solving: a case for complex cognition? , 2010, Cognitive Processing.

[50]  Jessica L. Wildman,et al.  Trust Development in Swift Starting Action Teams , 2012 .

[51]  A. Graesser,et al.  Handbook of discourse processes , 2003 .

[52]  John R. Austin.  Transactive memory in organizational groups: the effects of content, consensus, specialization, and accuracy on group performance. , 2003, The Journal of applied psychology.

[53]  Gautam Biswas,et al.  Measuring Self-Regulated Learning Skills through Social Interactions in a teachable Agent Environment , 2010, Res. Pract. Technol. Enhanc. Learn..

[54]  Pierre Dillenbourg,et al.  Collaborative Learning: Cognitive and Computational Approaches. Advances in Learning and Instruction Series. , 1999 .

[55]  Pierre Dillenbourg,et al.  Sharing Solutions: Persistence and Grounding in Multimodal Collaborative Problem Solving , 2006 .