The Computer-Based Assessment of Complex Problem Solving and How It Is Influenced by Students' Information and Communication Technology Literacy.

The 21st-century work environment places strong emphasis on nonroutine transversal skills. In an educational context, complex problem solving (CPS) is generally considered an important transversal skill that includes knowledge acquisition and its application in new and interactive situations. The dynamic and interactive nature of CPS requires a computer-based administration of CPS tests such that the assessment of CPS might be partially confounded with information and communication technology (ICT) literacy. To establish CPS as a distinct construct that involves complex cognitive processes not covered by other general cognitive abilities and not related to ICT literacy, it is necessary to investigate the influence of ICT literacy on CPS and on the power of CPS to predict external educational criteria. We did so in 3 different samples of either high school or university students using a variety of instruments to measure ICT literacy and general cognitive ability. Convergent results based on structural equation modeling and confirmatory factor analyses across the studies showed that ICT literacy was weakly or moderately related to CPS, and these associations were similar to those between ICT and other general cognitive abilities. Furthermore, the power of CPS to predict external educational criteria over and above general cognitive ability remained even if the influence of ICT literacy on CPS was controlled for. We conclude that CPS is a distinct construct that captures complex cognitive processes not generally found in other assessments of general cognitive ability or of ICT literacy.

[1]  Philipp Sonnleitner,et al.  Differential relations between facets of complex problem solving and students’ immigration background , 2014 .

[2]  David Autor,et al.  RECENT TECHNOLOGICAL CHANGE: AN EMPIRICAL EXPLORATION* , 2003 .

[3]  Cleotilde Gonzalez,et al.  The Relationships between Cognitive Ability and Dynamic Decision Making. , 2005 .

[4]  J. Funke,et al.  Beyond IQ: A Latent State-Trait Analysis of General Intelligence, Dynamic Decision Making, and Implicit Learning , 2011 .

[5]  Cynthia G. Parshall,et al.  Practical Considerations in Computer-Based Testing , 2002 .

[6]  R. P. McDonald,et al.  Structural Equations with Latent Variables , 1989 .

[7]  M. Prensky.  Digital Natives, Digital Immigrants , 2001 .

[8]  Roy B. Clariana,et al.  Paper-based versus computer-based assessment: key factors associated with the test mode effect , 2002, Br. J. Educ. Technol..

[9]  Joachim Funke,et al.  Complex problem solving: a case for complex cognition? , 2010, Cognitive Processing.

[10]  Kenneth A. Bollen,et al.  Structural Equations with Latent Variables , 1989 .

[11]  Joachim Wirth,et al.  Computer-based Assessment of Problem Solving Competence , 2003 .

[12]  Mark A. Davis,et al.  Cognitive and emotional components of anxiety: literature review and a revised worry-emotionality scale. , 1981, Journal of educational psychology.

[13]  Detlev Leutner,et al.  The fuzzy relationship of intelligence and problem solving in computer simulations , 2002, Comput. Hum. Behav..

[14]  A. Collins,et al.  Situated Cognition and the Culture of Learning , 1989 .

[15]  Klaus Oberauer,et al.  The multiple faces of working memory: Storage, processing, supervision, and coordination , 2003 .

[16]  David Autor,et al.  The Skill Content of Recent Technological Change: An Empirical Exploration , 2003 .

[17]  M. Prensky.  Digital Natives, Digital Immigrants Part 1 , 2001 .

[18]  Joan A. Ballantine,et al.  Computer usage and the validity of self-assessed computer competence among first-year business students , 2007, Comput. Educ..

[19]  J. Carroll.  Human Cognitive Abilities-a sur-vey of factor-analytic studies , 1993 .

[20]  K. McGrew.  CHC theory and the human cognitive abilities project: Standing on the shoulders of the giants of psychometric intelligence research , 2009 .

[21]  Samuel Greiff,et al.  Complex problem solving : More than reasoning ? , 2017 .

[22]  Robert J. Mislevy,et al.  Automated scoring of complex tasks in computer-based testing , 2006 .

[23]  Samuel Greiff,et al.  Validity of the MicroDYN approach: Complex problem solving predicts school grades beyond working memory capacity , 2013 .

[24]  David Dorn,et al.  This Job is &Apos;Getting Old:&Apos; Measuring Changes in Job Opportunities Using Occupational Age Structure , 2009 .

[25]  J. Funke.  Dynamic systems as tools for analysing human judgement , 2001 .

[26]  Joachim Funke,et al.  Finite-State Automata: Dynamic Task Environments in Problem-Solving Research , 1993 .

[27]  Philipp Sonnleitner,et al.  Students' complex problem-solving abilities: Their structure and relations to reasoning ability and educational success , 2013 .

[28]  W. Wittmann,et al.  Investigating the paths between working memory, intelligence, knowledge, and complex problem solving performances via Brunswik-symmetry , 1999 .

[29]  Regina Vollmeyer,et al.  Personen- und Aufgabenmerkmale beim komplexen Problemlösen , 1999 .

[30]  J. Raven,et al.  Manual for Raven's progressive matrices and vocabulary scales , 1962 .

[31]  P. Bentler,et al.  Cutoff criteria for fit indexes in covariance structure analysis : Conventional criteria versus new alternatives , 1999 .

[33]  F. Drasgow,et al.  Equivalence of computerized and paper-and-pencil cognitive ability tests: A meta-analysis. , 1993 .

[34]  Meng-Jung Tsai,et al.  Do Male Students Often Perform Better Than Female Students When Learning Computers?: A Study of Taiwanese Eighth Graders' Computer Education through Strategic and Cooperative Learning , 2002 .

[35]  T. Richter,et al.  Eine revidierte Fassung des Inventars zur Computerbildung (INCOBI-R) , 2010 .

[36]  Samuel Greiff,et al.  Funke Dynamic Problem Solving : A New Assessment Perspective , 2012 .

[37]  Samuel Greiff,et al.  Dynamic Problem Solving , 2012 .

[38]  Johan van Braak,et al.  Domains and determinants of university students' self-perceived computer competence , 2004, Comput. Educ..

[39]  Deanna Kuhn,et al.  Do Students Need to Be Taught How to Reason , 2009 .

[40]  Johannes Naumann,et al.  Assessing Individual Differences in Basic Computer Skills , 2013 .

[41]  Samuel Greiff,et al.  Complex Problem Solving in Educational Contexts—Something Beyond g: Concept, Assessment, Measurement Invariance, and Construct Validity , 2013 .

[42]  Alexandra Spitz-Oener.  Technical Change, Job Tasks, and Rising Educational Demands: Looking outside the Wage Structure , 2006, Journal of Labor Economics.

[43]  Samuel Greiff,et al.  A multitrait-multimethod study of assessment instruments for complex problem solving☆ , 2013 .

[44]  T. Little,et al.  To Parcel or Not to Parcel: Exploring the Question, Weighing the Merits , 2002 .

[45]  P. Frensch,et al.  Complex problem solving : the European perspective , 1995 .

[46]  N. Sander.  Inhibitory and executive functions in cognitive psychology : an individual differences approach examining structure and overlap with working memory capacity and intelligence , 2005 .

[47]  Richard D. Roberts,et al.  Learning and Individual Differences: Process, Trait and Content Determinants , 1999 .

[48]  J. Sweller.  Implications of Cognitive Load Theory for Multimedia Learning , 2005, The Cambridge Handbook of Multimedia Learning.

[49]  Samuel Greiff,et al.  Inductive reasoning, domain specific and complex problem solving: Relations and development , 2013 .

[50]  S. Ian Robertson,et al.  Problem-solving , 2001, Human Thinking.

[51]  David R. Michael,et al.  Serious Games: Games That Educate, Train, and Inform , 2005 .

[52]  O. Wilhelm,et al.  Working memory capacity - facets of a cognitive ability construct , 2000 .

[53]  Michael Russell,et al.  Computer-based Testing and Validity: a look back into the future , 2003 .

[54]  Samuel Greiff,et al.  The Genetics Lab: Acceptance and psychometric characteristics of a computer-based microworld assessing complex problem solving , 2012 .

[55]  Jan L. Plass,et al.  Intelligence Assessment with Computer Simulations. , 2005 .

[56]  Samuel Greiff,et al.  The Process of Solving Complex Problems , 2012, J. Probl. Solving.

[57]  Klaus Oberauer,et al.  Which Working Memory Functions Predict Intelligence , 2008 .

[58]  D. D. Bickerstaff,et al.  Computerized Adaptive Testing , 1989 .

[59]  Markus Appel,et al.  Are heavy users of computer games and social media more computer literate? , 2012, Comput. Educ..

[60]  R. Brennan.  Elements of generalizability theory , 1983 .