Creating an instrument for evaluating critical thinking apps for college students

The purpose of this study was to identify salient features for a critical thinking app and create an instrument to facilitate the app evaluation and selection process. Two questions guided the study: (1) What distinguishes critical thinking instructional apps from others? and (2) What design principles are essential to develop a critical thinking instructional app? The study was conducted in two phases, including a synthesis of existing research in Phase I and development of an evaluation instrument in Phase II. Three lines of research (on critical thinking, educational apps design principles, and tools for the evaluation of educational apps) informed the development of the instrument, which included three evaluation categories (content, pedagogy, and design). A synthesis of research used to create the instrument is included herein along with the instrument design process, rationale for this design, recommendations for usage, its limitations, and implications for future practice and research. Findings will enable app users to more wisely select critical thinking apps specific to their needs and assist app developers with distinguishing the salient qualities required to design apps for critical thinking. The study accordingly contributes to both software evaluation and its development with findings beneficial to both app users and developers.

[1]  Cecilia M. Plaza,et al.  An Environmental Scan on the Status of Critical Thinking and Problem Solving Skills in Colleges/Schools of Pharmacy: Report of the 2009–2010 Academic Affairs Standing Committee , 2010, American Journal of Pharmaceutical Education.

[2]  T. Chenault,et al.  An Act of Translation: The Need to Understand Students' Understanding of Critical Thinking in the Undergraduate Classroom. , 2008 .

[3]  L. Hedges,et al.  RESEARCH SYNTHESIS AS A SCIENTIFIC PROCESS , 2009, The Handbook of Research Synthesis and Meta-Analysis.

[4]  Russ Unger,et al.  A Project Guide to UX Design: For User Experience Designers in the Field or in the Making , 2009 .

[5]  Steven J. McGahan,et al.  Online Course Quality Assurance: Development of a Quality Checklist. , 2015 .

[6]  T. Gog,et al.  Improving students' critical thinking : Empirical support for explicit instructions combined with practice , 2014 .

[7]  P. Facione.  Critical Thinking: A Statement of Expert Consensus for Purposes of Educational Assessment and Instruction. Research Findings and Recommendations. , 1990 .

[8]  Jakob Nielsen,et al.  Enhancing the explanatory power of usability heuristics , 1994, CHI '94.

[9]  Robert H. Ennis.  A taxonomy of critical thinking dispositions and abilitities , 1987 .

[10]  Robert H. Ennis.  Is Critical Thinking Culturally Biased , 1998 .

[11]  Developing critical thinking through leadership education. , 2016 .

[12]  Magdalena Mateescu,et al.  Mobile and ubiquitous learning in higher education settings. A systematic review of empirical studies , 2016, Comput. Hum. Behav..

[13]  Fernando Rosell-Aguilar,et al.  State of the App: A Taxonomy and Framework for Evaluating Language Learning Mobile Applications , 2016 .

[14]  Robert H. Ennis.  Critical Thinking and Subject Specificity: Clarification and Needed Research , 1989 .

[15]  Mark Bullen,et al.  What is Instructional Design , 2014 .

[16]  D. Norman.  The Design of Everyday Things: Revised and Expanded Edition , 2013 .

[17]  Lucy Santos Green,et al.  Mobile app selection for 5th through 12th grade science: The development of the MASS rubric , 2014, Comput. Educ..

[18]  Thomas Cochrane.  Social Implications of Mobile Learning in Global Learning Environments , 2013 .

[19]  Cheng-Yuan Lee,et al.  A Comprehensive Evaluation Rubric for Assessing Instructional Apps , 2015, J. Inf. Technol. Educ. Res..

[20]  Ada Haynes,et al.  Moving beyond Assessment to Improving Students' Critical Thinking Skills: A Model for Implementing Change. , 2016 .

[21]  J. Mathias.  Thinking Like a Social Worker: Examining the Meaning of Critical Thinking in Social Work , 2015 .

[22]  Jan Elen,et al.  Effectiveness of Critical Thinking Instruction in Higher Education: A Systematic Review of Intervention Studies , 2013 .

[23]  Ivy Sui-Yu Yau,et al.  Effects of mobile apps for nursing students: learning motivation, social interaction and study performance , 2018 .

[24]  Catherine Nickerson,et al.  Mobile or Not? Assessing the Instructional Value of Mobile Learning , 2017 .

[25]  W. Smets,et al.  High Quality Differentiated Instruction - A Checklist for Teacher Professional Development on Handling Differences in the General Education Classroom , 2017 .

[26]  F. A,et al.  The Disposition Toward Critical Thinking : Its Character , Measurement , and Relationship to Critical Thinking Skill , 2007 .

[27]  Betty Tärning,et al.  Review of Feedback in Digital Applications - Does the Feedback They Provide Support Learning? , 2018, J. Inf. Technol. Educ. Res..

[28]  J. Hedberg,et al.  Visualizing Solutions : Apps as Cognitive SteppingStones in the Learning Process , 2015 .

[29]  Jennifer Sabourin,et al.  Mobile Learning: A Handbook for Developers, Educators, and Learners , 2015 .

[30]  Eduardo Garcia Villada,et al.  CALL Evaluation for Early Foreign Language Learning: A Review of the Literature and a Framework for Evaluation. , 2013 .

[31]  Robert H. Ennis,et al.  Evaluating Critical Thinking , 1989 .

[32]  Ralph H. Johnson,et al.  A Meta-Level Approach to the Problem of Defining ‘Critical Thinking’ , 2015 .

[33]  Andrea E Cladis.  A shifting paradigm: An evaluation of the pervasive effects of digital technologies on language expression, creativity, critical thinking, political discourse, and interactive processes of human communications , 2020 .

[34]  Brian R. Bryant,et al.  How to Find Good Apps , 2016 .

[35]  Timos Almpanis.  Staff Development and Institutional Support for Technology Enhanced Learning in UK Universities , 2015 .

[36]  Sharon Bailin,et al.  Conceptualizing critical thinking , 1999 .

[37]  L. A. Hausman.  How we Think , 1921 .

[38]  Todd Cherner,et al.  A Detailed Rubric for Assessing the Quality of Teacher Resource Apps , 2016, J. Inf. Technol. Educ. Innov. Pract..

[39]  Helen Crompton,et al.  The use of mobile learning in higher education: A systematic review , 2018, Comput. Educ..

[40]  S. Burns,et al.  Faculty Perceptions of Critical Thinking at a Health Sciences University. , 2013 .

[41]  Hollylynne S. Lee,et al.  iPads in the Mathematics Classroom: Developing Criteria for Selecting Appropriate Learning Apps. , 2018 .

[42]  Kristoffer Larsson.  Understanding and teaching critical thinking—A new approach , 2017 .

[43]  Theodore J. Kopcha,et al.  Instructional Design Models , 2014 .

[44]  David Parsons,et al.  The future of mobile learning and implications for education and training , 2016 .

[45]  Abdulkareem Al-Alwani,et al.  Evaluation Criterion for Quality Assessment of E-Learning Content , 2014 .

[46]  Kenneth J. Luterbach,et al.  Designing and Developing Robust Instructional Apps , 2018 .

[47]  Xingyu Pan,et al.  Tablet-Based Math Assessment: What Can We Learn from Math Apps? , 2015, J. Educ. Technol. Soc..

[48]  Diana Laurillard,et al.  The pedagogical challenges to collaborative technologies , 2009, Int. J. Comput. Support. Collab. Learn..

[49]  Ben Shneiderman,et al.  Designing the User Interface: Strategies for Effective Human-Computer Interaction , 1998 .

[50]  Philip C. Abrami,et al.  Strategies for Teaching Students to Think Critically , 2015 .

[51]  Jim Garrison,et al.  Philosophy of Education , 1936 .

[52]  Robert H. Ennis.  Critical Thinking Across the Curriculum: A Vision , 2018 .

[53]  Blanka Frydrychova Klimova,et al.  Mobile phones and/or smartphones and their apps for teaching English as a foreign language , 2018, Education and Information Technologies.

[54]  Michail Kalogiannakis,et al.  Designing and creating an educational app rubric for preschool teachers , 2017, Education and Information Technologies.

[55]  J. Rowsell,et al.  Free Play or Tight Spaces? Mapping Participatory Literacies in Apps , 2016 .

[56]  Monica W. Tracey,et al.  ID model construction and validation: a multiple intelligences case , 2007 .

[57]  Robert M. Bernard,et al.  Instructional Interventions Affecting Critical Thinking Skills and Dispositions: A Stage 1 Meta-Analysis , 2008 .

[58]  Gina S. Matkin,et al.  Deficient Critical Thinking Skills among College Graduates: Implications for leadership , 2012 .

[59]  Linda S. Behar-Horenstein,et al.  Do instructional interventions influence college students’ critical thinking skills? A meta-analysis , 2013 .

[60]  Kathy Hirsh-Pasek,et al.  Putting Education in “Educational” Apps , 2015, Psychological science in the public interest : a journal of the American Psychological Society.

[61]  Sean Lennon,et al.  Using Online Learning Platforms to Enhance Students' Reflective and Critical Thinking. , 2010 .