Explaining learning performance using response-time, self-regulation and satisfaction from content: an fsQCA approach

This study focuses on compiling students' response-time allocated to answer correctly or wrongly, their self-regulation, as well as their satisfaction from content, in order to explain high or medium/low learning performance. To this end, it proposes a conceptual model in conjunction with research propositions. For the evaluation of the approach, an empirical study with 452 students was conducted. The fuzzy set qualitative comparative analysis (fsQCA) revealed five configurations driven by the admitted factors that explain students' high performance, as well as five additional patterns, interpreting students' medium/low performance. These findings advance our understanding of the relations between actual usage and latent behavioral factors, as well as their combined effect on students' test score. Limitations and potential implications of these findings are also discussed.

[1]  Jeremy Burrus,et al.  IDENTIFYING THE MOST IMPORTANT 21ST CENTURY WORKFORCE COMPETENCIES: AN ANALYSIS OF THE OCCUPATIONAL INFORMATION NETWORK (O*NET) , 2013 .

[2]  Di Challis,et al.  Committing to quality learning through adaptive online assessment , 2005 .

[3]  Billie Eilam,et al.  Students’ planning in the process of self-regulated learning , 2003 .

[4]  Fethi A. Inan,et al.  Examining the effects of learner-learner interactions on satisfaction and learning in an online undergraduate course , 2017, Comput. Educ..

[5]  Nari Kim,et al.  Assessment in Online Distance Education: A Comparison of Three Online Programs at a University. , 2008 .

[6]  Michail N. Giannakos,et al.  Making Sense of Learning Analytics with a Configurational Approach , 2016, SE@VBL@LAK.

[7]  M. Alonso,et al.  Efectos del Olvido Dirigido con el Método de las Palabras: Una comparación entre pruebas directas e indirectas de memoria , 2000 .

[8]  Yan Piaw Chua,et al.  Effects of computer-based testing on test performance and testing motivation , 2012, Comput. Hum. Behav..

[9]  A. Woodside.  Moving beyond multiple regression analysis to algorithms: Calling for adoption of a paradigm shift from symmetric to asymmetric thinking in data analysis and crafting theory , 2013 .

[10]  Anastasia Kitsantas,et al.  Test Preparation and Performance: A Self-Regulatory Analysis , 2002 .

[11]  B. Clauser,et al.  Modeling Pacing Behavior and Test Speededness Using Latent Growth Curve Models , 2013 .

[12]  T. Wolsey.  Efficacy of Instructor Feedback on Written Work in an Online Program , 2008 .

[13]  Barbara S. Plake,et al.  Development and Application of Detection Indices for Measuring Guessing Behaviors and Test-Taking Effort in Computerized Adaptive Testing , 2011 .

[14]  J. C. Setzer,et al.  An Investigation of Examinee Test-Taking Effort on a Large-Scale Assessment , 2013 .

[15]  I. Arnold,et al.  Cheating at online formative tests: Does it pay off? , 2016, Internet High. Educ..

[16]  Willem J. van der Linden.  A bivariate lognormal response-time model for the detection of collusion between test takers , 2009 .

[17]  Donna L. Sundre,et al.  An exploration of the psychology of the examinee: Can examinee self-regulation and test-taking motivation predict consequential and non-consequential test performance? , 2004 .

[18]  Yi-Shun Wang,et al.  Assessment of learner satisfaction with asynchronous electronic learning systems , 2003, Inf. Manag..

[19]  Bradley A. Hanson,et al.  Development and Calibration of an Item Response Model That Incorporates Response Time , 2005 .

[20]  Jerry M. Mendel,et al.  Charles Ragin's Fuzzy Set Qualitative Comparative Analysis (fsQCA) used for linguistic summarizations , 2012, Inf. Sci..

[21]  Anne R. Fitzpatrick.  The Meaning of Content Validity , 1983 .

[22]  Michail N. Giannakos,et al.  Value Co-Creation and Trust in Social Commerce: an fsQCA Approach , 2017, ECIS.

[23]  William H. Glick,et al.  Fit, Equifinality, and Organizational Effectiveness: A Test of Two Configurational Theories , 1993 .

[24]  Andrew Davis,et al.  The Limits of Educational Assessment , 1999 .

[25]  Anastasios A. Economides,et al.  The acceptance and use of computer based assessment , 2011, Comput. Educ..

[26]  Michail N. Giannakos,et al.  Fuzzy set analysis as a means to understand users of 21st-century learning systems: The case of mobile learning and reflections on learning analytics research , 2017, Comput. Hum. Behav..

[27]  Sandra Lowe,et al.  Redesigning Social Inquiry Fuzzy Sets And Beyond , 2016 .

[28]  Michail N. Giannakos,et al.  Assessing Student Behavior in Computer Science Education with an fsQCA Approach , 2017, ACM Trans. Comput. Educ..

[29]  Charles B. Hodges,et al.  Email, Self-Regulation, Self-Efficacy, and Achievement in a College Online Mathematics Course , 2010 .

[30]  Patrick A. Mello.  Fuzzy-Set Qualitative Comparative Analysis , 2014 .

[31]  Anastasios A. Economides,et al.  Temporal learning analytics for computer based testing , 2014, LAK.

[32]  William P. Darrow,et al.  Exploring the impact of personality on performance: How time-on-task moderates the mediation by self-efficacy , 2009 .

[33]  Lutz F. Hornke,et al.  Item Response Times in Computerized Adaptive Testing , 2000 .

[34]  Anastasios A. Economides,et al.  Exhibiting achievement behavior during computer-based testing: What temporal trace data and personality traits tell us? , 2017, Comput. Hum. Behav..

[35]  Michail N. Giannakos,et al.  An Exploratory Study on the Influence of Cognitive and Affective Characteristics in Programming-Based Making Activities , 2017, 2017 IEEE 17th International Conference on Advanced Learning Technologies (ICALT).

[36]  Jeremy Burrus,et al.  EXAMINING THE EFFICACY OF A TIME MANAGEMENT INTERVENTION FOR HIGH SCHOOL STUDENTS , 2013 .

[37]  Yi-Hsuan Lee,et al.  A review of recent response-time analyses in educational testing , 2011 .

[38]  Michail N. Giannakos,et al.  Enhancing Student Digital Skills: Adopting an Ecosystemic School Analytics Approach , 2017, 2017 IEEE 17th International Conference on Advanced Learning Technologies (ICALT).

[39]  A. Parasuraman,et al.  When the Recipe Is More Important Than the Ingredients , 2014 .

[40]  Peer C. Fiss.  Building Better Causal Theories: A Fuzzy Set Approach to Typologies in Organization Research , 2011 .

[41]  Anastasios A. Economides,et al.  Towards the alignment of computer-based assessment outcome with learning goals: The LAERS architecture , 2013, 2013 IEEE Conference on e-Learning, e-Management and e-Services.

[42]  Shelby J. Haberman,et al.  Investigating Test-Taking Behaviors Using Timing and Process Data , 2016 .

[43]  Joseph E. Beck,et al.  Engagement tracing: using response times to model student disengagement , 2005, AIED.

[44]  R. L. Dipboye,et al.  College students' time management : correlations with academic performance and stress , 1990 .

[45]  Maria Puzziferro.  Online Technologies Self-Efficacy and Self-Regulated Learning as Predictors of Final Grade and Satisfaction in College-Level Online Courses , 2008 .

[46]  Jan Dul,et al.  Identifying single necessary conditions with NCA and fsQCA , 2016 .

[47]  B. Zimmerman,et al.  Development of a Structured Interview for Assessing Student Use of Self-Regulated Learning Strategies , 1986 .

[48]  Dowming Yeh,et al.  What drives a successful e-Learning? An empirical investigation of the critical factors influencing learner satisfaction , 2008, Comput. Educ..

[49]  C. C. Lo,et al.  How Student Satisfaction Factors Affect Perceived Learning. , 2010 .

[50]  William J. Doll,et al.  The Measurement of End-User Computing Satisfaction , 1988, MIS Q..

[51]  Howard Wainer,et al.  Computerized Adaptive Testing: A Primer , 2000 .

[52]  J. Broadbent,et al.  Self-regulated learning strategies & academic achievement in online higher education learning environments: A systematic review , 2015, Internet High. Educ..

[53]  Steven L. Wise,et al.  Response Time Effort: A New Measure of Examinee Motivation in Computer-Based Tests , 2005 .

[54]  Yi-Shun Wang,et al.  Multi-criteria evaluation of the web-based e-learning system: A methodology based on learner satisfaction and its applications , 2008, Comput. Educ..