Learning in Digital Networks - ICT literacy: A novel assessment of students' 21st century skills

The present investigation aims to fill some of the gaps revealed in the literature regarding the limited access to more advanced and novel assessment instruments for measuring students' ICT literacy. In particular, this study outlines the adaption, further development, and validation of the Learning in Digital NetworksICT literacy (LDN-ICT) test. The LDN-ICT test comprises an online performance-based assessment in which real-time student-student collaboration is facilitated through two different platforms (i.e., GoogleDocs and chat). The test attempts to measure students ability in handling digital information, to communicate and collaborate during problem solving. The data are derived from 144 students in grade 9 analyzed using item response theory models (unidimensional and multidimensional Rasch models). The appropriateness of the models was evaluated by examining the item fit statistics. To gather validity evidence for the test, we investigated the differential item functioning of the individual items and correlations with other constructs (e.g., self-efficacy, collective efficacy, perceived usefulness and academic aspirations). Our results supported the hypothesized structure of LDN-ICT as comprising four dimensions. No significant differences across gender groups were identified. In support of existing research, we found positive relations to self-efficacy, academic aspirations, and socio-economic background. In sum, our results provide evidence for the reliability and validity of the test. Further refinements and the future use of the test are discussed. The Learning in Digital Networks (LDN-ICT) test is comprised of multiple dimensions.Multidimensional Rasch model shows better fit than Unidimensional Rasch model.Evidence for reliability and validity of the LDN-ICT test is established.LDN-ICT is positively related to students' self-efficacy and academic aspirations.Differences in LDN-ICT across socio-economic status exit, yet not across gender.

[1]  Bengt Muthén,et al.  A Method for Studying the Homogeneity of Test Items with Respect to Other Relevant Variables , 1985 .

[2]  Linda C. Schmidt,et al.  Collective efficacy beliefs in student work teams: Relation to self-efficacy, cohesion, and performance , 2006 .

[3]  Marjolein Drent,et al.  Gender differences in computer attitudes: Does the school matter? , 2008, Comput. Hum. Behav..

[4]  Donald Hedeker,et al.  Full-information item bi-factor analysis , 1992 .

[5]  Alija Kulenović,et al.  Standards for Educational and Psychological Testing , 1999 .

[6]  Hye-Sook Kim,et al.  An analysis of variables affecting the ICT literacy level of Korean elementary school students , 2014, Comput. Educ..

[7]  Joke Voogt,et al.  Under which conditions does ICT have a positive effect on teaching and learning? A Call to Action , 2013, J. Comput. Assist. Learn..

[8]  Jörg Wittwer,et al.  The Test of Technological and Information Literacy (TILT) in the National Educational Panel Study: Development, empirical testing, and evidence for validity , 2013 .

[9]  Deborah Compeau,et al.  Application of Social Cognitive Theory to Training for Computer Skills , 1995, Inf. Syst. Res..

[10]  Antonio Fini,et al.  Are young generations in secondary school digitally competent? A study on Italian teenagers , 2012, Comput. Educ..

[11]  Johan van Braak,et al.  Gender and socioeconomic related differences in performance based ICT competences , 2015, Comput. Educ..

[12]  Ulrich Trautwein,et al.  Integration of multidimensional self-concept and core personality constructs: construct validation and relations to well-being and achievement. , 2006, Journal of personality.

[13]  G. Halász,et al.  Key Competences in Europe: Interpretation, Policy Formulation and Implementation. , 2011 .

[14]  Perman Gochyyev,et al.  Rethinking ICT literacy: From computer skills to social network settings , 2015 .

[15]  Miguel Nussbaum,et al.  Comparing marginal effects of Chilean students' economic, social and cultural status on digital versus reading and mathematics performance , 2015, Comput. Educ..

[16]  Friedrich W. Hesse,et al.  A Framework for Teachable Collaborative Problem Solving Skills , 2015 .

[17]  J. Fraillon,et al.  Preparing for Life in a Digital Age: The IEA International Computer and Information Literacy Study International Report , 2014 .

[18]  Sophia Rabe-Hesketh,et al.  Multilevel and Longitudinal Modeling Using Stata , 2005 .

[19]  Derek C. Briggs,et al.  An introduction to multidimensional measurement using Rasch models. , 2003, Journal of applied measurement.

[20]  Jean Gordon,et al.  Key Competences in Europe: Opening Doors for Lifelong Learners Across the School Curriculum and Teacher Education , 2009 .

[21]  Robert M. Klassen,et al.  Optimism and realism: A review of self‐efficacy from a cross‐cultural perspective , 2004 .

[22]  Alan Durndell,et al.  Computer self efficacy, computer anxiety, attitudes towards the Internet and reported experience with the Internet, by gender, in an East European sample , 2002, Comput. Hum. Behav..

[23]  Kathy E. Green,et al.  Stability of the Philadelphia Geriatric Center Morale Scale: A Multidimensional Item Response Model and Rasch Analysis , 2010 .

[24]  Michael Koller,et al.  A simulation study provided sample size guidance for differential item functioning (DIF) studies using short scales. , 2009, Journal of clinical epidemiology.

[25]  R. Hambleton,et al.  An NCME Instructional Module on Comparison of Classical Test Theory and Item Response Theory and Their Applications to Test Development. , 2005 .

[26]  R. Hambleton,et al.  Item Response Theory: Principles and Applications , 1984 .

[27]  Albert D. Ritzhaupt,et al.  Measuring Information and Communication Technology Literacy using a performance assessment: Validation of the Student Tool for Technology Literacy (ST2L) , 2014, Comput. Educ..

[28]  Rebecca Zwick,et al.  A Review of ETS Differential Item Functioning Assessment Procedures: Flagging Rules, Minimum Sample Size Requirements, and Criterion Refinement , 2012 .

[29]  Mark Wilson,et al.  Constructing Measures: An Item Response Modeling Approach , 2004 .

[30]  J. Voogt,et al.  A comparative analysis of international frameworks for 21st century competences: Implications for national curriculum policies , 2012 .

[31]  Robert L. Montgomery,et al.  Personality, Achievement Test Scores, and High School Percentile as Predictors of Academic Performance Across Four Years of Coursework , 2006 .

[32]  Mary Thorpe,et al.  Student attitudes towards and use of ICT in course study, work and social activity: A technology acceptance model approach , 2012, Br. J. Educ. Technol..

[33]  Albert V. Carron,et al.  Collective efficacy and group performance. , 1992 .

[34]  Ronny Scherer,et al.  Teachers' emphasis on developing students' digital information and communication skills (TEDDICS): A new construct in 21st century education , 2016, Comput. Educ..

[35]  Ioanna Vekiri,et al.  Gender issues in technology use: Perceived social support, computer self-efficacy and value beliefs, and computer use beyond school , 2008, Comput. Educ..

[36]  L R Bergman,et al.  A person-oriented approach in research on developmental psychopathology , 1997, Development and Psychopathology.

[37]  H. Cooper,et al.  The Relation Between Self-Beliefs and Academic Achievement: A Meta-Analytic Review , 2004 .

[38]  Anthony J. Nyberg,et al.  Collective efficacy, group potency, and group performance: meta-analyses of their relationships, and test of a mediation model. , 2009, The Journal of applied psychology.

[39]  J. Fraillon,et al.  IEA International Computer and Information Literacy Study 2018 Assessment Framework , 2019 .

[40]  F. Baker,et al.  Item response theory : parameter estimation techniques , 1993 .

[41]  Mark D. Reckase,et al.  An Extension of the Two-Parameter Logistic Model to the Multidimensional Latent Space. , 1983 .

[42]  Sophia Rabe-Hesketh,et al.  Multilevel and Longitudinal Modeling Using Stata, Second Edition , 2008 .

[43]  Margaret Wu,et al.  Properties of Rasch residual fit statistics. , 2013, Journal of applied measurement.

[44]  R. J. De Ayala,et al.  The IRT Tradition and its Applications , 2013 .

[45]  G. Masters.  A rasch model for partial credit scoring , 1982 .

[46]  A. Bandura,et al.  Longitudinal Analysis of the Role of Perceived Self-Efficacy for Self-Regulated Learning in Academic Continuance and Achievement. , 2008 .

[47]  Patrick Griffin,et al.  Assessment and Teaching of 21st Century Skills , 2011 .

[48]  Kristy J. Lauver,et al.  Do psychosocial and study skill factors predict college outcomes? A meta-analysis. , 2004, Psychological bulletin.

[49]  C. Fox,et al.  Applying the Rasch Model: Fundamental Measurement in the Human Sciences , 2001 .

[50]  A. Bandura.  GUIDE FOR CONSTRUCTING SELF-EFFICACY SCALES , 2006 .

[51]  Scheuermann Friedrich,et al.  Assessing the effects of ICT in education - Indicators, criteria and benchmarks for international comparisons , 2009 .

[52]  J. Linacre,et al.  Sample size and item calibration stability , 1994 .

[53]  Ioanna Vekiri,et al.  Socioeconomic differences in elementary students' ICT beliefs and out-of-school experiences , 2010, Comput. Educ..

[54]  Jared Keengwe,et al.  Faculty Integration of Technology into Instruction and Students’ Perceptions of Computer Technology to Improve Student Learning , 2007 .

[55]  Ove Edvard Hatlevik,et al.  An emerging digital divide in urban school children's information literacy: Challenging equity in the Norwegian school system , 2013, First Monday.

[56]  R. Olsen,et al.  Taking a future perspective by learning from the past – A systematic review of assessment instruments that aim to measure primary and secondary school students' ICT literacy , 2016 .

[57]  Ferrari Anusca.  DIGCOMP: A Framework for Developing and Understanding Digital Competence in Europe. , 2013 .

[58]  Chin-Chung Tsai,et al.  Elementary school students' attitudes and self-efficacy of using PDAs in a ubiquitous learning context , 2010 .

[59]  S. Duncan,et al.  Gender issues. , 2012, Handbook of clinical neurology.

[60]  M. Thomas.  The Value of Item Response Theory in Clinical Assessment: A Review , 2011, Assessment.

[61]  B. Wright,et al.  Best test design , 1979 .

[62]  Linda Price,et al.  Learners and learning in the twenty‐first century: what do we know about students’ attitudes towards and experiences of information and communication technologies that will help us design courses? , 2005 .

[63]  Tina N. Hohlfeld,et al.  Are gender differences in perceived and demonstrated technology literacy significant? It depends on the model , 2013, Educational Technology Research and Development.

[64]  R. Hambleton,et al.  Fundamentals of Item Response Theory , 1991 .

[65]  David Thissen,et al.  Marginal maximum likelihood estimation for the one-parameter logistic model , 1982 .

[66]  I. Ajzen.  The theory of planned behavior , 1991 .

[67]  Kathleen Scalise,et al.  Assessment of Learning in Digital Networks , 2015 .

[68]  Perman Gochyyev,et al.  Modeling Data from Collaborative Assessments: Learning in Digital Interactive Social Networks. , 2017 .

[69]  Dongil Kim,et al.  Assessing student ICT literacy on a national level , 2009 .

[70]  Neil Selwyn,et al.  Challenging educational expectations of the social web: a web 2.0 far? , 2009 .

[71]  J. Herman,et al.  Defining Twenty-First Century Skills , 2012 .

[72]  Erika D Peterson,et al.  Collective Efficacy and Aspects of Shared Mental Models as Predictors of Performance Over Time in Work Groups , 2000 .

[73]  Mark D. Reckase,et al.  The Difficulty of Test Items That Measure More Than One Ability , 1985 .

[74]  Johan van Braak,et al.  Direct measures of digital information processing and communication skills in primary education: Using item response theory for the development and validation of an ICT competence scale , 2014, Comput. Educ..

[75]  Claudio Barbaranelli,et al.  The contribution of personality traits and self-efficacy beliefs to academic achievement: a longitudinal study. , 2011, The British journal of educational psychology.

[76]  Jin-Shei Lai,et al.  Procedures for the Analysis of Differential Item Functioning (DIF) for Small Sample Sizes , 2005, Evaluation & the health professions.

[77]  S. Messick.  Validity of Psychological Assessment: Validation of Inferences from Persons' Responses and Performances as Scientific Inquiry into Score Meaning. Research Report RR-94-45. , 1994 .

[78]  Manuel Larrán Jorge,et al.  Gender differences between faculty members in higher education: A literature review of selected higher education journals , 2016 .

[79]  Insu Paek,et al.  Formulating the Rasch Differential Item Functioning Model Under the Marginal Maximum Likelihood Estimation Context and Its Comparison With Mantel–Haenszel Procedure in Short Test and Small Sample Conditions , 2011 .

[80]  R. Goddard,et al.  Collective efficacy : A neglected construct in the study of schools and student achievement , 2001 .

[81]  Perman Gochyyev,et al.  Assessment of Learning in Digital Interactive Social Networks: A Learning Analytics Approach , 2016 .

[82]  Kara A. Incalcaterra,et al.  A meta-analysis of team-efficacy, potency, and performance: interdependence and level of analysis as moderators of observed relationships. , 2002, The Journal of applied psychology.

[83]  Eric T. Bradlow,et al.  A General Bayesian Model for Testlets: Theory and Applications , 2002 .

[84]  Ching-Yi Wu,et al.  Multidimensional Rasch validation of the Frenchay Activities Index in stroke patients receiving rehabilitation. , 2012, Journal of rehabilitation medicine.

[85]  Raymond J. Adams,et al.  The Multidimensional Random Coefficients Multinomial Logit Model , 1997 .

[86]  Vivek Venkatesh,et al.  Perceptions of Technology Use and Course Effectiveness in the Age of Web 2.0 : A Large-Scale Survey of Québec University Students and Instructors , 2012 .

[87]  Miguel Nussbaum,et al.  Assessment of 21st century ICT skills in Chile: Test design and results from high school level students , 2012, Comput. Educ..

[88]  A. Bandura.  Self-Efficacy: The Exercise of Control , 1997, Journal of Cognitive Psychotherapy.

[89]  Katerina Ananiadou,et al.  21st Century Skills and Competences for New Millennium Learners in OECD Countries , 2009 .

[90]  J. Fraillon,et al.  Preparing for life in a digital age , 2014 .

[91]  Patrick Griffin,et al.  The ATC21S Method , 2015 .

[92]  J. V. Dijk.  Digital divide research, achievements and shortcomings , 2006 .

[93]  J. Graydon,et al.  The impact of collective efficacy beliefs on effort and persistence in a group task. , 1999, Journal of sports sciences.

[94]  Fred D. Davis.  Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology , 1989, MIS Q..

[95]  David Pepper,et al.  Assessing Key Competences across the Curriculum--And Europe. , 2011 .

[96]  E. Muraki,et al.  Full-Information Item Factor Analysis , 1988 .