A Review of Three Large-Scale Datasets Critiquing Item Design, Data Collection, and the Usefulness of Claims

Issues of validity and usefulness of three large-scale longitudinal data sets are reviewed in this chapter. The Trends in International Mathematics and Science Study (TIMSS), the National Assessment of Educational Progress (NAEP), and the Educational Longitudinal Study of 2002 (ELS:2002) are compared and contrasted with respect to differences in sampling frame, internal and external validity, and especially construct validity of assessment items. Conclusions about the usefulness of large-scale secondary data analysis show that the reviewed assessments have been critical for determining inequities of opportunity for gender, ethnicity, socioeconomic status, and across national boundaries. They have also been useful for researchers examining the effectiveness of curricular policy on student learning. Moreover, some stakeholders have used the results as evidence that a nation’s future GDP is predicted by the outcome on TIMSS, and that students need more mathematical knowledge and skills to compete in a world that has an ever increasing rate of technological expansion. Though longitudinal, the duration of the studies presents a problem, as none follow students’ mathematical abilities or development for any length of time (e.g., early childhood into adulthood), and few studies from large-scale assessments shed light onto the kinds of pedagogy or curricular tasks that positively impact student learning. Lastly, threats to validity for large-scale studies are critiqued, and shown to be underreported in the literature.

[1]  Laura M. Desimone,et al.  What Makes Professional Development Effective? Results From a National Sample of Teachers , 2001 .

[2]  F. C. Hemphill,et al.  Achievement Gaps: How Hispanic and White Students in Public Schools Perform in Mathematics and Reading on the National Assessment of Educational Progress. Statistical Analysis Report. NCES 2011-459. , 2011 .

[3]  George Lakoff,et al.  Women, Fire, and Dangerous Things , 1987 .

[4]  C. Gamble,et al.  Statistical analysis report , 2016 .

[5]  V. B. Griffo.  Examining NAEP: The Effect of Item Format on Struggling 4th Graders' Reading Comprehension , 2011 .

[6]  Christine Y. O'Sullivan,et al.  TIMSS 2011 Assessment Frameworks. , 2009 .

[7]  Xin Wei.  Are More Stringent NCLB State Accountability Systems Associated With Better Student Outcomes? An Analysis of NAEP Results Across States , 2012 .

[8]  William H. Schmidt,et al.  According to the Book , 2002 .

[9]  Gilbert A. Valverde.  According to the Book: Using TIMSS to investigate the translation of policy into practice through the world of textbooks , 2002 .

[10]  Alexander W. Wiseman.  Introduction: The Advantages and Disadvantages of National Education Policymaking Informed by International Achievement Studies , 2010 .

[11]  Leland S. Cogan,et al.  “Culture Shock” – Eighth-Grade Mathematics From an International Perspective , 2002 .

[12]  Joseph L. Devitis.  Chapter Twenty-Seven: David C. Berliner and Bruce J. Biddle, The Manufactured Crisis: Myths, Fraud, and the Attack on America’s Public Schools (1995) , 2016 .

[13]  T. Howard.  Why Race and Culture Matter in Schools: Closing the Achievement Gap in America's Classrooms (Multicultural Education Series) , 2010 .

[14]  Haggai Kupermintz.  Enhancing the Validity and Usefulness of Large-Scale Educational Assessments: III. , 2016 .

[15]  Katherine Ariemma.  Lessons Learned: What International Assessments Tell Us about Math Achievement , 2012 .

[16]  James A. Middleton,et al.  Large-scale studies in mathematics education , 2015 .

[17]  W. Shadish,et al.  Experimental and Quasi-Experimental Designs for Generalized Causal Inference , 2001 .

[18]  Alternative Approaches to Setting Performance Standards for the National Assessment of Educational Progress (NAEP). , 2012 .

[19]  Accessing and Analyzing National Databases , 2008 .

[20]  Lawrence C. Stedman.  Respecting the Evidence , 1996 .

[21]  Robert Bozick,et al.  Education Longitudinal Study of 2002 (ELS:2002): A First Look at the Initial Postsecondary Experiences of the High School Sophomore Class of 2002. , 2007 .

[22]  A. Mji,et al.  Alignment between South African mathematics assessment standards and the TIMSS assessment frameworks : original research , 2012 .

[23]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[24]  Berchie W. Holliday,et al.  Why Using International Comparative Math and Science Achievement Data from TIMSS Is Not Helpful , 2003 .

[25]  Gary N. Marks,et al.  Explaining socioeconomic inequalities in student achievement: The role of home and school factors , 2006 .

[26]  David C. Berliner,et al.  The Manufactured Crisis , 1995 .

[27]  Anthony Lutkus,et al.  The Nation's Report Card. NAEP 2004 Trends in Academic Progress: Three Decades of Student Performance in Reading, 1971-2004 and Mathematics, 1973-2004. NCES 2005?464. , 2005 .

[28]  Gerald W. Bracey.  The TIMSS “Final Year” Study and Report: A Critique , 2000 .

[29]  Haggai Kupermintz,et al.  Enhancing the Validity and Usefulness of Large-Scale Educational Assessments: III. NELS: 88 Mathematics Achievement to 12th Grade , 1997 .

[30]  D. Berliner,et al.  Making Molehills Out of Molehills: Reply to Lawrence Stedman's Review of The Manufactured Crisis , 1996 .

[31]  D. Macnab.  Raising standards in mathematics education: values, vision, and TIMSS , 2000 .

[32]  E. Rosch,et al.  Categorization of Natural Objects , 1981 .

[33]  G Pink.  Heads in the sand. , 1994, Nursing standard (Royal College of Nursing (Great Britain) : 1987).

[34]  T. Kowalski,et al.  Handbook of Data-Based Decision Making in Education , 2008 .

[35]  V. Lee.  Understanding High School Restructuring Effects on the Equitable Distribution of Learning in Mathematics and Science. Revised. , 1996 .

[36]  Pascal D. Forgione.  Responses to Frequently Asked Questions about 12th-Grade TIMSS , 1998 .

[37]  J. Stigler,et al.  A Proposal for Improving Classroom Teaching: Lessons from the TIMSS Video Study , 2000, The Elementary School Journal.

[38]  Trends in mathematics and science performance in 18 countries: Multiple regression analysis of the cohort effects of TIMSS 1995-2007 , 2012 .

[39]  Ian Westbury.  A Nation at Risk , 1984 .

[40]  Gerald W. Bracey.  The Seventh Bracey Report on the Condition of Public Education. , 1997 .

[41]  Lesia Lennex,et al.  A Comparison of Calculator Use in Eighth‐Grade Mathematics Classrooms in the United States, Japan, and Portugal: Results From the Third International Mathematics and Science Study , 2000 .

[42]  Jianjun Wang,et al.  TIMSS Primary and Middle School Data: Some Technical Concerns , 2001 .

[43]  B. Efron,et al.  The Jackknife Estimate of Variance , 1981 .