Statistical tests conducted with school environment data: The effect of teachers being clustered in schools

This article discusses the effect of clustering on statistical tests conducted with school environment data. Because most school environment studies involve the collection of data from teachers nested within schools, the hierarchical nature to these data cannot be ignored. In particular, this article considers the influence of intraschool correlations on tests of statistical significance conducted with the individual teacher as the unit of analysis. Theory that adjusts t test scores for nested data in two-group comparisons is presented and applied to school environment data. This article demonstrates that Type I error rates inflate greatly as the intraschool correlation increases. Because data analysis techniques that recognise the clustering of teachers in schools are essential, it is recommended that either multilevel analysis or adjustments to statistical parameters be undertaken in school environment studies involving nested data.

[1]  Dorian Wilson.  The interface of school climate and school connectedness and relationships with aggression and victimization. , 2004, The Journal of school health.

[2]  Barry J. Fraser,et al.  Classroom Learning Environments , 2003 .

[3]  M. Aitkin,et al.  Statistical Modelling Issues in School Effectiveness Studies , 1986 .

[4]  L. Hedges,et al.  Intraclass Correlation Values for Planning Group-Randomized Trials in Education , 2007 .

[5]  A. Mcevoy,et al.  Antisocial Behavior, Academic Failure, and School Climate , 2000 .

[6]  J. Vyskocil,et al.  Collective Bargaining and Supervision: A Matter of Climate. , 1979 .

[7]  J. Algina,et al.  Reconceptualizing the Debate on School Climate and Students' Academic Motivation and Achievement: A Multilevel Analysis. , 2000 .

[8]  K. Pepper,et al.  Making a Change: The Effects of the Leadership Role on School Climate , 2002 .

[9]  R. Moos.  Work environment scale manual , 1986 .

[10]  Dorothy L. Gabel,et al.  Handbook of Research on Science Teaching and Learning Project. , 1993 .

[11]  J. Levy.  Interpersonal Teacher Behavior and School Environment , 2005 .

[12]  B. Fraser,et al.  Use of WES‡ to assess science teachers’ perceptions of school environment , 1983 .

[13]  A. Kitsantas,et al.  Students’ Perceptions of School Safety: , 2004 .

[14]  M. Fish,et al.  Making Schools Safe , 2007 .

[15]  J. Gill.  Hierarchical Linear Models , 2005 .

[16]  Jeffrey P. Dorman,et al.  Relationship between school‐level and classroom‐level environments in secondary schools , 1997 .

[17]  Jeffrey P. Dorman.  School Environment Questionnaire: An Instrument Developed for Australian Catholic Secondary Schools , 1996 .

[18]  Jeffrey P. Dorman.  Use of Teacher Perceptions in School Environment Research. , 1996 .

[19]  Lindsay Paterson,et al.  Schools, Classrooms and Pupils , 1991 .

[20]  Kenneth Tobin,et al.  International handbook of science education , 1998 .

[21]  Joseph J. Stevens,et al.  Exploratory and Confirmatory Factor Analysis of the School Level Environment Questionnaire (SLEQ) , 2001 .

[22]  H. Alker.  A typology of ecological fallacies , 1969 .

[23]  B. Tabachnick,et al.  Using multivariate statistics, 5th ed. , 2007 .

[24]  Nathan B. Wood,et al.  Viewing the school environment through multiple lenses: In search of school-level variables tied to student achievement , 2006 .

[25]  Joseph M. Wisenbaker,et al.  Elementary School Social Climate and School Achievement1 , 1978 .

[26]  A. Loukas,et al.  Middle school student perceptions of school climate: Examining protective functions on subsequent adjustment problems. , 2007 .

[27]  Timothy J. Robinson,et al.  Multilevel Analysis: Techniques and Applications , 2002 .

[28]  James Noonan.  School Climate and the Safe School: Seven Contributing Factors. Safety in the Schools. , 2005 .

[29]  Norman G. Lederman,et al.  Handbook of Research on Science Education , 2023 .

[30]  B. Fraser.  Science Learning Environments: Assessment, Effects and Determinants , 1998 .

[31]  Lee J. Cronbach,et al.  Research on Classrooms and Schools: Formulation of Questions, Design and Analysis. , 1976 .

[32]  P. Brok,et al.  Multilevel issues in research using students’ perceptions of learning environments: The case of the Questionnaire on Teacher Interaction , 2006 .

[33]  Ian H. Langford,et al.  A User’s Guide to MLwiN, Version 2.10 , 2000 .

[34]  Stephen W. Raudenbush,et al.  Schools, Classrooms, and Pupils: International Studies of Schooling from a Multilevel Perspective , 1991 .

[35]  R. Plomin,et al.  A genetically sensitive investigation of the effects of the school environment and socio‐economic status on academic achievement in seven‐year‐olds , 2005 .

[36]  Barry J. Fraser,et al.  Relationships Between School-Level and Classroom-Level Environment. , 1982 .

[37]  Karissa D. Horton,et al.  Examining School Connectedness as a Mediator of School Climate Effects , 2006 .

[38]  Rosina Wright-Castro,et al.  Student Engagement, School Climate, and Future Expectations in High School. , 2003 .

[39]  B. Fraser,et al.  DEVELOPMENT OF A SCHOOL—LEVEL ENVIRONMENT QUESTIONNAIRE , 1983 .

[40]  Karen Elliot,et al.  But what does it mean? The use of effect sizes in educational research , 2004 .

[41]  W. Hoy,et al.  Secondary School Climate: A Revision of the OCDQ , 1987 .

[42]  B. Fraser,et al.  Differences in the Psychosocial Work Environment of Different Types of Schools , 1989 .

[43]  Larry V. Hedges,et al.  Correcting a Significance Test for Clustering , 2007 .

[44]  H. Goldstein.  Some observations on the definition and estimation of effect sizes , 2004 .

[45]  R. Skiba,et al.  Creating School Climates That Prevent School Violence , 2001 .

[46]  Valerie E. Lee,et al.  Using Hierarchical Linear Modeling to Study Social Contexts: The Case of School Effects , 2000 .

[47]  Joseph J. Stevens,et al.  Student achievement and elementary teachers’ perceptions of school climate , 2006 .

[48]  Bill Thornton,et al.  Relationships between Measures of Leadership and School Climate. , 2005 .

[49]  D. Fisher,et al.  Actual and Ideal Principal Interpersonal Behaviour , 1998 .

[50]  D. Fisher,et al.  A School Level Environment Study in Australia. , 1999 .

[51]  Mattei Dogan,et al.  Quantitative Ecological Analysis in the Social Sciences. , 1969 .

[52]  Jack Levy,et al.  Do You Know What You Look Like?: Interpersonal Relationships In Education , 1993 .

[53]  Harvey Goldstein,et al.  Partitioning variation in multilevel models , 2002 .

[54]  C. Anderson.  The Search for School Climate: A Review of the Research , 1982 .

[55]  Ibtesam Halawah,et al.  The Relationship between Effective Communication of High School Principal and School Climate. , 2005 .

[56]  Kathleen Vail.  Create Great School Climate. , 2005 .

[57]  Roel Bosker,et al.  Multilevel analysis : an introduction to basic and advanced multilevel modeling , 1999 .

[58]  Patrick Onghena,et al.  The Effects of Ignoring a Level in Multilevel Analysis , 2005 .

[59]  M. Ely.  The Management of Schools in New South Wales 1848–1880: Local Initiative Suppressed? , 1971 .