One Size Doesn’t Fit All: Using Factor Analysis to Gather Validity Evidence When Using Surveys in Your Research

Across all sciences, the quality of measurements is important. Survey measurements are only appropriate for use when researchers have validity evidence within their particular context. Yet, this step is frequently skipped or is not reported in educational research. This article briefly reviews the aspects of validity that researchers should consider when using surveys. It then focuses on factor analysis, a statistical method that can be used to collect an important type of validity evidence. Factor analysis helps researchers explore or confirm the relationships between survey items and identify the total number of dimensions represented on the survey. The essential steps to conduct and interpret a factor analysis are described. This use of factor analysis is illustrated throughout by a validation of Diekman and colleagues’ goal endorsement instrument for use with first-year undergraduate science, technology, engineering, and mathematics students. We provide example data, annotated code, and output for analyses in R, an open-source programming language and software environment for statistical computing. For education researchers using surveys, understanding the theoretical and statistical underpinnings of survey validity is fundamental for implementing rigorous education research.

[1]  R. Cattell.  The Scree Test For The Number Of Factors. , 1966, Multivariate behavioral research.

[2]  Todd D. Reeves,et al.  Contemporary Test Validity in Theory and Practice: A Primer for Discipline-Based Education Researchers , 2016, CBE life sciences education.

[3]  S. Finney.  Nonnormal and categorical data in structural equation modeling , 2013 .

[4]  Stephanie A. Fryberg,et al.  Unseen disadvantage: how American universities' focus on independence undermines the academic performance of first-generation college students. , 2012, Journal of personality and social psychology.

[5]  S. Finney,et al.  Factor Analysis , 2018, The Reviewer’s Guide to Quantitative Methods in the Social Sciences.

[6]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[7]  Sarah L. Eddy,et al.  Caution, Student Experience May Vary: Social Identities Impact a Student’s Experience in Peer Discussions , 2015, CBE life sciences education.

[8]  Steven W Rissing,et al.  Can an inquiry approach improve college student learning in a teaching laboratory? , 2009, CBE life sciences education.

[9]  R. Kline.  Principles and practice of structural equation modeling, 4th ed. , 2016 .

[10]  L. Crocker,et al.  Introduction to Classical and Modern Test Theory , 1986 .

[11]  John L. Perry,et al.  Assessing Model Fit: Caveats and Recommendations for Confirmatory Factor Analysis and Exploratory Structural Equation Modeling , 2015 .

[12]  P. Bentler,et al.  Cutoff criteria for fit indexes in covariance structure analysis : Conventional criteria versus new alternatives , 1999 .

[13]  Robert W. Lissitz,et al.  A Suggested Change in Terminology and Emphasis Regarding Validity and Education , 2007 .

[14]  E. Dolan,et al.  Do Biology Students Really Hate Math? Empirical Insights into Undergraduate Life Science Majors’ Emotions about Mathematics , 2017, CBE life sciences education.

[15]  B. Price.  A First Course in Factor Analysis , 1993 .

[16]  M. Kane.  Explicating validity , 2016 .

[17]  Tenko Raykov,et al.  Thanks Coefficient Alpha, We Still Need You! , 2019, Educational and psychological measurement.

[18]  Pedro M. Valero-Mora,et al.  Determining the Number of Factors to Retain in EFA: An easy-to-use computer program for carrying out Parallel Analysis , 2007 .

[19]  Samuel B. Green,et al.  Evaluation of Parallel Analysis Methods for Determining the Number of Factors , 2010 .

[20]  C. Distefano,et al.  Understanding and Using Factor Scores: Considerations for the Applied Researcher , 2009 .

[21]  Yves Rosseel,et al.  lavaan: An R Package for Structural Equation Modeling , 2012 .

[22]  Amanda B. Diekman,et al.  Seeking Congruity Between Goals and Roles , 2010, Psychological science.

[23]  Christina Moyer,et al.  Giving back or giving up: Native American student experiences in science and engineering. , 2014, Cultural diversity & ethnic minority psychology.

[24]  G. Cizek.  Validating test score meaning and defending test score use: different aims, different methods , 2016 .

[25]  J. Eccles,et al.  The development of achievement task values: A theoretical analysis , 1992 .

[26]  Patrick Ian Armstrong,et al.  Men and things, women and people: a meta-analysis of sex differences in interests. , 2009, Psychological bulletin.

[27]  W. Revelle.  psych: Procedures for Personality and Psychological Research , 2017 .

[28]  Sarah L. Eddy,et al.  Getting Under the Hood: How and for Whom Does Increasing Course Structure Work? , 2014, CBE life sciences education.

[29]  William A. Mehrens,et al.  The Consequences of Consequential Validity. , 2005 .

[30]  D. Watson,et al.  Constructing validity: Basic issues in objective scale development , 1995 .

[31]  Sarah L. Eddy,et al.  ASPECT: A Survey to Assess Student Perspective of Engagement in an Active-Learning Classroom , 2017, CBE life sciences education.

[32]  K. Slaney.  Construct Validity: Developments and Debates , 2017 .

[33]  T. Eckes,et al.  Social Role Theory of Sex Differences and Similarities : A Current Appraisal , 2012 .

[34]  S. Green,et al.  Coefficient Alpha: A Reliability Coefficient for the 21st Century? , 2011 .

[35]  Ryan J. Gagnon,et al.  Measurement Theory and Applications for the Social Sciences , 2019, Measurement: Interdisciplinary Research and Perspectives.

[36]  S. Messick.  Validity of Psychological Assessment: Validation of Inferences from Persons' Responses and Performances as Scientific Inquiry into Score Meaning. Research Report RR-94-45. , 1994 .

[37]  An Gie Yong,et al.  A Beginner's Guide to Factor Analysis: Focusing on Exploratory Factor Analysis , 2013 .

[38]  M. Tavakol,et al.  Making sense of Cronbach's alpha , 2011, International journal of medical education.

[39]  Deborah A. Prentice,et al.  What Women and Men Should Be, Shouldn't be, are Allowed to be, and don't Have to Be: The Contents of Prescriptive Gender Stereotypes , 2002 .

[40]  Howard B. Lee,et al.  A First Course in Factor Analysis 2nd Ed , 1973 .

[41]  S. E. Andrews,et al.  The Math–Biology Values Instrument: Development of a Tool to Measure Life Science Majors’ Task Values of Using Math in the Context of Biology , 2017, CBE life sciences education.

[42]  J. Calder.  Survey research methods , 1998, Medical education.

[43]  B. Tabachnick,et al.  Using multivariate statistics, 5th ed. , 2007 .

[44]  Klaas Sijtsma,et al.  On the Use, the Misuse, and the Very Limited Usefulness of Cronbach’s Alpha , 2008, Psychometrika.

[45]  Peter Armbruster,et al.  Active learning and student-centered pedagogy improve student attitudes and performance in introductory biology. , 2009, CBE life sciences education.

[46]  D. Bakan.  The duality of human existence : an essay on psychology and religion , 1966 .

[47]  M. Matsunaga.  How to factor-analyze your data right: do’s, don’ts, and how-to’s. , 2010 .

[48]  Jessi L. Smith,et al.  To Grab and To Hold: Cultivating communal goals to overcome cultural and structural barriers in first generation college students' science interest. , 2015, Translational issues in psychological science.

[49]  Karee E. Dunn,et al.  A Critical Review of Research on Formative Assessment: The Limited Scientific Evidence of the Impact of Formative Assessment in Education , 2009 .

[50]  Daniel M. McNeish,et al.  Psychological Methods Thanks Coefficient Alpha , We ’ ll Take It From Here , 2022 .

[51]  S. Messick.  Validity of Psychological Assessment: Validation of Inferences from Persons' Responses and Performances as Scientific Inquiry into Score Meaning. Research Report RR-94-45. , 1994 .

[52]  P. Gagné,et al.  Measurement Model Quality, Sample Size, and Solution Propriety in Confirmatory Factor Models , 2006, Multivariate behavioral research.

[53]  Tiffany A. Whittaker,et al.  Scale Development Research , 2006 .

[54]  N. Schmitt.  Uses and abuses of coefficient alpha. , 1996 .

[55]  M. Miller,et al.  Sample Size Requirements for Structural Equation Models , 2013, Educational and psychological measurement.

[56]  G. A. Marcoulides,et al.  An Introduction to Applied Multivariate Analysis , 2008 .

[57]  S A Mulaik,et al.  A Brief History of the Philosophical Foundations of Exploratory Factor Analysis. , 1987, Multivariate behavioral research.

[58]  Samuel B. Green,et al.  Commentary on Coefficient Alpha: A Cautionary Tale , 2009 .

[59]  D. Borsboom,et al.  The concept of validity. , 2004, Psychological review.

[60]  Diana Adler,et al.  Using Multivariate Statistics , 2016 .

[61]  L. Cronbach,et al.  Construct validity in psychological tests. , 1955, Psychological bulletin.

[62]  Richard L. Gorsuch.  Exploratory Factor Analysis , 1988 .

[63]  Laura M. Stapleton,et al.  The Reviewer’s Guide to Quantitative Methods in the Social Sciences , 2018 .

[64]  Theodore A. Walls,et al.  Non-Graphical Solutions for Cattell’s Scree Test , 2013 .

[65]  J. Ruscio,et al.  Determining the number of factors to retain in an exploratory factor analysis using comparison data of known factorial structure. , 2012, Psychological assessment.

[66]  H. Marsh,et al.  In Search of Golden Rules: Comment on Hypothesis-Testing Approaches to Setting Cutoff Values for Fit Indexes and Dangers in Overgeneralizing Hu and Bentler's (1999) Findings , 2004 .