Necessary steps in factor analysis: Enhancing validation studies of educational instruments. The PHEEM applied to clerks as an example

Background: The validation of educational instruments, in particular the employment of factor analysis, can be improved in many instances. Aims: To demonstrate the superiority of a sophisticated method of factor analysis, implying an integration of recommendations described in the factor analysis literature, over often employed limited applications of factor analysis. We demonstrate the essential steps, focusing on the Postgraduate Hospital Educational Environment Measure (PHEEM). Method: The PHEEM was completed by 279 clerks. We performed Principal Component Analysis (PCA) with varimax rotation. A combination of three psychometric criteria was applied: scree plot, eigenvalues >1.5 and a minimum percentage of additionally explained variance of approximately 5%. Furthermore, four interpretability criteria were used. Confirmatory factor analysis was performed to verify the original scale structure. Results: Our method yielded three interpretable and practically useful dimensions: learning content and coaching, beneficial affective climate and external regulation. Additionally, combining several criteria reduced the risk of overfactoring and underfactoring. Furthermore, the resulting dimensions corresponded with three learning functions essential to high-quality learning, thus strengthening our findings. Confirmatory factor analysis disproved the original scale structure. Conclusions: Our sophisticated approach yielded several advantages over methods applied in previous validation studies. Therefore, we recommend this method in validation studies to achieve best practice.

[1]  S. Roff,et al.  Educational environment, student approaches to learning and academic achievement in a Thai nursing school , 2000 .

[2]  D. Sobral.  Medical students' self-appraisal of first-year learning outcomes: use of the course valuing inventory , 2004, Medical teacher.

[3]  Elazar J. Pedhazur,et al.  Measurement, Design, and Analysis: An Integrated Approach , 1994 .

[4]  W F Velicer,et al.  Component Analysis versus Common Factor Analysis: Some issues in Selecting an Appropriate Procedure. , 1990, Multivariate behavioral research.

[5]  Andy Field,et al.  Discovering statistics using SPSS, 2nd ed. , 2005 .

[6]  C. Heckler.  A Step-by-Step Approach to Using the SAS™ System for Factor Analysis and Structural Equation Modeling , 1996 .

[7]  H. Kiers,et al.  The Empirical Verification of an Assignment of Items to Subtests , 2008 .

[8]  J. Jaccard,et al.  LISREL Approaches to Interaction Effects in Multiple Regression , 1998 .

[9]  T. Bonnesen,et al.  Validation of the PHEEM instrument in a Danish hospital setting , 2007, Medical teacher.

[10]  J. Vermunt.  Metacognitive, cognitive and affective aspects of learning styles and strategies: A phenomenographic analysis , 1996 .

[11]  S. Roff,et al.  Assessing the educational environment in the operating room-a measure of resident perception at one Canadian institution. , 2006, Surgery.

[12]  D. Wall,et al.  Educational environment in intensive care medicine—use of Postgraduate Hospital Educational Environment Measure (PHEEM) , 2007, Medical teacher.

[13]  S. Roff,et al.  Development and validation of an instrument to measure the postgraduate clinical learning and teaching educational environment for hospital-based junior doctors in the UK , 2005, Medical teacher.

[14]  Duane T. Wegener,et al.  Evaluating the use of exploratory factor analysis in psychological research. , 1999 .

[15]  S. Leinster,et al.  Medical students' perceptions of their educational environment: expected versus actual perceptions , 2007, Medical education.

[16]  R. Darlington,et al.  Factor Analysis , 2008 .

[17]  Stefaan Decoene,et al.  Measurement, design, and analysis - an integrated approach - pedhazur,ej, schmelkin,lp , 1995 .

[18]  Richard L. Gorsuch,et al.  Effects of under- and overextraction on principal axis factor analysis with varimax rotation. , 1996 .

[19]  R. Henson,et al.  Use of Exploratory Factor Analysis in Published Research , 2006 .

[20]  R. Rummel.  Applied Factor Analysis , 1970 .

[21]  S. Roff,et al.  Development and validation of the Anaesthetic Theatre Educational Environment Measure (ATEEM) , 2004, Medical teacher.

[22]  S. Roff,et al.  Development and validation of the Dundee Ready Education Environment Measure (DREEM) , 1997 .

[23]  J. Parry,et al.  Hostile teaching hospitals and friendly district general hospitals: final year students' views on clinical attachment locations , 2002, Medical education.

[24]  T. Shuell.  The role of the student in learning from instruction , 1988 .

[25]  J. Nunnally.  Psychometric Theory (2nd ed), New York: McGraw-Hill. , 1978 .

[26]  Nick Lee,et al.  The evolution of “classical mythology” within marketing measure development , 2005 .

[27]  J. Vermunt,et al.  Congruence and friction between learning and teaching , 1999 .

[28]  K. M. Kieffer,et al.  An Introductory Primer on the Appropriate Use of Exploratory and Confirmatory Factor Analysis. , 1999 .

[29]  D. Wall,et al.  Can STEEM be used to measure the educational environment within the operating theatre for undergraduate medical students? , 2006, Medical teacher.

[30]  K. Sijtsma,et al.  Tilburg University Psychometric properties of an instrument to measure the clinical learning environment , 2017 .

[31]  Geert Kelchtermans,et al.  Disentangling clinical learning experiences: an exploratory study on the dynamic tensions in internship , 2006, Medical education.

[32]  R. Sitgreaves.  Psychometric theory (2nd ed.). , 1979 .

[33]  K. Cassar.  Development of an instrument to measure the surgical operating theatre learning environment as perceived by basic surgical trainees , 2004, Medical teacher.

[34]  J. Vieira,et al.  Psychometric properties of the Dundee Ready Educational Environment Measure (DREEM) applied to medical residents , 2005, Medical teacher.

[35]  Andy P. Field,et al.  Discovering Statistics Using SPSS , 2000 .