Creating an Instrument to Measure Student Response to Instructional Practices

Background Calls for the reform of education in science, technology, engineering, and mathematics (STEM) have inspired many instructional innovations, some research based. Yet adoption of such instruction has been slow. Research has suggested that students' response may significantly affect an instructor's willingness to adopt different types of instruction. Purpose We created the Student Response to Instructional Practices (StRIP) instrument to measure the effects of several variables on student response to instructional practices. We discuss the step-by-step process for creating this instrument. Design/Method The development process had six steps: item generation and construct development, validity testing, implementation, exploratory factor analysis, confirmatory factor analysis, and instrument modification and replication. We discuss pilot testing of the initial instrument, construct development, and validation using exploratory and confirmatory factor analyses. Results This process produced 47 items measuring three parts of our framework. Types of instruction separated into four factors (interactive, constructive, active, and passive); strategies for using in-class activities into two factors (explanation and facilitation); and student responses to instruction into five factors (value, positivity, participation, distraction, and evaluation). Conclusions We describe the design process and final results for our instrument, a useful tool for understanding the relationship between type of instruction and students' response.

[1]  Matthew T. Hora,et al.  Findings from classroom observations of 58 math and science faculty , 2012 .

[2]  James J. Appleton,et al.  Student engagement with school: Critical conceptual and methodological issues of the construct. , 2008 .

[3]  R. Eisinga,et al.  The reliability of a two-item scale: Pearson, Cronbach, or Spearman-Brown? , 2013, International Journal of Public Health.

[4]  K. Tanner,et al.  “What if students revolt?”—Considering Student Resistance: Origins, Options, and Opportunities for Investigation , 2013, CBE life sciences education.

[5]  T. Goetz,et al.  Academic Emotions in Students' Self-Regulated Learning and Achievement: A Program of Qualitative and Quantitative Research , 2002 .

[6]  Charles Teddlie,et al.  Mixed Methods Sampling A Typology With Examples , 2016 .

[7]  D. Jonassen,et al.  Exploring the Tensions of Problem‐Based Learning: Insights From Research , 2003 .

[8]  Michael J. Prince,et al.  Use of Research-Based Instructional Strategies in Core Chemical Engineering Courses. , 2013 .

[9]  Michael Fowler,et al.  Use of Engineering Design Competitions for Undergraduate and Capstone Projects. , 2009 .

[10]  Katherine A. Friedrich,et al.  9: Thawing the Chilly Climate: Inclusive Teaching Resources for Science, Technology, Engineering, and Math , 2008 .

[11]  Barry J. Fraser,et al.  Classroom Environment Instruments: Development, Validity and Applications , 1998 .

[12]  B. Zimmerman.  Self-Regulated Learning and Academic Achievement: An Overview , 1990 .

[13]  V. Lee,et al.  Effects of High School Restructuring and Size on Early Gains in Achievement and Engagement , 1995 .

[14]  B. Byrne.  Structural equation modeling with EQS : basic concepts, applications, and programming , 2000 .

[15]  D. Betsy McCoach,et al.  Development of an Instrument to Measure Perspectives of Engineering Education Among College Students , 2008 .

[16]  Michael J. Prince,et al.  Inductive Teaching and Learning Methods: Definitions, Comparisons, and Research Bases , 2006 .

[17]  B. Tabachnick,et al.  Using Multivariate Statistics , 1983 .

[18]  David W. Johnson,et al.  Active Learning: Cooperation in the College Classroom , 2006 .

[19]  S. Haynes,et al.  Content validity in psychological assessment: A functional approach to concepts and methods. , 1995 .

[20]  Maryellen Weimer,et al.  Learner-Centered Teaching: Five Key Changes to Practice , 2013 .

[21]  Arianne M. Dantas,et al.  Promoting student-centered active learning in lectures with a personal response system. , 2009, Advances in physiology education.

[22]  E. Skinner,et al.  Motivation in the classroom: Reciprocal effects of teacher behavior and student engagement across the school year. , 1993 .

[23]  Prateek Shekhar,et al.  A mixed methods study to understand student resistance to active learning , 2015 .

[24]  Valerie J. Caracelli,et al.  Toward a Conceptual Framework for Mixed-Method Evaluation Designs , 1989 .

[25]  Shanna R. Daly,et al.  Bridging the Research‐to‐Practice Gap: Designing an Institutional Change Plan Using Local Evidence , 2014 .

[26]  Melissa H. Dancy,et al.  Barriers to the use of research-based instructional strategies: The influence of both individual and situational characteristics , 2007 .

[27]  Richard M. Felder,et al.  Best Practices Involving Teamwork in the Classroom: Results From a Survey of 6435 Engineering Student Respondents , 2007, IEEE Transactions on Education.

[28]  Maura Borrego,et al.  Diffusion of Engineering Education Innovations: A Survey of Awareness and Adoption Rates in U.S. Engineering Departments , 2010 .

[29]  Hyun Kyoung Ro,et al.  Validity of the Contextual Competence Scale for Engineering Students , 2015 .

[30]  Tapabrata Maiti,et al.  Principles and Practice of Structural Equation Modeling (2nd ed.) , 2006 .

[31]  Timothy R. Hinkin,et al.  A Brief Tutorial on the Development of Measures for Use in Survey Questionnaires , 1998 .

[32]  R. Fassinger.  Use of Structural Equation Modeling in Counseling Psychology Research. , 1987 .

[33]  Kim A. Stewart,et al.  Lessons from the Best and Worst Student Team Experiences: How a Teacher can make the Difference , 1999 .

[34]  Markus A. Maier,et al.  Achievement Goals and Achievement Emotions: Testing a Model of Their Joint Relations with Academic Performance. , 2009 .

[35]  D. Stipek,et al.  Good Instruction Is Motivating , 2002 .

[36]  K. Powell,et al.  Cognitive and Social Constructivism: Developing Tools for an Effective Classroom. , 2009 .

[37]  George D. Kuh.  High-Impact Educational Practices: What They Are, Who Has Access to Them, and Why They Matter , 2008 .

[38]  Michael J. Prince,et al.  Does Active Learning Work? A Review of the Research , 2004 .

[39]  Judith A. Ouimet,et al.  Using Focus Groups, Expert Advice, and Cognitive Interviews to Establish the Validity of a College Student Survey , 2004 .

[40]  Susan K. Donohue,et al.  Factors affecting student attitudes toward active learning activities in a graduate engineering statistics course , 2009, 2009 39th IEEE Frontiers in Education Conference.

[41]  H. Kaiser.  The varimax criterion for analytic rotation in factor analysis , 1958 .

[42]  C. Fornell,et al.  Evaluating structural equation models with unobservable variables and measurement error. , 1981 .

[43]  Lisa Linnenbrink-Garcia,et al.  Academic Emotions and Student Engagement , 2012 .

[44]  Jeremy D. Finn,et al.  Measuring Participation among Elementary Grade Students , 1991 .

[45]  Do they see it coming , 2008 .

[46]  R. Haase,et al.  Advanced Applications of Structural Equation Modeling in Counseling Psychology Research , 2006 .

[47]  David Lake,et al.  Student performance and perceptions of a lecture-based course compared with the same course utilizing group discussion. , 2001, Physical therapy.

[48]  Kerrie A. Douglas,et al.  Validity: Meaning and Relevancy in Assessment for Engineering Education Research , 2015 .

[49]  Tiffany A. Whittaker,et al.  Scale Development Research , 2006 .

[50]  Michael T. Brannick,et al.  Construction of a Job in General scale: A comparison of global, composite, and specific measures. , 1989 .

[51]  Thomas A. Angelo,et al.  Classroom Assessment Techniques: A Handbook for College Teachers. Second Edition. , 1993 .

[52]  R. Bagozzi,et al.  On the evaluation of structural equation models , 1988 .

[53]  P. Pintrich,et al.  Motivational and self-regulated learning components of classroom academic performance. , 1990 .

[54]  S. Reise,et al.  Factor analysis and scale revision. , 2000, Psychological assessment.

[55]  Rex B. Kline,et al.  Principles and Practice of Structural Equation Modeling , 1998 .

[56]  Melissa H. Dancy,et al.  Experiences of new faculty implementing research-based instructional strategies , 2012 .

[57]  B. Thompson.  Exploratory and Confirmatory Factor Analysis: Understanding Concepts and Applications , 2004 .

[58]  J. P. Mohsen,et al.  Creating a Culture for Scholarly and Systematic Innovation in Engineering Education , 2009 .

[59]  Katharine Semsar,et al.  How Not to Lose Your Students with Concept Maps. , 2011 .

[60]  K. Lynch,et al.  Group Project Work and Student-centred Active Learning: Two different experiences , 2000 .

[61]  Michelene T. H. Chi,et al.  Active-Constructive-Interactive: A Conceptual Framework for Differentiating Learning Activities , 2009, Top. Cogn. Sci..

[62]  Aman Yadav,et al.  Problem‐based Learning: Influence on Students' Learning in an Electrical Engineering Course , 2011 .

[63]  Hee-Sun Lee,et al.  Measuring Engineering Design Self‐Efficacy , 2010 .

[64]  Rebecca Brent,et al.  ACTIVE LEARNING: AN INTRODUCTION * , 2009 .

[65]  E. Skinner,et al.  Developmental Dynamics of Student Engagement, Coping, and Everyday Resilience , 2012 .

[66]  Michelle K. Smith,et al.  Active learning increases student performance in science, engineering, and mathematics , 2014, Proceedings of the National Academy of Sciences.

[67]  Joel A. Michael,et al.  Faculty Perceptions About Barriers to Active Learning , 2007 .

[68]  P. Bentler,et al.  Cutoff criteria for fit indexes in covariance structure analysis : Conventional criteria versus new alternatives , 1999 .

[69]  Jeffrey E. Froyd,et al.  Estimates of Use of Research-Based Instructional Strategies in Core Electrical or Computer Engineering Courses , 2013, IEEE Transactions on Education.

[70]  E. McColl.  Cognitive Interviewing. A Tool for Improving Questionnaire Design , 2006, Quality of Life Research.

[71]  Jennifer A. Fredricks,et al.  School Engagement: Potential of the Concept, State of the Evidence , 2004 .

[72]  Diane Ebert-May,et al.  Scientific Teaching , 2004, Science.

[73]  J. Armstrong,et al.  Are Student Ratings of Instruction Useful? , 2005 .

[74]  Rolph E. Anderson,et al.  Multivariate Data Analysis: Text and Readings , 1979 .

[75]  Robert J. Beichner,et al.  Do They See It Coming? Using Expectancy Violation to Gauge the Success of Pedagogical Reforms. , 2010 .

[76]  S. Christenson,et al.  Handbook of Research on Student Engagement , 2012 .

[77]  J. Connell,et al.  Competence, autonomy, and relatedness: A motivational analysis of self-system processes. , 1991 .

[78]  Susan R. Singer,et al.  Discipline-Based Education Research: Understanding and Improving Learning in Undergraduate Science and Engineering. , 2012 .

[79]  Prateek Shekhar,et al.  Development of an observation protocol to study undergraduate engineering student resistance to active learning , 2015 .

[80]  Hal A. Lawson,et al.  New Conceptual Frameworks for Student Engagement Research, Policy, and Practice , 2013 .

[81]  Howard B. Lee,et al.  A First Course in Factor Analysis 2nd Ed , 1973 .

[82]  D. Rubin.  INFERENCE AND MISSING DATA , 1975 .

[83]  M. Chi,et al.  The ICAP Framework: Linking Cognitive Engagement to Active Learning Outcomes , 2014 .