Measuring Student Content Knowledge, iSTEM, Self Efficacy, and Engagement through a Long-Term Engineering Design Intervention

The current study reports on the outcomes of a classroom-based long-term engineering design intervention intended to increase high school students’ perceptions of the integrated nature of STEM disciplines (iSTEM) and to assess the effect of the intervention on student participation in an extracurricular STEM activity (i.e., a research poster symposium). Cross-disciplinary teams of students (n=373) from high school mathematics, science, and engineering classrooms completed engineering design challenges. The results indicated that, consistent with our predictions, the intervention exhibited a positive impact on students that began the study with the lowest iSTEM scores. Furthermore, the classroom environment mattered. While no individual scores (i.e., posttest iSTEM scores) were predictive of participation in the poster symposium, the collective scores were (i.e., mean classroom iSTEM scores). Four measures were used in this study; Content knowledge quiz. Student content knowledge was assessed with a teacher made nine-item multiple-choice quiz; self-efficacy, task specific selfefficacy was assessed through a nine item measure; iSTEM perceptions. Participants responded to a nine-item iSTEM scale developed and validated by the authors in a previous study, to measure student perceptions of the interconnections between mathematics, science, and engineering; and STEM clubs. Participants responded “Yes” (1) or “No” (0) to the question regarding their involvement in extracurricular STEM club. Hierarchical linear modeling (HLM) was used in this analysis because it distinguishes variability in scores at the student-level (i.e., level-1) from variability in scores at the classroom level (i.e., level-2), which results in correctly estimating standard error. Therefore, HLM was used to conduct multilevel-paired sample t-tests. Further, all analyses were conducted with Restricted Maximum Likelihood estimation. The results indicated that, consistent with our predictions, the intervention exhibited a positive impact on students that began the study with the lowest iSTEM scores. Furthermore, the classroom environment mattered. While no individual scores (i.e., posttest iSTEM scores) were predictive of participation in the poster symposium, the collective scores were (i.e., mean classroom iSTEM scores). Introduction Many science, technology, engineering and mathematics (STEM) concepts, especially those learned in the critical formative years of pre-collegiate education are abstract in nature, often taught in vertically articulated course offerings that are frequently unconnected horizontally with other STEM course content. The lack of concept and content connections to authentic applications makes learning difficult for young learners. In addition, few opportunities exist within K-12 education for students to apply STEM learning in contextually authentic learning-indoing inquiry and design driven environments in which they are immersed over time greater than a few class periods. Combine these factors with student misconceptions of what engineering practice is and less than optimal instructional models yields a volatile combination for student attrition and low perceived value for learning STEM subjects. The aversion to learning basic STEM concepts due to their high abstractness, low perceived value, utility, and disconnection from applications has triggered a decrease in confidence in STEM learning among entering college students. This can be illustrated by the fact that enrollment in U.S. institutions of higher education has grown steadily at all levels rising from 14.5 million students in 1994 to 20.7 million in 2009, but such a growth is not fully reflected in science and engineering. Institutions of higher education in the United States granted engineering degrees in the mid-2000s at a lower rate than in the mid-1980s. The number of American students earning bachelor’s degrees increased by 16% over the past 10 years, however, the number of bachelor’s degrees earned in engineering decreased by 15%. Nationally, less than 50% of the students who enrolled in engineering curriculum complete the program. American Universities typically lose 50% of engineering freshmen and sophomore during the first two years of their engineering program. This trend is continuing in the foreseeable future and it can be attributed to (at least) several factors:  The traditional teaching of math, physics, and engineering concepts are isolated. Each discipline operates within its own silo. Students do not see the relationship of what is taught to what they are interested in learning.  Early engineering students fail to identify with and become part of the engineering community through practice, inclusion, and engagement.  Only small populations of high school students find themselves attracted to engineering schools and have never experienced doing research or engineering design. Addressing these significant factors in the learning of STEM and especially in coming to know, experience, and integrate engineering practices as part of the STEM learning continuum is becoming an imperative that pre-collegiate education must address. However, challenges exist when a shift in paradigmatic approach to learning and instruction is introduced to a wellestablished educational system. Shifting approaches to STEM education The recent release of the Next Generation Science Standards (NGSS) marks a significant shift in the core concepts and approaches guiding science, technology, engineering, and mathematics education content in the coming years. Most notable is the inclusion of engineering and technology concepts in a framework that emphasizes practices, crosscutting concepts, and core ideas. The repositioning of engineering and technology content within science education brings to light new opportunities and challenges when conceptualizing the design and delivery of instruction in STEM subjects. Moreover, realizing the full potential of the NGSS will require new conceptions of learning and instruction being adopted to include the richness of unifying practice, inquiry, and design across STEM concepts and contexts. The NGSS articulates a broad set of expectations for students in science grounded in practices and inquiry. Within these guiding standards are three major dimensions around which grades K12 science education needs to be integrated into standards, curriculum, instruction, and assessment. These dimensions include: scientific and engineering practices; crosscutting concepts that unify the study of science and engineering through their common application across fields; and core ideas in four disciplinary areas: physical sciences, life sciences, earth and space sciences, and engineering, technology, and applications of science. Integrating the three dimensions of scientific and engineering practice, crosscutting concepts, and disciplinary core ideas that cover traditional scientific fields of study (i.e. physical science, life science, and earth and space science) now includes the addition of engineering, technology, and applications of science. Integrating the three dimensions could prove illusive, however approaches informed by research on teaching and learning from cognitive sciences combined with aggressive methodological approaches to measuring student learning within the three dimensions can yield promising results. John Bruer 6 in his seminal book Schools for Thought argued that, ‘the National Assessment of Educational Progress (NAPE; often referred to as the Nations report card) results indicate that current curricula, teaching methods and instructional materials successfully impart facts and rote skills to most students but fail to impart high-order reasoning and learning skills’ (p. 5). This statement continues to resonate today as it did in 1993. Other researchers have explored transforming the classroom from “work sites where students perform assigned tasks under management of teachers into communities of learning and interpretation, where students are given significant opportunity to take charge of their own learning...attempting to engineer an innovative educational environment”. p.141 Grounding of the intervention design STEM content knowledge. The conceptualization and design of this study is informed by two perspectives; the first influenced from well researched areas of teaching and learning from the cognitive sciences; and second from the newly released NGSS. The consonance between models of classroom learning and teaching informed by research from the cognitive sciences and the new frameworks vision to actively engage students in scientific and engineering practices and apply crosscutting concepts to deepen their understanding of the core ideas in these fields. In addition, the new vision sets a goal of students gaining sufficient knowledge of the practices, crosscutting concepts, and core ideas of science and engineering to engage in public discussions about these subjects and communicate their understanding of science and engineering practices. One approach to this transformation process can be addressed through adopting well-researched approaches to learning and instruction grounded in the cognitive sciences. An example of this has been evidenced with the emergence of engineering and technology education as an integral component of STEM education. For many years technology education alone struggled to establish itself as an equal partner in general education and often was unable to gain recognition for the value of its instruction. Often technology educators touted the effectiveness of their hands-on “making” programs based on anecdotal evidence gathered from their classroom experiences reflecting how their instructional methods empower students to learn. Today’s engineering and technology education originated without any meaningful input from cognitive science research. However, it appears that engineering and technology education pra