Validity Evidence for the SUCCESS Survey: Measuring Non-Cognitive and Affective Traits of Engineering and Computing Students

This research paper examines the validity evidence from an exploratory factor analysis for a pilot of the SUCCESS survey (Studying Underlying Characteristics for Computing and Engineering Student Success). This survey was developed to measure underlying factors that may influence student success including personality, community, grit, thriving, identity, mindset, motivation, perceptions of faculty caring, stress, gratitude, self-control, mindfulness, and belongingness. We measure these underlying factors because each engineering and computing student admitted to a university has clear potential for academic and personal success in their undergraduate curriculum from admissions criteria, however, while some thrive academically, others flounder. In this project, we ask, “Why is it that highly credentialed and previously successful students do not see the same success in college?” We posit that some collection of characteristics— apparently not visible on their admission applications and perhaps not related to their talent or intelligence—is an important piece of the student performance puzzle. We developed a survey to measure various non-cognitive and affective factors that we believe are important for student achievement, academically, personally, and professionally. These non-cognitive and affective factors are representative of multifaceted aspects of undergraduate student success in prior literature. Each of the constructs we chose had validity evidence from prior studies, some within an engineering population. We piloted the survey across two different universities, one West Coast and one Midwest (n = 490), in Summer 2017. We used Exploratory Factor Analysis (EFA) to evaluate instrument performance to decide which items to include in the national release of the survey in Fall 2017. Our results provide preliminary validity evidence for items that measure various non-cognitive and affective factors. The wide-ranging constructs within the SUCCESS survey provide multiple pathways to understand students’ likelihood for success in engineering and computing. Our future work includes distributing this survey to over a dozen universities across the U.S., yielding a broad dataset of non-cognitive profiles of engineering and computing students broadly. In parallel, we will link these results with students’ registrar information at three study sites to develop predictive models for student success. Motivation for this study Engineering and computing education remains critical for U.S. workforce development and technological innovation now and into the future [1]–[3]. Many students recognize the importance and opportunity associated with studying STEM majors, and engineering and computing programs today have a talented applicant pool [4]. As a consequence, many institutions see relatively uniform and strong applicant credentials in terms of high school GPA, standardized test scores, and leadership experiences [5]. Each admitted student has the clear potential for academic success in the undergraduate curriculum. However, while some thrive at the university, many languish near the middle of the pack, or worse, they struggle academically. We want to know why highly credentialed and previously successful students sometimes do not see the same success in college. We posit that there are characteristics—apparently not visible on their admission applications—of such students that may make them more likely to navigate successfully the difficult pathways in their engineering and computing programs. We believe that an important piece of the student performance puzzle lies in the collection of non-cognitive and affective (NCA) factors including grit, study habits, personality, feelings of belongingness, and a sense of engineering or computing identity. To decide on these factors, and others, we engaged in an extensive process including a literature review, prioritization based on interests, and constructs with existing measurements to decide on this set of NCA factors. Each of the factors that we included in our pilot survey consisted of items used in other studies and are described in detail below. Differences in these traits, not asked on admissions materials and perhaps formed through the college experience, may explain particular reasons why some students thrive while others struggle. This project begins to answer the call from National Academy of Sciences [6] to see how interpersonal and intrapersonal factors contribute to student success, by focusing on how NCA factors influence student performance. The purpose of this paper is to introduce the SUCCESS project survey and to describe how we used a pilot study and exploratory factor analysis (EFA) as part of a decision-making process to evaluate items for inclusion on the national survey. In this process, we assume that our sample is reflective of our future national sample and do not anticipate our models to significantly improve [7] or become worse [8]. The results of this work also suggest how items used to measure NCA factors may be useful in future studies.