Normative Typologies Of Epics Students On Abet Ec Criterion 3: A Multistage Cluster Analysis

Using state-of-the-art profile/cluster analysis technique, this study aimed to derive normative profiles of the students in the Engineering Projects in Community Service (EPICS) program, based on their scores across eight noncognitive measures (e.g., communication and teamwork skills), as defined by the Accreditation Board for Engineering and Technology’s Engineering Criteria 2000 (ABET EC2000) Criterion 3. The results supported a four-profile solution. Profiles were described in terms of their level (means) and shape (peaks and valleys) of performance on the noncognitive subscales. These profiles will be used as a foundation for continuous improvement in the service-learning area within engineering education. Background/Theoretical Framework First established in 1995, Engineering Projects in Community Service (EPICS) is a service-learning program that enables long-term projects in which teams of engineering undergraduates are matched with community service agencies that request technical assistance. Within EPICS program, teams of undergraduates design, build, and deploy real systems to solve engineering-based problems for local community service and education organizations 1 . With a main objective to integrate engineering design with meeting the needs of the local community through a multi-disciplinary service learning curricular structure, EPICS programs are now operating at 15 universities nationwide with over 1350 students participated on 140 teams 1 . Accreditation Board for Engineering and Technology’s Engineering Criteria 2000 (ABET, 1999) Criterion 3 2 Programs Outcomes and Assessment specifies outcomes P ge 12110.2 college graduates are expected to know and demonstrate from accredited engineering programs. The generality of Criterion 3 objectives require engineering programs to articulate desired program outcomes related to professional skills that the participants can assess through self-report instruments 3 . In recognition of this complex task, EPICS ABET EC 3 self-report instruments were developed by a team of engineering educators and psychometricians. These scales provide educators critical information regarding students’ perception of ABET EC 3, and in turn, may provide a foundation for continuous students and program improvement. The term profile comes from the practice in applied work in which scores on a test battery are plotted in terms of graph or profile 4 . In profile analysis, groups of individuals are formed based on similarities in scores on a meaningful set of measures. Mathematically, the main objective of profile analysis is to aggregate cases/subjects to minimize intragroup multivariate variability while maximizing intergroup multivariate variability. By embodying the practice of interpreting score variation across a set of measures as indicative of an individual’s personal attributes, profile analysis has been widely used in the behavioral sciences to indicate performance on a test battery. In profile analysis, it is recommended that a normative typology of the profile types, or core profiles, in the population be developed prior to judging profile uniqueness prior to determining the clinical or educational relevance of a profile 6 , 5 . Cluster analysis is the statistical method to derive normative typologies to classify individuals with similar profiles/patterns of performance. Kachigan 7 defined cluster analysis as “set of techniques for accomplishing the task of partitioning a set of objects into relatively homogeneous subsets based on inter-object similarities”. Previous research P ge 12110.3 by Maller et al 8 showed that, although profile analysis has been used considerably to derive normative typologies for behavioral 9 and intelligence tests 11 , 10 , no research has applied cluster analytic techniques to derive normative typologies within the context of service-learning in engineering education area. Classification can provide critical insight into the relationship between students’ perceptions of the program and other important aspects of learning (e.g., academic achievement). Thus, profile analysis is useful for the EPICS program for at least two reasons: (a) for program evaluation to monitor how EPICS students perform on a variety of ABET criteria, and (b) to understand how EPICS students share common characteristics on ABET outcomes that may affect their educational and professional experiences. By using the McDermott’s 13 , 12 three-stage cluster analysis strategy, the main purpose of current study was to derive homogeneous subtypes of individual EPICS students, based upon their scores across measures of eight program outcomes. Specifically, the present study includes: (1) examination of how EPICS students were grouped in terms of their evaluation on the professional skills and objectives defined by ABET EC2000 Criterion 3, and analysis of the characteristics on specific profile pattern(s) found; (2) investigation of possible explanatory (e.g., demographic background variables) reasons of the way they were grouped. For instance, mean scores of the two gender groups were compared to see if significant difference existed between male and female in typal prevalence. Additionally, future research direction was also discussed. Method Participants. The sample was obtained from the 264 students (32% females, 68% males) registered in EPICS program at a major Midwestern U.S. university during the 2005-2006 academic year. Among these students, 75% were in Engineering-related P ge 12110.4 majors (e.g., Electronic Engineering and Civil Engineering), while the rest of 25% were from non-engineering majors. Instrumentation. All data were based on students’ self-ratings on the instrument designed by a team of engineering educators and psychometricians at the same institution. This self-report instrument aimed to conceptualize and measure specific professional skills of the EPICS students, and evaluate whether an engineering design course effectively promotes the program and Criterion 3 outcomes. The ABET Criterion 3 outcomes were formally defined based on theory, empirical evidence, Criterion 3, and the goals of the engineering program. Previous study reported the detailed process of scale construction and validation. The following eight subscales were included: socialresponsibility, design process, awareness of ethical issues, teamwork, lifelong learning, oral skills, written skills, and communication competence. For each individual item, students recorded their responses on a 5-point Likert scale (e.g., 1=strongly disagree to 5=strongly agree). The students also took a questionnaire with eighteen demographic questions (gender, race, etc.) along with the survey. Items were reviewed by faulty from the College of Engineering and the College of Education to ensure content validity. Item analysis was used to delete, modify and replace poorly performed items (e.g., low itemtotal correlation or item discrimination). All subscale Cronbach alphas were at or above .90, indicating acceptable reliability evidence and that the scales provided consistent scores. The construct validity of the scales has been supported by methods of confirmatory factor analysis. Data Analysis. McDermott’s Multistage Euclidean Grouping (MEG) cluster analytic strategy was used to generate profiles of the EPICS students based on subscale P ge 12110.5 standard scores. MEG is a comprehensive 3-step procedure for hierarchical agglomerative cluster analysis with subsequent k–means iterative partitioning that provides built-in replications, relocations, and descriptive measures to ensure correct number of clusters 13 , 12 . Three steps were involved in MEG: first, the whole sample was divided and assigned randomly into two mutually exclusive random blocks, with 132 subjects in each block, and clusters were derived for each independent data block. Based on simulation studies that have shown its ability to recover known data structures 15 , 14 , Ward’s method 16 was used to increase the degree of association of cases within clusters with maximal dissimilarity between clusters; second, a full similarity matrix based on the Squared Euclidean Distances was provided by the first-step clustering results to obtain higher order clusters for the entire sample using Ward’s method; finally, iterative partitioning using k-means centroid sorting was used to relocate cases to optimize withincluster homogeneity . It is desirable to derive clusters that would be replicable and account for the prevalent profiles that exist in the population, and not exclude profiles that do not readily conform to the profiles of the larger population. The following six criteria were used to determine the most representative normative typology: (a) a minimal loss in the error sum of squares (ESS) for merging clusters 16 ; (b) an average withincluster homogeneity coefficient, H 17 > .60; (c) solutions meeting Mojena’s first stopping rule , (d) simultaneous elevation of the pseudo-F statistic over pseudo-t 2 statistics 19 ; (e) an average between-cluster similarity coefficient, p r < .40 20 ; and (f) a 75% replication rate for each cluster, determined by absorption into the firstand second-stage partitioning 21 . Clusters were described by mean z-scores on the eight noncognitive measures to make subsequent comparisons meaningful. Chi-square analyses were P ge 12110.6 conducted to test the gender and major differences across clusters. Results Figure 1 displays the four-cluster (referred to as profiles) solution for EPICS students. Profiles are describes as: Profile 1 (high level with comparatively low scores on ethics subscale), Profile 2 (average level with comparatively high scores on ethics subscale), Profile 3 (average level), and Profile 4 (low level). The higher the profile level is, the more the EPICS students agree that the specific engineering design course effectively promotes the program and ABET Criterion 3 outcomes related to their professiona