Validity of the Contextual Competence Scale for Engineering Students

Background Engineering educators and practitioners increasingly value contextual competence. A psychometrically sound, practical, and useful scale for assessing undergraduate engineering students' contextual competence is needed. Purpose/Hypothesis This article provides comprehensive evidence of the content, structural, discriminant, and criterion-related validity of the contextual competence scale. Design/Method This study used student, alumni, and faculty survey data from a nationally representative sample of 120 U.S. engineering programs from 31 four-year institutions. Validity evidence was obtained by expert review of questions, a pilot test, factor analyses, and several analyses utilizing t-tests, correlations, and regression. Results Experts constructed the questions used in the scale (content validity). Those questions combined to measure a single concept (structural validity), the scale reliably measures (Cronbach's alpha = .91) that concept, and it focuses on contextual competence instead of other student outcomes (discriminant validity). The contextual competence scale varies according to students' characteristics and curricular experiences as well as similarities and differences in student and alumni levels of contextual competence in the same programs and institutions (criterion-related validity). Finally, the scale may be a more accurate measure of contextual competence than faculty members' perceptions of students' ability. Conclusions The contextual competence scale allows engineering programs to meet ABET and other self-study requirements or do large-scale research with relative ease and little expense. The process described in this article can be used by other researchers in engineering education for their scale development efforts.

[1]  Elizabeth J. Whitt,et al.  Differences Between Women and Men in Objectively Measured Outcomes, and the Factors That Influence Those Outcomes, in the First Three Years of College , 2003 .

[2]  Shannon K. Gilmartin,et al.  Assessing Response Rates and Nonresponse Bias in Web and Paper Surveys , 2003 .

[3]  Betsy Palmer,et al.  Best Paper PIC II: Design in Context: Where do the Engineers of 2020 Learn this Skill? , 2011 .

[4]  L. Hoffmann,et al.  An intervention study to enhance girls' interest, self-concept, and achievement in physics classes , 2002 .

[5]  Joan S. Stark,et al.  Faculty Perceptions of Professional Preparation Environments: Testing a Conceptual Framework , 1987 .

[6]  J. Bordogna,et al.  Engineering Education: Innovation Through Integration , 1993 .

[7]  Cynthia J. Atman,et al.  Engineering design factors: how broadly do students define problems? , 2000, 30th Annual Frontiers in Education Conference. Building on A Century of Progress in Engineering Education. Conference Proceedings (IEEE Cat. No.00CH37135).

[8]  Nicholas A. Bowman.  Can 1st-Year College Students Accurately Report Their Learning and Development? , 2010 .

[9]  Suzanne G. Brainard,et al.  A Six‐Year Longitudinal Study of Undergraduate Women in Engineering and Science * , 1998 .

[10]  Eric L. Dey.  Working with Low Survey Response Rates: The Efficacy of Weighting Adjustments , 1995 .

[11]  Gül E. Okudan Kremer,et al.  Approaches to engaging students in engineering design and problem solving , 2011 .

[12]  Cynthia J. Atman,et al.  Engineering Student Attitudes Assessment , 1998 .

[13]  Barbara M. Moskal,et al.  The Globally Competent Engineer: Working Effectively with People Who Define Problems Differently , 2006 .

[14]  L. Cronbach.  Coefficient alpha and the internal structure of tests , 1951 .

[15]  E. Thorndike.  A constant error in psychological ratings. , 1920 .

[16]  Andrew Lau.  Life-centered design - A paradigm for engineering in the 21 st century , 2004 .

[17]  J. Graham,et al.  Missing data analysis: making it work in the real world. , 2009, Annual review of psychology.

[18]  Brent K. Jesiek,et al.  Cultural Orientation and Global Competency: A Comparative Assessment of Engineering Students , 2011 .

[19]  Lisa R. Lattuca,et al.  Academic Environments in Detail: Holland’s Theory at the Subdiscipline Level , 2010 .

[20]  Yehuda Baruch,et al.  Response Rate in Academic Studies — A Comparative Analysis , 1999 .

[21]  Tom W. Smith.  TRENDS IN NON-RESPONSE RATES , 1995 .

[22]  C. Atman,et al.  Considering Context: A Study of First‐Year Engineering Students , 2007 .

[23]  P. Muchinsky.  WHEN THE PSYCHOMETRICS OF TEST DEVELOPMENT MEETS ORGANIZATIONAL REALITIES: A CONCEPTUAL FRAMEWORK FOR ORGANIZATIONAL CHANGE, EXAMPLES, AND RECOMMENDATIONS , 2004 .

[24]  D. Boud,et al.  Student Self-Assessment in Higher Education: A Meta-Analysis , 1989 .

[25]  W.E. Kastenberg,et al.  An Approach to Undergraduate Engineering Education for the 21st Century , 2006, Proceedings. Frontiers in Education. 36th Annual Conference.

[26]  Elaine Seymour,et al.  The loss of women from science, mathematics, and engineering undergraduate majors: An explanatory account , 1995 .

[27]  Hyun Kyoung Ro,et al.  Measuring engineering students' contextual competence , 2012 .

[28]  Clive L. Dym,et al.  Social Dimensions of Engineering Design: Observations from Mudd Design Workshop III (An Educational Brief) , 2001 .

[29]  Glenn Regehr,et al.  Measuring Self-assessment: Current State of the Art , 2002, Advances in health sciences education : theory and practice.

[30]  C. Steeh.  Trends in Nonresponse Rates, 1952–1979 , 1981 .

[31]  Ashley Ater Kranov,et al.  A Direct Method For Teaching And Assessing Professional Skills In Engineering Programs , 2008 .

[32]  D. Grasso,et al.  Socially‐Relevant Design: The TOYtech Project at Smith College , 2002 .

[33]  M. Loui.  Ethics and the Development of Professional Identities of Engineering Students , 2005 .

[34]  Joan S. Stark,et al.  Strengthening the Ties That Bind: Integrating Undergraduate Liberal and Professional Study. Report of the Professional Preparation Network. , 1988 .

[35]  Cynthia J. Atman,et al.  Comparing freshman and senior engineering design processes: an in-depth follow-up study , 2005 .

[36]  Yin Zhang.  Using the Internet for survey research: a case study , 2000 .