Sustainable Engineering Assessment Using Rubric-Based Analysis of Challenge Question Responses

AbstractA method to evaluate students’ application of sustainability principles into engineering design and problem solving was developed. Students were presented with two challenge questions that each posed a scenario. Students responded to one question and their answers were scored using a rubric that combined both analytic and holistic scoring criteria to account for sustainability principles. The rubric was iteratively developed and met standard measures of reliability and validity. Based on responses from primarily civil and environmental engineering students at three institutions, it was found that male students, seniors, and participants in many diverse learning activities (extracurricular service, internships, undergraduate research, and engineering design courses) achieved higher rubric scores. Scores on the challenge questions did not correlate to students’ interests in sustainable engineering, which were measured using a Likert-based survey. A challenge-question approach to formative assessment...

[1]  J. Leydens,et al.  Scoring Rubric Development: Validity and Reliability. , 2000 .

[2]  Monica E. Cardella,et al.  Defining "Sustainable Engineering": A Comparative Analysis of Published Sustainability Principles and Existing Courses , 2011 .

[3]  M. Birenbaum.  Assessment 2000: Towards a Pluralistic Approach to Assessment , 1996 .

[4]  Bart Cambré,et al.  Adjustment for Acquiescence in the Assessment of the Construct Equivalence of Likert-Type Score Items , 2003 .

[5]  Barbara M. Olds,et al.  Can Our Students Recognize And Resolve Ethical Dilemmas , 2004 .

[6]  Paul H. King,et al.  Concept Mapping as a Form of Student Assessment and Instruction , 2002 .

[7]  Frank G. Splitt,et al.  The Challenge to Change: On Realizing the New Paradigm for Engineering Education , 2003 .

[8]  A. Strauss,et al.  The discovery of grounded theory: strategies for qualitative research aldine de gruyter , 1968 .

[9]  A. Strauss,et al.  The Discovery of Grounded Theory , 1967 .

[10]  S. Brookhart.  The Art and Science of Classroom Assessment: The Missing Part of Pedagogy , 1999 .

[11]  Barbara M. Moskal,et al.  Qualitative Methods Used in the Assessment of Engineering Education , 2004 .

[12]  J. L. Elder.  Research and Solutions: Think Systemically, Act Cooperatively: The Key to Reaching a Tipping Point for the Sustainability Movement in Higher Education , 2008 .

[13]  Troy Devon Thomas,et al.  Response Styles in Survey Research: A Literature Review of Antecedents, Consequences, and Remedies , 2013 .

[14]  S. Benesch.  Critical Thinking: A Learning Process for Democracy , 1993 .

[15]  Alija Kulenović,et al.  Standards for Educational and Psychological Testing , 1999 .

[16]  David R. Riley,et al.  Teaching Sustainability in Building Design and Engineering , 2007 .

[17]  D. Schoen,et al.  The Reflective Practitioner: How Professionals Think in Action , 1985 .

[18]  Ning Wang,et al.  Validating the cognitive complexity and content quality of a mathematics performance assessment , 1994 .

[19]  Banjo Roxas,et al.  Social desirability bias in survey research on sustainable development in small firms: an exploratory analysis of survey mode effect , 2012 .

[20]  Melissa M. Bilec,et al.  Updating the benchmark Sustainable Engineering education report - Trends from 2005 to 2010 , 2011 .

[21]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[22]  Donna M. Randall,et al.  The social desirability response bias in ethics research , 1991 .

[23]  T. Bers,et al.  Assessment and testing: myths and realities , 1994 .

[24]  Filip Dochy,et al.  A new assessment era: different needs, new challenges , 2001 .

[25]  Barbara M. Moskal,et al.  Scoring Rubrics: What, When and How? , 2000 .

[26]  Thomas A. Angelo,et al.  Classroom Assessment Techniques: A Handbook for College Teachers. Second Edition. , 1993 .

[27]  M. Besterfield-Sacre,et al.  Assessing engineering students' abilities to resolve ethical dilemmas , 2003, 33rd Annual Frontiers in Education, 2003. FIE 2003..

[28]  Trudy H. Bers,et al.  Assessment and Testing: Myths and Realities. New Directions for Community Colleges, Number 88. , 1994 .

[29]  B. D. Vries,et al.  Conceptualizing sustainable development An assessment methodology connecting values, knowledge, worldviews and scenarios , 2009 .

[30]  Mary McCormick,et al.  Assessing Students' Motivation to Engage in Sustainable Engineering. , 2015 .

[31]  Steven E. Stemler.  Practical Assessment, Research, and Evaluation Practical Assessment, Research, and Evaluation A Comparison of Consensus, Consistency, and Measurement A Comparison of Consensus, Consistency, and Measurement Approaches to Estimating Interrater Reliability Approaches to Estimating Interrater Reliabilit , 2022 .

[32]  W. Popham.  What's Wrong--and What's Right--with Rubrics. , 1997 .

[33]  Mary Besterfield-Sacre,et al.  Scoring Concept Maps: An Integrated Rubric for Assessing Engineering Education , 2004 .

[34]  Elise Barrella,et al.  Development and Application of a Sustainable Design Rubric to Evaluate Student Abilities to Incorporate Sustainability into Capstone Design Projects , 2013 .

[35]  Mien Segers,et al.  An Alternative for Assessing Problem-Solving Skills: The Overall Test. , 1997 .

[36]  Anders Jonsson,et al.  The use of scoring rubrics: Reliability, validity, and educational consequences , 2007 .

[37]  Cynthia J. Atman,et al.  Concept maps for engineering education: a cognitively motivated tool supporting varied assessment functions , 2000, IEEE Trans. Educ..

[38]  K. Paterson,et al.  Development for the other 80% Engineering Hope , 2008 .

[39]  Rosalyn McKeown,et al.  Using Rubrics to Assess Student Knowledge Related to Sustainability , 2011 .

[40]  Paul J. Feltovich,et al.  Learning, Teaching and Testing for Complex Conceptual Understanding. , 1991 .

[41]  J. Lemke.  Talking Science: Language, Learning, and Values , 1990 .