Towards an Interactive Assessment Framework for Engineering Design Learning

In this paper we explore the use of text processing technology for on-line assessment in an engineering design project class. We present results from a 5-week classroom study in a capstone engineering design course in which we explore the potential benefits of such technology for student learning in this context. Furthermore, we present results from ongoing work assessing student productivity based on features extracted from their conversational behavior in the course discussion board. While we found that typical shallow productivity measures such as number of posts, length of posts, or number of files committed have no correlation with an instructor assigned grade, we can achieve a substantial improvement using simple linguistic patterns extracted from on-line conversational behavior.© 2007 ASME

[1]  C. Hmelo‐Silver.  Problem-Based Learning: What and How Do Students Learn? , 2004 .

[2]  Xin Liu,et al.  Generic text summarization using relevance measure and latent semantic analysis , 2001, SIGIR '01.

[3]  Armin Weinberger,et al.  Scripts for Computer-Supported Collaborative Learning , 2003 .

[4]  Kipling D. Williams,et al.  Interpersonal Relations and Group Processes Social Loafing: a Meta-analytic Review and Theoretical Integration , 2022 .

[5]  Eduard Hovy,et al.  A BE-based Multi-document Summarizer with Sentence Compression , 2005 .

[6]  Karen L. Tonso,et al.  Teams that Work: Campus Culture, Engineer Identity, and Social Interactions , 2006 .

[7]  Judith Donath,et al.  A SEMANTIC APPROACH TO VISUALIZING , 2002 .

[8]  N. Webb,et al.  Short Circuits or Superconductors? Effects of Group Composition on High-Achieving Students’ Science Assessment Performance , 2002 .

[9]  Cynthia J. Atman,et al.  Educating effective engineering designers: the role of reflective practice , 2003 .

[10]  Carolyn Penstein Rosé,et al.  Museli: A Multi-Source Evidence Integration Approach to Topic Segmentation of Spontaneous Dialogue , 2006, NAACL.

[11]  Carolyn Penstein Rosé,et al.  InfoMagnets: Making Sense of Corpus Data , 2006, HLT-NAACL.

[12]  Carolyn Penstein Rosé,et al.  Topic-Segmentation of Dialogue , 2006, HLT-NAACL 2006.

[13]  Shuang Song,et al.  Triangulation of Indicators of Successful Student Design Teams , 2006 .

[14]  JUDY KAY,et al.  School of It Technical Report Wattle Tree: What'll It Tell Us? , 2006 .

[15]  Andreas Krause,et al.  Context-aware mobile computing: learning context- dependent personal preferences from a wearable sensor array , 2006, IEEE Transactions on Mobile Computing.

[16]  Spencer P. Magleby,et al.  A Review of Literature on Teaching Engineering Design Through Project‐Oriented Capstone Courses , 1997 .

[17]  Timothy Koschmann,et al.  How Do People Learn , 2005 .

[18]  N. Webb,et al.  The Development of Students' Helping Behavior and Learning in Peer-Directed Small Groups , 2003 .

[19]  R. Atkinson,et al.  Structuring the Transition From Example Study to Problem Solving in Cognitive Skill Acquisition: A Cognitive Load Perspective , 2003 .

[20]  Neil T. Heffernan,et al.  Automatic and Semi-Automatic Skill Coding With a View Towards Supporting On-Line Assessment , 2005, AIED.

[21]  R. Atkinson,et al.  Transitioning From Studying Examples to Solving Problems: Effects of Self-Explanation Prompts and Fading Worked-Out Steps. , 2003 .

[22]  Klaus Zechner,et al.  Automatic generation of concise summaries of spoken dialogues in unrestricted domains , 2001, SIGIR '01.

[23]  Michael Callaghan,et al.  Conversation space: visualising multi-threaded conversation , 2000, AVI '00.

[24]  Warren Sack,et al.  Conversation Map: An Interface for Very Large-Scale Conversations , 2000, J. Manag. Inf. Syst..

[25]  Alice M. Agogino,et al.  A Document Analysis Method for Characterizing Design Team Performance , 2004 .

[26]  Peter W. Foltz,et al.  The Measurement of Textual Coherence with Latent Semantic Analysis. , 1998 .

[27]  Carolyn Penstein Rosé,et al.  Supporting CSCL with automatic corpus analysis technology , 2005, CSCL.

[28]  Leonard J. Bass,et al.  Adtranz: a mobile computing system for maintenance and collaboration , 1998, Digest of Papers. Second International Symposium on Wearable Computers (Cat. No.98EX215).

[29]  Judy Kay,et al.  The Big Five and Visualisations of Team Work Activity , 2006, Intelligent Tutoring Systems.

[30]  Paul A. Kirschner,et al.  Designing Sociable CSCL Environments , 2004 .

[31]  Francine Chen,et al.  A trainable document summarizer , 1995, SIGIR '95.

[32]  Cynthia J. Atman,et al.  A comparison of freshman and senior engineering design processes , 1999 .

[33]  Inderpal S. Bhandari,et al.  Orthogonal Defect Classification - A Concept for In-Process Measurements , 1992, IEEE Trans. Software Eng..

[34]  Alice M. Agogino,et al.  Towards Computational Tools for Supporting the Reflective Team , 2002, AID.

[35]  Pierre Dillenbourg,et al.  What we know about CSCL and implementing it in higher education , 2004 .

[36]  Susan Finger,et al.  Supporting collaborative learning in engineering design , 2005, CSCWD.

[37]  Ian H. Witten,et al.  Data mining - practical machine learning tools and techniques, Second Edition , 2005, The Morgan Kaufmann series in data management systems.

[38]  James W. Pennebaker,et al.  Linguistic Inquiry and Word Count (LIWC2007) , 2007 .

[39]  Jan-Willem Strijbos,et al.  The effect of roles on computer-supported collaborative learning. , 2004 .

[40]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques, 3rd Edition , 1999 .

[41]  A. Renkl.  Worked-out examples: instructional explanations support learning by self- explanations , 2002 .