2002 ECI Conference on e-Technologies in Engineering Education: Learning Outcomes Providing Future Possibilities

Effective Strategies to Assess the Impact of e-Learning A i UN ITED i ENG INEER ING FOUNDATIONiCONFERENCE Davos,iSwitzerlandi11-16iAugusti2002 http://www.coe.gatech.edu/eTEE TEE 2002 e-Technologies in Engineering Education Learning Outcomes Providing Future Possibilities Proceedings of the 2002 eTEE Conference 11-16 August 2002 Davos, Switzerland 158 e-Technologies in Engineering Education Learning Outcomes Providing Future Possibilities II. The Assessment Process As I work with people who are developing assessment plans, I see two common mistakes: 1) making the process too complicated; and 2) skipping some of the essential steps in developing an effective process. Because of this, my colleague Ron Miller and I developed a matrix that provides faculty with the structure they need to develop an effective assessment plan but also with the flexibility to be adapted for a variety of settings and purposes, including e-learning [6]. The matrix has been used successfully for course and program assessment at several institutions; it can also easily be used for educational project evaluation or individual course assessment. Our goal in developing the matrix was to help demystify assessment for faculty and assuage some of their fears about assessment. I first present a brief overview on developing an assessment plan and then introduce details of the matrix. A. Developing an Assessment Plan The following steps have been found valuable in developing an effective assessment plan: • Identify course or program objectives consistent with institutional goals and the needs of internal and external stakeholders including accrediting agencies. • Develop course or program outcomes and performance criteria for each objective. • Decide what course and program curricular and cocurricular activities will address each outcome. • Determine the best methods for assessing and evaluating each outcome and decide when assessment data will be collected. • Report results to stakeholders and use feedback to improve the program and the assessment process itself. Each of these steps will be discussed more fully in the next section. B. The Assessment Matrix A relatively easy way to begin developing a course or program evaluation plan is to use the assessment matrix, summarized in Table 1, which Ron Miller and I adapted and expanded from a similar matrix included in the National Science Foundation’s User Friendly Handbook for Project Evaluation [7]. We are also indebted to the outcomes assessment guidelines developed by Gloria Rogers and Jean Sando and published in Stepping Ahead: An Assessment Plan Development Guide [8]. Table 1. Assessment Matrix. ___________________________________________________

[1]  A. Maskrey,et al.  A Call for Action , 1997 .

[2]  Craig Zimring,et al.  Supporting collaborative design groups as design communities , 2000 .

[3]  Ian R. Simpson International Aspects Of Engineering Education In Europe , 1997 .

[4]  Andrew B. Hargadon Firms as Knowledge Brokers: Lessons in Pursuing Continuous Innovation , 1998 .

[5]  James A. Perkins,et al.  International Education in a Troubled World , 1980 .

[6]  James H. McClellan,et al.  ANIMATIONS AND GUIS FOR INTRODUCTORY ENGINEERING COURSES , 2001 .

[7]  Geri Gay,et al.  Use of Communication Resources in a Networked Collaborative Design Environment , 2006, J. Comput. Mediat. Commun..

[8]  John L. Gossage,et al.  Integrating best practice pedagogy with computer-aided modeling and simulation to improve undergraduate chemical engineering education , 2001 .

[9]  J. J. Carroll The Power of Problem-Based Learning. , 2005 .

[10]  Barbara M. Olds,et al.  An Assessment Matrix for Evaluating Engineering Programs , 1998 .

[11]  Curtis J. Bonk,et al.  Stating the State of E-learning: Surveys in College and Corporate Training Settings , 2001 .

[12]  D. J. Mook,et al.  Educational opportunities for US students abroad: how to internationalize and diversify your university , 2001, 31st Annual Frontiers in Education Conference. Impact on Engineering and Science Education. Conference Proceedings (Cat. No.01CH37193).

[13]  R. Oliver,et al.  Using situated learning and multimedia to investigate higher-order thinking , 1999 .

[14]  Jason L. Frand The Information-Age Mindset: Changes in Students and Implications for Higher Education , 2000 .

[15]  Alfred Bork,et al.  Multimedia in Learning , 2001 .

[16]  B. Olds e-Technologies in Engineering Education Learning Outcomes Providing Future Possibilities , 2003 .

[17]  Kathy Schmidt,et al.  The professor and the media laboratory: a case study in Web-based course creation , 2001, 31st Annual Frontiers in Education Conference. Impact on Engineering and Science Education. Conference Proceedings (Cat. No.01CH37193).

[18]  J. Prus,et al.  A critical review of student assessment options , 1994 .

[19]  H. Scott Fogler,et al.  Strategies for Creative Problem Solving , 1994 .

[20]  Russel C. Jones Educating Engineers for International Practice. , 1995 .

[21]  M. D. Merrill,et al.  Knowledge objects and mental models , 2000, Proceedings International Workshop on Advanced Learning Technologies. IWALT 2000. Advanced Learning Technology: Design and Development Issues.

[22]  H. Bourlard,et al.  S E a R C H , 2002 .

[23]  Debashish Dutta,et al.  Global Product Development: Using Global Resources Effectively for a Novel Course , 2002 .

[24]  Robert B. Gilbert,et al.  Development, implementation, and preliminary assessment of virtual laboratory , 1999 .

[25]  J. Bransford How people learn , 2000 .

[26]  B. Bloom,et al.  Taxonomy of Educational Objectives. Handbook I: Cognitive Domain , 1966 .

[27]  N. Carolina. Commission on Higher Education , 1984 .

[28]  Nigel Marlow,et al.  The role of student learning styles, gender, attitudes and perceptions on information and communication technology assisted learning , 1999, Comput. Educ..

[29]  Cynthia J. Atman,et al.  A comparison of freshman and senior engineering design processes , 1999 .

[30]  D Wolfle,et al.  Women in Science and Engineering. , 1964, Science.

[31]  Matthew C. Makel,et al.  Growing up digital , 2020, Transformative Digital Humanities.

[32]  John C. Strikwerda,et al.  Reversing the Lecture/Homework Paradigm Using eTEACH® Web‐based Streaming Video Software , 2002 .

[33]  Marc Prensky,et al.  Digital game-based learning , 2000, CIE.

[34]  Gary Klass Plato as Distance Education Pioneer: Status and Quality Threats of Internet Education , 2000, First Monday.

[35]  Ann L. Brown,et al.  How people learn: Brain, mind, experience, and school. , 1999 .

[36]  Lawrence W. Barsalou,et al.  The role of situations in concept learning , 1996 .

[37]  Shuichi Fukuda,et al.  What we have learned from our experience from TMIT-Stanford shared class , 1999, IEEE SMC'99 Conference Proceedings. 1999 IEEE International Conference on Systems, Man, and Cybernetics (Cat. No.99CH37028).

[38]  Mark Guzdial,et al.  Beyond Adoption to Invention: Teacher-Created Collaborative Activities in Higher Education , 2001 .

[39]  Hartmut Ewald,et al.  Performing Experiments by Remote Control Using the Internet , 2000 .

[40]  R. A. Kenyon,et al.  The International Virtual Design Studio , 1997, Proceedings Frontiers in Education 1997 27th Annual Conference. Teaching and Learning in an Era of Change.

[41]  John Seely Brown,et al.  Book Reviews : The Social Life of Information By John Seely Brown and Paul Duguid. Boston: Harvard Business School Press, 2000. 320 pages , 2000 .

[42]  Gary R. Martin A Co Op Based International Engineering Minor Degree , 1997 .

[43]  Alex H. B. Duffy,et al.  The "What" and "How" of Learning in Design , 1997, IEEE Expert.

[44]  Margot Brereton The role of hardware in learning engineering fundamentals: An empirical study of engineering design and product analysis activity , 1999 .

[45]  Larry Leifer,et al.  Design-Team Performance: Metrics and the Impact of Technology , 1998 .

[46]  J. M. Pullen,et al.  ClassWise: synchronous Internet desktop education , 1999 .

[47]  W. Hodgins,et al.  Into the future: A vision paper , 2000 .

[48]  S. Covey The seven habits of highly effective people. , 1991, National medical-legal journal.

[49]  Bay-Wei Chang,et al.  In-Place Editing of Web Pages: Sparrow Community-Shared Documents , 1998, Comput. Networks.

[50]  I. Nonaka,et al.  Enabling Knowledge Creation , 2000 .

[51]  Jane Seale,et al.  Reflection on-line or off-line: the role of learning technologies in encouraging students to reflect , 2000, Comput. Educ..

[52]  Erkki Sutinen,et al.  StoryML: An XML Extension for Woven Stories , 2002, Intelligent Tutoring Systems.

[53]  H. Barrows Problem‐based learning in medicine and beyond: A brief overview , 1996 .

[54]  M. Fischetti Working knowledge. , 2003, Scientific American.

[55]  Vincent Kiernan An Engineering Professor Uses the Web To Run a "Virtual Laboratory.". , 1997 .

[56]  G. Augusti Europe today and tomorrow: mutual recognition by harmonization or by comparison? , 1998 .

[57]  K. Charlton,et al.  IMAGINATION AND EDUCATION , 1967 .

[58]  Patricia Arnold,et al.  Communities of Practice: guidelines for the design of online seminars in higher education , 2001 .

[59]  B. E. Mulhall H3E-a thematic network in engineering , 1998 .

[60]  Ernest T. Pascarella,et al.  Students' Out-of-Class Experiences and Their Influence on Learning and Cognitive Development: A Literature Review. , 1996 .

[61]  James H. McClellan,et al.  Animating Theoretical Concepts for Signal Processing Courses , 2002 .

[62]  Piet Kommers,et al.  Theoretical backgrounds of educational and training technology , 2001 .

[63]  David H. Jonassen,et al.  Problems and issues in designing hypertext/hypermedia for learning , 1990 .

[64]  G. Lefrançois Psychology for teaching , 1979 .

[65]  Guillermo Vasquez de Velasco,et al.  The Internationalization of an Undergraduate Program Using Reciprocal Distance Education , 1999 .

[66]  R. Ivanič Writing and identity , 1998 .

[67]  Amy R. Pritchett,et al.  Software For The Automated Evaluation Of Web Delivered Instruction , 2002 .

[68]  Adegboyega Oladipo Mabogunje Measuring conceptual design process performance in mechanical engineering: a question based approach , 1998 .

[69]  R. Light,et al.  Making the Most of College: Students Speak Their Minds , 2001 .

[70]  Tara Brabazon Internet Teaching and the Administration of Knowledge , 2001, First Monday.

[71]  E. Seymour,et al.  Talking About Leaving: Why Undergraduates Leave The Sciences , 1997 .

[72]  P. Brouwer,et al.  Hold on a Minute Here: What Happened to Critical Thinking in the Information Age? , 1996 .

[73]  Bob Boice,et al.  Systematic Mentoring for New Faculty Teachers and Graduate Teaching Assistants , 1998 .

[74]  Adam L. Turk,et al.  The use of video-taped lectures and web-based communications in teaching: A distance-teaching and cross-Atlantic collaboration experiment , 2002 .

[75]  Samuli Kolari,et al.  Does Pedagogical Training Benefit the Engineering Educator , 2002 .

[76]  Alice M. Agogino,et al.  The impact and instructional benefit of using multimedia case studies to teach engineering design , 1994 .

[77]  Erkki Sutinen,et al.  Woven stories: collaboratively authoring microworlds via the Internet , 1999 .

[78]  David Wallace,et al.  A Comparative Evaluation of World Wide Web-Based and Classroom Teaching , 1997 .

[79]  Peter Elbow,et al.  A Community of Writers , 1989 .

[80]  Mark Guzdial,et al.  Using An Unstructured Collaboration Tool to Support Peer Interaction in Large College Classes , 2000 .

[81]  Linda Carswell The “Virtual University”: toward an Internet paradigm? , 1998, ITiCSE '98.

[82]  Leonard A. Van Gulick,et al.  Internationalization Of The Lafayette College Engineering Curriculum , 1997 .

[83]  Mark Guzdial,et al.  Teacher and student authoring on the web for shifting agency , 1999 .

[84]  Jörg Roche,et al.  Intercultural Challenges in Networked Learning: Hard Technologies Meet Soft Skills , 2002, First Monday.

[85]  J. Nash,et al.  NON-COOPERATIVE GAMES , 1951, Classics in Game Theory.

[86]  David Bargeron,et al.  Supporting interaction outside of class: anchored discussions vs. discussion boards , 2002, CSCL.

[87]  D. Winter,et al.  Adapting the internet: Comments from a women-only electronic forum , 1996 .

[88]  Mark Guzdial,et al.  A Catalog of CoWeb Uses , 2000 .

[89]  Julián Monge-Nájera,et al.  Internet, Multimedia and Virtual Laboratories in a 'Third World' Environment , 2001 .

[90]  Floraline I. Stevens,et al.  User-Friendly Handbook for Project Evaluation: Science, Mathematics, Engineering, and Technology Education. , 1993 .

[91]  Don Mitchell,et al.  The Architecture of a Distributed Virtual Worlds System , 1998, COOTS.

[92]  Carol A. Twigg,et al.  The Changing Definition of Learning. , 1994 .

[93]  James E. Pelkie Technological Innovation: Regaining the Competitive Edge , 1989 .

[94]  Albert L. Ingram,et al.  Beyond chat on the internet , 2000, Comput. Educ..

[95]  Lyle D. Feisel,et al.  A Colloquy on Learning Objectives For Engineering Education Laboratories , 2002 .

[96]  Pat Hutchings The Course Portfolio: How Faculty Can Examine Their Teaching To Advance Practice and Improve Student Learning. The Teaching Initiatives. , 1998 .

[97]  Bo Leuf,et al.  The Wiki Way: Quick Collaboration on the Web , 2001 .

[98]  Lora Aroyo,et al.  An agent-oriented approach for ideational support in learning—integration and impact , 2000 .

[99]  Andrew B. Hargadon,et al.  Technology brokering and innovation in a product development firm. , 1997 .

[100]  Oeit The Innovation Bubble , 2002 .

[101]  John R. Bourne,et al.  Virtual Engineering Laboratories: Design and Experiments , 1994 .

[102]  Richmond Campbell,et al.  Paradoxes of Rationality and Cooperation: Prisoner's Dilemma and Newcomb's Problem , 1986 .

[103]  B. Millis Making the Most of College: Students Speak Their Minds , 2002 .

[104]  G. W. Furnas,et al.  Generalized fisheye views , 1986, CHI '86.

[105]  G. Albertelli,et al.  Teaching with ALN Technology: Benefits and Costs * , 2001 .

[106]  David H. Gillette Using electronic tools to promote active learning , 1996 .

[107]  Greg Myers,et al.  Writing biology : texts in the social construction of scientific knowledge , 1990 .

[108]  M. Gladwell,et al.  The Tipping Point , 2011 .

[109]  L. A. Gerhardt The Global Engineering Education Exchange Program-its history, progress, and challenges , 2001, 31st Annual Frontiers in Education Conference. Impact on Engineering and Science Education. Conference Proceedings (Cat. No.01CH37193).

[110]  Kim LaScola Needy,et al.  Combining international experience and industrial relevance in a Capstone engineering design course , 1997, Proceedings Frontiers in Education 1997 27th Annual Conference. Teaching and Learning in an Era of Change.

[111]  Paul Bacsich,et al.  The costs of networked learning , 2001 .

[112]  Gail A. Herndon The chronicle of higher education , 1977 .

[113]  Robin Adams,et al.  Assessment of an International Freshmen Research and Design Experience: A Triangulation Study* , 2002 .

[114]  R. Clark Reconsidering Research on Learning from Media , 1983 .

[115]  Luis E. Anido-Rifón,et al.  Labware for the Internet , 2000, Comput. Appl. Eng. Educ..

[116]  Helmut M. Niegemann Selbstkontrolliertes Lernen und Didaktisches Design , 1998 .

[117]  Tony Clear,et al.  Using groupware for international collaborative learning , 2000, 30th Annual Frontiers in Education Conference. Building on A Century of Progress in Engineering Education. Conference Proceedings (IEEE Cat. No.00CH37135).

[118]  A. Chickering,et al.  Seven Principles for Good Practice in Undergraduate Education , 1987, CORE.

[119]  N. A. Mello How one institution provides a global perspective for engineers , 2001, 31st Annual Frontiers in Education Conference. Impact on Engineering and Science Education. Conference Proceedings (Cat. No.01CH37193).

[120]  Mark Guzdial,et al.  Computer supported collaborative learning for curriculum integration , 2000 .

[121]  R. Felder,et al.  Learning and Teaching Styles in Engineering Education. , 1988 .

[122]  Alan Mandell,et al.  The Given and the Made: Authenticity and Nature in Virtual Education , 2000, First Monday.

[123]  Fataneh Taghaboni-Dutta,et al.  Teaching in An e-Classroom: How To Be An Effective Teacher to E-Learners , 2002 .

[124]  Mark Guzdial,et al.  Collaborative learning at low cost: CoWeb use in English composition , 2002, CSCL.

[125]  Lee Sproull,et al.  Connections: New Ways of Working in the Networked Organization , 1991 .

[126]  Jose A. Romagnoli,et al.  Web-Based Courseware in Teaching Laboratory-Based Courses , 2000 .

[127]  Dale A. Harris Online distance education in the United States , 1999, IEEE Commun. Mag..

[128]  Bob Rehder,et al.  Using latent semantic analysis to assess knowledge: Some technical considerations , 1998 .

[129]  Mark Guzdial Use of collaborative multimedia in computer science classes , 2001 .

[130]  Elliot Soloway,et al.  Interface design: a neglected issue in educational software , 1986, CHI '87.

[131]  Mike Moore,et al.  Distance Education: A Systems View , 1995 .

[132]  Arvid Andersen Implementation of engineering product design using international student teamwork—to comply with future needs , 2001 .

[133]  Linda A. Suskie Questionnaire Survey Research: What Works , 1996 .

[134]  Larry Leifer,et al.  Facilitating Product Development Knowledge Acquisition: Interaction between the Expert and the Team* , 2003 .

[135]  Niall Sclater,et al.  Online collaborative design projects: overcoming barriers to communication , 2001 .

[136]  Julie Baher Articulate Virtual Labs in Thermodynamics Education: A Multiple Case Study* , 1999 .