Formative Feedback: Providing Bioengineering Professors With Quantitative Measures Of Their Teaching

The purpose of this work is to determine the effects of formative, quantitative feedback to bioengineering professors on the teaching of their lessons. For the past three years, members of the assessment thrust of the VaNTH Engineering Research Center at Vanderbilt University have worked to develop an observation instrument to capture specific elements in lesson delivery, including (1) various types of teacher-student interactions, student engagement levels related to type of instruction, levels of indicators reflecting HPL learning theory (knowledge-centeredness, student-centeredness, assessment-centeredness, and community-centeredness) (Bransford et al. 1999), and specific indicators of effective teaching. After three years of testing and revision, we have developed valid, quantitative measurements of the teaching of a lesson. We then set about to organize this data into categories that would profile a classroom lesson and allow professors to self-assess their lesson delivery and student engagement. Based on findings in an unpublished dissertation (Harris, 1988), our hypothesis is that providing the professors with quantitative measurements of specific classroom activities would cause them to modify their teaching in ways that would include more HPL elements and indicators of effective teaching. Professors receive both graphical profiles of various aspects of their teaching and the observer’s additional explanation of specific aspects. We predict that formative feedback will have an effect on professors’ structuring and teaching of lessons. This was supported primarily by the Engineering Research Centers Program of the National Science Foundation under Award Number EEC9876363. Background and Introduction In September of 1999, bioengineering and learning science faculties at four research universities (Vanderbilt University, Northwestern University, the University of Texas at Austin, and the Harvard/Massachusetts Institute of Technology Division of Health Science and Technology [VaNTH]) received a National Science Foundation grant to support a collaboration of these universities within an Engineering Research Center (ERC) for bioengineering educational technologies. Although there are several ERCs throughout the country, VaNTH is the only ERC with a focus on education; and for the past three years this group has worked toward the goal of the improvement of teaching in bioengineering. One endeavor of this project has been the development of an observation system to document classroom teaching and measure change. Educators, learning scientists, and bioengineering domain experts within the ERC have worked together to develop a classroom observation system that differentiates various types of teaching within a classroom, and thus can eventually help answer questions on what kinds of teaching are more effective. P ge 878.1 Proceedings of the 2003 American Society for Engineering Education Annual Conference & Exposition Copyright © 2003, American Society for Engineering Education The resulting VaNTH Observation System (VOS) is a four-part system, with each part reflecting a type of classroom observation proven useful in prior K-12 research 2, 3, . These four types are teacher-student interactions, student academic engagement, narrative notes, and global ratings of indicators of effective teaching. The observer cycles repeatedly through the first three of these, and then completes the fourth at the end of a class session: in other words, three minutes of CIO real-time data collection, followed by a time-sample of student engagement, followed by a description of the CIO segment, and back to the CIO and repeat through the class period; global ratings after class. The instrument as a whole provides primarily quantitative measures of teaching in the bioengineering classroom, with some slight quantitative data in the narrative notes that can assist in the interpretation of the quantitative. One innovation of the VOS not found in any other observation system is the ability to capture a class lesson’s “HPLness” – that is, indicators of the “How People Learn” learning theory, which is set forth in set forth in the National Research Council’s How People Learn: Brain, Mind, Experience, and School . Data Recorded by the Classroom Interaction Observation (CIO) Of the four VOS components, the Classroom Interaction Observation offers the richest source of information on teaching within a classroom. Within CIO data there are numerous strings of interaction recorded in each three-minute segment (approximately one string every six seconds), and each contains information on what the instructor is doing, how this relates to students, the HPLness of the interaction, and media use. The coding pattern of a string is who/to whom/said what/how/using what media. Forty-one separate possible codes within five coding categories provide over 150 different valid code strings. It is within the “how” category of these code strings that the four lesson “centerednesses” relating to HPL learning theory are captured: knowledge-, student-. assessment-, and community-centeredness. Data Recorded by the Student Engagement Observation (SEO) The Student Engagement Observation provides information on students’ academic engagement – or lack thereof. It classifies each student present as engaged in either “desirable” or “undesirable” behaviors. After each three-minute CIO segment, the observer makes a visual sweep of the classroom, focuses momentarily on each student, and records numbers: first, the total number of students in the room; next, the number engaged in each of four different categories of unsanctioned behaviors (e.g., sleeping); and last, the number of remaining students “definitely engaged” and “probably engaged” in each of six different categories of sanctioned behavior (e.g., lecture). Data Recorded by the Narrative Notes (NN) The Narrative Notes component provides descriptions (1) of the topic of a lesson segment (e.g., free body diagram), (2) of external factors that may affect the success of that segment (e.g., freezing room temperature), and (3) of other possibly relevant information not captured elsewhere in the VOS (e.g., student enthusiasm for donuts a professor provided). It also provides an identification of the type or types of instruction used in the prior three-minute lesson segment, based on 15 categories identified by professors of biomedical and chemical engineering. P ge 878.2 Proceedings of the 2003 American Society for Engineering Education Annual Conference & Exposition Copyright © 2003, American Society for Engineering Education Data Recorded by the Global Ratings (GR) The Global Ratings component uses a Likert scale to indicate the degree of presence of 17 specific indicators related to effective teaching and/or HPLness. Plan of Action For the past three years, we have worked to develop various lesson profiles based on each of the sections of the VOS that can be used to assist in data analysis. As we examined these charts, we became convinced of their possibility to provide formative feedback to professors – that is, feedback that would allow them to reflect on and assess their own teaching and to determine what changes they may wish to make in their teaching. The VOS provides massive amounts of data, especially in the CIO. Because of this, we had to develop a way to combine information into logical categories relevant to professors’ classrooms and teaching styles. We wanted to present data graphically in a way that a novice in learning science and pedagogy could easily interpret it, could gain useful information from it, and could make appropriate lesson analyses and adjustments based on it. We started with the CIO. As there are so many possibilities within the CIO code strings, we began by determining the categories most relevant to an understanding of a class lesson, and the specific CIO data that would feed into each of those categories. We next developed a computer program that would parse the data collected into the newly developed categories. Finally, we designed a bar graph format to present the data in a logical way that flowed easily from one category to another. We then turned to data in the SEO and in the GR. We found that student engagement was easily plotted directly from the categories as they stand. For each data sample in the SEO, a bar representing 100% could be divided into segments of “definitely engaged,” “probably engaged,” and “unengaged.” For each summary GR at the end of a lesson, we simply plotted the data as a straightforward bar graph. In the spring semester of 2003, we plan to provide observed professors with formative feedback based on data from their VOS observations. Eight observations will be spaced throughout the semester, with cumulative feedback provided after the fourth and eighth observations. Prior to receiving feedback, professors will participate in a one-hour learning forum, which will explain the various observed categories and their operational definitions (“what it looks like in a classroom”). Thus they will have a knowledge of what has been and will be measured. Professors will receive both graphic and numeric data, and will work to combine categories of numeric data to reflect the teaching patterns unique to their own classrooms (e.g., questioning patterns, feedback patterns, lecture patterns).