Assessing Engineering Students' Information Literacy Skills: An Alpha Version of a Multiple-choice Instrument

Information literacy is crucial component of developing life-long learning skills. However, few standard and easily gradable assessment tools exist to assess the information literacy competencies of learners. In this paper, we discuss the development of a multiple choice instrument designed to measure these competencies in an efficient and expedient manner, and we present results of data collected from 366 first-year engineering students. The instrument requires students to first read a technical memo and, based on the memo‟s arguments, answer eight multiple choice and two open-ended response questions. The mean score on the multiple choice portion was only 3.46 out of 8. A qualitative analysis of the open-ended responses provided more insights into students‟ abilities to identify and resolve conflicts between information sources, evaluate the reliability and relevancy of information sources, and use reliable information sources. Introduction One of the most important skills students can take away from a technical education is the ability to become curious, persistent, and life-long learners, and information literacy is crucial to developing those skills. For example, Shuman, et al, when describing requirements for lifelong learning, explicitly includes informational components, expecting students will Demonstrate reading, writing, listening, and speaking skills Demonstrate an awareness of what needs to be learned Follow a learning plan Identify, retrieve, and organize information Demonstrate critical thinking skills Reflect on one‟s own understanding. There is substantial overlap between Shuman‟s lifelong learning competencies and those identified by the Association of College and Research Libraries in their Information Literacy Competency Standards, which include Determining the extent of information needed Access the needed information effectively and efficiently Evaluate information and its sources critically Incorporate selected information into one‟s knowledge base Use information effectively to accomplish a specific purpose Understand the economic, legal, and social issues surrounding the use of information, and access and use information ethically and legally. 3 Despite the growing importance of lifelong learning skills in our rapidly evolving work environment, where technical knowledge has a half life of less than five years, Lattuca, Terenzini, and Volkwein reports that students assess their lifelong learning skills as the lowest of all the „category 3‟ student outcomes, with almost no improvement in those scores in the past ten years 2 . With regard to information literacy specifically, recent studies have found students skip steps in the research process 3 , and have a substantial overreliance on internet search engines 4 and web sites of marginal quality 5-7 Development of Information Literacy Assessment Tool Despite the growing importance of lifelong learning, few standard and easily gradable assessment tools exist to assess the information literacy components of lifelong learning skills, especially in the context of an engineering education. iSkills 8 requires students to carry out a task, such as organizing e-mails, but is expensive and takes a long time to administer, which are often non-starters for educators. The Information Literacy Test (ILT), is another multiple choice test that focuses on knowledge-level performance of students, rather than assessing processes students use 9 . Existing instruments generally involve self-reports rather than objectively measuring life-long learning skills 10 . We developed a multiple choice instrument to measure students‟ information literacy skills and not perceptions. The instrument included a technical memo on the top and asked students to answer eight multiple choice and two open-ended response questions based on the memo‟s arguments. The memo was authentic and included errors as it was written based on actual memos written by first-year engineering students. The technical memo proposed a method that would provide energy savings for the University Residence Halls (see Appendix). This memo included proposed solutions as well as information and citations (websites, blog, published scholarly literature, and informal interviews) used to support the argument made by the author. A table included in the memo also calculated potential savings for the Residence Halls. In this paper, we present results from the multiple choice instrument and provide an evaluation of students‟ information literacy competencies. Using the ILCS framework, objectives for this instrument were designed to assess if students can identify implicit and explicit assumptions, identify and resolve conflicts between information sources (including pre-existing knowledge), evaluate the reliability and relevancy of information sources and use reliable information sources, accurately acknowledge the sources referenced, evaluate overall quality of a written document, determine what is known and what new information is needed to make a strong argument, extract the main points from the information sources and critically evaluate and integrate information from a variety of sources.