Navigational Issues in the Design of On-Line Self-Administered Questionnaires : The Effect of Training and Familiarity

While completing a computerized self-administered questionnaire (CSAQ) for a business or an organization, a respondent may need to navigate between both the survey relevant database and records. Dual navigation refers to the act of navigating information records to retrieve specific information, while simultaneously navigating through the survey. In dual navigation the respondent may be either familiar or unfamiliar with the records on one hand and either familiar or unfamiliar with the survey on the other. As familiarity with the records increase, respondents dual navigation may change. In this study, two organizations were created: the University of Maryland Break Dancing Club and the University of Maryland Knitting Club. Additionally, each had a set of records for 1999 and 2000. Participants were asked to complete the same organizational questionnaire twice for the 1999 Break Dancing Club and the 2000 Knitting Club, the 1999 Knitting Club and the 2000 Break Dancing Club, the 1999 and 2000 Knitting Club, or the 1999 and 2000 Break Dancing Club. Accuracy, completion times, and navigational pattern were all recorded as well as subjective assessments. Completion times and navigational pattern varied between conditions, reflecting a transfer of training from one questionnaire to the next. Conducting surveys of businesses and organizations using self-administered questionnaires pose unique challenges to surveyors. Dillman (2000) notes, “Few survey undertakings are as difficult as defining, sampling, contacting, and obtaining responses to selfadministered questionnaires from businesses or other organizations” (p. 323). When both the survey and the records of the organization are on-line, respondents must often engage in dual navigation of both the records and the questionnaire (Norman, Friedman, Norman, & Stevenson, 2000; Norman, Slaughter, Friedman, Norman, & Stevenson, 2000). Dual navigation refers to the act of navigating information records to retrieve specific information, while simultaneously navigating through the survey. Navigation is facilitated by familiarity with the system and its interface. In dual navigation the respondent may be either familiar or unfamiliar with the records on one hand and either familiar or unfamiliar with the survey on the other. For example, a parttime employee at a small business may be given the task of completing a questionnaire on the general features of their employer. Most likely the records they consult to answer the questions will be new and unknown to the part-time employee. However, business and organizational surveys are often repeated annually or semi-annually, forcing the part-time employee to answer the same questions at every cycle, to maintain a file of essential data, and to become familiar with both the records and the survey. As various researchers have noted (see Lazar & Pierce, 1999; Synodinos, Papacostas, & Okimoto, 1994), the use of the World Wide Web and computerized self-administered questionnaires (CSAQ) has recently become a viable option to conduct survey research. As such, it will be beneficial to investigate the psychological aspects of completing business and/or organization dual navigation CSAQs in order to design better interfaces. Moreover, such research will add to our basic understanding of how respondents plan, learn, and perform question-answering tasks. Using CSAQs and the Web can make the same part-time employee’s job of completing the questionnaire easier because surveys can be formatted to specific organizations, preferences can be stored for later use, or data from electronic files can be automatically copied. Administratively, CSAQs can allow quickly dissemination, retrieval, and tabulation of questionnaire data. However, the use of CSAQs can pose a new challenge for survey designers as respondents search for answers to survey questions from different sources. Respondents may answer questions from memory, from consulting paper records, or increasingly from digital records. This research is an extension to the research of Norman, Friedman, Norman & Stevenson (2000a) and Norman, Slaughter, Friedman, Norman, & Stevenson (2000b). Their research was conducted to assist the U.S. Bureau of the Census in developing principles and guidelines for interactive questionnaires. Specifically, this research investigated the dual navigation of digital organizational records and a CSAQ and the transfer of training from one session to another session. Dual Navigation As previously mentioned, dual navigation refers to the act of navigating information records to retrieve specific information, which answers specific questions from the survey. Dual navigation of records and survey suggest that performance on filling out the questionnaire can be influenced by both the design of the questionnaire and the database. As Norman et al. (2000a) note, “Both internal and external sources are organized in ways that dictate or influence in some way the order of retrieval of information” (p. 3). When there is a match between the order of 1 This would be an external source of memory. As opposed to internal source; where internal source refers to long term memory. questions in a survey and the order or retrieval of information from an external source, processing is expected to be most efficient. When there is a mismatch, processing is less efficient and may result in errors (Norman et al., 2000a). Norman et al. (2000a) demonstrated this by presenting participants with long distantia surveys partitioned in four ways: a) whole/form-based, b) semantic/section-based, c) screen/page-based, and d) single item based. Furthermore, the questionnaires were presented with or without an index. Although neither completion times of the questionnaire nor subjective assessments differed among the eight versions, revision times reflected ease of finding items in the structure of the survey and the use of an index to the sections of the questionnaire. Performance during a dual navigation task can be influenced by the respondent’s familiarity with either the CSAQ or the records. Norman et al. (2000b) classified four possible scenarios for experience of the respondents. A respondent may be a) familiar with the organizational site, but new to the questionnaire, b) familiar with the questionnaire, but the organizational records are unfamiliar, c) familiar with both the questionnaire and the organizational records, or d) unfamiliar with both the questionnaire and the organizational records. As respondents interact with the records and the CSAQ, we hypothesize that they form a mental representation of the questionnaire and the database together. This representation could lead respondents to complete the questionnaire in a nonlinear pattern the next time they saw the questionnaire and a similar record. Figure 1 shows a schematic of the first time participants complete a dual navigation task. They do not have a mental representation of the record or the questionnaire. Consequently, they answer the survey in a linear format and navigate through the 2 The respondents were asked to revisit 16 questions based on only the topic of the question or on the topic and the question number and to change their answers. record to answer each question. Figure 2 illustrates the second time participants complete the dual navigation task with a database and questionnaire they are familiar with. They are hypothesized to have formed a mental representation of the records and the locations of the specific questions to be answered. Thus, they navigate through the survey in a nonlinear pattern as they answer several questions in different parts of the questionnaire that correspond to the current record they are viewing. Figure 1: Schematic of person completing randomly ordered questionnaire by accessing information from records. There is no mental representation of either, because it is the first time completing the dual navigation task. Figure 2: Schematic of person completing randomly ordered questionnaire by accessing information from records from which they are familiar with. There is a mental representation of both the questionnaire and the records, which assists their completion of the dual navigation task. An important question to be answered is what is the base for the mental representation of the participant. For example, we hypothesized that respondents in our task would use the record as a base mental representation. Thus, the questionnaire items are mapped onto specific areas of the record within the mental representation. If respondents used the questionnaire as a base, the nodes of information would be mapped onto the questionnaire. We hypothesized that subjects would use the database as a base for their mental representation because the record is the source of data and is the external store of information. Participants should view the questionnaire as referring to the record, not the other way around. Survey Design Lazar and Preece (1999) state, “Web-based surveys must follow many of the same rules for survey design as traditional paper surveys” (p. 63). For example, all survey researchers should verify that the questions are clear and well written by pilot testing the survey. However, the use of CSAQs necessitates the understanding and implementation of the issues and factors involved in the computer interface environment (Norman et al., 2000a). One aspect of interface design is the actual form of the questionnaire. Norman et al. (2000a) conceptualized the design of questionnaires as existing on a design continuum. At one end of the design continuum are form-based designs that present questionnaires as one long form in a scrollable window. At the other end of the design continuum are item-based questionnaires that present only a single item at a time. Additionally, each of the different designs from the continuum could also contain an index to the questions. An index allows respondents to navigate throu