Evaluating municipal websites: A methodological comparison of three think-aloud variants

Usability methods have received relatively little methodological attention within the field of E-Government. This paper aims to address this gap by reporting on a usability test of the municipal website of Deventer (the Netherlands), carried out by means of three variants of the think-aloud method (concurrent/retrospective think-aloud protocols and constructive interaction). These three methods had proved successful in a previous evaluation of a different municipal website, yet we decided to replicate our study in order to investigate whether the three methods would reveal different results when applied to another municipal website with a different information architecture. The results of our study showed that, as in the previous municipal website evaluation, the three evaluation methods were largely comparable in terms of output. Nevertheless, we did find a number of differences between the present and previous municipal website evaluation regarding the workings of the three methods—differences that could be explained by the different information architectures of the municipal websites tested. This suggests that the three evaluation methods might indeed work differently depending on the nature of the website that is being evaluated, and calls for more research into the effect of task type on the validity of evaluation methods.

[1]  Janet Kaaya,et al.  Implementing e-Government Services in East Africa : Assessing Status through Content Analysis of Government Websites , 2004 .

[2]  Debora Shaw,et al.  Handbook of usability testing: How to plan, design, and conduct effective tests , 1996 .

[3]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[4]  M. D. de Jong,et al.  Retrospective vs. concurrent think-aloud protocols: Testing the usability of an online library catalogue , 2003 .

[5]  Kuno Schedler,et al.  Customer orientation in electronic government: Motives and effects , 2007, Gov. Inf. Q..

[6]  Andrew Potter.  Accessibility of Alabama government Web sites , 2002 .

[7]  N. Ummelen,et al.  Measuring reading behavior in policy documents: a comparison of two instruments , 2000 .

[8]  R. R. Brown.  Effect of age on the speed-power relationship with reference to tests of intelligence. , 1938 .

[9]  E. Welch,et al.  Virtual Interactivity Between Government and Citizens: The Chicago Police Department's Citizen ICAM Application Demonstration Case , 2005 .

[10]  Joseph F. Dumas,et al.  A Practical Guide to Usability Testing , 1993 .

[11]  Leo Lentz,et al.  Scenario evaluation of municipal Web sites: Development and use of an expert-focused evaluation tool , 2006, Gov. Inf. Q..

[12]  K. A. Ericsson,et al.  Protocol Analysis: Verbal Reports as Data , 1984 .

[13]  Leo Lentz,et al.  Municipalities on the Web: User-Friendliness of Government Information on the Internet , 2006, EGOV.

[14]  Paul T. Jaeger,et al.  Assessing Section 508 compliance on federal e-government Web sites: A multi-method, user-centered evaluation of accessibility for persons with disabilities , 2006, Gov. Inf. Q..

[15]  Eric J. Johnson,et al.  The validity of verbal protocols , 1989, Memory & cognition.

[16]  Cor Aarnoutse,et al.  Third-grader's think-aloud protocols : types of reading activities in reading an expository text , 2006 .

[17]  Kyung-Sun Kim,et al.  Information-seeking on the Web: Effects of user and task variables , 2001 .

[18]  Edward F. Halpin,et al.  An Exploratory Evaluation of UK Local e-Government From an Accountability Perspective , 2005 .

[19]  Maarten van Someren,et al.  The Think Aloud Method: A Practical Guide to Modelling Cognitive Processes , 1994 .

[20]  Jean-Paul Dionne,et al.  Accessing Problem-Solving Strategy Knowledge: The Complementary Use of Concurrent Verbal Protocols and Retrospective Debriefing. , 2000 .

[21]  Jyoti Choudrie,et al.  Evaluating global e-government sites: A view using web diagnostics tools , 2004 .

[22]  Peter Jan Schellens,et al.  Evaluation of an Informational Web Site: Three Variants of the Think-aloud Method Compared , 2007 .

[23]  Richard Heeks,et al.  Analyzing e-government research: Perspectives, philosophies, theories, methods, and practice , 2007, Gov. Inf. Q..

[24]  Graeme Baxter,et al.  The effectiveness of parliamentary information services in the United Kingdom , 2003, Gov. Inf. Q..

[25]  Rowena Cullen,et al.  Democracy online: an assessment of New Zealand government web sites , 2000, Gov. Inf. Q..

[26]  C. May,et al.  Interaction between States and Citizens in the Age of the Internet: “e-Government” in the United States, Britain, and the European Union , 2003 .

[27]  J. F. Schneider,et al.  Exploring Ease in Thinking Aloud , 2006, Psychological reports.

[28]  K. Taylor,et al.  Accessing Problem-Solving Strategy Knowledge: The Complementary Use of Concurrent Verbal Protocols and Retrospective Debriefing. , 2000 .

[29]  A. Wong,et al.  Writers’ mental representations of the intended audience and of the rhetorical purpose for writing and the strategies that they employed when they composed , 2005 .

[30]  Kristin R. Eschenfelder,et al.  Examining the role of Web site information in facilitating different citizen-government relationships: A case study of state Chronic Wasting Disease Web sites , 2007, Gov. Inf. Q..

[31]  Sayyed Mohammad Hadi Alavi.  On the adequacy of verbal protocols in examining an underlying construct of a test , 2005 .

[32]  Kristin R. Eschenfelder,et al.  Behind the Web site: An inside look at the production of Web-based textual government information , 2004, Gov. Inf. Q..

[33]  A. Ek,et al.  Nurses' reasoning process during care planning taking pressure ulcer prevention as an example. A think-aloud study. , 2007, International journal of nursing studies.

[34]  Menno D. T. de Jong,et al.  Employing think-aloud protocols and constructive interaction to test the usability of online library catalogues: a methodological comparison , 2004, Interact. Comput..

[35]  David K. Farkas,et al.  Guidelines for Designing Web Navigation , 2000 .

[36]  Yuquan Shi,et al.  The accessibility of Chinese local government Web sites: An exploratory study , 2007, Gov. Inf. Q..

[37]  Jan van Dijk,et al.  Personalization in the public sector: An inventory of organizational and user obstacles towards personalization of electronic services in the public sector , 2007, Gov. Inf. Q..