The Power of the Crowd: Performing Usability Testing Using an On-Demand Workforce

For many business organizations, Web sites are an important tool for attracting new customers or to advertise or sell their products and services. Therefore, any problems in Web site usability can directly influence a company’s bottom line. Nevertheless, despite this importance, examples of poorly designed Web pages abound, and often, such poor design is blamed on not having paid attention to thorough usability testing. Whereas even a few tests are better than none at all, especially smaller organizations lack the resources to perform multiple rounds of in-depth user studies. Recently, crowdsourcing has become a popular way of recruiting an on-demand workforce for a variety of tasks. Our study demonstrates how crowdsourcing can be used as a relatively easy and inexpensive way to recruit participants for usability tests and suggests ways to overcome the lacking possibility to observe participants during the testing process. Further, we demonstrate that the crowds were able to detect at least as many usability problems as experts asked to evaluate the same sites.