Sampling the Web: The Development of a Custom Search Tool for Research

Research designed to study the Internet is beset with challenges. One of these challenges involves obtaining samples of Web pages. Methodologies used in previous studies may be categorized into random, purposeful, and purposeful random types of sampling. This paper contains an outline of these methodologies and information about the development of a custom sampling tool that may be used to obtain purposeful random samples of Web page links. The custom search application called Web Sampler works through the Google Web APIs service to collect a random sample of pages from search results returned from the Google index. Web Sampler is inexpensive to develop and may be easily customized for specialized search needs required by researchers who are investigating Web page content. The Internet is a vast network of interconnected computers that supports rapid access to and exchange of digitally encoded information including e-mail and Web pages. Working on top of the Internet is the World Wide Web, which is a hypertext system that enables retrieval and display of Web pages through the use of browser software. During the years since the idea for the Web was initially conceptualized by Berners Lee (1989, 1990), it has grown to include several billion publicly accessible pages that may be indexed by search engines (Gulli & Signorini, 2005). The Web has become a huge gateway to information that may be accessed by anyone with a computer, Web browser software, and Internet access. The quality and scope of information available on the Web is not fully documented through research due to the continual growth and dynamic nature of Web-based content. Web research comes with its own set of unique methodological challenges such as access issues, population definitions, sampling procedures, and selection or development of appropriate software applications for individual research needs. The purpose of this paper is to articulate the challenges associated with Web research, summarize several strategies that have been used to obtain samples of Web pages in previous studies, and describe the development of a custom search application designed to obtain purposeful random samples of Web pages. The Challenges of Web Research A variety of challenges need to be addressed in any research study designed to explore the Web. The process of finding solutions to these challenges may involve an evolution of traditional research methodologies. Consider the basic research procedure of defining a population and selecting a suitable sample to study. In Web-based research, complex problems emerge when defining populations or selecting samples. For example, if the population is defined to be all existing Web pages, then access problems are encountered. A portion of the total population of all existing Web pages is private, and public access is restricted. If the private pages are eliminated from consideration for research, then the population under study may be redefined as the set of Web pages open to the general public. Samples may be collected from the set of public Web pages without access problems since they are available to everyone. Unfortunately, removing the access barrier does not solve the entire sampling problem. Web page sampling has not proven to be simple or straightforward. Several variations of sampling procedures have been used or proposed in order to extract information from the Web (Henzinger & Lawrence, 2004). As shown in Table 1, an examination of sampling techniques that have been used in Web research reveals that they tend to fall within the broad categories of random sampling, purposive sampling, and purposive random sampling.