Web hunting: design of a simple intelligent Web search agent

The World Wide Web has become a vast resource of information. The problem is that finding the information that an individual desires is often quite difficult, because of the complexity in organization and the quantity of information stored. Information is stored in uniquely named files located within uniquely named directories on a Website identified by a series of four numbers or a corresponding symbolic name. The number of directories and files is limited only by storage capacity and content availability. Finding the right page, the one containing the information that is sought in one of those uniquely named files, via its Uniform Resource Locator (URL; protocol + host + file -e.g., http://acm.uta. edu/youngblood/index.html) would be nearly impossible if an individual had to guess where the information was located. Fortunately, search engines such as Alta Vista (www.altavista.com), Yahoo (www.yahoo.com), HotBot (www.hotbot.com), and many others have made navigating the Web easier. These services provide users access to large databases containing keyword cross references to URLs. The heart of a search engine is the search agent, which collects the information to be placed in the database.