ALIWEB - Archie-like Indexing in the WEB

Abstract ALIWEB is a framework for automatic collection and processing of resource indices in the World Wide Web. The current ALIWEB implementation regularly retrieves index files from many servers in the Web and combines them into a single searchable database. Using existing Web protocols and a simple index file format, server administrators can have descriptive and up-to-date information about their services incorporated into the ALIWEB database with little effort. As the indices are single files there is little overhead in the collection process and because the files are prepared for the purpose the resulting database is of high quality. This paper discusses the background and objectives of ALIWEB and gives an overview of its functionality and implementation, comparing it to other existing resource directories in the Web. It reviews the experiences with the first months of operation, and suggests possible future directions of ALIWEB.