Usenetfs: A Stackable File System for Large Article Directories

The Internet has grown much in popularity in the past few years. Numerous users read USENET newsgroups daily for entertainment, work, study, and more. USENET News servers have seen a gradual increase in the traffic exchanged between them, to a point where the hardware and software supporting the servers is no longer capable of meeting demand, at which point the servers begin “dropping” articles they could not process. The rate of this increase has been faster than software or hardware improvements were able to keep up, resulting in much time and effort spent by administrators upgrading their news systems. One of the primary reasons for the slowness of news servers has been the need to process many articles in very large flat directories representing newsgroups such as control.cancel and misc.jobs.offered. A large portion of the resources is spent on processing articles in these few newsgroups. Most Unix directories are organized as a linear unsorted sequence of entries. Large newsgroups can have hundreds of thousands of articles in one directory, resulting in significant delays processing any single article. Usenetfs is a file system that rearranges the directory structure from being flat to one with small directories containing fewer articles. By breaking the structure into smaller directories, it improves the performance of looking for, creating, or deleting files, since these operations occur on smaller directories. Usenetfs takes advantage of article numbers; knowing that file names representing articles are composed of digits helps to bound the size of the smaller directories. Usenetfs improves overall performance by at least 22% for average news servers; common news server operations such as looking up, adding, and deleting articles are sped up by as much as several orders of magnitude. Usenetfs was designed and implemented as a stackable Vnode layer loadable kernel module[Heidemann94, Rosenthal92, Skinner93]. It operates by “encapsulating” a client file system with a layer of directory management. To the process performing directory operations through a mounted Usenetfs, all directories appear flat; but when inspecting the underlying storage that it manages, small directories are visible. Usenetfs is small and is transparent to the user. It requires no change to News software, to other file systems, or to the rest of the operating system. Usenetfs is more portable than other native kernel-based file systems because it interacts with the Vnode interface which is similar on many different platforms.