The pragmatics of information retrieval experimentation

The novice information scientist, though he or she may have thoroughly studied the design and results of previous information retrieval tests and clearly described the purpose of his/her own test, may still, when faced with its implementation, have great difficulty in proceeding. Early information retrieval experiments were of necessity ad hoc, and it is only in recent years that a body of practice, based on the experiences of Cleverdon and later investigators, has made possible a few recommendations on the pragmatics of conducting information retrieval experiments. The following remarks, though based to some extent on a study of the major tests, including those described in later chapters of this book, are heavily dependent on the author's own trials, tribulations, and mistakes. If there is one lesson to be learned from experience, it is that the theoretically optimum design can never be achieved, and the art of information retrieval experimentation is to make the compromises that will least detract from the usefulness of the results. In determining experimental procedures, three aspects must be kept in mind: