Searching in document images: what does the appearance of a document tell us about what it means?

The document understanding problem can be informally defined as the automatic extraction of meaning from documents. In the Intelligent Sensory Information Systems group we have experimented with analyzing the visual appearance of documents in order to extract meaning. That is, we concentrate on how documents look, rather than on what they say. We motivate this approach with several applications from document image understanding. First, we describe how document genre classification can be used to group visually similar documents together, which simplifies the analysis task for an entire class of documents. Second, we consider the logical block labeling problem. We show how logical labels (e.g. title, author, header, footer) can be assigned to blocks of text using a few visual features. Third, we discuss our approach to detecting the reading order of text using the visual structure of a document. The examples are based on the work in the field of content-based image retrieval (CBIR). Content-based image retrieval aims at searching and browsing image repositories on the basis of a visual specification of the query. The query may be one or preferably more examples, and the presentation may be a linear list of items, or prefarably a similarity grouping. Our research on colour representations of real world objects and colour composition of documents has shown that CBIR techniques can be successfully applied in order to simplify the document understanding problem.