Interactive Learning Using a "Society of Models"

Digital library access is driven by features but features are often context dependent and noisy and their relevance for a query is not always obvious This paper describes an approach for utilizing many data dependent user dependent and task dependent features in a semi automated tool Instead of requiring universal similarity measures or manual selection of relevant features the approach provides a learning algorithm for selecting and combining groupings of the data where groupings can be induced by highly spe cialized and context dependent features The se lection process is guided by a rich example based interaction with the user The inherent com binatorics of using multiple features is reduced by a multistage grouping generation weighting and collection process The stages closest to the user are trained fastest and slowly propagate their adaptations back to earlier stages The weighting stage adapts the collection stage s search space across uses so that in later interactions good groupings are found given few examples from the user Described is an interactive time imple mentation of this architecture for semi automatic within image segmentation and across image la beling driven by concurrently active color mod els texture models or manually provided group ings Issues for digital libraries Digital libraries of images video and sound are a rich area for pattern recognition research They also introduce a host of new problems and requirements since the range of possible queries is immense and requires the utilization of many spe cialized features Also systems for retrieval browsing and annotation i e classifying regions often must perform with only a small number of examples from a user i e an insuf cient amount of training data by traditional requirements Thus the area is doubly exciting since it presents the eld of pattern recognition with new challenges while beckoning in new applications One important issue for digital libraries is nding good models and similarity measures for comparing database en tries A part of this di culty is that feature extraction and comparison methods are highly data dependent see Figure This work was supported in part by BT PLC Hewlett Packard Labs and NEC for an example with texture Similarity measures are also user and task dependent as demonstrated by Figure Un fortunately these dependencies are not at this point under stood well enough especially by the typical digital library user to permit careful selection of the optimal measure be forehand Note that the multi resolution simultaneous auto regressive MRSAR model of which fares poorly com pared to the shift invariant eigenvector EV model in the above two examples scores clearly above the EV model on the standard Brodatz database On the same test data but for a perceptually motivated similarity criteria based on periodicity directionality and randomness both the EV and MRSAR models are beat by a new Wold based model Attempts to use intuitive texture features like coarseness contrast and directionality are appropri ate in some cases but do not fully determine all the qualities people might use in judging similarity Thus an a priori opti mal context dependent selection among similarity measures either by human or computer seems unlikely Next the scope of queries that databases need to address is immense Current computational solutions attempt to of fer location of perceptual content nd round red objects and objective content nd pictures of people in Boston Desirable queries also extend to subjective content give me a scene of a romantic forest task speci c content I need something with open space to place text collaborative con tent show me pictures children like and more An swering such queries requires a variety of features or meta data to be attached to the data in a digital library some of which may not be computable directly from the data The implication for algorithms is that they cannot rely on one model or one small set of carefully picked features but will have to drink from a veritable feature hydrant from which only a few drops may be relevant for the query Finally there is a signi cant need for semi automated ver sus fully automated tools Human computer synergy can make ill de ned tasks manageable and has the power to over come many of the problems of current pattern recognition tools An important application of semi automated tools is to assist the population of a database viz the creation of metadata A crucial technical issue for such tools is the selec tion and combination of existing features which features are most useful for a given query or annotation how should they be combined and which combinations are useful for the sys tem to remember so that it gets smarter with increased use This last point is important since not only are the queries immensely variable but the amount of training data i e ex amples provided by a user of what they do and don t want available at any instant is usually limited Hence a tool should strive to improve its generalization ability