Semi-supervised annotation of brushwork in paintings domain using serial combinations of multiple experts

Many recent studies perform annotation of paintings based on brushwork. They model the brushwork indirectly as part of annotation of high-level artistic concepts such as artist name using low-level texture features and supervised inference methods. In this paper, we develop a framework for explicit annotation of paintings with brushwork classes. Brushwork classes serve as meta-level semantic concepts for artist names, paintings styles and periods of art and facilitate the incorporation of domain-specific ontologies. In particular, we employ the serial multi-expert framework with semi-supervised clustering methods to perform the annotation of brushwork patterns. Serial combination of multiple experts facilitates step-wise refinement of decisions based on the preferences of individual experts. Each individual expert performs focused subtasks using relevant feature set, which decreases the 'curse of dimensionality' and noise in the feature space. Each expert focuses on the annotation of the currently available samples from its unlabeled pool using semi-supervised agglomerative clustering. This approach is more appropriate as compared to the traditional classification methods since each brushwork class includes a variety of patterns and cannot be represented as a single distribution in the feature space. The experts exploit the distribution of unlabelled patterns and further minimize the annotation error. The multi-expert semi-supervised framework out-performs the conventional methods in annotation of patterns with brushwork classes. This framework will further be adopted to facilitate ontology-based annotation with higher-level semantic concepts such as the artist names, painting styles and periods of art.

[1]  Eric O. Postma,et al.  Discovering the Visual Signature of Painters , 2000 .

[2]  Fuad Rahman,et al.  Serial Combination of Multiple Experts: A Unified Evaluation , 1999, Pattern Analysis & Applications.

[3]  Dan Klein,et al.  From Instance-level Constraints to Space-Level Constraints: Making the Most of Prior Knowledge in Data Clustering , 2002, ICML.

[4]  Anthony K. H. Tung,et al.  Constraint-based clustering in large databases , 2001, ICDT.

[5]  Tat-Seng Chua,et al.  A learning-based approach for annotating large on-line image collection , 2004, 10th International Multimedia Modelling Conference, 2004. Proceedings..

[6]  Guojun Lu,et al.  Content-based shape retrieval using different shape descriptors: a comparative study , 2001, IEEE International Conference on Multimedia and Expo, 2001. ICME 2001..

[7]  C.-C. Jay Kuo,et al.  Texture Roughness Analysis and Synthesis via Extended Self-Similar (ESS) Model , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  M. Yelizaveta,et al.  Analysis and Retrieval of Paintings Using Artistic Color Concepts , 2005, 2005 IEEE International Conference on Multimedia and Expo.

[9]  James Ze Wang,et al.  Studying digital imagery of ancient paintings by mixtures of stochastic models , 2004, IEEE Transactions on Image Processing.

[10]  Tat-Seng Chua,et al.  Content-based retrieval of segmented images , 1994, MULTIMEDIA '94.

[11]  R. Arnheim,et al.  Art and Visual Perception: A Psychology of the Creative Eye. , 1956 .

[12]  M. Teague Image analysis via the general theory of moments , 1980 .

[13]  J. Kittler,et al.  Multistage pattern recognition with reject option , 1992, Proceedings., 11th IAPR International Conference on Pattern Recognition. Vol.II. Conference B: Pattern Recognition Methodology and Systems.

[14]  Thomas Melzer,et al.  Stroke detection of brush strokes in portrait miniatures using a semi-parametric and a model based approach , 1998, Proceedings. Fourteenth International Conference on Pattern Recognition (Cat. No.98EX170).

[15]  Wolfgang Doster,et al.  A decision theoretic approach to hierarchical classifier design , 1984, Pattern Recognit..

[16]  Ramesh C. Jain,et al.  Representation and retrieval of paintings based on art history concepts , 2004, 2004 IEEE International Conference on Multimedia and Expo (ICME) (IEEE Cat. No.04TH8763).

[17]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  B. S. Manjunath,et al.  Texture Features for Browsing and Retrieval of Image Data , 1996, IEEE Trans. Pattern Anal. Mach. Intell..

[19]  Stéphane Mallat,et al.  A Theory for Multiresolution Signal Decomposition: The Wavelet Representation , 1989, IEEE Trans. Pattern Anal. Mach. Intell..

[20]  V. Vapnik Estimation of Dependences Based on Empirical Data , 2006 .

[21]  Herbert A. Glucksman Multicategory Classification of Patterns Represented by High-Order Vectors of Multilevel Measurements , 1971, IEEE Transactions on Computers.

[22]  B. Mandelbrot Fractal Geometry of Nature , 1984 .

[23]  Jerome H. Friedman,et al.  An Overview of Predictive Learning and Function Approximation , 1994 .