A fuzzy rule based multimodal framework for face sketch-to-photo retrieval

Abstract Facial sketches are widely used by law enforcement agencies to assist in the identification and apprehension of suspects involved in criminal activities. Sketches used in forensic investigations are drawn by forensic artists following the verbal description provided by an eyewitness or the victim. In a forensic sketch, the facial description depends on the memory of the eyewitness, therefore, there is uncertainty in facial attributes. Most of the existing sketch-photo retrieval methods have ignored the uncertainty of facial attributes. In this paper, we have proposed a novel multimodal framework for sketch-photo retrieval. We have fused the facial attributes and the semantic color information using a fuzzy rule based layered classifier. We have also performed a survey to rank the importance of different facial attributes in the identification of a person. We have performed experiments using viewed, semi-forensic and forensic sketches dataset and results are compared with state-of-the-art methods. Experimental results show that the proposed fuzzy rule based method achieves promising performance as compared to state-of-art methods specifically in case of forensic sketches.