Chemical imaging analysis holds great potential in probing the chemical heterogeneity of samples with high spatial resolution and molecular specificity. This paper demonstrates the implementation of Raman mapping for microscopic characterization of tablets containing chloramphenicol palmitate polymorphs with the aid of a new multivariate image segmentation approach based on spatial directed agglomeration clustering. This approach performs the agglomeration clustering by stepwise merging the pixels possessing both spatial closeness and spectral similarity into clusters that define the image segmentation. The incorporation of spatial closeness into the clustering process enables the approach to improve the robustness and avoid poorly defined image segmentation arising from clusters with highly separated pixels. Additionally, the stepwise merging of clusters offers an F-statistic-based procedure to automatically ascertain the number of image segments. Raman mapping analysis of tablets containing two polymorphs of chloramphenicol palmitate followed by multivariate image segmentation reveals that the proposed technique offers the identification of each polymorph and a quantitative visualization of the spatial distribution of the polymorphs identified. This technique holds promise in rapid, noninvasive, and quantitative polymorph analysis for pharmaceutical production processes.