Abstract This research work introduces and describes a robust method for extracting harmonic color features (HCFs) and verifies its validity by predicting visual aesthetics of a large image dataset against a large human survey on the same dataset. This work is a continuation of our previous research (Firoze et al. [13] ) where we demonstrated a machine’s capability of understanding aesthetics with respect to the rule of thirds (Mai et al. [14] ). In this research, we have successfully devised a method of extracting HCFs and trained a model that can perceive beauty with a root mean squared error of 1.115 ± 0.196 on a scale of 0 to 5. In contrast to classic segmentation approaches, we have used HSV color scale gradients and differentials to extract the HCFs. Due to reduced computations, differential color harmony is quite suitable for big data and fast computing. We have used a large dataset of 5000 images from the standard MIRFLICKR (Huiskes and Lew [12] ) dataset and conducted a survey where participants measured the beauty of these images. We used these data to train classifiers and regression models, and we verified our approach by comparing machine perceived beauty against human perceived beauty.