Artificial Neural Networks and Deep Learning in the Visual Arts: a review

In this article, we perform an exhaustive analysis of the use of Artificial Neural Networks and Deep Learning in the Visual Arts. We begin by introducing changes in Artificial Intelligence over the years and examine in depth the latest work carried out in prediction, classification, evaluation, generation, and identification through Artificial Neural Networks for the different Visual Arts. While we highlight the contributions of photography and pictorial art, there are also other uses for 3D modeling, including video games, architecture, and comics. The results of the investigations discussed show that the use of Artificial Neural Networks in the Visual Arts continues to evolve and have recently experienced significant growth. To complement the text, we include a glossary and table with information about the most commonly employed image datasets.

[1]  Qinghua Huang,et al.  Synthesized computational aesthetic evaluation of photos , 2016, Neurocomputing.

[2]  Antonino Santos,et al.  Computerized measures of visual complexity. , 2015, Acta psychologica.

[3]  Xiangmin Xu,et al.  A multi-scene deep learning model for image aesthetic evaluation , 2016, Signal Process. Image Commun..

[4]  Zhou Wang,et al.  Blind Image Quality Assessment Using a Deep Bilinear Convolutional Neural Network , 2019, IEEE Transactions on Circuits and Systems for Video Technology.

[5]  Tristan Cazenave.  Residual Networks for Computer Go , 2018, IEEE Transactions on Games.

[6]  Alan C. Bovik,et al.  Massive Online Crowdsourced Study of Subjective and Objective Picture Quality , 2015, IEEE Transactions on Image Processing.

[7]  François Lemarchand,et al.  Fundamental visual features for aesthetic classification of photographs across datasets , 2018, Pattern Recognit. Lett..

[8]  Sonja Grgic,et al.  A Deep Learning Perspective on Beauty, Sentiment, and Remembrance of Art , 2019, IEEE Access.

[9]  Fernando Morgado Dias,et al.  Artificial neural networks: a review of commercial hardware , 2004, Eng. Appl. Artif. Intell..

[10]  Antonio Torralba,et al.  Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope , 2001, International Journal of Computer Vision.

[11]  Guy Lapalme,et al.  A systematic analysis of performance measures for classification tasks , 2009, Inf. Process. Manag..

[12]  Biswajeet Pradhan,et al.  Review of Traffic Accident Predictions with Neural Networks , 2019, Laser Scanning Systems in Highway and Safety Assessment.

[13]  James Zijun Wang,et al.  Rating Image Aesthetics Using Deep Learning , 2015, IEEE Transactions on Multimedia.

[14]  Nicolas A. Barriga.  A Short Introduction to Procedural Content Generation Algorithms for Videogames , 2019, Int. J. Artif. Intell. Tools.

[15]  M. Kramer.  Nonlinear principal component analysis using autoassociative neural networks , 1991 .

[16]  Jon McCormack,et al.  Understanding Aesthetics and Fitness Measures in Evolutionary Art Systems , 2019, Complex..

[17]  Kede Ma,et al.  Waterloo Exploration Database: New Challenges for Image Quality Assessment Models. , 2017, IEEE transactions on image processing : a publication of the IEEE Signal Processing Society.

[18]  Peyman Milanfar,et al.  NIMA: Neural Image Assessment , 2017, IEEE Transactions on Image Processing.

[19]  Michael Felsberg,et al.  Painting-91: a large scale database for computational painting categorization , 2014, Machine Vision and Applications.

[20]  Jan-Michael Frahm,et al.  USAC: A Universal Framework for Random Sample Consensus , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[21]  L. D. Couprie.  Iconclass: an iconographic classification system , 1983 .

[22]  F ROSENBLATT,et al.  The perceptron: a probabilistic model for information storage and organization in the brain. , 1958, Psychological review.

[23]  Frédéric Jurie,et al.  New public dataset for spotting patterns in medieval document images , 2016, J. Electronic Imaging.

[24]  Julian Togelius,et al.  Neuroevolution in Games: State of the Art and Open Challenges , 2014, IEEE Transactions on Computational Intelligence and AI in Games.

[25]  Tao Mei,et al.  Query-Dependent Aesthetic Model With Deep Learning for Photo Quality Assessment , 2015, IEEE Transactions on Multimedia.

[26]  Margaret Lech,et al.  Two-Stage Deep Learning Approach to the Classification of Fine-Art Paintings , 2019, IEEE Access.

[27]  Kaiqi Huang,et al.  Hierarchical aesthetic quality assessment using deep convolutional neural networks , 2016, Signal Process. Image Commun..

[28]  Richard Lippmann,et al.  Review of Neural Networks for Speech Recognition , 1989, Neural Computation.

[29]  A. Elgammal.  AI Is Blurring the Definition of Artist , 2019, American Scientist.

[30]  Mei-Chen Yeh,et al.  Artist-based painting classification using Markov random fields with convolution neural network , 2020, Multimedia Tools and Applications.

[31]  Jon McCormack,et al.  Complex Systems in Aesthetics and Arts , 2019, Complex..

[32]  Juan Romero,et al.  Transfer learning features for predicting aesthetics through a novel hybrid machine learning method , 2019, Neural Computing and Applications.

[33]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[34]  Tom White,et al.  Generative Adversarial Networks: An Overview , 2017, IEEE Signal Processing Magazine.

[35]  Kunihiko Fukushima,et al.  Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position , 1980, Biological Cybernetics.

[36]  Hongping Cai,et al.  Cross-depiction problem: Recognition and synthesis of photographs and artwork , 2015, Computational Visual Media.

[37]  Xinmei Tian,et al.  Multi-level photo quality assessment with multi-view features , 2015, Neurocomputing.

[38]  Fei Gao,et al.  DeepSim: Deep similarity for image quality assessment , 2017, Neurocomputing.

[39]  H. J. Eysenck,et al.  Comparative study of artists and nonartists on the maitland graves design judgment test. , 1971 .

[40]  Eric C. Larson,et al.  Most apparent distortion: full-reference image quality assessment and the role of strategy , 2010, J. Electronic Imaging.

[41]  Eric O. Postma,et al.  Toward Discovery of the Artist's Style: Learning to recognize artists by their artworks , 2015, IEEE Signal Processing Magazine.

[42]  J. Friedman.  Special Invited Paper-Additive logistic regression: A statistical view of boosting , 2000 .

[43]  Alan Dorin,et al.  Artificial Life Art, Creativity, and Techno-hybridization (editor's introduction) , 2015, Artificial Life.

[44]  Penousal Machado,et al.  Distinguishing paintings from photographs by complexity estimates , 2016, Neural Computing and Applications.

[45]  G. Greenfield,et al.  Guest editors' introduction , 2012 .

[46]  Lawrence D. Jackel,et al.  Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.

[47]  K. O. Götz,et al.  The Maitland Graves Design Judgment Test Judged by 22 Experts , 1974 .

[48]  Zunlei Feng,et al.  Neural Style Transfer: A Review , 2017, IEEE Transactions on Visualization and Computer Graphics.

[49]  David G. Lowe,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004, International Journal of Computer Vision.

[50]  Eero P. Simoncelli,et al.  A Parametric Texture Model Based on Joint Statistics of Complex Wavelet Coefficients , 2000, International Journal of Computer Vision.

[51]  Tinne Tuytelaars,et al.  Location recognition over large time lags , 2014, Comput. Vis. Image Underst..

[52]  Yimin Zhou,et al.  Photograph aesthetical evaluation and classification with deep convolutional neural networks , 2017, Neurocomputing.

[53]  A. Reiterer,et al.  Application of artificial neural networks for automated analysis of cystoscopic images: a review of the current status and future prospects , 2020, World Journal of Urology.

[54]  Calvin N. Mooers.  Preventing Software Piracy , 1977, Computer.

[55]  Wangmeng Zuo,et al.  Attention-guided CNN for image denoising , 2020, Neural Networks.

[56]  Luming Zhang,et al.  Media Quality Assessment by Perceptual Gaze-Shift Patterns Discovery , 2017, IEEE Transactions on Multimedia.

[57]  Hannu Toivonen,et al.  Data mining and machine learning in computational creativity , 2015, Wiley Interdiscip. Rev. Data Min. Knowl. Discov..

[58]  Alan C. Bovik,et al.  A Statistical Evaluation of Recent Full Reference Image Quality Assessment Algorithms , 2006, IEEE Transactions on Image Processing.

[59]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[60]  Wenhua Qian,et al.  Image aesthetic quality evaluation using convolution neural network embedded learning , 2017 .

[61]  Soteris A. Kalogirou,et al.  Artificial neural networks in renewable energy systems applications: a review , 2001 .

[62]  Xun Xu,et al.  Visual aesthetic understanding: Sample-specific aesthetic classification and deep activation map visualization , 2018, Signal Process. Image Commun..

[63]  Penousal Machado,et al.  The Art of Artificial Evolution , 2008 .

[64]  Yimin Zhou,et al.  Computational aesthetics of photos quality assessment based on improved artificial neural network combined with an autoencoder technique , 2016, Neurocomputing.

[65]  Haisheng Li,et al.  3D Model Generation and Reconstruction Using Conditional Generative Adversarial Network , 2019, Int. J. Comput. Intell. Syst..

[66]  Kiyoharu Aizawa,et al.  Sketch-based manga retrieval using manga109 dataset , 2015, Multimedia Tools and Applications.

[67]  Kim Young-Min.  Feature visualization in comic artist classification using deep neural networks , 2019 .

[68]  Nikolay N. Ponomarenko,et al.  Image database TID2013: Peculiarities, results and perspectives , 2015, Signal Process. Image Commun..

[69]  Jean-Christophe Burie,et al.  Digital Comics Image Indexing Based on Deep Learning , 2018, J. Imaging.

[70]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[71]  Le Wu,et al.  IDEA: A new dataset for image aesthetic scoring , 2018, Multimedia Tools and Applications.