Is Saki #delicious?: The Food Perception Gap on Instagram and Its Relation to Health

Food is an integral part of our life and what and how much we eat crucially affects our health. Our food choices largely depend on how we perceive certain characteristics of food, such as whether it is healthy, delicious or if it qualifies as a salad. But these perceptions differ from person to person and one person's "single lettuce leaf" might be another person's "side salad". Studying how food is perceived in relation to what it actually is typically involves a laboratory setup. Here we propose to use recent advances in image recognition to tackle this problem. Concretely, we use data for 1.9 million images from Instagram from the US to look at systematic differences in how a machine would objectively label an image compared to how a human subjectively does. We show that this difference, which we call the "perception gap", relates to a number of health outcomes observed at the county level. To the best of our knowledge, this is the first time that image recognition is being used to study the "misalignment" of how people describe food images vs. what they actually depict.

[1]  Christopher D. Manning,et al.  Baselines and Bigrams: Simple, Good Sentiment and Topic Classification , 2012, ACL.

[2]  References , 1971 .

[3]  P. Fletcher,et al.  The Presence of Real Food Usurps Hypothetical Health Value Judgment in Overweight People123 , 2016, eNeuro.

[4]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Adrian David Cheok,et al.  Eating with our eyes: From visual hunger to digital satiation , 2016, Brain and Cognition.

[6]  Guodong Guo,et al.  A computational approach to body mass index prediction from face images , 2013, Image Vis. Comput..

[7]  Xianfeng Zhao,et al.  An adaptive detecting strategy against motion vector-based steganography , 2015, 2015 IEEE International Conference on Multimedia and Expo (ICME).

[8]  C. Fischler,et al.  Food, self and identity , 1988 .

[9]  Captain Y. B. Nusfield.  Public Health , 1906, Canadian Medical Association journal.

[10]  Adrian David Cheok,et al.  Can Food Porn Make Us Slim? How Brains of Consumers React to Food in Digital Environments , 2016 .

[11]  Andrew Zisserman,et al.  Return of the Devil in the Details: Delving Deep into Convolutional Nets , 2014, BMVC.

[12]  Ingmar Weber,et al.  Crowdsourcing Health Labels: Inferring Body Weight from Profile Pictures , 2016, Digital Health.

[13]  Christophe G. Giraud-Carrier,et al.  Identifying Health-Related Topics on Twitter - An Exploration of Tobacco-Related Tweets as a Test Topic , 2011, SBP.

[14]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[15]  Hamed Haddadi,et al.  #FoodPorn: Obesity Patterns in Culinary Interactions , 2015, Digital Health.

[16]  J. Delwiche,et al.  You eat with your eyes first , 2012, Physiology & Behavior.

[17]  Keiji Yanai,et al.  Recognition of Multiple-Food Images by Detecting Candidate Regions , 2012, 2012 IEEE International Conference on Multimedia and Expo.

[18]  Ophir Frieder,et al.  A framework for detecting public health trends with Twitter , 2013, 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM 2013).

[19]  Matthieu Cord,et al.  Recipe recognition with large multimodal food dataset , 2015, 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW).

[20]  Sergio Guadarrama,et al.  Im2Calories: Towards an Automated Mobile Vision Food Diary , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[21]  B. Wansink,et al.  Looks Good Enough to Eat , 2012 .

[22]  T Noakes,et al.  It is time to bust the myth of physical inactivity and obesity: you cannot outrun a bad diet , 2015, British Journal of Sports Medicine.

[23]  Matthieu Guillaumin,et al.  Food-101 - Mining Discriminative Components with Random Forests , 2014, ECCV.

[24]  C. Forde,et al.  Sensory influences on food intake control: moving beyond palatability , 2016, Obesity reviews : an official journal of the International Association for the Study of Obesity.

[25]  Patty Kostkova.  A roadmap to integrated digital public health surveillance: the vision and the challenges , 2013, WWW '13 Companion.

[26]  Jonathan A. Zearfoss,et al.  It tastes as good as it looks! The effect of food presentation on liking for the flavor of food ☆ , 2014, Appetite.

[27]  Patty Kostkova.  Twitter: A Digital Socioscope: Public Health , 2015 .

[28]  Mark Dredze,et al.  How Social Media Will Change Public Health , 2012, IEEE Intelligent Systems.

[29]  Vinod Vokkarane,et al.  DeepFood: Deep Learning-Based Food Image Recognition for Computer-Aided Dietary Assessment , 2016, ICOST.

[30]  Nazanin Andalibi,et al.  Depression-related Imagery on Instagram , 2015, CSCW Companion.

[31]  Elia Gatti,et al.  A taste of Kandinsky: assessing the influence of the artistic visual presentation of food on the dining experience , 2014, Flavour.

[32]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[33]  Ophelia Deroy,et al.  Plating manifesto (II): the art and science of plating , 2014, Flavour.

[34]  Keiji Yanai,et al.  Automatic Expansion of a Food Image Dataset Leveraging Existing Categories with Domain Adaptation , 2014, ECCV Workshops.

[35]  Deborah A Yurgelun-Todd,et al.  Body mass predicts orbitofrontal activity during visual presentations of high-calorie foods , 2005, Neuroreport.

[36]  Venkata Rama Kiran Garimella,et al.  Social Media Image Analysis for Public Health , 2015, CHI.

[37]  Fei-Fei Li,et al.  Deep visual-semantic alignments for generating image descriptions , 2015, CVPR.

[38]  Aron Culotta,et al.  Estimating county health statistics with twitter , 2014, CHI.

[39]  Shengen Yan,et al.  Deep Image: Scaling up Image Recognition , 2015, ArXiv.

[40]  Hamed Haddadi,et al.  Towards Bottom-Up Analysis of Social Food , 2016, Digital Health.

[41]  Keiji Yanai,et al.  FoodCam: A real-time food recognition system on a smartphone , 2015, Multimedia Tools and Applications.

[42]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[43]  Giovanni Maria Farinella,et al.  A Benchmark Dataset to Study the Representation of Food Images , 2014, ECCV Workshops.

[44]  Saeed Abdullah,et al.  Collective Smile: Measuring Societal Happiness from Geolocated Images , 2015, CSCW.

[45]  Thomas Hillman,et al.  Adolescents' presentation of food in social media: An explorative study , 2016, Appetite.

[46]  A. Raben,et al.  Effect of sensory perception of foods on appetite and food intake: a review of studies on humans , 2003, International Journal of Obesity.

[47]  Yukinobu Taniguchi,et al.  Estimating nutritional value from food images based on semantic segmentation , 2014, UbiComp Adjunct.

[48]  Lei Yang,et al.  PFID: Pittsburgh fast-food image dataset , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).

[49]  Hsinchun Chen,et al.  A framework for authorship identification of online messages: Writing-style features and classification techniques , 2006 .

[50]  Michael J. Paul,et al.  Discovering Health Topics in Social Media Using Topic Models , 2014, PloS one.

[51]  Mark Dredze,et al.  You Are What You Tweet: Analyzing Twitter for Public Health , 2011, ICWSM.

[52]  Joy Hirsch,et al.  Leptin reverses weight loss-induced changes in regional neural activity responses to visual food stimuli. , 2008, The Journal of clinical investigation.

[53]  Graciela Gonzalez-Hernandez,et al.  Utilizing social media data for pharmacovigilance: A review , 2015, J. Biomed. Informatics.

[54]  Sofiane Abbar,et al.  Fetishizing Food in Digital Age: #foodporn Around the World , 2016, ICWSM.

[55]  Sofiane Abbar,et al.  You Tweet What You Eat: Studying Food Consumption Through Twitter , 2014, CHI.