Towards Bottom-Up Analysis of Social Food

Social media provide a wealth of information for research into public health by providing a rich mix of personal data, location, hashtags, and social network information. Among these, Instagram has been recently the subject of many computational social science studies. However despite Instagram's focus on image sharing, most studies have exclusively focused on the hashtag and social network structure. In this paper we perform the first large scale content analysis of Instagram posts, addressing both the image and the associated hashtags, aiming to understand the content of partially-labelled images taken in-the-wild and the relationship with hashtags that individuals use as noisy labels. In particular, we explore the possibility of learning to recognise food image content in a data driven way, discovering both the categories of food, and how to recognise them, purely from social network data. Notably, we demonstrate that our approach to food recognition can often achieve accuracies greater than 70% in recognising popular food-related image categories, despite using no manual annotation. We highlight the current capabilities and future challenges and opportunities for such data-driven analysis of image content and the relation to hashtags.

[1]  Hamed Haddadi,et al.  #FoodPorn: Obesity Patterns in Culinary Interactions , 2015, Digital Health.

[2]  Xinlei Chen,et al.  NEIL: Extracting Visual Knowledge from Web Data , 2013, 2013 IEEE International Conference on Computer Vision.

[3]  Subbarao Kambhampati,et al.  What We Instagram: A First Analysis of Instagram Photo Content and User Types , 2014, ICWSM.

[4]  Antonio Torralba,et al.  Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope , 2001, International Journal of Computer Vision.

[5]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[6]  Munmun De Choudhury,et al.  Measuring and Characterizing Nutritional Information of Food and Ingestion Content in Instagram , 2015, WWW.

[7]  Maeve Duggan,et al.  Social Media Update 2016 , 2016 .

[8]  Dumitru Erhan,et al.  Training Deep Neural Networks on Noisy Labels with Bootstrapping , 2014, ICLR.

[9]  Daniel Fried,et al.  Analyzing the language of food on social media , 2014, 2014 IEEE International Conference on Big Data (Big Data).

[10]  Alexander C. Berg,et al.  Automatic Attribute Discovery and Characterization from Noisy Web Data , 2010, ECCV.

[11]  Samy Bengio,et al.  Large-Scale Object Classification Using Label Relation Graphs , 2014, ECCV.

[12]  Joan Bruna,et al.  Training Convolutional Networks with Noisy Labels , 2014, ICLR 2014.

[13]  Chih-Jen Lin,et al.  LIBLINEAR: A Library for Large Linear Classification , 2008, J. Mach. Learn. Res..

[14]  Corby K. Martin,et al.  Quantification of food intake using food image analysis , 2009, 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[15]  L. Zepeda,et al.  Think before you eat: photographic food diaries as intervention tools to change dietary decision making and attitudes , 2008 .

[16]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[17]  Keiji Yanai,et al.  Food image recognition with deep convolutional features , 2014, UbiComp Adjunct.

[18]  Corby K. Martin,et al.  Measuring food intake with digital photography. , 2014, Journal of human nutrition and dietetics : the official journal of the British Dietetic Association.

[19]  Shaogang Gong,et al.  Re-id: Hunting Attributes in the Wild , 2014, BMVC.

[20]  S A Jebb,et al.  Measurements of total energy expenditure provide insights into the validity of dietary measurements of energy intake. , 1993, Journal of the American Dietetic Association.

[21]  Sofiane Abbar,et al.  You Tweet What You Eat: Studying Food Consumption Through Twitter , 2014, CHI.

[22]  Hamed Haddadi,et al.  Fruits and Vegetables Calorie Counter Using Convolutional Neural Networks , 2016, Digital Health.