DeepTaste: Augmented Reality Gustatory Manipulation with GAN-Based Real-Time Food-to-Food Translation

We have been studying augmented reality (AR)-based gustatory manipulation interfaces and previously proposed a gustatory manipulation interface using generative adversarial network (GAN)-based real time image-to-image translation. Unlike three-dimensional (3D) food model-based systems that only change the color or texture pattern of a particular type of food in an inflexible manner, our GAN-based system changes the appearance of food into multiple types of food in real time flexibly, dynamically, and interactively. In the present paper, we first describe in detail a user study on a vision-induced gustatory manipulation system using a 3D food model and report its successful experimental results. We then summarize identified problems of the 3D model-based system and describe implementation details of the GAN-based system. We finally report in detail the main user study in which we investigated the impact of the GAN-based system on gustatory sensations and food recognition when somen noodles were turned into ramen noodles or fried noodles, and steamed rice into curry and rice or fried rice. The experimental results revealed that our system successfully manipulates gustatory sensations to some extent and that the effectiveness seems to depend on the original and target types of food as well as the experience of each individual with the food.

[1]  Hyunsoo Kim,et al.  Learning to Discover Cross-Domain Relations with Generative Adversarial Networks , 2017, ICML.

[2]  S. Sicherer,et al.  Quality of life in food allergy , 2011, Current opinion in allergy and clinical immunology.

[3]  Alexei A. Efros,et al.  Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[4]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[6]  Takuji Narumi,et al.  Augmented perception of satiety: controlling food consumption by changing apparent size of food with augmented reality , 2012, CHI.

[7]  Hiroo Iwata,et al.  Food simulator: a haptic interface for biting , 2004, IEEE Virtual Reality 2004.

[8]  Takuji Narumi,et al.  Evaluating cross-sensory perception of superimposing virtual color onto real drink: toward realization of pseudo-gustatory displays , 2010, AH.

[9]  Brian Wansink,et al.  Why Visual Cues of Portion Size May Influence Intake , 2004, Obesity research.

[10]  Jaakko Lehtinen,et al.  Progressive Growing of GANs for Improved Quality, Stability, and Variation , 2017, ICLR.

[11]  Kiyoshi Kiyokawa,et al.  AR Timewarping: A Temporal Synchronization Framework for Real-Time Sensor Fusion in Head-Mounted Displays , 2018, AH.

[12]  Jan Kautz,et al.  High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[13]  Hidekazu Tanaka,et al.  Chewing jockey: augmented food texture by using sound based on the cross-modal effect , 2011, Advances in Computer Entertainment Technology.

[14]  Carlos Velasco,et al.  Assessing crossmodal correspondences in exotic fruit juices: The case of shape and sound symbolism , 2013 .

[15]  Léon Bottou,et al.  Wasserstein Generative Adversarial Networks , 2017, ICML.

[16]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[17]  Takuji Narumi,et al.  Enchanting Your Noodles: GAN-based Real-time Food-to-Food Translation and Its Impact on Vision-induced Gustatory Manipulation , 2019, 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[18]  Bernt Schiele,et al.  Generative Adversarial Text to Image Synthesis , 2016, ICML.

[19]  C. Spence,et al.  Does Food Color Influence Taste and Flavor Perception in Humans? , 2010 .

[20]  P. Todd,et al.  Dining in the dark. The importance of visual cues for food consumption and satiety , 2010, Appetite.

[21]  C. Spence,et al.  Multisensory flavor perception: Assessing the influence of fruit acids and color cues on the perception of fruit-flavored beverages , 2008 .

[22]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.

[23]  R. Boakes,et al.  Confusing tastes and smells: how odours can influence the perception of sweet and sour tastes. , 1999, Chemical senses.

[24]  Ellen Yi-Luen Do,et al.  Vocktail: A Virtual Cocktail for Pairing Digital Taste, Smell, and Color Sensations , 2017, ACM Multimedia.

[25]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Takuji Narumi,et al.  Affecting Our Perception of Satiety by Changing the Size of Virtual Dishes Displayed with a Tabletop Display , 2013, HCI.

[27]  Katsunori Okajima,et al.  Effects of Visual Food Texture on Taste Perception , 2011 .

[28]  M A Viergever,et al.  Representation of sweet and salty taste intensity in the brain. , 2010, Chemical senses.

[29]  Dimitris N. Metaxas,et al.  StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[30]  Jan Kautz,et al.  Unsupervised Image-to-Image Translation Networks , 2017, NIPS.

[31]  C. Spence,et al.  Is it the plate or is it the food? Assessing the influence of the color (black or white) and shape of the plate on the perception of the food placed on it , 2012 .

[32]  D. Dubourdieu,et al.  The Color of Odors , 2001, Brain and Language.

[33]  Randy F. Pausch,et al.  A Literature Survey for Virtual Environments: Military Flight Simulator Visual Systems and Simulator Sickness , 1992, Presence: Teleoperators & Virtual Environments.

[34]  C. Spence,et al.  Does the type of receptacle influence the crossmodal association between colour and flavour? A cross-cultural comparison , 2014, Flavour.

[35]  Katsunori Okajima,et al.  Projective-AR system for customizing the appearance and taste of food , 2016, MVAR '16.

[36]  P. Rozin,et al.  “Taste-smell confusions” and the duality of the olfactory sense , 1982, Perception & psychophysics.

[37]  Keiji Yanai,et al.  Recognition of Multiple-Food Images by Detecting Candidate Regions , 2012, 2012 IEEE International Conference on Multimedia and Expo.

[38]  Takuji Narumi,et al.  Augmented reality flavors: gustatory display based on edible marker and cross-modal interaction , 2011, CHI.

[39]  Andrea Vedaldi,et al.  Instance Normalization: The Missing Ingredient for Fast Stylization , 2016, ArXiv.

[40]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[41]  O. Genschow,et al.  The color red reduces snack food and soft drink intake , 2012, Appetite.

[42]  A. Burks,et al.  Food allergy QoL questionnaire for children aged 0–12 years: content, construct, and cross‐cultural validity , 2008, Clinical and experimental allergy : journal of the British Society for Allergy and Clinical Immunology.

[43]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Jung-Woo Ha,et al.  StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[45]  Max A. Viergever,et al.  The first taste is always with the eyes: A meta-analysis on the neural correlates of processing visual food cues , 2011, NeuroImage.

[46]  M. Ernst,et al.  Humans integrate visual and haptic information in a statistically optimal fashion , 2002, Nature.