EmoG: Supporting the Sketching of Emotional Expressions for Storyboarding

Storyboarding is an important ideation technique that uses sequential art to depict important scenarios of user experience. Existing data-driven support for storyboarding focuses on constructing user stories, but fail to address its benefit as a graphic narrative device. Instead, we propose to develop a data-driven design support tool that increases the expressiveness of user stories by facilitating sketching storyboards. To explore this, we focus on supporting the sketching of emotional expressions of characters in storyboards. In this paper, we present EmoG, an interactive system that generates sketches of characters with emotional expressions based on input strokes from the user. We evaluated EmoG with 21 participants in a controlled user study. The results showed that our tool has significantly better performance in usefulness, ease of use, and quality of results than the baseline system.

[1]  Francesc Moreno-Noguer,et al.  GANimation: Anatomically-aware Facial Animation from a Single Image , 2018, ECCV.

[2]  Jan Meskens,et al.  Supporting Multidisciplinary Teams and Early Design Stages Using Storyboards , 2009, HCI.

[3]  Yang Li,et al.  Activity-based prototyping of ubicomp applications for long-lived, everyday human activities , 2008, CHI.

[4]  Tien-Tsin Wong,et al.  Two-stage sketch colorization , 2018, ACM Trans. Graph..

[5]  Douglas Eck,et al.  A Neural Representation of Sketch Drawings , 2017, ICLR.

[6]  Xin Yan,et al.  AI-Sketcher : A Deep Generative Model for Producing High-Quality Sketches , 2019, AAAI.

[7]  Lingling Fan,et al.  StoryDroid: Automated Generation of Storyboard for Android Apps , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE).

[8]  Kiyoharu Aizawa,et al.  DrawFromDrawings: 2D Drawing Assistance via Stroke Interpolation with a Sketch Database , 2017, IEEE Transactions on Visualization and Computer Graphics.

[9]  Yong Man Ro,et al.  Differential Generative Adversarial Networks: Synthesizing Non-linear Facial Variations with Limited Number of Training Data , 2017, ArXiv.

[10]  Mary Beth Rosson,et al.  Scenario-based design , 2002 .

[11]  Yang Wang,et al.  IdeaWall: Improving Creative Collaboration through Combinatorial Visual Stimuli , 2017, CSCW.

[12]  Peter Dalsgaard,et al.  Mapping the Landscape of Creativity Support Tools in HCI , 2019, CHI.

[13]  Ioannis Pitas,et al.  Facial expression analysis and synthesis : A survey , 2003 .

[14]  F. Spillers.  Emotion as a Cognitive Artifact and the Design Implications for Products That are Perceived As Pleasurable , 2004 .

[15]  Gregory D. Abowd,et al.  Storyboarding: an empirical determination of best practices and effective guidelines , 2006, DIS '06.

[16]  Ravin Balakrishnan,et al.  Storeoboard: Sketching Stereoscopic Storyboards , 2016, CHI.

[17]  Ziwei Liu,et al.  Semantic Facial Expression Editing using Autoencoded Flow , 2016, ArXiv.

[18]  Marc Alexa,et al.  How do humans sketch objects? , 2012, ACM Trans. Graph..

[19]  A. D. Manning,et al.  Understanding Comics: The Invisible Art , 1993 .

[20]  Hans Pancherz,et al.  Divine proportions in attractive and nonattractive faces. , 2010, World journal of orthodontics.

[21]  Sungwoo Lee,et al.  I Lead, You Help but Only with Enough Details: Understanding User Experience of Co-Creation with Artificial Intelligence , 2018, CHI.

[22]  Rama Chellappa,et al.  ExprGAN: Facial Expression Editing with Controllable Expression Intensity , 2017, AAAI.

[23]  Hiroshi Ishikawa,et al.  Real-time data-driven interactive rough sketch inking , 2018, ACM Trans. Graph..

[24]  Andrés Lucero,et al.  May AI?: Design Ideation with Cooperative Contextual Bandits , 2019, CHI.

[25]  Bing Zhou,et al.  Facial expression cloning with elastic and muscle models , 2014, J. Vis. Commun. Image Represent..

[26]  Amit R.Sharma,et al.  Face Photo-Sketch Synthesis and Recognition , 2012 .

[27]  Tom White,et al.  Sampling Generative Networks: Notes on a Few Effective Techniques , 2016, ArXiv.

[28]  Anthony Jameson,et al.  Understanding and Dealing With Usability Side Effects of Intelligent Processing , 2009, AI Mag..

[29]  Ranjitha Kumar,et al.  An Experimentation Engine for Data-Driven Fashion Systems , 2017, AAAI Spring Symposia.

[30]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[31]  Jeffrey Nichols,et al.  Swire: Sketch-based User Interface Retrieval , 2019, CHI.

[32]  Jodi Forlizzi,et al.  Guidelines for Depicting Emotions in Storyboard Scenarios , 2012 .

[33]  A. Lewis.  Making Comics: Storytelling Secrets of Comics, Manga and Graphic Novels , 2007 .

[34]  Yunhong Wang,et al.  Facial Expression Synthesis by U-Net Conditional Generative Adversarial Networks , 2018, ICMR.

[35]  Yang Li,et al.  External representations in ubiquitous computing design and the implications for design tools , 2006, DIS '06.

[36]  Qiang Zhang,et al.  Mining Pain Points from Hotel Online Comments based on Sentiment Analysis , 2019, 2019 IEEE 8th Joint International Information Technology and Artificial Intelligence Conference (ITAIC).

[37]  Corrie van der Lelie,et al.  The value of storyboards in the product design process , 2006, Personal and Ubiquitous Computing.