From Writing to Painting: A Kinect-Based Cross-Modal Chinese Painting Generation System

As computer and interaction technologies mature, a much broader range of media is now used for input and output, each of which has its own rich repertoire of techniques, instruments, and cultural heritage. The combination of multiple media can produce novel multimedia human-computer interaction approaches which are more efficient and interesting than traditional single media methods. This paper presents CalliPaint, a system for cross-modal art generation that links together Chinese ink brush calligraphy writing and Chinese landscape painting. We investigate the mapping between the two modalities based on concepts of metaphoric congruence, and implement our findings into a prototype system. A multi-step evaluation experiment with real users suggests that CalliPaint provides a realistic and intuitive experience that allows even novice users to create attractive landscape paintings from writing. Comparison with a general-purpose digital painting software suggests that CalliPaint provides users with a more enjoyable experience. Finally, exhibiting CalliPaint in an open-access location for use by casual users without any training shows that the system is easy to learn.

[1]  D. Maurer,et al.  The shape of boubas: sound-shape correspondences in toddlers and adults. , 2006, Developmental science.

[2]  Hiroshi Ishii,et al.  I/O brush: drawing with everyday objects as ink , 2004, CHI.

[3]  Andrew D. Wilson.  Using a depth camera as a touch sensor , 2010, ITS '10.

[4]  Guang Li,et al.  ShadowStory: creative and collaborative digital storytelling inspired by cultural heritage , 2011, CHI.

[5]  Kwan-Liu Ma,et al.  A new form of traditional art: visual simulation of Chinese shadow play , 2003, SIGGRAPH '03.

[6]  Daniel J. Wigdor,et al.  Combining and measuring the benefits of bimanual pen and direct-touch interaction on horizontal interfaces , 2008, AVI '08.

[7]  Qunsheng Peng,et al.  Image-based synthesis of Chinese landscape painting , 2008, Journal of Computer Science and Technology.

[8]  J. Dolinsky,et al.  Synthesizing Handwritten Characters Using Naturalness Learning , 2007, 2007 IEEE International Conference on Computational Cybernetics.

[9]  Kok-Lim Low,et al.  SandCanvas: a multi-touch art medium inspired by sand animation , 2011, CHI.

[10]  Marcelo M. Wanderley,et al.  Gestural control of sound synthesis , 2004, Proceedings of the IEEE.

[11]  Stephen Chi-fai Chan,et al.  MelodicBrush: a novel system for cross-modal digital art creation linking calligraphy and music , 2012, DIS '12.

[12]  Xiaojun Bi,et al.  Acquiring and pointing: an empirical study of pen-tilt-based interaction , 2011, CHI.

[13]  Xiang Cao,et al.  Grips and gestures on a multi-touch pen , 2011, CHI.

[14]  Jiri Matas,et al.  Face-TLD: Tracking-Learning-Detection applied to faces , 2010, 2010 IEEE International Conference on Image Processing.

[15]  Xiaojun Bi,et al.  An exploration of pen rolling for pen-based interaction , 2008, UIST '08.

[16]  Steve Strassmann,et al.  Hairy brushes , 1986, SIGGRAPH.

[17]  Hao Jiang,et al.  An Intelligent System for Chinese Calligraphy , 2007, AAAI.

[18]  Susan Bush,et al.  Three Thousand Years of Chinese Painting , 1998 .

[19]  Nelson Siu-Hang Chu,et al.  Real-time painting with an expressive virtual Chinese brush , 2004, IEEE Computer Graphics and Applications.

[20]  Xiang Cao,et al.  Telling the whole story: anticipation, inspiration and reputation in a field deployment of TellTable , 2010, CSCW '10.

[21]  Ryohei Nakatsu,et al.  Cultural Computing with Context-Aware Application: ZENetic Computer , 2005, ICEC.

[22]  Boris Müller,et al.  Articulated paint: musical expression for non-musicians , 2007, NIME '07.

[23]  Congfu Xu,et al.  Virtual hairy brush for digital painting and calligraphy , 2005, Sci. China Ser. F Inf. Sci..

[24]  Fumihisa Shibata,et al.  MAI painting brush: an interactive device that realizes the feeling of real painting , 2010, UIST.

[25]  Xiaofeng Mi,et al.  Droplet: A virtual brush model to simulate Chinese calligraphy and painting , 2008, Journal of Computer Science and Technology.

[26]  Laewoo Kang,et al.  Hé: Calligraphy as a Musical Interface , 2010, NIME.

[27]  Hiroshi Ishii,et al.  Jabberstamp: embedding sound and voice in traditional drawings , 2007, SIGGRAPH '07.

[28]  I-Fan Shen,et al.  A new style of ancient culture: animated Chinese Dunhuang murals , 2004, SIGGRAPH '04.

[29]  V. Ramachandran,et al.  Synaesthesia? A window into perception, thought and language , 2001 .

[30]  Steven Schkolne.  Drawing with the Hand in Free Space: Creating 3D Shapes with Gesture in a Semi-Immersive Environment , 2002, Leonardo.

[31]  Johannes Taelman,et al.  FluidPaint: an interactive digital painting system using real wet brushes , 2009, ITS '09.

[32]  Ruigang Yang,et al.  TofCut: Towards Robust Real-time Foreground Extraction using Time-of-flight Camera , 2016 .

[33]  Geri Gay,et al.  Harmonic paper: interactive music interface for drawing , 2013, CHI Extended Abstracts.

[34]  William V. Baxter,et al.  DAB: Interactive Haptic Painting with 3D Virtual Brushes , 2001, SIGGRAPH Courses.

[35]  Patricia Buckley Ebrey,et al.  The Cambridge Illustrated History of China , 2022 .

[36]  Pedro Lopes,et al.  Combining bimanual manipulation and pen-based input for 3D modelling , 2011, SBIM '11.

[37]  Kazuhiro Jo,et al.  DrawSound: a drawing instrument for sound performance , 2008, TEI.

[38]  Horace Ho-Shing Ip,et al.  Virtual brush: a model-based synthesis of Chinese calligraphy , 2000, Comput. Graph..

[39]  Takeo Igarashi,et al.  Vignette: interactive texture design and manipulation with freeform gestures for pen-and-ink illustration , 2012, CHI.

[40]  Qinglian Guo,et al.  "Nijimi" rendering algorithm for creating quality black ink paintings , 2003, Proceedings Computer Graphics International 2003.