Flashpen: A High-Fidelity and High-Precision Multi-Surface Pen for Virtual Reality

Digital pen interaction has become a first-class input modality for precision tasks such as writing, annotating, drawing, and 2D manipulation. The key enablers of digital inking are the capacitive or resistive sensors that are integrated in contemporary tablet devices. In Virtual Reality (VR), however, users typically provide input across large regions, hence limiting the suitability of using additional tablet devices for accurate pen input. In this paper, we present Flashpen, a digital pen for VR whose sensing principle affords accurately digitizing hand writing and intricate drawing, including small and quick turns. Flashpen re-purposes an inexpensive gaming mouse sensor that digitizes extremely fine grained motions in the micrometer range at over 8 kHz when moving on a surface. We combine Flashpen's high-fidelity relative input with the absolute tracking cues from a VR headset to enable pen interaction across a variety of VR applications. In our two-block evaluation, which consists of a tracing task and a writing task, we compare Flashpen to a professional drawing tablet (Wacom). With this, we demonstrate that Flashpen's fidelity matches the performance of state-of-the-art digitizers and approaches the fidelity of analog pens, while adding the flexibility of supporting a wide range of flat surfaces.

[1]  William Buxton,et al.  Pen + touch = new tools , 2010, UIST.

[2]  Mayra Donaji Barrera Machuca,et al.  Multiplanes: Assisted Freehand VR Drawing , 2017, UIST.

[3]  Wolfgang Stuerzlinger,et al.  Is the Pen Mightier than the Controller? A Comparison of Input Devices for Selection in Virtual and Augmented Reality , 2019, VRST.

[4]  Eyal Ofek,et al.  CLAW: A Multifunctional Handheld Haptic Controller for Grasping, Touching, and Triggering in Virtual Reality , 2018, CHI.

[5]  Xing-Dong Yang,et al.  Magic finger: always-available input through finger instrumentation , 2012, UIST.

[6]  Ivan Poupyrev,et al.  3D User Interfaces: Theory and Practice , 2004 .

[7]  Rubaiat Habib Kazi,et al.  SymbiosisSketch: Combining 2D & 3D Sketching for Designing Detailed 3D Objects in Situ , 2018, CHI.

[8]  I. Scott MacKenzie,et al.  Phrase sets for evaluating text entry techniques , 2003, CHI Extended Abstracts.

[9]  Rubaiat Habib Kazi,et al.  MagicalHands: Mid-Air Hand Gestures for Animating in VR , 2019, UIST.

[10]  Rubaiat Habib Kazi,et al.  Experimental Evaluation of Sketching on Surfaces in VR , 2017, CHI.

[11]  Johann Habakuk Israel,et al.  Investigating the Learnability of Immersive Free-Hand Sketching , 2010, SBIM.

[12]  Fabrice Matulic,et al.  Eliciting Pen-Holding Postures for General Input with Suitability for EMG Armband Detection , 2019, ISS.

[13]  C. Lawrence Zitnick,et al.  Handwriting beautification using token means , 2013, ACM Trans. Graph..

[14]  Doug A. Bowman,et al.  Design and evaluation of menu systems for immersive virtual environments , 2001, Proceedings IEEE Virtual Reality 2001.

[15]  Nadir Weibel,et al.  PintAR: Sketching Spatial Experiences in Augmented Reality , 2019, Conference on Designing Interactive Systems.

[16]  David Kirsh,et al.  Thinking with external representations , 2010, AI & SOCIETY.

[17]  Jan O. Borchers,et al.  ARPen: Mid-Air Object Manipulation Techniques for a Bimanual AR System with Pen & Smartphone , 2019, CHI.

[18]  Enrico Rukzio,et al.  VRSketchIn: Exploring the Design Space of Pen and Tablet Interaction for 3D Sketching in Virtual Reality , 2020, CHI.

[19]  Zhen Li,et al.  HoloDoc: Enabling Mixed Reality Workspaces that Harness Physical and Digital Content , 2019, CHI.

[20]  Brandon Yee,et al.  Augmented Reality In-Situ 3D Sketching of Physical Objects , 2009 .

[21]  David H. Laidlaw,et al.  Drawing on Air: Input Techniques for Controlled 3D Line Illustration , 2007, IEEE Transactions on Visualization and Computer Graphics.

[22]  Tara Chittenden,et al.  Tilt Brush painting: Chronotopic adventures in a physical-virtual threshold , 2018, Journal of Contemporary Painting.

[23]  Audrey Girouard,et al.  FlexStylus: Leveraging Bend Input for Pen Interaction , 2017, UIST.

[24]  Roman Rädle,et al.  "In VR, everything is possible!": Sketching and Simulating Spatially-Aware Interactive Spaces in Virtual Reality , 2020, CHI.

[25]  Feng Tian,et al.  Get a Grip: Evaluating Grip Gestures for VR Input using a Lightweight Pen , 2020, CHI.

[26]  Anil Ufuk Batmaz,et al.  Precision vs. Power Grip: A Comparison of Pen Grip Styles for Selection in Virtual Reality , 2020, 2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW).

[27]  Bongshin Lee,et al.  ActiveInk: (Th)Inking with Data , 2019, CHI.

[28]  Tony DeRose,et al.  Toolglass and magic lenses: the see-through interface , 1993, SIGGRAPH.

[29]  Stephen DiVerdi,et al.  HelpingHand: example-based stroke stylization , 2012, ACM Trans. Graph..

[30]  Jean-Daniel Fekete,et al.  TicTacToon: a paperless system for professional 2D animation , 1995, SIGGRAPH.

[31]  Joseph J. LaViola,et al.  CavePainting: a fully immersive 3D artistic medium and interactive experience , 2001, I3D '01.

[32]  Fabrice Matulic,et al.  PenSight: Enhanced Interaction with a Pen-Top Camera , 2020, CHI.

[33]  Patrick Baudisch,et al.  Soap: a pointing device that works in mid-air , 2006, UIST.

[34]  Patrick Baudisch,et al.  Disappearing mobile devices , 2009, UIST '09.

[35]  Tyler A Jost,et al.  Quantitative analysis of the Oculus Rift S in controlled movement , 2019, Disability and rehabilitation. Assistive technology.

[36]  Brian Coltin,et al.  HTC Vive: Analysis and Accuracy Improvement , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[37]  Samuel B. Williams,et al.  ASSOCIATION FOR COMPUTING MACHINERY , 2000 .

[38]  Ming-Hsuan Yang,et al.  DodecaPen: Accurate 6DoF Tracking of a Passive Stylus , 2017, UIST.

[39]  Christophe Hurter,et al.  Dear Pictograph: Investigating the Role of Personalization and Immersion for Consuming and Enjoying Visualizations , 2020, CHI.

[40]  Hongbo Fu,et al.  Mobi3DSketch: 3D Sketching in Mobile AR , 2019, CHI.

[41]  Eyal Ofek,et al.  Haptic Revolver: Touch, Shear, Texture, and Shape Rendering on a Reconfigurable Virtual Reality Controller , 2018, CHI.

[42]  Max Mühlhäuser,et al.  VRSketchPen: Unconstrained Haptic Assistance for Sketching in Virtual 3D Environments , 2020, VRST.

[43]  Eyal Ofek,et al.  NormalTouch and TextureTouch: High-fidelity 3D Haptic Shape Rendering on Handheld Virtual Reality Controllers , 2016, UIST.

[44]  Walter F. Bischof,et al.  Hands, hover, and nibs: understanding stylus accuracy on tablets , 2015, Graphics Interface.

[45]  Ivan Poupyrev,et al.  Virtual Notepad: handwriting in immersive VR , 1998, Proceedings. IEEE 1998 Virtual Reality Annual International Symposium (Cat. No.98CB36180).

[46]  Per Ola Kristensson,et al.  Pen-based Interaction with Spreadsheets in Mobile Virtual Reality , 2020, 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[47]  Xiang Cao,et al.  Grips and gestures on a multi-touch pen , 2011, CHI.

[48]  Anoop Gupta,et al.  The pen is mightier: understanding stylus behaviour while inking on tablets , 2014, Graphics Interface.

[49]  Hans-Peter Seidel,et al.  FreeDrawer: a free-form sketching system on the responsive workbench , 2001, VRST '01.

[50]  Richard F. Lyon,et al.  The Optical Mouse, and an Architectural Methodology for Smart Digital Sensors , 1981 .