Computing surfaces : a platform for scalable interactive displays

Recent progress in electronic, display and sensing technologies makes possible a future with omnipresent, arbitrarily large interactive display surfaces. Nonetheless, current methods of designing display systems with multi-touch sensitivity do not scale. This thesis presents computing surfaces as a viable platform for resolving forthcoming scalability limitations. Computing surfaces are composed of a homogeneous network of physically adjoined, small sensitive displays with local computation and communication capabilities. In this platform, inherent scalability is provided by a distributed architecture. The regular spatial distribution of resources presents new demands on the way surface input and output information is managed and processed. Direct user input with touch based gestures needs to account for the distributed architecture of computing surfaces. A scalable middleware solution that conceals the tiled architecture is proposed for reasoning with touch-based gestures. The validity of this middleware is proven in a case study, where a fully distributed algorithm for online recognition of unistrokes – a particular class of touch-based gestures – is presented and evaluated. Novel interaction techniques based around interactive display surfaces involve direct manipulation with displayed digital objects. In order to facilitate such interactions in computing surfaces, an efficient distributed algorithm to perform 2D image transformations is introduced and evaluated. The performance of these transformations is heavily influenced by the arbitration policies of the interconnection network. One approach for improving the performance of these transformations in conventional network architectures is proposed and evaluated. More advanced applications in computing surfaces require the presence of some notion of time. An efficient algorithm for internal time synchronisation is presented and evaluated. A hardware solution is adopted to minimise the delay uncertainty of special timestamp messages. The proposed algorithm allows efficient, scalable time synchronisation among clusters of tiles. A hardware reference platform is constructed to demonstrate the basic principles and features of computing surfaces. This platform and a complementary simulation environment is used for extensive evaluation and analysis.

[1]  Debra J. Mascaro,et al.  Organic thin-film transistors: A review of recent advances , 2001, IBM J. Res. Dev..

[2]  J. R. Wullert,et al.  Electronic Information Display Technologies , 1997 .

[3]  Stephen R. Forrest,et al.  Micropatterning of small molecular weight organic semiconductor thin films using organic vapor phase deposition , 2003 .

[4]  Nick McKeown,et al.  The iSLIP scheduling algorithm for input-queued switches , 1999, TNET.

[5]  Greg Humphreys,et al.  Chromium: a stream-processing framework for interactive rendering on clusters , 2002, SIGGRAPH.

[6]  Henri E. Bal,et al.  VIRPI: A High-Level Toolkit for Interactive Scientific Visualization in Virtual Reality , 2001, EGVE/IPT.

[7]  M. Orhon.  The X Window System , 2005 .

[8]  Richard Han,et al.  TSync: a lightweight bidirectional time synchronization service for wireless sensor networks , 2004, MOCO.

[9]  William J. Dally,et al.  A delay model and speculative architecture for pipelined routers , 2001, Proceedings HPCA Seventh International Symposium on High-Performance Computer Architecture.

[10]  Ulrich Schmid,et al.  PSynUTC - Evaluation of a High Precision Time Synchronization Prototype System for Ethernet LANs , 2002 .

[11]  J. Jacobson,et al.  An electrophoretic ink for all-printed reflective electronic displays , 1998, Nature.

[12]  Eric Harslem,et al.  Designing the STAR User Interface , 1987, ECICS.

[13]  Donald P. Seraphim,et al.  30.3: Seamless Tiling Technology for Large Direct‐View Color AMLCD's , 2000 .

[14]  J. P. Krusius,et al.  8.1: Seamless Tiling of AMLCDs for Large Area Displays , 2002 .

[15]  Kimberly J. Allen.  Reel to Real: Prospects for Flexible Displays , 2005, Proceedings of the IEEE.

[16]  Hiroshi Ishii,et al.  The metaDESK: models and prototypes for tangible user interfaces , 1997, UIST '97.

[17]  Kay Römer.  Time synchronization in ad hoc networks , 2001, MobiHoc '01.

[18]  S. Lennart Johnsson,et al.  ROMM routing on mesh and torus networks , 1995, SPAA '95.

[19]  Tobias Isenberg,et al.  A Buffer Framework for Supporting Responsive Interaction in Information Visualization Interfaces , 2006, Fourth International Conference on Creating, Connecting and Collaborating through Computing (C5'06).

[20]  G. Blelloch.  Introduction to Data Compression * , 2022 .

[21]  Michael Kiy,et al.  Optical proximity and touch sensors based on monolithically integrated polymer photodiodes and polymer LEDs , 2006 .

[22]  Mihail L. Sichitiu,et al.  Simple, accurate time synchronization for wireless sensor networks , 2003, 2003 IEEE Wireless Communications and Networking, 2003. WCNC 2003..

[23]  J. Wade Davis,et al.  Statistical Pattern Recognition , 2003, Technometrics.

[24]  Jun‐Hyeok Yu,et al.  P‐7: 4 Inch a‐Si TFT‐LCD with an Embedded Color Image Scanner , 2007 .

[25]  Charles R. Johnson,et al.  Matrix analysis , 1985, Statistical Inference for Engineers and Data Scientists.

[26]  David L. Mills,et al.  Adaptive hybrid clock discipline algorithm for the network time protocol , 1998, TNET.

[27]  Martin Horauer,et al.  NTI: A Network Time Interface M-Module for High-Accuracy Clock-Synchronization , 1998, IPPS/SPDP Workshops.

[28]  Henning Sirringhaus,et al.  Device Physics of Solution‐Processed Organic Field‐Effect Transistors , 2005 .

[29]  Jun Rekimoto,et al.  SmartSkin: an infrastructure for freehand manipulation on interactive surfaces , 2002, CHI.

[30]  David Goldberg,et al.  Touch-typing with a stylus , 1993, INTERCHI.

[31]  Deborah Estrin,et al.  Proceedings of the 5th Symposium on Operating Systems Design and Implementation Fine-grained Network Time Synchronization Using Reference Broadcasts , 2022 .

[32]  T. Jackson,et al.  Stacked pentacene layer organic thin-film transistors with improved characteristics , 1997, IEEE Electron Device Letters.

[33]  Keith Marzullo,et al.  Maintaining the time in a distributed system , 1985, OPSR.

[34]  Adam Finkelstein,et al.  Automatic alignment of high-resolution multi-projector displays using an uncalibrated camera , 2000, Proceedings Visualization 2000. VIS 2000 (Cat. No.00CH37145).

[35]  Samuel Williams,et al.  The Landscape of Parallel Computing Research: A View from Berkeley , 2006 .

[36]  Saurabh Ganeriwal,et al.  Timing-sync protocol for sensor networks , 2003, SenSys '03.

[37]  Ching Y. Suen,et al.  The State of the Art in Online Handwriting Recognition , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[38]  Robert A. Street,et al.  All jet-printed polymer thin-film transistor active-matrix backplanes , 2004 .

[39]  Takao Someya,et al.  A large-area, flexible pressure sensor matrix with organic field-effect transistors for artificial skin applications. , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[40]  Garrison W. Cottrell,et al.  Time-delay neural networks: representation and induction of finite-state machines , 1997, IEEE Trans. Neural Networks.

[41]  Joseph Y. Halpern,et al.  Optimal precision in the presence of uncertainty , 1985, J. Complex..

[42]  H. Sirringhaus,et al.  Integrated optoelectronic devices based on conjugated polymers , 1998, Science.

[43]  V. R. Raju,et al.  Paper-like electronic displays: Large-area rubber-stamped plastic sheets of electronics and microencapsulated electrophoretic inks , 2001, Proceedings of the National Academy of Sciences of the United States of America.

[44]  Robert L. Grossman,et al.  TeraScope: distributed visual data mining of terascale data sets over photonic networks , 2003, Future Gener. Comput. Syst..

[45]  Ivan E. Sutherland,et al.  Reentrant polygon clipping , 1974, Commun. ACM.

[46]  P. Magnante,et al.  Electroluminescence in Organic Crystals , 1963 .

[47]  H. Sirringhaus,et al.  High-Resolution Ink-Jet Printing of All-Polymer Transistor Circuits , 2000, Science.

[48]  Bill Buxton,et al.  Multi-Touch Systems that I Have Known and Loved , 2009 .

[49]  Ian F. Akyildiz,et al.  Time-diffusion synchronization protocol for wireless sensor networks , 2005, IEEE/ACM Transactions on Networking.

[50]  Kai Li,et al.  Software environments for cluster-based display systems , 2001, Proceedings First IEEE/ACM International Symposium on Cluster Computing and the Grid.

[51]  Xi Wang,et al.  LambdaBridge: A Scalable Architecture for Future Generation Terabit Applications , 2006, 2006 3rd International Conference on Broadband Communications, Networks and Systems.

[52]  Luc Renambot,et al.  Scalable Graphics Architecture for High-Resolution Displays , 2005 .

[53]  Jefferson Y. Han.  Low-cost multi-touch sensing through frustrated total internal reflection , 2005, UIST.

[54]  Mike Wu,et al.  Multi-finger and whole hand gestural interaction techniques for multi-user tabletop displays , 2003, UIST '03.

[55]  Jun Liu,et al.  Fabrication of fully transparent nanowire transistors for transparent and flexible electronics. , 2007, Nature nanotechnology.

[56]  E·史密斯,et al.  Active matrix displays , 2009 .

[57]  Andrew D. Wilson.  TouchLight: an imaging touch screen and display for gesture-based interaction , 2004, ICMI '04.

[58]  Peter Robinson,et al.  T3: Rapid Prototyping of High-Resolution and Mixed-Presence Tabletop Applications , 2007, Second Annual IEEE International Workshop on Horizontal Interactive Human-Computer Systems (TABLETOP'07).

[59]  Anoop Gupta,et al.  Experiences with Scalability of Display Walls , 2002 .

[60]  Craig B. Stunkel,et al.  Clock Synchronization on a Multicomputer , 1997, J. Parallel Distributed Comput..

[61]  Akira J. Ikushima,et al.  Observation of degradation processes of Al electrodes in organic electroluminescence devices by electroluminescence microscopy, atomic force microscopy, scanning electron microscopy, and Auger electron spectroscopy , 1994 .

[62]  Andy Hopper,et al.  Global Teleporting with Java: Toward Ubiquitous Personalized Computing , 1997, Computer.

[63]  Scott E. Hudson.  Using light emitting diode arrays as touch-sensitive input and output devices , 2004, UIST '04.

[64]  Hiroshi Ishii,et al.  Bricks: laying the foundations for graspable user interfaces , 1995, CHI '95.

[65]  Kenneth A. Perrine,et al.  Parallel Graphics and Interactivity with the Scaleable Graphics Engine , 2001, ACM/IEEE SC 2001 Conference (SC'01).

[66]  Clifton Forlines,et al.  Sharing and building digital group histories , 2002, CSCW '02.

[67]  Ronald L. Graham,et al.  Concrete Mathematics, a Foundation for Computer Science , 1991, The Mathematical Gazette.

[68]  J. M. Shaw,et al.  Organic electronics: Introduction , 2001, IBM J. Res. Dev..

[69]  Gyula Simon,et al.  The flooding time synchronization protocol , 2004, SenSys '04.

[70]  Jan M. Rabaey,et al.  Lightweight time synchronization for sensor networks , 2003, WSNA '03.

[71]  Hiroyuki Ohshima,et al.  46.2: Invited Paper: High-Performance LTPS Technologies for Advanced Mobile Display Applications , 2007 .

[72]  David E. Culler,et al.  Elapsed time on arrival: a simple and versatile primitive for canonical time synchronisation services , 2006, Int. J. Ad Hoc Ubiquitous Comput..

[73]  M. Sheelagh T. Carpendale,et al.  Territoriality in collaborative tabletop workspaces , 2004, CSCW.

[74]  Richard H. Friend,et al.  Conjugated polymers. New materials for optoelectronic devices , 2001 .

[75]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[76]  David B. Mitzi,et al.  Synthesis, Structure, and Properties of Organic‐Inorganic Perovskites and Related Materials , 2007 .

[77]  Venkatram Vishwanath,et al.  JuxtaView - a tool for interactive visualization of large imagery on scalable tiled displays , 2004, 2004 IEEE International Conference on Cluster Computing (IEEE Cat. No.04EX935).

[78]  David B. Mitzi,et al.  Transport, Optical, and Magnetic Properties of the Conducting Halide Perovskite CH3NH3SnI3 , 1995 .

[79]  Cherie R. Kagan,et al.  Organic-inorganic hybrid materials as semiconducting channels in thin-film field-effect transistors , 1999, Science.

[80]  Adam Finkelstein,et al.  Data distribution strategies for high-resolution displays , 2001, Comput. Graph..

[81]  Kay Römer,et al.  Time Synchronization and Calibration in Wireless Sensor Networks , 2005, Handbook of Sensor Networks.

[82]  Hiroshi Ishii,et al.  Tangible bits: towards seamless interfaces between people, bits and atoms , 1997, CHI.

[83]  Leslie Lamport,et al.  Time, clocks, and the ordering of events in a distributed system , 1978, CACM.

[84]  Paul Horowitz,et al.  The Art of Electronics , 1980 .

[85]  Gordon Stoll,et al.  WireGL: a scalable graphics system for clusters , 2001, SIGGRAPH.

[86]  Sudhir P. Mudur,et al.  Mathematical Elements for Computer Graphics , 1985, Advances in Computer Graphics.

[87]  Myung-Koo Kang,et al.  35-1: Invited Paper: Advanced Technologies Based on a-Si or LTPS (Low Temperature Poly Si) TFT (Thin Film Transistor) for High Performance Mobile Display , 2007 .

[88]  P. Pleshko,et al.  Scanning limitations of liquid-crystal displays , 1974 .

[89]  Karl R. Amundson.  Electrophoretic Imaging Films for Electronic Paper Displays , 2005 .

[90]  Rahul Sukthankar,et al.  Scalable alignment of large-format multi-projector displays using camera homography trees , 2002, IEEE Visualization, 2002. VIS 2002..

[91]  Sargur N. Srihari,et al.  On-Line and Off-Line Handwriting Recognition: A Comprehensive Survey , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[92]  Wayne H. Wolf,et al.  Design and implementation of ubiquitous smart cameras , 2006, IEEE International Conference on Sensor Networks, Ubiquitous, and Trustworthy Computing (SUTC'06).

[93]  Robert A. Street,et al.  54.1: Flexible Electrophoretic Displays with Jet‐Printed Active‐Matrix Backplanes , 2005 .

[94]  Vittorio Ghergia.  New materials for optoelectronic devices , 1993 .

[95]  Qun Li,et al.  Global clock synchronization in sensor networks , 2006, IEEE Transactions on Computers.

[96]  Nancy A. Lynch,et al.  Gradient clock synchronization , 2004, PODC '04.

[97]  Shahzad Malik,et al.  Visual touchpad: a two-handed gestural input device , 2004, ICMI '04.

[98]  D. Davcev,et al.  Single exponential smoothing method and neural network in one method for time series prediction , 2004, IEEE Conference on Cybernetics and Intelligent Systems, 2004..

[99]  Stacey D. Scott,et al.  Investigating Tabletop Territoriality in Digital Tabletop Workspaces , 2006 .

[100]  Stephen R. Forrest,et al.  The path to ubiquitous and low-cost organic electronic appliances on plastic , 2004, Nature.

[101]  David F. Rogers,et al.  Mathematical elements for computer graphics , 1976 .

[102]  Darren Leigh,et al.  DiamondTouch: a multi-user touch technology , 2001, UIST '01.

[103]  Min Huang,et al.  13.3: A Novel Approach to Tiled Displays , 2003 .

[104]  Seif Haridi,et al.  Distributed Algorithms , 1992, Lecture Notes in Computer Science.

[105]  Niraj K. Jha,et al.  A 4.6Tbits/s 3.6GHz single-cycle NoC router with a novel switch allocator in 65nm CMOS , 2007, ICCD.

[106]  William J. Dally,et al.  Principles and Practices of Interconnection Networks , 2004 .

[107]  Developing the First Commercial Product that Uses Multi-Touch Technology , 2008 .

[108]  Mary Czerwinski,et al.  Large display research overview , 2006, Color Imaging Conference.

[109]  James H. Aylor,et al.  Computer for the 21st Century , 1999, Computer.

[110]  William Buxton,et al.  ThinSight: versatile multi-touch sensing for thin form-factor displays , 2007, UIST.

[111]  Nancy A. Lynch,et al.  An Upper and Lower Bound for Clock Synchronization , 1984, Inf. Control..

[112]  Kathy Kincade.  Flexible displays open new windows of opportunity , 2004 .

[113]  Andrew D. Wilson.  PlayAnywhere: a compact interactive tabletop projection-vision system , 2005, UIST.

[114]  Eric W. Forsythe,et al.  Flexible-display development for army applications , 2007 .

[115]  Sergi Jordà,et al.  The reacTable: exploring the synergy between live music performance and tabletop tangible interfaces , 2007, TEI.

[116]  Yuval Tamir,et al.  Symmetric Crossbar Arbiters for VLSI Communication Switches , 1993, IEEE Trans. Parallel Distributed Syst..

[117]  Kenneth C. Smith,et al.  A multi-touch three dimensional touch-sensitive tablet , 1985, CHI '85.

[118]  량롱-창,et al.  Composition and process for the sealing of microcups in roll-to-roll display manufacturing , 2002 .

[119]  William F. Feehery,et al.  69.1: Invited Paper: Solution Processing of Small‐Molecule OLEDs , 2007 .

[120]  Norbert A. Streitz,et al.  Connectables: dynamic coupling of displays for the flexible creation of shared workspaces , 2001, UIST '01.

[121]  Theo Mayer.  New options and considerations for creating enhanced viewing experiences , 1997, COMG.

[122]  Pierre David Wellner,et al.  Interacting with paper on the DigitalDesk , 1993, CACM.

[123]  Stephen R. Forrest,et al.  Direct, Mask‐ and Solvent‐Free Printing of Molecular Organic Semiconductors , 2004 .

[124]  Stephen R. Forrest,et al.  Ultrathin Organic Films Grown by Organic Molecular Beam Deposition and Related Techniques. , 1997, Chemical reviews.

[125]  R. N. Marks,et al.  Light-emitting diodes based on conjugated polymers , 1990, Nature.

[126]  W. Graham Richards,et al.  Art of electronics , 1983, Nature.

[127]  Kai Li,et al.  Color gamut matching for tiled display walls , 2003 .

[128]  J. Vig.  Introduction to Quartz Frequency Standards , 1992 .

[129]  Ken Hinckley,et al.  Synchronous gestures for multiple persons and computers , 2003, UIST '03.

[130]  Yukio Sato,et al.  54.5L: Late‐News Paper: A 24‐inch AM‐OLED Display with XGA Resolution by Novel Seamless Tiling Technology , 2003 .

[131]  A. Kumary,et al.  A 4.6Tbits/s 3.6GHz single-cycle NoC router with a novel switch allocator in 65nm CMOS , 2007 .

[132]  Tat-Jen Cham,et al.  Calibrating Scalable Multi-Projector Displays Using Camera Homography Trees , 2001 .

[133]  Bob Francis,et al.  Silicon Graphics Inc. , 1993 .

[134]  Katsuyuki Morii,et al.  Inkjet Printing of Light-Emitting Polymer Displays , 2003 .

[135]  Jun Rekimoto,et al.  HoloWall: designing a finger, hand, body, and object sensitive wall , 1997, UIST '97.

[136]  Carolina Cruz-Neira,et al.  Surround-Screen Projection-Based Virtual Reality: The Design and Implementation of the CAVE , 2023 .

[137]  A. L. Nel.  Hexagonal image processing , 1989, COMSIG 1989 Proceedings: Southern African Conference on Communications and Signal Processing.

[138]  C. Tang,et al.  Organic Electroluminescent Diodes , 1987 .

[139]  Nancy A. Lynch,et al.  Impossibility of distributed consensus with one faulty process , 1983, PODS '83.

[140]  Günther Greiner,et al.  Efficient clipping of arbitrary polygons , 1998, TOGS.

[141]  A. Sugimoto,et al.  Flexible OLED displays using plastic substrates , 2004, IEEE Journal of Selected Topics in Quantum Electronics.

[142]  Charles L. Seitz,et al.  Myrinet: A Gigabit-per-Second Local Area Network , 1995, IEEE Micro.

[143]  G. Gelinck,et al.  Flexible active-matrix displays and shift registers based on solution-processed organic transistors , 2004, Nature materials.

[144]  Jin Jang,et al.  P‐28: Flexibility Study of High‐Performance LTPS‐TFT on Flexible Metal Foil , 2007 .

[145]  Nancy A. Lynch,et al.  An Overview of Clock Synchronization , 1986, Fault-Tolerant Distributed Computing.

[146]  HongMei Zang,et al.  microcup ® electronic paper by roll-to-roll manufacturing processes , 2003 .

[147]  Norbert A. Streitz,et al.  i-LAND: an interactive landscape for creativity and innovation , 1999, CHI '99.

[148]  Akif Ali,et al.  Near-optimal worst-case throughput routing for two-dimensional mesh networks , 2005, 32nd International Symposium on Computer Architecture (ISCA'05).