Embedding 3D models in offline physical environments

This article introduces a novel approach for embedding 3D models in offline physical environments using quick response (QR) codes. Unlike conventional methods, we consider settings where 3D models cannot be retrieved from a remote server. Our method involves generating octree models from voxelized 3D models and storing them in QR codes using a space‐efficient data structure. This allows storing 3D models that are both intelligible and purposeful on standard QR codes while addressing the major storage constraint that is present in offline situations. Furthermore, we explore 3D convolutional neural networks (CNN) and autoencoders (AE) to compress 3D models with high resolutions where using octrees alone does not suffice. To the best of our knowledge, our AE network is the first to employ octrees to further compress its encoded data. Through user‐friendly desktop and mobile applications, we allow users to encode, decode and visualize 3D models in augmented reality (AR) using QR codes, thus experiment with our methods. The proposed approach enables unique applications and future research in ubiquitous computing, 3D data compression and transmission, 3D AEs, AR and Virtual Reality, low‐cost autonomous robots, and 3D printing.

[1]  Yang Liu,et al.  Adaptive O-CNN: A Patch-based Deep Representation of 3D Shapes , 2018 .

[2]  Silvio Savarese,et al.  3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction , 2016, ECCV.

[3]  Leonidas J. Guibas,et al.  FPNN: Field Probing Neural Networks for 3D Data , 2016, NIPS.

[4]  Thomas Brox,et al.  Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[5]  Tai-Wei Kan,et al.  Applying QR code in augmented reality applications , 2009, VRCAI '09.

[6]  Sebastian Scherer,et al.  VoxNet: A 3D Convolutional Neural Network for real-time object recognition , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[7]  Tadayoshi Kohno,et al.  Analyzing the Use of Quick Response Codes in the Wild , 2015, MobiSys.

[8]  Peter Deutsch,et al.  DEFLATE Compressed Data Format Specification version 1.3 , 1996, RFC.

[9]  David A. Huffman,et al.  A method for the construction of minimum-redundancy codes , 1952, Proceedings of the IRE.

[10]  Jianxiong Xiao,et al.  3D ShapeNets: A deep representation for volumetric shapes , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Leonidas J. Guibas,et al.  ShapeNet: An Information-Rich 3D Model Repository , 2015, ArXiv.

[13]  Peter Deutsch,et al.  GZIP file format specification version 4.3 , 1996, RFC.

[14]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[15]  Michael Gervautz,et al.  A simple method for color quantization: octree quantization , 1990 .

[16]  Oliver Bimber,et al.  Unsynchronized 4D Barcodes , 2007, ISVC.

[17]  Wolfram Burgard,et al.  OctoMap: an efficient probabilistic 3D mapping framework based on octrees , 2013, Autonomous Robots.

[18]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[19]  Ming Zeng,et al.  Octree-based fusion for realtime 3D reconstruction , 2013, Graph. Model..

[20]  Jung Hong Chuang Level of Detail for 3D Graphics , 2002 .

[21]  Michael I. Jordan,et al.  Advances in Neural Information Processing Systems 30 , 1995 .

[22]  James A. Storer,et al.  Data compression via textual substitution , 1982, JACM.

[23]  Thomas Dean,et al.  Quick Layered Response ( QLR ) Codes , 2012 .

[24]  Bruno Patrão,et al.  Graphic Code: A New Machine Readable Approach , 2018, 2018 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR).

[25]  Tack-Don Han,et al.  ColorCodeAR: Large identifiable ColorCode-based augmented reality system , 2016, 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[26]  Leonidas J. Guibas,et al.  Volumetric and Multi-view CNNs for Object Classification on 3D Data , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  Il-Yeol Song,et al.  Octree-R: An Adaptive Octree for Efficient Ray Tracing , 1995, IEEE Trans. Vis. Comput. Graph..

[28]  Pierre Vandergheynst,et al.  Geometric Deep Learning: Going beyond Euclidean data , 2016, IEEE Signal Process. Mag..

[29]  Greg Turk,et al.  Simplification and Repair of Polygonal Models Using Volumetric Techniques , 2003, IEEE Trans. Vis. Comput. Graph..

[30]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[31]  Ming Yang,et al.  3D Convolutional Neural Networks for Human Action Recognition , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Michael Rohs,et al.  Real-World Interaction with Camera Phones , 2004, UCS.

[33]  Bo Li,et al.  Large-Scale 3D Shape Retrieval from ShapeNet Core55 , 2016, 3DOR@Eurographics.

[34]  D. Huffman A Method for the Construction of Minimum-Redundancy Codes , 1952 .

[35]  Yang Liu,et al.  O-CNN , 2017, ACM Trans. Graph..

[36]  Reinhard Klein,et al.  Eurographics Symposium on Point-based Graphics (2006) Octree-based Point-cloud Compression , 2022 .

[37]  Matthew D. Zeiler ADADELTA: An Adaptive Learning Rate Method , 2012, ArXiv.