Ein echtzeitfähiges System zur Gewinnung von Tiefeninformation aus Stereobildpaaren für konfigurierbare Hardware

Diese Arbeit befasst sich mit der Entwicklung eines echtzeitfahigen Systems zur Erstellung von Tiefeninformation aus Stereobildpaaren, das in einer Reihe von Anwendungen zur dreidimensionalen Vermessung des Raumes herangezogen werden kann. Als Hauptanwendungsgebiete sind in erster Linie mobile Robotikapplikationen vorgesehen, die sehr strenge Anforderungen sowohl bezuglich des Ressourcenverbrauchs als auch im Hinblick auf die Messeigenschaften und das Laufzeitverhalten stellen. Ein Merkmal des in dieser Arbeit entworfenen Systems ist die in Echtzeit stattfindende Ausfuhrung der verwendeten Algorithmen in Kombination mit sehr guten Messeigenschaften. Das verwendete Stereo-Matching-Verfahren basiert auf einem globalen Ansatz und liefert im Vergleich zu den alternativen echtzeitfahigen Methoden sehr gute Ergebnisse. Im Vordergrund steht dabei der Semi-Global-Matching-Algorithmus. Aufgrund der Komplexitat globaler Ansatze finden in Echtzeitapplikationen nur lokale Stereo-Verfahren Verwendung. Lokale Verfahren liefern jedoch im Vergleich zu den globalen Methoden qualitativ schlechte Disparitatskarten. Ein neuer globaler Matching-Algorithmus Efficient-Semi-Global-Matching (eSGM) wird vorgestellt und in das Konzept fur mobile Robotikanwendungen umgesetzt. Wegen der begrenzten Ressourcen der realen Hardware wurde eine Weiterentwicklung des eSGM-Algorithmus fur die Realisierung genutzt. Abschliesend wird das System anhand der drei Kerneigenschaften Laufzeit, Ressourcenverbrauch und Qualitat der Tiefeninformation gegenuber den Verfahren nach dem Stand der Technik bewertet. Der in dieser Arbeit vorgestellte FPGA-Ansatz, die eingesetzte Entwurfsmethode und die vorgestellten Algorithmen ermoglichten es, ein leistungsfahiges Stereo-Bildverarbeitungssystem zu entwickeln, das den hohen Anforderungen bezuglich des Laufzeitverhaltens und der Qualitat des Ergebnisses gerecht wird.

[1]  Takeo Kanade,et al.  A stereo machine for video-rate dense depth mapping and its new applications , 1996, Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[2]  Kostas Daniilidis,et al.  Predicting Disparity Windows for Real-Time Stereo , 2000, ECCV.

[3]  Qingxiong Yang,et al.  Near Real-time Stereo for Weakly-Textured Scenes , 2008, BMVC.

[4]  Nanning Zheng,et al.  Stereo Matching Using Belief Propagation , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Olga Veksler,et al.  Fast variable window for stereo correspondence using integral images , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[6]  Erik Lindholm,et al.  NVIDIA Tesla: A Unified Graphics and Computing Architecture , 2008, IEEE Micro.

[7]  Ines Ernst,et al.  Mutual Information Based Semi-Global Stereo Matching on the GPU , 2008, ISVC.

[8]  Charles T. Loop,et al.  Computing rectifying homographies for stereo vision , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[9]  Martin Humenberger,et al.  A census-based stereo vision algorithm using modified Semi-Global Matching and plane fitting to improve matching quality , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[10]  Jesús M. Pérez,et al.  Real-Time Stereo Matching Using Memory-Efficient Belief Propagation for High-Definition 3D Tele-Presence Systems , 2009, CIARP.

[11]  Donald Geman,et al.  Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images , 1984 .

[12]  Markus Vincze,et al.  A fast stereo matching algorithm suitable for embedded real-time systems , 2010, Comput. Vis. Image Underst..

[13]  J. Rose,et al.  The effect of LUT and cluster size on deep-submicron FPGA performance and density , 2000, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[14]  Adrian Schischmanow,et al.  Stereo camera calibration with diffractive optical elements , 2009 .

[15]  Heiko Hirschmüller,et al.  Evaluation of Stereo Matching Costs on Images with Radiometric Differences , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Dah-Jye Lee,et al.  A Comparison Study on Implementing Optical Flow and Digital Communications on FPGAs and GPUs , 2010, TRETS.

[17]  Tsutomu Maruyama,et al.  A Real-Time Stereo Vision System with FPGA , 2003, FPL.

[18]  John Woodfill,et al.  Real-time stereo vision on the PARTS reconfigurable computer , 1997, Proceedings. The 5th Annual IEEE Symposium on Field-Programmable Custom Computing Machines Cat. No.97TB100186).

[19]  Samuel Williams,et al.  The Landscape of Parallel Computing Research: A View from Berkeley , 2006 .

[20]  In-So Kweon,et al.  Adaptive Support-Weight Approach for Correspondence Search , 2006, IEEE Trans. Pattern Anal. Mach. Intell..

[21]  Federico Tombari,et al.  Stereo Vision Enabling Precise Border Localization Within a Scanline Optimization Framework , 2007, ACCV.

[22]  Federico Tombari,et al.  Segmentation-Based Adaptive Support for Accurate Stereo Correspondence , 2007, PSIVT.

[23]  Rudy Lauwereins,et al.  Real-time accurate stereo with bitwise fast voting on CUDA , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[24]  Olga Veksler,et al.  Stereo correspondence by dynamic programming on a tree , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[25]  David Krutz.  Ein Betriebssystem für konfigurierbare Hardware , 2006 .

[26]  Federico Tombari,et al.  Near real-time stereo based on effective cost aggregation , 2008, 2008 19th International Conference on Pattern Recognition.

[27]  David Z. Pan,et al.  ISPD placement contest updates and ISPD 2007 global routing contest , 2007, ISPD '07.

[28]  Stefan K. Gehrig,et al.  A Real-Time Low-Power Stereo Vision Engine Using Semi-Global Matching , 2009, ICVS.

[29]  Ramin Zabih,et al.  Non-parametric Local Transforms for Computing Visual Correspondence , 1994, ECCV.

[30]  Maximilian Buder.  Dense real-time stereo matching using memory efficient semi-global-matching variant based on FPGAs , 2012, Real-Time Image and Video Processing.

[31]  Fay Huang,et al.  Proceedings of the 3rd Pacific Rim Symposium on Advances in Image and Video Technology , 2009 .

[32]  Ruigang Yang,et al.  A Performance Study on Different Cost Aggregation Approaches Used in Real-Time Stereo Matching , 2007, International Journal of Computer Vision.

[33]  Bum-Jae You,et al.  Architecture and Implementation of Real-Time Stereo Vision with Bilateral Background Subtraction , 2007, ICIC.

[34]  John Morris,et al.  Robustness to noise of stereo matching , 2003, 12th International Conference on Image Analysis and Processing, 2003.Proceedings..

[35]  Wayne Luk,et al.  Novel FPGA-based implementation of median and weighted median filters for image processing , 2005, International Conference on Field Programmable Logic and Applications, 2005..

[36]  P ? ? ? ? ? ? ? % ? ? ? ? , 1991 .

[37]  Karsten Scheibe,et al.  MFC - A Modular Line Camera for 3D World Modulling , 2008, RobVis.

[38]  Derek D. Lichti,et al.  Static Calibration and Analysis of the Velodyne HDL-64E S2 for High Accuracy Mobile Scanning , 2010, Remote. Sens..

[39]  William T. Freeman,et al.  Comparison of graph cuts with belief propagation for stereo, using identical MRF parameters , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[40]  Hans-Peter Seidel,et al.  Accurate Real-Time Disparity Estimation with Variational Methods , 2009, ISVC.

[41]  Michael J. Flynn,et al.  Very high-speed computing systems , 1966 .

[42]  Andreas Klaus,et al.  Segment-Based Stereo Matching Using Belief Propagation and a Self-Adapting Dissimilarity Measure , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[43]  Daniel P. Huttenlocher,et al.  Efficient Belief Propagation for Early Vision , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[44]  Gerhard Tröster,et al.  High-Level Area and Performance Estimation of Hardware Building Blocks on FPGAs , 2000, FPL.

[45]  Vladimir Kolmogorov,et al.  Multi-camera Scene Reconstruction via Graph Cuts , 2002, ECCV.

[46]  Jonathan Rose,et al.  Measuring the Gap Between FPGAs and ASICs , 2006, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems.

[47]  Heiko Hirschmüller,et al.  Stereo Processing by Semiglobal Matching and Mutual Information , 2008, IEEE Trans. Pattern Anal. Mach. Intell..

[48]  Ashutosh Saxena,et al.  3-D Depth Reconstruction from a Single Still Image , 2007, International Journal of Computer Vision.

[49]  Miodrag Potkonjak,et al.  MediaBench: a tool for evaluating and synthesizing multimedia and communications systems , 1997, Proceedings of 30th Annual International Symposium on Microarchitecture.

[50]  Paul F. Whelan,et al.  Projective rectification from the fundamental matrix , 2005, Image Vis. Comput..

[51]  Hong Jeong,et al.  Generalized Trellis Stereo Matching with Systolic Array , 2004, ISPA.

[52]  E. Hecht.  Optics 4th Edition , 1998 .

[53]  R. GaborSzedo.  Two-Dimensional Rank Order Filter , 2006 .

[54]  Federico Tombari,et al.  Classification and evaluation of cost aggregation methods for stereo correspondence , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[55]  Jia Yunde,et al.  Stereo Vision System on Programmable Chip (SVSoC) for Small Robot Navigation , 2006, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[56]  Masatoshi Okutomi,et al.  Sub-Pixel Estimation Error Cancellation on Area-Based Matching , 2005, International Journal of Computer Vision.

[57]  D. Scharstein,et al.  A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms , 2001, Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001).

[58]  Olaf Hochmuth,et al.  Werkzeuge der Signalverarbeitung - Grundlagen, Anwendungsbeispiele, Übungsaufgaben , 2004 .

[59]  Stefan Gehrig,et al.  Stereo matching in the presence of sub-pixel calibration errors , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[60]  Emanuele Trucco,et al.  Efficient stereo with multiple windowing , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[61]  Hong Jeong,et al.  Trellis Based Real-Time Depth Perception Chip Using Interline Constraint , 2008, New Directions in Intelligent Interactive Multimedia.

[62]  Richard Szeliski,et al.  Computer Vision - Algorithms and Applications , 2011, Texts in Computer Science.

[63]  Emanuele Trucco,et al.  A compact algorithm for rectification of stereo pairs , 2000, Machine Vision and Applications.

[64]  Dongil Han,et al.  A Novel Stereo Matching Method for Wide Disparity Range Detection , 2005, ICIAR.

[65]  V. Rodehorst,et al.  EFFICIENT SEMI-GLOBAL MATCHING FOR TRINOCULAR STEREO , 2007 .

[66]  Richard Szeliski,et al.  A Comparative Study of Energy Minimization Methods for Markov Random Fields with Smoothness-Based Priors , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[67]  Roberto Manduchi,et al.  Bilateral filtering for gray and color images , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[68]  Aaron F. Bobick,et al.  Large Occlusion Stereo , 1999, International Journal of Computer Vision.

[69]  Andreas Steininger,et al.  Hardware implementation of an SAD based stereo vision algorithm , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[70]  Barry McCullagh.  Real-time disparity map computation using the cell broadband engine , 2010, Journal of Real-Time Image Processing.

[71]  Jonathan Rose,et al.  Measuring the Gap Between FPGAs and ASICs , 2007, IEEE Trans. Comput. Aided Des. Integr. Circuits Syst..

[72]  Hong Jeong,et al.  High-speed parallel very large scale integration architecture for global stereo matching , 2008, J. Electronic Imaging.

[73]  T. Vaudrey,et al.  Differences between stereo and motion behaviour on synthetic and real-world stereo sequences , 2008, 2008 23rd International Conference Image and Vision Computing New Zealand.

[74]  Takeo Kanade,et al.  A Cooperative Algorithm for Stereo Matching and Occlusion Detection , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[75]  Miao Liao,et al.  Real-time Global Stereo Matching Using Hierarchical Belief Propagation , 2006, BMVC.

[76]  Kevin Skadron,et al.  Accelerating Compute-Intensive Applications with GPUs and FPGAs , 2008, 2008 Symposium on Application Specific Processors.

[77]  Minglun Gong,et al.  Near real-time reliable stereo matching using programmable graphics hardware , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[78]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[79]  Valerij Matrose,et al.  Optimierung der Verdrahtbarkeit unter Berücksichtigung heterogener Verdrahtungsressourcen hierarchischer FPGA-Architekturen , 2009 .

[80]  Alfred Schmitt,et al.  Real-Time Stereo by using Dynamic Programming , 2003, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[81]  Heiko Hirschmüller,et al.  Stereo Vision in Structured Environments by Consistent Semi-Global Matching , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[82]  Olga Veksler,et al.  Fast approximate energy minimization via graph cuts , 2001, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[83]  Ninad Thakoor,et al.  Real-time Planar Surface Segmentation in Disparity Space , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[84]  N. Felber,et al.  Efficient ASIC implementation of a real-time depth mapping stereo vision system , 2003, 2003 46th Midwest Symposium on Circuits and Systems.

[85]  Miao Liao,et al.  High-Quality Real-Time Stereo Using Adaptive Cost Aggregation and Dynamic Programming , 2006, Third International Symposium on 3D Data Processing, Visualization, and Transmission (3DPVT'06).

[87]  Clemens Rabe,et al.  Real-time Semi-Global Matching on the CPU , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[88]  W. James MacLean,et al.  A Real-Time Large Disparity Range Stereo-System using FPGAs , 2006, Fourth IEEE International Conference on Computer Vision Systems (ICVS'06).

[89]  Andreas Steininger,et al.  SAD-Based Stereo Matching Using FPGAs , 2009 .

[90]  Heiko Hirschmüller,et al.  Stereo vision based mapping and immediate virtual walkthroughs , 2003 .

[91]  Carlo Tomasi,et al.  Depth Discontinuities by Pixel-to-Pixel Stereo , 1999, International Journal of Computer Vision.

[92]  Brent E. Nelson,et al.  Genetic algorithms in software and in hardware-a performance analysis of workstation and custom computing machine implementations , 1996, 1996 Proceedings IEEE Symposium on FPGAs for Custom Computing Machines.

[93]  P. Reinartz,et al.  Semiglobal Matching Results on the ISPRS Stereo Matching Benchmark , 2012 .

[94]  Jonathan M. Garibaldi,et al.  Real-Time Correlation-Based Stereo Vision with Reduced Border Errors , 2002, International Journal of Computer Vision.

[95]  Chris Murphy,et al.  Low-Cost Stereo Vision on an FPGA , 2007, 15th Annual IEEE Symposium on Field-Programmable Custom Computing Machines (FCCM 2007).

[96]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[97]  Reinhard Klette,et al.  The Naked Truth about Cost Functions , 2009 .

[98]  Leon Hirsch.  Handbook Of Computer Vision And Applications , 2016 .

[99]  Dinesh Bhatia,et al.  Estimating pre-placement FPGA interconnection requirements , 2004, 17th International Conference on VLSI Design. Proceedings..