Application-Oriented On-Board Optical Technologies for HPCs

The increased communication bandwidth demands of high performance computing (HPC) systems calling at the same time for reduced latency and increased power efficiency have designated optical interconnects as the key technology in order to achieve the target of exascale performance. In this realm, technology advances have to be accompanied by the development of corresponding design and simulation tools that support end-to-end system modeling in order to evaluate the performance benefits offered by optical components at system scale. In this paper, we present recent advances on electro-optical printed circuit boards (EOPCB) technology development pursued within the European FP7 PhoxTroT research program and directed toward system-scale performance benefits in real HPC workload applications. We report on high-density and multilayered EOPCBs together with all necessary building blocks for enabling true optical blade technology, including multimode polymer-based single- and dual-layer EOPCBs, a board-compatible optically interfaced router chip, and passive board-level connectors. We also demonstrate a complete optical blade design and evaluation software simulation framework called OptoHPC that tailors optical blade technology development toward optimized performance at HPC system scale, allowing for its validation with synthetic workload benchmark traffic profiles and for reliable comparison with existing HPC platforms. The OptoHPC simulator is finally utilized for evaluating and comparing a 384-node HPC system relying on optically enabled blades with the state-of-the-art Cray XK7 HPC network when performing with a range of synthetic workload traffic profiles, revealing the significant throughput and delay improvements that can be released through application-oriented optical blade technology.

[1]  K. Hasharoni,et al.  A high end routing platform for core and edge applications based on chip to chip optical interconnect , 2013, 2013 Optical Fiber Communication Conference and Exposition and the National Fiber Optic Engineers Conference (OFC/NFOEC).

[2]  Y. Matsuoka,et al.  A High-Coupling-Efficiency Multilayer Optical Printed Wiring Board with a Cube-Core Structure for High-Density Optical Interconnections , 2007, 2007 Proceedings 57th Electronic Components and Technology Conference.

[3]  Nikos Pleros,et al.  Bringing OptoBoards to HPC-scale environments: An OptoHPC simulation engine , 2016, AISTECS '16.

[4]  Christian Engelmann,et al.  xSim: The extreme-scale simulator , 2011, 2011 International Conference on High Performance Computing & Simulation.

[5]  Alexander V. Rylyakov,et al.  Monolithic Silicon Integration of Scaled Photonic Switch Fabrics, CMOS Logic, and Device Driver Circuits , 2014, Journal of Lightwave Technology.

[6]  Lars Brusberg,et al.  Pluggable Electro-Optical Circuit Board Interconnect Based on Embedded Graded-Index Planar Glass Waveguides , 2015, Journal of Lightwave Technology.

[7]  Lars Brusberg,et al.  Thin glass based electro-optical circuit board (EOCB): Waveguide process, PCB technology, and coupling interfaces , 2015, 2015 IEEE CPMT Symposium Japan (ICSJ).

[8]  Shinji Nishimura,et al.  A 25-Gb/s 2.2-W 65-nm CMOS Optical Transceiver Using a Power-Supply-Variation-Tolerant Analog Front End and Data-Format Conversion , 2014, IEEE Journal of Solid-State Circuits.

[9]  P. Maniotis,et al.  Performance analysis and layout design of optical blades for HPCs using the OptoBoard-Sim simulator , 2015, 2015 IEEE Optical Interconnects Conference (OI).

[11]  Kai Wang,et al.  Demonstration of fully enabled data center subsystem with embedded optical interconnect , 2014, Photonics West - Optoelectronic Materials and Devices.

[12]  A. Wonfor,et al.  Terabit capacity passive polymer optical backplane , 2008, 2008 Conference on Lasers and Electro-Optics and 2008 Conference on Quantum Electronics and Laser Science.

[13]  Luca P. Carloni,et al.  PhoenixSim: A simulator for physical-layer analysis of chip-scale photonic interconnection networks , 2010, 2010 Design, Automation & Test in Europe Conference & Exhibition (DATE 2010).

[14]  Mikawa Takashi.  Low-cost, high-density optical parallel link modules and optical backplane for the last 1 meter regime applications , 2008 .

[15]  Matt Ezell,et al.  Understanding the Impact of Interconnect Failures on System Operation , 2013 .

[16]  J. Beals,et al.  Cost-Effective Multimode Polymer Waveguides for High-Speed On-Board Optical Interconnects , 2009, IEEE Journal of Quantum Electronics.

[17]  B. J. Offrein,et al.  FirstLight: Pluggable Optical Interconnect Technologies for Polymeric Electro-Optical Printed Circuit Boards in Data Centers , 2012, Journal of Lightwave Technology.

[18]  Yurii A. Vlasov,et al.  Optical technologies for data communication in large parallel systems , 2011 .

[19]  T. Rapala-Virtanen,et al.  Electro-optical backplane demonstrator with multimode polymer waveguides for board-to-board interconnects , 2014, Proceedings of the 5th Electronics System-integration Technology Conference (ESTC).

[20]  Henning Schröder,et al.  Display glass for low-loss and high-density optical interconnects in electro-optical circuit boards with eight optical layers. , 2015, Optics express.

[21]  Kai Wang,et al.  Electro-optical backplane demonstrator with integrated multimode gradient-index thin glass waveguide panel , 2015, Photonics West - Optoelectronic Materials and Devices.

[22]  Ray T. Chen,et al.  Fully embedded board-level guided-wave optoelectronic interconnects , 2000, Proceedings of the IEEE.

[23]  K. Adachi,et al.  20-Gb/s/ch High-Speed Low-Power 1-Tb/s Multilayer Optical Printed Circuit Board With Lens-Integrated Optical Devices and CMOS IC , 2011, IEEE Photonics Technology Letters.

[24]  Richard V. Penty,et al.  40 Gb/s Data Transmission Over a 1-m-Long Multimode Polymer Spiral Waveguide for Board-Level Optical Interconnects , 2015, Journal of Lightwave Technology.

[25]  Yan Yan,et al.  All-Optical Programmable Disaggregated Data Centre Network Realized by FPGA-Based Switch and Interface Card , 2016, Journal of Lightwave Technology.

[26]  Somayeh Sardashti,et al.  The gem5 simulator , 2011, CARN.

[27]  L. Schares,et al.  160-Gb/s, 16-Channel Full-Duplex, Single-Chip CMOS Optical Transceiver , 2007, OFC/NFOEC 2007 - 2007 Conference on Optical Fiber Communication and the National Fiber Optic Engineers Conference.

[28]  Emmanouel A. Varvarigos,et al.  Laying out interconnects on optical printed circuit boards , 2014, 2014 ACM/IEEE Symposium on Architectures for Networking and Communications Systems (ANCS).

[29]  C. Schow,et al.  Terabit/s-Class Optical PCB Links Incorporating 360-Gb/s Bidirectional 850 nm Parallel Optical Transceivers , 2012, Journal of Lightwave Technology.

[30]  Felix Betschon,et al.  960 Gb/s Optical Backplane Ecosystem Using Embedded Polymer Waveguides and Demonstration in a 12G SAS Storage Array , 2013, Journal of Lightwave Technology.

[31]  T. Mikawa.  Low-cost high-density optical parallel link modules and optical backplane for the last 1-meter regime applications , 2008, SPIE OPTO.

[32]  Yasuhiko Arakawa,et al.  Development of high-density single-mode polymer waveguides with low crosstalk for chip-to-chip optical interconnection. , 2013, Optics express.

[33]  K. Hasharoni,et al.  336-Channel electro-optical interconnect: Underfill process improvement, fiber bundle and reliability results , 2014, 2014 IEEE 64th Electronic Components and Technology Conference (ECTC).

[34]  Takaaki Ishigure,et al.  Low-loss design and fabrication of multimode polymer optical waveguide circuit with crossings for high-density optical PCB , 2013, 2013 IEEE 63rd Electronic Components and Technology Conference.

[35]  A. Varga,et al.  THE OMNET++ DISCRETE EVENT SIMULATION SYSTEM , 2003 .

[36]  Avi Mamman,et al.  On-chip optics for data center interconnects. Prospects and limitations , 2016, 2016 IEEE CPMT Symposium Japan (ICSJ).

[37]  P. Chiniwalla,et al.  MCM LGA package with optical I/O passively aligned to dual layer polymer waveguides in PCB , 2006, 56th Electronic Components and Technology Conference 2006.

[38]  William J. Dally,et al.  Principles and Practices of Interconnection Networks , 2004 .

[39]  Kevin T. Pedretti,et al.  SST + gem5 = a scalable simulation infrastructure for high performance computing , 2012, SimuTools.

[40]  Jung Ho Ahn,et al.  Corona: System Implications of Emerging Nanophotonic Technology , 2008, 2008 International Symposium on Computer Architecture.

[41]  L. Schares,et al.  Terabus: Terabit/Second-Class Card-Level Optical Interconnect Technologies , 2006, IEEE Journal of Selected Topics in Quantum Electronics.

[42]  T. Rapala-Virtanen,et al.  Development of electro-optical PCBs with embedded waveguides for data center and high performance computing applications , 2014, Photonics West - Optoelectronic Materials and Devices.

[43]  Jeffrey A. Kash,et al.  Optical interconnects for high performance computing , 2012, 2009 Asia Communications and Photonics conference and Exhibition (ACP).

[44]  D. Milward,et al.  Optical 8-channel, 10 Gb/s MT pluggable connector alignment technology for precision coupling of laser and photodiode arrays to polymer waveguide arrays for optical board-to-board interconnects , 2008, 2008 58th Electronic Components and Technology Conference.

[45]  Ravishankar K. Iyer,et al.  Lessons Learned from the Analysis of System Failures at Petascale: The Case of Blue Waters , 2014, 2014 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks.

[46]  R. Beyeler,et al.  Polymer-Waveguide-Based Board-Level Optical Interconnect Technology for Datacom Applications , 2008, IEEE Transactions on Advanced Packaging.