Providing Computing Power for High Level Controllers in MicroTCA-based LLRF Systems via PCI Express Extension

The MicroTCA.4 standard for crate architecture allows to use a PCI Express Generation 3 bus for data transmission between the modules in a crate. This enables a software, running on a CPU module in the crate, to directly access the data, processed, i.e., by a Field Programmable Gate Array (FPGA) on another module in the same crate. The CPU performance is limited, due to the limit of cooling capacity specified for each slot, by the MicroTCA.4 standard. This limitation can be circumvented, by extending the PCI Express bus from the crate to a high performance computer. This is already practised in Low Level Radio Frequency (LLRF) control systems. This article will discuss the advantages and disadvantages of this feature with a special focus on the use for high level control algorithms. THE MicroTCA STANDARD The Micro Telecommunications Computing Architecture (MicroTCA) standard [1], maintained by the PCI Industrial Computer Manufacturers Group (PICMG) provides guidelines and requirements for the design of reliable, remote maintainable computing infrastructures. It contains multiple sub-standards for different use-cases, with MicroTCA.4 being specialized towards scientific applications. Commonly, these are realized as 19” crates. The computing infrastructure of a MicroTCA.4 crate consists of a shelf manager, called MicroTCA Carrier Hub (MCH) and a backplane. The devices, connected to the backplane, have to adhere to the Advanced Mezzanine Card (AMC) standard [2], also maintained by PICMG, and are hence referred to as AMCs. Processing Power Limitation in a MicroTCA.4 Crate The maximal thermal power load per slot in a crate is limited to 80 W, due to cooling constraints (see Section 5.6 in [1]). Thus the Thermal Design Power (TDP) of a CPU, installed on an AMC, has to be well below 80 W, as other devices on the board might also produce some thermal load. As Fig. 1 shows, the processing power of a CPU, represented by its clock frequency and the number of cores, is limited by the TDP. This limitation can be circumvented by installing multiple CPU modules into a MicroTCA.4 crate. The drawback of this work-around would be increased cost and the reduction of slots, available in a crate. Additionally, the modules, ∗ Supported by MicroTCA Techlab. † patrick.nonn@desy.de Figure 1: CPU Performance, represented by the product of clock frequency and number of cores, versus thermal design power (TDP) of various CPUs from the Intel Xeon family. connected to different CPUs, can no longer communicate with each other via PCI Express.