A Roadmap for HEP Software and Computing

Particle physics has an ambitious and broad experimental programme for the coming decades. This programme requires large investments in detector hardware, either to build new facilities and experiments, or to upgrade existing ones. Similarly, it requires commensurate investment in the R&D of software to acquire, manage, process, and analyse the shear amounts of data to be recorded. In planning for the HL-LHC in particular, it is critical that all of the collaborating stakeholders agree on the software goals and priorities, and that the efforts complement each other. In this spirit, this white paper describes the R&D activities required to prepare for this software upgrade.

Formica | Gutsche | Fisk | Ivanov | Ivanchenko | Coles | Guan | Pearce | Albrecht | Chapman | Fitzpatrick | Hristov | Belforte | Ferber | Hodgkinson | Gligorov | Laycock | Farrell | Heinrich | Garonne | Gaede | Dimitrov | Franzoni | Hartmann | Crooks | Neubauer | McNab | Hauth | Perdue | Petzold | Geurts | Kuznetsov | Forti | Leggett | Konstantinov | Mitrevski | Gardner | Britton | McKee | Gray | Banerjee | Livny | Atzori | Bhimji | Asai | Mato | Chen | Huang | Linden | Childers | Kagan | Komarov | Amadio | Andronico | Aphecetche | Apostolakis | Babik | Bagliesi | Bandieramonte | Barisits | Bauerdick | Bernius | Bianchi | Bird | Biscarat | Blomer | Bloom | Boccali | Bockelman | Bonacorsi | Boveia | Bozzi | Bracko | Buckley | Buncic | Calafiura | Campaña | Canal | Canali | Castro | Cattaneo | Cerminara | Villanueva | Chang | Clarke | Clemencic | E. Cogneras | Collier | Colling | Corti | Cosmo | Couturier | Cranmer | Cranshaw | Cristella | Currie | Dallmeier ‐ Tiessen | Cian | Roeck | Peris | Dérue | Girolamo | Guida | Doglioni | Dotti | Duellmann | Duflot | Dykstra | Dziurda | Egede | Elmsheuser | Elvira | Eulisse | Filipcic | Flix | Fuess | Ganis | Gellrich | Genser | George | Gheata | Giacomini | Giagu | Giffels | Girone | Glushkov | Gohn | Caballero | Govi | Grasland | Grilló | Gyurjyan | Hanushevsky | Hariri | Harvey | Hegner | Heinemann | Heiss | Hernández | Hildreth | Hoeche | Holzman | Iven | Jashal | Jayatilaka | Jones | Jouvin | Kane | Karavakis | Katz | Kcira | Keeble | Kersevan | Kirby | Klimentov | Koppenburg | Kowalkowski | Kreczko | Kuhr | Kutschke | Lancon | Lange | Lassnig | Letts | Lewendel | Lima | Linacre | Presti | Lopienski | Magini | Marshall | Martelli | Mazumdar | McCauley | McFayden | Mehdiyev | Meinhard | Menasce | Michelotto | Moneta | Morgan | Moyse | Murray | Nairz | Norman | Novak | Oyanguren | Ozturk | Pansanel | Pascuzzi | Pearson | Pedro | Yzquierdo | Perrozzi | Petersen | Petrić | Piedra | Piilonen | Piparo | Pivarski | Pokorski | F. Polci | Potamianos | Psihas | Reuter | Ribon | Rinaldi | Polci | Bračko | Roeck | Dykstra | Gaede | Presti | Novak

[1]  H. Rabin.  Summary and perspectives, marmosets and oncology. , 1978, Primates in medicine.

[2]  F. Rademakers,et al.  ROOT — An object oriented data analysis framework , 1997 .

[3]  Iosif Legrand,et al.  Models Of Networked Analysis At Regional Centres For Lhc Experiments (monarc), Phase 2 Report, 24th March 2000 , 2000 .

[4]  Marco Cattaneo,et al.  GAUDI — A software architecture and framework for building HEP data processing applications , 2001 .

[5]  R. Pittau,et al.  ALPGEN, a generator for hard multiparton processes in hadronic collisions , 2002, hep-ph/0206293.

[6]  A. Dell'Acqua,et al.  Geant4 - A simulation toolkit , 2003 .

[7]  G Eulisse,et al.  IgProf profiling tool , 2005 .

[8]  F. Gaede.  Marlin and LCCD—Software tools for the ILC , 2006 .

[9]  Marcelino B. Santos,et al.  CMS Physics : Technical Design Report Volume 1: Detector Performance and Software , 2006 .

[10]  F. Siegert,et al.  Event generation with SHERPA 1.1 , 2008, 0811.4622.

[11]  P. Speckmayer,et al.  The toolkit for multivariate data analysis, TMVA 4 , 2010 .

[12]  Ryan Henson Creighton.  Unity 3D game development by example : beginner's guide: lite : get up and running as a Unity game developer , 2010 .

[13]  Andreas Moll,et al.  The Software Framework of the Belle II Experiment , 2011 .

[14]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[15]  K. Cranmer,et al.  RECAST — extending the impact of existing analyses , 2010, 1010.2506.

[16]  Predrag Buncic,et al.  Distributing LHC application software and conditions databases using the CernVM file system , 2011 .

[17]  Jason Weston,et al.  Natural Language Processing (Almost) from Scratch , 2011, J. Mach. Learn. Res..

[18]  Jim Kowalkowski,et al.  The art framework , 2012 .

[19]  Michelle Butler,et al.  Snowmass 2013 Computing Frontier Storage and Data Management , 2013, 1311.4580.

[20]  F. Riggi,et al.  The upgrade programme of the major experiments at the Large Hadron Collider , 2014 .

[21]  Ben Couturier,et al.  Measurements of the LHCb software stack on the ARM architecture , 2014 .

[22]  Jonathan L. Feng,et al.  Building for Discovery: Strategic Plan for U.S. Particle Physics in the Global Context , 2014 .

[23]  Andy R. Terrel,et al.  Blaze: Building A Foundation for Array-Oriented Computing in Python , 2014, SciPy.

[24]  Marco Cattaneo,et al.  Update of the Computing Models of the WLCG and the LHC Experiments , 2014 .

[25]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[26]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[27]  Christian Märtin.  Multicore Processors: Challenges, Opportunities, Emerging Trends , 2014 .

[28]  Paola Grosso,et al.  Power-aware applications for scientific cluster and distributed computing , 2014, ArXiv.

[29]  Balázs Kégl,et al.  The Higgs boson machine learning challenge , 2014, HEPML@NIPS.

[30]  James G. Shanahan,et al.  Large Scale Distributed Data Science using Apache Spark , 2015, KDD.

[31]  M. Mangano.  The Physics Landscape of the High Luminosity LHC , 2015 .

[32]  Matthew Rocklin,et al.  Dask: Parallel Computation with Blocked algorithms and Task Scheduling , 2015, SciPy.

[33]  C D Jones,et al.  Using the CMS Threaded Framework In A Production Environment , 2015 .

[34]  Lucio Rossi,et al.  High-Luminosity Large Hadron Collider (HL-LHC) : Preliminary Design Report , 2015 .

[35]  Joseph K. Bradley,et al.  Spark SQL: Relational Data Processing in Spark , 2015, SIGMOD Conference.

[36]  P. Buncic,et al.  Technical Design Report for the Upgrade of the Online-Offline Computing System , 2015 .

[37]  D Contardo,et al.  Technical Proposal for the Phase-II Upgrade of the CMS Detector , 2015 .

[38]  A Hamilton,et al.  Massive affordable computing using ARM processors in high energy physics , 2015 .

[39]  P. Mato,et al.  Gaudi components for concurrency: Concurrency for existing and future experiments , 2015 .

[40]  M. Schatz,et al.  Big Data: Astronomical or Genomical? , 2015, PLoS biology.

[41]  Tibor Simko,et al.  CERN Services for Long Term Data Preservation , 2016, iPRES.

[42]  Peter Sanders,et al.  Thrill: High-performance algorithmic distributed batch data processing with C++ , 2016, 2016 IEEE International Conference on Big Data (Big Data).

[43]  Espen Blikra,et al.  An SDN based approach for the ATLAS data acquisition network , 2016 .

[44]  M. Frank,et al.  Tesla: An application for real-time data analysis in High Energy Physics , 2016, Comput. Phys. Commun..

[45]  Andreas Pfeiffer,et al.  Functional tests of a prototype for the CMS-ATLAS common non-event data handling framework , 2017 .

[46]  Michela Paganini,et al.  CaloGAN: Simulating 3D High Energy Particle Showers in Multi-Layer Electromagnetic Calorimeters with Generative Adversarial Networks , 2017, ArXiv.

[47]  Walter Lampl,et al.  How to review 4 million lines of ATLAS code , 2017 .

[48]  Karen Y He,et al.  Big Data Analytics for Genomic Medicine , 2017, International journal of molecular sciences.

[49]  Joshua Bendavid,et al.  Efficient Monte Carlo Integration Using Boosted Decision Trees and Generative Deep Neural Networks , 2017, 1707.00028.

[50]  Lukas Heinrich,et al.  HEPData: a repository for high energy physics data , 2017, ArXiv.

[51]  B. Isildak.  Search for narrow resonances in dijet final states at s=8 TeV with the novel CMS technique of data scouting , 2017 .

[52]  T. Kuhr,et al.  HEP Software Foundation Community White Paper Working Group - Software Development, Deployment and Validation , 2017, 1712.07959.

[53]  Ilija Vukotic,et al.  A new experiment-agnostic mechanism to persistify and serve the detector geometry of ATLAS , 2017 .

[54]  Inder Monga,et al.  Beyond 100 Gb/s: capacity, flexibility, and network optimization [Invited] , 2017, IEEE/OSA Journal of Optical Communications and Networking.

[55]  Paul Laycock.  A Conditions Data Management System for HEP Experiments , 2018 .

[56]  E. Sexton-Kennedy,et al.  HEP Software Development in the Next Decade; the Views of the HSF Community , 2018, Journal of Physics: Conference Series.

[57]  Amsterdam,et al.  arXiv : HEP Community White Paper on Software trigger and event reconstruction , 2018, 1802.08638.

[58]  Massimo Lamanna,et al.  SWAN: A service for interactive analysis in the cloud , 2018, Future Gener. Comput. Syst..