Julia is a mature general-purpose programming language, with a large ecosystem of libraries and more than 12000 third-party packages, which specifically targets scientific computing. As a language, Julia is as dynamic, interactive, and accessible as Python with NumPy, but achieves run-time performance on par with C/C++. In this paper, we describe the state of adoption of Julia in HEP, where momentum has been gathering over a number of years. HEP-oriented Julia packages can already, via UnROOT.jl, read HEP's major file formats, including TTree and RNTuple. Interfaces to some of HEP's major software packages, such as through Geant4.jl, are available too. Jet reconstruction algorithms in Julia show excellent performance. A number of full HEP analyses have been performed in Julia. We show how, as the support for HEP has matured, developments have benefited from Julia's core design choices, which makes reuse from and integration with other packages easy. In particular, libraries developed outside HEP for plotting, statistics, fitting, and scientific machine learning are extremely useful. We believe that the powerful combination of flexibility and speed, the wide selection of scientific
A first differentiable analysis pipeline is presented for an example high-energy physics (HEP) use case with publicly available collision data from the Compact Muon Solenoid detector at the Large Hadron Collider. The pipeline combines tools from the Scikit-HEP ecosystem with JAX. The study is based on an existing search for a hypothetical particle, the $Z^{\prime}$ boson, and uses a realistic, yet simplified, statistical model. The gradient-based optimization techniques employed in this work can advance HEP workflows by enabling end-to-end tuning of analysis parameters, improving both computational scalability and overall sensitivity. The challenges of adopting such techniques in HEP workflows are highlighted, along with practical mitigation to those challenges. This framework results in a significant improvement in expected statistical significance compared to a baseline analysis by fine-tuning $\mathcal{O}(10^3)$ parameters in the pipeline. Perspectives on future applications and recommendations for broader engagement with differentiable techniques in the field are also outlined.
Particle physics has an ambitious and broad global experimental programme for the coming decades. Large investments in building new facilities are already underway or under consideration. Scaling the present processing power and data storage needs by the foreseen increase in data rates in the next decade for HL-LHC is not sustainable within the current budgets. As a result, a more efficient usage of computing resources is required in order to realise the physics potential of future experiments. Software and computing are an integral part of experimental design, trigger and data acquisition, simulation, reconstruction, and analysis, as well as related theoretical predictions. A significant investment in computing and software is therefore critical. Advances in software and computing, including artificial intelligence (AI) and machine learning (ML), will be key for solving these challenges. Making better use of new processing hardware such as graphical processing units (GPUs) or ARM chips is a growing trend. This forms part of a computing solution that makes efficient use of facilities and contributes to the reduction of the environmental footprint of HEP computing. The HEP community
In November 2022, the HEP Software Foundation and the Institute for Research and Innovation for Software in High-Energy Physics organized a workshop on the topic of Software Citation and Recognition in HEP. The goal of the workshop was to bring together different types of stakeholders whose roles relate to software citation, and the associated credit it provides, in order to engage the community in a discussion on: the ways HEP experiments handle citation of software, recognition for software efforts that enable physics results disseminated to the public, and how the scholarly publishing ecosystem supports these activities. Reports were given from the publication board leadership of the ATLAS, CMS, and LHCb experiments and HEP open source software community organizations (ROOT, Scikit-HEP, MCnet), and perspectives were given from publishers (Elsevier, JOSS) and related tool providers (INSPIRE, Zenodo). This paper summarizes key findings and recommendations from the workshop as presented at the 26th International Conference on Computing in High Energy and Nuclear Physics (CHEP 2023).
In view of the European Strategy for Particle Physics process, the French HEP community has organized a national process of collecting written contributions and has pursued a series of workshops culminating with a national symposium held in Paris on January 20-21, 2025 that involved over 280 scientists https://indico.in2p3.fr/event/34662/. The present document summarises the main conclusions of this bottom-up approach centred on the physics and technology motivations.
The U.S. CMS collaboration has designed a novel internship program for undergraduates to enhance the participation of students from under-represented populations, including those at minority serving institutions, in High Energy Physics (HEP). These students traditionally face several barriers including lack of research infrastructure and opportunities, insufficient mentoring, lack of support networks, and financial hardship, among many others, resulting in a lack of participation in STEM fields. We had recently reported about a fully virtual 10-week internship pilot program called "U.S. CMS - PURSUE (Program for Undergraduate Research SUmmer Experience)" to address dismantling such barriers. The 2023 iteration of this program builds on it by imparting not only an in-person summer internship experience but extends it into the academic semester as well. Students are selected predominantly from Minority Serving Institutions with no research program in HEP and from under-represented groups. They experience a structured hands-on research experience with an initial two-week "bootcamp" on software training modules followed by an 8-week HEP project targeting physics analysis, software, com
The INSPIRE platform -- the most widely-used discovery service specifically tailored to the needs of researchers in High Energy Physics (HEP) -- has become a central component of the information infrastructure for the discipline. Despite this, INSPIRE's continued sustainability is frequently endangered by resource constraints, recently made more acute by the loss of support from historical funders changing their research priorities. If the European particle physics community wishes to ensure INSPIRE's long-term sustainability, the community should secure international support and ensure appropriate funding.
The increasing complexity of modern neural network architectures demands fast and memory-efficient implementations to mitigate computational bottlenecks. In this work, we evaluate the recently proposed BitNet architecture in HEP applications, assessing its performance in classification, regression, and generative modeling tasks. Specifically, we investigate its suitability for quark-gluon discrimination, SMEFT parameter estimation, and detector simulation, comparing its efficiency and accuracy to state-of-the-art methods. Our results show that while BitNet consistently performs competitively in classification tasks, its performance in regression and generation varies with the size and type of the network, highlighting key limitations and potential areas for improvement.
Predicting the performance of various infrastructure design options in complex federated infrastructures with computing sites distributed over a wide area network that support a plethora of users and workflows, such as the Worldwide LHC Computing Grid (WLCG), is not trivial. Due to the complexity and size of these infrastructures, it is not feasible to deploy experimental test-beds at large scales merely for the purpose of comparing and evaluating alternate designs. An alternative is to study the behaviours of these systems using simulation. This approach has been used successfully in the past to identify efficient and practical infrastructure designs for High Energy Physics (HEP). A prominent example is the Monarc simulation framework, which was used to study the initial structure of the WLCG. New simulation capabilities are needed to simulate large-scale heterogeneous computing systems with complex networks, data access and caching patterns. A modern tool to simulate HEP workloads that execute on distributed computing infrastructures based on the SimGrid and WRENCH simulation frameworks is outlined. Studies of its accuracy and scalability are presented using HEP as a case-study.
We present a transformer architecture-based foundation model for tasks at high-energy particle colliders such as the Large Hadron Collider. We train the model to classify jets using a self-supervised strategy inspired by the Joint Embedding Predictive Architecture. We use the JetClass dataset containing 100M jets of various known particles to pre-train the model with a data-centric approach -- the model uses a fraction of the jet constituents as the context to predict the embeddings of the unseen target constituents. Our pre-trained model fares well with other datasets for standard classification benchmark tasks. We test our model on two additional downstream tasks: top tagging and differentiating light-quark jets from gluon jets. We also evaluate our model with task-specific metrics and baselines and compare it with state-of-the-art models in high-energy physics. Project site: https://hep-jepa.github.io/
The Caltech HEP Crystal Lab has been actively investigating novel inorganic scintillators along the following three directions. Fast and radiation hard inorganic scintillators to face the challenge of severe radiation environment expected by future HEP experiments at hadron colliders, such as the high luminosity LHC and FCC hh. Ultrafast inorganic scintillators to face the challenge of unprecedented event rate expected by future HEP experiments searching for rare decays, such as Mu2e II, and ultrafast time of flight system at hadron colliders. Cost effective inorganic scintillators for the homogeneous hadron calorimeter concept to face the challenge of both electromagnetic and jet mass resolutions required by the proposed Higgs factory. We report novel materials along all directions: LuAG:Ce ceramic fibers for the HL LHC, Lu2O3:Yb ceramic scintillators for ultrafast applications, and ABS:Ce and DSB:Ce glass scintillators for the proposed Higgs factory. The result of this investigation may also benefit nuclear physics experiments, GHz hard X ray imaging, medical imaging, and homeland security applications.
Many high-energy-physics (HEP) simulations for the LHC rely on Monte Carlo using importance sampling by means of the VEGAS algorithm. However, complex high-precision calculations have become a challenge for the standard toolbox, as this approach suffers from poor performance in complex cases. As a result, there has been keen interest in HEP for modern machine learning to power adaptive sampling. While previous studies have shown the potential of normalizing-flow-powered neural importance sampling (NIS) over VEGAS, there remains a gap in accessible tools tailored for non-experts. In response, we introduce ZüNIS, a fully automated NIS library designed to bridge this divide, while at the same time providing the infrastructure to customise the algorithm for dealing with challenging tasks. After a general introduction on NIS, we first show how to extend the original formulation of NIS to reuse samples over multiple gradient steps while guaranteeing a stable training, yielding a significant improvement for slow functions. Next, we introduce the structure of the library, which can be used by non-experts with minimal effort and is extensivly documented, which is crucial to become a mature
Long term sustainability of the high energy physics (HEP) research software ecosystem is essential for the field. With upgrades and new facilities coming online throughout the 2020s this will only become increasingly relevant throughout this decade. Meeting this sustainability challenge requires a workforce with a combination of HEP domain knowledge and advanced software skills. The required software skills fall into three broad groups. The first is fundamental and generic software engineering (e.g. Unix, version control,C++, continuous integration). The second is knowledge of domain specific HEP packages and practices (e.g., the ROOT data format and analysis framework). The third is more advanced knowledge involving more specialized techniques. These include parallel programming, machine learning and data science tools, and techniques to preserve software projects at all scales. This paper dis-cusses the collective software training program in HEP and its activities led by the HEP Software Foundation (HSF) and the Institute for Research and Innovation in Software in HEP (IRIS-HEP). The program equips participants with an array of software skills that serve as ingredients from whic
HEP data-processing software must support the disparate physics needs of many experiments. For both collider and neutrino environments, HEP experiments typically use data-processing frameworks to manage the computational complexities of their large-scale data processing needs. Data-processing frameworks are being faced with new challenges this decade. The computing landscape has changed from the past three decades of homogeneous single-core x86 batch jobs running on grid sites. Frameworks must now work on a heterogeneous mixture of different platforms: multi-core machines, different CPU architectures, and computational accelerators; and different computing sites: grid, cloud, and high-performance computing. We describe these challenges in more detail and how frameworks may confront them. Given their historic success, frameworks will continue to be critical software systems that enable HEP experiments to meet their computing needs. Frameworks have weathered computing revolutions in the past; they will do so again with support from the HEP community
Superconducting sensors are a key enabling technology for many HEP experiments with advances in sensor capabilities leading directly to expanded science reach. The unique materials and processes required for the fabrication of these sensors makes commercial sourcing impractical in comparison with semiconducting devices. Subsequently, the development and fabrication of new sensors are often performed at academic cleanrooms supported through HEP basic detector research and/or project funds. While this operational model has been successful to date, we are at a turning point in the history of superconducting electronics, as evidenced by the rapid growth in the field of quantum computing, when scale and sophistication of these sensors can lead to significant progress. In order to achieve this progress and meet the needs of the next generations of HEP experiments, continued support of all stages of the superconducting sensors development pipeline is necessary.
As particle physics experiments push their limits on both the energy and the intensity frontiers, the amount and complexity of the produced data are also expected to increase accordingly. With such large data volumes, next-generation efforts like the HL-LHC and DUNE will rely even more on both high-throughput (HTC) and high-performance (HPC) computing clusters. Full utilization of HPC resources requires scalable and efficient data-handling and I/O. For the last few decades, ROOT has been used by most HEP experiments to store data. However, other storage technologies like HDF5 may perform better in HPC environments. Initial explorations with HDF5 have begun using ATLAS, CMS and DUNE data; the DUNE experiment has also adopted HDF5 for its data-acquisition system. This paper presents the future outlook of the HEP computing and the role of HPC, and a summary of ongoing and future works to use HDF5 as a possible data storage technology for the HEP experiments to use in HPC environments.
Machine Learning (ML) will play a significant role in the success of the upcoming High-Luminosity LHC (HL-LHC) program at CERN. An unprecedented amount of data at the exascale will be collected by LHC experiments in the next decade, and this effort will require novel approaches to train and use ML models. In this paper, we discuss a Machine Learning as a Service pipeline for HEP (MLaaS4HEP) which provides three independent layers: a data streaming layer to read High-Energy Physics (HEP) data in their native ROOT data format; a data training layer to train ML models using distributed ROOT files; a data inference layer to serve predictions using pre-trained ML models via HTTP protocol. Such modular design opens up the possibility to train data at large scale by reading ROOT files from remote storage facilities, e.g. World-Wide LHC Computing Grid (WLCG) infrastructure, and feed the data to the user's favorite ML framework. The inference layer implemented as TensorFlow as a Service (TFaaS) may provide an easy access to pre-trained ML models in existing infrastructure and applications inside or outside of the HEP domain. In particular, we demonstrate the usage of the MLaaS4HEP architect
Benchmarking of CPU resources in WLCG has been based on the HEP-SPEC06 (HS06) suite for over a decade. It has recently become clear that HS06, which is based on real applications from non-HEP domains, no longer describes typical HEP workloads. The aim of the HEP-Benchmarks project is to develop a new benchmark suite for WLCG compute resources, based on real applications from the LHC experiments. By construction, these new benchmarks are thus guaranteed to have a score highly correlated to the throughputs of HEP applications, and a CPU usage pattern similar to theirs. Linux containers and the CernVM-FS filesystem are the two main technologies enabling this approach, which had been considered impossible in the past. In this paper, we review the motivation, implementation and outlook of the new benchmark suite.
Future HEP experiments at the energy and intensity frontiers present stringent challenges to inorganic scintillators in radiation tolerance, ultrafast time response and cost. This paper reports recent progress in radiation hard, ultrafast, and cost-effective inorganic scintillators for future HEP experiments. Examples are LYSO crystals for a precision time of flight detector, LuAG ceramics for an ultracompact, radiation hard shashlik sampling calorimeter, BaF2:Y crystals for an ultrafast calorimeter, and cost-effective scintillators for a homogeneous hadron calorimeter. Applications for Gigahertz hard X-ray imaging will also be discussed.
This draft report summarizes and details the findings, results, and recommendations derived from the ASCR/HEP Exascale Requirements Review meeting held in June, 2015. The main conclusions are as follows. 1) Larger, more capable computing and data facilities are needed to support HEP science goals in all three frontiers: Energy, Intensity, and Cosmic. The expected scale of the demand at the 2025 timescale is at least two orders of magnitude -- and in some cases greater -- than that available currently. 2) The growth rate of data produced by simulations is overwhelming the current ability, of both facilities and researchers, to store and analyze it. Additional resources and new techniques for data analysis are urgently needed. 3) Data rates and volumes from HEP experimental facilities are also straining the ability to store and analyze large and complex data volumes. Appropriately configured leadership-class facilities can play a transformational role in enabling scientific discovery from these datasets. 4) A close integration of HPC simulation and data analysis will aid greatly in interpreting results from HEP experiments. Such an integration will minimize data movement and facilita