ESnet Turns on 400G Circuits to Four DOE National Labs, Supercharging Multi-Site Scientific Research

Rendering of the ESnet6 network, showing the locations of the four sites with 400G capability in orange and the two in progress in yellow.

Today’s world-changing scientific research is being conducted by collaborators at far-flung national laboratories who require high-speed, low-latency access to high performance computing facilities and specialized instruments. The Energy Sciences Network (ESnet) is proud to announce that it has supercharged the current and future bandwidth for four of the Department of Energy’s (DOE’s) national laboratories and user facilities, unleashing 400 Gigabit per second (400G) capability for Argonne National Laboratory, National Energy Research Scientific Computing Center, Oak Ridge National Laboratory, and Pacific Northwest National Laboratory. With this boost in capacity, scientists can process, analyze, visualize, share, and store the enormous quantities of research data at speeds up to four times faster than previously possible.

“It’s of vital importance that scientific researchers not be hindered by where they or their project’s instruments, computational resources, and data might be located,” said Inder Monga, executive director of ESnet. “Enabling 400G, which represents the networking industry’s current gold standard, will help facilitate that kind of seamless collaboration. We look forward to turning on 400G for more sites — and to upgrading to 800G as the technology begins to be available.”   

The 400G circuit installations were made possible by the 2022 launch of ESnet6, the sixth iteration of ESnet’s critical data circulatory system for the DOE Office of Science research complex. ESnet6 was specifically designed to support multi-facility collaborations aligned with the DOE’s new Integrated Research Infrastructure (IRI) initiative, to help DOE researchers and their international collaborators effectively harness the barrage of data generated by artificial intelligence, high-resolution instrument imagery, complex long-term global studies, and more. ESnet’s traffic is increasing by a factor of 10 every 5.5 years; in 2022, the total exceeded 1.36 exabytes. (An exabyte is equal to 1,000 petabytes or 1 billion gigabytes.)

What 400G Means for Big Science

Argonne has been involved in several collaborations that demonstrate the efficacy of integrating its supercomputers at the Argonne Leadership Computing Facility (ALCF) with experiments to accelerate scientific discoveries. Under the lab’s Nexus initiative, Argonne researchers are working with the DIII-D National Fusion Facility to enable on-demand access to ALCF supercomputers for experiment-time data analysis and predictive simulations that can be used to inform the parameters of the facility’s fast-paced plasma physics experiments. Similarly, Argonne researchers are demonstrating IRI capabilities through its ongoing efforts to tightly couple ALCF computing resources with experiments at Argonne’s Advanced Photon Source (APS), which is undergoing an upgrade that is expected to increase the volume of data generated by APS instruments by multiple orders of magnitude.

“In recent years, we’ve seen a surge in the near experiment-time analysis workflows being employed by the DOE light sources, fusion research facilities, and other large-scale experiments,” said ALCF Director Michael Papka. “As these facilities continue to evolve and improve, they’ll produce greater data volumes faster than ever before, increasing the demand for high-speed networking to the computing facilities. The ESnet upgrade is essential to keep pace with this growing scientific data deluge and meet future data-intensive research challenges.”

Oak Ridge National Laboratory (ORNL) leads the Earth Systems Grid Federation project (ESGF2) to improve the discovery, access, and storage of data used for Earth systems models and simulations and climate change research. The collaborative project, which includes Argonne and Lawrence Livermore National Laboratory, is already taking advantage of ESnet to move the planet’s largest collection of Earth System model output data between DOE high-performance computing and data facilities. The integration of ESnet is dramatically enhancing data-intensive science, facilitating data sharing, and improving access to simulation and AI platforms.

“The ESnet bandwidth upgrades at ORNL will simplify and accelerate data-integration intensive campaigns across DOE and collaborator facilities, further enabling and supporting integrated research infrastructures and projects like ESGF2,” said Mallikarjun Shankar, section head for Advanced Technologies at ORNL. 

Scientists at Pacific Northwest National Laboratory extract a foot-long soil core for study through the Molecular Observation Network. Just a single teaspoon of soil contains tens of thousands microbial species. Sorting out the proteins, metabolites, and interactions in the complex environment where soil, plants, and microbes meet results in a tremendous data stream. PNNL's new 400G connection to ESnet will enable even faster data sharing and analysis by collaborators around the nation. (Credit: Andrea Starr, Pacific Northwest National Laboratory)

And at the Environmental Molecular Sciences Laboratory, an Office of Science user facility located at PNNL, researchers have embarked on a project known as the Molecular Observation Network, a nationwide effort to understand the processes that govern what happens to carbon in soil. More carbon resides in Earth’s soil than in the atmosphere and vegetation combined, and its fate plays a huge role in our climate. It’s an incredibly active environment, with scientists collecting reams of data about soil processes. The upgrade makes it possible to transfer hundreds of gigabytes of project data in just a few minutes, not the hours previously required. The near real-time, constant data feed makes the autonomous experimentation planned at EMSL easier to implement.

“This ESNet6 upgrade presents a tremendously exciting opportunity for pursuing big science at DOE user facilities such as EMSL and at the other laboratories,” said Douglas Mans, EMSL director. “Accessing, moving, and storing massive data sets more effectively ensures U.S. scientific leadership and economic benefit.”

Speeding Up Data for High-Performance Computing 

As one of ASCR’s three high performance computing facilities, NERSC will see an immediate benefit to the 400G upgrade. Increasingly, NERSC’s more than 9,000 users move data in and out of the data center in two main ways. The first is by transferring large datasets (simulated or from an experiment) between ASCR facilities as scientists take advantage of the specialized computing available at each site. The other is by moving data from DOE experimental and observational facilities for analysis. This analysis can be very time-sensitive, with scientists requiring quick turnarounds to inform running experiments.

“The ESnet6 upgrade is essential to maintaining scientific productivity as data is increasingly transferred between experiment and computing sites,” said Debbie Bard, group lead for Data Science Engagement at NERSC. “Enabling 400G is transformational – it opens up new ways of performing science integrated across multiple DOE facilities.”

More 400G Capability on the Way

In the coming months, most of the data-intensive national labs will be upgrading to 400G service. CERN, the particle physics laboratory in Geneva whose Large Hadron Collider is one of ESnet’s largest data sources, and Lawrence Berkeley National Laboratory upgrades are almost complete. The greatly expanded capabilities offered by ESnet6 have positioned the DOE research ecosystem well to support the explosive growth in data generated by experimental observational and simulation science as well as AI/ML.

About ESnet

The Energy Sciences Network (ESnet) is a high-performance, unclassified network built to support and facilitate large-scale scientific research across America and around the world. Funded by the U.S. Department of Energy’s Office of Science and managed by Lawrence Berkeley National Laboratory, ESnet connects DOE’s national laboratory system, supercomputing facilities, and major scientific instruments, as well as peers with more than 270 research and commercial networks for global collaboration on the world’s biggest scientific challenges. ESnet’s vision is that scientific progress will be completely unconstrained by the physical location of instruments, people, computational resources, or data. 

About Argonne National Laboratory

Argonne National Laboratory seeks solutions to pressing national problems in science and technology. The nation’s first national laboratory, Argonne conducts leading-edge basic and applied scientific research in virtually every scientific discipline. Argonne researchers work closely with researchers from hundreds of companies, universities, and federal, state and municipal agencies to help them solve their specific problems, advance America’s scientific leadership and prepare the nation for a better future. With employees from more than 60 nations, Argonne is managed by UChicago Argonne, LLC for the U.S. Department of Energy’s Office of Science.

About Oak Ridge National Laboratory

Oak Ridge National Laboratory is the Department of Energy’s largest science and technology laboratory and premier research institution delivering scientific discoveries and technical breakthroughs that support the nation’s security, energy, and economic goals. ORNL manages world-class tools and user facilities, including the Oak Ridge Leadership Computing Facility (OLCF), which houses the two fastest supercomputers in the US: Summit, an IBM AC922, 200-petaflop system, and the HPE Cray EX Frontier, the world’s fastest supercomputer and first to break the exascale barrier. The OLCF also manages multiple data storage and file systems that, together with the facility’s high-performance computing systems, support research by users around the world.

About Pacific Northwest National Laboratory

Pacific Northwest National Laboratory draws on its distinguishing strengths in chemistry, Earth sciences, biology and data science to advance scientific knowledge and address challenges in sustainable energy and national security. Founded in 1965, PNNL is operated by Battelle for the Department of Energy’s Office of Science, which is the single largest supporter of basic research in the physical sciences in the United States. DOE’s Office of Science is working to address some of the most pressing challenges of our time. For more information, visit https://energy.gov/science. For more information on PNNL, visit PNNL's News Center. Follow us on Twitter, Facebook, LinkedIn and Instagram. 

About the National Energy Research Scientific Computing Center (NERSC)

The National Energy Research Scientific Computing Center (NERSC) is a U.S. Department of Energy Office of Science User Facility that serves as the primary high-performance computing center for scientific research sponsored by the Office of Science. Located at Lawrence Berkeley National Laboratory, NERSC serves more than 9,000 scientists at national laboratories and universities researching a wide range of problems in combustion, climate modeling, fusion energy, materials science, physics, chemistry, computational biology, and other disciplines.