The Technological Heavyweight You’ve Probably Never Heard Of: ESnet

This article originally appeared on the DOE Office of Science website.

One lone computer, sitting in the Department of Energy’s (DOE) Lawrence Livermore National Laboratory (LLNL), didn’t seem all that significant at the time. While in 1974 computers weren’t as ubiquitous as they are today, they were widely used in the research community. In fact, the fusion researchers using the computer had borrowed it from other scientists at LLNL. The system’s single stand-out feature was that it had four dial-up lines that could send items to other computers at the astounding speed of 10 characters per second.

But those four lines were the first piece of a network that’s now the technological superhighway for DOE Office of Science data. Of course, that original computer has long since been retired. But its legacy lives on in the Energy Sciences Network (ESnet), a DOE Office of Science user facility.

Since that first computer more than 30 years ago, ESnet has expanded to connect more than 40 major research institutions at speeds 15,000 times faster than a home network. From acting as an early adopter of protocols that now run the Internet to making today’s scientific discoveries possible, ESnet is the big player in the Internet you’ve probably never heard of.

How ESnet Helped Lay the Foundations for the Internet

The fusion researchers at LLNL soon expanded that lone computer to a cluster of connected computers. In 1976, they opened up the cluster to remote users, creating the first computing network devoted to transferring scientific data. While other computer networks existed – such as ARPANET, considered the grandfather of the Internet – they were only used for researching network technology. By 1977, the LLNL network had users in 13 states, including other DOE national laboratories. Users could log on and take advantage of some of the fastest computers in the world as if they were in the office next door. In 1986, this network merged with one devoted to high energy physics and formally became ESnet.

This 1977 map outlined ESnet's reach (then located at LLNL and called MFEnet). Today, the high-speed network connects over 40 research sites and facilities and reaches across the Atlantic to bring data from the Large Hadron Collider to the U.S. »Learn more about ESnet's history.

Everything was humming along nicely – until the Internet broke. At the time – October 1986 – the Internet had about 10,000 users. The entire system went from sending 56,000 bits/second to 320 bits/second. (Today’s average speed across the entire U.S. system is 14.2 million bits/second.) An email that should have taken minutes to transmit took all day. After puzzling over the problem for months, two Lawrence Berkeley National Laboratory (LBNL) researchers who worked on ESnet met at a coffeehouse and had a moment of realization: the routers were sending information quickly from individual computers, overloading the system. Data was coming down the on-ramps of the information superhighway just fine, but hit a traffic jam once it got to the road. Every time the entire network rejected a packet of information, the router sent it again, jamming everything up even more. To resolve the problem, those same Berkeley researchers—Van Jacobson and Mike Karels—designed a protocol that required routers to wait a set period of time before trying again. ESnet was an early adopter of these protocols, and ESnet’s testing allowed other systems to adopt these protocols, which led to them becoming part of the foundation of today’s Internet.

“[ESnet] has always been pushing the boundaries of network technology,” said ESnet director Inder Monga.

ESnet continued its boundary-pushing in the late 1980s and 1990s, collaborating with Bell Labs, Apple, and HP on new network technologies. It supported the development of a high-speed system that allowed storing large amounts of data on a “virtual disk” – a forerunner to cloud computing. One of its first uses was an imaging program called TerraVision. Its ability to allow users to explore a 3D landscape was a precursor to both DOE’s earth systems models and Google Earth.

Tapping into the Fastest Computers in the World

Today, ESnet continues to make it possible for researchers to access data and supercomputing power far beyond their physical location. Unlike commercial services, ESnet exclusively serves the needs of scientists.

Just like they did in the 1970s, fusion scientists today use ESnet to access cutting-edge computers. Stephen Jardin at DOE’s Princeton Plasma Physics Laboratory regularly connects to the National Energy Research Scientific Computing Center (NERSC) at LBNL, one of the Office of Science’s supercomputing user facilities. He runs complex, nonlinear simulations of plasma in devices used to study fusion. “It’s effortless. We just connect to the new computer and voila, we have much higher computational speed,” said Jardin. “It’s pretty much invisible to the user.”

This specialized network is also essential for moving huge amounts of data at a time. “A network performance issue, no matter how small, means a lot of delay for the scientist,” said Monga. A delay could result in wasting the time researchers have been allocated on a supercomputer, where every hour of runtime counts.

This speed and efficiency played a key role in helping move data from the STAR detector at the Relativistic Heavy Ion Collider (RHIC), a DOE Office of Science user facility at DOE’s Brookhaven National Laboratory. To study the substance that existed at the beginning of the universe, STAR records billions of particle collisions. To analyze the overwhelming amount of data, the team wanted to tap into NERSC’s power. A team with staff members from ESnet and NERSC developed a new method to move data three times faster than the RHIC researchers needed. Once the new method debuted, scientists were able to access processed data in a week. In the past, they had to wait months for results.

Enabling Worldwide Collaboration

It’s one thing to move data cross-country. It’s another to move it around the world. ESnet enables both.

“The expertise and knowledge are distributed around the world,” said Bo Jayatilaka, a physicist at DOE’s Fermilab. “Coming together to build these projects means that we get more out of these resources than if we build a single facility in one country.”

The ESnet team recognized this need early on. The network first connected to CERN, the multinational, high energy physics facility based in Switzerland, back in 1988. This connection allowed scientists to draw on the storage capacity of multiple locations, share data easily, and keep several computers available for analysis. The need for this connection only increased when CERN debuted the Large Hadron Collider (LHC), the world’s largest particle accelerator, in 2008. To fulfill the LHC’s research goal, computers must scour data on more particle collisions than there are stars in the Milky Way to find a single “interesting” result.

“In order to be able to process the LHC data at all, we need to have this international network,” said Ian Bird, the LHC computing grid project leader. At the time, no one had established a network with the level of reliability the LHC team needed. CERN and the California Institute of Technology (CalTech) collaborated on the network technology. They then implemented it on ESnet and its international partners’ networks. Referring to the LHC’s global network, Bird said, “This is one of the most fantastic success stories.”

To strengthen that network, ESnet established four high-speed, underwater links between the U.S. and Europe in 2014. Each cable acts as a backup for the others. With American researchers responsible for analyzing 25 percent of CERN’s data, they need to ensure a stable connection. “We have to move an enormous amount of data across the Atlantic,” said Jayatilaka. “The speed of the ESnet resources that link all of these sites makes it possible.”

CERN is far from DOE’s only international collaboration. Katrin Heitmann at Argonne National Laboratory runs models that create data similar to what scientists expect to come from the Large Synoptic Survey Telescope (LSST). A massive camera that will take photos of the cosmos, the LSST is scheduled to launch in 2022. Heitmann is working to ensure the accuracy of scientists’ data analysis techniques. An international collaboration of scientists is creating and analyzing these simulations and transferring data between Argonne, NERSC, France, and the UK. ESnet connects the U.S. and European hubs. “The data amounts are large, so it has to be fast,” she said. “Distributed collaborations make it very important to have these very fast data transfers.”

Both collaborations’ work requires scientists be able to reserve space on the network. The ESnet team developed the On-demand Secure Circuits and Reservation System (OSCARS) to give them more control over how the data flows. Scientists can schedule major data transfers and give priority to big, critical data sets over less urgent ones. Using this software, the ESnet team set a record in 2018 for moving the biggest amount of data across the country in the shortest period of time. The software received an R&D100 award in 2013 and a DOE Secretary’s Achievement Award in 2015.

Analyzing Light Source Data on the Fly

Understanding the process of photosynthesis requires the fastest X-ray light source in the world, the Linac Coherent Light Source, a DOE Office of Science user facility at DOE’s SLAC National Accelerator Laboratory. In the past, researchers took images of the process and brought them back to their institution for analysis. An ESnet connection between LCLS and NERSC established in 2013 allows scientists to transfer their experimental data as the light source is running. They receive analyzed data back in five to 10 minutes. Using that data, scientists can then quickly judge the quality of the sample and experimental set-up and tweak as necessary in real time.

“Our beam time is so limited that we don’t want to waste the time,” said Junko Yano, a LBNL researcher. “We are basically building a very fast feedback loop.” This is critical for looking at the different states of Photosystem II, an assembly of large protein molecules that play a crucial role in photosynthesis. Yano and her colleagues are trying to develop a complete high-resolution atomic movie of how Photosystem II works.

Future light sources promise to deliver even more data for ESnet to carry. LCLS-II will produce 10 to 10,000 times more images per second than LCLS does now.

“As we get to bigger and bigger scale problems, the network is being used to share the resources and infrastructure effectively across the labs,” said Monga.

Connecting to the Future

The ESnet team has big plans to keep up with the evolution of DOE’s user facilities. They’re in the process of upgrading to ESnet6. With twice the capacity of the current system, the new system will be able to carry the massive amount of data produced and analyzed by the next generation of computers that DOE expects to launch in 2021.

But ESnet is more than a physical network. “We also do social networking to make sure we have all of the connections and relationships to get the data where it needs to go,” said Monga. About 80 percent of ESnet’s traffic is from research institutions that aren’t part of DOE’s main network. “By lifting all boats, we lift the capabilities that the scientists can use for discovery,” he said.

The ESnet team also releases open-source networking software and provides technical assistance to other institutions. More than 100 institutions around the world use the Science DMZ system, a set of tools developed by the ESnet team that makes it easier to move large data sets. To date, ESnet staff members have led many physical and virtual Science DMZ workshops. These workshops have reached an audience of about 800 people across 360 institutions in 39 states in the United States and 38 countries worldwide.

These days, ESnet functions so smoothly that most people don’t even notice it, just like the computer that started the whole network. But to fulfill ESnet’s mission of serving science, the team wouldn’t want it any other way.          

The Office of Science is the single largest supporter of basic research in the physical sciences in the United States and is working to address some of the most pressing challenges of our time. For more information please visit https://www.energy.gov/science.

Shannon Brescher Shea (shannon.shea@science.doe.gov) is a senior writer/editor in the Office of Science’s Office of Communication and Public Affairs.