ESnet4 Helps Researchers Seeking the Origins of Matter

Approaching the speed of light, millions of protons will collide per second when the Large Hadron Collider (LHC) comes online this fall. The experiment will generate more data than the international scientific community has ever tried to manage. Scientists suspect the outcome of these “subatomic smashups” will provide valuable insights into the origins of matter and dark energy in the Universe.

As thousands of researchers across the globe anxiously await the results of this experiment, getting the massive amounts of data to them is no insignificant task. Fortunately, network engineers at the U.S. Department of Energy’s (DOE) Energy Sciences Network (ESnet) foresaw this data challenge years ago and developed ESnet4, a new large-scale science data transport network with enough bandwidth to transport multiple streams of 10 gigabits of information per second — the equivalent of transmitting 500 hours of digital music per second for each 10 gigabit line.

The LHC, which straddles the Swiss and French borders on the outskirts of Geneva, will be the first experiment to fully utilize the advanced capabilities of this network, which connects DOE national laboratories to researchers across the country and collaborators worldwide.

“ESnet4 is one of the most robust scientific data networks in existence,” says Steve Cotter, Department Head for ESnet. “The science environment of today is very different from that of a few years ago. ESnet4 provides the high-speed, extremely reliable connectivity between labs and U.S. and international research institutions required to support the inherently collaborative, global nature of modern large-scale science.”

ESnet is funded by the DOE and based at the Lawrence Berkeley National Laboratory, Berkeley, Calif.

Flowing Information to America

The European Center for Nuclear Research (CERN), which manages the LHC, will initially collect the experiment’s data. The information will then migrate across the Atlantic Ocean via fiber optics on a network called USLHCnet, which is managed by researchers at the California Institute of Technology in Pasadena, Calif.

Like a virtual Ellis Island, an ESnet hub on 8th Street in Manhattan will be the US entry point for LHC data. From there, ESnet will deliver data from the LHC’s ATLAS detector to Brookhaven National Laboratory in Upton, N.Y., where it will be processed and stored. Meanwhile, data from the LHC’s CMS detector will go to the Fermi National Accelerator Laboratory in Batavia, Ill., for processing and storage.

Researchers at universities and DOE laboratories across the country will then be able to connect to these databases through ESnet4, the DOE’s next-generation scientific network. Internet2, the country’s leading education and research network, and ESnet officially launched a partnership in 2006 to develop and deploy ESnet4 just in time for the LHC experiment.

To maximize efficiency, ESnet4 utilizes three main elements:

• A circuit-oriented Science Data Network for moving terabytes of data. Like a direct line connecting two endpoints, this dedicated network allows information to flow directly at high data rates from one remote host to another.

• An Internet Protocol (IP) network for typical data transfers. Unlike the Science Data Network, the IP network is connected to many computers, and can have multiple endpoints. To reach a destination, information traveling on an IP network will constantly encounter “gateways” that quickly direct and redirect it. Like air-traffic controllers, the gateways virtually determine which routes are preferred and find the most efficient routes for travel. Because information will move through numerous gateways before reaching its destination, IP networks are not the most efficient tool for moving massive datasets. Like hundreds of cars trying to pass on a toll road, large datasets can cause virtual “traffic jams” if too much information is trying to pass through the gateways. Thus, the Science Data Network is ideal for moving large datasets.

• Metropolitan Area Networks (MANs) are the last component of ESnet4. This system carries both Science Data and IP networks to effectively connect research centers in the same geographic region. Currently, 11 ESnet sites are served by MANs. The Long Island MAN and the Chicago Area MAN were specifically built to facilitate the movement of data from the LHC experiments.

“LHC is just the beginning,” says Joe Burrescia, General Manager for ESnet. “ESnet4’s innovative and reliable infrastructure allows scientists from all over the world, and across disciplines, to exchange large datasets and analyses in an efficient way. It is these collaborations, this sharing of information, that allows us to better understand the world around us.”