ESnet Connects STAR to Asian Collaborators

The Department of Energy's Energy Sciences Network (ESnet), the Korea Research Environment Open Network2 and the Global Ring Network for Advanced Applications Development achieved a sustained data transfer rate of 1 gigabit per second (Gbps), equivalent to transporting 120 minutes of video per second, between the Brookhaven National Laboratory in New York and the Korea Institute of Science and Technology Information in Daejeon, South Korea.

This capability will make 20 percent more data available to the STAR nuclear physics community per year, and close a digital divide between U.S. and Asian physicists.

“This stable, high-speed data transfer capability will have a great impact on the international nuclear physics community. It will enhance our ability to redistribute data to our Asian colleagues — approximately 27 percent of our collaborators are located in China, South Korea and India,” says Jerome Lauret, the software and computing project leader of the Solenoidal Tracker at RHIC (STAR) experiment based at Department of Energy’s Brookhaven National Laboratory.

For years, physicists have searched for the fundamental building blocks of matter by smashing together speeding gold ions inside the Relativistic Heavy Ion Collider (RHIC) at BNL, and tracking the basic particles that burst out with the STAR detector located on the south side of the RHIC. Analyzing the results of these collisions helps researchers improve our understanding of basic particles, fields and forces.

The STAR detectors collect a tremendous amount of data, approximately 1,000 collision events per second, which then have to be mined and processed before they can be analyzed. This means that physicists may have to wait several months to a year after an experiment before they can use the information. The mined data is stored at BNL, so scientists in Asia often have to wait several weeks longer for the massive datasets to make their way across networks spanning the United States and the Pacific Ocean. Lauret believes that this delay is a disadvantage to physicists in Asia, and causes a digital divide among researchers on different continents.

“I see a lot of physics potential coming from our Asian colleagues, but the digital divide and the far distances do not provide equal scientific opportunities to our colleagues across the world. With the 1 Gbps sustained transfer rates and bringing data closer to our scientists, I suspect we will see a lot more papers coming out that reference data taken one or two years after the experiment,” says Lauret.

According to Lauret, the new steady 1 Gbps connection speeds up the distribution of research-ready STAR data in two ways. It would allow the team to export a portion of the raw STAR data to the Korea Institute of Science and Technology Information (KISTI) for mining, while BNL proceeds with other data-mining activities. The additional data-mining at KISTI will make 20 percent more STAR data available to the research community per year. As KISTI mines the data, it will also be storing the results. This means that collaborators across China, Korea and India will not have to wait for massive STAR datasets to transfer from New York. Now the information is much closer in Asia and will flow much faster.

According to Eli Dart, an ESnet network engineer, network monitoring and test tools played an integral role in optimizing the 1Gbps bandwidth performance between BNL and KISTI by providing engineers with the information necessary to find and fix network bottlenecks, as well as to properly tune the high-throughput data transfer servers. ESnet maintains a repository of network and data transfer tuning tips at http://fasterdata.es.net/.

“Several different configurations were tested, and while some were adequate – initial testing achieved 600Mbps – the final production configuration is stable at 1Gbps,” says Dart. “Enabling science and enhancing scientific productivity are key components of ESnet’s core mission. Working with the STAR collaboration and our partner networks to establish a stable, high-performance data transfer pipeline from BNL to KISTI has been very productive — the STAR collaboration has a new capability at its disposal, and the scientists will benefit greatly from enhanced access to vital research data.”

ESnet provides reliable, high-bandwidth connections linking more than 40 DOE research sites and National laboratories across the United States, and connects scientists and facilities to thousands of collaborators around the world via connections to other research networks. Science data traffic from BNL arrives at KISTI by traversing three different networks. From BNL, ESnet carries data to Seattle. The traffic is then forwarded to Korea Research Environment Open Network2 (KREONet2), which carries the traffic across the Pacific to South Korea on a circuit funded by Global Ring Network for Advanced Applications Development (GLORIAD). Funded by the National Science Foundation, the GLORIAD program cooperatively funds links from the United States to KISTI in South Korea, and the China Academy of Sciences in mainland China.

"We are pleased to see such science benefit from the DOE-KISTI partnership in utilizing the GLORIAD-KREONet infrastructure cooperatively established with our innovative Korean partners at KISTI," says Greg Cole, Principal Investigator for the US GLORIAD program. "This project suggests the need for sustained investment in cyberinfrastructure supporting needs of a growing global science community with vastly growing demands on network infrastructure."

“Research networks are a crucial infrastructure for global scientific collaborations. We have been helping this effort by collaborating with advanced NRENs (National Research and Education Networks) like ESnet to provide the performance necessary for their large-scale experiments. The STAR high-performance data transfer is certainly a very good example of how networks will help push the frontiers of science,” said Hyeak-Ro Lee, a leader of the Department of Computing and Networking Resources at the KISTI Supercomputing Center.

In a Memorandum of Understanding (MOU) established in January 2009 between KISTI and DOE’s Lawrence Berkeley National Laboratory, which manages ESnet, both organizations agreed to collaborate to optimize operations and performance of the GLORIAD/KREONnet network facilities.

About the DOE Office of Science
With a Fiscal Year 2009 budget of $4.8 billion, the DOE Office of Science is the largest funder of basic research in the physical sciences in the United States. The steward of ten National Laboratories, the Office of Science funds research at National Laboratories and over 300 universities nationwide through programs in advanced scientific computing research, basic energy sciences, biological and environmental research, fusion energy sciences, high energy physics, and nuclear physics.