ESnet’s Networking Prowess on Display at Singapore Conference

At the Supercomputing Asia 2019 conference (SCA19), held March 11-14 in Singapore, ESnet’s expertise in network design and operation took center stage as ESnet Director Inder Monga delivered a keynote address and the winner of the Data Mover Challenge (DMC) used an approach honed on ESnet's network.

In his presentation "Cracking open the network ‘black-box’," Monga discussed an emerging trend of integrating application workflows with compute, storage, and network resources in unique ways with experiences from ESnet, the science network super-highway supporting big-data research in the U.S. In his talk, Monga noted that data acquisition in its raw form and its transformation through data analysis into insights requires a network infrastructure that makes the collection and movement of data from instrument, to data center, to high-performance computing or the cloud a seamless experience.

The conference’s Data Mover Challenge featured seven teams, including Argonne National Laboratory and Fermilab, as well as Zettar, a Palo Alto startup that has developed multi-100+Gbps capable hyperscale data distribution software and been awarded grants by both the National Science Foundation and the Department of Energy (DOE)'s Office of Science. The DMC was held to showcase challengers by having them compete in deploying the best software tools on data transfer nodes (DTNs) that are set up within current 100Gbps international Research and Education Networks across the globe, according to the conference website.

Chin Fang, founder and CEO of Zettar, began collaborating with the DOE’s SLAC National Accelerator Laboratory in 2015 to address the challenge of supporting the data transfer demands of the Linac Coherent Light Source II (LCLS II), which is expected to come online in 2020. The LCLS is the world’s first hard X-ray free-electron laser, and its strobe-like pulses are just a few millionths of a billionth of a second long. It is also a billion times brighter than previous X-ray sources. LCLS II will provide a major jump in capability – moving from 120 pulses per second to 1 million pulses per second. The increased capability is expected to generate data transfers of multiple terabits per second as the experimental results are sent from SLAC to DOE’s supercomputing facilities.

Last fall, using a 5,000-mile loop of ESnet's fiber and ESnet's OSCARS (On-Demand Secure Circuits and Advance Reservation System) to reserve the bandwidth for the run, the SLAC-Zettar collaboration set a record by transferring 1 petabyte in 29 hours, with encryption and checksumming, beating the previous record by 5 hours, an almost 15 percent improvement.

Fang decided to enter a Zettar team in the SCA19 challenge at the encouragement of Monga and Amedeo Perazzo, Director of the Control and Data Systems Division for the LCLS at SLAC. Over seven consecutive weeks starting in January, each team had five days to achieve the maximum data transfer rate. When the awards were announced on March 13, the Zettar team was named overall winner for achieving "an amazing 68 Gbps transfer rate between Chicago and NSCC (Singapore)," the conference press release noted.

Fang said Zettar’s victory was made possible by "the rich experience of running our software over ESnet and using OSCARS" for the 2018 record-setting data transfer and many other pre-production trials. "If we had the chance to apply our best practices, I am sure nearly full bandwidth utilization is quite within reach indeed. Regardless, we are very pleased by the outcome," he added.

In his presentation after Zettar was named the DMC winner, Fang fielded questions from experts, including Fermilab's Dr. Wenji Wu, lead developer of the DOE-funded BigData Express, Argonne's Dr. Rajkumar Kettimuthu, technical lead of Globus, one of the creators of GridFTP, and Vincenzo Capone, head of Research Engagement & Support for GÉANT, the European research network, who wanted to know if Zettar's approach could be exported to other networks citing a planned 100 Gbps network around the globe.

In looking back at the conference, Fang said, "I was impressed by the lasting and deep influence of ESnet on the world's research data movements. ESnet has pioneered several important best practices, reference designs, and the Fasterdata knowledge base. All have been faithfully followed and studied in the U.S. by almost all research organizations, large and small, and around the world too, by international organizations such as the Asia Pacific Research Platform."

Some significant ESnet contributions cited by Fang include:

• The Science DMZ infrastructure architecture that helps research organizations support both science and enterprise workflows in a secure and high-performance manner
• DTN reference design with properly tuned DTNs at both ends of a data path dramatically improve the transfer rate.
• The fasterdata.es.net knowledge base originally developed in 2007 and now accessed by tens of thousands of visitors from around the world each year.

ESnet contributors Eli Dart, Eric Pouyoul and Brian Tierney were cited by Fang as some of the key people behind these accomplishments.

“Although networking has long been essential to HPC and science collaborations, It’s gratifying to see it’s profile raised even higher through competitions like the Data Mover Challenge,” Monga said. “Emphasizing data movement is an important factor as we strive toward a seamless research ecosystem of experiment facilities, computing resources and integrated network connectivity.”