Using Software to Better Define Network Functionality

Tracing its roots back to the days of dial-up modems in the 1970s, ESnet has consistently worked to make the network faster and more robust. Typically this has been done by taking advantage of new data transfer technologies and upgraded hardware.

But with the emergence of software-defined networking over the past decade, ESnet is pioneering new possibilities for automatable networking in real time. Most recently, ESnet supported a demonstration by researchers at the SLAC National Accelerator Laboratory in which one petabyte of data was transferred 5,000 miles across the United States and back in a record-setting 29 hours.

An early start
In early 1998, ESnet staff successfully marked selected Internet traffic for priority service over unmarked, lower-priority traffic between Lawrence Berkeley and Argonne national laboratories in a demonstration to show capabilities for production-mode scientific research. In a test sharing an intentionally congested path, the priority-marked stream moved at eight frames per second, while the standard version transmitted just one frame per second.

The experiment helped pave the way for more reliable and constant connectivity via priority bandwidth on the Internet and was also a first step toward the development of ESnet’s OSCARS, the On-demand Secure Circuits and Reservation System. What makes OSCARS so useful is that it can automatically create end-to-end circuits, crossing multiple network domains.

Before OSCARS, this was very time-consuming – in 2010, for example, ESnet engineers needed 10 hours of phone calls and about 100 emails over three months to do what one person can do in five minutes and one email using OSCARS. OSCARS was recognized for its technological importance with a 2012 R&D100 Award.

Setting the stage for SDN
OSCARS was a precursor to the subsequent development of software-defined networking, or SDN. Generally speaking, SDN is an emerging technology paradigm aimed at making it easier for software applications to automatically configure and control the various layers of the network to improve flexibility, predictability and reliability. As an example, SDN would give network administrators a measure of predictability by giving them more control over their dataflows. Without this ability, the larger science data flows would compete with other data transfers for bandwidth making it difficult for scientists to get critical data when it is needed, especially if the data is stored across multiple sites, which is increasingly common.

One of the critical challenges is developing an approach that is compatible across different networks and equipment produced by different vendors. Since 2012, ESnet staff have pushed the boundaries of SDN, working with vendors, other R&E networks and the researchers who rely on those networks, and have shared their work with the wider networking community. Here is an overview of some of those projects:

In late November of that year, ESnet staff at Berkeley Lab joined Infinera, a manufacturer of high capacity optical transmission equipment, to demonstrate the world’s first prototype of an SDN Open Transport Switch capable of dynamically controlling bandwidth services at the optical layer. The proof-of-concept demo was conducted on a testbed network ring in New York, connecting Brookhaven National Laboratory with a network hub in Manhattan. The demonstration marked a new approach where an open architecture with SDN was used to provide traffic-engineered paths at the optical layer and was accomplished through extensions to the OpenFlow protocol.

The following year, the Open Networking Foundation (ONF), a non-profit organization dedicated to promoting SDN, appointed Monga as one of nine new industry thought leaders named as a Research Associate for the coming year.

In November 2013, ESnet, in partnership with Brocade and Infinera, demonstrated how SDN can be used to provision services and automatically optimize network resources across a multi-layer network as traffic and service demands change. The demonstration leveraged ESnet’s OSCARS application along with the SDN-favored OpenFlow protocol.

Monga served as the lead organizer for a joint DOE-NSF-NITRD sponsored two-day workshop at the National Science foundation to plan a path forward to develop, deploy and operate a prototype SDN network. The December 2013 workshop was by invitation only and drew about 100 networking experts from academia, industry, national labs and federal agencies.

To support SDN-related research, ESnet deployed a 100 Gbps SDN testbed to provide network researchers with a realistic environment for 100G application/middleware experiments. To support testbed research, ESnet developed the ESnet Network Operating System, or ENOS, a prototype next-generation architecture for handling data-intensive science workflows. In February 2015, the Corporation for Education Network Initiatives in California (CENIC) awarded ESnet the 2015 Innovations in Networking Award for High-Performance Research Applications for the 100G SDN Testbed.

Three months later, ESnet was part of a coalition including the Open Networking Operating System (ONOS) Project, the Open Networking Foundation, Corsa Technology, AARNet and CSIRO in the successful deployment of an ONOS-based software-defined peering router. The router located at AARNet/CSIRO in Australia exchanged routes with the Vandervecken SDN controller stack at ESnet in California. The trans-Pacific deployment validated the benefits of open source SDN principles to flexibly deliver agile applications at a fraction of the cost of traditional proprietary networking solutions.

Using Faucet to improve the flow
ESnet and Berkeley Lab hosted a March 2017 meeting at which seven vendors of network equipment came together in a real-time demonstration to successfully test the interoperability of Faucet, an open-source SDN controller. The March 30 event was sponsored by Google, LBLnet (Lawrence Berkeley National Laboratory’s internal network) and ESnet. Originally developed in New Zealand by REANNZ and the University of Waikato, Faucet is a compact open source OpenFlow controller, which enables network operators to run their networks the same way they do server clusters. Faucet moves network control functions, like routing protocols, neighbor discovery and switching algorithms, to vendor-independent, server-based software.

At the SC18 conference in Dallas, ESnet collaborated with the University of Waikato and the Faucet Foundation on a very large, never-before-done deployment of the Faucet controller, which was created to bring the benefits of SDN to a typical enterprise network and has been deployed in various settings. Allied Telesis, Cisco, NoviFlow and CyberReboot also contributed to the effort to build a multi-terabit Faucet network which was used as part of the SCinet production network supporting over 10,000 attendees from all over the globe.

Also at SC18, ESnet collaborated with two universities and three national labs to demonstrate ESnet’s Software-defined network for End-to-end Networked Science at Exascale (SENSE) research project. SENSE is building smart network services to accelerate scientific discovery in the era of big data driven by exascale computing, cloud computing, machine learning and AI. The SENSE SC18 demonstration showcased a comprehensive approach to request and provision end-to-end network services across domains that combines deployment of infrastructure across multiple labs/campuses, SC booths and WAN with a focus on usability, performance and resilience.

A SENSE testbed consisting of network and end-system resources was deployed across DOE laboratories, including Fermilab and Argonne National Lab and the National Energy Research Scientific Computing Center at Berkeley Lab; facilities at Caltech and the University of Maryland; MAX (the Mid-Atlantic Crossroads) and ESnet. To control network resources, the SENSE system interacts with production provisioning system of ESnet and other regional and site networks. To control end-systems, SENSE software is deployed at the end-sites.

Leading up to SC18, in September 2018, researchers at the Department of Energy's SLAC National Accelerator Laboratory (SLAC) and Zettar Inc. used a 5,000-mile network loop operated by ESnet to transfer 1 petabyte of data in 29 hours, with encryption and checksumming, beating the team's own 2017 record by 5 hours, an almost 15 percent improvement. The demonstration used OSCARS to secure the needed network capacity.

The project is aimed at achieving the high data transfer rates needed to accommodate the amount of data to be generated by the Linac Coherent Light Source II (LCLS II), which is expected to come online in 2020. LCLS II will provide a major jump in capability over the current LCLS – moving from 120 pulses per second to 1 million pulses per second. Scientists use LCLS to take crisp pictures of atomic motions, watch chemical reactions unfold, probe the properties of materials and explore fundamental processes in living things. The increased capability is expected to generate data transfers of multiple terabits per second-- as the experimental results are sent from SLAC to Department of Energy supercomputing facilities for analysis.