ESnet, Infinera Successfully Test Innovative Approach for Guiding Large Data Flows

Since its launch about 30 years ago, the Internet has changed nearly every aspect of society, from how we communicate to how we conduct business and, in the case of Department of Energy national laboratories, to how we carry out scientific research. At the same time, this transformation has also pushed the physical limits of the networking infrastructure, from the optical fibers that carry the data to the protocols that ensure that data is delivered to the right destination. 

As the network serving tens of thousands of DOE researchers at national labs and universities, ESnet has seen the amount of data carried on the network double every 18 months. Science is increasingly data-intensive and as the network connecting researchers with DOE supercomputing centers and experimental facilities, ESnet plays a critical role in ensuring the DOE scientific community is able to access and share massive datasets for their research. However, this exponential growth rate over the last 20 years puts a tremendous challenge on ESnet staff to keep investigating and pushing for innovative approaches to scale the network economically while simultaneously running a production service with the current generation of network technologies.

One such innovative approach is Software Defined Networking, or SDN, an emerging field that makes it easier for software applications to automatically configure and control the various layers of the network. ESnet has been conducting localized tests to innovate, experiment, and demonstrate the value of SDN when applied towards end-to-end support of data-intensive science collaborations and applications.

If successful, SDN would give users a measure of predictiablity by giving them more control over their dataflows. Without this ability, the larger science data flows would compete with other data transfers for bandwidth making it difficult for scientists to get critical data when it is needed, especially if the data is stored across multiple sites, which is increasingly common. 

In late November, ESnet staff at Lawrence Berkeley National Laboratory joined Infinera, a manufacturer of high capacity optical transmission equipment, to demonstrate a prototype SDN Open Transport Switch (OTS) capable of dynamically controlling bandwidth services at the optical layer. The proof-of-concept demo was conducted on a testbed network ring in New York, connecting Brookhaven National Laboratory with a network hub in Manhattan.

The demonstration marked the first time an open architecture with SDN was used to provide traffic-engineered paths at the optical layer and was accomplished through extensions to the OpenFlow protocol. Networks are built with several standardized layers of functions, with each layer providing services to support the needs of the layer above it and being supported by the layer below. The optical transport layer, layered on top of the physical fiber optic cable, is the bottom bit transport layer upon which packet-based data transfer functions are built. 

Above the optical layer, each data packet has an address, which is read individually and sent on to its destination by packet-forwarding devices called switches or routers. Using current technologies, the optical layer, which is circuit-oriented, is statically planned and is difficult to change dynamically. But with the growth in the size of datasets, the ability to virtualize and dynamically provision bandwidth at the optical layer is an intriguing challenge. If successful, the amount of bandwidth between any two network nodes could be automatically increased without adversely impacting the packet layers. There has been a lot of research and protocol development done in this area, but no mechanisms have been deployed or used widely by the telecom carriers today.

According to ESnet Chief Technologist Inder Monga, the problem of moving lots of data at high speed between two locations can be analogous to a typical shipping problem. The problem is equivalent to a company needing to ship 200 containers to a single destination. The analogy for the packet approach would be to use one truck per container to deliver the entire shipment. Each truck has a limited load capacity, but has the flexibility to change its route at any time. However, due to traffic conditions, the arrival of the trucks may be unpredictable. This is similar to how packets are routed through the Internet today using best effort delivery. Alternatively, the 200 containers can be transported using a rail system. The rail system has the benefit of being optimized for bulk transport, has a deterministic delivery schedule (due to fixed routes), but typically requires a fair amount of time and effort to plan as well as recover from unforeseen issues.

Creating a network dynamic optical bypass layer to express data between two points is similar to setting up a train route to move cargo between two locations. The fundamental difference in the analogy of the truck and rail transport to networking is that in the network world, both the truck and rail use the same right of way. Using a dynamic optical bypass is a great way of moving large datasets, but it has to be done in such a way that is does not disrupt other network traffic, Monga says. The idea isn’t exactly new – it’s been under consideration for eight to 10 years, but hasn’t yet caught on with industry due to the complexity of the previous approaches. 

One method of controlling and manipulating data flows at the hardware layer is using OpenFlow, an interface that allows the software to reason about flow of packets and build an optical path to carry the entire flow like containers on train. This allows for more sophisticated multi-layer traffic management and is easily controlled by using the SDN approach.

For the November demonstration, Infinera extended the OpenFlow protocol to the optical layer by developing an early version of the Open Transport Switch architecture. The demo leveraged the capabilities of OSCARS, ESnet’s On-Demand Secure Circuits and Advance Reservation System (OSCARS), a centralized traffic-engineering application with multi-domain scheduling and reservation capabilities that can build paths to route flows through the network and is aligned with the SDN/OpenFlow architecture.

The goal of providing bandwidth on demand at the optical layer as a production service is a challenge Monga describes as “very hard, but desirable.”

“If this provisioning of bandwidth can be automated so the route is configured by software, the network will be better able to deal with projected data-intensive science’s data movement demands,” Monga said.

While successful, the demonstration is a first step towards the larger milestone and required a few months of planning by ESnet staff members Chin Guok and Michael O’Connor to deploy the SDN testbed, and Andy Lake who implemented optical extensions and OpenFlow support within OSCARS . Critical support also came from Brookhaven’s Scott Bradley, Dantong Yu and Tan Li. 

 Infinera’s strategy leaders, Chris Liou and Ping Pan drove the OpenFlow use-case and implementation architecture with support from their engineering team including Sharfuddin Syed and Abhinava Sadasivarao, who helped with architecting and implementing the OTS concepts within the on-switch software and management platform.