ESnet, Orange Silicon Valley, and Bay Microsystems Demonstrate the World’s First Long Distance 40Gbps RDMA Data Transfer

SOUTH SAN FRANCISCO, BERKELEY AND FREMONT, Calif. (Oct. 25, 2011) – Researchers at the Department of Energy’s Energy Sciences Network (ESnet), Orange Silicon Valley and Bay Microsystems have collaborated to showcase the world’s first 40 Gbps (gigabits per second) wide area network demonstration of RDMA over InfiniBand. RDMA, or Remote Direct Memory Access, is an alternative model to TCP/IP for bulk data transfer designed to significantly improve the utilization of available bandwidth resources and the processing efficiency of the end hosts – and does so without the need for any special network tuning.

While RDMA over InfiniBand is not a new concept, this experiment marks the first time that it has been demonstrated at 40 Gbps data rates over long-haul distances, in this case approximately 200 miles. The demonstration showed that using RDMA moves data at up to 96 percent of the peak capacity of the network as opposed to other network technologies where utilization can drop significantly over longer distances.

“The volume of science data that needs to be shared among global research sites is reaching levels beyond what current 10 Gbps networks and traditional network protocols can handle,” said Brian Tierney, ESnet Staff Scientist and lead for the Advanced Networking Initiative (ANI). “RDMA communication opens new possibilities for researchers by enabling quick and efficient data transfer for ‘big data’ applications. It does so by eliminating many of the typical network performance bottlenecks and helps users make much more efficient use of the WAN infrastructure – this helps to ensure that scientific progress is unimpeded even as data needs grow.”

The long-haul RDMA demonstration is part of a research project being conducted at Brookhaven National Laboratory (BNL) in New York and leverages ESnet’s ANI 100 Gbps testbed. The testbed is being made available to researchers both in the public and private sector as a first of its kind platform to test different and potentially disruptive approaches to new network protocols and architectures in a greater than 10 Gbps network environment. The testbed is currently available in the Long Island, New York region but will soon have a national footprint to enable experimentation at even greater distances.

“While RDMA over extended distances has been tested and shown to scale up to 10Gbps data rates, this is the first time RDMA over distance has been proven to work at full bandwidth for 40Gbps data rates” said Eric Dube, Senior ProductManager of Systems at Bay Microsystems, Inc. “Gaining access to a 40 Gbps wide area optical circuit is very costly and had prohibited this kind of research in the past. Using the ANI testbed, we are now able to prove these concepts in a live network environment setting the stage for deploying scalable RDMA-enabled applications over 100G networks. This is especially important as more geographically dispersed data centers and science sites will require this type of bandwidth and capability.

RDMA works by enabling network adapters to transfer data to or from application memory, eliminating the overhead of copying data between application memory and the data buffers in the operating system. When an application performs an RDMA operation, the application data is delivered directly to the network eliminating the additional latency the transfer would incur if the data were to be copied into the operating system memory buffers. This dramatically increases the application-messaging rate and increases the utilization of available network bandwidth.

Bay Microsystems provided its IBEx G40 40 Gbps InfiniBand WAN acceleration platform as well as provided input on the network design, bring-up, and overall support during the testing effort. Bay specializes in providing secure, high-performance networking solutions to commercial enterprises, government agencies and service providers. In particular, Bay delivers a portfolio of wide area network acceleration solutions to data centers to maximize the utilization of their computation, storage, and interconnect assets, and increase their security and reliability.

“Applications that combine RDMA with InfiniBand achieve additional benefits including reliable data transfer, sustained high bandwidth utilization, and reduced CPU overhead, all critical items to consider when ‘Big Data’ Cloud computing, high frequency trading, and high performance computing applications are deployed across a WAN infrastructure,” Dube said.

Orange is an international operator for mobile and broadband Internet services and, under the brand Orange Business Services, is one of the world leaders in providing telecommunication services to multinational companies. Orange Silicon Valley participated in the project to explore RDMA’s potential for meeting the growing challenge of the company and its enterprise customers. Orange Silicon Valley is a U.S. subsidiary of France Telecom-Orange. 

Soumik Sinharoy, Project Manager at Orange Silicon Valley, said “Orange believes that the high bandwidth efficiency that comes with RDMA would be beneficial considering the significant increase in traffic anticipated across our enterprise and consumer markets. Orange is also using this opportunity to evaluate how increased efficiency could help us increase the utilization of our existing assets.” 

By using the IBEx G40 40 Gbps InfiniBand WAN acceleration platform and Infinera’s DTN switched DWDM system along with high-performance InfiniBand-connected hosts provided by BNL, researchers were able to extend 4X InfiniBand QDR outside the data center and transport it on a 40 Gbps wavelength over the 370.36 km fiber loop of ESnet’s wide area network circuit from BNL in Upton, NY to Manhattan, NY. The IBEx G40, provides enhanced buffering and credit management allowing all native InfiniBand protocols to be extended over SONET OC-768/SDH STM-256 or ITU-T G.709 OTU3 at 40 Gbps data dates over thousands of miles.

The proof is in the results where application testing produced near theoretical line rate performance across the WAN circuit with RDMA-enabled application communication providing 96 percent link efficiency. These improvements can be attributed to the lossless nature of InfiniBand allowing for reliable transport when used as a wide area network technology and the reduced protocol overhead capabilities of RDMA.

Further demonstrations will be presented at the SC11 conference being held Nov. 12-18 in Seattle. Researchers will build a 40 Gbps transcontinental network circuit enabling RDMA application data transfers over InfiniBand at near line-rate performance.

About Orange

France Telecom-Orange is one of the world’s leading telecommunications operators with 170,000 employees worldwide, including 102,000 employees in France, and sales of 22.6 billion euros in the first semester 2011. Present in 35 countries, the Group had a customer base of 217.3 million customers at 30 June 2011, including 143 million customers under the Orange brand, the Group's single brand for internet, television and mobile services in the majority of countries where the company operates. At 30 June 2011, the Group had 158.3 million mobile customers and 14 million broadband internet (ADSL, fibre) customers worldwide. Orange is one of the main European operators for mobile and broadband internet services and, under the brand Orange Business Services, is one of the world leaders in providing telecommunication services to multinational companies. 

With its industrial project, "conquests 2015", Orange is simultaneously addressing its employees, customers and shareholders, as well as the society in which the company operates, through a concrete set of action plans. These commitments are expressed through a new vision of human resources for employees; through the deployment of a network infrastructure upon which the Group will build its future growth; through the Group's ambition to offer a superior customer experience thanks in particular to improved quality of service; and through the acceleration of international development

France Telecom (NYSE:FTE) is listed on Euronext Paris (compartment A) and on the New York Stock Exchange. For more information (on the internet and on your mobile): www.orange.com, www.orange-business.com, www.orange-innovation.tv. (Orange and any other Orange product or service names included in this material are trademarks of Orange Brand Services Limited, Orange France or France Telecom.)

About Bay Microsystems, Inc.

Bay Microsystems was founded in 2000 to deliver secure high performance networking solutions to commercial enterprises, government agencies, and service providers. Bay delivers a portfolio of wide area network acceleration solutions that transform how global data centers are architected by substantially increasing the performance of the applications they support, maximizing the utilization of their computation, storage, and interconnect assets, and increasing their security and reliability.