ESnet Director Bill Johnston Lays Out his Vision of ESnet’s New Directions

ESnet Director Bill Johnston lays out his vision of ESnet’s new directions in a two-pager, explaining that meeting DOE’s science needs “ entails not only increasing end-to-end bandwidth, but becoming actively involved with other domestic and international research and education (R&E) networks in developing and deploying cutting edge operational technologies. Such collaboration is necessary to produce a seamless interoperable infrastructure that will allow the advancement of DOE’s large scale science.”

Here is the full text:

ESnet Progress and New Directions

William E. Johnston (wej@es.net), ESnet Manager, Lawrence Berkeley National Laboratory

Summary: ESnet is driven by the requirements of the science Program Offices in DOE’s Office of Science (OSC). To that end, ESnet provides a high-bandwidth network connecting forty-two DOE sites to each other and to collaborators worldwide. Each year, the Office of Science facilities are used by more than 18,000 researchers from universities, other government agencies, and private industry. As DOE’s large scale science continues to move to a distributed international model, ESnet is providing the innovation and expertise to meet its networking needs.  This entails not only increasing end-to-end bandwidth, but becoming actively involved with other domestic and international research and education (R&E) networks in developing and deploying cutting edge operational technologies. Such collaboration is necessary to produce a seamless interoperable infrastructure that will allow the advancement of DOE’s large scale science. 

Overview

ESnet is evolving to meet the needs of DOE science as identified in an August 2002, DOE Office of Science-sponsored workshop. This evolution is tracking the results of a follow-on workshop, held in June 2003[1], which constructed a roadmap describing the network and related middleware services necessary to meet the ambitious networking needs of DOE’s large-scale science.

One of the most important findings of these workshops was to note the absolute importance of delivering reliable, high speed, end-to-end bandwidth. In the case of large scale science, this can mean one end of the connection is at a large experimental facility, such as the National Synchrotron Light Source, and the other end might be at a computing cluster located at a researcher’s home facility, which may be located at a US or international university.

This report describes how ESnet innovation is ensuring that end-to-end (application to application) communications challenges are met, and how the related services are being kept interoperable with SC’s domestic and international collaborators.

The Network Needed for Next-Generation DOE Science

DOE science programs that have currently defined requirements for high bandwidth and network quality-of-service include High Energy Physics, Climate (data and computations), NanoScience at the Spallation Neutron Source, Fusion Energy, Astrophysics, and Genomics (data and computations), the SC supercomputer centers, and several others.

The science applications must move massive amounts of data in a predictable way--network bandwidth, reliability and quality of service are critical. Given the global scope of these large science projects, it is the norm that the data needed to be shared will cross multiple networks and administrative domains, and so ESnet is designed to facilitate this.

Additionally, achieving end-to-end high bandwidth for distributed applications, on-line instruments, etc., requires extensive monitoring and diagnosis in the network in order to provide feedback for both debugging and operation.

ESnet Strategic Directions

In order to address the network needs for the next generation of DOE science, ESnet is taking a number of steps.

A New Architecture

A new ESnet architecture and a new implementation strategy have been developed, and the next-generation network is being incrementally deployed to increase the bandwidth, services, reliability and cost effectiveness of the network.

The elements of the architecture include multiple, independent, optical channel-based, national cores (each at 10-40 Gb/s) that independently connect to Metropolitan Area Network (MAN) rings, together with independent paths to the R&E networks of Europe and Japan by connecting to all of the available peering points.

The MAN rings provide redundant paths and on-demand high bandwidth point-to-point circuits for DOE Labs. The multiple cores connect to the MAN rings in different locations to ensure that the failure of a core node could not isolate the MAN. This is illustrated in the figure using the current ESnet IP core and the Science Data Network (SDN), built from National Lambda Rail (NLR) fiber as a second core. The first two segments of the second core – 10 Gb/s circuits from San Diego to Sunnyvale, CA to Seattle have been put into service.

Another aspect of the new architecture is high-speed peering with the US university community via the Internet2/Abilene network. US universities are an important component of DOE science and require state-of -the-art access to the DOE laboratories served by ESnet. The new architecture provides seamless, high-speed access between the university community and the DOE labs via multiple high-speed interconnects.

Implementation Strategy

The implementation involves building the network by taking advantage of the evolution of the telecom milieu – that is, using non-traditional sources of fiber, collaborations with existing R&E network confederations for lower cost transport, and vendor-neutral interconnect points for more easily achieving financial competition for the “last mile” tail circuits to ESnet sites.

Replacing the current point-to-point tail circuits with MAN optical rings is providing high-speed, high-quality production IP service, at least one backup path from DOE labs to ESnet hubs, scalable bandwidth options from sites to the ESnet core, and point-to-point provisioned high-speed circuits as an ESnet service. The newly completed SF Bay Area MAN (BAMAN) is the first ESnet MAN.

Involvement with the Networking R&D Community

A clear mandate from the Roadmap Workshop was that ESnet should be more closely involved with the network R&D community, both to assist that community and to more rapidly transition new technology into ESnet. To facilitate this, the new implementation strategy includes multiple interconnection points with NLR and UltraScienceNet – DOE’s network R&D testbed.

ESnet has been very active in collaborating with the R&D community, the European R&E network, GEANT, and the domestic R&E network, Internet2, in the areas of applied network research that are directly applicable to creating the seamless end-to-end paradigm required for science. Specifically, the OSCARS[2] project that dynamically creates end-to-end private virtual networks is a collaboration in order to ensure an interoperable, inter-domain approach that will allow scientists to run the specialized protocols needed to move vast quantities of data between the various networks. ESnet’s collaboration with the R&D community on perfSONAR[3], an inter-domain monitoring framework, is assisting in assuring that the end-to-end paths are functioning correctly.

ESnet is also sharing its expertise by participating in the DHS sponsored Secure Routing Workshops, whose purpose is to secure the fundamental reachability protocols on which the entire Internet is dependent.