ESnet SCinet Volunteers Help Create World’s Most Powerful Temporary Network

ESnet will provide multiple 400 Gbps connections, to sites in Los Angeles, Berkeley, CA, Chicago, and Washington, D.C. (Enlarge image)

In just a few days, the world’s fastest and most powerful volunteer-built network will go live in the Denver Convention Center, home to this year’s International Conference for High Performance Computing, Networking, Storage, and Analysis, or SC. SC23’s temporary network, known as SCinet, will achieve record speeds of up to 6.71 Terabits per second, up from 5.01 Tbps for SC22. This is not about linking up conference booths and running keynotes smoothly, although SCinet will also do that, over roughly 60 miles of fiber-optic infrastructure. The goal of SCinet and its nearly 200 volunteers is to push the boundaries of networking technologies by supporting workshops, demos, and collaborations that showcase the industry’s leading innovations. 

Those volunteers, who come from nine countries and represent 113 institutions, are split into 18 teams. Staff from the Department of Energy’s high-speed, cutting-edge scientific network, Energy Sciences Network (ESnet), have long been closely involved, as have their colleagues from Lawrence Berkeley National Laboratory and the National Energy Research Scientific Computing Center. This year, ESnet's high-level participation with SciNet this year includes Nathan Miller from ESnet’s Optical Network group serving as SCinet’s Executive Director, with Jason Zurawski of Science Engagement (who was SciNet’s 2022 Executive Director and 2018 Chair), SCinet Executive Advisor. Optical Network’s Kate Robinson is SCinet’s Technical Director. Optical Network’s Michael Blodgett is Wide Area Network (WAN) Co-Lead, and Site Solutions’ Cody Rotermund is Deputy WAN Lead. Core Routing’s Dylan Jacob is Co-Chair of the Routing Team; Orchestration and Core Data’s Ryan Vredenburg is a Routing team member.

Every year, SCinet rises from an empty floor, thanks to volunteers racking, stacking, and connecting servers and routers in the Denver Convention Center to the world.

Conventional wisdom has it that “SCinet takes a year to plan, a month to build, a week to operate, and a day to tear down,” and indeed, the Brainstorming and Research phase for SCinet at SC23 began in December 2022. The fifth and final phase — Staging, Setup, and Show — runs for two intense months through the end of the conference, with the busiest stretch starting October 18, when the volunteers began unloading the freight inventory into a cavernous room in the bowels of the convention center and racking and stacking…and racking and stacking servers and routers. They have been working nonstop since then to get everything connected, tested, and ready for the start of SC on November 13. 

ESnet’s Cody Rotermund (second from right), SCinet Deputy WAN Lead, addresses SCinet volunteers during staging, on Oct. 23.

In addition to person-hours, ESnet is one of several circuit and spectrum providers among the 30 companies and organizations that have donated an estimated $43 million worth of hardware, software, and services to SCinet. This year, ESnet has been awarded platinum status in recognition of its hardware and services contributions. 

ESnet will contribute up to 2.8 Tbps of WAN capacity to SC23 and SCinet, via a combination of different technologies and efforts led by Blodgett and Rotermund. With one approach, involving “alien wavelengths,” ESnet is providing spectrum over which SCinet can use their own optical transponders to light up capacity using the ESnet Open Line System (OLS) as the transport network. This was achieved by extending the ESnet OLS network into the Denver Convention Center, giving SCinet the ability to pack more bandwidth over a single fiber and using transponder modems for different vendors. The WAN team is also experimenting with using 400 Gbps ZR+ as a method of transport to external carrier networks between the convention center and the nearest colocation facility where other networks have a point of presence (POP).

In other cases, ESnet is providing connectivity to the ESnet router network through border gateway protocol (BGP) peering, which provides routed connectivity to other networks and DOE labs and user facilities already linked to ESnet’s backbone. This additional network capacity is designed to enable SC’s Network Research Exhibition participants to connect to the necessary high performance computing and research centers for their demos.

Dylan Jacob (far left) is Co-Chair of the SCinet Routing team.

ESnet is also assisting with the Routing team’s configuration automation efforts this year, with Jacob and Vredenburg sharing the organization’s expertise in designing services around the Cisco Network Services Orchestrator (NSO) framework. Meanwhile, Robinson is overseeing SCinet’s IPv6 adoption, helping SCinet to run Dynamic Host Configuration Protocol (DHCP) Option 108, which allows toggling off IPv4 on a client device that is able to communicate using IPv6. This allows for more devices to communicate using IPv6 only versus dual stack (IPv4 and IPv6 together).

“SCinet is a massive undertaking, and it’s so exciting to see these many months of planning and work paying off as everything comes together,” said Miller. “We at ESnet are proud to contribute what we know, and pick up so much from our peers at other user facilities and national research and education networks (NRENs). It's important for ESnet’s mission that we are continually learning about and demonstrating new ways computer networks can remove barriers for research and help accelerate scientific discoveries.”