Meet Scott Richmond, ESnet's Head of Software Orchestration and Core Data

Scott Richmond has been at the Department of Energy's Energy Sciences Network (ESnet) for 11 years, working as both a network engineer and a software engineer in those times. In November 2021, he took over as the group lead for ESnet's new Software Orchestration and Core Data group, a team of software engineers who create software solutions designed to simplify the deployment, management, and ongoing operations of the scientific network. They build software that the Network Operations Center and network engineers use at ESnet and facilities.

When Richmond first joined ESnet's team in April 2011 as a student assistant working on the ANI 100G Testbed, it was the beginning of an important career shift for him away from retail.

His interest in computers wasn't new – Richmond's initial post-college plans involved a lucrative career as a software engineer. But when he graduated in 2000, the dot-com bubble burst and jobs in that field evaporated. He transitioned into a nearly decade-long career as a regional manager of a chain of tabletop gaming retail stores in the San Francisco Bay Area.

But in 2009, Richmond enrolled in a local community college to get certified in network engineering. That's when he found out about ESnet.

The path to software orchestration

Not long after coming to ESnet, Richmond was hired on as a network engineer, a role he has held for most of his time with the organization. But he still hadn't lost the passion for software that was sparked during his first stint in college back in the late 90s, and he wanted to use that passion to make a difference at ESnet.

Having joined ESnet around the end of the deployment of ESnet5, the previous iteration of the network, he was intimately aware of how tedious and complex network engineering tasks could be, especially as a new hire.

"I remember doing the optical network deployment there and having to ship parts everywhere and having to go in and figure out the optical systems ... It was extremely laborious", said Richmond. "What I really wanted to do was build tools for my network engineer counterparts that would simplify operations."

So Richmond began to lend his passion for software to the task of simplifying those processes. After years of hard work, Richmond and the software engineering team have made major improvements.

"Having a system that now, ultimately, I can just push a button and it will define what I need? I think that's fantastic."

Turning institutional knowledge into automation

At ESnet, engineers are unconsciously competent. The expertise runs so deep that accomplishing tasks is nearly muscle memory, like getting into your car and auto-piloting your way home.

When ESnet comprised only 30-40 engineers, it was easy to have one person be an expert in an area and to have all that expertise live in their head rather than in detailed documentation. This makes automating software a challenge because you need to know, step by step, exactly how a human would handle the task before you can code it up in a piece of software. Right now, ESnet is going through a process of examining what it is that engineers do without thinking.

"When you get a call at two in the morning, you need to be able to quickly and easily find out what you have to do to troubleshoot that link. Being an on-call engineer, it was always a pain for me to go and search through documents and spreadsheets and figure out 'ok, where is this circuit attached to'? It was difficult unless you knew exactly where to look," said Richmond.

Working with a small team of software engineers, he addressed this challenge by creating ESnet's database (ESDB), a source of truth for all the physical and logical attributes of the network. The goal of ESDB is to reduce the amount of time and brain power required for engineers to comb through and find the information they need.

"Let's make this easy," he said. "Let's figure out a way to make it so that someone knows how to get their data with a snap of the fingers, right then and there."

Building a bridge between two groups

Scott's background as a network engineer allows him to understand the needs of network engineers on a deep, personal level. This has helped him to soothe anxieties that come along with redefining an existing workflow to include software automation.

"We're asking for a bit of faith on their part," said Richmond. "From a network engineering standpoint, you want to understand what the software's doing, because it's affecting configs that they're later going to have to troubleshoot."

Richmond's goal is to help network engineers feel empowered to dig their fingers into the code themselves. "There's no hard line here, right? We absolutely want network engineering to even code, make pull requests, and we'll teach them how to do it as software engineers."

Richmond is passionate about spreading knowledge between the worlds of network and software engineering. He's constantly trying to teach each side about how the other one works.

"As a regional manager, I basically taught and trained and worked with people. And so I've kind of brought that forward with me into ESnet." said Richmond. "I personally enjoy teaching. My kids call me Professor Dad."

With the upcoming unveiling of ESnet6, the facility's engineers are busy. But Scott hopes that once things slow down, they'll be able to devote more time to creating that bridge.

"Our doors are open. If you'd like to learn, please let us know and we'll walk you through what we're doing. And you're more than welcome to do it!"