Superfacility Model Brings COVID Research Into Real Time

For the better part of a year now, finding a way to end the coronavirus pandemic has required a global response and the efforts of thousands of people across scientific, medical, academic, and political entities. Among the collaborators are scientists working at the National Energy Research Scientific Computing Center (NERSC) at Lawrence Berkeley National Laboratory (Berkeley Lab) and the Linac Coherent Light Source (LCLS) at SLAC National Accelerator Laboratory, who are contributing to these goals through research partnerships that are leveraging the superfacility model for real-time data analysis.

For example, researchers from Turkey working at the LCLS used X-ray crystallography to capture detailed images of the structure of the SARS-CoV-2 virus, focusing on two of the viral proteases — enzymes that make the virus’s life cycle possible — and how to keep them from functioning. By understanding the molecular structure of the proteases, researchers can identify proteins that bind to them and interfere with their role in viral reproduction, a key step toward potential treatments for COVID-19. Analysis of the findings from these experiments is ongoing.

“We had been developing large-scale computing projects through the ExaFEL project, and when we went into the COVID-19 shutdown last year, LCLS management decided to invest additional resources into COVID-19 research and we got extra funding from the U.S. Department of Energy to accelerate that and get our users ready to do this work remotely,” said Chuck Yoon, an information systems specialist who leads the LCLS Data Analytics Department and the Advanced Methods for Analysis Group at SLAC.

During and after the experiments, the LCLS data analytics team utilized NERSC’s Cori supercomputer to process data and provide results in real time, allowing the researchers to monitor the experiment, begin analysis, and make changes nimbly as necessary. Like an increasing number of experiments across the scientific spectrum, their work generated large datasets, which the computing resources at NERSC are well prepared to handle. Data from the LCLS experiments traveled to NERSC via the Energy Sciences Network (ESnet), the high-speed computer network connecting U.S. Department of Energy scientists and their collaborators worldwide that is managed by staff at Berkeley Lab.

“We have developed this pipeline where we go from raw data to structure,” Yoon said. “We have the pieces ready, and the goal is to run it efficiently at NERSC and then display the results in real time at SLAC or for anyone with a laptop.”

The LCLS data analytics team utilized NERSC’s Cori supercomputer to process data and provide results in real time, allowing the researchers to monitor the experiment, begin analysis, and make changes nimbly as necessary.

Using NERSC allows for real-time data analysis and enables researchers to make decisions on the fly, added NERSC application performance specialist Johannes Blaschke, who has also been involved with this project. “Previously, you’d have to go home and crunch the numbers after the fact. This way, you can see any bad data while the experiment is still running and find out whether there’s a difference [in results] between the proteins, so you can move on to the next sample and use your precious beam time well.” 

Collaborating with NERSC and ESnet has been invaluable for real-time data processing, added research scientist Aaron Brewster, a project scientist in Berkeley Lab’s Molecular Biophysics and Integrated Bioimaging division who has been working with the LCLS team. “The data rates LCLS can produce can only be handled using advanced supercomputing facilities. They allow us to match the processing rate to the data production rate, letting us study small differences in atomic structure in near real time. This in turn guides decision-making processes during the experiment.”

Efficiency, Flexibility, Collaboration

This is the promise of the superfacility concept: connecting multiple facilities so they can be used by scientists seamlessly as one. As experimental data volumes continue to skyrocket, the power of supercomputers like those at NERSC become increasingly crucial tools that enable scientists to spot patterns that might be difficult to see because of the size and complexity of the datasets. Using the superfacility model allows experimenters to obtain real-time results, a capability that is proving to enhance the likelihood of success at, for example, light sources such as the LCLS.

“This is an emerging need, being able to move data between research sites and computing centers,” said Debbie Bard, who leads NERSC’s Data Science Engagement group. “By building this infrastructure, we’re making it easier for scientists to think about doing science in this way. This particular work with the LCLS is one of many experiments that need our resources. But part of the idea is that every experiment should have this capability.”

As the superfacility approach enables more sophisticated SARS-CoV-2 research, this research also shapes the development of the superfacility concept, exposing challenges and necessary improvements. NERSC developers see two particular areas of growth for the future: automation and resilience. A move toward automation in supercomputing and for the superfacility is a step forward for the goals of efficiency and collaboration. Automating job submission and data transfer allows scientists to use their time doing science instead of administrative work and cuts out the human middleman.

“The goal is for machines to talk to machines, with no humans in the middle to slow things down,” said Bard. “We’re not there yet, but we’re working toward submitting jobs and moving data around in an automated way.” The result would be easier and more seamless access and computing both for NERSC users and for NERSC developers. The collaboration with LCLS on SARS-CoV-2 proteases is just one of many that’s helping to identify parts of the process that need automating. 

In addition, like other world-class research facilities, NERSC has to occasionally take its systems offline for maintenance and to perform upgrades. In the superfacility model, this can have an impact beyond the NERSC facility, so NERSC is working to keep resources available during these events — for example, keeping some services and access to data up when the main compute systems are down.

Collaboration between LCLS and NERSC on COVID is forward-facing in more ways than one as we anticipate the eventual end of the pandemic and the future role and process of high performance computing in scientific research. Though it’s just one part of the research ecosystem, Bard noted, “this work is urgent in two senses: we need to analyze the data coming from the experiment urgently, and the science of COVID research is very urgent.”

The installation of the next-generation Perlmutter supercomputer at NERSC this year will further enhance these efforts, Yoon added.

“Looking into the future, we have developed algorithms and artificial intelligence to use the GPUs when Perlmutter comes online, and we have put a lot of work into automating the crystallography analysis,” he said. “The idea is to get humans out of the loop so that the machines can efficiently process the data and provide near-real-time feedback to help us optimize our experiments.”

NERSC, SLAC, and ESnet are U.S. Department of Energy Office of Science user facilities.

Research was supported by the Exascale Computing Project (17-SC-20-SC), a collaborative effort of the U.S. Department of Energy Office of Science and the National Nuclear Security Administration.

Use of the Linac Coherent Light Source (LCLS), SLAC National Accelerator Laboratory, is supported by the U.S. Department of Energy, Office of Science, Office of Basic Energy Sciences under Contract No. DE-AC02-76SF00515.

Research was supported by the DOE Office of Science through the National Virtual Biotechnology Laboratory, a consortium of DOE national laboratories focused on response to COVID-19, with funding provided by the Coronavirus CARES Act.