Software & Tools Development
Over the years, ESnet has developed several software tools which may be useful to other organizations. ESnet benefits greatly from many excellent open-source tools and strives to contribute back to the community whenever possible.
ESxSNMP is a system to efficiently collect and store large quantities of SNMP data. It exposes the collected data as a RESTful service allowing easy integration with other tools. Currently ESxSNMP is able to integrate with perfSONAR and Graphite. For more information on ESxSNMP, check out this video and presentation. For storing most data, ESxSNMP uses TSDB which is a simple database optimized for time series data.
esmond is a system for collecting, storing, visualizing and analyzing large sets of timeseries data. It was driven by the needs of the ESnet engineering team but is likely useful to a much wider audience. esmond has a RESTful API which allows easy access to the data which is collected. The original focus was on collecting SNMP timeseries data which is still the system's forte, but there is support for generalized timeseries data. The perfSONAR project has begun using esmond to store timeseries of network measurements. The code is available as open-source on github and the documentation has a lot more details.
The MyESnet portal visually showcases the sites ESnet serves, live traffic traversing the network, and other detailed information about ESnet's network and its services. The portal is made of many component technologies, some of them are open-sourced to the public and described below.
A set of modular charting components used for building flexible interactive charts. It was built for React from the ground up, specifically to visualize timeseries data and network traffic data in particular.
A set of React based mapping components which are used within the ESnet Portal, but are not tied to ESnet, or even to network visualization.
A library build on top of immutable.js to provide basic timeseries functionality within ESnet tools.
Graphite is a system for visualizing time series data which was developed by Orbitz as part of their in-house server monitoring tools. ESnet has modified it to work with ESxSNMP and is working integrating it with Net Almanac. Whenever possible, ESnet contributes code back to the Graphite main line.
Net Almanac provides a place to store and retrieve metadata that can help in the interpretation of network events. Often it is difficult to know what events might have caused by or been related to a network event. Metadata about things such as outages, conferences, and demonstrations can be very valuable when viewing graphs of network traffic months in the past (or even in the present). Net Almanac is a collective memory for events this kind of metadata. ESnet is working to integrate Net Almanac with Graphite to allow easy viewing of the metadata right alongside graphs of utilization or performance data.
Iperf is a modern tool for measuring maximum TCP and UDP bandwidth performance. It aims to be easy to use while still being powerful and flexible. It is able to measure achievable bandwidth, delay jitter, and datagram loss. Iperf was originally developed by NLANR/DAST, and has become one of the de facto tools for performing network measurements. The older code had become unmaintainable and so ESnet has completely rewritten Iperf starting with version 3.0.