Science DMZ for Pennsylvania State University & Virginia Tech Transportation Institute

The Pennsylvania State University's College of Engineering (CoE) collaborates with many partners on jointly funded activities. The Virginia Tech Transportation Institute (VTTI), housed at Virginia Polytechnic Institute and State University, is one such partner. VTTI chooses to collocate computing and storage resources at Penn State, whose network security and management is implemented by local staff. However, due to policy limitations for collocated equipment, a security mechanism in the form of a firewall was required to protect both the campus and VTTI equipment. Shortly after collocation, VTTI users noticed that performance for hosts that were seemingly connected using 1 Gbps local connections, were limited to around 50 Mbps overall; this observation was true in either direction of data flow.

Using perfSONAR, network engineers discovered that the size of the TCP window was not growing beyond the default value of 64 KB, despite the fact that hosts involved in data transfer and measurement testing were configured to use auto-tuning---a mechanism that would allow this value to grow as time, capacity, and demand dictated. To find the correct window size needed to achieve network speeds close to 1 Gbps, the sites were measured at 10ms away in terms of round-trip latency; which yielded a window size of:

Further investigation into the behavior of the network revealed that there was no packet loss observed along the path, and other perfSONAR test servers on campus showed performance to VTTI that exceeded 900 Mbps. From some continued performance monitoring, the investigation began to center on the performance of the CoE firewall.

A review of the firewall configuration revealed that a setting on the firewall, TCP flow sequence checking, modifies the TCP header field that specifies window size (e.g., a clear violation of tcp_window_scaling, set forth in RFC 1323). Disabling this firewall setting increased inbound performance by nearly 5 times, and outbound performance by close to 12 times the original observations. Figure 1 is a capture of overall network utilization to CoE, and shows an immediate increase in performance after the change to the firewall setting.

Figure 1: Penn State College of Engineering network utilization, collected passively from SNMP data.

Because CoE and VTTI were able to utilize the Science DMZ functions, like perfSONAR, engineers were able to locate and resolve the major network performance problem. Figure 1 also shows that numerous users, not just VTTI, were impacted by this abnormality. The alteration in behavior allowed TCP to reach higher levels of throughput, and allowed flows to complete in a shorter time than with a limited window.