Watch Out for That Hockey Stick: The Need for Network Device Product Evaluations
by Chuck McAuleyBy Chuck McAuley
The key to predicting a latency spike, or “hockey stick” curve, in the performance of a network or security device is to stress that device to failure using your own unique traffic mix — before you deploy it in production. Otherwise, one of two things is likely to happen:
- Your device will fail — probably at the worst possible time — while it’s carrying important business traffic.
- Not knowing for sure where the hockey sticks are hiding, you’ll be relegated to overbuilding your network — which translates into spending too much money, too soon, on capacity that you don’t actually need.
Today I’m going to show you how to identify these latency spikes during the product evaluation or “bakeoff” stage. Why during evaluation? Simple: before you buy that new network or data center device, you have leverage to get the vendor to either fix the issue or at least provide a discount for lower-than-advertised performance. The key during any bakeoff is to not leave yourself at the mercy of what the device’s data sheet says, which typically reflects idealized traffic scenarios in the vendor’s lab. Let’s look at how to discover latency spikes during a product evaluation using your own representative network traffic and user behavior.
Measuring Performance Latency During Firewall Evaluations
Latency hockey sticks came to mind recently during a performance evaluation of a proxy firewall. One of our customers was using our device to perform this evaluation. As results started coming in, I pulled up an analysis of the total time for Web page transactions versus the number of requests per second on a customer’s network. The resulting graph looked like this:
The y-axis is the socket lifetime in milliseconds. This is the amount of time it took to connect to a Web server, send a GET request, retrieve the page, and then close the connection with a FIN. The x-axis is the number of connections attempted per second. I ran 25 separate tests, each attempting a different number of connections per second. The average socket lifetime (you could consider it the “transaction latency” if you wanted) for the entire test run was measured.
What you can see in the results is very familiar to people working in performance test labs. It’s a telltale hockey stick curve that shows when resource utilization has maxed out. And that little blip up around 2500 CPS? That’s when we actually start failing transactions. We can safely ignore the rest of the curve beyond that, because we have run out of resources.
What is interesting is the nearly flat line up until 2,300 connections per second. When you think about it, this is how network devices should operate: almost perfect performance until you hit the device’s breaking point. If I were the maker of this equipment, my next job would be to figure out which resource it was that ran out. Was it CPU cycles? Available memory? Some bottleneck based on hashing algorithms? You can’t guess about this kind of stuff, because in real life the causes are too diverse and too unpredictable.
Evaluating Network Devices Based on Your Unique Traffic Behavior
Let’s now try to take this data and put it into terms that are meaningful for a live network. The big insight from the graph above is that knowing the level of resource utilization within a device might not be very helpful in terms of understanding its actual capacity to handle live traffic. If I measure CPU utilization, for instance, is 90% going to be OK, or is that entering the danger zone — right where the hockey stick starts to curve? Or is the danger zone at 75%? Or 96%? Again, you can’t afford to just guess.
Maybe for the device in question, it’s not about the CPU at all, so you could track CPU cycles as closely as you like — but then get hit with a hockey stick because of a problem with memory allocation or, worse still, a bottleneck between two other elements within your system.
Keep in mind that the graph above is simplified: it represents a generic process that I ran in a lab. But the lesson still holds for the real world, because you need to find the hockey sticks that will appear in your own network. Since the permutations of network traffic are endless, you can’t simulate every single scenario. But you can — and you should — use this approach with the same traffic that you expect to see on your network.
The only real way to find out is to stress the device until it breaks, and to do it with real representative traffic. That means you can’t rely on the vendor’s idealized mix or IMIX or PCAPs, or on simple HTTP sessions. You have to use an accurate simulation of the network traffic that the device will be forced to handle in the real world.
Pointing out the hockey sticks in latency may seem like Networking 101, but you’d be surprised at how often even well-trained IT people miss the implications of not knowing exactly where those hockey sticks are. And you would also be surprised at how many companies are not stressing devices like this and taking advantage of that data during the product evaluation.
Back to the Proxy Firewall Evaluation
When I was evaluating the proxy firewall, I had a chance to talk with the engineers responsible for securing that network — smart people responsible for a big Internet company’s infrastructure. Since they’d never had a tool before that would let them create their own unique network traffic and measure its exact effects on their devices to the point of failure, they had always taken the cautious route: when they saw devices hitting around 20% of their projected maximum load, they just bought more boxes. It was a safe way to do things, because they had plenty of capacity to spare. But it was also much more expensive than it needed to be, because in fact they were buying much more capacity than they ever needed.
Once they had that kind of factual, detailed information on a device’s breaking point — that place where the hockey stick crushes it — it really opened their eyes. In fact, they immediately started dialing up scenarios to probe the weaknesses of each component they had.
If I’ve convinced you of the danger of hockey sticks, you should try the same approach. You might be surprised at how much life is left in that old beast of a firewall in your data center. Or you might find out that you should have been replacing it yesterday.
