May 18, 2011

How You Set Up Your Next Competitive Product Evaluation Will Make or Break Your Network Infrastructure Upgrade

by Pam O'Neal

Late last year, Enterprise Strategy Group reported that less than a third of companies in critical infrastructure industries (banking, electric utilities, etc.) conducted consistent vendor evaluations. In the SANS Institute survey on network security BreakingPoint just conducted, only 18 percent of respondents said that they have a formal testing and validation program in place to harden elements of their infrastructure against attack.

Troubling statistics, but not surprising, considering it takes more than a CCIE certification and a traffic generator to truly validate today’s content-aware devices. And then there’s the endless organizational balancing act among the projects that “should” be done versus those that “must” be done. Most IT buyers agree that device evaluations should be done, but it’s a challenge to carve out the time to plan, stage, and execute a meaningful bakeoff given the constant fire drills in the typical IT organization. I’m betting that the big companies damaged by recent high-profile security breaches and outages might be rethinking those priorities right now.

What You Will Learn from a Proper Device Bakeoff

Most buyers haven’t considered the benefits—both before and after purchase—of conducting a proper bakeoff. They also don’t realize just how fast and easy it is to set up a proper bakeoff if you have the right roadmap (we have a handy product bakeoff guide available) and tools. And, of course, you can always outsource the job. However you proceed, going through the bakeoff process gives you the advance insight you need to not only make the right purchase, but also to configure, tune, and deploy new devices without incident. It even helps you rightsize your infrastructure as a whole.

In a recent two-day device evaluation that we performed, our clients learned so much about the actual flow of traffic across their network that they were able to save $80,000 on a firewall upgrade. The customer went into the bakeoff thinking that they would need two pairs of redundant firewalls to maintain security for a particular installation. But the evaluation surprised them by showing that properly tuned firewalls outperformed their expectations. The company was able to buy just one pair of firewalls and save 50% on their planned investment.

Now, performance above spec is not the most common case. More often than not, we see devices that perform well below their advertised numbers. In another instance, we saw a firewall that supposedly performed at 20 Gbps slowed to just 5 Gbps when it was subjected to the actual traffic seen on the network where it would be deployed. Whatever the bakeoff reveals, as a buyer you want to know what you’re getting into before you sign the purchase order.

Finally, there are the benefits that are difficult to quantify. Untested equipment requires weeks of post-deployment tuning and troubleshooting that’s frustrating for staff members and that often leads to both finger-pointing and expensive remediation steps. It’s even worse when device outages, security breaches, or unplanned bottlenecks impact the resiliency of entire infrastructures. Failures like that—the ones that have dominated tech headlines in the past couple of months—damage brands while leading to serious, even career-threatening, embarrassment for individuals. By contrast, conducting a rigorous bakeoff minimizes the risk of all these problems and saves hundreds of hours of staff time by eliminating surprises and guesswork.

Avoid Bakeoff Shortcuts: UDP, HTTP, and PCAPs

It shouldn’t be surprising that the key to getting value from your competitive device evaluation is to use your own application traffic—the real-world mix of applications and attacks seen in production. If you’re evaluating content-aware devices, you want real predictability about their future performance, security, and stability. The only way to get that predictability is to use authentic conditions. How you accomplish this will make all the difference in the value you gain from your bakeoff.

Unfortunately, too many organizations take shortcuts, performing evaluations with inferior alternatives to real application traffic. Here are a few of the common culprits:

Possibly the least useful approach is something like plain UDP traffic. Using this to “test” a device’s capabilities is like seeing how fast you can rev a race car’s engine when it’s not even installed in the car. It might be interesting in the abstract—but it won’t tell you how fast the car will run on the track.
Only slightly better is HTTP traffic. Think about it: How many of your devices in production have the luxury of handling only HTTP packets, all of a consistent size? These days, HTTP acts more like a transport protocol, and network operations engineers know to expect complex mixtures of other applications layered on top of it.
There are standard mixes like IMIX, but these are also inadequate—as our CTO Dennis Cox pointed out nearly three years ago in this post.
Finally, there are packet captures (PCAPs), which attempt to substitute a small slice of real traffic for a steady flow of it. PCAPs have enough problems to warrant a post of their own, but here’s the short version:
- A given PCAP is unlikely to reflect the actual mix of application protocols that traverse your network, and cannot reflect the variability of timing and interactions that applications routinely display.
- A PCAP can’t change to reflect changes in an application protocol, which means you have to recapture frequently.
- PCAPs are unwieldy to handle if they’re any bigger than tens of megabytes, yet you need many gigabytes of traffic to adequately test modern equipment.
- Caching is so good on content-aware devices these days that repetitive PCAPs are sniffed out immediately.

New Guide to the Best Bakeoffs

Now that we’ve laid out the pitfalls, what’s next? The experts in BreakingPoint’s Professional Services organization have compiled A Six-Step Plan for Competitive Device Evaluations. This 14-page guide is full of best practices they’ve developed in some of the world’s largest network and data center infrastructures. Download your own copy of this recently published guide by following this link. And if you know you still won’t have time to do your own bakeoff, take a look at our Device Evaluation Service to find out how we can do one for you in less than a week.