4 Product Bakeoff Pitfalls and How to Avoid Getting Played
by BreakingPoint LabsUPDATE: BreakingPoint recently published a Product Bakeoff White Paper, available for download.
Most IT buyers take product data sheet claims with a grain of salt. Capabilities are overblown and performance numbers are guesstimated using only Layer 2/3 or HTTP traffic. We all know this isn't real. So, we don't trick ourselves into believing that the numbers will hold true in a real network.
Beyond the performance numbers, there's also the question of features and functionality. When it comes to network and security equipment, one size does not fit all. Each reacts differently to traffic on your network and offers a unique set of features and performance levels. The top network equipment and server vendors are also combining multi-purpose capabilities to introduce all-in-one products in an effort to differentiate their offerings. Evaluating the best firewall, IPS, or server load balancer to meet your needs just became even more bewildering. How do make head-to-head comparisons with products that operate differently? How do you generate a realistic mix of traffic and still evaluate all of the products with the same mix?
A product bakeoff is the most important step in the hardware evaluation(PDF), negotiation, and planning process because it offers the best way to understand the real capabilities and performance of the devices you are considering for purchase. Yet lack of planning, bad bakeoff behavior, and the use of synthetic traffic often leave buyers with poor decision-making data and a false sense of security. As well all know, a bit of preparation on the front end is always required to get accurate results. Combine that with insider knowledge and you can ensure an accurate and deterministic network device evaluation. Here are 4 product bakeoff secrets every IT buyer should understand before embarking upon any head-to-head product comparison:
1. Bits will bite you. Bit blasting, the use of IMIX or even PCAPs will not give you an accurate view of how a product will perform under load and under attack--in other words--in your network. Mike Hamilton effectively characterizes the problem with IMIX in this post: "Frequently I am asked about UDP "packet blasting", IMIX, RFC 2544 and other testing procedures. My answer is always the same; these testing "methodologies" are not realistic. Testing a firewall with 100% stateless UDP traffic is pointless when the actual traffic it will see consists of a wide variety of applications over both UDP and TCP."
It is absolutely vital that you adequately categorize the makeup of the traffic on your network and simulate these conditions under load during your product bakeoff. As the chart below illustrates, you will likely see several magnitudes of difference in throughput between competing products when exposed to real network traffic.
2. You can have the best of both worlds: realistic and random. Once you've moved up the stack to realistic application layer traffic, you also need to ensure the traffic generated is somewhat random so that you can make direct comparisons between products and generate the same traffic in the same way to produce deterministic results. A pseudo-random mix of traffic will provide more accurate results, help you identify, resolve and validate fixes, and prevent vendors from gaming the system with clever programming tricks. This may sound mutually exclusive, but let me assure you it is not. The answer lies in the use of pseudo random number generator (PRNG).
While most testing products do not provide PRNG capabilities, BreakingPoint leverages PRNG to generate a pseudo-dynamic blend of traffic that is both real and repeatable. Once a seed value is set, BreakingPoint will create data by generating all data variants in the same way for each test executed. You can pick a number and generate the attacks or application traffic the same way every time to run repeated tests on different products or to repeat tests on the same product or network to determine the source or resolution to a problem.
3. Don't be left with a false sense of security. Another important reason to evaluate security equipment with random mix of traffic is because it is the only way to truly evaluate vulnerability vs. exploit variant detection capabilities. It's generally agreed that vulnerability filtering provides superior security coverage. Yet many products can’t perform the complex matching needed to deliver a vulnerability filter. They offer products that are programmed to identify exploits. The problem with this approach is that slight variants of the exploits these systems are programmed to identify can easily bypass the system. Vendors then release new filters and hackers develop new variants to bypass the same vendors again. The Conficker worm is a prime example of new variants bypassing security products every few months.
A dirty little secret of synthetic testing vendors is that their exploits are branded with trademarks or other recognizable content. Vendors can easily exploit this code by programming their products to recognize the code and trigger filters to easily pass product validation. While it may appear that these products are working as promised, this is no indication that the equipment is capable of recognizing and filtering real security attacks in a production network. PRNG eliminates the possibility that devices under test can be programmed to recognize and react to codes embedded in test traffic.
4. Stick to a proven test methodology. One of the challenges for buyers who are structuring their product bakeoff, is the lack of good solid test methodologies for thoroughly validating equipment. They use RFC's that are outdated, specifications become obsolete quickly, and most of the methodologies you’ll find online are many years old. Since they were published, network equipment has evolved, so you need a current product bakeoff test methodology designed specifically for the devices you are evaluating. Here are a few the BreakingPoint Labs team has published: Firewall Testing methodology, Server Load Balancer Testing methodology, IPS Testing methodology and other test methodologies. However, more are needed.
Tip of the Product Bakeoff Iceberg?
These are probably just the start so I’d like to hear your product bakeoff advice and your methodology wish list. By sharing tips, collaborating on test methodologies, and exposing secrets, we can all get the most out of product bakeoffs.
