Setting The Standard

Next week is my favorite week of the year. It's the Sales Operations meetings held at our headquarters in Austin. Each year we bring the sales people and sales engineers together to review the previous year and preview the year moving forward. More importantly I get to show off.

2009, from all facets, was an incredible year here at BreakingPoint. Sales had an amazing year, with huge growth. Our employee base grew by nearly 30%, much of that being our heavy investment in the security group. We put out 3 major releases and 3 minor releases of our firmware for the BreakingPoint Elite. And our application protocol list now tops more than 100 and our strikes are over 4,300.

This news is certainly exciting, but that was last year. And this is a completely new year and we are ramping up in engineering like you could not imagine. The next firmware release will once again improve the performance of everything from our application protocols, security engine and our SSL. And, of course this is all done without having to replace your blades and at no extra cost. Bet your other vendors don't say that every year.

Next month I'll be putting together a screencast showing you all the features in our next release. I'll save all the juicy bits for then, but here is a teaser of what to expect:

• Five new test labs, including huge strides for mobile networks.
• Changes in the way we are using the NetLogic network processor.
• Switch from using our network processor cores to do SSL, to leveraging the encryption engine on the chip itself (the impact this has on our number of handshakes is staggering).

Last year we changed the way people test their network equipment, this year we will set the standard.

Reminds me of when I worked at Cisco many years ago and Kevin Kennedy (Vice President) would show a slide in which Cisco was compared to other similar companies. There must have been 30 companies listed and at the time 3Com was below us, Lucent ahead of us and all the way at the top were companies like HP. At the time HP was 10x the size of Cisco. Today, Cisco is tens of billions of dollars ahead of HP, with a third of the employees. 

Every year that presentation showed Cisco passing yet another company. We have the same chart for our industry and the same goals, and some companies were ahead of us at the beginning of 2009. During 2009 we passed four of them and this year we will pass four more. And one day, like Cisco, we'll be at the top of everyone else's list.

NOTE: Sometimes Cisco didn't pass a company, the company fell. I'm seeing a lot of that lately, maybe I should send some flowers.

WAN Optimization

WAN optimization

Protocol Realism: HTTP POST Content-Type

As any semi-competent web application developer could tell you, when you fill out a form and submit it via HTTP, there are a few options your browser has in formatting the response. However, since I am not a semi-competent web application developer, I had to go look this up because we have a BreakingPoint Elite user who recently asked for exactly this functionality in our HTTP Application Simulator.

So, as of the latest Application Protocol and Strike Pack, the HTTP Application Simulator now asks for an optional "Content-boundary" value, to go along with the usual "Content-Type" header, like so:

To illustrate, if we had a test that specified the options as shown above, along with a text file with the specified boundary token of "BPS123" to demarcate each of the HTTP form's values, the result would look something like this:

POST /submit.html HTTP/1.1
Host: default
Connection: Keep-Alive
User-Agent: BreakingPoint/1.x (http://bpointsys.com/)
Accept: */*
Content-Type: multipart/form-data; boundary=BPS123
Content-Length: 395

--BPS123
Content-Disposition: form-data; name="value_one"

This is my first value. I just want to say
--BPS123
Content-Disposition: form-data; name="value_two"

Yo value_one, I'm really happy for you and I'mma let you finish,
but application/x-www-form-urlencoded is the one of the best form
encoding types of all time.

One of the best form encoding types of all time!
--BPS123--

If we had, instead, set the Content-Type to the more typical POST encoding of "application/x-www-form-urlencoded," we could specify another appropriately formatted text file and generate a result that HTTP-aware content inspection devices would understand:

POST /submit.html HTTP/1.1
Host: default
Connection: Keep-Alive
User-Agent: BreakingPoint/1.x (http://bpointsys.com/)
Accept: */*
Content-Type: application/x-www-form-urlencoded
Content-Length: 188

value_one=First+value&value_two=%09This+is+the+first+line%0d%
0aSecond+line,+and+honestly,+this+kanye+thing+isn%27t+all+that+funny+anymore.
%0d%0aNo,+really,+let%27s+just+end+this+now.

Of course, there's also the plain text encoding, which virtually nobody uses. But, just because it's unpopular doesn't mean that your inspection device shouldn't be tested on it. To do so, it's as simple as setting "text/plain" as the Content-Type, and providing the text to look for:

POST /submit.html HTTP/1.1
Host: default
Connection: Keep-Alive
User-Agent: BreakingPoint/1.x (http://bpointsys.com/)
Accept: */*
Content-Type: text/plain
Content-Length: 298

value_one=This is the first value.
value_two=Yo value_one, I'm really hap--
value_three=Yo value_two, I'm really happy for you, and I'mma let you 
finish, but the fake form value interruption in example one was the best 
"I'mma let you finish" joke of this entire blog post.

This entire blog post!

It's a little thing, of course -- just a few UI settings to make sure that the form data you're blasting out at dangerous speeds is formatted according to your fancy. But it's the little things like this that make the case for realistic testing. If a vendor claims a device inspects outgoing HTTP data, then, by gum, it ought to be handling all the vagaries of form encoding. And if the vendor claims it does that, then by double-gum, now you can test it to be sure.

A Look at Some of the New Features in BreakingPoint Elite

Today I wanted to take a look at the latest version of BreakingPoint Elite, which is available immediately to BreakingPoint users, and includes more than 30 new features. I thought I would post a few of the features that we featured in the news release and embed a product screencast from CTO Dennis Cox demonstrating some of the features.

1) Dual Stack IPv4/IPv6 Testing Capabilities and Support for Current IPv6 Standards

BreakingPoint has the unique ability to generate blended stateful application traffic mixed with live security attacks at line-rate speeds and high session counts, delivered from the same address space. Using BreakingPoint Elite's IPv6 dual stack testing you get the industry’s most comprehensive and up to date IPv6 capable testing, allowing you to:

• Simulate IPv6 traffic through each testing component including Client Simulator for load testing IPv6 capable application servers.
• Ensure compliance and validate performance with the latest IPv6 standards and blended IPv4 and IPv6 traffic.

2) Capture and Recreate Functionality

BreakingPoint Elite has the largest capture history buffer, with 8 Gigabytes per blade, to reduce the time it takes to debug network equipment and application servers. This latest product upgrade adds in advanced filtering of packet captures to capture and report on only the data needed. This includes:

• Post-processing ability to compress the capture export, allowing you to specify start and end frames.
• Berkley Packet Filter (BPF) capability to isolate traffic by IP address, port, protocol and more.
• Ingress pre-processing filtering parameters based on VLAN ID, Source IP, Destination IP, Source Port and Destination Port.

Additionally you now have the ability to incorporate data originating from a real network into tests, replaying stateful traffic using the TCP stack, including three-way handshakes and any necessary retransmissions. This includes raw playback of traffic to test for common issues with ARP, RARP, TCP/UDP headers, TCP SYN floods, DDoS, invalid packets and more.

3) Pattern Matching (PCRE) for Full Data Validation of Network Security Devices

BreakingPoint Elite regular expression pattern matching validates traffic flows utilizing a user-defined data pattern by comparing that pattern against incoming network traffic. QA and R&D can quickly identify sequence errors and data errors in a network device or application server, including validation that the network equipment is generating the appropriate network traffic. BreakingPoint Elite also supports PERL Compatible Regular Expressions (PCRE), to allow users to match substrings in data packets, as well as provide the ability to store the matched data for future use.

4) Impairment Support for Realistic Wide-Area Network (WAN) Simulation

Support for simulating impairments within IP traffic for realistic WAN simulation by configuring impairments within BreakingPoint Elite Network Neighborhood and easily replicating and fine-tuning for any DUT.

5) Enhanced Test Automation, Reporting and Layer 2/3 Testing Capabilities

More extensive test automation including Attack Plan Iteration that enables security testers to loop security attacks. Also, reporting enhancements include the ability to customize content to be included in reports, and export in additional formats. Finally, numerous Layer 2/3 testing capabilities have been added including UDP and enhanced HTTP support in Session Sender, addressing expansion, stateless TCP in Routing Robot and expanded frame sizes.

4 Product Bakeoff Pitfalls and How to Avoid Getting Played

Most IT buyers take product data sheet claims with a grain of salt. Capabilities are overblown and performance numbers are guesstimated using only Layer 2/3 or HTTP traffic. We all know this isn't real. So, we don't trick ourselves into believing that the numbers will hold true in a real network.

Beyond the performance numbers, there's also the question of features and functionality. When it comes to network and security equipment, one size does not fit all. Each reacts differently to traffic on your network and offers a unique set of features and performance levels. The top network equipment and server vendors are also combining multi-purpose capabilities to introduce all-in-one products in an effort to differentiate their offerings. Evaluating the best firewall, IPS, or server load balancer to meet your needs just became even more bewildering. How do make head-to-head comparisons with products that operate differently? How do you generate a realistic mix of traffic and still evaluate all of the products with the same mix?

A product bakeoff is the most important step in the hardware evaluation(PDF), negotiation, and planning process because it offers the best way to understand the real capabilities and performance of the devices you are considering for purchase. Yet lack of planning, bad bakeoff behavior, and the use of synthetic traffic often leave buyers with poor decision-making data and a false sense of security. As well all know, a bit of preparation on the front end is always required to get accurate results. Combine that with insider knowledge and you can ensure an accurate and deterministic network device evaluation. Here are 4 product bakeoff secrets every IT buyer should understand before embarking upon any head-to-head product comparison:

1. Bits will bite you. Bit blasting, the use of IMIX or even PCAPs will not give you an accurate view of how a product will perform under load and under attack--in other words--in your network. Mike Hamilton effectively characterizes the problem with IMIX in this post: "Frequently I am asked about UDP "packet blasting", IMIX, RFC 2544 and other testing procedures. My answer is always the same; these testing "methodologies" are not realistic. Testing a firewall with 100% stateless UDP traffic is pointless when the actual traffic it will see consists of a wide variety of applications over both UDP and TCP."

It is absolutely vital that you adequately categorize the makeup of the traffic on your network and simulate these conditions under load during your product bakeoff. As the chart below illustrates, you will likely see several magnitudes of difference in throughput between competing products when exposed to real network traffic.

2. You can have the best of both worlds: realistic and random. Once you've moved up the stack to realistic application layer traffic, you also need to ensure the traffic generated is somewhat random so that you can make direct comparisons between products and generate the same traffic in the same way to produce deterministic results. A pseudo-random mix of traffic will provide more accurate results, help you identify, resolve and validate fixes, and prevent vendors from gaming the system with clever programming tricks. This may sound mutually exclusive, but let me assure you it is not. The answer lies in the use of pseudo random number generator (PRNG).

While most testing products do not provide PRNG capabilities, BreakingPoint leverages PRNG to generate a pseudo-dynamic blend of traffic that is both real and repeatable. Once a seed value is set, BreakingPoint will create data by generating all data variants in the same way for each test executed. You can pick a number and generate the attacks or application traffic the same way every time to run repeated tests on different products or to repeat tests on the same product or network to determine the source or resolution to a problem.

3. Don't be left with a false sense of security. Another important reason to evaluate security equipment with random mix of traffic is because it is the only way to truly evaluate vulnerability vs. exploit variant detection capabilities. It's generally agreed that vulnerability filtering provides superior security coverage. Yet many products can’t perform the complex matching needed to deliver a vulnerability filter. They offer products that are programmed to identify exploits. The problem with this approach is that slight variants of the exploits these systems are programmed to identify can easily bypass the system. Vendors then release new filters and hackers develop new variants to bypass the same vendors again. The Conficker worm is a prime example of new variants bypassing security products every few months.

A dirty little secret of synthetic testing vendors is that their exploits are branded with trademarks or other recognizable content. Vendors can easily exploit this code by programming their products to recognize the code and trigger filters to easily pass product validation. While it may appear that these products are working as promised, this is no indication that the equipment is capable of recognizing and filtering real security attacks in a production network. PRNG eliminates the possibility that devices under test can be programmed to recognize and react to codes embedded in test traffic.

4. Stick to a proven test methodology. One of the challenges for buyers who are structuring their product bakeoff, is the lack of good solid test methodologies for thoroughly validating equipment. They use RFC's that are outdated, specifications become obsolete quickly, and most of the methodologies you’ll find online are many years old. Since they were published, network equipment has evolved, so you need a current product bakeoff test methodology designed specifically for the devices you are evaluating. Here are a few the BreakingPoint Labs team has published: Firewall Testing methodology, Server Load Balancer Testing methodology, IPS Testing methodology and other test methodologies. However, more are needed.

Tip of the Product Bakeoff Iceberg?
These are probably just the start so I’d like to hear your product bakeoff advice and your methodology wish list. By sharing tips, collaborating on test methodologies, and exposing secrets, we can all get the most out of product bakeoffs.