From the Floor at RSA 2010: Real-World Mobile Network Traffic Validation

On the show floor at RSA Conference there is a lot happening and overall the show seems much more well attended than last year. This show, as most of you know, is also a harbinger of news releases and product announcements. Crossbeam, providers of scalable software and hardware platforms, distributed a few pieces of news leading up to the show and at the conference itself. I went over to visit the Crossbeam booth (#545) while at RSA so check out a live demonstration of their X-Series security platform using four BreakingPoint Elite chassis. With this impressive demonstration in the background I talked with Crossbeam's Peter Doggart.

Q. First off Peter, can you provide us with an overview of what Crossbeam provides?

Crossbeam’s X-Series security platform lets customers virtualize third-party, best-in-class security applications and scale them to meet the needs of large, high-performance network environments. Today, more than 900 leading enterprises and service providers, including 10 of the top 11 telecom carriers worldwide, rely on Crossbeam as the underlying architecture for the delivery of security services.

Q. Crossbeam is demonstrating something very interesting here at RSA, can you tell us about what is going on and why?

In working with service providers over the past year, and in particular mobile network operators (MNOs), it has become evident that they are under enormous pressure to meet growing network demands while simultaneously delivering “clean” data pipes.

What we are showing at RSA is proof that our X-Series security platform delivers the world’s fastest firewall performance to meet the needs of mobile operators. Using BreakingPoint Elite, we are conducting a to stress-test the X-Series chassis. We are running a best-in-class application on the X-Series, Check Point Security Gateway R70 Firewall, to clean, inspect and secure the traffic.

This demonstration shows how service providers and mobile carriers can easily scale their network security infrastructure to cope with the next generation of mobile technology, 4G/LTE, under real-world conditions.

Q. You mention “real world” a few times in your answer and in the news release that went out. What does that mean to mobile network operators?

There is a growing gap between what vendors state on their data sheets and what we typically see out in the real world in terms of performance. There are two key elements at play in the real world. First, we are seeing more attacks, which place a greater burden on our security systems and, second, we are seeing smaller payload sizes, especially with the growing number of mobile devices. The result is that mobile operators need to buy and manage a lot more equipment than they budgeted for as the real-world demands are far greater than they ever anticipated. This is not only more costly to them, but it is also a lot more complex to manage.

Realistic tests like this one at RSA validate that we deliver the fastest-performing firewall on the market under real-world conditions which means that we can stand behind our performance claims and mobile network operators can be assured that their X-Series security infrastructure delivers the flexibility, superior performance and high availability required to handle the unpredictability of growing data traffic demands.

Q. How can this type of validation, throughout the industry, not just at Crossbeam, help the overall performance of MNOs?

Crossbeam’s policy is to be transparent when it comes to performance claims. We are doing the opposite of what many vendors do by actually creating tests that provide worst case metrics, not the best case. Take the RSA live demonstration. We are using BreakingPoint to generate 96 byte HTTP packets, which in the real mobile world would be the worst case payload size. At Crossbeam, we want to create some real-world industry guidelines that everyone follows so mobile operators, government and enterprise customers understand exactly what they are buying, and can capacity plan correctly.

Q. I noticed four BreakingPoint chassis in the Crossbeam booth generating the traffic for the demonstration. Why does Crossbeam use BreakingPoint for product validation?

First, we use the BreakingPoint Elite chassis because they can accurately simulate the type of traffic we see in the real world and, second, because BreakingPoint is the only vendor that can push the Crossbeam chassis to its current performance limits.

Q. How has using BreakingPoint helped the evolution of Crossbeam products?

Because BreakingPoint equipment pushes our chassis to its absolute limits, Crossbeam is better able to fine-tune its performance to address customer needs with the assurance that the X-Series can handle their network demands. In the latest release of the X-Series operating system, for instance, we boosted the number of concurrent IP connections we can support up to 10 million, and increased the new connection rates per second to 320,000. These numbers are critical to mobile operators who need to support the growing number of smartphones and other devices, which create more traffic than traditional mobile phones and are nearly always connected. Without BreakingPoint, we couldn’t have confidence in our real-world performance metrics.

Testing and Validation of Network Security Devices

While catching up on security news and blogs the other day, I came across a blog post from ICSA Labs entitled "Why a Test Lab Needs to be Wary of Commercial Exploit Packet Captures" and thought that it would be a good conversation starter to inform our readers about how BreakingPoint approaches developing test cases for security device testing, our methodology behind why we develop our test cases the way we do, and the thought processes and conclusions behind those decisions.

First, it's important to note that ICSA's blog post is primarily talking about test tools that replay packet captures as their security tests. While the BreakingPoint devices do provide a packet capture replay component, this component is not what we use for security testing. The BreakingPoint devices provide a dedicated security component that execute packaged attacks targeting individual vulnerabilities that we call "strikes". Strikes are not packet captures, and we'll discuss how strikes operate and the benefits derived from them a little later in this post.

Toward the beginning of their blog post, ICSA wrote the following:

"If ICSA Labs were to use one or more exploit packet captures created elsewhere, then we would be effectively vouching for the quality and accuracy of these packet captures. But that is the problem; we cannot vouch for their quality and accuracy."

This is also one of the primary reasons that we do not use packet captures of attack traffic that we have come across in our research. However, we take it one step further and and don't even use packet captures created in-house. We simply don't use packet captures for security testing at all, which brings me to the first subject I'd like to discuss:

Attack Realism

Let's look at what ICSA has to say on attack realism using third-party packet captures:

"ICSA Labs does not know whether the code for each would-be exploit actually works as expected. Even if it did work, we cannot confirm that the would-be exploit was run against a vulnerable system when the capture was made. And assuming it was a working exploit that was run against a vulnerable system, we do not know whether the attack succeeded when the packet capture was made. Also, information in the commercial tool typically indicates at which vulnerability each exploit packet capture is aimed. But again, a test lab has no reasonable way to confirm that. To use the tool in this way ICSA Labs would have to make many assumptions and essentially trust an entity outside of our control."

The BreakingPoint Labs team builds each strike by hand after performing our own analysis of the vulnerability. We have a high degree of certainty that our attacks are correct because we do this analysis and then we test the strikes afterward when possible against the actual vulnerable target. Then, we use these strikes (not packet captures of them) in testing performed using the BreakingPoint device. There are currently two ways to test using these strikes; passing attack traffic through an intermediary Device Under Test (DUT), and sending attack traffic directly to an endpoint DUT, which I'll cover next.

Attack Simulation

"But what happens if the vendor's IPS proxies traffic or alters the content of traffic as some IPS products do? Keep in mind that this is a replayed packet capture, not a live exploit. If the commercial tool with its packet capture of an exploit is run against an IPS that does one of these things and the IPS fails to block the attack, did the IPS really fail? Remember, the IPS modified the traffic on the line."

This is a valid concern when testing an intermediary DUT, and even more so when you're using static data from packet captures. In this scenario, our strikes act as both the attacker and target, and send the attack traffic from one port on our device, through the DUT, and back to a second port. In this way, it's really an attack "simulation" using real attack traffic because we're essentially sending traffic back to ourselves rather than a real target. Because we know what valid attack traffic looks like for each individual iteration of the strike, we know what data we're sending, and we know what the data should look like when we (the target) receive it, if the DUT modifies the attack traffic in transit we consider the attack blocked as it is no longer the attack traffic that we sent and is invalid.

"One-Arm" Strikes

"If the IPS vendor cannot reproduce the issue reported to them by the test lab, then the test lab should be able to confirm its findings in some way. But minus the real attack and actual vulnerable system, that is either a very tall order or impossible!"

Once again, we're in total agreement here, which is why we use real attacks. To the extent possible, strikes that target servers can be run in "one-arm" mode where rather than passing attack traffic through a DUT and back to ourselves, the traffic is sent to the DUT as the attack's target server. In this mode, strikes can be used to actually trigger vulnerabilities on actual vulnerable systems. This is what test houses that use BreakingPoint devices like NSS do to verify that the test cases they are using are indeed valid, even though they are provided by BreakingPoint, their vendor.

Custom Strikes

What if BreakingPoint doesn't have a strike for the vulnerability you want to test? Or what if, like ICSA, you don't trust third party content at all? Even though BreakingPoint provides you with real attacks packaged as strikes, users can easily develop their own strikes. I won't cover this topic in any detail here, as we've already had a three part series (1, 2, 3) on this subject posted to the blog.

Strike Development Goals

1. Trigger Just the Vulnerability & Use Unidentifiable Payloads

One of the most frequently raised concerns about our strikes is that they contain no active payloads or executable shellcode. This is by design. Sure, network security devices often have filters for well-known shellcode and common payload encoders, and we have specific strike categories to test those specific cases, however if you are relying on the detection of such by your IPS to protect you from actual vulnerabilities then you have already failed. Most network security devices are reactive in nature, and in order to detect a particular shellcode or payload encoder, it must first be aware of it and/or have a filter for it. We know there are payload encoders and shellcode out there that devices are unaware of, so we simulate this by using completely random data as our payloads. This forces the DUT to identify attacks based on the properties of the vulnerability, not by relying on detecting known shellcode or a decoder stub from an encoded payload. We focus entirely on triggering the vulnerability, not actually exploiting it with an operational payload.

2. Randomness & Uniqueness on the Wire

"ICSA Labs is unwilling to risk its reputation and the trust of end users through the use of packaged exploit packet captures in its testing. All of the exploit packet captures we use in network IPS testing were captured here in the lab by our experts. And in ALL cases, we are in a position to verify our coverage protection test results by running the real, live attack against the actual vulnerable system."

The problem with ICSA's approach here is that you're initially still testing with static packet captures. Consider the scenario where you replay your packet capture of a malicious TIFF file traversing the wire. The IPS under test blocks it, and you mark that as a success. How do you know that if some unrelated parts of the TIFF file are modified, that the IPS won't miss it? How do you know that if you add a whole lot of padding or superfluous structure to the file and move the evil from the beginning of the file to beyond the padding, that the IPS won't miss it? If you're initially relying on packet captures of static attack traffic and then only breaking out the real exploits and targets when something seems amiss or a customer questions your tests, you're not being thorough in your testing.

BreakingPoint's approach to providing these various attack permutations is to identify all of the components of the attack that are absolutely essential for the attack to work and trigger the vulnerability. We identify these values and their upper and lower bound thresholds as well as identify behavioral protocol and process interactions and what combination and permutations of these are valid. We then develop our strikes to randomize these properties as much as possible while still conforming to the identified valid parameters. Further, we randomize as much other data as possible that is not directly related to triggering the vulnerability while still remaining valid for whatever protocol, file format, or other data structure is being used in the attack. All of this context information and the flexibility provided by dynamic test cases such as strikes as opposed to packet captures is the benefit we get from performing the vulnerability analysis ourselves, understanding the operational bounds of the data involved, and developing strikes that launch attacks that actually utilize that knowledge. You can read more about this subject in one of my previous blog posts, File Format Vulnerabilities and Dynamic Exploit Generators.

3. Evasions

To further the previous point, BreakingPoint can optionally also mutate attack traffic by employing various evasion techniques. When you combine evasion techniques such as IP fragmentation with fragment reordering, using various text encoding methods, and HTTP chunked encoding transmission, among others, with the randomization of the attack traffic that we are already performing as outlined in the previous section, nearly endless permutations of a single attack are dynamically generated which using static packet captures simply can't compete with. Forgive me for quoting a deodorant commercial, but "anything less would be uncivilized." For much more in-depth information on the subject of evasions, please see our recent webcast entitled Harden Security Devices Against Increasingly Sophisticated Evasions or this previous blog post on the subject.

Conclusion

I hope you enjoyed this look into the BreakingPoint strike development and security device testing mindset and found the information both useful and enlightening. Please do follow some of the links above as there is much more information available about the topics discussed.

Setting The Standard

Next week is my favorite week of the year. It's the Sales Operations meetings held at our headquarters in Austin. Each year we bring the sales people and sales engineers together to review the previous year and preview the year moving forward. More importantly I get to show off.

2009, from all facets, was an incredible year here at BreakingPoint. Sales had an amazing year, with huge growth. Our employee base grew by nearly 30%, much of that being our heavy investment in the security group. We put out 3 major releases and 3 minor releases of our firmware for the BreakingPoint Elite. And our application protocol list now tops more than 100 and our strikes are over 4,300.

This news is certainly exciting, but that was last year. And this is a completely new year and we are ramping up in engineering like you could not imagine. The next firmware release will once again improve the performance of everything from our application protocols, security engine and our SSL. And, of course this is all done without having to replace your blades and at no extra cost. Bet your other vendors don't say that every year.

Next month I'll be putting together a screencast showing you all the features in our next release. I'll save all the juicy bits for then, but here is a teaser of what to expect:

• Five new test labs, including huge strides for mobile networks.
• Changes in the way we are using the NetLogic network processor.
• Switch from using our network processor cores to do SSL, to leveraging the encryption engine on the chip itself (the impact this has on our number of handshakes is staggering).

Last year we changed the way people test their network equipment, this year we will set the standard.

Reminds me of when I worked at Cisco many years ago and Kevin Kennedy (Vice President) would show a slide in which Cisco was compared to other similar companies. There must have been 30 companies listed and at the time 3Com was below us, Lucent ahead of us and all the way at the top were companies like HP. At the time HP was 10x the size of Cisco. Today, Cisco is tens of billions of dollars ahead of HP, with a third of the employees. 

Every year that presentation showed Cisco passing yet another company. We have the same chart for our industry and the same goals, and some companies were ahead of us at the beginning of 2009. During 2009 we passed four of them and this year we will pass four more. And one day, like Cisco, we'll be at the top of everyone else's list.

NOTE: Sometimes Cisco didn't pass a company, the company fell. I'm seeing a lot of that lately, maybe I should send some flowers.

Unified Threat Management

Unified Threat Management

A Look at Some of the New Features in BreakingPoint Elite

Today I wanted to take a look at the latest version of BreakingPoint Elite, which is available immediately to BreakingPoint users, and includes more than 30 new features. I thought I would post a few of the features that we featured in the news release and embed a product screencast from CTO Dennis Cox demonstrating some of the features.

1) Dual Stack IPv4/IPv6 Testing Capabilities and Support for Current IPv6 Standards

BreakingPoint has the unique ability to generate blended stateful application traffic mixed with live security attacks at line-rate speeds and high session counts, delivered from the same address space. Using BreakingPoint Elite's IPv6 dual stack testing you get the industry’s most comprehensive and up to date IPv6 capable testing, allowing you to:

• Simulate IPv6 traffic through each testing component including Client Simulator for load testing IPv6 capable application servers.
• Ensure compliance and validate performance with the latest IPv6 standards and blended IPv4 and IPv6 traffic.

2) Capture and Recreate Functionality

BreakingPoint Elite has the largest capture history buffer, with 8 Gigabytes per blade, to reduce the time it takes to debug network equipment and application servers. This latest product upgrade adds in advanced filtering of packet captures to capture and report on only the data needed. This includes:

• Post-processing ability to compress the capture export, allowing you to specify start and end frames.
• Berkley Packet Filter (BPF) capability to isolate traffic by IP address, port, protocol and more.
• Ingress pre-processing filtering parameters based on VLAN ID, Source IP, Destination IP, Source Port and Destination Port.

Additionally you now have the ability to incorporate data originating from a real network into tests, replaying stateful traffic using the TCP stack, including three-way handshakes and any necessary retransmissions. This includes raw playback of traffic to test for common issues with ARP, RARP, TCP/UDP headers, TCP SYN floods, DDoS, invalid packets and more.

3) Pattern Matching (PCRE) for Full Data Validation of Network Security Devices

BreakingPoint Elite regular expression pattern matching validates traffic flows utilizing a user-defined data pattern by comparing that pattern against incoming network traffic. QA and R&D can quickly identify sequence errors and data errors in a network device or application server, including validation that the network equipment is generating the appropriate network traffic. BreakingPoint Elite also supports PERL Compatible Regular Expressions (PCRE), to allow users to match substrings in data packets, as well as provide the ability to store the matched data for future use.

4) Impairment Support for Realistic Wide-Area Network (WAN) Simulation

Support for simulating impairments within IP traffic for realistic WAN simulation by configuring impairments within BreakingPoint Elite Network Neighborhood and easily replicating and fine-tuning for any DUT.

5) Enhanced Test Automation, Reporting and Layer 2/3 Testing Capabilities

More extensive test automation including Attack Plan Iteration that enables security testers to loop security attacks. Also, reporting enhancements include the ability to customize content to be included in reports, and export in additional formats. Finally, numerous Layer 2/3 testing capabilities have been added including UDP and enhanced HTTP support in Session Sender, addressing expansion, stateless TCP in Routing Robot and expanded frame sizes.