Testing and Validation of Network Security Devices

While catching up on security news and blogs the other day, I came across a blog post from ICSA Labs entitled "Why a Test Lab Needs to be Wary of Commercial Exploit Packet Captures" and thought that it would be a good conversation starter to inform our readers about how BreakingPoint approaches developing test cases for security device testing, our methodology behind why we develop our test cases the way we do, and the thought processes and conclusions behind those decisions.

First, it's important to note that ICSA's blog post is primarily talking about test tools that replay packet captures as their security tests. While the BreakingPoint devices do provide a packet capture replay component, this component is not what we use for security testing. The BreakingPoint devices provide a dedicated security component that execute packaged attacks targeting individual vulnerabilities that we call "strikes". Strikes are not packet captures, and we'll discuss how strikes operate and the benefits derived from them a little later in this post.

Toward the beginning of their blog post, ICSA wrote the following:

"If ICSA Labs were to use one or more exploit packet captures created elsewhere, then we would be effectively vouching for the quality and accuracy of these packet captures. But that is the problem; we cannot vouch for their quality and accuracy."

This is also one of the primary reasons that we do not use packet captures of attack traffic that we have come across in our research. However, we take it one step further and and don't even use packet captures created in-house. We simply don't use packet captures for security testing at all, which brings me to the first subject I'd like to discuss:

Attack Realism

Let's look at what ICSA has to say on attack realism using third-party packet captures:

"ICSA Labs does not know whether the code for each would-be exploit actually works as expected. Even if it did work, we cannot confirm that the would-be exploit was run against a vulnerable system when the capture was made. And assuming it was a working exploit that was run against a vulnerable system, we do not know whether the attack succeeded when the packet capture was made. Also, information in the commercial tool typically indicates at which vulnerability each exploit packet capture is aimed. But again, a test lab has no reasonable way to confirm that. To use the tool in this way ICSA Labs would have to make many assumptions and essentially trust an entity outside of our control."

The BreakingPoint Labs team builds each strike by hand after performing our own analysis of the vulnerability. We have a high degree of certainty that our attacks are correct because we do this analysis and then we test the strikes afterward when possible against the actual vulnerable target. Then, we use these strikes (not packet captures of them) in testing performed using the BreakingPoint device. There are currently two ways to test using these strikes; passing attack traffic through an intermediary Device Under Test (DUT), and sending attack traffic directly to an endpoint DUT, which I'll cover next.

Attack Simulation

"But what happens if the vendor's IPS proxies traffic or alters the content of traffic as some IPS products do? Keep in mind that this is a replayed packet capture, not a live exploit. If the commercial tool with its packet capture of an exploit is run against an IPS that does one of these things and the IPS fails to block the attack, did the IPS really fail? Remember, the IPS modified the traffic on the line."

This is a valid concern when testing an intermediary DUT, and even more so when you're using static data from packet captures. In this scenario, our strikes act as both the attacker and target, and send the attack traffic from one port on our device, through the DUT, and back to a second port. In this way, it's really an attack "simulation" using real attack traffic because we're essentially sending traffic back to ourselves rather than a real target. Because we know what valid attack traffic looks like for each individual iteration of the strike, we know what data we're sending, and we know what the data should look like when we (the target) receive it, if the DUT modifies the attack traffic in transit we consider the attack blocked as it is no longer the attack traffic that we sent and is invalid.

"One-Arm" Strikes

"If the IPS vendor cannot reproduce the issue reported to them by the test lab, then the test lab should be able to confirm its findings in some way. But minus the real attack and actual vulnerable system, that is either a very tall order or impossible!"

Once again, we're in total agreement here, which is why we use real attacks. To the extent possible, strikes that target servers can be run in "one-arm" mode where rather than passing attack traffic through a DUT and back to ourselves, the traffic is sent to the DUT as the attack's target server. In this mode, strikes can be used to actually trigger vulnerabilities on actual vulnerable systems. This is what test houses that use BreakingPoint devices like NSS do to verify that the test cases they are using are indeed valid, even though they are provided by BreakingPoint, their vendor.

Custom Strikes

What if BreakingPoint doesn't have a strike for the vulnerability you want to test? Or what if, like ICSA, you don't trust third party content at all? Even though BreakingPoint provides you with real attacks packaged as strikes, users can easily develop their own strikes. I won't cover this topic in any detail here, as we've already had a three part series (1, 2, 3) on this subject posted to the blog.

Strike Development Goals

1. Trigger Just the Vulnerability & Use Unidentifiable Payloads

One of the most frequently raised concerns about our strikes is that they contain no active payloads or executable shellcode. This is by design. Sure, network security devices often have filters for well-known shellcode and common payload encoders, and we have specific strike categories to test those specific cases, however if you are relying on the detection of such by your IPS to protect you from actual vulnerabilities then you have already failed. Most network security devices are reactive in nature, and in order to detect a particular shellcode or payload encoder, it must first be aware of it and/or have a filter for it. We know there are payload encoders and shellcode out there that devices are unaware of, so we simulate this by using completely random data as our payloads. This forces the DUT to identify attacks based on the properties of the vulnerability, not by relying on detecting known shellcode or a decoder stub from an encoded payload. We focus entirely on triggering the vulnerability, not actually exploiting it with an operational payload.

2. Randomness & Uniqueness on the Wire

"ICSA Labs is unwilling to risk its reputation and the trust of end users through the use of packaged exploit packet captures in its testing. All of the exploit packet captures we use in network IPS testing were captured here in the lab by our experts. And in ALL cases, we are in a position to verify our coverage protection test results by running the real, live attack against the actual vulnerable system."

The problem with ICSA's approach here is that you're initially still testing with static packet captures. Consider the scenario where you replay your packet capture of a malicious TIFF file traversing the wire. The IPS under test blocks it, and you mark that as a success. How do you know that if some unrelated parts of the TIFF file are modified, that the IPS won't miss it? How do you know that if you add a whole lot of padding or superfluous structure to the file and move the evil from the beginning of the file to beyond the padding, that the IPS won't miss it? If you're initially relying on packet captures of static attack traffic and then only breaking out the real exploits and targets when something seems amiss or a customer questions your tests, you're not being thorough in your testing.

BreakingPoint's approach to providing these various attack permutations is to identify all of the components of the attack that are absolutely essential for the attack to work and trigger the vulnerability. We identify these values and their upper and lower bound thresholds as well as identify behavioral protocol and process interactions and what combination and permutations of these are valid. We then develop our strikes to randomize these properties as much as possible while still conforming to the identified valid parameters. Further, we randomize as much other data as possible that is not directly related to triggering the vulnerability while still remaining valid for whatever protocol, file format, or other data structure is being used in the attack. All of this context information and the flexibility provided by dynamic test cases such as strikes as opposed to packet captures is the benefit we get from performing the vulnerability analysis ourselves, understanding the operational bounds of the data involved, and developing strikes that launch attacks that actually utilize that knowledge. You can read more about this subject in one of my previous blog posts, File Format Vulnerabilities and Dynamic Exploit Generators.

3. Evasions

To further the previous point, BreakingPoint can optionally also mutate attack traffic by employing various evasion techniques. When you combine evasion techniques such as IP fragmentation with fragment reordering, using various text encoding methods, and HTTP chunked encoding transmission, among others, with the randomization of the attack traffic that we are already performing as outlined in the previous section, nearly endless permutations of a single attack are dynamically generated which using static packet captures simply can't compete with. Forgive me for quoting a deodorant commercial, but "anything less would be uncivilized." For much more in-depth information on the subject of evasions, please see our recent webcast entitled Harden Security Devices Against Increasingly Sophisticated Evasions or this previous blog post on the subject.

Conclusion

I hope you enjoyed this look into the BreakingPoint strike development and security device testing mindset and found the information both useful and enlightening. Please do follow some of the links above as there is much more information available about the topics discussed.

Setting The Standard

Next week is my favorite week of the year. It's the Sales Operations meetings held at our headquarters in Austin. Each year we bring the sales people and sales engineers together to review the previous year and preview the year moving forward. More importantly I get to show off.

2009, from all facets, was an incredible year here at BreakingPoint. Sales had an amazing year, with huge growth. Our employee base grew by nearly 30%, much of that being our heavy investment in the security group. We put out 3 major releases and 3 minor releases of our firmware for the BreakingPoint Elite. And our application protocol list now tops more than 100 and our strikes are over 4,300.

This news is certainly exciting, but that was last year. And this is a completely new year and we are ramping up in engineering like you could not imagine. The next firmware release will once again improve the performance of everything from our application protocols, security engine and our SSL. And, of course this is all done without having to replace your blades and at no extra cost. Bet your other vendors don't say that every year.

Next month I'll be putting together a screencast showing you all the features in our next release. I'll save all the juicy bits for then, but here is a teaser of what to expect:

• Five new test labs, including huge strides for mobile networks.
• Changes in the way we are using the NetLogic network processor.
• Switch from using our network processor cores to do SSL, to leveraging the encryption engine on the chip itself (the impact this has on our number of handshakes is staggering).

Last year we changed the way people test their network equipment, this year we will set the standard.

Reminds me of when I worked at Cisco many years ago and Kevin Kennedy (Vice President) would show a slide in which Cisco was compared to other similar companies. There must have been 30 companies listed and at the time 3Com was below us, Lucent ahead of us and all the way at the top were companies like HP. At the time HP was 10x the size of Cisco. Today, Cisco is tens of billions of dollars ahead of HP, with a third of the employees. 

Every year that presentation showed Cisco passing yet another company. We have the same chart for our industry and the same goals, and some companies were ahead of us at the beginning of 2009. During 2009 we passed four of them and this year we will pass four more. And one day, like Cisco, we'll be at the top of everyone else's list.

NOTE: Sometimes Cisco didn't pass a company, the company fell. I'm seeing a lot of that lately, maybe I should send some flowers.

Virus and Spam Filters

Virus and Spam Filters

Cisco Becomes The Weakest Link In National Infrastructure Security

Last week Cisco released patches in their semi-annual security announcement. The publication includes 11 advisories that address 12 individual vulnerabilities. Ten of the advisories address vulnerabilities in Cisco IOS and one advisory addresses a vulnerability in Cisco Unified Communications Manager. Together these can affect routers and switches that not only use the Cisco Unified Communications Manager, but any device relying on the Cisco IOS operating system. To put it bluntly, this means a ton of devices critical to any network, and these vulnerabilities leave businesses and government agencies exposed to a barrage of attacks including denial-of-service (DDoS) or policy bypass.

Much has been written about the announcement of the vulnerabilities. However, details are lacking and there are more questions than answers. This lack of information leads me to believe Cisco does not take security seriously and continues to not know how to work with the security community. Considering the lack of details and opinions, I thought I would provide a few of my own.

1) Twice A Year Is Not Enough

The number of vulnerabilities patched by Cisco is not the issue. It is the potential danger these vulnerabilities pose. One of the IOS vulnerabilities allows unauthenticated attackers to bypass access control policies when the “Object Groups for Access Control Lists (ACLs)” feature is used. Your company is most likely protecting your critical components by leveraging ACLs, now imagine they are no longer in place. The human resources database with all that W-2 information? Hackers now have your salary, your direct deposit account, your medical history and of course your social security number. To make matters worse, replace that HR database with our government’s nuclear secrets; don’t you think Iran is aware of the Cisco vulnerabilities?

Scary stuff, for sure, but how long has the vulnerability been around and recognized. The answer is unknown. The only fact we have is that each of these eleven vulnerabilities may have been around for at least six months. That is an eternity in the security space and has given hackers too much time to walk in through an open door.

Microsoft is often a punching bag when it comes to vulnerabilities and it is sometimes warranted, but let’s be honest, the company does a good job of patching issues on a regular basis. With Microsoft, you know that you are going to get a patch each month and important details that help you make an informed security decision. Cisco should examine its patching schedule in light of the September 24th announcement; every six months is not acceptable.

2) Updating Routers and Switches is Now Critical

You can never diminish the importance of a switch or router to your network infrastructure. They are the core to any network whether in a home, a large Enterprise or the Federal Government. If one fails you know it. However, if a vulnerability let’s people through due to a hack do you know it? While everyone remembers to patch their Mac or Windows laptop, how often do they patch the router, firewall or switch?

To see how up-to-date folks are with their Cisco firmware I ran a quick test. During a 1-hour scan of the Internet I found 420 responding systems and NONE were patched with any fixes from this cycle or the last. That means 420 systems, at a minimum, are susceptible to a years worth of vulnerabilities.

Microsoft had enough of people not patching and now it force feeds the patches. While I’m not a fan of that solution, it does work. Cisco needs to apply the same method to its products. It is irresponsible for Cisco to run its business in a way that could cause mass disruption to critical network infrastructures including government and military services.

Cisco is not the only one to blame in this mess, the people responsible for getting their routers, switches and other network equipment up-to-date also must be held accountable. How many of you updated with the patches on September 24th, the day of the announcement? The quick scan I did is telling me not many. Kelly Jackson Higgins of Dark Reading put it best, “The dirty little secret about patching routers is that many enterprises don't bother for fear of the fallout any changes to their Cisco router software could have on the rest of the infrastructure.”

3) Testing, Testing, Testing

In this case we have a great example of why every network device needs to be realistically tested under a variety of scenarios, both security and performance driven. Obviously, testing must occur at the NEMs level throughout the product lifecycle, but the enterprise must also test this equipment before it is deployed and after updates like these are made. Having the ability to quickly test equipment and the network after making updates is critical.

There is no room for excuses anymore. We have been able to become more adept at updating and testing equipment and software that are given more regular patches. Just look at how Microsoft Tuesday has become a habit. Other vendors have realized that this approach, ultimately, is better for everyone. I would encourage manufacturers of any network equipment to do the same.

The reason this is important is because the United States is currently fighting in two wars, heavily dependent on network technologies. The Department of Defense and other military agencies have concluded that the next major war will be waged, in great part, in cyberspace. If Cisco and other vendors guilty of the same security concerns do not get their act together it will be a war we cannot win.

Until March 24, 2010, when the next Cisco bulletin is due.

A Look at Some of the New Features in BreakingPoint Elite

Today I wanted to take a look at the latest version of BreakingPoint Elite, which is available immediately to BreakingPoint users, and includes more than 30 new features. I thought I would post a few of the features that we featured in the news release and embed a product screencast from CTO Dennis Cox demonstrating some of the features.

1) Dual Stack IPv4/IPv6 Testing Capabilities and Support for Current IPv6 Standards

BreakingPoint has the unique ability to generate blended stateful application traffic mixed with live security attacks at line-rate speeds and high session counts, delivered from the same address space. Using BreakingPoint Elite's IPv6 dual stack testing you get the industry’s most comprehensive and up to date IPv6 capable testing, allowing you to:

• Simulate IPv6 traffic through each testing component including Client Simulator for load testing IPv6 capable application servers.
• Ensure compliance and validate performance with the latest IPv6 standards and blended IPv4 and IPv6 traffic.

2) Capture and Recreate Functionality

BreakingPoint Elite has the largest capture history buffer, with 8 Gigabytes per blade, to reduce the time it takes to debug network equipment and application servers. This latest product upgrade adds in advanced filtering of packet captures to capture and report on only the data needed. This includes:

• Post-processing ability to compress the capture export, allowing you to specify start and end frames.
• Berkley Packet Filter (BPF) capability to isolate traffic by IP address, port, protocol and more.
• Ingress pre-processing filtering parameters based on VLAN ID, Source IP, Destination IP, Source Port and Destination Port.

Additionally you now have the ability to incorporate data originating from a real network into tests, replaying stateful traffic using the TCP stack, including three-way handshakes and any necessary retransmissions. This includes raw playback of traffic to test for common issues with ARP, RARP, TCP/UDP headers, TCP SYN floods, DDoS, invalid packets and more.

3) Pattern Matching (PCRE) for Full Data Validation of Network Security Devices

BreakingPoint Elite regular expression pattern matching validates traffic flows utilizing a user-defined data pattern by comparing that pattern against incoming network traffic. QA and R&D can quickly identify sequence errors and data errors in a network device or application server, including validation that the network equipment is generating the appropriate network traffic. BreakingPoint Elite also supports PERL Compatible Regular Expressions (PCRE), to allow users to match substrings in data packets, as well as provide the ability to store the matched data for future use.

4) Impairment Support for Realistic Wide-Area Network (WAN) Simulation

Support for simulating impairments within IP traffic for realistic WAN simulation by configuring impairments within BreakingPoint Elite Network Neighborhood and easily replicating and fine-tuning for any DUT.

5) Enhanced Test Automation, Reporting and Layer 2/3 Testing Capabilities

More extensive test automation including Attack Plan Iteration that enables security testers to loop security attacks. Also, reporting enhancements include the ability to customize content to be included in reports, and export in additional formats. Finally, numerous Layer 2/3 testing capabilities have been added including UDP and enhanced HTTP support in Session Sender, addressing expansion, stateless TCP in Routing Robot and expanded frame sizes.