BreakingPoint Labs Blog

Resiliency. Don't Leave Home Without It

Mike Hamilton — 2010-03-09T14:45:00Z

This morning I found myself very frustrated as I attempted to fill out my expense report. Keeping track of expenses is a painful exercise. You must remember to get receipts, remember to save those receipts, keep those boarding passes and submit the report in a timely manner. Let’s face it. It is an enormous pain. Here is where my American Express card typically saves the day.

I do love my American Express card, it has never let me down and while I do a fairly good job of keeping up with my receipts, sometimes things slip my mind or I lose a receipt. In those cases the first thing I do is login to my American Express account check out all my statements (since I'm so environmentally friendly I've rid myself of those pesky paper statements). It is a great resource to double check that I haven't missed any expenses and it ensures I'm reimbursed for my work-related expenses.

Unfortunately, when I logged on this morning, this is what I saw:

Not only am I unable to access my account, they are unable to write an error message in proper English. Of course, once I saw this message I thought I’d simply hit the trusty back arrow and try again, except this time this is what I got:

“We’re Sorry”? That’s all you have to say about this? And you don't even have the decency to properly format the HTML to display in my browser?

In 2009 American Express did about $24 Billion in revenue with a net income of about $2.1 Billion, yet they haven't figured out whether their network and application infrastructure can handle the load of customers simply trying to check their recent card activity. You could probably argue that it is not a big deal if a handful of their customers cant login to see their statement and pay their bills. But, if their customers are unable to login and pay their bills, maybe those customers forget to “try again sometime” and pay their bill a lot later. If you've ever studied finance, you probably know about the time value of money. All other things being equal, American Express would much rather have my credit card payment today rather than next week, especially considering the financial climate over the last 18 months. Because of their network and application infrastructure failure, they aren't going to get my payment today.

While this in and of itself is a sizable issue, it really is more of a symptom of a larger and potentially far more catastrophic problem. Our security. American Express’ inability to support the required number of users logged into their network indicates a series of problems with their network infrastructure. Certain devices that make up their network infrastructure are unable to handle the load associated with everyday users logging in. It is very likely that the issue is isolated to a very small segment of their network infrastructure, but here is the problem: if one of the elements in their infrastructure is incapable of handling the requisite load, the entire infrastructure suffers.

Validating Infrastructure Resiliency

What my experience this morning has told me is that American Express hasn't validated the resiliency of their network infrastructure, particularly under load. What I'd really like to know is what would happen if American Express was subjected to a larger scale cyber attack. Would our personal information be vulnerable to evildoers?When validating a network or IT infrastructure there is a critical interrelationship between realistic applications, security, performance, currency and concurrency:

Realistic applications and security means that network infrastructure is subjected to an appropriate blend of applications and malicious traffic that occur in the real world.
Performance means that the infrastructure is kept under realistic load conditions.
Currency refers to whether the applications and security are the latest versions, while concurrency is the amalgamation of the four previous factors.

The ability to handle all of those factors at the same time is what defines a resilient cyber infrastructure. Here is where American Express has failed. They certainly have tried to validate their network is secure. And it might be. But, clearly they have not validated their infrastructure under stressful performance conditions; otherwise I would have been able to access my account. Thus, they haven't really validated their network is secure. They've missed the concurrency aspect of resiliency.

Naysayers might claim that it is too hard to do all of this validation and be prepared for any conditions. And up until recently, they may have actually been right. But things have changed. One example is our announcement of the BreakingPoint Resiliency Score last week. This concept brings a scientific method to validating cyber infrastructure resiliency and removes the guesswork from the entire process.

Maybe if American Express had validated their cyber infrastructure's resiliency, I'd have been able to submit my expense report.

From the Floor at RSA 2010: Real-World Mobile Network Traffic Validation

Kyle Flaherty — 2010-03-03T14:16:18Z

On the show floor at RSA Conference there is a lot happening and overall the show seems much more well attended than last year. This show, as most of you know, is also a harbinger of news releases and product announcements. Crossbeam, providers of scalable software and hardware platforms, distributed a few pieces of news leading up to the show and at the conference itself. I went over to visit the Crossbeam booth (#545) while at RSA so check out a live demonstration of their X-Series security platform using four BreakingPoint Elite chassis. With this impressive demonstration in the background I talked with Crossbeam's Peter Doggart.

Q. First off Peter, can you provide us with an overview of what Crossbeam provides?

Crossbeam’s X-Series security platform lets customers virtualize third-party, best-in-class security applications and scale them to meet the needs of large, high-performance network environments. Today, more than 900 leading enterprises and service providers, including 10 of the top 11 telecom carriers worldwide, rely on Crossbeam as the underlying architecture for the delivery of security services.

Q. Crossbeam is demonstrating something very interesting here at RSA, can you tell us about what is going on and why?

In working with service providers over the past year, and in particular mobile network operators (MNOs), it has become evident that they are under enormous pressure to meet growing network demands while simultaneously delivering “clean” data pipes.

What we are showing at RSA is proof that our X-Series security platform delivers the world’s fastest firewall performance to meet the needs of mobile operators. Using BreakingPoint Elite, we are conducting a to stress-test the X-Series chassis. We are running a best-in-class application on the X-Series, Check Point Security Gateway R70 Firewall, to clean, inspect and secure the traffic.

This demonstration shows how service providers and mobile carriers can easily scale their network security infrastructure to cope with the next generation of mobile technology, 4G/LTE, under real-world conditions.

Q. You mention “real world” a few times in your answer and in the news release that went out. What does that mean to mobile network operators?

There is a growing gap between what vendors state on their data sheets and what we typically see out in the real world in terms of performance. There are two key elements at play in the real world. First, we are seeing more attacks, which place a greater burden on our security systems and, second, we are seeing smaller payload sizes, especially with the growing number of mobile devices. The result is that mobile operators need to buy and manage a lot more equipment than they budgeted for as the real-world demands are far greater than they ever anticipated. This is not only more costly to them, but it is also a lot more complex to manage.

Realistic tests like this one at RSA validate that we deliver the fastest-performing firewall on the market under real-world conditions which means that we can stand behind our performance claims and mobile network operators can be assured that their X-Series security infrastructure delivers the flexibility, superior performance and high availability required to handle the unpredictability of growing data traffic demands.

Q. How can this type of validation, throughout the industry, not just at Crossbeam, help the overall performance of MNOs?

Crossbeam’s policy is to be transparent when it comes to performance claims. We are doing the opposite of what many vendors do by actually creating tests that provide worst case metrics, not the best case. Take the RSA live demonstration. We are using BreakingPoint to generate 96 byte HTTP packets, which in the real mobile world would be the worst case payload size. At Crossbeam, we want to create some real-world industry guidelines that everyone follows so mobile operators, government and enterprise customers understand exactly what they are buying, and can capacity plan correctly.

Q. I noticed four BreakingPoint chassis in the Crossbeam booth generating the traffic for the demonstration. Why does Crossbeam use BreakingPoint for product validation?

First, we use the BreakingPoint Elite chassis because they can accurately simulate the type of traffic we see in the real world and, second, because BreakingPoint is the only vendor that can push the Crossbeam chassis to its current performance limits.

Q. How has using BreakingPoint helped the evolution of Crossbeam products?

Because BreakingPoint equipment pushes our chassis to its absolute limits, Crossbeam is better able to fine-tune its performance to address customer needs with the assurance that the X-Series can handle their network demands. In the latest release of the X-Series operating system, for instance, we boosted the number of concurrent IP connections we can support up to 10 million, and increased the new connection rates per second to 320,000. These numbers are critical to mobile operators who need to support the growing number of smartphones and other devices, which create more traffic than traditional mobile phones and are nearly always connected. Without BreakingPoint, we couldn’t have confidence in our real-world performance metrics.

Replace Vendor Assurances With Measurable Answers

Kyle Flaherty — 2010-03-02T17:54:27Z

This morning on the floor of RSA Conference BreakingPoint unveiled the BreakingPoint Resiliency Score™, a new approach to objectively measure the resiliency of network and security equipment, putting an end to data sheet speculation. We've all been there, of course, reading a product data sheet that provides data on performance and security of a piece of network or data center equipment. But, we have all reached the point where we basically ignore much of this data.

The reason isn't that the information is fictitious, it is simply not based on real-world scenarios. BreakingPoint is all about real-world simulation, as anyone who reads this blog regularly knows. The BreakingPoint Resiliency Score takes this ability to simulate real-world applications, real-time security strikes and maximum load to provide an objective, repeatable and scientifically measured certification of the performance, security and stability of any network or network device.

The press release that went out this morning had a great quote from BreakingPoint CTO Dennis Cox. It summarizes why this is important for all of us:

“Certification for performance and security is nothing new; in fact, we have come to expect it for everything from our phones to our automobiles. Yet network equipment, which supports our businesses and governments, has no standardized certification for performance and security. Instead we rely on statements made in product marketing literature, which are based on best-case scenarios, not real-world truths. Organizations want measurable answers, not assurances, when it comes to cybersecurity. The BreakingPoint Resiliency Score cuts through all the speculation and confusion and uses a scientific methodology that provides a deterministic and repeatable certification of any vendor claim.”

If you are at RSA Conference stop by booth 1356 and we would gladly show you how Resiliency Score works.

Testing and Validation of Network Security Devices

Dustin D. Trammell — 2010-03-01T16:00:00Z

While catching up on security news and blogs the other day, I came across a blog post from ICSA Labs entitled "Why a Test Lab Needs to be Wary of Commercial Exploit Packet Captures" and thought that it would be a good conversation starter to inform our readers about how BreakingPoint approaches developing test cases for security device testing, our methodology behind why we develop our test cases the way we do, and the thought processes and conclusions behind those decisions.

First, it's important to note that ICSA's blog post is primarily talking about test tools that replay packet captures as their security tests. While the BreakingPoint devices do provide a packet capture replay component, this component is not what we use for security testing. The BreakingPoint devices provide a dedicated security component that execute packaged attacks targeting individual vulnerabilities that we call "strikes". Strikes are not packet captures, and we'll discuss how strikes operate and the benefits derived from them a little later in this post.

Toward the beginning of their blog post, ICSA wrote the following:

"If ICSA Labs were to use one or more exploit packet captures created elsewhere, then we would be effectively vouching for the quality and accuracy of these packet captures. But that is the problem; we cannot vouch for their quality and accuracy."

This is also one of the primary reasons that we do not use packet captures of attack traffic that we have come across in our research. However, we take it one step further and and don't even use packet captures created in-house. We simply don't use packet captures for security testing at all, which brings me to the first subject I'd like to discuss:

Attack Realism

Let's look at what ICSA has to say on attack realism using third-party packet captures:

"ICSA Labs does not know whether the code for each would-be exploit actually works as expected. Even if it did work, we cannot confirm that the would-be exploit was run against a vulnerable system when the capture was made. And assuming it was a working exploit that was run against a vulnerable system, we do not know whether the attack succeeded when the packet capture was made. Also, information in the commercial tool typically indicates at which vulnerability each exploit packet capture is aimed. But again, a test lab has no reasonable way to confirm that. To use the tool in this way ICSA Labs would have to make many assumptions and essentially trust an entity outside of our control."

The BreakingPoint Labs team builds each strike by hand after performing our own analysis of the vulnerability. We have a high degree of certainty that our attacks are correct because we do this analysis and then we test the strikes afterward when possible against the actual vulnerable target. Then, we use these strikes (not packet captures of them) in testing performed using the BreakingPoint device. There are currently two ways to test using these strikes; passing attack traffic through an intermediary Device Under Test (DUT), and sending attack traffic directly to an endpoint DUT, which I'll cover next.

Attack Simulation

"But what happens if the vendor's IPS proxies traffic or alters the content of traffic as some IPS products do? Keep in mind that this is a replayed packet capture, not a live exploit. If the commercial tool with its packet capture of an exploit is run against an IPS that does one of these things and the IPS fails to block the attack, did the IPS really fail? Remember, the IPS modified the traffic on the line."

This is a valid concern when testing an intermediary DUT, and even more so when you're using static data from packet captures. In this scenario, our strikes act as both the attacker and target, and send the attack traffic from one port on our device, through the DUT, and back to a second port. In this way, it's really an attack "simulation" using real attack traffic because we're essentially sending traffic back to ourselves rather than a real target. Because we know what valid attack traffic looks like for each individual iteration of the strike, we know what data we're sending, and we know what the data should look like when we (the target) receive it, if the DUT modifies the attack traffic in transit we consider the attack blocked as it is no longer the attack traffic that we sent and is invalid.

"One-Arm" Strikes

"If the IPS vendor cannot reproduce the issue reported to them by the test lab, then the test lab should be able to confirm its findings in some way. But minus the real attack and actual vulnerable system, that is either a very tall order or impossible!"

Once again, we're in total agreement here, which is why we use real attacks. To the extent possible, strikes that target servers can be run in "one-arm" mode where rather than passing attack traffic through a DUT and back to ourselves, the traffic is sent to the DUT as the attack's target server. In this mode, strikes can be used to actually trigger vulnerabilities on actual vulnerable systems. This is what test houses that use BreakingPoint devices like NSS do to verify that the test cases they are using are indeed valid, even though they are provided by BreakingPoint, their vendor.

Custom Strikes

What if BreakingPoint doesn't have a strike for the vulnerability you want to test? Or what if, like ICSA, you don't trust third party content at all? Even though BreakingPoint provides you with real attacks packaged as strikes, users can easily develop their own strikes. I won't cover this topic in any detail here, as we've already had a three part series (1, 2, 3) on this subject posted to the blog.

Strike Development Goals

Trigger Just the Vulnerability & Use Unidentifiable Payloads

One of the most frequently raised concerns about our strikes is that they contain no active payloads or executable shellcode. This is by design. Sure, network security devices often have filters for well-known shellcode and common payload encoders, and we have specific strike categories to test those specific cases, however if you are relying on the detection of such by your IPS to protect you from actual vulnerabilities then you have already failed. Most network security devices are reactive in nature, and in order to detect a particular shellcode or payload encoder, it must first be aware of it and/or have a filter for it. We know there are payload encoders and shellcode out there that devices are unaware of, so we simulate this by using completely random data as our payloads. This forces the DUT to identify attacks based on the properties of the vulnerability, not by relying on detecting known shellcode or a decoder stub from an encoded payload. We focus entirely on triggering the vulnerability, not actually exploiting it with an operational payload.

Randomness & Uniqueness on the Wire

"ICSA Labs is unwilling to risk its reputation and the trust of end users through the use of packaged exploit packet captures in its testing. All of the exploit packet captures we use in network IPS testing were captured here in the lab by our experts. And in ALL cases, we are in a position to verify our coverage protection test results by running the real, live attack against the actual vulnerable system."

The problem with ICSA's approach here is that you're initially still testing with static packet captures. Consider the scenario where you replay your packet capture of a malicious TIFF file traversing the wire. The IPS under test blocks it, and you mark that as a success. How do you know that if some unrelated parts of the TIFF file are modified, that the IPS won't miss it? How do you know that if you add a whole lot of padding or superfluous structure to the file and move the evil from the beginning of the file to beyond the padding, that the IPS won't miss it? If you're initially relying on packet captures of static attack traffic and then only breaking out the real exploits and targets when something seems amiss or a customer questions your tests, you're not being thorough in your testing.

BreakingPoint's approach to providing these various attack permutations is to identify all of the components of the attack that are absolutely essential for the attack to work and trigger the vulnerability. We identify these values and their upper and lower bound thresholds as well as identify behavioral protocol and process interactions and what combination and permutations of these are valid. We then develop our strikes to randomize these properties as much as possible while still conforming to the identified valid parameters. Further, we randomize as much other data as possible that is not directly related to triggering the vulnerability while still remaining valid for whatever protocol, file format, or other data structure is being used in the attack. All of this context information and the flexibility provided by dynamic test cases such as strikes as opposed to packet captures is the benefit we get from performing the vulnerability analysis ourselves, understanding the operational bounds of the data involved, and developing strikes that launch attacks that actually utilize that knowledge. You can read more about this subject in one of my previous blog posts, File Format Vulnerabilities and Dynamic Exploit Generators.

Evasions

To further the previous point, BreakingPoint can optionally also mutate attack traffic by employing various evasion techniques. When you combine evasion techniques such as IP fragmentation with fragment reordering, using various text encoding methods, and HTTP chunked encoding transmission, among others, with the randomization of the attack traffic that we are already performing as outlined in the previous section, nearly endless permutations of a single attack are dynamically generated which using static packet captures simply can't compete with. Forgive me for quoting a deodorant commercial, but "anything less would be uncivilized." For much more in-depth information on the subject of evasions, please see our recent webcast entitled Harden Security Devices Against Increasingly Sophisticated Evasions or this previous blog post on the subject.

Conclusion

I hope you enjoyed this look into the BreakingPoint strike development and security device testing mindset and found the information both useful and enlightening. Please do follow some of the links above as there is much more information available about the topics discussed.

Proxies

John Repa — 2010-02-19T21:40:47Z

Proxies

Application Protocol Fuzzing

Kyle Flaherty — 2010-02-19T21:21:05Z

Application protocol fuzzing resources

Anti-Malware

John Repa — 2010-02-12T21:13:34Z

Anti-Malware page test

Application Servers

John Repa — 2010-02-12T21:13:10Z

Application Servers

Load Balancers

John Repa — 2010-02-12T21:12:45Z

Load Balancers

Firewalls

John Repa — 2010-02-12T21:12:19Z

Firewalls

IDS IPS

John Repa — 2010-02-12T21:11:49Z

IDS-IPS

Network Management Tools

John Repa — 2010-02-12T21:11:19Z

Network Management Tools

Routers and Switches

John Repa — 2010-02-12T21:10:29Z

Routers and Switches

VPN Gateways

John Repa — 2010-02-12T21:10:01Z

VPN Gateways

Virus and Spam Filters

John Repa — 2010-02-12T21:09:20Z

Virus and Spam Filters

BreakingPoint Labs Blog

Resiliency. Don't Leave Home Without It

Validating Infrastructure Resiliency

From the Floor at RSA 2010: Real-World Mobile Network Traffic Validation

Replace Vendor Assurances With Measurable Answers

Testing and Validation of Network Security Devices

Attack Realism

Attack Simulation

"One-Arm" Strikes

Custom Strikes

Strike Development Goals

Trigger Just the Vulnerability & Use Unidentifiable Payloads

Randomness & Uniqueness on the Wire

Evasions

Conclusion

Proxies

Application Protocol Fuzzing

Anti-Malware

Application Servers

Load Balancers

Firewalls

IDS IPS

Network Management Tools

Routers and Switches

VPN Gateways

Virus and Spam Filters