Resiliency. Don't Leave Home Without It

This morning I found myself very frustrated as I attempted to fill out my expense report. Keeping track of expenses is a painful exercise. You must remember to get receipts, remember to save those receipts, keep those boarding passes and submit the report in a timely manner. Let’s face it. It is an enormous pain. Here is where my American Express card typically saves the day.

I do love my American Express card, it has never let me down and while I do a fairly good job of keeping up with my receipts, sometimes things slip my mind or I lose a receipt. In those cases the first thing I do is login to my American Express account check out all my statements (since I'm so environmentally friendly I've rid myself of those pesky paper statements). It is a great resource to double check that I haven't missed any expenses and it ensures I'm reimbursed for my work-related expenses.

Unfortunately, when I logged on this morning, this is what I saw:

Not only am I unable to access my account, they are unable to write an error message in proper English. Of course, once I saw this message I thought I’d simply hit the trusty back arrow and try again, except this time this is what I got:

“We’re Sorry”? That’s all you have to say about this? And you don't even have the decency to properly format the HTML to display in my browser?

In 2009 American Express did about $24 Billion in revenue with a net income of about $2.1 Billion, yet they haven't figured out whether their network and application infrastructure can handle the load of customers simply trying to check their recent card activity. You could probably argue that it is not a big deal if a handful of their customers cant login to see their statement and pay their bills. But, if their customers are unable to login and pay their bills, maybe those customers forget to “try again sometime” and pay their bill a lot later. If you've ever studied finance, you probably know about the time value of money. All other things being equal, American Express would much rather have my credit card payment today rather than next week, especially considering the financial climate over the last 18 months. Because of their network and application infrastructure failure, they aren't going to get my payment today.

While this in and of itself is a sizable issue, it really is more of a symptom of a larger and potentially far more catastrophic problem. Our security. American Express’ inability to support the required number of users logged into their network indicates a series of problems with their network infrastructure. Certain devices that make up their network infrastructure are unable to handle the load associated with everyday users logging in. It is very likely that the issue is isolated to a very small segment of their network infrastructure, but here is the problem: if one of the elements in their infrastructure is incapable of handling the requisite load, the entire infrastructure suffers.

Validating Infrastructure Resiliency

What my experience this morning has told me is that American Express hasn't validated the resiliency of their network infrastructure, particularly under load. What I'd really like to know is what would happen if American Express was subjected to a larger scale cyber attack. Would our personal information be vulnerable to evildoers?When validating a network or IT infrastructure there is a critical interrelationship between realistic applications, security, performance, currency and concurrency:

• Realistic applications and security means that network infrastructure is subjected to an appropriate blend of applications and malicious traffic that occur in the real world.
• Performance means that the infrastructure is kept under realistic load conditions.
• Currency refers to whether the applications and security are the latest versions, while concurrency is the amalgamation of the four previous factors.

The ability to handle all of those factors at the same time is what defines a resilient cyber infrastructure. Here is where American Express has failed. They certainly have tried to validate their network is secure. And it might be. But, clearly they have not validated their infrastructure under stressful performance conditions; otherwise I would have been able to access my account. Thus, they haven't really validated their network is secure. They've missed the concurrency aspect of resiliency.

Naysayers might claim that it is too hard to do all of this validation and be prepared for any conditions. And up until recently, they may have actually been right. But things have changed. One example is our announcement of the BreakingPoint Resiliency Score last week. This concept brings a scientific method to validating cyber infrastructure resiliency and removes the guesswork from the entire process.

Maybe if American Express had validated their cyber infrastructure's resiliency, I'd have been able to submit my expense report.

Setting The Standard

Next week is my favorite week of the year. It's the Sales Operations meetings held at our headquarters in Austin. Each year we bring the sales people and sales engineers together to review the previous year and preview the year moving forward. More importantly I get to show off.

2009, from all facets, was an incredible year here at BreakingPoint. Sales had an amazing year, with huge growth. Our employee base grew by nearly 30%, much of that being our heavy investment in the security group. We put out 3 major releases and 3 minor releases of our firmware for the BreakingPoint Elite. And our application protocol list now tops more than 100 and our strikes are over 4,300.

This news is certainly exciting, but that was last year. And this is a completely new year and we are ramping up in engineering like you could not imagine. The next firmware release will once again improve the performance of everything from our application protocols, security engine and our SSL. And, of course this is all done without having to replace your blades and at no extra cost. Bet your other vendors don't say that every year.

Next month I'll be putting together a screencast showing you all the features in our next release. I'll save all the juicy bits for then, but here is a teaser of what to expect:

• Five new test labs, including huge strides for mobile networks.
• Changes in the way we are using the NetLogic network processor.
• Switch from using our network processor cores to do SSL, to leveraging the encryption engine on the chip itself (the impact this has on our number of handshakes is staggering).

Last year we changed the way people test their network equipment, this year we will set the standard.

Reminds me of when I worked at Cisco many years ago and Kevin Kennedy (Vice President) would show a slide in which Cisco was compared to other similar companies. There must have been 30 companies listed and at the time 3Com was below us, Lucent ahead of us and all the way at the top were companies like HP. At the time HP was 10x the size of Cisco. Today, Cisco is tens of billions of dollars ahead of HP, with a third of the employees. 

Every year that presentation showed Cisco passing yet another company. We have the same chart for our industry and the same goals, and some companies were ahead of us at the beginning of 2009. During 2009 we passed four of them and this year we will pass four more. And one day, like Cisco, we'll be at the top of everyone else's list.

NOTE: Sometimes Cisco didn't pass a company, the company fell. I'm seeing a lot of that lately, maybe I should send some flowers.

Data Center Planning and Consolidation

Data center planning and consolidation resources

Open Letter to Twitter: Can We Help?

Dear Twitter,

In December 2006 I jumped onto Twitter, not realizing the impact it would have on my life. Twitter helped introduce me to Pam O’Neal (@poneal) at BreakingPoint (@breakingpoint) and subsequently move my family from Boston to Austin to work here. Now, each day at BreakingPoint, we use Twitter (30%+ of our employees are registered) to communicate with our community of server load testing professionals and network and security engineers, as well as communicate amongst one another.

As a company, BreakingPoint recognized the potential of Twitter early and in 2008, even added a product feature to test the ability of network devices' and application servers' to handle the load of Twitter traffic. Our team has worked for decades developing the latest in networking and data center technology and they know first-hand how hard it can be to sustain a reliable network and service. It is why they created a better network and load testing solution.

Yesterday, when Twitter was down due to a "bug triggered by an edge case in one of the core services", I thought about how important Twitter had become to our business and me. I watched the predictable posts complaining about the fail whale and it hit me; rather than throwing criticism, I would be best served getting my hands dirty and helping with the problem. An idea surfaced, which I talked through with our CTO and co-founder Dennis Cox (@denniscox), and the green flag was waved.

BreakingPoint wants to help Twitter by providing the use of its server load testing product and wicked smart folks (sorry, the Boston still in me) to help assure the resiliency of your company's network devices, servers and overall data center infrastructure. We want to provide the use of our BreakingPoint Elite, combined with a team of experts, many of them industry-recognized resources in security, Ruby, software engineering and load testing. As your company will see, we can provide server load testing on the scale a service like Twitter would need. I can have someone from our San Francisco office meet with your team today if needed (and I can be on the next flight).

BreakingPoint is a huge fan of Twitter and hopes that it can help! I've included the Twitter profiles for some of our team below. We are all looking forward to working with your company.

		denniscox - Dennis Cox CTO and founder of @breakingpoint			tmanning - Todd Manning Author of @breakingpoint Twitter testing protocol
		todb - Tod Beardsley @breakingpoint "protocol monkey" (his words, not mine)			poneal - Pam Oneal Vice President of Marketing @breakingpoint
		mikehamilton - Michael Hamilton Product Marketing @breakingpoint			kkuehl -Kirby Kuehl Writes protocol (de|en)coders in C/C++ @breakingpoint
		jgstroud - Jonathan Stroud Hardware design @breakingpoint			druidian - I)ruid @breakingpoint Labs
		chrisfenton - Chris Fenton @breakingpoint--West Region			KyleFlaherty - Kyle Flaherty Communications for @breakingpoint

 Many thanks for everything your company has provided our community,

@kyleflaherty

UPDATE: I'd be remiss if I didn't also link to @busterbcook and @rhythmx, two more great BreakingPoint Tweeters!

How To Create Custom Strikes in Testing: Finding, Validating and Understanding Vulnerabilities

UPDATE: To download the entire how-to guide, which includes all three parts, simply head over to our how-to guides section.

According to the National Institute of Standards and Technology’s National Vulnerability Database, there are currently more than 38,000 known security vulnerabilities. Not only is this number growing, but there are thousands of additional unknown vulnerabilities being discovered and written each day. Although BreakingPoint leads the testing industry in both security and application coverage, providing 4,200+ security strikes and 80+ application protocols, we also recognize the need to create customized vulnerability exploits. Last month we introduced the Custom Strike Toolkit to provide the ability for our users to create and generate any strike they want during testing (we provide a similar Custom Application Toolkit).

In order to detail how the Custom Strike Toolkit works we put together a “How-To” guide that not only describes the step-by-step process necessary to utilize the BreakingPoint Custom Strike Toolkit, but uses a real-world example of a zero-day exploit for the Internet Information Services (IIS) FTP service.

Today I look at part one of our "How To Create a Custom Strike" guide, examining how to find, validate and understand a vulnerability. Soon we will look at exploits versus vulnerabilities and finally finish it off with a detailed look at how to build and generate your own custom strike.

Finding and Validating the Vulnerability

For the purposes of the how-to guide, we will be using an FTP denial of service vulnerability, CVE-2009-2521, published by Nikolaos Rangos in September 2009.  Utilizing a specially crafted ‘ls’ command, an FTP client may cause the IIS service to crash.

The first step in the process is to validate that the vulnerability exists, and is exploitable. In this case, IIS 5.0 was installed on a standard PC platform running Windows XP.  The screenshot below illustrates the Microsoft IIS configuration utility.

Figure 1: Microsoft Configuration Utility

Once the FTP service has been installed and configured, the installation must be validated. The following screenshot shows the initial login screen illustrating that the FTP service is up and running.

Figure 2: Connected to Local FTP Service

As noted in the vulnerability publication, the only thing required to exploit this vulnerability is a directory that may be read by a user. To illustrate the severity of this vulnerability, an “anonymous” FTP user will read the “pub” folder in the root directory.

Figure 3: Anonymous User Successfully Logged In to IIS FTP Service

The next step in the exploit process is to issue the appropriate FTP command to trigger the vulnerability.

Figure 4: Specially Crafted ‘ls’ Command

Once this command has been issued, the damage has commenced. The following screenshot illustrates the output from the FTP client indicating that the server aborted the connection.

Figure 5: Many Directory Listings Followed By Server-side Close

In order to fully verify that the FTP service has been exploited, a quick look at the IIS management console shows that the service has stopped.

Figure 6: IIS FTP Service Crashed Due to Exploit

Understanding the Vulnerability

Now that we have successfully validated that the vulnerability does in fact exist, we now have to understand what the network traffic looks like that would actually exploit it. The following Wireshark stream flow shows the actual ASCII characters transmitted over the wire.

Figure 7: ASCII Output from Network Capture

Upon completion of the TCP handshake, the IIS FTP server responds with a “220 Microsoft FTP Service” message. The client must now authenticate, in this case, as an anonymous user with the “USER anonymous” command. The server responds with a “331” code allowing anonymous access to the FTP site. The “PORT” command then follows with the appropriate successful response. The key packet in the transaction happens when the “NLST –R p*/../” command is issued. A “150” response follows and the data channel starts streaming the directory listings recursively until the service dies. Once this happens, the server terminates the connection with a TCP Reset.

Figure 8: All Relevant FTP Control Channel Packets

Figure 9: All Relevant FTP Data Channel Packets