Setting The Standard

Next week is my favorite week of the year. It's the Sales Operations meetings held at our headquarters in Austin. Each year we bring the sales people and sales engineers together to review the previous year and preview the year moving forward. More importantly I get to show off.

2009, from all facets, was an incredible year here at BreakingPoint. Sales had an amazing year, with huge growth. Our employee base grew by nearly 30%, much of that being our heavy investment in the security group. We put out 3 major releases and 3 minor releases of our firmware for the BreakingPoint Elite. And our application protocol list now tops more than 100 and our strikes are over 4,300.

This news is certainly exciting, but that was last year. And this is a completely new year and we are ramping up in engineering like you could not imagine. The next firmware release will once again improve the performance of everything from our application protocols, security engine and our SSL. And, of course this is all done without having to replace your blades and at no extra cost. Bet your other vendors don't say that every year.

Next month I'll be putting together a screencast showing you all the features in our next release. I'll save all the juicy bits for then, but here is a teaser of what to expect:

• Five new test labs, including huge strides for mobile networks.
• Changes in the way we are using the NetLogic network processor.
• Switch from using our network processor cores to do SSL, to leveraging the encryption engine on the chip itself (the impact this has on our number of handshakes is staggering).

Last year we changed the way people test their network equipment, this year we will set the standard.

Reminds me of when I worked at Cisco many years ago and Kevin Kennedy (Vice President) would show a slide in which Cisco was compared to other similar companies. There must have been 30 companies listed and at the time 3Com was below us, Lucent ahead of us and all the way at the top were companies like HP. At the time HP was 10x the size of Cisco. Today, Cisco is tens of billions of dollars ahead of HP, with a third of the employees. 

Every year that presentation showed Cisco passing yet another company. We have the same chart for our industry and the same goals, and some companies were ahead of us at the beginning of 2009. During 2009 we passed four of them and this year we will pass four more. And one day, like Cisco, we'll be at the top of everyone else's list.

NOTE: Sometimes Cisco didn't pass a company, the company fell. I'm seeing a lot of that lately, maybe I should send some flowers.

Cisco Security Agent Exits the Market?

This morning, Ron Gula tweeted a link regarding the possible discontinuation of Cisco Security Agent (CSA). Gula, the CEO/CTO of Tenable Security, pondered whether this was the first of many Cisco security products to be discontinued. While I think he may be right in that regard, I was hoping CSA remained alive. Patrick Ogenstad wrote the actual blog post in Network Lore to which Gula referred in his tweet. It's a wonderful article and I agree with most of what Ogenstad writes, with the exception of a sentence in the last paragraph:

"Perhaps Cisco is the wrong vendor to have this specific product in its portfolio, and perhaps someone else will buy it."

While I do hope someone picks up the product, I actually think Cisco is the best company to own CSA. This was its trojan horse into the desktop to disseminate a whole host of other products and services. Perhaps even TelePresence?

Cisco has the manpower, a technical sales force and a strong technical support organization. Those are key factors, in my opinion, to make CSA successful. CSA reminds me a lot of Network Flight Recorder (NFR), acquired by Check Point in 2006. The products are (were?) both extremely powerful. You could do most anything you wanted and neither product required constant upgrading. The general feedback on both, however, was that they were complicated and required knowledgeable people to set them up and get the most out of them. I really dislike that as an argument for their demise: "I'm too lazy to read the manual and do a few Bing searches, Mr. Vendor. Just make it all auto-magically happen for me.".

Sorry, buddy, but networking doesn't work that way and network security definitely doesn't work that way. It's a detail-oriented profession and if you are not detailed enough to understand the difference between UDP and TCP, get out of networking. You are not doing anybody any favors by judging everything on presets and defaults. You sir, are the type of person being mocked in beer commercials.

We see this all the time in testing. Vendor A has built a new content-aware firewall, and its QA team tests the product using a bit blaster to see how many UDP packets can go through at any given packet size and any second. When does that happen on a real network? Never. The QA team is doing what they did in the past and is now simply being lazy. They are not helping the product succeed in the real world. Here is a suggestion to anyone with a content-aware firewall, test with some actual content, and you'll be surprised by the results.

As I noted in my last blog post, network administrators are once again failing to secure their networks properly, whether it's failing to update their routers and switches with the latest Cisco patches or not deploying solid security solutions such as CSA. It leads me to a couple of important questions for the peanut gallery: Why is CSA leaving the market (or is it)? And what could Cisco have done to save it?

Oh, one last thing, what are you doing to save your product? God knows I hope you're testing.

Cisco Becomes The Weakest Link In National Infrastructure Security

Last week Cisco released patches in their semi-annual security announcement. The publication includes 11 advisories that address 12 individual vulnerabilities. Ten of the advisories address vulnerabilities in Cisco IOS and one advisory addresses a vulnerability in Cisco Unified Communications Manager. Together these can affect routers and switches that not only use the Cisco Unified Communications Manager, but any device relying on the Cisco IOS operating system. To put it bluntly, this means a ton of devices critical to any network, and these vulnerabilities leave businesses and government agencies exposed to a barrage of attacks including denial-of-service (DDoS) or policy bypass.

Much has been written about the announcement of the vulnerabilities. However, details are lacking and there are more questions than answers. This lack of information leads me to believe Cisco does not take security seriously and continues to not know how to work with the security community. Considering the lack of details and opinions, I thought I would provide a few of my own.

1) Twice A Year Is Not Enough

The number of vulnerabilities patched by Cisco is not the issue. It is the potential danger these vulnerabilities pose. One of the IOS vulnerabilities allows unauthenticated attackers to bypass access control policies when the “Object Groups for Access Control Lists (ACLs)” feature is used. Your company is most likely protecting your critical components by leveraging ACLs, now imagine they are no longer in place. The human resources database with all that W-2 information? Hackers now have your salary, your direct deposit account, your medical history and of course your social security number. To make matters worse, replace that HR database with our government’s nuclear secrets; don’t you think Iran is aware of the Cisco vulnerabilities?

Scary stuff, for sure, but how long has the vulnerability been around and recognized. The answer is unknown. The only fact we have is that each of these eleven vulnerabilities may have been around for at least six months. That is an eternity in the security space and has given hackers too much time to walk in through an open door.

Microsoft is often a punching bag when it comes to vulnerabilities and it is sometimes warranted, but let’s be honest, the company does a good job of patching issues on a regular basis. With Microsoft, you know that you are going to get a patch each month and important details that help you make an informed security decision. Cisco should examine its patching schedule in light of the September 24th announcement; every six months is not acceptable.

2) Updating Routers and Switches is Now Critical

You can never diminish the importance of a switch or router to your network infrastructure. They are the core to any network whether in a home, a large Enterprise or the Federal Government. If one fails you know it. However, if a vulnerability let’s people through due to a hack do you know it? While everyone remembers to patch their Mac or Windows laptop, how often do they patch the router, firewall or switch?

To see how up-to-date folks are with their Cisco firmware I ran a quick test. During a 1-hour scan of the Internet I found 420 responding systems and NONE were patched with any fixes from this cycle or the last. That means 420 systems, at a minimum, are susceptible to a years worth of vulnerabilities.

Microsoft had enough of people not patching and now it force feeds the patches. While I’m not a fan of that solution, it does work. Cisco needs to apply the same method to its products. It is irresponsible for Cisco to run its business in a way that could cause mass disruption to critical network infrastructures including government and military services.

Cisco is not the only one to blame in this mess, the people responsible for getting their routers, switches and other network equipment up-to-date also must be held accountable. How many of you updated with the patches on September 24th, the day of the announcement? The quick scan I did is telling me not many. Kelly Jackson Higgins of Dark Reading put it best, “The dirty little secret about patching routers is that many enterprises don't bother for fear of the fallout any changes to their Cisco router software could have on the rest of the infrastructure.”

3) Testing, Testing, Testing

In this case we have a great example of why every network device needs to be realistically tested under a variety of scenarios, both security and performance driven. Obviously, testing must occur at the NEMs level throughout the product lifecycle, but the enterprise must also test this equipment before it is deployed and after updates like these are made. Having the ability to quickly test equipment and the network after making updates is critical.

There is no room for excuses anymore. We have been able to become more adept at updating and testing equipment and software that are given more regular patches. Just look at how Microsoft Tuesday has become a habit. Other vendors have realized that this approach, ultimately, is better for everyone. I would encourage manufacturers of any network equipment to do the same.

The reason this is important is because the United States is currently fighting in two wars, heavily dependent on network technologies. The Department of Defense and other military agencies have concluded that the next major war will be waged, in great part, in cyberspace. If Cisco and other vendors guilty of the same security concerns do not get their act together it will be a war we cannot win.

Until March 24, 2010, when the next Cisco bulletin is due.

Finding Bigfoot

Recently, one of our folks on the International team had a request for web mail protocols for use in testing; Gmail, Yahoo, etc. Pretty standard stuff, so Mike Hamilton, Director of Product Marketing, made the applications via our Custom Application Toolkit for the customer. When we discovered what the customer was planning on doing with the application protocols we decided to take it to the next level.

This customer wants to search for keywords in e-mails across webmail protocols. The ability to generate traffic with and without certain content is something we have been able to do with our product since day one. You simply create two (or more) Super Flows, assign them to an application protocol and use the weights to vary how much of any one Super Flow shows up. However, we thought it would be great to give the users a few more options than the standard user supplied parameters or random data.

Tod Beardsley, BreakingPoint Labs security researcher, decided to rewrite Mike's original webmail protocol with a slightly different angle. Check out the screenshots below.

[Click Image for Expanded View]

[Click Image for Expanded View]

As you can see you get the standard e-mail controls (to, from, subject, etc.). But you also get the ability to select a language dictionary from which to generate the random words. You can specify the minimum and maximum number of words to generate along with a list of keywords. The Super Flow also supports random generation of file attachments or user specified file attachments. You can even control the size of the attachments.

In this case you create two Super Flows, one with the keywords enabled and one with them disabled. Put both of the Super Flows in an Application Profile and assign that to an Application Simulator component and off you go. Remember to use the weights to assign how often you want to see one traffic over another.

My favorite part is that all this functionality was added via a Strike Pack. No software release. All of our customers get this new functionality along with the latest applications and security strikes that come out in the Strike Packs.

Don't Put All Your Eggs in One Carton

First, some quick definitions:

An Exploit is a working version of code that takes advantage of a vulnerability in a product. The exploit could crash the product, or it could make it open for remote access (for someone to get into your computer).

A Vulnerability is a flaw in a product. A well known flaw in programs is a buffer overflow. A buffer overflow is when a program tries to give more data then the program it's talking with can handle. In layman's terms, let's say you try to put 13 eggs in an egg carton, one's going to fall and crack. You overflowed the carton. The carton can only fit a dozen eggs and when you put too many eggs in bad things happen. Let's go with the egg analogy for a bit.

Now imagine you work at the market and your job is to check every carton of eggs. So you decide if you see 13 eggs then you will throw away the carton. This works okay - but you are checking for the exploit not the vulnerability. If I put in 14 eggs, you won't detect a problem, since you are only looking for 13 eggs. A true vulnerability detection would be "if there are more then 12 eggs".

Most security vendors say they are vulnerability-based not exploit-based, however it is a rare vendor that can do mostly vulnerability filters. Luckily there is an easy way for you to prove this fact. Look for filters that are enumerated (exploit.1, exploit.2, exploit.3, or exploit.A, exploit.B, exploit.C) or go social media on them and search their blogs.

After looking around for a bit you'll quickly find that vulnerability filters are a rare find. The reason vulnerability filters are rare is because they generally require intimate knowledge of how the program works internally (source code). On the other hand, exploit signatures are trivial to create. If I know what the packet looks like, then I just match on that packet. An exploit is generally a fixed packet(s), which makes signatures trivial to write. Since they are specific, they do not provide as effective security as a vulnerability filter.

Certain companies literally do not know how to write vulnerability filters. The two main reasons are either the product they develop can't do the complex matching needed for a vulnerability filter or they don't have the skillset to create them. For example, in late October W32.downadup was released (popular name Conficker), everybody released signatures for it fairly quickly. In December another variant of the exploit came out. It bypassed pretty much every antivirus and IPS system. Those vendors then released new filters, and by mid-March two more variants were found, which bypassed all the same vendors again. Variant E (number 5 if you are counting) came out in April followed by the subsequent release of filters from most of the vendors. What does this tell us? First and foremost, people are not writing vulnerability filters. And secondly, you are not getting good protection out of the exploit signatures. To write an exploit filter (signature) you need to have a copy of the exploit in the wild. That means the exploit is already out and you are doing damage control not prevention.

Let me pick on some vendors for a minute. I picked these three because to be honest, they have the BEST websites for finding out this information. If you are a customer of the below, be happy, your vendor isn't trying to hide anything from you and they each have great websites to boot.

For those not listed, don't think it's because you made a vulnerability filter, it's just I didn't need more then three examples. F-Secure, Cisco, 3Com and I'm sure others all have similar information online with multiple releases of the filters to block Conficker.

Why is this all important?

You want a vulnerability filter. Every vendor will tell you that the perfect solution is a filter that blocks the vulnerability. With a vulnerability filter, you remove the need to update every time a new variant comes out. You block the bad stuff every time, no matter what.

BreakingPoint has the tools to test vendor filters to see if they provide adequate protection against threats. We do this by using a pseudo random number generator (PRNG) which you initialize with a value. This value is generally referred to as a seed. The PRNG uses this seed to generate random data in the parts of the exploit that doesn't matter - for example the first 12 eggs. Instead of sending 13 eggs over and over again, it will generate 14, 16, 204, etc., eggs.

The PRNG however is not actually random - it's pseudo random. If you look up pseudo in the dictionary it will say "not genuine; sham". That's right, it's not random - and that's good. We want you to be able to pick a number and generate the attacks (or application data) the same way every time. Meaning, if I choose seed "4" it will generate [20, 21, 2045, 657,...]. If I choose "5", it will generate [516, 352, 6246, 2657, ...]. This way I'm generating all variants, but in the same way. If you want something repeatable - always use the same seed.

It's possible to write signatures that are exploit-based but are smart enough to block future variants, and BreakingPoint testing tools allow you to prove it during product bakeoffs. But more importantly you can find out if their engine can handle any vulnerability filters.