Denial of Service Attacks

Recently there was a Dark Reading article on a new type of TCP DOS attack, of course the authors have learned their PR lessons and it's all hush-hush until the next big conference or product launch. Of course that leaves us all guessing as to what the attack could be. So I thought I would cover some basics about Denial of Service Attacks for people interested.

Normal TCP Connection
Client             Server
SYN       ->
              <-       SYN-ACK
ACK       ->      
DATA      ->       
               <-       ACK
FIN         ->
               <-       FIN-ACK


SYN Flood
SYN Floods are one of the most basic Denial of Service attacks. The basic way they work is the attacker sends a large number of TCP requests to the victim (a server or network device). This ties up all his connection entries and he can no longer process incoming requests. To jump down to a even more basic level -

To make a connection to a server a number of things have to happen, one of them is generally a TCP connection. The connection involves a three way hand shake. The first is called the SYN, which the client sends, then the server says SYN-ACK - meaning I read you loud and clear. The next stage (third step in the three way handshake), is the client saying okay - we have a connection or ACK. Now we are in established state. So if my devices supports 5,000 connections then and you sent 5,000 SYNs I can't receive any more connections. So I have denied service. Make sense?

SYN Flood
Client             Server
SYN     ->
            <-       SYN-ACK

SYN     ->
            <-       SYN-ACK

repeat

This technique has had a solution since 1996 (I know, I have the first patent on it). There are so many ways to solve this problem. One is to make a list of connecting and established connections. When the connecting list is full you throw away the oldest of those in the connecting state or favor something existing in the established list. Hence, clients that you have already spoken to are more trusted then those talking to you the first time. Again, dozens of solutions to the SYN Flood problem.

Connection Attack
A connection attack is basically the same thing as Denial of Service, except in this case I complete the three way hand shake. Now, I've gone onto your established list and eaten up your entries. You can also do this (but it's painful) by focusing on a single IP address and eating up every possible port number (16 bit value), but that requires a lot of connections (~65k) and generally speaking big sites use load balancers and dozens of IP addresses, so it's not useful.

Connection Attack
Client             Server
SYN     ->
             <-       SYN-ACK
ACK     ->      

Other forms of Denial of Service
You can play with TCP stacks to create other types of Denial of Service issues. Sending resets can interrupt connections or playing with the Window Size. I could make the victim think that I'm on a really slow link and it will leave the connection open forever. This works great on network devices in particular because they tend to shortcut the TCP fields to save on memory and speeding up their data searching (hashing, lists, trees, etc).

I did a recent test the other day on making the victim think I am on a slow link. I found devices with specific field size for the TCP header were susceptible to having this type of attack more often. For example, a Windows 2003 Server took 10 packets per connection, while a popular network firewall took 4 packets per connection. This type of attack is intensive in the number of packets (not large one's mind you, but still enough) and it fools even the best load balancers and firewalls. It definitely can bypass SYN-Cookies without a problem.

A good example on TCP Window Size is:
"In theory the TCP window size should be set to the product of the available bandwidth of the network and the round trip time of data going over the network. For example if a network had a bandwidth of 100 Mbits/s and the round trip time was 5 msec, the TCP window should be 100x10^6 times 5x10^-3 or 500x10^3 bits (65 kilobytes)" - so If I want to go slower?

This type of attack is really interesting because an article in Dark Reading recently talks about some folks that think they found the next DOS attack, no details have surfaced so who knows, but my guess it's this type of attack.

Application Denial of Service
Another Denial of Service is to run a CPU intensive program/script/query on the device you are after. For example, if I run a select query that is highly complex on a database I may be denying service to users that want to query as well. Run enough of these and you can starve people out. You could also send malformed data to the program, if the program crashes or runs really slow you deny service. A recent favorite we saw was VoIP related, some compression algorithms are more CPU intensive then others. By selecting a horribly slow one and creating a dozen calls, all voice calls start to suffer in quality - or worse can't happen at all. So you can see Denial of Service isn't just TCP related and there are dozens of ways to make it happen.

Network Device Denial of Service
Network equipment also suffers from Denial of Service problems. How many ARP entries can a device handle? How big is the connection table? Does it have a control path for routing protocols? If I send routing protocol packets, like RIP updates does it slow down packet processing? What if I send a flood of them? If I create a worse case packet for an IDS that leverages Regular Expressions, does it miss traffic? If so, I can sneak my attack by unnoticed.

So it seems that Denial of Service that will end the Internet is the new craze. To be honest, Denial of Service attacks aren't rocket science, to find one you just need to look, it really is that easy. It all comes back to testing; if you test the worst case scenarios in your product, you will come up with simple solutions to protect from these and minimize the possibility of a Denial of Service attack.

BreakingPoint LiveLook: Talking Customer Support

Caught up with Greg Griffith here at BreakingPoint and chatted with him about his role in support and what that entails. If you are a BreakingPoint user you may have already worked with Greg to help you with our testing solutions.  Greg also chatted with us about his interests outside of providing support, including lending his film making skills to help us produce our test methodologies and his latest hobby...model rocketry.

BreakingPoint LiveLook: Engaging the Community

Today I decided to talk to Kyle our Director of Marketing at BreakingPoint. Kyle just joined the company a few months back and I really enjoy giving him a hard time for some unknown reason. We knew when we were starting BreakingPoint we wanted to market via the 'net and not traditional sources like magazines and tradeshows. Not to say we won't do those sorts of things, it was just that we wanted to be more about being online.

The online test methodologies, the screencasts, the large amount of product data available online, even this blog are examples of us focused on our communication over the 'net rather then traditional channels.

BreakingPoint LiveLook: Enter the Heat Chamber

This is where you can come to see what is going on inside of BreakingPoint.  We have our handy cameras and we are sitting down with some of the folks to talk about what they are currently working on, the network testing market and much more.

August 4, 2008: Hardware Engineer Greg Singleton discusses "The Heat Chamber":

 

Traffic Analysis

As I may have previously mentioned, I love traffic analysis reports. I have several locations where I grab information and a script that combines them and collates all the data for me on a monthly basis. One of the sites I get my data from is Internet2.

The data shown on the Internet2 backbone is vastly different in terms of application percentages (less P2P for the most part). However, all the other numbers line up with almost all other sources. Check this out; 90% of all packets are between 1401 and 1500 bytes. Makes you wonder what your test rig is set to? It also reminds me of our conversation about IMIX, the description from CAIDA's site again is: "IMIX derives from analysis of NLANR traces and is tri-modal (e.g., 58% at 40 bytes, 18% at 576 bytes, and 23% at 1518 bytes)."

Netflow on Internet2 says 100 byte to 1400 byte packets are 9.15% and sub 100 byte is 0.46%, whereas IMIX is 18% at 576 bytes and 58% at 40 bytes. Again - why are people running IMIX? Crazy. After all, IMIX is based on real data, just like the data I'm provided, it's just I'm using current data. Unless your sending your device back in a time machine, I suggest you stop using IMIX.

My final point - how many applications are you testing - because Internet2 is a small backbone (in terms of users) yet according to the netflow data they have more than 40 applications running that they can measure.

P.S. 41.33%  of the traffic is unidentifiable :) That's a big piece of data that's missing.

Do the Math

I love math, although the funny thing is, I was really bad at it going through school and had to have a math tutor in high school. These days, I actually consider going back to school to study math from time to time. Talk about a change. What intrigues me the most is the ability to use math to come up with clever ways to solve problems. This became apparent to me since I actually honed my math skills by majoring in Music Composition.

During freshman year my professor played a recording of a piece of music, roughly 5 minutes long. He would play the piece once and it was our job, after the piece was played, to write the general score down. The challenge; we could only write for the first 30 seconds and the last 10 seconds of the recording! Five minutes of music with 40 seconds of writing to get the general score? Impossible we all said, it would take hours to transcribe 5 minutes of music with this many parts. We would need to listen to it over and over again to get it right...we all stared at him like he was crazy.

By my Junior year I could do it with only 10 seconds of writing at the beginning of the piece and a second at the end. It was dead simple, because it was math. What you need is the data that starts it and ends it. You can determine the path from those two points. [NOTE: This applies mostly to classical music]

The same thing applies to network equipment testing - I see too many engineers test their equipment without data. What does traffic look like in the real world? Sometimes a customer will quote an RFC, but the RFC is 10 years old and the standards, while not documented, have changed completely. If you don't know what the network looks like, even just 10 seconds of the beginning and 1 second at the end, you can't test your product correctly. After all, just because you know networks have HTTP in it, doesn't mean that's the only thing running. To complete the analogy, it would be like listening just for the drums to figure out the tune; you need to hear/see the whole thing for a good picture.

This resonated with me again when I saw a competitor just announced P2P support. Just announced? P2P is over 30 percent of network traffic and some test tools are just adding it? Heck some don't even have it yet. If you don't have the data, things will never add up.

DPI for Bandwidth Shaping? Where's the news?

There seem to be several conversations happening on service providers using DPI to throttle traffic.  My reaction?  What's the big deal?

This is another case of the media looking for a story that isn't there. Service providers of all sorts have been using bandwidth shaping [DPI] since the early 90's. Technologies such as Asynchronous Transfer Mode (ATM), had bandwidth shaping in their specifications and ATM was one of the core transports of service providers. UBR , CBR, VBR anyone? Why is this a surprise?

It reminds me of the Global Warming debate. Al Gore has been talking about it since the 70's - yet when he was vice president nothing got done about it. Not his fault, he's just one man. But if he were vice president now with Global Warming such a hot topic, pardon the pun, the media's pressure would bend the will of the people to solve the "issue".

Just yesterday, based on some guys figuring out what statistical packet analysis can get you, everybody went crazy again on this topic. Statistical packet analysis was around WELL before this paper. A bunch of guys from MIT started Mazu, which built a company based on this concept. Arbor is another company that leveraged this, well before Mazu did. So what's the hype? I guess it's a slow news day.

When Looking Bad Makes You Good

Aren’t you afraid you’re going to make everybody look bad?

I often get asked this question when I am demonstrating the product, discussing test results, or explaining how my product works to the folks who are testing network devices. Why is it that everyone is afraid of looking bad? It seems like the whole network industry has been immersed in this fear for so long that they’re willing to do anything to look good.

Test gear companies are aware of this fear, so they’ve built their products to favor specific network equipment vendors. At the end of the day, test gear companies will do anything to ensure that their customers’ products don’t look bad – even if it means tweaking their test gear to work better with certain network equipment. My problem with this is: how will network equipment manufacturers ever know if their stuff really works if they’re not breaking their equipment?

One of the reasons why the companies I’ve been with have all been successful is because we focused on breaking our equipment. I wanted to find test gear that showed me the problems with our products; I wasn’t particularly looking for test gear that would help improve the specs on our data sheets. Obviously, that’ s not a bad thing, but it’s not the most important thing. The most important thing for us was finding the issues with our products before they were released into the field. After all, the field is the worse place to find an issue, the costs are a lot higher and the debugging time is infinitely longer.

So, my answer to the question, “Aren’t you afraid you’re going to make everyone look bad?” is no. I’m afraid of what will happen if someone installs network equipment that wasn’t properly tested. I’m afraid something terrible will happen, like a CEO not being able to access their e-mail or giant wombats overtaking the world.

What do you use your test gear for?

The Problem With IMIX

Andre Gironda was kind enough to post a comment regarding IMIX to yesterday's blog post, "The World is Lacking Background Traffic." Basically Andre writes that IMIX might be outdated but it's out there. This was something that I was planning on talking about in the future, but since I got the perfect setup, I'll do it now. I have no issue with IMIX, but I do have an issue with thinking that it can provide you good data for how a device will work, performance wise, in your network. Andre is right, it's a bit outdated, in fact I believe it's 7 years old this month and the NLANR (National Laboratory for Applied Network Research), who provided it, had it's funding dry up 2 years or so ago.

For those that don't know what IMIX is, I will steal from a description I found on CAIDA's site: "IMIX derives from analysis of NLANR traces and is tri-modal (e.g., 58% at 40 bytes, 18% at 576 bytes, and 23% at 1518 bytes)."

So to prove this out - I took a Juniper Networks device that is content-aware and ran three benchmarks. It was fairly easy since this device used to be in my network! The three benchmarks are as follows:

1. Setup a test to run IMIX at 20 megabits (yes, my device is wimpy);
   
2. Setup a test to run BreakingPoint's Application Profile for Enterprise Traffic (using release 1.2);
3. Monitor it via SNMP all day to see when I started dropping traffic (yes I have a 10-megabit pipe to the Internet);

(It should be noted that I knew the device couldn't handle our 10-megabit pipe to the Internet, I know this because we had to upgrade it to a larger device after we changed from a T1 to a 10-megabit pipe).  Here's a graph showing the results:

As you see, the IMIX told us the device could perform 18.9 megabits per second without packet loss. The Enterprise Application Profile clocked it at 4.6 and the Actual was fluxing between 4.7 and 4.9.

The World is Lacking Background Traffic

When I inherited the system test and quality assurance (QA) groups, along with my other responsibilities as Director of Engineering at my last Company, one of the things the QA engineers would do to make my head tilt was to configure all the tests with background traffic. Why would that make my head tilt? It was the background traffic they used; a popular traffic generator running layer 2 traffic with random data. I immediately said, "That's not background traffic - that traffic doesn't exist on any network". The engineer in charge of running the "background equipment" said that he didn't want the traffic to interfere with the test. Of course my next question was, "Then why run the traffic at all? If the background traffic is not to interfere with the traffic and it's not ANYWHERE near realistic - what's the point?"

I wanted realistic IP traffic the next time I visited and a week later I saw he had IPv4 traffic with random data being sent. Granted, this is a huge improvement over random Ethernet frames, but a long way to realistic traffic. I acknowledged this was a good improvement but wondered if he had any peer-to-peer traffic? The QA engineer asked why and I explained the recent feature we added for traffic shaping peer-to-peer traffic. He explained that this would get in the way. I suggested that he should be running P2P traffic all the time, "Think about it; if we have traffic shaping off - peer-to-peer traffic IS the background traffic and not the primary traffic." I told him to have 20 percent of the traffic peer-to-peer by next week.

Next week I come by and sure enough 80% of the traffic is IPv4 with random data, and 20% is a P2P datagram. Now, first knock, he didn't get the gist of what I was saying; he didn't take a look at all our features and pull in different traffic sources, he did the literal translation. Oh well.  However, and here is the second knock, there were 5 "streams" of traffic being sent. A stream being an instruction to the popular traffic generator to send out traffic. So what we had were first four packets of IPv4 with random data, then a single packet that is part of a P2P stream, then repeat. My not so surprising reaction, "Hey this isn't P2P, it's a packet."

Fast forward one week...the QA engineer is there with the rep for the test equipment. The sales engineer for the test equipment vendor has his head held high and says "I have a very complex TCL script here that will recreate the P2P flow exactly!" I pull out a copy of Ethereal (now Wireshark), take a look and the traffic is the same, except the IP address repeating over and over again. "This isn't background traffic, it's a single P2P flow played over and over again."  Pause.  Finally the SE for the test company says, "What do you expect, actual traffic?".

Yes!

 
We need to change the way people test, and it's a fundamental shift. After all, your product is going into a network with an average of 30+ different applications on the wire, don't you think your product should handle that traffic correctly? Background traffic needs to be real and this, along with other similar stories, led to the creation of BreakingPoint.  We understand the need for repeatable testing, so we want to dig into what background traffic should be, why you should do it this way and what NOT to do.

Background - it's a lot more important than most people realize.

Deep Packet Inspection (DPI)

Every month or so I write a blog post for dpacket.org. For those that don't know, dpacket.org, it's the online community site focused on deep packet inspection. It's run by Kyle Rosenthal out of San Francisco. If you are a vendor making DPI devices or interested in DPI in general I suggest you join and check it out. It's a new community and it's just starting to grow.

Most of the time I write about Network Processors, I'm very passionate about that technology and have been lucky to work on them throughout my career.

May 5th 2008: Common Problems with Network Processors
March 3rd 2008: Moving as Fast as Intel
Feburary 4th 2008: Deep Packet Inspection and the Role of Network Processors

 

Coming Full Circle

 

At my previous company the CEO at the time, John McHale, decided that we need to prove to the world something that we thought we already knew. That we had built the best Intrusion Prevention System (IPS) on the planet. Our mission was to have someone prove it. We didn't want to pay someone to prove our claims, we wanted a bake-off of every possible IPS vendor at the time and independently determine the winner. All parties should pay to be tested and the test should be the same for all vendors. That third party was NSS Labs.

NSS Labs had gained a reputation as a very strong test lab that published their full test methodology for the scrutiny of everyone. It was a published set of tests and you would be measured against these specific goals. At the time NSS had three levels, Tested (you showed up), Approved (works pretty good) and Gold (best product in the class). They rarely handed out Gold, and still rarely hand out of Gold. At the company we said Approved would be great and a huge achievement for our first time being tested. John said, nope he wanted Gold. We explained that Gold would be near impossible, we would have to get 100% on every test and that alone would take 6-9 months of development effort if we focused on nothing else. John made a gutsy call and the wisdom was - if we are Gold with NSS then we should never lose a deal because we are the best product. So we spent every night and weekend for many months recreating the NSS test lab and running tests over and over again. At the time NSS was in France and we visited Bob Walder (CTO of NSS Labs) many times to talk about the testing and make sure our results equaled his. In other words, our testing harness was correct.

We put in so many optimizations and changes in those months that it was amazing. This wasn't features; this was performance optimizations, UI changes and all the spit and polish that you never get around to do. These are the defects in your bug tracking systems that just keep getting moved to future because they aren't "important" enough. Well it turns out if you actually spent the time on those items, something amazing will happen.

We, TippingPoint, received the NSS Gold award. It was all worth it, the product was 10x more stable, faster and more accurate. Everything we did for NSS made the product better for everybody else and really made headway for sales. Not just as a marketing vehicle, but also as a superior product.

Last week we announced that NSS had switched to using BreakingPoint as it's primary test product. NSS was one of the reasons we started BreakingPoint.  The lab setup was complicated, hard to use, results would vary based on the stability of a number of the key test vendors. We would run them 3 times in a row just to make sure the results were the same, often from a reboot on the test gear. I wanted a device that could replicate all of NSS Labs' tests and then some. In fact, I wanted a device that would allow NSS Labs to create more realistic tests. Tests that were impossible in the past.

It's really neat that we have come full circle.