White Paper: Simulating Distributed Denial-of-Service with BreakingPoint

Today we have released a new white paper that I've been working on entitled "Simulating Distributed Denial-of-Service with BreakingPoint".  This paper describes how to configure your BreakingPoint product's Network Neighborhood to simulate the traffic profile normally associated with a DDoS attack and then outlines a number of DDoS attack scenarios.  I've also provided a link below to a packaged version that includes product test cases to simulate the scenarios described in the paper.

Of the scenarios presented, there are several recent real-world analogies.  For example, the group of HTTP scenarios in the paper are similar in nature to multiple DDoS attacks that were recently launched simultaneously against our very own HD Moore's Metasploit Project website, alongside other information security and hacking related websites. You can read his ongoing commentary from during and after the attacks on the Metasploit Blog, beginning with this post.

The last scenario discussed in the paper is one of my all-time favorite DDoSes from when I was focusing a lot of my research efforts within the scope of VoIP systems and technologies.  I regularly employed the tactic outlined by this scenario to demonstrate how a DDoS attack can effectively fly under most network security devices' radar by avoiding the usual DDoS traffic model and by shifting the target of the attack from the technology itself to elsewhere.  I won't ruin the surprise here, you'll have to download the paper to find out what I'm talking about...

Finally, some of the test cases were created via scripting within the BreakingPoint TCL interface, so the paper also provides an introduction to that topic as well as the TCL scripts themselves.  Todd Manning has recently been blogging here on this topic, the posts for which you can find by browsing this blog using the "tcl" tag.

We invite you to take a look at the paper, which can be found here (PDF).  The package which includes the paper as well as supporting materials such as test cases and TCL scripts can be found here.

340 Gigabits Per Second of Blended Layer 4-7 Application Traffic

The testing tools industry loves to throw around numbers when it comes to Gigabits per second (Gbps) and rightfully so, it is a critical discussion, particularly as we see more and more network devices capable of handling large Gbps. Truth in testing dictates that it is not only important to emulate traffic at the Gbps you need, but also do it using blended, stateful Layer 4-7 application traffic (and simply using HTTP is NOT realistic).

Below is a screencast we shot of a 5-box BreakingPoint Elite test and a 10-box BreakingPoint Elite test. The tests are performed from one common user interface (not to mention common reporting) and you can watch along with me to see how much Layer 4-7 blended application traffic we emulate. The story is in the headline, but it's fun to watch it happen and I edited this down so that you can watch it in a little over 120 seconds, but if you want a more in-depth look at BreakingPoint Elite check out all of our screencasts.

As you saw in the video, BreakingPoint testing tools supports more than 70 application protocols including HTTP, SMTP, AOL® IM, FIX, IBM DB2, VMware® VMotionTM, Microsoft® CIFS/SMB, MAPI, Oracle, Encrypted BitTorrent™, eDonkey, RADIUS, SIP, Windows Live Messenger, World of Warcraft®, Yahoo!® Messenger, and many others. Additionally, new application protocols and security strikes released each week and an API is available for accelerating proprietary protocols.

Time for Realism, Simplicity and Truth in Testing

UPDATED: Went back to the 16 Spirent Avalanche number after consulting with some folks, thanks!

Interesting to read NetworkWorld's review of the Juniper SRX 5800, particularly after we just had the opportunity to test the SRX during BreakingPoint Day at Juniper. It was entertaining to watch Twitter and read the thoughts of a variety of folks reading the review and recognizing that from start to finish, the test missed the boat on realistic testing and was shocking in its complexity and immensity due to the required Spirent test gear. I even wrote about my thoughts on Twitter while reading the review:

A few highlights from my point of view:

1) It took 16 Spirent Avalanches, 1 Spirent TestCenter and 1 Spirent ThreatEx to run a test of the Juniper SRX. That is crazy, particularly when you realize one BreakingPoint Elite generates 40 Gbps of Layer 4-7 traffic and 80 Gbps of Layer 2-3 traffic. Imagine the cost and time associated with needing that much test equipment.

2) On top of all that equipment, there seems to be zero use of stateful traffic to test the Juniper SRX. Testing without a realistic blend of traffic (beyond simply HTTP), Layer 4-7 stateful traffic, is unacceptable in today's era of content-aware high-performance network equipment.

3) To continue hammering this point home...using "13 UDP attacks" to test IPS functionality at load? As HD said on Twitter, "Static testing is pretty worthless".

4) Dennis reminded us all that relying on IMIX is a dangerous game, which was referred to in the test through the use of data from CAIDA. Again we must all make a more concerted effort towards realistic testing.

Having just witnessed our test of Juniper SRX it struck me how using a fourth of the test equipment (four BreakingPoint Elite) used by NetworkWorld, we tested the SRX with stateful traffic, live security strikes and even more throughput and bandwidth. This latest test should be a call to our industry to come together and demand more realism, transparency and simplicity from network equipment testing. We all recognize what it takes to perform realistic testing, but all too often we let test vendors, equipment vendors, test labs and industry publications get away with testing insufficiently. Now is the time to demand more truth in testing.

StackScrambler and the Tale of a Packet Parsing Bug

A core feature of the BreakingPoint product line is the ability to perform any kind of layer 2-7 testing with a single hardware device. BreakingPoint's functionality is divided into 7 test components; these are Bit Blaster for layer 2 ethernet traffic, Routing Robot for layer 3 IP traffic, Session Sender for layer 4 TCP and UDP traffic, AppSim for layer 7 application traffic, Recreate for replaying any captured layer 2-7 traffic, Security for malicious traffic, and finally Stack Scrambler for malformed or fuzzed but benign traffic. My previous posts have predominantly focused on testing using the Security component for testing IPS. Today, I want to take a look at using Stack Scrambler to find bugs in software and devices that inspect packets.

Stack Scrambler is a component that performs stack integrity checking for the Ethernet, IP, TCP, UDP and other stacks of a DUT. It is analogous to the ISIC suite of utilities. During a test, Stack Scrambler sends a stream of mutated packets and includes an ICMP echo request every 100 packets. The ping packets are sent to determine the functional state of the target. When testing a DUT inline, the BreakingPoint box will send pings out on the client interface and receive them on the server interface. When testing a host in endpoint mode (we call this 'one arm' testing), if the target is functioning properly it will respond to our pings with ping response packets. In either mode, we check to make sure that the number of pings sent equals the number of pings received. After the configured test duration, 5 consecutive pings are sent as a final test of the target's functional state.

A bug was reported by our excellent QA folks that the number of pings reported by the component did not match the number actually sent during a test. Troubleshooting this is easy - setup a packet capture, run the test, compare the report with what is seen on the wire.

The BreakingPoint Elite has a really cool feature the previous two products do not: a 2GB packet buffer. This packet buffer lets users export up to the last 2GB of the most recently-run test. The packet buffer on the Elite is handled by the FPGAs, which have their own memory for saving packets that are sent and received. This is one feature I am glad we have in the product, and I know that customers are going to get a ton of use from it. Before the Elite, doing packet captures either involved using an external host or using a private, development-only feature of our network processor. Having the packet buffer makes test setup much simpler and completely reliable. You don't have to worry about whether or not your external capture method can keep pace with 40Gbits/sec.

Accessing the packet buffer in the UI

Selecting ports and packet direction when exporting the packet buffer.

I setup a 10 minute test using Stack Scrambler, and left all other parameters at their default. After running the test, I exported a pcap file. In my testing, I did not use a DUT, but instead had the BreakingPoint box running in loopback between two interfaces. In this type of setup, the packets seen on Slot 1/Port 0 will be identical to those on Slot 1/Port 1, so I only exported the packet buffer for the client port. The report for this test run claims 2798 ping packets were sent.

Wireshark display of the packets from Stack Scrambler

I opened the 150k-packet capture file in Wireshark. Since this was a bug about the reported number of ICMP packets, I filtered the list on 'icmp' to get to a total of 1554 ICMP packets, so at this point I've already verified the reported bug. Visually picking out the diagnostic pings is pretty easy - they're a static size, have a static payload, a static ICMP ID field, and a sequential ICMP sequence number. The diagnostic pings also share source and destination IP addresses. In its default configuration, stack scrambler will send fuzzed pings in addition to the diagnostic pings, so I need to filter those out using 'icmp && ip.src==1.1.55.93'. Wireshark now shows 1404 diagnostic ping packets, definitely less than reported. It looks off by about a factor of 2, but not exactly. A quick check shows reported_pings = (diagnostic_pings - final_pings) * 2.

So a quick check of a single test shows that it looks like the stats are double counting all the diagnostic pings sent during the test, except the final burst of 5 consecutive pings. This sounds like a reasonable hypothesis. I wanted to run several tests and verify that the reported pings were always wrong as predicted by the formula above, but I wanted to automate the testing. Since Tod Beardsley wrote yet-another pcap library for ruby, I decided I'd put it to use here. I wrote a quick script that parsed the pcap file, counting ping packets if the ICMP ID matches Stack Scrambler's static value 0xDEAD. I really like blog posts with code examples, so here's a simplified version of what I came up with using PacketFu:

is pretty awesome? It's pretty awesome. So, I wrote a quick TCL script to have the Elite run several tests running for a randomized number of seconds, checking the reported pings for each test. The TCL code looks like this:

On the first test iteration, Stack Scrambler runs, and I export my pcap file. The ruby script crashes. Saywhat?! The second iteration runs, and that packet capture crashes the ruby script. Every packet capture crashes the ruby script. The crashes look like it's PacketFu. I modified my ruby script in a couple of ways. First, I wrap a ruby begin..rescue block around PacketFu's parse() call; secondly, I save off every packet that causes an exception. We'll come back to the PacketFu-Fail in a moment.

After gathering stats from the 10 iterations of Stack Scrambler, I conclude that my hypothesis was correct. Looking at the code, I see that the diagnostic pings sent during the test are counted as both sent and received on both test interfaces, instead of being counted as sent on the client interface and received on the server interface. After a 2-line fix and a checkin, the Stack Scrambler ping reporting bug was fixed.

Ok, so back to PacketFu. I went and told TodB that his library was broken. I looked at the list of exception-causing packets. They appeared to have at least one thing in common: they were packets with an ethertype field of 0x0800, which is IPv4, but the IP version in the IP header was not 4. I mentioned this finding to Tod, and he's committed a fix to PacketFu to address the issue.

Using Stack Scrambler and the BreakingPoint Elite might be a little overkill for testing a new packet parsing library written in ruby, but this example shows how easy it is to make assumptions when parsing network traffic. What if a similar bug existed in your IPS or router?

Lately I've been writing a series of posts on the BreakingPoint TCL API. This is the first time I show examples of new API calls that have been added for the BreakingPoint Elite. Next time, I'll be discussing some of the updates in the new API, and how to take advantage of those updates to script testing with the BreakingPoint Elite.

Update: corrected a typo in the ruby code.

BreakingPoint & VMware Networking

Today, I'll show you how you can use a VMware server to quickly switch between different test devices without ever leaving your desk. Then I'll fire up some sample tests against a virtual Linux host.

Why VMware

Working for BreakingPoint Labs, we are faced with rapid release cycles and a broad range of tasks. Sometimes it is working to provide timely Microsoft Patch Tuesday Strike coverage, other times it is reverse engineering a protocol and implementing it with Application Simulator. Frequently we have to quickly pick up strange new aspects of the computer world and just run with it.

Since we switch gears a lot, we make heavy use of VMware Workstation™ on our personal development machines. This works perfectly right up until we need to use BreakingPoint to target our environment. Our development BreakingPoint test products are located in a rack in a lab, but we are living in cubicle land.

VMware Server 2™ is a nice solution. It has a remote web console where all of the BreakingPoint Labs developers can add/configure new virtual servers without affecting each others setup. With a dedicated server in our rack for VMware, we can configure what kind of target our BreakingPoint product is connected to without ever leaving our cubes.

The Setup

You can grab a copy of VMware Server 2 here. You'll need a server with adequate RAM and disk space, as well as at least 3 physical network interfaces. That is one for a regular LAN connection and at least two that connect directly to the ports on you BreakingPoint product.

After getting VMware up and running, you'll need to add two new virtual interfaces in bridged mode. This can be very simply done by the "vmware-config.pl" wizard. I like to name these VMLAN1 and VMLAN2. If you like, you can also add new vmnet interfaces for the other two test ports. These will take the traffic your BreakingPoint product is generating and pipe it into your virtual environment.

When you create a new virtual machine, you should be able to see and add the custom bridged network interfaces via the "add new hardware" wizard. After your virtual machine is booted, you will want to make sure the virtual machine's network settings match up with a configured BreakingPoint "Network Neighborhood".

Blasting Linux with a Packet Cannon

As an example, I'll run a test against a Linux VM that I have set up. I would say the typical use case for Linux networking is a home NAT router/firewall. So, I've set up a standard Ubuntu Server in VMware and configured it to perform NAT for all LAN hosts with a simple iptables script. BreakingPoint is also configured to use a NAT Network Neighborhood so that it will recognize packets with the source address rewritten.

# eth2 is the LAN side, eth1 is the WAN side
echo 1 >> /proc/sys/net/ipv4/ip_forward
/sbin/iptables -F
/sbin/iptables -P FORWARD DROP
/sbin/iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE 
/sbin/iptables -A FORWARD -i eth2 -j ACCEPT # Let LAN traffic through
/sbin/iptables -A FORWARD -o eth2 -j ACCEPT
/sbin/iptables -A INPUT -i eth1 -m state --state ESTABLISHED,RELATED -j ACCEPT

This firewall script will enable NAT with connection tracking for the traffic that passes through the Linux device. The tracking of every session should be putting quite a load on the virtual CPU. I will also be running ntop on the VM to get some graphs of what traffic it sees from its perspective.

Application Simulator: Small to Medium Business Apps

I'll start off by running the Application Simulator preset for SOHO routers for 10 minutes. This will send a reasonable amount of traffic considering that it is virtual networking on commodity hardware. This test is about right for such a router. Note the CPU usage for the ntop daemon.

Ntop Stats from the Linux VM

Ntop CPU Usage

BreakingPoint Traffic (RX and TX rates match up)

Application Simulator: Max Bandwidth

This Application Simulator preset run as many of the highest bandwidth applications as fast as it can. Considering just how much that can be, this will simply DoS the Linux VM. The data rate is fast enough that even though the traffic is realistic, its effectively just a SYN flood. The gathered data shows effectively the same thing.

NTop Stats (Not much is getting through)

BreakingPoint Traffic (RX rate nowhere near TX rate)

Switching Gears

The real awesomeness of using VMware comes when you finish with one task and need to do something completely different. I started off with a Linux bandwidth test and now I need to run some Strikes against a Windows Vista machine. All I have to do is boot the Vista virtual machine and double check my network settings.

Strike Level 5 Running Against Windows Vista

Have it your way

There is no reason to limit yourself to just one or two test ports either. With VMware, you can configure any number of hosts connected by any number of LAN segments. You could daisy-chain a few software IPS machines and see how far malicous traffic can go within your network. Or you could run a one-armed test against a web server that is behind a firewall all while flooding the router with P2P traffic. The options are nearly limitless.