Attacking Critical Internet Infrastructure

Update: The presentation went off without a hitch and the full paper describing the attack, including proof is now available.

Taking a page from L0pht Heavy Industries, Alexander Sotirov, Jacob Appelbaum, and a team of researchers whose identities have to remain secret for now are making the theoretical possible this Tuesday at the 25th Chaos Communication Congress in Berlin. The details of their presentation have been heavily censored leading up the event, with only a handful of security researchers, journalists, and collaborators given early access to the materials. Fortunately, I was one of them, and I wanted to take the opportunity to talk about their research, why it is important, and why the pre-conference secrecy is justified.

First things first; the reason for secrecy. Their research combined a known weakness in one area with a massive resource investment in another to show that a third party was vulnerable to a practical attack that affects the security of all Internet users. Security researchers often release code and technical documentation to demonstrate a flaw, but in this case, they went a step further and used the attack in the real world to obtain proof that it works. This process required interaction with a third party that will likely do whatever they can to save face once the details become public.

To prepare for the fallout, Alexander and Jacob have been working with a legal team to review their work and advise them on the best way to disclose the issue without finding themselves at the receiving end of a lawsuit. From my own conversations with Alexander, I don't believe they broke any laws, but in the past few years there have been multiple unjustified legal actions and threats against researchers who have tried to warn the public about serious security issues.

The last ten years have shown us that it is usually better to ask for forgiveness than permission when it comes to vulnerability disclosure. Vendors have a financial interest in protecting their reputation and this is apparent in the number of pre-disclosure threats they make; however, once the proverbial cat is out of the bag, it is cheaper to address the problem than to proceed with legal action, since that legal action will usually result in even worse publicity for the vendor. If your organization is withholding vulnerability information due to concerns over a legal reaction from the affected vendor, then you have already lost the game and are doing a disservice to every user that relies on that vendor's product or service.

Switching back to the actual presentation; there are three things that make their research impressive. First, their work involved serious collaboration between academia and independent security researchers. This type of coordination is tough to manage and nearly impossible to actually publish anything under terms everyone can agree to. Jacob's previous work on cold-boot memory attacks faced similar challenges and the end result was partial disclosure of the developed tools (the BitLocker code was never released). Second, their research required massive computational resources that had to be utilized within a specific window of time. Although computing costs have dropped significantly over the last few years, the researchers estimated that commercially available computation resources such as Amazon EC2 put the technique within the grasp of a profitable criminal organization, large botnet operator and certainly state sponsors. The attack only has to be performed once in order to reap rewards for a long time afterward (months, if not years). This one-time investment model could pay for itself many times over if it was used to provide services to criminal organizations. Finally, they actually did it. This isn't a pie-in-the-sky talk about what may happen or what someone might be able to do, this is a demonstration of what they actually did with the results to prove it.

Their live presentation will be available online via streaming video, it is scheduled for Saal 1 at 3:15pm (8:15am CST, 9:15am EST, 6:15am PST) on Tuesday, December 30th. Keep an eye on the schedule in case their timeslot is moved.

Fuzzing the Internet Printing Protocol (IPP)

The Internet Printing Protocol (IPP) was designed as a next-generation printing and print job management protocol. Unlike the older line printer protocol (LP), IPP supports a number of advanced features, including authentication and encryption. IPP is loosely based around HTTP 1.1 and takes advantage of many of HTTP's features and conventions. The amount of code required to implement IPP is helped by its similarity to normal HTTP traffic. Microsoft Windows, Mac OS X, and most Linux distributions all support client and server modes of IPP. On the Windows side, the IPP server is integrated into the Microsoft IIS web server (/printers/). On Mac OS X and most Unix flavors, IPP is implemented via the Common Unix Printing System (CUPS), an open source implementation that is current maintained by Apple. Most network printers also support IPP and enable this protocol by default, however they tend to use custom implementations that have less features (and as we will demonstrate, stability) than the IPP implementations available for consumer operating systems.

In this article, we will be launching the BreakingPoint IPP fuzzer against a HP LaserJet 4250 network printer as well as an Ubuntu Linux system running CUPS. To get started, we access the web interface of a BreakingPoint BPS 1K appliance running the latest StrikePack. This system has interface 3 connected to the network that our targets are on and interfaces 1, 2, and 4 unplugged. After authenticating, the first thing we need to do is create a new Attack Series containing the list of Strikes we want to run. To create the Attack Series, we need to access the Attack Manager from the Managers drop down menu at the top of the interface. Once the Attack Manager has loaded, click on the Create Attack Series button. When the system prompts for a name, enter “IPP Fuzzer”. After the Series has been created, click the Add Strikes button in the right-hand pane. This will load the Strike Browser interface. From this interface, select the “ipp” keyword from the Keywords list and click the Strikes radio button. The results should look like Figure 1.

Figure 1: Searching for IPP strikes by keyword

Click the Search button to look for all individual Strikes which have the “ipp” keyword. The resulting list will contain a number of Strikes in the fuzzer category as well as an exploit. Since we are interested in fuzzing, we will select all of the fuzzer strikes and use the right-pointing arrow to add them to this test. The result should look like Figure 2.

Figure 2: Adding the IPP fuzzer Strikes to the Attack Group

Click the Add Strikes button to save this Attack Group and return to the main Attack Manager screen. The “IPP Fuzzer” Series should be selected and show the Strikes that were added to the list on the right-hand pane. This screen should look like Figure 3.

Figure 3: The configured Attack Series

The next thing we need to do is configure the IP address of the fuzzer and the target system. From the Control Center menu, select Network Neighborhood. Click the Create Neighborhood button and name the new setting “IPP Fuzzer Endpoint”. Once named, the right-hand pane will show a list of tabs for each interface and an additional interface labeled “External”. To configure the IP address used by the Fuzzer, select the tab for Interface 3. The network we are on is 10.10.10.0/23 and we want to confine the fuzzer to a single IP address, 10.10.11.165. Uncheck the “Use Address Range” box to ensure only a single IP address is used. Once configured, click the Apply Changes button. The result should look like Figure 4.

Figure 4: Configuring the fuzzer source IP address

Now that the source address has been configured, select the tab labeled “External”. This screen is used to configure the target IP address. First we need to select the preset range and use the delete button to remove it. Next we enter the target address into the Minimum Address field, uncheck the “Use Address Range” box, and click Add Range. The address we chose (10.10.10.251) points to the HP LaserJet printer that we want to test. Finally, click the Apply Changes button to save these settings. The result should look like Figure 5.

Figure 5: Configuring the target IP address

At this point, our Attack Series and Network Neighborhood is configured. The next step is to create a new test using these settings. From the Test menu at the top of the screen, select the New Test option. On the left hand side of the new test screen is a link for selecting the DUT and Network Neighborhood. Click this link, select the newly created “IPP Fuzzer Endpoint” option from the list of neighborhoods, and click the Accept button.

Once the interface has returned to the New Test screen, select the link labeled “Add a Test Component”. This will display a screen of 7 test components. Choose the component labeled Security from the bottom left of this screen. Once the Security component has been added, select the Interfaces tab, uncheck the presets, and use the checkboxes to select Interface 3 as the Client and the External interface as the Server. Click the Apply Changes button. The result should look like Figure 6.

Figure 6: Configuring the Security component interfaces

Select the Parameters tab of the Security component. Click the Attack Series option in the list of parameters and choose the IPP Fuzzer option that was created in the previous steps. Click the Apply Changes button. The result should look like Figure 7.

Figure 7: Selecting the IPP Fuzzer Attack Series

Finally, click the Save and Run button from the left side of the screen. The interface will prompt for a name for the test and start running the IPP Fuzzer Strikes against the target device. Once the test starts running, we will see a progress screen like the one shown in Figure 8.

Figure 8: Running the IPP Fuzzer Attack Series

In the graph above, we see a short spike of packets, followed by a long series of almost no traffic. Once the test completes, we can look at the report and determine what happened. Figure 9 shows a sample report for this test.

Figure 9: IPP Fuzzer vs HP LaserJet

In the Strike Detail section of the report, we see that the “IPP Print Operation Long Job Name” Strike reported a maximum job size of 64 bytes (of a maximum tested size of 65528). All of the following Strikes reported a maximum length of zero while the last Strike indicates that zero operations were accepted. If we try to telnet to the IPP port of the target device, we can see that the service is down and is no longer accepting requests. It appears that an IPP Job Name that is 65 bytes long will crash the HP LaserJet IPP service. The bandwidth graph showed a quick spike as the first 64 requests were sent to the IPP service, but after the 65^th test, the IPP service stopped responding and the subsequent Strikes quickly gave up after not being able to connect.

The second target we want to test is a Ubuntu Linux system running CUPS, which happens to expose an IPP service as well. To change the target IP address, we access the Network Neighborhood we created, click on the External tab, change the IP address, and click Apply Changes. The next time we run this test, the fuzzer will target the Ubuntu system instead. Figure 10 shows a sample run of the exact same test against a CUPS IPP server.

Figure 10: IPP Fuzzer vs CUPS

In this test, we see that the service refused requests with field lengths over a threshold (~14100), but did not crash. At the end of the test, the IPP service was still online and functional, and the logs of the CUPS server did not show any fatal errors. Since the IPP service did not crash during this test, all six Strikes were able to run to completion, extending the test time to about one hour.

The BreakingPoint architecture makes it easy to integrate fuzzing with performance, application, and live exploit testing. This combination provides a way to flush out bugs that only trigger when the device is under a heavy load, something that is difficult to do with other products on the market. While IPP is a simple example, the same technique applies to all of our fuzzers. We design them to test live services as well as content-inspection devices that route traffic. The reports make it possible to determine exactly what requests can break a service and allow our users to reverse-engineer a content-filter's traffic normalization rules.

Building a Better Botnet

Earlier this year, Dennis and Kyle put together a screencast overview of BreakingPoint's capabilities for simulating botnets. Today, I'd like to go into a little more detail about the AppSim component that's most associated with malicious traffic, IRC.

While most of our AppSim components are used to simulate good, normal traffic -- HTTP, Oracle, VoIP, etc. -- IRC stands out as a protocol that is, for better or worse, often associated with malicious traffic. Of course, not everything on IRC evil. However, IRC is the traditional medium for setting up command and control channels for botnets, those armies of compromised end-user machines performing attacks such as DDOSes, spam campaigns, vulnerability scanning, and other evil deeds. So, for most ISPs and enterprises, it's not very useful to just detect the presence of IRC traffic; it's much more worthwhile to detect particular patterns associated with botnet activity. Given that, let's build up a quick botnet that your fancy expensive detection device claims coverage for to see what sticks. In this example, we'll model a fictional botnet (DdosHax) that uses channel topic changes to signal to compromised hosts, from the point of view of the bot herder's local network.

First, we'll fire up the IRC AppSim Superflow editor, and populate a new superflow with the IRC actions we're interested in:

That gives us a channel join, a couple of other users joining the same channel (our bots), a topic change (the command), and the bot responses (command acknowledgements).

Next, we'll fill in the salient details of the bot herder's user name and the network he's joining.

And finally, we'll set the options for our flow actions so this all looks like a typical botnet communication. We can set the command of "!sf 192.168.1.1" to the channel topic:

And generate a response from several peers:

Here, we can see that the naming convention is pretty flexible for the number of bot hosts we'd like to simulate. In this case, we set a range between "a0" and "b9", which gives 20 remote peers, all of whom include the range indicator in their names (zombie-a0 through zombie-b9). If we'd prefer randomized names, we could uncheck the "Include Range in Name?" options, and if we wanted completely random names, we'd uncheck Peer Nick Prefix as well. If you wanted a hundred peers, simply set the range from 1 to 100, or 'aaa' to 'adv', or whatever other range naming scheme that strikes your fancy.

Once this AppSim is run on the wire, we can take a look at the traffic with a regular packet sniffer:

That looks pretty evil to me -- and best of all, the whole procedure took about five or six minutes to set up from scratch, without all the futzing around with setting up real IRC servers and clients, writing scripted actions and responses, or editing up packet capture files.

Firewall Testing WebCast: Video Replay

We put up the full video of our firewall testing webcast that was held last week and I wanted to be sure to share it with all of you. It is embedded below.

Just a reminder that if you also want to receive BreakingPoint's firewall test methodology when it is published you can just put your info within this form. Additionally, we have put together an online room on FriendFeed that gathers all the latest info, thoughts, strategies and opinions on firewall testing. Check it out and let us know what you think.

Test Automation: BPS TCL!

If you are a BreakingPoint customer, you probably know that each week we release a StrikePack to update our Security and AppSim components. These updates insure that our security coverage is as up-to-date as possible and give customers quick access to the most recently developed application protocol simulations. This type of rapid release schedule requires testing to happen quickly.

Enter the BreakingPoint TCL API. Since we're building test equipment, it stands to reason that people expect (nyuk-nyuk) the availability of TCL to automate their BreakingPoint appliances (and soon, our chassis the BreakingPoint Elite). BreakingPoint provides a custom TCL shell that provides the API for controlling your devices. This shell is used as the interpreter for TCL scripts, as well as an interactive shell. The interactive mode is a great way to explore and learn the API. Customers can also read chapter 14 of the BreakingPoint UserGuide, which can be downloaded from the BreakingPoint Systems Documentation page on StrikeCenter (authentication required). The examples in this post are intended for use with appliances running the 1.2.3 BreakingPoint software release.

To get started, download the BreakingPoint TCL shell from the system start page. When you run it from the command line (or from a script using the -i flag), you'll get a familiar-feeling '% ' prompt. To connect to your BreakingPoint box, issue something similar to the following:

This sequence is so common that I have wrapped it inside a script I call harness.tcl, which parses its commandline arguments, makes the initial connection, and then drops you into the interactive shell.

Once you're connected, you can take a quick look at the commands and variables available for use by issuing:

The listed commands will show usage information when you issue them without arguments, so you can just experiment and see what each of the commands accept. Many variables are defined, but the variable we care about at this point is the $bps variable, which represents the connection we have established to the BreakingPoint appliance.

The $bps object gives commands to control many aspects of the device: tests, components, application profiles, attack series, load profiles, DUTs, reports, and device management.

So let's dive in and configure a simple test with a single Security component. The basic steps are:

• Name the test
• Add a security component
• Set the test interfaces for the component
• Save and run the test
• Review the results

The code for this is pretty simple:

This test will use the default DUT profile of 'BreakingPoint Default,' and the default Network Neighborhood of 'BreakingPoint Switching.' If your test environment requires something different like routing, you'll need to issue the appropriate setDut and setNeighborhood commands before adding the component. The two setDomain commands set the server and client sides of the security component, and specify the default ethernet and IP addressing domains. After running the test, we display the summary results of the test. To get detail about what strikes were blocked, you need to view the report generated from the exportReport call.

The nice thing about this code is that it's general enough to run any component. You can set componentname in the first line to appsim, bitblaster, recreate, routingrobot, security, sessionsender, or stackscrambler. The list of available components can be found by issuing `$bps createComponent false false`. This will show both the canned (BreakingPoint defined) as well as custom (customer defined) components.

The automation afforded by using TCL has been invaluable in reducing the time it takes the BreakingPoint Labs team to test and deploy StrikePacks. I know it will benefit anyone that takes the time to learn it. Over the coming weeks, I'll be sharing hints and code that will help BreakingPoint customers leverage automation in their testing efforts. The next installment will focus on utilizing the Security component to test IPS signatures. In the meantime, please download the code archive for this post and get started with TCL!