10Gig E: Hot Technology of 2009?

This week I was catching up on many of the end-of-year prediction articles and "what to look for" in 2009 postings. Most of these lists are mundane and many of them don't go back and review their previous predictions to see how close they came in their prognosis. However, one list did catch my eye, Network World's "Nine Hot Technologies for '09" (nice work with the 9s folks). Number six was the one that stood out to me and we discussed it a bit on Twitter earlier in the week:

"With ongoing data-center server consolidation, not to mention the needs of service providers and high-volume Web sites, standards groups and vendors are hard at work on 40 Gigabit Ethernet and even 100 Gigabit Ethernet. For now, however, 10G Ethernet is the industry standard, and customers are flocking to 10G Ethernet switches. Switch-based 10G Ethernet port shipments grew by 140% in 2007, Infonetics Research reports. Worldwide revenue for 10G Ethernet services and equipment will hit nearly $9.5 billion by year-end, a 30% increase from last year, the firm predicts"

We wrote about this Infonetics research back in October when the report first came out. The market saw a large increase in switch-based 10G Ethernet port shipments in 2007 and, as Network World's Neil Weinberg writes, this was primarily due to per-port pricing dropping from $39K to $4K.

10G is definitely growing in the back room and Neil makes the point in the article that:

"If your Fast Ethernet boxes are becoming stressed, this might be the time to move to 10G Ethernet. Per-port prices are coming down and feature sets are going up."

I have not found comprehensive results from 2008 sales of 10G equipment (send along if you have a link), but I still have a few questions as we begin 2009:

1. How much will the economy effect purchasing of this equipment? We might be surprised in that companies will move resources to the data center in order to purchase 'future-looking' equipment.
   
2. When will we see the standards ironed out on 40G and/or 100G? This will obviously have an impact on the movement.
   
3. When will chipsets for 10G decrease, making 10G to the desktop a bit more feasible? There was an obvious increase in 10G switch-based Ethernet shipments because of the decrease in per-port price, the same will go for desktop usage.

Obviously our eye is always on the testing of this type of equipment and the fact that you need testing tools that can not only realistically test performance and security, but reduce time-to-test. This is part of the conversation not typically discussed in most of these projections, particularly the latter point around reducing time-to-test. As I said in #1 above, we may see an increase in infrastructure in the data center, meaning that equipment manufacturers and service providers are going to want to deploy updated equipment and confirm that it is ready to perform in deployment (including 10 Gigabit Ethernet testing). In that spirit we are going to spend much of next week writing on reducing time-to-test, including seven tips we have to help in make this happen.

 

Attacking Critical Internet Infrastructure

Update: The presentation went off without a hitch and the full paper describing the attack, including proof is now available.

Taking a page from L0pht Heavy Industries, Alexander Sotirov, Jacob Appelbaum, and a team of researchers whose identities have to remain secret for now are making the theoretical possible this Tuesday at the 25th Chaos Communication Congress in Berlin. The details of their presentation have been heavily censored leading up the event, with only a handful of security researchers, journalists, and collaborators given early access to the materials. Fortunately, I was one of them, and I wanted to take the opportunity to talk about their research, why it is important, and why the pre-conference secrecy is justified.

First things first; the reason for secrecy. Their research combined a known weakness in one area with a massive resource investment in another to show that a third party was vulnerable to a practical attack that affects the security of all Internet users. Security researchers often release code and technical documentation to demonstrate a flaw, but in this case, they went a step further and used the attack in the real world to obtain proof that it works. This process required interaction with a third party that will likely do whatever they can to save face once the details become public.

To prepare for the fallout, Alexander and Jacob have been working with a legal team to review their work and advise them on the best way to disclose the issue without finding themselves at the receiving end of a lawsuit. From my own conversations with Alexander, I don't believe they broke any laws, but in the past few years there have been multiple unjustified legal actions and threats against researchers who have tried to warn the public about serious security issues.

The last ten years have shown us that it is usually better to ask for forgiveness than permission when it comes to vulnerability disclosure. Vendors have a financial interest in protecting their reputation and this is apparent in the number of pre-disclosure threats they make; however, once the proverbial cat is out of the bag, it is cheaper to address the problem than to proceed with legal action, since that legal action will usually result in even worse publicity for the vendor. If your organization is withholding vulnerability information due to concerns over a legal reaction from the affected vendor, then you have already lost the game and are doing a disservice to every user that relies on that vendor's product or service.

Switching back to the actual presentation; there are three things that make their research impressive. First, their work involved serious collaboration between academia and independent security researchers. This type of coordination is tough to manage and nearly impossible to actually publish anything under terms everyone can agree to. Jacob's previous work on cold-boot memory attacks faced similar challenges and the end result was partial disclosure of the developed tools (the BitLocker code was never released). Second, their research required massive computational resources that had to be utilized within a specific window of time. Although computing costs have dropped significantly over the last few years, the researchers estimated that commercially available computation resources such as Amazon EC2 put the technique within the grasp of a profitable criminal organization, large botnet operator and certainly state sponsors. The attack only has to be performed once in order to reap rewards for a long time afterward (months, if not years). This one-time investment model could pay for itself many times over if it was used to provide services to criminal organizations. Finally, they actually did it. This isn't a pie-in-the-sky talk about what may happen or what someone might be able to do, this is a demonstration of what they actually did with the results to prove it.

Their live presentation will be available online via streaming video, it is scheduled for Saal 1 at 3:15pm (8:15am CST, 9:15am EST, 6:15am PST) on Tuesday, December 30th. Keep an eye on the schedule in case their timeslot is moved.

Fuzzing the Internet Printing Protocol (IPP)

The Internet Printing Protocol (IPP) was designed as a next-generation printing and print job management protocol. Unlike the older line printer protocol (LP), IPP supports a number of advanced features, including authentication and encryption. IPP is loosely based around HTTP 1.1 and takes advantage of many of HTTP's features and conventions. The amount of code required to implement IPP is helped by its similarity to normal HTTP traffic. Microsoft Windows, Mac OS X, and most Linux distributions all support client and server modes of IPP. On the Windows side, the IPP server is integrated into the Microsoft IIS web server (/printers/). On Mac OS X and most Unix flavors, IPP is implemented via the Common Unix Printing System (CUPS), an open source implementation that is current maintained by Apple. Most network printers also support IPP and enable this protocol by default, however they tend to use custom implementations that have less features (and as we will demonstrate, stability) than the IPP implementations available for consumer operating systems.

In this article, we will be launching the BreakingPoint IPP fuzzer against a HP LaserJet 4250 network printer as well as an Ubuntu Linux system running CUPS. To get started, we access the web interface of a BPS 1K appliance running the latest StrikePack. This system has interface 3 connected to the network that our targets are on and interfaces 1, 2, and 4 unplugged. After authenticating, the first thing we need to do is create a new Attack Series containing the list of Strikes we want to run. To create the Attack Series, we need to access the Attack Manager from the Managers drop down menu at the top of the interface. Once the Attack Manager has loaded, click on the Create Attack Series button. When the system prompts for a name, enter “IPP Fuzzer”. After the Series has been created, click the Add Strikes button in the right-hand pane. This will load the Strike Browser interface. From this interface, select the “ipp” keyword from the Keywords list and click the Strikes radio button. The results should look like Figure 1.

Figure 1: Searching for IPP strikes by keyword

Click the Search button to look for all individual Strikes which have the “ipp” keyword. The resulting list will contain a number of Strikes in the fuzzer category as well as an exploit. Since we are interested in fuzzing, we will select all of the fuzzer strikes and use the right-pointing arrow to add them to this test. The result should look like Figure 2.

Figure 2: Adding the IPP fuzzer Strikes to the Attack Group

Click the Add Strikes button to save this Attack Group and return to the main Attack Manager screen. The “IPP Fuzzer” Series should be selected and show the Strikes that were added to the list on the right-hand pane. This screen should look like Figure 3.

Figure 3: The configured Attack Series

The next thing we need to do is configure the IP address of the fuzzer and the target system. From the Control Center menu, select Network Neighborhood. Click the Create Neighborhood button and name the new setting “IPP Fuzzer Endpoint”. Once named, the right-hand pane will show a list of tabs for each interface and an additional interface labeled “External”. To configure the IP address used by the Fuzzer, select the tab for Interface 3. The network we are on is 10.10.10.0/23 and we want to confine the fuzzer to a single IP address, 10.10.11.165. Uncheck the “Use Address Range” box to ensure only a single IP address is used. Once configured, click the Apply Changes button. The result should look like Figure 4.

Figure 4: Configuring the fuzzer source IP address

Now that the source address has been configured, select the tab labeled “External”. This screen is used to configure the target IP address. First we need to select the preset range and use the delete button to remove it. Next we enter the target address into the Minimum Address field, uncheck the “Use Address Range” box, and click Add Range. The address we chose (10.10.10.251) points to the HP LaserJet printer that we want to test. Finally, click the Apply Changes button to save these settings. The result should look like Figure 5.

Figure 5: Configuring the target IP address

At this point, our Attack Series and Network Neighborhood is configured. The next step is to create a new test using these settings. From the Test menu at the top of the screen, select the New Test option. On the left hand side of the new test screen is a link for selecting the DUT and Network Neighborhood. Click this link, select the newly created “IPP Fuzzer Endpoint” option from the list of neighborhoods, and click the Accept button.

Once the interface has returned to the New Test screen, select the link labeled “Add a Test Component”. This will display a screen of 7 test components. Choose the component labeled Security from the bottom left of this screen. Once the Security component has been added, select the Interfaces tab, uncheck the presets, and use the checkboxes to select Interface 3 as the Client and the External interface as the Server. Click the Apply Changes button. The result should look like Figure 6.

Figure 6: Configuring the Security component interfaces

Select the Parameters tab of the Security component. Click the Attack Series option in the list of parameters and choose the IPP Fuzzer option that was created in the previous steps. Click the Apply Changes button. The result should look like Figure 7.

Figure 7: Selecting the IPP Fuzzer Attack Series

Finally, click the Save and Run button from the left side of the screen. The interface will prompt for a name for the test and start running the IPP Fuzzer Strikes against the target device. Once the test starts running, we will see a progress screen like the one shown in Figure 8.

Figure 8: Running the IPP Fuzzer Attack Series

In the graph above, we see a short spike of packets, followed by a long series of almost no traffic. Once the test completes, we can look at the report and determine what happened. Figure 9 shows a sample report for this test.

Figure 9: IPP Fuzzer vs HP LaserJet

In the Strike Detail section of the report, we see that the “IPP Print Operation Long Job Name” Strike reported a maximum job size of 64 bytes (of a maximum tested size of 65528). All of the following Strikes reported a maximum length of zero while the last Strike indicates that zero operations were accepted. If we try to telnet to the IPP port of the target device, we can see that the service is down and is no longer accepting requests. It appears that an IPP Job Name that is 65 bytes long will crash the HP LaserJet IPP service. The bandwidth graph showed a quick spike as the first 64 requests were sent to the IPP service, but after the 65^th test, the IPP service stopped responding and the subsequent Strikes quickly gave up after not being able to connect.

The second target we want to test is a Ubuntu Linux system running CUPS, which happens to expose an IPP service as well. To change the target IP address, we access the Network Neighborhood we created, click on the External tab, change the IP address, and click Apply Changes. The next time we run this test, the fuzzer will target the Ubuntu system instead. Figure 10 shows a sample run of the exact same test against a CUPS IPP server.

Figure 10: IPP Fuzzer vs CUPS

In this test, we see that the service refused requests with field lengths over a threshold (~14100), but did not crash. At the end of the test, the IPP service was still online and functional, and the logs of the CUPS server did not show any fatal errors. Since the IPP service did not crash during this test, all six Strikes were able to run to completion, extending the test time to about one hour.

The BreakingPoint architecture makes it easy to integrate fuzzing with performance, application, and live exploit testing. This combination provides a way to flush out bugs that only trigger when the device is under a heavy load, something that is difficult to do with other products on the market. While IPP is a simple example, the same technique applies to all of our fuzzers. We design them to test live services as well as content-inspection devices that route traffic. The reports make it possible to determine exactly what requests can break a service and allow our users to reverse-engineer a content-filter's traffic normalization rules.

Building a Better Botnet

Earlier this year, Dennis and Kyle put together a screencast overview of BreakingPoint's capabilities for simulating botnets. Today, I'd like to go into a little more detail about the AppSim component that's most associated with malicious traffic, IRC.

While most of our AppSim components are used to simulate good, normal traffic -- HTTP, Oracle, VoIP, etc. -- IRC stands out as a protocol that is, for better or worse, often associated with malicious traffic. Of course, not everything on IRC evil. However, IRC is the traditional medium for setting up command and control channels for botnets, those armies of compromised end-user machines performing attacks such as DDOSes, spam campaigns, vulnerability scanning, and other evil deeds. So, for most ISPs and enterprises, it's not very useful to just detect the presence of IRC traffic; it's much more worthwhile to detect particular patterns associated with botnet activity. Given that, let's build up a quick botnet that your fancy expensive detection device claims coverage for to see what sticks. In this example, we'll model a fictional botnet (DdosHax) that uses channel topic changes to signal to compromised hosts, from the point of view of the bot herder's local network.

First, we'll fire up the IRC AppSim Superflow editor, and populate a new superflow with the IRC actions we're interested in:

That gives us a channel join, a couple of other users joining the same channel (our bots), a topic change (the command), and the bot responses (command acknowledgements).

Next, we'll fill in the salient details of the bot herder's user name and the network he's joining.

And finally, we'll set the options for our flow actions so this all looks like a typical botnet communication. We can set the command of "!sf 192.168.1.1" to the channel topic:

And generate a response from several peers:

Here, we can see that the naming convention is pretty flexible for the number of bot hosts we'd like to simulate. In this case, we set a range between "a0" and "b9", which gives 20 remote peers, all of whom include the range indicator in their names (zombie-a0 through zombie-b9). If we'd prefer randomized names, we could uncheck the "Include Range in Name?" options, and if we wanted completely random names, we'd uncheck Peer Nick Prefix as well. If you wanted a hundred peers, simply set the range from 1 to 100, or 'aaa' to 'adv', or whatever other range naming scheme that strikes your fancy.

Once this AppSim is run on the wire, we can take a look at the traffic with a regular packet sniffer:

That looks pretty evil to me -- and best of all, the whole procedure took about five or six minutes to set up from scratch, without all the futzing around with setting up real IRC servers and clients, writing scripted actions and responses, or editing up packet capture files.

Firewall Testing WebCast: Video Replay

We put up the full video of our firewall testing webcast that was held last week and I wanted to be sure to share it with all of you. It is embedded below.

Just a reminder that if you also want to receive BreakingPoint's firewall test methodology when it is published you can just put your info within this form. Additionally, we have put together an online room on FriendFeed that gathers all the latest info, thoughts, strategies and opinions on firewall testing. Check it out and let us know what you think.