The Problem With IMIX

Andre Gironda was kind enough to post a comment regarding IMIX to yesterday's blog post, "The World is Lacking Background Traffic." Basically Andre writes that IMIX might be outdated but it's out there. This was something that I was planning on talking about in the future, but since I got the perfect setup, I'll do it now. I have no issue with IMIX, but I do have an issue with thinking that it can provide you good data for how a device will work, performance wise, in your network. Andre is right, it's a bit outdated, in fact I believe it's 7 years old this month and the NLANR (National Laboratory for Applied Network Research), who provided it, had it's funding dry up 2 years or so ago.

For those that don't know what IMIX is, I will steal from a description I found on CAIDA's site: "IMIX derives from analysis of NLANR traces and is tri-modal (e.g., 58% at 40 bytes, 18% at 576 bytes, and 23% at 1518 bytes)."

So to prove this out - I took a Juniper Networks device that is content-aware and ran three benchmarks. It was fairly easy since this device used to be in my network! The three benchmarks are as follows:

1. Setup a test to run IMIX at 20 megabits (yes, my device is wimpy);
   
2. Setup a test to run BreakingPoint's Application Profile for Enterprise Traffic (using release 1.2);
3. Monitor it via SNMP all day to see when I started dropping traffic (yes I have a 10-megabit pipe to the Internet);

(It should be noted that I knew the device couldn't handle our 10-megabit pipe to the Internet, I know this because we had to upgrade it to a larger device after we changed from a T1 to a 10-megabit pipe).  Here's a graph showing the results:

As you see, the IMIX told us the device could perform 18.9 megabits per second without packet loss. The Enterprise Application Profile clocked it at 4.6 and the Actual was fluxing between 4.7 and 4.9.

The World is Lacking Background Traffic

When I inherited the system test and quality assurance (QA) groups, along with my other responsibilities as Director of Engineering at my last Company, one of the things the QA engineers would do to make my head tilt was to configure all the tests with background traffic. Why would that make my head tilt? It was the background traffic they used; a popular traffic generator running layer 2 traffic with random data. I immediately said, "That's not background traffic - that traffic doesn't exist on any network". The engineer in charge of running the "background equipment" said that he didn't want the traffic to interfere with the test. Of course my next question was, "Then why run the traffic at all? If the background traffic is not to interfere with the traffic and it's not ANYWHERE near realistic - what's the point?"

I wanted realistic IP traffic the next time I visited and a week later I saw he had IPv4 traffic with random data being sent. Granted, this is a huge improvement over random Ethernet frames, but a long way to realistic traffic. I acknowledged this was a good improvement but wondered if he had any peer-to-peer traffic? The QA engineer asked why and I explained the recent feature we added for traffic shaping peer-to-peer traffic. He explained that this would get in the way. I suggested that he should be running P2P traffic all the time, "Think about it; if we have traffic shaping off - peer-to-peer traffic IS the background traffic and not the primary traffic." I told him to have 20 percent of the traffic peer-to-peer by next week.

Next week I come by and sure enough 80% of the traffic is IPv4 with random data, and 20% is a P2P datagram. Now, first knock, he didn't get the gist of what I was saying; he didn't take a look at all our features and pull in different traffic sources, he did the literal translation. Oh well.  However, and here is the second knock, there were 5 "streams" of traffic being sent. A stream being an instruction to the popular traffic generator to send out traffic. So what we had were first four packets of IPv4 with random data, then a single packet that is part of a P2P stream, then repeat. My not so surprising reaction, "Hey this isn't P2P, it's a packet."

Fast forward one week...the QA engineer is there with the rep for the test equipment. The sales engineer for the test equipment vendor has his head held high and says "I have a very complex TCL script here that will recreate the P2P flow exactly!" I pull out a copy of Ethereal (now Wireshark), take a look and the traffic is the same, except the IP address repeating over and over again. "This isn't background traffic, it's a single P2P flow played over and over again."  Pause.  Finally the SE for the test company says, "What do you expect, actual traffic?".

Yes!

 
We need to change the way people test, and it's a fundamental shift. After all, your product is going into a network with an average of 30+ different applications on the wire, don't you think your product should handle that traffic correctly? Background traffic needs to be real and this, along with other similar stories, led to the creation of BreakingPoint.  We understand the need for repeatable testing, so we want to dig into what background traffic should be, why you should do it this way and what NOT to do.

Background - it's a lot more important than most people realize.

IPS Test Methodology Intro

We've been busy with a lot of exciting developments in BreakingPoint Labs, one of which you will hear about on Tuesday. I thought that I would whet your appetite a bit though and post up this intro video starring our own Dennis Cox and Rik Maerz.  In this video, one of eight we will be posting along with materials, Dennis and Rik introduce the upcoming Intrusion Prevention System (IPS) test methodology being put together by BreakingPoint for testing a range of IPS devices.

I personally found the videos and ultimately the test methodology to be extremely helpful in understanding the full capabilities of our products.  What resonated with me the most was how you can apply your tests to measure a variety of instances, vulnerabilities, application protocols and more.  This remains true across different devices.  As you will hear in the video Dennis and Rik generated the methodologies by testing a spectrum of devices (and they also do a clever job of censoring the actual names).

 Go check out the intro here and stay tuned for episode 2, a discussion of past IPS tests.

Deep Packet Inspection (DPI)

Every month or so I write a blog post for dpacket.org. For those that don't know, dpacket.org, it's the online community site focused on deep packet inspection. It's run by Kyle Rosenthal out of San Francisco. If you are a vendor making DPI devices or interested in DPI in general I suggest you join and check it out. It's a new community and it's just starting to grow.

Most of the time I write about Network Processors, I'm very passionate about that technology and have been lucky to work on them throughout my career.

May 5th 2008: Common Problems with Network Processors
March 3rd 2008: Moving as Fast as Intel
Feburary 4th 2008: Deep Packet Inspection and the Role of Network Processors

 

Coming Full Circle

 

At my previous company the CEO at the time, John McHale, decided that we need to prove to the world something that we thought we already knew. That we had built the best Intrusion Prevention System (IPS) on the planet. Our mission was to have someone prove it. We didn't want to pay someone to prove our claims, we wanted a bake-off of every possible IPS vendor at the time and independently determine the winner. All parties should pay to be tested and the test should be the same for all vendors. That third party was NSS Labs.

NSS Labs had gained a reputation as a very strong test lab that published their full test methodology for the scrutiny of everyone. It was a published set of tests and you would be measured against these specific goals. At the time NSS had three levels, Tested (you showed up), Approved (works pretty good) and Gold (best product in the class). They rarely handed out Gold, and still rarely hand out of Gold. At the company we said Approved would be great and a huge achievement for our first time being tested. John said, nope he wanted Gold. We explained that Gold would be near impossible, we would have to get 100% on every test and that alone would take 6-9 months of development effort if we focused on nothing else. John made a gutsy call and the wisdom was - if we are Gold with NSS then we should never lose a deal because we are the best product. So we spent every night and weekend for many months recreating the NSS test lab and running tests over and over again. At the time NSS was in France and we visited Bob Walder (CTO of NSS Labs) many times to talk about the testing and make sure our results equaled his. In other words, our testing harness was correct.

We put in so many optimizations and changes in those months that it was amazing. This wasn't features; this was performance optimizations, UI changes and all the spit and polish that you never get around to do. These are the defects in your bug tracking systems that just keep getting moved to future because they aren't "important" enough. Well it turns out if you actually spent the time on those items, something amazing will happen.

We, TippingPoint, received the NSS Gold award. It was all worth it, the product was 10x more stable, faster and more accurate. Everything we did for NSS made the product better for everybody else and really made headway for sales. Not just as a marketing vehicle, but also as a superior product.

Last week we announced that NSS had switched to using BreakingPoint as it's primary test product. NSS was one of the reasons we started BreakingPoint.  The lab setup was complicated, hard to use, results would vary based on the stability of a number of the key test vendors. We would run them 3 times in a row just to make sure the results were the same, often from a reboot on the test gear. I wanted a device that could replicate all of NSS Labs' tests and then some. In fact, I wanted a device that would allow NSS Labs to create more realistic tests. Tests that were impossible in the past.

It's really neat that we have come full circle.

The IP Traffic "exaflood"

A colleague passed along this paper to me last week from The Discovery Institute.  Bret Swanson and George Gilder go into detail on the increasing amount of data, particularly 'rich media' traveling over our broadband pipes.  They estimate that in the U.S. by 2015:

movie downloads and P2P file sharing could be 100 exabytes

video calling and virtual windows could generate 400 exabytes

“cloud” computing and remote backup could total 50 exabytes

Internet video, gaming, and virtual worlds could produce 200 exabytes

non-Internet “IPTV” could reach 100 exabytes, and possibly much more

business IP traffic will generate some 100 exabytes

Certainly on a personal level we can all relate to the immense amount of data we are putting out there, whether it is the home movie or a large amount of product demonstrations (had to get that plug in...).  If you simply think about the future of IPTV, not to mention everything else, we are pushing towards immense levels of IP traffic.  We were also reminded over Twitter that Cisco has already started talking 'zetta' levels. Bret and George write:

"No one can know just how fast Internet traffic will grow, making capacity planning for network backbone engineers a difficult task. In fact, like network data itself, the macro trends discussed in this paper are “bursty”—they are unpredictable and potentially explosive. Networks will thus require much more raw bandwidth, yes, but also sophisticated new techniques to finely manage traffic flows. We do know for certain that if consumers and innovators are to enjoy the rising tide of the Exaflood, capacity in access networks to homes and businesses will have to expand by a factor of between 10 and 100 in just the next few years. One- and 10-megabit links will have to become 25- and 100-megabit links."

This is dead-on correct. The network backbone, and it's ability to handle the growing traffic, is certainly a topic of great importance.  It puts even more emphasis on the ability to test the equipment that makes up the backbone, and test it in a way that proves it can handle today's IP traffic AND the sudden burst of traffic growth that folks see over time.  Obviously we think this is one of the most important aspects to these 'growth predictions'. 

It's beneficial to prognosticate on IP traffic levels in 2010, 2015 and beyond.  People get excited by these prospects, but one area we certainly need more light on is the fact that as IP traffic continues to grow and NEMs continue to push innovation the testing industry has been somewhat stagnant in it's ability to validate the backbone.  It reminded me of the quote Des had in the NSS Labs news last week (emphasis mine):

"Legacy test products have not kept up with the pace of innovation and are not architected to properly test next-generation content-aware equipment. Evaluating and certifying content-aware equipment by testing with a single application, like HTTP, may seem suitable for other testing providers, but it is irresponsible when today's network devices are hammered by an increasing volume of business, recreational and malicious traffic.”

NSS Labs & BreakingPoint

Welcome to BreakingPoint's new blog.  Don't fret however, you're still going to get the info you've come to expect from BreakingPoint, but now we have some additional bloggers and new content.  As the 'new kid on the block' I'll be posting along with the entire blog team you see in the right sidebar.  My focus will be on providing the latest company information, including lots of video and podcasts that I (and hopefully you) find compelling.  As we move forward I'll elaborate a bit more and get feedback from you all, but let's get to the first thing on my mind.

You may have already heard about the recent announcement with NSS Labs and their selection of BreakingPoint as the standard for testing next-generation content-aware network equipment. I shot some video on my FlipCam (next time remind me to get a bit closer) at a recent company meeting where Des Wilson, our CEO, talked about the news and why it was important.  After watching the video I realized it gave added information about the news, things you simply can not get into a press release.  A few minutes of editing later, voila!  Also, be sure to head on over to NSS Labs' blog for more industry insight.