Testing With Twitter; The Importance of "Real" Applications
We have talked here about the overall importance of testing…particularly the need to test with REAL
application traffic, live security strikes and doing it all at speeds of 10Gps and faster. These days that first criterion, testing with real apps, is getting more and more important. A great example comes from many of the applications popping up, seemingly every day, in the ‘social networking’ and Web 2.0 space.
Twitter is one of those social networking apps that emerged seemingly overnight and rapidly became a social phenomenon. It transcends description but has caught fire and become so mainstream, so fast, it is now regularly mentioned on CNN, USA Today and even CSI (Miami, of course). This exposure, and the applications usefulness, has created enormous growth. Depending on who you talk with there are an estimated 340,000 public Twitter accounts with 60,000 new accounts starting each month, or potentially 2.2M Twitter users (as of July 22nd). Twitter has also seen great usage as a customer service tool, a social network and a lifesaver.
With great success comes great disappointment and scrutiny of course, and Twitter has been getting plenty of poor coverage because of their inability to keep their network functioning at peak performance. It’s a worry of every company, from Fortune 500 to a start-up…if you provide a service that relies on a network, you best know that the components of that network can handle all the application traffic, security strikes and do it with blazing fast speeds. We take this to heart and have actually added Twitter to the real applications folks can now perform protocol testing using BreakingPoint (check out our TweetLease).
Folks like Biz Stone at Twitter and other business heads (I’m talking to you Mr. Zuckerberg) must push their network equipment manufacturers to make sure they are properly testing network devices for today’s sophisticated application usage. Twitter is simply one example of apps that can affect your network performance, but it is something that was not tested on your network devices before they were deployed. Tomorrow there will be more and you need to make sure these are being used to test your equipment.
Of course, this topic brings us into the ‘architecture discussion’ and how it is crucial that your testing tool is being updated with realistic blended apps and security strikes on a regular basis, and by “regular basis” I don’t mean every three months. There is no telling where the next heavy use application traffic will come from but let’s just say my money is on Plurk…just because I love the name.
Latency and the Global Exchange
Earlier this week Cisco put out its study on global exchanges showing that there are at least 50 different cababilities an exchange needs to achieve peak performance. The study focuses on the top ten, which can be seen in the cross-hairs of this graphic. Coming in at number 5 was latency:
Ivy Schmerkmen at Wall Street & Technology has a terrific in-depth look at the study, including this nugget:
Before going through a major upgrade, the London Stock Exchange could handle up to 450 transactions at the peak second. After a four-year IT overhaul project, the LSE's new trading system TradeElect is now able to handle up to 2,500 transactions at the peak second and it cuts the time it took to process an order to 10 milliseconds from 140 milliseconds. Low-latency trading platforms are critical to exchanges that want to go after the algorithmic trading business, according to the study.
This hit home with me since this week I had also been reading a report put out by TABB Group in April, "The Value of a Millisecond" where they estimated, “Up to 10 milliseconds of latency could result in a 10% drop in revenues.” I knew from previous work in the financial services industry that time was money...but wow. It is obvious that exchanges need to have the most up to date network equipment, but in the rush to get these devices deployed, measuring their performance and security is, d'uh, critical (you knew I'd bring it back around to test, right?).
The reason I'm knee deep in all of this research was because this week we announced our support for the FIX/FIXT protocol. FIX (Financial Information eXchange) is the global messaging standard for the electronic communication of "trade-related data" between financial firms such as banks, broker-dealers, exchanges, and industry utilities. If these exchanges need the latest and greatest network equipment, they need that equipment to have been tested using the FIX protocol (and many others, obviously). Just trying to do our part to grab those milliseconds back!
What Do You Think?
It has been a few months since we launched our new website and we wanted to get your reaction and thoughts. We put together a quick survey (9 questions) and we really want to know what you think. For your trouble you will be entered into a drawing to receive an iPod Touch or one of several $25 gift certificates to Amazon.com.
Click Here to take survey
Traffic Analysis
As I may have previously mentioned, I love traffic analysis reports. I have several locations where I grab information and a script that combines them and collates all the data for me on a monthly basis. One of the sites I get my data from is Internet2.
The data shown on the Internet2 backbone is vastly different in terms
of application percentages (less P2P for the most part). However, all
the other numbers line up with almost all other sources. Check this
out; 90% of all packets are between 1401 and 1500 bytes. Makes you
wonder what your test rig is set to? It also reminds me of our
conversation about IMIX, the description from CAIDA's
site again is: "IMIX derives from analysis of NLANR traces and is
tri-modal (e.g., 58% at 40 bytes, 18% at 576 bytes, and 23% at 1518
bytes)."
Netflow on Internet2 says 100 byte to 1400 byte packets are 9.15% and sub 100 byte is 0.46%, whereas IMIX is 18% at 576 bytes and 58% at 40 bytes. Again - why are people running IMIX? Crazy. After all, IMIX is based on real data, just like the data I'm provided, it's just I'm using current data. Unless your sending your device back in a time machine, I suggest you stop using IMIX.
My final point - how many applications are you testing - because Internet2 is a small backbone (in terms of users) yet according to the netflow data they have more than 40 applications running that they can measure.
P.S. 41.33% of the traffic is unidentifiable :) That's a big piece of data that's missing.
Do the Math
I love math, although the funny thing is, I was really bad at it going through school and had to have a math tutor in high school. These days, I actually consider going back to school to study math from time to time. Talk about a change. What intrigues me the most is the ability to use math to come up with clever ways to solve problems. This became apparent to me since I actually honed my math skills by majoring in Music Composition.
During freshman year my professor played a recording of a piece of music, roughly 5 minutes long. He would play the piece once and it was our job, after the piece was played, to write the general score down. The challenge; we could only write for the first 30 seconds and the last 10 seconds of the recording! Five minutes of music with 40 seconds of writing to get the general score? Impossible we all said, it would take hours to transcribe 5 minutes of music with this many parts. We would need to listen to it over and over again to get it right...we all stared at him like he was crazy.
By my Junior year I could do it with only 10 seconds of writing at the beginning of the piece and a second at the end. It was dead simple, because it was math. What you need is the data that starts it and ends it. You can determine the path from those two points. [NOTE: This applies mostly to classical music]
The same thing applies to network equipment testing - I see too many engineers test their equipment without data. What does traffic look like in the real world? Sometimes a customer will quote an RFC, but the RFC is 10 years old and the standards, while not documented, have changed completely. If you don't know what the network looks like, even just 10 seconds of the beginning and 1 second at the end, you can't test your product correctly. After all, just because you know networks have HTTP in it, doesn't mean that's the only thing running. To complete the analogy, it would be like listening just for the drums to figure out the tune; you need to hear/see the whole thing for a good picture.
This resonated with me again when I saw a competitor just announced P2P support. Just announced? P2P is over 30 percent of network traffic and some test tools are just adding it? Heck some don't even have it yet. If you don't have the data, things will never add up.
A Few More Bugs in Ruby
We use Ruby all over the place in the BPS appliance and for our internal tools, so the latest run of vulnerabilities in Ruby discovered by Drew Yao, discussed in detail here, really caught our eye. While analyzing them, the BreakingPoint Labs team found several new unpatched bugs in Ruby's parser.
These bugs are rather low impact because they don't really appear to be exploitable and because triggering already require enough access to run arbitrary Ruby statements. So we've decided just to publish them here for your enjoyment. Breaking things is just what we like to do here.
Too much recursion when evaluating a statement
Examples:
ruby -e 'eval(“[]”*10000)' ruby -e 'eval(“+0”*8000)'
Each of these causes so many calls to rb_eval() that it eventually runs off and accesses invalid memory. The second seems to be the same crash, but through a slightly different vector and leaves the program in a slightly different state.
Program received signal SIGSEGV, Segmentation fault.
0x0805891d in rb_eval (self=3084609960, n=Cannot access memory at address 0xbf143ebc
) at eval.c:2927
#1 0x0805a45d in rb_eval (self=3084831140, n=0xb7ddbeec) at eval.c:3467
#2 0x0805a45d in rb_eval (self=3084831140, n=0xb7ddbed8) at eval.c:3467
#3 0x0805a45d in rb_eval (self=3084831140, n=0xb7ddbec4) at eval.c:3467
...
#7815 0x0805a45d in rb_eval (self=3084831140, n=0xb7db5aa8) at eval.c:3467
#7816 0x08055e6a in eval_node (self=3084831140, node=0xb7db5aa8) at eval.c:1428
#7817 0x08062649 in eval (self=3084831140, ...) at eval.c:6490
#7818 0x08062a9f in rb_f_eval (argc=1, ...) at eval.c:5691
#7820 0x080602dc in rb_call0 (klass=3084836000, ...) at eval.c:5847
#7821 0x08061965 in rb_call (klass=3084836000, ...) at eval.c:6094
#7822 0x0805a98b in rb_eval (self=3084831140, n=0xb7ddcdb0) at eval.c:3488
#7823 0x08055e6a in eval_node (self=3084831140, node=0xb7ddcdb0) at eval.c:1428
#7824 0x0805644c in ruby_exec_internal () at eval.c:1634
#7825 0x08056490 in ruby_exec () at eval.c:1654
#7826 0x080564b2 in ruby_run () at eval.c:1664
#7827 0x0805434e in main ()
at main.c:48
Extremely long syntax error causes invalid access
Examples:
ruby -e 'eval(“a”*10000000+”$”)' ruby -e 'eval("a("*10000000)' ruby -e 'eval("\x01!"*10000000)'
When faced with a syntax error on a huge line like this, Ruby fails to allocate enough memory for the error message and dies trying to copy the message buffer. This seems to work with just about any line with a syntax error that is around 10 megabytes in size or more. The call to “alloca()” in parse.y tries to load the huge error string onto the stack and blows up. Here's the error and the code that causes the crash:
Program received signal SIGSEGV, Segmentation fault. 0x0809bc3b in ruby_yyerror (msg=0xbfe81f68 "syntax error, unexpected $end")
at parse.y:2539
In parse.y:
2538 buf = ALLOCA_N(char, len+2);
2539 MEMCPY(buf, p, char, len);
The Difficulty With Testing Blacklists
I was recently directed to Greg Hoglund's new blog, Fast Horizon. So far the few posts he's made have been excellent reading, and I generally agree with most of what he's been saying over there. Just recently he put up a post entitled "Whitelists are the new snake-oil" where he convincingly outlines both why blacklisting is a failing approach to stopping malware (if it ever really worked in the first place), and why the whitelisting approach that many vendors are moving toward is equally doomed to failure. Blacklisting is the current industry standard approach to stopping malware and is used in technologies such as anti-virus and anti-spyware software. Blacklisting is also the approach that most network security device vendors take to blocking network attacks and exploits. While firewalls are the obvious exception to this rule as they are essentially whitelist based, the vast majority of packets that get blocked, filtered, or identified by other network security devices such as IDS and IPS systems are done so via a match against a blacklist of traffic signatures. These signatures are analogous to fingerprints of the known malicious data that traverses the network. The following quote from Greg's post nicely sums up the difficulties with this approach:
"Blacklisting sounds ideal, but it doesn’t work. New malware emerges daily that has no corresponding blacklist signature. The malware must first be detected, and then processed. There is always a time window where Enterprises have no defense. Recent figures suggest that the AV vendors are falling so far behind this curve that they will never catch up with the deluge of new malware arriving daily. It can take weeks for a signature to become available.
This deluge of new malware is due to several factors. First, there is more money behind malware development than ever before. Second, we weren’t really that good at capturing malware in the past. Today, new malware can be automatically collected, without human intervention. The slow trickle of malware turned into a flood as honeypot technology emerged. Sensor grids can obtain new malware samples with efficiency - they automatically ‘drive by’ (aka spidering) malicious websites to get infections and leave open ports on the ‘Net so automated scanners will exploit them. In parallel to the automated collection efforts, cybercrime has risen to epic levels. Finally, the barrier to entry has dropped for the cyber criminal. Cyber weapon toolkits have become commonly available. Anti-detection technology is standard fare. New variants of a malware program can be auto-generated. A safe bet is to expect thousands of new malware to hit the Internet per day."
What he describes happening in the malware space is also happening within the attack and exploitation landscape. Network attacks and exploits are becoming increasingly more dynamic and harder to fingerprint for a number of reasons. As individual exploits increase the number of targets they can attack, which usually consist of various combinations of target software, the operating system running that software, patch-levels of both the software and operating system, the hardware architecture that both are running on, and any number of other target environmental factors, identifying a single exploit or vulnerability based on it's related network traffic becomes much harder. When you then factor in all of the evasion techniques such as data massaging, reordering, randomizing, encoding, encrypting, tunneling, and any number of others, many of which have become standard in publicly available tools such as the Metasploit Exploitation Framework, the problem gets exponentially harder and you begin to approach the number of potential variants of a single attack or exploit that the malware folks are seeing in their space.
When faced with an ever-changing hostile network environment of malicious traffic, blacklists of signatures and filters that are developed to detect this data must be equally robust, and must be tested and verified with test cases that approach the dynamic nature and variation that you would see in the wild.
For further reading on dynamic security tests, please see my previous post entitled "File Format Vulnerabilities and Dynamic Exploit Generators" from a few weeks ago. It helps make Greg's point in that it illustrates how many variants of a malicious file must be properly detected when even only a handful of fields of data within that file can change and it still be a working file format exploit.
DPI for Bandwidth Shaping? Where's the news?
There seem to be several conversations happening on service providers using DPI to throttle traffic. My reaction? What's the big deal?
This is another case of the media looking for a story that isn't there. Service providers of all sorts have been using bandwidth shaping [DPI] since the early 90's. Technologies such as Asynchronous Transfer Mode (ATM), had bandwidth shaping in their specifications and ATM was one of the core transports of service providers. UBR , CBR, VBR anyone? Why is this a surprise?
It reminds me of the Global Warming debate. Al Gore has been talking about it since the 70's - yet when he was vice president nothing got done about it. Not his fault, he's just one man. But if he were vice president now with Global Warming such a hot topic, pardon the pun, the media's pressure would bend the will of the people to solve the "issue".
Just yesterday, based on some guys figuring out what statistical packet analysis can get you, everybody went crazy again on this topic. Statistical packet analysis was around WELL before this paper. A bunch of guys from MIT started Mazu, which built a company based on this concept. Arbor is another company that leveraged this, well before Mazu did. So what's the hype? I guess it's a slow news day.
More Intrusion Prevention Discussion
A few weeks ago we talked Intrusion Prevention Systems (IPS) testing and we posted up our videos and methodology. We've gotten some great feedback from the community and will be using some of it in our next set of videos (and we are always looking for more input). With that in the back of my head I was interested to see The Tolly Group's most recent custom test of TrustWave's Intrusion Prevention Appliance. You can download the test summary and look at how they are going to test (full disclosure that they are using BreakingPoint in the testing). We also have more info on security equipment testing.