There are new vulnerabilities popping up all the time, and some go unfixed for days or even months. But NetSupport Manager has been out for a full year now, and sad to say, the vulnerability that was reported still exists within the product. Today we review this vulnerability, not to pick on NetSupport Manager, but to illustrate an important lesson for all of us about auditing your network and testing for vulnerabilities.

First, for those who do not know, let’s look at what NetSupport Manager does. Here is a quote from the website:

"The latest evolution in Remote PC support and desktop management . . . Designed to operate over your network, via mobile communications or the Internet, securely and without the need for firewall configuration."

Usefulness of the product aside, a stack-smashing attack was reported while using this product a year ago. This attack would allow anyone to seize control of the machine. Smashing is typically accomplished by overwriting a buffer that sits on the stack, which then overwrites the return address of the function and exception handlers. Crafting the bytes carefully, a hacker can then have the program run arbitrary code.

The key point, at least for this post, is that despite a very nasty exploit against this software that has been public knowledge for over one year, you can buy it today. You might even have it deployed already.

Auditing and Testing Vulnerabilities in Your Own Network

Here are a couple of questions to keep you up at night:

1. How many third-party software programs, such as NetSupport Manager, does your company use?
2. How would you know that such a vulnerability exists in your company if the vendor doesn't provide an update?

Here is the two-pronged answer:

1. Do a periodic audit of your software.

A quick Google search for "NetSupport Manager vulnerability" provides, as the fourth link, a warning about this particular attack. (Note the release date of 2011-01-10.) It is important to continue searching for information about vulnerabilities within the software and devices that you use in your network.

2. Test your network using the attack.

Use a box, such as the BreakingPoint Storm or FireStorm, to generate traffic. Our box will generate this exact attack and test your device as well as the IDS/IPS that is supposed to protect you. This is critical, particularly after reading about the latest published vulnerabilities.

But there is more to it than simply auditing and testing. Let’s discuss a scenario. In an earlier blog post, I mentioned a job candidate waiting in the lobby who may have access to your VoIP system via the telephone. While that post described a particular VoIP implementation, there is another plausible avenue of attack:

Companies typically perform interviews in a conference room that will have at least one data port available. Good companies have several people interview a candidate to ensure that there is a good team fit and, for a high-tech company, that all the technical issues are covered. Between these interview sessions, a candidate can be left alone in the room - behind your firewall. Have you ever interviewed someone and decided there was no way you'd ever trust him in your company? How long would it take him to send a malicious packet or exploit a known long-term vuln such as the one presented by NetSupport Manager?

How You Can Prevent Vulnerability Exploitation

Perhaps my scenario above is a bit dramatic, but it is certainly realistic. And it can be prevented, because many vulnerabilities are publicly known for a long time - a year in the case of NetSupport Manager. So what can you do?

For the programmer:

• Remember: Stack buffer overflows still get discovered. When there is a push to get your code out the door on time, it's easy to miss this. A common pitfall is prototyping code and not going back to make sure it is correct.
• Never trust user input, and don't trust the caller to your function to have validated the size of the buffer.
• Unicode or http encoding is longer, so a string of "://" may have a "length" of 3 bytes, but your application may have converted that somewhere else to "%3a%2f%2f," which is 9 bytes.

For the business owner:

• You cannot wait for a vendor - you must periodically audit all your software for vulnerabilities. The Google search above took only seconds.
• Split the audit work up, and offer something very nice to the employee with the best find. It'll be worth every penny.
• It's basic knowledge but overlooked: All visitors to your company need to be escorted at all times.

Oh, and don’t forget: Test.

The room was full that day in February 2009 with an audience eager to witness what, at the time, would be the fastest public firewall test using stateful application traffic. Minutes later the BreakingPoint Storm CTM had put the Juniper SRX5800 through its paces, achieving 109 Gigabits per second (Gbps) of true application traffic. On that day the SRX5800 became the first network device tested with more than 100 Gbps of blended Layer 4-7 application traffic. During the past few years we've seen even more public firewall tests using real-world conditions, from a live test of the Cisco ASA 5500 during RSA Conference 2011 using applications ranging from eBay to Facebook, to today's announcement concerning the Fortinet FortiGate 5140B.

Fortinet, as a long time BreakingPoint customer, has embraced the idea of real-world testing using the actual mix of network traffic seen where their devices are deployed. When it came to capturing the performance data they would use to market their high end firewall, they chose to follow the same protocol. They performed their most recent test they used five BreakingPoint FireStorm CTMs to demonstrate their product under load from 526 Gbps of real-world traffic from applications such as Facebook, Pandora, and AOL Instant Messenger. 

Although the idea of real-world testing hasn't reached every corner of our industry, it is a hopeful sign for everyone that companies such as Fortinet (and Juniper, Cisco, and others) are taking this seriously in order to represent exactly what you should expect from their devices when deployed.

Here's a brief video discussing the test and Fortinet has more information on their site:

When you spend time on your mobile phone, you use dozens of different applications: Facebook, Netflix, email, YouTube, and more. And behind the scenes your carrier is registering and authenticating your device on the network, establishing HTTP, Android App Store, and other connections over bearer channels while blocking malicious attacks. THIS is the reality of mobile traffic today, and it can be a challenge. So when I was surfing around on my iPhone the other morning I was thrilled to see a new mobile security test, “Crossbeam and Spirent Partner to Define Real-World Security Test Methodology for Mobile Network Operators.”

Immediately I was intrigued. Since BreakingPoint is a leader in real-world network security testing for wireless and wired networks, we are always looking for interesting new security research, especially in the mobile space. But then I dove into the actual test. What a disappointment.

This test, while being touted as a test of “mobile firewalls for 4G-LTE networks” that was done “under extremely demanding real-world conditions,” was in truth devoid of realism when it comes to testing any firewall being deployed within a mobility network. It reminded me of Mike Myers’ Linda Richman, who would probably say, “The Crossbeam mobile security test was neither mobile nor security. Discuss.”

Unfortunately this isn’t a joking matter. Using tests like these, which falsely claim realism and security, is what has led to our networks being more vulnerable to attack. Let’s take a look at the three main parts of the test listed in the title – mobile, security, and real world –and why each was actually missing.

Mobility Testing Truth #1: Create Actual Mobile Network Behavior

First, let’s talk about the mobile part. Crossbeam conducted the test on the SGi interface. For those not versed in mobility-speak, this is the interface between the mobile network and the Internet. That means there was no GTP encapsulation, no LTE or 3G signaling, no firewall inspection of mobile traffic, absolutely nothing at all in this test in any way related to mobile networks [p.6 of the test report] besides the title optimized for search engines. This wasn’t a mobility test, for the same reason that sitting by the pool all day isn’t the same as swimming laps. A mobile network is a complex system of devices, technologies, protocols, and more, and as you can see in the diagram below, the SGi portion is simply the last part out to the Internet (or PDN in telco terms).

Mobility Testing Truth #2: Use Application Traffic Seen on a Mobile Network

Next, consider the “real world” part and how you use your phone each day. A mobility test must use application traffic that is actually seen on mobile networks, in the appropriate ratios, and behaving the way real traffic would. As a starting point, you can use a mobile traffic mix such as one of those researched by our own Chris Adams’ mobile traffic analysis, which contains a mix of encrypted and clear text web, mobile flash, streaming audio and video, and peer-to-peer file sharing.

Also, you can mirror the traffic mix published in the most recent Sandvine report, and be sure to include Facebook, Skype, and Netflix, which currently constitute between a third and a half of mobile network traffic during peak hours. Netflix and other streaming video would be particularly important to include, as documented in the recent Cisco report, which notes that not only does mobile video constitute a large percentage of mobile traffic now but it’s also projected to grow to 66% of mobile traffic by 2015. For a true mobile network simulation, you’d probably also note (as did the Cisco report) that roughly half of the traffic on mobile networks comes from laptops and netbooks, which will have different application traffic characteristics than smartphones do.

But in order to correctly model mobile network performance and understand the user experience, you must go beyond looking at the individual applications and make sure each mobile user is using a blend of these applications, along with the associated ancillary protocols (such as DNS to resolve the servers for my applications, email, and Twitter to distribute interesting stuff), because you must be able to model the interaction of the multiple applications in use.

Instead, while the report discussed the use of “real-world” applications, it was actually just simplistic HTTP grabbing five URLs with a few JPEG images [p.7]. It’s 2012, HTTP is just a transport – calling it an “application” is quite a stretch. Reading further, I see that the various applications in use weren’t even used together by any of the endpoints – 95% of them did HTTP only, 2% did SMTP, and 1% did DNS only [p.7]. That’s not the way the real world works, and evaluating web-request performance without considering the requisite DNS latency is faulty; it’s simply a limited best-case viewpoint (in direct contrast to Spirent’s stated objectives). The table in the Spirent blog post where DNS is detailed is slightly misleading; records don’t resolve to URLs, they resolve to IP addresses, and if you read the actual test results it’s clear that the DNS results weren’t used in the HTTP requests.

Real users want high performance. They don’t care if their web performance is slow because the firewall can’t process DNS or can’t process HTTP. They just care that it’s slow. And in this test, the two protocols were in no way correlated or interdependent, so bad firewall DNS processing wouldn’t be reflected in any measurement of HTTP user experience. And although Spirent stated in their blog post that they were trying to model real user experience, the omission of DNS in the actual user experience reflects either flawed test design or limited test-tool functionality. It’s not clear from the report what is the case.

We will assume that the statement in the Spirent blog that the latency variations they measured showed “only tens of nanoseconds of impacts on the Web browser page render time” is a typo, since the table on p.25 of the actual report shows variations from 10.2 ms to 121.0 ms – and the actual measurement in question was HTTP transaction latency, not web browser page render time.

Mobility Testing Truth #3: Firewall Testing Should Involve Security Attacks

And finally, we come to security, which is kind of important when we are talking about testing a firewall. I invite you to download the report and find a single security-related test. Were there any attacks detected or blocked? What about application-layer DDoS attacks or content inspection for all those emails and web pages? And it would seem obvious, but where was the use of mobile malware during a mobility security test? Security in this test was conspicuous only in its absence.

The Truth Hurts

At the end of the day, this test was neither mobile nor real-world nor about security. It demonstrated that eight Spirent appliances can produce 107Gb of simplistic traffic at no more than 97% CPU utilization and that a properly configured Crossbeam chassis running Check Point software can forward that amount of that type of traffic.

This post is not a condemnation of the quality of any of these products, nor their performance. Instead it is important for our industry not to look at this test and confuse it with an actual mobile security test. Real-world results, using an actual real-world test, would be very different. And relying on these types of tests, whether done by a vendor, a lab, or even yourself, creates a false sense of security that ultimately results in degraded performance and security.

That’s the truth – and sometimes the truth hurts.

Related Resources:

A Visitor's Guide to Telecom-Land

LTE Diaries: GTP Tunneling

Testing Mobile Network Infrastructure Easily and Economically at Massive Scale

During the last two weeks of the year we are recapping some of the most popular topics covered on the blog during 2011. So far we have reviewed mobility testing, security research, DDoS, and now today, cyber range deployment. Cyber ranges are critical tools used to recreate cyber war conditions in order to harden IT infrastructure, train cyber warriors, and perform cutting-edge cyber security research.

Take a look at some of our top blog posts around cyber range deployment in 2011:

Cyber War Testing and Training Exercises: EUCOM’s Cyber Endeavor and Combined Endeavor

“Cyber warfare training” is emerging as a new military discipline, and the U.S. military has now made cyber security education and conditioning mandatory during basic training. Cyber warfare training is done in various ways, from in-depth classroom studies to live exercises using federated cyber ranges. It has become obvious that nation states and military organizations must prepare themselves and train their personnel to recognize, prevent, and combat cyber attacks. This post dives into how the U.S. European Command (EUCOM) is conducting cyber range exercises.

Cyber Range Strategy: How to Get Small and Arm Defenders, Stat

In an era of steep budget cuts for the U.S. Department of Defense, the objective in government circles has been to “get small” — to do more without spending more. Although some budget allocations for cyber security may be protected from cuts, the imperative to get small still makes sense. Smaller typically means more agile, easier to deploy widely, and, of course, less costly. Use this post to learn how to get small and arm cyber defenders.

Red Team, Blue Team: A Better Approach to Cyber Security Training

Every organization, whether part of the government or the private sector, needs “battle-tested” IT personnel in order to defend its networks against attack. The most effective way to provide this experience is to recreate the exact scenarios, no matter how nefarious, they will see in the real world. This two-part post goes into 'cyber war-gaming', exercises that bring IT personnel from different specialties into color-coded red, white, and blue teams that perform specific roles in attacking and defending IT infrastructures.

Accelerating the Deployment of the Evolved Cyber Range

The above posts show how organizations worldwide face a dangerous shortage of personnel with the skills required to defend against cyber attack. This urgent situation is made worse by the weaknesses and vulnerabilities that continue to pervade critical IT infrastructures. Cyber range deployment helps answer these problems. Leveraging BreakingPoint’s extensive work in building cyber range technology, this white paper details how to deploy this evolved simulation environment.

Using Cyber Ranges to Arm, Train, and Certify Personnel

Earlier in the year I sat down with BreakingPoint’s Director of Product Management to discuss how organizations such as DISA, EUCOM, and Northrop Grumman have deployed cyber range technologies. Now you can revisit this informative webcast.

Additional Blog Rewind 2011 Topics:

2011 Blog Rewind: Mobility Testing

2011 Blog Rewind: Network Security Testing

2011 Blog Rewind: DDoS Testing

During these last two weeks of the year we are recapping some of the most popular topics covered on the blog during 2011. Today we take a look at network security testing and blog posts that detailed some of the very latest in security research. The threat landscape continues to shift radically because we have to worry about more sophisticated attacks, evolved use of vulnerabilities, and an expanded target with the growth in mobile malware. This made for some very interesting blog posts.

These are some of the topics that made our list of network security testing blog posts for 2011:

From Patch to Proof-of-Concept: MS10-081

In this blog post, one of the most read of 2011, we look at how our security research team reverse engineered a Microsoft patch to understand a vulnerability, then we go through the proof-of-concept exploit. And at the end of the post a screencast shows you how to execute the exploit yourself.

The CISO's Guide to Reducing Mobile Malware Threats

By 2015, more people will access the Internet through a mobile device than through a wired Ethernet connection. Unfortunately, adoption of good security habits isn’t keeping pace with the adoption of mobile computing. In this two-part series we provide some insight into the overall threat of mobile malware and pragmatic tips on how enterprises can educate employees to stop the spread of this hazard.

Creating the Son of Stuxnet

Many people thought Stuxnet, and SCADA-based attacks in general, were a rare danger and one they should not worry about. However, after the latest rounds of SCADA attacks on U.S. infrastructure, this topic has again become critical for network security. In this post Pam O’Neal and Mike Hamilton actually recreate Stuxnet and use it to test the resiliency of infrastructures.

Microsoft UDP Vulnerability: A Tester’s Perspective on MS11-083

Again our security research team shows off their chops and takes a look at a Microsoft UDP vulnerability. But this time we show how to use our testing equipment to recreate this vulnerability in a matter of hours, when some people thought it would take days to simulate. And we also show some possible concerns your company might have about the vuln.

Application and Threat Intelligence: Network Testing with Current Applications and Attacks

The application traffic and security threats traversing networks and data centers evolve all the time. This we know. But what a lot of folks don’t know is how up to date their testing gear is when it comes to the latest application and threat intelligence. Our colleague Steve McGregory provided an inside look into the BreakingPoint Application and Threat Intelligence (ATI) research team and how they keep our users up to date with the very latest intelligence.

Additional Blog Rewind 2011 Topics:

2011 Blog Rewind: Mobility Testing

2011 Blog Rewind: DDoS Testing

2011 Blog Rewind: Network Security Testing