Crossbeam Mobile Security Test Was Neither Mobile Nor Security. Discuss.

By Scott Register

When you spend time on your mobile phone, you use dozens of different applications: Facebook, Netflix, email, YouTube, and more. And behind the scenes your carrier is registering and authenticating your device on the network, establishing HTTP, Android App Store, and other connections over bearer channels while blocking malicious attacks. THIS is the reality of mobile traffic today, and it can be a challenge. So when I was surfing around on my iPhone the other morning I was thrilled to see a new mobile security test, “Crossbeam and Spirent Partner to Define Real-World Security Test Methodology for Mobile Network Operators.”

Immediately I was intrigued. Since BreakingPoint is a leader in real-world network security testing for wireless and wired networks, we are always looking for interesting new security research, especially in the mobile space. But then I dove into the actual test. What a disappointment.

This test, while being touted as a test of “mobile firewalls for 4G-LTE networks” that was done “under extremely demanding real-world conditions,” was in truth devoid of realism when it comes to testing any firewall being deployed within a mobility network. It reminded me of Mike Myers’ Linda Richman, who would probably say, “The Crossbeam mobile security test was neither mobile nor security. Discuss.”

Unfortunately this isn’t a joking matter. Using tests like these, which falsely claim realism and security, is what has led to our networks being more vulnerable to attack. Let’s take a look at the three main parts of the test listed in the title – mobile, security, and real world –and why each was actually missing.

Mobility Testing Truth #1: Create Actual Mobile Network Behavior

First, let’s talk about the mobile part. Crossbeam conducted the test on the SGi interface. For those not versed in mobility-speak, this is the interface between the mobile network and the Internet. That means there was no GTP encapsulation, no LTE or 3G signaling, no firewall inspection of mobile traffic, absolutely nothing at all in this test in any way related to mobile networks [p.6 of the test report] besides the title optimized for search engines. This wasn’t a mobility test, for the same reason that sitting by the pool all day isn’t the same as swimming laps. A mobile network is a complex system of devices, technologies, protocols, and more, and as you can see in the diagram below, the SGi portion is simply the last part out to the Internet (or PDN in telco terms).

Mobility Testing Truth #2: Use Application Traffic Seen on a Mobile Network

Next, consider the “real world” part and how you use your phone each day. A mobility test must use application traffic that is actually seen on mobile networks, in the appropriate ratios, and behaving the way real traffic would. As a starting point, you can use a mobile traffic mix such as one of those researched by our own Chris Adams’ mobile traffic analysis, which contains a mix of encrypted and clear text web, mobile flash, streaming audio and video, and peer-to-peer file sharing.

Also, you can mirror the traffic mix published in the most recent Sandvine report, and be sure to include Facebook, Skype, and Netflix, which currently constitute between a third and a half of mobile network traffic during peak hours. Netflix and other streaming video would be particularly important to include, as documented in the recent Cisco report, which notes that not only does mobile video constitute a large percentage of mobile traffic now but it’s also projected to grow to 66% of mobile traffic by 2015. For a true mobile network simulation, you’d probably also note (as did the Cisco report) that roughly half of the traffic on mobile networks comes from laptops and netbooks, which will have different application traffic characteristics than smartphones do.

But in order to correctly model mobile network performance and understand the user experience, you must go beyond looking at the individual applications and make sure each mobile user is using a blend of these applications, along with the associated ancillary protocols (such as DNS to resolve the servers for my applications, email, and Twitter to distribute interesting stuff), because you must be able to model the interaction of the multiple applications in use.

Instead, while the report discussed the use of “real-world” applications, it was actually just simplistic HTTP grabbing five URLs with a few JPEG images [p.7]. It’s 2012, HTTP is just a transport – calling it an “application” is quite a stretch. Reading further, I see that the various applications in use weren’t even used together by any of the endpoints – 95% of them did HTTP only, 2% did SMTP, and 1% did DNS only [p.7]. That’s not the way the real world works, and evaluating web-request performance without considering the requisite DNS latency is faulty; it’s simply a limited best-case viewpoint (in direct contrast to Spirent’s stated objectives). The table in the Spirent blog post where DNS is detailed is slightly misleading; records don’t resolve to URLs, they resolve to IP addresses, and if you read the actual test results it’s clear that the DNS results weren’t used in the HTTP requests.

Real users want high performance. They don’t care if their web performance is slow because the firewall can’t process DNS or can’t process HTTP. They just care that it’s slow. And in this test, the two protocols were in no way correlated or interdependent, so bad firewall DNS processing wouldn’t be reflected in any measurement of HTTP user experience. And although Spirent stated in their blog post that they were trying to model real user experience, the omission of DNS in the actual user experience reflects either flawed test design or limited test-tool functionality. It’s not clear from the report what is the case.

We will assume that the statement in the Spirent blog that the latency variations they measured showed “only tens of nanoseconds of impacts on the Web browser page render time” is a typo, since the table on p.25 of the actual report shows variations from 10.2 ms to 121.0 ms – and the actual measurement in question was HTTP transaction latency, not web browser page render time.

Mobility Testing Truth #3: Firewall Testing Should Involve Security Attacks

And finally, we come to security, which is kind of important when we are talking about testing a firewall. I invite you to download the report and find a single security-related test. Were there any attacks detected or blocked? What about application-layer DDoS attacks or content inspection for all those emails and web pages? And it would seem obvious, but where was the use of mobile malware during a mobility security test? Security in this test was conspicuous only in its absence.

The Truth Hurts

At the end of the day, this test was neither mobile nor real-world nor about security. It demonstrated that eight Spirent appliances can produce 107Gb of simplistic traffic at no more than 97% CPU utilization and that a properly configured Crossbeam chassis running Check Point software can forward that amount of that type of traffic.

This post is not a condemnation of the quality of any of these products, nor their performance. Instead it is important for our industry not to look at this test and confuse it with an actual mobile security test. Real-world results, using an actual real-world test, would be very different. And relying on these types of tests, whether done by a vendor, a lab, or even yourself, creates a false sense of security that ultimately results in degraded performance and security.

That’s the truth – and sometimes the truth hurts.

---

Related Resources:

A Visitor's Guide to Telecom-Land

LTE Diaries: GTP Tunneling

Testing Mobile Network Infrastructure Easily and Economically at Massive Scale