MILCOM Cyber Security Takeaways: Lessons from the Physical Battlefield

If the recent barrage of government security events is any indication, civilian and military personnel are energized about hardening cyber security. The largest event in recent months was MILCOM 2009, where I had three days to talk cyber simulation and cyber warfare with some of the best and brightest. The experience was both educational and a bit surreal, as BreakingPoint showcased resiliency testing products on an exhibit floor where networking products were intermingled closely (almost perilously, in my case) with military hardware like satellite receivers and armored vehicles.

These products once formed a strange combination, but now their mingling illustrates how the lines are blurring between the physical battleground and cyber warfare. Experts from academia and government have warned for years that new battles will be fought online, where traditionally weaker parties have a level playing field with the militaries of larger nations. Just this month the United Nations Telecommunications Agency chief warned that “the next world war could take place in cyberspace.”

Each day, thousands of individuals attack networks around the world for reasons ranging from personal amusement to organized cyber crime. As this graph taken from Infonetics Research data shows, cyber intruders and enemies of the state are becoming more sophisticated and aggressive in network attacks.

While government-operated networks are the targets of more than a million attacks each week, privately owned infrastructures are also increasingly vulnerable to attack. And more than data and communications are at risk. Here in the U.S., much of our critical infrastructure, such as energy, transportation, and financial networks, is private. A single attack taking out one of these networks could threaten the U.S. market and personal safety.

Both the public and private sectors are investing heavily in experimental and proven techniques for hardening the networks and data centers that form our critical information infrastructure. They are also scrambling to recruit and train security researchers in an effort to launch a proactive defense. This is particularly difficult in the world of information or digital warfare, where the terrain is complex and virtually invisible, conditions are ever changing, and attackers are widely distributed. To fight on this virtual battlefield, the U.S. is gearing up to hire some 4,000 specialists, and they are going to need actual hands-on experience to identify and block attacks that are ever morphing.

For the U.S. military, training was—and still is—one of the greatest challenges facing leaders in achieving the goal of Information Superiority, defined as “the capability to collect, process, analyze and disseminate information while denying an adversary’s ability to do the same.” It strikes me that there are a few lessons from the physical battlefront we can apply to the virtual world of cyber warfare, such as the use of simulation to prepare soldiers for battle.

Unlike training for the battlefield by using war games and battle simulations, our cyber warriors will conduct their missions online, and these cyber warriors must have Internet-scale cyber simulation capabilities to replicate commercial, military, and government network conditions. When it comes to cyber war, network mayhem is the enemy. Cyber warriors must know how to navigate the mayhem and spot the most deadly attacks in some of the most complex and confusing terrain imaginable—an ever-morphing online world filled with invisible enemies.

With an increasingly sophisticated army of guerrilla hackers operating in a highly distributed cyber battlefront, the ability to detect an attack and respond to prevent damage instantly is paramount. The enemy who strikes first can take out an entire network through coordinated distributed denial of service (DDoS) attacks. Before most products or experts even know anything is happening, the attack is over and critical systems are compromised.

Intelligence and the ability to spot and block attacks instantly are paramount to protect against devastating DDoS assaults. However, network-traffic-generation products or emulators cannot generate the evolving application and attack traffic or simulate the user load necessary to replicate realistic attacks and prepare our defenses to block them. In fact, from the reactions we received at MILCOM, it is evident that cyber simulation of the scale and sophistication created by BreakingPoint products has never before been possible. While that is exciting to hear, it is also frightening.

The U.S. government’s answer is to invest billions in building the U.S. Defense Advanced Research Projects Agency’s (DARPA) National Cyber Range (NCR). It’s a great idea and an ambitious undertaking, but it will take years to fully realize. Meanwhile, our military and intelligence communities do not have the tools they need now to properly prepare for the dangers of cyber warfare, and that is simply not acceptable when the technology exists in such a small form factor.

It is encouraging to speak with so many in the military who “get it” and are actively looking for solutions to train cyber soldiers to defend the network. With any hope, we can take yet another lesson learned from the physical battlefront and arm our soldiers with the tools they need to keep our infrastructure safe from growing cyber threats.

Emulate Chinese Peer-to-Peer Protocols - PPLive and QQLive to Test Network Equipment and Application Servers

This morning BreakingPoint announced that we’ve added realistic emulation of the peer-to-peer (P2P) streaming video network PPLive and online television platform QQLive to a growing library of 80+ application protocols. Now BreakingPoint customers can accurately simulate the world’s most used instant messaging, peer-to-peer and video streaming applications when testing network equipment and application servers. Below are more details from the news release:

• PPLive, the world's largest all-video network, is a peer-to-peer streaming video network created in Huazhong University of Science and Technology. PPLive streams live television and film programs on top of hundreds of millions of video clips, films and plays.
• QQLive is an interactive P2P distribution platform for streaming media developed by Tencent, China's largest and most used Internet service portal. QQLive has more than 100 TV channels and can be viewed in a variety of ways including through the Web and desktop programs.
• In October 2008, BreakingPoint introduced emulation of China's QQ IM (Instant Messaging) application, the most used IM application in the world. Today’s addition of QQLive and PPLive provide BreakingPoint users with the most complete and realistic testing of the world's most used peer-to-peer, instant messaging and video streaming applications.
• Overall BreakingPoint provides support for the world’s most popular IM, P2P and video streaming applications, including AIM®, QQIM, Windows Live Messenger, Yahoo!® Messenger, Encrypted BitTorrent™, eDonkey, Gnutella, PPLive and QQLive.

4 Product Bakeoff Pitfalls and How to Avoid Getting Played

Most IT buyers take product data sheet claims with a grain of salt. Capabilities are overblown and performance numbers are guesstimated using only Layer 2/3 or HTTP traffic. We all know this isn't real. So, we don't trick ourselves into believing that the numbers will hold true in a real network.

Beyond the performance numbers, there's also the question of features and functionality. When it comes to network and security equipment, one size does not fit all. Each reacts differently to traffic on your network and offers a unique set of features and performance levels. The top network equipment and server vendors are also combining multi-purpose capabilities to introduce all-in-one products in an effort to differentiate their offerings. Evaluating the best firewall, IPS, or server load balancer to meet your needs just became even more bewildering. How do make head-to-head comparisons with products that operate differently? How do you generate a realistic mix of traffic and still evaluate all of the products with the same mix?

A product bakeoff is the most important step in the hardware evaluation(PDF), negotiation, and planning process because it offers the best way to understand the real capabilities and performance of the devices you are considering for purchase. Yet lack of planning, bad bakeoff behavior, and the use of synthetic traffic often leave buyers with poor decision-making data and a false sense of security. As well all know, a bit of preparation on the front end is always required to get accurate results. Combine that with insider knowledge and you can ensure an accurate and deterministic network device evaluation. Here are 4 product bakeoff secrets every IT buyer should understand before embarking upon any head-to-head product comparison:

1. Bits will bite you. Bit blasting, the use of IMIX or even PCAPs will not give you an accurate view of how a product will perform under load and under attack--in other words--in your network. Mike Hamilton effectively characterizes the problem with IMIX in this post: "Frequently I am asked about UDP "packet blasting", IMIX, RFC 2544 and other testing procedures. My answer is always the same; these testing "methodologies" are not realistic. Testing a firewall with 100% stateless UDP traffic is pointless when the actual traffic it will see consists of a wide variety of applications over both UDP and TCP."

It is absolutely vital that you adequately categorize the makeup of the traffic on your network and simulate these conditions under load during your product bakeoff. As the chart below illustrates, you will likely see several magnitudes of difference in throughput between competing products when exposed to real network traffic.

2. You can have the best of both worlds: realistic and random. Once you've moved up the stack to realistic application layer traffic, you also need to ensure the traffic generated is somewhat random so that you can make direct comparisons between products and generate the same traffic in the same way to produce deterministic results. A pseudo-random mix of traffic will provide more accurate results, help you identify, resolve and validate fixes, and prevent vendors from gaming the system with clever programming tricks. This may sound mutually exclusive, but let me assure you it is not. The answer lies in the use of pseudo random number generator (PRNG).

While most testing products do not provide PRNG capabilities, BreakingPoint leverages PRNG to generate a pseudo-dynamic blend of traffic that is both real and repeatable. Once a seed value is set, BreakingPoint will create data by generating all data variants in the same way for each test executed. You can pick a number and generate the attacks or application traffic the same way every time to run repeated tests on different products or to repeat tests on the same product or network to determine the source or resolution to a problem.

3. Don't be left with a false sense of security. Another important reason to evaluate security equipment with random mix of traffic is because it is the only way to truly evaluate vulnerability vs. exploit variant detection capabilities. It's generally agreed that vulnerability filtering provides superior security coverage. Yet many products can’t perform the complex matching needed to deliver a vulnerability filter. They offer products that are programmed to identify exploits. The problem with this approach is that slight variants of the exploits these systems are programmed to identify can easily bypass the system. Vendors then release new filters and hackers develop new variants to bypass the same vendors again. The Conficker worm is a prime example of new variants bypassing security products every few months.

A dirty little secret of synthetic testing vendors is that their exploits are branded with trademarks or other recognizable content. Vendors can easily exploit this code by programming their products to recognize the code and trigger filters to easily pass product validation. While it may appear that these products are working as promised, this is no indication that the equipment is capable of recognizing and filtering real security attacks in a production network. PRNG eliminates the possibility that devices under test can be programmed to recognize and react to codes embedded in test traffic.

4. Stick to a proven test methodology. One of the challenges for buyers who are structuring their product bakeoff, is the lack of good solid test methodologies for thoroughly validating equipment. They use RFC's that are outdated, specifications become obsolete quickly, and most of the methodologies you’ll find online are many years old. Since they were published, network equipment has evolved, so you need a current product bakeoff test methodology designed specifically for the devices you are evaluating. Here are a few the BreakingPoint Labs team has published: Firewall Testing methodology, Server Load Balancer Testing methodology, IPS Testing methodology and other test methodologies. However, more are needed.

Tip of the Product Bakeoff Iceberg?
These are probably just the start so I’d like to hear your product bakeoff advice and your methodology wish list. By sharing tips, collaborating on test methodologies, and exposing secrets, we can all get the most out of product bakeoffs.

Test the Cloud Before Living In It to Avoid Tech Torture

We don’t often include CNN.com in our blogroll here on the BreakingPoint Labs blog, but this week the news site is offering up an intriguing video blog of Topher Kohan’s experience of living (working) in the cloud.  Topher has set a challenge for himself to conduct his daily tasks using only cloud-based apps while filming a diary of his experiences. According to Topher:

“I will try to do 100 percent of my job online and not use any desktop applications. So when I check my e-mail, schedule a meeting or write a document, I will do all of this online and then save it in online storage. If it works, I will be able to sit at any computer in the world, and as long as it has Internet access and a browser, I can log in to some Web sites and work.”

Cloud cynics will be disappointed to learn that the videos are not exactly living up to his headline: “Tech Torture with Topher". Wish I could say the same for my own cloud application performance experiences. But things are just starting to heat up as already Topher has run into issues with Twitter. What? The Twitter Fail Whale already? To be fair, an outage or capacity overload was not really the nature of Topher’s chief complaint. It was the lack of functionality for dealing with the overwhelming volume of tweets from the 400 plus Twitterers he follows.

Tune in to watch Topher as he moves on to more complex applications like Google Docs. Heads up Google, Amazon, Twitter and other cloud vendors – this is no time for a major outage or performance issue. Hope you’ve done your due diligence in the area of cloud testing. There’s a large audience of cloud enthusiasts as well as skeptics watching (and commenting) to see Topher be tortured by 12 hour application outages prior to a story deadline, performance issues that would try the patience of Mother Teresa, or perhaps a major security breach involving the loss of Topher’s personal credit card information. Alas, I find myself a bit too excited by the prospects.  Don’t let me down guys.

Update: Get instant access to BreakingPoint's step-by-step Server Load Balancer Testing Methodology.

Testing a Moving Target: Four Considerations for Ensuring the Performance and Security of Cloud Infrastructures

UPDATE: You may also be interested in our resiliency testing paper and our cloud computing testing section.

We took on the topic of cloud computing in my last post on testing in the cloud where we looked at the challenges vendors faced when conducting performance, security, and load testing for cloud-based environments. It’s no surprise that the difficulties scale right along with the environment. It became clear while talking with Julien Sobrier, QA engineer for Zscaler, a provider of multi-tenant SaaS security services. According to Sobrier “It is extremely difficult to replicate the behavior of a cloud in a lab: changing latency, packet loss, broken connections, with overlapping packets.” The list goes on and on.

The challenges of testing cloud-based environments go well beyond just the size and complexity of the environment.  The dynamic nature of cloud infrastructures means QA must effectively test for an ever-changing unknown: 

• Unlimited web services and applications
• Elastic demand
• Morphing usage patterns
• Dynamic resource allocation

…and again the list goes on.   The constantly evolving characteristics of this adaptive environment, and the users who access it, create an unlimited number of testing variables. It’s a bit like building a moving skyscraper on shifting sands. It can be done, you just need the right tools or a whole lot of time and money. 

When it comes to innovation, last generation performance, security and load testing products have lagged behind the hardware and software they were designed to test stifling the pace of delivering stable next-generation products and services. In my conversations with a number of cloud vendors, the same pattern appears to hold true.  Sobrier explains “Right now, we are using the same tools that appliance vendors are using:  Protos for fuzzing, regular HTTP performance tools (Autobench), etc., and custom tools to create a bigger variety of traffic.” 

In an attempt to emulate realistic conditions, cloud vendors like Zscaler and larger cloud vendors like Microsoft, Amazon and other must use legacy tools, some originally designed for traditional LAN-based environments, onto hundreds of servers to simulate load. The net result: an amalgamation of tools and workarounds that is costly, brittle and not ideally suited for the task at hand.

Testing Cloud Infrastructure: Four Important Factors to Consider
While the tools used to test cloud infrastructures are not unique, the scale of those issues is very unique because of the dynamic and shared characteristics of cloud infrastructures. In this real-time adaptive environment, four factors are paramount:  Elasticity, Scale, Realism and Security.

Elasticity
Renata Budko, VP of Products and Marketing of HyTrust, sums up the dynamic nature of the cloud: “Cloud infrastructure is very different from the traditional set-up where applications make exclusive use of the server resources. In the cloud, resources are pooled, access infrastructure is shared and resource allocation changes dynamically. The underlying infrastructure is flexible and changes often and every major revision of hardware or software can result in significant performance impact and, of course, needs to be tested. However, in the case of cloud, the process is so dynamic that discrete testing following major upgrades is simply not sufficient.  You need to understand how services behave when deployed together.”

Testing in such a dynamic environment must closely reflect the elastic nature of usage patterns:  conditions change frequently, demand is elastic, resources are shared, and more frequent releases require continuous testing.  There is little time for cumbersome test configuration and scripting. Extensive automation is a must-have to replicate a wide range of usage patterns. What’s more, dynamic resource allocation means applications cannot be tested in isolation from one another. In addition to high performance, highly scalable testing platform, vendors need more agile, automated, and easier to use testing products designed for a fast-paced, dynamic environment.

Realism
In today’s frenetic Web services/dynamic application/mashup world, it is impossible to emulate all of the different types of traffic that traverse the cloud, but vendors still need to emulate a broad mix of traffic. And, that means more realistic testing tools that support an ever-changing mix of applications, services, and incredibly high volume of sessions and high memory usage with sophisticated security attacks. Otherwise, you are left to run small tests with a limited mix of applications then extrapolating the results. Ultimately, you are making assumptions about how things might work with very few real data points. This is not sufficient to ensure the SLAs cloud vendors must deliver to business application users.

Gomez’ CTO Imad Mouline echoes the need for more realistic testing underscoring the need to create more realistic transactions with real-user monitoring and reporting. According to Mouline, “It is important to simulate load to the infrastructure that is coming in from different IP locations, different networks and from different places in the world.” 

Scalability
Mouline also talked a lot about the fact that we assume too much when we sign up for cloud computing services. One of the largest and most dangerous assumptions is stability in the face of peak demand. Prior to deployment vendors must offer assurances that services will perform reliably under a variety of load conditions.

Simulating that load is easier said than done, however. Again, Mr. Sorbier:  “Most tools I've worked with simulate one client and one server. We need to simulate thousands of clients and servers, with different IP addresses to be closer to reality.”  Imagine the number of servers and the millions in LoadRunner fees that it would take to run the tests needed to emulate the typical load these infrastructures see on a daily basis, much less under peak conditions. With the state of legacy testing tools, it would take a dedicated hydro-electric plant and a government bail-out. Many have suggested using the massive computational power of the cloud to simulate that load, but we have yet to see this live up to the performance needs of growing cloud vendors. 

Security
Possibly, the biggest challenge lies in the security arena, in part due to the historical practice of conducting security and performance testing in isolation. In traditional hardware/software testing scenarios, security and performance organizations are typically siloed. As security breaches become more frequent and the impacts more severe, and as network equipment vendors embed more and more security functionality into their core network products, this is changing. But change has not come fast enough for the cloud. In this more open and accessible environment, the stakes are higher.

Security attacks are not just dangerous; the protection against these strikes can have an immense effect on overall performance. Vendors must recognize the impact of security on application performance – specifically, web services—and test accordingly by emulating the real world where hackers are exploiting the cloud to spread viruses, malware, and attacking critical network infrastructure. 

In a recent presentation on Cloud Computing Security, Eva Chen, CEO of Trend Micro reported  “a new virus is created every 2 seconds”. Clearly, you have to have massive computing horsepower and a wide range of current security attacks to test for this in order to remain secure. That’s going to require a new type of testing product designed to evolve along with the security landscape. Ensuring effective protection for cloud networks will require constant vigilance to keep testing tools current. 

Advancing the State of Cloud Testing
There have been few advances in the last decade when it comes to cloud server and infrastructure testing. But, to live up to the vision of “truth in testing,” vendors need better options. New testing tools are emerging, but the unique obstacles presented by the dynamic, shared cloud infrastructure have set the bar almost impossibly high leading the vendors we spoke with to rely on home-grown in-house options. Clearly, these vendors are in need of new more scalable, flexible, realistic and cost-effective options. In the next post of our cloud testing series, I’ll look at how companies are trying to leverage cloud infrastructures for performance and load testing.