Testing a Moving Target: Four Considerations for Ensuring the Performance and Security of Cloud Infrastructures

UPDATE: You may also be interested in our resiliency testing paper and our cloud computing testing section.

We took on the topic of cloud computing in my last post on testing in the cloud where we looked at the challenges vendors faced when conducting performance, security, and load testing for cloud-based environments. It’s no surprise that the difficulties scale right along with the environment. It became clear while talking with Julien Sobrier, QA engineer for Zscaler, a provider of multi-tenant SaaS security services. According to Sobrier “It is extremely difficult to replicate the behavior of a cloud in a lab: changing latency, packet loss, broken connections, with overlapping packets.” The list goes on and on.

The challenges of testing cloud-based environments go well beyond just the size and complexity of the environment.  The dynamic nature of cloud infrastructures means QA must effectively test for an ever-changing unknown: 

• Unlimited web services and applications
• Elastic demand
• Morphing usage patterns
• Dynamic resource allocation

…and again the list goes on.   The constantly evolving characteristics of this adaptive environment, and the users who access it, create an unlimited number of testing variables. It’s a bit like building a moving skyscraper on shifting sands. It can be done, you just need the right tools or a whole lot of time and money. 

When it comes to innovation, last generation performance, security and load testing products have lagged behind the hardware and software they were designed to test stifling the pace of delivering stable next-generation products and services. In my conversations with a number of cloud vendors, the same pattern appears to hold true.  Sobrier explains “Right now, we are using the same tools that appliance vendors are using:  Protos for fuzzing, regular HTTP performance tools (Autobench), etc., and custom tools to create a bigger variety of traffic.” 

In an attempt to emulate realistic conditions, cloud vendors like Zscaler and larger cloud vendors like Microsoft, Amazon and other must use legacy tools, some originally designed for traditional LAN-based environments, onto hundreds of servers to simulate load. The net result: an amalgamation of tools and workarounds that is costly, brittle and not ideally suited for the task at hand.

Testing Cloud Infrastructure: Four Important Factors to Consider
While the tools used to test cloud infrastructures are not unique, the scale of those issues is very unique because of the dynamic and shared characteristics of cloud infrastructures. In this real-time adaptive environment, four factors are paramount:  Elasticity, Scale, Realism and Security.

Elasticity
Renata Budko, VP of Products and Marketing of HyTrust, sums up the dynamic nature of the cloud: “Cloud infrastructure is very different from the traditional set-up where applications make exclusive use of the server resources. In the cloud, resources are pooled, access infrastructure is shared and resource allocation changes dynamically. The underlying infrastructure is flexible and changes often and every major revision of hardware or software can result in significant performance impact and, of course, needs to be tested. However, in the case of cloud, the process is so dynamic that discrete testing following major upgrades is simply not sufficient.  You need to understand how services behave when deployed together.”

Testing in such a dynamic environment must closely reflect the elastic nature of usage patterns:  conditions change frequently, demand is elastic, resources are shared, and more frequent releases require continuous testing.  There is little time for cumbersome test configuration and scripting. Extensive automation is a must-have to replicate a wide range of usage patterns. What’s more, dynamic resource allocation means applications cannot be tested in isolation from one another. In addition to high performance, highly scalable testing platform, vendors need more agile, automated, and easier to use testing products designed for a fast-paced, dynamic environment.

Realism
In today’s frenetic Web services/dynamic application/mashup world, it is impossible to emulate all of the different types of traffic that traverse the cloud, but vendors still need to emulate a broad mix of traffic. And, that means more realistic testing tools that support an ever-changing mix of applications, services, and incredibly high volume of sessions and high memory usage with sophisticated security attacks. Otherwise, you are left to run small tests with a limited mix of applications then extrapolating the results. Ultimately, you are making assumptions about how things might work with very few real data points. This is not sufficient to ensure the SLAs cloud vendors must deliver to business application users.

Gomez’ CTO Imad Mouline echoes the need for more realistic testing underscoring the need to create more realistic transactions with real-user monitoring and reporting. According to Mouline, “It is important to simulate load to the infrastructure that is coming in from different IP locations, different networks and from different places in the world.” 

Scalability
Mouline also talked a lot about the fact that we assume too much when we sign up for cloud computing services. One of the largest and most dangerous assumptions is stability in the face of peak demand. Prior to deployment vendors must offer assurances that services will perform reliably under a variety of load conditions.

Simulating that load is easier said than done, however. Again, Mr. Sorbier:  “Most tools I've worked with simulate one client and one server. We need to simulate thousands of clients and servers, with different IP addresses to be closer to reality.”  Imagine the number of servers and the millions in LoadRunner fees that it would take to run the tests needed to emulate the typical load these infrastructures see on a daily basis, much less under peak conditions. With the state of legacy testing tools, it would take a dedicated hydro-electric plant and a government bail-out. Many have suggested using the massive computational power of the cloud to simulate that load, but we have yet to see this live up to the performance needs of growing cloud vendors. 

Security
Possibly, the biggest challenge lies in the security arena, in part due to the historical practice of conducting security and performance testing in isolation. In traditional hardware/software testing scenarios, security and performance organizations are typically siloed. As security breaches become more frequent and the impacts more severe, and as network equipment vendors embed more and more security functionality into their core network products, this is changing. But change has not come fast enough for the cloud. In this more open and accessible environment, the stakes are higher.

Security attacks are not just dangerous; the protection against these strikes can have an immense effect on overall performance. Vendors must recognize the impact of security on application performance – specifically, web services—and test accordingly by emulating the real world where hackers are exploiting the cloud to spread viruses, malware, and attacking critical network infrastructure. 

In a recent presentation on Cloud Computing Security, Eva Chen, CEO of Trend Micro reported  “a new virus is created every 2 seconds”. Clearly, you have to have massive computing horsepower and a wide range of current security attacks to test for this in order to remain secure. That’s going to require a new type of testing product designed to evolve along with the security landscape. Ensuring effective protection for cloud networks will require constant vigilance to keep testing tools current. 

Advancing the State of Cloud Testing
There have been few advances in the last decade when it comes to cloud server and infrastructure testing. But, to live up to the vision of “truth in testing,” vendors need better options. New testing tools are emerging, but the unique obstacles presented by the dynamic, shared cloud infrastructure have set the bar almost impossibly high leading the vendors we spoke with to rely on home-grown in-house options. Clearly, these vendors are in need of new more scalable, flexible, realistic and cost-effective options. In the next post of our cloud testing series, I’ll look at how companies are trying to leverage cloud infrastructures for performance and load testing.