- 1.866.352.6691 | INTL 1.512.821.6000
Will Cloud Computing Be the Tipping Point for Change in the Test Industry?
The cloud computing space is heating up once again with IBM, Sun, Cisco and others introducing the Open Cloud Manifesto this week. The document was designed, in the words of the group’s founders, to start a conversation around standards and help clients ask the right questions about cloud interoperability.
It strikes me that Cloud Computing is doomed to fail unless someone starts asking questions about performance, security, and load testing for cloud-based environments. How are the major cloud players actually testing their networks and applications within the cloud for availability, performance and security? Or are they even testing at all with the cloud in mind?
Most of us have experienced at least one negative experience with cloud apps such as Google Docs, Amazon’s S3, Microsoft Azure or another service. We’re not talking sluggish performance. We are seeing outages of 10 to 20 hours or more. When it comes to business applications, that’s not considered acceptable down time.
Clearly, better performance monitoring and testing is in order to avoid what Greg Ferro, of EtherealMind.com, calls the “Vista effect,” where constant coverage of technology failures makes the acceptance of the technology nearly impossible.
No one disputes that more comprehensive and realistic testing of cloud networks is critical to mainstream adoption of cloud computing. And, while availability is obviously the most critical factor, performance and security are also weighing heavy on the minds of IT organizations that must deliver to SLAs whether they manage the environment or not. What’s more, cloud networks make enticing targets for hackers as one breach exposes the content of many users from many different companies.
Cloud Testing Has Its Challenges
The real issues behind the lack of testing and subsequent performance problems are the cost and complexity of testing on this scale using legacy tools. Testing cloud applications and networks demands a wide mix of application traffic, current security coverage, and incredibly high-performance and throughput. Essentially, you need to create a realistic testing environment with an ever-changing mix of applications and increasingly sophisticated security attacks. That makes delivering high performance a moving target.
That’s part of the reason performance and load testing on an enterprise or Internet scale has always been so painful, inefficient and costly. Now, push this to the cloud and you can begin to see a hint of the challenges ahead. Consider the added dynamic of routine maintenance, patches and upgrades and the need to factor in the time to test how the patches and upgrades will impact performance.
Limited Options for Performance and Load Testing in the Cloud
Historically, vendors have been left with few options, most of which would erode the very value propositions that make cloud computing so attractive—cost savings and agility. You could purchase or rent hundreds of servers to emulate users and application traffic, buy a whole bunch of testing software, purchase a slew of applications, license security attacks, and tie it together with a bunch of custom scripting. Or you could contract with a testing service provider and wait for a test window. Unfortunately, it’s not just a one and done proposition. Networks and applications change.
No one is arguing that testing in a cloud-based environment isn’t complex. It’s about as challenging as it gets. I’m just saying that performance, security and load testing has been far too difficult and expensive for far too long. And, the time for change has been dramatically accelerated with the onset of cloud computing.
Traditional testing tools were simply not designed for this dynamic, complex and high performance computing environments, and haven’t evolved to keep pace. Even a heavy hitter like HP LoadRunner is considered past its prime for today’s market needs. Without radical change in the performance and realism of today’s testing products, what we continue to refer to "truth in testing", cloud vendors will continue to be stuck between a rock and a hard place leading to the ultimate FAIL of cloud computing. I mean, if Google can’t do it with their immense resources, can it be done?
In my next post on cloud testing I’ll take a look at the requirements for testing cloud networks and ways to introduce realistic high performance conditions without significant downtime or huge investments in legacy testing tools and server farms.
Download the BreakingPoint Server Load Resiliency Methodology, a step-by-step guide to measuring the performance, security and stability of your cloud infrastructure.
Comments
- Etherealmind
-
Testing is the elephant in the room
I think you have rightly picked on a core problem with almost every network. Only certain networks and applications are ever tested properly. Time and budgets are not allocated for enough testing unless its' identified early.
If the testing is not done, then its only an outage that finds the underlying flaw. Enough outages and you are losing customers....and it doesn't take long before that becomes like Microsoft's Vista. Once you have a bad reputation it never goes away - look at 3Com, they shafted us in 2001 and no one has ever forgotten.
The elephant in the room is that testing is not prioritised, and then there isn't enough skills to do it right when you do get the chance.
One day, one day.
Thanks for the link.
Etherealmind
- April 1, 2009, 10:06 AM
- Andre Gironda
-
IaaS and Paas
Quality/security testing has always made use of clouds. Virtual resources, distributing computing, grid technology, and even server virtualization have been in use by quality/security testers as soon as the tech gets on the shelf since day-one.
The difference today is how easily accessible, portable, and agile cloud compute/storage resources have become. Cloning/templating, live VM/storage migrations, and self-service portals such as Skytap have enhanced all faces of IT and dev.
Personally, I like what Elfriede Dustin had to say about R&D&T at the GTAC conference. "Test" is the new "development".
- April 1, 2009, 10:26 AM
- Pam O'Neal
-
R&D&T - Cloud Testing
Andre - Agree to a point, but we haven't yet seen these testing tools/services scale efficiently to the load and performance levels that it will take to simulate realistic conditions, not to mention real applications. On your final point, I am intrigued. Do you have any other info about Dustin's R&D&T prezo? Would like to explore this further.
- April 1, 2009, 12:52 PM
- Andre Gironda
-
Re: R&D&T - Cloud Testing
- April 1, 2009, 2:23 PM
- Mark Tomlinson
-
HP LoadRunner Not Past it's Prime
Pam,
I must refute the claim or rumor that HP LoadRunner is past its prime, although I hear that a lot. The majority of newcomers to automated load testing only find fault with the pricing (which I admit is higher than others, but matches the ROI value) and not the technology.
In fact, I would go so far as to share a story about a completely naive and misinformed engineer who was told by another developer that "LoadRunner cannot handle any dynamic data in the script." Of course, that was completely untrue - LoadRunner has supported dynamic data parameterization and correlation for nearly 15 years. In fact, it might be an honest claim that the LoadRunner guys invented the concept way back when. This same tester had no training, certification or had even read the tutorial for LoadRunner (which is included for free with the product). That's just plain lazy engineering.
Also, what I see out there with our customers (and even a few Industry Analysts) is a complete lack of understanding of the LoadRunner product and how it has evolved, changed, grown and expanded since 1991. It's almost like going to a family reunion and all your old relatives treat you like you're still 14 years old...even though you are 38.
HP LoadRunner: this is not your father's automated performance testing tool.
If you want to learn more, check out our blog: http://www.communities.hp.com/online/blogs/loadrunner/default.aspx
Thanks!
-mt
Disclosed: I am the HP LoadRunner Product Manager
- December 2, 2009, 8:11 AM
- Ray Tomany
-
When I hear 'LoadRunner is past its prime' it tends to be from a frustrated, anxious, or newbie competitor. I'd be keen to understand where you may have had problems with it as I work on the coalface daily in HP presales doing LoadRunner proof of concepts, and would like to know where others have had issues, as I haven't. If there are any genuine issues then it would be good to hear about them, so that they can be investigated, and put to bed. HP is listening.
- May 13, 2010, 6:14 PM
- Pam O'Neal
-
Hi Ray, thanks for reading about BreakingPoint and your question, I'd love to let you know what we and our customers are coming across. We hear a lot that LoadRunner maxes out at 50,000 users? Then you have to roll in the multi-million dollar server farm in an attempt to push data centers to capacity. We've seen it at dozens of your customers who are looking for a smarter, more cost-effective load testing solution that also factors in the critical aspects of security. I understand LoadRunner does a fine job of testing for functionality on small-scale projects. However, scaling to meet the demands of large cloud data centers, really most large data centers, is an entirely different matter. In those scenarios they are asking for high performance along with hardware/software that is flexible and scales along with changes in applications and security attacks. Many LoadRunner customers do not realize that this capability is possible or affordable, especially on an ongoing basis, resulting in companies who have no idea what their data center capacity or resiliency might be. This frustration was felt quite strongly by our founders and was one of the reasons BreakingPoint was created. And, fortunately for us, continues to drive our innovation and tremendous growth.
- May 14, 2010, 4:45 PM
- Cathy
-
Hi, Even though it's quite an old post, I completely agree with the comment made in the post about HP Loadrunner. Maybe even HP realized that, that's why it's also moving into cloud testng environment with it's new products.
- May 27, 2010, 3:42 AM
- Pam O'Neal
-
Hello Cathy. Thanks for joining the discussion. Not sure about their motivation for moving toward cloud testing and am very curious about the success that similar organizations are having on that front. However, using the cloud to test the cloud doesn't necessarily address the scalability or security concerns I raised. I'll take a look at this in a future blog post.
- May 27, 2010, 4:19 PM
- Cathy
-
Hi Pam, I'll surely look forward for your post on these issues. Regards, Cathy
- May 28, 2010, 7:29 AM
- Alex
-
I think the key aspect to consider that tools like Loadrunner enable end to end performance engineering helping identify pitfall, scalability and performance issue for any given application. Whereas utilization of tools like BreakingPoint are beneficial to determine capacity planning of a specific farm or to generate additional traffic (not representative of end user behavior though for any given application) in conjuction with end to end performance engineering conducted utilizing Loadrunner tool.
- September 21, 2010, 11:11 PM
Google+
Tags: Cloud Testing //