Nov 09, 2010

Evaluating Virtualized Environments with Realistic Load

by Kristi Thiele

UPDATE: BreakingPoint's Application Load Evaluation Service is now available to help you you properly test cloud and virtualized infrastructures.

Part of my job as a sales engineer at BreakingPoint is to support evaluations of our product for potential customers. Recently one of these on-site evaluation adventures took an interesting twist. The client asked me to run a simulation on their network to measure the effects on end-user experience when more than one web application was consuming resources at the same time. Let’s look at what we found, since this is a challenge that many organizations face.

Step Back: VMware vSphere 4 Training

To set the scene, let me take you back to last spring, when I went through VMware vSphere 4 training. I recall the instructor at one point polling the class (17 students representing about 10 different companies) to find out who had already gotten their VMotion configurations finalized and was happy with them? Only one student raised his hand. The module in which we teamed up and did a lab that would let us witness the dynamic reallocation of resources across different servers was left for the very end.

Having taught technical classes, I always leave the most advanced and riskiest labs for the very end, since you might run out of time and never get to them. In this case we did have time for the final VMware lab. But afterwards it surprised me that I didn’t see a mob of IT folks rushing to try this sort of resource reallocation.

Impact of Load on VMware Environments

Back to my recent eval. The goal of the simulation was to occupy the CPU and other shared Apache web services on the server so that the functional test performed by Hewlett-Packard’s LoadRunner could stress a second web application. As I said, we wanted to see any effects on the end-user experience if another resident web application was consuming resources at the same time.

The BreakingPoint Storm CTM was set up to use our Client Simulation module with a rate of nearly 15,000 connections per second. It wasn’t until we had already started looking at the results that one of the client engineers spoke up and said, “Oh, by the way, these servers are VMs in a more complex shared setup.” I was excited, because -- remembering my VMware training class -- I really wanted to see how the impacts of the simulation that the BreakingPoint Storm CTM was running would be seen by the vCenter Server console. We had already been monitoring CPU utilization via Linux commands on each of the three servers that I was targeting, but now I wanted to see how that translated to a resource allocation from vCenter Server.

As soon as we logged into the vCenter management console, we noticed that there were entries in the event log indicating that reallocation of resources had been triggered, since the CPUs for each of the three VMs had immediately gone from green to red. This triggered the event to shift the VMs around according to the Distributed Resource Scheduling (DRS) policy for automation. As soon as the BreakingPoint Storm CTM simulation was over, the status went from red to green. Take a look (you can click through for a larger image):

For those who are not familiar with these VMware graphs, there are two scales on the one graph. The left scale shows CPU usage in MHz while the right scale shows CPU usage as a percentage of allocated reserved resources. There are four lines on the graph: the amber-colored one at the very top is the line for utilization as a percentage for the specific server -- it basically sits at 100% until the simulation stops. The red line also near the top shows the CPU utilization in Mhz for the specific VM -- it is basically just below 6000 MHz (6Ghz). The bluish and purple lines near the middle show CPU utilization for the cores from the host assigned to the VM. These two resources combined are what the VM is able to use from a resource-reservation perspective; if you added the blue and the purple, they would equal the red.

As you can see, the VM ran out of CPU resources, so it had to do something. In this particular case, there weren’t enough available resources in the cluster where the VM could be moved to that didn’t also have the same limitations. That’s good to know, because this scenario can now help the client organization ensure that they have allocated enough resources to the VMs or to the cluster at a higher level.

Not only did I think that this simulation was cool from an engineer’s perspective, but the folks I was working with also indicated that they didn’t have any other way to cause this type of result without resorting to some artificial, unrealistic test case, as I had done in my VMware training lab last spring. Working with us, they can now:

Burden this shared, virtualized web application infrastructure while running HP LoadRunner tests for functional performance.
Take the virtualized environment to its limits so that the network engineers can verify resource reservation and VMotion designs.
Run these scenarios over and over again in pre-production until they are satisfied that they have it right -- without risking production outages or affecting end users.

Of course I’m biased, but this eval brings to light just how powerful the BreakingPoint Storm CTM can be for providing real-world simulations in virtualized environments. If you want to give it a go, just register for a personalized demo.