Talking Performance Testing with Steve Mitchell of F5 Networks

The life of a tester has changed dramatically over the past few years due to enhancements in content-aware network equipment, deep-packet inspection (DPI), more and more applications across the network and more dangerous security threats. Over the past few years testing professionals have come across a variety of new tools, including those from BreakingPoint, and now there seems to be a renewed focus and excitement around testing network equipment and servers.

Steve Mitchell, a performance tester at F5 Networks, a leader in Application Delivery Networking (ADN), focused on ensuring the secure, reliable, and fast delivery of applications. Steve is also the person behind PerfTesting.org has been working in the computing industry for the last 15 years and has always been driven by being able to measure and improve the performance of all of the products he has worked on; passion is a necessary trait for a successful tester.

Recently I spoke with Steve to discuss this evolution for performance testers and what it means for us all.

What is it about performance testing that attracts people to the role?

I have worked in the computing industry for the last 15 years, and in all of my experience, I have always been driven by being able to measure and improve performance of all of the products I've worked on. Not until F5 Networks did I make the transition to a full-time performance position about 5 years ago in our product development group. This was necessitated by the types of products that F5 delivers, and the need to have a dedicated group of people focused on industry-standardized and highly repeatable performance testing. I originally started in the group as a manager with one employee, and did a lot of testing myself. Now we're far larger, yet I still stay in touch with all of the performance solutions in the market.

This profession continues to drive me because of the ever changing atmosphere - I always look for opportunities in my career that are constantly interesting and changing, and performance testing and development definitely fit that. There's never a dull time with the speeds at which processors and technology has been leapfrogging - always a new chipset or idea that a developer has to make things faster, better, and more stable.

I think that whole mentality is why so many people, and good people at that, are attracted to performance testing in general. It's similar to tuning a car just to get 1 more horsepower - the physical and tangible pieces that you can touch and influence to eek out a bit more performance than your competitor to stay ahead in the race. I think also the types of challenges in increasing performance, or the development and test challenges in figuring out performance problems is also highly attractive to many.

Testers constantly use methodologies, but do you have a personal methodology or "core values" for how you test equipment? Your own personal rules for testing?

Yes, the most important thing being a scientific, step-by-step process to capture as much as possible. I'm not plodding or stuck in a rut -just fanatic about designing automation and scripts to capture as much as possible, and make things as absolutely repeatable as possible. I also have always had a knack for troubleshooting and finding problems, which lends itself well to investigating performance issues - thinking outside the normal box of issues that contribute to problems.

When you look back at the beginning of your career and today what are the major differences in what you do day to day? What are the major differences in how you get your job done?

The biggest difference is the maturity of the tools and automation available. Originally when I started working with performance equipment, it was all home grown - no real vendors to speak of. You were on your own for everything, which usually meant you built just enough to achieve what you were trying to test, and that was pretty minimal! It's progressed through horribly gross and unstable software from just about every vendor, with no thought to automation and features other than the basic requirements, to where we are today - lots of vendors to choose from, lots of options for features and functionality, and growing automation and control architectures. It's definitely come a long way in a pretty short amount of time compared to many other markets.

Today's economic environment obviously posses challenges for everyone, including performance and security testers in R&D/QA labs. Are their ways for perf testers to reduce testing costs without cutting corners?

I think there are a couple of ways that costs can be reduced without cutting corners. The first and most important is deeper automation - not just of tests, but of equipment and labs, and of reporting and delivery of results to not only testers, but management and otherwise. There are a lot of companies that have pseudo-solutions here, but nothing that really fits the bill for most of us. However, the good news is that most of this can be done with open source tools and standard operating system tools. We've done some of this at F5, and have seen a huge amount of work removed from individual testers, which means they can spend more time testing, which means products get better coverage, and faster to market.

The other thing that I think could help would be to adapt the automation above to utilize the very expensive performance test equipment more effectively, thereby getting more overall use out of it. Examples would include splitting up chassis into smaller parts so that more groups of tests can be run, dicing up ports in different amounts to optimize tests better so that there is less time per test.

All of these ideas have to do with optimizing the amount of time things are taking through automation or allocation.

Automation was an important topic on your forum site and now your blog. How has automation changed in performance testing over the years and where would you like to see it go in the near future?

As I alluded to above, automation was non-existent in the early years. It's come a long way, but there's still a lot left to do. Nowadays most vendors offer some sort of scripting language or interface to their products, typically via TCL, that allows you to do just about everything that their normal product GUIs can do. Some even allow you to do things more intelligently via these interfaces so you aren't taking as much time to configure things if you're only varying one parameter - this is extremely important in order to get the most out of your equipment.

Where does it need to go now? An open and transparent standard for all of the different pieces is key to the next step in the evolution of performance test automation. There are several efforts out there, one of which I'm a key contributor (TesLA), but the bottom line is that there needs to be a standard for all of the devices in the sandbox of performance testing to play with. The lab automation solution needs to be able to configure a switch, L1 patch panel, and a device under test, and then pass the relevant information on to the traffic generator, fuzzer, and other pieces in the test. Once a test is running, everyone involved needs to output common statistics and information in a centralized way so that the reporting aspect, which is always different for everyone out there, can be optimized.

I would predict that in the next 2 years this area will become more important to all vendors and consumers in the performance test industry than any new feature, functionality, or faster card or appliance. Being able to utilize all of these pieces as effectively as possible is not only financially responsible given the economy, but a requirement for many, if not all, organizations that have complex applications and solutions to test.

Posted by Kyle Flaherty

Printer Friendly Create PDF

0 comments
Tags: DPI Testing // Server Load Testing //