According to My Competitors I'm No Longer An Idiot

BreakingPoint is growing by leaps and bounds, as evidenced by our news release we put out a few weeks ago. We are really excited about our growth and that people are now demanding our approach to testing. Sales numbers aren’t the only way to measure growth; the reaction of your competitors is another way. This isn’t our first rodeo and one thing remains constant in business; you know you are growing faster than the competition when they stop calling your ideas dumb and start adopting them. I thought it would be fun to point out a few examples.

Layer 2-7 Testing

BreakingPoint was founded because of our frustration as equipment manufacturers with the testing industry. We knew the only way to test is by testing Layer 2-7 completely. Unfortunately at the time, and to this day, test vendors such as Ixia and Spirent have different solutions for Layer 2/3 and Layer 4-7 testing, meaning additional hardware, additional licenses and disparate reporting and of course Windows-only management software to run them.

At the latest round of trade shows, it was great to see our competitors (Ixia and Spirent) talking more and more about comprehensive Layer 2-7 testing. It means they are listening to the community now, and in the future hopefully provide comprehensive solutions. The first step is always to market it, and the second step is to build it. They just got to the first step but hopefully will advance to the second step. Granted the second step is a bit bigger than just updating a website.

Application Protocols

Each application protocol brings its own challenge to your network and the gear that runs your network. It comes down to a simple fact, if you aren't testing network equipment and servers with the same application protocols that they will be hit with in the real world, they will fail. Today we have more then 70 stateful applications for use in testing and we were the first to allow for testing with applications such as VMware VMotion, AIM, CIFS/SMB, BitTorrent and many others.

I beat the drum around application protocols as a customer of test equipment for 10 years and I guess I didn’t bang loud enough. Now, I see that Ixia supports 10 application protocols as of today, but please leave me a comment if you know of more. I know 10 doesn’t sound like a lot compared to 70, and it’s not. But it’s a huge increase from pure BitBlasting that they mostly focused on.

Security

Many of us who started BreakingPoint have backgrounds in security and security equipment. Not only have we discovered multiple security vulnerabilities, but also we have deep knowledge about how security affects performance. Any equipment can perform admirably with no security policies on and zero strikes hitting it, but show me performance when all the policies are on and the box is getting hammered. Much like our focus on application protocols, this is the reason we supply 4,000+ security strikes, 80+ evasion techniques, application layer fuzzing and complete Microsoft Tuesday coverage.

In the realm of security we are now seeing our competitors incorporate a dose of security into their solutions. An example is Ixia's partnership with Codenomicon (they were sharing a booth at RSA Conference), and Spirent’s ThreatEx product. It is good to see folks moving in this direction since it is so important. I would just stress to testers to make sure you test security and performance at the same time; otherwise you aren’t doing any testing. In the real world, security attacks happen at the same time as legitimate traffic, testing them separately provides almost no value.

Simplifying Licensing

In my past as an equipment manufacturer and testing tools customer, the complex pricing and service models presented by test vendors constantly frustrated me. I can remember too many times when I was sold test equipment and then had to buy additional licenses and modules to actually make it work properly…don’t even get me started on the multi-page proposals.

This not only stalls development, it leaves a bad taste in the mouths of engineers about testing. Our goal was to provide a complete service and support package for each user that gives them everything they need, from the latest firmware upgrades to all the application protocols and security strikes at a single price…eliminating the need for licenses. Our competitors’ current architectures make this technologically difficult, but their embracing of other elements tells me that perhaps the time is coming when we will see a more simplified pricing model throughout the industry.

Network Processors

Earlier this year we doubled the performance ability of our 3-slot BreakingPoint Elite chassis from 20 to 40 Gbps of Layer 4-7 app traffic through a firmware upgrade for all of our users. Our use of network processor technology allows us this flexibility and enormous benefit for users. The key, of course, is in how you program the NP and for nearly ten years we have been actively working at this skill.

This sentiment seems to not be shared by Spirent during a discussion in this LinkedIn message board (need to join the LinkedIn group), but Ixia announced at Toolapoolza they are coming out with a network processor powered blade: beta is May 11th and release is scheduled for June 11th. I am so excited about this; I’m a big fan of network processors no matter who uses them. I can’t wait to see what they get out of it. It was getting lonely being the only one able to do high performance application protocols (7.5 million concurrent connections at 20 Gbps per card as of release 1.3.1 and continues to increase).

Stalling

Our fellow test vendors have been convinced we are on the right path and are starting to follow along. Heck, from our website traffic analysis, folks from Ixia and Spirent are spending an average of 2.5 hours per day on our website. That would be fine, except they spend so much time downloading my picture I may file a stalker petition with the courts. Do they even have courts in Romania, home of the Ixia Development Center? ;)

The stage they are currently in is called the “Stalling stage”. Tell the customers you have the same thing and it’s cheaper and faster and available next quarter. And they say that every quarter – the truth is that annoys customers and causes more problems than it solves. It also sends more business my way when they fail to deliver.

Goliath vs. Daniel

My favorite thing about writing this post was that during proof reading it struck me how Spirent (LSE: SPT.L) has 1,500 employees and Ixia (NASDAQ: XXIA) has 850 employees. We are, of course, significantly smaller, but this allows us to develop faster, quicker and smarter then these guys, well at least according to them. It's not always the guy with the slingshot, sometimes all it takes is vision :)

Testing a Moving Target: Four Considerations for Ensuring the Performance and Security of Cloud Infrastructures

UPDATE: You may also be interested in our resiliency testing paper and our cloud computing testing section.

We took on the topic of cloud computing in my last post on testing in the cloud where we looked at the challenges vendors faced when conducting performance, security, and load testing for cloud-based environments. It’s no surprise that the difficulties scale right along with the environment. It became clear while talking with Julien Sobrier, QA engineer for Zscaler, a provider of multi-tenant SaaS security services. According to Sobrier “It is extremely difficult to replicate the behavior of a cloud in a lab: changing latency, packet loss, broken connections, with overlapping packets.” The list goes on and on.

The challenges of testing cloud-based environments go well beyond just the size and complexity of the environment.  The dynamic nature of cloud infrastructures means QA must effectively test for an ever-changing unknown: 

• Unlimited web services and applications
• Elastic demand
• Morphing usage patterns
• Dynamic resource allocation

…and again the list goes on.   The constantly evolving characteristics of this adaptive environment, and the users who access it, create an unlimited number of testing variables. It’s a bit like building a moving skyscraper on shifting sands. It can be done, you just need the right tools or a whole lot of time and money. 

When it comes to innovation, last generation performance, security and load testing products have lagged behind the hardware and software they were designed to test stifling the pace of delivering stable next-generation products and services. In my conversations with a number of cloud vendors, the same pattern appears to hold true.  Sobrier explains “Right now, we are using the same tools that appliance vendors are using:  Protos for fuzzing, regular HTTP performance tools (Autobench), etc., and custom tools to create a bigger variety of traffic.” 

In an attempt to emulate realistic conditions, cloud vendors like Zscaler and larger cloud vendors like Microsoft, Amazon and other must use legacy tools, some originally designed for traditional LAN-based environments, onto hundreds of servers to simulate load. The net result: an amalgamation of tools and workarounds that is costly, brittle and not ideally suited for the task at hand.

Testing Cloud Infrastructure: Four Important Factors to Consider
While the tools used to test cloud infrastructures are not unique, the scale of those issues is very unique because of the dynamic and shared characteristics of cloud infrastructures. In this real-time adaptive environment, four factors are paramount:  Elasticity, Scale, Realism and Security.

Elasticity
Renata Budko, VP of Products and Marketing of HyTrust, sums up the dynamic nature of the cloud: “Cloud infrastructure is very different from the traditional set-up where applications make exclusive use of the server resources. In the cloud, resources are pooled, access infrastructure is shared and resource allocation changes dynamically. The underlying infrastructure is flexible and changes often and every major revision of hardware or software can result in significant performance impact and, of course, needs to be tested. However, in the case of cloud, the process is so dynamic that discrete testing following major upgrades is simply not sufficient.  You need to understand how services behave when deployed together.”

Testing in such a dynamic environment must closely reflect the elastic nature of usage patterns:  conditions change frequently, demand is elastic, resources are shared, and more frequent releases require continuous testing.  There is little time for cumbersome test configuration and scripting. Extensive automation is a must-have to replicate a wide range of usage patterns. What’s more, dynamic resource allocation means applications cannot be tested in isolation from one another. In addition to high performance, highly scalable testing platform, vendors need more agile, automated, and easier to use testing products designed for a fast-paced, dynamic environment.

Realism
In today’s frenetic Web services/dynamic application/mashup world, it is impossible to emulate all of the different types of traffic that traverse the cloud, but vendors still need to emulate a broad mix of traffic. And, that means more realistic testing tools that support an ever-changing mix of applications, services, and incredibly high volume of sessions and high memory usage with sophisticated security attacks. Otherwise, you are left to run small tests with a limited mix of applications then extrapolating the results. Ultimately, you are making assumptions about how things might work with very few real data points. This is not sufficient to ensure the SLAs cloud vendors must deliver to business application users.

Gomez’ CTO Imad Mouline echoes the need for more realistic testing underscoring the need to create more realistic transactions with real-user monitoring and reporting. According to Mouline, “It is important to simulate load to the infrastructure that is coming in from different IP locations, different networks and from different places in the world.” 

Scalability
Mouline also talked a lot about the fact that we assume too much when we sign up for cloud computing services. One of the largest and most dangerous assumptions is stability in the face of peak demand. Prior to deployment vendors must offer assurances that services will perform reliably under a variety of load conditions.

Simulating that load is easier said than done, however. Again, Mr. Sorbier:  “Most tools I've worked with simulate one client and one server. We need to simulate thousands of clients and servers, with different IP addresses to be closer to reality.”  Imagine the number of servers and the millions in LoadRunner fees that it would take to run the tests needed to emulate the typical load these infrastructures see on a daily basis, much less under peak conditions. With the state of legacy testing tools, it would take a dedicated hydro-electric plant and a government bail-out. Many have suggested using the massive computational power of the cloud to simulate that load, but we have yet to see this live up to the performance needs of growing cloud vendors. 

Security
Possibly, the biggest challenge lies in the security arena, in part due to the historical practice of conducting security and performance testing in isolation. In traditional hardware/software testing scenarios, security and performance organizations are typically siloed. As security breaches become more frequent and the impacts more severe, and as network equipment vendors embed more and more security functionality into their core network products, this is changing. But change has not come fast enough for the cloud. In this more open and accessible environment, the stakes are higher.

Security attacks are not just dangerous; the protection against these strikes can have an immense effect on overall performance. Vendors must recognize the impact of security on application performance – specifically, web services—and test accordingly by emulating the real world where hackers are exploiting the cloud to spread viruses, malware, and attacking critical network infrastructure. 

In a recent presentation on Cloud Computing Security, Eva Chen, CEO of Trend Micro reported  “a new virus is created every 2 seconds”. Clearly, you have to have massive computing horsepower and a wide range of current security attacks to test for this in order to remain secure. That’s going to require a new type of testing product designed to evolve along with the security landscape. Ensuring effective protection for cloud networks will require constant vigilance to keep testing tools current. 

Advancing the State of Cloud Testing
There have been few advances in the last decade when it comes to cloud server and infrastructure testing. But, to live up to the vision of “truth in testing,” vendors need better options. New testing tools are emerging, but the unique obstacles presented by the dynamic, shared cloud infrastructure have set the bar almost impossibly high leading the vendors we spoke with to rely on home-grown in-house options. Clearly, these vendors are in need of new more scalable, flexible, realistic and cost-effective options. In the next post of our cloud testing series, I’ll look at how companies are trying to leverage cloud infrastructures for performance and load testing. 

BreakingPoint Live Presentation/Demonstration at RSA

Quick post for those of you at the RSA Conference today, BreakingPoint's CTO Dennis Cox will be doing a special demonstration with Juniper Networks in booth #1125 at 12pm PT and 12:30pm PT today (April 21st). You might remember the Juniper SRX 5800 test video we put up here a couple of months ago showing how the BreakingPoint Elite was used to test the SRX with stateful Layer 4-7 application traffic (109 Gbps specifically). Today Dennis and some folks at Juniper will be presenting how they performed that test and a live demo.

If you miss Dennis' presentation today be sure to stop by the booth during the show and our team will provide you with any information or demonstration that you might need.

We'll be live Tweeting the event as much as possible, so remember to follow us on Twitter.

Technology Growth is All About Proving Value Today And Tomorrow

Any number of surveys have been released this year taking a look at the possible spending patterns of companies when it comes to making investments in technology. Most recently Robert Half Technology released results from a survey of 1,400 CIOs throughout U.S. based companies with 100 or more employees. I originally saw Denise Dubie mention the survey on Twitter, leading to her article summarizing the survey. Many of the results were not too surprising, such as a bit more than a quarter of folks were looking to subscribe more to SaaS. I was struck however by the first sentence:

"Despite a challenging economy, seven out of 10 chief information officers (CIOs) interviewed recently said their companies will invest in information technology (IT) initiatives in the next 12 months."

The coverage centered around this stat, declaring that 70% of CIOs were looking to spend on IT in 2009. Seventy percent is certainly a very positive sign for any industry and folks should greet this with optimism, however I also asked myself what the other 30% were planning on doing in 2009. My guess was that these CIOs were saying "no" today in the face of tight budgets, but that most executives recognize that spending today will provide enormous benefits for the organization when economic conditions improve. Dave Willmer, executive director of Robert Half Technology, the conductors of the survey explained:

"We didn’t ask CIOs specifically why they don’t plan to spend on IT initiatives. Only a small percentage indicated “Other” areas (less than 2 percent), and the 30 percent of CIOs cited “No areas or no investments at this time.” However, we did ask IT executives in the same survey about their hiring plans and what factors are driving the expected decrease in staff in Q2. Forty percent attributed their response to reduced IT budgets and another 18 percent cited IT projects being put on hold, so this may shed some light on why some companies are not investing in IT initiatives at all."

Certainly a company that is downsizing staff will not answer yes to a survey questions like this one. Additionally, the budget halt or reduction at these companies may lead them to not be actively looking at implementing new IT initiatives. My guess however is that planning is still actively happening. But what impact will this type of viewpoint have for these companies when the economy turns? Again, Mr. Willmer:

"Given the challenging economy, it’s difficult to predict which companies will be successful and which will not in a year, or even several months from now. However, companies that are able to invest in critical IT projects, particularly those that will lead to increased security, cost savings and efficiencies, are at an advantage. Anytime you can help your business become more efficient or cost-effective, you stand to benefit."

Bingo! Even in the face of reduced or frozen budgets, it remains important to push forward IT initiatives, even if you can't spend the actual dime today. Even the executives saying "no" today are already planning for tomorrow. They are using resources to map out their plan, meet with vendors of the products they want, test products and ultimately be ready to move forward with purchase and implementation the minute budget gets approved. Today that CIO might be saying "no way" to planned initiatives, but it doesn't mean they aren't moving forward for tomorrow.

Since I was communicating with Mr. Willmer I thought I would ask what surprised him the most about the survey results, his response:

"I expected virtualization to be somewhat higher since many companies we work with have deployed virtualization. Among the CIOs we surveyed, however, a number may have already invested in virtualization and are likely investing more money in this area, particularly among the larger firms. We also see a lot of activity in Web 2.0, which had a lower response overall (17 percent), but it was several points higher for companies in the professional services industry (24 percent)."

Virtualization does seem to be an initiative that folks may have started last year, or prior, and therefore folks would not lump that into a new initiative, but it shows that money is spent in an area that adds simplicity, creates a competitive advantage and lowers costs over time. In fact, when you look at many of the initiatives folks listed they revolve around streamlining technology services, increasing simplicity and reducing costs.

The conclusion is perhaps not surprising. Companies are willing to spend if they see value for today AND tomorrow. The definition of value is different for companies of course, but they share a core desire to gain a competitive advantage, simplify current products and services, enhance innovation and reduce costs. Just last week BreakingPoint announced 285% growth from Q1 2008 to Q1 2009. Looking back at the news I was struck at the quote from our CEO Des Wilson in light of this topic of technology initiatives companies are planning in 2009:

“BreakingPoint is growing rapidly because we have and continue to listen to market needs and deliver the right product and pricing model to our customers. We continue to break new ground in the testing tools industry by driving the most realistic applications and services at extremely high performance levels using the smallest footprint. Organizations today must realize that it is never enough to simply reduce your pricing, customers want to know about how you and your solutions can better serve them and their business as the networking landscape changes around us.”

3 Reasons Application-Aware Network Equipment Fails to Perform in the Real World and How to Prevent It

You have probably been in the situation at one point where a product simply does not work when put into a real-world scenario. It might be as simple as that new hammock you put on your porch (long story) or high-performance network equipment. In both cases it benefits everyone to test the product under a real-world scenario prior to actual deployment, but as we've discussed on this blog numerous times, that simply is not happening in all cases.

Responsibly testing for real-world networks demands realism, transparency and simplicity from network equipment testing. When you are not testing in this manner you risk failures when the equipment is deployed in the real world. It is a topic we are certainly passionate about and if you are attending Cisco Toolapalooza you have the opportunity to attend the talk by Dennis Cox, our co-founder and CTO, "3 Reasons Application-Aware Network Equipment Fails to Perform in the Real World and How to Prevent It".

In his presentation Dennis gives practical information including common reasons for equipment failure and strategies for efficiently conducting meaningful Layer 2-7 testing. Having watched Dennis present many times, I know this will be an entertaining and educational talk.

Here is the poster that was created for the event, I thought this was the best way to share the details: