Coming Full Circle
At my previous company the CEO at the time, John McHale, decided that we need to prove to the world something that we thought we already knew. That we had built the best Intrusion Prevention System (IPS) on the planet. Our mission was to have someone prove it. We didn't want to pay someone to prove our claims, we wanted a bake-off of every possible IPS vendor at the time and independently determine the winner. All parties should pay to be tested and the test should be the same for all vendors. That third party was NSS Labs.
NSS Labs had gained a reputation as a very strong test lab that published their full test methodology for the scrutiny of everyone. It was a published set of tests and you would be measured against these specific goals. At the time NSS had three levels, Tested (you showed up), Approved (works pretty good) and Gold (best product in the class). They rarely handed out Gold, and still rarely hand out of Gold. At the company we said Approved would be great and a huge achievement for our first time being tested. John said, nope he wanted Gold. We explained that Gold would be near impossible, we would have to get 100% on every test and that alone would take 6-9 months of development effort if we focused on nothing else. John made a gutsy call and the wisdom was - if we are Gold with NSS then we should never lose a deal because we are the best product. So we spent every night and weekend for many months recreating the NSS test lab and running tests over and over again. At the time NSS was in France and we visited Bob Walder (CTO of NSS Labs) many times to talk about the testing and make sure our results equaled his. In other words, our testing harness was correct.
We put in so many optimizations and changes in those months that it was amazing. This wasn't features; this was performance optimizations, UI changes and all the spit and polish that you never get around to do. These are the defects in your bug tracking systems that just keep getting moved to future because they aren't "important" enough. Well it turns out if you actually spent the time on those items, something amazing will happen.
We, TippingPoint, received the NSS Gold award. It was all worth it, the product was 10x more stable, faster and more accurate. Everything we did for NSS made the product better for everybody else and really made headway for sales. Not just as a marketing vehicle, but also as a superior product.
Last week we announced that NSS had switched to using
BreakingPoint as it's primary test product. NSS was one of the reasons we
started BreakingPoint. The lab setup was complicated, hard to use, results would
vary based on the stability of a number of the key test vendors. We would run
them 3 times in a row just to make sure the results were the same, often from a
reboot on the test gear. I wanted a device that could replicate all of NSS Labs'
tests and then some. In fact, I wanted a device that would allow NSS Labs to create
more realistic tests. Tests that were impossible in the past.
It's really neat that we have come full circle.
