Time for Realism, Simplicity and Truth in Testing

UPDATED: Went back to the 16 Spirent Avalanche number after consulting with some folks, thanks!

Interesting to read NetworkWorld's review of the Juniper SRX 5800, particularly after we just had the opportunity to test the SRX during BreakingPoint Day at Juniper. It was entertaining to watch Twitter and read the thoughts of a variety of folks reading the review and recognizing that from start to finish, the test missed the boat on realistic testing and was shocking in its complexity and immensity due to the required Spirent test gear. I even wrote about my thoughts on Twitter while reading the review:

A few highlights from my point of view:

1) It took 16 Spirent Avalanches, 1 Spirent TestCenter and 1 Spirent ThreatEx to run a test of the Juniper SRX. That is crazy, particularly when you realize one BreakingPoint Elite generates 40 Gbps of Layer 4-7 traffic and 80 Gbps of Layer 2-3 traffic. Imagine the cost and time associated with needing that much test equipment.

2) On top of all that equipment, there seems to be zero use of stateful traffic to test the Juniper SRX. Testing without a realistic blend of traffic (beyond simply HTTP), Layer 4-7 stateful traffic, is unacceptable in today's era of content-aware high-performance network equipment.

3) To continue hammering this point home...using "13 UDP attacks" to test IPS functionality at load? As HD said on Twitter, "Static testing is pretty worthless".

4) Dennis reminded us all that relying on IMIX is a dangerous game, which was referred to in the test through the use of data from CAIDA. Again we must all make a more concerted effort towards realistic testing.

Having just witnessed our test of Juniper SRX it struck me how using a fourth of the test equipment (four BreakingPoint Elite) used by NetworkWorld, we tested the SRX with stateful traffic, live security strikes and even more throughput and bandwidth. This latest test should be a call to our industry to come together and demand more realism, transparency and simplicity from network equipment testing. We all recognize what it takes to perform realistic testing, but all too often we let test vendors, equipment vendors, test labs and industry publications get away with testing insufficiently. Now is the time to demand more truth in testing.

Posted by Kyle Flaherty

Printer Friendly Create PDF

0 comments
Tags: