Video: Cisco ASA Becomes the First Device to Face Live Validation with the BreakingPoint FireStorm CTM
by Mike MurphyNOTE: Want to validate your firewall? Check out our Rethink Firewall Testing Methodology for more information.
By Mike Murphy
Today at the 2011 RSA Conference, the newly announced BreakingPoint FireStorm CTM was used to demonstrate the ability of Cisco’s ASA-5585-X security appliance to handle 350,000 connections per second of blended application traffic, as you can see in this [updated]* screenshot from the BreakingPoint device:
Before I talk a little more about Cisco’s approach to this validation process, check out this short video that shows our validation of the ASA in progress. It was taken in the BreakingPoint booth at the end of a joint presentation with Cisco. (Apologies for the audio quality -- it was standing-room only in the booth and there was a lot of background noise.)
We often talk about how vendors lie with their data sheets, representing their products’ performance under idealized conditions instead of tough real-world conditions. Cisco deserves credit: they could have chosen to go that route, using the many tricks of the trade to pump up the ASA-5585-X’s numbers, but they didn’t. They put their device through a real beating at the hands of the BreakingPoint FireStorm CTM — one that reflects the heavy load and mix of application traffic that the ASA will have to handle in a production environment.
Common tactics for fudging a TCP connections number include:
- Reporting TCP connection open rate only (instead of open + L7 data + close)
- Not sending any L7 application data (open + close only)
- Piggybacking ACKs (This saves 2 packets per connection, since one ACK rides on the HTTP GET from the client and another ACK rides on the HTTP 200 OK from the server.)
- Disabling firewall features like content inspection and TCP normalizers
But Cisco did none of these. Furthermore, the validation at RSA was carried out using five protocols (FIXT, eBay, Oracle, Facebook, and NetBIOS) from BreakingPoint’s library of more than 150 application protocols. The traditional — unrealistic — approach is to use plain HTTP and pretend that that’s close enough to the real world, even though it clearly isn’t. You can see the specifics of the protocol traffic flows in this screenshot from the BreakingPoint FireStorm CTM:
As you can see from the ASA 5585-X data sheet, Cisco provides a throughput number of 35 Gbps for stateless UDP traffic, which is accurately labeled “Firewall Throughput (max).” But you also see that they publish a number for “Firewall Throughput (multi-protocol)” — a number based on more than just a stateless UDP-only or HTTP-only test. The multi-protocol number of 20 Gbps was arrived at by using a mixture of applications that a typical customer would see running on their network.
We’ve been working with equipment manufacturers’ marketing organizations for years, and find it refreshing that Cisco is choosing realism over best-case-scenario numbers for its ASA line, even though competitors will use their honesty against them. At the end of the day, though, Cisco knows it’s the right thing to do for their customers.
If you have questions or other follow-ups, please leave them in the comment field below.
* Update, February 18: The screenshot originally published with this post showed 350,043 attempted connections and 350,071 successful connections, which raised some questions in the comments and via e-mail. To minimize confusion, we substituted a different screenshot -- generated by the same process -- showing round numbers. The reason for the difference in the earlier version is that attempted and successful are measuring different things. Attempted means that our device sent a TCP SYN packet, while successful means we received the last ACK to close the connection. Some small variation is normal for these numbers throughout the process, depending on the current state of each emulated user.
By the way, you can check out additional statistics on our summary tab to confirm there aren’t any TCP connection or application failures. After the whole process is over, you can use this to confirm that all connections that were attempted were successful (which was the case here).
Related Content:
