Five for Friday: Matt Sarrel & Network Equipment Testing on Wall Street

This week for "Five for Friday" I talked with Matt Sarrel, the Founder and Executive Director of Sarrel Group. Sarrel Group is a New York City based product testing, editorial services and consulting firm. Matt and his staff have worked with an exclusive clientele of small hedge fund and financial management companies for more than 10 years building networks, implementing algorithm based trading systems, and building sophisticated data analysis clusters. As we discussed a month ago Matt's group is using BreakingPoint as part of their testing environment. It seemed to make sense, with all the economic news, that we start off tapping Matt's brain about testing on Wall Street. Enjoy!

1) How sophisticated is network equipment testing for firms on Wall Street?

I can’t speak for the larger firms because most of the work I’ve done on Wall Street has been with smaller hedge funds. I’m sure the larger firms must test network equipment before implementing it, but to tell you the truth, smaller firms take the network for granted. They are very focused on their software and the algorithms that they run and will test those almost infinitely. It surprises me that these firms assume that the network functions properly (the same way most people assume that the there will be a dial tone when they pick up the phone) because minimizing latency is crucial for many of these projects. In my opinion, network equipment and network links should be tested more often. I’ve seen companies spend hundreds of thousands of dollars optimizing code and then turn around and run it over a tediously slow WAN link. Taking the network for granted increases the number of unknowns (not a good thing in finance) and puts many projects in potential jeopardy.

2) How will the latest economic alter testing for these types of companies?

There are two ways to look at this. The first is that companies will simply cut their test budgets. The trouble is that most business people don’t understand that systems should be tested before implementation. That’s just some issue that techies need to deal with and times are tough so let’s cut the budget. What those people don’t realize is that the cost of not testing far exceeds the money saved by not testing. Would you want to invest in a fund that is going to use your money to test their system when it goes live? I didn’t think so.

On the other hand, and this is what I hope happens, is that Wall Street realizes that they no longer have the luxury of testing on a live system. In these troubled times, controlling and limiting risk is essential – both in your systems and in your investments. It would make more sense to me for companies to increase the amount of testing in order to limit the risk of systems failing when they go live. There is enough uncertainty in the market; you don’t need that uncertainty to extend into your systems. Spending the money to test and verify that systems work as needed will reduce risk and allow companies on Wall Street to focus on their financial management, investments, trades, and algorithms rather than being sidetracked by something as simple as an overloaded network.

3) No matter the organization, when you are running tests on devices what do you look to do from the start? What is the most important first few steps?

The first few steps occur before the actual testing starts. The most important aspect of any test is understanding how the device will be used in that particular environment and how the device itself works. Without that solid understanding, you’re testing without context, and testing without context is meaningless. It’s like if your son comes home from school and says, “I got a 25 on my test.” What does that mean? A 25 out of 25, a 25 out of 100? What was the average score? What did he get on the last test? Is this in an easy subject or a hard one? Understanding how a device will fit into a particular environment and then testing it congruent to that implementation is what we at Sarrel Group refer to as “real-world testing”. Testing in a void won’t teach you anything about how a device will perform in the real world. Sometimes it takes more time and resources to create an appropriate test bed than it does to run the tests. This is one thing in particular which BreakingPoint greatly simplifies because we can go to a client site and capture real network traffic, then return to the lab and replay it to test network equipment.

4) What are you looking for in your test results? Does this differ from what the vendors are looking to get out of the testing?

As I mentioned in my previous answer, Sarrel Group focuses on real-world testing. My technicians and I have years of consulting experience and have built hundreds if not thousands of networks. We know the issues that customers face when they implement a new technology, and Wall Street firms frequently implement new technology in order to gain a competitive advantage. Most test labs, and I’m being generous here because as far as I can tell it is almost all other test labs, don’t employ technicians with real-world experience. The result is testing under unrealistic conditions which yield results that are inappropriate for most implementations. Interestingly enough, this is almost exactly what most vendors want because almost every device will perform better in a sterile lab environment than in a complex real-world environment. Vendors want test results that max out their equipment and show how fast it is and many times test labs will resort to unrealistic test scenarios to meet these goals.

We see little value in that at Sarrel Group, and I would like to think that the industry has gotten smart enough over the years that corporate buyers see little value in that kind of testing also. For example, I tested an IPSec firewall earlier this year. The company said it could process traffic at wire speed. The sales and marketing materials all said “wire-speed”. When we got it into the lab it was a wholly different story. It could process non-encrypted traffic at wire speed. Turn on IPSec and performance was cut by a third. Then, we advanced from simple Layer 2 bit blasting to run advanced application traffic through the device and performance was cut by another third. So all of a sudden, the 100 Mbps VPN is now a 30 Mbps VPN. You can only get meaningful results by testing under real world conditions.

5) How has testing evolved since you started in the industry?

In the old days, meaning 1990, in order to test we needed to build complex networks with hundreds and sometimes thousands of PCs to generate load. This meant that we need to write all kinds of scripts to coordinate and manage the PC’s and need a few technicians around whose sole purpose was to maintain the testbed. Then automated test tools started to appear and it became possible to simulate an enterprise network environment more easily. But then network equipment and applications got more sophisticated and testing took a lot more knowledge and resources to do properly. I think that this is where most test labs stopped advancing and stayed focused on lower layer simple testing which many of us call bit blasting.

Other test labs, like Sarrel Group, developed methodology to test more complex layer 4-7 devices, but using the old equipment it wasn’t all that easy. One of the reasons why we’ve partnered with BreakingPoint is that these test devices were designed to conduct more complex testing at higher layers and this is the direction the industry is moving in. The days of simple bit blasting are over and the test labs who realize this earlier will be able to test bleeding edge products more quickly and easily than those that don’t.

Posted by Kyle Flaherty

Printer Friendly Create PDF

0 comments
Tags: 10/40/100 GigE // IPS Testing // Layer 2-7 Traffic //