Mar 24, 2011

The Elephants in the Room: Meeting the Challenges of Low Latency and High Volume for Financial Exchanges

by Kristi Thiele

By Kristi Thiele

If you’ve spent any time working with the IT infrastructures of companies involved in financial trading and exchanges, you know all about the pressure to do more things in less time. The obvious pressure is to achieve the lowest possible latency in exchange environments, such that each transaction happens in fewer microseconds. This mandate makes sense when you consider that losing even a few microseconds could result in a 10% drop in revenues. But as High Frequency Trading (HFT) becomes an essential component for staying competitive, the time it takes to get to market with new technology also becomes paramount.

Lately I've done a few weeks of work with several companies in Boston, New York, and Philadelphia. Some of these organizations provide trading and exchange services themselves, while others provide co-location access to market and exchange data. But all of them have to deal with the overlapping challenges of low latency, high data volume, and realism in simulations. That last one, unfortunately, is too often overlooked.

The Business Problem: Nanosecond Latency at High Data Volumes

According to Edgar Blum, Head of Application Engineering at SIX Swiss Exchange, the demand for HFT has led exchanges to look for ways to bring down latency while handling far more transactions. In a Market View magazine article on competition in the exchange industry, Blum said that, for his company, “latency is currently 800 microseconds and throughput is around 10,000 transactions per second on a single engine.” However, he continues, “We are aiming for below 200 microseconds and throughput of at least 50,000 transactions per second per engine.”

To be competitive, exchanges have to provide differentiated solutions quickly, which means doing things faster. But as another Market View article pointed out, “the elephant in the room” is data volume. The elephant article included statistics from the Aite Group with predictions of “volumes to average 1.2 billion messages per day by 2011.” With the amount of transactions increasing, infrastructure and applications must be validated to ensure that they can keep up and maintain the volume while reducing latency.

If all of this seems obvious, it’s still not common practice. On my tour of East Coast exchange companies, I had several conversations that focused on how organizations are validating vendor data sheets for performance. Every one of the IT professionals I talked to addressed latency; not one conversation discussed the impact that data volume might have on latency. At least as far as I can tell, that big gray elephant isn’t being talked about.

The Pink Elephant: Realistic Simulation

If industry pros aren’t talking about that gray elephant, maybe it shouldn’t be surprising that they’re also avoiding what I’m calling the pink elephant: realism in simulations. Let’s use the analogy of the road system to think about this. Using traditional network testing tools, simulating a superhighway full of different types of vehicles — much less coordinating them so that everything moves at top speed — would be only a dream, a pink elephant. But I’m here to tell you that the pink elephant is real. Realistic simulation is possible, and in fact it’s necessary for HFT exchanges. Until they implement processes for realistic simulation, they won’t meet the technical or business challenges of getting their products and services to market faster as volume goes up and latency requirements inexorably go down.

If you’re in this position, what do you do about it? The key is to validate the whole design of your infrastructure as you build it out. Validation needs to include scenarios that stress individual devices as well as the entire system. Each of these scenarios should mimic your real-world conditions — user load, sessions, mix of applications, and so on — as well as what-if cases for security attacks and malformed traffic.

Putting the Pink Elephant to Work: A Real-World Example Using the FIX Protocol

Here’s how I put the pink elephant through its paces at one brokerage house. The staff there wanted to find out whether the data sheet for a particular switch was accurate; it claimed that the device added only 600 nanoseconds of latency, but the staff had no way to validate that. Once I arrived on site, I set up my BreakingPoint Storm CTM and ran a baseline process to show that a piece of fiber cabled between two of the 10G interface ports on our product added an average of 825 nanoseconds of latency.

Rather than using generic traffic (simple TCP or HTTP sessions would hardly produce an accurate result for the exchange), the scenario used an accurate mix of real-world application traffic, with nearly 60% of the bandwidth from FIX protocol sessions [PDF link]. Using the same fiber, I ran the same scenario through the switch being validated, then compared the results. The result was an average latency of 1.425 µs, 600 nanoseconds more than the 825 nanoseconds of latency created by the fiber alone. Right away, the staff at the brokerage knew that the data sheet claim was valid — not in the abstract, but as the switch handled their kind of traffic.

Such realistic simulations can — and should — be repeated and extended across many scenarios of load, applications, attacks, and corrupted traffic. This will only become more important as HFT environments become ever more demanding in terms of latency. (Keeping up with these challenges is one reason that our newest product, the BreakingPoint FireStorm CTM, is built to measure latency down to 10 nanoseconds.)

Superior nanosecond latency numbers are what HFT organizations are trying to reach with each device. But dealing with the gray elephant of volume and the pink elephant of realistic simulation allows them to address the other challenge: getting to market quickly. That’s where the technical advantages of efficient, holistic validation of network and data center infrastructures meets up with the strategic objectives of exchange companies.

So, what about those two elephants in the room — are you ready to talk about them?