May 31, 2011

The Cost of Corrective Action vs. Proactive Network and Cloud Testing: Lessons from the Sony Breach

by BreakingPoint Labs

$171 million. That's the initial cost Sony announced for last month's breach of the PlayStation Network and the 24-day outage that affected about 100,000,000 customers. You can be sure those costs will keep mounting over time, maybe even surpassing $1 billion, now that more hackers are probing other vulnerabilities. Now that we have hard numbers straight from the source, it's worth asking a broader question: Is the Sony breach the trigger that other companies need to get smarter about cyber security testing?

Your grandma probably always told you “An ounce of prevention is worth a pound of cure.” And most would agree that it’s true. However, most of us ignore this good advice. In the Sony case, we knew that an “ounce” of network and data center resiliency testing would equate to a miniscule fraction of the costs of after-the-fact remediation. Top-performing companies have already gotten ahead of this issue: They have devoted the appropriate budget and built the resiliency practices needed to catch these problems before they ever happen. More companies need to follow in their footsteps.

Companies seem serious about cyber security, as they’ve shown by spending billions to build up layers of protection trying to thwart attacks. But the hackers are smart and agile, and it’s time for businesses to behave that way, too. Otherwise, those hit by breaches and other failures will continue to find out the hard way about the huge and costly ripple effects that such incidents create. The rest of this post gives you a few arguments you need to secure the budget for hardening your infrastructure.

The High Cost of Correcting Security Failures

The direct costs created by breaches and downtime are large and obvious, but they are only the beginning. For a less sensational example than Sony, consider the outage that hit the Commonwealth of Virginia's network last year. The total compensation agreed upon between Virginia and its IT contractor was about $4.75 million. But only $1.9 million of that "represents ... direct costs incurred by the Commonwealth as a result of the disruption." The rest of the total comes from $2.1 million in operational improvements such as database backup, plus $750,000 to implement corrective technologies. And all of that comes on top of $250,000 that the contractor had to pay earlier so a third party could audit Virginia's systems just to figure out the extent of the problem.

The Virginia example doesn't even get into some of the worst ripple effects. Even a company much smaller than Sony could face costs in all of the following areas for a data breach or similar problem:

Brand Damage — These incidents do long-term damage to a company's brand, and to the sense of trust that customers and potential customers have about that company. All of this could be summarized as the cost of lost credibility — something that Sony will be wrestling with for a long time. Like many companies faced with embarrassing situations, Sony is making a down payment on regaining trust by giving customers extras like free premium services — not to mention identity-theft monitoring.
Legal Costs — While settlements or damage awards from civil lawsuits will rightly draw the most attention, these costs extend in many other directions, too: internal investigations, handling lawsuits, preparing for civil and compliance hearings, et cetera. While we’re on this subject, it’s worth noting that Sony could be found noncompliant with the Payment Card Industry (PCI) security standard, which would be a particular headache. In fact, Sony has already been sued on those exact grounds.
Regulatory Costs — It’s hard to guess exactly what regulators will do, but it is safe to assume that Sony will face some combination of monetary fines and additional compliance costs for extra filings, hearings, and more. Sony has already started to feel the bite of increased regulatory scrutiny in its home country of Japan, where authorities put the brakes on the relaunch of the PlayStation Network until the company could satisfy them that it was putting better security measures in place. Even for smaller companies that operate with less of a global footprint, it is impossible for a company’s cost of compliance to go down when regulators begin combing through its business.
Technical Costs — Breached networks must be rebuilt or reconfigured to perform like they should have in the first place. But those costs multiply when a company must perform remediation in crisis mode. You should always take the time to plan infrastructures intelligently, test them thoroughly, and implement them systematically. But organizations responding to an emergency will see their IT staff — plus outside contractors — working day and night to come up with solutions. It's like trying to change a blown-out tire at 100 mph.
Administrative Costs — This covers everything from additional staff time spent emailing affected customers to the extra bills racked up as customer service reps deal with angry PlayStation Network users. While this category may seem routine, it has already accounted for millions of dollars in expense for Sony.

The Low Cost of Proactive Resiliency Testing to Validate Performance, Security, and Stability

Getting smart about IT infrastructure resiliency — and avoiding the landslide of costs mentioned above — means implementing systematic validation of devices, networks, and data centers. Each element of the infrastructure must be tested ahead of time under real-world conditions to ensure its performance, security, and stability once it is deployed in production. That’s exactly what we enable with our BreakingPoint CTM products and professional services. Taking this approach is similar to how telecommunication service providers have validated every component of voice and data infrastructures for years.

The good news is that today enterprises don’t have to build out their own massive labs or cyber ranges, because a BreakingPoint CTM compresses the power and adaptability of a full cyber range into a single 7-inch-high box. A lot of enterprises might be surprised to find out that our technological innovations have brought the cost of comprehensive testing equipment well under a million dollars. Even less costly are on-demand professional services that use such equipment to handle particular resiliency projects. That's a long way from the nine figures — and counting — that Sony has already racked up.

Forward-looking companies are making resiliency testing a regular part of their day-to-day operations. They ensure network and data center resiliency the same way that a traditional manufacturer would ensure fire safety in its factories, or that any legitimate business would audit its books to catch financial irregularities. Unfortunately, too many other companies fail to invest any resources in augmenting testing expertise, processes, or equipment. We can hope that Sony's experience will change that mindset.

We're All Networking Companies Now

Maybe you think you don't need to worry about these costs because your company doesn't provide cloud services like Sony does. True, the lesson of the Sony fiasco is particularly relevant for other cloud providers, but it's not exclusive to them. The truth is that many companies in industries seemingly unrelated to networking — fields like banking and pharmaceuticals — have become de facto networking companies, simply because they must run such large networks as a basic requirement for conducting business. Just like cloud providers and traditional telecommunications carriers, these businesses must think ahead about how they will ensure resilient online services for their customers or face increasing risks of costly failures.

While Sony's final tally will be a long time coming, you can be sure that $171 million is only the tip of the iceberg for the company. Meanwhile, you can also be sure that other companies are racking up losses from data breaches and other security failures, too. It's just happening out of the limelight — as in the Virginia case, which drew far less attention than the recent outages at Sony or Amazon. But whether they've made the headlines or not, those losses could have been eliminated, or at least radically diminished, with an ounce of smarter prevention.