From the Customer to BreakingPoint to You: Implementing TR-069 for Client Simulator
by Chris AdamsBy Chris Adams
Maybe you've heard the same horror stories I have about network and security device vendors taking forever to implement support for application protocols that customers need. Certainly there are a lot of enterprise users of this equipment who have had to live through these horror stories again and again. At BreakingPoint, we get around that problem with a unique product architecture, a dedicated team of application engineers (of which I'm one), and, if I say so myself, a real commitment to delivering what our customers need. In this post I'm going to walk you through our process of delivering – and delivering again – on the protocol needs of one of our telecom customers.
First, some background: last summer, we released support for the TR-069 broadband protocol in the Application Simulator component of the BreakingPoint Cyber Tomography Machine (CTM). You can read about that release in a post I wrote a little while back. For this post, I will assume you are familiar with the TR-069 protocol or have read the previous blog post to get the basics. As a reminder, Application Simulator is the component in our products which lets you send realistic traffic at Layers 4-7 by acting as both the client and server.
The overall protocol was not too difficult to implement, since TR-069 is an open standard and our systems engineer (SE) working with the customer gathered packet captures (PCAPs) demonstrating the specific protocol actions that the customer wanted implemented. (When we know it will help enough users in the long run, we often add protocol coverage at the request of individual customers.) Since we had such a solid base to work off of, the implementation was very straightforward.
However, the customer originally wanted TR-069 support in our Client Simulator component as well. The major difference here is that Client Simulator, as its name would suggest, simulates the client alone, so it works in one-arm mode to validate server-side equipment. In the rest of this post, I am going to discuss that part of the development process – the issues we encountered, how we solved them, and how we were able to implement TR-069 for Client Simulator without ever having any auto-configuration server (ACS) or customer premise equipment (CPE) devices in our lab.
The Challenge of a One-Arm TR-069 Implementation
Realism and accuracy are two of the highest priorities we have when implementing a new protocol. Thanks to the open standard and the PCAPs already mentioned, we knew we had the realism part down for TR-069. Our product was able to reliably simulate all of the actions seen in the PCAPs, but unfortunately, we had no ACS systems to test against in order to implement a Client Simulator version, so we initially shipped TR-069 with Application Simulator support only. (We would never want to ship a Client Simulator application that we had not tested and verified internally.)
Still, the challenge remained: the customer also wanted to test their ACS infrastructure, not the just intermediate components between the ACS and CPEs. What could we do? We wanted to deliver for them, but we just didn't have the resources to do so. It was time to get creative and look for alternate ways to solve the problem.
I went back to the drawing board and thought about how we had launched into TR-069 in the first place. When the project started, I had set out to write the Client Simulator implementation of the protocol first. After I got through the Inform and InformResponse actions, I moved on to SetParameterValues and SetParameterValuesResponse. Things were going great . . . until I got stuck at the point where the ACS sends a special SetParameterValues message which updates the CPE's authentication credentials. In the real world, this would happen as part of the bootstrap process through which the factory-default username and password get updated with the ISP's configured values.
So now I was stuck trying to implement the new SetParameterValuesResponse action from the CPE to the ACS. How could we confirm that our solution would work without testing it? In particular, how could we be sure that the ACS could send new authentication parameters to the CPE as part of the bootstrap process? If that step failed, all of our subsequent actions in our Super Flow would fail.
One option that we discussed for getting around this obstacle was to send code changes to our SE, have him run the tests on the customer's premises, and then have him email PCAPs and logs back to us. But we all agreed that this would drag out the development process, so we took a different approach. After some polite negotiation, our customer agreed to provide us with live access to their lab. Our SE would travel there and set up a BreakingPoint Storm CTM, and the customer would provide us with secure shell (SSH) access for our device. At that point, we could have full access to the ACS equipment in their lab.
To give you an idea of why we could not simply stand up an ACS in our own lab, we are talking about a multi-tier architecture with application Web servers and a backend database capable of managing thousands of CPE devices, not just a single application that we could run with a simple Web server. But while that set-up would be totally impractical to implement in our own lab, it was already up and running in our customer's facility. Now we were getting somewhere!
Adapting TR-069 for Client Simulator
Once all of these steps had been taken, I used SSH to get into our device and got to work. The first thing I did was to create a Super Flow using our existing Application Simulator actions for TR-069. Things worked surprisingly well up to the point that the SetParameterValues message was received by the server. I made sure that the Super Flow was configured with all of the right values specified by the customer so that our device would behave just like a CPE in their network. Up to this point, the only changes we had made were in configuration XML.
Now we were ready to start modifying our Ruby code. I modified the code that handles our SetParameterValuesResponse and Inform messages so that it would use the new authentication parameters provided by the ACS. At that point, we were able to get the system completely bootstrapped. The customer checked their database and confirmed that we had successfully registered as a provisioned CPE! After this, the remaining actions were pretty simple to work out – again, it was mainly XML editing rather than changing the underlying protocol implementation that we had already shipped for the Application Simulator.
Lessons Learned
What did we learn from this? First, that if we are given clear specifications, we are able to produce a very accurate protocol implementation without ever touching a real system. The original implementation really was as close to ideal as you can get and one which every engineer appreciates. So although we already had the application, in practice there could be minor incompatibilities which could have caused the whole thing to fail. As the saying goes “In theory there is no difference between theory and practice. In practice there is.” Don't get me wrong, we love to get as many different systems into our lab as possible, but we also realize that this is not always feasible.
Second, we learned that even when it's impossible to get a system, or several boxes comprising a single “application” in this case, into our lab, that need not stop us from developing a protocol. We're dedicated to our customers, and we are willing to go to great lengths to deliver realistic and accurate applications to meet their needs. In my experience, we would always be willing to listen to a customer who wanted to provide us with a similar arrangement so that we could implement a protocol for them.
Everyone in IT knows the disappointment of a vendor over-promising and under-delivering. By contrast, we were able to turn out TR-069 in a matter of months, and then a few months later deliver Client Simulator support with just one more week of development effort. In the end, both BreakingPoint and our customers came out ahead: we improved our product – already the best in the industry – by adding one-arm and two-arm implementations of an important application protocol, and all of our customers got the benefit of using them. We will continue to impress our customers by exceeding their expectations and giving them exactly what they want. And how many vendors can say that?
Related posts:
