AUGUST 27, 2010

LTE Diaries: GTP Tunneling

NOTE: This is one post in a long series, catch up on the most recent LTE development updates, or check out our recent LTE webcast.

The last time I posted to the blog we were talking about questions surrounding IPv6 deployment. You might remember from that post that I am a software architect at BreakingPoint Systems, specializing in design and performance for the network processors in our products. Today I get to continue our "LTE Diaries" series where we have outlined our plans for LTE support and the speed at which we are able to make this happen, thanks to the network processor-based architecture. If you haven't read the previous posts I would encourage you to read up on our overall LTE development, the S1AP implementation and our SCTP support. Today we will tackle the GPRS Tunneling Protocol, or GTP. GTP is a layer-4 protocol used for a number of different back-haul tasks in a 3GPP/LTE environment. It comes in two flavors, GTP-C (control plane) and GTP-U (for user data). I have had the opportunity to implement parts of both GTP-C and GTP-U for BreakingPoint's upcoming LTE support, and would like to share some development notes.

What Is GTP?

GTP-U is the simplest to explain. It is used to tunnel user data across an internal telecom network. These networks comprise of the various devices between a mobile device and the rest of the phone system and/or internet.

Some key functionality GTP intends to provide include:

Isolation of user traffic from the telecom network itself

GTP-U functions similarly to other network tunneling/isolation protocols, such as 802.1Q, PPP or GRE. GTP typically appears inside of private IPv4 networks (address encodings for tunnel endpoints might have space for IPv6 addresses, but it's not clear how to specify them), but tunnels can encapsulate any IP protocol version, or PPP directly.

Traffic classification, billing and QoS

Devices can request multiple tunnels and obtain different amounts of bandwidth, billing methods and QoS. This flexibility could allow a device to setup one tunnel for low-priority background updates, say for email or text messages, and another tunnel for voice traffic.

Packet ordering

GTP can optionally maintain packet ordering, but has no reliability mechanism.

The kitchen sink

GTP is somewhat like HTTP or SUN-RPC, as it gets used for a lot of miscellaneous purposes besides its main one, which is to tunnel data. The GTP-C protocol has several dozen extra messages besides the normal 'setup tunnel/tear down tunnel' variants. These tend to be device specific, as a pair of devices in a telecom closet may speak one dialect of GTP-C on a particular link, and another pair of devices may speak a different subset.

An Insider's Game

Like many protocols in the telecom industry, GTP is not a product of the Internet Engineering Task Force (IETF), though its specifications are managed similarly to the familiar RFCs of the internet. GTP and many other specs are managed by the 3rd Generation Partnership Project (3GPP). These specifications can be read as drafts (.DOC files) for free. One can also download final PDF specs with a free login, but only a limited number of times. According to their website, they have been producing protocol specs for about 12 years, though I have to admit they would have slipped under my radar if I hadn't been specifically looking for information on GTP.

Interoperability

While the specs may be open or at least free to access, it can be difficult to find open implementations in the wild for testing and experimenting. As with many internet protocols, the spec is realistically only half of what you need, as nothing really beats a reference implementation and a packet analyzer. Implementing a spec doesn't get you anything if your stack can't interoperate.

For one reference implementation, we licensed a commercial GTP stack. This stack provided a nice source of packet captures and a good baseline reference, especially since it is used in various real mobile network support devices. However, its code complexity is somewhat high and it has a lot of features (implementing the 'kitchen sink' mentioned above) that we didn't need.

Somewhat advantageously, there has been a lot of interest lately in implementing an open-source GSM stack, with Harald Welte leading the charge with the OpenBSC project. An adjunct to that project is a piece of software called OpenGGSN and with its libgtp stack, you can use a regular Linux PC as a GTP endpoint. This is a nice "lean and mean" stack and it even sets up interfaces for tunneling gateway->phone traffic. GPL licensing means we couldn't integrate it either into our realtime networking stack, but having a second implementation has been very helpful for testing. It is also worth checking out Harald's blog for insight into interoperating with these formerly esoteric protocols.

Reference implementations in place, I've been spending some time pitting them against each other as well as against the third lean and mean stack we're implementing for our product. Having two stacks to evaluate against (in addition the the third one I'm developing) means that we will have both a nice pile of bugs on all sides before release.

Fragile Specs: Flexible, But Also Fuzzable

One of the interesting things about GTP is how it is intended to run on private, isolated networks. At least according to block diagrams, there often appears to not even be a switch between various telecom devices, rather they are cabled directly to each other via specialized interfaces on each device. This sort of presumed isolation both simplifies implementations, but can also allow for implementations not terribly well exercised for resiliency and security.

For example, a GTP header contains a variable number of fixed and flexible-sized options fields. These fields, called 'Information Elements' are encoded in a half dozen ways (ranging from big-endian 4-byte integers to something called 'Telecom Binary Coded Decimal'), and on top of that, all information elements must be encoded in sorted order in the packet. Given how some routers were confused by Linux changing the order of TCP options, which are not specified as ordered in the first place, it will be interesting to see the effects of not following ordering. It will be especially interesting for options that build on each other (I see a NULL pointer dereference in my future).

Another oddity of GTP is its seeming disregard for alignment efficiency, at least in C or on modern processors. Historically, internet protocols have tried to align packet headers to convenient sizes for contemporary CPU architectures. You can date a protocol by the types of alignments it attempts. IPv4 options are aligned to 2 byte units, which was fine for 16-bit CPUs that were popular in the early to mid 80's. IPv6 is aligned to 8 byte units, to reflect the modern changes to 64-bit processing. At most, GTP, GTP-C in particular, only guarantees 1-byte alignment for header data. As a result, you typically have various headers starting at bytes 3, 5 or 7, spanning cache lines and registers. Of course, it could be worse. If you check out 'LTE: The UMTS Long Term Evolution's" chapter 4 on user plane protocols, this is an apparent improvement, fields in older protocols were not even byte aligned (something only an ASIC or FPGA developer would love).

Of course, to achieve even byte alignment, some sacrifices must be made. In places, the spec identifies 'spare' bits, which are used to pad out small fields to use whole bytes, and which may have a use at some time in the future. However, in my testing, these spare bits are required to be set literally to '1' and '1' only. Setting these supposedly unused spare bits to '0' yielded errors communicating with both of my reference stacks. So much for future use.

When configuring a GTP tunnel, the endpoint places a high-degree of trust in the initiator. So much so, the endpoint allows the initiator to decide the tunnel ID that it wants to use. Tunnel IDs are 32-bit values, though it is unlikely that any real device can support that many. However, since the initiator controls the tunnel ID, it can hijack any existing tunnel (tunnels can be reconnected at any time). It is also easy to DoS the endpoint, either by setting up many tunnels, or manipulating the tunnel ID so that it hashes poorly.

Other GTP Performance Notes

Another interesting note about GTP is how it encapsulates IPv4 and IPv6 packets, but does not specify any particular MTU to the endpoints. A GTP-U packet over regular ethernet adds 40 bytes overhead to the packet. Also, the GTP-C protocol does not specify a particular MTU for endpoints when a new tunnel is setup. This means that a single GTP-U tunneling 1500 byte MTU packets will be fragmented. Thus, GTP-tunneling devices, if not using jumbo packets on their interfaces, will suffer a considerable loss in network performance, compared to unfragmented traffic, because a normal 1500 byte IP packet will be split into two packets, one only 40 bytes. This is in addition to the extra CPU and memory used for reassembling these packets at the endpoints. A quick survey of current 3G phones reveals that they all have apparent MTUs of 1500 bytes.

GTP implements packet ordering preservation, which is optionally enabled. However, there is no acknowledge or retry mechanism, so if packets come out of order or are lost, reordering is only a best-effort mechanism. One could conceivably cause denial of service to a device by forcing all connections to maintain order. It will be interesting trying this out, although it may turn out this is not an often-used feature. Interestingly, early versions of GTP allowed the use of TCP or UDP as a transport. The latest versions only specify UDP. This makes me think that TCP also presented enough overhead (reliability plus ordering) that it was seen as redundant and overly-costly.

Summary

I'm having a good deal of fun implementing and testing GTP. On one hand, it is interesting to contrast the 3GPP-developed protocols to IETF ones. A similar protocol developed for the internet might utilize something like XMPP or XML-RPC in place of GTP-C, and PPP or GRE for tunneling instead of GTP-U. Primarily, the drive for GTP seems to have stemmed from the desire to isolate internal telecom network traffic from the potentially dangerous internet traffic. Also, it's not the most efficient protocol, being susceptible to fragmentation and packet misalignment. However, on the other hand, it is one of the simpler LTE protocols, and is fairly well specified (almost to a fault), and has multiple implementations, which are helpful for testing robustness and interoperability. You can look forward to GTP support in a BreakingPoint release near you.