Emulating Netflix: Delivering Video Through the Cloud
by Chris AdamsUPDATE: BreakingPoint's Application Load Evaluation Service is now available to help you you properly test applications within cloud and virtualized infrastructures.
By Chris Adams
I recently implemented Netflix streaming for the Application Simulator component of the BreakingPoint Storm CTM, and found out a lot about Netflix in the process. Netflix has been in the news a lot lately, and with good reason. If you're a service provider in the United States or Canada, then you have undoubtedly seen their traffic take them from being a bit player to one of the top five users of your bandwidth. Sandvine recently reported that Netflix streaming accounts for more than 20% of nightly traffic in North America. In fact, Netflix is such big news that Level 3 is caught up in a battle between Comcast (and therefore NBC and Hulu) and Netflix. Ars Technica has an excellent and thorough write-up of the situation.
In this post, I'm going to walk you through some technical background that will help you understand the Netflix architecture, then get into our specific implementation, which should be useful both for the cable and telecom service providers that stream Netflix to your set-top box, and for the makers of those set-top boxes. After that, I'll offer a few other thoughts on the business implications of the Netflix video-on-demand model.
(If you like this topic, you should also check this out: Agustin Schapira over at Pomelo, LLC did a great writeup of the Netflix security architecture in April 2010. In it, he highlights the key functions performed by the various servers and also mentions using the Tamper Data extension for Firefox. This is a great tool for inspecting SSL sessions in your browser to peek under the hood. Beware, however, that using Tamper Data will cause these sessions to be cached on your machine, so be sure to clear your cache if you're using it for anything sensitive.)
Technical Background
In order to stream a movie, the Silverlight player and web browser that Netflix uses must initiate connections with a handful of servers. Interestingly, most of these are not run by Netflix, which has gotten a lot of attention for embracing cloud services. The company uses Amazon Web Services (AWS) for many functions, but here we are primarily concerned with AWS as it pertains to movie streaming. Netflix's movie content is hosted as naked WMV or WMA files with their content delivery network (CDN) partners. This allows Netflix to avoid complex content management, but also requires a robust access control system, which we'll investigate in a bit.
After developing an understanding of the Netflix architecture, my next step was to examine the content that they were serving. I experimented with several short videos, some TV show espisodes, and a couple of feature-length films. In case you're curious to see diagnostic statistics for Netflix streaming, be sure to check out Matt Smith's writeup about the Silverlight player. Digging a little deeper, we can look at Neil Hunt's post on the Netflix blog about their encoding process.
One of the impressive feats of Netflix's streaming service is how they rely on Transmission Control Protocol (TCP) to handle congestion control and segmentation. Many streaming applications perform their own congestion control, either because they employ User Datagram Protocol (UDP), or because their authors felt that TCP's algorithms were insufficient. With Netflix, the client requests a chunk of the audio or video stream, and that chunk is returned in a single HTTP 200 response split across multiple TCP segments.
It turns out that this works very well in practice. My family has been using Netflix streaming since it was first introduced. Initially, we would have long buffering times. Now, however, the load times are very fast, and the quality is very good. Presumably any client player software or hardware would have sufficient memory to buffer the large chunks of data that occur as the tradeoff for using TCP's segmentation and congestion control.
Implementation
At this point, I had enough info to start writing code. We will need connections to three servers: the Netflix web server, the Netflix access control server, and a CDN server. Since the focus of our implementation is simulating high numbers of users streaming videos rather than the minutiae of the one-time client initialization, we can ignore many of the Silverlight-specific details. However, the Pomelo blog post already mentioned covers these in good detail. As I mentioned above, Netflix uses a robust access-control mechanism, which simplifies accessing content hosted on CDN servers.
Checking out our Tamper Data dumps, we see lots of XML mentioning "nccp" for the host agmoviecontrol.cloud.netflix.net, which turns out to be the AWS-hosted access control server. For example, here is a ping transaction between the client and server:
POST https://agmoviecontrol.cloud.netflix.net/nccp/controller/2.7/ping
<?xml version="1.0" encoding="utf-8"?>
<nccp:request xmlns:nccp="http://www.netflix.com/eds/nccp/2.7">
<nccp:header>
<nccp:softwareversion>1.0.28.2</nccp:softwareversion>
<nccp:payload>AhACOxVjqpnHUcLOsnuJXeHmgJCd4K0Bv/d17VKOv2lGkYk0/1tzha54ZMXpvVqgnkfjIoE7ku1Xn4yF2Pefsh/pRBvDfc0bpqO+QjracyZWsxLNONk58DX/U9TOoQT6qOiBG/uok6YmCLQoehUrTZUKIJe/tmamI3C7RCAO+odvoO3I36mKL7zH4+IRG38E+xzFrrZY21FqKCmHW3+GO8ENx1Q=</nccp:payload>
</nccp:header>
<nccp:ping />
And the server response:
<HMAC>F3DJGvVlNW9adVV63/t3nEdaCHwsni9AoHPe9vJuzB0=</HMAC>
<?xml version="1.0" encoding="utf-8"?><nccp:response xmlns:nccp="http://www.netflix.com/eds/nccp/2.7">
<nccp:responseheader>
<nccp:payload>AhD/Z6PfeMQOZKG4CHszAsCqgJB2N8RuBpjVrdYePi+9GLlVHIu6Hl/8ubFaT4et4xTepT2JuvHF54LIFcpXUf6/uCx3++2/R/+hCilqb3KCY1Dbrf/y1N9jK06HeIMiwxVxuF4/d5QkHcu0mFb+soR0HyKtgSrSLuh9jda/j6nd3JYMtUOa+XcNipv00b7wMufrpLNucr5kAhnBwceO9oRcnxQ=</nccp:payload>
</nccp:responseheader>
<nccp:result method="ping">
<nccp:status>
<nccp:success>true</nccp:success>
</nccp:status>
</nccp:result>
<nccp:parameters>
<nccp:retrycontrol>10</nccp:retrycontrol>
<nccp:mintimeout>60</nccp:mintimeout>
<nccp:lowfrequencypollinterval>28800</nccp:lowfrequencypollinterval>
<nccp:mediumfrequencypollinterval>60</nccp:mediumfrequencypollinterval>
<nccp:highfrequencypollinterval>15</nccp:highfrequencypollinterval>
<nccp:hightomediumpollswitchinterval>300</nccp:hightomediumpollswitchinterval>
<nccp:mediumtolowpollswitchinterval>600</nccp:mediumtolowpollswitchinterval>
<nccp:loginterval>60</nccp:loginterval>
<nccp:maxlogsize>2000000</nccp:maxlogsize>
<nccp:loglevel>error</nccp:loglevel>
<nccp:supportphone>866-579-7113</nccp:supportphone>
</nccp:parameters>
</nccp:response>
What do we see here? First, every message between the client and server has at least one payload which is a base64 encoded blob. This is undoubtedly related to the specific client player. All of these messages begin with 0x02 0x10, and are 164 bytes long when decoded. There are other base64 encoded strings of different lengths that turn up elsewhere, but the first two bytes are always 0x02 0x10. We also see that each server response has a hashed authentication code prepended to it as well.
The other POST Uniform Resource Identifiers (URIs) include authenticationrenewal, moviemetadata, usermoviemetadata, authorization, and license, with each doing roughly what its name implies. For the moviemetadata, the client sends the movieid, which is originally requested by the browser as:
GET /WiPlayer?movieid=<movieid>&trkid=<trkid>&strackid=<strackid>&strkid=<strkid>
The fields show up again later, with the movieid and trkid sent by the client with each request and the others showing up again in the authorization message. There are also a few other uses for a movieid, although they may be against the Netflix terms of service. For example, if you wanted to get the movie's cover art, you could fashion a URL like this:
http://cdn-<number>.nflximg.com/us/boxshots/<size>/<movieid>.jpg
The size choices are tiny, small, big3, large, gsd, and ghd, and a few other variants.
http://cdn-5.nflximg.com/us/boxshots/ghd/567905.jpg
As I mentioned, grabbing data like this is probably against the Netflix terms of service if you issue a large number of requests. But it is all readily available via the API if you sign up for their developer program, so go to the Netflix Developer Network if you want more details. [Editor's note: while the link is correct, that Netflix page was down at the time we published this post.]
Getting back to the metadata messages, we see the server response with the movie's metadata containing the movieid, the metadata expiration, and the actual metadata: preview URL, box art URLs, synopsis, release date, runtime (in seconds), actors, directors, rating, genres, and the availability start/end times (as Unix epoch seconds). Based on my samples, the availability times look like they run on a quarterly basis, which might indicate that they are related more to Netflix's licensing agreements than to any specific technical reason. (You can see my XML code for this at the bottom of this post, if you like – it's a bit long to include here.)
Next we see the POST to the authorization URI. The client sends the movieid and trackid (seen earlier as trkid), and the desired video and audio profiles. The server then responds with the CDN locations for the video and audio files in all of the available bitrates. Each download URL has an expiration tag, which was 3 hours after the request in my sampled data. Furthermore, this number is embedded in the URL either as "e=<expiration>" or as part of the "token=<expiration>..." depending on which CDN is used. Since the client receives URLs for all of the available bitrates, it can dynamically change between them based on bandwidth overhead. Netflix claims that players will choose bitrates that will maintain 40% headroom for available bandwidth.
Finally, the client POSTs to the license URI. The client sends a giant base64 encoded SOAP request which has all of the normal client PKI bits in it. The server responds back with a base64 encoded payload containing the decryption key.
That was a lot of work just for the setup. Let's hope there's a good payoff!
The client will now request the WMV and WMA video and audio files, respectively. Since access is so tightly controlled via the setup, these files can sit naked out on the CDN servers. The first thing we notice is that for each file, the client requests the first 64 bytes. Using the maximum bitrate of 1500kbps for video and 128kbps for audio, the remaining video byte ranges are requested in subsequent GET requests with sizes ranging from 136KB to just over 1MB, clustered between 250KB and 600KB. The audio was a bit different. Over several observed samples, the request size alternated between two fixed numbers. One set was 443974 +/- 40635 bytes, and the other was 246125 +/- 762 bytes. It's hard to infer much based on these two samples, but it is worth noting that audio requests are exactly two sizes and that they occur once for every 10 video requests.
Putting this all together, we can take a user-input movieid or generate one randomly, put it together with a user-selected bitrate and movie length (in minutes), and use them to create realistic Netflix metadata. The BreakingPoint Storm CTM encrypts the data using the SSL wrapping feature, then creates streams for the video and audio data. From a user perspective, this is as easy to use as our other Application Simulator Superflows, meaning that it takes only a few clicks to have a realistic simulation running.
That's it for the technical side.
Business Implications
As I researched the Netflix application, I came across some other interesting data that made me think about what video streaming means for Netflix's business. Working from publicly available documents, we can figure out that the company spends $500 to $700 million annually on postage for around 500 million DVDs. This gives us somewhere between $.70 and $1 in round trip costs per disc mailed.
Let's compare that to their streaming service. A quick check around the Web shows current CDN pricing of around $0.05 or $0.06 per gigabyte delivered, and this is consistent with reports of Netflix's negotiated rates (setting aside special or introductory pricing). If a movie is 90 minutes long, and delivered at 1500 kbps for the video stream and 128 kbps for the audio stream, then we see that it is (1500/8 + 128/8)*60*90 = 1,098,900 or just over one gigabyte. This is convenient for calculating per-movie delivery costs.
Let's take a look at Netflix's current plan pricing.
| DVDs | $ Cost | $ Cost per DVD |
Equiv. $ Cost to Stream Movies |
Marginal $ per DVD |
|---|---|---|---|---|
| 0 | 8 | N/A | N/A | N/A |
| 1 | 10 | 10 | 0.06 | 2 |
| 2 | 15 | 7.5 | 0.12 | 5 |
| 3 | 20 | 6.67 | 0.18 | 5 |
| 4 | 28 | 7 | 0.24 | 8 |
| 5 | 35 | 7 | 0.30 | 7 |
| 6 | 42 | 7 | 0.36 | 7 |
| 7 | 49 | 7 | 0.42 | 7 |
| 8 | 56 | 7 | 0.48 | 7 |
We see that the cost per DVD is lowest on the 3-at-a-time plan, while the marginal cost is highest on the 4-at-a-time plan. Does that seem like any guidance to consumers?
According to the New York Times, "Netflix is making less revenue per customer as streaming catches on because customers are subscribing to less expensive plans, with fewer discs and unlimited streaming." Let's examine that for a second. The most expensive plan offered by Netflix is the 8 at-a-time for $56. A customer would have to watch between 56 and 80 movies delivered via physical discs before Netflix loses money on delivery costs for this customer. Contrast this with the cheapest plan at $8, which offers no DVDs by mail. Such a customer could watch between 133 and 160 streamed movies before Netflix loses money on delivery. Netflix recovers $0.64 to $0.94 per movie just by shifting a customer to streaming. That sure beats sneakernet via the US Postal Service. The cost recovery of switching from mail delivery to streaming suggests that Netflix has a vested interest in moving customers toward streaming, especially the heaviest users.
Of course, my speculative numbers leave out other expenses related to delivering movies, such as the overhead for running shipping and receiving facilities and the IT expenses for ripping and managing content. There are also rumors that Netflix rate-limits both the heaviest DVD and streaming users. (To be fair, Netflix says that it doesn't impose rate limits, and it would probably avoid imposing them to prevent a repeat of the 2005 lawsuit—settled out of court—brought by a customer who claimed his DVD service was being throttled. It's also possible that limits are imposed quietly by service providers.)
We see, however, that even under the most extreme examples, the revenue per customer minus delivery costs can potentially shift the value equation drastically in Netflix's favor when streaming is considered. This is a win-win for Netflix and for customers. The only drawback for streaming customers is a limited selection of movies, yet Netflix will surely improve their offerings as they begin to compete more closely with Amazon, Apple, and Google.
Whatever Netflix does next with its movie streaming, BreakingPoint users will be well-equipped to emulate Netflix's unusual traffic accurately to see how it performs on their own networks. If you have thoughts about how to put our Netflix emulation to use, or questions about how I implemented this, please leave a comment.
Example XML for Iron Man:
Client POST:
POST https://agmoviecontrol.cloud.netflix.net/nccp/controller/2.7/moviemetadata
<headers omitted>
<?xml version="1.0" encoding="utf-8"?>
<nccp:request xmlns:nccp="http://www.netflix.com/eds/nccp/2.7">
<nccp:header>
<nccp:softwareversion>1.0.28.2</nccp:softwareversion>
<nccp:payload><220 byte base64 payload></nccp:payload>
</nccp:header>
<nccp:moviemetadata>
<nccp:movieid>70080038</nccp:movieid>
</nccp:moviemetadata>
</nccp:request>]
Server response (headers omitted):
<HMAC>5v2lcfvlc9Tjs5KXYSzA1zf28HEiHaOcKXAFDIJVqpQ=</HMAC>
<?xml version="1.0" encoding="utf-8"?><nccp:response xmlns:nccp="http://www.netflix.com/eds/nccp/2.7">
<nccp:responseheader>
<nccp:payload><220 byte base64 payload></nccp:payload>
</nccp:responseheader>
<nccp:result method="moviemetadata">
<nccp:moviedata expiration="1287765023" movie_id="70080038">
<nccp:contenttype>Movie</nccp:contenttype>
<nccp:name>Iron Man</nccp:name>
<nccp:moviepreviewurl format="SD">http://null/item/?x=oWNexpO_M6R2nasLOoI6ev-2tcekyLUhhkaj7q1c5pYn8WGMnnvrbMN78nYGiRcxQX_39xUDLS7n7lUYJ_1dWGod9Y4kGqdGZM5d4hFaiO97iwSxh2GHDkjWJ-Rx5qoaR7-piYNcX4qrrg..</nccp:moviepreviewurl>
<nccp:boxart>
<nccp:resolution>
<nccp:width>284</nccp:width>
<nccp:height>405</nccp:height>
</nccp:resolution>
<nccp:pixelaspect>
<nccp:width>1</nccp:width>
<nccp:height>1</nccp:height>
</nccp:pixelaspect>
<nccp:downloadurl>http://cdn-8.nflximg.com/en_US/boxshots/ghd_epx/70080038.jpg</nccp:downloadurl>
</nccp:boxart>
<nccp:boxart>
<nccp:resolution>
<nccp:width>210</nccp:width>
<nccp:height>270</nccp:height>
</nccp:resolution>
<nccp:pixelaspect>
<nccp:width>1</nccp:width>
<nccp:height>1</nccp:height>
</nccp:pixelaspect>
<nccp:downloadurl>http://cdn-8.nflximg.com/en_US/boxshots/gsd_epx/70080038.jpg</nccp:downloadurl>
</nccp:boxart>
<nccp:synopsis>After escaping from kidnappers using makeshift power armor, an ultrarich inventor and weapons maker turns his creation into a force for good by using it to fight crime. But his skills are stretched to the limit when he must face the evil Iron Monger.</nccp:synopsis>
<nccp:releasedate>2008</nccp:releasedate>
<nccp:runtime>7500</nccp:runtime>
<nccp:actors>
<nccp:actor>Robert Downey Jr.</nccp:actor>
<nccp:actor>Terrence Howard</nccp:actor>
<nccp:actor>Jeff Bridges</nccp:actor>
<nccp:actor>Gwyneth Paltrow</nccp:actor>
<nccp:actor>Leslie Bibb</nccp:actor>
<nccp:actor>Shaun Toub</nccp:actor>
<nccp:actor>Faran Tahir</nccp:actor>
<nccp:actor>Clark Gregg</nccp:actor>
<nccp:actor>Bill Smitrovich</nccp:actor>
<nccp:actor>Sayed Badreya</nccp:actor>
<nccp:actor>Stan Lee</nccp:actor>
</nccp:actors>
<nccp:directors>
<nccp:director>Jon Favreau</nccp:director>
</nccp:directors>
<nccp:agerating>
<nccp:us-rating>PG-13</nccp:us-rating>
</nccp:agerating>
<nccp:genres>
<nccp:genre>Action Thrillers</nccp:genre>
<nccp:genre>Comic Books and Superheroes</nccp:genre>
<nccp:genre>Blockbusters</nccp:genre>
</nccp:genres>
<nccp:availability>
<nccp:window>
<nccp:window_start>1283324400</nccp:window_start>
<nccp:window_end>1293436800</nccp:window_end>
</nccp:window>
</nccp:availability>
</nccp:moviedata>
<nccp:status>
<nccp:success>true</nccp:success>
</nccp:status>
</nccp:result>
<nccp:parameters>
<nccp:retrycontrol>10</nccp:retrycontrol>
<nccp:mintimeout>60</nccp:mintimeout>
<nccp:lowfrequencypollinterval>28800</nccp:lowfrequencypollinterval>
<nccp:mediumfrequencypollinterval>60</nccp:mediumfrequencypollinterval>
<nccp:highfrequencypollinterval>15</nccp:highfrequencypollinterval>
<nccp:hightomediumpollswitchinterval>300</nccp:hightomediumpollswitchinterval>
<nccp:mediumtolowpollswitchinterval>600</nccp:mediumtolowpollswitchinterval>
<nccp:loginterval>60</nccp:loginterval>
<nccp:maxlogsize>2000000</nccp:maxlogsize>
<nccp:loglevel>error</nccp:loglevel>
<nccp:supportphone>866-579-7113</nccp:supportphone>
</nccp:parameters>
</nccp:response>
