Load testing HLS with Ruby JMeter

In this post we're going to demonstrate how to test Apple's HTTP Live Streaming (HLS) using Ruby JMeter.

RTMP

Real Time Messaging Protocol (RTMP) was initially a proprietary protocol for streaming audio, video and data over the Internet, between a Flash player and a server.

First let's take a look at what RTMP looks like in the browser. Not much to see here! We have an embedded Flash player and in terms of HTTP traffic, not much going on. The reason is because the Flash player itself is communicating via an alternate TCP port to the server which is streaming the content.

Let's take another look with a packet sniffer. Now we can see the packet headers and what looks like encoded binary content being streamed to the client. This is going to be difficult to simulate using tools like JMeter or any other generic load testing tool that typically favours HTTP.

The biggest drawback is that RTMP only works in Flash and not in HTML5. New HTTP streaming protocols, like Apple's HTTP Live Streaming (HLS), have wider device support (e.g. iOS) and will likely replace RTMP over the coming years.

HLS

HLS works by breaking the overall stream into a sequence of small HTTP-based file downloads, each download loading one short chunk of an overall potentially unbounded transport stream. We can see the following in the browser.

Notice the request made to playlist.m3u8. This effectively serves as a pointer to the other chunks that will need to be downloaded as part of a stream. We can see this as follows:

curl http://wowzaec2demo.streamlock.net/vod/_definst_/smil:streaming_tutorial/streaming_tutorial.smil/playlist.m3u8  
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=3128000,CODECS="avc1.77.31,mp4a.40.2",RESOLUTION=1280x720
chunklist_w1057647775_b3128000.m3u8  
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1778000,CODECS="avc1.77.30,mp4a.40.2",RESOLUTION=852x480
chunklist_w1057647775_b1778000.m3u8  
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1048000,CODECS="avc1.77.30,mp4a.40.2",RESOLUTION=640x360
chunklist_w1057647775_b1048000.m3u8  
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=738000,CODECS="avc1.77.21,mp4a.40.2",RESOLUTION=428x240
chunklist_w1057647775_b738000.m3u8  
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=528000,CODECS="avc1.77.13,mp4a.40.2",RESOLUTION=312x176
chunklist_w1057647775_b528000.m3u8  

The first chunk is chunklist_w1057647775_b3128000 and you can see each subsequent chunk after that. Each chunk will reveal the media URI to be downloaded by the client. We can see this as follows:

curl http://wowzaec2demo.streamlock.net/vod/_definst_/smil:streaming_tutorial/streaming_tutorial.smil/chunklist_w570392994_b3128000.m3u8  
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:6
#EXT-X-MEDIA-SEQUENCE:0
#EXTINF:6.0,
media_w570392994_b3128000_0.ts  
#EXTINF:6.0,
media_w570392994_b3128000_1.ts  
#EXTINF:6.0,
media_w570392994_b3128000_2.ts  
#EXTINF:6.0,
media_w570392994_b3128000_3.ts  
#EXTINF:6.0,

media_w570392994_b3128000_0.ts is the first media sequence of the first chunk. For example:

curl -I http://wowzaec2demo.streamlock.net/vod/_definst_/smil:streaming_tutorial/streaming_tutorial.smil/media_w570392994_b528000_0.ts  
HTTP/1.1 200 OK  
Date: Mon, 09 Jun 2014 21:51:20 GMT  
Content-Type: video/MP2T  
Accept-Ranges: bytes  
Server: FlashCom/3.5.7  
Cache-Control: no-cache  
Content-Length: 426008  

Ruby JMeter

The logic to this is straightforward.

Essentially our first request is to get the playlist. In the response we will extract the chunklist. We then make a request to each chunk. In the response of each we extract the streams. We then make a request to each media stream.

We can simulate this in Ruby JMeter as follows:

get name: 'playlist', url: "#{base_path}/playlist.m3u8" do  
  random_timer 1000, 3000

  extract regex: 'chunklist_(.+?)\.m3u8',
          match_num: -1,
          name: 'chunklist'
end

exists 'chunklist_matchNr' do  
  foreach_controller inputVal: 'chunklist', returnVal: 'chunk' do
    get name: 'chunk', url: "#{base_path}/chunklist_${chunk}.m3u8" do

      extract regex: 'media_(.+?)\.ts',
              match_num: -1,
              name: 'streams'
    end

    exists 'streams_matchNr' do
      foreach_controller inputVal: 'streams', returnVal: 'stream' do
        get name: 'stream', url: "#{base_path}/media_${stream}.ts"
      end
    end
  end
end  

Notice how we've made use of the match_num: -1 in order to extract all response body matches for each of the items we're interested in, chunklist and streams. We check that the relevant capture has more than one matchNr before making subsequent requests. We also use the foreach_controller of JMeter to loop through each of these results.

You can see the full Ruby JMeter script in detail here. If you don't want to use ruby-jmeter you can also download a copy of the same JMeter test plan here.

Running it on Flood IO

You can upload the JMeter test plan or use the ruby-jmeter gem to execute this demonstration on Flood IO for free. Here are some example results.

What type of results would we be looking for? Flat response times would be of importance, especially for the stream transaction as this is the transaction that is delivering media to the client.

We'd obviously be interested in any response time variation of this transaction as we increase the concurrency (and subsequent network throughput) against the system under test.

Obviously this sort of testing with full HLS media is likely to generate enormous bandwidth between the simulated clients and system under test. This is a perfect reason to consider solutions like Flood IO to generate this type of load.