Dogfooding your performance test platform

When it comes to self assessment, Captain Riker sums it up:

"The game isn't big enough unless it scares you a little."
-- William Riker

At Flood IO we take performance seriously and are always looking to improve. We find an increasing number of customers come to us for advice and not just execution of their test plans. A lot of that advice centres around how to improve first visit (empty cache) performance. Luckily you don't need an expensive tool to figure this out. Tools such as Google Page Speed and Yahoo YSlow offer simple, straight forward advice on how to optimise web page performance.

There's been a lot of media attention on this lately, just follow the #webperf hash tag on Twitter for example. If you're not already familiar with the tools, go check them out. They're easy to install and you can test your own site with a simple browser extension.

Which brings me back to the topic of this post. With all the SaaS performance test vendors out there and barrage of advice being offered, are they dogfooding their own product?

We took a look at 12 vendors that advertise in this space including ourselves to see how we stand up.

The Sites

We chose from the following SaaS performance test sites to run the comparison tests:

Flood, Loadstorm, Apicasystem, Loader, Telerik, Neotys, Neustar, Loadimpact, Blazemeter, Blitz, Soasta and Smartbear.

The Comparison

We simulated a first visit to the home page of each product with an empty browser cache, using the free service of GTMetrix. This site runs reports from Vancouver Canada using Firefox (Desktop) 25.0.1, Page Speed 1.12.16 and YSlow 3.1.7. Tests were automated using the GTMetrix API and a simple bash script we cooked up for this purpose. This makes the test repeatable and can be expanded to include other sites if interested.

The Results

For each dimension of the test we rank sites from best to worst in terms of their individual score. You can view the summary results here which includes links to the individual GTMetrix formatted reports.

Page Load Times (ms)

This is a measure of page load time in milliseconds and is perhaps the most contentious dimension measured. Typical caveats to this score will be things like repeatability (hitting cold or warm servers), geographic distance from test to origin, TCP connection overheads, SSL negotiation and so on. In any case, Loadstorm was ranked number one here, with the majority under 3 seconds.

1639 http://loadstorm.com  
2104 http://apicasystem.com  
2215 http://loader.io  
2585 http://www.telerik.com  
2688 http://loadimpact.com  
2816 https://flood.io  
2876 http://www.neotys.com  
3019 http://neustar.com  
3153 http://soasta.com  
3472 http://blitz.io  
3683 http://smartbear.com  
3848 http://blazemeter.com  

Page Requests

AKA the how bad does my site suck on a high latency mobile network score or more commonly the I don't know how to combine files and minimise requests score.

80% of the end-user response time is spent on the front-end. Most of this time is tied up in downloading all the components in the page: images, stylesheets, scripts, Flash, etc. Reducing the number of components in turn reduces the number of HTTP requests required to render the page. This is the key to faster pages.

It's an area in which Flood IO is quite aggressive in terms of optimizations. In fact a second visit to our home page with a primed cache is just 2 requests. We take care to combine assets into single files, use CSS image sprites where possible and generally reduce the amount of HTTP requests required. Very important for high latency links like mobile users and often overlooked by clients including our competitors.

12 https://flood.io  
30 http://loadstorm.com  
38 http://www.telerik.com  
40 http://loadimpact.com  
41 http://apicasystem.com  
45 http://www.neotys.com  
46 http://loader.io  
55 http://blitz.io  
58 http://neustar.com  
75 http://smartbear.com  
87 http://soasta.com  
110 http://blazemeter.com  

Page Weight (B)

AKA the but I thought everyone was on 100 Gigabit Ethernet score.

Put simply, what is the total page weight in Bytes for all content on your page for a first visit with an empty cache.

Apparently web page size has grown +151% in the last 3 years with the average page size now 1575 KB. Flood IO is in the middle of that range with 719 KB and some of the worst offenders creeping above the average. It makes you wonder, just how good do those images need to be as part of your market-eering?

261625 http://loadstorm.com  
544241 http://loadimpact.com  
545797 http://apicasystem.com  
594202 http://www.neotys.com  
597154 http://www.telerik.com  
654106 http://loader.io  
690677 http://neustar.com  
719168 https://flood.io  
922117 http://blitz.io  
1094106 http://blazemeter.com  
1574796 http://smartbear.com  
1857024 http://soasta.com  

Google Page Speed

AKA the how does Google page speed rank my site score.

Google's Page Speed best practices cover many of the steps involved in page load time, including resolving DNS names, setting up TCP connections, transmitting HTTP requests, downloading resources, fetching resources from cache, parsing and executing scripts, and rendering objects on the page.

It's a useful comparison between sites because it's effectively a static analysis against a common set of rules. The higher the score the better. Flood IO came equal first with Loadimpact and the majority of sites score well in the +90% range. Some sites can definitely improve though.

96 https://flood.io  
96 http://loadimpact.com  
93 http://www.telerik.com  
93 http://loadstorm.com  
92 http://soasta.com  
92 http://smartbear.com  
92 http://blazemeter.com  
89 http://loader.io  
87 http://apicasystem.com  
84 http://www.neotys.com  
68 http://neustar.com  
52 http://blitz.io  

Note: this is slightly different to the Google PageSpeed Insights score not listed here, but this also gives you the ability to view your score from both mobile and desktop devices. It's worth trying for yourself. Flood IO scores an impressive 98/100 for both platforms.

Yahoo YSlow

AKA the I don't believe Google, what does Yahoo think score.

We've long been a fan of this browser extension. Some dimensions of the score you might need to take with a grain of salt, and it is generally a little more harsh but once again, as a static analysis for comparison, it's a great measure of performance. Looks like Flood IO comes out on top again, but no surprise since we focus a lot of our attention in improving these scores.

96 https://flood.io  
93 http://apicasystem.com  
90 http://loader.io  
89 http://loadstorm.com  
81 http://www.neotys.com  
81 http://loadimpact.com  
75 http://blazemeter.com  
72 http://www.telerik.com  
70 http://soasta.com  
68 http://neustar.com  
67 http://smartbear.com  
63 http://blitz.io  

Steps to Improve

So what have we been doing to improve our own score lately? We like to use another free service provided by WebPageTest which lets us test these different performance dimensions in our own time and from different locations around the world. Example results are shown here

We've since:

  1. ensured our resources are compressed and served via a CDN where possible.
  2. minified our resources to the extent we still maintain our core functionality, look and feel that we're happy with.
  3. taken steps to improve server response time by removing or tuning back end system calls through application profiling.
  4. leveraged browser caching as much as possible including correct use of expires and cache-control headers.
  5. optimised images as much as possible including use of CSS image sprites to further reduce number of requests to the origin servers.

The waterfall chart for our first visit performance is quite telling and gives us ideas for further optimizations, although how much is enough?!

First impressions count, and we hope this post demonstrates that Flood IO is indeed eating its own dog food.