Load testing with Selenium

Selenium Flood Tests

Selenium helps you test web applications by simulating how a real user would interact with your application using real browsers. The WebDriver API has language bindings and implementations to help drive different browsers.

Combine Selenium with WebDriver on Flood IO and you get distributed load testing across the globe.

Chrome and Firefox

On Flood IO you can launch browsers using ChromeDriver or FirefoxDriver that run in isolated Docker containers.

Since these containers run full browsers, CPU overheads are naturally higher so we limit the amount of browsers to 5 per grid node.

The benefit of running full browser automation for load testing comes from driving the browser itself. Being able to run existing browser based automation suites as a control sample in conjunction with more traditional load testing tools and techniques is valuable for the increased fidelity of the simulation. However there's nothing stopping you from running larger grids with multiple nodes to increase the level of concurrency if you want to run Selenium only load tests.

Java Bindings

We currently support native Java bindings for WebDriver so you can re-use existing tests already written in Java. Depending on demand, we may support additional bindings / test runners in other languages in future. At present we'll stick with Plain Old Java Objects. Let us know..

Example Test

For your tests to work on Flood IO you'll need to import our FloodAgent. This is a custom plugin that lets us pull together reporting information for your flood tests. The critical functions your script needs to call are .started() and .finished() for test boundaries. Inside that you can make calls to .passed_transaction(WebDriver driver, String label, Integer responseCode) or .failed_transaction(WebDriver driver, String label, Integer responseCode).

When your test is running we'll pass in the relevant WEBDRIVER_HOST and WEBDRIVER_PORT environment variables for your test to connect to.

Have a look at the following example test plan.

import java.net.URL;

import org.openqa.selenium.By;  
import org.openqa.selenium.WebDriver;  
import org.openqa.selenium.WebElement;

import org.openqa.selenium.JavascriptExecutor;  
import org.openqa.selenium.WebDriverException;

import org.openqa.selenium.remote.Augmenter;  
import org.openqa.selenium.remote.DesiredCapabilities;  
import org.openqa.selenium.remote.RemoteWebDriver;

import org.openqa.selenium.support.ui.Select;

import io.flood.selenium.FloodAgent;

public class Loadtest  {  
  public static void main(String[] args) throws Exception {
    int iterations = 0;

    // Create a new instance of the html unit driver
    // Notice that the remainder of the code relies on the interface,
    // not the implementation.
    WebDriver driver = new RemoteWebDriver(new URL("http://" + System.getenv("WEBDRIVER_HOST") + ":" + System.getenv("WEBDRIVER_PORT") + "/wd/hub"), DesiredCapabilities.chrome());
    JavascriptExecutor js = (JavascriptExecutor)driver;

    // Create a new instance of the Flood IO agent
    FloodAgent flood = new FloodAgent();

    // Inform Flood IO the test has started
    flood.started();

    // It's up to you to control test duration / iterations programatically.
    while( iterations < 1000 ) {
      try {
        // And now use this to visit the target site
        driver.get("https://loadtest.flood.io/usertiming");

        // Log a passed transaction in Flood IO
        flood.passed_transaction(driver);

        // Log a custom mark from the User Timing API
        flood.get_mark(driver, "mark_headers_loaded");

        // Log a custom measure from the User Timing API
        flood.get_measure(driver, "measure_page_load");

        iterations++;

        // Good idea to introduce some form of pacing / think time into your scripts
        Thread.sleep(3000);
      } catch(InterruptedException e) {
        Thread.currentThread().interrupt();
        String[] lines = e.getMessage().split("\\r?\\n");
        System.err.println("Browser terminated early: " + lines[0]);
      } catch (WebDriverException e) {
        String[] lines = e.getMessage().split("\\r?\\n");
        System.err.println(lines[0]);
      }
    }

    driver.quit();

    // Inform Flood IO the test has finished
    flood.finished();
  }
}

How it works

We use Docker containers that run standalone Selenium Node images provided by the Selenium project for Chrome and Firefox. Inside those our Java plugin makes calls to the same internal reporting API using Elastic that all of our load test runners use. The plugin itself is using the JavascriptExecutor within WebDriver to parse the Navigation Timing API, something along the lines of:

JavascriptExecutor js = (JavascriptExecutor)driver;

long loadEventEnd = (Long)js.executeScript("return window.performance.timing.loadEventEnd;");  
long navigationStart = (Long)js.executeScript("return window.performance.timing.navigationStart;");  
long responseStart = (Long)js.executeScript("return window.performance.timing.responseStart;");  
long connectStart = (Long)js.executeScript("return window.performance.timing.connectStart;");  
...

Beyond that, results are formatted in the same fashion as all of our load test reports. Here's an example. Importantly we're not relying on any proxies in between your browser and the target site so as to minimise impact on your test results as much as possible.

We're always interested in your feedback and requests for different ways to generate load.