Running Serenity tests in parallel

One of the most common ways to run UI test suites faster is to run them in parallel. There are ways you can make Serenity tests run in parallel, and in this section we will walk through a few of them.

Parallel execution with the Maven Failsafe Plugin

If you are using Maven, the easiest way to run tests in parallel is to use the maven-failsafe-plugin to run your tests. This feature is well documented but can be complicated to set up. However, when running Serenity tests, a few simple variations generally work well.

Running tests in parallel threads.

One of the simplest ways to run your tests in parallel is to run them in parallel threads. For example, the following (partial) configuration will run your test classes across two threads per CPU core:

<plugin>
    <artifactId>maven-failsafe-plugin</artifactId>
    <version>3.0.0-M3</version>
    <configuration>
        <!-- Parallel execution in the same JVM -->
        <parallel>classes</parallel>
        <threadCount>2</threadCount>
    </configuration>

The parallel configuration element takes several options, including methods, classes, both, suites, suitesAndClasses, suitesAndMethods, classesAndMethods or all. However in practice, the options that work best with Serenity are classes, suites and suitesAndClasses.

How many threads can I use?

Selenium’s ability to scale on a single machine is not unlimited. In practice, the more threads you use, the higher the chances of sporadic, hard-to-reproduce test failures due to timeouts and other related issues. The added performance gain of each thread also tends to drop off for higher numbers of threads. The Maven documentation has this to say: "the concurrency happens within the same JVM process. That is efficient in terms of memory and execution time, but you may be more vulnerable towards race conditions or other unexpected and hard to reproduce behavior".

Using forks

Another way to run your tests in parallel in Maven is to use Forked Test Execution. Using Forked Test Execution, new JVM processes are spun up to execute your tests, up to a configurable maximum number of processes. This creates better separation between your tests, which can improve their reliability. However the forked JVMs take longer to set up, so there is a minor performance hit.

You tell Maven to spawn new processes by adding the forkCount configuration element, as shown here below. This can either be a number (the maximum number of forks) or a multiplier (the number of forks per CPU). The example below uses the value of "2C", which means 2 forked processes per CPU:

<plugin>
    <artifactId>maven-failsafe-plugin</artifactId>
    <version>3.0.0-M3</version>
    <configuration>
        <!-- Parallel execution in the same JVM -->
        <parallel>classes</parallel>
        <threadCount>2</threadCount>
        <forkCount>2C</forkCount>
    </configuration>

Using multiple forks can be a good alternative to running all tests in a single JVM, and can reduce the risk of certain types of errors. The exact configuration that will work best for your environment will vary, so it is best to experiment with different options to see what gives the best results.

Flaky parallel tests

Parallel tests in Selenium are more subject to sporadic errors and failures, due to timeouts and resource contention.