Reporting Manual Test Results

One of the principles of BDD is to have a single source of truth for both the requirements that describe a feature, and the automated tests that verify them. And it is a natural step to want to include both automated and manual tests in this single source of truth.

Cucumber is primarily and traditionally used for automating executable specifications. But with Serenity BDD, you can add special tags to indicate that a scenario represents a manual test case.

Recording manual test cases

You can flag any Cucumber scenario as manual simply by using the @manual tag:

@manual
Scenario: Invoice details should be downloadable as a PDF file
  The layout and content of the PDF file should be verified

  Given Clive has made a purchase
  When he reviews his past orders
  Then he should be able to download his invoice as an PDF file

This scenario will now appear as a manual test case in the Serenity reports:

manual scenario
Figure 1. A manual scenario

Recording manual test results

By default, @manual scenarios are marked as pending in the Serenity reports. However, you can indicate a different result by adding the @manual-result tag as shown here:

  • A passing test: @manual-result:passed

  • A failing test: @manual-result:failed

  • A compromised test: @manual-result:compromised

Note that if you want to record the result of a manual test, you should include both the @manual and the @manual-result tags. This also makes it easier to run all of the manual tests in isolation.

@manual
@manual-result:passed
Scenario: Invoice details should be downloadable as a PDF file
   ...

These manual test results are indicated in a different shade in the Serenity reports, and also appear in the summary table on the home page:

dashboard with manual results
Figure 2. The Serenity dashboard showing some manual test results

Updating manual test results

Manual test results are generally only valid for a given version of an application - when a new version is built, the manual tests may need to be redone.

Different projects deal with manual testing in different ways. For example some teams refer to a target release version, and test against this version when a new feature or story is ready to test. They do not, for example, feel the need to redo every manual test for each commit. They assess on a case-by-case basis whether any given changes might impact the features that have already been tested.

For example, suppose that a team is working towards a release at the end of the 15th sprint of their project.

@manual
@manual-result:passed
@manual-last-tested:sprint-15
Scenario: Invoice details should be downloadable as a PDF file
   ...

In the Serenity properties or configuration file, the team also records the current version (or sprint number):

current.target.version = sprint-15

When the manual tests are processed, this scenario will be reported as passing, because the current.target.version matches the @last-tested tag in the scenario.

However, in the next sprint, the target version is updated:

current.target.version = sprint-16

Now, when the manual scenario is processed, it will be marked as pending, with a note indicating that a new manual test is required:

manual test pending retest
Figure 3. An out-of-date manual test

Attaching evidence

Sometimes we need to attach screenshots or other files to our manual test reports as additional evidence. The @manual-test-evidence tag allows you to do just this. You can either include a link to an external site, as shown here:

@manual
@manual-result:passed
@manual-last-tested:sprint-15
@manual-test-evidence:https://some.external/link.png
Scenario: Invoice details should be downloadable as a PDF file
   ...

Alternatively, you can place an image in the src/test/resources/assets folder and include a relative link to this file (starting with "assets/"):

@manual
@manual-result:passed
@manual-last-tested:sprint-15
@manual-test-evidence:assets/some-screenshot.png
Scenario: Invoice details should be downloadable as a PDF file
   ...

You can provide more than one piece of evidence by providing a comma-separated list:

@manual
@manual-result:passed
@manual-last-tested:sprint-15
@manual-test-evidence:assets/first-screenshot.png,assets/second-screenshot.png
Scenario: Invoice details should be downloadable as a PDF file
   ...

You can also use a markdown-like syntax to customise the text that will describe each piece of evidence:

@manual
@manual-result:passed
@manual-last-tested:sprint-15
@manual-test-evidence:[First_Screenshot](assets/first-screenshot.png)
Scenario: Invoice details should be downloadable as a PDF file
   ...

Since Cucumber does not allow spaces in tags, you can use underscores for spaced.

Test Evidence is only displayed if the @manual-last-tested annotation is defined.

Updating manual test results in an existing test suite

Manual test results are not always updated at the same time as code changes. You can update an existing set of test results by selectively running the manual scenarios.

 mvn verify -Dcucumber.options="--tags @manual"