What is Living documentation

Serenity BDD is more than a library that generates test reports. That is, while you can use Serenity BDD to produce great test reports just fine, the real goal of Serenity BDD is to produce living documentation for your product.

But what do we mean by "living documentation"? Living Documentation is a concept that comes from the world of Behaviour Driven Development. It is closely related to the idea of Executable Specifications. Living documentation is, as the name suggests, both documentation, and living.

  • It is documentation, because it describes how an application works, and what business rules it applies, in a way that normal users can understand. Well written living documentation can be used by new team members to understand what a product does, and how it works. It can be handed over to a maintenance team when the application goes into production, or it can be used as evidence for auditors to show that the application respects the relevant rules and regulations.

  • It is living, because it is generated by the automated test suite, and is therefore by definition it is always up-to-date.

In a Serenity BDD report, you can find the Living Documentation in the Requirements tab.

requirements tab
Figure 1. A test report generated by Serenity

Living Documentation and Test Reporting

Living Documentation should not be confused with conventional test reporting. There are a number of important differences, and understanding these differences makes it easier to write good quality living documentation.

Timing

Conventional test reports are designed and generated fairly late in the development process, typically once features have been delivered and the automated tests written. Testers will often start preparatory work on test cases once the specifications or user stories are finalised, but test reports are still an activity that happens later rather than earlier in the development process.

For a team practicing BDD, on the other hand, the work on Living Documentation starts much earlier. Teams practicing BDD will work together with product owners to define acceptance criteria for the stories to be developed, and these acceptance criteria will form the basis for the automated acceptance tests and for the living documentation that is generated with each build.

Authors

Test Reports are the responsibility of QA; testers are usually the ones who design the report structure (if any design is done) and generate the test reports.

Living Documentation, on the other hand, is written collaboratively by many different actors: BAs, developers, testers and product owners all play key roles in the BDD requirements discovery process, and help, together, to express the business needs in a form that can be executed as automated acceptance tests.

Audience and Language

Testers are also the primary consumers of test reports. In many projects, other team members will only ever see test report summaries or overviews.

Living Documentation is for the whole team. It describes what the application should do before a feature is implemented (in the form of pending scenarios), and demonstrates that the implemented feature behaves as expected (in the form of passing scenarios). For this reason, living documentation scenarios need to be written in business terms, in a way that is easily understandable by non-testers.

Purpose and scope

Produced by testers and for testers, a test report naturally has a very strong testing focus. The emphasis is primarily on knowing whether a test passes or fails, and, at a higher level, knowing what proportion of tests pass or fail.

Living Documentation is more like a very detailed illustrated user manual. The focus is about describing what the application does, in business terms.

For example, a scenario in a test report might be interested in checking whether all of the countries appear in a country drop-down list on a user registration page. A scenario appearing in a living documentation report would be more interested in describing the user registration process as a whole, and demonstrating how it works for different types of users, and what business constraints or limitations apply to the choice of countries.

These differences can be summed up in the following table:

Test Reports Living Documentation

Produced after development is done

Written before development starts

Written by testers

Written collaboratively by the whole team

Written mostly for testers

For the whole team and beyond

Reports on passing and failing tests

Describes worked examples of functionality

Living Documentation and other types of tests

Living Documentation is designed to work together with lower-level unit and integration testing to provide a high degree of confidence in the quality of the application. Living documentation will generally focus on key examples of positive and negative paths through the application, and leave the more exhaustive testing to the unit test layers.

In highly regulated environments, on the other hand, living documentation will often be more exhaustive and detailed, as it may be used for audits or regulatory reporting.

Organising requirements for living documentation

In Serenity BDD, living documentation is organised in a requirements hierarchy. This hierarchy is usually implemented as a directory tree (for JBehave and Cucumber) or as a package structure (for JUnit).

Common Requirements Structures

The exact organisation varies from project to project, but some common structures include:

  • Capabilities > Features

  • Capabilities > Features > Stories

  • Goals > Capabilities > Features

  • Epics > Stories

  • Themes > Epics > Stories

A two-level requirements hierarchy works well for most projects. A small project (for example, a micro-service) might need only a short list of features. Only very large or complex projects would typically need three or more levels.

Goals, Capabilities and Features

The aim of any software project is to help our stakeholders achieve their goals by providing them with capabilities. In BDD, a Capability is something that allows users to do something that they couldn’t do previously, or to do something that they could do previously, but do it more efficiently. A capability is technology-agnostic: it does not commit to a particular solution or implementation. An example of a capability might be the ability to pay online with a credit card.

A Feature is a concrete solution or implementation that delivers a capability. Some possible features that deliver the capability we mentioned earlier might be to pay via PayPal, via Stripe or to pay by integrating with a merchant banking platform.

Themes and Epics

In Scrum, an Epic is simply a big user story, one that can’t be delivered in a sprint. A Theme is just another way to group related User Stories, though many teams use Themes as groups of related epics. (Another way to implement the original idea of themes in Serenity BDD is to use tags).

User Stories

User Stories are commonly used in Agile as a way to organise work, but they are not always very useful when it comes to Living Documentation. This is because they reflect how a feature was sliced That’s why Cucumber prefers to group scenarios in Feature Files (which describe a feature). JBehave still uses the older convention of "Story Files", which were meant to contain the acceptance criteria of a given story).

Configuring your requirements structure in Serenity BDD

You can configure the way Serenity names the different levels in your own requirements structure using the serenity.requirements.types property. For example, if you want to describe your requirements in terms of themes, epics and stories, you would add the following to your Serenity configuration file:

serenity.requirements.types=theme,epic,story

If you don’t configure this parameter, Serenity will decide on a sensible default hierarchy. This hierarchy depends on whether you are using JUnit, Cucumber or JBehave, and on the depth of your requirements hierarchy:

Test Framework Default Hierarchy

JUnit

capability > feature > story

Cucumber

theme > capability > feature

JBehave

capability > feature > story

Requirements configuration applies to the container levels, not to the feature or story files themselves. If you are using Cucumber, feature files will always be represented as features. If you are using JBehave, story files will always be represented as stories.

Requirements Hierarchies for JUnit Tests

Many teams write automated acceptance tests with Serenity BDD using JUnit. The Screenplay Pattern in particular makes it easy to write highly maintainable tests using a business-readable DSL that produces excellent living documentation.

JUnit acceptance tests should be organised in a package structure that reflects your requirements hierarchy. Note that this means that they may not reflect the package structure in your application, as is usually done for unit and integration tests.

A simple two-level hierarchy is illustrated here:

com
└── acme
    └── myapps
        └── specs
            ├── multiple_todo_lists
            ├── sharing_lists
            └── simple_todo_lists
                ├── AddingNewItems.java
                ├── DeletingItems.java
                ├── FilteringItemsByStatus.java
                └── MarkingItemsAsComplete.java

You need to tell Serenity where to find the requirements hierarchy in your package structure, using the serenity.test.root property. For the example shown above, the root package is com.acme.myapp.specs:

serenity.test.root=com.acme.myapp.specs

In this case, the Serenity living documentation will treat the JUnit test cases ("Adding New Items", "Deleting Items" etc.) as Stories, and the packages directly underneath the com.acme.myapp.specs package ("Multiple Todo Lists", "Sharing Lists" etc.) as Features.

Requirements Hierarchies for Cucumber

When you use Cucumber, Serenity expects your feature files to be stored in the src/test/resources/features directory. Your requirements hierarchy goes directly underneath this directory:

src
└── test
    └── resources
        └── features
            ├── multiple_todo_lists
            ├── sharing_lists
            └── simple_todo_lists
                ├── adding_new_items.feature
                ├── deleting_items.feature
                ├── filtering_items_by_status.feature
                └── marking_items_as_complete.feature

When Cucumber is used with the default configuration, the feature files always represent Features, and the directories containing the features represent Capabilities. A Feature is typically bigger than a user story, and can contain the acceptance criteria (scenarios) from several user stories.

Requirements Hierarchies for JBehave

When you use JBehave, Serenity expects your Story files to be stored in the src/test/resources/stories directory. Your requirements hierarchy goes directly underneath this directory:

src
└── test
    └── resources
        └── stories
            ├── multiple_todo_lists
            ├── sharing_lists
            └── simple_todo_lists
                ├── adding_new_items.story
                ├── deleting_items.feature
                ├── filtering_items_by_status.story
                └── marking_items_as_complete.story

By default the Story files represent stories, and the directories above them are rendered as Features.

Adding more details

You can add a special markdown file called overview.md at any level in the requirements hierarchy. This file will be rendered on the corresponding page in the Requirements report.

An overview.md file placed in the root requirements directory (src/test/resources/features or src/test/resources/stories) will be rendered on the Requirements home page in the report. Overview files can also be called narrative.md or narrative.txt (a convention used in older versions of Serenity).

Markdown (https://daringfireball.net/projects/markdown/) is a convenient lightweight format that you can use to make your living documentation more readable. An example of a simple Markdown overview file is shown here:

## A fantastic Todo application.

This application lets you do **great things**, like:

* Add new todo items
* Mark your items as complete
* Share your todo lists with your friends

If you place this overview.md file in the requirements root directory, the generated report will include this text on the main Requirements page:

overview example
Figure 2. A rendered application overview

Report assets

You can also include images in the overview.md files. By default, Serenity will copy any files in the src/test/resources/assets directory into the target/site/serenity/assets directory when it generates the reports.

You can place any images you want to include in your living documentation here, and then refer to them using the Markdown image syntax, like this:

The application architecture is as shown below:

![Application architecture ](assets/architecture.png)

This image would then appear in your reports, as illustrated here:

illustrated overview
Figure 3. A rendered application overview including an image

If you need to override the location of the assets directory, you can do so using the report.assets.directory property, as shown here:

report.assets.directory=src/test/resources/my-special-resources

Note that the target directory in the target/site/serenity directory will always be called assets.