What is Living documentation
Serenity BDD is more than a library that generates test reports. That is, while you can use Serenity BDD to produce great test reports just fine, the real goal of Serenity BDD is to produce living documentation for your product.
But what do we mean by "living documentation"? Living Documentation is a concept that comes from the world of Behaviour Driven Development. It is closely related to the idea of Executable Specifications. Living documentation is, as the name suggests, both documentation, and living.
It is documentation, because it describes how an application works, and what business rules it applies, in a way that normal users can understand. Well written living documentation can be used by new team members to understand what a product does, and how it works. It can be handed over to a maintenance team when the application goes into production, or it can be used as evidence for auditors to show that the application respects the relevant rules and regulations.
It is living, because it is generated by the automated test suite, and is therefore by definition it is always up-to-date.
In a Serenity BDD report, you can find the Living Documentation in the
Living Documentation should not be confused with conventional test reporting. There are a number of important differences, and understanding these differences makes it easier to write good quality living documentation.
Conventional test reports are designed and generated fairly late in the development process, typically once features have been delivered and the automated tests written. Testers will often start preparatory work on test cases once the specifications or user stories are finalised, but test reports are still an activity that happens later rather than earlier in the development process.
For a team practicing BDD, on the other hand, the work on Living Documentation starts much earlier. Teams practicing BDD will work together with product owners to define acceptance criteria for the stories to be developed, and these acceptance criteria will form the basis for the automated acceptance tests and for the living documentation that is generated with each build.
Test Reports are the responsibility of QA; testers are usually the ones who design the report structure (if any design is done) and generate the test reports.
Living Documentation, on the other hand, is written collaboratively by many different actors: BAs, developers, testers and product owners all play key roles in the BDD requirements discovery process, and help, together, to express the business needs in a form that can be executed as automated acceptance tests.
Testers are also the primary consumers of test reports. In many projects, other team members will only ever see test report summaries or overviews.
Living Documentation is for the whole team. It describes what the application should do before a feature is implemented (in the form of pending scenarios), and demonstrates that the implemented feature behaves as expected (in the form of passing scenarios). For this reason, living documentation scenarios need to be written in business terms, in a way that is easily understandable by non-testers.
Produced by testers and for testers, a test report naturally has a very strong testing focus. The emphasis is primarily on knowing whether a test passes or fails, and, at a higher level, knowing what proportion of tests pass or fail.
Living Documentation is more like a very detailed illustrated user manual. The focus is about describing what the application does, in business terms.
For example, a scenario in a test report might be interested in checking whether all of the countries appear in a country drop-down list on a user registration page. A scenario appearing in a living documentation report would be more interested in describing the user registration process as a whole, and demonstrating how it works for different types of users, and what business constraints or limitations apply to the choice of countries.
These differences can be summed up in the following table:
|Test Reports||Living Documentation|
Produced after development is done
Written before development starts
Written by testers
Written collaboratively by the whole team
Written mostly for testers
For the whole team and beyond
Reports on passing and failing tests
Describes worked examples of functionality
Living Documentation is designed to work together with lower-level unit and integration testing to provide a high degree of confidence in the quality of the application. Living documentation will generally focus on key examples of positive and negative paths through the application, and leave the more exhaustive testing to the unit test layers.
In highly regulated environments, on the other hand, living documentation will often be more exhaustive and detailed, as it may be used for audits or regulatory reporting.
In Serenity BDD, living documentation is organised in a requirements hierarchy. This hierarchy is usually implemented as a directory tree (for JBehave and Cucumber) or as a package structure (for JUnit).
The exact organisation varies from project to project, but some common structures include:
Capabilities > Features
Capabilities > Features > Stories
Goals > Capabilities > Features
Epics > Stories
Themes > Epics > Stories
A two-level requirements hierarchy works well for most projects. A small project (for example, a micro-service) might need only a short list of features. Only very large or complex projects would typically need three or more levels.
The aim of any software project is to help our stakeholders achieve their goals by providing them with capabilities. In BDD, a Capability is something that allows users to do something that they couldn’t do previously, or to do something that they could do previously, but do it more efficiently. A capability is technology-agnostic: it does not commit to a particular solution or implementation. An example of a capability might be the ability to pay online with a credit card.
A Feature is a concrete solution or implementation that delivers a capability. Some possible features that deliver the capability we mentioned earlier might be to pay via PayPal, via Stripe or to pay by integrating with a merchant banking platform.
In Scrum, an Epic is simply a big user story, one that can’t be delivered in a sprint. A Theme is just another way to group related User Stories, though many teams use Themes as groups of related epics. (Another way to implement the original idea of themes in Serenity BDD is to use tags).
User Stories are commonly used in Agile as a way to organise work, but they are not always very useful when it comes to Living Documentation. This is because they reflect how a feature was sliced That’s why Cucumber prefers to group scenarios in Feature Files (which describe a feature). JBehave still uses the older convention of "Story Files", which were meant to contain the acceptance criteria of a given story).
You can configure the way Serenity names the different levels in your own requirements structure using the
serenity.requirements.types property. For example, if you want to describe your requirements in terms of themes, epics and stories, you would add the following to your Serenity configuration file:
If you don’t configure this parameter, Serenity will decide on a sensible default hierarchy. This hierarchy depends on whether you are using JUnit, Cucumber or JBehave, and on the depth of your requirements hierarchy:
|Test Framework||Default Hierarchy|
capability > feature > story
theme > capability > feature
capability > feature > story
Requirements configuration applies to the container levels, not to the feature or story files themselves. If you are using Cucumber, feature files will always be represented as features. If you are using JBehave, story files will always be represented as stories.
Many teams write automated acceptance tests with Serenity BDD using JUnit. The Screenplay Pattern in particular makes it easy to write highly maintainable tests using a business-readable DSL that produces excellent living documentation.
JUnit acceptance tests should be organised in a package structure that reflects your requirements hierarchy. Note that this means that they may not reflect the package structure in your application, as is usually done for unit and integration tests.
A simple two-level hierarchy is illustrated here:
com └── acme └── myapps └── specs ├── multiple_todo_lists ├── sharing_lists └── simple_todo_lists ├── AddingNewItems.java ├── DeletingItems.java ├── FilteringItemsByStatus.java └── MarkingItemsAsComplete.java
You need to tell Serenity where to find the requirements hierarchy in your package structure, using the
serenity.test.root property. For the example shown above, the root package is
In this case, the Serenity living documentation will treat the JUnit test cases ("Adding New Items", "Deleting Items" etc.) as Stories, and the packages directly underneath the
com.acme.myapp.specs package ("Multiple Todo Lists", "Sharing Lists" etc.) as Features.
When you use Cucumber, Serenity expects your feature files to be stored in the
src/test/resources/features directory. Your requirements hierarchy goes directly underneath this directory:
src └── test └── resources └── features ├── multiple_todo_lists ├── sharing_lists └── simple_todo_lists ├── adding_new_items.feature ├── deleting_items.feature ├── filtering_items_by_status.feature └── marking_items_as_complete.feature
When Cucumber is used with the default configuration, the feature files always represent Features, and the directories containing the features represent Capabilities. A Feature is typically bigger than a user story, and can contain the acceptance criteria (scenarios) from several user stories.
When you use JBehave, Serenity expects your Story files to be stored in the
src/test/resources/stories directory. Your requirements hierarchy goes directly underneath this directory:
src └── test └── resources └── stories ├── multiple_todo_lists ├── sharing_lists └── simple_todo_lists ├── adding_new_items.story ├── deleting_items.feature ├── filtering_items_by_status.story └── marking_items_as_complete.story
By default the Story files represent stories, and the directories above them are rendered as Features.
You can add a special markdown file called
overview.md at any level in the requirements hierarchy. This file will be rendered on the corresponding page in the Requirements report.
overview.md file placed in the root requirements directory (
src/test/resources/stories) will be rendered on the Requirements home page in the report. Overview files can also be called
narrative.txt (a convention used in older versions of Serenity).
Markdown (https://daringfireball.net/projects/markdown/) is a convenient lightweight format that you can use to make your living documentation more readable. An example of a simple Markdown overview file is shown here:
## A fantastic Todo application. This application lets you do **great things**, like: * Add new todo items * Mark your items as complete * Share your todo lists with your friends
If you place this
overview.md file in the requirements root directory, the generated report will include this text on the main Requirements page:
You can also include images in the
overview.md files. By default, Serenity will copy any files in the
src/test/resources/assets directory into the
target/site/serenity/assets directory when it generates the reports.
You can place any images you want to include in your living documentation here, and then refer to them using the Markdown image syntax, like this:
The application architecture is as shown below: ![Application architecture ](assets/architecture.png)
This image would then appear in your reports, as illustrated here:
If you need to override the location of the
assets directory, you can do so using the
report.assets.directory property, as shown here:
Note that the target directory in the
target/site/serenity directory will always be called