<

E2E Testing with Cucumber and Screenplay Design

Cucumber
Screenplay
Test

Automated testing is essential in web application development. Especially, the importance of E2E testing before release is high. This time, I will introduce my experience of incorporating the famous Cucumber and Screenplay design in BDD.

Test Design Approach

First, I think there are multiple approaches to E2E test design.

  • User Story-Based Testing
    • Used in agile development, tests are designed based on requirement descriptions that reflect the user's perspective.
  • Business Requirement-Based Testing
    • Design tests that directly reflect business purposes and requirements, focusing on business value and goals.
  • Scenario Testing
    • Evaluate system behavior using test cases that mimic real-world business processes and user scenarios.

I have often played the role of a developer in agile development in the web industry. When collaborating with product managers and other non-developer business side members, we often decide on the first user story and design the test. Therefore, this time, I will assume the first approach.

Next, at what stage do you create test cases from user stories? With the shift-left approach, doing it at a stage earlier than the implementation stage can reduce risks such as repair costs and release delays. Therefore, I will take the following approach.

  • Acceptance Test-Driven Development (ATDD)
    • An approach to proceed with development after creating test cases that meet the acceptance criteria defined before development.

We create the user story we want to achieve before development and the test cases that satisfy it. This work is done in collaboration with business side members. The format of the test case will adopt the Gherkin format, which is expressed in Given-When-Then.

  • Behavior-Driven Development (BDD)
    • Define system behavior using "Given-When-Then" behavior scenarios and create test cases based on it.

As a testing tool, we use Cucumber, which can read Gherkin and execute tests.

https://cucumber.io/

Example Gherkin Scenario

For example, consider the following scenario for a user story about finding a product in an online store.

# online-store.feature

Feature: Online Store

  Scenario: customer finds product by name

# - Apisitt, responsible for setting up test data using the REST API
# - Wendy, representing a customer interacting with the web UI

    Given Apisitt sets up product catalogue with:
      | name    | price |
      | Apples  | £2.50 |
    When Wendy looks for 'Apples'
    Then she should see top search result of:
      | name  | Apples |
      | price | £2.50  |

※ Quoted from Screenplay Pattern - serenity-js.org

This scenario is written in Gherkin format. The file name is *.feature. It can also be described in Markdown.

https://github.com/cucumber/gherkin/blob/main/MARKDOWN_WITH_GHERKIN.md

Furthermore, you can write scenarios in Japanese.

https://cucumber.io/docs/gherkin/languages/

For other ways of writing Gherkin and practices, please refer to the following.

https://cucumber.io/docs/gherkin/

Next, let's write a test that satisfies the scenario. The following is the test code that satisfies the scenario. The Given, When, and Then of the previous scenario correspond to the following test code. (You can ignore Actor)

// online-store.steps.ts

import { Given, When, Then, DataTable } from "@cucumber/cucumber";
import { Actor } from "@serenity-js/core";

Given(
  "{actor} sets up product catalogue with:",
  (actor: Actor, products: DataTable) => actor.attemptsTo()
);

When("{actor} looks for {string}", (actor: Actor, productName: string) =>
  actor.attemptsTo()
);

Then(
  "{pronoun} should see top search result of",
  (actor: Actor, expectedResult: string) => actor.attemptsTo()
);

To run the test, you can run it with the following command.

npx cucumber-js

If this test is successful, you can see that the function of online-store.feature is guaranteed. In other words, it becomes acceptable and can be released.

Test Implementation Methods

Now, let's implement the tests. There are several ways to implement tests, such as:

  • Page Object Model
    • A design pattern that models each page of a web application as an object, improving the maintenance and reusability of UI tests.
  • Keyword Driven Testing
    • A method of writing test scripts using keywords (actions or operations), making the tests easy to understand and maintain even for non-technical people.
  • Screenplay
    • A design pattern that models test scenarios based on "actors" and "tasks", improving the readability and flexibility of tests.

I thought Screenplay would be a good fit for tests based on user stories, so I tried it this time.

Library Selection

When trying out the Screenplay design, the following were the candidates:

I personally wanted to use Playwright for E2E testing, so I excluded cucumber/screenplay.js. Also, while Tallyb/cucumber-playwright could have been used just to incorporate the Screenplay design method, I excluded it because of the following issues:

  • Need to implement the five elements of Screenplay (explained later) by yourself.
  • Need to manage Actors for each scenario.

Therefore, I adopted serenity-js/serenity-js, which has implemented the elements necessary for Screenplay design.

What is Screenplay

Screenplay, summarized from Screenplay Pattern - serenity-js.org, is as follows:

The Screenplay Pattern is a user-centric approach to writing high-quality automated acceptance tests, effectively using layers of abstraction and incorporating business terminology into test scenarios. This pattern focuses on actors and their goals, promoting cooperation and understanding between engineers and business stakeholders by using domain language.

What I liked about the Screenplay Pattern are the following points:

  • User-centric approach
    • The ability to design tests around the user, represented as an Actor
  • Promotes cooperation and understanding between engineers and business stakeholders
    • The ability to create feature files in collaboration with business side members
  • Has layers of abstraction
    • The ability to improve test maintainability by reusing tasks (explained later)

The Five Elements in Screenplay

Screenplay has the following five elements:

Five elements of the Screenplay Pattern - https://serenity-js.org
Five elements of the Screenplay Pattern - https://serenity-js.org

I will introduce the five elements:

  • Actor
    • Represents people or external systems that interact with the system under test.
    • Examples
      • User
      • API
  • Ability
    • Used to handle integration libraries necessary for interaction with the system under test in a simplified manner.
    • Examples
      • Ability to access a web page
        • A wrapper for a library to operate a browser (such as Playwright)
      • Ability to send API requests
        • A wrapper for a library to send API requests (such as axios)
  • Interaction
    • Represents low-level activities that an actor can perform using a specific interface.
    • Examples
      • Logging in
        • Entering a username and password in the login form and clicking the login button
      • Adding a product to the cart
        • Accessing the product page and adding the product to the cart
  • Task
    • Used to model meaningful steps in business workflows within the domain.
    • Examples
      • Buying a product online
        • Logging in
        • Adding a product to the cart
        • Purchasing
  • Question
    • Used to obtain information from the system under test or the test execution environment.
    • Examples
      • What is the current account balance?
        • Getting the user's account balance

Also, in serenity-js, there is an element called Note that allows an Actor to remember information.

https://serenity-js.org/api/core/class/TakeNotes/

Here is an example of implementing five elements in the previous online-store.steps.ts.

// online-store.steps.ts

import { Given, When, Then, DataTable } from "@cucumber/cucumber";
import { Actor, Task } from "@serenity-js/core";
import { CallAnApi, PostRequest, Send, LastResponse } from "@serenity-js/rest";
import { BrowseTheWebWithPlaywright } from "@serenity-js/playwright";
import { Navigate, Page } from "@serenity-js/web";
import { Ensure, equals, endsWith } from "@serenity-js/assertions";

Given(
  "{actor} sets up product catalogue with:",
  (actor: Actor, products: DataTable) =>
    actor.attemptsTo(setupProductCatalogue(products.hashes()))
);

When("{actor} looks for {string}", (actor: Actor, productName: string) =>
  actor.attemptsTo(openOnlineStore(), findProductCalled(productName))
);

Then(
  "{pronoun} should see top search result of",
  (actor: Actor, expectedResult: string) =>
    actor.attemptsTo(
      // Question
      Ensure.that(
        topSearchResult().name,
        equals(expectedResult.rowsHash().name)
      ),
      // Question
      Ensure.that(
        topSearchResult().price,
        equals(expectedResult.rowsHash().price)
      )
    )
);

// Task
const setupProductCatalogue = (products: Product[]) =>
  Task.where(
    `#actor sets up the product catalogue`,
    // Interaction
    Send.a(PostRequest.to("/products").with(products)),
    // Question
    Ensure.that(LastResponse.status(), equals(201))
  );

// Task
const openOnlineStore = () =>
  Task.where(
    `#actor opens the online store`,
    // Interaction
    Navigate.to("https://example.org"),
    // Question
    Ensure.that(Page.current().title(), endsWith("My Example Shop"))
  );

// Task
const findProductCalled = () =>
  Task.where(`#actor looks for a product`, undefined); // コード例がなかったため、省略

Also, {actor} and {pronoun} can be defined as follows.

// parameter.steps.ts
import { defineParameterType } from "@cucumber/cucumber";
import { actorCalled, actorInTheSpotlight } from "@serenity-js/core";
import { CallAnApi } from "@serenity-js/rest";
import { BrowseTheWebWithPlaywright } from "@serenity-js/playwright";

defineParameterType({
  regexp: /[A-Z][a-z]+/,
  transformer(name: string) {
    if (name === "Apisitt") {
      return actorCalled(name).whoCan(
        // Ability
        CallAnApi.at("https://api.example.org")
      );
    }
    if (name === "Wendy") {
      return actorCalled(name).whoCan(
        // Ability
        BrowseTheWebWithPlaywright.using(browser)
      );
    }
  },
  name: "actor",
});

defineParameterType({
  regexp: /he|she|they|his|her|their/,
  transformer() {
    return actorInTheSpotlight();
  },
  name: "pronoun",
});

※ parameter.steps.ts - serenity-js/serenity-js-cucumber-playwright-template

That's all for the introduction to Screenplay.

Scenario Antipatterns

Since we're here, let's also introduce some scenario antipatterns.

https://www.thinkcode.se/blog/2016/06/22/cucumber-antipatterns

In summary, the following antipatterns exist:

  • Writing feature files after coding
    • Writing Gherkin feature files after software implementation. It's not for development promotion, but just for record.
  • Scenarios created solely by business stakeholders
    • When product owners or business analysts create scenarios on their own, they may not reflect actual business needs or test feasibility.
  • Scenarios created by developers or testers without consultation with business stakeholders
    • When developers or testers create scenarios on their own, they tend to be unrealistic or contain unrealistic data and user descriptions.
  • Scenarios that are too high-level
    • High-level, vague scenarios do not reflect specific business rules and are not reliable.
  • Dead documentation
    • Insufficient Gherkin does not function as a document to accurately convey the system's functionality.
  • Misunderstandings due to unnecessary details
    • When scenarios contain unnecessary details, the essence of the business rule to be tested becomes ambiguous.
  • Inappropriate scenario names
    • The name of a scenario should succinctly indicate its content, but inappropriate names hinder understanding of the content.
  • Beginner's mistakes
    • Mistakes that beginners are prone to make, such as focusing too much on UI details or using the personal pronoun "I".
  • Unclear distinction of Given/When/Then
    • When the distinction between Given, When, and Then is unclear, the intent of the scenario becomes unclear.

In Conclusion

This time, I introduced my experience of incorporating Screenplay design. Please feel free to use it as a reference.

If it was helpful, support me with a ☕!

Share

Related tags

Cucumber
Test