8 Common Reasons BDD Automation Fails

Software test automation is hard. A staggering 64% of the time, automated testing fails to deliver according to expectations (Reference https://www.cigniti.com/blog/4-reasons-why-test-automation-fails-2/). In other words, only three out of every ten projects successfully achieve test automation.

Moreover, behavior-driven development (BDD) automation is not immune to this. Many of the challenges you’ll encounter with any software test automation apply to BDD automation also. This article discusses the most common issues with BDD automation and how you can avoid them.

Problem

BDD is not testing a testing framework or tool. I have covered why BDD is not about testing elsewhere on my blog. When BDD is used as a testing tool, the Engineering teams might lose the advantages of collaborating and deriving business-critical scenarios.

How does this impact automation testing? When testers work in silos, it is only their understanding and perception of the requirements that get taken into account. The scenarios are not derived by collaborating with other members of the team, mainly the Product Owner and the Developer. As a result, the tester might end up automating scenarios that don’t cover all the business requirements. Thus, the whole team won’t see the benefits of test automation because the application won’t be testing all scenarios.

Solution

Understand what BDD is and how to roll it out in a project. Using techniques like OOPSI, example mapping, or functional mapping and collaboratively deriving scenarios really help flesh out all the required scenarios. Nailing this down is critical for any BDD automation. Getting this step wrong — automating without understanding or covering all business requirements — guarantees to make your test automation fail, as it doesn’t test all business requirements.

Problem

Not thinking about a test architecture before starting any test automation will come back to haunt you. Instead, treat test automation just like any development effort. Many software engineers embark on the test automation journey without thinking about the architecture or the design. Unfortunately, I have seen many teams not give this the same importance as development work, thinking that the BDD automation code will never make it to live, so they don’t have to treat it as a production code. This is a false approach. Your BDD automation code lives as long as your production code exists.

Solution

Before the start of BDD automation, defining a solid test architecture, design principles, coding standards, and guidelines helps immensely. One should follow the same guidelines as you would for any development work. So, define a solid peer review process, static code analysis, and all the other things that you would follow as part of your definition of done — but obviously tweak it a bit for test automation.

Problem

Automating BDD Scenarios for UI (web or mobile) doesn’t scale and is difficult to manage. Although this is a well-documented issue, when the application uses either web or mobile, most teams end up automating the BDD scenarios via the UI elements.

When BDD automation — or any automation, using BDD or not — is done at the UI layer, it could suffer from several, if not all, the following issues:

• Brittle 
• Flaky 
• Slow 
• Not scalable 
• Doesn’t test the behavior of your application 
• Paves the way to imperative BDD steps, which are similar to manual test scripts and don’t bode well for automation

Solution

Avoid automating BDD scenarios that depend completely on the UI layer. Instead, find a middle ground with a combination of UI and API. I wrote a blog, “Is BDD Good for UI Automation”, about this topic specifically and how to address it.

Problem

Fast feedback is extremely important so that the teams react quickly and fix issues and improve time to market. Imagine a test running for 5 hours while the team is waiting for the result either to release or to fix any issues. Five hours might sound extreme, but I have seen many projects where the test suites run for long hours.

Solution

There are several things you can consider to make your test suite conducive to fast feedback. Let’s look at some of the options that you should consider:

• Parallel run
• Fail fast
• Better Application design (Moving away from Monolith to Micro Services)
• Real-time error reporting (show errors as soon as they occur, and not generate error report after the entire test completes)

Problem

As the project grows, some parts of your API data would be in constant flux. As a result, for every change, you have to keep updating all the steps in which the API is called. The API changes may be completely irrelevant to your project, which might not affect your application behavior, but you still have to change your BDD steps since the API has changed.

Consider the following step:
When a User registers on the website with a username as “jerome@nocodebdd.com”.

For that step, when you define Step Definition (your automation code), let’s say you have to call the API and pass the following test data:

{

“employee”: {

“@id”: “be129”,

“firstname”: “Jane”,

“lastname”: “Doe”,

“title”: “Engineer”,

“division”: “Materials”,

“building”: “327”,

“room”: “19”,

“supervisor”: “be131”

}

}

Let’s assume that the above data is hard coded or is in a file, which you read and send it as part of the API.

Now, assume the same API is used in a different step, which passes slightly different information as shown below:

{

“employee”: {

“@id”: “be129”,

“firstname”: “Jane”,

“lastname”: “Doe”,

“title”: “Engineer”,

“division”: “Materials”,

“building”: “327”,

“section”: {

“sectionId”: “1”,

“room”: “19”

},

“supervisor”: “be131”

}

}

The new piece of information (i.e. section) added to your API/Service, might not impact the behavior of your application, but since the API/Service requirement has changed, the automation script needs to change as well.

Solution

To avoid this, the best approach would be to use templates, and there are several template libraries such as pebble. For every API call, you can then create a template with default values. When you call the steps, you just have to update the default values before calling. So, when the API changes, all you have to do is update the template and all the BDD steps that refer to this template would get automatically updated.

Problem

Typically, when you run an automated BDD suite, you run it on different machines (e.g., your peer’s machine) and in different environments (e.g., local, development, CI/CD pipelines, etc…). When you hard code environment-specific variables such as URLs, database credentials, API endpoints, or API credentials, you’ll end up with a BDD suite that can’t be supported and run in different environments.

Solution

Using NoCodeBDD, this can be easily achieved by defining these environment-specific variables in the environment section in NoCodeBDD and referring to them in your step definitions. When you want to run NoCodeBDD in a different environment, all you have to do is define the environment variables for that environment and NoCodeBDD would take care of the rest. This is shown in the video below:

https://nocodebdd.live/environment-variables

Problem

When writing BDD Steps, if the reusability of the steps is not taken care of, then the BDD automation test suite ends up having quite a lot of duplication steps. Over time, it’ll become unmanageable and difficult to scale.

Let’s take an example of how not reusing the steps would cause an issue. In the following screenshot, a scenario is defined to check for successful student registration:

If a new scenario has to be created to check for a negative scenario (i.e. Student registration process should fail if student age is not provided), then the user might end up creating something similar to the following scenario:

When you write automation scripts, you would end up writing separate scripts for the When and Then steps as they have a different description, though they test the same behavior which is the student registration process. When using something like Cucumber to automate this, the team would end up duplicating quite a lot of code.

Even if the code is well written and reused, the team would end up duplicating at least the method and also the unnecessary step duplication. From a readability and maintenance point of view, this is not ideal. Typically a project would scale to have hundreds of scenarios, if not thousands. If you don’t address these step duplication, your BDD scenarios and the accompanying automation scripts will soon become difficult to manage.

Solution

If you are testing the same behavior, then consider reusing the steps. For example, if you are calling the registration step in multiple places, you could easily parameterize and reuse it wherever it is required as shown below:

Problem

Having imperative test cases is not the way to execute BDD. Here is an excellent article that explains the difference between imperative and declarative scenarios. The big problem with the imperative scenario is that it doesn’t explain the business behavior or the rules. Instead, it explains the actual implementation. Imperative scenarios are hard to maintain, not scaleable and don’t explain the business intent.

Solution

Implement declarative scenarios that explain the business rules. In other words, write scenarios that explain the behavior of the system. Business users are not interested in the imperative scenario shown above. If you are the actual user of the above-mentioned site, I’m positive that no one needs to explain every single step to registering for you.

All you need to know is that if I enter all the necessary information, I can register. In other words, focus on “what” is being tested (in this case, the registration process) and not “how” it is tested (in this case, entering each and every value). Below, you’ll find a sample declarative scenario for the imperative scenario above.

Writing declarative scenarios improves business collaboration. Namely, the business users engage much better, and your BDD scenarios will still work without issue when your application (not the behavior!) changes. As a result, you drastically reduce the maintenance of your automation scripts.

Conclusion

BDD Automation is hard for that matter any test automation is hard. This was one of the main driver to craete NoCodeBDD, which incorporates most of these issues in to the tool so the users can start their BDD Automation from day one and avoid these issues, which are difficult to resolve at a later stage without major rework. NoCodeBDD team also works closely with their users through regular online catch ups to make sure the right approach been taken. You can download a free version of NoCodeBDD from www.nocodebdd.com/download