← All field notes

One framework to rule them all

When test automation grows, many organisations decide to build a single in-house framework that every team must use. It looks like the efficient choice. Here is how it usually turns into a bottleneck, what the organisation actually needed, and how to build that instead.

21 Nov 20269 min read#QA#Test automation#Architecture

Test automation often starts with one team. Then a second team joins, then a third. Each one picks its own tools and works in its own way.

After a while, the bill gets big. Someone asks a fair question: couldn't we do this once, for everyone?

A common answer is to build an enterprise test framework: one in-house toolkit that every team must use to write its automated tests. I call it "One framework to rule them all".

If you know your Tolkien, you know how that line ends: "and in the darkness bind them". We'll come back to that.

I first ran into this pattern a few years ago, in a large company that was part of an international group. I had no decision power at the time, so I watched it unfold and learned. Later, as a Quality Assurance Architect, I recognised the same pattern early enough to steer the organisation away from it before it could damage the automation effort.

The idea behind it is legitimate. What goes wrong is the answer: the organisation needs standards for how tests plug into its delivery process, and it builds a single technology instead.

Why it's a legitimate idea

Test automation is expensive, and not only because of the tools.

Multiply that by every team and every technology, and the bill gets impressive. So the organisation tries to rationalise: fewer technologies, less training, fewer licences, one infrastructure. The way to get there seems clear: build our own framework, once, and have everyone use it.

To be fair, those savings are real. One stack is cheaper to train for and to support. But there's a price that rarely appears in the business case: testing gets reduced to its lowest common denominator, the set of things every team can do in the same way.

How it goes in practice

It usually starts with a pilot team.

They imagine what their framework should look like, and they quickly reach the conclusion that, in an ideal world, testers should be able to write tests without writing code. So they go for a keyword-driven approach: the framework offers a list of ready-made actions, and testers assemble them like building blocks.

Open browser on    https://app.example.com
Enter              "jdoe"         in   Username
Enter              "secret"       in   Password
Click on           Log in
Check text         "Welcome"      is visible

Each line is a keyword (an action the framework knows how to perform) followed by its values. The tester never sees the code behind it. If this style looks familiar, I discussed its limits in a previous post on BDD: tests written as clicks and fields describe how someone uses the screen, and say little about what the system should do.

In the second organisation, this was exactly the early sign I noticed: management clearly wanted a single, unified list of keywords that every tester in the department would use to build their tests. That can work for a squad, a single team working on one product. It may even work for a tribe, a group of teams working on the same area. It doesn't work for a whole department, where ways of working and technologies differ from one team to the next. A team testing a mobile app and a team testing batch processes on a mainframe don't need the same building blocks.

But back to the pilot. At first, the results are impressive. The team delivers a first batch of tests in record time. The reports look good, and everything is green.

Well, not everything is perfect. The tests don't run automatically in the delivery pipeline (the CI/CD chain that builds, tests and deploys every change). There are no quality gates, the automatic checks that stop a release when tests fail. The framework's own code doesn't follow good development practices. But it was fast, it delivered, and what's missing looks like 10% or 15% of the work at most.

This is where the ninety-ninety rule kicks in. Tom Cargill of Bell Labs put it this way: "The first 90 percent of the code accounts for the first 90 percent of the development time. The remaining 10 percent of the code accounts for the other 90 percent of the development time."

The team works hard on those last points, and its delivery speed visibly drops. After a few months the framework is ready, the toolchain is taking shape, and the framework is still used by exactly one team. Now the organisation pushes to roll it out to everyone else.

The trap closes

Teams that already had automation in place are asked to rewrite or migrate all their tests into the new framework. That's months of work that adds no new test, only the same tests in a different tool.

While migrating, they discover the framework lacks things they need: a protocol, a type of application, a way of handling test data. They file feature requests. The pilot team, now the owner of the Enterprise Test Automation Framework, ends up with a backlog as long as Route 66.

Without anyone deciding it, the organisation has created a new internal product, with its own roadmap, releases, bugs and support requests. And every testing team in the department now depends on it.

Friction becomes the norm. Project managers get creative at avoiding test automation, because it really is slowing their projects down, and the framework team can't support them fast enough. Automation turns into a sensitive subject: management keeps pushing hard to get it done, while the people on the ground can't rely on it, so they avoid it.

Some teams quietly go back to their own tools, outside any control, under the radar.

And there's the end of Tolkien's line. The framework was meant to bring everyone together. Instead it binds teams to a tool they can't leave, and the real testing work moves into the darkness, where nobody can see it, measure it or count it.

What the organisation actually needed

Look at what the organisation wanted when it started. It wasn't that every tester should type the same keywords. It wanted every team's automated tests to fit into the company's software delivery process in the same way, so that results could be trusted, compared and audited.

An enterprise test framework, done right, is mostly a set of procedures, standards and rules, with a small technical part where it helps. Its job is to make sure that, without extra effort:

The technical part can come in several versions, one per programming language used in the organisation, all offering the same common features.

The problem is about standardisation and interfaces, rather than about a single technology.

By building one framework to rule them all, the organisation has instead:

Where to start

Instead of building the one framework, think in terms of process. Decide what every team's tests must provide to fit into the delivery process, and leave the testing community free to choose its tools for everything else.

That freedom still has limits. The organisation can keep a list of approved tools, check their licences and security, and retire the ones nobody can support. That's a matter of governance, and it works the same way as for the product code: development teams choose their languages and libraries within rules the organisation sets, and test code deserves the same treatment.

  1. Agree on the common settings. With your DevOps colleagues (the people who run the delivery pipeline), list the settings every automated test needs: target environment, configuration, secrets. Give them standard names that every pipeline provides.
  2. Define what a test result looks like. Agree on the minimum information every result must contain, whatever tool produced it: which test, which requirement, pass or fail, when, on which environment. Then build or adopt a tool that converts each tool's result files into that common shape. Most test tools already export a standard file format, so this is a translation job, not a rewrite.
  3. Connect results to your test management tool. Once results share one shape, sending them automatically to the tool that tracks tests and requirements is one integration instead of dozens. That's where traceability comes from.
  4. Provide a thin starter layer per language. For one programming language, build or adopt a small library that reads the standard settings, writes results in the right place and format, and sets up common things such as the browser. Teams must use this layer to be compliant with the delivery process, and nothing more is mandatory. How they write their tests on top of it is up to them.
  5. Repeat for every new language that teams actually use.

The enterprise framework has now become an enterprise capability. Instead of forcing people into a rigid frame, it gives them what they need to plug into the delivery process, and leaves them free to pick the libraries and tools that fit the job. One framework rules nobody. A few shared standards bind everyone in the light, where the work can be seen.

If you're facing this right now and want to talk it through, get in touch.