Owner: Architect:

Story (Required)

Acceptance tests should accept configuration options through a mechanism besides environment variables.

Background (Required)

We are starting to use a rather alarming number of environment variables to configure how we can run acceptance tests. While make has some decent features to handle environment variables, it's not entirely perfect. For instance, we have about 6 different ways to invoke tests, depending on if you want wip mode, behavex as a runner, or a dry-run of tests. With some of the improvements in the pipeline as well, such as provided images and provided clusters, I feel this might only get worse as time goes on.

As a way to work around this, we could have a test runner outside of our make files to run tests. We could still invoke it with make (and its invocation would replace the current acceptance test recipe targets), but a standalone mode might be useful as well. This would let us read test configuration from a provided manifest (some kind of yaml or toml), and it would figure out how exactly tests should be run.

Glossary

Test runner: a script designed to run acceptance tests.

Out of scope

Changes to primaza's agents or main controller shouldn't be necessary, and if they're necessary, should be captured in separate stories.

In Scope

If changes to acceptance tests or their implementations are determined to be necessary, then these changes are within scope of this story.

Approach(Required)

Write a script to accept testing configuration options from both a manifest and command-line options. The command-line should override the manifest if they conflict.

Using the provided options, this script should set up and invoke behave/behavex and run acceptance tests.

Demo requirements(Required)

A demo should show how options might be changed and how acceptance tests can be invoked.

Dependencies

None

Edge Case

None

Acceptance Criteria

The script should accept options from a manifest
The script should accept override from the command-line
Using provided options, the script should invoke acceptance tests.
Documentation: The book should have developer documentation on how to invoke acceptance tests.