Testing¶
Methodology¶
Testing the code in this repository is taken very seriously. We want to facilitate confidence that any change will have the intended side-effects and won’t regress behavior. We do this by providing a testing framework that is comprehensive and robust.
We currently support the following flavors of tests:
- Python unit tests
- Mercurial t tests
- Mercurial .py tests
The test driver is responsible for identifying which flavor a particular file is.
Many tests interact with services running locally, commonly inside Docker containers. Running actual services is encouraged over mocking.
Running¶
Tests are executed by running the following in a built environment:
$ ./run-tests
To see help on options that control execution:
$ ./run-tests --help
Unknown script arguments will be proxied to Mercurial’s run-tests.py
testing harness.
Common tasks are described below.
Obtain code coverage results (makes tests run slower):
$ ./run-tests --cover
Use Docker images from last run (makes tests run faster):
$ ./run-tests --use-last-images
Test a single file:
$ ./run-tests path/to/test.t
Run all tests in a directory:
$ ./run-tests hgext/pushlog
Run a test in debug mode (see progress, interact with a debugger):
$ ./run-tests -d path/to/test.t
Run tests against all supported Mercurial versions:
$ ./run-tests --all-hg-versions
Run tests with a specific Mercurial installation:
$ ./run-tests --with-hg=/path/to/hg
Do not run Selenium tests:
$ ./run-tests --headless
Do not run tests that require Docker:
$ ./run-tests --no-docker
Run tests 1 at a time:
$ ./run-tests -j1
By default, certain flaky tests do not run in the test suite. You can run them anyways with:
$ ./run-tests --run-flaky
Authoring Tests¶
Test File Naming¶
Mercurial .t
and .py
tests will be automatically discovered from
the following directories:
hgext/*/tests/
hghooks/tests/
Mercurial test filenames must be prefixed with test-
. e.g.
test-foo.t
.
Python unit tests will be discovered from the following directories:
pylib/**
Python unit test filenames must be prefixed with test
. e.g.
test_foo.py
.
To write a new test, simply put the test file in one of the
aforementioned directories and name it so that it will be discovered. If
you run run-tests path/test/test
and the specified
filename wouldn’t get discovered, an error will be raised saying so.
Choice of Test Flavor¶
Mercurial t tests are recommended for most tests.
Mercurial t tests are glorified shell scripts. Tests consist of a
series of commands that will be invoked in a shell. However, they are
much more than that. Expected output from commands is captured inline
in the .t
file. For example:
$ hg push
pushing to ssh://user@dummy/$TESTTMP/repos/test-repo
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 1 files
If the expected output differs from actual, the Mercurial test harness will print a diff of the changes.
.t tests are very useful for testing the behavior of command line programs.
Unless you are testing a headless Python module, you should probably be writing t tests.
Python APIs and Helper Scripts¶
Tests often want to instantiate services and interact with them. To facilitate this, there are various Python APIs and helper scripts.
The Python APIs are all available as part of the vcttesting package. There is typically a subpackage or module for each service you may want to interact with.
To facilitate testing from t tests, there are various command line
tools for interacting with specific services. For example, the
bugzilla
tool allows you to start up and stop Bugzilla instances
and perform common actions against them, such as create a bug.
These APIs and scripts exist only to support testing. Their APIs and arguments are not considered stable. They should not be relied on outside the context of the testing environment.
The CLI tools all use mach for command dispatching. Simply run
<tool> help
to see a list of what commands are available.