Methodology
Testing the code in this repository is taken very seriously. We want
to facilitate confidence that any change will have the intended
side-effects and won’t regress behavior. We do this by providing a
testing framework that is comprehensive and robust.
We currently support the following flavors of tests:
Python unit tests
Mercurial t tests
Mercurial .py tests
The test driver is responsible for identifying which flavor a particular
file is.
Many tests interact with services running locally, commonly inside
Docker containers. Running actual services is encouraged over
mocking.
Running
Tests are executed by running the following in a built
environment:
To see help on options that control execution:
Unknown script arguments will be proxied to Mercurial’s run-tests.py
testing harness.
Common tasks are described below.
Obtain code coverage results (makes tests run slower):
Use Docker images from last run (makes tests run faster):
$ ./run-tests --use-last-images
Test a single file:
$ ./run-tests path/to/test.t
Run all tests in a directory:
$ ./run-tests hgext/pushlog
Run a test in debug mode (see progress, interact with a debugger):
$ ./run-tests -d path/to/test.t
Run tests against all supported Mercurial versions:
$ ./run-tests --all-hg-versions
Run tests with a specific Mercurial installation:
$ ./run-tests --with-hg=/path/to/hg
Do not run Selenium tests:
Do not run tests that require Docker:
$ ./run-tests --no-docker
Run tests 1 at a time:
By default, certain flaky tests do not run in the test suite. You can run them anyways with:
$ ./run-tests --run-flaky
Authoring Tests
Test File Naming
Mercurial .t and .py tests will be automatically discovered from
the following directories:
hgext/*/tests/
hghooks/tests/
Mercurial test filenames must be prefixed with test-. e.g.
test-foo.t.
Python unit tests will be discovered from the following directories:
Python unit test filenames must be prefixed with test. e.g.
test_foo.py.
To write a new test, simply put the test file in one of the
aforementioned directories and name it so that it will be discovered. If
you run run-tests path/test/test and the specified
filename wouldn’t get discovered, an error will be raised saying so.
Choice of Test Flavor
Mercurial t tests
are recommended for most tests.
Mercurial t tests are glorified shell scripts. Tests consist of a
series of commands that will be invoked in a shell. However, they are
much more than that. Expected output from commands is captured inline
in the .t file. For example:
$ hg push
pushing to ssh://user@dummy/$TESTTMP/repos/test-repo
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 1 files
If the expected output differs from actual, the Mercurial test harness
will print a diff of the changes.
.t tests are very useful for testing the behavior of command line
programs.
Unless you are testing a headless Python module, you should probably
be writing t tests.
Python APIs and Helper Scripts
Tests often want to instantiate services and interact with them. To
facilitate this, there are various Python APIs and helper scripts.
The Python APIs are all available as part of the
vcttesting package. There is typically
a subpackage or module for each service you may want to interact with.
To facilitate testing from t tests, there are various command line
tools for interacting with specific services. For example, the
bugzilla tool allows you to start up and stop Bugzilla instances
and perform common actions against them, such as create a bug.
These APIs and scripts exist only to support testing. Their APIs and
arguments are not considered stable. They should not be relied on
outside the context of the testing environment.
The CLI tools all use mach for command dispatching. Simply run
<tool> help to see a list of what commands are available.