mirror of
https://github.com/Kitware/CMake.git
synced 2026-01-06 05:40:54 -06:00
Merge topic 'test-fixtures'
73f47c9e CTest: Add support for test fixturesThis commit is contained in:
Brad King
@@ -304,6 +304,9 @@ Properties on Tests
|
||||
/prop_test/DEPENDS
|
||||
/prop_test/ENVIRONMENT
|
||||
/prop_test/FAIL_REGULAR_EXPRESSION
|
||||
/prop_test/FIXTURES_CLEANUP
|
||||
/prop_test/FIXTURES_REQUIRED
|
||||
/prop_test/FIXTURES_SETUP
|
||||
/prop_test/LABELS
|
||||
/prop_test/MEASUREMENT
|
||||
/prop_test/PASS_REGULAR_EXPRESSION
|
||||
|
||||
@@ -3,4 +3,8 @@ DEPENDS
|
||||
|
||||
Specifies that this test should only be run after the specified list of tests.
|
||||
|
||||
Set this to a list of tests that must finish before this test is run.
|
||||
Set this to a list of tests that must finish before this test is run. The
|
||||
results of those tests are not considered, the dependency relationship is
|
||||
purely for order of execution (i.e. it is really just a *run after*
|
||||
relationship). Consider using test fixtures with setup tests if a dependency
|
||||
with successful completion is required (see :prop_test:`FIXTURES_REQUIRED`).
|
||||
|
||||
46
Help/prop_test/FIXTURES_CLEANUP.rst
Normal file
46
Help/prop_test/FIXTURES_CLEANUP.rst
Normal file
@@ -0,0 +1,46 @@
|
||||
FIXTURES_CLEANUP
|
||||
----------------
|
||||
|
||||
Specifies a list of fixtures for which the test is to be treated as a cleanup
|
||||
test.
|
||||
|
||||
Fixture cleanup tests are ordinary tests with all of the usual test
|
||||
functionality. Setting the ``FIXTURES_CLEANUP`` property for a test has two
|
||||
primary effects:
|
||||
|
||||
- CTest will ensure the test executes after all other tests which list any of
|
||||
the fixtures in its :prop_test:`FIXTURES_REQUIRED` property.
|
||||
|
||||
- If CTest is asked to run only a subset of tests (e.g. using regular
|
||||
expressions or the ``--rerun-failed`` option) and the cleanup test is not in
|
||||
the set of tests to run, it will automatically be added if any tests in the
|
||||
set require any fixture listed in ``FIXTURES_CLEANUP``.
|
||||
|
||||
A cleanup test can have multiple fixtures listed in its ``FIXTURES_CLEANUP``
|
||||
property. It will execute only once for the whole CTest run, not once for each
|
||||
fixture. A fixture can also have more than one cleanup test defined. If there
|
||||
are multiple cleanup tests for a fixture, projects can control their order with
|
||||
the usual :prop_test:`DEPENDS` test property if necessary.
|
||||
|
||||
A cleanup test is allowed to require other fixtures, but not any fixture listed
|
||||
in its ``FIXTURES_CLEANUP`` property. For example:
|
||||
|
||||
.. code-block:: cmake
|
||||
|
||||
# Ok: Dependent fixture is different to cleanup
|
||||
set_tests_properties(cleanupFoo PROPERTIES
|
||||
FIXTURES_CLEANUP Foo
|
||||
FIXTURES_REQUIRED Bar
|
||||
)
|
||||
|
||||
# Error: cannot require same fixture as cleanup
|
||||
set_tests_properties(cleanupFoo PROPERTIES
|
||||
FIXTURES_CLEANUP Foo
|
||||
FIXTURES_REQUIRED Foo
|
||||
)
|
||||
|
||||
Cleanup tests will execute even if setup or regular tests for that fixture fail
|
||||
or are skipped.
|
||||
|
||||
See :prop_test:`FIXTURES_REQUIRED` for a more complete discussion of how to use
|
||||
test fixtures.
|
||||
94
Help/prop_test/FIXTURES_REQUIRED.rst
Normal file
94
Help/prop_test/FIXTURES_REQUIRED.rst
Normal file
@@ -0,0 +1,94 @@
|
||||
FIXTURES_REQUIRED
|
||||
-----------------
|
||||
|
||||
Specifies a list of fixtures the test requires. Fixture names are case
|
||||
sensitive.
|
||||
|
||||
Fixtures are a way to attach setup and cleanup tasks to a set of tests. If a
|
||||
test requires a given fixture, then all tests marked as setup tasks for that
|
||||
fixture will be executed first (once for the whole set of tests, not once per
|
||||
test requiring the fixture). After all tests requiring a particular fixture
|
||||
have completed, CTest will ensure all tests marked as cleanup tasks for that
|
||||
fixture are then executed. Tests are marked as setup tasks with the
|
||||
:prop_test:`FIXTURES_SETUP` property and as cleanup tasks with the
|
||||
:prop_test:`FIXTURES_CLEANUP` property. If any of a fixture's setup tests fail,
|
||||
all tests listing that fixture in their ``FIXTURES_REQUIRED`` property will not
|
||||
be executed. The cleanup tests for the fixture will always be executed, even if
|
||||
some setup tests fail.
|
||||
|
||||
When CTest is asked to execute only a subset of tests (e.g. by the use of
|
||||
regular expressions or when run with the ``--rerun-failed`` command line
|
||||
option), it will automatically add any setup or cleanup tests for fixtures
|
||||
required by any of the tests that are in the execution set.
|
||||
|
||||
Since setup and cleanup tasks are also tests, they can have an ordering
|
||||
specified by the :prop_test:`DEPENDS` test property just like any other tests.
|
||||
This can be exploited to implement setup or cleanup using multiple tests for a
|
||||
single fixture to modularise setup or cleanup logic.
|
||||
|
||||
The concept of a fixture is different to that of a resource specified by
|
||||
:prop_test:`RESOURCE_LOCK`, but they may be used together. A fixture defines a
|
||||
set of tests which share setup and cleanup requirements, whereas a resource
|
||||
lock has the effect of ensuring a particular set of tests do not run in
|
||||
parallel. Some situations may need both, such as setting up a database,
|
||||
serialising test access to that database and deleting the database again at the
|
||||
end. For such cases, tests would populate both ``FIXTURES_REQUIRED`` and
|
||||
:prop_test:`RESOURCE_LOCK` to combine the two behaviours. Names used for
|
||||
:prop_test:`RESOURCE_LOCK` have no relationship with names of fixtures, so note
|
||||
that a resource lock does not imply a fixture and vice versa.
|
||||
|
||||
Consider the following example which represents a database test scenario
|
||||
similar to that mentioned above:
|
||||
|
||||
.. code-block:: cmake
|
||||
|
||||
add_test(NAME testsDone COMMAND emailResults)
|
||||
add_test(NAME fooOnly COMMAND testFoo)
|
||||
add_test(NAME dbOnly COMMAND testDb)
|
||||
add_test(NAME dbWithFoo COMMAND testDbWithFoo)
|
||||
add_test(NAME createDB COMMAND initDB)
|
||||
add_test(NAME setupUsers COMMAND userCreation)
|
||||
add_test(NAME cleanupDB COMMAND deleteDB)
|
||||
add_test(NAME cleanupFoo COMMAND removeFoos)
|
||||
|
||||
set_tests_properties(setupUsers PROPERTIES DEPENDS createDB)
|
||||
|
||||
set_tests_properties(createDB PROPERTIES FIXTURES_SETUP DB)
|
||||
set_tests_properties(setupUsers PROPERTIES FIXTURES_SETUP DB)
|
||||
set_tests_properties(cleanupDB PROPERTIES FIXTURES_CLEANUP DB)
|
||||
set_tests_properties(cleanupFoo PROPERTIES FIXTURES_CLEANUP Foo)
|
||||
set_tests_properties(testsDone PROPERTIES FIXTURES_CLEANUP "DB;Foo")
|
||||
|
||||
set_tests_properties(fooOnly PROPERTIES FIXTURES_REQUIRED Foo)
|
||||
set_tests_properties(dbOnly PROPERTIES FIXTURES_REQUIRED DB)
|
||||
set_tests_properties(dbWithFoo PROPERTIES FIXTURES_REQUIRED "DB;Foo")
|
||||
|
||||
set_tests_properties(dbOnly dbWithFoo createDB setupUsers cleanupDB
|
||||
PROPERTIES RESOURCE_LOCK DbAccess)
|
||||
|
||||
Key points from this example:
|
||||
|
||||
- Two fixtures are defined: ``DB`` and ``Foo``. Tests can require a single
|
||||
fixture as ``fooOnly`` and ``dbOnly`` do, or they can depend on multiple
|
||||
fixtures like ``dbWithFoo`` does.
|
||||
|
||||
- A ``DEPENDS`` relationship is set up to ensure ``setupUsers`` happens after
|
||||
``createDB``, both of which are setup tests for the ``DB`` fixture and will
|
||||
therefore be executed before the ``dbOnly`` and ``dbWithFoo`` tests
|
||||
automatically.
|
||||
|
||||
- No explicit ``DEPENDS`` relationships were needed to make the setup tests run
|
||||
before or the cleanup tests run after the regular tests.
|
||||
|
||||
- The ``Foo`` fixture has no setup tests defined, only a single cleanup test.
|
||||
|
||||
- ``testsDone`` is a cleanup test for both the ``DB`` and ``Foo`` fixtures.
|
||||
Therefore, it will only execute once regular tests for both fixtures have
|
||||
finished (i.e. after ``fooOnly``, ``dbOnly`` and ``dbWithFoo``). No
|
||||
``DEPENDS`` relationship was specified for ``testsDone``, so it is free to
|
||||
run before, after or concurrently with other cleanup tests for either
|
||||
fixture.
|
||||
|
||||
- The setup and cleanup tests never list the fixtures they are for in their own
|
||||
``FIXTURES_REQUIRED`` property, as that would result in a dependency on
|
||||
themselves and be considered an error.
|
||||
47
Help/prop_test/FIXTURES_SETUP.rst
Normal file
47
Help/prop_test/FIXTURES_SETUP.rst
Normal file
@@ -0,0 +1,47 @@
|
||||
FIXTURES_SETUP
|
||||
--------------
|
||||
|
||||
Specifies a list of fixtures for which the test is to be treated as a setup
|
||||
test.
|
||||
|
||||
Fixture setup tests are ordinary tests with all of the usual test
|
||||
functionality. Setting the ``FIXTURES_SETUP`` property for a test has two
|
||||
primary effects:
|
||||
|
||||
- CTest will ensure the test executes before any other test which lists the
|
||||
fixture(s) in its :prop_test:`FIXTURES_REQUIRED` property.
|
||||
|
||||
- If CTest is asked to run only a subset of tests (e.g. using regular
|
||||
expressions or the ``--rerun-failed`` option) and the setup test is not in
|
||||
the set of tests to run, it will automatically be added if any tests in the
|
||||
set require any fixture listed in ``FIXTURES_SETUP``.
|
||||
|
||||
A setup test can have multiple fixtures listed in its ``FIXTURES_SETUP``
|
||||
property. It will execute only once for the whole CTest run, not once for each
|
||||
fixture. A fixture can also have more than one setup test defined. If there are
|
||||
multiple setup tests for a fixture, projects can control their order with the
|
||||
usual :prop_test:`DEPENDS` test property if necessary.
|
||||
|
||||
A setup test is allowed to require other fixtures, but not any fixture listed
|
||||
in its ``FIXTURES_SETUP`` property. For example:
|
||||
|
||||
.. code-block:: cmake
|
||||
|
||||
# Ok: dependent fixture is different to setup
|
||||
set_tests_properties(setupFoo PROPERTIES
|
||||
FIXTURES_SETUP Foo
|
||||
FIXTURES_REQUIRED Bar
|
||||
)
|
||||
|
||||
# Error: cannot require same fixture as setup
|
||||
set_tests_properties(setupFoo PROPERTIES
|
||||
FIXTURES_SETUP Foo
|
||||
FIXTURES_REQUIRED Foo
|
||||
)
|
||||
|
||||
If any of a fixture's setup tests fail, none of the tests listing that fixture
|
||||
in its :prop_test:`FIXTURES_REQUIRED` property will be run. Cleanup tests will,
|
||||
however, still be executed.
|
||||
|
||||
See :prop_test:`FIXTURES_REQUIRED` for a more complete discussion of how to use
|
||||
test fixtures.
|
||||
@@ -5,3 +5,6 @@ Specify a list of resources that are locked by this test.
|
||||
|
||||
If multiple tests specify the same resource lock, they are guaranteed
|
||||
not to run concurrently.
|
||||
|
||||
See also :prop_test:`FIXTURES_REQUIRED` if the resource requires any setup or
|
||||
cleanup steps.
|
||||
|
||||
8
Help/release/dev/test-fixtures.rst
Normal file
8
Help/release/dev/test-fixtures.rst
Normal file
@@ -0,0 +1,8 @@
|
||||
test-fixtures
|
||||
-------------
|
||||
|
||||
* CTest now supports test fixtures through the new :prop_test:`FIXTURES_SETUP`,
|
||||
:prop_test:`FIXTURES_CLEANUP` and :prop_test:`FIXTURES_REQUIRED` test
|
||||
properties. When using regular expressions or ``--rerun-failed`` to limit
|
||||
the tests to be run, a fixture's setup and cleanup tests will automatically
|
||||
be added to the execution set if any test requires that fixture.
|
||||
Reference in New Issue
Block a user