The `TEST_FILTER` argument can be used to filter tests during the
discovery phase. It combines `--gtest_filter=<expr>` with the
`--gtest_list_tests` argument when invoking the test excutable for
listing defined tests.
Fixes: #17493
When using a file system which only has second resolution timestamps,
there is a reasonably high likelihood of timestamps being the same.
The IS_NEWER_THAN test returns true when timestamps are the same,
so don't redo test discovery when they match exactly.
The generated ..._include.cmake file contained an inclusion of
GoogleTestAddTests without any path. In general, this is a good
approach since it enables to correctly catch possibly
user-customized files.
However, in this case, it didn’t work this way since the
..._include.cmake file is evaluated by a separate CMake call under
a custom command. Because of this, the CMAKE_MODULE_PATH is not set
as expected by the user and the said inclusion catches CMake own
version of the module rather than the possibly user-customized one.
This change fixes this by making the inclusion with an absolute path
determined upon the ..._include.cmake file generation.
Introducing a new DISCOVERY_MODE mode option, which provides greater control
over when gtest_discover_tests perforsm test discovery.
It has two supported modes:
* POST_BUILD
* PRE_TEST
POST_BUILD is the default behavior, which adds a POST_BUILD command
to perform test discovery after the test has been built.
PRE_TEST is a new mode, which delays test discovery until test execution.
DISCOVERY_MODE can be controlled in two ways:
1. Setting the DISCOVERY_MODE when calling gtest_discover_tests
2. Setting the global CMAKE_GTEST_DISCOVER_TESTS_DISCOVERY_MODE
prior to calling gtest_discover_tests
By delaying test discovery until ctest runtime,
we can provide better cross compile support,
because we're more likely to be in an environment that can run the test executable.
This was achieved by moving the command for test discovery
into the generated ctest include file.
In PRE_TEST mode, the generated include file now has the form:
if(EXISTS "path/to/test.exe")
if("path/to/test.exe" IS_NEWER_THAN "path/to/ctest_file")
// command to discover tests
// and generate ctest_file
endif()
include("path/to/ctest_file")
endif()
elseif()
// test not built
endif()
Which generates the appropriate CTest files at runtime
and regenerates the CTest files when the executable is updated.
In the old approach, test discovery was always added as POST_BUILD step
and the executable was ran in order to scan for tests.
If the test executable failed to run for any reason,
this resulted in a link failure.
This failure could more commonly occur when cross compiling,
because your build environment might not have
the appropriate runtime environment dlls required to run the test executable.
I also ran into the issue when compiling a Qt application using Qt Creator.
Qt Creator manages its build and runtime environments separately
and only adds the Qt dlls to the environment during runtime -- not during build.
So when gtest_discover_tests ran my test executable,
it promptly crashed because the environment wasn't correct,
which resulted in a build failure.
Setting the DISCOVERY_MODE to PRE_TEST fixed my build failure
by delaying test discovery until runtime,
because this allowed the test exe to run in the proper environment.
A few non-trivial implementation details worth noting:
1. bracket_arguments
In the PRE_TEST side, parameters whose contents might contain special characters,
need to be handled with great care.
To aid in escaping these list arguments, backslashes, and other special characters,
and ensuring that they are preserved in the generated file,
bracket arguments (i.e. [== <...> ==]) are used.
For example:
gtest_discover_tests(
...
EXTRA_ARGS how now "\"brown\" cow"
)
Generates a file with the following call:
gtest_discover_tests_impl(
...
TEST_EXTRA_ARGS [==[how;now;"brown" cow]==]
)
This way the arguments are forwarded correctly.
2. multi-config generators
Multi-Config generators (e.g. MSBuild) now work more correctly in
PRE_TEST than POST_BUILD.
PRE_TEST is more correct because it will generate a unique CTest file for each configuration.
It generates a per config file responsible for test discovery:
foo[1]_include-Debug.cmake
foo[1]_include-MinSizeRel.cmake
foo[1]_include-Release.cmake
foo[1]_include-RelWithDebInfo.cmake
A per config file for containing the google tests:
foo[1]_tests-Debug.cmake
And an outer ctest file:
foo[1]_include-Debug.cmake
That is generically written to include the correct configuration file
by looking at the value of ${CTEST_CONFIGURATION_TYPE} when CTest runs.
POST_BUILD, in contrast, preserves the existing functionality.
Tests are disocvered based on the last configuration that was chosen
and only a single file is produced:
foo[1]_include.cmake
foo[1]_tests.cmake
But it runs with whatever executable requested by ctest --config,
which means it's possible to run the wrong tests
if some tests were enabled in one configuration but not another.
When executing googltests in parallel using 'ctest -j n' and using
'--gtest_output=xml' there is a race condition upon file creation.
See googletest issue https://github.com/google/googletest/issues/2506.
As all testcases (potentially) can be run in parallel each testcase has
to create it's own XML JUnit file. EXTRA_ARGS is not suitable because it
is identical per testsuite.
So instead a new (opitional) parameter has been introduced to specify
the storage location for each test of the testsuite.
If policy settings at the time the GoogleTest module is included are
such that CMP0057 is unset or set to OLD, the use of IN_LIST with
if() will lead to an error. Therefore, explicitly specify the policy
settings for the whole file to ensure the function implementations
have access to the required CMake features.
Fixes: #18198
In gtest_discover_tests(), the TIMEOUT keyword was making it
impossible to set the TIMEOUT test property via the PROPERTIES
keyword. This would be a frequent case, but it doesn't complain
and instead silently does something different to what would
normally be expected. The TIMEOUT keyword has been renamed
to DISCOVERY_TIMEOUT, thereby removing the clash.
This is a breaking change. 3.10.1 and 3.10.2 were the only versions
that supported the TIMEOUT keyword and uses of it were likely
not working as intended.
Fixes: #17801
Add a TIMEOUT option to gtest_discover_tests. This provides a
work-around in case a test goes out to lunch, rather than causing the
build to simply hang. (Although this is still a problem with the user's
project, hanging the build is not cool, especially in the case of
automatically running CI builds. It is much preferred that the build
should actively fail in this case, and it is trivially easy for us to
implement that.)
According to the documentation, tests can be discovered for a target
multiple times by using a different prefix and/or suffix to ensure name
uniqueness. However, while this worked for gtest_add_tests, it did not
work with gtest_discover_tests because the generated file that sets up
the tests was named based only on the target name, and so subsequent
discovery from the same target would clobber earlier discovery.
Fix this by introducing a counter that records how many times discovery
has been used on a target, and use this to generate unique names of the
generated test list files.
Add a new gtest_discover_tests function to GoogleTest.cmake,
implementing dynamic test discovery (i.e. tests are discovered by
actually running the test executable and asking for the list of
available tests, which is used to dynamically declare the tests) rather
than the source-parsing approach used by gtest_add_tests. Compared to
the source-parsing approach, this has the advantage of being robust
against users declaring tests in unusual ways, and much better support
for advanced features such as parameterized tests.
A unit test, modeled after the TEST_INCLUDE_DIR[S] test, is also
included. Note that the unit test does not actually require that Google
Test is available. The new functionality does not actually depend on
Google Test as such; it only requires that the test executable lists
tests in the expected format when invoked with --gtest_list_tests, which
the unit test can fake readily.
Now has keyword-based arguments (old syntax form is still supported).
Discovered tests can have a prefix and/or suffix added to the test names
and the list of discovered tests is available to the caller. The working
dir can also be set and the dependency on the source files is now
optional instead of mandatory.
Extract the `gtest_add_tests` macro from `FindGTest` into a separate
module. GTest or GoogleTest can be used by a project in a several
different ways, including installed libraries in the system, from an
ExternalProject, or adding the GTest source directory as a sub directory
of the project. As not all of these uses are supported by the FindGTest
module the useful `gtest_add_tests` macro is separated to easily enable
reuse.
Issue: #14151
