The `TEST_FILTER` argument can be used to filter tests during the
discovery phase. It combines `--gtest_filter=<expr>` with the
`--gtest_list_tests` argument when invoking the test excutable for
listing defined tests.
Fixes: #17493
Refactoring in commit 889a7146ff (GoogleTestAddTests: Refactor into
callable method, 2020-03-16, v3.18.0-rc1~450^2~3) accidentally
parsed `TEST_EXECUTOR` as a single-value argument instead of a list.
The Google test framework allows to write the results into an XML file
since commit e9ab39eb1d (GoogleTest: Add XML_OUTPUT_DIR parameter,
2020-03-06, v3.18.0-rc1~538^2~2). This file is passed on the command
line: `--gtest_output=xml:FILE_NAME`.
The module allows to specify a directory to save those files with
**TEST_XML_OUTPUT_PARAM**.
If the option is set, the filename will be set to
`${prefix}${pretty_suite}.${pretty_test}${suffix}.xml`.
The pretty names contain parameters for the tests, if
value-parameterized tests are used. These parameters may not be safe to
use in file names.
There are two possible options:
1. sanitize the file name
2. omit the values and use the internal numbering of gtest
This commit chose option 2. The testname needs to be a valid C++
identifier and should therefore be reasonable for a filename.
Note that the generated names contain slashes. This will lead to
subdirectories, but works on both Linux and Windows.
Fixes: #20877
The function gtest_discover_tests calls the passed test executable with
the parameter --gtest_list_tests and parses the output to find all
tests.
In case of value-parameterized tests ([1]), the test values are included
in the output. While test names are alphanumeric, the values can contain
arbitrary content.
First, the output is separated into lines with `foreach`. Included
semi-colons breaks this and need to get escaped.
Afterwards, the testname is passed on to the `add_command` helper. This
helper was converted into a macro in commit dac201442d (GoogleTest:
Optimize gtest_discover_tests, 2020-02-18). As a macro, its arguments
are re-evaluated. Therefore we need to escape `\`, `;` and to prevent
unwanted variable expansion `$`.
Fixes: #20661
[1] <https://github.com/google/googletest/blob/0eea2e9/googletest/docs/advanced.md#value-parameterized-tests>
Skipped tests are currently reported as successful. Using
SKIP_REGULAR_EXPRESSION on googletest's output prefix, skipped tests can
be detected and accounted accordingly.
Using SKIP_RETURN_CODE is not possible, googletests exit code is not
affected by skipped tests.
Fixes: #19669
Move test discovery logic into new gtest_discover_tests_impl method
and make GoogleTestAddTests aware of whether it is being launched in
CMake's script mode.
When launched in script mode, gtest_discover_tests_impl is called
passing arguments obtained from the definitions passed into the call to cmake.
(i.e. cmake -P GoogleTestAddTests -D <arg1> -D <arg2> ...)
This preserves the existing behavior assumed by GoogleTest.cmake.
Unit tests are unchanged and still pass.
Looking ahead, it also allows GoogleTestAddTests to be included in generated files
and call gtest_discover_tests_impl to perform test discovery at test runtime
with the new PRE_TEST discovery mode introduced later in this branch.
My original approach attempted to call execute_process(cmake -P ...) in
the generated file, the same way POST_BUILD is doing, but I ran into
difficulties serializing the command arguments correctly.
By exposing a way to call gtest_discover_tests_impl directly from our generated file,
we remove a layer of shell quoting / parsing that our arguments have to survive,
which simplifies the act of producing a generated file that behaves the
same as its POST_BUILD counterpart.
When executing googltests in parallel using 'ctest -j n' and using
'--gtest_output=xml' there is a race condition upon file creation.
See googletest issue https://github.com/google/googletest/issues/2506.
As all testcases (potentially) can be run in parallel each testcase has
to create it's own XML JUnit file. EXTRA_ARGS is not suitable because it
is identical per testsuite.
So instead a new (opitional) parameter has been introduced to specify
the storage location for each test of the testsuite.
Fixes regression introduced dac201442d (GoogleTest: Optimize gtest_discover_tests, 2020-02-18).
The generated CTest include files has the form:
if(EXISTS "foo_tests.cmake")
include("foo_tests.cmake")
else()
add_test(foo_NOT_BUILT foo_test_NOT_BUILT)
endif()
Starting in dac201442d, an empty discovery_timeout_test[1]_tests.cmake was written
as soon as GoogleTestAddTests was processed.
This meant, that even if test discovery would fail (due to a crash or timeout in the executable),
we would always produce an empty CTest file.
So instead of reporting:
Unable to find executable: foo_NOT_BUILT
Errors while running CTest
We instead get:
No tests were found!!!
To fix the problem, we WRITE the file on the first call to flush_script,
thus creating the file once we know we have valid output
and the call to gtest_discover_tests hasn't failed.
After creating the file, we then set the mode to APPEND
and append to the file for every subsequent call.
Prior to this, `gtest_discover_tests` could take multiple minutes if
many tests are present. This behavior was caused by a repeated addition
to the variable `script` in the `add_command` function using:
set(script "${script}${NAME}(${_args})\n" PARENT_SCOPE)
This takes very long for large variables.
This commit flushes the contents of the variable to ${CTEST_FILE} after
a certain size of the variable is reached.
In addition:
- cmake_minimum_required(VERSION ${CMAKE_VERSION}) is set to allow usage
of new policies. In particular, CMP0053 speeds up variable expansion.
- No longer appends strings using set(), but instead uses string(APPEND).
- An additional buffer for the tests variable is set.
Add a TIMEOUT option to gtest_discover_tests. This provides a
work-around in case a test goes out to lunch, rather than causing the
build to simply hang. (Although this is still a problem with the user's
project, hanging the build is not cool, especially in the case of
automatically running CI builds. It is much preferred that the build
should actively fail in this case, and it is trivially easy for us to
implement that.)
Add a new gtest_discover_tests function to GoogleTest.cmake,
implementing dynamic test discovery (i.e. tests are discovered by
actually running the test executable and asking for the list of
available tests, which is used to dynamically declare the tests) rather
than the source-parsing approach used by gtest_add_tests. Compared to
the source-parsing approach, this has the advantage of being robust
against users declaring tests in unusual ways, and much better support
for advanced features such as parameterized tests.
A unit test, modeled after the TEST_INCLUDE_DIR[S] test, is also
included. Note that the unit test does not actually require that Google
Test is available. The new functionality does not actually depend on
Google Test as such; it only requires that the test executable lists
tests in the expected format when invoked with --gtest_list_tests, which
the unit test can fake readily.
