The function gtest_discover_tests calls the passed test executable with
the parameter --gtest_list_tests and parses the output to find all
tests.
In case of value-parameterized tests ([1]), the test values are included
in the output. While test names are alphanumeric, the values can contain
arbitrary content.
First, the output is separated into lines with `foreach`. Included
semi-colons breaks this and need to get escaped.
Afterwards, the testname is passed on to the `add_command` helper. This
helper was converted into a macro in commit dac201442d (GoogleTest:
Optimize gtest_discover_tests, 2020-02-18). As a macro, its arguments
are re-evaluated. Therefore we need to escape `\`, `;` and to prevent
unwanted variable expansion `$`.
Fixes: #20661
[1] <https://github.com/google/googletest/blob/0eea2e9/googletest/docs/advanced.md#value-parameterized-tests>
PRE_TEST makes it possible to properly distinguish between test cases
that exist only in certain configurations.
In the new test scenario, debug tests are disabled in release builds,
and release tests are disabled in debug builds
when a multi config generator is used.
Note, this is a bit of a hack and *only* works for PRE_TEST mode.
POST_BUILD makes no attempt to get this right. It preserves the status quo
and you obtain the tests that were last discovered.
See further discussion in !4078
Ideally, the POST_BUILD behavior could be fixed
by using generator expressions in OUTPUT and BYPRODUCT expressions.
Then you could do something like:
set(ctest_include_file "${ctest_file_base}_include-$<CONFIG>.cmake")
set(ctest_tests_file "${ctest_file_base}_tests-$<CONFIG>.cmake")
Once #12877 lands, maybe this can be revisited.
Co-authored-by: Ryan Thornton <ThorntonRyan@JohnDeere.com>
Co-authored-by: Kevin Puetz <PuetzKevinA@JohnDeere.com>
Now, the unit tests are ran twice -- once with POST_BUILD (i.e. default mode)
and again with PRE_TEST (i.e. new discovery mode).
Both modes of setting gtest discovery mode are also tested:
1. Using the global override (i.e. CMAKE_GTEST_DISCOVER_TESTS_DISCOVERY_MODE)
2. Explicitly passing DISCOVERY_MODE in calls to gtest_discover_tests (in GoogleTestDiscoveryTimeout.cmake)
The goal is to show that the new PRE_TEST discovery mode does not break existing behavior
(i.e. should not break POST_BUILD mode) and should also pass the same tests
in the same way.
A few non trivial implementation details worth noting:
1. Refactoring discovery_timeout_test into own project
Originally, I tried doing:
```
run_GoogleTest(POST_BUILD)
run_GoogleTest(PRE_TEST)
```
Without changing the internal structure of run_GoogleTest.
But since discovery_timeout_test is part of the same project as the other tests,
and CTest include files always get evaluated and that's where test discovery occurs,
this means every other test now notices the timeout problem when running in PRE_TEST mode.
As a result, keeping the existing test structure meant that each existing test
(and any new test) would need to have its own PRE_TEST / POST_BUILD variant for stderr and stdout
in order to handle the case where discovery_timeout_test timed out.
This exponential increase in test output files introduced unnecessary complexity
and made it more cumbersome to work on test cases.
Why should an unrelated test case care about discovery_timeout_test?
So, to fix that issue, the tests were broken apart into two main groups:
1. run_GoogleTest_discovery_timeout (the test dealing with discovery_timeout_test)
2. run_GoogleTest (everything else)
This isolates the PRE_TEST / POST_BUILD timeout variants to a single test case.
And the other test cases remain unchanged -- further driving home the point that
DISCOVERY_MODE shouldn't change existing behavior.
2. Different number of PRE_TEST / POST_BUILD file variants
On the PRE_TEST path, different build systems / compilers (i.e. MSBuild and ninja/gcc)
produces different build output when building discovery_timeout_test,
but we don't actually care what it is, just as long as it builds
successfully.
This the fundamental difference in behavior between POST_BUILD (which would have failed)
and PRE_TEST (which doesn't) and is the reason why we don't need
a GoogleTest-discovery-build-result.txt or GoogleTest-discovery-build-stdout.txt
3. Fix flaky discovery timeout test
The test expects to see:
> Output:
> timeout
> case.
But sometimes, the test would only produce:
> Output:
> timout
In certain environments, specifically when built with OpenWatcom 1.4,
and while the build server was under heavy load (i.e. running many tests in parallel),
std::endl behaves inconsistently and doesn't completely
flush std::cout when the program is terminated due to timeout.
This results in inconsistent test failures because the actual output
doesn't fully match what's expected.
At first we tried adding an additional:
std::cout << std::flush
That didn't work. But using C-style printf() and fflush() appears to do
the trick:
> This time I managed to get on the machine while it was still busy doing other nightly builds
> and could reproduce the problem reliably. With that I was finally able to find a fix.
> It turns out my earlier hypothesis that C++ stream flushing was not working on the old compiler was correct,
> but even .flush() is not enough.
> I changed it to use C-style printf() and fflush() and now the test passes on that build.
> -- Brad King <brad.king@kitware.com>
Co-authored-by: Ryan Thornton <ThorntonRyan@JohnDeere.com>
Co-authored-by: Kevin Puetz <PuetzKevinA@JohnDeere.com>
Discovery timeout test needs split out into two components:
- build
- test
Building the project should result in a failure due to the timeout
when executing gtest discovery as a post build step.
Likewise, if you ran CTest after this build failure,
we should *also* detect that the discovery_timeout_test has not been built
because no discovery_timeout_test[1]_tests.cmake has been created
due to the test discovery failure (caused by the timeout).
Run the `clang-format.bash` script to update all our C and C++ code to a
new style defined by `.clang-format`. Use `clang-format` version 6.0.
* If you reached this commit for a line in `git blame`, re-run the blame
operation starting at the parent of this commit to see older history
for the content.
* See the parent commit for instructions to rebase a change across this
style transition commit.
In gtest_discover_tests(), the TIMEOUT keyword was making it
impossible to set the TIMEOUT test property via the PROPERTIES
keyword. This would be a frequent case, but it doesn't complain
and instead silently does something different to what would
normally be expected. The TIMEOUT keyword has been renamed
to DISCOVERY_TIMEOUT, thereby removing the clash.
This is a breaking change. 3.10.1 and 3.10.2 were the only versions
that supported the TIMEOUT keyword and uses of it were likely
not working as intended.
Fixes: #17801
Add a test to verify expected behavior when a test executable on which
gtest_discover_tests is invoked was not built. (The test for the newly
added timeout conveniently sets up this situation, so this new test was
almost trivial to add.)
Add a TIMEOUT option to gtest_discover_tests. This provides a
work-around in case a test goes out to lunch, rather than causing the
build to simply hang. (Although this is still a problem with the user's
project, hanging the build is not cool, especially in the case of
automatically running CI builds. It is much preferred that the build
should actively fail in this case, and it is trivially easy for us to
implement that.)
According to the documentation, tests can be discovered for a target
multiple times by using a different prefix and/or suffix to ensure name
uniqueness. However, while this worked for gtest_add_tests, it did not
work with gtest_discover_tests because the generated file that sets up
the tests was named based only on the target name, and so subsequent
discovery from the same target would clobber earlier discovery.
Fix this by introducing a counter that records how many times discovery
has been used on a target, and use this to generate unique names of the
generated test list files.
Add a new gtest_discover_tests function to GoogleTest.cmake,
implementing dynamic test discovery (i.e. tests are discovered by
actually running the test executable and asking for the list of
available tests, which is used to dynamically declare the tests) rather
than the source-parsing approach used by gtest_add_tests. Compared to
the source-parsing approach, this has the advantage of being robust
against users declaring tests in unusual ways, and much better support
for advanced features such as parameterized tests.
A unit test, modeled after the TEST_INCLUDE_DIR[S] test, is also
included. Note that the unit test does not actually require that Google
Test is available. The new functionality does not actually depend on
Google Test as such; it only requires that the test executable lists
tests in the expected format when invoked with --gtest_list_tests, which
the unit test can fake readily.
