There's no need to check if flushing is needed after every command
is added to the variable holding the script content. It is enough to only
check once per test name. This simplifies the flushing logic, removing
the need for a separate flush_script() command. Previously, we were
not clearing the flushed script contents in all cases, but this is now
rigorously enforced at the one location where flushing is performed.
Also simplify the flushing of the list of test names, since that too doesn't
need a separate command. It is simpler and safer to handle that
directly inline where the one call to flush_tests_buffer() was
previously being made.
Fixes: #26431
This will now preserve empty values in the TEST_LAUNCHER and
CROSSCOMPILING_EMULATOR target properties for tests added by:
- The add_test() command.
- The ExternalData_Add_Test() command from the ExternalData module.
- The gtest_add_tests() or gtest_discover_tests() commands from the
GoogleTest module.
For the gtest_add_tests() and gtest_discover_tests() commands,
empty elements in the values passed after the EXTRA_ARGS keyword
are also now preserved.
Policy CMP0178 is added to provide backward compatibility with the
old behavior where empty values were silently discarded from the
above cases.
Fixes: #26337
We run test executables to discover tests. Use the same launchers that
are used to run the tests. We already handle `CROSSCOMPILING_EMULATOR`.
Update the logic to account for the `TEST_LAUNCHER` property added by
commit 1ec0372ed4 (add_test: Optionally use a launcher for tests running
in-project targets, 2023-11-11), and for the `CROSSCOMPILING_EMULATOR`
behavior change in commit ca5a300d7f (add_test: Honor
CROSSCOMPILING_EMULATOR only when cross-compiling, 2023-11-02).
Fixes: #25603
Co-authored-by: Brad King <brad.king@kitware.com>
Signed-off-by: Ralf Habacker <ralf.habacker@freenet.de>
If the working directory didn't exist, the old message just says:
Error running test executable.
Path: '<Path to test executable>'
Result: No such file or directory
Output:
This leads the user to the conclusion that the test executable doesn't
exist, which isn't true. Make the true cause visible by reporting the
working directory in the error message.
The internal helper variable '_GOOGLETEST_DISCOVER_TESTS_SCRIPT' can have gone
out-of-scope when 'gtest_discover_tests()' is called, depending on where the
GoogleTest module is actually included. This leads to a silent failure of
dynamic test discovery, since the custom post-build commands actually does
nothing (it basically invokes 'cmake -P ""'). Ctest will then fail to run the
tests, considering them to be 'not built'.
Fix this by determining the path to the GoogleTest module based on
'${CMAKE_ROOT}' instead, which is always available.
A new test case was added to test suite 'RunCMake/GoogleTest' to ensure that
'gtest_discover_tests()' works correctly when invoked in a different variable
scope.
Fixes: #25477
Update commit 1c15eb39d2 (Tests: Suppress failures on macOS arm64 due to
separate Xcode signing phase, 2021-02-22, v3.20.0-rc2~11^2) for Xcode 15,
which ignores our `-Xlinker -adhoc_codesign` flag, prioritizes its own
`-Xlinker -no_adhoc_codesign` flag, and does not add an ad-hoc signature
at link time.
Issue: #21845
Fix a regression from commit 073dd1bd81 (GoogleTest: Change format for
typed tests, 2022-02-07, v3.23.0-rc1~4^2) in the suite name detection.
Co-authored-by: Evgeniy Shcherbina <ixsci@pm.me>
Fixes: #24563
Due to add_command() being a macro it introduced excessive and
nonobvious escaping in different parts of the script. Because of
one of such places the resulting script would have an erroneous
${TEST_LIST} if the user data (in test parameters) had a semicolon.
To eliminate this non-obvious escaping, add_command() was converted
to function. Updated the escaping accordingly.
Fixes: #23059
Before it would output a typed test as follows:
Suit/Type.Case
And now it would be:
Suit.Case<Type>
In case of NO_PRETTY_TYPES it would simply use the type number
instead of its text representation:
Suit.Case<0>
The change is introduced to make sure CTest outputs tests in a
similar fashion which is "*Suit.Case*" and angle brackets "<>"
emphasize that we are dealing with a typed (template) kind.
When there were many cases (two digits or more) the "prettier" would
fail to recognize the pretty part leaving the test name unprocessed.
The fix made sure the processing would work correctly, irrespective
of the case number.
Before the fix, for the following input:
TypedSuite/1. # TypeParam = int
case
TypedSuite/10. # TypeParam = char
case
The output would be:
TypedSuite/int.case
TypedSuite/10. # TypeParam = char.case
Now the output will be:
TypedSuite/int.case
TypedSuite/char.case
Before the fix the gtest_discover_tests() function would strip the
user data in test parameters (everything to the right of GetParam())
of spaces. Now the parameters aren't altered in any way.
Fixes#23058
If a test name contains a square bracket (due to parameters) then it
breaks gtest_discovery_test() function in some not-so-predictable
way. That happens due to the special meaning these brackets have in
the CMake language and they can't be escaped universally.
So the following treatment has been implemented:
* Every occurrence of ('[' | ']') in a test name gets replaced with
the corresponding placeholder ("__osb_*" | "__csb_*") before the
Google Test output processing and gets replaced back before adding
a new test to CTest, keeping the original test name intact in the
CTest output.
The placeholders are chosen that way to minimize the chance of
clashing with something in the user tests but even if the default
ones would clash with something then they are enhanced to not clash
with anything (hence "_*" at the placeholder's end).
* The GTest output gets searched for the default test name guards
("[=[" | "]=]") and if they are found a new one gets generated until
the one is found which can safely encompass any test name. The
search is quite simple: find the least amount of '=' which would
allow escaping any test.
* The resulting ${TEST_LIST} variable will contain every test but
tests with square brackets as there is no way to make sure such
tests won't break the list altogether.
Fixes: #23039
The `TEST_FILTER` argument can be used to filter tests during the
discovery phase. It combines `--gtest_filter=<expr>` with the
`--gtest_list_tests` argument when invoking the test excutable for
listing defined tests.
Fixes: #17493
The RunCMake infrastructure's `*-stdout.txt` expected output files
interpret `\n` as a `\` and `n` rather than a newline. Use a literal
newline instead. Otherwise the cases that were trying to match any
configuration name via `[^\n]*` would fail on `RelWithDebInfo` because
it contains the letter `n`.
Some tests fail because Xcode runs `POST_BUILD` commands before signing
the binaries they run. Tell the linker to perform ad-hoc codesign even
though Xcode normally tells it not to.
Other tests fail because `install_name_tool` does not revise ad-hoc
signatures without the codesign `linker-signed` flag. Add that flag
ourselves where needed by our tests.
For now these changes help our test suite pass so we can use it to cover
everything else. Both of these cases may need further investigation to
update CMake to help projects in general.
Issue: #21845, #21854
The function gtest_discover_tests calls the passed test executable with
the parameter --gtest_list_tests and parses the output to find all
tests.
In case of value-parameterized tests ([1]), the test values are included
in the output. While test names are alphanumeric, the values can contain
arbitrary content.
First, the output is separated into lines with `foreach`. Included
semi-colons breaks this and need to get escaped.
Afterwards, the testname is passed on to the `add_command` helper. This
helper was converted into a macro in commit dac201442d (GoogleTest:
Optimize gtest_discover_tests, 2020-02-18). As a macro, its arguments
are re-evaluated. Therefore we need to escape `\`, `;` and to prevent
unwanted variable expansion `$`.
Fixes: #20661
[1] <https://github.com/google/googletest/blob/0eea2e9/googletest/docs/advanced.md#value-parameterized-tests>
PRE_TEST makes it possible to properly distinguish between test cases
that exist only in certain configurations.
In the new test scenario, debug tests are disabled in release builds,
and release tests are disabled in debug builds
when a multi config generator is used.
Note, this is a bit of a hack and *only* works for PRE_TEST mode.
POST_BUILD makes no attempt to get this right. It preserves the status quo
and you obtain the tests that were last discovered.
See further discussion in !4078
Ideally, the POST_BUILD behavior could be fixed
by using generator expressions in OUTPUT and BYPRODUCT expressions.
Then you could do something like:
set(ctest_include_file "${ctest_file_base}_include-$<CONFIG>.cmake")
set(ctest_tests_file "${ctest_file_base}_tests-$<CONFIG>.cmake")
Once #12877 lands, maybe this can be revisited.
Co-authored-by: Ryan Thornton <ThorntonRyan@JohnDeere.com>
Co-authored-by: Kevin Puetz <PuetzKevinA@JohnDeere.com>
Now, the unit tests are ran twice -- once with POST_BUILD (i.e. default mode)
and again with PRE_TEST (i.e. new discovery mode).
Both modes of setting gtest discovery mode are also tested:
1. Using the global override (i.e. CMAKE_GTEST_DISCOVER_TESTS_DISCOVERY_MODE)
2. Explicitly passing DISCOVERY_MODE in calls to gtest_discover_tests (in GoogleTestDiscoveryTimeout.cmake)
The goal is to show that the new PRE_TEST discovery mode does not break existing behavior
(i.e. should not break POST_BUILD mode) and should also pass the same tests
in the same way.
A few non trivial implementation details worth noting:
1. Refactoring discovery_timeout_test into own project
Originally, I tried doing:
```
run_GoogleTest(POST_BUILD)
run_GoogleTest(PRE_TEST)
```
Without changing the internal structure of run_GoogleTest.
But since discovery_timeout_test is part of the same project as the other tests,
and CTest include files always get evaluated and that's where test discovery occurs,
this means every other test now notices the timeout problem when running in PRE_TEST mode.
As a result, keeping the existing test structure meant that each existing test
(and any new test) would need to have its own PRE_TEST / POST_BUILD variant for stderr and stdout
in order to handle the case where discovery_timeout_test timed out.
This exponential increase in test output files introduced unnecessary complexity
and made it more cumbersome to work on test cases.
Why should an unrelated test case care about discovery_timeout_test?
So, to fix that issue, the tests were broken apart into two main groups:
1. run_GoogleTest_discovery_timeout (the test dealing with discovery_timeout_test)
2. run_GoogleTest (everything else)
This isolates the PRE_TEST / POST_BUILD timeout variants to a single test case.
And the other test cases remain unchanged -- further driving home the point that
DISCOVERY_MODE shouldn't change existing behavior.
2. Different number of PRE_TEST / POST_BUILD file variants
On the PRE_TEST path, different build systems / compilers (i.e. MSBuild and ninja/gcc)
produces different build output when building discovery_timeout_test,
but we don't actually care what it is, just as long as it builds
successfully.
This the fundamental difference in behavior between POST_BUILD (which would have failed)
and PRE_TEST (which doesn't) and is the reason why we don't need
a GoogleTest-discovery-build-result.txt or GoogleTest-discovery-build-stdout.txt
3. Fix flaky discovery timeout test
The test expects to see:
> Output:
> timeout
> case.
But sometimes, the test would only produce:
> Output:
> timout
In certain environments, specifically when built with OpenWatcom 1.4,
and while the build server was under heavy load (i.e. running many tests in parallel),
std::endl behaves inconsistently and doesn't completely
flush std::cout when the program is terminated due to timeout.
This results in inconsistent test failures because the actual output
doesn't fully match what's expected.
At first we tried adding an additional:
std::cout << std::flush
That didn't work. But using C-style printf() and fflush() appears to do
the trick:
> This time I managed to get on the machine while it was still busy doing other nightly builds
> and could reproduce the problem reliably. With that I was finally able to find a fix.
> It turns out my earlier hypothesis that C++ stream flushing was not working on the old compiler was correct,
> but even .flush() is not enough.
> I changed it to use C-style printf() and fflush() and now the test passes on that build.
> -- Brad King <brad.king@kitware.com>
Co-authored-by: Ryan Thornton <ThorntonRyan@JohnDeere.com>
Co-authored-by: Kevin Puetz <PuetzKevinA@JohnDeere.com>
Discovery timeout test needs split out into two components:
- build
- test
Building the project should result in a failure due to the timeout
when executing gtest discovery as a post build step.
Likewise, if you ran CTest after this build failure,
we should *also* detect that the discovery_timeout_test has not been built
because no discovery_timeout_test[1]_tests.cmake has been created
due to the test discovery failure (caused by the timeout).
Run the `clang-format.bash` script to update all our C and C++ code to a
new style defined by `.clang-format`. Use `clang-format` version 6.0.
* If you reached this commit for a line in `git blame`, re-run the blame
operation starting at the parent of this commit to see older history
for the content.
* See the parent commit for instructions to rebase a change across this
style transition commit.
In gtest_discover_tests(), the TIMEOUT keyword was making it
impossible to set the TIMEOUT test property via the PROPERTIES
keyword. This would be a frequent case, but it doesn't complain
and instead silently does something different to what would
normally be expected. The TIMEOUT keyword has been renamed
to DISCOVERY_TIMEOUT, thereby removing the clash.
This is a breaking change. 3.10.1 and 3.10.2 were the only versions
that supported the TIMEOUT keyword and uses of it were likely
not working as intended.
Fixes: #17801
Add a test to verify expected behavior when a test executable on which
gtest_discover_tests is invoked was not built. (The test for the newly
added timeout conveniently sets up this situation, so this new test was
almost trivial to add.)
Add a TIMEOUT option to gtest_discover_tests. This provides a
work-around in case a test goes out to lunch, rather than causing the
build to simply hang. (Although this is still a problem with the user's
project, hanging the build is not cool, especially in the case of
automatically running CI builds. It is much preferred that the build
should actively fail in this case, and it is trivially easy for us to
implement that.)
According to the documentation, tests can be discovered for a target
multiple times by using a different prefix and/or suffix to ensure name
uniqueness. However, while this worked for gtest_add_tests, it did not
work with gtest_discover_tests because the generated file that sets up
the tests was named based only on the target name, and so subsequent
discovery from the same target would clobber earlier discovery.
Fix this by introducing a counter that records how many times discovery
has been used on a target, and use this to generate unique names of the
generated test list files.
Add a new gtest_discover_tests function to GoogleTest.cmake,
implementing dynamic test discovery (i.e. tests are discovered by
actually running the test executable and asking for the list of
available tests, which is used to dynamically declare the tests) rather
than the source-parsing approach used by gtest_add_tests. Compared to
the source-parsing approach, this has the advantage of being robust
against users declaring tests in unusual ways, and much better support
for advanced features such as parameterized tests.
A unit test, modeled after the TEST_INCLUDE_DIR[S] test, is also
included. Note that the unit test does not actually require that Google
Test is available. The new functionality does not actually depend on
Google Test as such; it only requires that the test executable lists
tests in the expected format when invoked with --gtest_list_tests, which
the unit test can fake readily.
