mirror/CMake
1
0
Fork 0
mirror of https://github.com/Kitware/CMake.git synced 2026-01-02 20:00:38 -06:00
Code Issues Packages Projects Releases Wiki Activity

GoogleTest: Add new DISCOVERY_MODE option to gtest_discover_tests

Browse Source 
Introducing a new DISCOVERY_MODE mode option, which provides greater control
over when gtest_discover_tests perforsm test discovery.

It has two supported modes:
* POST_BUILD
* PRE_TEST

POST_BUILD is the default behavior, which adds a POST_BUILD command
to perform test discovery after the test has been built.

PRE_TEST is a new mode, which delays test discovery until test execution.

DISCOVERY_MODE can be controlled in two ways:
1. Setting the DISCOVERY_MODE when calling gtest_discover_tests
2. Setting the global CMAKE_GTEST_DISCOVER_TESTS_DISCOVERY_MODE
   prior to calling gtest_discover_tests

By delaying test discovery until ctest runtime,
we can provide better cross compile support,
because we're more likely to be in an environment that can run the test executable.

This was achieved by moving the command for test discovery
into the generated ctest include file.

In PRE_TEST mode, the generated include file now has the form:

if(EXISTS "path/to/test.exe")
  if("path/to/test.exe" IS_NEWER_THAN "path/to/ctest_file")
    // command to discover tests
    // and generate ctest_file
  endif()
  include("path/to/ctest_file")
endif()
elseif()
  // test not built
endif()

Which generates the appropriate CTest files at runtime
and regenerates the CTest files when the executable is updated.

In the old approach, test discovery was always added as POST_BUILD step
and the executable was ran in order to scan for tests.

If the test executable failed to run for any reason,
this resulted in a link failure.

This failure could more commonly occur when cross compiling,
because your build environment might not have
the appropriate runtime environment dlls required to run the test executable.

I also ran into the issue when compiling a Qt application using Qt Creator.
Qt Creator manages its build and runtime environments separately
and only adds the Qt dlls to the environment during runtime -- not during build.

So when gtest_discover_tests ran my test executable,
it promptly crashed because the environment wasn't correct,
which resulted in a build failure.

Setting the DISCOVERY_MODE to PRE_TEST fixed my build failure
by delaying test discovery until runtime,
because this allowed the test exe to run in the proper environment.

A few non-trivial implementation details worth noting:

1. bracket_arguments

In the PRE_TEST side, parameters whose contents might contain special characters,
need to be handled with great care.

To aid in escaping these list arguments, backslashes, and other special characters,
and ensuring that they are preserved in the generated file,
bracket arguments (i.e. [== <...> ==]) are used.

For example:

    gtest_discover_tests(
      ...
      EXTRA_ARGS how now "\"brown\" cow"
    )

Generates a file with the following call:

    gtest_discover_tests_impl(
      ...
      TEST_EXTRA_ARGS [==[how;now;"brown" cow]==]
    )

This way the arguments are forwarded correctly.

2. multi-config generators

Multi-Config generators (e.g. MSBuild) now work more correctly in
PRE_TEST than POST_BUILD.

PRE_TEST is more correct because it will generate a unique CTest file for each configuration.

It generates a per config file responsible for test discovery:
    foo[1]_include-Debug.cmake
    foo[1]_include-MinSizeRel.cmake
    foo[1]_include-Release.cmake
    foo[1]_include-RelWithDebInfo.cmake

A per config file for containing the google tests:
    foo[1]_tests-Debug.cmake

And an outer ctest file:
    foo[1]_include-Debug.cmake

That is generically written to include the correct configuration file
by looking at the value of ${CTEST_CONFIGURATION_TYPE} when CTest runs.

POST_BUILD, in contrast, preserves the existing functionality.

Tests are disocvered based on the last configuration that was chosen
and only a single file is produced:
    foo[1]_include.cmake
    foo[1]_tests.cmake

But it runs with whatever executable requested by ctest --config,
which means it's possible to run the wrong tests
if some tests were enabled in one configuration but not another.
This commit is contained in:
Ryan Thornton
2020-03-16 12:04:35 -05:00
parent 889a7146ff
commit 75e82a13db
1 changed files with 104 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files

133
Modules/GoogleTest.cmake
View File

@@ -152,6 +152,7 @@ same as the Google Test name (i.e. ``suite.testcase``); see also
[TEST_LIST var]
[DISCOVERY_TIMEOUT seconds]
[XML_OUTPUT_DIR dir]
[DISCOVERY_MODE <POST_BUILD|PRE_TEST>]
)
``gtest_discover_tests`` sets up a post-build command on the test executable
@@ -244,6 +245,22 @@ same as the Google Test name (i.e. ``suite.testcase``); see also
``EXTRA_ARGS --gtest_output=xml`` to avoid race conditions writing the
XML result output when using parallel test execution.
``DISCOVERY_MODE``
Provides greater control over when ``gtest_discover_tests``performs test
discovery. By default, ``POST_BUILD`` sets up a post-build command
to perform test discovery at build time. In certain scenarios, like
cross-compiling, this ``POST_BUILD`` behavior is not desirable.
By contrast, ``PRE_TEST`` delays test discovery until just prior to test
execution. This way test discovery occurs in the target environment
where the test has a better chance at finding appropriate runtime
dependencies.
``DISCOVERY_MODE`` defaults to the value of the
``CMAKE_GTEST_DISCOVER_TESTS_DISCOVERY_MODE`` variable if it is not
passed when calling ``gtest_discover_tests``. This provides a mechanism
for globally selecting a preferred test discovery behavior without having
to modify each call site.
#]=======================================================================]
# Save project's policies
@@ -376,11 +393,12 @@ function(gtest_add_tests)
endfunction()
#------------------------------------------------------------------------------
function(gtest_discover_tests TARGET)
cmake_parse_arguments(
""
"NO_PRETTY_TYPES;NO_PRETTY_VALUES"
"TEST_PREFIX;TEST_SUFFIX;WORKING_DIRECTORY;TEST_LIST;DISCOVERY_TIMEOUT;XML_OUTPUT_DIR"
"TEST_PREFIX;TEST_SUFFIX;WORKING_DIRECTORY;TEST_LIST;DISCOVERY_TIMEOUT;XML_OUTPUT_DIR;DISCOVERY_MODE"
"EXTRA_ARGS;PROPERTIES"
${ARGN}
)
@@ -394,6 +412,12 @@ function(gtest_discover_tests TARGET)
if(NOT _DISCOVERY_TIMEOUT)
set(_DISCOVERY_TIMEOUT 5)
endif()
if(NOT _DISCOVERY_MODE)
if(NOT CMAKE_GTEST_DISCOVER_TESTS_DISCOVERY_MODE)
set(CMAKE_GTEST_DISCOVER_TESTS_DISCOVERY_MODE "POST_BUILD")
endif()
set(_DISCOVERY_MODE ${CMAKE_GTEST_DISCOVER_TESTS_DISCOVERY_MODE})
endif()
get_property(
has_counter
@@ -425,35 +449,86 @@ function(gtest_discover_tests TARGET)
TARGET ${TARGET}
PROPERTY CROSSCOMPILING_EMULATOR
)
add_custom_command(
TARGET ${TARGET} POST_BUILD
BYPRODUCTS "${ctest_tests_file}"
COMMAND "${CMAKE_COMMAND}"
-D "TEST_TARGET=${TARGET}"
-D "TEST_EXECUTABLE=$<TARGET_FILE:${TARGET}>"
-D "TEST_EXECUTOR=${crosscompiling_emulator}"
-D "TEST_WORKING_DIR=${_WORKING_DIRECTORY}"
-D "TEST_EXTRA_ARGS=${_EXTRA_ARGS}"
-D "TEST_PROPERTIES=${_PROPERTIES}"
-D "TEST_PREFIX=${_TEST_PREFIX}"
-D "TEST_SUFFIX=${_TEST_SUFFIX}"
-D "NO_PRETTY_TYPES=${_NO_PRETTY_TYPES}"
-D "NO_PRETTY_VALUES=${_NO_PRETTY_VALUES}"
-D "TEST_LIST=${_TEST_LIST}"
-D "CTEST_FILE=${ctest_tests_file}"
-D "TEST_DISCOVERY_TIMEOUT=${_DISCOVERY_TIMEOUT}"
-D "TEST_XML_OUTPUT_DIR=${_XML_OUTPUT_DIR}"
-P "${_GOOGLETEST_DISCOVER_TESTS_SCRIPT}"
VERBATIM
)
file(WRITE "${ctest_include_file}"
"if(EXISTS \"${ctest_tests_file}\")\n"
" include(\"${ctest_tests_file}\")\n"
"else()\n"
" add_test(${TARGET}_NOT_BUILT ${TARGET}_NOT_BUILT)\n"
"endif()\n"
)
if(_DISCOVERY_MODE STREQUAL "POST_BUILD")
add_custom_command(
TARGET ${TARGET} POST_BUILD
BYPRODUCTS "${ctest_tests_file}"
COMMAND "${CMAKE_COMMAND}"
-D "TEST_TARGET=${TARGET}"
-D "TEST_EXECUTABLE=$<TARGET_FILE:${TARGET}>"
-D "TEST_EXECUTOR=${crosscompiling_emulator}"
-D "TEST_WORKING_DIR=${_WORKING_DIRECTORY}"
-D "TEST_EXTRA_ARGS=${_EXTRA_ARGS}"
-D "TEST_PROPERTIES=${_PROPERTIES}"
-D "TEST_PREFIX=${_TEST_PREFIX}"
-D "TEST_SUFFIX=${_TEST_SUFFIX}"
-D "NO_PRETTY_TYPES=${_NO_PRETTY_TYPES}"
-D "NO_PRETTY_VALUES=${_NO_PRETTY_VALUES}"
-D "TEST_LIST=${_TEST_LIST}"
-D "CTEST_FILE=${ctest_tests_file}"
-D "TEST_DISCOVERY_TIMEOUT=${_DISCOVERY_TIMEOUT}"
-D "TEST_XML_OUTPUT_DIR=${_XML_OUTPUT_DIR}"
-P "${_GOOGLETEST_DISCOVER_TESTS_SCRIPT}"
VERBATIM
)
file(WRITE "${ctest_include_file}"
"if(EXISTS \"${ctest_tests_file}\")\n"
" include(\"${ctest_tests_file}\")\n"
"else()\n"
" add_test(${TARGET}_NOT_BUILT ${TARGET}_NOT_BUILT)\n"
"endif()\n"
)
elseif(_DISCOVERY_MODE STREQUAL "PRE_TEST")
get_property(GENERATOR_IS_MULTI_CONFIG GLOBAL
PROPERTY GENERATOR_IS_MULTI_CONFIG
)
if(GENERATOR_IS_MULTI_CONFIG)
set(ctest_tests_file "${ctest_file_base}_tests-$<CONFIG>.cmake")
endif()
string(CONCAT ctest_include_content
"if(EXISTS \"$<TARGET_FILE:${TARGET}>\")" "\n"
" if(\"$<TARGET_FILE:${TARGET}>\" IS_NEWER_THAN \"${ctest_tests_file}\")" "\n"
" include(GoogleTestAddTests)" "\n"
" gtest_discover_tests_impl(" "\n"
" TEST_EXECUTABLE" " [==[" "$<TARGET_FILE:${TARGET}>" "]==]" "\n"
" TEST_EXECUTOR" " [==[" "${crosscompiling_emulator}" "]==]" "\n"
" TEST_WORKING_DIR" " [==[" "${_WORKING_DIRECTORY}" "]==]" "\n"
" TEST_EXTRA_ARGS" " [==[" "${_EXTRA_ARGS}" "]==]" "\n"
" TEST_PROPERTIES" " [==[" "${_PROPERTIES}" "]==]" "\n"
" TEST_PREFIX" " [==[" "${_TEST_PREFIX}" "]==]" "\n"
" TEST_SUFFIX" " [==[" "${_TEST_SUFFIX}" "]==]" "\n"
" NO_PRETTY_TYPES" " [==[" "${_NO_PRETTY_TYPES}" "]==]" "\n"
" NO_PRETTY_VALUES" " [==[" "${_NO_PRETTY_VALUES}" "]==]" "\n"
" TEST_LIST" " [==[" "${_TEST_LIST}" "]==]" "\n"
" CTEST_FILE" " [==[" "${ctest_tests_file}" "]==]" "\n"
" TEST_DISCOVERY_TIMEOUT" " [==[" "${_DISCOVERY_TIMEOUT}" "]==]" "\n"
" TEST_XML_OUTPUT_DIR" " [==[" "${_XML_OUTPUT_DIR}" "]==]" "\n"
" )" "\n"
" endif()" "\n"
" include(\"${ctest_tests_file}\")" "\n"
"else()" "\n"
" add_test(${TARGET}_NOT_BUILT ${TARGET}_NOT_BUILT)" "\n"
"endif()" "\n"
)
if(GENERATOR_IS_MULTI_CONFIG)
foreach(_config ${CMAKE_CONFIGURATION_TYPES})
file(GENERATE OUTPUT "${ctest_file_base}_include-${_config}.cmake" CONTENT "${ctest_include_content}" CONDITION $<CONFIG:${_config}>)
endforeach()
file(WRITE "${ctest_include_file}" "include(\"${ctest_file_base}_include-\${CTEST_CONFIGURATION_TYPE}.cmake\")")
else()
file(GENERATE OUTPUT "${ctest_file_base}_include.cmake" CONTENT "${ctest_include_content}")
file(WRITE "${ctest_include_file}" "include(\"${ctest_file_base}_include.cmake\")")
endif()
else()
message(SEND_ERROR "Unknown DISCOVERY_MODE: ${_DISCOVERY_MODE}")
endif()
# Add discovered tests to directory TEST_INCLUDE_FILES
set_property(DIRECTORY