From aff4dc47c4dc80595e4d8b75215a193bbcc41543 Mon Sep 17 00:00:00 2001 From: Brad King Date: Thu, 23 May 2024 14:27:01 -0400 Subject: [PATCH 1/6] Tests/RunCMake: Cross-reference README from main Tests/README --- Tests/README.rst | 4 ++++ Tests/RunCMake/README.rst | 16 ++++++++++++++-- 2 files changed, 18 insertions(+), 2 deletions(-) diff --git a/Tests/README.rst b/Tests/README.rst index 3e989382b8..2810fec4ef 100644 --- a/Tests/README.rst +++ b/Tests/README.rst @@ -29,3 +29,7 @@ are organized as follows. Tests that run CMake and/or other tools while precisely checking their return code and stdout/stderr content. Useful for testing error cases and diagnostic output. + + See `RunCMake/README.rst`_. + +.. _`RunCMake/README.rst`: RunCMake/README.rst diff --git a/Tests/RunCMake/README.rst b/Tests/RunCMake/README.rst index 8850d7ad69..9ff7ed2498 100644 --- a/Tests/RunCMake/README.rst +++ b/Tests/RunCMake/README.rst @@ -1,5 +1,17 @@ -This directory contains tests that run CMake to configure a project -but do not actually build anything. To add a test: +CMake Tests/RunCMake Directory +****************************** + +This directory contains tests that run CMake and/or other tools while +precisely checking their return code and stdout/stderr content. +The RunCMake infrastructure is useful for testing error cases and +diagnostic output. + +See also `../README.rst`_ and the `CMake Source Code Guide`_. + +.. _`../README.rst`: ../README.rst +.. _`CMake Source Code Guide`: ../../Help/dev/source.rst + +To add a test: 1. Add a subdirectory named for the test, say ``/``. From 0add7a802fef644814e2144344d7522116fcc962 Mon Sep 17 00:00:00 2001 From: Brad King Date: Thu, 23 May 2024 14:36:06 -0400 Subject: [PATCH 2/6] Tests/RunCMake: Organize README as multiple sections --- Tests/RunCMake/README.rst | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/Tests/RunCMake/README.rst b/Tests/RunCMake/README.rst index 9ff7ed2498..90fe12615e 100644 --- a/Tests/RunCMake/README.rst +++ b/Tests/RunCMake/README.rst @@ -10,12 +10,16 @@ See also `../README.rst`_ and the `CMake Source Code Guide`_. .. _`../README.rst`: ../README.rst .. _`CMake Source Code Guide`: ../../Help/dev/source.rst +.. _`CMakeLists.txt`: CMakeLists.txt + +Adding a Test +============= To add a test: 1. Add a subdirectory named for the test, say ``/``. -2. In ``./CMakeLists.txt`` call ``add_RunCMake_test`` and pass the +2. In `CMakeLists.txt`_ call ``add_RunCMake_test`` and pass the test directory name ````. 3. Create script ``/RunCMakeTest.cmake`` in the directory containing:: @@ -96,6 +100,14 @@ To add a test: The check script may optionally set ``RunCMake_TEST_FAILURE_MESSAGE`` with additional text to be included in the message if the test fails. +Running a Test +============== + +Each call to ``add_RunCMake_test(Example)`` in `CMakeLists.txt`_ creates +a test named ``RunCMake.Example`` that may be run with ``ctest``:: + + $ ctest -R "^RunCMake\.Example$" + To speed up local testing, you can choose to run only a subset of ``run_cmake()`` tests in a ``RunCMakeTest.cmake`` script by using the ``RunCMake_TEST_FILTER`` environment variable. If this variable is set, From c2f8a6729f9d328b9b824205b067dc8f5eda9612 Mon Sep 17 00:00:00 2001 From: Brad King Date: Thu, 23 May 2024 15:18:50 -0400 Subject: [PATCH 3/6] Tests/RunCMake: Rename "sub-test" terminology to "case" We've long used the latter term in commit messages and comments. --- Tests/RunCMake/README.rst | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/Tests/RunCMake/README.rst b/Tests/RunCMake/README.rst index 90fe12615e..29f5d87d2f 100644 --- a/Tests/RunCMake/README.rst +++ b/Tests/RunCMake/README.rst @@ -25,25 +25,25 @@ To add a test: 3. Create script ``/RunCMakeTest.cmake`` in the directory containing:: include(RunCMake) - run_cmake(SubTest1) + run_cmake(Case1) ... - run_cmake(SubTestN) + run_cmake(CaseN) - where ``SubTest1`` through ``SubTestN`` are sub-test names each - corresponding to an independent CMake run and project configuration. + where ``Case1`` through ``CaseN`` are case names each corresponding to + an independent CMake run and project configuration. One may also add calls of the form:: - run_cmake_command(SubTestI ${CMAKE_COMMAND} ...) + run_cmake_command(CaseI ${CMAKE_COMMAND} ...) to fully customize the test case command-line. Alternatively, if the test is to cover running ``ctest -S`` then use:: include(RunCTest) - run_ctest(SubTest1) + run_ctest(Case1) ... - run_ctest(SubTestN) + run_ctest(CaseN) and create ``test.cmake.in``, ``CTestConfig.cmake.in``, and ``CMakeLists.txt.in`` files to be configured for each case. @@ -66,30 +66,30 @@ To add a test: where ``${RunCMake_TEST}`` is literal. A value for ``RunCMake_TEST`` will be passed to CMake by the ``run_cmake`` macro when running each - sub-test. + case. -5. Create a ``/.cmake`` file for each sub-test named +5. Create a ``/.cmake`` file for each case named above containing the actual test code. Optionally create files containing expected test results: - ``-result.txt`` + ``-result.txt`` Regex matching expected process result, if not ``0`` - ``-stdout.txt`` + ``-stdout.txt`` Regex matching expected stdout content - ``-stderr.txt`` + ``-stderr.txt`` Regex matching expected stderr content, if not ``^$`` - ``-check.cmake`` + ``-check.cmake`` Custom result check. Note that when a specific platform expects differing stdout or stderr that can be done by adding a platform specific output file. These follow the naming convention of: - ``-stdout-.txt`` - ``-stderr-.txt`` + ``-stdout-.txt`` + ``-stderr-.txt`` Note that trailing newlines will be stripped from actual and expected test output before matching against the stdout and stderr expressions. - The code in ``-check.cmake`` may use variables + The code in ``-check.cmake`` may use variables ``RunCMake_TEST_SOURCE_DIR`` Top of test source tree @@ -116,7 +116,7 @@ match the regular expression are not run. For example:: $ RunCMake_TEST_FILTER="^example" ctest -R '^RunCMake\.Example$' -This will only run subtests in ``RunCMake.Example`` that start with +This will only run cases in ``RunCMake.Example`` that start with ``example``. To speed up the process of creating a new ``RunCMake`` test, you can run a From aebf6b8b29fb93163f028c9c6be71913b9f58c85 Mon Sep 17 00:00:00 2001 From: Brad King Date: Thu, 23 May 2024 15:51:10 -0400 Subject: [PATCH 4/6] Tests/RunCMake: Clarify documentation of platform-specific output matching --- Tests/RunCMake/README.rst | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/Tests/RunCMake/README.rst b/Tests/RunCMake/README.rst index 29f5d87d2f..d41c41acea 100644 --- a/Tests/RunCMake/README.rst +++ b/Tests/RunCMake/README.rst @@ -81,11 +81,8 @@ To add a test: ``-check.cmake`` Custom result check. - Note that when a specific platform expects differing stdout or stderr that - can be done by adding a platform specific output file. These follow the - naming convention of: - ``-stdout-.txt`` - ``-stderr-.txt`` + To specify platform-specific matches, create files of the form + ``-{stdout,stderr}-.txt``. Note that trailing newlines will be stripped from actual and expected test output before matching against the stdout and stderr expressions. From 61a08b0856a9aafc75b6cac755787cf05d86a34f Mon Sep 17 00:00:00 2001 From: Brad King Date: Thu, 23 May 2024 14:49:54 -0400 Subject: [PATCH 5/6] Tests/RunCMake: Document commands/variables available to RunCMakeTest scripts --- Tests/RunCMake/README.rst | 140 +++++++++++++++++++++++++++++++++++--- 1 file changed, 132 insertions(+), 8 deletions(-) diff --git a/Tests/RunCMake/README.rst b/Tests/RunCMake/README.rst index d41c41acea..412b2b0eae 100644 --- a/Tests/RunCMake/README.rst +++ b/Tests/RunCMake/README.rst @@ -86,17 +86,141 @@ To add a test: Note that trailing newlines will be stripped from actual and expected test output before matching against the stdout and stderr expressions. - The code in ``-check.cmake`` may use variables - - ``RunCMake_TEST_SOURCE_DIR`` - Top of test source tree - ``RunCMake_TEST_BINARY_DIR`` - Top of test binary tree - - and an failure must store a message in ``RunCMake_TEST_FAILED``. + The code in ``-check.cmake`` may use the `RunCMake Variables`_. + On failure the script must store a message in ``RunCMake_TEST_FAILED``. The check script may optionally set ``RunCMake_TEST_FAILURE_MESSAGE`` with additional text to be included in the message if the test fails. +RunCMake Commands +================= + +A ``RunCMakeTest.cmake`` script, after ``include(RunCMake)``, may use +the following commands. + +``run_cmake()`` + Run CMake or another command and check expected results described by + ``-{result,stdout,stderr}.txt`` and ``-check.cmake``. + The command is executed by a call of the form:: + + execute_process( + COMMAND ${RunCMake_TEST_COMMAND} ${RunCMake_TEST_OPTIONS} + WORKING_DIRECTORY "${RunCMake_TEST_COMMAND_WORKING_DIRECTORY}" + [TIMEOUT "${RunCMake_TEST_TIMEOUT}"] + ... + ) + + Behavior may be customized by setting `RunCMake Variables`_ before + the call. + +``run_cmake_command( ...)`` + Sets ``RunCMake_TEST_COMMAND`` to ``;...`` + and calls ``run_cmake()``. + + This is useful to run an arbitrary command. + +``run_cmake_script( ...)`` + Sets ``RunCMake_TEST_COMMAND`` to + ``${CMAKE_COMMAND};...;-P;${RunCMake_SOURCE_DIR}/.cmake`` + and calls ``run_cmake()``. + + This is useful to run CMake in script mode without configuring a project. + +``run_cmake_with_options( ...)`` + Sets ``RunCMake_TEST_OPTIONS`` to ``...`` + and calls ``run_cmake()``. + +``run_cmake_with_raw_args( "")`` + Calls ``run_cmake()`` with the underlying ``execute_process()`` + call extended with the content of ```` treated as literal source + code of CMake language command arguments:: + + execute_process( + COMMAND ${RunCMake_TEST_COMMAND} ${RunCMake_TEST_OPTIONS} + ... + ) + + This is useful to pass arguments to the test command that cannot be + encoded in CMake language ``;``-separated lists. + +RunCMake Variables +================== + +The behavior of `RunCMake Commands`_ such as ``run_cmake()`` may be +customized by setting the following variables before a call. + +``RunCMake_GENERATOR`` + CMake generator to use when configuring projects. + This provided to ``RunCMakeTest.cmake`` scripts automatically + when they are executed, based on the CMake generator used to + configure the test suite. + + For some generators, additional variables are also provided: + + ``RunCMake_GENERATOR_PLATFORM`` + Specifies the ``CMAKE_GENERATOR_PLATFORM``. + + ``RunCMake_GENERATOR_TOOLSET`` + Specifies the ``CMAKE_GENERATOR_TOOLSET``. + + ``RunCMake_GENERATOR_INSTANCE`` + Specifies the ``CMAKE_GENERATOR_INSTANCE``. + +``RunCMake_GENERATOR_IS_MULTI_CONFIG`` + Boolean value indicating whether ``${RunCMake_GENERATOR}`` is a + multi-config generator. + This provided to ``RunCMakeTest.cmake`` scripts automatically + when they are executed, based on the CMake generator used to + configure the test suite. + +``RunCMake_SOURCE_DIR`` + Absolute path to the ``Tests/RunCMake/`` directory in + the CMake source tree. This provided to ``RunCMakeTest.cmake`` + scripts automatically when they are executed. + +``RunCMake_BINARY_DIR`` + Absolute path to the ``Tests/RunCMake/`` directory in + the CMake binary tree. This provided to ``RunCMakeTest.cmake`` + scripts automatically when they are executed. + +``RunCMake_TEST_SOURCE_DIR`` + Absolute path to the individual test case's source tree. + If not set, defaults to ``${RunCMake_SOURCE_DIR}``. + +``RunCMake_TEST_BINARY_DIR`` + Absolute path to the individual test case's binary tree. + If not set, defaults to ``${RunCMake_BINARY_DIR}/-build``. + +``RunCMake_TEST_NO_CLEAN`` + Boolean value indicating whether ``run_cmake()`` should remove the + ``${RunCMake_TEST_BINARY_DIR}`` directory before running the test case. + If not set, or if set to a false value, the directory is removed. + +``RunCMake_TEST_COMMAND`` + The command for ``run_cmake()`` to execute. + If not set, defaults to running CMake to generate a project:: + + ${CMAKE_COMMAND} ${RunCMake_TEST_SOURCE_DIR} \ + -G ${RunCMake_GENERATOR} ... -DRunCMake_TEST= + +``RunCMake_TEST_COMMAND_WORKING_DIRECTORY`` + The working directory in which ``run_cmake()`` to execute its command. + If not set, defaults to ``${RunCMake_TEST_BINARY_DIR}``. + +``RunCMake_TEST_OPTIONS`` + Additional command-line options for ``run_cmake()`` to pass to + CMake when configuring a project with a default ``RunCMake_TEST_COMMAND``. + If not set, defaults to empty. + If ``RunCMake_TEST_COMMAND`` is set, ``RunCMake_TEST_OPTIONS`` is forced + to empty. + +``RunCMake_TEST_OUTPUT_MERGE`` + Boolean value indicating whether ``run_cmake()`` should redirect + the test process's ``stderr`` into its ``stdout``. + +``RunCMake_TEST_TIMEOUT`` + Specify a timeout, in seconds, for ``run_cmake()`` to pass to its + underlying ``execute_process()`` call using the ``TIMEOUT`` option. + Running a Test ============== From 0dbf2c24bf59d2c2363939790d651bf872d5d52e Mon Sep 17 00:00:00 2001 From: Brad King Date: Thu, 23 May 2024 15:54:35 -0400 Subject: [PATCH 6/6] Tests/RunCMake: Document how to write multi-step test cases --- Tests/RunCMake/README.rst | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/Tests/RunCMake/README.rst b/Tests/RunCMake/README.rst index 412b2b0eae..ea6db54d50 100644 --- a/Tests/RunCMake/README.rst +++ b/Tests/RunCMake/README.rst @@ -195,6 +195,8 @@ customized by setting the following variables before a call. ``${RunCMake_TEST_BINARY_DIR}`` directory before running the test case. If not set, or if set to a false value, the directory is removed. + This is useful to run `Multi-Step Test Cases`_. + ``RunCMake_TEST_COMMAND`` The command for ``run_cmake()`` to execute. If not set, defaults to running CMake to generate a project:: @@ -221,6 +223,30 @@ customized by setting the following variables before a call. Specify a timeout, in seconds, for ``run_cmake()`` to pass to its underlying ``execute_process()`` call using the ``TIMEOUT`` option. +Multi-Step Test Cases +===================== + +Normally each ``run_cmake()`` call corresponds to one standalone +test case with its own build tree. However, some test cases may require +multiple steps to be performed in a single build tree. This can be +achieved as follows:: + + block() + set(RunCMake_TEST_BINARY_DIR ${RunCMake_BINARY_DIR}/example-build) + run_cmake(example) + set(RunCMake_TEST_NO_CLEAN 1) + set(RunCMake_TEST_OUTPUT_MERGE 1) + run_cmake_command(example-build ${CMAKE_COMMAND} --build . --config Debug) + endblock() + +In this example, ``block() ... endblock()`` is used to isolate the +variable settings from later cases. A single build tree is used for +all cases inside the block. The first step cleans the build tree and +runs CMake to configure the case's project. The second step runs +``cmake --build`` to drive the generated build system and merges the +build tool's ``stderr`` into its ``stdout``. Note that each call uses +a unique case name so that expected results can be expressed individually. + Running a Test ==============