Merge topic 'doc-testing'

8a688e556e Help/dev: Add a CMake Testing Guide
7cc124d4b5 Help/dev: Move integration testing to a more-specific document
d9ab1fb5da Help/dev: Fix formatting in CMake Documentation Guide
aa3fffa4bb README: Drop outdated link on running CMake

Acked-by: Kitware Robot <kwrobot@kitware.com>
Acked-by: Juan Ramos <juan.ramos@kitware.com>
Merge-request: !9576

CONTRIBUTING.rst

@@ -26,7 +26,8 @@ To contribute patches:
 #. Fork the upstream `CMake Repository`_ into a personal account.
 #. Run `Utilities/SetupForDevelopment.sh`_ for local git configuration.
 #. See `Building CMake`_ for building CMake locally.
-#. See the `CMake Source Code Guide`_ for coding guidelines.
+#. See the `CMake Source Code Guide`_ for coding guidelines
+   and the `CMake Testing Guide`_ for testing instructions.
 #. Create a topic branch named suitably for your work.
    Base all new work on the upstream ``master`` branch.
    Base work on the upstream ``release`` branch only if it fixes a
@@ -49,6 +50,7 @@ The merge request will enter the `CMake Review Process`_ for consideration.
 .. _`Utilities/SetupForDevelopment.sh`: Utilities/SetupForDevelopment.sh
 .. _`Building CMake`: README.rst#building-cmake
 .. _`CMake Source Code Guide`: Help/dev/source.rst
+.. _`CMake Testing Guide`: Help/dev/testing.rst
 .. _`commit messages`: Help/dev/review.rst#commit-messages
 .. _`CMake Review Process`: Help/dev/review.rst
 
@@ -61,10 +63,10 @@ drive testing and submit results to the `CMake CDash Page`_.  Anyone is
 welcome to provide testing machines in order to help keep support for their
 platforms working.
 
-See documentation on `CMake Testing Process`_ for more information.
+See documentation on `CMake Integration Testing`_ for more information.
 
 .. _`CMake CDash Page`: https://open.cdash.org/index.php?project=CMake
-.. _`CMake Testing Process`: Help/dev/testing.rst
+.. _`CMake Integration Testing`: Help/dev/integration-testing.rst
 
 License
 =======

Help/dev/README.rst

@@ -23,12 +23,13 @@ branches and tags.  Upstream development processes are covered by the
 following documents:
 
 * The `CMake Review Process`_ manages integration of changes.
-* The `CMake Testing Process`_ drives integration testing.
+* The `CMake Integration Testing`_ infrastructure tests changes
+  before and after merging.
 
 .. _`Kitware's GitLab Instance`: https://gitlab.kitware.com
 .. _`CMake Repository`: https://gitlab.kitware.com/cmake/cmake
 .. _`CMake Review Process`: review.rst
-.. _`CMake Testing Process`: testing.rst
+.. _`CMake Integration Testing`: integration-testing.rst
 
 Developer Documentation
 =======================
@@ -37,10 +38,12 @@ CMake developer documentation is provided by the following documents:
 
 * The `CMake Source Code Guide`_.
 * The `CMake Documentation Guide`_.
+* The `CMake Testing Guide`_.
 * The `CMake Experimental Features Guide`_.
 
 .. _`CMake Source Code Guide`: source.rst
 .. _`CMake Documentation Guide`: documentation.rst
+.. _`CMake Testing Guide`: testing.rst
 .. _`CMake Experimental Features Guide`: experimental.rst
 
 Maintainer Documentation

Help/dev/documentation.rst

@@ -310,17 +310,17 @@ are suppressed inside of square- or angle-brackets.  This behavior can be
 controlled using the ``:break:`` option; note, however, that there is no way
 to *force* a line break.  The default value is 'smart'.  Allowable values are:
 
-  ``all``
-    Allow line breaks at any whitespace.
+``all``
+  Allow line breaks at any whitespace.
 
-  ``smart`` (default)
-    Allow line breaks at whitespace, except between matched square- or
-    angle-brackets.  For example, if a signature contains the text
-    ``<input>... [OUTPUT_VARIABLE <out-var>]``, a line break would be allowed
-    after ``<input>...`` but not between ``OUTPUT_VARIABLE`` and ``<out-var>``.
+``smart`` (default)
+  Allow line breaks at whitespace, except between matched square- or
+  angle-brackets.  For example, if a signature contains the text
+  ``<input>... [OUTPUT_VARIABLE <out-var>]``, a line break would be allowed
+  after ``<input>...`` but not between ``OUTPUT_VARIABLE`` and ``<out-var>``.
 
-  ``verbatim``
-    Allow line breaks only where the source document contains a newline.
+``verbatim``
+  Allow line breaks only where the source document contains a newline.
 
 The directive treats its content as the documentation of the signature(s).
 Indent the signature documentation accordingly.

Help/dev/integration-testing.rst

@@ -0,0 +1,46 @@
+CMake Integration Testing
+*************************
+
+The following documents how to run integration testing builds.
+See documentation on `CMake Development`_ for more information.
+
+See the `CMake Testing Guide`_ for running the test suite locally.
+
+.. _`CMake Development`: README.rst
+.. _`CMake Testing Guide`: testing.rst
+
+CMake Dashboard Scripts
+=======================
+
+The *integration testing* step of the `CMake Review Process`_ uses a set of
+testing machines that follow an integration branch on their own schedule to
+drive testing and submit results to the `CMake CDash Page`_.  Anyone is
+welcome to provide testing machines in order to help keep support for their
+platforms working.
+
+The `CMake Dashboard Scripts Repository`_ provides CTest scripts to drive
+nightly, continuous, and experimental testing of CMake.  Use the following
+commands to set up a new integration testing client:
+
+.. code-block:: console
+
+  $ mkdir -p ~/Dashboards
+  $ cd ~/Dashboards
+  $ git clone https://gitlab.kitware.com/cmake/dashboard-scripts.git CMakeScripts
+  $ cd CMakeScripts
+
+The `cmake_common.cmake`_ script contains comments at the top with
+instructions to set up a testing client.  As it instructs, create a
+CTest script with local settings and include ``cmake_common.cmake``.
+
+.. _`CMake Review Process`: review.rst
+.. _`CMake CDash Page`: https://open.cdash.org/index.php?project=CMake
+.. _`CMake Dashboard Scripts Repository`: https://gitlab.kitware.com/cmake/dashboard-scripts
+.. _`cmake_common.cmake`: https://gitlab.kitware.com/cmake/dashboard-scripts/-/blob/master/cmake_common.cmake
+
+Nightly Start Time
+------------------
+
+The ``cmake_common.cmake`` script expects its includer to be run from a
+nightly scheduled task (cron job).  Schedule such tasks for sometime after
+``1:00am UTC``, the time at which our nightly testing branches fast-forward.

Help/dev/review.rst

@@ -348,7 +348,7 @@ Integration Testing
 
 The above `topic testing`_ tests the MR topic independent of other
 merge requests and on only a few key platforms and configurations.
-The `CMake Testing Process`_ also has a large number of machines
+`CMake Integration Testing`_ also uses a large number of machines
 provided by Kitware and generous volunteers that cover nearly all
 supported platforms, generators, and configurations.  In order to
 avoid overwhelming these resources, they do not test every MR
@@ -403,7 +403,7 @@ until one of the following occurs:
 Once a MR has been removed from the topic stage a new ``Do: stage``
 command is needed to stage it again.
 
-.. _`CMake Testing Process`: testing.rst
+.. _`CMake Integration Testing`: integration-testing.rst
 
 Resolve
 =======

Help/dev/source.rst

@@ -310,6 +310,7 @@ The CMake source tree is organized as follows.
 
 * ``Tests/``:
   The test suite.  See `Tests/README.rst`_.
+  To run the tests, see the `CMake Testing Guide`_.
 
 * ``Utilities/``:
   Scripts, third-party source code.
@@ -331,5 +332,6 @@ The CMake source tree is organized as follows.
     See `Utilities/Release/README.rst`_.
 
 .. _`CMake Documentation Guide`: documentation.rst
+.. _`CMake Testing Guide`: testing.rst
 .. _`Tests/README.rst`: ../../Tests/README.rst
 .. _`Utilities/Release/README.rst`: ../../Utilities/Release/README.rst

Help/dev/testing.rst

@@ -1,43 +1,111 @@
-CMake Testing Process
-*********************
+CMake Testing Guide
+*******************
 
-The following documents the process for running integration testing builds.
+The following is a guide to the CMake test suite for developers.
 See documentation on `CMake Development`_ for more information.
 
+See `CMake Integration Testing`_ for running integration testing builds.
+
+See `Tests/README.rst`_ for the test suite layout in the source tree.
+
 .. _`CMake Development`: README.rst
+.. _`CMake Integration Testing`: integration-testing.rst
+.. _`Tests/README.rst`: ../../Tests/README.rst
 
-CMake Dashboard Scripts
-=======================
+Running Tests in the Build Tree
+===============================
 
-The *integration testing* step of the `CMake Review Process`_ uses a set of
-testing machines that follow an integration branch on their own schedule to
-drive testing and submit results to the `CMake CDash Page`_.  Anyone is
-welcome to provide testing machines in order to help keep support for their
-platforms working.
+After `Building CMake`_, one may run the test suite in the build tree
+using `ctest(1)`_:
 
-The `CMake Dashboard Scripts Repository`_ provides CTest scripts to drive
-nightly, continuous, and experimental testing of CMake.  Use the following
-commands to set up a new integration testing client:
+* With a single-configuration CMake generator, such as ``Ninja``
+  or ``Unix Makefiles``, one may simply run ``ctest``:
+
+  .. code-block:: console
+
+    $ ctest
+
+* With a multi-configuration CMake generator, such as
+  ``Ninja Multi-Config``, ``Visual Studio``, or ``Xcode``,
+  one must tell ``ctest`` which configuration to test
+  by passing the ``-C <config>`` option:
+
+  .. code-block:: console
+
+    $ ctest -C Debug
+
+Some useful `ctest(1)`_ options include:
+
+``-N``
+  List test names without running them.
+
+``-V``
+  Show verbose output from each test.
+
+``-j <N>``
+  Run to run up to ``N`` tests concurrently.
+
+``-R <regex>``
+  Select tests for which the regular expression matches a substring
+  of their name.
+
+Cleaning Test Build Trees
+-------------------------
+
+Many CMake tests create their own test project build trees underneath
+the ``Tests/`` directory at the top of the CMake build tree.  These
+build trees are left behind after testing completes in order to
+facilitate manual investigation of results.  Many of the tests do *not*
+clean their build trees if they are run again, with the exception of
+tests using the `RunCMake`_ infrastructure.
+
+In order to clear test build trees, drive the ``test_clean`` custom target
+in the CMake build tree:
 
 .. code-block:: console
 
-  $ mkdir -p ~/Dashboards
-  $ cd ~/Dashboards
-  $ git clone https://gitlab.kitware.com/cmake/dashboard-scripts.git CMakeScripts
-  $ cd CMakeScripts
+  $ cmake --build . --target test_clean
 
-The `cmake_common.cmake`_ script contains comments at the top with
-instructions to set up a testing client.  As it instructs, create a
-CTest script with local settings and include ``cmake_common.cmake``.
+This removes the ``Tests/`` subdirectories created by individual tests
+so they will use a fresh directory next time they run.
 
-.. _`CMake Review Process`: review.rst
-.. _`CMake CDash Page`: https://open.cdash.org/index.php?project=CMake
-.. _`CMake Dashboard Scripts Repository`: https://gitlab.kitware.com/cmake/dashboard-scripts
-.. _`cmake_common.cmake`: https://gitlab.kitware.com/cmake/dashboard-scripts/-/blob/master/cmake_common.cmake
+.. _`Building CMake`: ../../README.rst#building-cmake
+.. _`ctest(1)`: https://cmake.org/cmake/help/latest/manual/ctest.1.html
+.. _`RunCMake`: ../../Tests/RunCMake/README.rst
 
-Nightly Start Time
-------------------
+Running Tests with a Different Generator
+========================================
 
-The ``cmake_common.cmake`` script expects its includer to be run from a
-nightly scheduled task (cron job).  Schedule such tasks for sometime after
-``1:00am UTC``, the time at which our nightly testing branches fast-forward.
+After `Building CMake`_ with one CMake generator, one may configure the
+test suite using a different generator in a separate build tree, without
+building CMake itself again, by defining ``CMake_TEST_EXTERNAL_CMAKE``
+to be the absolute path to the ``bin`` directory containing the ``cmake``,
+``ctest``, and ``cpack`` executables.
+
+For example, after building CMake with the ``Ninja`` generator:
+
+.. code-block:: console
+
+  $ cmake -B build-ninja -G Ninja -DCMAKE_BUILD_TYPE=Debug
+  $ cmake --build build-ninja
+
+one may configure a second build tree to drive tests with the
+``Ninja Multi-Config`` generator:
+
+.. code-block:: console
+
+  $ cmake -B build-nmc-tests -G "Ninja Multi-Config" \
+    -DCMake_TEST_EXTERNAL_CMAKE="$PWD/build-ninja/bin"
+  $ cmake --build build-nmc-tests --config Release
+
+The second build tree does not build CMake itself, but does configure
+the test suite and build test binaries.  One may then run tests normally:
+
+.. code-block:: console
+
+  $ cd build-nmc-tests
+  $ ctest -C Release
+
+Note that the configuration with which one drives tests in the second
+build tree is independent of the configuration with which CMake was
+built in the first.

README.rst

@@ -51,18 +51,19 @@ Building CMake with CMake
 -------------------------
 
 You can build CMake as any other project with a CMake-based build system:
-run the installed CMake on the sources of this CMake with your preferred
-options and generators. Then build it and install it.
-For instructions how to do this, see documentation on `Running CMake`_.
-
-.. _`Running CMake`: https://cmake.org/runningcmake
+run an already-installed CMake on this source tree with your preferred
+generator and options.  Then build it and install it.
 
 To build the documentation, install `Sphinx`_ and configure CMake with
 ``-DSPHINX_HTML=ON`` and/or ``-DSPHINX_MAN=ON`` to enable the "html" or
 "man" builder.  Add ``-DSPHINX_EXECUTABLE=/path/to/sphinx-build`` if the
 tool is not found automatically.
 
+To run the test suite, run ``ctest`` in the CMake build directory after
+building.  See the `CMake Testing Guide`_ for details.
+
 .. _`Sphinx`: https://sphinx-doc.org
+.. _`CMake Testing Guide`: Help/dev/testing.rst
 
 Building CMake from Scratch
 ---------------------------

Tests/README.rst

@@ -2,8 +2,9 @@ CMake Tests Directory
 *********************
 
 This directory contains the CMake test suite.
-See also the `CMake Source Code Guide`_.
+See also the `CMake Testing Guide`_ and the `CMake Source Code Guide`_.
 
+.. _`CMake Testing Guide`: ../Help/dev/testing.rst
 .. _`CMake Source Code Guide`: ../Help/dev/source.rst
 
 Many tests exist as immediate subdirectories, but some tests

Tests/RunCMake/README.rst

@@ -6,9 +6,11 @@ precisely checking their return code and stdout/stderr content.
 The RunCMake infrastructure is useful for testing error cases and
 diagnostic output.
 
-See also `../README.rst`_ and the `CMake Source Code Guide`_.
+See also `../README.rst`_, the `CMake Testing Guide`_,
+and the `CMake Source Code Guide`_.
 
 .. _`../README.rst`: ../README.rst
+.. _`CMake Testing Guide`: ../../Help/dev/testing.rst
 .. _`CMake Source Code Guide`: ../../Help/dev/source.rst
 .. _`CMakeLists.txt`: CMakeLists.txt