From b60789a758ddd755be635ae60b24364ecf7fdd34 Mon Sep 17 00:00:00 2001 From: Zack Galbreath Date: Wed, 21 Apr 2021 14:05:11 -0400 Subject: [PATCH] Help: Document CTest custom test measurements --- Help/command/ctest_test.rst | 80 +++++++++++++++++++++++++++++++++++++ Help/manual/ctest.1.rst | 2 + 2 files changed, 82 insertions(+) diff --git a/Help/command/ctest_test.rst b/Help/command/ctest_test.rst index 9081b3f4b1..c61d01e5d1 100644 --- a/Help/command/ctest_test.rst +++ b/Help/command/ctest_test.rst @@ -170,3 +170,83 @@ The options are: See also the :variable:`CTEST_CUSTOM_MAXIMUM_PASSED_TEST_OUTPUT_SIZE` and :variable:`CTEST_CUSTOM_MAXIMUM_FAILED_TEST_OUTPUT_SIZE` variables. + +.. _`Additional Test Measurements`: + +Additional Test Measurements +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +CTest can parse the output of your tests for extra measurements to report +to CDash. + +When run as a :ref:`Dashboard Client`, CTest will include these custom +measurements in the ``Test.xml`` file that gets uploaded to CDash. + +Check the `CDash test measurement documentation +`_ +for more information on the types of test measurements that CDash recognizes. + +The following example demonstrates how to output a variety of custom test +measurements. + +.. code-block:: c++ + + std::cout << + "28.3" + << std::endl; + + std::cout << + "red" + << std::endl; + + std::cout << + "https://cmake.org" + << std::endl; + + std::cout << + "" << + "line 1.\n" << + " \033[31;1m line 2. Bold red, and indented!\033[0;0ml\n" << + "line 3. Not bold or indented...\n" << + "" << std::endl; + +Image Measurements +"""""""""""""""""" + +The following example demonstrates how to upload test images to CDash. + +.. code-block:: c++ + + std::cout << + "" << + "/dir/to/test_img.jpg" << std::endl; + + std::cout << + "" << + "/dir/to/valid_img.gif" << std::endl; + + std::cout << + " << + "/dir/to/img.png" + << std::endl; + +Images will be displayed together in an interactive comparison mode on CDash +if they are provided with two or more of the following names. + +* ``TestImage`` +* ``ValidImage`` +* ``BaselineImage`` +* ``DifferenceImage2`` + +By convention, ``TestImage`` is the image generated by your test, and +``ValidImage`` (or ``BaselineImage``) is basis of comparison used to determine +if the test passed or failed. + +If another image name is used it will be displayed by CDash as a static image +separate from the interactive comparison UI. + +Attached Files +"""""""""""""" + +To associate other types of files with a test, use the +:prop_test:`ATTACHED_FILES` or :prop_test:`ATTACHED_FILES_ON_FAIL` test properties. diff --git a/Help/manual/ctest.1.rst b/Help/manual/ctest.1.rst index e35e162d8b..811997c554 100644 --- a/Help/manual/ctest.1.rst +++ b/Help/manual/ctest.1.rst @@ -1095,6 +1095,8 @@ Configuration settings include: * `CTest Script`_ variable: :variable:`CTEST_TEST_TIMEOUT` * :module:`CTest` module variable: ``DART_TESTING_TIMEOUT`` +To report extra test values to CDash, see :ref:`Additional Test Measurements`. + .. _`CTest Coverage Step`: CTest Coverage Step