Help:test prop WILL_FAIL clarify and example

This makes the WILL_FAIL behavior explicit. Users could be confused
because some other build systems treat equivalent test properties
as allowing a failure, but still pass on a passing test, and may
catch SIGABRT unlike CMake.

We show an example of how to wrap a test so that sigabrt, segfault,
etc. are also handled by WILL_FAIL.
This commit is contained in:
scivision
2023-11-01 11:47:34 -04:00
parent 51d985da3e
commit 7b4983536a

View File

@@ -1,8 +1,35 @@
WILL_FAIL
---------
If set to true, this will invert the pass/fail flag of the test.
If ``true``, inverts the pass / fail test criteria. Tests for which
``WILL_FAIL`` is ``true`` fail with return code 0 and pass with non-zero
return code. Tests that exceed the timeout specified by :prop_test:`TIMEOUT`
still fail regardless of ``WILL_FAIL``.
This property can be used for tests that are expected to fail and return a
non-zero return code. Note that system-level test failures such as segmentation
faults or heap errors will still fail the test even if ``WILL_FALL`` is true.
Caveat: system-level test failures including segmentation faults,
signal abort, or heap errors may fail the test even if ``WILL_FAIL`` is true.
Example of a test that would ordinarily pass, but fails because ``WILL_FAIL``
is ``true``:
.. code-block:: cmake
add_test(NAME failed COMMAND ${CMAKE_COMMAND} -E true)
set_property(TEST failed PROPERTY WILL_FAIL true)
To run a test that may have a system-level failure, but still pass if
``WILL_FAIL`` is set, use a CMake command to wrap the executable run like:
.. code-block:: cmake
add_executable(main main.c)
add_test(NAME sigabrt COMMAND ${CMAKE_COMMAND} -E env $<TARGET_FILE:main>)
set_property(TEST sigabrt PROPERTY WILL_FAIL TRUE)
.. code-block:: c
#include <signal.h>
int main(void){ raise(SIGABRT); return 0; }