CTest normally terminates test processes on timeout using `SIGKILL`.
Offer tests a chance to exit gracefully, on platforms supporting POSIX
signals, by setting `TIMEOUT_SIGNAL_{NAME,GRACE_PERIOD}` properties.
Fixes: #17288
For policy-specific tests, use the version before the policy was
introduced. Otherwise, use 3.5 where possible.
Also, remove `cmake_minimum_required()` and `project()` calls from
individual cases where they are handled by `CMakeLists.txt`.
Some RunCMake tests unset some environment variables that affect ctest's
behavior before running it. Any test using the RunCTest helper needs to
do this, so do it in the helper itself.
Issue: #24153
The `ctest_test` and `CTestCommandLine` truncation tests had multiple problems:
- escape expected result string to avoid regex matching
- specify the truncation size
- pass the truncation mode correctly into the test definition
- use unique test names
Issue: #23868
Add `--test-output-truncation` to `ctest`. This option can be used to
customize which part of the test output is being truncated. Currently
supported values are `tail`, `middle` and `head`.
Also add equivalent `CTEST_CUSTOM_TEST_OUTPUT_TRUNCATION` variable.
Fixes: #23206
Check for the existence of the Binary directory string before checking
for coverage files. If the directory is not specified, write an error
message and do not do any checking.
Fixes: #22102
In commit 49948f7221 (ctest_memcheck: Add support for ThreadSanitizer,
2014-07-07, v3.1.0-rc1~322^2~1) and commit 1e005eadbc (CTest: Fix
MemoryCheckType from 'ctest -T MemCheck', 2014-07-15, v3.1.0-rc1~298^2),
the `CMakeCommand` internal setting was left set only when using `ctest
-S` scripts. Instead simply use CTest's corresponding CMake directly
without passing it through an internal setting.
Fixes: #20584
If no tests were found, the default behavior of CTest is to always log an
error message but to return an error code in script mode only. This option
unifies the behavior of CTest by either returning an error code if no tests
were found or by ignoring it.
Signed-off-by: Stefan Dinkelacker <s.dinkelacker@dkfz-heidelberg.de>
Combine `--repeat-until-fail`, `--repeat-until-pass`, and
`--repeat-after-timeout` to create a single `--repeat <mode>:<n>`
option. Retain `--repeat-until-fail` too because that has been
available in previous releases.
The `PROCESSES` test property name added for CMake 3.16 is too close to
the existing `PROCESSORS` test property. Furthermore, the property in
principle specifies groups of resources organized in a way that is
meaningful to a particular test. The groups may often correspond to
processes but they could have other meanings. Since the property name
`PROCESSES` has not been in a final 3.16 release yet, simply rename it
to `RESOURCE_GROUPS`.
Fixes: #19914
A case added by commit 39ac8b4eb5 (ctest: Add --repeat-after-timeout
option, 2019-10-29) includes a success-case timeout of 2 seconds. That
is too short when running under valgrind, so extend it to 5 seconds.
For pass regex, display only the element that was found, rather
than all elements
Rename loop variable for fail regex, from pass to fail
For consistency, add space in output for pass
Add tests that find and don't find PASS_REGULAR_EXPRESSION, and a test
that finds FAIL_REGULAR_EXPRESSION, whose LastTest.log files are checked
using *-check.cmake.
Avoid exposing the internal special value that we use to track tests not
available in the tested configuration. This also prevents clients from
having to do the filtering themselves.
Since commit v3.11.0-rc1~117^2 (CTest: Re-implement test process
handling using libuv, 2017-12-10) we do not give the child test
processes any stdin. Prior to that change we let the child test
processes inherit stdin from ctest itself. Tests that run serially
might be able to use the real stdin meaningfully, so restore that
behavior and add a test case.
Fixes: #18591
The `ctest --test-load` option is implemented in `StartNextTests` by not
starting any tests when the load is too high and instead sleeping and
then returning. Prior to commit v3.11.0-rc1~117^2 (CTest: Re-implement
test process handling using libuv, 2017-12-10) our outer loop in
`RunTests` would immediately call `StartNextTests` again. However, now
the `uv_run` loop may simply terminate if there are no tests running
because no events are left pending.
Fix this by converting the sleep in `StartNextTests` into a libuv timer
that it starts instead. This avoids leaving `uv_run` with no pending
events. In the case that there are other running tests this also allows
CTest to detect when they finish even if it during the wait period where
we previously slept.
This regression was not caught by the test suite because it only
verified that we do not start new tests when the load was too high and
not that we proceed to start tests when the load drops. Revise the test
suite to cover both.
Fixes: #18338
In commit v2.8.0~170 (ENH: Added ctest test options PROCESSORS and
RUN_SERIAL, 2009-09-07) CTest learned to track the number of processors
allocated to running tests in order to balance it against the desired
level of parallelism. Extend this idea by introducing a new
`PROCESSOR_AFFINITY` test property to ask that CTest run a test
with the CPU affinity mask set. This will allow a set of tests
that are running concurrently to use disjoint CPU resources.
This commit changes the times reported by labels and subprojects to be
weighted by the PROCESSORS property. It is reported with `sec*proc`
instead of just `sec`.
Refactoring in commit v3.4.0-rc1~390^2~1 (cmCTestMultiProcessHandler:
Refactor RUN_SERIAL implementation, 2015-06-01) forgot to update a code
path for cleaning up after a failed RUN_SERIAL test. This causes an
infinite loop after a RUN_SERIAL test fails. Fix it and add a test.
Add ctest command-line options:
--test-output-size-passed <n>
--test-output-size-failed <n>
to set the amount of test output to store in Test.xml as a command-line
dashboard client.
Add a TestLoad setting to CTest that can be set via a new --test-load
command-line option, CTEST_TEST_LOAD variable, or TEST_LOAD option to
the ctest_test command. Teach cmCTestMultiProcessHandler to measure
the CPU load and avoid starting tests that may take more than the
spare load currently available. The expression
<current_load> + <test_processors> <= <max-load>
must be true to start a new test.
Co-Author: Zack Galbreath <zack.galbreath@kitware.com>
Use the KWSys Process "MergeOutput" option to give each test child
process the same pipe for both stdout and stderr. This allows natural
merging of stdout and stderr together instead of merging on arbitrary
buffered read boundaries as before.
