Files
CMake/Tests/RunCMake
Ryan Thornton 1ba4cb565e GoogleTest: Parameterize tests to check PRE_TEST/POST_BUILD discovery mode
Now, the unit tests are ran twice -- once with POST_BUILD (i.e. default mode)
and again with PRE_TEST (i.e. new discovery mode).

Both modes of setting gtest discovery mode are also tested:
1. Using the global override (i.e. CMAKE_GTEST_DISCOVER_TESTS_DISCOVERY_MODE)
2. Explicitly passing DISCOVERY_MODE in calls to gtest_discover_tests (in GoogleTestDiscoveryTimeout.cmake)

The goal is to show that the new PRE_TEST discovery mode does not break existing behavior
(i.e. should not break POST_BUILD mode) and should also pass the same tests
in the same way.

A few non trivial implementation details worth noting:

1. Refactoring discovery_timeout_test into own project

Originally, I tried doing:

```
run_GoogleTest(POST_BUILD)
run_GoogleTest(PRE_TEST)
```

Without changing the internal structure of run_GoogleTest.

But since discovery_timeout_test is part of the same project as the other tests,
and CTest include files always get evaluated and that's where test discovery occurs,
this means every other test now notices the timeout problem when running in PRE_TEST mode.

As a result, keeping the existing test structure meant that each existing test
(and any new test) would need to have its own PRE_TEST / POST_BUILD variant for stderr and stdout
in order to handle the case where discovery_timeout_test timed out.

This exponential increase in test output files introduced unnecessary complexity
and made it more cumbersome to work on test cases.

Why should an unrelated test case care about discovery_timeout_test?

So, to fix that issue, the tests were broken apart into two main groups:
1. run_GoogleTest_discovery_timeout (the test dealing with discovery_timeout_test)
2. run_GoogleTest (everything else)

This isolates the PRE_TEST / POST_BUILD timeout variants to a single test case.

And the other test cases remain unchanged -- further driving home the point that
DISCOVERY_MODE shouldn't change existing behavior.

2. Different number of PRE_TEST / POST_BUILD file variants

On the PRE_TEST path, different build systems / compilers (i.e. MSBuild and ninja/gcc)
produces different build output when building discovery_timeout_test,
but we don't actually care what it is, just as long as it builds
successfully.

This the fundamental difference in behavior between POST_BUILD (which would have failed)
and PRE_TEST (which doesn't) and is the reason why we don't need
a GoogleTest-discovery-build-result.txt or GoogleTest-discovery-build-stdout.txt

3. Fix flaky discovery timeout test

The test expects to see:

> Output:
>  timeout
>    case.

But sometimes, the test would only produce:

> Output:
>  timout

In certain environments, specifically when built with OpenWatcom 1.4,
and while the build server was under heavy load (i.e. running many tests in parallel),
std::endl behaves inconsistently and doesn't completely
flush std::cout when the program is terminated due to timeout.

This results in inconsistent test failures because the actual output
doesn't fully match what's expected.

At first we tried adding an additional:
    std::cout << std::flush

That didn't work. But using C-style printf() and fflush() appears to do
the trick:

> This time I managed to get on the machine while it was still busy doing other nightly builds
> and could reproduce the problem reliably. With that I was finally able to find a fix.
> It turns out my earlier hypothesis that C++ stream flushing was not working on the old compiler was correct,
> but even .flush() is not enough.
> I changed it to use C-style printf() and fflush() and now the test passes on that build.
> -- Brad King <brad.king@kitware.com>

Co-authored-by: Ryan Thornton <ThorntonRyan@JohnDeere.com>
Co-authored-by: Kevin Puetz <PuetzKevinA@JohnDeere.com>
2020-03-27 09:39:47 -05:00
..
2019-08-22 13:10:18 -04:00
2019-09-03 09:23:48 -04:00
2019-11-04 14:11:45 -05:00
2020-02-19 09:11:05 -05:00
2020-03-09 09:11:22 -04:00
2020-03-17 08:06:57 -04:00
2019-08-22 15:53:42 -04:00

This directory contains tests that run CMake to configure a project
but do not actually build anything.  To add a test:

1. Add a subdirectory named for the test, say ``<Test>/``.

2. In ``./CMakeLists.txt`` call ``add_RunCMake_test`` and pass the
   test directory name ``<Test>``.

3. Create script ``<Test>/RunCMakeTest.cmake`` in the directory containing::

    include(RunCMake)
    run_cmake(SubTest1)
    ...
    run_cmake(SubTestN)

   where ``SubTest1`` through ``SubTestN`` are sub-test names each
   corresponding to an independent CMake run and project configuration.

   One may also add calls of the form::

    run_cmake_command(SubTestI ${CMAKE_COMMAND} ...)

   to fully customize the test case command-line.

   Alternatively, if the test is to cover running ``ctest -S`` then use::

    include(RunCTest)
    run_ctest(SubTest1)
    ...
    run_ctest(SubTestN)

   and create ``test.cmake.in``, ``CTestConfig.cmake.in``, and
   ``CMakeLists.txt.in`` files to be configured for each case.

4. Create file ``<Test>/CMakeLists.txt`` in the directory containing::

    cmake_minimum_required(...)
    project(${RunCMake_TEST} NONE) # or languages needed
    include(${RunCMake_TEST}.cmake)

   where ``${RunCMake_TEST}`` is literal.  A value for ``RunCMake_TEST``
   will be passed to CMake by the ``run_cmake`` macro when running each
   sub-test.

5. Create a ``<Test>/<SubTest>.cmake`` file for each sub-test named
   above containing the actual test code.  Optionally create files
   containing expected test results:

   ``<SubTest>-result.txt``
    Regex matching expected process result, if not ``0``
   ``<SubTest>-stdout.txt``
    Regex matching expected stdout content
   ``<SubTest>-stderr.txt``
    Regex matching expected stderr content, if not ``^$``
   ``<SubTest>-check.cmake``
    Custom result check.

  Note that when a specific platform expects differing stdout or stderr that
  can be done by adding a platform specific output file. These follow the
  naming convention of:
   ``<SubTest>-stdout-<platform_lower_case>.txt``
   ``<SubTest>-stderr-<platform_lower_case>.txt``

   Note that trailing newlines will be stripped from actual and expected
   test output before matching against the stdout and stderr expressions.
   The code in ``<SubTest>-check.cmake`` may use variables

   ``RunCMake_TEST_SOURCE_DIR``
    Top of test source tree
   ``RunCMake_TEST_BINARY_DIR``
    Top of test binary tree

   and an failure must store a message in ``RunCMake_TEST_FAILED``.

To speed up local testing, you can choose to run only a subset of
``run_cmake()`` tests in a ``RunCMakeTest.cmake`` script by using the
``RunCMake_TEST_FILTER`` environment variable. If this variable is set,
it is treated as a regular expression, and any tests whose names don't
match the regular expression are not run. For example::

  $ RunCMake_TEST_FILTER="^example" ctest -R '^RunCMake\.Example$'

This will only run subtests in ``RunCMake.Example`` that start with
``example``.