mirror/CMake
1
0
Fork 0
mirror of https://github.com/Kitware/CMake.git synced 2026-02-21 14:40:26 -06:00
Code Issues Packages Projects Releases Wiki Activity

instrumentation: Store CDash settings in query files

Browse Source 
Adds new `cdashSubmit` and `cdashVerbose` options to allow enabling
instrumentation in CDash submissions using query files or the
`cmake_instrumentation` command.

Fixes: #26783, #26727
This commit is contained in:
Martin Duffy
2025-07-08 10:31:39 -04:00
parent afa94bae1e
commit 9dec460c8c
12 changed files with 155 additions and 96 deletions
Download Patch File Download Diff File Expand all files Collapse all files

45
Help/manual/cmake-instrumentation.7.rst
View File

@@ -100,7 +100,7 @@ Enabling Instrumentation for CDash Submissions
You can enable instrumentation when using CTest in :ref:`Dashboard Client`
mode by setting the :envvar:`CTEST_USE_INSTRUMENTATION` environment variable
to the current UUID for the ``CMAKE_EXPERIMENTAL_INSTRUMENTATION`` feature.
Doing so automatically enables the ``dynamicSystemInformation`` query.
Doing so automatically enables the ``dynamicSystemInformation`` option.
The following table shows how each type of instrumented command gets mapped
to a corresponding type of CTest XML file.
@@ -125,6 +125,9 @@ By default the command line reported to CDash is truncated at the first space.
You can instead choose to report the full command line (including arguments)
by setting :envvar:`CTEST_USE_VERBOSE_INSTRUMENTATION` to 1.
Alternatively, you can use the `v1 Query Files`_ to enable instrumentation for
CDash using the ``cdashSubmit`` and ``cdashVerbose`` options.
.. _`cmake-instrumentation API v1`:
API v1
@@ -146,8 +149,9 @@ subdirectories:
``query/generated/``
Holds query files generated by a CMake project with the
:command:`cmake_instrumentation` command. These files are owned by CMake and
are deleted and regenerated automatically during the CMake configure step.
:command:`cmake_instrumentation` command or the
:envvar:`CTEST_USE_INSTRUMENTATION` variable. These files are owned by CMake
and are deleted and regenerated automatically during the CMake configure step.
``data/``
Holds instrumentation data collected on the project. CMake owns all data
@@ -193,8 +197,9 @@ key is required, but all other fields are optional.
* ``postTest``
``options``
A list of strings specifying additional optional data to collect during
instrumentation. Elements in this list should be one of the following:
A list of strings used to enable certain optional behavior, including the
collection of certain additional data. Elements in this list should be one of
the following:
``staticSystemInformation``
Enables collection of the static information about the host machine CMake
@@ -207,13 +212,26 @@ key is required, but all other fields are optional.
generated by CMake, and includes information from immediately before and
after the command is executed.
``cdashSubmit``
Enables including instrumentation data in CDash. This does not
automatically enable ``dynamicSystemInformation``, but is otherwise
equivalent to having the :envvar:`CTEST_USE_INSTRUMENTATION` environment
variable enabled.
``cdashVerbose``
Enables including the full untruncated commands in data submitted to
CDash. Equivalent to having the
:envvar:`CTEST_USE_VERBOSE_INSTRUMENTATION` environment variable enabled.
The ``callbacks`` listed will be invoked during the specified hooks
*at a minimum*. When there are multiple query files, the ``callbacks``,
``hooks`` and ``options`` between them will be merged. Therefore, if any query
file includes any ``hooks``, every ``callback`` across all query files will be
executed at every ``hook`` across all query files. Additionally, if any query
file includes any optional ``options``, the optional query data will be present
in all data files.
file requests optional data using the ``options`` field, any related data will
be present in all snippet files. User written ``callbacks`` should be able to
handle the presence of this optional data, since it may be requested by an
unrelated query.
Example:
@@ -231,7 +249,8 @@ Example:
],
"options": [
"staticSystemInformation",
"dynamicSystemInformation"
"dynamicSystemInformation",
"cdashSubmit"
]
}
@@ -240,12 +259,14 @@ invocation, an index file ``index-<timestamp>.json`` will be generated in
``<build>/.cmake/instrumentation/v1/data`` containing a list of data snippet
files created since the previous indexing. The commands
``/usr/bin/python callback.py index-<timestamp>.json`` and
``/usr/bin/cmake -P callback.cmake arg index-<timestamp>.json`` will be executed in
that order. The index file will contain the ``staticSystemInformation`` data and
each snippet file listed in the index will contain the
``/usr/bin/cmake -P callback.cmake arg index-<timestamp>.json`` will be executed
in that order. The index file will contain the ``staticSystemInformation`` data
and each snippet file listed in the index will contain the
``dynamicSystemInformation`` data. Once both callbacks have completed, the index
file and all snippet files listed by it will be deleted from the project build
tree.
tree. The instrumentation data will be present in the XML files submitted to
CDash, but with truncated command strings because ``cdashVerbose`` was not
enabled.
.. _`cmake-instrumentation Data v1`:

14
Source/cmCTest.cxx
View File

@@ -114,7 +114,6 @@ struct cmCTest::Private
bool UseHTTP10 = false;
bool PrintLabels = false;
bool Failover = false;
bool UseVerboseInstrumentation = false;
cmJSONState parseState;
bool FlushTestProgressLine = false;
@@ -320,10 +319,6 @@ cmCTest::cmCTest()
this->Impl->TestProgressOutput = !cmIsOff(envValue);
}
envValue.clear();
if (cmSystemTools::GetEnv("CTEST_USE_VERBOSE_INSTRUMENTATION", envValue)) {
this->Impl->UseVerboseInstrumentation = !cmIsOff(envValue);
}
envValue.clear();
this->Impl->Parts[PartStart].SetName("Start");
this->Impl->Parts[PartUpdate].SetName("Update");
@@ -3629,11 +3624,6 @@ cmInstrumentation& cmCTest::GetInstrumentation()
return *this->Impl->Instrumentation;
}
bool cmCTest::GetUseVerboseInstrumentation() const
{
return this->Impl->UseVerboseInstrumentation;
}
void cmCTest::ConvertInstrumentationSnippetsToXML(cmXMLWriter& xml,
std::string const& subdir)
{
@@ -3662,6 +3652,8 @@ void cmCTest::ConvertInstrumentationSnippetsToXML(cmXMLWriter& xml,
bool cmCTest::ConvertInstrumentationJSONFileToXML(std::string const& fpath,
cmXMLWriter& xml)
{
bool verboseCommands = this->GetInstrumentation().HasOption(
cmInstrumentationQuery::Option::CDashVerbose);
Json::Value root;
this->Impl->parseState = cmJSONState(fpath, &root);
if (!this->Impl->parseState.errors.empty()) {
@@ -3710,7 +3702,7 @@ bool cmCTest::ConvertInstrumentationJSONFileToXML(std::string const& fpath,
}
// Truncate the full command line if verbose instrumentation
// was not requested.
if (key == "command" && !this->GetUseVerboseInstrumentation()) {
if (key == "command" && !verboseCommands) {
std::string command_str = root[key].asString();
std::string truncated = command_str.substr(0, command_str.find(' '));
if (command_str != truncated) {

1
Source/cmCTest.h
View File

@@ -436,7 +436,6 @@ public:
std::vector<std::string> GetCommandLineHttpHeaders() const;
cmInstrumentation& GetInstrumentation();
bool GetUseVerboseInstrumentation() const;
private:
int GenerateNotesFile(cmake* cm, std::string const& files);

142
Source/cmInstrumentation.cxx
View File

@@ -29,6 +29,53 @@
using LoadQueriesAfter = cmInstrumentation::LoadQueriesAfter;
std::map<std::string, std::string> cmInstrumentation::cdashSnippetsMap = {
{
"configure",
"configure",
},
{
"generate",
"configure",
},
{
"compile",
"build",
},
{
"link",
"build",
},
{
"custom",
"build",
},
{
"build",
"skip",
},
{
"cmakeBuild",
"build",
},
{
"cmakeInstall",
"build",
},
{
"install",
"build",
},
{
"ctest",
"build",
},
{
"test",
"test",
}
};
cmInstrumentation::cmInstrumentation(std::string const& binary_dir,
LoadQueriesAfter loadQueries)
{
@@ -38,6 +85,7 @@ cmInstrumentation::cmInstrumentation(std::string const& binary_dir,
this->binaryDir = binary_dir;
this->timingDirv1 =
cmStrCat(this->binaryDir, "/.cmake/instrumentation-", uuid, "/v1");
this->cdashDir = cmStrCat(this->timingDirv1, "/cdash");
if (cm::optional<std::string> configDir =
cmSystemTools::GetCMakeConfigDirectory()) {
this->userTimingDirv1 =
@@ -60,7 +108,10 @@ void cmInstrumentation::LoadQueries()
this->hasQuery = this->hasQuery ||
this->ReadJSONQueries(cmStrCat(this->userTimingDirv1, "/query"));
}
}
void cmInstrumentation::CheckCDashVariable()
{
std::string envVal;
if (cmSystemTools::GetEnv("CTEST_USE_INSTRUMENTATION", envVal) &&
!cmIsOff(envVal)) {
@@ -69,63 +120,23 @@ void cmInstrumentation::LoadQueries()
cmExperimental::Feature::Instrumentation)
.Uuid;
if (envVal == uuid) {
std::set<cmInstrumentationQuery::Option> options_ = {
cmInstrumentationQuery::Option::CDashSubmit,
cmInstrumentationQuery::Option::DynamicSystemInformation
};
if (cmSystemTools::GetEnv("CTEST_USE_VERBOSE_INSTRUMENTATION",
envVal) &&
!cmIsOff(envVal)) {
options_.insert(cmInstrumentationQuery::Option::CDashVerbose);
}
for (auto const& option : options_) {
this->AddOption(option);
}
std::set<cmInstrumentationQuery::Hook> hooks_ = {
cmInstrumentationQuery::Hook::PrepareForCDash
};
this->AddHook(cmInstrumentationQuery::Hook::PrepareForCDash);
this->AddOption(
cmInstrumentationQuery::Option::DynamicSystemInformation);
this->cdashDir = cmStrCat(this->timingDirv1, "/cdash");
cmSystemTools::MakeDirectory(this->cdashDir);
cmSystemTools::MakeDirectory(cmStrCat(this->cdashDir, "/configure"));
cmSystemTools::MakeDirectory(cmStrCat(this->cdashDir, "/build"));
cmSystemTools::MakeDirectory(
cmStrCat(this->cdashDir, "/build/commands"));
cmSystemTools::MakeDirectory(
cmStrCat(this->cdashDir, "/build/targets"));
cmSystemTools::MakeDirectory(cmStrCat(this->cdashDir, "/test"));
this->cdashSnippetsMap = { {
"configure",
"configure",
},
{
"generate",
"configure",
},
{
"compile",
"build",
},
{
"link",
"build",
},
{
"custom",
"build",
},
{
"build",
"skip",
},
{
"cmakeBuild",
"build",
},
{
"cmakeInstall",
"build",
},
{
"install",
"build",
},
{
"ctest",
"build",
},
{
"test",
"test",
} };
this->hasQuery = true;
this->WriteJSONQuery(options_, hooks_, {});
}
}
}
@@ -192,13 +203,9 @@ void cmInstrumentation::WriteJSONQuery(
for (auto const& callback : callbacks_) {
root["callbacks"].append(cmInstrumentation::GetCommandStr(callback));
}
cmsys::Directory d;
int n = 0;
if (d.Load(cmStrCat(this->timingDirv1, "/query/generated"))) {
n = (int)d.GetNumberOfFiles() - 2; // Don't count '.' or '..'
}
this->WriteInstrumentationJson(root, "query/generated",
cmStrCat("query-", n, ".json"));
this->WriteInstrumentationJson(
root, "query/generated",
cmStrCat("query-", this->writtenJsonQueries++, ".json"));
}
void cmInstrumentation::ClearGeneratedQueries()
@@ -306,7 +313,7 @@ int cmInstrumentation::CollectTimingData(cmInstrumentationQuery::Hook hook)
}
// Special case for CDash collation
if (this->HasHook(cmInstrumentationQuery::Hook::PrepareForCDash)) {
if (this->HasOption(cmInstrumentationQuery::Option::CDashSubmit)) {
this->PrepareDataForCDash(directory, index_path);
}
@@ -676,6 +683,13 @@ std::string const& cmInstrumentation::GetCDashDir()
void cmInstrumentation::PrepareDataForCDash(std::string const& data_dir,
std::string const& index_path)
{
cmSystemTools::MakeDirectory(this->cdashDir);
cmSystemTools::MakeDirectory(cmStrCat(this->cdashDir, "/configure"));
cmSystemTools::MakeDirectory(cmStrCat(this->cdashDir, "/build"));
cmSystemTools::MakeDirectory(cmStrCat(this->cdashDir, "/build/commands"));
cmSystemTools::MakeDirectory(cmStrCat(this->cdashDir, "/build/targets"));
cmSystemTools::MakeDirectory(cmStrCat(this->cdashDir, "/test"));
Json::Value root;
std::string error_msg;
cmJSONState parseState = cmJSONState(index_path, &root);

4
Source/cmInstrumentation.h
View File

@@ -33,6 +33,7 @@ public:
cmInstrumentation(std::string const& binary_dir,
LoadQueriesAfter loadQueries = LoadQueriesAfter::Yes);
void LoadQueries();
void CheckCDashVariable();
int InstrumentCommand(
std::string command_type, std::vector<std::string> const& command,
std::function<int()> const& callback,
@@ -91,7 +92,7 @@ private:
std::set<cmInstrumentationQuery::Hook> hooks;
std::vector<std::string> callbacks;
std::vector<std::string> queryFiles;
std::map<std::string, std::string> cdashSnippetsMap;
static std::map<std::string, std::string> cdashSnippetsMap;
Json::Value preTestStats;
std::string errorMsg;
bool hasQuery = false;
@@ -101,4 +102,5 @@ private:
std::unique_ptr<cmsys::SystemInformation> systemInformation;
cmsys::SystemInformation& GetSystemInformation();
#endif
int writtenJsonQueries = 0;
};

3
Source/cmInstrumentationQuery.cxx
View File

@@ -16,7 +16,8 @@
#include "cmStringAlgorithms.h"
std::vector<std::string> const cmInstrumentationQuery::OptionString{
"staticSystemInformation", "dynamicSystemInformation"
"staticSystemInformation", "dynamicSystemInformation", "cdashSubmit",
"cdashVerbose"
};
std::vector<std::string> const cmInstrumentationQuery::HookString{
"postGenerate", "preBuild", "postBuild",

4
Source/cmInstrumentationQuery.h
View File

@@ -15,7 +15,9 @@ public:
enum Option
{
StaticSystemInformation,
DynamicSystemInformation
DynamicSystemInformation,
CDashSubmit,
CDashVerbose
};
static std::vector<std::string> const OptionString;

1
Source/cmake.cxx
View File

@@ -2644,6 +2644,7 @@ int cmake::ActualConfigure()
cmStrCat(this->GetHomeOutputDirectory(), "/CMakeFiles"_s),
this->FileAPI->GetConfigureLogVersions());
this->Instrumentation->LoadQueries();
this->Instrumentation->CheckCDashVariable();
}
#endif

1
Tests/RunCMake/Instrumentation/query/cmake-command.cmake
View File

@@ -8,6 +8,7 @@
API_VERSION 1
DATA_VERSION 1
HOOKS postGenerate
OPTIONS cdashSubmit cdashVerbose
CALLBACK ${CMAKE_COMMAND} -E echo callback1
)
# Query 2

5
Tests/RunCMake/Instrumentation/query/generated/query-1.json.in
View File

@@ -7,6 +7,9 @@
[
"postGenerate"
],
"options" : [],
"options" : [
"cdashSubmit",
"cdashVerbose"
],
"version" : 1
}

17
Tests/RunCMake/ctest_instrumentation/InstrumentationInCTestXML-check.cmake
View File

@@ -1,3 +1,12 @@
set(timingDir "${RunCMake_TEST_BINARY_DIR}/.cmake/instrumentation-a37d1069-1972-4901-b9c9-f194aaf2b6e0/v1")
file(READ "${timingDir}/query/generated/query-0.json" jsonData)
string(JSON options GET "${jsonData}" options)
if (options MATCHES cdashVerbose AND NOT ${RunCMake_USE_VERBOSE_INSTRUMENTATION})
set(RunCMake_TEST_FAILED "cdashVerbose option not found in generated query despite environment variable")
elseif (NOT options MATCHES cdashVerbose AND ${RunCMake_USE_VERBOSE_INSTRUMENTATION})
set(RunCMake_TEST_FAILED "cdashVerbose option found in generated query despite environment variable")
endif()
foreach(xml_type Configure Build Test)
file(GLOB xml_file "${RunCMake_TEST_BINARY_DIR}/Testing/*/${xml_type}.xml")
if(xml_file)
@@ -29,6 +38,12 @@ foreach(xml_type Configure Build Test)
if(NOT xml_content MATCHES "<CmakeBuild")
set(RunCMake_TEST_FAILED "<CmakeBuild> element not found in Build.xml")
endif()
if(NOT RunCMake_USE_VERBOSE_INSTRUMENTATION AND NOT xml_content MATCHES "(truncated)")
set(RunCMake_TEST_FAILED "Commands not truncated despite cdashVerbose option")
endif()
if(verbose AND xml_content MATCHES "(truncated)")
set(RunCMake_TEST_FAILED "Commands truncated despite cdashVerbose option")
endif()
endif()
else()
set(RunCMake_TEST_FAILED "${xml_type}.xml not found")
@@ -37,7 +52,7 @@ endforeach()
foreach(dir_to_check "configure" "test" "build/targets" "build/commands")
file(GLOB leftover_cdash_snippets
"${RunCMake_TEST_BINARY_DIR}/.cmake/instrumentation-a37d1069-1972-4901-b9c9-f194aaf2b6e0/v1/cdash/${dir_to_check}/*")
"${timingDir}/cdash/${dir_to_check}/*")
if(leftover_cdash_snippets)
set(RunCMake_TEST_FAILED "Leftover snippets found in cdash dir: ${leftover_cdash_snippets}")
endif()

14
Tests/RunCMake/ctest_instrumentation/RunCMakeTest.cmake
View File

@@ -1,6 +1,13 @@
include(RunCTest)
function(run_InstrumentationInCTestXML USE_INSTRUMENTATION)
function(run_InstrumentationInCTestXML CASE_NAME USE_INSTRUMENTATION USE_VERBOSE_INSTRUMENTATION)
if(USE_VERBOSE_INSTRUMENTATION)
set(ENV{CTEST_USE_VERBOSE_INSTRUMENTATION} "1")
set(RunCMake_USE_VERBOSE_INSTRUMENTATION TRUE)
else()
set(ENV{CTEST_USE_VERBOSE_INSTRUMENTATION} "0")
set(RunCMake_USE_VERBOSE_INSTRUMENTATION FALSE)
endif()
if(USE_INSTRUMENTATION)
set(ENV{CTEST_USE_INSTRUMENTATION} "1")
set(ENV{CTEST_EXPERIMENTAL_INSTRUMENTATION} "a37d1069-1972-4901-b9c9-f194aaf2b6e0")
@@ -18,5 +25,6 @@ function(run_InstrumentationInCTestXML USE_INSTRUMENTATION)
unset(RunCMake_USE_LAUNCHERS)
unset(RunCMake_USE_INSTRUMENTATION)
endfunction()
run_InstrumentationInCTestXML(ON)
run_InstrumentationInCTestXML(OFF)
run_InstrumentationInCTestXML(InstrumentationInCTestXML ON OFF)
run_InstrumentationInCTestXML(VerboseInstrumentationInCTestXML ON ON)
run_InstrumentationInCTestXML(NoInstrumentationInCTestXML OFF OFF)