Help: Override pygments CMakeLexer to support <..> and [..]

* The code snippets in the docs consist of CMake code mixed
  with syntax definition punctuation like < > [ ] ... Therefore
  a pure CMake lexer is inadequate. Here it is replaced by a
  CMake syntax definition parser.
* Fixed syntax definition snippets in FindPkgConfig.cmake to
  make best use of syntax highlighting. This source file is the
  hardest to support because it contains comparison operators
  <= = >=, which need special attention to avoid confusion
  with the placeholder indicators <...>.
* Fixed syntax in execute_process.rst (there were unbalanced
  brackets).
* Disabled syntax highlighting for long string examples in
  cmake-language.7.rst.
* No highlighting of removed syntax in CMP0049
* To inspect the outcome of this patch, see e.g. the pages
  * manual/cmake-buildsystem.7.html
  * module/ExternalProject.html
  * module/FindPkgConfig.html
  which are particularly rich in complex code snippets.

Help/command/execute_process.rst

@@ -5,8 +5,8 @@ Execute one or more child processes.
 
 .. code-block:: cmake
 
-  execute_process(COMMAND <cmd1> [args1...]]
-                  [COMMAND <cmd2> [args2...] [...]]
+  execute_process(COMMAND <cmd1> [<arguments>]
+                  [COMMAND <cmd2> [<arguments>]]...
                   [WORKING_DIRECTORY <directory>]
                   [TIMEOUT <seconds>]
                   [RESULT_VARIABLE <variable>]

Help/manual/cmake-language.7.rst

@@ -206,9 +206,10 @@ enclosed content, such as `Escape Sequences`_ or `Variable References`_,
 is performed.  A bracket argument is always given to the command
 invocation as exactly one argument.
 
-For example:
+.. No code-block syntax highlighting in the following example
+   (long string literal not supported by our cmake.py)
 
-.. code-block:: cmake
+For example::
 
  message([=[
  This is the first line in a bracket argument with bracket length 1.
@@ -253,9 +254,10 @@ closing quotes.  Both `Escape Sequences`_ and `Variable References`_
 are evaluated.  A quoted argument is always given to the command
 invocation as exactly one argument.
 
-For example:
+.. No code-block syntax highlighting in the following example
+   (escape \" not supported by our cmake.py)
 
-::
+For example::
 
  message("This is a quoted argument containing multiple lines.
  This is always one argument even though it contains a ; character.
@@ -264,11 +266,12 @@ For example:
  It does end in an unescaped double quote.
  ")
 
+.. No code-block syntax highlighting in the following example
+   (for conformity with the two above examples)
+
 The final ``\`` on any line ending in an odd number of backslashes
 is treated as a line continuation and ignored along with the
-immediately following newline character.  For example:
-
-.. code-block:: cmake
+immediately following newline character.  For example::
 
  message("\
  This is the first line of a quoted argument. \

Help/policy/CMP0049.rst

@@ -3,14 +3,14 @@ CMP0049
 
 Do not expand variables in target source entries.
 
-CMake 2.8.12 and lower performed and extra layer of variable expansion
-when evaluating source file names:
-
-.. code-block:: cmake
+CMake 2.8.12 and lower performed an extra layer of variable expansion
+when evaluating source file names::
 
   set(a_source foo.c)
   add_executable(foo \${a_source})
 
+.. note: no cmake highlighting since this syntax is deprecated
+
 This was undocumented behavior.
 
 The OLD behavior for this policy is to expand such variables when processing

Modules/ExternalProject.cmake

@@ -16,7 +16,9 @@ External Project Definition
 
   The ``ExternalProject_Add()`` function creates a custom target to drive
   download, update/patch, configure, build, install and test steps of an
-  external project::
+  external project:
+
+  .. code-block:: cmake
 
     ExternalProject_Add(<name> [<option>...])
 
@@ -608,7 +610,9 @@ External Project Definition
       appended to them by following them with as many ``COMMAND ...`` options
       as needed
       (:manual:`generator expressions <cmake-generator-expressions(7)>` are
-      supported). For example::
+      supported). For example:
+
+      .. code-block:: cmake
 
         ExternalProject_Add(example
           ... # Download options, etc.
@@ -627,7 +631,9 @@ Obtaining Project Properties
 .. command:: ExternalProject_Get_Property
 
   The ``ExternalProject_Get_Property()`` function retrieves external project
-  target properties::
+  target properties:
+
+  .. code-block:: cmake
 
     ExternalProject_Get_Property(<name> <prop1> [<prop2>...])
 
@@ -655,7 +661,9 @@ control needed to implement such step-level capabilities.
 
   The ``ExternalProject_Add_Step()`` function specifies an additional custom
   step for an external project defined by an earlier call to
-  :command:`ExternalProject_Add`::
+  :command:`ExternalProject_Add`:
+
+  .. code-block:: cmake
 
     ExternalProject_Add_Step(<name> <step> [<option>...])
 
@@ -722,7 +730,9 @@ control needed to implement such step-level capabilities.
 
   The ``ExternalProject_Add_StepTargets()`` function generates targets for the
   steps listed. The name of each created target will be of the form
-  ``<name>-<step>``::
+  ``<name>-<step>``:
+
+  .. code-block:: cmake
 
     ExternalProject_Add_StepTargets(<name> [NO_DEPENDS] <step1> [<step2>...])
 
@@ -773,7 +783,9 @@ control needed to implement such step-level capabilities.
   The ``ExternalProject_Add_StepDependencies()`` function can be used to add
   dependencies to a step. The dependencies added must be targets CMake already
   knows about (these can be ordinary executable or library targets, custom
-  targets or even step targets of another external project)::
+  targets or even step targets of another external project):
+
+  .. code-block:: cmake
 
     ExternalProject_Add_StepDependencies(<name> <step> <target1> [<target2>...])
 

Modules/FindPkgConfig.cmake

@@ -88,7 +88,9 @@ endmacro()
 .. command:: pkg_get_variable
 
   Retrieves the value of a pkg-config variable ``varName`` and stores it in the
-  result variable ``resultVar`` in the calling scope. ::
+  result variable ``resultVar`` in the calling scope.
+
+  .. code-block:: cmake
 
     pkg_get_variable(<resultVar> <moduleName> <varName>)
 
@@ -514,7 +516,9 @@ endmacro()
 .. command:: pkg_check_modules
 
   Checks for all the given modules, setting a variety of result variables in
-  the calling scope. ::
+  the calling scope.
+
+  .. code-block:: cmake
 
     pkg_check_modules(<prefix>
                       [REQUIRED] [QUIET]
@@ -552,10 +556,10 @@ endmacro()
   - ``foo>=3.1`` matches any version from 3.1 or later.
   - ``foo=1.2.3`` requires that foo must be exactly version 1.2.3.
 
-  The following variables may be set upon return.  Two sets of values exist,
-  one for the common case (``<XXX> = <prefix>``) and another for the
-  information ``pkg-config`` provides when it is called with the ``--static``
-  option (``<XXX> = <prefix>_STATIC``)
+  The following variables may be set upon return.  Two sets of values exist:
+  One for the common case (``<XXX> = <prefix>``) and another for the
+  information ``pkg-config`` provides when called with the ``--static``
+  option (``<XXX> = <prefix>_STATIC``).
 
   ``<XXX>_FOUND``
     set to 1 if module(s) exist
@@ -582,7 +586,7 @@ endmacro()
   There are some special variables whose prefix depends on the number of
   ``<moduleSpec>`` given.  When there is only one ``<moduleSpec>``,
   ``<YYY>`` will simply be ``<prefix>``, but if two or more ``<moduleSpec>``
-  items are given, ``<YYY>`` will be ``<prefix>_<moduleName>``
+  items are given, ``<YYY>`` will be ``<prefix>_<moduleName>``.
 
   ``<YYY>_VERSION``
     version of the module
@@ -593,7 +597,7 @@ endmacro()
   ``<YYY>_LIBDIR``
     lib directory of the module
 
-  Examples
+  Examples:
 
   .. code-block:: cmake
 
@@ -653,7 +657,9 @@ endmacro()
 
   The behavior of this command is the same as :command:`pkg_check_modules`,
   except that rather than checking for all the specified modules, it searches
-  for just the first successful match. ::
+  for just the first successful match.
+
+  .. code-block:: cmake
 
     pkg_search_module(<prefix>
                       [REQUIRED] [QUIET]
@@ -662,7 +668,7 @@ endmacro()
                       [IMPORTED_TARGET [GLOBAL]]
                       <moduleSpec> [<moduleSpec>...])
 
-  Examples
+  Example:
 
   .. code-block:: cmake
 

Utilities/Sphinx/cmake.py

@@ -4,14 +4,55 @@
 import os
 import re
 
-# Monkey patch for pygments reporting an error when generator expressions are
-# used.
-# https://bitbucket.org/birkenfeld/pygments-main/issue/942/cmake-generator-expressions-not-handled
+# Override much of pygments' CMakeLexer.
+# We need to parse CMake syntax definitions, not CMake code.
+
+# For hard test cases that use much of the syntax below, see
+# - module/FindPkgConfig.html (with "glib-2.0>=2.10 gtk+-2.0" and similar)
+# - module/ExternalProject.html (with http:// https:// git@; also has command options -E --build)
+# - manual/cmake-buildsystem.7.html (with nested $<..>; relative and absolute paths, "::")
+
 from pygments.lexers import CMakeLexer
-from pygments.token import Name, Operator
+from pygments.token import Name, Operator, Punctuation, String, Text, Comment, Generic, Whitespace, Number
 from pygments.lexer import bygroups
-CMakeLexer.tokens["args"].append(('(\\$<)(.+?)(>)',
-                                  bygroups(Operator, Name.Variable, Operator)))
+
+# Notes on regular expressions below:
+# - [\.\+-] are needed for string constants like gtk+-2.0
+# - Unix paths are recognized by '/'; support for Windows paths may be added if needed
+# - (\\.) allows for \-escapes (used in manual/cmake-language.7)
+# - $<..$<..$>..> nested occurence in cmake-buildsystem
+
+CMakeLexer.tokens["root"] = [
+  (r'\b(\w+)([ \t]*)(\()', bygroups(Name.Function, Text, Name.Function), '#push'),     # fctn(
+  (r'\(', Name.Function, '#push'),
+  (r'\)', Name.Function, '#pop'),
+  (r'\[', Punctuation, '#push'),
+  (r'\]', Punctuation, '#pop'),
+  (r'[|;,.=*\-]', Punctuation),
+  (r'\\\\', Punctuation),                                   # used in commands/source_group
+  (r'[:]', Operator),
+  (r'[<>]=', Punctuation),                                  # used in FindPkgConfig.cmake
+  (r'\$<', Operator, '#push'),                              # $<...>
+  (r'<[^<|]+?>(\w*\.\.\.)?', Name.Variable),                # <expr>
+  (r'(\$\w*\{)(.+?)(\})', bygroups(Operator, Name.Tag, Operator)),   # ${..} $ENV{..}
+  (r'([A-Z]+\{)(.+?)(\})', bygroups(Operator, Name.Tag, Operator)),  # DATA{ ...}
+  (r'[a-z]+(@|(://))((\\.)|[\w.+-:/\\])+', Name.Attribute),          # URL, git@, ...
+  (r'/\w[\w\.\+-/\\]*', Name.Attribute),                    # absolute path
+  (r'/', Name.Attribute),
+  (r'\w[\w\.\+-]*/[\w.+-/\\]*', Name.Attribute),            # relative path
+  (r'[A-Z]((\\.)|[\w.+-])*[a-z]((\\.)|[\w.+-])*', Name.Builtin), # initial A-Z, contains a-z
+  (r'@?[A-Z][A-Z0-9_]*', Name.Constant),
+  (r'[a-z_]((\\;)|(\\ )|[\w.+-])*', Name.Builtin),
+  (r'[0-9][0-9\.]*', Number),
+  (r'(?s)"(\\"|[^"])*"', String),                           # "string"
+  (r'\.\.\.', Name.Variable),
+  (r'<', Operator, '#push'),                                # <..|..> is different from <expr>
+  (r'>', Operator, '#pop'),
+  (r'\n', Whitespace),
+  (r'[ \t]+', Whitespace),
+  (r'#.*\n', Comment),
+  #  (r'[^<>\])\}\|$"# \t\n]+', Name.Exception),            # fallback, for debugging only
+]
 
 # Monkey patch for sphinx generating invalid content for qcollectiongenerator
 # https://bitbucket.org/birkenfeld/sphinx/issue/1435/qthelp-builder-should-htmlescape-keywords

Utilities/Sphinx/colors.py

@@ -0,0 +1,29 @@
+# -*- coding: utf-8 -*-
+
+from pygments.style import Style
+from pygments.token import Name, Comment, String, Number, Operator, Whitespace
+
+class CMakeTemplateStyle(Style):
+    """
+        for more token names, see pygments/styles.default
+    """
+
+    background_color = "#f8f8f8"
+    default_style = ""
+
+    styles = {
+        Whitespace:                "#bbbbbb",
+        Comment:                   "italic #408080",
+        Operator:                  "bold #000000",
+        String:                    "#217A21",
+        Number:                    "#105030",
+        Name.Builtin:              "#400080",          # anything lowercase
+        Name.Function:             "bold #1010A0",     # function
+        Name.Variable:             "#1080B0",          # <..>
+        Name.Tag:                  "#19177C",          # ${..}
+        Name.Constant:             "#6020E0",          # uppercase only
+        Name.Entity:               "italic #70A020",   # @..@
+        Name.Attribute:            "#906060",          # paths, URLs
+        Name.Label:                "#A0A000",          # anything left over
+        Name.Exception:            "bold #FF0000",     # for debugging only
+    }

Utilities/Sphinx/conf.py.in

@@ -15,6 +15,7 @@ project = 'CMake'
 copyright = '@conf_copyright@'
 version = '@conf_version@' # feature version
 release = '@conf_release@' # full version string
+pygments_style = 'colors.CMakeTemplateStyle'
 
 primary_domain = 'cmake'
 highlight_language = 'none'