feat(appium): configuration file and schema support

Summary: This PR adds support for configuration files (e.g., `.appiumrc.json`) and schema definitions.  Command-line options _and_ configuration is now derived from the schema(s), for Appium _and_ its extensions (both plugins and drivers).

## Dependencies
### Development
- Added `through2` for a custom gulp task to convert `appium/lib/appium-config-schema.js` to a JSON file
- Added `json-schema-to-typescript` for converting aforementioned JSON file to a `.d.ts` file (which gets published)

### Production (applies to `appium` only)
- Added `libconfig` for multi-format config file loading, which is a smaller drop-in replacement for `cosmiconfig`
- Added `ajv` for schema support and validation
- Added `ajv-formats` to understand optional value formats (e.g., `ipv4`)
- Added `@sidvind/better-ajv-errors` which provides nice error output on the CLI if the config file is invalid
- Added `@oclif/errors` for better error output on the CLI if _arguments_ are invalid

## New Modules of Note

- `lib/schema.js` consumes `ajv` to create and modify a schema
    - It exposes a handful of public API functions, so that consumers can query the schemas without actually running Appium.
    - The flow is basically this: read the base schema (`appium-config-schema.js`).  When extensions load and have schemas, add those to a pot.  When `finalizeSchema()` is called, throw the base schema into the pot as well, stir, and add to singleton `ajv` instance.  Now we can validate against the stew.
    - The schema for an extension is read and registered via `readExtensionSchema()`
    - Provides a way to get at default (for showing those "non-default" options) and `flattenSchema()` which squishes the finalized schema into a single `Record` key/value object, for use by the CLI.  This may make sense to move into `schema-args.js`
    - Provides access to ajv-and-JSON-schema-defined _formatters_ (which are basically just validation functions).  For example, if we want any string, it's just a `string`.  But if we want the string to match a regexp, that's a formatter.  (e.g., the `hostname` type)
-  lib/config-file.js` consumes `lilconfig` to read config files, validate them against the schema, and provide the result as CLI args
    - Handles conversion between kebab-case and camelCase
    - Contains `formatErrors()` which will be called after validation of a config file fails, which shows _exactly_ where the config file is wrong (usually).
- `lib/cli/schema-args.js` which serves as an adapter between the schema and `argparse`
    - Translates between JSON schema and `argparse`, which...well, I did what I could.  It's pretty close.
    - Wraps the formatters as mentioned above, and provides custom ones.
    - We cannot, unfortunately, wholly use schema validation against CLI args.  The flag names are different, the structure is different, there's no actual _file_ to compare against, and `argparse` does not help.  We _can_ use anything `argparse` natively provides, in addition to `ajv` formatters, and any custom ones (e.g., allowable number ranges).
    - Will likely end up adding more formatters corresponding to various JSON schema features later.  We _may_ be able to use _more_ of Ajv to do this (e.g., reuse its internal validators, if they are public APIs).
- `lib/appium-config-schema.js` which is the root schema. Extensions add to this before it is registered & compiled by `ajv`.
    - The basic shape is that there are three top-level props, `server`, `driver`, and `plugin`.  `server` contains all of the server options as you'd expect, and `driver` and `plugin` will be filled in by extensions.  Each will have properties corresponding to the extension name.  You can find examples in `sample-code` and the test fixtures
    - There's a gulp task that takes this file, rewrites it as JSON, then pipes it into another tool which outputs a `.d.ts` file, nicely described using all of the metadata from the schema.  This will be useful for mainly contributors, extension authors, and consumers of Appium (though not end-users).
- `lib/ext-config-io.js`
    - A new module which handles reading/writing of YAML extension config files only.
    - This class is a singleton.
    - Some logic pulled out of `extension-config.js` and into here
    - Previously, we would read/write this file multiple times (once per subclass of `ExtensionConfig`)
    - This module uses a naive dirty-tracker to persist the extension data automatically.
    - Subclasses of `ExtensionConfig` only have access to their "portion" of the config file; drivers can't see plugins and vice-versa.
    - `DRIVER_TYPE` and `PLUGIN_TYPE` originate here, mainly to avoid a circular dependency

## Modifications

- The single source of truth for the CLI _and_ config file options is the schema, with the caveats that:
    - Command-line arguments (for `appium server` are effectively a flat list, but the schema/config file is not.
    - Extension-specific configuration lives in an extension-type-specific property (`driver` or `plugin`) in the config file
        - Under each (`driver` or `plugin`), each extension may have a configuration section corresponding to the extension name
        - The shape of the extension-specific options _can_ be defined by a schema (and referenced in the `appium.schema` prop of the extension's `package.json`)
        - If an extension does not provide a schema, no validation will occur on any options found here -- TODO needs tests
    - This does _not_ affect _CLI commands_ added by extensions; only options for `server`.
- The `--appium-home` flag is now removed, as we can't find extensions without `APPIUM_HOME`, which means we can't parse arguments (because extensions may have added some).
- `--plugin-args` and `--driver-args` are removed.  These are replaced by individual flags for each argument, defined by the extension in its schema, e.g., `--driver-fake-host=127.0.0.1`
- Extensions can still define argument "constraints" using `argConstraints`, but this is considered deprecated
- `lib/main.js`
    - Now exports a few public APIs
    - split into functions `init` and `main`.  `init` is intended to be run via a consumer, which does what the old `main` did up to starting the server.  This bisection is not perfect, as `init` still does some CLI-related things; it needs to be further broken up.
    - `driverConfig.read()` and `pluginConfig.read()` are now called in `lib/cli/parser.js`.
    - Merges the CLI args, config file args, and defaults, in that order of precedence.
- `lib/grid-register.js`
    - Supports embedded configuration instead of just a path to a JSON file.
    - Not clear to me if `nodeconfig` is a thing we need to keep?
- `lib/extension-config.js`
    - Given `APPIUM_HOME` is where we keep the extensions, this file now defines `APPIUM_HOME`. It is exported from the `appium` package.
    - Add `getSchemaProblems()` method, which will read/validate/register the schema for an extension, if one exists.
    - Added many type annotations
    - `read()` and `write()` now delegate to the singleton `ExtConfigIO` instnace
    - Re-exports `DRIVER_TYPE` and `PLUGIN_TYPE`
- `lib/driver-config.js`
    - **Fix** the case where two drivers attempt to use the same `automationName`.  It appears this was intended, but did not work correctly.  This involved overloading `read()`.
- `lib/plugin-config.js`
    - Literally nothing, which is neat.
- `lib/config.js`
    - Remove unused validations (now occur via schema validation or are extension-specific)
    - Rename `getNonDefaultArgs` to `getNonDefaultServerArgs` to be more specific
    - Rewrote `getNonDefaultServerArgs`, which became much more complicated due to adding a config file and schema defaults to the mix. Flexed my lodash-fu, which is still kind of crappy because I'm better with `lodas/fp`.
- `lib/cli/parser.js`
    - Created a wrapper for `argparse`: class `ArgParser`, which replaces the monkeypatching we were doing here.
    - This class also stores extension-specific args in a namespace.  For example, `--driver-foo-bar=baz` would normally be stored in prop `driverFooBar`, but this moves it into `driver: {foo: {bar: 'baz'}}`, which will be provided to its associated extension
    - `addExtensionsToParser` renamed to `addExtensionCommandsToParser` which is more specific.
    - Removed `getDefaultServerArgs` since those now come from the schema.
    - Removed "shared args" stuff
    - `getParser` is now async, and constructs an `ArgParser`.  By necessity, the extension config YAML is read here (which is still wonky because we have to call `read()` twice; it only actually ends up reading once)
    - `getParser` also _finalizes_ the schema, which means "katamari-up the base schema and all extension schemas, then tell ajv about the result"
- `lib/cli/parser-helpers.js`
    - Allow security features to just be an array (as allowed in schema)
    - TODO: needs test
- `lib/cli/extension.js`
    - Use `APPIUM_HOME` exported from `lib/extension-config` instead of `args.appiumHome`
    - Avoids re-instantiating `ExtensionConfig` subclasses if not needed
- `lib/cli/args.js`
    - Remove all server argument definitions in lieu of schema; leave a few that do not make sense in a config file (like `--config`, which determines the location of the config file!)
    - Server args are now pulled from the schema via a call to `toParserArgs()` with overrides for various validation functions (which are confusingly "types" in `argparse` parlance).  Overrides can probably be removed or relocated to `schema-args`
    - Instantiates the `DriverConfig` and `PluginConfig` instances, which are exported.   These might want to move somewhere else
- Tests
    - Added unit tests for `extension-config`, `grid-register` which were missing
    - Added unit tests for new `schema` and `schema-args` and `config-file` modules
    - Added e2e test for config file handling
    - Other necessary modifications
    - Fix path to `fake-driver` in `driver-e2e-specs`
    - Removed stuff from `parser-specs` which no longer applies

## Other

- Added `scripts/generate-schema-declarations.js` which uses `json-schema-to-typescript` to create `appium-config.d.ts`
- Added config file samples in `.json` and `.yaml` formats
- Added custom gulp task to `appium/gulpfile.js` to generate schema in JSON format, since its "single source of truth" is actually just a big JS object
- Moved `appium` CLI-related tests into `test/cli`
- Added some declarations into `packages/appium/types`; the schema-related one is generated by the script mentioend above, but `types.d.ts` is just some stuff that was not expressible in a docstring
This commit is contained in:
Christopher Hiller
2021-10-07 15:55:01 -07:00
parent 71357e52e1
commit d52c36e1ea
62 changed files with 5289 additions and 1075 deletions
+38
View File
@@ -0,0 +1,38 @@
/* eslint-disable no-console */
'use strict';
/*
* This module reads in the config file JSON schema and outputs a TypeScript declaration file (`.d.ts`).
*/
const path = require('path');
const {compileFromFile} = require('json-schema-to-typescript');
const {fs} = require('../packages/support');
const SCHEMA_PATH = require.resolve(
'../packages/appium/build/lib/appium-config.schema.json',
);
const DECLARATIONS_PATH = path.join(
__dirname,
'..',
'packages',
'appium',
'types',
'appium-config.d.ts',
);
async function main () {
try {
const ts = await compileFromFile(SCHEMA_PATH);
await fs.writeFile(DECLARATIONS_PATH, ts);
console.log(`wrote to ${DECLARATIONS_PATH}`);
} catch (err) {
console.error(err);
process.exitCode = 1;
}
}
if (require.main === module) {
main();
}