`@appium/schema` is a package which was extracted from `appium`. It _only_ contains the schema. The impetus here was to avoid cyclical dependencies, since `appium` depends on `@appium/types`, and `@appium/types` depends on the schema. This package breaks the cycle.
The `scripts/generate-schema-json.js` (which converts the compiled JS schema file into JSON) has moved into this package, and the generated JSON file is now under version control. This is because the `generate-schema-types.js` script (previously `generate-schema-declarations`) depends on it, and `@appium/types` depends on the generated types. The generated types are _also_ under version control. I do not expect the schema to change often, so I believe the tradeoff to be worth it.
## @appium/schema
`lib/appium-config-schema.js` is the "single source of truth" for the schema.
## appium
- Removed schema source
- Removed namespaces from internal types because they suck (see change to `types/index.d.ts`)
- Removed unused `continuation-local-storage`
- Add dep for `@appium/schema`
- Added a TS project dependency on `@appium/schema`
## @appium/types
- The "sources" have moved from `src` to `lib` for consistency.
- `lib/appium-config.ts` (the generated types) are now under version control.
- `lib/schema/` has been removed (it previously held the JSON schema artifact)
- `scripts/generate-schema-declarations.js` becomes `scripts/generate-schema-types.js`
- Updated scripts in `package.json`
- Added dep for `@appium/schema`
- Added a TS project dependency on `@appium/schema`
## Other
- Reorganize root `tsconfig.json`; add `schema` reference; remove unused `noEmit: true`
- Sort root `package.json`
- Remove schema JSON and schema declarations scripts, since those are now both handled in their respective packages
- Remove `lint-staged` config for schema file, since it didn't work right anyway (still need to do something about this, but probably doesn't need to be in a precommit hook)
- Remove stuff from `.gitignore`
This changes the TS configuration to to a) emit declarations for supported packages, and b) build declarations incrementally by package.
To begin, we are targeting `appium`, `@appium/base-driver` and `@appium/support` for declarations; these three are intended to be published with declarations generated via their JS.
Both `@appium/support` and `appium` are fully typechecked (sans implicit `any` types), but `@appium/base-driver` is mostly not. Regardless, declarations are generated for all three.
Each package will have its own `tsconfig.json` which "inherits" from `config/tsconfig.base.json`. Each will also need to declare relative paths to dependencies. e.g., the `appium` package must configure its `tsconfig.json` so that it depends on the declarations of `@appium/support` and also where to find `@appium/support`. Essentially, a small dependency tree gets built, and those at the "root" get their declarations built first; then the consumers use those declarations.
Also reorganized some scripts and added `npm-run-all` to help and added a "typecheck" job to CI.
Since `npm install` will build, it's pretty inconvenient to have `npm` abort installation if the build fails. In that case, if any of the build steps fail (see `build:loose`) the installation will continue. Added a `prepublishOnly` which runs a build in "strict" mode where everything must pass.
_Note: there's still a strong coupling of "what `BaseDriver` provides" and "what an external driver *may* provide". It's unclear to me if this is a problem, though there seem to be a handful of methods that external drivers *must* implement. The easiest way to get around this would probably be to extract these required properties and methods into their own interface (duplication) if we cannot otherwise declare methods as "abstract". As it stands, the `Driver` is basically just `BaseDriver`._
Also:
- Moved some configuration types out of `appium` and into here, since they are used both by `appium` and `base-driver` via `DriverOpts`
- Split `Driver` type into "the thing that BaseDriver is" and "all the other methods external drivers can implement"
- Added missing `proxyCommand` method
- Better generics for `LogType`
- Remove unused `Constructor` type
- Better types for `getSession`/`getSessions`
- Added util `Class` type
- Added many missing typings from DefinitelyTyped
- Added `ts-node` for `@wdio/types` (I need to send a PR to webdriver to eliminate this dependency)
- Type checks in their CI step
This change removes `gulp` and replaces it with plain ol' `babel` and `mocha`.
## `appium`
- No gulp.
- `test/` has been re-organized. All test files now end in the extension `.spec.js`, which is a recognized convention and understood by editors/IDEs. The tests are further split into `unit` and `e2e` subdirs. This makes working with both `babel` and `mocha` easier.
- The tests are _not_ run against the transpiled code; code is now transpiled on-the-fly via `@babel/register`. This only affects `appium`.
- `commands-yml/validator.js` moved to `scripts/parse-yml-commands.js`. It has been rewritten as a CJS module. `commands-yml/` should not contain `.js` files.
- `test/setup.js` is a new test harness specific to Appium, which loads & configures `@babel/register`. Eventually, the contents of this file can be moved into the root monorepo's `test/setup.js`, but the rest of the packages do not need `@babel/register`, so we avoid the overhead.
## `@appium/gulp-plugins`
- Unfortunately, due to the new `npm test` strategy (see below), we cannot use the `nyan` reporter. Or rather, we _can_, but will not see the animation. So the reporter is now set to default to `spec`.
- Modified tests so they didn't overwrite the actual `build` dir with transpiled fixtures, which is what was happening. This was causing failures depending on the transpile/test order.
## All Packages
- All packages now have their own `build`, `test`, `test:e2e` and `dev` (just build + watch mode) npm scripts. This makes running a complete build easier, since each package can now provide its own appropriate command. This is _mostly_ just `gulp once` (where `gulp` was used before), except for `appium`, where `@babel/cli` is used directly.
- This is what happens when `npm test` is run from the monorepo (in order):
1. All packages are built (in parallel) by running `npm run build` in each folder (via `lerna`). See `pretest`
2. All code is linted directly via `eslint`.
3. All _unit tests_ are run w/ `npm test` in each folder, having assumed transpilation already occurred (if necessary).
- `npm run e2e-test` runs the `npm run test:e2e` in each folder. No transpilation occurs! Careful.
- All scripts are expected to live in `packages/*/scripts`, and this directory enforces CJS via ESLint. `postinstall.js` moved, as well as `check-npm-pack-files.js`. `generate-schema-declarations.mjs` moved from the root `scripts/` dir into `packages/appium/scripts/`; it's now CJS instead of ESM.
- I had profiled running various things with `lerna run --parallel`, but it turns out this is not faster. I'm not sure why, but I think it may have something to do with streaming STDOUT/STDERR, or potentially the overhead of running child processes outweighs the time it takes to actually run tests and builds.
- The behavior of `lerna run` now mimicks `lerna exec`, which was "run stuff in serial, streaming the output". The default behavior of `lerna run` is "run stuff in parallel, buffering the output". Buffered output sucks, and streaming the output from all tasks at once creates chaos. To override this behavior, you'll see `--parallel --concurrency=8 --prefix` for tasks that can truly be parallelized, like transpilation. The number 8 is the "maximum", and CI usually won't have more than 2 cores available anyway.
- When run in CI, Mocha runs with `--forbid-only`, which will fail if there's an `.only()` somewhere in the tests. Further, it forces color output.
## Etc.
- Updated Wallaby config, `.gitignore`, `.eslintignore`.
- Sorted `package.json` files
- Removed `watch` and `coverage` scripts which were broken anyway.
chore(appium): fix schema json path issues
- the path to the JSON schema is now correct: `lib/appium-config-schema.json`
- add `log-symbols` to devdeps for niceness. this is already a transitive dep
`lint-staged` doesn't want to do it because reasons. I don't think it necessarily supports the use case where if file _A_ is staged, then file _B_ should be staged--it only acts upon file _A_.
also replace the `lint:fix` script with use of `eslint` proper, since using `gulp` misses files _not_ in `packages/`.
This now consumes Ajv directly to validate CLI args. Since a JSON schema is made of JSON schemas, we can simply validate an argument against _the schema for that argument_.
- Replaced parser transformation functions (`parseSecurityFeatures()`, `parseJsonStringOrFile()`) with custom transforms which are bound to keyword `appiumCliTransformer`. e.g., any option that can now accept a JSON string or filepath on the command line may use `appiumCliTransformer: 'json'`. This is the default behavior for `array` and `object` -type options, which cannot easily be expressed on the CLI. The config file does _not_ support this behavior, so the value must be whatever the schema `type` says (e.g., an `object` instead of a JSON string or filepath). To this end, removed `parser-helpers.js`
- Further, we now disallow union types in the schema. A prop must have a single type.
- Added custom keywords `appiumCliDest`, `appiumCliDescription`, and `appiumCliTransformer`. Previously (if existed), this could have been any value--these are now typed by the schema--e.g., a number for `appiumCliDest` will fail.
- Added more types and TS checks around
- Fixed misspelling of "successfully" in various places
- Removed `@oclifg/errors` because we can now use `better-ajv-errors` to display them, as intended. It looks great!
- Moved all schema-related stuff to `lib/schema/` because I had too many modules
- Added more constants to please @mykola-mokhnach 😄
- Handles untyped properties or `null`-typed properties
- `getSchema()`, `validate()` etc now accept a schema ID. The schema ID is derived from the argument name. This is how we handle validation for each argument individually. The config file is still validated against the entire schema at once.
- Fixed problem where only `driverConfigs` were ever used to call extension commands!
- Added some semaphore-like things to `ExtConfigIO` to avoid multiple reads/writes at once
- Fixed display of metavars in `--help`
- More tests; removed tests for removed code
Summary: This PR adds support for configuration files (e.g., `.appiumrc.json`) and schema definitions. Command-line options _and_ configuration is now derived from the schema(s), for Appium _and_ its extensions (both plugins and drivers).
## Dependencies
### Development
- Added `through2` for a custom gulp task to convert `appium/lib/appium-config-schema.js` to a JSON file
- Added `json-schema-to-typescript` for converting aforementioned JSON file to a `.d.ts` file (which gets published)
### Production (applies to `appium` only)
- Added `libconfig` for multi-format config file loading, which is a smaller drop-in replacement for `cosmiconfig`
- Added `ajv` for schema support and validation
- Added `ajv-formats` to understand optional value formats (e.g., `ipv4`)
- Added `@sidvind/better-ajv-errors` which provides nice error output on the CLI if the config file is invalid
- Added `@oclif/errors` for better error output on the CLI if _arguments_ are invalid
## New Modules of Note
- `lib/schema.js` consumes `ajv` to create and modify a schema
- It exposes a handful of public API functions, so that consumers can query the schemas without actually running Appium.
- The flow is basically this: read the base schema (`appium-config-schema.js`). When extensions load and have schemas, add those to a pot. When `finalizeSchema()` is called, throw the base schema into the pot as well, stir, and add to singleton `ajv` instance. Now we can validate against the stew.
- The schema for an extension is read and registered via `readExtensionSchema()`
- Provides a way to get at default (for showing those "non-default" options) and `flattenSchema()` which squishes the finalized schema into a single `Record` key/value object, for use by the CLI. This may make sense to move into `schema-args.js`
- Provides access to ajv-and-JSON-schema-defined _formatters_ (which are basically just validation functions). For example, if we want any string, it's just a `string`. But if we want the string to match a regexp, that's a formatter. (e.g., the `hostname` type)
- lib/config-file.js` consumes `lilconfig` to read config files, validate them against the schema, and provide the result as CLI args
- Handles conversion between kebab-case and camelCase
- Contains `formatErrors()` which will be called after validation of a config file fails, which shows _exactly_ where the config file is wrong (usually).
- `lib/cli/schema-args.js` which serves as an adapter between the schema and `argparse`
- Translates between JSON schema and `argparse`, which...well, I did what I could. It's pretty close.
- Wraps the formatters as mentioned above, and provides custom ones.
- We cannot, unfortunately, wholly use schema validation against CLI args. The flag names are different, the structure is different, there's no actual _file_ to compare against, and `argparse` does not help. We _can_ use anything `argparse` natively provides, in addition to `ajv` formatters, and any custom ones (e.g., allowable number ranges).
- Will likely end up adding more formatters corresponding to various JSON schema features later. We _may_ be able to use _more_ of Ajv to do this (e.g., reuse its internal validators, if they are public APIs).
- `lib/appium-config-schema.js` which is the root schema. Extensions add to this before it is registered & compiled by `ajv`.
- The basic shape is that there are three top-level props, `server`, `driver`, and `plugin`. `server` contains all of the server options as you'd expect, and `driver` and `plugin` will be filled in by extensions. Each will have properties corresponding to the extension name. You can find examples in `sample-code` and the test fixtures
- There's a gulp task that takes this file, rewrites it as JSON, then pipes it into another tool which outputs a `.d.ts` file, nicely described using all of the metadata from the schema. This will be useful for mainly contributors, extension authors, and consumers of Appium (though not end-users).
- `lib/ext-config-io.js`
- A new module which handles reading/writing of YAML extension config files only.
- This class is a singleton.
- Some logic pulled out of `extension-config.js` and into here
- Previously, we would read/write this file multiple times (once per subclass of `ExtensionConfig`)
- This module uses a naive dirty-tracker to persist the extension data automatically.
- Subclasses of `ExtensionConfig` only have access to their "portion" of the config file; drivers can't see plugins and vice-versa.
- `DRIVER_TYPE` and `PLUGIN_TYPE` originate here, mainly to avoid a circular dependency
## Modifications
- The single source of truth for the CLI _and_ config file options is the schema, with the caveats that:
- Command-line arguments (for `appium server` are effectively a flat list, but the schema/config file is not.
- Extension-specific configuration lives in an extension-type-specific property (`driver` or `plugin`) in the config file
- Under each (`driver` or `plugin`), each extension may have a configuration section corresponding to the extension name
- The shape of the extension-specific options _can_ be defined by a schema (and referenced in the `appium.schema` prop of the extension's `package.json`)
- If an extension does not provide a schema, no validation will occur on any options found here -- TODO needs tests
- This does _not_ affect _CLI commands_ added by extensions; only options for `server`.
- The `--appium-home` flag is now removed, as we can't find extensions without `APPIUM_HOME`, which means we can't parse arguments (because extensions may have added some).
- `--plugin-args` and `--driver-args` are removed. These are replaced by individual flags for each argument, defined by the extension in its schema, e.g., `--driver-fake-host=127.0.0.1`
- Extensions can still define argument "constraints" using `argConstraints`, but this is considered deprecated
- `lib/main.js`
- Now exports a few public APIs
- split into functions `init` and `main`. `init` is intended to be run via a consumer, which does what the old `main` did up to starting the server. This bisection is not perfect, as `init` still does some CLI-related things; it needs to be further broken up.
- `driverConfig.read()` and `pluginConfig.read()` are now called in `lib/cli/parser.js`.
- Merges the CLI args, config file args, and defaults, in that order of precedence.
- `lib/grid-register.js`
- Supports embedded configuration instead of just a path to a JSON file.
- Not clear to me if `nodeconfig` is a thing we need to keep?
- `lib/extension-config.js`
- Given `APPIUM_HOME` is where we keep the extensions, this file now defines `APPIUM_HOME`. It is exported from the `appium` package.
- Add `getSchemaProblems()` method, which will read/validate/register the schema for an extension, if one exists.
- Added many type annotations
- `read()` and `write()` now delegate to the singleton `ExtConfigIO` instnace
- Re-exports `DRIVER_TYPE` and `PLUGIN_TYPE`
- `lib/driver-config.js`
- **Fix** the case where two drivers attempt to use the same `automationName`. It appears this was intended, but did not work correctly. This involved overloading `read()`.
- `lib/plugin-config.js`
- Literally nothing, which is neat.
- `lib/config.js`
- Remove unused validations (now occur via schema validation or are extension-specific)
- Rename `getNonDefaultArgs` to `getNonDefaultServerArgs` to be more specific
- Rewrote `getNonDefaultServerArgs`, which became much more complicated due to adding a config file and schema defaults to the mix. Flexed my lodash-fu, which is still kind of crappy because I'm better with `lodas/fp`.
- `lib/cli/parser.js`
- Created a wrapper for `argparse`: class `ArgParser`, which replaces the monkeypatching we were doing here.
- This class also stores extension-specific args in a namespace. For example, `--driver-foo-bar=baz` would normally be stored in prop `driverFooBar`, but this moves it into `driver: {foo: {bar: 'baz'}}`, which will be provided to its associated extension
- `addExtensionsToParser` renamed to `addExtensionCommandsToParser` which is more specific.
- Removed `getDefaultServerArgs` since those now come from the schema.
- Removed "shared args" stuff
- `getParser` is now async, and constructs an `ArgParser`. By necessity, the extension config YAML is read here (which is still wonky because we have to call `read()` twice; it only actually ends up reading once)
- `getParser` also _finalizes_ the schema, which means "katamari-up the base schema and all extension schemas, then tell ajv about the result"
- `lib/cli/parser-helpers.js`
- Allow security features to just be an array (as allowed in schema)
- TODO: needs test
- `lib/cli/extension.js`
- Use `APPIUM_HOME` exported from `lib/extension-config` instead of `args.appiumHome`
- Avoids re-instantiating `ExtensionConfig` subclasses if not needed
- `lib/cli/args.js`
- Remove all server argument definitions in lieu of schema; leave a few that do not make sense in a config file (like `--config`, which determines the location of the config file!)
- Server args are now pulled from the schema via a call to `toParserArgs()` with overrides for various validation functions (which are confusingly "types" in `argparse` parlance). Overrides can probably be removed or relocated to `schema-args`
- Instantiates the `DriverConfig` and `PluginConfig` instances, which are exported. These might want to move somewhere else
- Tests
- Added unit tests for `extension-config`, `grid-register` which were missing
- Added unit tests for new `schema` and `schema-args` and `config-file` modules
- Added e2e test for config file handling
- Other necessary modifications
- Fix path to `fake-driver` in `driver-e2e-specs`
- Removed stuff from `parser-specs` which no longer applies
## Other
- Added `scripts/generate-schema-declarations.js` which uses `json-schema-to-typescript` to create `appium-config.d.ts`
- Added config file samples in `.json` and `.yaml` formats
- Added custom gulp task to `appium/gulpfile.js` to generate schema in JSON format, since its "single source of truth" is actually just a big JS object
- Moved `appium` CLI-related tests into `test/cli`
- Added some declarations into `packages/appium/types`; the schema-related one is generated by the script mentioend above, but `types.d.ts` is just some stuff that was not expressible in a docstring
This PR supersedes PR #14562.
This is my first attempt at making this repo a monorepo and moving appium-proper into `packages/appium`.
I had tried to make this work with npm's "workspaces", but it seems lifecycle scripts don't run as you'd expect them to upon install. Instead, I'm using Lerna, which provides more features for monorepos.
Changes include:
- The root `package.json` is now a package named `appium-monorepo`, which is `private`. The monorepo should not be published.
- All dev dependencies remain in the root `package.json`.
- `gulpfile.js` _moved_ into `packages/appium/gulpfile.js`, but this should be **temporary**--`appium-gulp-plugins` makes certain assumptions which a monorepo violates, and I was unable to get it working without patching `appium-gulp-plugins`.
- Most of the `package.json` `scripts` moved into `packages/appium/package.json` due to the above. This too should be considered **temporary**.
- Adds a root `postinstall` lifecycle script which calls `lerna bootstrap` to install/link packages. I don't see a `package-lock.json` in `packages/appium/`; this needs investigating.
- Adds a root `test` lifecycle script which executes `npm run test` in all packages. The tests do not pass, but I don't know if this is my fault.
- The `clean` script will purge all `node_modules` folders in each package, as well.
- I removed `scripts/release*.sh`, since it seems they would fail anyway--`lerna` should be used for releases instead
- The shrinkwrap-related stuff is untested.
- Renamed `tags` prop in `packages/appium/package.json` to `keywords`. I don't know if `tags` is supported.
