Files
cypress/guides/error-handling.md
T
Brian Mann 29841f32b9 feat: redesign server errors (#20072)
* chore: rename errors.js -> errors.ts

* refactor: type safety on errors

* refactor: add err_template for consistent error formatting

* fix a few system tests

* fix tests; update snapshots

* Fix types

* normalize snapshot - remove chalk ansi colors

* more unit test fixes

* more system test fixes

* circleci build

* backtick always in stdout, fix error formatting and failing snapshots

* refactor: create @packages/errors

* fix import

* fix import

* fixing build / tests

* remove extraneous file

* move warnIfExplicitCiBuildId

* fix build / tests

* Fix

* error, type fixes, documentation, standardize child process error serialization

* fix import

* build errors on install

* wrote specs generating visual images of all errors

* remove unused dep

* sanitize stack traces

* add image diffing

- if base images don't exist, create them
- if base images don't match and local, overwrite them, if in CI throw
- if base images are stale and local, delete them, if in CI throw

* remove Courier New + MesloLGS NF font

* type fixes, remove Bluebird, general cleanup

* TS Cleanup

* skip typecheck on tests for now

* yarn.lock

* fix @types/chai version

* fix yarn.lock

* Different version of mocha types so it isnt patched

* errors spec snapshot

* CI fix

* fixes

* store snapshot images in circle artifacts

* dont change artifact destination prefix

* use Courier Prime

* antialias the text

* decrease pixelmatch threshold, fail in CI only when changed pixels > 100

* increase timeout

* overflow: hidden, remove new Promise, add debug logging

Co-Authored-By: Tim Griesser <tgriesser@gmail.com>

* run unit tests w/ concurrency=1

* unique window per file

* disable app hardware acceleration + use in process gpu + single process

* do not do image diffing

- conditionally convert html to images
- store html snapshots
- do not store images in git

* store snapshot html

* Merge branch 'tgriesser/chore/refactor-errors' of https://github.com/cypress-io/cypress into tgriesser/chore/refactor-errors

* remove concurrency

* fix assertion

* fixing ci

* Link in readme

* pass the browsers to listItems

* fix: build @packages/errors in CI, defer import to prevent errors locally

* Merge branch 'develop' into tgriesser/chore/refactor-errors

* develop:
  chore: fix cypress npm package artifact upload path (#20023)
  chore(driver): move cy.within logic into it's own file (#20036)
  chore: update automerge workflows (#19982)
  fix(selectFile): use target window's File/DataTransfer classes (#20003)
  chore: Update Chrome (stable) to 98.0.4758.80 and Chrome (beta) to 98.0.4758.80 (#19995)
  fix: Adjust ffmpeg CLI args for performance (#19983)
  build: allow unified to run cypress on Apple Silicon (arm64) (backport #19067 to 9.x) (#19968)

* fix run-if-ci.sh

* remove dead code

* Mark the .html files as generated in gitattributes

* fix running single error case, slice out more of the brittle stack

* remove additional brittle stack line

* firefox web security error

* nest inside of describe

* reformat and redesign errors

* more error cleanup and standardization

* additional formatting of errors, code cleanup, refactoring

* update ansi colors to match terminal colors

* cleanup remaining loose ends, update several errors, compact excess formatters

* fix types

* additional formatting, remove TODO's, ensure no [object Object] in output

* add test for 412 server response on invalid schema

* update unknown dashboard error on creating run

* use fs.access instead of fs.stat for perf

* added PLUGINS_FILE_NOT_FOUND error

- separated out from PLUGINS_FILE_ERROR
- add system tests for both cases
- update snapshots
- remove stack trace from PLUGINS_FILE_NOT_FOUND fka PLUGINS_FILE_ERROR

* add plugins system test around plugins synchronously throwing on require

* remove forcing process.cwd() to be packages/server, update affected code

- this was a long needed hangover from very old code that was doing unnecessary things due to respawning electron from node and handling various entrypoints into booting cypress
- this also makes the root yarn dev and dev-debug work correctly because options.cwd is set correctly now

* perf: lazy load chalk

* remove excessive line since the file exists but is invalid

* fix types

* add system test when plugins function throws a synchronous error

* create new PLUGINS_INVALID_EVENT_ERROR, wire it up correctly for error template use

- properly pass error instance from child to ensure proper user stack frames
- move error display code into the right place

* only show a single stack trace, either originalError or internal cypressError

* push error html snapshots

* fix tests, types

* fix test

* fix failing tests

* fix tests

* fixes lots of broken tests

* more test fixes

* fixes more tests

* fix type checking

* wip: consistent handling of interpolated values

* feat: fixing up errors, added simple error comparison tool

* wrapping up error formatting

* Fixes for unit tests

* fix PLUGINS_VALIDATION_ERROR

* fix fs.readdir bug, show rows even if there's only markdown formatting [SKIP CI]

* when in base-list, show full width of errors

* Fix type errors

* added searching and filtering for files based on error name

* fix: system tests

* updated NO_SPECS_FOUND error to properly join searched folder + pattern

- join patterns off of process.cwd, not projectRoot
- highlight original specPattern in yellow, baseDir in blue
- add tests

* fixes failing tests

* fix test

* preserve original spec pattern, display relative to projectRoot for terminal banner

* make the nodeVersion path display in gray, not white

* fix tests, pass right variables

* fix chrome:canary snapshots

* update snapshots

* update snapshot

* strip newlines caused by "Still waiting to connect to {browser}..."

* don't remove the snapshotHtmlFolder in CI, add additional verification snapshots match to error keys symmetrically

* update snapshot

* update snapshot

* update snapshot

* update snapshot

* update snapshot

* update snapshot

* update gitignore

* fix snapshot

* update snapshot html

* update logic for parsing the resolve pattern matching, add tests

* update snapshots

* update snapshot

* update snapshot

* update snapshot

* fix failing test

* fix: error_message_spec

* fix snapshot

* run each variant through an it(...) so multiple failures are received

* add newlines to multiline formatters, add fmt.stringify, allow format overrides

* stringify invalid return values from plugins

* move config validation errors into packages/errors, properly highlight and stringify values

* add component testing yarn commands

* fix the arrow not showing on details

* fix typescript error

* fixed lots of poorly written tests that weren't actually testing anything. created settings validation error when given a string validation result.

* fixes tests

* fixes tests, adds new error template for validating within an array list (for browser validation)

* remove dupe line

* fix copy for consistency, update snapshots

* remove redundant errors, standardize formatting and phrasing

* more formatting

* remove excess snapshots

* prune out excessive error snapshot html files when not in CI

* add missing tests, add case for when config validation fails without a fileType

* fixes test

* update snapshot

* update snapshot

* update snapshot

* sort uniqErrors + errorKeys prior to assertion - assert one by one

* add system test for binding to an event with the wrong handler

* fixes tests

* set more descriptive errors when setting invalid plugin events, or during plugin event validation

* remove duplicate PLUGINS_EVENT_ERROR, collapse into existing error

* use the same multiline formatting as @packages/errors

* standardize verbiage and highlighting for consistency

* fix incorrect error path

* fixes tests, standardized and condensed more language

* Update packages/errors/src/errors.ts

Co-authored-by: Ryan Manuel <ryanm@cypress.io>

* Update guides/error-handling.md

Co-authored-by: Ryan Manuel <ryanm@cypress.io>

* Update guides/error-handling.md

Co-authored-by: Ryan Manuel <ryanm@cypress.io>

* added some final todo's

* fix types

Co-authored-by: Tim Griesser <tgriesser10@gmail.com>
Co-authored-by: Tim Griesser <tgriesser@gmail.com>
Co-authored-by: Ryan Manuel <ryanm@cypress.io>
2022-02-11 02:06:07 -05:00

4.3 KiB

Error Handling in Cypress

Clear, consistent, errors are one of the important parts of the Cypress experience. When something goes wrong, there should be clear, actionable feedback for the user about exactly what went wrong, where it went wrong, and next steps on how to fix.

@packages/errors

All error related logic for the server should be added to @packages/errors. This logic has been separated out from the @packages/server to enable strict type checking & use in other packages we have added in the 10.0-release branch.

Summary of the Errors package:

  • errors.ts: A key/value mapping of known errors to functions returning "ErrorTemplates", described below, also includes/re-exports several helper utilities:
    • get / getError: builds & retrieves the error as a CypressError, should be the main way we retrieve errors throughout Cypress. Aliased as errors.get for existing use in the server package
    • throw / throwErr: Get & throw the error, so we can spy/stub it in a test. Aliased as errors.throw for existing use in the server package
    • logWarning: Logs the error as a warning to the console, aliased as errors.log for existing use in the server package
  • errTemplate.ts: Tagged template literal formatting the error as described below
  • stackUtils.ts: Utilities for working with a stack trace, extended by the driver package

errTemplate

The errTemplate is a tagged template literal. It allows us to maintain consistent behavior & formatting in our error messages, when we see a variable, we format it depending on the target environment.

The error message returns a message that defaults to being formatted for the terminal, and has a forBrowser method which returns the error message where the variables are wrapped in backticks '`' for Markdown display in the browser.

Return Value of errTemplate (ErrTemplateResult):

{
  // Will always exist, this is the terminal-formatted error message
  message: string, 
  // Will always exist, this is the browser-formatted error message
  messageMarkdown: string, 
  details?: string, // Exists if there is `details()` call in the errTemplate
  originalError?: ErrorLike // Exists if an error was passed into the `details()`
}

Example:

CANNOT_TRASH_ASSETS: (arg1: string) => {
  return errTemplate`\
      Warning: We failed to trash the existing run results.

      This error will not alter the exit code.

      ${details(arg1)}`
},  

In this case, arg1 will be highlighted in yellow when printed to the terminal.

PLUGINS_FILE_ERROR: (arg1: string, arg2: Error) => {
  return errTemplate`\
      The plugins file is missing or invalid.

      Your \`pluginsFile\` is set to ${arg1}, but either the file is missing, it contains a syntax error, or threw an error when required. The \`pluginsFile\` must be a \`.js\`, \`.ts\`, or \`.coffee\` file.

      Or you might have renamed the extension of your \`pluginsFile\`. If that's the case, restart the test runner.

      Please fix this, or set \`pluginsFile\` to \`false\` if a plugins file is not necessary for your project.

      ${details(arg2)}
    `
},

arg1 will be highlighted in blue for the terminal, and wrapped in backticks when called with forBrowser. Details will be printed in yellow as a stack trace when printed to the terminal, or shown as a stack-trace in the browser.

Error Wrapping

Any time we know about an edge case that is an error, we should define an error in errors.ts. This error should be retrieved by getError, which converts it to a CypressError.

The CypressError is an Error containing the message returned from the errTemplate. The stack is set to that of the originalError if it exists (i.e. the error object passed into details), otherwise it's the stack from where the getError / throwError is called.

The CypressError has an isCypressErr prop which we use as a duck-type guard against exiting the process when logging exceptions. It also maintains a reference to the originalError if it exists.

Child-Process Errors

All errors that occur in a child process spawned by Cypress should be sent over the ipc bridge using util.serializeError.

This ensures the name, message, stack, and any other relevant details are preserved and can be handled by the standard process of Cypress' error standardization / wrapping.