* chore: set up feature/test-burn-in feature branch * feat: add burnIn Configuration option (currently a no-op) (#27377) * feat: add the burnIn Configuration to the config package. Option currently is a no-op * chore: make burn in experimental * chore: set experimentalBurnIn to false by default * feat: add new experimental retries configuration (#27412) * feat: implement the experimental retries configuration options to pair with test burn in * [run ci] * fix cache invalidation [run ci] * fix snapshot added in v13 for module api to include test burn in experimentalflag * chore: fix merge conflict * chore: add burnInTestAction capability (#27768) * add burnInTestAction capability * feat: add burn in capability for cloud * chore: fix snapshot for record_spec * feat: implement experimental retries (#27826) * chore: format the retries/runner snapshot files to make diff easier * feat: implement experimentalRetries strategies 'detect-flake-and-pass-on-threshold' and 'detect-flake-but-always-fail'. This should not be a breaking change, though it does modify mocha and the test object even when the experiment is not configured. This is to exercise the system and make sure things still work as expected even when we go GA. Test updates will follow in following commits. * chore: update snapshots from system tests and cy-in-cy tests that now have the cypress test metadata property _cypressTestStatusInfo. tests have been added in the fail-with-[before|after]each specs to visually see the suite being skipped when developing. * chore: add cy-in-cy tests to verify reporter behavior for pass/fail tests, as well as new mocha snapshots to verify attempts. New tests were needed for this as the 'retries' option in testConfigOverrides currently is and will be invalid for experiment and will function as an override. tests run in the cy-in-cy tests are using globally configured experimentalRetries for the given tested project, which showcases the different behavior between attempts/retries and pass/fail status. * chore: add unit test like driver test to verify the test object in mocha is decorated/handled properly in calculateTestStatus * chore: add sanity system tests to verify console reporter output for experimental retries logic. Currently there is a bug in the reporter where the logged status doesnt wait for the aftereach to complete, which impacts the total exitCode and printed status. * fix: aftereach console output. make sure to fail the test in the appropriate spot in runner.ts and not prematurely, which in turn updates the snapshots for cy-in-cy as the fail event comes later." * chore: address comments from code review * fix: make sure hook failures print outer status + attempts when the error is the hook itself. * chore: improve types within calculateTestStatus inside mocha.ts * Revert "feat: add burnIn Configuration option (currently a no-op) (#27377)" This reverts commitc428443079. * Revert "chore: add burnInTestAction capability (#27768)" This reverts commitae3df1a505. * chore: run snapshot and binary jobs against experimental retries feature branch * chore: add changelog entry (wip) * Revert "fix snapshot added in v13 for module api to include test burn in experimentalflag" This reverts commitbb5046c91e. * Fix system tests * Clear CircleCI cache * Normalize retries config for test execution * Fixed some unit tests * update snapshots for newer test metadata * Fix cy-in-cy snapshots * update snapshots * bump cache version * chore: ensure legacy retry overrides work; reject exp. retries overrides (#28045) * update changelog * flip if statement in experimental retries option validation * refactor invalid experimental retry override for more useful error msg * revert testConfigOverrides snapshot * update snapshots for test override sys test * Update packages/config/src/validation.ts Co-authored-by: Chris Breiding <chrisbreiding@users.noreply.github.com> * succinct changelog entry; links to docs for details * testConfigOverride system test snapshots * Update .github/workflows/update_v8_snapshot_cache.yml Co-authored-by: Ryan Manuel <ryanm@cypress.io> * Update cli/CHANGELOG.md Co-authored-by: Ryan Manuel <ryanm@cypress.io> * Update packages/driver/src/cypress.ts Co-authored-by: Ryan Manuel <ryanm@cypress.io> * updating cache-version * improve typescript usage when appending experimental retry options to experiments in Experimenets.vue * Revert "improve typescript usage when appending experimental retry options to experiments in Experimenets.vue" This reverts commitb459aba882. * refactor test config override validation for experimental retry subkeys * account for error throw differences in browsers in system tests * bump circle cache * bump circle cache again --------- Co-authored-by: astone123 <adams@cypress.io> Co-authored-by: mabela416 <mabel@cypress.io> Co-authored-by: Muaz Othman <muaz@cypress.io> Co-authored-by: Muaz Othman <muazweb@gmail.com> Co-authored-by: Cacie Prins <cacie@cypress.io> Co-authored-by: Cacie Prins <cacieprins@users.noreply.github.com> Co-authored-by: Chris Breiding <chrisbreiding@users.noreply.github.com> Co-authored-by: Ryan Manuel <ryanm@cypress.io> Co-authored-by: Matthew Schile <mschile@cypress.io>
CLI
The CLI is used to build the cypress npm module to be run within a terminal.
The CLI has the following responsibilities:
- Allow users to print CLI commands
- Allow users to install the Cypress executable
- Allow users to print their current Cypress version
- Allow users to run Cypress tests from the terminal
- Allow users to open Cypress in the interactive Test Runner.
- Allow users to verify that Cypress is installed correctly and executable
- Allow users to manages the Cypress binary cache
- Allow users to pass in options that change way tests are ran or recorded (browsers used, specfiles ran, grouping, parallelization)
Building
See scripts/build.js. Note that the built npm package will include NPM_README.md as its public README file.
Testing
Automated
From the repo's root, you can run unit tests with:
yarn test-unit --scope cypress
yarn test-watch --scope cypress
yarn test-debug --scope cypress
Updating snapshots
Prepend SNAPSHOT_UPDATE=1 to any test command. See snap-shot-it instructions for more info.
SNAPSHOT_UPDATE=1 yarn test-unit --scope cypress
Type Linting
When testing with dtslint, you may need to remove existing typescript installations before running the type linter (for instance, on OS X, you might rm -rf ~/.dts/typescript-installs) in order to reproduce issues with new versions of typescript (i.e., @next).
Manual
To build and test an NPM package:
yarnyarn build
This creates build folder.
cd build; yarn pack
This creates an archive, usually named cypress-v<version>.tgz. You can install this archive from other projects, but because there is no corresponding binary yet (probably), skip binary download. For example from inside cypress-example-kitchensink folder
yarn add ~/{your-dirs}/cypress/cli/build/cypress-3.3.1.tgz --ignore-scripts
Which installs the tgz file we have just built from folder Users/jane-lane/{your-dirs}/cypress/cli/build.
Sub-package API
How do deep imports from cypress/* get resolved?
The cypress npm package comes pre-assembled with mounting libraries for major front-end frameworks. These mounting libraries are the first examples of Cypress providing re-exported sub-packages. These sub-packages follow the same naming convention they do when they're published on npm, but without a leading @ sign. For example:
An example of a sub-package: @cypress/vue, @cypress/react, @cypress/mount-utils
Let's discuss the Vue mounting library that Cypress ships.
If you'd installed the @cypress/vue package from NPM, you could write the following code.
This would be necessary when trying to use a version of Vue, React, or other library that may be newer or older than the current version of cypress itself.
import { mount } from '@cypress/vue'
Now, with the sub-package API, you're able to import the latest APIs directly from Cypress without needing to install a separate dependency.
import { mount } from 'cypress/vue'
The only difference is the import name, and if you still need to use a specific version of one of our external sub-packages, you may install it and import it directly.
Adding a new sub-package
There are a few steps when adding a new sub-package.
- Make sure the sub-package's rollup build is self-contained or that any dependencies are also declared in the CLI's
package.json. - Now, in the
postbuildscript for the sub-package you'd like to embed, invokenode ./scripts/sync-exported-npm-with-cli.js(relative to the sub-package, seenpm/vuefor an example). - Add the sub-package's name to the following locations:
cli/.gitignorecli/scripts/post-build.js.eslintignore(under cli/sub-package)
- DO NOT manually update the package.json file. Running
yarn buildwill automate this process. - Commit the changed files.
Here is an example Pull Request
Module API
The module API can be tested locally using something like:
/* @ts-ignore */
import cypress from '../../cli/lib/cypress'
const run = cypress.run as (options?: Partial<CypressCommandLine.CypressRunOptions>) => Promise<CypressCommandLine.CypressRunResult | CypressCommandLine.CypressFailedRunResult>
run({
spec: './cypress/component/advanced/framer-motion/Motion.spec.tsx',
testingType: 'component',
/* @ts-ignore */
dev: true,
}).then(results => {
console.log(results)
})
Note that the dev flag is required for local testing, as otherwise the command will fail with a binary error.