mirror/cypress
1
0
Fork 0
mirror of https://github.com/cypress-io/cypress.git synced 2026-01-01 20:19:58 -06:00
Code Issues Packages Projects Releases Wiki Activity

feat: experimental retries (#27930)

Browse Source 
* chore: set up feature/test-burn-in feature branch

* feat: add burnIn Configuration option (currently a no-op) (#27377)

* feat: add the burnIn Configuration to the config package. Option
currently is a no-op

* chore: make burn in experimental

* chore: set experimentalBurnIn to false by default

* feat: add new experimental retries configuration (#27412)

* feat: implement the experimental retries configuration options to pair
with test burn in

* [run ci]

* fix cache invalidation [run ci]

* fix snapshot added in v13 for module api to include test burn in experimentalflag

* chore: fix merge conflict

* chore: add burnInTestAction capability (#27768)

* add burnInTestAction capability

* feat: add burn in capability for cloud

* chore: fix snapshot for record_spec

* feat: implement experimental retries (#27826)

* chore: format the retries/runner snapshot files to make diff easier

* feat: implement experimentalRetries strategies 'detect-flake-and-pass-on-threshold' and 'detect-flake-but-always-fail'. This should not be a breaking change, though it does modify mocha and the test object even when the experiment is not configured. This is to exercise the system and make sure things still work as expected even when we go GA. Test updates will follow in following commits.

* chore: update snapshots from system tests and cy-in-cy tests that now have the cypress test metadata property _cypressTestStatusInfo. tests have been added in the fail-with-[before|after]each specs to visually see the suite being skipped when developing.

* chore: add cy-in-cy tests to verify reporter behavior for pass/fail tests, as well as new mocha snapshots to verify attempts. New tests were needed for this as the 'retries' option in testConfigOverrides currently is and will be invalid for experiment and will function as an override. tests run in the cy-in-cy tests are using globally configured experimentalRetries for the given tested project, which showcases the different behavior between attempts/retries and pass/fail status.

* chore: add unit test like driver test to verify the test object in mocha is decorated/handled properly in calculateTestStatus

* chore: add sanity system tests to verify console reporter output for experimental retries logic. Currently there is a bug in the reporter where the logged status doesnt wait for the aftereach to complete, which impacts the total exitCode and printed status.

* fix: aftereach console output. make sure to fail the test in the appropriate spot in runner.ts and not prematurely, which in turn updates the snapshots for cy-in-cy as the fail event comes later."

* chore: address comments from code review

* fix: make sure hook failures print outer status + attempts when the error is the hook itself.

* chore: improve types within calculateTestStatus inside mocha.ts

* Revert "feat: add burnIn Configuration option (currently a no-op) (#27377)"

This reverts commit c428443079.

* Revert "chore: add burnInTestAction capability (#27768)"

This reverts commit ae3df1a505.

* chore: run snapshot and binary jobs against experimental retries feature branch

* chore: add changelog entry (wip)

* Revert "fix snapshot added in v13 for module api to include test burn in experimentalflag"

This reverts commit bb5046c91e.

* Fix system tests

* Clear CircleCI cache

* Normalize retries config for test execution

* Fixed some unit tests

* update snapshots for newer test metadata

* Fix cy-in-cy snapshots

* update snapshots

* bump cache version

* chore: ensure legacy retry overrides work; reject exp. retries overrides (#28045)

* update changelog

* flip if statement in experimental retries option validation

* refactor invalid experimental retry override for more useful error msg

* revert testConfigOverrides snapshot

* update snapshots for test override sys test

* Update packages/config/src/validation.ts

Co-authored-by: Chris Breiding <chrisbreiding@users.noreply.github.com>

* succinct changelog entry; links to docs for details

* testConfigOverride system test snapshots

* Update .github/workflows/update_v8_snapshot_cache.yml

Co-authored-by: Ryan Manuel <ryanm@cypress.io>

* Update cli/CHANGELOG.md

Co-authored-by: Ryan Manuel <ryanm@cypress.io>

* Update packages/driver/src/cypress.ts

Co-authored-by: Ryan Manuel <ryanm@cypress.io>

* updating cache-version

* improve typescript usage when appending experimental retry options to experiments in Experimenets.vue

* Revert "improve typescript usage when appending experimental retry options to experiments in Experimenets.vue"

This reverts commit b459aba882.

* refactor test config override validation for experimental retry subkeys

* account for error throw differences in browsers in system tests

* bump circle cache

* bump circle cache again

---------

Co-authored-by: astone123 <adams@cypress.io>
Co-authored-by: mabela416 <mabel@cypress.io>
Co-authored-by: Muaz Othman <muaz@cypress.io>
Co-authored-by: Muaz Othman <muazweb@gmail.com>
Co-authored-by: Cacie Prins <cacie@cypress.io>
Co-authored-by: Cacie Prins <cacieprins@users.noreply.github.com>
Co-authored-by: Chris Breiding <chrisbreiding@users.noreply.github.com>
Co-authored-by: Ryan Manuel <ryanm@cypress.io>
Co-authored-by: Matthew Schile <mschile@cypress.io>
This commit is contained in:
Bill Glesias
2023-10-26 14:06:14 -04:00
committed by GitHub
parent 80cc83d0aa
commit 201e9f366e
65 changed files with 100783 additions and 9355 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
.circleci/cache-version.txt
View File

@@ -1,3 +1,3 @@
# Bump this version to force CI to re-create the cache from scratch.
10-18-23
10-25-23.1

13
.circleci/workflows.yml
View File

@@ -29,6 +29,7 @@ mainBuildFilters: &mainBuildFilters
- develop
- /^release\/\d+\.\d+\.\d+$/
# use the following branch as well to ensure that v8 snapshot cache updates are fully tested
- 'feature/experimental-retries'
- 'publish-binary'
- 'ryanm/fix/better-sqlite3'
@@ -41,6 +42,9 @@ macWorkflowFilters: &darwin-workflow-filters
- equal: [ develop, << pipeline.git.branch >> ]
# use the following branch as well to ensure that v8 snapshot cache updates are fully tested
- equal: [ 'update-v8-snapshot-cache-on-develop', << pipeline.git.branch >> ]
- equal: [ 'feature/experimental-retries', << pipeline.git.branch >> ]
- equal: [ 'chore/update_webpack_deps_to_latest_webpack4_compat', << pipeline.git.branch >> ]
- equal: [ 'lerna-optimize-tasks', << pipeline.git.branch >> ]
- equal: [ 'ryanm/fix/better-sqlite3', << pipeline.git.branch >> ]
- matches:
pattern: /^release\/\d+\.\d+\.\d+$/
@@ -52,6 +56,10 @@ linuxArm64WorkflowFilters: &linux-arm64-workflow-filters
- equal: [ develop, << pipeline.git.branch >> ]
# use the following branch as well to ensure that v8 snapshot cache updates are fully tested
- equal: [ 'update-v8-snapshot-cache-on-develop', << pipeline.git.branch >> ]
- equal: [ 'feature/experimental-retries', << pipeline.git.branch >> ]
- equal: [ 'chore/update_webpack_deps_to_latest_webpack4_compat', << pipeline.git.branch >> ]
- equal: [ 'chore/bump_loaders_and_optimize_webpack', << pipeline.git.branch >> ]
- equal: [ 'lerna-optimize-tasks', << pipeline.git.branch >> ]
- equal: [ 'ryanm/fix/better-sqlite3', << pipeline.git.branch >> ]
- matches:
pattern: /^release\/\d+\.\d+\.\d+$/
@@ -75,6 +83,9 @@ windowsWorkflowFilters: &windows-workflow-filters
- equal: [ develop, << pipeline.git.branch >> ]
# use the following branch as well to ensure that v8 snapshot cache updates are fully tested
- equal: [ 'update-v8-snapshot-cache-on-develop', << pipeline.git.branch >> ]
- equal: [ 'feature/experimental-retries', << pipeline.git.branch >> ]
- equal: [ 'chore/update_webpack_deps_to_latest_webpack4_compat', << pipeline.git.branch >> ]
- equal: [ 'lerna-optimize-tasks', << pipeline.git.branch >> ]
- equal: [ 'ryanm/fix/better-sqlite3', << pipeline.git.branch >> ]
- matches:
pattern: /^release\/\d+\.\d+\.\d+$/
@@ -145,7 +156,7 @@ commands:
name: Set environment variable to determine whether or not to persist artifacts
command: |
echo "Setting SHOULD_PERSIST_ARTIFACTS variable"
echo 'if ! [[ "$CIRCLE_BRANCH" != "develop" && "$CIRCLE_BRANCH" != "release/"* && "$CIRCLE_BRANCH" != "publish-binary" && "$CIRCLE_BRANCH" != "update-v8-snapshot-cache-on-develop" && "$CIRCLE_BRANCH" != "ryanm/fix/better-sqlite3" ]]; then
echo 'if ! [[ "$CIRCLE_BRANCH" != "develop" && "$CIRCLE_BRANCH" != "release/"* && "$CIRCLE_BRANCH" != "publish-binary" && "$CIRCLE_BRANCH" != "lerna-optimize-tasks" && "$CIRCLE_BRANCH" != "feature/experimental-retries" ]]; then
export SHOULD_PERSIST_ARTIFACTS=true
fi' >> "$BASH_ENV"
# You must run `setup_should_persist_artifacts` command and be using bash before running this command

9
cli/CHANGELOG.md
View File

@@ -1,7 +1,12 @@
<!-- See the ../guides/writing-the-cypress-changelog.md for details on writing the changelog. -->
## 13.3.4
_Released 11/7/2023 (PENDING)_
## 13.4.0
_Released 10/25/2023 (PENDING)_
**Features:**
- Introduced experimental configuration options for advanced retry logic: adds `experimentalStrategy` and `experimentalOptions` keys to the `retry` configuration key. See [Experimental Flake Detection Features](https://docs.cypress.io/guides/references/experiments/#Experimental-Flake-Detection-Features) in the documentation. Addressed in [#27930](https://github.com/cypress-io/cypress/pull/27930).
**Bugfixes:**

26
cli/types/cypress.d.ts vendored
View File

@@ -2851,6 +2851,30 @@ declare namespace Cypress {
certs: PEMCert[] | PFXCert[]
}
type RetryStrategyWithModeSpecs = RetryStrategy & {
openMode: boolean; // defaults to false
runMode: boolean; // defaults to true
}
type RetryStrategy =
| RetryStrategyDetectFlakeAndPassOnThresholdType
| RetryStrategyDetectFlakeButAlwaysFailType
interface RetryStrategyDetectFlakeAndPassOnThresholdType {
experimentalStrategy: "detect-flake-and-pass-on-threshold"
experimentalOptions?: {
maxRetries: number; // defaults to 2 if experimentalOptions is not provided, must be a whole number > 0
passesRequired: number; // defaults to 2 if experimentalOptions is not provided, must be a whole number > 0 and <= maxRetries
}
}
interface RetryStrategyDetectFlakeButAlwaysFailType {
experimentalStrategy: "detect-flake-but-always-fail"
experimentalOptions?: {
maxRetries: number; // defaults to 2 if experimentalOptions is not provided, must be a whole number > 0
stopIfAnyPassed: boolean; // defaults to false if experimentalOptions is not provided
}
}
interface ResolvedConfigOptions<ComponentDevServerOpts = any> {
/**
* Url used as prefix for [cy.visit()](https://on.cypress.io/visit) or [cy.request()](https://on.cypress.io/request) command's url
@@ -3122,7 +3146,7 @@ declare namespace Cypress {
* To enable test retries only in runMode, set e.g. `{ openMode: null, runMode: 2 }`
* @default null
*/
retries: Nullable<number | { runMode?: Nullable<number>, openMode?: Nullable<number> }>
retries: Nullable<number | ({ runMode?: Nullable<number>, openMode?: Nullable<number> }) | RetryStrategyWithModeSpecs>
/**
* Enables including elements within the shadow DOM when using querying
* commands (e.g. cy.get(), cy.find()). Can be set globally in cypress.config.{js,ts,mjs,cjs},

33
cli/types/tests/cypress-tests.ts
View File

@@ -1159,6 +1159,39 @@ namespace CypressLocalStorageTests {
cy.clearAllSessionStorage({ log: 'true' }) // $ExpectError
}
namespace CypressRetriesSpec {
Cypress.config('retries', {
openMode: 0,
runMode: 1
})
Cypress.config('retries', {
openMode: false,
runMode: false,
experimentalStrategy: "detect-flake-and-pass-on-threshold",
experimentalOptions: {
maxRetries: 2,
passesRequired: 2
}
})
Cypress.config('retries', {
openMode: false,
runMode: false,
experimentalStrategy: "detect-flake-but-always-fail",
experimentalOptions: {
maxRetries: 2,
stopIfAnyPassed: true
}
})
Cypress.config('retries', { openMode: false, runMode: true, experimentalStrategy: "detect-flake-and-pass-on-threshold", experimentalOptions: { maxRetries: 2} }) // $ExpectError
Cypress.config('retries', { openMode: false, runMode: true, experimentalStrategy: "detect-flake-but-always-fail", experimentalOptions: { maxRetries: 2} }) // $ExpectError
Cypress.config('retries', { openMode: false, runMode: true, experimentalStrategy: "detect-flake-and-pass-on-threshold", experimentalOptions: { passesRequired: 2} }) // $ExpectError
Cypress.config('retries', { openMode: false, runMode: true, experimentalStrategy: "detect-flake-but-always-fail", experimentalOptions: { stopIfAnyPassed: true} }) // $ExpectError
}
namespace CypressTraversalTests {
cy.wrap({}).prevUntil('a') // $ExpectType Chainable<JQuery<HTMLAnchorElement>>
cy.wrap({}).prevUntil('#myItem') // $ExpectType Chainable<JQuery<HTMLElement>>

413
packages/app/cypress/e2e/runner/retries.experimentalRetries.mochaEvents.cy.ts Normal file
View File

@@ -0,0 +1,413 @@
import { loadSpec, runSpec } from './support/spec-loader'
import { runCypressInCypressMochaEventsTest } from './support/mochaEventsUtils'
import { snapshots } from './retries.experimentalRetries.mochaEvents.snapshots'
/**
* The mochaEvent tests require a spec to be loaded and executed within an inner Cypress context.
* The spec must load and execute within the duration of the Cypress command timeout.
* The execution time of the inner Cypress is resource/OS dependant and can exceed the default value (4s),
* so we have increased the command timeout to allow the inner spec more time to complete and report
* its mocha event log.
*/
/**
* In context to experimentalRetries, what is considered a passed/failed test is going to vary depending on the project.
* 'detect-flake-and-pass-on-threshold' will have a passed test as long as the config passesRequired is satisfied on retries.
* 'detect-flake-but-always-fail' projects should have the same passed/failed tests, but different number of attempts depending on stopIfAnyPassed
*/
describe('Experimental retries: mochaEvents & test status tests', { retries: 0, defaultCommandTimeout: 7500 }, () => {
const projects: ['detect-flake-and-pass-on-threshold', 'detect-flake-but-always-fail', 'detect-flake-but-always-fail-stop-any-passed'] = ['detect-flake-and-pass-on-threshold', 'detect-flake-but-always-fail', 'detect-flake-but-always-fail-stop-any-passed']
projects.forEach((project) => {
describe(project, () => {
// This will differ per strategy
// for each project:
// 'detect-flake-and-pass-on-threshold': will run a total of 6 times. The first attempt will fail and the next 5 attempts will pass. The test passes.
// 'detect-flake-but-always-fail': will run a total of 10 times. The first attempt will fail and the next 9 attempts will pass. The test fails.
// 'detect-flake-but-always-fail-stop-any-passed': will run a total of 2 times. The first attempt will fail and the second attempt will pass. The test fails.
describe('simple retry', () => {
it('matches mocha snapshot', (done) => {
const { assertMatchingSnapshot } = runCypressInCypressMochaEventsTest(
snapshots,
`"${project}": retries mochaEvents simple retry #1`,
done,
)
runSpec({
fileName: 'experimental-retries/simple-fail.retries.mochaEvents.cy.js',
projectName: project,
}).then((win) => {
assertMatchingSnapshot(win)
})
})
it('displays correct passed/failed tests', () => {
loadSpec({
filePath: 'runner/experimental-retries/simple-fail.retries.mochaEvents.cy.js',
projectName: project,
passCount: project === 'detect-flake-and-pass-on-threshold' ? 1 : 0,
failCount: project === 'detect-flake-and-pass-on-threshold' ? 0 : 1,
})
})
})
// This will differ per strategy
// for each project:
// 'detect-flake-and-pass-on-threshold': will run a total of 6 times. The first attempt will fail and the next 5 attempts will pass. The test passes.
// 'detect-flake-but-always-fail': will run a total of 10 times. The first attempt will fail and the next 9 attempts will pass. The test fails.
// 'detect-flake-but-always-fail-stop-any-passed': will run a total of 2 times. The first attempt will fail and the second attempt will pass. The test fails.
describe('test retry with hooks', () => {
it('matches mocha snapshot', (done) => {
const { assertMatchingSnapshot } = runCypressInCypressMochaEventsTest(
snapshots,
`"${project}": retries mochaEvents test retry with hooks #1`,
done,
)
runSpec({
fileName: 'experimental-retries/test-retry-with-hooks.retries.mochaEvents.cy.js',
projectName: project,
}).then((win) => {
assertMatchingSnapshot(win)
})
})
it('displays correct passed/failed tests', () => {
loadSpec({
filePath: 'runner/experimental-retries/test-retry-with-hooks.retries.mochaEvents.cy.js',
projectName: project,
passCount: project === 'detect-flake-and-pass-on-threshold' ? 1 : 0,
failCount: project === 'detect-flake-and-pass-on-threshold' ? 0 : 1,
})
})
})
// This will differ per strategy
// for each project:
// 'detect-flake-and-pass-on-threshold': will run a total of 6 times. The first attempt will fail and the next 5 attempts will pass. The test passes.
// 'detect-flake-but-always-fail': will run a total of 10 times. The first attempt will fail and the next 9 attempts will pass. The test fails.
// 'detect-flake-but-always-fail-stop-any-passed': will run a total of 2 times. The first attempt will fail and the second attempt will pass. The test fails.
describe('test retry with [only]', () => {
it('matches mocha snapshot', (done) => {
const { assertMatchingSnapshot } = runCypressInCypressMochaEventsTest(
snapshots,
`"${project}": retries mochaEvents test retry with [only] #1`,
done,
)
runSpec({
fileName: 'experimental-retries/test-retry-with-only.retries.mochaEvents.cy.js',
projectName: project,
}).then((win) => {
assertMatchingSnapshot(win)
})
})
it('displays correct passed/failed tests', () => {
loadSpec({
filePath: 'runner/experimental-retries/test-retry-with-only.retries.mochaEvents.cy.js',
projectName: project,
passCount: project === 'detect-flake-and-pass-on-threshold' ? 1 : 0,
failCount: project === 'detect-flake-and-pass-on-threshold' ? 0 : 1,
})
})
})
// This will differ per strategy
// for each project:
// 'detect-flake-and-pass-on-threshold': will run a total of 6 times. The first attempt will fail and the next 5 attempts will pass. The test passes.
// 'detect-flake-but-always-fail': will run a total of 10 times. The first attempt will fail and the next 9 attempts will pass. The test fails.
// 'detect-flake-but-always-fail-stop-any-passed': will run a total of 2 times. The first attempt will fail and the second attempt will pass. The test fails.
describe('can retry from [beforeEach]', () => {
it('matches mocha snapshot', (done) => {
const { assertMatchingSnapshot } = runCypressInCypressMochaEventsTest(
snapshots,
`"${project}": retries mochaEvents can retry from [beforeEach] #1`,
done,
)
runSpec({
fileName: 'experimental-retries/can-retry-from-beforeEach.retries.mochaEvents.cy.js',
projectName: project,
}).then((win) => {
assertMatchingSnapshot(win)
})
})
it('displays correct passed/failed tests', () => {
loadSpec({
filePath: 'runner/experimental-retries/can-retry-from-beforeEach.retries.mochaEvents.cy.js',
projectName: project,
passCount: project === 'detect-flake-and-pass-on-threshold' ? 1 : 0,
failCount: project === 'detect-flake-and-pass-on-threshold' ? 0 : 1,
})
})
})
/**
* This will differ per strategy. The spec under test is a bit more complex
* for each project:
*
* 'detect-flake-and-pass-on-threshold':
*
* Suite 1 (passed)
* Test 1 (passed)
* Attempt 1 (failed)
* Attempt 2 (passed)
* Attempt 3 (passed)
* Attempt 4 (passed)
* Attempt 5 (passed)
* Attempt 6 (passed) (passesRequired met)
* Test 2 (passed)
* Attempt 1 (passed)
* Test 3 (passed)
* Attempt 1 (passed)
*
* Suite 2 (passed)
* Test 1 (passed)
* Attempt 1 (failed)
* Attempt 2 (failed)
* Attempt 3 (passed)
* Attempt 4 (passed)
* Attempt 5 (passed)
* Attempt 6 (passed)
* Attempt 7 (passed) (passesRequired met)
*
* Suite 3 (passed)
* Test 1 (passed)
* Attempt 1 (passed)
*
* FINAL RESULT:
* 5 tests passed / 0 tests failed
*
* 'detect-flake-but-always-fail':
*
* Suite 1 (failed)
* Test 1 (failed)
* Attempt 1 (failed)
* Attempt 2 (passed)
* Attempt 3 (passed)
* Attempt 4 (passed)
* Attempt 5 (passed)
* Attempt 6 (passed)
* Attempt 7 (passed)
* Attempt 8 (passed)
* Attempt 9 (passed)
* Attempt 10 (passed) (maxRetries achieved because stopIfAnyPassed=false, fail the test because flaky)
* Test 2 (passed)
* Attempt 1 (passed)
* Test 3 (passed)
* Attempt 1 (passed)
*
* Suite 2 (failed)
* Test 1 (failed)
* Attempt 1 (failed)
* Attempt 2 (failed)
* Attempt 3 (passed)
* Attempt 4 (passed)
* Attempt 5 (passed)
* Attempt 6 (passed)
* Attempt 7 (passed)
* Attempt 8 (passed)
* Attempt 9 (passed)
* Attempt 10 (passed) (maxRetries achieved because stopIfAnyPassed=false, fail the test because flaky)
*
* Suite 3 (passed)
* Test 1 (passed)
* Attempt 1 (passed)
*
* FINAL RESULT:
* 3 tests passed / 2 tests failed
*
* 'detect-flake-but-always-fail-stop-any-passed':
*
* Suite 1 (failed)
* Test 1 (failed)
* Attempt 1 (failed)
* Attempt 2 (passed) (stopIfAnyPassed=true so stop attempts, fail the test because flaky)
* Test 2 (passed)
* Attempt 1 (passed)
* Test 3 (passed)
* Attempt 1 (passed)
*
* Suite 2 (failed)
* Test 1 (failed)
* Attempt 1 (failed)
* Attempt 2 (failed)
* Attempt 3 (passed) (stopIfAnyPassed=true so stop attempts, fail the test because flaky)
*
* Suite 3 (passed)
* Test 1 (passed)
* Attempt 1 (passed)
*
* FINAL RESULT:
* 3 tests passed / 2 tests failed
*/
describe('can retry from [afterEach]', () => {
it('matches mocha snapshot', (done) => {
const { assertMatchingSnapshot } = runCypressInCypressMochaEventsTest(
snapshots,
`"${project}": retries mochaEvents can retry from [afterEach] #1`,
done,
)
runSpec({
fileName: 'experimental-retries/can-retry-from-afterEach.retries.mochaEvents.cy.js',
projectName: project,
}).then((win) => {
assertMatchingSnapshot(win)
})
})
it('displays correct passed/failed tests', () => {
loadSpec({
filePath: 'runner/experimental-retries/can-retry-from-afterEach.retries.mochaEvents.cy.js',
projectName: project,
passCount: project === 'detect-flake-and-pass-on-threshold' ? 5 : 3,
failCount: project === 'detect-flake-and-pass-on-threshold' ? 0 : 2,
})
})
})
// this is the same for each test strategy. If the before hook fails the who suite is skipped and tests aren't run
describe('cant retry from [before]', () => {
it('matches mocha snapshot', (done) => {
const { assertMatchingSnapshot } = runCypressInCypressMochaEventsTest(
snapshots,
`"${project}": retries mochaEvents cant retry from [before] #1`,
done,
)
runSpec({
fileName: 'experimental-retries/cant-retry-from-before.retries.mochaEvents.cy.js',
projectName: project,
}).then((win) => {
assertMatchingSnapshot(win)
})
})
it('displays correct passed/failed tests', () => {
loadSpec({
filePath: 'runner/experimental-retries/cant-retry-from-before.retries.mochaEvents.cy.js',
projectName: project,
passCount: 0,
failCount: 1,
})
})
})
/**
* This will differ per strategy. The spec under test is a bit more complex
* for each project:
*
* 'detect-flake-and-pass-on-threshold':
*
* Suite 1 (passed)
* Test 1 (passed)
* Attempt 1 (passed)
* Test 2 (passed)
* Attempt 1 (failed)
* Attempt 2 (failed)
* Attempt 3 (passed)
* Attempt 4 (passed)
* Attempt 5 (passed)
* Attempt 6 (passed)
* Attempt 7 (passed) (passesRequired met)
* Test 3 (passed)
* Attempt 1 (passed)
*
* FINAL RESULT:
* 3 tests passed / 0 tests failed
*
* 'detect-flake-but-always-fail':
*
* Suite 1 (failed)
* Test 1 (passed)
* Attempt 1 (passed)
* Test 2 (failed)
* Attempt 1 (failed)
* Attempt 2 (failed)
* Attempt 3 (passed)
* Attempt 4 (passed)
* Attempt 5 (passed)
* Attempt 6 (passed)
* Attempt 7 (passed)
* Attempt 8 (passed)
* Attempt 9 (passed)
* Attempt 10 (passed) (maxRetries achieved because stopIfAnyPassed=false, fail the test because flaky)
* Test 3 (passed)
* Attempt 1 (passed)
*
* FINAL RESULT:
* 2 tests passed / 1 tests failed
*
* 'detect-flake-but-always-fail-stop-any-passed':
*
* Suite 1 (failed)
* Test 1 (passed)
* Attempt 1 (passed)
* Test 2 (failed)
* Attempt 1 (failed)
* Attempt 2 (failed)
* Attempt 3 (passed) (stopIfAnyPassed=true so stop attempts, fail the test because flaky)
* Test 3 (passed)
* Attempt 1 (passed)
*
* FINAL RESULT:
* 2 tests passed / 1 tests failed
*/
describe('three tests with retry', () => {
it('matches mocha snapshot', (done) => {
const { assertMatchingSnapshot } = runCypressInCypressMochaEventsTest(
snapshots,
`"${project}": retries mochaEvents three tests with retry #1`,
done,
)
runSpec({
fileName: 'experimental-retries/three-tests-with-retry.retries.mochaEvents.cy.js',
projectName: project,
}).then((win) => {
assertMatchingSnapshot(win)
})
})
it('displays correct passed/failed tests', () => {
loadSpec({
filePath: 'runner/experimental-retries/three-tests-with-retry.retries.mochaEvents.cy.js',
projectName: project,
passCount: project === 'detect-flake-and-pass-on-threshold' ? 3 : 2,
failCount: project === 'detect-flake-and-pass-on-threshold' ? 0 : 1,
})
})
})
// This will differ per strategy
// for each project:
// 'detect-flake-and-pass-on-threshold': will run a total of 6 times. All attempts fail. The test fails
// 'detect-flake-but-always-fail': will run a total of 10 times. All attempts fail. The test fails.
// 'detect-flake-but-always-fail-stop-any-passed': will run a total of 10 times. All attempts fail. The test fails.
describe('cleanses errors before emitting', () => {
it('does not try to serialize error with err.actual as DOM node', function (done) {
// because there are more attempts for 'detect-flake-but-always-fail', the timeout needs to be increased
this.timeout(15000)
const { assertMatchingSnapshot } = runCypressInCypressMochaEventsTest(
snapshots,
`"${project}": retries mochaEvents cleanses errors before emitting does not try to serialize error with err.actual as DOM node #1`,
done,
)
runSpec({
fileName: 'experimental-retries/does-not-serialize-dom-error.cy.js',
projectName: project,
}).then((win) => {
// should not have err.actual, expected properties since the subject is a DOM element
assertMatchingSnapshot(win)
})
})
})
})
})
})

54372
packages/app/cypress/e2e/runner/retries.experimentalRetries.mochaEvents.snapshots.ts Normal file
View File

File diff suppressed because it is too large Load Diff

12158
packages/app/cypress/e2e/runner/retries.mochaEvents.snapshots.ts
View File

File diff suppressed because it is too large Load Diff

173
packages/app/cypress/e2e/runner/runner.experimentalRetries.mochaEvents.cy.ts Normal file
View File

@@ -0,0 +1,173 @@
import { runSpec } from './support/spec-loader'
import { runCypressInCypressMochaEventsTest } from './support/mochaEventsUtils'
import { snapshots } from './runner.experimentalRetries.mochaEvents.snapshots'
/**
* The mochaEvent tests require a spec to be loaded and executed within an inner Cypress context.
* The spec must load and execute within the duration of the Cypress command timeout.
* The execution time of the inner Cypress is resource/OS dependant and can exceed the default value (4s),
* so we have increased the command timeout to allow the inner spec more time to complete and report
* its mocha event log.
*/
/**
* In context to experimentalRetries, the end state of the tests should be identical regardless of strategy for these tests.
* However, the total amount of attempts on a test will differ based on the strategy used and is documented below
*/
describe('experimental retries: runner tests', { defaultCommandTimeout: 7500 }, () => {
const projects: ['detect-flake-and-pass-on-threshold', 'detect-flake-but-always-fail', 'detect-flake-but-always-fail-stop-any-passed'] = ['detect-flake-and-pass-on-threshold', 'detect-flake-but-always-fail', 'detect-flake-but-always-fail-stop-any-passed']
projects.forEach((project) => {
describe(project, () => {
describe('tests finish with correct state', () => {
describe('hook failures', () => {
// regardless of strategy, this should fail the suite immediately and not run any additional attempts, so the snapshots should be near identical
it(`fail in [before]`, (done) => {
const { assertMatchingSnapshot } = runCypressInCypressMochaEventsTest(
snapshots,
`"${project}": tests finish with correct state hook failures fail in [before] #1`,
done,
)
runSpec({
fileName: 'fail-with-before.mochaEvents.cy.js',
projectName: project,
}).then((win) => {
assertMatchingSnapshot(win)
})
})
// This will differ per strategy
// the snapshots for 'detect-flake-and-always-fail' configurations should almost be identical, regardless of experimentalOptions configuration.
// for each project:
// 'detect-flake-and-pass-on-threshold': will run a total of 6 times and fail 6 times, config is satisfied, the test fails, and the suite is skipped
// 'detect-flake-but-always-fail': will run a total of 10 times and fail 10 times, config is satisfied, the test fails, and the suite is skipped
// 'detect-flake-but-always-fail-stop-any-passed': will run a total of 10 times and fail 10 times config is satisfied, the test fails, and the suite is skipped
it(`fail in [beforeEach]`, (done) => {
const { assertMatchingSnapshot } = runCypressInCypressMochaEventsTest(
snapshots,
`"${project}": tests finish with correct state hook failures fail in [beforeEach] #1`,
done,
)
runSpec({
fileName: 'fail-with-beforeEach.mochaEvents.cy.js',
projectName: project,
}).then((win) => {
assertMatchingSnapshot(win)
})
})
// regardless of strategy, this should fail the suite immediately and not run any additional attempts, so the snapshots should be near identical
it(`fail in [after]`, (done) => {
const { assertMatchingSnapshot } = runCypressInCypressMochaEventsTest(
snapshots,
`"${project}": tests finish with correct state hook failures fail in [after] #1`,
done,
)
runSpec({
fileName: 'fail-with-after.mochaEvents.cy.js',
projectName: project,
}).then((win) => {
assertMatchingSnapshot(win)
})
})
// This will differ per strategy
// the snapshots for 'detect-flake-and-always-fail' configurations should almost be identical, regardless of experimentalOptions configuration.
// for each project:
// 'detect-flake-and-pass-on-threshold': will run a total of 6 times and fail 6 times, config is satisfied, the test fails, and the suite is skipped
// 'detect-flake-but-always-fail': will run a total of 10 times and fail 10 times, config is satisfied, the test fails, and the suite is skipped
// 'detect-flake-but-always-fail-stop-any-passed': will run a total of 10 times and fail 10 times config is satisfied, the test fails, and the suite is skipped
it(`fail in [afterEach]`, (done) => {
const { assertMatchingSnapshot } = runCypressInCypressMochaEventsTest(
snapshots,
`"${project}": tests finish with correct state hook failures fail in [afterEach] #1`,
done,
)
runSpec({
fileName: 'fail-with-afterEach.mochaEvents.cy.js',
projectName: project,
}).then((win) => {
assertMatchingSnapshot(win)
})
})
})
})
describe('mocha grep', () => {
// This will differ per strategy
// the snapshots for 'detect-flake-and-always-fail' configurations should almost be identical, regardless of experimentalOptions configuration.
// for each project:
// 'detect-flake-and-pass-on-threshold': will run a total of 6 times and fail 6 times, config is satisfied, the test fails, but the suite is NOT skipped
// 'detect-flake-but-always-fail': will run a total of 10 times and fail 10 times, config is satisfied, the test fails, but the suite is NOT skipped
// 'detect-flake-but-always-fail-stop-any-passed': will run a total of 10 times and fail 10 times config is satisfied, the test fails,but the suite is NOT skipped
it('fail with [only]', (done) => {
const { assertMatchingSnapshot } = runCypressInCypressMochaEventsTest(
snapshots,
`"${project}": tests finish with correct state mocha grep fail with [only] #1`,
done,
)
runSpec({
fileName: 'fail-with-only.mochaEvents.cy.js',
projectName: project,
}).then((win) => {
assertMatchingSnapshot(win)
})
})
// This will be the same per strategy, as the test passes and retries don't get invoked
it('pass with [only]', (done) => {
const { assertMatchingSnapshot } = runCypressInCypressMochaEventsTest(
snapshots,
`"${project}": tests finish with correct state mocha grep pass with [only] #1`,
done,
)
runSpec({
fileName: 'pass-with-only.mochaEvents.cy.js',
projectName: project,
}).then((win) => {
assertMatchingSnapshot(win)
})
})
})
})
// these should be the same per strategy, as each test passes and retries is not invoked
describe('mocha events', () => {
it('simple single test', (done) => {
const { assertMatchingSnapshot } = runCypressInCypressMochaEventsTest(
snapshots,
`"${project}": mocha events simple single test #1`,
done,
)
runSpec({
fileName: 'simple-single-test.mochaEvents.cy.js',
projectName: project,
}).then((win) => {
assertMatchingSnapshot(win)
})
})
it('simple three tests', (done) => {
const { assertMatchingSnapshot } = runCypressInCypressMochaEventsTest(
snapshots,
`"${project}": mocha events simple three tests #1`,
done,
)
runSpec({
fileName: 'three-tests-with-hooks.mochaEvents.cy.js',
projectName: project,
}).then((win) => {
assertMatchingSnapshot(win)
})
})
})
})
})

30737
packages/app/cypress/e2e/runner/runner.experimentalRetries.mochaEvents.snapshots.ts Normal file
View File

File diff suppressed because it is too large Load Diff

6985
packages/app/cypress/e2e/runner/runner.mochaEvents.snapshots.ts
View File

File diff suppressed because it is too large Load Diff

4
packages/app/cypress/e2e/runner/support/mochaEventsUtils.ts
View File

@@ -108,6 +108,10 @@ const eventCleanseMap = {
const keysToEliminate = ['codeFrame', '_testConfig'] as const
function removeUnusedKeysForTestSnapshot<T> (obj: T): T {
// with experimental retries, mocha can fire a 'retry' event with an undefined error
// this is expected
if (obj === undefined) return obj
for (const key of keysToEliminate) {
delete obj[key]
}

13
packages/app/cypress/e2e/runner/support/spec-loader.ts
View File

@@ -1,3 +1,5 @@
import type { ProjectFixtureDir } from '@tooling/system-tests/lib/fixtureDirs'
export const shouldHaveTestResults = ({ passCount, failCount, pendingCount }) => {
passCount = passCount || '--'
failCount = failCount || '--'
@@ -13,6 +15,8 @@ export const shouldHaveTestResults = ({ passCount, failCount, pendingCount }) =>
})
}
type ExperimentalRetriesProjects = 'detect-flake-and-pass-on-threshold' | 'detect-flake-but-always-fail' | 'detect-flake-but-always-fail-stop-any-passed'
export type LoadSpecOptions = {
filePath: string
setup?: () => void
@@ -20,7 +24,7 @@ export type LoadSpecOptions = {
failCount?: number | string
pendingCount?: number | string
hasPreferredIde?: boolean
projectName?: 'runner-e2e-specs' | 'runner-ct-specs' | 'session-and-origin-e2e-specs'
projectName?: 'runner-e2e-specs' | 'runner-ct-specs' | 'session-and-origin-e2e-specs' | ExperimentalRetriesProjects
mode?: 'e2e' | 'component'
configFile?: string
scaffold?: boolean
@@ -82,9 +86,10 @@ export function loadSpec (options: LoadSpecOptions) {
shouldHaveTestResults({ passCount, failCount, pendingCount })
}
export function runSpec ({ fileName }: { fileName: string }) {
cy.scaffoldProject('runner-e2e-specs')
cy.openProject('runner-e2e-specs')
export function runSpec ({ fileName, projectName }: { fileName: string, projectName?: ProjectFixtureDir }) {
projectName = projectName || 'runner-e2e-specs'
cy.scaffoldProject(projectName)
cy.openProject(projectName)
cy.startAppServer()
cy.visitApp(`specs/runner?file=cypress/e2e/runner/${fileName}`)

7
packages/app/src/runner/event-manager.ts
View File

@@ -32,6 +32,7 @@ let crossOriginOnMessageRef = ({ data, source }: MessageEvent<{
return undefined
}
let crossOriginLogs: {[key: string]: Cypress.Log} = {}
let hasMochaRunEnded: boolean = false
interface AddGlobalListenerOptions {
element: AutomationElementId
@@ -564,12 +565,14 @@ export class EventManager {
})
Cypress.on('run:start', async () => {
hasMochaRunEnded = false
if (Cypress.config('experimentalMemoryManagement') && Cypress.isBrowser({ family: 'chromium' })) {
await Cypress.backend('start:memory:profiling', Cypress.config('spec'))
}
})
Cypress.on('run:end', async () => {
hasMochaRunEnded = true
if (Cypress.config('experimentalMemoryManagement') && Cypress.isBrowser({ family: 'chromium' })) {
await Cypress.backend('end:memory:profiling')
}
@@ -720,8 +723,8 @@ export class EventManager {
Cypress.primaryOriginCommunicator.on('after:screenshot', handleAfterScreenshot)
Cypress.primaryOriginCommunicator.on('log:added', (attrs) => {
// If the test is over and the user enters interactive snapshot mode, do not add cross origin logs to the test runner.
if (Cypress.state('test')?.final) return
// If the mocha run is over and the user enters interactive snapshot mode, do not add cross origin logs to the test runner.
if (hasMochaRunEnded) return
// Create a new local log representation of the cross origin log.
// It will be attached to the current command.

24
packages/app/src/settings/project/Experiments.vue
View File

@@ -54,13 +54,33 @@ const props = defineProps<{
}>()
const localExperiments = computed(() => {
return props.gql?.config ? (props.gql.config as CypressResolvedConfig).filter((item) => item.field.startsWith('experimental')).map((configItem) => {
// get experiments out of the config
const experimentalConfigurations = props.gql?.config ? (props.gql.config as CypressResolvedConfig).filter((item) => item.field.startsWith('experimental')) : []
// get experimental retry properties on the 'retries' config object. Mutate the experimentalConfigurations array as to not have side effects with props.gql.config
// TODO: remove this once experimentalRetries becomes GA. This is to be treated as a one off as supported nested experiments inside config is rare.
const { value: { experimentalStrategy, experimentalOptions, from } } = props.gql?.config.find((item) => item.field === 'retries')
experimentalConfigurations.push({
field: 'retries.experimentalStrategy',
from,
value: experimentalStrategy,
})
experimentalConfigurations.push({
field: 'retries.experimentalOptions',
from,
value: experimentalOptions,
})
// end TODO removal
return experimentalConfigurations.map((configItem) => {
return {
key: configItem.field,
name: t(`settingsPage.experiments.${configItem.field}.name`),
enabled: !!configItem.value,
description: t(`settingsPage.experiments.${configItem.field}.description`),
}
}) : []
})
})
</script>

238
packages/config/__snapshots__/validation.spec.ts.js
View File

@@ -28,7 +28,7 @@ exports['browsers list with a string'] = {
exports['invalid retry value'] = {
'key': 'mockConfigKey',
'value': '1',
'type': 'a positive number or null or an object with keys "openMode" and "runMode" with values of numbers or nulls',
'type': 'a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy',
}
exports['invalid retry object'] = {
@@ -36,7 +36,7 @@ exports['invalid retry object'] = {
'value': {
'fakeMode': 1,
},
'type': 'a positive number or null or an object with keys "openMode" and "runMode" with values of numbers or nulls',
'type': 'a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy',
}
exports['not qualified url'] = {
@@ -106,7 +106,7 @@ exports['null instead of a number'] = {
exports['config/src/validation .isValidClientCertificatesSet returns error message for certs not passed as an array array 1'] = {
'key': 'mockConfigKey',
'value': '1',
'type': 'a positive number or null or an object with keys "openMode" and "runMode" with values of numbers or nulls',
'type': 'a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy',
}
exports['config/src/validation .isValidClientCertificatesSet returns error message for certs object without url 1'] = {
@@ -261,3 +261,235 @@ exports['invalid upper bound'] = {
'value': 52,
'type': 'a valid CRF number between 1 & 51, 0 or false to disable compression, or true to use the default compression of 32',
}
exports['config/src/validation .isValidRetriesConfig experimental options fails with invalid strategy 1'] = {
'key': 'mockConfigKey',
'value': {
'experimentalStrategy': 'foo',
},
'type': 'a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy',
}
exports['config/src/validation .isValidRetriesConfig experimental options fails with invalid strategy w/ other options (valid) 1'] = {
'key': 'mockConfigKey',
'value': {
'runMode': true,
'openMode': false,
'experimentalStrategy': 'bar',
},
'type': 'a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy',
}
exports['config/src/validation .isValidRetriesConfig experimental options fails with detect-flake-but-always-fail: valid strategy w/ other invalid options with experiment 1'] = {
'key': 'mockConfigKey',
'value': {
'runMode': 1,
'openMode': 0,
'experimentalStrategy': 'detect-flake-but-always-fail',
},
'type': 'a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy',
}
exports['config/src/validation .isValidRetriesConfig experimental options fails with detect-flake-but-always-fail: maxRetries is negative 1'] = {
'key': 'mockConfigKey',
'value': {
'experimentalStrategy': 'detect-flake-but-always-fail',
'experimentalOptions': {
'maxRetries': -2,
},
},
'type': 'a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy',
}
exports['config/src/validation .isValidRetriesConfig experimental options fails with detect-flake-but-always-fail: maxRetries is 0 1'] = {
'key': 'mockConfigKey',
'value': {
'experimentalStrategy': 'detect-flake-but-always-fail',
'experimentalOptions': {
'maxRetries': 0,
},
},
'type': 'a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy',
}
exports['config/src/validation .isValidRetriesConfig experimental options fails with detect-flake-but-always-fail: maxRetries is floating 1'] = {
'key': 'mockConfigKey',
'value': {
'experimentalStrategy': 'detect-flake-but-always-fail',
'experimentalOptions': {
'maxRetries': 3.5,
},
},
'type': 'a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy',
}
exports['config/src/validation .isValidRetriesConfig experimental options fails with experimentalStrategy is "detect-flake-but-always-fail" with only "maxRetries" in "experimentalOptions" 1'] = {
'key': 'mockConfigKey',
'value': {
'experimentalStrategy': 'detect-flake-but-always-fail',
'experimentalOptions': {
'maxRetries': 4,
},
},
'type': 'a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy',
}
exports['config/src/validation .isValidRetriesConfig experimental options fails with detect-flake-and-pass-on-threshold: valid strategy w/ other invalid options with experiment 1'] = {
'key': 'mockConfigKey',
'value': {
'runMode': 1,
'openMode': 0,
'experimentalStrategy': 'detect-flake-and-pass-on-threshold',
},
'type': 'a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy',
}
exports['config/src/validation .isValidRetriesConfig experimental options fails with detect-flake-and-pass-on-threshold: maxRetries is negative 1'] = {
'key': 'mockConfigKey',
'value': {
'experimentalStrategy': 'detect-flake-and-pass-on-threshold',
'experimentalOptions': {
'maxRetries': -2,
},
},
'type': 'a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy',
}
exports['config/src/validation .isValidRetriesConfig experimental options fails with detect-flake-and-pass-on-threshold: maxRetries is 0 1'] = {
'key': 'mockConfigKey',
'value': {
'experimentalStrategy': 'detect-flake-and-pass-on-threshold',
'experimentalOptions': {
'maxRetries': 0,
},
},
'type': 'a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy',
}
exports['config/src/validation .isValidRetriesConfig experimental options fails with detect-flake-and-pass-on-threshold: maxRetries is floating 1'] = {
'key': 'mockConfigKey',
'value': {
'experimentalStrategy': 'detect-flake-and-pass-on-threshold',
'experimentalOptions': {
'maxRetries': 3.5,
},
},
'type': 'a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy',
}
exports['config/src/validation .isValidRetriesConfig experimental options fails with experimentalStrategy is "detect-flake-and-pass-on-threshold" with only "maxRetries" in "experimentalOptions" 1'] = {
'key': 'mockConfigKey',
'value': {
'experimentalStrategy': 'detect-flake-and-pass-on-threshold',
'experimentalOptions': {
'maxRetries': 4,
},
},
'type': 'a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy',
}
exports['config/src/validation .isValidRetriesConfig experimental options fails with detect-flake-and-pass-on-threshold passesRequired is negative 1'] = {
'key': 'mockConfigKey',
'value': {
'experimentalStrategy': 'detect-flake-and-pass-on-threshold',
'experimentalOptions': {
'maxRetries': 1,
'passesRequired': -4,
},
},
'type': 'a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy',
}
exports['config/src/validation .isValidRetriesConfig experimental options fails with detect-flake-and-pass-on-threshold passesRequired is 0 1'] = {
'key': 'mockConfigKey',
'value': {
'experimentalStrategy': 'detect-flake-and-pass-on-threshold',
'experimentalOptions': {
'maxRetries': 1,
'passesRequired': 0,
},
},
'type': 'a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy',
}
exports['config/src/validation .isValidRetriesConfig experimental options fails with detect-flake-and-pass-on-threshold passesRequired is floating 1'] = {
'key': 'mockConfigKey',
'value': {
'experimentalStrategy': 'detect-flake-and-pass-on-threshold',
'experimentalOptions': {
'maxRetries': 1,
'passesRequired': 3.5,
},
},
'type': 'a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy',
}
exports['config/src/validation .isValidRetriesConfig experimental options fails with detect-flake-and-pass-on-threshold provides passesRequired without maxRetries 1'] = {
'key': 'mockConfigKey',
'value': {
'experimentalStrategy': 'detect-flake-and-pass-on-threshold',
'experimentalOptions': {
'passesRequired': 3,
},
},
'type': 'a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy',
}
exports['config/src/validation .isValidRetriesConfig experimental options fails with detect-flake-and-pass-on-threshold provides passesRequired that is greater than maxRetries 1'] = {
'key': 'mockConfigKey',
'value': {
'experimentalStrategy': 'detect-flake-and-pass-on-threshold',
'experimentalOptions': {
'maxRetries': 3,
'passesRequired': 5,
},
},
'type': 'a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy',
}
exports['config/src/validation .isValidRetriesConfig experimental options fails with detect-flake-and-pass-on-threshold provides stopIfAnyPassed option 1'] = {
'key': 'mockConfigKey',
'value': {
'experimentalStrategy': 'detect-flake-and-pass-on-threshold',
'experimentalOptions': {
'maxRetries': 3,
'stopIfAnyPassed': true,
},
},
'type': 'a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy',
}
exports['config/src/validation .isValidRetriesConfig experimental options fails with detect-flake-but-always-fail provides passesRequired option 1'] = {
'key': 'mockConfigKey',
'value': {
'experimentalStrategy': 'detect-flake-but-always-fail',
'experimentalOptions': {
'maxRetries': 3,
'passesRequired': 2,
},
},
'type': 'a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy',
}
exports['config/src/validation .isValidRetriesConfig experimental options fails with detect-flake-but-always-fail provides stopIfAnyPassed without maxRetries 1'] = {
'key': 'mockConfigKey',
'value': {
'experimentalStrategy': 'detect-flake-but-always-fail',
'experimentalOptions': {
'stopIfAnyPassed': false,
},
},
'type': 'a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy',
}
exports['config/src/validation .isValidRetriesConfig experimental options fails with detect-flake-but-always-fail stopIfAnyPassed is a number (0 and 1 do not work) 1'] = {
'key': 'mockConfigKey',
'value': {
'experimentalStrategy': 'detect-flake-but-always-fail',
'experimentalOptions': {
'maxRetries': 2,
'stopIfAnyPassed': 1,
},
},
'type': 'a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy',
}

20
packages/config/src/browser.ts
View File

@@ -197,6 +197,26 @@ export const validateOverridableAtRunTime = (config: any, isSuiteLevelOverride:
return
}
// this is unique validation, not applied to the general cy config.
// it will be removed when we support defining experimental retries
// in test config overrides
// TODO: remove when experimental retry overriding is supported
if (configKey === 'retries') {
const experimentalRetryCfgKeys = [
'experimentalStrategy', 'experimentalOptions',
]
Object.keys(config.retries || {})
.filter((v) => experimentalRetryCfgKeys.includes(v))
.forEach((invalidExperimentalCfgOverride) => {
onErr({
invalidConfigKey: `retries.${invalidExperimentalCfgOverride}`,
supportedOverrideLevel: 'global_only',
})
})
}
// TODO: add a hook to ensure valid testing-type configuration is being set at runtime for all configuration values.
// https://github.com/cypress-io/cypress/issues/24365

22
packages/config/src/options.ts
View File

@@ -355,10 +355,32 @@ const driverConfigOptions: Array<DriverConfigOption> = [
validation: validate.isNumber,
overrideLevel: 'any',
}, {
/**
* if experimentalStrategy is `detect-flake-and-pass-on-threshold`
* an no experimentalOptions are configured, the following configuration
* should be implicitly used:
* experimentalStrategy: 'detect-flake-and-pass-on-threshold',
* experimentalOptions: {
* maxRetries: 2,
* passesRequired: 2
* }
*
* if experimentalStrategy is `detect-flake-but-always-fail`
* an no experimentalOptions are configured, the following configuration
* should be implicitly used:
* experimentalStrategy: 'detect-flake-but-always-fail',
* experimentalOptions: {
* maxRetries: 2,
* stopIfAnyPassed: false
* }
*/
name: 'retries',
defaultValue: {
runMode: 0,
openMode: 0,
// these values MUST be populated in order to display the experiment correctly inside the project settings in open mode
experimentalStrategy: undefined,
experimentalOptions: undefined,
},
validation: validate.isValidRetriesConfig,
overrideLevel: 'any',

74
packages/config/src/validation.ts
View File

@@ -119,8 +119,50 @@ export const isValidBrowserList = (_key: string, browsers: any): ErrResult | tru
return true
}
const isValidExperimentalRetryOptionsConfig = (options: any, strategy: 'detect-flake-but-always-fail' | 'detect-flake-and-pass-on-threshold'): boolean => {
if (options == null) return true
// retries must be an integer of 1 or greater
const isValidMaxRetries = _.isInteger(options.maxRetries) && options.maxRetries > 0
if (!isValidMaxRetries) {
return false
}
// if the strategy is 'detect-flake-but-always-fail', stopIfAnyPassed must be provided and must be a boolean
if (strategy === 'detect-flake-but-always-fail') {
if (options.passesRequired !== undefined) {
return false
}
const isValidStopIfAnyPasses = _.isBoolean(options.stopIfAnyPassed)
if (!isValidStopIfAnyPasses) {
return false
}
}
// if the strategy is 'detect-flake-and-pass-on-threshold', passesRequired must be provided and must be an integer greater than 0
if (strategy === 'detect-flake-and-pass-on-threshold') {
if (options.stopIfAnyPassed !== undefined) {
return false
}
const isValidPassesRequired = _.isInteger(options.passesRequired) && options.passesRequired > 0 && options.passesRequired <= options.maxRetries
if (!isValidPassesRequired) {
return false
}
}
return true
}
export const isValidRetriesConfig = (key: string, value: any): ErrResult | true => {
const optionalKeys = ['runMode', 'openMode']
const experimentalOptions = ['experimentalStrategy', 'experimentalOptions']
const experimentalStrategyOptions = ['detect-flake-but-always-fail', 'detect-flake-and-pass-on-threshold']
const isValidRetryValue = (val: any) => _.isNull(val) || (Number.isInteger(val) && val >= 0)
const optionalKeysAreValid = (val: any, k: string) => optionalKeys.includes(k) && isValidRetryValue(val)
@@ -128,11 +170,37 @@ export const isValidRetriesConfig = (key: string, value: any): ErrResult | true
return true
}
if (_.isObject(value) && _.every(value, optionalKeysAreValid)) {
return true
if (_.isObject(value)) {
const traditionalConfigOptions = _.omit(value, experimentalOptions)
const experimentalConfigOptions = _.pick<any>(value, experimentalOptions)
const isTraditionalConfigValid = _.every(traditionalConfigOptions, optionalKeysAreValid)
// if optionalKeys are only present and are valid, return true.
// The defaults for 'experimentalStrategy' and 'experimentalOptions' are undefined, but the keys exist, so we need to check for this
if (isTraditionalConfigValid && !Object.keys(experimentalConfigOptions).filter((key) => experimentalConfigOptions[key] !== undefined).length) {
return true
}
// check experimental configuration. experimentalStrategy MUST be present if experimental config is provided and set to one of the provided enumerations
if (experimentalConfigOptions.experimentalStrategy) {
// make sure the strategy provided is one of our valid enums
const isValidStrategy = experimentalStrategyOptions.includes(experimentalConfigOptions.experimentalStrategy)
// if a strategy is provided, and traditional options are also provided, such as runMode and openMode, then these values need to be booleans
const openAndRunModeConfigOptions = _.pick(value, optionalKeys)
const isValidRunAndOpenModeConfigWithStrategy = _.every(openAndRunModeConfigOptions, _.isBoolean)
// if options aren't present (either undefined or null) or are configured correctly, return true
if (isValidStrategy && isValidRunAndOpenModeConfigWithStrategy && (
experimentalConfigOptions.experimentalOptions == null ||
isValidExperimentalRetryOptionsConfig(experimentalConfigOptions.experimentalOptions, experimentalConfigOptions.experimentalStrategy))) {
return true
}
}
}
return errMsg(key, value, 'a positive number or null or an object with keys "openMode" and "runMode" with values of numbers or nulls')
return errMsg(key, value, 'a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy')
}
/**

4
packages/config/test/project/utils.spec.ts
View File

@@ -1090,7 +1090,7 @@ describe('config/src/project/utils', () => {
reporterOptions: { value: null, from: 'default' },
requestTimeout: { value: 5000, from: 'default' },
responseTimeout: { value: 30000, from: 'default' },
retries: { value: { runMode: 0, openMode: 0 }, from: 'default' },
retries: { value: { runMode: 0, openMode: 0, experimentalStrategy: undefined, experimentalOptions: undefined }, from: 'default' },
screenshotOnRunFailure: { value: true, from: 'default' },
screenshotsFolder: { value: 'cypress/screenshots', from: 'default' },
specPattern: { value: 'cypress/e2e/**/*.cy.{js,jsx,ts,tsx}', from: 'default' },
@@ -1208,7 +1208,7 @@ describe('config/src/project/utils', () => {
reporterOptions: { value: null, from: 'default' },
requestTimeout: { value: 5000, from: 'default' },
responseTimeout: { value: 30000, from: 'default' },
retries: { value: { runMode: 0, openMode: 0 }, from: 'default' },
retries: { value: { runMode: 0, openMode: 0, experimentalStrategy: undefined, experimentalOptions: undefined }, from: 'default' },
screenshotOnRunFailure: { value: true, from: 'default' },
screenshotsFolder: { value: 'cypress/screenshots', from: 'default' },
slowTestThreshold: { value: 10000, from: 'default' },

283
packages/config/test/validation.spec.ts
View File

@@ -166,6 +166,289 @@ describe('config/src/validation', () => {
expect(result).to.not.be.true
snapshot('invalid retry object', result)
})
it('returns true for valid retry object with experimental keys (default)', () => {
let result = validation.isValidRetriesConfig(mockKey, {
openMode: 0,
runMode: 0,
experimentalStrategy: undefined,
experimentalOptions: undefined,
})
expect(result).to.be.true
})
describe('experimental options', () => {
describe('passes with', () => {
['detect-flake-but-always-fail', 'detect-flake-and-pass-on-threshold'].forEach((strategy) => {
it(`experimentalStrategy is "${strategy}" with no "experimentalOptions" & valid runMode and openMode`, () => {
let result = validation.isValidRetriesConfig(mockKey, {
runMode: true,
openMode: false,
experimentalStrategy: strategy,
})
expect(result).to.be.true
result = validation.isValidRetriesConfig(mockKey, {
experimentalStrategy: strategy,
})
expect(result).to.be.true
})
})
it('experimentalStrategy is "detect-flake-but-always-fail" and has option "stopIfAnyPassed"', () => {
let result = validation.isValidRetriesConfig(mockKey, {
runMode: true,
openMode: false,
experimentalStrategy: 'detect-flake-but-always-fail',
experimentalOptions: {
maxRetries: 1,
stopIfAnyPassed: true,
},
})
expect(result).to.be.true
result = validation.isValidRetriesConfig(mockKey, {
experimentalStrategy: 'detect-flake-but-always-fail',
experimentalOptions: {
maxRetries: 4,
stopIfAnyPassed: false,
},
})
expect(result).to.be.true
})
it('experimentalStrategy is "detect-flake-and-pass-on-threshold" and has option "passesRequired"', () => {
let result = validation.isValidRetriesConfig(mockKey, {
runMode: true,
openMode: false,
experimentalStrategy: 'detect-flake-and-pass-on-threshold',
experimentalOptions: {
maxRetries: 1,
passesRequired: 1,
},
})
expect(result).to.be.true
result = validation.isValidRetriesConfig(mockKey, {
experimentalStrategy: 'detect-flake-and-pass-on-threshold',
experimentalOptions: {
maxRetries: 4,
passesRequired: 2,
},
})
expect(result).to.be.true
})
})
describe('fails with', () => {
it('invalid strategy', () => {
const result = validation.isValidRetriesConfig(mockKey, {
experimentalStrategy: 'foo',
})
expect(result).to.not.be.true
snapshot(result)
})
it('invalid strategy w/ other options (valid)', () => {
const result = validation.isValidRetriesConfig(mockKey, {
runMode: true,
openMode: false,
experimentalStrategy: 'bar',
})
expect(result).to.not.be.true
snapshot(result)
})
;['detect-flake-but-always-fail', 'detect-flake-and-pass-on-threshold'].forEach((strategy) => {
it(`${strategy}: valid strategy w/ other invalid options with experiment`, () => {
const result = validation.isValidRetriesConfig(mockKey, {
runMode: 1,
openMode: 0,
experimentalStrategy: strategy,
})
expect(result).to.not.be.true
snapshot(result)
})
it(`${strategy}: maxRetries is negative`, () => {
const result = validation.isValidRetriesConfig(mockKey, {
experimentalStrategy: strategy,
experimentalOptions: {
maxRetries: -2,
},
})
expect(result).to.not.be.true
snapshot(result)
})
it(`${strategy}: maxRetries is 0`, () => {
const result = validation.isValidRetriesConfig(mockKey, {
experimentalStrategy: strategy,
experimentalOptions: {
maxRetries: 0,
},
})
expect(result).to.not.be.true
snapshot(result)
})
it(`${strategy}: maxRetries is floating`, () => {
const result = validation.isValidRetriesConfig(mockKey, {
experimentalStrategy: strategy,
experimentalOptions: {
maxRetries: 3.5,
},
})
expect(result).to.not.be.true
snapshot(result)
})
it(`experimentalStrategy is "${strategy}" with only "maxRetries" in "experimentalOptions"`, () => {
const result = validation.isValidRetriesConfig(mockKey, {
experimentalStrategy: strategy,
experimentalOptions: {
maxRetries: 4,
},
})
expect(result).to.not.be.true
snapshot(result)
})
})
describe('detect-flake-and-pass-on-threshold', () => {
it(`passesRequired is negative`, () => {
const result = validation.isValidRetriesConfig(mockKey, {
experimentalStrategy: 'detect-flake-and-pass-on-threshold',
experimentalOptions: {
maxRetries: 1,
passesRequired: -4,
},
})
expect(result).to.not.be.true
snapshot(result)
})
it(`passesRequired is 0`, () => {
const result = validation.isValidRetriesConfig(mockKey, {
experimentalStrategy: 'detect-flake-and-pass-on-threshold',
experimentalOptions: {
maxRetries: 1,
passesRequired: 0,
},
})
expect(result).to.not.be.true
snapshot(result)
})
it(`passesRequired is floating`, () => {
const result = validation.isValidRetriesConfig(mockKey, {
experimentalStrategy: 'detect-flake-and-pass-on-threshold',
experimentalOptions: {
maxRetries: 1,
passesRequired: 3.5,
},
})
expect(result).to.not.be.true
snapshot(result)
})
it('provides passesRequired without maxRetries', () => {
const result = validation.isValidRetriesConfig(mockKey, {
experimentalStrategy: 'detect-flake-and-pass-on-threshold',
experimentalOptions: {
passesRequired: 3,
},
})
expect(result).to.not.be.true
snapshot(result)
})
it('provides passesRequired that is greater than maxRetries', () => {
const result = validation.isValidRetriesConfig(mockKey, {
experimentalStrategy: 'detect-flake-and-pass-on-threshold',
experimentalOptions: {
maxRetries: 3,
passesRequired: 5,
},
})
expect(result).to.not.be.true
snapshot(result)
})
it('provides stopIfAnyPassed option', () => {
const result = validation.isValidRetriesConfig(mockKey, {
experimentalStrategy: 'detect-flake-and-pass-on-threshold',
experimentalOptions: {
maxRetries: 3,
stopIfAnyPassed: true,
},
})
expect(result).to.not.be.true
snapshot(result)
})
})
describe('detect-flake-but-always-fail', () => {
it('provides passesRequired option', () => {
const result = validation.isValidRetriesConfig(mockKey, {
experimentalStrategy: 'detect-flake-but-always-fail',
experimentalOptions: {
maxRetries: 3,
passesRequired: 2,
},
})
expect(result).to.not.be.true
snapshot(result)
})
it('provides stopIfAnyPassed without maxRetries', () => {
const result = validation.isValidRetriesConfig(mockKey, {
experimentalStrategy: 'detect-flake-but-always-fail',
experimentalOptions: {
stopIfAnyPassed: false,
},
})
expect(result).to.not.be.true
snapshot(result)
})
it('stopIfAnyPassed is a number (0 and 1 do not work)', () => {
const result = validation.isValidRetriesConfig(mockKey, {
experimentalStrategy: 'detect-flake-but-always-fail',
experimentalOptions: {
maxRetries: 2,
stopIfAnyPassed: 1,
},
})
expect(result).to.not.be.true
snapshot(result)
})
})
})
})
})
describe('.isPlainObject', () => {

308
packages/driver/cypress/e2e/util/mocha_custom_methods.cy.js Normal file
View File

@@ -0,0 +1,308 @@
import { calculateTestStatus } from '../../../src/cypress/mocha.ts'
describe('mocha custom methods', () => {
describe('calculateTestStatus', () => {
let totalRetries = 2
const createMockTest = (state = 'passed', prevAttempts = []) => {
const mockTestContext = {
currentRetry () {
return prevAttempts.length
},
retries () {
return totalRetries
},
state,
prevAttempts,
}
return Cypress._.cloneDeep(mockTestContext)
}
it('should never attempt to retry a test that passes on the first try, regardless of strategy', function () {
const undefinedStrategyTest = createMockTest()
const noExperimentalRetries = calculateTestStatus(undefinedStrategyTest)
expect(noExperimentalRetries.outerStatus).to.equal('passed')
expect(noExperimentalRetries.attempts).to.equal(1)
expect(noExperimentalRetries.shouldAttemptsContinue).to.be.false
expect(noExperimentalRetries.strategy).to.be.undefined
expect(undefinedStrategyTest.final).to.be.true
const detectFlakeAndPassOnThresholdStrategyTest = createMockTest()
const detectFlakeAndPassOnThreshold = calculateTestStatus(detectFlakeAndPassOnThresholdStrategyTest, {
strategy: 'detect-flake-and-pass-on-threshold',
maxRetries: 8,
passesRequired: 5,
})
expect(detectFlakeAndPassOnThreshold.outerStatus).to.equal('passed')
expect(detectFlakeAndPassOnThreshold.attempts).to.equal(1)
expect(detectFlakeAndPassOnThreshold.shouldAttemptsContinue).to.be.false
expect(detectFlakeAndPassOnThreshold.strategy).to.equal('detect-flake-and-pass-on-threshold')
expect(detectFlakeAndPassOnThresholdStrategyTest.final).to.be.true
const detectFlakeButAlwaysFailStrategyTest = createMockTest()
const detectFlakeButAlwaysFail = calculateTestStatus(detectFlakeButAlwaysFailStrategyTest, {
strategy: 'detect-flake-but-always-fail',
maxRetries: 8,
stopIfAnyPassed: false,
})
expect(detectFlakeButAlwaysFail.outerStatus).to.equal('passed')
expect(detectFlakeButAlwaysFail.attempts).to.equal(1)
expect(detectFlakeButAlwaysFail.shouldAttemptsContinue).to.be.false
expect(detectFlakeButAlwaysFail.strategy).to.equal('detect-flake-but-always-fail')
expect(detectFlakeButAlwaysFailStrategyTest.final).to.be.true
})
describe('undefined (GA implementation/original)', () => {
it('passed: keeps signaling to retry until test passes', function () {
const mockTest1 = createMockTest('failed')
const attempt1 = calculateTestStatus(mockTest1)
expect(attempt1.outerStatus).to.be.undefined
expect(attempt1.attempts).to.equal(1)
expect(attempt1.shouldAttemptsContinue).to.be.true
expect(attempt1.strategy).to.be.undefined
const mockTest2 = createMockTest('passed', [mockTest1])
const attempt2 = calculateTestStatus(mockTest2)
expect(attempt2.outerStatus).to.equal('passed')
expect(attempt2.attempts).to.equal(2)
expect(attempt2.shouldAttemptsContinue).to.be.false
expect(attempt2.strategy).to.be.undefined
})
// this logic is NOT inclusive of after/afterEach hooks, which can still set the test state after the test has calculated the meta data properties.
// this happens inside ./driver/src/cypress/runner.ts
it('failed: keeps signaling to retry until retry limit is reached', function () {
const mockTest1 = createMockTest('failed')
const attempt1 = calculateTestStatus(mockTest1)
expect(attempt1.outerStatus).to.be.undefined
expect(attempt1.attempts).to.equal(1)
expect(attempt1.shouldAttemptsContinue).to.be.true
expect(attempt1.strategy).to.be.undefined
const mockTest2 = createMockTest('failed', [mockTest1])
const attempt2 = calculateTestStatus(mockTest2)
expect(attempt2.outerStatus).to.be.undefined
expect(attempt2.attempts).to.equal(2)
expect(attempt2.shouldAttemptsContinue).to.be.true
expect(attempt2.strategy).to.be.undefined
const mockTest3 = createMockTest('failed', [mockTest1, mockTest2])
const attempt3 = calculateTestStatus(mockTest3)
expect(attempt3.outerStatus).to.equal('failed')
expect(attempt3.attempts).to.equal(3)
expect(attempt3.shouldAttemptsContinue).to.be.false
expect(attempt3.strategy).to.be.undefined
})
})
describe('detect-flake-and-pass-on-threshold', () => {
it('passed: no longer signals to retry test after passesRequired threshold is reached', function () {
totalRetries = 5
const mockTest1 = createMockTest('failed')
const attempt1 = calculateTestStatus(mockTest1, {
strategy: 'detect-flake-and-pass-on-threshold',
maxRetries: totalRetries,
passesRequired: 2,
})
expect(attempt1.outerStatus).to.be.undefined
expect(attempt1.attempts).to.equal(1)
expect(attempt1.shouldAttemptsContinue).to.be.true
expect(attempt1.strategy).to.equal('detect-flake-and-pass-on-threshold')
expect(mockTest1.final).to.be.false
const mockTest2 = createMockTest('failed', [mockTest1])
const attempt2 = calculateTestStatus(mockTest2, {
strategy: 'detect-flake-and-pass-on-threshold',
maxRetries: totalRetries,
passesRequired: 2,
})
expect(attempt2.outerStatus).to.be.undefined
expect(attempt2.attempts).to.equal(2)
expect(attempt2.shouldAttemptsContinue).to.be.true
expect(attempt2.strategy).to.equal('detect-flake-and-pass-on-threshold')
expect(mockTest2.final).to.be.false
const mockTest3 = createMockTest('passed', [mockTest1, mockTest2])
const attempt3 = calculateTestStatus(mockTest3, {
strategy: 'detect-flake-and-pass-on-threshold',
maxRetries: totalRetries,
passesRequired: 2,
})
expect(attempt3.outerStatus).to.be.undefined
expect(attempt3.attempts).to.equal(3)
expect(attempt3.shouldAttemptsContinue).to.be.true
expect(attempt3.strategy).to.equal('detect-flake-and-pass-on-threshold')
expect(mockTest3.final).to.be.false
const mockTest4 = createMockTest('passed', [mockTest1, mockTest2, mockTest3])
const attempt4 = calculateTestStatus(mockTest4, {
strategy: 'detect-flake-and-pass-on-threshold',
maxRetries: totalRetries,
passesRequired: 2,
})
expect(attempt4.outerStatus).to.equal('passed')
expect(attempt4.attempts).to.equal(4)
expect(attempt4.shouldAttemptsContinue).to.be.false
expect(attempt4.strategy).to.equal('detect-flake-and-pass-on-threshold')
expect(mockTest4.final).to.be.true
})
it('failed: no longer signals to retry test if the passesRequired is impossible to meet', function () {
totalRetries = 4
const mockTest1 = createMockTest('failed')
const attempt1 = calculateTestStatus(mockTest1, {
strategy: 'detect-flake-and-pass-on-threshold',
maxRetries: totalRetries,
passesRequired: 2,
})
expect(attempt1.outerStatus).to.be.undefined
expect(attempt1.attempts).to.equal(1)
expect(attempt1.shouldAttemptsContinue).to.be.true
expect(attempt1.strategy).to.equal('detect-flake-and-pass-on-threshold')
expect(mockTest1.final).to.be.false
const mockTest2 = createMockTest('failed', [mockTest1])
const attempt2 = calculateTestStatus(mockTest2, {
strategy: 'detect-flake-and-pass-on-threshold',
maxRetries: totalRetries,
passesRequired: 2,
})
expect(attempt2.outerStatus).to.be.undefined
expect(attempt2.attempts).to.equal(2)
expect(attempt2.shouldAttemptsContinue).to.be.true
expect(attempt2.strategy).to.equal('detect-flake-and-pass-on-threshold')
expect(mockTest2.final).to.be.false
const mockTest3 = createMockTest('failed', [mockTest1, mockTest2])
const attempt3 = calculateTestStatus(mockTest3, {
strategy: 'detect-flake-and-pass-on-threshold',
maxRetries: totalRetries,
passesRequired: 2,
})
expect(attempt3.outerStatus).to.be.undefined
expect(attempt3.attempts).to.equal(3)
expect(attempt3.shouldAttemptsContinue).to.be.true
expect(attempt3.strategy).to.equal('detect-flake-and-pass-on-threshold')
expect(mockTest3.final).to.be.false
const mockTest4 = createMockTest('failed', [mockTest1, mockTest2, mockTest3])
const attempt4 = calculateTestStatus(mockTest4, {
strategy: 'detect-flake-and-pass-on-threshold',
maxRetries: totalRetries,
passesRequired: 2,
})
expect(attempt4.outerStatus).to.equal('failed')
expect(attempt4.attempts).to.equal(4)
expect(attempt4.shouldAttemptsContinue).to.be.false
expect(attempt4.strategy).to.equal('detect-flake-and-pass-on-threshold')
expect(mockTest4.final).to.be.true
})
})
describe('detect-flake-but-always-fail', () => {
it('failed: no longer signals to retry after retries are exhausted', function () {
totalRetries = 3
const mockTest1 = createMockTest('failed')
const attempt1 = calculateTestStatus(mockTest1, {
strategy: 'detect-flake-but-always-fail',
maxRetries: totalRetries,
stopIfAnyPassed: false,
})
expect(attempt1.outerStatus).to.be.undefined
expect(attempt1.attempts).to.equal(1)
expect(attempt1.shouldAttemptsContinue).to.be.true
expect(attempt1.strategy).to.equal('detect-flake-but-always-fail')
expect(mockTest1.final).to.be.false
const mockTest2 = createMockTest('failed', [mockTest1])
const attempt2 = calculateTestStatus(mockTest2, {
strategy: 'detect-flake-but-always-fail',
maxRetries: totalRetries,
stopIfAnyPassed: false,
})
expect(attempt2.outerStatus).to.be.undefined
expect(attempt2.attempts).to.equal(2)
expect(attempt2.shouldAttemptsContinue).to.be.true
expect(attempt2.strategy).to.equal('detect-flake-but-always-fail')
expect(mockTest2.final).to.be.false
const mockTest3 = createMockTest('passed', [mockTest1, mockTest2])
const attempt3 = calculateTestStatus(mockTest3, {
strategy: 'detect-flake-but-always-fail',
maxRetries: totalRetries,
stopIfAnyPassed: false,
})
expect(attempt3.outerStatus).to.be.undefined
expect(attempt3.attempts).to.equal(3)
expect(attempt3.shouldAttemptsContinue).to.be.true
expect(attempt3.strategy).to.equal('detect-flake-but-always-fail')
expect(mockTest3.final).to.be.false
const mockTest4 = createMockTest('passed', [mockTest1, mockTest2, mockTest3])
const attempt4 = calculateTestStatus(mockTest4, {
strategy: 'detect-flake-but-always-fail',
maxRetries: totalRetries,
stopIfAnyPassed: false,
})
expect(attempt4.outerStatus).to.equal('failed')
expect(attempt4.attempts).to.equal(4)
expect(attempt4.shouldAttemptsContinue).to.be.false
expect(attempt4.strategy).to.equal('detect-flake-but-always-fail')
expect(mockTest4.final).to.be.true
// make sure forceState is called on 'detect-flake-but-always-fail' in the case the last test attempt passed, but the outer status should indicate a failure
expect(mockTest4.forceState).to.equal('passed')
})
it('failed: short circuits after a retry has a passed test', function () {
totalRetries = 3
const mockTest1 = createMockTest('failed')
const attempt1 = calculateTestStatus(mockTest1, {
strategy: 'detect-flake-but-always-fail',
maxRetries: totalRetries,
stopIfAnyPassed: true,
})
expect(attempt1.outerStatus).to.be.undefined
expect(attempt1.attempts).to.equal(1)
expect(attempt1.shouldAttemptsContinue).to.be.true
expect(attempt1.strategy).to.equal('detect-flake-but-always-fail')
expect(mockTest1.final).to.be.false
const mockTest2 = createMockTest('passed', [mockTest1])
const attempt2 = calculateTestStatus(mockTest2, {
strategy: 'detect-flake-but-always-fail',
maxRetries: totalRetries,
stopIfAnyPassed: true,
})
expect(attempt2.outerStatus).to.equal('failed')
expect(attempt2.attempts).to.equal(2)
expect(attempt2.shouldAttemptsContinue).to.be.false
expect(attempt2.strategy).to.equal('detect-flake-but-always-fail')
expect(mockTest2.final).to.true
// make sure forceState is called on 'detect-flake-but-always-fail' in the case the last test attempt passed, but the outer status should indicate a failure
expect(mockTest2.forceState).to.equal('passed')
})
})
})
})

85
packages/driver/patches/mocha+7.0.1.dev.patch
View File

@@ -11,3 +11,88 @@ index 0b43004..588e195 100644
runner.checkLeaks = options.checkLeaks === true;
runner.fullStackTrace = options.fullTrace;
runner.asyncOnly = options.asyncOnly;
diff --git a/node_modules/mocha/lib/runner.js b/node_modules/mocha/lib/runner.js
index ceb1a24..113ac40 100644
--- a/node_modules/mocha/lib/runner.js
+++ b/node_modules/mocha/lib/runner.js
@@ -677,9 +677,45 @@ Runner.prototype.runTests = function(suite, fn) {
}
self.emit(constants.EVENT_TEST_END, test);
return self.hookUp(HOOK_TYPE_AFTER_EACH, next);
- } else if (err) {
+ }
+ else if (err || test.hasAttemptPassed) {
+ if(test.hasAttemptPassed){
+ // Currently, to get passing attempts to rerun in mocha,
+ // we signal to mocha that we MIGHT need to retry a passed test attempt.
+ // If the test is run and there are no errors present, we assume a
+ // passed test attempt(set in ./driver/src/cypress/runner.ts)
+ test.state = STATE_PASSED
+ } else {
+ // Otherwise, we can assume the test attempt failed as 'err' would have to be present here.
+ test.state = STATE_FAILED
+ }
+
+ // Evaluate if the test should continue based on 'calculateTestStatus'.
+ // This is a custom method added by Cypress in ./driver/src/cypress/mocha.ts
+ var testStatusInfo = test.calculateTestStatus()
+
+ if(!testStatusInfo.shouldAttemptsContinue){
+ // If the test has met the exit condition, we need to grab the metadata from
+ // 'calculateTestStatus' in order to display and interpret the test outerStatus correctly.
+ test._cypressTestStatusInfo = testStatusInfo
+
+ if(testStatusInfo.attempts > 1) {
+ // If the test has been run AT LEAST twice (i.e. we are retrying), and the exit condition is met,
+ // modify mocha '_retries' to be the max retries made in order to possibly short circuit a suite
+ // if a hook has failed on every attempt (which we may not know at this stage of the test run).
+
+ // We will need the original retries to 'reset' the possible retries
+ // if the test attempt passes and fits the exit condition, BUT an 'afterEach' hook fails.
+ // In this case, we need to know how many retries we can reapply to satisfy the config.
+ test._maxRetries = test._retries
+ test._retries = test._currentRetry
+ }
+ }
+
var retry = test.currentRetry();
- if (retry < test.retries()) {
+
+ // requeue the test if we have retries and haven't satisfied our retry configuration.
+ if (retry < test.retries() && testStatusInfo.shouldAttemptsContinue) {
var clonedTest = test.clone();
clonedTest.currentRetry(retry + 1);
tests.unshift(clonedTest);
@@ -689,8 +725,25 @@ Runner.prototype.runTests = function(suite, fn) {
// Early return + hook trigger so that it doesn't
// increment the count wrong
return self.hookUp(HOOK_TYPE_AFTER_EACH, next);
- } else {
- self.fail(test, err);
+ } else if(testStatusInfo.outerStatus === STATE_FAILED) {
+ // However, if we have fit the exit condition and the outerStatus of the test is marked as 'failed'.
+
+ // We need to check the state of the last test attempt.
+ // In this case, if the strategy is "detect-flake-but-always-fail",
+ // has an outerStatus of 'failed', but the last test attempt passed, we still want to call the 'fail' hooks on the test, but keep
+ // the test attempt marked as passed.
+
+ // However, since the test might have afterEach/after hooks that mutate the state of the test
+ // (from passed to failed), the hooks might actually affect how many retries are actually run in order to satisfy the config.
+ // In this case, we want to delay failing as long as possible to make sure the test is settled, all attempts are run, and hooks
+ // can no longer retry. For this edge case specifically, the failing of the test in the runner lives in ./driver/src/cypress/runner.ts
+ if(test.state === STATE_FAILED){
+ self.fail(test, err)
+ }
+ } else if (testStatusInfo?.outerStatus === STATE_PASSED){
+ // There is no case where a test can 'pass' and the last test attempt be a failure,
+ // meaning we can assume a 'passed' outerStatus has a final passed test attempt.
+ self.emit(constants.EVENT_TEST_PASS, test);
}
self.emit(constants.EVENT_TEST_END, test);
return self.hookUp(HOOK_TYPE_AFTER_EACH, next);
@@ -1029,3 +1082,4 @@ Runner.constants = constants;
* @external EventEmitter
* @see {@link https://nodejs.org/api/events.html#events_class_eventemitter}
*/
+

11
packages/driver/src/cypress.ts
View File

@@ -278,7 +278,16 @@ class $Cypress {
}
if (_.isObject(testRetries)) {
return testRetries[this.config('isInteractive') ? 'openMode' : 'runMode']
const retriesAsNumberOrBoolean = testRetries[this.config('isInteractive') ? 'openMode' : 'runMode']
// If experimentalRetries are configured, an experimentalStrategy is present, and the retries configured is a boolean
// then we need to set the mocha '_retries' to 'maxRetries' present in the 'experimentalOptions' configuration.
if (testRetries['experimentalStrategy'] && _.isBoolean(retriesAsNumberOrBoolean) && retriesAsNumberOrBoolean) {
return testRetries['experimentalOptions'].maxRetries
}
// Otherwise, this is a number and falls back to default
return retriesAsNumberOrBoolean
}
return null

4
packages/driver/src/cypress/error_messages.ts
View File

@@ -272,6 +272,10 @@ export default {
message: `The \`{{invalidConfigKey}}\` configuration can only be overridden from a suite-level override.`,
docsUrl: 'https://on.cypress.io/config',
},
global_only: {
message: `The \`{{invalidConfigKey}}\` configuration can only be set globally.`,
docsUrl: 'https://on.cypress.io/config',
},
},
invalid_test_override: {
message: `The config passed to your {{overrideLevel}}-level overrides has the following validation error:\n\n{{errMsg}}`,

165
packages/driver/src/cypress/mocha.ts
View File

@@ -37,6 +37,104 @@ delete (window as any).Mocha
export const SKIPPED_DUE_TO_BROWSER_MESSAGE = ' (skipped due to browser)'
interface CypressTest extends Mocha.Test {
prevAttempts: CypressTest[]
final?: boolean
forceState?: 'passed'
}
type Strategy = 'detect-flake-and-pass-on-threshold' | 'detect-flake-but-always-fail' | undefined
type NormalizedRetriesConfig = {
strategy?: Strategy
maxRetries?: number
passesRequired?: number
stopIfAnyPassed?: boolean
}
// NOTE: 'calculateTestStatus' is marked as an individual function to make functionality easier to test.
export function calculateTestStatus (test: CypressTest, config?: NormalizedRetriesConfig) {
// @ts-expect-error
const totalAttemptsAlreadyExecuted = test.currentRetry() + 1
let shouldAttemptsContinue: boolean = true
let outerTestStatus: 'passed' | 'failed' | undefined = undefined
const passedTests = _.filter(test.prevAttempts, (o) => o.state === 'passed')
const failedTests = _.filter(test.prevAttempts, (o) => o.state === 'failed')
// Additionally, if the current test attempt passed/failed, add it to the attempt list
if (test.state === 'passed') {
passedTests.push(test)
} else if (test.state === 'failed') {
failedTests.push(test)
}
// If there is AT LEAST one failed test attempt, we know we need to apply retry logic.
// Otherwise, the test might be burning in (not implemented yet) OR the test passed on the first attempt,
// meaning retry logic does NOT need to be applied.
if (failedTests.length > 0) {
const maxAttempts = test.retries() + 1
const remainingAttempts = maxAttempts - totalAttemptsAlreadyExecuted
const passingAttempts = passedTests.length
// Below variables are used for when strategy is "detect-flake-and-pass-on-threshold" or no strategy is defined
let passesRequired = config?.strategy !== 'detect-flake-but-always-fail' ?
(config?.passesRequired || 1) :
null
const neededPassingAttemptsLeft = config?.strategy !== 'detect-flake-but-always-fail' ?
(passesRequired as number) - passingAttempts :
null
// Below variables are used for when strategy is only "detect-flake-but-always-fail"
let stopIfAnyPassed = config?.strategy === 'detect-flake-but-always-fail' ?
(config.stopIfAnyPassed || false) :
null
// Do we have the required amount of passes? If yes, we no longer need to keep running the test.
if (config?.strategy !== 'detect-flake-but-always-fail' && passingAttempts >= (passesRequired as number)) {
outerTestStatus = 'passed'
test.final = true
shouldAttemptsContinue = false
} else if (totalAttemptsAlreadyExecuted < maxAttempts &&
(
// For strategy "detect-flake-and-pass-on-threshold" or no strategy (current GA retries):
// If we haven't met our max attempt limit AND we have enough remaining attempts that can satisfy the passing requirement.
// retry the test.
(config?.strategy !== 'detect-flake-but-always-fail' && remainingAttempts >= (neededPassingAttemptsLeft as number)) ||
// For strategy "detect-flake-but-always-fail":
// If we haven't met our max attempt limit AND
// stopIfAnyPassed is false OR
// stopIfAnyPassed is true and no tests have passed yet.
// retry the test.
(config?.strategy === 'detect-flake-but-always-fail' && (!stopIfAnyPassed || stopIfAnyPassed && passingAttempts === 0))
)) {
test.final = false
shouldAttemptsContinue = true
} else {
// Otherwise, we should stop retrying the test.
outerTestStatus = 'failed'
test.final = true
// If an outerStatus is 'failed', but the last test attempt was 'passed', we need to force the status so mocha doesn't flag the test attempt as failed.
// This is a common use case with 'detect-flake-but-always-fail', where we want to display the last attempt as 'passed' but fail the test.
test.forceState = test.state === 'passed' ? test.state : undefined
shouldAttemptsContinue = false
}
} else {
// retry logic did not need to be applied and the test passed.
outerTestStatus = 'passed'
shouldAttemptsContinue = false
test.final = true
}
return {
strategy: config?.strategy,
shouldAttemptsContinue,
attempts: totalAttemptsAlreadyExecuted,
outerStatus: outerTestStatus,
}
}
type MochaArgs = [string, Function | undefined]
function createRunnable (ctx, fnType: 'Test' | 'Suite', mochaArgs: MochaArgs, runnableFn: Function, testCallback: Function | string = '', _testConfig?: Record<string, any>) {
const runnable = runnableFn.apply(ctx, mochaArgs)
@@ -221,6 +319,10 @@ const restoreTestClone = () => {
Test.prototype.clone = testClone
}
const removeCalculateTestStatus = () => {
delete Test.prototype.calculateTestStatus
}
const restoreRunnerRunTests = () => {
Runner.prototype.runTests = runnerRunTests
}
@@ -326,11 +428,72 @@ function patchTestClone () {
ret._testConfig = this._testConfig
ret.id = this.id
ret.order = this.order
ret._currentRetry = this._currentRetry
return ret
}
}
function getNormalizedRetriesConfig (Cypress: Cypress.Cypress): NormalizedRetriesConfig {
const retriesConfig = Cypress.config('retries')
const isInOpenMode = Cypress.config('isInteractive')
if (retriesConfig == null) {
return {}
}
if (typeof retriesConfig === 'number') {
return {
strategy: 'detect-flake-and-pass-on-threshold',
maxRetries: retriesConfig,
passesRequired: 1,
}
}
const enablementKey: 'openMode'|'runMode' = isInOpenMode ? 'openMode' : 'runMode'
const enablementValue = retriesConfig[enablementKey]
// if retries are explicitly disabled, return an empty object
if (enablementValue === false) {
return {}
}
// by default, retries are disabled in open mode
if (!enablementValue && isInOpenMode) {
return {}
}
if (typeof enablementValue === 'number') {
return {
strategy: 'detect-flake-and-pass-on-threshold',
maxRetries: enablementValue,
passesRequired: 1,
}
}
const config = retriesConfig as Cypress.RetryStrategy
// TODO: For GA, rename experimentalStrategy to strategy, experimentalOptions to options
return {
strategy: config.experimentalStrategy,
maxRetries: config.experimentalOptions?.maxRetries,
passesRequired: config.experimentalOptions?.['passesRequired'],
stopIfAnyPassed: config.experimentalOptions?.['stopIfAnyPassed'],
}
}
function createCalculateTestStatus (Cypress: Cypress.Cypress) {
// Adds a method to the test object called 'calculateTestStatus'
// which is used inside our mocha patch (./driver/patches/mocha+7.0.1.dev.patch)
// in order to calculate test retries. This prototype functions as a light abstraction around
// 'calculateTestStatus', which makes the function easier to unit-test
Test.prototype.calculateTestStatus = function () {
const retriesConfig = getNormalizedRetriesConfig(Cypress)
return calculateTestStatus(this, retriesConfig)
}
}
function patchRunnerRunTests () {
Runner.prototype.runTests = function () {
const suite = arguments[0]
@@ -495,6 +658,7 @@ const restore = () => {
restoreHookRetries()
restoreRunnerRunTests()
restoreTestClone()
removeCalculateTestStatus()
restoreSuiteAddTest()
restoreSuiteAddSuite()
restoreSuiteHooks()
@@ -509,6 +673,7 @@ const override = (specWindow, Cypress, config) => {
patchHookRetries()
patchRunnerRunTests()
patchTestClone()
createCalculateTestStatus(Cypress)
patchSuiteAddTest(specWindow, config)
patchSuiteAddSuite(specWindow, config)
patchSuiteHooks(specWindow, config)

130
packages/driver/src/cypress/runner.ts
View File

@@ -25,7 +25,7 @@ const RUNNABLE_AFTER_RUN_ASYNC_EVENT = 'runner:runnable:after:run:async'
const RUNNABLE_LOGS = ['routes', 'agents', 'commands', 'hooks'] as const
const RUNNABLE_PROPS = [
'_testConfig', 'id', 'order', 'title', '_titlePath', 'root', 'hookName', 'hookId', 'err', 'state', 'pending', 'failedFromHookId', 'body', 'speed', 'type', 'duration', 'wallClockStartedAt', 'wallClockDuration', 'timings', 'file', 'originalTitle', 'invocationDetails', 'final', 'currentRetry', 'retries', '_slow',
'_cypressTestStatusInfo', '_testConfig', 'id', 'order', 'title', '_titlePath', 'root', 'hookName', 'hookId', 'err', 'state', 'pending', 'failedFromHookId', 'body', 'speed', 'type', 'duration', 'wallClockStartedAt', 'wallClockDuration', 'timings', 'file', 'originalTitle', 'invocationDetails', 'final', 'currentRetry', 'retries', '_slow',
] as const
const debug = debugFn('cypress:driver:runner')
@@ -481,7 +481,26 @@ const overrideRunnerHook = (Cypress, _runner, getTestById, getTest, setTest, get
if (test.final !== false) {
test.final = true
if (test.state === 'passed') {
Cypress.action('runner:pass', wrap(test))
if (test?._cypressTestStatusInfo?.outerStatus === 'failed') {
// We call _runner.fail here instead of in mocha because we need to make sure none of the hooks mutate the current test state, which might trigger
// another attempt. This affects our server reporter by reporting the test final status multiple times and incorrect attempt statuses.
// We can be sure here that the test is settled and we can fail it appropriately if the condition is met.
// In this case, since the last attempt of the test does not contain an error, we need to look one up from a previous attempt
// and fail the last attempt with this error to appropriate the correct runner lifecycle hooks. However, we still want the
// last attempt to be marked as 'passed'. This is where 'forceState' comes into play (see 'calculateTestStatus' in ./driver/src/cypress/mocha.ts)
// If there are other hooks (such as multiple afterEach hooks) that MIGHT impact the end conditions of the test, we only want to fail this ONCE!
const lastTestWithErr = (test.prevAttempts || []).find((t) => t.state === 'failed')
// TODO: figure out serialization with this looked up error as it isn't printed to the console reporter properly.
const err = lastTestWithErr?.err
// fail the test as it would in the mocha/lib/runner.js as we can now be certain that no other hooks will impact the state of the test (regardless of hierarchy)
_runner.fail(test, err)
} else {
// If the last test attempt passed, but the outerStatus isn't marked as failed, then we want to emit the mocha 'pass' event.
Cypress.action('runner:pass', wrap(test))
}
}
Cypress.action('runner:test:end', wrap(test))
@@ -915,6 +934,11 @@ const setHookFailureProps = (test, hook, err) => {
test.duration = hook.duration // TODO: nope (?)
test.hookName = hookName // TODO: why are we doing this?
test.failedFromHookId = hook.hookId
// There should never be a case where the outerStatus of a test is set AND the last test attempt failed on a hook and the state is passed.
// Therefore, if the last test attempt fails on a hook, the outerStatus should also indicate a failure.
if (test?._cypressTestStatusInfo?.outerStatus) {
test._cypressTestStatusInfo.outerStatus = test.state
}
}
function getTestFromRunnable (runnable) {
@@ -1133,11 +1157,28 @@ const _runnerListeners = (_runner, Cypress, _emissions, getTestById, getTest, se
} else {
err = $errUtils.appendErrMsg(err, errMessage)
}
// If the test never failed and only the hooks did,
// we need to attach the metadata of the test to the hook to report the failure correctly to the server reporter.
// We calculate it fresh here since it may not be available on the test, which is the case with a beforeEach hook.
// as well as maybe incorrect (test passed on first attempt, but after hooks failed)
const testStatus = test.calculateTestStatus()
runnable._cypressTestStatusInfo = {
attempts: testStatus.attempts,
strategy: testStatus.strategy,
// regardless of the test state, we should ultimately fail the test here.
outerStatus: runnable.state,
shouldAttemptsContinue: false,
}
}
// always set runnable err so we can tap into
// taking a screenshot on error
runnable.err = $errUtils.wrapErr(err)
// If the last test passed, but the outerStatus of a test failed, we need to correct the status of the test to say 'passed'
// (see 'calculateTestStatus' in ./driver/src/cypress/mocha.ts).
runnable.state = runnable.forceState || runnable.state
if (!runnable.alreadyEmittedMocha) {
// do not double emit this event
@@ -1355,9 +1396,7 @@ export default {
replaceTest(test, test.id)
}
const maybeHandleRetry = (runnable, err) => {
if (!err) return
const maybeHandleRetryOnFailure = (runnable, err) => {
const r = runnable
const isHook = r.type === 'hook'
const isTest = r.type === 'test'
@@ -1365,8 +1404,62 @@ export default {
const hookName = isHook && getHookName(r)
const isBeforeEachHook = isHook && !!hookName.match(/before each/)
const isAfterEachHook = isHook && !!hookName.match(/after each/)
const retryAbleRunnable = isTest || isBeforeEachHook || isAfterEachHook
const willRetry = (test._currentRetry < test._retries) && retryAbleRunnable
let isBeforeEachThatIsRetryable = false
let isAfterEachThatIsRetryable = false
if (isBeforeEachHook || isAfterEachHook) {
if (err) {
// If the beforeEach/afterEach hook failed, mark the test attempt as failed
test.state = 'failed'
}
// Then calculate the test status, accounting for the updated state if the hook errored
// to see if we should continue running the test.
const status = test.calculateTestStatus()
// If we have remaining attempts, inclusive of the beforeEach attempt if it failed, then the hook is retry-able
isBeforeEachThatIsRetryable = isBeforeEachHook && status.shouldAttemptsContinue
if (isAfterEachHook) {
// If we have remaining attempts, inclusive of the afterEach attempt if it failed, then the hook is retry-able
if (status.shouldAttemptsContinue) {
isAfterEachThatIsRetryable = true
} else if (!status.shouldAttemptsContinue && err) {
/**
* OR in the event the test attempt 'passed' and hit the exit condition,
* BUT the afterEach hook errored which MIGHT change the test exit condition (as the test attempt is now 'failed')
*
* In this case, we need to see if we MIGHT have additional retries (maxRetries) available to reapply to satisfy
* the test exit condition.
*
* Ex: This is important for 'detect-flake-but-always-fail' where stopIfAnyPassed=true, where the test itself might pass,
* the exit condition is met, but THEN the 'afterEach' hook itself fails, which MIGHT change the exit conditions of the test
* if there are remaining attempts that can be executed in order to satisfy the experimentalRetries configuration.
*
* To help with exit conditions on test skipping on repeated hook failures, test._retries
* is set to retries made inside our mocha patch (./driver/patches/mocha+7.0.1.dev.patch), assuming a retry is made.
* To show how many attempts are possible, we set '_maxRetries' to the total retries initially configured in order
* to reference here in the case we might need to 'reset'.
*
* When we fall into this scenario, we need to 'reset' the mocha '_retries' in order to continue attempts
* and requeue the test.
*/
// Since this is the afterEach, we can assume the currentRetry has already run
const canMoreAttemptsBeApplied = test._currentRetry === test._retries && test._currentRetry < test._maxRetries
if (canMoreAttemptsBeApplied) {
// The test in fact did NOT fit the exit condition because the 'afterEach' changed the status of the test.
// Reset the retries to apply more attempts to possibly satisfy the test retry conditions.
test._retries = test._maxRetries
isAfterEachThatIsRetryable = true
}
}
}
}
const willRetry = isBeforeEachThatIsRetryable || isAfterEachThatIsRetryable
const isTestConfigOverride = !fired(TEST_BEFORE_RUN_EVENT, test)
const fail = function () {
@@ -1376,10 +1469,21 @@ export default {
return
}
if (isTest) {
// If there is no error on the test attempt, then the test attempt passed!
// set a custom property on the test obj, hasTestAttemptPassed,
// to inform mocha (through patch-package) that we need to re-attempt a passed test attempt
// if experimentalRetries is enabled and there is at least one existing failure.
runnable.hasAttemptPassed = !err
}
if (err) {
if (willRetry) {
test.state = 'failed'
test.final = false
// If the test is being retried/re-attempted, delete the testStatusInfo metadata object if it is present
// that determines outer status as it is no longer needed and contributes to additional properties on the
// test runnable that are NOT needed.
delete test._cypressTestStatusInfo
}
if (isTestConfigOverride) {
@@ -1416,7 +1520,13 @@ export default {
newTest._currentRetry = test._currentRetry + 1
test.parent.testsQueue.unshift(newTest)
// Check to see if the test attempt maybe passed, but hasn't satisfied its retry config yet and requeued itself.
// In this case, we DON'T need to add the new test attempt as it is already queued to rerun.
const testRetryThatMatches = test.parent.testsQueue.find((t) => t.id === newTest.id && t._currentRetry === newTest._currentRetry)
if (!testRetryThatMatches) {
test.parent.testsQueue.unshift(newTest)
}
// this prevents afterEach hooks that exist at a deeper (or same) level than the failing one from running
test._skipHooksWithLevelGreaterThan = runnable.titlePath().length - 1
@@ -1643,7 +1753,7 @@ export default {
delete runnable.err
}
err = maybeHandleRetry(runnable, err)
err = maybeHandleRetryOnFailure(runnable, err)
return runnableAfterRunAsync(runnable, Cypress)
.then(() => {

4
packages/driver/src/util/config.ts
View File

@@ -103,7 +103,9 @@ export const validateConfig = (state: State, config: Record<string, any>, skipCo
validateOverridableAtRunTime(config, isSuiteOverride, (validationResult) => {
let errKey = 'config.cypress_config_api.read_only'
if (validationResult.supportedOverrideLevel === 'suite') {
if (validationResult.supportedOverrideLevel === 'global_only') {
errKey = 'config.invalid_mocha_config_override.global_only'
} else if (validationResult.supportedOverrideLevel === 'suite') {
errKey = 'config.invalid_mocha_config_override.suite_only'
} else if (mochaOverrideLevel) {
errKey = 'config.invalid_mocha_config_override.read_only'

9
packages/driver/types/internal-types.d.ts vendored
View File

@@ -96,4 +96,13 @@ interface CypressRunnable extends Mocha.Runnable {
hookName: string
id: any
err: any
// Added by Cypress to Tests in order to calculate continue conditions for retries
calculateTestStatus?: () => {
strategy: 'detect-flake-and-pass-on-threshold' | 'detect-flake-but-always-fail' | undefined
shouldAttemptsContinue: boolean
attempts: number
outerStatus: 'passed' | failed
}
// Added by Cypress to Tests in order to determine if the experimentalRetries test run passed so we can leverage in the retry logic.
hasAttemptPassed?: boolean
}

10
packages/frontend-shared/src/locales/en-US.json
View File

@@ -595,6 +595,16 @@
"name": "Modify obstructive third party code",
"description": "Applies `modifyObstructiveCode` to third party `.html` and `.js`, removes subresource integrity, and modifies the user agent in Electron."
},
"retries": {
"experimentalStrategy": {
"name": "Retries Strategy",
"description": "Applies a strategy for test retries according to your \"flake tolerance\"; options are `detect-flake-but-always-fail` or `detect-flake-and-pass-on-threshold`."
},
"experimentalOptions": {
"name": "Retries Strategy Options",
"description": "Sets retries strategy-specific options like `maxRetries`, `passesRequired`, and `stopIfAnyPassed`."
}
},
"experimentalSingleTabRunMode": {
"name": "Single tab run mode",
"description": "Runs all component specs in a single tab, trading spec isolation for faster run mode execution."

5
packages/reporter/src/attempts/attempt-model.ts
View File

@@ -23,6 +23,7 @@ export default class Attempt {
@observable isActive: boolean | null = null
@observable routes: Route[] = []
@observable _state?: TestState | null = null
@observable _testOuterStatus?: TestState = undefined
@observable _invocationCount: number = 0
@observable invocationDetails?: FileDetails
@observable hookCount: { [name in HookName]: number } = {
@@ -172,6 +173,10 @@ export default class Attempt {
this._state = props.state
}
if (props._cypressTestStatusInfo?.outerStatus) {
this._testOuterStatus = props._cypressTestStatusInfo.outerStatus
}
if (props.err) {
if (this.err) {
this.err.update(props.err)

4
packages/reporter/src/lib/events.ts
View File

@@ -98,7 +98,9 @@ const events: Events = {
runner.on('test:after:run', action('test:after:run', (runnable: TestProps, isInteractive: boolean) => {
runnablesStore.runnableFinished(runnable, isInteractive)
if (runnable.final && !appState.studioActive) {
statsStore.incrementCount(runnable.state!)
// When displaying the overall test status, we want to reference the test outerStatus
// as the last runnable (test attempt) may have passed, but the outerStatus might mark the test run as a failure.
statsStore.incrementCount(runnable?._cypressTestStatusInfo?.outerStatus || runnable.state!)
}
}))

22
packages/reporter/src/test/test-model.ts
View File

@@ -15,6 +15,10 @@ export type UpdateTestCallback = () => void
export interface TestProps extends RunnableProps {
state: TestState | null
// the final state of the test (the attempt might pass, but the test might be marked as failed)
_cypressTestStatusInfo?: {
outerStatus?: TestState
}
err?: ErrProps
isOpen?: boolean
agents?: Array<AgentProps>
@@ -31,6 +35,10 @@ export interface TestProps extends RunnableProps {
export interface UpdatableTestProps {
id: TestProps['id']
state?: TestProps['state']
// the final state of the test (the attempt might pass, but the test might be marked as failed)
_cypressTestStatusInfo?: {
outerStatus?: TestState
}
err?: TestProps['err']
hookId?: string
failedFromHookId?: string
@@ -89,7 +97,9 @@ export default class Test extends Runnable {
}
@computed get state () {
return this.lastAttempt ? this.lastAttempt.state : 'active'
// Use the outerStatus of the last attempt to determine overall test status, if present,
// as the last attempt may have 'passed', but the outerStatus may be marked as failed.
return this.lastAttempt ? (this.lastAttempt._testOuterStatus || this.lastAttempt.state) : 'active'
}
@computed get err () {
@@ -122,19 +132,21 @@ export default class Test extends Runnable {
}
addLog = (props: LogProps) => {
return this._withAttempt(props.testCurrentRetry || this.currentRetry, (attempt: Attempt) => {
// NOTE: The 'testCurrentRetry' prop may be zero, which means we really care about nullish coalescing the value
// to make sure logs on the first attempt are still accounted for even if the attempt has finished.
return this._withAttempt(props.testCurrentRetry ?? this.currentRetry, (attempt: Attempt) => {
return attempt.addLog(props)
})
}
updateLog (props: LogProps) {
this._withAttempt(props.testCurrentRetry || this.currentRetry, (attempt: Attempt) => {
this._withAttempt(props.testCurrentRetry ?? this.currentRetry, (attempt: Attempt) => {
attempt.updateLog(props)
})
}
removeLog (props: LogProps) {
this._withAttempt(props.testCurrentRetry || this.currentRetry, (attempt: Attempt) => {
this._withAttempt(props.testCurrentRetry ?? this.currentRetry, (attempt: Attempt) => {
attempt.removeLog(props)
})
}
@@ -189,7 +201,7 @@ export default class Test extends Runnable {
@action finish (props: UpdatableTestProps, isInteractive: boolean) {
this._isFinished = !(props.retries && props.currentRetry) || props.currentRetry >= props.retries
this._withAttempt(props.currentRetry || 0, (attempt: Attempt) => {
this._withAttempt(props.currentRetry ?? 0, (attempt: Attempt) => {
attempt.finish(props, isInteractive)
})
}

2
packages/server/lib/project-base.ts
View File

@@ -364,7 +364,7 @@ export class ProjectBase extends EE {
debug('received runnables %o', runnables)
if (reporterInstance) {
reporterInstance.setRunnables(runnables)
reporterInstance.setRunnables(runnables, this.getConfig())
}
if (this._recordTests) {

211
packages/server/lib/reporter.js
View File

@@ -18,6 +18,41 @@ const { overrideRequire } = require('./override_require')
// otherwise mocha will be resolved from project's node_modules and might not work with our code
const customReporterMochaPath = path.dirname(require.resolve('mocha-7.0.1'))
const buildAttemptMessage = (currentRetry, totalRetries) => {
return `(Attempt ${currentRetry} of ${totalRetries})`
}
// Used for experimentalRetries, where we want to display the passed/failed
// status of an attempt in the console
const getIconStatus = (status) => {
let overallStatusSymbol
let overallStatusSymbolColor
let overallStatusColor
switch (status) {
case 'passed':
overallStatusSymbol = mochaSymbols.ok
overallStatusSymbolColor = 'checkmark'
overallStatusColor = 'checkmark'
break
case 'failed':
overallStatusSymbol = mochaSymbols.err
overallStatusSymbolColor = 'bright fail'
overallStatusColor = 'error message'
break
default:
overallStatusSymbol = undefined
overallStatusSymbolColor = undefined
overallStatusColor = 'medium'
}
return {
overallStatusSymbol,
overallStatusSymbolColor,
overallStatusColor,
}
}
overrideRequire((depPath, _load) => {
if ((depPath === 'mocha') || depPath.startsWith('mocha/')) {
return _load(depPath.replace('mocha', customReporterMochaPath))
@@ -191,7 +226,10 @@ const mergeErr = function (runnable, runnables, stats) {
// dont mutate the test, and merge in the runnable title
// in the case its a hook so that we emit the right 'fail'
// event for reporters
test = _.extend({}, test, { title: runnable.title })
// However, we need to propagate the cypressTestStatusInfo to the reporter in order to print the test information correctly
// in the terminal, as well as updating the test state as it may have changed in the client-side runner.
test = _.extend({}, test, { title: runnable.title, _cypressTestStatusInfo: runnable._cypressTestStatusInfo, state: runnable.state })
return [test, test.err]
}
@@ -249,13 +287,14 @@ class Reporter {
this.normalizeTest = this.normalizeTest.bind(this)
}
setRunnables (rootRunnable) {
setRunnables (rootRunnable, cypressConfig) {
if (!rootRunnable) {
rootRunnable = { title: '' }
}
// manage stats ourselves
this.stats = { suites: 0, tests: 0, passes: 0, pending: 0, skipped: 0, failures: 0 }
this.retriesConfig = cypressConfig ? cypressConfig.retries : {}
this.runnables = {}
rootRunnable = this._createRunnable(rootRunnable, 'suite')
const reporter = Reporter.loadReporter(this.reporterName, this.projectRoot)
@@ -281,23 +320,102 @@ class Reporter {
--indents
})
const retriesConfig = this.retriesConfig
// Override the default reporter to always show test timing even for fast tests
// and display slow ones in yellow rather than red
this.runner._events.pass[2] = function (test) {
// can possibly be undefined if the test fails before being run, such as in before/beforeEach hooks
const cypressTestMetaData = test._cypressTestStatusInfo
const durationColor = test.speed === 'slow' ? 'medium' : 'fast'
const fmt =
let fmt
// Print the default if the experiment is not configured
if (!retriesConfig?.experimentalStrategy) {
fmt =
Array(indents).join(' ') +
mochaColor('checkmark', ` ${ mochaSymbols.ok}`) +
mochaColor('pass', ' %s') +
mochaColor(durationColor, ' (%dms)')
// Log: `✓ test title (300ms)` when a test passes
// eslint-disable-next-line no-console
console.log(fmt, test.title, test.duration)
} else {
// If there have been no retries and experimental retries is configured,
// DON'T decorate the last test in the console as an attempt.
if (cypressTestMetaData?.attempts > 1) {
const lastTestStatus = getIconStatus(test.state)
fmt =
Array(indents).join(' ') +
mochaColor(lastTestStatus.overallStatusColor, ` ${ lastTestStatus.overallStatusSymbol}${buildAttemptMessage(cypressTestMetaData.attempts, test.retries + 1)}`)
// or Log: `✓(Attempt 3 of 3) test title` when the overall outerStatus of a test has passed
// eslint-disable-next-line no-console
console.log(fmt, test.title)
}
const finalTestStatus = getIconStatus(cypressTestMetaData.outerStatus || test.state)
const finalMessaging =
Array(indents).join(' ') +
mochaColor(finalTestStatus.overallStatusSymbolColor, ` ${ finalTestStatus.overallStatusSymbol ? finalTestStatus.overallStatusSymbol : ''}`) +
mochaColor(finalTestStatus.overallStatusColor, ' %s') +
mochaColor(durationColor, ' (%dms)')
// Log: ✓`test title` when the overall outerStatus of a test has passed
// OR
// Log: ✖`test title` when the overall outerStatus of a test has failed
// eslint-disable-next-line no-console
console.log(finalMessaging, test.title, test.duration)
}
}
const originalFailPrint = this.runner._events.fail[2]
this.runner._events.fail[2] = function (test) {
// can possibly be undefined if the test fails before being run, such as in before/beforeEach hooks
const cypressTestMetaData = test._cypressTestStatusInfo
// print the default if the experiment is not configured
if (!retriesConfig?.experimentalStrategy) {
return originalFailPrint.call(this, test)
}
const durationColor = test.speed === 'slow' ? 'medium' : 'fast'
// If there have been no retries and experimental retries is configured,
// DON'T decorate the last test in the console as an attempt.
if (cypressTestMetaData?.attempts > 1) {
const lastTestStatus = getIconStatus(test.state)
const fmt =
Array(indents).join(' ') +
mochaColor(lastTestStatus.overallStatusColor, ` ${ lastTestStatus.overallStatusSymbol}${buildAttemptMessage(cypressTestMetaData.attempts, test.retries + 1)}`)
// Log: `✖(Attempt 3 of 3) test title (300ms)` when a test fails and none of the retries have passed
// eslint-disable-next-line no-console
console.log(fmt, test.title)
}
const finalTestStatus = getIconStatus(cypressTestMetaData?.outerStatus || test.state)
const finalMessaging =
Array(indents).join(' ') +
mochaColor('checkmark', ` ${ mochaSymbols.ok}`) +
mochaColor('pass', ' %s') +
mochaColor(finalTestStatus.overallStatusSymbolColor, ` ${ finalTestStatus.overallStatusSymbol ? finalTestStatus.overallStatusSymbol : ''}`) +
mochaColor(finalTestStatus.overallStatusColor, ' %s') +
mochaColor(durationColor, ' (%dms)')
// Log: `✓ test title (300ms)` when a test passes
// Log: ✖`test title` when the overall outerStatus of a test has failed
// eslint-disable-next-line no-console
console.log(fmt, test.title, test.duration)
console.log(finalMessaging, test.title, test.duration)
}
}
this.runner.ignoreLeaks = true
this.runner.ignoreLeaks = true
}
}
_createRunnable (runnableProps, type, parent) {
@@ -331,7 +449,15 @@ class Reporter {
}
emit (event, arg) {
if (event === 'retry' && this.reporterName === 'spec') {
// iI using GA retries, log the retry attempt as the status of the attempt is not used
if (event === 'retry' && this.reporterName === 'spec' && !this.retriesConfig?.experimentalStrategy) {
this._logRetry(arg)
}
// If using experimental retries, log the attempt after the test attempt runs to accurately represent the attempt pass/fail status
// We don't log the last attempt as this is handled by the main pass/fail handler defined above, and would ultimately log AFTER the test complete test status is reported
// from the mocha:pass/mocha:fail event
if (event === 'test:after:run' && this.reporterName === 'spec' && this.retriesConfig?.experimentalStrategy && !arg.final) {
this._logRetry(arg)
}
@@ -361,12 +487,39 @@ class Reporter {
_logRetry (test) {
const runnable = this.runnables[test.id]
const padding = ' '.repeat(runnable.titlePath().length)
const retryMessage = mochaColor('medium', `(Attempt ${test.currentRetry + 1} of ${test.retries + 1})`)
// Log: `(Attempt 1 of 2) test title` when a test retries
// Merge the runnable with the updated test props to gain most recent status from the app runnable (in the case a passed test is retried).
_.extend(runnable, test)
const padding = ' '.repeat(runnable.titlePath().length)
// Don't display a pass/fail symbol if we don't know the status.
let mochaSymbolToDisplay = ''
let mochaColorScheme = 'medium'
// If experimental retries are configured, we need to print the pass/fail status of each attempt as it is no longer implied.
if (this.retriesConfig?.experimentalStrategy) {
switch (runnable.state) {
case 'passed':
mochaSymbolToDisplay = mochaColor('checkmark', mochaSymbols.ok)
mochaColorScheme = 'green'
break
case 'failed':
mochaSymbolToDisplay = mochaColor('bright fail', mochaSymbols.err)
mochaColorScheme = 'error message'
break
default:
}
}
const attemptMessage = mochaColor(mochaColorScheme, buildAttemptMessage(test.currentRetry + 1, test.retries + 1))
// Log: `(Attempt 1 of 2) test title` when a test attempts without experimentalRetries configured.
// OR
// Log: `✓(Attempt 1 of 2) test title` when a test attempt passes with experimentalRetries configured.
// OR
// Log: `✖(Attempt 1 of 2) test title` when a test attempt fails with experimentalRetries configured.
// eslint-disable-next-line no-console
console.log(`${padding}${retryMessage} ${test.title}`)
return console.log(`${padding}${mochaSymbolToDisplay}${attemptMessage} ${test.title}`)
}
normalizeHook (hook = {}) {
@@ -379,13 +532,31 @@ class Reporter {
}
normalizeTest (test = {}) {
let outerTest = _.clone(test)
// In the case tests were skipped or another case where they haven't run,
// test._cypressTestStatusInfo.outerStatus will be undefined. In this case,
// the test state reflects the outer state.
outerTest.state = test._cypressTestStatusInfo?.outerStatus || test.state
// If the outerStatus is failed, but the last test passed, look up the first error that occurred
// and use this as the test overall error. This same logic is applied in the mocha patch to report
// the error correctly to the Cypress browser reporter (see ./driver/patches/mocha+7.0.1.dev.patch)
if (test.state === 'passed' && outerTest.state === 'failed') {
outerTest.err = (test.prevAttempts || []).find((t) => t.state === 'failed')?.err
// Otherwise, if the test failed, set the error on the outer test similar to the state
// as we can assume that if the last test failed, the state could NEVER be passing.
} else if (test.state === 'failed') {
outerTest.err = test.err
}
const normalizedTest = {
testId: orNull(test.id),
title: getTitlePath(test),
state: orNull(test.state),
body: orNull(test.body),
displayError: orNull(test.err && test.err.stack),
attempts: _.map((test.prevAttempts || []).concat([test]), (attempt) => {
testId: orNull(outerTest.id),
title: getTitlePath(outerTest),
state: orNull(outerTest.state),
body: orNull(outerTest.body),
displayError: orNull(outerTest.err && outerTest.err.stack),
attempts: _.map((outerTest.prevAttempts || []).concat([test]), (attempt) => {
const err = attempt.err && {
name: attempt.err.name,
message: attempt.err.message,

2
packages/server/test/unit/config_spec.js
View File

@@ -883,7 +883,7 @@ describe('lib/config', () => {
})
context('retries', () => {
const retriesError = 'a positive number or null or an object with keys "openMode" and "runMode" with values of numbers or nulls'
const retriesError = 'a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy'
// need to keep the const here or it'll get stripped by the build
// eslint-disable-next-line no-unused-vars

2302
system-tests/__snapshots__/experimental_retries.spec.ts.js Normal file
View File

File diff suppressed because it is too large Load Diff

276
system-tests/__snapshots__/protocol_spec.js
View File

@@ -610,6 +610,12 @@ exports['e2e events'] = `
"preAfterTest": [
{
"test": {
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [
{
@@ -681,6 +687,12 @@ exports['e2e events'] = `
},
{
"test": {
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [
{
@@ -763,6 +775,12 @@ exports['e2e events'] = `
},
{
"test": {
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [
{
@@ -845,6 +863,12 @@ exports['e2e events'] = `
},
{
"test": {
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [
{
@@ -927,6 +951,12 @@ exports['e2e events'] = `
},
{
"test": {
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [
{
@@ -1009,6 +1039,12 @@ exports['e2e events'] = `
},
{
"test": {
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [
{
@@ -1091,6 +1127,12 @@ exports['e2e events'] = `
},
{
"test": {
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [
{
@@ -1173,6 +1215,12 @@ exports['e2e events'] = `
},
{
"test": {
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [
{
@@ -1255,6 +1303,12 @@ exports['e2e events'] = `
},
{
"test": {
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [
{
@@ -1338,6 +1392,12 @@ exports['e2e events'] = `
],
"afterTest": [
{
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [
{
@@ -1405,6 +1465,12 @@ exports['e2e events'] = `
"_slow": 10000
},
{
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [
{
@@ -1483,6 +1549,12 @@ exports['e2e events'] = `
"_slow": 10000
},
{
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [
{
@@ -1561,6 +1633,12 @@ exports['e2e events'] = `
"_slow": 10000
},
{
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [
{
@@ -1639,6 +1717,12 @@ exports['e2e events'] = `
"_slow": 10000
},
{
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [
{
@@ -1717,6 +1801,12 @@ exports['e2e events'] = `
"_slow": 10000
},
{
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [
{
@@ -1795,6 +1885,12 @@ exports['e2e events'] = `
"_slow": 10000
},
{
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [
{
@@ -1873,6 +1969,12 @@ exports['e2e events'] = `
"_slow": 10000
},
{
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [
{
@@ -1951,6 +2053,12 @@ exports['e2e events'] = `
"_slow": 10000
},
{
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [
{
@@ -4974,7 +5082,7 @@ exports['component events - experimentalSingleTabRunMode: true'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:333:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-0.js:17:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:462:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-0.js:17:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)"
},
"currentRetry": 0,
"retries": 0,
@@ -5003,7 +5111,7 @@ exports['component events - experimentalSingleTabRunMode: true'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:333:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-0.js:21:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:462:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-0.js:21:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)"
},
"currentRetry": 0,
"retries": 0,
@@ -5032,7 +5140,7 @@ exports['component events - experimentalSingleTabRunMode: true'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:333:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-1.js:17:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:462:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-1.js:17:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)"
},
"currentRetry": 0,
"retries": 0,
@@ -5061,7 +5169,7 @@ exports['component events - experimentalSingleTabRunMode: true'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:333:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-1.js:21:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:462:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-1.js:21:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)"
},
"currentRetry": 0,
"retries": 0,
@@ -5071,6 +5179,12 @@ exports['component events - experimentalSingleTabRunMode: true'] = `
"preAfterTest": [
{
"test": {
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [],
"unverifiedTestConfig": {},
@@ -5102,7 +5216,7 @@ exports['component events - experimentalSingleTabRunMode: true'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:333:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-0.js:17:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:462:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-0.js:17:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)"
},
"final": true,
"currentRetry": 0,
@@ -5115,6 +5229,12 @@ exports['component events - experimentalSingleTabRunMode: true'] = `
},
{
"test": {
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [],
"unverifiedTestConfig": {},
@@ -5146,7 +5266,7 @@ exports['component events - experimentalSingleTabRunMode: true'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:333:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-0.js:21:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:462:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-0.js:21:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)"
},
"final": true,
"currentRetry": 0,
@@ -5159,6 +5279,12 @@ exports['component events - experimentalSingleTabRunMode: true'] = `
},
{
"test": {
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [],
"unverifiedTestConfig": {},
@@ -5190,7 +5316,7 @@ exports['component events - experimentalSingleTabRunMode: true'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:333:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-1.js:17:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:462:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-1.js:17:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)"
},
"final": true,
"currentRetry": 0,
@@ -5203,6 +5329,12 @@ exports['component events - experimentalSingleTabRunMode: true'] = `
},
{
"test": {
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [],
"unverifiedTestConfig": {},
@@ -5234,7 +5366,7 @@ exports['component events - experimentalSingleTabRunMode: true'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:333:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-1.js:21:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:462:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-1.js:21:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)"
},
"final": true,
"currentRetry": 0,
@@ -5248,6 +5380,12 @@ exports['component events - experimentalSingleTabRunMode: true'] = `
],
"afterTest": [
{
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [],
"unverifiedTestConfig": {},
@@ -5280,7 +5418,7 @@ exports['component events - experimentalSingleTabRunMode: true'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:333:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-0.js:17:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:462:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-0.js:17:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)"
},
"final": true,
"currentRetry": 0,
@@ -5288,6 +5426,12 @@ exports['component events - experimentalSingleTabRunMode: true'] = `
"_slow": 250
},
{
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [],
"unverifiedTestConfig": {},
@@ -5320,7 +5464,7 @@ exports['component events - experimentalSingleTabRunMode: true'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:333:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-0.js:21:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:462:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-0.js:21:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)"
},
"final": true,
"currentRetry": 0,
@@ -5328,6 +5472,12 @@ exports['component events - experimentalSingleTabRunMode: true'] = `
"_slow": 250
},
{
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [],
"unverifiedTestConfig": {},
@@ -5360,7 +5510,7 @@ exports['component events - experimentalSingleTabRunMode: true'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:333:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-1.js:17:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:462:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-1.js:17:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)"
},
"final": true,
"currentRetry": 0,
@@ -5368,6 +5518,12 @@ exports['component events - experimentalSingleTabRunMode: true'] = `
"_slow": 250
},
{
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [],
"unverifiedTestConfig": {},
@@ -5400,7 +5556,7 @@ exports['component events - experimentalSingleTabRunMode: true'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:333:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-1.js:21:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:462:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-1.js:21:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)"
},
"final": true,
"currentRetry": 0,
@@ -5437,7 +5593,7 @@ exports['component events - experimentalSingleTabRunMode: true'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addSuite (cypress:///../driver/src/cypress/mocha.ts:359:86)\\n at Suite.create (cypress:///../driver/node_modules/mocha/lib/suite.js:33:10)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:123:27)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at ./src/components/HelloEarth.cy.jsx (http://localhost:2121/__cypress/src/spec-0.js:16:1)\\n at Function.__webpack_require__ (http://localhost:2121/__cypress/src/main.js:114:42)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addSuite (cypress:///../driver/src/cypress/mocha.ts:488:86)\\n at Suite.create (cypress:///../driver/node_modules/mocha/lib/suite.js:33:10)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:123:27)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at ./src/components/HelloEarth.cy.jsx (http://localhost:2121/__cypress/src/spec-0.js:16:1)\\n at Function.__webpack_require__ (http://localhost:2121/__cypress/src/main.js:114:42)"
},
"retries": -1,
"_slow": 250,
@@ -5459,7 +5615,7 @@ exports['component events - experimentalSingleTabRunMode: true'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:333:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-0.js:17:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:462:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-0.js:17:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)"
},
"currentRetry": 0,
"retries": -1,
@@ -5490,7 +5646,7 @@ exports['component events - experimentalSingleTabRunMode: true'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:333:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-0.js:21:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:462:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-0.js:21:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)"
},
"currentRetry": 0,
"retries": -1,
@@ -5540,7 +5696,7 @@ exports['component events - experimentalSingleTabRunMode: true'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addSuite (cypress:///../driver/src/cypress/mocha.ts:359:86)\\n at Suite.create (cypress:///../driver/node_modules/mocha/lib/suite.js:33:10)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:123:27)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at ./src/components/HelloMars.cy.jsx (http://localhost:2121/__cypress/src/spec-1.js:16:1)\\n at Function.__webpack_require__ (http://localhost:2121/__cypress/src/main.js:114:42)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addSuite (cypress:///../driver/src/cypress/mocha.ts:488:86)\\n at Suite.create (cypress:///../driver/node_modules/mocha/lib/suite.js:33:10)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:123:27)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at ./src/components/HelloMars.cy.jsx (http://localhost:2121/__cypress/src/spec-1.js:16:1)\\n at Function.__webpack_require__ (http://localhost:2121/__cypress/src/main.js:114:42)"
},
"retries": -1,
"_slow": 250,
@@ -5562,7 +5718,7 @@ exports['component events - experimentalSingleTabRunMode: true'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:333:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-1.js:17:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:462:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-1.js:17:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)"
},
"currentRetry": 0,
"retries": -1,
@@ -5593,7 +5749,7 @@ exports['component events - experimentalSingleTabRunMode: true'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:333:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-1.js:21:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:462:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-1.js:21:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)"
},
"currentRetry": 0,
"retries": -1,
@@ -6669,7 +6825,7 @@ exports['component events - experimentalSingleTabRunMode: false'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:333:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-0.js:17:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:462:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-0.js:17:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)"
},
"currentRetry": 0,
"retries": 0,
@@ -6698,7 +6854,7 @@ exports['component events - experimentalSingleTabRunMode: false'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:333:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-0.js:21:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:462:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-0.js:21:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)"
},
"currentRetry": 0,
"retries": 0,
@@ -6727,7 +6883,7 @@ exports['component events - experimentalSingleTabRunMode: false'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:333:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-1.js:17:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:462:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-1.js:17:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)"
},
"currentRetry": 0,
"retries": 0,
@@ -6756,7 +6912,7 @@ exports['component events - experimentalSingleTabRunMode: false'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:333:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-1.js:21:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:462:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-1.js:21:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)"
},
"currentRetry": 0,
"retries": 0,
@@ -6766,6 +6922,12 @@ exports['component events - experimentalSingleTabRunMode: false'] = `
"preAfterTest": [
{
"test": {
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [],
"unverifiedTestConfig": {},
@@ -6797,7 +6959,7 @@ exports['component events - experimentalSingleTabRunMode: false'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:333:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-0.js:17:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:462:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-0.js:17:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)"
},
"final": true,
"currentRetry": 0,
@@ -6810,6 +6972,12 @@ exports['component events - experimentalSingleTabRunMode: false'] = `
},
{
"test": {
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [],
"unverifiedTestConfig": {},
@@ -6841,7 +7009,7 @@ exports['component events - experimentalSingleTabRunMode: false'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:333:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-0.js:21:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:462:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-0.js:21:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)"
},
"final": true,
"currentRetry": 0,
@@ -6854,6 +7022,12 @@ exports['component events - experimentalSingleTabRunMode: false'] = `
},
{
"test": {
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [],
"unverifiedTestConfig": {},
@@ -6885,7 +7059,7 @@ exports['component events - experimentalSingleTabRunMode: false'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:333:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-1.js:17:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:462:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-1.js:17:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)"
},
"final": true,
"currentRetry": 0,
@@ -6898,6 +7072,12 @@ exports['component events - experimentalSingleTabRunMode: false'] = `
},
{
"test": {
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [],
"unverifiedTestConfig": {},
@@ -6929,7 +7109,7 @@ exports['component events - experimentalSingleTabRunMode: false'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:333:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-1.js:21:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:462:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-1.js:21:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)"
},
"final": true,
"currentRetry": 0,
@@ -6943,6 +7123,12 @@ exports['component events - experimentalSingleTabRunMode: false'] = `
],
"afterTest": [
{
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [],
"unverifiedTestConfig": {},
@@ -6975,7 +7161,7 @@ exports['component events - experimentalSingleTabRunMode: false'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:333:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-0.js:17:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:462:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-0.js:17:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)"
},
"final": true,
"currentRetry": 0,
@@ -6983,6 +7169,12 @@ exports['component events - experimentalSingleTabRunMode: false'] = `
"_slow": 250
},
{
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [],
"unverifiedTestConfig": {},
@@ -7015,7 +7207,7 @@ exports['component events - experimentalSingleTabRunMode: false'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:333:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-0.js:21:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:462:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-0.js:21:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)"
},
"final": true,
"currentRetry": 0,
@@ -7023,6 +7215,12 @@ exports['component events - experimentalSingleTabRunMode: false'] = `
"_slow": 250
},
{
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [],
"unverifiedTestConfig": {},
@@ -7055,7 +7253,7 @@ exports['component events - experimentalSingleTabRunMode: false'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:333:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-1.js:17:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:462:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-1.js:17:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)"
},
"final": true,
"currentRetry": 0,
@@ -7063,6 +7261,12 @@ exports['component events - experimentalSingleTabRunMode: false'] = `
"_slow": 250
},
{
"_cypressTestStatusInfo": {
"strategy": "detect-flake-and-pass-on-threshold",
"shouldAttemptsContinue": false,
"attempts": 1,
"outerStatus": "passed"
},
"_testConfig": {
"testConfigList": [],
"unverifiedTestConfig": {},
@@ -7095,7 +7299,7 @@ exports['component events - experimentalSingleTabRunMode: false'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:333:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-1.js:21:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:462:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-1.js:21:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)"
},
"final": true,
"currentRetry": 0,
@@ -7132,7 +7336,7 @@ exports['component events - experimentalSingleTabRunMode: false'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addSuite (cypress:///../driver/src/cypress/mocha.ts:359:86)\\n at Suite.create (cypress:///../driver/node_modules/mocha/lib/suite.js:33:10)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:123:27)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at ./src/components/HelloEarth.cy.jsx (http://localhost:2121/__cypress/src/spec-0.js:16:1)\\n at Function.__webpack_require__ (http://localhost:2121/__cypress/src/main.js:114:42)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addSuite (cypress:///../driver/src/cypress/mocha.ts:488:86)\\n at Suite.create (cypress:///../driver/node_modules/mocha/lib/suite.js:33:10)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:123:27)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at ./src/components/HelloEarth.cy.jsx (http://localhost:2121/__cypress/src/spec-0.js:16:1)\\n at Function.__webpack_require__ (http://localhost:2121/__cypress/src/main.js:114:42)"
},
"retries": -1,
"_slow": 250,
@@ -7154,7 +7358,7 @@ exports['component events - experimentalSingleTabRunMode: false'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:333:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-0.js:17:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:462:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-0.js:17:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)"
},
"currentRetry": 0,
"retries": -1,
@@ -7185,7 +7389,7 @@ exports['component events - experimentalSingleTabRunMode: false'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:333:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-0.js:21:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:462:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-0.js:21:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)"
},
"currentRetry": 0,
"retries": -1,
@@ -7235,7 +7439,7 @@ exports['component events - experimentalSingleTabRunMode: false'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addSuite (cypress:///../driver/src/cypress/mocha.ts:359:86)\\n at Suite.create (cypress:///../driver/node_modules/mocha/lib/suite.js:33:10)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:123:27)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at ./src/components/HelloMars.cy.jsx (http://localhost:2121/__cypress/src/spec-1.js:16:1)\\n at Function.__webpack_require__ (http://localhost:2121/__cypress/src/main.js:114:42)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addSuite (cypress:///../driver/src/cypress/mocha.ts:488:86)\\n at Suite.create (cypress:///../driver/node_modules/mocha/lib/suite.js:33:10)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:123:27)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at ./src/components/HelloMars.cy.jsx (http://localhost:2121/__cypress/src/spec-1.js:16:1)\\n at Function.__webpack_require__ (http://localhost:2121/__cypress/src/main.js:114:42)"
},
"retries": -1,
"_slow": 250,
@@ -7257,7 +7461,7 @@ exports['component events - experimentalSingleTabRunMode: false'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:333:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-1.js:17:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:462:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-1.js:17:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)"
},
"currentRetry": 0,
"retries": -1,
@@ -7288,7 +7492,7 @@ exports['component events - experimentalSingleTabRunMode: false'] = `
"line": 94,
"column": 17,
"whitespace": " ",
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:333:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-1.js:21:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:54:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:115:14)"
"stack": "Error\\n at Object.getInvocationDetails (cypress:///../driver/src/cypress/stack_utils.ts:94:17)\\n at Suite.addTest (cypress:///../driver/src/cypress/mocha.ts:462:85)\\n at context.it.context.specify (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:88:13)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)\\n at Suite.<anonymous> (http://localhost:2121/__cypress/src/spec-1.js:21:3)\\n at Object.create (cypress:///../driver/node_modules/mocha/lib/interfaces/common.js:140:19)\\n at context.describe.context.context (cypress:///../driver/node_modules/mocha/lib/interfaces/bdd.js:41:27)\\n at createRunnable (cypress:///../driver/src/cypress/mocha.ts:126:31)\\n at eval (cypress:///../driver/src/cypress/mocha.ts:187:14)"
},
"currentRetry": 0,
"retries": -1,

304
system-tests/__snapshots__/testConfigOverrides_spec.ts.js
View File

@@ -310,7 +310,7 @@ https://on.cypress.io/config
runs:
CypressError: The config passed to your suite-level overrides has the following validation error:
Expected \`retries\` to be a positive number or null or an object with keys "openMode" and "runMode" with values of numbers or nulls.
Expected \`retries\` to be a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy.
Instead the value was: \`"1"\`
@@ -359,7 +359,7 @@ https://on.cypress.io/config
"before all" hook for "test config override throws error":
CypressError: The config passed to your test-level overrides has the following validation error:
Expected \`retries\` to be a positive number or null or an object with keys "openMode" and "runMode" with values of numbers or nulls.
Expected \`retries\` to be a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy.
Instead the value was: \`"1"\`
@@ -373,7 +373,7 @@ Because this error occurred during a \`before all\` hook we are skipping the rem
test config override throws error:
CypressError: The config passed to your test-level overrides has the following validation error:
Expected \`retries\` to be a positive number or null or an object with keys "openMode" and "runMode" with values of numbers or nulls.
Expected \`retries\` to be a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy.
Instead the value was: \`"1"\`
@@ -518,7 +518,7 @@ Instead the value was: \`"null"\`
runs:
CypressError: The config passed to your suite-level overrides has the following validation error:
Expected \`retries\` to be a positive number or null or an object with keys "openMode" and "runMode" with values of numbers or nulls.
Expected \`retries\` to be a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy.
Instead the value was: \`"1"\`
@@ -571,7 +571,7 @@ https://on.cypress.io/config
"before all" hook for "test config override throws error":
CypressError: The config passed to your test-level overrides has the following validation error:
Expected \`retries\` to be a positive number or null or an object with keys "openMode" and "runMode" with values of numbers or nulls.
Expected \`retries\` to be a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy.
Instead the value was: \`"1"\`
@@ -586,7 +586,7 @@ Because this error occurred during a \`before all\` hook we are skipping the rem
test config override throws error:
CypressError: The config passed to your test-level overrides has the following validation error:
Expected \`retries\` to be a positive number or null or an object with keys "openMode" and "runMode" with values of numbers or nulls.
Expected \`retries\` to be a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy.
Instead the value was: \`"1"\`
@@ -666,7 +666,7 @@ exports['testConfigOverrides / correctly fails when invalid config values for it
throws error at the correct line number:
CypressError: The config passed to your suite-level overrides has the following validation error:
Expected \`retries\` to be a positive number or null or an object with keys "openMode" and "runMode" with values of numbers or nulls.
Expected \`retries\` to be a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy.
Instead the value was: \`"1"\`
@@ -972,7 +972,7 @@ https://on.cypress.io/config
runs:
CypressError: The config passed to your suite-level overrides has the following validation error:
Expected \`retries\` to be a positive number or null or an object with keys "openMode" and "runMode" with values of numbers or nulls.
Expected \`retries\` to be a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy.
Instead the value was: \`"1"\`
@@ -1017,7 +1017,7 @@ https://on.cypress.io/config
"before all" hook for "test config override throws error":
CypressError: The config passed to your test-level overrides has the following validation error:
Expected \`retries\` to be a positive number or null or an object with keys "openMode" and "runMode" with values of numbers or nulls.
Expected \`retries\` to be a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy.
Instead the value was: \`"1"\`
@@ -1030,7 +1030,7 @@ Because this error occurred during a \`before all\` hook we are skipping the rem
test config override throws error:
CypressError: The config passed to your test-level overrides has the following validation error:
Expected \`retries\` to be a positive number or null or an object with keys "openMode" and "runMode" with values of numbers or nulls.
Expected \`retries\` to be a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy.
Instead the value was: \`"1"\`
@@ -1174,7 +1174,7 @@ Instead the value was: \`"null"\`
runs:
CypressError: The config passed to your suite-level overrides has the following validation error:
Expected \`retries\` to be a positive number or null or an object with keys "openMode" and "runMode" with values of numbers or nulls.
Expected \`retries\` to be a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy.
Instead the value was: \`"1"\`
@@ -1223,7 +1223,7 @@ https://on.cypress.io/config
"before all" hook for "test config override throws error":
CypressError: The config passed to your test-level overrides has the following validation error:
Expected \`retries\` to be a positive number or null or an object with keys "openMode" and "runMode" with values of numbers or nulls.
Expected \`retries\` to be a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy.
Instead the value was: \`"1"\`
@@ -1237,7 +1237,7 @@ Because this error occurred during a \`before all\` hook we are skipping the rem
test config override throws error:
CypressError: The config passed to your test-level overrides has the following validation error:
Expected \`retries\` to be a positive number or null or an object with keys "openMode" and "runMode" with values of numbers or nulls.
Expected \`retries\` to be a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy.
Instead the value was: \`"1"\`
@@ -1316,7 +1316,7 @@ exports['testConfigOverrides / correctly fails when invalid config values for it
throws error at the correct line number:
CypressError: The config passed to your suite-level overrides has the following validation error:
Expected \`retries\` to be a positive number or null or an object with keys "openMode" and "runMode" with values of numbers or nulls.
Expected \`retries\` to be a positive number or null or an object with keys "openMode" and "runMode" with values of numbers, booleans, or nulls, or experimental configuration with key "experimentalStrategy" with value "detect-flake-but-always-fail" or "detect-flake-and-pass-on-threshold" and key "experimentalOptions" to provide a valid configuration for your selected strategy.
Instead the value was: \`"1"\`
@@ -1354,3 +1354,279 @@ https://on.cypress.io/config
`
exports['testConfigOverrides / experimental retries specific behavior / fails when attempting to set experimental retries as override [chrome,electron]'] = `
====================================================================================================
(Run Starting)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Cypress: 1.2.3 │
│ Browser: FooBrowser 88 │
│ Specs: 1 found (override-with-experimental-retries.cy.js) │
│ Searched: cypress/e2e/override-with-experimental-retries.cy.js │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
────────────────────────────────────────────────────────────────────────────────────────────────────
Running: override-with-experimental-retries.cy.js (1 of 1)
overriding legacy retries with experimental retries
at the describe level
1) sets the config
at the test level
2) sets the config
0 passing
2 failing
1) overriding legacy retries with experimental retries
at the describe level
sets the config:
CypressError: The config passed to your suite-level overrides has the following validation error:
CypressError: The \`retries.experimentalStrategy\` configuration can only be set globally.
https://on.cypress.io/config
Error
[stack trace lines]
2) overriding legacy retries with experimental retries
at the test level
sets the config:
CypressError: The config passed to your test-level overrides has the following validation error:
CypressError: The \`retries.experimentalStrategy\` configuration can only be set globally.
https://on.cypress.io/config
Error
[stack trace lines]
(Results)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Tests: 2 │
│ Passing: 0 │
│ Failing: 2 │
│ Pending: 0 │
│ Skipped: 0 │
│ Screenshots: 0 │
│ Video: false │
│ Duration: X seconds │
│ Spec Ran: override-with-experimental-retries.cy.js │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
====================================================================================================
(Run Finished)
Spec Tests Passing Failing Pending Skipped
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ ✖ override-with-experimental-retries. XX:XX 2 - 2 - - │
│ cy.js │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
✖ 1 of 1 failed (100%) XX:XX 2 - 2 - -
`
exports['testConfigOverrides / experimental retries specific behavior / succeeds when setting legacy retries as an override to experimental retries [chrome,electron]'] = `
====================================================================================================
(Run Starting)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Cypress: 1.2.3 │
│ Browser: FooBrowser 88 │
│ Specs: 1 found (override-with-legacy-retries.cy.js) │
│ Searched: cypress/e2e/override-with-legacy-retries.cy.js │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
────────────────────────────────────────────────────────────────────────────────────────────────────
Running: override-with-legacy-retries.cy.js (1 of 1)
overriding experimental retries with legacy retries
✓ sets the config
1 passing
(Results)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Tests: 1 │
│ Passing: 1 │
│ Failing: 0 │
│ Pending: 0 │
│ Skipped: 0 │
│ Screenshots: 0 │
│ Video: false │
│ Duration: X seconds │
│ Spec Ran: override-with-legacy-retries.cy.js │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
====================================================================================================
(Run Finished)
Spec Tests Passing Failing Pending Skipped
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ ✔ override-with-legacy-retries.cy.js XX:XX 1 1 - - - │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
✔ All specs passed! XX:XX 1 1 - - -
`
exports['testConfigOverrides / experimental retries specific behavior / fails when attempting to set experimental retries as override [firefox]'] = `
====================================================================================================
(Run Starting)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Cypress: 1.2.3 │
│ Browser: FooBrowser 88 │
│ Specs: 1 found (override-with-experimental-retries.cy.js) │
│ Searched: cypress/e2e/override-with-experimental-retries.cy.js │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
────────────────────────────────────────────────────────────────────────────────────────────────────
Running: override-with-experimental-retries.cy.js (1 of 1)
overriding legacy retries with experimental retries
at the describe level
1) sets the config
at the test level
2) sets the config
0 passing
2 failing
1) overriding legacy retries with experimental retries
at the describe level
sets the config:
CypressError: The config passed to your suite-level overrides has the following validation error:
CypressError: The \`retries.experimentalStrategy\` configuration can only be set globally.
https://on.cypress.io/config
[stack trace lines]
2) overriding legacy retries with experimental retries
at the test level
sets the config:
CypressError: The config passed to your test-level overrides has the following validation error:
CypressError: The \`retries.experimentalStrategy\` configuration can only be set globally.
https://on.cypress.io/config
[stack trace lines]
(Results)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Tests: 2 │
│ Passing: 0 │
│ Failing: 2 │
│ Pending: 0 │
│ Skipped: 0 │
│ Screenshots: 0 │
│ Video: false │
│ Duration: X seconds │
│ Spec Ran: override-with-experimental-retries.cy.js │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
====================================================================================================
(Run Finished)
Spec Tests Passing Failing Pending Skipped
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ ✖ override-with-experimental-retries. XX:XX 2 - 2 - - │
│ cy.js │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
✖ 1 of 1 failed (100%) XX:XX 2 - 2 - -
`
exports['testConfigOverrides / experimental retries specific behavior / succeeds when setting legacy retries as an override to experimental retries [firefox]'] = `
====================================================================================================
(Run Starting)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Cypress: 1.2.3 │
│ Browser: FooBrowser 88 │
│ Specs: 1 found (override-with-legacy-retries.cy.js) │
│ Searched: cypress/e2e/override-with-legacy-retries.cy.js │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
────────────────────────────────────────────────────────────────────────────────────────────────────
Running: override-with-legacy-retries.cy.js (1 of 1)
overriding experimental retries with legacy retries
✓ sets the config
1 passing
(Results)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Tests: 1 │
│ Passing: 1 │
│ Failing: 0 │
│ Pending: 0 │
│ Skipped: 0 │
│ Screenshots: 0 │
│ Video: false │
│ Duration: X seconds │
│ Spec Ran: override-with-legacy-retries.cy.js │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
====================================================================================================
(Run Finished)
Spec Tests Passing Failing Pending Skipped
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ ✔ override-with-legacy-retries.cy.js XX:XX 1 1 - - - │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
✔ All specs passed! XX:XX 1 1 - - -
`

37
system-tests/project-fixtures/runner-specs/cypress/e2e/runner/experimental-retries/can-retry-from-afterEach.retries.mochaEvents.cy.js Normal file
View File

@@ -0,0 +1,37 @@
let i = 0
afterEach(() => {
if (i === 0) {
i++
throw new Error('')
}
})
describe('suite 1', () => {
before(() => {})
beforeEach(() => {})
beforeEach(() => {})
afterEach(() => {})
after(() => {})
it('test 1', () => {})
it('test 2', () => {})
it('test 3', () => {})
})
describe('suite 2', () => {
let j = 0
afterEach(() => {
if (j === 0 || j === 1) {
j++
throw new Error('')
}
})
it('test 1', () => {})
})
describe('suite 3', () => {
it('test 1', () => {})
})

20
system-tests/project-fixtures/runner-specs/cypress/e2e/runner/experimental-retries/can-retry-from-beforeEach.retries.mochaEvents.cy.js Normal file
View File

@@ -0,0 +1,20 @@
describe('suite 1', () => {
before(() => {})
beforeEach(() => {})
let i = 0
beforeEach(() => {
if (i === 0) {
i++
throw new Error('')
}
})
beforeEach(() => {})
afterEach(() => {})
after(() => {})
it('test 1', () => {})
})

17
system-tests/project-fixtures/runner-specs/cypress/e2e/runner/experimental-retries/cant-retry-from-before.retries.mochaEvents.cy.js Normal file
View File

@@ -0,0 +1,17 @@
describe('suite 1', () => {
let i = 0
before(() => {
if (i === 0) {
i++
throw new Error('')
}
})
beforeEach(() => {})
beforeEach(() => {})
afterEach(() => {})
afterEach(() => {})
after(() => {})
it('test 1', () => {})
})

4
system-tests/project-fixtures/runner-specs/cypress/e2e/runner/experimental-retries/does-not-serialize-dom-error.cy.js Normal file
View File

@@ -0,0 +1,4 @@
it('visits', { defaultCommandTimeout: 200 }, () => {
cy.visit('cypress/fixtures/dom.html')
cy.get('#button').should('not.be.visible')
})

10
system-tests/project-fixtures/runner-specs/cypress/e2e/runner/experimental-retries/simple-fail.retries.mochaEvents.cy.js Normal file
View File

@@ -0,0 +1,10 @@
describe('suite 1', () => {
let i = 0
it('test 1', () => {
if (i === 0) {
i++
throw new Error('test 1')
}
})
})

15
system-tests/project-fixtures/runner-specs/cypress/e2e/runner/experimental-retries/test-retry-with-hooks.retries.mochaEvents.cy.js Normal file
View File

@@ -0,0 +1,15 @@
describe('suite 1', () => {
before(() => {})
beforeEach(() => {})
after(() => {})
afterEach(() => {})
let i = 0
it('test 1', () => {
if (i === 0) {
i++
throw new Error('test 1')
}
})
})

20
system-tests/project-fixtures/runner-specs/cypress/e2e/runner/experimental-retries/test-retry-with-only.retries.mochaEvents.cy.js Normal file
View File

@@ -0,0 +1,20 @@
describe('suite 1', () => {
before(() => {})
beforeEach(() => {})
after(() => {})
afterEach(() => {})
it('test 1', () => {})
let i = 0
// eslint-disable-next-line mocha/no-exclusive-tests
it.only('test 2', () => {
if (i === 0) {
i++
throw new Error('test 2')
}
})
it('test 3', () => {})
})

19
system-tests/project-fixtures/runner-specs/cypress/e2e/runner/experimental-retries/three-tests-with-retry.retries.mochaEvents.cy.js Normal file
View File

@@ -0,0 +1,19 @@
describe('suite 1', () => {
before(() => {})
beforeEach(() => {})
afterEach(() => {})
after(() => {})
it('test 1', () => {})
let i = 0
it('test 2', () => {
if (i <= 1) {
i++
throw new Error('')
}
})
it('test 3', () => {})
})

1
system-tests/project-fixtures/runner-specs/cypress/e2e/runner/fail-with-afterEach.mochaEvents.cy.js
View File

@@ -4,4 +4,5 @@ describe('suite 1', () => {
})
it('test 1', () => {})
it('test 2', () => {})
})

1
system-tests/project-fixtures/runner-specs/cypress/e2e/runner/fail-with-beforeEach.mochaEvents.cy.js
View File

@@ -4,4 +4,5 @@ describe('suite 1', () => {
})
it('test 1', () => {})
it('test 2', () => {})
})

15
system-tests/projects/detect-flake-and-pass-on-threshold/cypress.config.js Normal file
View File

@@ -0,0 +1,15 @@
module.exports = {
numTestsKeptInMemory: 0,
e2e: {
supportFile: false,
retries: {
openMode: true,
runMode: true,
experimentalStrategy: 'detect-flake-and-pass-on-threshold',
experimentalOptions: {
maxRetries: 9,
passesRequired: 5,
},
},
},
}

3
system-tests/projects/detect-flake-and-pass-on-threshold/package.json Normal file
View File

@@ -0,0 +1,3 @@
{
"projectFixtureDirectory": "runner-specs"
}

15
system-tests/projects/detect-flake-but-always-fail-stop-any-passed/cypress.config.js Normal file
View File

@@ -0,0 +1,15 @@
module.exports = {
numTestsKeptInMemory: 0,
e2e: {
supportFile: false,
retries: {
openMode: true,
runMode: true,
experimentalStrategy: 'detect-flake-but-always-fail',
experimentalOptions: {
maxRetries: 9,
stopIfAnyPassed: true,
},
},
},
}

3
system-tests/projects/detect-flake-but-always-fail-stop-any-passed/package.json Normal file
View File

@@ -0,0 +1,3 @@
{
"projectFixtureDirectory": "runner-specs"
}

15
system-tests/projects/detect-flake-but-always-fail/cypress.config.js Normal file
View File

@@ -0,0 +1,15 @@
module.exports = {
numTestsKeptInMemory: 0,
e2e: {
supportFile: false,
retries: {
openMode: true,
runMode: true,
experimentalStrategy: 'detect-flake-but-always-fail',
experimentalOptions: {
maxRetries: 9,
stopIfAnyPassed: false,
},
},
},
}

3
system-tests/projects/detect-flake-but-always-fail/package.json Normal file
View File

@@ -0,0 +1,3 @@
{
"projectFixtureDirectory": "runner-specs"
}

23
system-tests/projects/experimental-retries/cypress-experimental-retries.config.js Normal file
View File

@@ -0,0 +1,23 @@
module.exports = {
e2e: {
supportFile: false,
setupNodeEvents (on, config) {
// in the case the tests needed to be debugged:
// on('before:browser:launch', (browser, launchOptions) => {
// launchOptions.args.push('--auto-open-devtools-for-tabs')
// return launchOptions
// })
},
},
retries: {
experimentalStrategy: 'detect-flake-and-pass-on-threshold',
runMode: true,
openMode: true,
experimentalOptions: {
maxRetries: 3,
passesRequired: 1,
},
},
}

18
system-tests/projects/experimental-retries/cypress-legacy-retries.config.js Normal file
View File

@@ -0,0 +1,18 @@
module.exports = {
e2e: {
supportFile: false,
setupNodeEvents (on, config) {
// in the case the tests needed to be debugged:
// on('before:browser:launch', (browser, launchOptions) => {
// launchOptions.args.push('--auto-open-devtools-for-tabs')
// return launchOptions
// })
},
},
retries: {
runMode: 0,
openMode: 0,
},
}

14
system-tests/projects/experimental-retries/cypress.config.js Normal file
View File

@@ -0,0 +1,14 @@
module.exports = {
e2e: {
supportFile: false,
setupNodeEvents (on, config) {
// in the case the tests needed to be debugged:
// on('before:browser:launch', (browser, launchOptions) => {
// launchOptions.args.push('--auto-open-devtools-for-tabs')
// return launchOptions
// })
},
},
}

5
system-tests/projects/experimental-retries/cypress/e2e/always-fails.cy.js Normal file
View File

@@ -0,0 +1,5 @@
describe('always fails', () => {
it('always fails', function () {
expect(true).to.be.false
})
})

5
system-tests/projects/experimental-retries/cypress/e2e/always-passes.cy.js Normal file
View File

@@ -0,0 +1,5 @@
describe('always passes', () => {
it('always passes', function () {
expect(true).to.be.true
})
})

15
system-tests/projects/experimental-retries/cypress/e2e/deterministic-flaky.cy.js Normal file
View File

@@ -0,0 +1,15 @@
describe('deterministic flaky test', () => {
it('deterministically runs pass/fail on this test', function () {
// this means the test WILL behave as
// first attempt (FAIL)
// second attempt (PASS)
// third attempt (FAIL)
// fourth attempt (PASS)
// fifth attempt (FAIL)...
if (this.test.currentRetry() % 2) {
expect(true).to.be.true
} else {
expect(true).to.be.false
}
})
})

51
system-tests/projects/experimental-retries/cypress/e2e/override-with-experimental-retries.cy.js Normal file
View File

@@ -0,0 +1,51 @@
describe('overriding legacy retries with experimental retries', () => {
const experimentalStrategy = 'detect-flake-and-pass-on-threshold'
const openMode = false
const runMode = true
const maxRetries = 3
const passesRequired = 1
describe('at the describe level', {
retries: {
experimentalStrategy,
openMode,
runMode,
experimentalOptions: {
maxRetries,
passesRequired,
},
},
}, () => {
it('sets the config', () => {
const retries = Cypress.config('retries')
expect(retries.experimentalStrategy).to.eq(experimentalStrategy)
expect(retries.experimentalOptions?.maxRetries).to.eq(maxRetries)
expect(retries.experimentalOptions?.passesRequired).to.eq(passesRequired)
expect(retries.runMode).to.eq(runMode)
expect(retries.openMode).to.eq(openMode)
})
})
describe('at the test level', () => {
it('sets the config', {
retries: {
experimentalStrategy,
openMode,
runMode,
experimentalOptions: {
maxRetries,
passesRequired,
},
},
}, () => {
const retries = Cypress.config('retries')
expect(retries.experimentalStrategy).to.eq(experimentalStrategy)
expect(retries.experimentalOptions?.maxRetries).to.eq(maxRetries)
expect(retries.experimentalOptions?.passesRequired).to.eq(passesRequired)
expect(retries.runMode).to.eq(runMode)
expect(retries.openMode).to.eq(openMode)
})
})
})

16
system-tests/projects/experimental-retries/cypress/e2e/override-with-legacy-retries.cy.js Normal file
View File

@@ -0,0 +1,16 @@
describe('overriding experimental retries with legacy retries', () => {
const openMode = 1
const runMode = 3
it('sets the config', {
retries: {
openMode,
runMode,
},
}, () => {
const retries = Cypress.config('retries')
expect(retries.runMode).to.eq(runMode)
expect(retries.openMode).to.eq(openMode)
})
})

309
system-tests/test/experimental_retries.spec.ts Normal file
View File

@@ -0,0 +1,309 @@
import systemTests from '../lib/system-tests'
describe('e2e retries.experimentalStrategy', () => {
systemTests.setup()
describe('"detect-flake-and-pass-on-threshold"', () => {
describe('passes', () => {
systemTests.it('only runs the first attempt of the test if the test passes', {
project: 'experimental-retries',
browser: '!webkit',
spec: 'always-passes.cy.js',
snapshot: true,
expectedExitCode: 0,
config: {
retries: {
openMode: false,
runMode: true,
experimentalStrategy: 'detect-flake-and-pass-on-threshold',
experimentalOptions: {
maxRetries: 10,
passesRequired: 3,
},
},
screenshotOnRunFailure: false,
},
})
systemTests.it('retries up to the "passesRequired" limit if the config can be satisfied', {
project: 'experimental-retries',
browser: '!webkit',
spec: 'deterministic-flaky.cy.js',
snapshot: true,
expectedExitCode: 0,
config: {
retries: {
openMode: false,
runMode: true,
experimentalStrategy: 'detect-flake-and-pass-on-threshold',
experimentalOptions: {
maxRetries: 9,
passesRequired: 3,
},
},
screenshotOnRunFailure: false,
},
})
systemTests.it('retries up to the "passesRequired" limit if the config can be satisfied (max attempts)', {
project: 'experimental-retries',
browser: '!webkit',
spec: 'deterministic-flaky.cy.js',
snapshot: true,
expectedExitCode: 0,
config: {
retries: {
openMode: false,
runMode: true,
experimentalStrategy: 'detect-flake-and-pass-on-threshold',
experimentalOptions: {
maxRetries: 9,
passesRequired: 5,
},
},
screenshotOnRunFailure: false,
},
})
})
describe('fails', () => {
systemTests.it('short-circuits if the needed "passesRequired" cannot be satisfied by the remaining attempts available', {
project: 'experimental-retries',
browser: '!webkit',
spec: 'deterministic-flaky.cy.js',
snapshot: true,
expectedExitCode: 1,
config: {
retries: {
openMode: false,
runMode: true,
experimentalStrategy: 'detect-flake-and-pass-on-threshold',
experimentalOptions: {
maxRetries: 5,
passesRequired: 5,
},
},
screenshotOnRunFailure: false,
},
})
systemTests.it('retries up to the "passesRequired" limit if the config can be satisfied (max attempts possible)', {
project: 'experimental-retries',
browser: '!webkit',
spec: 'deterministic-flaky.cy.js',
snapshot: true,
expectedExitCode: 1,
config: {
retries: {
openMode: false,
runMode: true,
experimentalStrategy: 'detect-flake-and-pass-on-threshold',
experimentalOptions: {
maxRetries: 6,
passesRequired: 4,
},
},
screenshotOnRunFailure: false,
},
})
})
/**
* exercised additionally in cy-in-cy tests to verify correct mocha snapshots and cypress reporter output:
* packages/app/cypress/e2e/runner/retries.experimentalRetries.mochaEvents.cy.ts
* packages/app/cypress/e2e/runner/runner.experimentalRetries.mochaEvents.cy.ts
*/
systemTests.it('exercises experimental-retries suite to verify console reporter and final status code are correct.', {
project: 'detect-flake-and-pass-on-threshold',
browser: '!webkit',
spec: 'runner/experimental-retries/*',
snapshot: true,
expectedExitCode: 2,
config: {
retries: {
openMode: false,
runMode: true,
experimentalStrategy: 'detect-flake-and-pass-on-threshold',
experimentalOptions: {
maxRetries: 9,
passesRequired: 5,
},
},
screenshotOnRunFailure: false,
},
})
systemTests.it('exercises experimental-retries hook failures to verify console reporter and final status code are correct.', {
project: 'detect-flake-and-pass-on-threshold',
browser: '!webkit',
spec: 'runner/fail-with-afterEach.mochaEvents.cy.js,runner/fail-with-beforeEach.mochaEvents.cy.js,runner/fail-with-after.mochaEvents.cy.js,runner/fail-with-before.mochaEvents.cy.js',
snapshot: true,
expectedExitCode: 4,
config: {
retries: {
openMode: false,
runMode: true,
experimentalStrategy: 'detect-flake-and-pass-on-threshold',
experimentalOptions: {
maxRetries: 9,
passesRequired: 5,
},
},
screenshotOnRunFailure: false,
},
})
})
describe('"detect-flake-but-always-fail"', () => {
describe('passes', () => {
systemTests.it('only runs the first attempt of the test if the test passes', {
project: 'experimental-retries',
browser: '!webkit',
spec: 'always-passes.cy.js',
snapshot: true,
expectedExitCode: 0,
config: {
retries: {
openMode: false,
runMode: true,
experimentalStrategy: 'detect-flake-but-always-fail',
experimentalOptions: {
maxRetries: 9,
stopIfAnyPassed: false,
},
},
screenshotOnRunFailure: false,
},
})
})
describe('fails', () => {
systemTests.it('runs all attempts of the test if the first attempt fails and "stopIfAnyPassed=false"', {
project: 'experimental-retries',
browser: '!webkit',
spec: 'deterministic-flaky.cy.js',
snapshot: true,
expectedExitCode: 1,
config: {
retries: {
openMode: false,
runMode: true,
experimentalStrategy: 'detect-flake-but-always-fail',
experimentalOptions: {
maxRetries: 9,
stopIfAnyPassed: false,
},
},
screenshotOnRunFailure: false,
},
})
systemTests.it('runs attempts of the test if the first attempt fails until the test passes if "stopIfAnyPassed=true"', {
project: 'experimental-retries',
browser: '!webkit',
spec: 'deterministic-flaky.cy.js',
snapshot: true,
expectedExitCode: 1,
config: {
retries: {
openMode: false,
runMode: true,
experimentalStrategy: 'detect-flake-but-always-fail',
experimentalOptions: {
maxRetries: 9,
stopIfAnyPassed: true,
},
},
screenshotOnRunFailure: false,
},
})
})
/**
* exercised additionally in cy-in-cy tests to verify correct mocha snapshots and cypress reporter output:
* packages/app/cypress/e2e/runner/retries.experimentalRetries.mochaEvents.cy.ts
* packages/app/cypress/e2e/runner/runner.experimentalRetries.mochaEvents.cy.ts
*/
systemTests.it('exercises experimental-retries suite to verify console reporter and final status code are correct.', {
project: 'detect-flake-but-always-fail',
browser: '!webkit',
spec: 'runner/experimental-retries/*',
snapshot: true,
// FIXME: this should be 8
expectedExitCode: 9,
config: {
retries: {
openMode: false,
runMode: true,
experimentalStrategy: 'detect-flake-but-always-fail',
experimentalOptions: {
maxRetries: 9,
stopIfAnyPassed: false,
},
},
screenshotOnRunFailure: false,
},
})
systemTests.it('exercises experimental-retries hook failures to verify console reporter and final status code are correct.', {
project: 'detect-flake-but-always-fail',
browser: '!webkit',
spec: 'runner/fail-with-afterEach.mochaEvents.cy.js,runner/fail-with-beforeEach.mochaEvents.cy.js,runner/fail-with-after.mochaEvents.cy.js,runner/fail-with-before.mochaEvents.cy.js',
snapshot: true,
expectedExitCode: 4,
config: {
retries: {
openMode: false,
runMode: true,
experimentalStrategy: 'detect-flake-but-always-fail',
experimentalOptions: {
maxRetries: 9,
stopIfAnyPassed: false,
},
},
screenshotOnRunFailure: false,
},
})
systemTests.it('exercises experimental-retries suite to verify console reporter and final status code are correct (stopIfAnyPassed=true).', {
project: 'detect-flake-but-always-fail-stop-any-passed',
browser: '!webkit',
spec: 'runner/experimental-retries/*',
snapshot: true,
// FIXME: this should be 8
expectedExitCode: 9,
config: {
retries: {
openMode: false,
runMode: true,
experimentalStrategy: 'detect-flake-but-always-fail',
experimentalOptions: {
maxRetries: 9,
stopIfAnyPassed: true,
},
},
screenshotOnRunFailure: false,
},
})
systemTests.it('exercises experimental-retries hook failures to verify console reporter and final status code are correct (stopIfAnyPassed=true).', {
project: 'detect-flake-but-always-fail',
browser: '!webkit',
spec: 'runner/fail-with-afterEach.mochaEvents.cy.js,runner/fail-with-beforeEach.mochaEvents.cy.js,runner/fail-with-after.mochaEvents.cy.js,runner/fail-with-before.mochaEvents.cy.js',
snapshot: true,
expectedExitCode: 4,
config: {
retries: {
openMode: false,
runMode: true,
experimentalStrategy: 'detect-flake-but-always-fail',
experimentalOptions: {
maxRetries: 9,
stopIfAnyPassed: true,
},
},
screenshotOnRunFailure: false,
},
})
})
})

20
system-tests/test/testConfigOverrides_spec.ts
View File

@@ -82,5 +82,25 @@ describe('testConfigOverrides', () => {
browser: browserList,
expectedExitCode: 1,
})
describe('experimental retries specific behavior', () => {
systemTests.it(`fails when attempting to set experimental retries as override [${browserList}]`, {
spec: 'override-with-experimental-retries.cy.js',
project: 'experimental-retries',
configFile: 'cypress-legacy-retries.config.js',
expectedExitCode: 2,
browser: browserList,
snapshot: true,
})
systemTests.it(`succeeds when setting legacy retries as an override to experimental retries [${browserList}]`, {
spec: 'override-with-legacy-retries.cy.js',
project: 'experimental-retries',
configFile: 'cypress-experimental-retries.config.js',
expectedExitCode: 0,
browser: browserList,
snapshot: true,
})
})
})
})