* chore: set up feature/test-burn-in feature branch
* feat: add burnIn Configuration option (currently a no-op) (#27377)
* feat: add the burnIn Configuration to the config package. Option
currently is a no-op
* chore: make burn in experimental
* chore: set experimentalBurnIn to false by default
* feat: add new experimental retries configuration (#27412)
* feat: implement the experimental retries configuration options to pair
with test burn in
* [run ci]
* fix cache invalidation [run ci]
* fix snapshot added in v13 for module api to include test burn in experimentalflag
* chore: fix merge conflict
* chore: add burnInTestAction capability (#27768)
* add burnInTestAction capability
* feat: add burn in capability for cloud
* chore: fix snapshot for record_spec
* feat: implement experimental retries (#27826)
* chore: format the retries/runner snapshot files to make diff easier
* feat: implement experimentalRetries strategies 'detect-flake-and-pass-on-threshold' and 'detect-flake-but-always-fail'. This should not be a breaking change, though it does modify mocha and the test object even when the experiment is not configured. This is to exercise the system and make sure things still work as expected even when we go GA. Test updates will follow in following commits.
* chore: update snapshots from system tests and cy-in-cy tests that now have the cypress test metadata property _cypressTestStatusInfo. tests have been added in the fail-with-[before|after]each specs to visually see the suite being skipped when developing.
* chore: add cy-in-cy tests to verify reporter behavior for pass/fail tests, as well as new mocha snapshots to verify attempts. New tests were needed for this as the 'retries' option in testConfigOverrides currently is and will be invalid for experiment and will function as an override. tests run in the cy-in-cy tests are using globally configured experimentalRetries for the given tested project, which showcases the different behavior between attempts/retries and pass/fail status.
* chore: add unit test like driver test to verify the test object in mocha is decorated/handled properly in calculateTestStatus
* chore: add sanity system tests to verify console reporter output for experimental retries logic. Currently there is a bug in the reporter where the logged status doesnt wait for the aftereach to complete, which impacts the total exitCode and printed status.
* fix: aftereach console output. make sure to fail the test in the appropriate spot in runner.ts and not prematurely, which in turn updates the snapshots for cy-in-cy as the fail event comes later."
* chore: address comments from code review
* fix: make sure hook failures print outer status + attempts when the error is the hook itself.
* chore: improve types within calculateTestStatus inside mocha.ts
* Revert "feat: add burnIn Configuration option (currently a no-op) (#27377)"
This reverts commit c428443079.
* Revert "chore: add burnInTestAction capability (#27768)"
This reverts commit ae3df1a505.
* chore: run snapshot and binary jobs against experimental retries feature branch
* chore: add changelog entry (wip)
* Revert "fix snapshot added in v13 for module api to include test burn in experimentalflag"
This reverts commit bb5046c91e.
* Fix system tests
* Clear CircleCI cache
* Normalize retries config for test execution
* Fixed some unit tests
* update snapshots for newer test metadata
* Fix cy-in-cy snapshots
* update snapshots
* bump cache version
* chore: ensure legacy retry overrides work; reject exp. retries overrides (#28045)
* update changelog
* flip if statement in experimental retries option validation
* refactor invalid experimental retry override for more useful error msg
* revert testConfigOverrides snapshot
* update snapshots for test override sys test
* Update packages/config/src/validation.ts
Co-authored-by: Chris Breiding <chrisbreiding@users.noreply.github.com>
* succinct changelog entry; links to docs for details
* testConfigOverride system test snapshots
* Update .github/workflows/update_v8_snapshot_cache.yml
Co-authored-by: Ryan Manuel <ryanm@cypress.io>
* Update cli/CHANGELOG.md
Co-authored-by: Ryan Manuel <ryanm@cypress.io>
* Update packages/driver/src/cypress.ts
Co-authored-by: Ryan Manuel <ryanm@cypress.io>
* updating cache-version
* improve typescript usage when appending experimental retry options to experiments in Experimenets.vue
* Revert "improve typescript usage when appending experimental retry options to experiments in Experimenets.vue"
This reverts commit b459aba882.
* refactor test config override validation for experimental retry subkeys
* account for error throw differences in browsers in system tests
* bump circle cache
* bump circle cache again
---------
Co-authored-by: astone123 <adams@cypress.io>
Co-authored-by: mabela416 <mabel@cypress.io>
Co-authored-by: Muaz Othman <muaz@cypress.io>
Co-authored-by: Muaz Othman <muazweb@gmail.com>
Co-authored-by: Cacie Prins <cacie@cypress.io>
Co-authored-by: Cacie Prins <cacieprins@users.noreply.github.com>
Co-authored-by: Chris Breiding <chrisbreiding@users.noreply.github.com>
Co-authored-by: Ryan Manuel <ryanm@cypress.io>
Co-authored-by: Matthew Schile <mschile@cypress.io>
* Declare function so we can import
* Fix ignore pattern to match api
* Fix lint
* Only force array if not already an array
Turn grep filtering on for own test
* Add a command to test grep tag @smoke
* adding tests
---------
Co-authored-by: Cacie Prins <cacieprins@users.noreply.github.com>
Co-authored-by: Jordan <jordan@jpdesigning.com>
Co-authored-by: Adam Stone-Lord <adams@cypress.io>
Co-authored-by: Matthew Schile <mschile@cypress.io>
* report correct number of failures if browser crashes in the middle of a test suite
* report failure correctly when browser crashes during test
* refactor crash handling
* correct exit option check, clean up debug
* exit on success if exit option !== false
* use default stats when reporter stats are unavailable
* fix error messaging
* move reporter types to an intermediate .ts file, to be combined with reporter.js when migrated
* debug tab close test in ci
* move debug env from pkg to ci yml
* set debug env in spec
* fix pckg
* adds some logging to cri-client
* remove event emit logging from project-base
* revert snapshot for tab close system test
* fixes console output for no exit on success
* changelog
* changelog wsp
* cleanup
* clean up tests
* refactor to more straightforward control flow
* rm export for unused type
* correct tab close snapshot for ci
* new system test for mid-test config crash
* update snapshots
* fix: yield waited requests in the order that they occured
* fix firefox
* add changelog entry
* add test case for sorting
* add multi-alias test
* preserve alias order when sorting via requestId
* removed value parse code
* messy but more robust
* add comment, rename variable
* reduce alias count to account for webkit
* collate new tests under existing describe
---------
Co-authored-by: young-robot <adam.stonelord+fake@gmail.com>
Co-authored-by: David Kasper <dave.m.kasper@gmail.com>
