* chore: set up feature/test-burn-in feature branch
* feat: add burnIn Configuration option (currently a no-op) (#27377)
* feat: add the burnIn Configuration to the config package. Option
currently is a no-op
* chore: make burn in experimental
* chore: set experimentalBurnIn to false by default
* feat: add new experimental retries configuration (#27412)
* feat: implement the experimental retries configuration options to pair
with test burn in
* [run ci]
* fix cache invalidation [run ci]
* fix snapshot added in v13 for module api to include test burn in experimentalflag
* chore: fix merge conflict
* chore: add burnInTestAction capability (#27768)
* add burnInTestAction capability
* feat: add burn in capability for cloud
* chore: fix snapshot for record_spec
* feat: implement experimental retries (#27826)
* chore: format the retries/runner snapshot files to make diff easier
* feat: implement experimentalRetries strategies 'detect-flake-and-pass-on-threshold' and 'detect-flake-but-always-fail'. This should not be a breaking change, though it does modify mocha and the test object even when the experiment is not configured. This is to exercise the system and make sure things still work as expected even when we go GA. Test updates will follow in following commits.
* chore: update snapshots from system tests and cy-in-cy tests that now have the cypress test metadata property _cypressTestStatusInfo. tests have been added in the fail-with-[before|after]each specs to visually see the suite being skipped when developing.
* chore: add cy-in-cy tests to verify reporter behavior for pass/fail tests, as well as new mocha snapshots to verify attempts. New tests were needed for this as the 'retries' option in testConfigOverrides currently is and will be invalid for experiment and will function as an override. tests run in the cy-in-cy tests are using globally configured experimentalRetries for the given tested project, which showcases the different behavior between attempts/retries and pass/fail status.
* chore: add unit test like driver test to verify the test object in mocha is decorated/handled properly in calculateTestStatus
* chore: add sanity system tests to verify console reporter output for experimental retries logic. Currently there is a bug in the reporter where the logged status doesnt wait for the aftereach to complete, which impacts the total exitCode and printed status.
* fix: aftereach console output. make sure to fail the test in the appropriate spot in runner.ts and not prematurely, which in turn updates the snapshots for cy-in-cy as the fail event comes later."
* chore: address comments from code review
* fix: make sure hook failures print outer status + attempts when the error is the hook itself.
* chore: improve types within calculateTestStatus inside mocha.ts
* Revert "feat: add burnIn Configuration option (currently a no-op) (#27377)"
This reverts commit c428443079.
* Revert "chore: add burnInTestAction capability (#27768)"
This reverts commit ae3df1a505.
* chore: run snapshot and binary jobs against experimental retries feature branch
* chore: add changelog entry (wip)
* Revert "fix snapshot added in v13 for module api to include test burn in experimentalflag"
This reverts commit bb5046c91e.
* Fix system tests
* Clear CircleCI cache
* Normalize retries config for test execution
* Fixed some unit tests
* update snapshots for newer test metadata
* Fix cy-in-cy snapshots
* update snapshots
* bump cache version
* chore: ensure legacy retry overrides work; reject exp. retries overrides (#28045)
* update changelog
* flip if statement in experimental retries option validation
* refactor invalid experimental retry override for more useful error msg
* revert testConfigOverrides snapshot
* update snapshots for test override sys test
* Update packages/config/src/validation.ts
Co-authored-by: Chris Breiding <chrisbreiding@users.noreply.github.com>
* succinct changelog entry; links to docs for details
* testConfigOverride system test snapshots
* Update .github/workflows/update_v8_snapshot_cache.yml
Co-authored-by: Ryan Manuel <ryanm@cypress.io>
* Update cli/CHANGELOG.md
Co-authored-by: Ryan Manuel <ryanm@cypress.io>
* Update packages/driver/src/cypress.ts
Co-authored-by: Ryan Manuel <ryanm@cypress.io>
* updating cache-version
* improve typescript usage when appending experimental retry options to experiments in Experimenets.vue
* Revert "improve typescript usage when appending experimental retry options to experiments in Experimenets.vue"
This reverts commit b459aba882.
* refactor test config override validation for experimental retry subkeys
* account for error throw differences in browsers in system tests
* bump circle cache
* bump circle cache again
---------
Co-authored-by: astone123 <adams@cypress.io>
Co-authored-by: mabela416 <mabel@cypress.io>
Co-authored-by: Muaz Othman <muaz@cypress.io>
Co-authored-by: Muaz Othman <muazweb@gmail.com>
Co-authored-by: Cacie Prins <cacie@cypress.io>
Co-authored-by: Cacie Prins <cacieprins@users.noreply.github.com>
Co-authored-by: Chris Breiding <chrisbreiding@users.noreply.github.com>
Co-authored-by: Ryan Manuel <ryanm@cypress.io>
Co-authored-by: Matthew Schile <mschile@cypress.io>
* report correct number of failures if browser crashes in the middle of a test suite
* report failure correctly when browser crashes during test
* refactor crash handling
* correct exit option check, clean up debug
* exit on success if exit option !== false
* use default stats when reporter stats are unavailable
* fix error messaging
* move reporter types to an intermediate .ts file, to be combined with reporter.js when migrated
* debug tab close test in ci
* move debug env from pkg to ci yml
* set debug env in spec
* fix pckg
* adds some logging to cri-client
* remove event emit logging from project-base
* revert snapshot for tab close system test
* fixes console output for no exit on success
* changelog
* changelog wsp
* cleanup
* clean up tests
* refactor to more straightforward control flow
* rm export for unused type
* correct tab close snapshot for ci
* new system test for mid-test config crash
* update snapshots
* fix: yield waited requests in the order that they occured
* fix firefox
* add changelog entry
* add test case for sorting
* add multi-alias test
* preserve alias order when sorting via requestId
* removed value parse code
* messy but more robust
* add comment, rename variable
* reduce alias count to account for webkit
* collate new tests under existing describe
---------
Co-authored-by: young-robot <adam.stonelord+fake@gmail.com>
Co-authored-by: David Kasper <dave.m.kasper@gmail.com>
* fix: set a crashed flag when cri detects a crash, throw on send if crashed
* changelog update
* Update cli/CHANGELOG.md
Co-authored-by: Ryan Manuel <ryanm@cypress.io>
* update snapshots
* updates changelog, updates crash handling snapshot
* electron tab crash snapshot fix
* electron tab crash snapshot...
* add wsp to eol on snapshot - maybe the diff?
* more whitespace?
* update snapshots again
* fix screenshot res in electron tab crash snap
* updates changelog
---------
Co-authored-by: Ryan Manuel <ryanm@cypress.io>
* chore: update the packages/electron README to update the electron updating process as it has changed/evolved since written
* chore: update images used in binary system tests
* Update packages/electron/README.md
Co-authored-by: Chris Breiding <chrisbreiding@users.noreply.github.com>
---------
Co-authored-by: Chris Breiding <chrisbreiding@users.noreply.github.com>
* feat: update to electron 25 and bump node dependencies. need to remove
custom docker image
* chore: update node versions to 18+ [run ci]
* chore: bump @types/node from v16 to v18 and bump the typescript supported CLI version from 3.9 to 4.4
* update mock-fs to 5.2.0 to fix BigInt issue (see https://github.com/tschaub/mock-fs/releases/tag/v5.1.4) [run ci]
* chore: update electron integrity check to pass create binary job
* chore: fix issues with achrinza/node-ipc not yet having node 18 engine support (only up to 17 by patching from 9.2.2 to 9.2.5 as seen in https://github.com/achrinza/node-ipc/pull/38. vue/cli-service is not maintained and we should migrate these over to vue create at some point in the near future to get rid of these resolutions
* allow for TLSv1 tests to work with node 18 as the express server running node 18 with tlsv1 needs to allow legacy ciphers
* chore: update snyk workflows to use node 18
* chore: add changelog
* update timeout
* more cleanup during binary build
* bump cache and run ci. include ignore engines for rwa and get rid of
16.16 image references
* chore: update FF tests to 115 as serialization now works with errors and click events do not get fired on buttons being typed into
* chore: don't error when symlink already exists
* chore: fix serialization test for newer versions of firefox
* chore: fix CI config
* chore: fix chrome system tests to work with chrome 114. updates mostly caused by bugs in screen height in chrome. see https://bugs.chromium.org/p/chromium/issues/detail?id=1416398
* chore: test binary against vite update in RWA
* remove wait for RWA as it shouldnt be needed for vite. link example recipes update to node 18
chore: remove commented out code
* chore: print message when DISABLE_SNAPSHOT_REQUIRE is set
* chore: clean out unneeded dependencies always
* chore: remove trailing space
* fix: propagate click events for enter and typing on firefox 106 or later
* chore: fix changelog failures
* fix: correctly simulate click events for buttons on keyup and space type in Firefox versions greater than 91 and simulate click for buttons on enter in Firefox versions greater than or equal to 106
* chore: add documentation to type to clarify firefox synthetic events
* chore: update protocol snapshot as order or log messages seems to have changed
* update comments
* sort commandLogChanged events for protocol
* chore: remove PR ids from CRA and CER as the PRs are merged into develop
---------
Co-authored-by: Ryan Manuel <ryanm@cypress.io>
Co-authored-by: Chris Breiding <chrisbreiding@gmail.com>
Co-authored-by: Matt Schile <mschile@cypress.io>
* basic structure and stubs for tests
* think this works, needs validation and icon
* partially working, needs i18n cleanup for failure count
* working with tests and i18n
* update to latest status icon
* fix debug integration tests
* removing ts-expect-error for isWindows util to match 13 branch
* add change log
* whatever
* speculate on release
* fix typo in template
