* chore: set up feature/test-burn-in feature branch
* feat: add burnIn Configuration option (currently a no-op) (#27377)
* feat: add the burnIn Configuration to the config package. Option
currently is a no-op
* chore: make burn in experimental
* chore: set experimentalBurnIn to false by default
* feat: add new experimental retries configuration (#27412)
* feat: implement the experimental retries configuration options to pair
with test burn in
* [run ci]
* fix cache invalidation [run ci]
* fix snapshot added in v13 for module api to include test burn in experimentalflag
* chore: fix merge conflict
* chore: add burnInTestAction capability (#27768)
* add burnInTestAction capability
* feat: add burn in capability for cloud
* chore: fix snapshot for record_spec
* feat: implement experimental retries (#27826)
* chore: format the retries/runner snapshot files to make diff easier
* feat: implement experimentalRetries strategies 'detect-flake-and-pass-on-threshold' and 'detect-flake-but-always-fail'. This should not be a breaking change, though it does modify mocha and the test object even when the experiment is not configured. This is to exercise the system and make sure things still work as expected even when we go GA. Test updates will follow in following commits.
* chore: update snapshots from system tests and cy-in-cy tests that now have the cypress test metadata property _cypressTestStatusInfo. tests have been added in the fail-with-[before|after]each specs to visually see the suite being skipped when developing.
* chore: add cy-in-cy tests to verify reporter behavior for pass/fail tests, as well as new mocha snapshots to verify attempts. New tests were needed for this as the 'retries' option in testConfigOverrides currently is and will be invalid for experiment and will function as an override. tests run in the cy-in-cy tests are using globally configured experimentalRetries for the given tested project, which showcases the different behavior between attempts/retries and pass/fail status.
* chore: add unit test like driver test to verify the test object in mocha is decorated/handled properly in calculateTestStatus
* chore: add sanity system tests to verify console reporter output for experimental retries logic. Currently there is a bug in the reporter where the logged status doesnt wait for the aftereach to complete, which impacts the total exitCode and printed status.
* fix: aftereach console output. make sure to fail the test in the appropriate spot in runner.ts and not prematurely, which in turn updates the snapshots for cy-in-cy as the fail event comes later."
* chore: address comments from code review
* fix: make sure hook failures print outer status + attempts when the error is the hook itself.
* chore: improve types within calculateTestStatus inside mocha.ts
* Revert "feat: add burnIn Configuration option (currently a no-op) (#27377)"
This reverts commit c428443079.
* Revert "chore: add burnInTestAction capability (#27768)"
This reverts commit ae3df1a505.
* chore: run snapshot and binary jobs against experimental retries feature branch
* chore: add changelog entry (wip)
* Revert "fix snapshot added in v13 for module api to include test burn in experimentalflag"
This reverts commit bb5046c91e.
* Fix system tests
* Clear CircleCI cache
* Normalize retries config for test execution
* Fixed some unit tests
* update snapshots for newer test metadata
* Fix cy-in-cy snapshots
* update snapshots
* bump cache version
* chore: ensure legacy retry overrides work; reject exp. retries overrides (#28045)
* update changelog
* flip if statement in experimental retries option validation
* refactor invalid experimental retry override for more useful error msg
* revert testConfigOverrides snapshot
* update snapshots for test override sys test
* Update packages/config/src/validation.ts
Co-authored-by: Chris Breiding <chrisbreiding@users.noreply.github.com>
* succinct changelog entry; links to docs for details
* testConfigOverride system test snapshots
* Update .github/workflows/update_v8_snapshot_cache.yml
Co-authored-by: Ryan Manuel <ryanm@cypress.io>
* Update cli/CHANGELOG.md
Co-authored-by: Ryan Manuel <ryanm@cypress.io>
* Update packages/driver/src/cypress.ts
Co-authored-by: Ryan Manuel <ryanm@cypress.io>
* updating cache-version
* improve typescript usage when appending experimental retry options to experiments in Experimenets.vue
* Revert "improve typescript usage when appending experimental retry options to experiments in Experimenets.vue"
This reverts commit b459aba882.
* refactor test config override validation for experimental retry subkeys
* account for error throw differences in browsers in system tests
* bump circle cache
* bump circle cache again
---------
Co-authored-by: astone123 <adams@cypress.io>
Co-authored-by: mabela416 <mabel@cypress.io>
Co-authored-by: Muaz Othman <muaz@cypress.io>
Co-authored-by: Muaz Othman <muazweb@gmail.com>
Co-authored-by: Cacie Prins <cacie@cypress.io>
Co-authored-by: Cacie Prins <cacieprins@users.noreply.github.com>
Co-authored-by: Chris Breiding <chrisbreiding@users.noreply.github.com>
Co-authored-by: Ryan Manuel <ryanm@cypress.io>
Co-authored-by: Matthew Schile <mschile@cypress.io>
* report correct number of failures if browser crashes in the middle of a test suite
* report failure correctly when browser crashes during test
* refactor crash handling
* correct exit option check, clean up debug
* exit on success if exit option !== false
* use default stats when reporter stats are unavailable
* fix error messaging
* move reporter types to an intermediate .ts file, to be combined with reporter.js when migrated
* debug tab close test in ci
* move debug env from pkg to ci yml
* set debug env in spec
* fix pckg
* adds some logging to cri-client
* remove event emit logging from project-base
* revert snapshot for tab close system test
* fixes console output for no exit on success
* changelog
* changelog wsp
* cleanup
* clean up tests
* refactor to more straightforward control flow
* rm export for unused type
* correct tab close snapshot for ci
* new system test for mid-test config crash
* update snapshots
* fix: set a crashed flag when cri detects a crash, throw on send if crashed
* changelog update
* Update cli/CHANGELOG.md
Co-authored-by: Ryan Manuel <ryanm@cypress.io>
* update snapshots
* updates changelog, updates crash handling snapshot
* electron tab crash snapshot fix
* electron tab crash snapshot...
* add wsp to eol on snapshot - maybe the diff?
* more whitespace?
* update snapshots again
* fix screenshot res in electron tab crash snap
* updates changelog
---------
Co-authored-by: Ryan Manuel <ryanm@cypress.io>
* feat: update to electron 25 and bump node dependencies. need to remove
custom docker image
* chore: update node versions to 18+ [run ci]
* chore: bump @types/node from v16 to v18 and bump the typescript supported CLI version from 3.9 to 4.4
* update mock-fs to 5.2.0 to fix BigInt issue (see https://github.com/tschaub/mock-fs/releases/tag/v5.1.4) [run ci]
* chore: update electron integrity check to pass create binary job
* chore: fix issues with achrinza/node-ipc not yet having node 18 engine support (only up to 17 by patching from 9.2.2 to 9.2.5 as seen in https://github.com/achrinza/node-ipc/pull/38. vue/cli-service is not maintained and we should migrate these over to vue create at some point in the near future to get rid of these resolutions
* allow for TLSv1 tests to work with node 18 as the express server running node 18 with tlsv1 needs to allow legacy ciphers
* chore: update snyk workflows to use node 18
* chore: add changelog
* update timeout
* more cleanup during binary build
* bump cache and run ci. include ignore engines for rwa and get rid of
16.16 image references
* chore: update FF tests to 115 as serialization now works with errors and click events do not get fired on buttons being typed into
* chore: don't error when symlink already exists
* chore: fix serialization test for newer versions of firefox
* chore: fix CI config
* chore: fix chrome system tests to work with chrome 114. updates mostly caused by bugs in screen height in chrome. see https://bugs.chromium.org/p/chromium/issues/detail?id=1416398
* chore: test binary against vite update in RWA
* remove wait for RWA as it shouldnt be needed for vite. link example recipes update to node 18
chore: remove commented out code
* chore: print message when DISABLE_SNAPSHOT_REQUIRE is set
* chore: clean out unneeded dependencies always
* chore: remove trailing space
* fix: propagate click events for enter and typing on firefox 106 or later
* chore: fix changelog failures
* fix: correctly simulate click events for buttons on keyup and space type in Firefox versions greater than 91 and simulate click for buttons on enter in Firefox versions greater than or equal to 106
* chore: add documentation to type to clarify firefox synthetic events
* chore: update protocol snapshot as order or log messages seems to have changed
* update comments
* sort commandLogChanged events for protocol
* chore: remove PR ids from CRA and CER as the PRs are merged into develop
---------
Co-authored-by: Ryan Manuel <ryanm@cypress.io>
Co-authored-by: Chris Breiding <chrisbreiding@gmail.com>
Co-authored-by: Matt Schile <mschile@cypress.io>
* feat: add v13 splash page content (wip)
chore: add video link for v13 inside graphql context and root resolver. add migration tests outside of migration manager and update component tests to test aspects
chore: update copy to most recent. Needs onlink deployments
chore: opt for named slot in major version welcome as we still want to show the changes whether or not the video is present
chore: fix video margin by shrinking the margin for a more appealing loadout
chore: update onlinks for what is expected in deploy
chore: adjust margins around video, move video up, and reduce margin on major welcome title
chore: bump release date to 8/29
chore: update stubbed query to reference actual video
* Update packages/launchpad/src/migration/MajorVersionWelcome.vue
Co-authored-by: Mark Noonan <mark@cypress.io>
* Update packages/launchpad/src/migration/MajorVersionWelcome.vue
Co-authored-by: Mark Noonan <mark@cypress.io>
* chore: update code elements to be inlinecodefragments
* chore: update utm links to reflect correct nomenclature
* stub migration src in network performance launchpad test on video as abortcontroller is difficult to test with current ctx setup
---------
Co-authored-by: Mark Noonan <mark@cypress.io>
* chore: update ts-loader from v8 to v9
* chore: update sass-loader from v10 to v13
* chore: update postcss-loader from v4 to v7
* chore: update mini-css-extract-plugin from v1 to v2
* chore: update html-webpack-plugin from v4 to v5
* chore: update css-loader from v5 to v6
* chore: update copy-webpack-plugin from v6 to v 11
* chore: update babel and babel loader to latest
* chore: remove Url import from validation as it should be available globally
* chore: update coffee-loader to latest and coffeescript to 2.6.0. coffee
script > 2.6.0 has issues with packheard transpilation, which likely
isnt worth the squeeze here to update it further
* chore: fix react component tests with webpack 4 as babl loader 9 only works with webpack 5, so we need ot pin the dep to 8 in the system-test project
* chore: remove dependency on file-loader and opt for ubilt in assert handling where applicable. Additionally, remove react-svg-loader and move to svgr loader inside the runner to bundle svgs needed for react components. Make sure that all SVGs are being imported correctly
chore: tell webpack preprocessor to process svgs inside of e2e tests for
the reporter. We did not need this before because we were using
react-svg-loader, which was used directly in source to process the svg
and allowed the svg to work under test. Since the source code is now
bundler independent, we need to tell cypress to bundle the svgs under
test. To do this, we pass our webpack config into
@cypress/webpack-preprocessor to allow loading the svgs. This is not a
breaking change.
* chore: switch from hash to contenthash
* chore: remove use of moduleIds for chunkIds as the defaults are likely better and handle caching/debugging better
* chore: mark tests that need to be updated as node 12 and 14 are no longer supported. node 12 doesnt ship with fs/promises, which causes fatal errors with babel-loader 9
try to test against new docker images [run ci]
* chore: updating v8 snapshot cache
* chore: updating v8 snapshot cache
* chore: updating v8 snapshot cache
* skip http/https form submittal unit tests for experimental webkit due to the clientRoute 404ing after submitting
* chore: pin webpack 5 for component tests in cy in cy to prevent using 4 from hoisted deps [run ci]
* build binaries [run ci]
* chore: bump circle cache [run ci]
* fix: turn off svgo from @svgr/webpack to preserve our scss class names within the svg. see https://react-svgr.com/docs/options/#svgo [run ci]
---------
Co-authored-by: cypress-bot[bot] <+cypress-bot[bot]@users.noreply.github.com>
* chore: [run ci] does further prerequisites for webpack 5:
https://webpack.js.org/migrate/5/#make-sure-your-build-has-no-errors-or-warningshttps://webpack.js.org/migrate/5/#make-sure-to-use-modehttps://webpack.js.org/migrate/5/#update-outdated-optionshttps://webpack.js.org/migrate/5/#test-webpack-5-compatibility
app builds and runs locally. Time to test out in CI and see if buffer or
process need to be polyfilled by the build
* chore: upgrade to webpack 5 and do the bare minimum to get it working
* chore: get @packages/extension working
* chore: add TODOs to finish after webpack 5 update
* chore: update the webpack config for npm/webpack-batteries-included-preprocessor to be webpack 5 compliant
* chore: patch whatwg-url 7.1.0. package 'source-map' uses
whatwg-url@7.1.0 which has a dependency on punycode node expected API.
since punycode is now polyfilled for us implicitly via the punycode npm
package, the API signatures are a bit different
https://github.com/mathiasbynens/punycode.js/blob/main/punycode.js#L101
vs https://nodejs.org/api/punycode.html#punycodeucs2. The patch uses the
punycode npm package expected API and is needed for source maps to work
for cy.origin() dependencies for Cypress.require()
* chore: convert whatwg patch into dev patch as source-map is not installed when building the binary / installing prod dependencies
* chore: only move production level patches into the binary dist directory for yarn install --production
* chore: remove --openssl-legacy-provider code for node versions 17 and over as webpack has been updated to v5
* chore: fix the webpack-batteries-included-preprocessor tests by shimming the correct node globals and built ins
* chore: provide the define plugin and evalDevtoolPlugin again as we need define in order to build the react-dom library correctly in the bundle to not include the development version
* chore: updating v8 snapshot cache
* chore: updating v8 snapshot cache
* chore: updating v8 snapshot cache
* chore: fix the webpack preprocessor not to change promise references under the hood when compiling the first bundle, as it was causing the webpack preprocessor to hang as the reference itself was different
* chore: fix issues from readFile that were caused by Webpack 5 using 'path-browserify'
* chore: update chrome component testing snapshots to match Webpack 5 changes
* chore: fix mismatched snapshots from webpack 5 update
* chore: use Cypress.Buffer instead of Buffer for selectFile system test to avoid having to polyfill Buffer from webpack
* chore: fix system test webpack path that now includes e2e workspace
* chore: patch enhanced-resolve to properly discover the pnp api for the yarn_v3.1.1_pnp_spec.ts system test. see https://github.com/webpack/enhanced-resolve/issues/263 for more details
* chore: set stats to 'none' for experimentalSingleTabMode to prevent different webpack compiled terminal formatting in the snapshot between local and CI.
* chore: fix node built in tests and configure webpack-batteries-included-preprocessor correctly
* chore: fallback to buffer correctly in config, even though there is no impact due to the provide plugin
* Update binary-cleanup.js to exclude added build dependencies for webpack
5 added by webpack-terser-plugin under the hood
* chore: add stream-browserify to webpack preprocessor batteries included as a dep as its used in the config [run ci]
* chore: make sure process and buffer are installed in the CLI for webpack provide
* chore: build cross platform binaries [run ci]
* chore: fix webpack evalDevToolPlugin instantiation [run ci]
* run all binary jobs [run ci]
* chore: updating v8 snapshot cache
* add find-up to the entry points that need to be kept
* chore: updating v8 snapshot cache
* chore: updating v8 snapshot cache
* chore: fix mocha build warnings
* chore: fix STRIPPED_INTEGRITY_TAG import warnings
* chore: add changelog event
---------
Co-authored-by: cypress-bot[bot] <+cypress-bot[bot]@users.noreply.github.com>
Co-authored-by: Ryan Manuel <ryanm@cypress.io>
* chore: upgrade webpack dependencies to latest sub 5 compat [run ci]
* chore: updating v8 snapshot cache
* chore: updating v8 snapshot cache
* chore: updating v8 snapshot cache
* chore: build binaries [run ci]
* chore: update snapshot for darwin
* chore: downgrade tsconfig-paths-webpack-loader from v4 to v3 [run ci]
* [run ci]
* chore: add rimraf in @packages/runner to use in prebuild command to clear out dist directory to get windows compat [run ci]
* don't pass -rf flags into rimraf as it is inherit with the command
---------
Co-authored-by: cypress-bot[bot] <+cypress-bot[bot]@users.noreply.github.com>
* basic check
* wonky to say the least
* need to figure exit mechanism, then add steps to validation
* partial cleanup for step validation
* umm working?
* remove comment
* remove old test code
* rename variable
* chore: add system test for desired behavior
* fix: issue where process was exiting when window was being closed while launch process was occuring
* chore: add changelog entry
* chore: update darwin v8 snapshot
* BREAKING CHANGE: set video to false by default (system tests need updating).
* Update cli/CHANGELOG.md
Co-authored-by: Emily Rohrbough <emilyrohrbough@users.noreply.github.com>
* chore: update type comments
* chore: update protocol snapshot
* run ci
* run ci
* set video to true for chrome browser crash test
* chore: put in workaround for failing system test spec to be fixed in 27062
* chore: allow retries on actionability tests to be at least one retry as the CI tests run faster without video on
* chore: fix flaky navigation test where done is called multiple times almsot always, but sometimes throws an error
---------
Co-authored-by: Emily Rohrbough <emilyrohrbough@users.noreply.github.com>
Ryan Manuel