mirror/cypress
1
0
Fork 0
mirror of https://github.com/cypress-io/cypress.git synced 2026-04-24 07:59:12 -05:00
Code Issues Packages Projects Releases Wiki Activity

feat: more informative error messages when Test Replay fails to capture (#29312)

Browse Source 
* new error messages

* errors for initialization and capturing test replay

* messaging the case where no protocol instance but also no fatal error

* adds suggested remedies to initialization errors

* differentiation between network and http errors, initial work on db missing error

* improve db not found error

* add new error type to schema

* rm commented dead code

* clean up network error creation in uploadStream

* make artifact confirmation error msg more general

* test display of put instance artifacts failure

* changelog

* ensure we are displaying errors even in quiet mode

* fix pending changelog

* new snapshots for protocol errors

* improve aggregate error

* resolved html error snapshots

* fix check-ts

* fix test

* sanitize temp dir in CLOUD_PROTOCOL_CAPTURE_FAILURE error msg

* changelog

* fix double entry of certain network errors, error message prop for network errors

* fix url arg for network error

* Update packages/server/lib/cloud/artifacts/upload_artifacts.ts

Co-authored-by: Ryan Manuel <ryanm@cypress.io>

* rm extra formatting from debug

* snapshot whitespace

* changelog update

* +pending

* Update cli/CHANGELOG.md

* resolve snapshots?

* does moving the stack trace fix whitespace in snapshots in ci?

* maybe fixes whitespace on electron now?

* fully normalize stack traces

* remove full normalization

* maybe debug stack normalization

* rm stack traces from error messages

---------

Co-authored-by: Jennifer Shehane <jennifer@cypress.io>
Co-authored-by: Ryan Manuel <ryanm@cypress.io>
This commit is contained in:
Cacie Prins
2024-05-01 14:32:53 -04:00
committed by GitHub
parent e62faff9de
commit fc88764ad5
28 changed files with 1276 additions and 74 deletions
Download Patch File Download Diff File Expand all files Collapse all files
cli/CHANGELOG.md
+5 -1
View File
@@ -1,8 +1,12 @@
<!-- See the ../guides/writing-the-cypress-changelog.md for details on writing the changelog. -->
## 13.8.2
## 13.9.0
_Released 5/7/2024 (PENDING)_
**Features:**
- Added more descriptive error messages when Test Replay fails to record or upload. Addresses [#29022](https://github.com/cypress-io/cypress/issues/29022).
**Dependency Updates:**
- Updated electron from `27.1.3` to `27.3.10` to address [CVE-2024-3156](https://nvd.nist.gov/vuln/detail/CVE-2024-3156). Addressed in [#29367](https://github.com/cypress-io/cypress/pull/29367).
packages/errors/__snapshot-html__/CLOUD_CANNOT_UPLOAD_ARTIFACTS_PROTOCOL.html → packages/errors/__snapshot-html__/CLOUD_CANNOT_CONFIRM_ARTIFACTS.html
Generated
+1 -1
View File
@@ -34,7 +34,7 @@
</style>
</head>
<body><pre><span style="color:#e05561">Warning: We encountered an error while confirming the upload of artifacts.<span style="color:#e6e6e6">
<body><pre><span style="color:#e05561">Warning: We encountered an error while confirming the upload of artifacts for this spec.<span style="color:#e6e6e6">
<span style="color:#e05561"><span style="color:#e6e6e6">
<span style="color:#e05561">These results will not display artifacts.<span style="color:#e6e6e6">
<span style="color:#e05561"><span style="color:#e6e6e6">
packages/errors/__snapshot-html__/CLOUD_PROTOCOL_CAPTURE_FAILURE.html
Generated
+49
View File
@@ -0,0 +1,49 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Courier+Prime&display=swap" rel="stylesheet">
<style>
body {
font-family: "Courier Prime", Courier, monospace;
padding: 0 1em;
line-height: 1.4;
color: #eee;
background-color: #111;
}
pre {
padding: 0 0;
margin: 0 0;
font-family: "Courier Prime", Courier, monospace;
}
body {
margin: 5px;
padding: 0;
overflow: hidden;
}
pre {
white-space: pre-wrap;
word-break: break-word;
-webkit-font-smoothing: antialiased;
}
</style>
</head>
<body><pre><span style="color:#e05561">Warning: We encountered an error while recording Test Replay data for this spec.<span style="color:#e6e6e6">
<span style="color:#e05561"><span style="color:#e6e6e6">
<span style="color:#e05561">These results will not display Test Replay recordings.<span style="color:#e6e6e6">
<span style="color:#e05561"><span style="color:#e6e6e6">
<span style="color:#e05561">This can happen for many reasons. If this problem persists:<span style="color:#e6e6e6">
<span style="color:#e05561"><span style="color:#e6e6e6">
<span style="color:#e05561">- Try increasing the available disk space.<span style="color:#e6e6e6">
<span style="color:#e05561">- Ensure that <span style="color:#4ec4ff">/os/tmpdir/cypress/protocol<span style="color:#e05561"> is both readable and writable.<span style="color:#e6e6e6">
<span style="color:#e05561"><span style="color:#e6e6e6">
<span style="color:#e05561">This error will not affect or change the exit code.<span style="color:#e6e6e6">
<span style="color:#e05561"><span style="color:#e6e6e6">
<span style="color:#e05561"><span style="color:#de73ff">Error: fail whale<span style="color:#e05561"><span style="color:#e6e6e6"></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span>
</pre></body></html>
packages/errors/__snapshot-html__/CLOUD_PROTOCOL_INITIALIZATION_FAILURE.html
Generated
+44
View File
@@ -0,0 +1,44 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Courier+Prime&display=swap" rel="stylesheet">
<style>
body {
font-family: "Courier Prime", Courier, monospace;
padding: 0 1em;
line-height: 1.4;
color: #eee;
background-color: #111;
}
pre {
padding: 0 0;
margin: 0 0;
font-family: "Courier Prime", Courier, monospace;
}
body {
margin: 5px;
padding: 0;
overflow: hidden;
}
pre {
white-space: pre-wrap;
word-break: break-word;
-webkit-font-smoothing: antialiased;
}
</style>
</head>
<body><pre><span style="color:#e05561">Warning: We encountered an error while initializing the Test Replay recording for this spec.<span style="color:#e6e6e6">
<span style="color:#e05561"><span style="color:#e6e6e6">
<span style="color:#e05561">These results will not display Test Replay recordings.<span style="color:#e6e6e6">
<span style="color:#e05561"><span style="color:#e6e6e6">
<span style="color:#e05561">This error will not affect or change the exit code.<span style="color:#e6e6e6">
<span style="color:#e05561"><span style="color:#e6e6e6">
<span style="color:#e05561"><span style="color:#de73ff">Error: fail whale<span style="color:#e05561"><span style="color:#e6e6e6"></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span>
</pre></body></html>
packages/errors/__snapshot-html__/CLOUD_PROTOCOL_UPLOAD_AGGREGATE_ERROR - withNetworkError.html
Generated
+46
View File
@@ -0,0 +1,46 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Courier+Prime&display=swap" rel="stylesheet">
<style>
body {
font-family: "Courier Prime", Courier, monospace;
padding: 0 1em;
line-height: 1.4;
color: #eee;
background-color: #111;
}
pre {
padding: 0 0;
margin: 0 0;
font-family: "Courier Prime", Courier, monospace;
}
body {
margin: 5px;
padding: 0;
overflow: hidden;
}
pre {
white-space: pre-wrap;
word-break: break-word;
-webkit-font-smoothing: antialiased;
}
</style>
</head>
<body><pre><span style="color:#e05561">Warning: We encountered multiple errors while uploading the Test Replay recording for this spec.<span style="color:#e6e6e6">
<span style="color:#e05561"><span style="color:#e6e6e6">
<span style="color:#e05561">We attempted to upload the Test Replay recording <span style="color:#de73ff">3<span style="color:#e05561"> times.<span style="color:#e6e6e6">
<span style="color:#e05561"><span style="color:#e6e6e6">
<span style="color:#e05561">Some or all of the errors encountered are system-level network errors. Please verify your network configuration for connecting to <span style="color:#de73ff">http://some/url<span style="color:#e05561"><span style="color:#e6e6e6">
<span style="color:#e05561"><span style="color:#e6e6e6">
<span style="color:#e05561"><span style="color:#4f5666"> - <span style="color:#e05561"><span style="color:#4ec4ff">http://some/url: ECONNRESET<span style="color:#e05561"><span style="color:#e6e6e6">
<span style="color:#e05561"><span style="color:#4f5666"> - <span style="color:#e05561"><span style="color:#4ec4ff">fail whale<span style="color:#e05561"><span style="color:#e6e6e6">
<span style="color:#e05561"><span style="color:#4f5666"> - <span style="color:#e05561"><span style="color:#4ec4ff">fail whale<span style="color:#e05561"><span style="color:#e6e6e6"></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span>
</pre></body></html>
packages/errors/__snapshot-html__/CLOUD_PROTOCOL_UPLOAD_AGGREGATE_ERROR.html
Generated
+44
View File
@@ -0,0 +1,44 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Courier+Prime&display=swap" rel="stylesheet">
<style>
body {
font-family: "Courier Prime", Courier, monospace;
padding: 0 1em;
line-height: 1.4;
color: #eee;
background-color: #111;
}
pre {
padding: 0 0;
margin: 0 0;
font-family: "Courier Prime", Courier, monospace;
}
body {
margin: 5px;
padding: 0;
overflow: hidden;
}
pre {
white-space: pre-wrap;
word-break: break-word;
-webkit-font-smoothing: antialiased;
}
</style>
</head>
<body><pre><span style="color:#e05561">Warning: We encountered multiple errors while uploading the Test Replay recording for this spec.<span style="color:#e6e6e6">
<span style="color:#e05561"><span style="color:#e6e6e6">
<span style="color:#e05561">We attempted to upload the Test Replay recording <span style="color:#de73ff">3<span style="color:#e05561"> times.<span style="color:#e6e6e6">
<span style="color:#e05561"><span style="color:#e6e6e6">
<span style="color:#e05561"><span style="color:#4f5666"> - <span style="color:#e05561"><span style="color:#4ec4ff">fail whale<span style="color:#e05561"><span style="color:#e6e6e6">
<span style="color:#e05561"><span style="color:#4f5666"> - <span style="color:#e05561"><span style="color:#4ec4ff">fail whale<span style="color:#e05561"><span style="color:#e6e6e6">
<span style="color:#e05561"><span style="color:#4f5666"> - <span style="color:#e05561"><span style="color:#4ec4ff">fail whale<span style="color:#e05561"><span style="color:#e6e6e6"></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span>
</pre></body></html>
packages/errors/__snapshot-html__/CLOUD_PROTOCOL_UPLOAD_HTTP_FAILURE.html
Generated
+44
View File
@@ -0,0 +1,44 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Courier+Prime&display=swap" rel="stylesheet">
<style>
body {
font-family: "Courier Prime", Courier, monospace;
padding: 0 1em;
line-height: 1.4;
color: #eee;
background-color: #111;
}
pre {
padding: 0 0;
margin: 0 0;
font-family: "Courier Prime", Courier, monospace;
}
body {
margin: 5px;
padding: 0;
overflow: hidden;
}
pre {
white-space: pre-wrap;
word-break: break-word;
-webkit-font-smoothing: antialiased;
}
</style>
</head>
<body><pre><span style="color:#e05561">Warning: We encountered an HTTP error while uploading the Test Replay recording for this spec.<span style="color:#e6e6e6">
<span style="color:#e05561"><span style="color:#e6e6e6">
<span style="color:#e05561">These results will not display Test Replay recordings.<span style="color:#e6e6e6">
<span style="color:#e05561"><span style="color:#e6e6e6">
<span style="color:#e05561">This error will not affect or change the exit code.<span style="color:#e6e6e6">
<span style="color:#e05561"><span style="color:#e6e6e6">
<span style="color:#e05561"><span style="color:#4ec4ff">https://some/url<span style="color:#e05561"> responded with HTTP <span style="color:#de73ff">500<span style="color:#e05561">: <span style="color:#de73ff">Internal Server Error<span style="color:#e05561"><span style="color:#e6e6e6"></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span>
</pre></body></html>
packages/errors/__snapshot-html__/CLOUD_PROTOCOL_UPLOAD_NEWORK_FAILURE.html
Generated
+46
View File
@@ -0,0 +1,46 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Courier+Prime&display=swap" rel="stylesheet">
<style>
body {
font-family: "Courier Prime", Courier, monospace;
padding: 0 1em;
line-height: 1.4;
color: #eee;
background-color: #111;
}
pre {
padding: 0 0;
margin: 0 0;
font-family: "Courier Prime", Courier, monospace;
}
body {
margin: 5px;
padding: 0;
overflow: hidden;
}
pre {
white-space: pre-wrap;
word-break: break-word;
-webkit-font-smoothing: antialiased;
}
</style>
</head>
<body><pre><span style="color:#e05561">Warning: We encountered a network error while uploading the Test Replay recording for this spec.<span style="color:#e6e6e6">
<span style="color:#e05561"><span style="color:#e6e6e6">
<span style="color:#e05561">Please verify your network configuration for accessing <span style="color:#4ec4ff">https://some/url<span style="color:#e05561"><span style="color:#e6e6e6">
<span style="color:#e05561"><span style="color:#e6e6e6">
<span style="color:#e05561">These results will not display Test Replay recordings.<span style="color:#e6e6e6">
<span style="color:#e05561"><span style="color:#e6e6e6">
<span style="color:#e05561">This error will not affect or change the exit code.<span style="color:#e6e6e6">
<span style="color:#e05561"><span style="color:#e6e6e6">
<span style="color:#e05561"><span style="color:#de73ff">Error: fail whale<span style="color:#e05561"><span style="color:#e6e6e6"></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span>
</pre></body></html>
packages/errors/src/errors.ts
+75 -2
View File
@@ -1,4 +1,5 @@
import AU from 'ansi_up'
import os from 'os'
/* eslint-disable no-console */
import chalk from 'chalk'
import _ from 'lodash'
@@ -529,9 +530,9 @@ export const AllCypressErrors = {
${fmt.highlightSecondary(apiErr)}`
},
CLOUD_CANNOT_UPLOAD_ARTIFACTS_PROTOCOL: (apiErr: Error) => {
CLOUD_CANNOT_CONFIRM_ARTIFACTS: (apiErr: Error) => {
return errTemplate`\
Warning: We encountered an error while confirming the upload of artifacts.
Warning: We encountered an error while confirming the upload of artifacts for this spec.
These results will not display artifacts.
@@ -539,6 +540,78 @@ export const AllCypressErrors = {
${fmt.highlightSecondary(apiErr)}`
},
CLOUD_PROTOCOL_INITIALIZATION_FAILURE: (error: Error) => {
return errTemplate`\
Warning: We encountered an error while initializing the Test Replay recording for this spec.
These results will not display Test Replay recordings.
This error will not affect or change the exit code.
${fmt.highlightSecondary(error)}`
},
CLOUD_PROTOCOL_CAPTURE_FAILURE: (error: Error) => {
return errTemplate`\
Warning: We encountered an error while recording Test Replay data for this spec.
These results will not display Test Replay recordings.
This can happen for many reasons. If this problem persists:
- Try increasing the available disk space.
- Ensure that ${fmt.path(path.join(os.tmpdir(), 'cypress', 'protocol'))} is both readable and writable.
This error will not affect or change the exit code.
${fmt.highlightSecondary(error)}`
},
CLOUD_PROTOCOL_UPLOAD_HTTP_FAILURE: (error: Error & { url: string, status: number, statusText: string }) => {
return errTemplate`\
Warning: We encountered an HTTP error while uploading the Test Replay recording for this spec.
These results will not display Test Replay recordings.
This error will not affect or change the exit code.
${fmt.url(error.url)} responded with HTTP ${fmt.stringify(error.status)}: ${fmt.highlightSecondary(error.statusText)}`
},
CLOUD_PROTOCOL_UPLOAD_NEWORK_FAILURE: (error: Error & { url: string }) => {
return errTemplate`\
Warning: We encountered a network error while uploading the Test Replay recording for this spec.
Please verify your network configuration for accessing ${fmt.url(error.url)}
These results will not display Test Replay recordings.
This error will not affect or change the exit code.
${fmt.highlightSecondary(error)}`
},
CLOUD_PROTOCOL_UPLOAD_AGGREGATE_ERROR: (error: {
errors: (Error & { kind?: 'NetworkError' | 'HttpError', url: string })[]
}) => {
if (error.errors.length === 1) {
if (error.errors[0]?.kind === 'NetworkError') {
return AllCypressErrors.CLOUD_PROTOCOL_UPLOAD_NEWORK_FAILURE(error.errors[0])
}
return AllCypressErrors.CLOUD_PROTOCOL_UPLOAD_HTTP_FAILURE(error.errors[0] as Error & { url: string, status: number, statusText: string})
}
let networkErr = error.errors.find((err) => {
return err.kind === 'NetworkError'
})
const recommendation = networkErr ? errPartial`Some or all of the errors encountered are system-level network errors. Please verify your network configuration for connecting to ${fmt.highlightSecondary(networkErr.url)}` : null
return errTemplate`\
Warning: We encountered multiple errors while uploading the Test Replay recording for this spec.
We attempted to upload the Test Replay recording ${fmt.stringify(error.errors.length)} times.
${recommendation}
${fmt.listItems(error.errors.map((error) => error.message))}`
},
CLOUD_CANNOT_CREATE_RUN_OR_INSTANCE: (apiErr: Error) => {
return errTemplate`\
Warning: We encountered an error communicating with our servers.
packages/errors/test/unit/visualSnapshotErrors_spec.ts
+63 -1
View File
@@ -8,6 +8,7 @@ import path from 'path'
import sinon, { SinonSpy } from 'sinon'
import * as errors from '../../src'
import { convertHtmlToImage } from '../support/utils'
import os from 'os'
// For importing the files below
process.env.CYPRESS_INTERNAL_ENV = 'test'
@@ -67,6 +68,7 @@ const sanitize = (str: string) => {
return str
.split(lineAndColNumsRe).join('')
.split(cypressRootPath).join('cypress')
.split(os.tmpdir()).join('/os/tmpdir')
}
const snapshotAndTestErrorConsole = async function (errorFileName: string) {
@@ -637,7 +639,7 @@ describe('visual error templates', () => {
default: [err],
}
},
CLOUD_CANNOT_UPLOAD_ARTIFACTS_PROTOCOL: () => {
CLOUD_CANNOT_CONFIRM_ARTIFACTS: () => {
return {
default: [makeErr()],
}
@@ -649,6 +651,66 @@ describe('visual error templates', () => {
default: [err],
}
},
CLOUD_PROTOCOL_INITIALIZATION_FAILURE: () => {
const err = makeErr()
return {
default: [err],
}
},
CLOUD_PROTOCOL_CAPTURE_FAILURE: () => {
const err = makeErr()
return {
default: [err],
}
},
CLOUD_PROTOCOL_UPLOAD_HTTP_FAILURE: () => {
// @ts-expect-error
const err: Error & { status: number, statusText: string, url: string } = makeErr()
err.status = 500
err.statusText = 'Internal Server Error'
err.url = 'https://some/url'
return {
default: [err],
}
},
CLOUD_PROTOCOL_UPLOAD_NEWORK_FAILURE: () => {
// @ts-expect-error
const err: Error & { url: string } = makeErr()
err.url = 'https://some/url'
return {
default: [err],
}
},
CLOUD_PROTOCOL_UPLOAD_AGGREGATE_ERROR: () => {
// @ts-expect-error
const aggregateError: Error & { errors: any[] } = makeErr()
// @ts-expect-error
const aggregateErrorWithNetworkError: Error & { errors: any[] } = makeErr()
const errOne = makeErr()
const errTwo = makeErr()
const errThree = makeErr()
aggregateError.errors = [errOne, errTwo, errThree]
// @ts-expect-error
const errNetworkErr: Error & { kind: string, url: string } = new Error('http://some/url: ECONNRESET')
errNetworkErr.kind = 'NetworkError'
errNetworkErr.url = 'http://some/url'
aggregateErrorWithNetworkError.errors = [errNetworkErr, errTwo, errThree]
return {
default: [aggregateError],
withNetworkError: [aggregateErrorWithNetworkError],
}
},
CLOUD_RECORD_KEY_NOT_VALID: () => {
return {
default: ['record-key-123', 'project-id-123'],
packages/graphql/schemas/schema.graphql
+6 -1
View File
@@ -1137,17 +1137,22 @@ enum ErrorTypeEnum {
CLOUD_AUTO_CANCEL_MISMATCH
CLOUD_AUTO_CANCEL_NOT_AVAILABLE_IN_PLAN
CLOUD_CANCEL_SKIPPED_SPEC
CLOUD_CANNOT_CONFIRM_ARTIFACTS
CLOUD_CANNOT_CREATE_RUN_OR_INSTANCE
CLOUD_CANNOT_PROCEED_IN_PARALLEL
CLOUD_CANNOT_PROCEED_IN_SERIAL
CLOUD_CANNOT_UPLOAD_ARTIFACTS
CLOUD_CANNOT_UPLOAD_ARTIFACTS_PROTOCOL
CLOUD_GRAPHQL_ERROR
CLOUD_INVALID_RUN_REQUEST
CLOUD_PARALLEL_DISALLOWED
CLOUD_PARALLEL_GROUP_PARAMS_MISMATCH
CLOUD_PARALLEL_REQUIRED
CLOUD_PROJECT_NOT_FOUND
CLOUD_PROTOCOL_CAPTURE_FAILURE
CLOUD_PROTOCOL_INITIALIZATION_FAILURE
CLOUD_PROTOCOL_UPLOAD_AGGREGATE_ERROR
CLOUD_PROTOCOL_UPLOAD_HTTP_FAILURE
CLOUD_PROTOCOL_UPLOAD_NEWORK_FAILURE
CLOUD_RECORD_KEY_NOT_VALID
CLOUD_RUN_GROUP_NAME_NOT_UNIQUE
CLOUD_STALE_RUN
packages/server/lib/cloud/api/http_error.ts
+15 -12
View File
@@ -1,32 +1,35 @@
const SENSITIVE_KEYS = Object.freeze(['x-amz-credential', 'x-amz-signature', 'Signature', 'AWSAccessKeyId'])
const scrubUrl = (url: string, sensitiveKeys: readonly string[]): string => {
const parsedUrl = new URL(url)
import { scrubUrl } from './scrub_url'
for (const [key, value] of parsedUrl.searchParams) {
if (sensitiveKeys.includes(key)) {
parsedUrl.searchParams.set(key, 'X'.repeat(value.length))
}
}
return parsedUrl.href
}
export const HttpErrorKind = 'HttpError'
export class HttpError extends Error {
public readonly kind = HttpErrorKind
constructor (
message: string,
public readonly url: string,
public readonly status: number,
public readonly statusText: string,
public readonly originalResponse: Response,
) {
super(message)
}
public static isHttpError (error: Error & { kind?: any, originalResponse?: Response }): error is HttpError {
return error?.kind === HttpErrorKind && Boolean(error.originalResponse)
}
public static async fromResponse (response: Response): Promise<HttpError> {
const status = response.status
const statusText = await (response.json().catch(() => {
return response.statusText
}))
const scrubbedUrl = scrubUrl(response.url)
return new HttpError(
`${status} ${statusText} (${scrubUrl(response.url, SENSITIVE_KEYS)})`,
`${status} ${statusText} (${scrubUrl(response.url)})`,
scrubbedUrl,
status,
statusText,
response,
)
}
packages/server/lib/cloud/api/network_error.ts
+15
View File
@@ -0,0 +1,15 @@
const NetworkErrorKind = 'NetworkError'
export class NetworkError extends Error {
public readonly kind = NetworkErrorKind
constructor (
public readonly originalError: Error,
public readonly url: string,
) {
super(originalError.message)
}
static isNetworkError (error: Error & { url?: string, kind?: string }): error is NetworkError {
return error?.kind === NetworkErrorKind
}
}
packages/server/lib/cloud/api/scrub_url.ts
+13
View File
@@ -0,0 +1,13 @@
const SENSITIVE_KEYS = Object.freeze(['x-amz-credential', 'x-amz-signature', 'Signature', 'AWSAccessKeyId'])
export const scrubUrl = (url: string): string => {
const parsedUrl = new URL(url)
for (const [key, value] of parsedUrl.searchParams) {
if (SENSITIVE_KEYS.includes(key)) {
parsedUrl.searchParams.set(key, 'X'.repeat(value.length))
}
}
return parsedUrl.href
}
packages/server/lib/cloud/artifacts/artifact.ts
+2
View File
@@ -37,6 +37,7 @@ export interface ArtifactUploadResult {
size: number
}
uploadDuration?: number
originalError?: Error
}
export type ArtifactUploadStrategy<T> = (filePath: string, uploadUrl: string, fileSize: number | bigint) => T
@@ -108,6 +109,7 @@ export class Artifact<T extends ArtifactUploadStrategy<UploadResponse>, UploadRe
...this.commonResultFields(),
success: false,
uploadDuration,
originalError: err,
}
}
}
packages/server/lib/cloud/artifacts/protocol_artifact.ts
+20 -4
View File
@@ -1,6 +1,9 @@
import fs from 'fs/promises'
import { existsSync } from 'fs'
import type { ProtocolManager } from '../protocol'
import { IArtifact, ArtifactUploadStrategy, ArtifactUploadResult, Artifact, ArtifactKinds } from './artifact'
import Debug from 'debug'
const debug = Debug('cypress:server:cloud:artifacts:protocol')
interface ProtocolUploadStrategyResult {
success: boolean
@@ -28,10 +31,21 @@ const createProtocolUploadStrategy = (protocolManager: ProtocolManager) => {
return strategy
}
export const createProtocolArtifact = async (filePath: string, uploadUrl: string, protocolManager: ProtocolManager): Promise<IArtifact> => {
const { size } = await fs.stat(filePath)
export const createProtocolArtifact = async (filePath: string, uploadUrl: string, protocolManager: ProtocolManager): Promise<IArtifact | undefined> => {
let size: number | undefined
return new Artifact('protocol', filePath, uploadUrl, size, createProtocolUploadStrategy(protocolManager))
debug('statting file path', filePath)
try {
const stat = await fs.stat(filePath)
debug('file stat', stat)
size = stat.size
} catch (e) {
debug('failed to stat protocol artifact filepath: ', e)
protocolManager.addFatalError('uploadCaptureArtifact', new Error(`File not found: ${filePath}`))
}
return size !== undefined ? new Artifact('protocol', filePath, uploadUrl, size, createProtocolUploadStrategy(protocolManager)) : undefined
}
export const composeProtocolErrorReportFromOptions = async ({
@@ -45,10 +59,12 @@ export const composeProtocolErrorReportFromOptions = async ({
}): Promise<ArtifactUploadResult> => {
const url = captureUploadUrl || protocolCaptureMeta.url
const pathToFile = protocolManager?.getArchivePath()
const fileSize = pathToFile ? (await fs.stat(pathToFile))?.size : 0
const fileSize = pathToFile && existsSync(pathToFile) ? (await fs.stat(pathToFile))?.size : 0
const fatalError = protocolManager?.getFatalError()
debug('fatalError via composeProtocolErrorReport', fatalError)
return {
key: ArtifactKinds.PROTOCOL,
url: url ?? 'UNKNOWN',
packages/server/lib/cloud/artifacts/upload_artifacts.ts
+93 -16
View File
@@ -1,5 +1,7 @@
import _ from 'lodash'
import Debug from 'debug'
import type ProtocolManager from '../protocol'
import { isProtocolInitializationError } from '@packages/types'
import api from '../api'
import { logUploadManifest, logUploadResults, beginUploadActivityOutput } from '../../util/print-run'
import type { UpdateInstanceArtifactsPayload, ArtifactMetadata, ProtocolMetadata } from '../api'
@@ -9,6 +11,8 @@ import { IArtifact, ArtifactUploadResult, ArtifactKinds } from './artifact'
import { createScreenshotArtifactBatch } from './screenshot_artifact'
import { createVideoArtifact } from './video_artifact'
import { createProtocolArtifact, composeProtocolErrorReportFromOptions } from './protocol_artifact'
import { HttpError } from '../api/http_error'
import { NetworkError } from '../api/network_error'
const debug = Debug('cypress:server:cloud:artifacts')
@@ -17,8 +21,10 @@ const toUploadReportPayload = (acc: {
video?: ArtifactMetadata
protocol?: ProtocolMetadata
}, { key, ...report }: ArtifactUploadResult): UpdateInstanceArtifactsPayload => {
const reportWithoutOriginalError = _.omit(report, 'originalError')
if (key === ArtifactKinds.PROTOCOL) {
let { error, errorStack, allErrors } = report
let { error, errorStack, allErrors } = reportWithoutOriginalError
if (allErrors) {
error = `Failed to upload Test Replay after ${allErrors.length} attempts. Errors: ${allErrors.map((error) => error.message).join(', ')}`
@@ -27,12 +33,12 @@ const toUploadReportPayload = (acc: {
error = `Failed to upload Test Replay: ${error}`
}
debug('protocol report %O', report)
debug('protocol report %O', reportWithoutOriginalError)
return {
...acc,
protocol: {
...report,
...reportWithoutOriginalError,
error,
errorStack,
},
@@ -41,7 +47,7 @@ const toUploadReportPayload = (acc: {
return {
...acc,
[key]: (key === 'screenshots') ? [...acc.screenshots, report] : report,
[key]: (key === 'screenshots') ? [...acc.screenshots, reportWithoutOriginalError] : reportWithoutOriginalError,
}
}
@@ -102,9 +108,23 @@ const extractArtifactsFromOptions = async ({
const protocolUploadUrl = captureUploadUrl || protocolCaptureMeta.url
debug('should add protocol artifact? %o, %o, %O', protocolFilePath, protocolUploadUrl, protocolManager)
if (protocolManager && protocolFilePath && protocolUploadUrl) {
artifacts.push(await createProtocolArtifact(protocolFilePath, protocolUploadUrl, protocolManager))
const shouldAddProtocolArtifact = protocolManager && protocolFilePath && protocolUploadUrl && !protocolManager.hasFatalError()
debug('should add protocol artifact? %o', {
protocolFilePath,
protocolUploadUrl,
protocolManager: !!protocolManager,
fatalError: protocolManager?.hasFatalError(),
shouldAddProtocolArtifact,
})
if (shouldAddProtocolArtifact) {
const protocolArtifact = await createProtocolArtifact(protocolFilePath, protocolUploadUrl, protocolManager)
debug(protocolArtifact)
if (protocolArtifact) {
artifacts.push(protocolArtifact)
}
}
} catch (e) {
debug('Error creating protocol artifact: %O', e)
@@ -122,13 +142,49 @@ export const uploadArtifacts = async (options: UploadArtifactOptions) => {
[ArtifactKinds.PROTOCOL]: 2,
}
// Checking protocol fatal errors here, because if there is no fatal error
// but protocol is enabled and there is no archive path, we want to detect
// and establish a fatal error
const preArtifactExtractionFatalError = protocolManager?.getFatalError()
/**
* sometimes, protocolManager initializes both without an archive path and without recording an internal
* fatal error. Test Replay initialization should be refactored in order to capture this state more appropriately.
*/
debug({
archivePath: protocolManager?.getArchivePath(),
protocolManager: !!protocolManager,
preArtifactExtractionFatalError,
protocolCaptureMeta,
})
if (protocolManager && (!protocolManager.getArchivePath() && !preArtifactExtractionFatalError && !protocolCaptureMeta.disabledMessage && protocolCaptureMeta.url)) {
protocolManager.addFatalError('UNKNOWN', new Error('Unable to determine Test Replay archive location'))
}
const artifacts = (await extractArtifactsFromOptions(options)).sort((a, b) => {
return priority[a.reportKey] - priority[b.reportKey]
})
let uploadReport: UpdateInstanceArtifactsPayload
let uploadReport: UpdateInstanceArtifactsPayload = { video: undefined, screenshots: [], protocol: undefined }
const postArtifactExtractionFatalError = protocolManager?.getFatalError()
if (postArtifactExtractionFatalError) {
if (isProtocolInitializationError(postArtifactExtractionFatalError)) {
errors.warning('CLOUD_PROTOCOL_INITIALIZATION_FAILURE', postArtifactExtractionFatalError.error)
} else {
errors.warning('CLOUD_PROTOCOL_CAPTURE_FAILURE', postArtifactExtractionFatalError.error)
}
}
if (!quiet) {
debug('logging upload manifest: %O', {
artifacts,
protocolCaptureMeta,
fatalProtocolError: protocolManager?.getFatalError(),
})
logUploadManifest(artifacts, protocolCaptureMeta, protocolManager?.getFatalError())
}
@@ -151,23 +207,42 @@ export const uploadArtifacts = async (options: UploadArtifactOptions) => {
logUploadResults(uploadResults, protocolManager?.getFatalError())
}
const protocolFatalError = protocolManager?.getFatalError()
const postUploadProtocolFatalError = protocolManager?.getFatalError()
/**
* Protocol instances with fatal errors prior to uploading will not have an uploadResult,
* but we still want to report them to updateInstanceArtifacts
*/
if (postUploadProtocolFatalError && postUploadProtocolFatalError.captureMethod === 'uploadCaptureArtifact') {
const error = postUploadProtocolFatalError.error
if ((error as AggregateError).errors) {
// eslint-disable-next-line no-console
console.log('')
errors.warning('CLOUD_PROTOCOL_UPLOAD_AGGREGATE_ERROR', postUploadProtocolFatalError.error as AggregateError)
} else if (HttpError.isHttpError(error)) {
// eslint-disable-next-line no-console
console.log('')
errors.warning('CLOUD_PROTOCOL_UPLOAD_HTTP_FAILURE', error)
} else if (NetworkError.isNetworkError(error)) {
// eslint-disable-next-line no-console
console.log('')
errors.warning('CLOUD_PROTOCOL_UPLOAD_NEWORK_FAILURE', error)
}
}
// there is no upload results entry for protocol if we did not attempt to upload protocol due to a fatal error
// during initialization or capture. however, we still want to report this failure with the rest of the upload
// results, so we extract what the upload failure report should be from the options passed to this fn
if (!uploadResults.find((result: ArtifactUploadResult) => {
return result.key === ArtifactKinds.PROTOCOL
}) && protocolFatalError) {
}) && postUploadProtocolFatalError) {
debug('composing error report from options')
uploadResults.push(await composeProtocolErrorReportFromOptions(options))
}
uploadReport = uploadResults.reduce(toUploadReportPayload, { video: undefined, screenshots: [], protocol: undefined })
} catch (err) {
debug('primary try/catch failure, ', err.stack)
errors.warning('CLOUD_CANNOT_UPLOAD_ARTIFACTS', err)
return exception.create(err)
await exception.create(err)
}
debug('checking for protocol errors', protocolManager?.hasErrors())
@@ -195,7 +270,9 @@ export const uploadArtifacts = async (options: UploadArtifactOptions) => {
stack: err.stack,
})
errors.warning('CLOUD_CANNOT_UPLOAD_ARTIFACTS_PROTOCOL', err)
// eslint-disable-next-line no-console
console.log('')
errors.warning('CLOUD_CANNOT_CONFIRM_ARTIFACTS', err)
if (err.statusCode !== 503) {
return exception.create(err)
packages/server/lib/cloud/protocol.ts
+1 -1
View File
@@ -235,7 +235,7 @@ export class ProtocolManager implements ProtocolManagerShape {
return !!this._errors.length
}
addFatalError (captureMethod: ProtocolCaptureMethod, error: Error, args: any) {
addFatalError (captureMethod: ProtocolCaptureMethod, error: Error, args?: any) {
this._errors.push({
fatal: true,
error,
packages/server/lib/cloud/upload/upload_stream.ts
+22 -10
View File
@@ -4,6 +4,7 @@ import type { ReadStream } from 'fs'
import type { StreamActivityMonitor } from './stream_activity_monitor'
import Debug from 'debug'
import { HttpError } from '../api/http_error'
import { NetworkError } from '../api/network_error'
import { agent } from '@packages/network'
const debug = Debug('cypress:server:cloud:uploadStream')
@@ -67,17 +68,16 @@ export const uploadStream = async (fileStream: ReadStream, destinationUrl: strin
debugVerbose('PUT %s Response: %O', destinationUrl, response)
debugVerbose('PUT %s Error: %O', destinationUrl, error)
// Record all HTTP errors encountered
if (response?.status && response?.status >= 400) {
errorPromises.push(HttpError.fromResponse(response))
}
const isHttpError = response?.status && response?.status >= 400
const isNetworkError = error && !timeoutMonitor?.getController().signal.reason
// Record network errors
if (error) {
errorPromises.push(Promise.resolve(error))
if (isHttpError) {
errorPromises.push(HttpError.fromResponse(response))
} else if (isNetworkError) {
errorPromises.push(Promise.resolve(new NetworkError(error, destinationUrl)))
}
const isUnderRetryLimit = attempt < retries
const isNetworkError = !!error
const isRetryableHttpError = (!!response?.status && RETRYABLE_STATUS_CODES.includes(response.status))
debug('checking if should retry: %s %O', destinationUrl, {
@@ -129,10 +129,22 @@ export const uploadStream = async (fileStream: ReadStream, destinationUrl: strin
resolve()
}
} catch (e) {
const error = abortController?.signal.reason ?? e
debug('error on upload:', e)
const signalError = abortController?.signal.reason
debug('PUT %s: %s', destinationUrl, error)
reject(error)
const errors = await Promise.all(errorPromises)
debug('errors on upload:')
errors.forEach((e) => debug(e))
if (signalError && !errors.includes(signalError)) {
errors.push(signalError)
}
if (errors.length > 1) {
reject(new AggregateError(errors, `${errors.length} errors encountered during upload`))
} else {
reject(errors[0])
}
}
})
}
packages/server/lib/util/print-run.ts
+15 -21
View File
@@ -19,6 +19,8 @@ import type { Table } from 'cli-table3'
import type { CypressRunResult } from '../modes/results'
import type { IArtifact, ArtifactUploadResult } from '../cloud/artifacts/artifact'
import { ArtifactKinds } from '../cloud/artifacts/artifact'
type Screenshot = {
width: number
height: number
@@ -601,9 +603,9 @@ export const logUploadManifest = (artifacts: IArtifact[], protocolCaptureMeta: {
disabledMessage?: string
}, protocolFatalError?: ProtocolError) => {
const labels = {
'video': 'Video',
'screenshots': 'Screenshot',
'protocol': 'Test Replay',
[ArtifactKinds.VIDEO]: 'Video',
[ArtifactKinds.SCREENSHOTS]: 'Screenshot',
[ArtifactKinds.PROTOCOL]: 'Test Replay',
}
// eslint-disable-next-line no-console
@@ -615,9 +617,9 @@ export const logUploadManifest = (artifacts: IArtifact[], protocolCaptureMeta: {
// eslint-disable-next-line no-console
console.log('')
const video = artifacts.find(({ reportKey }) => reportKey === 'video')
const screenshots = artifacts.filter(({ reportKey }) => reportKey === 'screenshots')
const protocol = artifacts.find(({ reportKey }) => reportKey === 'protocol')
const video = artifacts.find(({ reportKey }) => reportKey === ArtifactKinds.VIDEO)
const screenshots = artifacts.filter(({ reportKey }) => reportKey === ArtifactKinds.SCREENSHOTS)
const protocol = artifacts.find(({ reportKey }) => reportKey === ArtifactKinds.PROTOCOL)
if (video) {
printPendingArtifactUpload(video, labels)
@@ -674,22 +676,14 @@ export const printCompletedArtifactUpload = ({ pathToFile, key, fileSize, succes
}
export const logUploadResults = (results: ArtifactUploadResult[], protocolFatalError: ProtocolError | undefined) => {
const labels = {
'video': 'Video',
'screenshots': 'Screenshot',
'protocol': 'Test Replay',
if (!results.length) {
return
}
// if protocol did not attempt an upload due to a fatal error, there will still be an upload result - this is
// so we can report the failure properly to instance/artifacts. But, we do not want to display it here.
const trimmedResults = protocolFatalError && protocolFatalError.captureMethod !== 'uploadCaptureArtifact' ?
results.filter(((result) => {
return result.key !== 'protocol'
})) :
results
if (!trimmedResults.length) {
return
const labels = {
[ArtifactKinds.VIDEO]: 'Video',
[ArtifactKinds.SCREENSHOTS]: 'Screenshot',
[ArtifactKinds.PROTOCOL]: 'Test Replay',
}
// eslint-disable-next-line no-console
@@ -702,7 +696,7 @@ export const logUploadResults = (results: ArtifactUploadResult[], protocolFatalE
// eslint-disable-next-line no-console
console.log('')
trimmedResults.forEach(({ key, ...report }, i, { length }) => {
results.forEach(({ key, ...report }, i, { length }) => {
printCompletedArtifactUpload({ key, ...report }, labels, chalk.grey(`${i + 1}/${length}`))
})
}
packages/types/src/protocol.ts
+5 -1
View File
@@ -41,7 +41,7 @@ export interface AppCaptureProtocolInterface extends AppCaptureProtocolCommon {
beforeSpec ({ workingDirectory, archivePath, dbPath, db }: { workingDirectory: string, archivePath: string, dbPath: string, db: Database }): void
}
export type ProtocolCaptureMethod = keyof AppCaptureProtocolInterface | 'setupProtocol' | 'uploadCaptureArtifact' | 'getCaptureProtocolScript' | 'cdpClient.on' | 'getZippedDb'
export type ProtocolCaptureMethod = keyof AppCaptureProtocolInterface | 'setupProtocol' | 'uploadCaptureArtifact' | 'getCaptureProtocolScript' | 'cdpClient.on' | 'getZippedDb' | 'UNKNOWN' | 'createProtocolArtifact'
export interface ProtocolError {
args?: any
@@ -52,6 +52,10 @@ export interface ProtocolError {
isUploadError?: boolean
}
export const isProtocolInitializationError = (error: ProtocolError) => {
return ['setupProtocol', 'beforeSpec', 'getCaptureProtocolScript'].includes(error.captureMethod)
}
type ProtocolErrorReportEntry = Omit<ProtocolError, 'fatal' | 'error'> & {
message: string
name: string
system-tests/__snapshots__/record_spec.js
+340 -1
View File
@@ -3119,6 +3119,13 @@ exports['e2e record capture-protocol enabled protocol runtime errors error initi
- /XXX/XXX/XXX/cypress/screenshots/record_pass.cy.js/yay it passes.png (400x1022)
Warning: We encountered an error while initializing the Test Replay recording for this spec.
These results will not display Test Replay recordings.
This error will not affect or change the exit code.
Error: Error instantiating Protocol Capture
(Uploading Cloud Artifacts)
@@ -3202,6 +3209,13 @@ exports['e2e record capture-protocol enabled protocol runtime errors error in pr
- /XXX/XXX/XXX/cypress/screenshots/record_pass.cy.js/yay it passes.png (400x1022)
Warning: We encountered an error while initializing the Test Replay recording for this spec.
These results will not display Test Replay recordings.
This error will not affect or change the exit code.
Error: Error in beforeSpec
(Uploading Cloud Artifacts)
@@ -3285,6 +3299,18 @@ exports['e2e record capture-protocol enabled protocol runtime errors error in pr
- /XXX/XXX/XXX/cypress/screenshots/record_pass.cy.js/yay it passes.png (400x1022)
Warning: We encountered an error while recording Test Replay data for this spec.
These results will not display Test Replay recordings.
This can happen for many reasons. If this problem persists:
- Try increasing the available disk space.
- Ensure that /os/tmpdir/cypress/protocol is both readable and writable.
This error will not affect or change the exit code.
Error: error in beforeTest
(Uploading Cloud Artifacts)
@@ -3458,6 +3484,13 @@ We will retry 1 more time in X second(s)...
- /XXX/XXX/XXX/cypress/screenshots/record_pass.cy.js/yay it passes.png (400x1022)
Warning: We encountered an error while initializing the Test Replay recording for this spec.
These results will not display Test Replay recordings.
This error will not affect or change the exit code.
Error: Error downloading capture code: 500 - "500 - Internal Server Error"
(Uploading Cloud Artifacts)
@@ -3553,7 +3586,17 @@ exports['capture-protocol api errors error report 500 continues 1'] = `
(Uploaded Cloud Artifacts)
- Screenshot - Done Uploading 1 kB in Xm, Ys ZZ.ZZms 1/2 /XXX/XXX/XXX/cypress/screenshots/record_pass.cy.js/yay it passes.png
- Test Replay - Done Uploading 1 kB in Xm, Ys ZZ.ZZms 2/2
- Test Replay - Failed Uploading after Xm, Ys ZZ.ZZms 2/2 - request to http://fake.test/url failed, reason: getaddrinfo ENOTFOUND fake.test
Warning: We encountered multiple errors while uploading the Test Replay recording for this spec.
We attempted to upload the Test Replay recording 3 times.
Some or all of the errors encountered are system-level network errors. Please verify your network configuration for connecting to http://fake.test/url
- request to http://fake.test/url failed, reason: getaddrinfo ENOTFOUND fake.test
- request to http://fake.test/url failed, reason: getaddrinfo ENOTFOUND fake.test
- request to http://fake.test/url failed, reason: getaddrinfo ENOTFOUND fake.test
====================================================================================================
@@ -3723,6 +3766,14 @@ exports['capture-protocol api errors upload 500 - does not retry continues 1'] =
- Screenshot - Done Uploading 1 kB in Xm, Ys ZZ.ZZms 1/2 /XXX/XXX/XXX/cypress/screenshots/record_pass.cy.js/yay it passes.png
- Test Replay - Failed Uploading after Xm, Ys ZZ.ZZms 2/2 - 500 Internal Server Error (http://localhost:1234/capture-protocol/upload/?x-amz-credential=XXXXXXXX&x-amz-signature=XXXXXXXXXXXXX)
Warning: We encountered an HTTP error while uploading the Test Replay recording for this spec.
These results will not display Test Replay recordings.
This error will not affect or change the exit code.
http://localhost:1234/capture-protocol/upload/?x-amz-credential=XXXXXXXX&x-amz-signature=XXXXXXXXXXXXX responded with HTTP 500: Internal Server Error
====================================================================================================
(Run Finished)
@@ -3891,6 +3942,294 @@ exports['capture-protocol api errors upload 503 - tries 3 times and fails contin
- Screenshot - Done Uploading 1 kB in Xm, Ys ZZ.ZZms 1/2 /XXX/XXX/XXX/cypress/screenshots/record_pass.cy.js/yay it passes.png
- Test Replay - Failed Uploading after Xm, Ys ZZ.ZZms 2/2 - 503 Service Unavailable (http://localhost:1234/capture-protocol/upload/?x-amz-credential=XXXXXXXX&x-amz-signature=XXXXXXXXXXXXX)
Warning: We encountered multiple errors while uploading the Test Replay recording for this spec.
We attempted to upload the Test Replay recording 3 times.
- 503 Service Unavailable (http://localhost:1234/capture-protocol/upload/?x-amz-credential=XXXXXXXX&x-amz-signature=XXXXXXXXXXXXX)
- 503 Service Unavailable (http://localhost:1234/capture-protocol/upload/?x-amz-credential=XXXXXXXX&x-amz-signature=XXXXXXXXXXXXX)
- 503 Service Unavailable (http://localhost:1234/capture-protocol/upload/?x-amz-credential=XXXXXXXX&x-amz-signature=XXXXXXXXXXXXX)
====================================================================================================
(Run Finished)
Spec Tests Passing Failing Pending Skipped
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ ✔ record_pass.cy.js XX:XX 2 1 - 1 - │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
✔ All specs passed! XX:XX 2 1 - 1 -
───────────────────────────────────────────────────────────────────────────────────────────────────────
Recorded Run: https://dashboard.cypress.io/projects/cjvoj7/runs/12
`
exports['e2e record capture-protocol enabled protocol runtime errors db is unreadable displays warning and continues 1'] = `
====================================================================================================
(Run Starting)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Cypress: 1.2.3 │
│ Browser: FooBrowser 88 │
│ Specs: 1 found (record_pass.cy.js) │
│ Searched: cypress/e2e/record_pass* │
│ Params: Tag: false, Group: false, Parallel: false │
│ Run URL: https://dashboard.cypress.io/projects/cjvoj7/runs/12 │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
────────────────────────────────────────────────────────────────────────────────────────────────────
Running: record_pass.cy.js (1 of 1)
Estimated: X second(s)
record pass
✓ passes
- is pending
1 passing
1 pending
(Results)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Tests: 2 │
│ Passing: 1 │
│ Failing: 0 │
│ Pending: 1 │
│ Skipped: 0 │
│ Screenshots: 1 │
│ Video: false │
│ Duration: X seconds │
│ Estimated: X second(s) │
│ Spec Ran: record_pass.cy.js │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
(Screenshots)
- /XXX/XXX/XXX/cypress/screenshots/record_pass.cy.js/yay it passes.png (400x1022)
Warning: We encountered an error while recording Test Replay data for this spec.
These results will not display Test Replay recordings.
This can happen for many reasons. If this problem persists:
- Try increasing the available disk space.
- Ensure that /os/tmpdir/cypress/protocol is both readable and writable.
This error will not affect or change the exit code.
Error: File not found: /os/tmpdir/cypress/protocol/e9e81b5e-cc58-4026-b2ff-8ae3161435a6.tar
(Uploading Cloud Artifacts)
- Video - Nothing to upload
- Screenshot - 1 kB /XXX/XXX/XXX/cypress/screenshots/record_pass.cy.js/yay it passes.png
- Test Replay - Failed Capturing - File not found: /os/tmpdir/cypress/protocol/e9e81b5e-cc58-4026-b2ff-8ae3161435a6.tar
Uploading Cloud Artifacts: . . . . .
(Uploaded Cloud Artifacts)
- Screenshot - Done Uploading 1 kB in Xm, Ys ZZ.ZZms 1/1 /XXX/XXX/XXX/cypress/screenshots/record_pass.cy.js/yay it passes.png
====================================================================================================
(Run Finished)
Spec Tests Passing Failing Pending Skipped
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ ✔ record_pass.cy.js XX:XX 2 1 - 1 - │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
✔ All specs passed! XX:XX 2 1 - 1 -
───────────────────────────────────────────────────────────────────────────────────────────────────────
Recorded Run: https://dashboard.cypress.io/projects/cjvoj7/runs/12
`
exports['capture-protocol api errors upload network error retries 3 times, warns and continues 1'] = `
====================================================================================================
(Run Starting)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Cypress: 1.2.3 │
│ Browser: FooBrowser 88 │
│ Specs: 1 found (record_pass.cy.js) │
│ Searched: cypress/e2e/record_pass* │
│ Params: Tag: false, Group: false, Parallel: false │
│ Run URL: https://dashboard.cypress.io/projects/cjvoj7/runs/12 │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
────────────────────────────────────────────────────────────────────────────────────────────────────
Running: record_pass.cy.js (1 of 1)
Estimated: X second(s)
record pass
✓ passes
- is pending
1 passing
1 pending
(Results)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Tests: 2 │
│ Passing: 1 │
│ Failing: 0 │
│ Pending: 1 │
│ Skipped: 0 │
│ Screenshots: 1 │
│ Video: false │
│ Duration: X seconds │
│ Estimated: X second(s) │
│ Spec Ran: record_pass.cy.js │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
(Screenshots)
- /XXX/XXX/XXX/cypress/screenshots/record_pass.cy.js/yay it passes.png (400x1022)
(Uploading Cloud Artifacts)
- Video - Nothing to upload
- Screenshot - 1 kB /XXX/XXX/XXX/cypress/screenshots/record_pass.cy.js/yay it passes.png
- Test Replay
Uploading Cloud Artifacts: . . . . .
(Uploaded Cloud Artifacts)
- Screenshot - Done Uploading 1 kB in Xm, Ys ZZ.ZZms 1/2 /XXX/XXX/XXX/cypress/screenshots/record_pass.cy.js/yay it passes.png
- Test Replay - Failed Uploading after Xm, Ys ZZ.ZZms 2/2 - request to http://fake.test/url failed, reason: getaddrinfo ENOTFOUND fake.test
Warning: We encountered multiple errors while uploading the Test Replay recording for this spec.
We attempted to upload the Test Replay recording 3 times.
Some or all of the errors encountered are system-level network errors. Please verify your network configuration for connecting to http://fake.test/url
- request to http://fake.test/url failed, reason: getaddrinfo ENOTFOUND fake.test
- request to http://fake.test/url failed, reason: getaddrinfo ENOTFOUND fake.test
- request to http://fake.test/url failed, reason: getaddrinfo ENOTFOUND fake.test
====================================================================================================
(Run Finished)
Spec Tests Passing Failing Pending Skipped
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ ✔ record_pass.cy.js XX:XX 2 1 - 1 - │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
✔ All specs passed! XX:XX 2 1 - 1 -
───────────────────────────────────────────────────────────────────────────────────────────────────────
Recorded Run: https://dashboard.cypress.io/projects/cjvoj7/runs/12
`
exports['e2e record api interaction errors update instance artifacts warns but proceeds 1'] = `
====================================================================================================
(Run Starting)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Cypress: 1.2.3 │
│ Browser: FooBrowser 88 │
│ Specs: 1 found (record_pass.cy.js) │
│ Searched: cypress/e2e/record_pass* │
│ Params: Tag: false, Group: false, Parallel: false │
│ Run URL: https://dashboard.cypress.io/projects/cjvoj7/runs/12 │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
────────────────────────────────────────────────────────────────────────────────────────────────────
Running: record_pass.cy.js (1 of 1)
Estimated: X second(s)
record pass
✓ passes
- is pending
1 passing
1 pending
(Results)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Tests: 2 │
│ Passing: 1 │
│ Failing: 0 │
│ Pending: 1 │
│ Skipped: 0 │
│ Screenshots: 1 │
│ Video: false │
│ Duration: X seconds │
│ Estimated: X second(s) │
│ Spec Ran: record_pass.cy.js │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
(Screenshots)
- /XXX/XXX/XXX/cypress/screenshots/record_pass.cy.js/yay it passes.png (400x1022)
(Uploading Cloud Artifacts)
- Video - Nothing to upload
- Screenshot - 1 kB /XXX/XXX/XXX/cypress/screenshots/record_pass.cy.js/yay it passes.png
- Test Replay - Nothing to upload - Test Replay is disabled for this project. Enable Test Replay in Cloud project settings
Uploading Cloud Artifacts: . . . . .
(Uploaded Cloud Artifacts)
- Screenshot - Done Uploading 1 kB in Xm, Ys ZZ.ZZms 1/1 /XXX/XXX/XXX/cypress/screenshots/record_pass.cy.js/yay it passes.png
Warning: We encountered an error while confirming the upload of artifacts for this spec.
These results will not display artifacts.
This error will not affect or change the exit code.
StatusCodeError: 500 - "Internal Server Error"
====================================================================================================
(Run Finished)
system-tests/lib/normalizeStdout.ts
+7
View File
@@ -1,5 +1,6 @@
import Fixtures from './fixtures'
import _ from 'lodash'
import os from 'os'
export const e2ePath = Fixtures.projectPath('e2e')
@@ -98,6 +99,10 @@ export const replaceStackTraceLines = (str: string, browserName: 'electron' | 'f
return str.replace(stackTraceRegex, (match: string, ...parts: string[]) => {
let post = parts[0]
console.log('POST:')
console.log(`"${post}"`)
console.log('/POST')
if (browserName === 'firefox') {
post = post.replace(whiteSpaceBetweenNewlines, '\n')
}
@@ -115,6 +120,8 @@ export const normalizeStdout = function (str: string, options: any = {}) {
// /Users/jane/........../ -> //foo/bar/.projects/
// (Required when paths are printed outside of our own formatting)
.split(pathUpToProjectName).join('/foo/bar/.projects')
// temp dir may change from run to run, normalize it to a fake dir
.split(os.tmpdir()).join('/os/tmpdir')
// unless normalization is explicitly turned off then
// always normalize the stdout replacing the browser text
system-tests/lib/protocol-stubs/protocolStubResponse.ts
+2
View File
@@ -52,3 +52,5 @@ export const PROTOCOL_STUB_BEFORETEST_ERROR = stub('protocolStubWithBeforeTestEr
export const PROTOCOL_STUB_FONT_FLOODING = stub('protocolStubFontFlooding.ts')
export const PROTOCOL_STUB_SERVICE_WORKER = stub('protocolStubServiceWorker.ts')
export const PROTOCOL_STUB_NO_DB_WRITE = stub('protocolStubWithMissingArchive.ts')
system-tests/lib/protocol-stubs/protocolStubWithMissingArchive.ts
+183
View File
@@ -0,0 +1,183 @@
import path from 'path'
import fs from 'fs-extra'
import type { AppCaptureProtocolInterface, ResponseEndedWithEmptyBodyOptions, ResponseStreamOptions, ResponseStreamTimedOutOptions } from '@packages/types'
import type { Readable } from 'stream'
const getFilePath = (filename) => {
return path.join(
path.resolve(__dirname),
'cypress',
'system-tests-protocol-dbs',
`${filename}.json`,
)
}
export class AppCaptureProtocol implements AppCaptureProtocolInterface {
private filename: string
private events = {
beforeSpec: [],
afterSpec: [],
beforeTest: [],
preAfterTest: [],
afterTest: [],
addRunnables: [],
connectToBrowser: [],
commandLogAdded: [],
commandLogChanged: [],
viewportChanged: [],
urlChanged: [],
pageLoading: [],
resetTest: [],
responseEndedWithEmptyBody: [],
responseStreamTimedOut: [],
}
private cdpClient: any
private scriptToEvaluateId: any
private archivePath: string | undefined
getDbMetadata (): { offset: number, size: number } {
return {
offset: 0,
size: 0,
}
}
cdpReconnect (): Promise<void> {
return Promise.resolve()
}
responseStreamReceived (options: ResponseStreamOptions): Readable {
return options.responseStream
}
resetEvents () {
this.events.beforeTest = []
this.events.preAfterTest = []
this.events.afterTest = []
this.events.commandLogAdded = []
this.events.commandLogChanged = []
this.events.viewportChanged = []
this.events.urlChanged = []
this.events.pageLoading = []
this.events.responseEndedWithEmptyBody = []
this.events.responseStreamTimedOut = []
}
connectToBrowser = async (cdpClient) => {
if (cdpClient) {
this.events.connectToBrowser.push(true)
this.cdpClient = cdpClient
}
const scriptToEvaluateResult = await this.cdpClient.send(
'Page.addScriptToEvaluateOnNewDocument',
{
source: `(function () {})()`,
},
)
this.scriptToEvaluateId = scriptToEvaluateResult.identifier
}
addRunnables = (runnables) => {
this.events.addRunnables.push(runnables)
return Promise.resolve()
}
beforeSpec = ({ archivePath, db }) => {
this.archivePath = archivePath
this.events.beforeSpec.push(db)
this.filename = getFilePath(path.basename(db.name))
if (!fs.existsSync(archivePath)) {
// If a dummy file hasn't been created by the test, write a tar file so that it can be fake uploaded
fs.writeFileSync(archivePath, '')
}
}
async afterSpec (): Promise<void> {
this.events.afterSpec.push(true)
// since the order of the logs can vary per run, we sort them by id to ensure the snapshot can be compared
this.events.commandLogChanged.sort((log1, log2) => {
return log1.id.localeCompare(log2.id)
})
try {
fs.outputFileSync(this.filename, JSON.stringify(this.events, null, 2))
} catch (e) {
console.log('error writing protocol events', e)
}
await this.cdpClient.send('Page.removeScriptToEvaluateOnNewDocument', {
identifier: this.scriptToEvaluateId || '',
})
.catch(() => {
})
if (fs.existsSync(this.archivePath)) {
fs.rmSync(this.archivePath)
}
}
beforeTest = (test) => {
this.events.beforeTest.push(test)
return Promise.resolve()
}
commandLogAdded = (log) => {
// we only care about logs that occur during a test
if (log.testId) {
this.events.commandLogAdded.push(log)
}
}
commandLogChanged = (log) => {
// we only care about logs that occur during a test and
// since the number of log changes can vary per run, we only want to record
// the passed/failed ones to ensure the snapshot can be compared
if (log.testId && (log.state === 'passed' || log.state === 'failed')) {
this.events.commandLogChanged.push(log)
}
}
viewportChanged = (input) => {
this.events.viewportChanged.push(input)
}
urlChanged = (input) => {
this.events.urlChanged.push(input)
}
pageLoading = (input) => {
this.events.pageLoading.push(input)
}
preAfterTest = (test, options) => {
this.events.preAfterTest.push({ test, options })
return Promise.resolve()
}
afterTest = (test) => {
this.events.afterTest.push(test)
return Promise.resolve()
}
responseEndedWithEmptyBody = (options: ResponseEndedWithEmptyBodyOptions) => {
this.events.responseEndedWithEmptyBody.push(options)
}
responseStreamTimedOut (options: ResponseStreamTimedOutOptions): void {
this.events.responseStreamTimedOut.push(options)
}
resetTest (testId: string): void {
this.resetEvents()
this.events.resetTest.push(testId)
}
}
system-tests/lib/protocol-stubs/protocolStubWithRuntimeError.ts
+3
View File
@@ -5,6 +5,9 @@ export class AppCaptureProtocol implements AppCaptureProtocolInterface {
constructor () {
throw new Error('Error instantiating Protocol Capture')
}
cdpReconnect (): Promise<void> {
return Promise.resolve()
}
preAfterTest (test: Record<string, any>, options: Record<string, any>): Promise<void> {
return Promise.resolve()
system-tests/lib/serverStub.ts
+16 -1
View File
@@ -14,6 +14,7 @@ import systemTests from './system-tests'
let CAPTURE_PROTOCOL_ENABLED = false
let CAPTURE_PROTOCOL_MESSAGE: string | undefined
let CAPTURE_PROTOCOL_UPLOAD_ENABLED = true
import {
TEST_PRIVATE,
@@ -94,7 +95,11 @@ const sendUploadUrls = function (req, res) {
json.screenshotUploadUrls = screenshotUploadUrls
if (CAPTURE_PROTOCOL_ENABLED) {
json.captureUploadUrl = `http://localhost:1234${CAPTURE_PROTOCOL_UPLOAD_URL}`
if (CAPTURE_PROTOCOL_UPLOAD_ENABLED) {
json.captureUploadUrl = `http://localhost:1234${CAPTURE_PROTOCOL_UPLOAD_URL}`
} else {
json.captureUploadUrl = `http://fake.test/url`
}
}
return res.json(json)
@@ -455,6 +460,16 @@ export const disableCaptureProtocolWithMessage = (message: string) => {
})
}
export const disableCaptureProtocolUploadUrl = () => {
beforeEach(() => {
CAPTURE_PROTOCOL_UPLOAD_ENABLED = false
})
afterEach(() => {
CAPTURE_PROTOCOL_ENABLED = true
})
}
export const setupStubbedServer = (routes) => {
systemTests.setup({
servers: [{
system-tests/test/record_spec.js
+101 -1
View File
@@ -23,13 +23,14 @@ const {
postInstanceTestsResponse,
encryptBody,
disableCaptureProtocolWithMessage,
disableCaptureProtocolUploadUrl,
CAPTURE_PROTOCOL_UPLOAD_URL,
postRunResponseWithProtocolDisabled,
routeHandlers,
} = require('../lib/serverStub')
const { expectRunsToHaveCorrectTimings } = require('../lib/resultsUtils')
const { randomBytes } = require('crypto')
const { PROTOCOL_STUB_CONSTRUCTOR_ERROR, PROTOCOL_STUB_NONFATAL_ERROR, PROTOCOL_STUB_BEFORESPEC_ERROR, PROTOCOL_STUB_BEFORETEST_ERROR } = require('../lib/protocol-stubs/protocolStubResponse')
const { PROTOCOL_STUB_CONSTRUCTOR_ERROR, PROTOCOL_STUB_NONFATAL_ERROR, PROTOCOL_STUB_BEFORESPEC_ERROR, PROTOCOL_STUB_BEFORETEST_ERROR, PROTOCOL_STUB_NO_DB_WRITE } = require('../lib/protocol-stubs/protocolStubResponse')
const debug = require('debug')('cypress:system-tests:record_spec')
const e2ePath = Fixtures.projectPath('e2e')
const outputPath = path.join(e2ePath, 'output.json')
@@ -1528,6 +1529,44 @@ describe('e2e record', () => {
})
})
describe('update instance artifacts', () => {
const routes = createRoutes({
putArtifacts: {
res (_, res) {
return res.sendStatus(500)
},
},
})
setupStubbedServer(routes)
it('warns but proceeds', function () {
process.env.DISABLE_API_RETRIES = 'true'
return systemTests.exec(this, {
key: 'f858a2bc-b469-4e48-be67-0876339ee7e1',
configFile: 'cypress-with-project-id.config.js',
spec: 'record_pass*',
record: true,
snapshot: true,
})
.then(() => {
const urls = getRequestUrls()
expect(urls).to.deep.eq([
'POST /runs',
`POST /runs/${runId}/instances`,
`POST /instances/${instanceId}/tests`,
`POST /instances/${instanceId}/results`,
'PUT /screenshots/1.png',
`PUT /instances/${instanceId}/artifacts`,
`PUT /instances/${instanceId}/stdout`,
`POST /runs/${runId}/instances`,
])
})
})
})
describe('api retries on error', () => {
let count = 0
@@ -2401,6 +2440,37 @@ describe('e2e record', () => {
})
})
describe('db is unreadable', () => {
enableCaptureProtocol(PROTOCOL_STUB_NO_DB_WRITE)
beforeEach(() => {
if (fs.existsSync(archiveFile)) {
return fsPromise.rm(archiveFile)
}
})
it('displays warning and continues', function () {
return systemTests.exec(this, {
key: 'f858a2bc-b469-4e48-be67-0876339ee7e1',
configFile: 'cypress-with-project-id.config.js',
spec: 'record_pass*',
record: true,
snapshot: true,
}).then(() => {
const urls = getRequestUrls()
expect(urls).to.include.members([`PUT /instances/${instanceId}/artifacts`])
expect(urls).not.to.include.members([`PUT ${CAPTURE_PROTOCOL_UPLOAD_URL}`])
const artifactReport = getRequests().find(({ url }) => url === `PUT /instances/${instanceId}/artifacts`)?.body
expect(artifactReport?.protocol).to.exist()
expect(artifactReport?.protocol?.error).to.exist().and.not.to.be.empty()
expect(artifactReport?.protocol?.errorStack).to.exist().and.not.to.be.empty()
expect(artifactReport?.protocol?.url).to.exist().and.not.be.empty()
})
})
})
describe('error initializing protocol', () => {
enableCaptureProtocol(PROTOCOL_STUB_CONSTRUCTOR_ERROR)
@@ -2623,6 +2693,36 @@ describe('capture-protocol api errors', () => {
})
})
describe('upload network error', () => {
disableCaptureProtocolUploadUrl()
setupStubbedServer(createRoutes())
it('retries 3 times, warns and continues', function () {
process.env.API_RETRY_INTERVALS = '1000'
return systemTests.exec(this, {
key: 'f858a2bc-b469-4e48-be67-0876339ee7e1',
configFile: 'cypress-with-project-id.config.js',
spec: 'record_pass*',
record: true,
snapshot: true,
}).then(() => {
const urls = getRequestUrls()
expect(urls).to.include.members([`PUT /instances/${instanceId}/artifacts`])
const artifactReport = getRequests().find(({ url }) => url === `PUT /instances/${instanceId}/artifacts`)?.body
expect(artifactReport?.protocol).to.exist()
expect(artifactReport?.protocol?.error).to.equal(
'Failed to upload Test Replay after 3 attempts. Errors: request to http://fake.test/url failed, reason: getaddrinfo ENOTFOUND fake.test, request to http://fake.test/url failed, reason: getaddrinfo ENOTFOUND fake.test, request to http://fake.test/url failed, reason: getaddrinfo ENOTFOUND fake.test',
)
expect(artifactReport?.protocol?.errorStack).to.exist().and.not.to.be.empty()
})
})
})
describe('fetch script 500', () => {
stubbedServerWithErrorOn('getCaptureScript')
it('continues', function () {