Files
cypress/packages/errors/test/support/error-comparison-tool.ts
Brian Mann 29841f32b9 feat: redesign server errors (#20072)
* chore: rename errors.js -> errors.ts

* refactor: type safety on errors

* refactor: add err_template for consistent error formatting

* fix a few system tests

* fix tests; update snapshots

* Fix types

* normalize snapshot - remove chalk ansi colors

* more unit test fixes

* more system test fixes

* circleci build

* backtick always in stdout, fix error formatting and failing snapshots

* refactor: create @packages/errors

* fix import

* fix import

* fixing build / tests

* remove extraneous file

* move warnIfExplicitCiBuildId

* fix build / tests

* Fix

* error, type fixes, documentation, standardize child process error serialization

* fix import

* build errors on install

* wrote specs generating visual images of all errors

* remove unused dep

* sanitize stack traces

* add image diffing

- if base images don't exist, create them
- if base images don't match and local, overwrite them, if in CI throw
- if base images are stale and local, delete them, if in CI throw

* remove Courier New + MesloLGS NF font

* type fixes, remove Bluebird, general cleanup

* TS Cleanup

* skip typecheck on tests for now

* yarn.lock

* fix @types/chai version

* fix yarn.lock

* Different version of mocha types so it isnt patched

* errors spec snapshot

* CI fix

* fixes

* store snapshot images in circle artifacts

* dont change artifact destination prefix

* use Courier Prime

* antialias the text

* decrease pixelmatch threshold, fail in CI only when changed pixels > 100

* increase timeout

* overflow: hidden, remove new Promise, add debug logging

Co-Authored-By: Tim Griesser <tgriesser@gmail.com>

* run unit tests w/ concurrency=1

* unique window per file

* disable app hardware acceleration + use in process gpu + single process

* do not do image diffing

- conditionally convert html to images
- store html snapshots
- do not store images in git

* store snapshot html

* Merge branch 'tgriesser/chore/refactor-errors' of https://github.com/cypress-io/cypress into tgriesser/chore/refactor-errors

* remove concurrency

* fix assertion

* fixing ci

* Link in readme

* pass the browsers to listItems

* fix: build @packages/errors in CI, defer import to prevent errors locally

* Merge branch 'develop' into tgriesser/chore/refactor-errors

* develop:
  chore: fix cypress npm package artifact upload path (#20023)
  chore(driver): move cy.within logic into it's own file (#20036)
  chore: update automerge workflows (#19982)
  fix(selectFile): use target window's File/DataTransfer classes (#20003)
  chore: Update Chrome (stable) to 98.0.4758.80 and Chrome (beta) to 98.0.4758.80 (#19995)
  fix: Adjust ffmpeg CLI args for performance (#19983)
  build: allow unified to run cypress on Apple Silicon (arm64) (backport #19067 to 9.x) (#19968)

* fix run-if-ci.sh

* remove dead code

* Mark the .html files as generated in gitattributes

* fix running single error case, slice out more of the brittle stack

* remove additional brittle stack line

* firefox web security error

* nest inside of describe

* reformat and redesign errors

* more error cleanup and standardization

* additional formatting of errors, code cleanup, refactoring

* update ansi colors to match terminal colors

* cleanup remaining loose ends, update several errors, compact excess formatters

* fix types

* additional formatting, remove TODO's, ensure no [object Object] in output

* add test for 412 server response on invalid schema

* update unknown dashboard error on creating run

* use fs.access instead of fs.stat for perf

* added PLUGINS_FILE_NOT_FOUND error

- separated out from PLUGINS_FILE_ERROR
- add system tests for both cases
- update snapshots
- remove stack trace from PLUGINS_FILE_NOT_FOUND fka PLUGINS_FILE_ERROR

* add plugins system test around plugins synchronously throwing on require

* remove forcing process.cwd() to be packages/server, update affected code

- this was a long needed hangover from very old code that was doing unnecessary things due to respawning electron from node and handling various entrypoints into booting cypress
- this also makes the root yarn dev and dev-debug work correctly because options.cwd is set correctly now

* perf: lazy load chalk

* remove excessive line since the file exists but is invalid

* fix types

* add system test when plugins function throws a synchronous error

* create new PLUGINS_INVALID_EVENT_ERROR, wire it up correctly for error template use

- properly pass error instance from child to ensure proper user stack frames
- move error display code into the right place

* only show a single stack trace, either originalError or internal cypressError

* push error html snapshots

* fix tests, types

* fix test

* fix failing tests

* fix tests

* fixes lots of broken tests

* more test fixes

* fixes more tests

* fix type checking

* wip: consistent handling of interpolated values

* feat: fixing up errors, added simple error comparison tool

* wrapping up error formatting

* Fixes for unit tests

* fix PLUGINS_VALIDATION_ERROR

* fix fs.readdir bug, show rows even if there's only markdown formatting [SKIP CI]

* when in base-list, show full width of errors

* Fix type errors

* added searching and filtering for files based on error name

* fix: system tests

* updated NO_SPECS_FOUND error to properly join searched folder + pattern

- join patterns off of process.cwd, not projectRoot
- highlight original specPattern in yellow, baseDir in blue
- add tests

* fixes failing tests

* fix test

* preserve original spec pattern, display relative to projectRoot for terminal banner

* make the nodeVersion path display in gray, not white

* fix tests, pass right variables

* fix chrome:canary snapshots

* update snapshots

* update snapshot

* strip newlines caused by "Still waiting to connect to {browser}..."

* don't remove the snapshotHtmlFolder in CI, add additional verification snapshots match to error keys symmetrically

* update snapshot

* update snapshot

* update snapshot

* update snapshot

* update snapshot

* update snapshot

* update gitignore

* fix snapshot

* update snapshot html

* update logic for parsing the resolve pattern matching, add tests

* update snapshots

* update snapshot

* update snapshot

* update snapshot

* fix failing test

* fix: error_message_spec

* fix snapshot

* run each variant through an it(...) so multiple failures are received

* add newlines to multiline formatters, add fmt.stringify, allow format overrides

* stringify invalid return values from plugins

* move config validation errors into packages/errors, properly highlight and stringify values

* add component testing yarn commands

* fix the arrow not showing on details

* fix typescript error

* fixed lots of poorly written tests that weren't actually testing anything. created settings validation error when given a string validation result.

* fixes tests

* fixes tests, adds new error template for validating within an array list (for browser validation)

* remove dupe line

* fix copy for consistency, update snapshots

* remove redundant errors, standardize formatting and phrasing

* more formatting

* remove excess snapshots

* prune out excessive error snapshot html files when not in CI

* add missing tests, add case for when config validation fails without a fileType

* fixes test

* update snapshot

* update snapshot

* update snapshot

* sort uniqErrors + errorKeys prior to assertion - assert one by one

* add system test for binding to an event with the wrong handler

* fixes tests

* set more descriptive errors when setting invalid plugin events, or during plugin event validation

* remove duplicate PLUGINS_EVENT_ERROR, collapse into existing error

* use the same multiline formatting as @packages/errors

* standardize verbiage and highlighting for consistency

* fix incorrect error path

* fixes tests, standardized and condensed more language

* Update packages/errors/src/errors.ts

Co-authored-by: Ryan Manuel <ryanm@cypress.io>

* Update guides/error-handling.md

Co-authored-by: Ryan Manuel <ryanm@cypress.io>

* Update guides/error-handling.md

Co-authored-by: Ryan Manuel <ryanm@cypress.io>

* added some final todo's

* fix types

Co-authored-by: Tim Griesser <tgriesser10@gmail.com>
Co-authored-by: Tim Griesser <tgriesser@gmail.com>
Co-authored-by: Ryan Manuel <ryanm@cypress.io>
2022-02-11 02:06:07 -05:00

224 lines
6.7 KiB
TypeScript

/* eslint-disable no-console */
import express from 'express'
import fs from 'fs-extra'
import globby from 'globby'
import Markdown from 'markdown-it'
import path from 'path'
import { WIDTH } from './utils'
const ERRORS_DIR = path.join(__dirname, '..', '..')
const SNAPSHOT_HTML = path.join(ERRORS_DIR, '__snapshot-html__')
const SNAPSHOT_HTML_LOCAL = path.join(ERRORS_DIR, '__snapshot-html-local__')
const SNAPSHOT_MARKDOWN = path.join(ERRORS_DIR, '__snapshot-md__')
const app = express()
const LINKS = `<h5><a href="/">Ansi Compare</a> | <a href="/base-list">Ansi Base List</a> | <a href="/md">Markdown</a> | <input id="search" placeholder="search error by name"><button id="go">go</button></h5>`
const SEARCH_HANDLER = `
<script type="text/javascript">
$(document).on('click', '#go', (ev) => {
const val = $('#search').val()
window.location.search = 'spec=' + val
})
$(document).ready(() => {
const u = new URL(window.location.href)
const spec = u.searchParams.get('spec')
$('#search').val(spec)
})
</script>
`
const getFiles = async (baseDir: string, spec?: string) => {
const pattern = spec ? `${spec}*` : '**/*'
return (await globby(`${baseDir}/${pattern}`))
}
async function getRows (offset = 0, baseList: boolean = false, spec?: string) {
const baseDir = baseList ? SNAPSHOT_HTML : SNAPSHOT_HTML_LOCAL
const toCompare = await getFiles(baseDir, spec)
const rows = toCompare.filter((f) => f.endsWith('.html')).sort().slice(offset, offset + 10).map((f) => path.basename(f).split('.')[0]).map((name) => {
const width = baseList ? WIDTH : 550
// const height = baseList ? HEIGHT : 600
const height = 400
return `
<tr id="${name}">
<td>${name}</td>
<td colspan="2"><iframe src="/html/${name}/__snapshot-html__" width="${width}" height="${height}"></iframe></td>
${baseList ? '' : `
<td colspan="2"><iframe src="/html/${name}/__snapshot-html-local__" width="${width}" height="${height}"></iframe></td>
<td><button data-looks-good="${name}">Looks Good</button></td>
`}
</tr>`
})
if (toCompare.length > offset + 10) {
rows.push(`<tr id="loadMore"><td colspan="6"><button style="width:100%" data-load-more="${offset + 10}">Load More</button></td><tr>`)
}
return rows.join('\n')
}
app.get('/', async (req, res) => {
try {
const rows = await getRows()
res.type('html').send(`
<script src="https://code.jquery.com/jquery-3.6.0.js" integrity="sha256-H+K7U5CnXl1h5ywQfKtSj8PCmoN9aaq30gDh27Xc0jk=" crossorigin="anonymous"></script>
<script type="text/javascript">
$(document).on('click', '[data-looks-good]', (ev) => {
const id = $(ev.currentTarget).data('looksGood')
fetch('/looks-good/' + id)
.then((res) => res.json())
.then(() => {
$('#' + id).remove()
})
.catch(e => {
alert(e.stack)
})
})
$(document).on('click', '[data-load-more]', (ev) => {
const offset = $(ev.currentTarget).data('loadMore')
$('#loadMore').remove()
fetch('/load-more/' + offset)
.then(res => res.text())
.then((val) => {
$(val).appendTo('tbody')
})
})
</script>
${LINKS}
<table>
<thead>
<tr><th>Table</th><th colspan="2">Original</th><th colspan="2">New</th></tr>
<thead>
<tbody>
${rows}
</tbody>
</table>
`)
} catch (e) {
res.json({ errStack: e.stack })
}
})
app.get('/base-list', async (req, res) => {
const spec = req.query.spec as string
try {
const rows = await getRows(0, true, spec)
res.type('html').send(`
<script src="https://code.jquery.com/jquery-3.6.0.js" integrity="sha256-H+K7U5CnXl1h5ywQfKtSj8PCmoN9aaq30gDh27Xc0jk=" crossorigin="anonymous"></script>
<script type="text/javascript">
$(document).on('click', '[data-load-more]', (ev) => {
const offset = $(ev.currentTarget).data('loadMore')
$('#loadMore').remove()
fetch('/load-more-base/' + offset)
.then(res => res.text())
.then((val) => {
$(val).appendTo('tbody')
})
})
</script>
${LINKS}
${SEARCH_HANDLER}
<table>
<thead>
<tr><th>Table</th><th colspan="2">Original</th></tr>
<thead>
<tbody>
${rows}
</tbody>
</table>
`)
} catch (e) {
res.json({ errStack: e.stack })
}
})
app.get<{offset: number}>('/load-more/:offset', async (req, res) => {
const rows = await getRows(req.params.offset)
res.send(rows)
})
app.get<{offset: number}>('/load-more-base/:offset', async (req, res) => {
const rows = await getRows(req.params.offset, true)
res.send(rows)
})
app.get('/looks-good/:name', async (req, res) => {
try {
await fs.move(
path.join(SNAPSHOT_HTML_LOCAL, `${req.params.name}.html`),
path.join(SNAPSHOT_HTML, `${req.params.name}.html`),
{ overwrite: true },
)
res.json({ ok: true })
} catch (e) {
res.status(400).json({ stack: e.stack })
}
})
app.get<{name: string, type: string}>('/html/:name/:type', async (req, res) => {
const pathToFile = path.join(ERRORS_DIR, req.params.type, `${req.params.name}.html`)
try {
const contents = await fs.readFile(pathToFile, 'utf8')
res.type('html').send(contents.replace(/\<link(.*?)>/g, '').replace('overflow: hidden;', ''))
} catch (e) {
res.json({ errStack: e })
}
})
app.get('/md', async (req, res) => {
const spec = req.query.spec as string
try {
const toRender = (await getFiles(SNAPSHOT_MARKDOWN, spec)).filter((f) => f.endsWith('.md')).sort()
const markdownContents = await Promise.all(toRender.map((f) => fs.readFile(f, 'utf8')))
const md = new Markdown({
html: true,
linkify: true,
})
res.type('html').send(`
<script src="https://code.jquery.com/jquery-3.6.0.js" integrity="sha256-H+K7U5CnXl1h5ywQfKtSj8PCmoN9aaq30gDh27Xc0jk=" crossorigin="anonymous"></script>
${LINKS}
${SEARCH_HANDLER}
<style>
tr {
outline: 1px solid black;
}
pre {
overflow: scroll;
}
</style>
<div>
${toRender.map((r, i) => {
return `<div style="border: 1px solid grey; margin-bottom: 10px;">
<div>${path.basename(r).split('.')[0]}</div>
<div style="width: 100%; display: flex;">
<div style="width: 50%">${md.render(markdownContents[i] ?? '')}</div>
<div style="width: 50%; overflow: scroll;"><pre>${markdownContents[i] ?? ''}</pre></div>
</div>
</div>`
}).join('\n')}
</div>
`)
} catch (e) {
res.json({ stack: e.stack })
}
})
app.listen(5555, () => {
console.log(`Comparison server listening on: http://localhost:5555`)
})