mirror of
https://github.com/cypress-io/cypress.git
synced 2026-05-25 01:49:06 -05:00
Ryan Manuel
f2d8def94a
* chore(server): add bundle cache root resolver
First step of the cy-prompt/Studio bundle cache rework. Adds
getBundleCacheRoot() / getBundleCacheDir(kind) under
packages/server/lib/cloud/bundles/cache_root.ts, returning
<cypress-cache>/bundles[/<kind>] and honoring CYPRESS_CACHE_FOLDER
the same way the CLI does (untildify + path.resolve, falling back
to cachedir('Cypress')).
Adds cachedir@^2.4.0 and untildify@^4.0.0 to @packages/server deps
to match the versions already used by the CLI.
No callers wired yet — pure helper landing ahead of the streaming
verify+extract module.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(server): extract_atomic: extract retry harness, add renameAtomicWithRetry
Lifts the EPERM/EACCES retry loop from writeFileAtomicWithRetry into
a generic retryOnRenameError<T>(op) helper, then adds
renameAtomicWithRetry(src, dst) using the same harness.
renameAtomicWithRetry will be used by the upcoming bundle-cache
publish step, which moves staged files into the shared <hash>/
directory via per-file atomic rename. This preserves the
cross-process safety property from PRs #33034 / #33330: a concurrent
reader always sees either the prior version's complete bytes or the
new version's complete bytes, never absent or partial.
Behavior of writeFileAtomicWithRetry is unchanged — same retries
(3), same delay (100ms), same error semantics. Existing extractAtomic
spec passes unmodified; new spec block covers renameAtomicWithRetry
(success, EPERM/EACCES retry, exhausted budget, non-retryable codes).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(server): ensureSignedBundle module for unified bundle download
Adds the streaming verify+extract module under
packages/server/lib/cloud/bundles/ that will replace the legacy
download-to-tar + re-open + verify + re-extract flow used by
ensureCyPromptBundle / ensureStudioBundle.
Modules:
- bundle_error.ts : BundleError(kind, stage) for Sentry tagging
- parse_hash_from_bundle_url.ts : hash extraction shared with lifecycle managers
- sweep_orphan_staging.ts : best-effort cleanup of crashed-process staging dirs
- stream_download_verify_extract.ts : single-pass fetch -> SHA256 verify tee
-> tar.Parse, files written to a
per-process .staging-<rand>/ dir.
No on-disk tar artifact.
- publish_staging_to_final.ts : per-file renameAtomicWithRetry from staging
into shared <hash>/, manifest.json last.
- ensure_signed_bundle.ts : orchestrator. Both signatures must verify before
any byte lands in the shared <hash>/ dir.
encryption.ts: new createStreamingSignatureVerifier export so the streaming
tee can update a SHA256 verifier per chunk without buffering the tar.
Cross-process safety property from PR #33034 / #33330 is preserved: each
process has a private staging dir, publish is gated on signature verify,
and per-file atomic rename means a concurrent reader always sees either
prior-version or new-version bytes for any file -- never absent or partial.
Telemetry: errors carry a structured `stage` tag (network / signature /
extract / manifest / publish) so Sentry filtering reproduces the planned
`bundle.download.error.{...}` taxonomy. A net-new counter framework was
not built -- the existing telemetry surface is duration marks/measures
only, and the legacy `ENOENT bundle.tar-<rand>` Sentry signal disappears
structurally when step 7 deletes the legacy fetchers. Counters can be a
follow-up if Sentry filtering proves insufficient.
No callers wired yet -- pure new module landing ahead of caller swap.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(server): unit + cross-process tests for ensureSignedBundle module
Adds 20 tests covering the new bundles/ module:
- parse_hash_from_bundle_url_spec.ts (5) : url -> hash extraction edge cases
- bundle_error_spec.ts (3) : kind/stage tagging + isBundleError
- sweep_orphan_staging_spec.ts (4) : age-threshold filtering, non-staging
entries ignored, missing dir tolerated
- publish_staging_to_final_spec.ts (3) : happy path, manifest renamed last,
cross-process safety regression
- ensure_signed_bundle_spec.ts (5) : orchestrator happy path, signature
failure, missing manifest, network
error propagation, staging cleanup
on publish failure
The cross-process test is the load-bearing one: it spawns two real child
processes (via `node -r @packages/ts/register cross_process_publish_worker.ts`)
that each call publishStagingToFinal into a shared finalDir while the
parent runs a tight readFile loop. After both children exit, the test
asserts (a) zero ENOENT observations on the watched file, (b) zero
corrupt/partial reads, and (c) finalDir contains every file with
matching content. This directly guards the PR #33034 invariant the
rework had to preserve.
Worker script lives in test/support/ rather than test/unit/ so the
mocha glob doesn't try to load it as a spec.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(server): wire cy-prompt to ensureSignedBundle
ensure_cy_prompt_bundle.ts is now a thin wrapper around
ensureSignedBundle({ kind: 'cy-prompt' }) that returns
{ manifest, cyPromptPath } (where cyPromptPath is the bundleDir
the new helper computed under <cypress-cache>/bundles/cy-prompt/<hash>).
CyPromptLifecycleManager:
- Drops os.tmpdir()-based path construction.
- Hash extraction now uses parseHashFromBundleUrl (shared with the
bundle helper).
- hashLoadingMap value type updated from Promise<Record<string,string>>
to Promise<{ manifest, cyPromptPath }>; the path now flows out of
ensureCyPromptBundle rather than being computed twice.
CYPRESS_LOCAL_CY_PROMPT_PATH bypass is preserved verbatim.
Existing CyPromptLifecycleManager_spec assertions on tmpdir-based paths
still pass because the test stub returns that exact path -- the legacy
path is still a valid string in the test, just no longer baked into the
production code.
ensure_cy_prompt_bundle_spec was rewritten as a delegation test (3
cases: forwarding, undefined projectId, error propagation) since all
the prior assertions on download/extract/verify mechanics now live in
the bundles/ unit tests added in the previous commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(server): wire studio to ensureSignedBundle
Symmetric to the cy-prompt wiring. ensure_studio_bundle.ts is now a
thin wrapper around ensureSignedBundle({ kind: 'studio' }) that
returns { manifest, studioPath } where studioPath is the bundleDir
under <cypress-cache>/bundles/studio/<hash>.
StudioLifecycleManager:
- Drops os.tmpdir()-based path construction.
- Hash extraction now uses parseHashFromBundleUrl (shared with the
bundle helper and cy-prompt manager).
- hashLoadingMap value type updated; the path now flows out of
ensureStudioBundle rather than being computed twice.
CYPRESS_LOCAL_STUDIO_PATH bypass + currentStudioHash retry-clear
behavior preserved verbatim.
ensure_studio_bundle_spec rewritten as a delegation test (3 cases).
StudioLifecycleManager_spec assertions updated for the new
{manifest, studioPath} return shape; the 'clears cached bundle promises
on retry' test's dummyPromise was updated to match the new shape so
the awaiting code path in the lifecycle manager still works.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(server): delete get_cy_prompt_bundle / get_studio_bundle
These two legacy fetchers (and their specs) are now subsumed by
streamDownloadVerifyExtract under packages/server/lib/cloud/bundles/.
The lifecycle managers and ensure_*_bundle wrappers have been wired
to the new helper in the previous two commits, so nothing else
imports these.
extract_atomic.ts is intentionally kept -- it remains the home for
renameAtomicWithRetry, which the bundle publish step uses.
This also eliminates the legacy 'ENOENT bundle.tar-<rand>' Sentry
signal structurally: there's no longer any code path in the runtime
that writes a bundle.tar file to disk.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(cli): cache prune ignores non-semver entries; cache clear/prune cover bundles/
cypress cache prune now filters by util.isSemver before removing entries,
matching the existing behavior of getCachedVersions (used by cache list).
Without this filter, prune would also remove the new bundles/ subdir
that the server uses to cache cy-prompt and Studio bundles.
cypress cache clear is unchanged (still fs.remove the entire cache root)
and removes bundles/ along with binary version dirs, which is the
documented behavior.
Adds two regression tests:
- cache clear removes bundles/cy-prompt/<hash>/ and bundles/studio/<hash>/
alongside binary version dirs.
- cache prune preserves bundles/ while pruning old binary versions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(changelog): note bundle cache rework + ENOENT fix
Bugfix entry under 15.15.0 calling out:
- The intermittent ENOENT bundle.tar failure class on Linux
- The streaming verify+extract approach (no on-disk tar artifact)
- The new <cypress-cache>/bundles/<kind>/<hash>/ layout
- CYPRESS_CACHE_FOLDER honored, cache clear/prune behavior
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(server): drop dead extractAtomic / writeFileAtomicWithRetry
After the cy-prompt and Studio call sites were migrated to
streamDownloadVerifyExtract, neither extractAtomic nor its supporting
buffer-based writeFileAtomicWithRetry has any callers left. Both
removed along with the unused tar / createReadStream / writeFileAtomic
imports.
extract_atomic.ts now contains only renameAtomicWithRetry plus its
shared retry harness -- the load-bearing primitive used during the
bundle publish step. extract_atomic_spec.ts trimmed to just the 6
renameAtomicWithRetry tests.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(cli): cache prune skips bundles/ by name, still removes beta/prerelease binaries
The previous fix used util.isSemver to gate prune, but state.getVersionDir()
formats non-stable builds as 'beta-<version>-<branch>-<sha>' which is not a
valid semver. That spelling would orphan beta/prerelease binary caches forever.
Switched to a targeted name-based exclusion (NON_BINARY_CACHE_ENTRIES, currently
just 'bundles'). Restores the original 'anything not the current binary version'
prune semantics for both stable and beta dirs, while still preserving the
bundles/ tree.
Adds a regression test that pre-populates two beta-prefixed binary dirs and
asserts they are pruned alongside the existing bundles-preservation test.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(server): bundle fetch timeout now actually retries
Wrapping a fetch AbortError as BundleError(stage='network') stopped the
retry harness in its tracks: asyncRetry's shouldRetry uses isRetryableError,
which only matches SystemError or HttpError. A timeout would surface as a
non-retryable BundleError and fail on the first attempt instead of using
the configured retry budget (3 attempts, ~1.5s of linear backoff).
Switched to throwing a SystemError with code 'ETIMEDOUT' on AbortError
inside the retry loop. SystemError is matched by isRetryableError, so the
timeout now consumes the full retry budget like every other transient
network failure.
Adds a regression test that stubs cross-fetch to throw AbortError and
asserts (a) fetch is called 3 times (full budget), (b) every attempt's
error is a retryable SystemError with code 'ETIMEDOUT'.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(server): tag network/extract bundle errors with BundleError stage
streamDownloadVerifyExtract was rethrowing raw HttpError/SystemError values
on the network and extraction paths, so the only error surface that
actually carried the BundleError(kind, stage) tags promised by the rework
was the signature path. Sentry filtering on err.stage would have missed
every fetch failure, every mid-stream connection drop, every HTTP 4xx/5xx,
and every tar parse error.
Now both phases route through wrapAsBundleError, which:
- keeps the underlying HttpError/SystemError as `cause` so asyncRetry's
cause-aware shouldRetry preserves the existing isRetryableError-based
classification (HTTP 408/429/5xx + SystemError keep retrying;
HTTP 4xx + tar parse errors fail fast as before),
- emits stage='network' for fetch-phase failures, AbortError timeouts,
and POSIX-style syscall codes (ECONNRESET, ETIMEDOUT, EAI_AGAIN, ...)
during the pipeline,
- emits stage='extract' for non-syscall pipeline errors (overwhelmingly
tar.Parse complaining about malformed bytes).
Tightened the syscall-shape heuristic to /^E[A-Z]/ and explicitly excluded
Node ERR_* codes, since a TypeError thrown inside HttpError.fromResponse
(via scrubUrl) carries code 'ERR_INVALID_URL' and would otherwise be
miswrapped as a retryable SystemError.
Adds 4 regression tests covering: timeout retry, HTTP 4xx no-retry, HTTP
503 retry, tar parse no-retry. Each asserts both the BundleError shape
(stage + kind + cause type) and the retry call count, so a future shape
change can't silently regress either property.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(server): drop inline type import in ensure_signed_bundle
V8 snapshot processing can't handle mixed value/type imports on a
single line. Inline `type BundleKind` -> bare `BundleKind`. Same
treatment is already applied in stream_download_verify_extract.ts.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(server): drop test/integration/cloud/extract_atomic_spec.ts
This integration spec imported the now-removed `extractAtomic` export and
crashed at runtime with "extractAtomic is not a function" once that
function was deleted in the dead-code prune. Missed it because I only
inspected test/unit/.
Coverage of the underlying fs primitive lives on:
- renameAtomicWithRetry has full retry/EPERM-EACCES coverage in the unit
spec at test/unit/cloud/extract_atomic_spec.ts.
- The cross-process publish flow (which is the load-bearing tar-extract +
per-file rename path the deleted integration spec was sanity-checking)
has a real-FS regression test in
test/unit/cloud/bundles/publish_staging_to_final_spec.ts that spawns
two child processes against a shared finalDir.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(server): scope proxyquire.noPreserveCache() to TelemetryReporter spec
The TelemetryReporter spec called proxyquire.noPreserveCache() at module
load time, which mutates the global proxyquire instance for the rest of
the mocha process. Every subsequent proxyquire(...) call across the
entire suite then reloads transitive deps, so a downstream spec's
top-level imports and the proxyquired-module's transitive imports
end up resolving to *different instances* of the same module.
Concrete impact: in the full unit-suite run, CyPromptLifecycleManager
and StudioLifecycleManager would fail their setup `calledWith({...,
cloudApi: { CloudRequest, ... } })` assertions because the test's
imported CloudRequest and the lifecycle manager's imported CloudRequest
ended up as different `axios.create()` instances. The matcher
`sinon.match.func` also lost its identity (samsam's matcher detection
compared the matcher created with the pre-pollution sinon against a
post-pollution sinon, so it deep-equal'd as `{ test, message }`
instead of being recognized as a matcher).
Wrap the noPreserveCache() in before/after hooks scoped to this
describe block so the toggle is reverted by `proxyquire.preserveCache()`
on the way out -- no other spec is affected. The TelemetryReporter
spec's own 6 tests still pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(server): drop dead exports left behind by the bundle cache rework
Cleanup of unused symbols that became dead once the streaming verify+extract
helper replaced the legacy fetcher path:
- write-file-atomic (package.json): only consumer was the deleted
writeFileAtomicWithRetry buffer-based path in extract_atomic.ts.
- verifySignatureFromFile (encryption.ts) + its 2 spec tests: only callers
were the deleted get_cy_prompt_bundle.ts / get_studio_bundle.ts. The
streaming-verify path uses createStreamingSignatureVerifier instead.
- getBundleCacheRoot (cache_root.ts): only used internally by
getBundleCacheDir; downgrade from `export const` to `const`.
- EnsureSignedBundleOptions / EnsureSignedBundleResult interfaces: only
consumed inside ensure_signed_bundle.ts; drop `export` on both.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(server): surface cause's syscall code on BundleError so cert detection works
StudioLifecycleManager#updateStatus reads `error?.code` and feeds it into
isNonRetriableCertErrorCode() to decide whether to flip isCertError and
show the proxy-specific recovery message. Wrapping a SystemError inside
BundleError({cause: sysError}) made the top-level `code` undefined, so
certificate failures during studio bundle download silently lost their
cert classification and users got the generic error path instead.
BundleError now mirrors a string-typed `cause.code` onto its own `code`
property so any consumer that reads error.code keeps working.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(server): drain in-flight publish renames before throwing
publishStagingToFinal used Promise.all on the non-manifest renames, which
rejects on the first failure but leaves siblings running. The caller
(ensureSignedBundle) then runs `finally { remove(staging) }`, which races
the still-in-flight renames -- those renames find their src path missing
and reject with ENOENT, surfacing as unhandled rejections.
Mirrored the pattern stream_download_verify_extract.ts already uses for
its entryPromises: capture the promises in a let, await Promise.all in a
try, and Promise.allSettled them in the catch before re-throwing. The
function still rejects with the original error; in-flight siblings are
just guaranteed to settle first.
Adds a regression test that stubs renameAtomicWithRetry so one rel rejects
fast and the others resolve slowly, asserts no `unhandledRejection` event
fires during the test window, and confirms the slow renames settled
before the function returned.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(server): match CLI semantics for CYPRESS_CACHE_FOLDER override
cache_root.ts read process.env.CYPRESS_CACHE_FOLDER directly, but the
CLI's state.getCacheDir() routes the same variable through util.getEnv(),
which:
- trims leading/trailing whitespace,
- strips surrounding double quotes (so Windows CMD's
`set CYPRESS_CACHE_FOLDER="C:\cache"` -- which sets the value with
embedded quotes -- works the same on both sides), and
- falls back to npm_config_* / npm_package_config_* env-var spellings
so values from .npmrc or `--cypress-cache-folder=...` flags reach the
server too.
Without this, the server resolved `<cache>/bundles` to a different path
than the one `cypress cache clear` removes, and npm-config-based
overrides were silently ignored server-side.
Adds a small dequote()/readEnvVar() pair (intentionally local rather than
importing from cli/, since the two packages don't share util code) and
seven tests covering: clean path, surrounding quotes, whitespace trim,
npm_config_* fallback, lowercase variant, bare-var precedence, and
empty/whitespace-only-override falling through to cachedir().
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(changelog): trim the bundle-cache rework entry
Cut the wording to two sentences to match the surrounding entries' style.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(server): trim verbose comments in bundle cache files
Tightened comments to match the surrounding code style and dropped
rationale that wasn't load-bearing for a future reader.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(server): retry bundle fetches on HTTP 500 by passing GET to isRetryableError
isRetryableError(err, method) treats HTTP 500 as retryable only when the
method is one of the idempotent set (GET/HEAD/PUT/DELETE/OPTIONS). The
deleted get_cy_prompt_bundle.ts / get_studio_bundle.ts passed 'GET'
explicitly; shouldRetryBundleError was passing only the error, so a
transient 500 from the bundle CDN failed on the first attempt instead of
consuming the retry budget.
Pass 'GET' through both code paths (cause-unwrapped and bare). Adds a
regression test asserting fetchStub is called 3 times when the CDN
responds with 500.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Apply suggestion from @ryanthemanuel
* Update CHANGELOG.md
* fix(server): retry renameAtomicWithRetry on EBUSY for Windows
Adds EBUSY to the retry classifier alongside EPERM/EACCES. On Windows,
`fs.rename` can fail with EBUSY when the destination is held open by
another process (Defender / AV, an indexer, a file watcher) -- the same
failure class PR #33330 already retried for EPERM/EACCES, just under a
different code on a different transient holder.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(server): don't retry filesystem syscall errors during bundle extract
Filesystem errors (ENOSPC, EACCES, EROFS, EIO, ...) raised by tar-entry
writes during the pipeline phase were wrapped as SystemError and tagged
stage='network', causing isRetryableError to burn the full retry budget
on non-transient failures and mis-classifying them in Sentry. Now only
network-class POSIX codes (ECONNRESET, ETIMEDOUT, ...) retain
network/retryable wrapping mid-pipeline; everything else honors
defaultStage='extract' and keeps the raw error as cause.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(server): drop shared counter in sweepOrphanStaging
Return per-task booleans and tally after Promise.all instead of mutating
a shared counter from concurrent async callbacks.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Apply suggestion from @ryanthemanuel
* Apply suggestion from @ryanthemanuel
* Apply suggestion from @ryanthemanuel
* Update cli/CHANGELOG.md
Co-authored-by: Matt Schile <mschile@cypress.io>
* Update CHANGELOG.md
* Apply suggestion from @ryanthemanuel
* chore: PR review follow-ups for bundle cache rework
- Rename NON_BINARY_CACHE_ENTRIES to EXTERNAL_CACHE_ENTRIES in cli cache task.
- Refactor walkFiles in publish_staging_to_final to collect arrays via
Promise.all and flatten, instead of mutating a shared results array
from concurrent async callbacks. Matches sweepOrphanStaging's pattern.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(cli): align CYPRESS_CACHE_FOLDER trim/dequote with server bundle cache
state.getCacheDir() now passes trim=true to util.getEnv, applying
dequote() + trim per util.ts:538. Without this, Windows CMD's
`set CYPRESS_CACHE_FOLDER="C:\path"` left literal quotes in the resolved
path (cypress-io/cypress#4506), so `cypress cache clear` targeted a
different directory than the server wrote bundles to. Updates the
server-side comment in cache_root.ts to reference the trim flag and
adds CLI tests covering quoted, whitespace-padded, and whitespace-only
overrides so parity stays enforced.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Matt Schile <mschile@cypress.io>