mirror/cypress
1
0
Fork 0
mirror of https://github.com/cypress-io/cypress.git synced 2026-03-12 12:29:38 -05:00
Code Issues Packages Projects Releases Wiki Activity

Retry certain requests on failure (#4015)

Browse Source 
* retry requests, basic e2e test

* basic e2e test for chrome behavior

* don't use timeouts to test

* some minor cleanup

* validate google chrome's retry behavior w/ a proxy

* get retries on network errors workin

* cleanup

* final changes before switching approach

* Reverting previous approach in request.coffee, server.coffee

* add retryOnNetworkFailure

* now works with retryOnStatusCodeFailure

* retry 4 times in addition to the 1 initial attempt

* add tests for subresources

* much improved error handling

* have the e2e test really work

* e2e baseurl check

* retry baseurl check

* remove special handling for node 8.2.1 bug

* WIP: continue making progress building out request retry

- swap out passthrough’s for pumpify / duplexify / pump
- clean up error handling / messaging / retry logic

* pipe clientrequest events

* buffer req body to disk, restore error handling/retrying

* don't bubble up errors from failed attempts

* actually pipe reqstream, oops

* add some e2e tests for request body

* revert lib/request.coffee to 7be1051

* add almost-working lazy-stream

* manually fire the 'pipe' event on the reqStream to copy headers to the outgoing message

- restore the ‘error’ propagation so that all tests pass for now

* cleanup leaking 'undefined' into stdout, causing failing e2e tests

- do not set onWarning as a property of project, just pass as an
argument

* add new options to request_spec, deduplicate default opts

* use stream buffer in request.coffee

* revert request.coffee

* last stream_buffer changes before removing fs stuff

* remove fs stuff from stream_buffer, add stream piping tests

* it works! :tada::tada:🎉 using duplexify to separate delayStream and reqBodyBuffer

* retry for .5s max on ECONNREFUSED

* add error codes

* don't timeout proxied requests

* restore baseurl check

* update new e2e tests

* make delay work with rp

* propagate clientresponse events

* removing tests that don't do anything now that we don't ESOCKETTIMEOUT on proxied requests

* add new visit, req options to index.d.ts

* don't fail on server-performance-test

* make retries with status code work again

* account for different stack trace in ci

* fix test

* retry https requests

* add tests for https passthru retries working

* clean up error handling for https-proxy

* fix failing https-proxy tests, tweak agent error handling to prevent multiple callbacks

* make expectedExitCode actual vs. expected in the correct order

* bump up e2e test timeout so it can retry and still work

* update tests

* retry up to 500ms on proxied requests

* add tests for incompatible options errors

* remove .only

* maybe this will help it act more consistently

* help e2e test work in ci

* don't reconnect on already made connections

* clarify naming

* wip: testing https proxy

* better debug calls

* WIP: getting proxy passthrough retry tests going

* handle retrying upstream proxy errors

- add tests for successfully retrying proxy errors and for unsuccessful
retries
- fix onClose errors when proxy connection is closed before enough data
is received
- fix not returning setTimeout correctly

* group related code accordingly

* do not build typescript by default, drop extension from main

* more TODO notes

* don't set a default NO_PROXY if NO_PROXY = ''

* debugging-proxy@2.0.0

* null out reqBodyBuffer when req finishes

* wip: retry in agent, not https-proxy [skip-ci]

* update https-proxy to use @packages/network retries

* retry after connection but before proxy tunnel established

* appease my linty overlords

* update https-proxy tests

* update agent specs, decided to still use tls.connect

it's easier to test and has less complexity

* test retrying HTTPS passthru

* debugging-proxy@2.0.1

* increase defaultCommandTimeout 100 -> 200 to prevent flake in CI

* auto formatting

* fix test to be dynamic and not rely on magic constants

* copy types field when linking proxy images, update packages/network types field

* linting

* add network index.js file

* linting

* improve error messaging experience when verifying base url

* only insert 1 new line

* fix failing test not binding to localhost

* removed test that's covered by e2e specs

* remove dash in 're-try'

* some cleanup for readability

* use allocUnsafe per perf

* unset NO_PROXY with an empty string

* move retry ensuring the baseUrl into url, cleanup some imperative code

* if the head is already present during connect, make connection, else wait for first head bytes

* minor formatting, clarity around conditions, naming

* rename retryInterval -> retryIntervals

* set defaults for requests during creation for clarity

* rename send -> sendPromise to pair better with sendStream

* use retryIntervals instead of juggling MAX_REQUEST_RETRIES

- ensure debug messages are consistent between request streams +
promises
- set static constants

* DRY up status check + network failure + retry logic

- keeps the debug logic identical between promises + streams
- ensures all logic paths are also consistent
- consolidates the pop’ing of intervals in a single place

* find / replace fail

* derp

* make the logic actually correct, set intervals as cloned defaults for recursive lookup

* pass arg correctly

* reduce debugging noise, formatting

* rename intervals -> delaysRemaining for clarity

* added unit tests around getDelayForRetry

* set retryIntervals as default options correctly, add unit tests


Co-authored-by: Brian Mann <brian.mann86@gmail.com>
Co-authored-by: Jennifer Shehane <jennifer@cypress.io>
Co-authored-by: Gleb Bahmutov <gleb.bahmutov@gmail.com>
This commit is contained in:
Zach Bloomquist
2019-05-15 12:46:55 -04:00
committed by Brian Mann
parent d4d7326ebb
commit b8a6baaca3
42 changed files with 1849 additions and 486 deletions
Download Patch File Download Diff File Expand all files Collapse all files

202
packages/https-proxy/lib/server.coffee
View File

@@ -1,5 +1,5 @@
_ = require("lodash")
agent = require("@packages/network").agent
{ agent, connect } = require("@packages/network")
allowDestroy = require("server-destroy-vvo")
debug = require("debug")("cypress:https-proxy")
fs = require("fs-extra")
@@ -26,41 +26,57 @@ class Server
constructor: (@_ca, @_port) ->
@_onError = null
connect: (req, socket, head, options = {}) ->
connect: (req, browserSocket, head, options = {}) ->
## don't buffer writes - thanks a lot, Nagle
## https://github.com/cypress-io/cypress/issues/3192
socket.setNoDelay(true)
browserSocket.setNoDelay(true)
if not head or head.length is 0
debug("Writing socket connection headers for URL:", req.url)
debug("Writing browserSocket connection headers %o", { url: req.url })
socket.once "data", (data) =>
@connect(req, socket, data, options)
browserSocket.on "error", (err) =>
## TODO: shouldn't we destroy the upstream socket here?
## and also vise versa if the upstream socket throws?
## we may get this "for free" though because piping will
## automatically forward the TCP errors...?
socket.write "HTTP/1.1 200 OK\r\n"
## nothing to do except catch here, the browser has d/c'd
debug("received error on client browserSocket %o", {
err, url: req.url
})
if req.headers["proxy-connection"] is "keep-alive"
socket.write("Proxy-Connection: keep-alive\r\n")
socket.write("Connection: keep-alive\r\n")
browserSocket.write "HTTP/1.1 200 OK\r\n"
return socket.write("\r\n")
if req.headers["proxy-connection"] is "keep-alive"
browserSocket.write("Proxy-Connection: keep-alive\r\n")
browserSocket.write("Connection: keep-alive\r\n")
else
if odc = options.onDirectConnection
## if onDirectConnection return true
## then dont proxy, just pass this through
if odc.call(@, req, socket, head) is true
return @_makeDirectConnection(req, socket, head)
else
debug("Not making direct connection to #{req.url}")
browserSocket.write("\r\n")
socket.pause()
## if we somehow already have the head here
if _.get(head, "length")
## then immediately make up the connection
return @_onFirstHeadBytes(req, browserSocket, head, options)
@_onServerConnectData(req, socket, head)
## else once we get it make the connection later
browserSocket.once "data", (data) =>
@_onFirstHeadBytes(req, browserSocket, data, options)
_onUpgrade: (fn, req, socket, head) ->
_onFirstHeadBytes: (req, browserSocket, head, options) ->
browserSocket.pause()
if odc = options.onDirectConnection
## if onDirectConnection return true
## then dont proxy, just pass this through
if odc.call(@, req, browserSocket, head) is true
return @_makeDirectConnection(req, browserSocket, head)
else
debug("Not making direct connection %o", { url: req.url })
@_onServerConnectData(req, browserSocket, head)
_onUpgrade: (fn, req, browserSocket, head) ->
if fn
fn.call(@, req, socket, head)
fn.call(@, req, browserSocket, head)
_onRequest: (fn, req, res) ->
hostPort = parse.hostAndPort(req.url, req.headers, 443)
@@ -80,107 +96,93 @@ class Server
res.end()
.pipe(res)
_upstreamProxyForHostPort: (hostname, port) ->
getProxyForUrl("https://#{hostname}:#{port}")
_makeDirectConnection: (req, socket, head) ->
_makeDirectConnection: (req, browserSocket, head) ->
{ port, hostname } = url.parse("http://#{req.url}")
if upstreamProxy = @_upstreamProxyForHostPort(hostname, port)
return @_makeUpstreamProxyConnection(upstreamProxy, socket, head, port, hostname)
debug("Making connection to #{hostname}:#{port}")
@_makeConnection(browserSocket, head, port, hostname)
debug("Making direct connection to #{hostname}:#{port}")
@_makeConnection(socket, head, port, hostname)
_makeConnection: (browserSocket, head, port, hostname) ->
onSocket = (err, upstreamSocket) =>
debug('received upstreamSocket callback for request %o', { port, hostname, err })
_makeConnection: (socket, head, port, hostname) ->
onConnect = ->
socket.pipe(conn)
conn.pipe(socket)
conn.write(head)
onError = (err) =>
browserSocket.destroy(err)
socket.resume()
if @_onError
@_onError(err, browserSocket, head, port)
conn = new net.Socket()
conn.setNoDelay(true)
if err
return onError(err)
conn.on "error", (err) =>
if @_onError
@_onError(err, socket, head, port)
upstreamSocket.setNoDelay(true)
upstreamSocket.on "error", onError
## compact out hostname when undefined
args = _.compact([port, hostname, onConnect])
conn.connect.apply(conn, args)
browserSocket.pipe(upstreamSocket)
upstreamSocket.pipe(browserSocket)
upstreamSocket.write(head)
# todo: as soon as all requests are intercepted, this can go away since this is just for pass-through
_makeUpstreamProxyConnection: (upstreamProxy, socket, head, toPort, toHostname) ->
debug("making proxied connection to #{toHostname}:#{toPort} with upstream #{upstreamProxy}")
browserSocket.resume()
onUpstreamSock = (err, upstreamSock) ->
if @_onError
if err
return @_onError(err, socket, head, port)
upstreamSock.on "error", (err) =>
@_onError(err, socket, head, port)
if upstreamProxy = getProxyForUrl("https://#{hostname}:#{port}")
# todo: as soon as all requests are intercepted, this can go away since this is just for pass-through
debug("making proxied connection %o", {
host: "#{hostname}:#{port}",
proxy: upstreamProxy,
})
if not upstreamSock
## couldn't establish a proxy connection, fail gracefully
socket.resume()
return socket.destroy()
return agent.httpsAgent.createUpstreamProxyConnection {
proxy: upstreamProxy
href: "https://#{hostname}:#{port}"
uri: {
port
hostname
}
shouldRetry: true
}, onSocket
upstreamSock.setNoDelay(true)
upstreamSock.pipe(socket)
socket.pipe(upstreamSock)
upstreamSock.write(head)
return connect.createRetryingSocket({ port, host: hostname }, onSocket)
socket.resume()
agent.httpsAgent.createProxiedConnection {
proxy: upstreamProxy
href: "https://#{toHostname}:#{toPort}"
uri: {
port: toPort
hostname: toHostname
}
}, onUpstreamSock.bind(@)
_onServerConnectData: (req, socket, head) ->
_onServerConnectData: (req, browserSocket, head) ->
firstBytes = head[0]
makeConnection = (port) =>
debug("Making intercepted connection to %s", port)
@_makeConnection(socket, head, port)
@_makeConnection(browserSocket, head, port)
if firstBytes in SSL_RECORD_TYPES
{hostname} = url.parse("http://#{req.url}")
if firstBytes not in SSL_RECORD_TYPES
## if this isn't an SSL request then go
## ahead and make the connection now
return makeConnection(@_port)
## else spin up the SNI server
{ hostname } = url.parse("http://#{req.url}")
if sslServer = sslServers[hostname]
return makeConnection(sslServer.port)
## only be creating one SSL server per hostname at once
if not sem = sslSemaphores[hostname]
sem = sslSemaphores[hostname] = semaphore(1)
sem.take =>
leave = ->
process.nextTick ->
sem.leave()
if sslServer = sslServers[hostname]
leave()
return makeConnection(sslServer.port)
## only be creating one SSL server per hostname at once
if not sem = sslSemaphores[hostname]
sem = sslSemaphores[hostname] = semaphore(1)
@_getPortFor(hostname)
.then (port) ->
sslServers[hostname] = { port: port }
sem.take =>
leave = ->
process.nextTick ->
sem.leave()
leave()
if sslServer = sslServers[hostname]
leave()
return makeConnection(sslServer.port)
@_getPortFor(hostname)
.then (port) ->
sslServers[hostname] = { port: port }
leave()
makeConnection(port)
else
makeConnection(@_port)
makeConnection(port)
_normalizeKeyAndCert: (certPem, privateKeyPem) ->
return {

2
packages/https-proxy/package.json
View File

@@ -23,7 +23,7 @@
"bin-up": "1.2.0",
"chai": "3.5.0",
"cross-env": "5.2.0",
"@cypress/debugging-proxy": "1.6.0",
"@cypress/debugging-proxy": "2.0.1",
"request": "2.88.0",
"request-promise": "4.2.4",
"sinon": "1.17.7",

4
packages/https-proxy/test/helpers/proxy.coffee
View File

@@ -15,7 +15,7 @@ pipe = (req, res) ->
onConnect = (req, socket, head, proxy) ->
proxy.connect(req, socket, head, {
onDirectConnection: (req, socket, head) ->
req.url is "localhost:8444"
["localhost:8444", "localhost:12344"].includes(req.url)
})
onRequest = (req, res) ->
@@ -64,4 +64,4 @@ module.exports = {
prx.close(resolve)
.then ->
prx.proxy.close()
}
}

55
packages/https-proxy/test/integration/proxy_spec.coffee
View File

@@ -91,8 +91,21 @@ describe "Proxy", ->
.then (html) ->
expect(html).to.include("https server")
it "retries 5 times", ->
@sandbox.spy(net, 'connect')
request({
strictSSL: false
url: "https://localhost:12344"
proxy: "http://localhost:3333"
})
.then ->
throw new Error("should not reach")
.catch ->
expect(net.connect).to.have.callCount(5)
it "closes outgoing connections when client disconnects", ->
@sandbox.spy(net.Socket.prototype, 'connect')
@sandbox.spy(net, 'connect')
request({
strictSSL: false
@@ -104,10 +117,10 @@ describe "Proxy", ->
## ensure client has disconnected
expect(res.socket.destroyed).to.be.true
## ensure the outgoing socket created for this connection was destroyed
socket = net.Socket.prototype.connect.getCalls()
socket = net.connect.getCalls()
.find (call) =>
_.isEqual(call.args.slice(0,2), ["8444", "localhost"])
.thisValue
call.args[0].port == "8444" && call.args[0].host == "localhost"
.returnValue
expect(socket.destroyed).to.be.true
it "can boot the httpServer", ->
@@ -161,7 +174,6 @@ describe "Proxy", ->
context "with an upstream proxy", ->
beforeEach ->
@oldEnv = Object.assign({}, process.env)
process.env.NO_PROXY = ""
process.env.HTTP_PROXY = process.env.HTTPS_PROXY = "http://localhost:9001"
@@ -172,15 +184,16 @@ describe "Proxy", ->
@upstream.start(9001)
it "passes a request to an https server through the upstream", ->
@upstream._onConnect = (domain, port) ->
expect(domain).to.eq('localhost')
expect(port).to.eq('8444')
return true
request({
strictSSL: false
url: "https://localhost:8444/"
proxy: "http://localhost:3333"
}).then (res) =>
expect(@upstream.getRequests()[0]).to.include({
url: 'localhost:8444'
https: true
})
expect(res).to.contain("https server")
it "uses HTTP basic auth when provided", ->
@@ -189,6 +202,11 @@ describe "Proxy", ->
password: 'bar'
})
@upstream._onConnect = (domain, port) ->
expect(domain).to.eq('localhost')
expect(port).to.eq('8444')
return true
process.env.HTTP_PROXY = process.env.HTTPS_PROXY = "http://foo:bar@localhost:9001"
request({
@@ -196,14 +214,10 @@ describe "Proxy", ->
url: "https://localhost:8444/"
proxy: "http://localhost:3333"
}).then (res) =>
expect(@upstream.getRequests()[0]).to.include({
url: 'localhost:8444'
https: true
})
expect(res).to.contain("https server")
it "closes outgoing connections when client disconnects", ->
@sandbox.spy(net.Socket.prototype, 'connect')
@sandbox.spy(net, 'connect')
request({
strictSSL: false
@@ -217,13 +231,10 @@ describe "Proxy", ->
expect(res.socket.destroyed).to.be.true
## ensure the outgoing socket created for this connection was destroyed
socket = net.Socket.prototype.connect.getCalls()
socket = net.connect.getCalls()
.find (call) =>
_.isEqual(call.args[0][0], {
host: 'localhost'
port: 9001
})
.thisValue
call.args[0].port == 9001 && call.args[0].host == "localhost"
.returnValue
new Promise (resolve) ->
socket.on 'close', =>
@@ -232,4 +243,6 @@ describe "Proxy", ->
afterEach ->
@upstream.stop()
Object.assign(process.env, @oldEnv)
delete process.env.HTTP_PROXY
delete process.env.HTTPS_PROXY
delete process.env.NO_PROXY

54
packages/https-proxy/test/unit/server_spec.coffee
View File

@@ -13,6 +13,10 @@ describe "lib/server", ->
Server.create(@ca, @port, options)
afterEach ->
delete process.env.HTTPS_PROXY
delete process.env.NO_PROXY
context "#listen", ->
it "calls options.onUpgrade with req, socket head", ->
onUpgrade = @sandbox.stub()
@@ -34,22 +38,50 @@ describe "lib/server", ->
expect(onRequest).to.be.calledWith(req, res)
it "calls options.onError with err and port", (done) ->
onError = @sandbox.stub()
it "calls options.onError with err and port and destroys the client socket", (done) ->
socket = new EE()
socket.destroy = @sandbox.stub()
head = {}
onError = (err, socket, head, port) ->
expect(err.message).to.eq("connect ECONNREFUSED 127.0.0.1:8444")
expect(socket).to.eq(socket)
expect(head).to.eq(head)
expect(port).to.eq("8444")
expect(socket.destroy).to.be.calledOnce
done()
@setup({onError: onError})
.then (srv) ->
conn = srv._makeDirectConnection({url: "localhost:8444"}, socket, head)
conn.once "error", ->
err = onError.getCall(0).args[0]
expect(err.message).to.eq("connect ECONNREFUSED 127.0.0.1:8444")
expect(onError.getCall(0).args[1]).to.eq(socket)
expect(onError.getCall(0).args[2]).to.eq(head)
expect(onError.getCall(0).args[3]).to.eq("8444")
done()
return
it "with proxied connection calls options.onError with err and port and destroys the client socket", (done) ->
socket = new EE()
socket.destroy = @sandbox.stub()
head = {}
onError = (err, socket, head, port) ->
expect(err.message).to.eq("A connection to the upstream proxy could not be established: connect ECONNREFUSED 127.0.0.1:8444")
expect(socket).to.eq(socket)
expect(head).to.eq(head)
expect(port).to.eq("11111")
expect(socket.destroy).to.be.calledOnce
done()
process.env.HTTPS_PROXY = 'http://localhost:8444'
process.env.NO_PROXY = ''
@setup({onError: onError})
.then (srv) ->
conn = srv._makeDirectConnection({url: "should-not-reach.invalid:11111"}, socket, head)
return