Files
cypress/packages/https-proxy/lib/server.coffee
Zach Bloomquist c1a345dce2 Improved proxy support (#3531)
* https-proxy: unused file

* server: wrap all https requests that use a proxy

* server: use request lib in ensureUrl if proxy is in use. this makes runs tab work behind a proxy

* electron: pass --proxy-server to app itself, so the embedded github login page works

* cli: first attempt at env vars from windows registry

* cli: api cleanup

* cli: lint

* cli: fix crash on no proxy, add tests

* add desktop-gui watch to terminals.json

* cli: pass along --proxy-source

* electron: pass --proxy-bypass-list too

* server: whitelist proxy* args

* cli: better wording

* desktop-gui: display proxy settings

* extension: force proxy [wip]

* extension: finally, i am victorious over coffeescript

* extension: add -loopback to bypasslist

* extension: revert changes

Revert "extension: force proxy [wip]"

This reverts commit 3ab6ba42a763f25ee65f12eb8b79eb597efc9b11.

* desktop-gui: skip proxysettings if there aren't any

* https-proxy, server: proxy directConnections using https-proxy-agent

* https-agent: pool httpsAgents

* https-proxy: work when they're not on a proxy

* https-proxy: ci - use agent 1.0

* https-proxy: tests

* desktop-gui: hide proxy settings when not using proxy

* https-proxy: pass req through to https-proxy-agent callback

* cli: use get-windows-proxy

* desktop-gui: always show proxy settings

* server: use get-windows-proxy

* electron, server: supply electron proxy config when window launched

* server: fix

* https-proxy: cleanup

* server: clean up ensureUrl

* https-proxy: cleanup

* cli: fix

* cli: fix destructuring

* server: enable ForeverAgent to pool HTTPS/HTTP connections

#3192

* server: updating snapshot

* https-proxy: don't crash, do error if proxy unreachable

* https-proxy:

* get-windows-proxy@1.0.0

* https-proxy: use proxy-from-env to decide on a proxy for a url

* server: fallback to HTTP_PROXY globally if HTTPS_PROXY not set

* server: proxy args test

* cli: add proxy tests

* cli: add test that loadSystemProxySettings is called during download

* cli, server: account for the fact that CI has some proxy vars set

* https-proxy: ""

* cli, https-proxy, server: ""

* desktop-gui: update settings gui

* desktop-gui: cypress tests for proxy settings

* server: strict undefined check

* cli, server: move get-windows-proxy to scope, optionalDeps

* server, cli: use new and improved get-windows-proxy

* cli, server: 1.5.0

* server: re-check for proxy since cli may have failed to load the lib

* server, cli: 1.5.1

* server: NO_PROXY=localhost by default, clean up

* https-proxy: disable Nagle's on proxy sockets

\#3192

* https-proxy: use setNoDelay on upstream, cache https agent

* https-proxy: test basic auth

* https-proxy: add todo: remove this

* server: add custom HTTP(s) Agent implementation w keepalive, tunneling

* server: typescript for agent

* add ts to zunder

* server: more ts

* ts: add missing Agent type declaration

* server: create CombinedAgent

* server: use agent in more places

* ts: more declarations

* server: make script work even if debug port not supplied

* server: begin some testing

* server, ts: agent, tests

* server: test

* server: agent works with websockets now

* server: update snapshot

* server: work out some more bugs with websockets

* server: more websockets

* server: add net_profiler

* https-proxy: fix dangling socket on direct connection

* server: fix potential 'headers have already been sent'

* https-proxy: nab another dangler

* server: update test to expect agent

* https-proxy: fix failing test

* desktop-gui: change on-link

* server: add tests for empty response case

* server: tests

* server: send keep-alive with requests

* server: make net profiler hook on socket.connect

* server: only hook profiler once

* server: update tests, add keep-alive test

* server: only regen headers if needed

* server: move http_overrides into CombinedAgent, make it proxy-proof

for #112

* server: update snapshot

* server: undo

* server: avoid circular dependency

* https-proxy, server: use our Agent instead of https-proxy-agent

* server: add dependency back

* cli: actually use proxy for download

* server, launcher, ts: typescript

* Revert "server, launcher, ts: typescript"

This reverts commit d3f8b8bbb6.

* Revert "Revert "server, launcher, ts: typescript""

This reverts commit 818dfdfd00.

* ts, server: respond to PR

* server, ts: types

* ts: really fix types

* https-proxy, server: export CA from https-proxy

* agent, server, https-proxy: move agent to own package

* agent => networking, move connect into networking

* fix tests

* fix test

* networking: respond to PR changes, add more unit tests

* rename ctx

* networking, ts: add more tests

* server: add ensureUrl tests

* https-proxy: remove https-proxy-agent

* server: use CombinedAgent for API

* server: updates

* add proxy performance tests

* add perf tests to workflow

* circle

* run perf tests with --no-sandbox

* networking, ts: ch-ch-ch-ch-changes

* server, networking: pr changes

* run networking tests in circle

* server: fix performance test

* https-proxy: test that sockets are being closed

* https-proxy: write, not emit

* networking: fix test

* networking: bubble err in connect

* networking: style

* networking: clean p connect error handling

* networking => network

* server: make perf tests really work

* server: really report

* server: use args from browser

* server: use AI to determine max run time

* server: load electron only when needed


Co-authored-by: Brian Mann <brian@cypress.io>
2019-03-31 23:39:10 -04:00

248 lines
6.5 KiB
CoffeeScript

_ = require("lodash")
agent = require("@packages/network").agent
allowDestroy = require("server-destroy-vvo")
debug = require("debug")("cypress:https-proxy")
fs = require("fs-extra")
getProxyForUrl = require("proxy-from-env").getProxyForUrl
https = require("https")
net = require("net")
parse = require("./util/parse")
Promise = require("bluebird")
semaphore = require("semaphore")
url = require("url")
fs = Promise.promisifyAll(fs)
sslServers = {}
sslSemaphores = {}
## https://en.wikipedia.org/wiki/Transport_Layer_Security#TLS_record
SSL_RECORD_TYPES = [
22 ## Handshake
128, 0 ## TODO: what do these unknown types mean?
]
class Server
constructor: (@_ca, @_port) ->
@_onError = null
connect: (req, socket, head, options = {}) ->
## don't buffer writes - thanks a lot, Nagle
## https://github.com/cypress-io/cypress/issues/3192
socket.setNoDelay(true)
if not head or head.length is 0
debug("Writing socket connection headers for URL:", req.url)
socket.once "data", (data) =>
@connect(req, socket, data, options)
socket.write "HTTP/1.1 200 OK\r\n"
if req.headers["proxy-connection"] is "keep-alive"
socket.write("Proxy-Connection: keep-alive\r\n")
socket.write("Connection: keep-alive\r\n")
return socket.write("\r\n")
else
if odc = options.onDirectConnection
## if onDirectConnection return true
## then dont proxy, just pass this through
if odc.call(@, req, socket, head) is true
return @_makeDirectConnection(req, socket, head)
else
debug("Not making direct connection to #{req.url}")
socket.pause()
@_onServerConnectData(req, socket, head)
_onUpgrade: (fn, req, socket, head) ->
if fn
fn.call(@, req, socket, head)
_onRequest: (fn, req, res) ->
hostPort = parse.hostAndPort(req.url, req.headers, 443)
req.url = url.format({
protocol: "https:"
hostname: hostPort.host
port: hostPort.port
}) + req.url
if fn
return fn.call(@, req, res)
req.pipe(request(req.url))
.on "error", ->
res.statusCode = 500
res.end()
.pipe(res)
_upstreamProxyForHostPort: (hostname, port) ->
getProxyForUrl("https://#{hostname}:#{port}")
_makeDirectConnection: (req, socket, head) ->
{ port, hostname } = url.parse("http://#{req.url}")
if upstreamProxy = @_upstreamProxyForHostPort(hostname, port)
return @_makeUpstreamProxyConnection(upstreamProxy, socket, head, port, hostname)
debug("Making direct connection to #{hostname}:#{port}")
@_makeConnection(socket, head, port, hostname)
_makeConnection: (socket, head, port, hostname) ->
onConnect = ->
socket.pipe(conn)
conn.pipe(socket)
conn.write(head)
socket.resume()
conn = new net.Socket()
conn.setNoDelay(true)
conn.on "error", (err) =>
if @_onError
@_onError(err, socket, head, port)
## compact out hostname when undefined
args = _.compact([port, hostname, onConnect])
conn.connect.apply(conn, args)
# todo: as soon as all requests are intercepted, this can go away since this is just for pass-through
_makeUpstreamProxyConnection: (upstreamProxy, socket, head, toPort, toHostname) ->
debug("making proxied connection to #{toHostname}:#{toPort} with upstream #{upstreamProxy}")
onUpstreamSock = (err, upstreamSock) ->
if @_onError
if err
return @_onError(err, socket, head, port)
upstreamSock.on "error", (err) =>
@_onError(err, socket, head, port)
if not upstreamSock
## couldn't establish a proxy connection, fail gracefully
socket.resume()
return socket.destroy()
upstreamSock.setNoDelay(true)
upstreamSock.pipe(socket)
socket.pipe(upstreamSock)
upstreamSock.write(head)
socket.resume()
agent.httpsAgent.createProxiedConnection {
proxy: upstreamProxy
href: "https://#{toHostname}:#{toPort}"
uri: {
port: toPort
hostname: toHostname
}
}, onUpstreamSock.bind(@)
_onServerConnectData: (req, socket, head) ->
firstBytes = head[0]
makeConnection = (port) =>
debug("Making intercepted connection to %s", port)
@_makeConnection(socket, head, port)
if firstBytes in SSL_RECORD_TYPES
{hostname} = url.parse("http://#{req.url}")
if sslServer = sslServers[hostname]
return makeConnection(sslServer.port)
## only be creating one SSL server per hostname at once
if not sem = sslSemaphores[hostname]
sem = sslSemaphores[hostname] = semaphore(1)
sem.take =>
leave = ->
process.nextTick ->
sem.leave()
if sslServer = sslServers[hostname]
leave()
return makeConnection(sslServer.port)
@_getPortFor(hostname)
.then (port) ->
sslServers[hostname] = { port: port }
leave()
makeConnection(port)
else
makeConnection(@_port)
_normalizeKeyAndCert: (certPem, privateKeyPem) ->
return {
key: privateKeyPem
cert: certPem
}
_getCertificatePathsFor: (hostname) ->
@_ca.getCertificateKeysForHostname(hostname)
.spread(@_normalizeKeyAndCert)
_generateMissingCertificates: (hostname) ->
@_ca.generateServerCertificateKeys(hostname)
.spread(@_normalizeKeyAndCert)
_getPortFor: (hostname) ->
@_getCertificatePathsFor(hostname)
.catch (err) =>
@_generateMissingCertificates(hostname)
.then (data = {}) =>
@_sniServer.addContext(hostname, data)
return @_sniPort
listen: (options = {}) ->
new Promise (resolve) =>
@_onError = options.onError
@_sniServer = https.createServer({})
allowDestroy(@_sniServer)
@_sniServer.on "upgrade", @_onUpgrade.bind(@, options.onUpgrade)
@_sniServer.on "request", @_onRequest.bind(@, options.onRequest)
@_sniServer.listen 0, '127.0.0.1', =>
## store the port of our current sniServer
@_sniPort = @_sniServer.address().port
debug("Created SNI HTTPS Proxy on port %s", @_sniPort)
resolve()
close: ->
close = =>
new Promise (resolve) =>
@_sniServer.destroy(resolve)
close()
.finally ->
sslServers = {}
module.exports = {
reset: ->
sslServers = {}
create: (ca, port, options = {}) ->
srv = new Server(ca, port)
srv
.listen(options)
.return(srv)
}