This is needed by attic and it uses internal APIs so it has to be
exported 😢
A better path forward would be to fully implement intersection types
but that is a large change for an edge case feature.
For the longest time, there's been a 'go vet' warning telling us that
when we build a custom http.Client in http_batch_store.go, we were
copying a Mutex down in there. That seems like not what you want to do.
Turns out we should be creating our own http.Transport object and
sharing that, instead of copying the default one every time we build
a new httpBatchStore.
This adds optional fields to structs.
New version: 7.4
To create a struct type with optional fields use types.MakeStructType2
There are some API changes in this commit and there will be some more in followup commits.
Fixes#2327
My mistake was assuming that, when a network read error happens, the
Err field of the resulting net.OpError would directly contain a
unix.Errno. What actually happens is that there's an *os.SyscallError
in there, which is the thing that's actually wrapped around the
unix.Errno. Hence, my `IsConnReset()` function was always returning
false.
This patch has been tested in the field, so I'm confident it will
work to mitigate #3255 in practice.
A connection reset doesn't get caught by io.ReadFull and converted to
an io.ErrUnexpectedEOF like I thought. So, I need to try to type
assert the error I get to *net.OpError and then use that to see if it
was a connection reset.
Workaround for #3255
Apparently, not all failures while reading get translated to
an ErrUnexpectedEOF. In order to figure out what error we see
in practice in s3_table_reader.go, log the type.
This was something that evolved from the way that Dynamo stores
data, and a way to allow clients to make incremental write
progress. We never actually made the clients handle it
properly, though, and so much has changed since we wrote it
that it's only going to be in the way of building something
better.
Fixes#3234
We hit the point of diminishing returns with #3255.
Just do retries with an exponential backoff, logging
failures so we can at least go back in logs and see
how often we see this failure in future, and if we
get storms of retries or not.
Trying to run `ss` in a background process on every request wound up
opening too many FDs. This is because, in order to read stderr and
stdout from the child process, the server process needed to open a
couple of pipes. For every single one. Unless they exited VERY
fast, that means that there'd be many of these child processes alive
at the same time, each requiring 2 FDs -- and this is in addition to
the incoming network connections the server is handling and all its
connections to S3.
Whoops.
This patch only spins up `ss` in the event that a request has already
failed during read. The downside is that we're getting a snapshot from
_after_ the failure. The upside is that it's likely to actually
function! The hope is that the situation that lead to the read failing
persists long enough that we can see it even when sampling just after
the failure. Maybe not, but hopefully it's a start on figuring out
Run a tool called `ss` in the background while making s3 requests. Log the output iff reading from S3 fails in the manner that we're trying to diagnose in #3255
This exports a few functions so that we can pass variables into
the query:
- NewRootQueryObject (was constructQueryType)
- GetTypeName
- GetInputTypeName
- InputToNomsValue (was argToNomsValue)
* Noms type to GraphQL input types
We already had a way to generate GraphQL output types from Noms types
but no way to do input types. Input types are used in arguments
position.
We currently do not do any conversion from the JSON like data in
argument position. We leave it to the users to determine if they want
to convert the data into noms data structures or go to Go structures
or operate directly on the JSON like data.
* Introduce argToNomsValue
Using `marshal.Marshal` is not going to work because structs are done as
`map[string]interface{}` in GraphQL input types.
Instead introduce argToNomsValue which correctly handles
GraphQL -> Noms correctly even for structs.
This is needed because we do not have optional fields in noms structs.
Therefore the noms struct type has the field but the the noms struct may
not have it.
Erik Arvidsson