[](https://travis-ci.org/sirupsen/logrus) [](https://godoc.org/github.com/sirupsen/logrus)
+
+Logrus is a structured logger for Go (golang), completely API compatible with
+the standard library logger.
+
+**Logrus is in maintenance-mode.** We will not be introducing new features. It's
+simply too hard to do in a way that won't break many people's projects, which is
+the last thing you want from your Logging library (again...).
+
+This does not mean Logrus is dead. Logrus will continue to be maintained for
+security, (backwards compatible) bug fixes, and performance (where we are
+limited by the interface).
+
+I believe Logrus' biggest contribution is to have played a part in today's
+widespread use of structured logging in Golang. There doesn't seem to be a
+reason to do a major, breaking iteration into Logrus V2, since the fantastic Go
+community has built those independently. Many fantastic alternatives have sprung
+up. Logrus would look like those, had it been re-designed with what we know
+about structured logging in Go today. Check out, for example,
+[Zerolog][zerolog], [Zap][zap], and [Apex][apex].
+
+[zerolog]: https://github.com/rs/zerolog
+[zap]: https://github.com/uber-go/zap
+[apex]: https://github.com/apex/log
+
+**Seeing weird case-sensitive problems?** It's in the past been possible to
+import Logrus as both upper- and lower-case. Due to the Go package environment,
+this caused issues in the community and we needed a standard. Some environments
+experienced problems with the upper-case variant, so the lower-case was decided.
+Everything using `logrus` will need to use the lower-case:
+`github.com/sirupsen/logrus`. Any package that isn't, should be changed.
+
+To fix Glide, see [these
+comments](https://github.com/sirupsen/logrus/issues/553#issuecomment-306591437).
+For an in-depth explanation of the casing issue, see [this
+comment](https://github.com/sirupsen/logrus/issues/570#issuecomment-313933276).
+
+Nicely color-coded in development (when a TTY is attached, otherwise just
+plain text):
+
+
+
+With `log.SetFormatter(&log.JSONFormatter{})`, for easy parsing by logstash
+or Splunk:
+
+```json
+{"animal":"walrus","level":"info","msg":"A group of walrus emerges from the
+ocean","size":10,"time":"2014-03-10 19:57:38.562264131 -0400 EDT"}
+
+{"level":"warning","msg":"The group's number increased tremendously!",
+"number":122,"omg":true,"time":"2014-03-10 19:57:38.562471297 -0400 EDT"}
+
+{"animal":"walrus","level":"info","msg":"A giant walrus appears!",
+"size":10,"time":"2014-03-10 19:57:38.562500591 -0400 EDT"}
+
+{"animal":"walrus","level":"info","msg":"Tremendously sized cow enters the ocean.",
+"size":9,"time":"2014-03-10 19:57:38.562527896 -0400 EDT"}
+
+{"level":"fatal","msg":"The ice breaks!","number":100,"omg":true,
+"time":"2014-03-10 19:57:38.562543128 -0400 EDT"}
+```
+
+With the default `log.SetFormatter(&log.TextFormatter{})` when a TTY is not
+attached, the output is compatible with the
+[logfmt](http://godoc.org/github.com/kr/logfmt) format:
+
+```text
+time="2015-03-26T01:27:38-04:00" level=debug msg="Started observing beach" animal=walrus number=8
+time="2015-03-26T01:27:38-04:00" level=info msg="A group of walrus emerges from the ocean" animal=walrus size=10
+time="2015-03-26T01:27:38-04:00" level=warning msg="The group's number increased tremendously!" number=122 omg=true
+time="2015-03-26T01:27:38-04:00" level=debug msg="Temperature changes" temperature=-4
+time="2015-03-26T01:27:38-04:00" level=panic msg="It's over 9000!" animal=orca size=9009
+time="2015-03-26T01:27:38-04:00" level=fatal msg="The ice breaks!" err=&{0x2082280c0 map[animal:orca size:9009] 2015-03-26 01:27:38.441574009 -0400 EDT panic It's over 9000!} number=100 omg=true
+```
+To ensure this behaviour even if a TTY is attached, set your formatter as follows:
+
+```go
+ log.SetFormatter(&log.TextFormatter{
+ DisableColors: true,
+ FullTimestamp: true,
+ })
+```
+
+#### Logging Method Name
+
+If you wish to add the calling method as a field, instruct the logger via:
+```go
+log.SetReportCaller(true)
+```
+This adds the caller as 'method' like so:
+
+```json
+{"animal":"penguin","level":"fatal","method":"github.com/sirupsen/arcticcreatures.migrate","msg":"a penguin swims by",
+"time":"2014-03-10 19:57:38.562543129 -0400 EDT"}
+```
+
+```text
+time="2015-03-26T01:27:38-04:00" level=fatal method=github.com/sirupsen/arcticcreatures.migrate msg="a penguin swims by" animal=penguin
+```
+Note that this does add measurable overhead - the cost will depend on the version of Go, but is
+between 20 and 40% in recent tests with 1.6 and 1.7. You can validate this in your
+environment via benchmarks:
+```
+go test -bench=.*CallerTracing
+```
+
+
+#### Case-sensitivity
+
+The organization's name was changed to lower-case--and this will not be changed
+back. If you are getting import conflicts due to case sensitivity, please use
+the lower-case import: `github.com/sirupsen/logrus`.
+
+#### Example
+
+The simplest way to use Logrus is simply the package-level exported logger:
+
+```go
+package main
+
+import (
+ log "github.com/sirupsen/logrus"
+)
+
+func main() {
+ log.WithFields(log.Fields{
+ "animal": "walrus",
+ }).Info("A walrus appears")
+}
+```
+
+Note that it's completely api-compatible with the stdlib logger, so you can
+replace your `log` imports everywhere with `log "github.com/sirupsen/logrus"`
+and you'll now have the flexibility of Logrus. You can customize it all you
+want:
+
+```go
+package main
+
+import (
+ "os"
+ log "github.com/sirupsen/logrus"
+)
+
+func init() {
+ // Log as JSON instead of the default ASCII formatter.
+ log.SetFormatter(&log.JSONFormatter{})
+
+ // Output to stdout instead of the default stderr
+ // Can be any io.Writer, see below for File example
+ log.SetOutput(os.Stdout)
+
+ // Only log the warning severity or above.
+ log.SetLevel(log.WarnLevel)
+}
+
+func main() {
+ log.WithFields(log.Fields{
+ "animal": "walrus",
+ "size": 10,
+ }).Info("A group of walrus emerges from the ocean")
+
+ log.WithFields(log.Fields{
+ "omg": true,
+ "number": 122,
+ }).Warn("The group's number increased tremendously!")
+
+ log.WithFields(log.Fields{
+ "omg": true,
+ "number": 100,
+ }).Fatal("The ice breaks!")
+
+ // A common pattern is to re-use fields between logging statements by re-using
+ // the logrus.Entry returned from WithFields()
+ contextLogger := log.WithFields(log.Fields{
+ "common": "this is a common field",
+ "other": "I also should be logged always",
+ })
+
+ contextLogger.Info("I'll be logged with common and other field")
+ contextLogger.Info("Me too")
+}
+```
+
+For more advanced usage such as logging to multiple locations from the same
+application, you can also create an instance of the `logrus` Logger:
+
+```go
+package main
+
+import (
+ "os"
+ "github.com/sirupsen/logrus"
+)
+
+// Create a new instance of the logger. You can have any number of instances.
+var log = logrus.New()
+
+func main() {
+ // The API for setting attributes is a little different than the package level
+ // exported logger. See Godoc.
+ log.Out = os.Stdout
+
+ // You could set this to any `io.Writer` such as a file
+ // file, err := os.OpenFile("logrus.log", os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)
+ // if err == nil {
+ // log.Out = file
+ // } else {
+ // log.Info("Failed to log to file, using default stderr")
+ // }
+
+ log.WithFields(logrus.Fields{
+ "animal": "walrus",
+ "size": 10,
+ }).Info("A group of walrus emerges from the ocean")
+}
+```
+
+#### Fields
+
+Logrus encourages careful, structured logging through logging fields instead of
+long, unparseable error messages. For example, instead of: `log.Fatalf("Failed
+to send event %s to topic %s with key %d")`, you should log the much more
+discoverable:
+
+```go
+log.WithFields(log.Fields{
+ "event": event,
+ "topic": topic,
+ "key": key,
+}).Fatal("Failed to send event")
+```
+
+We've found this API forces you to think about logging in a way that produces
+much more useful logging messages. We've been in countless situations where just
+a single added field to a log statement that was already there would've saved us
+hours. The `WithFields` call is optional.
+
+In general, with Logrus using any of the `printf`-family functions should be
+seen as a hint you should add a field, however, you can still use the
+`printf`-family functions with Logrus.
+
+#### Default Fields
+
+Often it's helpful to have fields _always_ attached to log statements in an
+application or parts of one. For example, you may want to always log the
+`request_id` and `user_ip` in the context of a request. Instead of writing
+`log.WithFields(log.Fields{"request_id": request_id, "user_ip": user_ip})` on
+every line, you can create a `logrus.Entry` to pass around instead:
+
+```go
+requestLogger := log.WithFields(log.Fields{"request_id": request_id, "user_ip": user_ip})
+requestLogger.Info("something happened on that request") # will log request_id and user_ip
+requestLogger.Warn("something not great happened")
+```
+
+#### Hooks
+
+You can add hooks for logging levels. For example to send errors to an exception
+tracking service on `Error`, `Fatal` and `Panic`, info to StatsD or log to
+multiple places simultaneously, e.g. syslog.
+
+Logrus comes with [built-in hooks](hooks/). Add those, or your custom hook, in
+`init`:
+
+```go
+import (
+ log "github.com/sirupsen/logrus"
+ "gopkg.in/gemnasium/logrus-airbrake-hook.v2" // the package is named "airbrake"
+ logrus_syslog "github.com/sirupsen/logrus/hooks/syslog"
+ "log/syslog"
+)
+
+func init() {
+
+ // Use the Airbrake hook to report errors that have Error severity or above to
+ // an exception tracker. You can create custom hooks, see the Hooks section.
+ log.AddHook(airbrake.NewHook(123, "xyz", "production"))
+
+ hook, err := logrus_syslog.NewSyslogHook("udp", "localhost:514", syslog.LOG_INFO, "")
+ if err != nil {
+ log.Error("Unable to connect to local syslog daemon")
+ } else {
+ log.AddHook(hook)
+ }
+}
+```
+Note: Syslog hook also support connecting to local syslog (Ex. "/dev/log" or "/var/run/syslog" or "/var/run/log"). For the detail, please check the [syslog hook README](hooks/syslog/README.md).
+
+A list of currently known service hooks can be found in this wiki [page](https://github.com/sirupsen/logrus/wiki/Hooks)
+
+
+#### Level logging
+
+Logrus has seven logging levels: Trace, Debug, Info, Warning, Error, Fatal and Panic.
+
+```go
+log.Trace("Something very low level.")
+log.Debug("Useful debugging information.")
+log.Info("Something noteworthy happened!")
+log.Warn("You should probably take a look at this.")
+log.Error("Something failed but I'm not quitting.")
+// Calls os.Exit(1) after logging
+log.Fatal("Bye.")
+// Calls panic() after logging
+log.Panic("I'm bailing.")
+```
+
+You can set the logging level on a `Logger`, then it will only log entries with
+that severity or anything above it:
+
+```go
+// Will log anything that is info or above (warn, error, fatal, panic). Default.
+log.SetLevel(log.InfoLevel)
+```
+
+It may be useful to set `log.Level = logrus.DebugLevel` in a debug or verbose
+environment if your application has that.
+
+#### Entries
+
+Besides the fields added with `WithField` or `WithFields` some fields are
+automatically added to all logging events:
+
+1. `time`. The timestamp when the entry was created.
+2. `msg`. The logging message passed to `{Info,Warn,Error,Fatal,Panic}` after
+ the `AddFields` call. E.g. `Failed to send event.`
+3. `level`. The logging level. E.g. `info`.
+
+#### Environments
+
+Logrus has no notion of environment.
+
+If you wish for hooks and formatters to only be used in specific environments,
+you should handle that yourself. For example, if your application has a global
+variable `Environment`, which is a string representation of the environment you
+could do:
+
+```go
+import (
+ log "github.com/sirupsen/logrus"
+)
+
+init() {
+ // do something here to set environment depending on an environment variable
+ // or command-line flag
+ if Environment == "production" {
+ log.SetFormatter(&log.JSONFormatter{})
+ } else {
+ // The TextFormatter is default, you don't actually have to do this.
+ log.SetFormatter(&log.TextFormatter{})
+ }
+}
+```
+
+This configuration is how `logrus` was intended to be used, but JSON in
+production is mostly only useful if you do log aggregation with tools like
+Splunk or Logstash.
+
+#### Formatters
+
+The built-in logging formatters are:
+
+* `logrus.TextFormatter`. Logs the event in colors if stdout is a tty, otherwise
+ without colors.
+ * *Note:* to force colored output when there is no TTY, set the `ForceColors`
+ field to `true`. To force no colored output even if there is a TTY set the
+ `DisableColors` field to `true`. For Windows, see
+ [github.com/mattn/go-colorable](https://github.com/mattn/go-colorable).
+ * When colors are enabled, levels are truncated to 4 characters by default. To disable
+ truncation set the `DisableLevelTruncation` field to `true`.
+ * When outputting to a TTY, it's often helpful to visually scan down a column where all the levels are the same width. Setting the `PadLevelText` field to `true` enables this behavior, by adding padding to the level text.
+ * All options are listed in the [generated docs](https://godoc.org/github.com/sirupsen/logrus#TextFormatter).
+* `logrus.JSONFormatter`. Logs fields as JSON.
+ * All options are listed in the [generated docs](https://godoc.org/github.com/sirupsen/logrus#JSONFormatter).
+
+Third party logging formatters:
+
+* [`FluentdFormatter`](https://github.com/joonix/log). Formats entries that can be parsed by Kubernetes and Google Container Engine.
+* [`GELF`](https://github.com/fabienm/go-logrus-formatters). Formats entries so they comply to Graylog's [GELF 1.1 specification](http://docs.graylog.org/en/2.4/pages/gelf.html).
+* [`logstash`](https://github.com/bshuster-repo/logrus-logstash-hook). Logs fields as [Logstash](http://logstash.net) Events.
+* [`prefixed`](https://github.com/x-cray/logrus-prefixed-formatter). Displays log entry source along with alternative layout.
+* [`zalgo`](https://github.com/aybabtme/logzalgo). Invoking the Power of Zalgo.
+* [`nested-logrus-formatter`](https://github.com/antonfisher/nested-logrus-formatter). Converts logrus fields to a nested structure.
+* [`powerful-logrus-formatter`](https://github.com/zput/zxcTool). get fileName, log's line number and the latest function's name when print log; Sava log to files.
+* [`caption-json-formatter`](https://github.com/nolleh/caption_json_formatter). logrus's message json formatter with human-readable caption added.
+
+You can define your formatter by implementing the `Formatter` interface,
+requiring a `Format` method. `Format` takes an `*Entry`. `entry.Data` is a
+`Fields` type (`map[string]interface{}`) with all your fields as well as the
+default ones (see Entries section above):
+
+```go
+type MyJSONFormatter struct {
+}
+
+log.SetFormatter(new(MyJSONFormatter))
+
+func (f *MyJSONFormatter) Format(entry *Entry) ([]byte, error) {
+ // Note this doesn't include Time, Level and Message which are available on
+ // the Entry. Consult `godoc` on information about those fields or read the
+ // source of the official loggers.
+ serialized, err := json.Marshal(entry.Data)
+ if err != nil {
+ return nil, fmt.Errorf("Failed to marshal fields to JSON, %v", err)
+ }
+ return append(serialized, '\n'), nil
+}
+```
+
+#### Logger as an `io.Writer`
+
+Logrus can be transformed into an `io.Writer`. That writer is the end of an `io.Pipe` and it is your responsibility to close it.
+
+```go
+w := logger.Writer()
+defer w.Close()
+
+srv := http.Server{
+ // create a stdlib log.Logger that writes to
+ // logrus.Logger.
+ ErrorLog: log.New(w, "", 0),
+}
+```
+
+Each line written to that writer will be printed the usual way, using formatters
+and hooks. The level for those entries is `info`.
+
+This means that we can override the standard library logger easily:
+
+```go
+logger := logrus.New()
+logger.Formatter = &logrus.JSONFormatter{}
+
+// Use logrus for standard log output
+// Note that `log` here references stdlib's log
+// Not logrus imported under the name `log`.
+log.SetOutput(logger.Writer())
+```
+
+#### Rotation
+
+Log rotation is not provided with Logrus. Log rotation should be done by an
+external program (like `logrotate(8)`) that can compress and delete old log
+entries. It should not be a feature of the application-level logger.
+
+#### Tools
+
+| Tool | Description |
+| ---- | ----------- |
+|[Logrus Mate](https://github.com/gogap/logrus_mate)|Logrus mate is a tool for Logrus to manage loggers, you can initial logger's level, hook and formatter by config file, the logger will be generated with different configs in different environments.|
+|[Logrus Viper Helper](https://github.com/heirko/go-contrib/tree/master/logrusHelper)|An Helper around Logrus to wrap with spf13/Viper to load configuration with fangs! And to simplify Logrus configuration use some behavior of [Logrus Mate](https://github.com/gogap/logrus_mate). [sample](https://github.com/heirko/iris-contrib/blob/master/middleware/logrus-logger/example) |
+
+#### Testing
+
+Logrus has a built in facility for asserting the presence of log messages. This is implemented through the `test` hook and provides:
+
+* decorators for existing logger (`test.NewLocal` and `test.NewGlobal`) which basically just adds the `test` hook
+* a test logger (`test.NewNullLogger`) that just records log messages (and does not output any):
+
+```go
+import(
+ "github.com/sirupsen/logrus"
+ "github.com/sirupsen/logrus/hooks/test"
+ "github.com/stretchr/testify/assert"
+ "testing"
+)
+
+func TestSomething(t*testing.T){
+ logger, hook := test.NewNullLogger()
+ logger.Error("Helloerror")
+
+ assert.Equal(t, 1, len(hook.Entries))
+ assert.Equal(t, logrus.ErrorLevel, hook.LastEntry().Level)
+ assert.Equal(t, "Helloerror", hook.LastEntry().Message)
+
+ hook.Reset()
+ assert.Nil(t, hook.LastEntry())
+}
+```
+
+#### Fatal handlers
+
+Logrus can register one or more functions that will be called when any `fatal`
+level message is logged. The registered handlers will be executed before
+logrus performs an `os.Exit(1)`. This behavior may be helpful if callers need
+to gracefully shutdown. Unlike a `panic("Something went wrong...")` call which can be intercepted with a deferred `recover` a call to `os.Exit(1)` can not be intercepted.
+
+```
+...
+handler := func() {
+ // gracefully shutdown something...
+}
+logrus.RegisterExitHandler(handler)
+...
+```
+
+#### Thread safety
+
+By default, Logger is protected by a mutex for concurrent writes. The mutex is held when calling hooks and writing logs.
+If you are sure such locking is not needed, you can call logger.SetNoLock() to disable the locking.
+
+Situation when locking is not needed includes:
+
+* You have no hooks registered, or hooks calling is already thread-safe.
+
+* Writing to logger.Out is already thread-safe, for example:
+
+ 1) logger.Out is protected by locks.
+
+ 2) logger.Out is an os.File handler opened with `O_APPEND` flag, and every write is smaller than 4k. (This allows multi-thread/multi-process writing)
+
+ (Refer to http://www.notthewizard.com/2014/06/17/are-files-appends-really-atomic/)
diff --git a/vendor/github.com/Sirupsen/logrus/alt_exit.go b/vendor/github.com/Sirupsen/logrus/alt_exit.go
new file mode 100644
index 0000000..8fd189e
--- /dev/null
+++ b/vendor/github.com/Sirupsen/logrus/alt_exit.go
@@ -0,0 +1,76 @@
+package logrus
+
+// The following code was sourced and modified from the
+// https://github.com/tebeka/atexit package governed by the following license:
+//
+// Copyright (c) 2012 Miki Tebeka 
+[](https://semaphoreci.com/joeybloggs/locales)
+[](https://goreportcard.com/report/github.com/go-playground/locales)
+[](https://godoc.org/github.com/go-playground/locales)
+
+[](https://gitter.im/go-playground/locales?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)
+
+Locales is a set of locales generated from the [Unicode CLDR Project](http://cldr.unicode.org/) which can be used independently or within
+an i18n package; these were built for use with, but not exclusive to, [Universal Translator](https://github.com/go-playground/universal-translator).
+
+Features
+--------
+- [x] Rules generated from the latest [CLDR](http://cldr.unicode.org/index/downloads) data, v31.0.1
+- [x] Contains Cardinal, Ordinal and Range Plural Rules
+- [x] Contains Month, Weekday and Timezone translations built in
+- [x] Contains Date & Time formatting functions
+- [x] Contains Number, Currency, Accounting and Percent formatting functions
+- [x] Supports the "Gregorian" calendar only ( my time isn't unlimited, had to draw the line somewhere )
+
+Full Tests
+--------------------
+I could sure use your help adding tests for every locale, it is a huge undertaking and I just don't have the free time to do it all at the moment;
+any help would be **greatly appreciated!!!!** please see [issue](https://github.com/go-playground/locales/issues/1) for details.
+
+Installation
+-----------
+
+Use go get
+
+```shell
+go get github.com/go-playground/locales
+```
+
+NOTES
+--------
+You'll notice most return types are []byte, this is because most of the time the results will be concatenated with a larger body
+of text and can avoid some allocations if already appending to a byte array, otherwise just cast as string.
+
+Usage
+-------
+```go
+package main
+
+import (
+ "fmt"
+ "time"
+
+ "github.com/go-playground/locales/currency"
+ "github.com/go-playground/locales/en_CA"
+)
+
+func main() {
+
+ loc, _ := time.LoadLocation("America/Toronto")
+ datetime := time.Date(2016, 02, 03, 9, 0, 1, 0, loc)
+
+ l := en_CA.New()
+
+ // Dates
+ fmt.Println(l.FmtDateFull(datetime))
+ fmt.Println(l.FmtDateLong(datetime))
+ fmt.Println(l.FmtDateMedium(datetime))
+ fmt.Println(l.FmtDateShort(datetime))
+
+ // Times
+ fmt.Println(l.FmtTimeFull(datetime))
+ fmt.Println(l.FmtTimeLong(datetime))
+ fmt.Println(l.FmtTimeMedium(datetime))
+ fmt.Println(l.FmtTimeShort(datetime))
+
+ // Months Wide
+ fmt.Println(l.MonthWide(time.January))
+ fmt.Println(l.MonthWide(time.February))
+ fmt.Println(l.MonthWide(time.March))
+ // ...
+
+ // Months Abbreviated
+ fmt.Println(l.MonthAbbreviated(time.January))
+ fmt.Println(l.MonthAbbreviated(time.February))
+ fmt.Println(l.MonthAbbreviated(time.March))
+ // ...
+
+ // Months Narrow
+ fmt.Println(l.MonthNarrow(time.January))
+ fmt.Println(l.MonthNarrow(time.February))
+ fmt.Println(l.MonthNarrow(time.March))
+ // ...
+
+ // Weekdays Wide
+ fmt.Println(l.WeekdayWide(time.Sunday))
+ fmt.Println(l.WeekdayWide(time.Monday))
+ fmt.Println(l.WeekdayWide(time.Tuesday))
+ // ...
+
+ // Weekdays Abbreviated
+ fmt.Println(l.WeekdayAbbreviated(time.Sunday))
+ fmt.Println(l.WeekdayAbbreviated(time.Monday))
+ fmt.Println(l.WeekdayAbbreviated(time.Tuesday))
+ // ...
+
+ // Weekdays Short
+ fmt.Println(l.WeekdayShort(time.Sunday))
+ fmt.Println(l.WeekdayShort(time.Monday))
+ fmt.Println(l.WeekdayShort(time.Tuesday))
+ // ...
+
+ // Weekdays Narrow
+ fmt.Println(l.WeekdayNarrow(time.Sunday))
+ fmt.Println(l.WeekdayNarrow(time.Monday))
+ fmt.Println(l.WeekdayNarrow(time.Tuesday))
+ // ...
+
+ var f64 float64
+
+ f64 = -10356.4523
+
+ // Number
+ fmt.Println(l.FmtNumber(f64, 2))
+
+ // Currency
+ fmt.Println(l.FmtCurrency(f64, 2, currency.CAD))
+ fmt.Println(l.FmtCurrency(f64, 2, currency.USD))
+
+ // Accounting
+ fmt.Println(l.FmtAccounting(f64, 2, currency.CAD))
+ fmt.Println(l.FmtAccounting(f64, 2, currency.USD))
+
+ f64 = 78.12
+
+ // Percent
+ fmt.Println(l.FmtPercent(f64, 0))
+
+ // Plural Rules for locale, so you know what rules you must cover
+ fmt.Println(l.PluralsCardinal())
+ fmt.Println(l.PluralsOrdinal())
+
+ // Cardinal Plural Rules
+ fmt.Println(l.CardinalPluralRule(1, 0))
+ fmt.Println(l.CardinalPluralRule(1.0, 0))
+ fmt.Println(l.CardinalPluralRule(1.0, 1))
+ fmt.Println(l.CardinalPluralRule(3, 0))
+
+ // Ordinal Plural Rules
+ fmt.Println(l.OrdinalPluralRule(21, 0)) // 21st
+ fmt.Println(l.OrdinalPluralRule(22, 0)) // 22nd
+ fmt.Println(l.OrdinalPluralRule(33, 0)) // 33rd
+ fmt.Println(l.OrdinalPluralRule(34, 0)) // 34th
+
+ // Range Plural Rules
+ fmt.Println(l.RangePluralRule(1, 0, 1, 0)) // 1-1
+ fmt.Println(l.RangePluralRule(1, 0, 2, 0)) // 1-2
+ fmt.Println(l.RangePluralRule(5, 0, 8, 0)) // 5-8
+}
+```
+
+NOTES:
+-------
+These rules were generated from the [Unicode CLDR Project](http://cldr.unicode.org/), if you encounter any issues
+I strongly encourage contributing to the CLDR project to get the locale information corrected and the next time
+these locales are regenerated the fix will come with.
+
+I do however realize that time constraints are often important and so there are two options:
+
+1. Create your own locale, copy, paste and modify, and ensure it complies with the `Translator` interface.
+2. Add an exception in the locale generation code directly and once regenerated, fix will be in place.
+
+Please to not make fixes inside the locale files, they WILL get overwritten when the locales are regenerated.
+
+License
+------
+Distributed under MIT License, please see license file in code for more details.
diff --git a/vendor/github.com/go-playground/locales/currency/currency.go b/vendor/github.com/go-playground/locales/currency/currency.go
new file mode 100644
index 0000000..cdaba59
--- /dev/null
+++ b/vendor/github.com/go-playground/locales/currency/currency.go
@@ -0,0 +1,308 @@
+package currency
+
+// Type is the currency type associated with the locales currency enum
+type Type int
+
+// locale currencies
+const (
+ ADP Type = iota
+ AED
+ AFA
+ AFN
+ ALK
+ ALL
+ AMD
+ ANG
+ AOA
+ AOK
+ AON
+ AOR
+ ARA
+ ARL
+ ARM
+ ARP
+ ARS
+ ATS
+ AUD
+ AWG
+ AZM
+ AZN
+ BAD
+ BAM
+ BAN
+ BBD
+ BDT
+ BEC
+ BEF
+ BEL
+ BGL
+ BGM
+ BGN
+ BGO
+ BHD
+ BIF
+ BMD
+ BND
+ BOB
+ BOL
+ BOP
+ BOV
+ BRB
+ BRC
+ BRE
+ BRL
+ BRN
+ BRR
+ BRZ
+ BSD
+ BTN
+ BUK
+ BWP
+ BYB
+ BYN
+ BYR
+ BZD
+ CAD
+ CDF
+ CHE
+ CHF
+ CHW
+ CLE
+ CLF
+ CLP
+ CNH
+ CNX
+ CNY
+ COP
+ COU
+ CRC
+ CSD
+ CSK
+ CUC
+ CUP
+ CVE
+ CYP
+ CZK
+ DDM
+ DEM
+ DJF
+ DKK
+ DOP
+ DZD
+ ECS
+ ECV
+ EEK
+ EGP
+ ERN
+ ESA
+ ESB
+ ESP
+ ETB
+ EUR
+ FIM
+ FJD
+ FKP
+ FRF
+ GBP
+ GEK
+ GEL
+ GHC
+ GHS
+ GIP
+ GMD
+ GNF
+ GNS
+ GQE
+ GRD
+ GTQ
+ GWE
+ GWP
+ GYD
+ HKD
+ HNL
+ HRD
+ HRK
+ HTG
+ HUF
+ IDR
+ IEP
+ ILP
+ ILR
+ ILS
+ INR
+ IQD
+ IRR
+ ISJ
+ ISK
+ ITL
+ JMD
+ JOD
+ JPY
+ KES
+ KGS
+ KHR
+ KMF
+ KPW
+ KRH
+ KRO
+ KRW
+ KWD
+ KYD
+ KZT
+ LAK
+ LBP
+ LKR
+ LRD
+ LSL
+ LTL
+ LTT
+ LUC
+ LUF
+ LUL
+ LVL
+ LVR
+ LYD
+ MAD
+ MAF
+ MCF
+ MDC
+ MDL
+ MGA
+ MGF
+ MKD
+ MKN
+ MLF
+ MMK
+ MNT
+ MOP
+ MRO
+ MTL
+ MTP
+ MUR
+ MVP
+ MVR
+ MWK
+ MXN
+ MXP
+ MXV
+ MYR
+ MZE
+ MZM
+ MZN
+ NAD
+ NGN
+ NIC
+ NIO
+ NLG
+ NOK
+ NPR
+ NZD
+ OMR
+ PAB
+ PEI
+ PEN
+ PES
+ PGK
+ PHP
+ PKR
+ PLN
+ PLZ
+ PTE
+ PYG
+ QAR
+ RHD
+ ROL
+ RON
+ RSD
+ RUB
+ RUR
+ RWF
+ SAR
+ SBD
+ SCR
+ SDD
+ SDG
+ SDP
+ SEK
+ SGD
+ SHP
+ SIT
+ SKK
+ SLL
+ SOS
+ SRD
+ SRG
+ SSP
+ STD
+ STN
+ SUR
+ SVC
+ SYP
+ SZL
+ THB
+ TJR
+ TJS
+ TMM
+ TMT
+ TND
+ TOP
+ TPE
+ TRL
+ TRY
+ TTD
+ TWD
+ TZS
+ UAH
+ UAK
+ UGS
+ UGX
+ USD
+ USN
+ USS
+ UYI
+ UYP
+ UYU
+ UZS
+ VEB
+ VEF
+ VND
+ VNN
+ VUV
+ WST
+ XAF
+ XAG
+ XAU
+ XBA
+ XBB
+ XBC
+ XBD
+ XCD
+ XDR
+ XEU
+ XFO
+ XFU
+ XOF
+ XPD
+ XPF
+ XPT
+ XRE
+ XSU
+ XTS
+ XUA
+ XXX
+ YDD
+ YER
+ YUD
+ YUM
+ YUN
+ YUR
+ ZAL
+ ZAR
+ ZMK
+ ZMW
+ ZRN
+ ZRZ
+ ZWD
+ ZWL
+ ZWR
+)
diff --git a/vendor/github.com/go-playground/locales/logo.png b/vendor/github.com/go-playground/locales/logo.png
new file mode 100644
index 0000000..3038276
Binary files /dev/null and b/vendor/github.com/go-playground/locales/logo.png differ
diff --git a/vendor/github.com/go-playground/locales/rules.go b/vendor/github.com/go-playground/locales/rules.go
new file mode 100644
index 0000000..9202900
--- /dev/null
+++ b/vendor/github.com/go-playground/locales/rules.go
@@ -0,0 +1,293 @@
+package locales
+
+import (
+ "strconv"
+ "time"
+
+ "github.com/go-playground/locales/currency"
+)
+
+// // ErrBadNumberValue is returned when the number passed for
+// // plural rule determination cannot be parsed
+// type ErrBadNumberValue struct {
+// NumberValue string
+// InnerError error
+// }
+
+// // Error returns ErrBadNumberValue error string
+// func (e *ErrBadNumberValue) Error() string {
+// return fmt.Sprintf("Invalid Number Value '%s' %s", e.NumberValue, e.InnerError)
+// }
+
+// var _ error = new(ErrBadNumberValue)
+
+// PluralRule denotes the type of plural rules
+type PluralRule int
+
+// PluralRule's
+const (
+ PluralRuleUnknown PluralRule = iota
+ PluralRuleZero // zero
+ PluralRuleOne // one - singular
+ PluralRuleTwo // two - dual
+ PluralRuleFew // few - paucal
+ PluralRuleMany // many - also used for fractions if they have a separate class
+ PluralRuleOther // other - required—general plural form—also used if the language only has a single form
+)
+
+const (
+ pluralsString = "UnknownZeroOneTwoFewManyOther"
+)
+
+// Translator encapsulates an instance of a locale
+// NOTE: some values are returned as a []byte just in case the caller
+// wishes to add more and can help avoid allocations; otherwise just cast as string
+type Translator interface {
+
+ // The following Functions are for overriding, debugging or developing
+ // with a Translator Locale
+
+ // Locale returns the string value of the translator
+ Locale() string
+
+ // returns an array of cardinal plural rules associated
+ // with this translator
+ PluralsCardinal() []PluralRule
+
+ // returns an array of ordinal plural rules associated
+ // with this translator
+ PluralsOrdinal() []PluralRule
+
+ // returns an array of range plural rules associated
+ // with this translator
+ PluralsRange() []PluralRule
+
+ // returns the cardinal PluralRule given 'num' and digits/precision of 'v' for locale
+ CardinalPluralRule(num float64, v uint64) PluralRule
+
+ // returns the ordinal PluralRule given 'num' and digits/precision of 'v' for locale
+ OrdinalPluralRule(num float64, v uint64) PluralRule
+
+ // returns the ordinal PluralRule given 'num1', 'num2' and digits/precision of 'v1' and 'v2' for locale
+ RangePluralRule(num1 float64, v1 uint64, num2 float64, v2 uint64) PluralRule
+
+ // returns the locales abbreviated month given the 'month' provided
+ MonthAbbreviated(month time.Month) string
+
+ // returns the locales abbreviated months
+ MonthsAbbreviated() []string
+
+ // returns the locales narrow month given the 'month' provided
+ MonthNarrow(month time.Month) string
+
+ // returns the locales narrow months
+ MonthsNarrow() []string
+
+ // returns the locales wide month given the 'month' provided
+ MonthWide(month time.Month) string
+
+ // returns the locales wide months
+ MonthsWide() []string
+
+ // returns the locales abbreviated weekday given the 'weekday' provided
+ WeekdayAbbreviated(weekday time.Weekday) string
+
+ // returns the locales abbreviated weekdays
+ WeekdaysAbbreviated() []string
+
+ // returns the locales narrow weekday given the 'weekday' provided
+ WeekdayNarrow(weekday time.Weekday) string
+
+ // WeekdaysNarrowreturns the locales narrow weekdays
+ WeekdaysNarrow() []string
+
+ // returns the locales short weekday given the 'weekday' provided
+ WeekdayShort(weekday time.Weekday) string
+
+ // returns the locales short weekdays
+ WeekdaysShort() []string
+
+ // returns the locales wide weekday given the 'weekday' provided
+ WeekdayWide(weekday time.Weekday) string
+
+ // returns the locales wide weekdays
+ WeekdaysWide() []string
+
+ // The following Functions are common Formatting functionsfor the Translator's Locale
+
+ // returns 'num' with digits/precision of 'v' for locale and handles both Whole and Real numbers based on 'v'
+ FmtNumber(num float64, v uint64) string
+
+ // returns 'num' with digits/precision of 'v' for locale and handles both Whole and Real numbers based on 'v'
+ // NOTE: 'num' passed into FmtPercent is assumed to be in percent already
+ FmtPercent(num float64, v uint64) string
+
+ // returns the currency representation of 'num' with digits/precision of 'v' for locale
+ FmtCurrency(num float64, v uint64, currency currency.Type) string
+
+ // returns the currency representation of 'num' with digits/precision of 'v' for locale
+ // in accounting notation.
+ FmtAccounting(num float64, v uint64, currency currency.Type) string
+
+ // returns the short date representation of 't' for locale
+ FmtDateShort(t time.Time) string
+
+ // returns the medium date representation of 't' for locale
+ FmtDateMedium(t time.Time) string
+
+ // returns the long date representation of 't' for locale
+ FmtDateLong(t time.Time) string
+
+ // returns the full date representation of 't' for locale
+ FmtDateFull(t time.Time) string
+
+ // returns the short time representation of 't' for locale
+ FmtTimeShort(t time.Time) string
+
+ // returns the medium time representation of 't' for locale
+ FmtTimeMedium(t time.Time) string
+
+ // returns the long time representation of 't' for locale
+ FmtTimeLong(t time.Time) string
+
+ // returns the full time representation of 't' for locale
+ FmtTimeFull(t time.Time) string
+}
+
+// String returns the string value of PluralRule
+func (p PluralRule) String() string {
+
+ switch p {
+ case PluralRuleZero:
+ return pluralsString[7:11]
+ case PluralRuleOne:
+ return pluralsString[11:14]
+ case PluralRuleTwo:
+ return pluralsString[14:17]
+ case PluralRuleFew:
+ return pluralsString[17:20]
+ case PluralRuleMany:
+ return pluralsString[20:24]
+ case PluralRuleOther:
+ return pluralsString[24:]
+ default:
+ return pluralsString[:7]
+ }
+}
+
+//
+// Precision Notes:
+//
+// must specify a precision >= 0, and here is why https://play.golang.org/p/LyL90U0Vyh
+//
+// v := float64(3.141)
+// i := float64(int64(v))
+//
+// fmt.Println(v - i)
+//
+// or
+//
+// s := strconv.FormatFloat(v-i, 'f', -1, 64)
+// fmt.Println(s)
+//
+// these will not print what you'd expect: 0.14100000000000001
+// and so this library requires a precision to be specified, or
+// inaccurate plural rules could be applied.
+//
+//
+//
+// n - absolute value of the source number (integer and decimals).
+// i - integer digits of n.
+// v - number of visible fraction digits in n, with trailing zeros.
+// w - number of visible fraction digits in n, without trailing zeros.
+// f - visible fractional digits in n, with trailing zeros.
+// t - visible fractional digits in n, without trailing zeros.
+//
+//
+// Func(num float64, v uint64) // v = digits/precision and prevents -1 as a special case as this can lead to very unexpected behaviour, see precision note's above.
+//
+// n := math.Abs(num)
+// i := int64(n)
+// v := v
+//
+//
+// w := strconv.FormatFloat(num-float64(i), 'f', int(v), 64) // then parse backwards on string until no more zero's....
+// f := strconv.FormatFloat(n, 'f', int(v), 64) // then turn everything after decimal into an int64
+// t := strconv.FormatFloat(n, 'f', int(v), 64) // then parse backwards on string until no more zero's....
+//
+//
+//
+// General Inclusion Rules
+// - v will always be available inherently
+// - all require n
+// - w requires i
+//
+
+// W returns the number of visible fraction digits in N, without trailing zeros.
+func W(n float64, v uint64) (w int64) {
+
+ s := strconv.FormatFloat(n-float64(int64(n)), 'f', int(v), 64)
+
+ // with either be '0' or '0.xxxx', so if 1 then w will be zero
+ // otherwise need to parse
+ if len(s) != 1 {
+
+ s = s[2:]
+ end := len(s) + 1
+
+ for i := end; i >= 0; i-- {
+ if s[i] != '0' {
+ end = i + 1
+ break
+ }
+ }
+
+ w = int64(len(s[:end]))
+ }
+
+ return
+}
+
+// F returns the visible fractional digits in N, with trailing zeros.
+func F(n float64, v uint64) (f int64) {
+
+ s := strconv.FormatFloat(n-float64(int64(n)), 'f', int(v), 64)
+
+ // with either be '0' or '0.xxxx', so if 1 then f will be zero
+ // otherwise need to parse
+ if len(s) != 1 {
+
+ // ignoring error, because it can't fail as we generated
+ // the string internally from a real number
+ f, _ = strconv.ParseInt(s[2:], 10, 64)
+ }
+
+ return
+}
+
+// T returns the visible fractional digits in N, without trailing zeros.
+func T(n float64, v uint64) (t int64) {
+
+ s := strconv.FormatFloat(n-float64(int64(n)), 'f', int(v), 64)
+
+ // with either be '0' or '0.xxxx', so if 1 then t will be zero
+ // otherwise need to parse
+ if len(s) != 1 {
+
+ s = s[2:]
+ end := len(s) + 1
+
+ for i := end; i >= 0; i-- {
+ if s[i] != '0' {
+ end = i + 1
+ break
+ }
+ }
+
+ // ignoring error, because it can't fail as we generated
+ // the string internally from a real number
+ t, _ = strconv.ParseInt(s[:end], 10, 64)
+ }
+
+ return
+}
diff --git a/vendor/github.com/go-playground/universal-translator/.gitignore b/vendor/github.com/go-playground/universal-translator/.gitignore
new file mode 100644
index 0000000..2661785
--- /dev/null
+++ b/vendor/github.com/go-playground/universal-translator/.gitignore
@@ -0,0 +1,24 @@
+# Compiled Object files, Static and Dynamic libs (Shared Objects)
+*.o
+*.a
+*.so
+
+# Folders
+_obj
+_test
+
+# Architecture specific extensions/prefixes
+*.[568vq]
+[568vq].out
+
+*.cgo1.go
+*.cgo2.c
+_cgo_defun.c
+_cgo_gotypes.go
+_cgo_export.*
+
+_testmain.go
+
+*.exe
+*.test
+*.prof
\ No newline at end of file
diff --git a/vendor/github.com/go-playground/universal-translator/LICENSE b/vendor/github.com/go-playground/universal-translator/LICENSE
new file mode 100644
index 0000000..8d8aba1
--- /dev/null
+++ b/vendor/github.com/go-playground/universal-translator/LICENSE
@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2016 Go Playground
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
diff --git a/vendor/github.com/go-playground/universal-translator/README.md b/vendor/github.com/go-playground/universal-translator/README.md
new file mode 100644
index 0000000..24aef15
--- /dev/null
+++ b/vendor/github.com/go-playground/universal-translator/README.md
@@ -0,0 +1,90 @@
+## universal-translator
+
+
+[](https://semaphoreci.com/joeybloggs/universal-translator)
+[](https://coveralls.io/github/go-playground/universal-translator)
+[](https://goreportcard.com/report/github.com/go-playground/universal-translator)
+[](https://godoc.org/github.com/go-playground/universal-translator)
+
+[](https://gitter.im/go-playground/universal-translator?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)
+
+Universal Translator is an i18n Translator for Go/Golang using CLDR data + pluralization rules
+
+Why another i18n library?
+--------------------------
+Because none of the plural rules seem to be correct out there, including the previous implementation of this package,
+so I took it upon myself to create [locales](https://github.com/go-playground/locales) for everyone to use; this package
+is a thin wrapper around [locales](https://github.com/go-playground/locales) in order to store and translate text for
+use in your applications.
+
+Features
+--------
+- [x] Rules generated from the [CLDR](http://cldr.unicode.org/index/downloads) data, v30.0.3
+- [x] Contains Cardinal, Ordinal and Range Plural Rules
+- [x] Contains Month, Weekday and Timezone translations built in
+- [x] Contains Date & Time formatting functions
+- [x] Contains Number, Currency, Accounting and Percent formatting functions
+- [x] Supports the "Gregorian" calendar only ( my time isn't unlimited, had to draw the line somewhere )
+- [x] Support loading translations from files
+- [x] Exporting translations to file(s), mainly for getting them professionally translated
+- [ ] Code Generation for translation files -> Go code.. i.e. after it has been professionally translated
+- [ ] Tests for all languages, I need help with this, please see [here](https://github.com/go-playground/locales/issues/1)
+
+Installation
+-----------
+
+Use go get
+
+```shell
+go get github.com/go-playground/universal-translator
+```
+
+Usage & Documentation
+-------
+
+Please see https://godoc.org/github.com/go-playground/universal-translator for usage docs
+
+##### Examples:
+
+- [Basic](https://github.com/go-playground/universal-translator/tree/master/examples/basic)
+- [Full - no files](https://github.com/go-playground/universal-translator/tree/master/examples/full-no-files)
+- [Full - with files](https://github.com/go-playground/universal-translator/tree/master/examples/full-with-files)
+
+File formatting
+--------------
+All types, Plain substitution, Cardinal, Ordinal and Range translations can all be contained withing the same file(s);
+they are only separated for easy viewing.
+
+##### Examples:
+
+- [Formats](https://github.com/go-playground/universal-translator/tree/master/examples/file-formats)
+
+##### Basic Makeup
+NOTE: not all fields are needed for all translation types, see [examples](https://github.com/go-playground/universal-translator/tree/master/examples/file-formats)
+```json
+{
+ "locale": "en",
+ "key": "days-left",
+ "trans": "You have {0} day left.",
+ "type": "Cardinal",
+ "rule": "One",
+ "override": false
+}
+```
+|Field|Description|
+|---|---|
+|locale|The locale for which the translation is for.|
+|key|The translation key that will be used to store and lookup each translation; normally it is a string or integer.|
+|trans|The actual translation text.|
+|type|The type of translation Cardinal, Ordinal, Range or "" for a plain substitution(not required to be defined if plain used)|
+|rule|The plural rule for which the translation is for eg. One, Two, Few, Many or Other.(not required to be defined if plain used)|
+|override|If you wish to override an existing translation that has already been registered, set this to 'true'. 99% of the time there is no need to define it.|
+
+Help With Tests
+---------------
+To anyone interesting in helping or contributing, I sure could use some help creating tests for each language.
+Please see issue [here](https://github.com/go-playground/locales/issues/1) for details.
+
+License
+------
+Distributed under MIT License, please see license file in code for more details.
diff --git a/vendor/github.com/go-playground/universal-translator/errors.go b/vendor/github.com/go-playground/universal-translator/errors.go
new file mode 100644
index 0000000..38b163b
--- /dev/null
+++ b/vendor/github.com/go-playground/universal-translator/errors.go
@@ -0,0 +1,148 @@
+package ut
+
+import (
+ "errors"
+ "fmt"
+
+ "github.com/go-playground/locales"
+)
+
+var (
+ // ErrUnknowTranslation indicates the translation could not be found
+ ErrUnknowTranslation = errors.New("Unknown Translation")
+)
+
+var _ error = new(ErrConflictingTranslation)
+var _ error = new(ErrRangeTranslation)
+var _ error = new(ErrOrdinalTranslation)
+var _ error = new(ErrCardinalTranslation)
+var _ error = new(ErrMissingPluralTranslation)
+var _ error = new(ErrExistingTranslator)
+
+// ErrExistingTranslator is the error representing a conflicting translator
+type ErrExistingTranslator struct {
+ locale string
+}
+
+// Error returns ErrExistingTranslator's internal error text
+func (e *ErrExistingTranslator) Error() string {
+ return fmt.Sprintf("error: conflicting translator for locale '%s'", e.locale)
+}
+
+// ErrConflictingTranslation is the error representing a conflicting translation
+type ErrConflictingTranslation struct {
+ locale string
+ key interface{}
+ rule locales.PluralRule
+ text string
+}
+
+// Error returns ErrConflictingTranslation's internal error text
+func (e *ErrConflictingTranslation) Error() string {
+
+ if _, ok := e.key.(string); !ok {
+ return fmt.Sprintf("error: conflicting key '%#v' rule '%s' with text '%s' for locale '%s', value being ignored", e.key, e.rule, e.text, e.locale)
+ }
+
+ return fmt.Sprintf("error: conflicting key '%s' rule '%s' with text '%s' for locale '%s', value being ignored", e.key, e.rule, e.text, e.locale)
+}
+
+// ErrRangeTranslation is the error representing a range translation error
+type ErrRangeTranslation struct {
+ text string
+}
+
+// Error returns ErrRangeTranslation's internal error text
+func (e *ErrRangeTranslation) Error() string {
+ return e.text
+}
+
+// ErrOrdinalTranslation is the error representing an ordinal translation error
+type ErrOrdinalTranslation struct {
+ text string
+}
+
+// Error returns ErrOrdinalTranslation's internal error text
+func (e *ErrOrdinalTranslation) Error() string {
+ return e.text
+}
+
+// ErrCardinalTranslation is the error representing a cardinal translation error
+type ErrCardinalTranslation struct {
+ text string
+}
+
+// Error returns ErrCardinalTranslation's internal error text
+func (e *ErrCardinalTranslation) Error() string {
+ return e.text
+}
+
+// ErrMissingPluralTranslation is the error signifying a missing translation given
+// the locales plural rules.
+type ErrMissingPluralTranslation struct {
+ locale string
+ key interface{}
+ rule locales.PluralRule
+ translationType string
+}
+
+// Error returns ErrMissingPluralTranslation's internal error text
+func (e *ErrMissingPluralTranslation) Error() string {
+
+ if _, ok := e.key.(string); !ok {
+ return fmt.Sprintf("error: missing '%s' plural rule '%s' for translation with key '%#v' and locale '%s'", e.translationType, e.rule, e.key, e.locale)
+ }
+
+ return fmt.Sprintf("error: missing '%s' plural rule '%s' for translation with key '%s' and locale '%s'", e.translationType, e.rule, e.key, e.locale)
+}
+
+// ErrMissingBracket is the error representing a missing bracket in a translation
+// eg. This is a {0 <-- missing ending '}'
+type ErrMissingBracket struct {
+ locale string
+ key interface{}
+ text string
+}
+
+// Error returns ErrMissingBracket error message
+func (e *ErrMissingBracket) Error() string {
+ return fmt.Sprintf("error: missing bracket '{}', in translation. locale: '%s' key: '%v' text: '%s'", e.locale, e.key, e.text)
+}
+
+// ErrBadParamSyntax is the error representing a bad parameter definition in a translation
+// eg. This is a {must-be-int}
+type ErrBadParamSyntax struct {
+ locale string
+ param string
+ key interface{}
+ text string
+}
+
+// Error returns ErrBadParamSyntax error message
+func (e *ErrBadParamSyntax) Error() string {
+ return fmt.Sprintf("error: bad parameter syntax, missing parameter '%s' in translation. locale: '%s' key: '%v' text: '%s'", e.param, e.locale, e.key, e.text)
+}
+
+// import/export errors
+
+// ErrMissingLocale is the error representing an expected locale that could
+// not be found aka locale not registered with the UniversalTranslator Instance
+type ErrMissingLocale struct {
+ locale string
+}
+
+// Error returns ErrMissingLocale's internal error text
+func (e *ErrMissingLocale) Error() string {
+ return fmt.Sprintf("error: locale '%s' not registered.", e.locale)
+}
+
+// ErrBadPluralDefinition is the error representing an incorrect plural definition
+// usually found within translations defined within files during the import process.
+type ErrBadPluralDefinition struct {
+ tl translation
+}
+
+// Error returns ErrBadPluralDefinition's internal error text
+func (e *ErrBadPluralDefinition) Error() string {
+ return fmt.Sprintf("error: bad plural definition '%#v'", e.tl)
+}
diff --git a/vendor/github.com/go-playground/universal-translator/import_export.go b/vendor/github.com/go-playground/universal-translator/import_export.go
new file mode 100644
index 0000000..7bd76f2
--- /dev/null
+++ b/vendor/github.com/go-playground/universal-translator/import_export.go
@@ -0,0 +1,274 @@
+package ut
+
+import (
+ "encoding/json"
+ "fmt"
+ "io/ioutil"
+ "os"
+ "path/filepath"
+
+ "io"
+
+ "github.com/go-playground/locales"
+)
+
+type translation struct {
+ Locale string `json:"locale"`
+ Key interface{} `json:"key"` // either string or integer
+ Translation string `json:"trans"`
+ PluralType string `json:"type,omitempty"`
+ PluralRule string `json:"rule,omitempty"`
+ OverrideExisting bool `json:"override,omitempty"`
+}
+
+const (
+ cardinalType = "Cardinal"
+ ordinalType = "Ordinal"
+ rangeType = "Range"
+)
+
+// ImportExportFormat is the format of the file import or export
+type ImportExportFormat uint8
+
+// supported Export Formats
+const (
+ FormatJSON ImportExportFormat = iota
+)
+
+// Export writes the translations out to a file on disk.
+//
+// NOTE: this currently only works with string or int translations keys.
+func (t *UniversalTranslator) Export(format ImportExportFormat, dirname string) error {
+
+ _, err := os.Stat(dirname)
+ fmt.Println(dirname, err, os.IsNotExist(err))
+ if err != nil {
+
+ if !os.IsNotExist(err) {
+ return err
+ }
+
+ if err = os.MkdirAll(dirname, 0744); err != nil {
+ return err
+ }
+ }
+
+ // build up translations
+ var trans []translation
+ var b []byte
+ var ext string
+
+ for _, locale := range t.translators {
+
+ for k, v := range locale.(*translator).translations {
+ trans = append(trans, translation{
+ Locale: locale.Locale(),
+ Key: k,
+ Translation: v.text,
+ })
+ }
+
+ for k, pluralTrans := range locale.(*translator).cardinalTanslations {
+
+ for i, plural := range pluralTrans {
+
+ // leave enough for all plural rules
+ // but not all are set for all languages.
+ if plural == nil {
+ continue
+ }
+
+ trans = append(trans, translation{
+ Locale: locale.Locale(),
+ Key: k.(string),
+ Translation: plural.text,
+ PluralType: cardinalType,
+ PluralRule: locales.PluralRule(i).String(),
+ })
+ }
+ }
+
+ for k, pluralTrans := range locale.(*translator).ordinalTanslations {
+
+ for i, plural := range pluralTrans {
+
+ // leave enough for all plural rules
+ // but not all are set for all languages.
+ if plural == nil {
+ continue
+ }
+
+ trans = append(trans, translation{
+ Locale: locale.Locale(),
+ Key: k.(string),
+ Translation: plural.text,
+ PluralType: ordinalType,
+ PluralRule: locales.PluralRule(i).String(),
+ })
+ }
+ }
+
+ for k, pluralTrans := range locale.(*translator).rangeTanslations {
+
+ for i, plural := range pluralTrans {
+
+ // leave enough for all plural rules
+ // but not all are set for all languages.
+ if plural == nil {
+ continue
+ }
+
+ trans = append(trans, translation{
+ Locale: locale.Locale(),
+ Key: k.(string),
+ Translation: plural.text,
+ PluralType: rangeType,
+ PluralRule: locales.PluralRule(i).String(),
+ })
+ }
+ }
+
+ switch format {
+ case FormatJSON:
+ b, err = json.MarshalIndent(trans, "", " ")
+ ext = ".json"
+ }
+
+ if err != nil {
+ return err
+ }
+
+ err = ioutil.WriteFile(filepath.Join(dirname, fmt.Sprintf("%s%s", locale.Locale(), ext)), b, 0644)
+ if err != nil {
+ return err
+ }
+
+ trans = trans[0:0]
+ }
+
+ return nil
+}
+
+// Import reads the translations out of a file or directory on disk.
+//
+// NOTE: this currently only works with string or int translations keys.
+func (t *UniversalTranslator) Import(format ImportExportFormat, dirnameOrFilename string) error {
+
+ fi, err := os.Stat(dirnameOrFilename)
+ if err != nil {
+ return err
+ }
+
+ processFn := func(filename string) error {
+
+ f, err := os.Open(filename)
+ if err != nil {
+ return err
+ }
+ defer f.Close()
+
+ return t.ImportByReader(format, f)
+ }
+
+ if !fi.IsDir() {
+ return processFn(dirnameOrFilename)
+ }
+
+ // recursively go through directory
+ walker := func(path string, info os.FileInfo, err error) error {
+
+ if info.IsDir() {
+ return nil
+ }
+
+ switch format {
+ case FormatJSON:
+ // skip non JSON files
+ if filepath.Ext(info.Name()) != ".json" {
+ return nil
+ }
+ }
+
+ return processFn(path)
+ }
+
+ return filepath.Walk(dirnameOrFilename, walker)
+}
+
+// ImportByReader imports the the translations found within the contents read from the supplied reader.
+//
+// NOTE: generally used when assets have been embedded into the binary and are already in memory.
+func (t *UniversalTranslator) ImportByReader(format ImportExportFormat, reader io.Reader) error {
+
+ b, err := ioutil.ReadAll(reader)
+ if err != nil {
+ return err
+ }
+
+ var trans []translation
+
+ switch format {
+ case FormatJSON:
+ err = json.Unmarshal(b, &trans)
+ }
+
+ if err != nil {
+ return err
+ }
+
+ for _, tl := range trans {
+
+ locale, found := t.FindTranslator(tl.Locale)
+ if !found {
+ return &ErrMissingLocale{locale: tl.Locale}
+ }
+
+ pr := stringToPR(tl.PluralRule)
+
+ if pr == locales.PluralRuleUnknown {
+
+ err = locale.Add(tl.Key, tl.Translation, tl.OverrideExisting)
+ if err != nil {
+ return err
+ }
+
+ continue
+ }
+
+ switch tl.PluralType {
+ case cardinalType:
+ err = locale.AddCardinal(tl.Key, tl.Translation, pr, tl.OverrideExisting)
+ case ordinalType:
+ err = locale.AddOrdinal(tl.Key, tl.Translation, pr, tl.OverrideExisting)
+ case rangeType:
+ err = locale.AddRange(tl.Key, tl.Translation, pr, tl.OverrideExisting)
+ default:
+ return &ErrBadPluralDefinition{tl: tl}
+ }
+
+ if err != nil {
+ return err
+ }
+ }
+
+ return nil
+}
+
+func stringToPR(s string) locales.PluralRule {
+
+ switch s {
+ case "One":
+ return locales.PluralRuleOne
+ case "Two":
+ return locales.PluralRuleTwo
+ case "Few":
+ return locales.PluralRuleFew
+ case "Many":
+ return locales.PluralRuleMany
+ case "Other":
+ return locales.PluralRuleOther
+ default:
+ return locales.PluralRuleUnknown
+ }
+
+}
diff --git a/vendor/github.com/go-playground/universal-translator/logo.png b/vendor/github.com/go-playground/universal-translator/logo.png
new file mode 100644
index 0000000..a37aa8c
Binary files /dev/null and b/vendor/github.com/go-playground/universal-translator/logo.png differ
diff --git a/vendor/github.com/go-playground/universal-translator/translator.go b/vendor/github.com/go-playground/universal-translator/translator.go
new file mode 100644
index 0000000..cfafce8
--- /dev/null
+++ b/vendor/github.com/go-playground/universal-translator/translator.go
@@ -0,0 +1,420 @@
+package ut
+
+import (
+ "fmt"
+ "strconv"
+ "strings"
+
+ "github.com/go-playground/locales"
+)
+
+const (
+ paramZero = "{0}"
+ paramOne = "{1}"
+ unknownTranslation = ""
+)
+
+// Translator is universal translators
+// translator instance which is a thin wrapper
+// around locales.Translator instance providing
+// some extra functionality
+type Translator interface {
+ locales.Translator
+
+ // adds a normal translation for a particular language/locale
+ // {#} is the only replacement type accepted and are ad infinitum
+ // eg. one: '{0} day left' other: '{0} days left'
+ Add(key interface{}, text string, override bool) error
+
+ // adds a cardinal plural translation for a particular language/locale
+ // {0} is the only replacement type accepted and only one variable is accepted as
+ // multiple cannot be used for a plural rule determination, unless it is a range;
+ // see AddRange below.
+ // eg. in locale 'en' one: '{0} day left' other: '{0} days left'
+ AddCardinal(key interface{}, text string, rule locales.PluralRule, override bool) error
+
+ // adds an ordinal plural translation for a particular language/locale
+ // {0} is the only replacement type accepted and only one variable is accepted as
+ // multiple cannot be used for a plural rule determination, unless it is a range;
+ // see AddRange below.
+ // eg. in locale 'en' one: '{0}st day of spring' other: '{0}nd day of spring'
+ // - 1st, 2nd, 3rd...
+ AddOrdinal(key interface{}, text string, rule locales.PluralRule, override bool) error
+
+ // adds a range plural translation for a particular language/locale
+ // {0} and {1} are the only replacement types accepted and only these are accepted.
+ // eg. in locale 'nl' one: '{0}-{1} day left' other: '{0}-{1} days left'
+ AddRange(key interface{}, text string, rule locales.PluralRule, override bool) error
+
+ // creates the translation for the locale given the 'key' and params passed in
+ T(key interface{}, params ...string) (string, error)
+
+ // creates the cardinal translation for the locale given the 'key', 'num' and 'digit' arguments
+ // and param passed in
+ C(key interface{}, num float64, digits uint64, param string) (string, error)
+
+ // creates the ordinal translation for the locale given the 'key', 'num' and 'digit' arguments
+ // and param passed in
+ O(key interface{}, num float64, digits uint64, param string) (string, error)
+
+ // creates the range translation for the locale given the 'key', 'num1', 'digit1', 'num2' and
+ // 'digit2' arguments and 'param1' and 'param2' passed in
+ R(key interface{}, num1 float64, digits1 uint64, num2 float64, digits2 uint64, param1, param2 string) (string, error)
+
+ // VerifyTranslations checks to ensures that no plural rules have been
+ // missed within the translations.
+ VerifyTranslations() error
+}
+
+var _ Translator = new(translator)
+var _ locales.Translator = new(translator)
+
+type translator struct {
+ locales.Translator
+ translations map[interface{}]*transText
+ cardinalTanslations map[interface{}][]*transText // array index is mapped to locales.PluralRule index + the locales.PluralRuleUnknown
+ ordinalTanslations map[interface{}][]*transText
+ rangeTanslations map[interface{}][]*transText
+}
+
+type transText struct {
+ text string
+ indexes []int
+}
+
+func newTranslator(trans locales.Translator) Translator {
+ return &translator{
+ Translator: trans,
+ translations: make(map[interface{}]*transText), // translation text broken up by byte index
+ cardinalTanslations: make(map[interface{}][]*transText),
+ ordinalTanslations: make(map[interface{}][]*transText),
+ rangeTanslations: make(map[interface{}][]*transText),
+ }
+}
+
+// Add adds a normal translation for a particular language/locale
+// {#} is the only replacement type accepted and are ad infinitum
+// eg. one: '{0} day left' other: '{0} days left'
+func (t *translator) Add(key interface{}, text string, override bool) error {
+
+ if _, ok := t.translations[key]; ok && !override {
+ return &ErrConflictingTranslation{locale: t.Locale(), key: key, text: text}
+ }
+
+ lb := strings.Count(text, "{")
+ rb := strings.Count(text, "}")
+
+ if lb != rb {
+ return &ErrMissingBracket{locale: t.Locale(), key: key, text: text}
+ }
+
+ trans := &transText{
+ text: text,
+ }
+
+ var idx int
+
+ for i := 0; i < lb; i++ {
+ s := "{" + strconv.Itoa(i) + "}"
+ idx = strings.Index(text, s)
+ if idx == -1 {
+ return &ErrBadParamSyntax{locale: t.Locale(), param: s, key: key, text: text}
+ }
+
+ trans.indexes = append(trans.indexes, idx)
+ trans.indexes = append(trans.indexes, idx+len(s))
+ }
+
+ t.translations[key] = trans
+
+ return nil
+}
+
+// AddCardinal adds a cardinal plural translation for a particular language/locale
+// {0} is the only replacement type accepted and only one variable is accepted as
+// multiple cannot be used for a plural rule determination, unless it is a range;
+// see AddRange below.
+// eg. in locale 'en' one: '{0} day left' other: '{0} days left'
+func (t *translator) AddCardinal(key interface{}, text string, rule locales.PluralRule, override bool) error {
+
+ var verified bool
+
+ // verify plural rule exists for locale
+ for _, pr := range t.PluralsCardinal() {
+ if pr == rule {
+ verified = true
+ break
+ }
+ }
+
+ if !verified {
+ return &ErrCardinalTranslation{text: fmt.Sprintf("error: cardinal plural rule '%s' does not exist for locale '%s' key: '%v' text: '%s'", rule, t.Locale(), key, text)}
+ }
+
+ tarr, ok := t.cardinalTanslations[key]
+ if ok {
+ // verify not adding a conflicting record
+ if len(tarr) > 0 && tarr[rule] != nil && !override {
+ return &ErrConflictingTranslation{locale: t.Locale(), key: key, rule: rule, text: text}
+ }
+
+ } else {
+ tarr = make([]*transText, 7, 7)
+ t.cardinalTanslations[key] = tarr
+ }
+
+ trans := &transText{
+ text: text,
+ indexes: make([]int, 2, 2),
+ }
+
+ tarr[rule] = trans
+
+ idx := strings.Index(text, paramZero)
+ if idx == -1 {
+ tarr[rule] = nil
+ return &ErrCardinalTranslation{text: fmt.Sprintf("error: parameter '%s' not found, may want to use 'Add' instead of 'AddCardinal'. locale: '%s' key: '%v' text: '%s'", paramZero, t.Locale(), key, text)}
+ }
+
+ trans.indexes[0] = idx
+ trans.indexes[1] = idx + len(paramZero)
+
+ return nil
+}
+
+// AddOrdinal adds an ordinal plural translation for a particular language/locale
+// {0} is the only replacement type accepted and only one variable is accepted as
+// multiple cannot be used for a plural rule determination, unless it is a range;
+// see AddRange below.
+// eg. in locale 'en' one: '{0}st day of spring' other: '{0}nd day of spring' - 1st, 2nd, 3rd...
+func (t *translator) AddOrdinal(key interface{}, text string, rule locales.PluralRule, override bool) error {
+
+ var verified bool
+
+ // verify plural rule exists for locale
+ for _, pr := range t.PluralsOrdinal() {
+ if pr == rule {
+ verified = true
+ break
+ }
+ }
+
+ if !verified {
+ return &ErrOrdinalTranslation{text: fmt.Sprintf("error: ordinal plural rule '%s' does not exist for locale '%s' key: '%v' text: '%s'", rule, t.Locale(), key, text)}
+ }
+
+ tarr, ok := t.ordinalTanslations[key]
+ if ok {
+ // verify not adding a conflicting record
+ if len(tarr) > 0 && tarr[rule] != nil && !override {
+ return &ErrConflictingTranslation{locale: t.Locale(), key: key, rule: rule, text: text}
+ }
+
+ } else {
+ tarr = make([]*transText, 7, 7)
+ t.ordinalTanslations[key] = tarr
+ }
+
+ trans := &transText{
+ text: text,
+ indexes: make([]int, 2, 2),
+ }
+
+ tarr[rule] = trans
+
+ idx := strings.Index(text, paramZero)
+ if idx == -1 {
+ tarr[rule] = nil
+ return &ErrOrdinalTranslation{text: fmt.Sprintf("error: parameter '%s' not found, may want to use 'Add' instead of 'AddOrdinal'. locale: '%s' key: '%v' text: '%s'", paramZero, t.Locale(), key, text)}
+ }
+
+ trans.indexes[0] = idx
+ trans.indexes[1] = idx + len(paramZero)
+
+ return nil
+}
+
+// AddRange adds a range plural translation for a particular language/locale
+// {0} and {1} are the only replacement types accepted and only these are accepted.
+// eg. in locale 'nl' one: '{0}-{1} day left' other: '{0}-{1} days left'
+func (t *translator) AddRange(key interface{}, text string, rule locales.PluralRule, override bool) error {
+
+ var verified bool
+
+ // verify plural rule exists for locale
+ for _, pr := range t.PluralsRange() {
+ if pr == rule {
+ verified = true
+ break
+ }
+ }
+
+ if !verified {
+ return &ErrRangeTranslation{text: fmt.Sprintf("error: range plural rule '%s' does not exist for locale '%s' key: '%v' text: '%s'", rule, t.Locale(), key, text)}
+ }
+
+ tarr, ok := t.rangeTanslations[key]
+ if ok {
+ // verify not adding a conflicting record
+ if len(tarr) > 0 && tarr[rule] != nil && !override {
+ return &ErrConflictingTranslation{locale: t.Locale(), key: key, rule: rule, text: text}
+ }
+
+ } else {
+ tarr = make([]*transText, 7, 7)
+ t.rangeTanslations[key] = tarr
+ }
+
+ trans := &transText{
+ text: text,
+ indexes: make([]int, 4, 4),
+ }
+
+ tarr[rule] = trans
+
+ idx := strings.Index(text, paramZero)
+ if idx == -1 {
+ tarr[rule] = nil
+ return &ErrRangeTranslation{text: fmt.Sprintf("error: parameter '%s' not found, are you sure you're adding a Range Translation? locale: '%s' key: '%v' text: '%s'", paramZero, t.Locale(), key, text)}
+ }
+
+ trans.indexes[0] = idx
+ trans.indexes[1] = idx + len(paramZero)
+
+ idx = strings.Index(text, paramOne)
+ if idx == -1 {
+ tarr[rule] = nil
+ return &ErrRangeTranslation{text: fmt.Sprintf("error: parameter '%s' not found, a Range Translation requires two parameters. locale: '%s' key: '%v' text: '%s'", paramOne, t.Locale(), key, text)}
+ }
+
+ trans.indexes[2] = idx
+ trans.indexes[3] = idx + len(paramOne)
+
+ return nil
+}
+
+// T creates the translation for the locale given the 'key' and params passed in
+func (t *translator) T(key interface{}, params ...string) (string, error) {
+
+ trans, ok := t.translations[key]
+ if !ok {
+ return unknownTranslation, ErrUnknowTranslation
+ }
+
+ b := make([]byte, 0, 64)
+
+ var start, end, count int
+
+ for i := 0; i < len(trans.indexes); i++ {
+ end = trans.indexes[i]
+ b = append(b, trans.text[start:end]...)
+ b = append(b, params[count]...)
+ i++
+ start = trans.indexes[i]
+ count++
+ }
+
+ b = append(b, trans.text[start:]...)
+
+ return string(b), nil
+}
+
+// C creates the cardinal translation for the locale given the 'key', 'num' and 'digit' arguments and param passed in
+func (t *translator) C(key interface{}, num float64, digits uint64, param string) (string, error) {
+
+ tarr, ok := t.cardinalTanslations[key]
+ if !ok {
+ return unknownTranslation, ErrUnknowTranslation
+ }
+
+ rule := t.CardinalPluralRule(num, digits)
+
+ trans := tarr[rule]
+
+ b := make([]byte, 0, 64)
+ b = append(b, trans.text[:trans.indexes[0]]...)
+ b = append(b, param...)
+ b = append(b, trans.text[trans.indexes[1]:]...)
+
+ return string(b), nil
+}
+
+// O creates the ordinal translation for the locale given the 'key', 'num' and 'digit' arguments and param passed in
+func (t *translator) O(key interface{}, num float64, digits uint64, param string) (string, error) {
+
+ tarr, ok := t.ordinalTanslations[key]
+ if !ok {
+ return unknownTranslation, ErrUnknowTranslation
+ }
+
+ rule := t.OrdinalPluralRule(num, digits)
+
+ trans := tarr[rule]
+
+ b := make([]byte, 0, 64)
+ b = append(b, trans.text[:trans.indexes[0]]...)
+ b = append(b, param...)
+ b = append(b, trans.text[trans.indexes[1]:]...)
+
+ return string(b), nil
+}
+
+// R creates the range translation for the locale given the 'key', 'num1', 'digit1', 'num2' and 'digit2' arguments
+// and 'param1' and 'param2' passed in
+func (t *translator) R(key interface{}, num1 float64, digits1 uint64, num2 float64, digits2 uint64, param1, param2 string) (string, error) {
+
+ tarr, ok := t.rangeTanslations[key]
+ if !ok {
+ return unknownTranslation, ErrUnknowTranslation
+ }
+
+ rule := t.RangePluralRule(num1, digits1, num2, digits2)
+
+ trans := tarr[rule]
+
+ b := make([]byte, 0, 64)
+ b = append(b, trans.text[:trans.indexes[0]]...)
+ b = append(b, param1...)
+ b = append(b, trans.text[trans.indexes[1]:trans.indexes[2]]...)
+ b = append(b, param2...)
+ b = append(b, trans.text[trans.indexes[3]:]...)
+
+ return string(b), nil
+}
+
+// VerifyTranslations checks to ensures that no plural rules have been
+// missed within the translations.
+func (t *translator) VerifyTranslations() error {
+
+ for k, v := range t.cardinalTanslations {
+
+ for _, rule := range t.PluralsCardinal() {
+
+ if v[rule] == nil {
+ return &ErrMissingPluralTranslation{locale: t.Locale(), translationType: "plural", rule: rule, key: k}
+ }
+ }
+ }
+
+ for k, v := range t.ordinalTanslations {
+
+ for _, rule := range t.PluralsOrdinal() {
+
+ if v[rule] == nil {
+ return &ErrMissingPluralTranslation{locale: t.Locale(), translationType: "ordinal", rule: rule, key: k}
+ }
+ }
+ }
+
+ for k, v := range t.rangeTanslations {
+
+ for _, rule := range t.PluralsRange() {
+
+ if v[rule] == nil {
+ return &ErrMissingPluralTranslation{locale: t.Locale(), translationType: "range", rule: rule, key: k}
+ }
+ }
+ }
+
+ return nil
+}
diff --git a/vendor/github.com/go-playground/universal-translator/universal_translator.go b/vendor/github.com/go-playground/universal-translator/universal_translator.go
new file mode 100644
index 0000000..dbf707f
--- /dev/null
+++ b/vendor/github.com/go-playground/universal-translator/universal_translator.go
@@ -0,0 +1,113 @@
+package ut
+
+import (
+ "strings"
+
+ "github.com/go-playground/locales"
+)
+
+// UniversalTranslator holds all locale & translation data
+type UniversalTranslator struct {
+ translators map[string]Translator
+ fallback Translator
+}
+
+// New returns a new UniversalTranslator instance set with
+// the fallback locale and locales it should support
+func New(fallback locales.Translator, supportedLocales ...locales.Translator) *UniversalTranslator {
+
+ t := &UniversalTranslator{
+ translators: make(map[string]Translator),
+ }
+
+ for _, v := range supportedLocales {
+
+ trans := newTranslator(v)
+ t.translators[strings.ToLower(trans.Locale())] = trans
+
+ if fallback.Locale() == v.Locale() {
+ t.fallback = trans
+ }
+ }
+
+ if t.fallback == nil && fallback != nil {
+ t.fallback = newTranslator(fallback)
+ }
+
+ return t
+}
+
+// FindTranslator trys to find a Translator based on an array of locales
+// and returns the first one it can find, otherwise returns the
+// fallback translator.
+func (t *UniversalTranslator) FindTranslator(locales ...string) (trans Translator, found bool) {
+
+ for _, locale := range locales {
+
+ if trans, found = t.translators[strings.ToLower(locale)]; found {
+ return
+ }
+ }
+
+ return t.fallback, false
+}
+
+// GetTranslator returns the specified translator for the given locale,
+// or fallback if not found
+func (t *UniversalTranslator) GetTranslator(locale string) (trans Translator, found bool) {
+
+ if trans, found = t.translators[strings.ToLower(locale)]; found {
+ return
+ }
+
+ return t.fallback, false
+}
+
+// GetFallback returns the fallback locale
+func (t *UniversalTranslator) GetFallback() Translator {
+ return t.fallback
+}
+
+// AddTranslator adds the supplied translator, if it already exists the override param
+// will be checked and if false an error will be returned, otherwise the translator will be
+// overridden; if the fallback matches the supplied translator it will be overridden as well
+// NOTE: this is normally only used when translator is embedded within a library
+func (t *UniversalTranslator) AddTranslator(translator locales.Translator, override bool) error {
+
+ lc := strings.ToLower(translator.Locale())
+ _, ok := t.translators[lc]
+ if ok && !override {
+ return &ErrExistingTranslator{locale: translator.Locale()}
+ }
+
+ trans := newTranslator(translator)
+
+ if t.fallback.Locale() == translator.Locale() {
+
+ // because it's optional to have a fallback, I don't impose that limitation
+ // don't know why you wouldn't but...
+ if !override {
+ return &ErrExistingTranslator{locale: translator.Locale()}
+ }
+
+ t.fallback = trans
+ }
+
+ t.translators[lc] = trans
+
+ return nil
+}
+
+// VerifyTranslations runs through all locales and identifies any issues
+// eg. missing plural rules for a locale
+func (t *UniversalTranslator) VerifyTranslations() (err error) {
+
+ for _, trans := range t.translators {
+ err = trans.VerifyTranslations()
+ if err != nil {
+ return
+ }
+ }
+
+ return
+}
diff --git a/vendor/github.com/google/uuid/.travis.yml b/vendor/github.com/google/uuid/.travis.yml
new file mode 100644
index 0000000..d8156a6
--- /dev/null
+++ b/vendor/github.com/google/uuid/.travis.yml
@@ -0,0 +1,9 @@
+language: go
+
+go:
+ - 1.4.3
+ - 1.5.3
+ - tip
+
+script:
+ - go test -v ./...
diff --git a/vendor/github.com/google/uuid/CONTRIBUTING.md b/vendor/github.com/google/uuid/CONTRIBUTING.md
new file mode 100644
index 0000000..04fdf09
--- /dev/null
+++ b/vendor/github.com/google/uuid/CONTRIBUTING.md
@@ -0,0 +1,10 @@
+# How to contribute
+
+We definitely welcome patches and contribution to this project!
+
+### Legal requirements
+
+In order to protect both you and ourselves, you will need to sign the
+[Contributor License Agreement](https://cla.developers.google.com/clas).
+
+You may have already signed it for other Google projects.
diff --git a/vendor/github.com/google/uuid/CONTRIBUTORS b/vendor/github.com/google/uuid/CONTRIBUTORS
new file mode 100644
index 0000000..b4bb97f
--- /dev/null
+++ b/vendor/github.com/google/uuid/CONTRIBUTORS
@@ -0,0 +1,9 @@
+Paul Borman +BenchmarkValuesScan 500000 6265 ns/op 74 B/op 3 allocs/op +BenchmarkScan 500000 4994 ns/op 41 B/op 4 allocs/op +BenchmarkNamedScan 500000 4960 ns/op 93 B/op 7 allocs/op + +BenchmarkInsert 500000 4085 ns/op 16 B/op 1 allocs/op +BenchmarkNamedInsert 500000 4798 ns/op 64 B/op 4 allocs/op + +BenchmarkDisabledCache 100000 19841 ns/op 117 B/op 3 allocs/op +BenchmarkEnabledCache 2000000 790 ns/op 50 B/op 1 allocs/op + +BenchmarkLike 1000000 2605 ns/op 0 B/op 0 allocs/op +BenchmarkHalf 500000 4988 ns/op 33 B/op 1 allocs/op +BenchmarkRegexp 500000 5557 ns/op 8 B/op 1 allocs/op +diff --git a/vendor/github.com/gwenn/gosqlite/backup.go b/vendor/github.com/gwenn/gosqlite/backup.go new file mode 100644 index 0000000..e86178f --- /dev/null +++ b/vendor/github.com/gwenn/gosqlite/backup.go @@ -0,0 +1,124 @@ +// Copyright 2010 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package sqlite + +/* +#include
+BenchmarkParsing 5000 381518 ns/op 256.87 MB/s 4288 B/op 5 allocs/op +BenchmarkQuotedParsing 5000 487599 ns/op 209.19 MB/s 4288 B/op 5 allocs/op +BenchmarkEmbeddedNL 5000 594618 ns/op 201.81 MB/s 4288 B/op 5 allocs/op +BenchmarkStdParser 500 5026100 ns/op 23.88 MB/s 625499 B/op 16037 allocs/op +BenchmarkYacrParser 5000 593165 ns/op 202.30 MB/s 4288 B/op 5 allocs/op +BenchmarkYacrWriter 200000 9433 ns/op 98.05 MB/s 2755 B/op 0 allocs/op +BenchmarkStdWriter 100000 27804 ns/op 33.27 MB/s 2755 B/op 0 allocs/op ++ +USAGES +------ +* [csvdiff](https://github.com/gwenn/csvdiff) +* [csvgrep](https://github.com/gwenn/csvgrep) +* [SQLite import/export/module](https://github.com/gwenn/gosqlite/blob/master/csv.go) diff --git a/vendor/github.com/gwenn/yacr/reader.go b/vendor/github.com/gwenn/yacr/reader.go new file mode 100644 index 0000000..7d6ebd9 --- /dev/null +++ b/vendor/github.com/gwenn/yacr/reader.go @@ -0,0 +1,472 @@ +// Copyright 2011 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +// Package yacr is yet another CSV reader (and writer) with small memory usage. +package yacr + +import ( + "bufio" + "bytes" + "encoding" + "fmt" + "io" + "reflect" + "strconv" +) + +// Reader provides an interface for reading CSV data +// (compatible with rfc4180 and extended with the option of having a separator other than ","). +// Successive calls to the Scan method will step through the 'fields', skipping the separator/newline between the fields. +// The EndOfRecord method tells when a field is terminated by a line break. +type Reader struct { + *bufio.Scanner + sep byte // values separator + quoted bool // specify if values may be quoted (when they contain separator or newline) + guess bool // try to guess separator based on the file header + eor bool // true when the most recent field has been terminated by a newline (not a separator). + lineno int // current line number (not record number) + + Trim bool // trim spaces (only on unquoted values). Break rfc4180 rule: "Spaces are considered part of a field and should not be ignored." + Comment byte // character marking the start of a line comment. When specified (not 0), line comment appears as empty line. + Lazy bool // specify if quoted values may contains unescaped quote not followed by a separator or a newline + + Headers map[string]int // Index (first is 1) by header +} + +// DefaultReader creates a "standard" CSV reader (separator is comma and quoted mode active) +func DefaultReader(rd io.Reader) *Reader { + return NewReader(rd, ',', true, false) +} + +// NewReader returns a new CSV scanner to read from r. +// When quoted is false, values must not contain a separator or newline. +func NewReader(r io.Reader, sep byte, quoted, guess bool) *Reader { + s := &Reader{bufio.NewScanner(r), sep, quoted, guess, true, 1, false, 0, false, nil} + s.Split(s.ScanField) + return s +} + +// ScanHeaders loads current line as the header line. +func (s *Reader) ScanHeaders() error { + s.Headers = make(map[string]int) + for i := 1; s.Scan(); i++ { + s.Headers[s.Text()] = i + if s.EndOfRecord() { + break + } + } + return s.Err() +} + +// ScanRecordByName decodes one line fields by name (name1, value1, ...). +// Specified names must match Headers. +func (s *Reader) ScanRecordByName(args ...interface{}) (int, error) { + if len(args)%2 != 0 { + return 0, fmt.Errorf("expected an even number of arguments: %d", len(args)) + } + values := make([]interface{}, len(s.Headers)) + for i := 0; i < len(args); i += 2 { + name, ok := args[i].(string) + if !ok { + return 0, fmt.Errorf("non-string field name at %d: %T", i, args[i]) + } + index, ok := s.Headers[name] + if !ok { + return 0, fmt.Errorf("unknown field name: %s", name) + } + values[index-1] = args[i+1] + } + return s.ScanRecord(values...) +} + +// ScanRecord decodes one line fields to values. +// Empty lines are ignored/skipped. +// It's like fmt.Scan or database.sql.Rows.Scan. +// Returns (0, nil) on EOF, (*, err) on error +// and (n >= 1, nil) on success (n may be less or greater than len(values)). +// var n int +// var err error +// for { +// values := make([]string, N) +// if n, err = s.ScanRecord(&values[0]/*, &values[1], ...*/); err != nil || n == 0 { +// break // or error handling +// } else if (n > N) { +// n = N // ignore extra values +// } +// for _, value := range values[0:n] { +// // ... +// } +// } +// if err != nil { +// // error handling +// } +func (s *Reader) ScanRecord(values ...interface{}) (int, error) { + for i, value := range values { + if !s.Scan() { + return i, s.Err() + } + if i == 0 { // skip empty line (or line comment) + for s.EndOfRecord() && len(s.Bytes()) == 0 { + if !s.Scan() { + return i, s.Err() + } + } + } + if err := s.value(value, true); err != nil { + return i + 1, err + } else if s.EndOfRecord() && i != len(values)-1 { + return i + 1, nil + } + } + if !s.EndOfRecord() { + i := len(values) + for ; !s.EndOfRecord(); i++ { // Consume extra fields + if !s.Scan() { + return i, s.Err() + } + } + return i, nil + } + return len(values), nil +} + +// ScanValue advances to the next token and decodes field's content to value. +// The value may point to data that will be overwritten by a subsequent call to Scan. +func (s *Reader) ScanValue(value interface{}) error { + if !s.Scan() { + return s.Err() + } + return s.value(value, false) +} + +// Value decodes field's content to value. +// The value may point to data that will be overwritten by a subsequent call to Scan. +func (s *Reader) Value(value interface{}) error { + return s.value(value, false) +} +func (s *Reader) value(value interface{}, copied bool) error { + var err error + switch value := value.(type) { + case nil: + case *string: + *value = s.Text() + case *int: + *value, err = strconv.Atoi(s.Text()) + case *int32: + var i int64 + i, err = strconv.ParseInt(s.Text(), 10, 32) + *value = int32(i) + case *int64: + *value, err = strconv.ParseInt(s.Text(), 10, 64) + case *bool: + *value, err = strconv.ParseBool(s.Text()) + case *float64: + *value, err = strconv.ParseFloat(s.Text(), 64) + case *[]byte: + if copied { + v := s.Bytes() + c := make([]byte, len(v)) + copy(c, v) + *value = c + } else { + *value = s.Bytes() + } + case encoding.TextUnmarshaler: + err = value.UnmarshalText(s.Bytes()) + default: + return s.scanReflect(value) + } + return err +} + +func (s *Reader) scanReflect(v interface{}) (err error) { + rv := reflect.ValueOf(v) + if rv.Kind() != reflect.Ptr || rv.IsNil() { + return fmt.Errorf("unsupported type %T", v) + } + dv := reflect.Indirect(rv) + switch dv.Kind() { + case reflect.String: + dv.SetString(s.Text()) + case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: + var i int64 + i, err = strconv.ParseInt(s.Text(), 10, dv.Type().Bits()) + if err == nil { + dv.SetInt(i) + } + case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr: + var i uint64 + i, err = strconv.ParseUint(s.Text(), 10, dv.Type().Bits()) + if err == nil { + dv.SetUint(i) + } + case reflect.Bool: + var b bool + b, err = strconv.ParseBool(s.Text()) + if err == nil { + dv.SetBool(b) + } + case reflect.Float32, reflect.Float64: + var f float64 + f, err = strconv.ParseFloat(s.Text(), dv.Type().Bits()) + if err == nil { + dv.SetFloat(f) + } + default: + return fmt.Errorf("unsupported type: %T", v) + } + return +} + +// LineNumber returns current line number (not record number) +func (s *Reader) LineNumber() int { + return s.lineno +} + +// EndOfRecord returns true when the most recent field has been terminated by a newline (not a separator). +func (s *Reader) EndOfRecord() bool { + return s.eor +} + +// Sep returns the values separator used/guessed +func (s *Reader) Sep() byte { + return s.sep +} + +// SkipRecords skips n records/headers +func (s *Reader) SkipRecords(n int) error { + i := 0 + for { + if i == n { + return nil + } + if !s.Scan() { + return s.Err() + } + if s.eor { + i++ + } + } +} + +// ScanField implements bufio.SplitFunc for CSV. +// Lexing is adapted from csv_read_one_field function in SQLite3 shell sources. +func (s *Reader) ScanField(data []byte, atEOF bool) (advance int, token []byte, err error) { + var a int + for { + a, token, err = s.scanField(data, atEOF) + advance += a + if err != nil || a == 0 || token != nil { + return + } + data = data[a:] + } +} + +func (s *Reader) scanField(data []byte, atEOF bool) (advance int, token []byte, err error) { + if atEOF && len(data) == 0 && s.eor { + return 0, nil, nil + } + if s.guess { + s.guess = false + if b := guess(data); b > 0 { + s.sep = b + } + } + if s.quoted && len(data) > 0 && data[0] == '"' { // quoted field (may contains separator, newline and escaped quote) + startLineno := s.lineno + escapedQuotes := 0 + strict := true + var c, pc, ppc byte + // Scan until the separator or newline following the closing quote (and ignore escaped quote) + for i := 1; i < len(data); i++ { + c = data[i] + if c == '\n' { + s.lineno++ + } else if c == '"' { + if pc == c { // escaped quote + pc = 0 + escapedQuotes++ + continue + } + } + if pc == '"' && c == s.sep { + s.eor = false + return i + 1, unescapeQuotes(data[1:i-1], escapedQuotes, strict), nil + } else if pc == '"' && c == '\n' { + s.eor = true + return i + 1, unescapeQuotes(data[1:i-1], escapedQuotes, strict), nil + } else if c == '\n' && pc == '\r' && ppc == '"' { + s.eor = true + return i + 1, unescapeQuotes(data[1:i-2], escapedQuotes, strict), nil + } + if pc == '"' && c != '\r' { + if s.Lazy { + strict = false + } else { + return 0, nil, fmt.Errorf("unescaped %c character at line %d", pc, s.lineno) + } + } + ppc = pc + pc = c + } + if atEOF { + if c == '"' { + s.eor = true + return len(data), unescapeQuotes(data[1:len(data)-1], escapedQuotes, strict), nil + } + // If we're at EOF, we have a non-terminated field. + return 0, nil, fmt.Errorf("non-terminated quoted field between lines %d and %d", startLineno, s.lineno) + } + } else if s.eor && s.Comment != 0 && len(data) > 0 && data[0] == s.Comment { // line comment + for i, c := range data { + if c == '\n' { + s.lineno++ + return i + 1, nil, nil + } + } + if atEOF { + return len(data), nil, nil + } + } else { // unquoted field + // Scan until separator or newline, marking end of field. + for i, c := range data { + if c == s.sep { + s.eor = false + if s.Trim { + return i + 1, trim(data[0:i]), nil + } + return i + 1, data[0:i], nil + } else if c == '\n' { + s.lineno++ + if i > 0 && data[i-1] == '\r' { + s.eor = true + if s.Trim { + return i + 1, trim(data[0 : i-1]), nil + } + return i + 1, data[0 : i-1], nil + } + s.eor = true + if s.Trim { + return i + 1, trim(data[0:i]), nil + } + return i + 1, data[0:i], nil + } + } + // If we're at EOF, we have a final field. Return it. + if atEOF { + s.eor = true + if s.Trim { + return len(data), trim(data), nil + } + return len(data), data, nil + } + } + // Request more data. + return 0, nil, nil +} + +func unescapeQuotes(b []byte, count int, strict bool) []byte { + if count == 0 { + return b + } + for i, j := 0, 0; i < len(b); i, j = i+1, j+1 { + b[j] = b[i] + if b[i] == '"' && (strict || i < len(b)-1 && b[i+1] == '"') { + i++ + } + } + return b[:len(b)-count] +} + +func guess(data []byte) byte { + seps := []byte{',', ';', '\t', '|', ':'} + count := make(map[byte]uint) + for _, b := range data { + if bytes.IndexByte(seps, b) >= 0 { + count[b]++ + /*} else if b == '\n' { + break*/ + } + } + var max uint + var sep byte + for b, c := range count { + if c > max { + max = c + sep = b + } + } + return sep +} + +// bytes.TrimSpace may return nil... +func trim(s []byte) []byte { + t := bytes.TrimSpace(s) + if t == nil { + return s[0:0] + } + return t +} + +// IsNumber determines if the current token is a number or not. +// Only works for single-byte encodings (ASCII, ISO-8859-1) and UTF-8. +func (s *Reader) IsNumber() (isNum bool, isReal bool) { + return IsNumber(s.Bytes()) +} + +// Only works for single-byte encodings (ASCII, ISO-8859-1) and UTF-8. +func isDigit(c byte) bool { + return c >= '0' && c <= '9' +} + +// IsNumber determines if the string is a number or not. +// Only works for single-byte encodings (ASCII, ISO-8859-1) and UTF-8. +func IsNumber(s []byte) (isNum bool, isReal bool) { + if len(s) == 0 { + return false, false + } + i := 0 + if s[i] == '-' || s[i] == '+' { // sign + i++ + } + // Nor Hexadecimal nor octal supported + digit := false + for ; len(s) != i && isDigit(s[i]); i++ { + digit = true + } + if len(s) == i { // integer "[-+]?\d*" + return digit, false + } + if s[i] == '.' { // real + for i++; len(s) != i && isDigit(s[i]); i++ { // digit(s) optional + digit = true + } + } + if len(s) == i { // real "[-+]?\d*\.\d*" + if digit { + return true, true + } + // "[-+]?\." is not a number + return false, false + } + if s[i] == 'e' || s[i] == 'E' { // exponent + i++ + if !digit || len(s) == i { // nor "[-+]?\.?e" nor "[-+]?\d*\.?\d*e" is a number + return false, false + } + if s[i] == '-' || s[i] == '+' { // sign + i++ + } + if len(s) == i || !isDigit(s[i]) { // one digit expected + return false, false + } + for i++; len(s) != i && isDigit(s[i]); i++ { + } + } + if len(s) == i { + return true, true + } + return false, false +} diff --git a/vendor/github.com/gwenn/yacr/util.go b/vendor/github.com/gwenn/yacr/util.go new file mode 100644 index 0000000..b78e4d4 --- /dev/null +++ b/vendor/github.com/gwenn/yacr/util.go @@ -0,0 +1,53 @@ +// Copyright 2011 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package yacr + +import ( + "compress/bzip2" + "compress/gzip" + "io" + "io/ioutil" + "os" + "path" +) + +type zReadCloser struct { + f *os.File + rd io.ReadCloser +} + +// Zopen transparently opens gzip/bzip files (based on their extension). +func Zopen(filepath string) (io.ReadCloser, error) { + f, err := os.Open(filepath) + if err != nil { + return nil, err + } + var rd io.ReadCloser + // TODO zip + ext := path.Ext(f.Name()) + if ext == ".gz" { + rd, err = gzip.NewReader(f) + if err != nil { + _ = f.Close() + return nil, err + } + } else if ext == ".bz2" { + rd = ioutil.NopCloser(bzip2.NewReader(f)) + } else { + rd = f + } + return &zReadCloser{f, rd}, nil +} +func (z *zReadCloser) Read(b []byte) (n int, err error) { + return z.rd.Read(b) +} +func (z *zReadCloser) Close() (err error) { + err = z.rd.Close() + if err != nil { + _ = z.f.Close() + return + } + return z.f.Close() +} diff --git a/vendor/github.com/gwenn/yacr/writer.go b/vendor/github.com/gwenn/yacr/writer.go new file mode 100644 index 0000000..3766ee8 --- /dev/null +++ b/vendor/github.com/gwenn/yacr/writer.go @@ -0,0 +1,212 @@ +// Copyright 2011 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package yacr + +import ( + "bufio" + "encoding" + "errors" + "fmt" + "io" + "reflect" + "strconv" + "unsafe" +) + +// Writer provides an interface for writing CSV data +// (compatible with rfc4180 and extended with the option of having a separator other than ","). +// Successive calls to the Write method will automatically insert the separator. +// The EndOfRecord method tells when a line break is inserted. +type Writer struct { + b *bufio.Writer + sep byte // values separator + quoted bool // specify if values should be quoted (when they contain a separator, a double-quote or a newline) + sor bool // true at start of record + err error // sticky error. + bs []byte // byte slice used to write string with minimal/no alloc/copy + hb *reflect.SliceHeader // header of bs + + UseCRLF bool // True to use \r\n as the line terminator +} + +// DefaultWriter creates a "standard" CSV writer (separator is comma and quoted mode active) +func DefaultWriter(wr io.Writer) *Writer { + return NewWriter(wr, ',', true) +} + +// NewWriter returns a new CSV writer. +func NewWriter(w io.Writer, sep byte, quoted bool) *Writer { + wr := &Writer{b: bufio.NewWriter(w), sep: sep, quoted: quoted, sor: true} + wr.hb = (*reflect.SliceHeader)(unsafe.Pointer(&wr.bs)) + return wr +} + +// WriteRecord ensures that values are quoted when needed. +// It's like fmt.Println. +func (w *Writer) WriteRecord(values ...interface{}) bool { + for _, v := range values { + if !w.WriteValue(v) { + return false + } + } + w.EndOfRecord() + return w.err == nil +} + +// WriteValue ensures that value is quoted when needed. +// Value's type/kind is used to encode value to text. +func (w *Writer) WriteValue(value interface{}) bool { + switch value := value.(type) { + case nil: + return w.Write([]byte{}) + case string: + return w.WriteString(value) + case int: + return w.WriteString(strconv.Itoa(value)) + case int32: + return w.WriteString(strconv.FormatInt(int64(value), 10)) + case int64: + return w.WriteString(strconv.FormatInt(value, 10)) + case bool: + return w.WriteString(strconv.FormatBool(value)) + case float32: + return w.WriteString(strconv.FormatFloat(float64(value), 'f', -1, 32)) + case float64: + return w.WriteString(strconv.FormatFloat(value, 'f', -1, 64)) + case []byte: + return w.Write(value) + case encoding.TextMarshaler: // time.Time + if text, err := value.MarshalText(); err != nil { + w.setErr(err) + w.Write([]byte{}) // TODO Validate: write an empty field + return false + } else { + return w.Write(text) // please, ignore golint + } + default: + return w.writeReflect(value) + } +} + +// WriteReflect ensures that value is quoted when needed. +// Value's (reflect) Kind is used to encode value to text. +func (w *Writer) writeReflect(value interface{}) bool { + v := reflect.ValueOf(value) + switch v.Kind() { + case reflect.String: + return w.WriteString(v.String()) + case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: + return w.WriteString(strconv.FormatInt(v.Int(), 10)) + case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr: + return w.WriteString(strconv.FormatUint(v.Uint(), 10)) + case reflect.Bool: + return w.WriteString(strconv.FormatBool(v.Bool())) + case reflect.Float32, reflect.Float64: + return w.WriteString(strconv.FormatFloat(v.Float(), 'f', -1, v.Type().Bits())) + default: + w.setErr(fmt.Errorf("unsupported type: %T, %v", value, value)) + w.Write([]byte{}) // TODO Validate: write an empty field + return false + } +} + +// WriteString ensures that value is quoted when needed. +func (w *Writer) WriteString(value string) bool { + // To avoid making a copy... + hs := (*reflect.StringHeader)(unsafe.Pointer(&value)) + w.hb.Data = hs.Data + w.hb.Len = hs.Len + w.hb.Cap = hs.Len + return w.Write(w.bs) +} + +var ( + // ErrNewLine is the error returned when a value contains a newline in unquoted mode. + ErrNewLine = errors.New("yacr.Writer: newline character in value") + // ErrSeparator is the error returned when a value contains a separator in unquoted mode. + ErrSeparator = errors.New("yacr.Writer: separator in value") +) + +// Write ensures that value is quoted when needed. +func (w *Writer) Write(value []byte) bool { + if w.err != nil { + return false + } + if !w.sor { + w.setErr(w.b.WriteByte(w.sep)) + } + // In quoted mode, value is enclosed between quotes if it contains sep, quote or \n. + if w.quoted { + last := 0 + for i, c := range value { + switch c { + case '"', '\r', '\n', w.sep: + default: + continue + } + if last == 0 { + w.setErr(w.b.WriteByte('"')) + } + if _, err := w.b.Write(value[last : i+1]); err != nil { + w.setErr(err) + } + if c == '"' { + w.setErr(w.b.WriteByte(c)) // escaped with another double quote + } + last = i + 1 + } + if _, err := w.b.Write(value[last:]); err != nil { + w.setErr(err) + } + if last != 0 { + w.setErr(w.b.WriteByte('"')) + } + } else { + // check that value does not contain sep or \n + for _, c := range value { + switch c { + case '\n': + w.setErr(ErrNewLine) + return false + case w.sep: + w.setErr(ErrSeparator) + return false + default: + continue + } + } + if _, err := w.b.Write(value); err != nil { + w.setErr(err) + } + } + w.sor = false + return w.err == nil +} + +// EndOfRecord tells when a line break must be inserted. +func (w *Writer) EndOfRecord() { + if w.UseCRLF { + w.setErr(w.b.WriteByte('\r')) + } + w.setErr(w.b.WriteByte('\n')) + w.sor = true +} + +// Flush ensures the writer's buffer is flushed. +func (w *Writer) Flush() { + w.setErr(w.b.Flush()) +} + +// Err returns the first error that was encountered by the Writer. +func (w *Writer) Err() error { + return w.err +} + +// setErr records the first error encountered. +func (w *Writer) setErr(err error) { + if w.err == nil { + w.err = err + } +} diff --git a/vendor/github.com/hectane/go-nonblockingchan/.travis.yml b/vendor/github.com/hectane/go-nonblockingchan/.travis.yml new file mode 100644 index 0000000..f4f458a --- /dev/null +++ b/vendor/github.com/hectane/go-nonblockingchan/.travis.yml @@ -0,0 +1,5 @@ +language: go + +go: + - 1.5 + - tip diff --git a/vendor/github.com/hectane/go-nonblockingchan/LICENSE.txt b/vendor/github.com/hectane/go-nonblockingchan/LICENSE.txt new file mode 100644 index 0000000..fb72c83 --- /dev/null +++ b/vendor/github.com/hectane/go-nonblockingchan/LICENSE.txt @@ -0,0 +1,9 @@ +The MIT License (MIT) + +Copyright (c) 2015 Nathan Osman + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. diff --git a/vendor/github.com/hectane/go-nonblockingchan/README.md b/vendor/github.com/hectane/go-nonblockingchan/README.md new file mode 100644 index 0000000..a463000 --- /dev/null +++ b/vendor/github.com/hectane/go-nonblockingchan/README.md @@ -0,0 +1,36 @@ +## go-nonblockingchan + +[](https://travis-ci.org/hectane/go-nonblockingchan) +[](https://godoc.org/github.com/hectane/go-nonblockingchan) +[](http://opensource.org/licenses/MIT) + +A special type that mimics the behavior of a channel but does not block when items are sent. + +### Features + +- Send items without ever worrying that the send will block +- Check how many items are waiting to be received +- Synchronized access to members - use it from any goroutine + +### Usage + +To use the package, add the following import: + + import "github.com/hectane/go-nonblockingchan" + +Use the `New()` function to create a new instance: + + c := nbc.New() + +To send an item on the channel, use the `Send` field: + + c.Send <- true + +Sending will always succeed immediately. The item will be added to an internal buffer until it is received: + + v, ok := <-c.Recv + if ok { + // value was received + } else { + // channel was closed + } diff --git a/vendor/github.com/hectane/go-nonblockingchan/chan.go b/vendor/github.com/hectane/go-nonblockingchan/chan.go new file mode 100644 index 0000000..3dfc650 --- /dev/null +++ b/vendor/github.com/hectane/go-nonblockingchan/chan.go @@ -0,0 +1,73 @@ +// Non-blocking channel for Go. +package nbc + +import ( + "container/list" + "sync" +) + +// Special type that mimics the behavior of a channel but does not block when +// items are sent. Items are stored internally until received. Closing the Send +// channel will cause the Recv channel to be closed after all items have been +// received. +type NonBlockingChan struct { + mutex sync.Mutex + Send chan<- interface{} + Recv <-chan interface{} + items *list.List +} + +// Loop for buffering items between the Send and Recv channels until the Send +// channel is closed. +func (n *NonBlockingChan) run(send <-chan interface{}, recv chan<- interface{}) { + for { + if send == nil && n.items.Len() == 0 { + close(recv) + break + } + var ( + recvChan chan<- interface{} + recvVal interface{} + ) + if n.items.Len() > 0 { + recvChan = recv + recvVal = n.items.Front().Value + } + select { + case i, ok := <-send: + if ok { + n.mutex.Lock() + n.items.PushBack(i) + n.mutex.Unlock() + } else { + send = nil + } + case recvChan <- recvVal: + n.mutex.Lock() + n.items.Remove(n.items.Front()) + n.mutex.Unlock() + } + } +} + +// Create a new non-blocking channel. +func New() *NonBlockingChan { + var ( + send = make(chan interface{}) + recv = make(chan interface{}) + n = &NonBlockingChan{ + Send: send, + Recv: recv, + items: list.New(), + } + ) + go n.run(send, recv) + return n +} + +// Retrieve the number of items waiting to be received. +func (n *NonBlockingChan) Len() int { + n.mutex.Lock() + defer n.mutex.Unlock() + return n.items.Len() +} diff --git a/vendor/github.com/hectane/hectane/LICENSE.txt b/vendor/github.com/hectane/hectane/LICENSE.txt new file mode 100644 index 0000000..fb72c83 --- /dev/null +++ b/vendor/github.com/hectane/hectane/LICENSE.txt @@ -0,0 +1,9 @@ +The MIT License (MIT) + +Copyright (c) 2015 Nathan Osman + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. diff --git a/vendor/github.com/hectane/hectane/email/attachment.go b/vendor/github.com/hectane/hectane/email/attachment.go new file mode 100644 index 0000000..001fa66 --- /dev/null +++ b/vendor/github.com/hectane/hectane/email/attachment.go @@ -0,0 +1,48 @@ +package email + +import ( + "fmt" + "mime/multipart" + "mime/quotedprintable" + "net/textproto" +) + +// Email attachment. The content of the attachment is provided either as a +// UTF-8 string or as a Base64-encoded string ("encoded" set to "true"). +type Attachment struct { + Filename string `json:"filename"` + ContentType string `json:"content_type"` + Content string `json:"content"` + Encoded bool `json:"encoded"` +} + +// Write the attachment to the specified multipart writer. +func (a Attachment) Write(w *multipart.Writer) error { + headers := make(textproto.MIMEHeader) + if len(a.Filename) != 0 { + headers.Add("Content-Type", fmt.Sprintf("%s; name=%s", a.ContentType, a.Filename)) + } else { + headers.Add("Content-Type", a.ContentType) + } + if a.Encoded { + headers.Add("Content-Transfer-Encoding", "base64") + } else { + headers.Add("Content-Transfer-Encoding", "quoted-printable") + } + p, err := w.CreatePart(headers) + if err != nil { + return err + } + if a.Encoded { + if _, err := p.Write([]byte(a.Content)); err != nil { + return err + } + } else { + q := quotedprintable.NewWriter(p) + if _, err := q.Write([]byte(a.Content)); err != nil { + return err + } + return q.Close() + } + return nil +} diff --git a/vendor/github.com/hectane/hectane/email/email.go b/vendor/github.com/hectane/hectane/email/email.go new file mode 100644 index 0000000..d626997 --- /dev/null +++ b/vendor/github.com/hectane/hectane/email/email.go @@ -0,0 +1,143 @@ +package email + +import ( + "github.com/hectane/hectane/queue" + "github.com/kennygrant/sanitize" + + "bytes" + "fmt" + "io" + "mime/multipart" + "net/mail" + "net/textproto" + "strings" + "time" +) + +// Abstract representation of an email. +type Email struct { + From string `json:"from"` + To []string `json:"to"` + Cc []string `json:"cc"` + Bcc []string `json:"bcc"` + Subject string `json:"subject"` + Headers Headers `json:"headers"` + Text string `json:"text"` + Html string `json:"html"` + Attachments []Attachment `json:"attachments"` +} + +// Write the headers for the email to the specified writer. +func (e *Email) writeHeaders(w io.Writer, id, boundary string) error { + headers := Headers{ + "Message-Id": fmt.Sprintf("<%s@hectane>", id), + "From": e.From, + "To": strings.Join(e.To, ", "), + "Subject": e.Subject, + "Date": time.Now().Format("Mon, 02 Jan 2006 15:04:05 -0700"), + "MIME-Version": "1.0", + "Content-Type": fmt.Sprintf("multipart/mixed; boundary=%s", boundary), + } + for k, v := range e.Headers { + headers[k] = v + } + if len(e.Cc) > 0 { + headers["Cc"] = strings.Join(e.Cc, ", ") + } + return headers.Write(w) +} + +// Write the body of the email to the specified writer. +func (e *Email) writeBody(w *multipart.Writer) error { + var ( + buff = &bytes.Buffer{} + altWriter = multipart.NewWriter(buff) + ) + p, err := w.CreatePart(textproto.MIMEHeader{ + "Content-Type": []string{ + fmt.Sprintf("multipart/alternative; boundary=%s", altWriter.Boundary()), + }, + }) + if err != nil { + return err + } + if e.Text == "" { + e.Text = sanitize.HTML(e.Html) + } + if e.Html == "" { + e.Html = toHTML(e.Text) + } + if err := (Attachment{ + ContentType: "text/plain; charset=utf-8", + Content: e.Text, + }.Write(altWriter)); err != nil { + return err + } + if err := (Attachment{ + ContentType: "text/html; charset=utf-8", + Content: e.Html, + }.Write(altWriter)); err != nil { + return err + } + if err := altWriter.Close(); err != nil { + return err + } + if _, err := io.Copy(p, buff); err != nil { + return err + } + return nil +} + +// Create an array of messages with the specified body. +func (e *Email) newMessages(s *queue.Storage, from, body string) ([]*queue.Message, error) { + addresses := append(append(e.To, e.Cc...), e.Bcc...) + m, err := GroupAddressesByHost(addresses) + if err != nil { + return nil, err + } + messages := make([]*queue.Message, 0, 1) + for h, to := range m { + msg := &queue.Message{ + Host: h, + From: from, + To: to, + } + if err := s.SaveMessage(msg, body); err != nil { + return nil, err + } + messages = append(messages, msg) + } + return messages, nil +} + +// Convert the email into an array of messages grouped by host suitable for +// delivery to the mail queue. +func (e *Email) Messages(s *queue.Storage) ([]*queue.Message, error) { + from, err := mail.ParseAddress(e.From) + if err != nil { + return nil, err + } + w, body, err := s.NewBody() + if err != nil { + return nil, err + } + mpWriter := multipart.NewWriter(w) + if err := e.writeHeaders(w, body, mpWriter.Boundary()); err != nil { + return nil, err + } + if err := e.writeBody(mpWriter); err != nil { + return nil, err + } + for _, a := range e.Attachments { + if err := a.Write(mpWriter); err != nil { + return nil, err + } + } + if err := mpWriter.Close(); err != nil { + return nil, err + } + if err := w.Close(); err != nil { + return nil, err + } + return e.newMessages(s, from.Address, body) +} diff --git a/vendor/github.com/hectane/hectane/email/headers.go b/vendor/github.com/hectane/hectane/email/headers.go new file mode 100644 index 0000000..04ceb53 --- /dev/null +++ b/vendor/github.com/hectane/hectane/email/headers.go @@ -0,0 +1,20 @@ +package email + +import ( + "fmt" + "io" +) + +// Map of email headers. +type Headers map[string]string + +// Write the headers to the specified io.Writer. +func (e Headers) Write(w io.Writer) error { + for k, v := range e { + if _, err := w.Write([]byte(fmt.Sprintf("%s: %s\r\n", k, v))); err != nil { + return err + } + } + _, err := w.Write([]byte("\r\n")) + return err +} diff --git a/vendor/github.com/hectane/hectane/email/util.go b/vendor/github.com/hectane/hectane/email/util.go new file mode 100644 index 0000000..9ca4711 --- /dev/null +++ b/vendor/github.com/hectane/hectane/email/util.go @@ -0,0 +1,44 @@ +package email + +import ( + "fmt" + "html" + "net/mail" + "regexp" + "strings" +) + +// The link RegExp is adapted from http://stackoverflow.com/a/3809435/193619. +var ( + replaceCRLF = regexp.MustCompile(`\r?\n`) + replaceLinks = regexp.MustCompile(`https?:\/\/[-a-zA-Z0-9@:%._\+~#=]{2,256}\.[a-z]{2,6}\b(?:[-a-zA-Z0-9@:%_\+.~#?&//=]*)`) +) + +// Group a list of email addresses by their host. An error will be returned if +// any of the addresses are invalid. +func GroupAddressesByHost(addrs []string) (map[string][]string, error) { + m := make(map[string][]string) + for _, addr := range addrs { + a, err := mail.ParseAddress(addr) + if err != nil { + return nil, err + } + parts := strings.Split(a.Address, "@") + if m[parts[1]] == nil { + m[parts[1]] = make([]string, 0, 1) + } + m[parts[1]] = append(m[parts[1]], a.Address) + } + return m, nil +} + +// Convert the specified text to its HTML equivalent, preserving formatting +// where possible and converting URLs to elements. +func toHTML(data string) string { + data = html.EscapeString(data) + data = replaceCRLF.ReplaceAllString(data, "
") + data = replaceLinks.ReplaceAllStringFunc(data, func(m string) string { + return fmt.Sprintf("%s", m, m) + }) + return data +} diff --git a/vendor/github.com/hectane/hectane/queue/config.go b/vendor/github.com/hectane/hectane/queue/config.go new file mode 100644 index 0000000..0f99742 --- /dev/null +++ b/vendor/github.com/hectane/hectane/queue/config.go @@ -0,0 +1,7 @@ +package queue + +// Application configuration. +type Config struct { + Directory string `json:"directory"` + DisableSSLVerification bool `json:"disable-ssl-verification"` +} diff --git a/vendor/github.com/hectane/hectane/queue/host.go b/vendor/github.com/hectane/hectane/queue/host.go new file mode 100644 index 0000000..579205b --- /dev/null +++ b/vendor/github.com/hectane/hectane/queue/host.go @@ -0,0 +1,296 @@ +package queue + +import ( + "github.com/Sirupsen/logrus" + "github.com/hectane/go-nonblockingchan" + + "crypto/tls" + "errors" + "fmt" + "io" + "net" + "net/mail" + "net/smtp" + "net/textproto" + "strings" + "sync" + "syscall" + "time" +) + +// Host status information. +type HostStatus struct { + Active bool `json:"active"` + Length int `json:"length"` +} + +// Persistent connection to an SMTP host. +type Host struct { + m sync.Mutex + config *Config + storage *Storage + log *logrus.Entry + host string + newMessage *nbc.NonBlockingChan + lastActivity time.Time + stop chan bool +} + +// Receive the next message in the queue. The host queue is considered +// "inactive" while waiting for new messages to arrive. The current time is +// recorded before entering the select{} block so that the Idle() method can +// calculate the idle time. +func (h *Host) receiveMessage() *Message { + h.m.Lock() + h.lastActivity = time.Now() + h.m.Unlock() + defer func() { + h.m.Lock() + h.lastActivity = time.Time{} + h.m.Unlock() + }() + for { + select { + case i := <-h.newMessage.Recv: + return i.(*Message) + case <-h.stop: + return nil + } + } +} + +// Parse an email address and extract the hostname. +func (h *Host) parseHostname(addr string) (string, error) { + a, err := mail.ParseAddress(addr) + if err != nil { + return "", err + } + return strings.Split(a.Address, "@")[1], nil +} + +// Attempt to connect to the specified server. The connection attempt is +// performed in a separate goroutine, allowing it to be aborted if the host +// queue is shut down. +func (h *Host) tryMailServer(server, hostname string) (*smtp.Client, error) { + var ( + c *smtp.Client + err error + done = make(chan bool) + ) + go func() { + c, err = smtp.Dial(fmt.Sprintf("%s:25", server)) + close(done) + }() + select { + case <-done: + case <-h.stop: + return nil, nil + } + if err != nil { + return nil, err + } + if err := c.Hello(hostname); err != nil { + return nil, err + } + if ok, _ := c.Extension("STARTTLS"); ok { + config := &tls.Config{ServerName: server} + if h.config.DisableSSLVerification { + config.InsecureSkipVerify = true + } + if err := c.StartTLS(config); err != nil { + return nil, err + } + } + return c, nil +} + +// Attempt to find the mail servers for the specified host. MX records are +// checked first. If one or more were found, the records are converted into an +// array of strings (sorted by priority). If none were found, the original host +// is returned. +func (h *Host) findMailServers(host string) []string { + r, err := net.LookupMX(host) + if err != nil { + return []string{host} + } + servers := make([]string, len(r)) + for i, r := range r { + servers[i] = strings.TrimSuffix(r.Host, ".") + } + return servers +} + +// Attempt to connect to one of the mail servers. +func (h *Host) connectToMailServer(hostname string) (*smtp.Client, error) { + for _, s := range h.findMailServers(h.host) { + c, err := h.tryMailServer(s, hostname) + if err != nil { + h.log.Debugf("unable to connect to %s", s) + continue + } + return c, nil + } + return nil, errors.New("unable to connect to a mail server") +} + +// Attempt to send the specified message to the specified client. +func (h *Host) deliverToMailServer(c *smtp.Client, m *Message) error { + r, err := h.storage.GetMessageBody(m) + if err != nil { + return err + } + defer r.Close() + if err := c.Mail(m.From); err != nil { + return err + } + for _, t := range m.To { + if err := c.Rcpt(t); err != nil { + return err + } + } + w, err := c.Data() + if err != nil { + return err + } + defer w.Close() + if _, err := io.Copy(w, r); err != nil { + return err + } + return nil +} + +// Receive message and deliver them to their recipients. Due to the complicated +// algorithm for message delivery, the body of the method is broken up into a +// sequence of labeled sections. +func (h *Host) run() { + defer close(h.stop) + var ( + m *Message + hostname string + c *smtp.Client + err error + tries int + duration = time.Minute + ) +receive: + if m == nil { + m = h.receiveMessage() + if m == nil { + goto shutdown + } + h.log.Info("message received in queue") + } + hostname, err = h.parseHostname(m.From) + if err != nil { + h.log.Error(err.Error()) + goto cleanup + } +deliver: + if c == nil { + h.log.Debug("connecting to mail server") + c, err = h.connectToMailServer(hostname) + if c == nil { + if err != nil { + h.log.Error(err) + goto wait + } else { + goto shutdown + } + } + h.log.Debug("connection established") + } + err = h.deliverToMailServer(c, m) + if err != nil { + h.log.Error(err) + if _, ok := err.(syscall.Errno); ok { + c = nil + goto deliver + } + if e, ok := err.(*textproto.Error); ok { + if e.Code >= 400 && e.Code <= 499 { + c.Close() + c = nil + goto wait + } + c.Reset() + } + h.log.Error(err.Error()) + goto cleanup + } + h.log.Info("message delivered successfully") +cleanup: + h.log.Debug("deleting message from disk") + err = h.storage.DeleteMessage(m) + if err != nil { + h.log.Error(err.Error()) + } + m = nil + tries = 0 + goto receive +wait: + // We differ a tiny bit from the RFC spec here but this should work well + // enough - the goal is to retry lots of times early on and space out the + // remaining attempts as time goes on. (Roughly 48 hours total.) + switch { + case tries < 8: + duration *= 2 + case tries < 18: + default: + h.log.Error("maximum retry count exceeded") + goto cleanup + } + select { + case <-h.stop: + case <-time.After(duration): + goto receive + } + tries++ +shutdown: + h.log.Debug("shutting down") + if c != nil { + c.Close() + } +} + +// Create a new host connection. +func NewHost(host string, s *Storage, c *Config) *Host { + h := &Host{ + config: c, + storage: s, + log: logrus.WithField("context", host), + host: host, + newMessage: nbc.New(), + stop: make(chan bool), + } + go h.run() + return h +} + +// Attempt to deliver a message to the host. +func (h *Host) Deliver(m *Message) { + h.newMessage.Send <- m +} + +// Retrieve the connection idle time. +func (h *Host) Idle() time.Duration { + h.m.Lock() + defer h.m.Unlock() + if h.lastActivity.IsZero() { + return 0 + } + return time.Since(h.lastActivity) +} + +// Return the status of the host connection. +func (h *Host) Status() *HostStatus { + return &HostStatus{ + Active: h.Idle() == 0, + Length: h.newMessage.Len(), + } +} + +// Close the connection to the host. +func (h *Host) Stop() { + h.stop <- true + <-h.stop +} diff --git a/vendor/github.com/hectane/hectane/queue/queue.go b/vendor/github.com/hectane/hectane/queue/queue.go new file mode 100644 index 0000000..5504003 --- /dev/null +++ b/vendor/github.com/hectane/hectane/queue/queue.go @@ -0,0 +1,127 @@ +package queue + +import ( + "github.com/Sirupsen/logrus" + + "time" +) + +// Queue status information. +type QueueStatus struct { + Uptime int `json:"uptime"` + Hosts map[string]*HostStatus `json:"hosts"` +} + +// Mail queue managing the sending of messages to hosts. +type Queue struct { + config *Config + Storage *Storage + log *logrus.Entry + hosts map[string]*Host + newMessage chan *Message + getStats chan chan *QueueStatus + stop chan bool +} + +// Deliver the specified message to the appropriate host queue. +func (q *Queue) deliverMessage(m *Message) { + if _, ok := q.hosts[m.Host]; !ok { + q.hosts[m.Host] = NewHost(m.Host, q.Storage, q.config) + } + q.hosts[m.Host].Deliver(m) +} + +// Generate stats for the queue. This is done by obtaining the information +// asynchronously and delivering it on the supplied channel when available. +func (q *Queue) stats(c chan *QueueStatus, startTime time.Time) { + go func() { + s := &QueueStatus{ + Uptime: int(time.Now().Sub(startTime) / time.Second), + Hosts: map[string]*HostStatus{}, + } + for n, h := range q.hosts { + s.Hosts[n] = h.Status() + } + c <- s + close(c) + }() +} + +// Check for inactive host queues and shut them down. +func (q *Queue) checkForInactiveQueues() { + for n, h := range q.hosts { + if h.Idle() > time.Minute { + h.Stop() + delete(q.hosts, n) + } + } +} + +// Receive new messages and deliver them to the specified host queue. Check for +// idle queues every so often and shut them down if they haven't been used. +func (q *Queue) run() { + defer close(q.stop) + startTime := time.Now() + ticker := time.NewTicker(time.Minute) + defer ticker.Stop() +loop: + for { + select { + case m := <-q.newMessage: + q.deliverMessage(m) + case c := <-q.getStats: + q.stats(c, startTime) + case <-ticker.C: + q.checkForInactiveQueues() + case <-q.stop: + break loop + } + } + q.log.Info("stopping host queues") + for h := range q.hosts { + q.hosts[h].Stop() + } + q.log.Info("shutting down") +} + +// Create a new message queue. Any undelivered messages on disk will be added +// to the appropriate queue. +func NewQueue(c *Config) (*Queue, error) { + q := &Queue{ + config: c, + Storage: NewStorage(c.Directory), + log: logrus.WithField("context", "Queue"), + hosts: make(map[string]*Host), + newMessage: make(chan *Message), + getStats: make(chan chan *QueueStatus), + stop: make(chan bool), + } + messages, err := q.Storage.LoadMessages() + if err != nil { + return nil, err + } + q.log.Infof("loaded %d message(s) from %s", len(messages), c.Directory) + for _, m := range messages { + q.deliverMessage(m) + } + go q.run() + return q, nil +} + +// Provide the status of each host queue. +func (q *Queue) Status() *QueueStatus { + c := make(chan *QueueStatus) + q.getStats <- c + return <-c +} + +// Deliver the specified message to the appropriate host queue. +func (q *Queue) Deliver(m *Message) { + q.newMessage <- m +} + +// Stop all active host queues. +func (q *Queue) Stop() { + q.stop <- true + <-q.stop +} diff --git a/vendor/github.com/hectane/hectane/queue/storage.go b/vendor/github.com/hectane/hectane/queue/storage.go new file mode 100644 index 0000000..d1aa414 --- /dev/null +++ b/vendor/github.com/hectane/hectane/queue/storage.go @@ -0,0 +1,160 @@ +package queue + +import ( + "github.com/pborman/uuid" + + "encoding/json" + "io" + "io/ioutil" + "os" + "path" + "strings" + "sync" +) + +const ( + bodyFilename = "body" + messageExtension = ".message" +) + +// Message metadata. +type Message struct { + id string + body string + Host string + From string + To []string +} + +// Manager for message metadata and body on disk. All methods are safe to call +// from multiple goroutines. +type Storage struct { + m sync.Mutex + directory string +} + +// Determine the path to the directory containing the specified body. +func (s *Storage) bodyDirectory(body string) string { + return path.Join(s.directory, body) +} + +// Determine the filename of the specified body. +func (s *Storage) bodyFilename(body string) string { + return path.Join(s.bodyDirectory(body), bodyFilename) +} + +// Determine the filename of the specified message. +func (s *Storage) messageFilename(m *Message) string { + return path.Join(s.directory, m.body, m.id) + messageExtension +} + +// Load all messages with the specified body. +func (s *Storage) loadMessages(body string) []*Message { + messages := make([]*Message, 0, 1) + if files, err := ioutil.ReadDir(s.bodyDirectory(body)); err == nil { + for _, f := range files { + if strings.HasSuffix(f.Name(), messageExtension) { + m := &Message{ + id: strings.TrimSuffix(f.Name(), messageExtension), + body: body, + } + if r, err := os.Open(s.messageFilename(m)); err == nil { + if err := json.NewDecoder(r).Decode(m); err == nil { + messages = append(messages, m) + } + r.Close() + } + } + } + } + return messages +} + +// Create a Storage instance for the specified directory. +func NewStorage(directory string) *Storage { + return &Storage{ + directory: directory, + } +} + +// Create a new message body. The writer must be closed after writing the +// message body. +func (s *Storage) NewBody() (io.WriteCloser, string, error) { + body := uuid.New() + if err := os.MkdirAll(s.bodyDirectory(body), 0700); err != nil { + return nil, "", err + } + w, err := os.OpenFile(s.bodyFilename(body), os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0600) + if err != nil { + return nil, "", err + } + return w, body, nil +} + +// Load messages from the storage directory. Any messages that could not be +// loaded are ignored. +func (s *Storage) LoadMessages() ([]*Message, error) { + directories, err := ioutil.ReadDir(s.directory) + if err != nil { + if !os.IsNotExist(err) { + return nil, err + } + return []*Message{}, nil + } + var messages []*Message + for _, d := range directories { + if d.IsDir() { + if _, err := os.Stat(s.bodyFilename(d.Name())); err == nil { + messages = append(messages, s.loadMessages(d.Name())...) + } + } + } + return messages, nil +} + +// Save the specified message to disk. +func (s *Storage) SaveMessage(m *Message, body string) error { + s.m.Lock() + defer s.m.Unlock() + m.id = uuid.New() + m.body = body + w, err := os.OpenFile(s.messageFilename(m), os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0600) + if err != nil { + return err + } + defer w.Close() + if err := json.NewEncoder(w).Encode(m); err != nil { + return err + } + return nil +} + +// Retreive a reader for the message body. +func (s *Storage) GetMessageBody(m *Message) (io.ReadCloser, error) { + s.m.Lock() + defer s.m.Unlock() + return os.Open(s.bodyFilename(m.body)) +} + +// Delete the specified message. The message body is also deleted if no more +// messages exist. +func (s *Storage) DeleteMessage(m *Message) error { + s.m.Lock() + defer s.m.Unlock() + if err := os.Remove(s.messageFilename(m)); err != nil { + return err + } + d, err := os.Open(s.bodyDirectory(m.body)) + if err != nil { + return err + } + defer d.Close() + e, err := d.Readdir(2) + if err != nil { + return err + } + if len(e) == 1 { + return os.RemoveAll(s.bodyDirectory(m.body)) + } + return nil +} diff --git a/vendor/github.com/jackc/pgx/.gitignore b/vendor/github.com/jackc/pgx/.gitignore new file mode 100644 index 0000000..cb0cd90 --- /dev/null +++ b/vendor/github.com/jackc/pgx/.gitignore @@ -0,0 +1,24 @@ +# Compiled Object files, Static and Dynamic libs (Shared Objects) +*.o +*.a +*.so + +# Folders +_obj +_test + +# Architecture specific extensions/prefixes +*.[568vq] +[568vq].out + +*.cgo1.go +*.cgo2.c +_cgo_defun.c +_cgo_gotypes.go +_cgo_export.* + +_testmain.go + +*.exe + +conn_config_test.go diff --git a/vendor/github.com/jackc/pgx/.travis.yml b/vendor/github.com/jackc/pgx/.travis.yml new file mode 100644 index 0000000..d9ea43b --- /dev/null +++ b/vendor/github.com/jackc/pgx/.travis.yml @@ -0,0 +1,60 @@ +language: go + +go: + - 1.7.4 + - 1.6.4 + - tip + +# Derived from https://github.com/lib/pq/blob/master/.travis.yml +before_install: + - sudo apt-get remove -y --purge postgresql libpq-dev libpq5 postgresql-client-common postgresql-common + - sudo rm -rf /var/lib/postgresql + - wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add - + - sudo sh -c "echo deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main $PGVERSION >> /etc/apt/sources.list.d/postgresql.list" + - sudo apt-get update -qq + - sudo apt-get -y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::="--force-confnew" install postgresql-$PGVERSION postgresql-server-dev-$PGVERSION postgresql-contrib-$PGVERSION + - sudo chmod 777 /etc/postgresql/$PGVERSION/main/pg_hba.conf + - echo "local all postgres trust" > /etc/postgresql/$PGVERSION/main/pg_hba.conf + - echo "local all all trust" >> /etc/postgresql/$PGVERSION/main/pg_hba.conf + - echo "host all pgx_md5 127.0.0.1/32 md5" >> /etc/postgresql/$PGVERSION/main/pg_hba.conf + - echo "host all pgx_pw 127.0.0.1/32 password" >> /etc/postgresql/$PGVERSION/main/pg_hba.conf + - echo "hostssl all pgx_ssl 127.0.0.1/32 md5" >> /etc/postgresql/$PGVERSION/main/pg_hba.conf + - echo "host replication pgx_replication 127.0.0.1/32 md5" >> /etc/postgresql/$PGVERSION/main/pg_hba.conf + - echo "host pgx_test pgx_replication 127.0.0.1/32 md5" >> /etc/postgresql/$PGVERSION/main/pg_hba.conf + - sudo chmod 777 /etc/postgresql/$PGVERSION/main/postgresql.conf + - "[[ $PGVERSION < 9.6 ]] || echo \"wal_level='logical'\" >> /etc/postgresql/$PGVERSION/main/postgresql.conf" + - "[[ $PGVERSION < 9.6 ]] || echo \"max_wal_senders=5\" >> /etc/postgresql/$PGVERSION/main/postgresql.conf" + - "[[ $PGVERSION < 9.6 ]] || echo \"max_replication_slots=5\" >> /etc/postgresql/$PGVERSION/main/postgresql.conf" + - sudo /etc/init.d/postgresql restart + +env: + matrix: + - PGVERSION=9.6 + - PGVERSION=9.5 + - PGVERSION=9.4 + - PGVERSION=9.3 + - PGVERSION=9.2 + +# The tricky test user, below, has to actually exist so that it can be used in a test +# of aclitem formatting. It turns out aclitems cannot contain non-existing users/roles. +before_script: + - mv conn_config_test.go.travis conn_config_test.go + - psql -U postgres -c 'create database pgx_test' + - "[[ \"${PGVERSION}\" = '9.0' ]] && psql -U postgres -f /usr/share/postgresql/9.0/contrib/hstore.sql pgx_test || psql -U postgres pgx_test -c 'create extension hstore'" + - psql -U postgres -c "create user pgx_ssl SUPERUSER PASSWORD 'secret'" + - psql -U postgres -c "create user pgx_md5 SUPERUSER PASSWORD 'secret'" + - psql -U postgres -c "create user pgx_pw SUPERUSER PASSWORD 'secret'" + - psql -U postgres -c "create user pgx_replication with replication password 'secret'" + - psql -U postgres -c "create user \" tricky, ' } \"\" \\ test user \" superuser password 'secret'" + +install: + - go get -u github.com/shopspring/decimal + - go get -u gopkg.in/inconshreveable/log15.v2 + - go get -u github.com/jackc/fake + +script: + - go test -v -race -short ./... + +matrix: + allow_failures: + - go: tip diff --git a/vendor/github.com/jackc/pgx/CHANGELOG.md b/vendor/github.com/jackc/pgx/CHANGELOG.md new file mode 100644 index 0000000..2fa56bb --- /dev/null +++ b/vendor/github.com/jackc/pgx/CHANGELOG.md @@ -0,0 +1,184 @@ +# 2.11.0 (June 5, 2017) + +## Fixes + +* Fix race with concurrent execution of stdlib.OpenFromConnPool (Terin Stock) + +## Features + +* .pgpass support (j7b) +* Add missing CopyFrom delegators to Tx and ConnPool (Jack Christensen) +* Add ParseConnectionString (James Lawrence) + +## Performance + +* Optimize HStore encoding (René Kroon) + +# 2.10.0 (March 17, 2017) + +## Fixes + +* Oid underlying type changed to uint32, previously it was incorrectly int32 (Manni Wood) +* Explicitly close checked-in connections on ConnPool.Reset, previously they were closed by GC + +## Features + +* Add xid type support (Manni Wood) +* Add cid type support (Manni Wood) +* Add tid type support (Manni Wood) +* Add "char" type support (Manni Wood) +* Add NullOid type (Manni Wood) +* Add json/jsonb binary support to allow use with CopyTo +* Add named error ErrAcquireTimeout (Alexander Staubo) +* Add logical replication decoding (Kris Wehner) +* Add PgxScanner interface to allow types to simultaneously support database/sql and pgx (Jack Christensen) +* Add CopyFrom with schema support (Jack Christensen) + +## Compatibility + +* jsonb now defaults to binary format. This means passing a []byte to a jsonb column will no longer work. +* CopyTo is now deprecated but will continue to work. + +# 2.9.0 (August 26, 2016) + +## Fixes + +* Fix *ConnPool.Deallocate() not deleting prepared statement from map +* Fix stdlib not logging unprepared query SQL (Krzysztof Dryś) +* Fix Rows.Values() with varchar binary format +* Concurrent ConnPool.Acquire calls with Dialer timeouts now timeout in the expected amount of time (Konstantin Dzreev) + +## Features + +* Add CopyTo +* Add PrepareEx +* Add basic record to []interface{} decoding +* Encode and decode between all Go and PostgreSQL integer types with bounds checking +* Decode inet/cidr to net.IP +* Encode/decode [][]byte to/from bytea[] +* Encode/decode named types whose underlying types are string, int, int8, int16, int32, int64, uint, uint8, uint16, uint32, uint64 + +## Performance + +* Substantial reduction in memory allocations + +# 2.8.1 (March 24, 2016) + +## Features + +* Scan accepts nil argument to ignore a column + +## Fixes + +* Fix compilation on 32-bit architecture +* Fix Tx.status not being set on error on Commit +* Fix Listen/Unlisten with special characters + +# 2.8.0 (March 18, 2016) + +## Fixes + +* Fix unrecognized commit failure +* Fix msgReader.rxMsg bug when msgReader already has error +* Go float64 can no longer be encoded to a PostgreSQL float4 +* Fix connection corruption when query with error is closed early + +## Features + +This release adds multiple extension points helpful when wrapping pgx with +custom application behavior. pgx can now use custom types designed for the +standard database/sql package such as +[github.com/shopspring/decimal](https://github.com/shopspring/decimal). + +* Add *Tx.AfterClose() hook +* Add *Tx.Conn() +* Add *Tx.Status() +* Add *Tx.Err() +* Add *Rows.AfterClose() hook +* Add *Rows.Conn() +* Add *Conn.SetLogger() to allow changing logger +* Add *Conn.SetLogLevel() to allow changing log level +* Add ConnPool.Reset method +* Add support for database/sql.Scanner and database/sql/driver.Valuer interfaces +* Rows.Scan errors now include which argument caused error +* Add Encode() to allow custom Encoders to reuse internal encoding functionality +* Add Decode() to allow customer Decoders to reuse internal decoding functionality +* Add ConnPool.Prepare method +* Add ConnPool.Deallocate method +* Add Scan to uint32 and uint64 (utrack) +* Add encode and decode to []uint16, []uint32, and []uint64 (Max Musatov) + +## Performance + +* []byte skips encoding/decoding + +# 2.7.1 (October 26, 2015) + +* Disable SSL renegotiation + +# 2.7.0 (October 16, 2015) + +* Add RuntimeParams to ConnConfig +* ParseURI extracts RuntimeParams +* ParseDSN extracts RuntimeParams +* ParseEnvLibpq extracts PGAPPNAME +* Prepare is now idempotent +* Rows.Values now supports oid type +* ConnPool.Release automatically unlistens connections (Joseph Glanville) +* Add trace log level +* Add more efficient log leveling +* Retry automatically on ConnPool.Begin (Joseph Glanville) +* Encode from net.IP to inet and cidr +* Generalize encoding pointer to string to any PostgreSQL type +* Add UUID encoding from pointer to string (Joseph Glanville) +* Add null mapping to pointer to pointer (Jonathan Rudenberg) +* Add JSON and JSONB type support (Joseph Glanville) + +# 2.6.0 (September 3, 2015) + +* Add inet and cidr type support +* Add binary decoding to TimestampOid in stdlib driver (Samuel Stauffer) +* Add support for specifying sslmode in connection strings (Rick Snyder) +* Allow ConnPool to have MaxConnections of 1 +* Add basic PGSSLMODE to support to ParseEnvLibpq +* Add fallback TLS config +* Expose specific error for TSL refused +* More error details exposed in PgError +* Support custom dialer (Lewis Marshall) + +# 2.5.0 (April 15, 2015) + +* Fix stdlib nil support (Blaž Hrastnik) +* Support custom Scanner not reading entire value +* Fix empty array scanning (Laurent Debacker) +* Add ParseDSN (deoxxa) +* Add timestamp support to NullTime +* Remove unused text format scanners +* Return error when too many parameters on Prepare +* Add Travis CI integration (Jonathan Rudenberg) +* Large object support (Jonathan Rudenberg) +* Fix reading null byte arrays (Karl Seguin) +* Add timestamptz[] support +* Add timestamp[] support (Karl Seguin) +* Add bool[] support (Karl Seguin) +* Allow writing []byte into text and varchar columns without type conversion (Hari Bhaskaran) +* Fix ConnPool Close panic +* Add Listen / notify example +* Reduce memory allocations (Karl Seguin) + +# 2.4.0 (October 3, 2014) + +* Add per connection oid to name map +* Add Hstore support (Andy Walker) +* Move introductory docs to godoc from readme +* Fix documentation references to TextEncoder and BinaryEncoder +* Add keep-alive to TCP connections (Andy Walker) +* Add support for EmptyQueryResponse / Allow no-op Exec (Andy Walker) +* Allow reading any type into []byte +* WaitForNotification detects lost connections quicker + +# 2.3.0 (September 16, 2014) + +* Truncate logged strings and byte slices +* Extract more error information from PostgreSQL +* Fix data race with Rows and ConnPool diff --git a/vendor/github.com/jackc/pgx/LICENSE b/vendor/github.com/jackc/pgx/LICENSE new file mode 100644 index 0000000..7dee3da --- /dev/null +++ b/vendor/github.com/jackc/pgx/LICENSE @@ -0,0 +1,22 @@ +Copyright (c) 2013 Jack Christensen + +MIT License + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE +LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. \ No newline at end of file diff --git a/vendor/github.com/jackc/pgx/README.md b/vendor/github.com/jackc/pgx/README.md new file mode 100644 index 0000000..0061c7f --- /dev/null +++ b/vendor/github.com/jackc/pgx/README.md @@ -0,0 +1,125 @@ +[](https://godoc.org/github.com/jackc/pgx) + +# Pgx + +Pgx is a pure Go database connection library designed specifically for +PostgreSQL. Pgx is different from other drivers such as +[pq](http://godoc.org/github.com/lib/pq) because, while it can operate as a +database/sql compatible driver, pgx is primarily intended to be used directly. +It offers a native interface similar to database/sql that offers better +performance and more features. + +## Features + +Pgx supports many additional features beyond what is available through database/sql. + +* Listen / notify +* Transaction isolation level control +* Full TLS connection control +* Binary format support for custom types (can be much faster) +* Copy from protocol support for faster bulk data loads +* Logging support +* Configurable connection pool with after connect hooks to do arbitrary connection setup +* PostgreSQL array to Go slice mapping for integers, floats, and strings +* Hstore support +* JSON and JSONB support +* Maps inet and cidr PostgreSQL types to net.IPNet and net.IP +* Large object support +* Null mapping to Null* struct or pointer to pointer. +* Supports database/sql.Scanner and database/sql/driver.Valuer interfaces for custom types +* Logical replication connections, including receiving WAL and sending standby status updates + +## Performance + +Pgx performs roughly equivalent to [pq](http://godoc.org/github.com/lib/pq) and +[go-pg](https://github.com/go-pg/pg) for selecting a single column from a single +row, but it is substantially faster when selecting multiple entire rows (6893 +queries/sec for pgx vs. 3968 queries/sec for pq -- 73% faster). + +See this [gist](https://gist.github.com/jackc/d282f39e088b495fba3e) for the +underlying benchmark results or checkout +[go_db_bench](https://github.com/jackc/go_db_bench) to run tests for yourself. + +## database/sql + +Import the ```github.com/jackc/pgx/stdlib``` package to use pgx as a driver for +database/sql. It is possible to retrieve a pgx connection from database/sql on +demand. This allows using the database/sql interface in most places, but using +pgx directly when more performance or PostgreSQL specific features are needed. + +## Documentation + +pgx includes extensive documentation in the godoc format. It is viewable online at [godoc.org](https://godoc.org/github.com/jackc/pgx). + +## Testing + +pgx supports multiple connection and authentication types. Setting up a test +environment that can test all of them can be cumbersome. In particular, +Windows cannot test Unix domain socket connections. Because of this pgx will +skip tests for connection types that are not configured. + +### Normal Test Environment + +To setup the normal test environment, first install these dependencies: + + go get github.com/jackc/fake + go get github.com/shopspring/decimal + go get gopkg.in/inconshreveable/log15.v2 + +Then run the following SQL: + + create user pgx_md5 password 'secret'; + create user " tricky, ' } "" \ test user " password 'secret'; + create database pgx_test; + create user pgx_replication with replication password 'secret'; + +Connect to database pgx_test and run: + + create extension hstore; + +Next open conn_config_test.go.example and make a copy without the +.example. If your PostgreSQL server is accepting connections on 127.0.0.1, +then you are done. + +### Connection and Authentication Test Environment + +Complete the normal test environment setup and also do the following. + +Run the following SQL: + + create user pgx_none; + create user pgx_pw password 'secret'; + +Add the following to your pg_hba.conf: + +If you are developing on Unix with domain socket connections: + + local pgx_test pgx_none trust + local pgx_test pgx_pw password + local pgx_test pgx_md5 md5 + +If you are developing on Windows with TCP connections: + + host pgx_test pgx_none 127.0.0.1/32 trust + host pgx_test pgx_pw 127.0.0.1/32 password + host pgx_test pgx_md5 127.0.0.1/32 md5 + +### Replication Test Environment + +Add a replication user: + + create user pgx_replication with replication password 'secret'; + +Add a replication line to your pg_hba.conf: + + host replication pgx_replication 127.0.0.1/32 md5 + +Change the following settings in your postgresql.conf: + + wal_level=logical + max_wal_senders=5 + max_replication_slots=5 + +## Version Policy + +pgx follows semantic versioning for the documented public API on stable releases. Branch `v2` is the latest stable release. `master` can contain new features or behavior that will change or be removed before being merged to the stable `v2` branch (in practice, this occurs very rarely). diff --git a/vendor/github.com/jackc/pgx/conn.go b/vendor/github.com/jackc/pgx/conn.go new file mode 100644 index 0000000..a2d60e7 --- /dev/null +++ b/vendor/github.com/jackc/pgx/conn.go @@ -0,0 +1,1322 @@ +package pgx + +import ( + "bufio" + "crypto/md5" + "crypto/tls" + "encoding/binary" + "encoding/hex" + "errors" + "fmt" + "io" + "net" + "net/url" + "os" + "os/user" + "path/filepath" + "regexp" + "strconv" + "strings" + "time" +) + +// DialFunc is a function that can be used to connect to a PostgreSQL server +type DialFunc func(network, addr string) (net.Conn, error) + +// ConnConfig contains all the options used to establish a connection. +type ConnConfig struct { + Host string // host (e.g. localhost) or path to unix domain socket directory (e.g. /private/tmp) + Port uint16 // default: 5432 + Database string + User string // default: OS user name + Password string + TLSConfig *tls.Config // config for TLS connection -- nil disables TLS + UseFallbackTLS bool // Try FallbackTLSConfig if connecting with TLSConfig fails. Used for preferring TLS, but allowing unencrypted, or vice-versa + FallbackTLSConfig *tls.Config // config for fallback TLS connection (only used if UseFallBackTLS is true)-- nil disables TLS + Logger Logger + LogLevel int + Dial DialFunc + RuntimeParams map[string]string // Run-time parameters to set on connection as session default values (e.g. search_path or application_name) +} + +// Conn is a PostgreSQL connection handle. It is not safe for concurrent usage. +// Use ConnPool to manage access to multiple database connections from multiple +// goroutines. +type Conn struct { + conn net.Conn // the underlying TCP or unix domain socket connection + lastActivityTime time.Time // the last time the connection was used + reader *bufio.Reader // buffered reader to improve read performance + wbuf [1024]byte + writeBuf WriteBuf + Pid int32 // backend pid + SecretKey int32 // key to use to send a cancel query message to the server + RuntimeParams map[string]string // parameters that have been reported by the server + PgTypes map[Oid]PgType // oids to PgTypes + config ConnConfig // config used when establishing this connection + TxStatus byte + preparedStatements map[string]*PreparedStatement + channels map[string]struct{} + notifications []*Notification + alive bool + causeOfDeath error + logger Logger + logLevel int + mr msgReader + fp *fastpath + pgsqlAfInet *byte + pgsqlAfInet6 *byte + busy bool + poolResetCount int + preallocatedRows []Rows +} + +// PreparedStatement is a description of a prepared statement +type PreparedStatement struct { + Name string + SQL string + FieldDescriptions []FieldDescription + ParameterOids []Oid +} + +// PrepareExOptions is an option struct that can be passed to PrepareEx +type PrepareExOptions struct { + ParameterOids []Oid +} + +// Notification is a message received from the PostgreSQL LISTEN/NOTIFY system +type Notification struct { + Pid int32 // backend pid that sent the notification + Channel string // channel from which notification was received + Payload string +} + +// PgType is information about PostgreSQL type and how to encode and decode it +type PgType struct { + Name string // name of type e.g. int4, text, date + DefaultFormat int16 // default format (text or binary) this type will be requested in +} + +// CommandTag is the result of an Exec function +type CommandTag string + +// RowsAffected returns the number of rows affected. If the CommandTag was not +// for a row affecting command (such as "CREATE TABLE") then it returns 0 +func (ct CommandTag) RowsAffected() int64 { + s := string(ct) + index := strings.LastIndex(s, " ") + if index == -1 { + return 0 + } + n, _ := strconv.ParseInt(s[index+1:], 10, 64) + return n +} + +// Identifier a PostgreSQL identifier or name. Identifiers can be composed of +// multiple parts such as ["schema", "table"] or ["table", "column"]. +type Identifier []string + +// Sanitize returns a sanitized string safe for SQL interpolation. +func (ident Identifier) Sanitize() string { + parts := make([]string, len(ident)) + for i := range ident { + parts[i] = `"` + strings.Replace(ident[i], `"`, `""`, -1) + `"` + } + return strings.Join(parts, ".") +} + +// ErrNoRows occurs when rows are expected but none are returned. +var ErrNoRows = errors.New("no rows in result set") + +// ErrNotificationTimeout occurs when WaitForNotification times out. +var ErrNotificationTimeout = errors.New("notification timeout") + +// ErrDeadConn occurs on an attempt to use a dead connection +var ErrDeadConn = errors.New("conn is dead") + +// ErrTLSRefused occurs when the connection attempt requires TLS and the +// PostgreSQL server refuses to use TLS +var ErrTLSRefused = errors.New("server refused TLS connection") + +// ErrConnBusy occurs when the connection is busy (for example, in the middle of +// reading query results) and another action is attempts. +var ErrConnBusy = errors.New("conn is busy") + +// ErrInvalidLogLevel occurs on attempt to set an invalid log level. +var ErrInvalidLogLevel = errors.New("invalid log level") + +// ProtocolError occurs when unexpected data is received from PostgreSQL +type ProtocolError string + +func (e ProtocolError) Error() string { + return string(e) +} + +// Connect establishes a connection with a PostgreSQL server using config. +// config.Host must be specified. config.User will default to the OS user name. +// Other config fields are optional. +func Connect(config ConnConfig) (c *Conn, err error) { + return connect(config, nil, nil, nil) +} + +func connect(config ConnConfig, pgTypes map[Oid]PgType, pgsqlAfInet *byte, pgsqlAfInet6 *byte) (c *Conn, err error) { + c = new(Conn) + + c.config = config + + if pgTypes != nil { + c.PgTypes = make(map[Oid]PgType, len(pgTypes)) + for k, v := range pgTypes { + c.PgTypes[k] = v + } + } + + if pgsqlAfInet != nil { + c.pgsqlAfInet = new(byte) + *c.pgsqlAfInet = *pgsqlAfInet + } + if pgsqlAfInet6 != nil { + c.pgsqlAfInet6 = new(byte) + *c.pgsqlAfInet6 = *pgsqlAfInet6 + } + + if c.config.LogLevel != 0 { + c.logLevel = c.config.LogLevel + } else { + // Preserve pre-LogLevel behavior by defaulting to LogLevelDebug + c.logLevel = LogLevelDebug + } + c.logger = c.config.Logger + c.mr.log = c.log + c.mr.shouldLog = c.shouldLog + + if c.config.User == "" { + user, err := user.Current() + if err != nil { + return nil, err + } + c.config.User = user.Username + if c.shouldLog(LogLevelDebug) { + c.log(LogLevelDebug, "Using default connection config", "User", c.config.User) + } + } + + if c.config.Port == 0 { + c.config.Port = 5432 + if c.shouldLog(LogLevelDebug) { + c.log(LogLevelDebug, "Using default connection config", "Port", c.config.Port) + } + } + + network := "tcp" + address := fmt.Sprintf("%s:%d", c.config.Host, c.config.Port) + // See if host is a valid path, if yes connect with a socket + if _, err := os.Stat(c.config.Host); err == nil { + // For backward compatibility accept socket file paths -- but directories are now preferred + network = "unix" + address = c.config.Host + if !strings.Contains(address, "/.s.PGSQL.") { + address = filepath.Join(address, ".s.PGSQL.") + strconv.FormatInt(int64(c.config.Port), 10) + } + } + if c.config.Dial == nil { + c.config.Dial = (&net.Dialer{KeepAlive: 5 * time.Minute}).Dial + } + + if c.shouldLog(LogLevelInfo) { + c.log(LogLevelInfo, fmt.Sprintf("Dialing PostgreSQL server at %s address: %s", network, address)) + } + err = c.connect(config, network, address, config.TLSConfig) + if err != nil && config.UseFallbackTLS { + if c.shouldLog(LogLevelInfo) { + c.log(LogLevelInfo, fmt.Sprintf("Connect with TLSConfig failed, trying FallbackTLSConfig: %v", err)) + } + err = c.connect(config, network, address, config.FallbackTLSConfig) + } + + if err != nil { + if c.shouldLog(LogLevelError) { + c.log(LogLevelError, fmt.Sprintf("Connect failed: %v", err)) + } + return nil, err + } + + return c, nil +} + +func (c *Conn) connect(config ConnConfig, network, address string, tlsConfig *tls.Config) (err error) { + c.conn, err = c.config.Dial(network, address) + if err != nil { + return err + } + defer func() { + if c != nil && err != nil { + c.conn.Close() + c.alive = false + } + }() + + c.RuntimeParams = make(map[string]string) + c.preparedStatements = make(map[string]*PreparedStatement) + c.channels = make(map[string]struct{}) + c.alive = true + c.lastActivityTime = time.Now() + + if tlsConfig != nil { + if c.shouldLog(LogLevelDebug) { + c.log(LogLevelDebug, "Starting TLS handshake") + } + if err := c.startTLS(tlsConfig); err != nil { + return err + } + } + + c.reader = bufio.NewReader(c.conn) + c.mr.reader = c.reader + + msg := newStartupMessage() + + // Default to disabling TLS renegotiation. + // + // Go does not support (https://github.com/golang/go/issues/5742) + // PostgreSQL recommends disabling (http://www.postgresql.org/docs/9.4/static/runtime-config-connection.html#GUC-SSL-RENEGOTIATION-LIMIT) + if tlsConfig != nil { + msg.options["ssl_renegotiation_limit"] = "0" + } + + // Copy default run-time params + for k, v := range config.RuntimeParams { + msg.options[k] = v + } + + msg.options["user"] = c.config.User + if c.config.Database != "" { + msg.options["database"] = c.config.Database + } + + if err = c.txStartupMessage(msg); err != nil { + return err + } + + for { + var t byte + var r *msgReader + t, r, err = c.rxMsg() + if err != nil { + return err + } + + switch t { + case backendKeyData: + c.rxBackendKeyData(r) + case authenticationX: + if err = c.rxAuthenticationX(r); err != nil { + return err + } + case readyForQuery: + c.rxReadyForQuery(r) + if c.shouldLog(LogLevelInfo) { + c.log(LogLevelInfo, "Connection established") + } + + // Replication connections can't execute the queries to + // populate the c.PgTypes and c.pgsqlAfInet + if _, ok := msg.options["replication"]; ok { + return nil + } + + if c.PgTypes == nil { + err = c.loadPgTypes() + if err != nil { + return err + } + } + + if c.pgsqlAfInet == nil || c.pgsqlAfInet6 == nil { + err = c.loadInetConstants() + if err != nil { + return err + } + } + + return nil + default: + if err = c.processContextFreeMsg(t, r); err != nil { + return err + } + } + } +} + +func (c *Conn) loadPgTypes() error { + rows, err := c.Query(`select t.oid, t.typname +from pg_type t +left join pg_type base_type on t.typelem=base_type.oid +where ( + t.typtype='b' + and (base_type.oid is null or base_type.typtype='b') + ) + or t.typname in('record');`) + if err != nil { + return err + } + + c.PgTypes = make(map[Oid]PgType, 128) + + for rows.Next() { + var oid Oid + var t PgType + + rows.Scan(&oid, &t.Name) + + // The zero value is text format so we ignore any types without a default type format + t.DefaultFormat, _ = DefaultTypeFormats[t.Name] + + c.PgTypes[oid] = t + } + + return rows.Err() +} + +// Family is needed for binary encoding of inet/cidr. The constant is based on +// the server's definition of AF_INET. In theory, this could differ between +// platforms, so request an IPv4 and an IPv6 inet and get the family from that. +func (c *Conn) loadInetConstants() error { + var ipv4, ipv6 []byte + + err := c.QueryRow("select '127.0.0.1'::inet, '1::'::inet").Scan(&ipv4, &ipv6) + if err != nil { + return err + } + + c.pgsqlAfInet = &ipv4[0] + c.pgsqlAfInet6 = &ipv6[0] + + return nil +} + +// Close closes a connection. It is safe to call Close on a already closed +// connection. +func (c *Conn) Close() (err error) { + if !c.IsAlive() { + return nil + } + + wbuf := newWriteBuf(c, 'X') + wbuf.closeMsg() + + _, err = c.conn.Write(wbuf.buf) + + c.die(errors.New("Closed")) + if c.shouldLog(LogLevelInfo) { + c.log(LogLevelInfo, "Closed connection") + } + return err +} + +// ParseURI parses a database URI into ConnConfig +// +// Query parameters not used by the connection process are parsed into ConnConfig.RuntimeParams. +func ParseURI(uri string) (ConnConfig, error) { + var cp ConnConfig + + url, err := url.Parse(uri) + if err != nil { + return cp, err + } + + if url.User != nil { + cp.User = url.User.Username() + cp.Password, _ = url.User.Password() + } + + parts := strings.SplitN(url.Host, ":", 2) + cp.Host = parts[0] + if len(parts) == 2 { + p, err := strconv.ParseUint(parts[1], 10, 16) + if err != nil { + return cp, err + } + cp.Port = uint16(p) + } + cp.Database = strings.TrimLeft(url.Path, "/") + + err = configSSL(url.Query().Get("sslmode"), &cp) + if err != nil { + return cp, err + } + + ignoreKeys := map[string]struct{}{ + "sslmode": {}, + } + + cp.RuntimeParams = make(map[string]string) + + for k, v := range url.Query() { + if _, ok := ignoreKeys[k]; ok { + continue + } + + cp.RuntimeParams[k] = v[0] + } + if cp.Password == "" { + pgpass(&cp) + } + return cp, nil +} + +var dsnRegexp = regexp.MustCompile(`([a-zA-Z_]+)=((?:"[^"]+")|(?:[^ ]+))`) + +// ParseDSN parses a database DSN (data source name) into a ConnConfig +// +// e.g. ParseDSN("user=username password=password host=1.2.3.4 port=5432 dbname=mydb sslmode=disable") +// +// Any options not used by the connection process are parsed into ConnConfig.RuntimeParams. +// +// e.g. ParseDSN("application_name=pgxtest search_path=admin user=username password=password host=1.2.3.4 dbname=mydb") +// +// ParseDSN tries to match libpq behavior with regard to sslmode. See comments +// for ParseEnvLibpq for more information on the security implications of +// sslmode options. +func ParseDSN(s string) (ConnConfig, error) { + var cp ConnConfig + + m := dsnRegexp.FindAllStringSubmatch(s, -1) + + var sslmode string + + cp.RuntimeParams = make(map[string]string) + + for _, b := range m { + switch b[1] { + case "user": + cp.User = b[2] + case "password": + cp.Password = b[2] + case "host": + cp.Host = b[2] + case "port": + p, err := strconv.ParseUint(b[2], 10, 16) + if err != nil { + return cp, err + } + cp.Port = uint16(p) + case "dbname": + cp.Database = b[2] + case "sslmode": + sslmode = b[2] + default: + cp.RuntimeParams[b[1]] = b[2] + } + } + + err := configSSL(sslmode, &cp) + if err != nil { + return cp, err + } + if cp.Password == "" { + pgpass(&cp) + } + return cp, nil +} + +// ParseConnectionString parses either a URI or a DSN connection string. +// see ParseURI and ParseDSN for details. +func ParseConnectionString(s string) (ConnConfig, error) { + if strings.HasPrefix(s, "postgres://") || strings.HasPrefix(s, "postgresql://") { + return ParseURI(s) + } + return ParseDSN(s) +} + +// ParseEnvLibpq parses the environment like libpq does into a ConnConfig +// +// See http://www.postgresql.org/docs/9.4/static/libpq-envars.html for details +// on the meaning of environment variables. +// +// ParseEnvLibpq currently recognizes the following environment variables: +// PGHOST +// PGPORT +// PGDATABASE +// PGUSER +// PGPASSWORD +// PGSSLMODE +// PGAPPNAME +// +// Important TLS Security Notes: +// ParseEnvLibpq tries to match libpq behavior with regard to PGSSLMODE. This +// includes defaulting to "prefer" behavior if no environment variable is set. +// +// See http://www.postgresql.org/docs/9.4/static/libpq-ssl.html#LIBPQ-SSL-PROTECTION +// for details on what level of security each sslmode provides. +// +// "require" and "verify-ca" modes currently are treated as "verify-full". e.g. +// They have stronger security guarantees than they would with libpq. Do not +// rely on this behavior as it may be possible to match libpq in the future. If +// you need full security use "verify-full". +// +// Several of the PGSSLMODE options (including the default behavior of "prefer") +// will set UseFallbackTLS to true and FallbackTLSConfig to a disabled or +// weakened TLS mode. This means that if ParseEnvLibpq is used, but TLSConfig is +// later set from a different source that UseFallbackTLS MUST be set false to +// avoid the possibility of falling back to weaker or disabled security. +func ParseEnvLibpq() (ConnConfig, error) { + var cc ConnConfig + + cc.Host = os.Getenv("PGHOST") + + if pgport := os.Getenv("PGPORT"); pgport != "" { + if port, err := strconv.ParseUint(pgport, 10, 16); err == nil { + cc.Port = uint16(port) + } else { + return cc, err + } + } + + cc.Database = os.Getenv("PGDATABASE") + cc.User = os.Getenv("PGUSER") + cc.Password = os.Getenv("PGPASSWORD") + + sslmode := os.Getenv("PGSSLMODE") + + err := configSSL(sslmode, &cc) + if err != nil { + return cc, err + } + + cc.RuntimeParams = make(map[string]string) + if appname := os.Getenv("PGAPPNAME"); appname != "" { + cc.RuntimeParams["application_name"] = appname + } + if cc.Password == "" { + pgpass(&cc) + } + return cc, nil +} + +func configSSL(sslmode string, cc *ConnConfig) error { + // Match libpq default behavior + if sslmode == "" { + sslmode = "prefer" + } + + switch sslmode { + case "disable": + case "allow": + cc.UseFallbackTLS = true + cc.FallbackTLSConfig = &tls.Config{InsecureSkipVerify: true} + case "prefer": + cc.TLSConfig = &tls.Config{InsecureSkipVerify: true} + cc.UseFallbackTLS = true + cc.FallbackTLSConfig = nil + case "require", "verify-ca", "verify-full": + cc.TLSConfig = &tls.Config{ + ServerName: cc.Host, + } + default: + return errors.New("sslmode is invalid") + } + + return nil +} + +// Prepare creates a prepared statement with name and sql. sql can contain placeholders +// for bound parameters. These placeholders are referenced positional as $1, $2, etc. +// +// Prepare is idempotent; i.e. it is safe to call Prepare multiple times with the same +// name and sql arguments. This allows a code path to Prepare and Query/Exec without +// concern for if the statement has already been prepared. +func (c *Conn) Prepare(name, sql string) (ps *PreparedStatement, err error) { + return c.PrepareEx(name, sql, nil) +} + +// PrepareEx creates a prepared statement with name and sql. sql can contain placeholders +// for bound parameters. These placeholders are referenced positional as $1, $2, etc. +// It defers from Prepare as it allows additional options (such as parameter OIDs) to be passed via struct +// +// PrepareEx is idempotent; i.e. it is safe to call PrepareEx multiple times with the same +// name and sql arguments. This allows a code path to PrepareEx and Query/Exec without +// concern for if the statement has already been prepared. +func (c *Conn) PrepareEx(name, sql string, opts *PrepareExOptions) (ps *PreparedStatement, err error) { + if name != "" { + if ps, ok := c.preparedStatements[name]; ok && ps.SQL == sql { + return ps, nil + } + } + + if c.shouldLog(LogLevelError) { + defer func() { + if err != nil { + c.log(LogLevelError, fmt.Sprintf("Prepare `%s` as `%s` failed: %v", name, sql, err)) + } + }() + } + + // parse + wbuf := newWriteBuf(c, 'P') + wbuf.WriteCString(name) + wbuf.WriteCString(sql) + + if opts != nil { + if len(opts.ParameterOids) > 65535 { + return nil, fmt.Errorf("Number of PrepareExOptions ParameterOids must be between 0 and 65535, received %d", len(opts.ParameterOids)) + } + wbuf.WriteInt16(int16(len(opts.ParameterOids))) + for _, oid := range opts.ParameterOids { + wbuf.WriteInt32(int32(oid)) + } + } else { + wbuf.WriteInt16(0) + } + + // describe + wbuf.startMsg('D') + wbuf.WriteByte('S') + wbuf.WriteCString(name) + + // sync + wbuf.startMsg('S') + wbuf.closeMsg() + + _, err = c.conn.Write(wbuf.buf) + if err != nil { + c.die(err) + return nil, err + } + + ps = &PreparedStatement{Name: name, SQL: sql} + + var softErr error + + for { + var t byte + var r *msgReader + t, r, err := c.rxMsg() + if err != nil { + return nil, err + } + + switch t { + case parseComplete: + case parameterDescription: + ps.ParameterOids = c.rxParameterDescription(r) + + if len(ps.ParameterOids) > 65535 && softErr == nil { + softErr = fmt.Errorf("PostgreSQL supports maximum of 65535 parameters, received %d", len(ps.ParameterOids)) + } + case rowDescription: + ps.FieldDescriptions = c.rxRowDescription(r) + for i := range ps.FieldDescriptions { + t, _ := c.PgTypes[ps.FieldDescriptions[i].DataType] + ps.FieldDescriptions[i].DataTypeName = t.Name + ps.FieldDescriptions[i].FormatCode = t.DefaultFormat + } + case noData: + case readyForQuery: + c.rxReadyForQuery(r) + + if softErr == nil { + c.preparedStatements[name] = ps + } + + return ps, softErr + default: + if e := c.processContextFreeMsg(t, r); e != nil && softErr == nil { + softErr = e + } + } + } +} + +// Deallocate released a prepared statement +func (c *Conn) Deallocate(name string) (err error) { + delete(c.preparedStatements, name) + + // close + wbuf := newWriteBuf(c, 'C') + wbuf.WriteByte('S') + wbuf.WriteCString(name) + + // flush + wbuf.startMsg('H') + wbuf.closeMsg() + + _, err = c.conn.Write(wbuf.buf) + if err != nil { + c.die(err) + return err + } + + for { + var t byte + var r *msgReader + t, r, err := c.rxMsg() + if err != nil { + return err + } + + switch t { + case closeComplete: + return nil + default: + err = c.processContextFreeMsg(t, r) + if err != nil { + return err + } + } + } +} + +// Listen establishes a PostgreSQL listen/notify to channel +func (c *Conn) Listen(channel string) error { + _, err := c.Exec("listen " + quoteIdentifier(channel)) + if err != nil { + return err + } + + c.channels[channel] = struct{}{} + + return nil +} + +// Unlisten unsubscribes from a listen channel +func (c *Conn) Unlisten(channel string) error { + _, err := c.Exec("unlisten " + quoteIdentifier(channel)) + if err != nil { + return err + } + + delete(c.channels, channel) + return nil +} + +// WaitForNotification waits for a PostgreSQL notification for up to timeout. +// If the timeout occurs it returns pgx.ErrNotificationTimeout +func (c *Conn) WaitForNotification(timeout time.Duration) (*Notification, error) { + // Return already received notification immediately + if len(c.notifications) > 0 { + notification := c.notifications[0] + c.notifications = c.notifications[1:] + return notification, nil + } + + stopTime := time.Now().Add(timeout) + + for { + now := time.Now() + + if now.After(stopTime) { + return nil, ErrNotificationTimeout + } + + // If there has been no activity on this connection for a while send a nop message just to ensure + // the connection is alive + nextEnsureAliveTime := c.lastActivityTime.Add(15 * time.Second) + if nextEnsureAliveTime.Before(now) { + // If the server can't respond to a nop in 15 seconds, assume it's dead + err := c.conn.SetReadDeadline(now.Add(15 * time.Second)) + if err != nil { + return nil, err + } + + _, err = c.Exec("--;") + if err != nil { + return nil, err + } + + c.lastActivityTime = now + } + + var deadline time.Time + if stopTime.Before(nextEnsureAliveTime) { + deadline = stopTime + } else { + deadline = nextEnsureAliveTime + } + + notification, err := c.waitForNotification(deadline) + if err != ErrNotificationTimeout { + return notification, err + } + } +} + +func (c *Conn) waitForNotification(deadline time.Time) (*Notification, error) { + var zeroTime time.Time + + for { + // Use SetReadDeadline to implement the timeout. SetReadDeadline will + // cause operations to fail with a *net.OpError that has a Timeout() + // of true. Because the normal pgx rxMsg path considers any error to + // have potentially corrupted the state of the connection, it dies + // on any errors. So to avoid timeout errors in rxMsg we set the + // deadline and peek into the reader. If a timeout error occurs there + // we don't break the pgx connection. If the Peek returns that data + // is available then we turn off the read deadline before the rxMsg. + err := c.conn.SetReadDeadline(deadline) + if err != nil { + return nil, err + } + + // Wait until there is a byte available before continuing onto the normal msg reading path + _, err = c.reader.Peek(1) + if err != nil { + c.conn.SetReadDeadline(zeroTime) // we can only return one error and we already have one -- so ignore possiple error from SetReadDeadline + if err, ok := err.(*net.OpError); ok && err.Timeout() { + return nil, ErrNotificationTimeout + } + return nil, err + } + + err = c.conn.SetReadDeadline(zeroTime) + if err != nil { + return nil, err + } + + var t byte + var r *msgReader + if t, r, err = c.rxMsg(); err == nil { + if err = c.processContextFreeMsg(t, r); err != nil { + return nil, err + } + } else { + return nil, err + } + + if len(c.notifications) > 0 { + notification := c.notifications[0] + c.notifications = c.notifications[1:] + return notification, nil + } + } +} + +func (c *Conn) IsAlive() bool { + return c.alive +} + +func (c *Conn) CauseOfDeath() error { + return c.causeOfDeath +} + +func (c *Conn) sendQuery(sql string, arguments ...interface{}) (err error) { + if ps, present := c.preparedStatements[sql]; present { + return c.sendPreparedQuery(ps, arguments...) + } + return c.sendSimpleQuery(sql, arguments...) +} + +func (c *Conn) sendSimpleQuery(sql string, args ...interface{}) error { + + if len(args) == 0 { + wbuf := newWriteBuf(c, 'Q') + wbuf.WriteCString(sql) + wbuf.closeMsg() + + _, err := c.conn.Write(wbuf.buf) + if err != nil { + c.die(err) + return err + } + + return nil + } + + ps, err := c.Prepare("", sql) + if err != nil { + return err + } + + return c.sendPreparedQuery(ps, args...) +} + +func (c *Conn) sendPreparedQuery(ps *PreparedStatement, arguments ...interface{}) (err error) { + if len(ps.ParameterOids) != len(arguments) { + return fmt.Errorf("Prepared statement \"%v\" requires %d parameters, but %d were provided", ps.Name, len(ps.ParameterOids), len(arguments)) + } + + // bind + wbuf := newWriteBuf(c, 'B') + wbuf.WriteByte(0) + wbuf.WriteCString(ps.Name) + + wbuf.WriteInt16(int16(len(ps.ParameterOids))) + for i, oid := range ps.ParameterOids { + switch arg := arguments[i].(type) { + case Encoder: + wbuf.WriteInt16(arg.FormatCode()) + case string, *string: + wbuf.WriteInt16(TextFormatCode) + default: + switch oid { + case BoolOid, ByteaOid, Int2Oid, Int4Oid, Int8Oid, Float4Oid, Float8Oid, TimestampTzOid, TimestampTzArrayOid, TimestampOid, TimestampArrayOid, DateOid, BoolArrayOid, ByteaArrayOid, Int2ArrayOid, Int4ArrayOid, Int8ArrayOid, Float4ArrayOid, Float8ArrayOid, TextArrayOid, VarcharArrayOid, OidOid, InetOid, CidrOid, InetArrayOid, CidrArrayOid, RecordOid, JsonOid, JsonbOid: + wbuf.WriteInt16(BinaryFormatCode) + default: + wbuf.WriteInt16(TextFormatCode) + } + } + } + + wbuf.WriteInt16(int16(len(arguments))) + for i, oid := range ps.ParameterOids { + if err := Encode(wbuf, oid, arguments[i]); err != nil { + return err + } + } + + wbuf.WriteInt16(int16(len(ps.FieldDescriptions))) + for _, fd := range ps.FieldDescriptions { + wbuf.WriteInt16(fd.FormatCode) + } + + // execute + wbuf.startMsg('E') + wbuf.WriteByte(0) + wbuf.WriteInt32(0) + + // sync + wbuf.startMsg('S') + wbuf.closeMsg() + + _, err = c.conn.Write(wbuf.buf) + if err != nil { + c.die(err) + } + + return err +} + +// Exec executes sql. sql can be either a prepared statement name or an SQL string. +// arguments should be referenced positionally from the sql string as $1, $2, etc. +func (c *Conn) Exec(sql string, arguments ...interface{}) (commandTag CommandTag, err error) { + if err = c.lock(); err != nil { + return commandTag, err + } + + startTime := time.Now() + c.lastActivityTime = startTime + + defer func() { + if err == nil { + if c.shouldLog(LogLevelInfo) { + endTime := time.Now() + c.log(LogLevelInfo, "Exec", "sql", sql, "args", logQueryArgs(arguments), "time", endTime.Sub(startTime), "commandTag", commandTag) + } + } else { + if c.shouldLog(LogLevelError) { + c.log(LogLevelError, "Exec", "sql", sql, "args", logQueryArgs(arguments), "error", err) + } + } + + if unlockErr := c.unlock(); unlockErr != nil && err == nil { + err = unlockErr + } + }() + + if err = c.sendQuery(sql, arguments...); err != nil { + return + } + + var softErr error + + for { + var t byte + var r *msgReader + t, r, err = c.rxMsg() + if err != nil { + return commandTag, err + } + + switch t { + case readyForQuery: + c.rxReadyForQuery(r) + return commandTag, softErr + case rowDescription: + case dataRow: + case bindComplete: + case commandComplete: + commandTag = CommandTag(r.readCString()) + default: + if e := c.processContextFreeMsg(t, r); e != nil && softErr == nil { + softErr = e + } + } + } +} + +// Processes messages that are not exclusive to one context such as +// authentication or query response. The response to these messages +// is the same regardless of when they occur. +func (c *Conn) processContextFreeMsg(t byte, r *msgReader) (err error) { + switch t { + case 'S': + c.rxParameterStatus(r) + return nil + case errorResponse: + return c.rxErrorResponse(r) + case noticeResponse: + return nil + case emptyQueryResponse: + return nil + case notificationResponse: + c.rxNotificationResponse(r) + return nil + default: + return fmt.Errorf("Received unknown message type: %c", t) + } +} + +func (c *Conn) rxMsg() (t byte, r *msgReader, err error) { + if !c.alive { + return 0, nil, ErrDeadConn + } + + t, err = c.mr.rxMsg() + if err != nil { + c.die(err) + } + + c.lastActivityTime = time.Now() + + if c.shouldLog(LogLevelTrace) { + c.log(LogLevelTrace, "rxMsg", "type", string(t), "msgBytesRemaining", c.mr.msgBytesRemaining) + } + + return t, &c.mr, err +} + +func (c *Conn) rxAuthenticationX(r *msgReader) (err error) { + switch r.readInt32() { + case 0: // AuthenticationOk + case 3: // AuthenticationCleartextPassword + err = c.txPasswordMessage(c.config.Password) + case 5: // AuthenticationMD5Password + salt := r.readString(4) + digestedPassword := "md5" + hexMD5(hexMD5(c.config.Password+c.config.User)+salt) + err = c.txPasswordMessage(digestedPassword) + default: + err = errors.New("Received unknown authentication message") + } + + return +} + +func hexMD5(s string) string { + hash := md5.New() + io.WriteString(hash, s) + return hex.EncodeToString(hash.Sum(nil)) +} + +func (c *Conn) rxParameterStatus(r *msgReader) { + key := r.readCString() + value := r.readCString() + c.RuntimeParams[key] = value +} + +func (c *Conn) rxErrorResponse(r *msgReader) (err PgError) { + for { + switch r.readByte() { + case 'S': + err.Severity = r.readCString() + case 'C': + err.Code = r.readCString() + case 'M': + err.Message = r.readCString() + case 'D': + err.Detail = r.readCString() + case 'H': + err.Hint = r.readCString() + case 'P': + s := r.readCString() + n, _ := strconv.ParseInt(s, 10, 32) + err.Position = int32(n) + case 'p': + s := r.readCString() + n, _ := strconv.ParseInt(s, 10, 32) + err.InternalPosition = int32(n) + case 'q': + err.InternalQuery = r.readCString() + case 'W': + err.Where = r.readCString() + case 's': + err.SchemaName = r.readCString() + case 't': + err.TableName = r.readCString() + case 'c': + err.ColumnName = r.readCString() + case 'd': + err.DataTypeName = r.readCString() + case 'n': + err.ConstraintName = r.readCString() + case 'F': + err.File = r.readCString() + case 'L': + s := r.readCString() + n, _ := strconv.ParseInt(s, 10, 32) + err.Line = int32(n) + case 'R': + err.Routine = r.readCString() + + case 0: // End of error message + if err.Severity == "FATAL" { + c.die(err) + } + return + default: // Ignore other error fields + r.readCString() + } + } +} + +func (c *Conn) rxBackendKeyData(r *msgReader) { + c.Pid = r.readInt32() + c.SecretKey = r.readInt32() +} + +func (c *Conn) rxReadyForQuery(r *msgReader) { + c.TxStatus = r.readByte() +} + +func (c *Conn) rxRowDescription(r *msgReader) (fields []FieldDescription) { + fieldCount := r.readInt16() + fields = make([]FieldDescription, fieldCount) + for i := int16(0); i < fieldCount; i++ { + f := &fields[i] + f.Name = r.readCString() + f.Table = r.readOid() + f.AttributeNumber = r.readInt16() + f.DataType = r.readOid() + f.DataTypeSize = r.readInt16() + f.Modifier = r.readInt32() + f.FormatCode = r.readInt16() + } + return +} + +func (c *Conn) rxParameterDescription(r *msgReader) (parameters []Oid) { + // Internally, PostgreSQL supports greater than 64k parameters to a prepared + // statement. But the parameter description uses a 16-bit integer for the + // count of parameters. If there are more than 64K parameters, this count is + // wrong. So read the count, ignore it, and compute the proper value from + // the size of the message. + r.readInt16() + parameterCount := r.msgBytesRemaining / 4 + + parameters = make([]Oid, 0, parameterCount) + + for i := int32(0); i < parameterCount; i++ { + parameters = append(parameters, r.readOid()) + } + return +} + +func (c *Conn) rxNotificationResponse(r *msgReader) { + n := new(Notification) + n.Pid = r.readInt32() + n.Channel = r.readCString() + n.Payload = r.readCString() + c.notifications = append(c.notifications, n) +} + +func (c *Conn) startTLS(tlsConfig *tls.Config) (err error) { + err = binary.Write(c.conn, binary.BigEndian, []int32{8, 80877103}) + if err != nil { + return + } + + response := make([]byte, 1) + if _, err = io.ReadFull(c.conn, response); err != nil { + return + } + + if response[0] != 'S' { + return ErrTLSRefused + } + + c.conn = tls.Client(c.conn, tlsConfig) + + return nil +} + +func (c *Conn) txStartupMessage(msg *startupMessage) error { + _, err := c.conn.Write(msg.Bytes()) + return err +} + +func (c *Conn) txPasswordMessage(password string) (err error) { + wbuf := newWriteBuf(c, 'p') + wbuf.WriteCString(password) + wbuf.closeMsg() + + _, err = c.conn.Write(wbuf.buf) + + return err +} + +func (c *Conn) die(err error) { + c.alive = false + c.causeOfDeath = err + c.conn.Close() +} + +func (c *Conn) lock() error { + if c.busy { + return ErrConnBusy + } + c.busy = true + return nil +} + +func (c *Conn) unlock() error { + if !c.busy { + return errors.New("unlock conn that is not busy") + } + c.busy = false + return nil +} + +func (c *Conn) shouldLog(lvl int) bool { + return c.logger != nil && c.logLevel >= lvl +} + +func (c *Conn) log(lvl int, msg string, ctx ...interface{}) { + if c.Pid != 0 { + ctx = append(ctx, "pid", c.Pid) + } + + switch lvl { + case LogLevelTrace: + c.logger.Debug(msg, ctx...) + case LogLevelDebug: + c.logger.Debug(msg, ctx...) + case LogLevelInfo: + c.logger.Info(msg, ctx...) + case LogLevelWarn: + c.logger.Warn(msg, ctx...) + case LogLevelError: + c.logger.Error(msg, ctx...) + } +} + +// SetLogger replaces the current logger and returns the previous logger. +func (c *Conn) SetLogger(logger Logger) Logger { + oldLogger := c.logger + c.logger = logger + return oldLogger +} + +// SetLogLevel replaces the current log level and returns the previous log +// level. +func (c *Conn) SetLogLevel(lvl int) (int, error) { + oldLvl := c.logLevel + + if lvl < LogLevelNone || lvl > LogLevelTrace { + return oldLvl, ErrInvalidLogLevel + } + + c.logLevel = lvl + return lvl, nil +} + +func quoteIdentifier(s string) string { + return `"` + strings.Replace(s, `"`, `""`, -1) + `"` +} diff --git a/vendor/github.com/jackc/pgx/conn_config_test.go.example b/vendor/github.com/jackc/pgx/conn_config_test.go.example new file mode 100644 index 0000000..cac798b --- /dev/null +++ b/vendor/github.com/jackc/pgx/conn_config_test.go.example @@ -0,0 +1,25 @@ +package pgx_test + +import ( + "github.com/jackc/pgx" +) + +var defaultConnConfig *pgx.ConnConfig = &pgx.ConnConfig{Host: "127.0.0.1", User: "pgx_md5", Password: "secret", Database: "pgx_test"} + +// To skip tests for specific connection / authentication types set that connection param to nil +var tcpConnConfig *pgx.ConnConfig = nil +var unixSocketConnConfig *pgx.ConnConfig = nil +var md5ConnConfig *pgx.ConnConfig = nil +var plainPasswordConnConfig *pgx.ConnConfig = nil +var invalidUserConnConfig *pgx.ConnConfig = nil +var tlsConnConfig *pgx.ConnConfig = nil +var customDialerConnConfig *pgx.ConnConfig = nil +var replicationConnConfig *pgx.ConnConfig = nil + +// var tcpConnConfig *pgx.ConnConfig = &pgx.ConnConfig{Host: "127.0.0.1", User: "pgx_md5", Password: "secret", Database: "pgx_test"} +// var unixSocketConnConfig *pgx.ConnConfig = &pgx.ConnConfig{Host: "/private/tmp", User: "pgx_none", Database: "pgx_test"} +// var md5ConnConfig *pgx.ConnConfig = &pgx.ConnConfig{Host: "127.0.0.1", User: "pgx_md5", Password: "secret", Database: "pgx_test"} +// var plainPasswordConnConfig *pgx.ConnConfig = &pgx.ConnConfig{Host: "127.0.0.1", User: "pgx_pw", Password: "secret", Database: "pgx_test"} +// var invalidUserConnConfig *pgx.ConnConfig = &pgx.ConnConfig{Host: "127.0.0.1", User: "invalid", Database: "pgx_test"} +// var tlsConnConfig *pgx.ConnConfig = &pgx.ConnConfig{Host: "127.0.0.1", User: "pgx_md5", Password: "secret", Database: "pgx_test", TLSConfig: &tls.Config{InsecureSkipVerify: true}} +// var customDialerConnConfig *pgx.ConnConfig = &pgx.ConnConfig{Host: "127.0.0.1", User: "pgx_md5", Password: "secret", Database: "pgx_test"} diff --git a/vendor/github.com/jackc/pgx/conn_config_test.go.travis b/vendor/github.com/jackc/pgx/conn_config_test.go.travis new file mode 100644 index 0000000..75714bf --- /dev/null +++ b/vendor/github.com/jackc/pgx/conn_config_test.go.travis @@ -0,0 +1,30 @@ +package pgx_test + +import ( + "crypto/tls" + "github.com/jackc/pgx" + "os" + "strconv" +) + +var defaultConnConfig = &pgx.ConnConfig{Host: "127.0.0.1", User: "pgx_md5", Password: "secret", Database: "pgx_test"} +var tcpConnConfig = &pgx.ConnConfig{Host: "127.0.0.1", User: "pgx_md5", Password: "secret", Database: "pgx_test"} +var unixSocketConnConfig = &pgx.ConnConfig{Host: "/var/run/postgresql", User: "postgres", Database: "pgx_test"} +var md5ConnConfig = &pgx.ConnConfig{Host: "127.0.0.1", User: "pgx_md5", Password: "secret", Database: "pgx_test"} +var plainPasswordConnConfig = &pgx.ConnConfig{Host: "127.0.0.1", User: "pgx_pw", Password: "secret", Database: "pgx_test"} +var invalidUserConnConfig = &pgx.ConnConfig{Host: "127.0.0.1", User: "invalid", Database: "pgx_test"} +var tlsConnConfig = &pgx.ConnConfig{Host: "127.0.0.1", User: "pgx_ssl", Password: "secret", Database: "pgx_test", TLSConfig: &tls.Config{InsecureSkipVerify: true}} +var customDialerConnConfig = &pgx.ConnConfig{Host: "127.0.0.1", User: "pgx_md5", Password: "secret", Database: "pgx_test"} +var replicationConnConfig *pgx.ConnConfig = nil + +func init() { + version := os.Getenv("PGVERSION") + + if len(version) > 0 { + v, err := strconv.ParseFloat(version,64) + if err == nil && v >= 9.6 { + replicationConnConfig = &pgx.ConnConfig{Host: "127.0.0.1", User: "pgx_replication", Password: "secret", Database: "pgx_test"} + } + } +} + diff --git a/vendor/github.com/jackc/pgx/conn_pool.go b/vendor/github.com/jackc/pgx/conn_pool.go new file mode 100644 index 0000000..1913699 --- /dev/null +++ b/vendor/github.com/jackc/pgx/conn_pool.go @@ -0,0 +1,529 @@ +package pgx + +import ( + "errors" + "sync" + "time" +) + +type ConnPoolConfig struct { + ConnConfig + MaxConnections int // max simultaneous connections to use, default 5, must be at least 2 + AfterConnect func(*Conn) error // function to call on every new connection + AcquireTimeout time.Duration // max wait time when all connections are busy (0 means no timeout) +} + +type ConnPool struct { + allConnections []*Conn + availableConnections []*Conn + cond *sync.Cond + config ConnConfig // config used when establishing connection + inProgressConnects int + maxConnections int + resetCount int + afterConnect func(*Conn) error + logger Logger + logLevel int + closed bool + preparedStatements map[string]*PreparedStatement + acquireTimeout time.Duration + pgTypes map[Oid]PgType + pgsqlAfInet *byte + pgsqlAfInet6 *byte + txAfterClose func(tx *Tx) + rowsAfterClose func(rows *Rows) +} + +type ConnPoolStat struct { + MaxConnections int // max simultaneous connections to use + CurrentConnections int // current live connections + AvailableConnections int // unused live connections +} + +// ErrAcquireTimeout occurs when an attempt to acquire a connection times out. +var ErrAcquireTimeout = errors.New("timeout acquiring connection from pool") + +// NewConnPool creates a new ConnPool. config.ConnConfig is passed through to +// Connect directly. +func NewConnPool(config ConnPoolConfig) (p *ConnPool, err error) { + p = new(ConnPool) + p.config = config.ConnConfig + p.maxConnections = config.MaxConnections + if p.maxConnections == 0 { + p.maxConnections = 5 + } + if p.maxConnections < 1 { + return nil, errors.New("MaxConnections must be at least 1") + } + p.acquireTimeout = config.AcquireTimeout + if p.acquireTimeout < 0 { + return nil, errors.New("AcquireTimeout must be equal to or greater than 0") + } + + p.afterConnect = config.AfterConnect + + if config.LogLevel != 0 { + p.logLevel = config.LogLevel + } else { + // Preserve pre-LogLevel behavior by defaulting to LogLevelDebug + p.logLevel = LogLevelDebug + } + p.logger = config.Logger + if p.logger == nil { + p.logLevel = LogLevelNone + } + + p.txAfterClose = func(tx *Tx) { + p.Release(tx.Conn()) + } + + p.rowsAfterClose = func(rows *Rows) { + p.Release(rows.Conn()) + } + + p.allConnections = make([]*Conn, 0, p.maxConnections) + p.availableConnections = make([]*Conn, 0, p.maxConnections) + p.preparedStatements = make(map[string]*PreparedStatement) + p.cond = sync.NewCond(new(sync.Mutex)) + + // Initially establish one connection + var c *Conn + c, err = p.createConnection() + if err != nil { + return + } + p.allConnections = append(p.allConnections, c) + p.availableConnections = append(p.availableConnections, c) + + return +} + +// Acquire takes exclusive use of a connection until it is released. +func (p *ConnPool) Acquire() (*Conn, error) { + p.cond.L.Lock() + c, err := p.acquire(nil) + p.cond.L.Unlock() + return c, err +} + +// deadlinePassed returns true if the given deadline has passed. +func (p *ConnPool) deadlinePassed(deadline *time.Time) bool { + return deadline != nil && time.Now().After(*deadline) +} + +// acquire performs acquision assuming pool is already locked +func (p *ConnPool) acquire(deadline *time.Time) (*Conn, error) { + if p.closed { + return nil, errors.New("cannot acquire from closed pool") + } + + // A connection is available + if len(p.availableConnections) > 0 { + c := p.availableConnections[len(p.availableConnections)-1] + c.poolResetCount = p.resetCount + p.availableConnections = p.availableConnections[:len(p.availableConnections)-1] + return c, nil + } + + // Set initial timeout/deadline value. If the method (acquire) happens to + // recursively call itself the deadline should retain its value. + if deadline == nil && p.acquireTimeout > 0 { + tmp := time.Now().Add(p.acquireTimeout) + deadline = &tmp + } + + // Make sure the deadline (if it is) has not passed yet + if p.deadlinePassed(deadline) { + return nil, ErrAcquireTimeout + } + + // If there is a deadline then start a timeout timer + var timer *time.Timer + if deadline != nil { + timer = time.AfterFunc(deadline.Sub(time.Now()), func() { + p.cond.Broadcast() + }) + defer timer.Stop() + } + + // No connections are available, but we can create more + if len(p.allConnections)+p.inProgressConnects < p.maxConnections { + // Create a new connection. + // Careful here: createConnectionUnlocked() removes the current lock, + // creates a connection and then locks it back. + c, err := p.createConnectionUnlocked() + if err != nil { + return nil, err + } + c.poolResetCount = p.resetCount + p.allConnections = append(p.allConnections, c) + return c, nil + } + // All connections are in use and we cannot create more + if p.logLevel >= LogLevelWarn { + p.logger.Warn("All connections in pool are busy - waiting...") + } + + // Wait until there is an available connection OR room to create a new connection + for len(p.availableConnections) == 0 && len(p.allConnections)+p.inProgressConnects == p.maxConnections { + if p.deadlinePassed(deadline) { + return nil, ErrAcquireTimeout + } + p.cond.Wait() + } + + // Stop the timer so that we do not spawn it on every acquire call. + if timer != nil { + timer.Stop() + } + return p.acquire(deadline) +} + +// Release gives up use of a connection. +func (p *ConnPool) Release(conn *Conn) { + if conn.TxStatus != 'I' { + conn.Exec("rollback") + } + + if len(conn.channels) > 0 { + if err := conn.Unlisten("*"); err != nil { + conn.die(err) + } + conn.channels = make(map[string]struct{}) + } + conn.notifications = nil + + p.cond.L.Lock() + + if conn.poolResetCount != p.resetCount { + conn.Close() + p.cond.L.Unlock() + p.cond.Signal() + return + } + + if conn.IsAlive() { + p.availableConnections = append(p.availableConnections, conn) + } else { + p.removeFromAllConnections(conn) + } + p.cond.L.Unlock() + p.cond.Signal() +} + +// removeFromAllConnections Removes the given connection from the list. +// It returns true if the connection was found and removed or false otherwise. +func (p *ConnPool) removeFromAllConnections(conn *Conn) bool { + for i, c := range p.allConnections { + if conn == c { + p.allConnections = append(p.allConnections[:i], p.allConnections[i+1:]...) + return true + } + } + return false +} + +// Close ends the use of a connection pool. It prevents any new connections +// from being acquired, waits until all acquired connections are released, +// then closes all underlying connections. +func (p *ConnPool) Close() { + p.cond.L.Lock() + defer p.cond.L.Unlock() + + p.closed = true + + // Wait until all connections are released + if len(p.availableConnections) != len(p.allConnections) { + for len(p.availableConnections) != len(p.allConnections) { + p.cond.Wait() + } + } + + for _, c := range p.allConnections { + _ = c.Close() + } +} + +// Reset closes all open connections, but leaves the pool open. It is intended +// for use when an error is detected that would disrupt all connections (such as +// a network interruption or a server state change). +// +// It is safe to reset a pool while connections are checked out. Those +// connections will be closed when they are returned to the pool. +func (p *ConnPool) Reset() { + p.cond.L.Lock() + defer p.cond.L.Unlock() + + p.resetCount++ + p.allConnections = p.allConnections[0:0] + + for _, conn := range p.availableConnections { + conn.Close() + } + + p.availableConnections = p.availableConnections[0:0] +} + +// invalidateAcquired causes all acquired connections to be closed when released. +// The pool must already be locked. +func (p *ConnPool) invalidateAcquired() { + p.resetCount++ + + for _, c := range p.availableConnections { + c.poolResetCount = p.resetCount + } + + p.allConnections = p.allConnections[:len(p.availableConnections)] + copy(p.allConnections, p.availableConnections) +} + +// Stat returns connection pool statistics +func (p *ConnPool) Stat() (s ConnPoolStat) { + p.cond.L.Lock() + defer p.cond.L.Unlock() + + s.MaxConnections = p.maxConnections + s.CurrentConnections = len(p.allConnections) + s.AvailableConnections = len(p.availableConnections) + return +} + +func (p *ConnPool) createConnection() (*Conn, error) { + c, err := connect(p.config, p.pgTypes, p.pgsqlAfInet, p.pgsqlAfInet6) + if err != nil { + return nil, err + } + return p.afterConnectionCreated(c) +} + +// createConnectionUnlocked Removes the current lock, creates a new connection, and +// then locks it back. +// Here is the point: lets say our pool dialer's OpenTimeout is set to 3 seconds. +// And we have a pool with 20 connections in it, and we try to acquire them all at +// startup. +// If it happens that the remote server is not accessible, then the first connection +// in the pool blocks all the others for 3 secs, before it gets the timeout. Then +// connection #2 holds the lock and locks everything for the next 3 secs until it +// gets OpenTimeout err, etc. And the very last 20th connection will fail only after +// 3 * 20 = 60 secs. +// To avoid this we put Connect(p.config) outside of the lock (it is thread safe) +// what would allow us to make all the 20 connection in parallel (more or less). +func (p *ConnPool) createConnectionUnlocked() (*Conn, error) { + p.inProgressConnects++ + p.cond.L.Unlock() + c, err := Connect(p.config) + p.cond.L.Lock() + p.inProgressConnects-- + + if err != nil { + return nil, err + } + return p.afterConnectionCreated(c) +} + +// afterConnectionCreated executes (if it is) afterConnect() callback and prepares +// all the known statements for the new connection. +func (p *ConnPool) afterConnectionCreated(c *Conn) (*Conn, error) { + p.pgTypes = c.PgTypes + p.pgsqlAfInet = c.pgsqlAfInet + p.pgsqlAfInet6 = c.pgsqlAfInet6 + + if p.afterConnect != nil { + err := p.afterConnect(c) + if err != nil { + c.die(err) + return nil, err + } + } + + for _, ps := range p.preparedStatements { + if _, err := c.Prepare(ps.Name, ps.SQL); err != nil { + c.die(err) + return nil, err + } + } + + return c, nil +} + +// Exec acquires a connection, delegates the call to that connection, and releases the connection +func (p *ConnPool) Exec(sql string, arguments ...interface{}) (commandTag CommandTag, err error) { + var c *Conn + if c, err = p.Acquire(); err != nil { + return + } + defer p.Release(c) + + return c.Exec(sql, arguments...) +} + +// Query acquires a connection and delegates the call to that connection. When +// *Rows are closed, the connection is released automatically. +func (p *ConnPool) Query(sql string, args ...interface{}) (*Rows, error) { + c, err := p.Acquire() + if err != nil { + // Because checking for errors can be deferred to the *Rows, build one with the error + return &Rows{closed: true, err: err}, err + } + + rows, err := c.Query(sql, args...) + if err != nil { + p.Release(c) + return rows, err + } + + rows.AfterClose(p.rowsAfterClose) + + return rows, nil +} + +// QueryRow acquires a connection and delegates the call to that connection. The +// connection is released automatically after Scan is called on the returned +// *Row. +func (p *ConnPool) QueryRow(sql string, args ...interface{}) *Row { + rows, _ := p.Query(sql, args...) + return (*Row)(rows) +} + +// Begin acquires a connection and begins a transaction on it. When the +// transaction is closed the connection will be automatically released. +func (p *ConnPool) Begin() (*Tx, error) { + return p.BeginIso("") +} + +// Prepare creates a prepared statement on a connection in the pool to test the +// statement is valid. If it succeeds all connections accessed through the pool +// will have the statement available. +// +// Prepare creates a prepared statement with name and sql. sql can contain +// placeholders for bound parameters. These placeholders are referenced +// positional as $1, $2, etc. +// +// Prepare is idempotent; i.e. it is safe to call Prepare multiple times with +// the same name and sql arguments. This allows a code path to Prepare and +// Query/Exec/PrepareEx without concern for if the statement has already been prepared. +func (p *ConnPool) Prepare(name, sql string) (*PreparedStatement, error) { + return p.PrepareEx(name, sql, nil) +} + +// PrepareEx creates a prepared statement on a connection in the pool to test the +// statement is valid. If it succeeds all connections accessed through the pool +// will have the statement available. +// +// PrepareEx creates a prepared statement with name and sql. sql can contain placeholders +// for bound parameters. These placeholders are referenced positional as $1, $2, etc. +// It defers from Prepare as it allows additional options (such as parameter OIDs) to be passed via struct +// +// PrepareEx is idempotent; i.e. it is safe to call PrepareEx multiple times with the same +// name and sql arguments. This allows a code path to PrepareEx and Query/Exec/Prepare without +// concern for if the statement has already been prepared. +func (p *ConnPool) PrepareEx(name, sql string, opts *PrepareExOptions) (*PreparedStatement, error) { + p.cond.L.Lock() + defer p.cond.L.Unlock() + + if ps, ok := p.preparedStatements[name]; ok && ps.SQL == sql { + return ps, nil + } + + c, err := p.acquire(nil) + if err != nil { + return nil, err + } + + p.availableConnections = append(p.availableConnections, c) + + // Double check that the statement was not prepared by someone else + // while we were acquiring the connection (since acquire is not fully + // blocking now, see createConnectionUnlocked()) + if ps, ok := p.preparedStatements[name]; ok && ps.SQL == sql { + return ps, nil + } + + ps, err := c.PrepareEx(name, sql, opts) + if err != nil { + return nil, err + } + + for _, c := range p.availableConnections { + _, err := c.PrepareEx(name, sql, opts) + if err != nil { + return nil, err + } + } + + p.invalidateAcquired() + p.preparedStatements[name] = ps + + return ps, err +} + +// Deallocate releases a prepared statement from all connections in the pool. +func (p *ConnPool) Deallocate(name string) (err error) { + p.cond.L.Lock() + defer p.cond.L.Unlock() + + for _, c := range p.availableConnections { + if err := c.Deallocate(name); err != nil { + return err + } + } + + p.invalidateAcquired() + delete(p.preparedStatements, name) + + return nil +} + +// BeginIso acquires a connection and begins a transaction in isolation mode iso +// on it. When the transaction is closed the connection will be automatically +// released. +func (p *ConnPool) BeginIso(iso string) (*Tx, error) { + for { + c, err := p.Acquire() + if err != nil { + return nil, err + } + + tx, err := c.BeginIso(iso) + if err != nil { + alive := c.IsAlive() + p.Release(c) + + // If connection is still alive then the error is not something trying + // again on a new connection would fix, so just return the error. But + // if the connection is dead try to acquire a new connection and try + // again. + if alive { + return nil, err + } + continue + } + + tx.AfterClose(p.txAfterClose) + return tx, nil + } +} + +// Deprecated. Use CopyFrom instead. CopyTo acquires a connection, delegates the +// call to that connection, and releases the connection. +func (p *ConnPool) CopyTo(tableName string, columnNames []string, rowSrc CopyToSource) (int, error) { + c, err := p.Acquire() + if err != nil { + return 0, err + } + defer p.Release(c) + + return c.CopyTo(tableName, columnNames, rowSrc) +} + +// CopyFrom acquires a connection, delegates the call to that connection, and +// releases the connection. +func (p *ConnPool) CopyFrom(tableName Identifier, columnNames []string, rowSrc CopyFromSource) (int, error) { + c, err := p.Acquire() + if err != nil { + return 0, err + } + defer p.Release(c) + + return c.CopyFrom(tableName, columnNames, rowSrc) +} diff --git a/vendor/github.com/jackc/pgx/copy_from.go b/vendor/github.com/jackc/pgx/copy_from.go new file mode 100644 index 0000000..1f8a230 --- /dev/null +++ b/vendor/github.com/jackc/pgx/copy_from.go @@ -0,0 +1,241 @@ +package pgx + +import ( + "bytes" + "fmt" +) + +// CopyFromRows returns a CopyFromSource interface over the provided rows slice +// making it usable by *Conn.CopyFrom. +func CopyFromRows(rows [][]interface{}) CopyFromSource { + return ©FromRows{rows: rows, idx: -1} +} + +type copyFromRows struct { + rows [][]interface{} + idx int +} + +func (ctr *copyFromRows) Next() bool { + ctr.idx++ + return ctr.idx < len(ctr.rows) +} + +func (ctr *copyFromRows) Values() ([]interface{}, error) { + return ctr.rows[ctr.idx], nil +} + +func (ctr *copyFromRows) Err() error { + return nil +} + +// CopyFromSource is the interface used by *Conn.CopyFrom as the source for copy data. +type CopyFromSource interface { + // Next returns true if there is another row and makes the next row data + // available to Values(). When there are no more rows available or an error + // has occurred it returns false. + Next() bool + + // Values returns the values for the current row. + Values() ([]interface{}, error) + + // Err returns any error that has been encountered by the CopyFromSource. If + // this is not nil *Conn.CopyFrom will abort the copy. + Err() error +} + +type copyFrom struct { + conn *Conn + tableName Identifier + columnNames []string + rowSrc CopyFromSource + readerErrChan chan error +} + +func (ct *copyFrom) readUntilReadyForQuery() { + for { + t, r, err := ct.conn.rxMsg() + if err != nil { + ct.readerErrChan <- err + close(ct.readerErrChan) + return + } + + switch t { + case readyForQuery: + ct.conn.rxReadyForQuery(r) + close(ct.readerErrChan) + return + case commandComplete: + case errorResponse: + ct.readerErrChan <- ct.conn.rxErrorResponse(r) + default: + err = ct.conn.processContextFreeMsg(t, r) + if err != nil { + ct.readerErrChan <- ct.conn.processContextFreeMsg(t, r) + } + } + } +} + +func (ct *copyFrom) waitForReaderDone() error { + var err error + for err = range ct.readerErrChan { + } + return err +} + +func (ct *copyFrom) run() (int, error) { + quotedTableName := ct.tableName.Sanitize() + buf := &bytes.Buffer{} + for i, cn := range ct.columnNames { + if i != 0 { + buf.WriteString(", ") + } + buf.WriteString(quoteIdentifier(cn)) + } + quotedColumnNames := buf.String() + + ps, err := ct.conn.Prepare("", fmt.Sprintf("select %s from %s", quotedColumnNames, quotedTableName)) + if err != nil { + return 0, err + } + + err = ct.conn.sendSimpleQuery(fmt.Sprintf("copy %s ( %s ) from stdin binary;", quotedTableName, quotedColumnNames)) + if err != nil { + return 0, err + } + + err = ct.conn.readUntilCopyInResponse() + if err != nil { + return 0, err + } + + go ct.readUntilReadyForQuery() + defer ct.waitForReaderDone() + + wbuf := newWriteBuf(ct.conn, copyData) + + wbuf.WriteBytes([]byte("PGCOPY\n\377\r\n\000")) + wbuf.WriteInt32(0) + wbuf.WriteInt32(0) + + var sentCount int + + for ct.rowSrc.Next() { + select { + case err = <-ct.readerErrChan: + return 0, err + default: + } + + if len(wbuf.buf) > 65536 { + wbuf.closeMsg() + _, err = ct.conn.conn.Write(wbuf.buf) + if err != nil { + ct.conn.die(err) + return 0, err + } + + // Directly manipulate wbuf to reset to reuse the same buffer + wbuf.buf = wbuf.buf[0:5] + wbuf.buf[0] = copyData + wbuf.sizeIdx = 1 + } + + sentCount++ + + values, err := ct.rowSrc.Values() + if err != nil { + ct.cancelCopyIn() + return 0, err + } + if len(values) != len(ct.columnNames) { + ct.cancelCopyIn() + return 0, fmt.Errorf("expected %d values, got %d values", len(ct.columnNames), len(values)) + } + + wbuf.WriteInt16(int16(len(ct.columnNames))) + for i, val := range values { + err = Encode(wbuf, ps.FieldDescriptions[i].DataType, val) + if err != nil { + ct.cancelCopyIn() + return 0, err + } + + } + } + + if ct.rowSrc.Err() != nil { + ct.cancelCopyIn() + return 0, ct.rowSrc.Err() + } + + wbuf.WriteInt16(-1) // terminate the copy stream + + wbuf.startMsg(copyDone) + wbuf.closeMsg() + _, err = ct.conn.conn.Write(wbuf.buf) + if err != nil { + ct.conn.die(err) + return 0, err + } + + err = ct.waitForReaderDone() + if err != nil { + return 0, err + } + return sentCount, nil +} + +func (c *Conn) readUntilCopyInResponse() error { + for { + var t byte + var r *msgReader + t, r, err := c.rxMsg() + if err != nil { + return err + } + + switch t { + case copyInResponse: + return nil + default: + err = c.processContextFreeMsg(t, r) + if err != nil { + return err + } + } + } +} + +func (ct *copyFrom) cancelCopyIn() error { + wbuf := newWriteBuf(ct.conn, copyFail) + wbuf.WriteCString("client error: abort") + wbuf.closeMsg() + _, err := ct.conn.conn.Write(wbuf.buf) + if err != nil { + ct.conn.die(err) + return err + } + + return nil +} + +// CopyFrom uses the PostgreSQL copy protocol to perform bulk data insertion. +// It returns the number of rows copied and an error. +// +// CopyFrom requires all values use the binary format. Almost all types +// implemented by pgx use the binary format by default. Types implementing +// Encoder can only be used if they encode to the binary format. +func (c *Conn) CopyFrom(tableName Identifier, columnNames []string, rowSrc CopyFromSource) (int, error) { + ct := ©From{ + conn: c, + tableName: tableName, + columnNames: columnNames, + rowSrc: rowSrc, + readerErrChan: make(chan error), + } + + return ct.run() +} diff --git a/vendor/github.com/jackc/pgx/copy_to.go b/vendor/github.com/jackc/pgx/copy_to.go new file mode 100644 index 0000000..229e9a4 --- /dev/null +++ b/vendor/github.com/jackc/pgx/copy_to.go @@ -0,0 +1,222 @@ +package pgx + +import ( + "bytes" + "fmt" +) + +// Deprecated. Use CopyFromRows instead. CopyToRows returns a CopyToSource +// interface over the provided rows slice making it usable by *Conn.CopyTo. +func CopyToRows(rows [][]interface{}) CopyToSource { + return ©ToRows{rows: rows, idx: -1} +} + +type copyToRows struct { + rows [][]interface{} + idx int +} + +func (ctr *copyToRows) Next() bool { + ctr.idx++ + return ctr.idx < len(ctr.rows) +} + +func (ctr *copyToRows) Values() ([]interface{}, error) { + return ctr.rows[ctr.idx], nil +} + +func (ctr *copyToRows) Err() error { + return nil +} + +// Deprecated. Use CopyFromSource instead. CopyToSource is the interface used by +// *Conn.CopyTo as the source for copy data. +type CopyToSource interface { + // Next returns true if there is another row and makes the next row data + // available to Values(). When there are no more rows available or an error + // has occurred it returns false. + Next() bool + + // Values returns the values for the current row. + Values() ([]interface{}, error) + + // Err returns any error that has been encountered by the CopyToSource. If + // this is not nil *Conn.CopyTo will abort the copy. + Err() error +} + +type copyTo struct { + conn *Conn + tableName string + columnNames []string + rowSrc CopyToSource + readerErrChan chan error +} + +func (ct *copyTo) readUntilReadyForQuery() { + for { + t, r, err := ct.conn.rxMsg() + if err != nil { + ct.readerErrChan <- err + close(ct.readerErrChan) + return + } + + switch t { + case readyForQuery: + ct.conn.rxReadyForQuery(r) + close(ct.readerErrChan) + return + case commandComplete: + case errorResponse: + ct.readerErrChan <- ct.conn.rxErrorResponse(r) + default: + err = ct.conn.processContextFreeMsg(t, r) + if err != nil { + ct.readerErrChan <- ct.conn.processContextFreeMsg(t, r) + } + } + } +} + +func (ct *copyTo) waitForReaderDone() error { + var err error + for err = range ct.readerErrChan { + } + return err +} + +func (ct *copyTo) run() (int, error) { + quotedTableName := quoteIdentifier(ct.tableName) + buf := &bytes.Buffer{} + for i, cn := range ct.columnNames { + if i != 0 { + buf.WriteString(", ") + } + buf.WriteString(quoteIdentifier(cn)) + } + quotedColumnNames := buf.String() + + ps, err := ct.conn.Prepare("", fmt.Sprintf("select %s from %s", quotedColumnNames, quotedTableName)) + if err != nil { + return 0, err + } + + err = ct.conn.sendSimpleQuery(fmt.Sprintf("copy %s ( %s ) from stdin binary;", quotedTableName, quotedColumnNames)) + if err != nil { + return 0, err + } + + err = ct.conn.readUntilCopyInResponse() + if err != nil { + return 0, err + } + + go ct.readUntilReadyForQuery() + defer ct.waitForReaderDone() + + wbuf := newWriteBuf(ct.conn, copyData) + + wbuf.WriteBytes([]byte("PGCOPY\n\377\r\n\000")) + wbuf.WriteInt32(0) + wbuf.WriteInt32(0) + + var sentCount int + + for ct.rowSrc.Next() { + select { + case err = <-ct.readerErrChan: + return 0, err + default: + } + + if len(wbuf.buf) > 65536 { + wbuf.closeMsg() + _, err = ct.conn.conn.Write(wbuf.buf) + if err != nil { + ct.conn.die(err) + return 0, err + } + + // Directly manipulate wbuf to reset to reuse the same buffer + wbuf.buf = wbuf.buf[0:5] + wbuf.buf[0] = copyData + wbuf.sizeIdx = 1 + } + + sentCount++ + + values, err := ct.rowSrc.Values() + if err != nil { + ct.cancelCopyIn() + return 0, err + } + if len(values) != len(ct.columnNames) { + ct.cancelCopyIn() + return 0, fmt.Errorf("expected %d values, got %d values", len(ct.columnNames), len(values)) + } + + wbuf.WriteInt16(int16(len(ct.columnNames))) + for i, val := range values { + err = Encode(wbuf, ps.FieldDescriptions[i].DataType, val) + if err != nil { + ct.cancelCopyIn() + return 0, err + } + + } + } + + if ct.rowSrc.Err() != nil { + ct.cancelCopyIn() + return 0, ct.rowSrc.Err() + } + + wbuf.WriteInt16(-1) // terminate the copy stream + + wbuf.startMsg(copyDone) + wbuf.closeMsg() + _, err = ct.conn.conn.Write(wbuf.buf) + if err != nil { + ct.conn.die(err) + return 0, err + } + + err = ct.waitForReaderDone() + if err != nil { + return 0, err + } + return sentCount, nil +} + +func (ct *copyTo) cancelCopyIn() error { + wbuf := newWriteBuf(ct.conn, copyFail) + wbuf.WriteCString("client error: abort") + wbuf.closeMsg() + _, err := ct.conn.conn.Write(wbuf.buf) + if err != nil { + ct.conn.die(err) + return err + } + + return nil +} + +// Deprecated. Use CopyFrom instead. CopyTo uses the PostgreSQL copy protocol to +// perform bulk data insertion. It returns the number of rows copied and an +// error. +// +// CopyTo requires all values use the binary format. Almost all types +// implemented by pgx use the binary format by default. Types implementing +// Encoder can only be used if they encode to the binary format. +func (c *Conn) CopyTo(tableName string, columnNames []string, rowSrc CopyToSource) (int, error) { + ct := ©To{ + conn: c, + tableName: tableName, + columnNames: columnNames, + rowSrc: rowSrc, + readerErrChan: make(chan error), + } + + return ct.run() +} diff --git a/vendor/github.com/jackc/pgx/doc.go b/vendor/github.com/jackc/pgx/doc.go new file mode 100644 index 0000000..f527d11 --- /dev/null +++ b/vendor/github.com/jackc/pgx/doc.go @@ -0,0 +1,265 @@ +// Package pgx is a PostgreSQL database driver. +/* +pgx provides lower level access to PostgreSQL than the standard database/sql +It remains as similar to the database/sql interface as possible while +providing better speed and access to PostgreSQL specific features. Import +github.com/jack/pgx/stdlib to use pgx as a database/sql compatible driver. + +Query Interface + +pgx implements Query and Scan in the familiar database/sql style. + + var sum int32 + + // Send the query to the server. The returned rows MUST be closed + // before conn can be used again. + rows, err := conn.Query("select generate_series(1,$1)", 10) + if err != nil { + return err + } + + // rows.Close is called by rows.Next when all rows are read + // or an error occurs in Next or Scan. So it may optionally be + // omitted if nothing in the rows.Next loop can panic. It is + // safe to close rows multiple times. + defer rows.Close() + + // Iterate through the result set + for rows.Next() { + var n int32 + err = rows.Scan(&n) + if err != nil { + return err + } + sum += n + } + + // Any errors encountered by rows.Next or rows.Scan will be returned here + if rows.Err() != nil { + return err + } + + // No errors found - do something with sum + +pgx also implements QueryRow in the same style as database/sql. + + var name string + var weight int64 + err := conn.QueryRow("select name, weight from widgets where id=$1", 42).Scan(&name, &weight) + if err != nil { + return err + } + +Use Exec to execute a query that does not return a result set. + + commandTag, err := conn.Exec("delete from widgets where id=$1", 42) + if err != nil { + return err + } + if commandTag.RowsAffected() != 1 { + return errors.New("No row found to delete") + } + +Connection Pool + +Connection pool usage is explicit and configurable. In pgx, a connection can +be created and managed directly, or a connection pool with a configurable +maximum connections can be used. Also, the connection pool offers an after +connect hook that allows every connection to be automatically setup before +being made available in the connection pool. This is especially useful to +ensure all connections have the same prepared statements available or to +change any other connection settings. + +It delegates Query, QueryRow, Exec, and Begin functions to an automatically +checked out and released connection so you can avoid manually acquiring and +releasing connections when you do not need that level of control. + + var name string + var weight int64 + err := pool.QueryRow("select name, weight from widgets where id=$1", 42).Scan(&name, &weight) + if err != nil { + return err + } + +Base Type Mapping + +pgx maps between all common base types directly between Go and PostgreSQL. In +particular: + + Go PostgreSQL + ----------------------- + string varchar + text + + // Integers are automatically be converted to any other integer type if + // it can be done without overflow or underflow. + int8 + int16 smallint + int32 int + int64 bigint + int + uint8 + uint16 + uint32 + uint64 + uint + + // Floats are strict and do not automatically convert like integers. + float32 float4 + float64 float8 + + time.Time date + timestamp + timestamptz + + []byte bytea + + +Null Mapping + +pgx can map nulls in two ways. The first is Null* types that have a data field +and a valid field. They work in a similar fashion to database/sql. The second +is to use a pointer to a pointer. + + var foo pgx.NullString + var bar *string + err := conn.QueryRow("select foo, bar from widgets where id=$1", 42).Scan(&a, &b) + if err != nil { + return err + } + +Array Mapping + +pgx maps between int16, int32, int64, float32, float64, and string Go slices +and the equivalent PostgreSQL array type. Go slices of native types do not +support nulls, so if a PostgreSQL array that contains a null is read into a +native Go slice an error will occur. + +Hstore Mapping + +pgx includes an Hstore type and a NullHstore type. Hstore is simply a +map[string]string and is preferred when the hstore contains no nulls. NullHstore +follows the Null* pattern and supports null values. + +JSON and JSONB Mapping + +pgx includes built-in support to marshal and unmarshal between Go types and +the PostgreSQL JSON and JSONB. + +Inet and Cidr Mapping + +pgx encodes from net.IPNet to and from inet and cidr PostgreSQL types. In +addition, as a convenience pgx will encode from a net.IP; it will assume a /32 +netmask for IPv4 and a /128 for IPv6. + +Custom Type Support + +pgx includes support for the common data types like integers, floats, strings, +dates, and times that have direct mappings between Go and SQL. Support can be +added for additional types like point, hstore, numeric, etc. that do not have +direct mappings in Go by the types implementing ScannerPgx and Encoder. + +Custom types can support text or binary formats. Binary format can provide a +large performance increase. The natural place for deciding the format for a +value would be in ScannerPgx as it is responsible for decoding the returned +data. However, that is impossible as the query has already been sent by the time +the ScannerPgx is invoked. The solution to this is the global +DefaultTypeFormats. If a custom type prefers binary format it should register it +there. + + pgx.DefaultTypeFormats["point"] = pgx.BinaryFormatCode + +Note that the type is referred to by name, not by OID. This is because custom +PostgreSQL types like hstore will have different OIDs on different servers. When +pgx establishes a connection it queries the pg_type table for all types. It then +matches the names in DefaultTypeFormats with the returned OIDs and stores it in +Conn.PgTypes. + +See example_custom_type_test.go for an example of a custom type for the +PostgreSQL point type. + +pgx also includes support for custom types implementing the database/sql.Scanner +and database/sql/driver.Valuer interfaces. + +Raw Bytes Mapping + +[]byte passed as arguments to Query, QueryRow, and Exec are passed unmodified +to PostgreSQL. In like manner, a *[]byte passed to Scan will be filled with +the raw bytes returned by PostgreSQL. This can be especially useful for reading +varchar, text, json, and jsonb values directly into a []byte and avoiding the +type conversion from string. + +Transactions + +Transactions are started by calling Begin or BeginIso. The BeginIso variant +creates a transaction with a specified isolation level. + + tx, err := conn.Begin() + if err != nil { + return err + } + // Rollback is safe to call even if the tx is already closed, so if + // the tx commits successfully, this is a no-op + defer tx.Rollback() + + _, err = tx.Exec("insert into foo(id) values (1)") + if err != nil { + return err + } + + err = tx.Commit() + if err != nil { + return err + } + +Copy Protocol + +Use CopyFrom to efficiently insert multiple rows at a time using the PostgreSQL +copy protocol. CopyFrom accepts a CopyFromSource interface. If the data is already +in a [][]interface{} use CopyFromRows to wrap it in a CopyFromSource interface. Or +implement CopyFromSource to avoid buffering the entire data set in memory. + + rows := [][]interface{}{ + {"John", "Smith", int32(36)}, + {"Jane", "Doe", int32(29)}, + } + + copyCount, err := conn.CopyFrom( + "people", + []string{"first_name", "last_name", "age"}, + pgx.CopyFromRows(rows), + ) + +CopyFrom can be faster than an insert with as few as 5 rows. + +Listen and Notify + +pgx can listen to the PostgreSQL notification system with the +WaitForNotification function. It takes a maximum time to wait for a +notification. + + err := conn.Listen("channelname") + if err != nil { + return nil + } + + if notification, err := conn.WaitForNotification(time.Second); err != nil { + // do something with notification + } + +TLS + +The pgx ConnConfig struct has a TLSConfig field. If this field is +nil, then TLS will be disabled. If it is present, then it will be used to +configure the TLS connection. This allows total configuration of the TLS +connection. + +Logging + +pgx defines a simple logger interface. Connections optionally accept a logger +that satisfies this interface. The log15 package +(http://gopkg.in/inconshreveable/log15.v2) satisfies this interface and it is +simple to define adapters for other loggers. Set LogLevel to control logging +verbosity. +*/ +package pgx diff --git a/vendor/github.com/jackc/pgx/fastpath.go b/vendor/github.com/jackc/pgx/fastpath.go new file mode 100644 index 0000000..19b9878 --- /dev/null +++ b/vendor/github.com/jackc/pgx/fastpath.go @@ -0,0 +1,108 @@ +package pgx + +import ( + "encoding/binary" +) + +func newFastpath(cn *Conn) *fastpath { + return &fastpath{cn: cn, fns: make(map[string]Oid)} +} + +type fastpath struct { + cn *Conn + fns map[string]Oid +} + +func (f *fastpath) functionOID(name string) Oid { + return f.fns[name] +} + +func (f *fastpath) addFunction(name string, oid Oid) { + f.fns[name] = oid +} + +func (f *fastpath) addFunctions(rows *Rows) error { + for rows.Next() { + var name string + var oid Oid + if err := rows.Scan(&name, &oid); err != nil { + return err + } + f.addFunction(name, oid) + } + return rows.Err() +} + +type fpArg []byte + +func fpIntArg(n int32) fpArg { + res := make([]byte, 4) + binary.BigEndian.PutUint32(res, uint32(n)) + return res +} + +func fpInt64Arg(n int64) fpArg { + res := make([]byte, 8) + binary.BigEndian.PutUint64(res, uint64(n)) + return res +} + +func (f *fastpath) Call(oid Oid, args []fpArg) (res []byte, err error) { + wbuf := newWriteBuf(f.cn, 'F') // function call + wbuf.WriteInt32(int32(oid)) // function object id + wbuf.WriteInt16(1) // # of argument format codes + wbuf.WriteInt16(1) // format code: binary + wbuf.WriteInt16(int16(len(args))) // # of arguments + for _, arg := range args { + wbuf.WriteInt32(int32(len(arg))) // length of argument + wbuf.WriteBytes(arg) // argument value + } + wbuf.WriteInt16(1) // response format code (binary) + wbuf.closeMsg() + + if _, err := f.cn.conn.Write(wbuf.buf); err != nil { + return nil, err + } + + for { + var t byte + var r *msgReader + t, r, err = f.cn.rxMsg() + if err != nil { + return nil, err + } + switch t { + case 'V': // FunctionCallResponse + data := r.readBytes(r.readInt32()) + res = make([]byte, len(data)) + copy(res, data) + case 'Z': // Ready for query + f.cn.rxReadyForQuery(r) + // done + return + default: + if err := f.cn.processContextFreeMsg(t, r); err != nil { + return nil, err + } + } + } +} + +func (f *fastpath) CallFn(fn string, args []fpArg) ([]byte, error) { + return f.Call(f.functionOID(fn), args) +} + +func fpInt32(data []byte, err error) (int32, error) { + if err != nil { + return 0, err + } + n := int32(binary.BigEndian.Uint32(data)) + return n, nil +} + +func fpInt64(data []byte, err error) (int64, error) { + if err != nil { + return 0, err + } + return int64(binary.BigEndian.Uint64(data)), nil +} diff --git a/vendor/github.com/jackc/pgx/hstore.go b/vendor/github.com/jackc/pgx/hstore.go new file mode 100644 index 0000000..0ab9f77 --- /dev/null +++ b/vendor/github.com/jackc/pgx/hstore.go @@ -0,0 +1,222 @@ +package pgx + +import ( + "bytes" + "errors" + "fmt" + "unicode" + "unicode/utf8" +) + +const ( + hsPre = iota + hsKey + hsSep + hsVal + hsNul + hsNext +) + +type hstoreParser struct { + str string + pos int +} + +func newHSP(in string) *hstoreParser { + return &hstoreParser{ + pos: 0, + str: in, + } +} + +func (p *hstoreParser) Consume() (r rune, end bool) { + if p.pos >= len(p.str) { + end = true + return + } + r, w := utf8.DecodeRuneInString(p.str[p.pos:]) + p.pos += w + return +} + +func (p *hstoreParser) Peek() (r rune, end bool) { + if p.pos >= len(p.str) { + end = true + return + } + r, _ = utf8.DecodeRuneInString(p.str[p.pos:]) + return +} + +func parseHstoreToMap(s string) (m map[string]string, err error) { + keys, values, err := ParseHstore(s) + if err != nil { + return + } + m = make(map[string]string, len(keys)) + for i, key := range keys { + if !values[i].Valid { + err = fmt.Errorf("key '%s' has NULL value", key) + m = nil + return + } + m[key] = values[i].String + } + return +} + +func parseHstoreToNullHstore(s string) (store map[string]NullString, err error) { + keys, values, err := ParseHstore(s) + if err != nil { + return + } + + store = make(map[string]NullString, len(keys)) + + for i, key := range keys { + store[key] = values[i] + } + return +} + +// ParseHstore parses the string representation of an hstore column (the same +// you would get from an ordinary SELECT) into two slices of keys and values. it +// is used internally in the default parsing of hstores, but is exported for use +// in handling custom data structures backed by an hstore column without the +// overhead of creating a map[string]string +func ParseHstore(s string) (k []string, v []NullString, err error) { + if s == "" { + return + } + + buf := bytes.Buffer{} + keys := []string{} + values := []NullString{} + p := newHSP(s) + + r, end := p.Consume() + state := hsPre + + for !end { + switch state { + case hsPre: + if r == '"' { + state = hsKey + } else { + err = errors.New("String does not begin with \"") + } + case hsKey: + switch r { + case '"': //End of the key + if buf.Len() == 0 { + err = errors.New("Empty Key is invalid") + } else { + keys = append(keys, buf.String()) + buf = bytes.Buffer{} + state = hsSep + } + case '\\': //Potential escaped character + n, end := p.Consume() + switch { + case end: + err = errors.New("Found EOS in key, expecting character or \"") + case n == '"', n == '\\': + buf.WriteRune(n) + default: + buf.WriteRune(r) + buf.WriteRune(n) + } + default: //Any other character + buf.WriteRune(r) + } + case hsSep: + if r == '=' { + r, end = p.Consume() + switch { + case end: + err = errors.New("Found EOS after '=', expecting '>'") + case r == '>': + r, end = p.Consume() + switch { + case end: + err = errors.New("Found EOS after '=>', expecting '\"' or 'NULL'") + case r == '"': + state = hsVal + case r == 'N': + state = hsNul + default: + err = fmt.Errorf("Invalid character '%c' after '=>', expecting '\"' or 'NULL'", r) + } + default: + err = fmt.Errorf("Invalid character after '=', expecting '>'") + } + } else { + err = fmt.Errorf("Invalid character '%c' after value, expecting '='", r) + } + case hsVal: + switch r { + case '"': //End of the value + values = append(values, NullString{String: buf.String(), Valid: true}) + buf = bytes.Buffer{} + state = hsNext + case '\\': //Potential escaped character + n, end := p.Consume() + switch { + case end: + err = errors.New("Found EOS in key, expecting character or \"") + case n == '"', n == '\\': + buf.WriteRune(n) + default: + buf.WriteRune(r) + buf.WriteRune(n) + } + default: //Any other character + buf.WriteRune(r) + } + case hsNul: + nulBuf := make([]rune, 3) + nulBuf[0] = r + for i := 1; i < 3; i++ { + r, end = p.Consume() + if end { + err = errors.New("Found EOS in NULL value") + return + } + nulBuf[i] = r + } + if nulBuf[0] == 'U' && nulBuf[1] == 'L' && nulBuf[2] == 'L' { + values = append(values, NullString{String: "", Valid: false}) + state = hsNext + } else { + err = fmt.Errorf("Invalid NULL value: 'N%s'", string(nulBuf)) + } + case hsNext: + if r == ',' { + r, end = p.Consume() + switch { + case end: + err = errors.New("Found EOS after ',', expcting space") + case (unicode.IsSpace(r)): + r, end = p.Consume() + state = hsKey + default: + err = fmt.Errorf("Invalid character '%c' after ', ', expecting \"", r) + } + } else { + err = fmt.Errorf("Invalid character '%c' after value, expecting ','", r) + } + } + + if err != nil { + return + } + r, end = p.Consume() + } + if state != hsNext { + err = errors.New("Improperly formatted hstore") + return + } + k = keys + v = values + return +} diff --git a/vendor/github.com/jackc/pgx/large_objects.go b/vendor/github.com/jackc/pgx/large_objects.go new file mode 100644 index 0000000..a4922ef --- /dev/null +++ b/vendor/github.com/jackc/pgx/large_objects.go @@ -0,0 +1,147 @@ +package pgx + +import ( + "io" +) + +// LargeObjects is a structure used to access the large objects API. It is only +// valid within the transaction where it was created. +// +// For more details see: http://www.postgresql.org/docs/current/static/largeobjects.html +type LargeObjects struct { + // Has64 is true if the server is capable of working with 64-bit numbers + Has64 bool + fp *fastpath +} + +const largeObjectFns = `select proname, oid from pg_catalog.pg_proc +where proname in ( +'lo_open', +'lo_close', +'lo_create', +'lo_unlink', +'lo_lseek', +'lo_lseek64', +'lo_tell', +'lo_tell64', +'lo_truncate', +'lo_truncate64', +'loread', +'lowrite') +and pronamespace = (select oid from pg_catalog.pg_namespace where nspname = 'pg_catalog')` + +// LargeObjects returns a LargeObjects instance for the transaction. +func (tx *Tx) LargeObjects() (*LargeObjects, error) { + if tx.conn.fp == nil { + tx.conn.fp = newFastpath(tx.conn) + } + if _, exists := tx.conn.fp.fns["lo_open"]; !exists { + res, err := tx.Query(largeObjectFns) + if err != nil { + return nil, err + } + if err := tx.conn.fp.addFunctions(res); err != nil { + return nil, err + } + } + + lo := &LargeObjects{fp: tx.conn.fp} + _, lo.Has64 = lo.fp.fns["lo_lseek64"] + + return lo, nil +} + +type LargeObjectMode int32 + +const ( + LargeObjectModeWrite LargeObjectMode = 0x20000 + LargeObjectModeRead LargeObjectMode = 0x40000 +) + +// Create creates a new large object. If id is zero, the server assigns an +// unused OID. +func (o *LargeObjects) Create(id Oid) (Oid, error) { + newOid, err := fpInt32(o.fp.CallFn("lo_create", []fpArg{fpIntArg(int32(id))})) + return Oid(newOid), err +} + +// Open opens an existing large object with the given mode. +func (o *LargeObjects) Open(oid Oid, mode LargeObjectMode) (*LargeObject, error) { + fd, err := fpInt32(o.fp.CallFn("lo_open", []fpArg{fpIntArg(int32(oid)), fpIntArg(int32(mode))})) + return &LargeObject{fd: fd, lo: o}, err +} + +// Unlink removes a large object from the database. +func (o *LargeObjects) Unlink(oid Oid) error { + _, err := o.fp.CallFn("lo_unlink", []fpArg{fpIntArg(int32(oid))}) + return err +} + +// A LargeObject is a large object stored on the server. It is only valid within +// the transaction that it was initialized in. It implements these interfaces: +// +// io.Writer +// io.Reader +// io.Seeker +// io.Closer +type LargeObject struct { + fd int32 + lo *LargeObjects +} + +// Write writes p to the large object and returns the number of bytes written +// and an error if not all of p was written. +func (o *LargeObject) Write(p []byte) (int, error) { + n, err := fpInt32(o.lo.fp.CallFn("lowrite", []fpArg{fpIntArg(o.fd), p})) + return int(n), err +} + +// Read reads up to len(p) bytes into p returning the number of bytes read. +func (o *LargeObject) Read(p []byte) (int, error) { + res, err := o.lo.fp.CallFn("loread", []fpArg{fpIntArg(o.fd), fpIntArg(int32(len(p)))}) + if len(res) < len(p) { + err = io.EOF + } + return copy(p, res), err +} + +// Seek moves the current location pointer to the new location specified by offset. +func (o *LargeObject) Seek(offset int64, whence int) (n int64, err error) { + if o.lo.Has64 { + n, err = fpInt64(o.lo.fp.CallFn("lo_lseek64", []fpArg{fpIntArg(o.fd), fpInt64Arg(offset), fpIntArg(int32(whence))})) + } else { + var n32 int32 + n32, err = fpInt32(o.lo.fp.CallFn("lo_lseek", []fpArg{fpIntArg(o.fd), fpIntArg(int32(offset)), fpIntArg(int32(whence))})) + n = int64(n32) + } + return +} + +// Tell returns the current read or write location of the large object +// descriptor. +func (o *LargeObject) Tell() (n int64, err error) { + if o.lo.Has64 { + n, err = fpInt64(o.lo.fp.CallFn("lo_tell64", []fpArg{fpIntArg(o.fd)})) + } else { + var n32 int32 + n32, err = fpInt32(o.lo.fp.CallFn("lo_tell", []fpArg{fpIntArg(o.fd)})) + n = int64(n32) + } + return +} + +// Trunctes the large object to size. +func (o *LargeObject) Truncate(size int64) (err error) { + if o.lo.Has64 { + _, err = o.lo.fp.CallFn("lo_truncate64", []fpArg{fpIntArg(o.fd), fpInt64Arg(size)}) + } else { + _, err = o.lo.fp.CallFn("lo_truncate", []fpArg{fpIntArg(o.fd), fpIntArg(int32(size))}) + } + return +} + +// Close closees the large object descriptor. +func (o *LargeObject) Close() error { + _, err := o.lo.fp.CallFn("lo_close", []fpArg{fpIntArg(o.fd)}) + return err +} diff --git a/vendor/github.com/jackc/pgx/logger.go b/vendor/github.com/jackc/pgx/logger.go new file mode 100644 index 0000000..4423325 --- /dev/null +++ b/vendor/github.com/jackc/pgx/logger.go @@ -0,0 +1,81 @@ +package pgx + +import ( + "encoding/hex" + "errors" + "fmt" +) + +// The values for log levels are chosen such that the zero value means that no +// log level was specified and we can default to LogLevelDebug to preserve +// the behavior that existed prior to log level introduction. +const ( + LogLevelTrace = 6 + LogLevelDebug = 5 + LogLevelInfo = 4 + LogLevelWarn = 3 + LogLevelError = 2 + LogLevelNone = 1 +) + +// Logger is the interface used to get logging from pgx internals. +// https://github.com/inconshreveable/log15 is the recommended logging package. +// This logging interface was extracted from there. However, it should be simple +// to adapt any logger to this interface. +type Logger interface { + // Log a message at the given level with context key/value pairs + Debug(msg string, ctx ...interface{}) + Info(msg string, ctx ...interface{}) + Warn(msg string, ctx ...interface{}) + Error(msg string, ctx ...interface{}) +} + +// LogLevelFromString converts log level string to constant +// +// Valid levels: +// trace +// debug +// info +// warn +// error +// none +func LogLevelFromString(s string) (int, error) { + switch s { + case "trace": + return LogLevelTrace, nil + case "debug": + return LogLevelDebug, nil + case "info": + return LogLevelInfo, nil + case "warn": + return LogLevelWarn, nil + case "error": + return LogLevelError, nil + case "none": + return LogLevelNone, nil + default: + return 0, errors.New("invalid log level") + } +} + +func logQueryArgs(args []interface{}) []interface{} { + logArgs := make([]interface{}, 0, len(args)) + + for _, a := range args { + switch v := a.(type) { + case []byte: + if len(v) < 64 { + a = hex.EncodeToString(v) + } else { + a = fmt.Sprintf("%x (truncated %d bytes)", v[:64], len(v)-64) + } + case string: + if len(v) > 64 { + a = fmt.Sprintf("%s (truncated %d bytes)", v[:64], len(v)-64) + } + } + logArgs = append(logArgs, a) + } + + return logArgs +} diff --git a/vendor/github.com/jackc/pgx/messages.go b/vendor/github.com/jackc/pgx/messages.go new file mode 100644 index 0000000..317ba27 --- /dev/null +++ b/vendor/github.com/jackc/pgx/messages.go @@ -0,0 +1,159 @@ +package pgx + +import ( + "encoding/binary" +) + +const ( + protocolVersionNumber = 196608 // 3.0 +) + +const ( + backendKeyData = 'K' + authenticationX = 'R' + readyForQuery = 'Z' + rowDescription = 'T' + dataRow = 'D' + commandComplete = 'C' + errorResponse = 'E' + noticeResponse = 'N' + parseComplete = '1' + parameterDescription = 't' + bindComplete = '2' + notificationResponse = 'A' + emptyQueryResponse = 'I' + noData = 'n' + closeComplete = '3' + flush = 'H' + copyInResponse = 'G' + copyData = 'd' + copyFail = 'f' + copyDone = 'c' +) + +type startupMessage struct { + options map[string]string +} + +func newStartupMessage() *startupMessage { + return &startupMessage{map[string]string{}} +} + +func (s *startupMessage) Bytes() (buf []byte) { + buf = make([]byte, 8, 128) + binary.BigEndian.PutUint32(buf[4:8], uint32(protocolVersionNumber)) + for key, value := range s.options { + buf = append(buf, key...) + buf = append(buf, 0) + buf = append(buf, value...) + buf = append(buf, 0) + } + buf = append(buf, ("\000")...) + binary.BigEndian.PutUint32(buf[0:4], uint32(len(buf))) + return buf +} + +type FieldDescription struct { + Name string + Table Oid + AttributeNumber int16 + DataType Oid + DataTypeSize int16 + DataTypeName string + Modifier int32 + FormatCode int16 +} + +// PgError represents an error reported by the PostgreSQL server. See +// http://www.postgresql.org/docs/9.3/static/protocol-error-fields.html for +// detailed field description. +type PgError struct { + Severity string + Code string + Message string + Detail string + Hint string + Position int32 + InternalPosition int32 + InternalQuery string + Where string + SchemaName string + TableName string + ColumnName string + DataTypeName string + ConstraintName string + File string + Line int32 + Routine string +} + +func (pe PgError) Error() string { + return pe.Severity + ": " + pe.Message + " (SQLSTATE " + pe.Code + ")" +} + +func newWriteBuf(c *Conn, t byte) *WriteBuf { + buf := append(c.wbuf[0:0], t, 0, 0, 0, 0) + c.writeBuf = WriteBuf{buf: buf, sizeIdx: 1, conn: c} + return &c.writeBuf +} + +// WriteBuf is used build messages to send to the PostgreSQL server. It is used +// by the Encoder interface when implementing custom encoders. +type WriteBuf struct { + buf []byte + sizeIdx int + conn *Conn +} + +func (wb *WriteBuf) startMsg(t byte) { + wb.closeMsg() + wb.buf = append(wb.buf, t, 0, 0, 0, 0) + wb.sizeIdx = len(wb.buf) - 4 +} + +func (wb *WriteBuf) closeMsg() { + binary.BigEndian.PutUint32(wb.buf[wb.sizeIdx:wb.sizeIdx+4], uint32(len(wb.buf)-wb.sizeIdx)) +} + +func (wb *WriteBuf) WriteByte(b byte) { + wb.buf = append(wb.buf, b) +} + +func (wb *WriteBuf) WriteCString(s string) { + wb.buf = append(wb.buf, []byte(s)...) + wb.buf = append(wb.buf, 0) +} + +func (wb *WriteBuf) WriteInt16(n int16) { + b := make([]byte, 2) + binary.BigEndian.PutUint16(b, uint16(n)) + wb.buf = append(wb.buf, b...) +} + +func (wb *WriteBuf) WriteUint16(n uint16) { + b := make([]byte, 2) + binary.BigEndian.PutUint16(b, n) + wb.buf = append(wb.buf, b...) +} + +func (wb *WriteBuf) WriteInt32(n int32) { + b := make([]byte, 4) + binary.BigEndian.PutUint32(b, uint32(n)) + wb.buf = append(wb.buf, b...) +} + +func (wb *WriteBuf) WriteUint32(n uint32) { + b := make([]byte, 4) + binary.BigEndian.PutUint32(b, n) + wb.buf = append(wb.buf, b...) +} + +func (wb *WriteBuf) WriteInt64(n int64) { + b := make([]byte, 8) + binary.BigEndian.PutUint64(b, uint64(n)) + wb.buf = append(wb.buf, b...) +} + +func (wb *WriteBuf) WriteBytes(b []byte) { + wb.buf = append(wb.buf, b...) +} diff --git a/vendor/github.com/jackc/pgx/msg_reader.go b/vendor/github.com/jackc/pgx/msg_reader.go new file mode 100644 index 0000000..21db5d2 --- /dev/null +++ b/vendor/github.com/jackc/pgx/msg_reader.go @@ -0,0 +1,316 @@ +package pgx + +import ( + "bufio" + "encoding/binary" + "errors" + "io" +) + +// msgReader is a helper that reads values from a PostgreSQL message. +type msgReader struct { + reader *bufio.Reader + msgBytesRemaining int32 + err error + log func(lvl int, msg string, ctx ...interface{}) + shouldLog func(lvl int) bool +} + +// Err returns any error that the msgReader has experienced +func (r *msgReader) Err() error { + return r.err +} + +// fatal tells rc that a Fatal error has occurred +func (r *msgReader) fatal(err error) { + if r.shouldLog(LogLevelTrace) { + r.log(LogLevelTrace, "msgReader.fatal", "error", err, "msgBytesRemaining", r.msgBytesRemaining) + } + r.err = err +} + +// rxMsg reads the type and size of the next message. +func (r *msgReader) rxMsg() (byte, error) { + if r.err != nil { + return 0, r.err + } + + if r.msgBytesRemaining > 0 { + if r.shouldLog(LogLevelTrace) { + r.log(LogLevelTrace, "msgReader.rxMsg discarding unread previous message", "msgBytesRemaining", r.msgBytesRemaining) + } + + _, err := r.reader.Discard(int(r.msgBytesRemaining)) + if err != nil { + return 0, err + } + } + + b, err := r.reader.Peek(5) + if err != nil { + r.fatal(err) + return 0, err + } + msgType := b[0] + r.msgBytesRemaining = int32(binary.BigEndian.Uint32(b[1:])) - 4 + r.reader.Discard(5) + return msgType, nil +} + +func (r *msgReader) readByte() byte { + if r.err != nil { + return 0 + } + + r.msgBytesRemaining-- + if r.msgBytesRemaining < 0 { + r.fatal(errors.New("read past end of message")) + return 0 + } + + b, err := r.reader.ReadByte() + if err != nil { + r.fatal(err) + return 0 + } + + if r.shouldLog(LogLevelTrace) { + r.log(LogLevelTrace, "msgReader.readByte", "value", b, "byteAsString", string(b), "msgBytesRemaining", r.msgBytesRemaining) + } + + return b +} + +func (r *msgReader) readInt16() int16 { + if r.err != nil { + return 0 + } + + r.msgBytesRemaining -= 2 + if r.msgBytesRemaining < 0 { + r.fatal(errors.New("read past end of message")) + return 0 + } + + b, err := r.reader.Peek(2) + if err != nil { + r.fatal(err) + return 0 + } + + n := int16(binary.BigEndian.Uint16(b)) + + r.reader.Discard(2) + + if r.shouldLog(LogLevelTrace) { + r.log(LogLevelTrace, "msgReader.readInt16", "value", n, "msgBytesRemaining", r.msgBytesRemaining) + } + + return n +} + +func (r *msgReader) readInt32() int32 { + if r.err != nil { + return 0 + } + + r.msgBytesRemaining -= 4 + if r.msgBytesRemaining < 0 { + r.fatal(errors.New("read past end of message")) + return 0 + } + + b, err := r.reader.Peek(4) + if err != nil { + r.fatal(err) + return 0 + } + + n := int32(binary.BigEndian.Uint32(b)) + + r.reader.Discard(4) + + if r.shouldLog(LogLevelTrace) { + r.log(LogLevelTrace, "msgReader.readInt32", "value", n, "msgBytesRemaining", r.msgBytesRemaining) + } + + return n +} + +func (r *msgReader) readUint16() uint16 { + if r.err != nil { + return 0 + } + + r.msgBytesRemaining -= 2 + if r.msgBytesRemaining < 0 { + r.fatal(errors.New("read past end of message")) + return 0 + } + + b, err := r.reader.Peek(2) + if err != nil { + r.fatal(err) + return 0 + } + + n := uint16(binary.BigEndian.Uint16(b)) + + r.reader.Discard(2) + + if r.shouldLog(LogLevelTrace) { + r.log(LogLevelTrace, "msgReader.readUint16", "value", n, "msgBytesRemaining", r.msgBytesRemaining) + } + + return n +} + +func (r *msgReader) readUint32() uint32 { + if r.err != nil { + return 0 + } + + r.msgBytesRemaining -= 4 + if r.msgBytesRemaining < 0 { + r.fatal(errors.New("read past end of message")) + return 0 + } + + b, err := r.reader.Peek(4) + if err != nil { + r.fatal(err) + return 0 + } + + n := uint32(binary.BigEndian.Uint32(b)) + + r.reader.Discard(4) + + if r.shouldLog(LogLevelTrace) { + r.log(LogLevelTrace, "msgReader.readUint32", "value", n, "msgBytesRemaining", r.msgBytesRemaining) + } + + return n +} + +func (r *msgReader) readInt64() int64 { + if r.err != nil { + return 0 + } + + r.msgBytesRemaining -= 8 + if r.msgBytesRemaining < 0 { + r.fatal(errors.New("read past end of message")) + return 0 + } + + b, err := r.reader.Peek(8) + if err != nil { + r.fatal(err) + return 0 + } + + n := int64(binary.BigEndian.Uint64(b)) + + r.reader.Discard(8) + + if r.shouldLog(LogLevelTrace) { + r.log(LogLevelTrace, "msgReader.readInt64", "value", n, "msgBytesRemaining", r.msgBytesRemaining) + } + + return n +} + +func (r *msgReader) readOid() Oid { + return Oid(r.readInt32()) +} + +// readCString reads a null terminated string +func (r *msgReader) readCString() string { + if r.err != nil { + return "" + } + + b, err := r.reader.ReadBytes(0) + if err != nil { + r.fatal(err) + return "" + } + + r.msgBytesRemaining -= int32(len(b)) + if r.msgBytesRemaining < 0 { + r.fatal(errors.New("read past end of message")) + return "" + } + + s := string(b[0 : len(b)-1]) + + if r.shouldLog(LogLevelTrace) { + r.log(LogLevelTrace, "msgReader.readCString", "value", s, "msgBytesRemaining", r.msgBytesRemaining) + } + + return s +} + +// readString reads count bytes and returns as string +func (r *msgReader) readString(countI32 int32) string { + if r.err != nil { + return "" + } + + r.msgBytesRemaining -= countI32 + if r.msgBytesRemaining < 0 { + r.fatal(errors.New("read past end of message")) + return "" + } + + count := int(countI32) + var s string + + if r.reader.Buffered() >= count { + buf, _ := r.reader.Peek(count) + s = string(buf) + r.reader.Discard(count) + } else { + buf := make([]byte, count) + _, err := io.ReadFull(r.reader, buf) + if err != nil { + r.fatal(err) + return "" + } + s = string(buf) + } + + if r.shouldLog(LogLevelTrace) { + r.log(LogLevelTrace, "msgReader.readString", "value", s, "msgBytesRemaining", r.msgBytesRemaining) + } + + return s +} + +// readBytes reads count bytes and returns as []byte +func (r *msgReader) readBytes(count int32) []byte { + if r.err != nil { + return nil + } + + r.msgBytesRemaining -= count + if r.msgBytesRemaining < 0 { + r.fatal(errors.New("read past end of message")) + return nil + } + + b := make([]byte, int(count)) + + _, err := io.ReadFull(r.reader, b) + if err != nil { + r.fatal(err) + return nil + } + + if r.shouldLog(LogLevelTrace) { + r.log(LogLevelTrace, "msgReader.readBytes", "value", b, "msgBytesRemaining", r.msgBytesRemaining) + } + + return b +} diff --git a/vendor/github.com/jackc/pgx/pgpass.go b/vendor/github.com/jackc/pgx/pgpass.go new file mode 100644 index 0000000..b6f028d --- /dev/null +++ b/vendor/github.com/jackc/pgx/pgpass.go @@ -0,0 +1,85 @@ +package pgx + +import ( + "bufio" + "fmt" + "os" + "os/user" + "path/filepath" + "strings" +) + +func parsepgpass(cfg *ConnConfig, line string) *string { + const ( + backslash = "\r" + colon = "\n" + ) + const ( + host int = iota + port + database + username + pw + ) + line = strings.Replace(line, `\:`, colon, -1) + line = strings.Replace(line, `\\`, backslash, -1) + parts := strings.Split(line, `:`) + if len(parts) != 5 { + return nil + } + for i := range parts { + if parts[i] == `*` { + continue + } + parts[i] = strings.Replace(strings.Replace(parts[i], backslash, `\`, -1), colon, `:`, -1) + switch i { + case host: + if parts[i] != cfg.Host { + return nil + } + case port: + portstr := fmt.Sprintf(`%v`, cfg.Port) + if portstr == "0" { + portstr = "5432" + } + if parts[i] != portstr { + return nil + } + case database: + if parts[i] != cfg.Database { + return nil + } + case username: + if parts[i] != cfg.User { + return nil + } + } + } + return &parts[4] +} + +func pgpass(cfg *ConnConfig) (found bool) { + passfile := os.Getenv("PGPASSFILE") + if passfile == "" { + u, err := user.Current() + if err != nil { + return + } + passfile = filepath.Join(u.HomeDir, ".pgpass") + } + f, err := os.Open(passfile) + if err != nil { + return + } + defer f.Close() + scanner := bufio.NewScanner(f) + var pw *string + for scanner.Scan() { + pw = parsepgpass(cfg, scanner.Text()) + if pw != nil { + cfg.Password = *pw + return true + } + } + return false +} diff --git a/vendor/github.com/jackc/pgx/query.go b/vendor/github.com/jackc/pgx/query.go new file mode 100644 index 0000000..19b867e --- /dev/null +++ b/vendor/github.com/jackc/pgx/query.go @@ -0,0 +1,494 @@ +package pgx + +import ( + "database/sql" + "errors" + "fmt" + "time" +) + +// Row is a convenience wrapper over Rows that is returned by QueryRow. +type Row Rows + +// Scan works the same as (*Rows Scan) with the following exceptions. If no +// rows were found it returns ErrNoRows. If multiple rows are returned it +// ignores all but the first. +func (r *Row) Scan(dest ...interface{}) (err error) { + rows := (*Rows)(r) + + if rows.Err() != nil { + return rows.Err() + } + + if !rows.Next() { + if rows.Err() == nil { + return ErrNoRows + } + return rows.Err() + } + + rows.Scan(dest...) + rows.Close() + return rows.Err() +} + +// Rows is the result set returned from *Conn.Query. Rows must be closed before +// the *Conn can be used again. Rows are closed by explicitly calling Close(), +// calling Next() until it returns false, or when a fatal error occurs. +type Rows struct { + conn *Conn + mr *msgReader + fields []FieldDescription + vr ValueReader + rowCount int + columnIdx int + err error + startTime time.Time + sql string + args []interface{} + afterClose func(*Rows) + unlockConn bool + closed bool +} + +func (rows *Rows) FieldDescriptions() []FieldDescription { + return rows.fields +} + +func (rows *Rows) close() { + if rows.closed { + return + } + + if rows.unlockConn { + rows.conn.unlock() + rows.unlockConn = false + } + + rows.closed = true + + if rows.err == nil { + if rows.conn.shouldLog(LogLevelInfo) { + endTime := time.Now() + rows.conn.log(LogLevelInfo, "Query", "sql", rows.sql, "args", logQueryArgs(rows.args), "time", endTime.Sub(rows.startTime), "rowCount", rows.rowCount) + } + } else if rows.conn.shouldLog(LogLevelError) { + rows.conn.log(LogLevelError, "Query", "sql", rows.sql, "args", logQueryArgs(rows.args)) + } + + if rows.afterClose != nil { + rows.afterClose(rows) + } +} + +func (rows *Rows) readUntilReadyForQuery() { + for { + t, r, err := rows.conn.rxMsg() + if err != nil { + rows.close() + return + } + + switch t { + case readyForQuery: + rows.conn.rxReadyForQuery(r) + rows.close() + return + case rowDescription: + case dataRow: + case commandComplete: + case bindComplete: + case errorResponse: + err = rows.conn.rxErrorResponse(r) + if rows.err == nil { + rows.err = err + } + default: + err = rows.conn.processContextFreeMsg(t, r) + if err != nil { + rows.close() + return + } + } + } +} + +// Close closes the rows, making the connection ready for use again. It is safe +// to call Close after rows is already closed. +func (rows *Rows) Close() { + if rows.closed { + return + } + rows.readUntilReadyForQuery() + rows.close() +} + +func (rows *Rows) Err() error { + return rows.err +} + +// abort signals that the query was not successfully sent to the server. +// This differs from Fatal in that it is not necessary to readUntilReadyForQuery +func (rows *Rows) abort(err error) { + if rows.err != nil { + return + } + + rows.err = err + rows.close() +} + +// Fatal signals an error occurred after the query was sent to the server. It +// closes the rows automatically. +func (rows *Rows) Fatal(err error) { + if rows.err != nil { + return + } + + rows.err = err + rows.Close() +} + +// Next prepares the next row for reading. It returns true if there is another +// row and false if no more rows are available. It automatically closes rows +// when all rows are read. +func (rows *Rows) Next() bool { + if rows.closed { + return false + } + + rows.rowCount++ + rows.columnIdx = 0 + rows.vr = ValueReader{} + + for { + t, r, err := rows.conn.rxMsg() + if err != nil { + rows.Fatal(err) + return false + } + + switch t { + case readyForQuery: + rows.conn.rxReadyForQuery(r) + rows.close() + return false + case dataRow: + fieldCount := r.readInt16() + if int(fieldCount) != len(rows.fields) { + rows.Fatal(ProtocolError(fmt.Sprintf("Row description field count (%v) and data row field count (%v) do not match", len(rows.fields), fieldCount))) + return false + } + + rows.mr = r + return true + case commandComplete: + case bindComplete: + default: + err = rows.conn.processContextFreeMsg(t, r) + if err != nil { + rows.Fatal(err) + return false + } + } + } +} + +// Conn returns the *Conn this *Rows is using. +func (rows *Rows) Conn() *Conn { + return rows.conn +} + +func (rows *Rows) nextColumn() (*ValueReader, bool) { + if rows.closed { + return nil, false + } + if len(rows.fields) <= rows.columnIdx { + rows.Fatal(ProtocolError("No next column available")) + return nil, false + } + + if rows.vr.Len() > 0 { + rows.mr.readBytes(rows.vr.Len()) + } + + fd := &rows.fields[rows.columnIdx] + rows.columnIdx++ + size := rows.mr.readInt32() + rows.vr = ValueReader{mr: rows.mr, fd: fd, valueBytesRemaining: size} + return &rows.vr, true +} + +type scanArgError struct { + col int + err error +} + +func (e scanArgError) Error() string { + return fmt.Sprintf("can't scan into dest[%d]: %v", e.col, e.err) +} + +// Scan reads the values from the current row into dest values positionally. +// dest can include pointers to core types, values implementing the Scanner +// interface, []byte, and nil. []byte will skip the decoding process and directly +// copy the raw bytes received from PostgreSQL. nil will skip the value entirely. +func (rows *Rows) Scan(dest ...interface{}) (err error) { + if len(rows.fields) != len(dest) { + err = fmt.Errorf("Scan received wrong number of arguments, got %d but expected %d", len(dest), len(rows.fields)) + rows.Fatal(err) + return err + } + + for i, d := range dest { + vr, _ := rows.nextColumn() + + if d == nil { + continue + } + + // Check for []byte first as we allow sidestepping the decoding process and retrieving the raw bytes + if b, ok := d.(*[]byte); ok { + // If it actually is a bytea then pass it through decodeBytea (so it can be decoded if it is in text format) + // Otherwise read the bytes directly regardless of what the actual type is. + if vr.Type().DataType == ByteaOid { + *b = decodeBytea(vr) + } else { + if vr.Len() != -1 { + *b = vr.ReadBytes(vr.Len()) + } else { + *b = nil + } + } + } else if s, ok := d.(Scanner); ok { + err = s.Scan(vr) + if err != nil { + rows.Fatal(scanArgError{col: i, err: err}) + } + } else if s, ok := d.(PgxScanner); ok { + err = s.ScanPgx(vr) + if err != nil { + rows.Fatal(scanArgError{col: i, err: err}) + } + } else if s, ok := d.(sql.Scanner); ok { + var val interface{} + if 0 <= vr.Len() { + switch vr.Type().DataType { + case BoolOid: + val = decodeBool(vr) + case Int8Oid: + val = int64(decodeInt8(vr)) + case Int2Oid: + val = int64(decodeInt2(vr)) + case Int4Oid: + val = int64(decodeInt4(vr)) + case TextOid, VarcharOid: + val = decodeText(vr) + case OidOid: + val = int64(decodeOid(vr)) + case Float4Oid: + val = float64(decodeFloat4(vr)) + case Float8Oid: + val = decodeFloat8(vr) + case DateOid: + val = decodeDate(vr) + case TimestampOid: + val = decodeTimestamp(vr) + case TimestampTzOid: + val = decodeTimestampTz(vr) + default: + val = vr.ReadBytes(vr.Len()) + } + } + err = s.Scan(val) + if err != nil { + rows.Fatal(scanArgError{col: i, err: err}) + } + } else if vr.Type().DataType == JsonOid { + // Because the argument passed to decodeJSON will escape the heap. + // This allows d to be stack allocated and only copied to the heap when + // we actually are decoding JSON. This saves one memory allocation per + // row. + d2 := d + decodeJSON(vr, &d2) + } else if vr.Type().DataType == JsonbOid { + // Same trick as above for getting stack allocation + d2 := d + decodeJSONB(vr, &d2) + } else { + if err := Decode(vr, d); err != nil { + rows.Fatal(scanArgError{col: i, err: err}) + } + } + if vr.Err() != nil { + rows.Fatal(scanArgError{col: i, err: vr.Err()}) + } + + if rows.Err() != nil { + return rows.Err() + } + } + + return nil +} + +// Values returns an array of the row values +func (rows *Rows) Values() ([]interface{}, error) { + if rows.closed { + return nil, errors.New("rows is closed") + } + + values := make([]interface{}, 0, len(rows.fields)) + + for range rows.fields { + vr, _ := rows.nextColumn() + + if vr.Len() == -1 { + values = append(values, nil) + continue + } + + switch vr.Type().FormatCode { + // All intrinsic types (except string) are encoded with binary + // encoding so anything else should be treated as a string + case TextFormatCode: + values = append(values, vr.ReadString(vr.Len())) + case BinaryFormatCode: + switch vr.Type().DataType { + case TextOid, VarcharOid: + values = append(values, decodeText(vr)) + case BoolOid: + values = append(values, decodeBool(vr)) + case ByteaOid: + values = append(values, decodeBytea(vr)) + case Int8Oid: + values = append(values, decodeInt8(vr)) + case Int2Oid: + values = append(values, decodeInt2(vr)) + case Int4Oid: + values = append(values, decodeInt4(vr)) + case OidOid: + values = append(values, decodeOid(vr)) + case Float4Oid: + values = append(values, decodeFloat4(vr)) + case Float8Oid: + values = append(values, decodeFloat8(vr)) + case BoolArrayOid: + values = append(values, decodeBoolArray(vr)) + case Int2ArrayOid: + values = append(values, decodeInt2Array(vr)) + case Int4ArrayOid: + values = append(values, decodeInt4Array(vr)) + case Int8ArrayOid: + values = append(values, decodeInt8Array(vr)) + case Float4ArrayOid: + values = append(values, decodeFloat4Array(vr)) + case Float8ArrayOid: + values = append(values, decodeFloat8Array(vr)) + case TextArrayOid, VarcharArrayOid: + values = append(values, decodeTextArray(vr)) + case TimestampArrayOid, TimestampTzArrayOid: + values = append(values, decodeTimestampArray(vr)) + case DateOid: + values = append(values, decodeDate(vr)) + case TimestampTzOid: + values = append(values, decodeTimestampTz(vr)) + case TimestampOid: + values = append(values, decodeTimestamp(vr)) + case InetOid, CidrOid: + values = append(values, decodeInet(vr)) + case JsonOid: + var d interface{} + decodeJSON(vr, &d) + values = append(values, d) + case JsonbOid: + var d interface{} + decodeJSONB(vr, &d) + values = append(values, d) + default: + rows.Fatal(errors.New("Values cannot handle binary format non-intrinsic types")) + } + default: + rows.Fatal(errors.New("Unknown format code")) + } + + if vr.Err() != nil { + rows.Fatal(vr.Err()) + } + + if rows.Err() != nil { + return nil, rows.Err() + } + } + + return values, rows.Err() +} + +// AfterClose adds f to a LILO queue of functions that will be called when +// rows is closed. +func (rows *Rows) AfterClose(f func(*Rows)) { + if rows.afterClose == nil { + rows.afterClose = f + } else { + prevFn := rows.afterClose + rows.afterClose = func(rows *Rows) { + f(rows) + prevFn(rows) + } + } +} + +// Query executes sql with args. If there is an error the returned *Rows will +// be returned in an error state. So it is allowed to ignore the error returned +// from Query and handle it in *Rows. +func (c *Conn) Query(sql string, args ...interface{}) (*Rows, error) { + c.lastActivityTime = time.Now() + + rows := c.getRows(sql, args) + + if err := c.lock(); err != nil { + rows.abort(err) + return rows, err + } + rows.unlockConn = true + + ps, ok := c.preparedStatements[sql] + if !ok { + var err error + ps, err = c.Prepare("", sql) + if err != nil { + rows.abort(err) + return rows, rows.err + } + } + rows.sql = ps.SQL + rows.fields = ps.FieldDescriptions + err := c.sendPreparedQuery(ps, args...) + if err != nil { + rows.abort(err) + } + return rows, rows.err +} + +func (c *Conn) getRows(sql string, args []interface{}) *Rows { + if len(c.preallocatedRows) == 0 { + c.preallocatedRows = make([]Rows, 64) + } + + r := &c.preallocatedRows[len(c.preallocatedRows)-1] + c.preallocatedRows = c.preallocatedRows[0 : len(c.preallocatedRows)-1] + + r.conn = c + r.startTime = c.lastActivityTime + r.sql = sql + r.args = args + + return r +} + +// QueryRow is a convenience wrapper over Query. Any error that occurs while +// querying is deferred until calling Scan on the returned *Row. That *Row will +// error with ErrNoRows if no rows are returned. +func (c *Conn) QueryRow(sql string, args ...interface{}) *Row { + rows, _ := c.Query(sql, args...) + return (*Row)(rows) +} diff --git a/vendor/github.com/jackc/pgx/replication.go b/vendor/github.com/jackc/pgx/replication.go new file mode 100644 index 0000000..7b28d6b --- /dev/null +++ b/vendor/github.com/jackc/pgx/replication.go @@ -0,0 +1,429 @@ +package pgx + +import ( + "errors" + "fmt" + "net" + "time" +) + +const ( + copyBothResponse = 'W' + walData = 'w' + senderKeepalive = 'k' + standbyStatusUpdate = 'r' + initialReplicationResponseTimeout = 5 * time.Second +) + +var epochNano int64 + +func init() { + epochNano = time.Date(2000, 1, 1, 0, 0, 0, 0, time.UTC).UnixNano() +} + +// Format the given 64bit LSN value into the XXX/XXX format, +// which is the format reported by postgres. +func FormatLSN(lsn uint64) string { + return fmt.Sprintf("%X/%X", uint32(lsn>>32), uint32(lsn)) +} + +// Parse the given XXX/XXX format LSN as reported by postgres, +// into a 64 bit integer as used internally by the wire procotols +func ParseLSN(lsn string) (outputLsn uint64, err error) { + var upperHalf uint64 + var lowerHalf uint64 + var nparsed int + nparsed, err = fmt.Sscanf(lsn, "%X/%X", &upperHalf, &lowerHalf) + if err != nil { + return + } + + if nparsed != 2 { + err = errors.New(fmt.Sprintf("Failed to parsed LSN: %s", lsn)) + return + } + + outputLsn = (upperHalf << 32) + lowerHalf + return +} + +// The WAL message contains WAL payload entry data +type WalMessage struct { + // The WAL start position of this data. This + // is the WAL position we need to track. + WalStart uint64 + // The server wal end and server time are + // documented to track the end position and current + // time of the server, both of which appear to be + // unimplemented in pg 9.5. + ServerWalEnd uint64 + ServerTime uint64 + // The WAL data is the raw unparsed binary WAL entry. + // The contents of this are determined by the output + // logical encoding plugin. + WalData []byte +} + +func (w *WalMessage) Time() time.Time { + return time.Unix(0, (int64(w.ServerTime)*1000)+epochNano) +} + +func (w *WalMessage) ByteLag() uint64 { + return (w.ServerWalEnd - w.WalStart) +} + +func (w *WalMessage) String() string { + return fmt.Sprintf("Wal: %s Time: %s Lag: %d", FormatLSN(w.WalStart), w.Time(), w.ByteLag()) +} + +// The server heartbeat is sent periodically from the server, +// including server status, and a reply request field +type ServerHeartbeat struct { + // The current max wal position on the server, + // used for lag tracking + ServerWalEnd uint64 + // The server time, in microseconds since jan 1 2000 + ServerTime uint64 + // If 1, the server is requesting a standby status message + // to be sent immediately. + ReplyRequested byte +} + +func (s *ServerHeartbeat) Time() time.Time { + return time.Unix(0, (int64(s.ServerTime)*1000)+epochNano) +} + +func (s *ServerHeartbeat) String() string { + return fmt.Sprintf("WalEnd: %s ReplyRequested: %d T: %s", FormatLSN(s.ServerWalEnd), s.ReplyRequested, s.Time()) +} + +// The replication message wraps all possible messages from the +// server received during replication. At most one of the wal message +// or server heartbeat will be non-nil +type ReplicationMessage struct { + WalMessage *WalMessage + ServerHeartbeat *ServerHeartbeat +} + +// The standby status is the client side heartbeat sent to the postgresql +// server to track the client wal positions. For practical purposes, +// all wal positions are typically set to the same value. +type StandbyStatus struct { + // The WAL position that's been locally written + WalWritePosition uint64 + // The WAL position that's been locally flushed + WalFlushPosition uint64 + // The WAL position that's been locally applied + WalApplyPosition uint64 + // The client time in microseconds since jan 1 2000 + ClientTime uint64 + // If 1, requests the server to immediately send a + // server heartbeat + ReplyRequested byte +} + +// Create a standby status struct, which sets all the WAL positions +// to the given wal position, and the client time to the current time. +// The wal positions are, in order: +// WalFlushPosition +// WalApplyPosition +// WalWritePosition +// +// If only one position is provided, it will be used as the value for all 3 +// status fields. Note you must provide either 1 wal position, or all 3 +// in order to initialize the standby status. +func NewStandbyStatus(walPositions ...uint64) (status *StandbyStatus, err error) { + if len(walPositions) == 1 { + status = new(StandbyStatus) + status.WalFlushPosition = walPositions[0] + status.WalApplyPosition = walPositions[0] + status.WalWritePosition = walPositions[0] + } else if len(walPositions) == 3 { + status = new(StandbyStatus) + status.WalFlushPosition = walPositions[0] + status.WalApplyPosition = walPositions[1] + status.WalWritePosition = walPositions[2] + } else { + err = errors.New(fmt.Sprintf("Invalid number of wal positions provided, need 1 or 3, got %d", len(walPositions))) + return + } + status.ClientTime = uint64((time.Now().UnixNano() - epochNano) / 1000) + return +} + +func ReplicationConnect(config ConnConfig) (r *ReplicationConn, err error) { + if config.RuntimeParams == nil { + config.RuntimeParams = make(map[string]string) + } + config.RuntimeParams["replication"] = "database" + + c, err := Connect(config) + if err != nil { + return + } + return &ReplicationConn{c: c}, nil +} + +type ReplicationConn struct { + c *Conn +} + +// Send standby status to the server, which both acts as a keepalive +// message to the server, as well as carries the WAL position of the +// client, which then updates the server's replication slot position. +func (rc *ReplicationConn) SendStandbyStatus(k *StandbyStatus) (err error) { + writeBuf := newWriteBuf(rc.c, copyData) + writeBuf.WriteByte(standbyStatusUpdate) + writeBuf.WriteInt64(int64(k.WalWritePosition)) + writeBuf.WriteInt64(int64(k.WalFlushPosition)) + writeBuf.WriteInt64(int64(k.WalApplyPosition)) + writeBuf.WriteInt64(int64(k.ClientTime)) + writeBuf.WriteByte(k.ReplyRequested) + + writeBuf.closeMsg() + + _, err = rc.c.conn.Write(writeBuf.buf) + if err != nil { + rc.c.die(err) + } + + return +} + +func (rc *ReplicationConn) Close() error { + return rc.c.Close() +} + +func (rc *ReplicationConn) IsAlive() bool { + return rc.c.IsAlive() +} + +func (rc *ReplicationConn) CauseOfDeath() error { + return rc.c.CauseOfDeath() +} + +func (rc *ReplicationConn) readReplicationMessage() (r *ReplicationMessage, err error) { + var t byte + var reader *msgReader + t, reader, err = rc.c.rxMsg() + if err != nil { + return + } + + switch t { + case noticeResponse: + pgError := rc.c.rxErrorResponse(reader) + if rc.c.shouldLog(LogLevelInfo) { + rc.c.log(LogLevelInfo, pgError.Error()) + } + case errorResponse: + err = rc.c.rxErrorResponse(reader) + if rc.c.shouldLog(LogLevelError) { + rc.c.log(LogLevelError, err.Error()) + } + return + case copyBothResponse: + // This is the tail end of the replication process start, + // and can be safely ignored + return + case copyData: + var msgType byte + msgType = reader.readByte() + switch msgType { + case walData: + walStart := reader.readInt64() + serverWalEnd := reader.readInt64() + serverTime := reader.readInt64() + walData := reader.readBytes(reader.msgBytesRemaining) + walMessage := WalMessage{WalStart: uint64(walStart), + ServerWalEnd: uint64(serverWalEnd), + ServerTime: uint64(serverTime), + WalData: walData, + } + + return &ReplicationMessage{WalMessage: &walMessage}, nil + case senderKeepalive: + serverWalEnd := reader.readInt64() + serverTime := reader.readInt64() + replyNow := reader.readByte() + h := &ServerHeartbeat{ServerWalEnd: uint64(serverWalEnd), ServerTime: uint64(serverTime), ReplyRequested: replyNow} + return &ReplicationMessage{ServerHeartbeat: h}, nil + default: + if rc.c.shouldLog(LogLevelError) { + rc.c.log(LogLevelError, "Unexpected data playload message type %v", t) + } + } + default: + if rc.c.shouldLog(LogLevelError) { + rc.c.log(LogLevelError, "Unexpected replication message type %v", t) + } + } + return +} + +// Wait for a single replication message up to timeout time. +// +// Properly using this requires some knowledge of the postgres replication mechanisms, +// as the client can receive both WAL data (the ultimate payload) and server heartbeat +// updates. The caller also must send standby status updates in order to keep the connection +// alive and working. +// +// This returns pgx.ErrNotificationTimeout when there is no replication message by the specified +// duration. +func (rc *ReplicationConn) WaitForReplicationMessage(timeout time.Duration) (r *ReplicationMessage, err error) { + var zeroTime time.Time + + deadline := time.Now().Add(timeout) + + // Use SetReadDeadline to implement the timeout. SetReadDeadline will + // cause operations to fail with a *net.OpError that has a Timeout() + // of true. Because the normal pgx rxMsg path considers any error to + // have potentially corrupted the state of the connection, it dies + // on any errors. So to avoid timeout errors in rxMsg we set the + // deadline and peek into the reader. If a timeout error occurs there + // we don't break the pgx connection. If the Peek returns that data + // is available then we turn off the read deadline before the rxMsg. + err = rc.c.conn.SetReadDeadline(deadline) + if err != nil { + return nil, err + } + + // Wait until there is a byte available before continuing onto the normal msg reading path + _, err = rc.c.reader.Peek(1) + if err != nil { + rc.c.conn.SetReadDeadline(zeroTime) // we can only return one error and we already have one -- so ignore possiple error from SetReadDeadline + if err, ok := err.(*net.OpError); ok && err.Timeout() { + return nil, ErrNotificationTimeout + } + return nil, err + } + + err = rc.c.conn.SetReadDeadline(zeroTime) + if err != nil { + return nil, err + } + + return rc.readReplicationMessage() +} + +func (rc *ReplicationConn) sendReplicationModeQuery(sql string) (*Rows, error) { + rc.c.lastActivityTime = time.Now() + + rows := rc.c.getRows(sql, nil) + + if err := rc.c.lock(); err != nil { + rows.abort(err) + return rows, err + } + rows.unlockConn = true + + err := rc.c.sendSimpleQuery(sql) + if err != nil { + rows.abort(err) + } + + var t byte + var r *msgReader + t, r, err = rc.c.rxMsg() + if err != nil { + return nil, err + } + + switch t { + case rowDescription: + rows.fields = rc.c.rxRowDescription(r) + // We don't have c.PgTypes here because we're a replication + // connection. This means the field descriptions will have + // only Oids. Not much we can do about this. + default: + if e := rc.c.processContextFreeMsg(t, r); e != nil { + rows.abort(e) + return rows, e + } + } + + return rows, rows.err +} + +// Execute the "IDENTIFY_SYSTEM" command as documented here: +// https://www.postgresql.org/docs/9.5/static/protocol-replication.html +// +// This will return (if successful) a result set that has a single row +// that contains the systemid, current timeline, xlogpos and database +// name. +// +// NOTE: Because this is a replication mode connection, we don't have +// type names, so the field descriptions in the result will have only +// Oids and no DataTypeName values +func (rc *ReplicationConn) IdentifySystem() (r *Rows, err error) { + return rc.sendReplicationModeQuery("IDENTIFY_SYSTEM") +} + +// Execute the "TIMELINE_HISTORY" command as documented here: +// https://www.postgresql.org/docs/9.5/static/protocol-replication.html +// +// This will return (if successful) a result set that has a single row +// that contains the filename of the history file and the content +// of the history file. If called for timeline 1, typically this will +// generate an error that the timeline history file does not exist. +// +// NOTE: Because this is a replication mode connection, we don't have +// type names, so the field descriptions in the result will have only +// Oids and no DataTypeName values +func (rc *ReplicationConn) TimelineHistory(timeline int) (r *Rows, err error) { + return rc.sendReplicationModeQuery(fmt.Sprintf("TIMELINE_HISTORY %d", timeline)) +} + +// Start a replication connection, sending WAL data to the given replication +// receiver. This function wraps a START_REPLICATION command as documented +// here: +// https://www.postgresql.org/docs/9.5/static/protocol-replication.html +// +// Once started, the client needs to invoke WaitForReplicationMessage() in order +// to fetch the WAL and standby status. Also, it is the responsibility of the caller +// to periodically send StandbyStatus messages to update the replication slot position. +// +// This function assumes that slotName has already been created. In order to omit the timeline argument +// pass a -1 for the timeline to get the server default behavior. +func (rc *ReplicationConn) StartReplication(slotName string, startLsn uint64, timeline int64, pluginArguments ...string) (err error) { + var queryString string + if timeline >= 0 { + queryString = fmt.Sprintf("START_REPLICATION SLOT %s LOGICAL %s TIMELINE %d", slotName, FormatLSN(startLsn), timeline) + } else { + queryString = fmt.Sprintf("START_REPLICATION SLOT %s LOGICAL %s", slotName, FormatLSN(startLsn)) + } + + for _, arg := range pluginArguments { + queryString += fmt.Sprintf(" %s", arg) + } + + if err = rc.c.sendQuery(queryString); err != nil { + return + } + + // The first replication message that comes back here will be (in a success case) + // a empty CopyBoth that is (apparently) sent as the confirmation that the replication has + // started. This call will either return nil, nil or if it returns an error + // that indicates the start replication command failed + var r *ReplicationMessage + r, err = rc.WaitForReplicationMessage(initialReplicationResponseTimeout) + if err != nil && r != nil { + if rc.c.shouldLog(LogLevelError) { + rc.c.log(LogLevelError, "Unxpected replication message %v", r) + } + } + + return +} + +// Create the replication slot, using the given name and output plugin. +func (rc *ReplicationConn) CreateReplicationSlot(slotName, outputPlugin string) (err error) { + _, err = rc.c.Exec(fmt.Sprintf("CREATE_REPLICATION_SLOT %s LOGICAL %s", slotName, outputPlugin)) + return +} + +// Drop the replication slot for the given name +func (rc *ReplicationConn) DropReplicationSlot(slotName string) (err error) { + _, err = rc.c.Exec(fmt.Sprintf("DROP_REPLICATION_SLOT %s", slotName)) + return +} diff --git a/vendor/github.com/jackc/pgx/sql.go b/vendor/github.com/jackc/pgx/sql.go new file mode 100644 index 0000000..7ee0f2a --- /dev/null +++ b/vendor/github.com/jackc/pgx/sql.go @@ -0,0 +1,29 @@ +package pgx + +import ( + "strconv" +) + +// QueryArgs is a container for arguments to an SQL query. It is helpful when +// building SQL statements where the number of arguments is variable. +type QueryArgs []interface{} + +var placeholders []string + +func init() { + placeholders = make([]string, 64) + + for i := 1; i < 64; i++ { + placeholders[i] = "$" + strconv.Itoa(i) + } +} + +// Append adds a value to qa and returns the placeholder value for the +// argument. e.g. $1, $2, etc. +func (qa *QueryArgs) Append(v interface{}) string { + *qa = append(*qa, v) + if len(*qa) < len(placeholders) { + return placeholders[len(*qa)] + } + return "$" + strconv.Itoa(len(*qa)) +} diff --git a/vendor/github.com/jackc/pgx/tx.go b/vendor/github.com/jackc/pgx/tx.go new file mode 100644 index 0000000..deb6c01 --- /dev/null +++ b/vendor/github.com/jackc/pgx/tx.go @@ -0,0 +1,207 @@ +package pgx + +import ( + "errors" + "fmt" +) + +// Transaction isolation levels +const ( + Serializable = "serializable" + RepeatableRead = "repeatable read" + ReadCommitted = "read committed" + ReadUncommitted = "read uncommitted" +) + +const ( + TxStatusInProgress = 0 + TxStatusCommitFailure = -1 + TxStatusRollbackFailure = -2 + TxStatusCommitSuccess = 1 + TxStatusRollbackSuccess = 2 +) + +var ErrTxClosed = errors.New("tx is closed") + +// ErrTxCommitRollback occurs when an error has occurred in a transaction and +// Commit() is called. PostgreSQL accepts COMMIT on aborted transactions, but +// it is treated as ROLLBACK. +var ErrTxCommitRollback = errors.New("commit unexpectedly resulted in rollback") + +// Begin starts a transaction with the default isolation level for the current +// connection. To use a specific isolation level see BeginIso. +func (c *Conn) Begin() (*Tx, error) { + return c.begin("") +} + +// BeginIso starts a transaction with isoLevel as the transaction isolation +// level. +// +// Valid isolation levels (and their constants) are: +// serializable (pgx.Serializable) +// repeatable read (pgx.RepeatableRead) +// read committed (pgx.ReadCommitted) +// read uncommitted (pgx.ReadUncommitted) +func (c *Conn) BeginIso(isoLevel string) (*Tx, error) { + return c.begin(isoLevel) +} + +func (c *Conn) begin(isoLevel string) (*Tx, error) { + var beginSQL string + if isoLevel == "" { + beginSQL = "begin" + } else { + beginSQL = fmt.Sprintf("begin isolation level %s", isoLevel) + } + + _, err := c.Exec(beginSQL) + if err != nil { + return nil, err + } + + return &Tx{conn: c}, nil +} + +// Tx represents a database transaction. +// +// All Tx methods return ErrTxClosed if Commit or Rollback has already been +// called on the Tx. +type Tx struct { + conn *Conn + afterClose func(*Tx) + err error + status int8 +} + +// Commit commits the transaction +func (tx *Tx) Commit() error { + if tx.status != TxStatusInProgress { + return ErrTxClosed + } + + commandTag, err := tx.conn.Exec("commit") + if err == nil && commandTag == "COMMIT" { + tx.status = TxStatusCommitSuccess + } else if err == nil && commandTag == "ROLLBACK" { + tx.status = TxStatusCommitFailure + tx.err = ErrTxCommitRollback + } else { + tx.status = TxStatusCommitFailure + tx.err = err + } + + if tx.afterClose != nil { + tx.afterClose(tx) + } + return tx.err +} + +// Rollback rolls back the transaction. Rollback will return ErrTxClosed if the +// Tx is already closed, but is otherwise safe to call multiple times. Hence, a +// defer tx.Rollback() is safe even if tx.Commit() will be called first in a +// non-error condition. +func (tx *Tx) Rollback() error { + if tx.status != TxStatusInProgress { + return ErrTxClosed + } + + _, tx.err = tx.conn.Exec("rollback") + if tx.err == nil { + tx.status = TxStatusRollbackSuccess + } else { + tx.status = TxStatusRollbackFailure + } + + if tx.afterClose != nil { + tx.afterClose(tx) + } + return tx.err +} + +// Exec delegates to the underlying *Conn +func (tx *Tx) Exec(sql string, arguments ...interface{}) (commandTag CommandTag, err error) { + if tx.status != TxStatusInProgress { + return CommandTag(""), ErrTxClosed + } + + return tx.conn.Exec(sql, arguments...) +} + +// Prepare delegates to the underlying *Conn +func (tx *Tx) Prepare(name, sql string) (*PreparedStatement, error) { + return tx.PrepareEx(name, sql, nil) +} + +// PrepareEx delegates to the underlying *Conn +func (tx *Tx) PrepareEx(name, sql string, opts *PrepareExOptions) (*PreparedStatement, error) { + if tx.status != TxStatusInProgress { + return nil, ErrTxClosed + } + + return tx.conn.PrepareEx(name, sql, opts) +} + +// Query delegates to the underlying *Conn +func (tx *Tx) Query(sql string, args ...interface{}) (*Rows, error) { + if tx.status != TxStatusInProgress { + // Because checking for errors can be deferred to the *Rows, build one with the error + err := ErrTxClosed + return &Rows{closed: true, err: err}, err + } + + return tx.conn.Query(sql, args...) +} + +// QueryRow delegates to the underlying *Conn +func (tx *Tx) QueryRow(sql string, args ...interface{}) *Row { + rows, _ := tx.Query(sql, args...) + return (*Row)(rows) +} + +// Deprecated. Use CopyFrom instead. CopyTo delegates to the underlying *Conn +func (tx *Tx) CopyTo(tableName string, columnNames []string, rowSrc CopyToSource) (int, error) { + if tx.status != TxStatusInProgress { + return 0, ErrTxClosed + } + + return tx.conn.CopyTo(tableName, columnNames, rowSrc) +} + +// CopyFrom delegates to the underlying *Conn +func (tx *Tx) CopyFrom(tableName Identifier, columnNames []string, rowSrc CopyFromSource) (int, error) { + if tx.status != TxStatusInProgress { + return 0, ErrTxClosed + } + + return tx.conn.CopyFrom(tableName, columnNames, rowSrc) +} + +// Conn returns the *Conn this transaction is using. +func (tx *Tx) Conn() *Conn { + return tx.conn +} + +// Status returns the status of the transaction from the set of +// pgx.TxStatus* constants. +func (tx *Tx) Status() int8 { + return tx.status +} + +// Err returns the final error state, if any, of calling Commit or Rollback. +func (tx *Tx) Err() error { + return tx.err +} + +// AfterClose adds f to a LILO queue of functions that will be called when +// the transaction is closed (either Commit or Rollback). +func (tx *Tx) AfterClose(f func(*Tx)) { + if tx.afterClose == nil { + tx.afterClose = f + } else { + prevFn := tx.afterClose + tx.afterClose = func(tx *Tx) { + f(tx) + prevFn(tx) + } + } +} diff --git a/vendor/github.com/jackc/pgx/value_reader.go b/vendor/github.com/jackc/pgx/value_reader.go new file mode 100644 index 0000000..a489754 --- /dev/null +++ b/vendor/github.com/jackc/pgx/value_reader.go @@ -0,0 +1,156 @@ +package pgx + +import ( + "errors" +) + +// ValueReader is used by the Scanner interface to decode values. +type ValueReader struct { + mr *msgReader + fd *FieldDescription + valueBytesRemaining int32 + err error +} + +// Err returns any error that the ValueReader has experienced +func (r *ValueReader) Err() error { + return r.err +} + +// Fatal tells r that a Fatal error has occurred +func (r *ValueReader) Fatal(err error) { + r.err = err +} + +// Len returns the number of unread bytes +func (r *ValueReader) Len() int32 { + return r.valueBytesRemaining +} + +// Type returns the *FieldDescription of the value +func (r *ValueReader) Type() *FieldDescription { + return r.fd +} + +func (r *ValueReader) ReadByte() byte { + if r.err != nil { + return 0 + } + + r.valueBytesRemaining-- + if r.valueBytesRemaining < 0 { + r.Fatal(errors.New("read past end of value")) + return 0 + } + + return r.mr.readByte() +} + +func (r *ValueReader) ReadInt16() int16 { + if r.err != nil { + return 0 + } + + r.valueBytesRemaining -= 2 + if r.valueBytesRemaining < 0 { + r.Fatal(errors.New("read past end of value")) + return 0 + } + + return r.mr.readInt16() +} + +func (r *ValueReader) ReadUint16() uint16 { + if r.err != nil { + return 0 + } + + r.valueBytesRemaining -= 2 + if r.valueBytesRemaining < 0 { + r.Fatal(errors.New("read past end of value")) + return 0 + } + + return r.mr.readUint16() +} + +func (r *ValueReader) ReadInt32() int32 { + if r.err != nil { + return 0 + } + + r.valueBytesRemaining -= 4 + if r.valueBytesRemaining < 0 { + r.Fatal(errors.New("read past end of value")) + return 0 + } + + return r.mr.readInt32() +} + +func (r *ValueReader) ReadUint32() uint32 { + if r.err != nil { + return 0 + } + + r.valueBytesRemaining -= 4 + if r.valueBytesRemaining < 0 { + r.Fatal(errors.New("read past end of value")) + return 0 + } + + return r.mr.readUint32() +} + +func (r *ValueReader) ReadInt64() int64 { + if r.err != nil { + return 0 + } + + r.valueBytesRemaining -= 8 + if r.valueBytesRemaining < 0 { + r.Fatal(errors.New("read past end of value")) + return 0 + } + + return r.mr.readInt64() +} + +func (r *ValueReader) ReadOid() Oid { + return Oid(r.ReadUint32()) +} + +// ReadString reads count bytes and returns as string +func (r *ValueReader) ReadString(count int32) string { + if r.err != nil { + return "" + } + + r.valueBytesRemaining -= count + if r.valueBytesRemaining < 0 { + r.Fatal(errors.New("read past end of value")) + return "" + } + + return r.mr.readString(count) +} + +// ReadBytes reads count bytes and returns as []byte +func (r *ValueReader) ReadBytes(count int32) []byte { + if r.err != nil { + return nil + } + + if count < 0 { + r.Fatal(errors.New("count must not be negative")) + return nil + } + + r.valueBytesRemaining -= count + if r.valueBytesRemaining < 0 { + r.Fatal(errors.New("read past end of value")) + return nil + } + + return r.mr.readBytes(count) +} diff --git a/vendor/github.com/jackc/pgx/values.go b/vendor/github.com/jackc/pgx/values.go new file mode 100644 index 0000000..a189e18 --- /dev/null +++ b/vendor/github.com/jackc/pgx/values.go @@ -0,0 +1,3439 @@ +package pgx + +import ( + "bytes" + "database/sql/driver" + "encoding/json" + "fmt" + "io" + "math" + "net" + "reflect" + "regexp" + "strconv" + "strings" + "time" +) + +// PostgreSQL oids for common types +const ( + BoolOid = 16 + ByteaOid = 17 + CharOid = 18 + NameOid = 19 + Int8Oid = 20 + Int2Oid = 21 + Int4Oid = 23 + TextOid = 25 + OidOid = 26 + TidOid = 27 + XidOid = 28 + CidOid = 29 + JsonOid = 114 + CidrOid = 650 + CidrArrayOid = 651 + Float4Oid = 700 + Float8Oid = 701 + UnknownOid = 705 + InetOid = 869 + BoolArrayOid = 1000 + Int2ArrayOid = 1005 + Int4ArrayOid = 1007 + TextArrayOid = 1009 + ByteaArrayOid = 1001 + VarcharArrayOid = 1015 + Int8ArrayOid = 1016 + Float4ArrayOid = 1021 + Float8ArrayOid = 1022 + AclItemOid = 1033 + AclItemArrayOid = 1034 + InetArrayOid = 1041 + VarcharOid = 1043 + DateOid = 1082 + TimestampOid = 1114 + TimestampArrayOid = 1115 + TimestampTzOid = 1184 + TimestampTzArrayOid = 1185 + RecordOid = 2249 + UuidOid = 2950 + JsonbOid = 3802 +) + +// PostgreSQL format codes +const ( + TextFormatCode = 0 + BinaryFormatCode = 1 +) + +const maxUint = ^uint(0) +const maxInt = int(maxUint >> 1) +const minInt = -maxInt - 1 + +// DefaultTypeFormats maps type names to their default requested format (text +// or binary). In theory the Scanner interface should be the one to determine +// the format of the returned values. However, the query has already been +// executed by the time Scan is called so it has no chance to set the format. +// So for types that should always be returned in binary the format should be +// set here. +var DefaultTypeFormats map[string]int16 + +func init() { + DefaultTypeFormats = map[string]int16{ + "_aclitem": TextFormatCode, // Pg's src/backend/utils/adt/acl.c has only in/out (text) not send/recv (bin) + "_bool": BinaryFormatCode, + "_bytea": BinaryFormatCode, + "_cidr": BinaryFormatCode, + "_float4": BinaryFormatCode, + "_float8": BinaryFormatCode, + "_inet": BinaryFormatCode, + "_int2": BinaryFormatCode, + "_int4": BinaryFormatCode, + "_int8": BinaryFormatCode, + "_text": BinaryFormatCode, + "_timestamp": BinaryFormatCode, + "_timestamptz": BinaryFormatCode, + "_varchar": BinaryFormatCode, + "aclitem": TextFormatCode, // Pg's src/backend/utils/adt/acl.c has only in/out (text) not send/recv (bin) + "bool": BinaryFormatCode, + "bytea": BinaryFormatCode, + "char": BinaryFormatCode, + "cid": BinaryFormatCode, + "cidr": BinaryFormatCode, + "date": BinaryFormatCode, + "float4": BinaryFormatCode, + "float8": BinaryFormatCode, + "json": BinaryFormatCode, + "jsonb": BinaryFormatCode, + "inet": BinaryFormatCode, + "int2": BinaryFormatCode, + "int4": BinaryFormatCode, + "int8": BinaryFormatCode, + "name": BinaryFormatCode, + "oid": BinaryFormatCode, + "record": BinaryFormatCode, + "text": BinaryFormatCode, + "tid": BinaryFormatCode, + "timestamp": BinaryFormatCode, + "timestamptz": BinaryFormatCode, + "varchar": BinaryFormatCode, + "xid": BinaryFormatCode, + } +} + +// SerializationError occurs on failure to encode or decode a value +type SerializationError string + +func (e SerializationError) Error() string { + return string(e) +} + +// Deprecated: Scanner is an interface used to decode values from the PostgreSQL +// server. To allow types to support pgx and database/sql.Scan this interface +// has been deprecated in favor of PgxScanner. +type Scanner interface { + // Scan MUST check r.Type().DataType (to check by OID) or + // r.Type().DataTypeName (to check by name) to ensure that it is scanning an + // expected column type. It also MUST check r.Type().FormatCode before + // decoding. It should not assume that it was called on a data type or format + // that it understands. + Scan(r *ValueReader) error +} + +// PgxScanner is an interface used to decode values from the PostgreSQL server. +// It is used exactly the same as the Scanner interface. It simply has renamed +// the method. +type PgxScanner interface { + // ScanPgx MUST check r.Type().DataType (to check by OID) or + // r.Type().DataTypeName (to check by name) to ensure that it is scanning an + // expected column type. It also MUST check r.Type().FormatCode before + // decoding. It should not assume that it was called on a data type or format + // that it understands. + ScanPgx(r *ValueReader) error +} + +// Encoder is an interface used to encode values for transmission to the +// PostgreSQL server. +type Encoder interface { + // Encode writes the value to w. + // + // If the value is NULL an int32(-1) should be written. + // + // Encode MUST check oid to see if the parameter data type is compatible. If + // this is not done, the PostgreSQL server may detect the error if the + // expected data size or format of the encoded data does not match. But if + // the encoded data is a valid representation of the data type PostgreSQL + // expects such as date and int4, incorrect data may be stored. + Encode(w *WriteBuf, oid Oid) error + + // FormatCode returns the format that the encoder writes the value. It must be + // either pgx.TextFormatCode or pgx.BinaryFormatCode. + FormatCode() int16 +} + +// NullFloat32 represents an float4 that may be null. NullFloat32 implements the +// Scanner and Encoder interfaces so it may be used both as an argument to +// Query[Row] and a destination for Scan. +// +// If Valid is false then the value is NULL. +type NullFloat32 struct { + Float32 float32 + Valid bool // Valid is true if Float32 is not NULL +} + +func (n *NullFloat32) Scan(vr *ValueReader) error { + if vr.Type().DataType != Float4Oid { + return SerializationError(fmt.Sprintf("NullFloat32.Scan cannot decode OID %d", vr.Type().DataType)) + } + + if vr.Len() == -1 { + n.Float32, n.Valid = 0, false + return nil + } + n.Valid = true + n.Float32 = decodeFloat4(vr) + return vr.Err() +} + +func (n NullFloat32) FormatCode() int16 { return BinaryFormatCode } + +func (n NullFloat32) Encode(w *WriteBuf, oid Oid) error { + if oid != Float4Oid { + return SerializationError(fmt.Sprintf("NullFloat32.Encode cannot encode into OID %d", oid)) + } + + if !n.Valid { + w.WriteInt32(-1) + return nil + } + + return encodeFloat32(w, oid, n.Float32) +} + +// NullFloat64 represents an float8 that may be null. NullFloat64 implements the +// Scanner and Encoder interfaces so it may be used both as an argument to +// Query[Row] and a destination for Scan. +// +// If Valid is false then the value is NULL. +type NullFloat64 struct { + Float64 float64 + Valid bool // Valid is true if Float64 is not NULL +} + +func (n *NullFloat64) Scan(vr *ValueReader) error { + if vr.Type().DataType != Float8Oid { + return SerializationError(fmt.Sprintf("NullFloat64.Scan cannot decode OID %d", vr.Type().DataType)) + } + + if vr.Len() == -1 { + n.Float64, n.Valid = 0, false + return nil + } + n.Valid = true + n.Float64 = decodeFloat8(vr) + return vr.Err() +} + +func (n NullFloat64) FormatCode() int16 { return BinaryFormatCode } + +func (n NullFloat64) Encode(w *WriteBuf, oid Oid) error { + if oid != Float8Oid { + return SerializationError(fmt.Sprintf("NullFloat64.Encode cannot encode into OID %d", oid)) + } + + if !n.Valid { + w.WriteInt32(-1) + return nil + } + + return encodeFloat64(w, oid, n.Float64) +} + +// NullString represents an string that may be null. NullString implements the +// Scanner Encoder interfaces so it may be used both as an argument to +// Query[Row] and a destination for Scan. +// +// If Valid is false then the value is NULL. +type NullString struct { + String string + Valid bool // Valid is true if String is not NULL +} + +func (n *NullString) Scan(vr *ValueReader) error { + // Not checking oid as so we can scan anything into into a NullString - may revisit this decision later + + if vr.Len() == -1 { + n.String, n.Valid = "", false + return nil + } + + n.Valid = true + n.String = decodeText(vr) + return vr.Err() +} + +func (n NullString) FormatCode() int16 { return TextFormatCode } + +func (n NullString) Encode(w *WriteBuf, oid Oid) error { + if !n.Valid { + w.WriteInt32(-1) + return nil + } + + return encodeString(w, oid, n.String) +} + +// AclItem is used for PostgreSQL's aclitem data type. A sample aclitem +// might look like this: +// +// postgres=arwdDxt/postgres +// +// Note, however, that because the user/role name part of an aclitem is +// an identifier, it follows all the usual formatting rules for SQL +// identifiers: if it contains spaces and other special characters, +// it should appear in double-quotes: +// +// postgres=arwdDxt/"role with spaces" +// +type AclItem string + +// NullAclItem represents a pgx.AclItem that may be null. NullAclItem implements the +// Scanner and Encoder interfaces so it may be used both as an argument to +// Query[Row] and a destination for Scan for prepared and unprepared queries. +// +// If Valid is false then the value is NULL. +type NullAclItem struct { + AclItem AclItem + Valid bool // Valid is true if AclItem is not NULL +} + +func (n *NullAclItem) Scan(vr *ValueReader) error { + if vr.Type().DataType != AclItemOid { + return SerializationError(fmt.Sprintf("NullAclItem.Scan cannot decode OID %d", vr.Type().DataType)) + } + + if vr.Len() == -1 { + n.AclItem, n.Valid = "", false + return nil + } + + n.Valid = true + n.AclItem = AclItem(decodeText(vr)) + return vr.Err() +} + +// Particularly important to return TextFormatCode, seeing as Postgres +// only ever sends aclitem as text, not binary. +func (n NullAclItem) FormatCode() int16 { return TextFormatCode } + +func (n NullAclItem) Encode(w *WriteBuf, oid Oid) error { + if !n.Valid { + w.WriteInt32(-1) + return nil + } + + return encodeString(w, oid, string(n.AclItem)) +} + +// Name is a type used for PostgreSQL's special 63-byte +// name data type, used for identifiers like table names. +// The pg_class.relname column is a good example of where the +// name data type is used. +// +// Note that the underlying Go data type of pgx.Name is string, +// so there is no way to enforce the 63-byte length. Inputting +// a longer name into PostgreSQL will result in silent truncation +// to 63 bytes. +// +// Also, if you have custom-compiled PostgreSQL and set +// NAMEDATALEN to a different value, obviously that number of +// bytes applies, rather than the default 63. +type Name string + +// NullName represents a pgx.Name that may be null. NullName implements the +// Scanner and Encoder interfaces so it may be used both as an argument to +// Query[Row] and a destination for Scan for prepared and unprepared queries. +// +// If Valid is false then the value is NULL. +type NullName struct { + Name Name + Valid bool // Valid is true if Name is not NULL +} + +func (n *NullName) Scan(vr *ValueReader) error { + if vr.Type().DataType != NameOid { + return SerializationError(fmt.Sprintf("NullName.Scan cannot decode OID %d", vr.Type().DataType)) + } + + if vr.Len() == -1 { + n.Name, n.Valid = "", false + return nil + } + + n.Valid = true + n.Name = Name(decodeText(vr)) + return vr.Err() +} + +func (n NullName) FormatCode() int16 { return TextFormatCode } + +func (n NullName) Encode(w *WriteBuf, oid Oid) error { + if !n.Valid { + w.WriteInt32(-1) + return nil + } + + return encodeString(w, oid, string(n.Name)) +} + +// The pgx.Char type is for PostgreSQL's special 8-bit-only +// "char" type more akin to the C language's char type, or Go's byte type. +// (Note that the name in PostgreSQL itself is "char", in double-quotes, +// and not char.) It gets used a lot in PostgreSQL's system tables to hold +// a single ASCII character value (eg pg_class.relkind). +type Char byte + +// NullChar represents a pgx.Char that may be null. NullChar implements the +// Scanner and Encoder interfaces so it may be used both as an argument to +// Query[Row] and a destination for Scan for prepared and unprepared queries. +// +// If Valid is false then the value is NULL. +type NullChar struct { + Char Char + Valid bool // Valid is true if Char is not NULL +} + +func (n *NullChar) Scan(vr *ValueReader) error { + if vr.Type().DataType != CharOid { + return SerializationError(fmt.Sprintf("NullChar.Scan cannot decode OID %d", vr.Type().DataType)) + } + + if vr.Len() == -1 { + n.Char, n.Valid = 0, false + return nil + } + n.Valid = true + n.Char = decodeChar(vr) + return vr.Err() +} + +func (n NullChar) FormatCode() int16 { return BinaryFormatCode } + +func (n NullChar) Encode(w *WriteBuf, oid Oid) error { + if oid != CharOid { + return SerializationError(fmt.Sprintf("NullChar.Encode cannot encode into OID %d", oid)) + } + + if !n.Valid { + w.WriteInt32(-1) + return nil + } + + return encodeChar(w, oid, n.Char) +} + +// NullInt16 represents a smallint that may be null. NullInt16 implements the +// Scanner and Encoder interfaces so it may be used both as an argument to +// Query[Row] and a destination for Scan for prepared and unprepared queries. +// +// If Valid is false then the value is NULL. +type NullInt16 struct { + Int16 int16 + Valid bool // Valid is true if Int16 is not NULL +} + +func (n *NullInt16) Scan(vr *ValueReader) error { + if vr.Type().DataType != Int2Oid { + return SerializationError(fmt.Sprintf("NullInt16.Scan cannot decode OID %d", vr.Type().DataType)) + } + + if vr.Len() == -1 { + n.Int16, n.Valid = 0, false + return nil + } + n.Valid = true + n.Int16 = decodeInt2(vr) + return vr.Err() +} + +func (n NullInt16) FormatCode() int16 { return BinaryFormatCode } + +func (n NullInt16) Encode(w *WriteBuf, oid Oid) error { + if oid != Int2Oid { + return SerializationError(fmt.Sprintf("NullInt16.Encode cannot encode into OID %d", oid)) + } + + if !n.Valid { + w.WriteInt32(-1) + return nil + } + + return encodeInt16(w, oid, n.Int16) +} + +// NullInt32 represents an integer that may be null. NullInt32 implements the +// Scanner and Encoder interfaces so it may be used both as an argument to +// Query[Row] and a destination for Scan. +// +// If Valid is false then the value is NULL. +type NullInt32 struct { + Int32 int32 + Valid bool // Valid is true if Int32 is not NULL +} + +func (n *NullInt32) Scan(vr *ValueReader) error { + if vr.Type().DataType != Int4Oid { + return SerializationError(fmt.Sprintf("NullInt32.Scan cannot decode OID %d", vr.Type().DataType)) + } + + if vr.Len() == -1 { + n.Int32, n.Valid = 0, false + return nil + } + n.Valid = true + n.Int32 = decodeInt4(vr) + return vr.Err() +} + +func (n NullInt32) FormatCode() int16 { return BinaryFormatCode } + +func (n NullInt32) Encode(w *WriteBuf, oid Oid) error { + if oid != Int4Oid { + return SerializationError(fmt.Sprintf("NullInt32.Encode cannot encode into OID %d", oid)) + } + + if !n.Valid { + w.WriteInt32(-1) + return nil + } + + return encodeInt32(w, oid, n.Int32) +} + +// Oid (Object Identifier Type) is, according to https://www.postgresql.org/docs/current/static/datatype-oid.html, +// used internally by PostgreSQL as a primary key for various system tables. It is currently implemented +// as an unsigned four-byte integer. Its definition can be found in src/include/postgres_ext.h +// in the PostgreSQL sources. +type Oid uint32 + +// NullOid represents a Command Identifier (Oid) that may be null. NullOid implements the +// Scanner and Encoder interfaces so it may be used both as an argument to +// Query[Row] and a destination for Scan. +// +// If Valid is false then the value is NULL. +type NullOid struct { + Oid Oid + Valid bool // Valid is true if Oid is not NULL +} + +func (n *NullOid) Scan(vr *ValueReader) error { + if vr.Type().DataType != OidOid { + return SerializationError(fmt.Sprintf("NullOid.Scan cannot decode OID %d", vr.Type().DataType)) + } + + if vr.Len() == -1 { + n.Oid, n.Valid = 0, false + return nil + } + n.Valid = true + n.Oid = decodeOid(vr) + return vr.Err() +} + +func (n NullOid) FormatCode() int16 { return BinaryFormatCode } + +func (n NullOid) Encode(w *WriteBuf, oid Oid) error { + if oid != OidOid { + return SerializationError(fmt.Sprintf("NullOid.Encode cannot encode into OID %d", oid)) + } + + if !n.Valid { + w.WriteInt32(-1) + return nil + } + + return encodeOid(w, oid, n.Oid) +} + +// Xid is PostgreSQL's Transaction ID type. +// +// In later versions of PostgreSQL, it is the type used for the backend_xid +// and backend_xmin columns of the pg_stat_activity system view. +// +// Also, when one does +// +// select xmin, xmax, * from some_table; +// +// it is the data type of the xmin and xmax hidden system columns. +// +// It is currently implemented as an unsigned four byte integer. +// Its definition can be found in src/include/postgres_ext.h as TransactionId +// in the PostgreSQL sources. +type Xid uint32 + +// NullXid represents a Transaction ID (Xid) that may be null. NullXid implements the +// Scanner and Encoder interfaces so it may be used both as an argument to +// Query[Row] and a destination for Scan. +// +// If Valid is false then the value is NULL. +type NullXid struct { + Xid Xid + Valid bool // Valid is true if Xid is not NULL +} + +func (n *NullXid) Scan(vr *ValueReader) error { + if vr.Type().DataType != XidOid { + return SerializationError(fmt.Sprintf("NullXid.Scan cannot decode OID %d", vr.Type().DataType)) + } + + if vr.Len() == -1 { + n.Xid, n.Valid = 0, false + return nil + } + n.Valid = true + n.Xid = decodeXid(vr) + return vr.Err() +} + +func (n NullXid) FormatCode() int16 { return BinaryFormatCode } + +func (n NullXid) Encode(w *WriteBuf, oid Oid) error { + if oid != XidOid { + return SerializationError(fmt.Sprintf("NullXid.Encode cannot encode into OID %d", oid)) + } + + if !n.Valid { + w.WriteInt32(-1) + return nil + } + + return encodeXid(w, oid, n.Xid) +} + +// Cid is PostgreSQL's Command Identifier type. +// +// When one does +// +// select cmin, cmax, * from some_table; +// +// it is the data type of the cmin and cmax hidden system columns. +// +// It is currently implemented as an unsigned four byte integer. +// Its definition can be found in src/include/c.h as CommandId +// in the PostgreSQL sources. +type Cid uint32 + +// NullCid represents a Command Identifier (Cid) that may be null. NullCid implements the +// Scanner and Encoder interfaces so it may be used both as an argument to +// Query[Row] and a destination for Scan. +// +// If Valid is false then the value is NULL. +type NullCid struct { + Cid Cid + Valid bool // Valid is true if Cid is not NULL +} + +func (n *NullCid) Scan(vr *ValueReader) error { + if vr.Type().DataType != CidOid { + return SerializationError(fmt.Sprintf("NullCid.Scan cannot decode OID %d", vr.Type().DataType)) + } + + if vr.Len() == -1 { + n.Cid, n.Valid = 0, false + return nil + } + n.Valid = true + n.Cid = decodeCid(vr) + return vr.Err() +} + +func (n NullCid) FormatCode() int16 { return BinaryFormatCode } + +func (n NullCid) Encode(w *WriteBuf, oid Oid) error { + if oid != CidOid { + return SerializationError(fmt.Sprintf("NullCid.Encode cannot encode into OID %d", oid)) + } + + if !n.Valid { + w.WriteInt32(-1) + return nil + } + + return encodeCid(w, oid, n.Cid) +} + +// Tid is PostgreSQL's Tuple Identifier type. +// +// When one does +// +// select ctid, * from some_table; +// +// it is the data type of the ctid hidden system column. +// +// It is currently implemented as a pair unsigned two byte integers. +// Its conversion functions can be found in src/backend/utils/adt/tid.c +// in the PostgreSQL sources. +type Tid struct { + BlockNumber uint32 + OffsetNumber uint16 +} + +// NullTid represents a Tuple Identifier (Tid) that may be null. NullTid implements the +// Scanner and Encoder interfaces so it may be used both as an argument to +// Query[Row] and a destination for Scan. +// +// If Valid is false then the value is NULL. +type NullTid struct { + Tid Tid + Valid bool // Valid is true if Tid is not NULL +} + +func (n *NullTid) Scan(vr *ValueReader) error { + if vr.Type().DataType != TidOid { + return SerializationError(fmt.Sprintf("NullTid.Scan cannot decode OID %d", vr.Type().DataType)) + } + + if vr.Len() == -1 { + n.Tid, n.Valid = Tid{BlockNumber: 0, OffsetNumber: 0}, false + return nil + } + n.Valid = true + n.Tid = decodeTid(vr) + return vr.Err() +} + +func (n NullTid) FormatCode() int16 { return BinaryFormatCode } + +func (n NullTid) Encode(w *WriteBuf, oid Oid) error { + if oid != TidOid { + return SerializationError(fmt.Sprintf("NullTid.Encode cannot encode into OID %d", oid)) + } + + if !n.Valid { + w.WriteInt32(-1) + return nil + } + + return encodeTid(w, oid, n.Tid) +} + +// NullInt64 represents an bigint that may be null. NullInt64 implements the +// Scanner and Encoder interfaces so it may be used both as an argument to +// Query[Row] and a destination for Scan. +// +// If Valid is false then the value is NULL. +type NullInt64 struct { + Int64 int64 + Valid bool // Valid is true if Int64 is not NULL +} + +func (n *NullInt64) Scan(vr *ValueReader) error { + if vr.Type().DataType != Int8Oid { + return SerializationError(fmt.Sprintf("NullInt64.Scan cannot decode OID %d", vr.Type().DataType)) + } + + if vr.Len() == -1 { + n.Int64, n.Valid = 0, false + return nil + } + n.Valid = true + n.Int64 = decodeInt8(vr) + return vr.Err() +} + +func (n NullInt64) FormatCode() int16 { return BinaryFormatCode } + +func (n NullInt64) Encode(w *WriteBuf, oid Oid) error { + if oid != Int8Oid { + return SerializationError(fmt.Sprintf("NullInt64.Encode cannot encode into OID %d", oid)) + } + + if !n.Valid { + w.WriteInt32(-1) + return nil + } + + return encodeInt64(w, oid, n.Int64) +} + +// NullBool represents an bool that may be null. NullBool implements the Scanner +// and Encoder interfaces so it may be used both as an argument to Query[Row] +// and a destination for Scan. +// +// If Valid is false then the value is NULL. +type NullBool struct { + Bool bool + Valid bool // Valid is true if Bool is not NULL +} + +func (n *NullBool) Scan(vr *ValueReader) error { + if vr.Type().DataType != BoolOid { + return SerializationError(fmt.Sprintf("NullBool.Scan cannot decode OID %d", vr.Type().DataType)) + } + + if vr.Len() == -1 { + n.Bool, n.Valid = false, false + return nil + } + n.Valid = true + n.Bool = decodeBool(vr) + return vr.Err() +} + +func (n NullBool) FormatCode() int16 { return BinaryFormatCode } + +func (n NullBool) Encode(w *WriteBuf, oid Oid) error { + if oid != BoolOid { + return SerializationError(fmt.Sprintf("NullBool.Encode cannot encode into OID %d", oid)) + } + + if !n.Valid { + w.WriteInt32(-1) + return nil + } + + return encodeBool(w, oid, n.Bool) +} + +// NullTime represents an time.Time that may be null. NullTime implements the +// Scanner and Encoder interfaces so it may be used both as an argument to +// Query[Row] and a destination for Scan. It corresponds with the PostgreSQL +// types timestamptz, timestamp, and date. +// +// If Valid is false then the value is NULL. +type NullTime struct { + Time time.Time + Valid bool // Valid is true if Time is not NULL +} + +func (n *NullTime) Scan(vr *ValueReader) error { + oid := vr.Type().DataType + if oid != TimestampTzOid && oid != TimestampOid && oid != DateOid { + return SerializationError(fmt.Sprintf("NullTime.Scan cannot decode OID %d", vr.Type().DataType)) + } + + if vr.Len() == -1 { + n.Time, n.Valid = time.Time{}, false + return nil + } + + n.Valid = true + switch oid { + case TimestampTzOid: + n.Time = decodeTimestampTz(vr) + case TimestampOid: + n.Time = decodeTimestamp(vr) + case DateOid: + n.Time = decodeDate(vr) + } + + return vr.Err() +} + +func (n NullTime) FormatCode() int16 { return BinaryFormatCode } + +func (n NullTime) Encode(w *WriteBuf, oid Oid) error { + if oid != TimestampTzOid && oid != TimestampOid && oid != DateOid { + return SerializationError(fmt.Sprintf("NullTime.Encode cannot encode into OID %d", oid)) + } + + if !n.Valid { + w.WriteInt32(-1) + return nil + } + + return encodeTime(w, oid, n.Time) +} + +// Hstore represents an hstore column. It does not support a null column or null +// key values (use NullHstore for this). Hstore implements the Scanner and +// Encoder interfaces so it may be used both as an argument to Query[Row] and a +// destination for Scan. +type Hstore map[string]string + +func (h *Hstore) Scan(vr *ValueReader) error { + //oid for hstore not standardized, so we check its type name + if vr.Type().DataTypeName != "hstore" { + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode type %s into Hstore", vr.Type().DataTypeName))) + return nil + } + + if vr.Len() == -1 { + vr.Fatal(ProtocolError("Cannot decode null column into Hstore")) + return nil + } + + switch vr.Type().FormatCode { + case TextFormatCode: + m, err := parseHstoreToMap(vr.ReadString(vr.Len())) + if err != nil { + vr.Fatal(ProtocolError(fmt.Sprintf("Can't decode hstore column: %v", err))) + return nil + } + hm := Hstore(m) + *h = hm + return nil + case BinaryFormatCode: + vr.Fatal(ProtocolError("Can't decode binary hstore")) + return nil + default: + vr.Fatal(ProtocolError(fmt.Sprintf("Unknown field description format code: %v", vr.Type().FormatCode))) + return nil + } +} + +func (h Hstore) FormatCode() int16 { return TextFormatCode } + +func (h Hstore) Encode(w *WriteBuf, oid Oid) error { + var buf bytes.Buffer + + i := 0 + for k, v := range h { + i++ + ks := strings.Replace(k, `\`, `\\`, -1) + ks = strings.Replace(ks, `"`, `\"`, -1) + vs := strings.Replace(v, `\`, `\\`, -1) + vs = strings.Replace(vs, `"`, `\"`, -1) + buf.WriteString(`"`) + buf.WriteString(ks) + buf.WriteString(`"=>"`) + buf.WriteString(vs) + buf.WriteString(`"`) + if i < len(h) { + buf.WriteString(", ") + } + } + w.WriteInt32(int32(buf.Len())) + w.WriteBytes(buf.Bytes()) + return nil +} + +// NullHstore represents an hstore column that can be null or have null values +// associated with its keys. NullHstore implements the Scanner and Encoder +// interfaces so it may be used both as an argument to Query[Row] and a +// destination for Scan. +// +// If Valid is false, then the value of the entire hstore column is NULL +// If any of the NullString values in Store has Valid set to false, the key +// appears in the hstore column, but its value is explicitly set to NULL. +type NullHstore struct { + Hstore map[string]NullString + Valid bool +} + +func (h *NullHstore) Scan(vr *ValueReader) error { + //oid for hstore not standardized, so we check its type name + if vr.Type().DataTypeName != "hstore" { + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode type %s into NullHstore", vr.Type().DataTypeName))) + return nil + } + + if vr.Len() == -1 { + h.Valid = false + return nil + } + + switch vr.Type().FormatCode { + case TextFormatCode: + store, err := parseHstoreToNullHstore(vr.ReadString(vr.Len())) + if err != nil { + vr.Fatal(ProtocolError(fmt.Sprintf("Can't decode hstore column: %v", err))) + return nil + } + h.Valid = true + h.Hstore = store + return nil + case BinaryFormatCode: + vr.Fatal(ProtocolError("Can't decode binary hstore")) + return nil + default: + vr.Fatal(ProtocolError(fmt.Sprintf("Unknown field description format code: %v", vr.Type().FormatCode))) + return nil + } +} + +func (h NullHstore) FormatCode() int16 { return TextFormatCode } + +func (h NullHstore) Encode(w *WriteBuf, oid Oid) error { + var buf bytes.Buffer + + if !h.Valid { + w.WriteInt32(-1) + return nil + } + + i := 0 + for k, v := range h.Hstore { + i++ + ks := strings.Replace(k, `\`, `\\`, -1) + ks = strings.Replace(ks, `"`, `\"`, -1) + if v.Valid { + vs := strings.Replace(v.String, `\`, `\\`, -1) + vs = strings.Replace(vs, `"`, `\"`, -1) + buf.WriteString(fmt.Sprintf(`"%s"=>"%s"`, ks, vs)) + } else { + buf.WriteString(fmt.Sprintf(`"%s"=>NULL`, ks)) + } + if i < len(h.Hstore) { + buf.WriteString(", ") + } + } + w.WriteInt32(int32(buf.Len())) + w.WriteBytes(buf.Bytes()) + return nil +} + +// Encode encodes arg into wbuf as the type oid. This allows implementations +// of the Encoder interface to delegate the actual work of encoding to the +// built-in functionality. +func Encode(wbuf *WriteBuf, oid Oid, arg interface{}) error { + if arg == nil { + wbuf.WriteInt32(-1) + return nil + } + + switch arg := arg.(type) { + case Encoder: + return arg.Encode(wbuf, oid) + case driver.Valuer: + v, err := arg.Value() + if err != nil { + return err + } + return Encode(wbuf, oid, v) + case string: + return encodeString(wbuf, oid, arg) + case []AclItem: + return encodeAclItemSlice(wbuf, oid, arg) + case []byte: + return encodeByteSlice(wbuf, oid, arg) + case [][]byte: + return encodeByteSliceSlice(wbuf, oid, arg) + } + + refVal := reflect.ValueOf(arg) + + if refVal.Kind() == reflect.Ptr { + if refVal.IsNil() { + wbuf.WriteInt32(-1) + return nil + } + arg = refVal.Elem().Interface() + return Encode(wbuf, oid, arg) + } + + if oid == JsonOid { + return encodeJSON(wbuf, oid, arg) + } + if oid == JsonbOid { + return encodeJSONB(wbuf, oid, arg) + } + + switch arg := arg.(type) { + case []string: + return encodeStringSlice(wbuf, oid, arg) + case bool: + return encodeBool(wbuf, oid, arg) + case []bool: + return encodeBoolSlice(wbuf, oid, arg) + case int: + return encodeInt(wbuf, oid, arg) + case uint: + return encodeUInt(wbuf, oid, arg) + case Char: + return encodeChar(wbuf, oid, arg) + case AclItem: + // The aclitem data type goes over the wire using the same format as string, + // so just cast to string and use encodeString + return encodeString(wbuf, oid, string(arg)) + case Name: + // The name data type goes over the wire using the same format as string, + // so just cast to string and use encodeString + return encodeString(wbuf, oid, string(arg)) + case int8: + return encodeInt8(wbuf, oid, arg) + case uint8: + return encodeUInt8(wbuf, oid, arg) + case int16: + return encodeInt16(wbuf, oid, arg) + case []int16: + return encodeInt16Slice(wbuf, oid, arg) + case uint16: + return encodeUInt16(wbuf, oid, arg) + case []uint16: + return encodeUInt16Slice(wbuf, oid, arg) + case int32: + return encodeInt32(wbuf, oid, arg) + case []int32: + return encodeInt32Slice(wbuf, oid, arg) + case uint32: + return encodeUInt32(wbuf, oid, arg) + case []uint32: + return encodeUInt32Slice(wbuf, oid, arg) + case int64: + return encodeInt64(wbuf, oid, arg) + case []int64: + return encodeInt64Slice(wbuf, oid, arg) + case uint64: + return encodeUInt64(wbuf, oid, arg) + case []uint64: + return encodeUInt64Slice(wbuf, oid, arg) + case float32: + return encodeFloat32(wbuf, oid, arg) + case []float32: + return encodeFloat32Slice(wbuf, oid, arg) + case float64: + return encodeFloat64(wbuf, oid, arg) + case []float64: + return encodeFloat64Slice(wbuf, oid, arg) + case time.Time: + return encodeTime(wbuf, oid, arg) + case []time.Time: + return encodeTimeSlice(wbuf, oid, arg) + case net.IP: + return encodeIP(wbuf, oid, arg) + case []net.IP: + return encodeIPSlice(wbuf, oid, arg) + case net.IPNet: + return encodeIPNet(wbuf, oid, arg) + case []net.IPNet: + return encodeIPNetSlice(wbuf, oid, arg) + case Oid: + return encodeOid(wbuf, oid, arg) + case Xid: + return encodeXid(wbuf, oid, arg) + case Cid: + return encodeCid(wbuf, oid, arg) + default: + if strippedArg, ok := stripNamedType(&refVal); ok { + return Encode(wbuf, oid, strippedArg) + } + return SerializationError(fmt.Sprintf("Cannot encode %T into oid %v - %T must implement Encoder or be converted to a string", arg, oid, arg)) + } +} + +func stripNamedType(val *reflect.Value) (interface{}, bool) { + switch val.Kind() { + case reflect.Int: + return int(val.Int()), true + case reflect.Int8: + return int8(val.Int()), true + case reflect.Int16: + return int16(val.Int()), true + case reflect.Int32: + return int32(val.Int()), true + case reflect.Int64: + return int64(val.Int()), true + case reflect.Uint: + return uint(val.Uint()), true + case reflect.Uint8: + return uint8(val.Uint()), true + case reflect.Uint16: + return uint16(val.Uint()), true + case reflect.Uint32: + return uint32(val.Uint()), true + case reflect.Uint64: + return uint64(val.Uint()), true + case reflect.String: + return val.String(), true + } + + return nil, false +} + +// Decode decodes from vr into d. d must be a pointer. This allows +// implementations of the Decoder interface to delegate the actual work of +// decoding to the built-in functionality. +func Decode(vr *ValueReader, d interface{}) error { + switch v := d.(type) { + case *bool: + *v = decodeBool(vr) + case *int: + n := decodeInt(vr) + if n < int64(minInt) { + return fmt.Errorf("%d is less than minimum value for int", n) + } else if n > int64(maxInt) { + return fmt.Errorf("%d is greater than maximum value for int", n) + } + *v = int(n) + case *int8: + n := decodeInt(vr) + if n < math.MinInt8 { + return fmt.Errorf("%d is less than minimum value for int8", n) + } else if n > math.MaxInt8 { + return fmt.Errorf("%d is greater than maximum value for int8", n) + } + *v = int8(n) + case *int16: + n := decodeInt(vr) + if n < math.MinInt16 { + return fmt.Errorf("%d is less than minimum value for int16", n) + } else if n > math.MaxInt16 { + return fmt.Errorf("%d is greater than maximum value for int16", n) + } + *v = int16(n) + case *int32: + n := decodeInt(vr) + if n < math.MinInt32 { + return fmt.Errorf("%d is less than minimum value for int32", n) + } else if n > math.MaxInt32 { + return fmt.Errorf("%d is greater than maximum value for int32", n) + } + *v = int32(n) + case *int64: + n := decodeInt(vr) + if n < math.MinInt64 { + return fmt.Errorf("%d is less than minimum value for int64", n) + } else if n > math.MaxInt64 { + return fmt.Errorf("%d is greater than maximum value for int64", n) + } + *v = int64(n) + case *uint: + n := decodeInt(vr) + if n < 0 { + return fmt.Errorf("%d is less than zero for uint8", n) + } else if maxInt == math.MaxInt32 && n > math.MaxUint32 { + return fmt.Errorf("%d is greater than maximum value for uint", n) + } + *v = uint(n) + case *uint8: + n := decodeInt(vr) + if n < 0 { + return fmt.Errorf("%d is less than zero for uint8", n) + } else if n > math.MaxUint8 { + return fmt.Errorf("%d is greater than maximum value for uint8", n) + } + *v = uint8(n) + case *uint16: + n := decodeInt(vr) + if n < 0 { + return fmt.Errorf("%d is less than zero for uint16", n) + } else if n > math.MaxUint16 { + return fmt.Errorf("%d is greater than maximum value for uint16", n) + } + *v = uint16(n) + case *uint32: + n := decodeInt(vr) + if n < 0 { + return fmt.Errorf("%d is less than zero for uint32", n) + } else if n > math.MaxUint32 { + return fmt.Errorf("%d is greater than maximum value for uint32", n) + } + *v = uint32(n) + case *uint64: + n := decodeInt(vr) + if n < 0 { + return fmt.Errorf("%d is less than zero for uint64", n) + } + *v = uint64(n) + case *Char: + *v = decodeChar(vr) + case *AclItem: + // aclitem goes over the wire just like text + *v = AclItem(decodeText(vr)) + case *Name: + // name goes over the wire just like text + *v = Name(decodeText(vr)) + case *Oid: + *v = decodeOid(vr) + case *Xid: + *v = decodeXid(vr) + case *Tid: + *v = decodeTid(vr) + case *Cid: + *v = decodeCid(vr) + case *string: + *v = decodeText(vr) + case *float32: + *v = decodeFloat4(vr) + case *float64: + *v = decodeFloat8(vr) + case *[]AclItem: + *v = decodeAclItemArray(vr) + case *[]bool: + *v = decodeBoolArray(vr) + case *[]int16: + *v = decodeInt2Array(vr) + case *[]uint16: + *v = decodeInt2ArrayToUInt(vr) + case *[]int32: + *v = decodeInt4Array(vr) + case *[]uint32: + *v = decodeInt4ArrayToUInt(vr) + case *[]int64: + *v = decodeInt8Array(vr) + case *[]uint64: + *v = decodeInt8ArrayToUInt(vr) + case *[]float32: + *v = decodeFloat4Array(vr) + case *[]float64: + *v = decodeFloat8Array(vr) + case *[]string: + *v = decodeTextArray(vr) + case *[]time.Time: + *v = decodeTimestampArray(vr) + case *[][]byte: + *v = decodeByteaArray(vr) + case *[]interface{}: + *v = decodeRecord(vr) + case *time.Time: + switch vr.Type().DataType { + case DateOid: + *v = decodeDate(vr) + case TimestampTzOid: + *v = decodeTimestampTz(vr) + case TimestampOid: + *v = decodeTimestamp(vr) + default: + return fmt.Errorf("Can't convert OID %v to time.Time", vr.Type().DataType) + } + case *net.IP: + ipnet := decodeInet(vr) + if oneCount, bitCount := ipnet.Mask.Size(); oneCount != bitCount { + return fmt.Errorf("Cannot decode netmask into *net.IP") + } + *v = ipnet.IP + case *[]net.IP: + ipnets := decodeInetArray(vr) + ips := make([]net.IP, len(ipnets)) + for i, ipnet := range ipnets { + if oneCount, bitCount := ipnet.Mask.Size(); oneCount != bitCount { + return fmt.Errorf("Cannot decode netmask into *net.IP") + } + ips[i] = ipnet.IP + } + *v = ips + case *net.IPNet: + *v = decodeInet(vr) + case *[]net.IPNet: + *v = decodeInetArray(vr) + default: + if v := reflect.ValueOf(d); v.Kind() == reflect.Ptr { + el := v.Elem() + switch el.Kind() { + // if d is a pointer to pointer, strip the pointer and try again + case reflect.Ptr: + // -1 is a null value + if vr.Len() == -1 { + if !el.IsNil() { + // if the destination pointer is not nil, nil it out + el.Set(reflect.Zero(el.Type())) + } + return nil + } + if el.IsNil() { + // allocate destination + el.Set(reflect.New(el.Type().Elem())) + } + d = el.Interface() + return Decode(vr, d) + case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: + n := decodeInt(vr) + if el.OverflowInt(n) { + return fmt.Errorf("Scan cannot decode %d into %T", n, d) + } + el.SetInt(n) + return nil + case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64: + n := decodeInt(vr) + if n < 0 { + return fmt.Errorf("%d is less than zero for %T", n, d) + } + if el.OverflowUint(uint64(n)) { + return fmt.Errorf("Scan cannot decode %d into %T", n, d) + } + el.SetUint(uint64(n)) + return nil + case reflect.String: + el.SetString(decodeText(vr)) + return nil + } + } + return fmt.Errorf("Scan cannot decode into %T", d) + } + + return nil +} + +func decodeBool(vr *ValueReader) bool { + if vr.Len() == -1 { + vr.Fatal(ProtocolError("Cannot decode null into bool")) + return false + } + + if vr.Type().DataType != BoolOid { + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode oid %v into bool", vr.Type().DataType))) + return false + } + + if vr.Type().FormatCode != BinaryFormatCode { + vr.Fatal(ProtocolError(fmt.Sprintf("Unknown field description format code: %v", vr.Type().FormatCode))) + return false + } + + if vr.Len() != 1 { + vr.Fatal(ProtocolError(fmt.Sprintf("Received an invalid size for an bool: %d", vr.Len()))) + return false + } + + b := vr.ReadByte() + return b != 0 +} + +func encodeBool(w *WriteBuf, oid Oid, value bool) error { + if oid != BoolOid { + return fmt.Errorf("cannot encode Go %s into oid %d", "bool", oid) + } + + w.WriteInt32(1) + + var n byte + if value { + n = 1 + } + + w.WriteByte(n) + + return nil +} + +func decodeInt(vr *ValueReader) int64 { + switch vr.Type().DataType { + case Int2Oid: + return int64(decodeInt2(vr)) + case Int4Oid: + return int64(decodeInt4(vr)) + case Int8Oid: + return int64(decodeInt8(vr)) + } + + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode oid %v into any integer type", vr.Type().DataType))) + return 0 +} + +func decodeInt8(vr *ValueReader) int64 { + if vr.Len() == -1 { + vr.Fatal(ProtocolError("Cannot decode null into int64")) + return 0 + } + + if vr.Type().DataType != Int8Oid { + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode oid %v into int8", vr.Type().DataType))) + return 0 + } + + if vr.Type().FormatCode != BinaryFormatCode { + vr.Fatal(ProtocolError(fmt.Sprintf("Unknown field description format code: %v", vr.Type().FormatCode))) + return 0 + } + + if vr.Len() != 8 { + vr.Fatal(ProtocolError(fmt.Sprintf("Received an invalid size for an int8: %d", vr.Len()))) + return 0 + } + + return vr.ReadInt64() +} + +func decodeChar(vr *ValueReader) Char { + if vr.Len() == -1 { + vr.Fatal(ProtocolError("Cannot decode null into char")) + return Char(0) + } + + if vr.Type().DataType != CharOid { + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode oid %v into char", vr.Type().DataType))) + return Char(0) + } + + if vr.Type().FormatCode != BinaryFormatCode { + vr.Fatal(ProtocolError(fmt.Sprintf("Unknown field description format code: %v", vr.Type().FormatCode))) + return Char(0) + } + + if vr.Len() != 1 { + vr.Fatal(ProtocolError(fmt.Sprintf("Received an invalid size for a char: %d", vr.Len()))) + return Char(0) + } + + return Char(vr.ReadByte()) +} + +func decodeInt2(vr *ValueReader) int16 { + if vr.Len() == -1 { + vr.Fatal(ProtocolError("Cannot decode null into int16")) + return 0 + } + + if vr.Type().DataType != Int2Oid { + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode oid %v into int16", vr.Type().DataType))) + return 0 + } + + if vr.Type().FormatCode != BinaryFormatCode { + vr.Fatal(ProtocolError(fmt.Sprintf("Unknown field description format code: %v", vr.Type().FormatCode))) + return 0 + } + + if vr.Len() != 2 { + vr.Fatal(ProtocolError(fmt.Sprintf("Received an invalid size for an int2: %d", vr.Len()))) + return 0 + } + + return vr.ReadInt16() +} + +func encodeInt(w *WriteBuf, oid Oid, value int) error { + switch oid { + case Int2Oid: + if value < math.MinInt16 { + return fmt.Errorf("%d is less than min pg:int2", value) + } else if value > math.MaxInt16 { + return fmt.Errorf("%d is greater than max pg:int2", value) + } + w.WriteInt32(2) + w.WriteInt16(int16(value)) + case Int4Oid: + if value < math.MinInt32 { + return fmt.Errorf("%d is less than min pg:int4", value) + } else if value > math.MaxInt32 { + return fmt.Errorf("%d is greater than max pg:int4", value) + } + w.WriteInt32(4) + w.WriteInt32(int32(value)) + case Int8Oid: + if int64(value) <= int64(math.MaxInt64) { + w.WriteInt32(8) + w.WriteInt64(int64(value)) + } else { + return fmt.Errorf("%d is larger than max int64 %d", value, int64(math.MaxInt64)) + } + default: + return fmt.Errorf("cannot encode %s into oid %v", "int8", oid) + } + + return nil +} + +func encodeUInt(w *WriteBuf, oid Oid, value uint) error { + switch oid { + case Int2Oid: + if value > math.MaxInt16 { + return fmt.Errorf("%d is greater than max pg:int2", value) + } + w.WriteInt32(2) + w.WriteInt16(int16(value)) + case Int4Oid: + if value > math.MaxInt32 { + return fmt.Errorf("%d is greater than max pg:int4", value) + } + w.WriteInt32(4) + w.WriteInt32(int32(value)) + case Int8Oid: + //****** Changed value to int64(value) and math.MaxInt64 to int64(math.MaxInt64) + if int64(value) > int64(math.MaxInt64) { + return fmt.Errorf("%d is greater than max pg:int8", value) + } + w.WriteInt32(8) + w.WriteInt64(int64(value)) + + default: + return fmt.Errorf("cannot encode %s into oid %v", "uint8", oid) + } + + return nil +} + +func encodeChar(w *WriteBuf, oid Oid, value Char) error { + w.WriteInt32(1) + w.WriteByte(byte(value)) + return nil +} + +func encodeInt8(w *WriteBuf, oid Oid, value int8) error { + switch oid { + case Int2Oid: + w.WriteInt32(2) + w.WriteInt16(int16(value)) + case Int4Oid: + w.WriteInt32(4) + w.WriteInt32(int32(value)) + case Int8Oid: + w.WriteInt32(8) + w.WriteInt64(int64(value)) + default: + return fmt.Errorf("cannot encode %s into oid %v", "int8", oid) + } + + return nil +} + +func encodeUInt8(w *WriteBuf, oid Oid, value uint8) error { + switch oid { + case Int2Oid: + w.WriteInt32(2) + w.WriteInt16(int16(value)) + case Int4Oid: + w.WriteInt32(4) + w.WriteInt32(int32(value)) + case Int8Oid: + w.WriteInt32(8) + w.WriteInt64(int64(value)) + default: + return fmt.Errorf("cannot encode %s into oid %v", "uint8", oid) + } + + return nil +} + +func encodeInt16(w *WriteBuf, oid Oid, value int16) error { + switch oid { + case Int2Oid: + w.WriteInt32(2) + w.WriteInt16(value) + case Int4Oid: + w.WriteInt32(4) + w.WriteInt32(int32(value)) + case Int8Oid: + w.WriteInt32(8) + w.WriteInt64(int64(value)) + default: + return fmt.Errorf("cannot encode %s into oid %v", "int16", oid) + } + + return nil +} + +func encodeUInt16(w *WriteBuf, oid Oid, value uint16) error { + switch oid { + case Int2Oid: + if value <= math.MaxInt16 { + w.WriteInt32(2) + w.WriteInt16(int16(value)) + } else { + return fmt.Errorf("%d is greater than max int16 %d", value, math.MaxInt16) + } + case Int4Oid: + w.WriteInt32(4) + w.WriteInt32(int32(value)) + case Int8Oid: + w.WriteInt32(8) + w.WriteInt64(int64(value)) + default: + return fmt.Errorf("cannot encode %s into oid %v", "int16", oid) + } + + return nil +} + +func encodeInt32(w *WriteBuf, oid Oid, value int32) error { + switch oid { + case Int2Oid: + if value <= math.MaxInt16 { + w.WriteInt32(2) + w.WriteInt16(int16(value)) + } else { + return fmt.Errorf("%d is greater than max int16 %d", value, math.MaxInt16) + } + case Int4Oid: + w.WriteInt32(4) + w.WriteInt32(value) + case Int8Oid: + w.WriteInt32(8) + w.WriteInt64(int64(value)) + default: + return fmt.Errorf("cannot encode %s into oid %v", "int32", oid) + } + + return nil +} + +func encodeUInt32(w *WriteBuf, oid Oid, value uint32) error { + switch oid { + case Int2Oid: + if value <= math.MaxInt16 { + w.WriteInt32(2) + w.WriteInt16(int16(value)) + } else { + return fmt.Errorf("%d is greater than max int16 %d", value, math.MaxInt16) + } + case Int4Oid: + if value <= math.MaxInt32 { + w.WriteInt32(4) + w.WriteInt32(int32(value)) + } else { + return fmt.Errorf("%d is greater than max int32 %d", value, math.MaxInt32) + } + case Int8Oid: + w.WriteInt32(8) + w.WriteInt64(int64(value)) + default: + return fmt.Errorf("cannot encode %s into oid %v", "uint32", oid) + } + + return nil +} + +func encodeInt64(w *WriteBuf, oid Oid, value int64) error { + switch oid { + case Int2Oid: + if value <= math.MaxInt16 { + w.WriteInt32(2) + w.WriteInt16(int16(value)) + } else { + return fmt.Errorf("%d is greater than max int16 %d", value, math.MaxInt16) + } + case Int4Oid: + if value <= math.MaxInt32 { + w.WriteInt32(4) + w.WriteInt32(int32(value)) + } else { + return fmt.Errorf("%d is greater than max int32 %d", value, math.MaxInt32) + } + case Int8Oid: + w.WriteInt32(8) + w.WriteInt64(value) + default: + return fmt.Errorf("cannot encode %s into oid %v", "int64", oid) + } + + return nil +} + +func encodeUInt64(w *WriteBuf, oid Oid, value uint64) error { + switch oid { + case Int2Oid: + if value <= math.MaxInt16 { + w.WriteInt32(2) + w.WriteInt16(int16(value)) + } else { + return fmt.Errorf("%d is greater than max int16 %d", value, math.MaxInt16) + } + case Int4Oid: + if value <= math.MaxInt32 { + w.WriteInt32(4) + w.WriteInt32(int32(value)) + } else { + return fmt.Errorf("%d is greater than max int32 %d", value, math.MaxInt32) + } + case Int8Oid: + + if value <= math.MaxInt64 { + w.WriteInt32(8) + w.WriteInt64(int64(value)) + } else { + return fmt.Errorf("%d is greater than max int64 %d", value, int64(math.MaxInt64)) + } + default: + return fmt.Errorf("cannot encode %s into oid %v", "uint64", oid) + } + + return nil +} + +func decodeInt4(vr *ValueReader) int32 { + if vr.Len() == -1 { + vr.Fatal(ProtocolError("Cannot decode null into int32")) + return 0 + } + + if vr.Type().DataType != Int4Oid { + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode oid %v into int32", vr.Type().DataType))) + return 0 + } + + if vr.Type().FormatCode != BinaryFormatCode { + vr.Fatal(ProtocolError(fmt.Sprintf("Unknown field description format code: %v", vr.Type().FormatCode))) + return 0 + } + + if vr.Len() != 4 { + vr.Fatal(ProtocolError(fmt.Sprintf("Received an invalid size for an int4: %d", vr.Len()))) + return 0 + } + + return vr.ReadInt32() +} + +func decodeOid(vr *ValueReader) Oid { + if vr.Len() == -1 { + vr.Fatal(ProtocolError("Cannot decode null into Oid")) + return Oid(0) + } + + if vr.Type().DataType != OidOid { + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode oid %v into pgx.Oid", vr.Type().DataType))) + return Oid(0) + } + + // Oid needs to decode text format because it is used in loadPgTypes + switch vr.Type().FormatCode { + case TextFormatCode: + s := vr.ReadString(vr.Len()) + n, err := strconv.ParseUint(s, 10, 32) + if err != nil { + vr.Fatal(ProtocolError(fmt.Sprintf("Received invalid Oid: %v", s))) + } + return Oid(n) + case BinaryFormatCode: + if vr.Len() != 4 { + vr.Fatal(ProtocolError(fmt.Sprintf("Received an invalid size for an Oid: %d", vr.Len()))) + return Oid(0) + } + return Oid(vr.ReadInt32()) + default: + vr.Fatal(ProtocolError(fmt.Sprintf("Unknown field description format code: %v", vr.Type().FormatCode))) + return Oid(0) + } +} + +func encodeOid(w *WriteBuf, oid Oid, value Oid) error { + if oid != OidOid { + return fmt.Errorf("cannot encode Go %s into oid %d", "pgx.Oid", oid) + } + + w.WriteInt32(4) + w.WriteUint32(uint32(value)) + + return nil +} + +func decodeXid(vr *ValueReader) Xid { + if vr.Len() == -1 { + vr.Fatal(ProtocolError("Cannot decode null into Xid")) + return Xid(0) + } + + if vr.Type().DataType != XidOid { + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode oid %v into pgx.Xid", vr.Type().DataType))) + return Xid(0) + } + + // Unlikely Xid will ever go over the wire as text format, but who knows? + switch vr.Type().FormatCode { + case TextFormatCode: + s := vr.ReadString(vr.Len()) + n, err := strconv.ParseUint(s, 10, 32) + if err != nil { + vr.Fatal(ProtocolError(fmt.Sprintf("Received invalid Oid: %v", s))) + } + return Xid(n) + case BinaryFormatCode: + if vr.Len() != 4 { + vr.Fatal(ProtocolError(fmt.Sprintf("Received an invalid size for an Oid: %d", vr.Len()))) + return Xid(0) + } + return Xid(vr.ReadUint32()) + default: + vr.Fatal(ProtocolError(fmt.Sprintf("Unknown field description format code: %v", vr.Type().FormatCode))) + return Xid(0) + } +} + +func encodeXid(w *WriteBuf, oid Oid, value Xid) error { + if oid != XidOid { + return fmt.Errorf("cannot encode Go %s into oid %d", "pgx.Xid", oid) + } + + w.WriteInt32(4) + w.WriteUint32(uint32(value)) + + return nil +} + +func decodeCid(vr *ValueReader) Cid { + if vr.Len() == -1 { + vr.Fatal(ProtocolError("Cannot decode null into Cid")) + return Cid(0) + } + + if vr.Type().DataType != CidOid { + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode oid %v into pgx.Cid", vr.Type().DataType))) + return Cid(0) + } + + // Unlikely Cid will ever go over the wire as text format, but who knows? + switch vr.Type().FormatCode { + case TextFormatCode: + s := vr.ReadString(vr.Len()) + n, err := strconv.ParseUint(s, 10, 32) + if err != nil { + vr.Fatal(ProtocolError(fmt.Sprintf("Received invalid Oid: %v", s))) + } + return Cid(n) + case BinaryFormatCode: + if vr.Len() != 4 { + vr.Fatal(ProtocolError(fmt.Sprintf("Received an invalid size for an Oid: %d", vr.Len()))) + return Cid(0) + } + return Cid(vr.ReadUint32()) + default: + vr.Fatal(ProtocolError(fmt.Sprintf("Unknown field description format code: %v", vr.Type().FormatCode))) + return Cid(0) + } +} + +func encodeCid(w *WriteBuf, oid Oid, value Cid) error { + if oid != CidOid { + return fmt.Errorf("cannot encode Go %s into oid %d", "pgx.Cid", oid) + } + + w.WriteInt32(4) + w.WriteUint32(uint32(value)) + + return nil +} + +// Note that we do not match negative numbers, because neither the +// BlockNumber nor OffsetNumber of a Tid can be negative. +var tidRegexp *regexp.Regexp = regexp.MustCompile(`^\((\d*),(\d*)\)$`) + +func decodeTid(vr *ValueReader) Tid { + if vr.Len() == -1 { + vr.Fatal(ProtocolError("Cannot decode null into Tid")) + return Tid{BlockNumber: 0, OffsetNumber: 0} + } + + if vr.Type().DataType != TidOid { + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode oid %v into pgx.Tid", vr.Type().DataType))) + return Tid{BlockNumber: 0, OffsetNumber: 0} + } + + // Unlikely Tid will ever go over the wire as text format, but who knows? + switch vr.Type().FormatCode { + case TextFormatCode: + s := vr.ReadString(vr.Len()) + + match := tidRegexp.FindStringSubmatch(s) + if match == nil { + vr.Fatal(ProtocolError(fmt.Sprintf("Received invalid Oid: %v", s))) + return Tid{BlockNumber: 0, OffsetNumber: 0} + } + + blockNumber, err := strconv.ParseUint(s, 10, 16) + if err != nil { + vr.Fatal(ProtocolError(fmt.Sprintf("Received invalid BlockNumber part of a Tid: %v", s))) + } + + offsetNumber, err := strconv.ParseUint(s, 10, 16) + if err != nil { + vr.Fatal(ProtocolError(fmt.Sprintf("Received invalid offsetNumber part of a Tid: %v", s))) + } + return Tid{BlockNumber: uint32(blockNumber), OffsetNumber: uint16(offsetNumber)} + case BinaryFormatCode: + if vr.Len() != 6 { + vr.Fatal(ProtocolError(fmt.Sprintf("Received an invalid size for an Oid: %d", vr.Len()))) + return Tid{BlockNumber: 0, OffsetNumber: 0} + } + return Tid{BlockNumber: vr.ReadUint32(), OffsetNumber: vr.ReadUint16()} + default: + vr.Fatal(ProtocolError(fmt.Sprintf("Unknown field description format code: %v", vr.Type().FormatCode))) + return Tid{BlockNumber: 0, OffsetNumber: 0} + } +} + +func encodeTid(w *WriteBuf, oid Oid, value Tid) error { + if oid != TidOid { + return fmt.Errorf("cannot encode Go %s into oid %d", "pgx.Tid", oid) + } + + w.WriteInt32(6) + w.WriteUint32(value.BlockNumber) + w.WriteUint16(value.OffsetNumber) + + return nil +} + +func decodeFloat4(vr *ValueReader) float32 { + if vr.Len() == -1 { + vr.Fatal(ProtocolError("Cannot decode null into float32")) + return 0 + } + + if vr.Type().DataType != Float4Oid { + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode oid %v into float32", vr.Type().DataType))) + return 0 + } + + if vr.Type().FormatCode != BinaryFormatCode { + vr.Fatal(ProtocolError(fmt.Sprintf("Unknown field description format code: %v", vr.Type().FormatCode))) + return 0 + } + + if vr.Len() != 4 { + vr.Fatal(ProtocolError(fmt.Sprintf("Received an invalid size for an float4: %d", vr.Len()))) + return 0 + } + + i := vr.ReadInt32() + return math.Float32frombits(uint32(i)) +} + +func encodeFloat32(w *WriteBuf, oid Oid, value float32) error { + switch oid { + case Float4Oid: + w.WriteInt32(4) + w.WriteInt32(int32(math.Float32bits(value))) + case Float8Oid: + w.WriteInt32(8) + w.WriteInt64(int64(math.Float64bits(float64(value)))) + default: + return fmt.Errorf("cannot encode %s into oid %v", "float32", oid) + } + + return nil +} + +func decodeFloat8(vr *ValueReader) float64 { + if vr.Len() == -1 { + vr.Fatal(ProtocolError("Cannot decode null into float64")) + return 0 + } + + if vr.Type().DataType != Float8Oid { + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode oid %v into float64", vr.Type().DataType))) + return 0 + } + + if vr.Type().FormatCode != BinaryFormatCode { + vr.Fatal(ProtocolError(fmt.Sprintf("Unknown field description format code: %v", vr.Type().FormatCode))) + return 0 + } + + if vr.Len() != 8 { + vr.Fatal(ProtocolError(fmt.Sprintf("Received an invalid size for an float8: %d", vr.Len()))) + return 0 + } + + i := vr.ReadInt64() + return math.Float64frombits(uint64(i)) +} + +func encodeFloat64(w *WriteBuf, oid Oid, value float64) error { + switch oid { + case Float8Oid: + w.WriteInt32(8) + w.WriteInt64(int64(math.Float64bits(value))) + default: + return fmt.Errorf("cannot encode %s into oid %v", "float64", oid) + } + + return nil +} + +func decodeText(vr *ValueReader) string { + if vr.Len() == -1 { + vr.Fatal(ProtocolError("Cannot decode null into string")) + return "" + } + + return vr.ReadString(vr.Len()) +} + +func encodeString(w *WriteBuf, oid Oid, value string) error { + w.WriteInt32(int32(len(value))) + w.WriteBytes([]byte(value)) + return nil +} + +func decodeBytea(vr *ValueReader) []byte { + if vr.Len() == -1 { + return nil + } + + if vr.Type().DataType != ByteaOid { + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode oid %v into []byte", vr.Type().DataType))) + return nil + } + + if vr.Type().FormatCode != BinaryFormatCode { + vr.Fatal(ProtocolError(fmt.Sprintf("Unknown field description format code: %v", vr.Type().FormatCode))) + return nil + } + + return vr.ReadBytes(vr.Len()) +} + +func encodeByteSlice(w *WriteBuf, oid Oid, value []byte) error { + w.WriteInt32(int32(len(value))) + w.WriteBytes(value) + + return nil +} + +func decodeJSON(vr *ValueReader, d interface{}) error { + if vr.Len() == -1 { + return nil + } + + if vr.Type().DataType != JsonOid { + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode oid %v into json", vr.Type().DataType))) + } + + bytes := vr.ReadBytes(vr.Len()) + err := json.Unmarshal(bytes, d) + if err != nil { + vr.Fatal(err) + } + return err +} + +func encodeJSON(w *WriteBuf, oid Oid, value interface{}) error { + if oid != JsonOid { + return fmt.Errorf("cannot encode JSON into oid %v", oid) + } + + s, err := json.Marshal(value) + if err != nil { + return fmt.Errorf("Failed to encode json from type: %T", value) + } + + w.WriteInt32(int32(len(s))) + w.WriteBytes(s) + + return nil +} + +func decodeJSONB(vr *ValueReader, d interface{}) error { + if vr.Len() == -1 { + return nil + } + + if vr.Type().DataType != JsonbOid { + err := ProtocolError(fmt.Sprintf("Cannot decode oid %v into jsonb", vr.Type().DataType)) + vr.Fatal(err) + return err + } + + bytes := vr.ReadBytes(vr.Len()) + if vr.Type().FormatCode == BinaryFormatCode { + if bytes[0] != 1 { + err := ProtocolError(fmt.Sprintf("Unknown jsonb format byte: %x", bytes[0])) + vr.Fatal(err) + return err + } + bytes = bytes[1:] + } + + err := json.Unmarshal(bytes, d) + if err != nil { + vr.Fatal(err) + } + return err +} + +func encodeJSONB(w *WriteBuf, oid Oid, value interface{}) error { + if oid != JsonbOid { + return fmt.Errorf("cannot encode JSON into oid %v", oid) + } + + s, err := json.Marshal(value) + if err != nil { + return fmt.Errorf("Failed to encode json from type: %T", value) + } + + w.WriteInt32(int32(len(s) + 1)) + w.WriteByte(1) // JSONB format header + w.WriteBytes(s) + + return nil +} + +func decodeDate(vr *ValueReader) time.Time { + var zeroTime time.Time + + if vr.Len() == -1 { + vr.Fatal(ProtocolError("Cannot decode null into time.Time")) + return zeroTime + } + + if vr.Type().DataType != DateOid { + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode oid %v into time.Time", vr.Type().DataType))) + return zeroTime + } + + if vr.Type().FormatCode != BinaryFormatCode { + vr.Fatal(ProtocolError(fmt.Sprintf("Unknown field description format code: %v", vr.Type().FormatCode))) + return zeroTime + } + + if vr.Len() != 4 { + vr.Fatal(ProtocolError(fmt.Sprintf("Received an invalid size for an date: %d", vr.Len()))) + } + dayOffset := vr.ReadInt32() + return time.Date(2000, 1, int(1+dayOffset), 0, 0, 0, 0, time.Local) +} + +func encodeTime(w *WriteBuf, oid Oid, value time.Time) error { + switch oid { + case DateOid: + tUnix := time.Date(value.Year(), value.Month(), value.Day(), 0, 0, 0, 0, time.UTC).Unix() + dateEpoch := time.Date(2000, 1, 1, 0, 0, 0, 0, time.UTC).Unix() + + secSinceDateEpoch := tUnix - dateEpoch + daysSinceDateEpoch := secSinceDateEpoch / 86400 + + w.WriteInt32(4) + w.WriteInt32(int32(daysSinceDateEpoch)) + + return nil + case TimestampTzOid, TimestampOid: + microsecSinceUnixEpoch := value.Unix()*1000000 + int64(value.Nanosecond())/1000 + microsecSinceY2K := microsecSinceUnixEpoch - microsecFromUnixEpochToY2K + + w.WriteInt32(8) + w.WriteInt64(microsecSinceY2K) + + return nil + default: + return fmt.Errorf("cannot encode %s into oid %v", "time.Time", oid) + } +} + +const microsecFromUnixEpochToY2K = 946684800 * 1000000 + +func decodeTimestampTz(vr *ValueReader) time.Time { + var zeroTime time.Time + + if vr.Len() == -1 { + vr.Fatal(ProtocolError("Cannot decode null into time.Time")) + return zeroTime + } + + if vr.Type().DataType != TimestampTzOid { + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode oid %v into time.Time", vr.Type().DataType))) + return zeroTime + } + + if vr.Type().FormatCode != BinaryFormatCode { + vr.Fatal(ProtocolError(fmt.Sprintf("Unknown field description format code: %v", vr.Type().FormatCode))) + return zeroTime + } + + if vr.Len() != 8 { + vr.Fatal(ProtocolError(fmt.Sprintf("Received an invalid size for an timestamptz: %d", vr.Len()))) + return zeroTime + } + + microsecSinceY2K := vr.ReadInt64() + microsecSinceUnixEpoch := microsecFromUnixEpochToY2K + microsecSinceY2K + return time.Unix(microsecSinceUnixEpoch/1000000, (microsecSinceUnixEpoch%1000000)*1000) +} + +func decodeTimestamp(vr *ValueReader) time.Time { + var zeroTime time.Time + + if vr.Len() == -1 { + vr.Fatal(ProtocolError("Cannot decode null into timestamp")) + return zeroTime + } + + if vr.Type().DataType != TimestampOid { + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode oid %v into time.Time", vr.Type().DataType))) + return zeroTime + } + + if vr.Type().FormatCode != BinaryFormatCode { + vr.Fatal(ProtocolError(fmt.Sprintf("Unknown field description format code: %v", vr.Type().FormatCode))) + return zeroTime + } + + if vr.Len() != 8 { + vr.Fatal(ProtocolError(fmt.Sprintf("Received an invalid size for an timestamp: %d", vr.Len()))) + return zeroTime + } + + microsecSinceY2K := vr.ReadInt64() + microsecSinceUnixEpoch := microsecFromUnixEpochToY2K + microsecSinceY2K + return time.Unix(microsecSinceUnixEpoch/1000000, (microsecSinceUnixEpoch%1000000)*1000) +} + +func decodeInet(vr *ValueReader) net.IPNet { + var zero net.IPNet + + if vr.Len() == -1 { + vr.Fatal(ProtocolError("Cannot decode null into net.IPNet")) + return zero + } + + if vr.Type().FormatCode != BinaryFormatCode { + vr.Fatal(ProtocolError(fmt.Sprintf("Unknown field description format code: %v", vr.Type().FormatCode))) + return zero + } + + pgType := vr.Type() + if pgType.DataType != InetOid && pgType.DataType != CidrOid { + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode oid %v into %s", pgType.DataType, pgType.Name))) + return zero + } + if vr.Len() != 8 && vr.Len() != 20 { + vr.Fatal(ProtocolError(fmt.Sprintf("Received an invalid size for a %s: %d", pgType.Name, vr.Len()))) + return zero + } + + vr.ReadByte() // ignore family + bits := vr.ReadByte() + vr.ReadByte() // ignore is_cidr + addressLength := vr.ReadByte() + + var ipnet net.IPNet + ipnet.IP = vr.ReadBytes(int32(addressLength)) + ipnet.Mask = net.CIDRMask(int(bits), int(addressLength)*8) + + return ipnet +} + +func encodeIPNet(w *WriteBuf, oid Oid, value net.IPNet) error { + if oid != InetOid && oid != CidrOid { + return fmt.Errorf("cannot encode %s into oid %v", "net.IPNet", oid) + } + + var size int32 + var family byte + switch len(value.IP) { + case net.IPv4len: + size = 8 + family = *w.conn.pgsqlAfInet + case net.IPv6len: + size = 20 + family = *w.conn.pgsqlAfInet6 + default: + return fmt.Errorf("Unexpected IP length: %v", len(value.IP)) + } + + w.WriteInt32(size) + w.WriteByte(family) + ones, _ := value.Mask.Size() + w.WriteByte(byte(ones)) + w.WriteByte(0) // is_cidr is ignored on server + w.WriteByte(byte(len(value.IP))) + w.WriteBytes(value.IP) + + return nil +} + +func encodeIP(w *WriteBuf, oid Oid, value net.IP) error { + if oid != InetOid && oid != CidrOid { + return fmt.Errorf("cannot encode %s into oid %v", "net.IP", oid) + } + + var ipnet net.IPNet + ipnet.IP = value + bitCount := len(value) * 8 + ipnet.Mask = net.CIDRMask(bitCount, bitCount) + return encodeIPNet(w, oid, ipnet) +} + +func decodeRecord(vr *ValueReader) []interface{} { + if vr.Len() == -1 { + return nil + } + + if vr.Type().FormatCode != BinaryFormatCode { + vr.Fatal(ProtocolError(fmt.Sprintf("Unknown field description format code: %v", vr.Type().FormatCode))) + return nil + } + + if vr.Type().DataType != RecordOid { + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode oid %v into []interface{}", vr.Type().DataType))) + return nil + } + + valueCount := vr.ReadInt32() + record := make([]interface{}, 0, int(valueCount)) + + for i := int32(0); i < valueCount; i++ { + fd := FieldDescription{FormatCode: BinaryFormatCode} + fieldVR := ValueReader{mr: vr.mr, fd: &fd} + fd.DataType = vr.ReadOid() + fieldVR.valueBytesRemaining = vr.ReadInt32() + vr.valueBytesRemaining -= fieldVR.valueBytesRemaining + + switch fd.DataType { + case BoolOid: + record = append(record, decodeBool(&fieldVR)) + case ByteaOid: + record = append(record, decodeBytea(&fieldVR)) + case Int8Oid: + record = append(record, decodeInt8(&fieldVR)) + case Int2Oid: + record = append(record, decodeInt2(&fieldVR)) + case Int4Oid: + record = append(record, decodeInt4(&fieldVR)) + case OidOid: + record = append(record, decodeOid(&fieldVR)) + case Float4Oid: + record = append(record, decodeFloat4(&fieldVR)) + case Float8Oid: + record = append(record, decodeFloat8(&fieldVR)) + case DateOid: + record = append(record, decodeDate(&fieldVR)) + case TimestampTzOid: + record = append(record, decodeTimestampTz(&fieldVR)) + case TimestampOid: + record = append(record, decodeTimestamp(&fieldVR)) + case InetOid, CidrOid: + record = append(record, decodeInet(&fieldVR)) + case TextOid, VarcharOid, UnknownOid: + record = append(record, decodeText(&fieldVR)) + default: + vr.Fatal(fmt.Errorf("decodeRecord cannot decode oid %d", fd.DataType)) + return nil + } + + // Consume any remaining data + if fieldVR.Len() > 0 { + fieldVR.ReadBytes(fieldVR.Len()) + } + + if fieldVR.Err() != nil { + vr.Fatal(fieldVR.Err()) + return nil + } + } + + return record +} + +func decode1dArrayHeader(vr *ValueReader) (length int32, err error) { + numDims := vr.ReadInt32() + if numDims > 1 { + return 0, ProtocolError(fmt.Sprintf("Expected array to have 0 or 1 dimension, but it had %v", numDims)) + } + + vr.ReadInt32() // 0 if no nulls / 1 if there is one or more nulls -- but we don't care + vr.ReadInt32() // element oid + + if numDims == 0 { + return 0, nil + } + + length = vr.ReadInt32() + + idxFirstElem := vr.ReadInt32() + if idxFirstElem != 1 { + return 0, ProtocolError(fmt.Sprintf("Expected array's first element to start a index 1, but it is %d", idxFirstElem)) + } + + return length, nil +} + +func decodeBoolArray(vr *ValueReader) []bool { + if vr.Len() == -1 { + return nil + } + + if vr.Type().DataType != BoolArrayOid { + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode oid %v into []bool", vr.Type().DataType))) + return nil + } + + if vr.Type().FormatCode != BinaryFormatCode { + vr.Fatal(ProtocolError(fmt.Sprintf("Unknown field description format code: %v", vr.Type().FormatCode))) + return nil + } + + numElems, err := decode1dArrayHeader(vr) + if err != nil { + vr.Fatal(err) + return nil + } + + a := make([]bool, int(numElems)) + for i := 0; i < len(a); i++ { + elSize := vr.ReadInt32() + switch elSize { + case 1: + if vr.ReadByte() == 1 { + a[i] = true + } + case -1: + vr.Fatal(ProtocolError("Cannot decode null element")) + return nil + default: + vr.Fatal(ProtocolError(fmt.Sprintf("Received an invalid size for an bool element: %d", elSize))) + return nil + } + } + + return a +} + +func encodeBoolSlice(w *WriteBuf, oid Oid, slice []bool) error { + if oid != BoolArrayOid { + return fmt.Errorf("cannot encode Go %s into oid %d", "[]bool", oid) + } + + encodeArrayHeader(w, BoolOid, len(slice), 5) + for _, v := range slice { + w.WriteInt32(1) + var b byte + if v { + b = 1 + } + w.WriteByte(b) + } + + return nil +} + +func decodeByteaArray(vr *ValueReader) [][]byte { + if vr.Len() == -1 { + return nil + } + + if vr.Type().DataType != ByteaArrayOid { + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode oid %v into [][]byte", vr.Type().DataType))) + return nil + } + + if vr.Type().FormatCode != BinaryFormatCode { + vr.Fatal(ProtocolError(fmt.Sprintf("Unknown field description format code: %v", vr.Type().FormatCode))) + return nil + } + + numElems, err := decode1dArrayHeader(vr) + if err != nil { + vr.Fatal(err) + return nil + } + + a := make([][]byte, int(numElems)) + for i := 0; i < len(a); i++ { + elSize := vr.ReadInt32() + switch elSize { + case -1: + vr.Fatal(ProtocolError("Cannot decode null element")) + return nil + default: + a[i] = vr.ReadBytes(elSize) + } + } + + return a +} + +func encodeByteSliceSlice(w *WriteBuf, oid Oid, value [][]byte) error { + if oid != ByteaArrayOid { + return fmt.Errorf("cannot encode Go %s into oid %d", "[][]byte", oid) + } + + size := 20 // array header size + for _, el := range value { + size += 4 + len(el) + } + + w.WriteInt32(int32(size)) + + w.WriteInt32(1) // number of dimensions + w.WriteInt32(0) // no nulls + w.WriteInt32(int32(ByteaOid)) // type of elements + w.WriteInt32(int32(len(value))) // number of elements + w.WriteInt32(1) // index of first element + + for _, el := range value { + encodeByteSlice(w, ByteaOid, el) + } + + return nil +} + +func decodeInt2Array(vr *ValueReader) []int16 { + if vr.Len() == -1 { + return nil + } + + if vr.Type().DataType != Int2ArrayOid { + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode oid %v into []int16", vr.Type().DataType))) + return nil + } + + if vr.Type().FormatCode != BinaryFormatCode { + vr.Fatal(ProtocolError(fmt.Sprintf("Unknown field description format code: %v", vr.Type().FormatCode))) + return nil + } + + numElems, err := decode1dArrayHeader(vr) + if err != nil { + vr.Fatal(err) + return nil + } + + a := make([]int16, int(numElems)) + for i := 0; i < len(a); i++ { + elSize := vr.ReadInt32() + switch elSize { + case 2: + a[i] = vr.ReadInt16() + case -1: + vr.Fatal(ProtocolError("Cannot decode null element")) + return nil + default: + vr.Fatal(ProtocolError(fmt.Sprintf("Received an invalid size for an int2 element: %d", elSize))) + return nil + } + } + + return a +} + +func decodeInt2ArrayToUInt(vr *ValueReader) []uint16 { + if vr.Len() == -1 { + return nil + } + + if vr.Type().DataType != Int2ArrayOid { + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode oid %v into []uint16", vr.Type().DataType))) + return nil + } + + if vr.Type().FormatCode != BinaryFormatCode { + vr.Fatal(ProtocolError(fmt.Sprintf("Unknown field description format code: %v", vr.Type().FormatCode))) + return nil + } + + numElems, err := decode1dArrayHeader(vr) + if err != nil { + vr.Fatal(err) + return nil + } + + a := make([]uint16, int(numElems)) + for i := 0; i < len(a); i++ { + elSize := vr.ReadInt32() + switch elSize { + case 2: + tmp := vr.ReadInt16() + if tmp < 0 { + vr.Fatal(ProtocolError(fmt.Sprintf("%d is less than zero for uint16", tmp))) + return nil + } + a[i] = uint16(tmp) + case -1: + vr.Fatal(ProtocolError("Cannot decode null element")) + return nil + default: + vr.Fatal(ProtocolError(fmt.Sprintf("Received an invalid size for an int2 element: %d", elSize))) + return nil + } + } + + return a +} + +func encodeInt16Slice(w *WriteBuf, oid Oid, slice []int16) error { + if oid != Int2ArrayOid { + return fmt.Errorf("cannot encode Go %s into oid %d", "[]int16", oid) + } + + encodeArrayHeader(w, Int2Oid, len(slice), 6) + for _, v := range slice { + w.WriteInt32(2) + w.WriteInt16(v) + } + + return nil +} + +func encodeUInt16Slice(w *WriteBuf, oid Oid, slice []uint16) error { + if oid != Int2ArrayOid { + return fmt.Errorf("cannot encode Go %s into oid %d", "[]uint16", oid) + } + + encodeArrayHeader(w, Int2Oid, len(slice), 6) + for _, v := range slice { + if v <= math.MaxInt16 { + w.WriteInt32(2) + w.WriteInt16(int16(v)) + } else { + return fmt.Errorf("%d is greater than max smallint %d", v, math.MaxInt16) + } + } + + return nil +} + +func decodeInt4Array(vr *ValueReader) []int32 { + if vr.Len() == -1 { + return nil + } + + if vr.Type().DataType != Int4ArrayOid { + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode oid %v into []int32", vr.Type().DataType))) + return nil + } + + if vr.Type().FormatCode != BinaryFormatCode { + vr.Fatal(ProtocolError(fmt.Sprintf("Unknown field description format code: %v", vr.Type().FormatCode))) + return nil + } + + numElems, err := decode1dArrayHeader(vr) + if err != nil { + vr.Fatal(err) + return nil + } + + a := make([]int32, int(numElems)) + for i := 0; i < len(a); i++ { + elSize := vr.ReadInt32() + switch elSize { + case 4: + a[i] = vr.ReadInt32() + case -1: + vr.Fatal(ProtocolError("Cannot decode null element")) + return nil + default: + vr.Fatal(ProtocolError(fmt.Sprintf("Received an invalid size for an int4 element: %d", elSize))) + return nil + } + } + + return a +} + +func decodeInt4ArrayToUInt(vr *ValueReader) []uint32 { + if vr.Len() == -1 { + return nil + } + + if vr.Type().DataType != Int4ArrayOid { + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode oid %v into []uint32", vr.Type().DataType))) + return nil + } + + if vr.Type().FormatCode != BinaryFormatCode { + vr.Fatal(ProtocolError(fmt.Sprintf("Unknown field description format code: %v", vr.Type().FormatCode))) + return nil + } + + numElems, err := decode1dArrayHeader(vr) + if err != nil { + vr.Fatal(err) + return nil + } + + a := make([]uint32, int(numElems)) + for i := 0; i < len(a); i++ { + elSize := vr.ReadInt32() + switch elSize { + case 4: + tmp := vr.ReadInt32() + if tmp < 0 { + vr.Fatal(ProtocolError(fmt.Sprintf("%d is less than zero for uint32", tmp))) + return nil + } + a[i] = uint32(tmp) + case -1: + vr.Fatal(ProtocolError("Cannot decode null element")) + return nil + default: + vr.Fatal(ProtocolError(fmt.Sprintf("Received an invalid size for an int4 element: %d", elSize))) + return nil + } + } + + return a +} + +func encodeInt32Slice(w *WriteBuf, oid Oid, slice []int32) error { + if oid != Int4ArrayOid { + return fmt.Errorf("cannot encode Go %s into oid %d", "[]int32", oid) + } + + encodeArrayHeader(w, Int4Oid, len(slice), 8) + for _, v := range slice { + w.WriteInt32(4) + w.WriteInt32(v) + } + + return nil +} + +func encodeUInt32Slice(w *WriteBuf, oid Oid, slice []uint32) error { + if oid != Int4ArrayOid { + return fmt.Errorf("cannot encode Go %s into oid %d", "[]uint32", oid) + } + + encodeArrayHeader(w, Int4Oid, len(slice), 8) + for _, v := range slice { + if v <= math.MaxInt32 { + w.WriteInt32(4) + w.WriteInt32(int32(v)) + } else { + return fmt.Errorf("%d is greater than max integer %d", v, math.MaxInt32) + } + } + + return nil +} + +func decodeInt8Array(vr *ValueReader) []int64 { + if vr.Len() == -1 { + return nil + } + + if vr.Type().DataType != Int8ArrayOid { + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode oid %v into []int64", vr.Type().DataType))) + return nil + } + + if vr.Type().FormatCode != BinaryFormatCode { + vr.Fatal(ProtocolError(fmt.Sprintf("Unknown field description format code: %v", vr.Type().FormatCode))) + return nil + } + + numElems, err := decode1dArrayHeader(vr) + if err != nil { + vr.Fatal(err) + return nil + } + + a := make([]int64, int(numElems)) + for i := 0; i < len(a); i++ { + elSize := vr.ReadInt32() + switch elSize { + case 8: + a[i] = vr.ReadInt64() + case -1: + vr.Fatal(ProtocolError("Cannot decode null element")) + return nil + default: + vr.Fatal(ProtocolError(fmt.Sprintf("Received an invalid size for an int8 element: %d", elSize))) + return nil + } + } + + return a +} + +func decodeInt8ArrayToUInt(vr *ValueReader) []uint64 { + if vr.Len() == -1 { + return nil + } + + if vr.Type().DataType != Int8ArrayOid { + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode oid %v into []uint64", vr.Type().DataType))) + return nil + } + + if vr.Type().FormatCode != BinaryFormatCode { + vr.Fatal(ProtocolError(fmt.Sprintf("Unknown field description format code: %v", vr.Type().FormatCode))) + return nil + } + + numElems, err := decode1dArrayHeader(vr) + if err != nil { + vr.Fatal(err) + return nil + } + + a := make([]uint64, int(numElems)) + for i := 0; i < len(a); i++ { + elSize := vr.ReadInt32() + switch elSize { + case 8: + tmp := vr.ReadInt64() + if tmp < 0 { + vr.Fatal(ProtocolError(fmt.Sprintf("%d is less than zero for uint64", tmp))) + return nil + } + a[i] = uint64(tmp) + case -1: + vr.Fatal(ProtocolError("Cannot decode null element")) + return nil + default: + vr.Fatal(ProtocolError(fmt.Sprintf("Received an invalid size for an int8 element: %d", elSize))) + return nil + } + } + + return a +} + +func encodeInt64Slice(w *WriteBuf, oid Oid, slice []int64) error { + if oid != Int8ArrayOid { + return fmt.Errorf("cannot encode Go %s into oid %d", "[]int64", oid) + } + + encodeArrayHeader(w, Int8Oid, len(slice), 12) + for _, v := range slice { + w.WriteInt32(8) + w.WriteInt64(v) + } + + return nil +} + +func encodeUInt64Slice(w *WriteBuf, oid Oid, slice []uint64) error { + if oid != Int8ArrayOid { + return fmt.Errorf("cannot encode Go %s into oid %d", "[]uint64", oid) + } + + encodeArrayHeader(w, Int8Oid, len(slice), 12) + for _, v := range slice { + if v <= math.MaxInt64 { + w.WriteInt32(8) + w.WriteInt64(int64(v)) + } else { + return fmt.Errorf("%d is greater than max bigint %d", v, int64(math.MaxInt64)) + } + } + + return nil +} + +func decodeFloat4Array(vr *ValueReader) []float32 { + if vr.Len() == -1 { + return nil + } + + if vr.Type().DataType != Float4ArrayOid { + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode oid %v into []float32", vr.Type().DataType))) + return nil + } + + if vr.Type().FormatCode != BinaryFormatCode { + vr.Fatal(ProtocolError(fmt.Sprintf("Unknown field description format code: %v", vr.Type().FormatCode))) + return nil + } + + numElems, err := decode1dArrayHeader(vr) + if err != nil { + vr.Fatal(err) + return nil + } + + a := make([]float32, int(numElems)) + for i := 0; i < len(a); i++ { + elSize := vr.ReadInt32() + switch elSize { + case 4: + n := vr.ReadInt32() + a[i] = math.Float32frombits(uint32(n)) + case -1: + vr.Fatal(ProtocolError("Cannot decode null element")) + return nil + default: + vr.Fatal(ProtocolError(fmt.Sprintf("Received an invalid size for an float4 element: %d", elSize))) + return nil + } + } + + return a +} + +func encodeFloat32Slice(w *WriteBuf, oid Oid, slice []float32) error { + if oid != Float4ArrayOid { + return fmt.Errorf("cannot encode Go %s into oid %d", "[]float32", oid) + } + + encodeArrayHeader(w, Float4Oid, len(slice), 8) + for _, v := range slice { + w.WriteInt32(4) + w.WriteInt32(int32(math.Float32bits(v))) + } + + return nil +} + +func decodeFloat8Array(vr *ValueReader) []float64 { + if vr.Len() == -1 { + return nil + } + + if vr.Type().DataType != Float8ArrayOid { + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode oid %v into []float64", vr.Type().DataType))) + return nil + } + + if vr.Type().FormatCode != BinaryFormatCode { + vr.Fatal(ProtocolError(fmt.Sprintf("Unknown field description format code: %v", vr.Type().FormatCode))) + return nil + } + + numElems, err := decode1dArrayHeader(vr) + if err != nil { + vr.Fatal(err) + return nil + } + + a := make([]float64, int(numElems)) + for i := 0; i < len(a); i++ { + elSize := vr.ReadInt32() + switch elSize { + case 8: + n := vr.ReadInt64() + a[i] = math.Float64frombits(uint64(n)) + case -1: + vr.Fatal(ProtocolError("Cannot decode null element")) + return nil + default: + vr.Fatal(ProtocolError(fmt.Sprintf("Received an invalid size for an float4 element: %d", elSize))) + return nil + } + } + + return a +} + +func encodeFloat64Slice(w *WriteBuf, oid Oid, slice []float64) error { + if oid != Float8ArrayOid { + return fmt.Errorf("cannot encode Go %s into oid %d", "[]float64", oid) + } + + encodeArrayHeader(w, Float8Oid, len(slice), 12) + for _, v := range slice { + w.WriteInt32(8) + w.WriteInt64(int64(math.Float64bits(v))) + } + + return nil +} + +func decodeTextArray(vr *ValueReader) []string { + if vr.Len() == -1 { + return nil + } + + if vr.Type().DataType != TextArrayOid && vr.Type().DataType != VarcharArrayOid { + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode oid %v into []string", vr.Type().DataType))) + return nil + } + + if vr.Type().FormatCode != BinaryFormatCode { + vr.Fatal(ProtocolError(fmt.Sprintf("Unknown field description format code: %v", vr.Type().FormatCode))) + return nil + } + + numElems, err := decode1dArrayHeader(vr) + if err != nil { + vr.Fatal(err) + return nil + } + + a := make([]string, int(numElems)) + for i := 0; i < len(a); i++ { + elSize := vr.ReadInt32() + if elSize == -1 { + vr.Fatal(ProtocolError("Cannot decode null element")) + return nil + } + + a[i] = vr.ReadString(elSize) + } + + return a +} + +// escapeAclItem escapes an AclItem before it is added to +// its aclitem[] string representation. The PostgreSQL aclitem +// datatype itself can need escapes because it follows the +// formatting rules of SQL identifiers. Think of this function +// as escaping the escapes, so that PostgreSQL's array parser +// will do the right thing. +func escapeAclItem(acl string) (string, error) { + var escapedAclItem bytes.Buffer + reader := strings.NewReader(acl) + for { + rn, _, err := reader.ReadRune() + if err != nil { + if err == io.EOF { + // Here, EOF is an expected end state, not an error. + return escapedAclItem.String(), nil + } + // This error was not expected + return "", err + } + if needsEscape(rn) { + escapedAclItem.WriteRune('\\') + } + escapedAclItem.WriteRune(rn) + } +} + +// needsEscape determines whether or not a rune needs escaping +// before being placed in the textual representation of an +// aclitem[] array. +func needsEscape(rn rune) bool { + return rn == '\\' || rn == ',' || rn == '"' || rn == '}' +} + +// encodeAclItemSlice encodes a slice of AclItems in +// their textual represention for PostgreSQL. +func encodeAclItemSlice(w *WriteBuf, oid Oid, aclitems []AclItem) error { + strs := make([]string, len(aclitems)) + var escapedAclItem string + var err error + for i := range strs { + escapedAclItem, err = escapeAclItem(string(aclitems[i])) + if err != nil { + return err + } + strs[i] = string(escapedAclItem) + } + + var buf bytes.Buffer + buf.WriteRune('{') + buf.WriteString(strings.Join(strs, ",")) + buf.WriteRune('}') + str := buf.String() + w.WriteInt32(int32(len(str))) + w.WriteBytes([]byte(str)) + return nil +} + +// parseAclItemArray parses the textual representation +// of the aclitem[] type. The textual representation is chosen because +// Pg's src/backend/utils/adt/acl.c has only in/out (text) not send/recv (bin). +// See https://www.postgresql.org/docs/current/static/arrays.html#ARRAYS-IO +// for formatting notes. +func parseAclItemArray(arr string) ([]AclItem, error) { + reader := strings.NewReader(arr) + // Difficult to guess a performant initial capacity for a slice of + // aclitems, but let's go with 5. + aclItems := make([]AclItem, 0, 5) + // A single value + aclItem := AclItem("") + for { + // Grab the first/next/last rune to see if we are dealing with a + // quoted value, an unquoted value, or the end of the string. + rn, _, err := reader.ReadRune() + if err != nil { + if err == io.EOF { + // Here, EOF is an expected end state, not an error. + return aclItems, nil + } + // This error was not expected + return nil, err + } + + if rn == '"' { + // Discard the opening quote of the quoted value. + aclItem, err = parseQuotedAclItem(reader) + } else { + // We have just read the first rune of an unquoted (bare) value; + // put it back so that ParseBareValue can read it. + err := reader.UnreadRune() + if err != nil { + return nil, err + } + aclItem, err = parseBareAclItem(reader) + } + + if err != nil { + if err == io.EOF { + // Here, EOF is an expected end state, not an error.. + aclItems = append(aclItems, aclItem) + return aclItems, nil + } + // This error was not expected. + return nil, err + } + aclItems = append(aclItems, aclItem) + } +} + +// parseBareAclItem parses a bare (unquoted) aclitem from reader +func parseBareAclItem(reader *strings.Reader) (AclItem, error) { + var aclItem bytes.Buffer + for { + rn, _, err := reader.ReadRune() + if err != nil { + // Return the read value in case the error is a harmless io.EOF. + // (io.EOF marks the end of a bare aclitem at the end of a string) + return AclItem(aclItem.String()), err + } + if rn == ',' { + // A comma marks the end of a bare aclitem. + return AclItem(aclItem.String()), nil + } else { + aclItem.WriteRune(rn) + } + } +} + +// parseQuotedAclItem parses an aclitem which is in double quotes from reader +func parseQuotedAclItem(reader *strings.Reader) (AclItem, error) { + var aclItem bytes.Buffer + for { + rn, escaped, err := readPossiblyEscapedRune(reader) + if err != nil { + if err == io.EOF { + // Even when it is the last value, the final rune of + // a quoted aclitem should be the final closing quote, not io.EOF. + return AclItem(""), fmt.Errorf("unexpected end of quoted value") + } + // Return the read aclitem in case the error is a harmless io.EOF, + // which will be determined by the caller. + return AclItem(aclItem.String()), err + } + if !escaped && rn == '"' { + // An unescaped double quote marks the end of a quoted value. + // The next rune should either be a comma or the end of the string. + rn, _, err := reader.ReadRune() + if err != nil { + // Return the read value in case the error is a harmless io.EOF, + // which will be determined by the caller. + return AclItem(aclItem.String()), err + } + if rn != ',' { + return AclItem(""), fmt.Errorf("unexpected rune after quoted value") + } + return AclItem(aclItem.String()), nil + } + aclItem.WriteRune(rn) + } +} + +// Returns the next rune from r, unless it is a backslash; +// in that case, it returns the rune after the backslash. The second +// return value tells us whether or not the rune was +// preceeded by a backslash (escaped). +func readPossiblyEscapedRune(reader *strings.Reader) (rune, bool, error) { + rn, _, err := reader.ReadRune() + if err != nil { + return 0, false, err + } + if rn == '\\' { + // Discard the backslash and read the next rune. + rn, _, err = reader.ReadRune() + if err != nil { + return 0, false, err + } + return rn, true, nil + } + return rn, false, nil +} + +func decodeAclItemArray(vr *ValueReader) []AclItem { + if vr.Len() == -1 { + vr.Fatal(ProtocolError("Cannot decode null into []AclItem")) + return nil + } + + str := vr.ReadString(vr.Len()) + + // Short-circuit empty array. + if str == "{}" { + return []AclItem{} + } + + // Remove the '{' at the front and the '}' at the end, + // so that parseAclItemArray doesn't have to deal with them. + str = str[1 : len(str)-1] + aclItems, err := parseAclItemArray(str) + if err != nil { + vr.Fatal(ProtocolError(err.Error())) + return nil + } + return aclItems +} + +func encodeStringSlice(w *WriteBuf, oid Oid, slice []string) error { + var elOid Oid + switch oid { + case VarcharArrayOid: + elOid = VarcharOid + case TextArrayOid: + elOid = TextOid + default: + return fmt.Errorf("cannot encode Go %s into oid %d", "[]string", oid) + } + + var totalStringSize int + for _, v := range slice { + totalStringSize += len(v) + } + + size := 20 + len(slice)*4 + totalStringSize + w.WriteInt32(int32(size)) + + w.WriteInt32(1) // number of dimensions + w.WriteInt32(0) // no nulls + w.WriteInt32(int32(elOid)) // type of elements + w.WriteInt32(int32(len(slice))) // number of elements + w.WriteInt32(1) // index of first element + + for _, v := range slice { + w.WriteInt32(int32(len(v))) + w.WriteBytes([]byte(v)) + } + + return nil +} + +func decodeTimestampArray(vr *ValueReader) []time.Time { + if vr.Len() == -1 { + return nil + } + + if vr.Type().DataType != TimestampArrayOid && vr.Type().DataType != TimestampTzArrayOid { + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode oid %v into []time.Time", vr.Type().DataType))) + return nil + } + + if vr.Type().FormatCode != BinaryFormatCode { + vr.Fatal(ProtocolError(fmt.Sprintf("Unknown field description format code: %v", vr.Type().FormatCode))) + return nil + } + + numElems, err := decode1dArrayHeader(vr) + if err != nil { + vr.Fatal(err) + return nil + } + + a := make([]time.Time, int(numElems)) + for i := 0; i < len(a); i++ { + elSize := vr.ReadInt32() + switch elSize { + case 8: + microsecSinceY2K := vr.ReadInt64() + microsecSinceUnixEpoch := microsecFromUnixEpochToY2K + microsecSinceY2K + a[i] = time.Unix(microsecSinceUnixEpoch/1000000, (microsecSinceUnixEpoch%1000000)*1000) + case -1: + vr.Fatal(ProtocolError("Cannot decode null element")) + return nil + default: + vr.Fatal(ProtocolError(fmt.Sprintf("Received an invalid size for an time.Time element: %d", elSize))) + return nil + } + } + + return a +} + +func encodeTimeSlice(w *WriteBuf, oid Oid, slice []time.Time) error { + var elOid Oid + switch oid { + case TimestampArrayOid: + elOid = TimestampOid + case TimestampTzArrayOid: + elOid = TimestampTzOid + default: + return fmt.Errorf("cannot encode Go %s into oid %d", "[]time.Time", oid) + } + + encodeArrayHeader(w, int(elOid), len(slice), 12) + for _, t := range slice { + w.WriteInt32(8) + microsecSinceUnixEpoch := t.Unix()*1000000 + int64(t.Nanosecond())/1000 + microsecSinceY2K := microsecSinceUnixEpoch - microsecFromUnixEpochToY2K + w.WriteInt64(microsecSinceY2K) + } + + return nil +} + +func decodeInetArray(vr *ValueReader) []net.IPNet { + if vr.Len() == -1 { + return nil + } + + if vr.Type().DataType != InetArrayOid && vr.Type().DataType != CidrArrayOid { + vr.Fatal(ProtocolError(fmt.Sprintf("Cannot decode oid %v into []net.IP", vr.Type().DataType))) + return nil + } + + if vr.Type().FormatCode != BinaryFormatCode { + vr.Fatal(ProtocolError(fmt.Sprintf("Unknown field description format code: %v", vr.Type().FormatCode))) + return nil + } + + numElems, err := decode1dArrayHeader(vr) + if err != nil { + vr.Fatal(err) + return nil + } + + a := make([]net.IPNet, int(numElems)) + for i := 0; i < len(a); i++ { + elSize := vr.ReadInt32() + if elSize == -1 { + vr.Fatal(ProtocolError("Cannot decode null element")) + return nil + } + + vr.ReadByte() // ignore family + bits := vr.ReadByte() + vr.ReadByte() // ignore is_cidr + addressLength := vr.ReadByte() + + var ipnet net.IPNet + ipnet.IP = vr.ReadBytes(int32(addressLength)) + ipnet.Mask = net.CIDRMask(int(bits), int(addressLength)*8) + + a[i] = ipnet + } + + return a +} + +func encodeIPNetSlice(w *WriteBuf, oid Oid, slice []net.IPNet) error { + var elOid Oid + switch oid { + case InetArrayOid: + elOid = InetOid + case CidrArrayOid: + elOid = CidrOid + default: + return fmt.Errorf("cannot encode Go %s into oid %d", "[]net.IPNet", oid) + } + + size := int32(20) // array header size + for _, ipnet := range slice { + size += 4 + 4 + int32(len(ipnet.IP)) // size of element + inet/cidr metadata + IP bytes + } + w.WriteInt32(int32(size)) + + w.WriteInt32(1) // number of dimensions + w.WriteInt32(0) // no nulls + w.WriteInt32(int32(elOid)) // type of elements + w.WriteInt32(int32(len(slice))) // number of elements + w.WriteInt32(1) // index of first element + + for _, ipnet := range slice { + encodeIPNet(w, elOid, ipnet) + } + + return nil +} + +func encodeIPSlice(w *WriteBuf, oid Oid, slice []net.IP) error { + var elOid Oid + switch oid { + case InetArrayOid: + elOid = InetOid + case CidrArrayOid: + elOid = CidrOid + default: + return fmt.Errorf("cannot encode Go %s into oid %d", "[]net.IPNet", oid) + } + + size := int32(20) // array header size + for _, ip := range slice { + size += 4 + 4 + int32(len(ip)) // size of element + inet/cidr metadata + IP bytes + } + w.WriteInt32(int32(size)) + + w.WriteInt32(1) // number of dimensions + w.WriteInt32(0) // no nulls + w.WriteInt32(int32(elOid)) // type of elements + w.WriteInt32(int32(len(slice))) // number of elements + w.WriteInt32(1) // index of first element + + for _, ip := range slice { + encodeIP(w, elOid, ip) + } + + return nil +} + +func encodeArrayHeader(w *WriteBuf, oid, length, sizePerItem int) { + w.WriteInt32(int32(20 + length*sizePerItem)) + w.WriteInt32(1) // number of dimensions + w.WriteInt32(0) // no nulls + w.WriteInt32(int32(oid)) // type of elements + w.WriteInt32(int32(length)) // number of elements + w.WriteInt32(1) // index of first element +} diff --git a/vendor/github.com/kennygrant/sanitize/.gitignore b/vendor/github.com/kennygrant/sanitize/.gitignore new file mode 100644 index 0000000..0026861 --- /dev/null +++ b/vendor/github.com/kennygrant/sanitize/.gitignore @@ -0,0 +1,22 @@ +# Compiled Object files, Static and Dynamic libs (Shared Objects) +*.o +*.a +*.so + +# Folders +_obj +_test + +# Architecture specific extensions/prefixes +*.[568vq] +[568vq].out + +*.cgo1.go +*.cgo2.c +_cgo_defun.c +_cgo_gotypes.go +_cgo_export.* + +_testmain.go + +*.exe diff --git a/vendor/github.com/kennygrant/sanitize/.travis.yml b/vendor/github.com/kennygrant/sanitize/.travis.yml new file mode 100644 index 0000000..4f2ee4d --- /dev/null +++ b/vendor/github.com/kennygrant/sanitize/.travis.yml @@ -0,0 +1 @@ +language: go diff --git a/vendor/github.com/kennygrant/sanitize/LICENSE b/vendor/github.com/kennygrant/sanitize/LICENSE new file mode 100644 index 0000000..749ebb2 --- /dev/null +++ b/vendor/github.com/kennygrant/sanitize/LICENSE @@ -0,0 +1,27 @@ +Copyright (c) 2017 Mechanism Design. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. \ No newline at end of file diff --git a/vendor/github.com/kennygrant/sanitize/README.md b/vendor/github.com/kennygrant/sanitize/README.md new file mode 100644 index 0000000..4401ef7 --- /dev/null +++ b/vendor/github.com/kennygrant/sanitize/README.md @@ -0,0 +1,62 @@ +sanitize [](https://godoc.org/github.com/kennygrant/sanitize) [](https://goreportcard.com/report/github.com/kennygrant/sanitize) [](https://circleci.com/gh/kennygrant/sanitize) +======== + +Package sanitize provides functions to sanitize html and paths with go (golang). + +FUNCTIONS + + +```go +sanitize.Accents(s string) string +``` + +Accents replaces a set of accented characters with ascii equivalents. + +```go +sanitize.BaseName(s string) string +``` + +BaseName makes a string safe to use in a file name, producing a sanitized basename replacing . or / with -. Unlike Name no attempt is made to normalise text as a path. + +```go +sanitize.HTML(s string) string +``` + +HTML strips html tags with a very simple parser, replace common entities, and escape < and > in the result. The result is intended to be used as plain text. + +```go +sanitize.HTMLAllowing(s string, args...[]string) (string, error) +``` + +HTMLAllowing parses html and allow certain tags and attributes from the lists optionally specified by args - args[0] is a list of allowed tags, args[1] is a list of allowed attributes. If either is missing default sets are used. + +```go +sanitize.Name(s string) string +``` + +Name makes a string safe to use in a file name by first finding the path basename, then replacing non-ascii characters. + +```go +sanitize.Path(s string) string +``` + +Path makes a string safe to use as an url path. + + +Changes +------- + +Version 1.2 + +Adjusted HTML function to avoid linter warning +Added more tests from https://githubengineering.com/githubs-post-csp-journey/ +Chnaged name of license file +Added badges and change log to readme + +Version 1.1 +Fixed type in comments. +Merge pull request from Povilas Balzaravicius Pawka + - replace br tags with newline even when they contain a space + +Version 1.0 +First release \ No newline at end of file diff --git a/vendor/github.com/kennygrant/sanitize/sanitize.go b/vendor/github.com/kennygrant/sanitize/sanitize.go new file mode 100644 index 0000000..2932209 --- /dev/null +++ b/vendor/github.com/kennygrant/sanitize/sanitize.go @@ -0,0 +1,388 @@ +// Package sanitize provides functions for sanitizing text. +package sanitize + +import ( + "bytes" + "html" + "html/template" + "io" + "path" + "regexp" + "strings" + + parser "golang.org/x/net/html" +) + +var ( + ignoreTags = []string{"title", "script", "style", "iframe", "frame", "frameset", "noframes", "noembed", "embed", "applet", "object", "base"} + + defaultTags = []string{"h1", "h2", "h3", "h4", "h5", "h6", "div", "span", "hr", "p", "br", "b", "i", "strong", "em", "ol", "ul", "li", "a", "img", "pre", "code", "blockquote", "article", "section"} + + defaultAttributes = []string{"id", "class", "src", "href", "title", "alt", "name", "rel"} +) + +// HTMLAllowing sanitizes html, allowing some tags. +// Arrays of allowed tags and allowed attributes may optionally be passed as the second and third arguments. +func HTMLAllowing(s string, args ...[]string) (string, error) { + + allowedTags := defaultTags + if len(args) > 0 { + allowedTags = args[0] + } + allowedAttributes := defaultAttributes + if len(args) > 1 { + allowedAttributes = args[1] + } + + // Parse the html + tokenizer := parser.NewTokenizer(strings.NewReader(s)) + + buffer := bytes.NewBufferString("") + ignore := "" + + for { + tokenType := tokenizer.Next() + token := tokenizer.Token() + + switch tokenType { + + case parser.ErrorToken: + err := tokenizer.Err() + if err == io.EOF { + return buffer.String(), nil + } + return "", err + + case parser.StartTagToken: + + if len(ignore) == 0 && includes(allowedTags, token.Data) { + token.Attr = cleanAttributes(token.Attr, allowedAttributes) + buffer.WriteString(token.String()) + } else if includes(ignoreTags, token.Data) { + ignore = token.Data + } + + case parser.SelfClosingTagToken: + + if len(ignore) == 0 && includes(allowedTags, token.Data) { + token.Attr = cleanAttributes(token.Attr, allowedAttributes) + buffer.WriteString(token.String()) + } else if token.Data == ignore { + ignore = "" + } + + case parser.EndTagToken: + if len(ignore) == 0 && includes(allowedTags, token.Data) { + token.Attr = []parser.Attribute{} + buffer.WriteString(token.String()) + } else if token.Data == ignore { + ignore = "" + } + + case parser.TextToken: + // We allow text content through, unless ignoring this entire tag and its contents (including other tags) + if ignore == "" { + buffer.WriteString(token.String()) + } + case parser.CommentToken: + // We ignore comments by default + case parser.DoctypeToken: + // We ignore doctypes by default - html5 does not require them and this is intended for sanitizing snippets of text + default: + // We ignore unknown token types by default + + } + + } + +} + +// HTML strips html tags, replace common entities, and escapes <>&;'" in the result. +// Note the returned text may contain entities as it is escaped by HTMLEscapeString, and most entities are not translated. +func HTML(s string) (output string) { + + // Shortcut strings with no tags in them + if !strings.ContainsAny(s, "<>") { + output = s + } else { + + // First remove line breaks etc as these have no meaning outside html tags (except pre) + // this means pre sections will lose formatting... but will result in less unintentional paras. + s = strings.Replace(s, "\n", "", -1) + + // Then replace line breaks with newlines, to preserve that formatting + s = strings.Replace(s, "", "\n", -1) + s = strings.Replace(s, "
", "\n", -1) + s = strings.Replace(s, "", "\n", -1) + s = strings.Replace(s, "
", "\n", -1) + s = strings.Replace(s, "
", "\n", -1) + + // Walk through the string removing all tags + b := bytes.NewBufferString("") + inTag := false + for _, r := range s { + switch r { + case '<': + inTag = true + case '>': + inTag = false + default: + if !inTag { + b.WriteRune(r) + } + } + } + output = b.String() + } + + // Remove a few common harmless entities, to arrive at something more like plain text + output = strings.Replace(output, "‘", "'", -1) + output = strings.Replace(output, "’", "'", -1) + output = strings.Replace(output, "“", "\"", -1) + output = strings.Replace(output, "”", "\"", -1) + output = strings.Replace(output, " ", " ", -1) + output = strings.Replace(output, """, "\"", -1) + output = strings.Replace(output, "'", "'", -1) + + // Translate some entities into their plain text equivalent (for example accents, if encoded as entities) + output = html.UnescapeString(output) + + // In case we have missed any tags above, escape the text - removes <, >, &, ' and ". + output = template.HTMLEscapeString(output) + + // After processing, remove some harmless entities &, ' and " which are encoded by HTMLEscapeString + output = strings.Replace(output, """, "\"", -1) + output = strings.Replace(output, "'", "'", -1) + output = strings.Replace(output, "& ", "& ", -1) // NB space after + output = strings.Replace(output, "& ", "& ", -1) // NB space after + + return output +} + +// We are very restrictive as this is intended for ascii url slugs +var illegalPath = regexp.MustCompile(`[^[:alnum:]\~\-\./]`) + +// Path makes a string safe to use as a URL path, +// removing accents and replacing separators with -. +// The path may still start at / and is not intended +// for use as a file system path without prefix. +func Path(s string) string { + // Start with lowercase string + filePath := strings.ToLower(s) + filePath = strings.Replace(filePath, "..", "", -1) + filePath = path.Clean(filePath) + + // Remove illegal characters for paths, flattening accents + // and replacing some common separators with - + filePath = cleanString(filePath, illegalPath) + + // NB this may be of length 0, caller must check + return filePath +} + +// Remove all other unrecognised characters apart from +var illegalName = regexp.MustCompile(`[^[:alnum:]-.]`) + +// Name makes a string safe to use in a file name by first finding the path basename, then replacing non-ascii characters. +func Name(s string) string { + // Start with lowercase string + fileName := strings.ToLower(s) + fileName = path.Clean(path.Base(fileName)) + + // Remove illegal characters for names, replacing some common separators with - + fileName = cleanString(fileName, illegalName) + + // NB this may be of length 0, caller must check + return fileName +} + +// Replace these separators with - +var baseNameSeparators = regexp.MustCompile(`[./]`) + +// BaseName makes a string safe to use in a file name, producing a sanitized basename replacing . or / with -. +// No attempt is made to normalise a path or normalise case. +func BaseName(s string) string { + + // Replace certain joining characters with a dash + baseName := baseNameSeparators.ReplaceAllString(s, "-") + + // Remove illegal characters for names, replacing some common separators with - + baseName = cleanString(baseName, illegalName) + + // NB this may be of length 0, caller must check + return baseName +} + +// A very limited list of transliterations to catch common european names translated to urls. +// This set could be expanded with at least caps and many more characters. +var transliterations = map[rune]string{ + 'À': "A", + 'Á': "A", + 'Â': "A", + 'Ã': "A", + 'Ä': "A", + 'Å': "AA", + 'Æ': "AE", + 'Ç': "C", + 'È': "E", + 'É': "E", + 'Ê': "E", + 'Ë': "E", + 'Ì': "I", + 'Í': "I", + 'Î': "I", + 'Ï': "I", + 'Ð': "D", + 'Ł': "L", + 'Ñ': "N", + 'Ò': "O", + 'Ó': "O", + 'Ô': "O", + 'Õ': "O", + 'Ö': "OE", + 'Ø': "OE", + 'Œ': "OE", + 'Ù': "U", + 'Ú': "U", + 'Ü': "UE", + 'Û': "U", + 'Ý': "Y", + 'Þ': "TH", + 'ẞ': "SS", + 'à': "a", + 'á': "a", + 'â': "a", + 'ã': "a", + 'ä': "ae", + 'å': "aa", + 'æ': "ae", + 'ç': "c", + 'è': "e", + 'é': "e", + 'ê': "e", + 'ë': "e", + 'ì': "i", + 'í': "i", + 'î': "i", + 'ï': "i", + 'ð': "d", + 'ł': "l", + 'ñ': "n", + 'ń': "n", + 'ò': "o", + 'ó': "o", + 'ô': "o", + 'õ': "o", + 'ō': "o", + 'ö': "oe", + 'ø': "oe", + 'œ': "oe", + 'ś': "s", + 'ù': "u", + 'ú': "u", + 'û': "u", + 'ū': "u", + 'ü': "ue", + 'ý': "y", + 'ÿ': "y", + 'ż': "z", + 'þ': "th", + 'ß': "ss", +} + +// Accents replaces a set of accented characters with ascii equivalents. +func Accents(s string) string { + // Replace some common accent characters + b := bytes.NewBufferString("") + for _, c := range s { + // Check transliterations first + if val, ok := transliterations[c]; ok { + b.WriteString(val) + } else { + b.WriteRune(c) + } + } + return b.String() +} + +var ( + // If the attribute contains data: or javascript: anywhere, ignore it + // we don't allow this in attributes as it is so frequently used for xss + // NB we allow spaces in the value, and lowercase. + illegalAttr = regexp.MustCompile(`(d\s*a\s*t\s*a|j\s*a\s*v\s*a\s*s\s*c\s*r\s*i\s*p\s*t\s*)\s*:`) + + // We are far more restrictive with href attributes. + legalHrefAttr = regexp.MustCompile(`\A[/#][^/\\]?|mailto:|http://|https://`) +) + +// cleanAttributes returns an array of attributes after removing malicious ones. +func cleanAttributes(a []parser.Attribute, allowed []string) []parser.Attribute { + if len(a) == 0 { + return a + } + + var cleaned []parser.Attribute + for _, attr := range a { + if includes(allowed, attr.Key) { + + val := strings.ToLower(attr.Val) + + // Check for illegal attribute values + if illegalAttr.FindString(val) != "" { + attr.Val = "" + } + + // Check for legal href values - / mailto:// http:// or https:// + if attr.Key == "href" { + if legalHrefAttr.FindString(val) == "" { + attr.Val = "" + } + } + + // If we still have an attribute, append it to the array + if attr.Val != "" { + cleaned = append(cleaned, attr) + } + } + } + return cleaned +} + +// A list of characters we consider separators in normal strings and replace with our canonical separator - rather than removing. +var ( + separators = regexp.MustCompile(`[ &_=+:]`) + + dashes = regexp.MustCompile(`[\-]+`) +) + +// cleanString replaces separators with - and removes characters listed in the regexp provided from string. +// Accents, spaces, and all characters not in A-Za-z0-9 are replaced. +func cleanString(s string, r *regexp.Regexp) string { + + // Remove any trailing space to avoid ending on - + s = strings.Trim(s, " ") + + // Flatten accents first so that if we remove non-ascii we still get a legible name + s = Accents(s) + + // Replace certain joining characters with a dash + s = separators.ReplaceAllString(s, "-") + + // Remove all other unrecognised characters - NB we do allow any printable characters + s = r.ReplaceAllString(s, "") + + // Remove any multiple dashes caused by replacements above + s = dashes.ReplaceAllString(s, "-") + + return s +} + +// includes checks for inclusion of a string in a []string. +func includes(a []string, s string) bool { + for _, as := range a { + if as == s { + return true + } + } + return false +} diff --git a/vendor/github.com/konsorten/go-windows-terminal-sequences/LICENSE b/vendor/github.com/konsorten/go-windows-terminal-sequences/LICENSE new file mode 100644 index 0000000..14127cd --- /dev/null +++ b/vendor/github.com/konsorten/go-windows-terminal-sequences/LICENSE @@ -0,0 +1,9 @@ +(The MIT License) + +Copyright (c) 2017 marvin + konsorten GmbH (open-source@konsorten.de) + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the 'Software'), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. diff --git a/vendor/github.com/konsorten/go-windows-terminal-sequences/README.md b/vendor/github.com/konsorten/go-windows-terminal-sequences/README.md new file mode 100644 index 0000000..09a4a35 --- /dev/null +++ b/vendor/github.com/konsorten/go-windows-terminal-sequences/README.md @@ -0,0 +1,42 @@ +# Windows Terminal Sequences + +This library allow for enabling Windows terminal color support for Go. + +See [Console Virtual Terminal Sequences](https://docs.microsoft.com/en-us/windows/console/console-virtual-terminal-sequences) for details. + +## Usage + +```go +import ( + "syscall" + + sequences "github.com/konsorten/go-windows-terminal-sequences" +) + +func main() { + sequences.EnableVirtualTerminalProcessing(syscall.Stdout, true) +} + +``` + +## Authors + +The tool is sponsored by the [marvin + konsorten GmbH](http://www.konsorten.de). + +We thank all the authors who provided code to this library: + +* Felix Kollmann +* Nicolas Perraut +* @dirty49374 + +## License + +(The MIT License) + +Copyright (c) 2018 marvin + konsorten GmbH (open-source@konsorten.de) + +Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the 'Software'), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. diff --git a/vendor/github.com/konsorten/go-windows-terminal-sequences/go.mod b/vendor/github.com/konsorten/go-windows-terminal-sequences/go.mod new file mode 100644 index 0000000..716c613 --- /dev/null +++ b/vendor/github.com/konsorten/go-windows-terminal-sequences/go.mod @@ -0,0 +1 @@ +module github.com/konsorten/go-windows-terminal-sequences diff --git a/vendor/github.com/konsorten/go-windows-terminal-sequences/sequences.go b/vendor/github.com/konsorten/go-windows-terminal-sequences/sequences.go new file mode 100644 index 0000000..57f530a --- /dev/null +++ b/vendor/github.com/konsorten/go-windows-terminal-sequences/sequences.go @@ -0,0 +1,35 @@ +// +build windows + +package sequences + +import ( + "syscall" +) + +var ( + kernel32Dll *syscall.LazyDLL = syscall.NewLazyDLL("Kernel32.dll") + setConsoleMode *syscall.LazyProc = kernel32Dll.NewProc("SetConsoleMode") +) + +func EnableVirtualTerminalProcessing(stream syscall.Handle, enable bool) error { + const ENABLE_VIRTUAL_TERMINAL_PROCESSING uint32 = 0x4 + + var mode uint32 + err := syscall.GetConsoleMode(syscall.Stdout, &mode) + if err != nil { + return err + } + + if enable { + mode |= ENABLE_VIRTUAL_TERMINAL_PROCESSING + } else { + mode &^= ENABLE_VIRTUAL_TERMINAL_PROCESSING + } + + ret, _, err := setConsoleMode.Call(uintptr(stream), uintptr(mode)) + if ret == 0 { + return err + } + + return nil +} diff --git a/vendor/github.com/konsorten/go-windows-terminal-sequences/sequences_dummy.go b/vendor/github.com/konsorten/go-windows-terminal-sequences/sequences_dummy.go new file mode 100644 index 0000000..df61a6f --- /dev/null +++ b/vendor/github.com/konsorten/go-windows-terminal-sequences/sequences_dummy.go @@ -0,0 +1,11 @@ +// +build linux darwin + +package sequences + +import ( + "fmt" +) + +func EnableVirtualTerminalProcessing(stream uintptr, enable bool) error { + return fmt.Errorf("windows only package") +} diff --git a/vendor/github.com/leodido/go-urn/.gitignore b/vendor/github.com/leodido/go-urn/.gitignore new file mode 100644 index 0000000..a30b5ab --- /dev/null +++ b/vendor/github.com/leodido/go-urn/.gitignore @@ -0,0 +1,9 @@ +*.exe +*.dll +*.so +*.dylib + +*.test + +*.out +*.txt \ No newline at end of file diff --git a/vendor/github.com/leodido/go-urn/.travis.yml b/vendor/github.com/leodido/go-urn/.travis.yml new file mode 100644 index 0000000..913b641 --- /dev/null +++ b/vendor/github.com/leodido/go-urn/.travis.yml @@ -0,0 +1,15 @@ +language: go + +go: + - 1.9.x + - 1.10.x + - tip + +before_install: + - go get -t -v ./... + +script: + - go test -race -coverprofile=coverage.txt -covermode=atomic + +after_success: + - bash <(curl -s https://codecov.io/bash) \ No newline at end of file diff --git a/vendor/github.com/leodido/go-urn/README.md b/vendor/github.com/leodido/go-urn/README.md new file mode 100644 index 0000000..cc902ec --- /dev/null +++ b/vendor/github.com/leodido/go-urn/README.md @@ -0,0 +1,55 @@ +[](https://travis-ci.org/leodido/go-urn) [](https://codecov.io/gh/leodido/go-urn) [](https://godoc.org/github.com/leodido/go-urn) + +**A parser for URNs**. + +> As seen on [RFC 2141](https://tools.ietf.org/html/rfc2141#ref-1). + +[API documentation](https://godoc.org/github.com/leodido/go-urn). + +## Installation + +``` +go get github.com/leodido/go-urn +``` + +## Performances + +This implementation results to be really fast. + +Usually below ½ microsecond on my machine[1](#mymachine). + +Notice it also performs, while parsing: + +1. fine-grained and informative erroring +2. specific-string normalization + +``` +ok/00/urn:a:b______________________________________/-4 20000000 265 ns/op 182 B/op 6 allocs/op +ok/01/URN:foo:a123,456_____________________________/-4 30000000 296 ns/op 200 B/op 6 allocs/op +ok/02/urn:foo:a123%2c456___________________________/-4 20000000 331 ns/op 208 B/op 6 allocs/op +ok/03/urn:ietf:params:scim:schemas:core:2.0:User___/-4 20000000 430 ns/op 280 B/op 6 allocs/op +ok/04/urn:ietf:params:scim:schemas:extension:enterp/-4 20000000 411 ns/op 312 B/op 6 allocs/op +ok/05/urn:ietf:params:scim:schemas:extension:enterp/-4 20000000 472 ns/op 344 B/op 6 allocs/op +ok/06/urn:burnout:nss______________________________/-4 30000000 257 ns/op 192 B/op 6 allocs/op +ok/07/urn:abcdefghilmnopqrstuvzabcdefghilm:x_______/-4 20000000 375 ns/op 213 B/op 6 allocs/op +ok/08/urn:urnurnurn:urn____________________________/-4 30000000 265 ns/op 197 B/op 6 allocs/op +ok/09/urn:ciao:@!=%2c(xyz)+a,b.*@g=$_'_____________/-4 20000000 307 ns/op 248 B/op 6 allocs/op +ok/10/URN:x:abc%1dz%2f%3az_________________________/-4 30000000 259 ns/op 212 B/op 6 allocs/op +no/11/URN:-xxx:x___________________________________/-4 20000000 445 ns/op 320 B/op 6 allocs/op +no/12/urn::colon:nss_______________________________/-4 20000000 461 ns/op 320 B/op 6 allocs/op +no/13/urn:abcdefghilmnopqrstuvzabcdefghilmn:specifi/-4 10000000 660 ns/op 320 B/op 6 allocs/op +no/14/URN:a!?:x____________________________________/-4 20000000 507 ns/op 320 B/op 6 allocs/op +no/15/urn:urn:NSS__________________________________/-4 20000000 429 ns/op 288 B/op 6 allocs/op +no/16/urn:white_space:NSS__________________________/-4 20000000 482 ns/op 320 B/op 6 allocs/op +no/17/urn:concat:no_spaces_________________________/-4 20000000 539 ns/op 328 B/op 7 allocs/op +no/18/urn:a:/______________________________________/-4 20000000 470 ns/op 320 B/op 7 allocs/op +no/19/urn:UrN:NSS__________________________________/-4 20000000 399 ns/op 288 B/op 6 allocs/op +``` + +--- + +* [1]: Intel Core i7-7600U CPU @ 2.80GHz + +--- + +[](https://github.com/igrigorik/ga-beacon) \ No newline at end of file diff --git a/vendor/github.com/leodido/go-urn/machine.go b/vendor/github.com/leodido/go-urn/machine.go new file mode 100644 index 0000000..d621ea6 --- /dev/null +++ b/vendor/github.com/leodido/go-urn/machine.go @@ -0,0 +1,1670 @@ +package urn + +import ( + "fmt" +) + +var ( + errPrefix = "expecting the prefix to be the \"urn\" string (whatever case) [col %d]" + errIdentifier = "expecting the identifier to be string (1..31 alnum chars, also containing dashes but not at its start) [col %d]" + errSpecificString = "expecting the specific string to be a string containing alnum, hex, or others ([()+,-.:=@;$_!*']) chars [col %d]" + errNoUrnWithinID = "expecting the identifier to not contain the \"urn\" reserved string [col %d]" + errHex = "expecting the specific string hex chars to be well-formed (%%alnum{2}) [col %d]" + errParse = "parsing error [col %d]" +) + + +const start int = 1 +const first_final int = 44 + +const en_fail int = 46 +const en_main int = 1 + + +// Machine is the interface representing the FSM +type Machine interface { + Error() error + Parse(input []byte) (*URN, error) +} + +type machine struct { + data []byte + cs int + p, pe, eof, pb int + err error + tolower []int +} + +// NewMachine creates a new FSM able to parse RFC 2141 strings. +func NewMachine() Machine { + m := &machine{} + + return m +} + +// Err returns the error that occurred on the last call to Parse. +// +// If the result is nil, then the line was parsed successfully. +func (m *machine) Error() error { + return m.err +} + +func (m *machine) text() []byte { + return m.data[m.pb:m.p] +} + +// Parse parses the input byte array as a RFC 2141 string. +func (m *machine) Parse(input []byte) (*URN, error) { + m.data = input + m.p = 0 + m.pb = 0 + m.pe = len(input) + m.eof = len(input) + m.err = nil + m.tolower = []int{} + output := &URN{} + + { + m.cs = start + } + + + { + if (m.p) == (m.pe) { + goto _test_eof + } + switch m.cs { + case 1: + goto st_case_1 + case 0: + goto st_case_0 + case 2: + goto st_case_2 + case 3: + goto st_case_3 + case 4: + goto st_case_4 + case 5: + goto st_case_5 + case 6: + goto st_case_6 + case 7: + goto st_case_7 + case 8: + goto st_case_8 + case 9: + goto st_case_9 + case 10: + goto st_case_10 + case 11: + goto st_case_11 + case 12: + goto st_case_12 + case 13: + goto st_case_13 + case 14: + goto st_case_14 + case 15: + goto st_case_15 + case 16: + goto st_case_16 + case 17: + goto st_case_17 + case 18: + goto st_case_18 + case 19: + goto st_case_19 + case 20: + goto st_case_20 + case 21: + goto st_case_21 + case 22: + goto st_case_22 + case 23: + goto st_case_23 + case 24: + goto st_case_24 + case 25: + goto st_case_25 + case 26: + goto st_case_26 + case 27: + goto st_case_27 + case 28: + goto st_case_28 + case 29: + goto st_case_29 + case 30: + goto st_case_30 + case 31: + goto st_case_31 + case 32: + goto st_case_32 + case 33: + goto st_case_33 + case 34: + goto st_case_34 + case 35: + goto st_case_35 + case 36: + goto st_case_36 + case 37: + goto st_case_37 + case 38: + goto st_case_38 + case 44: + goto st_case_44 + case 39: + goto st_case_39 + case 40: + goto st_case_40 + case 45: + goto st_case_45 + case 41: + goto st_case_41 + case 42: + goto st_case_42 + case 43: + goto st_case_43 + case 46: + goto st_case_46 + } + goto st_out + st_case_1: + switch (m.data)[(m.p)] { + case 85: + goto tr1 + case 117: + goto tr1 + } + goto tr0 + tr0: + m.err = fmt.Errorf(errParse, m.p) + (m.p)-- + + { + goto st46 + } + + goto st0 + tr3: + m.err = fmt.Errorf(errPrefix, m.p) + (m.p)-- + + { + goto st46 + } + + m.err = fmt.Errorf(errParse, m.p) + (m.p)-- + + { + goto st46 + } + + goto st0 + tr6: + m.err = fmt.Errorf(errIdentifier, m.p) + (m.p)-- + + { + goto st46 + } + + m.err = fmt.Errorf(errParse, m.p) + (m.p)-- + + { + goto st46 + } + + goto st0 + tr41: + m.err = fmt.Errorf(errSpecificString, m.p) + (m.p)-- + + { + goto st46 + } + + m.err = fmt.Errorf(errParse, m.p) + (m.p)-- + + { + goto st46 + } + + goto st0 + tr44: + m.err = fmt.Errorf(errHex, m.p) + (m.p)-- + + { + goto st46 + } + + m.err = fmt.Errorf(errSpecificString, m.p) + (m.p)-- + + { + goto st46 + } + + m.err = fmt.Errorf(errParse, m.p) + (m.p)-- + + { + goto st46 + } + + goto st0 + tr50: + m.err = fmt.Errorf(errPrefix, m.p) + (m.p)-- + + { + goto st46 + } + + m.err = fmt.Errorf(errIdentifier, m.p) + (m.p)-- + + { + goto st46 + } + + m.err = fmt.Errorf(errParse, m.p) + (m.p)-- + + { + goto st46 + } + + goto st0 + tr52: + m.err = fmt.Errorf(errNoUrnWithinID, m.p) + (m.p)-- + + { + goto st46 + } + + m.err = fmt.Errorf(errIdentifier, m.p) + (m.p)-- + + { + goto st46 + } + + m.err = fmt.Errorf(errParse, m.p) + (m.p)-- + + { + goto st46 + } + + goto st0 + st_case_0: + st0: + m.cs = 0 + goto _out + tr1: + m.pb = m.p + + goto st2 + st2: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof2 + } + st_case_2: + switch (m.data)[(m.p)] { + case 82: + goto st3 + case 114: + goto st3 + } + goto tr0 + st3: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof3 + } + st_case_3: + switch (m.data)[(m.p)] { + case 78: + goto st4 + case 110: + goto st4 + } + goto tr3 + st4: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof4 + } + st_case_4: + if (m.data)[(m.p)] == 58 { + goto tr5 + } + goto tr0 + tr5: + output.prefix = string(m.text()) + + goto st5 + st5: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof5 + } + st_case_5: + switch (m.data)[(m.p)] { + case 85: + goto tr8 + case 117: + goto tr8 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto tr7 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto tr7 + } + default: + goto tr7 + } + goto tr6 + tr7: + m.pb = m.p + + goto st6 + st6: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof6 + } + st_case_6: + switch (m.data)[(m.p)] { + case 45: + goto st7 + case 58: + goto tr10 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st7 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st7 + } + default: + goto st7 + } + goto tr6 + st7: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof7 + } + st_case_7: + switch (m.data)[(m.p)] { + case 45: + goto st8 + case 58: + goto tr10 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st8 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st8 + } + default: + goto st8 + } + goto tr6 + st8: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof8 + } + st_case_8: + switch (m.data)[(m.p)] { + case 45: + goto st9 + case 58: + goto tr10 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st9 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st9 + } + default: + goto st9 + } + goto tr6 + st9: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof9 + } + st_case_9: + switch (m.data)[(m.p)] { + case 45: + goto st10 + case 58: + goto tr10 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st10 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st10 + } + default: + goto st10 + } + goto tr6 + st10: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof10 + } + st_case_10: + switch (m.data)[(m.p)] { + case 45: + goto st11 + case 58: + goto tr10 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st11 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st11 + } + default: + goto st11 + } + goto tr6 + st11: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof11 + } + st_case_11: + switch (m.data)[(m.p)] { + case 45: + goto st12 + case 58: + goto tr10 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st12 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st12 + } + default: + goto st12 + } + goto tr6 + st12: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof12 + } + st_case_12: + switch (m.data)[(m.p)] { + case 45: + goto st13 + case 58: + goto tr10 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st13 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st13 + } + default: + goto st13 + } + goto tr6 + st13: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof13 + } + st_case_13: + switch (m.data)[(m.p)] { + case 45: + goto st14 + case 58: + goto tr10 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st14 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st14 + } + default: + goto st14 + } + goto tr6 + st14: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof14 + } + st_case_14: + switch (m.data)[(m.p)] { + case 45: + goto st15 + case 58: + goto tr10 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st15 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st15 + } + default: + goto st15 + } + goto tr6 + st15: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof15 + } + st_case_15: + switch (m.data)[(m.p)] { + case 45: + goto st16 + case 58: + goto tr10 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st16 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st16 + } + default: + goto st16 + } + goto tr6 + st16: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof16 + } + st_case_16: + switch (m.data)[(m.p)] { + case 45: + goto st17 + case 58: + goto tr10 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st17 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st17 + } + default: + goto st17 + } + goto tr6 + st17: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof17 + } + st_case_17: + switch (m.data)[(m.p)] { + case 45: + goto st18 + case 58: + goto tr10 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st18 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st18 + } + default: + goto st18 + } + goto tr6 + st18: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof18 + } + st_case_18: + switch (m.data)[(m.p)] { + case 45: + goto st19 + case 58: + goto tr10 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st19 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st19 + } + default: + goto st19 + } + goto tr6 + st19: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof19 + } + st_case_19: + switch (m.data)[(m.p)] { + case 45: + goto st20 + case 58: + goto tr10 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st20 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st20 + } + default: + goto st20 + } + goto tr6 + st20: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof20 + } + st_case_20: + switch (m.data)[(m.p)] { + case 45: + goto st21 + case 58: + goto tr10 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st21 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st21 + } + default: + goto st21 + } + goto tr6 + st21: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof21 + } + st_case_21: + switch (m.data)[(m.p)] { + case 45: + goto st22 + case 58: + goto tr10 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st22 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st22 + } + default: + goto st22 + } + goto tr6 + st22: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof22 + } + st_case_22: + switch (m.data)[(m.p)] { + case 45: + goto st23 + case 58: + goto tr10 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st23 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st23 + } + default: + goto st23 + } + goto tr6 + st23: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof23 + } + st_case_23: + switch (m.data)[(m.p)] { + case 45: + goto st24 + case 58: + goto tr10 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st24 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st24 + } + default: + goto st24 + } + goto tr6 + st24: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof24 + } + st_case_24: + switch (m.data)[(m.p)] { + case 45: + goto st25 + case 58: + goto tr10 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st25 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st25 + } + default: + goto st25 + } + goto tr6 + st25: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof25 + } + st_case_25: + switch (m.data)[(m.p)] { + case 45: + goto st26 + case 58: + goto tr10 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st26 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st26 + } + default: + goto st26 + } + goto tr6 + st26: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof26 + } + st_case_26: + switch (m.data)[(m.p)] { + case 45: + goto st27 + case 58: + goto tr10 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st27 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st27 + } + default: + goto st27 + } + goto tr6 + st27: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof27 + } + st_case_27: + switch (m.data)[(m.p)] { + case 45: + goto st28 + case 58: + goto tr10 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st28 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st28 + } + default: + goto st28 + } + goto tr6 + st28: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof28 + } + st_case_28: + switch (m.data)[(m.p)] { + case 45: + goto st29 + case 58: + goto tr10 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st29 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st29 + } + default: + goto st29 + } + goto tr6 + st29: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof29 + } + st_case_29: + switch (m.data)[(m.p)] { + case 45: + goto st30 + case 58: + goto tr10 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st30 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st30 + } + default: + goto st30 + } + goto tr6 + st30: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof30 + } + st_case_30: + switch (m.data)[(m.p)] { + case 45: + goto st31 + case 58: + goto tr10 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st31 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st31 + } + default: + goto st31 + } + goto tr6 + st31: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof31 + } + st_case_31: + switch (m.data)[(m.p)] { + case 45: + goto st32 + case 58: + goto tr10 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st32 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st32 + } + default: + goto st32 + } + goto tr6 + st32: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof32 + } + st_case_32: + switch (m.data)[(m.p)] { + case 45: + goto st33 + case 58: + goto tr10 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st33 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st33 + } + default: + goto st33 + } + goto tr6 + st33: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof33 + } + st_case_33: + switch (m.data)[(m.p)] { + case 45: + goto st34 + case 58: + goto tr10 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st34 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st34 + } + default: + goto st34 + } + goto tr6 + st34: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof34 + } + st_case_34: + switch (m.data)[(m.p)] { + case 45: + goto st35 + case 58: + goto tr10 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st35 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st35 + } + default: + goto st35 + } + goto tr6 + st35: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof35 + } + st_case_35: + switch (m.data)[(m.p)] { + case 45: + goto st36 + case 58: + goto tr10 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st36 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st36 + } + default: + goto st36 + } + goto tr6 + st36: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof36 + } + st_case_36: + switch (m.data)[(m.p)] { + case 45: + goto st37 + case 58: + goto tr10 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st37 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st37 + } + default: + goto st37 + } + goto tr6 + st37: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof37 + } + st_case_37: + if (m.data)[(m.p)] == 58 { + goto tr10 + } + goto tr6 + tr10: + output.ID = string(m.text()) + + goto st38 + st38: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof38 + } + st_case_38: + switch (m.data)[(m.p)] { + case 33: + goto tr42 + case 36: + goto tr42 + case 37: + goto tr43 + case 61: + goto tr42 + case 95: + goto tr42 + } + switch { + case (m.data)[(m.p)] < 48: + if 39 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 46 { + goto tr42 + } + case (m.data)[(m.p)] > 59: + switch { + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto tr42 + } + case (m.data)[(m.p)] >= 64: + goto tr42 + } + default: + goto tr42 + } + goto tr41 + tr42: + m.pb = m.p + + goto st44 + st44: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof44 + } + st_case_44: + switch (m.data)[(m.p)] { + case 33: + goto st44 + case 36: + goto st44 + case 37: + goto st39 + case 61: + goto st44 + case 95: + goto st44 + } + switch { + case (m.data)[(m.p)] < 48: + if 39 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 46 { + goto st44 + } + case (m.data)[(m.p)] > 59: + switch { + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st44 + } + case (m.data)[(m.p)] >= 64: + goto st44 + } + default: + goto st44 + } + goto tr41 + tr43: + m.pb = m.p + + goto st39 + st39: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof39 + } + st_case_39: + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st40 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st40 + } + default: + goto tr46 + } + goto tr44 + tr46: + m.tolower = append(m.tolower, m.p-m.pb) + + goto st40 + st40: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof40 + } + st_case_40: + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st45 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st45 + } + default: + goto tr48 + } + goto tr44 + tr48: + m.tolower = append(m.tolower, m.p-m.pb) + + goto st45 + st45: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof45 + } + st_case_45: + switch (m.data)[(m.p)] { + case 33: + goto st44 + case 36: + goto st44 + case 37: + goto st39 + case 61: + goto st44 + case 95: + goto st44 + } + switch { + case (m.data)[(m.p)] < 48: + if 39 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 46 { + goto st44 + } + case (m.data)[(m.p)] > 59: + switch { + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st44 + } + case (m.data)[(m.p)] >= 64: + goto st44 + } + default: + goto st44 + } + goto tr44 + tr8: + m.pb = m.p + + goto st41 + st41: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof41 + } + st_case_41: + switch (m.data)[(m.p)] { + case 45: + goto st7 + case 58: + goto tr10 + case 82: + goto st42 + case 114: + goto st42 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st7 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st7 + } + default: + goto st7 + } + goto tr6 + st42: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof42 + } + st_case_42: + switch (m.data)[(m.p)] { + case 45: + goto st8 + case 58: + goto tr10 + case 78: + goto st43 + case 110: + goto st43 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st8 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st8 + } + default: + goto st8 + } + goto tr50 + st43: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof43 + } + st_case_43: + if (m.data)[(m.p)] == 45 { + goto st9 + } + switch { + case (m.data)[(m.p)] < 65: + if 48 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 57 { + goto st9 + } + case (m.data)[(m.p)] > 90: + if 97 <= (m.data)[(m.p)] && (m.data)[(m.p)] <= 122 { + goto st9 + } + default: + goto st9 + } + goto tr52 + st46: + if (m.p)++; (m.p) == (m.pe) { + goto _test_eof46 + } + st_case_46: + switch (m.data)[(m.p)] { + case 10: + goto st0 + case 13: + goto st0 + } + goto st46 + st_out: + _test_eof2: + m.cs = 2 + goto _test_eof + _test_eof3: + m.cs = 3 + goto _test_eof + _test_eof4: + m.cs = 4 + goto _test_eof + _test_eof5: + m.cs = 5 + goto _test_eof + _test_eof6: + m.cs = 6 + goto _test_eof + _test_eof7: + m.cs = 7 + goto _test_eof + _test_eof8: + m.cs = 8 + goto _test_eof + _test_eof9: + m.cs = 9 + goto _test_eof + _test_eof10: + m.cs = 10 + goto _test_eof + _test_eof11: + m.cs = 11 + goto _test_eof + _test_eof12: + m.cs = 12 + goto _test_eof + _test_eof13: + m.cs = 13 + goto _test_eof + _test_eof14: + m.cs = 14 + goto _test_eof + _test_eof15: + m.cs = 15 + goto _test_eof + _test_eof16: + m.cs = 16 + goto _test_eof + _test_eof17: + m.cs = 17 + goto _test_eof + _test_eof18: + m.cs = 18 + goto _test_eof + _test_eof19: + m.cs = 19 + goto _test_eof + _test_eof20: + m.cs = 20 + goto _test_eof + _test_eof21: + m.cs = 21 + goto _test_eof + _test_eof22: + m.cs = 22 + goto _test_eof + _test_eof23: + m.cs = 23 + goto _test_eof + _test_eof24: + m.cs = 24 + goto _test_eof + _test_eof25: + m.cs = 25 + goto _test_eof + _test_eof26: + m.cs = 26 + goto _test_eof + _test_eof27: + m.cs = 27 + goto _test_eof + _test_eof28: + m.cs = 28 + goto _test_eof + _test_eof29: + m.cs = 29 + goto _test_eof + _test_eof30: + m.cs = 30 + goto _test_eof + _test_eof31: + m.cs = 31 + goto _test_eof + _test_eof32: + m.cs = 32 + goto _test_eof + _test_eof33: + m.cs = 33 + goto _test_eof + _test_eof34: + m.cs = 34 + goto _test_eof + _test_eof35: + m.cs = 35 + goto _test_eof + _test_eof36: + m.cs = 36 + goto _test_eof + _test_eof37: + m.cs = 37 + goto _test_eof + _test_eof38: + m.cs = 38 + goto _test_eof + _test_eof44: + m.cs = 44 + goto _test_eof + _test_eof39: + m.cs = 39 + goto _test_eof + _test_eof40: + m.cs = 40 + goto _test_eof + _test_eof45: + m.cs = 45 + goto _test_eof + _test_eof41: + m.cs = 41 + goto _test_eof + _test_eof42: + m.cs = 42 + goto _test_eof + _test_eof43: + m.cs = 43 + goto _test_eof + _test_eof46: + m.cs = 46 + goto _test_eof + + _test_eof: + { + } + if (m.p) == (m.eof) { + switch m.cs { + case 44, 45: + raw := m.text() + output.SS = string(raw) + // Iterate upper letters lowering them + for _, i := range m.tolower { + raw[i] = raw[i] + 32 + } + output.norm = string(raw) + + case 1, 2, 4: + m.err = fmt.Errorf(errParse, m.p) + (m.p)-- + + { + goto st46 + } + + case 3: + m.err = fmt.Errorf(errPrefix, m.p) + (m.p)-- + + { + goto st46 + } + + m.err = fmt.Errorf(errParse, m.p) + (m.p)-- + + { + goto st46 + } + + case 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 41: + m.err = fmt.Errorf(errIdentifier, m.p) + (m.p)-- + + { + goto st46 + } + + m.err = fmt.Errorf(errParse, m.p) + (m.p)-- + + { + goto st46 + } + + case 38: + m.err = fmt.Errorf(errSpecificString, m.p) + (m.p)-- + + { + goto st46 + } + + m.err = fmt.Errorf(errParse, m.p) + (m.p)-- + + { + goto st46 + } + + case 42: + m.err = fmt.Errorf(errPrefix, m.p) + (m.p)-- + + { + goto st46 + } + + m.err = fmt.Errorf(errIdentifier, m.p) + (m.p)-- + + { + goto st46 + } + + m.err = fmt.Errorf(errParse, m.p) + (m.p)-- + + { + goto st46 + } + + case 43: + m.err = fmt.Errorf(errNoUrnWithinID, m.p) + (m.p)-- + + { + goto st46 + } + + m.err = fmt.Errorf(errIdentifier, m.p) + (m.p)-- + + { + goto st46 + } + + m.err = fmt.Errorf(errParse, m.p) + (m.p)-- + + { + goto st46 + } + + case 39, 40: + m.err = fmt.Errorf(errHex, m.p) + (m.p)-- + + { + goto st46 + } + + m.err = fmt.Errorf(errSpecificString, m.p) + (m.p)-- + + { + goto st46 + } + + m.err = fmt.Errorf(errParse, m.p) + (m.p)-- + + { + goto st46 + } + + } + } + + _out: + { + } + } + + if m.cs < first_final || m.cs == en_fail { + return nil, m.err + } + + return output, nil +} diff --git a/vendor/github.com/leodido/go-urn/machine.go.rl b/vendor/github.com/leodido/go-urn/machine.go.rl new file mode 100644 index 0000000..3bc05a6 --- /dev/null +++ b/vendor/github.com/leodido/go-urn/machine.go.rl @@ -0,0 +1,159 @@ +package urn + +import ( + "fmt" +) + +var ( + errPrefix = "expecting the prefix to be the \"urn\" string (whatever case) [col %d]" + errIdentifier = "expecting the identifier to be string (1..31 alnum chars, also containing dashes but not at its start) [col %d]" + errSpecificString = "expecting the specific string to be a string containing alnum, hex, or others ([()+,-.:=@;$_!*']) chars [col %d]" + errNoUrnWithinID = "expecting the identifier to not contain the \"urn\" reserved string [col %d]" + errHex = "expecting the specific string hex chars to be well-formed (%%alnum{2}) [col %d]" + errParse = "parsing error [col %d]" +) + +%%{ +machine urn; + +# unsigned alphabet +alphtype uint8; + +action mark { + m.pb = m.p +} + +action tolower { + m.tolower = append(m.tolower, m.p - m.pb) +} + +action set_pre { + output.prefix = string(m.text()) +} + +action set_nid { + output.ID = string(m.text()) +} + +action set_nss { + raw := m.text() + output.SS = string(raw) + // Iterate upper letters lowering them + for _, i := range m.tolower { + raw[i] = raw[i] + 32 + } + output.norm = string(raw) +} + +action err_pre { + m.err = fmt.Errorf(errPrefix, m.p) + fhold; + fgoto fail; +} + +action err_nid { + m.err = fmt.Errorf(errIdentifier, m.p) + fhold; + fgoto fail; +} + +action err_nss { + m.err = fmt.Errorf(errSpecificString, m.p) + fhold; + fgoto fail; +} + +action err_urn { + m.err = fmt.Errorf(errNoUrnWithinID, m.p) + fhold; + fgoto fail; +} + +action err_hex { + m.err = fmt.Errorf(errHex, m.p) + fhold; + fgoto fail; +} + +action err_parse { + m.err = fmt.Errorf(errParse, m.p) + fhold; + fgoto fail; +} + +pre = ([uU][rR][nN] @err(err_pre)) >mark %set_pre; + +nid = (alnum >mark (alnum | '-'){0,31}) %set_nid; + +hex = '%' (digit | lower | upper >tolower){2} $err(err_hex); + +sss = (alnum | [()+,\-.:=@;$_!*']); + +nss = (sss | hex)+ $err(err_nss); + +fail := (any - [\n\r])* @err{ fgoto main; }; + +main := (pre ':' (nid - pre %err(err_urn)) $err(err_nid) ':' nss >mark %set_nss) $err(err_parse); + +}%% + +%% write data noerror noprefix; + +// Machine is the interface representing the FSM +type Machine interface { + Error() error + Parse(input []byte) (*URN, error) +} + +type machine struct { + data []byte + cs int + p, pe, eof, pb int + err error + tolower []int +} + +// NewMachine creates a new FSM able to parse RFC 2141 strings. +func NewMachine() Machine { + m := &machine{} + + %% access m.; + %% variable p m.p; + %% variable pe m.pe; + %% variable eof m.eof; + %% variable data m.data; + + return m +} + +// Err returns the error that occurred on the last call to Parse. +// +// If the result is nil, then the line was parsed successfully. +func (m *machine) Error() error { + return m.err +} + +func (m *machine) text() []byte { + return m.data[m.pb:m.p] +} + +// Parse parses the input byte array as a RFC 2141 string. +func (m *machine) Parse(input []byte) (*URN, error) { + m.data = input + m.p = 0 + m.pb = 0 + m.pe = len(input) + m.eof = len(input) + m.err = nil + m.tolower = []int{} + output := &URN{} + + %% write init; + %% write exec; + + if m.cs < first_final || m.cs == en_fail { + return nil, m.err + } + + return output, nil +} diff --git a/vendor/github.com/leodido/go-urn/makefile b/vendor/github.com/leodido/go-urn/makefile new file mode 100644 index 0000000..362137a --- /dev/null +++ b/vendor/github.com/leodido/go-urn/makefile @@ -0,0 +1,17 @@ +SHELL := /bin/bash + +machine.go: machine.go.rl + ragel -Z -G2 -e -o $@ $< + @gofmt -w -s $@ + @sed -i '/^\/\/line/d' $@ + +.PHONY: build +build: machine.go + +.PHONY: bench +bench: *_test.go machine.go + go test -bench=. -benchmem -benchtime=5s ./... + +.PHONY: tests +tests: *_test.go machine.go + go test -race -timeout 10s -coverprofile=coverage.out -covermode=atomic -v ./... \ No newline at end of file diff --git a/vendor/github.com/leodido/go-urn/urn.go b/vendor/github.com/leodido/go-urn/urn.go new file mode 100644 index 0000000..b903b7b --- /dev/null +++ b/vendor/github.com/leodido/go-urn/urn.go @@ -0,0 +1,63 @@ +package urn + +import ( + "strings" +) + +// URN represents an Uniform Resource Name. +// +// The general form represented is: +// +// urn:
+
+```
+
+To pass through mostly unaltered (it gained a rel="nofollow" which is a good thing for user generated content):
+```html
+
+
+
+```
+
+It protects sites from [XSS](http://en.wikipedia.org/wiki/Cross-site_scripting) attacks. There are many [vectors for an XSS attack](https://www.owasp.org/index.php/XSS_Filter_Evasion_Cheat_Sheet) and the best way to mitigate the risk is to sanitize user input against a known safe list of HTML elements and attributes.
+
+You should **always** run bluemonday **after** any other processing.
+
+If you use [blackfriday](https://github.com/russross/blackfriday) or [Pandoc](http://johnmacfarlane.net/pandoc/) then bluemonday should be run after these steps. This ensures that no insecure HTML is introduced later in your process.
+
+bluemonday is heavily inspired by both the [OWASP Java HTML Sanitizer](https://code.google.com/p/owasp-java-html-sanitizer/) and the [HTML Purifier](http://htmlpurifier.org/).
+
+## Technical Summary
+
+Whitelist based, you need to either build a policy describing the HTML elements and attributes to permit (and the `regexp` patterns of attributes), or use one of the supplied policies representing good defaults.
+
+The policy containing the whitelist is applied using a fast non-validating, forward only, token-based parser implemented in the [Go net/html library](https://godoc.org/golang.org/x/net/html) by the core Go team.
+
+We expect to be supplied with well-formatted HTML (closing elements for every applicable open element, nested correctly) and so we do not focus on repairing badly nested or incomplete HTML. We focus on simply ensuring that whatever elements do exist are described in the policy whitelist and that attributes and links are safe for use on your web page. [GIGO](http://en.wikipedia.org/wiki/Garbage_in,_garbage_out) does apply and if you feed it bad HTML bluemonday is not tasked with figuring out how to make it good again.
+
+### Supported Go Versions
+
+bluemonday is tested against Go 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, and tip.
+
+We do not support Go 1.0 as we depend on `golang.org/x/net/html` which includes a reference to `io.ErrNoProgress` which did not exist in Go 1.0.
+
+## Is it production ready?
+
+*Yes*
+
+We are using bluemonday in production having migrated from the widely used and heavily field tested OWASP Java HTML Sanitizer.
+
+We are passing our extensive test suite (including AntiSamy tests as well as tests for any issues raised). Check for any [unresolved issues](https://github.com/microcosm-cc/bluemonday/issues?page=1&state=open) to see whether anything may be a blocker for you.
+
+We invite pull requests and issues to help us ensure we are offering comprehensive protection against various attacks via user generated content.
+
+## Usage
+
+Install in your `${GOPATH}` using `go get -u github.com/microcosm-cc/bluemonday`
+
+Then call it:
+```go
+package main
+
+import (
+ "fmt"
+
+ "github.com/microcosm-cc/bluemonday"
+)
+
+func main() {
+ // Do this once for each unique policy, and use the policy for the life of the program
+ // Policy creation/editing is not safe to use in multiple goroutines
+ p := bluemonday.UGCPolicy()
+
+ // The policy can then be used to sanitize lots of input and it is safe to use the policy in multiple goroutines
+ html := p.Sanitize(
+ `Google`,
+ )
+
+ // Output:
+ // Google
+ fmt.Println(html)
+}
+```
+
+We offer three ways to call Sanitize:
+```go
+p.Sanitize(string) string
+p.SanitizeBytes([]byte) []byte
+p.SanitizeReader(io.Reader) bytes.Buffer
+```
+
+If you are obsessed about performance, `p.SanitizeReader(r).Bytes()` will return a `[]byte` without performing any unnecessary casting of the inputs or outputs. Though the difference is so negligible you should never need to care.
+
+You can build your own policies:
+```go
+package main
+
+import (
+ "fmt"
+
+ "github.com/microcosm-cc/bluemonday"
+)
+
+func main() {
+ p := bluemonday.NewPolicy()
+
+ // Require URLs to be parseable by net/url.Parse and either:
+ // mailto: http:// or https://
+ p.AllowStandardURLs()
+
+ // We only allow and
+ p.AllowAttrs("href").OnElements("a")
+ p.AllowElements("p")
+
+ html := p.Sanitize(
+ `Google`,
+ )
+
+ // Output:
+ // Google
+ fmt.Println(html)
+}
+```
+
+We ship two default policies:
+
+1. `bluemonday.StrictPolicy()` which can be thought of as equivalent to stripping all HTML elements and their attributes as it has nothing on its whitelist. An example usage scenario would be blog post titles where HTML tags are not expected at all and if they are then the elements *and* the content of the elements should be stripped. This is a *very* strict policy.
+2. `bluemonday.UGCPolicy()` which allows a broad selection of HTML elements and attributes that are safe for user generated content. Note that this policy does *not* whitelist iframes, object, embed, styles, script, etc. An example usage scenario would be blog post bodies where a variety of formatting is expected along with the potential for TABLEs and IMGs.
+
+## Policy Building
+
+The essence of building a policy is to determine which HTML elements and attributes are considered safe for your scenario. OWASP provide an [XSS prevention cheat sheet](https://www.owasp.org/index.php/XSS_(Cross_Site_Scripting)_Prevention_Cheat_Sheet) to help explain the risks, but essentially:
+
+1. Avoid anything other than the standard HTML elements
+1. Avoid `script`, `style`, `iframe`, `object`, `embed`, `base` elements that allow code to be executed by the client or third party content to be included that can execute code
+1. Avoid anything other than plain HTML attributes with values matched to a regexp
+
+Basically, you should be able to describe what HTML is fine for your scenario. If you do not have confidence that you can describe your policy please consider using one of the shipped policies such as `bluemonday.UGCPolicy()`.
+
+To create a new policy:
+```go
+p := bluemonday.NewPolicy()
+```
+
+To add elements to a policy either add just the elements:
+```go
+p.AllowElements("b", "strong")
+```
+
+Or add elements as a virtue of adding an attribute:
+```go
+// Not the recommended pattern, see the recommendation on using .Matching() below
+p.AllowAttrs("nowrap").OnElements("td", "th")
+```
+
+Attributes can either be added to all elements:
+```go
+p.AllowAttrs("dir").Matching(regexp.MustCompile("(?i)rtl|ltr")).Globally()
+```
+
+Or attributes can be added to specific elements:
+```go
+// Not the recommended pattern, see the recommendation on using .Matching() below
+p.AllowAttrs("value").OnElements("li")
+```
+
+It is **always** recommended that an attribute be made to match a pattern. XSS in HTML attributes is very easy otherwise:
+```go
+// \p{L} matches unicode letters, \p{N} matches unicode numbers
+p.AllowAttrs("title").Matching(regexp.MustCompile(`[\p{L}\p{N}\s\-_',:\[\]!\./\\\(\)&]*`)).Globally()
+```
+
+You can stop at any time and call .Sanitize():
+```go
+// string htmlIn passed in from a HTTP POST
+htmlOut := p.Sanitize(htmlIn)
+```
+
+And you can take any existing policy and extend it:
+```go
+p := bluemonday.UGCPolicy()
+p.AllowElements("fieldset", "select", "option")
+```
+
+### Links
+
+Links are difficult beasts to sanitise safely and also one of the biggest attack vectors for malicious content.
+
+It is possible to do this:
+```go
+p.AllowAttrs("href").Matching(regexp.MustCompile(`(?i)mailto|https?`)).OnElements("a")
+```
+
+But that will not protect you as the regular expression is insufficient in this case to have prevented a malformed value doing something unexpected.
+
+We provide some additional global options for safely working with links.
+
+`RequireParseableURLs` will ensure that URLs are parseable by Go's `net/url` package:
+```go
+p.RequireParseableURLs(true)
+```
+
+If you have enabled parseable URLs then the following option will `AllowRelativeURLs`. By default this is disabled (bluemonday is a whitelist tool... you need to explicitly tell us to permit things) and when disabled it will prevent all local and scheme relative URLs (i.e. `href="localpage.html"`, `href="../home.html"` and even `href="//www.google.com"` are relative):
+```go
+p.AllowRelativeURLs(true)
+```
+
+If you have enabled parseable URLs then you can whitelist the schemes (commonly called protocol when thinking of `http` and `https`) that are permitted. Bear in mind that allowing relative URLs in the above option will allow for a blank scheme:
+```go
+p.AllowURLSchemes("mailto", "http", "https")
+```
+
+Regardless of whether you have enabled parseable URLs, you can force all URLs to have a rel="nofollow" attribute. This will be added if it does not exist, but only when the `href` is valid:
+```go
+// This applies to "a" "area" "link" elements that have a "href" attribute
+p.RequireNoFollowOnLinks(true)
+```
+
+We provide a convenience method that applies all of the above, but you will still need to whitelist the linkable elements for the URL rules to be applied to:
+```go
+p.AllowStandardURLs()
+p.AllowAttrs("cite").OnElements("blockquote", "q")
+p.AllowAttrs("href").OnElements("a", "area")
+p.AllowAttrs("src").OnElements("img")
+```
+
+An additional complexity regarding links is the data URI as defined in [RFC2397](http://tools.ietf.org/html/rfc2397). The data URI allows for images to be served inline using this format:
+
+```html
+ Hello World
+```
+
+We have provided a helper to verify the mimetype followed by base64 content of data URIs links:
+
+```go
+p.AllowDataURIImages()
+```
+
+That helper will enable GIF, JPEG, PNG and WEBP images.
+
+It should be noted that there is a potential [security](http://palizine.plynt.com/issues/2010Oct/bypass-xss-filters/) [risk](https://capec.mitre.org/data/definitions/244.html) with the use of data URI links. You should only enable data URI links if you already trust the content.
+
+We also have some features to help deal with user generated content:
+```go
+p.AddTargetBlankToFullyQualifiedLinks(true)
+```
+
+This will ensure that anchor `` links that are fully qualified (the href destination includes a host name) will get `target="_blank"` added to them.
+
+Additionally any link that has `target="_blank"` after the policy has been applied will also have the `rel` attribute adjusted to add `noopener`. This means a link may start like `` and will end up as ``. It is important to note that the addition of `noopener` is a security feature and not an issue. There is an unfortunate feature to browsers that a browser window opened as a result of `target="_blank"` can still control the opener (your web page) and this protects against that. The background to this can be found here: [https://dev.to/ben/the-targetblank-vulnerability-by-example](https://dev.to/ben/the-targetblank-vulnerability-by-example)
+
+### Policy Building Helpers
+
+We also bundle some helpers to simplify policy building:
+```go
+
+// Permits the "dir", "id", "lang", "title" attributes globally
+p.AllowStandardAttributes()
+
+// Permits the "img" element and its standard attributes
+p.AllowImages()
+
+// Permits ordered and unordered lists, and also definition lists
+p.AllowLists()
+
+// Permits HTML tables and all applicable elements and non-styling attributes
+p.AllowTables()
+```
+
+### Invalid Instructions
+
+The following are invalid:
+```go
+// This does not say where the attributes are allowed, you need to add
+// .Globally() or .OnElements(...)
+// This will be ignored without error.
+p.AllowAttrs("value")
+
+// This does not say where the attributes are allowed, you need to add
+// .Globally() or .OnElements(...)
+// This will be ignored without error.
+p.AllowAttrs(
+ "type",
+).Matching(
+ regexp.MustCompile("(?i)^(circle|disc|square|a|A|i|I|1)$"),
+)
+```
+
+Both examples exhibit the same issue, they declare attributes but do not then specify whether they are whitelisted globally or only on specific elements (and which elements). Attributes belong to one or more elements, and the policy needs to declare this.
+
+## Limitations
+
+We are not yet including any tools to help whitelist and sanitize CSS. Which means that unless you wish to do the heavy lifting in a single regular expression (inadvisable), **you should not allow the "style" attribute anywhere**.
+
+It is not the job of bluemonday to fix your bad HTML, it is merely the job of bluemonday to prevent malicious HTML getting through. If you have mismatched HTML elements, or non-conforming nesting of elements, those will remain. But if you have well-structured HTML bluemonday will not break it.
+
+## TODO
+
+* Add support for CSS sanitisation to allow some CSS properties based on a whitelist, possibly using the [Gorilla CSS3 scanner](http://www.gorillatoolkit.org/pkg/css/scanner) - PRs welcome so long as testing covers XSS and demonstrates safety first
+* Investigate whether devs want to blacklist elements and attributes. This would allow devs to take an existing policy (such as the `bluemonday.UGCPolicy()` ) that encapsulates 90% of what they're looking for but does more than they need, and to remove the extra things they do not want to make it 100% what they want
+* Investigate whether devs want a validating HTML mode, in which the HTML elements are not just transformed into a balanced tree (every start tag has a closing tag at the correct depth) but also that elements and character data appear only in their allowed context (i.e. that a `table` element isn't a descendent of a `caption`, that `colgroup`, `thead`, `tbody`, `tfoot` and `tr` are permitted, and that character data is not permitted)
+
+## Development
+
+If you have cloned this repo you will probably need the dependency:
+
+`go get golang.org/x/net/html`
+
+Gophers can use their familiar tools:
+
+`go build`
+
+`go test`
+
+I personally use a Makefile as it spares typing the same args over and over whilst providing consistency for those of us who jump from language to language and enjoy just typing `make` in a project directory and watch magic happen.
+
+`make` will build, vet, test and install the library.
+
+`make clean` will remove the library from a *single* `${GOPATH}/pkg` directory tree
+
+`make test` will run the tests
+
+`make cover` will run the tests and *open a browser window* with the coverage report
+
+`make lint` will run golint (install via `go get github.com/golang/lint/golint`)
+
+## Long term goals
+
+1. Open the code to adversarial peer review similar to the [Attack Review Ground Rules](https://code.google.com/p/owasp-java-html-sanitizer/wiki/AttackReviewGroundRules)
+1. Raise funds and pay for an external security review
diff --git a/vendor/github.com/microcosm-cc/bluemonday/doc.go b/vendor/github.com/microcosm-cc/bluemonday/doc.go
new file mode 100644
index 0000000..71dab60
--- /dev/null
+++ b/vendor/github.com/microcosm-cc/bluemonday/doc.go
@@ -0,0 +1,104 @@
+// Copyright (c) 2014, David Kitchen
+
+
+To pass through mostly unaltered (it gained a rel="nofollow"):
+
+
+
+
+
+The primary purpose of bluemonday is to take potentially unsafe user generated
+content (from things like Markdown, HTML WYSIWYG tools, etc) and make it safe
+for you to put on your website.
+
+It protects sites against XSS (http://en.wikipedia.org/wiki/Cross-site_scripting)
+and other malicious content that a user interface may deliver. There are many
+vectors for an XSS attack (https://www.owasp.org/index.php/XSS_Filter_Evasion_Cheat_Sheet)
+and the safest thing to do is to sanitize user input against a known safe list
+of HTML elements and attributes.
+
+Note: You should always run bluemonday after any other processing.
+
+If you use blackfriday (https://github.com/russross/blackfriday) or
+Pandoc (http://johnmacfarlane.net/pandoc/) then bluemonday should be run after
+these steps. This ensures that no insecure HTML is introduced later in your
+process.
+
+bluemonday is heavily inspired by both the OWASP Java HTML Sanitizer
+(https://code.google.com/p/owasp-java-html-sanitizer/) and the HTML Purifier
+(http://htmlpurifier.org/).
+
+We ship two default policies, one is bluemonday.StrictPolicy() and can be
+thought of as equivalent to stripping all HTML elements and their attributes as
+it has nothing on its whitelist.
+
+The other is bluemonday.UGCPolicy() and allows a broad selection of HTML
+elements and attributes that are safe for user generated content. Note that
+this policy does not whitelist iframes, object, embed, styles, script, etc.
+
+The essence of building a policy is to determine which HTML elements and
+attributes are considered safe for your scenario. OWASP provide an XSS
+prevention cheat sheet ( https://www.google.com/search?q=xss+prevention+cheat+sheet )
+to help explain the risks, but essentially:
+
+ 1. Avoid whitelisting anything other than plain HTML elements
+ 2. Avoid whitelisting `script`, `style`, `iframe`, `object`, `embed`, `base`
+ elements
+ 3. Avoid whitelisting anything other than plain HTML elements with simple
+ values that you can match to a regexp
+*/
+package bluemonday
diff --git a/vendor/github.com/microcosm-cc/bluemonday/go.mod b/vendor/github.com/microcosm-cc/bluemonday/go.mod
new file mode 100644
index 0000000..fa8453c
--- /dev/null
+++ b/vendor/github.com/microcosm-cc/bluemonday/go.mod
@@ -0,0 +1,5 @@
+module github.com/microcosm-cc/bluemonday
+
+go 1.9
+
+require golang.org/x/net v0.0.0-20181220203305-927f97764cc3
diff --git a/vendor/github.com/microcosm-cc/bluemonday/go.sum b/vendor/github.com/microcosm-cc/bluemonday/go.sum
new file mode 100644
index 0000000..bee241d
--- /dev/null
+++ b/vendor/github.com/microcosm-cc/bluemonday/go.sum
@@ -0,0 +1,2 @@
+golang.org/x/net v0.0.0-20181220203305-927f97764cc3 h1:eH6Eip3UpmR+yM/qI9Ijluzb1bNv/cAU/n+6l8tRSis=
+golang.org/x/net v0.0.0-20181220203305-927f97764cc3/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
diff --git a/vendor/github.com/microcosm-cc/bluemonday/helpers.go b/vendor/github.com/microcosm-cc/bluemonday/helpers.go
new file mode 100644
index 0000000..dfa5868
--- /dev/null
+++ b/vendor/github.com/microcosm-cc/bluemonday/helpers.go
@@ -0,0 +1,297 @@
+// Copyright (c) 2014, David Kitchen . To prevent those being removed we maintain
+ // a list of elements that are allowed to have no attributes and that will
+ // be maintained in the output HTML.
+ setOfElementsAllowedWithoutAttrs map[string]struct{}
+
+ setOfElementsToSkipContent map[string]struct{}
+}
+
+type attrPolicy struct {
+
+ // optional pattern to match, when not nil the regexp needs to match
+ // otherwise the attribute is removed
+ regexp *regexp.Regexp
+}
+
+type attrPolicyBuilder struct {
+ p *Policy
+
+ attrNames []string
+ regexp *regexp.Regexp
+ allowEmpty bool
+}
+
+type urlPolicy func(url *url.URL) (allowUrl bool)
+
+// init initializes the maps if this has not been done already
+func (p *Policy) init() {
+ if !p.initialized {
+ p.elsAndAttrs = make(map[string]map[string]attrPolicy)
+ p.globalAttrs = make(map[string]attrPolicy)
+ p.allowURLSchemes = make(map[string]urlPolicy)
+ p.setOfElementsAllowedWithoutAttrs = make(map[string]struct{})
+ p.setOfElementsToSkipContent = make(map[string]struct{})
+ p.initialized = true
+ }
+}
+
+// NewPolicy returns a blank policy with nothing whitelisted or permitted. This
+// is the recommended way to start building a policy and you should now use
+// AllowAttrs() and/or AllowElements() to construct the whitelist of HTML
+// elements and attributes.
+func NewPolicy() *Policy {
+
+ p := Policy{}
+
+ p.addDefaultElementsWithoutAttrs()
+ p.addDefaultSkipElementContent()
+
+ return &p
+}
+
+// AllowAttrs takes a range of HTML attribute names and returns an
+// attribute policy builder that allows you to specify the pattern and scope of
+// the whitelisted attribute.
+//
+// The attribute policy is only added to the core policy when either Globally()
+// or OnElements(...) are called.
+func (p *Policy) AllowAttrs(attrNames ...string) *attrPolicyBuilder {
+
+ p.init()
+
+ abp := attrPolicyBuilder{
+ p: p,
+ allowEmpty: false,
+ }
+
+ for _, attrName := range attrNames {
+ abp.attrNames = append(abp.attrNames, strings.ToLower(attrName))
+ }
+
+ return &abp
+}
+
+// AllowDataAttributes whitelists all data attributes. We can't specify the name
+// of each attribute exactly as they are customized.
+//
+// NOTE: These values are not sanitized and applications that evaluate or process
+// them without checking and verification of the input may be at risk if this option
+// is enabled. This is a 'caveat emptor' option and the person enabling this option
+// needs to fully understand the potential impact with regards to whatever application
+// will be consuming the sanitized HTML afterwards, i.e. if you know you put a link in a
+// data attribute and use that to automatically load some new window then you're giving
+// the author of a HTML fragment the means to open a malicious destination automatically.
+// Use with care!
+func (p *Policy) AllowDataAttributes() {
+ p.allowDataAttributes = true
+}
+
+// AllowNoAttrs says that attributes on element are optional.
+//
+// The attribute policy is only added to the core policy when OnElements(...)
+// are called.
+func (p *Policy) AllowNoAttrs() *attrPolicyBuilder {
+
+ p.init()
+
+ abp := attrPolicyBuilder{
+ p: p,
+ allowEmpty: true,
+ }
+ return &abp
+}
+
+// AllowNoAttrs says that attributes on element are optional.
+//
+// The attribute policy is only added to the core policy when OnElements(...)
+// are called.
+func (abp *attrPolicyBuilder) AllowNoAttrs() *attrPolicyBuilder {
+
+ abp.allowEmpty = true
+
+ return abp
+}
+
+// Matching allows a regular expression to be applied to a nascent attribute
+// policy, and returns the attribute policy. Calling this more than once will
+// replace the existing regexp.
+func (abp *attrPolicyBuilder) Matching(regex *regexp.Regexp) *attrPolicyBuilder {
+
+ abp.regexp = regex
+
+ return abp
+}
+
+// OnElements will bind an attribute policy to a given range of HTML elements
+// and return the updated policy
+func (abp *attrPolicyBuilder) OnElements(elements ...string) *Policy {
+
+ for _, element := range elements {
+ element = strings.ToLower(element)
+
+ for _, attr := range abp.attrNames {
+
+ if _, ok := abp.p.elsAndAttrs[element]; !ok {
+ abp.p.elsAndAttrs[element] = make(map[string]attrPolicy)
+ }
+
+ ap := attrPolicy{}
+ if abp.regexp != nil {
+ ap.regexp = abp.regexp
+ }
+
+ abp.p.elsAndAttrs[element][attr] = ap
+ }
+
+ if abp.allowEmpty {
+ abp.p.setOfElementsAllowedWithoutAttrs[element] = struct{}{}
+
+ if _, ok := abp.p.elsAndAttrs[element]; !ok {
+ abp.p.elsAndAttrs[element] = make(map[string]attrPolicy)
+ }
+ }
+ }
+
+ return abp.p
+}
+
+// Globally will bind an attribute policy to all HTML elements and return the
+// updated policy
+func (abp *attrPolicyBuilder) Globally() *Policy {
+
+ for _, attr := range abp.attrNames {
+ if _, ok := abp.p.globalAttrs[attr]; !ok {
+ abp.p.globalAttrs[attr] = attrPolicy{}
+ }
+
+ ap := attrPolicy{}
+ if abp.regexp != nil {
+ ap.regexp = abp.regexp
+ }
+
+ abp.p.globalAttrs[attr] = ap
+ }
+
+ return abp.p
+}
+
+// AllowElements will append HTML elements to the whitelist without applying an
+// attribute policy to those elements (the elements are permitted
+// sans-attributes)
+func (p *Policy) AllowElements(names ...string) *Policy {
+ p.init()
+
+ for _, element := range names {
+ element = strings.ToLower(element)
+
+ if _, ok := p.elsAndAttrs[element]; !ok {
+ p.elsAndAttrs[element] = make(map[string]attrPolicy)
+ }
+ }
+
+ return p
+}
+
+// RequireNoFollowOnLinks will result in all tags having a rel="nofollow"
+// added to them if one does not already exist
+//
+// Note: This requires p.RequireParseableURLs(true) and will enable it.
+func (p *Policy) RequireNoFollowOnLinks(require bool) *Policy {
+
+ p.requireNoFollow = require
+ p.requireParseableURLs = true
+
+ return p
+}
+
+// RequireNoFollowOnFullyQualifiedLinks will result in all tags that point
+// to a non-local destination (i.e. starts with a protocol and has a host)
+// having a rel="nofollow" added to them if one does not already exist
+//
+// Note: This requires p.RequireParseableURLs(true) and will enable it.
+func (p *Policy) RequireNoFollowOnFullyQualifiedLinks(require bool) *Policy {
+
+ p.requireNoFollowFullyQualifiedLinks = require
+ p.requireParseableURLs = true
+
+ return p
+}
+
+// AddTargetBlankToFullyQualifiedLinks will result in all tags that point
+// to a non-local destination (i.e. starts with a protocol and has a host)
+// having a target="_blank" added to them if one does not already exist
+//
+// Note: This requires p.RequireParseableURLs(true) and will enable it.
+func (p *Policy) AddTargetBlankToFullyQualifiedLinks(require bool) *Policy {
+
+ p.addTargetBlankToFullyQualifiedLinks = require
+ p.requireParseableURLs = true
+
+ return p
+}
+
+// RequireParseableURLs will result in all URLs requiring that they be parseable
+// by "net/url" url.Parse()
+// This applies to:
+// - a.href
+// - area.href
+// - blockquote.cite
+// - img.src
+// - link.href
+// - script.src
+func (p *Policy) RequireParseableURLs(require bool) *Policy {
+
+ p.requireParseableURLs = require
+
+ return p
+}
+
+// AllowRelativeURLs enables RequireParseableURLs and then permits URLs that
+// are parseable, have no schema information and url.IsAbs() returns false
+// This permits local URLs
+func (p *Policy) AllowRelativeURLs(require bool) *Policy {
+
+ p.RequireParseableURLs(true)
+ p.allowRelativeURLs = require
+
+ return p
+}
+
+// AllowURLSchemes will append URL schemes to the whitelist
+// Example: p.AllowURLSchemes("mailto", "http", "https")
+func (p *Policy) AllowURLSchemes(schemes ...string) *Policy {
+ p.init()
+
+ p.RequireParseableURLs(true)
+
+ for _, scheme := range schemes {
+ scheme = strings.ToLower(scheme)
+
+ // Allow all URLs with matching scheme.
+ p.allowURLSchemes[scheme] = nil
+ }
+
+ return p
+}
+
+// AllowURLSchemeWithCustomPolicy will append URL schemes with
+// a custom URL policy to the whitelist.
+// Only the URLs with matching schema and urlPolicy(url)
+// returning true will be allowed.
+func (p *Policy) AllowURLSchemeWithCustomPolicy(
+ scheme string,
+ urlPolicy func(url *url.URL) (allowUrl bool),
+) *Policy {
+
+ p.init()
+
+ p.RequireParseableURLs(true)
+
+ scheme = strings.ToLower(scheme)
+
+ p.allowURLSchemes[scheme] = urlPolicy
+
+ return p
+}
+
+// AddSpaceWhenStrippingTag states whether to add a single space " " when
+// removing tags that are not whitelisted by the policy.
+//
+// This is useful if you expect to strip tags in dense markup and may lose the
+// value of whitespace.
+//
+// For example: "
is valid, but isn't valid as the "dir" attr
+// is mandatory
+func (p *Policy) addDefaultElementsWithoutAttrs() {
+ p.init()
+
+ p.setOfElementsAllowedWithoutAttrs["abbr"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["acronym"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["address"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["article"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["aside"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["audio"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["b"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["bdi"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["blockquote"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["body"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["br"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["button"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["canvas"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["caption"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["center"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["cite"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["code"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["col"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["colgroup"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["datalist"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["dd"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["del"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["details"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["dfn"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["div"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["dl"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["dt"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["em"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["fieldset"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["figcaption"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["figure"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["footer"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["h1"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["h2"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["h3"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["h4"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["h5"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["h6"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["head"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["header"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["hgroup"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["hr"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["html"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["i"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["ins"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["kbd"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["li"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["mark"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["marquee"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["nav"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["ol"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["optgroup"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["option"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["p"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["pre"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["q"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["rp"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["rt"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["ruby"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["s"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["samp"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["script"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["section"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["select"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["small"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["span"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["strike"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["strong"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["style"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["sub"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["summary"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["sup"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["svg"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["table"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["tbody"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["td"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["textarea"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["tfoot"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["th"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["thead"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["title"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["time"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["tr"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["tt"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["u"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["ul"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["var"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["video"] = struct{}{}
+ p.setOfElementsAllowedWithoutAttrs["wbr"] = struct{}{}
+
+}
+
+// addDefaultSkipElementContent adds the HTML elements that we should skip
+// rendering the character content of, if the element itself is not allowed.
+// This is all character data that the end user would not normally see.
+// i.e. if we exclude a tag.
+func (p *Policy) addDefaultSkipElementContent() {
+ p.init()
+
+ p.setOfElementsToSkipContent["frame"] = struct{}{}
+ p.setOfElementsToSkipContent["frameset"] = struct{}{}
+ p.setOfElementsToSkipContent["iframe"] = struct{}{}
+ p.setOfElementsToSkipContent["noembed"] = struct{}{}
+ p.setOfElementsToSkipContent["noframes"] = struct{}{}
+ p.setOfElementsToSkipContent["noscript"] = struct{}{}
+ p.setOfElementsToSkipContent["nostyle"] = struct{}{}
+ p.setOfElementsToSkipContent["object"] = struct{}{}
+ p.setOfElementsToSkipContent["script"] = struct{}{}
+ p.setOfElementsToSkipContent["style"] = struct{}{}
+ p.setOfElementsToSkipContent["title"] = struct{}{}
+}
diff --git a/vendor/github.com/microcosm-cc/bluemonday/sanitize.go b/vendor/github.com/microcosm-cc/bluemonday/sanitize.go
new file mode 100644
index 0000000..65ed89b
--- /dev/null
+++ b/vendor/github.com/microcosm-cc/bluemonday/sanitize.go
@@ -0,0 +1,581 @@
+// Copyright (c) 2014, David Kitchen
AccessDenied
+ abcxxxxxxdef
+ // -> abcxxxdef
+ // e.g: xxxabcdefxxx
+ // -> defxxxabc
+ // Only extract an overlap if it is as big as the edit ahead or behind it.
+ pointer = 1
+ for pointer < len(diffs) {
+ if diffs[pointer-1].Type == DiffDelete &&
+ diffs[pointer].Type == DiffInsert {
+ deletion := diffs[pointer-1].Text
+ insertion := diffs[pointer].Text
+ overlapLength1 := dmp.DiffCommonOverlap(deletion, insertion)
+ overlapLength2 := dmp.DiffCommonOverlap(insertion, deletion)
+ if overlapLength1 >= overlapLength2 {
+ if float64(overlapLength1) >= float64(len(deletion))/2 ||
+ float64(overlapLength1) >= float64(len(insertion))/2 {
+
+ // Overlap found. Insert an equality and trim the surrounding edits.
+ diffs = append(
+ diffs[:pointer],
+ append([]Diff{Diff{DiffEqual, insertion[:overlapLength1]}}, diffs[pointer:]...)...)
+
+ diffs[pointer-1].Text =
+ deletion[0 : len(deletion)-overlapLength1]
+ diffs[pointer+1].Text = insertion[overlapLength1:]
+ pointer++
+ }
+ } else {
+ if float64(overlapLength2) >= float64(len(deletion))/2 ||
+ float64(overlapLength2) >= float64(len(insertion))/2 {
+ // Reverse overlap found. Insert an equality and swap and trim the surrounding edits.
+ overlap := Diff{DiffEqual, deletion[:overlapLength2]}
+ diffs = append(
+ diffs[:pointer],
+ append([]Diff{overlap}, diffs[pointer:]...)...)
+
+ diffs[pointer-1].Type = DiffInsert
+ diffs[pointer-1].Text = insertion[0 : len(insertion)-overlapLength2]
+ diffs[pointer+1].Type = DiffDelete
+ diffs[pointer+1].Text = deletion[overlapLength2:]
+ pointer++
+ }
+ }
+ pointer++
+ }
+ pointer++
+ }
+
+ return diffs
+}
+
+// Define some regex patterns for matching boundaries.
+var (
+ nonAlphaNumericRegex = regexp.MustCompile(`[^a-zA-Z0-9]`)
+ whitespaceRegex = regexp.MustCompile(`\s`)
+ linebreakRegex = regexp.MustCompile(`[\r\n]`)
+ blanklineEndRegex = regexp.MustCompile(`\n\r?\n$`)
+ blanklineStartRegex = regexp.MustCompile(`^\r?\n\r?\n`)
+)
+
+// diffCleanupSemanticScore computes a score representing whether the internal boundary falls on logical boundaries.
+// Scores range from 6 (best) to 0 (worst). Closure, but does not reference any external variables.
+func diffCleanupSemanticScore(one, two string) int {
+ if len(one) == 0 || len(two) == 0 {
+ // Edges are the best.
+ return 6
+ }
+
+ // Each port of this function behaves slightly differently due to subtle differences in each language's definition of things like 'whitespace'. Since this function's purpose is largely cosmetic, the choice has been made to use each language's native features rather than force total conformity.
+ rune1, _ := utf8.DecodeLastRuneInString(one)
+ rune2, _ := utf8.DecodeRuneInString(two)
+ char1 := string(rune1)
+ char2 := string(rune2)
+
+ nonAlphaNumeric1 := nonAlphaNumericRegex.MatchString(char1)
+ nonAlphaNumeric2 := nonAlphaNumericRegex.MatchString(char2)
+ whitespace1 := nonAlphaNumeric1 && whitespaceRegex.MatchString(char1)
+ whitespace2 := nonAlphaNumeric2 && whitespaceRegex.MatchString(char2)
+ lineBreak1 := whitespace1 && linebreakRegex.MatchString(char1)
+ lineBreak2 := whitespace2 && linebreakRegex.MatchString(char2)
+ blankLine1 := lineBreak1 && blanklineEndRegex.MatchString(one)
+ blankLine2 := lineBreak2 && blanklineEndRegex.MatchString(two)
+
+ if blankLine1 || blankLine2 {
+ // Five points for blank lines.
+ return 5
+ } else if lineBreak1 || lineBreak2 {
+ // Four points for line breaks.
+ return 4
+ } else if nonAlphaNumeric1 && !whitespace1 && whitespace2 {
+ // Three points for end of sentences.
+ return 3
+ } else if whitespace1 || whitespace2 {
+ // Two points for whitespace.
+ return 2
+ } else if nonAlphaNumeric1 || nonAlphaNumeric2 {
+ // One point for non-alphanumeric.
+ return 1
+ }
+ return 0
+}
+
+// DiffCleanupSemanticLossless looks for single edits surrounded on both sides by equalities which can be shifted sideways to align the edit to a word boundary.
+// E.g: The cat came. -> The cat came.
+func (dmp *DiffMatchPatch) DiffCleanupSemanticLossless(diffs []Diff) []Diff {
+ pointer := 1
+
+ // Intentionally ignore the first and last element (don't need checking).
+ for pointer < len(diffs)-1 {
+ if diffs[pointer-1].Type == DiffEqual &&
+ diffs[pointer+1].Type == DiffEqual {
+
+ // This is a single edit surrounded by equalities.
+ equality1 := diffs[pointer-1].Text
+ edit := diffs[pointer].Text
+ equality2 := diffs[pointer+1].Text
+
+ // First, shift the edit as far left as possible.
+ commonOffset := dmp.DiffCommonSuffix(equality1, edit)
+ if commonOffset > 0 {
+ commonString := edit[len(edit)-commonOffset:]
+ equality1 = equality1[0 : len(equality1)-commonOffset]
+ edit = commonString + edit[:len(edit)-commonOffset]
+ equality2 = commonString + equality2
+ }
+
+ // Second, step character by character right, looking for the best fit.
+ bestEquality1 := equality1
+ bestEdit := edit
+ bestEquality2 := equality2
+ bestScore := diffCleanupSemanticScore(equality1, edit) +
+ diffCleanupSemanticScore(edit, equality2)
+
+ for len(edit) != 0 && len(equality2) != 0 {
+ _, sz := utf8.DecodeRuneInString(edit)
+ if len(equality2) < sz || edit[:sz] != equality2[:sz] {
+ break
+ }
+ equality1 += edit[:sz]
+ edit = edit[sz:] + equality2[:sz]
+ equality2 = equality2[sz:]
+ score := diffCleanupSemanticScore(equality1, edit) +
+ diffCleanupSemanticScore(edit, equality2)
+ // The >= encourages trailing rather than leading whitespace on edits.
+ if score >= bestScore {
+ bestScore = score
+ bestEquality1 = equality1
+ bestEdit = edit
+ bestEquality2 = equality2
+ }
+ }
+
+ if diffs[pointer-1].Text != bestEquality1 {
+ // We have an improvement, save it back to the diff.
+ if len(bestEquality1) != 0 {
+ diffs[pointer-1].Text = bestEquality1
+ } else {
+ diffs = splice(diffs, pointer-1, 1)
+ pointer--
+ }
+
+ diffs[pointer].Text = bestEdit
+ if len(bestEquality2) != 0 {
+ diffs[pointer+1].Text = bestEquality2
+ } else {
+ diffs = append(diffs[:pointer+1], diffs[pointer+2:]...)
+ pointer--
+ }
+ }
+ }
+ pointer++
+ }
+
+ return diffs
+}
+
+// DiffCleanupEfficiency reduces the number of edits by eliminating operationally trivial equalities.
+func (dmp *DiffMatchPatch) DiffCleanupEfficiency(diffs []Diff) []Diff {
+ changes := false
+ // Stack of indices where equalities are found.
+ type equality struct {
+ data int
+ next *equality
+ }
+ var equalities *equality
+ // Always equal to equalities[equalitiesLength-1][1]
+ lastequality := ""
+ pointer := 0 // Index of current position.
+ // Is there an insertion operation before the last equality.
+ preIns := false
+ // Is there a deletion operation before the last equality.
+ preDel := false
+ // Is there an insertion operation after the last equality.
+ postIns := false
+ // Is there a deletion operation after the last equality.
+ postDel := false
+ for pointer < len(diffs) {
+ if diffs[pointer].Type == DiffEqual { // Equality found.
+ if len(diffs[pointer].Text) < dmp.DiffEditCost &&
+ (postIns || postDel) {
+ // Candidate found.
+ equalities = &equality{
+ data: pointer,
+ next: equalities,
+ }
+ preIns = postIns
+ preDel = postDel
+ lastequality = diffs[pointer].Text
+ } else {
+ // Not a candidate, and can never become one.
+ equalities = nil
+ lastequality = ""
+ }
+ postIns = false
+ postDel = false
+ } else { // An insertion or deletion.
+ if diffs[pointer].Type == DiffDelete {
+ postDel = true
+ } else {
+ postIns = true
+ }
+
+ // Five types to be split:
+ // ABXYCD
+ // AXCD
+ // ABXC
+ // AXCD
+ // ABXC
+ var sumPres int
+ if preIns {
+ sumPres++
+ }
+ if preDel {
+ sumPres++
+ }
+ if postIns {
+ sumPres++
+ }
+ if postDel {
+ sumPres++
+ }
+ if len(lastequality) > 0 &&
+ ((preIns && preDel && postIns && postDel) ||
+ ((len(lastequality) < dmp.DiffEditCost/2) && sumPres == 3)) {
+
+ insPoint := equalities.data
+
+ // Duplicate record.
+ diffs = append(diffs[:insPoint],
+ append([]Diff{Diff{DiffDelete, lastequality}}, diffs[insPoint:]...)...)
+
+ // Change second copy to insert.
+ diffs[insPoint+1].Type = DiffInsert
+ // Throw away the equality we just deleted.
+ equalities = equalities.next
+ lastequality = ""
+
+ if preIns && preDel {
+ // No changes made which could affect previous entry, keep going.
+ postIns = true
+ postDel = true
+ equalities = nil
+ } else {
+ if equalities != nil {
+ equalities = equalities.next
+ }
+ if equalities != nil {
+ pointer = equalities.data
+ } else {
+ pointer = -1
+ }
+ postIns = false
+ postDel = false
+ }
+ changes = true
+ }
+ }
+ pointer++
+ }
+
+ if changes {
+ diffs = dmp.DiffCleanupMerge(diffs)
+ }
+
+ return diffs
+}
+
+// DiffCleanupMerge reorders and merges like edit sections. Merge equalities.
+// Any edit section can move as long as it doesn't cross an equality.
+func (dmp *DiffMatchPatch) DiffCleanupMerge(diffs []Diff) []Diff {
+ // Add a dummy entry at the end.
+ diffs = append(diffs, Diff{DiffEqual, ""})
+ pointer := 0
+ countDelete := 0
+ countInsert := 0
+ commonlength := 0
+ textDelete := []rune(nil)
+ textInsert := []rune(nil)
+
+ for pointer < len(diffs) {
+ switch diffs[pointer].Type {
+ case DiffInsert:
+ countInsert++
+ textInsert = append(textInsert, []rune(diffs[pointer].Text)...)
+ pointer++
+ break
+ case DiffDelete:
+ countDelete++
+ textDelete = append(textDelete, []rune(diffs[pointer].Text)...)
+ pointer++
+ break
+ case DiffEqual:
+ // Upon reaching an equality, check for prior redundancies.
+ if countDelete+countInsert > 1 {
+ if countDelete != 0 && countInsert != 0 {
+ // Factor out any common prefixies.
+ commonlength = commonPrefixLength(textInsert, textDelete)
+ if commonlength != 0 {
+ x := pointer - countDelete - countInsert
+ if x > 0 && diffs[x-1].Type == DiffEqual {
+ diffs[x-1].Text += string(textInsert[:commonlength])
+ } else {
+ diffs = append([]Diff{Diff{DiffEqual, string(textInsert[:commonlength])}}, diffs...)
+ pointer++
+ }
+ textInsert = textInsert[commonlength:]
+ textDelete = textDelete[commonlength:]
+ }
+ // Factor out any common suffixies.
+ commonlength = commonSuffixLength(textInsert, textDelete)
+ if commonlength != 0 {
+ insertIndex := len(textInsert) - commonlength
+ deleteIndex := len(textDelete) - commonlength
+ diffs[pointer].Text = string(textInsert[insertIndex:]) + diffs[pointer].Text
+ textInsert = textInsert[:insertIndex]
+ textDelete = textDelete[:deleteIndex]
+ }
+ }
+ // Delete the offending records and add the merged ones.
+ if countDelete == 0 {
+ diffs = splice(diffs, pointer-countInsert,
+ countDelete+countInsert,
+ Diff{DiffInsert, string(textInsert)})
+ } else if countInsert == 0 {
+ diffs = splice(diffs, pointer-countDelete,
+ countDelete+countInsert,
+ Diff{DiffDelete, string(textDelete)})
+ } else {
+ diffs = splice(diffs, pointer-countDelete-countInsert,
+ countDelete+countInsert,
+ Diff{DiffDelete, string(textDelete)},
+ Diff{DiffInsert, string(textInsert)})
+ }
+
+ pointer = pointer - countDelete - countInsert + 1
+ if countDelete != 0 {
+ pointer++
+ }
+ if countInsert != 0 {
+ pointer++
+ }
+ } else if pointer != 0 && diffs[pointer-1].Type == DiffEqual {
+ // Merge this equality with the previous one.
+ diffs[pointer-1].Text += diffs[pointer].Text
+ diffs = append(diffs[:pointer], diffs[pointer+1:]...)
+ } else {
+ pointer++
+ }
+ countInsert = 0
+ countDelete = 0
+ textDelete = nil
+ textInsert = nil
+ break
+ }
+ }
+
+ if len(diffs[len(diffs)-1].Text) == 0 {
+ diffs = diffs[0 : len(diffs)-1] // Remove the dummy entry at the end.
+ }
+
+ // Second pass: look for single edits surrounded on both sides by equalities which can be shifted sideways to eliminate an equality. E.g: ABAC -> ABAC
+ changes := false
+ pointer = 1
+ // Intentionally ignore the first and last element (don't need checking).
+ for pointer < (len(diffs) - 1) {
+ if diffs[pointer-1].Type == DiffEqual &&
+ diffs[pointer+1].Type == DiffEqual {
+ // This is a single edit surrounded by equalities.
+ if strings.HasSuffix(diffs[pointer].Text, diffs[pointer-1].Text) {
+ // Shift the edit over the previous equality.
+ diffs[pointer].Text = diffs[pointer-1].Text +
+ diffs[pointer].Text[:len(diffs[pointer].Text)-len(diffs[pointer-1].Text)]
+ diffs[pointer+1].Text = diffs[pointer-1].Text + diffs[pointer+1].Text
+ diffs = splice(diffs, pointer-1, 1)
+ changes = true
+ } else if strings.HasPrefix(diffs[pointer].Text, diffs[pointer+1].Text) {
+ // Shift the edit over the next equality.
+ diffs[pointer-1].Text += diffs[pointer+1].Text
+ diffs[pointer].Text =
+ diffs[pointer].Text[len(diffs[pointer+1].Text):] + diffs[pointer+1].Text
+ diffs = splice(diffs, pointer+1, 1)
+ changes = true
+ }
+ }
+ pointer++
+ }
+
+ // If shifts were made, the diff needs reordering and another shift sweep.
+ if changes {
+ diffs = dmp.DiffCleanupMerge(diffs)
+ }
+
+ return diffs
+}
+
+// DiffXIndex returns the equivalent location in s2.
+func (dmp *DiffMatchPatch) DiffXIndex(diffs []Diff, loc int) int {
+ chars1 := 0
+ chars2 := 0
+ lastChars1 := 0
+ lastChars2 := 0
+ lastDiff := Diff{}
+ for i := 0; i < len(diffs); i++ {
+ aDiff := diffs[i]
+ if aDiff.Type != DiffInsert {
+ // Equality or deletion.
+ chars1 += len(aDiff.Text)
+ }
+ if aDiff.Type != DiffDelete {
+ // Equality or insertion.
+ chars2 += len(aDiff.Text)
+ }
+ if chars1 > loc {
+ // Overshot the location.
+ lastDiff = aDiff
+ break
+ }
+ lastChars1 = chars1
+ lastChars2 = chars2
+ }
+ if lastDiff.Type == DiffDelete {
+ // The location was deleted.
+ return lastChars2
+ }
+ // Add the remaining character length.
+ return lastChars2 + (loc - lastChars1)
+}
+
+// DiffPrettyHtml converts a []Diff into a pretty HTML report.
+// It is intended as an example from which to write one's own display functions.
+func (dmp *DiffMatchPatch) DiffPrettyHtml(diffs []Diff) string {
+ var buff bytes.Buffer
+ for _, diff := range diffs {
+ text := strings.Replace(html.EscapeString(diff.Text), "\n", "¶
", -1)
+ switch diff.Type {
+ case DiffInsert:
+ _, _ = buff.WriteString("")
+ _, _ = buff.WriteString(text)
+ _, _ = buff.WriteString("")
+ case DiffDelete:
+ _, _ = buff.WriteString("")
+ _, _ = buff.WriteString(text)
+ _, _ = buff.WriteString("")
+ case DiffEqual:
+ _, _ = buff.WriteString("")
+ _, _ = buff.WriteString(text)
+ _, _ = buff.WriteString("")
+ }
+ }
+ return buff.String()
+}
+
+// DiffPrettyText converts a []Diff into a colored text report.
+func (dmp *DiffMatchPatch) DiffPrettyText(diffs []Diff) string {
+ var buff bytes.Buffer
+ for _, diff := range diffs {
+ text := diff.Text
+
+ switch diff.Type {
+ case DiffInsert:
+ _, _ = buff.WriteString("\x1b[32m")
+ _, _ = buff.WriteString(text)
+ _, _ = buff.WriteString("\x1b[0m")
+ case DiffDelete:
+ _, _ = buff.WriteString("\x1b[31m")
+ _, _ = buff.WriteString(text)
+ _, _ = buff.WriteString("\x1b[0m")
+ case DiffEqual:
+ _, _ = buff.WriteString(text)
+ }
+ }
+
+ return buff.String()
+}
+
+// DiffText1 computes and returns the source text (all equalities and deletions).
+func (dmp *DiffMatchPatch) DiffText1(diffs []Diff) string {
+ //StringBuilder text = new StringBuilder()
+ var text bytes.Buffer
+
+ for _, aDiff := range diffs {
+ if aDiff.Type != DiffInsert {
+ _, _ = text.WriteString(aDiff.Text)
+ }
+ }
+ return text.String()
+}
+
+// DiffText2 computes and returns the destination text (all equalities and insertions).
+func (dmp *DiffMatchPatch) DiffText2(diffs []Diff) string {
+ var text bytes.Buffer
+
+ for _, aDiff := range diffs {
+ if aDiff.Type != DiffDelete {
+ _, _ = text.WriteString(aDiff.Text)
+ }
+ }
+ return text.String()
+}
+
+// DiffLevenshtein computes the Levenshtein distance that is the number of inserted, deleted or substituted characters.
+func (dmp *DiffMatchPatch) DiffLevenshtein(diffs []Diff) int {
+ levenshtein := 0
+ insertions := 0
+ deletions := 0
+
+ for _, aDiff := range diffs {
+ switch aDiff.Type {
+ case DiffInsert:
+ insertions += len(aDiff.Text)
+ case DiffDelete:
+ deletions += len(aDiff.Text)
+ case DiffEqual:
+ // A deletion and an insertion is one substitution.
+ levenshtein += max(insertions, deletions)
+ insertions = 0
+ deletions = 0
+ }
+ }
+
+ levenshtein += max(insertions, deletions)
+ return levenshtein
+}
+
+// DiffToDelta crushes the diff into an encoded string which describes the operations required to transform text1 into text2.
+// E.g. =3\t-2\t+ing -> Keep 3 chars, delete 2 chars, insert 'ing'. Operations are tab-separated. Inserted text is escaped using %xx notation.
+func (dmp *DiffMatchPatch) DiffToDelta(diffs []Diff) string {
+ var text bytes.Buffer
+ for _, aDiff := range diffs {
+ switch aDiff.Type {
+ case DiffInsert:
+ _, _ = text.WriteString("+")
+ _, _ = text.WriteString(strings.Replace(url.QueryEscape(aDiff.Text), "+", " ", -1))
+ _, _ = text.WriteString("\t")
+ break
+ case DiffDelete:
+ _, _ = text.WriteString("-")
+ _, _ = text.WriteString(strconv.Itoa(utf8.RuneCountInString(aDiff.Text)))
+ _, _ = text.WriteString("\t")
+ break
+ case DiffEqual:
+ _, _ = text.WriteString("=")
+ _, _ = text.WriteString(strconv.Itoa(utf8.RuneCountInString(aDiff.Text)))
+ _, _ = text.WriteString("\t")
+ break
+ }
+ }
+ delta := text.String()
+ if len(delta) != 0 {
+ // Strip off trailing tab character.
+ delta = delta[0 : utf8.RuneCountInString(delta)-1]
+ delta = unescaper.Replace(delta)
+ }
+ return delta
+}
+
+// DiffFromDelta given the original text1, and an encoded string which describes the operations required to transform text1 into text2, comAdde the full diff.
+func (dmp *DiffMatchPatch) DiffFromDelta(text1 string, delta string) (diffs []Diff, err error) {
+ i := 0
+ runes := []rune(text1)
+
+ for _, token := range strings.Split(delta, "\t") {
+ if len(token) == 0 {
+ // Blank tokens are ok (from a trailing \t).
+ continue
+ }
+
+ // Each token begins with a one character parameter which specifies the operation of this token (delete, insert, equality).
+ param := token[1:]
+
+ switch op := token[0]; op {
+ case '+':
+ // Decode would Diff all "+" to " "
+ param = strings.Replace(param, "+", "%2b", -1)
+ param, err = url.QueryUnescape(param)
+ if err != nil {
+ return nil, err
+ }
+ if !utf8.ValidString(param) {
+ return nil, fmt.Errorf("invalid UTF-8 token: %q", param)
+ }
+
+ diffs = append(diffs, Diff{DiffInsert, param})
+ case '=', '-':
+ n, err := strconv.ParseInt(param, 10, 0)
+ if err != nil {
+ return nil, err
+ } else if n < 0 {
+ return nil, errors.New("Negative number in DiffFromDelta: " + param)
+ }
+
+ i += int(n)
+ // Break out if we are out of bounds, go1.6 can't handle this very well
+ if i > len(runes) {
+ break
+ }
+ // Remember that string slicing is by byte - we want by rune here.
+ text := string(runes[i-int(n) : i])
+
+ if op == '=' {
+ diffs = append(diffs, Diff{DiffEqual, text})
+ } else {
+ diffs = append(diffs, Diff{DiffDelete, text})
+ }
+ default:
+ // Anything else is an error.
+ return nil, errors.New("Invalid diff operation in DiffFromDelta: " + string(token[0]))
+ }
+ }
+
+ if i != len(runes) {
+ return nil, fmt.Errorf("Delta length (%v) is different from source text length (%v)", i, len(text1))
+ }
+
+ return diffs, nil
+}
diff --git a/vendor/github.com/sergi/go-diff/diffmatchpatch/diffmatchpatch.go b/vendor/github.com/sergi/go-diff/diffmatchpatch/diffmatchpatch.go
new file mode 100644
index 0000000..d3acc32
--- /dev/null
+++ b/vendor/github.com/sergi/go-diff/diffmatchpatch/diffmatchpatch.go
@@ -0,0 +1,46 @@
+// Copyright (c) 2012-2016 The go-diff authors. All rights reserved.
+// https://github.com/sergi/go-diff
+// See the included LICENSE file for license details.
+//
+// go-diff is a Go implementation of Google's Diff, Match, and Patch library
+// Original library is Copyright (c) 2006 Google Inc.
+// http://code.google.com/p/google-diff-match-patch/
+
+// Package diffmatchpatch offers robust algorithms to perform the operations required for synchronizing plain text.
+package diffmatchpatch
+
+import (
+ "time"
+)
+
+// DiffMatchPatch holds the configuration for diff-match-patch operations.
+type DiffMatchPatch struct {
+ // Number of seconds to map a diff before giving up (0 for infinity).
+ DiffTimeout time.Duration
+ // Cost of an empty edit operation in terms of edit characters.
+ DiffEditCost int
+ // How far to search for a match (0 = exact location, 1000+ = broad match). A match this many characters away from the expected location will add 1.0 to the score (0.0 is a perfect match).
+ MatchDistance int
+ // When deleting a large block of text (over ~64 characters), how close do the contents have to be to match the expected contents. (0.0 = perfection, 1.0 = very loose). Note that MatchThreshold controls how closely the end points of a delete need to match.
+ PatchDeleteThreshold float64
+ // Chunk size for context length.
+ PatchMargin int
+ // The number of bits in an int.
+ MatchMaxBits int
+ // At what point is no match declared (0.0 = perfection, 1.0 = very loose).
+ MatchThreshold float64
+}
+
+// New creates a new DiffMatchPatch object with default parameters.
+func New() *DiffMatchPatch {
+ // Defaults.
+ return &DiffMatchPatch{
+ DiffTimeout: time.Second,
+ DiffEditCost: 4,
+ MatchThreshold: 0.5,
+ MatchDistance: 1000,
+ PatchDeleteThreshold: 0.5,
+ PatchMargin: 4,
+ MatchMaxBits: 32,
+ }
+}
diff --git a/vendor/github.com/sergi/go-diff/diffmatchpatch/match.go b/vendor/github.com/sergi/go-diff/diffmatchpatch/match.go
new file mode 100644
index 0000000..17374e1
--- /dev/null
+++ b/vendor/github.com/sergi/go-diff/diffmatchpatch/match.go
@@ -0,0 +1,160 @@
+// Copyright (c) 2012-2016 The go-diff authors. All rights reserved.
+// https://github.com/sergi/go-diff
+// See the included LICENSE file for license details.
+//
+// go-diff is a Go implementation of Google's Diff, Match, and Patch library
+// Original library is Copyright (c) 2006 Google Inc.
+// http://code.google.com/p/google-diff-match-patch/
+
+package diffmatchpatch
+
+import (
+ "math"
+)
+
+// MatchMain locates the best instance of 'pattern' in 'text' near 'loc'.
+// Returns -1 if no match found.
+func (dmp *DiffMatchPatch) MatchMain(text, pattern string, loc int) int {
+ // Check for null inputs not needed since null can't be passed in C#.
+
+ loc = int(math.Max(0, math.Min(float64(loc), float64(len(text)))))
+ if text == pattern {
+ // Shortcut (potentially not guaranteed by the algorithm)
+ return 0
+ } else if len(text) == 0 {
+ // Nothing to match.
+ return -1
+ } else if loc+len(pattern) <= len(text) && text[loc:loc+len(pattern)] == pattern {
+ // Perfect match at the perfect spot! (Includes case of null pattern)
+ return loc
+ }
+ // Do a fuzzy compare.
+ return dmp.MatchBitap(text, pattern, loc)
+}
+
+// MatchBitap locates the best instance of 'pattern' in 'text' near 'loc' using the Bitap algorithm.
+// Returns -1 if no match was found.
+func (dmp *DiffMatchPatch) MatchBitap(text, pattern string, loc int) int {
+ // Initialise the alphabet.
+ s := dmp.MatchAlphabet(pattern)
+
+ // Highest score beyond which we give up.
+ scoreThreshold := dmp.MatchThreshold
+ // Is there a nearby exact match? (speedup)
+ bestLoc := indexOf(text, pattern, loc)
+ if bestLoc != -1 {
+ scoreThreshold = math.Min(dmp.matchBitapScore(0, bestLoc, loc,
+ pattern), scoreThreshold)
+ // What about in the other direction? (speedup)
+ bestLoc = lastIndexOf(text, pattern, loc+len(pattern))
+ if bestLoc != -1 {
+ scoreThreshold = math.Min(dmp.matchBitapScore(0, bestLoc, loc,
+ pattern), scoreThreshold)
+ }
+ }
+
+ // Initialise the bit arrays.
+ matchmask := 1 << uint((len(pattern) - 1))
+ bestLoc = -1
+
+ var binMin, binMid int
+ binMax := len(pattern) + len(text)
+ lastRd := []int{}
+ for d := 0; d < len(pattern); d++ {
+ // Scan for the best match; each iteration allows for one more error. Run a binary search to determine how far from 'loc' we can stray at this error level.
+ binMin = 0
+ binMid = binMax
+ for binMin < binMid {
+ if dmp.matchBitapScore(d, loc+binMid, loc, pattern) <= scoreThreshold {
+ binMin = binMid
+ } else {
+ binMax = binMid
+ }
+ binMid = (binMax-binMin)/2 + binMin
+ }
+ // Use the result from this iteration as the maximum for the next.
+ binMax = binMid
+ start := int(math.Max(1, float64(loc-binMid+1)))
+ finish := int(math.Min(float64(loc+binMid), float64(len(text))) + float64(len(pattern)))
+
+ rd := make([]int, finish+2)
+ rd[finish+1] = (1 << uint(d)) - 1
+
+ for j := finish; j >= start; j-- {
+ var charMatch int
+ if len(text) <= j-1 {
+ // Out of range.
+ charMatch = 0
+ } else if _, ok := s[text[j-1]]; !ok {
+ charMatch = 0
+ } else {
+ charMatch = s[text[j-1]]
+ }
+
+ if d == 0 {
+ // First pass: exact match.
+ rd[j] = ((rd[j+1] << 1) | 1) & charMatch
+ } else {
+ // Subsequent passes: fuzzy match.
+ rd[j] = ((rd[j+1]<<1)|1)&charMatch | (((lastRd[j+1] | lastRd[j]) << 1) | 1) | lastRd[j+1]
+ }
+ if (rd[j] & matchmask) != 0 {
+ score := dmp.matchBitapScore(d, j-1, loc, pattern)
+ // This match will almost certainly be better than any existing match. But check anyway.
+ if score <= scoreThreshold {
+ // Told you so.
+ scoreThreshold = score
+ bestLoc = j - 1
+ if bestLoc > loc {
+ // When passing loc, don't exceed our current distance from loc.
+ start = int(math.Max(1, float64(2*loc-bestLoc)))
+ } else {
+ // Already passed loc, downhill from here on in.
+ break
+ }
+ }
+ }
+ }
+ if dmp.matchBitapScore(d+1, loc, loc, pattern) > scoreThreshold {
+ // No hope for a (better) match at greater error levels.
+ break
+ }
+ lastRd = rd
+ }
+ return bestLoc
+}
+
+// matchBitapScore computes and returns the score for a match with e errors and x location.
+func (dmp *DiffMatchPatch) matchBitapScore(e, x, loc int, pattern string) float64 {
+ accuracy := float64(e) / float64(len(pattern))
+ proximity := math.Abs(float64(loc - x))
+ if dmp.MatchDistance == 0 {
+ // Dodge divide by zero error.
+ if proximity == 0 {
+ return accuracy
+ }
+
+ return 1.0
+ }
+ return accuracy + (proximity / float64(dmp.MatchDistance))
+}
+
+// MatchAlphabet initialises the alphabet for the Bitap algorithm.
+func (dmp *DiffMatchPatch) MatchAlphabet(pattern string) map[byte]int {
+ s := map[byte]int{}
+ charPattern := []byte(pattern)
+ for _, c := range charPattern {
+ _, ok := s[c]
+ if !ok {
+ s[c] = 0
+ }
+ }
+ i := 0
+
+ for _, c := range charPattern {
+ value := s[c] | int(uint(1)<...
`-wrapped list to display line numbers.
+
+```go
+package syntaxhighlight_test
+
+import (
+ "fmt"
+ "os"
+
+ "github.com/sourcegraph/syntaxhighlight"
+)
+
+func Example() {
+ src := []byte(`
+/* hello, world! */
+var a = 3;
+
+// b is a cool function
+function b() {
+ return 7;
+}`)
+
+ highlighted, err := syntaxhighlight.AsHTML(src)
+ if err != nil {
+ fmt.Println(err)
+ os.Exit(1)
+ }
+
+ fmt.Println(string(highlighted))
+
+ // Output:
+ // /* hello, world! */
+ // var a = 3;
+ //
+ // // b is a cool function
+ // function b() {
+ // return 7;
+ // }
+}
+```
+
+## Contributors
+
+* [Quinn Slack](https://sourcegraph.com/sqs)
+
+Contributions are welcome! Submit a pull request on GitHub.
diff --git a/vendor/github.com/sourcegraph/syntaxhighlight/benchmark.txt b/vendor/github.com/sourcegraph/syntaxhighlight/benchmark.txt
new file mode 100644
index 0000000..2923a0f
--- /dev/null
+++ b/vendor/github.com/sourcegraph/syntaxhighlight/benchmark.txt
@@ -0,0 +1,3 @@
+PASS
+BenchmarkAnnotate-12 20000 80528 ns/op 18192 B/op 428 allocs/op
+ok github.com/sourcegraph/syntaxhighlight 2.424s
diff --git a/vendor/github.com/sourcegraph/syntaxhighlight/highlight.go b/vendor/github.com/sourcegraph/syntaxhighlight/highlight.go
new file mode 100644
index 0000000..9ff794c
--- /dev/null
+++ b/vendor/github.com/sourcegraph/syntaxhighlight/highlight.go
@@ -0,0 +1,295 @@
+// Package syntaxhighlight provides syntax highlighting for code. It currently
+// uses a language-independent lexer and performs decently on JavaScript, Java,
+// Ruby, Python, Go, and C.
+package syntaxhighlight
+
+import (
+ "bytes"
+ "io"
+ "strings"
+ "text/scanner"
+ "text/template"
+ "unicode"
+ "unicode/utf8"
+
+ "github.com/sourcegraph/annotate"
+)
+
+// Kind represents a syntax highlighting kind (class) which will be assigned to tokens.
+// A syntax highlighting scheme (style) maps text style properties to each token kind.
+type Kind uint8
+
+// A set of supported highlighting kinds
+const (
+ Whitespace Kind = iota
+ String
+ Keyword
+ Comment
+ Type
+ Literal
+ Punctuation
+ Plaintext
+ Tag
+ HTMLTag
+ HTMLAttrName
+ HTMLAttrValue
+ Decimal
+)
+
+//go:generate gostringer -type=Kind
+
+// Printer implements an interface to render highlighted output
+// (see HTMLPrinter for the implementation of this interface)
+type Printer interface {
+ Print(w io.Writer, kind Kind, tokText string) error
+}
+
+// HTMLConfig holds the HTML class configuration to be used by annotators when
+// highlighting code.
+type HTMLConfig struct {
+ String string
+ Keyword string
+ Comment string
+ Type string
+ Literal string
+ Punctuation string
+ Plaintext string
+ Tag string
+ HTMLTag string
+ HTMLAttrName string
+ HTMLAttrValue string
+ Decimal string
+ Whitespace string
+
+ AsOrderedList bool
+}
+
+// HTMLPrinter implements Printer interface and is used to produce
+// HTML-based highligher
+type HTMLPrinter HTMLConfig
+
+// Class returns the set class for a given token Kind.
+func (c HTMLConfig) Class(kind Kind) string {
+ switch kind {
+ case String:
+ return c.String
+ case Keyword:
+ return c.Keyword
+ case Comment:
+ return c.Comment
+ case Type:
+ return c.Type
+ case Literal:
+ return c.Literal
+ case Punctuation:
+ return c.Punctuation
+ case Plaintext:
+ return c.Plaintext
+ case Tag:
+ return c.Tag
+ case HTMLTag:
+ return c.HTMLTag
+ case HTMLAttrName:
+ return c.HTMLAttrName
+ case HTMLAttrValue:
+ return c.HTMLAttrValue
+ case Decimal:
+ return c.Decimal
+ }
+ return ""
+}
+
+// Print is the function that emits highlighted source code using
+// ... wrapper tags
+func (p HTMLPrinter) Print(w io.Writer, kind Kind, tokText string) error {
+ if p.AsOrderedList {
+ if i := strings.Index(tokText, "\n"); i > -1 {
+ if err := p.Print(w, kind, tokText[:i]); err != nil {
+ return err
+ }
+ w.Write([]byte("\n\n
"))
+ }
+ if err != nil {
+ return nil, err
+ }
+ return buf.Bytes(), nil
+}
+
+// NewScanner is a helper that takes a []byte src, wraps it in a reader and creates a Scanner.
+func NewScanner(src []byte) *scanner.Scanner {
+ return NewScannerReader(bytes.NewReader(src))
+}
+
+// NewScannerReader takes a reader src and creates a Scanner.
+func NewScannerReader(src io.Reader) *scanner.Scanner {
+ var s scanner.Scanner
+ s.Init(src)
+ s.Error = func(_ *scanner.Scanner, _ string) {}
+ s.Whitespace = 0
+ s.Mode = s.Mode ^ scanner.SkipComments
+ return &s
+}
+
+func tokenKind(tok rune, tokText string) Kind {
+ switch tok {
+ case scanner.Ident:
+ if _, isKW := keywords[tokText]; isKW {
+ return Keyword
+ }
+ if r, _ := utf8.DecodeRuneInString(tokText); unicode.IsUpper(r) {
+ return Type
+ }
+ return Plaintext
+ case scanner.Float, scanner.Int:
+ return Decimal
+ case scanner.Char, scanner.String, scanner.RawString:
+ return String
+ case scanner.Comment:
+ return Comment
+ }
+ if unicode.IsSpace(tok) {
+ return Whitespace
+ }
+ return Punctuation
+}
diff --git a/vendor/github.com/sourcegraph/syntaxhighlight/keywords.go b/vendor/github.com/sourcegraph/syntaxhighlight/keywords.go
new file mode 100644
index 0000000..85729d6
--- /dev/null
+++ b/vendor/github.com/sourcegraph/syntaxhighlight/keywords.go
@@ -0,0 +1,177 @@
+package syntaxhighlight
+
+var keywords = map[string]struct{}{
+ "BEGIN": {},
+ "END": {},
+ "False": {},
+ "Infinity": {},
+ "NaN": {},
+ "None": {},
+ "True": {},
+ "abstract": {},
+ "alias": {},
+ "align_union": {},
+ "alignof": {},
+ "and": {},
+ "append": {},
+ "as": {},
+ "asm": {},
+ "assert": {},
+ "auto": {},
+ "axiom": {},
+ "begin": {},
+ "bool": {},
+ "boolean": {},
+ "break": {},
+ "byte": {},
+ "caller": {},
+ "case": {},
+ "catch": {},
+ "char": {},
+ "class": {},
+ "concept": {},
+ "concept_map": {},
+ "const": {},
+ "const_cast": {},
+ "constexpr": {},
+ "continue": {},
+ "debugger": {},
+ "decltype": {},
+ "def": {},
+ "default": {},
+ "defined": {},
+ "del": {},
+ "delegate": {},
+ "delete": {},
+ "die": {},
+ "do": {},
+ "double": {},
+ "dump": {},
+ "dynamic_cast": {},
+ "elif": {},
+ "else": {},
+ "elsif": {},
+ "end": {},
+ "ensure": {},
+ "enum": {},
+ "eval": {},
+ "except": {},
+ "exec": {},
+ "exit": {},
+ "explicit": {},
+ "export": {},
+ "extends": {},
+ "extern": {},
+ "false": {},
+ "final": {},
+ "finally": {},
+ "float": {},
+ "float32": {},
+ "float64": {},
+ "for": {},
+ "foreach": {},
+ "friend": {},
+ "from": {},
+ "func": {},
+ "function": {},
+ "generic": {},
+ "get": {},
+ "global": {},
+ "goto": {},
+ "if": {},
+ "implements": {},
+ "import": {},
+ "in": {},
+ "inline": {},
+ "instanceof": {},
+ "int": {},
+ "int8": {},
+ "int16": {},
+ "int32": {},
+ "int64": {},
+ "interface": {},
+ "is": {},
+ "lambda": {},
+ "last": {},
+ "late_check": {},
+ "local": {},
+ "long": {},
+ "make": {},
+ "map": {},
+ "module": {},
+ "mutable": {},
+ "my": {},
+ "namespace": {},
+ "native": {},
+ "new": {},
+ "next": {},
+ "nil": {},
+ "no": {},
+ "nonlocal": {},
+ "not": {},
+ "null": {},
+ "nullptr": {},
+ "operator": {},
+ "or": {},
+ "our": {},
+ "package": {},
+ "pass": {},
+ "print": {},
+ "private": {},
+ "property": {},
+ "protected": {},
+ "public": {},
+ "raise": {},
+ "redo": {},
+ "register": {},
+ "reinterpret_cast": {},
+ "require": {},
+ "rescue": {},
+ "retry": {},
+ "return": {},
+ "self": {},
+ "set": {},
+ "short": {},
+ "signed": {},
+ "sizeof": {},
+ "static": {},
+ "static_assert": {},
+ "static_cast": {},
+ "strictfp": {},
+ "struct": {},
+ "sub": {},
+ "super": {},
+ "switch": {},
+ "synchronized": {},
+ "template": {},
+ "then": {},
+ "this": {},
+ "throw": {},
+ "throws": {},
+ "transient": {},
+ "true": {},
+ "try": {},
+ "type": {},
+ "typedef": {},
+ "typeid": {},
+ "typename": {},
+ "typeof": {},
+ "undef": {},
+ "undefined": {},
+ "union": {},
+ "unless": {},
+ "unsigned": {},
+ "until": {},
+ "use": {},
+ "using": {},
+ "var": {},
+ "virtual": {},
+ "void": {},
+ "volatile": {},
+ "wantarray": {},
+ "when": {},
+ "where": {},
+ "while": {},
+ "with": {},
+ "yield": {},
+}
diff --git a/vendor/github.com/sourcegraph/syntaxhighlight/kind_gostring.go b/vendor/github.com/sourcegraph/syntaxhighlight/kind_gostring.go
new file mode 100644
index 0000000..2fc73ec
--- /dev/null
+++ b/vendor/github.com/sourcegraph/syntaxhighlight/kind_gostring.go
@@ -0,0 +1,16 @@
+// generated by gostringer -type=Kind; DO NOT EDIT
+
+package syntaxhighlight
+
+import "fmt"
+
+const _Kind_name = "WhitespaceStringKeywordCommentTypeLiteralPunctuationPlaintextTagHTMLTagHTMLAttrNameHTMLAttrValueDecimal"
+
+var _Kind_index = [...]uint8{0, 10, 16, 23, 30, 34, 41, 52, 61, 64, 71, 83, 96, 103}
+
+func (i Kind) GoString() string {
+ if i+1 >= Kind(len(_Kind_index)) {
+ return fmt.Sprintf("syntaxhighlight.Kind(%d)", i)
+ }
+ return "syntaxhighlight." + _Kind_name[_Kind_index[i]:_Kind_index[i+1]]
+}
diff --git a/vendor/github.com/sqlitebrowser/blackfriday/.gitignore b/vendor/github.com/sqlitebrowser/blackfriday/.gitignore
new file mode 100644
index 0000000..75623dc
--- /dev/null
+++ b/vendor/github.com/sqlitebrowser/blackfriday/.gitignore
@@ -0,0 +1,8 @@
+*.out
+*.swp
+*.8
+*.6
+_obj
+_test*
+markdown
+tags
diff --git a/vendor/github.com/sqlitebrowser/blackfriday/.travis.yml b/vendor/github.com/sqlitebrowser/blackfriday/.travis.yml
new file mode 100644
index 0000000..a1687f1
--- /dev/null
+++ b/vendor/github.com/sqlitebrowser/blackfriday/.travis.yml
@@ -0,0 +1,30 @@
+sudo: false
+language: go
+go:
+ - 1.5.4
+ - 1.6.2
+ - tip
+matrix:
+ include:
+ - go: 1.2.2
+ script:
+ - go get -t -v ./...
+ - go test -v -race ./...
+ - go: 1.3.3
+ script:
+ - go get -t -v ./...
+ - go test -v -race ./...
+ - go: 1.4.3
+ script:
+ - go get -t -v ./...
+ - go test -v -race ./...
+ allow_failures:
+ - go: tip
+ fast_finish: true
+install:
+ - # Do nothing. This is needed to prevent default install action "go get -t -v ./..." from happening here (we want it to happen inside script step).
+script:
+ - go get -t -v ./...
+ - diff -u <(echo -n) <(gofmt -d -s .)
+ - go tool vet .
+ - go test -v -race ./...
diff --git a/vendor/github.com/sqlitebrowser/blackfriday/LICENSE.txt b/vendor/github.com/sqlitebrowser/blackfriday/LICENSE.txt
new file mode 100644
index 0000000..2885af3
--- /dev/null
+++ b/vendor/github.com/sqlitebrowser/blackfriday/LICENSE.txt
@@ -0,0 +1,29 @@
+Blackfriday is distributed under the Simplified BSD License:
+
+> Copyright © 2011 Russ Ross
+> All rights reserved.
+>
+> Redistribution and use in source and binary forms, with or without
+> modification, are permitted provided that the following conditions
+> are met:
+>
+> 1. Redistributions of source code must retain the above copyright
+> notice, this list of conditions and the following disclaimer.
+>
+> 2. Redistributions in binary form must reproduce the above
+> copyright notice, this list of conditions and the following
+> disclaimer in the documentation and/or other materials provided with
+> the distribution.
+>
+> THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+> "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+> LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+> FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+> COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+> INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+> BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+> LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+> CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+> LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
+> ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+> POSSIBILITY OF SUCH DAMAGE.
diff --git a/vendor/github.com/sqlitebrowser/blackfriday/README.md b/vendor/github.com/sqlitebrowser/blackfriday/README.md
new file mode 100644
index 0000000..0d1ac9a
--- /dev/null
+++ b/vendor/github.com/sqlitebrowser/blackfriday/README.md
@@ -0,0 +1,292 @@
+Blackfriday [](https://travis-ci.org/russross/blackfriday) [](https://godoc.org/github.com/russross/blackfriday)
+===========
+
+Blackfriday is a [Markdown][1] processor implemented in [Go][2]. It
+is paranoid about its input (so you can safely feed it user-supplied
+data), it is fast, it supports common extensions (tables, smart
+punctuation substitutions, etc.), and it is safe for all utf-8
+(unicode) input.
+
+HTML output is currently supported, along with Smartypants
+extensions. An experimental LaTeX output engine is also included.
+
+It started as a translation from C of [Sundown][3].
+
+
+Installation
+------------
+
+Blackfriday is compatible with Go 1. If you are using an older
+release of Go, consider using v1.1 of blackfriday, which was based
+on the last stable release of Go prior to Go 1. You can find it as a
+tagged commit on github.
+
+With Go 1 and git installed:
+
+ go get github.com/russross/blackfriday
+
+will download, compile, and install the package into your `$GOPATH`
+directory hierarchy. Alternatively, you can achieve the same if you
+import it into a project:
+
+ import "github.com/russross/blackfriday"
+
+and `go get` without parameters.
+
+Usage
+-----
+
+For basic usage, it is as simple as getting your input into a byte
+slice and calling:
+
+ output := blackfriday.MarkdownBasic(input)
+
+This renders it with no extensions enabled. To get a more useful
+feature set, use this instead:
+
+ output := blackfriday.MarkdownCommon(input)
+
+### Sanitize untrusted content
+
+Blackfriday itself does nothing to protect against malicious content. If you are
+dealing with user-supplied markdown, we recommend running blackfriday's output
+through HTML sanitizer such as
+[Bluemonday](https://github.com/microcosm-cc/bluemonday).
+
+Here's an example of simple usage of blackfriday together with bluemonday:
+
+``` go
+import (
+ "github.com/microcosm-cc/bluemonday"
+ "github.com/russross/blackfriday"
+)
+
+// ...
+unsafe := blackfriday.MarkdownCommon(input)
+html := bluemonday.UGCPolicy().SanitizeBytes(unsafe)
+```
+
+### Custom options
+
+If you want to customize the set of options, first get a renderer
+(currently either the HTML or LaTeX output engines), then use it to
+call the more general `Markdown` function. For examples, see the
+implementations of `MarkdownBasic` and `MarkdownCommon` in
+`markdown.go`.
+
+You can also check out `blackfriday-tool` for a more complete example
+of how to use it. Download and install it using:
+
+ go get github.com/russross/blackfriday-tool
+
+This is a simple command-line tool that allows you to process a
+markdown file using a standalone program. You can also browse the
+source directly on github if you are just looking for some example
+code:
+
+*
tag
+ if size := p.htmlHr(out, data, doRender); size > 0 {
+ return size
+ }
+
+ // check for HTML CDATA
+ if size := p.htmlCDATA(out, data, doRender); size > 0 {
+ return size
+ }
+
+ // no special case recognized
+ return 0
+ }
+
+ // look for an unindented matching closing tag
+ // followed by a blank line
+ found := false
+ /*
+ closetag := []byte("\n")
+ j = len(curtag) + 1
+ for !found {
+ // scan for a closing tag at the beginning of a line
+ if skip := bytes.Index(data[j:], closetag); skip >= 0 {
+ j += skip + len(closetag)
+ } else {
+ break
+ }
+
+ // see if it is the only thing on the line
+ if skip := p.isEmpty(data[j:]); skip > 0 {
+ // see if it is followed by a blank line/eof
+ j += skip
+ if j >= len(data) {
+ found = true
+ i = j
+ } else {
+ if skip := p.isEmpty(data[j:]); skip > 0 {
+ j += skip
+ found = true
+ i = j
+ }
+ }
+ }
+ }
+ */
+
+ // if not found, try a second pass looking for indented match
+ // but not if tag is "ins" or "del" (following original Markdown.pl)
+ if !found && curtag != "ins" && curtag != "del" {
+ i = 1
+ for i < len(data) {
+ i++
+ for i < len(data) && !(data[i-1] == '<' && data[i] == '/') {
+ i++
+ }
+
+ if i+2+len(curtag) >= len(data) {
+ break
+ }
+
+ j = p.htmlFindEnd(curtag, data[i-1:])
+
+ if j > 0 {
+ i += j - 1
+ found = true
+ break
+ }
+ }
+ }
+
+ if !found {
+ return 0
+ }
+
+ // the end of the block has been found
+ if doRender {
+ // trim newlines
+ end := i
+ for end > 0 && data[end-1] == '\n' {
+ end--
+ }
+ p.r.BlockHtml(out, data[:end])
+ }
+
+ return i
+}
+
+func (p *parser) renderHTMLBlock(out *bytes.Buffer, data []byte, start int, doRender bool) int {
+ // html block needs to end with a blank line
+ if i := p.isEmpty(data[start:]); i > 0 {
+ size := start + i
+ if doRender {
+ // trim trailing newlines
+ end := size
+ for end > 0 && data[end-1] == '\n' {
+ end--
+ }
+ p.r.BlockHtml(out, data[:end])
+ }
+ return size
+ }
+ return 0
+}
+
+// HTML comment, lax form
+func (p *parser) htmlComment(out *bytes.Buffer, data []byte, doRender bool) int {
+ i := p.inlineHTMLComment(out, data)
+ return p.renderHTMLBlock(out, data, i, doRender)
+}
+
+// HTML CDATA section
+func (p *parser) htmlCDATA(out *bytes.Buffer, data []byte, doRender bool) int {
+ const cdataTag = "') {
+ i++
+ }
+ i++
+ // no end-of-comment marker
+ if i >= len(data) {
+ return 0
+ }
+ return p.renderHTMLBlock(out, data, i, doRender)
+}
+
+// HR, which is the only self-closing block tag considered
+func (p *parser) htmlHr(out *bytes.Buffer, data []byte, doRender bool) int {
+ if data[0] != '<' || (data[1] != 'h' && data[1] != 'H') || (data[2] != 'r' && data[2] != 'R') {
+ return 0
+ }
+ if data[3] != ' ' && data[3] != '/' && data[3] != '>' {
+ // not an
tag after all; at least not a valid one
+ return 0
+ }
+
+ i := 3
+ for data[i] != '>' && data[i] != '\n' {
+ i++
+ }
+
+ if data[i] == '>' {
+ return p.renderHTMLBlock(out, data, i+1, doRender)
+ }
+
+ return 0
+}
+
+func (p *parser) htmlFindTag(data []byte) (string, bool) {
+ i := 0
+ for isalnum(data[i]) {
+ i++
+ }
+ key := string(data[:i])
+ if _, ok := blockTags[key]; ok {
+ return key, true
+ }
+ return "", false
+}
+
+func (p *parser) htmlFindEnd(tag string, data []byte) int {
+ // assume data[0] == '<' && data[1] == '/' already tested
+
+ // check if tag is a match
+ closetag := []byte("")
+ if !bytes.HasPrefix(data, closetag) {
+ return 0
+ }
+ i := len(closetag)
+
+ // check that the rest of the line is blank
+ skip := 0
+ if skip = p.isEmpty(data[i:]); skip == 0 {
+ return 0
+ }
+ i += skip
+ skip = 0
+
+ if i >= len(data) {
+ return i
+ }
+
+ if p.flags&EXTENSION_LAX_HTML_BLOCKS != 0 {
+ return i
+ }
+ if skip = p.isEmpty(data[i:]); skip == 0 {
+ // following line must be blank
+ return 0
+ }
+
+ return i + skip
+}
+
+func (*parser) isEmpty(data []byte) int {
+ // it is okay to call isEmpty on an empty buffer
+ if len(data) == 0 {
+ return 0
+ }
+
+ var i int
+ for i = 0; i < len(data) && data[i] != '\n'; i++ {
+ if data[i] != ' ' && data[i] != '\t' {
+ return 0
+ }
+ }
+ return i + 1
+}
+
+func (*parser) isHRule(data []byte) bool {
+ i := 0
+
+ // skip up to three spaces
+ for i < 3 && data[i] == ' ' {
+ i++
+ }
+
+ // look at the hrule char
+ if data[i] != '*' && data[i] != '-' && data[i] != '_' {
+ return false
+ }
+ c := data[i]
+
+ // the whole line must be the char or whitespace
+ n := 0
+ for data[i] != '\n' {
+ switch {
+ case data[i] == c:
+ n++
+ case data[i] != ' ':
+ return false
+ }
+ i++
+ }
+
+ return n >= 3
+}
+
+// isFenceLine checks if there's a fence line (e.g., ``` or ``` go) at the beginning of data,
+// and returns the end index if so, or 0 otherwise. It also returns the marker found.
+// If syntax is not nil, it gets set to the syntax specified in the fence line.
+// A final newline is mandatory to recognize the fence line, unless newlineOptional is true.
+func isFenceLine(data []byte, syntax *string, oldmarker string, newlineOptional bool) (end int, marker string) {
+ i, size := 0, 0
+
+ // skip up to three spaces
+ for i < len(data) && i < 3 && data[i] == ' ' {
+ i++
+ }
+
+ // check for the marker characters: ~ or `
+ if i >= len(data) {
+ return 0, ""
+ }
+ if data[i] != '~' && data[i] != '`' {
+ return 0, ""
+ }
+
+ c := data[i]
+
+ // the whole line must be the same char or whitespace
+ for i < len(data) && data[i] == c {
+ size++
+ i++
+ }
+
+ // the marker char must occur at least 3 times
+ if size < 3 {
+ return 0, ""
+ }
+ marker = string(data[i-size : i])
+
+ // if this is the end marker, it must match the beginning marker
+ if oldmarker != "" && marker != oldmarker {
+ return 0, ""
+ }
+
+ // TODO(shurcooL): It's probably a good idea to simplify the 2 code paths here
+ // into one, always get the syntax, and discard it if the caller doesn't care.
+ if syntax != nil {
+ syn := 0
+ i = skipChar(data, i, ' ')
+
+ if i >= len(data) {
+ if newlineOptional && i == len(data) {
+ return i, marker
+ }
+ return 0, ""
+ }
+
+ syntaxStart := i
+
+ if data[i] == '{' {
+ i++
+ syntaxStart++
+
+ for i < len(data) && data[i] != '}' && data[i] != '\n' {
+ syn++
+ i++
+ }
+
+ if i >= len(data) || data[i] != '}' {
+ return 0, ""
+ }
+
+ // strip all whitespace at the beginning and the end
+ // of the {} block
+ for syn > 0 && isspace(data[syntaxStart]) {
+ syntaxStart++
+ syn--
+ }
+
+ for syn > 0 && isspace(data[syntaxStart+syn-1]) {
+ syn--
+ }
+
+ i++
+ } else {
+ for i < len(data) && !isspace(data[i]) {
+ syn++
+ i++
+ }
+ }
+
+ *syntax = string(data[syntaxStart : syntaxStart+syn])
+ }
+
+ i = skipChar(data, i, ' ')
+ if i >= len(data) || data[i] != '\n' {
+ if newlineOptional && i == len(data) {
+ return i, marker
+ }
+ return 0, ""
+ }
+
+ return i + 1, marker // Take newline into account.
+}
+
+// fencedCodeBlock returns the end index if data contains a fenced code block at the beginning,
+// or 0 otherwise. It writes to out if doRender is true, otherwise it has no side effects.
+// If doRender is true, a final newline is mandatory to recognize the fenced code block.
+func (p *parser) fencedCodeBlock(out *bytes.Buffer, data []byte, doRender bool) int {
+ var syntax string
+ beg, marker := isFenceLine(data, &syntax, "", false)
+ if beg == 0 || beg >= len(data) {
+ return 0
+ }
+
+ var work bytes.Buffer
+
+ for {
+ // safe to assume beg < len(data)
+
+ // check for the end of the code block
+ newlineOptional := !doRender
+ fenceEnd, _ := isFenceLine(data[beg:], nil, marker, newlineOptional)
+ if fenceEnd != 0 {
+ beg += fenceEnd
+ break
+ }
+
+ // copy the current line
+ end := skipUntilChar(data, beg, '\n') + 1
+
+ // did we reach the end of the buffer without a closing marker?
+ if end >= len(data) {
+ return 0
+ }
+
+ // verbatim copy to the working buffer
+ if doRender {
+ work.Write(data[beg:end])
+ }
+ beg = end
+ }
+
+ if doRender {
+ p.r.BlockCode(out, work.Bytes(), syntax)
+ }
+
+ return beg
+}
+
+func (p *parser) table(out *bytes.Buffer, data []byte) int {
+ var header bytes.Buffer
+ i, columns := p.tableHeader(&header, data)
+ if i == 0 {
+ return 0
+ }
+
+ var body bytes.Buffer
+
+ for i < len(data) {
+ pipes, rowStart := 0, i
+ for ; data[i] != '\n'; i++ {
+ if data[i] == '|' {
+ pipes++
+ }
+ }
+
+ if pipes == 0 {
+ i = rowStart
+ break
+ }
+
+ // include the newline in data sent to tableRow
+ i++
+ p.tableRow(&body, data[rowStart:i], columns, false)
+ }
+
+ p.r.Table(out, header.Bytes(), body.Bytes(), columns)
+
+ return i
+}
+
+// check if the specified position is preceded by an odd number of backslashes
+func isBackslashEscaped(data []byte, i int) bool {
+ backslashes := 0
+ for i-backslashes-1 >= 0 && data[i-backslashes-1] == '\\' {
+ backslashes++
+ }
+ return backslashes&1 == 1
+}
+
+func (p *parser) tableHeader(out *bytes.Buffer, data []byte) (size int, columns []int) {
+ i := 0
+ colCount := 1
+ for i = 0; data[i] != '\n'; i++ {
+ if data[i] == '|' && !isBackslashEscaped(data, i) {
+ colCount++
+ }
+ }
+
+ // doesn't look like a table header
+ if colCount == 1 {
+ return
+ }
+
+ // include the newline in the data sent to tableRow
+ header := data[:i+1]
+
+ // column count ignores pipes at beginning or end of line
+ if data[0] == '|' {
+ colCount--
+ }
+ if i > 2 && data[i-1] == '|' && !isBackslashEscaped(data, i-1) {
+ colCount--
+ }
+
+ columns = make([]int, colCount)
+
+ // move on to the header underline
+ i++
+ if i >= len(data) {
+ return
+ }
+
+ if data[i] == '|' && !isBackslashEscaped(data, i) {
+ i++
+ }
+ i = skipChar(data, i, ' ')
+
+ // each column header is of form: / *:?-+:? *|/ with # dashes + # colons >= 3
+ // and trailing | optional on last column
+ col := 0
+ for data[i] != '\n' {
+ dashes := 0
+
+ if data[i] == ':' {
+ i++
+ columns[col] |= TABLE_ALIGNMENT_LEFT
+ dashes++
+ }
+ for data[i] == '-' {
+ i++
+ dashes++
+ }
+ if data[i] == ':' {
+ i++
+ columns[col] |= TABLE_ALIGNMENT_RIGHT
+ dashes++
+ }
+ for data[i] == ' ' {
+ i++
+ }
+
+ // end of column test is messy
+ switch {
+ case dashes < 3:
+ // not a valid column
+ return
+
+ case data[i] == '|' && !isBackslashEscaped(data, i):
+ // marker found, now skip past trailing whitespace
+ col++
+ i++
+ for data[i] == ' ' {
+ i++
+ }
+
+ // trailing junk found after last column
+ if col >= colCount && data[i] != '\n' {
+ return
+ }
+
+ case (data[i] != '|' || isBackslashEscaped(data, i)) && col+1 < colCount:
+ // something else found where marker was required
+ return
+
+ case data[i] == '\n':
+ // marker is optional for the last column
+ col++
+
+ default:
+ // trailing junk found after last column
+ return
+ }
+ }
+ if col != colCount {
+ return
+ }
+
+ p.tableRow(out, header, columns, true)
+ size = i + 1
+ return
+}
+
+func (p *parser) tableRow(out *bytes.Buffer, data []byte, columns []int, header bool) {
+ i, col := 0, 0
+ var rowWork bytes.Buffer
+
+ if data[i] == '|' && !isBackslashEscaped(data, i) {
+ i++
+ }
+
+ for col = 0; col < len(columns) && i < len(data); col++ {
+ for data[i] == ' ' {
+ i++
+ }
+
+ cellStart := i
+
+ for (data[i] != '|' || isBackslashEscaped(data, i)) && data[i] != '\n' {
+ i++
+ }
+
+ cellEnd := i
+
+ // skip the end-of-cell marker, possibly taking us past end of buffer
+ i++
+
+ for cellEnd > cellStart && data[cellEnd-1] == ' ' {
+ cellEnd--
+ }
+
+ var cellWork bytes.Buffer
+ p.inline(&cellWork, data[cellStart:cellEnd])
+
+ if header {
+ p.r.TableHeaderCell(&rowWork, cellWork.Bytes(), columns[col])
+ } else {
+ p.r.TableCell(&rowWork, cellWork.Bytes(), columns[col])
+ }
+ }
+
+ // pad it out with empty columns to get the right number
+ for ; col < len(columns); col++ {
+ if header {
+ p.r.TableHeaderCell(&rowWork, nil, columns[col])
+ } else {
+ p.r.TableCell(&rowWork, nil, columns[col])
+ }
+ }
+
+ // silently ignore rows with too many cells
+
+ p.r.TableRow(out, rowWork.Bytes())
+}
+
+// returns blockquote prefix length
+func (p *parser) quotePrefix(data []byte) int {
+ i := 0
+ for i < 3 && data[i] == ' ' {
+ i++
+ }
+ if data[i] == '>' {
+ if data[i+1] == ' ' {
+ return i + 2
+ }
+ return i + 1
+ }
+ return 0
+}
+
+// blockquote ends with at least one blank line
+// followed by something without a blockquote prefix
+func (p *parser) terminateBlockquote(data []byte, beg, end int) bool {
+ if p.isEmpty(data[beg:]) <= 0 {
+ return false
+ }
+ if end >= len(data) {
+ return true
+ }
+ return p.quotePrefix(data[end:]) == 0 && p.isEmpty(data[end:]) == 0
+}
+
+// parse a blockquote fragment
+func (p *parser) quote(out *bytes.Buffer, data []byte) int {
+ var raw bytes.Buffer
+ beg, end := 0, 0
+ for beg < len(data) {
+ end = beg
+ // Step over whole lines, collecting them. While doing that, check for
+ // fenced code and if one's found, incorporate it altogether,
+ // irregardless of any contents inside it
+ for data[end] != '\n' {
+ if p.flags&EXTENSION_FENCED_CODE != 0 {
+ if i := p.fencedCodeBlock(out, data[end:], false); i > 0 {
+ // -1 to compensate for the extra end++ after the loop:
+ end += i - 1
+ break
+ }
+ }
+ end++
+ }
+ end++
+
+ if pre := p.quotePrefix(data[beg:]); pre > 0 {
+ // skip the prefix
+ beg += pre
+ } else if p.terminateBlockquote(data, beg, end) {
+ break
+ }
+
+ // this line is part of the blockquote
+ raw.Write(data[beg:end])
+ beg = end
+ }
+
+ var cooked bytes.Buffer
+ p.block(&cooked, raw.Bytes())
+ p.r.BlockQuote(out, cooked.Bytes())
+ return end
+}
+
+// returns prefix length for block code
+func (p *parser) codePrefix(data []byte) int {
+ if data[0] == ' ' && data[1] == ' ' && data[2] == ' ' && data[3] == ' ' {
+ return 4
+ }
+ return 0
+}
+
+func (p *parser) code(out *bytes.Buffer, data []byte) int {
+ var work bytes.Buffer
+
+ i := 0
+ for i < len(data) {
+ beg := i
+ for data[i] != '\n' {
+ i++
+ }
+ i++
+
+ blankline := p.isEmpty(data[beg:i]) > 0
+ if pre := p.codePrefix(data[beg:i]); pre > 0 {
+ beg += pre
+ } else if !blankline {
+ // non-empty, non-prefixed line breaks the pre
+ i = beg
+ break
+ }
+
+ // verbatim copy to the working buffeu
+ if blankline {
+ work.WriteByte('\n')
+ } else {
+ work.Write(data[beg:i])
+ }
+ }
+
+ // trim all the \n off the end of work
+ workbytes := work.Bytes()
+ eol := len(workbytes)
+ for eol > 0 && workbytes[eol-1] == '\n' {
+ eol--
+ }
+ if eol != len(workbytes) {
+ work.Truncate(eol)
+ }
+
+ work.WriteByte('\n')
+
+ p.r.BlockCode(out, work.Bytes(), "")
+
+ return i
+}
+
+// returns unordered list item prefix
+func (p *parser) uliPrefix(data []byte) int {
+ i := 0
+
+ // start with up to 3 spaces
+ for i < 3 && data[i] == ' ' {
+ i++
+ }
+
+ // need a *, +, or - followed by a space
+ if (data[i] != '*' && data[i] != '+' && data[i] != '-') ||
+ data[i+1] != ' ' {
+ return 0
+ }
+ return i + 2
+}
+
+// returns ordered list item prefix
+func (p *parser) oliPrefix(data []byte) int {
+ i := 0
+
+ // start with up to 3 spaces
+ for i < 3 && data[i] == ' ' {
+ i++
+ }
+
+ // count the digits
+ start := i
+ for data[i] >= '0' && data[i] <= '9' {
+ i++
+ }
+
+ // we need >= 1 digits followed by a dot and a space
+ if start == i || data[i] != '.' || data[i+1] != ' ' {
+ return 0
+ }
+ return i + 2
+}
+
+// returns definition list item prefix
+func (p *parser) dliPrefix(data []byte) int {
+ i := 0
+
+ // need a : followed by a spaces
+ if data[i] != ':' || data[i+1] != ' ' {
+ return 0
+ }
+ for data[i] == ' ' {
+ i++
+ }
+ return i + 2
+}
+
+// parse ordered or unordered list block
+func (p *parser) list(out *bytes.Buffer, data []byte, flags int) int {
+ i := 0
+ flags |= LIST_ITEM_BEGINNING_OF_LIST
+ work := func() bool {
+ for i < len(data) {
+ skip := p.listItem(out, data[i:], &flags)
+ i += skip
+
+ if skip == 0 || flags&LIST_ITEM_END_OF_LIST != 0 {
+ break
+ }
+ flags &= ^LIST_ITEM_BEGINNING_OF_LIST
+ }
+ return true
+ }
+
+ p.r.List(out, work, flags)
+ return i
+}
+
+// Parse a single list item.
+// Assumes initial prefix is already removed if this is a sublist.
+func (p *parser) listItem(out *bytes.Buffer, data []byte, flags *int) int {
+ // keep track of the indentation of the first line
+ itemIndent := 0
+ for itemIndent < 3 && data[itemIndent] == ' ' {
+ itemIndent++
+ }
+
+ i := p.uliPrefix(data)
+ if i == 0 {
+ i = p.oliPrefix(data)
+ }
+ if i == 0 {
+ i = p.dliPrefix(data)
+ // reset definition term flag
+ if i > 0 {
+ *flags &= ^LIST_TYPE_TERM
+ }
+ }
+ if i == 0 {
+ // if in defnition list, set term flag and continue
+ if *flags&LIST_TYPE_DEFINITION != 0 {
+ *flags |= LIST_TYPE_TERM
+ } else {
+ return 0
+ }
+ }
+
+ // skip leading whitespace on first line
+ for data[i] == ' ' {
+ i++
+ }
+
+ // find the end of the line
+ line := i
+ for i > 0 && data[i-1] != '\n' {
+ i++
+ }
+
+ // get working buffer
+ var raw bytes.Buffer
+
+ // put the first line into the working buffer
+ raw.Write(data[line:i])
+ line = i
+
+ // process the following lines
+ containsBlankLine := false
+ sublist := 0
+
+gatherlines:
+ for line < len(data) {
+ i++
+
+ // find the end of this line
+ for data[i-1] != '\n' {
+ i++
+ }
+
+ // if it is an empty line, guess that it is part of this item
+ // and move on to the next line
+ if p.isEmpty(data[line:i]) > 0 {
+ containsBlankLine = true
+ raw.Write(data[line:i])
+ line = i
+ continue
+ }
+
+ // calculate the indentation
+ indent := 0
+ for indent < 4 && line+indent < i && data[line+indent] == ' ' {
+ indent++
+ }
+
+ chunk := data[line+indent : i]
+
+ // evaluate how this line fits in
+ switch {
+ // is this a nested list item?
+ case (p.uliPrefix(chunk) > 0 && !p.isHRule(chunk)) ||
+ p.oliPrefix(chunk) > 0 ||
+ p.dliPrefix(chunk) > 0:
+
+ if containsBlankLine {
+ // end the list if the type changed after a blank line
+ if indent <= itemIndent &&
+ ((*flags&LIST_TYPE_ORDERED != 0 && p.uliPrefix(chunk) > 0) ||
+ (*flags&LIST_TYPE_ORDERED == 0 && p.oliPrefix(chunk) > 0)) {
+
+ *flags |= LIST_ITEM_END_OF_LIST
+ break gatherlines
+ }
+ *flags |= LIST_ITEM_CONTAINS_BLOCK
+ }
+
+ // to be a nested list, it must be indented more
+ // if not, it is the next item in the same list
+ if indent <= itemIndent {
+ break gatherlines
+ }
+
+ // is this the first item in the nested list?
+ if sublist == 0 {
+ sublist = raw.Len()
+ }
+
+ // is this a nested prefix header?
+ case p.isPrefixHeader(chunk):
+ // if the header is not indented, it is not nested in the list
+ // and thus ends the list
+ if containsBlankLine && indent < 4 {
+ *flags |= LIST_ITEM_END_OF_LIST
+ break gatherlines
+ }
+ *flags |= LIST_ITEM_CONTAINS_BLOCK
+
+ // anything following an empty line is only part
+ // of this item if it is indented 4 spaces
+ // (regardless of the indentation of the beginning of the item)
+ case containsBlankLine && indent < 4:
+ if *flags&LIST_TYPE_DEFINITION != 0 && i < len(data)-1 {
+ // is the next item still a part of this list?
+ next := i
+ for data[next] != '\n' {
+ next++
+ }
+ for next < len(data)-1 && data[next] == '\n' {
+ next++
+ }
+ if i < len(data)-1 && data[i] != ':' && data[next] != ':' {
+ *flags |= LIST_ITEM_END_OF_LIST
+ }
+ } else {
+ *flags |= LIST_ITEM_END_OF_LIST
+ }
+ break gatherlines
+
+ // a blank line means this should be parsed as a block
+ case containsBlankLine:
+ *flags |= LIST_ITEM_CONTAINS_BLOCK
+ }
+
+ containsBlankLine = false
+
+ // add the line into the working buffer without prefix
+ raw.Write(data[line+indent : i])
+
+ line = i
+ }
+
+ // If reached end of data, the Renderer.ListItem call we're going to make below
+ // is definitely the last in the list.
+ if line >= len(data) {
+ *flags |= LIST_ITEM_END_OF_LIST
+ }
+
+ rawBytes := raw.Bytes()
+
+ // render the contents of the list item
+ var cooked bytes.Buffer
+ if *flags&LIST_ITEM_CONTAINS_BLOCK != 0 && *flags&LIST_TYPE_TERM == 0 {
+ // intermediate render of block item, except for definition term
+ if sublist > 0 {
+ p.block(&cooked, rawBytes[:sublist])
+ p.block(&cooked, rawBytes[sublist:])
+ } else {
+ p.block(&cooked, rawBytes)
+ }
+ } else {
+ // intermediate render of inline item
+ if sublist > 0 {
+ p.inline(&cooked, rawBytes[:sublist])
+ p.block(&cooked, rawBytes[sublist:])
+ } else {
+ p.inline(&cooked, rawBytes)
+ }
+ }
+
+ // render the actual list item
+ cookedBytes := cooked.Bytes()
+ parsedEnd := len(cookedBytes)
+
+ // strip trailing newlines
+ for parsedEnd > 0 && cookedBytes[parsedEnd-1] == '\n' {
+ parsedEnd--
+ }
+ p.r.ListItem(out, cookedBytes[:parsedEnd], *flags)
+
+ return line
+}
+
+// render a single paragraph that has already been parsed out
+func (p *parser) renderParagraph(out *bytes.Buffer, data []byte) {
+ if len(data) == 0 {
+ return
+ }
+
+ // trim leading spaces
+ beg := 0
+ for data[beg] == ' ' {
+ beg++
+ }
+
+ // trim trailing newline
+ end := len(data) - 1
+
+ // trim trailing spaces
+ for end > beg && data[end-1] == ' ' {
+ end--
+ }
+
+ work := func() bool {
+ p.inline(out, data[beg:end])
+ return true
+ }
+ p.r.Paragraph(out, work)
+}
+
+func (p *parser) paragraph(out *bytes.Buffer, data []byte) int {
+ // prev: index of 1st char of previous line
+ // line: index of 1st char of current line
+ // i: index of cursor/end of current line
+ var prev, line, i int
+
+ // keep going until we find something to mark the end of the paragraph
+ for i < len(data) {
+ // mark the beginning of the current line
+ prev = line
+ current := data[i:]
+ line = i
+
+ // did we find a blank line marking the end of the paragraph?
+ if n := p.isEmpty(current); n > 0 {
+ // did this blank line followed by a definition list item?
+ if p.flags&EXTENSION_DEFINITION_LISTS != 0 {
+ if i < len(data)-1 && data[i+1] == ':' {
+ return p.list(out, data[prev:], LIST_TYPE_DEFINITION)
+ }
+ }
+
+ p.renderParagraph(out, data[:i])
+ return i + n
+ }
+
+ // an underline under some text marks a header, so our paragraph ended on prev line
+ if i > 0 {
+ if level := p.isUnderlinedHeader(current); level > 0 {
+ // render the paragraph
+ p.renderParagraph(out, data[:prev])
+
+ // ignore leading and trailing whitespace
+ eol := i - 1
+ for prev < eol && data[prev] == ' ' {
+ prev++
+ }
+ for eol > prev && data[eol-1] == ' ' {
+ eol--
+ }
+
+ // render the header
+ // this ugly double closure avoids forcing variables onto the heap
+ work := func(o *bytes.Buffer, pp *parser, d []byte) func() bool {
+ return func() bool {
+ pp.inline(o, d)
+ return true
+ }
+ }(out, p, data[prev:eol])
+
+ id := ""
+ if p.flags&EXTENSION_AUTO_HEADER_IDS != 0 {
+ id = SanitizedAnchorName(string(data[prev:eol]))
+ }
+
+ p.r.Header(out, work, level, id)
+
+ // find the end of the underline
+ for data[i] != '\n' {
+ i++
+ }
+ return i
+ }
+ }
+
+ // if the next line starts a block of HTML, then the paragraph ends here
+ if p.flags&EXTENSION_LAX_HTML_BLOCKS != 0 {
+ if data[i] == '<' && p.html(out, current, false) > 0 {
+ // rewind to before the HTML block
+ p.renderParagraph(out, data[:i])
+ return i
+ }
+ }
+
+ // if there's a prefixed header or a horizontal rule after this, paragraph is over
+ if p.isPrefixHeader(current) || p.isHRule(current) {
+ p.renderParagraph(out, data[:i])
+ return i
+ }
+
+ // if there's a fenced code block, paragraph is over
+ if p.flags&EXTENSION_FENCED_CODE != 0 {
+ if p.fencedCodeBlock(out, current, false) > 0 {
+ p.renderParagraph(out, data[:i])
+ return i
+ }
+ }
+
+ // if there's a definition list item, prev line is a definition term
+ if p.flags&EXTENSION_DEFINITION_LISTS != 0 {
+ if p.dliPrefix(current) != 0 {
+ return p.list(out, data[prev:], LIST_TYPE_DEFINITION)
+ }
+ }
+
+ // if there's a list after this, paragraph is over
+ if p.flags&EXTENSION_NO_EMPTY_LINE_BEFORE_BLOCK != 0 {
+ if p.uliPrefix(current) != 0 ||
+ p.oliPrefix(current) != 0 ||
+ p.quotePrefix(current) != 0 ||
+ p.codePrefix(current) != 0 {
+ p.renderParagraph(out, data[:i])
+ return i
+ }
+ }
+
+ // otherwise, scan to the beginning of the next line
+ for data[i] != '\n' {
+ i++
+ }
+ i++
+ }
+
+ p.renderParagraph(out, data[:i])
+ return i
+}
+
+// SanitizedAnchorName returns a sanitized anchor name for the given text.
+//
+// It implements the algorithm specified in the package comment.
+func SanitizedAnchorName(text string) string {
+ var anchorName []rune
+ futureDash := false
+ for _, r := range text {
+ switch {
+ case unicode.IsLetter(r) || unicode.IsNumber(r):
+ if futureDash && len(anchorName) > 0 {
+ anchorName = append(anchorName, '-')
+ }
+ futureDash = false
+ anchorName = append(anchorName, unicode.ToLower(r))
+ default:
+ futureDash = true
+ }
+ }
+ return string(anchorName)
+}
diff --git a/vendor/github.com/sqlitebrowser/blackfriday/doc.go b/vendor/github.com/sqlitebrowser/blackfriday/doc.go
new file mode 100644
index 0000000..9656c42
--- /dev/null
+++ b/vendor/github.com/sqlitebrowser/blackfriday/doc.go
@@ -0,0 +1,32 @@
+// Package blackfriday is a Markdown processor.
+//
+// It translates plain text with simple formatting rules into HTML or LaTeX.
+//
+// Sanitized Anchor Names
+//
+// Blackfriday includes an algorithm for creating sanitized anchor names
+// corresponding to a given input text. This algorithm is used to create
+// anchors for headings when EXTENSION_AUTO_HEADER_IDS is enabled. The
+// algorithm is specified below, so that other packages can create
+// compatible anchor names and links to those anchors.
+//
+// The algorithm iterates over the input text, interpreted as UTF-8,
+// one Unicode code point (rune) at a time. All runes that are letters (category L)
+// or numbers (category N) are considered valid characters. They are mapped to
+// lower case, and included in the output. All other runes are considered
+// invalid characters. Invalid characters that preceed the first valid character,
+// as well as invalid character that follow the last valid character
+// are dropped completely. All other sequences of invalid characters
+// between two valid characters are replaced with a single dash character '-'.
+//
+// SanitizedAnchorName exposes this functionality, and can be used to
+// create compatible links to the anchor names generated by blackfriday.
+// This algorithm is also implemented in a small standalone package at
+// github.com/shurcooL/sanitized_anchor_name. It can be useful for clients
+// that want a small package and don't need full functionality of blackfriday.
+package blackfriday
+
+// NOTE: Keep Sanitized Anchor Name algorithm in sync with package
+// github.com/shurcooL/sanitized_anchor_name.
+// Otherwise, users of sanitized_anchor_name will get anchor names
+// that are incompatible with those generated by blackfriday.
diff --git a/vendor/github.com/sqlitebrowser/blackfriday/html.go b/vendor/github.com/sqlitebrowser/blackfriday/html.go
new file mode 100644
index 0000000..bb9268f
--- /dev/null
+++ b/vendor/github.com/sqlitebrowser/blackfriday/html.go
@@ -0,0 +1,950 @@
+//
+// Blackfriday Markdown Processor
+// Available at http://github.com/russross/blackfriday
+//
+// Copyright © 2011 Russ Ross