mirror/opencloud
1
0
Fork 0
mirror of https://github.com/opencloud-eu/opencloud.git synced 2025-12-30 17:00:57 -06:00
Code Issues Packages Projects Releases Wiki Activity

[full-ci] enhancement: add more kql spec tests and simplify ast normalization (#7254)

Browse Source 
* enhancement: add more kql spec tests and simplify ast normalization

* enhancement: kql parser error if query starts with AND

* enhancement: add kql docs and support for date and time only dateTimeRestriction queries

* enhancement: add the ability to decide how kql nodes get connected

connecting nodes (with edges) seem straight forward when not using group, the default connection for nodes with the same node is always OR. THis only applies for first level nodes, for grouped nodes it is defined differently. The KQL docs are saying, nodes inside a grouped node, with the same key are connected by a AND edge.

* enhancement: explicit error handling for falsy group nodes and queries with leading binary operator

* enhancement: use optimized grammar for kql parser and toolify pigeon

* enhancement: simplify error handling

* fix: kql implicit 'AND' and 'OR' follows the ms html spec instead of the pdf spec
This commit is contained in:
Florian Schade
2023-09-11 13:49:53 +02:00
committed by GitHub
parent dbb666babf
commit c0553c7273
72 changed files with 35393 additions and 1913 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
changelog/unreleased/kql-fix-unary.md
View File

@@ -1,7 +0,0 @@
Bugfix: Fixed cunary in the beginning
Fixed case when the unary in the beginning lead to panic
https://github.com/owncloud/ocis/pull/7247
git

4
changelog/unreleased/kql-search-query-language.md
View File

@@ -19,7 +19,11 @@ Complex queries:
https://github.com/owncloud/ocis/pull/7212
https://github.com/owncloud/ocis/pull/7043
https://github.com/owncloud/ocis/pull/7247
https://github.com/owncloud/ocis/pull/7248
https://github.com/owncloud/ocis/pull/7254
https://github.com/owncloud/web/pull/9653
https://github.com/owncloud/web/pull/9672
https://github.com/owncloud/ocis/issues/7042
https://github.com/owncloud/ocis/issues/7179
https://github.com/owncloud/ocis/issues/7114

2
go.mod
View File

@@ -8,6 +8,7 @@ require (
github.com/Masterminds/semver v1.5.0
github.com/MicahParks/keyfunc v1.9.0
github.com/Nerzal/gocloak/v13 v13.8.0
github.com/araddon/dateparse v0.0.0-20210429162001-6b43995a97de
github.com/bbalet/stopwords v1.0.0
github.com/blevesearch/bleve/v2 v2.3.9
github.com/coreos/go-oidc v2.2.1+incompatible
@@ -54,6 +55,7 @@ require (
github.com/libregraph/idm v0.4.1-0.20230221143410-3503963047a5
github.com/libregraph/lico v0.60.1-0.20230811070109-1d4140be554d
github.com/mitchellh/mapstructure v1.5.0
github.com/mna/pigeon v1.1.0
github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826
github.com/nats-io/nats-server/v2 v2.9.22
github.com/oklog/run v1.1.0

8
go.sum
View File

@@ -859,6 +859,8 @@ github.com/apache/arrow/go/v11 v11.0.0/go.mod h1:Eg5OsL5H+e299f7u5ssuXsuHQVEGC4x
github.com/apache/arrow/go/v12 v12.0.0/go.mod h1:d+tV/eHZZ7Dz7RPrFKtPK02tpr+c9/PEd/zm8mDS9Vg=
github.com/apache/thrift v0.12.0/go.mod h1:cp2SuWMxlEZw2r+iP2GNCdIi4C1qmUzdZFSVb+bacwQ=
github.com/apache/thrift v0.16.0/go.mod h1:PHK3hniurgQaNMZYaCLEqXKsYK8upmhPbmdP2FXSqgU=
github.com/araddon/dateparse v0.0.0-20210429162001-6b43995a97de h1:FxWPpzIjnTlhPwqqXc4/vE0f7GvRjuAsbW+HOIe8KnA=
github.com/araddon/dateparse v0.0.0-20210429162001-6b43995a97de/go.mod h1:DCaWoUhZrYW9p1lxo/cm8EmUOOzAPSEZNGF2DK1dJgw=
github.com/arbovm/levenshtein v0.0.0-20160628152529-48b4e1c0c4d0 h1:jfIu9sQUG6Ig+0+Ap1h4unLjW6YQJpKZVmUzxsD4E/Q=
github.com/arbovm/levenshtein v0.0.0-20160628152529-48b4e1c0c4d0/go.mod h1:t2tdKJDJF9BV14lnkjHmOQgcvEKgtqs5a1N3LNdJhGE=
github.com/armon/circbuf v0.0.0-20150827004946-bbbad097214e/go.mod h1:3U/XgcO3hCbHZ8TKRvWD2dDTCfh9M9ya+I9JpbB7O8o=
@@ -1640,6 +1642,7 @@ github.com/mattn/go-isatty v0.0.17/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/
github.com/mattn/go-runewidth v0.0.2/go.mod h1:LwmH8dsx7+W8Uxz3IHJYH5QSwggIsqBzpuz5H//U1FU=
github.com/mattn/go-runewidth v0.0.6/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI=
github.com/mattn/go-runewidth v0.0.9/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI=
github.com/mattn/go-runewidth v0.0.10/go.mod h1:RAqKPSqVFrSLVXbA8x7dzmKdmGzieGRCM46jaSJTDAk=
github.com/mattn/go-runewidth v0.0.13 h1:lTGmDsbAYt5DmK6OnoV7EuIF1wEIFAcxld6ypU4OSgU=
github.com/mattn/go-runewidth v0.0.13/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
github.com/mattn/go-sqlite3 v1.14.14/go.mod h1:NyWgC/yNuGj7Q9rpYnZvas74GogHl5/Z4A/KQRfk6bU=
@@ -1697,6 +1700,8 @@ github.com/mitchellh/mapstructure v1.5.0 h1:jeMsZIYE/09sWLaz43PL7Gy6RuMjD2eJVyua
github.com/mitchellh/mapstructure v1.5.0/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo=
github.com/mitchellh/reflectwalk v1.0.2 h1:G2LzWKi524PWgd3mLHV8Y5k7s6XUvT0Gef6zxSIeXaQ=
github.com/mitchellh/reflectwalk v1.0.2/go.mod h1:mSTlrgnPZtwu0c4WaC2kGObEpuNDbx0jmZXqmk4esnw=
github.com/mna/pigeon v1.1.0 h1:EjlvVbkGnNGemf8OrjeJX0nH8orujY/HkJgzJtd7kxc=
github.com/mna/pigeon v1.1.0/go.mod h1:rkFeDZ0gc+YbnrXPw0q2RlI0QRuKBBPu67fgYIyGRNg=
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
@@ -1861,6 +1866,7 @@ github.com/remyoudompheng/bigfft v0.0.0-20200410134404-eec4a21b6bb0/go.mod h1:qq
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
github.com/riandyrn/otelchi v0.5.1 h1:0/45omeqpP7f/cvdL16GddQBfAEmZvUyl2QzLSE6uYo=
github.com/riandyrn/otelchi v0.5.1/go.mod h1:ZxVxNEl+jQ9uHseRYIxKWRb3OY8YXFEu+EkNiiSNUEA=
github.com/rivo/uniseg v0.1.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
github.com/rivo/uniseg v0.4.2 h1:YwD0ulJSJytLpiaWua0sBDusfsCZohxjxzVTYjwxfV8=
github.com/rivo/uniseg v0.4.2/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
@@ -1892,6 +1898,7 @@ github.com/sacloud/libsacloud v1.36.2/go.mod h1:P7YAOVmnIn3DKHqCZcUKYUXmSwGBm3yS
github.com/scaleway/scaleway-sdk-go v1.0.0-beta.7.0.20210127161313-bd30bebeac4f/go.mod h1:CJJ5VAbozOl0yEw7nHB9+7BXTJbIn6h7W+f6Gau5IP8=
github.com/sciencemesh/meshdirectory-web v1.0.4 h1:1YSctF6PAXhoHUYCaeRTj7rHaF7b3rYrZf2R0VXBIbo=
github.com/sciencemesh/meshdirectory-web v1.0.4/go.mod h1:fJSThTS3xf+sTdL0iXQoaQJssLI7tn7DetHMHUl4SRk=
github.com/scylladb/termtables v0.0.0-20191203121021-c4c0b6d42ff4/go.mod h1:C1a7PQSMz9NShzorzCiG2fk9+xuCgLkPeCvMHYR2OWg=
github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529 h1:nn5Wsu0esKSJiIVhscUtVbo7ada43DJhG55ua/hjS5I=
github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529/go.mod h1:DxrIzT+xaE7yg65j358z/aeFdxmN0P9QXhEzd20vsDc=
github.com/segmentio/ksuid v1.0.4 h1:sBo2BdShXjmcugAMwjugoGUdUV0pcxY5mW4xKRn3v4c=
@@ -2526,6 +2533,7 @@ golang.org/x/tools v0.0.0-20190606124116-d0a3d012864b/go.mod h1:/rFqwRUd4F7ZHNgw
golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190628153133-6cdbf07be9d0/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190816200558-6889da9d5479/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20190830223141-573d9926052a/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20190907020128-2ca718005c18/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20190911174233-4f2ddba30aff/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20190927191325-030b2cf1153e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=

3
ocis-pkg/tools/tools.go
View File

@@ -1,7 +1,8 @@
// +build tools
//go:build tools
package tools
import (
_ "github.com/mna/pigeon"
_ "github.com/onsi/ginkgo/ginkgo"
)

2
services/search/Makefile
View File

@@ -29,7 +29,7 @@ ci-go-generate: $(PIGEON) $(MOCKERY) # CI runs ci-node-generate automatically be
$(MOCKERY) --dir pkg/content --output pkg/content/mocks --case underscore --name Extractor
$(MOCKERY) --dir pkg/content --output pkg/content/mocks --case underscore --name Retriever
$(MOCKERY) --dir pkg/search --output pkg/search/mocks --case underscore --name Searcher
$(PIGEON) -o pkg/query/kql/dictionary_gen.go pkg/query/kql/dictionary.peg
$(PIGEON) -optimize-grammar -optimize-parser -o pkg/query/kql/dictionary_gen.go pkg/query/kql/dictionary.peg
.PHONY: ci-node-generate
ci-node-generate:

32
services/search/pkg/query/ast/ast.go
View File

@@ -73,3 +73,35 @@ type GroupNode struct {
Key string
Nodes []Node
}
// NodeKey tries to return the node key
func NodeKey(n Node) string {
switch node := n.(type) {
case *StringNode:
return node.Key
case *DateTimeNode:
return node.Key
case *BooleanNode:
return node.Key
case *GroupNode:
return node.Key
default:
return ""
}
}
// NodeValue tries to return the node key
func NodeValue(n Node) interface{} {
switch node := n.(type) {
case *StringNode:
return node.Value
case *DateTimeNode:
return node.Value
case *BooleanNode:
return node.Value
case *GroupNode:
return node.Nodes
default:
return ""
}
}

35
services/search/pkg/query/kql/cast.go
View File

@@ -4,16 +4,11 @@ import (
"fmt"
"time"
"github.com/araddon/dateparse"
"github.com/owncloud/ocis/v2/services/search/pkg/query/ast"
)
func toIfaceSlice(in interface{}) []interface{} {
if in == nil {
return nil
}
return in.([]interface{})
}
func toNode[T ast.Node](in interface{}) (T, error) {
var t T
out, ok := in.(T)
@@ -25,25 +20,27 @@ func toNode[T ast.Node](in interface{}) (T, error) {
}
func toNodes[T ast.Node](in interface{}) ([]T, error) {
switch v := in.(type) {
case []T:
return v, nil
case T:
return []T{v}, nil
case []interface{}:
var nodes []T
for _, el := range toIfaceSlice(v) {
node, err := toNode[T](el)
var ts []T
for _, inter := range v {
n, err := toNodes[T](inter)
if err != nil {
return nil, err
}
nodes = append(nodes, node)
ts = append(ts, n...)
}
return nodes, nil
case []T:
return v, nil
return ts, nil
case nil:
return nil, nil
default:
return nil, fmt.Errorf("can't convert '%T' to []ast.Node", in)
var t T
return nil, fmt.Errorf("can't convert '%T' to '%T'", in, t)
}
}
@@ -77,5 +74,5 @@ func toTime(in interface{}) (time.Time, error) {
return time.Time{}, err
}
return time.Parse(time.RFC3339Nano, ts)
return dateparse.ParseLocal(ts)
}

129
services/search/pkg/query/kql/connect.go Normal file
View File

@@ -0,0 +1,129 @@
package kql
import (
"strings"
"github.com/owncloud/ocis/v2/services/search/pkg/query/ast"
)
// connectNodes connects given nodes
func connectNodes(c Connector, nodes ...ast.Node) []ast.Node {
var connectedNodes []ast.Node
for i := range nodes {
ri := len(nodes) - 1 - i
head := nodes[ri]
if connectionNodes := connectNode(c, head, connectedNodes...); len(connectionNodes) > 0 {
connectedNodes = append(connectionNodes, connectedNodes...)
}
connectedNodes = append([]ast.Node{head}, connectedNodes...)
}
return connectedNodes
}
// connectNode connects a tip node with the rest
func connectNode(c Connector, headNode ast.Node, tailNodes ...ast.Node) []ast.Node {
var nearestNeighborNode ast.Node
var nearestNeighborOperators []*ast.OperatorNode
l:
for _, tailNode := range tailNodes {
switch node := tailNode.(type) {
case *ast.OperatorNode:
nearestNeighborOperators = append(nearestNeighborOperators, node)
default:
nearestNeighborNode = node
break l
}
}
if nearestNeighborNode == nil {
return nil
}
return c.Connect(headNode, nearestNeighborNode, nearestNeighborOperators)
}
// Connector is responsible to decide what node connections are needed
type Connector interface {
Connect(head ast.Node, neighbor ast.Node, connections []*ast.OperatorNode) []ast.Node
}
// DefaultConnector is the default node connector
type DefaultConnector struct {
sameKeyOPValue string
}
// Connect implements the Connector interface and is used to connect the nodes using
// the default logic defined by the kql spec.
func (c DefaultConnector) Connect(head ast.Node, neighbor ast.Node, connections []*ast.OperatorNode) []ast.Node {
switch head.(type) {
case *ast.OperatorNode:
return nil
}
headKey := strings.ToLower(ast.NodeKey(head))
neighborKey := strings.ToLower(ast.NodeKey(neighbor))
connection := &ast.OperatorNode{
Base: &ast.Base{Loc: &ast.Location{Source: &[]string{"implicitly operator"}[0]}},
Value: BoolAND,
}
// if the current node and the neighbor node have the same key
// the connection is of type OR
//
// spec: same
// author:"John Smith" author:"Jane Smith"
// author:"John Smith" OR author:"Jane Smith"
//
// if the nodes have NO key, the edge is a AND connection
//
// spec: same
// cat dog
// cat AND dog
// from the spec:
// To construct complex queries, you can combine multiple
// free-text expressions with KQL query operators.
// If there are multiple free-text expressions without any
// operators in between them, the query behavior is the same
// as using the AND operator.
//
// nodes inside of group node are handled differently,
// if no explicit operator given, it uses AND
//
// spec: same
// author:"John Smith" AND author:"Jane Smith"
// author:("John Smith" "Jane Smith")
if headKey == neighborKey && headKey != "" && neighborKey != "" {
connection.Value = c.sameKeyOPValue
}
// decisions based on nearest neighbor operators
for i, node := range connections {
// consider direct neighbor operator only
if i == 0 {
// no connection is necessary here because an `AND` or `OR` edge is already present
// exit
for _, skipValue := range []string{BoolOR, BoolAND} {
if node.Value == skipValue {
return nil
}
}
// if neighbor node negotiates, an AND edge is needed
//
// spec: same
// cat -dog
// cat AND NOT dog
if node.Value == BoolNOT {
connection.Value = BoolAND
}
}
}
return []ast.Node{connection}
}

8
services/search/pkg/query/kql/const.go
View File

@@ -1,8 +0,0 @@
package kql
// The operator node value definition
const (
BoolAND = "AND"
BoolOR = "OR"
BoolNOT = "NOT"
)

68
services/search/pkg/query/kql/dictionary.peg
View File

@@ -7,23 +7,22 @@
////////////////////////////////////////////////////////
AST <-
_ nodes:Nodes _ {
return buildAST(nodes, c.text, c.pos)
n:Nodes {
return buildAST(n, c.text, c.pos)
}
////////////////////////////////////////////////////////
// nodes
////////////////////////////////////////////////////////
Nodes <-
n:(
_
(
GroupNode /
PropertyRestrictionNodes /
OperatorBooleanNode /
FreeTextKeywordNodes
)
_
)+ {
return buildNodes(n)
}
(_ Node)+
Node <-
GroupNode /
PropertyRestrictionNodes /
OperatorBooleanNodes /
FreeTextKeywordNodes
////////////////////////////////////////////////////////
// nesting
@@ -49,7 +48,18 @@ YesNoPropertyRestrictionNode <-
}
DateTimeRestrictionNode <-
k:Char+ o:(OperatorGreaterOrEqualNode / OperatorLessOrEqualNode / OperatorGreaterNode / OperatorLessNode / OperatorEqualNode / OperatorColonNode) '"'? v:(FullDate "T" FullTime) '"'? {
k:Char+ o:(
OperatorGreaterOrEqualNode /
OperatorLessOrEqualNode /
OperatorGreaterNode /
OperatorLessNode /
OperatorEqualNode /
OperatorColonNode
) '"'? v:(
DateTime /
FullDate /
FullTime
) '"'? {
return buildDateTimeNode(k, o, v, c.text, c.pos)
}
@@ -80,8 +90,23 @@ WordNode <-
// operators
////////////////////////////////////////////////////////
OperatorBooleanNode <-
("AND" / "OR" / "NOT") {
OperatorBooleanNodes <-
OperatorBooleanAndNode /
OperatorBooleanNotNode /
OperatorBooleanOrNode
OperatorBooleanAndNode <-
("AND" / "+") {
return buildOperatorNode(c.text, c.pos)
}
OperatorBooleanNotNode <-
("NOT" / "-") {
return buildOperatorNode(c.text, c.pos)
}
OperatorBooleanOrNode <-
("OR") {
return buildOperatorNode(c.text, c.pos)
}
@@ -160,6 +185,11 @@ FullTime <-
return c.text, nil
}
DateTime
= FullDate "T" FullTime {
return c.text, nil
}
////////////////////////////////////////////////////////
// misc
////////////////////////////////////////////////////////
@@ -180,4 +210,6 @@ Digit <-
}
_ <-
[ \t]*
[ \t]* {
return nil, nil
}

3880
services/search/pkg/query/kql/dictionary_gen.go
View File

File diff suppressed because it is too large Load Diff

794
services/search/pkg/query/kql/dictionary_test.go
View File

File diff suppressed because it is too large Load Diff

28
services/search/pkg/query/kql/doc.go Normal file
View File

@@ -0,0 +1,28 @@
/*
Package kql provides the ability to work with kql queries.
Not every aspect of the spec is implemented yet.
The language support will grow over time if needed.
The following spec parts are supported and tested:
- 2.1.2 AND Operator
- 2.1.6 NOT Operator
- 2.1.8 OR Operator
- 2.1.12 Parentheses
- 2.3.5 Date Tokens
- Human tokens not implemented
- 3.1.11 Implicit Operator
- 3.1.12 Parentheses
- 3.1.2 AND Operator
- 3.1.6 NOT Operator
- 3.1.8 OR Operator
- 3.2.3 Implicit Operator for Property Restriction
- 3.3.1.1.1 Implicit AND Operator
- 3.3.5 Date Tokens
References:
- https://learn.microsoft.com/en-us/sharepoint/dev/general-development/keyword-query-language-kql-syntax-reference
- https://learn.microsoft.com/en-us/openspecs/sharepoint_protocols/ms-kql/3bbf06cd-8fc1-4277-bd92-8661ccd3c9b0
- https://msopenspecs.azureedge.net/files/MS-KQL/%5bMS-KQL%5d.pdf
*/
package kql

26
services/search/pkg/query/kql/error.go
View File

@@ -1,10 +1,30 @@
package kql
import (
"fmt"
"github.com/owncloud/ocis/v2/services/search/pkg/query/ast"
)
// StartsWithBinaryOperatorError records an error and the operation that caused it.
type StartsWithBinaryOperatorError struct {
Op string
Node *ast.OperatorNode
}
func (e *StartsWithBinaryOperatorError) Error() string {
return "the expression can't begin from a binary operator: '" + e.Op + "'"
func (e StartsWithBinaryOperatorError) Error() string {
return "the expression can't begin from a binary operator: '" + e.Node.Value + "'"
}
// NamedGroupInvalidNodesError records an error and the operation that caused it.
type NamedGroupInvalidNodesError struct {
Node ast.Node
}
func (e NamedGroupInvalidNodesError) Error() string {
return fmt.Errorf(
"'%T' - '%v' - '%v' is not valid",
e.Node,
ast.NodeKey(e.Node),
ast.NodeValue(e.Node),
).Error()
}

48
services/search/pkg/query/kql/factory.go
View File

@@ -38,31 +38,16 @@ func buildAST(n interface{}, text []byte, pos position) (*ast.Ast, error) {
return nil, err
}
normalizedNodes, err := NormalizeNodes(nodes)
if err != nil {
a := &ast.Ast{
Base: b,
Nodes: connectNodes(DefaultConnector{sameKeyOPValue: BoolOR}, nodes...),
}
if err := validateAst(a); err != nil {
return nil, err
}
return &ast.Ast{
Base: b,
Nodes: normalizedNodes,
}, nil
}
func buildNodes(e interface{}) ([]ast.Node, error) {
maybeNodesGroups := toIfaceSlice(e)
nodes := make([]ast.Node, len(maybeNodesGroups))
for i, maybeNodesGroup := range maybeNodesGroups {
node, err := toNode[ast.Node](toIfaceSlice(maybeNodesGroup)[1])
if err != nil {
return nil, err
}
nodes[i] = node
}
return nodes, nil
return a, nil
}
func buildStringNode(k, v interface{}, text []byte, pos position) (*ast.StringNode, error) {
@@ -151,6 +136,13 @@ func buildOperatorNode(text []byte, pos position) (*ast.OperatorNode, error) {
return nil, err
}
switch value {
case "+":
value = BoolAND
case "-":
value = BoolNOT
}
return &ast.OperatorNode{
Base: b,
Value: value,
@@ -170,9 +162,15 @@ func buildGroupNode(k, n interface{}, text []byte, pos position) (*ast.GroupNode
return nil, err
}
return &ast.GroupNode{
gn := &ast.GroupNode{
Base: b,
Key: key,
Nodes: nodes,
}, nil
Nodes: connectNodes(DefaultConnector{sameKeyOPValue: BoolOR}, nodes...),
}
if err := validateGroupNode(gn); err != nil {
return nil, err
}
return gn, nil
}

2
services/search/pkg/query/kql/gen.go
View File

@@ -1,3 +1,3 @@
package kql
//go:generate go run github.com/mna/pigeon -o dictionary_gen.go dictionary.peg
//go:generate go run github.com/mna/pigeon -optimize-grammar -optimize-parser -o dictionary_gen.go dictionary.peg

28
services/search/pkg/query/kql/kql.go
View File

@@ -2,9 +2,21 @@
package kql
import (
"errors"
"github.com/owncloud/ocis/v2/services/search/pkg/query/ast"
)
// The operator node value definition
const (
// BoolAND connect two nodes with "AND"
BoolAND = "AND"
// BoolOR connect two nodes with "OR"
BoolOR = "OR"
// BoolNOT connect two nodes with "NOT"
BoolNOT = "NOT"
)
// Builder implements kql Builder interface
type Builder struct{}
@@ -12,7 +24,21 @@ type Builder struct{}
func (b Builder) Build(q string) (*ast.Ast, error) {
f, err := Parse("", []byte(q))
if err != nil {
return nil, err
var list errList
errors.As(err, &list)
for _, listError := range list {
var parserError *parserError
switch {
case errors.As(listError, &parserError):
if parserError.Inner != nil {
return nil, parserError.Inner
}
return nil, listError
}
}
}
return f.(*ast.Ast), nil
}

32
services/search/pkg/query/kql/kql_test.go
View File

@@ -5,23 +5,26 @@ import (
tAssert "github.com/stretchr/testify/assert"
"github.com/owncloud/ocis/v2/services/search/pkg/query/ast"
"github.com/owncloud/ocis/v2/services/search/pkg/query/kql"
)
func TestNewAST(t *testing.T) {
tests := []struct {
name string
givenQuery string
shouldError bool
name string
givenQuery string
expectedError error
}{
{
name: "success",
givenQuery: "foo:bar",
},
{
name: "error",
givenQuery: "AND",
shouldError: true,
name: "error",
givenQuery: kql.BoolAND,
expectedError: kql.StartsWithBinaryOperatorError{
Node: &ast.OperatorNode{Value: kql.BoolAND},
},
},
}
@@ -32,13 +35,20 @@ func TestNewAST(t *testing.T) {
t.Run(tt.name, func(t *testing.T) {
got, err := kql.Builder{}.Build(tt.givenQuery)
if tt.shouldError {
assert.NotNil(err)
if tt.expectedError != nil {
if tt.expectedError.Error() != "" {
assert.Equal(err.Error(), tt.expectedError.Error())
} else {
assert.NotNil(err)
}
assert.Nil(got)
} else {
assert.Nil(err)
assert.NotNil(got)
return
}
assert.Nil(err)
assert.NotNil(got)
})
}
}

128
services/search/pkg/query/kql/normalize.go
View File

@@ -1,128 +0,0 @@
package kql
import (
"github.com/owncloud/ocis/v2/services/search/pkg/query/ast"
)
var implicitOperatorNodeSource = "implicitly operator"
var operatorNodeAnd = ast.OperatorNode{Base: &ast.Base{Loc: &ast.Location{Source: &implicitOperatorNodeSource}}, Value: BoolAND}
var operatorNodeOr = ast.OperatorNode{Base: &ast.Base{Loc: &ast.Location{Source: &implicitOperatorNodeSource}}, Value: BoolOR}
// NormalizeNodes Populate the implicit logical operators in the ast
//
// https://learn.microsoft.com/en-us/sharepoint/dev/general-development/keyword-query-language-kql-syntax-reference#constructing-free-text-queries-using-kql
// If there are multiple free-text expressions without any operators in between them, the query behavior is the same as using the AND operator.
// "John Smith" "Jane Smith"
// This functionally is the same as using the AND Boolean operator, as follows:
// "John Smith" AND "Jane Smith"
//
// https://learn.microsoft.com/en-us/sharepoint/dev/general-development/keyword-query-language-kql-syntax-reference#using-multiple-property-restrictions-within-a-kql-query
// When you use multiple instances of the same property restriction, matches are based on the union of the property restrictions in the KQL query.
// author:"John Smith" author:"Jane Smith"
// This functionally is the same as using the OR Boolean operator, as follows:
// author:"John Smith" OR author:"Jane Smith"
//
// When you use different property restrictions, matches are based on an intersection of the property restrictions in the KQL query, as follows:
// author:"John Smith" filetype:docx
// This is the same as using the AND Boolean operator, as follows:
// author:"John Smith" AND filetype:docx
//
// https://learn.microsoft.com/en-us/sharepoint/dev/general-development/keyword-query-language-kql-syntax-reference#grouping-property-restrictions-within-a-kql-query
// author:("John Smith" "Jane Smith")
// This is the same as using the AND Boolean operator, as follows:
// author:"John Smith" AND author:"Jane Smith"
func NormalizeNodes(nodes []ast.Node) ([]ast.Node, error) {
res := make([]ast.Node, 0, len(nodes))
var currentNode ast.Node
var prevKey, currentKey *string
var operator *ast.OperatorNode
for _, node := range nodes {
switch n := node.(type) {
case *ast.StringNode:
if prevKey == nil {
prevKey = &n.Key
res = append(res, node)
continue
}
currentNode = n
currentKey = &n.Key
case *ast.DateTimeNode:
if prevKey == nil {
prevKey = &n.Key
res = append(res, node)
continue
}
currentNode = n
currentKey = &n.Key
case *ast.BooleanNode:
if prevKey == nil {
prevKey = &n.Key
res = append(res, node)
continue
}
currentNode = n
currentKey = &n.Key
case *ast.GroupNode:
var err error
n.Nodes, err = NormalizeNodes(n.Nodes)
if err != nil {
return nil, err
}
if prevKey == nil {
prevKey = &n.Key
res = append(res, n)
continue
}
currentNode = n
currentKey = &n.Key
case *ast.OperatorNode:
if n.Value == BoolNOT {
if prevKey == nil {
res = append(res, n)
} else {
operator = n
}
} else {
if prevKey == nil {
return nil, &StartsWithBinaryOperatorError{Op: n.Value}
}
prevKey = nil
res = append(res, node)
}
default:
prevKey = nil
res = append(res, node)
}
if prevKey != nil && currentKey != nil {
if *prevKey == *currentKey && *prevKey != "" {
res = append(res, &operatorNodeOr)
} else {
res = append(res, &operatorNodeAnd)
}
if operator != nil {
res = append(res, operator)
operator = nil
}
res = append(res, currentNode)
prevKey = currentKey
currentNode = nil
currentKey = nil
continue
}
}
return trimOrphan(res), nil
}
func trimOrphan(nodes []ast.Node) []ast.Node {
offset := len(nodes)
for i := len(nodes) - 1; i >= 0; i-- {
if _, ok := nodes[i].(*ast.OperatorNode); ok {
offset--
} else {
break
}
}
return nodes[:offset]
}

126
services/search/pkg/query/kql/normalize_test.go
View File

@@ -1,126 +0,0 @@
package kql_test
import (
"testing"
"time"
tAssert "github.com/stretchr/testify/assert"
"github.com/owncloud/ocis/v2/services/search/pkg/query/ast"
"github.com/owncloud/ocis/v2/services/search/pkg/query/ast/test"
"github.com/owncloud/ocis/v2/services/search/pkg/query/kql"
)
var now = time.Now()
func TestNormalizeNodes(t *testing.T) {
tests := []struct {
name string
givenNodes []ast.Node
expectedNodes []ast.Node
fixme bool
expectedError error
}{
{
name: "start with binary operator",
givenNodes: []ast.Node{
&ast.OperatorNode{Value: "OR"},
},
expectedError: &kql.StartsWithBinaryOperatorError{Op: "OR"},
},
{
name: "same key implicit OR",
givenNodes: []ast.Node{
&ast.StringNode{Key: "author", Value: "John Smith"},
&ast.StringNode{Key: "author", Value: "Jane Smith"},
},
expectedNodes: []ast.Node{
&ast.StringNode{Key: "author", Value: "John Smith"},
&ast.OperatorNode{Value: "OR"},
&ast.StringNode{Key: "author", Value: "Jane Smith"},
},
},
{
name: "no key implicit AND",
givenNodes: []ast.Node{
&ast.StringNode{Value: "John Smith"},
&ast.StringNode{Value: "Jane Smith"},
},
expectedNodes: []ast.Node{
&ast.StringNode{Value: "John Smith"},
&ast.OperatorNode{Value: "AND"},
&ast.StringNode{Value: "Jane Smith"},
},
},
{
name: "same key explicit AND",
givenNodes: []ast.Node{
&ast.StringNode{Key: "author", Value: "John Smith"},
&ast.OperatorNode{Value: "AND"},
&ast.StringNode{Key: "author", Value: "Jane Smith"},
},
expectedNodes: []ast.Node{
&ast.StringNode{Key: "author", Value: "John Smith"},
&ast.OperatorNode{Value: "AND"},
&ast.StringNode{Key: "author", Value: "Jane Smith"},
},
},
{
name: "key-group implicit AND",
// https://learn.microsoft.com/en-us/sharepoint/dev/general-development/keyword-query-language-kql-syntax-reference#grouping-property-restrictions-within-a-kql-query
fixme: true,
givenNodes: []ast.Node{
&ast.GroupNode{Key: "author", Nodes: []ast.Node{
&ast.StringNode{Key: "author", Value: "John Smith"},
&ast.StringNode{Key: "author", Value: "Jane Smith"},
}},
},
expectedNodes: []ast.Node{
&ast.GroupNode{Key: "author", Nodes: []ast.Node{
&ast.StringNode{Key: "author", Value: "John Smith"},
&ast.OperatorNode{Value: "AND"},
&ast.StringNode{Key: "author", Value: "Jane Smith"},
}},
},
},
{
name: "different key implicit AND",
givenNodes: []ast.Node{
&ast.StringNode{Key: "author", Value: "John Smith"},
&ast.StringNode{Key: "filetype", Value: "docx"},
&ast.DateTimeNode{Key: "mtime", Operator: &ast.OperatorNode{Value: "="}, Value: now},
},
expectedNodes: []ast.Node{
&ast.StringNode{Key: "author", Value: "John Smith"},
&ast.OperatorNode{Value: "AND"},
&ast.StringNode{Key: "filetype", Value: "docx"},
&ast.OperatorNode{Value: "AND"},
&ast.DateTimeNode{Key: "mtime", Operator: &ast.OperatorNode{Value: "="}, Value: now},
},
},
}
assert := tAssert.New(t)
for _, tt := range tests {
tt := tt
t.Run(tt.name, func(t *testing.T) {
if tt.fixme {
t.Skip("not implemented")
}
normalizedNodes, err := kql.NormalizeNodes(tt.givenNodes)
if tt.expectedError != nil {
assert.Equal(err, tt.expectedError)
assert.Nil(normalizedNodes)
return
}
if diff := test.DiffAst(tt.expectedNodes, normalizedNodes); diff != "" {
t.Fatalf("Nodes mismatch (-want +got): %s", diff)
}
})
}
}

37
services/search/pkg/query/kql/validate.go Normal file
View File

@@ -0,0 +1,37 @@
package kql
import (
"github.com/owncloud/ocis/v2/services/search/pkg/query/ast"
)
func validateAst(a *ast.Ast) error {
switch node := a.Nodes[0].(type) {
case *ast.OperatorNode:
switch node.Value {
case BoolAND, BoolOR:
return StartsWithBinaryOperatorError{Node: node}
}
}
return nil
}
func validateGroupNode(n *ast.GroupNode) error {
switch node := n.Nodes[0].(type) {
case *ast.OperatorNode:
switch node.Value {
case BoolAND, BoolOR:
return StartsWithBinaryOperatorError{Node: node}
}
}
if n.Key != "" {
for _, node := range n.Nodes {
if ast.NodeKey(node) != "" {
return NamedGroupInvalidNodesError{Node: node}
}
}
}
return nil
}

13
vendor/github.com/araddon/dateparse/.travis.yml generated vendored Normal file
View File

@@ -0,0 +1,13 @@
language: go
go:
- 1.13.x
before_install:
- go get -t -v ./...
script:
- go test -race -coverprofile=coverage.txt -covermode=atomic
after_success:
- bash <(curl -s https://codecov.io/bash)

21
vendor/github.com/araddon/dateparse/LICENSE generated vendored Normal file
View File

@@ -0,0 +1,21 @@
The MIT License (MIT)
Copyright (c) 2015-2017 Aaron Raddon
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

323
vendor/github.com/araddon/dateparse/README.md generated vendored Normal file
View File

@@ -0,0 +1,323 @@
Go Date Parser
---------------------------
Parse many date strings without knowing format in advance. Uses a scanner to read bytes and use a state machine to find format. Much faster than shotgun based parse methods. See [bench_test.go](https://github.com/araddon/dateparse/blob/master/bench_test.go) for performance comparison.
[![Code Coverage](https://codecov.io/gh/araddon/dateparse/branch/master/graph/badge.svg)](https://codecov.io/gh/araddon/dateparse)
[![GoDoc](https://godoc.org/github.com/araddon/dateparse?status.svg)](http://godoc.org/github.com/araddon/dateparse)
[![Build Status](https://travis-ci.org/araddon/dateparse.svg?branch=master)](https://travis-ci.org/araddon/dateparse)
[![Go ReportCard](https://goreportcard.com/badge/araddon/dateparse)](https://goreportcard.com/report/araddon/dateparse)
**MM/DD/YYYY VS DD/MM/YYYY** Right now this uses mm/dd/yyyy WHEN ambiguous if this is not desired behavior, use `ParseStrict` which will fail on ambiguous date strings.
**Timezones** The location your server is configured affects the results! See example or https://play.golang.org/p/IDHRalIyXh and last paragraph here https://golang.org/pkg/time/#Parse.
```go
// Normal parse. Equivalent Timezone rules as time.Parse()
t, err := dateparse.ParseAny("3/1/2014")
// Parse Strict, error on ambigous mm/dd vs dd/mm dates
t, err := dateparse.ParseStrict("3/1/2014")
> returns error
// Return a string that represents the layout to parse the given date-time.
layout, err := dateparse.ParseFormat("May 8, 2009 5:57:51 PM")
> "Jan 2, 2006 3:04:05 PM"
```
cli tool for testing dateformats
----------------------------------
[Date Parse CLI](https://github.com/araddon/dateparse/blob/master/dateparse)
Extended example
-------------------
https://github.com/araddon/dateparse/blob/master/example/main.go
```go
package main
import (
"flag"
"fmt"
"time"
"github.com/scylladb/termtables"
"github.com/araddon/dateparse"
)
var examples = []string{
"May 8, 2009 5:57:51 PM",
"oct 7, 1970",
"oct 7, '70",
"oct. 7, 1970",
"oct. 7, 70",
"Mon Jan 2 15:04:05 2006",
"Mon Jan 2 15:04:05 MST 2006",
"Mon Jan 02 15:04:05 -0700 2006",
"Monday, 02-Jan-06 15:04:05 MST",
"Mon, 02 Jan 2006 15:04:05 MST",
"Tue, 11 Jul 2017 16:28:13 +0200 (CEST)",
"Mon, 02 Jan 2006 15:04:05 -0700",
"Mon 30 Sep 2018 09:09:09 PM UTC",
"Mon Aug 10 15:44:11 UTC+0100 2015",
"Thu, 4 Jan 2018 17:53:36 +0000",
"Fri Jul 03 2015 18:04:07 GMT+0100 (GMT Daylight Time)",
"Sun, 3 Jan 2021 00:12:23 +0800 (GMT+08:00)",
"September 17, 2012 10:09am",
"September 17, 2012 at 10:09am PST-08",
"September 17, 2012, 10:10:09",
"October 7, 1970",
"October 7th, 1970",
"12 Feb 2006, 19:17",
"12 Feb 2006 19:17",
"14 May 2019 19:11:40.164",
"7 oct 70",
"7 oct 1970",
"03 February 2013",
"1 July 2013",
"2013-Feb-03",
// dd/Mon/yyy alpha Months
"06/Jan/2008:15:04:05 -0700",
"06/Jan/2008 15:04:05 -0700",
// mm/dd/yy
"3/31/2014",
"03/31/2014",
"08/21/71",
"8/1/71",
"4/8/2014 22:05",
"04/08/2014 22:05",
"4/8/14 22:05",
"04/2/2014 03:00:51",
"8/8/1965 12:00:00 AM",
"8/8/1965 01:00:01 PM",
"8/8/1965 01:00 PM",
"8/8/1965 1:00 PM",
"8/8/1965 12:00 AM",
"4/02/2014 03:00:51",
"03/19/2012 10:11:59",
"03/19/2012 10:11:59.3186369",
// yyyy/mm/dd
"2014/3/31",
"2014/03/31",
"2014/4/8 22:05",
"2014/04/08 22:05",
"2014/04/2 03:00:51",
"2014/4/02 03:00:51",
"2012/03/19 10:11:59",
"2012/03/19 10:11:59.3186369",
// yyyy:mm:dd
"2014:3:31",
"2014:03:31",
"2014:4:8 22:05",
"2014:04:08 22:05",
"2014:04:2 03:00:51",
"2014:4:02 03:00:51",
"2012:03:19 10:11:59",
"2012:03:19 10:11:59.3186369",
// Chinese
"2014年04月08日",
// yyyy-mm-ddThh
"2006-01-02T15:04:05+0000",
"2009-08-12T22:15:09-07:00",
"2009-08-12T22:15:09",
"2009-08-12T22:15:09.988",
"2009-08-12T22:15:09Z",
"2017-07-19T03:21:51:897+0100",
"2019-05-29T08:41-04", // no seconds, 2 digit TZ offset
// yyyy-mm-dd hh:mm:ss
"2014-04-26 17:24:37.3186369",
"2012-08-03 18:31:59.257000000",
"2014-04-26 17:24:37.123",
"2013-04-01 22:43",
"2013-04-01 22:43:22",
"2014-12-16 06:20:00 UTC",
"2014-12-16 06:20:00 GMT",
"2014-04-26 05:24:37 PM",
"2014-04-26 13:13:43 +0800",
"2014-04-26 13:13:43 +0800 +08",
"2014-04-26 13:13:44 +09:00",
"2012-08-03 18:31:59.257000000 +0000 UTC",
"2015-09-30 18:48:56.35272715 +0000 UTC",
"2015-02-18 00:12:00 +0000 GMT",
"2015-02-18 00:12:00 +0000 UTC",
"2015-02-08 03:02:00 +0300 MSK m=+0.000000001",
"2015-02-08 03:02:00.001 +0300 MSK m=+0.000000001",
"2017-07-19 03:21:51+00:00",
"2014-04-26",
"2014-04",
"2014",
"2014-05-11 08:20:13,787",
// yyyy-mm-dd-07:00
"2020-07-20+08:00",
// mm.dd.yy
"3.31.2014",
"03.31.2014",
"08.21.71",
"2014.03",
"2014.03.30",
// yyyymmdd and similar
"20140601",
"20140722105203",
// yymmdd hh:mm:yy mysql log
// 080313 05:21:55 mysqld started
"171113 14:14:20",
// unix seconds, ms, micro, nano
"1332151919",
"1384216367189",
"1384216367111222",
"1384216367111222333",
}
var (
timezone = ""
)
func main() {
flag.StringVar(&timezone, "timezone", "UTC", "Timezone aka `America/Los_Angeles` formatted time-zone")
flag.Parse()
if timezone != "" {
// NOTE: This is very, very important to understand
// time-parsing in go
loc, err := time.LoadLocation(timezone)
if err != nil {
panic(err.Error())
}
time.Local = loc
}
table := termtables.CreateTable()
table.AddHeaders("Input", "Parsed, and Output as %v")
for _, dateExample := range examples {
t, err := dateparse.ParseLocal(dateExample)
if err != nil {
panic(err.Error())
}
table.AddRow(dateExample, fmt.Sprintf("%v", t))
}
fmt.Println(table.Render())
}
/*
+-------------------------------------------------------+-----------------------------------------+
| Input | Parsed, and Output as %v |
+-------------------------------------------------------+-----------------------------------------+
| May 8, 2009 5:57:51 PM | 2009-05-08 17:57:51 +0000 UTC |
| oct 7, 1970 | 1970-10-07 00:00:00 +0000 UTC |
| oct 7, '70 | 1970-10-07 00:00:00 +0000 UTC |
| oct. 7, 1970 | 1970-10-07 00:00:00 +0000 UTC |
| oct. 7, 70 | 1970-10-07 00:00:00 +0000 UTC |
| Mon Jan 2 15:04:05 2006 | 2006-01-02 15:04:05 +0000 UTC |
| Mon Jan 2 15:04:05 MST 2006 | 2006-01-02 15:04:05 +0000 MST |
| Mon Jan 02 15:04:05 -0700 2006 | 2006-01-02 15:04:05 -0700 -0700 |
| Monday, 02-Jan-06 15:04:05 MST | 2006-01-02 15:04:05 +0000 MST |
| Mon, 02 Jan 2006 15:04:05 MST | 2006-01-02 15:04:05 +0000 MST |
| Tue, 11 Jul 2017 16:28:13 +0200 (CEST) | 2017-07-11 16:28:13 +0200 +0200 |
| Mon, 02 Jan 2006 15:04:05 -0700 | 2006-01-02 15:04:05 -0700 -0700 |
| Mon 30 Sep 2018 09:09:09 PM UTC | 2018-09-30 21:09:09 +0000 UTC |
| Mon Aug 10 15:44:11 UTC+0100 2015 | 2015-08-10 15:44:11 +0000 UTC |
| Thu, 4 Jan 2018 17:53:36 +0000 | 2018-01-04 17:53:36 +0000 UTC |
| Fri Jul 03 2015 18:04:07 GMT+0100 (GMT Daylight Time) | 2015-07-03 18:04:07 +0100 GMT |
| Sun, 3 Jan 2021 00:12:23 +0800 (GMT+08:00) | 2021-01-03 00:12:23 +0800 +0800 |
| September 17, 2012 10:09am | 2012-09-17 10:09:00 +0000 UTC |
| September 17, 2012 at 10:09am PST-08 | 2012-09-17 10:09:00 -0800 PST |
| September 17, 2012, 10:10:09 | 2012-09-17 10:10:09 +0000 UTC |
| October 7, 1970 | 1970-10-07 00:00:00 +0000 UTC |
| October 7th, 1970 | 1970-10-07 00:00:00 +0000 UTC |
| 12 Feb 2006, 19:17 | 2006-02-12 19:17:00 +0000 UTC |
| 12 Feb 2006 19:17 | 2006-02-12 19:17:00 +0000 UTC |
| 14 May 2019 19:11:40.164 | 2019-05-14 19:11:40.164 +0000 UTC |
| 7 oct 70 | 1970-10-07 00:00:00 +0000 UTC |
| 7 oct 1970 | 1970-10-07 00:00:00 +0000 UTC |
| 03 February 2013 | 2013-02-03 00:00:00 +0000 UTC |
| 1 July 2013 | 2013-07-01 00:00:00 +0000 UTC |
| 2013-Feb-03 | 2013-02-03 00:00:00 +0000 UTC |
| 06/Jan/2008:15:04:05 -0700 | 2008-01-06 15:04:05 -0700 -0700 |
| 06/Jan/2008 15:04:05 -0700 | 2008-01-06 15:04:05 -0700 -0700 |
| 3/31/2014 | 2014-03-31 00:00:00 +0000 UTC |
| 03/31/2014 | 2014-03-31 00:00:00 +0000 UTC |
| 08/21/71 | 1971-08-21 00:00:00 +0000 UTC |
| 8/1/71 | 1971-08-01 00:00:00 +0000 UTC |
| 4/8/2014 22:05 | 2014-04-08 22:05:00 +0000 UTC |
| 04/08/2014 22:05 | 2014-04-08 22:05:00 +0000 UTC |
| 4/8/14 22:05 | 2014-04-08 22:05:00 +0000 UTC |
| 04/2/2014 03:00:51 | 2014-04-02 03:00:51 +0000 UTC |
| 8/8/1965 12:00:00 AM | 1965-08-08 00:00:00 +0000 UTC |
| 8/8/1965 01:00:01 PM | 1965-08-08 13:00:01 +0000 UTC |
| 8/8/1965 01:00 PM | 1965-08-08 13:00:00 +0000 UTC |
| 8/8/1965 1:00 PM | 1965-08-08 13:00:00 +0000 UTC |
| 8/8/1965 12:00 AM | 1965-08-08 00:00:00 +0000 UTC |
| 4/02/2014 03:00:51 | 2014-04-02 03:00:51 +0000 UTC |
| 03/19/2012 10:11:59 | 2012-03-19 10:11:59 +0000 UTC |
| 03/19/2012 10:11:59.3186369 | 2012-03-19 10:11:59.3186369 +0000 UTC |
| 2014/3/31 | 2014-03-31 00:00:00 +0000 UTC |
| 2014/03/31 | 2014-03-31 00:00:00 +0000 UTC |
| 2014/4/8 22:05 | 2014-04-08 22:05:00 +0000 UTC |
| 2014/04/08 22:05 | 2014-04-08 22:05:00 +0000 UTC |
| 2014/04/2 03:00:51 | 2014-04-02 03:00:51 +0000 UTC |
| 2014/4/02 03:00:51 | 2014-04-02 03:00:51 +0000 UTC |
| 2012/03/19 10:11:59 | 2012-03-19 10:11:59 +0000 UTC |
| 2012/03/19 10:11:59.3186369 | 2012-03-19 10:11:59.3186369 +0000 UTC |
| 2014:3:31 | 2014-03-31 00:00:00 +0000 UTC |
| 2014:03:31 | 2014-03-31 00:00:00 +0000 UTC |
| 2014:4:8 22:05 | 2014-04-08 22:05:00 +0000 UTC |
| 2014:04:08 22:05 | 2014-04-08 22:05:00 +0000 UTC |
| 2014:04:2 03:00:51 | 2014-04-02 03:00:51 +0000 UTC |
| 2014:4:02 03:00:51 | 2014-04-02 03:00:51 +0000 UTC |
| 2012:03:19 10:11:59 | 2012-03-19 10:11:59 +0000 UTC |
| 2012:03:19 10:11:59.3186369 | 2012-03-19 10:11:59.3186369 +0000 UTC |
| 2014年04月08日 | 2014-04-08 00:00:00 +0000 UTC |
| 2006-01-02T15:04:05+0000 | 2006-01-02 15:04:05 +0000 UTC |
| 2009-08-12T22:15:09-07:00 | 2009-08-12 22:15:09 -0700 -0700 |
| 2009-08-12T22:15:09 | 2009-08-12 22:15:09 +0000 UTC |
| 2009-08-12T22:15:09.988 | 2009-08-12 22:15:09.988 +0000 UTC |
| 2009-08-12T22:15:09Z | 2009-08-12 22:15:09 +0000 UTC |
| 2017-07-19T03:21:51:897+0100 | 2017-07-19 03:21:51.897 +0100 +0100 |
| 2019-05-29T08:41-04 | 2019-05-29 08:41:00 -0400 -0400 |
| 2014-04-26 17:24:37.3186369 | 2014-04-26 17:24:37.3186369 +0000 UTC |
| 2012-08-03 18:31:59.257000000 | 2012-08-03 18:31:59.257 +0000 UTC |
| 2014-04-26 17:24:37.123 | 2014-04-26 17:24:37.123 +0000 UTC |
| 2013-04-01 22:43 | 2013-04-01 22:43:00 +0000 UTC |
| 2013-04-01 22:43:22 | 2013-04-01 22:43:22 +0000 UTC |
| 2014-12-16 06:20:00 UTC | 2014-12-16 06:20:00 +0000 UTC |
| 2014-12-16 06:20:00 GMT | 2014-12-16 06:20:00 +0000 UTC |
| 2014-04-26 05:24:37 PM | 2014-04-26 17:24:37 +0000 UTC |
| 2014-04-26 13:13:43 +0800 | 2014-04-26 13:13:43 +0800 +0800 |
| 2014-04-26 13:13:43 +0800 +08 | 2014-04-26 13:13:43 +0800 +0800 |
| 2014-04-26 13:13:44 +09:00 | 2014-04-26 13:13:44 +0900 +0900 |
| 2012-08-03 18:31:59.257000000 +0000 UTC | 2012-08-03 18:31:59.257 +0000 UTC |
| 2015-09-30 18:48:56.35272715 +0000 UTC | 2015-09-30 18:48:56.35272715 +0000 UTC |
| 2015-02-18 00:12:00 +0000 GMT | 2015-02-18 00:12:00 +0000 UTC |
| 2015-02-18 00:12:00 +0000 UTC | 2015-02-18 00:12:00 +0000 UTC |
| 2015-02-08 03:02:00 +0300 MSK m=+0.000000001 | 2015-02-08 03:02:00 +0300 +0300 |
| 2015-02-08 03:02:00.001 +0300 MSK m=+0.000000001 | 2015-02-08 03:02:00.001 +0300 +0300 |
| 2017-07-19 03:21:51+00:00 | 2017-07-19 03:21:51 +0000 UTC |
| 2014-04-26 | 2014-04-26 00:00:00 +0000 UTC |
| 2014-04 | 2014-04-01 00:00:00 +0000 UTC |
| 2014 | 2014-01-01 00:00:00 +0000 UTC |
| 2014-05-11 08:20:13,787 | 2014-05-11 08:20:13.787 +0000 UTC |
| 2020-07-20+08:00 | 2020-07-20 00:00:00 +0800 +0800 |
| 3.31.2014 | 2014-03-31 00:00:00 +0000 UTC |
| 03.31.2014 | 2014-03-31 00:00:00 +0000 UTC |
| 08.21.71 | 1971-08-21 00:00:00 +0000 UTC |
| 2014.03 | 2014-03-01 00:00:00 +0000 UTC |
| 2014.03.30 | 2014-03-30 00:00:00 +0000 UTC |
| 20140601 | 2014-06-01 00:00:00 +0000 UTC |
| 20140722105203 | 2014-07-22 10:52:03 +0000 UTC |
| 171113 14:14:20 | 2017-11-13 14:14:20 +0000 UTC |
| 1332151919 | 2012-03-19 10:11:59 +0000 UTC |
| 1384216367189 | 2013-11-12 00:32:47.189 +0000 UTC |
| 1384216367111222 | 2013-11-12 00:32:47.111222 +0000 UTC |
| 1384216367111222333 | 2013-11-12 00:32:47.111222333 +0000 UTC |
+-------------------------------------------------------+-----------------------------------------+
*/
```

2189
vendor/github.com/araddon/dateparse/parseany.go generated vendored Normal file
View File

File diff suppressed because it is too large Load Diff

6
vendor/github.com/mna/pigeon/.editorconfig generated vendored Normal file
View File

@@ -0,0 +1,6 @@
# See http://editorconfig.org
# In Go files we indent with tabs but still
# set indent_size to control the GitHub web viewer.
[*.go]
indent_size=4

2
vendor/github.com/mna/pigeon/.gitattributes generated vendored Normal file
View File

@@ -0,0 +1,2 @@
*.go text eol=lf
*.peg text eol=lf

22
vendor/github.com/mna/pigeon/.gitignore generated vendored Normal file
View File

@@ -0,0 +1,22 @@
# Binaries for programs and plugins
*.exe
*.dll
*.so
*.dylib
pigeon
bin/
bootstrap/cmd/bootstrap-pigeon/bootstrap-pigeon
bootstrap/cmd/bootstrap-build/bootstrap-build
bootstrap/cmd/pegscan/pegscan
bootstrap/cmd/pegparse/pegparse
# Test binary, build with `go test -c`
*.test
# Output of the go coverage tool, specifically when used with LiteIDE
*.out
# Temporary and swap files
*.swp
*.swo
*~

8
vendor/github.com/mna/pigeon/.travis.yml generated vendored Normal file
View File

@@ -0,0 +1,8 @@
language: go
script: make test
go:
- 1.11.x
- 1.12.x
- tip

33
vendor/github.com/mna/pigeon/CONTRIBUTING.md generated vendored Normal file
View File

@@ -0,0 +1,33 @@
# Contributing to pigeon
There are various ways to help support this open source project:
* if you use pigeon and find it useful, talk about it - that's probably the most basic way to help any open-source project: getting the word out that it exists and that it can be useful
* if you use pigeon and find bugs, please [file an issue][0]
* if something is poorly documented, or doesn't work as documented, this is also a bug, please [file an issue][0]
* if you can fix the issue (whether it is documentation- or code-related), then [submit a pull-request][1] - but read on to see what should be done to get it merged
* if you would like to see some new feature/behaviour being implemented, please first [open an issue][0] to discuss it because features are less likely to get merged compared to bug fixes
## Submitting a pull request
Assuming you already have a copy of the repository (either via `go get`, a github fork, a clone, etc.), you will also need `make` to regenerate all tools and files generated when a dependency changes. I use GNU make version 4.1, other versions of make may work too but haven't been tested.
Run `make` in the root directory of the repository. That will create the bootstrap builder, the bootstrap parser, and the final parser, along with some generated Go files. Once `make` is run successfully, run `go test ./...` in the root directory to make sure all tests pass.
Once this is done and tests pass, you can start implementing the bug fix (or the new feature provided **it has already been discussed and agreed in a github issue** first).
For a bug fix, the best way to proceed is to first write a test that proves the bug, then write the code that fixes the bug and makes the test pass. All other tests should still pass too (unless it relied on the buggy behaviour, in which case existing tests must be fixed).
For a new feature, it must be thoroughly tested. New code without new test(s) is unlikely to get merged.
Respect the coding style of the repository, which means essentially to respect the [coding guidelines of the Go community][2]. Use `gofmt` to format your code, and `goimports` to add and format the list of imported packages (or do it manually, but in a `goimports`-style).
Once all code is done and tests pass, regenerate the whole tree with `make`, run `make lint` to make sure the code is correct, and run tests again. You are now ready to submit the pull request.
## Licensing
All pull requests that get merged will be made available under the BSD 3-Clause license (see the LICENSE file for details), as the rest of the pigeon repository. Do not submit pull requests if you do not want your contributions to be made available under those terms.
[0]: https://github.com/mna/pigeon/issues/new
[1]: https://github.com/mna/pigeon/pulls
[2]: https://github.com/golang/go/wiki/CodeReviewComments

12
vendor/github.com/mna/pigeon/LICENSE generated vendored Normal file
View File

@@ -0,0 +1,12 @@
Copyright (c) 2015, Martin Angers & Contributors
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
* Neither the name of the author nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

198
vendor/github.com/mna/pigeon/Makefile generated vendored Normal file
View File

@@ -0,0 +1,198 @@
SHELL = /bin/bash
# force the use of go modules
export GO111MODULE = on
# directories and source code lists
ROOT = .
ROOT_SRC = $(ROOT)/*.go
BINDIR = ./bin
EXAMPLES_DIR = $(ROOT)/examples
TEST_DIR = $(ROOT)/test
# builder and ast packages
BUILDER_DIR = $(ROOT)/builder
BUILDER_SRC = $(BUILDER_DIR)/*.go
AST_DIR = $(ROOT)/ast
AST_SRC = $(AST_DIR)/*.go
# bootstrap tools variables
BOOTSTRAP_DIR = $(ROOT)/bootstrap
BOOTSTRAP_SRC = $(BOOTSTRAP_DIR)/*.go
BOOTSTRAPBUILD_DIR = $(BOOTSTRAP_DIR)/cmd/bootstrap-build
BOOTSTRAPBUILD_SRC = $(BOOTSTRAPBUILD_DIR)/*.go
BOOTSTRAPPIGEON_DIR = $(BOOTSTRAP_DIR)/cmd/bootstrap-pigeon
BOOTSTRAPPIGEON_SRC = $(BOOTSTRAPPIGEON_DIR)/*.go
STATICCODEGENERATOR_DIR = $(BOOTSTRAP_DIR)/cmd/static_code_generator
STATICCODEGENERATOR_SRC = $(STATICCODEGENERATOR_DIR)/*.go
# grammar variables
GRAMMAR_DIR = $(ROOT)/grammar
BOOTSTRAP_GRAMMAR = $(GRAMMAR_DIR)/bootstrap.peg
PIGEON_GRAMMAR = $(GRAMMAR_DIR)/pigeon.peg
TEST_GENERATED_SRC = $(patsubst %.peg,%.go,$(shell echo ./{examples,test}/**/*.peg))
all: $(BUILDER_DIR)/generated_static_code.go $(BINDIR)/static_code_generator \
$(BUILDER_DIR)/generated_static_code_range_table.go \
$(BINDIR)/bootstrap-build $(BOOTSTRAPPIGEON_DIR)/bootstrap_pigeon.go \
$(BINDIR)/bootstrap-pigeon $(ROOT)/pigeon.go $(BINDIR)/pigeon \
$(TEST_GENERATED_SRC)
$(BINDIR)/static_code_generator: $(STATICCODEGENERATOR_SRC)
go build -o $@ $(STATICCODEGENERATOR_DIR)
$(BINDIR)/bootstrap-build: $(BOOTSTRAPBUILD_SRC) $(BOOTSTRAP_SRC) $(BUILDER_SRC) \
$(AST_SRC)
go build -o $@ $(BOOTSTRAPBUILD_DIR)
$(BOOTSTRAPPIGEON_DIR)/bootstrap_pigeon.go: $(BINDIR)/bootstrap-build \
$(BOOTSTRAP_GRAMMAR)
$(BINDIR)/bootstrap-build $(BOOTSTRAP_GRAMMAR) > $@
$(BINDIR)/bootstrap-pigeon: $(BOOTSTRAPPIGEON_SRC) \
$(BOOTSTRAPPIGEON_DIR)/bootstrap_pigeon.go
go build -o $@ $(BOOTSTRAPPIGEON_DIR)
$(ROOT)/pigeon.go: $(BINDIR)/bootstrap-pigeon $(PIGEON_GRAMMAR)
$(BINDIR)/bootstrap-pigeon $(PIGEON_GRAMMAR) > $@
$(BINDIR)/pigeon: $(ROOT_SRC) $(ROOT)/pigeon.go
go build -o $@ $(ROOT)
$(BUILDER_DIR)/generated_static_code.go: $(BUILDER_DIR)/static_code.go $(BINDIR)/static_code_generator
$(BINDIR)/static_code_generator $(BUILDER_DIR)/static_code.go $@ staticCode
$(BUILDER_DIR)/generated_static_code_range_table.go: $(BUILDER_DIR)/static_code_range_table.go $(BINDIR)/static_code_generator
$(BINDIR)/static_code_generator $(BUILDER_DIR)/static_code_range_table.go $@ rangeTable0
$(BOOTSTRAP_GRAMMAR):
$(PIGEON_GRAMMAR):
# surely there's a better way to define the examples and test targets
$(EXAMPLES_DIR)/json/json.go: $(EXAMPLES_DIR)/json/json.peg $(EXAMPLES_DIR)/json/optimized/json.go $(EXAMPLES_DIR)/json/optimized-grammar/json.go $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint $< > $@
$(EXAMPLES_DIR)/json/optimized/json.go: $(EXAMPLES_DIR)/json/json.peg $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint -optimize-parser -optimize-basic-latin $< > $@
$(EXAMPLES_DIR)/json/optimized-grammar/json.go: $(EXAMPLES_DIR)/json/json.peg $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint -optimize-grammar $< > $@
$(EXAMPLES_DIR)/calculator/calculator.go: $(EXAMPLES_DIR)/calculator/calculator.peg $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint $< > $@
$(EXAMPLES_DIR)/indentation/indentation.go: $(EXAMPLES_DIR)/indentation/indentation.peg $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint $< > $@
$(TEST_DIR)/andnot/andnot.go: $(TEST_DIR)/andnot/andnot.peg $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint $< > $@
$(TEST_DIR)/predicates/predicates.go: $(TEST_DIR)/predicates/predicates.peg $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint $< > $@
$(TEST_DIR)/issue_1/issue_1.go: $(TEST_DIR)/issue_1/issue_1.peg $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint $< > $@
$(TEST_DIR)/linear/linear.go: $(TEST_DIR)/linear/linear.peg $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint $< > $@
$(TEST_DIR)/issue_18/issue_18.go: $(TEST_DIR)/issue_18/issue_18.peg $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint $< > $@
$(TEST_DIR)/runeerror/runeerror.go: $(TEST_DIR)/runeerror/runeerror.peg $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint $< > $@
$(TEST_DIR)/errorpos/errorpos.go: $(TEST_DIR)/errorpos/errorpos.peg $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint $< > $@
$(TEST_DIR)/global_store/global_store.go: $(TEST_DIR)/global_store/global_store.peg $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint $< > $@
$(TEST_DIR)/goto/goto.go: $(TEST_DIR)/goto/goto.peg $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint $< > $@
$(TEST_DIR)/goto_state/goto_state.go: $(TEST_DIR)/goto_state/goto_state.peg $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint $< > $@
$(TEST_DIR)/max_expr_cnt/maxexpr.go: $(TEST_DIR)/max_expr_cnt/maxexpr.peg $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint $< > $@
$(TEST_DIR)/labeled_failures/labeled_failures.go: $(TEST_DIR)/labeled_failures/labeled_failures.peg $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint $< > $@
$(TEST_DIR)/thrownrecover/thrownrecover.go: $(TEST_DIR)/thrownrecover/thrownrecover.peg $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint $< > $@
$(TEST_DIR)/alternate_entrypoint/altentry.go: $(TEST_DIR)/alternate_entrypoint/altentry.peg $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint -optimize-grammar -alternate-entrypoints Entry2,Entry3,C $< > $@
$(TEST_DIR)/state/state.go: $(TEST_DIR)/state/state.peg $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint -optimize-grammar $< > $@
$(TEST_DIR)/stateclone/stateclone.go: $(TEST_DIR)/stateclone/stateclone.peg $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint $< > $@
$(TEST_DIR)/statereadonly/statereadonly.go: $(TEST_DIR)/statereadonly/statereadonly.peg $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint $< > $@
$(TEST_DIR)/staterestore/staterestore.go: $(TEST_DIR)/staterestore/staterestore.peg $(TEST_DIR)/staterestore/standard/staterestore.go $(TEST_DIR)/staterestore/optimized/staterestore.go $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint $< > $@
$(TEST_DIR)/staterestore/standard/staterestore.go: $(TEST_DIR)/staterestore/staterestore.peg $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint $< > $@
$(TEST_DIR)/staterestore/optimized/staterestore.go: $(TEST_DIR)/staterestore/staterestore.peg $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint -optimize-grammar -optimize-parser -alternate-entrypoints TestAnd,TestNot $< > $@
$(TEST_DIR)/emptystate/emptystate.go: $(TEST_DIR)/emptystate/emptystate.peg $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint $< > $@
$(TEST_DIR)/issue_65/issue_65.go: $(TEST_DIR)/issue_65/issue_65.peg $(TEST_DIR)/issue_65/optimized/issue_65.go $(TEST_DIR)/issue_65/optimized-grammar/issue_65.go $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint $< > $@
$(TEST_DIR)/issue_65/optimized/issue_65.go: $(TEST_DIR)/issue_65/issue_65.peg $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint -optimize-parser -optimize-basic-latin $< > $@
$(TEST_DIR)/issue_65/optimized-grammar/issue_65.go: $(TEST_DIR)/issue_65/issue_65.peg $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint -optimize-grammar $< > $@
$(TEST_DIR)/issue_70/issue_70.go: $(TEST_DIR)/issue_70/issue_70.peg $(TEST_DIR)/issue_70/optimized/issue_70.go $(TEST_DIR)/issue_70/optimized-grammar/issue_70.go $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint $< > $@
$(TEST_DIR)/issue_70/optimized/issue_70.go: $(TEST_DIR)/issue_70/issue_70.peg $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint -optimize-parser -optimize-basic-latin $< > $@
$(TEST_DIR)/issue_70/optimized-grammar/issue_70.go: $(TEST_DIR)/issue_70/issue_70.peg $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint -optimize-grammar $< > $@
$(TEST_DIR)/issue_70b/issue_70b.go: $(TEST_DIR)/issue_70b/issue_70b.peg $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint --optimize-grammar $< > $@
$(TEST_DIR)/issue_80/issue_80.go: $(TEST_DIR)/issue_80/issue_80.peg $(BINDIR)/pigeon
$(BINDIR)/pigeon -nolint $< > $@
lint:
golint ./...
go vet ./...
gometalinter:
gometalinter ./...
cmp:
@boot=$$(mktemp) && $(BINDIR)/bootstrap-pigeon $(PIGEON_GRAMMAR) > $$boot && \
official=$$(mktemp) && $(BINDIR)/pigeon $(PIGEON_GRAMMAR) > $$official && \
cmp $$boot $$official && \
unlink $$boot && \
unlink $$official
test:
go test -v ./...
clean:
rm -f $(BUILDER_DIR)/generated_static_code.go $(BUILDER_DIR)/generated_static_code_range_table.go
rm -f $(BOOTSTRAPPIGEON_DIR)/bootstrap_pigeon.go $(ROOT)/pigeon.go $(TEST_GENERATED_SRC) $(EXAMPLES_DIR)/json/optimized/json.go $(EXAMPLES_DIR)/json/optimized-grammar/json.go $(TEST_DIR)/staterestore/optimized/staterestore.go $(TEST_DIR)/staterestore/standard/staterestore.go $(TEST_DIR)/issue_65/optimized/issue_65.go $(TEST_DIR)/issue_65/optimized-grammar/issue_65.go
rm -rf $(BINDIR)
.PHONY: all clean lint gometalinter cmp test

148
vendor/github.com/mna/pigeon/README.md generated vendored Normal file
View File

@@ -0,0 +1,148 @@
# pigeon - a PEG parser generator for Go
[![GoDoc](https://godoc.org/github.com/mna/pigeon?status.png)](https://godoc.org/github.com/mna/pigeon)
[![build status](https://secure.travis-ci.org/mna/pigeon.png?branch=master)](http://travis-ci.org/mna/pigeon)
[![GoReportCard](https://goreportcard.com/badge/github.com/mna/pigeon)](https://goreportcard.com/report/github.com/mna/pigeon)
[![Software License](https://img.shields.io/badge/license-BSD-blue.svg)](LICENSE)
The pigeon command generates parsers based on a [parsing expression grammar (PEG)][0]. Its grammar and syntax is inspired by the [PEG.js project][1], while the implementation is loosely based on the [parsing expression grammar for C# 3.0][2] article. It parses Unicode text encoded in UTF-8.
See the [godoc page][3] for detailed usage. Also have a look at the [Pigeon Wiki](https://github.com/mna/pigeon/wiki) for additional information about Pigeon and PEG in general.
## Releases
* v1.0.0 is the tagged release of the original implementation.
* Work has started on v2.0.0 with some planned breaking changes.
Github user [@mna][6] created the package in April 2015, and [@breml][5] is the package's maintainer as of May 2017.
### Breaking Changes since v1.0.0
* Removed support for Go < v1.11 to support go modules for dependency tracking.
* Removed support for Go < v1.9 due to the requirement [golang.org/x/tools/imports](https://godoc.org/golang.org/x/tools/imports), which was updated to reflect changes in recent versions of Go. This is in compliance with the [Go Release Policy](https://golang.org/doc/devel/release.html#policy) respectively the [Go Release Maintenance](https://github.com/golang/go/wiki/Go-Release-Cycle#release-maintenance), which states support for each major release until there are two newer major releases.
## Installation
Provided you have Go correctly installed with the $GOPATH and $GOBIN environment variables set, run:
```
$ go get -u github.com/mna/pigeon
```
This will install or update the package, and the `pigeon` command will be installed in your $GOBIN directory. Neither this package nor the parsers generated by this command require any third-party dependency, unless such a dependency is used in the code blocks of the grammar.
## Basic usage
```
$ pigeon [options] [PEG_GRAMMAR_FILE]
```
By default, the input grammar is read from `stdin` and the generated code is printed to `stdout`. You may save it in a file using the `-o` flag.
## Example
Given the following grammar:
```
{
// part of the initializer code block omitted for brevity
var ops = map[string]func(int, int) int {
"+": func(l, r int) int {
return l + r
},
"-": func(l, r int) int {
return l - r
},
"*": func(l, r int) int {
return l * r
},
"/": func(l, r int) int {
return l / r
},
}
func toIfaceSlice(v interface{}) []interface{} {
if v == nil {
return nil
}
return v.([]interface{})
}
func eval(first, rest interface{}) int {
l := first.(int)
restSl := toIfaceSlice(rest)
for _, v := range restSl {
restExpr := toIfaceSlice(v)
r := restExpr[3].(int)
op := restExpr[1].(string)
l = ops[op](l, r)
}
return l
}
}
Input <- expr:Expr EOF {
return expr, nil
}
Expr <- _ first:Term rest:( _ AddOp _ Term )* _ {
return eval(first, rest), nil
}
Term <- first:Factor rest:( _ MulOp _ Factor )* {
return eval(first, rest), nil
}
Factor <- '(' expr:Expr ')' {
return expr, nil
} / integer:Integer {
return integer, nil
}
AddOp <- ( '+' / '-' ) {
return string(c.text), nil
}
MulOp <- ( '*' / '/' ) {
return string(c.text), nil
}
Integer <- '-'? [0-9]+ {
return strconv.Atoi(string(c.text))
}
_ "whitespace" <- [ \n\t\r]*
EOF <- !.
```
The generated parser can parse simple arithmetic operations, e.g.:
```
18 + 3 - 27 * (-18 / -3)
=> -141
```
More examples can be found in the `examples/` subdirectory.
See the [godoc page][3] for detailed usage.
## Contributing
See the CONTRIBUTING.md file.
## License
The [BSD 3-Clause license][4]. See the LICENSE file.
[0]: http://en.wikipedia.org/wiki/Parsing_expression_grammar
[1]: http://pegjs.org/
[2]: http://www.codeproject.com/Articles/29713/Parsing-Expression-Grammar-Support-for-C-Part
[3]: https://godoc.org/github.com/mna/pigeon
[4]: http://opensource.org/licenses/BSD-3-Clause
[5]: https://github.com/breml
[6]: https://github.com/mna

3
vendor/github.com/mna/pigeon/TODO generated vendored Normal file
View File

@@ -0,0 +1,3 @@
- refactor implementation as a VM to avoid stack overflow in pathological cases (and maybe better performance): in branch wip-vm
? options like current receiver name read directly from the grammar file
? type annotations for generated code functions

662
vendor/github.com/mna/pigeon/ast/ast.go generated vendored Normal file
View File

@@ -0,0 +1,662 @@
// Package ast defines the abstract syntax tree for the PEG grammar.
//
// The parser generator's PEG grammar generates a tree using this package
// that is then converted by the builder to the simplified AST used in
// the generated parser.
package ast
import (
"bytes"
"fmt"
"strconv"
"strings"
)
// Pos represents a position in a source file.
type Pos struct {
Filename string
Line int
Col int
Off int
}
// String returns the textual representation of a position.
func (p Pos) String() string {
if p.Filename != "" {
return fmt.Sprintf("%s:%d:%d (%d)", p.Filename, p.Line, p.Col, p.Off)
}
return fmt.Sprintf("%d:%d (%d)", p.Line, p.Col, p.Off)
}
// Grammar is the top-level node of the AST for the PEG grammar.
type Grammar struct {
p Pos
Init *CodeBlock
Rules []*Rule
}
// NewGrammar creates a new grammar at the specified position.
func NewGrammar(p Pos) *Grammar {
return &Grammar{p: p}
}
// Pos returns the starting position of the node.
func (g *Grammar) Pos() Pos { return g.p }
// String returns the textual representation of a node.
func (g *Grammar) String() string {
var buf bytes.Buffer
buf.WriteString(fmt.Sprintf("%s: %T{Init: %v, Rules: [\n",
g.p, g, g.Init))
for _, r := range g.Rules {
buf.WriteString(fmt.Sprintf("%s,\n", r))
}
buf.WriteString("]}")
return buf.String()
}
// Rule represents a rule in the PEG grammar. It has a name, an optional
// display name to be used in error messages, and an expression.
type Rule struct {
p Pos
Name *Identifier
DisplayName *StringLit
Expr Expression
}
// NewRule creates a rule with at the specified position and with the
// specified name as identifier.
func NewRule(p Pos, name *Identifier) *Rule {
return &Rule{p: p, Name: name}
}
// Pos returns the starting position of the node.
func (r *Rule) Pos() Pos { return r.p }
// String returns the textual representation of a node.
func (r *Rule) String() string {
return fmt.Sprintf("%s: %T{Name: %v, DisplayName: %v, Expr: %v}",
r.p, r, r.Name, r.DisplayName, r.Expr)
}
// Expression is the interface implemented by all expression types.
type Expression interface {
Pos() Pos
}
// ChoiceExpr is an ordered sequence of expressions. The parser tries to
// match any of the alternatives in sequence and stops at the first one
// that matches.
type ChoiceExpr struct {
p Pos
Alternatives []Expression
}
// NewChoiceExpr creates a choice expression at the specified position.
func NewChoiceExpr(p Pos) *ChoiceExpr {
return &ChoiceExpr{p: p}
}
// Pos returns the starting position of the node.
func (c *ChoiceExpr) Pos() Pos { return c.p }
// String returns the textual representation of a node.
func (c *ChoiceExpr) String() string {
var buf bytes.Buffer
buf.WriteString(fmt.Sprintf("%s: %T{Alternatives: [\n", c.p, c))
for _, e := range c.Alternatives {
buf.WriteString(fmt.Sprintf("%s,\n", e))
}
buf.WriteString("]}")
return buf.String()
}
// FailureLabel is an identifier, which can by thrown and recovered in a grammar
type FailureLabel string
// RecoveryExpr is an ordered sequence of expressions. The parser tries to
// match any of the alternatives in sequence and stops at the first one
// that matches.
type RecoveryExpr struct {
p Pos
Expr Expression
RecoverExpr Expression
Labels []FailureLabel
}
// NewRecoveryExpr creates a choice expression at the specified position.
func NewRecoveryExpr(p Pos) *RecoveryExpr {
return &RecoveryExpr{p: p}
}
// Pos returns the starting position of the node.
func (r *RecoveryExpr) Pos() Pos { return r.p }
// String returns the textual representation of a node.
func (r *RecoveryExpr) String() string {
var buf bytes.Buffer
buf.WriteString(fmt.Sprintf("%s: %T{Expr: %v, RecoverExpr: %v", r.p, r, r.Expr, r.RecoverExpr))
buf.WriteString(fmt.Sprintf(", Labels: [\n"))
for _, e := range r.Labels {
buf.WriteString(fmt.Sprintf("%s,\n", e))
}
buf.WriteString("]}")
return buf.String()
}
// ActionExpr is an expression that has an associated block of code to
// execute when the expression matches.
type ActionExpr struct {
p Pos
Expr Expression
Code *CodeBlock
FuncIx int
}
// NewActionExpr creates a new action expression at the specified position.
func NewActionExpr(p Pos) *ActionExpr {
return &ActionExpr{p: p}
}
// Pos returns the starting position of the node.
func (a *ActionExpr) Pos() Pos { return a.p }
// String returns the textual representation of a node.
func (a *ActionExpr) String() string {
return fmt.Sprintf("%s: %T{Expr: %v, Code: %v}", a.p, a, a.Expr, a.Code)
}
// ThrowExpr is an expression that throws an FailureLabel to be catched by a
// RecoveryChoiceExpr.
type ThrowExpr struct {
p Pos
Label string
}
// NewThrowExpr creates a new throw expression at the specified position.
func NewThrowExpr(p Pos) *ThrowExpr {
return &ThrowExpr{p: p}
}
// Pos returns the starting position of the node.
func (t *ThrowExpr) Pos() Pos { return t.p }
// String returns the textual representation of a node.
func (t *ThrowExpr) String() string {
return fmt.Sprintf("%s: %T{Label: %v}", t.p, t, t.Label)
}
// SeqExpr is an ordered sequence of expressions, all of which must match
// if the SeqExpr is to be a match itself.
type SeqExpr struct {
p Pos
Exprs []Expression
}
// NewSeqExpr creates a new sequence expression at the specified position.
func NewSeqExpr(p Pos) *SeqExpr {
return &SeqExpr{p: p}
}
// Pos returns the starting position of the node.
func (s *SeqExpr) Pos() Pos { return s.p }
// String returns the textual representation of a node.
func (s *SeqExpr) String() string {
var buf bytes.Buffer
buf.WriteString(fmt.Sprintf("%s: %T{Exprs: [\n", s.p, s))
for _, e := range s.Exprs {
buf.WriteString(fmt.Sprintf("%s,\n", e))
}
buf.WriteString("]}")
return buf.String()
}
// LabeledExpr is an expression that has an associated label. Code blocks
// can access the value of the expression using that label, that becomes
// a local variable in the code.
type LabeledExpr struct {
p Pos
Label *Identifier
Expr Expression
}
// NewLabeledExpr creates a new labeled expression at the specified position.
func NewLabeledExpr(p Pos) *LabeledExpr {
return &LabeledExpr{p: p}
}
// Pos returns the starting position of the node.
func (l *LabeledExpr) Pos() Pos { return l.p }
// String returns the textual representation of a node.
func (l *LabeledExpr) String() string {
return fmt.Sprintf("%s: %T{Label: %v, Expr: %v}", l.p, l, l.Label, l.Expr)
}
// AndExpr is a zero-length matcher that is considered a match if the
// expression it contains is a match.
type AndExpr struct {
p Pos
Expr Expression
}
// NewAndExpr creates a new and (&) expression at the specified position.
func NewAndExpr(p Pos) *AndExpr {
return &AndExpr{p: p}
}
// Pos returns the starting position of the node.
func (a *AndExpr) Pos() Pos { return a.p }
// String returns the textual representation of a node.
func (a *AndExpr) String() string {
return fmt.Sprintf("%s: %T{Expr: %v}", a.p, a, a.Expr)
}
// NotExpr is a zero-length matcher that is considered a match if the
// expression it contains is not a match.
type NotExpr struct {
p Pos
Expr Expression
}
// NewNotExpr creates a new not (!) expression at the specified position.
func NewNotExpr(p Pos) *NotExpr {
return &NotExpr{p: p}
}
// Pos returns the starting position of the node.
func (n *NotExpr) Pos() Pos { return n.p }
// String returns the textual representation of a node.
func (n *NotExpr) String() string {
return fmt.Sprintf("%s: %T{Expr: %v}", n.p, n, n.Expr)
}
// ZeroOrOneExpr is an expression that can be matched zero or one time.
type ZeroOrOneExpr struct {
p Pos
Expr Expression
}
// NewZeroOrOneExpr creates a new zero or one expression at the specified
// position.
func NewZeroOrOneExpr(p Pos) *ZeroOrOneExpr {
return &ZeroOrOneExpr{p: p}
}
// Pos returns the starting position of the node.
func (z *ZeroOrOneExpr) Pos() Pos { return z.p }
// String returns the textual representation of a node.
func (z *ZeroOrOneExpr) String() string {
return fmt.Sprintf("%s: %T{Expr: %v}", z.p, z, z.Expr)
}
// ZeroOrMoreExpr is an expression that can be matched zero or more times.
type ZeroOrMoreExpr struct {
p Pos
Expr Expression
}
// NewZeroOrMoreExpr creates a new zero or more expression at the specified
// position.
func NewZeroOrMoreExpr(p Pos) *ZeroOrMoreExpr {
return &ZeroOrMoreExpr{p: p}
}
// Pos returns the starting position of the node.
func (z *ZeroOrMoreExpr) Pos() Pos { return z.p }
// String returns the textual representation of a node.
func (z *ZeroOrMoreExpr) String() string {
return fmt.Sprintf("%s: %T{Expr: %v}", z.p, z, z.Expr)
}
// OneOrMoreExpr is an expression that can be matched one or more times.
type OneOrMoreExpr struct {
p Pos
Expr Expression
}
// NewOneOrMoreExpr creates a new one or more expression at the specified
// position.
func NewOneOrMoreExpr(p Pos) *OneOrMoreExpr {
return &OneOrMoreExpr{p: p}
}
// Pos returns the starting position of the node.
func (o *OneOrMoreExpr) Pos() Pos { return o.p }
// String returns the textual representation of a node.
func (o *OneOrMoreExpr) String() string {
return fmt.Sprintf("%s: %T{Expr: %v}", o.p, o, o.Expr)
}
// RuleRefExpr is an expression that references a rule by name.
type RuleRefExpr struct {
p Pos
Name *Identifier
}
// NewRuleRefExpr creates a new rule reference expression at the specified
// position.
func NewRuleRefExpr(p Pos) *RuleRefExpr {
return &RuleRefExpr{p: p}
}
// Pos returns the starting position of the node.
func (r *RuleRefExpr) Pos() Pos { return r.p }
// String returns the textual representation of a node.
func (r *RuleRefExpr) String() string {
return fmt.Sprintf("%s: %T{Name: %v}", r.p, r, r.Name)
}
// StateCodeExpr is an expression which can modify the internal state of the parser.
type StateCodeExpr struct {
p Pos
Code *CodeBlock
FuncIx int
}
// NewStateCodeExpr creates a new state (#) code expression at the specified
// position.
func NewStateCodeExpr(p Pos) *StateCodeExpr {
return &StateCodeExpr{p: p}
}
// Pos returns the starting position of the node.
func (s *StateCodeExpr) Pos() Pos { return s.p }
// String returns the textual representation of a node.
func (s *StateCodeExpr) String() string {
return fmt.Sprintf("%s: %T{Code: %v}", s.p, s, s.Code)
}
// AndCodeExpr is a zero-length matcher that is considered a match if the
// code block returns true.
type AndCodeExpr struct {
p Pos
Code *CodeBlock
FuncIx int
}
// NewAndCodeExpr creates a new and (&) code expression at the specified
// position.
func NewAndCodeExpr(p Pos) *AndCodeExpr {
return &AndCodeExpr{p: p}
}
// Pos returns the starting position of the node.
func (a *AndCodeExpr) Pos() Pos { return a.p }
// String returns the textual representation of a node.
func (a *AndCodeExpr) String() string {
return fmt.Sprintf("%s: %T{Code: %v}", a.p, a, a.Code)
}
// NotCodeExpr is a zero-length matcher that is considered a match if the
// code block returns false.
type NotCodeExpr struct {
p Pos
Code *CodeBlock
FuncIx int
}
// NewNotCodeExpr creates a new not (!) code expression at the specified
// position.
func NewNotCodeExpr(p Pos) *NotCodeExpr {
return &NotCodeExpr{p: p}
}
// Pos returns the starting position of the node.
func (n *NotCodeExpr) Pos() Pos { return n.p }
// String returns the textual representation of a node.
func (n *NotCodeExpr) String() string {
return fmt.Sprintf("%s: %T{Code: %v}", n.p, n, n.Code)
}
// LitMatcher is a string literal matcher. The value to match may be a
// double-quoted string, a single-quoted single character, or a back-tick
// quoted raw string.
type LitMatcher struct {
posValue // can be str, rstr or char
IgnoreCase bool
}
// NewLitMatcher creates a new literal matcher at the specified position and
// with the specified value.
func NewLitMatcher(p Pos, v string) *LitMatcher {
return &LitMatcher{posValue: posValue{p: p, Val: v}}
}
// Pos returns the starting position of the node.
func (l *LitMatcher) Pos() Pos { return l.p }
// String returns the textual representation of a node.
func (l *LitMatcher) String() string {
return fmt.Sprintf("%s: %T{Val: %q, IgnoreCase: %t}", l.p, l, l.Val, l.IgnoreCase)
}
// CharClassMatcher is a character class matcher. The value to match must
// be one of the specified characters, in a range of characters, or in the
// Unicode classes of characters.
type CharClassMatcher struct {
posValue
IgnoreCase bool
Inverted bool
Chars []rune
Ranges []rune // pairs of low/high range
UnicodeClasses []string
}
// NewCharClassMatcher creates a new character class matcher at the specified
// position and with the specified raw value. It parses the raw value into
// the list of characters, ranges and Unicode classes.
func NewCharClassMatcher(p Pos, raw string) *CharClassMatcher {
c := &CharClassMatcher{posValue: posValue{p: p, Val: raw}}
c.parse()
return c
}
func (c *CharClassMatcher) parse() {
raw := c.Val
c.IgnoreCase = strings.HasSuffix(raw, "i")
if c.IgnoreCase {
raw = raw[:len(raw)-1]
}
// "unquote" the character classes
raw = raw[1 : len(raw)-1]
if len(raw) == 0 {
return
}
c.Inverted = raw[0] == '^'
if c.Inverted {
raw = raw[1:]
if len(raw) == 0 {
return
}
}
// content of char class is necessarily valid, so escapes are correct
r := strings.NewReader(raw)
var chars []rune
var buf bytes.Buffer
outer:
for {
rn, _, err := r.ReadRune()
if err != nil {
break outer
}
consumeN := 0
switch rn {
case '\\':
rn, _, _ := r.ReadRune()
switch rn {
case ']':
chars = append(chars, rn)
continue
case 'p':
rn, _, _ := r.ReadRune()
if rn == '{' {
buf.Reset()
for {
rn, _, _ := r.ReadRune()
if rn == '}' {
break
}
buf.WriteRune(rn)
}
c.UnicodeClasses = append(c.UnicodeClasses, buf.String())
} else {
c.UnicodeClasses = append(c.UnicodeClasses, string(rn))
}
continue
case 'x':
consumeN = 2
case 'u':
consumeN = 4
case 'U':
consumeN = 8
case '0', '1', '2', '3', '4', '5', '6', '7':
consumeN = 2
}
buf.Reset()
buf.WriteRune(rn)
for i := 0; i < consumeN; i++ {
rn, _, _ := r.ReadRune()
buf.WriteRune(rn)
}
rn, _, _, _ = strconv.UnquoteChar("\\"+buf.String(), 0)
chars = append(chars, rn)
default:
chars = append(chars, rn)
}
}
// extract ranges and chars
inRange, wasRange := false, false
for i, r := range chars {
if inRange {
c.Ranges = append(c.Ranges, r)
inRange = false
wasRange = true
continue
}
if r == '-' && !wasRange && len(c.Chars) > 0 && i < len(chars)-1 {
inRange = true
wasRange = false
// start of range is the last Char added
c.Ranges = append(c.Ranges, c.Chars[len(c.Chars)-1])
c.Chars = c.Chars[:len(c.Chars)-1]
continue
}
wasRange = false
c.Chars = append(c.Chars, r)
}
}
// Pos returns the starting position of the node.
func (c *CharClassMatcher) Pos() Pos { return c.p }
// String returns the textual representation of a node.
func (c *CharClassMatcher) String() string {
return fmt.Sprintf("%s: %T{Val: %q, IgnoreCase: %t, Inverted: %t}",
c.p, c, c.Val, c.IgnoreCase, c.Inverted)
}
// AnyMatcher is a matcher that matches any character except end-of-file.
type AnyMatcher struct {
posValue
}
// NewAnyMatcher creates a new any matcher at the specified position. The
// value is provided for completeness' sake, but it is always the dot.
func NewAnyMatcher(p Pos, v string) *AnyMatcher {
return &AnyMatcher{posValue{p, v}}
}
// Pos returns the starting position of the node.
func (a *AnyMatcher) Pos() Pos { return a.p }
// String returns the textual representation of a node.
func (a *AnyMatcher) String() string {
return fmt.Sprintf("%s: %T{Val: %q}", a.p, a, a.Val)
}
// CodeBlock represents a code block.
type CodeBlock struct {
posValue
}
// NewCodeBlock creates a new code block at the specified position and with
// the specified value. The value includes the outer braces.
func NewCodeBlock(p Pos, code string) *CodeBlock {
return &CodeBlock{posValue{p, code}}
}
// Pos returns the starting position of the node.
func (c *CodeBlock) Pos() Pos { return c.p }
// String returns the textual representation of a node.
func (c *CodeBlock) String() string {
return fmt.Sprintf("%s: %T{Val: %q}", c.p, c, c.Val)
}
// Identifier represents an identifier.
type Identifier struct {
posValue
}
// NewIdentifier creates a new identifier at the specified position and
// with the specified name.
func NewIdentifier(p Pos, name string) *Identifier {
return &Identifier{posValue{p: p, Val: name}}
}
// Pos returns the starting position of the node.
func (i *Identifier) Pos() Pos { return i.p }
// String returns the textual representation of a node.
func (i *Identifier) String() string {
return fmt.Sprintf("%s: %T{Val: %q}", i.p, i, i.Val)
}
// StringLit represents a string literal.
type StringLit struct {
posValue
}
// NewStringLit creates a new string literal at the specified position and
// with the specified value.
func NewStringLit(p Pos, val string) *StringLit {
return &StringLit{posValue{p: p, Val: val}}
}
// Pos returns the starting position of the node.
func (s *StringLit) Pos() Pos { return s.p }
// String returns the textual representation of a node.
func (s *StringLit) String() string {
return fmt.Sprintf("%s: %T{Val: %q}", s.p, s, s.Val)
}
type posValue struct {
p Pos
Val string
}

469
vendor/github.com/mna/pigeon/ast/ast_optimize.go generated vendored Normal file
View File

@@ -0,0 +1,469 @@
package ast
import (
"bytes"
"strconv"
"strings"
)
type grammarOptimizer struct {
rule string
protectedRules map[string]struct{}
rules map[string]*Rule
ruleUsesRules map[string]map[string]struct{}
ruleUsedByRules map[string]map[string]struct{}
visitor func(expr Expression) Visitor
optimized bool
}
func newGrammarOptimizer(protectedRules []string) *grammarOptimizer {
pr := make(map[string]struct{}, len(protectedRules))
for _, nm := range protectedRules {
pr[nm] = struct{}{}
}
r := grammarOptimizer{
protectedRules: pr,
rules: make(map[string]*Rule),
ruleUsesRules: make(map[string]map[string]struct{}),
ruleUsedByRules: make(map[string]map[string]struct{}),
}
r.visitor = r.init
return &r
}
// Visit is a generic Visitor to be used with Walk
// The actual function, which should be used during Walk
// is held in ruleRefOptimizer.visitor
func (r *grammarOptimizer) Visit(expr Expression) Visitor {
return r.visitor(expr)
}
// init is a Visitor, which is used with the Walk function
// The purpose of this function is to initialize the reference
// maps rules, ruleUsesRules and ruleUsedByRules.
func (r *grammarOptimizer) init(expr Expression) Visitor {
switch expr := expr.(type) {
case *Rule:
// Keep track of current rule, which is processed
r.rule = expr.Name.Val
r.rules[expr.Name.Val] = expr
case *RuleRefExpr:
// Fill ruleUsesRules and ruleUsedByRules for every RuleRefExpr
set(r.ruleUsesRules, r.rule, expr.Name.Val)
set(r.ruleUsedByRules, expr.Name.Val, r.rule)
}
return r
}
// Add element to map of maps, initialize the inner map
// if necessary.
func set(m map[string]map[string]struct{}, src, dst string) {
if _, ok := m[src]; !ok {
m[src] = make(map[string]struct{})
}
m[src][dst] = struct{}{}
}
// optimize is a Visitor, which is used with the Walk function
// The purpose of this function is to perform the actual optimizations.
// See Optimize for a detailed list of the performed optimizations.
func (r *grammarOptimizer) optimize(expr0 Expression) Visitor {
switch expr := expr0.(type) {
case *ActionExpr:
expr.Expr = r.optimizeRule(expr.Expr)
case *AndExpr:
expr.Expr = r.optimizeRule(expr.Expr)
case *ChoiceExpr:
expr.Alternatives = r.optimizeRules(expr.Alternatives)
// Optimize choice nested in choice
for i := 0; i < len(expr.Alternatives); i++ {
if choice, ok := expr.Alternatives[i].(*ChoiceExpr); ok {
r.optimized = true
if i+1 < len(expr.Alternatives) {
expr.Alternatives = append(expr.Alternatives[:i], append(choice.Alternatives, expr.Alternatives[i+1:]...)...)
} else {
expr.Alternatives = append(expr.Alternatives[:i], choice.Alternatives...)
}
}
// Combine sequence of single char LitMatcher to CharClassMatcher
if i > 0 {
l0, lok0 := expr.Alternatives[i-1].(*LitMatcher)
l1, lok1 := expr.Alternatives[i].(*LitMatcher)
c0, cok0 := expr.Alternatives[i-1].(*CharClassMatcher)
c1, cok1 := expr.Alternatives[i].(*CharClassMatcher)
combined := false
switch {
// Combine two LitMatcher to CharClassMatcher
// "a" / "b" => [ab]
case lok0 && lok1 && len([]rune(l0.Val)) == 1 && len([]rune(l1.Val)) == 1 && l0.IgnoreCase == l1.IgnoreCase:
combined = true
cm := CharClassMatcher{
Chars: append([]rune(l0.Val), []rune(l1.Val)...),
IgnoreCase: l0.IgnoreCase,
posValue: l0.posValue,
}
expr.Alternatives[i-1] = &cm
// Combine LitMatcher with CharClassMatcher
// "a" / [bc] => [abc]
case lok0 && cok1 && len([]rune(l0.Val)) == 1 && l0.IgnoreCase == c1.IgnoreCase && !c1.Inverted:
combined = true
c1.Chars = append(c1.Chars, []rune(l0.Val)...)
expr.Alternatives[i-1] = c1
// Combine CharClassMatcher with LitMatcher
// [ab] / "c" => [abc]
case cok0 && lok1 && len([]rune(l1.Val)) == 1 && c0.IgnoreCase == l1.IgnoreCase && !c0.Inverted:
combined = true
c0.Chars = append(c0.Chars, []rune(l1.Val)...)
// Combine CharClassMatcher with CharClassMatcher
// [ab] / [cd] => [abcd]
case cok0 && cok1 && c0.IgnoreCase == c1.IgnoreCase && c0.Inverted == c1.Inverted:
combined = true
c0.Chars = append(c0.Chars, c1.Chars...)
c0.Ranges = append(c0.Ranges, c1.Ranges...)
c0.UnicodeClasses = append(c0.UnicodeClasses, c1.UnicodeClasses...)
}
// If one of the optimizations was applied, remove the second element from Alternatives
if combined {
r.optimized = true
if i+1 < len(expr.Alternatives) {
expr.Alternatives = append(expr.Alternatives[:i], expr.Alternatives[i+1:]...)
} else {
expr.Alternatives = expr.Alternatives[:i]
}
}
}
}
case *Grammar:
// Reset optimized at the start of each Walk.
r.optimized = false
for i := 0; i < len(expr.Rules); i++ {
rule := expr.Rules[i]
// Remove Rule, if it is no longer used by any other Rule and it is not the first Rule.
_, used := r.ruleUsedByRules[rule.Name.Val]
_, protected := r.protectedRules[rule.Name.Val]
if !used && !protected {
expr.Rules = append(expr.Rules[:i], expr.Rules[i+1:]...)
// Compensate for the removed item
i--
for k, v := range r.ruleUsedByRules {
for kk := range v {
if kk == rule.Name.Val {
delete(r.ruleUsedByRules[k], kk)
if len(r.ruleUsedByRules[k]) == 0 {
delete(r.ruleUsedByRules, k)
}
}
}
}
r.optimized = true
continue
}
}
case *LabeledExpr:
expr.Expr = r.optimizeRule(expr.Expr)
case *NotExpr:
expr.Expr = r.optimizeRule(expr.Expr)
case *OneOrMoreExpr:
expr.Expr = r.optimizeRule(expr.Expr)
case *Rule:
r.rule = expr.Name.Val
expr.Expr = r.optimizeRule(expr.Expr)
case *SeqExpr:
expr.Exprs = r.optimizeRules(expr.Exprs)
for i := 0; i < len(expr.Exprs); i++ {
// Optimize nested sequences
if seq, ok := expr.Exprs[i].(*SeqExpr); ok {
r.optimized = true
if i+1 < len(expr.Exprs) {
expr.Exprs = append(expr.Exprs[:i], append(seq.Exprs, expr.Exprs[i+1:]...)...)
} else {
expr.Exprs = append(expr.Exprs[:i], seq.Exprs...)
}
}
// Combine sequence of LitMatcher
if i > 0 {
l0, ok0 := expr.Exprs[i-1].(*LitMatcher)
l1, ok1 := expr.Exprs[i].(*LitMatcher)
if ok0 && ok1 && l0.IgnoreCase == l1.IgnoreCase {
r.optimized = true
l0.Val += l1.Val
expr.Exprs[i-1] = l0
if i+1 < len(expr.Exprs) {
expr.Exprs = append(expr.Exprs[:i], expr.Exprs[i+1:]...)
} else {
expr.Exprs = expr.Exprs[:i]
}
}
}
}
case *ZeroOrMoreExpr:
expr.Expr = r.optimizeRule(expr.Expr)
case *ZeroOrOneExpr:
expr.Expr = r.optimizeRule(expr.Expr)
}
return r
}
func (r *grammarOptimizer) optimizeRules(exprs []Expression) []Expression {
for i := 0; i < len(exprs); i++ {
exprs[i] = r.optimizeRule(exprs[i])
}
return exprs
}
func (r *grammarOptimizer) optimizeRule(expr Expression) Expression {
// Optimize RuleRefExpr
if ruleRef, ok := expr.(*RuleRefExpr); ok {
if _, ok := r.ruleUsesRules[ruleRef.Name.Val]; !ok {
r.optimized = true
delete(r.ruleUsedByRules[ruleRef.Name.Val], r.rule)
if len(r.ruleUsedByRules[ruleRef.Name.Val]) == 0 {
delete(r.ruleUsedByRules, ruleRef.Name.Val)
}
delete(r.ruleUsesRules[r.rule], ruleRef.Name.Val)
if len(r.ruleUsesRules[r.rule]) == 0 {
delete(r.ruleUsesRules, r.rule)
}
// TODO: Check if reference exists, otherwise raise an error, which reference is missing!
return cloneExpr(r.rules[ruleRef.Name.Val].Expr)
}
}
// Remove Choices with only one Alternative left
if choice, ok := expr.(*ChoiceExpr); ok {
if len(choice.Alternatives) == 1 {
r.optimized = true
return choice.Alternatives[0]
}
}
// Remove Sequence with only one Expression
if seq, ok := expr.(*SeqExpr); ok {
if len(seq.Exprs) == 1 {
r.optimized = true
return seq.Exprs[0]
}
}
return expr
}
// cloneExpr takes an Expression and deep clones it (including all children)
// This is necessary because referenced Rules are denormalized and therefore
// have to become independent from their original Expression
func cloneExpr(expr Expression) Expression {
switch expr := expr.(type) {
case *ActionExpr:
return &ActionExpr{
Code: expr.Code,
Expr: cloneExpr(expr.Expr),
FuncIx: expr.FuncIx,
p: expr.p,
}
case *AndExpr:
return &AndExpr{
Expr: cloneExpr(expr.Expr),
p: expr.p,
}
case *AndCodeExpr:
return &AndCodeExpr{
Code: expr.Code,
FuncIx: expr.FuncIx,
p: expr.p,
}
case *CharClassMatcher:
return &CharClassMatcher{
Chars: append([]rune{}, expr.Chars...),
IgnoreCase: expr.IgnoreCase,
Inverted: expr.Inverted,
posValue: expr.posValue,
Ranges: append([]rune{}, expr.Ranges...),
UnicodeClasses: append([]string{}, expr.UnicodeClasses...),
}
case *ChoiceExpr:
alts := make([]Expression, 0, len(expr.Alternatives))
for i := 0; i < len(expr.Alternatives); i++ {
alts = append(alts, cloneExpr(expr.Alternatives[i]))
}
return &ChoiceExpr{
Alternatives: alts,
p: expr.p,
}
case *LabeledExpr:
return &LabeledExpr{
Expr: cloneExpr(expr.Expr),
Label: expr.Label,
p: expr.p,
}
case *NotExpr:
return &NotExpr{
Expr: cloneExpr(expr.Expr),
p: expr.p,
}
case *NotCodeExpr:
return &NotCodeExpr{
Code: expr.Code,
FuncIx: expr.FuncIx,
p: expr.p,
}
case *OneOrMoreExpr:
return &OneOrMoreExpr{
Expr: cloneExpr(expr.Expr),
p: expr.p,
}
case *SeqExpr:
exprs := make([]Expression, 0, len(expr.Exprs))
for i := 0; i < len(expr.Exprs); i++ {
exprs = append(exprs, cloneExpr(expr.Exprs[i]))
}
return &SeqExpr{
Exprs: exprs,
p: expr.p,
}
case *StateCodeExpr:
return &StateCodeExpr{
p: expr.p,
Code: expr.Code,
FuncIx: expr.FuncIx,
}
case *ZeroOrMoreExpr:
return &ZeroOrMoreExpr{
Expr: cloneExpr(expr.Expr),
p: expr.p,
}
case *ZeroOrOneExpr:
return &ZeroOrOneExpr{
Expr: cloneExpr(expr.Expr),
p: expr.p,
}
}
return expr
}
// cleanupCharClassMatcher is a Visitor, which is used with the Walk function
// The purpose of this function is to cleanup the redundancies created by the
// optimize Visitor. This includes to remove redundant entries in Chars, Ranges
// and UnicodeClasses of the given CharClassMatcher as well as regenerating the
// correct content for the Val field (string representation of the CharClassMatcher)
func (r *grammarOptimizer) cleanupCharClassMatcher(expr0 Expression) Visitor {
// We are only interested in nodes of type *CharClassMatcher
if chr, ok := expr0.(*CharClassMatcher); ok {
// Remove redundancies in Chars
chars := make([]rune, 0, len(chr.Chars))
charsMap := make(map[rune]struct{})
for _, c := range chr.Chars {
if _, ok := charsMap[c]; !ok {
charsMap[c] = struct{}{}
chars = append(chars, c)
}
}
if len(chars) > 0 {
chr.Chars = chars
} else {
chr.Chars = nil
}
// Remove redundancies in Ranges
ranges := make([]rune, 0, len(chr.Ranges))
rangesMap := make(map[string]struct{})
for i := 0; i < len(chr.Ranges); i += 2 {
rangeKey := string(chr.Ranges[i]) + "-" + string(chr.Ranges[i+1])
if _, ok := rangesMap[rangeKey]; !ok {
rangesMap[rangeKey] = struct{}{}
ranges = append(ranges, chr.Ranges[i], chr.Ranges[i+1])
}
}
if len(ranges) > 0 {
chr.Ranges = ranges
} else {
chr.Ranges = nil
}
// Remove redundancies in UnicodeClasses
unicodeClasses := make([]string, 0, len(chr.UnicodeClasses))
unicodeClassesMap := make(map[string]struct{})
for _, u := range chr.UnicodeClasses {
if _, ok := unicodeClassesMap[u]; !ok {
unicodeClassesMap[u] = struct{}{}
unicodeClasses = append(unicodeClasses, u)
}
}
if len(unicodeClasses) > 0 {
chr.UnicodeClasses = unicodeClasses
} else {
chr.UnicodeClasses = nil
}
// Regenerate the content for Val
var val bytes.Buffer
val.WriteString("[")
if chr.Inverted {
val.WriteString("^")
}
for _, c := range chr.Chars {
val.WriteString(escapeRune(c))
}
for i := 0; i < len(chr.Ranges); i += 2 {
val.WriteString(escapeRune(chr.Ranges[i]))
val.WriteString("-")
val.WriteString(escapeRune(chr.Ranges[i+1]))
}
for _, u := range chr.UnicodeClasses {
val.WriteString("\\p" + u)
}
val.WriteString("]")
if chr.IgnoreCase {
val.WriteString("i")
}
chr.posValue.Val = val.String()
}
return r
}
func escapeRune(r rune) string {
return strings.Trim(strconv.QuoteRune(r), `'`)
}
// Optimize walks a given grammar and optimizes the grammar in regards
// of parsing performance. This is done with several optimizations:
// * removal of unreferenced rules
// * replace rule references with a copy of the referenced Rule, if the
// referenced rule it self has no references.
// * resolve nested choice expressions
// * resolve choice expressions with only one alternative
// * resolve nested sequences expression
// * resolve sequence expressions with only one element
// * combine character class matcher and literal matcher, where possible
func Optimize(g *Grammar, alternateEntrypoints ...string) {
entrypoints := alternateEntrypoints
if len(g.Rules) > 0 {
entrypoints = append(entrypoints, g.Rules[0].Name.Val)
}
r := newGrammarOptimizer(entrypoints)
Walk(r, g)
r.visitor = r.optimize
r.optimized = true
for r.optimized {
Walk(r, g)
}
r.visitor = r.cleanupCharClassMatcher
Walk(r, g)
}

87
vendor/github.com/mna/pigeon/ast/ast_walk.go generated vendored Normal file
View File

@@ -0,0 +1,87 @@
package ast
import "fmt"
// A Visitor implements a Visit method, which is invoked for each Expression
// encountered by Walk.
// If the result visitor w is not nil, Walk visits each of the children
// of Expression with the visitor w, followed by a call of w.Visit(nil).
type Visitor interface {
Visit(expr Expression) (w Visitor)
}
// Walk traverses an AST in depth-first order: It starts by calling
// v.Visit(expr); Expression must not be nil. If the visitor w returned by
// v.Visit(expr) is not nil, Walk is invoked recursively with visitor
// w for each of the non-nil children of Expression, followed by a call of
// w.Visit(nil).
//
func Walk(v Visitor, expr Expression) {
if v = v.Visit(expr); v == nil {
return
}
switch expr := expr.(type) {
case *ActionExpr:
Walk(v, expr.Expr)
case *AndCodeExpr:
// Nothing to do
case *AndExpr:
Walk(v, expr.Expr)
case *AnyMatcher:
// Nothing to do
case *CharClassMatcher:
// Nothing to do
case *ChoiceExpr:
for _, e := range expr.Alternatives {
Walk(v, e)
}
case *Grammar:
for _, e := range expr.Rules {
Walk(v, e)
}
case *LabeledExpr:
Walk(v, expr.Expr)
case *LitMatcher:
// Nothing to do
case *NotCodeExpr:
// Nothing to do
case *NotExpr:
Walk(v, expr.Expr)
case *OneOrMoreExpr:
Walk(v, expr.Expr)
case *Rule:
Walk(v, expr.Expr)
case *RuleRefExpr:
// Nothing to do
case *SeqExpr:
for _, e := range expr.Exprs {
Walk(v, e)
}
case *StateCodeExpr:
// Nothing to do
case *ZeroOrMoreExpr:
Walk(v, expr.Expr)
case *ZeroOrOneExpr:
Walk(v, expr.Expr)
default:
panic(fmt.Sprintf("unknown expression type %T", expr))
}
}
type inspector func(Expression) bool
func (f inspector) Visit(expr Expression) Visitor {
if f(expr) {
return f
}
return nil
}
// Inspect traverses an AST in depth-first order: It starts by calling
// f(expr); expr must not be nil. If f returns true, Inspect invokes f
// recursively for each of the non-nil children of expr, followed by a
// call of f(nil).
func Inspect(expr Expression, f func(Expression) bool) {
Walk(inspector(f), expr)
}

817
vendor/github.com/mna/pigeon/builder/builder.go generated vendored Normal file
View File

@@ -0,0 +1,817 @@
// Package builder generates the parser code for a given grammar. It makes
// no attempt to verify the correctness of the grammar.
package builder
import (
"bytes"
"fmt"
"io"
"strconv"
"strings"
"text/template"
"unicode"
"regexp"
"github.com/mna/pigeon/ast"
)
const codeGeneratedComment = "// Code generated by pigeon; DO NOT EDIT.\n\n"
// generated function templates
var (
onFuncTemplate = `func (%s *current) %s(%s) (interface{}, error) {
%s
}
`
onPredFuncTemplate = `func (%s *current) %s(%s) (bool, error) {
%s
}
`
onStateFuncTemplate = `func (%s *current) %s(%s) (error) {
%s
}
`
callFuncTemplate = `func (p *parser) call%s() (interface{}, error) {
stack := p.vstack[len(p.vstack)-1]
_ = stack
return p.cur.%[1]s(%s)
}
`
callPredFuncTemplate = `func (p *parser) call%s() (bool, error) {
stack := p.vstack[len(p.vstack)-1]
_ = stack
return p.cur.%[1]s(%s)
}
`
callStateFuncTemplate = `func (p *parser) call%s() error {
stack := p.vstack[len(p.vstack)-1]
_ = stack
return p.cur.%[1]s(%s)
}
`
)
// Option is a function that can set an option on the builder. It returns
// the previous setting as an Option.
type Option func(*builder) Option
// ReceiverName returns an option that specifies the receiver name to
// use for the current struct (which is the struct on which all code blocks
// except the initializer are generated).
func ReceiverName(nm string) Option {
return func(b *builder) Option {
prev := b.recvName
b.recvName = nm
return ReceiverName(prev)
}
}
// Optimize returns an option that specifies the optimize option
// If optimize is true, the Debug and Memoize code is completely
// removed from the resulting parser
func Optimize(optimize bool) Option {
return func(b *builder) Option {
prev := b.optimize
b.optimize = optimize
return Optimize(prev)
}
}
// Nolint returns an option that specifies the nolint option
// If nolint is true, special '// nolint: ...' comments are added
// to the generated parser to suppress warnings by gometalinter.
func Nolint(nolint bool) Option {
return func(b *builder) Option {
prev := b.nolint
b.nolint = nolint
return Optimize(prev)
}
}
// BasicLatinLookupTable returns an option that specifies the basicLatinLookup option
// If basicLatinLookup is true, a lookup slice for the first 128 chars of
// the Unicode table (Basic Latin) is generated for each CharClassMatcher
// to increase the character matching.
func BasicLatinLookupTable(basicLatinLookupTable bool) Option {
return func(b *builder) Option {
prev := b.basicLatinLookupTable
b.basicLatinLookupTable = basicLatinLookupTable
return BasicLatinLookupTable(prev)
}
}
// BuildParser builds the PEG parser using the provider grammar. The code is
// written to the specified w.
func BuildParser(w io.Writer, g *ast.Grammar, opts ...Option) error {
b := &builder{w: w, recvName: "c"}
b.setOptions(opts)
return b.buildParser(g)
}
type builder struct {
w io.Writer
err error
// options
recvName string
optimize bool
basicLatinLookupTable bool
globalState bool
nolint bool
ruleName string
exprIndex int
argsStack [][]string
rangeTable bool
}
func (b *builder) setOptions(opts []Option) {
for _, opt := range opts {
opt(b)
}
}
func (b *builder) buildParser(g *ast.Grammar) error {
b.writeInit(g.Init)
b.writeGrammar(g)
for _, rule := range g.Rules {
b.writeRuleCode(rule)
}
b.writeStaticCode()
return b.err
}
func (b *builder) writeInit(init *ast.CodeBlock) {
if init == nil {
return
}
// remove opening and closing braces
val := codeGeneratedComment + init.Val[1:len(init.Val)-1]
b.writelnf("%s", val)
}
func (b *builder) writeGrammar(g *ast.Grammar) {
// transform the ast grammar to the self-contained, no dependency version
// of the parser-generator grammar.
b.writelnf("var g = &grammar {")
b.writelnf("\trules: []*rule{")
for _, r := range g.Rules {
b.writeRule(r)
}
b.writelnf("\t},")
b.writelnf("}")
}
func (b *builder) writeRule(r *ast.Rule) {
if r == nil || r.Name == nil {
return
}
b.exprIndex = 0
b.ruleName = r.Name.Val
b.writelnf("{")
b.writelnf("\tname: %q,", r.Name.Val)
if r.DisplayName != nil && r.DisplayName.Val != "" {
b.writelnf("\tdisplayName: %q,", r.DisplayName.Val)
}
pos := r.Pos()
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
b.writef("\texpr: ")
b.writeExpr(r.Expr)
b.writelnf("},")
}
func (b *builder) writeExpr(expr ast.Expression) {
b.exprIndex++
switch expr := expr.(type) {
case *ast.ActionExpr:
b.writeActionExpr(expr)
case *ast.AndCodeExpr:
b.writeAndCodeExpr(expr)
case *ast.AndExpr:
b.writeAndExpr(expr)
case *ast.AnyMatcher:
b.writeAnyMatcher(expr)
case *ast.CharClassMatcher:
b.writeCharClassMatcher(expr)
case *ast.ChoiceExpr:
b.writeChoiceExpr(expr)
case *ast.LabeledExpr:
b.writeLabeledExpr(expr)
case *ast.LitMatcher:
b.writeLitMatcher(expr)
case *ast.NotCodeExpr:
b.writeNotCodeExpr(expr)
case *ast.NotExpr:
b.writeNotExpr(expr)
case *ast.OneOrMoreExpr:
b.writeOneOrMoreExpr(expr)
case *ast.RecoveryExpr:
b.writeRecoveryExpr(expr)
case *ast.RuleRefExpr:
b.writeRuleRefExpr(expr)
case *ast.SeqExpr:
b.writeSeqExpr(expr)
case *ast.StateCodeExpr:
b.writeStateCodeExpr(expr)
case *ast.ThrowExpr:
b.writeThrowExpr(expr)
case *ast.ZeroOrMoreExpr:
b.writeZeroOrMoreExpr(expr)
case *ast.ZeroOrOneExpr:
b.writeZeroOrOneExpr(expr)
default:
b.err = fmt.Errorf("builder: unknown expression type %T", expr)
}
}
func (b *builder) writeActionExpr(act *ast.ActionExpr) {
if act == nil {
b.writelnf("nil,")
return
}
if act.FuncIx == 0 {
act.FuncIx = b.exprIndex
}
b.writelnf("&actionExpr{")
pos := act.Pos()
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
b.writelnf("\trun: (*parser).call%s,", b.funcName(act.FuncIx))
b.writef("\texpr: ")
b.writeExpr(act.Expr)
b.writelnf("},")
}
func (b *builder) writeAndCodeExpr(and *ast.AndCodeExpr) {
if and == nil {
b.writelnf("nil,")
return
}
b.writelnf("&andCodeExpr{")
pos := and.Pos()
if and.FuncIx == 0 {
and.FuncIx = b.exprIndex
}
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
b.writelnf("\trun: (*parser).call%s,", b.funcName(and.FuncIx))
b.writelnf("},")
}
func (b *builder) writeAndExpr(and *ast.AndExpr) {
if and == nil {
b.writelnf("nil,")
return
}
b.writelnf("&andExpr{")
pos := and.Pos()
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
b.writef("\texpr: ")
b.writeExpr(and.Expr)
b.writelnf("},")
}
func (b *builder) writeAnyMatcher(any *ast.AnyMatcher) {
if any == nil {
b.writelnf("nil,")
return
}
b.writelnf("&anyMatcher{")
pos := any.Pos()
b.writelnf("\tline: %d, col: %d, offset: %d,", pos.Line, pos.Col, pos.Off)
b.writelnf("},")
}
func (b *builder) writeCharClassMatcher(ch *ast.CharClassMatcher) {
if ch == nil {
b.writelnf("nil,")
return
}
b.writelnf("&charClassMatcher{")
pos := ch.Pos()
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
b.writelnf("\tval: %q,", ch.Val)
if len(ch.Chars) > 0 {
b.writef("\tchars: []rune{")
for _, rn := range ch.Chars {
if ch.IgnoreCase {
b.writef("%q,", unicode.ToLower(rn))
} else {
b.writef("%q,", rn)
}
}
b.writelnf("},")
}
if len(ch.Ranges) > 0 {
b.writef("\tranges: []rune{")
for _, rn := range ch.Ranges {
if ch.IgnoreCase {
b.writef("%q,", unicode.ToLower(rn))
} else {
b.writef("%q,", rn)
}
}
b.writelnf("},")
}
if len(ch.UnicodeClasses) > 0 {
b.rangeTable = true
b.writef("\tclasses: []*unicode.RangeTable{")
for _, cl := range ch.UnicodeClasses {
b.writef("rangeTable(%q),", cl)
}
b.writelnf("},")
}
if b.basicLatinLookupTable {
b.writelnf("\tbasicLatinChars: %#v,", BasicLatinLookup(ch.Chars, ch.Ranges, ch.UnicodeClasses, ch.IgnoreCase))
}
b.writelnf("\tignoreCase: %t,", ch.IgnoreCase)
b.writelnf("\tinverted: %t,", ch.Inverted)
b.writelnf("},")
}
// BasicLatinLookup calculates the decision results for the first 256 characters of the UTF-8 character
// set for a given set of chars, ranges and unicodeClasses to speedup the CharClassMatcher.
func BasicLatinLookup(chars, ranges []rune, unicodeClasses []string, ignoreCase bool) (basicLatinChars [128]bool) {
for _, rn := range chars {
if rn < 128 {
basicLatinChars[rn] = true
if ignoreCase {
if unicode.IsLower(rn) {
basicLatinChars[unicode.ToUpper(rn)] = true
} else {
basicLatinChars[unicode.ToLower(rn)] = true
}
}
}
}
for i := 0; i < len(ranges); i += 2 {
if ranges[i] < 128 {
for j := ranges[i]; j < 128 && j <= ranges[i+1]; j++ {
basicLatinChars[j] = true
if ignoreCase {
if unicode.IsLower(j) {
basicLatinChars[unicode.ToUpper(j)] = true
} else {
basicLatinChars[unicode.ToLower(j)] = true
}
}
}
}
}
for _, cl := range unicodeClasses {
rt := rangeTable(cl)
for r := rune(0); r < 128; r++ {
if unicode.Is(rt, r) {
basicLatinChars[r] = true
}
}
}
return
}
func (b *builder) writeChoiceExpr(ch *ast.ChoiceExpr) {
if ch == nil {
b.writelnf("nil,")
return
}
b.writelnf("&choiceExpr{")
pos := ch.Pos()
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
if len(ch.Alternatives) > 0 {
b.writelnf("\talternatives: []interface{}{")
for _, alt := range ch.Alternatives {
b.writeExpr(alt)
}
b.writelnf("\t},")
}
b.writelnf("},")
}
func (b *builder) writeLabeledExpr(lab *ast.LabeledExpr) {
if lab == nil {
b.writelnf("nil,")
return
}
b.writelnf("&labeledExpr{")
pos := lab.Pos()
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
if lab.Label != nil && lab.Label.Val != "" {
b.writelnf("\tlabel: %q,", lab.Label.Val)
}
b.writef("\texpr: ")
b.writeExpr(lab.Expr)
b.writelnf("},")
}
func (b *builder) writeLitMatcher(lit *ast.LitMatcher) {
if lit == nil {
b.writelnf("nil,")
return
}
b.writelnf("&litMatcher{")
pos := lit.Pos()
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
if lit.IgnoreCase {
b.writelnf("\tval: %q,", strings.ToLower(lit.Val))
} else {
b.writelnf("\tval: %q,", lit.Val)
}
b.writelnf("\tignoreCase: %t,", lit.IgnoreCase)
ignoreCaseFlag := ""
if lit.IgnoreCase {
ignoreCaseFlag = "i"
}
b.writelnf("\twant: %q,", strconv.Quote(lit.Val)+ignoreCaseFlag)
b.writelnf("},")
}
func (b *builder) writeNotCodeExpr(not *ast.NotCodeExpr) {
if not == nil {
b.writelnf("nil,")
return
}
b.writelnf("&notCodeExpr{")
pos := not.Pos()
if not.FuncIx == 0 {
not.FuncIx = b.exprIndex
}
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
b.writelnf("\trun: (*parser).call%s,", b.funcName(not.FuncIx))
b.writelnf("},")
}
func (b *builder) writeNotExpr(not *ast.NotExpr) {
if not == nil {
b.writelnf("nil,")
return
}
b.writelnf("&notExpr{")
pos := not.Pos()
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
b.writef("\texpr: ")
b.writeExpr(not.Expr)
b.writelnf("},")
}
func (b *builder) writeOneOrMoreExpr(one *ast.OneOrMoreExpr) {
if one == nil {
b.writelnf("nil,")
return
}
b.writelnf("&oneOrMoreExpr{")
pos := one.Pos()
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
b.writef("\texpr: ")
b.writeExpr(one.Expr)
b.writelnf("},")
}
func (b *builder) writeRecoveryExpr(recover *ast.RecoveryExpr) {
if recover == nil {
b.writelnf("nil,")
return
}
b.writelnf("&recoveryExpr{")
pos := recover.Pos()
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
b.writef("\texpr: ")
b.writeExpr(recover.Expr)
b.writef("\trecoverExpr: ")
b.writeExpr(recover.RecoverExpr)
b.writelnf("\tfailureLabel: []string{")
for _, label := range recover.Labels {
b.writelnf("%q,", label)
}
b.writelnf("\t},")
b.writelnf("},")
}
func (b *builder) writeRuleRefExpr(ref *ast.RuleRefExpr) {
if ref == nil {
b.writelnf("nil,")
return
}
b.writelnf("&ruleRefExpr{")
pos := ref.Pos()
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
if ref.Name != nil && ref.Name.Val != "" {
b.writelnf("\tname: %q,", ref.Name.Val)
}
b.writelnf("},")
}
func (b *builder) writeSeqExpr(seq *ast.SeqExpr) {
if seq == nil {
b.writelnf("nil,")
return
}
b.writelnf("&seqExpr{")
pos := seq.Pos()
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
if len(seq.Exprs) > 0 {
b.writelnf("\texprs: []interface{}{")
for _, e := range seq.Exprs {
b.writeExpr(e)
}
b.writelnf("\t},")
}
b.writelnf("},")
}
func (b *builder) writeStateCodeExpr(state *ast.StateCodeExpr) {
if state == nil {
b.writelnf("nil,")
return
}
b.globalState = true
b.writelnf("&stateCodeExpr{")
pos := state.Pos()
if state.FuncIx == 0 {
state.FuncIx = b.exprIndex
}
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
b.writelnf("\trun: (*parser).call%s,", b.funcName(state.FuncIx))
b.writelnf("},")
}
func (b *builder) writeThrowExpr(throw *ast.ThrowExpr) {
if throw == nil {
b.writelnf("nil,")
return
}
b.writelnf("&throwExpr{")
pos := throw.Pos()
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
b.writelnf("\tlabel: %q,", throw.Label)
b.writelnf("},")
}
func (b *builder) writeZeroOrMoreExpr(zero *ast.ZeroOrMoreExpr) {
if zero == nil {
b.writelnf("nil,")
return
}
b.writelnf("&zeroOrMoreExpr{")
pos := zero.Pos()
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
b.writef("\texpr: ")
b.writeExpr(zero.Expr)
b.writelnf("},")
}
func (b *builder) writeZeroOrOneExpr(zero *ast.ZeroOrOneExpr) {
if zero == nil {
b.writelnf("nil,")
return
}
b.writelnf("&zeroOrOneExpr{")
pos := zero.Pos()
b.writelnf("\tpos: position{line: %d, col: %d, offset: %d},", pos.Line, pos.Col, pos.Off)
b.writef("\texpr: ")
b.writeExpr(zero.Expr)
b.writelnf("},")
}
func (b *builder) writeRuleCode(rule *ast.Rule) {
if rule == nil || rule.Name == nil {
return
}
// keep trace of the current rule, as the code blocks are created
// in functions named "on<RuleName><#ExprIndex>".
b.ruleName = rule.Name.Val
b.pushArgsSet()
b.writeExprCode(rule.Expr)
b.popArgsSet()
}
func (b *builder) pushArgsSet() {
b.argsStack = append(b.argsStack, nil)
}
func (b *builder) popArgsSet() {
b.argsStack = b.argsStack[:len(b.argsStack)-1]
}
func (b *builder) addArg(arg *ast.Identifier) {
if arg == nil {
return
}
ix := len(b.argsStack) - 1
b.argsStack[ix] = append(b.argsStack[ix], arg.Val)
}
func (b *builder) writeExprCode(expr ast.Expression) {
switch expr := expr.(type) {
case *ast.ActionExpr:
b.writeExprCode(expr.Expr)
b.writeActionExprCode(expr)
case *ast.AndCodeExpr:
b.writeAndCodeExprCode(expr)
case *ast.LabeledExpr:
b.addArg(expr.Label)
b.pushArgsSet()
b.writeExprCode(expr.Expr)
b.popArgsSet()
case *ast.NotCodeExpr:
b.writeNotCodeExprCode(expr)
case *ast.AndExpr:
b.pushArgsSet()
b.writeExprCode(expr.Expr)
b.popArgsSet()
case *ast.ChoiceExpr:
for _, alt := range expr.Alternatives {
b.pushArgsSet()
b.writeExprCode(alt)
b.popArgsSet()
}
case *ast.NotExpr:
b.pushArgsSet()
b.writeExprCode(expr.Expr)
b.popArgsSet()
case *ast.OneOrMoreExpr:
b.pushArgsSet()
b.writeExprCode(expr.Expr)
b.popArgsSet()
case *ast.RecoveryExpr:
b.pushArgsSet()
b.writeExprCode(expr.Expr)
b.writeExprCode(expr.RecoverExpr)
b.popArgsSet()
case *ast.SeqExpr:
for _, sub := range expr.Exprs {
b.writeExprCode(sub)
}
case *ast.StateCodeExpr:
b.writeStateCodeExprCode(expr)
case *ast.ZeroOrMoreExpr:
b.pushArgsSet()
b.writeExprCode(expr.Expr)
b.popArgsSet()
case *ast.ZeroOrOneExpr:
b.pushArgsSet()
b.writeExprCode(expr.Expr)
b.popArgsSet()
}
}
func (b *builder) writeActionExprCode(act *ast.ActionExpr) {
if act == nil {
return
}
if act.FuncIx > 0 {
b.writeFunc(act.FuncIx, act.Code, callFuncTemplate, onFuncTemplate)
act.FuncIx = 0 // already rendered, prevent duplicates
}
}
func (b *builder) writeAndCodeExprCode(and *ast.AndCodeExpr) {
if and == nil {
return
}
if and.FuncIx > 0 {
b.writeFunc(and.FuncIx, and.Code, callPredFuncTemplate, onPredFuncTemplate)
and.FuncIx = 0 // already rendered, prevent duplicates
}
}
func (b *builder) writeNotCodeExprCode(not *ast.NotCodeExpr) {
if not == nil {
return
}
if not.FuncIx > 0 {
b.writeFunc(not.FuncIx, not.Code, callPredFuncTemplate, onPredFuncTemplate)
not.FuncIx = 0 // already rendered, prevent duplicates
}
}
func (b *builder) writeStateCodeExprCode(state *ast.StateCodeExpr) {
if state == nil {
return
}
if state.FuncIx > 0 {
b.writeFunc(state.FuncIx, state.Code, callStateFuncTemplate, onStateFuncTemplate)
state.FuncIx = 0 // already rendered, prevent duplicates
}
}
func (b *builder) writeFunc(funcIx int, code *ast.CodeBlock, callTpl, funcTpl string) {
if code == nil {
return
}
val := strings.TrimSpace(code.Val)[1 : len(code.Val)-1]
if len(val) > 0 && val[0] == '\n' {
val = val[1:]
}
if len(val) > 0 && val[len(val)-1] == '\n' {
val = val[:len(val)-1]
}
var args bytes.Buffer
ix := len(b.argsStack) - 1
if ix >= 0 {
for i, arg := range b.argsStack[ix] {
if i > 0 {
args.WriteString(", ")
}
args.WriteString(arg)
}
}
if args.Len() > 0 {
args.WriteString(" interface{}")
}
fnNm := b.funcName(funcIx)
b.writelnf(funcTpl, b.recvName, fnNm, args.String(), val)
args.Reset()
if ix >= 0 {
for i, arg := range b.argsStack[ix] {
if i > 0 {
args.WriteString(", ")
}
args.WriteString(fmt.Sprintf(`stack[%q]`, arg))
}
}
b.writelnf(callTpl, fnNm, args.String())
}
func (b *builder) writeStaticCode() {
buffer := bytes.NewBufferString("")
params := struct {
Optimize bool
BasicLatinLookupTable bool
GlobalState bool
Nolint bool
}{
Optimize: b.optimize,
BasicLatinLookupTable: b.basicLatinLookupTable,
GlobalState: b.globalState,
Nolint: b.nolint,
}
t := template.Must(template.New("static_code").Parse(staticCode))
err := t.Execute(buffer, params)
if err != nil {
// This is very unlikely to ever happen
panic("executing template: " + err.Error())
}
// Clean the ==template== comments from the generated parser
lines := strings.Split(buffer.String(), "\n")
buffer.Reset()
re := regexp.MustCompile(`^\s*//\s*(==template==\s*)+$`)
reLineEnd := regexp.MustCompile(`//\s*==template==\s*$`)
for _, line := range lines {
if !re.MatchString(line) {
line = reLineEnd.ReplaceAllString(line, "")
_, err := buffer.WriteString(line + "\n")
if err != nil {
// This is very unlikely to ever happen
panic("unable to write to byte buffer: " + err.Error())
}
}
}
b.writeln(buffer.String())
if b.rangeTable {
b.writeln(rangeTable0)
}
}
func (b *builder) funcName(ix int) string {
return "on" + b.ruleName + strconv.Itoa(ix)
}
func (b *builder) writef(f string, args ...interface{}) {
if b.err == nil {
_, b.err = fmt.Fprintf(b.w, f, args...)
}
}
func (b *builder) writelnf(f string, args ...interface{}) {
b.writef(f+"\n", args...)
}
func (b *builder) writeln(f string) {
if b.err == nil {
_, b.err = fmt.Fprint(b.w, f+"\n")
}
}

1450
vendor/github.com/mna/pigeon/builder/generated_static_code.go generated vendored Normal file
View File

File diff suppressed because it is too large Load Diff

21
vendor/github.com/mna/pigeon/builder/generated_static_code_range_table.go generated vendored Normal file
View File

@@ -0,0 +1,21 @@
// Code generated by static_code_generator with go generate; DO NOT EDIT.
package builder
var rangeTable0 = `
func rangeTable(class string) *unicode.RangeTable {
if rt, ok := unicode.Categories[class]; ok {
return rt
}
if rt, ok := unicode.Properties[class]; ok {
return rt
}
if rt, ok := unicode.Scripts[class]; ok {
return rt
}
// cannot happen
panic(fmt.Sprintf("invalid Unicode class: %s", class))
}
`

1466
vendor/github.com/mna/pigeon/builder/static_code.go generated vendored Normal file
View File

File diff suppressed because it is too large Load Diff

24
vendor/github.com/mna/pigeon/builder/static_code_range_table.go generated vendored Normal file
View File

@@ -0,0 +1,24 @@
//go:generate go run ../bootstrap/cmd/static_code_generator/main.go -- $GOFILE generated_$GOFILE rangeTable0
package builder
import (
"fmt"
"unicode"
)
// IMPORTANT: All code below this line is added to the parser as static code
func rangeTable(class string) *unicode.RangeTable {
if rt, ok := unicode.Categories[class]; ok {
return rt
}
if rt, ok := unicode.Properties[class]; ok {
return rt
}
if rt, ok := unicode.Scripts[class]; ok {
return rt
}
// cannot happen
panic(fmt.Sprintf("invalid Unicode class: %s", class))
}

594
vendor/github.com/mna/pigeon/doc.go generated vendored Normal file
View File

@@ -0,0 +1,594 @@
/*
Command pigeon generates parsers in Go from a PEG grammar.
From Wikipedia [0]:
A parsing expression grammar is a type of analytic formal grammar, i.e.
it describes a formal language in terms of a set of rules for recognizing
strings in the language.
Its features and syntax are inspired by the PEG.js project [1], while
the implementation is loosely based on [2]. Formal presentation of the
PEG theory by Bryan Ford is also an important reference [3]. An introductory
blog post can be found at [4].
[0]: http://en.wikipedia.org/wiki/Parsing_expression_grammar
[1]: http://pegjs.org/
[2]: http://www.codeproject.com/Articles/29713/Parsing-Expression-Grammar-Support-for-C-Part
[3]: http://pdos.csail.mit.edu/~baford/packrat/popl04/peg-popl04.pdf
[4]: http://0value.com/A-PEG-parser-generator-for-Go
Command-line usage
The pigeon tool must be called with PEG input as defined
by the accepted PEG syntax below. The grammar may be provided by a
file or read from stdin. The generated parser is written to stdout
by default.
pigeon [options] [GRAMMAR_FILE]
The following options can be specified:
-cache : cache parser results to avoid exponential parsing time in
pathological cases. Can make the parsing slower for typical
cases and uses more memory (default: false).
-debug : boolean, print debugging info to stdout (default: false).
-nolint: add '// nolint: ...' comments for generated parser to suppress
warnings by gometalinter (https://github.com/alecthomas/gometalinter).
-no-recover : boolean, if set, do not recover from a panic. Useful
to access the panic stack when debugging, otherwise the panic
is converted to an error (default: false).
-o=FILE : string, output file where the generated parser will be
written (default: stdout).
-optimize-basic-latin : boolean, if set, a lookup table for the first 128
characters of the Unicode table (Basic Latin) is generated for each character
class matcher. This speeds up the parsing, if parsed data mainly consists
of characters from this range (default: false).
-optimize-grammar : boolean, (EXPERIMENTAL FEATURE) if set, several performance
optimizations on the grammar are performed, with focus to the reduction of the
grammar depth.
Optimization:
* removal of unreferenced rules
* replace rule references with a copy of the referenced Rule, if the
referenced rule it self has no references.
* resolve nested choice expressions
* resolve choice expressions with only one alternative
* resolve nested sequences expression
* resolve sequence expressions with only one element
* combine character class matcher and literal matcher, where possible
The resulting grammar is usually more memory consuming, but faster for parsing.
The optimization of the grammar is done in multiple rounds (optimize until no
more optimizations have applied). This process takes some time, depending on the
optimization potential of the grammar.
-optimize-parser : boolean, if set, the options Debug, Memoize and Statistics are
removed from the resulting parser. The global "state" is optimized as well by
either removing all related code if no state change expression is present in the
grammar or by removing the restoration of the global "state" store after action
and predicate code blocks. This saves a few cpu cycles, when using the generated
parser (default: false).
-x : boolean, if set, do not build the parser, just parse the input grammar
(default: false).
-receiver-name=NAME : string, name of the receiver variable for the generated
code blocks. Non-initializer code blocks in the grammar end up as methods on the
*current type, and this option sets the name of the receiver (default: c).
-alternate-entrypoints=RULE[,RULE...] : string, comma-separated list of rule names
that may be used as alternate entrypoints for the parser, in addition to the
default entrypoint (the first rule in the grammar) (default: none).
Such entrypoints can be specified in the call to Parse by passing an
Entrypoint option that specifies the alternate rule name to use. This is only
necessary if the -optimize-parser flag is set, as some rules may be optimized
out of the resulting parser.
If the code blocks in the grammar (see below, section "Code block") are golint-
and go vet-compliant, then the resulting generated code will also be golint-
and go vet-compliant.
The generated code doesn't use any third-party dependency unless code blocks
in the grammar require such a dependency.
PEG syntax
The accepted syntax for the grammar is formally defined in the
grammar/pigeon.peg file, using the PEG syntax. What follows is an informal
description of this syntax.
Identifiers, whitespace, comments and literals follow the same
notation as the Go language, as defined in the language specification
(http://golang.org/ref/spec#Source_code_representation):
// single line comment*/
// /* multi-line comment */
/* 'x' (single quotes for single char literal)
"double quotes for string literal"
`backtick quotes for raw string literal`
RuleName (a valid identifier)
The grammar must be Unicode text encoded in UTF-8. New lines are identified
by the \n character (U+000A). Space (U+0020), horizontal tabs (U+0009) and
carriage returns (U+000D) are considered whitespace and are ignored except
to separate tokens.
Rules
A PEG grammar consists of a set of rules. A rule is an identifier followed
by a rule definition operator and an expression. An optional display name -
a string literal used in error messages instead of the rule identifier - can
be specified after the rule identifier. E.g.:
RuleA "friendly name" = 'a'+ // RuleA is one or more lowercase 'a's
The rule definition operator can be any one of those:
=, <-, ← (U+2190), ⟵ (U+27F5)
Expressions
A rule is defined by an expression. The following sections describe the
various expression types. Expressions can be grouped by using parentheses,
and a rule can be referenced by its identifier in place of an expression.
Choice expression
The choice expression is a list of expressions that will be tested in the
order they are defined. The first one that matches will be used. Expressions
are separated by the forward slash character "/". E.g.:
ChoiceExpr = A / B / C // A, B and C should be rules declared in the grammar
Because the first match is used, it is important to think about the order
of expressions. For example, in this rule, "<=" would never be used because
the "<" expression comes first:
BadChoiceExpr = "<" / "<="
Sequence expression
The sequence expression is a list of expressions that must all match in
that same order for the sequence expression to be considered a match.
Expressions are separated by whitespace. E.g.:
SeqExpr = "A" "b" "c" // matches "Abc", but not "Acb"
Labeled expression
A labeled expression consists of an identifier followed by a colon ":"
and an expression. A labeled expression introduces a variable named with
the label that can be referenced in the code blocks in the same scope.
The variable will have the value of the expression that follows the colon.
E.g.:
LabeledExpr = value:[a-z]+ {
fmt.Println(value)
return value, nil
}
The variable is typed as an empty interface, and the underlying type depends
on the following:
For terminals (character and string literals, character classes and
the any matcher), the value is []byte. E.g.:
Rule = label:'a' { // label is []byte }
For predicates (& and !), the value is always nil. E.g.:
Rule = label:&'a' { // label is nil }
For a sequence, the value is a slice of empty interfaces, one for each
expression value in the sequence. The underlying types of each value
in the slice follow the same rules described here, recursively. E.g.:
Rule = label:('a' 'b') { // label is []interface{} }
For a repetition (+ and *), the value is a slice of empty interfaces, one for
each repetition. The underlying types of each value in the slice follow
the same rules described here, recursively. E.g.:
Rule = label:[a-z]+ { // label is []interface{} }
For a choice expression, the value is that of the matching choice. E.g.:
Rule = label:('a' / 'b') { // label is []byte }
For the optional expression (?), the value is nil or the value of the
expression. E.g.:
Rule = label:'a'? { // label is nil or []byte }
Of course, the type of the value can be anything once an action code block
is used. E.g.:
RuleA = label:'3' {
return 3, nil
}
RuleB = label:RuleA { // label is int }
And and not expressions
An expression prefixed with the ampersand "&" is the "and" predicate
expression: it is considered a match if the following expression is a match,
but it does not consume any input.
An expression prefixed with the exclamation point "!" is the "not" predicate
expression: it is considered a match if the following expression is not
a match, but it does not consume any input. E.g.:
AndExpr = "A" &"B" // matches "A" if followed by a "B" (does not consume "B")
NotExpr = "A" !"B" // matches "A" if not followed by a "B" (does not consume "B")
The expression following the & and ! operators can be a code block. In that
case, the code block must return a bool and an error. The operator's semantic
is the same, & is a match if the code block returns true, ! is a match if the
code block returns false. The code block has access to any labeled value
defined in its scope. E.g.:
CodeAndExpr = value:[a-z] &{
// can access the value local variable...
return true, nil
}
Repeating expressions
An expression followed by "*", "?" or "+" is a match if the expression
occurs zero or more times ("*"), zero or one time "?" or one or more times
("+") respectively. The match is greedy, it will match as many times as
possible. E.g.
ZeroOrMoreAs = "A"*
Literal matcher
A literal matcher tries to match the input against a single character or a
string literal. The literal may be a single-quoted single character, a
double-quoted string or a backtick-quoted raw string. The same rules as in Go
apply regarding the allowed characters and escapes.
The literal may be followed by a lowercase "i" (outside the ending quote)
to indicate that the match is case-insensitive. E.g.:
LiteralMatch = "Awesome\n"i // matches "awesome" followed by a newline
Character class matcher
A character class matcher tries to match the input against a class of characters
inside square brackets "[...]". Inside the brackets, characters represent
themselves and the same escapes as in string literals are available, except
that the single- and double-quote escape is not valid, instead the closing
square bracket "]" must be escaped to be used.
Character ranges can be specified using the "[a-z]" notation. Unicode
classes can be specified using the "[\pL]" notation, where L is a
single-letter Unicode class of characters, or using the "[\p{Class}]"
notation where Class is a valid Unicode class (e.g. "Latin").
As for string literals, a lowercase "i" may follow the matcher (outside
the ending square bracket) to indicate that the match is case-insensitive.
A "^" as first character inside the square brackets indicates that the match
is inverted (it is a match if the input does not match the character class
matcher). E.g.:
NotAZ = [^a-z]i
Any matcher
The any matcher is represented by the dot ".". It matches any character
except the end of file, thus the "!." expression is used to indicate "match
the end of file". E.g.:
AnyChar = . // match a single character
EOF = !.
Code block
Code blocks can be added to generate custom Go code. There are three kinds
of code blocks: the initializer, the action and the predicate. All code blocks
appear inside curly braces "{...}".
The initializer must appear first in the grammar, before any rule. It is
copied as-is (minus the wrapping curly braces) at the top of the generated
parser. It may contain function declarations, types, variables, etc. just
like any Go file. Every symbol declared here will be available to all other
code blocks. Although the initializer is optional in a valid grammar, it is
usually required to generate a valid Go source code file (for the package
clause). E.g.:
{
package main
func someHelper() {
// ...
}
}
Action code blocks are code blocks declared after an expression in a rule.
Those code blocks are turned into a method on the "*current" type in the
generated source code. The method receives any labeled expression's value
as argument (as interface{}) and must return two values, the first being
the value of the expression (an interface{}), and the second an error.
If a non-nil error is returned, it is added to the list of errors that the
parser will return. E.g.:
RuleA = "A"+ {
// return the matched string, "c" is the default name for
// the *current receiver variable.
return string(c.text), nil
}
Predicate code blocks are code blocks declared immediately after the and "&"
or the not "!" operators. Like action code blocks, predicate code blocks
are turned into a method on the "*current" type in the generated source code.
The method receives any labeled expression's value as argument (as interface{})
and must return two opt, the first being a bool and the second an error.
If a non-nil error is returned, it is added to the list of errors that the
parser will return. E.g.:
RuleAB = [ab]i+ &{
return true, nil
}
State change code blocks are code blocks starting with "#". In contrast to
action and predicate code blocks, state change code blocks are allowed to
modify values in the global "state" store (see below).
State change code blocks are turned into a method on the "*current" type
in the generated source code.
The method is passed any labeled expression's value as an argument (of type
interface{}) and must return a value of type error.
If a non-nil error is returned, it is added to the list of errors that the
parser will return, note that the parser does NOT backtrack if a non-nil
error is returned.
E.g:
Rule = [a] #{
c.state["a"]++
if c.state["a"] > 5 {
return fmt.Errorf("we have seen more than 5 a's") // parser will not backtrack
}
return nil
}
The "*current" type is a struct that provides four useful fields that can be
accessed in action, state change, and predicate code blocks: "pos", "text",
"state" and "globalStore".
The "pos" field indicates the current position of the parser in the source
input. It is itself a struct with three fields: "line", "col" and "offset".
Line is a 1-based line number, col is a 1-based column number that counts
runes from the start of the line, and offset is a 0-based byte offset.
The "text" field is the slice of bytes of the current match. It is empty
in a predicate code block.
The "state" field is a global store, with backtrack support, of type
"map[string]interface{}". The values in the store are tied to the parser's
backtracking, in particular if a rule fails to match then all updates to the
state that occurred in the process of matching the rule are rolled back. For a
key-value store that is not tied to the parser's backtracking, see the
"globalStore".
The values in the "state" store are available for read access in action and
predicate code blocks, any changes made to the "state" store will be reverted
once the action or predicate code block is finished running. To update values
in the "state" use state change code blocks ("#{}").
IMPORTANT:
- In order to properly roll back the state if a rule fails to match the
parser must clone the state before trying to match a rule.
- The default clone mechanism makes a "shallow" copy of each value in the
"state", this implies that pointers, maps, slices, channels, and structs
containing any of the previous types are not properly copied.
- To support theses cases pigeon offers the "Cloner" interface which
consists of a single method "Clone". If a value stored in the "state"
store implements this interface, the "Clone" method is used to obtain a
proper copy.
- If a general solution is needed, external libraries which provide deep
copy functionality may be used in the "Clone" method
(e.g. https://github.com/mitchellh/copystructure).
The "globalStore" field is a global store of type "map[string]interface{}",
which allows to store arbitrary values, which are available in action and
predicate code blocks for read as well as write access.
It is important to notice, that the global store is completely independent from
the backtrack mechanism of PEG and is therefore not set back to its old state
during backtrack.
The initialization of the global store may be achieved by using the GlobalStore
function (http://godoc.org/github.com/mna/pigeon/test/predicates#GlobalStore).
Be aware, that all keys starting with "_pigeon" are reserved for internal use
of pigeon and should not be used nor modified. Those keys are treated as
internal implementation details and therefore there are no guarantees given in
regards of API stability.
Failure labels, throw and recover
pigeon supports an extension of the classical PEG syntax called failure labels,
proposed by Maidl et al. in their paper "Error Reporting in Parsing Expression Grammars" [7].
The used syntax for the introduced expressions is borrowed from their lpeglabel [8]
implementation.
This extension allows to signal different kinds of errors and to specify, which
recovery pattern should handle a given label.
With labeled failures it is possible to distinguish between an ordinary failure
and an error. Usually, an ordinary failure is produced when the matching of a
character fails, and this failure is caught by ordered choice. An error
(a non-ordinary failure), by its turn, is produced by the throw operator and
may be caught by the recovery operator.
In pigeon, the recovery expression consists of the regular expression, the recovery
expression and a set of labels to be matched. First, the regular expression is tried.
If this fails with one of the provided labels, the recovery expression is tried. If
this fails as well, the error is propagated. E.g.:
FailureRecoveryExpr = RegularExpr //{FailureLabel1, FailureLabel2} RecoveryExpr
To signal a failure condition, the throw expression is used. E.g.:
ThrowExpr = %{FailureLabel1}
For concrete examples, how to use throw and recover, have a look at the examples
"labeled_failures" and "thrownrecover" in the "test" folder.
The implementation of the throw and recover operators work as follows:
The failure recover expression adds the recover expression for every failure label
to the recovery stack and runs the regular expression.
The throw expression checks the recovery stack in reversed order for the provided
failure label. If the label is found, the respective recovery expression is run. If
this expression is successful, the parser continues the processing of the input. If
the recovery expression is not successful, the parsing fails and the parser starts
to backtrack.
If throw and recover expressions are used together with global state, it is the
responsibility of the author of the grammar to reset the global state to a valid
state during the recovery operation.
[7]: https://arxiv.org/pdf/1405.6646v3.pdf
[8]: https://github.com/sqmedeiros/lpeglabel
Using the generated parser
The parser generated by pigeon exports a few symbols so that it can be used
as a package with public functions to parse input text. The exported API is:
- Parse(string, []byte, ...Option) (interface{}, error)
- ParseFile(string, ...Option) (interface{}, error)
- ParseReader(string, io.Reader, ...Option) (interface{}, error)
- AllowInvalidUTF8(bool) Option
- Debug(bool) Option
- Entrypoint(string) Option
- GlobalStore(string, interface{}) Option
- MaxExpressions(uint64) Option
- Memoize(bool) Option
- Recover(bool) Option
- Statistics(*Stats) Option
See the godoc page of the generated parser for the test/predicates grammar
for an example documentation page of the exported API:
http://godoc.org/github.com/mna/pigeon/test/predicates.
Like the grammar used to generate the parser, the input text must be
UTF-8-encoded Unicode.
The start rule of the parser is the first rule in the PEG grammar used
to generate the parser. A call to any of the Parse* functions returns
the value generated by executing the grammar on the provided input text,
and an optional error.
Typically, the grammar should generate some kind of abstract syntax tree (AST),
but for simple grammars it may evaluate the result immediately, such as in
the examples/calculator example. There are no constraints imposed on the
author of the grammar, it can return whatever is needed.
Error reporting
When the parser returns a non-nil error, the error is always of type errList,
which is defined as a slice of errors ([]error). Each error in the list is
of type *parserError. This is a struct that has an "Inner" field that can be
used to access the original error.
So if a code block returns some well-known error like:
{
return nil, io.EOF
}
The original error can be accessed this way:
_, err := ParseFile("some_file")
if err != nil {
list := err.(errList)
for _, err := range list {
pe := err.(*parserError)
if pe.Inner == io.EOF {
// ...
}
}
}
By defaut the parser will continue after an error is returned and will
cumulate all errors found during parsing. If the grammar reaches a point
where it shouldn't continue, a panic statement can be used to terminate
parsing. The panic will be caught at the top-level of the Parse* call
and will be converted into a *parserError like any error, and an errList
will still be returned to the caller.
The divide by zero error in the examples/calculator grammar leverages this
feature (no special code is needed to handle division by zero, if it
happens, the runtime panics and it is recovered and returned as a parsing
error).
Providing good error reporting in a parser is not a trivial task. Part
of it is provided by the pigeon tool, by offering features such as
filename, position, expected literals and rule name in the error message,
but an important part of good error reporting needs to be done by the grammar
author.
For example, many programming languages use double-quotes for string literals.
Usually, if the opening quote is found, the closing quote is expected, and if
none is found, there won't be any other rule that will match, there's no need
to backtrack and try other choices, an error should be added to the list
and the match should be consumed.
In order to do this, the grammar can look something like this:
StringLiteral = '"' ValidStringChar* '"' {
// this is the valid case, build string literal node
// node = ...
return node, nil
} / '"' ValidStringChar* !'"' {
// invalid case, build a replacement string literal node or build a BadNode
// node = ...
return node, errors.New("string literal not terminated")
}
This is just one example, but it illustrates the idea that error reporting
needs to be thought out when designing the grammar.
Because the above mentioned error types (errList and parserError) are not
exported, additional steps have to be taken, ff the generated parser is used as
library package in other packages (e.g. if the same parser is used in multiple
command line tools).
One possible implementation for exported errors (based on interfaces) and
customized error reporting (caret style formatting of the position, where
the parsing failed) is available in the json example and its command line tool:
http://godoc.org/github.com/mna/pigeon/examples/json
API stability
Generated parsers have user-provided code mixed with pigeon code
in the same package, so there is no package
boundary in the resulting code to prevent access to unexported symbols.
What is meant to be implementation
details in pigeon is also available to user code - which doesn't mean
it should be used.
For this reason, it is important to precisely define what is intended to be
the supported API of pigeon, the parts that will be stable
in future versions.
The "stability" of the version 1.0 API attempts to make a similar guarantee
as the Go 1 compatibility [5]. The following lists what part of the
current pigeon code falls under that guarantee (features may be added in
the future):
- The pigeon command-line flags and arguments: those will not be removed
and will maintain the same semantics.
- The explicitly exported API generated by pigeon. See [6] for the
documentation of this API on a generated parser.
- The PEG syntax, as documented above.
- The code blocks (except the initializer) will always be generated as
methods on the *current type, and this type is guaranteed to have
the fields pos (type position) and text (type []byte). There are no
guarantees on other fields and methods of this type.
- The position type will always have the fields line, col and offset,
all defined as int. There are no guarantees on other fields and methods
of this type.
- The type of the error value returned by the Parse* functions, when
not nil, will always be errList defined as a []error. There are no
guarantees on methods of this type, other than the fact it implements the
error interface.
- Individual errors in the errList will always be of type *parserError,
and this type is guaranteed to have an Inner field that contains the
original error value. There are no guarantees on other fields and methods
of this type.
The above guarantee is given to the version 1.0 (https://github.com/mna/pigeon/releases/tag/v1.0.0)
of pigeon, which has entered maintenance mode (bug fixes only). The current
master branch includes the development toward a future version 2.0, which
intends to further improve pigeon.
While the given API stability should be maintained as far as it makes sense,
breaking changes may be necessary to be able to improve pigeon.
The new version 2.0 API has not yet stabilized and therefore changes to the API
may occur at any time.
References:
[5]: https://golang.org/doc/go1compat
[6]: http://godoc.org/github.com/mna/pigeon/test/predicates
*/
package main

294
vendor/github.com/mna/pigeon/main.go generated vendored Normal file
View File

@@ -0,0 +1,294 @@
package main
import (
"bufio"
"bytes"
"errors"
"flag"
"fmt"
"io"
"os"
"strconv"
"strings"
"golang.org/x/tools/imports"
"github.com/mna/pigeon/ast"
"github.com/mna/pigeon/builder"
)
// exit function mockable for tests
var exit = os.Exit
// ruleNamesFlag is a custom flag that parses a comma-separated
// list of rule names. It implements flag.Value.
type ruleNamesFlag []string
func (r *ruleNamesFlag) String() string {
return fmt.Sprint(*r)
}
func (r *ruleNamesFlag) Set(value string) error {
names := strings.Split(value, ",")
*r = append(*r, names...)
return nil
}
func main() {
fs := flag.NewFlagSet(os.Args[0], flag.ExitOnError)
// define command-line flags
var (
cacheFlag = fs.Bool("cache", false, "cache parsing results")
dbgFlag = fs.Bool("debug", false, "set debug mode")
shortHelpFlag = fs.Bool("h", false, "show help page")
longHelpFlag = fs.Bool("help", false, "show help page")
nolint = fs.Bool("nolint", false, "add '// nolint: ...' comments to suppress warnings by gometalinter")
noRecoverFlag = fs.Bool("no-recover", false, "do not recover from panic")
outputFlag = fs.String("o", "", "output file, defaults to stdout")
optimizeBasicLatinFlag = fs.Bool("optimize-basic-latin", false, "generate optimized parser for Unicode Basic Latin character sets")
optimizeGrammar = fs.Bool("optimize-grammar", false, "optimize the given grammar (EXPERIMENTAL FEATURE)")
optimizeParserFlag = fs.Bool("optimize-parser", false, "generate optimized parser without Debug and Memoize options")
recvrNmFlag = fs.String("receiver-name", "c", "receiver name for the generated methods")
noBuildFlag = fs.Bool("x", false, "do not build, only parse")
altEntrypointsFlag ruleNamesFlag
)
fs.Var(&altEntrypointsFlag, "alternate-entrypoints", "comma-separated list of rule names that may be used as entrypoints")
fs.Usage = usage
err := fs.Parse(os.Args[1:])
if err != nil {
fmt.Fprintln(os.Stderr, "args parse error:\n", err)
exit(6)
}
if *shortHelpFlag || *longHelpFlag {
fs.Usage()
exit(0)
}
if fs.NArg() > 1 {
argError(1, "expected one argument, got %q", strings.Join(fs.Args(), " "))
}
// get input source
infile := ""
if fs.NArg() == 1 {
infile = fs.Arg(0)
}
nm, rc := input(infile)
defer func() {
err = rc.Close()
if err != nil {
fmt.Fprintln(os.Stderr, "close file error:\n", err)
}
if r := recover(); r != nil {
panic(r)
}
if err != nil {
exit(7)
}
}()
// parse input
g, err := ParseReader(nm, rc, Debug(*dbgFlag), Memoize(*cacheFlag), Recover(!*noRecoverFlag))
if err != nil {
fmt.Fprintln(os.Stderr, "parse error(s):\n", err)
exit(3)
}
// validate alternate entrypoints
grammar := g.(*ast.Grammar)
rules := make(map[string]struct{}, len(grammar.Rules))
for _, rule := range grammar.Rules {
rules[rule.Name.Val] = struct{}{}
}
for _, entrypoint := range altEntrypointsFlag {
if entrypoint == "" {
continue
}
if _, ok := rules[entrypoint]; !ok {
fmt.Fprintf(os.Stderr, "argument error:\nunknown rule name %s used as alternate entrypoint\n", entrypoint)
exit(9)
}
}
if !*noBuildFlag {
if *optimizeGrammar {
ast.Optimize(grammar, altEntrypointsFlag...)
}
// generate parser
out := output(*outputFlag)
defer func() {
err := out.Close()
if err != nil {
fmt.Fprintln(os.Stderr, "close file error:\n", err)
exit(8)
}
}()
outBuf := bytes.NewBuffer([]byte{})
curNmOpt := builder.ReceiverName(*recvrNmFlag)
optimizeParser := builder.Optimize(*optimizeParserFlag)
basicLatinOptimize := builder.BasicLatinLookupTable(*optimizeBasicLatinFlag)
nolintOpt := builder.Nolint(*nolint)
if err := builder.BuildParser(outBuf, grammar, curNmOpt, optimizeParser, basicLatinOptimize, nolintOpt); err != nil {
fmt.Fprintln(os.Stderr, "build error: ", err)
exit(5)
}
// Defaults from golang.org/x/tools/cmd/goimports
options := &imports.Options{
TabWidth: 8,
TabIndent: true,
Comments: true,
Fragment: true,
}
formattedBuf, err := imports.Process("filename", outBuf.Bytes(), options)
if err != nil {
if _, err := out.Write(outBuf.Bytes()); err != nil {
fmt.Fprintln(os.Stderr, "write error: ", err)
exit(7)
}
fmt.Fprintln(os.Stderr, "format error: ", err)
exit(6)
}
if _, err := out.Write(formattedBuf); err != nil {
fmt.Fprintln(os.Stderr, "write error: ", err)
exit(7)
}
}
}
var usagePage = `usage: %s [options] [GRAMMAR_FILE]
Pigeon generates a parser based on a PEG grammar.
By default, pigeon reads the grammar from stdin and writes the
generated parser to stdout. If GRAMMAR_FILE is specified, the
grammar is read from this file instead. If the -o flag is set,
the generated code is written to this file instead.
-cache
cache parser results to avoid exponential parsing time in
pathological cases. Can make the parsing slower for typical
cases and uses more memory.
-debug
output debugging information while parsing the grammar.
-h -help
display this help message.
-nolint
add '// nolint: ...' comments for generated parser to suppress
warnings by gometalinter (https://github.com/alecthomas/gometalinter).
-no-recover
do not recover from a panic. Useful to access the panic stack
when debugging, otherwise the panic is converted to an error.
-o OUTPUT_FILE
write the generated parser to OUTPUT_FILE. Defaults to stdout.
-optimize-basic-latin
generate optimized parser for Unicode Basic Latin character set
-optimize-grammar
performes several performance optimizations on the grammar (EXPERIMENTAL FEATURE)
-optimize-parser
generate optimized parser without Debug and Memoize options and
with some other optimizations applied.
-receiver-name NAME
use NAME as for the receiver name of the generated methods
for the grammar's code blocks. Defaults to "c".
-x
do not generate the parser, only parse the grammar.
-alternate-entrypoints RULE[,RULE...]
comma-separated list of rule names that may be used as alternate
entrypoints for the parser, in addition to the first rule in the
grammar.
See https://godoc.org/github.com/mna/pigeon for more information.
`
// usage prints the help page of the command-line tool.
func usage() {
fmt.Printf(usagePage, os.Args[0])
}
// argError prints an error message to stderr, prints the command usage
// and exits with the specified exit code.
func argError(exitCode int, msg string, args ...interface{}) {
fmt.Fprintf(os.Stderr, msg, args...)
fmt.Fprintln(os.Stderr)
usage()
exit(exitCode)
}
// input gets the name and reader to get input text from.
func input(filename string) (nm string, rc io.ReadCloser) {
nm = "stdin"
inf := os.Stdin
if filename != "" {
f, err := os.Open(filename)
if err != nil {
fmt.Fprintln(os.Stderr, err)
exit(2)
}
inf = f
nm = filename
}
r := bufio.NewReader(inf)
return nm, makeReadCloser(r, inf)
}
// output gets the writer to write the generated parser to.
func output(filename string) io.WriteCloser {
out := os.Stdout
if filename != "" {
f, err := os.Create(filename)
if err != nil {
fmt.Fprintln(os.Stderr, err)
exit(4)
}
out = f
}
return out
}
// create a ReadCloser that reads from r and closes c.
func makeReadCloser(r io.Reader, c io.Closer) io.ReadCloser {
rc := struct {
io.Reader
io.Closer
}{r, c}
return io.ReadCloser(rc)
}
// astPos is a helper method for the PEG grammar parser. It returns the
// position of the current match as an ast.Pos.
func (c *current) astPos() ast.Pos {
return ast.Pos{Line: c.pos.line, Col: c.pos.col, Off: c.pos.offset}
}
// toIfaceSlice is a helper function for the PEG grammar parser. It converts
// v to a slice of empty interfaces.
func toIfaceSlice(v interface{}) []interface{} {
if v == nil {
return nil
}
return v.([]interface{})
}
// validateUnicodeEscape checks that the provided escape sequence is a
// valid Unicode escape sequence.
func validateUnicodeEscape(escape, errMsg string) (interface{}, error) {
r, _, _, err := strconv.UnquoteChar("\\"+escape, '"')
if err != nil {
return nil, errors.New(errMsg)
}
if 0xD800 <= r && r <= 0xDFFF {
return nil, errors.New(errMsg)
}
return nil, nil
}

4526
vendor/github.com/mna/pigeon/pigeon.go generated vendored Normal file
View File

File diff suppressed because it is too large Load Diff

71
vendor/github.com/mna/pigeon/reserved_words.go generated vendored Normal file
View File

@@ -0,0 +1,71 @@
package main
var reservedWords = map[string]bool{
// Go keywords http://golang.org/ref/spec#Keywords
"break": true,
"case": true,
"chan": true,
"const": true,
"continue": true,
"default": true,
"defer": true,
"else": true,
"fallthrough": true,
"for": true,
"func": true,
"goto": true,
"go": true,
"if": true,
"import": true,
"interface": true,
"map": true,
"package": true,
"range": true,
"return": true,
"select": true,
"struct": true,
"switch": true,
"type": true,
"var": true,
// predeclared identifiers http://golang.org/ref/spec#Predeclared_identifiers
"bool": true,
"byte": true,
"complex64": true,
"complex128": true,
"error": true,
"float32": true,
"float64": true,
"int8": true,
"int16": true,
"int32": true,
"int64": true,
"int": true,
"rune": true,
"string": true,
"uint8": true,
"uint16": true,
"uint32": true,
"uint64": true,
"uintptr": true,
"uint": true,
"true": true,
"false": true,
"iota": true,
"nil": true,
"append": true,
"cap": true,
"close": true,
"complex": true,
"copy": true,
"delete": true,
"imag": true,
"len": true,
"make": true,
"new": true,
"panic": true,
"println": true,
"print": true,
"real": true,
"recover": true,
}

200
vendor/github.com/mna/pigeon/unicode_classes.go generated vendored Normal file
View File

@@ -0,0 +1,200 @@
// This file is generated by the misc/cmd/unicode-classes tool.
// Do not edit.
package main
var unicodeClasses = map[string]bool{
"ASCII_Hex_Digit": true,
"Arabic": true,
"Armenian": true,
"Avestan": true,
"Balinese": true,
"Bamum": true,
"Bassa_Vah": true,
"Batak": true,
"Bengali": true,
"Bidi_Control": true,
"Bopomofo": true,
"Brahmi": true,
"Braille": true,
"Buginese": true,
"Buhid": true,
"C": true,
"Canadian_Aboriginal": true,
"Carian": true,
"Caucasian_Albanian": true,
"Cc": true,
"Cf": true,
"Chakma": true,
"Cham": true,
"Cherokee": true,
"Co": true,
"Common": true,
"Coptic": true,
"Cs": true,
"Cuneiform": true,
"Cypriot": true,
"Cyrillic": true,
"Dash": true,
"Deprecated": true,
"Deseret": true,
"Devanagari": true,
"Diacritic": true,
"Duployan": true,
"Egyptian_Hieroglyphs": true,
"Elbasan": true,
"Ethiopic": true,
"Extender": true,
"Georgian": true,
"Glagolitic": true,
"Gothic": true,
"Grantha": true,
"Greek": true,
"Gujarati": true,
"Gurmukhi": true,
"Han": true,
"Hangul": true,
"Hanunoo": true,
"Hebrew": true,
"Hex_Digit": true,
"Hiragana": true,
"Hyphen": true,
"IDS_Binary_Operator": true,
"IDS_Trinary_Operator": true,
"Ideographic": true,
"Imperial_Aramaic": true,
"Inherited": true,
"Inscriptional_Pahlavi": true,
"Inscriptional_Parthian": true,
"Javanese": true,
"Join_Control": true,
"Kaithi": true,
"Kannada": true,
"Katakana": true,
"Kayah_Li": true,
"Kharoshthi": true,
"Khmer": true,
"Khojki": true,
"Khudawadi": true,
"L": true,
"Lao": true,
"Latin": true,
"Lepcha": true,
"Limbu": true,
"Linear_A": true,
"Linear_B": true,
"Lisu": true,
"Ll": true,
"Lm": true,
"Lo": true,
"Logical_Order_Exception": true,
"Lt": true,
"Lu": true,
"Lycian": true,
"Lydian": true,
"M": true,
"Mahajani": true,
"Malayalam": true,
"Mandaic": true,
"Manichaean": true,
"Mc": true,
"Me": true,
"Meetei_Mayek": true,
"Mende_Kikakui": true,
"Meroitic_Cursive": true,
"Meroitic_Hieroglyphs": true,
"Miao": true,
"Mn": true,
"Modi": true,
"Mongolian": true,
"Mro": true,
"Myanmar": true,
"N": true,
"Nabataean": true,
"Nd": true,
"New_Tai_Lue": true,
"Nko": true,
"Nl": true,
"No": true,
"Noncharacter_Code_Point": true,
"Ogham": true,
"Ol_Chiki": true,
"Old_Italic": true,
"Old_North_Arabian": true,
"Old_Permic": true,
"Old_Persian": true,
"Old_South_Arabian": true,
"Old_Turkic": true,
"Oriya": true,
"Osmanya": true,
"Other_Alphabetic": true,
"Other_Default_Ignorable_Code_Point": true,
"Other_Grapheme_Extend": true,
"Other_ID_Continue": true,
"Other_ID_Start": true,
"Other_Lowercase": true,
"Other_Math": true,
"Other_Uppercase": true,
"P": true,
"Pahawh_Hmong": true,
"Palmyrene": true,
"Pattern_Syntax": true,
"Pattern_White_Space": true,
"Pau_Cin_Hau": true,
"Pc": true,
"Pd": true,
"Pe": true,
"Pf": true,
"Phags_Pa": true,
"Phoenician": true,
"Pi": true,
"Po": true,
"Ps": true,
"Psalter_Pahlavi": true,
"Quotation_Mark": true,
"Radical": true,
"Rejang": true,
"Runic": true,
"S": true,
"STerm": true,
"Samaritan": true,
"Saurashtra": true,
"Sc": true,
"Sharada": true,
"Shavian": true,
"Siddham": true,
"Sinhala": true,
"Sk": true,
"Sm": true,
"So": true,
"Soft_Dotted": true,
"Sora_Sompeng": true,
"Sundanese": true,
"Syloti_Nagri": true,
"Syriac": true,
"Tagalog": true,
"Tagbanwa": true,
"Tai_Le": true,
"Tai_Tham": true,
"Tai_Viet": true,
"Takri": true,
"Tamil": true,
"Telugu": true,
"Terminal_Punctuation": true,
"Thaana": true,
"Thai": true,
"Tibetan": true,
"Tifinagh": true,
"Tirhuta": true,
"Ugaritic": true,
"Unified_Ideograph": true,
"Vai": true,
"Variation_Selector": true,
"Warang_Citi": true,
"White_Space": true,
"Yi": true,
"Z": true,
"Zl": true,
"Zp": true,
"Zs": true,
}

78
vendor/golang.org/x/mod/internal/lazyregexp/lazyre.go generated vendored Normal file
View File

@@ -0,0 +1,78 @@
// Copyright 2018 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package lazyregexp is a thin wrapper over regexp, allowing the use of global
// regexp variables without forcing them to be compiled at init.
package lazyregexp
import (
"os"
"regexp"
"strings"
"sync"
)
// Regexp is a wrapper around [regexp.Regexp], where the underlying regexp will be
// compiled the first time it is needed.
type Regexp struct {
str string
once sync.Once
rx *regexp.Regexp
}
func (r *Regexp) re() *regexp.Regexp {
r.once.Do(r.build)
return r.rx
}
func (r *Regexp) build() {
r.rx = regexp.MustCompile(r.str)
r.str = ""
}
func (r *Regexp) FindSubmatch(s []byte) [][]byte {
return r.re().FindSubmatch(s)
}
func (r *Regexp) FindStringSubmatch(s string) []string {
return r.re().FindStringSubmatch(s)
}
func (r *Regexp) FindStringSubmatchIndex(s string) []int {
return r.re().FindStringSubmatchIndex(s)
}
func (r *Regexp) ReplaceAllString(src, repl string) string {
return r.re().ReplaceAllString(src, repl)
}
func (r *Regexp) FindString(s string) string {
return r.re().FindString(s)
}
func (r *Regexp) FindAllString(s string, n int) []string {
return r.re().FindAllString(s, n)
}
func (r *Regexp) MatchString(s string) bool {
return r.re().MatchString(s)
}
func (r *Regexp) SubexpNames() []string {
return r.re().SubexpNames()
}
var inTest = len(os.Args) > 0 && strings.HasSuffix(strings.TrimSuffix(os.Args[0], ".exe"), ".test")
// New creates a new lazy regexp, delaying the compiling work until it is first
// needed. If the code is being run as part of tests, the regexp compiling will
// happen immediately.
func New(str string) *Regexp {
lr := &Regexp{str: str}
if inTest {
// In tests, always compile the regexps early.
lr.re()
}
return lr
}

841
vendor/golang.org/x/mod/module/module.go generated vendored Normal file
View File

@@ -0,0 +1,841 @@
// Copyright 2018 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package module defines the module.Version type along with support code.
//
// The [module.Version] type is a simple Path, Version pair:
//
// type Version struct {
// Path string
// Version string
// }
//
// There are no restrictions imposed directly by use of this structure,
// but additional checking functions, most notably [Check], verify that
// a particular path, version pair is valid.
//
// # Escaped Paths
//
// Module paths appear as substrings of file system paths
// (in the download cache) and of web server URLs in the proxy protocol.
// In general we cannot rely on file systems to be case-sensitive,
// nor can we rely on web servers, since they read from file systems.
// That is, we cannot rely on the file system to keep rsc.io/QUOTE
// and rsc.io/quote separate. Windows and macOS don't.
// Instead, we must never require two different casings of a file path.
// Because we want the download cache to match the proxy protocol,
// and because we want the proxy protocol to be possible to serve
// from a tree of static files (which might be stored on a case-insensitive
// file system), the proxy protocol must never require two different casings
// of a URL path either.
//
// One possibility would be to make the escaped form be the lowercase
// hexadecimal encoding of the actual path bytes. This would avoid ever
// needing different casings of a file path, but it would be fairly illegible
// to most programmers when those paths appeared in the file system
// (including in file paths in compiler errors and stack traces)
// in web server logs, and so on. Instead, we want a safe escaped form that
// leaves most paths unaltered.
//
// The safe escaped form is to replace every uppercase letter
// with an exclamation mark followed by the letter's lowercase equivalent.
//
// For example,
//
// github.com/Azure/azure-sdk-for-go -> github.com/!azure/azure-sdk-for-go.
// github.com/GoogleCloudPlatform/cloudsql-proxy -> github.com/!google!cloud!platform/cloudsql-proxy
// github.com/Sirupsen/logrus -> github.com/!sirupsen/logrus.
//
// Import paths that avoid upper-case letters are left unchanged.
// Note that because import paths are ASCII-only and avoid various
// problematic punctuation (like : < and >), the escaped form is also ASCII-only
// and avoids the same problematic punctuation.
//
// Import paths have never allowed exclamation marks, so there is no
// need to define how to escape a literal !.
//
// # Unicode Restrictions
//
// Today, paths are disallowed from using Unicode.
//
// Although paths are currently disallowed from using Unicode,
// we would like at some point to allow Unicode letters as well, to assume that
// file systems and URLs are Unicode-safe (storing UTF-8), and apply
// the !-for-uppercase convention for escaping them in the file system.
// But there are at least two subtle considerations.
//
// First, note that not all case-fold equivalent distinct runes
// form an upper/lower pair.
// For example, U+004B ('K'), U+006B ('k'), and U+212A ('K' for Kelvin)
// are three distinct runes that case-fold to each other.
// When we do add Unicode letters, we must not assume that upper/lower
// are the only case-equivalent pairs.
// Perhaps the Kelvin symbol would be disallowed entirely, for example.
// Or perhaps it would escape as "!!k", or perhaps as "(212A)".
//
// Second, it would be nice to allow Unicode marks as well as letters,
// but marks include combining marks, and then we must deal not
// only with case folding but also normalization: both U+00E9 ('é')
// and U+0065 U+0301 ('e' followed by combining acute accent)
// look the same on the page and are treated by some file systems
// as the same path. If we do allow Unicode marks in paths, there
// must be some kind of normalization to allow only one canonical
// encoding of any character used in an import path.
package module
// IMPORTANT NOTE
//
// This file essentially defines the set of valid import paths for the go command.
// There are many subtle considerations, including Unicode ambiguity,
// security, network, and file system representations.
//
// This file also defines the set of valid module path and version combinations,
// another topic with many subtle considerations.
//
// Changes to the semantics in this file require approval from rsc.
import (
"errors"
"fmt"
"path"
"sort"
"strings"
"unicode"
"unicode/utf8"
"golang.org/x/mod/semver"
)
// A Version (for clients, a module.Version) is defined by a module path and version pair.
// These are stored in their plain (unescaped) form.
type Version struct {
// Path is a module path, like "golang.org/x/text" or "rsc.io/quote/v2".
Path string
// Version is usually a semantic version in canonical form.
// There are three exceptions to this general rule.
// First, the top-level target of a build has no specific version
// and uses Version = "".
// Second, during MVS calculations the version "none" is used
// to represent the decision to take no version of a given module.
// Third, filesystem paths found in "replace" directives are
// represented by a path with an empty version.
Version string `json:",omitempty"`
}
// String returns a representation of the Version suitable for logging
// (Path@Version, or just Path if Version is empty).
func (m Version) String() string {
if m.Version == "" {
return m.Path
}
return m.Path + "@" + m.Version
}
// A ModuleError indicates an error specific to a module.
type ModuleError struct {
Path string
Version string
Err error
}
// VersionError returns a [ModuleError] derived from a [Version] and error,
// or err itself if it is already such an error.
func VersionError(v Version, err error) error {
var mErr *ModuleError
if errors.As(err, &mErr) && mErr.Path == v.Path && mErr.Version == v.Version {
return err
}
return &ModuleError{
Path: v.Path,
Version: v.Version,
Err: err,
}
}
func (e *ModuleError) Error() string {
if v, ok := e.Err.(*InvalidVersionError); ok {
return fmt.Sprintf("%s@%s: invalid %s: %v", e.Path, v.Version, v.noun(), v.Err)
}
if e.Version != "" {
return fmt.Sprintf("%s@%s: %v", e.Path, e.Version, e.Err)
}
return fmt.Sprintf("module %s: %v", e.Path, e.Err)
}
func (e *ModuleError) Unwrap() error { return e.Err }
// An InvalidVersionError indicates an error specific to a version, with the
// module path unknown or specified externally.
//
// A [ModuleError] may wrap an InvalidVersionError, but an InvalidVersionError
// must not wrap a ModuleError.
type InvalidVersionError struct {
Version string
Pseudo bool
Err error
}
// noun returns either "version" or "pseudo-version", depending on whether
// e.Version is a pseudo-version.
func (e *InvalidVersionError) noun() string {
if e.Pseudo {
return "pseudo-version"
}
return "version"
}
func (e *InvalidVersionError) Error() string {
return fmt.Sprintf("%s %q invalid: %s", e.noun(), e.Version, e.Err)
}
func (e *InvalidVersionError) Unwrap() error { return e.Err }
// An InvalidPathError indicates a module, import, or file path doesn't
// satisfy all naming constraints. See [CheckPath], [CheckImportPath],
// and [CheckFilePath] for specific restrictions.
type InvalidPathError struct {
Kind string // "module", "import", or "file"
Path string
Err error
}
func (e *InvalidPathError) Error() string {
return fmt.Sprintf("malformed %s path %q: %v", e.Kind, e.Path, e.Err)
}
func (e *InvalidPathError) Unwrap() error { return e.Err }
// Check checks that a given module path, version pair is valid.
// In addition to the path being a valid module path
// and the version being a valid semantic version,
// the two must correspond.
// For example, the path "yaml/v2" only corresponds to
// semantic versions beginning with "v2.".
func Check(path, version string) error {
if err := CheckPath(path); err != nil {
return err
}
if !semver.IsValid(version) {
return &ModuleError{
Path: path,
Err: &InvalidVersionError{Version: version, Err: errors.New("not a semantic version")},
}
}
_, pathMajor, _ := SplitPathVersion(path)
if err := CheckPathMajor(version, pathMajor); err != nil {
return &ModuleError{Path: path, Err: err}
}
return nil
}
// firstPathOK reports whether r can appear in the first element of a module path.
// The first element of the path must be an LDH domain name, at least for now.
// To avoid case ambiguity, the domain name must be entirely lower case.
func firstPathOK(r rune) bool {
return r == '-' || r == '.' ||
'0' <= r && r <= '9' ||
'a' <= r && r <= 'z'
}
// modPathOK reports whether r can appear in a module path element.
// Paths can be ASCII letters, ASCII digits, and limited ASCII punctuation: - . _ and ~.
//
// This matches what "go get" has historically recognized in import paths,
// and avoids confusing sequences like '%20' or '+' that would change meaning
// if used in a URL.
//
// TODO(rsc): We would like to allow Unicode letters, but that requires additional
// care in the safe encoding (see "escaped paths" above).
func modPathOK(r rune) bool {
if r < utf8.RuneSelf {
return r == '-' || r == '.' || r == '_' || r == '~' ||
'0' <= r && r <= '9' ||
'A' <= r && r <= 'Z' ||
'a' <= r && r <= 'z'
}
return false
}
// importPathOK reports whether r can appear in a package import path element.
//
// Import paths are intermediate between module paths and file paths: we allow
// disallow characters that would be confusing or ambiguous as arguments to
// 'go get' (such as '@' and ' ' ), but allow certain characters that are
// otherwise-unambiguous on the command line and historically used for some
// binary names (such as '++' as a suffix for compiler binaries and wrappers).
func importPathOK(r rune) bool {
return modPathOK(r) || r == '+'
}
// fileNameOK reports whether r can appear in a file name.
// For now we allow all Unicode letters but otherwise limit to pathOK plus a few more punctuation characters.
// If we expand the set of allowed characters here, we have to
// work harder at detecting potential case-folding and normalization collisions.
// See note about "escaped paths" above.
func fileNameOK(r rune) bool {
if r < utf8.RuneSelf {
// Entire set of ASCII punctuation, from which we remove characters:
// ! " # $ % & ' ( ) * + , - . / : ; < = > ? @ [ \ ] ^ _ ` { | } ~
// We disallow some shell special characters: " ' * < > ? ` |
// (Note that some of those are disallowed by the Windows file system as well.)
// We also disallow path separators / : and \ (fileNameOK is only called on path element characters).
// We allow spaces (U+0020) in file names.
const allowed = "!#$%&()+,-.=@[]^_{}~ "
if '0' <= r && r <= '9' || 'A' <= r && r <= 'Z' || 'a' <= r && r <= 'z' {
return true
}
return strings.ContainsRune(allowed, r)
}
// It may be OK to add more ASCII punctuation here, but only carefully.
// For example Windows disallows < > \, and macOS disallows :, so we must not allow those.
return unicode.IsLetter(r)
}
// CheckPath checks that a module path is valid.
// A valid module path is a valid import path, as checked by [CheckImportPath],
// with three additional constraints.
// First, the leading path element (up to the first slash, if any),
// by convention a domain name, must contain only lower-case ASCII letters,
// ASCII digits, dots (U+002E), and dashes (U+002D);
// it must contain at least one dot and cannot start with a dash.
// Second, for a final path element of the form /vN, where N looks numeric
// (ASCII digits and dots) must not begin with a leading zero, must not be /v1,
// and must not contain any dots. For paths beginning with "gopkg.in/",
// this second requirement is replaced by a requirement that the path
// follow the gopkg.in server's conventions.
// Third, no path element may begin with a dot.
func CheckPath(path string) (err error) {
defer func() {
if err != nil {
err = &InvalidPathError{Kind: "module", Path: path, Err: err}
}
}()
if err := checkPath(path, modulePath); err != nil {
return err
}
i := strings.Index(path, "/")
if i < 0 {
i = len(path)
}
if i == 0 {
return fmt.Errorf("leading slash")
}
if !strings.Contains(path[:i], ".") {
return fmt.Errorf("missing dot in first path element")
}
if path[0] == '-' {
return fmt.Errorf("leading dash in first path element")
}
for _, r := range path[:i] {
if !firstPathOK(r) {
return fmt.Errorf("invalid char %q in first path element", r)
}
}
if _, _, ok := SplitPathVersion(path); !ok {
return fmt.Errorf("invalid version")
}
return nil
}
// CheckImportPath checks that an import path is valid.
//
// A valid import path consists of one or more valid path elements
// separated by slashes (U+002F). (It must not begin with nor end in a slash.)
//
// A valid path element is a non-empty string made up of
// ASCII letters, ASCII digits, and limited ASCII punctuation: - . _ and ~.
// It must not end with a dot (U+002E), nor contain two dots in a row.
//
// The element prefix up to the first dot must not be a reserved file name
// on Windows, regardless of case (CON, com1, NuL, and so on). The element
// must not have a suffix of a tilde followed by one or more ASCII digits
// (to exclude paths elements that look like Windows short-names).
//
// CheckImportPath may be less restrictive in the future, but see the
// top-level package documentation for additional information about
// subtleties of Unicode.
func CheckImportPath(path string) error {
if err := checkPath(path, importPath); err != nil {
return &InvalidPathError{Kind: "import", Path: path, Err: err}
}
return nil
}
// pathKind indicates what kind of path we're checking. Module paths,
// import paths, and file paths have different restrictions.
type pathKind int
const (
modulePath pathKind = iota
importPath
filePath
)
// checkPath checks that a general path is valid. kind indicates what
// specific constraints should be applied.
//
// checkPath returns an error describing why the path is not valid.
// Because these checks apply to module, import, and file paths,
// and because other checks may be applied, the caller is expected to wrap
// this error with [InvalidPathError].
func checkPath(path string, kind pathKind) error {
if !utf8.ValidString(path) {
return fmt.Errorf("invalid UTF-8")
}
if path == "" {
return fmt.Errorf("empty string")
}
if path[0] == '-' && kind != filePath {
return fmt.Errorf("leading dash")
}
if strings.Contains(path, "//") {
return fmt.Errorf("double slash")
}
if path[len(path)-1] == '/' {
return fmt.Errorf("trailing slash")
}
elemStart := 0
for i, r := range path {
if r == '/' {
if err := checkElem(path[elemStart:i], kind); err != nil {
return err
}
elemStart = i + 1
}
}
if err := checkElem(path[elemStart:], kind); err != nil {
return err
}
return nil
}
// checkElem checks whether an individual path element is valid.
func checkElem(elem string, kind pathKind) error {
if elem == "" {
return fmt.Errorf("empty path element")
}
if strings.Count(elem, ".") == len(elem) {
return fmt.Errorf("invalid path element %q", elem)
}
if elem[0] == '.' && kind == modulePath {
return fmt.Errorf("leading dot in path element")
}
if elem[len(elem)-1] == '.' {
return fmt.Errorf("trailing dot in path element")
}
for _, r := range elem {
ok := false
switch kind {
case modulePath:
ok = modPathOK(r)
case importPath:
ok = importPathOK(r)
case filePath:
ok = fileNameOK(r)
default:
panic(fmt.Sprintf("internal error: invalid kind %v", kind))
}
if !ok {
return fmt.Errorf("invalid char %q", r)
}
}
// Windows disallows a bunch of path elements, sadly.
// See https://docs.microsoft.com/en-us/windows/desktop/fileio/naming-a-file
short := elem
if i := strings.Index(short, "."); i >= 0 {
short = short[:i]
}
for _, bad := range badWindowsNames {
if strings.EqualFold(bad, short) {
return fmt.Errorf("%q disallowed as path element component on Windows", short)
}
}
if kind == filePath {
// don't check for Windows short-names in file names. They're
// only an issue for import paths.
return nil
}
// Reject path components that look like Windows short-names.
// Those usually end in a tilde followed by one or more ASCII digits.
if tilde := strings.LastIndexByte(short, '~'); tilde >= 0 && tilde < len(short)-1 {
suffix := short[tilde+1:]
suffixIsDigits := true
for _, r := range suffix {
if r < '0' || r > '9' {
suffixIsDigits = false
break
}
}
if suffixIsDigits {
return fmt.Errorf("trailing tilde and digits in path element")
}
}
return nil
}
// CheckFilePath checks that a slash-separated file path is valid.
// The definition of a valid file path is the same as the definition
// of a valid import path except that the set of allowed characters is larger:
// all Unicode letters, ASCII digits, the ASCII space character (U+0020),
// and the ASCII punctuation characters
// “!#$%&()+,-.=@[]^_{}~”.
// (The excluded punctuation characters, " * < > ? ` ' | / \ and :,
// have special meanings in certain shells or operating systems.)
//
// CheckFilePath may be less restrictive in the future, but see the
// top-level package documentation for additional information about
// subtleties of Unicode.
func CheckFilePath(path string) error {
if err := checkPath(path, filePath); err != nil {
return &InvalidPathError{Kind: "file", Path: path, Err: err}
}
return nil
}
// badWindowsNames are the reserved file path elements on Windows.
// See https://docs.microsoft.com/en-us/windows/desktop/fileio/naming-a-file
var badWindowsNames = []string{
"CON",
"PRN",
"AUX",
"NUL",
"COM1",
"COM2",
"COM3",
"COM4",
"COM5",
"COM6",
"COM7",
"COM8",
"COM9",
"LPT1",
"LPT2",
"LPT3",
"LPT4",
"LPT5",
"LPT6",
"LPT7",
"LPT8",
"LPT9",
}
// SplitPathVersion returns prefix and major version such that prefix+pathMajor == path
// and version is either empty or "/vN" for N >= 2.
// As a special case, gopkg.in paths are recognized directly;
// they require ".vN" instead of "/vN", and for all N, not just N >= 2.
// SplitPathVersion returns with ok = false when presented with
// a path whose last path element does not satisfy the constraints
// applied by [CheckPath], such as "example.com/pkg/v1" or "example.com/pkg/v1.2".
func SplitPathVersion(path string) (prefix, pathMajor string, ok bool) {
if strings.HasPrefix(path, "gopkg.in/") {
return splitGopkgIn(path)
}
i := len(path)
dot := false
for i > 0 && ('0' <= path[i-1] && path[i-1] <= '9' || path[i-1] == '.') {
if path[i-1] == '.' {
dot = true
}
i--
}
if i <= 1 || i == len(path) || path[i-1] != 'v' || path[i-2] != '/' {
return path, "", true
}
prefix, pathMajor = path[:i-2], path[i-2:]
if dot || len(pathMajor) <= 2 || pathMajor[2] == '0' || pathMajor == "/v1" {
return path, "", false
}
return prefix, pathMajor, true
}
// splitGopkgIn is like SplitPathVersion but only for gopkg.in paths.
func splitGopkgIn(path string) (prefix, pathMajor string, ok bool) {
if !strings.HasPrefix(path, "gopkg.in/") {
return path, "", false
}
i := len(path)
if strings.HasSuffix(path, "-unstable") {
i -= len("-unstable")
}
for i > 0 && ('0' <= path[i-1] && path[i-1] <= '9') {
i--
}
if i <= 1 || path[i-1] != 'v' || path[i-2] != '.' {
// All gopkg.in paths must end in vN for some N.
return path, "", false
}
prefix, pathMajor = path[:i-2], path[i-2:]
if len(pathMajor) <= 2 || pathMajor[2] == '0' && pathMajor != ".v0" {
return path, "", false
}
return prefix, pathMajor, true
}
// MatchPathMajor reports whether the semantic version v
// matches the path major version pathMajor.
//
// MatchPathMajor returns true if and only if [CheckPathMajor] returns nil.
func MatchPathMajor(v, pathMajor string) bool {
return CheckPathMajor(v, pathMajor) == nil
}
// CheckPathMajor returns a non-nil error if the semantic version v
// does not match the path major version pathMajor.
func CheckPathMajor(v, pathMajor string) error {
// TODO(jayconrod): return errors or panic for invalid inputs. This function
// (and others) was covered by integration tests for cmd/go, and surrounding
// code protected against invalid inputs like non-canonical versions.
if strings.HasPrefix(pathMajor, ".v") && strings.HasSuffix(pathMajor, "-unstable") {
pathMajor = strings.TrimSuffix(pathMajor, "-unstable")
}
if strings.HasPrefix(v, "v0.0.0-") && pathMajor == ".v1" {
// Allow old bug in pseudo-versions that generated v0.0.0- pseudoversion for gopkg .v1.
// For example, gopkg.in/yaml.v2@v2.2.1's go.mod requires gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405.
return nil
}
m := semver.Major(v)
if pathMajor == "" {
if m == "v0" || m == "v1" || semver.Build(v) == "+incompatible" {
return nil
}
pathMajor = "v0 or v1"
} else if pathMajor[0] == '/' || pathMajor[0] == '.' {
if m == pathMajor[1:] {
return nil
}
pathMajor = pathMajor[1:]
}
return &InvalidVersionError{
Version: v,
Err: fmt.Errorf("should be %s, not %s", pathMajor, semver.Major(v)),
}
}
// PathMajorPrefix returns the major-version tag prefix implied by pathMajor.
// An empty PathMajorPrefix allows either v0 or v1.
//
// Note that [MatchPathMajor] may accept some versions that do not actually begin
// with this prefix: namely, it accepts a 'v0.0.0-' prefix for a '.v1'
// pathMajor, even though that pathMajor implies 'v1' tagging.
func PathMajorPrefix(pathMajor string) string {
if pathMajor == "" {
return ""
}
if pathMajor[0] != '/' && pathMajor[0] != '.' {
panic("pathMajor suffix " + pathMajor + " passed to PathMajorPrefix lacks separator")
}
if strings.HasPrefix(pathMajor, ".v") && strings.HasSuffix(pathMajor, "-unstable") {
pathMajor = strings.TrimSuffix(pathMajor, "-unstable")
}
m := pathMajor[1:]
if m != semver.Major(m) {
panic("pathMajor suffix " + pathMajor + "passed to PathMajorPrefix is not a valid major version")
}
return m
}
// CanonicalVersion returns the canonical form of the version string v.
// It is the same as [semver.Canonical] except that it preserves the special build suffix "+incompatible".
func CanonicalVersion(v string) string {
cv := semver.Canonical(v)
if semver.Build(v) == "+incompatible" {
cv += "+incompatible"
}
return cv
}
// Sort sorts the list by Path, breaking ties by comparing [Version] fields.
// The Version fields are interpreted as semantic versions (using [semver.Compare])
// optionally followed by a tie-breaking suffix introduced by a slash character,
// like in "v0.0.1/go.mod".
func Sort(list []Version) {
sort.Slice(list, func(i, j int) bool {
mi := list[i]
mj := list[j]
if mi.Path != mj.Path {
return mi.Path < mj.Path
}
// To help go.sum formatting, allow version/file.
// Compare semver prefix by semver rules,
// file by string order.
vi := mi.Version
vj := mj.Version
var fi, fj string
if k := strings.Index(vi, "/"); k >= 0 {
vi, fi = vi[:k], vi[k:]
}
if k := strings.Index(vj, "/"); k >= 0 {
vj, fj = vj[:k], vj[k:]
}
if vi != vj {
return semver.Compare(vi, vj) < 0
}
return fi < fj
})
}
// EscapePath returns the escaped form of the given module path.
// It fails if the module path is invalid.
func EscapePath(path string) (escaped string, err error) {
if err := CheckPath(path); err != nil {
return "", err
}
return escapeString(path)
}
// EscapeVersion returns the escaped form of the given module version.
// Versions are allowed to be in non-semver form but must be valid file names
// and not contain exclamation marks.
func EscapeVersion(v string) (escaped string, err error) {
if err := checkElem(v, filePath); err != nil || strings.Contains(v, "!") {
return "", &InvalidVersionError{
Version: v,
Err: fmt.Errorf("disallowed version string"),
}
}
return escapeString(v)
}
func escapeString(s string) (escaped string, err error) {
haveUpper := false
for _, r := range s {
if r == '!' || r >= utf8.RuneSelf {
// This should be disallowed by CheckPath, but diagnose anyway.
// The correctness of the escaping loop below depends on it.
return "", fmt.Errorf("internal error: inconsistency in EscapePath")
}
if 'A' <= r && r <= 'Z' {
haveUpper = true
}
}
if !haveUpper {
return s, nil
}
var buf []byte
for _, r := range s {
if 'A' <= r && r <= 'Z' {
buf = append(buf, '!', byte(r+'a'-'A'))
} else {
buf = append(buf, byte(r))
}
}
return string(buf), nil
}
// UnescapePath returns the module path for the given escaped path.
// It fails if the escaped path is invalid or describes an invalid path.
func UnescapePath(escaped string) (path string, err error) {
path, ok := unescapeString(escaped)
if !ok {
return "", fmt.Errorf("invalid escaped module path %q", escaped)
}
if err := CheckPath(path); err != nil {
return "", fmt.Errorf("invalid escaped module path %q: %v", escaped, err)
}
return path, nil
}
// UnescapeVersion returns the version string for the given escaped version.
// It fails if the escaped form is invalid or describes an invalid version.
// Versions are allowed to be in non-semver form but must be valid file names
// and not contain exclamation marks.
func UnescapeVersion(escaped string) (v string, err error) {
v, ok := unescapeString(escaped)
if !ok {
return "", fmt.Errorf("invalid escaped version %q", escaped)
}
if err := checkElem(v, filePath); err != nil {
return "", fmt.Errorf("invalid escaped version %q: %v", v, err)
}
return v, nil
}
func unescapeString(escaped string) (string, bool) {
var buf []byte
bang := false
for _, r := range escaped {
if r >= utf8.RuneSelf {
return "", false
}
if bang {
bang = false
if r < 'a' || 'z' < r {
return "", false
}
buf = append(buf, byte(r+'A'-'a'))
continue
}
if r == '!' {
bang = true
continue
}
if 'A' <= r && r <= 'Z' {
return "", false
}
buf = append(buf, byte(r))
}
if bang {
return "", false
}
return string(buf), true
}
// MatchPrefixPatterns reports whether any path prefix of target matches one of
// the glob patterns (as defined by [path.Match]) in the comma-separated globs
// list. This implements the algorithm used when matching a module path to the
// GOPRIVATE environment variable, as described by 'go help module-private'.
//
// It ignores any empty or malformed patterns in the list.
// Trailing slashes on patterns are ignored.
func MatchPrefixPatterns(globs, target string) bool {
for globs != "" {
// Extract next non-empty glob in comma-separated list.
var glob string
if i := strings.Index(globs, ","); i >= 0 {
glob, globs = globs[:i], globs[i+1:]
} else {
glob, globs = globs, ""
}
glob = strings.TrimSuffix(glob, "/")
if glob == "" {
continue
}
// A glob with N+1 path elements (N slashes) needs to be matched
// against the first N+1 path elements of target,
// which end just before the N+1'th slash.
n := strings.Count(glob, "/")
prefix := target
// Walk target, counting slashes, truncating at the N+1'th slash.
for i := 0; i < len(target); i++ {
if target[i] == '/' {
if n == 0 {
prefix = target[:i]
break
}
n--
}
}
if n > 0 {
// Not enough prefix elements.
continue
}
matched, _ := path.Match(glob, prefix)
if matched {
return true
}
}
return false
}

250
vendor/golang.org/x/mod/module/pseudo.go generated vendored Normal file
View File

@@ -0,0 +1,250 @@
// Copyright 2018 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Pseudo-versions
//
// Code authors are expected to tag the revisions they want users to use,
// including prereleases. However, not all authors tag versions at all,
// and not all commits a user might want to try will have tags.
// A pseudo-version is a version with a special form that allows us to
// address an untagged commit and order that version with respect to
// other versions we might encounter.
//
// A pseudo-version takes one of the general forms:
//
// (1) vX.0.0-yyyymmddhhmmss-abcdef123456
// (2) vX.Y.(Z+1)-0.yyyymmddhhmmss-abcdef123456
// (3) vX.Y.(Z+1)-0.yyyymmddhhmmss-abcdef123456+incompatible
// (4) vX.Y.Z-pre.0.yyyymmddhhmmss-abcdef123456
// (5) vX.Y.Z-pre.0.yyyymmddhhmmss-abcdef123456+incompatible
//
// If there is no recently tagged version with the right major version vX,
// then form (1) is used, creating a space of pseudo-versions at the bottom
// of the vX version range, less than any tagged version, including the unlikely v0.0.0.
//
// If the most recent tagged version before the target commit is vX.Y.Z or vX.Y.Z+incompatible,
// then the pseudo-version uses form (2) or (3), making it a prerelease for the next
// possible semantic version after vX.Y.Z. The leading 0 segment in the prerelease string
// ensures that the pseudo-version compares less than possible future explicit prereleases
// like vX.Y.(Z+1)-rc1 or vX.Y.(Z+1)-1.
//
// If the most recent tagged version before the target commit is vX.Y.Z-pre or vX.Y.Z-pre+incompatible,
// then the pseudo-version uses form (4) or (5), making it a slightly later prerelease.
package module
import (
"errors"
"fmt"
"strings"
"time"
"golang.org/x/mod/internal/lazyregexp"
"golang.org/x/mod/semver"
)
var pseudoVersionRE = lazyregexp.New(`^v[0-9]+\.(0\.0-|\d+\.\d+-([^+]*\.)?0\.)\d{14}-[A-Za-z0-9]+(\+[0-9A-Za-z-]+(\.[0-9A-Za-z-]+)*)?$`)
const PseudoVersionTimestampFormat = "20060102150405"
// PseudoVersion returns a pseudo-version for the given major version ("v1")
// preexisting older tagged version ("" or "v1.2.3" or "v1.2.3-pre"), revision time,
// and revision identifier (usually a 12-byte commit hash prefix).
func PseudoVersion(major, older string, t time.Time, rev string) string {
if major == "" {
major = "v0"
}
segment := fmt.Sprintf("%s-%s", t.UTC().Format(PseudoVersionTimestampFormat), rev)
build := semver.Build(older)
older = semver.Canonical(older)
if older == "" {
return major + ".0.0-" + segment // form (1)
}
if semver.Prerelease(older) != "" {
return older + ".0." + segment + build // form (4), (5)
}
// Form (2), (3).
// Extract patch from vMAJOR.MINOR.PATCH
i := strings.LastIndex(older, ".") + 1
v, patch := older[:i], older[i:]
// Reassemble.
return v + incDecimal(patch) + "-0." + segment + build
}
// ZeroPseudoVersion returns a pseudo-version with a zero timestamp and
// revision, which may be used as a placeholder.
func ZeroPseudoVersion(major string) string {
return PseudoVersion(major, "", time.Time{}, "000000000000")
}
// incDecimal returns the decimal string incremented by 1.
func incDecimal(decimal string) string {
// Scan right to left turning 9s to 0s until you find a digit to increment.
digits := []byte(decimal)
i := len(digits) - 1
for ; i >= 0 && digits[i] == '9'; i-- {
digits[i] = '0'
}
if i >= 0 {
digits[i]++
} else {
// digits is all zeros
digits[0] = '1'
digits = append(digits, '0')
}
return string(digits)
}
// decDecimal returns the decimal string decremented by 1, or the empty string
// if the decimal is all zeroes.
func decDecimal(decimal string) string {
// Scan right to left turning 0s to 9s until you find a digit to decrement.
digits := []byte(decimal)
i := len(digits) - 1
for ; i >= 0 && digits[i] == '0'; i-- {
digits[i] = '9'
}
if i < 0 {
// decimal is all zeros
return ""
}
if i == 0 && digits[i] == '1' && len(digits) > 1 {
digits = digits[1:]
} else {
digits[i]--
}
return string(digits)
}
// IsPseudoVersion reports whether v is a pseudo-version.
func IsPseudoVersion(v string) bool {
return strings.Count(v, "-") >= 2 && semver.IsValid(v) && pseudoVersionRE.MatchString(v)
}
// IsZeroPseudoVersion returns whether v is a pseudo-version with a zero base,
// timestamp, and revision, as returned by [ZeroPseudoVersion].
func IsZeroPseudoVersion(v string) bool {
return v == ZeroPseudoVersion(semver.Major(v))
}
// PseudoVersionTime returns the time stamp of the pseudo-version v.
// It returns an error if v is not a pseudo-version or if the time stamp
// embedded in the pseudo-version is not a valid time.
func PseudoVersionTime(v string) (time.Time, error) {
_, timestamp, _, _, err := parsePseudoVersion(v)
if err != nil {
return time.Time{}, err
}
t, err := time.Parse("20060102150405", timestamp)
if err != nil {
return time.Time{}, &InvalidVersionError{
Version: v,
Pseudo: true,
Err: fmt.Errorf("malformed time %q", timestamp),
}
}
return t, nil
}
// PseudoVersionRev returns the revision identifier of the pseudo-version v.
// It returns an error if v is not a pseudo-version.
func PseudoVersionRev(v string) (rev string, err error) {
_, _, rev, _, err = parsePseudoVersion(v)
return
}
// PseudoVersionBase returns the canonical parent version, if any, upon which
// the pseudo-version v is based.
//
// If v has no parent version (that is, if it is "vX.0.0-[…]"),
// PseudoVersionBase returns the empty string and a nil error.
func PseudoVersionBase(v string) (string, error) {
base, _, _, build, err := parsePseudoVersion(v)
if err != nil {
return "", err
}
switch pre := semver.Prerelease(base); pre {
case "":
// vX.0.0-yyyymmddhhmmss-abcdef123456 → ""
if build != "" {
// Pseudo-versions of the form vX.0.0-yyyymmddhhmmss-abcdef123456+incompatible
// are nonsensical: the "vX.0.0-" prefix implies that there is no parent tag,
// but the "+incompatible" suffix implies that the major version of
// the parent tag is not compatible with the module's import path.
//
// There are a few such entries in the index generated by proxy.golang.org,
// but we believe those entries were generated by the proxy itself.
return "", &InvalidVersionError{
Version: v,
Pseudo: true,
Err: fmt.Errorf("lacks base version, but has build metadata %q", build),
}
}
return "", nil
case "-0":
// vX.Y.(Z+1)-0.yyyymmddhhmmss-abcdef123456 → vX.Y.Z
// vX.Y.(Z+1)-0.yyyymmddhhmmss-abcdef123456+incompatible → vX.Y.Z+incompatible
base = strings.TrimSuffix(base, pre)
i := strings.LastIndexByte(base, '.')
if i < 0 {
panic("base from parsePseudoVersion missing patch number: " + base)
}
patch := decDecimal(base[i+1:])
if patch == "" {
// vX.0.0-0 is invalid, but has been observed in the wild in the index
// generated by requests to proxy.golang.org.
//
// NOTE(bcmills): I cannot find a historical bug that accounts for
// pseudo-versions of this form, nor have I seen such versions in any
// actual go.mod files. If we find actual examples of this form and a
// reasonable theory of how they came into existence, it seems fine to
// treat them as equivalent to vX.0.0 (especially since the invalid
// pseudo-versions have lower precedence than the real ones). For now, we
// reject them.
return "", &InvalidVersionError{
Version: v,
Pseudo: true,
Err: fmt.Errorf("version before %s would have negative patch number", base),
}
}
return base[:i+1] + patch + build, nil
default:
// vX.Y.Z-pre.0.yyyymmddhhmmss-abcdef123456 → vX.Y.Z-pre
// vX.Y.Z-pre.0.yyyymmddhhmmss-abcdef123456+incompatible → vX.Y.Z-pre+incompatible
if !strings.HasSuffix(base, ".0") {
panic(`base from parsePseudoVersion missing ".0" before date: ` + base)
}
return strings.TrimSuffix(base, ".0") + build, nil
}
}
var errPseudoSyntax = errors.New("syntax error")
func parsePseudoVersion(v string) (base, timestamp, rev, build string, err error) {
if !IsPseudoVersion(v) {
return "", "", "", "", &InvalidVersionError{
Version: v,
Pseudo: true,
Err: errPseudoSyntax,
}
}
build = semver.Build(v)
v = strings.TrimSuffix(v, build)
j := strings.LastIndex(v, "-")
v, rev = v[:j], v[j+1:]
i := strings.LastIndex(v, "-")
if j := strings.LastIndex(v, "."); j > i {
base = v[:j] // "vX.Y.Z-pre.0" or "vX.Y.(Z+1)-0"
timestamp = v[j+1:]
} else {
base = v[:i] // "vX.0.0"
timestamp = v[i+1:]
}
return base, timestamp, rev, build, nil
}

636
vendor/golang.org/x/tools/go/ast/astutil/enclosing.go generated vendored Normal file
View File

@@ -0,0 +1,636 @@
// Copyright 2013 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package astutil
// This file defines utilities for working with source positions.
import (
"fmt"
"go/ast"
"go/token"
"sort"
"golang.org/x/tools/internal/typeparams"
)
// PathEnclosingInterval returns the node that encloses the source
// interval [start, end), and all its ancestors up to the AST root.
//
// The definition of "enclosing" used by this function considers
// additional whitespace abutting a node to be enclosed by it.
// In this example:
//
// z := x + y // add them
// <-A->
// <----B----->
//
// the ast.BinaryExpr(+) node is considered to enclose interval B
// even though its [Pos()..End()) is actually only interval A.
// This behaviour makes user interfaces more tolerant of imperfect
// input.
//
// This function treats tokens as nodes, though they are not included
// in the result. e.g. PathEnclosingInterval("+") returns the
// enclosing ast.BinaryExpr("x + y").
//
// If start==end, the 1-char interval following start is used instead.
//
// The 'exact' result is true if the interval contains only path[0]
// and perhaps some adjacent whitespace. It is false if the interval
// overlaps multiple children of path[0], or if it contains only
// interior whitespace of path[0].
// In this example:
//
// z := x + y // add them
// <--C--> <---E-->
// ^
// D
//
// intervals C, D and E are inexact. C is contained by the
// z-assignment statement, because it spans three of its children (:=,
// x, +). So too is the 1-char interval D, because it contains only
// interior whitespace of the assignment. E is considered interior
// whitespace of the BlockStmt containing the assignment.
//
// The resulting path is never empty; it always contains at least the
// 'root' *ast.File. Ideally PathEnclosingInterval would reject
// intervals that lie wholly or partially outside the range of the
// file, but unfortunately ast.File records only the token.Pos of
// the 'package' keyword, but not of the start of the file itself.
func PathEnclosingInterval(root *ast.File, start, end token.Pos) (path []ast.Node, exact bool) {
// fmt.Printf("EnclosingInterval %d %d\n", start, end) // debugging
// Precondition: node.[Pos..End) and adjoining whitespace contain [start, end).
var visit func(node ast.Node) bool
visit = func(node ast.Node) bool {
path = append(path, node)
nodePos := node.Pos()
nodeEnd := node.End()
// fmt.Printf("visit(%T, %d, %d)\n", node, nodePos, nodeEnd) // debugging
// Intersect [start, end) with interval of node.
if start < nodePos {
start = nodePos
}
if end > nodeEnd {
end = nodeEnd
}
// Find sole child that contains [start, end).
children := childrenOf(node)
l := len(children)
for i, child := range children {
// [childPos, childEnd) is unaugmented interval of child.
childPos := child.Pos()
childEnd := child.End()
// [augPos, augEnd) is whitespace-augmented interval of child.
augPos := childPos
augEnd := childEnd
if i > 0 {
augPos = children[i-1].End() // start of preceding whitespace
}
if i < l-1 {
nextChildPos := children[i+1].Pos()
// Does [start, end) lie between child and next child?
if start >= augEnd && end <= nextChildPos {
return false // inexact match
}
augEnd = nextChildPos // end of following whitespace
}
// fmt.Printf("\tchild %d: [%d..%d)\tcontains interval [%d..%d)?\n",
// i, augPos, augEnd, start, end) // debugging
// Does augmented child strictly contain [start, end)?
if augPos <= start && end <= augEnd {
_, isToken := child.(tokenNode)
return isToken || visit(child)
}
// Does [start, end) overlap multiple children?
// i.e. left-augmented child contains start
// but LR-augmented child does not contain end.
if start < childEnd && end > augEnd {
break
}
}
// No single child contained [start, end),
// so node is the result. Is it exact?
// (It's tempting to put this condition before the
// child loop, but it gives the wrong result in the
// case where a node (e.g. ExprStmt) and its sole
// child have equal intervals.)
if start == nodePos && end == nodeEnd {
return true // exact match
}
return false // inexact: overlaps multiple children
}
// Ensure [start,end) is nondecreasing.
if start > end {
start, end = end, start
}
if start < root.End() && end > root.Pos() {
if start == end {
end = start + 1 // empty interval => interval of size 1
}
exact = visit(root)
// Reverse the path:
for i, l := 0, len(path); i < l/2; i++ {
path[i], path[l-1-i] = path[l-1-i], path[i]
}
} else {
// Selection lies within whitespace preceding the
// first (or following the last) declaration in the file.
// The result nonetheless always includes the ast.File.
path = append(path, root)
}
return
}
// tokenNode is a dummy implementation of ast.Node for a single token.
// They are used transiently by PathEnclosingInterval but never escape
// this package.
type tokenNode struct {
pos token.Pos
end token.Pos
}
func (n tokenNode) Pos() token.Pos {
return n.pos
}
func (n tokenNode) End() token.Pos {
return n.end
}
func tok(pos token.Pos, len int) ast.Node {
return tokenNode{pos, pos + token.Pos(len)}
}
// childrenOf returns the direct non-nil children of ast.Node n.
// It may include fake ast.Node implementations for bare tokens.
// it is not safe to call (e.g.) ast.Walk on such nodes.
func childrenOf(n ast.Node) []ast.Node {
var children []ast.Node
// First add nodes for all true subtrees.
ast.Inspect(n, func(node ast.Node) bool {
if node == n { // push n
return true // recur
}
if node != nil { // push child
children = append(children, node)
}
return false // no recursion
})
// Then add fake Nodes for bare tokens.
switch n := n.(type) {
case *ast.ArrayType:
children = append(children,
tok(n.Lbrack, len("[")),
tok(n.Elt.End(), len("]")))
case *ast.AssignStmt:
children = append(children,
tok(n.TokPos, len(n.Tok.String())))
case *ast.BasicLit:
children = append(children,
tok(n.ValuePos, len(n.Value)))
case *ast.BinaryExpr:
children = append(children, tok(n.OpPos, len(n.Op.String())))
case *ast.BlockStmt:
children = append(children,
tok(n.Lbrace, len("{")),
tok(n.Rbrace, len("}")))
case *ast.BranchStmt:
children = append(children,
tok(n.TokPos, len(n.Tok.String())))
case *ast.CallExpr:
children = append(children,
tok(n.Lparen, len("(")),
tok(n.Rparen, len(")")))
if n.Ellipsis != 0 {
children = append(children, tok(n.Ellipsis, len("...")))
}
case *ast.CaseClause:
if n.List == nil {
children = append(children,
tok(n.Case, len("default")))
} else {
children = append(children,
tok(n.Case, len("case")))
}
children = append(children, tok(n.Colon, len(":")))
case *ast.ChanType:
switch n.Dir {
case ast.RECV:
children = append(children, tok(n.Begin, len("<-chan")))
case ast.SEND:
children = append(children, tok(n.Begin, len("chan<-")))
case ast.RECV | ast.SEND:
children = append(children, tok(n.Begin, len("chan")))
}
case *ast.CommClause:
if n.Comm == nil {
children = append(children,
tok(n.Case, len("default")))
} else {
children = append(children,
tok(n.Case, len("case")))
}
children = append(children, tok(n.Colon, len(":")))
case *ast.Comment:
// nop
case *ast.CommentGroup:
// nop
case *ast.CompositeLit:
children = append(children,
tok(n.Lbrace, len("{")),
tok(n.Rbrace, len("{")))
case *ast.DeclStmt:
// nop
case *ast.DeferStmt:
children = append(children,
tok(n.Defer, len("defer")))
case *ast.Ellipsis:
children = append(children,
tok(n.Ellipsis, len("...")))
case *ast.EmptyStmt:
// nop
case *ast.ExprStmt:
// nop
case *ast.Field:
// TODO(adonovan): Field.{Doc,Comment,Tag}?
case *ast.FieldList:
children = append(children,
tok(n.Opening, len("(")), // or len("[")
tok(n.Closing, len(")"))) // or len("]")
case *ast.File:
// TODO test: Doc
children = append(children,
tok(n.Package, len("package")))
case *ast.ForStmt:
children = append(children,
tok(n.For, len("for")))
case *ast.FuncDecl:
// TODO(adonovan): FuncDecl.Comment?
// Uniquely, FuncDecl breaks the invariant that
// preorder traversal yields tokens in lexical order:
// in fact, FuncDecl.Recv precedes FuncDecl.Type.Func.
//
// As a workaround, we inline the case for FuncType
// here and order things correctly.
//
children = nil // discard ast.Walk(FuncDecl) info subtrees
children = append(children, tok(n.Type.Func, len("func")))
if n.Recv != nil {
children = append(children, n.Recv)
}
children = append(children, n.Name)
if tparams := typeparams.ForFuncType(n.Type); tparams != nil {
children = append(children, tparams)
}
if n.Type.Params != nil {
children = append(children, n.Type.Params)
}
if n.Type.Results != nil {
children = append(children, n.Type.Results)
}
if n.Body != nil {
children = append(children, n.Body)
}
case *ast.FuncLit:
// nop
case *ast.FuncType:
if n.Func != 0 {
children = append(children,
tok(n.Func, len("func")))
}
case *ast.GenDecl:
children = append(children,
tok(n.TokPos, len(n.Tok.String())))
if n.Lparen != 0 {
children = append(children,
tok(n.Lparen, len("(")),
tok(n.Rparen, len(")")))
}
case *ast.GoStmt:
children = append(children,
tok(n.Go, len("go")))
case *ast.Ident:
children = append(children,
tok(n.NamePos, len(n.Name)))
case *ast.IfStmt:
children = append(children,
tok(n.If, len("if")))
case *ast.ImportSpec:
// TODO(adonovan): ImportSpec.{Doc,EndPos}?
case *ast.IncDecStmt:
children = append(children,
tok(n.TokPos, len(n.Tok.String())))
case *ast.IndexExpr:
children = append(children,
tok(n.Lbrack, len("[")),
tok(n.Rbrack, len("]")))
case *typeparams.IndexListExpr:
children = append(children,
tok(n.Lbrack, len("[")),
tok(n.Rbrack, len("]")))
case *ast.InterfaceType:
children = append(children,
tok(n.Interface, len("interface")))
case *ast.KeyValueExpr:
children = append(children,
tok(n.Colon, len(":")))
case *ast.LabeledStmt:
children = append(children,
tok(n.Colon, len(":")))
case *ast.MapType:
children = append(children,
tok(n.Map, len("map")))
case *ast.ParenExpr:
children = append(children,
tok(n.Lparen, len("(")),
tok(n.Rparen, len(")")))
case *ast.RangeStmt:
children = append(children,
tok(n.For, len("for")),
tok(n.TokPos, len(n.Tok.String())))
case *ast.ReturnStmt:
children = append(children,
tok(n.Return, len("return")))
case *ast.SelectStmt:
children = append(children,
tok(n.Select, len("select")))
case *ast.SelectorExpr:
// nop
case *ast.SendStmt:
children = append(children,
tok(n.Arrow, len("<-")))
case *ast.SliceExpr:
children = append(children,
tok(n.Lbrack, len("[")),
tok(n.Rbrack, len("]")))
case *ast.StarExpr:
children = append(children, tok(n.Star, len("*")))
case *ast.StructType:
children = append(children, tok(n.Struct, len("struct")))
case *ast.SwitchStmt:
children = append(children, tok(n.Switch, len("switch")))
case *ast.TypeAssertExpr:
children = append(children,
tok(n.Lparen-1, len(".")),
tok(n.Lparen, len("(")),
tok(n.Rparen, len(")")))
case *ast.TypeSpec:
// TODO(adonovan): TypeSpec.{Doc,Comment}?
case *ast.TypeSwitchStmt:
children = append(children, tok(n.Switch, len("switch")))
case *ast.UnaryExpr:
children = append(children, tok(n.OpPos, len(n.Op.String())))
case *ast.ValueSpec:
// TODO(adonovan): ValueSpec.{Doc,Comment}?
case *ast.BadDecl, *ast.BadExpr, *ast.BadStmt:
// nop
}
// TODO(adonovan): opt: merge the logic of ast.Inspect() into
// the switch above so we can make interleaved callbacks for
// both Nodes and Tokens in the right order and avoid the need
// to sort.
sort.Sort(byPos(children))
return children
}
type byPos []ast.Node
func (sl byPos) Len() int {
return len(sl)
}
func (sl byPos) Less(i, j int) bool {
return sl[i].Pos() < sl[j].Pos()
}
func (sl byPos) Swap(i, j int) {
sl[i], sl[j] = sl[j], sl[i]
}
// NodeDescription returns a description of the concrete type of n suitable
// for a user interface.
//
// TODO(adonovan): in some cases (e.g. Field, FieldList, Ident,
// StarExpr) we could be much more specific given the path to the AST
// root. Perhaps we should do that.
func NodeDescription(n ast.Node) string {
switch n := n.(type) {
case *ast.ArrayType:
return "array type"
case *ast.AssignStmt:
return "assignment"
case *ast.BadDecl:
return "bad declaration"
case *ast.BadExpr:
return "bad expression"
case *ast.BadStmt:
return "bad statement"
case *ast.BasicLit:
return "basic literal"
case *ast.BinaryExpr:
return fmt.Sprintf("binary %s operation", n.Op)
case *ast.BlockStmt:
return "block"
case *ast.BranchStmt:
switch n.Tok {
case token.BREAK:
return "break statement"
case token.CONTINUE:
return "continue statement"
case token.GOTO:
return "goto statement"
case token.FALLTHROUGH:
return "fall-through statement"
}
case *ast.CallExpr:
if len(n.Args) == 1 && !n.Ellipsis.IsValid() {
return "function call (or conversion)"
}
return "function call"
case *ast.CaseClause:
return "case clause"
case *ast.ChanType:
return "channel type"
case *ast.CommClause:
return "communication clause"
case *ast.Comment:
return "comment"
case *ast.CommentGroup:
return "comment group"
case *ast.CompositeLit:
return "composite literal"
case *ast.DeclStmt:
return NodeDescription(n.Decl) + " statement"
case *ast.DeferStmt:
return "defer statement"
case *ast.Ellipsis:
return "ellipsis"
case *ast.EmptyStmt:
return "empty statement"
case *ast.ExprStmt:
return "expression statement"
case *ast.Field:
// Can be any of these:
// struct {x, y int} -- struct field(s)
// struct {T} -- anon struct field
// interface {I} -- interface embedding
// interface {f()} -- interface method
// func (A) func(B) C -- receiver, param(s), result(s)
return "field/method/parameter"
case *ast.FieldList:
return "field/method/parameter list"
case *ast.File:
return "source file"
case *ast.ForStmt:
return "for loop"
case *ast.FuncDecl:
return "function declaration"
case *ast.FuncLit:
return "function literal"
case *ast.FuncType:
return "function type"
case *ast.GenDecl:
switch n.Tok {
case token.IMPORT:
return "import declaration"
case token.CONST:
return "constant declaration"
case token.TYPE:
return "type declaration"
case token.VAR:
return "variable declaration"
}
case *ast.GoStmt:
return "go statement"
case *ast.Ident:
return "identifier"
case *ast.IfStmt:
return "if statement"
case *ast.ImportSpec:
return "import specification"
case *ast.IncDecStmt:
if n.Tok == token.INC {
return "increment statement"
}
return "decrement statement"
case *ast.IndexExpr:
return "index expression"
case *typeparams.IndexListExpr:
return "index list expression"
case *ast.InterfaceType:
return "interface type"
case *ast.KeyValueExpr:
return "key/value association"
case *ast.LabeledStmt:
return "statement label"
case *ast.MapType:
return "map type"
case *ast.Package:
return "package"
case *ast.ParenExpr:
return "parenthesized " + NodeDescription(n.X)
case *ast.RangeStmt:
return "range loop"
case *ast.ReturnStmt:
return "return statement"
case *ast.SelectStmt:
return "select statement"
case *ast.SelectorExpr:
return "selector"
case *ast.SendStmt:
return "channel send"
case *ast.SliceExpr:
return "slice expression"
case *ast.StarExpr:
return "*-operation" // load/store expr or pointer type
case *ast.StructType:
return "struct type"
case *ast.SwitchStmt:
return "switch statement"
case *ast.TypeAssertExpr:
return "type assertion"
case *ast.TypeSpec:
return "type specification"
case *ast.TypeSwitchStmt:
return "type switch"
case *ast.UnaryExpr:
return fmt.Sprintf("unary %s operation", n.Op)
case *ast.ValueSpec:
return "value specification"
}
panic(fmt.Sprintf("unexpected node type: %T", n))
}

485
vendor/golang.org/x/tools/go/ast/astutil/imports.go generated vendored Normal file
View File

@@ -0,0 +1,485 @@
// Copyright 2013 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package astutil contains common utilities for working with the Go AST.
package astutil // import "golang.org/x/tools/go/ast/astutil"
import (
"fmt"
"go/ast"
"go/token"
"strconv"
"strings"
)
// AddImport adds the import path to the file f, if absent.
func AddImport(fset *token.FileSet, f *ast.File, path string) (added bool) {
return AddNamedImport(fset, f, "", path)
}
// AddNamedImport adds the import with the given name and path to the file f, if absent.
// If name is not empty, it is used to rename the import.
//
// For example, calling
//
// AddNamedImport(fset, f, "pathpkg", "path")
//
// adds
//
// import pathpkg "path"
func AddNamedImport(fset *token.FileSet, f *ast.File, name, path string) (added bool) {
if imports(f, name, path) {
return false
}
newImport := &ast.ImportSpec{
Path: &ast.BasicLit{
Kind: token.STRING,
Value: strconv.Quote(path),
},
}
if name != "" {
newImport.Name = &ast.Ident{Name: name}
}
// Find an import decl to add to.
// The goal is to find an existing import
// whose import path has the longest shared
// prefix with path.
var (
bestMatch = -1 // length of longest shared prefix
lastImport = -1 // index in f.Decls of the file's final import decl
impDecl *ast.GenDecl // import decl containing the best match
impIndex = -1 // spec index in impDecl containing the best match
isThirdPartyPath = isThirdParty(path)
)
for i, decl := range f.Decls {
gen, ok := decl.(*ast.GenDecl)
if ok && gen.Tok == token.IMPORT {
lastImport = i
// Do not add to import "C", to avoid disrupting the
// association with its doc comment, breaking cgo.
if declImports(gen, "C") {
continue
}
// Match an empty import decl if that's all that is available.
if len(gen.Specs) == 0 && bestMatch == -1 {
impDecl = gen
}
// Compute longest shared prefix with imports in this group and find best
// matched import spec.
// 1. Always prefer import spec with longest shared prefix.
// 2. While match length is 0,
// - for stdlib package: prefer first import spec.
// - for third party package: prefer first third party import spec.
// We cannot use last import spec as best match for third party package
// because grouped imports are usually placed last by goimports -local
// flag.
// See issue #19190.
seenAnyThirdParty := false
for j, spec := range gen.Specs {
impspec := spec.(*ast.ImportSpec)
p := importPath(impspec)
n := matchLen(p, path)
if n > bestMatch || (bestMatch == 0 && !seenAnyThirdParty && isThirdPartyPath) {
bestMatch = n
impDecl = gen
impIndex = j
}
seenAnyThirdParty = seenAnyThirdParty || isThirdParty(p)
}
}
}
// If no import decl found, add one after the last import.
if impDecl == nil {
impDecl = &ast.GenDecl{
Tok: token.IMPORT,
}
if lastImport >= 0 {
impDecl.TokPos = f.Decls[lastImport].End()
} else {
// There are no existing imports.
// Our new import, preceded by a blank line, goes after the package declaration
// and after the comment, if any, that starts on the same line as the
// package declaration.
impDecl.TokPos = f.Package
file := fset.File(f.Package)
pkgLine := file.Line(f.Package)
for _, c := range f.Comments {
if file.Line(c.Pos()) > pkgLine {
break
}
// +2 for a blank line
impDecl.TokPos = c.End() + 2
}
}
f.Decls = append(f.Decls, nil)
copy(f.Decls[lastImport+2:], f.Decls[lastImport+1:])
f.Decls[lastImport+1] = impDecl
}
// Insert new import at insertAt.
insertAt := 0
if impIndex >= 0 {
// insert after the found import
insertAt = impIndex + 1
}
impDecl.Specs = append(impDecl.Specs, nil)
copy(impDecl.Specs[insertAt+1:], impDecl.Specs[insertAt:])
impDecl.Specs[insertAt] = newImport
pos := impDecl.Pos()
if insertAt > 0 {
// If there is a comment after an existing import, preserve the comment
// position by adding the new import after the comment.
if spec, ok := impDecl.Specs[insertAt-1].(*ast.ImportSpec); ok && spec.Comment != nil {
pos = spec.Comment.End()
} else {
// Assign same position as the previous import,
// so that the sorter sees it as being in the same block.
pos = impDecl.Specs[insertAt-1].Pos()
}
}
if newImport.Name != nil {
newImport.Name.NamePos = pos
}
newImport.Path.ValuePos = pos
newImport.EndPos = pos
// Clean up parens. impDecl contains at least one spec.
if len(impDecl.Specs) == 1 {
// Remove unneeded parens.
impDecl.Lparen = token.NoPos
} else if !impDecl.Lparen.IsValid() {
// impDecl needs parens added.
impDecl.Lparen = impDecl.Specs[0].Pos()
}
f.Imports = append(f.Imports, newImport)
if len(f.Decls) <= 1 {
return true
}
// Merge all the import declarations into the first one.
var first *ast.GenDecl
for i := 0; i < len(f.Decls); i++ {
decl := f.Decls[i]
gen, ok := decl.(*ast.GenDecl)
if !ok || gen.Tok != token.IMPORT || declImports(gen, "C") {
continue
}
if first == nil {
first = gen
continue // Don't touch the first one.
}
// We now know there is more than one package in this import
// declaration. Ensure that it ends up parenthesized.
first.Lparen = first.Pos()
// Move the imports of the other import declaration to the first one.
for _, spec := range gen.Specs {
spec.(*ast.ImportSpec).Path.ValuePos = first.Pos()
first.Specs = append(first.Specs, spec)
}
f.Decls = append(f.Decls[:i], f.Decls[i+1:]...)
i--
}
return true
}
func isThirdParty(importPath string) bool {
// Third party package import path usually contains "." (".com", ".org", ...)
// This logic is taken from golang.org/x/tools/imports package.
return strings.Contains(importPath, ".")
}
// DeleteImport deletes the import path from the file f, if present.
// If there are duplicate import declarations, all matching ones are deleted.
func DeleteImport(fset *token.FileSet, f *ast.File, path string) (deleted bool) {
return DeleteNamedImport(fset, f, "", path)
}
// DeleteNamedImport deletes the import with the given name and path from the file f, if present.
// If there are duplicate import declarations, all matching ones are deleted.
func DeleteNamedImport(fset *token.FileSet, f *ast.File, name, path string) (deleted bool) {
var delspecs []*ast.ImportSpec
var delcomments []*ast.CommentGroup
// Find the import nodes that import path, if any.
for i := 0; i < len(f.Decls); i++ {
decl := f.Decls[i]
gen, ok := decl.(*ast.GenDecl)
if !ok || gen.Tok != token.IMPORT {
continue
}
for j := 0; j < len(gen.Specs); j++ {
spec := gen.Specs[j]
impspec := spec.(*ast.ImportSpec)
if importName(impspec) != name || importPath(impspec) != path {
continue
}
// We found an import spec that imports path.
// Delete it.
delspecs = append(delspecs, impspec)
deleted = true
copy(gen.Specs[j:], gen.Specs[j+1:])
gen.Specs = gen.Specs[:len(gen.Specs)-1]
// If this was the last import spec in this decl,
// delete the decl, too.
if len(gen.Specs) == 0 {
copy(f.Decls[i:], f.Decls[i+1:])
f.Decls = f.Decls[:len(f.Decls)-1]
i--
break
} else if len(gen.Specs) == 1 {
if impspec.Doc != nil {
delcomments = append(delcomments, impspec.Doc)
}
if impspec.Comment != nil {
delcomments = append(delcomments, impspec.Comment)
}
for _, cg := range f.Comments {
// Found comment on the same line as the import spec.
if cg.End() < impspec.Pos() && fset.Position(cg.End()).Line == fset.Position(impspec.Pos()).Line {
delcomments = append(delcomments, cg)
break
}
}
spec := gen.Specs[0].(*ast.ImportSpec)
// Move the documentation right after the import decl.
if spec.Doc != nil {
for fset.Position(gen.TokPos).Line+1 < fset.Position(spec.Doc.Pos()).Line {
fset.File(gen.TokPos).MergeLine(fset.Position(gen.TokPos).Line)
}
}
for _, cg := range f.Comments {
if cg.End() < spec.Pos() && fset.Position(cg.End()).Line == fset.Position(spec.Pos()).Line {
for fset.Position(gen.TokPos).Line+1 < fset.Position(spec.Pos()).Line {
fset.File(gen.TokPos).MergeLine(fset.Position(gen.TokPos).Line)
}
break
}
}
}
if j > 0 {
lastImpspec := gen.Specs[j-1].(*ast.ImportSpec)
lastLine := fset.PositionFor(lastImpspec.Path.ValuePos, false).Line
line := fset.PositionFor(impspec.Path.ValuePos, false).Line
// We deleted an entry but now there may be
// a blank line-sized hole where the import was.
if line-lastLine > 1 || !gen.Rparen.IsValid() {
// There was a blank line immediately preceding the deleted import,
// so there's no need to close the hole. The right parenthesis is
// invalid after AddImport to an import statement without parenthesis.
// Do nothing.
} else if line != fset.File(gen.Rparen).LineCount() {
// There was no blank line. Close the hole.
fset.File(gen.Rparen).MergeLine(line)
}
}
j--
}
}
// Delete imports from f.Imports.
for i := 0; i < len(f.Imports); i++ {
imp := f.Imports[i]
for j, del := range delspecs {
if imp == del {
copy(f.Imports[i:], f.Imports[i+1:])
f.Imports = f.Imports[:len(f.Imports)-1]
copy(delspecs[j:], delspecs[j+1:])
delspecs = delspecs[:len(delspecs)-1]
i--
break
}
}
}
// Delete comments from f.Comments.
for i := 0; i < len(f.Comments); i++ {
cg := f.Comments[i]
for j, del := range delcomments {
if cg == del {
copy(f.Comments[i:], f.Comments[i+1:])
f.Comments = f.Comments[:len(f.Comments)-1]
copy(delcomments[j:], delcomments[j+1:])
delcomments = delcomments[:len(delcomments)-1]
i--
break
}
}
}
if len(delspecs) > 0 {
panic(fmt.Sprintf("deleted specs from Decls but not Imports: %v", delspecs))
}
return
}
// RewriteImport rewrites any import of path oldPath to path newPath.
func RewriteImport(fset *token.FileSet, f *ast.File, oldPath, newPath string) (rewrote bool) {
for _, imp := range f.Imports {
if importPath(imp) == oldPath {
rewrote = true
// record old End, because the default is to compute
// it using the length of imp.Path.Value.
imp.EndPos = imp.End()
imp.Path.Value = strconv.Quote(newPath)
}
}
return
}
// UsesImport reports whether a given import is used.
func UsesImport(f *ast.File, path string) (used bool) {
spec := importSpec(f, path)
if spec == nil {
return
}
name := spec.Name.String()
switch name {
case "<nil>":
// If the package name is not explicitly specified,
// make an educated guess. This is not guaranteed to be correct.
lastSlash := strings.LastIndex(path, "/")
if lastSlash == -1 {
name = path
} else {
name = path[lastSlash+1:]
}
case "_", ".":
// Not sure if this import is used - err on the side of caution.
return true
}
ast.Walk(visitFn(func(n ast.Node) {
sel, ok := n.(*ast.SelectorExpr)
if ok && isTopName(sel.X, name) {
used = true
}
}), f)
return
}
type visitFn func(node ast.Node)
func (fn visitFn) Visit(node ast.Node) ast.Visitor {
fn(node)
return fn
}
// imports reports whether f has an import with the specified name and path.
func imports(f *ast.File, name, path string) bool {
for _, s := range f.Imports {
if importName(s) == name && importPath(s) == path {
return true
}
}
return false
}
// importSpec returns the import spec if f imports path,
// or nil otherwise.
func importSpec(f *ast.File, path string) *ast.ImportSpec {
for _, s := range f.Imports {
if importPath(s) == path {
return s
}
}
return nil
}
// importName returns the name of s,
// or "" if the import is not named.
func importName(s *ast.ImportSpec) string {
if s.Name == nil {
return ""
}
return s.Name.Name
}
// importPath returns the unquoted import path of s,
// or "" if the path is not properly quoted.
func importPath(s *ast.ImportSpec) string {
t, err := strconv.Unquote(s.Path.Value)
if err != nil {
return ""
}
return t
}
// declImports reports whether gen contains an import of path.
func declImports(gen *ast.GenDecl, path string) bool {
if gen.Tok != token.IMPORT {
return false
}
for _, spec := range gen.Specs {
impspec := spec.(*ast.ImportSpec)
if importPath(impspec) == path {
return true
}
}
return false
}
// matchLen returns the length of the longest path segment prefix shared by x and y.
func matchLen(x, y string) int {
n := 0
for i := 0; i < len(x) && i < len(y) && x[i] == y[i]; i++ {
if x[i] == '/' {
n++
}
}
return n
}
// isTopName returns true if n is a top-level unresolved identifier with the given name.
func isTopName(n ast.Expr, name string) bool {
id, ok := n.(*ast.Ident)
return ok && id.Name == name && id.Obj == nil
}
// Imports returns the file imports grouped by paragraph.
func Imports(fset *token.FileSet, f *ast.File) [][]*ast.ImportSpec {
var groups [][]*ast.ImportSpec
for _, decl := range f.Decls {
genDecl, ok := decl.(*ast.GenDecl)
if !ok || genDecl.Tok != token.IMPORT {
break
}
group := []*ast.ImportSpec{}
var lastLine int
for _, spec := range genDecl.Specs {
importSpec := spec.(*ast.ImportSpec)
pos := importSpec.Path.ValuePos
line := fset.Position(pos).Line
if lastLine > 0 && pos > 0 && line-lastLine > 1 {
groups = append(groups, group)
group = []*ast.ImportSpec{}
}
group = append(group, importSpec)
lastLine = line
}
groups = append(groups, group)
}
return groups
}

488
vendor/golang.org/x/tools/go/ast/astutil/rewrite.go generated vendored Normal file
View File

@@ -0,0 +1,488 @@
// Copyright 2017 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package astutil
import (
"fmt"
"go/ast"
"reflect"
"sort"
"golang.org/x/tools/internal/typeparams"
)
// An ApplyFunc is invoked by Apply for each node n, even if n is nil,
// before and/or after the node's children, using a Cursor describing
// the current node and providing operations on it.
//
// The return value of ApplyFunc controls the syntax tree traversal.
// See Apply for details.
type ApplyFunc func(*Cursor) bool
// Apply traverses a syntax tree recursively, starting with root,
// and calling pre and post for each node as described below.
// Apply returns the syntax tree, possibly modified.
//
// If pre is not nil, it is called for each node before the node's
// children are traversed (pre-order). If pre returns false, no
// children are traversed, and post is not called for that node.
//
// If post is not nil, and a prior call of pre didn't return false,
// post is called for each node after its children are traversed
// (post-order). If post returns false, traversal is terminated and
// Apply returns immediately.
//
// Only fields that refer to AST nodes are considered children;
// i.e., token.Pos, Scopes, Objects, and fields of basic types
// (strings, etc.) are ignored.
//
// Children are traversed in the order in which they appear in the
// respective node's struct definition. A package's files are
// traversed in the filenames' alphabetical order.
func Apply(root ast.Node, pre, post ApplyFunc) (result ast.Node) {
parent := &struct{ ast.Node }{root}
defer func() {
if r := recover(); r != nil && r != abort {
panic(r)
}
result = parent.Node
}()
a := &application{pre: pre, post: post}
a.apply(parent, "Node", nil, root)
return
}
var abort = new(int) // singleton, to signal termination of Apply
// A Cursor describes a node encountered during Apply.
// Information about the node and its parent is available
// from the Node, Parent, Name, and Index methods.
//
// If p is a variable of type and value of the current parent node
// c.Parent(), and f is the field identifier with name c.Name(),
// the following invariants hold:
//
// p.f == c.Node() if c.Index() < 0
// p.f[c.Index()] == c.Node() if c.Index() >= 0
//
// The methods Replace, Delete, InsertBefore, and InsertAfter
// can be used to change the AST without disrupting Apply.
type Cursor struct {
parent ast.Node
name string
iter *iterator // valid if non-nil
node ast.Node
}
// Node returns the current Node.
func (c *Cursor) Node() ast.Node { return c.node }
// Parent returns the parent of the current Node.
func (c *Cursor) Parent() ast.Node { return c.parent }
// Name returns the name of the parent Node field that contains the current Node.
// If the parent is a *ast.Package and the current Node is a *ast.File, Name returns
// the filename for the current Node.
func (c *Cursor) Name() string { return c.name }
// Index reports the index >= 0 of the current Node in the slice of Nodes that
// contains it, or a value < 0 if the current Node is not part of a slice.
// The index of the current node changes if InsertBefore is called while
// processing the current node.
func (c *Cursor) Index() int {
if c.iter != nil {
return c.iter.index
}
return -1
}
// field returns the current node's parent field value.
func (c *Cursor) field() reflect.Value {
return reflect.Indirect(reflect.ValueOf(c.parent)).FieldByName(c.name)
}
// Replace replaces the current Node with n.
// The replacement node is not walked by Apply.
func (c *Cursor) Replace(n ast.Node) {
if _, ok := c.node.(*ast.File); ok {
file, ok := n.(*ast.File)
if !ok {
panic("attempt to replace *ast.File with non-*ast.File")
}
c.parent.(*ast.Package).Files[c.name] = file
return
}
v := c.field()
if i := c.Index(); i >= 0 {
v = v.Index(i)
}
v.Set(reflect.ValueOf(n))
}
// Delete deletes the current Node from its containing slice.
// If the current Node is not part of a slice, Delete panics.
// As a special case, if the current node is a package file,
// Delete removes it from the package's Files map.
func (c *Cursor) Delete() {
if _, ok := c.node.(*ast.File); ok {
delete(c.parent.(*ast.Package).Files, c.name)
return
}
i := c.Index()
if i < 0 {
panic("Delete node not contained in slice")
}
v := c.field()
l := v.Len()
reflect.Copy(v.Slice(i, l), v.Slice(i+1, l))
v.Index(l - 1).Set(reflect.Zero(v.Type().Elem()))
v.SetLen(l - 1)
c.iter.step--
}
// InsertAfter inserts n after the current Node in its containing slice.
// If the current Node is not part of a slice, InsertAfter panics.
// Apply does not walk n.
func (c *Cursor) InsertAfter(n ast.Node) {
i := c.Index()
if i < 0 {
panic("InsertAfter node not contained in slice")
}
v := c.field()
v.Set(reflect.Append(v, reflect.Zero(v.Type().Elem())))
l := v.Len()
reflect.Copy(v.Slice(i+2, l), v.Slice(i+1, l))
v.Index(i + 1).Set(reflect.ValueOf(n))
c.iter.step++
}
// InsertBefore inserts n before the current Node in its containing slice.
// If the current Node is not part of a slice, InsertBefore panics.
// Apply will not walk n.
func (c *Cursor) InsertBefore(n ast.Node) {
i := c.Index()
if i < 0 {
panic("InsertBefore node not contained in slice")
}
v := c.field()
v.Set(reflect.Append(v, reflect.Zero(v.Type().Elem())))
l := v.Len()
reflect.Copy(v.Slice(i+1, l), v.Slice(i, l))
v.Index(i).Set(reflect.ValueOf(n))
c.iter.index++
}
// application carries all the shared data so we can pass it around cheaply.
type application struct {
pre, post ApplyFunc
cursor Cursor
iter iterator
}
func (a *application) apply(parent ast.Node, name string, iter *iterator, n ast.Node) {
// convert typed nil into untyped nil
if v := reflect.ValueOf(n); v.Kind() == reflect.Ptr && v.IsNil() {
n = nil
}
// avoid heap-allocating a new cursor for each apply call; reuse a.cursor instead
saved := a.cursor
a.cursor.parent = parent
a.cursor.name = name
a.cursor.iter = iter
a.cursor.node = n
if a.pre != nil && !a.pre(&a.cursor) {
a.cursor = saved
return
}
// walk children
// (the order of the cases matches the order of the corresponding node types in go/ast)
switch n := n.(type) {
case nil:
// nothing to do
// Comments and fields
case *ast.Comment:
// nothing to do
case *ast.CommentGroup:
if n != nil {
a.applyList(n, "List")
}
case *ast.Field:
a.apply(n, "Doc", nil, n.Doc)
a.applyList(n, "Names")
a.apply(n, "Type", nil, n.Type)
a.apply(n, "Tag", nil, n.Tag)
a.apply(n, "Comment", nil, n.Comment)
case *ast.FieldList:
a.applyList(n, "List")
// Expressions
case *ast.BadExpr, *ast.Ident, *ast.BasicLit:
// nothing to do
case *ast.Ellipsis:
a.apply(n, "Elt", nil, n.Elt)
case *ast.FuncLit:
a.apply(n, "Type", nil, n.Type)
a.apply(n, "Body", nil, n.Body)
case *ast.CompositeLit:
a.apply(n, "Type", nil, n.Type)
a.applyList(n, "Elts")
case *ast.ParenExpr:
a.apply(n, "X", nil, n.X)
case *ast.SelectorExpr:
a.apply(n, "X", nil, n.X)
a.apply(n, "Sel", nil, n.Sel)
case *ast.IndexExpr:
a.apply(n, "X", nil, n.X)
a.apply(n, "Index", nil, n.Index)
case *typeparams.IndexListExpr:
a.apply(n, "X", nil, n.X)
a.applyList(n, "Indices")
case *ast.SliceExpr:
a.apply(n, "X", nil, n.X)
a.apply(n, "Low", nil, n.Low)
a.apply(n, "High", nil, n.High)
a.apply(n, "Max", nil, n.Max)
case *ast.TypeAssertExpr:
a.apply(n, "X", nil, n.X)
a.apply(n, "Type", nil, n.Type)
case *ast.CallExpr:
a.apply(n, "Fun", nil, n.Fun)
a.applyList(n, "Args")
case *ast.StarExpr:
a.apply(n, "X", nil, n.X)
case *ast.UnaryExpr:
a.apply(n, "X", nil, n.X)
case *ast.BinaryExpr:
a.apply(n, "X", nil, n.X)
a.apply(n, "Y", nil, n.Y)
case *ast.KeyValueExpr:
a.apply(n, "Key", nil, n.Key)
a.apply(n, "Value", nil, n.Value)
// Types
case *ast.ArrayType:
a.apply(n, "Len", nil, n.Len)
a.apply(n, "Elt", nil, n.Elt)
case *ast.StructType:
a.apply(n, "Fields", nil, n.Fields)
case *ast.FuncType:
if tparams := typeparams.ForFuncType(n); tparams != nil {
a.apply(n, "TypeParams", nil, tparams)
}
a.apply(n, "Params", nil, n.Params)
a.apply(n, "Results", nil, n.Results)
case *ast.InterfaceType:
a.apply(n, "Methods", nil, n.Methods)
case *ast.MapType:
a.apply(n, "Key", nil, n.Key)
a.apply(n, "Value", nil, n.Value)
case *ast.ChanType:
a.apply(n, "Value", nil, n.Value)
// Statements
case *ast.BadStmt:
// nothing to do
case *ast.DeclStmt:
a.apply(n, "Decl", nil, n.Decl)
case *ast.EmptyStmt:
// nothing to do
case *ast.LabeledStmt:
a.apply(n, "Label", nil, n.Label)
a.apply(n, "Stmt", nil, n.Stmt)
case *ast.ExprStmt:
a.apply(n, "X", nil, n.X)
case *ast.SendStmt:
a.apply(n, "Chan", nil, n.Chan)
a.apply(n, "Value", nil, n.Value)
case *ast.IncDecStmt:
a.apply(n, "X", nil, n.X)
case *ast.AssignStmt:
a.applyList(n, "Lhs")
a.applyList(n, "Rhs")
case *ast.GoStmt:
a.apply(n, "Call", nil, n.Call)
case *ast.DeferStmt:
a.apply(n, "Call", nil, n.Call)
case *ast.ReturnStmt:
a.applyList(n, "Results")
case *ast.BranchStmt:
a.apply(n, "Label", nil, n.Label)
case *ast.BlockStmt:
a.applyList(n, "List")
case *ast.IfStmt:
a.apply(n, "Init", nil, n.Init)
a.apply(n, "Cond", nil, n.Cond)
a.apply(n, "Body", nil, n.Body)
a.apply(n, "Else", nil, n.Else)
case *ast.CaseClause:
a.applyList(n, "List")
a.applyList(n, "Body")
case *ast.SwitchStmt:
a.apply(n, "Init", nil, n.Init)
a.apply(n, "Tag", nil, n.Tag)
a.apply(n, "Body", nil, n.Body)
case *ast.TypeSwitchStmt:
a.apply(n, "Init", nil, n.Init)
a.apply(n, "Assign", nil, n.Assign)
a.apply(n, "Body", nil, n.Body)
case *ast.CommClause:
a.apply(n, "Comm", nil, n.Comm)
a.applyList(n, "Body")
case *ast.SelectStmt:
a.apply(n, "Body", nil, n.Body)
case *ast.ForStmt:
a.apply(n, "Init", nil, n.Init)
a.apply(n, "Cond", nil, n.Cond)
a.apply(n, "Post", nil, n.Post)
a.apply(n, "Body", nil, n.Body)
case *ast.RangeStmt:
a.apply(n, "Key", nil, n.Key)
a.apply(n, "Value", nil, n.Value)
a.apply(n, "X", nil, n.X)
a.apply(n, "Body", nil, n.Body)
// Declarations
case *ast.ImportSpec:
a.apply(n, "Doc", nil, n.Doc)
a.apply(n, "Name", nil, n.Name)
a.apply(n, "Path", nil, n.Path)
a.apply(n, "Comment", nil, n.Comment)
case *ast.ValueSpec:
a.apply(n, "Doc", nil, n.Doc)
a.applyList(n, "Names")
a.apply(n, "Type", nil, n.Type)
a.applyList(n, "Values")
a.apply(n, "Comment", nil, n.Comment)
case *ast.TypeSpec:
a.apply(n, "Doc", nil, n.Doc)
a.apply(n, "Name", nil, n.Name)
if tparams := typeparams.ForTypeSpec(n); tparams != nil {
a.apply(n, "TypeParams", nil, tparams)
}
a.apply(n, "Type", nil, n.Type)
a.apply(n, "Comment", nil, n.Comment)
case *ast.BadDecl:
// nothing to do
case *ast.GenDecl:
a.apply(n, "Doc", nil, n.Doc)
a.applyList(n, "Specs")
case *ast.FuncDecl:
a.apply(n, "Doc", nil, n.Doc)
a.apply(n, "Recv", nil, n.Recv)
a.apply(n, "Name", nil, n.Name)
a.apply(n, "Type", nil, n.Type)
a.apply(n, "Body", nil, n.Body)
// Files and packages
case *ast.File:
a.apply(n, "Doc", nil, n.Doc)
a.apply(n, "Name", nil, n.Name)
a.applyList(n, "Decls")
// Don't walk n.Comments; they have either been walked already if
// they are Doc comments, or they can be easily walked explicitly.
case *ast.Package:
// collect and sort names for reproducible behavior
var names []string
for name := range n.Files {
names = append(names, name)
}
sort.Strings(names)
for _, name := range names {
a.apply(n, name, nil, n.Files[name])
}
default:
panic(fmt.Sprintf("Apply: unexpected node type %T", n))
}
if a.post != nil && !a.post(&a.cursor) {
panic(abort)
}
a.cursor = saved
}
// An iterator controls iteration over a slice of nodes.
type iterator struct {
index, step int
}
func (a *application) applyList(parent ast.Node, name string) {
// avoid heap-allocating a new iterator for each applyList call; reuse a.iter instead
saved := a.iter
a.iter.index = 0
for {
// must reload parent.name each time, since cursor modifications might change it
v := reflect.Indirect(reflect.ValueOf(parent)).FieldByName(name)
if a.iter.index >= v.Len() {
break
}
// element x may be nil in a bad AST - be cautious
var x ast.Node
if e := v.Index(a.iter.index); e.IsValid() {
x = e.Interface().(ast.Node)
}
a.iter.step = 1
a.apply(parent, name, &a.iter, x)
a.iter.index += a.iter.step
}
a.iter = saved
}

18
vendor/golang.org/x/tools/go/ast/astutil/util.go generated vendored Normal file
View File

@@ -0,0 +1,18 @@
// Copyright 2015 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package astutil
import "go/ast"
// Unparen returns e with any enclosing parentheses stripped.
func Unparen(e ast.Expr) ast.Expr {
for {
p, ok := e.(*ast.ParenExpr)
if !ok {
return e
}
e = p.X
}
}

77
vendor/golang.org/x/tools/imports/forward.go generated vendored Normal file
View File

@@ -0,0 +1,77 @@
// Copyright 2019 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package imports implements a Go pretty-printer (like package "go/format")
// that also adds or removes import statements as necessary.
package imports // import "golang.org/x/tools/imports"
import (
"io/ioutil"
"log"
"golang.org/x/tools/internal/gocommand"
intimp "golang.org/x/tools/internal/imports"
)
// Options specifies options for processing files.
type Options struct {
Fragment bool // Accept fragment of a source file (no package statement)
AllErrors bool // Report all errors (not just the first 10 on different lines)
Comments bool // Print comments (true if nil *Options provided)
TabIndent bool // Use tabs for indent (true if nil *Options provided)
TabWidth int // Tab width (8 if nil *Options provided)
FormatOnly bool // Disable the insertion and deletion of imports
}
// Debug controls verbose logging.
var Debug = false
// LocalPrefix is a comma-separated string of import path prefixes, which, if
// set, instructs Process to sort the import paths with the given prefixes
// into another group after 3rd-party packages.
var LocalPrefix string
// Process formats and adjusts imports for the provided file.
// If opt is nil the defaults are used, and if src is nil the source
// is read from the filesystem.
//
// Note that filename's directory influences which imports can be chosen,
// so it is important that filename be accurate.
// To process data “as if” it were in filename, pass the data as a non-nil src.
func Process(filename string, src []byte, opt *Options) ([]byte, error) {
var err error
if src == nil {
src, err = ioutil.ReadFile(filename)
if err != nil {
return nil, err
}
}
if opt == nil {
opt = &Options{Comments: true, TabIndent: true, TabWidth: 8}
}
intopt := &intimp.Options{
Env: &intimp.ProcessEnv{
GocmdRunner: &gocommand.Runner{},
},
LocalPrefix: LocalPrefix,
AllErrors: opt.AllErrors,
Comments: opt.Comments,
FormatOnly: opt.FormatOnly,
Fragment: opt.Fragment,
TabIndent: opt.TabIndent,
TabWidth: opt.TabWidth,
}
if Debug {
intopt.Env.Logf = log.Printf
}
return intimp.Process(filename, src, intopt)
}
// VendorlessPath returns the devendorized version of the import path ipath.
// For example, VendorlessPath("foo/bar/vendor/a/b") returns "a/b".
func VendorlessPath(ipath string) string {
return intimp.VendorlessPath(ipath)
}

196
vendor/golang.org/x/tools/internal/fastwalk/fastwalk.go generated vendored Normal file
View File

@@ -0,0 +1,196 @@
// Copyright 2016 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package fastwalk provides a faster version of [filepath.Walk] for file system
// scanning tools.
package fastwalk
import (
"errors"
"os"
"path/filepath"
"runtime"
"sync"
)
// ErrTraverseLink is used as a return value from WalkFuncs to indicate that the
// symlink named in the call may be traversed.
var ErrTraverseLink = errors.New("fastwalk: traverse symlink, assuming target is a directory")
// ErrSkipFiles is a used as a return value from WalkFuncs to indicate that the
// callback should not be called for any other files in the current directory.
// Child directories will still be traversed.
var ErrSkipFiles = errors.New("fastwalk: skip remaining files in directory")
// Walk is a faster implementation of [filepath.Walk].
//
// [filepath.Walk]'s design necessarily calls [os.Lstat] on each file,
// even if the caller needs less info.
// Many tools need only the type of each file.
// On some platforms, this information is provided directly by the readdir
// system call, avoiding the need to stat each file individually.
// fastwalk_unix.go contains a fork of the syscall routines.
//
// See golang.org/issue/16399.
//
// Walk walks the file tree rooted at root, calling walkFn for
// each file or directory in the tree, including root.
//
// If Walk returns [filepath.SkipDir], the directory is skipped.
//
// Unlike [filepath.Walk]:
// - file stat calls must be done by the user.
// The only provided metadata is the file type, which does not include
// any permission bits.
// - multiple goroutines stat the filesystem concurrently. The provided
// walkFn must be safe for concurrent use.
// - Walk can follow symlinks if walkFn returns the TraverseLink
// sentinel error. It is the walkFn's responsibility to prevent
// Walk from going into symlink cycles.
func Walk(root string, walkFn func(path string, typ os.FileMode) error) error {
// TODO(bradfitz): make numWorkers configurable? We used a
// minimum of 4 to give the kernel more info about multiple
// things we want, in hopes its I/O scheduling can take
// advantage of that. Hopefully most are in cache. Maybe 4 is
// even too low of a minimum. Profile more.
numWorkers := 4
if n := runtime.NumCPU(); n > numWorkers {
numWorkers = n
}
// Make sure to wait for all workers to finish, otherwise
// walkFn could still be called after returning. This Wait call
// runs after close(e.donec) below.
var wg sync.WaitGroup
defer wg.Wait()
w := &walker{
fn: walkFn,
enqueuec: make(chan walkItem, numWorkers), // buffered for performance
workc: make(chan walkItem, numWorkers), // buffered for performance
donec: make(chan struct{}),
// buffered for correctness & not leaking goroutines:
resc: make(chan error, numWorkers),
}
defer close(w.donec)
for i := 0; i < numWorkers; i++ {
wg.Add(1)
go w.doWork(&wg)
}
todo := []walkItem{{dir: root}}
out := 0
for {
workc := w.workc
var workItem walkItem
if len(todo) == 0 {
workc = nil
} else {
workItem = todo[len(todo)-1]
}
select {
case workc <- workItem:
todo = todo[:len(todo)-1]
out++
case it := <-w.enqueuec:
todo = append(todo, it)
case err := <-w.resc:
out--
if err != nil {
return err
}
if out == 0 && len(todo) == 0 {
// It's safe to quit here, as long as the buffered
// enqueue channel isn't also readable, which might
// happen if the worker sends both another unit of
// work and its result before the other select was
// scheduled and both w.resc and w.enqueuec were
// readable.
select {
case it := <-w.enqueuec:
todo = append(todo, it)
default:
return nil
}
}
}
}
}
// doWork reads directories as instructed (via workc) and runs the
// user's callback function.
func (w *walker) doWork(wg *sync.WaitGroup) {
defer wg.Done()
for {
select {
case <-w.donec:
return
case it := <-w.workc:
select {
case <-w.donec:
return
case w.resc <- w.walk(it.dir, !it.callbackDone):
}
}
}
}
type walker struct {
fn func(path string, typ os.FileMode) error
donec chan struct{} // closed on fastWalk's return
workc chan walkItem // to workers
enqueuec chan walkItem // from workers
resc chan error // from workers
}
type walkItem struct {
dir string
callbackDone bool // callback already called; don't do it again
}
func (w *walker) enqueue(it walkItem) {
select {
case w.enqueuec <- it:
case <-w.donec:
}
}
func (w *walker) onDirEnt(dirName, baseName string, typ os.FileMode) error {
joined := dirName + string(os.PathSeparator) + baseName
if typ == os.ModeDir {
w.enqueue(walkItem{dir: joined})
return nil
}
err := w.fn(joined, typ)
if typ == os.ModeSymlink {
if err == ErrTraverseLink {
// Set callbackDone so we don't call it twice for both the
// symlink-as-symlink and the symlink-as-directory later:
w.enqueue(walkItem{dir: joined, callbackDone: true})
return nil
}
if err == filepath.SkipDir {
// Permit SkipDir on symlinks too.
return nil
}
}
return err
}
func (w *walker) walk(root string, runUserCallback bool) error {
if runUserCallback {
err := w.fn(root, os.ModeDir)
if err == filepath.SkipDir {
return nil
}
if err != nil {
return err
}
}
return readDir(root, w.onDirEnt)
}

119
vendor/golang.org/x/tools/internal/fastwalk/fastwalk_darwin.go generated vendored Normal file
View File

@@ -0,0 +1,119 @@
// Copyright 2022 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
//go:build darwin && cgo
// +build darwin,cgo
package fastwalk
/*
#include <dirent.h>
// fastwalk_readdir_r wraps readdir_r so that we don't have to pass a dirent**
// result pointer which triggers CGO's "Go pointer to Go pointer" check unless
// we allocat the result dirent* with malloc.
//
// fastwalk_readdir_r returns 0 on success, -1 upon reaching the end of the
// directory, or a positive error number to indicate failure.
static int fastwalk_readdir_r(DIR *fd, struct dirent *entry) {
struct dirent *result;
int ret = readdir_r(fd, entry, &result);
if (ret == 0 && result == NULL) {
ret = -1; // EOF
}
return ret;
}
*/
import "C"
import (
"os"
"syscall"
"unsafe"
)
func readDir(dirName string, fn func(dirName, entName string, typ os.FileMode) error) error {
fd, err := openDir(dirName)
if err != nil {
return &os.PathError{Op: "opendir", Path: dirName, Err: err}
}
defer C.closedir(fd)
skipFiles := false
var dirent syscall.Dirent
for {
ret := int(C.fastwalk_readdir_r(fd, (*C.struct_dirent)(unsafe.Pointer(&dirent))))
if ret != 0 {
if ret == -1 {
break // EOF
}
if ret == int(syscall.EINTR) {
continue
}
return &os.PathError{Op: "readdir", Path: dirName, Err: syscall.Errno(ret)}
}
if dirent.Ino == 0 {
continue
}
typ := dtToType(dirent.Type)
if skipFiles && typ.IsRegular() {
continue
}
name := (*[len(syscall.Dirent{}.Name)]byte)(unsafe.Pointer(&dirent.Name))[:]
name = name[:dirent.Namlen]
for i, c := range name {
if c == 0 {
name = name[:i]
break
}
}
// Check for useless names before allocating a string.
if string(name) == "." || string(name) == ".." {
continue
}
if err := fn(dirName, string(name), typ); err != nil {
if err != ErrSkipFiles {
return err
}
skipFiles = true
}
}
return nil
}
func dtToType(typ uint8) os.FileMode {
switch typ {
case syscall.DT_BLK:
return os.ModeDevice
case syscall.DT_CHR:
return os.ModeDevice | os.ModeCharDevice
case syscall.DT_DIR:
return os.ModeDir
case syscall.DT_FIFO:
return os.ModeNamedPipe
case syscall.DT_LNK:
return os.ModeSymlink
case syscall.DT_REG:
return 0
case syscall.DT_SOCK:
return os.ModeSocket
}
return ^os.FileMode(0)
}
// openDir wraps opendir(3) and handles any EINTR errors. The returned *DIR
// needs to be closed with closedir(3).
func openDir(path string) (*C.DIR, error) {
name, err := syscall.BytePtrFromString(path)
if err != nil {
return nil, err
}
for {
fd, err := C.opendir((*C.char)(unsafe.Pointer(name)))
if err != syscall.EINTR {
return fd, err
}
}
}

14
vendor/golang.org/x/tools/internal/fastwalk/fastwalk_dirent_fileno.go generated vendored Normal file
View File

@@ -0,0 +1,14 @@
// Copyright 2016 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
//go:build freebsd || openbsd || netbsd
// +build freebsd openbsd netbsd
package fastwalk
import "syscall"
func direntInode(dirent *syscall.Dirent) uint64 {
return uint64(dirent.Fileno)
}

15
vendor/golang.org/x/tools/internal/fastwalk/fastwalk_dirent_ino.go generated vendored Normal file
View File

@@ -0,0 +1,15 @@
// Copyright 2016 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
//go:build (linux || (darwin && !cgo)) && !appengine
// +build linux darwin,!cgo
// +build !appengine
package fastwalk
import "syscall"
func direntInode(dirent *syscall.Dirent) uint64 {
return dirent.Ino
}

14
vendor/golang.org/x/tools/internal/fastwalk/fastwalk_dirent_namlen_bsd.go generated vendored Normal file
View File

@@ -0,0 +1,14 @@
// Copyright 2018 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
//go:build (darwin && !cgo) || freebsd || openbsd || netbsd
// +build darwin,!cgo freebsd openbsd netbsd
package fastwalk
import "syscall"
func direntNamlen(dirent *syscall.Dirent) uint64 {
return uint64(dirent.Namlen)
}

29
vendor/golang.org/x/tools/internal/fastwalk/fastwalk_dirent_namlen_linux.go generated vendored Normal file
View File

@@ -0,0 +1,29 @@
// Copyright 2018 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
//go:build linux && !appengine
// +build linux,!appengine
package fastwalk
import (
"bytes"
"syscall"
"unsafe"
)
func direntNamlen(dirent *syscall.Dirent) uint64 {
const fixedHdr = uint16(unsafe.Offsetof(syscall.Dirent{}.Name))
nameBuf := (*[unsafe.Sizeof(dirent.Name)]byte)(unsafe.Pointer(&dirent.Name[0]))
const nameBufLen = uint16(len(nameBuf))
limit := dirent.Reclen - fixedHdr
if limit > nameBufLen {
limit = nameBufLen
}
nameLen := bytes.IndexByte(nameBuf[:limit], 0)
if nameLen < 0 {
panic("failed to find terminating 0 byte in dirent")
}
return uint64(nameLen)
}

38
vendor/golang.org/x/tools/internal/fastwalk/fastwalk_portable.go generated vendored Normal file
View File

@@ -0,0 +1,38 @@
// Copyright 2016 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
//go:build appengine || (!linux && !darwin && !freebsd && !openbsd && !netbsd)
// +build appengine !linux,!darwin,!freebsd,!openbsd,!netbsd
package fastwalk
import (
"io/ioutil"
"os"
)
// readDir calls fn for each directory entry in dirName.
// It does not descend into directories or follow symlinks.
// If fn returns a non-nil error, readDir returns with that error
// immediately.
func readDir(dirName string, fn func(dirName, entName string, typ os.FileMode) error) error {
fis, err := ioutil.ReadDir(dirName)
if err != nil {
return err
}
skipFiles := false
for _, fi := range fis {
if fi.Mode().IsRegular() && skipFiles {
continue
}
if err := fn(dirName, fi.Name(), fi.Mode()&os.ModeType); err != nil {
if err == ErrSkipFiles {
skipFiles = true
continue
}
return err
}
}
return nil
}

153
vendor/golang.org/x/tools/internal/fastwalk/fastwalk_unix.go generated vendored Normal file
View File

@@ -0,0 +1,153 @@
// Copyright 2016 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
//go:build (linux || freebsd || openbsd || netbsd || (darwin && !cgo)) && !appengine
// +build linux freebsd openbsd netbsd darwin,!cgo
// +build !appengine
package fastwalk
import (
"fmt"
"os"
"syscall"
"unsafe"
)
const blockSize = 8 << 10
// unknownFileMode is a sentinel (and bogus) os.FileMode
// value used to represent a syscall.DT_UNKNOWN Dirent.Type.
const unknownFileMode os.FileMode = os.ModeNamedPipe | os.ModeSocket | os.ModeDevice
func readDir(dirName string, fn func(dirName, entName string, typ os.FileMode) error) error {
fd, err := open(dirName, 0, 0)
if err != nil {
return &os.PathError{Op: "open", Path: dirName, Err: err}
}
defer syscall.Close(fd)
// The buffer must be at least a block long.
buf := make([]byte, blockSize) // stack-allocated; doesn't escape
bufp := 0 // starting read position in buf
nbuf := 0 // end valid data in buf
skipFiles := false
for {
if bufp >= nbuf {
bufp = 0
nbuf, err = readDirent(fd, buf)
if err != nil {
return os.NewSyscallError("readdirent", err)
}
if nbuf <= 0 {
return nil
}
}
consumed, name, typ := parseDirEnt(buf[bufp:nbuf])
bufp += consumed
if name == "" || name == "." || name == ".." {
continue
}
// Fallback for filesystems (like old XFS) that don't
// support Dirent.Type and have DT_UNKNOWN (0) there
// instead.
if typ == unknownFileMode {
fi, err := os.Lstat(dirName + "/" + name)
if err != nil {
// It got deleted in the meantime.
if os.IsNotExist(err) {
continue
}
return err
}
typ = fi.Mode() & os.ModeType
}
if skipFiles && typ.IsRegular() {
continue
}
if err := fn(dirName, name, typ); err != nil {
if err == ErrSkipFiles {
skipFiles = true
continue
}
return err
}
}
}
func parseDirEnt(buf []byte) (consumed int, name string, typ os.FileMode) {
// golang.org/issue/37269
dirent := &syscall.Dirent{}
copy((*[unsafe.Sizeof(syscall.Dirent{})]byte)(unsafe.Pointer(dirent))[:], buf)
if v := unsafe.Offsetof(dirent.Reclen) + unsafe.Sizeof(dirent.Reclen); uintptr(len(buf)) < v {
panic(fmt.Sprintf("buf size of %d smaller than dirent header size %d", len(buf), v))
}
if len(buf) < int(dirent.Reclen) {
panic(fmt.Sprintf("buf size %d < record length %d", len(buf), dirent.Reclen))
}
consumed = int(dirent.Reclen)
if direntInode(dirent) == 0 { // File absent in directory.
return
}
switch dirent.Type {
case syscall.DT_REG:
typ = 0
case syscall.DT_DIR:
typ = os.ModeDir
case syscall.DT_LNK:
typ = os.ModeSymlink
case syscall.DT_BLK:
typ = os.ModeDevice
case syscall.DT_FIFO:
typ = os.ModeNamedPipe
case syscall.DT_SOCK:
typ = os.ModeSocket
case syscall.DT_UNKNOWN:
typ = unknownFileMode
default:
// Skip weird things.
// It's probably a DT_WHT (http://lwn.net/Articles/325369/)
// or something. Revisit if/when this package is moved outside
// of goimports. goimports only cares about regular files,
// symlinks, and directories.
return
}
nameBuf := (*[unsafe.Sizeof(dirent.Name)]byte)(unsafe.Pointer(&dirent.Name[0]))
nameLen := direntNamlen(dirent)
// Special cases for common things:
if nameLen == 1 && nameBuf[0] == '.' {
name = "."
} else if nameLen == 2 && nameBuf[0] == '.' && nameBuf[1] == '.' {
name = ".."
} else {
name = string(nameBuf[:nameLen])
}
return
}
// According to https://golang.org/doc/go1.14#runtime
// A consequence of the implementation of preemption is that on Unix systems, including Linux and macOS
// systems, programs built with Go 1.14 will receive more signals than programs built with earlier releases.
//
// This causes syscall.Open and syscall.ReadDirent sometimes fail with EINTR errors.
// We need to retry in this case.
func open(path string, mode int, perm uint32) (fd int, err error) {
for {
fd, err := syscall.Open(path, mode, perm)
if err != syscall.EINTR {
return fd, err
}
}
}
func readDirent(fd int, buf []byte) (n int, err error) {
for {
nbuf, err := syscall.ReadDirent(fd, buf)
if err != syscall.EINTR {
return nbuf, err
}
}
}

260
vendor/golang.org/x/tools/internal/gopathwalk/walk.go generated vendored Normal file
View File

@@ -0,0 +1,260 @@
// Copyright 2018 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package gopathwalk is like filepath.Walk but specialized for finding Go
// packages, particularly in $GOPATH and $GOROOT.
package gopathwalk
import (
"bufio"
"bytes"
"log"
"os"
"path/filepath"
"strings"
"time"
"golang.org/x/tools/internal/fastwalk"
)
// Options controls the behavior of a Walk call.
type Options struct {
// If Logf is non-nil, debug logging is enabled through this function.
Logf func(format string, args ...interface{})
// Search module caches. Also disables legacy goimports ignore rules.
ModulesEnabled bool
}
// RootType indicates the type of a Root.
type RootType int
const (
RootUnknown RootType = iota
RootGOROOT
RootGOPATH
RootCurrentModule
RootModuleCache
RootOther
)
// A Root is a starting point for a Walk.
type Root struct {
Path string
Type RootType
}
// Walk walks Go source directories ($GOROOT, $GOPATH, etc) to find packages.
// For each package found, add will be called (concurrently) with the absolute
// paths of the containing source directory and the package directory.
// add will be called concurrently.
func Walk(roots []Root, add func(root Root, dir string), opts Options) {
WalkSkip(roots, add, func(Root, string) bool { return false }, opts)
}
// WalkSkip walks Go source directories ($GOROOT, $GOPATH, etc) to find packages.
// For each package found, add will be called (concurrently) with the absolute
// paths of the containing source directory and the package directory.
// For each directory that will be scanned, skip will be called (concurrently)
// with the absolute paths of the containing source directory and the directory.
// If skip returns false on a directory it will be processed.
// add will be called concurrently.
// skip will be called concurrently.
func WalkSkip(roots []Root, add func(root Root, dir string), skip func(root Root, dir string) bool, opts Options) {
for _, root := range roots {
walkDir(root, add, skip, opts)
}
}
// walkDir creates a walker and starts fastwalk with this walker.
func walkDir(root Root, add func(Root, string), skip func(root Root, dir string) bool, opts Options) {
if _, err := os.Stat(root.Path); os.IsNotExist(err) {
if opts.Logf != nil {
opts.Logf("skipping nonexistent directory: %v", root.Path)
}
return
}
start := time.Now()
if opts.Logf != nil {
opts.Logf("scanning %s", root.Path)
}
w := &walker{
root: root,
add: add,
skip: skip,
opts: opts,
}
w.init()
if err := fastwalk.Walk(root.Path, w.walk); err != nil {
logf := opts.Logf
if logf == nil {
logf = log.Printf
}
logf("scanning directory %v: %v", root.Path, err)
}
if opts.Logf != nil {
opts.Logf("scanned %s in %v", root.Path, time.Since(start))
}
}
// walker is the callback for fastwalk.Walk.
type walker struct {
root Root // The source directory to scan.
add func(Root, string) // The callback that will be invoked for every possible Go package dir.
skip func(Root, string) bool // The callback that will be invoked for every dir. dir is skipped if it returns true.
opts Options // Options passed to Walk by the user.
ignoredDirs []os.FileInfo // The ignored directories, loaded from .goimportsignore files.
}
// init initializes the walker based on its Options
func (w *walker) init() {
var ignoredPaths []string
if w.root.Type == RootModuleCache {
ignoredPaths = []string{"cache"}
}
if !w.opts.ModulesEnabled && w.root.Type == RootGOPATH {
ignoredPaths = w.getIgnoredDirs(w.root.Path)
ignoredPaths = append(ignoredPaths, "v", "mod")
}
for _, p := range ignoredPaths {
full := filepath.Join(w.root.Path, p)
if fi, err := os.Stat(full); err == nil {
w.ignoredDirs = append(w.ignoredDirs, fi)
if w.opts.Logf != nil {
w.opts.Logf("Directory added to ignore list: %s", full)
}
} else if w.opts.Logf != nil {
w.opts.Logf("Error statting ignored directory: %v", err)
}
}
}
// getIgnoredDirs reads an optional config file at <path>/.goimportsignore
// of relative directories to ignore when scanning for go files.
// The provided path is one of the $GOPATH entries with "src" appended.
func (w *walker) getIgnoredDirs(path string) []string {
file := filepath.Join(path, ".goimportsignore")
slurp, err := os.ReadFile(file)
if w.opts.Logf != nil {
if err != nil {
w.opts.Logf("%v", err)
} else {
w.opts.Logf("Read %s", file)
}
}
if err != nil {
return nil
}
var ignoredDirs []string
bs := bufio.NewScanner(bytes.NewReader(slurp))
for bs.Scan() {
line := strings.TrimSpace(bs.Text())
if line == "" || strings.HasPrefix(line, "#") {
continue
}
ignoredDirs = append(ignoredDirs, line)
}
return ignoredDirs
}
// shouldSkipDir reports whether the file should be skipped or not.
func (w *walker) shouldSkipDir(fi os.FileInfo, dir string) bool {
for _, ignoredDir := range w.ignoredDirs {
if os.SameFile(fi, ignoredDir) {
return true
}
}
if w.skip != nil {
// Check with the user specified callback.
return w.skip(w.root, dir)
}
return false
}
// walk walks through the given path.
func (w *walker) walk(path string, typ os.FileMode) error {
if typ.IsRegular() {
dir := filepath.Dir(path)
if dir == w.root.Path && (w.root.Type == RootGOROOT || w.root.Type == RootGOPATH) {
// Doesn't make sense to have regular files
// directly in your $GOPATH/src or $GOROOT/src.
return fastwalk.ErrSkipFiles
}
if !strings.HasSuffix(path, ".go") {
return nil
}
w.add(w.root, dir)
return fastwalk.ErrSkipFiles
}
if typ == os.ModeDir {
base := filepath.Base(path)
if base == "" || base[0] == '.' || base[0] == '_' ||
base == "testdata" ||
(w.root.Type == RootGOROOT && w.opts.ModulesEnabled && base == "vendor") ||
(!w.opts.ModulesEnabled && base == "node_modules") {
return filepath.SkipDir
}
fi, err := os.Lstat(path)
if err == nil && w.shouldSkipDir(fi, path) {
return filepath.SkipDir
}
return nil
}
if typ == os.ModeSymlink {
base := filepath.Base(path)
if strings.HasPrefix(base, ".#") {
// Emacs noise.
return nil
}
if w.shouldTraverse(path) {
return fastwalk.ErrTraverseLink
}
}
return nil
}
// shouldTraverse reports whether the symlink fi, found in dir,
// should be followed. It makes sure symlinks were never visited
// before to avoid symlink loops.
func (w *walker) shouldTraverse(path string) bool {
ts, err := os.Stat(path)
if err != nil {
logf := w.opts.Logf
if logf == nil {
logf = log.Printf
}
logf("%v", err)
return false
}
if !ts.IsDir() {
return false
}
if w.shouldSkipDir(ts, filepath.Dir(path)) {
return false
}
// Check for symlink loops by statting each directory component
// and seeing if any are the same file as ts.
for {
parent := filepath.Dir(path)
if parent == path {
// Made it to the root without seeing a cycle.
// Use this symlink.
return true
}
parentInfo, err := os.Stat(parent)
if err != nil {
return false
}
if os.SameFile(ts, parentInfo) {
// Cycle. Don't traverse.
return false
}
path = parent
}
}

1766
vendor/golang.org/x/tools/internal/imports/fix.go generated vendored Normal file
View File

File diff suppressed because it is too large Load Diff

356
vendor/golang.org/x/tools/internal/imports/imports.go generated vendored Normal file
View File

@@ -0,0 +1,356 @@
// Copyright 2013 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
//go:generate go run mkstdlib.go
// Package imports implements a Go pretty-printer (like package "go/format")
// that also adds or removes import statements as necessary.
package imports
import (
"bufio"
"bytes"
"context"
"fmt"
"go/ast"
"go/format"
"go/parser"
"go/printer"
"go/token"
"io"
"regexp"
"strconv"
"strings"
"golang.org/x/tools/go/ast/astutil"
"golang.org/x/tools/internal/event"
)
// Options is golang.org/x/tools/imports.Options with extra internal-only options.
type Options struct {
Env *ProcessEnv // The environment to use. Note: this contains the cached module and filesystem state.
// LocalPrefix is a comma-separated string of import path prefixes, which, if
// set, instructs Process to sort the import paths with the given prefixes
// into another group after 3rd-party packages.
LocalPrefix string
Fragment bool // Accept fragment of a source file (no package statement)
AllErrors bool // Report all errors (not just the first 10 on different lines)
Comments bool // Print comments (true if nil *Options provided)
TabIndent bool // Use tabs for indent (true if nil *Options provided)
TabWidth int // Tab width (8 if nil *Options provided)
FormatOnly bool // Disable the insertion and deletion of imports
}
// Process implements golang.org/x/tools/imports.Process with explicit context in opt.Env.
func Process(filename string, src []byte, opt *Options) (formatted []byte, err error) {
fileSet := token.NewFileSet()
file, adjust, err := parse(fileSet, filename, src, opt)
if err != nil {
return nil, err
}
if !opt.FormatOnly {
if err := fixImports(fileSet, file, filename, opt.Env); err != nil {
return nil, err
}
}
return formatFile(fileSet, file, src, adjust, opt)
}
// FixImports returns a list of fixes to the imports that, when applied,
// will leave the imports in the same state as Process. src and opt must
// be specified.
//
// Note that filename's directory influences which imports can be chosen,
// so it is important that filename be accurate.
func FixImports(ctx context.Context, filename string, src []byte, opt *Options) (fixes []*ImportFix, err error) {
ctx, done := event.Start(ctx, "imports.FixImports")
defer done()
fileSet := token.NewFileSet()
file, _, err := parse(fileSet, filename, src, opt)
if err != nil {
return nil, err
}
return getFixes(ctx, fileSet, file, filename, opt.Env)
}
// ApplyFixes applies all of the fixes to the file and formats it. extraMode
// is added in when parsing the file. src and opts must be specified, but no
// env is needed.
func ApplyFixes(fixes []*ImportFix, filename string, src []byte, opt *Options, extraMode parser.Mode) (formatted []byte, err error) {
// Don't use parse() -- we don't care about fragments or statement lists
// here, and we need to work with unparseable files.
fileSet := token.NewFileSet()
parserMode := parser.Mode(0)
if opt.Comments {
parserMode |= parser.ParseComments
}
if opt.AllErrors {
parserMode |= parser.AllErrors
}
parserMode |= extraMode
file, err := parser.ParseFile(fileSet, filename, src, parserMode)
if file == nil {
return nil, err
}
// Apply the fixes to the file.
apply(fileSet, file, fixes)
return formatFile(fileSet, file, src, nil, opt)
}
// formatFile formats the file syntax tree.
// It may mutate the token.FileSet.
//
// If an adjust function is provided, it is called after formatting
// with the original source (formatFile's src parameter) and the
// formatted file, and returns the postpocessed result.
func formatFile(fset *token.FileSet, file *ast.File, src []byte, adjust func(orig []byte, src []byte) []byte, opt *Options) ([]byte, error) {
mergeImports(file)
sortImports(opt.LocalPrefix, fset.File(file.Pos()), file)
var spacesBefore []string // import paths we need spaces before
for _, impSection := range astutil.Imports(fset, file) {
// Within each block of contiguous imports, see if any
// import lines are in different group numbers. If so,
// we'll need to put a space between them so it's
// compatible with gofmt.
lastGroup := -1
for _, importSpec := range impSection {
importPath, _ := strconv.Unquote(importSpec.Path.Value)
groupNum := importGroup(opt.LocalPrefix, importPath)
if groupNum != lastGroup && lastGroup != -1 {
spacesBefore = append(spacesBefore, importPath)
}
lastGroup = groupNum
}
}
printerMode := printer.UseSpaces
if opt.TabIndent {
printerMode |= printer.TabIndent
}
printConfig := &printer.Config{Mode: printerMode, Tabwidth: opt.TabWidth}
var buf bytes.Buffer
err := printConfig.Fprint(&buf, fset, file)
if err != nil {
return nil, err
}
out := buf.Bytes()
if adjust != nil {
out = adjust(src, out)
}
if len(spacesBefore) > 0 {
out, err = addImportSpaces(bytes.NewReader(out), spacesBefore)
if err != nil {
return nil, err
}
}
out, err = format.Source(out)
if err != nil {
return nil, err
}
return out, nil
}
// parse parses src, which was read from filename,
// as a Go source file or statement list.
func parse(fset *token.FileSet, filename string, src []byte, opt *Options) (*ast.File, func(orig, src []byte) []byte, error) {
parserMode := parser.Mode(0)
if opt.Comments {
parserMode |= parser.ParseComments
}
if opt.AllErrors {
parserMode |= parser.AllErrors
}
// Try as whole source file.
file, err := parser.ParseFile(fset, filename, src, parserMode)
if err == nil {
return file, nil, nil
}
// If the error is that the source file didn't begin with a
// package line and we accept fragmented input, fall through to
// try as a source fragment. Stop and return on any other error.
if !opt.Fragment || !strings.Contains(err.Error(), "expected 'package'") {
return nil, nil, err
}
// If this is a declaration list, make it a source file
// by inserting a package clause.
// Insert using a ;, not a newline, so that parse errors are on
// the correct line.
const prefix = "package main;"
psrc := append([]byte(prefix), src...)
file, err = parser.ParseFile(fset, filename, psrc, parserMode)
if err == nil {
// Gofmt will turn the ; into a \n.
// Do that ourselves now and update the file contents,
// so that positions and line numbers are correct going forward.
psrc[len(prefix)-1] = '\n'
fset.File(file.Package).SetLinesForContent(psrc)
// If a main function exists, we will assume this is a main
// package and leave the file.
if containsMainFunc(file) {
return file, nil, nil
}
adjust := func(orig, src []byte) []byte {
// Remove the package clause.
src = src[len(prefix):]
return matchSpace(orig, src)
}
return file, adjust, nil
}
// If the error is that the source file didn't begin with a
// declaration, fall through to try as a statement list.
// Stop and return on any other error.
if !strings.Contains(err.Error(), "expected declaration") {
return nil, nil, err
}
// If this is a statement list, make it a source file
// by inserting a package clause and turning the list
// into a function body. This handles expressions too.
// Insert using a ;, not a newline, so that the line numbers
// in fsrc match the ones in src.
fsrc := append(append([]byte("package p; func _() {"), src...), '}')
file, err = parser.ParseFile(fset, filename, fsrc, parserMode)
if err == nil {
adjust := func(orig, src []byte) []byte {
// Remove the wrapping.
// Gofmt has turned the ; into a \n\n.
src = src[len("package p\n\nfunc _() {"):]
src = src[:len(src)-len("}\n")]
// Gofmt has also indented the function body one level.
// Remove that indent.
src = bytes.Replace(src, []byte("\n\t"), []byte("\n"), -1)
return matchSpace(orig, src)
}
return file, adjust, nil
}
// Failed, and out of options.
return nil, nil, err
}
// containsMainFunc checks if a file contains a function declaration with the
// function signature 'func main()'
func containsMainFunc(file *ast.File) bool {
for _, decl := range file.Decls {
if f, ok := decl.(*ast.FuncDecl); ok {
if f.Name.Name != "main" {
continue
}
if len(f.Type.Params.List) != 0 {
continue
}
if f.Type.Results != nil && len(f.Type.Results.List) != 0 {
continue
}
return true
}
}
return false
}
func cutSpace(b []byte) (before, middle, after []byte) {
i := 0
for i < len(b) && (b[i] == ' ' || b[i] == '\t' || b[i] == '\n') {
i++
}
j := len(b)
for j > 0 && (b[j-1] == ' ' || b[j-1] == '\t' || b[j-1] == '\n') {
j--
}
if i <= j {
return b[:i], b[i:j], b[j:]
}
return nil, nil, b[j:]
}
// matchSpace reformats src to use the same space context as orig.
// 1. If orig begins with blank lines, matchSpace inserts them at the beginning of src.
// 2. matchSpace copies the indentation of the first non-blank line in orig
// to every non-blank line in src.
// 3. matchSpace copies the trailing space from orig and uses it in place
// of src's trailing space.
func matchSpace(orig []byte, src []byte) []byte {
before, _, after := cutSpace(orig)
i := bytes.LastIndex(before, []byte{'\n'})
before, indent := before[:i+1], before[i+1:]
_, src, _ = cutSpace(src)
var b bytes.Buffer
b.Write(before)
for len(src) > 0 {
line := src
if i := bytes.IndexByte(line, '\n'); i >= 0 {
line, src = line[:i+1], line[i+1:]
} else {
src = nil
}
if len(line) > 0 && line[0] != '\n' { // not blank
b.Write(indent)
}
b.Write(line)
}
b.Write(after)
return b.Bytes()
}
var impLine = regexp.MustCompile(`^\s+(?:[\w\.]+\s+)?"(.+?)"`)
func addImportSpaces(r io.Reader, breaks []string) ([]byte, error) {
var out bytes.Buffer
in := bufio.NewReader(r)
inImports := false
done := false
for {
s, err := in.ReadString('\n')
if err == io.EOF {
break
} else if err != nil {
return nil, err
}
if !inImports && !done && strings.HasPrefix(s, "import") {
inImports = true
}
if inImports && (strings.HasPrefix(s, "var") ||
strings.HasPrefix(s, "func") ||
strings.HasPrefix(s, "const") ||
strings.HasPrefix(s, "type")) {
done = true
inImports = false
}
if inImports && len(breaks) > 0 {
if m := impLine.FindStringSubmatch(s); m != nil {
if m[1] == breaks[0] {
out.WriteByte('\n')
breaks = breaks[1:]
}
}
}
fmt.Fprint(&out, s)
}
return out.Bytes(), nil
}

724
vendor/golang.org/x/tools/internal/imports/mod.go generated vendored Normal file
View File

@@ -0,0 +1,724 @@
// Copyright 2019 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package imports
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io/ioutil"
"os"
"path"
"path/filepath"
"regexp"
"sort"
"strconv"
"strings"
"golang.org/x/mod/module"
"golang.org/x/tools/internal/event"
"golang.org/x/tools/internal/gocommand"
"golang.org/x/tools/internal/gopathwalk"
)
// ModuleResolver implements resolver for modules using the go command as little
// as feasible.
type ModuleResolver struct {
env *ProcessEnv
moduleCacheDir string
dummyVendorMod *gocommand.ModuleJSON // If vendoring is enabled, the pseudo-module that represents the /vendor directory.
roots []gopathwalk.Root
scanSema chan struct{} // scanSema prevents concurrent scans and guards scannedRoots.
scannedRoots map[gopathwalk.Root]bool
initialized bool
mains []*gocommand.ModuleJSON
mainByDir map[string]*gocommand.ModuleJSON
modsByModPath []*gocommand.ModuleJSON // All modules, ordered by # of path components in module Path...
modsByDir []*gocommand.ModuleJSON // ...or number of path components in their Dir.
// moduleCacheCache stores information about the module cache.
moduleCacheCache *dirInfoCache
otherCache *dirInfoCache
}
func newModuleResolver(e *ProcessEnv) *ModuleResolver {
r := &ModuleResolver{
env: e,
scanSema: make(chan struct{}, 1),
}
r.scanSema <- struct{}{}
return r
}
func (r *ModuleResolver) init() error {
if r.initialized {
return nil
}
goenv, err := r.env.goEnv()
if err != nil {
return err
}
inv := gocommand.Invocation{
BuildFlags: r.env.BuildFlags,
ModFlag: r.env.ModFlag,
ModFile: r.env.ModFile,
Env: r.env.env(),
Logf: r.env.Logf,
WorkingDir: r.env.WorkingDir,
}
vendorEnabled := false
var mainModVendor *gocommand.ModuleJSON
// Module vendor directories are ignored in workspace mode:
// https://go.googlesource.com/proposal/+/master/design/45713-workspace.md
if len(r.env.Env["GOWORK"]) == 0 {
vendorEnabled, mainModVendor, err = gocommand.VendorEnabled(context.TODO(), inv, r.env.GocmdRunner)
if err != nil {
return err
}
}
if mainModVendor != nil && vendorEnabled {
// Vendor mode is on, so all the non-Main modules are irrelevant,
// and we need to search /vendor for everything.
r.mains = []*gocommand.ModuleJSON{mainModVendor}
r.dummyVendorMod = &gocommand.ModuleJSON{
Path: "",
Dir: filepath.Join(mainModVendor.Dir, "vendor"),
}
r.modsByModPath = []*gocommand.ModuleJSON{mainModVendor, r.dummyVendorMod}
r.modsByDir = []*gocommand.ModuleJSON{mainModVendor, r.dummyVendorMod}
} else {
// Vendor mode is off, so run go list -m ... to find everything.
err := r.initAllMods()
// We expect an error when running outside of a module with
// GO111MODULE=on. Other errors are fatal.
if err != nil {
if errMsg := err.Error(); !strings.Contains(errMsg, "working directory is not part of a module") && !strings.Contains(errMsg, "go.mod file not found") {
return err
}
}
}
if gmc := r.env.Env["GOMODCACHE"]; gmc != "" {
r.moduleCacheDir = gmc
} else {
gopaths := filepath.SplitList(goenv["GOPATH"])
if len(gopaths) == 0 {
return fmt.Errorf("empty GOPATH")
}
r.moduleCacheDir = filepath.Join(gopaths[0], "/pkg/mod")
}
sort.Slice(r.modsByModPath, func(i, j int) bool {
count := func(x int) int {
return strings.Count(r.modsByModPath[x].Path, "/")
}
return count(j) < count(i) // descending order
})
sort.Slice(r.modsByDir, func(i, j int) bool {
count := func(x int) int {
return strings.Count(r.modsByDir[x].Dir, string(filepath.Separator))
}
return count(j) < count(i) // descending order
})
r.roots = []gopathwalk.Root{
{Path: filepath.Join(goenv["GOROOT"], "/src"), Type: gopathwalk.RootGOROOT},
}
r.mainByDir = make(map[string]*gocommand.ModuleJSON)
for _, main := range r.mains {
r.roots = append(r.roots, gopathwalk.Root{Path: main.Dir, Type: gopathwalk.RootCurrentModule})
r.mainByDir[main.Dir] = main
}
if vendorEnabled {
r.roots = append(r.roots, gopathwalk.Root{Path: r.dummyVendorMod.Dir, Type: gopathwalk.RootOther})
} else {
addDep := func(mod *gocommand.ModuleJSON) {
if mod.Replace == nil {
// This is redundant with the cache, but we'll skip it cheaply enough.
r.roots = append(r.roots, gopathwalk.Root{Path: mod.Dir, Type: gopathwalk.RootModuleCache})
} else {
r.roots = append(r.roots, gopathwalk.Root{Path: mod.Dir, Type: gopathwalk.RootOther})
}
}
// Walk dependent modules before scanning the full mod cache, direct deps first.
for _, mod := range r.modsByModPath {
if !mod.Indirect && !mod.Main {
addDep(mod)
}
}
for _, mod := range r.modsByModPath {
if mod.Indirect && !mod.Main {
addDep(mod)
}
}
r.roots = append(r.roots, gopathwalk.Root{Path: r.moduleCacheDir, Type: gopathwalk.RootModuleCache})
}
r.scannedRoots = map[gopathwalk.Root]bool{}
if r.moduleCacheCache == nil {
r.moduleCacheCache = &dirInfoCache{
dirs: map[string]*directoryPackageInfo{},
listeners: map[*int]cacheListener{},
}
}
if r.otherCache == nil {
r.otherCache = &dirInfoCache{
dirs: map[string]*directoryPackageInfo{},
listeners: map[*int]cacheListener{},
}
}
r.initialized = true
return nil
}
func (r *ModuleResolver) initAllMods() error {
stdout, err := r.env.invokeGo(context.TODO(), "list", "-m", "-e", "-json", "...")
if err != nil {
return err
}
for dec := json.NewDecoder(stdout); dec.More(); {
mod := &gocommand.ModuleJSON{}
if err := dec.Decode(mod); err != nil {
return err
}
if mod.Dir == "" {
if r.env.Logf != nil {
r.env.Logf("module %v has not been downloaded and will be ignored", mod.Path)
}
// Can't do anything with a module that's not downloaded.
continue
}
// golang/go#36193: the go command doesn't always clean paths.
mod.Dir = filepath.Clean(mod.Dir)
r.modsByModPath = append(r.modsByModPath, mod)
r.modsByDir = append(r.modsByDir, mod)
if mod.Main {
r.mains = append(r.mains, mod)
}
}
return nil
}
func (r *ModuleResolver) ClearForNewScan() {
<-r.scanSema
r.scannedRoots = map[gopathwalk.Root]bool{}
r.otherCache = &dirInfoCache{
dirs: map[string]*directoryPackageInfo{},
listeners: map[*int]cacheListener{},
}
r.scanSema <- struct{}{}
}
func (r *ModuleResolver) ClearForNewMod() {
<-r.scanSema
*r = ModuleResolver{
env: r.env,
moduleCacheCache: r.moduleCacheCache,
otherCache: r.otherCache,
scanSema: r.scanSema,
}
r.init()
r.scanSema <- struct{}{}
}
// findPackage returns the module and directory that contains the package at
// the given import path, or returns nil, "" if no module is in scope.
func (r *ModuleResolver) findPackage(importPath string) (*gocommand.ModuleJSON, string) {
// This can't find packages in the stdlib, but that's harmless for all
// the existing code paths.
for _, m := range r.modsByModPath {
if !strings.HasPrefix(importPath, m.Path) {
continue
}
pathInModule := importPath[len(m.Path):]
pkgDir := filepath.Join(m.Dir, pathInModule)
if r.dirIsNestedModule(pkgDir, m) {
continue
}
if info, ok := r.cacheLoad(pkgDir); ok {
if loaded, err := info.reachedStatus(nameLoaded); loaded {
if err != nil {
continue // No package in this dir.
}
return m, pkgDir
}
if scanned, err := info.reachedStatus(directoryScanned); scanned && err != nil {
continue // Dir is unreadable, etc.
}
// This is slightly wrong: a directory doesn't have to have an
// importable package to count as a package for package-to-module
// resolution. package main or _test files should count but
// don't.
// TODO(heschi): fix this.
if _, err := r.cachePackageName(info); err == nil {
return m, pkgDir
}
}
// Not cached. Read the filesystem.
pkgFiles, err := ioutil.ReadDir(pkgDir)
if err != nil {
continue
}
// A module only contains a package if it has buildable go
// files in that directory. If not, it could be provided by an
// outer module. See #29736.
for _, fi := range pkgFiles {
if ok, _ := r.env.matchFile(pkgDir, fi.Name()); ok {
return m, pkgDir
}
}
}
return nil, ""
}
func (r *ModuleResolver) cacheLoad(dir string) (directoryPackageInfo, bool) {
if info, ok := r.moduleCacheCache.Load(dir); ok {
return info, ok
}
return r.otherCache.Load(dir)
}
func (r *ModuleResolver) cacheStore(info directoryPackageInfo) {
if info.rootType == gopathwalk.RootModuleCache {
r.moduleCacheCache.Store(info.dir, info)
} else {
r.otherCache.Store(info.dir, info)
}
}
func (r *ModuleResolver) cacheKeys() []string {
return append(r.moduleCacheCache.Keys(), r.otherCache.Keys()...)
}
// cachePackageName caches the package name for a dir already in the cache.
func (r *ModuleResolver) cachePackageName(info directoryPackageInfo) (string, error) {
if info.rootType == gopathwalk.RootModuleCache {
return r.moduleCacheCache.CachePackageName(info)
}
return r.otherCache.CachePackageName(info)
}
func (r *ModuleResolver) cacheExports(ctx context.Context, env *ProcessEnv, info directoryPackageInfo) (string, []string, error) {
if info.rootType == gopathwalk.RootModuleCache {
return r.moduleCacheCache.CacheExports(ctx, env, info)
}
return r.otherCache.CacheExports(ctx, env, info)
}
// findModuleByDir returns the module that contains dir, or nil if no such
// module is in scope.
func (r *ModuleResolver) findModuleByDir(dir string) *gocommand.ModuleJSON {
// This is quite tricky and may not be correct. dir could be:
// - a package in the main module.
// - a replace target underneath the main module's directory.
// - a nested module in the above.
// - a replace target somewhere totally random.
// - a nested module in the above.
// - in the mod cache.
// - in /vendor/ in -mod=vendor mode.
// - nested module? Dunno.
// Rumor has it that replace targets cannot contain other replace targets.
//
// Note that it is critical here that modsByDir is sorted to have deeper dirs
// first. This ensures that findModuleByDir finds the innermost module.
// See also golang/go#56291.
for _, m := range r.modsByDir {
if !strings.HasPrefix(dir, m.Dir) {
continue
}
if r.dirIsNestedModule(dir, m) {
continue
}
return m
}
return nil
}
// dirIsNestedModule reports if dir is contained in a nested module underneath
// mod, not actually in mod.
func (r *ModuleResolver) dirIsNestedModule(dir string, mod *gocommand.ModuleJSON) bool {
if !strings.HasPrefix(dir, mod.Dir) {
return false
}
if r.dirInModuleCache(dir) {
// Nested modules in the module cache are pruned,
// so it cannot be a nested module.
return false
}
if mod != nil && mod == r.dummyVendorMod {
// The /vendor pseudomodule is flattened and doesn't actually count.
return false
}
modDir, _ := r.modInfo(dir)
if modDir == "" {
return false
}
return modDir != mod.Dir
}
func (r *ModuleResolver) modInfo(dir string) (modDir string, modName string) {
readModName := func(modFile string) string {
modBytes, err := ioutil.ReadFile(modFile)
if err != nil {
return ""
}
return modulePath(modBytes)
}
if r.dirInModuleCache(dir) {
if matches := modCacheRegexp.FindStringSubmatch(dir); len(matches) == 3 {
index := strings.Index(dir, matches[1]+"@"+matches[2])
modDir := filepath.Join(dir[:index], matches[1]+"@"+matches[2])
return modDir, readModName(filepath.Join(modDir, "go.mod"))
}
}
for {
if info, ok := r.cacheLoad(dir); ok {
return info.moduleDir, info.moduleName
}
f := filepath.Join(dir, "go.mod")
info, err := os.Stat(f)
if err == nil && !info.IsDir() {
return dir, readModName(f)
}
d := filepath.Dir(dir)
if len(d) >= len(dir) {
return "", "" // reached top of file system, no go.mod
}
dir = d
}
}
func (r *ModuleResolver) dirInModuleCache(dir string) bool {
if r.moduleCacheDir == "" {
return false
}
return strings.HasPrefix(dir, r.moduleCacheDir)
}
func (r *ModuleResolver) loadPackageNames(importPaths []string, srcDir string) (map[string]string, error) {
if err := r.init(); err != nil {
return nil, err
}
names := map[string]string{}
for _, path := range importPaths {
_, packageDir := r.findPackage(path)
if packageDir == "" {
continue
}
name, err := packageDirToName(packageDir)
if err != nil {
continue
}
names[path] = name
}
return names, nil
}
func (r *ModuleResolver) scan(ctx context.Context, callback *scanCallback) error {
ctx, done := event.Start(ctx, "imports.ModuleResolver.scan")
defer done()
if err := r.init(); err != nil {
return err
}
processDir := func(info directoryPackageInfo) {
// Skip this directory if we were not able to get the package information successfully.
if scanned, err := info.reachedStatus(directoryScanned); !scanned || err != nil {
return
}
pkg, err := r.canonicalize(info)
if err != nil {
return
}
if !callback.dirFound(pkg) {
return
}
pkg.packageName, err = r.cachePackageName(info)
if err != nil {
return
}
if !callback.packageNameLoaded(pkg) {
return
}
_, exports, err := r.loadExports(ctx, pkg, false)
if err != nil {
return
}
callback.exportsLoaded(pkg, exports)
}
// Start processing everything in the cache, and listen for the new stuff
// we discover in the walk below.
stop1 := r.moduleCacheCache.ScanAndListen(ctx, processDir)
defer stop1()
stop2 := r.otherCache.ScanAndListen(ctx, processDir)
defer stop2()
// We assume cached directories are fully cached, including all their
// children, and have not changed. We can skip them.
skip := func(root gopathwalk.Root, dir string) bool {
if r.env.SkipPathInScan != nil && root.Type == gopathwalk.RootCurrentModule {
if root.Path == dir {
return false
}
if r.env.SkipPathInScan(filepath.Clean(dir)) {
return true
}
}
info, ok := r.cacheLoad(dir)
if !ok {
return false
}
// This directory can be skipped as long as we have already scanned it.
// Packages with errors will continue to have errors, so there is no need
// to rescan them.
packageScanned, _ := info.reachedStatus(directoryScanned)
return packageScanned
}
// Add anything new to the cache, and process it if we're still listening.
add := func(root gopathwalk.Root, dir string) {
r.cacheStore(r.scanDirForPackage(root, dir))
}
// r.roots and the callback are not necessarily safe to use in the
// goroutine below. Process them eagerly.
roots := filterRoots(r.roots, callback.rootFound)
// We can't cancel walks, because we need them to finish to have a usable
// cache. Instead, run them in a separate goroutine and detach.
scanDone := make(chan struct{})
go func() {
select {
case <-ctx.Done():
return
case <-r.scanSema:
}
defer func() { r.scanSema <- struct{}{} }()
// We have the lock on r.scannedRoots, and no other scans can run.
for _, root := range roots {
if ctx.Err() != nil {
return
}
if r.scannedRoots[root] {
continue
}
gopathwalk.WalkSkip([]gopathwalk.Root{root}, add, skip, gopathwalk.Options{Logf: r.env.Logf, ModulesEnabled: true})
r.scannedRoots[root] = true
}
close(scanDone)
}()
select {
case <-ctx.Done():
case <-scanDone:
}
return nil
}
func (r *ModuleResolver) scoreImportPath(ctx context.Context, path string) float64 {
if _, ok := stdlib[path]; ok {
return MaxRelevance
}
mod, _ := r.findPackage(path)
return modRelevance(mod)
}
func modRelevance(mod *gocommand.ModuleJSON) float64 {
var relevance float64
switch {
case mod == nil: // out of scope
return MaxRelevance - 4
case mod.Indirect:
relevance = MaxRelevance - 3
case !mod.Main:
relevance = MaxRelevance - 2
default:
relevance = MaxRelevance - 1 // main module ties with stdlib
}
_, versionString, ok := module.SplitPathVersion(mod.Path)
if ok {
index := strings.Index(versionString, "v")
if index == -1 {
return relevance
}
if versionNumber, err := strconv.ParseFloat(versionString[index+1:], 64); err == nil {
relevance += versionNumber / 1000
}
}
return relevance
}
// canonicalize gets the result of canonicalizing the packages using the results
// of initializing the resolver from 'go list -m'.
func (r *ModuleResolver) canonicalize(info directoryPackageInfo) (*pkg, error) {
// Packages in GOROOT are already canonical, regardless of the std/cmd modules.
if info.rootType == gopathwalk.RootGOROOT {
return &pkg{
importPathShort: info.nonCanonicalImportPath,
dir: info.dir,
packageName: path.Base(info.nonCanonicalImportPath),
relevance: MaxRelevance,
}, nil
}
importPath := info.nonCanonicalImportPath
mod := r.findModuleByDir(info.dir)
// Check if the directory is underneath a module that's in scope.
if mod != nil {
// It is. If dir is the target of a replace directive,
// our guessed import path is wrong. Use the real one.
if mod.Dir == info.dir {
importPath = mod.Path
} else {
dirInMod := info.dir[len(mod.Dir)+len("/"):]
importPath = path.Join(mod.Path, filepath.ToSlash(dirInMod))
}
} else if !strings.HasPrefix(importPath, info.moduleName) {
// The module's name doesn't match the package's import path. It
// probably needs a replace directive we don't have.
return nil, fmt.Errorf("package in %q is not valid without a replace statement", info.dir)
}
res := &pkg{
importPathShort: importPath,
dir: info.dir,
relevance: modRelevance(mod),
}
// We may have discovered a package that has a different version
// in scope already. Canonicalize to that one if possible.
if _, canonicalDir := r.findPackage(importPath); canonicalDir != "" {
res.dir = canonicalDir
}
return res, nil
}
func (r *ModuleResolver) loadExports(ctx context.Context, pkg *pkg, includeTest bool) (string, []string, error) {
if err := r.init(); err != nil {
return "", nil, err
}
if info, ok := r.cacheLoad(pkg.dir); ok && !includeTest {
return r.cacheExports(ctx, r.env, info)
}
return loadExportsFromFiles(ctx, r.env, pkg.dir, includeTest)
}
func (r *ModuleResolver) scanDirForPackage(root gopathwalk.Root, dir string) directoryPackageInfo {
subdir := ""
if dir != root.Path {
subdir = dir[len(root.Path)+len("/"):]
}
importPath := filepath.ToSlash(subdir)
if strings.HasPrefix(importPath, "vendor/") {
// Only enter vendor directories if they're explicitly requested as a root.
return directoryPackageInfo{
status: directoryScanned,
err: fmt.Errorf("unwanted vendor directory"),
}
}
switch root.Type {
case gopathwalk.RootCurrentModule:
importPath = path.Join(r.mainByDir[root.Path].Path, filepath.ToSlash(subdir))
case gopathwalk.RootModuleCache:
matches := modCacheRegexp.FindStringSubmatch(subdir)
if len(matches) == 0 {
return directoryPackageInfo{
status: directoryScanned,
err: fmt.Errorf("invalid module cache path: %v", subdir),
}
}
modPath, err := module.UnescapePath(filepath.ToSlash(matches[1]))
if err != nil {
if r.env.Logf != nil {
r.env.Logf("decoding module cache path %q: %v", subdir, err)
}
return directoryPackageInfo{
status: directoryScanned,
err: fmt.Errorf("decoding module cache path %q: %v", subdir, err),
}
}
importPath = path.Join(modPath, filepath.ToSlash(matches[3]))
}
modDir, modName := r.modInfo(dir)
result := directoryPackageInfo{
status: directoryScanned,
dir: dir,
rootType: root.Type,
nonCanonicalImportPath: importPath,
moduleDir: modDir,
moduleName: modName,
}
if root.Type == gopathwalk.RootGOROOT {
// stdlib packages are always in scope, despite the confusing go.mod
return result
}
return result
}
// modCacheRegexp splits a path in a module cache into module, module version, and package.
var modCacheRegexp = regexp.MustCompile(`(.*)@([^/\\]*)(.*)`)
var (
slashSlash = []byte("//")
moduleStr = []byte("module")
)
// modulePath returns the module path from the gomod file text.
// If it cannot find a module path, it returns an empty string.
// It is tolerant of unrelated problems in the go.mod file.
//
// Copied from cmd/go/internal/modfile.
func modulePath(mod []byte) string {
for len(mod) > 0 {
line := mod
mod = nil
if i := bytes.IndexByte(line, '\n'); i >= 0 {
line, mod = line[:i], line[i+1:]
}
if i := bytes.Index(line, slashSlash); i >= 0 {
line = line[:i]
}
line = bytes.TrimSpace(line)
if !bytes.HasPrefix(line, moduleStr) {
continue
}
line = line[len(moduleStr):]
n := len(line)
line = bytes.TrimSpace(line)
if len(line) == n || len(line) == 0 {
continue
}
if line[0] == '"' || line[0] == '`' {
p, err := strconv.Unquote(string(line))
if err != nil {
return "" // malformed quoted string or multiline module path
}
return p
}
return string(line)
}
return "" // missing module path
}

236
vendor/golang.org/x/tools/internal/imports/mod_cache.go generated vendored Normal file
View File

@@ -0,0 +1,236 @@
// Copyright 2019 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package imports
import (
"context"
"fmt"
"sync"
"golang.org/x/tools/internal/gopathwalk"
)
// To find packages to import, the resolver needs to know about all of
// the packages that could be imported. This includes packages that are
// already in modules that are in (1) the current module, (2) replace targets,
// and (3) packages in the module cache. Packages in (1) and (2) may change over
// time, as the client may edit the current module and locally replaced modules.
// The module cache (which includes all of the packages in (3)) can only
// ever be added to.
//
// The resolver can thus save state about packages in the module cache
// and guarantee that this will not change over time. To obtain information
// about new modules added to the module cache, the module cache should be
// rescanned.
//
// It is OK to serve information about modules that have been deleted,
// as they do still exist.
// TODO(suzmue): can we share information with the caller about
// what module needs to be downloaded to import this package?
type directoryPackageStatus int
const (
_ directoryPackageStatus = iota
directoryScanned
nameLoaded
exportsLoaded
)
type directoryPackageInfo struct {
// status indicates the extent to which this struct has been filled in.
status directoryPackageStatus
// err is non-nil when there was an error trying to reach status.
err error
// Set when status >= directoryScanned.
// dir is the absolute directory of this package.
dir string
rootType gopathwalk.RootType
// nonCanonicalImportPath is the package's expected import path. It may
// not actually be importable at that path.
nonCanonicalImportPath string
// Module-related information.
moduleDir string // The directory that is the module root of this dir.
moduleName string // The module name that contains this dir.
// Set when status >= nameLoaded.
packageName string // the package name, as declared in the source.
// Set when status >= exportsLoaded.
exports []string
}
// reachedStatus returns true when info has a status at least target and any error associated with
// an attempt to reach target.
func (info *directoryPackageInfo) reachedStatus(target directoryPackageStatus) (bool, error) {
if info.err == nil {
return info.status >= target, nil
}
if info.status == target {
return true, info.err
}
return true, nil
}
// dirInfoCache is a concurrency safe map for storing information about
// directories that may contain packages.
//
// The information in this cache is built incrementally. Entries are initialized in scan.
// No new keys should be added in any other functions, as all directories containing
// packages are identified in scan.
//
// Other functions, including loadExports and findPackage, may update entries in this cache
// as they discover new things about the directory.
//
// The information in the cache is not expected to change for the cache's
// lifetime, so there is no protection against competing writes. Users should
// take care not to hold the cache across changes to the underlying files.
//
// TODO(suzmue): consider other concurrency strategies and data structures (RWLocks, sync.Map, etc)
type dirInfoCache struct {
mu sync.Mutex
// dirs stores information about packages in directories, keyed by absolute path.
dirs map[string]*directoryPackageInfo
listeners map[*int]cacheListener
}
type cacheListener func(directoryPackageInfo)
// ScanAndListen calls listener on all the items in the cache, and on anything
// newly added. The returned stop function waits for all in-flight callbacks to
// finish and blocks new ones.
func (d *dirInfoCache) ScanAndListen(ctx context.Context, listener cacheListener) func() {
ctx, cancel := context.WithCancel(ctx)
// Flushing out all the callbacks is tricky without knowing how many there
// are going to be. Setting an arbitrary limit makes it much easier.
const maxInFlight = 10
sema := make(chan struct{}, maxInFlight)
for i := 0; i < maxInFlight; i++ {
sema <- struct{}{}
}
cookie := new(int) // A unique ID we can use for the listener.
// We can't hold mu while calling the listener.
d.mu.Lock()
var keys []string
for key := range d.dirs {
keys = append(keys, key)
}
d.listeners[cookie] = func(info directoryPackageInfo) {
select {
case <-ctx.Done():
return
case <-sema:
}
listener(info)
sema <- struct{}{}
}
d.mu.Unlock()
stop := func() {
cancel()
d.mu.Lock()
delete(d.listeners, cookie)
d.mu.Unlock()
for i := 0; i < maxInFlight; i++ {
<-sema
}
}
// Process the pre-existing keys.
for _, k := range keys {
select {
case <-ctx.Done():
return stop
default:
}
if v, ok := d.Load(k); ok {
listener(v)
}
}
return stop
}
// Store stores the package info for dir.
func (d *dirInfoCache) Store(dir string, info directoryPackageInfo) {
d.mu.Lock()
_, old := d.dirs[dir]
d.dirs[dir] = &info
var listeners []cacheListener
for _, l := range d.listeners {
listeners = append(listeners, l)
}
d.mu.Unlock()
if !old {
for _, l := range listeners {
l(info)
}
}
}
// Load returns a copy of the directoryPackageInfo for absolute directory dir.
func (d *dirInfoCache) Load(dir string) (directoryPackageInfo, bool) {
d.mu.Lock()
defer d.mu.Unlock()
info, ok := d.dirs[dir]
if !ok {
return directoryPackageInfo{}, false
}
return *info, true
}
// Keys returns the keys currently present in d.
func (d *dirInfoCache) Keys() (keys []string) {
d.mu.Lock()
defer d.mu.Unlock()
for key := range d.dirs {
keys = append(keys, key)
}
return keys
}
func (d *dirInfoCache) CachePackageName(info directoryPackageInfo) (string, error) {
if loaded, err := info.reachedStatus(nameLoaded); loaded {
return info.packageName, err
}
if scanned, err := info.reachedStatus(directoryScanned); !scanned || err != nil {
return "", fmt.Errorf("cannot read package name, scan error: %v", err)
}
info.packageName, info.err = packageDirToName(info.dir)
info.status = nameLoaded
d.Store(info.dir, info)
return info.packageName, info.err
}
func (d *dirInfoCache) CacheExports(ctx context.Context, env *ProcessEnv, info directoryPackageInfo) (string, []string, error) {
if reached, _ := info.reachedStatus(exportsLoaded); reached {
return info.packageName, info.exports, info.err
}
if reached, err := info.reachedStatus(nameLoaded); reached && err != nil {
return "", nil, err
}
info.packageName, info.exports, info.err = loadExportsFromFiles(ctx, env, info.dir, false)
if info.err == context.Canceled || info.err == context.DeadlineExceeded {
return info.packageName, info.exports, info.err
}
// The cache structure wants things to proceed linearly. We can skip a
// step here, but only if we succeed.
if info.status == nameLoaded || info.err == nil {
info.status = exportsLoaded
} else {
info.status = nameLoaded
}
d.Store(info.dir, info)
return info.packageName, info.exports, info.err
}

297
vendor/golang.org/x/tools/internal/imports/sortimports.go generated vendored Normal file
View File

@@ -0,0 +1,297 @@
// Copyright 2013 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Hacked up copy of go/ast/import.go
// Modified to use a single token.File in preference to a FileSet.
package imports
import (
"go/ast"
"go/token"
"log"
"sort"
"strconv"
)
// sortImports sorts runs of consecutive import lines in import blocks in f.
// It also removes duplicate imports when it is possible to do so without data loss.
//
// It may mutate the token.File.
func sortImports(localPrefix string, tokFile *token.File, f *ast.File) {
for i, d := range f.Decls {
d, ok := d.(*ast.GenDecl)
if !ok || d.Tok != token.IMPORT {
// Not an import declaration, so we're done.
// Imports are always first.
break
}
if len(d.Specs) == 0 {
// Empty import block, remove it.
f.Decls = append(f.Decls[:i], f.Decls[i+1:]...)
}
if !d.Lparen.IsValid() {
// Not a block: sorted by default.
continue
}
// Identify and sort runs of specs on successive lines.
i := 0
specs := d.Specs[:0]
for j, s := range d.Specs {
if j > i && tokFile.Line(s.Pos()) > 1+tokFile.Line(d.Specs[j-1].End()) {
// j begins a new run. End this one.
specs = append(specs, sortSpecs(localPrefix, tokFile, f, d.Specs[i:j])...)
i = j
}
}
specs = append(specs, sortSpecs(localPrefix, tokFile, f, d.Specs[i:])...)
d.Specs = specs
// Deduping can leave a blank line before the rparen; clean that up.
// Ignore line directives.
if len(d.Specs) > 0 {
lastSpec := d.Specs[len(d.Specs)-1]
lastLine := tokFile.PositionFor(lastSpec.Pos(), false).Line
if rParenLine := tokFile.PositionFor(d.Rparen, false).Line; rParenLine > lastLine+1 {
tokFile.MergeLine(rParenLine - 1) // has side effects!
}
}
}
}
// mergeImports merges all the import declarations into the first one.
// Taken from golang.org/x/tools/ast/astutil.
// This does not adjust line numbers properly
func mergeImports(f *ast.File) {
if len(f.Decls) <= 1 {
return
}
// Merge all the import declarations into the first one.
var first *ast.GenDecl
for i := 0; i < len(f.Decls); i++ {
decl := f.Decls[i]
gen, ok := decl.(*ast.GenDecl)
if !ok || gen.Tok != token.IMPORT || declImports(gen, "C") {
continue
}
if first == nil {
first = gen
continue // Don't touch the first one.
}
// We now know there is more than one package in this import
// declaration. Ensure that it ends up parenthesized.
first.Lparen = first.Pos()
// Move the imports of the other import declaration to the first one.
for _, spec := range gen.Specs {
spec.(*ast.ImportSpec).Path.ValuePos = first.Pos()
first.Specs = append(first.Specs, spec)
}
f.Decls = append(f.Decls[:i], f.Decls[i+1:]...)
i--
}
}
// declImports reports whether gen contains an import of path.
// Taken from golang.org/x/tools/ast/astutil.
func declImports(gen *ast.GenDecl, path string) bool {
if gen.Tok != token.IMPORT {
return false
}
for _, spec := range gen.Specs {
impspec := spec.(*ast.ImportSpec)
if importPath(impspec) == path {
return true
}
}
return false
}
func importPath(s ast.Spec) string {
t, err := strconv.Unquote(s.(*ast.ImportSpec).Path.Value)
if err == nil {
return t
}
return ""
}
func importName(s ast.Spec) string {
n := s.(*ast.ImportSpec).Name
if n == nil {
return ""
}
return n.Name
}
func importComment(s ast.Spec) string {
c := s.(*ast.ImportSpec).Comment
if c == nil {
return ""
}
return c.Text()
}
// collapse indicates whether prev may be removed, leaving only next.
func collapse(prev, next ast.Spec) bool {
if importPath(next) != importPath(prev) || importName(next) != importName(prev) {
return false
}
return prev.(*ast.ImportSpec).Comment == nil
}
type posSpan struct {
Start token.Pos
End token.Pos
}
// sortSpecs sorts the import specs within each import decl.
// It may mutate the token.File.
func sortSpecs(localPrefix string, tokFile *token.File, f *ast.File, specs []ast.Spec) []ast.Spec {
// Can't short-circuit here even if specs are already sorted,
// since they might yet need deduplication.
// A lone import, however, may be safely ignored.
if len(specs) <= 1 {
return specs
}
// Record positions for specs.
pos := make([]posSpan, len(specs))
for i, s := range specs {
pos[i] = posSpan{s.Pos(), s.End()}
}
// Identify comments in this range.
// Any comment from pos[0].Start to the final line counts.
lastLine := tokFile.Line(pos[len(pos)-1].End)
cstart := len(f.Comments)
cend := len(f.Comments)
for i, g := range f.Comments {
if g.Pos() < pos[0].Start {
continue
}
if i < cstart {
cstart = i
}
if tokFile.Line(g.End()) > lastLine {
cend = i
break
}
}
comments := f.Comments[cstart:cend]
// Assign each comment to the import spec preceding it.
importComment := map[*ast.ImportSpec][]*ast.CommentGroup{}
specIndex := 0
for _, g := range comments {
for specIndex+1 < len(specs) && pos[specIndex+1].Start <= g.Pos() {
specIndex++
}
s := specs[specIndex].(*ast.ImportSpec)
importComment[s] = append(importComment[s], g)
}
// Sort the import specs by import path.
// Remove duplicates, when possible without data loss.
// Reassign the import paths to have the same position sequence.
// Reassign each comment to abut the end of its spec.
// Sort the comments by new position.
sort.Sort(byImportSpec{localPrefix, specs})
// Dedup. Thanks to our sorting, we can just consider
// adjacent pairs of imports.
deduped := specs[:0]
for i, s := range specs {
if i == len(specs)-1 || !collapse(s, specs[i+1]) {
deduped = append(deduped, s)
} else {
p := s.Pos()
tokFile.MergeLine(tokFile.Line(p)) // has side effects!
}
}
specs = deduped
// Fix up comment positions
for i, s := range specs {
s := s.(*ast.ImportSpec)
if s.Name != nil {
s.Name.NamePos = pos[i].Start
}
s.Path.ValuePos = pos[i].Start
s.EndPos = pos[i].End
nextSpecPos := pos[i].End
for _, g := range importComment[s] {
for _, c := range g.List {
c.Slash = pos[i].End
nextSpecPos = c.End()
}
}
if i < len(specs)-1 {
pos[i+1].Start = nextSpecPos
pos[i+1].End = nextSpecPos
}
}
sort.Sort(byCommentPos(comments))
// Fixup comments can insert blank lines, because import specs are on different lines.
// We remove those blank lines here by merging import spec to the first import spec line.
firstSpecLine := tokFile.Line(specs[0].Pos())
for _, s := range specs[1:] {
p := s.Pos()
line := tokFile.Line(p)
for previousLine := line - 1; previousLine >= firstSpecLine; {
// MergeLine can panic. Avoid the panic at the cost of not removing the blank line
// golang/go#50329
if previousLine > 0 && previousLine < tokFile.LineCount() {
tokFile.MergeLine(previousLine) // has side effects!
previousLine--
} else {
// try to gather some data to diagnose how this could happen
req := "Please report what the imports section of your go file looked like."
log.Printf("panic avoided: first:%d line:%d previous:%d max:%d. %s",
firstSpecLine, line, previousLine, tokFile.LineCount(), req)
}
}
}
return specs
}
type byImportSpec struct {
localPrefix string
specs []ast.Spec // slice of *ast.ImportSpec
}
func (x byImportSpec) Len() int { return len(x.specs) }
func (x byImportSpec) Swap(i, j int) { x.specs[i], x.specs[j] = x.specs[j], x.specs[i] }
func (x byImportSpec) Less(i, j int) bool {
ipath := importPath(x.specs[i])
jpath := importPath(x.specs[j])
igroup := importGroup(x.localPrefix, ipath)
jgroup := importGroup(x.localPrefix, jpath)
if igroup != jgroup {
return igroup < jgroup
}
if ipath != jpath {
return ipath < jpath
}
iname := importName(x.specs[i])
jname := importName(x.specs[j])
if iname != jname {
return iname < jname
}
return importComment(x.specs[i]) < importComment(x.specs[j])
}
type byCommentPos []*ast.CommentGroup
func (x byCommentPos) Len() int { return len(x) }
func (x byCommentPos) Swap(i, j int) { x[i], x[j] = x[j], x[i] }
func (x byCommentPos) Less(i, j int) bool { return x[i].Pos() < x[j].Pos() }

11115
vendor/golang.org/x/tools/internal/imports/zstdlib.go generated vendored Normal file
View File

File diff suppressed because it is too large Load Diff

15
vendor/modules.txt vendored
View File

@@ -78,6 +78,9 @@ github.com/alexedwards/argon2id
# github.com/amoghe/go-crypt v0.0.0-20220222110647-20eada5f5964
## explicit
github.com/amoghe/go-crypt
# github.com/araddon/dateparse v0.0.0-20210429162001-6b43995a97de
## explicit; go 1.12
github.com/araddon/dateparse
# github.com/armon/go-metrics v0.4.1
## explicit; go 1.12
github.com/armon/go-metrics
@@ -1321,6 +1324,11 @@ github.com/mitchellh/mapstructure
# github.com/mitchellh/reflectwalk v1.0.2
## explicit
github.com/mitchellh/reflectwalk
# github.com/mna/pigeon v1.1.0
## explicit
github.com/mna/pigeon
github.com/mna/pigeon/ast
github.com/mna/pigeon/builder
# github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
## explicit
github.com/modern-go/concurrent
@@ -1958,6 +1966,8 @@ golang.org/x/image/tiff/lzw
golang.org/x/image/vector
# golang.org/x/mod v0.12.0
## explicit; go 1.17
golang.org/x/mod/internal/lazyregexp
golang.org/x/mod/module
golang.org/x/mod/semver
# golang.org/x/net v0.15.0
## explicit; go 1.17
@@ -2037,18 +2047,23 @@ golang.org/x/time/rate
# golang.org/x/tools v0.12.0
## explicit; go 1.18
golang.org/x/tools/cmd/stringer
golang.org/x/tools/go/ast/astutil
golang.org/x/tools/go/ast/inspector
golang.org/x/tools/go/gcexportdata
golang.org/x/tools/go/internal/packagesdriver
golang.org/x/tools/go/packages
golang.org/x/tools/go/types/objectpath
golang.org/x/tools/imports
golang.org/x/tools/internal/event
golang.org/x/tools/internal/event/core
golang.org/x/tools/internal/event/keys
golang.org/x/tools/internal/event/label
golang.org/x/tools/internal/event/tag
golang.org/x/tools/internal/fastwalk
golang.org/x/tools/internal/gcimporter
golang.org/x/tools/internal/gocommand
golang.org/x/tools/internal/gopathwalk
golang.org/x/tools/internal/imports
golang.org/x/tools/internal/packagesinternal
golang.org/x/tools/internal/pkgbits
golang.org/x/tools/internal/tokeninternal