v2.3.0

Add ability to globally set environment variables

CHANGELOG.md

@@ -0,0 +1,151 @@
+# Changelog
+
+## v2.3.0 - 2019-01-02
+
+- On Windows, Task can now be installed using [Scoop](https://scoop.sh/)
+  ([#152](https://github.com/go-task/task/pull/152));
+- Fixed issue with file/directory globing
+  ([#153](https://github.com/go-task/task/issues/153));
+- Added ability to globally set environment variables
+  (
+    [#138](https://github.com/go-task/task/pull/138),
+    [#159](https://github.com/go-task/task/pull/159)
+  ).
+
+## v2.2.1 - 2018-12-09
+
+- This repository now uses Go Modules (#143). We'll still keep the `vendor` directory in sync for some time, though;
+- Fixing a bug when the Taskfile has no tasks but includes another Taskfile (#150);
+- Fix a bug when calling another task or a dependency in an included Taskfile (#151).
+
+## v2.2.0 - 2018-10-25
+
+- Added support for [including other Taskfiles](https://taskfile.org/#/usage?id=including-other-taskfiles) (#98)
+  - This should be considered experimental. For now, only including local files is supported, but support for including remote Taskfiles is being discussed. If you have any feedback, please comment on #98.
+- Task now have a dedicated documentation site: https://taskfile.org
+  - Thanks to [Docsify](https://docsify.js.org/) for making this pretty easy. To check the source code, just take a look at the [docs](https://github.com/go-task/task/tree/master/docs) directory of this repository. Contributions to the documentation is really appreciated.
+
+## v2.1.1 - 2018-09-17
+
+- Fix suggestion to use `task --init` not being shown anymore (when a `Taskfile.yml` is not found)
+- Fix error when using checksum method and no file exists for a source glob (#131)
+- Fix signal handling when the `--watch` flag is given (#132)
+
+## v2.1.0 - 2018-08-19
+
+- Add a `ignore_error` option to task and command (#123)
+- Add a dry run mode (`--dry` flag) (#126)
+
+## v2.0.3 - 2018-06-24
+
+- Expand environment variables on "dir", "sources" and "generates" (#116)
+- Fix YAML merging syntax (#112)
+- Add ZSH completion (#111)
+- Implement new `output` option. Please check out the [documentation](https://github.com/go-task/task#output-syntax)
+
+## v2.0.2 - 2018-05-01
+
+- Fix merging of YAML anchors (#112)
+
+## v2.0.1 - 2018-03-11
+
+- Fixes panic on `task --list`
+
+## v2.0.0 - 2018-03-08
+
+Version 2.0.0 is here, with a new Taskfile format.
+
+Please, make sure to read the [Taskfile versions](https://github.com/go-task/task/blob/master/TASKFILE_VERSIONS.md) document, since it describes in depth what changed for this version.
+
+* New Taskfile version 2 (https://github.com/go-task/task/issues/77)
+* Possibility to have global variables in the `Taskfile.yml` instead of `Taskvars.yml` (https://github.com/go-task/task/issues/66)
+* Small improvements and fixes
+
+## v1.4.4 - 2017-11-19
+
+- Handle SIGINT and SIGTERM (#75);
+- List: print message with there's no task with description;
+- Expand home dir ("~" symbol) on paths (#74);
+- Add Snap as an installation method;
+- Move examples to its own repo;
+- Watch: also walk on tasks called on on "cmds", and not only on "deps";
+- Print logs to stderr instead of stdout (#68);
+- Remove deprecated `set` keyword;
+- Add checksum based status check, alternative to timestamp based.
+
+## v1.4.3 - 2017-09-07
+
+- Allow assigning variables to tasks at run time via CLI (#33)
+- Added suport for multiline variables from sh (#64)
+- Fixes env: remove square braces and evaluate shell (#62)
+- Watch: change watch library and few fixes and improvements
+- When use watching, cancel and restart long running process on file change (#59 and #60)
+
+## v1.4.2 - 2017-07-30
+
+- Flag to set directory of execution
+- Always echo command if is verbose mode
+- Add silent mode to disable echoing of commands
+- Fixes and improvements of variables (#56)
+
+## v1.4.1 - 2017-07-15
+
+- Allow use of YAML for dynamic variables instead of $ prefix
+  - `VAR: {sh: echo Hello}` instead of `VAR: $echo Hello`
+- Add `--list` (or `-l`) flag to print existing tasks
+- OS specific Taskvars file (e.g. `Taskvars_windows.yml`, `Taskvars_linux.yml`, etc)
+- Consider task up-to-date on equal timestamps (#49)
+- Allow absolute path in generates section (#48)
+- Bugfix: allow templating when calling deps (#42)
+- Fix panic for invalid task in cyclic dep detection
+- Better error output for dynamic variables in Taskvars.yml (#41)
+- Allow template evaluation in parameters
+
+## v1.4.0 - 2017-07-06
+
+- Cache dynamic variables
+- Add verbose mode (`-v` flag)
+- Support to task parameters (overriding vars) (#31) (#32)
+- Print command, also when "set:" is specified (#35)
+- Improve task command help text (#35)
+
+## v1.3.1 - 2017-06-14
+
+- Fix glob not working on commands (#28)
+- Add ExeExt template function
+- Add `--init` flag to create a new Taskfile
+- Add status option to prevent task from running (#27)
+- Allow interpolation on `generates` and `sources` attributes (#26)
+
+## v1.3.0 - 2017-04-24
+
+- Migrate from os/exec.Cmd to a native Go sh/bash interpreter
+  - This is a potentially breaking change if you use Windows.
+  - Now, `cmd` is not used anymore on Windows. Always use Bash-like syntax for your commands, even on Windows.
+- Add "ToSlash" and "FromSlash" to template functions
+- Use functions defined on github.com/Masterminds/sprig
+- Do not redirect stdin while running variables commands
+- Using `context` and `errgroup` packages (this will make other tasks to be cancelled, if one returned an error)
+
+## v1.2.0 - 2017-04-02
+
+- More tests and Travis integration
+- Watch a task (experimental)
+- Possibility to call another task
+- Fix "=" not being reconized in variables/environment variables
+- Tasks can now have a description, and help will print them (#10)
+- Task dependencies now run concurrently
+- Support for a default task (#16)
+
+## v1.1.0 - 2017-03-08
+
+- Support for YAML, TOML and JSON (#1)
+- Support running command in another directory (#4)
+- `--force` or `-f` flag to force execution of task even when it's up-to-date
+- Detection of cyclic dependencies (#5)
+- Support for variables (#6, #9, #14)
+- Operation System specific commands and variables (#13)
+
+## v1.0.0 - 2017-02-28
+
+- Add LICENSE file

README.md

@@ -3,10 +3,20 @@
 # Task
 
 Task is a task runner / build tool that aims to be simpler and easier to use
-than, for example, [GNU Make][make].
+than, for example, [GNU Make](https://www.gnu.org/software/make/).
-
----
 
 See [taskfile.org](https://taskfile.org) for documentation.
 
-[make]: https://www.gnu.org/software/make/
+---
+
+## Sponsors
+
+[![Sponsors](https://opencollective.com/task/sponsors.svg?width=890)](https://opencollective.com/task)
+
+## Backers
+
+[![Backers](https://opencollective.com/task/backers.svg?width=890)](https://opencollective.com/task)
+
+## Contributors
+
+[![Contributors](https://opencollective.com/task/contributors.svg?width=890)](https://github.com/go-task/task/graphs/contributors)

Taskfile.yml

@@ -1,5 +1,8 @@
 version: '2'
 
+includes:
+  docs: ./docs
+
 vars:
   GIT_COMMIT:
     sh: git log -n 1 --format=%h
@@ -83,13 +86,3 @@ tasks:
     cmds:
       - echo '{{.GO_PACKAGES}}'
     silent: true
-
-  docs:install:
-    desc: Installs docsify to work the on the documentation site
-    cmds:
-      - npm install docsify-cli -g
-
-  docs:serve:
-    desc: Serves the documentation site locally
-    cmds:
-      - docsify serve docs

docs/README.md

@@ -28,8 +28,8 @@ guide to check the full schema documentation and Task features.
 ## Features
 
 - [Easy installation](installation.md): just download a single binary, add to
-  $PATH and you're done! Or you can also install using [Homebrew][homebrew] or
+  $PATH and you're done! Or you can also install using [Homebrew][homebrew],
-  [Snapcraft][snapcraft] if you want;
+  [Snapcraft][snapcraft], or [Scoop][scoop] if you want;
 - Available on CIs: by adding [this simple command](installation.md#install-script)
   to install on your CI script and you're done to use Task as part of your CI pipeline;
 - Truly cross-platform: while most build tools only work well on Linux or macOS,
@@ -38,9 +38,22 @@ guide to check the full schema documentation and Task features.
   if a given set of files haven't changed since last run (based either on its
   timestamp or content).
 
+## Sponsors
+
+[![Sponsors](https://opencollective.com/task/sponsors.svg?width=890)](https://opencollective.com/task)
+
+## Backers
+
+[![Backers](https://opencollective.com/task/backers.svg?width=890)](https://opencollective.com/task)
+
+## Contributors
+
+[![Contributors](https://opencollective.com/task/contributors.svg?width=890)](https://github.com/go-task/task/graphs/contributors)
+
 [make]: https://www.gnu.org/software/make/
 [go]: https://golang.org/
 [yaml]: http://yaml.org/
 [homebrew]: https://brew.sh/
 [snapcraft]: https://snapcraft.io/
+[scoop]: https://scoop.sh/
 [sh]: https://mvdan.cc/sh

docs/Taskfile.yml

@@ -0,0 +1,12 @@
+version: '2'
+
+tasks:
+  install:
+    desc: Installs docsify to work the on the documentation site
+    cmds:
+      - npm install docsify-cli -g
+
+  serve:
+    desc: Serves the documentation site locally
+    cmds:
+      - docsify serve docs

docs/_sidebar.md

@@ -4,5 +4,4 @@
 - [Examples](examples.md)
 - [Releasing Task](releasing_task.md)
 - [Alternative Task Runners](alternative_task_runners.md)
-- [Sponsors and Backers](sponsors_and_backers.md)
 - [![Github](https://icongram.jgog.in/simple/github.svg?color=808080&size=16)Github](https://github.com/go-task/task)

docs/installation.md

@@ -25,6 +25,19 @@ right:
 sudo snap install task
 ```
 
+## Scoop
+
+If you're on Windows and have [Scoop][scoop] installed, use `extras` bucket
+to install Task like:
+
+```cmd
+scoop bucket add extras
+scoop install task
+```
+
+This installation method is community owned. After a new release of Task, it
+may take some time until it's available on Scoop.
+
 ## Go
 
 Task now uses [Go Modules](https://github.com/golang/go/wiki/Modules), which
@@ -59,7 +72,7 @@ Both methods requires having the [Go][go] environment properly setup locally.
 ## Install script
 
 We also have a [install script][installscript], which is very useful on
-scanarios like CIs. Many thanks to [godownloader][godownloader] for allowing
+scenarios like CIs. Many thanks to [godownloader][godownloader] for allowing
 easily generating this script.
 
 ```bash
@@ -74,3 +87,4 @@ curl -s https://taskfile.org/install.sh | sh
 [installscript]: https://github.com/go-task/task/blob/master/install-task.sh
 [releases]: https://github.com/go-task/task/releases
 [godownloader]: https://github.com/goreleaser/godownloader
+[scoop]: https://scoop.sh/

docs/releasing_task.md

@@ -1,6 +1,6 @@
 # Releasing Task
 
-The release process of Task is done is done with the help of
+The release process of Task is done with the help of
 [GoReleaser][goreleaser]. You can test the release process locally by calling
 the `test-release` task of the Taskfile.
 
@@ -24,6 +24,13 @@ the binaries:
 * Moving both `i386` and `amd64` new artifacts to the stable channel on
 the [Snapscraft dashboard][snapcraftdashboard]
 
+# Scoop
+
+Scoop is a community owned installation method. Scoop owners usually take care
+of updating versions there by editing
+[this file](https://github.com/lukesampson/scoop-extras/blob/master/task.json).
+If you think its Task version is outdated, open an issue to let us know.
+
 [goreleaser]: https://goreleaser.com/#continuous_integration
 [homebrewtap]: https://github.com/go-task/homebrew-tap
 [gotaskrb]: https://github.com/go-task/homebrew-tap/blob/master/Formula/go-task.rb

docs/sponsors_and_backers.md

@@ -1,16 +0,0 @@
-# Sponsors and Backers
-
-## Sponsors
-
-[![Sponsors](https://opencollective.com/task/sponsors.svg?width=890)][opencollective]
-
-## Backers
-
-[![Backers](https://opencollective.com/task/backers.svg?width=890)][opencollective]
-
-## Contributors
-
-[![Contributors](https://opencollective.com/task/contributors.svg?width=890)][contributors]
-
-[opencollective]: https://opencollective.com/task
-[contributors]: https://github.com/go-task/task/graphs/contributors

docs/usage.md

@@ -31,23 +31,41 @@ interpreter. So you can write sh/bash commands and it will work even on
 Windows, where `sh` or `bash` are usually not available. Just remember any
 executable called must be available by the OS or in PATH.
 
-If you ommit a task name, "default" will be assumed.
+If you omit a task name, "default" will be assumed.
 
 ## Environment
 
-You can specify environment variables that are added when running a command:
+You can use `env` to set custom environment variables for a specific task:
 
 ```yaml
 version: '2'
 
 tasks:
-  build:
+  greet:
     cmds:
-      - echo $hallo
+      - echo $GREETING
     env:
-      hallo: welt
+      GREETING: Hey, there!
 ```
 
+Additionally, you can set globally environment variables, that'll be available
+to all tasks:
+
+```yaml
+version: '2'
+
+env:
+  GREETING: Hey, there!
+
+tasks:
+  greet:
+    cmds:
+      - echo $GREETING
+```
+
+> NOTE: `env` supports expansion and and retrieving output from a shell command
+> just like variables, as you can see on the [Variables](#variables) section.
+
 ## Operating System specific tasks
 
 If you add a `Taskfile_{{GOOS}}.yml` you can override or amend your Taskfile
@@ -83,7 +101,7 @@ Keep in mind that the version of the files should match. Also, when redefining
 a task the whole task is replaced, properties of the task are not merged.
 
 It's also possible to have an OS specific `Taskvars.yml` file, like
-`Taskvars_windows.yml`, `Taskfile_linux.yml`, or `Taskvars_darwin.yml`. See the
+`Taskvars_windows.yml`, `Taskvars_linux.yml`, or `Taskvars_darwin.yml`. See the
 [variables section](#variables) below.
 
 ## Including other Taskfiles
@@ -455,7 +473,7 @@ Task also adds the following functions:
 - `catLines`: Replaces Unix (\n) and Windows (\r\n) styled newlines with a space.
 - `toSlash`: Does nothing on Unix, but on Windows converts a string from `\`
   path format to `/`.
-- `fromSlash`: Oposite of `toSlash`. Does nothing on Unix, but on Windows
+- `fromSlash`: Opposite of `toSlash`. Does nothing on Unix, but on Windows
   converts a string from `\` path format to `/`.
 - `exeExt`: Returns the right executable extension for the current OS
   (`".exe"` for Windows, `""` for others).
@@ -488,7 +506,7 @@ tasks:
 ## Help
 
 Running `task --list` (or `task -l`) lists all tasks with a description.
-The following taskfile:
+The following Taskfile:
 
 ```yaml
 version: '2'
@@ -575,7 +593,7 @@ tasks:
 
 * Or globally with `--silent` or `-s` flag
 
-If you want to suppress stdout instead, just redirect a command to `/dev/null`:
+If you want to suppress STDOUT instead, just redirect a command to `/dev/null`:
 
 ```yaml
 version: '2'

go.mod

@@ -20,5 +20,5 @@ require (
 	golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f
 	golang.org/x/sys v0.0.0-20180831094639-fa5fdf94c789 // indirect
 	gopkg.in/yaml.v2 v2.2.1
-	mvdan.cc/sh v0.0.0-20180829163519-3a244a89e2e5
+	mvdan.cc/sh v2.6.3-0.20181216173157-8aeb0734cd0f+incompatible
 )

go.sum

@@ -41,5 +41,5 @@ gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/yaml.v2 v2.2.1 h1:mUhvW9EsL+naU5Q3cakzfE91YhliOondGd6ZrsDBHQE=
 gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
-mvdan.cc/sh v0.0.0-20180829163519-3a244a89e2e5 h1:FKi9XtQO5aNipfQ/qnnLCoM6gdFwPQY702RRbNRxjK8=
+mvdan.cc/sh v2.6.3-0.20181216173157-8aeb0734cd0f+incompatible h1:jf0jjqiqwKXdH3JBKY+K3tFXGtUQZr/pFIO+cn0tQCc=
-mvdan.cc/sh v0.0.0-20180829163519-3a244a89e2e5/go.mod h1:IeeQbZq+x2SUGBensq/jge5lLQbS3XT2ktyp3wrt4x8=
+mvdan.cc/sh v2.6.3-0.20181216173157-8aeb0734cd0f+incompatible/go.mod h1:IeeQbZq+x2SUGBensq/jge5lLQbS3XT2ktyp3wrt4x8=

internal/execext/exec.go

@@ -7,7 +7,9 @@ import (
 	"os"
 	"strings"
 
+	"mvdan.cc/sh/expand"
 	"mvdan.cc/sh/interp"
+	"mvdan.cc/sh/shell"
 	"mvdan.cc/sh/syntax"
 )
 
@@ -41,14 +43,10 @@ func RunCommand(ctx context.Context, opts *RunCommandOptions) error {
 	if len(environ) == 0 {
 		environ = os.Environ()
 	}
-	env, err := interp.EnvFromList(environ)
-	if err != nil {
-		return err
-	}
 
 	r, err := interp.New(
 		interp.Dir(opts.Dir),
-		interp.Env(env),
+		interp.Env(expand.ListEnviron(environ...)),
 
 		interp.Module(interp.DefaultExec),
 		interp.Module(interp.OpenDevImpls(interp.DefaultOpen)),
@@ -70,3 +68,16 @@ func IsExitError(err error) bool {
 		return false
 	}
 }
+
+// Expand is a helper to mvdan.cc/shell.Fields that returns the first field
+// if available.
+func Expand(s string) (string, error) {
+	fields, err := shell.Fields(s, nil)
+	if err != nil {
+		return "", err
+	}
+	if len(fields) > 0 {
+		return fields[0], nil
+	}
+	return "", nil
+}

internal/status/glob.go

@@ -4,8 +4,9 @@ import (
 	"path/filepath"
 	"sort"
 
+	"github.com/go-task/task/v2/internal/execext"
+
 	"github.com/mattn/go-zglob"
-	"mvdan.cc/sh/shell"
 )
 
 func glob(dir string, globs []string) (files []string, err error) {
@@ -13,7 +14,7 @@ func glob(dir string, globs []string) (files []string, err error) {
 		if !filepath.IsAbs(g) {
 			g = filepath.Join(dir, g)
 		}
-		g, err = shell.Expand(g, nil)
+		g, err = execext.Expand(g)
 		if err != nil {
 			return nil, err
 		}

internal/taskfile/merge.go

@@ -35,6 +35,13 @@ func Merge(t1, t2 *Taskfile, namespaces ...string) error {
 		t1.Vars[k] = v
 	}
 
+	if t1.Env == nil {
+		t1.Env = make(Vars)
+	}
+	for k, v := range t2.Vars {
+		t1.Env[k] = v
+	}
+
 	if t1.Tasks == nil {
 		t1.Tasks = make(Tasks)
 	}

internal/taskfile/taskfile.go

@@ -7,6 +7,7 @@ type Taskfile struct {
 	Output     string
 	Includes   map[string]string
 	Vars       Vars
+	Env        Vars
 	Tasks      Tasks
 }
 
@@ -23,6 +24,7 @@ func (tf *Taskfile) UnmarshalYAML(unmarshal func(interface{}) error) error {
 		Output     string
 		Includes   map[string]string
 		Vars       Vars
+		Env        Vars
 		Tasks      Tasks
 	}
 	if err := unmarshal(&taskfile); err != nil {
@@ -33,6 +35,7 @@ func (tf *Taskfile) UnmarshalYAML(unmarshal func(interface{}) error) error {
 	tf.Output = taskfile.Output
 	tf.Includes = taskfile.Includes
 	tf.Vars = taskfile.Vars
+	tf.Env = taskfile.Env
 	tf.Tasks = taskfile.Tasks
 	if tf.Expansions <= 0 {
 		tf.Expansions = 2

task.go

@@ -268,9 +268,9 @@ func getEnviron(t *taskfile.Task) []string {
 		return nil
 	}
 
-	envs := os.Environ()
+	environ := os.Environ()
 	for k, v := range t.Env.ToStringMap() {
-		envs = append(envs, fmt.Sprintf("%s=%s", k, v))
+		environ = append(environ, fmt.Sprintf("%s=%s", k, v))
 	}
-	return envs
+	return environ
 }

task_test.go

@@ -61,7 +61,8 @@ func TestEnv(t *testing.T) {
 		Target:    "default",
 		TrimSpace: false,
 		Files: map[string]string{
-			"env.txt": "GOOS='linux' GOARCH='amd64' CGO_ENABLED='0'\n",
+			"local.txt":  "GOOS='linux' GOARCH='amd64' CGO_ENABLED='0'\n",
+			"global.txt": "FOO='foo' BAR='overriden' BAZ='baz'\n",
 		},
 	}
 	tt.Run(t)

testdata/env/.gitignore

@@ -1 +1 @@
-env.txt
+*.txt

testdata/env/Taskfile.yml

@@ -1,10 +1,33 @@
-default:
+version: '2'
-  vars:
+
-    AMD64: amd64
+vars:
-  env:
+  BAZ:
-    GOOS: linux
+    sh: echo baz
-    GOARCH: "{{.AMD64}}"
+
-    CGO_ENABLED:
+env:
-      sh: echo '0'
+  FOO: foo
-  cmds:
+  BAR: bar
-    - echo "GOOS='$GOOS' GOARCH='$GOARCH' CGO_ENABLED='$CGO_ENABLED'" > env.txt
+  BAZ: "{{.BAZ}}"
+
+tasks:
+  default:
+    cmds:
+      - task: local
+      - task: global
+
+  local:
+    vars:
+      AMD64: amd64
+    env:
+      GOOS: linux
+      GOARCH: "{{.AMD64}}"
+      CGO_ENABLED:
+        sh: echo '0'
+    cmds:
+      - echo "GOOS='$GOOS' GOARCH='$GOARCH' CGO_ENABLED='$CGO_ENABLED'" > local.txt
+
+  global:
+    env:
+      BAR: overriden
+    cmds:
+      - echo "FOO='$FOO' BAR='$BAR' BAZ='$BAZ'" > global.txt

variables.go

@@ -3,10 +3,9 @@ package task
 import (
 	"path/filepath"
 
+	"github.com/go-task/task/v2/internal/execext"
 	"github.com/go-task/task/v2/internal/taskfile"
 	"github.com/go-task/task/v2/internal/templater"
-
-	"mvdan.cc/sh/shell"
 )
 
 // CompiledTask returns a copy of a task, but replacing variables in almost all
@@ -31,13 +30,13 @@ func (e *Executor) CompiledTask(call taskfile.Call) (*taskfile.Task, error) {
 		Status:      r.ReplaceSlice(origTask.Status),
 		Dir:         r.Replace(origTask.Dir),
 		Vars:        nil,
-		Env:         r.ReplaceVars(origTask.Env),
+		Env:         nil,
 		Silent:      origTask.Silent,
 		Method:      r.Replace(origTask.Method),
 		Prefix:      r.Replace(origTask.Prefix),
 		IgnoreError: origTask.IgnoreError,
 	}
-	new.Dir, err = shell.Expand(new.Dir, nil)
+	new.Dir, err = execext.Expand(new.Dir)
 	if err != nil {
 		return nil, err
 	}
@@ -47,6 +46,14 @@ func (e *Executor) CompiledTask(call taskfile.Call) (*taskfile.Task, error) {
 	if new.Prefix == "" {
 		new.Prefix = new.Task
 	}
+
+	new.Env = make(taskfile.Vars, len(e.Taskfile.Env)+len(origTask.Env))
+	for k, v := range r.ReplaceVars(e.Taskfile.Env) {
+		new.Env[k] = v
+	}
+	for k, v := range r.ReplaceVars(origTask.Env) {
+		new.Env[k] = v
+	}
 	for k, v := range new.Env {
 		static, err := e.Compiler.HandleDynamicVar(v)
 		if err != nil {

vendor/modules.txt

@@ -38,7 +38,8 @@ golang.org/x/sys/unix
 golang.org/x/sys/windows
 # gopkg.in/yaml.v2 v2.2.1
 gopkg.in/yaml.v2
-# mvdan.cc/sh v0.0.0-20180829163519-3a244a89e2e5
+# mvdan.cc/sh v2.6.3-0.20181216173157-8aeb0734cd0f+incompatible
-mvdan.cc/sh/shell
+mvdan.cc/sh/expand
 mvdan.cc/sh/interp
+mvdan.cc/sh/shell
 mvdan.cc/sh/syntax

vendor/mvdan.cc/sh/expand/arith.go

@@ -1,57 +1,66 @@
 // Copyright (c) 2017, Daniel Martí <mvdan@mvdan.cc>
 // See LICENSE for licensing information
 
-package interp
+package expand
 
 import (
-	"context"
 	"fmt"
 	"strconv"
 
 	"mvdan.cc/sh/syntax"
 )
 
-func (r *Runner) arithm(ctx context.Context, expr syntax.ArithmExpr) int {
+func Arithm(cfg *Config, expr syntax.ArithmExpr) (int, error) {
 	switch x := expr.(type) {
 	case *syntax.Word:
-		str := r.loneWord(ctx, x)
+		str, err := Literal(cfg, x)
+		if err != nil {
+			return 0, err
+		}
 		// recursively fetch vars
-		for str != "" {
+		i := 0
-			val := r.getVar(str)
+		for str != "" && syntax.ValidName(str) {
+			val := cfg.envGet(str)
 			if val == "" {
 				break
 			}
+			if i++; i >= maxNameRefDepth {
+				break
+			}
 			str = val
 		}
 		// default to 0
-		return atoi(str)
+		return atoi(str), nil
 	case *syntax.ParenArithm:
-		return r.arithm(ctx, x.X)
+		return Arithm(cfg, x.X)
 	case *syntax.UnaryArithm:
 		switch x.Op {
 		case syntax.Inc, syntax.Dec:
-			name := x.X.(*syntax.Word).Parts[0].(*syntax.Lit).Value
+			name := x.X.(*syntax.Word).Lit()
-			old := atoi(r.getVar(name))
+			old := atoi(cfg.envGet(name))
 			val := old
 			if x.Op == syntax.Inc {
 				val++
 			} else {
 				val--
 			}
-			r.setVarString(ctx, name, strconv.Itoa(val))
+			cfg.envSet(name, strconv.Itoa(val))
 			if x.Post {
-				return old
+				return old, nil
 			}
-			return val
+			return val, nil
+		}
+		val, err := Arithm(cfg, x.X)
+		if err != nil {
+			return 0, err
 		}
-		val := r.arithm(ctx, x.X)
 		switch x.Op {
 		case syntax.Not:
-			return oneIf(val == 0)
+			return oneIf(val == 0), nil
 		case syntax.Plus:
-			return val
+			return val, nil
 		default: // syntax.Minus
-			return -val
+			return -val, nil
 		}
 	case *syntax.BinaryArithm:
 		switch x.Op {
@@ -59,16 +68,27 @@ func (r *Runner) arithm(ctx context.Context, expr syntax.ArithmExpr) int {
 			syntax.MulAssgn, syntax.QuoAssgn, syntax.RemAssgn,
 			syntax.AndAssgn, syntax.OrAssgn, syntax.XorAssgn,
 			syntax.ShlAssgn, syntax.ShrAssgn:
-			return r.assgnArit(ctx, x)
+			return cfg.assgnArit(x)
 		case syntax.Quest: // Colon can't happen here
-			cond := r.arithm(ctx, x.X)
+			cond, err := Arithm(cfg, x.X)
+			if err != nil {
+				return 0, err
+			}
 			b2 := x.Y.(*syntax.BinaryArithm) // must have Op==Colon
 			if cond == 1 {
-				return r.arithm(ctx, b2.X)
+				return Arithm(cfg, b2.X)
 			}
-			return r.arithm(ctx, b2.Y)
+			return Arithm(cfg, b2.Y)
 		}
-		return binArit(x.Op, r.arithm(ctx, x.X), r.arithm(ctx, x.Y))
+		left, err := Arithm(cfg, x.X)
+		if err != nil {
+			return 0, err
+		}
+		right, err := Arithm(cfg, x.Y)
+		if err != nil {
+			return 0, err
+		}
+		return binArit(x.Op, left, right), nil
 	default:
 		panic(fmt.Sprintf("unexpected arithm expr: %T", x))
 	}
@@ -88,10 +108,13 @@ func atoi(s string) int {
 	return n
 }
 
-func (r *Runner) assgnArit(ctx context.Context, b *syntax.BinaryArithm) int {
+func (cfg *Config) assgnArit(b *syntax.BinaryArithm) (int, error) {
-	name := b.X.(*syntax.Word).Parts[0].(*syntax.Lit).Value
+	name := b.X.(*syntax.Word).Lit()
-	val := atoi(r.getVar(name))
+	val := atoi(cfg.envGet(name))
-	arg := r.arithm(ctx, b.Y)
+	arg, err := Arithm(cfg, b.Y)
+	if err != nil {
+		return 0, err
+	}
 	switch b.Op {
 	case syntax.Assgn:
 		val = arg
@@ -116,8 +139,8 @@ func (r *Runner) assgnArit(ctx context.Context, b *syntax.BinaryArithm) int {
 	case syntax.ShrAssgn:
 		val >>= uint(arg)
 	}
-	r.setVarString(ctx, name, strconv.Itoa(val))
+	cfg.envSet(name, strconv.Itoa(val))
-	return val
+	return val, nil
 }
 
 func intPow(a, b int) int {

vendor/mvdan.cc/sh/expand/braces.go

@@ -0,0 +1,24 @@
+// Copyright (c) 2018, Daniel Martí <mvdan@mvdan.cc>
+// See LICENSE for licensing information
+
+package expand
+
+import "mvdan.cc/sh/syntax"
+
+// Braces performs Bash brace expansion on words. For example, passing it a
+// literal word "foo{bar,baz}" will return two literal words, "foobar" and
+// "foobaz".
+//
+// It does not return an error; malformed brace expansions are simply skipped.
+// For example, "a{b{c,d}" results in the words "a{bc" and "a{bd".
+//
+// Note that the resulting words may have more word parts than necessary, such
+// as contiguous *syntax.Lit nodes, and that these parts may be shared between
+// words.
+func Braces(words ...*syntax.Word) []*syntax.Word {
+	var res []*syntax.Word
+	for _, word := range words {
+		res = append(res, syntax.ExpandBraces(word)...)
+	}
+	return res
+}

vendor/mvdan.cc/sh/expand/doc.go

@@ -0,0 +1,5 @@
+// Copyright (c) 2018, Daniel Martí <mvdan@mvdan.cc>
+// See LICENSE for licensing information
+
+// Package expand contains code to perform various shell expansions.
+package expand

vendor/mvdan.cc/sh/expand/environ.go

@@ -0,0 +1,195 @@
+// Copyright (c) 2018, Daniel Martí <mvdan@mvdan.cc>
+// See LICENSE for licensing information
+
+package expand
+
+import (
+	"runtime"
+	"sort"
+	"strings"
+)
+
+// Environ is the base interface for a shell's environment, allowing it to fetch
+// variables by name and to iterate over all the currently set variables.
+type Environ interface {
+	// Get retrieves a variable by its name. To check if the variable is
+	// set, use Variable.IsSet.
+	Get(name string) Variable
+
+	// Each iterates over all the currently set variables, calling the
+	// supplied function on each variable. Iteration is stopped if the
+	// function returns false.
+	//
+	// The names used in the calls aren't required to be unique or sorted.
+	// If a variable name appears twice, the latest occurrence takes
+	// priority.
+	//
+	// Each is required to forward exported variables when executing
+	// programs.
+	Each(func(name string, vr Variable) bool)
+}
+
+// WriteEnviron is an extension on Environ that supports modifying and deleting
+// variables.
+type WriteEnviron interface {
+	Environ
+	// Set sets a variable by name. If !vr.IsSet(), the variable is being
+	// unset; otherwise, the variable is being replaced.
+	//
+	// It is the implementation's responsibility to handle variable
+	// attributes correctly. For example, changing an exported variable's
+	// value does not unexport it, and overwriting a name reference variable
+	// should modify its target.
+	Set(name string, vr Variable)
+}
+
+// Variable describes a shell variable, which can have a number of attributes
+// and a value.
+//
+// A Variable is unset if its Value field is untyped nil, which can be checked
+// via Variable.IsSet. The zero value of a Variable is thus a valid unset
+// variable.
+//
+// If a variable is set, its Value field will be a []string if it is an indexed
+// array, a map[string]string if it's an associative array, or a string
+// otherwise.
+type Variable struct {
+	Local    bool
+	Exported bool
+	ReadOnly bool
+	NameRef  bool        // if true, Value must be string
+	Value    interface{} // string, []string, or map[string]string
+}
+
+// IsSet returns whether the variable is set. An empty variable is set, but an
+// undeclared variable is not.
+func (v Variable) IsSet() bool {
+	return v.Value != nil
+}
+
+// String returns the variable's value as a string. In general, this only makes
+// sense if the variable has a string value or no value at all.
+func (v Variable) String() string {
+	switch x := v.Value.(type) {
+	case string:
+		return x
+	case []string:
+		if len(x) > 0 {
+			return x[0]
+		}
+	case map[string]string:
+		// nothing to do
+	}
+	return ""
+}
+
+// maxNameRefDepth defines the maximum number of times to follow references when
+// resolving a variable. Otherwise, simple name reference loops could crash a
+// program quite easily.
+const maxNameRefDepth = 100
+
+// Resolve follows a number of nameref variables, returning the last reference
+// name that was followed and the variable that it points to.
+func (v Variable) Resolve(env Environ) (string, Variable) {
+	name := ""
+	for i := 0; i < maxNameRefDepth; i++ {
+		if !v.NameRef {
+			return name, v
+		}
+		name = v.Value.(string)
+		v = env.Get(name)
+	}
+	return name, Variable{}
+}
+
+// FuncEnviron wraps a function mapping variable names to their string values,
+// and implements Environ. Empty strings returned by the function will be
+// treated as unset variables. All variables will be exported.
+//
+// Note that the returned Environ's Each method will be a no-op.
+func FuncEnviron(fn func(string) string) Environ {
+	return funcEnviron(fn)
+}
+
+type funcEnviron func(string) string
+
+func (f funcEnviron) Get(name string) Variable {
+	value := f(name)
+	if value == "" {
+		return Variable{}
+	}
+	return Variable{Exported: true, Value: value}
+}
+
+func (f funcEnviron) Each(func(name string, vr Variable) bool) {}
+
+// ListEnviron returns an Environ with the supplied variables, in the form
+// "key=value". All variables will be exported.
+//
+// On Windows, where environment variable names are case-insensitive, the
+// resulting variable names will all be uppercase.
+func ListEnviron(pairs ...string) Environ {
+	return listEnvironWithUpper(runtime.GOOS == "windows", pairs...)
+}
+
+// listEnvironWithUpper implements ListEnviron, but letting the tests specify
+// whether to uppercase all names or not.
+func listEnvironWithUpper(upper bool, pairs ...string) Environ {
+	list := append([]string{}, pairs...)
+	if upper {
+		// Uppercase before sorting, so that we can remove duplicates
+		// without the need for linear search nor a map.
+		for i, s := range list {
+			if sep := strings.IndexByte(s, '='); sep > 0 {
+				list[i] = strings.ToUpper(s[:sep]) + s[sep:]
+			}
+		}
+	}
+	sort.Strings(list)
+	last := ""
+	for i := 0; i < len(list); {
+		s := list[i]
+		sep := strings.IndexByte(s, '=')
+		if sep <= 0 {
+			// invalid element; remove it
+			list = append(list[:i], list[i+1:]...)
+			continue
+		}
+		name := s[:sep]
+		if last == name {
+			// duplicate; the last one wins
+			list = append(list[:i-1], list[i:]...)
+			continue
+		}
+		last = name
+		i++
+	}
+	return listEnviron(list)
+}
+
+type listEnviron []string
+
+func (l listEnviron) Get(name string) Variable {
+	// TODO: binary search
+	prefix := name + "="
+	for _, pair := range l {
+		if val := strings.TrimPrefix(pair, prefix); val != pair {
+			return Variable{Exported: true, Value: val}
+		}
+	}
+	return Variable{}
+}
+
+func (l listEnviron) Each(fn func(name string, vr Variable) bool) {
+	for _, pair := range l {
+		i := strings.IndexByte(pair, '=')
+		if i < 0 {
+			// can't happen; see above
+			panic("expand.listEnviron: did not expect malformed name-value pair: " + pair)
+		}
+		name, value := pair[:i], pair[i+1:]
+		if !fn(name, Variable{Exported: true, Value: value}) {
+			return
+		}
+	}
+}

vendor/mvdan.cc/sh/expand/expand.go

@@ -0,0 +1,784 @@
+// Copyright (c) 2017, Daniel Martí <mvdan@mvdan.cc>
+// See LICENSE for licensing information
+
+package expand
+
+import (
+	"bytes"
+	"fmt"
+	"io"
+	"os"
+	"os/user"
+	"path/filepath"
+	"regexp"
+	"runtime"
+	"strconv"
+	"strings"
+
+	"mvdan.cc/sh/syntax"
+)
+
+// A Config specifies details about how shell expansion should be performed. The
+// zero value is a valid configuration.
+type Config struct {
+	// Env is used to get and set environment variables when performing
+	// shell expansions. Some special parameters are also expanded via this
+	// interface, such as:
+	//
+	//   * "#", "@", "*", "0"-"9" for the shell's parameters
+	//   * "?", "$", "PPID" for the shell's status and process
+	//   * "HOME foo" to retrieve user foo's home directory (if unset,
+	//     os/user.Lookup will be used)
+	//
+	// If nil, there are no environment variables set. Use
+	// ListEnviron(os.Environ()...) to use the system's environment
+	// variables.
+	Env Environ
+
+	// TODO(mvdan): consider replacing NoGlob==true with ReadDir==nil.
+
+	// NoGlob corresponds to the shell option that disables globbing.
+	NoGlob bool
+	// GlobStar corresponds to the shell option that allows globbing with
+	// "**".
+	GlobStar bool
+
+	// CmdSubst expands a command substitution node, writing its standard
+	// output to the provided io.Writer.
+	//
+	// If nil, encountering a command substitution will result in an
+	// UnexpectedCommandError.
+	CmdSubst func(io.Writer, *syntax.CmdSubst) error
+
+	// ReadDir is used for file path globbing. If nil, globbing is disabled.
+	// Use ioutil.ReadDir to use the filesystem directly.
+	ReadDir func(string) ([]os.FileInfo, error)
+
+	bufferAlloc bytes.Buffer
+	fieldAlloc  [4]fieldPart
+	fieldsAlloc [4][]fieldPart
+
+	ifs string
+	// A pointer to a parameter expansion node, if we're inside one.
+	// Necessary for ${LINENO}.
+	curParam *syntax.ParamExp
+}
+
+// UnexpectedCommandError is returned if a command substitution is encountered
+// when Config.CmdSubst is nil.
+type UnexpectedCommandError struct {
+	Node *syntax.CmdSubst
+}
+
+func (u UnexpectedCommandError) Error() string {
+	return fmt.Sprintf("unexpected command substitution at %s", u.Node.Pos())
+}
+
+var zeroConfig = &Config{}
+
+func prepareConfig(cfg *Config) *Config {
+	if cfg == nil {
+		cfg = zeroConfig
+	}
+	if cfg.Env == nil {
+		cfg.Env = FuncEnviron(func(string) string { return "" })
+	}
+
+	cfg.ifs = " \t\n"
+	if vr := cfg.Env.Get("IFS"); vr.IsSet() {
+		cfg.ifs = vr.String()
+	}
+	return cfg
+}
+
+func (cfg *Config) ifsRune(r rune) bool {
+	for _, r2 := range cfg.ifs {
+		if r == r2 {
+			return true
+		}
+	}
+	return false
+}
+
+func (cfg *Config) ifsJoin(strs []string) string {
+	sep := ""
+	if cfg.ifs != "" {
+		sep = cfg.ifs[:1]
+	}
+	return strings.Join(strs, sep)
+}
+
+func (cfg *Config) strBuilder() *bytes.Buffer {
+	b := &cfg.bufferAlloc
+	b.Reset()
+	return b
+}
+
+func (cfg *Config) envGet(name string) string {
+	return cfg.Env.Get(name).String()
+}
+
+func (cfg *Config) envSet(name, value string) {
+	wenv, ok := cfg.Env.(WriteEnviron)
+	if !ok {
+		// TODO: we should probably error here
+		return
+	}
+	wenv.Set(name, Variable{Value: value})
+}
+
+// Literal expands a single shell word. It is similar to Fields, but the result
+// is a single string. This is the behavior when a word is used as the value in
+// a shell variable assignment, for example.
+//
+// The config specifies shell expansion options; nil behaves the same as an
+// empty config.
+func Literal(cfg *Config, word *syntax.Word) (string, error) {
+	if word == nil {
+		return "", nil
+	}
+	cfg = prepareConfig(cfg)
+	field, err := cfg.wordField(word.Parts, quoteNone)
+	if err != nil {
+		return "", err
+	}
+	return cfg.fieldJoin(field), nil
+}
+
+// Document expands a single shell word as if it were within double quotes. It
+// is simlar to Literal, but without brace expansion, tilde expansion, and
+// globbing.
+//
+// The config specifies shell expansion options; nil behaves the same as an
+// empty config.
+func Document(cfg *Config, word *syntax.Word) (string, error) {
+	if word == nil {
+		return "", nil
+	}
+	cfg = prepareConfig(cfg)
+	field, err := cfg.wordField(word.Parts, quoteDouble)
+	if err != nil {
+		return "", err
+	}
+	return cfg.fieldJoin(field), nil
+}
+
+// Pattern expands a single shell word as a pattern, using syntax.QuotePattern
+// on any non-quoted parts of the input word. The result can be used on
+// syntax.TranslatePattern directly.
+//
+// The config specifies shell expansion options; nil behaves the same as an
+// empty config.
+func Pattern(cfg *Config, word *syntax.Word) (string, error) {
+	cfg = prepareConfig(cfg)
+	field, err := cfg.wordField(word.Parts, quoteNone)
+	if err != nil {
+		return "", err
+	}
+	buf := cfg.strBuilder()
+	for _, part := range field {
+		if part.quote > quoteNone {
+			buf.WriteString(syntax.QuotePattern(part.val))
+		} else {
+			buf.WriteString(part.val)
+		}
+	}
+	return buf.String(), nil
+}
+
+// Format expands a format string with a number of arguments, following the
+// shell's format specifications. These include printf(1), among others.
+//
+// The resulting string is returned, along with the number of arguments used.
+//
+// The config specifies shell expansion options; nil behaves the same as an
+// empty config.
+func Format(cfg *Config, format string, args []string) (string, int, error) {
+	cfg = prepareConfig(cfg)
+	buf := cfg.strBuilder()
+	esc := false
+	var fmts []rune
+	initialArgs := len(args)
+
+	for _, c := range format {
+		switch {
+		case esc:
+			esc = false
+			switch c {
+			case 'n':
+				buf.WriteRune('\n')
+			case 'r':
+				buf.WriteRune('\r')
+			case 't':
+				buf.WriteRune('\t')
+			case '\\':
+				buf.WriteRune('\\')
+			default:
+				buf.WriteRune('\\')
+				buf.WriteRune(c)
+			}
+
+		case len(fmts) > 0:
+			switch c {
+			case '%':
+				buf.WriteByte('%')
+				fmts = nil
+			case 'c':
+				var b byte
+				if len(args) > 0 {
+					arg := ""
+					arg, args = args[0], args[1:]
+					if len(arg) > 0 {
+						b = arg[0]
+					}
+				}
+				buf.WriteByte(b)
+				fmts = nil
+			case '+', '-', ' ':
+				if len(fmts) > 1 {
+					return "", 0, fmt.Errorf("invalid format char: %c", c)
+				}
+				fmts = append(fmts, c)
+			case '0', '1', '2', '3', '4', '5', '6', '7', '8', '9':
+				fmts = append(fmts, c)
+			case 's', 'd', 'i', 'u', 'o', 'x':
+				arg := ""
+				if len(args) > 0 {
+					arg, args = args[0], args[1:]
+				}
+				var farg interface{} = arg
+				if c != 's' {
+					n, _ := strconv.ParseInt(arg, 0, 0)
+					if c == 'i' || c == 'd' {
+						farg = int(n)
+					} else {
+						farg = uint(n)
+					}
+					if c == 'i' || c == 'u' {
+						c = 'd'
+					}
+				}
+				fmts = append(fmts, c)
+				fmt.Fprintf(buf, string(fmts), farg)
+				fmts = nil
+			default:
+				return "", 0, fmt.Errorf("invalid format char: %c", c)
+			}
+		case c == '\\':
+			esc = true
+		case args != nil && c == '%':
+			// if args == nil, we are not doing format
+			// arguments
+			fmts = []rune{c}
+		default:
+			buf.WriteRune(c)
+		}
+	}
+	if len(fmts) > 0 {
+		return "", 0, fmt.Errorf("missing format char")
+	}
+	return buf.String(), initialArgs - len(args), nil
+}
+
+func (cfg *Config) fieldJoin(parts []fieldPart) string {
+	switch len(parts) {
+	case 0:
+		return ""
+	case 1: // short-cut without a string copy
+		return parts[0].val
+	}
+	buf := cfg.strBuilder()
+	for _, part := range parts {
+		buf.WriteString(part.val)
+	}
+	return buf.String()
+}
+
+func (cfg *Config) escapedGlobField(parts []fieldPart) (escaped string, glob bool) {
+	buf := cfg.strBuilder()
+	for _, part := range parts {
+		if part.quote > quoteNone {
+			buf.WriteString(syntax.QuotePattern(part.val))
+			continue
+		}
+		buf.WriteString(part.val)
+		if syntax.HasPattern(part.val) {
+			glob = true
+		}
+	}
+	if glob { // only copy the string if it will be used
+		escaped = buf.String()
+	}
+	return escaped, glob
+}
+
+// Fields expands a number of words as if they were arguments in a shell
+// command. This includes brace expansion, tilde expansion, parameter expansion,
+// command substitution, arithmetic expansion, and quote removal.
+func Fields(cfg *Config, words ...*syntax.Word) ([]string, error) {
+	cfg = prepareConfig(cfg)
+	fields := make([]string, 0, len(words))
+	dir := cfg.envGet("PWD")
+	for _, expWord := range Braces(words...) {
+		wfields, err := cfg.wordFields(expWord.Parts)
+		if err != nil {
+			return nil, err
+		}
+		for _, field := range wfields {
+			path, doGlob := cfg.escapedGlobField(field)
+			var matches []string
+			abs := filepath.IsAbs(path)
+			if doGlob && !cfg.NoGlob {
+				base := ""
+				if !abs {
+					base = dir
+				}
+				matches, err = cfg.glob(base, path)
+				if err != nil {
+					return nil, err
+				}
+			}
+			if len(matches) == 0 {
+				fields = append(fields, cfg.fieldJoin(field))
+				continue
+			}
+			for _, match := range matches {
+				if !abs {
+					match = strings.TrimPrefix(match, dir)
+				}
+				fields = append(fields, match)
+			}
+		}
+	}
+	return fields, nil
+}
+
+type fieldPart struct {
+	val   string
+	quote quoteLevel
+}
+
+type quoteLevel uint
+
+const (
+	quoteNone quoteLevel = iota
+	quoteDouble
+	quoteSingle
+)
+
+func (cfg *Config) wordField(wps []syntax.WordPart, ql quoteLevel) ([]fieldPart, error) {
+	var field []fieldPart
+	for i, wp := range wps {
+		switch x := wp.(type) {
+		case *syntax.Lit:
+			s := x.Value
+			if i == 0 && ql == quoteNone {
+				s = cfg.expandUser(s)
+			}
+			if ql == quoteDouble && strings.Contains(s, "\\") {
+				buf := cfg.strBuilder()
+				for i := 0; i < len(s); i++ {
+					b := s[i]
+					if b == '\\' && i+1 < len(s) {
+						switch s[i+1] {
+						case '\n': // remove \\\n
+							i++
+							continue
+						case '"', '\\', '$', '`': // special chars
+							continue
+						}
+					}
+					buf.WriteByte(b)
+				}
+				s = buf.String()
+			}
+			field = append(field, fieldPart{val: s})
+		case *syntax.SglQuoted:
+			fp := fieldPart{quote: quoteSingle, val: x.Value}
+			if x.Dollar {
+				fp.val, _, _ = Format(cfg, fp.val, nil)
+			}
+			field = append(field, fp)
+		case *syntax.DblQuoted:
+			wfield, err := cfg.wordField(x.Parts, quoteDouble)
+			if err != nil {
+				return nil, err
+			}
+			for _, part := range wfield {
+				part.quote = quoteDouble
+				field = append(field, part)
+			}
+		case *syntax.ParamExp:
+			val, err := cfg.paramExp(x)
+			if err != nil {
+				return nil, err
+			}
+			field = append(field, fieldPart{val: val})
+		case *syntax.CmdSubst:
+			val, err := cfg.cmdSubst(x)
+			if err != nil {
+				return nil, err
+			}
+			field = append(field, fieldPart{val: val})
+		case *syntax.ArithmExp:
+			n, err := Arithm(cfg, x.X)
+			if err != nil {
+				return nil, err
+			}
+			field = append(field, fieldPart{val: strconv.Itoa(n)})
+		default:
+			panic(fmt.Sprintf("unhandled word part: %T", x))
+		}
+	}
+	return field, nil
+}
+
+func (cfg *Config) cmdSubst(cs *syntax.CmdSubst) (string, error) {
+	if cfg.CmdSubst == nil {
+		return "", UnexpectedCommandError{Node: cs}
+	}
+	buf := cfg.strBuilder()
+	if err := cfg.CmdSubst(buf, cs); err != nil {
+		return "", err
+	}
+	return strings.TrimRight(buf.String(), "\n"), nil
+}
+
+func (cfg *Config) wordFields(wps []syntax.WordPart) ([][]fieldPart, error) {
+	fields := cfg.fieldsAlloc[:0]
+	curField := cfg.fieldAlloc[:0]
+	allowEmpty := false
+	flush := func() {
+		if len(curField) == 0 {
+			return
+		}
+		fields = append(fields, curField)
+		curField = nil
+	}
+	splitAdd := func(val string) {
+		for i, field := range strings.FieldsFunc(val, cfg.ifsRune) {
+			if i > 0 {
+				flush()
+			}
+			curField = append(curField, fieldPart{val: field})
+		}
+	}
+	for i, wp := range wps {
+		switch x := wp.(type) {
+		case *syntax.Lit:
+			s := x.Value
+			if i == 0 {
+				s = cfg.expandUser(s)
+			}
+			if strings.Contains(s, "\\") {
+				buf := cfg.strBuilder()
+				for i := 0; i < len(s); i++ {
+					b := s[i]
+					if b == '\\' {
+						i++
+						b = s[i]
+					}
+					buf.WriteByte(b)
+				}
+				s = buf.String()
+			}
+			curField = append(curField, fieldPart{val: s})
+		case *syntax.SglQuoted:
+			allowEmpty = true
+			fp := fieldPart{quote: quoteSingle, val: x.Value}
+			if x.Dollar {
+				fp.val, _, _ = Format(cfg, fp.val, nil)
+			}
+			curField = append(curField, fp)
+		case *syntax.DblQuoted:
+			allowEmpty = true
+			if len(x.Parts) == 1 {
+				pe, _ := x.Parts[0].(*syntax.ParamExp)
+				if elems := cfg.quotedElems(pe); elems != nil {
+					for i, elem := range elems {
+						if i > 0 {
+							flush()
+						}
+						curField = append(curField, fieldPart{
+							quote: quoteDouble,
+							val:   elem,
+						})
+					}
+					continue
+				}
+			}
+			wfield, err := cfg.wordField(x.Parts, quoteDouble)
+			if err != nil {
+				return nil, err
+			}
+			for _, part := range wfield {
+				part.quote = quoteDouble
+				curField = append(curField, part)
+			}
+		case *syntax.ParamExp:
+			val, err := cfg.paramExp(x)
+			if err != nil {
+				return nil, err
+			}
+			splitAdd(val)
+		case *syntax.CmdSubst:
+			val, err := cfg.cmdSubst(x)
+			if err != nil {
+				return nil, err
+			}
+			splitAdd(val)
+		case *syntax.ArithmExp:
+			n, err := Arithm(cfg, x.X)
+			if err != nil {
+				return nil, err
+			}
+			curField = append(curField, fieldPart{val: strconv.Itoa(n)})
+		default:
+			panic(fmt.Sprintf("unhandled word part: %T", x))
+		}
+	}
+	flush()
+	if allowEmpty && len(fields) == 0 {
+		fields = append(fields, curField)
+	}
+	return fields, nil
+}
+
+// quotedElems checks if a parameter expansion is exactly ${@} or ${foo[@]}
+func (cfg *Config) quotedElems(pe *syntax.ParamExp) []string {
+	if pe == nil || pe.Excl || pe.Length || pe.Width {
+		return nil
+	}
+	if pe.Param.Value == "@" {
+		return cfg.Env.Get("@").Value.([]string)
+	}
+	if nodeLit(pe.Index) != "@" {
+		return nil
+	}
+	val := cfg.Env.Get(pe.Param.Value).Value
+	if x, ok := val.([]string); ok {
+		return x
+	}
+	return nil
+}
+
+func (cfg *Config) expandUser(field string) string {
+	if len(field) == 0 || field[0] != '~' {
+		return field
+	}
+	name := field[1:]
+	rest := ""
+	if i := strings.Index(name, "/"); i >= 0 {
+		rest = name[i:]
+		name = name[:i]
+	}
+	if name == "" {
+		return cfg.Env.Get("HOME").String() + rest
+	}
+	if vr := cfg.Env.Get("HOME " + name); vr.IsSet() {
+		return vr.String() + rest
+	}
+
+	u, err := user.Lookup(name)
+	if err != nil {
+		return field
+	}
+	return u.HomeDir + rest
+}
+
+func findAllIndex(pattern, name string, n int) [][]int {
+	expr, err := syntax.TranslatePattern(pattern, true)
+	if err != nil {
+		return nil
+	}
+	rx := regexp.MustCompile(expr)
+	return rx.FindAllStringIndex(name, n)
+}
+
+// TODO: use this again to optimize globbing; see
+// https://github.com/mvdan/sh/issues/213
+func hasGlob(path string) bool {
+	magicChars := `*?[`
+	if runtime.GOOS != "windows" {
+		magicChars = `*?[\`
+	}
+	return strings.ContainsAny(path, magicChars)
+}
+
+var rxGlobStar = regexp.MustCompile(".*")
+
+// pathJoin2 is a simpler version of filepath.Join without cleaning the result,
+// since that's needed for globbing.
+func pathJoin2(elem1, elem2 string) string {
+	if elem1 == "" {
+		return elem2
+	}
+	if strings.HasSuffix(elem1, string(filepath.Separator)) {
+		return elem1 + elem2
+	}
+	return elem1 + string(filepath.Separator) + elem2
+}
+
+// pathSplit splits a file path into its elements, retaining empty ones. Before
+// splitting, slashes are replaced with filepath.Separator, so that splitting
+// Unix paths on Windows works as well.
+func pathSplit(path string) []string {
+	path = filepath.FromSlash(path)
+	return strings.Split(path, string(filepath.Separator))
+}
+
+func (cfg *Config) glob(base, pattern string) ([]string, error) {
+	parts := pathSplit(pattern)
+	matches := []string{""}
+	if filepath.IsAbs(pattern) {
+		if parts[0] == "" {
+			// unix-like
+			matches[0] = string(filepath.Separator)
+		} else {
+			// windows (for some reason it won't work without the
+			// trailing separator)
+			matches[0] = parts[0] + string(filepath.Separator)
+		}
+		parts = parts[1:]
+	}
+	for _, part := range parts {
+		switch {
+		case part == "", part == ".", part == "..":
+			var newMatches []string
+			for _, dir := range matches {
+				// TODO(mvdan): reuse the previous ReadDir call
+				if cfg.ReadDir == nil {
+					continue // no globbing
+				} else if _, err := cfg.ReadDir(filepath.Join(base, dir)); err != nil {
+					continue // not actually a dir
+				}
+				newMatches = append(newMatches, pathJoin2(dir, part))
+			}
+			matches = newMatches
+			continue
+		case part == "**" && cfg.GlobStar:
+			for i, match := range matches {
+				// "a/**" should match "a/ a/b a/b/cfg ..."; note
+				// how the zero-match case has a trailing
+				// separator.
+				matches[i] = pathJoin2(match, "")
+			}
+			// expand all the possible levels of **
+			latest := matches
+			for {
+				var newMatches []string
+				for _, dir := range latest {
+					var err error
+					newMatches, err = cfg.globDir(base, dir, rxGlobStar, newMatches)
+					if err != nil {
+						return nil, err
+					}
+				}
+				if len(newMatches) == 0 {
+					// not another level of directories to
+					// try; stop
+					break
+				}
+				matches = append(matches, newMatches...)
+				latest = newMatches
+			}
+			continue
+		}
+		expr, err := syntax.TranslatePattern(part, true)
+		if err != nil {
+			// If any glob part is not a valid pattern, don't glob.
+			return nil, nil
+		}
+		rx := regexp.MustCompile("^" + expr + "$")
+		var newMatches []string
+		for _, dir := range matches {
+			newMatches, err = cfg.globDir(base, dir, rx, newMatches)
+			if err != nil {
+				return nil, err
+			}
+		}
+		matches = newMatches
+	}
+	return matches, nil
+}
+
+func (cfg *Config) globDir(base, dir string, rx *regexp.Regexp, matches []string) ([]string, error) {
+	if cfg.ReadDir == nil {
+		// TODO(mvdan): check this at the beginning of a glob?
+		return nil, nil
+	}
+	infos, err := cfg.ReadDir(filepath.Join(base, dir))
+	if err != nil {
+		return nil, err
+	}
+	for _, info := range infos {
+		name := info.Name()
+		if !strings.HasPrefix(rx.String(), `^\.`) && name[0] == '.' {
+			continue
+		}
+		if rx.MatchString(name) {
+			matches = append(matches, pathJoin2(dir, name))
+		}
+	}
+	return matches, nil
+}
+
+//
+// The config specifies shell expansion options; nil behaves the same as an
+// empty config.
+func ReadFields(cfg *Config, s string, n int, raw bool) []string {
+	cfg = prepareConfig(cfg)
+	type pos struct {
+		start, end int
+	}
+	var fpos []pos
+
+	runes := make([]rune, 0, len(s))
+	infield := false
+	esc := false
+	for _, r := range s {
+		if infield {
+			if cfg.ifsRune(r) && (raw || !esc) {
+				fpos[len(fpos)-1].end = len(runes)
+				infield = false
+			}
+		} else {
+			if !cfg.ifsRune(r) && (raw || !esc) {
+				fpos = append(fpos, pos{start: len(runes), end: -1})
+				infield = true
+			}
+		}
+		if r == '\\' {
+			if raw || esc {
+				runes = append(runes, r)
+			}
+			esc = !esc
+			continue
+		}
+		runes = append(runes, r)
+		esc = false
+	}
+	if len(fpos) == 0 {
+		return nil
+	}
+	if infield {
+		fpos[len(fpos)-1].end = len(runes)
+	}
+
+	switch {
+	case n == 1:
+		// include heading/trailing IFSs
+		fpos[0].start, fpos[0].end = 0, len(runes)
+		fpos = fpos[:1]
+	case n != -1 && n < len(fpos):
+		// combine to max n fields
+		fpos[n-1].end = fpos[len(fpos)-1].end
+		fpos = fpos[:n]
+	}
+
+	var fields = make([]string, len(fpos))
+	for i, p := range fpos {
+		fields[i] = string(runes[p.start:p.end])
+	}
+	return fields
+}

vendor/mvdan.cc/sh/expand/param.go

@@ -1,12 +1,10 @@
 // Copyright (c) 2017, Daniel Martí <mvdan@mvdan.cc>
 // See LICENSE for licensing information
 
-package interp
+package expand
 
 import (
-	"context"
 	"fmt"
-	"os"
 	"regexp"
 	"sort"
 	"strconv"
@@ -17,146 +15,133 @@ import (
 	"mvdan.cc/sh/syntax"
 )
 
-func anyOfLit(v interface{}, vals ...string) string {
+func nodeLit(node syntax.Node) string {
-	word, _ := v.(*syntax.Word)
+	if word, ok := node.(*syntax.Word); ok {
-	if word == nil || len(word.Parts) != 1 {
+		return word.Lit()
-		return ""
-	}
-	lit, ok := word.Parts[0].(*syntax.Lit)
-	if !ok {
-		return ""
-	}
-	for _, val := range vals {
-		if lit.Value == val {
-			return val
-		}
 	}
 	return ""
 }
 
-// quotedElems checks if a parameter expansion is exactly ${@} or ${foo[@]}
+type UnsetParameterError struct {
-func (r *Runner) quotedElems(pe *syntax.ParamExp) []string {
+	Node    *syntax.ParamExp
-	if pe == nil || pe.Excl || pe.Length || pe.Width {
+	Message string
-		return nil
-	}
-	if pe.Param.Value == "@" {
-		return r.Params
-	}
-	if anyOfLit(pe.Index, "@") == "" {
-		return nil
-	}
-	val, _ := r.lookupVar(pe.Param.Value)
-	if x, ok := val.Value.(IndexArray); ok {
-		return x
-	}
-	return nil
 }
 
-func (r *Runner) paramExp(ctx context.Context, pe *syntax.ParamExp) string {
+func (u UnsetParameterError) Error() string {
+	return u.Message
+}
+
+func (cfg *Config) paramExp(pe *syntax.ParamExp) (string, error) {
+	oldParam := cfg.curParam
+	cfg.curParam = pe
+	defer func() { cfg.curParam = oldParam }()
+
 	name := pe.Param.Value
-	var vr Variable
-	set := false
 	index := pe.Index
 	switch name {
-	case "#":
-		vr.Value = StringVal(strconv.Itoa(len(r.Params)))
 	case "@", "*":
-		vr.Value = IndexArray(r.Params)
 		index = &syntax.Word{Parts: []syntax.WordPart{
 			&syntax.Lit{Value: name},
 		}}
-	case "?":
+	}
-		vr.Value = StringVal(strconv.Itoa(r.exit))
+	var vr Variable
-	case "$":
+	switch name {
-		vr.Value = StringVal(strconv.Itoa(os.Getpid()))
-	case "PPID":
-		vr.Value = StringVal(strconv.Itoa(os.Getppid()))
 	case "LINENO":
-		line := uint64(pe.Pos().Line())
+		// This is the only parameter expansion that the environment
-		vr.Value = StringVal(strconv.FormatUint(line, 10))
+		// interface cannot satisfy.
-	case "DIRSTACK":
+		line := uint64(cfg.curParam.Pos().Line())
-		vr.Value = IndexArray(r.dirStack)
+		vr.Value = strconv.FormatUint(line, 10)
 	default:
-		if n, err := strconv.Atoi(name); err == nil {
+		vr = cfg.Env.Get(name)
-			if i := n - 1; i < len(r.Params) {
-				vr.Value, set = StringVal(r.Params[i]), true
-			}
-		} else {
-			vr, set = r.lookupVar(name)
-		}
 	}
-	str := r.varStr(vr, 0)
+	orig := vr
-	if index != nil {
+	_, vr = vr.Resolve(cfg.Env)
-		str = r.varInd(ctx, vr, index, 0)
+	str, err := cfg.varInd(vr, index)
+	if err != nil {
+		return "", err
 	}
-	slicePos := func(expr syntax.ArithmExpr) int {
+	slicePos := func(n int) int {
-		p := r.arithm(ctx, expr)
+		if n < 0 {
-		if p < 0 {
+			n = len(str) + n
-			p = len(str) + p
+			if n < 0 {
-			if p < 0 {
+				n = len(str)
-				p = len(str)
 			}
-		} else if p > len(str) {
+		} else if n > len(str) {
-			p = len(str)
+			n = len(str)
 		}
-		return p
+		return n
 	}
 	elems := []string{str}
-	if anyOfLit(index, "@", "*") != "" {
+	switch nodeLit(index) {
+	case "@", "*":
 		switch x := vr.Value.(type) {
 		case nil:
 			elems = nil
-		case IndexArray:
+		case []string:
 			elems = x
 		}
 	}
 	switch {
 	case pe.Length:
 		n := len(elems)
-		if anyOfLit(index, "@", "*") == "" {
+		switch nodeLit(index) {
+		case "@", "*":
+		default:
 			n = utf8.RuneCountInString(str)
 		}
 		str = strconv.Itoa(n)
 	case pe.Excl:
 		var strs []string
 		if pe.Names != 0 {
-			strs = r.namesByPrefix(pe.Param.Value)
+			strs = cfg.namesByPrefix(pe.Param.Value)
-		} else if vr.NameRef {
+		} else if orig.NameRef {
-			strs = append(strs, string(vr.Value.(StringVal)))
+			strs = append(strs, orig.Value.(string))
-		} else if x, ok := vr.Value.(IndexArray); ok {
+		} else if x, ok := vr.Value.([]string); ok {
 			for i, e := range x {
 				if e != "" {
 					strs = append(strs, strconv.Itoa(i))
 				}
 			}
-		} else if x, ok := vr.Value.(AssocArray); ok {
+		} else if x, ok := vr.Value.(map[string]string); ok {
 			for k := range x {
 				strs = append(strs, k)
 			}
 		} else if str != "" {
-			vr, _ = r.lookupVar(str)
+			vr = cfg.Env.Get(str)
-			strs = append(strs, r.varStr(vr, 0))
+			strs = append(strs, vr.String())
 		}
 		sort.Strings(strs)
 		str = strings.Join(strs, " ")
 	case pe.Slice != nil:
 		if pe.Slice.Offset != nil {
-			offset := slicePos(pe.Slice.Offset)
+			n, err := Arithm(cfg, pe.Slice.Offset)
-			str = str[offset:]
+			if err != nil {
+				return "", err
+			}
+			str = str[slicePos(n):]
 		}
 		if pe.Slice.Length != nil {
-			length := slicePos(pe.Slice.Length)
+			n, err := Arithm(cfg, pe.Slice.Length)
-			str = str[:length]
+			if err != nil {
+				return "", err
+			}
+			str = str[:slicePos(n)]
 		}
 	case pe.Repl != nil:
-		orig := r.lonePattern(ctx, pe.Repl.Orig)
+		orig, err := Pattern(cfg, pe.Repl.Orig)
-		with := r.loneWord(ctx, pe.Repl.With)
+		if err != nil {
+			return "", err
+		}
+		with, err := Literal(cfg, pe.Repl.With)
+		if err != nil {
+			return "", err
+		}
 		n := 1
 		if pe.Repl.All {
 			n = -1
 		}
 		locs := findAllIndex(orig, str, n)
-		buf := r.strBuilder()
+		buf := cfg.strBuilder()
 		last := 0
 		for _, loc := range locs {
 			buf.WriteString(str[last:loc[0]])
@@ -166,7 +151,10 @@ func (r *Runner) paramExp(ctx context.Context, pe *syntax.ParamExp) string {
 		buf.WriteString(str[last:])
 		str = buf.String()
 	case pe.Exp != nil:
-		arg := r.loneWord(ctx, pe.Exp.Word)
+		arg, err := Literal(cfg, pe.Exp.Word)
+		if err != nil {
+			return "", err
+		}
 		switch op := pe.Exp.Op; op {
 		case syntax.SubstColPlus:
 			if str == "" {
@@ -174,11 +162,11 @@ func (r *Runner) paramExp(ctx context.Context, pe *syntax.ParamExp) string {
 			}
 			fallthrough
 		case syntax.SubstPlus:
-			if set {
+			if vr.IsSet() {
 				str = arg
 			}
 		case syntax.SubstMinus:
-			if set {
+			if vr.IsSet() {
 				break
 			}
 			fallthrough
@@ -187,24 +175,25 @@ func (r *Runner) paramExp(ctx context.Context, pe *syntax.ParamExp) string {
 				str = arg
 			}
 		case syntax.SubstQuest:
-			if set {
+			if vr.IsSet() {
 				break
 			}
 			fallthrough
 		case syntax.SubstColQuest:
 			if str == "" {
-				r.errf("%s\n", arg)
+				return "", UnsetParameterError{
-				r.exit = 1
+					Node:    pe,
-				r.setErr(ShellExitStatus(r.exit))
+					Message: arg,
+				}
 			}
 		case syntax.SubstAssgn:
-			if set {
+			if vr.IsSet() {
 				break
 			}
 			fallthrough
 		case syntax.SubstColAssgn:
 			if str == "" {
-				r.setVarString(ctx, name, arg)
+				cfg.envSet(name, arg)
 				str = arg
 			}
 		case syntax.RemSmallPrefix, syntax.RemLargePrefix,
@@ -229,7 +218,7 @@ func (r *Runner) paramExp(ctx context.Context, pe *syntax.ParamExp) string {
 			// empty string means '?'; nothing to do there
 			expr, err := syntax.TranslatePattern(arg, false)
 			if err != nil {
-				return str
+				return str, nil
 			}
 			rx := regexp.MustCompile(expr)
 
@@ -266,7 +255,7 @@ func (r *Runner) paramExp(ctx context.Context, pe *syntax.ParamExp) string {
 			}
 		}
 	}
-	return str
+	return str, nil
 }
 
 func removePattern(str, pattern string, fromEnd, greedy bool) string {
@@ -293,3 +282,67 @@ func removePattern(str, pattern string, fromEnd, greedy bool) string {
 	}
 	return str
 }
+
+func (cfg *Config) varInd(vr Variable, idx syntax.ArithmExpr) (string, error) {
+	if idx == nil {
+		return vr.String(), nil
+	}
+	switch x := vr.Value.(type) {
+	case string:
+		n, err := Arithm(cfg, idx)
+		if err != nil {
+			return "", err
+		}
+		if n == 0 {
+			return x, nil
+		}
+	case []string:
+		switch nodeLit(idx) {
+		case "@":
+			return strings.Join(x, " "), nil
+		case "*":
+			return cfg.ifsJoin(x), nil
+		}
+		i, err := Arithm(cfg, idx)
+		if err != nil {
+			return "", err
+		}
+		if len(x) > 0 {
+			return x[i], nil
+		}
+	case map[string]string:
+		switch lit := nodeLit(idx); lit {
+		case "@", "*":
+			var strs []string
+			keys := make([]string, 0, len(x))
+			for k := range x {
+				keys = append(keys, k)
+			}
+			sort.Strings(keys)
+			for _, k := range keys {
+				strs = append(strs, x[k])
+			}
+			if lit == "*" {
+				return cfg.ifsJoin(strs), nil
+			}
+			return strings.Join(strs, " "), nil
+		}
+		val, err := Literal(cfg, idx.(*syntax.Word))
+		if err != nil {
+			return "", err
+		}
+		return x[val], nil
+	}
+	return "", nil
+}
+
+func (cfg *Config) namesByPrefix(prefix string) []string {
+	var names []string
+	cfg.Env.Each(func(name string, vr Variable) bool {
+		if strings.HasPrefix(name, prefix) {
+			names = append(names, name)
+		}
+		return true
+	})
+	return names
+}

vendor/mvdan.cc/sh/interp/builtin.go

@@ -13,6 +13,7 @@ import (
 	"strconv"
 	"strings"
 
+	"mvdan.cc/sh/expand"
 	"mvdan.cc/sh/syntax"
 )
 
@@ -29,6 +30,20 @@ func isBuiltin(name string) bool {
 	return false
 }
 
+func oneIf(b bool) int {
+	if b {
+		return 1
+	}
+	return 0
+}
+
+// atoi is just a shorthand for strconv.Atoi that ignores the error,
+// just like shells do.
+func atoi(s string) int {
+	n, _ := strconv.Atoi(s)
+	return n
+}
+
 func (r *Runner) builtinCode(ctx context.Context, pos syntax.Pos, name string, args []string) int {
 	switch name {
 	case "true", ":":
@@ -91,7 +106,7 @@ func (r *Runner) builtinCode(ctx context.Context, pos syntax.Pos, name string, a
 		}
 
 		for _, arg := range args {
-			if _, ok := r.lookupVar(arg); ok && vars {
+			if vr := r.lookupVar(arg); vr.IsSet() && vars {
 				r.delVar(arg)
 				continue
 			}
@@ -100,14 +115,14 @@ func (r *Runner) builtinCode(ctx context.Context, pos syntax.Pos, name string, a
 			}
 		}
 	case "echo":
-		newline, expand := true, false
+		newline, doExpand := true, false
 	echoOpts:
 		for len(args) > 0 {
 			switch args[0] {
 			case "-n":
 				newline = false
 			case "-e":
-				expand = true
+				doExpand = true
 			case "-E": // default
 			default:
 				break echoOpts
@@ -118,8 +133,8 @@ func (r *Runner) builtinCode(ctx context.Context, pos syntax.Pos, name string, a
 			if i > 0 {
 				r.out(" ")
 			}
-			if expand {
+			if doExpand {
-				_, arg, _ = r.expandFormat(arg, nil)
+				arg, _, _ = expand.Format(r.ecfg, arg, nil)
 			}
 			r.out(arg)
 		}
@@ -133,7 +148,7 @@ func (r *Runner) builtinCode(ctx context.Context, pos syntax.Pos, name string, a
 		}
 		format, args := args[0], args[1:]
 		for {
-			n, s, err := r.expandFormat(format, args)
+			s, n, err := expand.Format(r.ecfg, format, args)
 			if err != nil {
 				r.errf("%v\n", err)
 				return 1
@@ -144,49 +159,35 @@ func (r *Runner) builtinCode(ctx context.Context, pos syntax.Pos, name string, a
 				break
 			}
 		}
-	case "break":
+	case "break", "continue":
 		if !r.inLoop {
-			r.errf("break is only useful in a loop")
+			r.errf("%s is only useful in a loop", name)
 			break
 		}
+		enclosing := &r.breakEnclosing
+		if name == "continue" {
+			enclosing = &r.contnEnclosing
+		}
 		switch len(args) {
 		case 0:
-			r.breakEnclosing = 1
+			*enclosing = 1
 		case 1:
 			if n, err := strconv.Atoi(args[0]); err == nil {
-				r.breakEnclosing = n
+				*enclosing = n
 				break
 			}
 			fallthrough
 		default:
-			r.errf("usage: break [n]\n")
+			r.errf("usage: %s [n]\n", name)
-			return 2
-		}
-	case "continue":
-		if !r.inLoop {
-			r.errf("continue is only useful in a loop")
-			break
-		}
-		switch len(args) {
-		case 0:
-			r.contnEnclosing = 1
-		case 1:
-			if n, err := strconv.Atoi(args[0]); err == nil {
-				r.contnEnclosing = n
-				break
-			}
-			fallthrough
-		default:
-			r.errf("usage: continue [n]\n")
 			return 2
 		}
 	case "pwd":
-		r.outf("%s\n", r.getVar("PWD"))
+		r.outf("%s\n", r.envGet("PWD"))
 	case "cd":
 		var path string
 		switch len(args) {
 		case 0:
-			path = r.getVar("HOME")
+			path = r.envGet("HOME")
 		case 1:
 			path = args[0]
 		default:
@@ -462,13 +463,13 @@ func (r *Runner) builtinCode(ctx context.Context, pos syntax.Pos, name string, a
 			args = append(args, "REPLY")
 		}
 
-		values := r.ifsFields(string(line), len(args), raw)
+		values := expand.ReadFields(r.ecfg, string(line), len(args), raw)
 		for i, name := range args {
 			val := ""
 			if i < len(values) {
 				val = values[i]
 			}
-			r.setVar(ctx, name, nil, Variable{Value: StringVal(val)})
+			r.setVar(name, nil, expand.Variable{Value: val})
 		}
 
 		return 0
@@ -478,7 +479,7 @@ func (r *Runner) builtinCode(ctx context.Context, pos syntax.Pos, name string, a
 			r.errf("getopts: usage: getopts optstring name [arg]\n")
 			return 2
 		}
-		optind, _ := strconv.Atoi(r.getVar("OPTIND"))
+		optind, _ := strconv.Atoi(r.envGet("OPTIND"))
 		if optind-1 != r.optState.argidx {
 			if optind < 1 {
 				optind = 1
@@ -499,7 +500,7 @@ func (r *Runner) builtinCode(ctx context.Context, pos syntax.Pos, name string, a
 
 		opt, optarg, done := r.optState.Next(optstr, args)
 
-		r.setVarString(ctx, name, string(opt))
+		r.setVarString(name, string(opt))
 		r.delVar("OPTARG")
 		switch {
 		case opt == '?' && diagnostics && !done:
@@ -508,11 +509,11 @@ func (r *Runner) builtinCode(ctx context.Context, pos syntax.Pos, name string, a
 			r.errf("getopts: option requires an argument -- %q\n", optarg)
 		default:
 			if optarg != "" {
-				r.setVarString(ctx, "OPTARG", optarg)
+				r.setVarString("OPTARG", optarg)
 			}
 		}
 		if optind-1 != r.optState.argidx {
-			r.setVarString(ctx, "OPTIND", strconv.FormatInt(int64(r.optState.argidx+1), 10))
+			r.setVarString("OPTIND", strconv.FormatInt(int64(r.optState.argidx+1), 10))
 		}
 
 		return oneIf(done)
@@ -559,6 +560,7 @@ func (r *Runner) builtinCode(ctx context.Context, pos syntax.Pos, name string, a
 				r.printOptLine(arg, *opt)
 			}
 		}
+		r.updateExpandOpts()
 
 	default:
 		// "trap", "umask", "alias", "unalias", "fg", "bg",
@@ -575,62 +577,6 @@ func (r *Runner) printOptLine(name string, enabled bool) {
 	r.outf("%s\t%s\n", name, status)
 }
 
-func (r *Runner) ifsFields(s string, n int, raw bool) []string {
-	type pos struct {
-		start, end int
-	}
-	var fpos []pos
-
-	runes := make([]rune, 0, len(s))
-	infield := false
-	esc := false
-	for _, c := range s {
-		if infield {
-			if r.ifsRune(c) && (raw || !esc) {
-				fpos[len(fpos)-1].end = len(runes)
-				infield = false
-			}
-		} else {
-			if !r.ifsRune(c) && (raw || !esc) {
-				fpos = append(fpos, pos{start: len(runes), end: -1})
-				infield = true
-			}
-		}
-		if c == '\\' {
-			if raw || esc {
-				runes = append(runes, c)
-			}
-			esc = !esc
-			continue
-		}
-		runes = append(runes, c)
-		esc = false
-	}
-	if len(fpos) == 0 {
-		return nil
-	}
-	if infield {
-		fpos[len(fpos)-1].end = len(runes)
-	}
-
-	switch {
-	case n == 1:
-		// include heading/trailing IFSs
-		fpos[0].start, fpos[0].end = 0, len(runes)
-		fpos = fpos[:1]
-	case n != -1 && n < len(fpos):
-		// combine to max n fields
-		fpos[n-1].end = fpos[len(fpos)-1].end
-		fpos = fpos[:n]
-	}
-
-	var fields = make([]string, len(fpos))
-	for i, p := range fpos {
-		fields[i] = string(runes[p.start:p.end])
-	}
-	return fields
-}
-
 func (r *Runner) readLine(raw bool) ([]byte, error) {
 	var line []byte
 	esc := false
@@ -675,7 +621,7 @@ func (r *Runner) changeDir(path string) int {
 	}
 	r.Dir = path
 	r.Vars["OLDPWD"] = r.Vars["PWD"]
-	r.Vars["PWD"] = Variable{Value: StringVal(path)}
+	r.Vars["PWD"] = expand.Variable{Value: path}
 	return 0
 }
 

vendor/mvdan.cc/sh/interp/doc.go

@@ -4,7 +4,4 @@
 // Package interp implements an interpreter that executes shell
 // programs. It aims to support POSIX, but its support is not complete
 // yet. It also supports some Bash features.
-//
-// This package is a work in progress and EXPERIMENTAL; its API is not
-// subject to the 1.x backwards compatibility guarantee.
 package interp

vendor/mvdan.cc/sh/interp/expand.go

@@ -1,508 +0,0 @@
-// Copyright (c) 2017, Daniel Martí <mvdan@mvdan.cc>
-// See LICENSE for licensing information
-
-package interp
-
-import (
-	"context"
-	"fmt"
-	"os"
-	"os/user"
-	"path/filepath"
-	"regexp"
-	"runtime"
-	"sort"
-	"strconv"
-	"strings"
-
-	"mvdan.cc/sh/syntax"
-)
-
-func (r *Runner) expandFormat(format string, args []string) (int, string, error) {
-	buf := r.strBuilder()
-	esc := false
-	var fmts []rune
-	initialArgs := len(args)
-
-	for _, c := range format {
-		switch {
-		case esc:
-			esc = false
-			switch c {
-			case 'n':
-				buf.WriteRune('\n')
-			case 'r':
-				buf.WriteRune('\r')
-			case 't':
-				buf.WriteRune('\t')
-			case '\\':
-				buf.WriteRune('\\')
-			default:
-				buf.WriteRune('\\')
-				buf.WriteRune(c)
-			}
-
-		case len(fmts) > 0:
-			switch c {
-			case '%':
-				buf.WriteByte('%')
-				fmts = nil
-			case 'c':
-				var b byte
-				if len(args) > 0 {
-					arg := ""
-					arg, args = args[0], args[1:]
-					if len(arg) > 0 {
-						b = arg[0]
-					}
-				}
-				buf.WriteByte(b)
-				fmts = nil
-			case '+', '-', ' ':
-				if len(fmts) > 1 {
-					return 0, "", fmt.Errorf("invalid format char: %c", c)
-				}
-				fmts = append(fmts, c)
-			case '0', '1', '2', '3', '4', '5', '6', '7', '8', '9':
-				fmts = append(fmts, c)
-			case 's', 'd', 'i', 'u', 'o', 'x':
-				arg := ""
-				if len(args) > 0 {
-					arg, args = args[0], args[1:]
-				}
-				var farg interface{} = arg
-				if c != 's' {
-					n, _ := strconv.ParseInt(arg, 0, 0)
-					if c == 'i' || c == 'd' {
-						farg = int(n)
-					} else {
-						farg = uint(n)
-					}
-					if c == 'i' || c == 'u' {
-						c = 'd'
-					}
-				}
-				fmts = append(fmts, c)
-				fmt.Fprintf(buf, string(fmts), farg)
-				fmts = nil
-			default:
-				return 0, "", fmt.Errorf("invalid format char: %c", c)
-			}
-		case c == '\\':
-			esc = true
-		case args != nil && c == '%':
-			// if args == nil, we are not doing format
-			// arguments
-			fmts = []rune{c}
-		default:
-			buf.WriteRune(c)
-		}
-	}
-	if len(fmts) > 0 {
-		return 0, "", fmt.Errorf("missing format char")
-	}
-	return initialArgs - len(args), buf.String(), nil
-}
-
-func (r *Runner) fieldJoin(parts []fieldPart) string {
-	switch len(parts) {
-	case 0:
-		return ""
-	case 1: // short-cut without a string copy
-		return parts[0].val
-	}
-	buf := r.strBuilder()
-	for _, part := range parts {
-		buf.WriteString(part.val)
-	}
-	return buf.String()
-}
-
-func (r *Runner) escapedGlobField(parts []fieldPart) (escaped string, glob bool) {
-	buf := r.strBuilder()
-	for _, part := range parts {
-		quoted := syntax.QuotePattern(part.val)
-		if quoted != part.val {
-			if part.quote > quoteNone {
-				buf.WriteString(quoted)
-			} else {
-				buf.WriteString(part.val)
-				glob = true
-			}
-		}
-	}
-	if glob { // only copy the string if it will be used
-		escaped = buf.String()
-	}
-	return escaped, glob
-}
-
-func (r *Runner) Fields(ctx context.Context, words ...*syntax.Word) ([]string, error) {
-	if !r.didReset {
-		r.Reset()
-	}
-	return r.fields(ctx, words...), r.err
-}
-
-func (r *Runner) fields(ctx context.Context, words ...*syntax.Word) []string {
-	fields := make([]string, 0, len(words))
-	baseDir := syntax.QuotePattern(r.Dir)
-	for _, word := range words {
-		for _, expWord := range syntax.ExpandBraces(word) {
-			for _, field := range r.wordFields(ctx, expWord.Parts) {
-				path, doGlob := r.escapedGlobField(field)
-				var matches []string
-				abs := filepath.IsAbs(path)
-				if doGlob && !r.opts[optNoGlob] {
-					if !abs {
-						path = filepath.Join(baseDir, path)
-					}
-					matches = glob(path, r.opts[optGlobStar])
-				}
-				if len(matches) == 0 {
-					fields = append(fields, r.fieldJoin(field))
-					continue
-				}
-				for _, match := range matches {
-					if !abs {
-						endSeparator := strings.HasSuffix(match, string(filepath.Separator))
-						match, _ = filepath.Rel(r.Dir, match)
-						if endSeparator {
-							match += string(filepath.Separator)
-						}
-					}
-					fields = append(fields, match)
-				}
-			}
-		}
-	}
-	return fields
-}
-
-func (r *Runner) loneWord(ctx context.Context, word *syntax.Word) string {
-	if word == nil {
-		return ""
-	}
-	field := r.wordField(ctx, word.Parts, quoteDouble)
-	return r.fieldJoin(field)
-}
-
-func (r *Runner) lonePattern(ctx context.Context, word *syntax.Word) string {
-	field := r.wordField(ctx, word.Parts, quoteSingle)
-	buf := r.strBuilder()
-	for _, part := range field {
-		if part.quote > quoteNone {
-			buf.WriteString(syntax.QuotePattern(part.val))
-		} else {
-			buf.WriteString(part.val)
-		}
-	}
-	return buf.String()
-}
-
-func (r *Runner) expandAssigns(ctx context.Context, as *syntax.Assign) []*syntax.Assign {
-	// Convert "declare $x" into "declare value".
-	// Don't use syntax.Parser here, as we only want the basic
-	// splitting by '='.
-	if as.Name != nil {
-		return []*syntax.Assign{as} // nothing to do
-	}
-	var asgns []*syntax.Assign
-	for _, field := range r.fields(ctx, as.Value) {
-		as := &syntax.Assign{}
-		parts := strings.SplitN(field, "=", 2)
-		as.Name = &syntax.Lit{Value: parts[0]}
-		if len(parts) == 1 {
-			as.Naked = true
-		} else {
-			as.Value = &syntax.Word{Parts: []syntax.WordPart{
-				&syntax.Lit{Value: parts[1]},
-			}}
-		}
-		asgns = append(asgns, as)
-	}
-	return asgns
-}
-
-type fieldPart struct {
-	val   string
-	quote quoteLevel
-}
-
-type quoteLevel uint
-
-const (
-	quoteNone quoteLevel = iota
-	quoteDouble
-	quoteSingle
-)
-
-func (r *Runner) wordField(ctx context.Context, wps []syntax.WordPart, ql quoteLevel) []fieldPart {
-	var field []fieldPart
-	for i, wp := range wps {
-		switch x := wp.(type) {
-		case *syntax.Lit:
-			s := x.Value
-			if i == 0 {
-				s = r.expandUser(s)
-			}
-			if ql == quoteDouble && strings.Contains(s, "\\") {
-				buf := r.strBuilder()
-				for i := 0; i < len(s); i++ {
-					b := s[i]
-					if b == '\\' && i+1 < len(s) {
-						switch s[i+1] {
-						case '\n': // remove \\\n
-							i++
-							continue
-						case '"', '\\', '$', '`': // special chars
-							continue
-						}
-					}
-					buf.WriteByte(b)
-				}
-				s = buf.String()
-			}
-			field = append(field, fieldPart{val: s})
-		case *syntax.SglQuoted:
-			fp := fieldPart{quote: quoteSingle, val: x.Value}
-			if x.Dollar {
-				_, fp.val, _ = r.expandFormat(fp.val, nil)
-			}
-			field = append(field, fp)
-		case *syntax.DblQuoted:
-			for _, part := range r.wordField(ctx, x.Parts, quoteDouble) {
-				part.quote = quoteDouble
-				field = append(field, part)
-			}
-		case *syntax.ParamExp:
-			field = append(field, fieldPart{val: r.paramExp(ctx, x)})
-		case *syntax.CmdSubst:
-			field = append(field, fieldPart{val: r.cmdSubst(ctx, x)})
-		case *syntax.ArithmExp:
-			field = append(field, fieldPart{
-				val: strconv.Itoa(r.arithm(ctx, x.X)),
-			})
-		default:
-			panic(fmt.Sprintf("unhandled word part: %T", x))
-		}
-	}
-	return field
-}
-
-func (r *Runner) cmdSubst(ctx context.Context, cs *syntax.CmdSubst) string {
-	r2 := r.sub()
-	buf := r.strBuilder()
-	r2.Stdout = buf
-	r2.stmts(ctx, cs.StmtList)
-	r.setErr(r2.err)
-	return strings.TrimRight(buf.String(), "\n")
-}
-
-func (r *Runner) wordFields(ctx context.Context, wps []syntax.WordPart) [][]fieldPart {
-	fields := r.fieldsAlloc[:0]
-	curField := r.fieldAlloc[:0]
-	allowEmpty := false
-	flush := func() {
-		if len(curField) == 0 {
-			return
-		}
-		fields = append(fields, curField)
-		curField = nil
-	}
-	splitAdd := func(val string) {
-		for i, field := range strings.FieldsFunc(val, r.ifsRune) {
-			if i > 0 {
-				flush()
-			}
-			curField = append(curField, fieldPart{val: field})
-		}
-	}
-	for i, wp := range wps {
-		switch x := wp.(type) {
-		case *syntax.Lit:
-			s := x.Value
-			if i == 0 {
-				s = r.expandUser(s)
-			}
-			if strings.Contains(s, "\\") {
-				buf := r.strBuilder()
-				for i := 0; i < len(s); i++ {
-					b := s[i]
-					if b == '\\' {
-						i++
-						b = s[i]
-					}
-					buf.WriteByte(b)
-				}
-				s = buf.String()
-			}
-			curField = append(curField, fieldPart{val: s})
-		case *syntax.SglQuoted:
-			allowEmpty = true
-			fp := fieldPart{quote: quoteSingle, val: x.Value}
-			if x.Dollar {
-				_, fp.val, _ = r.expandFormat(fp.val, nil)
-			}
-			curField = append(curField, fp)
-		case *syntax.DblQuoted:
-			allowEmpty = true
-			if len(x.Parts) == 1 {
-				pe, _ := x.Parts[0].(*syntax.ParamExp)
-				if elems := r.quotedElems(pe); elems != nil {
-					for i, elem := range elems {
-						if i > 0 {
-							flush()
-						}
-						curField = append(curField, fieldPart{
-							quote: quoteDouble,
-							val:   elem,
-						})
-					}
-					continue
-				}
-			}
-			for _, part := range r.wordField(ctx, x.Parts, quoteDouble) {
-				part.quote = quoteDouble
-				curField = append(curField, part)
-			}
-		case *syntax.ParamExp:
-			splitAdd(r.paramExp(ctx, x))
-		case *syntax.CmdSubst:
-			splitAdd(r.cmdSubst(ctx, x))
-		case *syntax.ArithmExp:
-			curField = append(curField, fieldPart{
-				val: strconv.Itoa(r.arithm(ctx, x.X)),
-			})
-		default:
-			panic(fmt.Sprintf("unhandled word part: %T", x))
-		}
-	}
-	flush()
-	if allowEmpty && len(fields) == 0 {
-		fields = append(fields, curField)
-	}
-	return fields
-}
-
-func (r *Runner) expandUser(field string) string {
-	if len(field) == 0 || field[0] != '~' {
-		return field
-	}
-	name := field[1:]
-	rest := ""
-	if i := strings.Index(name, "/"); i >= 0 {
-		rest = name[i:]
-		name = name[:i]
-	}
-	if name == "" {
-		return r.getVar("HOME") + rest
-	}
-	u, err := user.Lookup(name)
-	if err != nil {
-		return field
-	}
-	return u.HomeDir + rest
-}
-
-func match(pattern, name string) bool {
-	expr, err := syntax.TranslatePattern(pattern, true)
-	if err != nil {
-		return false
-	}
-	rx := regexp.MustCompile("^" + expr + "$")
-	return rx.MatchString(name)
-}
-
-func findAllIndex(pattern, name string, n int) [][]int {
-	expr, err := syntax.TranslatePattern(pattern, true)
-	if err != nil {
-		return nil
-	}
-	rx := regexp.MustCompile(expr)
-	return rx.FindAllStringIndex(name, n)
-}
-
-func hasGlob(path string) bool {
-	magicChars := `*?[`
-	if runtime.GOOS != "windows" {
-		magicChars = `*?[\`
-	}
-	return strings.ContainsAny(path, magicChars)
-}
-
-var rxGlobStar = regexp.MustCompile(".*")
-
-func glob(pattern string, globStar bool) []string {
-	parts := strings.Split(pattern, string(filepath.Separator))
-	matches := []string{"."}
-	if filepath.IsAbs(pattern) {
-		if parts[0] == "" {
-			// unix-like
-			matches[0] = string(filepath.Separator)
-		} else {
-			// windows (for some reason it won't work without the
-			// trailing separator)
-			matches[0] = parts[0] + string(filepath.Separator)
-		}
-		parts = parts[1:]
-	}
-	for _, part := range parts {
-		if part == "**" && globStar {
-			for i := range matches {
-				// "a/**" should match "a/ a/b a/b/c ..."; note
-				// how the zero-match case has a trailing
-				// separator.
-				matches[i] += string(filepath.Separator)
-			}
-			// expand all the possible levels of **
-			latest := matches
-			for {
-				var newMatches []string
-				for _, dir := range latest {
-					newMatches = globDir(dir, rxGlobStar, newMatches)
-				}
-				if len(newMatches) == 0 {
-					// not another level of directories to
-					// try; stop
-					break
-				}
-				matches = append(matches, newMatches...)
-				latest = newMatches
-			}
-			continue
-		}
-		expr, err := syntax.TranslatePattern(part, true)
-		if err != nil {
-			return nil
-		}
-		rx := regexp.MustCompile("^" + expr + "$")
-		var newMatches []string
-		for _, dir := range matches {
-			newMatches = globDir(dir, rx, newMatches)
-		}
-		matches = newMatches
-	}
-	return matches
-}
-
-func globDir(dir string, rx *regexp.Regexp, matches []string) []string {
-	d, err := os.Open(dir)
-	if err != nil {
-		return nil
-	}
-	defer d.Close()
-
-	names, _ := d.Readdirnames(-1)
-	sort.Strings(names)
-
-	for _, name := range names {
-		if !strings.HasPrefix(rx.String(), `^\.`) && name[0] == '.' {
-			continue
-		}
-		if rx.MatchString(name) {
-			matches = append(matches, filepath.Join(dir, name))
-		}
-	}
-	return matches
-}

vendor/mvdan.cc/sh/interp/interp.go

@@ -13,6 +13,7 @@ import (
 	"os"
 	"os/user"
 	"path/filepath"
+	"regexp"
 	"runtime"
 	"strings"
 	"sync"
@@ -20,6 +21,7 @@ import (
 
 	"golang.org/x/sync/errgroup"
 
+	"mvdan.cc/sh/expand"
 	"mvdan.cc/sh/syntax"
 )
 
@@ -46,10 +48,10 @@ func New(opts ...func(*Runner) error) (*Runner, error) {
 		}
 	}
 	if r.Exec == nil {
-		Module(nil)(r)
+		Module(ModuleExec(nil))(r)
 	}
 	if r.Open == nil {
-		Module(nil)(r)
+		Module(ModuleOpen(nil))(r)
 	}
 	if r.Stdout == nil || r.Stderr == nil {
 		StdIO(r.Stdin, r.Stdout, r.Stderr)(r)
@@ -57,12 +59,127 @@ func New(opts ...func(*Runner) error) (*Runner, error) {
 	return r, nil
 }
 
-// Env sets the interpreter's environment. If nil, the current process's
+func (r *Runner) fillExpandConfig(ctx context.Context) {
-// environment is used.
+	r.ectx = ctx
-func Env(env Environ) func(*Runner) error {
+	r.ecfg = &expand.Config{
+		Env: expandEnv{r},
+		CmdSubst: func(w io.Writer, cs *syntax.CmdSubst) error {
+			switch len(cs.Stmts) {
+			case 0: // nothing to do
+				return nil
+			case 1: // $(<file)
+				word := catShortcutArg(cs.Stmts[0])
+				if word == nil {
+					break
+				}
+				path := r.literal(word)
+				f, err := r.open(ctx, r.relPath(path), os.O_RDONLY, 0, true)
+				if err != nil {
+					return err
+				}
+				_, err = io.Copy(w, f)
+				return err
+			}
+			r2 := r.sub()
+			r2.Stdout = w
+			r2.stmts(ctx, cs.StmtList)
+			return r2.err
+		},
+		ReadDir: ioutil.ReadDir,
+	}
+	r.updateExpandOpts()
+}
+
+// catShortcutArg checks if a statement is of the form "$(<file)". The redirect
+// word is returned if there's a match, and nil otherwise.
+func catShortcutArg(stmt *syntax.Stmt) *syntax.Word {
+	if stmt.Cmd != nil || stmt.Negated || stmt.Background || stmt.Coprocess {
+		return nil
+	}
+	if len(stmt.Redirs) != 1 {
+		return nil
+	}
+	redir := stmt.Redirs[0]
+	if redir.Op != syntax.RdrIn {
+		return nil
+	}
+	return redir.Word
+}
+
+func (r *Runner) updateExpandOpts() {
+	r.ecfg.NoGlob = r.opts[optNoGlob]
+	r.ecfg.GlobStar = r.opts[optGlobStar]
+}
+
+func (r *Runner) expandErr(err error) {
+	switch err := err.(type) {
+	case nil:
+	case expand.UnsetParameterError:
+		r.errf("%s\n", err.Message)
+		r.exit = 1
+		r.setErr(ShellExitStatus(r.exit))
+	default:
+		r.setErr(err)
+		r.exit = 1
+	}
+}
+
+func (r *Runner) arithm(expr syntax.ArithmExpr) int {
+	n, err := expand.Arithm(r.ecfg, expr)
+	r.expandErr(err)
+	return n
+}
+
+func (r *Runner) fields(words ...*syntax.Word) []string {
+	strs, err := expand.Fields(r.ecfg, words...)
+	r.expandErr(err)
+	return strs
+}
+
+func (r *Runner) literal(word *syntax.Word) string {
+	str, err := expand.Literal(r.ecfg, word)
+	r.expandErr(err)
+	return str
+}
+
+func (r *Runner) document(word *syntax.Word) string {
+	str, err := expand.Document(r.ecfg, word)
+	r.expandErr(err)
+	return str
+}
+
+func (r *Runner) pattern(word *syntax.Word) string {
+	str, err := expand.Pattern(r.ecfg, word)
+	r.expandErr(err)
+	return str
+}
+
+// expandEnv exposes Runner's variables to the expand package.
+type expandEnv struct {
+	r *Runner
+}
+
+func (e expandEnv) Get(name string) expand.Variable {
+	return e.r.lookupVar(name)
+}
+func (e expandEnv) Set(name string, vr expand.Variable) {
+	e.r.setVarInternal(name, vr)
+}
+func (e expandEnv) Each(fn func(name string, vr expand.Variable) bool) {
+	e.r.Env.Each(fn)
+	for name, vr := range e.r.Vars {
+		if !fn(name, vr) {
+			return
+		}
+	}
+}
+
+// Env sets the interpreter's environment. If nil, a copy of the current
+// process's environment is used.
+func Env(env expand.Environ) func(*Runner) error {
 	return func(r *Runner) error {
 		if env == nil {
-			env, _ = EnvFromList(os.Environ())
+			env = expand.ListEnviron(os.Environ()...)
 		}
 		r.Env = env
 		return nil
@@ -143,6 +260,7 @@ func Params(args ...string) func(*Runner) error {
 			args = args[1:]
 		}
 		r.Params = args
+		r.updateExpandOpts()
 		return nil
 	}
 }
@@ -203,7 +321,7 @@ func StdIO(in io.Reader, out, err io.Writer) func(*Runner) error {
 type Runner struct {
 	// Env specifies the environment of the interpreter, which must be
 	// non-nil.
-	Env Environ
+	Env expand.Environ
 
 	// Dir specifies the working directory of the command, which must be an
 	// absolute path.
@@ -223,12 +341,15 @@ type Runner struct {
 	Stdout io.Writer
 	Stderr io.Writer
 
-	// Separate maps, note that bash allows a name to be both a var
+	// Separate maps - note that bash allows a name to be both a var and a
-	// and a func simultaneously
+	// func simultaneously
-	// TODO: merge into Env?
+
-	Vars  map[string]Variable
+	Vars  map[string]expand.Variable
 	Funcs map[string]*syntax.Stmt
 
+	ecfg *expand.Config
+	ectx context.Context // just so that Runner.Sub can use it again
+
 	// didReset remembers whether the runner has ever been reset. This is
 	// used so that Reset is automatically called when running any program
 	// or node for the first time on a Runner.
@@ -239,7 +360,7 @@ type Runner struct {
 	filename string // only if Node was a File
 
 	// like Vars, but local to a func i.e. "local foo=bar"
-	funcVars map[string]Variable
+	funcVars map[string]expand.Variable
 
 	// like Vars, but local to a cmd i.e. "foo=bar prog args..."
 	cmdVars map[string]string
@@ -262,9 +383,6 @@ type Runner struct {
 
 	optState getopts
 
-	ifsJoin string
-	ifsRune func(rune) bool
-
 	// keepRedirs is used so that "exec" can make any redirections
 	// apply to the current shell, and not just the command.
 	keepRedirs bool
@@ -281,17 +399,6 @@ type Runner struct {
 	// On Windows, the kill signal is always sent immediately,
 	// because Go doesn't currently support sending Interrupt on Windows.
 	KillTimeout time.Duration
-
-	fieldAlloc  [4]fieldPart
-	fieldsAlloc [4][]fieldPart
-	bufferAlloc bytes.Buffer
-	oneWord     [1]*syntax.Word
-}
-
-func (r *Runner) strBuilder() *bytes.Buffer {
-	b := &r.bufferAlloc
-	b.Reset()
-	return b
 }
 
 func (r *Runner) optByFlag(flag string) *bool {
@@ -380,7 +487,7 @@ func (r *Runner) Reset() {
 		usedNew:  r.usedNew,
 	}
 	if r.Vars == nil {
-		r.Vars = make(map[string]Variable)
+		r.Vars = make(map[string]expand.Variable)
 	} else {
 		for k := range r.Vars {
 			delete(r.Vars, k)
@@ -393,29 +500,22 @@ func (r *Runner) Reset() {
 			delete(r.cmdVars, k)
 		}
 	}
-	if _, ok := r.Env.Get("HOME"); !ok {
+	if vr := r.Env.Get("HOME"); !vr.IsSet() {
 		u, _ := user.Current()
-		r.Vars["HOME"] = Variable{Value: StringVal(u.HomeDir)}
+		r.Vars["HOME"] = expand.Variable{Value: u.HomeDir}
 	}
-	r.Vars["PWD"] = Variable{Value: StringVal(r.Dir)}
+	r.Vars["PWD"] = expand.Variable{Value: r.Dir}
-	r.Vars["IFS"] = Variable{Value: StringVal(" \t\n")}
+	r.Vars["IFS"] = expand.Variable{Value: " \t\n"}
-	r.ifsUpdated()
+	r.Vars["OPTIND"] = expand.Variable{Value: "1"}
-	r.Vars["OPTIND"] = Variable{Value: StringVal("1")}
 
 	if runtime.GOOS == "windows" {
 		// convert $PATH to a unix path list
-		path, _ := r.Env.Get("PATH")
+		path := r.Env.Get("PATH").String()
 		path = strings.Join(filepath.SplitList(path), ":")
-		r.Vars["PATH"] = Variable{Value: StringVal(path)}
+		r.Vars["PATH"] = expand.Variable{Value: path}
 	}
 
 	r.dirStack = append(r.dirStack, r.Dir)
-	if r.Exec == nil {
-		r.Exec = DefaultExec
-	}
-	if r.Open == nil {
-		r.Open = DefaultOpen
-	}
 	if r.KillTimeout == 0 {
 		r.KillTimeout = 2 * time.Second
 	}
@@ -424,23 +524,26 @@ func (r *Runner) Reset() {
 
 func (r *Runner) modCtx(ctx context.Context) context.Context {
 	mc := ModuleCtx{
-		Env:         r.Env,
 		Dir:         r.Dir,
 		Stdin:       r.Stdin,
 		Stdout:      r.Stdout,
 		Stderr:      r.Stderr,
 		KillTimeout: r.KillTimeout,
 	}
-	mc.Env = r.Env.Copy()
+	oenv := overlayEnviron{
+		parent: r.Env,
+		values: make(map[string]expand.Variable),
+	}
 	for name, vr := range r.Vars {
-		if !vr.Exported {
+		oenv.Set(name, vr)
-			continue
-		}
-		mc.Env.Set(name, r.varStr(vr, 0))
 	}
-	for name, val := range r.cmdVars {
+	for name, vr := range r.funcVars {
-		mc.Env.Set(name, val)
+		oenv.Set(name, vr)
 	}
+	for name, value := range r.cmdVars {
+		oenv.Set(name, expand.Variable{Exported: true, Value: value})
+	}
+	mc.Env = oenv
 	return context.WithValue(ctx, moduleCtxKey{}, mc)
 }
 
@@ -471,6 +574,7 @@ func (r *Runner) Run(ctx context.Context, node syntax.Node) error {
 	if !r.didReset {
 		r.Reset()
 	}
+	r.fillExpandConfig(ctx)
 	r.err = nil
 	r.filename = ""
 	switch x := node.(type) {
@@ -482,7 +586,7 @@ func (r *Runner) Run(ctx context.Context, node syntax.Node) error {
 	case syntax.Command:
 		r.cmd(ctx, x)
 	default:
-		return fmt.Errorf("Node can only be File, Stmt, or Command: %T", x)
+		return fmt.Errorf("node can only be File, Stmt, or Command: %T", x)
 	}
 	if r.exit > 0 {
 		r.setErr(ExitStatus(r.exit))
@@ -564,6 +668,7 @@ func (r *Runner) sub() *Runner {
 	// Keep in sync with the Runner type. Manually copy fields, to not copy
 	// sensitive ones like errgroup.Group, and to do deep copies of slices.
 	r2 := &Runner{
+		Env:         r.Env,
 		Dir:         r.Dir,
 		Params:      r.Params,
 		Exec:        r.Exec,
@@ -576,19 +681,20 @@ func (r *Runner) sub() *Runner {
 		filename:    r.filename,
 		opts:        r.opts,
 	}
-	// TODO: perhaps we could do a lazy copy here, or some sort of
+	r2.Vars = make(map[string]expand.Variable, len(r.Vars))
-	// overlay to avoid copying all the time
-	r2.Env = r.Env.Copy()
-	r2.Vars = make(map[string]Variable, len(r.Vars))
 	for k, v := range r.Vars {
 		r2.Vars[k] = v
 	}
+	r2.funcVars = make(map[string]expand.Variable, len(r.funcVars))
+	for k, v := range r.funcVars {
+		r2.funcVars[k] = v
+	}
 	r2.cmdVars = make(map[string]string, len(r.cmdVars))
 	for k, v := range r.cmdVars {
 		r2.cmdVars[k] = v
 	}
 	r2.dirStack = append([]string(nil), r.dirStack...)
-	r2.ifsUpdated()
+	r2.fillExpandConfig(r.ectx)
 	r2.didReset = true
 	return r2
 }
@@ -606,23 +712,19 @@ func (r *Runner) cmd(ctx context.Context, cm syntax.Command) {
 		r.exit = r2.exit
 		r.setErr(r2.err)
 	case *syntax.CallExpr:
-		fields := r.fields(ctx, x.Args...)
+		fields := r.fields(x.Args...)
 		if len(fields) == 0 {
 			for _, as := range x.Assigns {
-				vr, _ := r.lookupVar(as.Name.Value)
+				vr := r.lookupVar(as.Name.Value)
-				vr.Value = r.assignVal(ctx, as, "")
+				vr.Value = r.assignVal(as, "")
-				r.setVar(ctx, as.Name.Value, as.Index, vr)
+				r.setVar(as.Name.Value, as.Index, vr)
 			}
 			break
 		}
 		for _, as := range x.Assigns {
-			val := r.assignVal(ctx, as, "")
+			val := r.assignVal(as, "")
 			// we know that inline vars must be strings
-			r.cmdVars[as.Name.Value] = string(val.(StringVal))
+			r.cmdVars[as.Name.Value] = val.(string)
-			if as.Name.Value == "IFS" {
-				r.ifsUpdated()
-				defer r.ifsUpdated()
-			}
 		}
 		r.call(ctx, x.Args[0].Pos(), fields)
 		// cmdVars can be nuked here, as they are never useful
@@ -689,37 +791,37 @@ func (r *Runner) cmd(ctx context.Context, cm syntax.Command) {
 		switch y := x.Loop.(type) {
 		case *syntax.WordIter:
 			name := y.Name.Value
-			for _, field := range r.fields(ctx, y.Items...) {
+			for _, field := range r.fields(y.Items...) {
-				r.setVarString(ctx, name, field)
+				r.setVarString(name, field)
 				if r.loopStmtsBroken(ctx, x.Do) {
 					break
 				}
 			}
 		case *syntax.CStyleLoop:
-			r.arithm(ctx, y.Init)
+			r.arithm(y.Init)
-			for r.arithm(ctx, y.Cond) != 0 {
+			for r.arithm(y.Cond) != 0 {
 				if r.loopStmtsBroken(ctx, x.Do) {
 					break
 				}
-				r.arithm(ctx, y.Post)
+				r.arithm(y.Post)
 			}
 		}
 	case *syntax.FuncDecl:
 		r.setFunc(x.Name.Value, x.Body)
 	case *syntax.ArithmCmd:
-		r.exit = oneIf(r.arithm(ctx, x.X) == 0)
+		r.exit = oneIf(r.arithm(x.X) == 0)
 	case *syntax.LetClause:
 		var val int
 		for _, expr := range x.Exprs {
-			val = r.arithm(ctx, expr)
+			val = r.arithm(expr)
 		}
 		r.exit = oneIf(val == 0)
 	case *syntax.CaseClause:
-		str := r.loneWord(ctx, x.Word)
+		str := r.literal(x.Word)
 		for _, ci := range x.Items {
 			for _, word := range ci.Patterns {
-				pat := r.lonePattern(ctx, word)
+				pattern := r.pattern(word)
-				if match(pat, str) {
+				if match(pattern, str) {
 					r.stmts(ctx, ci.StmtList)
 					return
 				}
@@ -732,13 +834,13 @@ func (r *Runner) cmd(ctx context.Context, cm syntax.Command) {
 			r.exit = 1
 		}
 	case *syntax.DeclClause:
-		local := false
+		local, global := false, false
 		var modes []string
 		valType := ""
 		switch x.Variant.Value {
 		case "declare":
-			// When used in a function, "declare" acts as
+			// When used in a function, "declare" acts as "local"
-			// "local" unless the "-g" option is used.
+			// unless the "-g" option is used.
 			local = r.inFunc
 		case "local":
 			if !r.inFunc {
@@ -755,13 +857,13 @@ func (r *Runner) cmd(ctx context.Context, cm syntax.Command) {
 			modes = append(modes, "-n")
 		}
 		for _, opt := range x.Opts {
-			switch s := r.loneWord(ctx, opt); s {
+			switch s := r.literal(opt); s {
 			case "-x", "-r", "-n":
 				modes = append(modes, s)
 			case "-a", "-A":
 				valType = s
 			case "-g":
-				local = false
+				global = true
 			default:
 				r.errf("declare: invalid option %q\n", s)
 				r.exit = 2
@@ -769,11 +871,20 @@ func (r *Runner) cmd(ctx context.Context, cm syntax.Command) {
 			}
 		}
 		for _, as := range x.Assigns {
-			for _, as := range r.expandAssigns(ctx, as) {
+			for _, as := range r.flattenAssign(as) {
 				name := as.Name.Value
-				vr, _ := r.lookupVar(as.Name.Value)
+				if !syntax.ValidName(name) {
-				vr.Value = r.assignVal(ctx, as, valType)
+					r.errf("declare: invalid name %q\n", name)
-				vr.Local = local
+					r.exit = 1
+					return
+				}
+				vr := r.lookupVar(as.Name.Value)
+				vr.Value = r.assignVal(as, valType)
+				if global {
+					vr.Local = false
+				} else if local {
+					vr.Local = true
+				}
 				for _, mode := range modes {
 					switch mode {
 					case "-x":
@@ -784,7 +895,7 @@ func (r *Runner) cmd(ctx context.Context, cm syntax.Command) {
 						vr.NameRef = true
 					}
 				}
-				r.setVar(ctx, name, as.Index, vr)
+				r.setVar(name, as.Index, vr)
 			}
 		}
 	case *syntax.TimeClause:
@@ -808,6 +919,39 @@ func (r *Runner) cmd(ctx context.Context, cm syntax.Command) {
 	}
 }
 
+func (r *Runner) flattenAssign(as *syntax.Assign) []*syntax.Assign {
+	// Convert "declare $x" into "declare value".
+	// Don't use syntax.Parser here, as we only want the basic
+	// splitting by '='.
+	if as.Name != nil {
+		return []*syntax.Assign{as} // nothing to do
+	}
+	var asgns []*syntax.Assign
+	for _, field := range r.fields(as.Value) {
+		as := &syntax.Assign{}
+		parts := strings.SplitN(field, "=", 2)
+		as.Name = &syntax.Lit{Value: parts[0]}
+		if len(parts) == 1 {
+			as.Naked = true
+		} else {
+			as.Value = &syntax.Word{Parts: []syntax.WordPart{
+				&syntax.Lit{Value: parts[1]},
+			}}
+		}
+		asgns = append(asgns, as)
+	}
+	return asgns
+}
+
+func match(pattern, name string) bool {
+	expr, err := syntax.TranslatePattern(pattern, true)
+	if err != nil {
+		return false
+	}
+	rx := regexp.MustCompile("^" + expr + "$")
+	return rx.MatchString(name)
+}
+
 func elapsedString(d time.Duration, posix bool) string {
 	if posix {
 		return fmt.Sprintf("%.2f", d.Seconds())
@@ -823,10 +967,42 @@ func (r *Runner) stmts(ctx context.Context, sl syntax.StmtList) {
 	}
 }
 
+func (r *Runner) hdocReader(rd *syntax.Redirect) io.Reader {
+	if rd.Op != syntax.DashHdoc {
+		hdoc := r.document(rd.Hdoc)
+		return strings.NewReader(hdoc)
+	}
+	var buf bytes.Buffer
+	var cur []syntax.WordPart
+	flushLine := func() {
+		if buf.Len() > 0 {
+			buf.WriteByte('\n')
+		}
+		buf.WriteString(r.document(&syntax.Word{Parts: cur}))
+		cur = cur[:0]
+	}
+	for _, wp := range rd.Hdoc.Parts {
+		lit, ok := wp.(*syntax.Lit)
+		if !ok {
+			cur = append(cur, wp)
+			continue
+		}
+		for i, part := range strings.Split(lit.Value, "\n") {
+			if i > 0 {
+				flushLine()
+				cur = cur[:0]
+			}
+			part = strings.TrimLeft(part, "\t")
+			cur = append(cur, &syntax.Lit{Value: part})
+		}
+	}
+	flushLine()
+	return &buf
+}
+
 func (r *Runner) redir(ctx context.Context, rd *syntax.Redirect) (io.Closer, error) {
 	if rd.Hdoc != nil {
-		hdoc := r.loneWord(ctx, rd.Hdoc)
+		r.Stdin = r.hdocReader(rd)
-		r.Stdin = strings.NewReader(hdoc)
 		return nil, nil
 	}
 	orig := &r.Stdout
@@ -837,7 +1013,7 @@ func (r *Runner) redir(ctx context.Context, rd *syntax.Redirect) (io.Closer, err
 			orig = &r.Stderr
 		}
 	}
-	arg := r.loneWord(ctx, rd.Word)
+	arg := r.literal(rd.Word)
 	switch rd.Op {
 	case syntax.WordHdoc:
 		r.Stdin = strings.NewReader(arg + "\n")
@@ -860,9 +1036,9 @@ func (r *Runner) redir(ctx context.Context, rd *syntax.Redirect) (io.Closer, err
 	mode := os.O_RDONLY
 	switch rd.Op {
 	case syntax.AppOut, syntax.AppAll:
-		mode = os.O_RDWR | os.O_CREATE | os.O_APPEND
+		mode = os.O_WRONLY | os.O_CREATE | os.O_APPEND
 	case syntax.RdrOut, syntax.RdrAll:
-		mode = os.O_RDWR | os.O_CREATE | os.O_TRUNC
+		mode = os.O_WRONLY | os.O_CREATE | os.O_TRUNC
 	}
 	f, err := r.open(ctx, r.relPath(arg), mode, 0644, true)
 	if err != nil {
@@ -1039,7 +1215,7 @@ func splitList(path string) []string {
 }
 
 func (r *Runner) lookPath(file string) string {
-	pathList := splitList(r.getVar("PATH"))
+	pathList := splitList(r.envGet("PATH"))
 	chars := `/`
 	if runtime.GOOS == "windows" {
 		chars = `:\/`
@@ -1070,7 +1246,7 @@ func (r *Runner) pathExts() []string {
 	if runtime.GOOS != "windows" {
 		return nil
 	}
-	pathext := r.getVar("PATHEXT")
+	pathext := r.envGet("PATHEXT")
 	if pathext == "" {
 		return []string{".com", ".exe", ".bat", ".cmd"}
 	}

vendor/mvdan.cc/sh/interp/module.go

@@ -13,6 +13,8 @@ import (
 	"strings"
 	"syscall"
 	"time"
+
+	"mvdan.cc/sh/expand"
 )
 
 // FromModuleContext returns the ModuleCtx value stored in ctx, if any.
@@ -27,7 +29,7 @@ type moduleCtxKey struct{}
 // It contains some of the current state of the Runner, as well as some fields
 // necessary to implement some of the modules.
 type ModuleCtx struct {
-	Env         Environ
+	Env         expand.Environ
 	Dir         string
 	Stdin       io.Reader
 	Stdout      io.Writer

vendor/mvdan.cc/sh/interp/test.go

@@ -19,22 +19,22 @@ import (
 func (r *Runner) bashTest(ctx context.Context, expr syntax.TestExpr, classic bool) string {
 	switch x := expr.(type) {
 	case *syntax.Word:
-		return r.loneWord(ctx, x)
+		return r.literal(x)
 	case *syntax.ParenTest:
 		return r.bashTest(ctx, x.X, classic)
 	case *syntax.BinaryTest:
 		switch x.Op {
 		case syntax.TsMatch, syntax.TsNoMatch:
-			str := r.loneWord(ctx, x.X.(*syntax.Word))
+			str := r.literal(x.X.(*syntax.Word))
 			yw := x.Y.(*syntax.Word)
 			if classic { // test, [
-				lit := r.loneWord(ctx, yw)
+				lit := r.literal(yw)
 				if (str == lit) == (x.Op == syntax.TsMatch) {
 					return "1"
 				}
 			} else { // [[
-				pat := r.lonePattern(ctx, yw)
+				pattern := r.pattern(yw)
-				if match(pat, str) == (x.Op == syntax.TsMatch) {
+				if match(pattern, str) == (x.Op == syntax.TsMatch) {
 					return "1"
 				}
 			}
@@ -173,11 +173,9 @@ func (r *Runner) unTest(ctx context.Context, op syntax.UnTestOperator, x string)
 		}
 		return false
 	case syntax.TsVarSet:
-		_, e := r.lookupVar(x)
+		return r.lookupVar(x).IsSet()
-		return e
 	case syntax.TsRefVar:
-		v, _ := r.lookupVar(x)
+		return r.lookupVar(x).NameRef
-		return v.NameRef
 	case syntax.TsNot:
 		return x == ""
 	default:

vendor/mvdan.cc/sh/interp/vars.go

@@ -4,259 +4,137 @@
 package interp
 
 import (
-	"context"
+	"os"
-	"fmt"
 	"runtime"
-	"sort"
+	"strconv"
 	"strings"
 
+	"mvdan.cc/sh/expand"
 	"mvdan.cc/sh/syntax"
 )
 
-type Environ interface {
+type overlayEnviron struct {
-	Get(name string) (value string, exists bool)
+	parent expand.Environ
-	Set(name, value string)
+	values map[string]expand.Variable
-	Delete(name string)
-	Names() []string
-	Copy() Environ
 }
 
-type mapEnviron struct {
+func (o overlayEnviron) Get(name string) expand.Variable {
-	names  []string
+	if vr, ok := o.values[name]; ok {
-	values map[string]string
+		return vr
-}
-
-func (m *mapEnviron) Get(name string) (string, bool) {
-	val, ok := m.values[name]
-	return val, ok
-}
-
-func (m *mapEnviron) Set(name, value string) {
-	_, ok := m.values[name]
-	if !ok {
-		m.names = append(m.names, name)
-		sort.Strings(m.names)
 	}
-	m.values[name] = value
+	return o.parent.Get(name)
 }
 
-func (m *mapEnviron) Delete(name string) {
+func (o overlayEnviron) Set(name string, vr expand.Variable) {
-	if _, ok := m.values[name]; !ok {
+	o.values[name] = vr
-		return
+}
-	}
+
-	delete(m.values, name)
+func (o overlayEnviron) Each(f func(name string, vr expand.Variable) bool) {
-	for i, iname := range m.names {
+	o.parent.Each(f)
-		if iname == name {
+	for name, vr := range o.values {
-			m.names = append(m.names[:i], m.names[i+1:]...)
+		if !f(name, vr) {
 			return
 		}
 	}
 }
 
-func (m *mapEnviron) Names() []string {
+func execEnv(env expand.Environ) []string {
-	return m.names
+	list := make([]string, 0, 32)
-}
+	env.Each(func(name string, vr expand.Variable) bool {
-
+		if vr.Exported {
-func (m *mapEnviron) Copy() Environ {
+			list = append(list, name+"="+vr.String())
-	m2 := &mapEnviron{
+		}
-		names:  make([]string, len(m.names)),
+		return true
-		values: make(map[string]string, len(m.values)),
+	})
-	}
-	copy(m2.names, m.names)
-	for name, val := range m.values {
-		m2.values[name] = val
-	}
-	return m2
-}
-
-func execEnv(env Environ) []string {
-	names := env.Names()
-	list := make([]string, len(names))
-	for i, name := range names {
-		val, _ := env.Get(name)
-		list[i] = name + "=" + val
-	}
 	return list
 }
 
-func EnvFromList(list []string) (Environ, error) {
+func (r *Runner) lookupVar(name string) expand.Variable {
-	m := mapEnviron{
-		names:  make([]string, 0, len(list)),
-		values: make(map[string]string, len(list)),
-	}
-	for _, kv := range list {
-		i := strings.IndexByte(kv, '=')
-		if i < 0 {
-			return nil, fmt.Errorf("env not in the form key=value: %q", kv)
-		}
-		name, val := kv[:i], kv[i+1:]
-		if runtime.GOOS == "windows" {
-			name = strings.ToUpper(name)
-		}
-		m.names = append(m.names, name)
-		m.values[name] = val
-	}
-	sort.Strings(m.names)
-	return &m, nil
-}
-
-type FuncEnviron func(string) string
-
-func (f FuncEnviron) Get(name string) (string, bool) {
-	val := f(name)
-	return val, val != ""
-}
-
-func (f FuncEnviron) Set(name, value string) {}
-func (f FuncEnviron) Delete(name string)     {}
-func (f FuncEnviron) Names() []string        { return nil }
-func (f FuncEnviron) Copy() Environ          { return f }
-
-type Variable struct {
-	Local    bool
-	Exported bool
-	ReadOnly bool
-	NameRef  bool
-	Value    VarValue
-}
-
-// VarValue is one of:
-//
-//     StringVal
-//     IndexArray
-//     AssocArray
-type VarValue interface{}
-
-type StringVal string
-
-type IndexArray []string
-
-type AssocArray map[string]string
-
-func (r *Runner) lookupVar(name string) (Variable, bool) {
 	if name == "" {
 		panic("variable name must not be empty")
 	}
-	if val, e := r.cmdVars[name]; e {
+	var value interface{}
-		return Variable{Value: StringVal(val)}, true
+	switch name {
+	case "#":
+		value = strconv.Itoa(len(r.Params))
+	case "@", "*":
+		value = r.Params
+	case "?":
+		value = strconv.Itoa(r.exit)
+	case "$":
+		value = strconv.Itoa(os.Getpid())
+	case "PPID":
+		value = strconv.Itoa(os.Getppid())
+	case "DIRSTACK":
+		value = r.dirStack
+	case "0":
+		if r.filename != "" {
+			value = r.filename
+		} else {
+			value = "gosh"
+		}
+	case "1", "2", "3", "4", "5", "6", "7", "8", "9":
+		i := int(name[0] - '1')
+		if i < len(r.Params) {
+			value = r.Params[i]
+		} else {
+			value = ""
+		}
+	}
+	if value != nil {
+		return expand.Variable{Value: value}
+	}
+	if value, e := r.cmdVars[name]; e {
+		return expand.Variable{Value: value}
 	}
 	if vr, e := r.funcVars[name]; e {
-		return vr, true
+		vr.Local = true
+		return vr
 	}
 	if vr, e := r.Vars[name]; e {
-		return vr, true
+		return vr
 	}
-	if str, e := r.Env.Get(name); e {
+	if vr := r.Env.Get(name); vr.IsSet() {
-		return Variable{Value: StringVal(str)}, true
+		return vr
 	}
 	if runtime.GOOS == "windows" {
 		upper := strings.ToUpper(name)
-		if str, e := r.Env.Get(upper); e {
+		if vr := r.Env.Get(upper); vr.IsSet() {
-			return Variable{Value: StringVal(str)}, true
+			return vr
 		}
 	}
 	if r.opts[optNoUnset] {
 		r.errf("%s: unbound variable\n", name)
 		r.setErr(ShellExitStatus(1))
 	}
-	return Variable{}, false
+	return expand.Variable{}
 }
 
-func (r *Runner) getVar(name string) string {
+func (r *Runner) envGet(name string) string {
-	val, _ := r.lookupVar(name)
+	return r.lookupVar(name).String()
-	return r.varStr(val, 0)
 }
 
 func (r *Runner) delVar(name string) {
-	val, _ := r.lookupVar(name)
+	vr := r.lookupVar(name)
-	if val.ReadOnly {
+	if vr.ReadOnly {
 		r.errf("%s: readonly variable\n", name)
 		r.exit = 1
 		return
 	}
-	delete(r.Vars, name)
+	if vr.Local {
-	delete(r.funcVars, name)
+		// don't overwrite a non-local var with the same name
-	delete(r.cmdVars, name)
+		r.funcVars[name] = expand.Variable{}
-	r.Env.Delete(name)
+	} else {
+		r.Vars[name] = expand.Variable{} // to not query r.Env
+	}
 }
 
-// maxNameRefDepth defines the maximum number of times to follow
+func (r *Runner) setVarString(name, value string) {
-// references when expanding a variable. Otherwise, simple name
+	r.setVar(name, nil, expand.Variable{Value: value})
-// reference loops could crash the interpreter quite easily.
-const maxNameRefDepth = 100
-
-func (r *Runner) varStr(vr Variable, depth int) string {
-	if depth > maxNameRefDepth {
-		return ""
-	}
-	switch x := vr.Value.(type) {
-	case StringVal:
-		if vr.NameRef {
-			vr, _ = r.lookupVar(string(x))
-			return r.varStr(vr, depth+1)
-		}
-		return string(x)
-	case IndexArray:
-		if len(x) > 0 {
-			return x[0]
-		}
-	case AssocArray:
-		// nothing to do
-	}
-	return ""
 }
 
-func (r *Runner) varInd(ctx context.Context, vr Variable, e syntax.ArithmExpr, depth int) string {
+func (r *Runner) setVarInternal(name string, vr expand.Variable) {
-	if depth > maxNameRefDepth {
+	if _, ok := vr.Value.(string); ok {
-		return ""
-	}
-	switch x := vr.Value.(type) {
-	case StringVal:
-		if vr.NameRef {
-			vr, _ = r.lookupVar(string(x))
-			return r.varInd(ctx, vr, e, depth+1)
-		}
-		if r.arithm(ctx, e) == 0 {
-			return string(x)
-		}
-	case IndexArray:
-		switch anyOfLit(e, "@", "*") {
-		case "@":
-			return strings.Join(x, " ")
-		case "*":
-			return strings.Join(x, r.ifsJoin)
-		}
-		i := r.arithm(ctx, e)
-		if len(x) > 0 {
-			return x[i]
-		}
-	case AssocArray:
-		if lit := anyOfLit(e, "@", "*"); lit != "" {
-			var strs IndexArray
-			keys := make([]string, 0, len(x))
-			for k := range x {
-				keys = append(keys, k)
-			}
-			sort.Strings(keys)
-			for _, k := range keys {
-				strs = append(strs, x[k])
-			}
-			if lit == "*" {
-				return strings.Join(strs, r.ifsJoin)
-			}
-			return strings.Join(strs, " ")
-		}
-		return x[r.loneWord(ctx, e.(*syntax.Word))]
-	}
-	return ""
-}
-
-func (r *Runner) setVarString(ctx context.Context, name, val string) {
-	r.setVar(ctx, name, nil, Variable{Value: StringVal(val)})
-}
-
-func (r *Runner) setVarInternal(name string, vr Variable) {
-	if _, ok := vr.Value.(StringVal); ok {
 		if r.opts[optAllExport] {
 			vr.Exported = true
 		}
@@ -265,28 +143,31 @@ func (r *Runner) setVarInternal(name string, vr Variable) {
 	}
 	if vr.Local {
 		if r.funcVars == nil {
-			r.funcVars = make(map[string]Variable)
+			r.funcVars = make(map[string]expand.Variable)
 		}
 		r.funcVars[name] = vr
 	} else {
 		r.Vars[name] = vr
 	}
-	if name == "IFS" {
-		r.ifsUpdated()
-	}
 }
 
-func (r *Runner) setVar(ctx context.Context, name string, index syntax.ArithmExpr, vr Variable) {
+func (r *Runner) setVar(name string, index syntax.ArithmExpr, vr expand.Variable) {
-	cur, _ := r.lookupVar(name)
+	cur := r.lookupVar(name)
 	if cur.ReadOnly {
 		r.errf("%s: readonly variable\n", name)
 		r.exit = 1
 		return
 	}
-	_, isIndexArray := cur.Value.(IndexArray)
+	if name2, var2 := cur.Resolve(r.Env); name2 != "" {
-	_, isAssocArray := cur.Value.(AssocArray)
+		name = name2
+		cur = var2
+		vr.NameRef = false
+		cur.NameRef = false
+	}
+	_, isIndexArray := cur.Value.([]string)
+	_, isAssocArray := cur.Value.(map[string]string)
 
-	if _, ok := vr.Value.(StringVal); ok && index == nil {
+	if _, ok := vr.Value.(string); ok && index == nil {
 		// When assigning a string to an array, fall back to the
 		// zero value for the index.
 		if isIndexArray {
@@ -304,33 +185,33 @@ func (r *Runner) setVar(ctx context.Context, name string, index syntax.ArithmExp
 		return
 	}
 
-	// from the syntax package, we know that val must be a string if
+	// from the syntax package, we know that value must be a string if index
-	// index is non-nil; nested arrays are forbidden.
+	// is non-nil; nested arrays are forbidden.
-	valStr := string(vr.Value.(StringVal))
+	valStr := vr.Value.(string)
 
 	// if the existing variable is already an AssocArray, try our best
 	// to convert the key to a string
 	if isAssocArray {
-		amap := cur.Value.(AssocArray)
+		amap := cur.Value.(map[string]string)
 		w, ok := index.(*syntax.Word)
 		if !ok {
 			return
 		}
-		k := r.loneWord(ctx, w)
+		k := r.literal(w)
 		amap[k] = valStr
 		cur.Value = amap
 		r.setVarInternal(name, cur)
 		return
 	}
-	var list IndexArray
+	var list []string
 	switch x := cur.Value.(type) {
-	case StringVal:
+	case string:
-		list = append(list, string(x))
+		list = append(list, x)
-	case IndexArray:
+	case []string:
 		list = x
-	case AssocArray: // done above
+	case map[string]string: // done above
 	}
-	k := r.arithm(ctx, index)
+	k := r.arithm(index)
 	for len(list) < k+1 {
 		list = append(list, "")
 	}
@@ -358,32 +239,33 @@ func stringIndex(index syntax.ArithmExpr) bool {
 	return false
 }
 
-func (r *Runner) assignVal(ctx context.Context, as *syntax.Assign, valType string) VarValue {
+func (r *Runner) assignVal(as *syntax.Assign, valType string) interface{} {
-	prev, prevOk := r.lookupVar(as.Name.Value)
+	prev := r.lookupVar(as.Name.Value)
 	if as.Naked {
 		return prev.Value
 	}
 	if as.Value != nil {
-		s := r.loneWord(ctx, as.Value)
+		s := r.literal(as.Value)
-		if !as.Append || !prevOk {
+		if !as.Append || !prev.IsSet() {
-			return StringVal(s)
+			return s
 		}
 		switch x := prev.Value.(type) {
-		case StringVal:
+		case string:
-			return x + StringVal(s)
+			return x + s
-		case IndexArray:
+		case []string:
 			if len(x) == 0 {
 				x = append(x, "")
 			}
 			x[0] += s
 			return x
-		case AssocArray:
+		case map[string]string:
 			// TODO
 		}
-		return StringVal(s)
+		return s
 	}
 	if as.Array == nil {
-		return nil
+		// don't return nil, as that's an unset variable
+		return ""
 	}
 	elems := as.Array.Elems
 	if valType == "" {
@@ -395,12 +277,12 @@ func (r *Runner) assignVal(ctx context.Context, as *syntax.Assign, valType strin
 	}
 	if valType == "-A" {
 		// associative array
-		amap := AssocArray(make(map[string]string, len(elems)))
+		amap := make(map[string]string, len(elems))
 		for _, elem := range elems {
-			k := r.loneWord(ctx, elem.Index.(*syntax.Word))
+			k := r.literal(elem.Index.(*syntax.Word))
-			amap[k] = r.loneWord(ctx, elem.Value)
+			amap[k] = r.literal(elem.Value)
 		}
-		if !as.Append || !prevOk {
+		if !as.Append || !prev.IsSet() {
 			return amap
 		}
 		// TODO
@@ -414,7 +296,7 @@ func (r *Runner) assignVal(ctx context.Context, as *syntax.Assign, valType strin
 			indexes[i] = i
 			continue
 		}
-		k := r.arithm(ctx, elem.Index)
+		k := r.arithm(elem.Index)
 		indexes[i] = k
 		if k > maxIndex {
 			maxIndex = k
@@ -422,50 +304,18 @@ func (r *Runner) assignVal(ctx context.Context, as *syntax.Assign, valType strin
 	}
 	strs := make([]string, maxIndex+1)
 	for i, elem := range elems {
-		strs[indexes[i]] = r.loneWord(ctx, elem.Value)
+		strs[indexes[i]] = r.literal(elem.Value)
 	}
-	if !as.Append || !prevOk {
+	if !as.Append || !prev.IsSet() {
-		return IndexArray(strs)
+		return strs
 	}
 	switch x := prev.Value.(type) {
-	case StringVal:
+	case string:
-		prevList := IndexArray([]string{string(x)})
+		return append([]string{x}, strs...)
-		return append(prevList, strs...)
+	case []string:
-	case IndexArray:
 		return append(x, strs...)
-	case AssocArray:
+	case map[string]string:
 		// TODO
 	}
-	return IndexArray(strs)
+	return strs
-}
-
-func (r *Runner) ifsUpdated() {
-	runes := r.getVar("IFS")
-	r.ifsJoin = ""
-	if len(runes) > 0 {
-		r.ifsJoin = runes[:1]
-	}
-	r.ifsRune = func(r rune) bool {
-		for _, r2 := range runes {
-			if r == r2 {
-				return true
-			}
-		}
-		return false
-	}
-}
-
-func (r *Runner) namesByPrefix(prefix string) []string {
-	var names []string
-	for _, name := range r.Env.Names() {
-		if strings.HasPrefix(name, prefix) {
-			names = append(names, name)
-		}
-	}
-	for name := range r.Vars {
-		if strings.HasPrefix(name, prefix) {
-			names = append(names, name)
-		}
-	}
-	return names
 }

vendor/mvdan.cc/sh/shell/doc.go

@@ -1,9 +1,6 @@
 // Copyright (c) 2017, Daniel Martí <mvdan@mvdan.cc>
 // See LICENSE for licensing information
 
-// Package shell contains high-level features that use the syntax and
+// Package shell contains high-level features that use the syntax, expand, and
 // interp packages under the hood.
-//
-// This package is a work in progress and EXPERIMENTAL; its API is not
-// subject to the 1.x backwards compatibility guarantee.
 package shell

vendor/mvdan.cc/sh/shell/expand.go

@@ -4,41 +4,60 @@
 package shell
 
 import (
-	"context"
+	"os"
 	"strings"
 
-	"mvdan.cc/sh/interp"
+	"mvdan.cc/sh/expand"
 	"mvdan.cc/sh/syntax"
 )
 
-// Expand performs shell expansion on s, using env to resolve variables.
+// Expand performs shell expansion on s as if it were within double quotes,
-// The expansion will apply to parameter expansions like $var and
+// using env to resolve variables. This includes parameter expansion, arithmetic
-// ${#var}, but also to arithmetic expansions like $((var + 3)), and
+// expansion, and quote removal.
-// command substitutions like $(echo foo).
 //
-// If env is nil, the current environment variables are used.
+// If env is nil, the current environment variables are used. Empty variables
+// are treated as unset; to support variables which are set but empty, use the
+// expand package directly.
 //
-// Any side effects or modifications to the system are forbidden when
+// Command subsitutions like $(echo foo) aren't supported to avoid running
-// interpreting the program. This is enforced via whitelists when
+// arbitrary code. To support those, use an interpreter with the expand package.
-// executing programs and opening paths. The interpreter also has a timeout of
+//
-// two seconds.
+// An error will be reported if the input string had invalid syntax.
 func Expand(s string, env func(string) string) (string, error) {
 	p := syntax.NewParser()
-	src := "<<EXPAND_EOF\n" + s + "\nEXPAND_EOF"
+	word, err := p.Document(strings.NewReader(s))
-	f, err := p.Parse(strings.NewReader(src), "")
 	if err != nil {
 		return "", err
 	}
-	word := f.Stmts[0].Redirs[0].Hdoc
+	if env == nil {
-	last := word.Parts[len(word.Parts)-1].(*syntax.Lit)
+		env = os.Getenv
-	// since the heredoc implies a trailing newline
-	last.Value = strings.TrimSuffix(last.Value, "\n")
-	r := pureRunner()
-	if env != nil {
-		r.Env = interp.FuncEnviron(env)
 	}
-	ctx, cancel := context.WithTimeout(context.Background(), pureRunnerTimeout)
+	cfg := &expand.Config{Env: expand.FuncEnviron(env)}
-	defer cancel()
+	return expand.Document(cfg, word)
-	fields, err := r.Fields(ctx, word)
+}
-	return strings.Join(fields, ""), err
+
+// Fields performs shell expansion on s as if it were a command's arguments,
+// using env to resolve variables. It is similar to Expand, but includes brace
+// expansion, tilde expansion, and globbing.
+//
+// If env is nil, the current environment variables are used. Empty variables
+// are treated as unset; to support variables which are set but empty, use the
+// expand package directly.
+//
+// An error will be reported if the input string had invalid syntax.
+func Fields(s string, env func(string) string) ([]string, error) {
+	p := syntax.NewParser()
+	var words []*syntax.Word
+	err := p.Words(strings.NewReader(s), func(w *syntax.Word) bool {
+		words = append(words, w)
+		return true
+	})
+	if err != nil {
+		return nil, err
+	}
+	if env == nil {
+		env = os.Getenv
+	}
+	cfg := &expand.Config{Env: expand.FuncEnviron(env)}
+	return expand.Fields(cfg, words...)
 }

vendor/mvdan.cc/sh/shell/source.go

@@ -6,81 +6,42 @@ package shell
 import (
 	"context"
 	"fmt"
-	"io"
 	"os"
-	"time"
 
+	"mvdan.cc/sh/expand"
 	"mvdan.cc/sh/interp"
 	"mvdan.cc/sh/syntax"
 )
 
 // SourceFile sources a shell file from disk and returns the variables
-// declared in it.
+// declared in it. It is a convenience function that uses a default shell
+// parser, parses a file from disk, and calls SourceNode.
 //
-// A default parser is used; to set custom options, use SourceNode
+// This function should be used with caution, as it can interpret arbitrary
-// instead.
+// code. Untrusted shell programs shoudn't be sourced outside of a sandbox
-func SourceFile(path string) (map[string]interp.Variable, error) {
+// environment.
+func SourceFile(ctx context.Context, path string) (map[string]expand.Variable, error) {
 	f, err := os.Open(path)
 	if err != nil {
 		return nil, fmt.Errorf("could not open: %v", err)
 	}
 	defer f.Close()
-	p := syntax.NewParser()
+	file, err := syntax.NewParser().Parse(f, path)
-	file, err := p.Parse(f, path)
 	if err != nil {
 		return nil, fmt.Errorf("could not parse: %v", err)
 	}
-	return SourceNode(file)
+	return SourceNode(ctx, file)
-}
-
-// purePrograms holds a list of common programs that do not have side
-// effects, or otherwise cannot modify or harm the system that runs
-// them.
-var purePrograms = []string{
-	// string handling
-	"sed", "grep", "tr", "cut", "cat", "head", "tail", "seq", "yes",
-	"wc",
-	// paths
-	"ls", "pwd", "basename", "realpath",
-	// others
-	"env", "sleep", "uniq", "sort",
-}
-
-var pureRunnerTimeout = 2 * time.Second
-
-func pureRunner() *interp.Runner {
-	// forbid executing programs that might cause trouble
-	exec := interp.ModuleExec(func(ctx context.Context, path string, args []string) error {
-		for _, name := range purePrograms {
-			if args[0] == name {
-				return interp.DefaultExec(ctx, path, args)
-			}
-		}
-		return fmt.Errorf("program not in whitelist: %s", args[0])
-	})
-	// forbid opening any real files
-	open := interp.OpenDevImpls(func(ctx context.Context, path string, flags int, mode os.FileMode) (io.ReadWriteCloser, error) {
-		mc, _ := interp.FromModuleContext(ctx)
-		return nil, fmt.Errorf("cannot open path: %s", mc.UnixPath(path))
-	})
-	r, err := interp.New(interp.Module(exec), interp.Module(open))
-	if err != nil {
-		panic(err)
-	}
-	return r
 }
 
 // SourceNode sources a shell program from a node and returns the
-// variables declared in it.
+// variables declared in it. It accepts the same set of node types that
+// interp/Runner.Run does.
 //
-// Any side effects or modifications to the system are forbidden when
+// This function should be used with caution, as it can interpret arbitrary
-// interpreting the program. This is enforced via whitelists when
+// code. Untrusted shell programs shoudn't be sourced outside of a sandbox
-// executing programs and opening paths. The interpreter also has a timeout of
+// environment.
-// two seconds.
+func SourceNode(ctx context.Context, node syntax.Node) (map[string]expand.Variable, error) {
-func SourceNode(node syntax.Node) (map[string]interp.Variable, error) {
+	r, _ := interp.New()
-	r := pureRunner()
-	ctx, cancel := context.WithTimeout(context.Background(), pureRunnerTimeout)
-	defer cancel()
 	if err := r.Run(ctx, node); err != nil {
 		return nil, fmt.Errorf("could not run: %v", err)
 	}

vendor/mvdan.cc/sh/syntax/expand.go

@@ -5,7 +5,8 @@ package syntax
 
 import "strconv"
 
-// TODO: consider making these special syntax nodes
+// TODO(v3): Consider making these special syntax nodes.
+// Among other things, we can make use of Word.Lit.
 
 type brace struct {
 	seq   bool // {x..y[..incr]} instead of {x,y[,...]}
@@ -265,17 +266,13 @@ func expandRec(bw *braceWord) []*Word {
 	return []*Word{{Parts: left}}
 }
 
+// TODO(v3): remove
+
 // ExpandBraces performs Bash brace expansion on a word. For example,
 // passing it a single-literal word "foo{bar,baz}" will return two
 // single-literal words, "foobar" and "foobaz".
 //
-// It does not return an error; malformed brace expansions are simply
+// Deprecated: use mvdan.cc/sh/expand.Braces instead.
-// skipped. For example, "a{b{c,d}" results in the words "a{bc" and
-// "a{bd".
-//
-// Note that the resulting words may have more word parts than
-// necessary, such as contiguous *Lit nodes, and that these parts may be
-// shared between words.
 func ExpandBraces(word *Word) []*Word {
 	topBrace, any := splitBraces(word)
 	if !any {

vendor/mvdan.cc/sh/syntax/lexer.go

@@ -60,10 +60,9 @@ func (p *Parser) rune() rune {
 		// p.r instead of b so that newline
 		// character positions don't have col 0.
 		p.npos.line++
-		p.npos.col = 1
+		p.npos.col = 0
-	} else {
-		p.npos.col += p.w
 	}
+	p.npos.col += p.w
 	bquotes := 0
 retry:
 	if p.bsp < len(p.bs) {
@@ -87,9 +86,8 @@ retry:
 			p.w, p.r = 1, rune(b)
 			return p.r
 		}
-		if p.bsp+utf8.UTFMax >= len(p.bs) {
+		if !utf8.FullRune(p.bs[p.bsp:]) {
-			// we might need up to 4 bytes to read a full
+			// we need more bytes to read a full non-ascii rune
-			// non-ascii rune
 			p.fill()
 		}
 		var w int
@@ -122,14 +120,18 @@ func (p *Parser) fill() {
 	p.offs += p.bsp
 	left := len(p.bs) - p.bsp
 	copy(p.readBuf[:left], p.readBuf[p.bsp:])
+readAgain:
 	n, err := 0, p.readErr
 	if err == nil {
 		n, err = p.src.Read(p.readBuf[left:])
 		p.readErr = err
 	}
 	if n == 0 {
+		if err == nil {
+			goto readAgain
+		}
 		// don't use p.errPass as we don't want to overwrite p.tok
-		if err != nil && err != io.EOF {
+		if err != io.EOF {
 			p.err = err
 		}
 		if left > 0 {
@@ -238,6 +240,7 @@ skipSpace:
 			return
 		}
 	}
+changedState:
 	p.pos = p.getPos()
 	switch {
 	case p.quote&allRegTokens != 0:
@@ -292,15 +295,21 @@ skipSpace:
 	case p.quote&allParamExp != 0 && paramOps(r):
 		p.tok = p.paramToken(r)
 	case p.quote == testRegexp:
+		if !p.rxFirstPart && p.spaced {
+			p.quote = noState
+			goto changedState
+		}
+		p.rxFirstPart = false
 		switch r {
 		case ';', '"', '\'', '$', '&', '>', '<', '`':
 			p.tok = p.regToken(r)
 		case ')':
-			if p.reOpenParens > 0 {
+			if p.rxOpenParens > 0 {
 				// continuation of open paren
 				p.advanceLitRe(r)
 			} else {
 				p.tok = rightParen
+				p.quote = noState
 			}
 		default: // including '(', '|'
 			p.advanceLitRe(r)
@@ -900,7 +909,6 @@ func (p *Parser) advanceLitHdoc(r rune) {
 	p.newLit(r)
 	if p.quote == hdocBodyTabs {
 		for r == '\t' {
-			p.discardLit(1)
 			r = p.rune()
 		}
 	}
@@ -916,7 +924,12 @@ func (p *Parser) advanceLitHdoc(r rune) {
 		case '\\': // escaped byte follows
 			p.rune()
 		case '\n', utf8.RuneSelf:
-			if bytes.HasPrefix(p.litBs[lStart:], p.hdocStop) {
+			if p.parsingDoc {
+				if r == utf8.RuneSelf {
+					p.val = p.endLit()
+					return
+				}
+			} else if bytes.HasPrefix(p.litBs[lStart:], p.hdocStop) {
 				p.val = p.endLit()[:lStart]
 				if p.val == "" {
 					p.tok = _Newl
@@ -930,7 +943,6 @@ func (p *Parser) advanceLitHdoc(r rune) {
 			if p.quote == hdocBodyTabs {
 				for p.peekByte('\t') {
 					p.rune()
-					p.discardLit(1)
 				}
 			}
 			lStart = len(p.litBs)
@@ -938,7 +950,7 @@ func (p *Parser) advanceLitHdoc(r rune) {
 	}
 }
 
-func (p *Parser) hdocLitWord() *Word {
+func (p *Parser) quotedHdocWord() *Word {
 	r := p.r
 	p.newLit(r)
 	pos := p.getPos()
@@ -948,7 +960,6 @@ func (p *Parser) hdocLitWord() *Word {
 		}
 		if p.quote == hdocBodyTabs {
 			for r == '\t' {
-				p.discardLit(1)
 				r = p.rune()
 			}
 		}
@@ -976,19 +987,25 @@ func (p *Parser) advanceLitRe(r rune) {
 		case '\\':
 			p.rune()
 		case '(':
-			p.reOpenParens++
+			p.rxOpenParens++
 		case ')':
-			if p.reOpenParens--; p.reOpenParens < 0 {
+			if p.rxOpenParens--; p.rxOpenParens < 0 {
 				p.tok, p.val = _LitWord, p.endLit()
+				p.quote = noState
 				return
 			}
 		case ' ', '\t', '\r', '\n':
-			if p.reOpenParens <= 0 {
+			if p.rxOpenParens <= 0 {
 				p.tok, p.val = _LitWord, p.endLit()
+				p.quote = noState
 				return
 			}
-		case utf8.RuneSelf, ';', '"', '\'', '$', '&', '>', '<', '`':
+		case '"', '\'', '$', '`':
+			p.tok, p.val = _Lit, p.endLit()
+			return
+		case utf8.RuneSelf, ';', '&', '>', '<':
 			p.tok, p.val = _LitWord, p.endLit()
+			p.quote = noState
 			return
 		}
 	}

vendor/mvdan.cc/sh/syntax/nodes.go

@@ -3,7 +3,10 @@
 
 package syntax
 
-import "fmt"
+import (
+	"fmt"
+	"strings"
+)
 
 // Node represents a syntax tree node.
 type Node interface {
@@ -243,7 +246,12 @@ func (r *Redirect) Pos() Pos {
 	}
 	return r.OpPos
 }
-func (r *Redirect) End() Pos { return r.Word.End() }
+func (r *Redirect) End() Pos {
+	if r.Hdoc != nil {
+		return r.Hdoc.End()
+	}
+	return r.Word.End()
+}
 
 // CallExpr represents a command execution or function call, otherwise known as
 // a "simple command".
@@ -289,6 +297,10 @@ type Block struct {
 func (b *Block) Pos() Pos { return b.Lbrace }
 func (b *Block) End() Pos { return posAddCol(b.Rbrace, 1) }
 
+// TODO(v3): Refactor and simplify elif/else. For example, we could likely make
+// Else an *IfClause, remove ElsePos, make IfPos also do opening "else"
+// positions, and join the comment slices as Last []Comment.
+
 // IfClause represents an if statement.
 type IfClause struct {
 	Elif    bool // whether this IfClause begins with "elif"
@@ -302,6 +314,7 @@ type IfClause struct {
 	Else StmtList
 
 	ElseComments []Comment // comments on the "else"
+	FiComments   []Comment // comments on the "fi"
 }
 
 func (c *IfClause) Pos() Pos { return c.IfPos }
@@ -415,6 +428,28 @@ type Word struct {
 func (w *Word) Pos() Pos { return w.Parts[0].Pos() }
 func (w *Word) End() Pos { return w.Parts[len(w.Parts)-1].End() }
 
+// Lit returns the word as a literal value, if the word consists of *syntax.Lit
+// nodes only. An empty string is returned otherwise. Words with multiple
+// literals, which can appear in some edge cases, are handled properly.
+//
+// For example, the word "foo" will return "foo", but the word "foo${bar}" will
+// return "".
+func (w *Word) Lit() string {
+	// In the usual case, we'll have either a single part that's a literal,
+	// or one of the parts being a non-literal. Using strings.Join instead
+	// of a strings.Builder avoids extra work in these cases, since a single
+	// part is a shortcut, and many parts don't incur string copies.
+	lits := make([]string, 0, 1)
+	for _, part := range w.Parts {
+		lit, ok := part.(*Lit)
+		if !ok {
+			return ""
+		}
+		lits = append(lits, lit.Value)
+	}
+	return strings.Join(lits, "")
+}
+
 // WordPart represents all nodes that can form part of a word.
 //
 // These are *Lit, *SglQuoted, *DblQuoted, *ParamExp, *CmdSubst, *ArithmExp,

vendor/mvdan.cc/sh/syntax/parser.go

@@ -113,6 +113,135 @@ func (p *Parser) Stmts(r io.Reader, fn func(*Stmt) bool) error {
 	return p.err
 }
 
+type wrappedReader struct {
+	*Parser
+	io.Reader
+
+	lastLine    uint16
+	accumulated []*Stmt
+	fn          func([]*Stmt) bool
+}
+
+func (w *wrappedReader) Read(p []byte) (n int, err error) {
+	// If we lexed a newline for the first time, we just finished a line, so
+	// we may need to give a callback for the edge cases below not covered
+	// by Parser.Stmts.
+	if w.r == '\n' && w.npos.line > w.lastLine {
+		if w.Incomplete() {
+			// Incomplete statement; call back to print "> ".
+			if !w.fn(w.accumulated) {
+				return 0, io.EOF
+			}
+		} else if len(w.accumulated) == 0 {
+			// Nothing was parsed; call back to print another "$ ".
+			if !w.fn(nil) {
+				return 0, io.EOF
+			}
+		}
+		w.lastLine = w.npos.line
+	}
+	return w.Reader.Read(p)
+}
+
+// Interactive implements what is necessary to parse statements in an
+// interactive shell. The parser will call the given function under two
+// circumstances outlined below.
+//
+// If a line containing any number of statements is parsed, the function will be
+// called with said statements.
+//
+// If a line ending in an incomplete statement is parsed, the function will be
+// called with any fully parsed statents, and Parser.Incomplete will return
+// true.
+//
+// One can imagine a simple interactive shell implementation as follows:
+//
+//         fmt.Fprintf(os.Stdout, "$ ")
+//         parser.Interactive(os.Stdin, func(stmts []*syntax.Stmt) bool {
+//                 if parser.Incomplete() {
+//                         fmt.Fprintf(os.Stdout, "> ")
+//                         return true
+//                 }
+//                 run(stmts)
+//                 fmt.Fprintf(os.Stdout, "$ ")
+//                 return true
+//         }
+//
+// If the callback function returns false, parsing is stopped and the function
+// is not called again.
+func (p *Parser) Interactive(r io.Reader, fn func([]*Stmt) bool) error {
+	w := wrappedReader{Parser: p, Reader: r, fn: fn}
+	return p.Stmts(&w, func(stmt *Stmt) bool {
+		w.accumulated = append(w.accumulated, stmt)
+		// We finished parsing a statement and we're at a newline token,
+		// so we finished fully parsing a number of statements. Call
+		// back to run the statements and print "$ ".
+		if p.tok == _Newl {
+			if !fn(w.accumulated) {
+				return false
+			}
+			w.accumulated = w.accumulated[:0]
+			// The callback above would already print "$ ", so we
+			// don't want the subsequent wrappedReader.Read to cause
+			// another "$ " print thinking that nothing was parsed.
+			w.lastLine = w.npos.line + 1
+		}
+		return true
+	})
+}
+
+// Words reads and parses words one at a time, calling a function each time one
+// is parsed. If the function returns false, parsing is stopped and the function
+// is not called again.
+//
+// Newlines are skipped, meaning that multi-line input will work fine. If the
+// parser encounters a token that isn't a word, such as a semicolon, an error
+// will be returned.
+//
+// Note that the lexer doesn't currently tokenize spaces, so it may need to read
+// a non-space byte such as a newline or a letter before finishing the parsing
+// of a word. This will be fixed in the future.
+func (p *Parser) Words(r io.Reader, fn func(*Word) bool) error {
+	p.reset()
+	p.f = &File{}
+	p.src = r
+	p.rune()
+	p.next()
+	for {
+		p.got(_Newl)
+		w := p.getWord()
+		if w == nil {
+			if p.tok != _EOF {
+				p.curErr("%s is not a valid word", p.tok)
+			}
+			return p.err
+		}
+		if !fn(w) {
+			return nil
+		}
+	}
+}
+
+// Document parses a single here-document word. That is, it parses the input as
+// if they were lines following a <<EOF redirection.
+//
+// In practice, this is the same as parsing the input as if it were within
+// double quotes, but without having to escape all double quote characters.
+// Similarly, the here-document word parsed here cannot be ended by any
+// delimiter other than reaching the end of the input.
+func (p *Parser) Document(r io.Reader) (*Word, error) {
+	p.reset()
+	p.f = &File{}
+	p.src = r
+	p.rune()
+	p.quote = hdocBody
+	p.hdocStop = []byte("MVDAN_CC_SH_SYNTAX_EOF")
+	p.parsingDoc = true
+	p.next()
+	w := p.getWord()
+	return w, p.err
+}
+
 // Parser holds the internal state of the parsing mechanism of a
 // program.
 type Parser struct {
@@ -150,18 +279,23 @@ type Parser struct {
 	buriedHdocs int
 	heredocs    []*Redirect
 	hdocStop    []byte
+	parsingDoc  bool
 
-	// openBquotes is how many levels of backquotes are open at the
+	// openStmts is how many entire statements we're currently parsing. A
-	// moment
+	// non-zero number means that we require certain tokens or words before
+	// reaching EOF.
+	openStmts int
+	// openBquotes is how many levels of backquotes are open at the moment.
 	openBquotes int
-	// lastBquoteEsc is how many times the last backquote token was
+
-	// escaped
+	// lastBquoteEsc is how many times the last backquote token was escaped
 	lastBquoteEsc int
-	// buriedBquotes is like openBquotes, but saved for when the
+	// buriedBquotes is like openBquotes, but saved for when the parser
-	// parser comes out of single quotes
+	// comes out of single quotes
 	buriedBquotes int
 
-	reOpenParens int
+	rxOpenParens int
+	rxFirstPart  bool
 
 	accComs []Comment
 	curComs *[]Comment
@@ -180,6 +314,14 @@ type Parser struct {
 	litBs   []byte
 }
 
+func (p *Parser) Incomplete() bool {
+	// If we're in a quote state other than noState, we're parsing a node
+	// such as a double-quoted string.
+	// If there are any open statements, we need to finish them.
+	// If we're constructing a literal, we need to finish it.
+	return p.quote != noState || p.openStmts > 0 || p.litBs != nil
+}
+
 const bufSize = 1 << 10
 
 func (p *Parser) reset() {
@@ -191,9 +333,10 @@ func (p *Parser) reset() {
 	p.r, p.w = 0, 0
 	p.err, p.readErr = nil, nil
 	p.quote, p.forbidNested = noState, false
+	p.openStmts = 0
 	p.heredocs, p.buriedHdocs = p.heredocs[:0], 0
+	p.parsingDoc = false
 	p.openBquotes, p.buriedBquotes = 0, 0
-	p.reOpenParens = 0
 	p.accComs, p.curComs = nil, &p.accComs
 }
 
@@ -270,6 +413,8 @@ func (p *Parser) call(w *Word) *CallExpr {
 	return ce
 }
 
+//go:generate stringer -type=quoteState
+
 type quoteState uint32
 
 const (
@@ -372,7 +517,7 @@ func (p *Parser) doHeredocs() {
 			p.rune()
 		}
 		if quoted {
-			r.Hdoc = p.hdocLitWord()
+			r.Hdoc = p.quotedHdocWord()
 		} else {
 			p.next()
 			r.Hdoc = p.getWord()
@@ -597,7 +742,9 @@ loop:
 		if p.tok == _EOF {
 			break
 		}
+		p.openStmts++
 		s := p.getStmt(true, false, false)
+		p.openStmts--
 		if s == nil {
 			p.invalidStmtStart()
 			break
@@ -619,7 +766,7 @@ func (p *Parser) stmtList(stops ...string) (sl StmtList) {
 	}
 	p.stmts(fn, stops...)
 	split := len(p.accComs)
-	if p.tok == _LitWord && (p.val == "elif" || p.val == "else") {
+	if p.tok == _LitWord && (p.val == "elif" || p.val == "else" || p.val == "fi") {
 		// Split the comments, so that any aligned with an opening token
 		// get attached to it. For example:
 		//
@@ -630,7 +777,7 @@ func (p *Parser) stmtList(stops ...string) (sl StmtList) {
 		//     fi
 		// TODO(mvdan): look into deduplicating this with similar logic
 		// in caseItems.
-		for i := len(p.accComs)-1; i >= 0; i-- {
+		for i := len(p.accComs) - 1; i >= 0; i-- {
 			c := p.accComs[i]
 			if c.Pos().Col() != p.pos.Col() {
 				break
@@ -1250,15 +1397,8 @@ func (p *Parser) paramExp() *ParamExp {
 		default:
 			pe.Exp = p.paramExpExp()
 		}
-	case plus, colPlus, minus, colMinus, quest, colQuest, assgn, colAssgn:
+	case plus, colPlus, minus, colMinus, quest, colQuest, assgn, colAssgn,
-		// if unset/null actions
+		perc, dblPerc, hash, dblHash:
-		switch pe.Param.Value {
-		case "#", "$", "?", "!":
-			p.curErr("$%s can never be unset or null", pe.Param.Value)
-		}
-		pe.Exp = p.paramExpExp()
-	case perc, dblPerc, hash, dblHash:
-		// pattern string manipulation
 		pe.Exp = p.paramExpExp()
 	case _EOF:
 	default:
@@ -1333,6 +1473,9 @@ func (p *Parser) backquoteEnd() bool {
 
 // ValidName returns whether val is a valid name as per the POSIX spec.
 func ValidName(val string) bool {
+	if val == "" {
+		return false
+	}
 	for i, r := range val {
 		switch {
 		case 'a' <= r && r <= 'z':
@@ -1797,6 +1940,8 @@ func (p *Parser) ifClause(s *Stmt) {
 		curIf.ElsePos = elsePos
 		curIf.Else = p.followStmts("else", curIf.ElsePos, "fi")
 	}
+	curIf.FiComments = p.accComs
+	p.accComs = nil
 	rif.FiPos = p.stmtEnd(rif, "if", "fi")
 	curIf.FiPos = rif.FiPos
 	s.Cmd = rif
@@ -1952,7 +2097,7 @@ func (p *Parser) caseItems(stop string) (items []*CaseItem) {
 		p.got(_Newl)
 		split := len(p.accComs)
 		if p.tok == _LitWord && p.val != stop {
-			for i := len(p.accComs)-1; i >= 0; i-- {
+			for i := len(p.accComs) - 1; i >= 0; i-- {
 				c := p.accComs[i]
 				if c.Pos().Col() != p.pos.Col() {
 					break
@@ -1982,6 +2127,7 @@ func (p *Parser) testClause(s *Stmt) {
 }
 
 func (p *Parser) testExpr(ftok token, fpos Pos, pastAndOr bool) TestExpr {
+	p.got(_Newl)
 	var left TestExpr
 	if pastAndOr {
 		left = p.testExprBase(ftok, fpos)
@@ -1991,6 +2137,7 @@ func (p *Parser) testExpr(ftok token, fpos Pos, pastAndOr bool) TestExpr {
 	if left == nil {
 		return left
 	}
+	p.got(_Newl)
 	switch p.tok {
 	case andAnd, orOr:
 	case _LitWord:
@@ -2015,10 +2162,12 @@ func (p *Parser) testExpr(ftok token, fpos Pos, pastAndOr bool) TestExpr {
 		Op:    BinTestOperator(p.tok),
 		X:     left,
 	}
+	// Save the previous quoteState, since we change it in TsReMatch.
+	oldQuote := p.quote
+
 	switch b.Op {
 	case AndTest, OrTest:
 		p.next()
-		p.got(_Newl)
 		if b.Y = p.testExpr(token(b.Op), b.OpPos, false); b.Y == nil {
 			p.followErrExp(b.OpPos, b.Op.String())
 		}
@@ -2026,12 +2175,12 @@ func (p *Parser) testExpr(ftok token, fpos Pos, pastAndOr bool) TestExpr {
 		if p.lang != LangBash {
 			p.langErr(p.pos, "regex tests", LangBash)
 		}
-		oldReOpenParens := p.reOpenParens
+		p.rxOpenParens = 0
-		old := p.preNested(testRegexp)
+		p.rxFirstPart = true
-		defer func() {
+		// TODO(mvdan): Using nested states within a regex will break in
-			p.postNested(old)
+		// all sorts of ways. The better fix is likely to use a stop
-			p.reOpenParens = oldReOpenParens
+		// token, like we do with heredocs.
-		}()
+		p.quote = testRegexp
 		fallthrough
 	default:
 		if _, ok := b.X.(*Word); !ok {
@@ -2041,6 +2190,7 @@ func (p *Parser) testExpr(ftok token, fpos Pos, pastAndOr bool) TestExpr {
 		p.next()
 		b.Y = p.followWordTok(token(b.Op), b.OpPos)
 	}
+	p.quote = oldQuote
 	return b
 }
 
@@ -2079,14 +2229,12 @@ func (p *Parser) testExprBase(ftok token, fpos Pos) TestExpr {
 	case leftParen:
 		pe := &ParenTest{Lparen: p.pos}
 		p.next()
-		p.got(_Newl)
 		if pe.X = p.testExpr(leftParen, pe.Lparen, false); pe.X == nil {
 			p.followErrExp(pe.Lparen, "(")
 		}
 		pe.Rparen = p.matched(pe.Lparen, leftParen, rightParen)
 		return pe
 	default:
-		p.got(_Newl)
 		return p.followWordTok(ftok, fpos)
 	}
 }

vendor/mvdan.cc/sh/syntax/pattern.go

@@ -32,17 +32,16 @@ func charClass(s string) (string, error) {
 	return s[:len(name)+6], nil
 }
 
-// TranslatePattern turns a shell pattern expression into a regular
+// TranslatePattern turns a shell wildcard pattern into a regular expression
-// expression that can be used with regexp.Compile. It will return an
+// that can be used with regexp.Compile. It will return an error if the input
-// error if the input pattern was incorrect. Otherwise, the returned
+// pattern was incorrect. Otherwise, the returned expression can be passed to
-// expression can be passed to regexp.MustCompile.
+// regexp.MustCompile.
 //
 // For example, TranslatePattern(`foo*bar?`, true) returns `foo.*bar.`.
 //
-// Note that this function (and QuotePattern) should not be directly
+// Note that this function (and QuotePattern) should not be directly used with
-// used with file paths if Windows is supported, as the path separator
+// file paths if Windows is supported, as the path separator on that platform is
-// on that platform is the same character as the escaping character for
+// the same character as the escaping character for shell patterns.
-// shell patterns.
 func TranslatePattern(pattern string, greedy bool) (string, error) {
 	any := false
 loop:
@@ -122,9 +121,31 @@ loop:
 	return buf.String(), nil
 }
 
-// QuotePattern returns a string that quotes all special characters in
+// HasPattern returns whether a string contains any unescaped wildcard
-// the given pattern. The returned string is a pattern that matches the
+// characters: '*', '?', or '['. When the function returns false, the given
-// literal string.
+// pattern can only match at most one string.
+//
+// For example, HasPattern(`foo\*bar`) returns false, but HasPattern(`foo*bar`)
+// returns true.
+//
+// This can be useful to avoid extra work, like TranslatePattern. Note that this
+// function cannot be used to avoid QuotePattern, as backslashes are quoted by
+// that function but ignored here.
+func HasPattern(pattern string) bool {
+	for i := 0; i < len(pattern); i++ {
+		switch pattern[i] {
+		case '\\':
+			i++
+		case '*', '?', '[':
+			return true
+		}
+	}
+	return false
+}
+
+// QuotePattern returns a string that quotes all special characters in the given
+// wildcard pattern. The returned string is a pattern that matches the literal
+// string.
 //
 // For example, QuotePattern(`foo*bar?`) returns `foo\*bar\?`.
 func QuotePattern(pattern string) string {

vendor/mvdan.cc/sh/syntax/printer.go

@@ -5,6 +5,8 @@ package syntax
 
 import (
 	"bufio"
+	"bytes"
+	"fmt"
 	"io"
 	"strings"
 	"unicode"
@@ -63,8 +65,9 @@ func NewPrinter(options ...func(*Printer)) *Printer {
 // Print "pretty-prints" the given syntax tree node to the given writer. Writes
 // to w are buffered.
 //
-// The node types supported at the moment are *File, *Stmt, *Word, and any
+// The node types supported at the moment are *File, *Stmt, *Word, any Command
-// Command node. A trailing newline will only be printed when a *File is used.
+// node, and any WordPart node. A trailing newline will only be printed when a
+// *File is used.
 func (p *Printer) Print(w io.Writer, node Node) error {
 	p.reset()
 	p.bufWriter.Reset(w)
@@ -74,10 +77,15 @@ func (p *Printer) Print(w io.Writer, node Node) error {
 		p.newline(x.End())
 	case *Stmt:
 		p.stmtList(StmtList{Stmts: []*Stmt{x}})
+	case Command:
+		p.line = x.Pos().Line()
+		p.command(x, nil)
 	case *Word:
 		p.word(x)
-	case Command:
+	case WordPart:
-		p.command(x, nil)
+		p.wordPart(x, nil)
+	default:
+		return fmt.Errorf("unsupported node type: %T", x)
 	}
 	p.flushHeredocs()
 	p.flushComments()
@@ -85,39 +93,46 @@ func (p *Printer) Print(w io.Writer, node Node) error {
 }
 
 type bufWriter interface {
-	WriteByte(byte) error
+	Write([]byte) (int, error)
 	WriteString(string) (int, error)
+	WriteByte(byte) error
 	Reset(io.Writer)
 	Flush() error
 }
 
 type colCounter struct {
 	*bufio.Writer
-	column int
+	column    int
+	lineStart bool
 }
 
 func (c *colCounter) WriteByte(b byte) error {
-	if b == '\n' {
+	switch b {
-		c.column = 1
+	case '\n':
-	} else {
+		c.column = 0
-		c.column++
+		c.lineStart = true
+	case '\t', ' ':
+	default:
+		c.lineStart = false
 	}
+	c.column++
 	return c.Writer.WriteByte(b)
 }
 
 func (c *colCounter) WriteString(s string) (int, error) {
+	c.lineStart = false
 	for _, r := range s {
 		if r == '\n' {
-			c.column = 1
+			c.column = 0
-		} else {
-			c.column++
 		}
+		c.column++
 	}
 	return c.Writer.WriteString(s)
 }
 
 func (c *colCounter) Reset(w io.Writer) {
 	c.column = 1
+	c.lineStart = true
 	c.Writer.Reset(w)
 }
 
@@ -204,7 +219,12 @@ func (p *Printer) spacePad(pos Pos) {
 		p.WriteByte(' ')
 		p.wantSpace = false
 	}
-	for p.cols.column > 0 && p.cols.column < int(pos.col) {
+	if p.cols.lineStart {
+		// Never add padding at the start of a line, since this may
+		// result in broken indentation or mixing of spaces and tabs.
+		return
+	}
+	for !p.cols.lineStart && p.cols.column > 0 && p.cols.column < int(pos.col) {
 		p.WriteByte(' ')
 	}
 }
@@ -329,9 +349,9 @@ func (p *Printer) flushHeredocs() {
 			!p.minify && p.tabsPrinter != nil {
 			if r.Hdoc != nil {
 				extra := extraIndenter{
-					bufWriter: p.bufWriter,
+					bufWriter:   p.bufWriter,
-					afterNewl: true,
+					baseIndent:  int(p.level + 1),
-					level:     p.level + 1,
+					firstIndent: -1,
 				}
 				*p.tabsPrinter = Printer{
 					bufWriter: &extra,
@@ -396,13 +416,6 @@ func (p *Printer) semiRsrv(s string, pos Pos) {
 	p.wantSpace = true
 }
 
-func (p *Printer) comment(c Comment) {
-	if p.minify {
-		return
-	}
-	p.pendingComments = append(p.pendingComments, c)
-}
-
 func (p *Printer) flushComments() {
 	for i, c := range p.pendingComments {
 		p.firstLine = false
@@ -434,11 +447,11 @@ func (p *Printer) flushComments() {
 	p.pendingComments = nil
 }
 
-func (p *Printer) comments(cs []Comment) {
+func (p *Printer) comments(comments ...Comment) {
 	if p.minify {
 		return
 	}
-	p.pendingComments = append(p.pendingComments, cs...)
+	p.pendingComments = append(p.pendingComments, comments...)
 }
 
 func (p *Printer) wordParts(wps []WordPart) {
@@ -487,7 +500,7 @@ func (p *Printer) wordPart(wp, next WordPart) {
 		}
 	case *ParamExp:
 		litCont := ";"
-		if nextLit, ok := next.(*Lit); ok {
+		if nextLit, ok := next.(*Lit); ok && nextLit.Value != "" {
 			litCont = nextLit.Value[:1]
 		}
 		name := x.Param.Value
@@ -758,13 +771,13 @@ func (p *Printer) casePatternJoin(pats []*Word) {
 func (p *Printer) elemJoin(elems []*ArrayElem, last []Comment) {
 	p.incLevel()
 	for _, el := range elems {
-		var left *Comment
+		var left []Comment
 		for _, c := range el.Comments {
 			if c.Pos().After(el.Pos()) {
-				left = &c
+				left = append(left, c)
 				break
 			}
-			p.comment(c)
+			p.comments(c)
 		}
 		if el.Pos().Line() > p.line {
 			p.newline(el.Pos())
@@ -776,12 +789,10 @@ func (p *Printer) elemJoin(elems []*ArrayElem, last []Comment) {
 			p.WriteByte('=')
 		}
 		p.word(el.Value)
-		if left != nil {
+		p.comments(left...)
-			p.comment(*left)
-		}
 	}
 	if len(last) > 0 {
-		p.comments(last)
+		p.comments(last...)
 		p.flushComments()
 	}
 	p.decLevel()
@@ -924,14 +935,14 @@ func (p *Printer) command(cmd Command, redirs []*Redirect) (startRedirs int) {
 				p.wantSpace = false
 				p.newline(Pos{})
 				p.indent()
-				p.comments(x.Y.Comments)
+				p.comments(x.Y.Comments...)
 				p.newline(Pos{})
 				p.indent()
 			}
 		} else {
 			p.spacedToken(x.Op.String(), x.OpPos)
 			p.line = x.OpPos.Line()
-			p.comments(x.Y.Comments)
+			p.comments(x.Y.Comments...)
 			p.newline(Pos{})
 			p.indent()
 		}
@@ -952,7 +963,7 @@ func (p *Printer) command(cmd Command, redirs []*Redirect) (startRedirs int) {
 			p.space()
 		}
 		p.line = x.Body.Pos().Line()
-		p.comments(x.Body.Comments)
+		p.comments(x.Body.Comments...)
 		p.stmt(x.Body)
 	case *CaseClause:
 		p.WriteString("case ")
@@ -968,7 +979,7 @@ func (p *Printer) command(cmd Command, redirs []*Redirect) (startRedirs int) {
 					last = ci.Comments[i:]
 					break
 				}
-				p.comment(c)
+				p.comments(c)
 			}
 			p.newlines(ci.Pos())
 			p.casePatternJoin(ci.Patterns)
@@ -987,11 +998,11 @@ func (p *Printer) command(cmd Command, redirs []*Redirect) (startRedirs int) {
 				// avoid ; directly after tokens like ;;
 				p.wroteSemi = true
 			}
-			p.comments(last)
+			p.comments(last...)
 			p.flushComments()
 			p.level--
 		}
-		p.comments(x.Last)
+		p.comments(x.Last...)
 		if p.swtCaseIndent {
 			p.flushComments()
 			p.decLevel()
@@ -1048,20 +1059,30 @@ func (p *Printer) ifClause(ic *IfClause, elif bool) {
 	p.nestedStmts(ic.Cond, Pos{})
 	p.semiOrNewl("then", ic.ThenPos)
 	p.nestedStmts(ic.Then, ic.bodyEndPos())
-	p.comments(ic.ElseComments)
+
+	var left []Comment
+	for _, c := range ic.ElseComments {
+		if c.Pos().After(ic.ElsePos) {
+			left = append(left, c)
+			break
+		}
+		p.comments(c)
+	}
 	if ic.FollowedByElif() {
 		s := ic.Else.Stmts[0]
-		p.comments(s.Comments)
+		p.comments(s.Comments...)
 		p.semiRsrv("elif", ic.ElsePos)
 		p.ifClause(s.Cmd.(*IfClause), true)
 		return
 	}
 	if !ic.Else.empty() {
 		p.semiRsrv("else", ic.ElsePos)
+		p.comments(left...)
 		p.nestedStmts(ic.Else, ic.FiPos)
 	} else if ic.ElsePos.IsValid() {
 		p.line = ic.ElsePos.Line()
 	}
+	p.comments(ic.FiComments...)
 	p.semiRsrv("fi", ic.FiPos)
 }
 
@@ -1091,18 +1112,17 @@ func (p *Printer) stmtList(sl StmtList) {
 	lastIndentedLine := uint(0)
 	for i, s := range sl.Stmts {
 		pos := s.Pos()
-		var endCom *Comment
+		var midComs, endComs []Comment
-		var midComs []Comment
 		for _, c := range s.Comments {
 			if c.End().After(s.End()) {
-				endCom = &c
+				endComs = append(endComs, c)
 				break
 			}
 			if c.Pos().After(s.Pos()) {
 				midComs = append(midComs, c)
 				continue
 			}
-			p.comment(c)
+			p.comments(c)
 		}
 		if !p.minify || p.wantSpace {
 			p.newlines(pos)
@@ -1111,12 +1131,12 @@ func (p *Printer) stmtList(sl StmtList) {
 		if !p.hasInline(s) {
 			inlineIndent = 0
 			p.commentPadding = 0
-			p.comments(midComs)
+			p.comments(midComs...)
 			p.stmt(s)
 			p.wantNewline = true
 			continue
 		}
-		p.comments(midComs)
+		p.comments(midComs...)
 		p.stmt(s)
 		if s.Pos().Line() > lastIndentedLine+1 {
 			inlineIndent = 0
@@ -1137,15 +1157,13 @@ func (p *Printer) stmtList(sl StmtList) {
 			}
 			lastIndentedLine = p.line
 		}
-		if endCom != nil {
+		p.comments(endComs...)
-			p.comment(*endCom)
-		}
 		p.wantNewline = true
 	}
 	if len(sl.Stmts) == 1 && !sep {
 		p.wantNewline = false
 	}
-	p.comments(sl.Last)
+	p.comments(sl.Last...)
 }
 
 type byteCounter int
@@ -1160,6 +1178,9 @@ func (c *byteCounter) WriteByte(b byte) error {
 	}
 	return nil
 }
+func (c *byteCounter) Write(p []byte) (int, error) {
+	return c.WriteString(string(p))
+}
 func (c *byteCounter) WriteString(s string) (int, error) {
 	switch {
 	case *c < 0:
@@ -1173,20 +1194,41 @@ func (c *byteCounter) WriteString(s string) (int, error) {
 func (c *byteCounter) Reset(io.Writer) { *c = 0 }
 func (c *byteCounter) Flush() error    { return nil }
 
+// extraIndenter ensures that all lines in a '<<-' heredoc body have at least
+// baseIndent leading tabs. Those that had more tab indentation than the first
+// heredoc line will keep that relative indentation.
 type extraIndenter struct {
 	bufWriter
-	afterNewl bool
+	baseIndent int
-	level     uint
+
+	firstIndent int
+	firstChange int
+	curLine     []byte
 }
 
 func (e *extraIndenter) WriteByte(b byte) error {
-	if e.afterNewl {
+	e.curLine = append(e.curLine, b)
-		for i := uint(0); i < e.level; i++ {
+	if b != '\n' {
-			e.bufWriter.WriteByte('\t')
+		return nil
+	}
+	trimmed := bytes.TrimLeft(e.curLine, "\t")
+	lineIndent := len(e.curLine) - len(trimmed)
+	if e.firstIndent < 0 {
+		e.firstIndent = lineIndent
+		e.firstChange = e.baseIndent - lineIndent
+		lineIndent = e.baseIndent
+	} else {
+		if lineIndent < e.firstIndent {
+			lineIndent = e.firstIndent
+		} else {
+			lineIndent += e.firstChange
 		}
 	}
-	e.bufWriter.WriteByte(b)
+	for i := 0; i < lineIndent; i++ {
-	e.afterNewl = b == '\n'
+		e.bufWriter.WriteByte('\t')
+	}
+	e.bufWriter.Write(trimmed)
+	e.curLine = e.curLine[:0]
 	return nil
 }
 
@@ -1220,7 +1262,7 @@ func (p *Printer) nestedStmts(sl StmtList, closing Pos) {
 		//     { stmt; stmt; }
 		p.wantNewline = true
 	case closing.Line() > p.line && len(sl.Stmts) > 0 &&
-		sl.end().Line() <= p.line:
+		sl.end().Line() < closing.Line():
 		// Force a newline if we find:
 		//     { stmt
 		//     }

vendor/mvdan.cc/sh/syntax/quotestate_string.go

@@ -0,0 +1,35 @@
+// Code generated by "stringer -type=quoteState"; DO NOT EDIT.
+
+package syntax
+
+import "strconv"
+
+const _quoteState_name = "noStatesubCmdsubCmdBckquodblQuoteshdocWordhdocBodyhdocBodyTabsarithmExprarithmExprLetarithmExprCmdarithmExprBracktestRegexpswitchCaseparamExpNameparamExpSliceparamExpReplparamExpExparrayElems"
+
+var _quoteState_map = map[quoteState]string{
+	1:      _quoteState_name[0:7],
+	2:      _quoteState_name[7:13],
+	4:      _quoteState_name[13:25],
+	8:      _quoteState_name[25:34],
+	16:     _quoteState_name[34:42],
+	32:     _quoteState_name[42:50],
+	64:     _quoteState_name[50:62],
+	128:    _quoteState_name[62:72],
+	256:    _quoteState_name[72:85],
+	512:    _quoteState_name[85:98],
+	1024:   _quoteState_name[98:113],
+	2048:   _quoteState_name[113:123],
+	4096:   _quoteState_name[123:133],
+	8192:   _quoteState_name[133:145],
+	16384:  _quoteState_name[145:158],
+	32768:  _quoteState_name[158:170],
+	65536:  _quoteState_name[170:181],
+	131072: _quoteState_name[181:191],
+}
+
+func (i quoteState) String() string {
+	if str, ok := _quoteState_map[i]; ok {
+		return str
+	}
+	return "quoteState(" + strconv.FormatInt(int64(i), 10) + ")"
+}

vendor/mvdan.cc/sh/syntax/simplify.go

@@ -16,9 +16,6 @@ import "bytes"
 //     Remove redundant quotes                  [[ "$var" == str ]]
 //     Merge negations with unary operators     [[ ! -n $var ]]
 //     Use single quotes to shorten literals    "\$foo"
-//
-// This function is EXPERIMENTAL; it may change or disappear at any
-// point until this notice is removed.
 func Simplify(n Node) bool {
 	s := simplifier{}
 	Walk(n, s.visit)

vendor/mvdan.cc/sh/syntax/walk.go

@@ -39,7 +39,7 @@ func Walk(node Node, f func(Node) bool) {
 	case *Comment:
 	case *Stmt:
 		for _, c := range x.Comments {
-			if c.Pos().After(x.Pos()) {
+			if !x.End().After(c.Pos()) {
 				defer Walk(&c, f)
 				break
 			}