Thanks for filing this issue. In my comment in the original proposal I did suggest filing a new issue from the point of view of a bug report, so I agree with you that we should begin with the problem and not multiple potential solutions. I hope this doesn't mean I get more thumbs down reactions :)

I fully agree that this needs a decision before 1.16, even if the decision is that we're okay with the current semantics. I also tend to agree that excluding files by accident is less harmful than including them by accident, because the former is easy to spot but the latter could go unnoticed for a long time.

I'm still uneasy about introducing the notion of "hidden files" in the Go toolchain, but they do have a sort of precedent:

$ go help packages
[...]

Directory and file names that begin with "." or "_" are ignored
by the go tool, as are directories named "testdata".

Perhaps we could copy the same rule here. * is about files and not packages, but since builds happen inside package directories, I think it could make sense to be consistent. And the notion of "ignored filenames" by the toolchain would remain easy to remember.

I think that would be a fine approach (as long as explicit listing or a .*-glob would still allow you to include them). Just to be clear though:

* is about files and not packages

I want to make sure we are in agreement that this isn't just about globs specifically. As I said, I think //go:embed assets also shouldn't embed assets/.DS_Store (for example), even though it doesn't contain a glob.

I think this issue has made a very strongly case for "fail safe" instead of "fail unsafe but document/provide an escape hatch" semantics.

One additional thought: if we get the semantics "wrong" in 1.16, would it be possible to revise them in 1.17 with a go.mod directive? If so, that to me suggests doing something "conservative" in 1.16 and revising to be "liberal" in later versions of Go if the conservative thing was found to be too conservative.

CC @rsc.

@Merovius agreed. The rule would limit both recursing into directories, and the * glob. I think any other glob shouldn't be affected, so that one may use globs such as .* or _*.

Icon\r isn’t dotfile, but it is hidden. Should Go include it?

In a similar vein, any file can be marked hidden on NTFS. Should //go:embed read filesystem attributes, even on a Linux host compiling code on an NTFS volume, to ignore thumbs.db?

For sake of completeness: we need the ability to exclude files on purpose, thunk assets.go. Might also solve excluding dotfiles.

#42325 was closed to focus discussion here.

The proposal there was to add a magic comment for negative globs, like //go:embed-exclude *.go. The objection was that it would lead to a lot of OS/IDE specific exclusions being listed and still miss some things that should filtered when encountering a new environment.

My current feeling is that a combination of approaches should be taken:

• Dot files should be excluded unless specifically included. (Definitely do this.)
• Hidden files should probably also be excluded to the extent that we can figure out what files are hidden on the system. Since hidden files this is system specific, it may not always work properly, e.g. across a network. (Maybe do this if it can be made portable.)
• A way to specifically exclude files might be nice. (Can be left for Go 1.17 if we exclude dotfiles now.)
• Go source files could be automatically excluded at the cost of losing a few blog posts about easy Quines. (Probably do this unless there is a compelling usecase.)

@andig I don't agree that that's a necessity for the usecase you mention. There is no need to have assets.go in the same directory, you can have it in a directory further up and use a StripPrefixFS (if it's not in the stdlib, it can always be provided by a third party).

A well tested approach, see https://docs.docker.com/engine/reference/builder/#dockerignore-file , is providing something like a .goembedignore file. The .gitignore file is another such example.

Those are well understood semantics, easy to share via dotfile collection, easy to configure in template repositories, easy to add to UIs for repository creation like github.com or gitlab.com as well as repository compliance checkers often found in enterprises can easily verify its existence, syntax and content.

The extensions star-star-match, match-exceptions as well as the ability to add comments mentioned in the .dockerignore documentation also proved critical to practical use.

Adding such information to each mention of go:embed using wildcards seems unpractical to me and too easy to forget.

This problem has been solved pretty well before and I don't think Go needs yet another way to be special here.

Followup questions now are:

• Where are those .goembedignore files allowed?
• Should the go:embed directive allow mentioning such a file?
• Can mentioning this file to go:embed pragma be the only way that this information is passed?
• Do we still need sensible defaults then or is explicit better than implicit here?

Interesting idea. Maybe the default is to exclude dot files and .go files and then tweaking the go.embedignore file for a module could change the parameters to something else. It’s an interesting idea that could wait for Go 1.17 if the window for making it into 1.16 is too narrow now.

@rsc / @dmitshur For Go 1.16 I would suggest to revert the ability to add wildcard based trees to go:embed until this issue is solved so we can explore the solution space without releasing a feature with a potential security risk given such precedents in Docker and git before each had a way to ignore certain file patterns recursively.

Even without a wildcard pattern, the problem of including a directory that has an unexpected file will be there.

Personally, I'm opposed to .goembedignore and go:embed-exclude in general, for a similar reason why we rejected a .goignore in #30058. Here, you have two options (besides my proposal in #42328 (comment)):

A) To only include a subset of the files, use go:embed on a subdirectory and place them there.

B) To exclulde some sub-directory if option A isn't desirable, you could always drop a go.mod in there to exclude the directory from the current module.

The files referenced by //go:embed are effectively part of the build, just like *.go, *.c, etc.. Ideally it would be similarly easy to understand which embedded files are included. The rule to determine which files are included should be simple and avoid surprising results.

Excluding all files starting with "." seems simplest and least prone to confusion. This avoids including files which typically aren't visible in the directory, and entries which typically related to other system purposes (eg, .DS_Store). build.Context.Import also excludes _, so it might be reasonable to exclude it here as well.

Perhaps this rule could be extended to include dot-files when the pattern/filenames explicitly starts with ".". Another option would be including explicitly referenced files. Eg: //go:embed data data/.mysecrets. However, I suspect few codebases would be motivated to use this option.

I think it's counterintuitive to exclude certain files, especially when I specify a directory, I expect everything inside it. Furthermore, most (all?) of the existing tools do not have such behaviour.

From the original issue description, I would argue it is fine for in-development builds to contain extra files such as .DS_Store, building from a clean repo for release is something we should promote and is more or less equivalent to what already happens for go get / go install

I really dislike the inclusion of hidden files. However, this is what http.Dir for example says:

Note that Dir could expose sensitive files and directories. Dir will follow symlinks pointing out of the directory tree, which can be especially dangerous if serving from a directory in which users are able to create arbitrary symlinks. Dir will also allow access to files and directories starting with a period, which could expose sensitive directories like .git or sensitive files like .htpasswd. To exclude files with a leading period, remove the files/directories from the server or create a custom FileSystem implementation.

Hi all. Thanks for the lively discussion and thanks to @Merovius for the summary.

Regarding the open questions in the summary, the current proposal's answers are:

Will hidden files also be excluded if a directory is matched by a glob, or just if it is mentioned by name?

Yes. A name is a glob. There is no distinction. To process //go:embed, you figure out what is matched and then also walk directories that were matched.

Does "hidden" also apply to extended filesystem attributes?

No, just to .* and _* names.

Should _* and testdata really be skipped (comment by @mpx)?

testdata definitely not. That's not in the proposal. (I understand the confusion, but it was deliberately omitted compared to the go command.) But _* is there for a reason and so I think it makes sense to preserve. In particular the go command reads a _netrc file on Windows instead of .netrc, so at least some tools do treat _ as the Windows equivalent.

Regarding the alternatives, it's important not to deviate too much from what we already discussed and accepted. It's clear that being able to say //go:embed static and embed a directory tree is a very important case. We're not going to remove directory walking entirely. It's also clear that //go:embed static/*.jpg is important. We're not going to remove globs entirely. I think the proposal would not have been accepted without both of these, and we should respect that it was.

I would be interested to hear estimates of how often "everything on disk" will come up versus "ignore dot files". Clearly lots of people don't want dot files, hence this and other issues. And I also like that Hugo is ignoring them too.

Overall while there's lots of discussion since two weeks ago, I'm not seeing any clear change in consensus since we marked this "likely accept". It's a difficult call, but since embed is a new feature, "fall back to doing nothing" is not as strong. We are introducing a new feature, and we need to decide what is best for that new landing. There's no prior behavior to bias toward. More data would be helpful, as I mentioned in the previous comment, if anyone has any.

Overall it still sounds like we should make the directory change. If it's truly awful and wrong for the vast majority of Go users, it's easy to change in the future based on the go version in go.mod. But I doubt we're at that point. Instead, it's closer to a 50-50 call and we have to decide something. Both choices are good (and bad). On balance, the choice that doesn't accidentally vacuum up potentially sensitive files seems like the better one.

But _* is there for a reason and so I think it makes sense to preserve. In particular the go command reads a _netrc file on Windows instead of .netrc, so at least some tools do treat _ as the Windows equivalent.

I appreciate we have accepted a solution for dot-files (thanks, it greatly reduces the risk in my environment). However, I think it's worth looking at underscore-files a little closer since there are no examples of problems with _* in this thread and it appears the same justification does not apply:

• tools automatically create dot-files (not underscore-files?)
• dot-files are hidden under Unix (underscore files are not hidden)

The decision for using _netrc for Go was in https://go-review.googlesource.com/c/vgo/+/103866/. os.UserHomeDir is almost certainly outside any embed data directory.

I suspect _netrc is a consequence of porting ftp to early DOS/Windows and git continues to use it today - not a wider trend. Afaik, underscore config files are not common under Windows (but I'm far from an expert :) ). There is probably some historical reluctance to use dot-files under Windows due to it's FAT heritage with 8.3 filenames -- but this is no longer valid with "modern" filesystems (since Windows 95).

Can anyone suggest examples of tooling that creates underscore files under Windows or Unix that might interfere with //go:embed?

To be clear, I'm sure there are many people who would prefer to include underscore files (many comments above), no need to reply on that point - I'm just trying to find examples where excluding underscore-files might help.

There's been a lot of discussion but I don't see a change in the consensus from when this got marked likely accept. Let's make the change for the beta (hopefully next week) and then get more feedback. For now, accepted.

Change https://golang.org/cl/275092 mentions this issue: cmd/go, embed: exclude .* and _* from embedded directory trees

I just hit this while working with 1.16beta1. I have a //go:embed of ~700 files, ~70 directories and where some files have a leading underscore; these are not dot files, nor go source files, and it took me some time to work out that the reason I couldn't open some of them without error was that they were excluded from the build.

I've since found and read through the discussion here. Please let me know if there is a more appropriate place to provide feedback now that this issue is closed. Otherwise, please read on...

The draft proposal doesn't mention this restriction (I now realize it just hasn't been updated since July), and it was counterintuitive to me that files would be arbitrarily excluded because they're considered "hidden" on some platforms. Why counterintuitive? I don't know of any other tools that automatically exclude dot or leading-underscore files, other than ls for dot files, which provides the well-known -a escape hatch. Git, find, tar, etc., include such files by default. Even the macOS Finder allows you to toggle display dot files with ⌘-shift-..

Sometimes we need to include everything in an embed of a directory tree. Please give us a way out! :) One suggestion is a simple flag like //go:embed -all dir. Otherwise having some explicit filtering option in the directive (with an "otherwise include-all") would be welcomed, as was suggested in #42325. The proposed //go:embed dir vs. //go:embed dir/** is very subtle and reminds me of the arcane build constraint syntax that was thankfully recently revisited. An explicit inclusion-list, written by hand or code generated, is a big turn-off.

Right now I am debating whether to write more code and wrap my file tree in a store-only ZIP file and //go:embed that, or just use a 3rd party embed code-generator tool and eat the slower build times. Neither approach is preferred because they burden the project with an external generator. (It's been helpful during development to have the embed available as-is on the filesystem, so if using a ZIP file or generated embed.go file the content will be kept in both forms.) Another approach which I dislike is to use some encoding to escape filenames that //go:embed otherwise denies me from using naturally. That would force design changes to a larger surface area of an otherwise already complex project just to simplify the build.

I just hit this while working with 1.16beta1. I have a //go:embed of ~700 files, ~70 directories and where some files have a leading underscore; these are not dot files,

It sounds like simply including underscore files would work. I'm not aware of any situations where including underscore files would be problematic and I haven't seen any in this issue (it's highly unlikely _netrc would inadvertently appear within a module).

Unfortunately providing a -all option would be problematic too:

• Extra complexity: supporting the inclusion of file named -all would require escaping as well.
• Code using -all might incorrectly include dot-files when compiled on some systems (see the many examples in this issue).
• Any changes this late in the cycle would have to be very simple.

@rsc Perhaps excluding underscore files could be reconsidered?

@jpap I'm not sure, but it may help to open a separate issue to provide this feedback since this is closed and may not get attention.

Better documentation will solve everything.

Unfortunately providing a -all option would be problematic too:

• Extra complexity: supporting the inclusion of file named -all would require escaping as well.

That is easily avoided though a "--" terminator, and easily implemented using the flag package.

For example, to include a file called -all, you would simply write //go:embed -all -- -all. This pattern is likely familiar to most go developers, because we often use it on the command-line. This is also likely an edge case.

• Code using -all might incorrectly include dot-files when compiled on some systems (see the many examples in this issue).

Using -all is opt-in, an escape hatch, so yes -- you would get all dot-files because maybe that's what you want! :)

• Any changes this late in the cycle would have to be very simple.

A change that leveraged the flag package is likely smaller than CL 275092 above.

In src/cmd/go/internal/load/pkg.go(*Package).resolveEmbed, you could feed patterns into a flag.FlagSet having an "all" bool flag, then use it to short-circuit the offending condition:

if path != file && (isBadEmbedName(name) || name[0] == '.' || name[0] == '_') {
	// Ignore bad names, assuming they won't go into modules.
	// Also avoid hidden files that user may not know about.
	// See golang.org/issue/42328.
	if info.IsDir() {
		return fs.SkipDir
	}
	return nil
}

This is probably the wrong place for discussion but if you just need to embed some files with underscores, a quick workaround is that you can write a wrapper fs.FS that changes file names from -whatever to _whatever automatically and then rename the files on disk to start with a dash.

The workaround we've discussed is to write a small go generate program that generates a //go:embed comment. Something along the lines of

package main

func main() {
    const base = "embed"
    var dirs []string
    err := filepath.Walk(base, func(path string, info os.FileInfo, err error) {
        if err != nil {
            return err
        }
        if info.IsDir() {
            rel, _ := filepath.Rel(base, path)
            dirs = append(dirs, filepath.Join(rel, "*"))
        }
    })
    if err != nil {
        log.Fatal(err)
    }
    err = ioutil.WriteFile("embed.gen.go", []byte(fmt.Sprintf(template, strings.Join(dirs, " "))), 0644)
    if err != nil {
        log.Fatal(err)
    }
}

const template = `// Code generated by go run embed.go; DO NOT EDIT.

package pkg

import "embed"

//go:embed %s
var embedded embed.FS`

That way you still get the efficiency and integration of the native embedding support (which you lack if you use a third-party tool). I agree that it's less convenient to still have to run go generate when the set of files changes, but I'm still convinced that this is enough of an edge-case that it's fine if it,s less convenient.

In any case, I think the release of go 1.16 is imminent, so I think it's too late to change it for this release. I see that you've opened a new issue, I think we can discuss options there.

This is probably the wrong place for discussion but if you just need to embed some files with underscores, a quick workaround is that you can write a wrapper fs.FS that changes file names from -whatever to _whatever automatically and then rename the files on disk to start with a dash.

That doesn't work because these files aren't included in the executable to begin with: there's nothing to filter.

I've opened #43854 if you want to continue the discussion.

The workaround we've discussed is to write a small go generate program that generates a //go:embed comment.

This is an undesirable workaround for the reasons already discussed:

• Use of an external generator.
• Not automatically kept fresh.
• Increased complexity compared to a simple -all flag, or just not excluding files with leading underscore.

Let's continue the discussion in #43854.

That doesn't work because these files aren't included in the executable to begin with: there's nothing to filter.

The idea is that you change a/_whatever.txt to a/-whatever.txt and then add a wrapper than interprets Open("a/_whatever.txt") as meaning Open("a/-whatever.txt") and similarly changes the ReadDir calls. It's an ugly hack, but probably could be done in like 20 LoC plus a bulk rename.

The idea is that you change a/_whatever.txt to a/-whatever.txt and then add a wrapper than interprets Open("a/_whatever.txt") as meaning Open("a/-whatever.txt") and similarly changes the ReadDir calls. It's an ugly hack, but probably could be done in like 20 LoC plus a bulk rename.

Thanks for the clarification. That's the approach I also found undesirable in my original post, quoted again below.

Another approach which I dislike is to use some encoding to escape filenames that //go:embed otherwise denies me from using naturally. That would force design changes to a larger surface area of an otherwise already complex project just to simplify the build.

In my project that change isn't limited to a one-time bulk rename: it means changing the design of the all the code that uses this directory/file tree structure. The project uses many such directory trees; most are used outside of //go:generate on a live filesystem. The motivation for using //go:generate here is that there are some special trees that are "constant" and need to be distributed with the executable. Others that live on the filesystem are specific to the environment in which they run.

Although my initial inclination was to add a -all flag, or similar, I would worry about the additional complexities that incurs. Specifically that escaping or other escape hatches have to be added to the mix.

One alternative may be to have a second directive with different semantics:

• //go:embed — keeps the existing behavior
• //go:embed-all — embeds all files

This could become unwieldy if many permutations were anticipated to be added in the future.

One additional thought: does the module proxy respect embed directives? If not, the files may be excluded before the embed directive takes place anyway.

Edit: I'll go add this comment to the discussion on #43854 as well.

Change https://golang.org/cl/359413 mentions this issue: cmd/go: add //go:embed all:pattern