I like this, I find it annoying to use the tools.go solution, though I'll admit I don't have a better complaint than it being annoying/weird.

If this proposal moves forward, where does the dependency go in the go.mod file? (assuming the 1.17 format with multiple require blocks). Will it have a dedicated block for tools? Or are tools treated like // indirect and placed in the same block?

CC @bcmills @jayconrod

If this proposal moves forward, where does the dependency go in the go.mod file? (assuming the 1.17 format with multiple require blocks). Will it have a dedicated block for tools? Or are tools treated like // indirect and placed in the same block?

Good question! I'm not so familiar with the reasoning behind the multiple blocks... something to do with lazy loading? I'd defer to those with more experience in this area

Personally, I think #42088 is already a pretty good solution. With it, one can write go generate lines like:

//go:generate go run golang.org/x/tools/cmd/stringer@1a7ca93429 -type=Foo

Similarly, go run pkg@version can be used in scripts, makefiles, and so on. Plus, it doesn't even require a go.mod file to be used; you can use this method anywhere, just like go install pkg@version.

Another big advantage is that you can pick specific versions of tools, and they won't interfere with your main go.mod module dependency graph. Perhaps I want to use a generator that pulls in an unstable master version of a library that my project also uses, and I don't want my project to be forced into using the same newer unstable version.

The only downside to #42088 is that, if you repeat the same go run pkg@version commands across multiple files, it can get a bit repetitive. Luckily, you have multiple solutions at hand: sed scripts to keep the versions in sync, short script files to deduplicate the commands, or even a module-aware tool that could sync go run pkg@version strings with a go.mod file, if you wanted to do that.

Or GOBIN=local-dir go install pkg@version, always run from the local directory and not clobber whatever version the user may have globally installed.
I think it would be a mistake for modules to implicitly rely on shared mutable global bin dir for a first class workflow

Oh interesting, thanks @mvdan I wasn't aware of that solution. 🤔

A few concerns immediately come to mind...

1. You mean go run hack.me/now@v1.0.0 will just download and run some random go code 😱 That is slightly unexpected to me, equivalent to a curl | bash command. My assumption was always that go run ran local code or modules already specified in go.mod, but seems that assumption is incorrect

2. Should gqlgen instructions always be to specify version with go run github.com/99designs/gqlgen@0.14.0? That seems verbose

3. Repetition across multiple files, keeping version in sync, yep your comment above nails it

Also this go run solution should probably be added to the Go Modules FAQ if this is now considered best-practice for go:generate tools

In module mode, go run can always download, build, and run arbitrary code. The difference between go run pkg relying on go.mod and go run pkg@version is how you specify the version and how it's verified. With a go.mod, you are forced into a specific version recorded in go.mod and go.sum. Without one, it's up to you what version you specify; @master is obviously risky, @full-commit-hash is safest, and @v1.2.3 is a middle ground that would probably be best for most people. Even if a malicious upstream rewrites a tag to inject code, GOPROXY and GOSUMDB should protect you from that.

Also this go run solution should probably be added to the Go Modules FAQ if this is now considered best-practice for go:generate tools

It certainly warrants a mention. I'm not sure we should bless it as the only best practice, though, because there can be legitimate reasons for versioning, downloading, and running tools some other way. Perhaps some of your tools aren't written in Go, such as protoc, so you use a "tool bundler" that's entirely separate to Go. Or perhaps you do need your tools to share the same MVS graph with your main module for proper compatibility, so you want them to share a go.mod file.

Gotta say though... go run pkg@version seems like a massive security footgun to me.

go install I understand well that it can download code from a remote location and build a binary. It's not obvious at all that go run directly executes code from a remote location, and I wonder how widely that is understood.

So even with the go run pkg@version approach, I still think this proposal has value for specifying tool dependency versions in the context of a module. This approach avoids requiring a tools.go file (as with the existing best-practice), and avoids specifying the tool version for every file that uses it (with the go run approach)

Also worth noting: codegen tools like gqlgen and protobuf are often comprised of a generator command and a runtime, both of which typically need to be versioned in lock-step.

This proposal solves that case rather neatly, allowing go.mod to manage both generator and runtime versions.

Personally, I think #42088 is already a pretty good solution. With it, one can write go generate lines like:

//go:generate go run golang.org/x/tools/cmd/stringer@1a7ca93429 -type=Foo

Similarly, go run pkg@version can be used in scripts, makefiles, and so on. Plus, it doesn't even require a go.mod file to be used; you can use this method anywhere, just like go install pkg@version.

We used to do that. Then people would have that replicated across different files and the version wouldn't always match, and we wanted to automate tool updating, so we figured that migrating to tools.go + having everything in go.mod would be better for compatibility with the ecosystem built around go modules (vs rolling our own tool to keep modules used directly in //go:generate up to date).

Again, tools.go works, but it's weird (not very scientific, I know 🙈). I think this proposal makes version management of tools better because it enables people to manage them using solely go commands (vs things like the bash oneliner shared by the OP).

@jayconrod has previously suggested something similar, using a new directive (perhaps tool?) instead of a // tool comment.

Personally, I prefer the approach of adding a new directive — today we do treat requirements with // indirect comments a bit specially in terms of syntax, but they are semantically still just comments, and I would rather keep them that way at least to the extent possible.

A new tool directive, on the other hand, would allow us to preserve the existing semantics of go mod tidy without special treatment for // tool comments.

@bcmills would such tool requirements be part of the same MVS module graph?

The tool directive would list package paths (not module requirements), and the named packages would be treated as if imported in a .go source file in the main module.

In particular:

• go mod tidy would ensure that the packages transitively imported by the named package (and its test) can be resolved from the module graph.
• go mod vendor would copy the packages transitively imported by the name package into the vendor directory (but would omit its test code and dependencies as usual).
• go list direct (cmd/go: enable listing direct dependency updates #40364) would report the named packages as direct imports.

Or go list tools

I like this proposal. I've had something similar in my drafts folder for a while. @bcmills touched on the main difference. go.mod would have a tool directive that would name the full package path for the tool. You'd still need a separate require directive for the containing module, and that would be treated like a normal require directive by MVS.

module example.com/use

go 1.18

require golang.org/x/tools v0.1.6

tool golang.org/x/tools/cmd/stringer

I don't think go run tool@version and go install tool@version completely replace go run tool and go install tool. When the @version suffix is used, it ignores the go.mod file for the current module. That's useful most of the time, but not if you want to track your tool dependencies together with other dependencies, or if you want to use a patched version of a tool (applying replace directives).

Yeah I like the tool directive. There might be a couple of tradeoffs with compatibility with older go versions. A tool directive wouldn't be recognised by older go versions, and presumably ignored. A require directive with // tool would be recognised, but would be removed by a go mod tidy.

A tool directive would keep the dependency tree separate - as they should be. For example, I don't think indirect dependencies would need to be tracked for tools, or shared by the module. Essentially a tool directive would specify a version when running go run tool instead of needing go run tool@version

Or have I got that wrong? Is sharing indirect dependencies between tools and other dependencies a desirable feature?

A tool directive wouldn't be recognised by older go versions, and presumably ignored. A require directive with // tool would be recognised, but would be removed by a go mod tidy.

Right. The go command reports errors for unknown directives in the main module's go.mod file, but it ignores unknown directives in dependencies' go.mod files. So everyone working on a module that used this would need to upgrade to a version of Go that supports it (same as most other new features), but their users would be unaffected.

A tool directive would keep the dependency tree separate - as they should be. For example, I don't think indirect dependencies would need to be tracked for tools, or shared by the module. Essentially a tool directive would specify a version when running go run tool instead of needing go run tool@version

Or have I got that wrong? Is sharing indirect dependencies between tools and other dependencies a desirable feature?

My suggestion is to have tool act as a disembodied import declaration: it's just in go.mod instead of tools.go. You'd still need a require directive for the module providing the tool, and it would be treated as a regular requirement by go mod tidy and everything else.

If you don't want to mix tool and library dependencies in go.mod, it's probably better to either use go run tool@version or to have a separate tools.mod file, then go run -modfile=tools.mod tool.

Yep that makes a lot of sense @jayconrod

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@jayconrod Did you want to write up the tool directive approach that we could incorporate as an option into this proposal? I'm happy to collaborate on it with you. Positive feedback on that approach so far in this thread, and it would be good to compare the options directly against each other, now that this proposal will be considered by the go-powers-that-be

sorry if I missed it but will there be an easy way to list the tools available? Handy for exploring new code or for a tool to check if there's a local version that should be invoked instead

@jimmyfrasche yes, you’ll be able to run go tool with no arguments to get a list

In the case that two tool directives end in the same path segment, go tool X will error.

Any reason not to support aliasing the tool name, the same way imports work? If I have two tools named stringer, should I be able to do:

tool github.com/path/proj/cmd/stringer 
tool stringer2 github.com/path/proj2/cmd/stringer 

That would allow me to choose the output name of the tool.

We considered this, but decided it added additional complexity for little reason. Currently tools are installed to $PATH, and so tend to have unique names. You will still be able to use the full package path to distinguish between the two if this situation does arise

The spec says:

Subsequent runs of go tool X will then check that the built binary is up to date, and only rebuild it if necessary to speed up re-using tools.

Is it decided whether the "up to date" check here will compare the Go version used to build a cached tool against the current Go toolchain? My company ran into a golang.org/x/tools tools warning about mismatched Go toolchains from https://github.com/golang/tools/blob/559c4300daa4efe55422df9bba86d125cdf1d9ef/go/packages/packages.go#L973 using gqlgen, and it would be great if the cached runs from go tool continued to resolve this as go run does today. Thanks!

@whiten That is a great call-out, thank you. I believe that is already handled by Go's caching, but will make sure that that is the case.

@bcmills / @matloob I've sent a few intro PRs to this several weeks ago, but haven't heard anything back. I'm not familiar with Gerrit, so please let me know if there anything else I need to do to ask for a review.

• https://go-review.googlesource.com/c/mod/+/508355
• https://go-review.googlesource.com/c/go/+/521958
• https://go-review.googlesource.com/c/go/+/521959

(If the changes make no sense, I'm happy to jump on a phone call and talk them through if that's helpful, otherwise happy to collaborate async).

@ConradIrwin, my apologies for the delay. I am still planning to get to these reviews, but it may take me a while to find the bandwidth for a proper review.

Thanks! I will also stop blocking on your review then, and try to send the rest of the patch set. I think it's unlikely that changes to the patches submitted so far will require significant rework of later patches.

Change https://go.dev/cl/534817 mentions this issue: cmd/go: add support for mod tools

@ConradIrwin @bcmills
Any updates ?
Are the reviews of the 3 mentioned PRs on Gerrit still planned?

@piroux thanks for checking in!

I have one more commit to send (to add go get support), and am still awaiting review.

The go get was a bit trickier than expected (I hadn't considered that go get supports wildcards), but I hope to have that in a good state by the end of January, if not sooner.

Change https://go.dev/cl/563175 mentions this issue: cmd/go: add support for go get -tool

@bcmills @matloob I've uploaded the final pieces of this patchset (the changes to go get), and would love to try and get this merged for go1.23.

Please let me know if it would be helpful to talk through any of these changes, otherwise I look forward to your feedback in Gerrit!

@ConradIrwin, unfortunately I won't be able to review these changes after all; I'm leaving Google (and the Go team) on this Friday, March 15.

@bcmills wow, big change! I hope that everything turns out well for you.

@rsc / @matloob is there anyone else that would be interested in taking this on?

Hi all. I've been quietly watching this work for a few months now and I'm quite excited for this new functionality! That said, the last comments on this issue are a little discouraging. If possible, I would appreciate an update. Has anyone from the Go team taken over the code review? Is this progressing?

Thanks everyone for working toward this contribution!!

Unfortunately, at this point I'm not sure if we're going to have the time to review this before the freeze.

That said, I'll try to start reviewing the CLs at the bottom of the stack and we can submit them together if we are able to get the stack done in time.

@matloob I would love to have it reviewed before the next freeze if that's possible :D. Should I connect with you when the window opens again?

Yes that sounds good. I'll try to see if I can start during the freeze so we can maybe make progress before the window opens.