Wouldn't this be solved by CI caching?

Yes, except that (at least with GitHub Actions), CI caching of Go stuff doesn’t always provide noticeable improvements. Long story, but this is a pain point.

This is a real downside. It exacerbates an existing problem, which is modules being redownloaded every time. But modules are smaller, of course. I hope this would be handled by CI systems running caching proxies, which would speed up module downloads too. We also have plans to make the Go toolchains smaller by dropping most or all of the .a files.

On the other hand, given the choice between (1) having to wait for a CI system (or a Linux distribution, or a cloud provider) to update the available version of Go and (2) being able to use any Go version at the cost of slightly slower builds, I'd definitely choose (2).

And of course, CI systems that care about never downloading could force GOTOOLCHAIN=local in the environment, and then the build will break if a newer go line slips into go.mod.

Yes, except that (at least with GitHub Actions), CI caching of Go stuff doesn’t always provide noticeable improvements. Long story, but this is a pain point.

Hi @josharian 👋 , no rush, but do you happen to have a link handy that has some details, or is it possible to boil it down to a ~1-2 sentence description of the area of the problem?

In short: The CI cache is slow. For example, except in pathological cases, it's often faster to recompile than cache build artifacts. People currently have to experiment: Do I cache nothing? The module download cache? The build artifact cache? And then occasionally re-tune it. This would increase that burden.

(I suspect that there are additional issues, like if you do a bunch of work before the cache cleaning mechanism kicks in, the CI cache ends up transferring back and forth a LOT of build artifacts, many/most of which are no longer relevant. Accumulated cache cruft usually doesn't matter all that much on a developer's machine, where the cost is just disk space, but that calculus changes in CI, where the entire cache has to get copied across the network twice--start and end--regardless of how much of it you're going to use. This is a bit OT, and I suppose it should probably should have its own bug to discuss. But I don't want to invest the time right now to chase down the concrete data that would be appropriate when filing said bug. Feel free to file one if you'd like, though!)

It would apply to all go commands, even go commands that don't exist in the earlier version of Go. The go command would find the go.mod and reinvoke the newer go command before even parsing the full command line.

Note that go mod tidy already does change behavior based on the go version: Go 1.17 go mod tidy knows how to emulate Go 1.16 go mod tidy for modules that say 'go 1.16'. That's the backward compatibility half. The forward half doesn't work; this would make it work.

Yes, exactly. This is making the go version work very analogously to module dependency versions. Originally we thought that was too onerous because it required manual updating to a new Go version to resolve a problem. The automatic updating should make it more reasonable.

I agree that it's a good thing to have, but Go modules don't specify the exact version of their deps, just the minimum version.

This is not true. The current module's go.mod (the one that's in your working directory, or somewhere above your working directory in the file tree) lists the exact versions of the modules that are being used in the build. It is true that for a dependency incorporated into a larger program, the larger program may use a newer version of what that dependency's go.mod lists. But at the top level of the build, the go.mod has an explicit list, providing a reproducible build.

The current module's go.mod (the one that's in your working directory, or somewhere above your working directory in the file tree) lists the exact versions of the modules that are being used in the build.

Sorry, I don't understand. If main module M requires A v1.0.0 and B v1.0.0, and B v1.0.0 requires A v1.1.0 because it provides a new generic function, main module M has to be built with A v1.1.0, otherwise B won't compile. From the mod reference:

MVS starts at the main modules (special vertices in the graph that have no version) and traverses the graph, tracking the highest required version of each module. At the end of the traversal, the highest required versions comprise the build list: they are the minimum versions that satisfy all requirements.

It's my understanding that the main module can constrain versions with replace and exclude directives, but it can't just say "use only version V for this module, no matter what other modules require." From the mod reference:

The graph may be modified by exclude and replace directives in the go.mod file(s) of the main module(s) and by replace directives in the go.work file.

I must be missing something.

Sorry, I don't understand. If main module M requires A v1.0.0 and B v1.0.0, and B v1.0.0 requires A v1.1.0 because it provides a new generic function, main module M has to be built with A v1.1.0, otherwise B won't compile. From the mod reference:

Your understanding is accurate up to Go 1.16. The story changed with Go 1.17. See https://go.dev/doc/go1.17#go-command, https://go.dev/ref/mod#go-mod-file-go, https://go.dev/ref/mod#build-commands, and https://go.dev/ref/mod#go-mod-file-updates.

In summary:

• If the go.mod file has a go directive for Go 1.17+ then "[t]he go.mod file includes an explicit require directive for each module that provides any package transitively imported by a package or test in the main module."
• Most [go tool] commands report an error if go.mod is missing information or doesn’t accurately reflect reality. The go get and go mod tidy commands may be used to fix most of these problems. Additionally, the -mod=mod flag may be used with most module-aware commands (go build, go test, and so on) to instruct the go command to fix problems in go.mod and go.sum automatically.

When these two behaviors are combined (for Go 1.17+ main modules built by a Go 1.17+ toolchain) then I believe @rsc assertion holds. Either the go.mod specifies all the versions required for the build or the build will report an error about an inconsistent go.mod file and push you to fix it.

The only exceptions I can think of are the presence of local replacements in the main go.mod (which specify a local path without a version) and a go.work file in play (which could introduce newer versions of dependencies if other modules in the workspace require them).

When these two behaviors are combined (for Go 1.17+ main modules built by a Go 1.17+ toolchain) then I believe @rsc assertion holds. Either the go.mod specifies all the versions required for the build or the build will report an error about an inconsistent go.mod file and push you to fix it.

OK, but that's just putting all the transitive requirements into the module, not overriding/controlling them. In my M/A/B example, M still couldn't force the use of both A v1.0.0 and B v1.0.0. It simply can't build in that case, regardless of the module-fu we try (the new function wouldn't exist). It looks like if you do try to do that, the go command will report an error before it ever gets to the point of trying to build it. The only control I see is the ability to bump the minor version higher, but then again, so does every module, not just the main module.

I think I understand now. If I understand correctly, @rsc was saying that the main module's requirements are complete and explicit about the specific version of each dependency. If the main module requires a lower version than what MVS concludes is required, the go command will report an error, and suggest that the mod deps be updated.

To go back to the original context:

The situation with the Go toolchain today is exactly like what GOPATH did for dependencies before modules. [...] In fact you could argue the Go toolchain may be the most critical dependency any build has, yet it's the only one that we don't provide these kinds of versioning guarantees for.

I agree that it's a good thing to have, but Go modules don't specify the exact version of their deps, just the minimum version. Doesn't the go directive in modules accomplish the same thing for the language version? In your Docker/Kube example, the MVS conclusion would be that v1.20.4 should be used for both. Doesn't that provide a reproducible build?

If you take the go directive to be the exact Go version "required," then it seems to me that there's still a parallel with required module versions. If main module M only needs A v1.0.0, but has to use A v1.1.0 due to another module's requirement, then there's risk there that something will break. If main module M only needs Go v1.18, but has to use Go v1.19 due to another module's requirement, then there's risk there that something will break. In either case, you upgrade at your own risk.

do we have a way of saying I want the latest 1.80 toolchain or at least easily hiding this complexity from developers?

No, for the same reasons we don't have a way of saying I want the latest version of a given module: that changes day by day and you lose reproducible builds: the same source can produce different compiled binaries and possibly even different behavior from one day to the next. Instead the build toolchain version in the go.mod is completely precise and users control when it changes.

Of course, we should make it easy to change. Maybe something like 'go get -u toolchain' or 'go get -p toolchain'.

Sure, that seems reasonable. It is almost indistinguishable from not having a toolchain line. The only difference would be that if the toolchain is too old, the build breaks instead of automatically grabbing a new enough one.

Seeing users wanting toolchain local to opt out, I wonder why other approaches like nvm or rustup that provides an easy way for users to switch between versions when they find a different version of go is not a good solution.

@hyangah from personal experience, it's a people problem. With a large enough team, it's hard to get everyone to use the same versioning tool consistently. People are used to running go build and changes are hard and painful. I think part of why modules succeeded as a transition is because, largely, the build and test commands remained the same.

In our organization, tooling updates are done in org-level and they should be security verified. I think showing the warning that suggests actionable messages (e.g. "hey you need this version for best functionality, run go up 1.900.1") seem better than that. Another issue to consider is go is not the only tool developers depend on but some tools are sensitive to what version of go command is being used. For example, dlv needs to be a certain version to work with a certain version of go. gopls is also planning to implement version support window policy. By automatically picking the tool versions, that means other tools or ides have to implement similar mechanism to pick the right versions of tools.

Thanks for mentioning nvm and rustup. I had not looked closely at either before your comment. Both are analogous to what this discussion is about the go command doing. I don't see why it should be 'goup' instead of 'go' that does it. Why use two tools when one tool will do? We can do a much better job inside the 'go' command than any tool on the side can.

Rustup has 4 different ways to select the toolchain; at least what I described only has 2. :-)
https://rust-lang.github.io/rustup/overrides.html

Definitely true.

Worth noting that if you have C -> B -> A in one project and D -> B -> A2 in a different project, you're already getting two different compiled copies of B, because of the different As sending different export data up to B's imports. And if you are working hard to share exactly the same set of dependency versions across projects, it seems like the same sync mechanism could handle toolchain too.

Perhaps this proposal should go hand in hand with a slightly better heuristic around the sizes of Go caches, like #29561.

When it comes to Go minor versions I would maybe want some way to at blacklist some of them.

I have had programs that breaks/panics on a couple of 1.x.0 Go releases that were fixed in the .1 fix release. I don't know if a minimum minor Go version is the right solution for that situation because both v1.100.1 and v1.99.0 could be compatible while v1.100.0 isn't because of compiler/runtime bugs. A subtle aspect of this issue is that it can be a bug that only happens on a particular OS or architecture but for simplicity sake I would be satisfied with flat out reject a whole Go minor version anyway, the most important thing is that go install/build makes a fuss when you try to build something that is known to be broken.

It could also be argued that exempting Go version from the same semver rules that applies to Go modules would be a bad idea because it is less straight forward and not with it. The scenario described above isn't really possible to express within MVS though.

For the go.mod go line, the toolchain will only auto-upgrade, not auto-downgrade.
For the toolchain line, yes, it would make it possible to downgrade to an older toolchain.
Remote build environments should almost certainly be sandboxed anyway,
but they could also force GOTOOLCHAIN=auto to override the toolchain line but still allow upgrades to newer versions.

If you've gone to the trouble of installing Go 1.90 on your machine, I think that's a strong sign that you want to use it, even to compile older Go code. You can always override that if you want, of course, by setting GOTOOLCHAIN or adding a toolchain line to go.mod.

I assume that people using those toolchains would be unaffected. But part of the reason for the 'go' prefix in 'toolchain go1.90' is so that we could distinguish 'toolchain gccgo1.90' or 'toolchain tinygo1.90' at some point in the future.

1. Automatic binary download/execution seems a bit dirty (something that doesn't feel right to me)

I second this concern.

Is it possible to download the source code of the requested toolchain and build it on the fly? It might alleviate some of the concern on downloading & executing new binaries. This might be a little trickier to do but as for now, Go module builds generally build everything from source (except .syso files) and do not download or run any binary blobs (again, with the exception of syso files; but at least those are not executed during the build.)

At the very least, I'd want a documented way to instruct the Go command to not download unavailable toolchains and instead error out and in the error message, point out the exact reason (which module requires which Go toolchain version, etc.)

To opt out you would set GOTOOLCHAIN=local or go env -w GOTOOLCHAIN=local.

Obviously we have to get the implementation right, but at the moment you already trust commands like 'go run' or 'go test' to download bits from the internet, read those bits into memory, interpret them in some way, write out different bits to a new file, set the execute bit on that file, and run it. I honestly don't see how any security scanner will distinguish what the go command already does from what's contemplated in this discussion. Perhaps an overzealous security scanner could intercept the HTTPS connection and see that the zip file being downloaded contains an executable and block the download?

If I understand the problem to solve I think this should be opt-in, as in, trying non-released versions seems a non-standard use case to me. Even if it's no the case I would still prefer an explicit opt-in for this for at least some point release given how big is the change in behavior.

Obviously we have to get the implementation right, but at the moment you already trust commands like 'go run' or 'go test'

From a security perspective I think there's a big difference because of go.sum. In the case of "go run" and "go test" the download is of known files because there's an hash to validate the downloads. In the case of downloading a new toolchain it would be unknown because the local "go" binary has no information even if the "toolchain 1.73" exists.

Perhaps an overzealous security scanner could intercept the HTTPS connection and see that the zip file being downloaded contains an executable and block the download?

I have seen that behaviour in corporate machines. Usually developers ask an opt-out of those security scanners/firewalls rules or need to ask IT to add it to the allow-list.

Is it possible to download the source code of the requested toolchain and build it on the fly?

Note that there's no guarantee that the existing toolchain will be able to compile the requested toolchain. You might need to go through some intermediate toolchain versions if it relies on language features not present in the existing toolchain.

That may not be true in Go 1.20. I made some changes at the start of the current dev cycle to try to make the distributed toolchains work equally well on musl and non-musl systems. If the toolchain for Go 1.20 beta 1 doesn't work out of the box on Alpine, please file an issue. Thanks!

For reference, I think that would be https://go-review.googlesource.com/c/go/+/420774.

Yes, that's the idea. If Go 1.119 is still in beta, then you will get an error when you try the next command. You could fix that with 'toolchain go1.119beta1'.

Thanks for sharing your experiences with the current system. These kinds of examples are very helpful.

go mod download would in fact fetch the new Go toolchain, but because of the go, not the mod download.
Other commands like go version or go help would also download the new Go toolchain.

The specific problem in #43996 is caused by shipping .o files built on one machine in the distribution and expecting them to work on a different machine. We would like to not do that at all (#4719) and are moving slowly toward that.

Thanks for the clarification. #4719 is a blast from the past, I'd forgotten that was still ongoing. Your comment here #4719 (comment) was a good reminder to me that:

$GOROOT/pkg is difficult to kill because we want to keep allowing the building of programs with cgo-enabled package net from systems without a C compiler. We will probably have $GOROOT/pkg or an equivalent for quite some time.

At the risk of being off topic, is it still a goal to support building cgo-enabled package net without a C compiler? If so, how would that work without shipping those .o files with the downloaded toolchain?

Either (1) we drop that goal or (2) we find a way to make the default builds not use any C code. The latter may be possible on any system with a dynamically linked C library, which at least covers Linux, Mac, and Windows. We'd have to maintain by hand the linker directives that cgo derives automatically today, and we'd have to make sure that rebuilds still happen properly when important settings change (like trying to do a static build). It's still up in the air but I think doable.

(Not content-related, but how did you fix my list so that the two-paragraph first bullet rendered properly? I noticed that but did not know how to avoid it.)

Thanks for the careful thoughts. To reply to a few of them:

(1) Builds using Go's dev branches would certainly handle any go version listed in go.mod, so the only possible time they'd delegate to an alternate toolchain would be an explicit toolchain line. Since this is a discussion and not a proposal, I don't have every detail worked out: maybe such builds would default to GOTOOLCHAIN=local instead of GOTOOLCHAIN=auto, under the theory that you're working on a bleeding edge Go and want to use it in preference to others. I'm not sure.

(2) There's always the possibility of added confusion, but most problems I can think of would also be caused simply by having two different Go toolchains on a machine. If things like 'go install golang.org/dl/go1.19@latest; go1.19 download; go1.19 build' are OK today, then this should be OK too. We were pretty careful in the design of the module cache and build cache to allow different versions of Go to coexist.

(3) I believe the point is that 'toolchain go1.17' makes it easy for a project to keep using an older version of Go, and contributors working on it will "auto-downgrade" and not necessarily provide motivation for updating. I suppose that is true, although I'm not sure how common that would be. Perhaps we can do something to flag this more aggressively, although I am not quite sure what.

(4) Yes, definitely. The go.mod file has no multiline comments, so it suffices to do something like a search for (?m)^\s*go\s+(.*)$ in the file.

Thanks again.

(In Markdown you make extra paragraphs continue the list item by indenting them at least 3 spaces.)

Thanks for your reply!

1. Making tip default to the local toolchain is an interesting idea. I think I would like that, personally. If I'm using Go tip, it is to explicitly test out that newer Go version. I wonder where that would leave beta and RC pre-releases; I guess those should work like stable releases.

2. It's true that this is already a problem - it would just make using multiple Go versions at once much more common. I guess this depends on how much you anticipate to change how Go's caches work, or e.g. the default location of GOMODCACHE. I am worried about changes like cmd/go: add GOMODCACHE #34527, for example, which could lead to different Go versions using different cache locations.

3. I am indeed thinking that we'll want some way to flag this to users. Perfectly fine to leave that out of the initial implementation, but I think it should be mentioned in the proposal somehow.

Thanks for sharing this use case. I really appreciate these kinds of details.

In this case, it sounds like you control both the toolchain and the source code being loaded into the SBC, including the go.mod files. As long as the go.mod files do not specify a newer Go version in their 'go' lines, nor a toolchain line listing a different version of Go than what's installed, everything would continue to behave as before.

Setting GOTOOLCHAIN=local would override any go.mod toolchain line. If the go.mod says 'go 1.101' when the toolchain is Go 1.100, then GOTOOLCHAIN=local will cause the build to fail, but so would not having the internet, so there's not too much difference. Either way, since you control the go.mod files on the computer, it sounds like you can easily ensure that they don't ask for something that the local toolchain can't already supply.

Thanks for the note. It should suffice to use something like 'toolchain local' in the go.mod you give your students or else export GOTOOLCHAIN=local.

The problem I have is that I do not want my compiler to be reaching out to the Internet and installing a new version of the compiler somewhere without my control.

@prologic You can do GOTOOLCHAIN=local for that.

The go tool is not the compiler, it's an all-round tool to make a Go programmer's life easy.

@prologic You can do GOTOOLCHAIN=local for that.

I shouldn't have to do this 🤦‍♂️

I somewhat agree, but if you run go env -w GOTOOLCHAIN=local it'll set it for all future calls by that user.