Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: preserve basic GOPATH mode indefinitely #60915

Closed
rsc opened this issue Jun 21, 2023 · 19 comments
Closed

cmd/go: preserve basic GOPATH mode indefinitely #60915

rsc opened this issue Jun 21, 2023 · 19 comments

Comments

@rsc
Copy link
Contributor

rsc commented Jun 21, 2023

In Feb 2021 (https://go.dev/blog/go116-module-changes), we wrote:

It's still possible to build packages in GOPATH mode by setting the GO111MODULE environment variable to off. ... We plan to drop support for GOPATH mode in Go 1.17. In other words, Go 1.17 will ignore GO111MODULE. If you have projects that do not build in module-aware mode, now is the time to migrate. If there is a problem preventing you from migrating, please consider filing an issue or an experience report.

The time has come to decide what to do about GOPATH mode. There are at least two important problems with maintaining GOPATH mode as it exists today:

  • The old, GOPATH-mode go get is being left behind as far as security improvements like the module proxy, checksum databases, and so on. All the focus is on modules.

  • The old, GOPATH-mode source layout does not provide a way to identify the language version used by the source code. There is a bit of divergence here in that a go.mod with no go line assumes Go 1.16, while in GOPATH mode we assume the latest version of Go. This means if someone has downloadable packages they develop in GOPATH mode, they can use language features introduced after Go 1.16, like generics, but when clients download that code in module mode as a "legacy" module, the code is interpreted as Go 1.16 and does not compile. The divergence will become greater over time as other language features or changes are made, such as spec: less error-prone loop variable scoping #60078.

On the other hand, GOPATH mode remains the only way to work on dependency-free legacy packages that existing module code may still depend on (so-called "+incompatible" modules), and it remains the only way to build historical source code that predates modules, especially code that depends on the multiple layers of vendoring that Go 1.5 introduced (and modules removed).

Given these tensions, I propose that we do the following:

  1. Commit to preserving the ability to build GOPATH-layout source trees when GO111MODULE=off, indefinitely.
  2. Disable go get completely in GOPATH mode, since there is increasingly little code that it can download successfully and provides a worse and worse experience.
  3. Assume a language setting of Go 1.21 in GOPATH mode, instead of continuing to assume "the latest Go version". This will ensure that if we change for loops in Go 1.22 (spec: less error-prone loop variable scoping #60078) or make other changes in the future, the old, legacy code we are aiming to keep compiling does keep compiling as written.

We would make these changes for Go 1.22. The GOPATH mode enabler would continue to be GO111MODULE=off rather than introducing a cleaner name like GOMODULE=off, because there's little point to upsetting whatever scripts people with legacy trees have already written.

@rsc rsc added the Proposal label Jun 21, 2023
@gopherbot gopherbot added this to the Proposal milestone Jun 21, 2023
@bcmills bcmills added the GoCommand cmd/go label Jun 21, 2023
@dominikh
Copy link
Member

Do you expect that tools like gopls will continue to support GOPATH mode? While go list takes care of a lot of the details, it still requires some special handling in go/packages AFAIU — and possibly in the tools' themselves.

@rsc
Copy link
Contributor Author

rsc commented Jun 21, 2023

I would expect gopls and go/packages to continue to support GOPATH as well as they do today. There should be very little special code they need, and it's already written.

@rsc
Copy link
Contributor Author

rsc commented Jun 21, 2023

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@willfaught
Copy link
Contributor

What examples are there of GOPATH packages we want to keep working with the latest toolchains?

The steps to convert GOPATH packages to modules are:

  • go mod init ...
  • go get ...
  • go mod tidy

Unless I missed something, that's not much effort. This is basically equivalent to forking a package that you know will be deleted soon. In a sense, GOPATH packages would be "deleted." Anyone who truly cares can fork their own copy, and turn it into a module for others to use too.

How much simpler would the toolchain be without GOPATH support?

@ChrisHines
Copy link
Contributor

An anecdotal data point: My project at work still depends on GOPATH mode to build. Dependencies are managed with a 2015 era tool that automates populating the GOPATH with the desired versions of dependencies, but provides little help deciding what the versions should be. We would have upgraded to Go modules long ago had it been easy, but due to the size of the project, the accumulated complexity of its dependency tree, and other reasons it hasn't happened yet. Not for a lack of desire, by the way. The project policy for the toolchain version is to stay one major version behind, so it's currently built with Go 1.19 and would upgrade to 1.20 shortly after 1.21 is released.

Fortunately we have been able to remove some of the barriers to adopting modules over the last several months and I believe we will be upgraded sometime in the next several months. But there may be other projects similar to ours that are not publicly visible, that still need GOPATH, are still actively worked on, and regularly upgrade the toolchain, but are farther away from using modules. Continuing to support GOPATH mode helps projects like that.

@bcmills
Copy link
Contributor

bcmills commented Jun 27, 2023

How much simpler would the toolchain be without GOPATH support?

The main quirks remaining to support GOPATH mode have to do with how we resolve dependencies in vendor directories, although even that has some overlap with how we support the vendored dependencies of the standard library.

Beyond that, as Russ mentions, the go get implementation for GOPATH mode is complex and fairly subtle.

Speaking as one of the primary maintainers of cmd/go, I feel that the cost/benefit tradeoff supports Russ's proposal. The go get support is subtle and not worth the cost to maintain, but maintaining go build support for the time being is fairly inexpensive and may be valuable for the long tail of migrations (especially for enterprise users).

@willfaught
Copy link
Contributor

@ChrisHines Thanks for your example. I'm curious, how many man hours, working full-time 8 hour days without distraction, do you think it would take to convert that project to an equivalent module? I realize that it's probably non-trivial to do, but I suspect that a lot of situations like that (not necessarily yours, I'm speaking generally) are more a result of management deciding to spend their money on shiny new features instead of maintaining their code, especially since, until now, others have been willing to pay the price to keep them afloat in their current condition. I'm asking so we can have a better sense of the trade-off costs here.

@rsc
Copy link
Contributor Author

rsc commented Jun 27, 2023

Being able to run old (self-contained) code that predates modules is also worthwhile. That requires having GOPATH build support. Keeping build working is easy and it has real benefits.

@ChrisHines
Copy link
Contributor

@willfaught It would not be a single module. The project has dozens of repositories that will each become a module. About a dozen of those contain packages depended on a few levels deep by the rest of them. We've already invested hundreds of man hours to get things into a state that will make the transition easier (while being careful not to break things or interrupt regular development). At this point the source code is in a good state for a migration. I've already written a tool that can create an equivalent go.mod from the file format our existing tool uses and we have a map of our internal dependencies.

The biggest hurdle for us now is that the tool we've been using doesn't enforce repeatable builds. It allows depending on a branch head. We only use that for internal dependencies, but that implies a certain workflow that developers are used to and that our CI systems have been adapted to. Next steps are working out how to adapt our CI systems and developer workflows to the fact modules don't allow depending on branch heads. Trying things out in a small corner of the project where problems will have minimal impact, then educating people. Prior to the addition of workspaces (go.work) in Go 1.18 the workflow problem looked much more daunting, so I'm glad we have that tool available.

The issue has not been management being focused on shiny new features. The project was part way through a migration three years ago, but that was aborted after a key staff member left. I joined the project two years ago and was told almost immediately by management that adopting Go modules was a goal. It's been on the roadmap for a while. But it's a big and old code base with skeletons in many closets that we've been carefully working through.

It's really not a matter of 8 hour focus days as much as managing the smooth evolution of an entrenched social-technical system into a better state.

@zhsj
Copy link
Contributor

zhsj commented Jun 28, 2023

Speaking as Debian packager, I really appreciate that upstream can keep supporting GOPATH for go build. I'm sorry to say that Debian still relies on it. We don't use go get so it's fine to drop GOPATH for it.

@ChrisHines
Copy link
Contributor

It may not be clear from my previous messages that in my case we also don't use go get in GOPATH mode. It is the go build behavior in GOPATH mode that we still rely on.

@rsc
Copy link
Contributor Author

rsc commented Jun 28, 2023

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

@rsc
Copy link
Contributor Author

rsc commented Jul 5, 2023

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

@rsc rsc changed the title proposal: cmd/go: preserve basic GOPATH mode indefinitely cmd/go: preserve basic GOPATH mode indefinitely Jul 5, 2023
@rsc rsc modified the milestones: Proposal, Backlog Jul 5, 2023
@dominikh
Copy link
Member

dominikh commented Jul 5, 2023

If I am not mistaken, this will require changes to go/packages, for two reasons:

  • to pass the correct Go version (1.21) to go/types
  • to inform the user about the Go 1.21 restriction in case the user doesn't rely on go/packages to do the type checking

If we expose the standard library's "std" and "cmd" modules (as per #61174 (comment)), would go list -json include this module information even when GO111MODULE=off? If that were the case, we (both go/packages and the user) could default to Go 1.21 whenever module version information is absent. Otherwise, we'd have to inspect the environment to look for GO111MODULE=off, and either add new API to go/packages to report it to the user, or expect the user to check the environment themselves.

@dominikh
Copy link
Member

dominikh commented Jul 5, 2023

Assume a language setting of Go 1.21 in GOPATH mode

Will //go:build be able to upgrade to a higher version? That is, would code built in GOPATH-mode be able to opt into the new loopvar semantics?

@rsc
Copy link
Contributor Author

rsc commented Jul 24, 2023

@dominikh Yes, that should work.

@gopherbot
Copy link

Change https://go.dev/cl/518775 mentions this issue: cmd/go: delete GOPATH-mode get

@dmitshur dmitshur removed this from the Backlog milestone Aug 16, 2023
@dmitshur dmitshur added this to the Go1.22 milestone Aug 16, 2023
@ChrisHines
Copy link
Contributor

Linking some related proposals here for posterity's sake:

@gopherbot
Copy link

Change https://go.dev/cl/548875 mentions this issue: doc: document removal of 'go get' support in GOPATH mode

gopherbot pushed a commit that referenced this issue Dec 19, 2023
For #61422.
Updates #60915.

Change-Id: Ia8ca12c163a02223b26c5e4cd4c1b6093978aba4
Reviewed-on: https://go-review.googlesource.com/c/go/+/548875
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
ezz-no pushed a commit to ezz-no/go-ezzno that referenced this issue Feb 18, 2024
For golang#61422.
Updates golang#60915.

Change-Id: Ia8ca12c163a02223b26c5e4cd4c1b6093978aba4
Reviewed-on: https://go-review.googlesource.com/c/go/+/548875
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Accepted
Development

No branches or pull requests

8 participants