New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cmd/cgo: add -ldflags to augment CGO_LDFLAGS #66456
Comments
What is the specific change? Cgo already understands response files for command line arguments; the problem here is that the info is being passed to cgo in $CGO_LDFLAGS, and that's exacerbated by the go command and Bazel rules_go both not deduplicating the CGO_LDFLAGS info they gather from every cgo-using package in the build. (If 10 packages all say the same CGO_LDFLAGS, no need to say that part 10 times, just say it once.) Perhaps they should both deduplicate and then the length limit is not a problem anymore. We may need to deduplicate some linker arguments to appease the Mac linker anyway. |
Our I had tried de-duplication, but there were some ordering dependencies in the linker command. |
How did you use a response file at all? Response files are for command-line arguments, not environment variables. |
If we need to do something, we should add a -ldflags argument to the cgo command, and then that can be put in a response file. Interpreting @ signs in environment variables is a big scary door to open. |
The linked bazelbuild/rules_go#2654 has an actual CGO_LDFLAGS line in the top comment. It is one line but it is very very long and GitHub has added a scroll bar, so you don't notice how long it is. That CGO_LDFLAGS has:
So deduplicating is not going to cut it. My suggestion is we add -ldflags to cgo, and then the go command and Bazel can use the support cgo already has for response files on its command line. |
In short, Support for this in
The fixed environment-variable length limit on Linux/Windows makes it currently impossible to compile larger projects. Thinking about whether |
It would help because cgo accepts response files on the command line already. |
Sorry for taking so long to understand - if adding |
This proposal has been added to the active column of the proposals project |
Thank you. |
Based on the discussion above, this proposal seems like a likely accept. The proposal is to add a -ldflags argument to cmd/cgo and then use that argument inside the go command. When the command line gets too long, the go command will automatically switch to response files, avoiding the possibility of $CGO_LDFLAGS getting too long. Bazel could then do the same (adopt -ldflags and use the existing response file functionality for command line arguments) to avoid the same problem. |
No change in consensus, so accepted. 🎉 The proposal is to add a -ldflags argument to cmd/cgo and then use that argument inside the go command. When the command line gets too long, the go command will automatically switch to response files, avoiding the possibility of $CGO_LDFLAGS getting too long. Bazel could then do the same (adopt -ldflags and use the existing response file functionality for command line arguments) to avoid the same problem. |
Proposal Details
cgo
proposal to avoid "argument list too long" (E2BIG
) issues whenCGO_LDFLAGS
outgrows system limits.Problem Description
The
cgo
command writes the contents of theCGO_LDFLAGS
environment variable into the archive.C/C++ linker flags and dependencies are passed via this environment variable.
When there is a large list of C/C++ dependencies, it is easy to hit the system size limits, causing compilation to fail with
cgo: argument list too long
.MAX_ARG_STRLEN
(128kiB).Related issues
Overly large lists dependencies can be written into a response file, using the
@file
syntax, passingCGO_LDFLAGS=@<path-to-file>
instead of failing the compilation.This is a convention supported by many compilers, such as
gcc
orclang
, and mirrors thepassLongArgsInResponseFiles
fix for long argument lists in #18468.Multiple
go
commands also support response file arguments viaobjabi.Flagparse
.Proposal: Since build systems such as
bazel
may execute compile and link stages in different locations,cmd/cgo
should@response_file
,expandArgs
function can be used to do this.The text was updated successfully, but these errors were encountered: