Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: use MADV_FREE on linux as well #23687

Closed
CAFxX opened this issue Feb 4, 2018 · 5 comments
Closed

runtime: use MADV_FREE on linux as well #23687

CAFxX opened this issue Feb 4, 2018 · 5 comments
Labels
FrozenDueToAge help wanted NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Performance
Milestone

Comments

@CAFxX
Copy link
Contributor

CAFxX commented Feb 4, 2018

On linux currently sysUnused uses madvise(MADV_DONTNEED) to signal the kernel that a range of allocated memory contains unneeded data. After a successful call, the range (but not the data it contained before the call to madvise) is still available but the first access to that range will unconditionally incur a page fault (needed to 0-fill the range).

A potentially faster alternative is MADV_FREE, available since linux 4.5 (and already used for bsd). The mechanism is very similar, but the page fault will only be incurred if the kernel, between the call to madvise and the first access, decides to reuse that memory for something else.

Off the top of my head, there are two ways to support both pre-4.5 and 4.5+ kernels:

  • Use a fallback approach (i.e. try MADV_FREE first and MADV_DONTNEED if MADV_FREE fails): this is e.g. what node does
  • Detect at runtime startup whether the running kernel supports MADV_FREE (e.g. by attempting to call madvise(MADV_FREE) on an allocated but unused range)
@bradfitz
Copy link
Contributor

bradfitz commented Feb 4, 2018

/cc @aclements

@bradfitz bradfitz added this to the Go1.11 milestone Feb 4, 2018
@bradfitz bradfitz added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Feb 4, 2018
@ianlancetaylor ianlancetaylor added help wanted NeedsFix The path to resolution is known, but the work has not been done. labels Jul 10, 2018
@ianlancetaylor ianlancetaylor modified the milestones: Go1.11, Unplanned Jul 10, 2018
@gopherbot gopherbot removed the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jul 10, 2018
@ianlancetaylor ianlancetaylor added Performance NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jul 10, 2018
@gopherbot gopherbot removed the NeedsFix The path to resolution is known, but the work has not been done. label Jul 10, 2018
@ianlancetaylor ianlancetaylor modified the milestones: Unplanned, Go1.12 Jul 10, 2018
@gopherbot
Copy link

Change https://golang.org/cl/135395 mentions this issue: runtime: use MADV_FREE on Linux if available

@alk
Copy link

alk commented May 28, 2019

Hi all. I wondering if there is any data backing up claims about MADV_FREE actually improving anything.

Asking because:

a) I have not seen any data pointing that it does help (I co-maintain gperftools and over there we're defaulting MADV_FREE to off on Linux).

b) there are numerous questions about implementation of this feature in kernel. First, while it is somewhat cheaper then MADV_DONTNEED, it is not that cheap, since page tables have to be walked and modified. It doesn't get reflected in RSS and thus interaction of this and memory reclaim facility in kernel, especially in settings with heavy usage of control groups isn't trivial. And also defaults about how this memory is preferred compared other memory during reclaim is also not entirely clear.

Since this feature was lobbied by jemalloc folks, maybe there are some Facebook folks involved in golang who do have some data?

I think this ticket should be re-opened. We should either gather and publish the data and see that indeed it helps, or revert defaults to off. IMHO.

@thepudds
Copy link
Contributor

@alk I'm curious if you have tried any experiments with GODEBUG=madvdontneed=1? From the Go 1.12 release notes:

To revert to the Go 1.11 behavior (MADV_DONTNEED), set the environment variable GODEBUG=madvdontneed=1

Separately, given the volume of issues, I think some people automatically filter out notifications from closed issues, so you might want to open a new issue with your comments above.

@gopherbot
Copy link

Change https://golang.org/cl/267100 mentions this issue: runtime: default to MADV_DONTNEED on Linux

gopherbot pushed a commit that referenced this issue Nov 2, 2020
In Go 1.12, we changed the runtime to use MADV_FREE when available on
Linux (falling back to MADV_DONTNEED) in CL 135395 to address issue
 #23687. While MADV_FREE is somewhat faster than MADV_DONTNEED, it
doesn't affect many of the statistics that MADV_DONTNEED does until
the memory is actually reclaimed under OS memory pressure. This
generally leads to poor user experience, like confusing stats in top
and other monitoring tools; and bad integration with management
systems that respond to memory usage.

We've seen numerous issues about this user experience, including
 #41818, #39295, #37585, #33376, and #30904, many questions on Go
mailing lists, and requests for mechanisms to change this behavior at
run-time, such as #40870. There are also issues that may be a result
of this, but root-causing it can be difficult, such as #41444 and
 #39174. And there's some evidence it may even be incompatible with
Android's process management in #37569.

This CL changes the default to prefer MADV_DONTNEED over MADV_FREE, to
favor user-friendliness and minimal surprise over performance. I think
it's become clear that Linux's implementation of MADV_FREE ultimately
doesn't meet our needs. We've also made many improvements to the
scavenger since Go 1.12. In particular, it is now far more prompt and
it is self-paced, so it will simply trickle memory back to the system
a little more slowly with this change. This can still be overridden by
setting GODEBUG=madvdontneed=0.

Fixes #42330 (meta-issue).

Fixes #41818, #39295, #37585, #33376, #30904 (many of which were
already closed as "working as intended").

Change-Id: Ib6aa7f2dc8419b32516cc5a5fc402faf576c92e4
Reviewed-on: https://go-review.googlesource.com/c/go/+/267100
Trust: Austin Clements <austin@google.com>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
@golang golang locked and limited conversation to collaborators Nov 2, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge help wanted NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Performance
Projects
None yet
Development

No branches or pull requests

6 participants