Go Wiki: PerformanceMonitoring

The Go project monitors the performance characteristics of the Go implementation as well as that of subrepositories like golang.org/x/tools.

Benchmarks

golang.org/x/benchmarks/cmd/bench is the entrypoint for our performance tests. For Go implementations, this runs both the Sweet (end-to-end benchmarks) and bent (microbenchmarks) benchmarking suites.

For the golang.org/x/tools project, it runs the repository’s benchmarks.

These benchmarks can all be invoked manually, as can cmd/bench, but using both Sweet and bent directly will likely offer a better user experience. See their documentation for more details.

Performance testing principles

Change with the times

Our set of benchmarks is curated. It is allowed to change over time. Sticking to a single benchmark set over a long period of time can easily land us in a situation where we’re optimizing for the wrong thing.

Always perform a comparison

We never report performance numbers in isolation, and only relative to some baseline. This strategy comes from the fact that comparing performance data taken far apart in time, even on the same hardware, can result in a lot of noise that goes unaccounted for. The state of a machine or VM on one day is likely to be very different than the state of a machine or VM on the next day.

We refer to the tested version of source code as the “experiment” and the baseline version of source code as the “baseline.”

Presubmit

Do you have a Gerrit change that you want to run against our benchmarks?

Select a builder containing the word perf in the “Choose Tryjobs” dialog that appears when selecting a SlowBot.

There are two kinds of presubmit builders for performance testing:

• perf_vs_parent, which measures the performance delta of a change in isolation.
• perf_vs_tip, which measures the performance delta versus the current tip-of-tree for whichever repository the change is for. (Remember to rebase your change(s) before using this one!)

There’s a third special presubmit builder for the tools repository as well which contains the string perf_vs_gopls_0_11. This measures the performance delta versus the release-branch-gopls.0.11 branch of the tools repository.

Postsubmit

The performance dashboard provides continuous monitoring of benchmark performance for every commit that is made to the main Go repository and other subrepositories. The dashboard, more specifically, displays graphs showing the change in certain performance metrics (also called “units”) over time for different benchmarks. Use the navigation interface at the top of the page to explore further.

The regressions page displays all benchmarks in order of biggest regression to biggest improvement, followed by all benchmarks for which there is no statistically clear answer.

On the graphs, red means regression, blue means improvement.

Baselines

In post-submit, the baseline version for Go repository performance tests is automatically determined. For performance tests against changes on release branches, the baseline is always the latest release for that branch (for example, the latest minor release for Go 1.21 on release-branch.go1.21). For performance tests against tip-of-tree, the baseline is always the latest overall release of Go. This is indicated by the name of the builder that produces these benchmark esults, which contains the string perf_vs_release. What this means is that on every minor release of Go, the baseline shifts. These baseline shifts can be observed in the per-metric view.

Performance tests on subrepositories typically operate against some known long-term fixed baseline. For the tools repository, it’s the tip of the release-branch-gopls.0.11 branch.

Per-metric view

Click on any graph’s performance metric name to view a more detailed timeline of the performance deltas for that metric.

This view is particularly useful for identifying the culprit behind a regression and for pinpointing the source of an improvement.

Sometimes, a performance change happens because a benchmark has changed or because the baseline version being used as changed. This view also displays information about the baseline versions and the version of golang.org/x/benchmarks that was used to produce the results to help identify when this happens.

This content is part of the Go Wiki.