net/http, x/net/http2: rapid stream resets can cause excessive work (CVE-2023-39325) #63417

neild · 2023-10-06T15:52:23Z

A malicious HTTP/2 client which rapidly creates requests and
immediately resets them can cause excessive server resource consumption.
While the total number of requests is bounded to the
http2.Server.MaxConcurrentStreams setting, resetting an in-progress
request allows the attacker to create a new request while the existing
one is still executing.

HTTP/2 servers now bound the number of simultaneously executing
handler goroutines to the stream concurrency limit. New requests
arriving when at the limit (which can only happen after the client
has reset an existing, in-flight request) will be queued until a
handler exits. If the request queue grows too large, the server
will terminate the connection.

This issue is also fixed in golang.org/x/net/http2 v0.17.0,
for users manually configuring HTTP/2.

The default stream concurrency limit is 250 streams (requests)
per HTTP/2 connection. This value may be adjusted using the
golang.org/x/net/http2 package; see the Server.MaxConcurrentStreams
setting and the ConfigureServer function.

This is CVE-2023-39325 and Go issue https://go.dev/issue/63417.
This is also tracked by CVE-2023-44487.

This is a PRIVATE issue for CVE-2023-39325, tracked in http://b/303836512.

/cc @golang/security and @golang/release

neild · 2023-10-06T20:37:47Z

@gopherbot please open backport issues

gopherbot · 2023-10-06T20:38:49Z

Backport issue(s) opened: #63426 (for 1.20), #63427 (for 1.21).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

gopherbot · 2023-10-10T15:42:20Z

Change https://go.dev/cl/534215 mentions this issue: http2: limit maximum handler goroutines to MaxConcurrentStreams

gopherbot · 2023-10-10T15:43:12Z

Change https://go.dev/cl/534235 mentions this issue: [release-branch.go1.21] net/http: regenerate h2_bundle.go

gopherbot · 2023-10-10T15:43:15Z

Change https://go.dev/cl/534255 mentions this issue: [release-branch.go1.20] net/http: regenerate h2_bundle.go

dmitshur · 2023-10-10T15:46:31Z

Reopening to track bundling into the main tree, for Go 1.22.

Pull in a security fix from x/net/http2: http2: limit maximum handler goroutines to MaxConcurrentStreamso For #63417 Fixes #63427 Fixes CVE-2023-39325 Change-Id: I70626734e6d56edf508f27a5b055ddf96d806eeb Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/2047402 Reviewed-by: Tatiana Bradley <tatianabradley@google.com> Run-TryBot: Damien Neil <dneil@google.com> Reviewed-by: Ian Cottrell <iancottrell@google.com> Reviewed-on: https://go-review.googlesource.com/c/go/+/534235 Auto-Submit: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Michael Pratt <mpratt@google.com> Reviewed-by: Damien Neil <dneil@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

gopherbot · 2023-10-10T16:18:10Z

Change https://go.dev/cl/534295 mentions this issue: all: pull in x/net v0.17.0 and its dependencies

Pull in a security fix from x/net/http2: http2: limit maximum handler goroutines to MaxConcurrentStreamso For #63417 Fixes #63426 Fixes CVE-2023-39325 Change-Id: I6e32397323cd9b4114c990fcc9d19557a7f5f619 Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/2047401 Reviewed-by: Tatiana Bradley <tatianabradley@google.com> TryBot-Result: Security TryBots <security-trybots@go-security-trybots.iam.gserviceaccount.com> Run-TryBot: Damien Neil <dneil@google.com> Reviewed-by: Ian Cottrell <iancottrell@google.com> Reviewed-on: https://go-review.googlesource.com/c/go/+/534255 Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Damien Neil <dneil@google.com> TryBot-Bypass: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Michael Pratt <mpratt@google.com> Auto-Submit: Dmitri Shuralyov <dmitshur@google.com>

gopherbot · 2023-10-10T17:10:30Z

Change https://go.dev/cl/534218 mentions this issue: [internal-branch.go1.21-vendor] http2: limit maximum handler goroutines to MaxConcurrentStreams

gopherbot · 2023-10-10T17:13:40Z

Change https://go.dev/cl/534236 mentions this issue: [internal-branch.go1.20-vendor] http2: limit maximum handler goroutines to MaxConcurrentStreams

…es to MaxConcurrentStreams When the peer opens a new stream while we have MaxConcurrentStreams handler goroutines running, defer starting a handler until one of the existing handlers exits. For golang/go#63417. For golang/go#63427. For CVE-2023-39325. Change-Id: If0531e177b125700f3e24c5ebd24b1023098fa6d Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/2047391 Reviewed-by: Tatiana Bradley <tatianabradley@google.com> Run-TryBot: Damien Neil <dneil@google.com> Reviewed-by: Ian Cottrell <iancottrell@google.com> Reviewed-on: https://go-review.googlesource.com/c/net/+/534218 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Auto-Submit: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Michael Pratt <mpratt@google.com>

…es to MaxConcurrentStreams When the peer opens a new stream while we have MaxConcurrentStreams handler goroutines running, defer starting a handler until one of the existing handlers exits. For golang/go#63417. For golang/go#63426. For CVE-2023-39325. Change-Id: If0531e177b125700f3e24c5ebd24b1023098fa6d Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/2047553 Reviewed-by: Ian Cottrell <iancottrell@google.com> Run-TryBot: Damien Neil <dneil@google.com> Reviewed-by: Tatiana Bradley <tatianabradley@google.com> TryBot-Result: Security TryBots <security-trybots@go-security-trybots.iam.gserviceaccount.com> Reviewed-on: https://go-review.googlesource.com/c/net/+/534236 Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Michael Pratt <mpratt@google.com> Auto-Submit: Dmitri Shuralyov <dmitshur@google.com>

gopherbot · 2023-10-10T17:29:43Z

Change https://go.dev/cl/534297 mentions this issue: [release-branch.go1.20] all: tidy dependency versioning after release

gopherbot · 2023-10-10T17:42:53Z

Change https://go.dev/cl/534415 mentions this issue: [release-branch.go1.21] all: tidy dependency versioning after release

See golang/go#63417 Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>

This change mitigates CVE-2023-44487 by disabling HTTP2 by default and forcing HTTP/1.1 until the Go standard library and golang.org/x/net are fully fixed. Right now, it is possible for authenticated and unauthenticated users to hold open HTTP2 connections and consume huge amounts of memory. It is possible to revert back the change by using the `--web.enable-http2` argument. Before this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted h2 [...] * using HTTP/2 * h2h3 [:method: GET] * h2h3 [:path: /metrics] * h2h3 [:scheme: https] * h2h3 [:authority: localhost:8443] * h2h3 [user-agent: curl/8.0.1] * h2h3 [accept: */*] * Using Stream ID: 1 (easy handle 0x5594d4614b10) [...] > GET /metrics HTTP/2 [...] ``` After this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted http/1.1 [...] * using HTTP/1.1 > GET /metrics HTTP/1.1 > Host: localhost:8443 > User-Agent: curl/8.0.1 > Accept: */* [...] < HTTP/1.1 200 OK [...] ``` See also: * kubernetes/kubernetes#121120 * kubernetes/kubernetes#121197 * golang/go#63417 (comment) Signed-off-by: Simon Pasquier <spasquie@redhat.com>

Pull in a security fix from x/net/http2: http2: limit maximum handler goroutines to MaxConcurrentStreams Fixes golang#63417. Fixes CVE-2023-39325. Change-Id: I01e7774912e81007a7cf70f33e5989fb50a0b708 Cq-Include-Trybots: luci.golang.try:gotip-linux-amd64-longtest Reviewed-on: https://go-review.googlesource.com/c/go/+/534295 Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Michael Pratt <mpratt@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

This change mitigates CVE-2023-44487 by disabling HTTP2 by default and forcing HTTP/1.1 until the Go standard library and golang.org/x/net are fully fixed. Right now, it is possible for authenticated and unauthenticated users to hold open HTTP2 connections and consume huge amounts of memory. It is possible to revert back the change by using the `--web.enable-http2` argument. Before this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted h2 [...] * using HTTP/2 * h2h3 [:method: GET] * h2h3 [:path: /metrics] * h2h3 [:scheme: https] * h2h3 [:authority: localhost:8443] * h2h3 [user-agent: curl/8.0.1] * h2h3 [accept: */*] * Using Stream ID: 1 (easy handle 0x5594d4614b10) [...] > GET /metrics HTTP/2 [...] ``` After this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted http/1.1 [...] * using HTTP/1.1 > GET /metrics HTTP/1.1 > Host: localhost:8443 > User-Agent: curl/8.0.1 > Accept: */* [...] < HTTP/1.1 200 OK [...] ``` See also: * kubernetes/kubernetes#121120 * kubernetes/kubernetes#121197 * golang/go#63417 (comment) Signed-off-by: Simon Pasquier <spasquie@redhat.com> (cherry picked from commit a62e814)

ATP3530 · 2023-11-06T10:44:58Z

Does this CVE-2023-39325 impacts Go version 1.18? If yes is there a plan to backport required fix into version 1.18? Or we have to move to the latest version of go (1.20 or 1.21) to have this CVE fixed

neild · 2023-11-06T17:41:11Z

1.18 is affected. There are no plans to backport to 1.18, which is out of the support window; see: https://go.dev/doc/devel/release#policy

As an alternative to upgrading to a recent Go version, you can use a recent version of the golang.org/x/net/http2 package and http2.ConfigureServer to replace the standard library's bundled HTTP/2 implementation, although I'd really recommend upgrading to a Go version that's still getting security fixes.

This change mitigates CVE-2023-44487 by disabling HTTP2 by default and forcing HTTP/1.1 until the Go standard library and golang.org/x/net are fully fixed. Right now, it is possible for authenticated and unauthenticated users to hold open HTTP2 connections and consume huge amounts of memory. It is possible to revert back the change by using the `--web.enable-http2` argument. Before this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted h2 [...] * using HTTP/2 * h2h3 [:method: GET] * h2h3 [:path: /metrics] * h2h3 [:scheme: https] * h2h3 [:authority: localhost:8443] * h2h3 [user-agent: curl/8.0.1] * h2h3 [accept: */*] * Using Stream ID: 1 (easy handle 0x5594d4614b10) [...] > GET /metrics HTTP/2 [...] ``` After this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted http/1.1 [...] * using HTTP/1.1 > GET /metrics HTTP/1.1 > Host: localhost:8443 > User-Agent: curl/8.0.1 > Accept: */* [...] < HTTP/1.1 200 OK [...] ``` See also: * kubernetes/kubernetes#121120 * kubernetes/kubernetes#121197 * golang/go#63417 (comment) Signed-off-by: Simon Pasquier <spasquie@redhat.com>

# AWS EKS Backported To: go-1.19.13-eks Backported On: Thu, 12 Oct 2023 Backported By: rcrozean@amazon.com Backported From: release-branch.go1.20 Source Commit: golang@e175f27 # Original Information Pull in a security fix from x/net/http2: http2: limit maximum handler goroutines to MaxConcurrentStreamso For golang#63417 Fixes golang#63426 Fixes CVE-2023-39325 Change-Id: I6e32397323cd9b4114c990fcc9d19557a7f5f619 Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/2047401 Reviewed-by: Tatiana Bradley <tatianabradley@google.com> TryBot-Result: Security TryBots <security-trybots@go-security-trybots.iam.gserviceaccount.com> Run-TryBot: Damien Neil <dneil@google.com> Reviewed-by: Ian Cottrell <iancottrell@google.com> Reviewed-on: https://go-review.googlesource.com/c/go/+/534255 Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Damien Neil <dneil@google.com> TryBot-Bypass: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Michael Pratt <mpratt@google.com> Auto-Submit: Dmitri Shuralyov <dmitshur@google.com>

aojea · 2023-12-26T11:15:13Z

FYI, this fix is not sufficient for Kube, and possibly other projects. We are working on this in kubernetes/kubernetes#121120. pprof svg from a single client with a single connection attempting to DOS a Kube API server that has this fix and max streams set to 100 (memory usage grew to 5 GB in a few mins before it stabilized - with multiple connections the API server would have easily OOM'd):

@enj from the pprof attached the bulk of the memory consumed comes from the 4kb buffer allocated to the responseWriter that is created after the Headers frame is received. The buffer is also obtained from the sync.Pool that will make the memory usage more efficient.

https://github.com/golang/net/blob/f12db26b1c9293fa3eb95c936e548d2c1fba4ba9/http2/server.go#L69-L75

In this case, since that the attack is based only on headers frames, postponing the allocation of the responseWriter will reduce a lot the impact on memory usage, however, I've been checking around the code and I don't know how feasible is to do that, based on the rfc7540

Sending or receiving a HEADERS frame causes the stream to become "open"

A stream in the "open" state may be used by both peers to send frames of any type. In this state, sending peers observe
advertised stream-level flow-control limits (Section 5.2)

Pull in a security fix from x/net/http2: http2: limit maximum handler goroutines to MaxConcurrentStreamso For golang#63417 Fixes golang#63427 Fixes CVE-2023-39325 Change-Id: I70626734e6d56edf508f27a5b055ddf96d806eeb Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/2047402 Reviewed-by: Tatiana Bradley <tatianabradley@google.com> Run-TryBot: Damien Neil <dneil@google.com> Reviewed-by: Ian Cottrell <iancottrell@google.com> Reviewed-on: https://go-review.googlesource.com/c/go/+/534235 Auto-Submit: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Michael Pratt <mpratt@google.com> Reviewed-by: Damien Neil <dneil@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

Done with: go get golang.org/x/net@internal-branch.go1.21-vendor go mod tidy go mod vendor go generate net/http # zero diff since CL 534235 already did this For golang#63417. For golang#63427. For CVE-2023-39325. Change-Id: Ib258e0d8165760a1082e02c2f4c5ce7d2a3c3c90 Reviewed-on: https://go-review.googlesource.com/c/go/+/534415 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Auto-Submit: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Michael Pratt <mpratt@google.com>

Pull in a security fix from x/net/http2: http2: limit maximum handler goroutines to MaxConcurrentStreamso For golang#63417 Fixes golang#63427 Fixes CVE-2023-39325 Change-Id: I70626734e6d56edf508f27a5b055ddf96d806eeb Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/2047402 Reviewed-by: Tatiana Bradley <tatianabradley@google.com> Run-TryBot: Damien Neil <dneil@google.com> Reviewed-by: Ian Cottrell <iancottrell@google.com> Reviewed-on: https://go-review.googlesource.com/c/go/+/534235 Auto-Submit: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Michael Pratt <mpratt@google.com> Reviewed-by: Damien Neil <dneil@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

Done with: go get golang.org/x/net@internal-branch.go1.21-vendor go mod tidy go mod vendor go generate net/http # zero diff since CL 534235 already did this For golang#63417. For golang#63427. For CVE-2023-39325. Change-Id: Ib258e0d8165760a1082e02c2f4c5ce7d2a3c3c90 Reviewed-on: https://go-review.googlesource.com/c/go/+/534415 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Auto-Submit: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Michael Pratt <mpratt@google.com>

neild self-assigned this Oct 6, 2023

seankhliao added Security NeedsFix The path to resolution is known, but the work has not been done. release-blocker labels Oct 6, 2023

seankhliao added this to the Go1.22 milestone Oct 6, 2023

gopherbot mentioned this issue Oct 6, 2023

security: fix CVE-2023-39325 [1.20 backport] #63426

Closed

gopherbot mentioned this issue Oct 6, 2023

security: fix CVE-2023-39325 [1.21 backport] #63427

Closed

gopherbot closed this as completed in golang/net@b225e7c Oct 10, 2023

dmitshur reopened this Oct 10, 2023

gopherbot closed this as completed in f34964a Oct 10, 2023

dmitshur changed the title ~~security: fix CVE-2023-39325~~ net/http, x/net/http2: rapid stream resets can cause excessive work (CVE-2023-39325) Oct 10, 2023

This was referenced Oct 10, 2023

go@1.20 1.20.10 Homebrew/homebrew-core#150459

Merged

go 1.21.3 Homebrew/homebrew-core#150460

Merged

piroux mentioned this issue Oct 10, 2023

HTTP/2 Rapid Reset : CVE-2023-44487 caddyserver/caddy#5877

Closed

smira added a commit to smira/tools that referenced this issue Oct 10, 2023

feat: update Go to 1.21.3

35948af

See golang/go#63417 Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>

simonpasquier mentioned this issue Oct 19, 2023

fix: disable HTTP2 connections by default prometheus-operator/prometheus-operator#6028

Merged

5 tasks

johannes94 mentioned this issue Oct 20, 2023

fix: force HTTP1.1 for operator webhooks to remidiate CVE-2023-44487 stackrox/stackrox#8279

Merged

1 task

ingvagabund mentioned this issue Oct 22, 2023

Bump dependencies to address CVE-2023-44487 kubernetes-sigs/descheduler#1264

Merged

martin-neotech mentioned this issue Oct 25, 2023

[TT-10384] Is Tyk (gateway) affected by CVE-2023-44487 ? TykTechnologies/tyk#5672

Closed

realshuting mentioned this issue Oct 31, 2023

fix: backport CVE fixes kyverno/kyverno#8798

Merged

slashpai mentioned this issue Nov 3, 2023

OCPBUGS-21637: fix: disable HTTP2 connections by default openshift/prometheus-operator#252

Merged

5 tasks

gsoltis mentioned this issue Nov 3, 2023

fix(Turborepo): Bump x/net, go mod tidy vercel/turbo#6355

Merged

yeya24 mentioned this issue Nov 5, 2023

Update go version to 1.21.3 cortexproject/cortex#5630

Merged

3 tasks

lucianomda mentioned this issue Nov 6, 2023

Bump golang.org/x/net from 0.8.0 to 0.17.0 go-playground/validator#1179

Closed

MichaelSnowden mentioned this issue Nov 18, 2023

Patch x/net CVE fix temporalio/ringpop-go#13

Merged

racheljpg mentioned this issue Dec 15, 2023

Bump golang.org/x/net to v0.19.0 kubernetes/cloud-provider-alibaba-cloud#389

Open

iwata mentioned this issue Feb 8, 2024

Update vulnerable package versions cloudspannerecosystem/wrench#103

Merged

jmlMetaswitch mentioned this issue Mar 8, 2024

Bump golang.org/x/net from 0.7.0 to 0.17.0 antchfx/xmlquery#110

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

net/http, x/net/http2: rapid stream resets can cause excessive work (CVE-2023-39325) #63417

net/http, x/net/http2: rapid stream resets can cause excessive work (CVE-2023-39325) #63417

neild commented Oct 6, 2023 •

edited by dmitshur

neild commented Oct 6, 2023

gopherbot commented Oct 6, 2023

gopherbot commented Oct 10, 2023

gopherbot commented Oct 10, 2023

gopherbot commented Oct 10, 2023

dmitshur commented Oct 10, 2023

gopherbot commented Oct 10, 2023

gopherbot commented Oct 10, 2023

gopherbot commented Oct 10, 2023

gopherbot commented Oct 10, 2023

gopherbot commented Oct 10, 2023

ATP3530 commented Nov 6, 2023 •

edited

neild commented Nov 6, 2023

aojea commented Dec 26, 2023

net/http, x/net/http2: rapid stream resets can cause excessive work (CVE-2023-39325) #63417

net/http, x/net/http2: rapid stream resets can cause excessive work (CVE-2023-39325) #63417

Comments

neild commented Oct 6, 2023 • edited by dmitshur

neild commented Oct 6, 2023

gopherbot commented Oct 6, 2023

gopherbot commented Oct 10, 2023

gopherbot commented Oct 10, 2023

gopherbot commented Oct 10, 2023

dmitshur commented Oct 10, 2023

gopherbot commented Oct 10, 2023

gopherbot commented Oct 10, 2023

gopherbot commented Oct 10, 2023

gopherbot commented Oct 10, 2023

gopherbot commented Oct 10, 2023

ATP3530 commented Nov 6, 2023 • edited

neild commented Nov 6, 2023

aojea commented Dec 26, 2023

neild commented Oct 6, 2023 •

edited by dmitshur

ATP3530 commented Nov 6, 2023 •

edited