Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: BenchmarkClientServerParallel4 hangs and is useless #20166

Open
bradfitz opened this issue Apr 28, 2017 · 6 comments
Open

net/http: BenchmarkClientServerParallel4 hangs and is useless #20166

bradfitz opened this issue Apr 28, 2017 · 6 comments
Labels
help wanted Testing An issue that has been verified to require only test changes, not just a test failure.
Milestone

Comments

@bradfitz
Copy link
Contributor

It seems the BenchmarkClientServerParallel4 benchmark sometimes hangs while running. Sometimes it needs to be killed (first example) and sometimes it recovers on its own, leading to useless numbers.

In any case, I can't get good numbers out of it.

I suspect it's consuming too many sockets and running out of addresses or something.

bradfitz@gdev:~/go/src/net/http$ go test -benchtime=1s -bench=BenchmarkClientServerParallel4 -count=3 -run=XXXX | tee before
goos: linux
goarch: amd64
pkg: net/http
BenchmarkClientServerParallel4-4           30000             39470 ns/op            7539 B/op         73 allocs/op
BenchmarkClientServerParallel4-4           50000             40934 ns/op            7563 B/op         73 allocs/op
BenchmarkClientServerParallel4-4        ^C
bradfitz@gdev:~/go/src/net/http$ go test -benchtime=1s -bench=BenchmarkClientServerParallel4 -count=3 -run=XXXX | tee before
goos: linux
goarch: amd64
pkg: net/http
BenchmarkClientServerParallel4-4            5000            283132 ns/op            7244 B/op         85 allocs/op
BenchmarkClientServerParallel4-4               1        7011289330 ns/op           54720 B/op        527 allocs/op
BenchmarkClientServerParallel4-4               1        7010777787 ns/op           37472 B/op        238 allocs/op
PASS
ok      net/http        16.585s
@bradfitz bradfitz added help wanted Testing An issue that has been verified to require only test changes, not just a test failure. labels Apr 28, 2017
@bradfitz bradfitz added this to the Unplanned milestone Apr 28, 2017
@bradfitz
Copy link
Contributor Author

(At d286399, on linux/amd64)

@kennygrant
Copy link
Contributor

Might be related to #18753 with BenchmarkClientServerParallel64 on mac os - which is the same test just with more parallelism. I've see a similar problems on mac os with BenchmarkClientServerParallel4 and variants. The first 3 runs are fine, second time it is run thrice it dies with the error: can't assign requested address. Waiting a minute and trying again results in the same pattern - fine for first run, fails afterward, as opposed to trying again straight away which fails repeatedly, so it does seem to be exhausting resources by tying them up for too long.

netstat -a after the test shows thousands of connections in state TIME_WAIT for a minute or two, but I guess that's normal?

kenny$ cd src/net/http
kenny$ go test -benchtime=1s -bench=BenchmarkClientServerParallel4 -count=3 -run=XXXX | tee before
goos: darwin
goarch: amd64
pkg: net/http
BenchmarkClientServerParallel4-4   	   30000	     51827 ns/op	    7677 B/op	      75 allocs/op
BenchmarkClientServerParallel4-4   	   10000	    156941 ns/op	    7948 B/op	      76 allocs/op
BenchmarkClientServerParallel4-4   	   10000	    149905 ns/op	    7936 B/op	      76 allocs/op
PASS
ok  	net/http	5.105s
kenny$ go test -benchtime=1s -bench=BenchmarkClientServerParallel4 -count=3 -run=XXXX | tee before
goos: darwin
goarch: amd64
pkg: net/http
BenchmarkClientServerParallel4-4   	   10000	    284811 ns/op	    7809 B/op	      76 allocs/op
--- BENCH: BenchmarkClientServerParallel4-4
	serve_test.go:4756: Get: Get http://127.0.0.1:58703: dial tcp 127.0.0.1:58703: connect: can't assign requested address
	serve_test.go:4756: Get: Get http://127.0.0.1:58703: dial tcp 127.0.0.1:58703: connect: can't assign requested address
	serve_test.go:4756: Get: Get http://127.0.0.1:58703: dial tcp 127.0.0.1:58703: connect: can't assign requested address

@gopherbot
Copy link

CL https://golang.org/cl/49031 mentions this issue.

@xiam
Copy link

xiam commented Jul 16, 2017

I think this problem is caused by how are we using ts.Client().

httptest.Server has a Client function that returns the same client every time it's called.

When we use ts.Client() inside a parallel test without calling CloseIdleConnections() on the transport, a lot of idle connections start to pile up and the OS may decide not to grant any one else more.

httpserver.Server does close the transport's idle connections, but only does so when the server itself is closed.

I submitted a solution that I think fits with the httptest.Server's design, but this could also be fixed by using something like:

tr := &Transport{}
defer tr.CloseIdleConnections()
c := &Client{Transport: tr}

instead of c = ts.Client()

@odeke-em
Copy link
Member

Thank you for the investigation @xiam and for sending the CL. So for an updated, since Go1.12, we now have Client.CloseIdleConnections() https://golang.org/pkg/net/http/#Client.CloseIdleConnections -- committed on August 20th 2018 and in CL https://go-review.googlesource.com/130115.

Following up with your suggestion, perhaps in that defer, we could invoke defer client.CloseIdleConnections()?

If so, and if you have the time, a CL from you following up your previous one https://go-review.googlesource.com/c/go/+/49031/, would be great for Go1.15. @bradfitz would also be happy to see more folks involved with net/http.

Thank you!

@stippi2
Copy link

stippi2 commented May 27, 2021

To disable the automatic connection cache (if that's what you need) you can configure the Transport:

tr := &Transport{DisableKeepAlives: true}
c := &Client{Transport: tr}

Depending on the timing, closing idle connections may not work as reliably or have no effect at all (because a connection may not be considered "idle", yet). You can also configure a maximum number of idle connections on the Transport.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Testing An issue that has been verified to require only test changes, not just a test failure.
Projects
None yet
Development

No branches or pull requests

6 participants