x/build: frequent "communication error to buildlet" failures on plan9-amd64-0intro #49756

bcmills · 2021-11-23T16:26:22Z

#!watchflakes
post <- builder == "plan9-amd64-0intro" && `communication error to buildlet`

greplogs --dashboard -md -l -e 'communication error to buildlet' --since=2021-11-01

CC @millerresearch @0intro

@golang/release, what provokes this error message in the coordinator? (Is this a failing keepalive of some sort?)

The text was updated successfully, but these errors were encountered:

dmitshur · 2021-11-23T16:43:41Z

This looks relevant to:

// If a build fails multiple times due to communication
// problems with the buildlet, assume something's wrong with
// the buildlet or machine and fail the build, rather than
// looping forever. This promotes the err (communication
// error) to a remoteErr (an error that occurred remotely and
// is terminal).
if rerr := st.repeatedCommunicationError(err); rerr != nil {
	remoteErr = rerr
	err = nil
	doneMsg = "communication error to buildlet (promoted to terminal error): " + rerr.Error()
	fmt.Fprintf(st, "\n%s\n", doneMsg)
}

The current implementation of repeatedCommunicationError is implemented only for Plan 9, and doesn't allow for any retries yet:

// For now, only do this for plan9, which is flaky (Issue 31261)
if strings.HasPrefix(st.Name, "plan9-") && execErr == errBuildletsGone {
	// TODO: give it two tries at least later (store state
	// somewhere; global map?). But for now we're going to
	// only give it one try.
	return fmt.Errorf("network error promoted to terminal error: %v", execErr)
}

So this is related to issues #31261 and #13026.

millerresearch · 2021-11-23T20:10:53Z

When I observe this on the plan9-arm builders, it's because the builder machine has crashed or rebooted for some reason unrelated to the go tests (eg power flicker, file server disk full). I often use the retrybuilds utility to clear and restart the failed build, and it's fine.

bcmills · 2021-12-09T17:29:04Z

The rate of these for the plan9-arm builder seems to have gone down over the past couple of weeks, but the plan9-amd64-0intro builder appears to be particularly unreliable.

@0intro, could you please investigate?

greplogs --dashboard -md -l -e 'communication error to buildlet' --since=2021-11-24

0intro · 2021-12-09T18:39:02Z

I've recently reworked the plan9-386 and plan9-amd64 builders so they reboot automatically after running the tests.
This should prevent the builders from remaining stuck in case of failure.

I'll investigate the issues with the plan9-amd64 builder.

bcmills · 2022-01-05T19:58:07Z

@0intro, any progress on the plan9-amd64 builder? It is currently showing completely broken on the build dashboard.

greplogs --dashboard -md -l -e 'communication error to buildlet' --since=2021-12-10

bcmills · 2022-01-27T16:36:50Z

Still ongoing — essentially every run for plan9-amd64-0intro for the main Go repo.

greplogs --dashboard -md -l -e 'communication error to buildlet' --since=2022-01-06

0intro · 2022-01-28T13:19:06Z

The plan9-amd64 builder was running out of memory while executing GOMAXPROCS=2 go test runtime -cpu=1,2,4 -quick.
I think it should be better now.

@bcmills Thank you for your time tracking the various Plan 9 issues. It's very appreciated.

gopherbot · 2022-05-26T15:47:32Z

Change https://go.dev/cl/408698 mentions this issue: dashboard: add known issues for plan9-386, plan9-386-0intro, and plan9-amd64-0intro

…9-amd64-0intro For golang/go#52653. Updates golang/go#29801. Updates golang/go#49756. Updates golang/go#50137. Change-Id: Ided04da5c1e403c4a194c8c645c5179534a00907 Reviewed-on: https://go-review.googlesource.com/c/build/+/408698 TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Bryan Mills <bcmills@google.com> Reviewed-by: Alex Rakoczy <alex@golang.org> Auto-Submit: Bryan Mills <bcmills@google.com>

bcmills · 2022-06-28T15:16:51Z

Still ongoing as of last month, but possibly improved for this builder in June?

greplogs --dashboard -md -l -e '(?ms)\Aplan9-amd64.*communication error to buildlet' --since=2022-05-01
2022-05-24T17:02:34-70668a4/plan9-amd64-0intro
2022-05-19T21:32:46-dd83fd8/plan9-amd64-0intro
2022-05-17T20:05:58-13147f7/plan9-amd64-0intro
2022-05-08T17:28:26-1b86ef4/plan9-amd64-0intro

gopherbot · 2023-04-16T13:22:47Z

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

bcmills added Builders x/build issues (builders, bots, dashboards) OS-Plan9 labels Nov 23, 2021

bcmills added this to the Unreleased milestone Nov 23, 2021

heschi added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 24, 2021

bcmills changed the title ~~x/build: frequent "communication error to buildlet" failures on all plan9 builders~~ x/build: frequent "communication error to buildlet" failures on plan9-amd64-0intro Jan 27, 2022

bcmills mentioned this issue May 3, 2022

x/build: frequent "communication error to buildlet" failures on plan9-arm #52677

Closed

4a6f656c mentioned this issue Aug 8, 2022

clarify Go support policy for secondary ports #53383

Closed

bcmills added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Mar 16, 2023

gopherbot closed this as completed Apr 16, 2023

golang locked and limited conversation to collaborators Apr 15, 2024

gopherbot added the FrozenDueToAge label Apr 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/build: frequent "communication error to buildlet" failures on plan9-amd64-0intro #49756

x/build: frequent "communication error to buildlet" failures on plan9-amd64-0intro #49756

bcmills commented Nov 23, 2021 •

edited

dmitshur commented Nov 23, 2021

millerresearch commented Nov 23, 2021

bcmills commented Dec 9, 2021

0intro commented Dec 9, 2021

bcmills commented Jan 5, 2022

bcmills commented Jan 27, 2022

0intro commented Jan 28, 2022

gopherbot commented May 26, 2022

bcmills commented Jun 28, 2022

gopherbot commented Apr 16, 2023

x/build: frequent "communication error to buildlet" failures on plan9-amd64-0intro #49756

x/build: frequent "communication error to buildlet" failures on plan9-amd64-0intro #49756

Comments

bcmills commented Nov 23, 2021 • edited

dmitshur commented Nov 23, 2021

millerresearch commented Nov 23, 2021

bcmills commented Dec 9, 2021

0intro commented Dec 9, 2021

bcmills commented Jan 5, 2022

bcmills commented Jan 27, 2022

0intro commented Jan 28, 2022

gopherbot commented May 26, 2022

bcmills commented Jun 28, 2022

gopherbot commented Apr 16, 2023

bcmills commented Nov 23, 2021 •

edited