all: ensure that Go toolchain meets Apple’s notarization requirements #34986

artyom · 2019-10-18T10:10:03Z

What version of Go are you using (`go version`)?

$ go version
go version go1.13.3 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/artyom/Library/Caches/go-build"
GOENV="/Users/artyom/Library/Application Support/go/env"
GOEXE=""
GOFLAGS="-ldflags=-w -trimpath"
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/artyom/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/Users/artyom/Library/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/artyom/Library/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/artyom/Repositories/artyom/rex/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/lb/3rk8rqs53czgb4v35w_342xc0000gn/T/go-build499526677=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Installed go from https://dl.google.com/go/go1.13.3.darwin-amd64.tar.gz and then tried to use it to build project. This is on macOS Catalina 10.15:

Darwin MacBook-Air.local 19.0.0 Darwin Kernel Version 19.0.0: Wed Sep 25 20:18:50 PDT 2019; root:xnu-6153.11.26~2/RELEASE_X86_64 x86_64

What did you expect to see?

Go works as usual.

What did you see instead?

On first go call I'm presented with OS pop-up that "macOS cannot verify that this app is free from malware", with options either to discard this window (the process is killed then) or move the binary to trash. After whitelisting this binary via Preferences → Security & Privacy → General → Allow Anyway, go can be called fine.

Next I tried to build project with go install and got the came pop-up for "compile" tool.

The text was updated successfully, but these errors were encountered:

artyom · 2019-10-18T10:17:14Z

To be able to run updated Go 1.13.3 and compile my project with it, I had to whitelist so far these binaries:

go
compile
asm
cgo
link

Whitelisting each takes the following:

presented with os pop-up about binary not being verified;
click cancel, at this moment binary is killed;
go to Preferences → Security & Privacy → General, click "Allow Anyway" button;
try running the same command again;
presented with os pop-up about binary not being verified, but now with another option to run binary anyway;
click "open" button on that pop-up.

bradfitz · 2019-10-18T15:47:23Z

Good times.

/cc @golang/osp-team

bradfitz · 2019-10-18T15:48:07Z

Related: #34748 (we need a Catalina builder)

networkimprov · 2019-10-18T16:51:57Z

Discussion on Hacker News: https://news.ycombinator.com/item?id=21179970

Also, Catalina is reported to be a disastrous upgrade.

andybons · 2019-10-18T17:38:13Z

I’ll take a look at this.

dmitshur · 2019-10-18T18:09:11Z

I can reproduce on my personal MBP with macOS 10.15 (19A602) when installing from the .pkg installer too, however it's a different (less intrusive) experience.

The "macOS cannot verify that this app is free from malware" prompt appears when first running the installer. After accepting and installing, Go binaries such as go, compile, do not produce the same prompt. Also, re-running the .pkg installer doesn't ask again.

andybons · 2019-10-21T16:35:39Z

In the process of submitting the installer to Apple's notary service, but the process is very slow due to a service issue on Apple's side and our internal tooling keeps timing out. Diagnosing this has been difficult.

It may be that we will need to enable the Hardened Runtime which likely requires a newer version of Xcode that we're using to build releases and an audit of whatever entitlements are required for our binaries to run.

@aclements can you take a look at the link above and let me know which you think we'll need (if any)?

aclements · 2019-10-21T18:14:43Z

@andybons, I don't think we need any of the runtime exceptions from that page. The only one that we may even potentially want is "Disable Library Validation Entitlement", which would let users use the the plugin package without signing their plugins, but it seems like developers probably should sign their plugins.

vp2177 · 2019-10-22T09:34:11Z

The (.pkg) installer doesn't even start for me. Is that the same issue, or unrelated?

"go1.13.3.darwin-amd64.pkg can't be opened because Apple cannot check it for malicious software."

Meanwhile, the older Go version I had installed from before the OS X Catalina update, continues to work fine, no warning dialogs.

andybons · 2019-10-22T14:33:03Z

@vp2177 it’s the same issue.

andybons · 2019-10-22T14:51:32Z

Good news: according to Apple, we don’t have to enable the hardened runtime until January 2020. That said, we’ll likely have to start signing the binaries themselves (not just the installer as we do now), so this issue should stay open even if we fix the problem in the short term via notarization of the installer.

andybons · 2019-10-22T15:13:04Z

@dmitshur to follow up on your experience, do you have SIP (System Integrity Protection) turned on? csrutil status should give you that info.

@artyom do you have SIP turned on? Thanks.

dmitshur · 2019-10-22T15:46:22Z

@andybons Yes.

$ csrutil status
System Integrity Protection status: enabled.

artyom · 2019-10-22T16:44:41Z

@andybons yes, it's enabled:

¶ csrutil status
System Integrity Protection status: enabled.

andybons · 2019-10-22T17:01:17Z

Our installer is getting stuck in limbo with the Notarization service. We’ve contacted Apple and are awaiting a response. It’s unclear what the ETA on that will be, though.

andybons · 2019-10-22T18:28:23Z

https://developer.apple.com/system-status/ now shows an issue with the Notary service:

Performance: Today, 10:10 AM - ongoing

Some users are affected

Users are experiencing a problem with this service. We are working to resolve this issue.

andybons · 2019-10-23T15:36:28Z

Notarization of the installer package has completed successfully. We are testing locally to ensure no scary popups are shown.

andybons · 2019-10-23T16:15:28Z

While we believe we have the installer package issue resolved, we have to deal with the .tar.gz bundles, which require notarization of each individual binary. Working on this.

andybons · 2019-10-24T19:28:15Z

As an update, this is what we are going to do:

Sign all Mach-O binaries within $GOROOT/bin/ and $GOROOT/pkg/tool/darwin_amd64 (without the Hardened Runtime enabled as it’s not required just yet)
Construct the .pkg installer with the signed binaries included
Sign the .pkg installer
Send the .pkg installer off to the notarization service and staple the resulting ticket to the installer

Since the notarization process is recursive (and the bits being distributed in the .tar.gz file are the same that will be in the installer), this should solve the warnings for users that choose to install using either the .pkg file or the .tar.gz file.

I will update this thread once this work is complete.

andybons · 2019-10-29T00:13:11Z

The initial work mentioned in #34986 (comment) is now complete. We will likely issue a point release this week with the changes after some further testing.

andybons · 2019-11-21T16:28:35Z

@gopherbot please open a backport for 1.12 and 1.13 since otherwise Apple will reject future point releases of those versions

gopherbot · 2019-11-21T16:28:43Z

Backport issue(s) opened: #35747 (for 1.12), #35748 (for 1.13).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

gopherbot · 2019-11-21T16:28:49Z

Backport issue(s) opened: #35747 (for 1.12), #35748 (for 1.13).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

nightlyone · 2019-11-21T17:19:58Z

I wonder why tests and test data need to be shipped in releases. People interested in that can always retrieve it from the source repository. Has that been considered?

bradfitz · 2019-11-21T17:28:40Z

@nightlyone, it has, several times. The main reason we continue to include them is that when a user reports a problem, it's nice to be able to ask them "What does go test syscall or go test -v os say?". That's a question they can easily answer. If we make them do a new download, build, and parallel install, they're very unlikely to do so. Admittedly this debugging procedure doesn't happen too often, though.

…pple notarization to fail Starting with macOS 10.15 (Catalina), Apple now requires all software distributed outside of the App Store to be notarized. Any binaries we distribute must abide by a strict set of requirements like code-signing and having a minimum target SDK of 10.9 (amongst others). Apple’s notarization service will recursively inspect archives looking to find notarization candidate binaries. If it finds a binary that does not meet the requirements or is unable to decompress an archive, it will reject the entire distribution. From cursory testing, it seems that the service uses content sniffing to determine file types, so changing the file extension will not work. There are some binaries and archives included in our distribution that are being detected by Apple’s service as potential candidates for notarization or decompression. As these are files used by tests and some are intentionally invalid, we don’t intend to ever make them compliant. As a workaround for this, we base64-encode any binaries or archives that Apple’s notarization service issues a warning for, as these warnings will become errors in January 2020. Updates #34986 Fixes #35748 Change-Id: I106fbb6227b61eb221755568f047ee11103c1680 Reviewed-on: https://go-review.googlesource.com/c/go/+/208118 Run-TryBot: Andrew Bonventre <andybons@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> (cherry picked from commit 8bbfc51) Reviewed-on: https://go-review.googlesource.com/c/go/+/208219 Reviewed-by: Alexander Rakoczy <alex@golang.org>

…pple notarization to fail Starting with macOS 10.15 (Catalina), Apple now requires all software distributed outside of the App Store to be notarized. Any binaries we distribute must abide by a strict set of requirements like code-signing and having a minimum target SDK of 10.9 (amongst others). Apple’s notarization service will recursively inspect archives looking to find notarization candidate binaries. If it finds a binary that does not meet the requirements or is unable to decompress an archive, it will reject the entire distribution. From cursory testing, it seems that the service uses content sniffing to determine file types, so changing the file extension will not work. There are some binaries and archives included in our distribution that are being detected by Apple’s service as potential candidates for notarization or decompression. As these are files used by tests and some are intentionally invalid, we don’t intend to ever make them compliant. As a workaround for this, we base64-encode any binaries or archives that Apple’s notarization service issues a warning for, as these warnings will become errors in January 2020. Updates #34986 Updates #35747 Change-Id: I106fbb6227b61eb221755568f047ee11103c1680 Reviewed-on: https://go-review.googlesource.com/c/go/+/208118 Run-TryBot: Andrew Bonventre <andybons@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> (cherry picked from commit 8bbfc51) Reviewed-on: https://go-review.googlesource.com/c/go/+/208220 Reviewed-by: Alexander Rakoczy <alex@golang.org>

bcmills · 2019-11-21T17:37:26Z

It also seems desirable for go test all to have the same behavior regardless of how the Go toolchain was installed, particularly in module mode (where all is actually a reasonable thing to want to test).

gopherbot · 2019-11-21T19:37:51Z

Change https://golang.org/cl/208227 mentions this issue: [release-branch.go1.12] cmd/vendor: remove _test.go and testdata files

gopherbot · 2019-11-22T00:24:50Z

Change https://golang.org/cl/208266 mentions this issue: cmd/release, cmd/releasebot: clean up unused code

+ The .pkg installer is now constructed by an internal process, since the binaries within the installer need to be signed using internal-only certs + 1.11 is no longer supported, so the tour is no longer shipped with the release Updates golang/go#34986 Change-Id: Ic05198dec2fdbfb26d9011944051a97c777e3898 Reviewed-on: https://go-review.googlesource.com/c/build/+/208266 Run-TryBot: Andrew Bonventre <andybons@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

Binary files included in testdata directories can cause Apple’s notarization service to reject us since they don’t abide by their strict requirements. To emulate go mod vendor, remove all _test.go and testdata files from the vendor directory and update the instructions. Updates #34986 Fixes #35747 Change-Id: I5cde905fc78838d2e3b1519dab4aeee13d8d5356 Reviewed-on: https://go-review.googlesource.com/c/go/+/208227 Run-TryBot: Andrew Bonventre <andybons@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Alexander Rakoczy <alex@golang.org>

andybons · 2019-11-26T22:49:57Z

I've run an altered all.bash that signs (with the hardened runtime enabled) the toolchain binaries and all tests passed on macOS 10.15.1.

We plan to enable the hardened runtime for the 1.14 beta (scheduled for early December) and see if anything breaks. If everything looks good, we'll enable it for all releases moving forward.

toothrot · 2019-12-16T19:26:47Z

Just to reiterate what Andy said, the hardened runtime will be enabled for the go1.14 beta release.

cagedmantis · 2020-01-30T20:16:25Z

I've confirmed with @bcmills that #36572 should not be considered a blocker for go1.14rc1. I'm going to recommend that we close this issue. @andybons @dmitshur @toothrot

dmitshur · 2020-02-04T00:37:46Z

Sounds good. With the hardened runtime enabled as of Go 1.14 beta 1, I don't believe there's anything left to do here other than to watch for any issues if they're reported, but new issues can be opened for that.

I'll let one of @andybons or @toothrot confirm and close this issue since they're currently assigned to it.

andybons · 2020-02-04T14:33:31Z

Yes this looks to be fixed. If we discover any issues during the notarization stage of a release we'll open a new issue.

bradfitz added this to the Go1.14 milestone Oct 18, 2019

bradfitz added release-blocker NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Oct 18, 2019

andybons self-assigned this Oct 18, 2019

andybons pinned this issue Oct 22, 2019

andybons added the OS-Darwin label Oct 22, 2019

andybons changed the title ~~Go 1.13.3 distribution lacks notarization for macOS Catalina, requiring whitelisting of each binary~~ Notarize our installer and binaries (required as of macOS 10.15 Catalina) Oct 22, 2019

rsc changed the title ~~Notarize our installer and binaries (required as of macOS 10.15 Catalina)~~ macOS 10.15 Catalina rejects non-notarized Go installer, binaries Oct 24, 2019

rsc unpinned this issue Oct 30, 2019

rsc pinned this issue Oct 30, 2019

This was referenced Nov 21, 2019

all: ensure that Go toolchain meets Apple’s notarization requirements [1.12 backport] #35747

Closed

all: ensure that Go toolchain meets Apple’s notarization requirements [1.13 backport] #35748

Closed

russorat mentioned this issue Nov 21, 2019

Dealing with MacOS Catalina Notary Service influxdata/influxdb#16009

Open

mohammedzee1000 mentioned this issue Nov 27, 2019

Notarize odo in order to run on macOS Catalina redhat-developer/odo#2408

Closed

andybons assigned toothrot Dec 6, 2019

oneslash mentioned this issue Jan 3, 2020

Notirize application for macOS simplesurance/baur#92

Closed

rodrigc mentioned this issue Jan 14, 2020

macOS Catalina: "eksctl" cannot be opened because the developer cannot be verified eksctl-io/eksctl#1718

Closed

cagedmantis mentioned this issue Jan 15, 2020

cmd/go: dylib on macos with rpath fails to run with go run/test #36572

Open

andybons closed this as completed Feb 4, 2020

golang locked and limited conversation to collaborators Feb 3, 2021

gopherbot added the FrozenDueToAge label Feb 3, 2021

rsc unassigned toothrot and andybons Jun 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

all: ensure that Go toolchain meets Apple’s notarization requirements #34986

all: ensure that Go toolchain meets Apple’s notarization requirements #34986

artyom commented Oct 18, 2019

artyom commented Oct 18, 2019

bradfitz commented Oct 18, 2019

bradfitz commented Oct 18, 2019

networkimprov commented Oct 18, 2019

andybons commented Oct 18, 2019

dmitshur commented Oct 18, 2019

andybons commented Oct 21, 2019

aclements commented Oct 21, 2019

vp2177 commented Oct 22, 2019 •

edited

andybons commented Oct 22, 2019

andybons commented Oct 22, 2019 •

edited

andybons commented Oct 22, 2019

dmitshur commented Oct 22, 2019

artyom commented Oct 22, 2019

andybons commented Oct 22, 2019

andybons commented Oct 22, 2019

andybons commented Oct 23, 2019

andybons commented Oct 23, 2019

andybons commented Oct 24, 2019

andybons commented Oct 29, 2019

andybons commented Nov 21, 2019

gopherbot commented Nov 21, 2019

gopherbot commented Nov 21, 2019

nightlyone commented Nov 21, 2019

bradfitz commented Nov 21, 2019

bcmills commented Nov 21, 2019

gopherbot commented Nov 21, 2019

gopherbot commented Nov 22, 2019

andybons commented Nov 26, 2019 •

edited

toothrot commented Dec 16, 2019

cagedmantis commented Jan 30, 2020

dmitshur commented Feb 4, 2020

andybons commented Feb 4, 2020

all: ensure that Go toolchain meets Apple’s notarization requirements #34986

all: ensure that Go toolchain meets Apple’s notarization requirements #34986

Comments

artyom commented Oct 18, 2019

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

artyom commented Oct 18, 2019

bradfitz commented Oct 18, 2019

bradfitz commented Oct 18, 2019

networkimprov commented Oct 18, 2019

andybons commented Oct 18, 2019

dmitshur commented Oct 18, 2019

andybons commented Oct 21, 2019

aclements commented Oct 21, 2019

vp2177 commented Oct 22, 2019 • edited

andybons commented Oct 22, 2019

andybons commented Oct 22, 2019 • edited

andybons commented Oct 22, 2019

dmitshur commented Oct 22, 2019

artyom commented Oct 22, 2019

andybons commented Oct 22, 2019

andybons commented Oct 22, 2019

andybons commented Oct 23, 2019

andybons commented Oct 23, 2019

andybons commented Oct 24, 2019

andybons commented Oct 29, 2019

andybons commented Nov 21, 2019

gopherbot commented Nov 21, 2019

gopherbot commented Nov 21, 2019

nightlyone commented Nov 21, 2019

bradfitz commented Nov 21, 2019

bcmills commented Nov 21, 2019

gopherbot commented Nov 21, 2019

gopherbot commented Nov 22, 2019

andybons commented Nov 26, 2019 • edited

toothrot commented Dec 16, 2019

cagedmantis commented Jan 30, 2020

dmitshur commented Feb 4, 2020

andybons commented Feb 4, 2020

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

vp2177 commented Oct 22, 2019 •

edited

andybons commented Oct 22, 2019 •

edited

andybons commented Nov 26, 2019 •

edited