Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

all: stop using non-libc system calls on OpenBSD #36435

Open
4a6f656c opened this issue Jan 7, 2020 · 98 comments
Open

all: stop using non-libc system calls on OpenBSD #36435

4a6f656c opened this issue Jan 7, 2020 · 98 comments
Assignees
Labels
help wanted NeedsFix The path to resolution is known, but the work has not been done. OS-OpenBSD
Milestone

Comments

@4a6f656c
Copy link
Contributor

4a6f656c commented Jan 7, 2020

Upcoming changes to the OpenBSD kernel will prevent system calls from being made unless they are coming from libc.so (with some exceptions, for example, a static binary). There are also likely to be changes to the APIs used for system calls. As such, the Go runtime (and other packages) need to stop using direct syscalls, rather calling into libc functions instead (as has always been done for Solaris and now also for macOS). This will avoid these issues and allow Go to continue to function correctly on OpenBSD.

A version of Go with openbsd/amd64 being modified to perform all calls via libc is available at:

https://github.com/4a6f656c/go/tree/openbsd-syscall

Further work is still required to convert openbsd/386, openbsd/arm and openbsd/arm64.

@randall77
Copy link
Contributor

with some exceptions, for example, a static binary

But Go fits squarely within that exception. Or are you worried about nonstandard build modes?

@bradfitz bradfitz added this to the Go1.15 milestone Jan 7, 2020
@FiloSottile
Copy link
Contributor

That exception exists specifically for Go. https://lwn.net/Articles/806863/

@beoran
Copy link

beoran commented Jan 8, 2020

I think this move to make kernels callable only though a blessed C library is a bad idea, in particular for all users of programming languages that are not C-like. It introduces a performance overhead for every kernel call that could be avoided by direct kernel calls. The security benefits are debatable, fix the kernel, not hide the flaws and incompatibilities behind a C library.

Kernels should do like Linux does, and provide a safe, stable, kernel API, and not try to shift the interface to C libraries. And even then, graphic on Linux is a disaster because they exactly did that, provide most functionality in a C library, not in the kernel itself. We should protest at least once to mr. De Raadt. against this debatable decision.

@4a6f656c
Copy link
Contributor Author

4a6f656c commented Jan 8, 2020

@randall77 - in many cases Go on OpenBSD will generate a dynamically linked executable, unless cgo is disabled:

$ cat n.go
package main

import (
        "fmt"
        "net"
)

func main() {
        fmt.Println(net.IPv4len)
}
$ go build -o n n.go 
$ ldd n              
n:
        Start            End              Type  Open Ref GrpRef Name
        0000000000010000 00000000001d3000 exe   2    0   0      n
        0000000453cfd000 0000000453d3e000 rlib  0    1   0      /usr/lib/libpthread.so.26.1
        000000048614c000 000000048624c000 rlib  0    1   0      /usr/lib/libc.so.95.1
        0000000497c90000 0000000497c90000 ld.so 0    1   0      /usr/libexec/ld.so
$ CGO_ENABLED=0 go build -o n n.go
ldd n
$ ldd n
n:
not a dynamic executable

This would mean that we'd have to drop cgo support and only support statically compiled Go executables.

@FiloSottile you're correct in that there is currently an exception for dynamically linked executables to allow syscalls from both libc.so and the main text segment, so that Go continues to work. However, in addition to this there will always likely be an exception for static binaries, since in that case there is no libc.so and parts of libc.a have likely been linked into the main text segment (otherwise a static binary could not make syscalls).

@ghost
Copy link

ghost commented Jan 12, 2020

@4a6f656c why does it generate dynamically linked executable in the first place?

@gopherbot
Copy link

Change https://golang.org/cl/234381 mentions this issue: runtime, syscall: correct openbsd/arm and openbsd/arm64 syscalls for OpenBSD 6.7

gopherbot pushed a commit that referenced this issue May 26, 2020
…OpenBSD 6.7

Add two no op instructions following svc on openbsd/arm64 and swi on openbsd/arm.

All except some of the most recent arm64 processors have a speculative execution
flaw that occurs across a syscall boundary, which cannot be mitigated in the
kernel. In order to protect against this leak a speculation barrier needs to be
placed after an svc or swi instruction.

In order to avoid the performance impact of these instructions, the OpenBSD 6.7
kernel returns execution two instructions past the svc or swi call. For now two
hardware no ops are added, which allows syscalls to work with both 6.6 and 6.7.
These should be replaced with real speculation barriers once OpenBSD 6.8 is
released.

Updates #36435

Change-Id: I06153cb0998199242cca8761450e53599c3e7de4
Reviewed-on: https://go-review.googlesource.com/c/go/+/234381
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@ianlancetaylor
Copy link
Contributor

Moving to backlog milestone.

@ianlancetaylor ianlancetaylor modified the milestones: Go1.15, Backlog Jun 16, 2020
@ianlancetaylor ianlancetaylor added help wanted NeedsFix The path to resolution is known, but the work has not been done. labels Jun 16, 2020
@bw-86
Copy link

bw-86 commented Jul 17, 2020

I would like to work on this. After a brief look, Solaris seems to be a good role model for OpenBSD. What would be the best strategy here? OpenBSD seems to share definitions with the other BSD platforms, which result in conflicting definitions when I try to hook it into mksyscall_libc.pl.
Or should I try to do things more like MacOS? Is that an easier way? How should I proceed?

@ianlancetaylor
Copy link
Contributor

I don't think there is any fundamental difference between the Solaris approach and the macOS approach. Following the Solaris approach seems fine if that is easier.

Yes, you will undoubtedly have to shuffle the build tags to separate openbsd from the other BSD systems in some cases.

@bw-86
Copy link

bw-86 commented Jul 18, 2020

Well, I started. My WIP commits will be pushed here, if anyone is interested in them: https://github.com/bw-86/go

Currently, nothing is building, of course. I am chasing from one linking error to the next one while imitating what Solaris does. I will notify you when I'm getting somewhere (or when I have questions I cannot answer myself).

EDIT: I changed the default branch of my fork to 4a6f656c's work. My branch is openbsd-syscalls (plural), if anyone wants to see what I tried myself.

@ghost
Copy link

ghost commented Jul 18, 2020

Does that mean Go will always use dynamic linking or raw syscalls will still be used when linking statically?

I agree with @beoran and iirc OpenBSD will not enforce libc.so stubs if you won't play their security theater, so you don't have to sacrifice both efficiency and actual security.

@4a6f656c
Copy link
Contributor Author

I would like to work on this. After a brief look, Solaris seems to be a good role model for OpenBSD. What would be the best strategy here? OpenBSD seems to share definitions with the other BSD platforms, which result in conflicting definitions when I try to hook it into mksyscall_libc.pl.
Or should I try to do things more like MacOS? Is that an easier way? How should I proceed?

@bw-86, the git repo referenced in the first comment contains a full implementation for openbsd/amd64:

https://github.com/4a6f656c/go/tree/openbsd-syscall

I've since rebased and cleaned some aspects of this up, but need to push a new branch.

As noted, there is still work required to complete the migration for openbsd/386, openbsd/arm and openbsd/arm64. There are also some issues relating to the linker that impact this work (see #39257 and #39256). Aside from the remaining architectures, most of the remaining work is around being able to split this up to merge upstream and I have some concerns about cross-compiling and bootstrapping that I've not yet had time to investigate.

@4a6f656c
Copy link
Contributor Author

Does that mean Go will always use dynamic linking or raw syscalls will still be used when linking statically?

Go will always use dynamic linking (and possibly always external linking) - it would be a significant amount of effort to maintain both libc based syscalls and raw syscalls and switch between to two. Any Go binary on OpenBSD that pulls in the net package is already going to result in a dynamic cgo-based binary (unless cgo is explicitly disabled - see #36435 (comment)).

I agree with @beoran and iirc OpenBSD will not enforce libc.so stubs if you won't play their security theater, so you don't have to sacrifice both efficiency and actual security.

If you do not want OpenBSD's "security theater" then I would suggest not using OpenBSD. Additionally, have you actually measured the performance difference? Both Solaris and macOS already both use libc-style syscalls and IIRC the switch on macOS resulted in a minimal performance impact.

@bw-86
Copy link

bw-86 commented Jul 18, 2020

Heh, how could I miss that? Well, at least I learned something by trying to do it myself.

@4a6f656c: Can I do something to accelerate this? Should I try to adapt your work to i386? I have a system here were I could develop and test it.

@ghost

This comment has been minimized.

@4a6f656c
Copy link
Contributor Author

Heh, how could I miss that? Well, at least I learned something by trying to do it myself.

That's always a good thing :)

@4a6f656c: Can I do something to accelerate this? Should I try to adapt your work to i386? I have a system here were I could develop and test it.

Getting it working on openbsd/386 would be great!

I've just pushed an update to that repo that is based on Go -tip.

@4a6f656c
Copy link
Contributor Author

Does that mean Go will always use dynamic linking or raw syscalls will still be used when linking statically?

Go will always use dynamic linking (and possibly always external linking) - it would be a significant amount of effort to maintain both libc based syscalls and raw syscalls and switch between to two. Any Go binary on OpenBSD that pulls in the net package is already going to result in a dynamic cgo-based binary (unless cgo is explicitly disabled - see #36435 (comment)).

Actually, there is another thing that I need to investigate - in many cases we should be able to statically link against libc.a, which would actually allow us to produce static Go binaries in more cases then we currently do on OpenBSD.

@jrick
Copy link
Contributor

jrick commented Jul 18, 2020

I can chime in as both an OpenBSD user and someone who works on a project that builds releases for OpenBSD. I am happy with this change but hope that cross compiling to OpenBSD and reproducible builds remain possible. This probably negates requiring the external linker and having libc.a available.

The current behavior for macOS is ideal in this case. When cross compiling, it is still able to create a dynamically linked executable, and does so reproducibly, without depending on the external macOS linker.

$ uname
OpenBSD
$ cat importnet.go                                                             
package main

import _ "net"

func main() {}
$ CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build importnet.go 
$ echo put importnet | sftp machost
Connected to machost.
sftp> put importnet
Uploading importnet to /Users/jrick/importnet
importnet                                     100% 2047KB   2.6MB/s   00:00    
$ ssh machost './importnet && otool -L importnet'
importnet:
        /usr/lib/libSystem.B.dylib (compatibility version 0.0.0, current version 0.0.0)
        /System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 0.0.0, current version 0.0.0)
        /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 0.0.0, current version 0.0.0)

@bw-86
Copy link

bw-86 commented Jul 19, 2020

I have to admit defeat for the moment. I pushed some minor work to my repository; feel free to take it.
I will have to learn some more assembly and more about the calling conventions of Go and OpenBSD before I can tackle this. To be precise, I don't know how to write the syscall functions (syscall6, syscall9, ...).

So, can someone help me with this one? If I had just one of them, I might adapt it for the other use cases.

@golang golang deleted a comment Jul 21, 2020
@gopherbot
Copy link

Change https://golang.org/cl/250180 mentions this issue: cmd/dist,cmd/link,runtime: switch openbsd/amd64 to pthreads

@gopherbot
Copy link

Change https://go.dev/cl/421798 mentions this issue: unix: convert openbsd/386 to direct libc calls

@gopherbot
Copy link

Change https://go.dev/cl/421800 mentions this issue: unix: convert openbsd/arm to direct libc calls

gopherbot pushed a commit to golang/sys that referenced this issue Aug 8, 2022
Rename mkasm_darwin.go to mkasm.go and pass the operating system as an
argument. This will allow mkasm to be used to generate assembly for
OpenBSD.

Updates golang/go#36435

Change-Id: I42f54f5c6edc3a18a156e0e8e3c38d07ecf26d0b
Reviewed-on: https://go-review.googlesource.com/c/sys/+/421795
Run-TryBot: Joel Sing <joel@sing.id.au>
Run-TryBot: Ian Lance Taylor <iant@google.com>
Reviewed-by: Than McIntosh <thanm@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Auto-Submit: Ian Lance Taylor <iant@google.com>
gopherbot pushed a commit to golang/sys that referenced this issue Aug 11, 2022
The current code has continued to work on OpenBSD, since it has been using
syscall(2) via libc. However, the system call numbers are still hardcoded in
golang.org/sys/unix. Various system call changes have been made in OpenBSD,
resulting in changes to the system call numbers and arguments, which now
fail when this package is used.

Switch to calling various system calls directly via libc, rather than calling
via libc using syscall(2).

Updates golang/go#36435

Change-Id: I836a484b14e0a427ac565315e27f0de1e9a5d021
Reviewed-on: https://go-review.googlesource.com/c/sys/+/421796
Reviewed-by: Ian Lance Taylor <iant@google.com>
Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com>
Auto-Submit: Ian Lance Taylor <iant@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Ian Lance Taylor <iant@google.com>
Reviewed-by: Than McIntosh <thanm@google.com>
gopherbot pushed a commit to golang/sys that referenced this issue Aug 11, 2022
The current code has continued to work on OpenBSD, since it has been using
syscall(2) via libc. However, the system call numbers are still hardcoded in
golang.org/sys/unix. Various system call changes have been made in OpenBSD,
resulting in changes to the system call numbers and arguments, which now
fail when this package is used.

Switch to calling various system calls directly via libc, rather than calling
via libc using syscall(2).

Updates golang/go#36435

Change-Id: Iadd5734fd4a2421d758883293bd8a6759fc52ca7
Reviewed-on: https://go-review.googlesource.com/c/sys/+/421797
Reviewed-by: Ian Lance Taylor <iant@google.com>
Auto-Submit: Ian Lance Taylor <iant@google.com>
Run-TryBot: Joel Sing <joel@sing.id.au>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com>
Run-TryBot: Ian Lance Taylor <iant@google.com>
gopherbot pushed a commit to golang/sys that referenced this issue Aug 11, 2022
The current code has continued to work on OpenBSD, since it has been using
syscall(2) via libc. However, the system call numbers are still hardcoded in
golang.org/sys/unix. Various system call changes have been made in OpenBSD,
resulting in changes to the system call numbers and arguments, which now
fail when this package is used.

Switch to calling various system calls directly via libc, rather than calling
via libc using syscall(2).

Updates golang/go#36435

Change-Id: Ib42d5415ef35c7f7b9cbc55adaf7ca15973ab287
Reviewed-on: https://go-review.googlesource.com/c/sys/+/421798
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Run-TryBot: Ian Lance Taylor <iant@google.com>
Auto-Submit: Ian Lance Taylor <iant@google.com>
gopherbot pushed a commit to golang/sys that referenced this issue Aug 17, 2022
The current code has continued to work on OpenBSD, since it has been using
syscall(2) via libc. However, the system call numbers are still hardcoded in
golang.org/x/sys/unix. Various system call changes have been made in OpenBSD,
resulting in changes to the system call numbers and arguments, which now
fail when this package is used.

Switch to calling various system calls directly via libc, rather than calling
via libc using syscall(2).

Updates golang/go#36435

Change-Id: If8e403f0fda7a8b68da71c1f4efba7785b14edf5
Reviewed-on: https://go-review.googlesource.com/c/sys/+/421800
Run-TryBot: Ian Lance Taylor <iant@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Joel Sing <joel@sing.id.au>
Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
@gopherbot
Copy link

Change https://go.dev/cl/425637 mentions this issue: internal/syscall/unix: convert openbsd (except mips64) to direct libc calls

gopherbot pushed a commit that referenced this issue Aug 31, 2022
…except mips64) to direct libc calls

Call libc wrappers directly rather than calling using syscall(2).

Updates #36435

Change-Id: I40be410c7472f7d89cbec2ebdc7c841c7726ca4a
Reviewed-on: https://go-review.googlesource.com/c/go/+/425637
Reviewed-by: David Chase <drchase@google.com>
Reviewed-by: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
Reviewed-by: Joel Sing <joel@sing.id.au>
@gopherbot
Copy link

Change https://go.dev/cl/428776 mentions this issue: internal/poll, syscall: convert writev to direct libc call on openbsd (except mips64)

gopherbot pushed a commit that referenced this issue Sep 9, 2022
… (except mips64)

Add and use a writev libc wrapper instead of using syscall(2).

Updates #36435

Change-Id: I3c67665388ac276d9ef36868e368e91efb92800e
Reviewed-on: https://go-review.googlesource.com/c/go/+/428776
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
Reviewed-by: Joel Sing <joel@sing.id.au>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Benny Siegert <bsiegert@gmail.com>
Auto-Submit: Tobias Klauser <tobias.klauser@gmail.com>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
@gopherbot
Copy link

Change https://go.dev/cl/439975 mentions this issue: unix: flip openbsd libc build tags

gopherbot pushed a commit to golang/sys that referenced this issue Oct 10, 2022
Rather than listing OpenBSD architectures that are using libc, list the one
that is still blocked on direct system calls. This will simplify the addition
of new OpenBSD ports.

Updates golang/go#36435

Change-Id: I8d1d16a033245af436d955b5fa6ceeaa1f2a9f6b
Reviewed-on: https://go-review.googlesource.com/c/sys/+/439975
Run-TryBot: Joel Sing <joel@sing.id.au>
Reviewed-by: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com>
@gopherbot
Copy link

Change https://go.dev/cl/459497 mentions this issue: unix: convert openbsd/mips64 to direct libc calls

gopherbot pushed a commit to golang/sys that referenced this issue Dec 30, 2022
The current code has continued to work on OpenBSD, since it has been using
syscall(2) via libc. However, the system call numbers are still hardcoded in
golang.org/x/sys/unix. Various system call changes have been made in OpenBSD,
resulting in changes to the system call numbers and arguments, which now
fail when this package is used.

Switch to calling various system calls directly via libc, rather than calling
via libc using syscall(2).

Unfortunately, this will no longer work on upstream Go since the changes
needed to convert it to libc are still blocked pending review. This means
that we're in a less than ideal situation where upstream Go openbsd/mips64
does not work on any supported OpenBSD release (in fact has not since
OpenBSD 6.8, which was EOL over a year ago), however golang.org/x/sys/unix
is now unusable with the Go package that ships with supported releases via
OpenBSD ports. It would seem that being able to actually use Go software on
a supported OpenBSD release trumps maintaining compatibility with the
unusable upstream Go source.

Updates golang/go#36435

Change-Id: Id8947cd0e4e05709e96c3d4478ac8789b924d416
Reviewed-on: https://go-review.googlesource.com/c/sys/+/459497
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Run-TryBot: Ian Lance Taylor <iant@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
@gopherbot
Copy link

Change https://go.dev/cl/463982 mentions this issue: cmd/link: make .dynamic section read-only for MIPS ELF

gopherbot pushed a commit that referenced this issue Jan 31, 2023
For #36435

Change-Id: Ie733b641f20ca5bcee3784c088eb27699890a151
Reviewed-on: https://go-review.googlesource.com/c/go/+/463982
Reviewed-by: Joel Sing <joel@sing.id.au>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Auto-Submit: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
johanbrandhorst pushed a commit to Pryz/go that referenced this issue Feb 12, 2023
For golang#36435

Change-Id: Ie733b641f20ca5bcee3784c088eb27699890a151
Reviewed-on: https://go-review.googlesource.com/c/go/+/463982
Reviewed-by: Joel Sing <joel@sing.id.au>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Auto-Submit: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
gopherbot pushed a commit that referenced this issue Feb 22, 2023
Add internal linking support for calling SDYNIMPORT symbols on mips64. This adds
code to generate appropriate PLT and GOT entries, along with the various dynamic
entries needed for the dynamic loader.

Updates #36435, #46178

Change-Id: I783e0d028510ca2bca82bcbc745f2375770813fe
Reviewed-on: https://go-review.googlesource.com/c/go/+/415815
Reviewed-by: Rong Zhang <rongrong@oss.cipunited.com>
Run-TryBot: Ian Lance Taylor <iant@google.com>
Run-TryBot: Joel Sing <joel@sing.id.au>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Reviewed-by: Than McIntosh <thanm@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Ian Lance Taylor <iant@google.com>
@gopherbot
Copy link

Change https://go.dev/cl/516016 mentions this issue: runtime,syscall: invert openbsd architecture tests

gopherbot pushed a commit that referenced this issue Aug 5, 2023
Rather than testing for architectures that use libc-based system calls,
test that it is not the single architecture that Go is still using direct
system calls. This reduces the number of changes needed for new openbsd
ports.

Updates #36435
Updates #61546

Change-Id: I79c4597c629b8b372e9efcda79e8f6ff778b9e8e
Reviewed-on: https://go-review.googlesource.com/c/go/+/516016
Reviewed-by: Ian Lance Taylor <iant@google.com>
Auto-Submit: Ian Lance Taylor <iant@google.com>
Run-TryBot: Ian Lance Taylor <iant@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Run-TryBot: Joel Sing <joel@sing.id.au>
TryBot-Result: Gopher Robot <gobot@golang.org>
@4a6f656c 4a6f656c changed the title all: stop using direct syscalls on OpenBSD all: stop using non-libc syscalls on OpenBSD Nov 6, 2023
@4a6f656c 4a6f656c changed the title all: stop using non-libc syscalls on OpenBSD all: stop using non-libc system calls on OpenBSD Nov 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted NeedsFix The path to resolution is known, but the work has not been done. OS-OpenBSD
Projects
None yet
Development

No branches or pull requests

14 participants