Go for Java Programmers

Sameer Ajmani

Tech Lead Manager, Go team

Google

Video

This talk was presented at NYJavaSIG on April 23, 2015.

Watch the talk on YouTube

2

Outline

1. What is Go, and who uses it?
2. Comparing Go and Java
3. Examples
4. Concurrency
5. Tools

3

What is Go?

"Go is an open source programming language that makes it easy to build simple, reliable, and efficient software."

go.dev

4

History

Design began in late 2007.

Open source since 2009 with a very active community.

Language stable as of Go 1, early 2012.

5

Why Go?

Go is an answer to problems of scale at Google.

6

System Scale

Solution: great support for concurrency

7

A Second Problem: Engineering Scale

In 2011:

Solution: design the language for large code bases

8

Who uses Go at Google?

Lots of projects. Thousands of Go programmers. Millions of lines of Go code.

Public examples:

9

Who uses Go at Google?

Lots of projects. Thousands of Go programmers. Millions of lines of Go code.

Public examples:

The target is networked servers, but it's a great general-purpose language.

10

Who uses Go besides Google?

go.dev/wiki/GoUsers

Apcera, Bitbucket, bitly, Canonical, CloudFlare, Core OS, Digital Ocean, Docker, Dropbox, Facebook, Getty Images, GitHub, Heroku, Iron.io, Kubernetes, Medium, MongoDB services, Mozilla services, New York Times, pool.ntp.org, Secret, SmugMug, SoundCloud, Stripe, Square, Thomson Reuters, Tumblr, ...

11

Comparing Go and Java

12

Go and Java have much in common

13

Go differs from Java in several ways

14

Go intentionally leaves out many features

15

Why does Go leave out those features?

Clarity is critical.

When reading code, it should be clear what the program will do.

When writing code, it should be clear how to make the program do what you want.

Sometimes this means writing out a loop instead of invoking an obscure function.

(Don't DRY out.)

For more background on design:

16

Examples

17

Go looks familiar to Java programmers

Main.java 

public class Main {
    public static void main(String[] args) {
        System.out.println("Hello, world!");
    }
}

hello.go 

package main

import "fmt"

func main() {
    fmt.Println("Hello, 世界!")
}
18

Hello, web server

package main

import (
    "fmt"
    "log"
    "net/http"
)

func main() {
    http.HandleFunc("/hello", handleHello)
    fmt.Println("serving on http://localhost:7777/hello")
    log.Fatal(http.ListenAndServe("localhost:7777", nil))
}

func handleHello(w http.ResponseWriter, req *http.Request) {
    log.Println("serving", req.URL)
    fmt.Fprintln(w, "Hello, 世界!")
}

Types follow names in declarations.
Exported names are Capitalized. Unexported names are not.

19

Example: Google Search frontend

// +build ignore,OMIT

// The server program issues Google search requests. It serves on port 8080.
//
// The /search endpoint accepts these query params:
//   q=the Google search query
//
// For example, http://localhost:8080/search?q=golang serves the first
// few Google search results for "golang".
package main

import (
	"encoding/json"
	"fmt"
	"html/template"
	"log"
	"net/http"
	"net/url"
	"time"
)


func main() {
    http.HandleFunc("/search", handleSearch)
    fmt.Println("serving on http://localhost:8080/search")
    log.Fatal(http.ListenAndServe("localhost:8080", nil))
}

// handleSearch handles URLs like "/search?q=golang" by running a
// Google search for "golang" and writing the results as HTML to w.
func handleSearch(w http.ResponseWriter, req *http.Request) {

	log.Println("serving", req.URL)

	// Check the search query.
	query := req.FormValue("q") // HL
	if query == "" {
		http.Error(w, `missing "q" URL parameter`, http.StatusBadRequest)
		return
	}
	// ENDQUERY OMIT

	// Run the Google search.
	start := time.Now()
	results, err := Search(query) // HL
	elapsed := time.Since(start)
	if err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
		return
	}
	// ENDSEARCH OMIT

	// Render the results.
	type templateData struct {
		Results []Result
		Elapsed time.Duration
	}
	if err := resultsTemplate.Execute(w, templateData{ // HL
		Results: results,
		Elapsed: elapsed,
	}); err != nil {
		log.Print(err)
		return
	}
	// ENDRENDER OMIT
}

// A Result contains the title and URL of a search result.
type Result struct { // HL
	Title, URL string // HL
} // HL

var resultsTemplate = template.Must(template.New("results").Parse(`
<html>
<head/>
<body>
  <ol>
  {{range .Results}}
    <li>{{.Title}} - <a href="{{.URL}}">{{.URL}}</a></li>
  {{end}}
  </ol>
  <p>{{len .Results}} results in {{.Elapsed}}</p>
</body>
</html>
`))

// Search sends query to Google search and returns the results.
func Search(query string) ([]Result, error) {
	// Prepare the Google Search API request.
	u, err := url.Parse("https://ajax.googleapis.com/ajax/services/search/web?v=1.0")
	if err != nil {
		return nil, err
	}
	q := u.Query()
	q.Set("q", query) // HL
	u.RawQuery = q.Encode()

	// Issue the HTTP request and handle the response.
	resp, err := http.Get(u.String()) // HL
	if err != nil {
		return nil, err
	}
	defer resp.Body.Close() // HL

	// Parse the JSON search result.
	// https://developers.google.com/web-search/docs/#fonje
	var jsonResponse struct {
		ResponseData struct {
			Results []struct {
				TitleNoFormatting, URL string
			}
		}
	}
	if err := json.NewDecoder(resp.Body).Decode(&jsonResponse); err != nil { // HL
		return nil, err
	}

	// Extract the Results from jsonResponse and return them.
	var results []Result
	for _, r := range jsonResponse.ResponseData.Results { // HL
		results = append(results, Result{Title: r.TitleNoFormatting, URL: r.URL})
	}
	return results, nil
}
20

Validate the query

func handleSearch(w http.ResponseWriter, req *http.Request) {
    log.Println("serving", req.URL)

    // Check the search query.
    query := req.FormValue("q")
    if query == "" {
        http.Error(w, `missing "q" URL parameter`, http.StatusBadRequest)
        return
    }

FormValue is a method on the type *http.Request: 

package http
type Request struct {...}
func (r *Request) FormValue(key string) string {...}

query := req.FormValue("q") initializes a new variable query with
the type of the expression on the right hand side, string.

21

Fetch the search results

    // Run the Google search.
    start := time.Now()
    results, err := Search(query)
    elapsed := time.Since(start)
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }

Search returns two values, a slice of results and an error. 

func Search(query string) ([]Result, error) {...}

The results are valid only if the error is nil. 

type error interface {
    Error() string // a useful human-readable error message
}

Errors may contain additional information, accessed via type assertions.

22

Render the search results

    // Render the results.
    type templateData struct {
        Results []Result
        Elapsed time.Duration
    }
    if err := resultsTemplate.Execute(w, templateData{
        Results: results,
        Elapsed: elapsed,
    }); err != nil {
        log.Print(err)
        return
    }

resultsTemplate.Execute generates HTML and writes it to an io.Writer: 

type Writer interface {
        Write(p []byte) (n int, err error)
}

http.ResponseWriter implements the io.Writer interface.

23

HTML templates operate on Go values

// +build ignore,OMIT

// The server program issues Google search requests. It serves on port 8080.
//
// The /search endpoint accepts these query params:
//   q=the Google search query
//
// For example, http://localhost:8080/search?q=golang serves the first
// few Google search results for "golang".
package main

import (
	"encoding/json"
	"fmt"
	"html/template"
	"log"
	"net/http"
	"net/url"
	"time"
)

func main() {
	http.HandleFunc("/search", handleSearch) // HL
	fmt.Println("serving on http://localhost:8080/search")
	log.Fatal(http.ListenAndServe("localhost:8080", nil))
}

// handleSearch handles URLs like "/search?q=golang" by running a
// Google search for "golang" and writing the results as HTML to w.
func handleSearch(w http.ResponseWriter, req *http.Request) {
	log.Println("serving", req.URL)

	// Check the search query.
	query := req.FormValue("q") // HL
	if query == "" {
		http.Error(w, `missing "q" URL parameter`, http.StatusBadRequest)
		return
	}
	// ENDQUERY OMIT

	// Run the Google search.
	start := time.Now()
	results, err := Search(query) // HL
	elapsed := time.Since(start)
	if err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
		return
	}
	// ENDSEARCH OMIT

	// Render the results.
	type templateData struct {
		Results []Result
		Elapsed time.Duration
	}
	if err := resultsTemplate.Execute(w, templateData{ // HL
		Results: results,
		Elapsed: elapsed,
	}); err != nil {
		log.Print(err)
		return
	}
	// ENDRENDER OMIT
}


// A Result contains the title and URL of a search result.
type Result struct {
    Title, URL string
}

var resultsTemplate = template.Must(template.New("results").Parse(`
<html>
<head/>
<body>
  <ol>
  {{range .Results}}
    <li>{{.Title}} - <a href="{{.URL}}">{{.URL}}</a></li>
  {{end}}
  </ol>
  <p>{{len .Results}} results in {{.Elapsed}}</p>
</body>
</html>
`))


// Search sends query to Google search and returns the results.
func Search(query string) ([]Result, error) {
	// Prepare the Google Search API request.
	u, err := url.Parse("https://ajax.googleapis.com/ajax/services/search/web?v=1.0")
	if err != nil {
		return nil, err
	}
	q := u.Query()
	q.Set("q", query) // HL
	u.RawQuery = q.Encode()

	// Issue the HTTP request and handle the response.
	resp, err := http.Get(u.String()) // HL
	if err != nil {
		return nil, err
	}
	defer resp.Body.Close() // HL

	// Parse the JSON search result.
	// https://developers.google.com/web-search/docs/#fonje
	var jsonResponse struct {
		ResponseData struct {
			Results []struct {
				TitleNoFormatting, URL string
			}
		}
	}
	if err := json.NewDecoder(resp.Body).Decode(&jsonResponse); err != nil { // HL
		return nil, err
	}

	// Extract the Results from jsonResponse and return them.
	var results []Result
	for _, r := range jsonResponse.ResponseData.Results { // HL
		results = append(results, Result{Title: r.TitleNoFormatting, URL: r.URL})
	}
	return results, nil
}
24

Issue the query to the Google Search API

func Search(query string) ([]Result, error) {
    // Prepare the Google Search API request.
    u, err := url.Parse("https://ajax.googleapis.com/ajax/services/search/web?v=1.0")
    if err != nil {
        return nil, err
    }
    q := u.Query()
    q.Set("q", query)
    u.RawQuery = q.Encode()

    // Issue the HTTP request and handle the response.
    resp, err := http.Get(u.String())
    if err != nil {
        return nil, err
    }
    defer resp.Body.Close()

The defer statement arranges for resp.Body.Close to run when Search returns.

25

Parse the JSON response into a Go struct

developers.google.com/web-search/docs/#fonje

    var jsonResponse struct {
        ResponseData struct {
            Results []struct {
                TitleNoFormatting, URL string
            }
        }
    }
    if err := json.NewDecoder(resp.Body).Decode(&jsonResponse); err != nil {
        return nil, err
    }

    // Extract the Results from jsonResponse and return them.
    var results []Result
    for _, r := range jsonResponse.ResponseData.Results {
        results = append(results, Result{Title: r.TitleNoFormatting, URL: r.URL})
    }
    return results, nil
}
26

That's it for the frontend

All the packages are from the standard library: 

import (
    "encoding/json"
    "fmt"
    "html/template"
    "log"
    "net/http"
    "net/url"
    "time"
)

Go servers scale well: each request runs in its own goroutine.

Let's talk about concurrency.

27

Communicating Sequential Processes (Hoare, 1978)

Concurrent programs are structured as independent processes that
execute sequentially and communicate by passing messages.

Sequential execution is easy to understand. Async callbacks are not.

"Don't communicate by sharing memory, share memory by communicating."

Go primitives: goroutines, channels, and the select statement.

28

Goroutines

Goroutines are like lightweight threads.

They start with tiny stacks and resize as needed.

Go programs can have hundreds of thousands of them.

Start a goroutine using the go statement: 

go f(args)

The Go runtime schedules goroutines onto OS threads.

Blocked goroutines don't use a thread.

29

Channels

Channels provide communication between goroutines. 

c := make(chan string)

// goroutine 1
c <- "hello!"

// goroutine 2
s := <-c
fmt.Println(s) // "hello!"
30

Select

A select statement blocks until communication can proceed. 

select {
case n := <-in:
  fmt.Println("received", n)
case out <- v:
  fmt.Println("sent", v)
}

Only the selected case runs.

31

Example: Google Search (backend)

Q: What does Google search do?

A: Given a query, return a page of search results (and some ads).

Q: How do we get the search results?

A: Send the query to Web search, Image search, YouTube, Maps, News, etc., then mix the results.

How do we implement this?

32

Google Search: A fake framework

We can simulate a Search function with a random timeout up to 100ms. 

var (
    Web   = fakeSearch("web")
    Image = fakeSearch("image")
    Video = fakeSearch("video")
)

type Search func(query string) Result

func fakeSearch(kind string) Search {
    return func(query string) Result {
        time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)
        return Result(fmt.Sprintf("%s result for %q\n", kind, query))
    }
}
33

Google Search: Test the framework

// +build ignore,OMIT

package main

import (
	"fmt"
	"math/rand"
	"time"
)

type Result string

// START1 OMIT
func Google(query string) (results []Result) {
	results = append(results, Web(query))
	results = append(results, Image(query))
	results = append(results, Video(query))
	return
}

// STOP1 OMIT

// START2 OMIT
var (
	Web   = fakeSearch("web")
	Image = fakeSearch("image")
	Video = fakeSearch("video")
)

type Search func(query string) Result // HL

func fakeSearch(kind string) Search {
	return func(query string) Result {
		time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)
		return Result(fmt.Sprintf("%s result for %q\n", kind, query))
	}
}

// STOP2 OMIT

func init() {
	rand.Seed(time.Now().UnixNano())
}


func main() {
    start := time.Now()
    results := Google("golang")
    elapsed := time.Since(start)
    fmt.Println(results)
    fmt.Println(elapsed)
}
34

Google Search (serial)

The Google function takes a query and returns a slice of Results (which are just strings).

Google invokes Web, Image, and Video searches serially, appending them to the results slice. 

// +build ignore,OMIT

package main

import (
	"fmt"
	"math/rand"
	"time"
)

type Result string


func Google(query string) (results []Result) {
    results = append(results, Web(query))
    results = append(results, Image(query))
    results = append(results, Video(query))
    return
}


// START2 OMIT
var (
	Web   = fakeSearch("web")
	Image = fakeSearch("image")
	Video = fakeSearch("video")
)

type Search func(query string) Result // HL

func fakeSearch(kind string) Search {
	return func(query string) Result {
		time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)
		return Result(fmt.Sprintf("%s result for %q\n", kind, query))
	}
}

// STOP2 OMIT

func init() {
	rand.Seed(time.Now().UnixNano())
}

func main() {
	start := time.Now()
	results := Google("golang") // HL
	elapsed := time.Since(start)
	fmt.Println(results)
	fmt.Println(elapsed)
}
35

Google Search (parallel)

Run the Web, Image, and Video searches concurrently, and wait for all results.

The func literals are closures over query and c. 

// +build ignore,OMIT

package main

import (
	"fmt"
	"math/rand"
	"time"
)

type Result string
type Search func(query string) Result

var (
	Web   = fakeSearch("web")
	Image = fakeSearch("image")
	Video = fakeSearch("video")
)


func Google(query string) (results []Result) {
    c := make(chan Result)
    go func() { c <- Web(query) }()
    go func() { c <- Image(query) }()
    go func() { c <- Video(query) }()

    for i := 0; i < 3; i++ {
        result := <-c
        results = append(results, result)
    }
    return
}


func fakeSearch(kind string) Search {
	return func(query string) Result {
		time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)
		return Result(fmt.Sprintf("%s result for %q\n", kind, query))
	}
}

func init() {
	rand.Seed(time.Now().UnixNano())
}

func main() {
	start := time.Now()
	results := Google("golang")
	elapsed := time.Since(start)
	fmt.Println(results)
	fmt.Println(elapsed)
}
36

Google Search (timeout)

Don't wait for slow servers.

No locks. No condition variables. No callbacks. 

// +build ignore,OMIT

package main

import (
	"fmt"
	"math/rand"
	"time"
)

type Result string
type Search func(query string) Result

var (
	Web   = fakeSearch("web")
	Image = fakeSearch("image")
	Video = fakeSearch("video")
)

func Google(query string) (results []Result) {

    c := make(chan Result, 3)
    go func() { c <- Web(query) }()
    go func() { c <- Image(query) }()
    go func() { c <- Video(query) }()

    timeout := time.After(80 * time.Millisecond)
    for i := 0; i < 3; i++ {
        select {
        case result := <-c:
            results = append(results, result)
        case <-timeout:
            fmt.Println("timed out")
            return
        }
    }
    return

}

func fakeSearch(kind string) Search {
	return func(query string) Result {
		time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)
		return Result(fmt.Sprintf("%s result for %q\n", kind, query))
	}
}

func init() {
	rand.Seed(time.Now().UnixNano())
}

func main() {
	start := time.Now()
	results := Google("golang")
	elapsed := time.Since(start)
	fmt.Println(results)
	fmt.Println(elapsed)
}
37

Avoid timeout

Q: How do we avoid discarding results from slow servers?

A: Replicate the servers. Send requests to multiple replicas, and use the first response. 

func First(query string, replicas ...Search) Result {
    c := make(chan Result, len(replicas))
    searchReplica := func(i int) { c <- replicas[i](query) }
    for i := range replicas {
        go searchReplica(i)
    }
    return <-c
}
38

Using the First function

// +build ignore,OMIT

package main

import (
	"fmt"
	"math/rand"
	"time"
)

type Result string
type Search func(query string) Result

// START1 OMIT
func First(query string, replicas ...Search) Result {
	c := make(chan Result, len(replicas))
	searchReplica := func(i int) { c <- replicas[i](query) }
	for i := range replicas {
		go searchReplica(i)
	}
	return <-c
}

// STOP1 OMIT

func init() {
	rand.Seed(time.Now().UnixNano())
}


func main() {
    start := time.Now()
    result := First("golang",
        fakeSearch("replica 1"),
        fakeSearch("replica 2"))
    elapsed := time.Since(start)
    fmt.Println(result)
    fmt.Println(elapsed)
}


func fakeSearch(kind string) Search {
	return func(query string) Result {
		time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)
		return Result(fmt.Sprintf("%s result for %q\n", kind, query))
	}
}
39

Google Search (replicated)

Reduce tail latency using replicated search servers. 

// +build ignore,OMIT

package main

import (
	"fmt"
	"math/rand"
	"time"
)

type Result string
type Search func(query string) Result

var (
	Web1   = fakeSearch("web1")
	Web2   = fakeSearch("web2")
	Image1 = fakeSearch("image1")
	Image2 = fakeSearch("image2")
	Video1 = fakeSearch("video1")
	Video2 = fakeSearch("video2")
)

func Google(query string) (results []Result) {

    c := make(chan Result, 3)
    go func() { c <- First(query, Web1, Web2) }()
    go func() { c <- First(query, Image1, Image2) }()
    go func() { c <- First(query, Video1, Video2) }()
    timeout := time.After(80 * time.Millisecond)
    for i := 0; i < 3; i++ {
        select {
        case result := <-c:
            results = append(results, result)
        case <-timeout:
            fmt.Println("timed out")
            return
        }
    }
    return

}

func First(query string, replicas ...Search) Result {
	c := make(chan Result, len(replicas))
	searchReplica := func(i int) {
		c <- replicas[i](query)
	}
	for i := range replicas {
		go searchReplica(i)
	}
	return <-c
}

func fakeSearch(kind string) Search {
	return func(query string) Result {
		time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)
		return Result(fmt.Sprintf("%s result for %q\n", kind, query))
	}
}

func init() {
	rand.Seed(time.Now().UnixNano())
}

func main() {
	start := time.Now()
	results := Google("golang")
	elapsed := time.Since(start)
	fmt.Println(results)
	fmt.Println(elapsed)
}
40

And still…

No locks. No condition variables. No callbacks.

41

Summary

In just a few simple transformations we used Go's concurrency primitives to convert a

program into one that is

42

Tools

43

Go has great tools

The language is designed for tooling.

44

gofmt and goimports

Gofmt formats code automatically. No options.

Goimports updates import statements based on your workspace.

Most people run these tools on save.

go.dev/play/p/GPqra77cBK

45

The go tool

The go tool builds Go programs from source in a conventional directory layout.
No Makefiles or other configs.

Fetch the present tool and its dependencies, build it, and install it: 

% go get golang.org/x/tools/cmd/present

Run it: 

% present
46

godoc

Generated documentation for the world's open-source Go code:

pkg.go.dev

47

IDE and editor support

Eclipse, IntelliJ, emacs, vim, many others.

There's no "Go IDE".

Go tools meet you where you are.

48

Where to Go next

Take the Go Tour online.

go.dev/tour

Lots more material.

go.dev/wiki/Learn

Great community.

go.dev/project

49

Thank you

Sameer Ajmani

Tech Lead Manager, Go team

Google

@Sajma

sameer@golang.org

Use the left and right arrow keys or click the left and right edges of the page to navigate between slides.
(Press 'H' or navigate to hide this message.)