JSON, interfaces, and go generate

Francesc Campoy

Developer, Advocate, and Gopher

Your mission

Your mission, should you choose to accept it, is to decode this message: 

{
    "name": "Gopher",
    "birthdate": "2009/11/10",
    "shirt-size": "XS"
}

into: 

type Person struct {
    Name string
    Born time.Time
    Size ShirtSize
}
2

Your mission (cont.)

Where ShirtSize is an enum (1): 

type ShirtSize byte

const (
    NA ShirtSize = iota
    XS
    S
    M
    L
    XL
)

(1): Go doesn't have enums.
In this talk I will refer to constants of integer types as enums.

3

Using a map

4

Using a map

Pros: very simple

Cons: too simple? we have to write extra code 

func (p *Person) Parse(s string) error {
    fields := map[string]string{}

    dec := json.NewDecoder(strings.NewReader(s))
    if err := dec.Decode(&fields); err != nil {
        return fmt.Errorf("decode person: %v", err)
    }

    // Once decoded we can access the fields by name.
    p.Name = fields["name"]
5

Parsing dates

Time format based on a "magic" date: 

Mon Jan 2 15:04:05 -0700 MST 2006

An example: 

// +build ignore,OMIT

package main

import (
	"fmt"
	"time"
)


func main() {
    now := time.Now()
    fmt.Printf("Standard format: %v\n", now)
    fmt.Printf("American format: %v\n", now.Format("Jan 2 2006"))
    fmt.Printf("European format: %v\n", now.Format("02/01/2006"))
    fmt.Printf("Chinese format: %v\n", now.Format("2006/01/02"))
}
6

Why that date?

Let's reorder: 

Mon Jan 2 15:04:05 -0700 MST 2006

into: 

01/02 03:04:05 PM 2006 -07:00 MST

which is:

7

1 2 3 4 5 6 7!

8

Parsing the birth date:

Since our input was: 

{
    "name": "Gopher",
    "birthdate": "2009/11/10",
    "shirt-size": "XS"
}

Parse the birth date: 

    born, err := time.Parse("2006/01/02", fields["birthdate"])
    if err != nil {
        return fmt.Errorf("invalid date: %v", err)
    }
    p.Born = born
9

Parsing the shirt size

Many ways of writing this, this is a pretty bad one: 

func ParseShirtSize(s string) (ShirtSize, error) {
    sizes := map[string]ShirtSize{"XS": XS, "S": S, "M": M, "L": L, "XL": XL}
    ss, ok := sizes[s]
    if !ok {
        return NA, fmt.Errorf("invalid ShirtSize %q", s)
    }
    return ss, nil
}

Use a switch statement, but a map is more compact.

10

Parsing the shirt size

Our complete parsing function: 

func (p *Person) Parse(s string) error {
    fields := map[string]string{}

    dec := json.NewDecoder(strings.NewReader(s))
    if err := dec.Decode(&fields); err != nil {
        return fmt.Errorf("decode person: %v", err)
    }

    // Once decoded we can access the fields by name.
    p.Name = fields["name"]

    born, err := time.Parse("2006/01/02", fields["birthdate"])
    if err != nil {
        return fmt.Errorf("invalid date: %v", err)
    }
    p.Born = born

    p.Size, err = ParseShirtSize(fields["shirt-size"])
    return err
}
11

Does this work?

// +build ignore,OMIT

package main

import (
	"encoding/json"
	"fmt"
	"log"
	"strings"
	"time"
)

const input = `
{
    "name": "Gopher",
    "birthdate": "2009/11/10",
    "shirt-size": "XS"
}
`

type Person struct {
	Name string
	Born time.Time
	Size ShirtSize
}

type ShirtSize byte

const (
	NA ShirtSize = iota
	XS
	S
	M
	L
	XL
)

func (ss ShirtSize) String() string {
	sizes := map[ShirtSize]string{XS: "XS", S: "S", M: "M", L: "L", XL: "XL"}
	s, ok := sizes[ss]
	if !ok {
		return "invalid ShirtSize"
	}
	return s
}

func ParseShirtSize(s string) (ShirtSize, error) {
	sizes := map[string]ShirtSize{"XS": XS, "S": S, "M": M, "L": L, "XL": XL}
	ss, ok := sizes[s]
	if !ok {
		return NA, fmt.Errorf("invalid ShirtSize %q", s)
	}
	return ss, nil
}

func (p *Person) Parse(s string) error {
	fields := map[string]string{}

	dec := json.NewDecoder(strings.NewReader(s))
	if err := dec.Decode(&fields); err != nil {
		return fmt.Errorf("decode person: %v", err)
	}

	// Once decoded we can access the fields by name.
	p.Name = fields["name"]

	born, err := time.Parse("2006/01/02", fields["birthdate"])
	if err != nil {
		return fmt.Errorf("invalid date: %v", err)
	}
	p.Born = born

	p.Size, err = ParseShirtSize(fields["shirt-size"])
	return err
}


func main() {
    var p Person
    if err := p.Parse(input); err != nil {
        log.Fatalf("parse person: %v", err)
    }
    fmt.Println(p)
}

Note: ShirtSize is a fmt.Stringer

12

JSON decoding into structs

13

JSON decoding into structs

Use tags to adapt field names: 

type Person struct {
    Name string    `json:"name"`
    Born time.Time `json:"birthdate"`
    Size ShirtSize `json:"shirt-size"`
}

But this doesn't fit: 

// +build ignore,OMIT

package main

import (
	"encoding/json"
	"fmt"
	"log"
	"strings"
	"time"
)

const input = `
    {
        "name":"Gopher",
        "birthdate": "2009/11/10",
        "shirt-size": "XS"
    }
    `

type Person struct {
	Name string    `json:"name"`
	Born time.Time `json:"birthdate"`
	Size ShirtSize `json:"shirt-size"`
}

type ShirtSize byte

const (
	NA ShirtSize = iota
	XS
	S
	M
	L
	XL
)

func (ss ShirtSize) String() string {
	s, ok := map[ShirtSize]string{XS: "XS", S: "S", M: "M", L: "L", XL: "XL"}[ss]
	if !ok {
		return "invalid ShirtSize"
	}
	return s
}


func main() {
    var p Person
    dec := json.NewDecoder(strings.NewReader(input))
    if err := dec.Decode(&p); err != nil {
        log.Fatalf("parse person: %v", err)
    }
    fmt.Println(p)
}
14

Let's use an auxiliary struct type

Use string fields and do any decoding manually afterwards. 

    var aux struct {
        Name string
        Born string `json:"birthdate"`
        Size string `json:"shirt-size"`
    }

Note: the field tag for Name is not needed; the JSON decoder performs a case
insensitive match if the exact form is not found.

15

Let's use an auxiliary struct type (cont.)

The rest of the Parse function doesn't change much: 

func (p *Person) Parse(s string) error {
    var aux struct {
        Name string
        Born string `json:"birthdate"`
        Size string `json:"shirt-size"`
    }

    dec := json.NewDecoder(strings.NewReader(s))
    if err := dec.Decode(&aux); err != nil {
        return fmt.Errorf("decode person: %v", err)
    }

    p.Name = aux.Name
    born, err := time.Parse("2006/01/02", aux.Born)
    if err != nil {
        return fmt.Errorf("invalid date: %v", err)
    }
    p.Born = born
    p.Size, err = ParseShirtSize(aux.Size)
    return err
}
16

Can we do better?

17

Current solution

Repetition if other types have fields with:

Let's make the types smarter so json.Decoder will do all the work transparently.

Goal: json.Decoder should do all the work for me!

18

Meet Marshaler and Unmarshaler

Types satisfying json.Marshaler define how to be encoded into json. 

type Marshaler interface {
    MarshalJSON() ([]byte, error)
}

And json.Unmarshaler for the decoding part. 

type Unmarshaler interface {
    UnmarshalJSON([]byte) error
}
19

UnmarshalJSON all the things!

20

Let's make Person a json.Unmarshaler

Replace: 

func (p *Person) Parse(s string) error {

with: 

func (p *Person) UnmarshalJSON(data []byte) error {
    var aux struct {
        Name string
        Born string `json:"birthdate"`
        Size string `json:"shirt-size"`
    }

    dec := json.NewDecoder(bytes.NewReader(data))
    if err := dec.Decode(&aux); err != nil {
        return fmt.Errorf("decode person: %v", err)
    }
    p.Name = aux.Name
    // ... rest of function omitted ...
21

Let's make Person a json.Unmarshaler (cont.)

And our main function becomes: 

// +build ignore,OMIT

package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"log"
	"strings"
	"time"
)

const input = `
{
	"name": "Gopher",
	"birthdate": "2009/11/10",
	"shirt-size": "XS"
}
`

type Person struct {
	Name string
	Born time.Time
	Size ShirtSize
}

type ShirtSize byte

const (
	NA ShirtSize = iota
	XS
	S
	M
	L
	XL
)

func (ss ShirtSize) String() string {
	s, ok := map[ShirtSize]string{XS: "XS", S: "S", M: "M", L: "L", XL: "XL"}[ss]
	if !ok {
		return "invalid ShirtSize"
	}
	return s
}

func ParseShirtSize(s string) (ShirtSize, error) {
	ss, ok := map[string]ShirtSize{"XS": XS, "S": S, "M": M, "L": L, "XL": XL}[s]
	if !ok {
		return NA, fmt.Errorf("invalid ShirtSize %q", s)
	}
	return ss, nil
}

func (p *Person) UnmarshalJSON(data []byte) error {
	var aux struct {
		Name string
		Born string `json:"birthdate"`
		Size string `json:"shirt-size"`
	}

	dec := json.NewDecoder(bytes.NewReader(data)) // HL
	if err := dec.Decode(&aux); err != nil {
		return fmt.Errorf("decode person: %v", err)
	}
	p.Name = aux.Name
	// ... rest of function omitted ...
	born, err := time.Parse("2006/01/02", aux.Born)
	if err != nil {
		return fmt.Errorf("invalid date: %v", err)
	}
	p.Born = born
	p.Size, err = ParseShirtSize(aux.Size)
	return err
}


func main() {
    var p Person
    dec := json.NewDecoder(strings.NewReader(input))
    if err := dec.Decode(&p); err != nil {
        log.Fatalf("parse person: %v", err)
    }
    fmt.Println(p)
}
22

UnmarshalJSON for enums

Substitute ParseShirtSize: 

func ParseShirtSize(s string) (ShirtSize, error) {

with UnmarshalJSON: 

func (ss *ShirtSize) UnmarshalJSON(data []byte) error {
    // Extract the string from data.
    var s string
    if err := json.Unmarshal(data, &s); err != nil {
        return fmt.Errorf("shirt-size should be a string, got %s", data)
    }

    // The rest is equivalent to ParseShirtSize.
    got, ok := map[string]ShirtSize{"XS": XS, "S": S, "M": M, "L": L, "XL": XL}[s]
    if !ok {
        return fmt.Errorf("invalid ShirtSize %q", s)
    }
    *ss = got
    return nil
}
23

UnmarshalJSON for enums (cont.)

Now use ShirtSize in the aux struct: 

//go:build ignore && OMIT
// +build ignore,OMIT

package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"log"
	"strings"
	"time"
)

const input = `{
    "name":"Gopher",
    "birthdate": "2009/11/10",
    "shirt-size": "XS"
}`

type Person struct {
	Name string
	Born time.Time
	Size ShirtSize
}

type ShirtSize byte

const (
	NA ShirtSize = iota
	XS
	S
	M
	L
	XL
)

func (ss ShirtSize) String() string {
	s, ok := map[ShirtSize]string{XS: "XS", S: "S", M: "M", L: "L", XL: "XL"}[ss]
	if !ok {
		return "invalid ShirtSize"
	}
	return s
}

func (ss *ShirtSize) UnmarshalJSON(data []byte) error {
	// Extract the string from data.
	var s string
	if err := json.Unmarshal(data, &s); err != nil { // HL
		return fmt.Errorf("shirt-size should be a string, got %s", data)
	}

	// The rest is equivalent to ParseShirtSize.
	got, ok := map[string]ShirtSize{"XS": XS, "S": S, "M": M, "L": L, "XL": XL}[s]
	if !ok {
		return fmt.Errorf("invalid ShirtSize %q", s)
	}
	*ss = got // HL
	return nil
}


func (p *Person) UnmarshalJSON(data []byte) error {
    var aux struct {
        Name string
        Born string    `json:"birthdate"`
        Size ShirtSize `json:"shirt-size"`
    }

    dec := json.NewDecoder(bytes.NewReader(data))
    if err := dec.Decode(&aux); err != nil {
        return fmt.Errorf("decode person: %v", err)
    }
    p.Name = aux.Name
    p.Size = aux.Size
    // ... rest of function omitted ...

	born, err := time.Parse("2006/01/02", aux.Born)
	if err != nil {
		return fmt.Errorf("invalid date: %v", err)
	}
	p.Born = born
	return nil
}

func main() {
	var p Person
	dec := json.NewDecoder(strings.NewReader(input))
	if err := dec.Decode(&p); err != nil {
		log.Fatalf("parse person: %v", err)
	}
	fmt.Println(p)
}

Use the same trick to parse the birthdate.

24

Unmarshaling differently formatted dates

Create a new type Date: 

type Date struct{ time.Time }

And make it a json.Unmarshaler: 

func (d *Date) UnmarshalJSON(data []byte) error {
    var s string
    if err := json.Unmarshal(data, &s); err != nil {
        return fmt.Errorf("birthdate should be a string, got %s", data)
    }
    t, err := time.Parse("2006/01/02", s)
    if err != nil {
        return fmt.Errorf("invalid date: %v", err)
    }
    d.Time = t
    return nil
}
25

Unmarshaling differently formatted dates (cont.)

Now use Date in the aux struct: 

// +build ignore,OMIT

package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"log"
	"strings"
	"time"
)

const input = `{
    "name":"Gopher",
    "birthdate": "2009/11/10",
    "shirt-size": "XS"
}`

type Person struct {
	Name string
	Born Date
	Size ShirtSize
}

type ShirtSize byte

const (
	NA ShirtSize = iota
	XS
	S
	M
	L
	XL
)

func (ss ShirtSize) String() string {
	s, ok := map[ShirtSize]string{XS: "XS", S: "S", M: "M", L: "L", XL: "XL"}[ss]
	if !ok {
		return "invalid ShirtSize"
	}
	return s
}

func (ss *ShirtSize) UnmarshalJSON(data []byte) error {
	var s string
	if err := json.Unmarshal(data, &s); err != nil {
		return fmt.Errorf("shirt-size should be a string, got %s", data)
	}
	got, ok := map[string]ShirtSize{"XS": XS, "S": S, "M": M, "L": L, "XL": XL}[s]
	if !ok {
		return fmt.Errorf("invalid ShirtSize %q", s)
	}
	*ss = got
	return nil
}

type Date struct{ time.Time }

func (d Date) String() string { return d.Format("2006/01/02") }

func (d *Date) UnmarshalJSON(data []byte) error {
	var s string
	if err := json.Unmarshal(data, &s); err != nil {
		return fmt.Errorf("birthdate should be a string, got %s", data)
	}
	t, err := time.Parse("2006/01/02", s) // HL
	if err != nil {
		return fmt.Errorf("invalid date: %v", err)
	}
	d.Time = t
	return nil
}


func (p *Person) UnmarshalJSON(data []byte) error {
    r := bytes.NewReader(data)
    var aux struct {
        Name string
        Born Date      `json:"birthdate"`
        Size ShirtSize `json:"shirt-size"`
    }
    if err := json.NewDecoder(r).Decode(&aux); err != nil {
        return fmt.Errorf("decode person: %v", err)
    }
    p.Name = aux.Name
    p.Size = aux.Size
    p.Born = aux.Born
    return nil
}


func main() {
	var p Person
	dec := json.NewDecoder(strings.NewReader(input))
	if err := dec.Decode(&p); err != nil {
		log.Fatalf("parse person: %v", err)
	}
	fmt.Println(p)
}

Can this code be shorter?

26

Yes!

By making the Born field in Person of type Date.

Person.UnmarshalJSON is then equivalent to the default behavior!

It can be safely removed. 

// +build ignore,OMIT

package main

import (
	"encoding/json"
	"fmt"
	"log"
	"strings"
	"time"
)

const input = `{
    "name":"Gopher",
    "birthdate": "2009/11/10",
    "shirt-size": "XS"
}`

type Person struct {
	Name string    `json:"name"`
	Born Date      `json:"birthdate"`
	Size ShirtSize `json:"shirt-size"`
}

type ShirtSize byte

const (
	NA ShirtSize = iota
	XS
	S
	M
	L
	XL
)

func (ss ShirtSize) String() string {
	s, ok := map[ShirtSize]string{XS: "XS", S: "S", M: "M", L: "L", XL: "XL"}[ss]
	if !ok {
		return "invalid ShirtSize"
	}
	return s
}

func (ss *ShirtSize) UnmarshalJSON(data []byte) error {
	var s string
	if err := json.Unmarshal(data, &s); err != nil {
		return fmt.Errorf("shirt-size should be a string, got %s", data)
	}
	got, ok := map[string]ShirtSize{"XS": XS, "S": S, "M": M, "L": L, "XL": XL}[s]
	if !ok {
		return fmt.Errorf("invalid ShirtSize %q", s)
	}
	*ss = got
	return nil
}

type Date struct{ time.Time }

func (d Date) String() string { return d.Format("2006/01/02") }

func (d *Date) UnmarshalJSON(data []byte) error {
	var s string
	if err := json.Unmarshal(data, &s); err != nil {
		return fmt.Errorf("birthdate should be a string, got %s", data)
	}
	t, err := time.Parse("2006/01/02", s)
	if err != nil {
		return fmt.Errorf("invalid date: %v", err)
	}
	d.Time = t
	return nil
}


func main() {
    var p Person
    dec := json.NewDecoder(strings.NewReader(input))
    if err := dec.Decode(&p); err != nil {
        log.Fatalf("parse person: %v", err)
    }
    fmt.Println(p)
}
27

Was this really better?

28

Other ideas

29

Roman numerals

30

Roman numerals

Because why not? 

type romanNumeral int

And because Roman numerals are classier 

type Movie struct {
    Title string
    Year  romanNumeral
}
31

Roman numerals (cont.)

// +build ignore,OMIT

package main

import (
	"encoding/json"
	"fmt"
	"log"
	"strings"
)

type romanNumeral int

var numerals = []struct {
	s string
	v int
}{
	{"M", 1000}, {"CM", 900},
	{"D", 500}, {"CD", 400},
	{"C", 100}, {"XC", 90},
	{"L", 50}, {"XL", 40},
	{"X", 10}, {"IX", 9},
	{"V", 5}, {"IV", 4},
	{"I", 1},
}

func (n romanNumeral) String() string {
	res := ""
	v := int(n)
	for _, num := range numerals {
		res += strings.Repeat(num.s, v/num.v)
		v %= num.v
	}
	return res
}

func parseRomanNumeral(s string) (romanNumeral, error) {
	res := 0
	for _, num := range numerals {
		for strings.HasPrefix(s, num.s) {
			res += num.v
			s = s[len(num.s):]
		}
	}
	return romanNumeral(res), nil
}

func (n romanNumeral) MarshalJSON() ([]byte, error) {
	if n <= 0 {
		return nil, fmt.Errorf("Romans had only natural (=>1) numbers")
	}
	return json.Marshal(n.String())
}

func (n *romanNumeral) UnmarshalJSON(data []byte) error {
	var s string
	if err := json.Unmarshal(data, &s); err != nil {
		return err
	}
	p, err := parseRomanNumeral(s)
	if err == nil {
		*n = p
	}
	return err
}

type Movie struct {
	Title string
	Year  romanNumeral
}


func main() {
    // Encoding
    movies := []Movie{{"E.T.", 1982}, {"The Matrix", 1999}, {"Casablanca", 1942}}
    res, err := json.MarshalIndent(movies, "", "\t")
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("Movies: %s\n", res)

    // Decoding
    var m Movie
    inputText := `{"Title": "Alien", "Year":"MCMLXXIX"}`
    if err := json.NewDecoder(strings.NewReader(inputText)).Decode(&m); err != nil {
        log.Fatal(err)
    }
    fmt.Printf("%s was released in %d\n", m.Title, m.Year)
}
32

Secret data

33

Secret data

Some data is never to be encoded in clear text. 

type Person struct {
    Name string `json:"name"`
    SSN  secret `json:"ssn"`
}

type secret string

Use cryptography to make sure this is safe: 

func (s secret) MarshalJSON() ([]byte, error) {
    m, err := rsa.EncryptOAEP(crypto.SHA512.New(), rand.Reader, key.Public().(*rsa.PublicKey), []byte(s), nil)
    if err != nil {
        return nil, err
    }
    return json.Marshal(base64.StdEncoding.EncodeToString(m))
}

Note: This solution is just a toy; don't use it for real systems.

34

Secret data (cont.)

And use the same key to decode it when it comes back: 

func (s *secret) UnmarshalJSON(data []byte) error {
    var text string
    if err := json.Unmarshal(data, &text); err != nil {
        return fmt.Errorf("deocde secret string: %v", err)
    }
    cypher, err := base64.StdEncoding.DecodeString(text)
    if err != nil {
        return err
    }
    raw, err := rsa.DecryptOAEP(crypto.SHA512.New(), rand.Reader, key, cypher, nil)
    if err == nil {
        *s = secret(raw)
    }
    return err
}
35

Secret data (cont.)

Let's try it: 

// +build ignore,OMIT

package main

import (
	"crypto"
	"crypto/rand"
	"crypto/rsa"
	_ "crypto/sha512"
	"encoding/base64"
	"encoding/json"
	"fmt"
	"log"
)

var key *rsa.PrivateKey

func init() {
	k, err := rsa.GenerateKey(rand.Reader, 2048)
	if err != nil {
		log.Fatalf("generate key: %v", err)
	}
	key = k
}

type Person struct {
	Name string `json:"name"`
	SSN  secret `json:"ssn"`
}

type secret string

func (s secret) MarshalJSON() ([]byte, error) {
	m, err := rsa.EncryptOAEP(crypto.SHA512.New(), rand.Reader, key.Public().(*rsa.PublicKey), []byte(s), nil)
	if err != nil {
		return nil, err
	}
	return json.Marshal(base64.StdEncoding.EncodeToString(m))
}

func (s *secret) UnmarshalJSON(data []byte) error {
	var text string
	if err := json.Unmarshal(data, &text); err != nil { // HL
		return fmt.Errorf("deocde secret string: %v", err)
	}
	cypher, err := base64.StdEncoding.DecodeString(text) // HL
	if err != nil {
		return err
	}
	raw, err := rsa.DecryptOAEP(crypto.SHA512.New(), rand.Reader, key, cypher, nil) // HL
	if err == nil {
		*s = secret(raw)
	}
	return err
}


func main() {
    p := Person{
        Name: "Francesc",
        SSN:  "123456789",
    }

    b, err := json.MarshalIndent(p, "", "\t")
    if err != nil {
        log.Fatalf("Encode person: %v", err)
    }
    fmt.Printf("%s\n", b)

    var d Person
    if err := json.Unmarshal(b, &d); err != nil {
        log.Fatalf("Decode person: %v", err)
    }
    fmt.Println(d)
}
36

But most JSON enums are boring

37

go generate to the rescue!

go generate:

You will see it as comments in the code like: 

//go:generate go tool yacc -o gopher.go -p parser gopher.y

More information in the blog post.

38

code generation tools: stringer

stringer generates String methods for enum types. 

package painkiller

//go:generate stringer -type=Pill

type Pill int

const (
    Placebo Pill = iota
    Aspirin
    Ibuprofen
    Paracetamol
)

Call go generate: 

$ go generate $GOPATH/src/path_to_painkiller

which will create a new file containing the String definition for Pill.

39

jsonenums

Around 200 lines of code.

Parses and analyses a package using:

And generates the code using:

And it's on github: github.com/campoy/jsonenums

40

Demo

41

Thank you

Francesc Campoy

Developer, Advocate, and Gopher

@francesc

campoy@golang.org

Use the left and right arrow keys or click the left and right edges of the page to navigate between slides.
(Press 'H' or navigate to hide this message.)