For all future comments about options, continue the discussion here: #63397 (comment)

A correspondence with JSON Merge Patch is not always achievable and is also not desirable for JSON null.

Correct. That is why our semantic "takes inspiration from JSON Merge Patch (RFC 7386)". It is not defined as being exactly RFC 7286. We want some type of semantic that is easy to explain and implement towards. The current behavior is somewhat incomprehensible.

In the case of JSON Merge Patch, JSON null is given special meaning to make it possible to remove a name/value pair when patching. However, when unmarshalling, allowing JSON null to set a value to its zero value doesn't provide any new capabilities beyond what's already possible.

I find that the behaviors of jsonv1 and jsonv2 regarding JSON null are the least expected.

It would be just as easy to explain and implement if JSON null were only allowed if Go nil can be assigned to the value. If it can't, an error would be returned.

I'm not exactly following what you're proposing.

Adhering strictly to JSON Merge Patch semantics isn't possible since most Go types do not have presence bits. There is simply no way in many native data structures to indicate that a Go struct field, for example, has been deleted.

Were you instead proposing that null simply be rejected as a valid input in general? Would that also include Go pointers and interfaces where there is an obvious meaning of how a JSON null maps to the Go type?

I propose accepting JSON null as valid input only for slices, maps, interfaces, and pointers (i.e., where Go nil can be assigned to the type). For all other Go types, JSON null is considered invalid and unmarshal fails with a SemanticError.

For all other Go types, JSON null is considered invalid and unmarshal fails with a SemanticError

Whether a JSON document (or value) is valid or not is defined outside of the Go type system. What you are proposing is defining another format practically: GO-JSON but definitly not JSON

@dsnet
For objects, JSON keys are defined to be of type string and cannot be of type integer; your latter example is correct. Thankfully, from an idiomatic perspective, using integer-as-string for JSON keys is likely bad practice. While technically possible, it should be avoided.

The correct canonicalization is:

{"0":null,"300":null,"500":null,"70":null,"9":null,"90":null}

@zamicol, may you kindly stop bringing RFC 8785 into the discussion? Thank you. This thread is about the ordering of Go maps under json.Deterministic. The only relevance of RFC 8785 is for inspiration to the latter, but criticisms about RFC 8785 or the implementation is out of scope here (please start a new discussion as asked for above).

Let's start with arrays. The JSON RFC states that arrays may have mixed types.

I think "mixed types" is being confused for "mixed ordering" or "unordered".
Deterministic ordering must not affect the ordering of JSON arrays, which are "an ordered sequence of zero or more values."

Now let's examine deduplication: deduplication isn't a concern for arrays, but it is for objects. Escapes complicate matters.

Escape sequences are a red herring.

In the case of map[int]string, there is a potential use of ordering based on int. There's no escaping here. (BTW, I'm not saying it's a good idea to sort on numerical ordering or that use of map[int]string is even a good idea. Use of it already exists in v1, and I'm simply raising the question of whether we should sort it specially or not.)

In the case on non-numeric keys, sorting is based on Unicode codepoints of the resulting string (regardless of whether there are any escapes). This can be done based on UTF-8 or UTF-16 (though prior discussion seems to favor UTF-8).

arrays, which are "an ordered sequence of zero or more values."

Totally slipped my mind! That's correct.

stop bringing RFC

I'm not understanding, please accept my apology. I will move all discussion regarding the Canonicalize method to a new discussion.

Thanks @zamicol! For those who want to discuss canonicalization further, see #63397 (comment).

As long as we keep the deterministic ordering as an option (I like the contents of my json logs to be ordered alphabetically, after the time, level, and msg), I don't care what the default is and am good with the more performant non-deterministic ordering as default.

@willfaught, by default the only way to get a JSON null is through a pointer or interface. Non-default ways are as follows:

• Specifying FormatNilMapAsNull(true) or FormatNilMapAsNull(true) at the Marshal call
• Specifying a format:emitnull tag option on Go struct field for a slice or map
• Implementing a type-specified MarshalJSON or MarshalJSONV2 method that encodes a JSON null
• Implementing a caller-specified MarshalFuncV1 or MarshalFuncV2 function that encodes a JSON null

In general, we place control in both the hands of the authors of types and the callers of Marshal.

Also, if you're marshalling a field whose type is an interface then you're kind of already saying that this field can be literally any type to begin with, so trying to adhere to a schema for that field seems kinda silly.

It's also possible to just make the value be what you want to say.

the v2 default behaviour is more frendly to consumers. Json gets consumed most of time by clients in other languages and mostly they expect an empty array especially most consumers in JS or python world where working directly on lists is a common idiom and they expect things like arrayItem.forEach to work by default or at least only having to check for the presence or not of the property (ommited or not) and not also it's nullness.

Retracted: Would be more suitable to discuss on the #63397 (comment) topic. Also omitzero struct tag appears to handle this.

There are a few cases where at least the difference between an empty array and omitted really matters. E.g. consider the following two MongoDB queries:

• {"id": {"$in": []} : Matches no items.
• {"id": {}}: Matches all items.

So even if encoding a nil array as an empty array by default, I do not think that an empty array should be omitted by default:

type Comarison struct{
    In []any `json:"$in,omitempty"` // Should only treat `nil` as omitted.
}

Would it be helpful to have a way to express the three possible states of omitted, null and T as Go types? This would be especially useful when ingesting fields that convey different information depending on those states.

For example, consider a hypothetical Omittable[T] type. We could then define a simple update type:

type EventUpdate struct {
  // OtherID is the event A's other ID.
  // If omitted, then the field is unchanged.
  // If null, then the field has been reset to null.
  // If ID, then the field's value has been updated.
  OtherID json.Omittable[*ID] `json:"other_id"`
}

Users could then explicitly specify whether they want the OtherID field:

// update event that makes no change to OtherID
EventUpdate{}
// an update event that sets OtherID to something else
EventUpdate{OtherID: json.Present(ptr(123))}
// an update event that resets OtherID to null
EventUpdate{OtherID: json.Present(null)}

or detect if a field is present or not:

var update EventUpdate
json.Unmarshal(b, &update)

if !update.OtherID.IsZero() {
  // Add into a SQL UPDATE statement.
  q = append(q, "other_id = ?, ")
  v = append(v, update.OtherID.Value())
}

Implementing an Omittable[T] type would be relatively easy with the v2 API:

// Ommitable marks that a value of its type is either omitted (not in JSON at all)
// or is present and is of type T. NOTE: the user MUST use this with `,omitzero`!
type Omittable[T] struct {
  value T
  valid bool
}

// IsZero returns true if the field should be omitted.
func (o Omittable[T]) IsZero() bool { return o.valid }

// Value returns the present value. If none, then its zero value is returned.
func (o Omittable[T]) Value() T { return o.value }

// Present returns v wrapped in a new Omittable[T] that will not be omitted.
func Present[T any](v T) Omittable[T] {
  return Omittable[T]{v, true}
}

One downside of this implementation is that the user still has to add ,omitzero manually. If the ShouldOmit() function is implemented (#63397 (comment)), then this implementation could be slightly easier.

I'm also not sure if json.Present is a good name. A better name should probably be used.

The main question is: should this use case be covered by an additional type? Or should users write their own type boilerplate when they need to?

I'd like to think about optional/omittable/nullable generic types more broadly across std, like #48702 - there is little here that would be JSON-specific. And if such a generic type were to appear later on, I imagine we could add support for it in encoding/json/v2 in a backwards compatible way.

It's worth noting that #60370 was added, but I'd prefer to not simply follow that example and end up with very similar types across a handful of std packages.

I'd like to think about optional/omittable/nullable generic types more broadly across std

I definitely agree! I think in an ideal world, this would be the best solution.

I only suggested a type in encoding/json/v2 because I wasn't sure how practical it would be to add this into the stdlib, given that the proposal itself is for encoding/json/v2.

This remark echoes @bradfitz's regrets about oversharing types between client and server in his design of net/http.

It's unclear to me the benefit of this.

Dead code elimination (DCE) has gotten much better in recent years such that the cost of importing a package (but not using it) has dropped significantly. Thus, splitting this out doesn't seem to benefit binary bloat. That said, the prototype v2 implementation generates the marshal/unmarshal functions together, so depending on only one would link in the implementation for both (today). But that's an implementation detail; we can modify the code to assist in DCE.

The other benefit of splitting it out is to have a conceptual separation between marshal/unmarshal, but doing some brings a few detriments as well:

• As it stands, having both in the "json" package makes the call site unambiguous. It's clear what json.Marshal and json.Unmarshal do. However, encode.Marshal and decode.Unmarshal is less clear unless we go with jsonencode.Marshal and jsondecode.Unmarshal, but that stutters quite a bit.
  • We could make it non-idiomatic and rely on the package name as a verb and have it be encode.JSON or decode.JSON, but we'd be making use of a potentially contended namespace for "encode" and "decode". How would we extend this same pattern to XML or YAML? It still bothers me that I can't tell whether rand.Read refers to "math/rand" or "crypto/rand".
• json.Marshal is almost the entire opposite of json.Unmarshal. The "json" package provides a unified place to describe that mapping. I'd argue that there's closer correlation between Marshal and Unmarshal than there is between http.Client and http.Server.

We could also write a regression test that would build a tiny Go program only using json.Marshal and then check that json.Unmarshal and the other decoding functions and types aren't included in the binary - and vice versa. This would keep us honest and ensure that we don't make any changes in the future that could tie the encoder and decoder implementations together.

"It still bothers me that I can't tell whether rand.Read refers to "math/rand" or "crypto/rand".

This. Very much this.

Alternatively, carrying forth @prashantv's idea...

These are the current set of declarations with V1 and V2 suffixes:

• DefaultOptionsV2
• MarshalerV1
• MarshalerV2
• MarshalerV2.MarshalJSONV2
• UnmarshalerV1
• UnmarshalerV2
• UnmarshalerV2.UnmarshalJSONV2
• MarshalFuncV1
• MarshalFuncV2
• UnmarshalFuncV1
• UnmarshalFuncV2

DefaultOptionsV2 quite literally describes v2 semantics, so I think it's fine and actually more explicit that it carries a v2 suffix. Also, this option would only ever be used when you're dealing with a v1-to-v2 migration. If you're in a pure-v2 world, then you'll never need to reference this.

For the others, they could potentially be renamed as:

• MarshalerV1 -> Marshaler
• MarshalerV2 -> MarshalerTo
• MarshalerV2.MarshalJSONV2 -> MarshalerTo.MarshalJSONTo
• UnmarshalerV1 -> Unmarshaler
• UnmarshalerV2 -> UnmarshalerFrom
• UnmarshalerV2.UnmarshalJSONV2 -> UnmarshalerFrom.UnmarshalJSONFrom
• MarshalFuncV1 -> MarshalFunc
• MarshalFuncV2 -> MarshalToFunc
• UnmarshalFuncV1 -> UnmarshalFunc
• UnmarshalFuncV2 -> UnmarshalFromFunc

However, this introduces a discrepancy between the above and the existing set of top-level functions:

• Marshal
• MarshalWrite
• MarshalEncode
• Unmarshal
• UnmarshalRead
• UnmarshalDecode

We can't rename both MarshalWrite and MarshalEncode to MarshalTo,
nor can we rename both UnmarshalRead and UnmarshalDecode to UnmarshalFrom.

To be honest, I personally don't think it's problematic for MarshalerV1, MarshalerV2, UnmarshalerV2, and UnmarshalerV2 to carry the V1 and V2 suffixes since they're interface types and are rarely referenced in user code. I can see why there's a distaste to having V2 in the method name, though, since this appears in all user-code implementing custom JSON.

"I personally don't think it's problematic for MarshalerV1, MarshalerV2, UnmarshalerV2, and UnmarshalerV2 to carry the V1 and V2 suffixes since they're interface types and are rarely referenced in user code."

Agreed.

"I can see why there's a distaste to having V2 in the method name, though, since this appears in all user-code implementing custom JSON."

Confirming your perception on this, at least for me.

I think it's more important for the names to be clear then for them to order well.

As a curiosity though, If we go for FormatJSON and ParseJSON, then FormatJSON would order before MarshalJSON and ParseJSON would order before UnmarshalJSON. Same goes with Parser v.s. Marshaller and Formatter v.s. Unmarshaller. so the new method would at least order before the "old" methods, if not directly above.

As another alternative, the V1 interface could be implemented either as a private type or as an implicit interface similar to As and Is in the error package. This way old names would not show up in the docs at all, and the docs could just document that v1 interfaces are detected without publicly redeclaring them.

I like @prashantv suggestion, but could we re-order the words so it is MarshalToJSON and UnmarshalFromJSON? I know the ordering would be a bit off, but I feel like it would be more clear. As a new developer I had to regularly think about which method meant what. Also MarshalJSONTo makes it sound like it is taking JSON input and vice versa for UnmarshalJSONFrom.

To have a type keep its V1 interface whilst also enable forward compatibility with V2 (assuming naming collisions), there could be a bridging interface:

type UnmarshalerV2Supplier interface {
    AsUnmarshalerV2() UnmarshalerV2
}

An existing V1 type such as:

type Foo int

func (a *Foo) UnmarshalJSON(val []byte) error { ... }

Can be extended like so:

type foo2 Foo

func (a *Foo) AsUnmarshalerV2() json.UnmarshalerV2 {
    return (*foo2)(a)
}

func (a *foo2) UnmarshalJSON(dec *jsontext.Decoder, opts json.Options) error {...}

And a type that does not require backwards compatibility of course needs only implement v2 interface directly:

type Bar int

func (a *Bar) UnmarshalJSON(dec *jsontext.Decoder, opts json.Options) error {...}

Could also have a UnmarshalerV1Supplier interface to do the same principle in reverse. That seems less ideal for
backwards compatibility with older Go versions as UnmarshalerV1Supplier would not be understood by the existing json package.

This avoids the suffix and having to invent new names for each new major version. As V1 code is upgraded to the V2
interfaces, it can drop the *Supplier. The ugliness is only retained whilst both versions need to be concurrently supported, i.e. initial migration period.

Not suggesting this is necessarily a good solution, but shows there is viability in overloaded names without losing compatibility.

At this point. I'm going to somewhat disengage from the conversation. Others are free to continue discussing.
I work on Go as an external contributor and don't have the bandwidth to continually engage in long threads of response.
Unlike most other golang discussions, which are hosted by members of the Go team, I'm not employed by Google.
Thank you for your time.

@dsnet — Once again, thank you for taking the time to engage.

Given additional debate would be moot, I accept your position on a hybrid API.

However, please allow developers to build their own Hybrid API; that will only take a tiny bit of code change.

Here is the (updated) PR to that effect, ready to merge:

go-json-experiment/json#17

Since it is an additive approach as you acknowledge, allowing developers to implement their own will allow the Go team to see if there is strong interest in this, or if nobody other than a few like me care enough to write their own hybrid API.

But if developers cannot do it themselves, it will be hard to ever know.

-Mike

P.S. I would not even see the need to document this extensibility. People who want it will find it and use it. And if enough people do want to use it, then we can document it.

Marshaler, Unmarshaler, Encoder, and Decoder structs (not separate option types, the actual types) that contain option fields don't have this issue.

To be clear, you would actually need both a Marshaler and EncodeMarshaler type, where Marshaler has just options related to marshaling and EncodeMarshaler and options related to both marshaling and encoding.

@dsnet I don't follow. The encoder option fields would be mixed in with the marshal option fields in Marshaler, then copied to the Marshaler's Encoder option fields. You only need to deal with one option struct when directly un/marshaling or de/encoding.

Which is why each type has its own fields. If there's some obvious overlap, then fine, make a shared struct, but not at the expense of useless fields in some contexts, as pointed out here.

There is overlap in a number of options. A shared type makes the ergonomics of calling worse. Anything that introduces more references to declarations in general hurts ergonomics.

So then don't do that. Have each type have its own fields.

That sounds less clean and more verbose to me.

The variadic approach is objectively still less verbose:
json.Marshal(v, json.Deterministic(true))
versus
json.Marshaler{Deterministic: true}.Marshal(v)

In that context, if I remember correctly, I was addressing the complexity and verbosity of the API declarations, not call sites.

For those specific call sites, sure, but in practice, you'd never write that. You'd write:

m := json.Marshaler{Deterministic: true}
m.Marshal(v)
...
m.Marshal(v)
...
m.Marshal(v)

Using functional options, you'd have to write:

json.Marshal(v, json.Deterministic(true))
...
json.Marshal(v, json.Deterministic(true))
...
json.Marshal(v, json.Deterministic(true))

Your counterargument might be that you'd instead write:

o := json.Deterministic(true)
json.Marshal(v, o)
...
json.Marshal(v, o)
...
json.Marshal(v, o)

But in that case, it's a wash in terms of verbosity.

However, methods hung off an options struct has an additional problem in that you can't as easily use them in an expression.

I would argue that it's idiomatic to construct a value and then mutate it before use, as in option fields. We can find plenty of examples in the stdlib. Not everything needs to be expression-friendly. I imagine that most uses of options will be to get v1 behavior, and if I recall correctly, that requires more than one option to be changed, so I would expect most instances to be created in this way.

Speaking of idioms, are there any examples of functional options in the stdlib that you can point to? I'm not aware of any.

v2 options should be declared with v2. If v2 has options to behave like v1, then that's fine, that's the design, and v2 should reflect that.

If you don't want v2 to be "polluted" by v1, then don't make it behave like v1. This is trying to have your cake and eat it too, and as I've pointed out in my top-level thread, which you haven't addressed here yet, there are downsides to doing that (as there always are with trying to have your cake and eat it too).

This is just going to be a point that we're going to have to disagree on.

I don't think so. If you can think of a concrete rebuttal, I'm all ears.

Option structs [...] interfere with dead-code analysis.

Can you explain this?

If the *Marshalers type is a field in MarshalOptions, it must be linked in if MarshalOptions is referenced. With the variadic API, it's possible to implement it in such a way that caller-specified marshalers is entirely dead-code eliminated if WithMarshalers is never called.

OK, this sounds like it's about reflect, and not wanting to link that in, unless needed.

concrete binary and memory space costs of importing reflect

Importing reflect cannot be quantified as an X number of bytes of binary bloat because it is not simply a fixed number of instructions that needs to be linked in.

Thanks for the great explanation. It makes sense why we'd want to minimize links to reflect in general. So if jsonv2 needs reflect just to do its basic job, then what option contains a direct reference to reflect that you're trying to avoid if possible? I assume it's a jsontext option? Skimming above, I don't see mention of reflect anywhere in encoding options, and none of the encoding options seem to do reflection-related things.

This sounds more and more like a framework with plugins.

Whether or not it uses dependency injection is purely an implementation detail. The "json" API does not expose this detail. Thus, it is not a "framework".

Well, it does expose the detail in the option itself, which is part of the total API, but I agree that the core API does not. This was a minor point, and I can see if you squint at it right, it's not a framework. Fair enough. I actually like dependency injection.

jsonv1 required reflect, so it's unsurprising that jsonv2 would too

Yes, and that has been constraining on a few use cases for embedded or mobile applications. A flaw of v1 is not an argument for why it should exist in v2.

It might be if we insist on v2 being able to emulate v1 behavior. There are other considerations, like API ergonomics and complexity.

Saving 50KB of RAM while running in iOS with a 150MB RAM limit is not a valid reason, in my humble opinion

It's funny you say that...

I've read that, which is what I was vaguely alluding to when I wrote that. I couldn't find the link at the time, so thanks for sharing. Like I said above:

I'm looking for a concrete scenario, like "we have to simulate the universe in 32 bytes of RAM because of X, Y, and Z." Saving 50KB of RAM while running in iOS with a 150MB RAM limit is not a valid reason, in my humble opinion. If you need to do crazy things or need extreme optimization, don't expect the stdlib to cater to all your needs. Or don't expect backward compatibility with jsonv1.

We should either go with the stdlib catering to all needs, or go with backward compatibility with jsonv1, but not both, because going with both has this complicated API design. Users can just convert their JSON from v1 to v2 instead, and use v1 in the meantime.

I don't follow. Can you explain how slog is currently planned to work with jsonv2?

See #63397 (reply in thread)

This is just sticking a jsonopts.Options field in slog. You could do the same thing with a Marshaler.

If some other package has its own configuration of jsonv2 that it wants users to override

That doesn't work in packages (e.g., "slog") where it manages the marshaling of JSON on your behalf.

You can just pass it a Marshaler. Configure it however you want. If you want to "override" some custom behavior that's done by default, read the documentation to know what that is, then configure the Marshaler appropriately.

type SomeSlogStructSomewhere struct {
    ...
    // Marshaler is...
    // If unset, these options are used:
    // - ...
    // - ...
    // - ...
    Marshaler *json.Marshaler
    ...
}

That's all you need to tailor a Marshaler to exactly what you need for slog.

It's not a loss of type safety, it's a loss of immutability.

Immutability is a sub-category of type safety.

There are languages that have immutable values and no types or type system, so this isn't true on its face. I would say that immutability is a property that is a consequence of the absence of semantics that enable mutation. Type systems can describe values and expressions in such systems, but they aren't the same thing.

This subject will probably go off-topic, so I'd be happy to continue this rabbit hole elsewhere if you'd like. :)

I don't see how. Can you explain?

In an earlier version of the v2 "json" prototype, we exposed an option called:
EscapeRune func(rune) bool
that specifies which runes to escape in a JSON string. (we eventually removed this overly powerful option and replaced it with
EscapeForJS and EscapeForHTML). For performance, we pre-executed the function on all ASCII characters and built a small bit-map of which ASCII characters needed escaping, which vastly improved performance as it avoided a function call for a majority of characters. However, if the EscapeRune option were mutable (or non-deterministic), then the cache would become incorrect.

I don't see how mutability of boolean options relates to this, though. As you said, EscapeRune is gone, precisely because it prevented this optimization. And I don't see how EscapeRune pertains to the option struct vs. function options decision. You can specify EscapeRune either way, so EscapeRune as a functional option would have the same mutability problem.

Since jsontext.Encoder is passed to json.MarshalerV2.MarshalJSONV2, it would be problematic if the implementation of MarshalJSONV2 can change the behavior of encoding by toggling jsontext.Encoder.Indent. It makes no sense to change such parameters partway through encoding.

See "So Don't Do That" above.

I disagree.

Looking back, I misread your original comment. I see what you mean now about other types' implementations of MarshalJSONV2 mutating Encoder. That is indeed a potential problem, one that I don't have an easy answer for right now. Perhaps an Encoder interface used in that context would prevent mutation of the fields. I'm not clear on why Options is required for MarshalJSONV2, but assuming they are, then there's the question of how to supply the option struct too.

Good point!

Note that v1 compatibility has already been discussed elsewhere, like #63397 (comment). I worry that this thread has a big scope and is getting very long, to the point that we're duplicating a number of the other discussions.

@mvdan Unless you can be specific, I don't see an issue here. Functional options touch on compatibility, optimization, and other issues.

Let's not micromanage the partial overlap of comments. Topics naturally evolve and touch on other topics, or refer back to themselves. If you're not getting any more value out of this discussion, then I suggest unsubscribing. This thread, and this discussion in general, have wound down naturally without intervention. I don't think @dsnet plans to post more, and I don't have anything else to add.

Perhaps, I'm not understanding how you plan on extending PeekKind in a hypothetical future. Right now, it only ever returns 9 possible characters:

• 'n' for null
• 'f' for false
• 't' for true
• '"' for string
• '0' for number
• '[' for starting array
• ']' for closing array
• '{' for starting object
• '}' for closing object

It would be an unfortunate hindrance to usages today if all callers of PeekKind need to care about colons and commas. One of the nice properties of JSON over JWCC is that commas and can always be perfectly inferred.

I probably don't have any deep insights here. My general experience has been that many tools using JWCC only care about it as input, and the "obvious" ways to efficiently parse and represent it do not adapt well when you want to write it back out (especially if you also need the ability to edit). I think/agree there is real value in distinguishing the "one way" case from the "two way" case in the API.

Extending the scanner and parser to (efficiently) handle the JWCC extensions is straightforward, but you need quite a bit of API mass to preserve a comment-decorated AST, and if you want tools to be able to add, edit, and remove comments, the memory footprint of that representation is substantial. (One could almost certainly do better than I did in that package, but the problem is not specific to it).

Moreover, even basic JSON API functions like marshal/unmarshal don't map to JWCC very well: For example, there is no nice way to "compactify" JWCC text, without mangling the legibility or throwing away the comments. Any useful round-tripping of JWCC is probably going to need nice indentation, and then we get into the whole "jwcc fmt" issue where the API either has to be opinionated (and invite an unending future of option requests) or add a gallimaufry of knobs.

My intuition is that none of this belongs in a (baseline) JSON package, and that despite its superficial similarity and one-way compatibility with JSON, JWCC as an objet d'arte should be treated as its own thing, and programs that care to round-trip it should use a different API.

One of the nice properties of JSON over JWCC is that commas and [colons??] can always be perfectly inferred.

Perhaps we are misunderstanding each other, but with JWCC, commas and colons can also always be perfectly inferred. Provided that, when writing JWCC, you're writing in some canonical comma style. One option for "canonical comma style" is to simply output commas as per JSON. Another option is to always write commas after an element. Either way, you're writing per an opinionated formatter, the same way that gofmt writes an opinionated format even though, as input, Go's semi-colons are optional. Just like gofmt, it doesn't really matter what the canonical style is, just that there is one.

If you're worried that round-tripping will not produce byte-for-byte identical output, then that's already a problem for JSON (due to whitespace, which your package jsontext discards; see also capital-C RFC 8785 Canonicalization), not just JWCC.

Right now, it only ever returns 9 possible characters:

Well, 10, if you include returning const invalidKind Kind = 0 at EOF. Or maybe more if you want to discriminate EOF from other io.Reader errors. (See also the "negative Kind values mean different errors" idea elsewhere in this issue discussion.)

how you plan on extending PeekKind in a hypothetical future

That nitpick aside, my proposal is to add ' ' (the U+0020 SPACE byte) as a Kind for filler. Code like this:

for d.PeekKind() != '}' {
  // etc
}

becomes

for {
  if k := d.PeekKind(); k == '}' {
    break
  } else if k == ' ' {
    d.SkipFiller()  // or repurpose the existing SkipValue method,
                    // or allow a Value to represent filler.
    continue
  }
  // etc
}

But if you're calling Decoder.ReadToken in a loop, instead of rolling your own Decoder.PeekKind based loop, then you're unaffected. Likewise if you're using the higher level marshal/unmarshal API.

In case my point wasn't clear, I don't think we can extend PeekKind in the future. If we want to have the ability to support comments, whether from day zero or in the future, we'll need to have filler tokens from day zero.

Having said all that, "many tools using JWCC only care about it as input" sounds plausible to me. Perhaps a "90% of the benefits, 10% of the costs" middle ground would have a boolean Options to allow comments and trailing commas, only on the decode side, and simply treat them the same as whitespace (and not preserved on round-trip). No need then for a filler Kind.

That would let people opt in to consuming JWCC. Producing JWCC (including automated edits) would be punted to a separate package.

I wonder whether the coroutine proposal might end up helping in that respect.

When I saw the coroutine package, I immediately started thinking about how it could solve the inverted io.Reader and io.Writer problem, but I haven't quite come to terms with exactly how. Since this problem exists outside of just "json", but also with gzip, base64, etc., it'd be great if we had a single solution for all packages.

Also, a thought about the proposal process and release schedule: would it be a dealbreaker to, for example, ship this encoding/json/v2 API as-is without first-class support for unions or sum types? As far as I can tell, it should be possible to add support later, as an incremental step. For example, we could later start treating some structs as unions like @josharian's example with a new option like json:",union". If that sort of later addition doesn't seem possible for any reason, please say so.

I only mention this because there's a balance at play here; if we try to cram too many features into encoding/json/v2 from day one, the proposal process and implementation will take longer, and it might take an extra year for any of this to release. Holding up the proposal seems right if there are any blockers or design issues that will become a limiting factor later on, though.

There's also multiple ways people encode union types ( { "type": "variant", "data": { ... } }, { "variant": { ... } }, [ "variant", { ... } ], { "type": "variant", ... }) and the possibility of some form of them landing in Go. Holding back on adding this functionality seems wise.

As far as I can tell, it should be possible to add support later, as an incremental step. For example, we could later start treating some structs as unions like @josharian's example with a new option like json:",union". If that sort of later addition doesn't seem possible for any reason, please say so.

I'm not sure we would even need an explicit union option.

Suppose a future Go adds interface{ A, B } where Go reflection allows us to see that this is a list of types, then we can have "json" unmarshal different JSON kinds into the appropriate type. This assumes that the "json" package can discern that JSON numbers go to type A and JSON strings go to type B. This is possible for types without any custom JSON methods. Once you add custom representation, it gets tricky. We would need some way to annotate the interface type to know the mapping. As an aside, this is one reason why it would be useful to have type tags similar to struct field tags.

Since type lists in interface types isn't possible today, there are zero occurrences of them in the wild. So long as we add support in "json" for unions the moment they land in the language, then we're okay.

The conversation up to now has focused mostly on the Go language representation. I'm going to pivot and take it from the perspective of a JSON parser.

Dynamic JSON roughly falls into two categories:

1. A dynamic JSON value where the resulting concrete Go type is determined entirely based on JSON kind itself (e.g., use of a Go string for a JSON string and a []string for a JSON array such as spec.StringOrArray).

   • The important thing to note is that we know which Go type to use simply by peeking at the kind of the upcoming token. This is a relatively easy case where we could potentially provide first-class support in the "json" package. That said, we'll probably want to see where native union types in Go eventually ends up.

2. A dynamic JSON value where the resulting concrete Go type is determined by some other JSON value. This situation is harder and breaks down into two sub-categories: whether we can know for sure the type in a forward parsing (easy) and whether we can't (hard):

   • A. In the easy case, we are guaranteed to encounter the type name before we encounter the dynamic value. For example, one such construction is something like {"TypeName": ...} where TypeName determines the concrete Go type to use and ... is unmarshaled into that concrete Go type. The syntactic construction guarantees that TypeName must be encountered first. Thus, this can always be handled in a purely streaming manner.

   • B. In the hard case, we may not see the type name before we encounter the dynamic value. This is unfortunately the more common case since many dynamic JSON uses constructions like {"type": "TypeName", "value": "..."} (e.g., google.protobuf.Any). Since JSON objects are unordered name-value pairs, there is no guarantee that we will see the "type" member before the "value" member. It is entirely valid that we encounter {"value": ..., "type": "TypeName"}, where we don't know the concrete Go type to use by the time we encounter the value itself. This greatly complicates the implementation where it must implement a double parse to temporarily hold the value in a jsontext.Value until it encounters the type (which also implies O(n²) for recursive situations). As an optimization, we can hope that most JSON inputs have the "type" come first, but that doesn't change the fact that we still need the double-parse fallback.

   • This form of dynamic JSON value is highly varied that it's unclear whether the "json" package should provide first class support for this. For example, not every schema uses a "type" field (e.g., protobuf uses "@type") or whether there is a "value" field or whether the value is inlined at the same level as the "type". The v2 package does at least make this situation easier to handle manually, though.

I was pleased to learn you can implement streaming unmarshaling of union types with UnmarshalFuncV2 by creating an unmarshal func where T is a pointer to an interface:

func unmarshalAttr(dec *jsontext.Decoder, v *Attr, opts json.Options) error {
	var proxy struct {
		Width  *int    `json:"width"`
		Height *int    `json:"height"`
		Name   *string `json:"name"`
	}
	err := json.UnmarshalDecode(dec, &proxy)
	if err != nil {
		return err
	}
	switch {
	case proxy.Width != nil:
		*v = Width(*proxy.Width)
	case proxy.Height != nil:
		*v = Height(*proxy.Height)
	case proxy.Name != nil:
		*v = Name(*proxy.Name)
	}
	return nil
}

type Attr interface {
	isAttr()
}

Full working code here: https://github.com/ydnar/scratch/blob/main/cmd/json-union/main.go