doc comment revisions: headings, lists, and links #48305

rsc · 2021-09-10T00:26:46Z

rsc
Sep 10, 2021
Maintainer

I am looking into the possibility of revising Go's doc comment syntax, specifically adjusting headings and adding lists and links. This discussion is meant to gather feedback before writing an official proposal.

The current Go doc comment format has served us well since their introduction in 2009. There has only been one significant change, which was the addition of headings in 2011. But there are also a few long-open issues and proposals about doc comments, including:

#7349 points out that the headings rule does not work well with non-Roman scripts.
#31739 points out that lines ending with double quotes cannot be headings.
#34377 points out that lines ending with parens cannot be headings.
#7873 asks for list support.
#45533 proposes linking of symbols, written as [io.EOF], to make it easier to write good top-level doc comments and cross-reference with other packages.

It makes sense, as we approach a decade of experience, to take what we've learned and make one coherent revision, setting the syntax for the next 10 or so years.

Goals and non-goals

The primary design criteria for Go doc comments was to make them readable as ordinary comments when viewing the source code directly, in contrast to systems like C#'s Xmldoc, Java's Javadoc, and Perl's Perlpod. The goal was to prioritize readability, avoiding syntactic ceremony and complexity. This remains as a primary goal.

Another concern, new since 2009, is backwards compatibility. Whatever changes we make, existing doc comments must generally continue to render well. Less important, but still something to keep in mind, is forward compatibility: keeping new doc comments rendering well in older Go versions, for a smoother transition.

Another goal for the revamp is that it include writing a separate, standalone web page explaining how to write Go doc comments. Today that information is squirreled away in the doc.ToHTML comment and is not easily found or widely known.

Within those constraints, the focus I have set for this revamp is to address the issues listed above. Specifically:

Make the header syntax more predictable. The headings rule is clearly difficult to remember and has too many false negatives. But further adjustments of the current rule run the risk of false positives.
Add support for lists. There are many times in documentation when a bullet or numbered list is called for. Those appear in many doc comments today, as indented <pre> blocks.
Add support for links to URLs. Today the only way to link to something is by writing the URL directly, but those can sometimes be quite unreadable and interrupt the text.
Add support for links to Go API documentation, in the current package and in other packages. This would have multiple benefits, but one is the ability in large packages to write top-level doc comments that give a good overview and link directly to the functions and types being described.

I believe it also makes sense to add another goal:

Add formatting of doc comments to gofmt, to promote consistent appearance and create more room for future changes.

It is not a goal to support every possible kind of documentation or markup. For example:

Plain text has served us very well so far, and while some might prefer that comments allow font changes, the syntactic ceremony and complexity involved seems not worth the benefit, no matter how it is done.
People have asked for support for embedding images in documentation (see #39513), but that adds significant complexity as well: image size hints, different resolutions, image sets, images suitable for both light and dark mode presentation, and so on. It is also difficult (but not impossible) to render them on the command line. Although images clearly have important uses, all this complexity is in direct conflict with the primary goal. For these reasons, images are out of scope. I also note that C#'s Xmldoc and Perl's Perlpod seem not to have image support, although Java's Javadoc does.

Markdown is not the answer, but we can borrow good ideas

An obvious suggestion is to switch to Markdown; this is especially obvious given the discussion being hosted on GitHub where all comments are written in Markdown. I am fairly convinced Markdown is not the answer, for a few reasons.

First, there is no single definition of Markdown, as explained on the CommonMark site. CommonMark is roughly what is used on GitHub, Reddit, and Stack Overflow (although even among those there can be significant variation). Even so, let's define Markdown as CommonMark and continue.

Second, Markdown is not backwards compatible with existing doc comments. Go doc comments require only a single space of indentation to start a <pre> block, while Markdown requires more. Also, it is common for Go doc comments to use Go expressions like `raw strings` or formulas like a*x^2+b*x+c. Markdown would instead interpret those as syntactic markup and render as “raw strings or formulas like ax^2+bx+c”. Existing comments would need to be revised to make them Markdown-safe.

Third, many features in Markdown are not terribly readable. The basics of Markdown can be simple and punctuation-free, but once you get into more advanced uses, there is a surfeit of notation which directly works against the goal of being able to read (and write) program comments in source files without special tooling. Markdown doc comments would end up full of backquotes and underscores and stars, along with backslashes to escape punctuation that would otherwise be interpreted specially. (Here is my favorite recent example of a particularly subtle issue.)

Fourth, Markdown is surprisingly complex. Markdown, befitting its Perl roots, provides more than one way to do just about anything: _i_, *i*, and <em>i</em>; Setext and ATX headings; indented code blocks and fenced code blocks; three different ways to write a link; and so on. There are subtle rules about exactly how many spaces of indentation are required or allowed in different circumstances. All of this harms not just readability but also comprehensibility, learnability, and consistency. The ability to embed arbitrary HTML adds even more complexity. Developers should be spending their time on the code, not on arcane details of documentation formatting.

Of course, Markdown is widely used and therefore familiar to many users. Even though it would be a serious mistake to adopt Markdown in its entirety, it does make sense to look to Markdown for conventions that users would already be familiar with, that we can tailor to Go's needs. If you are a fan of Markdown, you can view this revision as making Go adopt a (very limited) subset of Markdown. If not, you can view it as Go adopting a couple extra conventions that can be defined separately from any Markdown implementation or spec.

Headings

The current rule is:

A span that consists of a single line, is followed by another paragraph span, begins with a capital letter, and contains no punctuation other than parentheses and commas is formatted as a heading.

I can never remember the details of this exact rule, despite having chosen it. Every time I write a heading, I worry about whether it's going to be recognized as such. Others clearly have the same problem (#7349, #31739, #34377). The rule avoided the need for visible syntax, but in retrospect visible syntax would have been simpler. As Markdown shows us, that syntax can be very lightweight: a single “#” would suffice. Therefore I suggest the following:

New Rule: If a span of non-blank lines is a single line beginning with # followed by a space or tab and then additional text, then that line is a heading.

# This is a heading

Here are some examples of variations that do not satisfy the rule and are therefore not headings:

#This is not a heading, because there is no space.

# This is not a heading,
# because it is multiple lines.

The next span is not a heading, because there is no additional text:

#

In the middle of a span of non-blank lines,
# this is not a heading either.

Transition: The old heading rule will remain valid, which is acceptable since it mainly has false negatives, not false positives.
Gofmt will rewrite old-style headings into new-style headings, so that the fact of being a heading is made clearer to readers.

Lists

There is no support for lists today. As noted before, documentation needing lists uses indented <pre> blocks instead.

For example, here are the docs for cookiejar.PublicSuffixList:

// PublicSuffixList provides the public suffix of a domain. For example:
//      - the public suffix of "example.com" is "com",
//      - the public suffix of "foo1.foo2.foo3.co.uk" is "co.uk", and
//      - the public suffix of "bar.pvt.k12.ma.us" is "pvt.k12.ma.us".
//
// Implementations of PublicSuffixList must be safe for concurrent use by
// multiple goroutines.

And here are the docs for url.URL.String:

// In the second form, the following rules apply:
//      - if u.Scheme is empty, scheme: is omitted.
//      - if u.User is nil, userinfo@ is omitted.
//      - if u.Host is empty, host/ is omitted.
//      - if u.Scheme and u.Host are empty and u.User is nil,
//         the entire scheme://userinfo@host/ is omitted.
//      - if u.Host is non-empty and u.Path begins with a /,
//         the form host/path does not add its own /.
//      - if u.RawQuery is empty, ?query is omitted.
//      - if u.Fragment is empty, #fragment is omitted.

Ideally, we'd like to adopt a rule that makes these into bullet lists without any edits at all. (Markdown's space-counting rules would make these <pre> blocks, not lists.)

Today, a span of lines all indented by one or more spaces or tabs is always a <pre> block.
I suggest the following:

New Rule: In a span of lines all blank or indented by one or more spaces or tabs (which would otherwise be a <pre> block),
if the first indented line begins with a bullet list marker or a numbered list marker,
then that span of indented lines is a bullet list or numbered list.
A bullet list marker is a dash, star, or plus followed by a space or tab and then text.
In a bullet list, each line beginning with a bullet list marker starts a new list item.
A numbered list marker is a decimal number followed by a period or right parenthesis, then a space or tab, and then text.
In a numbered list, each line beginning with a number list marker starts a new list item.
Item numbers are left as is, never renumbered (unlike Markdown).

Using this rule, the two doc comments above are both recognized and formatted as bullet lists, not as <pre> blocks.

Note that the rule means that a list item followed by a blank line followed by additional indented text continues the list item (regardless of comparative indentation level):

// Text.
//
//  - A bullet.
//
//     Another paragraph of that first bullet.
//
// - A second bullet.

Note also that there are no code blocks inside list items—any indented paragraph following a list item continues the list item, and the list ends at the next unindented line—nor are there nested lists. This avoids all of the space-counting subtlety of Markdown.

To re-emphasize, a critical property of this definition of lists is that it makes existing doc comments written with pseudo-lists turn into doc comments with real lists.

Transition: Gofmt will rewrite recognized bullet and numbered lists to use a standard format. For example, the two doc comments above would reformat to:

// PublicSuffixList provides the public suffix of a domain. For example:
//
//   - the public suffix of "example.com" is "com",
//   - the public suffix of "foo1.foo2.foo3.co.uk" is "co.uk", and
//   - the public suffix of "bar.pvt.k12.ma.us" is "pvt.k12.ma.us".
//
// Implementations of PublicSuffixList must be safe for concurrent use by
// multiple goroutines.

and:

// In the second form, the following rules apply:
//
//   - if u.Scheme is empty, scheme: is omitted.
//   - if u.User is nil, userinfo@ is omitted.
//   - if u.Host is empty, host/ is omitted.
//   - if u.Scheme and u.Host are empty and u.User is nil,
//     the entire scheme://userinfo@host/ is omitted.
//   - if u.Host is non-empty and u.Path begins with a /,
//     the form host/path does not add its own /.
//   - if u.RawQuery is empty, ?query is omitted.
//   - if u.Fragment is empty, #fragment is omitted.

The specific formatting rules are discussed in the Formatting section below.

Markdown recognizes three different bullets: -, *, and +. In the main Go repo, the dash is dominant: in comments of the form `//[ \t]+[-+*] ` (grepping, so some of these may not be in doc comments), 84% use -, 14% use *, and 2% use +. In a now slightly dated corpus of external Go code, the star is dominant: 37.6% -, 61.8% *, 0.7% +.

Markdown also recognizes two different numeric list item suffixes: “1.” and “1)”. In the main Go repo, 66% of comments use “1.” (versus 34% for “1)”). In the external corpus, “1.” is again the dominant choice, 81% to 19%.

We have two conflicting goals: handle existing comments well, and avoid needless variation. To satisfy both, all three bullets and both forms of numbers will be recognized, but gofmt (see below) will rewrite them to a single canonical form: dash for bullets, and “N.” for numbers. (Why dashes and not asterisks? Proper typesetting of bullet lists sometimes does use dashes, but never uses asterisks, so using dashes keeps the comments looking as typographically clean as possible.)

Links to URLs

Documentation is more useful with clear links to other web pages. For example, the encoding/json package doc today says:

// Package json implements encoding and decoding of JSON as defined in
// RFC 7159. The mapping between JSON and Go values is described
// in the documentation for the Marshal and Unmarshal functions.
//
// See "JSON and Go" for an introduction to this package:
// https://golang.org/doc/articles/json_and_go.html

There is no link to the actual RFC 7159, leaving the reader to Google it. And the link to the “JSON and Go” article must be copied and pasted. Loosely following the Markdown shortcut reference link format, I suggest the following:

New Rule: A span of unindented non-blank lines defines link targets when each line is of the form “[Text]: URL”. In other text, “[Text]” represents a link to URL using the given text—in HTML, <a href="URL">Text</a>.

For example:

// Package json implements encoding and decoding of JSON as defined in
// [RFC 7159]. The mapping between JSON and Go values is described
// in the documentation for the Marshal and Unmarshal functions.
//
// For an introduction to this package, see the article
// “[JSON and Go].”
//
// [RFC 7159]: https://tools.ietf.org/html/rfc7159
// [JSON and Go]: https://golang.org/doc/articles/json_and_go.html

Note that the link definitions can only be given in their own “paragraph” (span of non-blank unindented lines), which can contain more than one such definition, one per line. If there is no corresponding URL declaration, then (except for doc links, described in the next section) the text is not a hyperlink, and the square brackets are preserved.

This format only minimally interrupts the flow of the actual text, since the URLs are moved to a separate section. As already noted, it also roughly matches the Markdown shortcut reference link format, without the optional title text.

Transition: Gofmt will move link definitions to the end of the overall doc comment. Go vet will flag unused link targets. Older versions of Go will show the text verbatim, which is fairly readable.

Links to Go API documentation

Documentation is also more useful with clear links to other documentation, whether it's one function linking to another, preferred version or a top-level doc comment summarizing the overall API of the package, with links to the key types and functions. Today there is no way to do this. Names can be mentioned, of course, but users must find the docs on their own.

Following discussion on #45533, I suggest to treat doc links like the links in the previous section, without target definitions. Specifically:

New Rule: Doc links are links of the form “[Name1]” or “[Name1.Name2]” to refer to exported identifiers in the current package, or “[pkg]”, “[pkg.Name1]”, or “[pkg.Name1.Name2]” to refer to identifiers in other packages.
In the second form, “pkg” can be either a full import path or the assumed package name of an existing import.
The assumed package name is either the identifier in a renamed import or else the name assumed by goimports. (Goimports inserts renamings when that assumption is not correct, so this rule should work for essentially all Go code.)
A “pkg” is only assumed to be a full import path if it starts with a domain name (a path element with a dot) or is one of the packages from the standard library (“[os]”, “[encoding/json]”, and so on).
To avoid problems with maps, generics, and array types, doc links must be both preceded and followed by punctuation, spaces, tabs, or the start or end of a line.

For example, if the current package imports encoding/json, then “[json.Decoder]” can be written in place of “[encoding/json.Decoder]” to link to the docs for encoding/json's Decoder.

The implications and potential false positives of this implied URL link are presented by Joe Tsai here. In particular, the false positive rate appears to be low enough not to worry about.

To illustrate the need for the punctuation restriction, consider:

// Constant folding computes the exact constant value ([constant.Value])
// for every expression ([ast.Expr]) that is a compile-time constant.

versus

// The Types field, a map[ast.Expr]TypeAndValue,
// holds type-checking results for all AST expressions.

and

// A SHA1 hash is a [Size]byte.

Transition: Older versions of Go will show the text verbatim, which is still fairly readable.

Formatter

Along with the changes, I suggest we add to go/doc a function

func Format(text string) string

that reformats a doc comment in the conventional presentation, and then to have go/printer and gofmt invoke this formatter.

The formatter would canonicalize the input so that it formatted exactly as before but with the following properties:

All paragraphs are separated by single blank lines.
Legacy headings are converted to “#” headings.
All code blocks are indented by a single tab.
All code blocks have a blank line before and after.
All list markers are written as “␠␠-␠“ (space space dash space) or “␠N.␠“ (space number dot space).
All list item continuation text, including additional paragraphs, is indented by four spaces.
All lists have a blank line before and after.
If there is a blank line anywhere in a list, there are blank lines between all list elements.
All bullet list items use - as the bullet (* and + are converted).
All numbered list items use the “N.” form (“N)” is converted).
The ASCII double-single-quote forms that have always been
defined to render as “ and ” are replaced with those.
Link URLs are moved to the end of the doc comment.

The formatter would not reflow paragraphs, so as not to prohibit use of the semantic linefeeds convention.

This canonical formatting has the benefit for Markdown aficionados of being compatible with the Markdown equivalents. The output would still not be exactly Markdown, since various punctuation would not be (and does not need to be) escaped, but the block structure Go doc comments and Markdown have in common would be rendered as valid Markdown.

Additional API

The current doc.ToHTML is given only the comment text and therefore cannot implement import-based links to other identifiers. To address this, we would need to add ToHTML and ToText methods to the Package type, and define that the top-level functions are as though calling the methods on a zero value of the struct. The ToHTML method will need to take a new Config struct that, at the least, allows specifying the URL prefix of the documentation server (for example, / or https://pkg.go.dev/ or https://golang.org/pkg/). It would also need to specify the HTML tag for headings, which will depend on the surrounding page where the docs will be presented.

There is an accepted proposal to add doc.ToMarkdown for easy conversion of Go doc comments to Markdown, and we would implement and update that as part of this work. It too would be added to the Package type.

scott-cotton · 2021-09-10T01:06:20Z

scott-cotton
Sep 10, 2021

Great to see this overview, thanks!

Generally:
I think that a sort of unspoken looming problem for the doc format is related to the tooling. 10 years ago there was godoc -http= and godoc.org. Now it appears work is shifting to pkg.go.dev, and I have heard that there is work on a command we can run to host docs locally. So now I, and at least some others, use godoc locally and pkg.go.dev otherwise. The problem w.r.t. the format here is that 10 years ago there was more or less one tool. Now there is a doc.ToMarkdown proposal, and momentum is shifting towards pkg.go.dev, without any interest in replicating things like godoc -analysis=xyz.

In such an ecosystem, there is a natural drive to diversify formats, which I do not think is good for Go. However, perhaps it would be fruitful to think of the doc format for the next 10 years as serving a wider set of tools than godoc.org. To me, the principal of "to make [docs] readable as ordinary comments when viewing the source code directly" is not bad because different tools can present the information without needing to delve into rapidly evolving formats or formats with several variants.

Specifically:
I think that the lists as they are are difficult to read in the source code, happy to see proposals to use a list format. I also spend time working around the lack of links, especially to local symbols. It would be great if a new doc format supported links to local symbols, and if, like in the source code, we could refer to those symbols in the scope of where the source code is (taking into account imports).

Looking forward to seeing how this evolves, and personally hoping something a little bit nicer than ascii for math can be included (though unicode math support in editors, eg … ⊧ ∃x ax²+bx+c seems to help a little).

11 replies

leetrout Sep 10, 2021

I'll toss in another vote for AST. I spent a weekend pulling comments out and trying to move them to unist so that we could offer a public rendering of our private package and docs.

I think it would be very useful to have this officially supported so folks could trim down and render their docs however best suits them.

rsc Sep 12, 2021
Maintainer Author

Personally, I think we should only provide an AST, and not get into the business of formatting HTML or Markdown.

Even if we do add an AST, it would be a mistake to remove ToText, ToHTML, and ToMarkdown. Those are incredibly useful. Not everyone wants to reimplement them.

rsc Sep 12, 2021
Maintainer Author

And for what it's worth. I'm mostly convinced about the AST in some form. I will think more about how to make it most convenient.

dsnet Sep 12, 2021
Collaborator

Even if we do add an AST, it would be a mistake to remove ToText, ToHTML, and ToMarkdown. Those are incredibly useful.

As a data point, I should note that pkg.go.dev avoids using doc.ToHTML, but has it's own HTML formatter since it needs more customization than what ToHTML provides. I think a ToHTML function is fine to provide, so long as we accept that it's intended for basic usages. I'm primarily concerned about the feature bloat the ToHTML API will grow in order to accommodate all the reasonable ways that a higher-level HTML renderer might want to tweak its output.

rsc Sep 16, 2021
Maintainer Author

FWIW I looked at the pkg.go.dev formatter, and the situation is at the least unclear. It has a comment saying it could be removed once Go 1.14 was out, but clearly other changes have been made since then, and the comment appears stale. But looking at the commits it appears that many of the changes are bug fixes that we probably should put back into the standard go/doc. I have yet to find anything that requires pkg.go.dev to be permanently diverged from go/doc's ToHTML (but it's hard to reconstruct everything). Instead it seems like they have a copy just to be able to make fixes quicker. And the fixes I see are not about HTML customization but about corner cases - real bugs.

dmitshur · 2021-09-10T01:18:52Z

dmitshur
Sep 10, 2021
Maintainer

Issue #38056 may be worth considering, if it hasn't already been, as it has potential intersection with what's described in the "Links to URLs" section.

To add some historical background, it appears godoc.org had automatic RFC linking since its initial commit in 2014, with additional enhancements (golang/gddo#276, golang/gddo#282) in 2015.

2 replies

rsc Sep 10, 2021
Maintainer Author

I'm a little uneasy with special-casing RFCs.
Why are they above the bar?
What other things might be special cased, and why are they below the bar?

dsnet Sep 10, 2021
Collaborator

I feel like the explicit linking syntax with [] is sufficient.

I had code where I wrote "RFC 8785, Appendix B." and pkg.go.dev failed to auto-link it. Automatically linking RFC works in some common cases, but it's annoying when it doesn't. We could improve RFC auto-linking, but the advantage with explicit linking is that the author has full control over making it work as we intended.

ghost · 2021-09-10T01:21:57Z

ghost
Sep 10, 2021

A couple of the documentation websites automatically link to RFCs (godocs, pkg.go). Perhaps gofmt should apply the linking logic used by these sites to create the explicit links shown in the issue.

5 replies

rsc Sep 10, 2021
Maintainer Author

I've definitely considered that, but I am not entirely convinced we should special-case RFCs. It feels like overfitting to one very specific use case for Go. Surely there are other kinds of standards that would merit this too? Where do we draw the line? Or: why are RFCs above the line and not other things?

rsc Sep 10, 2021
Maintainer Author

Let's call this thread a duplicate of #48305 (comment).

ghost Sep 10, 2021

I am suggesting that gofmt (perhaps behind a fix switch) create explicit links for what pkg.go.dev does automatically. The line is what go.pkg.dev currently implements. The justification for this line is that people are accustomed to RFC linking on godocs.org and its successors. The end goal is to remove the special case for RFCs from the rest of the ecosystem.

Speaking of a fix switch, should some of the proposed gofmt transformations be behind a switch? Perhaps somebody will want to write text that happens to match the legacy header rule.

rsc Sep 10, 2021
Maintainer Author

Perhaps somebody will want to write text that happens to match the legacy header rule.

In that case I hope it will be useful to see more clearly that it's actually going to be rendered as a header.
This is analogous to gofmt rewriting 1+2 * 3+4 into 1 + 2*3 + 4.
The rewrite also makes it possible, in some future version, to disable the old header rule, but I'm not suggesting that now.

(It's also very easy to avoid matching the header rule: end the text with a period, or split one line into two.)

arp242 Sep 12, 2021

If an AST is exposed (as discuses earlier) then people can write tools to automatically create/generate links to RFCs, GitHub issues, Debian bug tracker, PornHub videos, or whatever you want.

For example, personally I'd think that writing a tool to automatically create a link from #666 to [#666] and [#666]: https://github.com/issues/666 would be pretty useful, but I don't think it needs to be in the standard Go formatter, and by linking this explicitly rather than doing it in the HTML conversation step also means you have more control (e.g. for linking to specific sections, comments, using a different text, etc.)

jimmyfrasche · 2021-09-10T01:38:41Z

jimmyfrasche
Sep 10, 2021
Collaborator

The ToHTML Config should let you specify the level of heading tag to use, which is important for screen reader UX.

2 replies

rsc Sep 10, 2021
Maintainer Author

Good point, thanks, added.

jimmyfrasche Sep 10, 2021
Collaborator

Similarly it would be nice to be able to override the IDs for the headings to make them play nice with the surrounding html.

ghost · 2021-09-10T02:54:22Z

ghost
Sep 10, 2021

For example, if the current package imports encoding/json, then “[json.Decoder]” can be written in place of “[encoding/json.Decoder]” to link to the docs for encoding/json's Decoder.

Because imports are scoped to a file, should this say "if the current file imports" instead of "the current package imports"?

13 replies

dsnet Sep 13, 2021
Collaborator

Since I'm advocating for "current file"-only semantics, I think there are several reasonable workarounds.

Option 1: Use package-path qualified names. For example: [crypto/rand.Reader]

Option 2: Import the given package and make use of it:

// This package uses [rand.Reader].
package mypkg

import "crypto/rand"

var _ = rand.Reader

Option 3: Depending on exactly how the rules operate, you could imagine a blank import also being sufficient:
```
// This package uses [rand.Reader].
package mypkg

import _ "crypto/rand"
```

dsnet Sep 13, 2021
Collaborator

Well, OK, but what about doc.go when there is an unambiguous reference to json?

My concern is that it might be an unambiguous reference today, but then someone adds an import in another file that introduces ambiguities for references in doc.go. I'm not sure if it's a good user experience if changing the imports in a specific file requires thinking about how it may affect references in another file.

AndrewHarrisSPU Sep 13, 2021

I'd rather accept that linking isn't very magical and manage the mapping between inline markup and well-qualified links myself. Whether or not the tooling can interpret doc symbols unambiguously, I feel like it's fair to ask that the author interpret them unambiguously. It's something I find holistically useful when the author is me.

This code uses different packages for [-rand.Rand] and [-rand.Reader]. See [-blog] for details.

[rand.Rand]: doc:math/rand.Rand
[rand.Reader] doc:crypto/rand.Reader
[blog] https://www.blog.com/

If there is an opportunity to examine an AST generated from a comment, the symbol could be changed to a •, making that an easy way to spot where the AST is hanging, and unobtrusive in plain text.

This code uses multiple rand imports.

  • [•rand.Rand] is from math/rand
  • [•rand.Reader] is from crypto/rand

[rand.Rand]: doc:math/rand.Rand
[rand.Reader] doc:crypto/rand.Reader
[blog] https://www.blog.com/

ghost Sep 14, 2021

Attributing the package doc to a single file is not always possible because the package doc is the the concatenation of the non-empty package comments. One workaround is to discourage multiple package comments with a go vet warning.

earthboundkid Sep 21, 2021

I favor a hybrid, package/file approach. Suppose a.go imports math/rand, and doc.go imports nothing but says "This package uses [rand.Rand]." If crypto/rand isn't imported in another file, the docs should link [rand.Rand] to math/rand, but if b.go imports crypto/rand, then the package name is ambiguous and it should fail to link. Link imports per package if unambiguous, per file if ambiguous.

diamondburned · 2021-09-10T03:07:04Z

diamondburned
Sep 10, 2021

How exactly would GoDoc handle links that have the same inner text, such as the
snippet below?

// Here is a paragraph. See link 1 [here]; link 2 [here].
//
// [here]: ???
// [here]: ???

The first paragraph contains 2 links that have the same inner text, so there
doesn't seem to be a way to declare them properly.

I think having an additional syntax like [here][here-1] (or [here][1])
similarly to Markdown's hyperlinks would solve this issue.

5 replies

dsnet Sep 10, 2021
Collaborator

What's the use case for this? The text within the link needs to be a unique reference. It seems more sensible to do:

// Here is a paragraph. See [link 1]; [link 2].
//
// [link 1]: ???
// [link 2]: ???

diamondburned Sep 10, 2021

There's not really a particular "use case" for this; it's just something that
might often come up when writing things up. For example, it's often natural for
me to say "refer to the documentation [here] or [here]", but that might not
always be translatable over.

With the current design, the issue goes beyond just the word "here" as well. I
think hyperlinking on any single word will be quite difficult, since the common
words would collide more often within a single link paragraph.

I suppose enumerating the text might be easier, but it doesn't always feel
natural, especially when I might not have control over what the comment is,
especially during code generation. If I stumble upon a link with the same inner
text, then I will likely have to enumerate that, making it ugly for the reader.

fzipp Sep 10, 2021

@diamondburned Using "here" (or even worse "click here") as a link text is bad practice anyway, see:

https://www.washington.edu/accessibility/checklist/links/
https://www.w3.org/QA/Tips/noClickHere

I don't think we have to support it.

diamondburned Sep 10, 2021

So what about code generation? Do I have to enumerate the texts then?

rsc Sep 12, 2021
Maintainer Author

@diamondburned I don't understand your question.

kylelemons · 2021-09-10T03:11:33Z

kylelemons
Sep 10, 2021

For assumed package names, I also (lightly) propose handling a few other common corner cases that I've noticed. These usually come up as the names for GitHub repositories (to avoid conflicts and make the language clear), and so if there are any Go source files at the root of the repository, the assumed package names would be invalid identifiers.

"pkg-go" is assumed to be named "pkg" (after ignoring any "/vN" suffix)
"pkg.go" is assumed to be named "pkg" (after ignoring any "/vN" suffix)
(maybe) "go-foo" is assumed to be named "pkg" -- this hasn't come up for us, but my memory says I've seen this more than once.

(we have had to write some of this down in our style guide since this has come up a few times, though this is at least partly our own fault, as we have Go source files at the root of the baseplate.go repository, following the prior convention from baseplate.py)

3 replies

dsnet Sep 10, 2021
Collaborator

The evidence for this seems strong.

I found:

8530 cases where stripping a go- prefix results in the correct package name
2029 cases where stripping a -go suffix results in the correct package name
204 cases where stripping a go. prefix results in the correct package name
114 cases where stripping a .go suffix results in the correct package name

For comparisons, I found:

6047 cases where stripping a /vN suffix results in the correct package name
1278 cases where stripping a .vN suffix results in the correct pacakge name

rsc Sep 10, 2021
Maintainer Author

Thanks very much for pointing this out.

I thought I was describing what goimports does, but I was not.
Now I've changed to be "what goimports does".
When that doesn't produce the correct identifier, goimports (and gopls) inserts a renaming anyway, so that should make the rule work for essentially all Go code.

kylelemons Sep 11, 2021

👍 Tying it to how goimports assumes names definitely makes sense to me.

For the curious, it looks like goimports handles all but the 204 go.-prefixed names, which seems like a pretty good batting average:

https://play.golang.org/p/Xbcmy1EaLKZ

ImportPathToAssumedName("foo")       = "foo"
ImportPathToAssumedName("foo/v1")    = "foo"
ImportPathToAssumedName("foo.v1")    = "foo"
ImportPathToAssumedName("foo.go")    = "foo"
ImportPathToAssumedName("foo.go/v1") = "foo"
ImportPathToAssumedName("foo-go")    = "foo"
ImportPathToAssumedName("foo-go/v1") = "foo"
ImportPathToAssumedName("go-foo")    = "foo"
ImportPathToAssumedName("go-foo/v1") = "foo"
ImportPathToAssumedName("go.foo")    = "go"
ImportPathToAssumedName("go.foo/v1") = "go"

And thanks for the canonical reference! I definitely need to add that to the style guide...

kortschak · 2021-09-10T04:14:11Z

kortschak
Sep 10, 2021

Is there a reason for choosing the link digraph sets with their low false positive rate over the essentially zero false positive rate with the lighter weight on the page `...' pairing?

16 replies

kortschak Sep 14, 2021

I've raised accessibility here three times. No response has addressed that. Note that this is not a theoretical concern.

dsnet Sep 14, 2021
Collaborator

@kortschak I'm not sure anyone disagrees that [...] delimiters have more visual noise than `...' (this quantitatively easy to prove based on the number of pixels needed to print it). As with most design decisions, there are different factors at play such as the familiarity of [...] (due to Markdown) over `...'. Pointing at the advantage of different approach is (in my opinion) a valid response, even if it doesn't directly address your concern of accessibility.

kylelemons Sep 14, 2021

I think it's important to note that concerns over accessibility can cut many ways. My screen reader, for example, is much happier with [text] than it is with `text'. For the former, it simply ignores the brackets and reads straight through, but for the latter it narrates the backtick but not the apostrophe. This is likely to vary between software, but given that the quotes are two different kinds of punctuation and they are not opposites, I can imagine that it is going to be pretty common to hear this kind of asymmetry. (I would also be happy for it to narrate both delimiters, which it does in some cases like [...] when there is no word to read, but reading one and not the other is annoying, at least to me)

kortschak Sep 14, 2021

Yeah. I'm not saying that I'm right about the solution or that the choice is simple (I come from a family of typographers), but I do feel that the concerns have been ignored.

rsc Sep 16, 2021
Maintainer Author

@kortschak, re:

It avoids needless variation.

I'm trying to understand how requiring a novel pair of digraphs to delimit a link is not variation and how addressing readability on the page is needless. This latter point has not been addressed anywhere here.

The full comment was:

Although I believe it is a mistake to adopt Markdown wholesale, it seems clear that lots of people do know Markdown. It seems like leaning into that a little and using the syntaxes that those people already know is a win. It avoids needless variation.

In that context "It" = reusing Markdown notations. The needless variation is doing something gratuitously different from Markdown.

You have raised readability a few times but I can't fathom how `text' is somehow more readable for a link than [text]. If I saw the former I would think it meant quotation marks (which it does in LaTeX).

I think we have to agree to disagree about which is more readable.

halturin · 2021-09-10T12:52:21Z

halturin
Sep 10, 2021

Interfaces, please, let's make them readable in the doc.

1 reply

dsnet Sep 10, 2021
Collaborator

Are you talking about Go interfaces? If so, it seems that this is a concern for the rendering of how interfaces look, rather than the syntax of documentation. I think it's out of scope for this discussion.

azr · 2021-09-10T14:11:26Z

azr
Sep 10, 2021

👋🏼 , Super nice to see this topic. In Packer, we often go generate docs from comments on structs (code, generated, live docs, generator) and then aggregate/import docs into pages. It's rather awesome because changes often happen only in one targeted place for us, and it makes documenting a bit easier and important. Moreover the code becomes the single source of truth.

For links, we use the markdown synthax: [word](url) and some pages can be quite big, containing a lot of links containing the same words/description; and having the description/link grouped together seemed simpler in terms of treatment. It can be slightly less easy on the eye when reading comments though.

11 replies

Gregory-Ledray Sep 10, 2021

Perhaps it makes more sense to have gofmt put the link definition immediately after its use, to make that easier

If that is what gofmt did - perhaps putting the links at the end of the current paragraph - then my argument really boils down to personal preference. In that case, I would very much appreciate it if gofmt automatically converted [text](https://link.io) to

[text]
[text]: https://link.io

Although I would understand if the language did not support that and I had to write a macro to do it instead.

dsnet Sep 10, 2021
Collaborator

I prefer the inline syntax because when reading code and I encounter a link, I often want to click on it.

Once a standard GoDoc syntax is ratified, I suspect the tooling ecosystem will adapt. One can imagine that in a future version of VSCode with the Go plugin, you can ctrl+click on [RFC 7159] and it will know fetch the link reference declared later on.

Gregory-Ledray Sep 10, 2021

True. But Go should not depend on an IDE to provide a good user experience, and IDEs are not guaranteed to implement such features by default. Case in point: despite Visual Studio being capable of rendering Xmldoc when I'm reading C# code, it does not always do so by default to make editing easier. Consequently I have to read ugly XML when I'm reading through C# code.

ChrisHines Sep 10, 2021

Your point about finding the link is a good one. Perhaps it makes more sense to have gofmt put the link definition immediately after its use, to make that easier.

This is the approach I have been using when writing markdown manually because I think it has the best balance of readability of the prose and ease of finding the referenced links.

arp242 Sep 12, 2021

I prefer the inline syntax because when reading code and I encounter a link, I often want to click on it. With the reference link syntax, if the doc.go file is long or there is an example in the comment, you may have to scroll down quite a ways and hunt for the link among a forest of other links. I know I don't have to say this, but I want to reinforce it anyway: a link is only useful if you can click on it.

Personally I strongly prefer putting the links externally in anything of substance I write (as outlines in the proposal) as it makes the text "flow" a lot better; I usually try to make documentation read well, and I find having a (possible very long) link smack-down in the middle of it makes it a lot harder. I also find it harder to read: 9 times out of 10 I don't really care about the link, so it makes the text harder to read.

It's true that this makes actually clicking the link a bit harder, but that's a problem solvable with fairly simple tooling as mentioned. Conversely, making the text "flow better" is a problem that's harder to solve with tooling.

In the end, both methods have their own set of advantages and disadvantages – which is why Markdown supports both – but the more pragmatical solution for Go seems to me to be to use [text]: https://link as the downsides with that are much easier solvable.

Anecdotally, I've had a lot of trouble in the past remembering whether Markdown's link syntax is text, (text)[url], url, or (url)[text]

The way I remember this is that it's the same as you would write it without any formatting: put the link in parenthesis after the text:

For more information, see the documentation (https://example.com) which has detailed examples.

I still get it wrong on occasion though, especially if I'm in a hurry, and I've heard from many people over the years that they have trouble remembering this.

peterbourgon · 2021-09-10T18:28:06Z

peterbourgon
Sep 10, 2021

Markdown is not the answer, but we can borrow good ideas

Markdown is the de facto standard solution for the problems you're trying to solve here. It certainly has flaws, but in practice they're solvable without major effort. And the benefits of using a more-or-less universally understood syntax hugely outweigh those costs.

Go shouldn't spend an innovation token on documentation syntax. It's a solved problem, and the marginal improvements offered by alternatives don't outweigh their concomitant toil.

—

I am fairly convinced Markdown is not the answer, for a few reasons.

To respond to each point...

First, there is no single definition of Markdown . . . Even so, let's define Markdown as CommonMark and continue.

That's just how these things go 🤷 and while it definitely introduces complexity to the parsers and renderers, it's not intractable.

(GitHub-flavored Markdown is what you should be targeting, for the record. It's by far the most widely used.)

Second, Markdown is not backwards compatible with existing doc comments . . .

It seems like all of these issues can be solved individually without too much fuss. You would end up forking GitHub-flavored Markdown and creating Go-flavored Markdown, or something like that. That's an ideal outcome.

Third, many features in Markdown are not terribly readable.

This is subjective, of course, so I'm not sure how much weight you can give to it in the decision-making. Furthermore, most of the enumerated feature requirements are core Markdown stuff, which means basically all Go programmers already know how to read and write them. (Thanks, GitHub!) That usability surely trumps aesthetics in the calculus.

Fourth, Markdown is surprisingly complex . . .

It's true, and in some sense a shame, that Markdown lets you express 1 thing in N ways. IMO, a Go-flavored Markdown could reasonably elide a lot of those "features" — I wouldn't miss HTML elements, or alternate demarcations for headings, for example. Maybe some people would say that it isn't actually Markdown without those features. Fair enough. But that's fine, there's no standards body involved here.

Just a point of feedback. Take it as you will.

11 replies

peterbourgon Sep 12, 2021

In-line links take up a ton of space in a paragraph

I agree that the readability of inline links is decreased in proportion to the length of the link. But the readability of footnote links is decreased in proportion to the distance between the link and the end of the paragraph. Neither is strictly more or less readable than the other; both have their place.

1 lists don’t render correctly as plain text.

That's true, and a good point. If the proposed syntax has "equivalent readability between source form and rendered form" as a design goal, then I agree it makes sense to elide this feature.

—

I also wanted to give a big +1 to not supporting images. Thanks for that.

peterbourgon Sep 13, 2021

Markdown is the de facto standard [on GitHub] . . . Go aims to serve a much broader audience than just people who live on GitHub.

And GitHub is the de facto standard VCS. I'm not sure there are many Go programmers in the world who don't use GitHub on a regular basis.

edit: I have no idea why you'd thumbs-down this, @kylelemons. It's a statement of fact followed by a (clearly labeled) statement of opinion. If you want to object, please do so in a productive way.

azr Sep 13, 2021

👋🏼 I've been thinking about this !

I also kinda like the renumbering of Markdown, because when I add a new item, I don't have to renumber/recommit the rest. Making it easier to find history in git and making PRs smaller. But the thing here, is that if I use a "simple/dumb" text editor; this magic numbering will be lost. And for editors that do renumber things, this makes it hard to edit docs. Maybe they suddenly need to have 'preview/raw' modes ?

Same thing for links: I also kinda like supporting more ways to display URLs. That's just a personal preference. But it's true that the readability of [description](url) can be a bit 'meh' on the eye. I have used it for a while in go docs, and they sometimes make lines of comments extra super long, breaking the style of godocs. They also very often break the flow of reading documentation. For example, when you're reading/understanding something, and suddenly sort of have to switch your eyes in 'skip mode'. It can be hard for me and I often lose the context and have to re-read again.

This would be solvable by a magic editor too, but I think that optimizing for a 'simple/dumb' text editor is just simpler and makes more sense.

zephyrtronium Sep 13, 2021

@peterbourgon
On the Go issue tracker, we use thumbs-down reactions to mean "I disagree." It is not an attack, just a way to indicate one's sentiment regarding an idea without flooding threads with "+1" or "-1" comments.

rsc Sep 13, 2021
Maintainer Author

It's hard to think of a clearer example of prioritizing writability over readability than 1, 1, 1 lists.
As noted, it remains a primary goal to prioritize readability of the actual comments.
Therefore we will not do any automatic renumbering of 1, 1, 1 lists.

benjaminjkraft · 2021-09-10T20:52:11Z

benjaminjkraft
Sep 10, 2021

Minor point on links to Go documentation: I wonder if it's feasible to allow links surrounded by spaces (in addition to punctuation/linebreaks), so you can do e.g.

The [File.Comments] list is not changed.

Of course [Size] byte is legal Go, but it's already not the clearest in GoDoc, and probably quite rare.

4 replies

rsc Sep 10, 2021
Maintainer Author

Fixed, thanks.

ChrisHines Sep 10, 2021

Thanks for this suggestion @benjaminjkraft. The punctuation requirement around this type of link was my biggest concern so far because I thought it would be subtle and hard to remember while also adding unfortunate noise in many cases. This change removes that concern for me.

rsc Sep 11, 2021
Maintainer Author

I was mentally including spaces in punctuation, as you can see in the examples. Sorry for the confusion.

ChrisHines Sep 13, 2021

Fair enough, but in the interest of closing the feedback loop here, the examples by themselves seem ambiguous on this point, at least to me. Note that the only positive example of "punctuation" is using parentheses. The other examples show how normal Go syntax could be mistaken for links if the requirement didn't exist. Since spaces are not normally considered punctuation in English grammar, it didn't seem reasonable to assume they would be in comments.

The problem is solved by the updated explanation. So no worries now.

AndrewHarrisSPU · 2021-09-10T20:57:46Z

AndrewHarrisSPU
Sep 10, 2021

I have a possibly useful hypothetical situation I mind: Consider an IDE that is cramming documentation into a hovering tooltip. I generally hate this but people have preferences …

There’s a particularly strict approach here that might suggest even the smallest amount of poorly handled inline markup leads to a failure to correctly display documentation. I tend to read this way - if the docs have any garbage, my brain instantly switches to solving the garbage problem rather than reading the doc. (Fans of a particular IDE won’t attribute the garbage to the IDE, either…)

In the IDE hypothetical, what are the chances and burdens of implementation versus displaying plain text? My reactionary take: the plain text must be pristine. ‘-‘ lists seem reasonable. Links should be more like footnotes - at the bottom, not inline. Emitting a very light AST prior to e.g. HTML, go doc terminal output seems pragmatic but the plain text should always be the assumed mode of rendering.

0 replies

rothskeller · 2021-09-10T21:57:54Z

rothskeller
Sep 10, 2021

I believe the plan for continuation lines in lists will be widely condemned as aesthetically ugly. Continuation lines should start in the same column as the text of the first line after the marker, thus:

- This is a test of the
  emergency broadcast
  system.

not

- This is a test of the
    emergency broadcast
    system.

It may be that keeping the four-space indentation of continuation lines is valuable for Markdown compatibility. But if so, the initial line should be formatted to match:

-   This is a test of the
    emergency broadcast
    system.

This approach may be particularly useful when dealing with numbered lists, whose numbers may not all have the same length:

9.  The ninth item.
    And its continuation.
10. The tenth item.
    And its continuation.

Also, as a minor note, the proposal specifies that numbered list items can start with any decimal number, but I think in practice it would need to be a non-negative number.

7 replies

kylelemons Sep 11, 2021

I think I had probably forgotten the examples by the time I got to the rules. Here's what I was keying off of:

All list markers are written as “ - “ (space space dash space) or “ N. “ (space number dot space).

I thought that was (space dash space) at the time, which could easily have been faulty reading or our tendency to combine when a word wraps to an identical word.

All list item continuation text, including additional paragraphs, is indented by four spaces.

The four spaces of indentation doesn't match the three characters " - " or the 5+ characters of " 10. " so I was assuming it wouldn't vertically align.

rsc Sep 11, 2021
Maintainer Author

Thanks. I changed the spaces to ␠ in the quotation.

kylelemons Sep 11, 2021

As for 9 vs 10, there is an additional option (that admittedly results in more diff lines and harder logic):

//   9. Line 1
//      Line 2
//  10. Line 1
//      Line 2

(i.e. "re"indent earlier numbered bullets based on the indentation required by the final number)

I would be very slightly in favor of this, but it has its downsides.

rsc Sep 11, 2021
Maintainer Author

Yes, the point of space space dash space for bullet lists is to keep all list text at the same indentation level, rather than having bullet list text one space to the left of number-list text. Typographically, I believe that consistency takes priority.

Beyond typography, one real problem is that when you add item 10 to a long list, every line above has to change in the diff. That seems suboptimal too. The current rule is much simpler.

rothskeller Sep 12, 2021

@rsc Thanks, the clarification of the space space dash space resolves my indentation concern. (Although I agree with another comment that space space bullet space would be better.) As regards numbered lists, I agree with the fixed four space indent of continuation lines for the reasons you cited.

dsnet · 2021-09-11T20:25:26Z

dsnet
Sep 11, 2021
Collaborator

If we're going with the route of gofmt-ing comments, can we instead rewrite the list marker as the Unicode Bullet (•, U+2022)? This is the most aesthetically pleasant character to use and it's a character whose purpose is to "introduce items in a list". On the other hand, both - and * seem like a historical hack in the computer industry to use that character for a similar purpose.

For usability, I'd still be able to use the - or * character, but then the moment I save the file, it gets auto-formatted as a •. Thus, I wouldn't have to worry about manually typing Unicode bullet points.

12 replies

rothskeller Sep 12, 2021

@rsc said, "I don't think Mac and Plan 9 is a majority of Go developers." Probably true, but do we have data on that? I suspect that Mac users, while not a majority, might be a pretty significant fraction.

I would argue at the very least that (a) the doc tools ought to recognize • as one of the valid characters introducing a list item and (b) gofmt should not change them to dashes. Don't force people away from the solution that everyone agrees is the most aesthetic.

My preference would be for gofmt to change all unnumbered lists to use • as the list marker. I believe new coders become very quickly aware that gofmt reformats their code for them. They will come to rely on it for bullet formatting just as they now come to rely on it to address many other shortcuts.

natefinch Sep 14, 2021

I really don't want to rely on gofmt to change my - into a •. There are a lot of places where gofmt doesn't run, like if you're editing something via the github UI. I agree that as a newbie to the language, I would go and copy and paste that symbol (like I did to write it in that first sentence) and be super annoyed that I couldn't just use a - like a normal person.

And as Russ said, - is suuuper common for a list marker. let's just keep the one that actually has a physical key right there on your keyboard.

bcmills Sep 16, 2021

Do any of the popular Linux or Windows editing environments make this any easier?

The vast majority of Linux environments support .XCompose key sequences, which are straightforward to configure.
Personally, I use an international layout and remap the ¤ (“currency”) character to produce the Unicode bullet:

<currency>:	U2022 # BULLET

The default XCompose configuration produces the character from the sequence ⎄ . =.

An XCompose interpreter is also available for Windows (https://github.com/samhocevar/wincompose), but my impression is that it is not widely used.

bcmills Sep 16, 2021

At any rate: I don't have a strong preference as to whether gofmt rewrites lists to use the Unicode bullet character, but I do feel strongly that the character should be included in the list of “bullet list marker” characters, and I have a slight preference for gofmt keeping lists that already use that character as-is.

Today, you can take a bulleted list typed elsewhere, paste it into a Go source file, and use a few quick editor commands to add a // prefix to make it into a pseudo-list. It would be unfortunate if going forward that results in a code block instead of a proper bulleted list.

beoran Feb 10, 2022

In the past, the Golang project has always decided against using non ASCII characters syntactically. I think the reasons for this are sound and we should stick to them.

benhoyt · 2021-09-12T00:29:24Z

benhoyt
Sep 12, 2021

I like this a lot, particularly how simple and close to the current documentation format it is (definitely agree about not importing all of Markdown).

Regarding numbered lists: "Item numbers are left as is, never renumbered (unlike Markdown)." What if they have a list with just "2." and "4." in it (or even out-of-order items)? I was unsure how this would be done in HTML, but it looks like there's an explicit value attribute on the <li> element, so definitely possible. Or would we just disallow out-of-order numbering (or correct it in gofmt)?

4 replies

rsc Sep 12, 2021
Maintainer Author

You would just have a list with items numbered 2 and 4.

benhoyt Sep 12, 2021

Fair enough, that's what you asked for. And I guess it's extremely unlikely in practice.

telemachus Sep 12, 2021

And I guess it's extremely unlikely in practice.

I agree that it's unlikely that anyone would intentionally or accidentally produce lists with, e.g., just 2 and 4.

But there is a mistake that may be more likely. People who write Markdown often may default—without even thinking about it—to numbered lists with only 1. And since so many people already know Markdown, newcomers to Go may miss the specific rules about numbering if they skim Go's documentation.

I don't think these possible mistakes should change the current proposal, but maybe the documentation should stress that people must number their own lists and that tools (e.g., gofmt) will not rewrite numbered lists. Another way to put this is that the docs should be careful in how they describe the relationship between Go's doc comments and Markdown. Things like "subset of Markdown" can be true but still misleading. So it may pay to stress "very limited" in "a (very limited) subset of Markdown."

natefinch Sep 14, 2021

The whole point of this kind of documentation is to have the plaintext match the rich text. 1 1 1 numbering explicitly breaks this promise. It's a bad idea in markdown (though I understand the impetus), let's not take that bad idea and move it into our doc format. The whole point of having numbered lists is so you can easily reference specific items... otherwise you might as well just have bullets. But if your numbers can change out from under you, then your reference to a numbered item can also change out from under you.

cooldarkdryplace · 2021-09-12T11:39:21Z

cooldarkdryplace
Sep 12, 2021

I have noticed that the feature set matches what is used in the Gemini mark-up.
Similarities do not end here. They have also rejected Markdown for the same reason: it is more complex than it seems to be at first glance. The intention was to make a markdown language that is parsable without any complexity, line by line.

I am not suggesting to start using Gemini markdown; even I would love that.
What can be considered is to work with the links in a similar way. Not showing them inline, but using as a block of lines, for example, at the bottom of a godoc section.
This may sound too limiting, but I think it will encourage users to follow a simpler structure and not become a wiki.

2 replies

rsc Sep 13, 2021
Maintainer Author

Thanks for the reference. I had seen this a while ago but forgotten about it.
As I understand it, Gemini markup does not allow linking the words "the Gemini mark-up" in your comment.
Ultimately I think the benefits of specific linked text far outweigh the syntactic weight of the [ ]:
it's an explicit goal to support that well.

cooldarkdryplace Sep 13, 2021

As I understand it, Gemini markup does not allow linking the words "the Gemini mark-up" in your comment.

This is true; I have proved myself wrong by using an in-line link.

arp242 · 2021-09-12T19:57:15Z

arp242
Sep 12, 2021

One concern with allowing lists inside pre blocks is that there may be cases where you want to write a pre block that starts with a bullet list marker but don't want it converted to a list. For example sometimes I incorporate some "ASCII art":

//   +-----------------------------+
//   | Some ASCII table or whatnot |
//   +-----------------------------+

This would accidentally start a list I think?

Especially combined with gofmt rewriting comments, I feel that pre blocks really should be left alone without exceptions. I fear that anything else will introduce some really unexpected and surprising behaviour in various edge cases.

Overall I feel that long-term design should be (strongly) prioritized over short-term compatibility. Nothing will break if pre-block lists are just treated identical as they are now. This has worked well enough for over ten years, and while some updating may be required to make existing comments better by taking advantage of the new syntax – which is never great – it also means we'll still be "stuck" with these short-term compatibility rules in ten years time.

Similarly, I feel having just one bullet marker would be better as it's more consistent with Go/gofmt's general design aesthetic of "one correct way to write/format it". I don't really care which as such. I appreciate the short-term compatibility advantages and that there's no clear "winner" on which character to use, but it makes Go better in the long term.

2 replies

benhoyt Sep 12, 2021

I like the idea of one bullet marker as the "one true bullet style" (which would probably be - given what Russ wrote above).

Regarding your ASCII art -- this may happen, but not in the specific example you gave, as the + bullet marker doesn't have a space after it. "A bullet list marker is a dash, star, or plus followed by a space or tab and then text."

rsc Sep 13, 2021
Maintainer Author

As long as the bullet marker does not appear on the first line, the whole indented span is a pre block.
In the worst case, you start with a line that clearly identifies a non-list.
The false positive rate here is going to be very small.

ajstarks · 2021-09-13T11:09:50Z

ajstarks
Sep 13, 2021

I'd like to address the explicit omission of illustrations as proposed in #39513

the original illustration proposal was written to explicitly avoid the complexity of a markdown-like language, and used //godoc: image... Now that we are considering "Go-Flavored Markdown", perhaps we can use:

![text]: https://link/to/image

and still accomplish the primary goals of simplicity and readability.

The argument for omission seems to be "illustrations are too complex, and not worth it", citing image size hints, resolutions, rendering, light and dark mode images -- all of these can and should be addressed by the document writer. Just as we want the writer will use good grammar, proper syntax, and clear writing, it is the writer's responsibility to use (or not use) appropriate illustrations in the service to good communication. For example not using images that are too large, or using the alternative text where image rendering is not practical.
It seems like a missed opportunity go another 10 years without having a fundamental part of documentation that have existed for centuries: illustrations
If illustrations are not supported, then the programmer is forced to use alternative and perhaps duplicate methods like READMEs which do support illustrations. It would be more productive to maintain documentation in one place.

Finally, under the the new proposal, what is the behavior of:

[text]: https://link/to/image

8 replies

ajstarks Sep 13, 2021

Re: image sizing, it would be interesting to see how github approaches this.
I can say ![text](file.png) and the correct thing happens.

I'd love to the the data that supports the claim 99% of packages don't need images in their docs. In fact, I don't think we know how many packages will need illustrations; tradition and capability prevent us from knowing.

peterbourgon Sep 13, 2021

Images seem a nonstarter, given the explicit goal of keeping the source and rendered forms of docs equally readable. Linking to an image feels like a good compromise: it satisfies this goal, it's already supported, and by avoiding inline images we maintain the virtuous property that all information displayed in documentation is part of the source tree itself — there's no risk of important information being unavailable due to broken links, bad networks, misconfiguration, bit rot, etc.

fgm Sep 13, 2021

Regarding images, even if they are not explicitly rendered as such, a doc server could relatively easily provide such a render without needing any extra in-format information: for links that are alone on a comment line, fetch the target, if it is an image use it and build derived images based on what the source resolution allows, with derived names per format, and render the link as an img srcset to these images probably linked to the original.

jimmyfrasche Sep 13, 2021
Collaborator

@fgm That's true but that means you can't specify alt text.

beoran Feb 10, 2022

The solution to having scalable images with a single image is to use SVG in stead of PNG. With that said, godoc could allow inlining SVG files, or more radically, render ASCII graphs to SVG like the ditaa tool does, thus keeping the graphs readable in plain text format.

deanveloper · 2021-09-21T18:23:49Z

deanveloper
Sep 21, 2021

Having sub-lists be defined would also be very convenient. Maybe sometime in the future, maybe now. I have, a couple of times, needed to explain certain guarantees about the return value of a function, and I annotated this by using lists for guarantees, and billeted descriptions to these guarantees in sub-lists.

0 replies

jhenstridge · 2021-09-22T07:59:16Z

jhenstridge
Sep 22, 2021

Transition: The old heading rule will remain valid, which is acceptable since it mainly has false negatives, not false positives.
Gofmt will rewrite old-style headings into new-style headings, so that the fact of being a heading is made clearer to readers.

Most of the other proposed gofmt cleanups are neutral w.r.t. processing documentation with old tools. This one would convert headings into non-headings from the point of view of existing software.

Maybe this is acceptable since use of headings seems fairly rare, and it's not going to make a particularly noticeable difference to how documentation is formatted in the terminal.

0 replies

dsnet · 2021-09-23T00:41:25Z

dsnet
Sep 23, 2021
Collaborator

Should we support linking to a heading declared in the package-level documentation from a specific declaration?

For example, lets say my package-level godoc says:

// Package hujson contains a parser and packer for the HuJSON format.
// Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
// tempor incididunt ut labore et dolore magna aliqua.
//
//
// Use with the Standard Library
//
// Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
// tempor incididunt ut labore et dolore magna aliqua.
// Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi
// ut aliquip ex ea commodo consequat.
package hujson

This contains a heading called "Use with the Standard Library".

Elsewhere in the godoc for a declaration, I would like to do:

// Deprecated: Do not use. See [Use with the Standard Library] for alternatives.
func Unmarshal([]byte, interface{}) error

2 replies

ChrisHines Sep 23, 2021

I would use this feature.

Question: How should tools deal with ambiguity due to duplicate headers or headers that also match another valid link target?

dsnet Sep 23, 2021
Collaborator

How should tools deal with ambiguity due to duplicate headers

The fact the "Overview" section on godoc already links to headings suggests to me that there's already some implicit behavior in place for dealing with duplicates. The HTML anchor ID is derived from the heading in what I suspect (but did not verify) is a stateless function (i.e., it does not consider what other headings exist). This would imply that two duplicate headings today would already result in conflicting HTML anchor IDs.

headers that also match another valid link target?

If the user provides an explicit link target, I would expect that to take precedence. In general, I would expect the "link symbol" to follow some sensible scoping rules:

Check the local comment block for any explicitly declared link targets
Check the package-level comment for any explicitly declared link targets
Check the package-level comment for any implicit link targets from headings
Check the global scope for any implicit link targets from Go declarations

earthboundkid · 2021-10-14T17:54:20Z

earthboundkid
Oct 14, 2021

Probably worth reading https://swift.org/blog/swift-docc/ to see if they have ideas worth stealing.

0 replies

jsshapiro · 2022-05-23T08:35:39Z

jsshapiro
May 23, 2022

1 reply

beoran May 23, 2022

I think you are a bit late to the party.

In stead of markdown, asciidoc is a lot more extensible and consistent. But i doubt we could find a way to combine good with asciidoc.

This comment has been hidden.

Sign in to view

This comment has been hidden.

Sign in to view

This comment has been hidden.

Sign in to view

This comment has been hidden.

Sign in to view

This comment has been hidden.

Sign in to view

This comment has been hidden.

Sign in to view

This comment has been hidden.

Sign in to view

This comment has been hidden.

Sign in to view

This comment has been hidden.

Sign in to view

doc comment revisions: headings, lists, and links #48305

rsc Sep 10, 2021 Maintainer

Goals and non-goals

Markdown is not the answer, but we can borrow good ideas

Headings

Lists

Links to URLs

Links to Go API documentation

Formatter

Additional API

Replies: 27 comments · 125 replies

rsc Sep 12, 2021 Maintainer Author

rsc Sep 12, 2021 Maintainer Author

dsnet Sep 12, 2021 Collaborator

rsc Sep 16, 2021 Maintainer Author

dmitshur Sep 10, 2021 Maintainer

rsc Sep 10, 2021 Maintainer Author

dsnet Sep 10, 2021 Collaborator

rsc Sep 10, 2021 Maintainer Author

rsc Sep 10, 2021 Maintainer Author

rsc Sep 10, 2021 Maintainer Author

jimmyfrasche Sep 10, 2021 Collaborator

rsc Sep 10, 2021 Maintainer Author

jimmyfrasche Sep 10, 2021 Collaborator

dsnet Sep 13, 2021 Collaborator

dsnet Sep 13, 2021 Collaborator

dsnet Sep 10, 2021 Collaborator

rsc Sep 12, 2021 Maintainer Author

dsnet Sep 10, 2021 Collaborator

rsc Sep 10, 2021 Maintainer Author

dsnet Sep 14, 2021 Collaborator

rsc Sep 16, 2021 Maintainer Author

This comment has been hidden.

This comment has been hidden.

This comment has been hidden.

This comment has been hidden.

dsnet Sep 10, 2021 Collaborator

dsnet Sep 10, 2021 Collaborator

rsc
Sep 10, 2021
Maintainer

Replies: 27 comments 125 replies

rsc Sep 12, 2021
Maintainer Author

rsc Sep 12, 2021
Maintainer Author

dsnet Sep 12, 2021
Collaborator

rsc Sep 16, 2021
Maintainer Author

dmitshur
Sep 10, 2021
Maintainer

rsc Sep 10, 2021
Maintainer Author

dsnet Sep 10, 2021
Collaborator

rsc Sep 10, 2021
Maintainer Author

rsc Sep 10, 2021
Maintainer Author

rsc Sep 10, 2021
Maintainer Author

jimmyfrasche
Sep 10, 2021
Collaborator

rsc Sep 10, 2021
Maintainer Author

jimmyfrasche Sep 10, 2021
Collaborator

dsnet Sep 13, 2021
Collaborator

dsnet Sep 13, 2021
Collaborator

dsnet Sep 10, 2021
Collaborator

rsc Sep 12, 2021
Maintainer Author

dsnet Sep 10, 2021
Collaborator

rsc Sep 10, 2021
Maintainer Author

dsnet Sep 14, 2021
Collaborator

rsc Sep 16, 2021
Maintainer Author

dsnet Sep 10, 2021
Collaborator

dsnet Sep 10, 2021
Collaborator