Really enthusiastic about this, seems like an excellent improvement to me!

To support this API, the net/http package adds a new method to Request:

package http

func (*Request) PathValue(wildcardName string) string

Is the status of setting PathValues still as mentioned here: #60227 (comment) - omitted for minimal proposal?

(I convinced myself something like a one-shot Request.ParsePathValues(pattern string), slicing the clean URL path, was the least hazard-free thing. I guess I'd be surprised if Handlers start to depend on PathValue but there was no way to populate independently of ServeMux.)

@AndrewHarrisSPU, yes, I deliberately omitted a way to set path values to be minimal. But I do see what you mean about having no way to set it. At the very least, tests should be able to set path values on a request without going through the matching process.

I don't think that exposing the parsing logic should be done as a method on Request; I'm not sure it should be done at all.

I would go with something simpler. Perhaps:

// SetPathValue sets key to value, so that subsequent calls to r.PathValue(key) return value.
func (r *Request) SetPathValue(key, value string)

Could you expose the pro and con about adding SetPathValue in the current proposal ?

@flibustenet, the pros of being able to set values are that it would be easier to write tests, and it would make it possible for middleware to transmit information via the path values.

That second pro is also a con: once we allow setting, then we have introduced this bag of key-value pairs to http.Request, which anyone can use for whatever they want. Middleware could overwrite path wildcards that a handler depended on. Or the storage could be co-opted for something completely unrelated to the path, much as people abuse contexts (only here it's worse, since setting a path value is a destructive change, so it is visible to everyone who has a pointer to the same request).

As someone who's played around with various Go routing techniques, I really like this, and think it'll more or less eliminate the need for third-party routers, except for very specialized cases.

I've tested it using the reference implementation in my go-routing suite, and it was very easy to use. I just copied the chi version and tweaked it a bit. Lack of regex matching like [0-9]+ means you have to error-check any strconv.Atoi calls for integer path values, but that's not a big deal. I much prefer the simplicity of just handling strings rather than fancy regex and type features.

A few comments/questions:

1. One thing I noticed is that old versions of ServeMux accept patterns like HandleFunc("GET /foo", ...). This would mean code written for the enhanced ServeMux would compile and appear to run fine on older versions of Go, but of course the routes wouldn't match anything. I think that's a bit error-prone, and wonder if there's a workaround. Possibilities are:
   • Instead of overloading the existing methods, use new methods like HandleMatch and HandleMatchFunc. Obviously a fairly big departure from the current proposal, but at least clear what's old vs new.
   • Update old versions of Go (maybe the last couple of versions) to panic on patterns that have a space in them, or perhaps patterns that use any of the enhanced features. This might be too much of a breaking change.
2. How would you customise the 405 response? For example, for API servers that need to always return JSON responses. The 404 handler is sort of easy to override with a catch-all / pattern. But for 405 it's harder. You'd have to wrap the ResponseWriter in something that recorded the status code and add some middleware that checked that -- quite tricky to get right.
3. {$} is a bit awkward. Is $ by itself ever used in URLs, and would it be backwards-incompatible just to use it directly?
4. Regarding SetPathValue: I agree something like this would be good for testing. Perhaps instead of being a method on Request, it could use httptest.NewRequestWithPathValues or similar, so it's clearly a function marked for testing and not general use? I'm not sure how that would be implemented in terms of package-internals though.

In Caddy, we have the concept of request matchers. I'm seeing isn't covered by this proposal is how to route requests based on headers. For example, something commonly done is multiplexing websockets with regular HTTP traffic based on the presence of Connection: Upgrade and Upgrade: websocket regardless of the path. What's the recommendation for routing by header?

Another learning from Caddy is we decided to go with "path matching is exact by default" and we use * as a fast suffix match, so /foo/* matches /foo/ and /foo/bar but not /foo, and /foo* matches /foo and /foo/bar and /foobar. Two paths can be defined for a route like path /foo /foo/* to disallow /foobar but still work without the trailing slash. We went with this approach because it gives ultimate flexibility by having routes that match exactly one URL nothing else.

Finally, there's an experimental https://developer.mozilla.org/en-US/docs/Web/API/URL_Pattern_API which has a lot of great ideas for matching URL patterns. I'd suggest considering some of the syntax from there, for example ? making the trailing slash optional.

Although wildcard matches occur against the escaped path, wildcard values are unescaped. For example, if a wildcard matches a%2Fb, its value is a/b.

Does that apply for "..." wildcards as well? If so, would that mean the handler couldn't differentiate literal slashes from escaped slashes in "..." wildcard values?

@muirdm: Yes, in a "..." wildcard you couldn't tell the difference between "a%2Fb" and "a/b". If a handler cared it would have to look at Request.EscapedPath.

@francislavoie: Thanks for all that info.

The Caddy routing framework seems very powerful. But I think routing by headers is out of scope for this proposal. Same for many of the features Caddy routing and the URL pattern API you link to. I think we expect that people will just write code for all those cases. By the usual Go standards, this proposal is already a big change.

@benhoyt, thanks for playing around with this.

1. I don't know what to do about older versions of ServeMux.Handle accepting "GET /". We can't change the behavior in older Go versions. If we added new methods we could make Handle panic in the new version, which I guess would solve the problem for code that was written in the new version but ended up being compiled with the old version. But that's probably rare. Maybe a vet check would make sense.

2. Again, no good answer to how to customize the 405. You could write a method-less pattern for each pattern that had a method, but that's not very satisfactory. But really, this is a general problem with the Handler signature: you can't find the status code without writing a ResponseWriter wrapper. That's probably pretty common—I know I've written one or two—so maybe it's not a big deal to have to use it to customize the 405 response.

3. $ is not escaped in URLs, so we can't use it as a metacharacter by itself.

4. I like the idea that only httptest can set the path values. We can always get around the visibility issues with an internal package, so I'm not worried about that.

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

I see two others usages of SetPathValue

To can start with standard router and switch to an improved one if needed without changing all the methods to retrieve the PathValue in the handlers. @jba you said that it's anyway a big work to switch the routers but I don't think so, I every time begin by standard router and only switch when needed without so much works, and we'll do even more with this proposal.

There is also the case where we want to route from a handler to an other handler or middleware and adjust the PathValues. For example with a complex parsing /api/{complex...} we'll want to expand complex in some PathValues and like to use them with the same method PathValue(). Or we'll need to use request.Context bag and it's worse.

It will prevent the temptation to make mux overcomplicated when we can improve it easily by adding/setting PathValues. We already do this with Form.

One small addition to this I would really like to see is the option to read back from a request which pattern it was matched with. We find this functionality in other frameworks really helpful for logging and metrics, and for resolving ambiguity in how a request reached a handler. Where it doesn't exist, we add it by using a middleware to add a request tag string to the context, but this causes a stutter wherever it's declared and pollutes logical concerns (routing) with cross-cutting ones (observability) in a way that makes otherwise-simple routing code harder to read and write.

Currently this doesn't exist in net/http, but could sometimes be fudged if a handler separately knew the path prefix it was registered under. With this change the template could be even more different from the URL's Path, so being able to explicitly pull it out becomes more important.

My proposed API would be only adding a Pattern string field to http.Request, which would be set by the router. I realise this brings up similar potential concerns about providing more places for data to be smuggled into requests, but this placement would match most of the information already on a Request, and would be able to be set by other frameworks as well.

Similar functionality exists already in gorilla/mux and chi.

I realise this change is kind of orthogonal to the routing additions, although I think it is made more pressing by them. Happy to open this as a separate issue if that's preferred.

@flibustenet, good point about Request.Form already being a bag of values. In fact, there are several others as well: PostForm, MultipartForm, Header and Trailer. Given those, SetPathValue seems relatively benign.

@treuherz, I see the usefulness for retrieving the pattern, but it feels out of scope for this issue. I'd suggest a separate proposal.

Regarding response codes, would a 405 response automatically include the "Allow" header with appropriate methods?

What is the state of the request's method?

package http

func (*Request) PathValue(wildcardName string) string

I can see that in reference implementation it's part of the ServeMux

package muxpatterns

func (mux *ServeMux) PathValue(r *http.Request, name string) string

IMO, it should stay this way.

Alternatively, if the method is still intended to be a request method, this proposal should make a commitment to allow setting the path values by 3rd party routers.

It's part of ServeMux in the reference implementation because I couldn't see how to modify http.Request without making muxpatterns unusable with net/http.

If it stayed there then every handler would have to have some way to access its ServeMux. But handlers only take ResponseWriters and Requests.

We are leaning to adding a Set method. As was pointed out, it wouldn't be the only bag of values in Request.

@iamdlfl, apparently the spec for a 405 says we MUST do it, so we will. I'm just not sure what to put there if there is a matching pattern that has no method. I guess all the valid HTTP methods?

Sounds good. And if someone doesn't define the optional method when registering the route, I don't think we'll have to worry about responding with a 405, right? If I'm understanding right, it will respond normally to any method type.

Good point!

Some common related features are naming patterns and taking a pattern and its arguments and outputting a url. This typically gets bundled as something you pass to templates so you can do something like {{ url "user_page" user.id }} so the template doesn't need to be updated if the named pattern changes from /u/{id} to /user/{id} or the like. I don't think this should necessarily be included per se but it would be nice if now or in a later proposal there were enough added so that things of this nature could be built. Sorry if this has come up I haven't been able to pay a great deal of attention to this proposal's discussion.

I like the idea of richer routing capabilities in the standard library, but I don't love how overloading Handle and HandleFunc to just accept a new string format makes Go programs using the new syntax backwards-incompatible in a way that can't be detected at compile time.

I like @benhoyt's suggestion of infixing Match into new method names, but I would prefer to also see the method+space pattern prefix omitted for two reasons:

1. Sometimes a route may wish to accept multiple (but not all) methods - does this string format allow this?
2. We have the HTTP methods conveniently defined as constants in net/http, but they can't be used here without some ugly and awkward string concatenation.

I would suggest these method signatures:

func (s *ServeMux) HandleMatch(pattern string, handler http.Handler, methods ...string)

func (s *ServeMux) HandleMatchFunc(pattern string, handler http.HandlerFunc, methods ...string)

If no methods are passed in the variadic arg, behave exactly like HandleFunc with the addition of wildcard matching, otherwise return a 405 if the request method doesn't match one in the methods slice. The method(s) being a dedicated but optional argument is self-documenting, IMO, compared to putting all that information into a single string.

This will feel very similar to the .Methods() chaining used commonly in gorilla/mux, which itself could be another approach to consider here.

@c9845

You could also just write a function that'll handle all of them:

func Route(method, path string) string {
  return fmt.Sprintf("%s %s", method, path)
}

Then you can use, for example, http.Handle(Route(http.MethodGet, "/"), ...). It's slightly more error prone, but makes it simpler to use the existing constants. It would be better if the constants had their own type so that the first argument could be type checked, but that would be a backwards incompatible change, so it's not likely.

@DeedleFake Thanks for the suggestion. I am kind of just seeing where this whole development/implementation/whatever ends up and then going from there. I imagine a lot of people are in the same boat.

is this feature going to be delivered in go 1.22?

@lucianboboc Yes, at least according to the provisional release notes for Go 1.22:

net/http

[...] TODO: https://go.dev/issue/61410: enhanced ServeMux routing
[...] TODO: https://go.dev/cl/528355: net/http: implement path value methods on Request; modified api/next/61410.txt

Change https://go.dev/cl/552195 mentions this issue: doc: fill out net/http.Request.PathValue section

But one outcome of the discussion on this issue is that we could find no evidence that routing speed matters, and some evidence that it doesn't. If you have an application where routing speed matters, we'd love to hear about it.

@jba could you provide that evidence that it doesn't matter?
On projects I work on, almost always there is a discussion as to which router to choose. Mainly because of performance. It would be great if we had evidence that the performance of the router is not that relevant.
Thanks!

@phenpessoa What @jba likely meant is that, in most applications, execution time is dominated by I/O with the network. Although a router should preferably minimise or even eliminate heap allocations, its execution speed (unless an atrocious choice were made for the data structure used to represent the routes) is unlikely to factor much into the overall performance of the server.

@phenpessoa The burden of proof is typically on the side that claims that something matters: that a piece of software affects performance, that a drug affects the outcome of a disease, that adding a substance speeds up a reaction, and so on. The null hypothesis is always that nothing has any effect.

The evidence here, besides what @jub0bs mentioned, is that no one has come forward with any evidence to the contrary for real systems. You will certainly find that router speed matters for a toy server that does very little work, but what about production servers? I've asked several times without any response. It sounds like you may have some real examples, though. Would you mind doing some performance measurements and posting them here?

Double checking some behavior reported in Gophers slack.

Given this program, should a PUT to /foo return a 405 or should it hit the / handler? It currently hits the / handler.

@danp

The behaviour you're observing is in accordance with the spec:

if two patterns overlap (have some requests in common), then the more specific pattern takes precedence.

mux.HandleFunc("/", /* omitted */)
mux.HandleFunc("GET /foo", /* omitted */)

Pattern GET /foo is more specific than pattern / because the first matches only GET and HEAD requests to /foo whereas the second matches any request.

A PUT request to /foo is not matched by pattern GET /foo, but is matched by pattern /.

👍 thanks for the double check. Here's what is said about 405:

If no pattern matches a request, ServeMux typically serves a 404 (Not Found). But if there is a pattern that matches with a different method, then it serves a 405 (Method Not Allowed) instead.

Are we sure 405 should only be returned if no other pattern matches? That does seem to be different from at least chi. For example this program returns a 405 for a PUT to /foo.

I see mentions of customizing 405 responses but best I can tell nothing specifically about the order of precedence for when to return it. Apologies if I've missed that.

I believe this is WAI. By writing the pattern /, the server author is saying that they want to serve all requests, regardless of method or path. It would be wrong for any request to fail without running the handler for /.

If the author wants PUTs to /foo or anything else to return 405, they can do that by defining GET /, POST /, and so on, but not PUT /.

So if my understanding is correct, in order to serve a custom 404 for any unmatched request, I can't use the catch-all "/" route.
Therefore, I need something like a custom-wrapped ResponseWriter to capture the status code, and then, based on that, return the appropriate 405/404 response?
Sorry if I missed something.

I guess the 404 handler would need to be at GET /? Does seem unfortunate if there’s not a good way to set the 404 handler and still have 405 messages.

My understanding is that by setting the 404 handler to /, we lose the ability to handle 405 at all. All requests will end up at / even when we have a 'matching' route, albeit with a different method.

Will this change impact http.StripPrefix? When I was playing with muxpatterns and added a dynamic prefix, I found all my routes broke--makes sense for a proof of concept, but I would expect this to work in go1.22:

r.Handle("/{id}/*", http.StripPrefix("/{id}", idHandler))

For what it's worth, this replacement for http.StripPrefix works for me as a hack:

// stripDynamicPrefix adjusts the URL Path of the request by dynamically
// stripping the specified prefix.
func stripDynamicPrefix(prefix string, h http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		// Split the request path and prefix into segments.
		pathSegments := strings.Split(r.URL.Path, "/")
		prefixSegments := strings.Split(prefix, "/")

		matched := true
		for i, segment := range prefixSegments {
			// Skip dynamic segments.
			if strings.Contains(segment, "{") && strings.Contains(segment, "}") {
				continue
			}

			// Check for segment match.
			if i >= len(pathSegments) || segment != pathSegments[i] {
				matched = false
				break
			}
		}
		if !matched {
			http.NotFound(w, r)
			return
		}
		newPath := "/" + strings.Join(pathSegments[len(prefixSegments):], "/")

		// Create a new request with the updated path.
		r2 := new(http.Request)
		*r2 = *r
		r2.URL = new(url.URL)
		*r2.URL = *r.URL
		r2.URL.Path = newPath

		h.ServeHTTP(w, r2)
	})
}

Additionally, I was surprised that this did not route:

r.Handle("GET  /my-route", handler) // note the 2nd space!

I wanted to align my routes neatly, e.g.

r.Handle("GET    /my-route", handler1)
r.Handle("POST   /my-route", handler2)
r.Handle("DELETE /my-route", handler3)

For what it's worth, this replacement for http.StripPrefix works for me as a hack:

I think it's a good sign that keeping the ServeMux rules minimal still allows flexibility on top. The minimal approach may more frequently need some translation - starting from how one would like to structure everything about route configuration, resulting in ServeMux registration calls - but is a good fit for static-plus-path-variables kinds of routing.

Allowing spaces in the handle route string and understanding placeholders in StripPrefix both seem like pretty good ideas to me, but they should probably be new proposals in the issue tracker so people can debate whether there's some downside to them. I don't think either should be a blocker to Go 1.22. They can be add ons for Go 1.23.