metrics for the standard library #61638
Replies: 9 comments 13 replies
-
|
for the net or net/http packages, it will be useful to know about the number of incoming requests, request rates, response times, maybe insight about the connection and memory usages, then error metrics as well. |
Beta Was this translation helpful? Give feedback.
-
|
Some of those could be broken down by response code, and having access to that info might be helpful as well. (ie 500's p99 etc.) |
Beta Was this translation helpful? Give feedback.
-
|
A lot of these metrics could be captured at a HTTP middleware layer. If the net/http modules themself provide metrics I think it should probably be ones that must be done there (e.g. memory, fine grained timings) and others implemented in a separate module as middleware. |
Beta Was this translation helpful? Give feedback.
-
|
Connection Statistics: For the net package, it would be beneficial to have metrics related to the number of open connections, connection states (idle, active, closed), connection errors, etc. These metrics could aid in diagnosing issues related to network connections and potentially optimize connection management. HTTP Performance Metrics: For the net/http package, metrics on request/response times, the number of active connections, and error rates could be insightful. This information could be used to analyze the overall health and performance of HTTP servers, identify slow endpoints, or detect potential performance regressions. Resource Utilization: Metrics that provide insights into resource utilization, such as memory and CPU consumption for specific HTTP handlers or net connections, could be valuable for optimizing the efficiency of servers and identifying resource-heavy components. Error Metrics: Knowing the frequency and types of errors encountered in the net and net/http packages could help pinpoint common issues and improve error handling strategies. TLS Handshakes: For secure connections, having metrics related to TLS handshakes (e.g., handshake duration, handshake failures) could aid in understanding the security overhead and optimizing TLS configuration. |
Beta Was this translation helpful? Give feedback.
-
http server:
http client:
something about the connection pools sql
something about the connection pools for distributions, it would be great if they could be exponential rather than fixed bucket. opentelemetry have some experimental conventions https://opentelemetry.io/docs/specs/otel/metrics/semantic_conventions/http-metrics/ |
Beta Was this translation helpful? Give feedback.
-
|
For sql, I'd also include information about the sql connection pool and connection state transitions to help appropriately tune better similar to pgBouncer. |
Beta Was this translation helpful? Give feedback.
-
|
for http paths, typically I've seen it masked by (templated) handler route, either within the instrumentation sdk, or later in post processing (eg using opentelemetry collector) |
Beta Was this translation helpful? Give feedback.
-
Notably, the (Note that Otel semantic conventions are going through a massive change from v1.21 and beyond, so |
Beta Was this translation helpful? Give feedback.
-
|
Ya, the metric should key on the pattern which was matched, not the actual HTTP Path |
Beta Was this translation helpful? Give feedback.
-
|
being able to get the pattern back out to do this kind of instrumentation is exactly what I had in mind when proposing #61551. I think having aggregation follow routing is the only way to aggregate requests in a way that's semantically meaningful |
Beta Was this translation helpful? Give feedback.
-
|
@seankhliao, it seems like many of those can be done already, without adding anything to the standard library. I'm not sure if you can measure panics recovered or rejected malformed requests, and I know the connection pool stuff would need internal support. But that's it—everything else you mention can be done outside. Am I wrong about that? |
Beta Was this translation helpful? Give feedback.
-
|
It certainly can be done outside std, but it would be nice not to need to import https://pkg.go.dev/github.com/felixge/httpsnoop every time for the server side, not sure about client side libraries. Plus there are ordering issues which you have to be very careful with when chaining middleware together. If #61410 were to be accepted, it would be a shame if you can't easily get back the parameterized pattern for use as the cleaned path attribute for the metric. More importantly, I'd like it to be observable by default. Having standard metrics that instrumentation libraries can hook into mean I can trust the numbers mean the same thing, regardless of which way they were reported into the monitoring backend. |
Beta Was this translation helpful? Give feedback.
-
|
If you look at https://github.com/open-telemetry/opentelemetry-go-contrib/tree/main/instrumentation , instrumentation needs be (re)written for every http router people may use, with varying levels of feature completeness. Having it in the server just after it's parsed but before it's routed would instrumentation libraries only have to be written once. |
Beta Was this translation helpful? Give feedback.
-
|
You've got a point. Also agree to it being observable by default. |
Beta Was this translation helpful? Give feedback.
-
|
Same question for @niiadoteibrown and @LinuxSploit. The ones you can't already get seem to be about the underlying connections, TLS handshakes, and handler-specific or connection-specific resource usage. What am I missing? |
Beta Was this translation helpful? Give feedback.
-
|
Seems about right, though insight about the underlying connections and connection-specific resource usage will be very useful. Error metrics as pointed out by @LinuxSploit can really improve error handling strategies. Having the net or net/http emit their own detailed metrics on the stated areas, like you said, the ones we can't already get, can go a long way to ease development process for example. |
Beta Was this translation helpful? Give feedback.
-
|
Is it possible to track connection reuse for incoming/outgoing requests? |
Beta Was this translation helpful? Give feedback.
-
|
👋🏽 That's an epic idea, especially for net and net/http packages. I would love to help! I'm part of the Prometheus team and I maintain https://github.com/prometheus/client_golang. It gives us quite an overview of what's used and how often, as the package is imported by over 38k packages in OSS. Related to
There is also https://github.com/mwitkow/go-conntrack for connection tracking (asked by @lrweck) That's only for the HTTP part, we could walk through other packages and identify what others use and built in OSS for those. Providing them natively by Go std packages might definitely improve accessibility, consistent semantics and convenience. However we should discuss how to also allow certain flexibility and to what degree, to make it useful. For example, middlewares allow you to change dimensions (e.g. code/method) per certain HTTP paths, or skip instrumentation for some paths/methods completely. |
Beta Was this translation helpful? Give feedback.
-
|
I wonder if #49881 is a useful precedent for how to decide which metrics are useful and appropriate here. That added a sum of That same data is available by enabling mutex profiles with Several of the metrics mentioned in this discussion already could be collected at the application layer by user-provided instrumentation on a |
Beta Was this translation helpful? Give feedback.
-
|
I've arrived here after posting #63494 I think request lifecycle stuff is reasonably well-serviced by I would say the highest priority areas to expand metrics on should be areas that are currently invisible, by which I mostly mean connection pools. The interface exposed by |
Beta Was this translation helpful? Give feedback.
-
|
Note that |
Beta Was this translation helpful? Give feedback.
-
The runtime/metrics package exposes wealth of detailed information about the Go runtime. Would other packages in the standard library benefit from their own metrics? What would be useful to know about the internal state, of, say, the
netornet/httppackages?Beta Was this translation helpful? Give feedback.
All reactions