sniff the MIME ContentType and seek at the start afterward (by buffering the first 512 bytes)

Isn't it seeking to the start already?

(Update) This is the problem (also explained in #48781):

CC @neild @bradfitz

I don't think we want the complexity of wrapping a non-io.Seeker in something that tries to implement Seek.

If you want to serve a non-seekable filesystem, like the contents of a tar, you can do so fairly simply with a handler that opens the file and uses io.Copy to serve it. This won't handle Range requests (serving the entire file in response to one), but I believe it will handle Content-Type detection.

But it doesn't seem unreasonable to relax the requirement that fs.File files served by net/http don't need to implement io.Seeker, especially since #51971 is adding ServeFileFS.

If we do that, then I think the way is to change serveContent to not require a seekable input. It can defer content-type detection to ResponseWrite.Write (I'm not sure why it isn't doing that now--historical accident, or is there a better reason?), and either ignore Range requests for non-seekable files or implement them by discarding the skipped portions of the file.

I think this could also be achieved by a wrapper FS that would wrap Files to implement "stream seek" (forward seek by discarding or backwards seek not further than size of buffered last written bytes).

I'm not sure why it isn't doing that now--historical accident, or is there a better reason?

Answering my own question: ServeContent can detect a content type when the response doesn't include the first bytes of the file, such as when responding to a HEAD request or a range request.

If we do that, then I think the way is to change serveContent to not require a seekable input. It can defer content-type detection to ResponseWrite.Write (I'm not sure why it isn't doing that now--historical accident, or is there a better reason?), and either ignore Range requests for non-seekable files or implement them by discarding the skipped portions of the file.

I think my prototype is implementing this suggestion.

The ioFileSeekFaker struct is just a way to have this functionality implemented without having to add more logic to the already quite complex serveContent. Besides it only slows down the non-seeker case. Implementing this inside serveContent would mean something like:

• move the DetectContentType buffer at the top level, to be able to serve it later (sendContent = io.MultiReader(bytes.NewReader(ctypeBuf), content))
• add the io.CopyN(io.Discard, ...) on "simple" range request
• add the io.CopyN(io.Discard, ...) on "multiple" range request and stop on non-ascending range (or respond with StatusRequestedRangeNotSatisfiable immediately)

PS: I think supporting DetectContentType and "simple" range request would cover most usecases (I have never seen multi-range requests in the wild).

I don't believe this is a good idea. Essentially everything passed to net/http should implement Seek. Otherwise range requests fail, and if range requests fail, then downloads can't be resumed, structured files can't be fetched incrementally, and so on.

Chrome and most browsers that preview PDF files fetch the specific sections use range requests to fetch the specific file sections they need to render the current page, instead of having to download the entire file just to show page 1. Long ago, before Go handled range requests well, I noticed that reading PDFs on Go servers was incredibly slow. It turned out this was because Chrome was assuming that range requests are always supported, and so it would do a range request for a specific block of the file, Go would send back the entire file, and Chrome would extract the one block it wanted. Then the next block that needed to be read, same thing. It turned viewing a single page from what should have been sub-linear amounts of bandwidth to almost quadratic. Maybe Chrome has been fixed now, but that experience taught me that in the modern world you're just not a real web server if you don't support range requests.

If we did the "seek by reading and discarding" implementation of range requests, that would fix the bandwidth issue in the PDF failure but not the overall cost. It would create a nice low-bandwidth denial-of-service for the server: a client could connect and ask for the very last byte of the file and cause the server to process the entire compressed data stream.

Given that modern web servers must support range requests and range requests must have Seek to be implemented efficiently, it seems OK to me to leave things as they are and strongly encourage fs.File implementations used with net/http to implement Seek.

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

Chrome and most browsers that preview PDF files fetch the specific sections use range requests to fetch the specific file sections they need to render the current page, instead of having to download the entire file just to show page 1.

Apparently it only performs "single-range" requests. Or do you know of any "multiple-range" requests widely used?

It would create a nice low-bandwidth denial-of-service for the server

Yes, this has been acknowledged in the first message. A mitigation would be to limit the max possible seek, however it would mean that the "good" actors will have to download the entire file in case they make such a range-request (making for a terrible user experience, as you noted).

strongly encourage fs.File implementations used with net/http to implement Seek

Currently it is more of an obligation (than just a mere encouragement :)

My current understanding is that such "hack" should be better served by a package outside of the stdlib (with sufficient warning regarding DoS/bad user experience).

And regarding my first argument "make it stable", it is probably not even relevant since the numbers of bytes read for MIME sniffing is unlikely to change in the future.

Based on the discussion above, this proposal seems like a likely decline.
— rsc for the proposal review group

No change in consensus, so declined.
— rsc for the proposal review group