liaoxin01 opened a new pull request, #66557:
URL: https://github.com/apache/doris/pull/66557
### What problem does this PR solve?
Problem Summary:
Reading from object storage fails from time to time with
```
[INTERNAL_ERROR]failed to read from <key>: Failed to flush response stream
(eof: 0, bad: 1) code=-1 type=1, request_id=failed to read
```
and succeeds when the same statement is run again. It has been hit by queries
reading a rowset, by compaction, by an outfile export and by the download of
an
inverted index, always on an object storage that was answering `429` or
`503` at
that moment.
`S3ObjStorageClient::get_object()` hands the buffer of the caller to the SDK
as
the response stream of the request, sized exactly like the requested range.
The
SDK writes the body of every response into that stream, the body of an error
response included. The XML document of a `429 SlowDown` is a few hundred
bytes,
so a small ranged read cannot hold it - the read of the footer of a packed
file
asks for 12 bytes. `PreallocatedStreamBuf` does not implement `overflow()`,
so
the stream turns bad, the write callback of curl reports a short write and
curl
aborts the transfer with `CURLE_WRITE_ERROR`.
The status code of the response is lost from there on:
`CurlHttpClient::MakeRequest()` reads `CURLINFO_RESPONSE_CODE` only when curl
succeeded, so the code stays at `REQUEST_NOT_MADE` (-1), and the flush check
at
the end of the same function replaces the retryable `NETWORK_CONNECTION`
classification with `INTERNAL_FAILURE` (1).
`S3CustomRetryStrategy::ShouldRetry()`
declines to retry an error classified that way, and so does
`S3FileReader::read_at_impl()`, which retries on `429` alone. A throttling
error
the server asked us to retry cancels the statement of the user instead,
which is
why running it again works.
This also means the error carries no evidence of what really happened: the
code
of the response, the exception name and the request id of the object storage
are
all gone by the time the message is built.
The fix lets the response stream grow: the body is written into the buffer
of the
caller as long as it fits, which is the case for every successful ranged
read and
keeps that path free of copies, and the remainder spills into a buffer of the
stream itself, truncated at 1MB because only error documents are expected to
overflow. The stream never turns bad, so curl completes the transfer, the SDK
records the real status code and parses the error out of the body, and both
the
retry of the SDK and the retry of `S3FileReader` on `429` work again.
A server or a proxy answering a ranged read with the whole object overflows
the
buffer as well. Such a read is still rejected, by the length check that
follows
the request, and now with a message that says so.
Two misleading messages are fixed along the way:
- `request_id=failed to read` is not a request id of the object storage. It
is
the string `S3FileReader` appended behind the empty request id of a failure
raised by the client itself. The append is dropped and an empty request id
is
printed as `<empty>`.
- The message of a failed read named neither the bucket nor the offset,
leaving
`failed to read from :` in the log whenever the key was empty.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- `be/test/io/fs/s3_response_stream_test.cpp` covers a body that fits, an
error body overflowing in one write, across writes and character by
character, the truncation of an oversized body, the rewind the SDK does
before parsing an error, and an empty body.
- Not tested end to end against a rate limited object storage.
- Behavior changed: No
- Does this need documentation: No
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
