mirror of
https://github.com/apache/doris.git
synced 2026-11-04 16:03:23 +00:00
Xin Liao
6b5c53fdb4
Reading from object storage fails from time to time with ``` [INTERNAL_ERROR]failed to read from <key>: Failed to flush response stream (eof: 0, bad: 1) code=-1 type=1, request_id=failed to read ``` and succeeds when the same statement is run again. It has been hit by queries reading a rowset, by compaction, by an outfile export and by the download of an inverted index, always on an object storage that was answering `429` or `503` at that moment. `S3ObjStorageClient::get_object()` hands the buffer of the caller to the SDK as the response stream of the request, sized exactly like the requested range. The SDK writes the body of every response into that stream, the body of an error response included. The XML document of a `429 SlowDown` is a few hundred bytes, so a small ranged read cannot hold it - the read of the footer of a packed file asks for 12 bytes. `PreallocatedStreamBuf` does not implement `overflow()`, so the stream turns bad, the write callback of curl reports a short write and curl aborts the transfer with `CURLE_WRITE_ERROR`. The status code of the response is lost from there on: `CurlHttpClient::MakeRequest()` reads `CURLINFO_RESPONSE_CODE` only when curl succeeded, so the code stays at `REQUEST_NOT_MADE` (-1), and the flush check at the end of the same function replaces the retryable `NETWORK_CONNECTION` classification with `INTERNAL_FAILURE` (1). `S3CustomRetryStrategy::ShouldRetry()` declines to retry an error classified that way, and so does `S3FileReader::read_at_impl()`, which retries on `429` alone. A throttling error the server asked us to retry cancels the statement of the user instead, which is why running it again works. This also means the error carries no evidence of what really happened: the code of the response, the exception name and the request id of the object storage are all gone by the time the message is built. The fix lets the response stream grow: the body is written into the buffer of the caller as long as it fits, which is the case for every successful ranged read and keeps that path free of copies, and the remainder spills into a buffer of the stream itself, truncated at 1MB because only error documents are expected to overflow. The stream never turns bad, so curl completes the transfer, the SDK records the real status code and parses the error out of the body, and both the retry of the SDK and the retry of `S3FileReader` on `429` work again. A server or a proxy answering a ranged read with the whole object overflows the buffer as well. Such a read is still rejected, by the length check that follows the request, and now with a message that says so. Two misleading messages are fixed along the way: - `request_id=failed to read` is not a request id of the object storage. It is the string `S3FileReader` appended behind the empty request id of a failure raised by the client itself. The append is dropped and an empty request id is printed as `<empty>`. - The message of a failed read named neither the bucket nor the offset, leaving `failed to read from :` in the log whenever the key was empty.