wushilin opened a new pull request, #13:
URL: https://github.com/apache/doris-sdk/pull/13
## Problem
The Go stream load SDK only retried HTTP 429/408/5xx. Doris reports almost
every load failure as **HTTP 200 + `Status: Fail`** — including transient ones
such as `[E-235] too many versions`, `[E-238] too many segments`, `current
running txns on db ... larger than limit`, memory limit exceeded and
txn/publish timeouts — so none of those were ever retried. The Flink connector
(`DorisBatchStreamLoad.load()`) retries every failed load up to
`sink.max-retries` with a fresh `label_N`.
## Changes
- **`classifyResponseError`**: every non-success outcome (non-2xx HTTP, or
2xx whose `Status` is not `Success`/`Publish Timeout`) is now retriable,
matching the Flink connector. The HTTP status code is no longer used to guess
retriability. The only exception is `Label Already Exists` +
`ExistingJobStatus: RUNNING`, which stays *ambiguous* and is resolved by
polling `get_load_state`.
- **`Config.MaxRetries` / `LoaderConfig.max_retries`**: default 3 (same as
`sink.max-retries`). A batch is sent at most `MaxRetries+1` times, still
bounded by `DorisUploadTimeout`. `-1` disables retries.
- **Fresh label per retry**: `<label>_<attempt>`, mirroring Flink's `label +
"_" + retry`.
- `streamLoadError.Error()` includes `ErrorURL` (like the Flink error
message) and keeps both the wrapper message and the last Doris response message.
- README: new config row, rewritten failure table, "Retry policy" section.
Deliberately **kept** from the Go SDK: ambiguous outcomes (response lost
after the request was sent, or label `RUNNING`) are resolved via
`get_load_state` before any retry — `VISIBLE`/`COMMITTED` is success,
`ABORTED`/`UNKNOWN` retries — so data is never loaded twice. Flink batch mode
re-sends blindly in that situation.
## Tests
- Updated `TestClassifyResponseError` table; new fakeSender tests for
`MaxRetries`, fresh labels, `-1`, and that ABORTED-poll retries count against
the cap.
- New `mock_backend_test.go`: a scripted mock Doris (`_stream_load` +
`get_load_state`, dropped TCP connections, FE 307 → BE redirect) driving the
real `httpSender` for each scenario:
- `Fail` (too many versions) retried with fresh label, same body, auth kept
- txn-limit `Fail` exhausts `MaxRetries`, last Doris message + ErrorURL
surfaced
- non-2xx retried; `MaxRetries=-1` fails fast; `DorisUploadTimeout` bounds
retries
- connection refused retried without polling
- dropped response + `VISIBLE` → success **without re-send**;
`ABORTED`/`UNKNOWN` → retried; `PREPARE`/`PRECOMMITTED` polled until terminal;
poll timeout → failure without re-send
- `Label Already Exists`: `FINISHED` → success, `RUNNING` → polled (then
success or retry), other → retried
- retry after FE redirect goes back through the FE with a new label and
keeps basic auth
- concurrent batches keep independent retry state
`go vet`, `go test ./...` and `go test -race ./...` pass; the mock tests
were run with `-count=5` to check for flakiness.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01UzpE493uFnJ1UQWx5N55sh
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
