wushilin opened a new pull request, #693:
URL: https://github.com/apache/doris-flink-connector/pull/693
# Proposed changes
Issue Number: close #xxx
## Problem Summary:
In batch mode (`sink.enable.batch-mode=true`), `DorisBatchStreamLoad.load()`
retries every failed attempt by re-sending the buffer under a new label
(`label_N`). That is correct when Doris answered with a definitive failure, but
when the failure is **not** an answer from Doris the first attempt may already
have been committed, and the retry loads the same data twice:
- socket timeout / connection reset / `NoHttpResponseException` after the
request body was sent;
- an unparseable or truncated 200 response;
- `Label Already Exists` with `ExistingJobStatus: FINISHED` — the earlier
attempt with this label was committed, yet it was treated as a failure and
re-sent under `label_1`.
On Duplicate-key tables these show up as duplicate rows; on Unique/Aggregate
tables they are masked by the key model.
### Fix
Before re-sending an attempt whose outcome is unknown, poll `GET
/api/{db}/get_load_state?label=...` on the configured FE nodes:
| state | action |
|---|---|
| `VISIBLE` / `COMMITTED` | data is in Doris → treat as success, **do not
re-send** |
| `ABORTED` / `UNKNOWN` | nothing was loaded → retry with a new label (as
before) |
| `PREPARE` / `PRECOMMITTED` | keep polling (1s → 5s backoff) until final or
the 5 min poll timeout; if still unresolved, **fail the load instead of
re-sending** |
- `Label Already Exists` + `FINISHED` is treated as already loaded;
`RUNNING` is resolved through `get_load_state`.
- Definitive failures — HTTP 200 + `Status: Fail` (e.g. `[E-235] too many
versions`), non-200 status codes, connection refused / unknown host / no route
— are retried immediately with a new label exactly as before.
`sink.max-retries` still bounds the retries.
- Group commit has no label, so its behaviour is unchanged.
- The 2PC streaming path (`DorisStreamLoad` / `DorisWriter`) is not touched:
it already handles this through precommit + label abort.
## Checklist(Required)
1. Does it affect the original behavior: (Yes) — batch mode no longer
re-sends a buffer whose earlier attempt was committed; a load whose label state
cannot be determined within 5 min now fails instead of being re-sent.
Definitive failures retry exactly as before.
2. Has unit tests been added: (Yes) — `TestDorisBatchStreamLoad` gains a
scripted mock of the Doris HTTP API (stream load PUTs + `get_load_state` GETs)
with 11 new cases: Status Fail retried without polling; connection refused
retried without polling; lost response + `VISIBLE` not re-sent (and auth header
on the poll); lost response + `ABORTED` / `UNKNOWN` retried with `label_1`;
pending state polled until final; unresolved state fails without re-send;
`Label Already Exists` `FINISHED` treated as loaded; `RUNNING` polled then
success / retried after `ABORTED`; `sink.max-retries` still bounds retries;
group commit unchanged. `testLoadFail` now uses a genuine `Status: Fail`
response since `Label Already Exists` + `FINISHED` is no longer a failure. Full
`flink-doris-connector-base` unit suite passes (431 tests) with checkstyle +
spotless.
3. Has document been added or modified: (No Need)
4. Does it need to update dependencies: (No)
5. Are there any changes that cannot be rolled back: (No)
## Further comments
The same gap existed in the Go stream load SDK (apache/doris-sdk#13 aligns
its retry policy with this connector and keeps its `get_load_state`
resolution); this PR brings the batch writer to the same guarantee: every
failed load is retried, but a load that may already be committed is never
blindly re-sent.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01UzpE493uFnJ1UQWx5N55sh
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
