bobhan1 opened a new pull request, #69606:
URL: https://github.com/apache/doris/pull/69606
### What problem does this PR solve?
Related PR: #67989
A transaction commit retry can scan the temporary rowsets left by concurrent
lazy publication, then start its write transaction after those rowsets and
their pending markers have been removed. The transaction is still COMMITTED, so
processing the old scan can publish the same rowsets again or return only part
of the transaction's partition versions.
Handle COMMITTED explicitly in the immediate, eventual and sub-transaction
commit paths. After the non-snapshot txn_info read in the actual write
transaction, release that transaction, submit or join the existing lazy task,
propagate its error, and restart after success. The next iteration returns the
durable VISIBLE transaction through the existing idempotent branch, with empty
version results so FE invalidates its version caches. A concurrent transition
from PREPARED to COMMITTED conflicts with the same txn_info read at KV commit.
Also clear the response before falling back from a failed immediate attempt
to lazy commit. Immediate may populate version results before its KV commit
fails with TXN_BYTES_TOO_LARGE; those results must not be combined with the
successful lazy response or retained in an already-visible response.
#### Commit structure
1. Add three deterministic real-FDB retry tests and strengthen the existing
fallback response test.
2. Resume COMMITTED transactions through lazy publication in all three
commit paths.
3. Clear failed immediate response data before lazy fallback.
#### Validation
- Before the repair, all four focused cases failed. The immediate retry
produced real version-3 duplicates; the lazy retry attempted to mark COMMITTED
again; lazy task failure was reported as success; fallback returned duplicate
partition versions.
- After the state-handling commit, the three new retry cases passed and only
the independent fallback case remained failing.
- Final ASAN run of the complete `txn_lazy_commit_test`: **30 passed, one
pre-existing disabled, zero failures/errors**. FDB 7.1.57 was deployed as a
private instance with the repository's `tools/fdb/fdb_ctl.sh`; its processes
were stopped and temporary configuration restored afterward.
- Changed-line clang-format 16 checks, build hygiene and `git diff --check`
passed. Clang-tidy was not run.
The tests synchronize actual partition cleanup and final visibility with
bounded gates, compare business metadata including statistics, check rowset
identities and versions, and verify the empty VISIBLE response used by FE cache
invalidation. The lazy retry's unfixed-path hook fails the test before its
existing DCHECK terminates the whole binary.
### Release note
Prevent duplicate publication and incomplete version responses when
transaction commit retries race lazy publication.
### Check List (For Author)
- Test
- [x] Unit Test
- Behavior changed:
- [x] Yes. Finish existing lazy publication for COMMITTED retries and
discard failed immediate response data before fallback.
- Does this need documentation?
- [x] No. No new user-facing setting or syntax.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
