kevinjqliu opened a new pull request, #3183:
URL: https://github.com/apache/iceberg-rust/pull/3183
## Which issue does this PR close?
None. Fixes a flaky integration test observed locally.
## What changes are included in this PR?
Switches the REST catalog fixture's SQLite database to WAL mode.
**Problem.**
`iceberg-catalog-loader::table_rename_suite::test_catalog_rename_table_across_namespaces::case_1_rest_catalog`
fails intermittently under parallel load with a server-side `SQLITE_BUSY
("database is locked")`. It passes in isolation and for every other catalog
backend. The fixture's JdbcCatalog sits on a file-backed SQLite database in the
default rollback-journal mode, where a committing writer needs an exclusive
lock that waits out every active reader. When two connections both hold a read
lock and try to upgrade, SQLite returns `SQLITE_BUSY` immediately rather than
waiting, to avoid deadlock. nextest runs every test in its own process, so the
loader suites plus the integration tests hit this server at exactly the
concurrency that triggers it.
**Why WAL.** WAL mode is the one setting that changes the shape of the
contention rather than the wait for it: readers keep reading the last committed
snapshot while the single writer appends to the log, so readers never block the
writer and the immediate-BUSY path no longer exists. Other options treat the
symptom: serializing the REST tests slows CI and misses the integration-tests
crate, a longer `busy_timeout` only moves the cliff, and client-side retries
are the wrong layer since rename is not idempotent.
`transaction_mode=IMMEDIATE` was also evaluated and does nothing here because
the JdbcCatalog runs every statement in autocommit.
Note the old URI's `mode=memory` suffix was part of the filename, not a
query parameter, so the database was already file-backed. The new path just
drops the misleading suffix.
## Are these changes tested?
Stress-tested against throwaway `apache/iceberg-rest-fixture:1.10.1`
containers with a harness that mirrors the flaky test (create two namespaces,
create a table, rename across namespaces, check, drop) from N concurrent
clients:
| config | first `SQLITE_BUSY` | max clients tested | `SQLITE_BUSY` total |
|---|---|---|---|
| old (rollback journal) | 16 clients | 128 | 9 / ~24k ops |
| `journal_mode=WAL` | none | 1024 | 0 / ~155k ops |
WAL also roughly halves p50 latency at 16+ clients, since reads no longer
queue behind writes. The existing loader suites pass unchanged against the new
config.
## AI Disclosure
Root-cause analysis, stress harness, and this PR were produced with
assistance from Claude Code; the change and the measurements were reviewed by
the author.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
