Croway opened a new pull request, #30162:
URL: https://github.com/apache/camel/pull/30162

   JIRA: https://issues.apache.org/jira/browse/CAMEL-27013
   
   **What was wrong**
   
   When an instance cannot reach the database for longer than 
`lockMaxAgeMillis`, its keep-alive fails on every round and the leases of the 
keys it holds grow older than `lockMaxAgeMillis`. Once the database is back, 
the orphan check treats them as orphans by age only, also on the instance that 
holds them: `queryForInt` counts the row only when `createdAt >= now - 
lockMaxAgeMillis OR createdAt IS NULL`, and `insert()` takes the row over with 
`UPDATE ... AND createdAt < ?`. Neither looks at `processorNameMessageIdSet`. A 
copy of a message the instance is still processing (a redelivery, the same file 
polled again) is accepted and processed a second time on the same instance, 
until the next keep-alive. Found with fault-injection tests (Toxiproxy in front 
of a real PostgreSQL, the database unreachable for longer than 
`lockMaxAgeMillis` while a message is processed). CAMEL-26816 already fixed the 
same symptom for the keys processed before the outage (they are confirmed now); 
the keys in f
 light were still taken over.
   
   **The fix**
   
   - `queryForInt` returns 1 for a key in `processorNameMessageIdSet`, before 
the age check: a key this instance holds is not an orphan for this instance, 
whatever the age of its lease. `contains()` returns true and `add()` false for 
it; the next keep-alive refreshes its `createdAt` as before.
   - The keep-alive logs one WARN when it has not refreshed the leases for 
longer than `lockMaxAgeMillis` (while the instance holds keys), saying that 
other instances may now take them over, and one INFO when it refreshes them 
again. The ERROR of each failed round is unchanged.
   - Docs (Orphan Lock section of `sql-component.adoc`, and the catalog copy): 
the new case, and what an outage longer than `lockMaxAgeMillis` means across 
instances. Another instance that reaches the database while this one cannot (a 
partition), or right after the outage before the next keep-alive, cannot tell 
this instance from one that crashed and takes the lease over; this is a 
limitation of a lease without fencing, which this PR documents (choose 
`lockMaxAgeMillis` longer than the longest outage to survive) and does not 
change. Possible follow-ups are in the JIRA.
   
   **Tests**
   
   - New `JdbcOrphanLockAwareIdempotentRepositoryOutageTest` (H2, two 
repository instances; node A's `DataSource` throws during the "outage", the 
lease is aged with SQL as the other tests of the class do):
     - a key in flight on node A is still a duplicate on node A after an outage 
longer than `lockMaxAgeMillis`, and the next keep-alive refreshes it;
     - a key whose remove and confirm failed during the outage is still a 
duplicate, and can be removed afterwards;
     - node B does take over the aged lease (the documented limitation), and 
node A still does not take the key again;
     - the WARN is logged once per outage, and the INFO when the keep-alive 
works again (real time, `lockMaxAgeMillis` 500 ms, Awaitility).
   - Without the fix, the first, second and fourth tests fail (`add()` returns 
true, no WARN); with it they pass.
   - Two existing assertions checked the age of a lease with `contains()` of 
the instance that holds the key, which is the behaviour changed here: 
`JdbcOrphanLockAwareIdempotentRepositoryTest.testLockKeepAliveWorks` now checks 
it from a second instance, and 
`JdbcOrphanLockAwareIdempotentRepositoryRestartTest.testRestartKeepsUpdateTimestampQuery`
 checks the refreshed `createdAt`.
   - All camel-sql unit tests pass.
   
   No upgrade-guide entry: no option or default changes; only a key the 
instance itself still holds is no longer taken over by that instance.
   
   **For reviewers**
   
   - The check is in `queryForInt`, so `contains()` and `add()` agree, and a 
held key no longer needs a database round trip. `insert()` is reached only when 
`queryForInt` returns 0, so its takeover is not changed.
   - A held key whose row was removed by someone else (a manual purge) is now a 
duplicate on the instance that holds it, until that instance removes or 
confirms it.
   
   _Claude Code on behalf of Croway_
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to