allthingssecurity opened a new pull request, #28038: URL: https://github.com/apache/camel/pull/28038
# Description [CAMEL-25727](https://issues.apache.org/jira/browse/CAMEL-25727) The `aws-cloudtrail` consumer started each `LookupEvents` call at the newest event time it had seen. AWS documents a delivery delay, not an order: the AWS CloudTrail FAQ says "Typically, CloudTrail delivers an event within 5 minutes of the API call", and the user guide says for trails that this time is not guaranteed. With a delay that varies per event, an event can become visible after a newer one was already returned. Its event time is then before the cursor, and the consumer never routes it. Nothing is logged. Change: - the normal polls are unchanged: the same `LookupEvents` requests, from the newest event time seen; - new late-event lookup: every `lookBackInterval` (new option, duration, default 1 minute) the consumer also looks up the events from `lookBackWindow` (new option, duration, default 10 minutes) before now up to the newest event time seen, never before the consumer start, and routes those it has not routed yet (by event id). It sends one page per poll; a failed page is logged (WARN, the stack trace only at DEBUG) and does not affect the events of the normal lookup, the lookup starts again at the next interval; - the ids of routed events are kept with their event time and dropped once that time is before every later lookup, so the set only holds the events of the window; - `lookBackWindow=0` disables the lookup and gives exactly the previous behaviour. Defaults: 10 minutes is twice the "typically within 5 minutes" of the FAQ. 1 minute means a late event is routed about a minute after it becomes visible, for one extra request per minute on a quiet account. Cost: the late-event lookup adds `ceil(N / maxResults)` requests per `lookBackInterval`, at most one per poll, where N is the number of events (of the `eventSource`, if set) between `lookBackWindow` before now and the newest event time seen, and none when that time is older than the window. With the defaults: one request per minute up to 50 events in 10 minutes, 20 per minute with 1000 events in 10 minutes. AWS allows two `LookupEvents` requests per second per account and Region (120 per minute); the polls send at least one request every `delay` (500 ms by default, so up to about two per second, as on main), and one more per further page of new events, as before. With the default `delay` the polls alone can already use most of that limit; the late-event lookup adds one request per minute on a quiet account. The option description, the component page and the upgrade note state this. Upgrade guide (4.24): a note under `camel-aws-cloudtrail`. Tests: - New `CloudtrailConsumerLateEventTest` (5 tests, mocked `CloudTrailClient` that returns the events in [`startTime`, `endTime`] newest first, page by page; a test clock; polls called directly, no sleeps): - `lateEventWithEarlierEventTimeIsRoutedOnceByTheLookBack`: an event with an earlier event time becomes visible after a newer one and is routed exactly once, by the late-event lookup. Without the change it fails (`expected: <1> but was: <0>` at the poll at T+60 s, 3 of 3 runs). It uses the new `now()` hook, so it was run against main's consumer with that `@Override` removed (main ignores the test clock, which does not change this sequence) and the four tests of the new options left out. With the change, removing only the call of the late-event lookup fails it the same way. - `normalPollsSendTheSameRequestsAsBefore`: between two late-event lookups each poll sends only the lookup from the newest event time (all pages, no `endTime`), counted request by request. - `lookBackSendsOnePagePerPollAndDoesNotRouteAgain`: one page per poll, start at the consumer start, no event routed twice, id set empty once the window has passed. - `failedLookBackDoesNotLoseTheEventsOfThePoll`: a failing late-event lookup is logged, the events of the normal lookup of that poll are routed, and the next interval routes the late event. - `lookBackWindowZeroBehavesAsBefore`: only the normal requests, and the late event is not routed, as before. - With the change the camel-aws-cloudtrail tests pass: 7 run, 0 failures, 0 errors, 0 skipped (5 new). - Generated files (configurers, component JSON, catalog, component and endpoint DSL) regenerated. What is not changed: an event with an event time before the consumer start is still not routed, and an event that becomes visible more than `lookBackWindow` after its event time is still missed. The CloudTrail refresh task of camel-aws-secrets-manager moves its position in a similar way; not changed here. What was not run: the full root build; no test against AWS. # Target - [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch) # Tracking - [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it): [CAMEL-25727](https://issues.apache.org/jira/browse/CAMEL-25727). # Apache Camel coding standards and style - [x] I checked that each commit in the pull request has a meaningful subject line and body. - [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes. (I built and tested camel-aws-cloudtrail, including the formatter and import-sort plugins, and regenerated the component and endpoint DSL. I did not run the full root build.) # AI-assisted contributions - [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used. This PR was prepared with Claude Code (Claude Opus 5.5). The commit carries a `Co-Authored-By` trailer. _Claude Code on behalf of allthingssecurity_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
