jakubmalek opened a new pull request, #14926: URL: https://github.com/apache/camel/pull/14926
# Description Hi, I'm using [camel-aws-sqs-source-kafka-connector](https://camel.apache.org/camel-kafka-connector/4.0.x/reference/connectors/camel-aws-sqs-source-kafka-source-connector.html), and recently I've found that I'm unable to poll all the messages from SQS queue in time. At first I wanted to increase the maximum number of messages polled in a single request. By default the client is configured to receive only `1` message and the maximum messages allowed by AWS is `10`. Here I had to add `maxMessagesPerPoll` parameter to the [Kamelet](https://github.com/apache/camel-kamelets/issues/2118) in order to configure it for the connector. _I'm mentioning the Kamelet because the parameter there has `maximum: 10` but this change renders that constraint obsolete._ Without the Kamelet there is a workaround to configure the `queueNameOrArn` with `?maxMessagesPerPoll=10&` suffix, to sneak in the parameter to the endpoint. But even with that and enabled `greedy` polling I was unable to keep up, and poll all messages in time. Increasing the number of concurrent polling threads is an option here, but I'm interested in processing the messages in the correct order. Looking at the Spring implementation, I've found that they are invoking receive requests with multiple threads and then merging the results together: https://github.com/awspring/spring-cloud-aws/blob/a81b879d1a6b46f8a26794b98a601f7450d84907/spring-cloud-aws-sqs/src/main/java/io/awspring/cloud/sqs/listener/source/AbstractSqsMessageSource.java#L194-L198 I think a similar approach could be applied in Apache Camel, where the number of concurrent requests is calculated from `ceil(maxMessagesPerPoll / 10D)` formula. ## Changes * I've moved the logic for invoking the SQS requests into a dedicated class `Sqs2PollingClient` * If the `maxMessagesPerPoll` is greater than 10, it executes multiple requests asynchronously and merge the results together * To maintain the right ordering, the messages are ordered by the sequence number attribute if available * In case of queue auto-creation I made sure that only a single request handling missing queue error will create the queue * In case of recently deleted queue, I've removed the `Thread.sleep(30_000)` in favor of postponed creation with next polling call after `30_000` back-off time has passed * I also noticed that `ReceiveMessageRequest.Builder#attributeNamesWithStrings` is now deprecated, so I've replaced it with `ReceiveMessageRequest.Builder#messageSystemAttributeNamesWithStrings` # Target - [ ] I checked that the commit is targeting the correct branch (note that Camel 3 uses `camel-3.x`, whereas Camel 4 uses the `main` branch) # Tracking - [ ] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it). _Do I need create a ticket for that ? I don't have the account in JIRA :)_ <!-- # *Note*: trivial changes like, typos, minor documentation fixes and other small items do not require a JIRA issue. In this case your pull request should address just this issue, without pulling in other changes. --> # Apache Camel coding standards and style - [ ] I checked that each commit in the pull request has a meaningful subject line and body. <!-- If you're unsure, you can format the pull request title like `[CAMEL-XXX] Fixes bug in camel-file component`, where you replace `CAMEL-XXX` with the appropriate JIRA issue. --> - [ ] I have run `mvn clean install -DskipTests` locally and I have committed all auto-generated changes <!-- You can run the aforementioned command in your module so that the build auto-formats your code. This will also be verified as part of the checks and your PR may be rejected if if there are uncommited changes after running `mvn clean install -DskipTests`. You can learn more about the contribution guidelines at https://github.com/apache/camel/blob/main/CONTRIBUTING.md --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@camel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
