davsclaus commented on a change in pull request #629: URL: https://github.com/apache/camel-website/pull/629#discussion_r713919344
########## File path: content/blog/2021/09/camel-kafka-consumer-changes/index.md ########## @@ -0,0 +1,68 @@ +--- +title: "Upcoming changes to the consumer on Kafka component" +date: 2021-09-27 +draft: true +authors: [orpiske] +categories: ["Features", "Camel"] +preview: "Upcoming changes to the consumer on Kafka component" +--- + +We recently started to review, investigate and fix reliability issues with the [Kafka](https://camel.apache.org/components/latest/kafka-component.html) component. As part of that, we refactored the code to avoid problematic concurrency patterns, remove calls to deprecated Kafka client code and simplify the overall maintainability of the code. This led to many changes in the internal implementation of the component, which we will talk about in this blog post. + +## Motivation + +The ground work related to these changes can be traced back to the following set of issues reported on our Jira. Some of these issues were relatively new, but some have been reported for some time. In no particular order, we have: + +* [camel-kafka: possible corruption of idempotency messages when using KafkaIdempotentRepository](https://issues.apache.org/jira/browse/CAMEL-16914) +* [camel-kafka: usage of deprecated method may cause Camel to block indefinitely](https://issues.apache.org/jira/browse/CAMEL-16928) +* [camel-kafka: investigate alternatives to remove Thread.sleep from the record fetcher](https://issues.apache.org/jira/browse/CAMEL-16949) +* [camel-kafka: commits may be skipped on stop or shutdown](https://issues.apache.org/jira/browse/CAMEL-16973) +* [camel-kafka: improve the documentation about consumer streams vs consumers count](https://issues.apache.org/jira/browse/CAMEL-16981) +* [camel-kafka -consumerCount vs consumerStreams problem with pool size](https://issues.apache.org/jira/browse/CAMEL-13215) +* [camel-kafka: make the resume strategy configurable](https://issues.apache.org/jira/browse/CAMEL-16974) +* [Seek to specific offset and KafkaConsumer access](https://issues.apache.org/jira/browse/CAMEL-13768) + +The main driver for these changes was our intention to move away from blocking consumer calls from the Kafka Consumer API, in order to leverage the modified APIs introduced on [KIP-266](https://cwiki.apache.org/confluence/display/KAFKA/KIP-266%3A+Fix+consumer+indefinite+blocking+behavior). Although the API changes by themselves are simple, the original behavior of the component consumer was not optimized for making the best use of them. + +## Concurrency Changes + +While reviewing and refactoring the code to implement the aforementioned set of changes, we used the opportunity to tweak the mechanism used for concurrently fetch records from Kafra. In previous versions, Camel had two configurations controlling the concurrency of the consumer threads: consumerStreams and consumersCount. The former was used to configure the size of the thread pool used by the consumers, whereas the latter configured the number of concurrent consumer threads used by the component. As part of these changes, the size of the thread pool is now defined by the number of concurrent consumers. + +## Resume Strategy + +This refactoring also allowed us to introduce one new feature that should provide greater flexibility for integrations willing to manage the resume process. It is now possible to define a custom resume strategy through the resumeStrategy configuration. A resume strategy can be recorded in the registry and referenced using the `resumeStrategy` configuration (i.e.: `resumeStrategy=#beanNameOnRegistry`). Review comment: Can you configure the resume strategy on the component level, in case you want the same for all your Camel kafka endpoints? ########## File path: content/blog/2021/09/camel-kafka-consumer-changes/index.md ########## @@ -0,0 +1,68 @@ +--- +title: "Upcoming changes to the consumer on Kafka component" +date: 2021-09-27 +draft: true +authors: [orpiske] +categories: ["Features", "Camel"] +preview: "Upcoming changes to the consumer on Kafka component" +--- + +We recently started to review, investigate and fix reliability issues with the [Kafka](https://camel.apache.org/components/latest/kafka-component.html) component. As part of that, we refactored the code to avoid problematic concurrency patterns, remove calls to deprecated Kafka client code and simplify the overall maintainability of the code. This led to many changes in the internal implementation of the component, which we will talk about in this blog post. + +## Motivation + +The ground work related to these changes can be traced back to the following set of issues reported on our Jira. Some of these issues were relatively new, but some have been reported for some time. In no particular order, we have: + +* [camel-kafka: possible corruption of idempotency messages when using KafkaIdempotentRepository](https://issues.apache.org/jira/browse/CAMEL-16914) +* [camel-kafka: usage of deprecated method may cause Camel to block indefinitely](https://issues.apache.org/jira/browse/CAMEL-16928) +* [camel-kafka: investigate alternatives to remove Thread.sleep from the record fetcher](https://issues.apache.org/jira/browse/CAMEL-16949) +* [camel-kafka: commits may be skipped on stop or shutdown](https://issues.apache.org/jira/browse/CAMEL-16973) +* [camel-kafka: improve the documentation about consumer streams vs consumers count](https://issues.apache.org/jira/browse/CAMEL-16981) +* [camel-kafka -consumerCount vs consumerStreams problem with pool size](https://issues.apache.org/jira/browse/CAMEL-13215) +* [camel-kafka: make the resume strategy configurable](https://issues.apache.org/jira/browse/CAMEL-16974) +* [Seek to specific offset and KafkaConsumer access](https://issues.apache.org/jira/browse/CAMEL-13768) Review comment: would look nicer if this also starts with camel-kafka: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@camel.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
