Manjeet Duhan created KAFKA-7749:
------------------------------------
Summary: confluent does not provide option to set consumer
properties at connector level
Key: KAFKA-7749
URL: https://issues.apache.org/jira/browse/KAFKA-7749
Project: Kafka
Issue Type: Improvement
Components: KafkaConnect
Reporter: Manjeet Duhan
_We want to increase consumer.max.poll.record to increase performance but this
value can only be set in worker properties which is applicable to all
connectors given cluster._
__
_Operative Situation :- We have one project which is communicating with
Elasticsearch and we set consumer.max.poll.record=500 after multiple
performance tests which worked fine for an year._
_Then one more project onboarded in the same cluster which required
consumer.max.poll.record=5000 based on their performance tests. This
configuration is moved to production._
_Admetric started failing as it was taking more than 5 minutes to process
5000 polled records and started throwing commitfailed exception which is
vicious cycle as it will process same data over and over again._
__
_We can control above if start consumer using plain java but this control was
not available at each consumer level in confluent connector._
_I have overridden kafka code to accept connector properties which will be
applied to single connector and others will keep on using default properties .
These changes are already running in production for more than 5 months._
_Some of the properties which were useful for us._
max.poll.records
max.poll.interval.ms
request.timeout.ms
key.deserializer
value.deserializer
heartbeat.interval.ms
session.timeout.ms
auto.offset.reset
connections.max.idle.ms
enable.auto.commit
auto.commit.interval.ms
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
