npawar commented on issue #3998: Upgrade to use Kafka release 2.1.1 URL: https://github.com/apache/incubator-pinot/issues/3998#issuecomment-494581187 We're planning to work on this issue this quarter. We (@mcvsubbu and I) will do some refactoring, such that the kafka code will be separated out as a stream plug in. This will make it easier for folks to plug in the versions of kafka they wish to use. Here's the tentative steps we plan to take: 1) Many of our tests are using kafka related classes. We plan to introduce a fake stream, which can be used in all tests. This will help us remove dependency of kafka related classes in all tests. 2) There are 2 implementations of PartitionFunction, which are using kafka libraries. We plan to abstract those out, and make the stream implementation provide stream specific partition functions. 3) After completion of above 2 steps, we should be able to pull out kafka related classes into a separate module, say "pinot-kafka-0.9.0.1". This should let us remove kafka dependency from all other modules (i think pinot-common, pinot-core, pinot-controller have kafka as a dependency in their pom). 4) We can then decide whether to keep "pinot-kafka-0.9.0.1" as a module in pinot, or separate it out to another open source repo. This module will serve as an example of exactly what needs to be done to add a plug in) Will add more details as we iron things out. After this is completed, you will be able to plug-in your implementations for other kafka versions. @ananthdurai @kishoreg
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
