ankitsultana commented on issue #17331: URL: https://github.com/apache/pinot/issues/17331#issuecomment-3658934381
@udaysagar2177 : have you folks evaluated Warpstream? With the proposed approach I think we'll end up implementing part of what Warpstream's consumer does in Pinot. (hasty generalization but I suppose you get the idea) Also, I haven't taken a deeper look yet, but I suppose there must be a way to do this already with the abstractions we already have? For instance, the offset is just a number which could be a key in some database that each vendor can control completely. Similarly, Kafka and stream consumers already do micro batching so instead of fetching a batch of messages from kafka one can simply fetch it from object store. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
