navina opened a new issue, #10996:
URL: https://github.com/apache/pinot/issues/10996
Issue is to open-up a discussion around:
1. Why and in what use-cases, does it make sense to use the stream level
consumption model in Pinot ?
2. What are the semantics offered by the stream level consumption model. Eg.
how does data from the source get partitioned into Pinot tables ? How is the
consumption monitored in this model? Iiuc, segment name convention is also
different?
3. Some feature differences I have noticed are (please correct, if I am
mistaken). I am sure there are more.
Feature | HLC | LLC
---|---|---
Force commit | No | Yes
Stream Message metadata extraction | No (can potentially be extended) | Yes
Ingestion throttling | No | Yes
4. Documentation is sparse about this usage and its guarantees. Iirc, there
were a few examples in OSS documentation which used high level consumer. Users
have mistakenly used these samples with `ConsumerType.HIGHLEVEL` and ended up
in long debugging sessions. One example is
https://apache-pinot.slack.com/archives/CDRCA57FC/p1687987849496959?thread_ts=1687912445.703689&cid=CDRCA57FC.
(when the original incident happened, we spent ~1-2 days debugging before
realizing that the stream type is high level)
I would like to propose that we find a path to sunset the stream level
consumption model. but I don't want to proceed without understanding the above
questions. Please help clarify.
I also see comments like "This can be removed once we remove HLC
implementation from the code"
[link](https://github.com/apache/pinot/blob/master/pinot-spi/src/main/java/org/apache/pinot/spi/stream/PartitionLevelStreamConfig.java#L30)
. So, I am assuming this topic has come up before for discussion :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
