vmarchaud opened a new issue #6556: URL: https://github.com/apache/incubator-pinot/issues/6556
GCP's pubsub implemention doesn't have any kind of partition/offset system so current LLC implementation obviously can't work with it. If people are interested [here is an implementation](https://github.com/reelevant-tech/pinot-pubsub-plugin/tree/v2.0.0/src/main/java/com/reelevant/pinot/plugins/stream/pubsub) that essentially "fake" offsets and always pull latest messages. However if for some reason the server is stopped while a segment is consuming, the plugin will not re-ingest events that were lost We also tried to implement it as a HLC ([implementation is here](https://github.com/reelevant-tech/pinot-pubsub-plugin/tree/v1.0.0/src/main/java/com/reelevant/pinot/plugins/stream/pubsub)) but its pretty much unusable since segments created in HL mode aren't saved to deep storage so you must rely on replicas created through pinot (not handled in the implementation we've made but could be by creating multiple subs). Since both approach are obviously not doable in production, we have decided to move on to a different pubsub system (most likely apache pulsar) so i don't actually request it to be implemented, i'm pretty much opening it so people looking for this will not waste weeks of research :) ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
