mcvsubbu commented on issue #10712: URL: https://github.com/apache/pinot/issues/10712#issuecomment-1735924785
@Jackie-Jiang and I had a discussion yesterday. It seems like he has in mind to tackle two problems with this. - The znode size limitation: The idea here is that we can have multiple tables with the data partitioned across the tables in some way (say, by time). A query to one is a query to all. - Single-topic limitation : Pinot currently limits consumption to one topic only. There have been requests to enable consumption of multiple topics. A logical table abstraction can then funnel queries to all the tables, each of which consumes a different topic. In both of these use cases, it seems to me that the logical table abstraction should also be extended to data push. On the offline side, this will mean some functionality in Pinot that will take the data pushed to the logical table, and distribute it correctly across the physical tables. On the realtime side, it is more murky, maybe physical tables need to be created to consume from single topic, and a logical table created to declare that these be combined. What _I_ was looking for, on the other hand, was just to be able to substitute the table name in the query with a physical table name (i.e. only for query purposes). The use cases I have in mind have _duplcate_ data across the tables, so it does not make sense to route queries to both tables. Some mechanism needs to be provided in pinot to _choose_ between the tables when a query comes in. From a data push standpoint, each table may push its own data (with the data being a bit different between them). TBD whether the three requirements should be (or can be) combined into one feature (while still keeping it simple) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
