Jackie-Jiang commented on issue #10712: URL: https://github.com/apache/pinot/issues/10712#issuecomment-1734199067
@mcvsubbu The main issue is the ZNode size limit (1M by default) which can be hit by the table's ideal state, external view or current state when a physical table contains too many segments (~10K). One workaround is to increase the jute buffer size, but that can potentially cause memory issue on ZK, and is not a thorough fix of the scale problem. By introducing logical table, we can split the segments into multiple physical tables and query them all together. This way we can horizontally scale the segments. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
