tibrewalpratik17 commented on issue #12667: URL: https://github.com/apache/pinot/issues/12667#issuecomment-2008252101
> for high-qps and high-ingest, we could refresh the copy of bitmaps periodically e.g. at seconds level vs. refreshing the bitmap copy at individual record or individual query, to amortize the cost of copying bitmaps, but with drawback of loss of data freshness at seconds level. I assume this is like what you meant by 'in batches' in your issue description. could you help elaborate the concern on scaling in this way? Freshness of few seconds should still be okay i guess. Yes "in batches" is very similar to this. But we should ensure we are refreshing the copy before `addRecord` workflow so that we don't take a snapshot in intermittent state. Maybe a read-lock might help but not sure if it might lead to huge scalability challenges. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
