tibrewalpratik17 commented on issue #12667: URL: https://github.com/apache/pinot/issues/12667#issuecomment-2008179502
> To solve this, we're thinking about to provide a consistent table view for queries for upsert tables. In general, we can take copy of bitmaps to form the consistent view for queries to use, while keeping ingesting data. Similar with what's proposed above. We can take the copy on read path or write path: query threads to lock the consuming thread (and helix threads that are adding/replacing segments) and take copy of validDocId bitmaps, then release the lock and process the segments with bitmap copies. Pros: fresh view; Cons: costly and blocks ingestion as the query rate becomes higher. consuming thread (and also the helix threads) take copy of validDocId bitmaps, then atomically replace the old set of bitmaps with the new set. Queries use the bitmap copy to process segments. We can take copy periodically to amortize cost but sacrifice some data freshness, or take copy at handling individual record if we can do the copy very efficiently (only swap the updated 1 or 2 bitmaps into the consistent view atomically). Those options are not exclusive. We may add all of them and use per need (query heavy or ingest heavy or strict freshness etc.) @klsince do you think we have any other solutions? As called out above, both these solutions might not scale at high-qps / high-ingestion-throughput :') -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
