klsince commented on code in PR #12976:
URL: https://github.com/apache/pinot/pull/12976#discussion_r1607346400
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java:
##########
@@ -1058,11 +1149,31 @@ protected void removeDocId(IndexSegment segment, int
docId) {
* Use the segmentContexts to collect the contexts for selected segments.
Reuse the segmentContext object if
* present, to avoid overwriting the contexts specified at the others places.
*/
- public void setSegmentContexts(List<SegmentContext> segmentContexts) {
+ public void setSegmentContexts(List<SegmentContext> segmentContexts,
Map<String, String> queryOptions) {
+ if (_consistencyMode == UpsertConfig.ConsistencyMode.NONE) {
+ setSegmentContexts(segmentContexts);
+ return;
+ }
+ if (_consistencyMode == UpsertConfig.ConsistencyMode.SYNC) {
+ _upsertViewLock.readLock().lock();
+ try {
+ setSegmentContexts(segmentContexts);
+ return;
+ } finally {
+ _upsertViewLock.readLock().unlock();
+ }
+ }
+ // If batch refresh is enabled, the copy of bitmaps is kept updated and
ready to use for a consistent view.
+ // The locking between query threads and upsert threads can be avoided
when using batch refresh.
+ // Besides, queries can share the copy of bitmaps, w/o cloning the bitmaps
by every single query.
+ // If query has specified a need for certain freshness, check the view and
refresh it as needed.
+ // When refreshing the copy of map, we need to take the WLock so only one
thread is refreshing view.
+ long upsertViewFreshnessMs =
QueryOptionsUtils.getUpsertViewFreshnessMs(queryOptions);
+ doBatchRefreshUpsertView(upsertViewFreshnessMs);
for (SegmentContext segmentContext : segmentContexts) {
IndexSegment segment = segmentContext.getIndexSegment();
- if (_trackedSegments.contains(segment)) {
-
segmentContext.setQueryableDocIdsSnapshot(getQueryableDocIdsSnapshotFromSegment(segment));
+ if (_segmentQueryableDocIdsMap.containsKey(segment)) {
Review Comment:
good point
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
