noob-se7en commented on code in PR #15296:
URL: https://github.com/apache/pinot/pull/15296#discussion_r2014111927
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java:
##########
@@ -1723,18 +1729,27 @@ public RealtimeSegmentDataManager(SegmentZKMetadata
segmentZKMetadata, TableConf
}
}
- // Consumption while downloading and replacing the slow replicas is not
allowed for the following tables:
+ // Consumption while downloading and replacing consuming segment is not
allowed for the following tables:
// 1. Partial Upserts
// 2. Dedup Tables
// For the above table types, we would be looking into the metadata
information when inserting a new record,
// so it is not right to allow consumption while downloading and replacing
the consuming segment as we might see
// duplicates in dedup tables and inconsistent entries compared to lead
replicas for partial
// upsert tables. If tables are dedup/partial upsert enabled check for table
and server config properties to see if
// consumption is allowed
- private boolean isConsumptionAllowedDuringCommit() {
- return (!_realtimeTableDataManager.isDedupEnabled() ||
_tableConfig.getDedupConfig()
- .isAllowDedupConsumptionDuringCommit()) &&
(!_realtimeTableDataManager.isPartialUpsertEnabled()
- ||
_tableConfig.getUpsertConfig().isAllowPartialUpsertConsumptionDuringCommit());
+ @VisibleForTesting
+ ParallelSegmentConsumptionPolicy getParallelConsumptionPolicy() {
+ if (_realtimeTableDataManager.isDedupEnabled() &&
_tableConfig.getDedupConfig()
+ .isAllowDedupConsumptionDuringCommit()) {
+ // if dedup is enabled
+ return ParallelSegmentConsumptionPolicy.ALLOW_DURING_BUILD_ONLY;
Review Comment:
> Do you foresee any scenario where user want to block consumption during
build?
No.
We should keep only 1 flag: _allowConsumptionDuringDownload then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
