Jackie-Jiang commented on a change in pull request #7743:
URL: https://github.com/apache/pinot/pull/7743#discussion_r747876433
##########
File path:
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java
##########
@@ -872,8 +872,14 @@ public void ensureAllPartitionsConsuming(TableConfig
tableConfig, PartitionLevel
if (idealState.isEnabled()) {
List<PartitionGroupConsumptionStatus>
currentPartitionGroupConsumptionStatusList =
getPartitionGroupConsumptionStatusList(idealState, streamConfig);
+
+ // Read the smallest offset when a new partition is detected
+ OffsetCriteria originalOffsetCriteria =
streamConfig.getOffsetCriteria();
+
streamConfig.setOffsetCriteria(OffsetCriteria.SMALLEST_OFFSET_CRITERIA);
Review comment:
Good question. When creating the `StreamConfig`, the map from the table
config is copied, and maintained separately, so there is no race condition on
that.
I actually started with the suggested approach by adding OffsetCriteria
override to the method, then found that I need to carry the extra parameter all
the way to the `StreamMetadataProvider` interface, and could pollute the
interface and potentially cause unexpected behavior for the implementations of
the interface. Then I decided to go with the changing stream config approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
