tibrewalpratik17 commented on code in PR #14506:
URL: https://github.com/apache/pinot/pull/14506#discussion_r1857995417
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/PinotLLCRealtimeSegmentManager.java:
##########
@@ -1593,6 +1599,16 @@ public void uploadToDeepStoreIfMissing(TableConfig
tableConfig, List<SegmentZKMe
}
}
+ @VisibleForTesting
+ String getSegmentCrcFromServer(String tableNameWithType, String segmentName,
String endpoint) {
Review Comment:
> Is this backward compatible when the server is still running the old pinot
version?
The proposed change is not fully backward compatible during the interim
period when the controller has been rolled out but the server is still running
an older version. This is a common challenge for changes that involve
interactions between the controller and server. Below are the considerations
and potential solutions to address this issue:
1. Soft Validation on Controller: Modify the controller to perform a soft
validation during this transitional period. If the controller cannot connect to
the older server, it proceeds with uploading to deep store as per the existing
flow.
Downside: This approach may result in a mismatch between Zookeeper's CRC and
deep store's CRC, introducing risks of data inconsistency.
2. Config-Driven Rollout: Introduce a configuration flag to enable the
change in controller only after the server rollout is complete. This ensures
that the new flow is activated only when both controller and server are running
the compatible versions.
Downside: Adds more configuration options for features that are meant to be
defaults, leading to potential clutter and management overhead.
3. Pause Deep Store Uploads During Rollout (Proposed):
In the worst-case scenario, it might be acceptable to halt deep store
uploads temporarily during the period between the controller and server
rollouts. Once the rollout is complete, the process resumes normally.
Rationale: This minimizes risk (e.g., CRC mismatches or partial uploads) and
avoids introducing temporary configurations or complex validation mechanisms.
Downside: There is a short period where new segments may not be uploaded to
deep store, but this is generally manageable if the window is small.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
