chenboat commented on code in PR #12317:
URL: https://github.com/apache/pinot/pull/12317#discussion_r1480634051
##########
pinot-common/src/main/java/org/apache/pinot/common/utils/fetcher/BaseSegmentFetcher.java:
##########
@@ -109,6 +111,46 @@ public File fetchUntarSegmentToLocalStreamed(URI uri, File
dest, long rateLimit,
throw new UnsupportedOperationException();
}
+ /**
+ * @param segmentName
+ * @param uriSupplier the supplier to the list of segment download uris.
+ * @param dest The destination to put the downloaded segment.
+ * @return true if and only if the segment fetch is successful. This method
keeps retrying (with exponential backoff)
+ * of the following steps until segment download is successful or the retry
limit is reached whichever comes first 1)
+ * Find servers hosting the segment in ONLINE state from the External View
of the table. 2) Shuffle the list of
+ * servers. 3) Go through the list of server http download URIs to fetch the
segment until success.
+ * @throws Exception
+ */
+ @Override
+ public boolean fetchSegmentToLocal(String segmentName, Supplier<List<URI>>
uriSupplier, File dest)
+ throws Exception {
+ try {
+ int attempt =
+ RetryPolicies.exponentialBackoffRetryPolicy(_retryCount,
_retryWaitMs, _retryDelayScaleFactor).attempt(() -> {
+ // First find servers hosting the segment in ONLINE state.
+ List<URI> peerSegmentURIs = uriSupplier.get();
+ // Shuffle the list of URIs.
+ Collections.shuffle(peerSegmentURIs);
+ // Next go through the list of URIs to fetch the segment until
success.
+ for (URI uri : peerSegmentURIs) {
+ try {
+ fetchSegmentToLocalWithoutRetry(uri, dest);
+ return true;
+ } catch (Exception e) {
+ _logger.warn("Download segment {} from peer {} failed.",
segmentName, uri, e);
+ }
+ }
+ // None of the URI works. Return false for retry.
+ return false;
+ });
+ _logger.info("Download segment {} successfully with {} attempts.",
segmentName, attempt + 1);
+ return true;
+ } catch (Exception e) {
Review Comment:
Good point. I looked through the code bases of the caller site
RealtimeTableDataManager and the interface of SegmentFetcher. My PR right now
tried to mix bool status return and exception. I think to be consistent with
the rest of the code bases and also for consistent error handling, Exception
throw is the most clean option. So I remove the boolean return status.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
