Copilot commented on code in PR #16810:
URL: https://github.com/apache/pinot/pull/16810#discussion_r2346836840
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java:
##########
@@ -850,8 +850,19 @@ public void run() {
break;
case KEEP: {
if (_segmentCompletionMode == CompletionMode.DOWNLOAD) {
- _state = State.DISCARDED;
- break;
+ // Fetch fresh metadata from ZooKeeper to check if download
URL has been set by another replica
+ String downloadUrl =
_realtimeTableDataManager.fetchZKMetadata(_segmentNameStr).getDownloadUrl();
+ if (StringUtils.isNotEmpty(downloadUrl)) {
+ _segmentLogger.info(
+ "CompletionMode is DOWNLOAD and download URL is
available for segment: {}. URL: {}",
+ _segmentNameStr, downloadUrl);
+ _state = State.DISCARDED;
+ break;
+ } else {
+ _segmentLogger.warn(
+ "CompletionMode is DOWNLOAD but download URL is missing
or invalid for segment: {}. "
Review Comment:
[nitpick] The log message mentions 'invalid' download URL but the code only
checks if it's empty/null with StringUtils.isNotEmpty(). Consider clarifying
the log message to only mention 'missing' or add validation for invalid URLs.
```suggestion
"CompletionMode is DOWNLOAD but download URL is
missing for segment: {}. "
```
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java:
##########
@@ -850,8 +850,19 @@ public void run() {
break;
case KEEP: {
if (_segmentCompletionMode == CompletionMode.DOWNLOAD) {
- _state = State.DISCARDED;
- break;
+ // Fetch fresh metadata from ZooKeeper to check if download
URL has been set by another replica
+ String downloadUrl =
_realtimeTableDataManager.fetchZKMetadata(_segmentNameStr).getDownloadUrl();
Review Comment:
Fetching metadata from ZooKeeper on every KEEP operation could introduce
performance overhead. Consider caching the metadata or checking if the segment
metadata has been updated since the last check to avoid unnecessary ZK calls.
##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeSegmentDataManager.java:
##########
@@ -1460,58 +1472,60 @@ public void goOnlineFromConsuming(SegmentZKMetadata
segmentZKMetadata)
case CATCHING_UP:
case HOLDING:
case INITIAL_CONSUMING:
- switch (_segmentCompletionMode) {
- case DOWNLOAD:
- _segmentLogger.info("State {}. CompletionMode {}. Downloading to
replace", _state.toString(),
- _segmentCompletionMode);
+ if (_segmentCompletionMode == CompletionMode.DOWNLOAD) {
+ // Fetch fresh metadata from ZooKeeper to check if download URL
has been set by another replica
+ String downloadUrl = segmentZKMetadata.getDownloadUrl();
+ if (StringUtils.isNotEmpty(downloadUrl)) {
+ _segmentLogger.info("State {}. CompletionMode {}. Downloading to
replace with fresh metadata. URL: {}",
+ _state.toString(), _segmentCompletionMode, downloadUrl);
downloadSegmentAndReplace(segmentZKMetadata);
break;
- case DEFAULT:
- // Allow to catch up upto final offset, and then replace.
- if (_currentOffset.compareTo(endOffset) > 0) {
- // We moved ahead of the offset that is committed in ZK.
- _segmentLogger
- .warn("Current offset {} ahead of the offset in zk {}.
Downloading to replace", _currentOffset,
- endOffset);
- downloadSegmentAndReplace(segmentZKMetadata);
- } else if (_currentOffset.compareTo(endOffset) == 0) {
- _segmentLogger
- .info("Current offset {} matches offset in zk {}.
Replacing segment", _currentOffset, endOffset);
- if (!buildSegmentAndReplace()) {
- _segmentLogger.warn("Failed to build the segment: {} and
replace. Downloading to replace",
- _segmentNameStr);
- downloadSegmentAndReplace(segmentZKMetadata);
- }
- } else {
- boolean success = false;
- // Since online helix transition for a segment can arrive
before segment's consumer acquires the
- // semaphore, check _consumerSemaphoreAcquired before catching
up.
- // This is to avoid consuming in parallel to another segment's
consumer.
- if (_consumerSemaphoreAcquired.get()) {
- _segmentLogger.info("Attempting to catch up from offset {}
to {} ", _currentOffset, endOffset);
- success = catchupToFinalOffset(endOffset,
-
TimeUnit.MILLISECONDS.convert(MAX_TIME_FOR_CONSUMING_TO_ONLINE_IN_SECONDS,
TimeUnit.SECONDS));
- } else {
- _segmentLogger.warn("Consumer semaphore was not acquired,
Skipping catch up from offset {} to {} ",
- _currentOffset, endOffset);
- }
+ } else {
+ _segmentLogger.warn("State {}. CompletionMode is DOWNLOAD but
download URL is missing: {}. "
+ + "Falling back to building segment locally. Download
URL: {}", _state.toString(),
+ _segmentNameStr, downloadUrl);
Review Comment:
[nitpick] The log message format includes a colon after 'missing' but then
uses a period, creating inconsistent punctuation. Also, the placeholder order
doesn't match the arguments - the log suggests the URL comes after the segment
name, but the arguments are state, segment name, then URL.
```suggestion
_segmentLogger.warn("State {}. CompletionMode is DOWNLOAD but
download URL is missing ({}). "
+ "Falling back to building segment locally for
segment: {}", _state.toString(),
downloadUrl, _segmentNameStr);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
