klsince commented on a change in pull request #7969:
URL: https://github.com/apache/pinot/pull/7969#discussion_r778357692
##########
File path:
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
##########
@@ -333,94 +306,156 @@ public void reloadSegment(String segmentName,
IndexLoadingConfig indexLoadingCon
@Override
public void addOrReplaceSegment(String segmentName, IndexLoadingConfig
indexLoadingConfig,
- SegmentZKMetadata zkMetadata, @Nullable SegmentMetadata localMetadata)
+ SegmentZKMetadata zkMetadata, @Nullable SegmentMetadata segmentMetadata)
throws Exception {
- if (!isNewSegment(zkMetadata, localMetadata)) {
- LOGGER.info("Segment: {} of table: {} has crc: {} same as before,
already loaded, do nothing", segmentName,
- _tableNameWithType, localMetadata.getCrc());
+ // Non-null segment metadata means the segment has already been loaded.
+ if (segmentMetadata != null) {
+ if (hasSameCRC(zkMetadata, segmentMetadata)) {
+ // Simply returns if the CRC hasn't changed. The table config may have
changed
+ // since segment is loaded, but that is handled by reloadSegment()
method.
+ LOGGER.info("Segment: {} of table: {} has crc: {} same as before,
already loaded, do nothing", segmentName,
+ _tableNameWithType, segmentMetadata.getCrc());
+ } else {
+ // Download the raw segment, reprocess and load it if the CRC has
changed.
+ LOGGER.info("Segment: {} of table: {} already loaded but its crc: {}
differs from new crc: {}", segmentName,
+ _tableNameWithType, segmentMetadata.getCrc(), zkMetadata.getCrc());
+ downloadRawSegmentAndProcess(segmentName, indexLoadingConfig,
zkMetadata,
+ ZKMetadataProvider.getTableSchema(_propertyStore,
_tableNameWithType));
+ }
return;
}
- // Try to recover if no local metadata is provided.
- if (localMetadata == null) {
- LOGGER.info("Segment: {} of table: {} is not loaded, checking disk",
segmentName, _tableNameWithType);
- localMetadata = recoverSegmentQuietly(segmentName);
- if (!isNewSegment(zkMetadata, localMetadata)) {
- LOGGER.info("Segment: {} of table {} has crc: {} same as before,
loading", segmentName, _tableNameWithType,
- localMetadata.getCrc());
- if (loadSegmentQuietly(segmentName, indexLoadingConfig)) {
- return;
- }
- // Set local metadata to null to indicate that the local segment fails
to load,
- // although it exists and has same crc with the remote one.
- localMetadata = null;
- }
- }
+ // For local tier backend, try to recover the segment from potential
+ // reload failure. Continue upon any failure.
+ File indexDir = getSegmentDataDir(segmentName);
+ recoverReloadFailureQuietly(_tableNameWithType, segmentName, indexDir);
- Preconditions.checkState(allowDownload(segmentName, zkMetadata), "Segment:
%s of table: %s does not allow download",
- segmentName, _tableNameWithType);
+ // Creates the SegmentDirectory object to access the segment metadata that
Review comment:
If I understand the question correctly:
1) one may implement SegmentDirectory interface to fetch metadata files in
remote tier backend to initialize the SegmentMetadata object
2) SegmentMetadata object is used to check if CRC changes or if the segment
is still consistent with the latest table config and schema
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
