klsince commented on a change in pull request #7969:
URL: https://github.com/apache/pinot/pull/7969#discussion_r778346750
##########
File path:
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
##########
@@ -275,53 +278,23 @@ public void addSegmentError(String segmentName,
SegmentErrorInfo segmentErrorInf
@Override
public void reloadSegment(String segmentName, IndexLoadingConfig
indexLoadingConfig, SegmentZKMetadata zkMetadata,
- SegmentMetadata localMetadata, @Nullable Schema schema, boolean
forceDownload)
+ SegmentMetadata segmentMetadata, @Nullable Schema schema, boolean
forceDownload)
throws Exception {
- File indexDir = localMetadata.getIndexDir();
- Preconditions.checkState(indexDir.isDirectory(), "Index directory: %s is
not a directory", indexDir);
-
- File parentFile = indexDir.getParentFile();
- File segmentBackupDir =
- new File(parentFile, indexDir.getName() +
CommonConstants.Segment.SEGMENT_BACKUP_DIR_SUFFIX);
-
+ File indexDir = getSegmentDataDir(segmentName);
+ // Create backup dir to make segment reloading atomic for local tier
backend.
+ LoaderUtils.createBackup(indexDir);
try {
- // First rename index directory to segment backup directory so that
original segment have all file descriptors
- // point to the segment backup directory to ensure original segment
serves queries properly
-
- // Rename index directory to segment backup directory (atomic)
- Preconditions.checkState(indexDir.renameTo(segmentBackupDir),
- "Failed to rename index directory: %s to segment backup directory:
%s", indexDir, segmentBackupDir);
-
- // Download from remote or copy from local backup directory into index
directory,
- // and then continue to load the segment from index directory.
- boolean shouldDownload = forceDownload || !hasSameCRC(zkMetadata,
localMetadata);
- if (shouldDownload && allowDownload(segmentName, zkMetadata)) {
- if (forceDownload) {
- LOGGER.info("Segment: {} of table: {} is forced to download",
segmentName, _tableNameWithType);
- } else {
- LOGGER.info("Download segment:{} of table: {} as local crc: {}
mismatches remote crc: {}", segmentName,
- _tableNameWithType, localMetadata.getCrc(), zkMetadata.getCrc());
- }
- indexDir = downloadSegment(segmentName, zkMetadata);
+ boolean shouldDownloadRawSegment = forceDownload ||
!hasSameCRC(zkMetadata, segmentMetadata);
+ if (shouldDownloadRawSegment && allowDownloadRawSegment(segmentName,
zkMetadata)) {
+ downloadRawSegmentAndProcess(segmentName, indexLoadingConfig,
zkMetadata, schema);
} else {
- LOGGER.info("Reload the local copy of segment: {} of table: {}",
segmentName, _tableNameWithType);
- FileUtils.copyDirectory(segmentBackupDir, indexDir);
+ downloadTierSegmentAndProcess(segmentName, indexLoadingConfig, schema);
}
-
- // Load from index directory and replace the old segment in memory.
- addSegment(ImmutableSegmentLoader.load(indexDir, indexLoadingConfig,
schema));
-
- // Rename segment backup directory to segment temporary directory
(atomic)
- // The reason to first rename then delete is that, renaming is an atomic
operation, but deleting is not. When we
- // rename the segment backup directory to segment temporary directory,
we know the reload already succeeded, so
- // that we can safely delete the segment temporary directory
- File segmentTempDir = new File(parentFile, indexDir.getName() +
CommonConstants.Segment.SEGMENT_TEMP_DIR_SUFFIX);
- Preconditions.checkState(segmentBackupDir.renameTo(segmentTempDir),
- "Failed to rename segment backup directory: %s to segment temporary
directory: %s", segmentBackupDir,
- segmentTempDir);
- FileUtils.deleteDirectory(segmentTempDir);
+ // Remove backup dir to mark the completion of reloading for local tier
backend.
+ LoaderUtils.removeBackup(indexDir);
Review comment:
hmm.. do you mean to do renameTo outside the createBackup() and
removeBackup() util methods?
I think `rename, then copy or delete` are pretty tightly coupled to finish
one operation atomically, and I feel separating them would increase complexity
for future maintenance.
btw, I extracted the two new util methods createBackup() and removeBackup()
into LoaderUtils after seeing `reloadFailureRecovery` method already there. I
kinda prefer to having those methods in one place, as they are all for the
failure handling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
