klsince commented on a change in pull request #7969:
URL: https://github.com/apache/pinot/pull/7969#discussion_r778505597
##########
File path:
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
##########
@@ -275,53 +278,23 @@ public void addSegmentError(String segmentName,
SegmentErrorInfo segmentErrorInf
@Override
public void reloadSegment(String segmentName, IndexLoadingConfig
indexLoadingConfig, SegmentZKMetadata zkMetadata,
- SegmentMetadata localMetadata, @Nullable Schema schema, boolean
forceDownload)
+ SegmentMetadata segmentMetadata, @Nullable Schema schema, boolean
forceDownload)
throws Exception {
- File indexDir = localMetadata.getIndexDir();
- Preconditions.checkState(indexDir.isDirectory(), "Index directory: %s is
not a directory", indexDir);
-
- File parentFile = indexDir.getParentFile();
- File segmentBackupDir =
- new File(parentFile, indexDir.getName() +
CommonConstants.Segment.SEGMENT_BACKUP_DIR_SUFFIX);
-
+ File indexDir = getSegmentDataDir(segmentName);
+ // Create backup dir to make segment reloading atomic for local tier
backend.
+ LoaderUtils.createBackup(indexDir);
try {
- // First rename index directory to segment backup directory so that
original segment have all file descriptors
- // point to the segment backup directory to ensure original segment
serves queries properly
-
- // Rename index directory to segment backup directory (atomic)
- Preconditions.checkState(indexDir.renameTo(segmentBackupDir),
- "Failed to rename index directory: %s to segment backup directory:
%s", indexDir, segmentBackupDir);
-
- // Download from remote or copy from local backup directory into index
directory,
- // and then continue to load the segment from index directory.
- boolean shouldDownload = forceDownload || !hasSameCRC(zkMetadata,
localMetadata);
- if (shouldDownload && allowDownload(segmentName, zkMetadata)) {
- if (forceDownload) {
- LOGGER.info("Segment: {} of table: {} is forced to download",
segmentName, _tableNameWithType);
- } else {
- LOGGER.info("Download segment:{} of table: {} as local crc: {}
mismatches remote crc: {}", segmentName,
- _tableNameWithType, localMetadata.getCrc(), zkMetadata.getCrc());
- }
- indexDir = downloadSegment(segmentName, zkMetadata);
+ boolean shouldDownloadRawSegment = forceDownload ||
!hasSameCRC(zkMetadata, segmentMetadata);
+ if (shouldDownloadRawSegment && allowDownloadRawSegment(segmentName,
zkMetadata)) {
+ downloadRawSegmentAndProcess(segmentName, indexLoadingConfig,
zkMetadata, schema);
Review comment:
If I understand the questions correctly, lemme use an example to add
some clarity:
The segments can move between tiers depends selection criteria like segment
age (as said in the docs/wiki ^). The most recent segments may be kept in a
tier where the servers in that tier use local disk; older segments may move to
a tier where the servers use remote store to keep them. But in those tiers, the
segments are preprocessed and ready for queries, just that some tier has low
latency and some higher depending on servers' storage backend in those tiers.
As in a [reply
below](https://github.com/apache/pinot/pull/7969#discussion_r778489953), the
segments in server's backend (or tier backend) are preprocessed with latest
table config and schema and ready for queries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
