[
https://issues.apache.org/jira/browse/HBASE-29572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
huginn updated HBASE-29572:
---------------------------
Description:
If an export snapshot fails in copying .snapshotInfo and manifest when we use
export snapshot tool to export snapshot to other cluster, which may leave
behind an empty snapshot directory under .snapshot (snapshot.export.skip.tmp =
true) or .snapshot/.tmp (snapshot.export.skip.tmp = false) without
.snapshotInfo file and manifest.
SnapshotFileCache periodically refreshes the cache by traversing all snapshot
directories under .snapshot and reading the .snapshotInfo file and manifest to
update the cache. If it can not read .snapshotInfo from a snapshot directory,
it will throw CorruptedSnapshotException. *getSnapshotsInProgress* may also
result in such a situation when it attempts to retrieve the .snapshotInfo under
each snapshot directory under .snapshot/.tmp.
So we need to clean up the output snapshot dir before throwing exception and
aborting this job if copying manifest fails.
{code:java}
cache = new SnapshotFileCache(fs, rootDir, workingFs, workingDir,
cacheRefreshPeriod,
cacheRefreshPeriod, "snapshot-hfile-cleaner-cache-refresher",
new SnapshotFileCache.SnapshotFileInspector() {
@Override
public Collection<String> filesUnderSnapshot(final FileSystem fs,
final Path snapshotDir)
throws IOException {
return SnapshotReferenceUtil.getHFileNames(conf, fs, snapshotDir);
}
});
public static Set<String> getHFileNames(final Configuration conf, final
FileSystem fs,
final Path snapshotDir) throws IOException {
SnapshotDescription desc = SnapshotDescriptionUtils.readSnapshotInfo(fs,
snapshotDir);
return getHFileNames(conf, fs, snapshotDir, desc);
}
public static SnapshotDescription readSnapshotInfo(FileSystem fs, Path
snapshotDir)
throws CorruptedSnapshotException {
Path snapshotInfo = new Path(snapshotDir, SNAPSHOTINFO_FILE);
try (FSDataInputStream in = fs.open(snapshotInfo)) {
return SnapshotDescription.parseFrom(in);
} catch (IOException e) {
throw new CorruptedSnapshotException("Couldn't read snapshot info from:"
+ snapshotInfo, e);
}
}
{code}
was:
If an export snapshot fails in copying .snapshotInfo and manifest, which may
leave behind an empty snapshot directory under .snapshot
(snapshot.export.skip.tmp = true) or .snapshot/.tmp (snapshot.export.skip.tmp =
false) without .snapshotInfo file and manifest.
SnapshotFileCache periodically refreshes the cache by traversing all snapshot
directories under .snapshot and reading the .snapshotInfo file and manifest to
update the cache. If it can not read .snapshotInfo from a snapshot directory,
it will throw CorruptedSnapshotException. *getSnapshotsInProgress* may also
result in such a situation when it attempts to retrieve the .snapshotInfo under
each snapshot directory under .snapshot/.tmp.
So we need to clean up the output snapshot dir before throwing exception and
aborting this job if copying manifest fails.
{code:java}
cache = new SnapshotFileCache(fs, rootDir, workingFs, workingDir,
cacheRefreshPeriod,
cacheRefreshPeriod, "snapshot-hfile-cleaner-cache-refresher",
new SnapshotFileCache.SnapshotFileInspector() {
@Override
public Collection<String> filesUnderSnapshot(final FileSystem fs,
final Path snapshotDir)
throws IOException {
return SnapshotReferenceUtil.getHFileNames(conf, fs, snapshotDir);
}
});
public static Set<String> getHFileNames(final Configuration conf, final
FileSystem fs,
final Path snapshotDir) throws IOException {
SnapshotDescription desc = SnapshotDescriptionUtils.readSnapshotInfo(fs,
snapshotDir);
return getHFileNames(conf, fs, snapshotDir, desc);
}
public static SnapshotDescription readSnapshotInfo(FileSystem fs, Path
snapshotDir)
throws CorruptedSnapshotException {
Path snapshotInfo = new Path(snapshotDir, SNAPSHOTINFO_FILE);
try (FSDataInputStream in = fs.open(snapshotInfo)) {
return SnapshotDescription.parseFrom(in);
} catch (IOException e) {
throw new CorruptedSnapshotException("Couldn't read snapshot info from:"
+ snapshotInfo, e);
}
}
{code}
> Clean up output directory when copying the manifest fails during snapshot
> export
> --------------------------------------------------------------------------------
>
> Key: HBASE-29572
> URL: https://issues.apache.org/jira/browse/HBASE-29572
> Project: HBase
> Issue Type: Bug
> Components: snapshots
> Affects Versions: 2.4.11
> Reporter: huginn
> Assignee: huginn
> Priority: Minor
> Labels: pull-request-available
>
> If an export snapshot fails in copying .snapshotInfo and manifest when we use
> export snapshot tool to export snapshot to other cluster, which may leave
> behind an empty snapshot directory under .snapshot (snapshot.export.skip.tmp
> = true) or .snapshot/.tmp (snapshot.export.skip.tmp = false) without
> .snapshotInfo file and manifest.
> SnapshotFileCache periodically refreshes the cache by traversing all snapshot
> directories under .snapshot and reading the .snapshotInfo file and manifest
> to update the cache. If it can not read .snapshotInfo from a snapshot
> directory, it will throw CorruptedSnapshotException.
> *getSnapshotsInProgress* may also result in such a situation when it attempts
> to retrieve the .snapshotInfo under each snapshot directory under
> .snapshot/.tmp.
> So we need to clean up the output snapshot dir before throwing exception and
> aborting this job if copying manifest fails.
> {code:java}
> cache = new SnapshotFileCache(fs, rootDir, workingFs, workingDir,
> cacheRefreshPeriod,
> cacheRefreshPeriod, "snapshot-hfile-cleaner-cache-refresher",
> new SnapshotFileCache.SnapshotFileInspector() {
> @Override
> public Collection<String> filesUnderSnapshot(final FileSystem fs,
> final Path snapshotDir)
> throws IOException {
> return SnapshotReferenceUtil.getHFileNames(conf, fs, snapshotDir);
> }
> });
> public static Set<String> getHFileNames(final Configuration conf, final
> FileSystem fs,
> final Path snapshotDir) throws IOException {
> SnapshotDescription desc = SnapshotDescriptionUtils.readSnapshotInfo(fs,
> snapshotDir);
> return getHFileNames(conf, fs, snapshotDir, desc);
> }
> public static SnapshotDescription readSnapshotInfo(FileSystem fs, Path
> snapshotDir)
> throws CorruptedSnapshotException {
> Path snapshotInfo = new Path(snapshotDir, SNAPSHOTINFO_FILE);
> try (FSDataInputStream in = fs.open(snapshotInfo)) {
> return SnapshotDescription.parseFrom(in);
> } catch (IOException e) {
> throw new CorruptedSnapshotException("Couldn't read snapshot info
> from:" + snapshotInfo, e);
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
