umamaheswararao commented on a change in pull request #2010:
URL: https://github.com/apache/hadoop/pull/2010#discussion_r425469062
##########
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
##########
@@ -1226,7 +1229,38 @@ public FileStatus getFileStatus(Path f) throws
IOException {
myUri, null));
}
}
- return result;
+ if (fallbackStatuses.length > 0) {
+ return consolidateFileStatuses(fallbackStatuses, result);
+ } else {
+ return result;
+ }
+ }
+
+ private FileStatus[] consolidateFileStatuses(FileStatus[] fallbackStatuses,
+ FileStatus[] mountPointStatuses) {
+ ArrayList<FileStatus> result = new ArrayList<>();
+ Set<String> pathSet = new HashSet<>();
+ int i = 0;
+ for (FileStatus status : mountPointStatuses) {
+ result.add(status);
+ pathSet.add(status.getPath().getName());
+ }
+ for (FileStatus status : fallbackStatuses) {
+ if (!pathSet.contains(status.getPath().getName())) {
+ result.add(status);
+ }
+ }
+ return result.toArray(new FileStatus[0]);
+ }
+
+ private FileStatus[] listStatusForFallbackLink() throws IOException {
+ if (theInternalDir.isRoot() && theInternalDir.getFallbackLink() != null)
{
+ URI fallBackUri =
theInternalDir.getFallbackLink().targetDirLinkList[0];
Review comment:
Why you might have issue was because you are passing fallBackUri.
ChrootedFS already contains /fallBackDir and you are again passing a uri which
contains /fallbackDir. SO it will fail with "FNFE". If you use chrootedfs, you
should just pass relative path by cutting chrooteduri. Since we support only at
root, it will be simply "/" on chorootedfs.
I have one another concern that we are returning absolute path with fall
back in ls. But with regular mountlinks we will return relative to viewfs
scheme ex: [viewfs://default/data, viewfs://default/targetRoot,
viewfs://default/danglingLink, [viewfs://default/user2, viewfs://default/user,
viewfs://default/internalDir, viewfs://default/linkToAFile]
Since fallback also part of viewfs and its actually a child fs, should we
return fallback paths also relatively constructed with respective to viewfs://
scheme?
exmaple: your fallback path is hdfs://nn1/fallBackDir and you have a dir
under it. That is /user1
So, when you do ls, should it return viewfs://default/user1 ?
if you use this ls result path to create some directory like
viewfs://default/user1/test1, it will/should work ( IMO) and it should create
at physical location of hdfs://nn1/fallBackDir/user1/test1
Because you did not have any other mount link with /user1, it will fallback
to create at fallback link.
But with the current behavior, we are returning hdfs://nn1/fallBackDir/user1
and if you use this path to create a dir under it, it will actually go to
regular hdfs instance but not via viewfs. This could be a potentially problem
in that case overloadScheme impl. There hdfs can go to ViewFSOverloadScheme and
try to create path as hdfs://nn1/fallBackDir/user1. Now fallBackDir/user1 will
be attempted to create in viewfsOS env, it checks whether we have any link with
fallBackDir. Of course we will not have any regular link. So, it will create at
fallback link.
Our fallback link is already hdfs://nn1/fallbackDir and now it will attempt
to create /fallbackDir/user1
So, it will result to create hdfs://nn1/fallbackDir/fallbackDir/user1. This
seems not correct.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
