bgaborg commented on issue #1123: HADOOP-16380 S3Guard to determine empty directory status for all non-root directories URL: https://github.com/apache/hadoop/pull/1123#issuecomment-513832536 -Dscale against ireland with dynamo: some test failed, so after the rerun: ``` [INFO] Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir [ERROR] Tests run: 9, Failures: 1, Errors: 2, Skipped: 0, Time elapsed: 224.725 s <<< FAILURE! - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir [ERROR] testRecursiveRootListing(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir) Time elapsed: 180.018 s <<< ERROR! org.junit.runners.model.TestTimedOutException: test timed out after 180000 milliseconds [ERROR] testRmEmptyRootDirNonRecursive(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir) Time elapsed: 31.867 s <<< FAILURE! java.lang.AssertionError: After 12 attempts: listing after rm /* not empty final [00] S3AFileStatus{path=s3a://gabota-versioned-bucket-ireland/tests3ascale; isDirectory=true; modification_time=0; access_time=0; owner=gaborbota; group=gaborbota; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=UNKNOWN eTag=null versionId=null deleted [00] S3AFileStatus{path=s3a://gabota-versioned-bucket-ireland/tests3ascale; isDirectory=true; modification_time=0; access_time=0; owner=gaborbota; group=gaborbota; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=UNKNOWN eTag=null versionId=null original [00] S3AFileStatus{path=s3a://gabota-versioned-bucket-ireland/test; isDirectory=true; modification_time=0; access_time=0; owner=gaborbota; group=gaborbota; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=FALSE eTag=null versionId=null [01] S3AFileStatus{path=s3a://gabota-versioned-bucket-ireland/tests3ascale; isDirectory=true; modification_time=0; access_time=0; owner=gaborbota; group=gaborbota; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=UNKNOWN eTag=null versionId=null [ERROR] testListEmptyRootDirectory(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir) Time elapsed: 2.905 s <<< ERROR! java.io.FileNotFoundException: about to be deleted file: not found s3a://gabota-versioned-bucket-ireland/tests3ascale in s3a://gabota-versioned-bucket-ireland/ at org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir.testListEmptyRootDirectory(ITestS3AContractRootDir.java:85) Caused by: java.io.FileNotFoundException: No such file or directory: s3a://gabota-versioned-bucket-ireland/tests3ascale at org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir.testListEmptyRootDirectory(ITestS3AContractRootDir.java:85) [INFO] Running org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortMagicCommitter [ERROR] Tests run: 7, Failures: 1, Errors: 2, Skipped: 0, Time elapsed: 67.798 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortMagicCommitter [ERROR] test_110_teragen(org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortMagicCommitter) Time elapsed: 16.361 s <<< FAILURE! java.lang.AssertionError: Teragen(1000, s3a://gabota-versioned-bucket-ireland/terasort-ITestTerasortMagicCommitter/sortin) failed expected:<0> but was:<1> [ERROR] test_120_terasort(org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortMagicCommitter) Time elapsed: 2.044 s <<< ERROR! java.io.FileNotFoundException: Output directory s3a://gabota-versioned-bucket-ireland/terasort-ITestTerasortMagicCommitter/sortin from previous teragen stage not found: Job may not have executed: not found s3a://gabota-versioned-bucket-ireland/terasort-ITestTerasortMagicCommitter/sortin in s3a://gabota-versioned-bucket-ireland/terasort-ITestTerasortMagicCommitter Caused by: java.io.FileNotFoundException: No such file or directory: s3a://gabota-versioned-bucket-ireland/terasort-ITestTerasortMagicCommitter/sortin [ERROR] test_130_teravalidate(org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortMagicCommitter) Time elapsed: 2.033 s <<< ERROR! java.io.FileNotFoundException: Output directory s3a://gabota-versioned-bucket-ireland/terasort-ITestTerasortMagicCommitter/sortout from previous terasort stage not found: Job may not have executed: not found s3a://gabota-versioned-bucket-ireland/terasort-ITestTerasortMagicCommitter/sortout in s3a://gabota-versioned-bucket-ireland/terasort-ITestTerasortMagicCommitter Caused by: java.io.FileNotFoundException: No such file or directory: s3a://gabota-versioned-bucket-ireland/terasort-ITestTerasortMagicCommitter/sortout [INFO] [INFO] Results: [INFO] [ERROR] Failures: [ERROR] ITestS3AContractRootDir>AbstractContractRootDirectoryTest.testRmEmptyRootDirNonRecursive:97->Assert.fail:88 After 12 attempts: listing after rm /* not empty final [00] S3AFileStatus{path=s3a://gabota-versioned-bucket-ireland/tests3ascale; isDirectory=true; modification_time=0; access_time=0; owner=gaborbota; group=gaborbota; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=UNKNOWN eTag=null versionId=null deleted [00] S3AFileStatus{path=s3a://gabota-versioned-bucket-ireland/tests3ascale; isDirectory=true; modification_time=0; access_time=0; owner=gaborbota; group=gaborbota; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=UNKNOWN eTag=null versionId=null original [00] S3AFileStatus{path=s3a://gabota-versioned-bucket-ireland/test; isDirectory=true; modification_time=0; access_time=0; owner=gaborbota; group=gaborbota; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=FALSE eTag=null versionId=null [01] S3AFileStatus{path=s3a://gabota-versioned-bucket-ireland/tests3ascale; isDirectory=true; modification_time=0; access_time=0; owner=gaborbota; group=gaborbota; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=UNKNOWN eTag=null versionId=null [ERROR] ITestTerasortMagicCommitter>AbstractCommitTerasortIT.test_110_teragen:173->AbstractCommitTerasortIT.executeStage:148->Assert.assertEquals:645->Assert.failNotEquals:834->Assert.fail:88 Teragen(1000, s3a://gabota-versioned-bucket-ireland/terasort-ITestTerasortMagicCommitter/sortin) failed expected:<0> but was:<1> [ERROR] Errors: [ERROR] ITestS3AContractRootDir.testListEmptyRootDirectory:85->AbstractContractRootDirectoryTest.testListEmptyRootDirectory:192 ? FileNotFound [ERROR] ITestS3AContractRootDir>AbstractContractRootDirectoryTest.testRecursiveRootListing:254 ? TestTimedOut [ERROR] ITestTerasortMagicCommitter>AbstractCommitTerasortIT.test_120_terasort:185->AbstractCommitITest.loadSuccessFile:473 ? FileNotFound [ERROR] ITestTerasortMagicCommitter>AbstractCommitTerasortIT.test_130_teravalidate:201->AbstractCommitITest.loadSuccessFile:473 ? FileNotFound [INFO] [ERROR] Tests run: 16, Failures: 2, Errors: 4, Skipped: 0 ``` Ireland -Dlocal: no unknown test failures other than `testInitNegativeRead` and `testMRJob` Summary: +1 pending on fixing org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir#testRmEmptyRootDirNonRecursive, testListEmptyRootDirectory failures I don't think that org.apache.hadoop.fs.s3a.commit.terasort.ITestTerasortMagicCommitter#test_110_teragen,test_120_terasort,test_130_teravalidate would be related to this change.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
