steveloughran commented on a change in pull request #1134: HADOOP-16433.
S3Guard: Filter expired entries and tombstones when lis…
URL: https://github.com/apache/hadoop/pull/1134#discussion_r306284285
##########
File path:
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3GuardTtl.java
##########
@@ -286,4 +290,70 @@ public void testCreateParentHasTombstone() throws
Exception {
}
}
+ /**
+ * Test that listing of metadatas is filtered from expired items.
+ */
+ @Test
+ public void testListingFilteredExpiredItems() throws Exception {
+ LOG.info("Authoritative mode: {}", authoritative);
+ final S3AFileSystem fs = getFileSystem();
+
+ long oldTime = 100L;
+ long newTime = 110L;
+ long ttl = 9L;
+ final String basedir = getMethodName();
+ final Path tombstonedPath = path(basedir + "/tombstonedPath");
+ final Path baseDirPath = path(basedir);
+ final List<Path> filesToCreate = new ArrayList<>();
+ final MetadataStore ms = fs.getMetadataStore();
+
+ for (int i = 0; i < 10; i++) {
+ filesToCreate.add(path(basedir + "/file" + i));
+ }
+
+ ITtlTimeProvider mockTimeProvider = mock(ITtlTimeProvider.class);
+ ITtlTimeProvider originalTimeProvider = fs.getTtlTimeProvider();
+
+ try {
+ fs.setTtlTimeProvider(mockTimeProvider);
+ when(mockTimeProvider.getMetadataTtl()).thenReturn(ttl);
+
+ // add and delete entry with the oldtime
+ when(mockTimeProvider.getNow()).thenReturn(oldTime);
+ touch(fs, tombstonedPath);
+ fs.delete(tombstonedPath, false);
+
+ // create items with newTime
+ when(mockTimeProvider.getNow()).thenReturn(newTime);
+ for (Path path : filesToCreate) {
+ touch(fs, path);
+ }
+
+ // listing will contain the tombstone with oldtime
+ when(mockTimeProvider.getNow()).thenReturn(oldTime);
+ final DirListingMetadata fullDLM = ms.listChildren(baseDirPath);
+ List<Path> containedPaths = fullDLM.getListing().stream()
+ .map(pm -> pm.getFileStatus().getPath())
+ .collect(Collectors.toList());
+ Assertions.assertThat(containedPaths)
Review comment:
welcome to the world of AssertJ! I still don't fully understand how best to
use it -but it does handle collections nicely.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
