nastra commented on code in PR #8925:
URL: https://github.com/apache/iceberg/pull/8925#discussion_r1375034993
##########
core/src/test/java/org/apache/iceberg/TestSplitPlanning.java:
##########
@@ -216,6 +217,34 @@ public void testSplitPlanningWithOffsets() {
"We should get one task per row group", 32,
Iterables.size(scan.planTasks()));
}
+ @Test
+ public void testSplitPlanningWithCorruptedOffsets() throws IOException {
+ List<DataFile> files16Mb = newFilesWithInvalidOffset(16, 16 * 1024 * 1024,
2);
+ appendFiles(files16Mb);
+
+ // There will be 4 tasks per file, instead of 2 tasks per file, since the
offsets are invalid
+ // and will be ignored.
+ TableScan scan =
+ table
+ .newScan()
+ .option(TableProperties.SPLIT_OPEN_FILE_COST, String.valueOf(0))
+ .option(TableProperties.SPLIT_SIZE, String.valueOf(4L * 1024 *
1024));
+
+ Assert.assertEquals(
+ "We should get four tasks per file since the offsets will be ignored",
+ 64,
+ Iterables.size(scan.planTasks()));
+ try (CloseableIterable<CombinedScanTask> tasks = scan.planTasks()) {
+ CombinedScanTask task = tasks.iterator().next().asCombinedScanTask();
+ Collection<FileScanTask> files = task.files();
+ for (FileScanTask fileScanTask : files) {
+ DataFile dataFile = fileScanTask.file();
+ Assert.assertNull(
Review Comment:
same as above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
