amogh-jahagirdar commented on code in PR #18740:
URL: https://github.com/apache/iceberg/pull/18740#discussion_r4437696677
##########
core/src/test/java/org/apache/iceberg/TestFilePlanner.java:
##########
@@ -273,6 +274,58 @@ void filterHonorsCaseSensitivity(FileFormat format) throws
IOException {
.hasMessageContaining("Cannot find field 'ID'");
}
+ @ParameterizedTest
+ @FieldSource("MANIFEST_FORMATS")
+ void columnStatsOmittedByDefault(FileFormat format) throws IOException {
+ ManifestFile root =
+ writeManifest(
+ format, EMPTY_PARTITION,
ImmutableList.of(dataFileWithStats(FILE_A, idStats(0, 99))));
+
+ DataFile file = Iterables.getOnlyElement(plan(root,
UNPARTITIONED_SPECS)).file();
+
+ assertThat(file.lowerBounds())
+ .as("no column stats are returned unless projectStats requests them")
+ .isNull();
+ assertThat(file.valueCounts()).as("value counts are also omitted by
default").isNull();
+ }
+
+ @ParameterizedTest
+ @FieldSource("MANIFEST_FORMATS")
+ void projectStatsForwardedToRootAndLeafReaders(FileFormat format) throws
IOException {
+ String rootFile = "s3://bucket/db/table/root.parquet";
+ String leafFile = "s3://bucket/db/table/leaf.parquet";
+ ManifestFile leaf =
+ writeManifest(
+ format,
+ EMPTY_PARTITION,
+ ImmutableList.of(dataFileWithStats(leafFile, idStats(100, 199))));
+ ManifestFile root =
+ writeManifest(
+ format,
+ EMPTY_PARTITION,
+ ImmutableList.of(
+ dataFileWithStats(rootFile, idStats(0, 99)),
dataManifest(leaf.path())));
+
+ int idFieldId = TABLE_SCHEMA.findField("id").fieldId();
+ List<FileScanTask> tasks =
+ plan(root, UNPARTITIONED_SPECS, planner ->
planner.projectStats(idFieldId));
+
+ assertThat(tasks)
Review Comment:
On the assertions: this could be a single extracting/tuple assertion like
the one in `residualForMixedPartitionedAndUnpartitioned`, which keeps both
bounds and lets us drop the local paths:
```
assertThat(tasks)
.extracting(
task -> task.file().location(),
task -> lowerBound(task.file(), idFieldId),
task -> upperBound(task.file(), idFieldId))
.containsExactlyInAnyOrder(tuple(FILE_A, 0, 99), tuple(FILE_C, 100,
199));
```
Separately, the fixture only has `id` stats, so this test would also pass if
the planner turned on stats for every field instead of the requested ones.
Adding `data` stats to the metadata and asserting they're absent after
projection of only id would close that gap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
