huaxingao commented on code in PR #10659:
URL: https://github.com/apache/iceberg/pull/10659#discussion_r1680162100
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java:
##########
@@ -175,7 +181,25 @@ public Statistics estimateStatistics() {
protected Statistics estimateStatistics(Snapshot snapshot) {
// its a fresh table, no data
if (snapshot == null) {
- return new Stats(0L, 0L);
+ return new Stats(0L, 0L, Maps.newHashMap());
+ }
+
+ Map<NamedReference, ColumnStatistics> map = Maps.newHashMap();
Review Comment:
Changed to `colStatsMap` and also only allocate if enableColumnStats and CBO
are enabled.
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java:
##########
@@ -175,7 +181,25 @@ public Statistics estimateStatistics() {
protected Statistics estimateStatistics(Snapshot snapshot) {
// its a fresh table, no data
if (snapshot == null) {
- return new Stats(0L, 0L);
+ return new Stats(0L, 0L, Maps.newHashMap());
+ }
+
+ Map<NamedReference, ColumnStatistics> map = Maps.newHashMap();
+
+ if (readConf.enableColumnStats()) {
+ List<StatisticsFile> files = table.statisticsFiles();
+ if (!files.isEmpty()) {
+ List<BlobMetadata> metadataList = (files.get(0)).blobMetadata();
+
+ for (BlobMetadata blobMetadata : metadataList) {
+ long ndv = Long.parseLong(blobMetadata.properties().get("ndv"));
+ ColumnStatistics colStats = new ColStats(ndv, null, null, 0L, 0L,
0L, null);
Review Comment:
In aggregate push down, I have some code to get min/max/nulls from the
manifest file. I will extract the common code and reuse them. Will have a
separate PR for this after this PR is merged. I have put a `// TODO: Fill min,
max and null from the manifest file`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
