huaxingao commented on code in PR #6622:
URL: https://github.com/apache/iceberg/pull/6622#discussion_r1115161587
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java:
##########
@@ -158,6 +178,134 @@ public Filter[] pushedFilters() {
return pushedFilters;
}
+ @Override
+ public boolean pushAggregation(Aggregation aggregation) {
+ if (!canPushDownAggregation(aggregation)) {
+ return false;
+ }
+
+ AggregateEvaluator aggregateEvaluator;
+ try {
+ List<Expression> aggregates =
+ Arrays.stream(aggregation.aggregateExpressions())
+ .map(agg -> SparkAggregates.convert(agg))
+ .collect(Collectors.toList());
+ aggregateEvaluator = AggregateEvaluator.create(schema, aggregates);
+ } catch (UnsupportedOperationException | IllegalArgumentException e) {
+ LOG.info("Skipped aggregate pushdown: " + e);
+ return false;
+ }
+
+ if
(!metricsModeSupportsAggregatePushDown(aggregateEvaluator.aggregates())) {
+ return false;
+ }
+
+ TableScan scan = table.newScan();
+ ((DataTableScan) scan).setStats(true);
+ Snapshot snapshot = readSnapshot();
+ if (snapshot == null) {
+ LOG.info("Skipped aggregate pushdown: table snapshot is null");
+ return false;
+ }
+ scan = scan.useSnapshot(snapshot.snapshotId());
+ scan = configureSplitPlanning(scan);
+
+ try (CloseableIterable<FileScanTask> fileScanTasks = scan.planFiles()) {
Review Comment:
Added this. Thanks
##########
api/src/main/java/org/apache/iceberg/TableScan.java:
##########
@@ -101,4 +101,11 @@ default TableScan appendsAfter(long fromSnapshotId) {
* @return the Snapshot this scan will use
*/
Snapshot snapshot();
+
+ /**
+ * Create a new {@link TableScan} from this scan's configuration that will
have column stats
+ *
+ * @return a new scan based on this with column stats
+ */
+ TableScan withColStats();
Review Comment:
Done
##########
core/src/main/java/org/apache/iceberg/TableScanContext.java:
##########
@@ -36,7 +36,7 @@ final class TableScanContext {
private final Expression rowFilter;
private final boolean ignoreResiduals;
private final boolean caseSensitive;
- private final boolean colStats;
+ private boolean colStats;
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
