wuwenchi commented on code in PR #46911:
URL: https://github.com/apache/doris/pull/46911#discussion_r1972810852
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java:
##########
@@ -198,10 +197,51 @@ private void setPaimonParams(TFileRangeDesc rangeDesc,
PaimonSplit paimonSplit)
tDeletionFile.setLength(deletionFile.length());
fileDesc.setDeletionFile(tDeletionFile);
}
+ if (paimonSplit.getRowCount().isPresent()) {
+
tableFormatFileDesc.setTableLevelRowCount(paimonSplit.getRowCount().get());
+ }
tableFormatFileDesc.setPaimonParams(fileDesc);
rangeDesc.setTableFormatParams(tableFormatFileDesc);
}
+ @VisibleForTesting
+ public static Optional<Long>
calcuteTableLevelCount(List<org.apache.paimon.table.source.Split> paimonSplits)
{
+ // check if all splits don't have deletion vector or cardinality of
every
+ // deletion vector is not null
+ boolean applyTableCountPushdown = true;
+ long totalCount = 0;
+ long deletionVectorCount = 0;
+
+ for (org.apache.paimon.table.source.Split s : paimonSplits) {
+ totalCount += s.rowCount();
+
+ Optional<List<DeletionFile>> deletionFiles = s.deletionFiles();
+ if (deletionFiles.isPresent()) {
+ for (DeletionFile dv : deletionFiles.get()) {
+ if (dv != null) {
+ Long cardinality = dv.cardinality();
+ if (cardinality == null) {
+ applyTableCountPushdown = false;
+ break;
Review Comment:
You can directly return 'Optional.empty()' here.
And there is no need to judge this `applyTableCountPushdown` any more later.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java:
##########
@@ -295,33 +331,21 @@ public List<Split> getSplits(int numBackends) throws
UserException {
splitStats.add(splitStat);
}
+ // if applyCountPushdown is true, calcute row count for count pushdown
+ if (applyCountPushdown) {
+ Optional<Long> optTableLevelCount =
calcuteTableLevelCount(paimonSplits);
+ if (optTableLevelCount.isPresent()) {
+ long tableLevelRowCount = optTableLevelCount.get();
+ setPushDownCount(tableLevelRowCount);
+ assignCountToSplits(splits, tableLevelRowCount);
Review Comment:
We can make an optimization here. If there are too many splits, we can only
select a small number of splits. For specific details, you can refer to Iceberg.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
