github-actions[bot] commented on code in PR #66597:
URL: https://github.com/apache/doris/pull/66597#discussion_r3747577840
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanNode.java:
##########
@@ -154,14 +154,23 @@ public List<Split> getSplits(int numBackends) throws
UserException {
plannedVersion = metadata.getVersion();
plannedFragments = metadata.getFragments().size();
Set<Long> fragmentIds = new HashSet<>();
- List<Split> splits = new ArrayList<>(plannedFragments);
+ long targetRows = 1;
for (LanceTableMetadata.LanceFragmentInfo fragment :
metadata.getFragments()) {
if (!fragmentIds.add(fragment.getId())) {
throw new UserException("Duplicate Lance fragment id " +
fragment.getId()
+ " at dataset version " + metadata.getVersion());
}
- splits.add(new LanceSplit(metadata.getDatasetUri(),
metadata.getVersion(),
- fragment.getId(), fragment.getRowCount()));
+ targetRows = Math.max(targetRows,
Math.max(fragment.getRowCount(), 1));
Review Comment:
[P2] Base legacy-fragment weights on physical scan work
`fragment.getRowCount()` comes from Java `FragmentMetadata#getNumRows()`,
which is physical rows minus deletions. However, the BE links lance-c
0.1.2/Lance 4.0.1, whose legacy fragment reader reads and merges the projected
physical batches before applying the deletion vector. A legacy fragment with
1,000,000 physical rows and 990,000 tombstones can therefore receive only a 1
percent weight even though its scan still does close to a full fragment's
I/O/decoding work; compared with the previous equal weights, this can actively
co-locate expensive tombstone-heavy fragments. Please use physical rows or
another format-aware scan-cost metric for this normalization and cover a
deletion-heavy legacy fragment.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanNode.java:
##########
@@ -154,14 +154,23 @@ public List<Split> getSplits(int numBackends) throws
UserException {
plannedVersion = metadata.getVersion();
plannedFragments = metadata.getFragments().size();
Set<Long> fragmentIds = new HashSet<>();
- List<Split> splits = new ArrayList<>(plannedFragments);
+ long targetRows = 1;
for (LanceTableMetadata.LanceFragmentInfo fragment :
metadata.getFragments()) {
if (!fragmentIds.add(fragment.getId())) {
throw new UserException("Duplicate Lance fragment id " +
fragment.getId()
+ " at dataset version " + metadata.getVersion());
}
- splits.add(new LanceSplit(metadata.getDatasetUri(),
metadata.getVersion(),
- fragment.getId(), fragment.getRowCount()));
+ targetRows = Math.max(targetRows,
Math.max(fragment.getRowCount(), 1));
+ }
+
+ // Use the largest fragment as one standard split so smaller
fragments keep
+ // their relative row-count weight during backend assignment.
+ List<Split> splits = new ArrayList<>(plannedFragments);
+ for (LanceTableMetadata.LanceFragmentInfo fragment :
metadata.getFragments()) {
+ LanceSplit split = new LanceSplit(metadata.getDatasetUri(),
metadata.getVersion(),
+ fragment.getId(), fragment.getRowCount());
+ split.setTargetSplitSize(targetRows);
Review Comment:
[P2] Keep S3/file TVF fragment weights in sync
This sets the common target only for catalog-backed `LanceScanNode` splits.
The S3 Lance TVF path loads the same fragment row counts via
`loadLatestForTvf()`, but `setLanceTableMetadata()` retains only IDs;
`TVFScanNode#getLanceSplits()` then passes `rowCount=1` and never sets a
target. The `file(...)` TVF delegates S3 Lance scans to this same path, so both
entry points still give every fragment the standard scheduling weight and keep
the skew this change is intended to remove. Please retain row counts through
the TVF metadata boundary, apply the same normalization there, and add a parity
test for that split path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
