pzhdfy commented on code in PR #68988:
URL: https://github.com/apache/doris/pull/68988#discussion_r4321186436
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java:
##########
@@ -1413,6 +1531,87 @@ public Map<String, String> getIncrReadParams() throws
UserException {
return paimonScanParams;
}
+ /**
+ * Plan a primary-key vector (ANN) search. Produces one Doris split per
+ * (partition, bucket) {@link BucketVectorSearchSplit}, serialized with the
+ * native "PKVSPLIT"/v1 protocol so paimon-rust on BE can decode it.
Planning
+ * only needs the vector column and the partition-half of the predicate;
the
+ * query vector and data (residual) predicates are handled by BE.
+ */
+ @VisibleForTesting
+ public List<Split> getVectorSearchSplits() throws UserException {
+ Table paimonTable = getProcessedTable();
+ // Only the partition-half of the predicate is pushed into Paimon
planning.
+ // The data (residual) half is left for BE's vector-search filter, and
passing
+ // it here would trip PrimaryKeyVectorScan's DV/merge-on-read
pre-filter guard.
+ PartitionPredicate partitionPredicate = null;
+ if (predicates != null && !predicates.isEmpty()) {
+ Pair<Optional<PartitionPredicate>, List<Predicate>> partSplit =
+
PartitionPredicate.splitPartitionPredicatesAndDataPredicates(
+ predicates, paimonTable.rowType(),
paimonTable.partitionKeys());
+ partitionPredicate = partSplit.getLeft().orElse(null);
+ }
+
+ // Resolve the actual Paimon field name (case-insensitive); the
pushed-down
+ // column name comes from the Doris column, which may differ in case.
+ String vectorColumn = resolvePaimonFieldName(paimonTable,
annSortColumnName);
+
+ VectorScan.Plan plan;
+ try {
+ plan = paimonTable.newBatchVectorSearchBuilder()
+ .withVectorColumn(vectorColumn)
+ .withPartitionFilter(partitionPredicate)
+ .newVectorScan()
+ .scan();
+ } catch (Exception e) {
+ throw new UserException("Paimon PK-vector search planning failed:
" + e.getMessage(), e);
+ }
+
+ List<Split> splits = new ArrayList<>();
+ Set<BinaryRow> partitions = new HashSet<>();
+ for (VectorSearchSplit vectorSplit : plan.splits()) {
+ if (!(vectorSplit instanceof BucketVectorSearchSplit)) {
+ throw new UserException("Unexpected PK-vector split type: "
+ + vectorSplit.getClass().getName());
+ }
+ BucketVectorSearchSplit bucketSplit = (BucketVectorSearchSplit)
vectorSplit;
+ // Serialize with Paimon's own DataOutputSerializer: the bytes
must be
+ // exactly BucketVectorSearchSplit.serialize's DataOutputView form
+ // (magic "PKVSPLIT", version 1, big-endian), which
+ // paimon_bucket_vector_search_split_deserialize on BE decodes.
This
+ // repository's Paimon no longer ships a
BucketVectorSearchSplitSerializer.
+ byte[] payloadBytes;
+ try {
+ DataOutputSerializer output = new DataOutputSerializer(256);
+ bucketSplit.serialize(output);
+ payloadBytes = output.getCopyOfBuffer();
+ } catch (IOException e) {
+ throw new UserException("Failed to serialize PK-vector split:
" + e.getMessage(), e);
+ }
+ DataSplit dataSplit = bucketSplit.dataSplit();
+ PaimonSplit split = new PaimonSplit(dataSplit);
+ split.setVectorPayloadBytes(payloadBytes);
Review Comment:
Confirmed — getVectorSearchSplits never called setPaimonPartitionValues,
and setPartitionValues throws "Missing partition values for Paimon partitioned
table" whenever the table has path partition keys. Fixed by mirroring the
ordinary split loop: partitionInfoMaps.computeIfAbsent per BinaryRow partition
via PaimonUtil.getPartitionInfoMap, set on every vector split. The regression
now carries a PARTITIONED BY (dt) PK-vector fixture with a brute-force
differential — this exact query failed with "Missing partition values" before
the fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
