This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a commit to branch branch-4.2 in repository https://gitbox.apache.org/repos/asf/doris.git
commit 98739b59da51bc2d74fba568aac55bc971dbf5ec Author: Gabriel <[email protected]> AuthorDate: Sat Sep 26 16:51:42 2026 +0800 [opt](lance) skip full-snapshot row-id prefilters and add profile counters (#68375) ### What problem does this PR solve? An unfiltered Lance vector search explicitly scoped to every fragment in its snapshot can still scan all indexed row IDs and construct a redundant allow set. This adds work to every query, particularly under concurrency, and the existing profile cannot isolate set construction from input scanning. Integrate the narrowly scoped Lance v11 fix in https://github.com/lance-format/lance/pull/9460 through the existing lance-c patch chain. Pin every Lance crate and the lockfile to commit `f202fe41ac18323ca0cd7bc6efd8fd30b8722116`; the commit is fetchable from the official repository. The upstream PR is pending review. Full-snapshot unfiltered queries omit the row-ID loader; fragment subsets, real predicates, deletion/overlay masks, indexed segment selection, and unindexed fallback retain their semantics. The explicit fragment list is preserved. Expose seven profile metrics: `LancePrefilterLoads`, `LancePrefilterInputRows`, `LancePrefilterInputBatches`, `LancePrefilterRowIds`, `LancePrefilterLoadTime`, `LancePrefilterInputTime`, and `LancePrefilterBuildTime`. Document their scope and overlapping wall-time semantics, and extend indexed multi-vector reader tests to check filtered/non-filtered metrics after close. There is no C ABI or FE protocol change. ### ANN result compatibility Removing an all-row prefilter activates Lance's unfiltered distance calculation path. For 4-bit IVF_PQ, that path additionally quantizes the distance table and can change ANN candidates at a fixed refinement budget. The frozen dot fixture reproduces a Top-5 recall change from 4/5 to 3/5; this PR does not change the underlying PQ algorithm. As a test-only workaround, the dot-score golden probes all partitions and uses `refine_factor=256`, covering all 1024 fixture rows before exact reranking. Update its golden to the exact dot results. Existing index-selection and single-partition ANN checks retain their original settings. This workaround does not establish unchanged recall for production queries. ### Release note Avoid redundant row-ID prefilter materialization for full-snapshot Lance vector searches and add detailed prefilter profile counters. ### Validation - Lance: 84 prefilter tests and 30 fragment-scope/FTS tests passed, including execution-summary export, deleted rows, old snapshots, stable row IDs, pinned segments, and unindexed appends. - Lance: workspace rustfmt and `cargo clippy --profile ci --all --tests --benches -- -D warnings` passed. - Doris: syntax compilation of both changed C++ files passed with Lance enabled; clang-format 16 and shell syntax checks passed. - Applied the full nine-patch chain to a fresh lance-c v0.1.9 archive with `--fuzz=0`; Rust 1.91 locked Linux dependency metadata resolved all 21 Lance crates to the pinned commit without updating the lockfile. - Test workaround: Groovy compilation and 16 fixture checks against Lance 11.0.0 passed across filtered/unfiltered paths, including all five score goldens and the three single-partition discriminators. The full Doris regression suite was not run locally. - The full Doris BE unit-test binary and production concurrency benchmark were not run locally. CI and a benchmark at fixed recall are still needed; no end-to-end latency reduction is claimed. ### Dependency review follow-up The dependency pin includes the applicable review fixes from [Lance #9471](https://github.com/lance-format/lance/pull/9471), backported into the [v11 PR #9460](https://github.com/lance-format/lance/pull/9460). FTS row-ID materialization now exports the same seven loader metrics as ANN. Coverage includes full-fragment ANN/FTS scopes and a deterministic 4-bit PQ recall fixture that actually exercises bulk distance-table quantization, including default postfilter queries. PQ scoring itself is unchanged. Keep the original dependency patch as the upgrade baseline and apply `lance-c-0.1.9-prefilter-fts.patch` with an independent marker. Existing caches containing the old prefilter patch must receive the new pin; simply replacing the old patch contents would leave those caches unchanged. Both Doris PRs carry identical patch payloads. Follow-up validation: - Lance v11: 84 prefilter tests and 30 additional fragment-scope/FTS tests passed; workspace rustfmt and Clippy passed. - Verified the release archive checksum, applied both actual Doris patch blocks with `--fuzz=0` to fresh and previously patched sources, and verified identical resulting trees and idempotent reruns. - Incompatible cached input fails without writing the new success marker. - Rust 1.91 locked Linux dependency metadata resolves all 21 Lance-source packages to the new revision without modifying Cargo.lock. - Shell syntax and whitespace checks passed. This refresh does not include a full third-party build or Doris regression run; PR CI is required. ### Check List (For Author) - [x] Unit Test: extend the existing Lance reader test and add upstream planner/loader tests. - [x] Behavior changed: redundant full-snapshot prefilter work is skipped; 4-bit PQ ANN candidates may change as described above. - [x] Documentation: `docs/lance-prefilter-profile.md` describes the counters and validation method. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label --- be/src/format_v2/table/lance_reader.cpp | 29 +++- be/test/format_v2/table/lance_reader_test.cpp | 16 +++ docs/lance-prefilter-profile.md | 58 ++++++++ .../lance/test_lance_vector_search_metrics.out | 2 +- .../lance/test_lance_vector_search_metrics.groovy | 5 +- thirdparty/download-thirdparty.sh | 11 +- .../patches/lance-c-0.1.9-prefilter-fts.patch | 148 +++++++++++++++++++++ thirdparty/patches/lance-c-0.1.9-prefilter.patch | 147 ++++++++++++++++++++ 8 files changed, 409 insertions(+), 7 deletions(-) diff --git a/be/src/format_v2/table/lance_reader.cpp b/be/src/format_v2/table/lance_reader.cpp index 840338e4b33..724495bf990 100644 --- a/be/src/format_v2/table/lance_reader.cpp +++ b/be/src/format_v2/table/lance_reader.cpp @@ -651,10 +651,21 @@ void LanceTableReader::_init_scanner_profile() { _index_comparisons = ADD_CHILD_COUNTER_WITH_LEVEL(_scanner_profile, "LanceIndexComparisons", TUnit::UNIT, LANCE_READER_PROFILE, 1); - // These scan counts are emitted by Lance's FilteredRead execution node. For vector searches - // with an explicit fragment set, they normally describe the fragments, ranges, and rows read - // while applying the row-id prefilter. They are scan input counts, not ANN result counts. + // Prefilter counters isolate row-id materialization. The generic scan counts below come + // from Lance's FilteredRead execution node and are scan inputs, not ANN result counts. _lance_count_metrics = { + {"prefilter_loads", + ADD_CHILD_COUNTER_WITH_LEVEL(_scanner_profile, "LancePrefilterLoads", TUnit::UNIT, + LANCE_READER_PROFILE, 1)}, + {"prefilter_input_rows", + ADD_CHILD_COUNTER_WITH_LEVEL(_scanner_profile, "LancePrefilterInputRows", TUnit::UNIT, + LANCE_READER_PROFILE, 1)}, + {"prefilter_input_batches", + ADD_CHILD_COUNTER_WITH_LEVEL(_scanner_profile, "LancePrefilterInputBatches", + TUnit::UNIT, LANCE_READER_PROFILE, 1)}, + {"prefilter_row_ids", + ADD_CHILD_COUNTER_WITH_LEVEL(_scanner_profile, "LancePrefilterRowIds", TUnit::UNIT, + LANCE_READER_PROFILE, 1)}, {"fragments_scanned", ADD_CHILD_COUNTER_WITH_LEVEL(_scanner_profile, "LanceFragmentsScanned", TUnit::UNIT, LANCE_READER_PROFILE, 1)}, @@ -686,6 +697,18 @@ void LanceTableReader::_init_scanner_profile() { TUnit::UNIT, LANCE_READER_PROFILE, 1)}, }; _lance_time_metrics = { + // These are wall times in the ANN row-id loader. LoadTime includes input polling + // and set construction; it must not be added to its component timers. + {"prefilter_load_time", + ADD_CHILD_TIMER_WITH_LEVEL(_scanner_profile, "LancePrefilterLoadTime", + LANCE_READER_PROFILE, 1)}, + {"prefilter_input_time", + ADD_CHILD_TIMER_WITH_LEVEL(_scanner_profile, "LancePrefilterInputTime", + LANCE_READER_PROFILE, 1)}, + {"prefilter_build_time", + ADD_CHILD_TIMER_WITH_LEVEL(_scanner_profile, "LancePrefilterBuildTime", + LANCE_READER_PROFILE, 1)}, + // This is wait time reported by the same Lance scan execution node described above, // rather than Doris scanner scheduling wait time. {"task_wait_time", ADD_CHILD_TIMER_WITH_LEVEL(_scanner_profile, "LanceTaskWaitTime", diff --git a/be/test/format_v2/table/lance_reader_test.cpp b/be/test/format_v2/table/lance_reader_test.cpp index b0e1b2bb831..a751e3aa80e 100644 --- a/be/test/format_v2/table/lance_reader_test.cpp +++ b/be/test/format_v2/table/lance_reader_test.cpp @@ -911,6 +911,22 @@ TEST(LanceTableReaderVectorSearchTest, MultiVectorScoresFiltersOffsetsAndIndexed } } EXPECT_TRUE(reader.close().ok()); + if (indexed) { + // Read metrics after close: lance-c publishes its final execution summary + // when the stream is released, including for an early top-k stop. + for (const char* name : {"LancePrefilterLoads", "LancePrefilterInputRows", + "LancePrefilterInputBatches", "LancePrefilterRowIds", + "LancePrefilterLoadTime", "LancePrefilterInputTime", + "LancePrefilterBuildTime"}) { + auto* counter = profile.get_counter(name); + ASSERT_NE(nullptr, counter) << name; + if (filtered) { + EXPECT_GT(counter->value(), 0) << name; + } else { + EXPECT_EQ(counter->value(), 0) << name; + } + } + } } } } diff --git a/docs/lance-prefilter-profile.md b/docs/lance-prefilter-profile.md new file mode 100644 index 00000000000..c387a076d77 --- /dev/null +++ b/docs/lance-prefilter-profile.md @@ -0,0 +1,58 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Lance vector row-ID prefilter profiling + +An explicit fragment list that covers every fragment in a fixed dataset snapshot does +not restrict an unfiltered vector query. Lance can omit the row-ID prefilter scan in +this case. The fragment selection remains attached to the scanner: indexed segment +selection, unindexed-fragment fallback, snapshot visibility, deletion masks and overlay +handling keep their existing semantics. A strict fragment subset or an actual filter +continues to use the normal prefilter path. + +The following Doris counters describe Lance's ANN **row-ID prefilter loader**, not +returned TopK rows, HNSW comparisons, or the deletion mask. Scalar-index selection +vectors use a different loader and are not included in these counters. + +| Counter | Meaning | +| --- | --- | +| `LancePrefilterLoads` | Number of row-ID prefilter loader executions started. | +| `LancePrefilterInputBatches` | Successfully consumed input batches. | +| `LancePrefilterInputRows` | Non-null input row IDs, including duplicates. | +| `LancePrefilterRowIds` | Sum of distinct row IDs in successfully completed allow sets. | +| `LancePrefilterLoadTime` | Total loader wall time, including input polling and set construction. | +| `LancePrefilterInputTime` | Wall time polling input batches, including upstream execution, I/O, decoding and scheduling. | +| `LancePrefilterBuildTime` | Wall time inserting row IDs into the allow set, measured once per batch. | + +The timers overlap: do not add LoadTime to InputTime or BuildTime. They are not CPU +timers. Across multiple loaders or scanners they accumulate and can exceed query +wall time. RowIds is not peak resident memory and can count the same ID again when +separate loaders build separate sets. An interrupted or failed load may contribute +partial input counts without a completed set cardinality. + +For a full-snapshot, unfiltered ANN query, no row-ID loader is needed and these +counters remain zero. Zero does not prove that no filtering occurred: native +visibility/deletion filtering and scalar-index selection vectors are independent. +The generic `LanceRowsScanned` counter can still include result materialization or +unindexed fallback work; it is not an exact count of distance comparisons. + +To validate performance, hold the dataset version, query vectors, search parameters, +cache state and recall target constant. Compare serial and concurrent runs using QPS, +latency percentiles, process CPU and these counters. Removal of the redundant row-ID +scan does not by itself establish the size of the end-to-end latency improvement. diff --git a/regression-test/data/external_table_p0/lance/test_lance_vector_search_metrics.out b/regression-test/data/external_table_p0/lance/test_lance_vector_search_metrics.out index a155038bb0a..cadaee651b7 100644 --- a/regression-test/data/external_table_p0/lance/test_lance_vector_search_metrics.out +++ b/regression-test/data/external_table_p0/lance/test_lance_vector_search_metrics.out @@ -33,10 +33,10 @@ row_id bigint No false \N -- !ivf_pq_dot -- 1 item-0001 -8.262586 +408 item-0408 -5.387761 819 item-0819 -5.1706505 229 item-0229 -5.1260695 800 item-0800 -4.9694424 -452 item-0452 -4.5769033 -- !flat_cosine -- 1 item-0001 0.0 diff --git a/regression-test/suites/external_table_p0/lance/test_lance_vector_search_metrics.groovy b/regression-test/suites/external_table_p0/lance/test_lance_vector_search_metrics.groovy index b557bc910a4..2e5686fb8bb 100644 --- a/regression-test/suites/external_table_p0/lance/test_lance_vector_search_metrics.groovy +++ b/regression-test/suites/external_table_p0/lance/test_lance_vector_search_metrics.groovy @@ -220,9 +220,12 @@ suite("test_lance_vector_search_metrics", "p0,external") { // generator skips the "the query's own row is at distance 0" assertion whenever the metric // is dot. Lance reports the score as a negated inner product, so ORDER BY _distance ASC // still puts the best match first and every distance below is negative. + // Work around different candidate rankings in Lance's filtered and unfiltered 4-bit + // PQ paths: refine all 1024 fixture rows (5 * 256 > 1024) while probing all partitions. + // This golden checks dot scores; the single-probe checks above retain ANN coverage. qt_ivf_pq_dot """ SELECT row_id, label, _distance - FROM ${search("vs_ivf_pq_f32_dot", headQuery, "5", "4", "dot", tables["vs_ivf_pq_f32_dot"].refine)} + FROM ${search("vs_ivf_pq_f32_dot", headQuery, "5", "4", "dot", ', "refine_factor"="256"')} ORDER BY _distance, row_id """ diff --git a/thirdparty/download-thirdparty.sh b/thirdparty/download-thirdparty.sh index d0c2eb0511c..e40a6f3d8b8 100755 --- a/thirdparty/download-thirdparty.sh +++ b/thirdparty/download-thirdparty.sh @@ -721,7 +721,7 @@ fi # Apply Doris lance-c patches as one chain to the pinned release archive. if [[ " ${TP_ARCHIVES[*]} " =~ " LANCE_C " ]]; then cd "${TP_SOURCE_DIR}/${LANCE_C_SOURCE}" - LANCE_C_PATCHED_MARK="${PATCHED_MARK}_community_pr83" + LANCE_C_PATCHED_MARK="${PATCHED_MARK}_community_pr83_prefilter" # Older source caches carry a different PR #73 and cannot accept this chain incrementally. if [[ -f "${PATCHED_MARK}" && ! -f "${LANCE_C_PATCHED_MARK}" ]]; then echo "The lance-c patch chain changed; remove ${TP_SOURCE_DIR}/${LANCE_C_SOURCE} and rebuild." @@ -730,12 +730,19 @@ if [[ " ${TP_ARCHIVES[*]} " =~ " LANCE_C " ]]; then if [[ ! -f "${LANCE_C_PATCHED_MARK}" ]]; then # PR #77 provides Lance v11 for the following community patches. PR #83 # retains PR #79's scalar-segment path when adding multi-vector execution. - for lance_patch in pr-74 pr-75-pr-78 pr-77 pr-73 pr-79 pr-80 pr-83; do + # The final patch pins the full-snapshot prefilter fix and its execution metrics. + for lance_patch in pr-74 pr-75-pr-78 pr-77 pr-73 pr-79 pr-80 pr-83 prefilter; do patch --batch --forward --reject-file=- --fuzz=0 --no-backup-if-mismatch -s \ -p1 <"${TP_PATCH_DIR}/${LANCE_C_SOURCE}-${lance_patch}.patch" done touch "${PATCHED_MARK}" "${LANCE_C_PATCHED_MARK}" fi + # Cached sources may carry the earlier prefilter pin; upgrade FTS metrics independently. + if [[ ! -f "${PATCHED_MARK}_prefilter_fts" ]]; then + patch --batch --forward --reject-file=- --fuzz=0 --no-backup-if-mismatch -s \ + -p1 <"${TP_PATCH_DIR}/${LANCE_C_SOURCE}-prefilter-fts.patch" + touch "${PATCHED_MARK}_prefilter_fts" + fi cd - echo "Finished patching ${LANCE_C_SOURCE}" fi diff --git a/thirdparty/patches/lance-c-0.1.9-prefilter-fts.patch b/thirdparty/patches/lance-c-0.1.9-prefilter-fts.patch new file mode 100644 index 00000000000..890093886fb --- /dev/null +++ b/thirdparty/patches/lance-c-0.1.9-prefilter-fts.patch @@ -0,0 +1,148 @@ +Subject: [PATCH] Include FTS prefilter metrics and full-snapshot test coverage + +Upstream: https://github.com/lance-format/lance/pull/9460 +Commit: f202fe41ac18323ca0cd7bc6efd8fd30b8722116 + +Backport the applicable review updates from Lance PR #9471 to v11. +Keep the C API and other dependencies unchanged, and upgrade previously +patched source caches with the same revision as fresh builds. + +diff --git a/Cargo.toml b/Cargo.toml +--- a/Cargo.toml ++++ b/Cargo.toml +@@ -20,10 +20,10 @@ + [dependencies] +-lance = { git = "https://github.com/lance-format/lance.git";, rev = "f75f3343b5e125c42da1bd7acc8d8217cd5660a6", features = ["substrait"] } +-lance-core = { git = "https://github.com/lance-format/lance.git";, rev = "f75f3343b5e125c42da1bd7acc8d8217cd5660a6" } +-lance-file = { git = "https://github.com/lance-format/lance.git";, rev = "f75f3343b5e125c42da1bd7acc8d8217cd5660a6" } +-lance-index = { git = "https://github.com/lance-format/lance.git";, rev = "f75f3343b5e125c42da1bd7acc8d8217cd5660a6" } +-lance-io = { git = "https://github.com/lance-format/lance.git";, rev = "f75f3343b5e125c42da1bd7acc8d8217cd5660a6" } +-lance-linalg = { git = "https://github.com/lance-format/lance.git";, rev = "f75f3343b5e125c42da1bd7acc8d8217cd5660a6" } +-lance-table = { git = "https://github.com/lance-format/lance.git";, rev = "f75f3343b5e125c42da1bd7acc8d8217cd5660a6" } +-lance-datafusion = { git = "https://github.com/lance-format/lance.git";, rev = "f75f3343b5e125c42da1bd7acc8d8217cd5660a6", features = ["substrait"] } ++lance = { git = "https://github.com/lance-format/lance.git";, rev = "f202fe41ac18323ca0cd7bc6efd8fd30b8722116", features = ["substrait"] } ++lance-core = { git = "https://github.com/lance-format/lance.git";, rev = "f202fe41ac18323ca0cd7bc6efd8fd30b8722116" } ++lance-file = { git = "https://github.com/lance-format/lance.git";, rev = "f202fe41ac18323ca0cd7bc6efd8fd30b8722116" } ++lance-index = { git = "https://github.com/lance-format/lance.git";, rev = "f202fe41ac18323ca0cd7bc6efd8fd30b8722116" } ++lance-io = { git = "https://github.com/lance-format/lance.git";, rev = "f202fe41ac18323ca0cd7bc6efd8fd30b8722116" } ++lance-linalg = { git = "https://github.com/lance-format/lance.git";, rev = "f202fe41ac18323ca0cd7bc6efd8fd30b8722116" } ++lance-table = { git = "https://github.com/lance-format/lance.git";, rev = "f202fe41ac18323ca0cd7bc6efd8fd30b8722116" } ++lance-datafusion = { git = "https://github.com/lance-format/lance.git";, rev = "f202fe41ac18323ca0cd7bc6efd8fd30b8722116", features = ["substrait"] } + datafusion = { version = "54.0.0", default-features = false } +@@ -53,5 +53,5 @@ + [dev-dependencies] +-lance = { git = "https://github.com/lance-format/lance.git";, rev = "f75f3343b5e125c42da1bd7acc8d8217cd5660a6", features = ["substrait"] } +-lance-datagen = { git = "https://github.com/lance-format/lance.git";, rev = "f75f3343b5e125c42da1bd7acc8d8217cd5660a6" } +-lance-file = { git = "https://github.com/lance-format/lance.git";, rev = "f75f3343b5e125c42da1bd7acc8d8217cd5660a6" } ++lance = { git = "https://github.com/lance-format/lance.git";, rev = "f202fe41ac18323ca0cd7bc6efd8fd30b8722116", features = ["substrait"] } ++lance-datagen = { git = "https://github.com/lance-format/lance.git";, rev = "f202fe41ac18323ca0cd7bc6efd8fd30b8722116" } ++lance-file = { git = "https://github.com/lance-format/lance.git";, rev = "f202fe41ac18323ca0cd7bc6efd8fd30b8722116" } + tokio = { version = "1", features = ["rt-multi-thread", "macros"] } +diff --git a/Cargo.lock b/Cargo.lock +--- a/Cargo.lock ++++ b/Cargo.lock +@@ -2608,3 +2608,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; ++source = "git+https://github.com/lance-format/lance.git?rev=f202fe41ac18323ca0cd7bc6efd8fd30b8722116#f202fe41ac18323ca0cd7bc6efd8fd30b8722116"; + dependencies = [ +@@ -3820,3 +3820,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; ++source = "git+https://github.com/lance-format/lance.git?rev=f202fe41ac18323ca0cd7bc6efd8fd30b8722116#f202fe41ac18323ca0cd7bc6efd8fd30b8722116"; + dependencies = [ +@@ -3892,3 +3892,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; ++source = "git+https://github.com/lance-format/lance.git?rev=f202fe41ac18323ca0cd7bc6efd8fd30b8722116#f202fe41ac18323ca0cd7bc6efd8fd30b8722116"; + dependencies = [ +@@ -3914,3 +3914,3 @@ + version = "58.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; ++source = "git+https://github.com/lance-format/lance.git?rev=f202fe41ac18323ca0cd7bc6efd8fd30b8722116#f202fe41ac18323ca0cd7bc6efd8fd30b8722116"; + dependencies = [ +@@ -3928,3 +3928,3 @@ + version = "58.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; ++source = "git+https://github.com/lance-format/lance.git?rev=f202fe41ac18323ca0cd7bc6efd8fd30b8722116#f202fe41ac18323ca0cd7bc6efd8fd30b8722116"; + dependencies = [ +@@ -3938,3 +3938,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; ++source = "git+https://github.com/lance-format/lance.git?rev=f202fe41ac18323ca0cd7bc6efd8fd30b8722116#f202fe41ac18323ca0cd7bc6efd8fd30b8722116"; + dependencies = [ +@@ -3984,3 +3984,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; ++source = "git+https://github.com/lance-format/lance.git?rev=f202fe41ac18323ca0cd7bc6efd8fd30b8722116#f202fe41ac18323ca0cd7bc6efd8fd30b8722116"; + dependencies = [ +@@ -4022,3 +4022,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; ++source = "git+https://github.com/lance-format/lance.git?rev=f202fe41ac18323ca0cd7bc6efd8fd30b8722116#f202fe41ac18323ca0cd7bc6efd8fd30b8722116"; + dependencies = [ +@@ -4054,3 +4054,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; ++source = "git+https://github.com/lance-format/lance.git?rev=f202fe41ac18323ca0cd7bc6efd8fd30b8722116#f202fe41ac18323ca0cd7bc6efd8fd30b8722116"; + dependencies = [ +@@ -4072,3 +4072,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; ++source = "git+https://github.com/lance-format/lance.git?rev=f202fe41ac18323ca0cd7bc6efd8fd30b8722116#f202fe41ac18323ca0cd7bc6efd8fd30b8722116"; + dependencies = [ +@@ -4082,3 +4082,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; ++source = "git+https://github.com/lance-format/lance.git?rev=f202fe41ac18323ca0cd7bc6efd8fd30b8722116#f202fe41ac18323ca0cd7bc6efd8fd30b8722116"; + dependencies = [ +@@ -4116,3 +4116,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; ++source = "git+https://github.com/lance-format/lance.git?rev=f202fe41ac18323ca0cd7bc6efd8fd30b8722116#f202fe41ac18323ca0cd7bc6efd8fd30b8722116"; + dependencies = [ +@@ -4148,3 +4148,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; ++source = "git+https://github.com/lance-format/lance.git?rev=f202fe41ac18323ca0cd7bc6efd8fd30b8722116#f202fe41ac18323ca0cd7bc6efd8fd30b8722116"; + dependencies = [ +@@ -4163,3 +4163,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; ++source = "git+https://github.com/lance-format/lance.git?rev=f202fe41ac18323ca0cd7bc6efd8fd30b8722116#f202fe41ac18323ca0cd7bc6efd8fd30b8722116"; + dependencies = [ +@@ -4231,3 +4231,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; ++source = "git+https://github.com/lance-format/lance.git?rev=f202fe41ac18323ca0cd7bc6efd8fd30b8722116#f202fe41ac18323ca0cd7bc6efd8fd30b8722116"; + dependencies = [ +@@ -4254,3 +4254,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; ++source = "git+https://github.com/lance-format/lance.git?rev=f202fe41ac18323ca0cd7bc6efd8fd30b8722116#f202fe41ac18323ca0cd7bc6efd8fd30b8722116"; + dependencies = [ +@@ -4294,3 +4294,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; ++source = "git+https://github.com/lance-format/lance.git?rev=f202fe41ac18323ca0cd7bc6efd8fd30b8722116#f202fe41ac18323ca0cd7bc6efd8fd30b8722116"; + dependencies = [ +@@ -4309,3 +4309,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; ++source = "git+https://github.com/lance-format/lance.git?rev=f202fe41ac18323ca0cd7bc6efd8fd30b8722116#f202fe41ac18323ca0cd7bc6efd8fd30b8722116"; + dependencies = [ +@@ -4336,3 +4336,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; ++source = "git+https://github.com/lance-format/lance.git?rev=f202fe41ac18323ca0cd7bc6efd8fd30b8722116#f202fe41ac18323ca0cd7bc6efd8fd30b8722116"; + dependencies = [ +@@ -4351,3 +4351,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; ++source = "git+https://github.com/lance-format/lance.git?rev=f202fe41ac18323ca0cd7bc6efd8fd30b8722116#f202fe41ac18323ca0cd7bc6efd8fd30b8722116"; + dependencies = [ +@@ -4390,3 +4390,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; ++source = "git+https://github.com/lance-format/lance.git?rev=f202fe41ac18323ca0cd7bc6efd8fd30b8722116#f202fe41ac18323ca0cd7bc6efd8fd30b8722116"; + dependencies = [ diff --git a/thirdparty/patches/lance-c-0.1.9-prefilter.patch b/thirdparty/patches/lance-c-0.1.9-prefilter.patch new file mode 100644 index 00000000000..c3df2f5d262 --- /dev/null +++ b/thirdparty/patches/lance-c-0.1.9-prefilter.patch @@ -0,0 +1,147 @@ +Subject: [PATCH] Use Lance full-snapshot prefilter optimization and loader metrics + +Upstream: https://github.com/lance-format/lance/pull/9460 +Commit: f75f3343b5e125c42da1bd7acc8d8217cd5660a6 + +Pin the tested Lance v11 change without upgrading the release or changing +the C API. Keep all Lance crates on the same revision. + +diff --git a/Cargo.toml b/Cargo.toml +--- a/Cargo.toml ++++ b/Cargo.toml +@@ -20,10 +20,10 @@ + [dependencies] +-lance = { git = "https://github.com/lance-format/lance.git";, rev = "ab6b5bbe", features = ["substrait"] } +-lance-core = { git = "https://github.com/lance-format/lance.git";, rev = "ab6b5bbe" } +-lance-file = { git = "https://github.com/lance-format/lance.git";, rev = "ab6b5bbe" } +-lance-index = { git = "https://github.com/lance-format/lance.git";, rev = "ab6b5bbe" } +-lance-io = { git = "https://github.com/lance-format/lance.git";, rev = "ab6b5bbe" } +-lance-linalg = { git = "https://github.com/lance-format/lance.git";, rev = "ab6b5bbe" } +-lance-table = { git = "https://github.com/lance-format/lance.git";, rev = "ab6b5bbe" } +-lance-datafusion = { git = "https://github.com/lance-format/lance.git";, rev = "ab6b5bbe", features = ["substrait"] } ++lance = { git = "https://github.com/lance-format/lance.git";, rev = "f75f3343b5e125c42da1bd7acc8d8217cd5660a6", features = ["substrait"] } ++lance-core = { git = "https://github.com/lance-format/lance.git";, rev = "f75f3343b5e125c42da1bd7acc8d8217cd5660a6" } ++lance-file = { git = "https://github.com/lance-format/lance.git";, rev = "f75f3343b5e125c42da1bd7acc8d8217cd5660a6" } ++lance-index = { git = "https://github.com/lance-format/lance.git";, rev = "f75f3343b5e125c42da1bd7acc8d8217cd5660a6" } ++lance-io = { git = "https://github.com/lance-format/lance.git";, rev = "f75f3343b5e125c42da1bd7acc8d8217cd5660a6" } ++lance-linalg = { git = "https://github.com/lance-format/lance.git";, rev = "f75f3343b5e125c42da1bd7acc8d8217cd5660a6" } ++lance-table = { git = "https://github.com/lance-format/lance.git";, rev = "f75f3343b5e125c42da1bd7acc8d8217cd5660a6" } ++lance-datafusion = { git = "https://github.com/lance-format/lance.git";, rev = "f75f3343b5e125c42da1bd7acc8d8217cd5660a6", features = ["substrait"] } + datafusion = { version = "54.0.0", default-features = false } +@@ -53,5 +53,5 @@ + [dev-dependencies] +-lance = { git = "https://github.com/lance-format/lance.git";, rev = "ab6b5bbe", features = ["substrait"] } +-lance-datagen = { git = "https://github.com/lance-format/lance.git";, rev = "ab6b5bbe" } +-lance-file = { git = "https://github.com/lance-format/lance.git";, rev = "ab6b5bbe" } ++lance = { git = "https://github.com/lance-format/lance.git";, rev = "f75f3343b5e125c42da1bd7acc8d8217cd5660a6", features = ["substrait"] } ++lance-datagen = { git = "https://github.com/lance-format/lance.git";, rev = "f75f3343b5e125c42da1bd7acc8d8217cd5660a6" } ++lance-file = { git = "https://github.com/lance-format/lance.git";, rev = "f75f3343b5e125c42da1bd7acc8d8217cd5660a6" } + tokio = { version = "1", features = ["rt-multi-thread", "macros"] } +diff --git a/Cargo.lock b/Cargo.lock +--- a/Cargo.lock ++++ b/Cargo.lock +@@ -2608,3 +2608,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=ab6b5bbe#ab6b5bbe46009ed78746b444df8db59a8bc5d842"; ++source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; + dependencies = [ +@@ -3820,3 +3820,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=ab6b5bbe#ab6b5bbe46009ed78746b444df8db59a8bc5d842"; ++source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; + dependencies = [ +@@ -3892,3 +3892,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=ab6b5bbe#ab6b5bbe46009ed78746b444df8db59a8bc5d842"; ++source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; + dependencies = [ +@@ -3914,3 +3914,3 @@ + version = "58.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=ab6b5bbe#ab6b5bbe46009ed78746b444df8db59a8bc5d842"; ++source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; + dependencies = [ +@@ -3928,3 +3928,3 @@ + version = "58.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=ab6b5bbe#ab6b5bbe46009ed78746b444df8db59a8bc5d842"; ++source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; + dependencies = [ +@@ -3938,3 +3938,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=ab6b5bbe#ab6b5bbe46009ed78746b444df8db59a8bc5d842"; ++source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; + dependencies = [ +@@ -3984,3 +3984,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=ab6b5bbe#ab6b5bbe46009ed78746b444df8db59a8bc5d842"; ++source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; + dependencies = [ +@@ -4022,3 +4022,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=ab6b5bbe#ab6b5bbe46009ed78746b444df8db59a8bc5d842"; ++source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; + dependencies = [ +@@ -4054,3 +4054,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=ab6b5bbe#ab6b5bbe46009ed78746b444df8db59a8bc5d842"; ++source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; + dependencies = [ +@@ -4072,3 +4072,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=ab6b5bbe#ab6b5bbe46009ed78746b444df8db59a8bc5d842"; ++source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; + dependencies = [ +@@ -4082,3 +4082,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=ab6b5bbe#ab6b5bbe46009ed78746b444df8db59a8bc5d842"; ++source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; + dependencies = [ +@@ -4116,3 +4116,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=ab6b5bbe#ab6b5bbe46009ed78746b444df8db59a8bc5d842"; ++source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; + dependencies = [ +@@ -4148,3 +4148,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=ab6b5bbe#ab6b5bbe46009ed78746b444df8db59a8bc5d842"; ++source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; + dependencies = [ +@@ -4163,3 +4163,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=ab6b5bbe#ab6b5bbe46009ed78746b444df8db59a8bc5d842"; ++source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; + dependencies = [ +@@ -4231,3 +4231,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=ab6b5bbe#ab6b5bbe46009ed78746b444df8db59a8bc5d842"; ++source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; + dependencies = [ +@@ -4254,3 +4254,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=ab6b5bbe#ab6b5bbe46009ed78746b444df8db59a8bc5d842"; ++source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; + dependencies = [ +@@ -4294,3 +4294,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=ab6b5bbe#ab6b5bbe46009ed78746b444df8db59a8bc5d842"; ++source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; + dependencies = [ +@@ -4309,3 +4309,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=ab6b5bbe#ab6b5bbe46009ed78746b444df8db59a8bc5d842"; ++source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; + dependencies = [ +@@ -4336,3 +4336,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=ab6b5bbe#ab6b5bbe46009ed78746b444df8db59a8bc5d842"; ++source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; + dependencies = [ +@@ -4351,3 +4351,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=ab6b5bbe#ab6b5bbe46009ed78746b444df8db59a8bc5d842"; ++source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; + dependencies = [ +@@ -4390,3 +4390,3 @@ + version = "11.0.0" +-source = "git+https://github.com/lance-format/lance.git?rev=ab6b5bbe#ab6b5bbe46009ed78746b444df8db59a8bc5d842"; ++source = "git+https://github.com/lance-format/lance.git?rev=f75f3343b5e125c42da1bd7acc8d8217cd5660a6#f75f3343b5e125c42da1bd7acc8d8217cd5660a6"; + dependencies = [ --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
