morningman opened a new pull request, #69581:
URL: https://github.com/apache/doris/pull/69581
### What problem does this PR solve?
Issue Number: None
Related PR: #65730 #66608 #66805 #66841 #66999 #67005 #67011 #67110 #67268
#67289 #67325 #67650 #67921 #68028 #68335 #68375 #68615 #68690
Problem Summary:
**In short.** This ports the Lance BE read path of branch-4.1 / branch-4.2
to master, in its branch-4.2 form as of `cabcf45c1af`; it gathers the BE parts
of the PRs listed above. Master links `liblance_c` but has no Lance reader, so
every `FORMAT_LANCE` scan fails today. With this PR the BE reads Lance datasets
through FileScannerV2: plain scans, vector search and full-text search,
predicate and runtime-filter pushdown, the schema fetch of a local Lance TVF,
and the second phase of TopN lazy materialization. The PR is BE only. The FE
connector plugin and the regression suites come in separate PRs, so until then
nothing on master plans a Lance scan, and the BE unit tests carry the proof.
Original authors: @zhangstar333 @Gabriel39 @wangzhaobo957-cloud @Jay-ju
@FANNG1
**Background.** Lance shipped in 4.1.4: the Lance catalog, `local()` /
`s3()` with `format=lance`, and the `vector_search` / `full_text_search` TVFs.
On master the FE side is being rebuilt as a connector plugin
(`fe-connector-lance`), and the BE needs the reader from 4.x. The thrift ids
are the branch-4.1 / 4.2 ones, so a BE built from master also reads the Lance
scan ranges that a 4.x FE sends during a rolling upgrade.
**The problem, and what it cost.**
| | master today | with this PR |
|---|---|---|
| A `FORMAT_LANCE` scan | `file_scan_operator` keeps it off FileScannerV2
and V1 has no Lance path, so the query fails | read by `LanceTableReader` in
FileScannerV2 |
| `fetch_table_schema` for a local Lance TVF | not handled | returns the
Arrow schema mapped to Doris types; Null leaves keep `NULL_TYPE` |
| The row location of a Lance row in TopN lazy materialization |
`GlobalRowLoacationV2` only has a 32-bit row id, but a Lance row id is a 64-bit
id within the snapshot | a version 1 layout carries the 64-bit row id; phase
two reads the rows with the dataset's `take_rows` |
**How this PR fixes it.**
- Reader: `be/src/format_v2/lance/*` and
`be/src/format_v2/table/lance_reader.*`, as on branch-4.2 except one local
renamed for master's `-Wshadow`. The unit tests construct
`RuntimeState(TQueryOptions(), globals)`, because master removed the
globals-only constructor.
- FileScannerV2: `is_supported`, the reader dispatch and `_to_file_format`
handle `FORMAT_LANCE` / `"lance"`; `FileFormat::LANCE` is added to its two
exhaustive switches. Lance queries use FileScannerV2 whatever
`enable_file_scanner_v2` says, as ADBC and fluss do: there is no V1 Lance
reader, and the variable is fuzzy in the regression harness. On branch-4.2 a
Lance query fails with NotSupported when the variable is off. Loads are
unchanged.
- Row locations: `ROW_VERSION {FILE_LOCAL_ROW_ID = 0, LANCE_DATASET_ROW_ID =
1}` and a union layout for `GlobalRowLoacationV2`. Version 0 keeps its byte
layout, which static asserts pin. `PRequestBlockDesc.row_id` becomes `uint64`
(a proto2 varint, so the wire format is compatible). The materialization
operator decodes row locations by version, rejects an unknown version or a
wrong size, and stamps `row_location_version`. `RowIdStorageReader` validates
the counts and the version, range-checks internal row ids against `uint32`, and
groups Lance rows by plan node, snapshot version and dataset URI, so each
dataset is read with one `take_rows`. Master's segment batching, scan-slot
dedup and fetch concurrency stay as they are.
- Thrift (`PlanNodes.thrift`): the external search request body
(`TVectorElementType`, `TVectorMetric`, `TSearchVector`, `TVectorSearchParams`,
`TFtsCoverageMode`, `TFtsQueryType`, `TFtsMatchOperator`,
`TFullTextSearchParams`, `TSearchFilterFormat`, `TSearchFilter`,
`TVectorSearchOptions`, `TExternalSearchQuery`), `TExternalSearchRequest` 2-4
and `TLanceFileDesc` 4-6, all with the branch-4.1 / 4.2 ids. The enums spell
out their 4.x values. No id is reused.
- Core: `to_protobuf` keeps `NULL_TYPE` for a NULL literal type, so a Null
leaf in ARRAY / MAP / STRUCT no longer comes back as BOOLEAN. The nullable
serde reads an Arrow Null array as NULLs without touching buffers that the
array does not have.
- Configs, caches and metrics, with the 4.x names and defaults:
| Config | Default |
|---|---|
| `lance_index_cache_size_bytes` | 10 GiB |
| `lance_metadata_cache_size_bytes` | 1 GiB |
| `enable_lance_data_cache` | true |
| `lance_data_cache_path` | `${DORIS_HOME}/lance_data_cache` |
| `lance_data_cache_disk_capacity_bytes` | 100 GiB |
| `lance_data_cache_read_block_size_bytes` | 1 MiB |
| `lance_io_buffer_size_bytes` (mutable) | 2 GiB |
| `lance_batch_readahead` (mutable) | 5 |
| `lance_fragment_readahead` (mutable) | 5 |
The BE-wide Lance session (index cache, metadata cache and the optional
Foyer disk cache) is created by the first Lance scan, so a BE that never reads
Lance creates no cache directory. The first six configs take effect after a
restart. Ten `lance_session_*` metrics report the two in-memory caches. With
the defaults, the disk tier keeps 51,200 sparse 2 MiB block files open from the
first Lance scan on (100 GiB in blocks of twice the read block size), the same
as on branch-4.1 / 4.2. Size `lance_data_cache_disk_capacity_bytes` and
`lance_data_cache_read_block_size_bytes` against the BE's open-file limit, or
set `enable_lance_data_cache = false`.
- Build: macOS links `-framework IOKit`. This is the first time objects of
`liblance_c.a` are pulled into `doris_be`, so the binary and the link time grow.
Not included: the `MaterializationOperator::push` timer fix (#69579); the
Lance index metadata scanner (`LANCE_INDEX_ENTRIES`); the FE side, including
the rolling-upgrade guard that keeps Lance lazy materialization off while an
old BE could decode a version 1 row location as version 0; the regression
suites; and the 4.x changes after `cabcf45c1af` (#68707, #68813, #68815). The
`InitReaderAvgTime` / `GetBlockAvgTime` profile fix that #67005 carried goes in
a separate PR.
**Results.**
| | master | with this PR |
|---|---|---|
| A `FORMAT_LANCE` query | fails before reading anything | read by
`LanceTableReader` in FileScannerV2 |
| A Lance row in the second phase of TopN lazy materialization | cannot be
represented | a version 1 row location, read back with `take_rows` |
| A row location of an unknown version or size in phase two | decoded as
version 0 | rejected with an error |
BE unit tests (ASAN): 238 cases. They cover the Lance suites (78:
`LanceTableReader*` 57, `LanceNestedNullTest` 10,
`LanceRuntimeFilterHelperTest` 8, `LanceSessionManagerTest` 3); the
row-location protocol (`MaterializationSharedStateTest`,
`RowIdStorageReaderTest`, and the new `RowIdFetchRequestValidationTest`: count
mismatch, unknown version, a Lance version on an internal mapping, an internal
row id of 2^32); `FileScannerV2Test` with the new
`LanceQueriesAlwaysUseScannerV2`, `FileScanOperatorAdbcTest` and
`FileScanOperatorFlussTest`; the ORC / Parquet global row id tests that now
read `.file_local`; and the Arrow / protobuf serde suites. 237 pass locally.
`LanceTableReaderVectorSearchTest.MultiVectorScoresFiltersOffsetsAndIndexedCoverage`
checks ANN timing metrics that the lance-c revision pinned by master reports;
the local third-party build predates that revision, so this case is left to CI.
### Release note
Add nine `lance_*` BE configs and `lance_session_*` BE metrics. They take
effect once the FE can plan Lance scans.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- `run-be-ut.sh --run --filter=...` with the suites listed under
Results: 237 of 238 pass on macOS arm64 (ASAN). The remaining case needs the
lance-c revision that master pins (see Results).
- `build-support/check-build-hygiene.sh`, clang-format 16 and
`build-support/run-clang-tidy.sh` on the changed files: all pass; clang-tidy
reports nothing on the changed lines of the 38 changed C++ files.
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- Behavior changed:
- [ ] No.
- [x] Yes. `PRequestBlockDesc.row_id` is `uint64`; phase two of TopN
lazy materialization rejects row locations of an unknown version or size;
`FORMAT_LANCE` scans run on FileScannerV2.
- Does this need documentation?
- [x] No. Lance is documented with the FE connector.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
