pzhdfy opened a new pull request, #66227:
URL: https://github.com/apache/doris/pull/66227
### What problem does this PR solve?
Related PR: #65883
Problem Summary:
Adds a third BE-side reader for Paimon FORMAT_JNI splits, backed by the
`paimon-rust` C bindings (`libpaimon_c.a`, statically linked). Rust Reader
is gated by the session variable
`enable_paimon_rust_reader` (default `false`).
Compared with the existing readers:
- **vs paimon-jni**: no JVM round-trip on the hot path — the split is
deserialized in-process and rows arrive via the Arrow C Data Interface,
which shortens the scanner critical section and removes JNI copy cost.
- **vs paimon-cpp**: reuses paimon's official Rust implementation.
paimon-rust tracks upstream paimon spec directly
(schema, DV, snapshots, predicates), so features/fixes propagate may be
more fast and different.
Our goal is not to replace paimon-cpp, just give user a choice to switch at
query time.
**Architecture**
FE (`PaimonScanNode`): when `enable_paimon_rust_reader` is on and the split
is a `DataSplit`, encode it with paimon's native binary serialization
(`PaimonUtil.encodeDataSplitToString`), stamp the range with
`TPaimonReaderType.PAIMON_RUST`, and attach `db_name` / `table_name` /
`paimon_table` so BE can open the table through a catalog.
BE (`FileScanner::_get_next_reader`): dispatches on `reader_type`. On
`PAIMON_RUST` it instantiates `PaimonRustReader`, which:
1. `paimon_catalog_create({warehouse, metastore=filesystem, ...})`
- warehouse is derived from the table root path by stripping
`/<db>.db/<table>` (paimon FS convention).
2. `paimon_identifier_new(db, tbl)` → `paimon_catalog_get_table`
3. `paimon_table_new_read_builder` → projection (case-insensitive) →
optional filter from `PaimonRustPredicateConverter` (translates Doris
push-down conjuncts into paimon-rust predicates, best-effort so any
non-representable conjunct is silently dropped and re-applied by the
engine).
4. `paimon_plan_from_split_bytes(split_bytes)` — one BE scanner reads
exactly its FE-planned split (no re-planning).
5. `paimon_table_read_to_arrow` → streamed arrow record batches → block.
Partition columns are back-filled in `on_after_read_block` from
FE-provided `columns_from_path*`, mirroring `PaimonCppReader`. OSS/S3
hadoop-style credential keys (`fs.oss.*` / `fs.s3a.*`) are remapped to
`AWS_ACCESS_KEY` / `AWS_ENDPOINT` / ... so paimon-rust's FileIO can
authenticate.
**Thirdparty**
- Introduces `paimon-rust-2fb5e49` (built with Rust 1.91.0, feature
`paimon/storage-hdfs`, `--locked`).
- Auto-installs `cbindgen` and regenerates the C header from the Rust
extern "C" surface via a locally-written `cbindgen.toml`
(`language=C, include_guard=PAIMON_C_H, pragma_once=true,
cpp_compat=true`),
so the shipped `paimon.h` always matches the built `.a`.
### Release note
Add `enable_paimon_rust_reader` session variable. When enabled, Paimon
non-native reads on BE use the paimon-rust reader instead of paimon-jni /
paimon-cpp.
### Check List (For Author)
- Test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
1. `CREATE CATALOG paimon_fs PROPERTIES ("type"="paimon",
"paimon.catalog.type"="filesystem",
"warehouse"="file:///media/ssd1/test_paimon/warehouse/");` when using
"warehouse"="hdfs://.../warehouse/", add HADOOP_CONF_DIR=xx in be.conf
2. `SET enable_paimon_rust_reader=true; SET force_jni_scanner=1;set
enable_file_scanner_v2=0;`
3. `SELECT` from a partitioned/DV paimon table; results must match
`enable_paimon_rust_reader=false and force_jni_scanner=0 and
enable_file_scanner_v2=1`.
4. `SELECT count(*)` — should hit table-level row count push-down.
5. `WHERE a=1 AND b>2` — BE INFO log should show `paimon-rust
predicate pushdown: applied`.
- [ ] Regression test
- [ ] No need to test or manual test.
- Behavior changed:
- [x] No. Guarded by `enable_paimon_rust_reader` (default `false`);
when off, all paths are unchanged.
- Does this need documentation?
- [ ] No.
- [x] Yes. <!-- Add doris-website PR link here for the new session
variable -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
