suxiaogang223 opened a new pull request, #70401: URL: https://github.com/apache/doris/pull/70401
### Background Paimon stores deleted row positions as bitmap entries in index files. During a native scan, Doris reads the entry for each split and applies its bitmap as a row-position delete predicate. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Bugfix Problem Summary: Native Doris scans fail on Paimon bitmap64 deletion vectors. Doris treats every descriptor length as bitmap32 and adds four bytes, so a bitmap64 entry ending at EOF fails the file-range check. This change identifies the format from each entry header and applies the corresponding bitmap, including when an index contains both versions. ### Details - **Frequency:** A native scan fails whenever it must apply a bitmap64 entry. - **Root cause:** Bitmap32 descriptor lengths count the magic and bitmap. Bitmap64 descriptor lengths also count the prefix and checksum. The previous reader assumed bitmap32 length semantics and decoded only Roaring32. - **Why existing tests missed it:** Native reader coverage used bitmap32 fixtures. - **Fix:** Validate the descriptor and physical range using the entry header, then use the shared Roaring64 and CRC32 decoder for bitmap64. Keep the decoded bitmap in the query-local cache after successful validation. - **Other paths and branches:** This change covers the native FileScannerV2 path on master. The JNI reader serves as the reference in the Fedora smoke test. - **Behavior change:** Native Paimon scans can apply bitmap64 deletion vectors. The first read of a DV on a decoded-cache miss reads the header and bitmap separately. ### Testing - Fedora x86_64 ASAN BE build completed. - Fedora SQL smoke test with Hive Metastore and Paimon 2.1.0. Both bitmap32 and bitmap64 scans selected the native reader and returned `(1,10)` and `(3,30)`, matching the JNI reader. The original bitmap64 query failed before the fix. - Fedora `run-be-ut.sh --run --filter='PaimonReaderTest.*:IcebergDeleteFileReaderHelperTest.*'` with ASAN_UT passed all 40 tests (19 Paimon, 21 Iceberg). The local build scoped the test target to these suites and their test support sources. - Clang-format 16, build hygiene, and changed production lines checked with clang-tidy. ### Release note Native Paimon scans support 64-bit deletion vectors, including snapshots that also reference 32-bit vectors. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: Codex (GPT-6) - Generated: Implementation, BE unit tests, and this description - Human review: Pending -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
