Gabriel39 opened a new pull request, #70248: URL: https://github.com/apache/doris/pull/70248
### Background Doris has per-file Iceberg encryption parameters and a native Parquet reader, but that reader cannot consume encrypted footers or pages. Parquet modular encryption authenticates each module with file AAD and row-group, physical-column and data-page ordinals. ### What problem does this PR solve? Issue Number: #70245 Related PR: #70087 (PME dependency build), #70192 (Iceberg metadata reads), #70213 (file-key wire contract), #70246 (per-file key extraction) PR Type: Feature Problem Summary: Add the physical reader needed to read AES_GCM_V1 Parquet files with encrypted footers and footer-key columns. Authenticate and decrypt modules through parquet-cpp before passing bytes into the existing native metadata and value decoders. Java/Iceberg interoperability, bounded parsing and cache isolation require changes beyond the crypto wrapper. Encrypted SQL admission remains closed while index, delete-file and final scan integration are completed separately. ### Details - **Change map:** 867 production lines changed, 659 test/generator lines, 38 documentation lines and one 3.6 KB generated synthetic fixture. Production changes comprise a roughly 420-line module crypto/parsing wrapper, footer ownership, native page/column plumbing and sequential scan guards. Cached, plaintext and encrypted header parsing use separate helpers to keep their state and validation boundaries clear. Tests include an independently generated Java/Iceberg file and a native/reference reader comparison. - **Scope:** AES_GCM_V1 encrypted footers, encrypted page headers, dictionary pages and Data Page V1/V2; 128/192/256-bit footer keys; scalar and nested physical decoding. AES_GCM_CTR_V1, plaintext-footer encryption and separate column keys are explicitly unsupported. - **Design:** Keep physical decoding in format_v2. The crypto wrapper uses parquet-cpp AES and module-AAD helpers, retains an immutable SecureString file key, and creates mutable decryptors per physical column reader. Track ciphertext disk sizes separately from decrypted compressed sizes and count only data pages in AAD ordinals. Authenticate metadata before accepting Parquet Java's zero-filled serialization padding. Bound encrypted modules and decompressed pages to 64 MiB and impose Thrift container, recursion and aggregate allocation limits before decoding. Bypass shared decrypted metadata/page/condition caches and index/dictionary/bloom pruning for the sequential stage. - **Usage:** Controlled physical-reader callers can supply the existing TFileEncryptionInfo as an optional constructor parameter. No new configuration, Thrift field, capability negotiation or user SQL entry is added. Encrypted indexes and delete-file integration remain follow-up work. - **Default behavior:** Existing plaintext callers retain default parameters and their existing cache/pruning paths. FE encrypted-table admission and BE scan dispatch guards remain closed. - **Compatibility:** Reuses the existing descriptor. PME requires the internal headers and actual encryption implementation from the previously enabled third-party build; older prebuilt bundles compile and explicitly report PME_DEPENDENCY_UNAVAILABLE. No storage format or dependency version changes. - **Behavior change:** The native physical reader can decrypt the supported layout when explicitly given file parameters; encrypted files without keys and unsupported layouts fail explicitly. This PR does not enable encrypted SQL queries. ### Testing - Locally, 235 focused ASAN tests passed across nine Parquet/native-decoder, page-cache and existing file-encryption/scan-range guard suites, including 12 encryption tests. Used run-be-ut.sh with a local CMake test-source filter; production libraries were built normally. This is not the full BE unit-test suite. - Positive coverage: dictionary/plain encoding, Data Page V1/V2, three AES key sizes, multiple pages and row groups, null/empty nested shapes, levels-only reads and the physical ParquetReader block interface. The checked-in Iceberg 1.12.0 Java sample is compared field-by-field with the independent parquet-cpp reference reader. - Negative coverage: missing/invalid/wrong keys and AAD, footer/header/payload corruption, truncation, amplified Thrift containers, cached reads with another key, and avoiding encrypted index reads. Errors do not expose keys or AAD. The missing-key contract failed before implementation and passes afterward. - Locally rebuilt PME with the Doris toolchain for positive tests. With the original prebuilt dependency bundle, all four fallback/metadata contract tests passed and decryption was explicitly rejected. - clang-format 16, clang-tidy, build hygiene and diff whitespace checks passed. - Full BE CI and encrypted SQL end-to-end validation are not claimed; encrypted SQL is intentionally gated at this stage. ### Notes for Reviewers Start with parquet_crypto, then encrypted footer parsing and native page_reader. The direct format_common include fixes a pre-existing unity-neighbor include dependency exposed by adding a source file. The fixture README includes its generator and public synthetic parameters. ### Release note None. This prepares the native encrypted Parquet reader; encrypted SQL admission remains disabled. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: No user-facing enablement; fixture and stage limitations documented alongside the tests. - Pre-submission status: Partially verified: focused local ASAN suites passed; full BE CI is pending and encrypted SQL remains gated. ### AI Disclosure - Tool: Codex (GPT-6). - Generated: Code, tests, synthetic fixture generator and this description. - Human review: Pending maintainer review; agent self-review completed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
