hubgeter opened a new pull request, #68593: URL: https://github.com/apache/doris/pull/68593
### What problem does this PR solve? Related PR: https://github.com/apache/doris-thirdparty/pull/417 Problem Summary: Parquet GZIP pages and ORC ZLIB streams are decompressed with libdeflate only on x86 (#27542, #27669). On AArch64 both fall back to zlib `inflate`, which is several times slower. The x86-only restriction was never needed: - libdeflate is built for every platform by `build-thirdparty.sh`. - The official `doris-thirdparty-prebuilt-linux-aarch64` package ships an aarch64 `libdeflate.a`, reachable through the usual `installed/lib -> lib64` link. This PR: - links `deflate` unconditionally in `be/cmake/thirdparty.cmake`; - removes the architecture guards in `be/src/util/block_compression.cpp`, so Parquet `GZIP` always uses `GzipBlockCompressionByLibdeflate`; - will bump `contrib/apache-orc` once apache/doris-thirdparty#417 is merged. That PR does the same for ORC `CompressionKind_ZLIB` (`ZlibDecompressionStreamByLibDeflate`). x86 behavior is unchanged. StarRocks made the same change in StarRocks/starrocks#78457 and reports 583 MB/s → 2678 MB/s (4.57x) decompression throughput on Neoverse-N1. ### Release note Use libdeflate to decompress Parquet GZIP and ORC ZLIB data on ARM64. ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> New `BlockCompressionTest.parquet_gzip`: - compares the Parquet GZIP codec with the zlib gzip codec for blocks from 1 byte to 8 MB; - checks empty input, an undersized output buffer, truncated streams, corrupted gzip header / deflate / CRC32 bytes, and non-gzip input. New ORC `Compression.zlib_malformed_chunks`. `BlockCompressionTest.*`, `TestCompression.*` and `Compression.*` all pass (32 tests), and so do the ORC and Parquet reader unit tests (1037 tests) under ASAN. - Behavior changed: - [x] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [x] No. - [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
