Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24059 )
Change subject: IMPALA-14700: Add read support for Parquet LZ4_RAW compression ...................................................................... IMPALA-14700: Add read support for Parquet LZ4_RAW compression This change requires a Parquet Version higher than the current one (1.12.3), because of the LZ4_RAW Thrift enum value. For that reason, APACHE_PARQUET_VERSION is increased to 1.15.2 in this patch, and is used instead of CDP_PARQUET_VERSION, until CDP_PARQUET_VERSION gets to a high enough version. Parquet deprecated LZ4 compression, and added a new one, LZ4_RAW. This patch adds read support for LZ4_RAW. It uses Lz4Compressor (Corresponding to THdfsCompression::LZ4). The write path hasn't changed and continues to use LZ4_BLOCKED. Testing: -Added a small test file using lz4_raw compression, from the parquet-testing repository. -Added a test case to test_scanners.py to check we can read the file. Change-Id: I22ee4e5bf9abec37be941c1dca8019a563343d34 Reviewed-on: http://gerrit.cloudera.org:8080/24059 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/exec/parquet/parquet-common.cc M be/src/exec/parquet/parquet-metadata-utils.cc M be/src/util/codec.cc M be/src/util/codec.h M bin/impala-config.sh M common/thrift/parquet.thrift A testdata/data/lz4_raw_compressed.parquet A testdata/workloads/functional-query/queries/QueryTest/parquet-lz4-raw-compression.test M tests/query_test/test_scanners.py 9 files changed, 31 insertions(+), 6 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/24059 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I22ee4e5bf9abec37be941c1dca8019a563343d34 Gerrit-Change-Number: 24059 Gerrit-PatchSet: 8 Gerrit-Owner: Balazs Hevele <[email protected]> Gerrit-Reviewer: Balazs Hevele <[email protected]> Gerrit-Reviewer: Daniel Vanko <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Surya Hebbar <[email protected]>
