alamb opened a new pull request, #24373: URL: https://github.com/apache/datafusion/pull/24373
## Which issue does this PR close? - Part of testing https://github.com/apache/arrow-rs/pull/9372 ## Rationale for this change Test PR (not for merge) to try out the Parquet ALP encoder/decoder support proposed in https://github.com/apache/arrow-rs/pull/9372 from DataFusion, so ALP encoded parquet files can be written and read with `datafusion-cli`. Based on https://github.com/apache/datafusion/pull/24366 (update to arrow-rs main), with the `[patch.crates-io]` section repointed at the `alp` branch from https://github.com/sdf-jkl/arrow-rs. ## What changes are included in this PR? - Pin the arrow/parquet git patch to `sdf-jkl/arrow-rs` branch `alp` (currently commit `5e9ad1c6`) - Adjust `try_fb_to_schema` back to `fb_to_schema` (the `alp` branch is based on an arrow-rs main commit that predates that API) - Add `alp` as a recognized value for the parquet `encoding` writer option ## Are these changes tested? Manually, with `datafusion-cli`: ```sql > COPY (SELECT random() AS r, CAST(random() AS FLOAT) AS f FROM generate_series(1,10000)) TO '/tmp/alp_test.parquet' OPTIONS ('format.encoding' 'alp', 'format.dictionary_enabled' 'false'); > SELECT path_in_schema, type, encodings FROM parquet_metadata('/tmp/alp_test.parquet'); +----------------+--------+------------+ | path_in_schema | type | encodings | +----------------+--------+------------+ | "r" | DOUBLE | [RLE, ALP] | | "f" | FLOAT | [RLE, ALP] | +----------------+--------+------------+ > SELECT count(*), min(r) >= 0, max(r) <= 1, min(f) >= 0 FROM '/tmp/alp_test.parquet'; -- 10000, true, true, true ``` Only `datafusion-cli` is expected to compile/run; other tests are not expected to pass. ## Are there any user-facing changes? No (test PR, not intended for merge). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
