joechenrh opened a new issue, #865: URL: https://github.com/apache/arrow-go/issues/865
### Describe the enhancement requested We have a valid Parquet file with a very large data page. The page itself is not malformed or illegal; it is just large enough that materializing the whole uncompressed page body is expensive. One observed page has metadata similar to: ```text type: DATA_PAGE compressedSize: 1079936746 uncompressedSize: 1426870008 numValues: 233 encoding: PLAIN definitionLevelEncoding: RLE repetitionLevelEncoding: BIT_PACKED hasCrc: true ``` The column is a string-like BYTE_ARRAY column. The page contains a small number of very large values, so the uncompressed page body is large even though `numValues` is small. Today the Parquet reader materializes the full uncompressed page buffer before decoding values. Even with buffered stream reading enabled for the column chunk, this still requires allocating memory proportional to the page's uncompressed size. For the example above, that means allocating around 1.4 GiB for a single page before any batch-level output memory is considered. It would be useful for the reader to support a streaming/lazy data page read path for large pages. For Data Page V1, the reader could parse repetition and definition levels first into small buffers, then leave the decompressed stream positioned at the encoded values and decode PLAIN BYTE_ARRAY values on demand. This avoids allocating the entire uncompressed page body while preserving the existing page layout semantics. A possible first step would be an opt-in reader property or strategy that enables streaming value decoding for supported page shapes, for example: - Data Page V1 - PLAIN encoding - BYTE_ARRAY / string-like columns - RLE definition levels - RLE or BIT_PACKED repetition levels - codecs that can be exposed as an `io.Reader` stream Unsupported encodings/codecs/page layouts could continue to fall back to the existing materialized page path. The goal is not to reject or special-case large pages as invalid, but to allow applications to read valid Parquet files with very large pages while keeping peak reader memory closer to the requested batch size plus small level buffers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
