punkeel opened a new issue, #882: URL: https://github.com/apache/arrow-go/issues/882
### Describe the bug, including details regarding any error messages, version, and platform. We hit this in production with Parquet files written by a Go writer and read by Rust/DataFusion. The original bad Parquet file is large, contains PII, and was produced from internal business data, so it cannot be shared. This report uses a small synthetic reproducer instead. Repro repository: [https://github.com/punkeel/parquet-repro-1](https://github.com/punkeel/parquet-repro-1) The repro writes a Parquet file using Arrow Go's public `pqarrow` writer API: ```go pqarrow.NewFileWriter(...) writer.WriteBuffered(record) ``` The generated schema is intentionally small: ```text c1: required list<optional string> c2: required list<required struct<key: string, value: string>> ``` `c1` is the important column. Because its list element is nullable, `pqarrow` uses the `ByteArrayColumnChunkWriter.WriteBatchSpaced` path. With DataPageV2 enabled, Arrow Go emits pages for `c1` that start with repetition level `1`, i.e. inside a row/list rather than at a row boundary. The generator and inspector are intentionally split. The generator only writes the file; the inspector reads the generated Parquet pages and prints the row-boundary violation. Inspector output from the repro: ```text inspecting repro.parquet column 0: c1.list.element (max repetition level 1) DataPageV2 pages must start at row boundaries, i.e. first_rep must be 0. OK c1 page 01: num_rows=1 num_values=1 first_rep=0 reps=[0] BAD c1 page 02: num_rows=0 num_values=1 first_rep=1 reps=[1] BAD c1 page 03: num_rows=0 num_values=1 first_rep=1 reps=[1] BAD c1 page 04: num_rows=0 num_values=1 first_rep=1 reps=[1] BAD c1 page 05: num_rows=0 num_values=1 first_rep=1 reps=[1] BAD c1 page 06: num_rows=0 num_values=1 first_rep=1 reps=[1] BAD c1 page 07: num_rows=0 num_values=1 first_rep=1 reps=[1] BAD c1 page 08: num_rows=1 num_values=2 first_rep=1 reps=[1 0] FOUND: this file contains DataPageV2 pages for repeated c1 that start with rep_level != 0. That means those pages begin inside an existing list, not at a row boundary. ``` This appears to violate the Parquet DataPageV2 row-boundary rule. The Apache Parquet format spec says, in [`src/main/thrift/parquet.thrift`](https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift): > NumRows: Number of rows in this data page. Every page must begin at a row boundary (repetition_level = 0): rows must **not** be split across page boundaries when using V2 data pages. The same file says for the older data page header: > If a OffsetIndex is present, a page must begin at a row boundary (repetition_level = 0). Otherwise, pages may begin within a row (repetition_level > 0). So page starts with `repetition_level > 0` are allowed for V1 pages without an offset index, but not for DataPageV2. DuckDB can still read the generated file, which is useful for compatibility context: ```text ┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬──────────────────────────────────────────┐ │ c1 │ c2 │ │ varchar[] │ struct("key" varchar, "value" varchar)[] │ ├──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────┤ │ [xxxxxxxxxxxxxxxx, xxxxxxxxxxxxxxxx, xxxxxxxxxxxxxxxx, xxxxxxxxxxxxxxxx, xxxxxxxxxxxxxxxx, xxxxxxxxxxxxxxxx, xxxxxxxxxxxxxxxx, NULL] │ [] │ │ [tail] │ [] │ └──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴──────────────────────────────────────────┘ ``` However, Rust `parquet = 59.0.0` fails on it: ```text FAIL SELECT c1 error=read projection ["c1"]: Parquet argument error: Parquet error: first repetition level of batch must be 0 OK SELECT c2 rows=2 FAIL SELECT c1, c2 error=read projection ["c1", "c2"]: Parquet argument error: Parquet error: Not all children array length are the same! Error: one or more projections failed ``` The `c1` reader-side error seems correct for a non-compliant DataPageV2 file. The combined projection failure is the older Arrow Rust symptom that led to [apache/arrow-rs#10243](https://github.com/apache/arrow-rs/issues/10243). This report is about Arrow Go emitting the non-compliant file in the first place. # To Reproduce ```sh git clone https://github.com/punkeel/parquet-repro-1 cd parquet-repro-1 go run . -out repro.parquet go run ./cmd/inspect -in repro.parquet ``` The first command writes `repro.parquet`. The second command reads the generated file and inspects the first column's DataPageV2 repetition levels. It should print the page trace shown above and end with: ```text FOUND: this file contains DataPageV2 pages for repeated c1 that start with rep_level != 0. ``` Optional DuckDB check: ```sh nix-shell -p duckdb --run \ "duckdb -c \"SELECT * FROM './repro.parquet';\"" ``` Optional Rust reader check: ```sh cd rust-read cargo run --release --locked -- ../repro.parquet ``` # Current behavior Arrow Go `pqarrow` writes a DataPageV2 for a repeated column where later pages begin with repetition level `1`. The checked-in `repro.parquet` has: ```text size: 1624 bytes sha256: f645f175885c4ac0060aa9d4f866a7c6e6fd58c66b0f0543eaa1b448b3a50c5a ``` # Expected behavior When writing DataPageV2, Arrow Go should not emit pages that begin inside a row/list. Concretely, for repeated columns written as DataPageV2, each page should begin at `repetition_level = 0`. If the writer cannot satisfy that for a given write path, it should either: - align the page/batch split to a row boundary, or - return an error rather than writing a non-compliant file. # Version information Repro generator: ```text Go tool used locally: go1.26.4 darwin/arm64 Go module directive: go 1.25.5 github.com/apache/arrow-go/v18 v18.6.0 ``` The production writer that first exposed this used: ```text github.com/apache/arrow-go/v18 v18.6.0 github.com/apache/iceberg-go v0.6.0 ``` The repro omits Iceberg catalog/table code because the invalid bytes are emitted by the Arrow Go Parquet writer after the Arrow record/schema already exists. Rust context: ```text rustc 1.96.0 (ac68faa20 2026-05-25) cargo 1.96.0 (30a34c682 2026-05-25) parquet = 59.0.0, features = ["arrow"] ``` # Related Arrow Rust issue This investigation originally surfaced as an Arrow Rust reader failure: [https://github.com/apache/arrow-rs/issues/10243](https://github.com/apache/arrow-rs/issues/10243) That issue is expected to be closed or treated as reader-side clarification because the file is non-compliant. The remaining bug is on the writer side: Arrow Go should not produce DataPageV2 pages that start with `repetition_level > 0`. # Suspected source The likely area is the Arrow Go byte-array column writer's spaced path. In `github.com/apache/arrow-go/v18 v18.6.0`, `ByteArrayColumnChunkWriter.WriteBatch` has DataPageV2 row-boundary alignment for repeated columns before committing each batch/page. The analogous `ByteArrayColumnChunkWriter.WriteBatchSpaced` path appears not to apply the same alignment. `pqarrow` uses `WriteBatchSpaced` for nullable byte-array/string leaves, which matches `c1: list<optional string>` and the real production column shape that exposed this. ### Component(s) Parquet -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
