Hi All, Following a lively discussion on this list, I thought I’d take a stab at addressing one pain point in the Parquet footer. I’ve put up a proposal [1] and PR [2] to switch path_in_schema in the ColumnMetaData from “required” to “optional”. I’ve also whipped up PoCs in Rust [3] and Java [4].
Please take a look and let’s discuss in the PR. Thanks, Ed [1] https://github.com/apache/parquet-format/issues/563 [2] https://github.com/apache/parquet-format/pull/564 [3] https://github.com/apache/arrow-rs/pull/9678 [4] https://github.com/apache/parquet-java/pull/3470
