> I also agree the statistics are a mess. But then, I think a bigger problem is overpopulation of the statistics. There is very little benefit to simple min/max statistics on unsorted columns. If writers were a little more conservative and simply omitted these optional statistics for columns that have no chance of benefiting from them that would reduce a great deal of bloat.
This is a great idea about how to take advantage of the (existing) metadata better. Something Xiangpeng Hao, Jigao Luo and I have been exploring is a parquet-linter[1] (still in the early phase) to help users find the best settings for their data when writing Parquet (without changing the format). This might be helpful to identify such sources of bloat for existing files Andrew [1]: https://github.com/XiangpengHao/parquet-linter On Mon, Mar 30, 2026 at 1:59 PM Ed Seidl <[email protected]> wrote: > Thanks for the perspective, Alkis. I'd just like to add a few comments. > > On 2026/03/27 13:37:46 Alkis Evlogimenos via dev wrote: > > 1. Dedup. The Thrift footer repeats path_in_schema (a list of strings) > for > > every column in every row group. For a 10K-column, 4-RG file that's 40K > > string lists and it's the single biggest source of footer bloat. The > > FlatBuffer footer drops it entirely — it's derivable from schema + column > > ordinal. Same for type (already in the schema), the full encodings list, > > and encoding_stats (replaced by a single bool). > > I agree path_in_schema is pretty useless, but we could just make that > field optional. Yes this would break old readers, but then so would adding > a new encoding or compression codec. Old readers can't be expected to work > forever. > > > 2. Compact stats. Thrift Statistics stores min/max as variable-length > > binary with per-field framing. The FlatBuffer footer uses fixed-width > > integers for numeric types and a prefix+truncated-suffix scheme for byte > > arrays. Across thousands of columns this adds up. > > I also agree the statistics are a mess. But then, I think a bigger problem > is overpopulation of the statistics. There is very little benefit to simple > min/max statistics on unsorted columns. If writers were a little more > conservative and simply omitted these optional statistics for columns that > have no chance of benefiting from them that would reduce a great deal of > bloat. > > > 3. Dropped dead weight. ConvertedType, deprecated min/max, > distinct_count, > > SizeStatistics > > I'll grant the first two, but already I've seen calls to do something with > distinct_count, and I personally use the size statistics, so I do not agree > to the "dead weight" label for those. I do agree that their current form is > not ideal, but was a compromise at the time. I think one benefit of the > flatbuffers work would be to separate out metadata needed for traversing > the file from metadata supporting indexes/other purposes. If we can easily > add new specialized structures that are easy to ignore I think that would > be a win. > > > A jump table into the existing Thrift footer preserves all of this > > duplication and bloat. You still have to decode the same fat > ColumnMetaData > > structs, you just get to skip to the right one faster. > > Given that most of the ColumnMetaData bloat is at the tail end of the > struct, the jump table allows for stopping parsing early and skipping to > the next column. No need to parse the bloat, but it is still there. > > > And the index itself > > adds at least 12 bytes plus framing per column per row group (you need > > offset+length since Thrift fields are variable-width), so the total > footer > > actually gets bigger. > > Not quite. Given that row groups and column chunks are serialized > back-to-back, one simply needs N+1 offsets, the lengths can then be > derived. Alternatively, if we use 0 offsets for the start of the row groups > and the first column chunk in a row group, you could instead just encode N > lengths and do an exclusive scan to deduce the offsets. This would allow > for using fewer bytes to encode the lengths at the expense of a little more > computation when instantiating the table. > > > Now, if we accept a breaking change is needed to meaningfully shrink the > > footer, then why not break into a format that also gives us zero-copy > > access natively? > > I do agree that if we are going to completely redo the metadata, then why > not change to flatbuffers, so long as we're good with the trade offs > (zero-copy and random access for larger representations). > > Cheers, > Ed > > >
