Hi Ed, Alkis, Raphael, and all, Ed, I just saw your reply come in while I was drafting this, and I completely agree! I would absolutely love to collaborate with you on merging our PoCs and drafting a formal proposal for the jump table. I also agree that the deeper PAR3/FlatBuffer architectural discussion is best continued separately.
Before we do, I wanted to address Alkis's points regarding name resolution and parsing speeds, just to get the data on the record: On name resolution: Alkis, you mentioned that O(1) name resolution doesn't work in general because of locale-dependent case folding. I think this is solvable: if the spec mandates Simple Case Folding (Unicode CaseFolding.txt, status S/C) for the hash table, every engine simply applies that same fold before probing, regardless of its native matching strategy. Collisions from folding are handled naturally -- the table returns candidate ordinals and the engine disambiguates with its own rules. The hash table is an acceleration structure, not the source of truth. You also noted that at 10K columns, name resolution doesn't show up in profiles. We benchmarked beyond 10K using the standard Java FlatBuffers API, and name lookup is actually the slowest FlatBuffer operation -- at wide column counts it is slower than iterating all column metadata, because it requires an O(N) linear scan of schema string comparisons. A zero-copy byte comparison implementation would improve the constant factor but not the complexity. A name hash table baked in makes it O(1). For field_id resolution, a linear scan over integers in FlatBuffers is fast enough that I wouldn't worry about it. For the jump table approach, I have a few compact alternatives to a hash table for field_id lookup depending on how dense the ID space is. On parsing speeds & format limitations: In my experience, many of the performance problems motivating the FlatBuffer footer are primarily library implementation issues rather than format limitations -- both in parsing overhead (heap allocation, object models) and in writer defaults that produce unnecessary statistics and bloat. FlatBuffers do win on full linear parsing, where zero-copy access avoids deserialization entirely. But how often does an engine need to parse the entire footer of a very wide file? Most queries touch a handful of columns. The jump table makes that case O(1) in Thrift, where the per-column decoding cost is manageable. For sequential parsing, we have a Java Thrift footer parser running at just under 4 cycles per touched byte -- multiple times faster than arrow-rs, an order of magnitude faster than Apache Thrift Java, and significantly faster than C++ Arrow -- on the same format, with no SIMD or Unsafe. A Rust port with unsafe is slightly faster still. The C++ Arrow bottleneck, for example, is heap allocation in the object model, not Thrift decoding. And work on tail-call-optimized interpreters for binary formats (as explored for protobuf -- see https://blog.reverberate.org/2021/04/21/musttail-efficient-interpreters.html) suggests there may be further headroom for Thrift compact protocol parsing as well. On the paths forward: Since there's broad support for the jump table as an immediate, backward-compatible optimization, that gives us a lot of breathing room. Regarding PAR3, I understand Ed's desire for a clean, modular rethink. My main hesitation, echoing Raphael, is the risk of getting stuck partially migrated. If we do go down the PAR3 route, I'd personally want it to ship with everything in the first version -- statistics, encryption, page indexes, and topics like fixed-size vectors that haven't been fully discussed yet -- to avoid the thrash of piecemeal adoption. But as Ed noted, we can hash out those architectural tradeoffs separately. Ed, I'll reach out so we can start syncing up on the index PoC! Cheers, Will On Wed, 1 Apr 2026 at 22:43, Ed Seidl <[email protected]> wrote: > > path_in_schema being optional. Making it optional doesn't avoid a > breaking > > change - it is a breaking change. Every existing reader uses > path_in_schema > > to reconstruct the column→schema mapping. The moment writers stop > emitting > > it, old readers break. Same ecosystem coordination cost as a new footer > > format. If we're paying that cost, we should get more than one fewer > field > > for it. > > Well, there are breaking changes and then there are breaking changes. ;-) > I think the thrash from removing a single field is much more manageable > than a second embedded version of the metadata. And I never said it wasn't > a breaking change, but it's no more so than adding a new encoding...at the > end of the day old readers won't be able to read either. > > > > > Moving forward. There is a tradeoff between complexity and scope. I see > > three paths: > > > > A. Ship the full FlatBuffer footer as proposed. Move forward with PR #544 > > as-is, logically compatible with the Thrift footer - schema, column chunk > > metadata, statistics, page indexes, encryption, all of it. One > transition, > > one spec. Risk: the scope keeps generating debate and we > > stay stuck. > > > > B. Ship a minimal FlatBuffer core, add modules later. Strip the > FlatBuffer > > footer to schema + column chunk placement (file offset, compressed size, > > uncompressed size) - the minimum a reader needs to plan I/O. Statistics, > > size statistics, page indexes, encryption become separate > > optional FlatBuffer modules that live before the footer and are > referenced > > by pointer from the core. Ratify the core now, add modules as independent > > work streams. This unblocks the part everyone agrees on and lets us > iterate > > on the contentious pieces without re-litigating the core. > > > > C. Improve statistics and page indexes within the current format. Hold > off > > on the FlatBuffer footer. Focus on smarter writer defaults, tooling like > > parquet-linter, and Will's jump table for O(1) access to existing files. > No > > format break, but we accept the structural limitations of Thrift. > > > > My preference is A or B, whichever lands faster. > > I think I'd prefer B. If we're going through the effort, rather than just > putting flatbuffer lipstick on the current metadata pig, I was hoping we'd > explore a complete rethink of the metadata with an eye to making it more > modular to support more experimentation. I like the idea of breaking it up > into typed sections, with file navigation info living in one place, schema > another, indexes yet another, things to be dreamt up etc. I think there's > sufficient interest and momentum to avoid the stagnation Raphael fears. > > Anyway, I think we've come to a consensus that it's worth pursuing the > skip index irrespective of the flatbuffer work. Perhaps Will and I can > collaborate and merge our PoCs and make a more formal proposal. > > I think the flatbuffer discussion should likely continue on the PR [1]. > > Thanks all for the lively discussion! > > Ed > > [1] https://github.com/apache/parquet-format/pull/544 >
