pvary commented on PR #14153:
URL: https://github.com/apache/iceberg/pull/14153#issuecomment-3338236549
> > If our goal is to support all V3 features across all file formats
>
> what if variant shredding won't ever be supported by orc or avro?
That’s a fair question. When I proposed introducing new file formats like
Vortex, the first concern raised was that we must ensure feature parity across
all supported formats. Otherwise, users would have to navigate a complex matrix
of supported vs. unsupported features — which is a valid usability concern.
To me, that implies we either implement these features across all formats or
reconsider the long-term viability of formats that can’t support them.
> > You might want to check out the File Format API proposal, since it will
impact Hive integration as well.
>
> yes, I intended to, but never found the time. Now it seems I can’t
postpone it any longer.
Totally understandable — keeping up with multiple open proposals is no small
task. Sorry for adding to the pile!
> > the API has planned to expose an inputSchema(S type) method, allowing
engines to pass in the input schema — which implicitly defines the shredding
configuration.
>
> Is this API available, and can we use it right now and pass shredding
configuration?
Not yet. I’d like to finalize the design first. In the meantime, I’ve been
maintaining a PoC implementation to test the concepts — you can find it here:
#12298.
Note: variant handling hasn’t been added to it yet.
> > while we do need to expose this functionality, we might want to rethink
how we expose it.
>
> That’s totally fine, we’ve only just begun work on variant type support.
👍
Thanks for all the info, and understanding!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
