ajantha-bhat commented on PR #11216:
URL: https://github.com/apache/iceberg/pull/11216#issuecomment-2434651434
@aokolnychyi, @rdblue:
I already tried POC for internal writers on another branch,
https://github.com/apache/iceberg/commit/c209bc94c3a43e3fb8262cffa121dc450889dfd8
The problems:
a) I am using PartitionData instead of Record for partition value, but the
`PartitionData` get() method wraps the byte array which is a problem for
internal writers, they expect byte[]. So, I didn't felt like using a new class
instead of `PartitionData` just for this.
b) Also, Using partitionData in StructLikeMap is not working fine. Some keys
are missing in the map (looks like equals() logic), If I use Record, it is
fine.
Maybe in the next version we can have optimized writer and reader (without
converter using internal reader and writers).
For end user it doesn't make any difference as new readers can also read the
old partition stats parquet file and old readers can read the new partition
stats parquet file. So, can we merge this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
