kinolaev commented on PR #17196:
URL: https://github.com/apache/iceberg/pull/17196#issuecomment-5443956554
I rebased the branch to resolve a conflict - `PositionAndRowDatumWriter` was
recently removed.
I started thinking about a proposal for v4. My personal preference is to get
rid of all inconsistencies between the specs at once:
1. `timestamp(_ns)` <-> `local-timestamp-*`
1. `timestamptz(_ns)` <-> `timestamp-*`
1. drop the `adjust-to-utc` property from v4
I realized that we don't actually need custom table properties. On the write
side, we can just use `format-version < 4 ? legacy iceberg timestamps : avro
timestamps`. On the read side, we need a property that tells us at which
`format-version` a manifest or a data file was written. Manifests already have
a required `format-version` metadata field that we can use to interpret the
`identity/void(timestamp(_ns))` partition field schema. And I think we could
add a `file_format_version` field to `manifest_entry`, which should be `null`
unless it differs from its manifest's `format-version` metadata field. Then, on
the read side, we could do the following:
```java
int fileFormatVersion = entry.fileFormatVersion() != null
? entry.fileFormatVersion()
: manifest.formatVersion() != null ? manifest.formatVersion() : 1;
if (fileFormatVersion < 4) {
// read `adjust-to-utc` with default `false`
} else {
// follow the Avro spec, ignore `adjust-to-utc`
}
```
As a bonus, it allows us to add external Avro data files with
`file_format_version=3` or `file_format_version=4` depending on their schema.
And of course it leaves room for further file format changes.
I'm still exploring alternatives. Just wanted to share the best idea I have
for now, in case you'd like to give early feedback - I'd really appreciate it!
:)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
