szehon-ho commented on code in PR #10981: URL: https://github.com/apache/iceberg/pull/10981#discussion_r1723950848
########## format/spec.md: ########## @@ -373,6 +378,13 @@ Notes: 3. Strings are truncated to a valid UTF-8 string with no more than `L` code points. 4. In contrast to strings, binary values do not have an assumed encoding and are truncated to `L` bytes. +#### XZ2 Transform Details + +XZ2 is based on the paper [XZ-Ordering: A Space-Filling Curve for Objects with Spatial Extensions]. + +Notes: +1. Resolution must be a positive integer. Defaults to TODO Review Comment: @jiayuasu do you have any suggestion for default here? ########## format/spec.md: ########## @@ -985,59 +997,66 @@ Values should be stored in Parquet using the types and logical type annotations Lists must use the [3-level representation](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#lists). -| Type | Parquet physical type | Logical type | Notes | -|--------------------|--------------------------------------------------------------------|---------------------------------------------|----------------------------------------------------------------| -| **`boolean`** | `boolean` | | | -| **`int`** | `int` | | | -| **`long`** | `long` | | | -| **`float`** | `float` | | | -| **`double`** | `double` | | | -| **`decimal(P,S)`** | `P <= 9`: `int32`,<br />`P <= 18`: `int64`,<br />`fixed` otherwise | `DECIMAL(P,S)` | Fixed must use the minimum number of bytes that can store `P`. | -| **`date`** | `int32` | `DATE` | Stores days from 1970-01-01. | -| **`time`** | `int64` | `TIME_MICROS` with `adjustToUtc=false` | Stores microseconds from midnight. | -| **`timestamp`** | `int64` | `TIMESTAMP_MICROS` with `adjustToUtc=false` | Stores microseconds from 1970-01-01 00:00:00.000000. | -| **`timestamptz`** | `int64` | `TIMESTAMP_MICROS` with `adjustToUtc=true` | Stores microseconds from 1970-01-01 00:00:00.000000 UTC. | -| **`timestamp_ns`** | `int64` | `TIMESTAMP_NANOS` with `adjustToUtc=false` | Stores nanoseconds from 1970-01-01 00:00:00.000000000. | -| **`timestamptz_ns`** | `int64` | `TIMESTAMP_NANOS` with `adjustToUtc=true` | Stores nanoseconds from 1970-01-01 00:00:00.000000000 UTC. | -| **`string`** | `binary` | `UTF8` | Encoding must be UTF-8. | -| **`uuid`** | `fixed_len_byte_array[16]` | `UUID` | | -| **`fixed(L)`** | `fixed_len_byte_array[L]` | | | -| **`binary`** | `binary` | | | -| **`struct`** | `group` | | | -| **`list`** | `3-level list` | `LIST` | See Parquet docs for 3-level representation. | -| **`map`** | `3-level map` | `MAP` | See Parquet docs for 3-level representation. | +| Type | Parquet physical type | Logical type | Notes | +|--------------------|--------------------------------------------------------------------|---------------------------------------------|---------------------------------------------------------------------------------------------------------| +| **`boolean`** | `boolean` | | | +| **`int`** | `int` | | | +| **`long`** | `long` | | | +| **`float`** | `float` | | | +| **`double`** | `double` | | | +| **`decimal(P,S)`** | `P <= 9`: `int32`,<br />`P <= 18`: `int64`,<br />`fixed` otherwise | `DECIMAL(P,S)` | Fixed must use the minimum number of bytes that can store `P`. | +| **`date`** | `int32` | `DATE` | Stores days from 1970-01-01. | +| **`time`** | `int64` | `TIME_MICROS` with `adjustToUtc=false` | Stores microseconds from midnight. | +| **`timestamp`** | `int64` | `TIMESTAMP_MICROS` with `adjustToUtc=false` | Stores microseconds from 1970-01-01 00:00:00.000000. | +| **`timestamptz`** | `int64` | `TIMESTAMP_MICROS` with `adjustToUtc=true` | Stores microseconds from 1970-01-01 00:00:00.000000 UTC. | +| **`timestamp_ns`** | `int64` | `TIMESTAMP_NANOS` with `adjustToUtc=false` | Stores nanoseconds from 1970-01-01 00:00:00.000000000. | +| **`timestamptz_ns`** | `int64` | `TIMESTAMP_NANOS` with `adjustToUtc=true` | Stores nanoseconds from 1970-01-01 00:00:00.000000000 UTC. | +| **`string`** | `binary` | `UTF8` | Encoding must be UTF-8. | +| **`uuid`** | `fixed_len_byte_array[16]` | `UUID` | | +| **`fixed(L)`** | `fixed_len_byte_array[L]` | | | +| **`binary`** | `binary` | | | +| **`struct`** | `group` | | | +| **`list`** | `3-level list` | `LIST` | See Parquet docs for 3-level representation. | +| **`map`** | `3-level map` | `MAP` | See Parquet docs for 3-level representation. | +| **`geometry`** | `binary` | `GEOMETRY` | WKB format, see Appendix G. Logical type annotation optional for supported Parquet format versions [1]. | Review Comment: I could add this section later too, once its implemented (same for ORC below) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
