jqin61 opened a new issue, #398:
URL: https://github.com/apache/iceberg-python/issues/398

   ### Question
   
   Hi I was trying to understand whether there is any special design initiative 
for:
   1. The Python representation of partition of datetime-related type is 
integer. For example:
   ```
   
iceberg_table.current_snapshot().manifests(iceberg_table.io)[0].fetch_manifest_entry(iceberg_table.io)[0].data_file.partition
   ``` 
   would show DataFile instance having partition as 
   ```
   Record[SNAPSHOT_DATE=18628]
   ```
   
   Since DataFile class is a Python object, would it make more sense to 
represent it using native Python date/datetime?
   
   2. As a result, conversion between date/timestamp(tz) and integer stays 
outside of avro reader/writer. For example, we have the source code of :
   ```
   @dataclass(frozen=True)
   class DateWriter(Writer):
       def write(self, encoder: BinaryEncoder, val: int) -> None:
           encoder.write_int(val)
   ```
   which can only take int as input.
   
   
   I asked because I was trying to implement partition write support and during 
writing the manifest list, I instantiated the DataFile using partition = 
Record(field=raw date-typed Python object) which breaks the avro writer for 
writing the manifest file. So I wonder whether I should 1. do the conversion 
from date -> into before instantiating the DataFile's partition field or maybe 
2.change the avro DateWriter to handle the date-integer conversion (and do this 
for avro reader as well).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org

Reply via email to