Re: [I] [feat] `add_files` support parquet files with field ids [iceberg-python]

2024-10-15 Thread via GitHub
kevinjqliu commented on issue #1227: URL: https://github.com/apache/iceberg-python/issues/1227#issuecomment-241482 @MrDerecho would you like to contribute this feature? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [I] [feat] `add_files` support parquet files with field ids [iceberg-python]

2024-10-15 Thread via GitHub
kevinjqliu commented on issue #1227: URL: https://github.com/apache/iceberg-python/issues/1227#issuecomment-2414821679 I think we can relax the constraints for `add_files` to allow field ids that are aligned, such as one written by an external engine like Trino. A use case I can thin

Re: [I] [feat] `add_files` support parquet files with field ids [iceberg-python]

2024-10-15 Thread via GitHub
MrDerecho commented on issue #1227: URL: https://github.com/apache/iceberg-python/issues/1227#issuecomment-2413900767 For context: right now I manage a very large data lake of time partitioned data. The use case has to do with the archival process put into place wherein after a rollin

Re: [I] [feat] `add_files` support parquet files with field ids [iceberg-python]

2024-10-15 Thread via GitHub
sungwy commented on issue #1227: URL: https://github.com/apache/iceberg-python/issues/1227#issuecomment-2413868478 Thanks for raising this @MrDerecho - in the initial version of add_files, we wanted to limit it to just parquet files that that were created in an external system. The assumpt

Re: [I] [feat] `add_files` support parquet files with field ids [iceberg-python]

2024-10-14 Thread via GitHub
kevinjqliu commented on issue #1227: URL: https://github.com/apache/iceberg-python/issues/1227#issuecomment-2411717630 Ty, updated the title. LMK if this is something you would like to work on 😄 -- This is an automated message from the Apache Git Service. To respond to the message, pleas