jqin61 commented on issue #208: URL: https://github.com/apache/iceberg-python/issues/208#issuecomment-1885365903
> In Iceberg it can be that some files are still on an older partitioning, we should make sure that we handle those correctly based on the that we provide. It seems Spark's iceberg support has such overwrite behaviors under schema evolution: - dynamic overwrite: data files generated from old partition spec will not be replaced even if some of the records match the overwriting data - static overwrite with PARTITION values specified: same as above - static overwrite without PARTITION values: all data is deleted regardless of what partition specs they conform to. As Fokko mentioned, we need to make sure in the implementation we use the latest partition spec_id when overwriting partitions so that the data in the old partition spec is not touched. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
