Re: [I] Support dynamic overwrite [iceberg-python]

2024-12-19 Thread via GitHub
sungwy closed issue #1287: Support dynamic overwrite URL: https://github.com/apache/iceberg-python/issues/1287 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e

Re: [I] Support dynamic overwrite [iceberg-python]

2024-12-08 Thread via GitHub
sundaresanr commented on issue #1287: URL: https://github.com/apache/iceberg-python/issues/1287#issuecomment-2526399105 It would be nice to support multiple partition overwrites in a single transaction => produce single snapshot -- This is an automated message from the Apache Git Service

Re: [I] Support dynamic overwrite [iceberg-python]

2024-11-07 Thread via GitHub
kevinjqliu commented on issue #1287: URL: https://github.com/apache/iceberg-python/issues/1287#issuecomment-2462831829 I see thanks for the explanation. When writing in fast append mode (DELETE+APPEND), it's possible to accidentally time travel to the DELETE snapshot and only see the delet

Re: [I] Support dynamic overwrite [iceberg-python]

2024-11-06 Thread via GitHub
koenvo commented on issue #1287: URL: https://github.com/apache/iceberg-python/issues/1287#issuecomment-2460958032 Ah good question. In our normal process the Iceberg tables are only queried using our own application. The application will always (for now at least) use the latest snapshot.

Re: [I] Support dynamic overwrite [iceberg-python]

2024-11-06 Thread via GitHub
kevinjqliu commented on issue #1287: URL: https://github.com/apache/iceberg-python/issues/1287#issuecomment-2460326291 yes, thats right. This will create 2 snapshots and if you time travel to the first one, you will only see the table with the data deleted. What if your use case here to

Re: [I] Support dynamic overwrite [iceberg-python]

2024-11-05 Thread via GitHub
koenvo commented on issue #1287: URL: https://github.com/apache/iceberg-python/issues/1287#issuecomment-2458808080 I also believe those two snapshots are added in a single transaction. What I mean is that it’s possible to time-travel to the delete snapshot. In that case you are looking at

Re: [I] Support dynamic overwrite [iceberg-python]

2024-11-05 Thread via GitHub
kevinjqliu commented on issue #1287: URL: https://github.com/apache/iceberg-python/issues/1287#issuecomment-2458722288 Thanks @Fokko didnt see that one, I'll close this issue when that PR is merged > It seems that the PR does a delete+append. If I understand correctly, this could le

Re: [I] Support dynamic overwrite [iceberg-python]

2024-11-04 Thread via GitHub
koenvo commented on issue #1287: URL: https://github.com/apache/iceberg-python/issues/1287#issuecomment-2455736587 Ah the PR contains quite some similar functionality indeed. It seems that the PR does a delete+append. If I understand correctly, this could lead to reading incomplete d

Re: [I] Support dynamic overwrite [iceberg-python]

2024-11-04 Thread via GitHub
Fokko commented on issue #1287: URL: https://github.com/apache/iceberg-python/issues/1287#issuecomment-2455407661 Good one @kevinjqliu, however I think this is a duplicate of https://github.com/apache/iceberg-python/pull/931. There is also a PR https://github.com/apache/iceberg-python/pull