Re: [I] REPLACE TABLE Support [iceberg-python]

2024-09-22 Thread via GitHub
github-actions[bot] commented on issue #281: URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-2367035445 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity oc

Re: [I] REPLACE TABLE Support [iceberg-python]

2024-02-29 Thread via GitHub
Fokko commented on issue #281: URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-1971024124 @syun64 Yes, that sounds like a great suggestion to me. I think it makes a lot of sense to create a table including a snapshot. I'm not sure if `as_select` is the best name since

Re: [I] REPLACE TABLE Support [iceberg-python]

2024-02-22 Thread via GitHub
syun64 commented on issue #281: URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-1959949307 There's a PR in progress that will introduce 'REPLACE TABLE' support, but I don't think we've come to a consensus yet on how we would want to support 'AS SELECT' semantics in PyI

Re: [I] REPLACE TABLE Support [iceberg-python]

2024-01-22 Thread via GitHub
syun64 commented on issue #281: URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-1905187291 Hi @Fokko - sounds like you beat me to it 😄 Please let me know if you need any additional heavy lifting on #284 . Happy to help as always. The reason I was curious if there

Re: [I] REPLACE TABLE Support [iceberg-python]

2024-01-22 Thread via GitHub
Fokko commented on issue #281: URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-1904323323 @syun64 I started on #284 today. It re-uses the `UpdateSchema` API which already sets the right IDs, and maintains IDs for the existing fields. I don't think that doesn't help us

Re: [I] REPLACE TABLE Support [iceberg-python]

2024-01-22 Thread via GitHub
syun64 commented on issue #281: URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-1904310699 In order to reduce duplication of code, would it make sense to combine the job of [TypeUtil.assignFreshIds](https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apac

Re: [I] REPLACE TABLE Support [iceberg-python]

2024-01-19 Thread via GitHub
syun64 commented on issue #281: URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-1901289940 Makes sense @Fokko . Thank you very much for taking the time to lay all these options where a user may have to handle compatible or incompatible schema updates. As you sug

Re: [I] REPLACE TABLE Support [iceberg-python]

2024-01-19 Thread via GitHub
syun64 commented on issue #281: URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-1901252873 Makes sense @Fokko . Thank you very much for taking the time to lay all these options out for the cases where a user may have to handle schema updates. As you suggested, I

Re: [I] REPLACE TABLE Support [iceberg-python]

2024-01-19 Thread via GitHub
Fokko commented on issue #281: URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-1901165986 > Evolving schema would be nice, also would it be possible to evolve partitions? e.g. after a specific overwrite I want to have schema evolution and possibility to replace the tab

Re: [I] REPLACE TABLE Support [iceberg-python]

2024-01-19 Thread via GitHub
syun64 commented on issue #281: URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-1900741649 Thank you for the great points @Fokko and @nicor88 . Just like @nicor88 mentioned, I think RTAS will be slightly different from overwrite in the sense that the schema, the

Re: [I] REPLACE TABLE Support [iceberg-python]

2024-01-19 Thread via GitHub
nicor88 commented on issue #281: URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-1900185062 > The next step would be to automatically evolve the schema as well (with flags for compatible/incompatible changes). @Fokko evolving schema would be nice, also would it be po

Re: [I] REPLACE TABLE Support [iceberg-python]

2024-01-19 Thread via GitHub
Fokko commented on issue #281: URL: https://github.com/apache/iceberg-python/issues/281#issuecomment-1900076148 @syun64 thanks for raising this. With the freshly merged write support, we can do: ```python cat = load_catalog('default') tbl = cat.load_table('default.some_table')