duoxoud commented on PR #12664: URL: https://github.com/apache/iceberg/pull/12664#issuecomment-2769595436
> Displaying non-current columns is intentional in Glue, as users may use LakeFormation and need to access dropped columns. Users should not rely on Glue for the latest table status, Iceberg metadata should always be considered the source of truth. Hello Xiaoyuan 👋, Thank you for your thoughtful feedback on this PR 😃. I completely understand your point about displaying non-current columns and there are legit use cases. However, as discussed in [issue #7584](https://github.com/apache/iceberg/issues/7584), there are important use cases where the current behavior creates challenges: - When integrating with Athena, Redshift Spectrum, or third-party data catalogs that don't support this behavior - In my specific case, we're using CDC logs to reconcile Postgres tables where we often don't control schema changes, and our end users rely exclusively on the latest Glue table status To be clear, this PR doesn't aim to change the default behavior, but rather to add a new configuration option that would allow users to choose whether non-current columns are displayed. This provides flexibility for both use cases: - Users who need access to dropped columns can maintain the current behavior - Users who need Glue to reflect only the current schema can opt into that alternative Would you be open to considering this approach since it preserves the existing functionality while adding an option for users with different requirements? Thanks again for your review 🙇 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
