pan3793 commented on PR #7733: URL: https://github.com/apache/iceberg/pull/7733#issuecomment-1567857761
> I was thinking maybe we could inject a new `ResolutionRule` in IcebergSparkSessionExtensions that collects session configurations(including `spark.datasource.iceberg.xxx` and `spark.sql.iceberg.xxx` values) and set options to Iceberg tables. In that way, configurations are unified for DataFrame and Catalog tables. WDYT? In my experience, allowing the user to control some behaviors by using `SET xxx=yyy` is fantastic, but it seems that Iceberg only allows a small set of configurations to be overwritten by SQL session configuration. So here the questions are: - are there principles for which type of configurations should be exposed to session conf? - since we are building a DataSource upon Spark DSv2 API, the API does have such limitations to allow overwrite options using SQL syntax, I would rather respect API design or promote the API change upstream than do such a hackly thing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
