liucao-dd commented on code in PR #16154:
URL: https://github.com/apache/iceberg/pull/16154#discussion_r3398915313
##########
docs/docs/spark-configuration.md:
##########
@@ -199,6 +199,8 @@ val spark = SparkSession.builder()
| spark.sql.iceberg.data-planning-mode | AUTO
| Scan planning mode for data files
(`AUTO`, `LOCAL`, `DISTRIBUTED`)
|
| spark.sql.iceberg.delete-planning-mode | AUTO
| Scan planning mode for delete
files (`AUTO`, `LOCAL`, `DISTRIBUTED`)
|
| spark.sql.iceberg.advisory-partition-size | Table default
| Advisory size (bytes) used for
writing to the Table when Spark's Adaptive Query Execution is enabled. Used to
size output files |
+| spark.sql.iceberg.split-size | Table default
| Overrides `read.split.target-size`
for scan planning. Session values are honored like read options and disable
adaptive split-size adjustment |
+| spark.sql.iceberg.split-size.<table-name> | Global session
default | Table-scoped split size
override using the fully qualified table name as a suffix
|
Review Comment:
Could we consider making table-scoped session configs a generic
identity-first pattern instead of a split-size-specific suffix? Having the
table name at the end of the config seems confusing.
An alternative is to resolve table-scoped session configs as:
` spark.sql.iceberg.<catalog>.<namespace...>.<table>.<setting-suffix>`
For this config, if the global key becomes
`spark.sql.iceberg.read.split-size`, the table-scoped key would be:
` spark.sql.iceberg.<catalog>.<namespace...>.<table>.read.split-size`
That keeps the table identity together, then applies the same setting suffix
used by the global session config, making it a generic pattern to support table
specific session config.
It would be a slightly larger change admittedly, and i'm happy to help or
open a separate PR just to add the table level generic session config support
if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
