manuzhang opened a new issue, #17719:
URL: https://github.com/apache/iceberg/issues/17719
### Problem
Spark 4.2 `SparkCatalog` must distinguish properties managed by Iceberg from
properties supplied through Spark when replacing a view. Iceberg's
`ViewBuilder.replace` merges properties, so Spark performs a best-effort
follow-up removal for properties omitted from the replacement. Iceberg-managed
properties must be excluded from those removals.
PR #14984 currently maintains this list in the Spark module:
```java
private static final Set<String> ICEBERG_MANAGED_VIEW_PROPERTIES =
ImmutableSet.of(
ViewProperties.VERSION_HISTORY_SIZE,
ViewProperties.METADATA_COMPRESSION,
ViewProperties.WRITE_METADATA_LOCATION,
ViewProperties.REPLACE_DROP_DIALECT_ALLOWED);
```
The list is accurate today, but it duplicates knowledge from
`org.apache.iceberg.view.ViewProperties`. If core adds another Iceberg-managed
property and the Spark copy is not updated, a subsequent `CREATE OR REPLACE
VIEW` can silently remove that setting.
`ViewProperties.COMMENT` is intentionally not preserved by this mechanism
because Spark owns comment replacement.
### Proposed fix
- Define an authoritative immutable collection or classifier for
Iceberg-managed view properties next to `ViewProperties`.
- Update Spark and other consumers to use the core definition rather than
maintaining copies.
- Preserve the intentional treatment of `COMMENT` as an engine-owned
replacement field.
- Add coverage that prevents newly introduced Iceberg-managed properties
from being treated as removable Spark properties.
### Context
Identified while reviewing Spark 4.2 support in #14984:
https://github.com/apache/iceberg/pull/14984#discussion_r3788259688
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
