mxm opened a new pull request, #13883:
URL: https://github.com/apache/iceberg/pull/13883
This feature allows modifying table properties on the fly. Users have
reported that they need to update table properties of both new and existing
tables. Some catalogues even check for certain properties and reject table
changes if those properties are not present.
Below the doc added as part of this PR:
---
#### Dynamic Table Properties
The Dynamic Sink supports dynamically updating table properties. This
feature allows you to:
- Set different properties for different tables based on table names
- Apply properties during table creation
- Update properties for existing tables
#### TablePropertiesUpdater Interface
The `TablePropertiesUpdater` is an interface that receives the
fully-qualified table name and current properties, then returns the updated
properties:
```java
interface TablePropertiesUpdater extends Serializable {
Map<String, String> apply(String tableName, Map<String, String>
currentProperties);
}
```
#### Usage Example
```java
TablePropertiesUpdater updater = (tableName, currentProps) -> {
Map<String, String> updatedProps = new HashMap<>(currentProps);
// Set compression based on table name
if (tableName.contains("logs")) {
updatedProps.put("write.parquet.compression-codec", "gzip");
} else {
updatedProps.put("write.parquet.compression-codec", "snappy");
}
// Set format properties
updatedProps.put("write.format.default", "parquet");
// Add custom metadata
updatedProps.put("created.by", "dynamic-sink");
updatedProps.put("table.identifier", tableName);
// Remove table properties
updatedProps.remove("to.be.removed.prop");
return updatedProps;
};
DynamicIcebergSink.forInput(dataStream)
.generator(new CustomRecordGenerator())
.catalogLoader(catalogLoader)
.tablePropertiesUpdater(updater)
.append();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
