mxm opened a new pull request, #13304:
URL: https://github.com/apache/iceberg/pull/13304
This completes the breakdown of #12424. Most importantly, it adds the
following missing components:
#### DynamicIcebergSink
The actual Flink sink which ties together the previously merged components.
The sink will be instantiated by the user like this:
```java
DynamicIcebergSink
.forInput(inputStream)
.withConverter(new MyConverter())
.catalogLoader(..)
.append();
```
An example for a converter implementation would be:
```java
class MyConverter implements DynamicRecordConverter<MyInput> {
final RuntimeParams params;
MyConverter(RuntimeParams params) {
this.params = params;
}
@Override
public void convert(MyInput inputRecord, Collector<DynamicRecord> out) {
DynamicRecord record = new DynamicRecord(
inputRecord.getTable(),
inputRecord.getBranch(),
inputRecord.buildSchema(),
inputRecord.convertData(),
PartitionSpec.unpartitioned(),
params.distributionMode(),
params.writeParallelism());
out.collect(record);
}
}
```
#### DynamicRecordProcessor
The core processing logic which takes user-provided input to write to
Iceberg tables and make the necessary changes to the Iceberg metadata.
#### Benchmarks
- DynamicRecordSerializerDeserializerBenchmark
- TestDynamicIcebergSinkPerf
### Previous PRs
The following previous PRs have been merged already:
1. #12996
2. #13032
3. #13080
4. #13277
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
