rlokugamage opened a new issue, #13623:
URL: https://github.com/apache/iceberg/issues/13623
### Apache Iceberg version
1.9.2 (latest release)
### Query engine
Kafka Connect
### Please describe the bug 🐞
I'm trying out the kafka connect plugin to write messages to an iceberg table
my connector config
```json
{
"name": "test-sink",
"config": {
"tasks.max": "1",
"connector.class": "org.apache.iceberg.connect.IcebergSinkConnector",
"topics": "test-topic",
"iceberg.tables.auto-create-enabled": "true",
"iceberg.tables.evolve-schema-enabled": "true",
"iceberg.tables": "default.test_table",
"iceberg.tables.default-id-columns": "id",
"iceberg.tables.default-partition-by": "partition_id",
"iceberg.catalog.type": "rest",
"iceberg.catalog.client.region": "us-east-1",
"iceberg.catalog.uri": "http://iceberg:8181";,
"iceberg.catalog.warehouse": "s3://bucket/warehouse/",
"iceberg.catalog.s3.secret-access-key": "minioadmin",
"iceberg.catalog.s3.endpoint": "http://minio:9000";,
"iceberg.catalog.s3.path-style-access": "true",
"iceberg.catalog.s3.access-key-id": "minioadmin",
"iceberg.control.commit.interval-ms": 10000
}
}
```
I'm using the docker-compose in the runtime folder, which has been working
great. However I noticed that setting `iceberg.tables.auto-create-enabled` to
`true` and also specify an identifier column in
`iceberg.tables.default-id-columns`, it doesn't add the `identifier-field-ids`
property to the schema when it creates the table on that first commit.
This ends up with the connector duplicating the same message, despite
specifying the identifier field.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
