rmoff commented on issue #13457:
URL: https://github.com/apache/iceberg/issues/13457#issuecomment-3032947176
The problem here is the SMT. If I use dynamic routing and just rely on an
existing field in the data, it works fine:
```
echo '{"target":"tmp.scratch","order_id": "001", "customer_id": "cust_123",
"product": "laptop", "quantity": 1, "price": 999.99}
{"target":"tmp.scratch","order_id": "002", "customer_id": "cust_456",
"product": "mouse", "quantity": 2, "price": 25.50}
{"target":"tmp.scratch","order_id": "003", "customer_id": "cust_789",
"product": "keyboard", "quantity": 1, "price": 75.00}
{"target":"tmp.scratch","order_id": "004", "customer_id": "cust_321",
"product": "monitor", "quantity": 1, "price": 299.99}
{"target":"tmp.scratch","order_id": "005", "customer_id": "cust_654",
"product": "headphones", "quantity": 1, "price": 149.99}' | docker compose exec
-T kcat kcat -P -b broker:9092 -t scratch
```
```
kcctl apply -f - <<EOF
{
"name": "iceberg-sink-scratch",
"config": {
"connector.class": "io.tabular.iceberg.connect.IcebergSinkConnector",
"topics": "scratch",
"iceberg.tables.dynamic-enabled": "true",
"iceberg.tables.route-field":"target",
"iceberg.tables.auto-create-enabled": "true",
"iceberg.tables.evolve-schema-enabled": "true",
"iceberg.catalog.catalog-impl":
"org.apache.iceberg.aws.glue.GlueCatalog",
"iceberg.catalog.warehouse": "s3://rmoff-lakehouse/00/",
"iceberg.catalog.io-impl": "org.apache.iceberg.aws.s3.S3FileIO",
"key.converter":"org.apache.kafka.connect.json.JsonConverter",
"key.converter.schemas.enable":"false",
"value.converter":"org.apache.kafka.connect.json.JsonConverter",
"value.converter.schemas.enable":"false",
"iceberg.control.commit.interval-ms": "1000"
}
}
EOF
```
```
❯ aws glue get-tables --database-name tmp --query 'TableList[].Name'
--output table --region us-east-1
-------------------------
| GetTables |
+-----------------------+
| dynamic_orders_json |
| scratch |
| static_orders_json |
+-----------------------+
❯ aws s3 --recursive ls s3://rmoff-lakehouse/00
2025-07-03 17:39:04 2058
00/tmp.db/dynamic_orders_json/data/00001-1751560743209-4cbce9d3-4021-445c-858c-894b727ff56d-00001.parquet
2025-07-03 17:39:03 1430
00/tmp.db/dynamic_orders_json/metadata/00000-3cf5ad66-e360-498c-a9c6-1127f6f7387b.metadata.json
2025-07-03 17:56:20 1968
00/tmp.db/scratch/data/00001-1751561779319-b449237b-a86d-4b11-adbc-2a9019f51660-00001.parquet
2025-07-03 17:56:19 1416
00/tmp.db/scratch/metadata/00000-2de3477c-25bf-4c9b-a7ce-2d5df380f7f4.metadata.json
2025-07-03 17:56:22 2620
00/tmp.db/scratch/metadata/00001-2cfff70d-d034-4ef8-9eea-22630081b06d.metadata.json
2025-07-03 17:56:21 7023
00/tmp.db/scratch/metadata/82ad629d-b787-4075-b5da-27703059fd08-m0.avro
2025-07-03 17:56:21 4234
00/tmp.db/scratch/metadata/snap-6145769438534152212-1-82ad629d-b787-4075-b5da-27703059fd08.avro
2025-07-03 17:34:21 1635
00/tmp.db/static_orders_json/data/00001-1751560405549-e180131f-fd33-46db-af68-6cdadd0f360f-00001.parquet
2025-07-03 17:33:25 1327
00/tmp.db/static_orders_json/metadata/00000-27c1ccf9-c5cc-436c-9246-fe6bd9d0041f.metadata.json
2025-07-03 17:34:23 2550
00/tmp.db/static_orders_json/metadata/00001-53d33894-c660-42b2-8eac-5c695b9edada.metadata.json
2025-07-03 17:34:22 6961
00/tmp.db/static_orders_json/metadata/18cfb817-5b80-4ae7-bd99-a927e96d64ae-m0.avro
2025-07-03 17:34:22 4242
00/tmp.db/static_orders_json/metadata/snap-6206227594069763789-1-18cfb817-5b80-4ae7-bd99-a927e96d64ae.avro
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
