davsclaus opened a new pull request, #3203:
URL: https://github.com/apache/camel-kamelets/pull/3203

   Part of #3183 and #3144. `kafka-source` and `kafka-sink` can now read and 
write records through a schema registry. The vendor logic is in camel-kafka 
([CAMEL-27130](https://issues.apache.org/jira/browse/CAMEL-27130), Camel 4.24), 
and the Kamelets only pass the options through.
   
   ### New options on `kafka-source` and `kafka-sink`
   
   | Option | Values | Maps to |
   |---|---|---|
   | `schemaRegistryVendor` | `confluent`, `apicurio` (optional) | 
`schemaRegistryVendor` |
   | `schemaRegistryUrl` | `http(s)://...` (optional) | `schemaRegistryURL` |
   | `schemaFormat` | `avro` (default), `json` | `schemaFormat` |
   
   - **`confluent`** also covers registries with the Confluent API, such as 
Redpanda and Karapace.
   - **`apicurio`** is Apicurio Registry 3.x.
   - **Without `schemaRegistryVendor`**, nothing changes.
   
   ### What the body becomes
   
   - **kafka-source:** the value read with its schema is turned into **JSON 
text**, so the next step gets text, as it does without a registry. An Avro 
record gives its JSON form, such as `{"id": 7, "item": "camel"}`, and a JSON 
Schema value goes through the json data format. Without this, the body was a 
`GenericRecord` or a `Map`, which `json-serialize-action` and the text sinks 
cannot use. That is why both Kamelets now depend on `camel:jackson3`.
   - **kafka-sink, with `json`:** JSON text is turned into an object first, so 
the record is written with an object schema instead of `"string"`.
   - **kafka-sink, with `avro`:** a text body is written as an Avro string.
   
   ### Tests
   
   **New Citrus test, `kafka-schema-registry-route-test`:** `timer-source` 
sends JSON to `kafka-sink`, which writes it with a JSON Schema through the 
schema registry of Redpanda (Confluent serdes, `--repos=confluent`). 
`kafka-source` reads it back as JSON text, and the test checks that the 
registry has a JSON schema for the topic. All 4 KafkaIT tests pass locally.
   
   **Manual runs with Camel JBang 4.24.0-SNAPSHOT:**
   
   | Case | Result |
   |---|---|
   | No registry | Unchanged, plain text |
   | Confluent JSON, JSON text in | `{"id":1,"item":"camel"}` out |
   | Confluent Avro, text in | Text out; the schema `"string"` is registered |
   | Confluent Avro record written by another application | `{"id": 7, "item": 
"camel"}` as `String` |
   | Apicurio 3.0.8 Avro, text in | Text out |
   
   **Limits, stated in the docs:**
   
   - **Schema check:** the sink registers a schema made from the message. With 
`json` that schema accepts any JSON object, and the sink does not check the 
message against a schema that is already registered.
   - **Registry login:** a registry that requires one is not supported yet, 
since camel-kafka has no options for it.
   - **`apicurio` with `json`:** it needs the schema registered first, and even 
then the producer fails on Camel JBang with an `IncompatibleClassChangeError` 
in `com.networknt.schema`. Camel pins json-schema-validator 3.0.9, while the 
Apicurio serde was built against an older version. So the docs say to use 
`avro` with `apicurio`.
   
   ### Not in this PR (follow-ups for #3183)
   
   - `protobuf`;
   - `kafka-batch-source`;
   - the registry login;
   - removing the deprecated Apicurio Kamelets (#3182).
   
   _Claude Code on behalf of Claus Ibsen_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to