davsclaus opened a new pull request, #3363:
URL: https://github.com/apache/camel-kamelets/pull/3363

   ## What it does
   
   A new source, `debezium-postgresql-source`, on the Camel `debezium-postgres` 
component. It is the catalog's first change data capture (CDC) source: it sends 
a message for each row inserted, updated or deleted in a PostgreSQL database, 
as it happens. Use it from a route:
   
   ```yaml
   - route:
       from:
         uri: "kamelet:debezium-postgresql-source"
         parameters:
           serverName: db.example.com
           username: replicator
           password: "{{db.password}}"
           databaseName: shop
           topicPrefix: shop
           tableIncludeList: public.orders
           offsetStorageFileName: /data/offsets.dat
         steps:
           - to: "kamelet:log-sink"
   ```
   
   ## Options
   
   - Connection: `serverName`, `serverPort` (5432), `username`, `password` 
(marked `format: password` and `urn:camel:group:credentials`), `databaseName`.
   - `topicPrefix` (required): the logical name. It is also the connector name, 
under which the position is saved. It has the pattern `^[a-zA-Z0-9._-]+$`.
   - What is sent: `tableIncludeList`, and `snapshotMode` (`initial`, 
`no_data`, `always`, `initial_only` or `when_needed`).
   - PostgreSQL objects: `slotName` (`debezium`, pattern `^[a-z0-9_]+$`), 
`publicationName` (`dbz_publication`, pattern `^[a-zA-Z0-9_]+$`), 
`publicationAutocreateMode` (`all_tables`, `filtered`, `disabled` or 
`no_tables`), and `pluginName`. `pluginName` defaults to `pgoutput`, which is 
built into PostgreSQL; the component's own default is `decoderbufs`.
   - Position store: `offsetStorageFileName` (default 
`debezium-postgresql-offsets.dat`), or `offsetStorage` = 
`org.apache.kafka.connect.storage.KafkaOffsetBackingStore`. The Kafka store 
uses `offsetStorageBootstrapServers` (passed as 
`additionalProperties.bootstrap.servers`), `offsetStorageTopic`, 
`offsetStoragePartitions` and `offsetStorageReplicationFactor`.
   - Every other connector option is set as 
`camel.component.debezium-postgres.<option>`. The Kamelet uses the global 
component, not a local bean, so these properties reach it. The description says 
so, and the test checks it with `columnExcludeList`.
   
   ## Input / Output
   
   - **Output:** the body is JSON text in bytes. It is the row after the 
change, or before it for a delete. Without `REPLICA IDENTITY FULL`, a deleted 
row has only its key columns set, and the other columns are `null`.
   - **Headers:** `debezium-operation` (`c`, `u`, `d` or `r`), `debezium-table` 
(`schema.table`), `debezium-key` (the primary key as JSON text) and 
`Content-Type: application/json`. The `CamelDebezium*` headers are removed: 
they hold Kafka Connect `Struct` objects, which other systems cannot read.
   - Events without a row are filtered out: tombstones, heartbeats and logical 
decoding messages.
   - `out` declares the `json` type with `default: json`, as 
`postgresql-source` does. The template marshals the JSON itself, after 
converting the `Struct` to a Map with the component's type converter.
   
   ## Tests
   
   Both tests are in `DatabaseIT`, under `database/debezium`. They use 
`postgres:17.12-alpine` with `-c wal_level=logical`, and `sql.groovy` gains a 
`postgres` case that runs `psql` in the container.
   
   - `debezium-postgresql-source-route-test` checks:
     - The snapshot row (`r`), then an insert, an update and a delete, each 
with the right operation, table, key and body.
     - The body type is `[B`, with `Content-Type` `application/json`.
     - The `CamelDebezium*` headers are gone.
     - A column left out by 
`camel.component.debezium-postgres.columnExcludeList` is not in the body.
     - After a stop, a row inserted in a table outside `tableIncludeList` is 
not sent. A restart with the same position file sends only the change made 
while it was stopped.
   - `debezium-postgresql-source-kafka-route-test`: the same restart check with 
the Kafka position store (`apache/kafka` 4.4.0). The log confirms the Kafka 
store was used and no position file was written.
   
   Local results:
   - `./mvnw -o clean install -DskipITs` passes.
   - `DatabaseIT` passes in full: 10 tests, 0 failures.
   
   The Kamelet is `Preview` and labelled `verified`.
   
   Fixes #3360.
   
   ## Upgrade note
   
   - New Kamelet `debezium-postgresql-source` (Preview): sends the row changes 
of PostgreSQL tables as they happen (change data capture with Debezium), one 
JSON message per insert, update or delete, with the `debezium-operation`, 
`debezium-table` and `debezium-key` headers; its position is saved in a file or 
in a Kafka topic.
   
   _Claude Code on behalf of Claus Ibsen_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   
   https://claude.ai/code/session_016Y4dDmBDsSWV117kccuNf9
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to