davsclaus opened a new pull request, #3363:
URL: https://github.com/apache/camel-kamelets/pull/3363
## What it does
A new source, `debezium-postgresql-source`, on the Camel `debezium-postgres`
component. It is the catalog's first change data capture (CDC) source: it sends
a message for each row inserted, updated or deleted in a PostgreSQL database,
as it happens. Use it from a route:
```yaml
- route:
from:
uri: "kamelet:debezium-postgresql-source"
parameters:
serverName: db.example.com
username: replicator
password: "{{db.password}}"
databaseName: shop
topicPrefix: shop
tableIncludeList: public.orders
offsetStorageFileName: /data/offsets.dat
steps:
- to: "kamelet:log-sink"
```
## Options
- Connection: `serverName`, `serverPort` (5432), `username`, `password`
(marked `format: password` and `urn:camel:group:credentials`), `databaseName`.
- `topicPrefix` (required): the logical name. It is also the connector name,
under which the position is saved. It has the pattern `^[a-zA-Z0-9._-]+$`.
- What is sent: `tableIncludeList`, and `snapshotMode` (`initial`,
`no_data`, `always`, `initial_only` or `when_needed`).
- PostgreSQL objects: `slotName` (`debezium`, pattern `^[a-z0-9_]+$`),
`publicationName` (`dbz_publication`, pattern `^[a-zA-Z0-9_]+$`),
`publicationAutocreateMode` (`all_tables`, `filtered`, `disabled` or
`no_tables`), and `pluginName`. `pluginName` defaults to `pgoutput`, which is
built into PostgreSQL; the component's own default is `decoderbufs`.
- Position store: `offsetStorageFileName` (default
`debezium-postgresql-offsets.dat`), or `offsetStorage` =
`org.apache.kafka.connect.storage.KafkaOffsetBackingStore`. The Kafka store
uses `offsetStorageBootstrapServers` (passed as
`additionalProperties.bootstrap.servers`), `offsetStorageTopic`,
`offsetStoragePartitions` and `offsetStorageReplicationFactor`.
- Every other connector option is set as
`camel.component.debezium-postgres.<option>`. The Kamelet uses the global
component, not a local bean, so these properties reach it. The description says
so, and the test checks it with `columnExcludeList`.
## Input / Output
- **Output:** the body is JSON text in bytes. It is the row after the
change, or before it for a delete. Without `REPLICA IDENTITY FULL`, a deleted
row has only its key columns set, and the other columns are `null`.
- **Headers:** `debezium-operation` (`c`, `u`, `d` or `r`), `debezium-table`
(`schema.table`), `debezium-key` (the primary key as JSON text) and
`Content-Type: application/json`. The `CamelDebezium*` headers are removed:
they hold Kafka Connect `Struct` objects, which other systems cannot read.
- Events without a row are filtered out: tombstones, heartbeats and logical
decoding messages.
- `out` declares the `json` type with `default: json`, as
`postgresql-source` does. The template marshals the JSON itself, after
converting the `Struct` to a Map with the component's type converter.
## Tests
Both tests are in `DatabaseIT`, under `database/debezium`. They use
`postgres:17.12-alpine` with `-c wal_level=logical`, and `sql.groovy` gains a
`postgres` case that runs `psql` in the container.
- `debezium-postgresql-source-route-test` checks:
- The snapshot row (`r`), then an insert, an update and a delete, each
with the right operation, table, key and body.
- The body type is `[B`, with `Content-Type` `application/json`.
- The `CamelDebezium*` headers are gone.
- A column left out by
`camel.component.debezium-postgres.columnExcludeList` is not in the body.
- After a stop, a row inserted in a table outside `tableIncludeList` is
not sent. A restart with the same position file sends only the change made
while it was stopped.
- `debezium-postgresql-source-kafka-route-test`: the same restart check with
the Kafka position store (`apache/kafka` 4.4.0). The log confirms the Kafka
store was used and no position file was written.
Local results:
- `./mvnw -o clean install -DskipITs` passes.
- `DatabaseIT` passes in full: 10 tests, 0 failures.
The Kamelet is `Preview` and labelled `verified`.
Fixes #3360.
## Upgrade note
- New Kamelet `debezium-postgresql-source` (Preview): sends the row changes
of PostgreSQL tables as they happen (change data capture with Debezium), one
JSON message per insert, update or delete, with the `debezium-operation`,
`debezium-table` and `debezium-key` headers; its position is saved in a file or
in a Kafka topic.
_Claude Code on behalf of Claus Ibsen_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_016Y4dDmBDsSWV117kccuNf9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]