davsclaus opened a new pull request, #3298:
URL: https://github.com/apache/camel-kamelets/pull/3298
Batch 5 of #3117 (databases, search, NoSQL): a default data type, `json`
(`application-json`), on 10 sources (`out`) and 4 sinks (`in`).
> **Draft:** the source defaults need a Camel 4.24.0-SNAPSHOT with
CAMEL-27258 (the output type of a source Kamelet is applied before the
consuming route), and the `mongodb-sink` default needs CAMEL-27259
(camel-mongodb insert of a `byte[]` body). Both were merged in Camel main on
2026-11-19, but the published snapshot that CI uses predates them. With that
snapshot, the source tests still get the stream and `MongoIT` fails on the
`mongodb-sink` insert. Rerun CI once a snapshot with both is published.
### What changes
| Kamelet | in | out | what changes (before → after) |
|---|---|---|---|
| `mariadb-source`, `mysql-source`, `oracle-database-source`,
`postgresql-source`, `sqlserver-source` | n/a | `json` | body: JSON text as a
stream (`InputStreamCache`) → JSON text in bytes (`byte[]`). `Content-Type:
application/json` as before. A JSON array (`useIterator` false) is kept as it
is. |
| `aws-redshift-source`, `databricks-source`, `snowflake-source` | n/a |
`json` | as the SQL sources (same template; no Citrus test) |
| `mongodb-source` | n/a | `json` | body: stream → `byte[]`; `Content-Type:
application/json` as before (now in the description) |
| `solr-source` | n/a | `json` | body: stream → `byte[]`; `Content-Type:
application/json` as before |
| `aws-redshift-sink`, `databricks-sink`, `snowflake-sink` | `json` | n/a |
as the SQL sinks of #3276: the body becomes JSON text in bytes before the
template parses it; a `Map` or `List` body is now written as JSON first;
afterwards `Content-Type` is `application/json` (before, the incoming one was
kept). No Citrus test: the template is the SQL one. |
| `mongodb-sink` | `json` | n/a | the body becomes JSON text in bytes, and
`Content-Type: application/json`, for `insert`, `save` and `update`; a `Map`
(or a `List` for `update`) is now written as JSON first. The stored document is
the same. Afterwards `Content-Type` is `application/json`. |
The Kamelets do the same as before in what they store or send: each source
already marshalled its result to JSON, and each sink already parsed JSON text.
The change is the Java class of the body that the next step sees (sources), and
the `Content-Type` and the accepted body kinds (sinks).
No row was dropped. Not in this PR: `mongodb-changes-stream-source` (waits
for C2, and #3283 changes it), and the rows the proposal marks "none" or "after
a Camel change".
### Risk to review
`mongodb-sink` with a `Map` or `Document` body is not documented input (the
description asks for JSON text), but the template accepted it. With the `json`
default, such a body goes through Jackson first: a `camel run` of the
transformer on `{_id: ObjectId(...), when: Date(0), qty: 2}` gave
`{"_id":{"date":"2024-05-24T14:18:26.000Z","timestamp":1716560306},"when":"1970-01-01T00:00:00.000Z","qty":2}`.
JSON text in the documented form (`$oid`, `$date`) is unchanged. I kept the
default, as for the SQL sinks of #3276; drop it from `mongodb-sink` if that
case matters.
### Descriptions
- Sources: the `Output:` paragraph says "JSON text in bytes";
`mongodb-source` now also says `Content-Type` is `application/json`.
- Sinks: the `Output:` paragraph says that `Content-Type` is set to
`application/json`.
### Tests
- `DatabaseIT` (mariadb, mysql, oracle, sqlserver sources), `SearchIT`
(solr-source): expect `X-Body-Type` `[B` instead of `InputStreamCache`.
- `PostgresIT`: gets an HTTP server; `postgresql-source` now records the
body type, the `Content-Type` and the row (it only checked a log line).
- `MongoIT`:
- `mongodb-source` records the body type, the `Content-Type` and the
document (it only checked a log line).
- `mongodb-sink` now checks the stored document after `insert`, `save` and
`update` of the same document, and what each leaves on the message (body type,
`Content-Type`, `CamelMongoOid`, matched and changed counts). Before, it
checked a log line.
### Overlap
- **#3294** (descriptions) changes the same `Output:` sentences of
`mariadb-`, `mysql-`, `oracle-database-`, `sqlserver-` and `solr-source` (it
adds "(a stream)"), and the `solr-source` test route. Whichever merges second
needs a rebase. With this PR the body is bytes, not a stream, so "(a stream)"
should become "in bytes".
- **#3283** changes `mongodb-changes-stream-source`, which this PR does not
touch.
### Upgrade note (for #3190)
```
* The SQL, MongoDB and Solr sources now give their JSON as bytes:
`mariadb-source`, `mysql-source`, `oracle-database-source`,
`postgresql-source`, `sqlserver-source`, `aws-redshift-source`,
`databricks-source`, `snowflake-source`, `mongodb-source` and `solr-source`
have an output data type of `json`. The body the next step sees is a `byte[]`
holding the same JSON text, where it was a stream before. `Content-Type` is
`application/json`, as before.
* `aws-redshift-sink`, `databricks-sink`, `snowflake-sink` and
`mongodb-sink` have an input data type of `json`, as the other SQL sinks have
since 4.24. They turn their input into JSON bytes and set `Content-Type:
application/json`, which stays on the message afterwards. They now also accept
a `Map` (and a `List`), which is written as JSON first. For `mongodb-sink`, a
`Map` or a MongoDB `Document` body that holds an `ObjectId` or a date is
therefore stored differently: an `ObjectId` becomes an object of its time
(`{"date":...,"timestamp":...}`) and a date becomes ISO text. Send such
documents as JSON text, with `{"$oid":"..."}` and `{"$date":"..."}`.
```
### Test results
- **Citrus, local** (Camel main with CAMEL-27258 in `~/.m2`):
- `DatabaseIT`, `PostgresIT`, `SearchIT`: 14 of 14 pass.
- `MongoIT`: 3 of 3 pass with a camel-mongodb that has CAMEL-27259. The
camel-mongodb jar in `~/.m2` predates CAMEL-27259; with it, the `mongodb-sink`
insert fails with `ClassCastException: [B cannot be cast to
[Ljava.lang.Object;`, as expected. To check the fix without installing Camel
into `~/.m2`, `MongoDbProducer` from Camel main was compiled into a copy of
that jar, which was passed to `camel run` only for this local run (not
committed).
- **Build:** `./mvnw clean install` from the root passes, run twice.
- Redshift, Databricks and Snowflake have no Citrus test; their templates
are the SQL ones that `DatabaseIT` and `PostgresIT` cover.
_Claude Code on behalf of Claus Ibsen_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_016Y4dDmBDsSWV117kccuNf9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]