davsclaus opened a new pull request, #3297:
URL: https://github.com/apache/camel-kamelets/pull/3297
Batch 1 (AWS) of #3117: a default data type on the AWS Kamelets whose body
is fixed. Camel runs the transformer of a default data type on every message,
so each default changes the body class or the `Content-Type`. The table shows
what changes. It was checked with the Citrus tests and with `camel run`, on a
local Camel main build that has CAMEL-27258.
> **Draft:** the `aws-cloudtrail-source` and `aws-kinesis-source` tests need
CAMEL-27258, which applies a source's output type before the consuming route.
That fix was merged today, after the published Camel 4.24.0-SNAPSHOT that CI
uses was built. This PR waits for a newer snapshot.
## What changes
| Kamelet | in | out | What changes (before → after) |
|---|---|---|---|
| `aws-cloudtrail-source` | - | `json` (was `text`) | `byte[]`, no
`Content-Type` → `byte[]` (JSON text), `application/json`. With CAMEL-27258 and
the old `text` default it would have been `String`, `text/plain`. Adds
`camel:jackson3`. |
| `aws-kinesis-source` | - | `binary` | `byte[]`, no `Content-Type` →
`byte[]`, `application/octet-stream`. With `useKclConsumers`: `ByteBuffer` →
`byte[]`. |
| `aws-kinesis-sink` | `binary` | - | `String` → `byte[]`, a stream cache is
kept; `Content-Type` → `application/octet-stream`. A Map, a List or a number
now fails. |
| `aws-kinesis-firehose-sink` | `binary` | - | same as `aws-kinesis-sink` |
| `aws-s3-streaming-upload-sink` | `binary` | - | same as `aws-kinesis-sink`
|
| `aws-lambda-sink` | `json` | - | JSON text `String` → JSON `byte[]`. A Map
or a List is written as JSON. Plain text `hello` → `"hello"`. A number now
fails. After the call the body is the `String` result as before, but
`Content-Type` is now `application/json`. Adds `camel:jackson3`. |
| `aws-eventbridge-sink` | `json` | - | as `aws-lambda-sink`. After the
call: `PutEventsResponse`, `Content-Type` → `application/json`. Adds
`camel:jackson3`. |
| `aws-bedrock-text-sink` | `json` | - | as `aws-lambda-sink`. After the
call: `String`, `Content-Type` → `application/json`. Adds `camel:jackson3`. |
| `aws-bedrock-agent-runtime-sink` | `text` | - | bytes or a stream →
`String`. After the call: `String`, `Content-Type` → `text/plain`. |
| `aws-comprehend-sink` | `text` | - | bytes or a stream → `String`. After
the call: the SDK List, `Content-Type` → `text/plain`. |
| `aws-polly-sink` | `text` | - | bytes or a stream → `String`. After the
call: the audio stream (`InputStreamCache`), `Content-Type` → `text/plain` (not
the audio type: that is in `CamelAwsPollyContentType`). |
| `aws-secrets-manager-sink` | `text` | - | bytes or a stream → `String`.
After the call: `CreateSecretResponse`, `Content-Type` → `text/plain`. |
| `aws-ses-sink` | `text` | - | bytes or a stream → `String`. After the
call: `String`, `Content-Type` → `text/plain`. |
| `aws-translate-action` | `text` | `text` | Input: text, bytes and a stream
are all sent as the text. Output: `String` as before. `Content-Type`: the
incoming one was kept → `text/plain`. |
The `Input:`/`Output:` paragraphs of these Kamelets now say this.
## Not done
- **`aws-ddb-sink`: no default.** The proposal had the
`aws2-ddb:application-json` type here, with the template's own
`transformDataType` step kept. A `camel run` against Floci showed a problem.
The default runs at the start of the Kamelet, before the template sets the
`operation` property. A body with an `operation` field, such as
`{"operation":"PutItem","id":1}` with `operation=PutItem`, then fails in
`Ddb2JsonDataTypeTransformer.checkBodyOperation`. Without the default this body
is stored, because Camel accepts an `operation` field that names the configured
operation. The default would also give the validator nothing. The YAML comment
now gives this reason.
- The other AWS Kamelets of the proposal wait on a Camel change:
`aws-s3-sink` waits on C3, and the sinks that ignore the body wait on C1. The
SQS and SNS sinks keep no default, because they forward `Content-Type` to AWS
(D2).
## Tests
-
`AwsIT#awsKinesis+awsFirehose+awsS3+awsLambda+awsEventBridge+awsBedrock+awsDdb+awsComprehend+awsPolly+awsSecretsManager+awsSes+awsCloudTrail`:
24 tests pass, run locally with Camel main + CAMEL-27258 in `~/.m2`.
- The expected `X-Body-Type`/`X-Content-Type` values are updated.
- The kinesis source and sink, eventbridge and ddb sink routes did not
record the body. They now log the body class and `Content-Type`, and the tests
verify that line.
- The firehose and streaming-upload routes now send the body to the Citrus
server as text, because Citrus reads an `application/octet-stream` body as a
byte array.
- `aws-translate-action` has no Citrus test here (#3290 adds one, see
below). I checked it with `camel run` against a local stand-in, before and
after the change.
- I also checked the `aws-lambda-sink` Map, List, plain-text and number
bodies with `camel run` against a stand-in.
- `./mvnw clean install` from the root passes, run twice.
## Overlap with open PRs
- **#3290** changes `aws-lambda-sink`, `aws-secrets-manager-sink` and
`aws-translate-action`, and their tests. Its translate test checks only
`X-Body-Type` (`String`), which this PR keeps. The lambda and secrets-manager
test files may conflict on rebase; the expected `X-Content-Type` there becomes
`ct=application/json` and `ct=text/plain`.
- **#3285** changes `aws-lambda-sink` and `aws-bedrock-text-sink`, and adds
`aws-lambda-sink-async` and `aws-bedrock-text-sink-converse` tests. Whichever
of the two PRs merges second must expect `ct=application/json` in those new
tests.
- In this PR the edits to those Kamelets are limited to the `dataTypes`
block, `camel:jackson3` and the `Input:`/`Output:` sentences.
## Upgrade note for #3190
```markdown
- **Default data types on the AWS Kamelets (#NNNN):**
- `aws-cloudtrail-source` changes from `text` to `json`. The record is
JSON in `byte[]` with `Content-Type: application/json`, instead of a `String`
with `text/plain`.
- `aws-kinesis-source` (`out: binary`) sets `Content-Type:
application/octet-stream`. With `useKclConsumers` the body is `byte[]` instead
of a `ByteBuffer`.
- `aws-kinesis-sink`, `aws-kinesis-firehose-sink` and
`aws-s3-streaming-upload-sink` (`in: binary`) turn text into `byte[]` and set
`Content-Type: application/octet-stream`. A stream is kept. A `Map`, a `List`
or a number now fails: turn it into JSON text first, for example with
`json-serialize-action`. AWS receives the same bytes as before.
- `aws-lambda-sink`, `aws-eventbridge-sink` and `aws-bedrock-text-sink`
(`in: json`) send JSON bytes and set `Content-Type: application/json`, which
stays on the message after the sink. A `Map` or a `List` is now written as
JSON. Text that is not a JSON object or array is sent as a JSON string (`hello`
becomes `"hello"`). A number fails.
- `aws-bedrock-agent-runtime-sink`, `aws-comprehend-sink`,
`aws-polly-sink`, `aws-secrets-manager-sink` and `aws-ses-sink` (`in: text`)
read the body as a `String` and set `Content-Type: text/plain`, which stays on
the message after the sink. After `aws-polly-sink` it is not the type of the
audio: use `CamelAwsPollyContentType`.
- `aws-translate-action` (`in` and `out: text`) sets `Content-Type:
text/plain` on the translated text. Before, the incoming `Content-Type` was
kept.
```
The CAMEL-27258 bullet in #3190 lists `aws-cloudtrail-source` under `text`.
Move it to `json`.
_Claude Code on behalf of Claus Ibsen_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_016Y4dDmBDsSWV117kccuNf9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]