davsclaus opened a new pull request, #3209:
URL: https://github.com/apache/camel-kamelets/pull/3209
Part of #3170: the AWS Kamelets can connect to AWS through an HTTP proxy,
with three new optional options, `proxyHost`, `proxyPort` and `proxyProtocol`.
### The problem
None of the AWS Kamelets exposed the proxy options of their components. In a
network where only a proxy reaches the internet, which is common in companies,
they could not be configured to use it. (The JVM system properties
`-Dhttp.proxyHost` / `-Dhttps.proxyHost` happen to be picked up by the AWS
SDK's HTTP client, and that works today, but it is global for the whole JVM and
not visible in the Kamelet. The `HTTP_PROXY` environment variable was not
picked up in a test.)
### The options
All 29 AWS Kamelets that use the AWS SDK get the same three options, with
the same wording:
| Option | Template mapping | What it does |
|---|---|---|
| `proxyHost` | `proxyHost: "{{?proxyHost}}"`, or
`&proxyHost={{?proxyHost}}` in an inline URI | The host name or IP address of
the proxy. Used only together with `proxyPort`. `pattern: "^[A-Za-z0-9._-]+$"`.
|
| `proxyPort` | `proxyPort: "{{?proxyPort}}"`, or
`&proxyPort={{?proxyPort}}` | The port of the proxy, such as `3128` for Squid.
Integer. |
| `proxyProtocol` | `proxyProtocol: "{{?proxyProtocol}}"`, or
`&proxyProtocol={{?proxyProtocol}}` | How to talk to the proxy itself: `HTTP`
or `HTTPS` (enum). |
- **`proxyProtocol`:** in every AWS component it is the scheme of the proxy,
not of the calls to AWS (`AwsClientBuilderUtil` builds
`proxyProtocol://proxyHost:proxyPort`). The component default is `HTTPS`, which
makes every call fail against a plain HTTP proxy such as Squid, with
`Unsupported or unrecognized SSL message`. The description therefore tells
users to set `HTTP` for such a proxy. The calls to AWS stay HTTPS: the proxy
log shows `CONNECT sqs.us-east-1.amazonaws.com:443` tunnels.
- **Defaults:** all three are optional and have no `default`, as in #3174.
When they are unset, the optional placeholders drop them, and the endpoint URIs
are identical to `main`'s.
- **Templates:** the parameters are appended at the end of each endpoint,
without reordering the existing ones. Kamelets with more than one AWS endpoint
pass them to all of them: `aws-sqs-fifo-sink` and `aws-sns-fifo-sink` (two
endpoints each), and `aws-s3-event-based-source` (the SQS consumer and the
`aws2-s3` `toD` of `getObject`).
- **Left out:** `aws-redshift-sink` and `aws-redshift-source`, which use
JDBC (`camel:sql`), not the AWS SDK. Every other AWS component used by the
catalog (`aws2-*`, `aws-bedrock*`, `aws-cloudtrail`, `aws-secrets-manager`) has
the three options.
- **`aws-kinesis-source` with `useKclConsumers`:** the KCL builds its own
DynamoDB and CloudWatch clients (`KclKinesis2Consumer`), without the proxy and
without the endpoint override. Its description and partial say so. This is a
limitation of the component.
- **Descriptions:**
- The "Good to know" paragraph of each Kamelet ends with: "To reach AWS
through an HTTP proxy, set `proxyHost` and `proxyPort`, and set `proxyProtocol`
to `HTTP` for a proxy that takes plain HTTP, such as Squid."
`aws-bedrock-agent-runtime-sink` and `aws-sns-sink` had no "Good to know"
paragraph, so they get one with this sentence.
- The descriptions say that the Kamelet has no option for a proxy user or
password: the components have none.
- **Partial pages:** each has a new "Through an HTTP Proxy" section with an
example.
### Verification
Tested with Camel JBang 4.24.0-SNAPSHOT, with `floci/floci` as the AWS
stand-in and `ubuntu/squid` as the forward proxy, both on one Docker network.
`uriEndpointOverride` was `http://floci:4566`: the name `floci` resolves only
inside the Docker network, so a call can succeed only through the proxy. Squid
logged each request with its `X-Amz-Target` header. The Kamelets without
`uriEndpointOverride` (`aws-lambda-sink`, `aws-secrets-manager-sink`,
`aws-ses-sink`, `aws-translate-action`) went to real AWS through the proxy,
with fake credentials.
| Case | Result |
|---|---|
| Options unset, all 29 Kamelets in one integration, `main` and this branch
| `camel get endpoint --json`: the 32 AWS endpoint URIs are identical, and so
is the `toD` `aws2-s3` URI of `aws-s3-event-based-source` (checked in a second
run of `main`), with no `proxy*` parameter |
| `aws-sqs-sink` + `aws-sqs-source`, proxy set (`HTTP`) | Message sent and
received. Squid: `CreateQueue`, `ListQueues`, `SendMessage`, `ReceiveMessage`,
`DeleteMessage` to `http://floci:4566/` |
| `aws-s3-sink`, proxy set | File written. Squid: `HEAD /bktp`, `PUT /bktp`,
`PUT /bktp/hello.txt` |
| `aws-sns-sink`, proxy set | Published. Squid: 3 `POST http://floci:4566/` |
| `aws-kinesis-sink` + `aws-kinesis-source`, proxy set | Record written and
read. Squid: `PutRecord`, `ListShards`, `GetShardIterator`, `GetRecords` |
| The same with `asyncClient: true` on the source (Netty client) | Record
read. Squid: `CONNECT floci:4566`, as Netty tunnels |
| `aws-ddb-sink`, proxy set | Item written. Squid: `DescribeTable`,
`PutItem` |
| `aws-s3-event-based-source` with `getObject: true`, proxy set | Squid: the
SQS calls to floci and `CONNECT
kamelets-proxy-test-bucket.s3.amazonaws.com:443` for the `getObject` call,
which then fails at AWS with the fake key: both endpoints use the proxy |
| The 20 other Kamelets, proxy set | Each one's requests appear in the Squid
log: `aws-sqs-fifo-sink`, `aws-sns-fifo-sink`, `aws-sqs-batch-sink`,
`aws-s3-source`, `aws-s3-streaming-upload-sink`, `aws-cloudwatch-sink`,
`aws-eventbridge-sink`, `aws-kinesis-firehose-sink`, `aws-ddb-streams-source`,
`aws-cloudtrail-source`, `aws-comprehend-sink`, `aws-ec2-sink`,
`aws-polly-sink`, `aws-timestream-query-sink`, the three Bedrock sinks (floci),
and `aws-lambda-sink`, `aws-secrets-manager-sink`, `aws-ses-sink`,
`aws-translate-action` (`CONNECT <service>.us-east-1.amazonaws.com:443`, then
`The security token included in the request is invalid` from AWS) |
| `proxyPort` set to a closed port, sqs, s3, sns, kinesis (sync and async),
ddb | Every call fails with `Connect to http://localhost:13999 failed:
Connection refused`; nothing in the Squid log |
| `proxyProtocol` unset (component default `HTTPS`) against Squid, sqs, s3,
sns, kinesis | Every call fails with `Unsupported or unrecognized SSL message`;
Squid logs `error:invalid-request` |
| `proxyHost` set without `proxyPort`, sqs and s3 to
`http://localhost:14566` | Works, and nothing in the Squid log: the proxy is
not used |
| No `uriEndpointOverride`, real AWS through the proxy: s3, sqs, sns,
kinesis, ddb | Squid: `CONNECT s3.amazonaws.com:443`, `sqs`, `sns`, `kinesis`,
`dynamodb.us-east-1.amazonaws.com:443`; AWS answers `The security token
included in the request is invalid` |
| `aws-kinesis-source` with `useKclConsumers: true`, proxy set | The Kinesis
calls go through the proxy; the DynamoDB calls of the KCL go straight to AWS
(`DynamoDbException: The security token included in the request is invalid`, no
`CONNECT dynamodb...` in the Squid log) |
`uriEndpointOverride` does not bypass the proxy: the SDK sends the requests
for the overridden endpoint through the proxy too, as plain HTTP when the
override is `http://`.
`./mvnw clean verify -DskipTests` and then `./mvnw clean verify` passed from
the root. Before building, the library copies of the 29 Kamelets were refreshed
with `./mvnw -pl library/camel-kamelets
resources:copy-resources@copy-resource-one`. `KameletsCatalogTest` counts the
options of `aws-sqs-source`: 23 became 26. The build regenerated
`kamelets-index.json`.
### Merge order
This PR and #3206 both change `aws-s3-sink`, `aws-s3-streaming-upload-sink`
and `aws-s3-event-based-source`; both append at the ends of the same URIs and
paragraphs. Whichever merges second will be rebased and rebuilt.
_Claude Code on behalf of Claus Ibsen_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]