davsclaus opened a new pull request, #3226:
URL: https://github.com/apache/camel-kamelets/pull/3226

   Part of #3170: `elasticsearch-index-sink` and `opensearch-index-sink` take a 
connection and a socket timeout, and their `certificate` now also takes a DER 
certificate. `solr-sink` and `solr-source` take a request handler, a connection 
timeout and a request timeout.
   
   ### The options
   
   | Option | Kamelets | Template mapping | What it does |
   |---|---|---|---|
   | `connectionTimeout` | `elasticsearch-index-sink`, `opensearch-index-sink` 
| `connectionTimeout={{?connectionTimeout}}` | How long to wait for the 
connection, in ms. Unset: 30000. |
   | `socketTimeout` | `elasticsearch-index-sink`, `opensearch-index-sink` | 
`socketTimeout={{?socketTimeout}}` | How long to wait for data from the 
cluster, in ms. Unset: 30000. |
   | `requestHandler` | `solr-sink`, `solr-source` | 
`requestHandler={{?requestHandler}}` | The path of another handler of the 
collection, such as `/update/catalog` or `/query`. Unset: `/update` for the 
sink, `/select` for the source. It has a `pattern:` (`^/[A-Za-z0-9_./-]+$`). |
   | `connectionTimeout` | `solr-sink`, `solr-source` | 
`connectionTimeout={{?connectionTimeout}}` | How long to wait for the 
connection, in ms. Unset: 60000. |
   | `requestTimeout` | `solr-sink`, `solr-source` | 
`requestTimeout={{?requestTimeout}}` | How long to wait for the answer of Solr, 
in ms. Unset: 600000, which is 10 minutes. |
   
   - **Defaults:** all the new options are optional and have no `default`, as 
in #3174. When they are unset, the optional placeholders drop them, and the 
endpoint URIs are the same as on `main`.
   - **`certificate` on Elasticsearch and OpenSearch:** this option already was 
the trust store for a private CA, and it works: the Elasticsearch `http_ca.crt` 
and the OpenSearch demo `root-ca.pem` make HTTPS work, and without them the 
message fails with `PKIX path building failed`. But it took only PEM text 
encoded in base64. A DER certificate encoded in base64 contains `+`, which 
became a space, and the integration failed to start with `Illegal base64 
character 20`. The mapping is now 
`certificatePath=RAW(base64:{{?certificate}})`, the same as #3173 did for Solr, 
so both PEM and DER work. The option descriptions now say so, and give the 
`base64` command.
   - **`size` on `solr-source`:** the number of rows was already added by #3173 
as `size`. It is tested again below, and the partial page has an example with 
`size: 500`.
   - **Descriptions:** the "Good to know" paragraphs say how long a call waits 
when it is not answered, and which option changes that. The Elasticsearch and 
OpenSearch paragraphs say what to do for a private CA. The Solr paragraphs name 
the default handlers. The partial pages have new sections and an example each.
   
   ### Left out
   
   - **`maxRetryTimeout`:** in camel-elasticsearch and camel-opensearch 4.24 it 
is deprecated and has no effect, as the current REST client ignores it.
   - **A trust store file, or no hostname check:** without a bean, the 
components take one CA certificate (`certificatePath`), which `certificate` 
maps. A JKS or PKCS12 trust store needs an `SSLContextParameters` bean: 
`opensearch-index-sink` already takes one in `sslContextParameters`, 
`elasticsearch-index-sink` has no such option yet. The hostname verifier of 
camel-opensearch is a bean too. With the demo certificates, the host name check 
passes, as `esnode.pem` names `localhost` and `127.0.0.1`.
   
   ### Verification
   
   Tested with Camel JBang 4.24.0-SNAPSHOT against Elasticsearch 9.1.0 
(security on, HTTPS with the CA it generates), OpenSearch 3.9.0 (HTTPS with the 
demo certificates) and Solr 9.10.1, all in Docker. A slow server was made with 
toxiproxy (3 s of latency on every packet from the server), a server that never 
answers with a local TCP socket that accepts and stays silent, and a connection 
that never opens with the address `10.255.255.1`.
   
   | Case | Result |
   |---|---|
   | Options unset, all four Kamelets, `main` and this branch | `camel get 
endpoint --json` shows identical URIs |
   | Elasticsearch / OpenSearch, `certificate` = the CA as base64 PEM | 
Document `order-42` stored |
   | The same, `certificate` = the CA as base64 DER (has `+`), `main` | The 
integration does not start: `Illegal base64 character 20` |
   | The same, this branch | Document stored, on both |
   | Elasticsearch / OpenSearch, no `certificate` | Message fails: `PKIX path 
building failed`, on both |
   | Elasticsearch with the OpenSearch CA in `certificate` | Message fails: 
`PKIX path building failed` (only the given CA is trusted) |
   | Elasticsearch, slow server, timeouts unset | Stored after 6.2 s |
   | Elasticsearch, slow server, `socketTimeout: 1000` | Fails after 1.1 s: 
`SocketTimeoutException: 1.000 milliseconds timeout` |
   | Elasticsearch, slow server, `socketTimeout: 5000` | Stored after 6.2 s |
   | OpenSearch, slow server, timeouts unset | Stored after 9.2 s |
   | OpenSearch, slow server, `socketTimeout: 1000` | Fails after 4.2 s: 
`Connection is closed` (the HTTP/2 client closes the connection on the timeout) 
|
   | OpenSearch, slow server, `socketTimeout: 5000` | Stored after 9.2 s |
   | Elasticsearch / OpenSearch, server that never answers, timeouts unset | 
Fails after 30 s with `SocketTimeoutException`, on both |
   | Elasticsearch / OpenSearch, `10.255.255.1`, `connectionTimeout: 1000` | 
Fails after 1.1 s with a connect timeout, on both |
   | Elasticsearch / OpenSearch, `10.255.255.1`, timeouts unset | Fails after 
30 s with a connect timeout, on both |
   | Elasticsearch / OpenSearch, `connectionTimeout: 5000`, `socketTimeout: 
7000` | Both appear in the endpoint URI |
   | `solr-source`, 25 documents, `size` unset / 20 / 50 | 10 / 20 / 25 
documents |
   | `solr-source`, `requestHandler: /newOrders` (a search handler with the 
invariant `fq=status_s:new`), `size: 50` | 13 documents, only the new ones |
   | `solr-source`, `requestHandler: /nothere` | The query fails with 
`RemoteSolrException` from `/solr/products/nothere`, and is logged |
   | `solr-sink`, `requestHandler` unset | 25 documents stored, none with 
`via_s` |
   | `solr-sink`, `requestHandler: /update/tagged` (an update handler that adds 
`via_s:tagged` with the template processor) | 25 documents stored, all with 
`via_s: tagged` |
   | `solr-sink`, `requestHandler: /update/nothere` | Message fails with 
`RemoteSolrException` |
   | `solr-sink`, slow server, timeouts unset / `requestTimeout: 10000` | 
Stored after 6.1 s |
   | `solr-sink`, slow server, `requestTimeout: 1000` | Fails after 2.1 s: 
`HttpTimeoutException: request timed out` |
   | `solr-source`, slow server, `requestTimeout: 1000` | Each query fails with 
`HttpTimeoutException`, is logged, and the next one runs 15 s later |
   | `solr-sink`, server that never answers, timeouts unset | Fails after 20 
min: `HttpTimeoutException: request timed out` (a `HEAD` request, then the 
request, each waited 10 min) |
   | `solr-sink`, server that never answers, `requestTimeout: 3000`, two 
messages | The first fails after 6 s, the second after 3 s |
   | `solr-source`, server that never answers, `requestTimeout: 3000` | The 
query fails after 3 s |
   | `solr-sink`, `10.255.255.1`, `connectionTimeout: 1000` | Fails after 2.1 
s: `HttpConnectTimeoutException` |
   | `solr-sink`, `10.255.255.1`, timeouts unset | Fails after 120 s: 
`HttpConnectTimeoutException` |
   | `solr-source`, all three options set | `connectionTimeout=5000`, 
`requestHandler=/newOrders`, `requestTimeout=7000` and `size=50` in the 
endpoint URI |
   | The examples of the four partial pages, run against the containers | All 
work; the OpenSearch one with a DER CA |
   
   Over plain HTTP, the Solr client checks the server with a `HEAD` request 
before its first `POST`, once per client. Each request waits for its own 
timeouts, so the first message of `solr-sink` waits up to twice as long, for 
example 20 minutes when Solr never answers, or two minutes when it cannot be 
reached. The descriptions of `solr-sink` say so. The queries of `solr-source` 
are `GET` requests, which wait only once.
   
   `./mvnw clean verify -DskipTests` and then `./mvnw clean verify` passed from 
the root. Before building, the library copies of the four Kamelets were 
refreshed with `./mvnw -pl library/camel-kamelets 
resources:copy-resources@copy-resource-one`. The build regenerated 
`kamelets-index.json`.
   
   _Claude Code on behalf of Claus Ibsen_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to