davsclaus opened a new pull request, #3221:
URL: https://github.com/apache/camel-kamelets/pull/3221

   Part of #3170: the file Kamelets can read only some files, wait until a file 
is completely written, stream large files, sort and limit what a poll reads, 
and set their timeouts. `ftps-*` can use implicit TLS, `sftp-*` can check that 
the server still answers, `sftp-sink` and `scp-sink` can set file permissions, 
`scp-sink` can write to a directory, and `file-watch-source` can filter files 
and stop watching subdirectories.
   
   Earlier PRs already added `move`, `moveFailed` and `idempotentRepository` 
(#3178), `tempPrefix` and `doneFileName` (#3199), and the SSH key login 
(#3197). `scp-sink` already had a known hosts file (`knownHostsUri`, mapped to 
`knownHostsFile`), so it is not added again.
   
   ### The options
   
   `ftp-source`, `ftps-source`, `sftp-source`:
   
   | Option | Template mapping | What it does |
   |---|---|---|
   | `include` | `include: "RAW({{?include}})"` | A regular expression on the 
file name (not the directory), case-insensitive |
   | `exclude` | `exclude: "RAW({{?exclude}})"` | A regular expression for the 
names not to read; wins over `include` and `antInclude` |
   | `antInclude` | `antInclude: "RAW({{?antInclude}})"` | Comma-separated 
wildcards on the path below `directoryName`, case-sensitive |
   | `preMove` | `preMove: "{{?preMove}}"` | Moves the file before it is read; 
`move` then acts from there (`.inprogress/.done`) |
   | `readLock` | `readLock: "{{?readLock}}"` | `none`, `changed` or `rename` 
(enum: the only values the FTP and SFTP components act on) |
   | `readLockMinAge` | `readLockMinAge: "{{?readLockMinAge}}"` | With 
`changed`: the minimum age of a file, in milliseconds |
   | `delay` | `delay: "{{?delay}}"` | Milliseconds between two polls |
   | `maxMessagesPerPoll` | `maxMessagesPerPoll: "{{?maxMessagesPerPoll}}"` | 
The most files read in one poll |
   | `sortBy` | `sortBy: "RAW({{?sortBy}})"` | `file:name`, 
`reverse:file:modified`, groups separated by `;` |
   | `streamDownload` | `streamDownload: "{{?streamDownload}}"` | Reads the 
file while it is handed on, instead of into memory first |
   | `connectTimeout` | `connectTimeout: "{{?connectTimeout}}"` | Milliseconds 
to wait for the connection (and, on FTP, the first answer) |
   | `socketTimeout` | `soTimeout: "{{?socketTimeout}}"` | FTP: milliseconds to 
wait for the answer to a command. SFTP: how long the session may stay without 
an answer |
   
   `ftp-source` and `ftps-source` only:
   
   | Option | Template mapping | What it does |
   |---|---|---|
   | `dataTimeout` | `timeout: "{{?dataTimeout}}"` | Milliseconds to wait for 
data while a file or a listing is transferred |
   | `stepwise` | `stepwise: "{{?stepwise}}"` | Must be false with 
`streamDownload` (see below) |
   
   Others:
   
   | Kamelet | Option | Template mapping |
   |---|---|---|
   | `ftps-source`, `ftps-sink` | `implicit` | `implicit: "{{?implicit}}"` |
   | `sftp-source`, `sftp-sink` | `serverAliveInterval` | `serverAliveInterval: 
"{{?serverAliveInterval}}"` |
   | `sftp-sink` | `chmod` (pattern `^[0-7]{3}$`) | `chmod: "{{?chmod}}"` |
   | `scp-sink` | `directoryName` (pattern `^[A-Za-z0-9_./~@+=,-]*$`) | 
`scp:{{serverName}}:{{serverPort}}/{{?directoryName}}?...` |
   | `scp-sink` | `chmod` (pattern `^[0-7]{3}$`) | `...?chmod={{?chmod}}&...` |
   | `file-watch-source` | `antInclude` | `antInclude: "RAW({{?antInclude}})"` |
   | `file-watch-source` | `recursive` (default `true`) | `recursive: 
"{{recursive}}"` |
   
   - **Defaults:** every new option is an optional placeholder without a 
`default`, as in #3174, so with the options unset the endpoint URIs are 
byte-identical to `main`'s for `ftp-source`, `ftps-source`, `sftp-source`, 
`ftps-sink` and `sftp-sink`. Two exceptions, both with the same behaviour as 
before: `file-watch-source` gains `recursive=true` (the component's default, 
and what the task asked for), and the `scp-sink` URI path gains a trailing `/` 
(`scp://host:2233/?...`), which the component strips, so the files still land 
in the home directory.
   - **Names:** `timeout` is taken by the `kamelet:` component (#3201), so the 
FTP data timeout is `dataTimeout`, and the socket timeout is `socketTimeout` 
for consistency.
   - **`RAW()`:** for the regular expressions, wildcards and `sortBy`. A `+` 
and a space in a regular expression reached the component as written (`a+\.csv` 
matched `aaa.csv`; `a b\.csv|b\.csv` matched both).
   - **`streamDownload` needs `stepwise` false on FTP and FTPS:** the FTP 
component throws `The option stepwise is not supported for stream downloading` 
for every file otherwise, so `stepwise` is added too and the descriptions say 
so. SFTP has no such restriction.
   - **`streamDownload` alone does not save memory:** Camel caches a stream 
body in memory at the start of the route, so a 400 MB file still ran out of a 
256 MB heap. It works with `camel.main.streamCachingSpoolEnabled=true` (the 
body is spooled to a file) or with stream caching off. The descriptions and the 
partial pages say so.
   - **`readLockMinAge` on FTP:** FTP servers list the modification time to the 
minute and in their own time zone. A vsftpd on UTC seen from an integration on 
CET made a fresh file look one hour old, so it was read at once; with the 
integration on UTC it was read 28 s after it was written (the minute boundary). 
The descriptions say so; SFTP is exact. `sortBy` `file:modified` is also only 
to the minute on FTP.
   - **`maxMessagesPerPoll` with `sortBy`:** the limit is applied first, so a 
poll takes the first files the server lists and sorts only these. Said in both 
descriptions.
   - **`sftp-source` template fix:** its `setBody: ${body.getBody()}` failed 
with `preMove` (the body is then bytes: `Method with name: getBody() not found 
on bean: [B`) and with `streamDownload` (a cached stream). It now unwraps the 
body only when it is a `GenericFile`; without the new options the body is bytes 
as before.
   - **`scp-sink` `directoryName`:** it is not created. When it is missing, SCP 
writes each file as a file with the name of the directory, which the 
descriptions say. Subdirectories in `CamelFileName` are created, and a name 
that points outside the directory still fails.
   - **`scp-sink` `chmod`:** it applies to new files only, and the server umask 
applies: the default `664` gave `644`, `666` gave `644`, `640` gave `640`; an 
existing file kept its `600`. `sftp-sink` `chmod` is set after the upload, also 
on an existing file.
   - **Left out:** the SFTP proxy, which needs a `com.jcraft.jsch.Proxy` bean. 
`connectTimeout` and the socket and data timeouts were not added to the sinks, 
as the issue lists them for the sources.
   - **Descriptions:** each new option says what it does, its unit and what 
happens when it is not set; "Good to know" points to the filters and 
`readLock`; `ftps-*` no longer say implicit TLS is unsupported; `scp-sink` and 
`file-watch-source` say where files go and what is watched. Each partial page 
has new examples (filters and files still being written, `preMove`, large 
files, timeouts, implicit TLS, file permissions, a file pattern).
   
   ### Verification
   
   Tested with Camel JBang 4.24.0-SNAPSHOT:
   - FTP: `delfer/alpine-ftp-server` (vsftpd).
   - FTPS explicit: the same image with TLS (`require_ssl_reuse=NO`); FTPS 
implicit: the same with `implicit_ssl=YES` on port 990.
   - SFTP: `atmoz/sftp`. SCP: `linuxserver/openssh-server`.
   - Timeouts: a non-routable address for the connect timeout, `docker pause` 
of the server for the socket timeout and `serverAliveInterval`, and a small 
fake FTP server that accepts the data connection but never sends, for the data 
timeout.
   - `file-watch-source`: a local directory.
   
   | Case | Result |
   |---|---|
   | Options unset, `ftp-source`, `ftps-source`, `sftp-source`, `ftps-sink`, 
`sftp-sink`, `main` and this branch | `camel get endpoint` shows identical URIs 
|
   | Unset, `file-watch-source` | Branch adds `recursive=true`; same events as 
`main` (CREATE of `top.csv`, `notes.txt`, `sub`, `sub/deep.csv`, `old/old.csv`) 
|
   | Unset, `scp-sink` | Branch URI `scp://127.0.0.1:2233/?...`, `main` 
`scp://127.0.0.1:2233?...`; both write to the home directory |
   | All options set, every Kamelet | Each option is on the endpoint URI under 
its component name |
   | `include: .*\.csv`, recursive, FTP / SFTP | `orders.csv`, `ORDERS2.CSV`, 
`sub/deep.csv`; not `notes.txt`, `data.xml`, `part.csv.tmp` |
   | `exclude: .*\.(tmp\|txt)`, FTP / SFTP | `orders.csv`, `ORDERS2.CSV`, 
`data.xml`, `sub/deep.csv` |
   | `include: .*\.csv` + `exclude: orders2.*`, FTP | `orders.csv` only 
(`ORDERS2.CSV` excluded whatever the case) |
   | `antInclude: *.csv,*.xml`, recursive, FTP / SFTP | `orders.csv`, 
`data.xml`; not `ORDERS2.CSV` (case) nor `sub/deep.csv` |
   | `antInclude: **/*.csv`, recursive, FTP / SFTP | `orders.csv`, 
`sub/deep.csv` |
   | `antInclude: *.txt`, FTPS explicit; `include: .*\.csv`, FTPS implicit | 
`b.txt`; `a.csv` |
   | `include: a+\.csv` + `exclude: a b\.csv`; `include: a b\.csv\|b\.csv` | 
`aaa.csv`; `a b.csv` and `b.csv` |
   | `preMove: .inprogress` + `move: .done`, FTP / SFTP / FTPS | Files read, 
then in `.inprogress/.done/` |
   | `preMove: .inprogress` only, FTP | Files read, then in `.inprogress/` |
   | Five files, unset, FTP | Read in the server's (alphabetical) order |
   | `sortBy: reverse:file:name`, FTP | `e d c b a` |
   | `sortBy: reverse:file:modified`, files written 1 s apart over two minutes, 
FTP | Newest minute first, alphabetical within a minute (`b d e a c`) |
   | `sortBy: reverse:file:name` + `maxMessagesPerPoll: 2` + `delay: 3000`, FTP 
| Polls 3 s apart: `b a`, `d c`, `e` (the limit picks the first listed, then 
sorts) |
   | `sortBy: reverse:file:modified` + `maxMessagesPerPoll: 3`, SFTP | `d a c`, 
then `b e` (the same) |
   | `delay` unset / `3000`, `idempotent: false`, FTP | The same file read 
every ~0.52 s / ~3.02 s |
   | `readLock: changed`, FTP | `a.csv` read ~6 s after start; the empty file 
never in 30 s |
   | `readLock: rename`, FTP | File read |
   | `readLock: changed` + `readLockMinAge: 60000`, SFTP, `fresh.csv` and a 
5-minute-old `old.csv` | `old.csv` after the 20 s skip of `fresh.csv`; 
`fresh.csv` exactly 60 s after it was written |
   | The same, FTP (server UTC, integration CET) | Both read at once: the 
listed time is parsed one hour too old |
   | The same, FTP, integration on UTC | `fresh.csv` read 28 s after it was 
written (listed to the minute) |
   | `streamDownload: true`, FTP, stepwise unset | Every file fails: `The 
option stepwise is not supported for stream downloading` |
   | `streamDownload` + `stepwise: false` + `recursive` + `move`, FTP | Both 
files read and moved; the body is a stream |
   | 400 MB file, 256 MB heap, FTP: unset / stream / stream + spool / stream + 
caching off | Out of memory / out of memory / read (`FileInputStreamCache`) / 
read (`SocketInputStream`) |
   | 400 MB file, 256 MB heap, SFTP: unset / stream / stream + spool | Out of 
memory / out of memory / read |
   | `streamDownload`, SFTP | Body is a stream; without it, bytes as on `main` |
   | Many options together, FTPS implicit (`exclude`, `preMove`, `move`, 
`sortBy`, `maxMessagesPerPoll: 2`, `streamDownload`, `stepwise: false`, 
`readLock: changed`) | `b a`, `d c`, `e`, ~5 s apart each; `x.tmp` left; files 
in `.inprogress/.done` |
   | `connectTimeout: 2000` / unset, unroutable address, FTP | Poll skipped 
after 10 s / 34 s (3 attempts) |
   | The same, SFTP | 10 s / 34 s |
   | Silent server that never greets, FTP | Each attempt times out after 
`connectTimeout` (`Timed out waiting for initial connect reply`), not 
`socketTimeout` |
   | `socketTimeout: 3000` / unset, FTP server paused | `Read timed out` 3 s 
after the pause / nothing in 45 s |
   | `dataTimeout: 3000` / unset, data connection that never sends, FTP | `Read 
timed out` 3.7 s / 30 s after `LIST` |
   | `socketTimeout: 3000` / `serverAliveInterval: 2000` / unset, SFTP server 
paused | Poll fails 6 s / 4.5 s after the pause / nothing in 40 s |
   | `serverAliveInterval: 2000` / unset, `sftp-sink`, two messages 25 s apart, 
server paused between them | Second message: new connection, written / old 
connection reused, written |
   | `implicit: true`, port 990, `ftps-source` / `ftps-sink` | File read / 
`up/orders.csv` written |
   | Implicit unset, port 990 | `Timed out waiting for initial connect reply` |
   | `implicit: true`, explicit server on port 21 | `Unsupported or 
unrecognized SSL message` |
   | `chmod: 640` / unset / `644` over an existing `600` file, `sftp-sink` | 
`-rw-r-----` / `-rw-r--r--` / `-rw-r--r--` |
   | `scp-sink` `directoryName: incoming` + `chmod: 640` | `incoming/dir.csv`, 
`-rw-r-----` |
   | `scp-sink` `directoryName: /tmp/abs` | `/tmp/abs/abs.csv` |
   | `scp-sink` `directoryName: missing` (not there) | A file named `missing` 
in the home directory, with the content |
   | `scp-sink` `directoryName: incoming`, `CamelFileName: sub/deep.csv` | 
`incoming/sub/deep.csv`, `sub` created |
   | `scp-sink` `CamelFileName: ../escape.csv` | Fails: `Cannot write file with 
name: ../escape.csv`; nothing written |
   | `scp-sink` `chmod: 666` / unset / `644` over an existing `600` file | 
`-rw-r--r--` (umask 022) / `-rw-r--r--` / stays `-rw-------` |
   | `file-watch-source` `antInclude: *.csv` | `top.csv` only |
   | `file-watch-source` `antInclude: **/*.csv` | `top.csv`, `sub/deep.csv`, 
`old/old.csv` |
   | `file-watch-source` `recursive: false` | `top.csv`, `notes.txt`, `sub`; 
not `sub/deep.csv`, `old/old.csv` |
   
   `./mvnw clean verify -DskipTests` and then `./mvnw clean verify` passed from 
the root. Before building, the library copies of the seven Kamelets were 
refreshed with `./mvnw -pl library/camel-kamelets 
resources:copy-resources@copy-resource-one`. The build regenerated 
`kamelets-index.json`. `KameletsCatalogTest` needed no change (no headers were 
added).
   
   _Claude Code on behalf of Claus Ibsen_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to