This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel-kamelets.git
The following commit(s) were added to refs/heads/main by this push:
new 6826af9df chore: the file Kamelets can filter, wait for complete
files, stream large files and set timeouts (#3170) (#3221)
6826af9df is described below
commit 6826af9df71af26922f49f988a24c91aefdfe2b1
Author: Claus Ibsen <[email protected]>
AuthorDate: Wed Nov 18 22:33:19 2026 +0100
chore: the file Kamelets can filter, wait for complete files, stream large
files and set timeouts (#3170) (#3221)
ftp-source, ftps-source and sftp-source take include, exclude and
antInclude to read only some files; preMove; readLock (none, changed,
rename) and readLockMinAge to leave a file alone while it is still being
written; delay, maxMessagesPerPoll and sortBy; streamDownload; and
connectTimeout plus a socket timeout. The timeouts are named
socketTimeout (soTimeout) and, on FTP and FTPS, dataTimeout (timeout),
since timeout is taken by the kamelet: component. ftp-source and
ftps-source also take stepwise, because the FTP component refuses
streamDownload while stepwise is true.
ftps-source and ftps-sink take implicit, for servers that use implicit
TLS on port 990. sftp-source and sftp-sink take serverAliveInterval, and
sftp-sink takes chmod. scp-sink takes directoryName and chmod; it already
had knownHostsUri, mapped to knownHostsFile. file-watch-source takes
antInclude and recursive, which defaults to true as before.
All the new options are optional placeholders without a default, so the
endpoints are unchanged when they are not set, except file-watch-source,
whose URI now has recursive=true, and scp-sink, whose URI path now ends
with a slash; both behave as before.
sftp-source called getBody() on the body, which fails with preMove (the
body is then bytes) and with streamDownload (a cached stream); it now
unwraps the body only when it is a remote file.
The SFTP proxy is left out: it needs a bean.
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
.../partials/file-watch-source-description.adoc | 28 +++-
.../ROOT/partials/ftp-source-description.adoc | 130 ++++++++++++++++-
.../ROOT/partials/ftps-sink-description.adoc | 29 +++-
.../ROOT/partials/ftps-source-description.adoc | 157 ++++++++++++++++++++-
.../ROOT/partials/scp-sink-description.adoc | 33 ++++-
.../ROOT/partials/sftp-sink-description.adoc | 26 ++++
.../ROOT/partials/sftp-source-description.adoc | 123 +++++++++++++++-
kamelets/file-watch-source.kamelet.yaml | 16 ++-
kamelets/ftp-source.kamelet.yaml | 85 ++++++++++-
kamelets/ftps-sink.kamelet.yaml | 9 +-
kamelets/ftps-source.kamelet.yaml | 92 +++++++++++-
kamelets/scp-sink.kamelet.yaml | 20 ++-
kamelets/sftp-sink.kamelet.yaml | 13 ++
kamelets/sftp-source.kamelet.yaml | 96 ++++++++++++-
.../main/resources/META-INF/kamelets-index.json | 14 +-
.../kamelets/file-watch-source.kamelet.yaml | 16 ++-
.../resources/kamelets/ftp-source.kamelet.yaml | 85 ++++++++++-
.../main/resources/kamelets/ftps-sink.kamelet.yaml | 9 +-
.../resources/kamelets/ftps-source.kamelet.yaml | 92 +++++++++++-
.../main/resources/kamelets/scp-sink.kamelet.yaml | 20 ++-
.../main/resources/kamelets/sftp-sink.kamelet.yaml | 13 ++
.../resources/kamelets/sftp-source.kamelet.yaml | 96 ++++++++++++-
22 files changed, 1151 insertions(+), 51 deletions(-)
diff --git a/docs/modules/ROOT/partials/file-watch-source-description.adoc
b/docs/modules/ROOT/partials/file-watch-source-description.adoc
index 9ed845900..22f988471 100644
--- a/docs/modules/ROOT/partials/file-watch-source-description.adoc
+++ b/docs/modules/ROOT/partials/file-watch-source-description.adoc
@@ -6,6 +6,8 @@ The File Watch Source Kamelet monitors files or directories for
changes and emit
- **File Path**: The path of the directory to monitor (required). It must be a
directory, not a single file; it is created when it does not exist
- **Events**: The types of filesystem events to watch for (default:
"CREATE,MODIFY,DELETE")
+- **Include Files (Wildcards)**: A wildcard pattern for the path of the files
whose changes to send, such as `**/*.csv` (not set by default: all files)
+- **Recursive**: Whether the changes in the subdirectories are sent too
(default: true)
=== Event Types
@@ -19,7 +21,7 @@ You can specify multiple events by separating them with
commas.
=== Output Format
-The body of each message is the file that changed. The `CamelFileEventType`
header holds the kind of event (`CREATE`, `MODIFY` or `DELETE`),
`CamelFileName` the path relative to the watched directory, and
`CamelFileAbsolutePath` the full path. Subdirectories are watched too. A
deleted file, or a directory, has no content to read.
+The body of each message is the file that changed. The `CamelFileEventType`
header holds the kind of event (`CREATE`, `MODIFY` or `DELETE`),
`CamelFileName` the path relative to the watched directory, and
`CamelFileAbsolutePath` the full path. Subdirectories are watched too, unless
**Recursive** is false. A deleted file, or a directory, has no content to read.
=== Volume Mounting
@@ -72,4 +74,26 @@ This example only monitors for modifications of the files in
the `/var/log/myapp
uri: "kamelet:log-sink"
----
-This example monitors for file creation and deletion events in the documents
directory.
\ No newline at end of file
+This example monitors for file creation and deletion events in the documents
directory.
+
+=== Example with a File Pattern
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:file-watch-source"
+ parameters:
+ filePath: "/data/incoming"
+ events: "CREATE"
+ antInclude: "**/*.csv"
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
+This sends a message for each new CSV file in `/data/incoming` and its
subdirectories, and for no other file or
+directory. The pattern matches the path below `filePath`, such as
`2026/orders.csv`: `*` matches any part of a name and
+`**` any number of directories, so `*.csv` alone matches only the files
directly in `/data/incoming`. Upper and lower
+case differ. With `recursive: false`, only the changes directly in `filePath`
are sent, which includes a new
+subdirectory itself but not the files in it.
diff --git a/docs/modules/ROOT/partials/ftp-source-description.adoc
b/docs/modules/ROOT/partials/ftp-source-description.adoc
index 8d7d66bad..e693e8f21 100644
--- a/docs/modules/ROOT/partials/ftp-source-description.adoc
+++ b/docs/modules/ROOT/partials/ftp-source-description.adoc
@@ -21,6 +21,20 @@ The FTP Source Kamelet supports the following configurations:
- **Delete**: Delete files after successful processing (default: false). When
false, and **Move** is not set, files are left on the server; with
**Idempotent**, they are skipped while the integration runs but read again
after a restart
- **Move**: Where to move each file on the server after it has been read, such
as `.done`, so that it is not read again, also after a restart (not set by
default). It cannot be combined with **Delete**
- **Move Failed**: Where to move a file on the server when it could not be
handed on, such as `.error` (not set by default). When not set, a file that
fails is read again at every poll
+- **Include Files**: A regular expression that the name of a file must match
to be read, such as `.*\.csv`; upper and lower case count as the same (not set
by default)
+- **Exclude Files**: A regular expression for the names of the files not to
read; it wins over **Include Files** and **Include Files (Wildcards)** (not set
by default)
+- **Include Files (Wildcards)**: Wildcard patterns for the path below the
directory, such as `*.csv` or `**/*.csv`; upper and lower case differ (not set
by default)
+- **Move Before Read**: Where to move each file before it is read, such as
`.inprogress` (not set by default)
+- **Wait Until Written**: `changed` reads a file only once it has stopped
changing, `rename` once it can be renamed (default: none)
+- **Minimum File Age**: With `changed`, how old a file must be before it is
read, in milliseconds (not set by default)
+- **Poll Interval**: How long to wait between two polls, in milliseconds
(default: 500)
+- **Max Files Per Poll**: The largest number of files read in one poll (not
set by default: all)
+- **Sort By**: The order of the files in a poll, such as `file:modified` or
`reverse:file:name` (not set by default: the order of the server)
+- **Stream Download**: Read each file from the server while it is handed on,
instead of into memory first (default: false)
+- **Change Directory Stepwise**: Change into the directory of a file one level
at a time; must be false with **Stream Download** (default: true)
+- **Connect Timeout**: How long to wait for the connection, in milliseconds
(default: 10000)
+- **Socket Timeout**: How long to wait for the answer to a command, in
milliseconds (default: 300000)
+- **Data Timeout**: How long to wait for data while a file or a listing is
transferred, in milliseconds (default: 30000)
=== Output Format
@@ -138,4 +152,118 @@ put on the server again under the same name is not read
again.
uri: "kamelet:log-sink"
----
-This example deletes files after processing and disables idempotency to allow
re-processing of files with the same name.
\ No newline at end of file
+This example deletes files after processing and disables idempotency to allow
re-processing of files with the same name.
+
+=== Example with File Filters and Files Still Being Written
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:ftp-source"
+ parameters:
+ connectionHost: "ftp.example.com"
+ username: "ftpuser"
+ password: "ftppass"
+ directoryName: "/incoming"
+ recursive: true
+ antInclude: "**/*.csv"
+ exclude: "draft-.*"
+ readLock: "changed"
+ sortBy: "file:modified"
+ maxMessagesPerPoll: 100
+ delay: 60000
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
+Once a minute, this reads the CSV files in `/incoming` and its subdirectories,
except those whose names start with
+`draft-`, at most 100 of them in each poll, the oldest first. **Include Files
(Wildcards)** matches the path below the
+directory, and upper and lower case differ: `*.csv` alone would match only the
files directly in `/incoming`.
+**Include Files** and **Exclude Files** are regular expressions that match the
name of the file alone, whatever the
+case. With **Max Files Per Poll**, a poll takes the first files the server
lists and sorts only these.
+
+`readLock: changed` reads a file only once its size and modification time have
not changed for 5 seconds, so a file
+that another program is still uploading is left alone. It never reads an empty
file, and while it checks a file the
+other files wait. With **Minimum File Age**, a file is read only once it is
that old instead. FTP servers list the
+modification time only to the minute and in their own time zone, so use
**Minimum File Age** only when the server and
+the integration are in the same time zone; otherwise every file looks older or
younger by the difference. **Sort By**
+`file:modified` is also only to the minute.
+
+The files a poll reads can be moved out of the way first with **Move Before
Read**, so that a second integration
+reading the same directory does not take them too:
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:ftp-source"
+ parameters:
+ connectionHost: "ftp.example.com"
+ username: "ftpuser"
+ password: "ftppass"
+ directoryName: "/incoming"
+ preMove: ".inprogress"
+ move: ".done"
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
+Each file is moved to `.inprogress` before it is read, and from there to
`.done` once it has been handed on, so it
+ends up in `/incoming/.inprogress/.done`.
+
+=== Example with Large Files
+
+By default each file is downloaded into memory before it is handed on. For
files too large for that, set
+**Stream Download**:
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:ftp-source"
+ parameters:
+ connectionHost: "ftp.example.com"
+ username: "ftpuser"
+ password: "ftppass"
+ directoryName: "/exports"
+ streamDownload: true
+ stepwise: false
+ binary: true
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
+Camel keeps a message body that is a stream in memory too, unless told
otherwise, so also set
+`camel.main.streamCachingSpoolEnabled=true` in the application properties:
large bodies are then written to a
+temporary file. In a test with 256 MB of memory, a 400 MB file failed with an
out of memory error without
+**Stream Download**, and with it but without spooling, and was read with both.
**Stream Download** needs
+**Change Directory Stepwise** set to false, or every file fails.
+
+=== Example with Timeouts
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:ftp-source"
+ parameters:
+ connectionHost: "ftp.example.com"
+ username: "ftpuser"
+ password: "ftppass"
+ directoryName: "/incoming"
+ connectTimeout: 5000
+ socketTimeout: 60000
+ dataTimeout: 30000
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
+**Connect Timeout** is how long to wait for the connection and the first
answer of the server; a connection that
+fails is tried 3 times before the poll is skipped. **Socket Timeout** is how
long to wait for the answer to a command,
+5 minutes by default, and **Data Timeout** how long to wait for data while a
file or the list of the files is
+transferred. When one of them runs out, the poll fails and the next poll
connects again.
diff --git a/docs/modules/ROOT/partials/ftps-sink-description.adoc
b/docs/modules/ROOT/partials/ftps-sink-description.adoc
index 037f3221c..47607b1b5 100644
--- a/docs/modules/ROOT/partials/ftps-sink-description.adoc
+++ b/docs/modules/ROOT/partials/ftps-sink-description.adoc
@@ -12,7 +12,7 @@ FTPS provides secure file transfer over encrypted connections
using SSL/TLS prot
Requires:
- Connection host (FTPS server hostname)
-- Connection port (defaults to 21)
+- Connection port (defaults to 21; usually 990 with implicit TLS)
- Username and password credentials
- Directory name for file operations
@@ -24,6 +24,7 @@ Requires:
- **Auto-create Directories**: Automatically creates missing directories
(enabled by default)
- **Temporary Name Prefix**: Upload each file under a temporary name, such as
`.tmp-orders.csv`, and rename it to its real name once it is complete (not set
by default)
- **Done File Name**: Write an empty done file next to each file once it is
complete, such as `orders.csv.done` (not set by default)
+- **Implicit TLS**: Use implicit TLS, usually on port 990, instead of explicit
TLS on the normal FTP port (default: false)
=== Optional Headers
@@ -70,3 +71,29 @@ with a dot, so it never reads a half-written file; another
reader can wait for t
`${file:name}` stands for the name of the file and `${file:name.noext}` for
the name without its extension; no other
`${...}` is allowed. The user needs the right to rename files on the server.
**tempPrefix** cannot be combined with
**fileExist** set to `Append`, and with `Override` a file of the same name is
deleted when the upload starts.
+
+=== Example with Implicit TLS
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:timer-source"
+ parameters:
+ message: "Hello"
+ steps:
+ - to:
+ uri: "kamelet:ftps-sink"
+ parameters:
+ connectionHost: "ftps.example.com"
+ connectionPort: "990"
+ username: "ftpsuser"
+ password: "ftpspass"
+ directoryName: "/upload"
+ implicit: true
+ passiveMode: true
+----
+
+By default the Kamelet uses explicit TLS: it connects to the normal FTP port,
usually 21, and then switches to TLS.
+A server that uses implicit TLS, usually on port 990, expects an encrypted
connection from the start: set
+**Implicit TLS** for it.
diff --git a/docs/modules/ROOT/partials/ftps-source-description.adoc
b/docs/modules/ROOT/partials/ftps-source-description.adoc
index 7cc128259..b5e20b151 100644
--- a/docs/modules/ROOT/partials/ftps-source-description.adoc
+++ b/docs/modules/ROOT/partials/ftps-source-description.adoc
@@ -22,6 +22,21 @@ The FTPS Source Kamelet supports the following
configurations:
- **Delete**: Delete files after successful processing (default: false)
- **Move**: Where to move each file on the server after it has been read, such
as `.done`, so that it is not read again, also after a restart (not set by
default). It cannot be combined with **Delete**
- **Move Failed**: Where to move a file on the server when it could not be
handed on, such as `.error` (not set by default). When not set, a file that
fails is read again at every poll
+- **Include Files**: A regular expression that the name of a file must match
to be read, such as `.*\.csv`; upper and lower case count as the same (not set
by default)
+- **Exclude Files**: A regular expression for the names of the files not to
read; it wins over **Include Files** and **Include Files (Wildcards)** (not set
by default)
+- **Include Files (Wildcards)**: Wildcard patterns for the path below the
directory, such as `*.csv` or `**/*.csv`; upper and lower case differ (not set
by default)
+- **Move Before Read**: Where to move each file before it is read, such as
`.inprogress` (not set by default)
+- **Wait Until Written**: `changed` reads a file only once it has stopped
changing, `rename` once it can be renamed (default: none)
+- **Minimum File Age**: With `changed`, how old a file must be before it is
read, in milliseconds (not set by default)
+- **Poll Interval**: How long to wait between two polls, in milliseconds
(default: 500)
+- **Max Files Per Poll**: The largest number of files read in one poll (not
set by default: all)
+- **Sort By**: The order of the files in a poll, such as `file:modified` or
`reverse:file:name` (not set by default: the order of the server)
+- **Stream Download**: Read each file from the server while it is handed on,
instead of into memory first (default: false)
+- **Change Directory Stepwise**: Change into the directory of a file one level
at a time; must be false with **Stream Download** (default: true)
+- **Connect Timeout**: How long to wait for the connection, in milliseconds
(default: 10000)
+- **Socket Timeout**: How long to wait for the answer to a command, in
milliseconds (default: 300000)
+- **Data Timeout**: How long to wait for data while a file or a listing is
transferred, in milliseconds (default: 30000)
+- **Implicit TLS**: Use implicit TLS, usually on port 990, instead of explicit
TLS (default: false)
=== Output Format
@@ -31,7 +46,7 @@ The Kamelet outputs file content as an `InputStream` and sets
headers with file
=== Security
-FTPS encrypts the connection of standard FTP with TLS. The Kamelet uses
explicit FTPS (it switches to TLS on the normal FTP port).
+FTPS encrypts the connection of standard FTP with TLS. By default the Kamelet
uses explicit FTPS: it switches to TLS on the normal FTP port. Set **Implicit
TLS** for a server that uses implicit FTPS, usually on port 990.
=== Usage Example
@@ -72,7 +87,7 @@ FTPS encrypts the connection of standard FTP with TLS. The
Kamelet uses explicit
uri: "kamelet:log-sink"
----
-This example enables passive mode, binary transfer, and recursive directory
processing. The Kamelet uses explicit FTPS: it connects to the normal FTP port
and then switches to TLS. Servers that only accept implicit FTPS (usually on
port 990) are not supported.
+This example enables passive mode, binary transfer, and recursive directory
processing. The Kamelet uses explicit FTPS here: it connects to the normal FTP
port and then switches to TLS.
=== Example with Files Moved After Reading
@@ -124,3 +139,141 @@ read in a file. Declare a `FileIdempotentRepository` bean
that writes the list t
The file must be on storage that survives a restart of the integration, such
as a persistent volume. A file that is
put on the server again under the same name is not read again.
+
+=== Example with File Filters and Files Still Being Written
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:ftps-source"
+ parameters:
+ connectionHost: "ftps.example.com"
+ username: "ftpsuser"
+ password: "ftpspass"
+ directoryName: "/incoming"
+ recursive: true
+ antInclude: "**/*.csv"
+ exclude: "draft-.*"
+ readLock: "changed"
+ sortBy: "file:modified"
+ maxMessagesPerPoll: 100
+ delay: 60000
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
+Once a minute, this reads the CSV files in `/incoming` and its subdirectories,
except those whose names start with
+`draft-`, at most 100 of them in each poll, the oldest first. **Include Files
(Wildcards)** matches the path below the
+directory, and upper and lower case differ: `*.csv` alone would match only the
files directly in `/incoming`.
+**Include Files** and **Exclude Files** are regular expressions that match the
name of the file alone, whatever the
+case. With **Max Files Per Poll**, a poll takes the first files the server
lists and sorts only these.
+
+`readLock: changed` reads a file only once its size and modification time have
not changed for 5 seconds, so a file
+that another program is still uploading is left alone. It never reads an empty
file, and while it checks a file the
+other files wait. With **Minimum File Age**, a file is read only once it is
that old instead. FTP servers list the
+modification time only to the minute and in their own time zone, so use
**Minimum File Age** only when the server and
+the integration are in the same time zone; otherwise every file looks older or
younger by the difference. **Sort By**
+`file:modified` is also only to the minute.
+
+The files a poll reads can be moved out of the way first with **Move Before
Read**, so that a second integration
+reading the same directory does not take them too:
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:ftps-source"
+ parameters:
+ connectionHost: "ftps.example.com"
+ username: "ftpsuser"
+ password: "ftpspass"
+ directoryName: "/incoming"
+ preMove: ".inprogress"
+ move: ".done"
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
+Each file is moved to `.inprogress` before it is read, and from there to
`.done` once it has been handed on, so it
+ends up in `/incoming/.inprogress/.done`.
+
+=== Example with Large Files
+
+By default each file is downloaded into memory before it is handed on. For
files too large for that, set
+**Stream Download**:
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:ftps-source"
+ parameters:
+ connectionHost: "ftps.example.com"
+ username: "ftpsuser"
+ password: "ftpspass"
+ directoryName: "/exports"
+ streamDownload: true
+ stepwise: false
+ binary: true
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
+Camel keeps a message body that is a stream in memory too, unless told
otherwise, so also set
+`camel.main.streamCachingSpoolEnabled=true` in the application properties:
large bodies are then written to a
+temporary file. In a test with 256 MB of memory, a 400 MB file failed with an
out of memory error without
+**Stream Download**, and with it but without spooling, and was read with both.
**Stream Download** needs
+**Change Directory Stepwise** set to false, or every file fails.
+
+=== Example with Timeouts
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:ftps-source"
+ parameters:
+ connectionHost: "ftps.example.com"
+ username: "ftpsuser"
+ password: "ftpspass"
+ directoryName: "/incoming"
+ connectTimeout: 5000
+ socketTimeout: 60000
+ dataTimeout: 30000
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
+**Connect Timeout** is how long to wait for the connection and the first
answer of the server; a connection that
+fails is tried 3 times before the poll is skipped. **Socket Timeout** is how
long to wait for the answer to a command,
+5 minutes by default, and **Data Timeout** how long to wait for data while a
file or the list of the files is
+transferred. When one of them runs out, the poll fails and the next poll
connects again.
+
+=== Example with Implicit TLS
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:ftps-source"
+ parameters:
+ connectionHost: "ftps.example.com"
+ connectionPort: "990"
+ username: "ftpsuser"
+ password: "ftpspass"
+ directoryName: "/incoming"
+ implicit: true
+ passiveMode: true
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
+With implicit TLS the connection is encrypted from the start, usually on port
990. Use the mode the server expects:
+explicit TLS against an implicit server waits for an answer until **Connect
Timeout** runs out, and implicit TLS
+against an explicit server fails at once.
diff --git a/docs/modules/ROOT/partials/scp-sink-description.adoc
b/docs/modules/ROOT/partials/scp-sink-description.adoc
index 454183797..51b1c0b30 100644
--- a/docs/modules/ROOT/partials/scp-sink-description.adoc
+++ b/docs/modules/ROOT/partials/scp-sink-description.adoc
@@ -10,7 +10,7 @@ Uses SSH protocol for authentication and encryption,
providing strong security f
=== Remote File Placement
-Transfers files to the home directory of `username` on the remote server. The
file name comes from the `CamelFileName` header, which may include
subdirectories; without it, a unique name is generated. A file with the same
name is replaced. To choose another directory on the server, use the SFTP Sink
Kamelet.
+Transfers files to `directoryName` on the remote server, or to the home
directory of `username` when it is not set. A path without a leading `/` is
relative to the home directory. The directory must exist: when it does not,
each file is written as a file with the name of the directory. The file name
comes from the `CamelFileName` header, which may include subdirectories, which
are created; without it, a unique name is generated. A file with the same name
is replaced. To create a missing [...]
=== Network Efficiency
@@ -31,4 +31,33 @@ Supports multiple authentication methods:
=== Cross-Platform Support
-Works across different operating systems and platforms, providing consistent
secure file transfer capabilities regardless of the target system architecture.
\ No newline at end of file
+Works across different operating systems and platforms, providing consistent
secure file transfer capabilities regardless of the target system architecture.
+
+=== Usage Example
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:timer-source"
+ parameters:
+ message: "Hello"
+ steps:
+ - setHeader:
+ name: CamelFileName
+ constant: "hello.txt"
+ - to:
+ uri: "kamelet:scp-sink"
+ parameters:
+ serverName: "files.example.com"
+ serverPort: "22"
+ username: "scpuser"
+ password: "scppass"
+ knownHostsUri: "/etc/scp/known_hosts"
+ directoryName: "incoming/reports"
+ chmod: "640"
+----
+
+This copies `hello.txt` to `incoming/reports` in the home directory of
`scpuser`, with the permissions `640`. The
+umask of the server can take permissions away: with the usual umask 022, the
default `664` gives `644`. A file that
+already exists keeps its permissions, and only its content is replaced.
diff --git a/docs/modules/ROOT/partials/sftp-sink-description.adoc
b/docs/modules/ROOT/partials/sftp-sink-description.adoc
index b57d6fbce..49edc43e9 100644
--- a/docs/modules/ROOT/partials/sftp-sink-description.adoc
+++ b/docs/modules/ROOT/partials/sftp-sink-description.adoc
@@ -69,3 +69,29 @@ with a dot, so it never reads a half-written file; another
reader can wait for t
`${file:name}` stands for the name of the file and `${file:name.noext}` for
the name without its extension; no other
`${...}` is allowed. The user needs the right to rename files on the server.
**tempPrefix** cannot be combined with
**fileExist** set to `Append`, and with `Override` a file of the same name is
deleted when the upload starts.
+
+=== Example with File Permissions
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:timer-source"
+ parameters:
+ message: "Hello"
+ steps:
+ - to:
+ uri: "kamelet:sftp-sink"
+ parameters:
+ connectionHost: "sftp.example.com"
+ username: "sftpuser"
+ password: "sftppass"
+ directoryName: "/upload"
+ chmod: "640"
+ serverAliveInterval: 30000
+----
+
+Each file gets the permissions `640` once it is written, read and write for
its owner and read for the group, also
+when it replaced a file with other permissions. Without **File Permissions**,
a new file gets the permissions the
+server gives it, usually `644`. **Server Alive Interval** checks every 30
seconds without data that the server still
+answers, and closes the connection when it does not, so that the next message
makes a new one.
diff --git a/docs/modules/ROOT/partials/sftp-source-description.adoc
b/docs/modules/ROOT/partials/sftp-source-description.adoc
index 11cbdf038..00a0c0992 100644
--- a/docs/modules/ROOT/partials/sftp-source-description.adoc
+++ b/docs/modules/ROOT/partials/sftp-source-description.adoc
@@ -20,10 +20,23 @@ The SFTP Source Kamelet supports the following
configurations:
- **Delete**: Delete files after successful processing (default: false). When
false, and **Move** is not set, files are left on the server; with
**Idempotent**, they are skipped while the integration runs but read again
after a restart
- **Move**: Where to move each file on the server after it has been read, such
as `.done`, so that it is not read again, also after a restart (not set by
default). It cannot be combined with **Delete**
- **Move Failed**: Where to move a file on the server when it could not be
handed on, such as `.error` (not set by default). When not set, a file that
fails is read again at every poll
+- **Include Files**: A regular expression that the name of a file must match
to be read, such as `.*\.csv`; upper and lower case count as the same (not set
by default)
+- **Exclude Files**: A regular expression for the names of the files not to
read; it wins over **Include Files** and **Include Files (Wildcards)** (not set
by default)
+- **Include Files (Wildcards)**: Wildcard patterns for the path below the
directory, such as `*.csv` or `**/*.csv`; upper and lower case differ (not set
by default)
+- **Move Before Read**: Where to move each file before it is read, such as
`.inprogress` (not set by default)
+- **Wait Until Written**: `changed` reads a file only once it has stopped
changing, `rename` once it can be renamed (default: none)
+- **Minimum File Age**: With `changed`, how old a file must be before it is
read, in milliseconds (not set by default)
+- **Poll Interval**: How long to wait between two polls, in milliseconds
(default: 500)
+- **Max Files Per Poll**: The largest number of files read in one poll (not
set by default: all)
+- **Sort By**: The order of the files in a poll, such as `file:modified` or
`reverse:file:name` (not set by default: the order of the server)
+- **Stream Download**: Read each file from the server while it is handed on,
instead of into memory first (default: false)
+- **Connect Timeout**: How long to wait for the connection, in milliseconds
(default: 10000)
+- **Socket Timeout**: How long the connection may stay without an answer, in
milliseconds (default: 300000)
+- **Server Alive Interval**: How often to check that the server still answers,
in milliseconds (not set by default)
=== Output Format
-The Kamelet outputs the file content as bytes and sets headers with file
information:
+The Kamelet outputs the file content as bytes, or as a stream when **Stream
Download** is true, and sets headers with file information:
- `file`: The name of the processed file
- `ce-file`: Cloud Events compatible file name header
@@ -138,4 +151,110 @@ put on the server again under the same name is not read
again.
uri: "kamelet:log-sink"
----
-This example deletes files after processing and disables idempotency to allow
re-processing of files with the same name.
\ No newline at end of file
+This example deletes files after processing and disables idempotency to allow
re-processing of files with the same name.
+
+=== Example with File Filters and Files Still Being Written
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:sftp-source"
+ parameters:
+ connectionHost: "sftp.example.com"
+ username: "sftpuser"
+ password: "sftppass"
+ directoryName: "/incoming"
+ recursive: true
+ antInclude: "**/*.csv"
+ exclude: "draft-.*"
+ readLock: "changed"
+ sortBy: "file:modified"
+ maxMessagesPerPoll: 100
+ delay: 60000
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
+Once a minute, this reads the CSV files in `/incoming` and its subdirectories,
except those whose names start with
+`draft-`, at most 100 of them in each poll, the oldest first. **Include Files
(Wildcards)** matches the path below the
+directory, and upper and lower case differ: `*.csv` alone would match only the
files directly in `/incoming`.
+**Include Files** and **Exclude Files** are regular expressions that match the
name of the file alone, whatever the
+case. With **Max Files Per Poll**, a poll takes the first files the server
lists and sorts only these.
+
+`readLock: changed` reads a file only once its size and modification time have
not changed for 5 seconds, so a file
+that another program is still uploading is left alone. It never reads an empty
file, and while it checks a file the
+other files wait. With **Minimum File Age**, a file is read only once it is
that old instead.
+
+The files a poll reads can be moved out of the way first with **Move Before
Read**, so that a second integration
+reading the same directory does not take them too:
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:sftp-source"
+ parameters:
+ connectionHost: "sftp.example.com"
+ username: "sftpuser"
+ password: "sftppass"
+ directoryName: "/incoming"
+ preMove: ".inprogress"
+ move: ".done"
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
+Each file is moved to `.inprogress` before it is read, and from there to
`.done` once it has been handed on, so it
+ends up in `/incoming/.inprogress/.done`.
+
+=== Example with Large Files
+
+By default each file is downloaded into memory before it is handed on. For
files too large for that, set
+**Stream Download**:
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:sftp-source"
+ parameters:
+ connectionHost: "sftp.example.com"
+ username: "sftpuser"
+ password: "sftppass"
+ directoryName: "/exports"
+ streamDownload: true
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
+Camel keeps a message body that is a stream in memory too, unless told
otherwise, so also set
+`camel.main.streamCachingSpoolEnabled=true` in the application properties:
large bodies are then written to a
+temporary file. In a test with 256 MB of memory, a 400 MB file failed with an
out of memory error without
+**Stream Download**, and with it but without spooling, and was read with both.
+
+=== Example with a Connection Check
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:sftp-source"
+ parameters:
+ connectionHost: "sftp.example.com"
+ username: "sftpuser"
+ password: "sftppass"
+ directoryName: "/incoming"
+ connectTimeout: 5000
+ serverAliveInterval: 30000
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+----
+
+Every 30 seconds without data from the server, the integration checks that the
server still answers. When it does
+not, the connection is closed, the poll fails, and the next poll connects
again. Without **Server Alive Interval**, a
+connection that no longer answers is closed after **Socket Timeout**, 5
minutes by default.
diff --git a/kamelets/file-watch-source.kamelet.yaml
b/kamelets/file-watch-source.kamelet.yaml
index a70a6c352..28ad21dae 100644
--- a/kamelets/file-watch-source.kamelet.yaml
+++ b/kamelets/file-watch-source.kamelet.yaml
@@ -37,7 +37,7 @@ spec:
Output: the body is the file that changed, which the next step can read
as text or bytes; a deleted file or a directory has no content to read. The
`CamelFileEventType` header is `CREATE`, `MODIFY` or `DELETE`, `CamelFileName`
is the path of the file relative to the watched directory, and
`CamelFileAbsolutePath` is its full path.
- Good to know: it watches the subdirectories too, new ones included. It
never changes or deletes the files. Only changes made while it runs are sent,
not the files that are already there when it starts. The path must be a
directory, not a single file; it is created when it does not exist. On
Kubernetes, mount the directory into the pod as a volume.
+ Good to know: it watches the subdirectories too, new ones included,
unless `recursive` is false. To send only the changes of some files, set
`antInclude`. It never changes or deletes the files. Only changes made while it
runs are sent, not the files that are already there when it starts. The path
must be a directory, not a single file; it is created when it does not exist.
On Kubernetes, mount the directory into the pod as a volume.
required:
- filePath
- events
@@ -45,7 +45,7 @@ spec:
properties:
filePath:
title: Directory to Watch
- description: "The full path of the local directory to watch. It must
be a directory, not a single file: to follow one file, watch its directory.
Changes in its subdirectories are sent too."
+ description: "The full path of the local directory to watch. It must
be a directory, not a single file: to follow one file, watch its directory.
Changes in its subdirectories are sent too, unless `recursive` is false."
type: string
example: /data/incoming
events:
@@ -54,6 +54,16 @@ spec:
type: string
default: "CREATE,MODIFY,DELETE"
example: CREATE
+ antInclude:
+ title: Include Files (Wildcards)
+ description: "A wildcard pattern that the path of a file must match
for its changes to be sent, such as `**/*.csv`. The path is relative to
`filePath`, such as `2026/orders.csv`. `*` matches any part of a name, and `**`
any number of directories: `*.csv` matches the CSV files in `filePath` only,
and `**/*.csv` those in the subdirectories too. Upper and lower case differ.
When it is not set, the changes of all the files are sent."
+ type: string
+ example: "**/*.csv"
+ recursive:
+ title: Recursive
+ description: If true, the changes in the subdirectories of `filePath`
are sent too, new subdirectories included. If false, only the changes of the
files and directories directly in `filePath` are sent.
+ type: boolean
+ default: true
dataTypes:
out:
headers:
@@ -77,7 +87,9 @@ spec:
uri: file-watch
parameters:
path: "{{filePath}}"
+ antInclude: "RAW({{?antInclude}})"
events: "{{events}}"
+ recursive: "{{recursive}}"
steps:
- to:
uri: kamelet:sink
diff --git a/kamelets/ftp-source.kamelet.yaml b/kamelets/ftp-source.kamelet.yaml
index 0c77a7c79..411f6ceda 100644
--- a/kamelets/ftp-source.kamelet.yaml
+++ b/kamelets/ftp-source.kamelet.yaml
@@ -37,7 +37,7 @@ spec:
Output: the body is the content of the file, as a stream. The `file` and
`ce-file` headers hold the name of the file, with its path below
`directoryName` when `recursive` is true, such as `2026/orders.csv`.
`CamelFileName` holds the same name, and `CamelFileLength` the size of the file
in bytes.
- Good to know: files are not deleted or moved after they are read, unless
`delete` is true or `move` is set. A file is read only once while the
integration runs (`idempotent`), but the files left on the server are all read
again after a restart; to read each file only once, also across restarts, set
`move` to a directory such as `.done`, set `delete` to true, or keep the list
of the files read in a file with `idempotentRepository`. The connection and the
login are not encrypted. Fil [...]
+ Good to know: files are not deleted or moved after they are read, unless
`delete` is true or `move` is set. A file is read only once while the
integration runs (`idempotent`), but the files left on the server are all read
again after a restart; to read each file only once, also across restarts, set
`move` to a directory such as `.done`, set `delete` to true, or keep the list
of the files read in a file with `idempotentRepository`. To read only some of
the files, set `include`, `exc [...]
required:
- connectionHost
- connectionPort
@@ -119,6 +119,75 @@ spec:
description: Where to move a file on the server when it could not be
handed on, for example because the next step failed, so that it is not read
over and over. The same format as `move`, such as `.error`. When it is not set,
a file that fails stays where it is and is read again at every poll. It can be
set together with `delete` or `move`.
type: string
example: ".error"
+ include:
+ title: Include Files
+ description: A regular expression that the name of a file must match
to be read, such as `.*\.csv` for the CSV files. The whole name must match,
upper and lower case count as the same, and only the name is matched, not its
directory. The other files stay on the server. When it is not set, all the
files are read.
+ type: string
+ example: '.*\.csv'
+ exclude:
+ title: Exclude Files
+ description: A regular expression for the names of the files not to
read, such as `.*\.(tmp|part)`. The whole name must match, upper and lower case
count as the same, and only the name is matched, not its directory. A file it
matches is not read, even when `include` or `antInclude` matches it too.
+ type: string
+ example: '.*\.(tmp|part)'
+ antInclude:
+ title: Include Files (Wildcards)
+ description: Wildcard patterns, separated by commas, that the path of
a file must match to be read, such as `*.csv,*.xml`. The path is relative to
`directoryName`, such as `2026/orders.csv` when `recursive` is true. `*`
matches any part of a name and `**` any number of directories, so `*.csv`
matches the CSV files in `directoryName` only, and `**/*.csv` those in the
subdirectories too. Upper and lower case differ. When it is not set, all the
files are read.
+ type: string
+ example: '**/*.csv'
+ preMove:
+ title: Move Before Read
+ description: Where to move each file on the server before it is read,
so that another reader of the same directory does not take it too. The same
format as `move`, such as `.inprogress`, a directory next to the file. `move`,
`moveFailed` and `delete` then act on the file in its new place, so with `move`
set to `.done` the file ends up in `.inprogress/.done`. When it is not set, a
file is read where it is.
+ type: string
+ example: ".inprogress"
+ readLock:
+ title: Wait Until Written
+ description: How to make sure that another program has finished
writing a file before it is read. `changed` reads a file once its size and
modification time have not changed between two checks 5 seconds apart, or, with
`readLockMinAge`, once the file is that old. A file still changing after 20
seconds is skipped until the next poll, and while a file is checked the other
files wait. `changed` never reads an empty file. `rename` reads a file once it
can be renamed. `none` reads the [...]
+ type: string
+ enum: ["none", "changed", "rename"]
+ example: changed
+ readLockMinAge:
+ title: Minimum File Age
+ description: 'Only used when `readLock` is `changed`: how old a file
must be, in milliseconds, before it is read, counted from its modification time
on the server. For example, 60000 leaves a file alone until it has not been
changed for one minute. A file that is not old enough is skipped until the next
poll, after 20 seconds of checks. The clocks of the server and of the
integration must agree. FTP servers give the modification time only to the
minute, and in their own time zone [...]
+ type: integer
+ example: 60000
+ delay:
+ title: Poll Interval
+ description: How long to wait between two polls of the directory, in
milliseconds. For example, 60000 polls once a minute. When it is not set, 500.
+ type: integer
+ example: 60000
+ maxMessagesPerPoll:
+ title: Max Files Per Poll
+ description: The largest number of files to read in one poll, such as
100. The other files are read at the next polls. The poll takes the first files
the server lists, and `sortBy` sorts only these. When it is not set, or 0, all
the files found are read.
+ type: integer
+ example: 100
+ sortBy:
+ title: Sort By
+ description: The order in which the files of a poll are read.
`file:name` sorts them by name, `file:modified` by modification time and
`file:length` by size. Put `reverse:` before it for the reverse order, such as
`reverse:file:modified` for the newest first. Separate several sorts with `;`,
such as `file:modified;file:name`. FTP servers give the modification time only
to the minute, so files changed in the same minute keep the order of the
server. When it is not set, the files a [...]
+ type: string
+ example: file:modified
+ streamDownload:
+ title: Stream Download
+ description: 'If true, each file is read from the server while it is
handed on, instead of being downloaded into memory first, and the body is a
stream. Use it for files too large for memory. It saves memory only when the
integration does not keep the stream in memory either: set
`camel.main.streamCachingSpoolEnabled=true`, which writes large bodies to a
temporary file, or turn stream caching off. It needs `stepwise` set to false,
or every file fails. When it is not set, false.'
+ type: boolean
+ stepwise:
+ title: Change Directory Stepwise
+ description: If true, the integration changes into the directory of a
file, one level at a time, to read it. If false, it uses the full path of the
file instead, which some servers need. It must be false when `streamDownload`
is true. When it is not set, true.
+ type: boolean
+ connectTimeout:
+ title: Connect Timeout
+ description: How long to wait for the connection to the FTP server, in
milliseconds, and then for its first answer. A connection that fails is tried 3
times before the poll is skipped. When it is not set, 10000.
+ type: integer
+ example: 10000
+ socketTimeout:
+ title: Socket Timeout
+ description: How long to wait for the answer of the server to a
command, in milliseconds, before the poll fails. When it is not set, 300000 (5
minutes).
+ type: integer
+ example: 60000
+ dataTimeout:
+ title: Data Timeout
+ description: How long to wait for data while a file or the list of the
files is transferred, in milliseconds, before the poll fails. When it is not
set, 30000.
+ type: integer
+ example: 30000
dataTypes:
out:
headers:
@@ -148,16 +217,30 @@ spec:
host: "{{username}}@{{connectionHost}}"
port: "{{connectionPort}}"
directoryName: "{{directoryName}}"
+ antInclude: "RAW({{?antInclude}})"
autoCreate: "{{autoCreate}}"
binary: "{{binary}}"
+ connectTimeout: "{{?connectTimeout}}"
+ delay: "{{?delay}}"
delete: "{{delete}}"
+ exclude: "RAW({{?exclude}})"
idempotent: "{{idempotent}}"
idempotentRepository: "#bean:{{?idempotentRepository}}"
+ include: "RAW({{?include}})"
+ maxMessagesPerPoll: "{{?maxMessagesPerPoll}}"
move: "{{?move}}"
moveFailed: "{{?moveFailed}}"
passiveMode: "{{passiveMode}}"
password: "RAW({{password}})"
+ preMove: "{{?preMove}}"
+ readLock: "{{?readLock}}"
+ readLockMinAge: "{{?readLockMinAge}}"
recursive: "{{recursive}}"
+ sortBy: "RAW({{?sortBy}})"
+ soTimeout: "{{?socketTimeout}}"
+ stepwise: "{{?stepwise}}"
+ streamDownload: "{{?streamDownload}}"
+ timeout: "{{?dataTimeout}}"
steps:
- setHeader:
name: file
diff --git a/kamelets/ftps-sink.kamelet.yaml b/kamelets/ftps-sink.kamelet.yaml
index 1636b2980..72b9b0dbe 100644
--- a/kamelets/ftps-sink.kamelet.yaml
+++ b/kamelets/ftps-sink.kamelet.yaml
@@ -39,7 +39,7 @@ spec:
Output: the message is unchanged. The `CamelFileNameProduced` header is
set to the path of the file on the server.
- Good to know: a file with the same name is replaced, unless `fileExist`
says otherwise. A file is uploaded under its real name from the start, so a
reader that polls the directory can pick it up half-written. To prevent that,
set `tempPrefix` to upload it under a temporary name and rename it once it is
complete, or `doneFileName` to write an empty done file next to it once it is
complete. It uses explicit TLS: it connects to the normal FTP port, usually 21,
and then switches to TLS [...]
+ Good to know: a file with the same name is replaced, unless `fileExist`
says otherwise. A file is uploaded under its real name from the start, so a
reader that polls the directory can pick it up half-written. To prevent that,
set `tempPrefix` to upload it under a temporary name and rename it once it is
complete, or `doneFileName` to write an empty done file next to it once it is
complete. By default it uses explicit TLS: it connects to the normal FTP port,
usually 21, and then swit [...]
required:
- connectionHost
- connectionPort
@@ -55,7 +55,7 @@ spec:
example: ftps.example.com
connectionPort:
title: Connection Port
- description: The port of the FTPS server, usually 21. The Kamelet
connects with explicit TLS, so use the normal FTP port, not the implicit TLS
port 990.
+ description: The port of the FTPS server, usually 21, or 990 when
`implicit` is true.
type: string
default: 21
username:
@@ -107,6 +107,10 @@ spec:
description: The name of an empty file to write next to each file once
the file is complete, so that a reader knows the file can be read.
`$\{file:name}` stands for the name of the file and `$\{file:name.noext}` for
the name without its extension, so `$\{file:name}.done` writes
`orders.csv.done`, and `$\{file:name.noext}.ready` writes `orders.ready`. A
fixed name, such as `ready`, is written again after each file. No other
`$\{...}` is allowed, and with one the file is uploaded b [...]
type: string
example: "${file:name}.done"
+ implicit:
+ title: Implicit TLS
+ description: 'If true, the connection uses implicit TLS: it is
encrypted from the start, usually on port 990. If false, it uses explicit TLS:
it connects to the normal FTP port, usually 21, and then switches to TLS. Use
the mode the server expects; the other one fails to connect. When it is not
set, false.'
+ type: boolean
dataTypes:
in:
headers:
@@ -175,6 +179,7 @@ spec:
autoCreate: "{{autoCreate}}"
binary: "{{binary}}"
fileExist: "{{fileExist}}"
+ implicit: "{{?implicit}}"
passiveMode: "{{passiveMode}}"
password: "RAW({{password}})"
tempPrefix: "RAW({{?tempPrefix}})"
diff --git a/kamelets/ftps-source.kamelet.yaml
b/kamelets/ftps-source.kamelet.yaml
index 9564bbcf9..4af08f227 100644
--- a/kamelets/ftps-source.kamelet.yaml
+++ b/kamelets/ftps-source.kamelet.yaml
@@ -37,7 +37,7 @@ spec:
Output: the body is the content of the file, as a stream. The `file` and
`ce-file` headers hold the name of the file, with its path below
`directoryName` when `recursive` is true, such as `2026/orders.csv`.
`CamelFileName` holds the same name, and `CamelFileLength` the size of the file
in bytes.
- Good to know: files are not deleted or moved after they are read, unless
`delete` is true or `move` is set. A file is read only once while the
integration runs (`idempotent`), but the files left on the server are all read
again after a restart; to read each file only once, also across restarts, set
`move` to a directory such as `.done`, set `delete` to true, or keep the list
of the files read in a file with `idempotentRepository`. It uses explicit TLS:
it connects to the normal FTP [...]
+ Good to know: files are not deleted or moved after they are read, unless
`delete` is true or `move` is set. A file is read only once while the
integration runs (`idempotent`), but the files left on the server are all read
again after a restart; to read each file only once, also across restarts, set
`move` to a directory such as `.done`, set `delete` to true, or keep the list
of the files read in a file with `idempotentRepository`. To read only some of
the files, set `include`, `exc [...]
required:
- connectionHost
- connectionPort
@@ -53,7 +53,7 @@ spec:
example: ftps.example.com
connectionPort:
title: Connection Port
- description: The port of the FTPS server, usually 21. The Kamelet
connects with explicit TLS, so use the normal FTP port, not the implicit TLS
port 990.
+ description: The port of the FTPS server, usually 21, or 990 when
`implicit` is true.
type: string
default: 21
username:
@@ -119,6 +119,79 @@ spec:
description: Where to move a file on the server when it could not be
handed on, for example because the next step failed, so that it is not read
over and over. The same format as `move`, such as `.error`. When it is not set,
a file that fails stays where it is and is read again at every poll. It can be
set together with `delete` or `move`.
type: string
example: ".error"
+ include:
+ title: Include Files
+ description: A regular expression that the name of a file must match
to be read, such as `.*\.csv` for the CSV files. The whole name must match,
upper and lower case count as the same, and only the name is matched, not its
directory. The other files stay on the server. When it is not set, all the
files are read.
+ type: string
+ example: '.*\.csv'
+ exclude:
+ title: Exclude Files
+ description: A regular expression for the names of the files not to
read, such as `.*\.(tmp|part)`. The whole name must match, upper and lower case
count as the same, and only the name is matched, not its directory. A file it
matches is not read, even when `include` or `antInclude` matches it too.
+ type: string
+ example: '.*\.(tmp|part)'
+ antInclude:
+ title: Include Files (Wildcards)
+ description: Wildcard patterns, separated by commas, that the path of
a file must match to be read, such as `*.csv,*.xml`. The path is relative to
`directoryName`, such as `2026/orders.csv` when `recursive` is true. `*`
matches any part of a name and `**` any number of directories, so `*.csv`
matches the CSV files in `directoryName` only, and `**/*.csv` those in the
subdirectories too. Upper and lower case differ. When it is not set, all the
files are read.
+ type: string
+ example: '**/*.csv'
+ preMove:
+ title: Move Before Read
+ description: Where to move each file on the server before it is read,
so that another reader of the same directory does not take it too. The same
format as `move`, such as `.inprogress`, a directory next to the file. `move`,
`moveFailed` and `delete` then act on the file in its new place, so with `move`
set to `.done` the file ends up in `.inprogress/.done`. When it is not set, a
file is read where it is.
+ type: string
+ example: ".inprogress"
+ readLock:
+ title: Wait Until Written
+ description: How to make sure that another program has finished
writing a file before it is read. `changed` reads a file once its size and
modification time have not changed between two checks 5 seconds apart, or, with
`readLockMinAge`, once the file is that old. A file still changing after 20
seconds is skipped until the next poll, and while a file is checked the other
files wait. `changed` never reads an empty file. `rename` reads a file once it
can be renamed. `none` reads the [...]
+ type: string
+ enum: ["none", "changed", "rename"]
+ example: changed
+ readLockMinAge:
+ title: Minimum File Age
+ description: 'Only used when `readLock` is `changed`: how old a file
must be, in milliseconds, before it is read, counted from its modification time
on the server. For example, 60000 leaves a file alone until it has not been
changed for one minute. A file that is not old enough is skipped until the next
poll, after 20 seconds of checks. The clocks of the server and of the
integration must agree. FTP servers give the modification time only to the
minute, and in their own time zone [...]
+ type: integer
+ example: 60000
+ delay:
+ title: Poll Interval
+ description: How long to wait between two polls of the directory, in
milliseconds. For example, 60000 polls once a minute. When it is not set, 500.
+ type: integer
+ example: 60000
+ maxMessagesPerPoll:
+ title: Max Files Per Poll
+ description: The largest number of files to read in one poll, such as
100. The other files are read at the next polls. The poll takes the first files
the server lists, and `sortBy` sorts only these. When it is not set, or 0, all
the files found are read.
+ type: integer
+ example: 100
+ sortBy:
+ title: Sort By
+ description: The order in which the files of a poll are read.
`file:name` sorts them by name, `file:modified` by modification time and
`file:length` by size. Put `reverse:` before it for the reverse order, such as
`reverse:file:modified` for the newest first. Separate several sorts with `;`,
such as `file:modified;file:name`. FTP servers give the modification time only
to the minute, so files changed in the same minute keep the order of the
server. When it is not set, the files a [...]
+ type: string
+ example: file:modified
+ streamDownload:
+ title: Stream Download
+ description: 'If true, each file is read from the server while it is
handed on, instead of being downloaded into memory first, and the body is a
stream. Use it for files too large for memory. It saves memory only when the
integration does not keep the stream in memory either: set
`camel.main.streamCachingSpoolEnabled=true`, which writes large bodies to a
temporary file, or turn stream caching off. It needs `stepwise` set to false,
or every file fails. When it is not set, false.'
+ type: boolean
+ stepwise:
+ title: Change Directory Stepwise
+ description: If true, the integration changes into the directory of a
file, one level at a time, to read it. If false, it uses the full path of the
file instead, which some servers need. It must be false when `streamDownload`
is true. When it is not set, true.
+ type: boolean
+ connectTimeout:
+ title: Connect Timeout
+ description: How long to wait for the connection to the FTPS server,
in milliseconds, and then for its first answer. A connection that fails is
tried 3 times before the poll is skipped. When it is not set, 10000.
+ type: integer
+ example: 10000
+ socketTimeout:
+ title: Socket Timeout
+ description: How long to wait for the answer of the server to a
command, in milliseconds, before the poll fails. When it is not set, 300000 (5
minutes).
+ type: integer
+ example: 60000
+ dataTimeout:
+ title: Data Timeout
+ description: How long to wait for data while a file or the list of the
files is transferred, in milliseconds, before the poll fails. When it is not
set, 30000.
+ type: integer
+ example: 30000
+ implicit:
+ title: Implicit TLS
+ description: 'If true, the connection uses implicit TLS: it is
encrypted from the start, usually on port 990. If false, it uses explicit TLS:
it connects to the normal FTP port, usually 21, and then switches to TLS. Use
the mode the server expects; the other one fails to connect. When it is not
set, false.'
+ type: boolean
dataTypes:
out:
headers:
@@ -148,16 +221,31 @@ spec:
host: "{{username}}@{{connectionHost}}"
port: "{{connectionPort}}"
directoryName: "{{directoryName}}"
+ antInclude: "RAW({{?antInclude}})"
autoCreate: "{{autoCreate}}"
binary: "{{binary}}"
+ connectTimeout: "{{?connectTimeout}}"
+ delay: "{{?delay}}"
delete: "{{delete}}"
+ exclude: "RAW({{?exclude}})"
idempotent: "{{idempotent}}"
idempotentRepository: "#bean:{{?idempotentRepository}}"
+ implicit: "{{?implicit}}"
+ include: "RAW({{?include}})"
+ maxMessagesPerPoll: "{{?maxMessagesPerPoll}}"
move: "{{?move}}"
moveFailed: "{{?moveFailed}}"
passiveMode: "{{passiveMode}}"
password: "RAW({{password}})"
+ preMove: "{{?preMove}}"
+ readLock: "{{?readLock}}"
+ readLockMinAge: "{{?readLockMinAge}}"
recursive: "{{recursive}}"
+ sortBy: "RAW({{?sortBy}})"
+ soTimeout: "{{?socketTimeout}}"
+ stepwise: "{{?stepwise}}"
+ streamDownload: "{{?streamDownload}}"
+ timeout: "{{?dataTimeout}}"
steps:
- setHeader:
name: file
diff --git a/kamelets/scp-sink.kamelet.yaml b/kamelets/scp-sink.kamelet.yaml
index 6a1397ae3..a85b74581 100644
--- a/kamelets/scp-sink.kamelet.yaml
+++ b/kamelets/scp-sink.kamelet.yaml
@@ -32,15 +32,15 @@ spec:
definition:
title: "SCP Sink"
description: |-
- Copies each message as a file to the home directory of a user on a
remote server, with SCP (secure copy) over an SSH connection.
+ Copies each message as a file to a directory on a remote server, with
SCP (secure copy) over an SSH connection.
- Use it to copy files to a server you reach with SSH that does not offer
SFTP, for example to drop a configuration or a report on a host. To choose the
directory on the server, or to append to or keep existing files, use
`sftp-sink` instead.
+ Use it to copy files to a server you reach with SSH that does not offer
SFTP, for example to drop a configuration or a report on a host. To append to
or keep existing files, or to create the directory when it is missing, use
`sftp-sink` instead.
Input: the body is the content of the file, as text, bytes or a stream.
The name of the file comes from the `CamelFileName` header, which may hold
subdirectories, such as `reports/data.csv`; without it, a unique name is
generated.
Output: the message is unchanged.
- Good to know: files are written in the home directory of `username`,
with the rights of that user, and a file with the same name is replaced. A file
name that points outside the home directory fails the message. The key of the
server is checked against the file in `knownHostsUri`, or else
`~/.ssh/known_hosts` of the user that runs the integration, unless
`strictHostKeyChecking` is `no`. Do not place it after messages from an
untrusted source.
+ Good to know: files are written in `directoryName`, or else in the home
directory of `username`, with the rights of that user, and a file with the same
name is replaced. `directoryName` must exist: when it does not, every file is
written as a file with the name of the directory. A file name that points
outside the directory fails the message. The key of the server is checked
against the file in `knownHostsUri`, or else `~/.ssh/known_hosts` of the user
that runs the integration, unl [...]
required:
- serverName
- serverPort
@@ -95,6 +95,18 @@ spec:
description: If true, the key of the server is looked up in
`~/.ssh/known_hosts` of the user that runs the integration.
type: boolean
default: true
+ directoryName:
+ title: Directory Name
+ description: The directory on the server to write the files to, such
as `incoming/reports` or `/data/incoming`. A path without a leading `/` is
relative to the home directory of `username`. It must exist, as it is not
created; when it is missing, each file is written as a file with the name of
the directory instead. When it is not set, the files are written in the home
directory.
+ type: string
+ pattern: "^[A-Za-z0-9_./~@+=,-]*$"
+ example: incoming/reports
+ chmod:
+ title: File Permissions
+ description: The permissions to give each new file, as three octal
digits, such as `640` for read and write by the owner and read by the group.
The umask of the server can take some away, so with the usual umask 022 the
`664` used when it is not set gives `644`. A file that already exists keeps its
permissions, and only its content is replaced.
+ type: string
+ pattern: "^[0-7]{3}$"
+ example: "640"
dataTypes:
in:
headers:
@@ -110,4 +122,4 @@ spec:
uri: kamelet:source
steps:
- to:
- uri:
"scp:{{serverName}}:{{serverPort}}?knownHostsFile={{?knownHostsUri}}&password=RAW({{?password}})&privateKeyFile=RAW({{?privateKeyFile}})&privateKeyFilePassphrase=RAW({{?privateKeyPassphrase}})&strictHostKeyChecking={{?strictHostKeyChecking}}&useUserKnownHostsFile={{?useUserKnownHostsFile}}&username=RAW({{?username}})"
+ uri:
"scp:{{serverName}}:{{serverPort}}/{{?directoryName}}?chmod={{?chmod}}&knownHostsFile={{?knownHostsUri}}&password=RAW({{?password}})&privateKeyFile=RAW({{?privateKeyFile}})&privateKeyFilePassphrase=RAW({{?privateKeyPassphrase}})&strictHostKeyChecking={{?strictHostKeyChecking}}&useUserKnownHostsFile={{?useUserKnownHostsFile}}&username=RAW({{?username}})"
diff --git a/kamelets/sftp-sink.kamelet.yaml b/kamelets/sftp-sink.kamelet.yaml
index 5f732ea54..b0b41c3d0 100644
--- a/kamelets/sftp-sink.kamelet.yaml
+++ b/kamelets/sftp-sink.kamelet.yaml
@@ -140,6 +140,17 @@ spec:
description: The name of an empty file to write next to each file once
the file is complete, so that a reader knows the file can be read.
`$\{file:name}` stands for the name of the file and `$\{file:name.noext}` for
the name without its extension, so `$\{file:name}.done` writes
`orders.csv.done`, and `$\{file:name.noext}.ready` writes `orders.ready`. A
fixed name, such as `ready`, is written again after each file. No other
`$\{...}` is allowed, and with one the file is uploaded b [...]
type: string
example: "${file:name}.done"
+ chmod:
+ title: File Permissions
+ description: The permissions to give each file once it is written, as
three octal digits, such as `640` for read and write by the owner and read by
the group. They are set on a file that already existed too. When it is not set,
a new file gets the permissions the server gives it, usually `644`, and a file
that already existed keeps its own.
+ type: string
+ pattern: "^[0-7]{3}$"
+ example: "640"
+ serverAliveInterval:
+ title: Server Alive Interval
+ description: How often to check, in milliseconds, that the server
still answers while no data comes from it. When the server misses a check, the
connection is closed, and the next message makes a new one. Use it to notice a
lost connection soon, for example behind a firewall that drops idle
connections. When it is not set, no checks are made.
+ type: integer
+ example: 30000
dataTypes:
in:
headers:
@@ -204,6 +215,7 @@ spec:
host: "{{connectionHost}}"
port: "{{connectionPort}}"
directoryName: "{{directoryName}}"
+ chmod: "{{?chmod}}"
doneFileName: "RAW({{?doneFileName}})"
autoCreate: "{{autoCreate}}"
binary: "{{binary}}"
@@ -214,6 +226,7 @@ spec:
privateKeyFile: "RAW({{?privateKeyFile}})"
privateKeyPassphrase: "RAW({{?privateKeyPassphrase}})"
privateKeyUri: "RAW({{?privateKeyUri}})"
+ serverAliveInterval: "{{?serverAliveInterval}}"
strictHostKeyChecking: "{{?strictHostKeyChecking}}"
tempPrefix: "RAW({{?tempPrefix}})"
useUserKnownHostsFile: "{{?useUserKnownHostsFile}}"
diff --git a/kamelets/sftp-source.kamelet.yaml
b/kamelets/sftp-source.kamelet.yaml
index f4697f038..21ba7d6e4 100644
--- a/kamelets/sftp-source.kamelet.yaml
+++ b/kamelets/sftp-source.kamelet.yaml
@@ -35,9 +35,9 @@ spec:
Use it to collect the files that a partner or another system drops on a
server you reach with SSH, such as orders, invoices or exports. For an FTP
server use `ftp-source` or `ftps-source`.
- Output: the body is the content of the file, as bytes. The `file` and
`ce-file` headers hold the name of the file, with its path below
`directoryName` when `recursive` is true, such as `2026/orders.csv`.
`CamelFileName` holds the same name, and `CamelFileLength` the size of the file
in bytes.
+ Output: the body is the content of the file, as bytes, or as a stream
when `streamDownload` is true. The `file` and `ce-file` headers hold the name
of the file, with its path below `directoryName` when `recursive` is true, such
as `2026/orders.csv`. `CamelFileName` holds the same name, and
`CamelFileLength` the size of the file in bytes.
- Good to know: files are not deleted or moved after they are read, unless
`delete` is true or `move` is set. A file is read only once while the
integration runs (`idempotent`), but the files left on the server are all read
again after a restart; to read each file only once, also across restarts, set
`move` to a directory such as `.done`, set `delete` to true, or keep the list
of the files read in a file with `idempotentRepository`. Log in with `username`
and either `password` or a p [...]
+ Good to know: files are not deleted or moved after they are read, unless
`delete` is true or `move` is set. A file is read only once while the
integration runs (`idempotent`), but the files left on the server are all read
again after a restart; to read each file only once, also across restarts, set
`move` to a directory such as `.done`, set `delete` to true, or keep the list
of the files read in a file with `idempotentRepository`. To read only some of
the files, set `include`, `exc [...]
required:
- connectionHost
- connectionPort
@@ -157,6 +157,71 @@ spec:
description: Where to move a file on the server when it could not be
handed on, for example because the next step failed, so that it is not read
over and over. The same format as `move`, such as `.error`. When it is not set,
a file that fails stays where it is and is read again at every poll. It can be
set together with `delete` or `move`.
type: string
example: ".error"
+ include:
+ title: Include Files
+ description: A regular expression that the name of a file must match
to be read, such as `.*\.csv` for the CSV files. The whole name must match,
upper and lower case count as the same, and only the name is matched, not its
directory. The other files stay on the server. When it is not set, all the
files are read.
+ type: string
+ example: '.*\.csv'
+ exclude:
+ title: Exclude Files
+ description: A regular expression for the names of the files not to
read, such as `.*\.(tmp|part)`. The whole name must match, upper and lower case
count as the same, and only the name is matched, not its directory. A file it
matches is not read, even when `include` or `antInclude` matches it too.
+ type: string
+ example: '.*\.(tmp|part)'
+ antInclude:
+ title: Include Files (Wildcards)
+ description: Wildcard patterns, separated by commas, that the path of
a file must match to be read, such as `*.csv,*.xml`. The path is relative to
`directoryName`, such as `2026/orders.csv` when `recursive` is true. `*`
matches any part of a name and `**` any number of directories, so `*.csv`
matches the CSV files in `directoryName` only, and `**/*.csv` those in the
subdirectories too. Upper and lower case differ. When it is not set, all the
files are read.
+ type: string
+ example: '**/*.csv'
+ preMove:
+ title: Move Before Read
+ description: Where to move each file on the server before it is read,
so that another reader of the same directory does not take it too. The same
format as `move`, such as `.inprogress`, a directory next to the file. `move`,
`moveFailed` and `delete` then act on the file in its new place, so with `move`
set to `.done` the file ends up in `.inprogress/.done`. When it is not set, a
file is read where it is.
+ type: string
+ example: ".inprogress"
+ readLock:
+ title: Wait Until Written
+ description: How to make sure that another program has finished
writing a file before it is read. `changed` reads a file once its size and
modification time have not changed between two checks 5 seconds apart, or, with
`readLockMinAge`, once the file is that old. A file still changing after 20
seconds is skipped until the next poll, and while a file is checked the other
files wait. `changed` never reads an empty file. `rename` reads a file once it
can be renamed. `none` reads the [...]
+ type: string
+ enum: ["none", "changed", "rename"]
+ example: changed
+ readLockMinAge:
+ title: Minimum File Age
+ description: 'Only used when `readLock` is `changed`: how old a file
must be, in milliseconds, before it is read, counted from its modification time
on the server. For example, 60000 leaves a file alone until it has not been
changed for one minute. A file that is not old enough is skipped until the next
poll, after 20 seconds of checks. The clocks of the server and of the
integration must agree.'
+ type: integer
+ example: 60000
+ delay:
+ title: Poll Interval
+ description: How long to wait between two polls of the directory, in
milliseconds. For example, 60000 polls once a minute. When it is not set, 500.
+ type: integer
+ example: 60000
+ maxMessagesPerPoll:
+ title: Max Files Per Poll
+ description: The largest number of files to read in one poll, such as
100. The other files are read at the next polls. The poll takes the first files
the server lists, and `sortBy` sorts only these. When it is not set, or 0, all
the files found are read.
+ type: integer
+ example: 100
+ sortBy:
+ title: Sort By
+ description: The order in which the files of a poll are read.
`file:name` sorts them by name, `file:modified` by modification time and
`file:length` by size. Put `reverse:` before it for the reverse order, such as
`reverse:file:modified` for the newest first. Separate several sorts with `;`,
such as `file:modified;file:name`. When it is not set, the files are read in
the order the server lists them.
+ type: string
+ example: file:modified
+ streamDownload:
+ title: Stream Download
+ description: 'If true, each file is read from the server while it is
handed on, instead of being downloaded into memory first, and the body is a
stream. Use it for files too large for memory. It saves memory only when the
integration does not keep the stream in memory either: set
`camel.main.streamCachingSpoolEnabled=true`, which writes large bodies to a
temporary file, or turn stream caching off. When it is not set, false.'
+ type: boolean
+ connectTimeout:
+ title: Connect Timeout
+ description: How long to wait for the connection to the SFTP server,
in milliseconds. A connection that fails is tried 3 times before the poll is
skipped. When it is not set, 10000.
+ type: integer
+ example: 10000
+ socketTimeout:
+ title: Socket Timeout
+ description: How long the connection may stay without an answer from
the server, in milliseconds, before it is closed and the poll fails. It is not
used when `serverAliveInterval` is set. When it is not set, 300000 (5 minutes).
+ type: integer
+ example: 60000
+ serverAliveInterval:
+ title: Server Alive Interval
+ description: How often to check, in milliseconds, that the server
still answers while no data comes from it. When the server misses a check, the
connection is closed and the poll fails, and a new connection is made at the
next poll. Use it to notice a lost connection soon, for example behind a
firewall that drops idle connections. When it is set, `socketTimeout` is not
used. When it is not set, no checks are made.
+ type: integer
+ example: 30000
dataTypes:
out:
headers:
@@ -186,29 +251,48 @@ spec:
host: "{{connectionHost}}"
port: "{{connectionPort}}"
directoryName: "{{directoryName}}"
+ antInclude: "RAW({{?antInclude}})"
autoCreate: "{{autoCreate}}"
binary: "{{binary}}"
+ connectTimeout: "{{?connectTimeout}}"
+ delay: "{{?delay}}"
delete: "{{delete}}"
+ exclude: "RAW({{?exclude}})"
idempotent: "{{idempotent}}"
idempotentRepository: "#bean:{{?idempotentRepository}}"
ignoreFileNotFoundOrPermissionError:
"{{ignoreFileNotFoundOrPermissionError}}"
+ include: "RAW({{?include}})"
knownHostsUri: "RAW({{?knownHostsUri}})"
+ maxMessagesPerPoll: "{{?maxMessagesPerPoll}}"
move: "{{?move}}"
moveFailed: "{{?moveFailed}}"
passiveMode: "{{passiveMode}}"
password: "RAW({{?password}})"
+ preMove: "{{?preMove}}"
privateKeyFile: "RAW({{?privateKeyFile}})"
privateKeyPassphrase: "RAW({{?privateKeyPassphrase}})"
privateKeyUri: "RAW({{?privateKeyUri}})"
+ readLock: "{{?readLock}}"
+ readLockMinAge: "{{?readLockMinAge}}"
recursive: "{{recursive}}"
+ serverAliveInterval: "{{?serverAliveInterval}}"
+ sortBy: "RAW({{?sortBy}})"
+ soTimeout: "{{?socketTimeout}}"
+ streamDownload: "{{?streamDownload}}"
strictHostKeyChecking: "{{?strictHostKeyChecking}}"
useUserKnownHostsFile: "{{?useUserKnownHostsFile}}"
username: "RAW({{?username}})"
steps:
- - setBody:
- expression:
- simple:
- expression: "${body.getBody()}"
+ - choice:
+ when:
+ - expression:
+ simple:
+ expression: "${body} is
'org.apache.camel.component.file.GenericFile'"
+ steps:
+ - setBody:
+ expression:
+ simple:
+ expression: "${body.getBody()}"
- setHeader:
name: file
expression:
diff --git
a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
index 93a103e17..fe7326cc5 100644
--- a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
+++ b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
@@ -765,7 +765,7 @@
"supportLevel" : "Stable",
"title" : "File Watch Source",
"summary" : "Sends a message each time a file in a local directory is
created, changed or deleted, with the kind of change and the path of the file.",
- "description" : "Sends a message each time a file in a local directory is
created, changed or deleted, with the kind of change and the path of the
file.\n\nUse it to react to files that another program writes into a directory
on the same machine, for example to start processing when a new export appears,
or to keep track of the changes in a folder. To read files from a remote server
instead, use `ftp-source`, `sftp-source` or `aws-s3-source`.\n\nOutput: the
body is the file that chan [...]
+ "description" : "Sends a message each time a file in a local directory is
created, changed or deleted, with the kind of change and the path of the
file.\n\nUse it to react to files that another program writes into a directory
on the same machine, for example to start processing when a new export appears,
or to keep track of the changes in a folder. To read files from a remote server
instead, use `ftp-source`, `sftp-source` or `aws-s3-source`.\n\nOutput: the
body is the file that chan [...]
"group" : "File Watch",
"namespace" : "File",
"provider" : "Apache Software Foundation"
@@ -795,7 +795,7 @@
"supportLevel" : "Stable",
"title" : "FTP Source",
"summary" : "Reads the files in a directory on an FTP server, one message
per file, and by default leaves them on the server.",
- "description" : "Reads the files in a directory on an FTP server, one
message per file, and by default leaves them on the server.\n\nUse it to
collect the files that a partner or a legacy system drops on an FTP server,
such as orders, invoices or exports. For an encrypted connection use
`ftps-source`, and for a server you reach with SSH use
`sftp-source`.\n\nOutput: the body is the content of the file, as a stream. The
`file` and `ce-file` headers hold the name of the file, with its [...]
+ "description" : "Reads the files in a directory on an FTP server, one
message per file, and by default leaves them on the server.\n\nUse it to
collect the files that a partner or a legacy system drops on an FTP server,
such as orders, invoices or exports. For an encrypted connection use
`ftps-source`, and for a server you reach with SSH use
`sftp-source`.\n\nOutput: the body is the content of the file, as a stream. The
`file` and `ce-file` headers hold the name of the file, with its [...]
"group" : "FTP",
"namespace" : "File",
"provider" : "Apache Software Foundation"
@@ -805,7 +805,7 @@
"supportLevel" : "Stable",
"title" : "FTPS Sink",
"summary" : "Uploads each message as a file to a directory on an FTPS
server, which is FTP with TLS encryption.",
- "description" : "Uploads each message as a file to a directory on an FTPS
server, which is FTP with TLS encryption.\n\nUse it to deliver exports, reports
or data files to a partner or a legacy system that receives files over FTPS.
For a server without TLS use `ftp-sink`, and for a server you reach with SSH
use `sftp-sink`.\n\nInput: the body is the content of the file, as text, bytes
or a stream. The name of the file comes from the `file` header, or from
`ce-file` when `file` is not [...]
+ "description" : "Uploads each message as a file to a directory on an FTPS
server, which is FTP with TLS encryption.\n\nUse it to deliver exports, reports
or data files to a partner or a legacy system that receives files over FTPS.
For a server without TLS use `ftp-sink`, and for a server you reach with SSH
use `sftp-sink`.\n\nInput: the body is the content of the file, as text, bytes
or a stream. The name of the file comes from the `file` header, or from
`ce-file` when `file` is not [...]
"group" : "FTP",
"namespace" : "File",
"provider" : "Apache Software Foundation"
@@ -815,7 +815,7 @@
"supportLevel" : "Stable",
"title" : "FTPS Source",
"summary" : "Reads the files in a directory on an FTPS server (FTP with
TLS encryption), one message per file, and by default leaves them on the
server.",
- "description" : "Reads the files in a directory on an FTPS server (FTP
with TLS encryption), one message per file, and by default leaves them on the
server.\n\nUse it to collect the files that a partner or a legacy system drops
on an FTPS server, such as orders, invoices or exports. For a server without
TLS use `ftp-source`, and for a server you reach with SSH use
`sftp-source`.\n\nOutput: the body is the content of the file, as a stream. The
`file` and `ce-file` headers hold the nam [...]
+ "description" : "Reads the files in a directory on an FTPS server (FTP
with TLS encryption), one message per file, and by default leaves them on the
server.\n\nUse it to collect the files that a partner or a legacy system drops
on an FTPS server, such as orders, invoices or exports. For a server without
TLS use `ftp-source`, and for a server you reach with SSH use
`sftp-source`.\n\nOutput: the body is the content of the file, as a stream. The
`file` and `ce-file` headers hold the nam [...]
"group" : "FTP",
"namespace" : "File",
"provider" : "Apache Software Foundation"
@@ -1954,8 +1954,8 @@
"type" : "sink",
"supportLevel" : "Stable",
"title" : "SCP Sink",
- "summary" : "Copies each message as a file to the home directory of a user
on a remote server, with SCP (secure copy) over an SSH connection.",
- "description" : "Copies each message as a file to the home directory of a
user on a remote server, with SCP (secure copy) over an SSH connection.\n\nUse
it to copy files to a server you reach with SSH that does not offer SFTP, for
example to drop a configuration or a report on a host. To choose the directory
on the server, or to append to or keep existing files, use `sftp-sink`
instead.\n\nInput: the body is the content of the file, as text, bytes or a
stream. The name of the file co [...]
+ "summary" : "Copies each message as a file to a directory on a remote
server, with SCP (secure copy) over an SSH connection.",
+ "description" : "Copies each message as a file to a directory on a remote
server, with SCP (secure copy) over an SSH connection.\n\nUse it to copy files
to a server you reach with SSH that does not offer SFTP, for example to drop a
configuration or a report on a host. To append to or keep existing files, or to
create the directory when it is missing, use `sftp-sink` instead.\n\nInput: the
body is the content of the file, as text, bytes or a stream. The name of the
file comes from the [...]
"group" : "SCP",
"namespace" : "File",
"provider" : "Apache Software Foundation"
@@ -1995,7 +1995,7 @@
"supportLevel" : "Stable",
"title" : "SFTP Source",
"summary" : "Reads the files in a directory on an SFTP server, over an
encrypted SSH connection, one message per file, and by default leaves them on
the server.",
- "description" : "Reads the files in a directory on an SFTP server, over an
encrypted SSH connection, one message per file, and by default leaves them on
the server.\n\nUse it to collect the files that a partner or another system
drops on a server you reach with SSH, such as orders, invoices or exports. For
an FTP server use `ftp-source` or `ftps-source`.\n\nOutput: the body is the
content of the file, as bytes. The `file` and `ce-file` headers hold the name
of the file, with its path [...]
+ "description" : "Reads the files in a directory on an SFTP server, over an
encrypted SSH connection, one message per file, and by default leaves them on
the server.\n\nUse it to collect the files that a partner or another system
drops on a server you reach with SSH, such as orders, invoices or exports. For
an FTP server use `ftp-source` or `ftps-source`.\n\nOutput: the body is the
content of the file, as bytes, or as a stream when `streamDownload` is true.
The `file` and `ce-file` he [...]
"group" : "FTP",
"namespace" : "File",
"provider" : "Apache Software Foundation"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/file-watch-source.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/file-watch-source.kamelet.yaml
index a70a6c352..28ad21dae 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/file-watch-source.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/file-watch-source.kamelet.yaml
@@ -37,7 +37,7 @@ spec:
Output: the body is the file that changed, which the next step can read
as text or bytes; a deleted file or a directory has no content to read. The
`CamelFileEventType` header is `CREATE`, `MODIFY` or `DELETE`, `CamelFileName`
is the path of the file relative to the watched directory, and
`CamelFileAbsolutePath` is its full path.
- Good to know: it watches the subdirectories too, new ones included. It
never changes or deletes the files. Only changes made while it runs are sent,
not the files that are already there when it starts. The path must be a
directory, not a single file; it is created when it does not exist. On
Kubernetes, mount the directory into the pod as a volume.
+ Good to know: it watches the subdirectories too, new ones included,
unless `recursive` is false. To send only the changes of some files, set
`antInclude`. It never changes or deletes the files. Only changes made while it
runs are sent, not the files that are already there when it starts. The path
must be a directory, not a single file; it is created when it does not exist.
On Kubernetes, mount the directory into the pod as a volume.
required:
- filePath
- events
@@ -45,7 +45,7 @@ spec:
properties:
filePath:
title: Directory to Watch
- description: "The full path of the local directory to watch. It must
be a directory, not a single file: to follow one file, watch its directory.
Changes in its subdirectories are sent too."
+ description: "The full path of the local directory to watch. It must
be a directory, not a single file: to follow one file, watch its directory.
Changes in its subdirectories are sent too, unless `recursive` is false."
type: string
example: /data/incoming
events:
@@ -54,6 +54,16 @@ spec:
type: string
default: "CREATE,MODIFY,DELETE"
example: CREATE
+ antInclude:
+ title: Include Files (Wildcards)
+ description: "A wildcard pattern that the path of a file must match
for its changes to be sent, such as `**/*.csv`. The path is relative to
`filePath`, such as `2026/orders.csv`. `*` matches any part of a name, and `**`
any number of directories: `*.csv` matches the CSV files in `filePath` only,
and `**/*.csv` those in the subdirectories too. Upper and lower case differ.
When it is not set, the changes of all the files are sent."
+ type: string
+ example: "**/*.csv"
+ recursive:
+ title: Recursive
+ description: If true, the changes in the subdirectories of `filePath`
are sent too, new subdirectories included. If false, only the changes of the
files and directories directly in `filePath` are sent.
+ type: boolean
+ default: true
dataTypes:
out:
headers:
@@ -77,7 +87,9 @@ spec:
uri: file-watch
parameters:
path: "{{filePath}}"
+ antInclude: "RAW({{?antInclude}})"
events: "{{events}}"
+ recursive: "{{recursive}}"
steps:
- to:
uri: kamelet:sink
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/ftp-source.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/ftp-source.kamelet.yaml
index 0c77a7c79..411f6ceda 100644
--- a/library/camel-kamelets/src/main/resources/kamelets/ftp-source.kamelet.yaml
+++ b/library/camel-kamelets/src/main/resources/kamelets/ftp-source.kamelet.yaml
@@ -37,7 +37,7 @@ spec:
Output: the body is the content of the file, as a stream. The `file` and
`ce-file` headers hold the name of the file, with its path below
`directoryName` when `recursive` is true, such as `2026/orders.csv`.
`CamelFileName` holds the same name, and `CamelFileLength` the size of the file
in bytes.
- Good to know: files are not deleted or moved after they are read, unless
`delete` is true or `move` is set. A file is read only once while the
integration runs (`idempotent`), but the files left on the server are all read
again after a restart; to read each file only once, also across restarts, set
`move` to a directory such as `.done`, set `delete` to true, or keep the list
of the files read in a file with `idempotentRepository`. The connection and the
login are not encrypted. Fil [...]
+ Good to know: files are not deleted or moved after they are read, unless
`delete` is true or `move` is set. A file is read only once while the
integration runs (`idempotent`), but the files left on the server are all read
again after a restart; to read each file only once, also across restarts, set
`move` to a directory such as `.done`, set `delete` to true, or keep the list
of the files read in a file with `idempotentRepository`. To read only some of
the files, set `include`, `exc [...]
required:
- connectionHost
- connectionPort
@@ -119,6 +119,75 @@ spec:
description: Where to move a file on the server when it could not be
handed on, for example because the next step failed, so that it is not read
over and over. The same format as `move`, such as `.error`. When it is not set,
a file that fails stays where it is and is read again at every poll. It can be
set together with `delete` or `move`.
type: string
example: ".error"
+ include:
+ title: Include Files
+ description: A regular expression that the name of a file must match
to be read, such as `.*\.csv` for the CSV files. The whole name must match,
upper and lower case count as the same, and only the name is matched, not its
directory. The other files stay on the server. When it is not set, all the
files are read.
+ type: string
+ example: '.*\.csv'
+ exclude:
+ title: Exclude Files
+ description: A regular expression for the names of the files not to
read, such as `.*\.(tmp|part)`. The whole name must match, upper and lower case
count as the same, and only the name is matched, not its directory. A file it
matches is not read, even when `include` or `antInclude` matches it too.
+ type: string
+ example: '.*\.(tmp|part)'
+ antInclude:
+ title: Include Files (Wildcards)
+ description: Wildcard patterns, separated by commas, that the path of
a file must match to be read, such as `*.csv,*.xml`. The path is relative to
`directoryName`, such as `2026/orders.csv` when `recursive` is true. `*`
matches any part of a name and `**` any number of directories, so `*.csv`
matches the CSV files in `directoryName` only, and `**/*.csv` those in the
subdirectories too. Upper and lower case differ. When it is not set, all the
files are read.
+ type: string
+ example: '**/*.csv'
+ preMove:
+ title: Move Before Read
+ description: Where to move each file on the server before it is read,
so that another reader of the same directory does not take it too. The same
format as `move`, such as `.inprogress`, a directory next to the file. `move`,
`moveFailed` and `delete` then act on the file in its new place, so with `move`
set to `.done` the file ends up in `.inprogress/.done`. When it is not set, a
file is read where it is.
+ type: string
+ example: ".inprogress"
+ readLock:
+ title: Wait Until Written
+ description: How to make sure that another program has finished
writing a file before it is read. `changed` reads a file once its size and
modification time have not changed between two checks 5 seconds apart, or, with
`readLockMinAge`, once the file is that old. A file still changing after 20
seconds is skipped until the next poll, and while a file is checked the other
files wait. `changed` never reads an empty file. `rename` reads a file once it
can be renamed. `none` reads the [...]
+ type: string
+ enum: ["none", "changed", "rename"]
+ example: changed
+ readLockMinAge:
+ title: Minimum File Age
+ description: 'Only used when `readLock` is `changed`: how old a file
must be, in milliseconds, before it is read, counted from its modification time
on the server. For example, 60000 leaves a file alone until it has not been
changed for one minute. A file that is not old enough is skipped until the next
poll, after 20 seconds of checks. The clocks of the server and of the
integration must agree. FTP servers give the modification time only to the
minute, and in their own time zone [...]
+ type: integer
+ example: 60000
+ delay:
+ title: Poll Interval
+ description: How long to wait between two polls of the directory, in
milliseconds. For example, 60000 polls once a minute. When it is not set, 500.
+ type: integer
+ example: 60000
+ maxMessagesPerPoll:
+ title: Max Files Per Poll
+ description: The largest number of files to read in one poll, such as
100. The other files are read at the next polls. The poll takes the first files
the server lists, and `sortBy` sorts only these. When it is not set, or 0, all
the files found are read.
+ type: integer
+ example: 100
+ sortBy:
+ title: Sort By
+ description: The order in which the files of a poll are read.
`file:name` sorts them by name, `file:modified` by modification time and
`file:length` by size. Put `reverse:` before it for the reverse order, such as
`reverse:file:modified` for the newest first. Separate several sorts with `;`,
such as `file:modified;file:name`. FTP servers give the modification time only
to the minute, so files changed in the same minute keep the order of the
server. When it is not set, the files a [...]
+ type: string
+ example: file:modified
+ streamDownload:
+ title: Stream Download
+ description: 'If true, each file is read from the server while it is
handed on, instead of being downloaded into memory first, and the body is a
stream. Use it for files too large for memory. It saves memory only when the
integration does not keep the stream in memory either: set
`camel.main.streamCachingSpoolEnabled=true`, which writes large bodies to a
temporary file, or turn stream caching off. It needs `stepwise` set to false,
or every file fails. When it is not set, false.'
+ type: boolean
+ stepwise:
+ title: Change Directory Stepwise
+ description: If true, the integration changes into the directory of a
file, one level at a time, to read it. If false, it uses the full path of the
file instead, which some servers need. It must be false when `streamDownload`
is true. When it is not set, true.
+ type: boolean
+ connectTimeout:
+ title: Connect Timeout
+ description: How long to wait for the connection to the FTP server, in
milliseconds, and then for its first answer. A connection that fails is tried 3
times before the poll is skipped. When it is not set, 10000.
+ type: integer
+ example: 10000
+ socketTimeout:
+ title: Socket Timeout
+ description: How long to wait for the answer of the server to a
command, in milliseconds, before the poll fails. When it is not set, 300000 (5
minutes).
+ type: integer
+ example: 60000
+ dataTimeout:
+ title: Data Timeout
+ description: How long to wait for data while a file or the list of the
files is transferred, in milliseconds, before the poll fails. When it is not
set, 30000.
+ type: integer
+ example: 30000
dataTypes:
out:
headers:
@@ -148,16 +217,30 @@ spec:
host: "{{username}}@{{connectionHost}}"
port: "{{connectionPort}}"
directoryName: "{{directoryName}}"
+ antInclude: "RAW({{?antInclude}})"
autoCreate: "{{autoCreate}}"
binary: "{{binary}}"
+ connectTimeout: "{{?connectTimeout}}"
+ delay: "{{?delay}}"
delete: "{{delete}}"
+ exclude: "RAW({{?exclude}})"
idempotent: "{{idempotent}}"
idempotentRepository: "#bean:{{?idempotentRepository}}"
+ include: "RAW({{?include}})"
+ maxMessagesPerPoll: "{{?maxMessagesPerPoll}}"
move: "{{?move}}"
moveFailed: "{{?moveFailed}}"
passiveMode: "{{passiveMode}}"
password: "RAW({{password}})"
+ preMove: "{{?preMove}}"
+ readLock: "{{?readLock}}"
+ readLockMinAge: "{{?readLockMinAge}}"
recursive: "{{recursive}}"
+ sortBy: "RAW({{?sortBy}})"
+ soTimeout: "{{?socketTimeout}}"
+ stepwise: "{{?stepwise}}"
+ streamDownload: "{{?streamDownload}}"
+ timeout: "{{?dataTimeout}}"
steps:
- setHeader:
name: file
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/ftps-sink.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/ftps-sink.kamelet.yaml
index 1636b2980..72b9b0dbe 100644
--- a/library/camel-kamelets/src/main/resources/kamelets/ftps-sink.kamelet.yaml
+++ b/library/camel-kamelets/src/main/resources/kamelets/ftps-sink.kamelet.yaml
@@ -39,7 +39,7 @@ spec:
Output: the message is unchanged. The `CamelFileNameProduced` header is
set to the path of the file on the server.
- Good to know: a file with the same name is replaced, unless `fileExist`
says otherwise. A file is uploaded under its real name from the start, so a
reader that polls the directory can pick it up half-written. To prevent that,
set `tempPrefix` to upload it under a temporary name and rename it once it is
complete, or `doneFileName` to write an empty done file next to it once it is
complete. It uses explicit TLS: it connects to the normal FTP port, usually 21,
and then switches to TLS [...]
+ Good to know: a file with the same name is replaced, unless `fileExist`
says otherwise. A file is uploaded under its real name from the start, so a
reader that polls the directory can pick it up half-written. To prevent that,
set `tempPrefix` to upload it under a temporary name and rename it once it is
complete, or `doneFileName` to write an empty done file next to it once it is
complete. By default it uses explicit TLS: it connects to the normal FTP port,
usually 21, and then swit [...]
required:
- connectionHost
- connectionPort
@@ -55,7 +55,7 @@ spec:
example: ftps.example.com
connectionPort:
title: Connection Port
- description: The port of the FTPS server, usually 21. The Kamelet
connects with explicit TLS, so use the normal FTP port, not the implicit TLS
port 990.
+ description: The port of the FTPS server, usually 21, or 990 when
`implicit` is true.
type: string
default: 21
username:
@@ -107,6 +107,10 @@ spec:
description: The name of an empty file to write next to each file once
the file is complete, so that a reader knows the file can be read.
`$\{file:name}` stands for the name of the file and `$\{file:name.noext}` for
the name without its extension, so `$\{file:name}.done` writes
`orders.csv.done`, and `$\{file:name.noext}.ready` writes `orders.ready`. A
fixed name, such as `ready`, is written again after each file. No other
`$\{...}` is allowed, and with one the file is uploaded b [...]
type: string
example: "${file:name}.done"
+ implicit:
+ title: Implicit TLS
+ description: 'If true, the connection uses implicit TLS: it is
encrypted from the start, usually on port 990. If false, it uses explicit TLS:
it connects to the normal FTP port, usually 21, and then switches to TLS. Use
the mode the server expects; the other one fails to connect. When it is not
set, false.'
+ type: boolean
dataTypes:
in:
headers:
@@ -175,6 +179,7 @@ spec:
autoCreate: "{{autoCreate}}"
binary: "{{binary}}"
fileExist: "{{fileExist}}"
+ implicit: "{{?implicit}}"
passiveMode: "{{passiveMode}}"
password: "RAW({{password}})"
tempPrefix: "RAW({{?tempPrefix}})"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/ftps-source.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/ftps-source.kamelet.yaml
index 9564bbcf9..4af08f227 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/ftps-source.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/ftps-source.kamelet.yaml
@@ -37,7 +37,7 @@ spec:
Output: the body is the content of the file, as a stream. The `file` and
`ce-file` headers hold the name of the file, with its path below
`directoryName` when `recursive` is true, such as `2026/orders.csv`.
`CamelFileName` holds the same name, and `CamelFileLength` the size of the file
in bytes.
- Good to know: files are not deleted or moved after they are read, unless
`delete` is true or `move` is set. A file is read only once while the
integration runs (`idempotent`), but the files left on the server are all read
again after a restart; to read each file only once, also across restarts, set
`move` to a directory such as `.done`, set `delete` to true, or keep the list
of the files read in a file with `idempotentRepository`. It uses explicit TLS:
it connects to the normal FTP [...]
+ Good to know: files are not deleted or moved after they are read, unless
`delete` is true or `move` is set. A file is read only once while the
integration runs (`idempotent`), but the files left on the server are all read
again after a restart; to read each file only once, also across restarts, set
`move` to a directory such as `.done`, set `delete` to true, or keep the list
of the files read in a file with `idempotentRepository`. To read only some of
the files, set `include`, `exc [...]
required:
- connectionHost
- connectionPort
@@ -53,7 +53,7 @@ spec:
example: ftps.example.com
connectionPort:
title: Connection Port
- description: The port of the FTPS server, usually 21. The Kamelet
connects with explicit TLS, so use the normal FTP port, not the implicit TLS
port 990.
+ description: The port of the FTPS server, usually 21, or 990 when
`implicit` is true.
type: string
default: 21
username:
@@ -119,6 +119,79 @@ spec:
description: Where to move a file on the server when it could not be
handed on, for example because the next step failed, so that it is not read
over and over. The same format as `move`, such as `.error`. When it is not set,
a file that fails stays where it is and is read again at every poll. It can be
set together with `delete` or `move`.
type: string
example: ".error"
+ include:
+ title: Include Files
+ description: A regular expression that the name of a file must match
to be read, such as `.*\.csv` for the CSV files. The whole name must match,
upper and lower case count as the same, and only the name is matched, not its
directory. The other files stay on the server. When it is not set, all the
files are read.
+ type: string
+ example: '.*\.csv'
+ exclude:
+ title: Exclude Files
+ description: A regular expression for the names of the files not to
read, such as `.*\.(tmp|part)`. The whole name must match, upper and lower case
count as the same, and only the name is matched, not its directory. A file it
matches is not read, even when `include` or `antInclude` matches it too.
+ type: string
+ example: '.*\.(tmp|part)'
+ antInclude:
+ title: Include Files (Wildcards)
+ description: Wildcard patterns, separated by commas, that the path of
a file must match to be read, such as `*.csv,*.xml`. The path is relative to
`directoryName`, such as `2026/orders.csv` when `recursive` is true. `*`
matches any part of a name and `**` any number of directories, so `*.csv`
matches the CSV files in `directoryName` only, and `**/*.csv` those in the
subdirectories too. Upper and lower case differ. When it is not set, all the
files are read.
+ type: string
+ example: '**/*.csv'
+ preMove:
+ title: Move Before Read
+ description: Where to move each file on the server before it is read,
so that another reader of the same directory does not take it too. The same
format as `move`, such as `.inprogress`, a directory next to the file. `move`,
`moveFailed` and `delete` then act on the file in its new place, so with `move`
set to `.done` the file ends up in `.inprogress/.done`. When it is not set, a
file is read where it is.
+ type: string
+ example: ".inprogress"
+ readLock:
+ title: Wait Until Written
+ description: How to make sure that another program has finished
writing a file before it is read. `changed` reads a file once its size and
modification time have not changed between two checks 5 seconds apart, or, with
`readLockMinAge`, once the file is that old. A file still changing after 20
seconds is skipped until the next poll, and while a file is checked the other
files wait. `changed` never reads an empty file. `rename` reads a file once it
can be renamed. `none` reads the [...]
+ type: string
+ enum: ["none", "changed", "rename"]
+ example: changed
+ readLockMinAge:
+ title: Minimum File Age
+ description: 'Only used when `readLock` is `changed`: how old a file
must be, in milliseconds, before it is read, counted from its modification time
on the server. For example, 60000 leaves a file alone until it has not been
changed for one minute. A file that is not old enough is skipped until the next
poll, after 20 seconds of checks. The clocks of the server and of the
integration must agree. FTP servers give the modification time only to the
minute, and in their own time zone [...]
+ type: integer
+ example: 60000
+ delay:
+ title: Poll Interval
+ description: How long to wait between two polls of the directory, in
milliseconds. For example, 60000 polls once a minute. When it is not set, 500.
+ type: integer
+ example: 60000
+ maxMessagesPerPoll:
+ title: Max Files Per Poll
+ description: The largest number of files to read in one poll, such as
100. The other files are read at the next polls. The poll takes the first files
the server lists, and `sortBy` sorts only these. When it is not set, or 0, all
the files found are read.
+ type: integer
+ example: 100
+ sortBy:
+ title: Sort By
+ description: The order in which the files of a poll are read.
`file:name` sorts them by name, `file:modified` by modification time and
`file:length` by size. Put `reverse:` before it for the reverse order, such as
`reverse:file:modified` for the newest first. Separate several sorts with `;`,
such as `file:modified;file:name`. FTP servers give the modification time only
to the minute, so files changed in the same minute keep the order of the
server. When it is not set, the files a [...]
+ type: string
+ example: file:modified
+ streamDownload:
+ title: Stream Download
+ description: 'If true, each file is read from the server while it is
handed on, instead of being downloaded into memory first, and the body is a
stream. Use it for files too large for memory. It saves memory only when the
integration does not keep the stream in memory either: set
`camel.main.streamCachingSpoolEnabled=true`, which writes large bodies to a
temporary file, or turn stream caching off. It needs `stepwise` set to false,
or every file fails. When it is not set, false.'
+ type: boolean
+ stepwise:
+ title: Change Directory Stepwise
+ description: If true, the integration changes into the directory of a
file, one level at a time, to read it. If false, it uses the full path of the
file instead, which some servers need. It must be false when `streamDownload`
is true. When it is not set, true.
+ type: boolean
+ connectTimeout:
+ title: Connect Timeout
+ description: How long to wait for the connection to the FTPS server,
in milliseconds, and then for its first answer. A connection that fails is
tried 3 times before the poll is skipped. When it is not set, 10000.
+ type: integer
+ example: 10000
+ socketTimeout:
+ title: Socket Timeout
+ description: How long to wait for the answer of the server to a
command, in milliseconds, before the poll fails. When it is not set, 300000 (5
minutes).
+ type: integer
+ example: 60000
+ dataTimeout:
+ title: Data Timeout
+ description: How long to wait for data while a file or the list of the
files is transferred, in milliseconds, before the poll fails. When it is not
set, 30000.
+ type: integer
+ example: 30000
+ implicit:
+ title: Implicit TLS
+ description: 'If true, the connection uses implicit TLS: it is
encrypted from the start, usually on port 990. If false, it uses explicit TLS:
it connects to the normal FTP port, usually 21, and then switches to TLS. Use
the mode the server expects; the other one fails to connect. When it is not
set, false.'
+ type: boolean
dataTypes:
out:
headers:
@@ -148,16 +221,31 @@ spec:
host: "{{username}}@{{connectionHost}}"
port: "{{connectionPort}}"
directoryName: "{{directoryName}}"
+ antInclude: "RAW({{?antInclude}})"
autoCreate: "{{autoCreate}}"
binary: "{{binary}}"
+ connectTimeout: "{{?connectTimeout}}"
+ delay: "{{?delay}}"
delete: "{{delete}}"
+ exclude: "RAW({{?exclude}})"
idempotent: "{{idempotent}}"
idempotentRepository: "#bean:{{?idempotentRepository}}"
+ implicit: "{{?implicit}}"
+ include: "RAW({{?include}})"
+ maxMessagesPerPoll: "{{?maxMessagesPerPoll}}"
move: "{{?move}}"
moveFailed: "{{?moveFailed}}"
passiveMode: "{{passiveMode}}"
password: "RAW({{password}})"
+ preMove: "{{?preMove}}"
+ readLock: "{{?readLock}}"
+ readLockMinAge: "{{?readLockMinAge}}"
recursive: "{{recursive}}"
+ sortBy: "RAW({{?sortBy}})"
+ soTimeout: "{{?socketTimeout}}"
+ stepwise: "{{?stepwise}}"
+ streamDownload: "{{?streamDownload}}"
+ timeout: "{{?dataTimeout}}"
steps:
- setHeader:
name: file
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/scp-sink.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/scp-sink.kamelet.yaml
index 6a1397ae3..a85b74581 100644
--- a/library/camel-kamelets/src/main/resources/kamelets/scp-sink.kamelet.yaml
+++ b/library/camel-kamelets/src/main/resources/kamelets/scp-sink.kamelet.yaml
@@ -32,15 +32,15 @@ spec:
definition:
title: "SCP Sink"
description: |-
- Copies each message as a file to the home directory of a user on a
remote server, with SCP (secure copy) over an SSH connection.
+ Copies each message as a file to a directory on a remote server, with
SCP (secure copy) over an SSH connection.
- Use it to copy files to a server you reach with SSH that does not offer
SFTP, for example to drop a configuration or a report on a host. To choose the
directory on the server, or to append to or keep existing files, use
`sftp-sink` instead.
+ Use it to copy files to a server you reach with SSH that does not offer
SFTP, for example to drop a configuration or a report on a host. To append to
or keep existing files, or to create the directory when it is missing, use
`sftp-sink` instead.
Input: the body is the content of the file, as text, bytes or a stream.
The name of the file comes from the `CamelFileName` header, which may hold
subdirectories, such as `reports/data.csv`; without it, a unique name is
generated.
Output: the message is unchanged.
- Good to know: files are written in the home directory of `username`,
with the rights of that user, and a file with the same name is replaced. A file
name that points outside the home directory fails the message. The key of the
server is checked against the file in `knownHostsUri`, or else
`~/.ssh/known_hosts` of the user that runs the integration, unless
`strictHostKeyChecking` is `no`. Do not place it after messages from an
untrusted source.
+ Good to know: files are written in `directoryName`, or else in the home
directory of `username`, with the rights of that user, and a file with the same
name is replaced. `directoryName` must exist: when it does not, every file is
written as a file with the name of the directory. A file name that points
outside the directory fails the message. The key of the server is checked
against the file in `knownHostsUri`, or else `~/.ssh/known_hosts` of the user
that runs the integration, unl [...]
required:
- serverName
- serverPort
@@ -95,6 +95,18 @@ spec:
description: If true, the key of the server is looked up in
`~/.ssh/known_hosts` of the user that runs the integration.
type: boolean
default: true
+ directoryName:
+ title: Directory Name
+ description: The directory on the server to write the files to, such
as `incoming/reports` or `/data/incoming`. A path without a leading `/` is
relative to the home directory of `username`. It must exist, as it is not
created; when it is missing, each file is written as a file with the name of
the directory instead. When it is not set, the files are written in the home
directory.
+ type: string
+ pattern: "^[A-Za-z0-9_./~@+=,-]*$"
+ example: incoming/reports
+ chmod:
+ title: File Permissions
+ description: The permissions to give each new file, as three octal
digits, such as `640` for read and write by the owner and read by the group.
The umask of the server can take some away, so with the usual umask 022 the
`664` used when it is not set gives `644`. A file that already exists keeps its
permissions, and only its content is replaced.
+ type: string
+ pattern: "^[0-7]{3}$"
+ example: "640"
dataTypes:
in:
headers:
@@ -110,4 +122,4 @@ spec:
uri: kamelet:source
steps:
- to:
- uri:
"scp:{{serverName}}:{{serverPort}}?knownHostsFile={{?knownHostsUri}}&password=RAW({{?password}})&privateKeyFile=RAW({{?privateKeyFile}})&privateKeyFilePassphrase=RAW({{?privateKeyPassphrase}})&strictHostKeyChecking={{?strictHostKeyChecking}}&useUserKnownHostsFile={{?useUserKnownHostsFile}}&username=RAW({{?username}})"
+ uri:
"scp:{{serverName}}:{{serverPort}}/{{?directoryName}}?chmod={{?chmod}}&knownHostsFile={{?knownHostsUri}}&password=RAW({{?password}})&privateKeyFile=RAW({{?privateKeyFile}})&privateKeyFilePassphrase=RAW({{?privateKeyPassphrase}})&strictHostKeyChecking={{?strictHostKeyChecking}}&useUserKnownHostsFile={{?useUserKnownHostsFile}}&username=RAW({{?username}})"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/sftp-sink.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/sftp-sink.kamelet.yaml
index 5f732ea54..b0b41c3d0 100644
--- a/library/camel-kamelets/src/main/resources/kamelets/sftp-sink.kamelet.yaml
+++ b/library/camel-kamelets/src/main/resources/kamelets/sftp-sink.kamelet.yaml
@@ -140,6 +140,17 @@ spec:
description: The name of an empty file to write next to each file once
the file is complete, so that a reader knows the file can be read.
`$\{file:name}` stands for the name of the file and `$\{file:name.noext}` for
the name without its extension, so `$\{file:name}.done` writes
`orders.csv.done`, and `$\{file:name.noext}.ready` writes `orders.ready`. A
fixed name, such as `ready`, is written again after each file. No other
`$\{...}` is allowed, and with one the file is uploaded b [...]
type: string
example: "${file:name}.done"
+ chmod:
+ title: File Permissions
+ description: The permissions to give each file once it is written, as
three octal digits, such as `640` for read and write by the owner and read by
the group. They are set on a file that already existed too. When it is not set,
a new file gets the permissions the server gives it, usually `644`, and a file
that already existed keeps its own.
+ type: string
+ pattern: "^[0-7]{3}$"
+ example: "640"
+ serverAliveInterval:
+ title: Server Alive Interval
+ description: How often to check, in milliseconds, that the server
still answers while no data comes from it. When the server misses a check, the
connection is closed, and the next message makes a new one. Use it to notice a
lost connection soon, for example behind a firewall that drops idle
connections. When it is not set, no checks are made.
+ type: integer
+ example: 30000
dataTypes:
in:
headers:
@@ -204,6 +215,7 @@ spec:
host: "{{connectionHost}}"
port: "{{connectionPort}}"
directoryName: "{{directoryName}}"
+ chmod: "{{?chmod}}"
doneFileName: "RAW({{?doneFileName}})"
autoCreate: "{{autoCreate}}"
binary: "{{binary}}"
@@ -214,6 +226,7 @@ spec:
privateKeyFile: "RAW({{?privateKeyFile}})"
privateKeyPassphrase: "RAW({{?privateKeyPassphrase}})"
privateKeyUri: "RAW({{?privateKeyUri}})"
+ serverAliveInterval: "{{?serverAliveInterval}}"
strictHostKeyChecking: "{{?strictHostKeyChecking}}"
tempPrefix: "RAW({{?tempPrefix}})"
useUserKnownHostsFile: "{{?useUserKnownHostsFile}}"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/sftp-source.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/sftp-source.kamelet.yaml
index f4697f038..21ba7d6e4 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/sftp-source.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/sftp-source.kamelet.yaml
@@ -35,9 +35,9 @@ spec:
Use it to collect the files that a partner or another system drops on a
server you reach with SSH, such as orders, invoices or exports. For an FTP
server use `ftp-source` or `ftps-source`.
- Output: the body is the content of the file, as bytes. The `file` and
`ce-file` headers hold the name of the file, with its path below
`directoryName` when `recursive` is true, such as `2026/orders.csv`.
`CamelFileName` holds the same name, and `CamelFileLength` the size of the file
in bytes.
+ Output: the body is the content of the file, as bytes, or as a stream
when `streamDownload` is true. The `file` and `ce-file` headers hold the name
of the file, with its path below `directoryName` when `recursive` is true, such
as `2026/orders.csv`. `CamelFileName` holds the same name, and
`CamelFileLength` the size of the file in bytes.
- Good to know: files are not deleted or moved after they are read, unless
`delete` is true or `move` is set. A file is read only once while the
integration runs (`idempotent`), but the files left on the server are all read
again after a restart; to read each file only once, also across restarts, set
`move` to a directory such as `.done`, set `delete` to true, or keep the list
of the files read in a file with `idempotentRepository`. Log in with `username`
and either `password` or a p [...]
+ Good to know: files are not deleted or moved after they are read, unless
`delete` is true or `move` is set. A file is read only once while the
integration runs (`idempotent`), but the files left on the server are all read
again after a restart; to read each file only once, also across restarts, set
`move` to a directory such as `.done`, set `delete` to true, or keep the list
of the files read in a file with `idempotentRepository`. To read only some of
the files, set `include`, `exc [...]
required:
- connectionHost
- connectionPort
@@ -157,6 +157,71 @@ spec:
description: Where to move a file on the server when it could not be
handed on, for example because the next step failed, so that it is not read
over and over. The same format as `move`, such as `.error`. When it is not set,
a file that fails stays where it is and is read again at every poll. It can be
set together with `delete` or `move`.
type: string
example: ".error"
+ include:
+ title: Include Files
+ description: A regular expression that the name of a file must match
to be read, such as `.*\.csv` for the CSV files. The whole name must match,
upper and lower case count as the same, and only the name is matched, not its
directory. The other files stay on the server. When it is not set, all the
files are read.
+ type: string
+ example: '.*\.csv'
+ exclude:
+ title: Exclude Files
+ description: A regular expression for the names of the files not to
read, such as `.*\.(tmp|part)`. The whole name must match, upper and lower case
count as the same, and only the name is matched, not its directory. A file it
matches is not read, even when `include` or `antInclude` matches it too.
+ type: string
+ example: '.*\.(tmp|part)'
+ antInclude:
+ title: Include Files (Wildcards)
+ description: Wildcard patterns, separated by commas, that the path of
a file must match to be read, such as `*.csv,*.xml`. The path is relative to
`directoryName`, such as `2026/orders.csv` when `recursive` is true. `*`
matches any part of a name and `**` any number of directories, so `*.csv`
matches the CSV files in `directoryName` only, and `**/*.csv` those in the
subdirectories too. Upper and lower case differ. When it is not set, all the
files are read.
+ type: string
+ example: '**/*.csv'
+ preMove:
+ title: Move Before Read
+ description: Where to move each file on the server before it is read,
so that another reader of the same directory does not take it too. The same
format as `move`, such as `.inprogress`, a directory next to the file. `move`,
`moveFailed` and `delete` then act on the file in its new place, so with `move`
set to `.done` the file ends up in `.inprogress/.done`. When it is not set, a
file is read where it is.
+ type: string
+ example: ".inprogress"
+ readLock:
+ title: Wait Until Written
+ description: How to make sure that another program has finished
writing a file before it is read. `changed` reads a file once its size and
modification time have not changed between two checks 5 seconds apart, or, with
`readLockMinAge`, once the file is that old. A file still changing after 20
seconds is skipped until the next poll, and while a file is checked the other
files wait. `changed` never reads an empty file. `rename` reads a file once it
can be renamed. `none` reads the [...]
+ type: string
+ enum: ["none", "changed", "rename"]
+ example: changed
+ readLockMinAge:
+ title: Minimum File Age
+ description: 'Only used when `readLock` is `changed`: how old a file
must be, in milliseconds, before it is read, counted from its modification time
on the server. For example, 60000 leaves a file alone until it has not been
changed for one minute. A file that is not old enough is skipped until the next
poll, after 20 seconds of checks. The clocks of the server and of the
integration must agree.'
+ type: integer
+ example: 60000
+ delay:
+ title: Poll Interval
+ description: How long to wait between two polls of the directory, in
milliseconds. For example, 60000 polls once a minute. When it is not set, 500.
+ type: integer
+ example: 60000
+ maxMessagesPerPoll:
+ title: Max Files Per Poll
+ description: The largest number of files to read in one poll, such as
100. The other files are read at the next polls. The poll takes the first files
the server lists, and `sortBy` sorts only these. When it is not set, or 0, all
the files found are read.
+ type: integer
+ example: 100
+ sortBy:
+ title: Sort By
+ description: The order in which the files of a poll are read.
`file:name` sorts them by name, `file:modified` by modification time and
`file:length` by size. Put `reverse:` before it for the reverse order, such as
`reverse:file:modified` for the newest first. Separate several sorts with `;`,
such as `file:modified;file:name`. When it is not set, the files are read in
the order the server lists them.
+ type: string
+ example: file:modified
+ streamDownload:
+ title: Stream Download
+ description: 'If true, each file is read from the server while it is
handed on, instead of being downloaded into memory first, and the body is a
stream. Use it for files too large for memory. It saves memory only when the
integration does not keep the stream in memory either: set
`camel.main.streamCachingSpoolEnabled=true`, which writes large bodies to a
temporary file, or turn stream caching off. When it is not set, false.'
+ type: boolean
+ connectTimeout:
+ title: Connect Timeout
+ description: How long to wait for the connection to the SFTP server,
in milliseconds. A connection that fails is tried 3 times before the poll is
skipped. When it is not set, 10000.
+ type: integer
+ example: 10000
+ socketTimeout:
+ title: Socket Timeout
+ description: How long the connection may stay without an answer from
the server, in milliseconds, before it is closed and the poll fails. It is not
used when `serverAliveInterval` is set. When it is not set, 300000 (5 minutes).
+ type: integer
+ example: 60000
+ serverAliveInterval:
+ title: Server Alive Interval
+ description: How often to check, in milliseconds, that the server
still answers while no data comes from it. When the server misses a check, the
connection is closed and the poll fails, and a new connection is made at the
next poll. Use it to notice a lost connection soon, for example behind a
firewall that drops idle connections. When it is set, `socketTimeout` is not
used. When it is not set, no checks are made.
+ type: integer
+ example: 30000
dataTypes:
out:
headers:
@@ -186,29 +251,48 @@ spec:
host: "{{connectionHost}}"
port: "{{connectionPort}}"
directoryName: "{{directoryName}}"
+ antInclude: "RAW({{?antInclude}})"
autoCreate: "{{autoCreate}}"
binary: "{{binary}}"
+ connectTimeout: "{{?connectTimeout}}"
+ delay: "{{?delay}}"
delete: "{{delete}}"
+ exclude: "RAW({{?exclude}})"
idempotent: "{{idempotent}}"
idempotentRepository: "#bean:{{?idempotentRepository}}"
ignoreFileNotFoundOrPermissionError:
"{{ignoreFileNotFoundOrPermissionError}}"
+ include: "RAW({{?include}})"
knownHostsUri: "RAW({{?knownHostsUri}})"
+ maxMessagesPerPoll: "{{?maxMessagesPerPoll}}"
move: "{{?move}}"
moveFailed: "{{?moveFailed}}"
passiveMode: "{{passiveMode}}"
password: "RAW({{?password}})"
+ preMove: "{{?preMove}}"
privateKeyFile: "RAW({{?privateKeyFile}})"
privateKeyPassphrase: "RAW({{?privateKeyPassphrase}})"
privateKeyUri: "RAW({{?privateKeyUri}})"
+ readLock: "{{?readLock}}"
+ readLockMinAge: "{{?readLockMinAge}}"
recursive: "{{recursive}}"
+ serverAliveInterval: "{{?serverAliveInterval}}"
+ sortBy: "RAW({{?sortBy}})"
+ soTimeout: "{{?socketTimeout}}"
+ streamDownload: "{{?streamDownload}}"
strictHostKeyChecking: "{{?strictHostKeyChecking}}"
useUserKnownHostsFile: "{{?useUserKnownHostsFile}}"
username: "RAW({{?username}})"
steps:
- - setBody:
- expression:
- simple:
- expression: "${body.getBody()}"
+ - choice:
+ when:
+ - expression:
+ simple:
+ expression: "${body} is
'org.apache.camel.component.file.GenericFile'"
+ steps:
+ - setBody:
+ expression:
+ simple:
+ expression: "${body.getBody()}"
- setHeader:
name: file
expression: