This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel-kamelets.git
The following commit(s) were added to refs/heads/main by this push:
new f6d0ac47b chore: Tika and Docling pick the output format, Docling logs
in and sets OCR, PQC uses a key store (#3170) (#3230)
f6d0ac47b is described below
commit f6d0ac47bc7b14c52b4b8cdb654950895ce39bff
Author: Claus Ibsen <[email protected]>
AuthorDate: Wed Nov 18 22:36:39 2026 +0100
chore: Tika and Docling pick the output format, Docling logs in and sets
OCR, PQC uses a key store (#3170) (#3230)
tika-extract-text-action: outputFormat (text, html, xml), mapped to
tikaParseOutputFormat, default text as before.
docling-convert-action: operation (CONVERT_TO_MARKDOWN, the default as
before, CONVERT_TO_HTML, EXTRACT_TEXT); the component's outputFormat
applies only to async submits, so the format is chosen by the operation.
enableOcr (mapped to both enableOCR for the docling CLI and doOcr for
Docling Serve), ocrLanguage, forceOcr, ocrEngine, processTimeout,
documentTimeout, and a Docling Serve login with authenticationScheme
(API_KEY, BEARER) and authenticationToken, a secret in RAW().
pqc-signature-action and pqc-kem-action: keyStore and keyPair (bean
references), keyPairAlias and keyStorePassword (a secret in RAW()), so a
signature or an encapsulation can be checked or opened after a restart.
pqc-kem-action also takes symmetricKeyLength, which SEED and SM4 need.
The descriptions say that camel-pqc autowires a single KeyStore or
KeyPair of the application.
langchain4j-ingest-file-source: initialDelay, maxMessagesPerPoll and
readLockMinAge.
All new options are optional; with them unset the endpoint URIs are the
same as before. New partial pages document each Kamelet with an example.
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
.../docling-convert-action-description.adoc | 63 +++++++++++++++++++++
...langchain4j-ingest-file-source-description.adoc | 34 +++++++++++
.../ROOT/partials/pqc-kem-action-description.adoc | 65 ++++++++++++++++++++++
.../partials/pqc-signature-action-description.adoc | 49 ++++++++++++++++
.../tika-extract-text-action-description.adoc | 41 ++++++++++++++
kamelets/docling-convert-action.kamelet.yaml | 65 ++++++++++++++++++++--
.../langchain4j-ingest-file-source.kamelet.yaml | 20 ++++++-
kamelets/pqc-kem-action.kamelet.yaml | 36 +++++++++++-
kamelets/pqc-signature-action.kamelet.yaml | 28 +++++++++-
kamelets/tika-extract-text-action.kamelet.yaml | 14 +++--
.../main/resources/META-INF/kamelets-index.json | 14 ++---
.../kamelets/docling-convert-action.kamelet.yaml | 65 ++++++++++++++++++++--
.../langchain4j-ingest-file-source.kamelet.yaml | 20 ++++++-
.../resources/kamelets/pqc-kem-action.kamelet.yaml | 36 +++++++++++-
.../kamelets/pqc-signature-action.kamelet.yaml | 28 +++++++++-
.../kamelets/tika-extract-text-action.kamelet.yaml | 14 +++--
16 files changed, 561 insertions(+), 31 deletions(-)
diff --git a/docs/modules/ROOT/partials/docling-convert-action-description.adoc
b/docs/modules/ROOT/partials/docling-convert-action-description.adoc
new file mode 100644
index 000000000..e800b25fe
--- /dev/null
+++ b/docs/modules/ROOT/partials/docling-convert-action-description.adoc
@@ -0,0 +1,63 @@
+== Docling Convert Action Kamelet Description
+
+=== Docling Serve
+
+Set `useDoclingServe` to true and `doclingServeUrl` to the address of a
Docling Serve server, such as one started from the
`quay.io/docling-project/docling-serve` image. Without them, the Kamelet runs
the `docling` command-line program on the machine of the integration.
+
+A server started with an API key, in the `DOCLING_SERVE_API_KEY` environment
variable, answers `Unauthorized` to any request without it. Set
`authenticationScheme` to `API_KEY` and the key in `authenticationToken`: it is
sent in the `X-Api-Key` header. For a server behind a gateway that asks for a
token in the `Authorization` header, set `authenticationScheme` to `BEARER`
instead.
+
+=== Timeouts
+
+`processTimeout` is how long the Kamelet waits for the answer of the server,
in milliseconds, 30000 when it is not set. A server without a GPU can take
longer than that for the first document after it starts, while it loads its
models, and for a large PDF: the message then fails. Set it to a few minutes
for such a server.
+
+`documentTimeout` is how long the server may work on one document, in seconds.
When the time is up, the body holds only the part of the document converted so
far, and the message does not fail.
+
+=== OCR
+
+OCR (optical character recognition) reads the text in images, such as the
pages of a scanned PDF.
+
+[width="100%",cols="1,3",options="header"]
+|===
+| Option | Effect with Docling Serve
+| `enableOcr` not set
+| The server decides; the Docling Serve image reads the images.
+| `enableOcr: false`
+| The images are not read, which is faster. A scanned page gives no text.
+| `enableOcr: true`
+| The images are read, in the language of `ocrLanguage` (`en` when it is not
set).
+| `ocrLanguage`
+| A code the OCR engine knows, such as `de`; `eng` or `iso:en` for Tesseract.
Used only with `enableOcr: true`. A language the engine has no model for fails
the message.
+| `forceOcr: true`
+| Every page is read with OCR, and its text layer is replaced, for a PDF whose
text layer is wrong.
+| `ocrEngine`
+| The engine, which must be installed on the server. `TESSERACT` needs
`enableOcr: true` and an `ocrLanguage` installed for it; `OCRMAC` works only on
a Mac.
+|===
+
+=== Example
+
+The following route converts the PDF files of a directory with a Docling Serve
server that has an API key, reading scanned pages in German, and writes each
one as Markdown to another directory.
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:langchain4j-ingest-file-source"
+ parameters:
+ directory: "/data/scans"
+ include: "**/*.pdf"
+ steps:
+ - to:
+ uri: "kamelet:docling-convert-action"
+ parameters:
+ useDoclingServe: true
+ doclingServeUrl: "http://docling-serve:5001"
+ authenticationScheme: API_KEY
+ authenticationToken: "{{docling.api.key}}"
+ processTimeout: 300000
+ enableOcr: true
+ ocrLanguage: de
+ - to:
+ uri: "file:/data/scans-markdown"
+ parameters:
+ fileName: "$\{file:name.noext}.md"
+----
diff --git
a/docs/modules/ROOT/partials/langchain4j-ingest-file-source-description.adoc
b/docs/modules/ROOT/partials/langchain4j-ingest-file-source-description.adoc
new file mode 100644
index 000000000..89021618b
--- /dev/null
+++ b/docs/modules/ROOT/partials/langchain4j-ingest-file-source-description.adoc
@@ -0,0 +1,34 @@
+== LangChain4j Ingest File Source Kamelet Description
+
+=== When Files Are Read
+
+The Kamelet looks at the directory one second after it starts, or
`initialDelay` milliseconds, and then every `delay` milliseconds.
+
+By default, it reads each file only once its size and modification time have
not changed for one second, so that a file still being copied in is not read
half-written. That check costs one second per file, also for a file that has
not changed in months. With `readLockMinAge`, a file is read when it last
changed at least that many milliseconds ago: an older file is read at once, and
a newer one once it is old enough. Set it for a copy that can pause for more
than a second, such as over a [...]
+
+`maxMessagesPerPoll` limits how many files are read in one look at the
directory; the others are read in the next looks. It spreads the first load of
a large directory over time, such as for an embedding model with a rate limit.
+
+=== Example
+
+The following route loads the Markdown and PDF files of a directory into a
vector store. It starts 30 seconds after the application, reads at most 100
files a minute, and reads a file only one minute after it last changed.
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:langchain4j-ingest-file-source"
+ parameters:
+ directory: "/data/knowledge-base"
+ include: "**/*.md,**/*.pdf"
+ initialDelay: 30000
+ delay: 60000
+ maxMessagesPerPoll: 100
+ readLockMinAge: 60000
+ steps:
+ - to:
+ uri: "kamelet:tika-extract-text-action"
+ - to:
+ uri: "kamelet:langchain4j-ingest-sink"
+----
+
+The embedding store and the embedding model of `langchain4j-ingest-sink` are
beans of the application.
diff --git a/docs/modules/ROOT/partials/pqc-kem-action-description.adoc
b/docs/modules/ROOT/partials/pqc-kem-action-description.adoc
new file mode 100644
index 000000000..15d9c3f91
--- /dev/null
+++ b/docs/modules/ROOT/partials/pqc-kem-action-description.adoc
@@ -0,0 +1,65 @@
+== PQC Key Encapsulation/Decapsulation Action Kamelet Description
+
+=== Key Pair
+
+Without `keyStore` or `keyPair`, the key pair of the post-quantum algorithm is
created when the application starts. An encapsulation can then be opened only
by the same running application.
+
+To open an encapsulation after a restart, or in another application, keep the
key pair in a key store. Java 24 and later can make an ML-KEM key pair with
`keytool`. A key pair of a KEM cannot sign its own certificate, so `keytool`
needs a signing key pair in the same key store, given with `-signer`, such as
an ML-DSA one:
+
+[source,bash]
+----
+keytool -genkeypair -alias signer -keyalg ML-DSA -dname CN=signer \
+ -validity 365 -keystore pqc.p12 -storetype PKCS12 -storepass changeit
+keytool -genkeypair -alias kem -keyalg ML-KEM -groupname ML-KEM-768 -dname
CN=kem -signer signer \
+ -validity 365 -keystore pqc.p12 -storetype PKCS12 -storepass changeit
+----
+
+The key store is a Java `KeyStore` object, defined in the application, and
referred to with `#bean:` followed by its name. Set `keyPairAlias` to the alias
of the key pair, and `keyStorePassword` to the password of the key store, which
must also be the password of the key pair.
+
+When the application has exactly one Java `KeyStore`, or one `KeyPair`, the
Camel PQC component uses it even when `keyStore` or `keyPair` is not set. With
a key store, `keyPairAlias` and `keyStorePassword` must then be set too, else
the messages fail.
+
+=== Key Length
+
+`symmetricKeyLength` is the length of the secret key in bits, 256 when it is
not set. Use the same length to create and to open an encapsulation. Set it to
128 for `SEED` and `SM4`, which take only 128-bit keys: with 256, the
integration fails to start.
+
+=== Example
+
+The following route creates a 128-bit AES key and its encapsulation with the
key pair `kem` of the key store `pqc.p12`, and then takes the same key out of
the encapsulation again. The key store is loaded by a Groovy script in the
`beans` section, so that no Java class has to be written; Groovy needs the
`camel-groovy` dependency, which Camel JBang adds by itself.
+
+[source,yaml,subs='+attributes,macros']
+----
+- beans:
+ - name: pqcKeyStore
+ type: java.security.KeyStore
+ scriptLanguage: groovy
+ script: "java.security.KeyStore.getInstance(new File('pqc.p12'),
'{{pqc.keystore.password}}'.toCharArray())"
+- route:
+ from:
+ uri: "kamelet:timer-source"
+ parameters:
+ period: 10000
+ message: "start"
+ steps:
+ - to:
+ uri: "kamelet:pqc-kem-action"
+ parameters:
+ operation: generateSecretKeyEncapsulation
+ keyEncapsulationAlgorithm: MLKEM
+ symmetricKeyAlgorithm: AES
+ symmetricKeyLength: 128
+ keyStore: "#bean:pqcKeyStore"
+ keyPairAlias: kem
+ keyStorePassword: "{{pqc.keystore.password}}"
+ - to:
+ uri: "kamelet:pqc-kem-action"
+ parameters:
+ operation: extractSecretKeyEncapsulation
+ keyEncapsulationAlgorithm: MLKEM
+ symmetricKeyAlgorithm: AES
+ symmetricKeyLength: 128
+ keyStore: "#bean:pqcKeyStore"
+ keyPairAlias: kem
+ keyStorePassword: "{{pqc.keystore.password}}"
+ - to:
+ uri: "kamelet:log-sink"
+----
diff --git a/docs/modules/ROOT/partials/pqc-signature-action-description.adoc
b/docs/modules/ROOT/partials/pqc-signature-action-description.adoc
new file mode 100644
index 000000000..25f80ddc2
--- /dev/null
+++ b/docs/modules/ROOT/partials/pqc-signature-action-description.adoc
@@ -0,0 +1,49 @@
+== PQC Signature Action Kamelet Description
+
+=== Key Pair
+
+Without `keyStore` or `keyPair`, the key pair is created when the application
starts. A signature can then be checked only by the same running application,
and not after a restart.
+
+To sign with a key pair that stays the same, keep it in a key store. Java 24
and later can make an ML-DSA key pair with `keytool`:
+
+[source,bash]
+----
+keytool -genkeypair -alias signer -keyalg ML-DSA -groupname ML-DSA-65 -dname
CN=signer \
+ -validity 365 -keystore pqc.p12 -storetype PKCS12 -storepass changeit
+----
+
+The key store is a Java `KeyStore` object, defined in the application, and
referred to with `#bean:` followed by its name. Set `keyPairAlias` to the alias
of the key pair, and `keyStorePassword` to the password of the key store, which
must also be the password of the key pair. A signature made this way can be
checked after a restart, and by any application that has the public key, which
`keytool -exportcert` writes out as a certificate.
+
+When the application has exactly one Java `KeyStore`, or one `KeyPair`, the
Camel PQC component uses it even when `keyStore` or `keyPair` is not set. With
a key store, `keyPairAlias` and `keyStorePassword` must then be set too, else
the messages fail.
+
+=== Example
+
+The following route signs a message every 10 seconds with the key pair
`signer` of the key store `pqc.p12`. The key store is loaded by a Groovy script
in the `beans` section, so that no Java class has to be written; Groovy needs
the `camel-groovy` dependency, which Camel JBang adds by itself.
+
+[source,yaml,subs='+attributes,macros']
+----
+- beans:
+ - name: pqcKeyStore
+ type: java.security.KeyStore
+ scriptLanguage: groovy
+ script: "java.security.KeyStore.getInstance(new File('pqc.p12'),
'{{pqc.keystore.password}}'.toCharArray())"
+- route:
+ from:
+ uri: "kamelet:timer-source"
+ parameters:
+ period: 10000
+ message: "Hello"
+ steps:
+ - to:
+ uri: "kamelet:pqc-signature-action"
+ parameters:
+ operation: sign
+ signatureAlgorithm: MLDSA
+ keyStore: "#bean:pqcKeyStore"
+ keyPairAlias: signer
+ keyStorePassword: "{{pqc.keystore.password}}"
+ - to:
+ uri: "kamelet:log-sink"
+ parameters:
+ showHeaders: true
+----
diff --git
a/docs/modules/ROOT/partials/tika-extract-text-action-description.adoc
b/docs/modules/ROOT/partials/tika-extract-text-action-description.adoc
new file mode 100644
index 000000000..38745ac88
--- /dev/null
+++ b/docs/modules/ROOT/partials/tika-extract-text-action-description.adoc
@@ -0,0 +1,41 @@
+== Tika Extract Text Action Kamelet Description
+
+=== Output Formats
+
+`outputFormat` chooses what the body becomes:
+
+[width="100%",cols="1,3",options="header"]
+|===
+| `outputFormat` | Body
+| `text` (the default)
+| The plain text of the document, with its blocks on separate lines.
+| `html`
+| The document as an XHTML page. The `head` holds the metadata Tika found,
such as the title and the content type, as `meta` elements. The `body` holds
the text in paragraphs, and for some formats, such as HTML, in headings, lists
and tables too. A PDF gives a `div` per page.
+| `xml`
+| The same XHTML page, with an XML declaration on top.
+|===
+
+With `html` and `xml`, a document whose metadata holds a character that XML
does not allow, such as the NUL character that some PDF writers put in it,
fails the message; `text` reads it.
+
+=== Example
+
+The following route reads the PDF files of a directory and writes each one as
an XHTML page to another directory. The Tika PDF parser module must be added to
the integration, in the Tika version of your Camel, such as with `camel run
route.yaml --dep=org.apache.tika:tika-parser-pdf-module:4.1.0`.
+
+[source,yaml,subs='+attributes,macros']
+----
+- route:
+ from:
+ uri: "kamelet:langchain4j-ingest-file-source"
+ parameters:
+ directory: "/data/manuals"
+ include: "**/*.pdf"
+ steps:
+ - to:
+ uri: "kamelet:tika-extract-text-action"
+ parameters:
+ outputFormat: html
+ - to:
+ uri: "file:/data/manuals-html"
+ parameters:
+ fileName: "$\{file:name.noext}.html"
+----
diff --git a/kamelets/docling-convert-action.kamelet.yaml
b/kamelets/docling-convert-action.kamelet.yaml
index 30827997d..fd1358b93 100644
--- a/kamelets/docling-convert-action.kamelet.yaml
+++ b/kamelets/docling-convert-action.kamelet.yaml
@@ -32,15 +32,15 @@ spec:
definition:
title: "Docling Convert Action"
description: |-
- Turns each document, such as a PDF, an office document or an HTML page,
into Markdown text with Docling, keeping its headings, lists and tables.
+ Turns each document, such as a PDF, an office document or an HTML page,
into Markdown, HTML or plain text with Docling, keeping its headings, lists and
tables.
Use it to get well-structured text out of documents before an AI step,
for example in front of `langchain4j-ingest-sink` to load them into a vector
store. For plain text without a Docling server, use `tika-extract-text-action`.
Input: the document, as bytes, a stream, a file or text. The header
named in `documentIdHeader` (by default `CamelLangChain4jIngestDocumentId`) is
the id of the document, if any; it is kept for `langchain4j-ingest-sink`
further on.
- Output: the body is the document as Markdown, as text. The headers of
the message are kept.
+ Output: the body is the document as Markdown, as text, or as HTML or
plain text with `operation`. The headers of the message are kept.
- Good to know: to use a Docling Serve server, which is the usual setup,
set both `useDoclingServe` to true and `doclingServeUrl`; otherwise it runs the
`docling` command-line program, which must be installed on the machine of the
integration. With `maxDocumentSize`, a larger document fails the message
without being sent.
+ Good to know: to use a Docling Serve server, which is the usual setup,
set both `useDoclingServe` to true and `doclingServeUrl`; otherwise it runs the
`docling` command-line program, which must be installed on the machine of the
integration. A server started with an API key needs `authenticationScheme` set
to `API_KEY` and the key in `authenticationToken`. A conversion that takes
longer than `processTimeout`, 30 seconds when it is not set, fails the message;
the first conversion af [...]
type: object
properties:
documentIdHeader:
@@ -64,6 +64,53 @@ spec:
title: Use Docling Serve
description: If true, the documents are converted by the Docling Serve
server at `doclingServeUrl`. When it is not set, the `docling` command-line
program is run on the machine of the integration.
type: boolean
+ operation:
+ title: Operation
+ description: What the document is turned into. `CONVERT_TO_MARKDOWN`,
the default, gives Markdown text. `CONVERT_TO_HTML` gives a whole HTML page,
with its styles, as text. `EXTRACT_TEXT` gives the text without the Markdown
marks of headings and images; tables are still drawn with `|` characters.
+ type: string
+ enum: ["CONVERT_TO_MARKDOWN", "CONVERT_TO_HTML", "EXTRACT_TEXT"]
+ default: CONVERT_TO_MARKDOWN
+ enableOcr:
+ title: Enable OCR
+ description: If false, the text in images, such as the pages of a
scanned PDF, is not read (OCR, optical character recognition), which is faster,
and a scanned page gives no text. If true, OCR is asked for, in `ocrLanguage`.
When it is not set, a Docling Serve server uses its own setting, which is to
read the images.
+ type: boolean
+ ocrLanguage:
+ title: OCR Language
+ description: The language of the text in images, as a code that the
OCR engine knows, such as `en` or `de`, or `eng` or `iso:en` for Tesseract. The
engine must have a model for the language, else the message fails. With Docling
Serve, it is used only when `enableOcr` is true. When it is not set, `en` is
used.
+ type: string
+ pattern: "[A-Za-z_:-]+"
+ example: de
+ forceOcr:
+ title: Force OCR
+ description: If true, every page is read with OCR, even a page that
holds text, whose text is then replaced by what OCR reads. Use it for a PDF
whose text layer is wrong. Only used with Docling Serve.
+ type: boolean
+ ocrEngine:
+ title: OCR Engine
+ description: The OCR engine that reads the images, which must be
installed on the Docling Serve server, else the message fails; `OCRMAC` works
only on a Mac. With `TESSERACT`, also set `enableOcr` to true and `ocrLanguage`
to a language installed for it. When it is not set, the server chooses. Only
used with Docling Serve.
+ type: string
+ enum: ["AUTO", "EASYOCR", "RAPIDOCR", "TESSERACT", "TESSEROCR",
"OCRMAC"]
+ processTimeout:
+ title: Process Timeout
+ description: How long to wait for a document to be converted, in
milliseconds. A slower conversion fails the message. With Docling Serve, it is
how long to wait for the answer of the server. When it is not set, 30000 (30
seconds) is used, which is often too short for a large PDF.
+ type: integer
+ example: 300000
+ documentTimeout:
+ title: Document Timeout
+ description: How long the Docling Serve server may work on one
document, in seconds. When the time is up, the server stops, and the body holds
only the part of the document it converted, without failing the message. Only
used with Docling Serve. When it is not set, the server uses its own limit.
+ type: integer
+ example: 120
+ authenticationScheme:
+ title: Authentication Scheme
+ description: How to send `authenticationToken` to the Docling Serve
server. `API_KEY` sends it in the `X-Api-Key` header, which is what a Docling
Serve server started with an API key (`DOCLING_SERVE_API_KEY`) expects.
`BEARER` sends it in the `Authorization` header as `Bearer` followed by the
token, for a server behind a gateway that asks for one. Set it together with
`authenticationToken`.
+ type: string
+ enum: ["API_KEY", "BEARER"]
+ authenticationToken:
+ title: Authentication Token
+ description: The API key or token to log in to the Docling Serve
server with, sent as `authenticationScheme` says. Only used with Docling Serve.
+ type: string
+ format: password
+ x-descriptors:
+ - urn:camel:group:credentials
dataTypes:
in:
headers:
@@ -172,6 +219,16 @@ spec:
parameters:
operationId: convert
contentInBody: true
+ authenticationScheme: "{{?authenticationScheme}}"
+ authenticationToken: "RAW({{?authenticationToken}})"
doclingServeUrl: "{{?doclingServeUrl}}"
- operation: CONVERT_TO_MARKDOWN
+ # the OCR switch is read as enableOCR by the docling command
line, and as doOcr by Docling Serve
+ doOcr: "{{?enableOcr}}"
+ documentTimeout: "{{?documentTimeout}}"
+ enableOCR: "{{?enableOcr}}"
+ forceOcr: "{{?forceOcr}}"
+ ocrEngine: "{{?ocrEngine}}"
+ ocrLanguage: "{{?ocrLanguage}}"
+ operation: "{{operation}}"
+ processTimeout: "{{?processTimeout}}"
useDoclingServe: "{{?useDoclingServe}}"
diff --git a/kamelets/langchain4j-ingest-file-source.kamelet.yaml
b/kamelets/langchain4j-ingest-file-source.kamelet.yaml
index 96abbe3d0..e938a1374 100644
--- a/kamelets/langchain4j-ingest-file-source.kamelet.yaml
+++ b/kamelets/langchain4j-ingest-file-source.kamelet.yaml
@@ -38,7 +38,7 @@ spec:
Output: the body is the file, read as bytes unless `charset` is set. The
`CamelLangChain4jIngestDocumentId` header is the path of the file relative to
`directory`, such as `manuals/pump.pdf`, which `langchain4j-ingest-sink` uses
as the document id. The usual file headers, such as `CamelFileName`, are set
too.
- Good to know: the files are never deleted or moved. A file is read again
when its modification time or size changes, and `langchain4j-ingest-sink` then
adds the new version next to the old one, unless it has an
`idempotentRepository`, which skips it. The files already read are remembered
in memory only, by default the last 1000, so after a restart every file is read
again; set `idempotentRepository` to keep them. A file that is still being
copied in is read only once it stops changing.
+ Good to know: the files are never deleted or moved. A file is read again
when its modification time or size changes, and `langchain4j-ingest-sink` then
adds the new version next to the old one, unless it has an
`idempotentRepository`, which skips it. The files already read are remembered
in memory only, by default the last 1000, so after a restart every file is read
again; set `idempotentRepository` to keep them. A file that is still being
copied in is read only once it stops chang [...]
required:
- directory
type: object
@@ -69,6 +69,21 @@ spec:
description: How long to wait between two looks at the directory, in
milliseconds. When it is not set, the directory is looked at every 500
milliseconds; a longer delay, such as one minute, is usually enough for a
knowledge base.
type: integer
example: 60000
+ initialDelay:
+ title: Initial Delay
+ description: How long to wait after the integration starts before the
first look at the directory, in milliseconds. When it is not set, 1000 (one
second) is used.
+ type: integer
+ example: 30000
+ maxMessagesPerPoll:
+ title: Max Messages Per Poll
+ description: The largest number of files to read in one look at the
directory; the others are read in the next looks, one `delay` later. Use it to
spread out the first load of a large directory. Which files come first is not
defined. When it is not set, there is no limit.
+ type: integer
+ example: 100
+ readLockMinAge:
+ title: Read Lock Min Age
+ description: How long ago a file must have last changed before it is
read, in milliseconds, such as 60000 for a file copied in slowly. It replaces
the usual check, which reads each file only once its size and modification time
have not changed for one second, so that an older file is read at once, and a
newer one once it is old enough.
+ type: integer
+ example: 60000
charset:
title: Charset
description: The character set to read text files with, such as
`UTF-8`; the body is then text. Leave it unset when a
`tika-extract-text-action`, a `docling-convert-action` or a
`langchain4j-ingest-sink` with `modality` set to `media` comes next, because
they need the bytes of the file, and a conversion breaks a binary file such as
a PDF.
@@ -106,8 +121,11 @@ spec:
# an edited file gets a new key and re-ingests
idempotentKey: "${file:absolute.path}:${file:modified}:${file:size}"
idempotentRepository: "{{?idempotentRepository}}"
+ initialDelay: "{{?initialDelay}}"
+ maxMessagesPerPoll: "{{?maxMessagesPerPoll}}"
noop: true
readLock: changed
+ readLockMinAge: "{{?readLockMinAge}}"
recursive: "{{recursive}}"
steps:
- setHeader:
diff --git a/kamelets/pqc-kem-action.kamelet.yaml
b/kamelets/pqc-kem-action.kamelet.yaml
index 3a642859a..8b50686bd 100644
--- a/kamelets/pqc-kem-action.kamelet.yaml
+++ b/kamelets/pqc-kem-action.kamelet.yaml
@@ -39,7 +39,7 @@ spec:
Output: `generateSecretKeyEncapsulation` and
`extractSecretKeyEncapsulation` give a Java object that holds the secret key
and its encapsulation. `extractSecretKeyFromEncapsulation` gives the secret
key, as a Java object. None of them is text or bytes, so the next step must be
one that works with Java key objects.
- Good to know: the key pair of the post-quantum algorithm is created when
the application starts, and no option can set it, so an encapsulation can be
opened only by the same running application.
+ Good to know: unless the key pair of the post-quantum algorithm comes
from `keyStore` or `keyPair`, it is created when the application starts, so an
encapsulation can be opened only by the same running application. With a key
pair from a key store, it can be opened after a restart, and by any application
that has the key store. When the application has exactly one Java `KeyStore`,
or one `KeyPair`, it is used even when `keyStore` or `keyPair` is not set;
`keyPairAlias` and `keyStor [...]
required:
- keyEncapsulationAlgorithm
- operation
@@ -58,9 +58,36 @@ spec:
enum:
["generateSecretKeyEncapsulation","extractSecretKeyEncapsulation","extractSecretKeyFromEncapsulation"]
symmetricKeyAlgorithm:
title: Symmetric Key Algorithm
- description: The symmetric cipher the secret key is for, such as
`AES`, the usual choice. The cipher must accept a key of the length the Camel
PQC component creates (256 bits in recent Camel versions); with one that does
not, such as `SEED` or `SM4`, the integration fails to start.
+ description: The symmetric cipher the secret key is for, such as
`AES`, the usual choice. The cipher must accept a key of the length in
`symmetricKeyLength`, 256 bits when it is not set; with one that does not, such
as `SEED` or `SM4` with 256 bits, the integration fails to start.
type: string
enum:
["AES","RC2","RC5","ARIA","CAMELLIA","CAST5","CAST6","CHACHA7539","DSTU7624","GOST28147","GOST3412_2015","GRAIN128","HC128","HC256","SALSA20","SEED","SM4","DESEDE"]
+ symmetricKeyLength:
+ title: Symmetric Key Length
+ description: The length of the secret key, in bits, such as 128 or 256
for `AES`. Use the same length in the steps that create and that open an
encapsulation. When it is not set, 256 is used; set it to 128 for a cipher that
takes only 128-bit keys, such as `SEED` or `SM4`.
+ type: integer
+ example: 128
+ keyStore:
+ title: Key Store
+ description: A Java `KeyStore` defined in the application, as `#bean:`
followed by its name, that holds the key pair to use, such as a PKCS12 file
made with `keytool` (Java 24 or later makes ML-KEM keys). The key pair is then
the same after a restart, and in other applications that load the same key
store. Set `keyPairAlias` and `keyStorePassword` with it; without both, no key
pair is taken from the key store, none is created either, and the messages fail.
+ type: string
+ example: "#bean:pqcKeyStore"
+ keyPairAlias:
+ title: Key Pair Alias
+ description: The alias, the name in `keyStore`, of the ML-KEM key pair
to use.
+ type: string
+ example: kem
+ keyStorePassword:
+ title: Key Store Password
+ description: The password of `keyStore`. It is also the password of
the key pair in it, so both must be the same.
+ type: string
+ format: password
+ x-descriptors:
+ - urn:camel:group:credentials
+ keyPair:
+ title: Key Pair
+ description: A Java `KeyPair` of the algorithm, defined in the
application, as `#bean:` followed by its name, to use instead of `keyStore`.
+ type: string
+ example: "#bean:pqcKeyPair"
dependencies:
- "camel:pqc"
- "camel:kamelet"
@@ -73,5 +100,10 @@ spec:
parameters:
label: kem
keyEncapsulationAlgorithm: "{{keyEncapsulationAlgorithm}}"
+ keyPair: "{{?keyPair}}"
+ keyPairAlias: "{{?keyPairAlias}}"
+ keyStore: "{{?keyStore}}"
+ keyStorePassword: "RAW({{?keyStorePassword}})"
operation: "{{operation}}"
symmetricKeyAlgorithm: "{{symmetricKeyAlgorithm}}"
+ symmetricKeyLength: "{{?symmetricKeyLength}}"
diff --git a/kamelets/pqc-signature-action.kamelet.yaml
b/kamelets/pqc-signature-action.kamelet.yaml
index dadec73b4..a6ed14113 100644
--- a/kamelets/pqc-signature-action.kamelet.yaml
+++ b/kamelets/pqc-signature-action.kamelet.yaml
@@ -39,7 +39,7 @@ spec:
Output: the body is unchanged. `sign` sets the `CamelPQCSignature`
header to the signature, as bytes. `verify` sets the `CamelPQCVerification`
header to true when the signature matches the body, and to false when it does
not.
- Good to know: the key pair is created when the application starts, and
no option can set it, so a signature can be checked only by the same running
application, and not after a restart. A `verify` without a `CamelPQCSignature`
header fails the message.
+ Good to know: unless the key pair comes from `keyStore` or `keyPair`, it
is created when the application starts, so a signature can be checked only by
the same running application, and not after a restart. With a key pair from a
key store, a signature can be checked after a restart, and by any application
that has the public key. When the application has exactly one Java `KeyStore`,
or one `KeyPair`, it is used even when `keyStore` or `keyPair` is not set;
`keyPairAlias` and `keySt [...]
required:
- signatureAlgorithm
- operation
@@ -55,6 +55,28 @@ spec:
description: What to do with each message. `sign` adds a signature of
the body in the `CamelPQCSignature` header; `verify` checks the body against
that header and sets `CamelPQCVerification`.
type: string
enum: ["sign","verify"]
+ keyStore:
+ title: Key Store
+ description: A Java `KeyStore` defined in the application, as `#bean:`
followed by its name, that holds the key pair to use, such as a PKCS12 file
made with `keytool` (Java 24 or later makes ML-DSA keys). The key pair is then
the same after a restart, and in other applications that load the same key
store. Set `keyPairAlias` and `keyStorePassword` with it; without both, no key
pair is taken from the key store, none is created either, and the messages fail.
+ type: string
+ example: "#bean:pqcKeyStore"
+ keyPairAlias:
+ title: Key Pair Alias
+ description: The alias, the name in `keyStore`, of the ML-DSA key pair
to use.
+ type: string
+ example: signer
+ keyStorePassword:
+ title: Key Store Password
+ description: The password of `keyStore`. It is also the password of
the key pair in it, so both must be the same.
+ type: string
+ format: password
+ x-descriptors:
+ - urn:camel:group:credentials
+ keyPair:
+ title: Key Pair
+ description: A Java `KeyPair` of the algorithm, defined in the
application, as `#bean:` followed by its name, to use instead of `keyStore`.
+ type: string
+ example: "#bean:pqcKeyPair"
dataTypes:
in:
headers:
@@ -85,5 +107,9 @@ spec:
uri: pqc
parameters:
label: sign
+ keyPair: "{{?keyPair}}"
+ keyPairAlias: "{{?keyPairAlias}}"
+ keyStore: "{{?keyStore}}"
+ keyStorePassword: "RAW({{?keyStorePassword}})"
operation: "{{operation}}"
signatureAlgorithm: "{{signatureAlgorithm}}"
diff --git a/kamelets/tika-extract-text-action.kamelet.yaml
b/kamelets/tika-extract-text-action.kamelet.yaml
index ccb54b918..3beb9a6e1 100644
--- a/kamelets/tika-extract-text-action.kamelet.yaml
+++ b/kamelets/tika-extract-text-action.kamelet.yaml
@@ -32,13 +32,13 @@ spec:
definition:
title: "Tika Extract Text Action"
description: |-
- Turns each document, such as an HTML page, a text file or a PDF, into
its plain text with Apache Tika, inside the integration.
+ Turns each document, such as an HTML page, a text file or a PDF, into
its plain text, or into XHTML, with Apache Tika, inside the integration.
Use it to get the text of documents before searching, indexing or
summarising them, for example in front of `langchain4j-ingest-sink` to load
documents into a vector store for an AI assistant. For a richer conversion to
Markdown that keeps headings and tables, use `docling-convert-action`, which
needs a Docling server.
Input: the document, as bytes, a stream, a file or text. The header
named in `documentIdHeader` (by default `CamelLangChain4jIngestDocumentId`) is
the id of the document, if any; it is kept for `langchain4j-ingest-sink`
further on.
- Output: the body is the plain text of the document, as text in UTF-8,
with its blocks on separate lines. All the headers of the message are removed,
except the Camel ones, such as the file name headers of
`langchain4j-ingest-file-source`.
+ Output: the body is the plain text of the document, as text in UTF-8,
with its blocks on separate lines; with `outputFormat` set to `html` or `xml`,
it is the document as XHTML text instead. All the headers of the message are
removed, except the Camel ones, such as the file name headers of
`langchain4j-ingest-file-source`.
Good to know: only HTML and plain text are read out of the box. For PDF,
add the Tika PDF parser module (`org.apache.tika:tika-parser-pdf-module`, in
the Tika version of your Camel) to the integration, and add the other Tika
parser modules for other formats; without the module, a PDF gives an empty
text, and does not fail. With `maxDocumentSize`, a larger document fails the
message without being read.
type: object
@@ -49,6 +49,12 @@ spec:
type: string
default: CamelLangChain4jIngestDocumentId
example: CamelFileName
+ outputFormat:
+ title: Output Format
+ description: What the body becomes. `text`, the default, is the plain
text of the document. `html` is the document as an XHTML page, with its
paragraphs, and for some formats its headings, lists and tables, as HTML
elements, and with the metadata Tika found, such as the title, as `meta`
elements in the `head`. `xml` is the same page with an XML declaration on top.
With `html` and `xml`, a document whose metadata holds a character that XML
does not allow, such as the NUL character [...]
+ type: string
+ enum: ["text", "html", "xml"]
+ default: text
maxDocumentSize:
title: Max Document Size
description: The largest document to read, in bytes. A larger document
fails the message without being read. 0, the default, means no limit.
@@ -155,13 +161,13 @@ spec:
- throwException:
exceptionType:
java.lang.IllegalArgumentException
message: "Document
'${exchangeProperty.CamelLangChain4jIngestDocumentId}' exceeds maxDocumentSize
(more than {{maxDocumentSize}} bytes)"
- # tika answers plain text (body only, blocks newline-separated) at the
pinned encoding
+ # tika answers plain text (body only, blocks newline-separated), or
XHTML, at the pinned encoding
- to:
uri: tika
parameters:
operation: parse
tikaParseOutputEncoding: UTF-8
- tikaParseOutputFormat: text
+ tikaParseOutputFormat: "{{outputFormat}}"
# a message- or document-supplied charset header outranks the charset
the conversion
# below sets, so it is swept before the bytes are decoded at the
pinned UTF-8.
# These three steps are the declarative twin of the
camel-langchain4j-ingest
diff --git
a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
index 46ee4bd73..a653ac9e9 100644
--- a/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
+++ b/library/camel-kamelets/src/main/resources/META-INF/kamelets-index.json
@@ -674,8 +674,8 @@
"type" : "action",
"supportLevel" : "Preview",
"title" : "Docling Convert Action",
- "summary" : "Turns each document, such as a PDF, an office document or an
HTML page, into Markdown text with Docling, keeping its headings, lists and
tables.",
- "description" : "Turns each document, such as a PDF, an office document or
an HTML page, into Markdown text with Docling, keeping its headings, lists and
tables.\n\nUse it to get well-structured text out of documents before an AI
step, for example in front of `langchain4j-ingest-sink` to load them into a
vector store. For plain text without a Docling server, use
`tika-extract-text-action`.\n\nInput: the document, as bytes, a stream, a file
or text. The header named in `documentIdHead [...]
+ "summary" : "Turns each document, such as a PDF, an office document or an
HTML page, into Markdown, HTML or plain text with Docling, keeping its
headings, lists and tables.",
+ "description" : "Turns each document, such as a PDF, an office document or
an HTML page, into Markdown, HTML or plain text with Docling, keeping its
headings, lists and tables.\n\nUse it to get well-structured text out of
documents before an AI step, for example in front of `langchain4j-ingest-sink`
to load them into a vector store. For plain text without a Docling server, use
`tika-extract-text-action`.\n\nInput: the document, as bytes, a stream, a file
or text. The header named in [...]
"group" : "Actions",
"namespace" : "AI",
"provider" : "Apache Software Foundation"
@@ -1425,7 +1425,7 @@
"supportLevel" : "Preview",
"title" : "LangChain4j Ingest File Source",
"summary" : "Reads the documents in a directory and its subdirectories,
one message per file, leaves them in place, and reads a file again when it
changes.",
- "description" : "Reads the documents in a directory and its
subdirectories, one message per file, leaves them in place, and reads a file
again when it changes.\n\nUse it to feed a knowledge base for an AI assistant:
put it in front of `langchain4j-ingest-sink`, with `tika-extract-text-action`
or `docling-convert-action` in between for PDF or office documents.\n\nOutput:
the body is the file, read as bytes unless `charset` is set. The
`CamelLangChain4jIngestDocumentId` header is the p [...]
+ "description" : "Reads the documents in a directory and its
subdirectories, one message per file, leaves them in place, and reads a file
again when it changes.\n\nUse it to feed a knowledge base for an AI assistant:
put it in front of `langchain4j-ingest-sink`, with `tika-extract-text-action`
or `docling-convert-action` in between for PDF or office documents.\n\nOutput:
the body is the file, read as bytes unless `charset` is set. The
`CamelLangChain4jIngestDocumentId` header is the p [...]
"group" : "LangChain4j Ingest",
"namespace" : "AI",
"provider" : "Apache Software Foundation"
@@ -1765,7 +1765,7 @@
"supportLevel" : "Stable",
"title" : "PQC Key Encapsulation/Decapsulation Action",
"summary" : "Creates a new secret key for a symmetric cipher, such as AES,
together with its encapsulation by a post-quantum algorithm such as ML-KEM, or
takes the secret key back out of such an encapsulation.",
- "description" : "Creates a new secret key for a symmetric cipher, such as
AES, together with its encapsulation by a post-quantum algorithm such as
ML-KEM, or takes the secret key back out of such an encapsulation.\n\nUse it to
try out post-quantum key establishment (a KEM, key encapsulation mechanism):
one step creates a secret key and its encapsulation, the protected form of the
key that may be sent to the holder of the private key, and a later step takes
the same secret key out of [...]
+ "description" : "Creates a new secret key for a symmetric cipher, such as
AES, together with its encapsulation by a post-quantum algorithm such as
ML-KEM, or takes the secret key back out of such an encapsulation.\n\nUse it to
try out post-quantum key establishment (a KEM, key encapsulation mechanism):
one step creates a secret key and its encapsulation, the protected form of the
key that may be sent to the holder of the private key, and a later step takes
the same secret key out of [...]
"group" : "PQC",
"namespace" : "PQC",
"provider" : "Apache Software Foundation"
@@ -1775,7 +1775,7 @@
"supportLevel" : "Stable",
"title" : "PQC Signature Action",
"summary" : "Signs each message with a post-quantum digital signature
algorithm, such as ML-DSA, or checks the signature of a message signed before.",
- "description" : "Signs each message with a post-quantum digital signature
algorithm, such as ML-DSA, or checks the signature of a message signed
before.\n\nUse it to try out post-quantum signatures, or to check inside one
running application that a message was not changed between two steps: sign it
with `operation` set to `sign`, and check it later with a second copy of this
Kamelet with `operation` set to `verify` and the same algorithm.\n\nInput: the
body is the data to sign or to [...]
+ "description" : "Signs each message with a post-quantum digital signature
algorithm, such as ML-DSA, or checks the signature of a message signed
before.\n\nUse it to try out post-quantum signatures, or to check inside one
running application that a message was not changed between two steps: sign it
with `operation` set to `sign`, and check it later with a second copy of this
Kamelet with `operation` set to `verify` and the same algorithm.\n\nInput: the
body is the data to sign or to [...]
"group" : "PQC",
"namespace" : "PQC",
"provider" : "Apache Software Foundation"
@@ -2214,8 +2214,8 @@
"type" : "action",
"supportLevel" : "Preview",
"title" : "Tika Extract Text Action",
- "summary" : "Turns each document, such as an HTML page, a text file or a
PDF, into its plain text with Apache Tika, inside the integration.",
- "description" : "Turns each document, such as an HTML page, a text file or
a PDF, into its plain text with Apache Tika, inside the integration.\n\nUse it
to get the text of documents before searching, indexing or summarising them,
for example in front of `langchain4j-ingest-sink` to load documents into a
vector store for an AI assistant. For a richer conversion to Markdown that
keeps headings and tables, use `docling-convert-action`, which needs a Docling
server.\n\nInput: the docume [...]
+ "summary" : "Turns each document, such as an HTML page, a text file or a
PDF, into its plain text, or into XHTML, with Apache Tika, inside the
integration.",
+ "description" : "Turns each document, such as an HTML page, a text file or
a PDF, into its plain text, or into XHTML, with Apache Tika, inside the
integration.\n\nUse it to get the text of documents before searching, indexing
or summarising them, for example in front of `langchain4j-ingest-sink` to load
documents into a vector store for an AI assistant. For a richer conversion to
Markdown that keeps headings and tables, use `docling-convert-action`, which
needs a Docling server.\n\nI [...]
"group" : "Actions",
"namespace" : "AI",
"provider" : "Apache Software Foundation"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/docling-convert-action.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/docling-convert-action.kamelet.yaml
index 30827997d..fd1358b93 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/docling-convert-action.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/docling-convert-action.kamelet.yaml
@@ -32,15 +32,15 @@ spec:
definition:
title: "Docling Convert Action"
description: |-
- Turns each document, such as a PDF, an office document or an HTML page,
into Markdown text with Docling, keeping its headings, lists and tables.
+ Turns each document, such as a PDF, an office document or an HTML page,
into Markdown, HTML or plain text with Docling, keeping its headings, lists and
tables.
Use it to get well-structured text out of documents before an AI step,
for example in front of `langchain4j-ingest-sink` to load them into a vector
store. For plain text without a Docling server, use `tika-extract-text-action`.
Input: the document, as bytes, a stream, a file or text. The header
named in `documentIdHeader` (by default `CamelLangChain4jIngestDocumentId`) is
the id of the document, if any; it is kept for `langchain4j-ingest-sink`
further on.
- Output: the body is the document as Markdown, as text. The headers of
the message are kept.
+ Output: the body is the document as Markdown, as text, or as HTML or
plain text with `operation`. The headers of the message are kept.
- Good to know: to use a Docling Serve server, which is the usual setup,
set both `useDoclingServe` to true and `doclingServeUrl`; otherwise it runs the
`docling` command-line program, which must be installed on the machine of the
integration. With `maxDocumentSize`, a larger document fails the message
without being sent.
+ Good to know: to use a Docling Serve server, which is the usual setup,
set both `useDoclingServe` to true and `doclingServeUrl`; otherwise it runs the
`docling` command-line program, which must be installed on the machine of the
integration. A server started with an API key needs `authenticationScheme` set
to `API_KEY` and the key in `authenticationToken`. A conversion that takes
longer than `processTimeout`, 30 seconds when it is not set, fails the message;
the first conversion af [...]
type: object
properties:
documentIdHeader:
@@ -64,6 +64,53 @@ spec:
title: Use Docling Serve
description: If true, the documents are converted by the Docling Serve
server at `doclingServeUrl`. When it is not set, the `docling` command-line
program is run on the machine of the integration.
type: boolean
+ operation:
+ title: Operation
+ description: What the document is turned into. `CONVERT_TO_MARKDOWN`,
the default, gives Markdown text. `CONVERT_TO_HTML` gives a whole HTML page,
with its styles, as text. `EXTRACT_TEXT` gives the text without the Markdown
marks of headings and images; tables are still drawn with `|` characters.
+ type: string
+ enum: ["CONVERT_TO_MARKDOWN", "CONVERT_TO_HTML", "EXTRACT_TEXT"]
+ default: CONVERT_TO_MARKDOWN
+ enableOcr:
+ title: Enable OCR
+ description: If false, the text in images, such as the pages of a
scanned PDF, is not read (OCR, optical character recognition), which is faster,
and a scanned page gives no text. If true, OCR is asked for, in `ocrLanguage`.
When it is not set, a Docling Serve server uses its own setting, which is to
read the images.
+ type: boolean
+ ocrLanguage:
+ title: OCR Language
+ description: The language of the text in images, as a code that the
OCR engine knows, such as `en` or `de`, or `eng` or `iso:en` for Tesseract. The
engine must have a model for the language, else the message fails. With Docling
Serve, it is used only when `enableOcr` is true. When it is not set, `en` is
used.
+ type: string
+ pattern: "[A-Za-z_:-]+"
+ example: de
+ forceOcr:
+ title: Force OCR
+ description: If true, every page is read with OCR, even a page that
holds text, whose text is then replaced by what OCR reads. Use it for a PDF
whose text layer is wrong. Only used with Docling Serve.
+ type: boolean
+ ocrEngine:
+ title: OCR Engine
+ description: The OCR engine that reads the images, which must be
installed on the Docling Serve server, else the message fails; `OCRMAC` works
only on a Mac. With `TESSERACT`, also set `enableOcr` to true and `ocrLanguage`
to a language installed for it. When it is not set, the server chooses. Only
used with Docling Serve.
+ type: string
+ enum: ["AUTO", "EASYOCR", "RAPIDOCR", "TESSERACT", "TESSEROCR",
"OCRMAC"]
+ processTimeout:
+ title: Process Timeout
+ description: How long to wait for a document to be converted, in
milliseconds. A slower conversion fails the message. With Docling Serve, it is
how long to wait for the answer of the server. When it is not set, 30000 (30
seconds) is used, which is often too short for a large PDF.
+ type: integer
+ example: 300000
+ documentTimeout:
+ title: Document Timeout
+ description: How long the Docling Serve server may work on one
document, in seconds. When the time is up, the server stops, and the body holds
only the part of the document it converted, without failing the message. Only
used with Docling Serve. When it is not set, the server uses its own limit.
+ type: integer
+ example: 120
+ authenticationScheme:
+ title: Authentication Scheme
+ description: How to send `authenticationToken` to the Docling Serve
server. `API_KEY` sends it in the `X-Api-Key` header, which is what a Docling
Serve server started with an API key (`DOCLING_SERVE_API_KEY`) expects.
`BEARER` sends it in the `Authorization` header as `Bearer` followed by the
token, for a server behind a gateway that asks for one. Set it together with
`authenticationToken`.
+ type: string
+ enum: ["API_KEY", "BEARER"]
+ authenticationToken:
+ title: Authentication Token
+ description: The API key or token to log in to the Docling Serve
server with, sent as `authenticationScheme` says. Only used with Docling Serve.
+ type: string
+ format: password
+ x-descriptors:
+ - urn:camel:group:credentials
dataTypes:
in:
headers:
@@ -172,6 +219,16 @@ spec:
parameters:
operationId: convert
contentInBody: true
+ authenticationScheme: "{{?authenticationScheme}}"
+ authenticationToken: "RAW({{?authenticationToken}})"
doclingServeUrl: "{{?doclingServeUrl}}"
- operation: CONVERT_TO_MARKDOWN
+ # the OCR switch is read as enableOCR by the docling command
line, and as doOcr by Docling Serve
+ doOcr: "{{?enableOcr}}"
+ documentTimeout: "{{?documentTimeout}}"
+ enableOCR: "{{?enableOcr}}"
+ forceOcr: "{{?forceOcr}}"
+ ocrEngine: "{{?ocrEngine}}"
+ ocrLanguage: "{{?ocrLanguage}}"
+ operation: "{{operation}}"
+ processTimeout: "{{?processTimeout}}"
useDoclingServe: "{{?useDoclingServe}}"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/langchain4j-ingest-file-source.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/langchain4j-ingest-file-source.kamelet.yaml
index 96abbe3d0..e938a1374 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/langchain4j-ingest-file-source.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/langchain4j-ingest-file-source.kamelet.yaml
@@ -38,7 +38,7 @@ spec:
Output: the body is the file, read as bytes unless `charset` is set. The
`CamelLangChain4jIngestDocumentId` header is the path of the file relative to
`directory`, such as `manuals/pump.pdf`, which `langchain4j-ingest-sink` uses
as the document id. The usual file headers, such as `CamelFileName`, are set
too.
- Good to know: the files are never deleted or moved. A file is read again
when its modification time or size changes, and `langchain4j-ingest-sink` then
adds the new version next to the old one, unless it has an
`idempotentRepository`, which skips it. The files already read are remembered
in memory only, by default the last 1000, so after a restart every file is read
again; set `idempotentRepository` to keep them. A file that is still being
copied in is read only once it stops changing.
+ Good to know: the files are never deleted or moved. A file is read again
when its modification time or size changes, and `langchain4j-ingest-sink` then
adds the new version next to the old one, unless it has an
`idempotentRepository`, which skips it. The files already read are remembered
in memory only, by default the last 1000, so after a restart every file is read
again; set `idempotentRepository` to keep them. A file that is still being
copied in is read only once it stops chang [...]
required:
- directory
type: object
@@ -69,6 +69,21 @@ spec:
description: How long to wait between two looks at the directory, in
milliseconds. When it is not set, the directory is looked at every 500
milliseconds; a longer delay, such as one minute, is usually enough for a
knowledge base.
type: integer
example: 60000
+ initialDelay:
+ title: Initial Delay
+ description: How long to wait after the integration starts before the
first look at the directory, in milliseconds. When it is not set, 1000 (one
second) is used.
+ type: integer
+ example: 30000
+ maxMessagesPerPoll:
+ title: Max Messages Per Poll
+ description: The largest number of files to read in one look at the
directory; the others are read in the next looks, one `delay` later. Use it to
spread out the first load of a large directory. Which files come first is not
defined. When it is not set, there is no limit.
+ type: integer
+ example: 100
+ readLockMinAge:
+ title: Read Lock Min Age
+ description: How long ago a file must have last changed before it is
read, in milliseconds, such as 60000 for a file copied in slowly. It replaces
the usual check, which reads each file only once its size and modification time
have not changed for one second, so that an older file is read at once, and a
newer one once it is old enough.
+ type: integer
+ example: 60000
charset:
title: Charset
description: The character set to read text files with, such as
`UTF-8`; the body is then text. Leave it unset when a
`tika-extract-text-action`, a `docling-convert-action` or a
`langchain4j-ingest-sink` with `modality` set to `media` comes next, because
they need the bytes of the file, and a conversion breaks a binary file such as
a PDF.
@@ -106,8 +121,11 @@ spec:
# an edited file gets a new key and re-ingests
idempotentKey: "${file:absolute.path}:${file:modified}:${file:size}"
idempotentRepository: "{{?idempotentRepository}}"
+ initialDelay: "{{?initialDelay}}"
+ maxMessagesPerPoll: "{{?maxMessagesPerPoll}}"
noop: true
readLock: changed
+ readLockMinAge: "{{?readLockMinAge}}"
recursive: "{{recursive}}"
steps:
- setHeader:
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/pqc-kem-action.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/pqc-kem-action.kamelet.yaml
index 3a642859a..8b50686bd 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/pqc-kem-action.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/pqc-kem-action.kamelet.yaml
@@ -39,7 +39,7 @@ spec:
Output: `generateSecretKeyEncapsulation` and
`extractSecretKeyEncapsulation` give a Java object that holds the secret key
and its encapsulation. `extractSecretKeyFromEncapsulation` gives the secret
key, as a Java object. None of them is text or bytes, so the next step must be
one that works with Java key objects.
- Good to know: the key pair of the post-quantum algorithm is created when
the application starts, and no option can set it, so an encapsulation can be
opened only by the same running application.
+ Good to know: unless the key pair of the post-quantum algorithm comes
from `keyStore` or `keyPair`, it is created when the application starts, so an
encapsulation can be opened only by the same running application. With a key
pair from a key store, it can be opened after a restart, and by any application
that has the key store. When the application has exactly one Java `KeyStore`,
or one `KeyPair`, it is used even when `keyStore` or `keyPair` is not set;
`keyPairAlias` and `keyStor [...]
required:
- keyEncapsulationAlgorithm
- operation
@@ -58,9 +58,36 @@ spec:
enum:
["generateSecretKeyEncapsulation","extractSecretKeyEncapsulation","extractSecretKeyFromEncapsulation"]
symmetricKeyAlgorithm:
title: Symmetric Key Algorithm
- description: The symmetric cipher the secret key is for, such as
`AES`, the usual choice. The cipher must accept a key of the length the Camel
PQC component creates (256 bits in recent Camel versions); with one that does
not, such as `SEED` or `SM4`, the integration fails to start.
+ description: The symmetric cipher the secret key is for, such as
`AES`, the usual choice. The cipher must accept a key of the length in
`symmetricKeyLength`, 256 bits when it is not set; with one that does not, such
as `SEED` or `SM4` with 256 bits, the integration fails to start.
type: string
enum:
["AES","RC2","RC5","ARIA","CAMELLIA","CAST5","CAST6","CHACHA7539","DSTU7624","GOST28147","GOST3412_2015","GRAIN128","HC128","HC256","SALSA20","SEED","SM4","DESEDE"]
+ symmetricKeyLength:
+ title: Symmetric Key Length
+ description: The length of the secret key, in bits, such as 128 or 256
for `AES`. Use the same length in the steps that create and that open an
encapsulation. When it is not set, 256 is used; set it to 128 for a cipher that
takes only 128-bit keys, such as `SEED` or `SM4`.
+ type: integer
+ example: 128
+ keyStore:
+ title: Key Store
+ description: A Java `KeyStore` defined in the application, as `#bean:`
followed by its name, that holds the key pair to use, such as a PKCS12 file
made with `keytool` (Java 24 or later makes ML-KEM keys). The key pair is then
the same after a restart, and in other applications that load the same key
store. Set `keyPairAlias` and `keyStorePassword` with it; without both, no key
pair is taken from the key store, none is created either, and the messages fail.
+ type: string
+ example: "#bean:pqcKeyStore"
+ keyPairAlias:
+ title: Key Pair Alias
+ description: The alias, the name in `keyStore`, of the ML-KEM key pair
to use.
+ type: string
+ example: kem
+ keyStorePassword:
+ title: Key Store Password
+ description: The password of `keyStore`. It is also the password of
the key pair in it, so both must be the same.
+ type: string
+ format: password
+ x-descriptors:
+ - urn:camel:group:credentials
+ keyPair:
+ title: Key Pair
+ description: A Java `KeyPair` of the algorithm, defined in the
application, as `#bean:` followed by its name, to use instead of `keyStore`.
+ type: string
+ example: "#bean:pqcKeyPair"
dependencies:
- "camel:pqc"
- "camel:kamelet"
@@ -73,5 +100,10 @@ spec:
parameters:
label: kem
keyEncapsulationAlgorithm: "{{keyEncapsulationAlgorithm}}"
+ keyPair: "{{?keyPair}}"
+ keyPairAlias: "{{?keyPairAlias}}"
+ keyStore: "{{?keyStore}}"
+ keyStorePassword: "RAW({{?keyStorePassword}})"
operation: "{{operation}}"
symmetricKeyAlgorithm: "{{symmetricKeyAlgorithm}}"
+ symmetricKeyLength: "{{?symmetricKeyLength}}"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/pqc-signature-action.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/pqc-signature-action.kamelet.yaml
index dadec73b4..a6ed14113 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/pqc-signature-action.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/pqc-signature-action.kamelet.yaml
@@ -39,7 +39,7 @@ spec:
Output: the body is unchanged. `sign` sets the `CamelPQCSignature`
header to the signature, as bytes. `verify` sets the `CamelPQCVerification`
header to true when the signature matches the body, and to false when it does
not.
- Good to know: the key pair is created when the application starts, and
no option can set it, so a signature can be checked only by the same running
application, and not after a restart. A `verify` without a `CamelPQCSignature`
header fails the message.
+ Good to know: unless the key pair comes from `keyStore` or `keyPair`, it
is created when the application starts, so a signature can be checked only by
the same running application, and not after a restart. With a key pair from a
key store, a signature can be checked after a restart, and by any application
that has the public key. When the application has exactly one Java `KeyStore`,
or one `KeyPair`, it is used even when `keyStore` or `keyPair` is not set;
`keyPairAlias` and `keySt [...]
required:
- signatureAlgorithm
- operation
@@ -55,6 +55,28 @@ spec:
description: What to do with each message. `sign` adds a signature of
the body in the `CamelPQCSignature` header; `verify` checks the body against
that header and sets `CamelPQCVerification`.
type: string
enum: ["sign","verify"]
+ keyStore:
+ title: Key Store
+ description: A Java `KeyStore` defined in the application, as `#bean:`
followed by its name, that holds the key pair to use, such as a PKCS12 file
made with `keytool` (Java 24 or later makes ML-DSA keys). The key pair is then
the same after a restart, and in other applications that load the same key
store. Set `keyPairAlias` and `keyStorePassword` with it; without both, no key
pair is taken from the key store, none is created either, and the messages fail.
+ type: string
+ example: "#bean:pqcKeyStore"
+ keyPairAlias:
+ title: Key Pair Alias
+ description: The alias, the name in `keyStore`, of the ML-DSA key pair
to use.
+ type: string
+ example: signer
+ keyStorePassword:
+ title: Key Store Password
+ description: The password of `keyStore`. It is also the password of
the key pair in it, so both must be the same.
+ type: string
+ format: password
+ x-descriptors:
+ - urn:camel:group:credentials
+ keyPair:
+ title: Key Pair
+ description: A Java `KeyPair` of the algorithm, defined in the
application, as `#bean:` followed by its name, to use instead of `keyStore`.
+ type: string
+ example: "#bean:pqcKeyPair"
dataTypes:
in:
headers:
@@ -85,5 +107,9 @@ spec:
uri: pqc
parameters:
label: sign
+ keyPair: "{{?keyPair}}"
+ keyPairAlias: "{{?keyPairAlias}}"
+ keyStore: "{{?keyStore}}"
+ keyStorePassword: "RAW({{?keyStorePassword}})"
operation: "{{operation}}"
signatureAlgorithm: "{{signatureAlgorithm}}"
diff --git
a/library/camel-kamelets/src/main/resources/kamelets/tika-extract-text-action.kamelet.yaml
b/library/camel-kamelets/src/main/resources/kamelets/tika-extract-text-action.kamelet.yaml
index ccb54b918..3beb9a6e1 100644
---
a/library/camel-kamelets/src/main/resources/kamelets/tika-extract-text-action.kamelet.yaml
+++
b/library/camel-kamelets/src/main/resources/kamelets/tika-extract-text-action.kamelet.yaml
@@ -32,13 +32,13 @@ spec:
definition:
title: "Tika Extract Text Action"
description: |-
- Turns each document, such as an HTML page, a text file or a PDF, into
its plain text with Apache Tika, inside the integration.
+ Turns each document, such as an HTML page, a text file or a PDF, into
its plain text, or into XHTML, with Apache Tika, inside the integration.
Use it to get the text of documents before searching, indexing or
summarising them, for example in front of `langchain4j-ingest-sink` to load
documents into a vector store for an AI assistant. For a richer conversion to
Markdown that keeps headings and tables, use `docling-convert-action`, which
needs a Docling server.
Input: the document, as bytes, a stream, a file or text. The header
named in `documentIdHeader` (by default `CamelLangChain4jIngestDocumentId`) is
the id of the document, if any; it is kept for `langchain4j-ingest-sink`
further on.
- Output: the body is the plain text of the document, as text in UTF-8,
with its blocks on separate lines. All the headers of the message are removed,
except the Camel ones, such as the file name headers of
`langchain4j-ingest-file-source`.
+ Output: the body is the plain text of the document, as text in UTF-8,
with its blocks on separate lines; with `outputFormat` set to `html` or `xml`,
it is the document as XHTML text instead. All the headers of the message are
removed, except the Camel ones, such as the file name headers of
`langchain4j-ingest-file-source`.
Good to know: only HTML and plain text are read out of the box. For PDF,
add the Tika PDF parser module (`org.apache.tika:tika-parser-pdf-module`, in
the Tika version of your Camel) to the integration, and add the other Tika
parser modules for other formats; without the module, a PDF gives an empty
text, and does not fail. With `maxDocumentSize`, a larger document fails the
message without being read.
type: object
@@ -49,6 +49,12 @@ spec:
type: string
default: CamelLangChain4jIngestDocumentId
example: CamelFileName
+ outputFormat:
+ title: Output Format
+ description: What the body becomes. `text`, the default, is the plain
text of the document. `html` is the document as an XHTML page, with its
paragraphs, and for some formats its headings, lists and tables, as HTML
elements, and with the metadata Tika found, such as the title, as `meta`
elements in the `head`. `xml` is the same page with an XML declaration on top.
With `html` and `xml`, a document whose metadata holds a character that XML
does not allow, such as the NUL character [...]
+ type: string
+ enum: ["text", "html", "xml"]
+ default: text
maxDocumentSize:
title: Max Document Size
description: The largest document to read, in bytes. A larger document
fails the message without being read. 0, the default, means no limit.
@@ -155,13 +161,13 @@ spec:
- throwException:
exceptionType:
java.lang.IllegalArgumentException
message: "Document
'${exchangeProperty.CamelLangChain4jIngestDocumentId}' exceeds maxDocumentSize
(more than {{maxDocumentSize}} bytes)"
- # tika answers plain text (body only, blocks newline-separated) at the
pinned encoding
+ # tika answers plain text (body only, blocks newline-separated), or
XHTML, at the pinned encoding
- to:
uri: tika
parameters:
operation: parse
tikaParseOutputEncoding: UTF-8
- tikaParseOutputFormat: text
+ tikaParseOutputFormat: "{{outputFormat}}"
# a message- or document-supplied charset header outranks the charset
the conversion
# below sets, so it is swept before the bytes are decoded at the
pinned UTF-8.
# These three steps are the declarative twin of the
camel-langchain4j-ingest