This is an automated email from the ASF dual-hosted git repository. davsclaus pushed a commit to branch fix/CAMEL-23806-split-azure-blob-docs in repository https://gitbox.apache.org/repos/asf/camel.git
commit a0293cdeaec2befe9de6e503c5ecca45834a7e73 Author: Claus Ibsen <[email protected]> AuthorDate: Sat Jun 20 17:13:12 2026 +0200 CAMEL-23806: Split azure-storage-blob docs into focused sub-pages Split the Azure Storage Blob component documentation (1,956 lines) into 2 focused sub-pages, reducing the main page to 264 lines. - azure-storage-blob-operations: all 30 producer operation examples, blob snapshots/versions, and SAS token generation - azure-storage-blob-consumer: consumer patterns including delete after read and move after read Co-Authored-By: Claude <[email protected]> Signed-off-by: Claus Ibsen <[email protected]> --- .../main/docs/azure-storage-blob-component.adoc | 1699 +------------------- .../src/main/docs/azure-storage-blob-consumer.adoc | 282 ++++ ...ent.adoc => azure-storage-blob-operations.adoc} | 547 +------ docs/components/modules/others/nav.adoc | 2 + .../others/pages/azure-storage-blob-consumer.adoc | 1 + .../pages/azure-storage-blob-operations.adoc | 1 + 6 files changed, 297 insertions(+), 2235 deletions(-) diff --git a/components/camel-azure/camel-azure-storage-blob/src/main/docs/azure-storage-blob-component.adoc b/components/camel-azure/camel-azure-storage-blob/src/main/docs/azure-storage-blob-component.adoc index 6c02d53d2284..250efd0ace8a 100644 --- a/components/camel-azure/camel-azure-storage-blob/src/main/docs/azure-storage-blob-component.adoc +++ b/components/camel-azure/camel-azure-storage-blob/src/main/docs/azure-storage-blob-component.adoc @@ -239,1703 +239,12 @@ and existing blocks together. Any blocks not specified in the block list and per Refer to the example section in this page to learn how to use these operations into your camel application. -== Examples +== Sub-Pages -=== Consumer Examples +For more details on specific features, see: -To consume a blob into a file using the file component, this can be done like this: -[tabs] -==== -Java:: -+ -[source,java] ----- -from("azure-storage-blob://camelazure/container1?blobName=hello.txt&accountName=yourAccountName&accessKey=yourAccessKey") - .to("file://blobdirectory"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="azure-storage-blob://camelazure/container1?blobName=hello.txt&accountName=yourAccountName&accessKey=yourAccessKey"/> - <to uri="file://blobdirectory"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: azure-storage-blob://camelazure/container1 - parameters: - blobName: hello.txt - accountName: yourAccountName - accessKey: yourAccessKey - steps: - - to: - uri: file://blobdirectory ----- -==== - -However, you can also write to file directly without using the file component, you will need to specify `fileDir` folder path to save your blob in your machine. - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("azure-storage-blob://camelazure/container1?blobName=hello.txt&accountName=yourAccountName&accessKey=yourAccessKey&fileDir=/var/to/awesome/dir") - .to("mock:results"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="azure-storage-blob://camelazure/container1?blobName=hello.txt&accountName=yourAccountName&accessKey=yourAccessKey&fileDir=/var/to/awesome/dir"/> - <to uri="mock:results"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: azure-storage-blob://camelazure/container1 - parameters: - blobName: hello.txt - accountName: yourAccountName - accessKey: yourAccessKey - fileDir: /var/to/awesome/dir - steps: - - to: - uri: mock:results ----- -==== - -Also, the component supports batch consumer, hence you can consume multiple blobs with only specifying the container name, the consumer will -return multiple exchanges depending on the number of the blobs in the container. Example: - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("azure-storage-blob://camelazure/container1?accountName=yourAccountName&accessKey=yourAccessKey&fileDir=/var/to/awesome/dir") - .to("mock:results"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="azure-storage-blob://camelazure/container1?accountName=yourAccountName&accessKey=yourAccessKey&fileDir=/var/to/awesome/dir"/> - <to uri="mock:results"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: azure-storage-blob://camelazure/container1 - parameters: - accountName: yourAccountName - accessKey: yourAccessKey - fileDir: /var/to/awesome/dir - steps: - - to: - uri: mock:results ----- -==== - -==== Delete After Read - -The consumer supports automatic deletion of blobs after they have been successfully processed. This is useful when you want to ensure that blobs are only processed once. - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("azure-storage-blob://camelazure/container1?deleteAfterRead=true&accessKey=RAW(yourAccessKey)") - .log("Processing blob: ${header.CamelAzureStorageBlobBlobName}") - .to("direct:processBlob"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="azure-storage-blob://camelazure/container1?deleteAfterRead=true&accessKey=RAW(yourAccessKey)"/> - <log message="Processing blob: ${header.CamelAzureStorageBlobBlobName}"/> - <to uri="direct:processBlob"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: azure-storage-blob://camelazure/container1 - parameters: - deleteAfterRead: true - accessKey: "RAW(yourAccessKey)" - steps: - - log: - message: "Processing blob: ${header.CamelAzureStorageBlobBlobName}" - - to: - uri: direct:processBlob ----- -==== - -IMPORTANT: The delete operation is only performed if the Exchange is successfully committed. If processing fails or a rollback occurs, the blob will not be deleted and can be reprocessed on the next poll. - -NOTE: When `deleteAfterRead` is set to `false` (the default), the same blobs will be retrieved repeatedly in subsequent polls. In this case, you should use the Idempotent Consumer EIP to filter out duplicates based on the `CamelAzureStorageBlobBlobName` header. - -==== Move After Read - -The consumer can move blobs to a different container after successful processing. This is useful for archiving processed blobs or implementing a processing pipeline across containers. - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("azure-storage-blob://camelazure/incoming?moveAfterRead=true&destinationContainer=archive&accessKey=RAW(yourAccessKey)") - .log("Processing blob: ${header.CamelAzureStorageBlobBlobName}") - .to("direct:processBlob"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="azure-storage-blob://camelazure/incoming?moveAfterRead=true&destinationContainer=archive&accessKey=RAW(yourAccessKey)"/> - <log message="Processing blob: ${header.CamelAzureStorageBlobBlobName}"/> - <to uri="direct:processBlob"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: azure-storage-blob://camelazure/incoming - parameters: - moveAfterRead: true - destinationContainer: archive - accessKey: "RAW(yourAccessKey)" - steps: - - log: - message: "Processing blob: ${header.CamelAzureStorageBlobBlobName}" - - to: - uri: direct:processBlob ----- -==== - -You can also customize the destination blob name by adding a prefix and/or suffix: - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("azure-storage-blob://camelazure/source?moveAfterRead=true&destinationContainer=archive&prefix=incoming/&removePrefixOnMove=true&destinationBlobPrefix=processed/&destinationBlobSuffix=.done&accessKey=RAW(yourAccessKey)") - .log("Processing: ${header.CamelAzureStorageBlobBlobName}") - .to("direct:processBlob"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="azure-storage-blob://camelazure/source?moveAfterRead=true&destinationContainer=archive&prefix=incoming/&removePrefixOnMove=true&destinationBlobPrefix=processed/&destinationBlobSuffix=.done&accessKey=RAW(yourAccessKey)"/> - <log message="Processing: ${header.CamelAzureStorageBlobBlobName}"/> - <to uri="direct:processBlob"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: azure-storage-blob://camelazure/source - parameters: - moveAfterRead: true - destinationContainer: archive - prefix: "incoming/" - removePrefixOnMove: true - destinationBlobPrefix: "processed/" - destinationBlobSuffix: ".done" - accessKey: "RAW(yourAccessKey)" - steps: - - log: - message: "Processing: ${header.CamelAzureStorageBlobBlobName}" - - to: - uri: direct:processBlob ----- -==== - -The move operation works as follows: - -1. The blob is copied to the destination container with the new name -2. The original blob is deleted from the source container -3. Both operations only occur after the Exchange is successfully committed - -The following options control the move behavior: - -[width="100%",cols="25%,75%",options="header",] -|=== -|Option |Description -|`moveAfterRead` |Enable moving blobs after successful processing -|`destinationContainer` |Target container for moved blobs (required when `moveAfterRead=true`) -|`destinationBlobPrefix` |Prefix to add to the blob name in the destination -|`destinationBlobSuffix` |Suffix to add to the blob name in the destination -|`removePrefixOnMove` |Remove the source prefix from the blob name before adding destination prefix -|=== - -=== Producer Operations Examples -- `listBlobContainers`: - -NOTE: The `CamelAzureStorageBlobListBlobContainersOptions` header requires a `ListBlobContainersOptions` object, which must be set from a bean or processor. - -._Java-only: programmatic ListBlobContainersOptions_ - -[source,java] ----- -from("direct:start") - .process(exchange -> { - exchange.getIn().setHeader("CamelAzureStorageBlobListBlobContainersOptions", - new ListBlobContainersOptions().setMaxResultsPerPage(10)); - }) - .to("azure-storage-blob://camelazure?operation=listBlobContainers&serviceClient=#client") - .to("mock:result"); ----- - -- `createBlobContainer`: - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("direct:start") - .setHeader("CamelAzureStorageBlobBlobContainerName", constant("newContainerName")) - .to("azure-storage-blob://camelazure/container1?operation=createBlobContainer&serviceClient=#client") - .to("mock:result"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="direct:start"/> - <setHeader name="CamelAzureStorageBlobBlobContainerName"> - <constant>newContainerName</constant> - </setHeader> - <to uri="azure-storage-blob://camelazure/container1?operation=createBlobContainer&serviceClient=#client"/> - <to uri="mock:result"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: direct:start - steps: - - setHeader: - name: CamelAzureStorageBlobBlobContainerName - constant: newContainerName - - to: - uri: azure-storage-blob://camelazure/container1 - parameters: - operation: createBlobContainer - serviceClient: "#client" - - to: - uri: mock:result ----- -==== - -- `deleteBlobContainer`: - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("direct:start") - .setHeader("CamelAzureStorageBlobBlobContainerName", constant("overridenName")) - .to("azure-storage-blob://camelazure/container1?operation=deleteBlobContainer&serviceClient=#client") - .to("mock:result"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="direct:start"/> - <setHeader name="CamelAzureStorageBlobBlobContainerName"> - <constant>overridenName</constant> - </setHeader> - <to uri="azure-storage-blob://camelazure/container1?operation=deleteBlobContainer&serviceClient=#client"/> - <to uri="mock:result"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: direct:start - steps: - - setHeader: - name: CamelAzureStorageBlobBlobContainerName - constant: overridenName - - to: - uri: azure-storage-blob://camelazure/container1 - parameters: - operation: deleteBlobContainer - serviceClient: "#client" - - to: - uri: mock:result ----- -==== - -- `listBlobs`: - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("direct:start") - .setHeader("CamelAzureStorageBlobBlobContainerName", constant("overridenName")) - .to("azure-storage-blob://camelazure/container1?operation=listBlobs&serviceClient=#client") - .to("mock:result"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="direct:start"/> - <setHeader name="CamelAzureStorageBlobBlobContainerName"> - <constant>overridenName</constant> - </setHeader> - <to uri="azure-storage-blob://camelazure/container1?operation=listBlobs&serviceClient=#client"/> - <to uri="mock:result"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: direct:start - steps: - - setHeader: - name: CamelAzureStorageBlobBlobContainerName - constant: overridenName - - to: - uri: azure-storage-blob://camelazure/container1 - parameters: - operation: listBlobs - serviceClient: "#client" - - to: - uri: mock:result ----- -==== - - -- `listBlobVersions`: - -Returns every version of every blob in the container. Versioning must be enabled on the storage -account. Each `BlobItem` in the result carries its own `versionId` and `isCurrentVersion` flag. -The `prefix` and `regex` options can be used to narrow the result down to a single blob name. - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("direct:start") - .setHeader("CamelAzureStorageBlobPrefix", constant("invoice.pdf")) - .to("azure-storage-blob://camelazure/container1?operation=listBlobVersions&serviceClient=#client") - .log("${body}") - .to("mock:result"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="direct:start"/> - <setHeader name="CamelAzureStorageBlobPrefix"> - <constant>invoice.pdf</constant> - </setHeader> - <to uri="azure-storage-blob://camelazure/container1?operation=listBlobVersions&serviceClient=#client"/> - <log message="${body}"/> - <to uri="mock:result"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: direct:start - steps: - - setHeader: - name: CamelAzureStorageBlobPrefix - constant: invoice.pdf - - to: - uri: azure-storage-blob://camelazure/container1 - parameters: - operation: listBlobVersions - serviceClient: "#client" - - log: - message: "${body}" - - to: - uri: mock:result ----- -==== - - -- `getBlob`: - -We can either set an `outputStream` in the exchange body and write the data to it: - -._Java-only: programmatic OutputStream handling_ - -[source,java] ----- -from("direct:start") - .process(exchange -> { - exchange.getIn().setHeader("CamelAzureStorageBlobBlobContainerName", "overridenName"); - exchange.getIn().setBody(outputStream); - }) - .to("azure-storage-blob://camelazure/container1?blobName=blob&operation=getBlob&serviceClient=#client") - .to("mock:result"); ----- - -If we don't set a body, then this operation will give us an `InputStream` instance which can be processed further downstream: - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("direct:start") - .to("azure-storage-blob://camelazure/container1?blobName=blob&operation=getBlob&serviceClient=#client") - .log("${body}") - .to("mock:result"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="direct:start"/> - <to uri="azure-storage-blob://camelazure/container1?blobName=blob&operation=getBlob&serviceClient=#client"/> - <log message="${body}"/> - <to uri="mock:result"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: direct:start - steps: - - to: - uri: azure-storage-blob://camelazure/container1 - parameters: - blobName: blob - operation: getBlob - serviceClient: "#client" - - log: - message: "${body}" - - to: - uri: mock:result ----- -==== - -- `deleteBlob`: - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("direct:start") - .setHeader("CamelAzureStorageBlobBlobName", constant("overridenName")) - .to("azure-storage-blob://camelazure/container1?blobName=blob&operation=deleteBlob&serviceClient=#client") - .to("mock:result"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="direct:start"/> - <setHeader name="CamelAzureStorageBlobBlobName"> - <constant>overridenName</constant> - </setHeader> - <to uri="azure-storage-blob://camelazure/container1?blobName=blob&operation=deleteBlob&serviceClient=#client"/> - <to uri="mock:result"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: direct:start - steps: - - setHeader: - name: CamelAzureStorageBlobBlobName - constant: overridenName - - to: - uri: azure-storage-blob://camelazure/container1 - parameters: - blobName: blob - operation: deleteBlob - serviceClient: "#client" - - to: - uri: mock:result ----- -==== - -- `downloadBlobToFile`: - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("direct:start") - .setHeader("CamelAzureStorageBlobBlobName", constant("overridenName")) - .to("azure-storage-blob://camelazure/container1?blobName=blob&operation=downloadBlobToFile&fileDir=/var/mydir&serviceClient=#client") - .to("mock:result"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="direct:start"/> - <setHeader name="CamelAzureStorageBlobBlobName"> - <constant>overridenName</constant> - </setHeader> - <to uri="azure-storage-blob://camelazure/container1?blobName=blob&operation=downloadBlobToFile&fileDir=/var/mydir&serviceClient=#client"/> - <to uri="mock:result"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: direct:start - steps: - - setHeader: - name: CamelAzureStorageBlobBlobName - constant: overridenName - - to: - uri: azure-storage-blob://camelazure/container1 - parameters: - blobName: blob - operation: downloadBlobToFile - fileDir: /var/mydir - serviceClient: "#client" - - to: - uri: mock:result ----- -==== - -- `downloadLink` - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("direct:start") - .to("azure-storage-blob://camelazure/container1?blobName=blob&operation=downloadLink&serviceClient=#client") - .log("My link ${header.CamelAzureStorageBlobDownloadLink}") - .to("mock:result"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="direct:start"/> - <to uri="azure-storage-blob://camelazure/container1?blobName=blob&operation=downloadLink&serviceClient=#client"/> - <log message="My link ${header.CamelAzureStorageBlobDownloadLink}"/> - <to uri="mock:result"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: direct:start - steps: - - to: - uri: azure-storage-blob://camelazure/container1 - parameters: - blobName: blob - operation: downloadLink - serviceClient: "#client" - - log: - message: "My link ${header.CamelAzureStorageBlobDownloadLink}" - - to: - uri: mock:result ----- -==== - -- `uploadBlockBlob` - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("direct:start") - .setHeader("CamelAzureStorageBlobBlobName", constant("overridenName")) - .setBody(constant("Block Blob")) - .to("azure-storage-blob://camelazure/container1?blobName=blob&operation=uploadBlockBlob&serviceClient=#client") - .to("mock:result"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="direct:start"/> - <setHeader name="CamelAzureStorageBlobBlobName"> - <constant>overridenName</constant> - </setHeader> - <setBody> - <constant>Block Blob</constant> - </setBody> - <to uri="azure-storage-blob://camelazure/container1?blobName=blob&operation=uploadBlockBlob&serviceClient=#client"/> - <to uri="mock:result"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: direct:start - steps: - - setHeader: - name: CamelAzureStorageBlobBlobName - constant: overridenName - - setBody: - constant: Block Blob - - to: - uri: azure-storage-blob://camelazure/container1 - parameters: - blobName: blob - operation: uploadBlockBlob - serviceClient: "#client" - - to: - uri: mock:result ----- -==== - -- `uploadBlockBlobChunked` - -This operation is recommended for uploading large files (larger than 256MB) as it uses chunked parallel uploads for memory efficiency. - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("file://data?noop=true") - .log("Uploading file: ${header.CamelFileName}") - .toD("azure-storage-blob://camelazure/container1?blobName=${header.CamelFileName}&operation=uploadBlockBlobChunked&blockSize=52428800&maxConcurrency=4&serviceClient=#client") - .log("Upload completed: ${header.CamelFileName}"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="file://data?noop=true"/> - <log message="Uploading file: ${header.CamelFileName}"/> - <toD uri="azure-storage-blob://camelazure/container1?blobName=${header.CamelFileName}&operation=uploadBlockBlobChunked&blockSize=52428800&maxConcurrency=4&serviceClient=#client"/> - <log message="Upload completed: ${header.CamelFileName}"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: file://data - parameters: - noop: true - steps: - - log: - message: "Uploading file: ${header.CamelFileName}" - - toD: - uri: azure-storage-blob://camelazure/container1 - parameters: - blobName: "${header.CamelFileName}" - operation: uploadBlockBlobChunked - blockSize: 52428800 - maxConcurrency: 4 - serviceClient: "#client" - - log: - message: "Upload completed: ${header.CamelFileName}" ----- -==== - -The `blockSize` and `maxConcurrency` options control memory usage and upload speed: - -* `blockSize`: Size of each chunk (default: 4MB, max: 4000MB). Larger blocks = fewer requests but more memory. -* `maxConcurrency`: Number of parallel uploads (default: auto-detected based on CPU cores). Higher = faster but more memory. -* Memory usage is approximately `blockSize × maxConcurrency`. - -- `stageBlockBlobList` - -._Java-only: requires BlobBlock objects in the body_ - -[source,java] ----- -from("direct:start") - .process(exchange -> { - final List<BlobBlock> blocks = new LinkedList<>(); - blocks.add(BlobBlock.createBlobBlock(new ByteArrayInputStream("Hello".getBytes()))); - blocks.add(BlobBlock.createBlobBlock(new ByteArrayInputStream("From".getBytes()))); - blocks.add(BlobBlock.createBlobBlock(new ByteArrayInputStream("Camel".getBytes()))); - exchange.getIn().setBody(blocks); - }) - .to("azure-storage-blob://camelazure/container1?blobName=blob&operation=stageBlockBlobList&serviceClient=#client") - .to("mock:result"); ----- - -- `commitBlockBlobList` - -._Java-only: requires Block objects in the body_ - -[source,java] ----- -from("direct:start") - .process(exchange -> { - final List<Block> blockIds = new LinkedList<>(); - blockIds.add(new Block().setName("id-1")); - blockIds.add(new Block().setName("id-2")); - blockIds.add(new Block().setName("id-3")); - exchange.getIn().setBody(blockIds); - }) - .to("azure-storage-blob://camelazure/container1?blobName=blob&operation=commitBlockBlobList&serviceClient=#client") - .to("mock:result"); ----- - -- `getBlobBlockList` - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("direct:start") - .to("azure-storage-blob://camelazure/container1?blobName=blob&operation=getBlobBlockList&serviceClient=#client") - .log("${body}") - .to("mock:result"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="direct:start"/> - <to uri="azure-storage-blob://camelazure/container1?blobName=blob&operation=getBlobBlockList&serviceClient=#client"/> - <log message="${body}"/> - <to uri="mock:result"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: direct:start - steps: - - to: - uri: azure-storage-blob://camelazure/container1 - parameters: - blobName: blob - operation: getBlobBlockList - serviceClient: "#client" - - log: - message: "${body}" - - to: - uri: mock:result ----- -==== - - -- `createAppendBlob` - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("direct:start") - .to("azure-storage-blob://camelazure/container1?blobName=blob&operation=createAppendBlob&serviceClient=#client") - .to("mock:result"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="direct:start"/> - <to uri="azure-storage-blob://camelazure/container1?blobName=blob&operation=createAppendBlob&serviceClient=#client"/> - <to uri="mock:result"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: direct:start - steps: - - to: - uri: azure-storage-blob://camelazure/container1 - parameters: - blobName: blob - operation: createAppendBlob - serviceClient: "#client" - - to: - uri: mock:result ----- -==== - -- `commitAppendBlob` - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("direct:start") - .setBody(constant("Hello world from my awesome tests!")) - .to("azure-storage-blob://camelazure/container1?blobName=blob&operation=commitAppendBlob&serviceClient=#client") - .to("mock:result"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="direct:start"/> - <setBody> - <constant>Hello world from my awesome tests!</constant> - </setBody> - <to uri="azure-storage-blob://camelazure/container1?blobName=blob&operation=commitAppendBlob&serviceClient=#client"/> - <to uri="mock:result"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: direct:start - steps: - - setBody: - constant: "Hello world from my awesome tests!" - - to: - uri: azure-storage-blob://camelazure/container1 - parameters: - blobName: blob - operation: commitAppendBlob - serviceClient: "#client" - - to: - uri: mock:result ----- -==== - -- `createPageBlob` - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("direct:start") - .to("azure-storage-blob://camelazure/container1?blobName=blob&operation=createPageBlob&serviceClient=#client") - .to("mock:result"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="direct:start"/> - <to uri="azure-storage-blob://camelazure/container1?blobName=blob&operation=createPageBlob&serviceClient=#client"/> - <to uri="mock:result"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: direct:start - steps: - - to: - uri: azure-storage-blob://camelazure/container1 - parameters: - blobName: blob - operation: createPageBlob - serviceClient: "#client" - - to: - uri: mock:result ----- -==== - -- `uploadPageBlob` - -NOTE: The `CamelAzureStorageBlobPageBlobRange` header requires a `PageRange` object, which must be set from a bean or processor. - -._Java-only: requires PageRange object_ - -[source,java] ----- -from("direct:start") - .process(exchange -> { - byte[] dataBytes = new byte[512]; - new Random().nextBytes(dataBytes); - final InputStream dataStream = new ByteArrayInputStream(dataBytes); - final PageRange pageRange = new PageRange().setStart(0).setEnd(511); - exchange.getIn().setHeader("CamelAzureStorageBlobPageBlobRange", pageRange); - exchange.getIn().setBody(dataStream); - }) - .to("azure-storage-blob://camelazure/container1?blobName=blob&operation=uploadPageBlob&serviceClient=#client") - .to("mock:result"); ----- - -- `resizePageBlob` - -._Java-only: requires PageRange object_ - -[source,java] ----- -from("direct:start") - .process(exchange -> { - exchange.getIn().setHeader("CamelAzureStorageBlobPageBlobRange", - new PageRange().setStart(0).setEnd(511)); - }) - .to("azure-storage-blob://camelazure/container1?blobName=blob&operation=resizePageBlob&serviceClient=#client") - .to("mock:result"); ----- - -- `clearPageBlob` - -._Java-only: requires PageRange object_ - -[source,java] ----- -from("direct:start") - .process(exchange -> { - exchange.getIn().setHeader("CamelAzureStorageBlobPageBlobRange", - new PageRange().setStart(0).setEnd(511)); - }) - .to("azure-storage-blob://camelazure/container1?blobName=blob&operation=clearPageBlob&serviceClient=#client") - .to("mock:result"); ----- - -- `getPageBlobRanges` - -._Java-only: requires PageRange object_ - -[source,java] ----- -from("direct:start") - .process(exchange -> { - exchange.getIn().setHeader("CamelAzureStorageBlobPageBlobRange", - new PageRange().setStart(0).setEnd(511)); - }) - .to("azure-storage-blob://camelazure/container1?blobName=blob&operation=getPageBlobRanges&serviceClient=#client") - .log("${body}") - .to("mock:result"); ----- - -- `copyBlob` - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("direct:copyBlob") - .setHeader("CamelAzureStorageBlobBlobName", constant("file.txt")) - .setHeader("CamelAzureStorageBlobSourceBlobContainerName", constant("containerblob1")) - .setHeader("CamelAzureStorageBlobSourceBlobAccountName", constant("account")) - .to("azure-storage-blob://account/containerblob2?operation=copyBlob&sourceBlobAccessKey=RAW(accessKey)") - .to("mock:result"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="direct:copyBlob"/> - <setHeader name="CamelAzureStorageBlobBlobName"> - <constant>file.txt</constant> - </setHeader> - <setHeader name="CamelAzureStorageBlobSourceBlobContainerName"> - <constant>containerblob1</constant> - </setHeader> - <setHeader name="CamelAzureStorageBlobSourceBlobAccountName"> - <constant>account</constant> - </setHeader> - <to uri="azure-storage-blob://account/containerblob2?operation=copyBlob&sourceBlobAccessKey=RAW(accessKey)"/> - <to uri="mock:result"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: direct:copyBlob - steps: - - setHeader: - name: CamelAzureStorageBlobBlobName - constant: file.txt - - setHeader: - name: CamelAzureStorageBlobSourceBlobContainerName - constant: containerblob1 - - setHeader: - name: CamelAzureStorageBlobSourceBlobAccountName - constant: account - - to: - uri: azure-storage-blob://account/containerblob2 - parameters: - operation: copyBlob - sourceBlobAccessKey: "RAW(accessKey)" - - to: - uri: mock:result ----- -==== - -In this way the `file.txt` in the container `containerblob1` of the account `account`, will be copied to the container `containerblob2` of the same account. - -- `createBlobSnapshot` - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("direct:createBlobSnapshot") - .to("azure-storage-blob://camelazure/container1?blobName=hello.txt&operation=createBlobSnapshot&serviceClient=#client") - .log("Snapshot ID: ${header.CamelAzureStorageBlobSnapshotId}") - .to("mock:result"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="direct:createBlobSnapshot"/> - <to uri="azure-storage-blob://camelazure/container1?blobName=hello.txt&operation=createBlobSnapshot&serviceClient=#client"/> - <log message="Snapshot ID: ${header.CamelAzureStorageBlobSnapshotId}"/> - <to uri="mock:result"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: direct:createBlobSnapshot - steps: - - to: - uri: azure-storage-blob://camelazure/container1 - parameters: - blobName: hello.txt - operation: createBlobSnapshot - serviceClient: "#client" - - log: - message: "Snapshot ID: ${header.CamelAzureStorageBlobSnapshotId}" - - to: - uri: mock:result ----- -==== - -=== Reading a specific blob snapshot - -The `getBlob`, `downloadBlobToFile` and `downloadLink` operations can target a specific snapshot by setting the -`snapshotId` URI parameter or the `CamelAzureStorageBlobSnapshotId` exchange header. When set, the read is scoped -to the snapshot version of the blob instead of the live one. The header takes precedence over the URI parameter. - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("direct:readSnapshot") - .setHeader("CamelAzureStorageBlobSnapshotId", constant("2026-04-15T10:00:00.0000000Z")) - .to("azure-storage-blob://camelazure/container1?blobName=hello.txt&operation=getBlob&serviceClient=#client") - .to("mock:result"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="direct:readSnapshot"/> - <setHeader name="CamelAzureStorageBlobSnapshotId"> - <constant>2026-04-15T10:00:00.0000000Z</constant> - </setHeader> - <to uri="azure-storage-blob://camelazure/container1?blobName=hello.txt&operation=getBlob&serviceClient=#client"/> - <to uri="mock:result"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: direct:readSnapshot - steps: - - setHeader: - name: CamelAzureStorageBlobSnapshotId - constant: "2026-04-15T10:00:00.0000000Z" - - to: - uri: azure-storage-blob://camelazure/container1 - parameters: - blobName: hello.txt - operation: getBlob - serviceClient: "#client" - - to: - uri: mock:result ----- -==== - -=== Reading a specific blob version - -When blob versioning is enabled on the storage account, the `getBlob`, `downloadBlobToFile` and `downloadLink` -operations can target a specific version by setting the `versionId` URI parameter or the -`CamelAzureStorageBlobVersionId` exchange header. When set, the read is scoped to the version of the blob instead -of the live one. The header takes precedence over the URI parameter. - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("direct:readVersion") - .setHeader("CamelAzureStorageBlobVersionId", constant("2026-04-15T10:00:00.0000000Z")) - .to("azure-storage-blob://camelazure/container1?blobName=hello.txt&operation=getBlob&serviceClient=#client") - .to("mock:result"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="direct:readVersion"/> - <setHeader name="CamelAzureStorageBlobVersionId"> - <constant>2026-04-15T10:00:00.0000000Z</constant> - </setHeader> - <to uri="azure-storage-blob://camelazure/container1?blobName=hello.txt&operation=getBlob&serviceClient=#client"/> - <to uri="mock:result"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: direct:readVersion - steps: - - setHeader: - name: CamelAzureStorageBlobVersionId - constant: "2026-04-15T10:00:00.0000000Z" - - to: - uri: azure-storage-blob://camelazure/container1 - parameters: - blobName: hello.txt - operation: getBlob - serviceClient: "#client" - - to: - uri: mock:result ----- -==== - -- `setBlobTags` - -NOTE: The `CamelAzureStorageBlobTags` header requires a `Map<String, String>` value, which must be set from a bean or processor in XML/YAML. - -._Java-only: Map.of() for blob tags_ - -[source,java] ----- -from("direct:setBlobTags") - .setHeader("CamelAzureStorageBlobTags", constant(Map.of("status", "quarantine", "category", "document"))) - .to("azure-storage-blob://camelazure/container1?blobName=hello.txt&operation=setBlobTags&serviceClient=#client") - .to("mock:result"); ----- - -- `getBlobTags` - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("direct:getBlobTags") - .to("azure-storage-blob://camelazure/container1?blobName=hello.txt&operation=getBlobTags&serviceClient=#client") - .log("Tags: ${body}") - .to("mock:result"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="direct:getBlobTags"/> - <to uri="azure-storage-blob://camelazure/container1?blobName=hello.txt&operation=getBlobTags&serviceClient=#client"/> - <log message="Tags: ${body}"/> - <to uri="mock:result"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: direct:getBlobTags - steps: - - to: - uri: azure-storage-blob://camelazure/container1 - parameters: - blobName: hello.txt - operation: getBlobTags - serviceClient: "#client" - - log: - message: "Tags: ${body}" - - to: - uri: mock:result ----- -==== - -- `findBlobsByTags` - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("direct:findBlobsByTags") - .setHeader("CamelAzureStorageBlobTagFilter", constant("\"Environment\" = 'Production' AND \"Status\" = 'Active'")) - .to("azure-storage-blob://camelazure?operation=findBlobsByTags&serviceClient=#client") - .log("Matching blobs: ${body}") - .to("mock:result"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="direct:findBlobsByTags"/> - <setHeader name="CamelAzureStorageBlobTagFilter"> - <constant>"Environment" = 'Production' AND "Status" = 'Active'</constant> - </setHeader> - <to uri="azure-storage-blob://camelazure?operation=findBlobsByTags&serviceClient=#client"/> - <log message="Matching blobs: ${body}"/> - <to uri="mock:result"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: direct:findBlobsByTags - steps: - - setHeader: - name: CamelAzureStorageBlobTagFilter - constant: "\"Environment\" = 'Production' AND \"Status\" = 'Active'" - - to: - uri: azure-storage-blob://camelazure - parameters: - operation: findBlobsByTags - serviceClient: "#client" - - log: - message: "Matching blobs: ${body}" - - to: - uri: mock:result ----- -==== - -- `setBlobLegalHold` - -[tabs] -==== -Java:: -+ -[source,java] ----- -// place a legal hold on the blob -from("direct:setLegalHold") - .setHeader("CamelAzureStorageBlobLegalHold", constant(true)) - .to("azure-storage-blob://camelazure/container1?blobName=hello.txt&operation=setBlobLegalHold&serviceClient=#client") - .to("mock:result"); - -// clear the legal hold -from("direct:clearLegalHold") - .setHeader("CamelAzureStorageBlobLegalHold", constant(false)) - .to("azure-storage-blob://camelazure/container1?blobName=hello.txt&operation=setBlobLegalHold&serviceClient=#client") - .to("mock:result"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="direct:setLegalHold"/> - <setHeader name="CamelAzureStorageBlobLegalHold"> - <constant>true</constant> - </setHeader> - <to uri="azure-storage-blob://camelazure/container1?blobName=hello.txt&operation=setBlobLegalHold&serviceClient=#client"/> - <to uri="mock:result"/> -</route> - -<route> - <from uri="direct:clearLegalHold"/> - <setHeader name="CamelAzureStorageBlobLegalHold"> - <constant>false</constant> - </setHeader> - <to uri="azure-storage-blob://camelazure/container1?blobName=hello.txt&operation=setBlobLegalHold&serviceClient=#client"/> - <to uri="mock:result"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: direct:setLegalHold - steps: - - setHeader: - name: CamelAzureStorageBlobLegalHold - constant: true - - to: - uri: azure-storage-blob://camelazure/container1 - parameters: - blobName: hello.txt - operation: setBlobLegalHold - serviceClient: "#client" - - to: - uri: mock:result - -- route: - from: - uri: direct:clearLegalHold - steps: - - setHeader: - name: CamelAzureStorageBlobLegalHold - constant: false - - to: - uri: azure-storage-blob://camelazure/container1 - parameters: - blobName: hello.txt - operation: setBlobLegalHold - serviceClient: "#client" - - to: - uri: mock:result ----- -==== - -- `setBlobImmutabilityPolicy` - -._Java-only: requires OffsetDateTime and BlobImmutabilityPolicyMode objects_ - -[source,java] ----- -from("direct:setImmutabilityPolicy") - .setHeader("CamelAzureStorageBlobImmutabilityPolicyExpiryTime", constant(OffsetDateTime.now().plusDays(7))) - .setHeader("CamelAzureStorageBlobImmutabilityPolicyMode", constant(BlobImmutabilityPolicyMode.UNLOCKED)) - .to("azure-storage-blob://camelazure/container1?blobName=hello.txt&operation=setBlobImmutabilityPolicy&serviceClient=#client") - .to("mock:result"); ----- - -- `undeleteBlob` - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("direct:undeleteBlob") - .to("azure-storage-blob://camelazure/container1?blobName=hello.txt&operation=undeleteBlob&serviceClient=#client") - .to("mock:result"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="direct:undeleteBlob"/> - <to uri="azure-storage-blob://camelazure/container1?blobName=hello.txt&operation=undeleteBlob&serviceClient=#client"/> - <to uri="mock:result"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: direct:undeleteBlob - steps: - - to: - uri: azure-storage-blob://camelazure/container1 - parameters: - blobName: hello.txt - operation: undeleteBlob - serviceClient: "#client" - - to: - uri: mock:result ----- -==== - -- `setBlobTier` - -._Java-only: requires AccessTier and RehydratePriority enum objects_ - -[source,java] ----- -// move a blob to the COOL tier for less frequent access -from("direct:moveToCool") - .setHeader("CamelAzureStorageBlobAccessTier", constant(AccessTier.COOL)) - .to("azure-storage-blob://camelazure/container1?blobName=hello.txt&operation=setBlobTier&serviceClient=#client") - .to("mock:result"); - -// rehydrate a blob from ARCHIVE to HOT with high priority -from("direct:rehydrate") - .setHeader("CamelAzureStorageBlobAccessTier", constant(AccessTier.HOT)) - .setHeader("CamelAzureStorageBlobRehydratePriority", constant(RehydratePriority.HIGH)) - .to("azure-storage-blob://camelazure/container1?blobName=archived.txt&operation=setBlobTier&serviceClient=#client") - .to("mock:result"); ----- - -=== Blob modification during download (ConditionNotMet) - -When downloading a blob via streaming (the `getBlob` operation or the consumer without `fileDir`), the Azure SDK reads the blob in chunks. -On the first chunk it captures the blob's ETag and uses it as an `if-match` condition on subsequent chunk requests. -If the blob is modified by another process between chunks (which changes its ETag), Azure returns HTTP 412 `ConditionNotMet`. - -This is by design in the Azure SDK — it ensures read consistency so that you do not receive half of one version concatenated with half of another. -The error is more likely with larger blobs and slower network connections (e.g., in real Azure environments versus local emulators). - -Workarounds: - -* Use the `fileDir` option so the consumer downloads the blob atomically to a file via `downloadBlobToFile` instead of streaming. -* Use blob snapshots — read from an immutable snapshot ID via the `snapshotId` option or the `CamelAzureStorageBlobSnapshotId` header, which cannot be modified. -* Use blob versioning — read from a specific version via the `versionId` option or the `CamelAzureStorageBlobVersionId` header. -* Avoid modifying blobs while they are being consumed. - -=== SAS Token generation example - -SAS Blob Container tokens can be generated programmatically or via Azure UI. To generate the token with Java code, the following can be done: - -._Java-only: programmatic SAS token generation_ - -[source,java] ----- -BlobContainerClient blobClient = new BlobContainerClientBuilder() - .endpoint(String.format("https://%s.blob.core.windows.net/%s";, accountName, accessKey)) - .containerName(containerName) - .credential(new StorageSharedKeyCredential(accountName, accessKey)) - .buildClient(); - -OffsetDateTime expiryTime = OffsetDateTime.now().plusDays(1); - -BlobContainerSasPermission blobContainerSasPermission = new BlobContainerSasPermission() - .setWritePermission(true) - .setListPermission(true) - .setCreatePermission(true) - .setDeletePermission(true) - .setAddPermission(true) - .setReadPermission(true); - -BlobServiceSasSignatureValues sasSignatureValues = new BlobServiceSasSignatureValues(expiryTime, blobContainerSasPermission); - -return blobClient.generateSas(sasSignatureValues); ----- - -The generated SAS token can be then stored to an application.properties file so that it can be loaded by the Camel route, for example: - -[source,properties] ----- -camel.component.azure-storage-blob.sas-token=MY_TOKEN_HERE ----- - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("direct:uploadBlob") - .to("azure-storage-blob://account/containerblob2?operation=uploadBlockBlob&credentialType=AZURE_SAS"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="direct:uploadBlob"/> - <to uri="azure-storage-blob://account/containerblob2?operation=uploadBlockBlob&credentialType=AZURE_SAS"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: direct:uploadBlob - steps: - - to: - uri: azure-storage-blob://account/containerblob2 - parameters: - operation: uploadBlockBlob - credentialType: AZURE_SAS ----- -==== +* xref:others:azure-storage-blob-operations.adoc[Producer Operations] - All producer operation examples, blob snapshots/versions, and SAS token generation +* xref:others:azure-storage-blob-consumer.adoc[Consumer Examples] - Consumer patterns including delete after read and move after read == Important Development Notes diff --git a/components/camel-azure/camel-azure-storage-blob/src/main/docs/azure-storage-blob-consumer.adoc b/components/camel-azure/camel-azure-storage-blob/src/main/docs/azure-storage-blob-consumer.adoc new file mode 100644 index 000000000000..a49f9cc45a4a --- /dev/null +++ b/components/camel-azure/camel-azure-storage-blob/src/main/docs/azure-storage-blob-consumer.adoc @@ -0,0 +1,282 @@ += Azure Storage Blob - Consumer Examples +:tabs-sync-option: + +xref:ROOT:azure-storage-blob-component.adoc[Back to Azure Storage Blob Component] + +== Consumer Examples + +To consume a blob into a file using the file component, this can be done like this: +[tabs] +==== +Java:: ++ +[source,java] +---- +from("azure-storage-blob://camelazure/container1?blobName=hello.txt&accountName=yourAccountName&accessKey=yourAccessKey") + .to("file://blobdirectory"); +---- + +XML:: ++ +[source,xml] +---- +<route> + <from uri="azure-storage-blob://camelazure/container1?blobName=hello.txt&accountName=yourAccountName&accessKey=yourAccessKey"/> + <to uri="file://blobdirectory"/> +</route> +---- + +YAML:: ++ +[source,yaml] +---- +- route: + from: + uri: azure-storage-blob://camelazure/container1 + parameters: + blobName: hello.txt + accountName: yourAccountName + accessKey: yourAccessKey + steps: + - to: + uri: file://blobdirectory +---- +==== + +However, you can also write to file directly without using the file component, you will need to specify `fileDir` folder path to save your blob in your machine. + +[tabs] +==== +Java:: ++ +[source,java] +---- +from("azure-storage-blob://camelazure/container1?blobName=hello.txt&accountName=yourAccountName&accessKey=yourAccessKey&fileDir=/var/to/awesome/dir") + .to("mock:results"); +---- + +XML:: ++ +[source,xml] +---- +<route> + <from uri="azure-storage-blob://camelazure/container1?blobName=hello.txt&accountName=yourAccountName&accessKey=yourAccessKey&fileDir=/var/to/awesome/dir"/> + <to uri="mock:results"/> +</route> +---- + +YAML:: ++ +[source,yaml] +---- +- route: + from: + uri: azure-storage-blob://camelazure/container1 + parameters: + blobName: hello.txt + accountName: yourAccountName + accessKey: yourAccessKey + fileDir: /var/to/awesome/dir + steps: + - to: + uri: mock:results +---- +==== + +Also, the component supports batch consumer, hence you can consume multiple blobs with only specifying the container name, the consumer will +return multiple exchanges depending on the number of the blobs in the container. Example: + +[tabs] +==== +Java:: ++ +[source,java] +---- +from("azure-storage-blob://camelazure/container1?accountName=yourAccountName&accessKey=yourAccessKey&fileDir=/var/to/awesome/dir") + .to("mock:results"); +---- + +XML:: ++ +[source,xml] +---- +<route> + <from uri="azure-storage-blob://camelazure/container1?accountName=yourAccountName&accessKey=yourAccessKey&fileDir=/var/to/awesome/dir"/> + <to uri="mock:results"/> +</route> +---- + +YAML:: ++ +[source,yaml] +---- +- route: + from: + uri: azure-storage-blob://camelazure/container1 + parameters: + accountName: yourAccountName + accessKey: yourAccessKey + fileDir: /var/to/awesome/dir + steps: + - to: + uri: mock:results +---- +==== + +=== Delete After Read + +The consumer supports automatic deletion of blobs after they have been successfully processed. This is useful when you want to ensure that blobs are only processed once. + +[tabs] +==== +Java:: ++ +[source,java] +---- +from("azure-storage-blob://camelazure/container1?deleteAfterRead=true&accessKey=RAW(yourAccessKey)") + .log("Processing blob: ${header.CamelAzureStorageBlobBlobName}") + .to("direct:processBlob"); +---- + +XML:: ++ +[source,xml] +---- +<route> + <from uri="azure-storage-blob://camelazure/container1?deleteAfterRead=true&accessKey=RAW(yourAccessKey)"/> + <log message="Processing blob: ${header.CamelAzureStorageBlobBlobName}"/> + <to uri="direct:processBlob"/> +</route> +---- + +YAML:: ++ +[source,yaml] +---- +- route: + from: + uri: azure-storage-blob://camelazure/container1 + parameters: + deleteAfterRead: true + accessKey: "RAW(yourAccessKey)" + steps: + - log: + message: "Processing blob: ${header.CamelAzureStorageBlobBlobName}" + - to: + uri: direct:processBlob +---- +==== + +IMPORTANT: The delete operation is only performed if the Exchange is successfully committed. If processing fails or a rollback occurs, the blob will not be deleted and can be reprocessed on the next poll. + +NOTE: When `deleteAfterRead` is set to `false` (the default), the same blobs will be retrieved repeatedly in subsequent polls. In this case, you should use the Idempotent Consumer EIP to filter out duplicates based on the `CamelAzureStorageBlobBlobName` header. + +=== Move After Read + +The consumer can move blobs to a different container after successful processing. This is useful for archiving processed blobs or implementing a processing pipeline across containers. + +[tabs] +==== +Java:: ++ +[source,java] +---- +from("azure-storage-blob://camelazure/incoming?moveAfterRead=true&destinationContainer=archive&accessKey=RAW(yourAccessKey)") + .log("Processing blob: ${header.CamelAzureStorageBlobBlobName}") + .to("direct:processBlob"); +---- + +XML:: ++ +[source,xml] +---- +<route> + <from uri="azure-storage-blob://camelazure/incoming?moveAfterRead=true&destinationContainer=archive&accessKey=RAW(yourAccessKey)"/> + <log message="Processing blob: ${header.CamelAzureStorageBlobBlobName}"/> + <to uri="direct:processBlob"/> +</route> +---- + +YAML:: ++ +[source,yaml] +---- +- route: + from: + uri: azure-storage-blob://camelazure/incoming + parameters: + moveAfterRead: true + destinationContainer: archive + accessKey: "RAW(yourAccessKey)" + steps: + - log: + message: "Processing blob: ${header.CamelAzureStorageBlobBlobName}" + - to: + uri: direct:processBlob +---- +==== + +You can also customize the destination blob name by adding a prefix and/or suffix: + +[tabs] +==== +Java:: ++ +[source,java] +---- +from("azure-storage-blob://camelazure/source?moveAfterRead=true&destinationContainer=archive&prefix=incoming/&removePrefixOnMove=true&destinationBlobPrefix=processed/&destinationBlobSuffix=.done&accessKey=RAW(yourAccessKey)") + .log("Processing: ${header.CamelAzureStorageBlobBlobName}") + .to("direct:processBlob"); +---- + +XML:: ++ +[source,xml] +---- +<route> + <from uri="azure-storage-blob://camelazure/source?moveAfterRead=true&destinationContainer=archive&prefix=incoming/&removePrefixOnMove=true&destinationBlobPrefix=processed/&destinationBlobSuffix=.done&accessKey=RAW(yourAccessKey)"/> + <log message="Processing: ${header.CamelAzureStorageBlobBlobName}"/> + <to uri="direct:processBlob"/> +</route> +---- + +YAML:: ++ +[source,yaml] +---- +- route: + from: + uri: azure-storage-blob://camelazure/source + parameters: + moveAfterRead: true + destinationContainer: archive + prefix: "incoming/" + removePrefixOnMove: true + destinationBlobPrefix: "processed/" + destinationBlobSuffix: ".done" + accessKey: "RAW(yourAccessKey)" + steps: + - log: + message: "Processing: ${header.CamelAzureStorageBlobBlobName}" + - to: + uri: direct:processBlob +---- +==== + +The move operation works as follows: + +1. The blob is copied to the destination container with the new name +2. The original blob is deleted from the source container +3. Both operations only occur after the Exchange is successfully committed + +The following options control the move behavior: + +[width="100%",cols="25%,75%",options="header",] +|=== +|Option |Description +|`moveAfterRead` |Enable moving blobs after successful processing +|`destinationContainer` |Target container for moved blobs (required when `moveAfterRead=true`) +|`destinationBlobPrefix` |Prefix to add to the blob name in the destination +|`destinationBlobSuffix` |Suffix to add to the blob name in the destination +|`removePrefixOnMove` |Remove the source prefix from the blob name before adding destination prefix +|=== diff --git a/components/camel-azure/camel-azure-storage-blob/src/main/docs/azure-storage-blob-component.adoc b/components/camel-azure/camel-azure-storage-blob/src/main/docs/azure-storage-blob-operations.adoc similarity index 61% copy from components/camel-azure/camel-azure-storage-blob/src/main/docs/azure-storage-blob-component.adoc copy to components/camel-azure/camel-azure-storage-blob/src/main/docs/azure-storage-blob-operations.adoc index 6c02d53d2284..8d7bc7cec7af 100644 --- a/components/camel-azure/camel-azure-storage-blob/src/main/docs/azure-storage-blob-component.adoc +++ b/components/camel-azure/camel-azure-storage-blob/src/main/docs/azure-storage-blob-operations.adoc @@ -1,525 +1,9 @@ -= Azure Storage Blob Service Component -:doctitle: Azure Storage Blob Service -:shortname: azure-storage-blob -:artifactid: camel-azure-storage-blob -:description: Store and retrieve blobs from Azure Storage Blob Service. -:since: 3.3 -:supportlevel: Stable += Azure Storage Blob - Producer Operations :tabs-sync-option: -:component-header: Both producer and consumer are supported -//Manually maintained attributes -:group: Azure -*Since Camel {since}* +xref:ROOT:azure-storage-blob-component.adoc[Back to Azure Storage Blob Component] -*{component-header}* - -The Azure Storage Blob component is used for storing and retrieving blobs from https://azure.microsoft.com/services/storage/blobs/[Azure Storage Blob] Service using *Azure APIs v12*. -However, in the case of versions above v12, we will see if this component can adopt these changes depending on how much breaking changes can result. - -Prerequisites - -You must have a valid Windows Azure Storage account. More information is available at -https://docs.microsoft.com/azure/[Azure Documentation Portal]. - -Maven users will need to add the following dependency to their `pom.xml` -for this component: - -[source,xml] ----- -<dependency> - <groupId>org.apache.camel</groupId> - <artifactId>camel-azure-storage-blob</artifactId> - <version>x.x.x</version> - <!-- use the same version as your Camel core version --> -</dependency> ----- - - -== URI Format - -[source,text] ----- -azure-storage-blob://accountName[/containerName][?options] ----- - -In the case of a consumer, `accountName`, `containerName` are required. - -In the case of a producer, it depends on the operation that is being requested, for example, if operation is on a container level, e.b: createContainer, accountName and containerName are only required, but in case of operation being requested in blob level, e.g: getBlob, accountName, containerName and blobName are required. - -The blob will be created if it does not already exist. -You can append query options to the URI in the following format, `?options=value&option2=value&...` - - -// component options: START -include::partial$component-configure-options.adoc[] -include::partial$component-endpoint-options.adoc[] -include::partial$component-endpoint-headers.adoc[] -// component options: END - -*Required information options:* - -To use this component, you have multiple options to provide the required Azure authentication information: - -- By providing your own https://azuresdkdocs.blob.core.windows.net/$web/java/azure-storage-blob/12.0.0/com/azure/storage/blob/BlobServiceClient.html[BlobServiceClient] instance which can be injected into `blobServiceClient`. Note: You don't need to create a specific client, e.g.: BlockBlobClient, the BlobServiceClient represents the upper level which -can be used to retrieve lower level clients. -- Via Azure Identity, when specifying `credentialType=AZURE_IDENTITY` and providing required https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/identity/azure-identity#environment-variables[environment variables]. This enables service principal (e.g. app registration) authentication with secret/certificate as well as username password. Note that this is the default authentication strategy. -- Via shared storage account key, when specifying `credentialType=SHARED_ACCOUNT_KEY` and providing `accountName` and `accessKey` for your Azure account, this is the simplest way to get started. The accessKey can be generated through your Azure portal. -- Via shared storage account key, when specifying `credentialType=SHARED_KEY_CREDENTIAL` and providing a https://azuresdkartifacts.blob.core.windows.net/azure-sdk-for-java/staging/apidocs/com/azure/storage/common/StorageSharedKeyCredential.html[StorageSharedKeyCredential] instance which can be injected into `credentials` option. -- Via Azure SAS, when specifying `credentialType=AZURE_SAS` and providing a SAS Token parameter through the `sasToken` parameter. - -== Usage - -For example, to download a blob content from the block blob `hello.txt` -located on the `container1` in the `camelazure` storage account, use the following snippet: - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("azure-storage-blob://camelazure/container1?blobName=hello.txt&credentialType=SHARED_ACCOUNT_KEY&accessKey=RAW(yourAccessKey)") - .to("file://blobdirectory"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="azure-storage-blob://camelazure/container1?blobName=hello.txt&credentialType=SHARED_ACCOUNT_KEY&accessKey=RAW(yourAccessKey)"/> - <to uri="file://blobdirectory"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: azure-storage-blob://camelazure/container1 - parameters: - blobName: hello.txt - credentialType: SHARED_ACCOUNT_KEY - accessKey: RAW(yourAccessKey) - steps: - - to: - uri: file://blobdirectory ----- -==== - -=== Advanced Azure Storage Blob configuration - -If your Camel Application is running behind a firewall or if you need to -have more control over the `BlobServiceClient` instance configuration, you can -create your own instance: - -._Java-only: programmatic BlobServiceClient setup_ - -[source,java] ----- -StorageSharedKeyCredential credential = new StorageSharedKeyCredential("yourAccountName", "yourAccessKey"); -String uri = String.format("https://%s.blob.core.windows.net";, "yourAccountName"); - -BlobServiceClient client = new BlobServiceClientBuilder() - .endpoint(uri) - .credential(credential) - .buildClient(); -// This is camel context -context.getRegistry().bind("client", client); ----- - -Then refer to this instance in your Camel `azure-storage-blob` component configuration: - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("azure-storage-blob://cameldev/container1?blobName=myblob&serviceClient=#client") - .to("mock:result"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="azure-storage-blob://cameldev/container1?blobName=myblob&serviceClient=#client"/> - <to uri="mock:result"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: azure-storage-blob://cameldev/container1 - parameters: - blobName: myblob - serviceClient: "#client" - steps: - - to: - uri: mock:result ----- -==== - -=== Automatic detection of BlobServiceClient client in registry - -The component is capable of detecting the presence of an BlobServiceClient bean into the registry. -If it's the only instance of that type, it will be used as the client, and you won't have to define it as uri parameter, like the example above. -This may be really useful for smarter configuration of the endpoint. - -=== Azure Storage Blob Producer operations - -Camel Azure Storage Blob component provides a wide range of operations on the producer side: - -*Operations on the service level* - -For these operations, `accountName` is *required*. -[width="100%",cols="10%,90%",options="header",] -|=== -|Operation |Description -|`listBlobContainers` |Get the content of the blob. You can restrict the output of this operation to a blob range. -|`getChangeFeed` | Returns transaction logs of all the changes that occur to the blobs and the blob metadata in your storage account. The change feed provides ordered, guaranteed, durable, immutable, read-only log of these changes. -|`findBlobsByTags` | Returns a list of blobs across the storage account whose index tags match the SQL-like filter expression provided via the `CamelAzureStorageBlobTagFilter` header or the message body. Optionally honours `CamelAzureStorageBlobMaxResultsPerPage` and `CamelAzureStorageBlobTimeout`. The result body is a `List<TaggedBlobItem>`. -|=== - -*Operations on the container level* - -For these operations, `accountName` and `containerName` are *required*. -[width="100%",cols="10%,90%",options="header",] -|=== -|Operation |Description -|`createBlobContainer` | Create a new container within a storage account. If a container with the same name already exists, the producer will ignore it. -|`deleteBlobContainer` | Delete the specified container in the storage account. If the container doesn't exist, the operation fails. -|`listBlobs`| Returns a list of blobs in this container, with folder structures flattened. -|`listBlobVersions`| Returns a list of blobs and their versions in this container. Each `BlobItem` in the result carries its own `versionId` and `isCurrentVersion` flag, allowing the full version history of every blob to be inspected. Requires versioning to be enabled on the storage account. Honours the same `prefix`, `regex` and `maxResultsPerPage` filters as `listBlobs`. -|=== - -*Operations on the blob level* - -For these operations, `accountName`, `containerName` and `blobName` are *required*. -[width="100%",cols="10%,10%,80%",options="header",] -|=== -|Operation |Blob Type|Description -|`getBlob` |Common|Get the content of the blob. You can restrict the output of this operation to a blob range. -|`deleteBlob` |Common|Delete a blob. -|`downloadBlobToFile` |Common|Download the entire blob into a file specified by the path. The file will be created and must not exist, if the file already exists a `FileAlreadyExistsException` will be thrown. -|`downloadLink` |Common| Generate the download link for the specified blob using shared access signatures (SAS). This by default only limits to 1hour of allowed access. However, you can override the default expiration duration through the headers. -|`uploadBlockBlob` |BlockBlob|Creates a new block blob, or updates the content of an existing block blob. Updating an existing block blob overwrites any existing metadata on the blob. Partial updates are not supported with PutBlob; the content of the existing blob is overwritten with the new content. Note: For larger files, use `uploadBlockBlobChunked`. -|`uploadBlockBlobChunked` |BlockBlob|Creates or updates a block blob with support for large files (up to several GB). Uses chunked parallel uploads for memory efficiency. Accepts File, Path, WrappedFile, or InputStream as body. Configure `blockSize` (default: 4MB) and `maxConcurrency` (default: auto) for performance tuning. Recommended for files larger than 256MB. -|`stageBlockBlobList`|`BlockBlob`|Uploads the specified block to the block blob's "staging area" to be later committed by a call to commitBlobBlockList. However, in case header `CamelAzureStorageBlobCommitBlobBlockListLater` or config `commitBlockListLater` is set to false, this will commit the blocks immediately after staging the blocks. -|`commitBlobBlockList`|`BlockBlob`|Write a blob by specifying the list of block IDs that are to make up the blob. To be written as part -of a blob, a block must have been successfully written to the server in a prior `stageBlockBlobList` operation. You can -call `commitBlobBlockList` to update a blob by uploading only those blocks that have changed, then committing the new -and existing blocks together. Any blocks not specified in the block list and permanently deleted. -|`getBlobBlockList` |`BlockBlob`|Returns the list of blocks that have been uploaded as part of a block blob using the specified blocklist filter. -|`createAppendBlob` |`AppendBlob`|Creates a 0-length append blob. Call commitAppendBlo`b operation to append data to an append blob. -|`commitAppendBlob` |`AppendBlob`|Commits a new block of data to the end of the existing append blob. In case of header `CamelAzureStorageBlobCreateAppendBlob` or config `createAppendBlob` is set to true, it will attempt to create the appendBlob through internal call to `createAppendBlob` operation first before committing. -|`createPageBlob`|`PageBlob`|Creates a page blob of the specified length. Call `uploadPageBlob` operation to upload data to a page blob. -|`uploadPageBlob`|`PageBlob`|Write one or more pages to the page blob. The size must be a multiple of 512. In case of header `CamelAzureStorageBlobCreatePageBlob` or config `createPageBlob` is set to true, it will attempt to create the appendBlob through internal call to `createPageBlob` operation first before uploading. -|`resizePageBlob`|`PageBlob`| Resizes the page blob to the specified size, which must be a multiple of 512. -|`clearPageBlob`|`PageBlob`| Free the specified pages from the page blob. The size of the range must be a multiple of 512. -|`getPageBlobRanges`|`PageBlob`|Returns the list of valid page ranges for a page blob or snapshot of a page blob. -|`copyBlob`|`Common`|Copy a blob from one container to another one, even from different accounts. -|`createBlobSnapshot`|`Common`|Creates a read-only snapshot of a blob. The snapshot ID is returned in the `CamelAzureStorageBlobSnapshotId` header. -|`setBlobTags`|`Common`|Sets user-defined index tags on a blob. Tags are key-value pairs that can be used to filter and query blobs across containers. Tags can be provided via the `CamelAzureStorageBlobTags` header or as the message body (`Map<String, String>`). -|`getBlobTags`|`Common`|Retrieves user-defined index tags from a blob. The tags are returned as the message body (`Map<String, String>`) and also set in the `CamelAzureStorageBlobTags` header. -|`setBlobLegalHold`|`Common`|Sets a legal hold on a blob. The legal hold flag (`Boolean`) is provided via the `CamelAzureStorageBlobLegalHold` header or the message body. While a legal hold is set, the blob cannot be modified or deleted until the hold is explicitly cleared by calling the operation again with `false`. -|`setBlobImmutabilityPolicy`|`Common`|Sets a time-based immutability policy on a blob. The policy expiry time (`OffsetDateTime`) is read from the `CamelAzureStorageBlobImmutabilityPolicyExpiryTime` header and the policy mode (`BlobImmutabilityPolicyMode`, defaults to `UNLOCKED`) from `CamelAzureStorageBlobImmutabilityPolicyMode`. A pre-built `BlobImmutabilityPolicy` may also be supplied via the message body or `CamelAzureStorageBlobImmutabilityPolicy` header. -|`undeleteBlob`|`Common`|Restores the contents and metadata of a soft-deleted blob and any associated soft-deleted snapshots. Soft delete must be enabled on the storage account for this operation to succeed. -|`setBlobTier`|`Common`|Sets the access tier of an existing blob. The target tier (`AccessTier` such as `HOT`, `COOL`, `COLD`, or `ARCHIVE`) is read from the `CamelAzureStorageBlobAccessTier` header or the message body. When rehydrating a blob from `ARCHIVE`, the optional `CamelAzureStorageBlobRehydratePriority` header (`RehydratePriority`, `STANDARD` or `HIGH`) controls the rehydration priority. -|=== - -Refer to the example section in this page to learn how to use these operations into your camel application. - -== Examples - -=== Consumer Examples - -To consume a blob into a file using the file component, this can be done like this: -[tabs] -==== -Java:: -+ -[source,java] ----- -from("azure-storage-blob://camelazure/container1?blobName=hello.txt&accountName=yourAccountName&accessKey=yourAccessKey") - .to("file://blobdirectory"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="azure-storage-blob://camelazure/container1?blobName=hello.txt&accountName=yourAccountName&accessKey=yourAccessKey"/> - <to uri="file://blobdirectory"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: azure-storage-blob://camelazure/container1 - parameters: - blobName: hello.txt - accountName: yourAccountName - accessKey: yourAccessKey - steps: - - to: - uri: file://blobdirectory ----- -==== - -However, you can also write to file directly without using the file component, you will need to specify `fileDir` folder path to save your blob in your machine. - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("azure-storage-blob://camelazure/container1?blobName=hello.txt&accountName=yourAccountName&accessKey=yourAccessKey&fileDir=/var/to/awesome/dir") - .to("mock:results"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="azure-storage-blob://camelazure/container1?blobName=hello.txt&accountName=yourAccountName&accessKey=yourAccessKey&fileDir=/var/to/awesome/dir"/> - <to uri="mock:results"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: azure-storage-blob://camelazure/container1 - parameters: - blobName: hello.txt - accountName: yourAccountName - accessKey: yourAccessKey - fileDir: /var/to/awesome/dir - steps: - - to: - uri: mock:results ----- -==== - -Also, the component supports batch consumer, hence you can consume multiple blobs with only specifying the container name, the consumer will -return multiple exchanges depending on the number of the blobs in the container. Example: - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("azure-storage-blob://camelazure/container1?accountName=yourAccountName&accessKey=yourAccessKey&fileDir=/var/to/awesome/dir") - .to("mock:results"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="azure-storage-blob://camelazure/container1?accountName=yourAccountName&accessKey=yourAccessKey&fileDir=/var/to/awesome/dir"/> - <to uri="mock:results"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: azure-storage-blob://camelazure/container1 - parameters: - accountName: yourAccountName - accessKey: yourAccessKey - fileDir: /var/to/awesome/dir - steps: - - to: - uri: mock:results ----- -==== - -==== Delete After Read - -The consumer supports automatic deletion of blobs after they have been successfully processed. This is useful when you want to ensure that blobs are only processed once. - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("azure-storage-blob://camelazure/container1?deleteAfterRead=true&accessKey=RAW(yourAccessKey)") - .log("Processing blob: ${header.CamelAzureStorageBlobBlobName}") - .to("direct:processBlob"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="azure-storage-blob://camelazure/container1?deleteAfterRead=true&accessKey=RAW(yourAccessKey)"/> - <log message="Processing blob: ${header.CamelAzureStorageBlobBlobName}"/> - <to uri="direct:processBlob"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: azure-storage-blob://camelazure/container1 - parameters: - deleteAfterRead: true - accessKey: "RAW(yourAccessKey)" - steps: - - log: - message: "Processing blob: ${header.CamelAzureStorageBlobBlobName}" - - to: - uri: direct:processBlob ----- -==== - -IMPORTANT: The delete operation is only performed if the Exchange is successfully committed. If processing fails or a rollback occurs, the blob will not be deleted and can be reprocessed on the next poll. - -NOTE: When `deleteAfterRead` is set to `false` (the default), the same blobs will be retrieved repeatedly in subsequent polls. In this case, you should use the Idempotent Consumer EIP to filter out duplicates based on the `CamelAzureStorageBlobBlobName` header. - -==== Move After Read - -The consumer can move blobs to a different container after successful processing. This is useful for archiving processed blobs or implementing a processing pipeline across containers. - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("azure-storage-blob://camelazure/incoming?moveAfterRead=true&destinationContainer=archive&accessKey=RAW(yourAccessKey)") - .log("Processing blob: ${header.CamelAzureStorageBlobBlobName}") - .to("direct:processBlob"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="azure-storage-blob://camelazure/incoming?moveAfterRead=true&destinationContainer=archive&accessKey=RAW(yourAccessKey)"/> - <log message="Processing blob: ${header.CamelAzureStorageBlobBlobName}"/> - <to uri="direct:processBlob"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: azure-storage-blob://camelazure/incoming - parameters: - moveAfterRead: true - destinationContainer: archive - accessKey: "RAW(yourAccessKey)" - steps: - - log: - message: "Processing blob: ${header.CamelAzureStorageBlobBlobName}" - - to: - uri: direct:processBlob ----- -==== - -You can also customize the destination blob name by adding a prefix and/or suffix: - -[tabs] -==== -Java:: -+ -[source,java] ----- -from("azure-storage-blob://camelazure/source?moveAfterRead=true&destinationContainer=archive&prefix=incoming/&removePrefixOnMove=true&destinationBlobPrefix=processed/&destinationBlobSuffix=.done&accessKey=RAW(yourAccessKey)") - .log("Processing: ${header.CamelAzureStorageBlobBlobName}") - .to("direct:processBlob"); ----- - -XML:: -+ -[source,xml] ----- -<route> - <from uri="azure-storage-blob://camelazure/source?moveAfterRead=true&destinationContainer=archive&prefix=incoming/&removePrefixOnMove=true&destinationBlobPrefix=processed/&destinationBlobSuffix=.done&accessKey=RAW(yourAccessKey)"/> - <log message="Processing: ${header.CamelAzureStorageBlobBlobName}"/> - <to uri="direct:processBlob"/> -</route> ----- - -YAML:: -+ -[source,yaml] ----- -- route: - from: - uri: azure-storage-blob://camelazure/source - parameters: - moveAfterRead: true - destinationContainer: archive - prefix: "incoming/" - removePrefixOnMove: true - destinationBlobPrefix: "processed/" - destinationBlobSuffix: ".done" - accessKey: "RAW(yourAccessKey)" - steps: - - log: - message: "Processing: ${header.CamelAzureStorageBlobBlobName}" - - to: - uri: direct:processBlob ----- -==== - -The move operation works as follows: - -1. The blob is copied to the destination container with the new name -2. The original blob is deleted from the source container -3. Both operations only occur after the Exchange is successfully committed - -The following options control the move behavior: - -[width="100%",cols="25%,75%",options="header",] -|=== -|Option |Description -|`moveAfterRead` |Enable moving blobs after successful processing -|`destinationContainer` |Target container for moved blobs (required when `moveAfterRead=true`) -|`destinationBlobPrefix` |Prefix to add to the blob name in the destination -|`destinationBlobSuffix` |Suffix to add to the blob name in the destination -|`removePrefixOnMove` |Remove the source prefix from the blob name before adding destination prefix -|=== - -=== Producer Operations Examples +== Producer Operations Examples - `listBlobContainers`: NOTE: The `CamelAzureStorageBlobListBlobContainersOptions` header requires a `ListBlobContainersOptions` object, which must be set from a bean or processor. @@ -1468,7 +952,7 @@ YAML:: ---- ==== -=== Reading a specific blob snapshot +== Reading a specific blob snapshot The `getBlob`, `downloadBlobToFile` and `downloadLink` operations can target a specific snapshot by setting the `snapshotId` URI parameter or the `CamelAzureStorageBlobSnapshotId` exchange header. When set, the read is scoped @@ -1522,7 +1006,7 @@ YAML:: ---- ==== -=== Reading a specific blob version +== Reading a specific blob version When blob versioning is enabled on the storage account, the `getBlob`, `downloadBlobToFile` and `downloadLink` operations can target a specific version by setting the `versionId` URI parameter or the @@ -1849,7 +1333,7 @@ from("direct:rehydrate") .to("mock:result"); ---- -=== Blob modification during download (ConditionNotMet) +== Blob modification during download (ConditionNotMet) When downloading a blob via streaming (the `getBlob` operation or the consumer without `fileDir`), the Azure SDK reads the blob in chunks. On the first chunk it captures the blob's ETag and uses it as an `if-match` condition on subsequent chunk requests. @@ -1865,7 +1349,7 @@ Workarounds: * Use blob versioning — read from a specific version via the `versionId` option or the `CamelAzureStorageBlobVersionId` header. * Avoid modifying blobs while they are being consumed. -=== SAS Token generation example +== SAS Token generation example SAS Blob Container tokens can be generated programmatically or via Azure UI. To generate the token with Java code, the following can be done: @@ -1936,20 +1420,3 @@ YAML:: credentialType: AZURE_SAS ---- ==== - -== Important Development Notes - -All integration tests use https://www.testcontainers.org/[Testcontainers] and run by default. -Obtaining of Azure accessKey and accountName is needed to be able to run all integration tests using Azure services. -In addition to the mocked unit tests, you *will need to run the integration tests -with every change you make or even client upgrade as the Azure client can break things even on minor versions upgrade.* -To run the integration tests, on this component directory, run the following maven command: - -[source,bash] ----- -mvn verify -DaccountName=myacc -DaccessKey=mykey -DcredentialType=SHARED_ACCOUNT_KEY ----- - -Whereby `accountName` is your Azure account name and `accessKey` is the access key being generated from Azure portal. - - diff --git a/docs/components/modules/others/nav.adoc b/docs/components/modules/others/nav.adoc index e8ea5aefaa88..d9f6b5052929 100644 --- a/docs/components/modules/others/nav.adoc +++ b/docs/components/modules/others/nav.adoc @@ -4,6 +4,8 @@ * xref:others:index.adoc[Miscellaneous Components] ** xref:attachments.adoc[Attachments] *** xref:azure-schema-registry.adoc[Azure Schema Registry] +** xref:azure-storage-blob-consumer.adoc[Azure Storage Blob - Consumer Examples] +** xref:azure-storage-blob-operations.adoc[Azure Storage Blob - Producer Operations] ** xref:camel-yaml-dsl-validator-maven-plugin.adoc[Camel YAML DSL Validator Maven Plugin] ** xref:cli-connector.adoc[CLI Connector] ** xref:cli-debug.adoc[CLI Debug] diff --git a/docs/components/modules/others/pages/azure-storage-blob-consumer.adoc b/docs/components/modules/others/pages/azure-storage-blob-consumer.adoc new file mode 120000 index 000000000000..6f430bda17db --- /dev/null +++ b/docs/components/modules/others/pages/azure-storage-blob-consumer.adoc @@ -0,0 +1 @@ +../../../../../components/camel-azure/camel-azure-storage-blob/src/main/docs/azure-storage-blob-consumer.adoc \ No newline at end of file diff --git a/docs/components/modules/others/pages/azure-storage-blob-operations.adoc b/docs/components/modules/others/pages/azure-storage-blob-operations.adoc new file mode 120000 index 000000000000..b8c2b8eddd08 --- /dev/null +++ b/docs/components/modules/others/pages/azure-storage-blob-operations.adoc @@ -0,0 +1 @@ +../../../../../components/camel-azure/camel-azure-storage-blob/src/main/docs/azure-storage-blob-operations.adoc \ No newline at end of file
