This is an automated email from the ASF dual-hosted git repository.
acosentino pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/camel.git
commit e093f6523f4df6085dfd4075c62bfc30e4b0bae0
Author: Andrea Cosentino <anco...@gmail.com>
AuthorDate: Mon Jun 8 08:34:48 2020 +0200
Regen website docs
---
.../modules/ROOT/pages/aws2-athena-component.adoc | 564 +++++++++++----------
1 file changed, 298 insertions(+), 266 deletions(-)
diff --git a/docs/components/modules/ROOT/pages/aws2-athena-component.adoc
b/docs/components/modules/ROOT/pages/aws2-athena-component.adoc
index fdfb1ab..d5a4106 100644
--- a/docs/components/modules/ROOT/pages/aws2-athena-component.adoc
+++ b/docs/components/modules/ROOT/pages/aws2-athena-component.adoc
@@ -4,69 +4,97 @@
:page-source:
components/camel-aws2-athena/src/main/docs/aws2-athena-component.adoc
:docTitle: AWS 2 Athena
:artifactId: camel-aws2-athena
-:description: Access AWS Athena service with SQL queries using AWS SDK version
2.x.
+:description: Access AWS Athena service using AWS SDK version 2.x.
:since: 3.4
:supportLevel: Preview
:component-header: Only producer is supported
*Since Camel {since}*
-*Since Camel 3.4*
-
-
*{component-header}*
-The S3 component supports storing and retrieving objects from/to
-https://aws.amazon.com/athena[Amazon's S3] service.
+The Athena component supports running queries with
https://aws.amazon.com/athena/[AWS Athena] and working with results.
Prerequisites
You must have a valid Amazon Web Services developer account, and be
-signed up to use Amazon S3. More information is available at
-https://aws.amazon.com/s3[Amazon S3].
+signed up to use Amazon Athena. More information is available at
+https://aws.amazon.com/athena/[AWS Athena].
== URI Format
[source,java]
------------------------------
-aws2-s3://bucketNameOrArn[?options]
+aws2-athena://label[?options]
------------------------------
-The bucket will be created if it don't already exists. +
- You can append query options to the URI in the following format,
+You can append query options to the URI in the following format,
?options=value&option2=value&...
-For example in order to read file `hello.txt` from bucket `helloBucket`, use
the following snippet:
+For example, to run a simple query, wait up to 60 seconds for completion, and
log the results:
[source,java]
--------------------------------------------------------------------------------
-from("aws2-s3://helloBucket?accessKey=yourAccessKey&secretKey=yourSecretKey&prefix=hello.txt")
- .to("file:/var/downloaded");
+from("direct:start")
+ .setBody(constant("SELECT 1"))
+
.to("aws2-athena://label?waitTimeout=60000&outputLocation=s3://bucket/path/")
+ .to("aws2-athena://label?operation=getQueryResults&outputType=StreamList")
+ .split(body()).streaming()
+ .to("log:out")
+ .to("mock:result");
--------------------------------------------------------------------------------
+Similarly, running the query and returning a path to the results in S3:
+
+[source,java]
+--------------------------------------------------------------------------------
+from("direct:start")
+ .setBody(constant("SELECT 1"))
+
.to("aws2-athena://label?waitTimeout=60000&outputLocation=s3://bucket/path/")
+ .to("aws2-athena://label?operation=getQueryResults&outputType=S3Pointer")
+ .to("mock:result");
+--------------------------------------------------------------------------------
== URI Options
// component options: START
-The AWS 2 Athena component supports 11 options, which are listed below.
+The AWS 2 Athena component supports 29 options, which are listed below.
[width="100%",cols="2,5,^1,2",options="header"]
|===
| Name | Description | Default | Type
-| *amazonAthenaClient* (producer) | To use the AmazonAthena as the client | |
AthenaClient
-| *configuration* (producer) | The component configuration | |
Athena2Configuration
+| *accessKey* (producer) | Amazon AWS Access Key. | | String
+| *amazonAthenaClient* (producer) | The AmazonAthena instance to use as the
client. | | AthenaClient
+| *configuration* (producer) | The component configuration. | |
Athena2Configuration
+| *database* (producer) | The Athena database to use. | | String
+| *delay* (producer) | Milliseconds before the next poll for query execution
status. See the section 'Waiting for Query Completion and Retrying Failed
Queries' to learn more. | 2000 | long
+| *encryptionOption* (producer) | The encryption type to use when storing
query results in S3. One of SSE_S3, SSE_KMS, or CSE_KMS. The value can be one
of: SSE_S3, SSE_KMS, CSE_KMS, null | | EncryptionOption
+| *includeTrace* (producer) | Include useful trace information at the
beginning of queries as an SQL comment (prefixed with --). | false | boolean
+| *initialDelay* (producer) | Milliseconds before the first poll for query
execution status. See the section 'Waiting for Query Completion and Retrying
Failed Queries' to learn more. | 1000 | long
+| *kmsKey* (producer) | For SSE-KMS and CSE-KMS, this is the KMS key ARN or
ID. | | String
| *lazyStartProducer* (producer) | Whether the producer should be started lazy
(on the first message). By starting lazy you can use this to allow CamelContext
and routes to startup in situations where a producer may otherwise fail during
starting and cause the route to fail being started. By deferring this startup
to be lazy then the startup failure can be handled during routing messages via
Camel's routing error handlers. Beware that when the first message is processed
then creating and [...]
-| *operation* (producer) | The operation to perform. It can be
startQueryExecution. The value can be one of: startQueryExecution,
listQueryExecutions | startQueryExecution | Athena2Operations
-| *proxyHost* (producer) | To define a proxy host when instantiating the
Athena client | | String
-| *proxyPort* (producer) | To define a proxy port when instantiating the
Athena client | | Integer
+| *maxAttempts* (producer) | Maximum number of times to attempt a query. Set
to 1 to disable retries. See the section 'Waiting for Query Completion and
Retrying Failed Queries' to learn more. | 1 | int
+| *maxResults* (producer) | Max number of results to return for the given
operation (if supported by the Athena API endpoint). If not set, will use the
Athena API default for the given operation. | | Integer
+| *nextToken* (producer) | Pagination token to use in the case where the
response from the previous request was truncated. | | String
+| *operation* (producer) | The Athena API function to call. The value can be
one of: getQueryExecution, getQueryResults, listQueryExecutions,
startQueryExecution | startQueryExecution | Athena2Operations
+| *outputLocation* (producer) | The location in Amazon S3 where query results
are stored, such as s3://path/to/query/bucket/. Ensure this value ends with a
forward slash ('/'). | | String
+| *outputType* (producer) | How query results should be returned. One of
StreamList (default - return a GetQueryResultsIterable that can page through
all results), SelectList (returns at most 1,000 rows at a time, plus a
NextToken value as a header than can be used for manual pagination of results),
S3Pointer (return an S3 path pointing to the results). The value can be one of:
StreamList, SelectList, S3Pointer | StreamList | Athena2OutputType
+| *proxyHost* (producer) | To define a proxy host when instantiating the
Athena client. | | String
+| *proxyPort* (producer) | To define a proxy port when instantiating the
Athena client. | | Integer
| *proxyProtocol* (producer) | To define a proxy protocol when instantiating
the Athena client. The value can be one of: HTTP, HTTPS | HTTPS | Protocol
-| *region* (producer) | The region in which Athena client needs to work. When
using this parameter, the configuration will expect the lowercase name of the
region (for example ap-east-1) You'll need to use the name
Region.EU_WEST_1.id() | | String
+| *queryExecutionId* (producer) | The unique ID identifying the query
execution. | | String
+| *queryString* (producer) | The SQL query to run. Except for simple queries,
prefer setting this as the body of the Exchange or as a header using
Athena2Constants.QUERY_STRING to avoid having to deal with URL encoding issues.
| | String
+| *region* (producer) | The region in which Athena client needs to work. When
using this parameter, the configuration will expect the lowercase name of the
region (for example ap-east-1). You'll need to use the name
Region.EU_WEST_1.id(). | | String
+| *resetWaitTimeoutOnRetry* (producer) | Reset the waitTimeout countdown in
the event of a query retry. If set to true, potential max time spent waiting
for queries is equal to waitTimeout x maxAttempts. See the section 'Waiting for
Query Completion and Retrying Failed Queries' to learn more. | true | boolean
+| *retry* (producer) | Optional comma separated list of error types to retry
the query for. Use 'retryable' to retry all retryable failure conditions (e.g.
generic errors and resources exhausted), 'generic' to retry
'GENERIC_INTERNAL_ERROR' failures, 'exhausted' to retry queries that have
exhausted resource limits, 'always' to always retry regardless of failure
condition, or 'never' or null to never retry (default). See the section
'Waiting for Query Completion and Retrying Failed Querie [...]
+| *secretKey* (producer) | Amazon AWS Secret Key. | | String
+| *waitTimeout* (producer) | Optional max wait time in millis to wait for a
successful query completion. See the section 'Waiting for Query Completion and
Retrying Failed Queries' to learn more. | 0 | long
+| *workGroup* (producer) | The workgroup to use for running the query. | |
String
| *basicPropertyBinding* (advanced) | Whether the component should use basic
property binding (Camel 2.x) or the newer property binding with additional
capabilities | false | boolean
-| *accessKey* (security) | Amazon AWS Access Key | | String
-| *secretKey* (security) | Amazon AWS Secret Key | | String
+| *clientRequestToken* (advanced) | A unique string to ensure issues queries
are idempotent. It is unlikely you will need to set this. | | String
|===
// component options: END
@@ -84,7 +112,7 @@ The AWS 2 Athena component supports 11 options, which are
listed below.
The AWS 2 Athena endpoint is configured using URI syntax:
----
-aws2-athena:query
+aws2-athena:label
----
with the following path and query parameters:
@@ -99,23 +127,41 @@ with the following path and query parameters:
|===
-=== Query Parameters (11 parameters):
+=== Query Parameters (29 parameters):
[width="100%",cols="2,5,^1,2",options="header"]
|===
| Name | Description | Default | Type
-| *amazonAthenaClient* (producer) | To use the AmazonAthena as the client | |
AthenaClient
+| *accessKey* (producer) | Amazon AWS Access Key. | | String
+| *amazonAthenaClient* (producer) | The AmazonAthena instance to use as the
client. | | AthenaClient
+| *database* (producer) | The Athena database to use. | | String
+| *delay* (producer) | Milliseconds before the next poll for query execution
status. See the section 'Waiting for Query Completion and Retrying Failed
Queries' to learn more. | 2000 | long
+| *encryptionOption* (producer) | The encryption type to use when storing
query results in S3. One of SSE_S3, SSE_KMS, or CSE_KMS. The value can be one
of: SSE_S3, SSE_KMS, CSE_KMS, null | | EncryptionOption
+| *includeTrace* (producer) | Include useful trace information at the
beginning of queries as an SQL comment (prefixed with --). | false | boolean
+| *initialDelay* (producer) | Milliseconds before the first poll for query
execution status. See the section 'Waiting for Query Completion and Retrying
Failed Queries' to learn more. | 1000 | long
+| *kmsKey* (producer) | For SSE-KMS and CSE-KMS, this is the KMS key ARN or
ID. | | String
| *lazyStartProducer* (producer) | Whether the producer should be started lazy
(on the first message). By starting lazy you can use this to allow CamelContext
and routes to startup in situations where a producer may otherwise fail during
starting and cause the route to fail being started. By deferring this startup
to be lazy then the startup failure can be handled during routing messages via
Camel's routing error handlers. Beware that when the first message is processed
then creating and [...]
-| *operation* (producer) | The operation to perform. It can be
startQueryExecution. The value can be one of: startQueryExecution,
listQueryExecutions | startQueryExecution | Athena2Operations
-| *proxyHost* (producer) | To define a proxy host when instantiating the
Athena client | | String
-| *proxyPort* (producer) | To define a proxy port when instantiating the
Athena client | | Integer
+| *maxAttempts* (producer) | Maximum number of times to attempt a query. Set
to 1 to disable retries. See the section 'Waiting for Query Completion and
Retrying Failed Queries' to learn more. | 1 | int
+| *maxResults* (producer) | Max number of results to return for the given
operation (if supported by the Athena API endpoint). If not set, will use the
Athena API default for the given operation. | | Integer
+| *nextToken* (producer) | Pagination token to use in the case where the
response from the previous request was truncated. | | String
+| *operation* (producer) | The Athena API function to call. The value can be
one of: getQueryExecution, getQueryResults, listQueryExecutions,
startQueryExecution | startQueryExecution | Athena2Operations
+| *outputLocation* (producer) | The location in Amazon S3 where query results
are stored, such as s3://path/to/query/bucket/. Ensure this value ends with a
forward slash ('/'). | | String
+| *outputType* (producer) | How query results should be returned. One of
StreamList (default - return a GetQueryResultsIterable that can page through
all results), SelectList (returns at most 1,000 rows at a time, plus a
NextToken value as a header than can be used for manual pagination of results),
S3Pointer (return an S3 path pointing to the results). The value can be one of:
StreamList, SelectList, S3Pointer | StreamList | Athena2OutputType
+| *proxyHost* (producer) | To define a proxy host when instantiating the
Athena client. | | String
+| *proxyPort* (producer) | To define a proxy port when instantiating the
Athena client. | | Integer
| *proxyProtocol* (producer) | To define a proxy protocol when instantiating
the Athena client. The value can be one of: HTTP, HTTPS | HTTPS | Protocol
-| *region* (producer) | The region in which Athena client needs to work. When
using this parameter, the configuration will expect the lowercase name of the
region (for example ap-east-1) You'll need to use the name
Region.EU_WEST_1.id() | | String
+| *queryExecutionId* (producer) | The unique ID identifying the query
execution. | | String
+| *queryString* (producer) | The SQL query to run. Except for simple queries,
prefer setting this as the body of the Exchange or as a header using
Athena2Constants.QUERY_STRING to avoid having to deal with URL encoding issues.
| | String
+| *region* (producer) | The region in which Athena client needs to work. When
using this parameter, the configuration will expect the lowercase name of the
region (for example ap-east-1). You'll need to use the name
Region.EU_WEST_1.id(). | | String
+| *resetWaitTimeoutOnRetry* (producer) | Reset the waitTimeout countdown in
the event of a query retry. If set to true, potential max time spent waiting
for queries is equal to waitTimeout x maxAttempts. See the section 'Waiting for
Query Completion and Retrying Failed Queries' to learn more. | true | boolean
+| *retry* (producer) | Optional comma separated list of error types to retry
the query for. Use 'retryable' to retry all retryable failure conditions (e.g.
generic errors and resources exhausted), 'generic' to retry
'GENERIC_INTERNAL_ERROR' failures, 'exhausted' to retry queries that have
exhausted resource limits, 'always' to always retry regardless of failure
condition, or 'never' or null to never retry (default). See the section
'Waiting for Query Completion and Retrying Failed Querie [...]
+| *secretKey* (producer) | Amazon AWS Secret Key. | | String
+| *waitTimeout* (producer) | Optional max wait time in millis to wait for a
successful query completion. See the section 'Waiting for Query Completion and
Retrying Failed Queries' to learn more. | 0 | long
+| *workGroup* (producer) | The workgroup to use for running the query. | |
String
| *basicPropertyBinding* (advanced) | Whether the endpoint should use basic
property binding (Camel 2.x) or the newer property binding with additional
capabilities | false | boolean
+| *clientRequestToken* (advanced) | A unique string to ensure issues queries
are idempotent. It is unlikely you will need to set this. | | String
| *synchronous* (advanced) | Sets whether synchronous processing should be
strictly used, or Camel is allowed to use asynchronous processing (if
supported). | false | boolean
-| *accessKey* (security) | Amazon AWS Access Key | | String
-| *secretKey* (security) | Amazon AWS Secret Key | | String
|===
// endpoint options: END
@@ -128,363 +174,349 @@ with the following path and query parameters:
-Required S3 component options
+Required Athena component options
-You have to provide the amazonS3Client in the
+You have to provide the amazonAthenaClient in the
Registry or your accessKey and secretKey to access
-the https://aws.amazon.com/s3[Amazon's S3].
-
-== Batch Consumer
-
-This component implements the Batch Consumer.
-
-This allows you for instance to know how many messages exists in this
-batch and for instance let the Aggregator
-aggregate this number of messages.
+the https://aws.amazon.com/athena/[AWS Athena] service.
== Usage
-=== Message headers evaluated by the S3 producer
-
-[width="100%",cols="10%,10%,80%",options="header",]
-|=======================================================================
-|Header |Type |Description
+=== Message headers evaluated by the Athena producer
-|`CamelAwsS3BucketName` |`String` |The bucket Name which this object will be
stored or which will be used for the current operation
+Message headers listed here override the corresponding
+query parameters listed in "Query Parameters".
-|`CamelAwsS3BucketDestinationName` |`String` |The bucket Destination Name
which will be used for the current operation
-
-|`CamelAwsS3ContentLength` |`Long` |The content length of this object.
+[width="100%",cols="5%,5%,10%,75%,5%",options="header",]
+|=======================================================================
+|Operation |Header |Type |Description |Required
-|`CamelAwsS3ContentType` |`String` |The content type of this object.
+|All |`CamelAwsAthenaOperation` |`String` |The operation to perform. Permitted
values are getQueryExecution, getQueryResults, listQueryExecutions,
startQueryExecution. Default is startQueryExecution. | No
-|`CamelAwsS3ContentControl` |`String` |The content control of this object.
+|startQueryExecution |`CamelAwsAthenaDatabase` |`String` |The Athena database
to use. | No
-|`CamelAwsS3ContentDisposition` |`String` |The content disposition of this
object.
+|getQueryExecution, getQueryResults |`CamelAwsAthenaQueryExecutionId`
|`String` |The unique ID identifying the query execution. | No
-|`CamelAwsS3ContentEncoding` |`String` |The content encoding of this object.
+|listQueryExecutions, startQueryExecution |`CamelAwsAthenaWorkGroup` |`String`
|The workgroup to use for running the query. | No
-|`CamelAwsS3ContentMD5` |`String` |The md5 checksum of this object.
+|getQueryResults, listQueryExecutions |`CamelAwsAthenaNextToken` |`String`
|Pagination token to use in the case where the response from the previous
request was truncated. | No
-|`CamelAwsS3DestinationKey` |`String` |The Destination key which will be used
for the current operation
+|getQueryResults, listQueryExecutions |`CamelAwsAthenaMaxResults` |`Integer`
|Max number of results to return for the given operation (if supported by the
Athena API endpoint).
+If not set, will use the Athena API default for the given operation. | No
-|`CamelAwsS3Key` |`String` |The key under which this object will be stored or
which will be used for the current operation
+|startQueryExecution |`CamelAwsAthenaIncludeTrace` |`boolean` |Include useful
trace information at the beginning of queries as an SQL comment (prefixed with
"--"). | No
-|`CamelAwsS3LastModified` |`java.util.Date` |The last modified timestamp of
this object.
+|startQueryExecution |`CamelAwsAthenaOutputLocation` |`String` |The location
in Amazon S3 where query results are stored, such as s3://path/to/query/bucket/.
+Ensure this value ends with a forward slash ('/'). | No
-|`CamelAwsS3Operation` |`String` |The operation to perform. Permitted values
are copyObject, deleteObject, listBuckets, deleteBucket, downloadLink,
listObjects
+|getQueryResults |`CamelAwsAthenaOutputType` |`Athena2OutputType` |How query
results should be returned. One of
+StreamList (default - return a GetQueryResultsIterable that can page through
all results),
+SelectList (returns at most 1,000 rows at a time, plus a NextToken value as a
header than can be used for manual pagination of results),
+S3Pointer (return an S3 path pointing to the results). | No
-|`CamelAwsS3StorageClass` |`String` |The storage class of this object.
+|startQueryExecution |`CamelAwsAthenaClientRequestToken` |`String` |A unique
string to ensure issues queries are idempotent. It is unlikely you will need
to set this. | No
-|`CamelAwsS3CannedAcl` |`String` |The canned acl that will be applied to the
object. see
-`com.amazonaws.services.s3.model.CannedAccessControlList` for allowed
-values.
+|startQueryExecution |`CamelAwsAthenaQueryString` |`String` |The SQL query to
run. Except for simple queries, prefer setting this as the `body` of the
+Exchange or as this header to avoid having to deal with URL encoding issues. |
No
-|`CamelAwsS3Acl` |`com.amazonaws.services.s3.model.AccessControlList` |A well
constructed Amazon S3 Access Control List object.
-see `com.amazonaws.services.s3.model.AccessControlList` for more details
+|startQueryExecution |`CamelAwsAthenaEncryptionOption` |`String` |The
encryption type to use when storing query results in S3. One of SSE_S3,
SSE_KMS, or CSE_KMS. | No
-|`CamelAwsS3Headers` |`Map<String,String>` |Support to get or set custom
objectMetadata headers.
+|startQueryExecution |`CamelAwsAthenaKmsKey` |`String` |For SSE-KMS and
CSE-KMS, this is the KMS key ARN or ID. | No
-|`CamelAwsS3ServerSideEncryption` |String |Sets the server-side encryption
algorithm when encrypting
-the object using AWS-managed keys. For example use AES256.
+|startQueryExecution |`CamelAwsAthenaWaitTimeout` |`long` |Optional max wait
time in millis to wait for a successful query completion.
+See the section 'Waiting for Query Completion and Retrying Failed Queries' to
learn more. | No
-|`CamelAwsS3VersionId` |`String` |The version Id of the object to be stored or
returned from the current operation
-|=======================================================================
+|startQueryExecution |`CamelAwsAthenaInitialDelay` |`long` |Milliseconds
before the first poll for query execution status.
+See the section 'Waiting for Query Completion and Retrying Failed Queries' to
learn more. | No
-=== Message headers set by the S3 producer
+|startQueryExecution |`CamelAwsAthenaDelay` |`long` |Milliseconds before the
next poll for query execution status.
+See the section 'Waiting for Query Completion and Retrying Failed Queries' to
learn more. | No
-[width="100%",cols="10%,10%,80%",options="header",]
-|=======================================================================
-|Header |Type |Description
-|`CamelAwsS3ETag` |`String` |The ETag value for the newly uploaded object.
+|startQueryExecution |`CamelAwsAthenaMaxAttempts` |`int` |Maximum number of
times to attempt a query. Set to 1 to disable retries.
+See the section 'Waiting for Query Completion and Retrying Failed Queries' to
learn more. | No
-|`CamelAwsS3VersionId` |`String` |The *optional* version ID of the newly
uploaded object.
+|startQueryExecution |`CamelAwsAthenaRetry` |`String` |Optional comma
separated list of error types to retry the query for. Use
+'retryable' to retry all retryable failure conditions (e.g. generic errors and
resources exhausted),
+'generic' to retry 'GENERIC_INTERNAL_ERROR' failures,
+'exhausted' to retry queries that have exhausted resource limits,
+'always' to always retry regardless of failure condition, or
+'never' or null to never retry (default).
+See the section 'Waiting for Query Completion and Retrying Failed Queries' to
learn more. | No
-|`CamelAwsS3DownloadLinkExpiration` | `String` | The expiration (millis) of
URL download link. The link will be stored into *CamelAwsS3DownloadLink*
response header.
+|startQueryExecution |`CamelAwsAthenaResetWaitTimeoutOnRetry` |`boolean`
|Reset the waitTimeout countdown in the event of a query retry.
+If set to true, potential max time spent waiting for queries is equal to
waitTimeout x maxAttempts.
+See the section 'Waiting for Query Completion and Retrying Failed Queries' to
learn more. | No
|=======================================================================
-=== Message headers set by the S3 consumer
+=== Message headers set by the Athena producer
-[width="100%",cols="10%,10%,80%",options="header",]
+[width="100%",cols="5%,5%,10%,80%",options="header",]
|=======================================================================
-|Header |Type |Description
-
-|`CamelAwsS3Key` |`String` |The key under which this object is stored.
-
-|`CamelAwsS3BucketName` |`String` |The name of the bucket in which this object
is contained.
-
-|`CamelAwsS3ETag` |`String` |The hex encoded 128-bit MD5 digest of the
associated object according to
-RFC 1864. This data is used as an integrity check to verify that the
-data received by the caller is the same data that was sent by Amazon S3.
+|Operation |Header |Type |Description
-|`CamelAwsS3LastModified` |`Date` |The value of the Last-Modified header,
indicating the date and time at
-which Amazon S3 last recorded a modification to the associated object.
+|getQueryExecution, getQueryResults, startQueryExecution
|`CamelAwsAthenaQueryExecutionId` |`String` |The unique ID identifying the
query execution.
-|`CamelAwsS3VersionId` |`String` |The version ID of the associated Amazon S3
object if available. Version
-IDs are only assigned to objects when an object is uploaded to an Amazon
-S3 bucket that has object versioning enabled.
+|getQueryExecution, getQueryResults, startQueryExecution
|`CamelAwsAthenaQueryExecutionState`
|`software.amazon.awssdk.services.athena.model.QueryExecutionState` |The state
of the query execution. One of `QUEUED`, `RUNNING`, `SUCCEEDED`, `FAILED`,
`CANCELLED`.
-|`CamelAwsS3ContentType` |`String` |The Content-Type HTTP header, which
indicates the type of content stored
-in the associated object. The value of this header is a standard MIME
-type.
+|getQueryExecution, getQueryResults, startQueryExecution
|`CamelAwsAthenaOutputLocation` |`String` |The location in Amazon S3 where
query results are stored, such as s3://path/to/query/bucket/.
-|`CamelAwsS3ContentMD5` |`String` |The base64 encoded 128-bit MD5 digest of
the associated object (content
-- not including headers) according to RFC 1864. This data is used as a
-message integrity check to verify that the data received by Amazon S3 is
-the same data that the caller sent.
+|getQueryExecution, getQueryResults, startQueryExecution
|`CamelAwsAthenaQueryExecutionId` |`String` |The unique ID identifying the
query execution.
-|`CamelAwsS3ContentLength` |`Long` |The Content-Length HTTP header indicating
the size of the associated
-object in bytes.
+|getQueryResults, listQueryExecutions |`CamelAwsAthenaNextToken` |`String`
|Pagination token to use in the case where the response from the previous
request was truncated.
-|`CamelAwsS3ContentEncoding` |`String` |The *optional* Content-Encoding HTTP
header specifying what content
-encodings have been applied to the object and what decoding mechanisms
-must be applied in order to obtain the media-type referenced by the
-Content-Type field.
+|startQueryExecution |`CamelAwsAthenaStartQueryExecutionAttempts` |`int`
|Total number of attempts made to run the query. Will be greater than 1 if the
query is retried.
-|`CamelAwsS3ContentDisposition` |`String` |The *optional* Content-Disposition
HTTP header, which specifies
-presentational information such as the recommended filename for the
-object to be saved as.
+|startQueryExecution |`CamelAwsAthenaStartQueryExecutionElapsedMillis` |`long`
|Total time in millis taken in startQueryExecution (mostly relevant when
waiting for query completion within startQueryExecution).
-|`CamelAwsS3ContentControl` |`String` |The *optional* Cache-Control HTTP
header which allows the user to
-specify caching behavior along the HTTP request/reply chain.
-
-|`CamelAwsS3ServerSideEncryption` |String |The server-side encryption
algorithm when encrypting the
-object using AWS-managed keys.
|=======================================================================
-=== S3 Producer operations
+=== Athena Producer operations
-Camel-AWS s3 component provides the following operation on the producer side:
+The Camel-AWS Athena component provides the following operation on the
producer side:
-- copyObject
-- deleteObject
-- listBuckets
-- deleteBucket
-- listObjects
-- getObject (this will return an S3Object instance)
-- getObjectRange (this will return an S3Object instance)
+ - getQueryExecution
+ - getQueryResults
+ - listQueryExecutions
+ - startQueryExecution
-=== Advanced AmazonS3 configuration
+=== Advanced AmazonAthena configuration
If your Camel Application is running behind a firewall or if you need to
-have more control over the `S3Client` instance configuration, you can
-create your own instance and refer to it in your Camel aws2-s3 component
configuration:
+have more control over the `AthenaClient` instance configuration, you can
+create your own instance and refer to it in your Camel aws2-athena component
configuration:
[source,java]
--------------------------------------------------------------------------------
-from("aws2-s3://MyBucket?amazonS3Client=#client&delay=5000&maxMessagesPerPoll=5")
+from("aws2-athena://MyQuery?amazonAthenaClient=#client&...")
.to("mock:result");
--------------------------------------------------------------------------------
-=== Use KMS with the S3 component
+=== Overriding query parameters with message headers
+
+Message headers listed in "Message headers evaluated by the Athena producer"
override the corresponding
+query parameters listed in "Query Parameters".
-To use AWS KMS to encrypt/decrypt data by using AWS infrastructure you can use
the options introduced in 2.21.x like in the following example
+For example:
[source,java]
--------------------------------------------------------------------------------
-from("file:tmp/test?fileName=test.txt")
- .setHeader(S3Constants.KEY, constant("testFile"))
-
.to("aws2-s3://mybucket?amazonS3Client=#client&useAwsKMS=true&awsKMSKeyId=3f0637ad-296a-3dfe-a796-e60654fb128c");
+from("direct:start")
+ .setHeader(Athena2Constants.OUTPUT_LOCATION,
constant("s3://other/location/"))
+ .to("aws2-athena:label?outputLocation=s3://foo/bar/")
+ .to("mock:result");
--------------------------------------------------------------------------------
-In this way you'll ask to S3, to use the KMS key
3f0637ad-296a-3dfe-a796-e60654fb128c, to encrypt the file test.txt. When you'll
ask to download this file, the decryption will be done directly before the
download.
-
-=== Use "useIAMCredentials" with the s3 component
+Will cause the output location to be `s3://other/location/`.
-To use AWS IAM credentials, you must first verify that the EC2 in which you
are launching the Camel application on has an IAM role associated with it
containing the appropriate policies attached to run effectively.
-Keep in mind that this feature should only be set to "true" on remote
instances. To clarify even further, you must still use static credentials
locally since IAM is an AWS specific component,
-but AWS environments should now be easier to manage. After this is implemented
and understood, you can set the query parameter "useIAMCredentials" to "true"
for AWS environments! To effectively toggle this
-on and off based on local and remote environments, you can consider enabling
this query parameter with system environment variables. For example, your code
could set the "useIAMCredentials" query parameter to "true",
-when the system environment variable called "isRemote" is set to true (there
are many other ways to do this and this should act as a simple example).
Although it doesn't take away the need for static credentials completely,
-using IAM credentials on AWS environments takes away the need to refresh on
remote environments and adds a major security boost (IAM credentials are
refreshed automatically every 6 hours and update when their
-policies are updated). This is the AWS recommended way to manage credentials
and therefore should be used as often as possible.
+=== Athena Producer Operation examples
-=== S3 Producer Operation examples
-
-- CopyObject: this operation copy an object from one bucket to a different one
+- getQueryExecution: this operation returns information about a query given
its query execution ID
[source,java]
--------------------------------------------------------------------------------
- from("direct:start").process(new Processor() {
-
- @Override
- public void process(Exchange exchange) throws Exception {
- exchange.getIn().setHeader(S3Constants.BUCKET_DESTINATION_NAME,
"camelDestinationBucket");
- exchange.getIn().setHeader(S3Constants.KEY, "camelKey");
- exchange.getIn().setHeader(S3Constants.DESTINATION_KEY,
"camelDestinationKey");
- }
- })
-
.to("aws2-s3://mycamelbucket?amazonS3Client=#amazonS3Client&operation=copyObject")
- .to("mock:result");
+from("direct:start")
+
.to("aws2-athena://label?operation=getQueryExecution&queryExecutionId=11111111-1111-1111-1111-111111111111")
+ .to("mock:result");
--------------------------------------------------------------------------------
-This operation will copy the object with the name expressed in the header
camelDestinationKey to the camelDestinationBucket bucket, from the bucket
mycamelbucket.
+The preceding example will yield an
+https://docs.aws.amazon.com/athena/latest/APIReference/API_QueryExecution.html[Athena
QueryExecution] in the body.
-- DeleteObject: this operation deletes an object from a bucket
+The getQueryExecution operation also supports retreiving the query execution
ID from a header
+(`CamelAwsAthenaQueryExecutionId`), and since startQueryExecution sets the
same header upon starting a query,
+these operations can be used together:
[source,java]
--------------------------------------------------------------------------------
- from("direct:start").process(new Processor() {
-
- @Override
- public void process(Exchange exchange) throws Exception {
- exchange.getIn().setHeader(S3Constants.KEY, "camelKey");
- }
- })
-
.to("aws2-s3://mycamelbucket?amazonS3Client=#amazonS3Client&operation=deleteObject")
- .to("mock:result");
+from("direct:start")
+ .setBody(constant("SELECT 1"))
+
.to("aws2-athena://label?operation=startQueryExecution&outputLocation=s3://bucket/path/")
+ .to("aws2-athena://label?operation=getQueryExecution")
+ .to("mock:result");
--------------------------------------------------------------------------------
-This operation will delete the object camelKey from the bucket mycamelbucket.
+The preceding example will yield an Athena QueryExecution in the body for the
query that was just started.
-- ListBuckets: this operation list the buckets for this account in this region
+- getQueryResults: this operation returns the results of a query that has
succeeded. The results are returned in the
+body in one of three formats.
-[source,java]
---------------------------------------------------------------------------------
- from("direct:start")
-
.to("aws2-s3://mycamelbucket?amazonS3Client=#amazonS3Client&operation=listBuckets")
- .to("mock:result");
---------------------------------------------------------------------------------
-
-This operation will list the buckets for this account
-
-- DeleteBucket: this operation delete the bucket specified as URI parameter or
header
+`StreamList` - the default - returns a
+https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/athena/paginators/GetQueryResultsIterable.html[GetQueryResultsIterable]
+in the body that can page through all results:
[source,java]
--------------------------------------------------------------------------------
- from("direct:start")
-
.to("aws2-s3://mycamelbucket?amazonS3Client=#amazonS3Client&operation=deleteBucket")
- .to("mock:result");
+from("direct:start")
+ .setBody(constant("SELECT 1"))
+
.to("aws2-athena://label?operation=startQueryExecution&waitTimeout=60000&outputLocation=s3://bucket/path/")
+ .to("aws2-athena://label?operation=getQueryResults&outputType=StreamList")
+ .to("mock:result");
--------------------------------------------------------------------------------
-This operation will delete the bucket mycamelbucket
-
-- DownloadLink: this operation create a download link for the file specified
in the key header
+The output of StreamList can be processed in various ways:
[source,java]
--------------------------------------------------------------------------------
- from("direct:start").process(new Processor() {
+from("direct:start")
+ .setBody(constant(
+ "SELECT * FROM ("
+ + " VALUES"
+ + " (1, 'a'),"
+ + " (2, 'b')"
+ + ") AS t (id, name)"))
+
.to("aws2-athena://label?operation=startQueryExecution&waitTimeout=60000&outputLocation=s3://bucket/path/")
+ .to("aws2-athena://label?operation=getQueryResults&outputType=StreamList")
+ .split(body()).streaming()
+ .process(new Processor() {
@Override
- public void process(Exchange exchange) throws Exception {
- exchange.getIn().setHeader(S3Constants.KEY, "camelKey");
+ public void process(Exchange exchange) {
+ GetQueryResultsResponse page = exchange
+ .getMessage()
+
.getBody(GetQueryResultsResponse.class);
+ for (Row row : page.resultSet().rows()) {
+ String line = row.data()
+ .stream()
+ .map(Datum::varCharValue)
+ .collect(Collectors.joining(","));
+ System.out.println(line);
+ }
}
- })
-
.to("aws2-s3://mycamelbucket?amazonS3Client=#amazonS3Client&operation=downloadLink")
- .to("mock:result");
+ })
+ .to("mock:result");
--------------------------------------------------------------------------------
-This operation will create a downloadLink for the file camelKey in the bucket
mycamelbucket
+The preceding example will print the results of the query as CSV to the
console.
-- ListObjects: this operation list object in a specific bucket
+`SelectList` - returns a
+https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/athena/model/GetQueryResultsResponse.html[GetQueryResponse]
+in the body containing at most 1,000 rows, plus the NextToken value as a
header (`CamelAwsAthenaNextToken`),
+which can be used for manual pagination of results:
[source,java]
--------------------------------------------------------------------------------
- from("direct:start")
-
.to("aws2-s3://mycamelbucket?amazonS3Client=#amazonS3Client&operation=listObjects")
- .to("mock:result");
+from("direct:start")
+ .setBody(constant("SELECT 1"))
+
.to("aws2-athena://label?operation=startQueryExecution&waitTimeout=60000&outputLocation=s3://bucket/path/")
+ .to("aws2-athena://label?operation=getQueryResults&outputType=SelectList")
+ .to("mock:result");
--------------------------------------------------------------------------------
-This operation will list the objects in the mycamelbucket bucket
+The preceding example will return a
+https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/athena/model/GetQueryResultsResponse.html[GetQueryResponse]
+in the body plus the NextToken value as a header (`CamelAwsAthenaNextToken`),
which can be used to manually page
+through the results 1,000 rows at a time.
-- GetObject: this operation get a single object in a specific bucket
+`S3Pointer` - return an S3 path (e.g. `s3://bucket/path/`) pointing to the
results:
[source,java]
--------------------------------------------------------------------------------
- from("direct:start").process(new Processor() {
-
- @Override
- public void process(Exchange exchange) throws Exception {
- exchange.getIn().setHeader(S3Constants.KEY, "camelKey");
- }
- })
-
.to("aws2-s3://mycamelbucket?amazonS3Client=#amazonS3Client&operation=getObject")
- .to("mock:result");
+from("direct:start")
+ .setBody(constant("SELECT 1"))
+
.to("aws2-athena://label?operation=startQueryExecution&waitTimeout=60000&outputLocation=s3://bucket/path/")
+ .to("aws2-athena://label?operation=getQueryResults&outputType=S3Pointer")
+ .to("mock:result");
--------------------------------------------------------------------------------
-This operation will return an S3Object instance related to the camelKey object
in mycamelbucket bucket.
+The preceding example will return an
+S3 path (e.g. `s3://bucket/path/`) in the body pointing to the results. The
path will also be set in a header
+(`CamelAwsAthenaOutputLocation`).
-- GetObjectRange: this operation get a single object range in a specific bucket
+- listQueryExecutions: this operation returns a list of query execution IDs
[source,java]
--------------------------------------------------------------------------------
- from("direct:start").process(new Processor() {
-
- @Override
- public void process(Exchange exchange) throws Exception {
- exchange.getIn().setHeader(S3Constants.KEY, "camelKey");
- exchange.getIn().setHeader(S3Constants.RANGE_START, "0");
- exchange.getIn().setHeader(S3Constants.RANGE_END, "9");
- }
- })
-
.to("aws2-s3://mycamelbucket?amazonS3Client=#amazonS3Client&operation=getObjectRange")
- .to("mock:result");
+from("direct:start")
+ .to("aws2-athena://label?operation=listQueryExecutions")
+ .to("mock:result");
--------------------------------------------------------------------------------
-This operation will return an S3Object instance related to the camelKey object
in mycamelbucket bucket, containing a the bytes from 0 to 9.
+The preceding example will return a list of query executions in the body, plus
the NextToken value as a
+header (`CamelAwsAthenaNextToken`) than can be used for manual pagination of
results.
-== Bucket Autocreation
+- startQueryExecution: this operation starts the execution of a query. It
supports waiting for the query to
+complete before proceeding, and retrying the query based on a set of
configurable failure conditions:
-With the option `autoCreateBucket` users are able to avoid the autocreation of
an S3 Bucket in case it doesn't exist. The default for this option is `true`.
-If set to false any operation on a not-existent bucket in AWS won't be
successful and an error will be returned.
+[source,java]
+--------------------------------------------------------------------------------
+from("direct:start")
+ .setBody(constant("SELECT 1"))
+
.to("aws2-athena://label?operation=startQueryExecution&outputLocation=s3://bucket/path/")
+ .to("mock:result");
+--------------------------------------------------------------------------------
-== Automatic detection of AmazonS3 client in registry
+The preceding example will start the query `SELECT 1` and configure the
+results to be saved to `s3://bucket/path/`, but will not wait for the query
+to complete.
-The component is capable of detecting the presence of an AmazonS3 bean into
the registry.
-If it's the only instance of that type it will be used as client and you won't
have to define it as uri parameter, like the example above.
-This may be really useful for smarter configuration of the endpoint.
+[source,java]
+--------------------------------------------------------------------------------
+from("direct:start")
+ .setBody(constant("SELECT 1"))
+
.to("aws2-athena://label?operation=startQueryExecution&waitTimeout=60000&outputLocation=s3://bucket/path/")
+ .to("mock:result");
+--------------------------------------------------------------------------------
-== Moving stuff between a bucket and another bucket
+The preceding example will start a query and wait up to 60 seconds for it to
+reach a status that indicates it is complete (one of SUCCEEDED, FAILED,
CANCELLED,
+or UNKNOWN_TO_SDK_VERSION). Upon failure, the query would not be retried.
-Some users like to consume stuff from a bucket and move the content in a
different one without using the copyObject feature of this component.
-If this is case for you, don't forget to remove the bucketName header from the
incoming exchange of the consumer, otherwise the file will be always
overwritten on the same
-original bucket.
+[source,java]
+--------------------------------------------------------------------------------
+from("direct:start")
+ .setBody(constant("SELECT 1"))
+
.to("aws2-athena://label?operation=startQueryExecution&waitTimeout=60000&initialDelay=10000&delay=1000&maxAttempts=3&retry=retryable&outputLocation=s3://bucket/path/")
+ .to("mock:result");
+--------------------------------------------------------------------------------
-== MoveAfterRead consumer option
+The preceding example will start a query and wait up to 60 seconds for it to
reach
+a status that indicates it is complete (one of SUCCEEDED, FAILED, CANCELLED,
+or UNKNOWN_TO_SDK_VERSION). Upon failure, the query would be automatically
+retried up to 2 more times if the failure state indicates the query may
+succeed upon retry (Athena queries that fail with states such as
`GENERIC_INTERNAL_ERROR`
+or "resource limit exhaustion" will sometimes succeed if retried). While
waiting
+for the query to complete, the query status would first be checked after an
+initial delay of 10 seconds, and subsequently every 1 second until the query
completes.
-In addition to deleteAfterRead it has been added another option,
moveAfterRead. With this option enabled the consumed object will be moved to a
target destinationBucket instead of being only deleted.
-This will require specifying the destinationBucket option. As example:
+=== Putting it all together
[source,java]
--------------------------------------------------------------------------------
-
from("aws2-s3://mycamelbucket?amazonS3Client=#amazonS3Client&moveAfterRead=true&destinationBucket=myothercamelbucket")
- .to("mock:result");
+from("direct:start")
+ .setBody(constant("SELECT 1"))
+
.to("aws2-athena://label?waitTimeout=60000&&maxAttempts=3&retry=retryable&outputLocation=s3://bucket/path/")
+ .to("aws2-athena://label?operation=getQueryResults&outputType=StreamList")
+ .to("mock:result");
--------------------------------------------------------------------------------
-In this case the objects consumed will be moved to myothercamelbucket bucket
and deleted from the original one (because of deleteAfterRead set to true as
default).
-
-== Using customer key as encryption
+The preceding example will start the query and wait up to 60 seconds for it to
+complete. Upon completion, getQueryResults put the results of the query into
+the body of the message for further processing.
-We introduced also the customer key support (an alternative of using KMS). The
following code shows an example.
+For the sake of completeness, a similar outcome could be achieved with the
following:
[source,java]
--------------------------------------------------------------------------------
-String key = UUID.randomUUID().toString();
-byte[] secretKey = generateSecretKey();
-String b64Key = Base64.getEncoder().encodeToString(secretKey);
-String b64KeyMd5 = Md5Utils.md5AsBase64(secretKey);
-
-String awsEndpoint =
"aws2-s3://mycamel?autoCreateBucket=false&useCustomerKey=true&customerKeyId=RAW("
+ b64Key + ")&customerKeyMD5=RAW(" + b64KeyMd5 + ")&customerAlgorithm=" +
AES256.name();
-
-from("direct:putObject")
- .setHeader(AWS2S3Constants.KEY, constant("test.txt"))
- .setBody(constant("Test"))
- .to(awsEndpoint);
+from("direct:start")
+ .setBody(constant("SELECT 1"))
+
.to("aws2-athena://label?operation=startQueryExecution&outputLocation=s3://bucket/path/")
+ .loopDoWhile(simple("${header." + Athena2Constants.QUERY_EXECUTION_STATE +
"} != 'SUCCEEDED'"))
+ .delay(1_000)
+ .to("aws2-athena://label?operation=getQueryExecution")
+ .end()
+ .to("aws2-athena://label?operation=getQueryResults&outputType=StreamList")
+ .to("mock:result");
--------------------------------------------------------------------------------
-== Using a POJO as body
-
-Sometimes build an AWS Request can be complex, because of multiple options. We
introduce the possibility to use a POJO as body.
-In AWS S3 there are multiple operations you can submit, as an example for List
brokers request, you can do something like:
+Caution: the preceding example would block indefinitely, however, if the query
did not complete with a
+status of SUCCEEDED.
-------------------------------------------------------------------------------------------------------
-from("direct:aws2-s3")
- .setBody(ListObjectsRequest.builder().bucket(bucketName).build())
-
.to("aws2-s3://test?amazonS3Client=#amazonS3Client&operation=listObjects&pojoRequest=true")
-------------------------------------------------------------------------------------------------------
+== Automatic detection of AmazonAthena client in registry
-In this way you'll pass the request directly without the need of passing
headers and options specifically related to this operation.
+The component is capable of detecting the presence of an AmazonAthena bean in
the registry.
+If it's the only instance of that type it will be used as the client and you
won't have to define it as URI parameter, like the example above.
+This may be really useful for smarter configuration of the endpoint.
== Dependencies
@@ -496,7 +528,7 @@ Maven users will need to add the following dependency to
their pom.xml.
---------------------------------------
<dependency>
<groupId>org.apache.camel</groupId>
- <artifactId>camel-aws2-s3</artifactId>
+ <artifactId>camel-aws2-athena</artifactId>
<version>${camel-version}</version>
</dependency>
---------------------------------------
@@ -504,4 +536,4 @@ Maven users will need to add the following dependency to
their pom.xml.
where `$\{camel-version\}` must be replaced by the actual version of Camel.
-include::camel-spring-boot::page$aws2-s3-starter.adoc[]
+include::camel-spring-boot::page$aws2-athena-starter.adoc[]
