This is an automated email from the ASF dual-hosted git repository.
acosentino pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/camel.git
commit d7c437eb3f476e366787bac80a384bed17f463fe
Author: Anderson Vaz <d...@andersonvaz.com>
AuthorDate: Tue Sep 22 09:27:34 2020 +0200
Changes S3Object manipulation to avoid consume large S3 into the memory and
consume it as stream instead.
---
.../org/apache/camel/catalog/docs/aws2-s3-component.adoc | 4 ++--
.../camel-aws2-s3/src/main/docs/aws2-s3-component.adoc | 4 ++--
.../org/apache/camel/component/aws2/s3/AWS2S3Endpoint.java | 14 ++++++++------
3 files changed, 12 insertions(+), 10 deletions(-)
diff --git
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/aws2-s3-component.adoc
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/aws2-s3-component.adoc
index f537122..1c2525b 100644
---
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/aws2-s3-component.adoc
+++
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/aws2-s3-component.adoc
@@ -78,11 +78,11 @@ The AWS 2 S3 Storage Service component supports 42 options,
which are listed bel
| *destinationBucketPrefix* (consumer) | Define the destination bucket prefix
to use when an object must be moved and moveAfterRead is set to true. | |
String
| *destinationBucketSuffix* (consumer) | Define the destination bucket suffix
to use when an object must be moved and moveAfterRead is set to true. | |
String
| *fileName* (consumer) | To get the object from the bucket with the given
file name | | String
-| *includeBody* (consumer) | If it is true, the exchange body will be set to a
stream to the contents of the file. If false, the headers will be set with the
S3 object metadata, but the body will be null. This option is strongly related
to autocloseBody option. In case of setting includeBody to true and
autocloseBody to false, it will be up to the caller to close the S3Object
stream. Setting autocloseBody to true, will close the S3Object stream
automatically. | true | boolean
+| *includeBody* (consumer) | If it is true, the S3Object exchange will be
consumed and put into the body and closed. If false the S3Object stream will be
put raw into the body and the headers will be set with the S3 object metadata.
This option is strongly related to autocloseBody option. In case of setting
includeBody to true because the S3Object stream will be consumed then it will
also be closed in case of includeBody false then it will be up to the caller to
close the S3Object stream [...]
| *includeFolders* (consumer) | If it is true, the folders/directories will be
consumed. If it is false, they will be ignored, and Exchanges will not be
created for those | true | boolean
| *moveAfterRead* (consumer) | Move objects from S3 bucket to a different
bucket after they have been retrieved. To accomplish the operation the
destinationBucket option must be set. The copy bucket operation is only
performed if the Exchange is committed. If a rollback occurs, the object is not
moved. | false | boolean
| *prefix* (consumer) | The prefix which is used in the
com.amazonaws.services.s3.model.ListObjectsRequest to only consume objects we
are interested in. | | String
-| *autocloseBody* (consumer) | If this option is true and includeBody is true,
then the S3Object.close() method will be called on exchange completion. This
option is strongly related to includeBody option. In case of setting
includeBody to true and autocloseBody to false, it will be up to the caller to
close the S3Object stream. Setting autocloseBody to true, will close the
S3Object stream automatically. | true | boolean
+| *autocloseBody* (consumer) | If this option is true and includeBody is
false, then the S3Object.close() method will be called on exchange completion.
This option is strongly related to includeBody option. In case of setting
includeBody to false and autocloseBody to false, it will be up to the caller to
close the S3Object stream. Setting autocloseBody to true, will close the
S3Object stream automatically. | true | boolean
| *deleteAfterWrite* (producer) | Delete file object after the S3 file has
been uploaded | false | boolean
| *keyName* (producer) | Setting the key name for an element in the bucket
through endpoint parameter | | String
| *lazyStartProducer* (producer) | Whether the producer should be started lazy
(on the first message). By starting lazy you can use this to allow CamelContext
and routes to startup in situations where a producer may otherwise fail during
starting and cause the route to fail being started. By deferring this startup
to be lazy then the startup failure can be handled during routing messages via
Camel's routing error handlers. Beware that when the first message is processed
then creating and [...]
diff --git a/components/camel-aws2-s3/src/main/docs/aws2-s3-component.adoc
b/components/camel-aws2-s3/src/main/docs/aws2-s3-component.adoc
index f537122..1c2525b 100644
--- a/components/camel-aws2-s3/src/main/docs/aws2-s3-component.adoc
+++ b/components/camel-aws2-s3/src/main/docs/aws2-s3-component.adoc
@@ -78,11 +78,11 @@ The AWS 2 S3 Storage Service component supports 42 options,
which are listed bel
| *destinationBucketPrefix* (consumer) | Define the destination bucket prefix
to use when an object must be moved and moveAfterRead is set to true. | |
String
| *destinationBucketSuffix* (consumer) | Define the destination bucket suffix
to use when an object must be moved and moveAfterRead is set to true. | |
String
| *fileName* (consumer) | To get the object from the bucket with the given
file name | | String
-| *includeBody* (consumer) | If it is true, the exchange body will be set to a
stream to the contents of the file. If false, the headers will be set with the
S3 object metadata, but the body will be null. This option is strongly related
to autocloseBody option. In case of setting includeBody to true and
autocloseBody to false, it will be up to the caller to close the S3Object
stream. Setting autocloseBody to true, will close the S3Object stream
automatically. | true | boolean
+| *includeBody* (consumer) | If it is true, the S3Object exchange will be
consumed and put into the body and closed. If false the S3Object stream will be
put raw into the body and the headers will be set with the S3 object metadata.
This option is strongly related to autocloseBody option. In case of setting
includeBody to true because the S3Object stream will be consumed then it will
also be closed in case of includeBody false then it will be up to the caller to
close the S3Object stream [...]
| *includeFolders* (consumer) | If it is true, the folders/directories will be
consumed. If it is false, they will be ignored, and Exchanges will not be
created for those | true | boolean
| *moveAfterRead* (consumer) | Move objects from S3 bucket to a different
bucket after they have been retrieved. To accomplish the operation the
destinationBucket option must be set. The copy bucket operation is only
performed if the Exchange is committed. If a rollback occurs, the object is not
moved. | false | boolean
| *prefix* (consumer) | The prefix which is used in the
com.amazonaws.services.s3.model.ListObjectsRequest to only consume objects we
are interested in. | | String
-| *autocloseBody* (consumer) | If this option is true and includeBody is true,
then the S3Object.close() method will be called on exchange completion. This
option is strongly related to includeBody option. In case of setting
includeBody to true and autocloseBody to false, it will be up to the caller to
close the S3Object stream. Setting autocloseBody to true, will close the
S3Object stream automatically. | true | boolean
+| *autocloseBody* (consumer) | If this option is true and includeBody is
false, then the S3Object.close() method will be called on exchange completion.
This option is strongly related to includeBody option. In case of setting
includeBody to false and autocloseBody to false, it will be up to the caller to
close the S3Object stream. Setting autocloseBody to true, will close the
S3Object stream automatically. | true | boolean
| *deleteAfterWrite* (producer) | Delete file object after the S3 file has
been uploaded | false | boolean
| *keyName* (producer) | Setting the key name for an element in the bucket
through endpoint parameter | | String
| *lazyStartProducer* (producer) | Whether the producer should be started lazy
(on the first message). By starting lazy you can use this to allow CamelContext
and routes to startup in situations where a producer may otherwise fail during
starting and cause the route to fail being started. By deferring this startup
to be lazy then the startup failure can be handled during routing messages via
Camel's routing error handlers. Beware that when the first message is processed
then creating and [...]
diff --git
a/components/camel-aws2-s3/src/main/java/org/apache/camel/component/aws2/s3/AWS2S3Endpoint.java
b/components/camel-aws2-s3/src/main/java/org/apache/camel/component/aws2/s3/AWS2S3Endpoint.java
index c9e3b9c..03b47e5 100644
---
a/components/camel-aws2-s3/src/main/java/org/apache/camel/component/aws2/s3/AWS2S3Endpoint.java
+++
b/components/camel-aws2-s3/src/main/java/org/apache/camel/component/aws2/s3/AWS2S3Endpoint.java
@@ -175,7 +175,7 @@ public class AWS2S3Endpoint extends ScheduledPollEndpoint {
throw new RuntimeCamelException(e);
}
} else {
- message.setBody(null);
+ message.setBody(s3Object);
}
message.setHeader(AWS2S3Constants.KEY, key);
@@ -193,12 +193,14 @@ public class AWS2S3Endpoint extends ScheduledPollEndpoint
{
message.setHeader(AWS2S3Constants.REPLICATION_STATUS,
s3Object.response().replicationStatus());
message.setHeader(AWS2S3Constants.STORAGE_CLASS,
s3Object.response().storageClass());
- /**
- * If includeBody != true, it is safe to close the object here. If
includeBody == true, the caller is
- * responsible for closing the stream and object once the body has
been fully consumed. As of 2.17, the consumer
- * does not close the stream or object on commit.
+ /*
+ * If includeBody == true, it is safe to close the object here because
the S3Object
+ * was consumed already. If includeBody != true, the caller is
responsible for
+ * closing the stream once the body has been fully consumed or use the
autoCloseBody
+ * configuration to automatically schedule the body closing at the end
of exchange.
+ * TODO REVIEW THIS STATEMENT As of 2.17, the consumer does not close
the stream or object on commit.
*/
- if (!configuration.isIncludeBody()) {
+ if (configuration.isIncludeBody()) {
IOHelper.close(s3Object);
} else {
if (configuration.isAutocloseBody()) {
