Repository: camel Updated Branches: refs/heads/master 1ed016f5d -> 87c393acd
CAMEL-10083 - Add "disconnectOnBatchComplete" option to close FTP connection immediately after Batch of upload complete Project: http://git-wip-us.apache.org/repos/asf/camel/repo Commit: http://git-wip-us.apache.org/repos/asf/camel/commit/87c393ac Tree: http://git-wip-us.apache.org/repos/asf/camel/tree/87c393ac Diff: http://git-wip-us.apache.org/repos/asf/camel/diff/87c393ac Branch: refs/heads/master Commit: 87c393acd6f6179eb943f61754dd18090472fcd9 Parents: 1ed016f Author: Andrea Cosentino <anco...@gmail.com> Authored: Fri Jun 24 15:37:12 2016 +0200 Committer: Andrea Cosentino <anco...@gmail.com> Committed: Fri Jun 24 15:39:32 2016 +0200 ---------------------------------------------------------------------- .../component/file/GenericFileProducer.java | 4 +- components/camel-ftp/src/main/docs/ftp.adoc | 5 +- .../file/remote/RemoteFileEndpoint.java | 15 ++++++ .../file/remote/RemoteFileProducer.java | 7 ++- ...tpProducerDisconnectOnBatchCompleteTest.java | 48 ++++++++++++++++++++ components/camel-jsch/src/main/docs/scp.adoc | 5 +- 6 files changed, 79 insertions(+), 5 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/camel/blob/87c393ac/camel-core/src/main/java/org/apache/camel/component/file/GenericFileProducer.java ---------------------------------------------------------------------- diff --git a/camel-core/src/main/java/org/apache/camel/component/file/GenericFileProducer.java b/camel-core/src/main/java/org/apache/camel/component/file/GenericFileProducer.java index 82f2e3b..b27af40 100644 --- a/camel-core/src/main/java/org/apache/camel/component/file/GenericFileProducer.java +++ b/camel-core/src/main/java/org/apache/camel/component/file/GenericFileProducer.java @@ -227,7 +227,7 @@ public class GenericFileProducer<T> extends DefaultProducer { handleFailedWrite(exchange, e); } - postWriteCheck(); + postWriteCheck(exchange); } /** @@ -248,7 +248,7 @@ public class GenericFileProducer<T> extends DefaultProducer { /** * Perform any actions that need to occur after we are done such as disconnecting. */ - public void postWriteCheck() { + public void postWriteCheck(Exchange exchange) { // nothing needed to check } http://git-wip-us.apache.org/repos/asf/camel/blob/87c393ac/components/camel-ftp/src/main/docs/ftp.adoc ---------------------------------------------------------------------- diff --git a/components/camel-ftp/src/main/docs/ftp.adoc b/components/camel-ftp/src/main/docs/ftp.adoc index 7c5d2ab..f6f9df4 100644 --- a/components/camel-ftp/src/main/docs/ftp.adoc +++ b/components/camel-ftp/src/main/docs/ftp.adoc @@ -114,8 +114,9 @@ The FTP component has no options. + // endpoint options: START -The FTP component supports 102 endpoint options which are listed below: +The FTP component supports 103 endpoint options which are listed below: {% raw %} [width="100%",cols="2s,1,1m,1m,5",options="header"] @@ -127,6 +128,7 @@ The FTP component supports 102 endpoint options which are listed below: | binary | common | false | boolean | Specifies the file transfer mode BINARY or ASCII. Default is ASCII (false). | charset | common | | String | This option is used to specify the encoding of the file. You can use this on the consumer to specify the encodings of the files which allow Camel to know the charset it should load the file content in case the file content is being accessed. Likewise when writing a file you can use this option to specify which charset to write the file as well. | disconnect | common | false | boolean | Whether or not to disconnect from remote FTP server right after use. Disconnect will only disconnect the current connection to the FTP server. If you have a consumer which you want to stop then you need to stop the consumer/route instead. +| disconnectOnBatchComplete | common | false | boolean | Whether or not to disconnect from remote FTP server right after a Batch is complete. disconnectOnBatchComplete will only disconnect the current connection to the FTP server. If you have a consumer which you want to stop then you need to stop the consumer/route instead. | download | common | false | boolean | Whether the FTP consumer should download the file. If this option is set to false then the message body will be null but the consumer will still trigger a Camel Exchange that has details about the file such as file name file size etc. It's just that the file will not be downloaded. | fastExistsCheck | common | false | boolean | If set this option to be true camel-ftp will use the list file directly to check if the file exists. Since some FTP server may not support to list the file directly if the option is false camel-ftp will use the old way to list the directory and check if the file exists. This option also influences readLock=changed to control whether it performs a fast check to update file information or not. This can be used to speed up the process if the FTP server has a lot of files. | fileName | common | | String | Use Expression such as File Language to dynamically set the filename. For consumers it's used as a filename filter. For producers it's used to evaluate the filename to write. If an expression is set it take precedence over the CamelFileName header. (Note: The header itself can also be an Expression). The expression options support both String and Expression types. If the expression is a String type it is always evaluated using the File Language. If the expression is an Expression type the specified Expression type is used - this allows you for instance to use OGNL expressions. For the consumer you can use it to filter filenames so you can for instance consume today's file using the File Language syntax: mydata-$date:now:yyyyMMdd.txt. The producers support the CamelOverruleFileName header which takes precedence over any existing CamelFileName header; the CamelOverruleFileName is a header that is used only once and makes it easier as this avoids to te mporary store CamelFileName and have to restore it afterwards. @@ -232,6 +234,7 @@ The FTP component supports 102 endpoint options which are listed below: + [Info] ==== FTPS component default trust store http://git-wip-us.apache.org/repos/asf/camel/blob/87c393ac/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/RemoteFileEndpoint.java ---------------------------------------------------------------------- diff --git a/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/RemoteFileEndpoint.java b/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/RemoteFileEndpoint.java index d334d4f..f6e3641 100644 --- a/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/RemoteFileEndpoint.java +++ b/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/RemoteFileEndpoint.java @@ -40,6 +40,8 @@ public abstract class RemoteFileEndpoint<T> extends GenericFileEndpoint<T> { @UriParam private boolean disconnect; @UriParam + private boolean disconnectOnBatchComplete; + @UriParam private boolean fastExistsCheck; @UriParam private boolean download = true; @@ -219,6 +221,19 @@ public abstract class RemoteFileEndpoint<T> extends GenericFileEndpoint<T> { this.disconnect = disconnect; } + public boolean isDisconnectOnBatchComplete() { + return disconnectOnBatchComplete; + } + + /** + * Whether or not to disconnect from remote FTP server right after a Batch is complete. + * disconnectOnBatchComplete will only disconnect the current connection to the FTP server. + * If you have a consumer which you want to stop, then you need to stop the consumer/route instead. + */ + public void setDisconnectOnBatchComplete(boolean disconnectOnBatchComplete) { + this.disconnectOnBatchComplete = disconnectOnBatchComplete; + } + public boolean isFastExistsCheck() { return fastExistsCheck; } http://git-wip-us.apache.org/repos/asf/camel/blob/87c393ac/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/RemoteFileProducer.java ---------------------------------------------------------------------- diff --git a/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/RemoteFileProducer.java b/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/RemoteFileProducer.java index 32a8a62..af4fc3d 100644 --- a/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/RemoteFileProducer.java +++ b/components/camel-ftp/src/main/java/org/apache/camel/component/file/remote/RemoteFileProducer.java @@ -144,8 +144,13 @@ public class RemoteFileProducer<T> extends GenericFileProducer<T> implements Ser } @Override - public void postWriteCheck() { + public void postWriteCheck(Exchange exchange) { try { + boolean isLast = exchange.getProperty(Exchange.BATCH_COMPLETE, true, Boolean.class); + if (isLast && getEndpoint().isDisconnectOnBatchComplete()) { + log.trace("postWriteCheck disconnect on batch complete from: {}", getEndpoint()); + disconnect(); + } if (getEndpoint().isDisconnect()) { log.trace("postWriteCheck disconnect from: {}", getEndpoint()); disconnect(); http://git-wip-us.apache.org/repos/asf/camel/blob/87c393ac/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/FtpProducerDisconnectOnBatchCompleteTest.java ---------------------------------------------------------------------- diff --git a/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/FtpProducerDisconnectOnBatchCompleteTest.java b/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/FtpProducerDisconnectOnBatchCompleteTest.java new file mode 100644 index 0000000..0a2e97f --- /dev/null +++ b/components/camel-ftp/src/test/java/org/apache/camel/component/file/remote/FtpProducerDisconnectOnBatchCompleteTest.java @@ -0,0 +1,48 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.camel.component.file.remote; + +import org.apache.commons.net.ftp.FTPClient; +import org.junit.Test; + +public class FtpProducerDisconnectOnBatchCompleteTest extends FtpServerTestSupport { + + @Override + public void setUp() throws Exception { + super.setUp(); + + // ask the singleton FtpEndpoint to make use of a custom FTPClient + // so that we can hold a reference on it inside the test below + FtpEndpoint<?> endpoint = context.getEndpoint(getFtpUrl(), FtpEndpoint.class); + endpoint.setFtpClient(new FTPClient()); + } + + private String getFtpUrl() { + return "ftp://admin@localhost:"; + getPort() + "/done?password=admin&disconnectOnBatchComplete=true"; + } + + @Test + public void testDisconnectOnBatchComplete() throws Exception { + sendFile(getFtpUrl(), "Hello World", "claus.txt"); + + Thread.sleep(2000); + FtpEndpoint<?> endpoint = context.getEndpoint(getFtpUrl(), FtpEndpoint.class); + assertFalse("The FTPClient should be already disconnected", endpoint.getFtpClient().isConnected()); + assertTrue("The FtpEndpoint should be configured to disconnect", endpoint.isDisconnectOnBatchComplete()); + } + +} \ No newline at end of file http://git-wip-us.apache.org/repos/asf/camel/blob/87c393ac/components/camel-jsch/src/main/docs/scp.adoc ---------------------------------------------------------------------- diff --git a/components/camel-jsch/src/main/docs/scp.adoc b/components/camel-jsch/src/main/docs/scp.adoc index 4f57c4c..d9a578b 100644 --- a/components/camel-jsch/src/main/docs/scp.adoc +++ b/components/camel-jsch/src/main/docs/scp.adoc @@ -63,8 +63,9 @@ The SCP component supports 1 options which are listed below. + // endpoint options: START -The SCP component supports 21 endpoint options which are listed below: +The SCP component supports 22 endpoint options which are listed below: {% raw %} [width="100%",cols="2s,1,1m,1m,5",options="header"] @@ -75,6 +76,7 @@ The SCP component supports 21 endpoint options which are listed below: | directoryName | producer | | String | The starting directory | chmod | producer | 664 | String | Allows you to set chmod on the stored file. For example chmod=664. | disconnect | producer | false | boolean | Whether or not to disconnect from remote FTP server right after use. Disconnect will only disconnect the current connection to the FTP server. If you have a consumer which you want to stop then you need to stop the consumer/route instead. +| disconnectOnBatchComplete | producer | false | boolean | Whether or not to disconnect from remote FTP server right after a Batch is complete. disconnectOnBatchComplete will only disconnect the current connection to the FTP server. If you have a consumer which you want to stop then you need to stop the consumer/route instead. | fileName | producer | | String | Use Expression such as File Language to dynamically set the filename. For consumers it's used as a filename filter. For producers it's used to evaluate the filename to write. If an expression is set it take precedence over the CamelFileName header. (Note: The header itself can also be an Expression). The expression options support both String and Expression types. If the expression is a String type it is always evaluated using the File Language. If the expression is an Expression type the specified Expression type is used - this allows you for instance to use OGNL expressions. For the consumer you can use it to filter filenames so you can for instance consume today's file using the File Language syntax: mydata-$date:now:yyyyMMdd.txt. The producers support the CamelOverruleFileName header which takes precedence over any existing CamelFileName header; the CamelOverruleFileName is a header that is used only once and makes it easier as this avoids to temporary store CamelFileName and have to restore it afterwards. | flatten | producer | false | boolean | Flatten is used to flatten the file name path to strip any leading paths so it's just the file name. This allows you to consume recursively into sub-directories but when you eg write the files to another directory they will be written in a single directory. Setting this to true on the producer enforces that any file name in CamelFileName header will be stripped for any leading paths. | strictHostKeyChecking | producer | no | String | Sets whether to use strict host key checking. Possible values are: no yes @@ -99,6 +101,7 @@ The SCP component supports 21 endpoint options which are listed below: + [[Jsch-Limitations]] Limitations ^^^^^^^^^^^
