Croway opened a new pull request, #30087: URL: https://github.com/apache/camel/pull/30087
[CAMEL-26999](https://issues.apache.org/jira/browse/CAMEL-26999) and [CAMEL-27000](https://issues.apache.org/jira/browse/CAMEL-27000): an FTP server, or the network path to it, that stops answering on an open connection (a hung server, a connection dropped by a firewall without a reset) costs `soTimeout` (5 minutes by default) again and again. Found with fault-injection tests; reproduced with an in-process fake FTP server (no Docker). ### CAMEL-26999: disconnect waits up to soTimeout for the reply to QUIT `FtpOperations.doDisconnect()` calls `client.logout()`, which in commons-net 3.13.0 sends QUIT and reads the reply with the SO_TIMEOUT of the control connection, i.e. `soTimeout`. So: * `RemoteFileConsumer.doStop()` / `RemoteFileProducer.doStop()` hold `CamelContext.stop()` for `soTimeout` per connected endpoint, also with nothing in flight (the shutdown timeout does not bound it: the second stop from the caller waits on the lock of `BaseService.stop()`). * `forceDisconnect()` called the same `doDisconnect()`, so after a failed poll `RemoteFilePollingConsumerPollStrategy.rollback()` sent QUIT on the dead connection and waited `soTimeout` once more before the next poll could connect. Fix (in `FtpOperations` only): * `forceDisconnect()` closes the connection without QUIT: it is the hard disconnect of the `RemoteFileOperations` contract, used to recover from a failure (`SftpOperations.forceDisconnect()` already closes the session without waiting). * `doDisconnect()` still sends QUIT, but waits for the reply at most 2 s (`soTimeout` when it is shorter, also when `soTimeout=0`). A healthy server answers QUIT at once; the connection is closed either way. ### CAMEL-27000: after a read timeout, the rest of the batch runs on the dead connection A failed download does not leave `GenericFileConsumer.processBatch()`, and commons-net leaves the timed-out connection open (with the reply to RETR unread, so a late reply would be read as the reply to the next command). Each remaining file sent its PWD (`stepwise=true`) on it and waited `soTimeout`: recovery took (remaining files + 2) x `soTimeout` and one WARN per file. The next poll's NOOP waited once more, then `connect()` replaced the socket without closing it. Fix: * `FtpConsumer.handleException()` (camel-ftp; no change to the shared `RemoteFileConsumer`): when a file failed because the server did not answer in time (a `SocketTimeoutException` in the cause: `soTimeout` on the control connection, `timeout` on the data connection), force-disconnect (now without QUIT) and end the batch through `isBatchAllowed()`. `processBatch()` releases the remaining files, as when the consumer stops, and the next poll connects again and picks them up. * `FtpOperations.doConnect()` closes a client that is still connected before connecting again, so a connection whose NOOP timed out is not left open. Only a timeout ends the batch. Other I/O failures keep the behaviour of CAMEL-25915 (the NOOP before the next poll decides): a reset connection fails fast, and a local error such as a full disk does not drop a good connection (`RemoteFileConsumerSendNoopTest` covers both and is unchanged). The two fixes are in one PR because the batch fix relies on the forced disconnect not sending QUIT (otherwise it would still wait `soTimeout` for that reply), and both tests use the same new fake FTP server. Two commits, one per JIRA. ### Tests New unit tests with `FakeFtpServer`, an in-process FTP server (passive mode) that can stop answering on an open connection while it serves new ones: * `FtpDisconnectFromSilentServerTest` (CAMEL-26999): `CamelContext.stop()` with an idle consumer / producer whose server stopped answering (`soTimeout` 15 s) returns in less than 10 s and sends QUIT; a failed poll (LIST timed out) does not send QUIT on the dead connection. * `FtpConsumerBatchAfterReadTimeoutTest` (CAMEL-27000): 10 files, the server stops answering at the RETR of the 3rd, before the reply or in the middle of the transfer: only `f-03` fails, nothing more is sent on the dead connection, it is closed, and all 10 files are delivered; a connection that stops answering between two polls is closed when the consumer connects again. Before the fix (main): all 6 fail, deterministically: * `stopIsNotHeldByAnIdleConsumerWhoseServerStoppedAnswering` / `...Producer...`: `CamelContext.stop() took 15002 ms with soTimeout 15000 ms` * `failedPollDoesNotSendQuitOnTheConnectionThatStoppedAnswering`: `expected: <[]> but was: <[QUIT]>` * `readTimeoutFailsOnlyTheFileBeingTransferred` (both stall modes): `expected: <[f-03.txt]> but was: <[f-03.txt, f-04.txt, ..., f-10.txt]>` * `connectionThatStoppedAnsweringIsClosedBeforeConnectingAgain`: the connection that stopped answering is still open 5 s after the reconnect After the fix: the 6 new tests pass (about 5 s per class), and all camel-ftp unit tests (188) and ITs (500, embedded FTP/SFTP servers) pass but one, `FtpProducerHealthCheckIT`: closing the old connection before connecting resets the reply string of commons-net, so a refused connect got the reason `null Connection refused`. The connect failure now uses an empty reply string, as before; `FtpProducerHealthCheckIT` and the connect, reconnect, disconnect and login-leak ITs pass again. ### Docs and upgrade guide * `ftp-component.adoc`: a new section "A server that stops answering", after "Concurrency". It is not in "About timeouts" because #30062 rewrites that section; it can move there once both are merged. * `camel-4x-upgrade-guide-4_24.adoc`: the wait for the reply to QUIT is bounded to 2 s, the forced disconnect sends no QUIT, and a timeout while retrieving a file ends the batch. _Claude Code on behalf of Croway_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
