Alailson Andrade do Nascimento created GUACAMOLE-2180:
---------------------------------------------------------
Summary: Improve SFTP upload performance
Key: GUACAMOLE-2180
URL: https://issues.apache.org/jira/browse/GUACAMOLE-2180
Project: Guacamole
Issue Type: Improvement
Components: SSH
Affects Versions: 1.6.0
Environment: The test environment consists of:
- Guacamole environment
- An Ubuntu virtual machine running in a cloud provider
- The machine hosts a Docker Compose setup containing:
- postgres (Guacamole database)
- guacamole-server
- guacamole-client
- SSH Target
- The actual SSH server is located on a user’s private LAN (on-prem)
- It is not directly reachable from the cloud VM
Connectivity setup
The user initiates an SSH remote port forward from their local network to the
cloud VM
(e.g., ssh -R <cloud-vm-port>:localhost:22 cloud-vm)
As a result, the cloud VM can access the on-prem SSH target through the
forwarded port
Guacamole uses that forwarded port as its SFTP/SSH target endpoint
Reporter: Alailson Andrade do Nascimento
Fix For: 1.6.1
Attachments: image-2025-12-05-12-46-35-184.png,
image-2025-12-05-13-43-06-876.png, image-2025-12-05-13-44-19-318.png,
image-2025-12-05-13-45-49-566.png, image-2025-12-05-13-46-10-386.png,
image-2025-12-05-13-50-56-071.png, image-2025-12-05-13-51-03-874.png,
image-2025-12-05-13-51-54-825.png, image-2025-12-05-13-52-01-566.png
h2. *Overview*
SFTP upload performance in Apache Guacamole is significantly slower than
expected under typical usage conditions.
Guacamole’s SSH/SFTP subsystem currently relies on *libssh2* for SFTP
operations. While widely used, libssh2 has {*}known throughput limitations{*}.
Relevant upstream references:
* libssh2 original performance issue:
[https://github.com/libssh2/libssh2/issues/90|https://github.com/libssh2/libssh2/issues/90?utm_source=chatgpt.com]
* Recent community discussion on SFTP throughput constraints:
[https://github.com/libssh2/libssh2/discussions/1064]
These discussions highlight structural constraints in libssh2’s SFTP
implementation that can severely limit effective transfer speeds—especially
across higher-latency networks or when handling large files.
----
h2. *Summary of the Problem*
File uploads via Guacamole’s SFTP subsystem currently perform {*}far below
expected throughput levels{*}, especially when transferring large files to SSH
targets located on different networks.
h3. *Observed behavior*
* When uploading a large file to an SSH target located across a different
network segment (moderate latency), upload performance degrades dramatically.
* In real testing, *each 4 MB chunk required ~1.1 minutes* to complete.
* This results in multi-hundred-MB uploads taking *tens of minutes* instead of
seconds.
!image-2025-12-05-13-50-56-071.png!
!image-2025-12-05-13-51-03-874.png!
h3. *Underlying cause*
These symptoms match known limitations in libssh2’s SFTP implementation,
including:
* *Inefficient or absent request pipelining*
(single outstanding request → multiple round trips)
* High sensitivity to network latency due to per-request round-trip delays
As a result, Guacamole’s SFTP performance falls well below that of other SFTP
clients that optimize their SFTP pipelines (e.g., OpenSSH’s {{{}sftp{}}}).
----
h2. *Proof of Concept / Prototype Fix*
A working proof-of-concept implementation demonstrating substantial throughput
improvement is available here:
🔗 [https://github.com/alailsonko/guacamole-server/pull/1]
h3. *Changes introduced in the prototype*
The prototype improves SFTP upload throughput by:
* Increasing libssh2 SFTP write buffer sizes
* Allowing multiple outstanding SFTP write requests (pipelined writes)
* Reducing the number of round trips required for each data chunk
* Optimizing batching behavior during sequential writes
h3. *Results*
On the exact same environment and SSH target:
* Current implementation:
*4 MB upload takes ~1.1 minutes*
* Prototype implementation:
*4 MB upload takes ~10 seconds*
This represents a {*}6× improvement{*}, with room for further optimization. It
was validated with verification of *checksum* of the file transferred.
!image-2025-12-05-13-51-54-825.png!
!image-2025-12-05-13-52-01-566.png!
----
h2. *Expected Behavior*
Guacamole’s SFTP upload performance should scale in line with:
* Available network bandwidth
* Server-side SSH capabilities
* Optimized libssh2 SFTP behavior (buffered + pipelined writes)
Users should experience upload speeds comparable to standard tools such as:
* OpenSSH {{sftp}}
* {{scp}}
* libssh2-tuned SFTP clients
----
h2. *Actual Behavior*
SFTP uploads through Guacamole are:
* *Significantly slower* than standard OpenSSH SFTP transfers
* Disproportionately impacted by network latency
Large file transfers may become effectively unusable.
----
h2. *Impact*
* Users experience extremely slow upload speeds when interacting with SSH/SFTP
targets through Guacamole.
* Large file transfers (10 MB+) become impractical, especially across WAN
links.
* The performance gap between Guacamole and native SSH clients reduces user
confidence and usability.
* Administrators often assume a network issue when the root cause is the
internal SFTP pipeline implementation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
