[
https://issues.apache.org/jira/browse/GUACAMOLE-2180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Couchman updated GUACAMOLE-2180:
-------------------------------------
Affects Version/s: (was: 1.6.0)
> Improve SFTP upload performance
> -------------------------------
>
> Key: GUACAMOLE-2180
> URL: https://issues.apache.org/jira/browse/GUACAMOLE-2180
> Project: Guacamole
> Issue Type: Improvement
> Components: SSH
> Environment: The test environment consists of:
> - Guacamole environment
> - An Ubuntu virtual machine running in a cloud provider
> - The machine hosts a Docker Compose setup containing:
> - postgres (Guacamole database)
> - guacamole-server
> - guacamole-client
> - SSH Target
> - The actual SSH server is located on a user’s private LAN (on-prem)
> - It is not directly reachable from the cloud VM
> Connectivity setup
> The user initiates an SSH remote port forward from their local network to the
> cloud VM
> (e.g., ssh -R <cloud-vm-port>:localhost:22 cloud-vm)
> As a result, the cloud VM can access the on-prem SSH target through the
> forwarded port
> Guacamole uses that forwarded port as its SFTP/SSH target endpoint
> Reporter: Alailson Andrade do Nascimento
> Priority: Major
> Labels: performance, sftp, ssh
> Fix For: 1.6.1
>
> Attachments: image-2025-12-05-12-46-35-184.png,
> image-2025-12-05-13-43-06-876.png, image-2025-12-05-13-44-19-318.png,
> image-2025-12-05-13-45-49-566.png, image-2025-12-05-13-46-10-386.png,
> image-2025-12-05-13-50-56-071.png, image-2025-12-05-13-51-03-874.png,
> image-2025-12-05-13-51-54-825.png, image-2025-12-05-13-52-01-566.png
>
> Original Estimate: 40h
> Remaining Estimate: 40h
>
> h2. *Overview*
> SFTP upload performance in Apache Guacamole is significantly slower than
> expected under typical usage conditions.
> Guacamole’s SSH/SFTP subsystem currently relies on *libssh2* for SFTP
> operations. While widely used, libssh2 has {*}known throughput limitations{*}.
> Relevant upstream references:
> * libssh2 original performance issue:
> [https://github.com/libssh2/libssh2/issues/90|https://github.com/libssh2/libssh2/issues/90?utm_source=chatgpt.com]
> * Recent community discussion on SFTP throughput constraints:
> [https://github.com/libssh2/libssh2/discussions/1064]
> These discussions highlight structural constraints in libssh2’s SFTP
> implementation that can severely limit effective transfer speeds—especially
> across higher-latency networks or when handling large files.
> ----
> h2. *Summary of the Problem*
> File uploads via Guacamole’s SFTP subsystem currently perform {*}far below
> expected throughput levels{*}, especially when transferring large files to
> SSH targets located on different networks.
> h3. *Observed behavior*
> * When uploading a large file to an SSH target located across a different
> network segment (moderate latency), upload performance degrades dramatically.
> * In real testing, *each 4 MB chunk required ~1.1 minutes* to complete.
> * This results in multi-hundred-MB uploads taking *tens of minutes* instead
> of seconds.
> !image-2025-12-05-13-50-56-071.png!
> !image-2025-12-05-13-51-03-874.png!
> h3. *Underlying cause*
> These symptoms match known limitations in libssh2’s SFTP implementation,
> including:
> * *Inefficient or absent request pipelining*
> (single outstanding request → multiple round trips)
> * High sensitivity to network latency due to per-request round-trip delays
> As a result, Guacamole’s SFTP performance falls well below that of other SFTP
> clients that optimize their SFTP pipelines (e.g., OpenSSH’s {{{}sftp{}}}).
> ----
> h2. *Proof of Concept / Prototype Fix*
> A working proof-of-concept implementation demonstrating substantial
> throughput improvement is available here:
> 🔗 [https://github.com/alailsonko/guacamole-server/pull/1]
> h3. *Changes introduced in the prototype*
> The prototype improves SFTP upload throughput by:
> * Increasing libssh2 SFTP write buffer sizes
> * Allowing multiple outstanding SFTP write requests (pipelined writes)
> * Reducing the number of round trips required for each data chunk
> * Optimizing batching behavior during sequential writes
> h3. *Results*
> On the exact same environment and SSH target:
> * Current implementation:
> *4 MB upload takes ~1.1 minutes*
> * Prototype implementation:
> *4 MB upload takes ~10 seconds*
> This represents a {*}6× improvement{*}, with room for further optimization.
> It was validated with verification of *checksum* of the file transferred.
> !image-2025-12-05-13-51-54-825.png!
> !image-2025-12-05-13-52-01-566.png!
> ----
> h2. *Expected Behavior*
> Guacamole’s SFTP upload performance should scale in line with:
> * Available network bandwidth
> * Server-side SSH capabilities
> * Optimized libssh2 SFTP behavior (buffered + pipelined writes)
> Users should experience upload speeds comparable to standard tools such as:
> * OpenSSH {{sftp}}
> * {{scp}}
> * libssh2-tuned SFTP clients
> ----
> h2. *Actual Behavior*
> SFTP uploads through Guacamole are:
> * *Significantly slower* than standard OpenSSH SFTP transfers
> * Disproportionately impacted by network latency
> Large file transfers may become effectively unusable.
> ----
> h2. *Impact*
> * Users experience extremely slow upload speeds when interacting with
> SSH/SFTP targets through Guacamole.
> * Large file transfers (10 MB+) become impractical, especially across WAN
> links.
> * The performance gap between Guacamole and native SSH clients reduces user
> confidence and usability.
> * Administrators often assume a network issue when the root cause is the
> internal SFTP pipeline implementation.
> Thanks, the point is to contribute to the project
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
