[
https://issues.apache.org/jira/browse/GUACAMOLE-2180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alailson Andrade do Nascimento updated GUACAMOLE-2180:
------------------------------------------------------
Description:
h2. *Overview*
SFTP upload performance in Apache Guacamole is significantly slower than
expected under typical usage conditions.
Guacamole’s SSH/SFTP subsystem currently relies on *libssh2* for SFTP
operations. While widely used, libssh2 has {*}known throughput limitations{*}.
Relevant upstream references:
* libssh2 original performance issue:
[https://github.com/libssh2/libssh2/issues/90|https://github.com/libssh2/libssh2/issues/90?utm_source=chatgpt.com]
* Recent community discussion on SFTP throughput constraints:
[https://github.com/libssh2/libssh2/discussions/1064]
These discussions highlight structural constraints in libssh2’s SFTP
implementation that can severely limit effective transfer speeds—especially
across higher-latency networks or when handling large files.
----
h2. *Summary of the Problem*
File uploads via Guacamole’s SFTP subsystem currently perform {*}far below
expected throughput levels{*}, especially when transferring large files to SSH
targets located on different networks.
h3. *Observed behavior*
* When uploading a large file to an SSH target located across a different
network segment (moderate latency), upload performance degrades dramatically.
* In real testing, *each 4 MB chunk required ~1.1 minutes* to complete.
* This results in multi-hundred-MB uploads taking *tens of minutes* instead of
seconds.
!image-2025-12-05-13-50-56-071.png!
!image-2025-12-05-13-51-03-874.png!
h3. *Underlying cause*
These symptoms match known limitations in libssh2’s SFTP implementation,
including:
* *Inefficient or absent request pipelining*
(single outstanding request → multiple round trips)
* High sensitivity to network latency due to per-request round-trip delays
As a result, Guacamole’s SFTP performance falls well below that of other SFTP
clients that optimize their SFTP pipelines (e.g., OpenSSH’s {{{}sftp{}}}).
----
h2. *Proof of Concept / Prototype Fix*
A working proof-of-concept implementation demonstrating substantial throughput
improvement is available here:
🔗 [https://github.com/alailsonko/guacamole-server/pull/1]
h3. *Changes introduced in the prototype*
The prototype improves SFTP upload throughput by:
* Increasing libssh2 SFTP write buffer sizes
* Allowing multiple outstanding SFTP write requests (pipelined writes)
* Reducing the number of round trips required for each data chunk
* Optimizing batching behavior during sequential writes
h3. *Results*
On the exact same environment and SSH target:
* Current implementation:
*4 MB upload takes ~1.1 minutes*
* Prototype implementation:
*4 MB upload takes ~10 seconds*
This represents a {*}6× improvement{*}, with room for further optimization. It
was validated with verification of *checksum* of the file transferred.
!image-2025-12-05-13-51-54-825.png!
!image-2025-12-05-13-52-01-566.png!
----
h2. *Expected Behavior*
Guacamole’s SFTP upload performance should scale in line with:
* Available network bandwidth
* Server-side SSH capabilities
* Optimized libssh2 SFTP behavior (buffered + pipelined writes)
Users should experience upload speeds comparable to standard tools such as:
* OpenSSH {{sftp}}
* {{scp}}
* libssh2-tuned SFTP clients
----
h2. *Actual Behavior*
SFTP uploads through Guacamole are:
* *Significantly slower* than standard OpenSSH SFTP transfers
* Disproportionately impacted by network latency
Large file transfers may become effectively unusable.
----
h2. *Impact*
* Users experience extremely slow upload speeds when interacting with SSH/SFTP
targets through Guacamole.
* Large file transfers (10 MB+) become impractical, especially across WAN
links.
* The performance gap between Guacamole and native SSH clients reduces user
confidence and usability.
* Administrators often assume a network issue when the root cause is the
internal SFTP pipeline implementation.
Thanks, the point is to contribute to the project
was:
h2. *Overview*
SFTP upload performance in Apache Guacamole is significantly slower than
expected under typical usage conditions.
Guacamole’s SSH/SFTP subsystem currently relies on *libssh2* for SFTP
operations. While widely used, libssh2 has {*}known throughput limitations{*}.
Relevant upstream references:
* libssh2 original performance issue:
[https://github.com/libssh2/libssh2/issues/90|https://github.com/libssh2/libssh2/issues/90?utm_source=chatgpt.com]
* Recent community discussion on SFTP throughput constraints:
[https://github.com/libssh2/libssh2/discussions/1064]
These discussions highlight structural constraints in libssh2’s SFTP
implementation that can severely limit effective transfer speeds—especially
across higher-latency networks or when handling large files.
----
h2. *Summary of the Problem*
File uploads via Guacamole’s SFTP subsystem currently perform {*}far below
expected throughput levels{*}, especially when transferring large files to SSH
targets located on different networks.
h3. *Observed behavior*
* When uploading a large file to an SSH target located across a different
network segment (moderate latency), upload performance degrades dramatically.
* In real testing, *each 4 MB chunk required ~1.1 minutes* to complete.
* This results in multi-hundred-MB uploads taking *tens of minutes* instead of
seconds.
!image-2025-12-05-13-50-56-071.png!
!image-2025-12-05-13-51-03-874.png!
h3. *Underlying cause*
These symptoms match known limitations in libssh2’s SFTP implementation,
including:
* *Inefficient or absent request pipelining*
(single outstanding request → multiple round trips)
* High sensitivity to network latency due to per-request round-trip delays
As a result, Guacamole’s SFTP performance falls well below that of other SFTP
clients that optimize their SFTP pipelines (e.g., OpenSSH’s {{{}sftp{}}}).
----
h2. *Proof of Concept / Prototype Fix*
A working proof-of-concept implementation demonstrating substantial throughput
improvement is available here:
🔗 [https://github.com/alailsonko/guacamole-server/pull/1]
h3. *Changes introduced in the prototype*
The prototype improves SFTP upload throughput by:
* Increasing libssh2 SFTP write buffer sizes
* Allowing multiple outstanding SFTP write requests (pipelined writes)
* Reducing the number of round trips required for each data chunk
* Optimizing batching behavior during sequential writes
h3. *Results*
On the exact same environment and SSH target:
* Current implementation:
*4 MB upload takes ~1.1 minutes*
* Prototype implementation:
*4 MB upload takes ~10 seconds*
This represents a {*}6× improvement{*}, with room for further optimization. It
was validated with verification of *checksum* of the file transferred.
!image-2025-12-05-13-51-54-825.png!
!image-2025-12-05-13-52-01-566.png!
----
h2. *Expected Behavior*
Guacamole’s SFTP upload performance should scale in line with:
* Available network bandwidth
* Server-side SSH capabilities
* Optimized libssh2 SFTP behavior (buffered + pipelined writes)
Users should experience upload speeds comparable to standard tools such as:
* OpenSSH {{sftp}}
* {{scp}}
* libssh2-tuned SFTP clients
----
h2. *Actual Behavior*
SFTP uploads through Guacamole are:
* *Significantly slower* than standard OpenSSH SFTP transfers
* Disproportionately impacted by network latency
Large file transfers may become effectively unusable.
----
h2. *Impact*
* Users experience extremely slow upload speeds when interacting with SSH/SFTP
targets through Guacamole.
* Large file transfers (10 MB+) become impractical, especially across WAN
links.
* The performance gap between Guacamole and native SSH clients reduces user
confidence and usability.
* Administrators often assume a network issue when the root cause is the
internal SFTP pipeline implementation.
> Improve SFTP upload performance
> -------------------------------
>
> Key: GUACAMOLE-2180
> URL: https://issues.apache.org/jira/browse/GUACAMOLE-2180
> Project: Guacamole
> Issue Type: Improvement
> Components: SSH
> Affects Versions: 1.6.0
> Environment: The test environment consists of:
> - Guacamole environment
> - An Ubuntu virtual machine running in a cloud provider
> - The machine hosts a Docker Compose setup containing:
> - postgres (Guacamole database)
> - guacamole-server
> - guacamole-client
> - SSH Target
> - The actual SSH server is located on a user’s private LAN (on-prem)
> - It is not directly reachable from the cloud VM
> Connectivity setup
> The user initiates an SSH remote port forward from their local network to the
> cloud VM
> (e.g., ssh -R <cloud-vm-port>:localhost:22 cloud-vm)
> As a result, the cloud VM can access the on-prem SSH target through the
> forwarded port
> Guacamole uses that forwarded port as its SFTP/SSH target endpoint
> Reporter: Alailson Andrade do Nascimento
> Priority: Major
> Labels: performance, sftp, ssh
> Fix For: 1.6.1
>
> Attachments: image-2025-12-05-12-46-35-184.png,
> image-2025-12-05-13-43-06-876.png, image-2025-12-05-13-44-19-318.png,
> image-2025-12-05-13-45-49-566.png, image-2025-12-05-13-46-10-386.png,
> image-2025-12-05-13-50-56-071.png, image-2025-12-05-13-51-03-874.png,
> image-2025-12-05-13-51-54-825.png, image-2025-12-05-13-52-01-566.png
>
> Original Estimate: 40h
> Remaining Estimate: 40h
>
> h2. *Overview*
> SFTP upload performance in Apache Guacamole is significantly slower than
> expected under typical usage conditions.
> Guacamole’s SSH/SFTP subsystem currently relies on *libssh2* for SFTP
> operations. While widely used, libssh2 has {*}known throughput limitations{*}.
> Relevant upstream references:
> * libssh2 original performance issue:
> [https://github.com/libssh2/libssh2/issues/90|https://github.com/libssh2/libssh2/issues/90?utm_source=chatgpt.com]
> * Recent community discussion on SFTP throughput constraints:
> [https://github.com/libssh2/libssh2/discussions/1064]
> These discussions highlight structural constraints in libssh2’s SFTP
> implementation that can severely limit effective transfer speeds—especially
> across higher-latency networks or when handling large files.
> ----
> h2. *Summary of the Problem*
> File uploads via Guacamole’s SFTP subsystem currently perform {*}far below
> expected throughput levels{*}, especially when transferring large files to
> SSH targets located on different networks.
> h3. *Observed behavior*
> * When uploading a large file to an SSH target located across a different
> network segment (moderate latency), upload performance degrades dramatically.
> * In real testing, *each 4 MB chunk required ~1.1 minutes* to complete.
> * This results in multi-hundred-MB uploads taking *tens of minutes* instead
> of seconds.
> !image-2025-12-05-13-50-56-071.png!
> !image-2025-12-05-13-51-03-874.png!
> h3. *Underlying cause*
> These symptoms match known limitations in libssh2’s SFTP implementation,
> including:
> * *Inefficient or absent request pipelining*
> (single outstanding request → multiple round trips)
> * High sensitivity to network latency due to per-request round-trip delays
> As a result, Guacamole’s SFTP performance falls well below that of other SFTP
> clients that optimize their SFTP pipelines (e.g., OpenSSH’s {{{}sftp{}}}).
> ----
> h2. *Proof of Concept / Prototype Fix*
> A working proof-of-concept implementation demonstrating substantial
> throughput improvement is available here:
> 🔗 [https://github.com/alailsonko/guacamole-server/pull/1]
> h3. *Changes introduced in the prototype*
> The prototype improves SFTP upload throughput by:
> * Increasing libssh2 SFTP write buffer sizes
> * Allowing multiple outstanding SFTP write requests (pipelined writes)
> * Reducing the number of round trips required for each data chunk
> * Optimizing batching behavior during sequential writes
> h3. *Results*
> On the exact same environment and SSH target:
> * Current implementation:
> *4 MB upload takes ~1.1 minutes*
> * Prototype implementation:
> *4 MB upload takes ~10 seconds*
> This represents a {*}6× improvement{*}, with room for further optimization.
> It was validated with verification of *checksum* of the file transferred.
> !image-2025-12-05-13-51-54-825.png!
> !image-2025-12-05-13-52-01-566.png!
> ----
> h2. *Expected Behavior*
> Guacamole’s SFTP upload performance should scale in line with:
> * Available network bandwidth
> * Server-side SSH capabilities
> * Optimized libssh2 SFTP behavior (buffered + pipelined writes)
> Users should experience upload speeds comparable to standard tools such as:
> * OpenSSH {{sftp}}
> * {{scp}}
> * libssh2-tuned SFTP clients
> ----
> h2. *Actual Behavior*
> SFTP uploads through Guacamole are:
> * *Significantly slower* than standard OpenSSH SFTP transfers
> * Disproportionately impacted by network latency
> Large file transfers may become effectively unusable.
> ----
> h2. *Impact*
> * Users experience extremely slow upload speeds when interacting with
> SSH/SFTP targets through Guacamole.
> * Large file transfers (10 MB+) become impractical, especially across WAN
> links.
> * The performance gap between Guacamole and native SSH clients reduces user
> confidence and usability.
> * Administrators often assume a network issue when the root cause is the
> internal SFTP pipeline implementation.
> Thanks, the point is to contribute to the project
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
