** Description changed:
[ Impact ]
Creating a TERMINATED_HTTPS listener in an amphora with >=32GB of memory
- triggers the OOM-killer during listener startup (and any subsequent
- `systemctl reload` of haproxy in the amphora).
+ triggers the OOM-killer during pool setup (and any subsequent `systemctl
+ reload` of haproxy in the amphora).
```
os loadbalancer listener create --name thttps_xxlarge --protocol
TERMINATED_HTTPS --protocol-port 443 --default-tls-container-ref <URL> --wait
xxlarge1
```
This was originally reported in a Caracal cloud using an Ubuntu 22.04
Amphora image.
I've been able to reproduce this reliably in my lab using the latest
devstack and an Ubuntu 24.04 Amphora image.
Workaround by setting a higher connection limit on one listener in
proportion to the 50000 default and the memory on the system. So for an
amphora with 32GiB of RAM, use --connection-limit 200000 for one
listener.
[ Test Plan ]
Deploy charmed Jammy/Caracal with Octavia and Barbican. For Noble
verification, DRU the Octavia unit to Noble before completing the rest
of the test plan.
+
+ From the machine hosting Octavia, create an ssh key and upload the public key
to Openstack as `amphora-backdoor` (for shell access to the amphora):
+ $ os keypair create --public path/to/key amphora-backdoor
Get a compute flavor id to use for the Octavia flavorprofile:
$ openstack flavor create --ram 32768 --disk 10 --vcpus 4 --id 10 m1.xxlarge
+----------------------------+------------+
| Field | Value |
+----------------------------+------------+
| OS-FLV-DISABLED:disabled | False |
| OS-FLV-EXT-DATA:ephemeral | 0 |
| description | None |
| disk | 10 |
| id | 10 |
| name | m1.xxlarge |
| os-flavor-access:is_public | True |
| properties | |
| ram | 32768 |
| rxtx_factor | 1.0 |
| swap | 0 |
| vcpus | 4 |
+----------------------------+------------+
Create the flavor profile:
$ openstack loadbalancer flavorprofile create \
--name o1.xxlarge \
--provider amphora \
--flavor-data '{"compute_flavor": "10"}'
+---------------+--------------------------------------+
| Field | Value |
+---------------+--------------------------------------+
| id | f3aac848-7e77-449a-96af-bf0312c45ef9 |
| name | o1.xxlarge |
| provider_name | amphora |
| flavor_data | {"compute_flavor": "10"} |
+---------------+--------------------------------------+
And Amphora flavor:
$ openstack loadbalancer flavor create \
--name o1.xxlarge \
--flavorprofile o1.xxlarge \
--description "Extra large LB" \
--enable
+-------------------+--------------------------------------+
| Field | Value |
+-------------------+--------------------------------------+
| id | 350f607c-9189-49ac-a585-84960322137a |
| name | o1.xxlarge |
| flavor_profile_id | f3aac848-7e77-449a-96af-bf0312c45ef9 |
| enabled | True |
| description | Extra large LB |
+-------------------+--------------------------------------+
For certificates I created a complete PKI following Jamie's Guide [1];
could use a self-signed cert. Attaching xxlarge.devstack.p12 for
convenience (as this test plan does not require the cert to function).
$ openssl pkcs12 -export \
-inkey xxlarge.devstack.key.pem \
-in xxlarge.devstack.cert.pem \
-certfile ca-chain.cert.pem \
-passout pass: -out xxlarge.devstack.p12
$ openstack secret store \
--name="xxlarge.devstack.p12" \
-t 'application/octet-stream' \
-e 'base64' \
--payload="$(base64 < xxlarge.devstack.p12)"
+---------------+-------------------------------------------------------------------+
| Field | Value
|
+---------------+-------------------------------------------------------------------+
| Secret href |
https://None:9312/v1/secrets/86f4113d-e9f6-43e3-aa10-459a6acc14b3 |
| Name | xxlarge.devstack.p12
|
| Created | None
|
| Status | None
|
| Content types | {'default': 'application/octet-stream'}
|
| Algorithm | aes
|
| Bit length | 256
|
| Secret type | opaque
|
| Mode | cbc
|
| Expiration | None
|
+---------------+-------------------------------------------------------------------+
Create an LB (provisions the Amphora):
$ openstack loadbalancer create --wait \
--name lb1-xxlarge \
--vip-subnet-id ext_net_subnet \
--flavor o1.xxlarge
+ +---------------------+--------------------------------------+
+ | Field | Value |
+ +---------------------+--------------------------------------+
+ | admin_state_up | True |
+ | availability_zone | None |
+ | created_at | 2026-02-02T22:16:05 |
+ | description | |
+ | flavor_id | 350f607c-9189-49ac-a585-84960322137a |
+ | id | 19d431d8-3ce7-4942-af7f-8fca6bd8e88c |
+ | listeners | |
+ | name | lb1-xxlarge |
+ | operating_status | ONLINE |
+ | pools | |
+ | project_id | 3ac0c8861bee423b9983a57f73bed0da |
+ | provider | amphora |
+ | provisioning_status | ACTIVE |
+ | updated_at | 2026-02-02T22:17:03 |
+ | vip_address | 10.149.109.90 |
+ | vip_network_id | adaf097d-c89a-4df4-b97a-4a09105c0c75 |
+ | vip_port_id | df4bd66e-a61b-4da4-810f-9e98112cb081 |
+ | vip_qos_policy_id | None |
+ | vip_subnet_id | 3a1853de-5caa-46ce-a5eb-7e2ca70898d3 |
+ | vip_vnic_type | normal |
+ | tags | |
+ | additional_vips | [] |
+ +---------------------+--------------------------------------+
+
+ In my test environment (using nested virt) the allocation of the haproxy
+ TLS session cache takes so long that the systemd unit times out. To
+ avoid this, ssh into the amphora and set DefaultTimeoutStartSec=infinity
+ and DefaultTimeoutStopSec=infinity in /etc/systemd/system.conf:
+
+ Get the amphora's management IP (using your `service` project's UUID):
+
+ $ openstack server list --project 0086b84a8dcf44b5b425abbddbbbd8c3
+
+--------------------------------------+--------------------------------------------+--------+--------------------------------------------+----------------------------------------------+------------+
+ | ID | Name
| Status | Networks | Image
| Flavor |
+
+--------------------------------------+--------------------------------------------+--------+--------------------------------------------+----------------------------------------------+------------+
+ | 7d00e939-41e4-4017-9757-b7656961b114 | amphora-f9436cbb-
| ACTIVE | ext_net=10.149.109.99; lb-mgmt-net=fc00:8c |
amphora-haproxy-x86_64-ubuntu-22.04-20250702 | m1.xxlarge |
+ | | bfa8-41e0-b1fc-a4bd7855bf3d
| | 8e:8682:b866:f816:3eff:fe15:4398 |
| |
+
+--------------------------------------+--------------------------------------------+--------+--------------------------------------------+----------------------------------------------+------------+
+
+ Log in to the machine hosting the Octavia API services, SSH to the
+ amphora IP, modify the systemd config and reload it:
+
+ $ ssh -i .ssh/id_amphora ubuntu@fc00:8c8e:8682:b866:f816:3eff:fe15:4398
+ $ sudo sed -i
's/#DefaultTimeoutStartSec=90s/DefaultTimeoutStartSec=infinity/'
/etc/systemd/system.conf
+ $ sudo sed -i 's/#DefaultTimeoutStopSec=90s/DefaultTimeoutStopSec=infinity/'
/etc/systemd/system.conf
+ $ sudo systemctl daemon-reload
Create the listener (this configures haproxy in the Amphora):
$ openstack loadbalancer listener create --wait \
--name thttps \
--protocol TERMINATED_HTTPS \
--protocol-port 443 \
--default-tls-container-ref
https://None:9312/v1/secrets/86f4113d-e9f6-43e3-aa10-459a6acc14b3 \
lb1-xxlarge
+
+-----------------------------+-----------------------------------------------------------------------------+
+ | Field | Value
|
+
+-----------------------------+-----------------------------------------------------------------------------+
+ | admin_state_up | True
|
+ | connection_limit | -1
|
+ | created_at | 2026-02-02T22:22:59
|
+ | default_pool_id | None
|
+ | default_tls_container_ref |
https://None:9312/v1/secrets/86f4113d-e9f6-43e3-aa10-459a6acc14b3 |
+ | description |
|
+ | id | 7799303a-fb43-4e2f-b3f8-6f6e187f2002
|
+ | insert_headers | None
|
+ | l7policies |
|
+ | loadbalancers | 19d431d8-3ce7-4942-af7f-8fca6bd8e88c
|
+ | name | thttps
|
+ | operating_status | OFFLINE
|
+ | project_id | 3ac0c8861bee423b9983a57f73bed0da
|
+ | protocol | TERMINATED_HTTPS
|
+ | protocol_port | 443
|
+ | provisioning_status | ACTIVE
|
+ | sni_container_refs | []
|
+ | timeout_client_data | 50000
|
+ | timeout_member_connect | 5000
|
+ | timeout_member_data | 50000
|
+ | timeout_tcp_inspect | 0
|
+ | updated_at | 2026-02-02T22:24:03
|
+ | client_ca_tls_container_ref | None
|
+ | client_authentication | NONE
|
+ | client_crl_container_ref | None
|
+ | allowed_cidrs | None
|
+ | tls_ciphers |
TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256: |
+ | |
DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM- |
+ | |
SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-SHA256:DHE-RSA-AES128- |
+ | |
SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256 |
+ | tls_versions | ['TLSv1.2', 'TLSv1.3']
|
+ | alpn_protocols | ['h2', 'http/1.1', 'http/1.0']
|
+ | tags |
|
+ | hsts_max_age | None
|
+ | hsts_include_subdomains | False
|
+ | hsts_preload | False
|
+
+-----------------------------+-----------------------------------------------------------------------------+
+
+ And create a pool (triggers haproxy restart in the amphora):
+ $ openstack loadbalancer pool create \
+ --name xxlarge_p1 \
+ --lb-algorithm ROUND_ROBIN \
+ --listener thttps \
+ --protocol HTTP \
+ --wait
+ +----------------------+--------------------------------------+
+ | Field | Value |
+ +----------------------+--------------------------------------+
+ | admin_state_up | True |
+ | created_at | 2026-02-02T22:34:15 |
+ | description | |
+ | healthmonitor_id | |
+ | id | 6a76ea3a-c4eb-4f1f-9277-b3bf76247462 |
+ | lb_algorithm | ROUND_ROBIN |
+ | listeners | 7799303a-fb43-4e2f-b3f8-6f6e187f2002 |
+ | loadbalancers | 19d431d8-3ce7-4942-af7f-8fca6bd8e88c |
+ | members | |
+ | name | xxlarge_p1 |
+ | operating_status | OFFLINE |
+ | project_id | 3ac0c8861bee423b9983a57f73bed0da |
+ | protocol | HTTP |
+ | provisioning_status | ACTIVE |
+ | session_persistence | None |
+ | updated_at | 2026-02-02T22:35:02 |
+ | tls_container_ref | None |
+ | ca_tls_container_ref | None |
+ | crl_container_ref | None |
+ | tls_enabled | False |
+ | tls_ciphers | None |
+ | tls_versions | None |
+ | tags | |
+ | alpn_protocols | None |
+ +----------------------+--------------------------------------+
Expected behavior:
- Success
+ Checking the journal from the amphora's shell should show no OOM
Actual behavior:
- The resource did not successfully reach ACTIVE status. (HTTP n/a)
- (Request-ID: None)
-
- [1] https://jamielinux.com/docs/openssl-certificate-
- authority/introduction.html
+ Observe 1 or more OOM-killer invocations in the amphora. The process
+ list is likely to contain two haproxy processes with slightly more than
+ 16GB in total_vm:
+
+ Feb 02 22:34:24 amphora-f9436cbb-bfa8-41e0-b1fc-a4bd7855bf3d kernel: [
2055] 65534 2055 4739383 2824 126976 0 0 haproxy
+ Feb 02 22:34:24 amphora-f9436cbb-bfa8-41e0-b1fc-a4bd7855bf3d kernel: [
2074] 0 2074 4687328 3373674 27123712 0 0 haproxy
+
+ [1]
https://jamielinux.com/docs/openssl-certificate-authority/introduction.html
+ [2]
https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#TimeoutSec=
[ Where problems could occur ]
The patch affects only the code that produces the `ssl.tune.cachesize`
haproxy configuration option. `ssl.tune.cachesize` is only included in
the haproxy configuration for listeners with protocol TERMINATED_HTTPS.
If the change is wrong, we'd expect to see failures related memory
consumption in Amphora that host TERMINATED_HTTPS listeners; most likely
these would be OOM-killer invocations visible on the system log, but
issues could also manifest as haproxy failing as a result of invalid
configuration.
Under very specific circumstances (recurring SSL connections balanced
well with the existing `ssl.tune.cachesize` on a TERMINATED_HTTPS
loadbalancer) it's also possible that this change could moderately
regress performance. A performance regression of this nature can be
prevented by provisioning an Amphora with more available memory.
-
[ Root Cause ]
454cff5 (in Zed+ IIUC) introduces the use of haproxy's
`tune.ssl.cachesize` for TERMINATED_HTTPS listeners [1][2].
The commit does not make clear that during a reload of haproxy
(SIGUSR2), the old worker process stays running until the new worker
process is ready [3][4]. This means that two TLS session caches are
allocated/held simultaneously during a reload of the service [5].
For small Amphorae, this works fine. The default connection limit is
50000, which takes enough of a chunk out of the 50% allocation that
there is enough wiggle room for the new haproxy worker to allocate its
cache and coexist with the old worker for some time.
However, as the available memory in the system increases, the memory
consumed by the session cache approaches 50%, and increases the worker's
memory usage beyond 50% (as something else in the worker is also using
memory in proportion to the configured cachesize).
I tested 10 values of tune.ssl.cachesize in an amphora with 32GiB of
RAM, reloading the haproxy service each time:
- vsz here is the value reported by `ps -ax -o pid,vsz,rss,uss,pmem,args |
grep haproxy`
- overhead is `tune.ssl.cachesize_MiB - vsz_MiB - 261`
- overhead% is `floor((overhead / tune.ssl.cachesize_MiB) * 100)`
tune.ssl.cachesize | tune.ssl.cachesize_MiB | vsz | vsz_MiB | overhead
| overhead%
0 | 0 | 267416 | 261 | 0
| 0%
7741606 | 1476 | 2142472 | 2092 | 355
| 24%
15483212 | 2953 | 4017260 | 3923 | 709
| 24%
23224818 | 4429 | 5892180 | 5754 | 1064
| 24%
30966424 | 5906 | 7767100 | 7585 | 1418
| 24%
38708030 | 7382 | 9642020 | 9416 | 1773
| 24%
46449636 | 8859 | 11516940 | 11247 | 2127
| 24%
54191242 | 10336 | 13391860 | 13077 | 2480
| 23%
61932848 | 11812 | 15266780 | 14908 | 2835
| 24%
69674454 | 13289 | 17141700 | 16739 | 3189
| 23%
77416060 | 14765 | 19016744 | 18571 | 3545
| 24%
Note that this listener was not configured with a pool, so there was no
load on the system when I gathered this data.
As shown, haproxy to consumes additional memory proportional to the size
of the TLS session cache. The allocation for the cache occurs at [6],
referring to [7].
I verified the documentation's assertion that tune.ssl.cachesize is 200
bytes on amd64; sizeof(struct shared_block) is 48 bytes on the same
hardware [8].
Octavia should allocate closer to 1/3 than 1/2 for the TLS session
cache. I'll test and propose a patch against master shortly.
[1]
https://opendev.org/openstack/octavia/commit/454cff587ed10b5e504da93b074b77cb85055b13
[2]
https://www.haproxy.com/documentation/haproxy-configuration-manual/new/2-8r1/#section-3.2.-tunesslcachesize
[3] https://github.com/haproxy/haproxy/issues/217#issuecomment-544515990
[4] https://manpages.ubuntu.com/manpages/jammy/en/man1/haproxy.1.html
[5]
https://opendev.org/openstack/octavia/src/branch/master/octavia/amphorae/backends/agent/api_server/templates/systemd.conf.j2
[6]
https://git.launchpad.net/ubuntu/+source/haproxy/tree/src/ssl_sock.c?h=applied/ubuntu/noble-devel#n5346
[7]
https://git.launchpad.net/ubuntu/+source/haproxy/tree/src/shctx.c?h=applied/ubuntu/noble-devel#n300
[8]
https://git.launchpad.net/ubuntu/+source/haproxy/tree/include/haproxy/shctx-t.h?h=applied/ubuntu/noble-devel#n38
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2119987
Title:
haproxy reload triggers OOM-killer for TERMINATED_HTTPS loadbalancers
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2119987/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
