All VPN servers have been rolled back to 4.4
Additional log collection is not possible.
Setting status to confirmed.
** Changed in: linux (Ubuntu)
Status: Incomplete => Confirmed
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1853197
Title:
Memory leak in net/xfrm/xfrm_state.c - 8 pages per ipsec connection
Status in linux package in Ubuntu:
Confirmed
Bug description:
Ubuntu linux distro, 4.15.0-62 kernel, server platform.
This OS is used as an IPSec VPN gateway. It serves up to several hundred
concurrent connections
In an attempt to upgrade from the 4.4 kernel to 4.15, the team noticed
that VPN gateway VMs were running out of physical memory after 12-48
hours, depending on load.
Attachments from a server machine in this state in attached leakinfo.txt
output of free -t
output of /proc/meminfo in out of memory condition
output of /slabtop -o -sc
/sys/kernel/debug/page_owner sorted and aggregated after server ran for 12
hrs and ran out of memory
Patches for 4.15 and 5.4
Highlight from page_owner, we can see the leak is a buffer associated
with the ipsec impelementation. Each connection leaks 32k of memory
via alloc_page with order=3
Page allocated via order 3, mask
0x1085220(GFP_ATOMIC|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP)
get_page_from_freelist+0xd64/0x1250
__alloc_pages_nodemask+0x11c/0x2e0
alloc_pages_current+0x6a/0xe0
skb_page_frag_refill+0x71/0x100
esp_output_head+0x265/0x3e0 [esp4]
esp_output+0xbc/0x180 [esp4]
xfrm_output_resume+0x179/0x530
xfrm_output+0x8e/0x230
xfrm4_output_finish+0x2b/0x30
__xfrm4_output+0x3a/0x50
xfrm4_output+0x43/0xc0
ip_forward_finish+0x51/0x80
ip_forward+0x38a/0x480
ip_rcv_finish+0x122/0x410
ip_rcv+0x292/0x360
__netif_receive_skb_core+0x815/0xbd0
Patch to fix this issue in 4.15 (tested and verified on same server
exhibiting above leak):
diff --git a/net/xfrm/xfrm_state.c b/net/xfrm/xfrm_state.c
index 728272f..7842f83 100644
--- a/net/xfrm/xfrm_state.c
+++ b/net/xfrm/xfrm_state.c
@@ -451,6 +451,10 @@ static void xfrm_state_gc_destroy(struct xfrm_state *x)
}
xfrm_dev_state_free(x);
security_xfrm_state_free(x);
+
+ if(x->xfrag.page)
+ put_page(x->xfrag.page);
+
kfree(x);
}
Patch for master branch (5.4 I believe) from Paul Wouters (p...@nohats.ca)
diff --git a/net/xfrm/xfrm_state.c b/net/xfrm/xfrm_state.c
index c6f3c4a1bd99..f3423562d933 100644
--- a/net/xfrm/xfrm_state.c
+++ b/net/xfrm/xfrm_state.c
@@ -495,6 +495,8 @@ static void ___xfrm_state_destroy(struct xfrm_state *x)
x->type->destructor(x);
xfrm_put_type(x->type);
}
+ if (x->xfrag.page)
+ put_page(x->xfrag.page);
xfrm_dev_state_free(x);
security_xfrm_state_free(x);
xfrm_state_free(x);
Severity: Critical - we are unable to use any kernel later than 4.11,
and are sticking with 4.4 in production.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1853197/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
