Package: lxc
Version: 1:2.0.7-2
Severity: important
Tags: patch
Dear Maintainer,
I am using LXC on stretch, running a number of stretch containers.
When running around 15 containers, systemd starts to fail, both on the host
and inside containers:
# journalctl -f
Failed to get journal fd: Too many open files
# systemctl stop nginx
Failed to allocate directory watch: Too many open files
The problem seems to be that systemd uses inotify quite a lot, and the limit
on inotify listeners is very low by default:
# sysctl fs.inotify.max_user_instances
fs.inotify.max_user_instances = 128
Increasing this value indeed solves the problem. The issue was described
in this blog post:
https://kdecherf.com/blog/2015/09/12/systemd-and-the-fd-exhaustion/
Attached is a patch for lxc that configures this sysctl setting to a higher
value. I don't know if this is the "Debian way" or if some other solution
is preferred.
-- System Information:
Debian Release: 9.0
APT prefers testing
APT policy: (500, 'testing')
Architecture: amd64
(x86_64)
Kernel: Linux 4.9.0-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
Versions of packages lxc depends on:
ii init-system-helpers 1.47
ii libapparmor1 2.11.0-3
ii libc6 2.24-9
ii libcap2 1:2.25-1
ii libgnutls30 3.5.8-5
ii liblxc1 1:2.0.7-2
ii libseccomp2 2.3.1-2.1
ii libselinux1 2.6-3+b1
ii lsb-base 9.20161125
ii python3-lxc 1:2.0.7-2
pn python3:any <none>
Versions of packages lxc recommends:
pn bridge-utils <none>
ii debootstrap 1.0.89
ii dirmngr 2.1.18-6
pn dnsmasq-base <none>
ii gnupg 2.1.18-6
ii iptables 1.6.0+snapshot20161117-6
pn libpam-cgfs <none>
pn lxcfs <none>
ii openssl 1.1.0e-1
ii rsync 3.1.2-1
pn uidmap <none>
Versions of packages lxc suggests:
pn apparmor <none>
pn btrfs-tools <none>
ii lvm2 2.02.168-2
-- no debconf information
>From 023651197841198cd10e797451ae591ec523ff45 Mon Sep 17 00:00:00 2001
From: Baptiste Jonglez <g...@bitsofnetworks.org>
Date: Sat, 22 Apr 2017 23:51:22 +0200
Subject: [PATCH] Increase the maximum number of inotify listeners
This avoid issues with systemd containers, where as little as 15 LXC
containers can trigger "Too many open files" errors in systemd.
---
debian/40-lxc-inotify.conf | 8 ++++++++
debian/rules | 4 ++++
2 files changed, 12 insertions(+)
create mode 100644 debian/40-lxc-inotify.conf
diff --git a/debian/40-lxc-inotify.conf b/debian/40-lxc-inotify.conf
new file mode 100644
index 0000000..c16dd5f
--- /dev/null
+++ b/debian/40-lxc-inotify.conf
@@ -0,0 +1,8 @@
+# Defines the maximum number of inotify listeners.
+# By default, this value is 128, which is quickly exhausted when using
+# systemd-based LXC containers (15 containers are enough).
+# When the limit is reached, systemd becomes mostly unusable, throwing
+# "Too many open files" all around (both on the host and in containers).
+# See https://kdecherf.com/blog/2015/09/12/systemd-and-the-fd-exhaustion/
+
+fs.inotify.max_user_instances=4096
diff --git a/debian/rules b/debian/rules
index 7d334ad..8443d0f 100755
--- a/debian/rules
+++ b/debian/rules
@@ -48,6 +48,10 @@ override_dh_auto_install:
rm -f debian/tmp/usr/share/man/*/lxc-top.*
rm -f debian/tmp/usr/share/man/*/*/lxc-top.*
+ # increase limit of inotify listeners
+ mkdir -p debian/tmp/etc/sysctl.d
+ cp debian/40-lxc-inotify.conf debian/tmp/etc/sysctl.d/
+
dh_apparmor -p lxc --profile-name=usr.bin.lxc-start
override_dh_compress:
--
2.12.2
