Re: [lxc-users] lxd process using lots of CPU

[Stéphane Graber]

On Thu, Mar 09, 2017 at 11:01:34PM +0900, Tomasz Chmielewski wrote:
> On a server with several ~idlish containers:
> 
> 
>   PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
> 19104 root       20   0 2548M 44132 15236 S 140.  0.0 58h03:17 /usr/bin/lxd
> --group lxd --logfile=/var/log/lxd/lxd.log
> 24966 root       20   0 2548M 44132 15236 S 18.2  0.0  2h45:36 /usr/bin/lxd
> --group lxd --logfile=/var/log/lxd/lxd.log
> 19162 root       20   0 2548M 44132 15236 S 17.5  0.0  3h31:49 /usr/bin/lxd
> --group lxd --logfile=/var/log/lxd/lxd.log
> 19120 root       20   0 2548M 44132 15236 S 16.2  0.0  3h16:11 /usr/bin/lxd
> --group lxd --logfile=/var/log/lxd/lxd.log
> 19244 root       20   0 2548M 44132 15236 S 11.0  0.0  1h48:56 /usr/bin/lxd
> --group lxd --logfile=/var/log/lxd/lxd.log
> 19123 root       20   0 2548M 44132 15236 S 11.0  0.0  3h34:42 /usr/bin/lxd
> --group lxd --logfile=/var/log/lxd/lxd.log
> 19243 root       20   0 2548M 44132 15236 S 10.4  0.0  1h06:13 /usr/bin/lxd
> --group lxd --logfile=/var/log/lxd/lxd.log
> 14962 root       20   0 2548M 44132 15236 R 10.4  0.0  3h17:27 /usr/bin/lxd
> --group lxd --logfile=/var/log/lxd/lxd.log
> 19356 root       20   0 2548M 44132 15236 S  9.7  0.0  2h16:44 /usr/bin/lxd
> --group lxd --logfile=/var/log/lxd/lxd.log
> 19161 root       20   0 2548M 44132 15236 R  9.7  0.0  1h26:40 /usr/bin/lxd
> --group lxd --logfile=/var/log/lxd/lxd.log
> 19126 root       20   0 2548M 44132 15236 R  9.1  0.0 22:11.20 /usr/bin/lxd
> --group lxd --logfile=/var/log/lxd/lxd.log
> 19115 root       20   0 2548M 44132 15236 R  8.4  0.0  2h55:21 /usr/bin/lxd
> --group lxd --logfile=/var/log/lxd/lxd.log
>   693 root       20   0 2548M 44132 15236 R  8.4  0.0  2h28:02 /usr/bin/lxd
> --group lxd --logfile=/var/log/lxd/lxd.log
> 
> 
> That's actually one lxd process with many threads; view from htop.
> 
> Expected?
> 
> ii  liblxc1                            2.0.7-0ubuntu1~16.04.1
> amd64        Linux Containers userspace tools (library)
> ii  lxc-common                         2.0.7-0ubuntu1~16.04.1
> amd64        Linux Containers userspace tools (common tools)
> ii  lxcfs                              2.0.6-0ubuntu1~16.04.1
> amd64        FUSE based filesystem for LXC
> ii  lxd                                2.0.9-0ubuntu1~16.04.2
> amd64        Container hypervisor based on LXC - daemon
> ii  lxd-client                         2.0.9-0ubuntu1~16.04.2
> amd64        Container hypervisor based on LXC - client
> 
> 
> 
> strace of the process mainly shows:
> 
> [pid 19124] poll([{fd=28, events=POLLIN|POLLPRI|POLLERR|POLLHUP|0x2000}], 1,
> -1 <unfinished ...>
> [pid 19120] poll([{fd=13, events=POLLIN|POLLPRI|POLLERR|POLLHUP|0x2000}], 1,
> -1 <unfinished ...>
> [pid 19124] <... poll resumed> )        = 1 ([{fd=28, revents=POLLNVAL}])
> [pid 19120] <... poll resumed> )        = 1 ([{fd=13, revents=POLLNVAL}])
> [pid 19124] poll([{fd=28, events=POLLIN|POLLPRI|POLLERR|POLLHUP|0x2000}], 1,
> -1 <unfinished ...>
> [pid 19120] poll([{fd=13, events=POLLIN|POLLPRI|POLLERR|POLLHUP|0x2000}], 1,
> -1 <unfinished ...>
> [pid 19124] <... poll resumed> )        = 1 ([{fd=28, revents=POLLNVAL}])
> [pid 19120] <... poll resumed> )        = 1 ([{fd=13, revents=POLLNVAL}])
> [pid 19124] poll([{fd=28, events=POLLIN|POLLPRI|POLLERR|POLLHUP|0x2000}], 1,
> -1 <unfinished ...>
> [pid 19120] poll([{fd=13, events=POLLIN|POLLPRI|POLLERR|POLLHUP|0x2000}], 1,
> -1 <unfinished ...>
> [pid 19124] <... poll resumed> )        = 1 ([{fd=28, revents=POLLNVAL}])
> [pid 19120] <... poll resumed> )        = 1 ([{fd=13, revents=POLLNVAL}])
> [pid 19124] poll([{fd=28, events=POLLIN|POLLPRI|POLLERR|POLLHUP|0x2000}], 1,
> -1 <unfinished ...>
> [pid 19120] poll([{fd=13, events=POLLIN|POLLPRI|POLLERR|POLLHUP|0x2000}], 1,
> -1 <unfinished ...>
> [pid 19124] <... poll resumed> )        = 1 ([{fd=28, revents=POLLNVAL}])
> [pid 19120] <... poll resumed> )        = 1 ([{fd=13, revents=POLLNVAL}])
> [pid 19124] poll([{fd=28, events=POLLIN|POLLPRI|POLLERR|POLLHUP|0x2000}], 1,
> -1 <unfinished ...>
> [pid 19120] poll([{fd=13, events=POLLIN|POLLPRI|POLLERR|POLLHUP|0x2000}], 1,
> -1 <unfinished ...>
> [pid 19124] <... poll resumed> )        = 1 ([{fd=28, revents=POLLNVAL}])
> [pid 19120] <... poll resumed> )        = 1 ([{fd=13, revents=POLLNVAL}])
> [pid 19124] poll([{fd=28, events=POLLIN|POLLPRI|POLLERR|POLLHUP|0x2000}], 1,
> -1 <unfinished ...>
> [pid 19120] poll([{fd=13, events=POLLIN|POLLPRI|POLLERR|POLLHUP|0x2000}], 1,
> -1 <unfinished ...>
> 
> 
> Tomasz Chmielewski
> https://lxadm.com

Do you see a flood of events if you run "lxc monitor --type=logging"?

-- 
Stéphane Graber
Ubuntu developer
http://www.ubuntu.com

signature.asc
Description: PGP signature

_______________________________________________
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users