I just updated the status to Incomplete for both bug tasks, as I may have been too confident in the original bug description given to me by the customer.
The logrotate script as written is not meant to "restart" rsyslogd. Looking at the man page, the HUP signal (as sent by the /usr/lib/rsyslog /rsyslog-rotate script on postrotate) is meant to trigger rsyslogd to close all open files. My testing shows that this does seem to be working OK on both 16.04 classic and Core 16. I suspect that the problem may be due the way the customer is integrating with rsyslogd, and that this in turn might be why rsyslogd is holding on to the deleted files. I'm going to leave this bug open until we can determine the actual root cause, because as mentioned in the bug description, this can lead to a device running out of storage. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to rsyslog in Ubuntu. https://bugs.launchpad.net/bugs/1875493 Title: [core] log rotation doesn't properly restart rsyslogd Status in Snappy: Incomplete Status in rsyslog package in Ubuntu: Incomplete Bug description: One of our commercial partners is developing a device agent snap for UC16. One of their engineers just shared the following bug report: I wanted to give you a heads up on an issue that we encountered with one of our gateways (a Dell 3000) regarding a full disk (/writable). It seems as though rsyslog had a bad logrotate config and was holding on to deleted files as described here: - https://www.claudiokuenzler.com/blog/861/after-ubuntu-update-trusty-xenial-disk-filled-syslog-logrotate - https://bugzilla.redhat.com/show_bug.cgi?id=1713358 Both df and du where showing different results for the “/writable” path on the Dell gateway. We were able to copy the lsof binary over and found this: $ sudo /tmp/lsof | grep deleted ... rsyslogd 1765 syslog 5w REG 179,4 1993793536 130599 /var/log/syslog (deleted) rsyslogd 1765 syslog 6w REG 179,4 45056 130836 /var/log/auth.log (deleted) rsyslogd 1765 syslog 7w REG 179,4 211976192 130782 /var/log/kern.log (deleted) in:imuxso 1765 1776 syslog 5w REG 179,4 1993793536 130599 /var/log/syslog (deleted) in:imuxso 1765 1776 syslog 6w REG 179,4 45056 130836 /var/log/auth.log (deleted) in:imuxso 1765 1776 syslog 7w REG 179,4 211976192 130782 /var/log/kern.log (deleted) in:imklog 1765 1777 syslog 5w REG 179,4 1993793536 130599 /var/log/syslog (deleted) in:imklog 1765 1777 syslog 6w REG 179,4 45056 130836 /var/log/auth.log (deleted) in:imklog 1765 1777 syslog 7w REG 179,4 211976192 130782 /var/log/kern.log (deleted) rs:main 1765 1778 syslog 5w REG 179,4 1993793536 130599 /var/log/syslog (deleted) rs:main 1765 1778 syslog 6w REG 179,4 45056 130836 /var/log/auth.log (deleted) rs:main 1765 1778 syslog 7w REG 179,4 211976192 130782 /var/log/kern.log (deleted) ... Restarting the service freed up the deleted files / disk space: sudo service rsyslog restart The rsyslog config (/etc/logrotate.d/rsyslog) calls this script post rotate: /usr/lib/rsyslog/rsyslog-rotate Which does not actually kill the rsyslog process: admin@GR0GP42:~$ ps aux | grep rsyslog syslog 2305 0.7 0.1 262692 3432 ? Ssl 20:35 0:00 /usr/sbin/rsyslogd -n admin 2464 0.0 0.0 12984 932 pts/0 S+ 20:36 0:00 grep --color=auto rsyslog admin@GR0GP42:~$ sudo /usr/lib/rsyslog/rsyslog-rotate admin@GR0GP42:~$ ps aux | grep rsyslog syslog 2305 0.6 0.1 262692 3432 ? Ssl 20:35 0:00 /usr/sbin/rsyslogd -n admin 2469 0.0 0.0 12984 972 pts/0 S+ 20:36 0:00 grep --color=auto rsyslog The fix appears to be using the following command: sudo systemctl restart rsyslog >/dev/null 2>&1 || true admin@GR0GP42:~$ ps aux | grep rsyslog syslog 2305 0.6 0.1 262692 3432 ? Ssl 20:35 0:00 /usr/sbin/rsyslogd -n admin 2482 0.0 0.0 12984 972 pts/0 S+ 20:36 0:00 grep --color=auto rsyslog admin@GR0GP42:~$ sudo systemctl restart rsyslog >/dev/null 2>&1 || true admin@GR0GP42:~$ ps aux | grep rsyslog syslog 2487 1.0 0.1 262692 3432 ? Ssl 20:36 0:00 /usr/sbin/rsyslogd -n admin 2496 0.0 0.0 12984 980 pts/0 S+ 20:36 0:00 grep --color=auto rsyslog To manage notifications about this bug go to: https://bugs.launchpad.net/snappy/+bug/1875493/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
