Am 06.07.21 um 11:51 schrieb Ben Wong:
Package: rsyslog
Version: 8.2102.0-2
Severity: important

Dear Maintainer,

My Debian box's hard disk filled up due to a single buggy user
application spewing messages at syslog. While I blame the application,
Debian's rsyslog should be more robust by default.

This is something I've seen happening more and more frequently.
A typical response people get when googling the problem is
"sudo rm /var/log/* and then set that up to run daily as a cronjob."
That's terrible advice and makes Debian seem shoddy.

Ideally, I'd like to see some sort of rate-limiter added to rsyslog.

There is a rater limiter in imuxsock.
https://www.rsyslog.com/how-to-use-rate-limiting-in-rsyslog/

The problem is: a couple of GB of log data is something that a busy log server can generate in a couple of minutes which perfectly valid.

On a laptop, a couple of GB of log data is probably unusual.

The problem is to pick a default which suits everyone, which is guess is not possible.




In the meantime, here is a simple three step fix which would improve
stability for many people using Debian.


1. /etc/rsyslog.conf: add user.none to /var/log/syslog, debug, and messages.
    That way, the spewage is limited to a single file, user.log.

I'm not really sure we can win this game.
Next time, it's not a user application generating a lot of data but some system service.

That said, there is certainly value in cleaning up the default Debian rsyslog default config, see e.g.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=508376
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=580552
to avoid duplication in log messages.


2. /etc/logrotate.d/rsyslog: add "maxsize" so that log files that are
    greater than a certain size are rotated even if their time
    criteria are not due. I suggest "maxsize 1G" is reasonable for
    most people and can be adjusted for those who want more.

Depends. I'd argue that the systemd journal nowadays is better suited for end user systems anyway and syslog/rsyslog more targetted at the enterprise area. So I don't think it makes sense to optimize for the former.

3a. Move /etc/cron.daily/logrotate to /etc/cron.hourly/ so that file
     sizes are checked more often. (This also fixes the "bug" where an
     hourly entry in logrotate.conf only gets rotated daily.)

3b. /usr/lib/systemd/system/logrotate.timer: Likewise for systemd.

     [Timer]                                    [Timer]
     OnCalendar=daily           -->          OnCalendar=hourly
     AccuracySec=1h                             AccuracySec=1m
     Persistent=true                            Persistent=true


Steps 1 and 2 ensure that the disk will not fill up and that important
system messages won't be rotated away too quickly based on file size.

Step 3 is necessary because the time in which a log file can fill up
/var is no longer measured in days. Currently, I have a single
chromium process sending over thirty thousand messages per second and
I doubt that's even close to the maximum possible.

I don't think, changing the cron job to hourly is really going to help if you have an application running amok. 1 hour is more then sufficient to fill up your disk.


Fwiw, my plan is for bookworm to have rsyslog removed from a default installation and rely on journald only (which does size-based log truncation), as I think journald is a better fit for end-user systems.

rsyslog would then only be installed by those who explicitly need it (those users will most likely will tweak its settings anyway to suit their needs).

Regards,
Michael


Attachment: OpenPGP_signature
Description: OpenPGP digital signature

Reply via email to