Rich Freeman posted on Tue, 24 Feb 2015 16:44:59 -0500 as excerpted: > On Tue, Feb 24, 2015 at 3:15 PM, Marc Joliet <mar...@gmx.de> wrote: >> >> == Things I have *not* gotten rid of (yet) == >> >> I plan on uninstalling syslog-ng, but haven't done so yet. I simply >> feel better waiting a bit, even though I don't run it anymore. Man, I >> feel silly after typing that... > > Yeah, took me a while to uninstall it, but I don't find much value in > running both.
First, welcome to systemd! =:^) I may respond to other parts too but this one's reasonably easy, while my choice is different, so I'll explain what and why... Intro: I chose to keep syslog-ng running here, while I (of course) have journald running as well. I believe the way I have configured both plays to the advantages of each, while discounting the weak areas of each as the other one covers that area. What: 1) Journald is configured to use volatile storage (the /run/log/journal subdir, with /run being tmpfs) ONLY, effectively limiting it to single- session. 2) Syslog-ng is configured pretty much as before, except that I have its systemd USE flag turned on, which builds it with journald support so the two cooperate a bit better. Without looking too much into details, I believe USE=systemd for syslog-ng tells it to build against a library systemd/journald provides for that purpose, such that syslog-ng's "system" source listens to journald instead of /dev/log and the kernel (dmesg) directly. What I can say is that while I've seen complaints from others about loggers getting duplicate entries (from systemd and the kernel both, for instance) in some cases, or missing entries in others (the OP, here), I've not noticed anything like that, here, and I believe it's due to the above -- only having the "system" and internal syslog-ng sources configured so syslog-ng isn't getting the same message from two sources, and building with the systemd USE flag, which basically configures syslog- ng's system source to be lazy and let journald feed it, instead of doing all the work of fetching and parsing the kernel and syslog socket on its own. Why: I love the journald/systemd integration -- having the last few log messages on systemctl status <service> makes troubleshooting within the same session **MUCH** easier, and the other information journald keeps is also useful. But, for me at least, that's most useful in the same session. Once I reboot and the information I'm looking for is in a previous session, the usefulness of that extra information goes down quite a bit. Meanwhile, for logs beyond one session, I still prefer traditional text- based logs, tho I suppose that's partly because that's what I'm most used to. But there's several advantages to it that I see: a) If the system crashes, partially corrupted in-the-crash text logs can be of at least some use after a reboot. Binary journals, not so much. b) As far as I could find, journald has absolutely no mechanism to filter incoming (pre-storage) log entries, while syslog-ng has had (pre-storage) filter mechanisms for ages. Journald seems to have all sorts of outgoing post-storage filter-on-output options, some of which are pretty nifty, but there have been several times in the past where logs filled up with "noise", thousands of repeat entries for something or other that occurs regularly, say once a minute, that I simply do not care about and don't want in my logs at all, but which I can't directly stop at the source for whatever reason. One of them was a kernel bug that triggered for a few kernel cycles, then eventually went away, another was a ksysguard triggered message, with that ksysguard graph updating every second, etc. Unfortunately, while journald makes it easy to filter these out on output, I found no way of preventing their appearance in the journal at all. But syslog-ng lets me dump them without ever actually logging them, or route them to a different log file if I prefer, keeping my primary logs clean. =:^) As a counter-point, however, at least journald does compression to some degree, so tens of thousands of identical or nearly identical messages don't take up as much room as you might expect as they compress very well. =:^) I guess after the first instance, dups basically consist of a timestamp and a pointer to the first message. So they don't take much space. Which is fortunate when some "noise" message starts appearing once a second... and you're storing it in tmpfs, thus in memory! Obviously persistent logs would be similarly space efficient, but I still don't like the idea of not being able to filter out the noise, BEFORE it hits permanent storage, so I just make sure it /doesn't/ hit permanent storage. =:^) c) I use btrfs for my primary filesystems, and btrfs and journald's binary-format journals don't play so well together. FWIW, the latest systemd-219 has lots of improvements for btrfs users, including some new features that are btrfs-specific (which I never-the-less don't expect to use right away as they're mostly VM/container oriented and I don't do much of that sort of thing, but I may well eventually use them), as well as some changes that should make journaling on btrfs work somewhat better. So this point should be to some degree eliminated after that stuff stabilizes tho part of the fix is journald setting nocow on the journal files and that has other implications I'll skip further discussion of here. However, by limiting journald to tmpfs only, I still get the best benefits of systemd/journald integration (like status including the last few log messages as mentioned above), while eliminating the entire class of btrfs/journald issues. Best of both worlds. =:^) d) Tho I don't do anything with it in my syslog-ng config here, syslog-ng can actually make use of some of the extra information journald passes to it, including trusted per-app filtering and routing. One of the problems that the old syslog approach had was that anything that could log messages could claim to be anything it wanted, and syslog really couldn't tell whether it was telling the truth or not, all it could do is sort and report the messages based on what was claimed. A big bullet-point feature of systemd/journald is that because systemd enforces service separation via control-classes and the like, there's a much stronger guarantee that what systemd/journald says is service X, is REALLY service X and not service Y! But that's not limited to journald. Syslog-ng has for several versions now had the ability to filter, route and report on out-of-log-channel information such as the extra properties that journald associates with each message, that make features such as systemctl status reporting the last few log messages from that service possible, because systemd/journald actually tracks it, AND passes it on to other loggers as well now, providing they're built to use the API provided by the systemd library I mentioned above. And again, for gentooers, syslog-ng is built with that API, against that library, based on USE=systemd. So once that's there, if desired, you can do all sorts of fancy filtering/log- routing/etc based on information such as the application/service name, that systemd passes along to syslog-ng. And again, with syslog-ng, this can be done BEFORE the actual logging if desired, instead of logging everything and filtering only on output, as journald does. =:^) e) I'm simply more comfortable dealing with text-based logs, since I'm used to them and can use whatever tools I want on them, as well as being able to split/sort/route to multiple log files in as simple or complex a setup as I want. =:^) Summary: For me, splitting up logging duties so journald does the collection, but only does session journaling to tmpfs, while syslog-ng still handles the "persistent" logging, gives me the best of both worlds. I love systemd/ journald features such as log-messages-in-status and having seen how useful it is, would seriously find it hard to do without. But at the same time, I can't bring myself to accept persistent logging without the ability to filter out the noise before it hits persistent storage, and journal storage on btrfs is problematic anyway, with both problems nicely solved by handing off to syslog-ng for my persistent storage needs while keeping journald for same-session use, so that's exactly what I do. =:^) -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
