I think we might be encountering the same issue. At least, we're also trying to enable imjournal in rsyslog because we want all of the structured log fields from systemd journal, and we're encountering the same error messages when starting rsyslog.service.
We are running an x86 EC2 instance: $ uname -a Linux ip-10-XXX-YYY-ZZZ 6.8.0-1016-aws #17-Ubuntu SMP Mon Sep 2 13:48:07 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux $ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 24.04.1 LTS Release: 24.04 Codename: noble $ dpkg -l rsyslog Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-==============-=================-============-========================================= ii rsyslog 8.2312.0-3ubuntu9 amd64 reliable system and kernel logging daemon I can also confirm that there are messages related to AppArmor denying rsyslog at approximately the same time in our dmesg: [Wed Oct 16 11:15:39 2024] audit: type=1400 audit(1729077335.160:679): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="rsyslogd" pid=506096 comm="apparmor_parser" [Wed Oct 16 11:15:39 2024] audit: type=1400 audit(1729077335.187:680): apparmor="DENIED" operation="open" class="file" profile="rsyslogd" name="/run/log/journal/" pid=506098 comm="rsyslogd" requested_mask="r" denied_mask="r" fsuid=102 ouid=0 [Wed Oct 16 11:15:39 2024] audit: type=1400 audit(1729077335.187:681): apparmor="DENIED" operation="open" class="file" profile="rsyslogd" name="/etc/machine-id" pid=506098 comm="rsyslogd" requested_mask="r" denied_mask="r" fsuid=102 ouid=0 [Wed Oct 16 11:15:39 2024] audit: type=1400 audit(1729077335.187:682): apparmor="DENIED" operation="open" class="file" profile="rsyslogd" name="/run/log/journal/" pid=506098 comm="rsyslogd" requested_mask="r" denied_mask="r" fsuid=102 ouid=0 [Wed Oct 16 11:15:39 2024] audit: type=1400 audit(1729077335.187:683): apparmor="DENIED" operation="open" class="file" profile="rsyslogd" name="/etc/machine-id" pid=506098 comm="rsyslogd" requested_mask="r" denied_mask="r" fsuid=102 ouid=0 [Wed Oct 16 11:15:39 2024] audit: type=1400 audit(1729077335.187:684): apparmor="DENIED" operation="open" class="file" profile="rsyslogd" name="/run/log/journal/" pid=506098 comm="in:imjournal" requested_mask="r" denied_mask="r" fsuid=102 ouid=0 [Wed Oct 16 11:15:39 2024] audit: type=1400 audit(1729077335.187:685): apparmor="DENIED" operation="open" class="file" profile="rsyslogd" name="/etc/machine-id" pid=506098 comm="in:imjournal" requested_mask="r" denied_mask="r" fsuid=102 ouid=0 [Wed Oct 16 11:15:39 2024] audit: type=1400 audit(1729077335.187:686): apparmor="DENIED" operation="open" class="file" profile="rsyslogd" name="/run/log/journal/" pid=506098 comm="in:imjournal" requested_mask="r" denied_mask="r" fsuid=102 ouid=0 [Wed Oct 16 11:15:39 2024] audit: type=1400 audit(1729077335.187:687): apparmor="DENIED" operation="open" class="file" profile="rsyslogd" name="/etc/machine-id" pid=506098 comm="in:imjournal" requested_mask="r" denied_mask="r" fsuid=102 ouid=0 [Wed Oct 16 11:15:39 2024] audit: type=1400 audit(1729077335.192:688): apparmor="DENIED" operation="open" class="file" profile="rsyslogd" name="/run/log/journal/" pid=506098 comm="in:imjournal" requested_mask="r" denied_mask="r" fsuid=102 ouid=0 As you may notice the rsyslog service itself is logging that it can't create the systemd journal state file under /var/spool/rsyslog, but it appears AppArmor is actually preventing rsyslog & imjournal from reading /run/log/journal/ and /etc/machine-id. I tried stopping and disabling AppArmor, and I also tried symlinking /etc/apparmor.d/usr.sbin.rsyslog from /etc/apparmor.d/disable/ and running apparmor_parser -R /etc/apparmor.d/usr.sbin.rsyslog, and confirmed /usr/sbin/rsyslog was not being enforced by running aa-status. However, that did NOT allow rsyslog & imjournal to work as now imjournal is segfaulting: [Wed Oct 16 11:50:35 2024] in:imjournal[516014]: segfault at 40 ip 000058bd6b96eb21 sp 000071bcd45ff9e0 error 6 in rsyslogd[58bd6b93f000+6f000] likely on CPU 1 (core 0, socket 0) [Wed Oct 16 11:50:35 2024] Code: b7 10 66 41 89 56 08 0f b6 40 02 41 88 46 0a e9 3f fe ff ff e8 b0 1f fd ff f3 0f 1e fa 55 48 89 e5 41 54 49 89 fc 53 48 8b 1f <f0> 83 6b 40 01 0f 85 c8 01 00 00 48 8b 7b 70 48 8d 83 50 01 00 00 [Wed Oct 16 11:50:51 2024] rs:main Q:Reg[516078]: segfault at 0 ip 000055e61b25f3d0 sp 000079c6479ff5e8 error 4 in rsyslogd[55e61b225000+6f000] likely on CPU 1 (core 0, socket 0) [Wed Oct 16 11:50:51 2024] Code: 01 4c 63 c0 41 89 c1 4d 69 c0 ab aa aa 2a 41 c1 f9 1f 49 c1 f8 21 45 29 c8 47 8d 04 40 41 c1 e0 02 44 29 c0 48 98 48 8b 04 c2 <0f> b6 00 88 01 0f be 47 01 83 e8 01 4c 63 c0 41 89 c1 4d 69 c0 ab [Wed Oct 16 11:50:51 2024] in:imjournal[516144]: segfault at 7a160c000090 ip 00007a160c000090 sp 00007a16415ff9c8 error 15 likely on CPU 1 (core 0, socket 0) [Wed Oct 16 11:50:51 2024] Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <a0> de 00 0c 16 7a 00 00 40 c4 00 0c 16 7a 00 00 d0 61 00 0c 16 7a [Wed Oct 16 11:50:52 2024] in:imjournal[516155]: segfault at 73f1f40054b0 ip 000073f1f40054b0 sp 000073f23e3ff878 error 15 likely on CPU 0 (core 0, socket 0) [Wed Oct 16 11:50:52 2024] Code: 00 00 e0 8f 00 f4 f1 73 00 00 10 01 00 00 00 00 00 00 24 00 00 00 00 00 00 00 45 13 1f cb f6 73 00 00 45 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 11 01 00 00 00 00 00 00 40 45 00 f4 f1 73 This should be pretty easy to reproduce as I can trigger it with a minimal config in /etc/rsyslog.d/: module(load="imjournal" StateFile="systemd_journald_state" IgnorePreviousMessages="on") module(load="mmjsonparse") module(load="omfwd") template(name="systemd_journal_json" type="string" string="%$!all- json%\n" ) action(type="mmjsonparse") user.* action(type="omfwd" target="remote-rsyslog" port="514" protocol="tcp" template="systemd_journal_json") -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to rsyslog in Ubuntu. https://bugs.launchpad.net/bugs/2073628 Title: imjournal module works with rsyslog package of ubuntu 22.04 but not with ubuntu 24.04 Status in rsyslog package in Ubuntu: Incomplete Bug description: imjournal module fails to create /var/spool/rsyslog/journal-state file in ubuntu 24.04, rsyslog version(8.2312.0) x86 and s390x both, but works well in ubuntu 22.04 , rsyslog version(8.2112.0) x86 and s390x ******* Ubuntu 24.04 s390x lsb_release -rd No LSB modules are available. Description: Ubuntu 24.04 LTS Release: 24.04 # apt-cache policy rsyslog rsyslog: Installed: 8.2312.0-3ubuntu9 Candidate: 8.2312.0-3ubuntu9 Version table: *** 8.2312.0-3ubuntu9 500 500 http://ports.ubuntu.com/ubuntu-ports noble/main s390x Packages 100 /var/lib/dpkg/status Have below line in /etc/rsyslog.conf module(load="imjournal" fileCreateMode="0666" PersistStateInterval="999" StateFile="/var/spool/rsyslog/journal_state") ul 19 18:39:35 latest-logs systemd[1]: Starting rsyslog.service - System Logging Service... Jul 19 18:39:35 latest-logs rsyslogd[8647]: rsyslogd's groupid changed to 102 Jul 19 18:39:35 latest-logs rsyslogd[8647]: rsyslogd's userid changed to 102 Jul 19 18:39:35 latest-logs systemd[1]: Started rsyslog.service - System Logging Service. Jul 19 18:39:35 latest-logs rsyslogd[8647]: [origin software="rsyslogd" swVersion="8.2312.0" x-pid="8647" x-info="https://www.rsyslog.com";] start Jul 19 18:39:35 latest-logs rsyslogd[8647]: imjournal: No statefile exists, /var/spool/rsyslog/journal_state will be created (ignore if this is first run): No such file or directory > Jul 19 18:39:35 latest-logs rsyslogd[8647]: imjournal: Journal indicates no msgs when positioned at head. [v8.2312.0 try https://www.rsyslog.com/e/0 ] Jul 19 18:39:35 latest-logs rsyslogd[8647]: imjournal: journal files changed, reloading... [v8.2312.0 try https://www.rsyslog.com/e/0 ] Jul 19 18:39:35 latest-logs rsyslogd[8647]: imjournal: No statefile exists, /var/spool/rsyslog/journal_state will be created (ignore if this is first run): No such file or directory > Jul 19 18:39:35 latest-logs rsyslogd[8647]: imjournal: Journal indicates no msgs when positioned at head. [v8.2312.0 try https://www.rsyslog.com/e/0 ] lines 1-25/25 (END) FIle /var/spool/rsyslog/journal_state should have created and logs should have redirected to rsyslog server ****** In Ubuntu 22.04 all is working as expected # lsb_release -rd Description: Ubuntu 22.04.4 LTS Release: 22.04 #apt-cache policy rsyslog rsyslog: Installed: 8.2112.0-2ubuntu2.2 Candidate: 8.2112.0-2ubuntu2.2 Version table: *** 8.2112.0-2ubuntu2.2 100 100 /var/lib/dpkg/status Use the same line as above in /etc/rsyslog.conf restart service. it did gave error about fileCreateMode which got ignored and proceeded to create the journal-state file and continued without any error Jul 19 18:44:37 systemd[1]: Starting System Logging Service... Jul 19 18:44:37 rsyslogd[13664]: error during parsing file /etc/rsyslog.conf, on or before line 16: parameter 'fileCreateMode' not known -- typo in co> Jul 19 18:44:37 systemd[1]: Started System Logging Service. Jul 19 18:44:37 rsyslogd[13664]: rsyslogd's groupid changed to 111 Jul 19 18:44:37 rsyslogd[13664]: rsyslogd's userid changed to 104 Jul 19 18:44:37 rsyslogd[13664]: [origin software="rsyslogd" swVersion="8.2112.0" x-pid="13664" x-info="https://www.rsyslog.com";] start Jul 19 18:44:37 rsyslogd[13664]: imjournal: journal files changed, reloading... [v8.2112.0 try https://www.rsyslog.com/e/0 ] /var/spool/rsyslog# ls journal_state ***** please help with this issue To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsyslog/+bug/2073628/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
