Public bug reported: OS provided by AKS is currently Ubuntu 16.04.6 LTS, kernel 4.15.0-1077-azure.
Every block written by a k8s pod to a ceph CSI volume generates 2 warning lines in the node's system logs (kern.log, syslog, messages, warn): "Apr 24 09:37:46 aks-<nodename> kernel: [242123.654538] ceph: Failed to find inode for 1" Under production load, eventually the node succumbs to DiskPressure as the drive fills up. Also performance is noticeably degraded. Background here: https://tracker.ceph.com/issues/45283 Luis Hernandez indicates 4 commits relating to this issue, just 2 of which have been backported to Ubuntu 16. d557c48db730 ("ceph: quota: add counter for snaprealms with quota") <== e3161f17d926 ("ceph: quota: cache inode pointer in ceph_snap_realm") 0eb6bbe4d9cf ("ceph: fix root quota realm check") <== 2596366907f8 ("ceph: don't check quota for snap inode") Quoth Luis: "I've done a quick test and, after compiling the bionic kernel 4.15.0-96.97 (the latest released), I can reproduce the issue. Cherry-picking the 2 missing commits (2596366907f8 and e3161f17d926) fixes it." In my testing Ubuntu 18 does not exhibit the bug, but Azure support tells me it will be months before they make it GA in AKS. Can we get those commits backported to Ubuntu 16? ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-azure (not installed) ProcVersionSignature: Ubuntu 4.15.0-1077.82-azure 4.15.18 Uname: Linux 4.15.0-1077-azure x86_64 ApportVersion: 2.20.1-0ubuntu2.23 Architecture: amd64 Date: Wed Apr 29 12:45:12 2020 ProcEnviron: TERM=xterm-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash SourcePackage: linux-meta-azure UpgradeStatus: No upgrade log present (probably fresh install) ** Affects: linux-meta-azure (Ubuntu) Importance: Undecided Status: New ** Tags: amd64 apport-bug uec-images xenial -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-meta-azure in Ubuntu. https://bugs.launchpad.net/bugs/1875884 Title: Kernel log flood "ceph: Failed to find inode for 1" Status in linux-meta-azure package in Ubuntu: New Bug description: OS provided by AKS is currently Ubuntu 16.04.6 LTS, kernel 4.15.0-1077-azure. Every block written by a k8s pod to a ceph CSI volume generates 2 warning lines in the node's system logs (kern.log, syslog, messages, warn): "Apr 24 09:37:46 aks-<nodename> kernel: [242123.654538] ceph: Failed to find inode for 1" Under production load, eventually the node succumbs to DiskPressure as the drive fills up. Also performance is noticeably degraded. Background here: https://tracker.ceph.com/issues/45283 Luis Hernandez indicates 4 commits relating to this issue, just 2 of which have been backported to Ubuntu 16. d557c48db730 ("ceph: quota: add counter for snaprealms with quota") <== e3161f17d926 ("ceph: quota: cache inode pointer in ceph_snap_realm") 0eb6bbe4d9cf ("ceph: fix root quota realm check") <== 2596366907f8 ("ceph: don't check quota for snap inode") Quoth Luis: "I've done a quick test and, after compiling the bionic kernel 4.15.0-96.97 (the latest released), I can reproduce the issue. Cherry-picking the 2 missing commits (2596366907f8 and e3161f17d926) fixes it." In my testing Ubuntu 18 does not exhibit the bug, but Azure support tells me it will be months before they make it GA in AKS. Can we get those commits backported to Ubuntu 16? ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-azure (not installed) ProcVersionSignature: Ubuntu 4.15.0-1077.82-azure 4.15.18 Uname: Linux 4.15.0-1077-azure x86_64 ApportVersion: 2.20.1-0ubuntu2.23 Architecture: amd64 Date: Wed Apr 29 12:45:12 2020 ProcEnviron: TERM=xterm-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash SourcePackage: linux-meta-azure UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-meta-azure/+bug/1875884/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
