[Kernel-packages] [Bug 1687267] [NEW] kernel BUG at /build/linux-2NWldV/linux-4.10.0/include/linux/swapops.h:129!
Public bug reported: First traceback looks to be the cause. The second traceback is the effect that followed (repeatedly). Don't think it's specific to system76. Apr 30 09:56:10 galago kernel: [97085.090114] [ cut here ] Apr 30 09:56:10 galago kernel: [97085.090137] kernel BUG at /build/linux-2NWldV/linux-4.10.0/include/linux/swapops.h:129! Apr 30 09:56:10 galago kernel: [97085.090157] invalid opcode: [#1] SMP Apr 30 09:56:10 galago kernel: [97085.090167] Modules linked in: nls_iso8859_1 uas usb_storage ccm rfcomm ipt_MASQUERADE nf_nat_masquerade_ipv4 xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack libcrc32c br_netfilter bridge stp llc aufs ec_sys cmac bnep arc4 intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel snd_hda_codec_hdmi kvm irqbypass crct10dif_pclmul crc32_pclmul snd_hda_codec_via ghash_clmulni_intel snd_hda_codec_generic pcbc snd_hda_intel snd_hda_codec iwlmvm snd_hda_core snd_hwdep aesni_intel snd_pcm uvcvideo mac80211 aes_x86_64 snd_seq_midi videobuf2_vmalloc crypto_simd snd_seq_midi_event glue_helper videobuf2_memops cryptd videobuf2_v4l2 btusb snd_rawmidi videobuf2_core btrtl snd_seq btbcm intel_cstate videodev btintel Apr 30 09:56:10 galago kernel: [97085.090334] bluetooth snd_seq_device media intel_rapl_perf iwlwifi snd_timer joydev input_leds cfg80211 rtsx_pci_ms snd memstick mei_me soundcore mei lpc_ich shpchp serio_raw tpm_infineon mac_hid binfmt_misc cuse parport_pc ppdev lp parport ip_tables x_tables autofs4 btrfs xor raid6_pq dm_mirror dm_region_hash dm_log i915 rtsx_pci_sdmmc i2c_algo_bit drm_kms_helper psmouse e1000e syscopyarea sysfillrect ahci sysimgblt fb_sys_fops libahci drm ptp rtsx_pci pps_core wmi fjes video Apr 30 09:56:10 galago kernel: [97085.090439] CPU: 0 PID: 3534 Comm: JS Helper Not tainted 4.10.0-20-generic #22-Ubuntu Apr 30 09:56:10 galago kernel: [97085.090457] Hardware name: System76, Inc. Galago UltraPro/W740SU , BIOS 4.6.5 12/12/2013 Apr 30 09:56:10 galago kernel: [97085.090481] task: 9003eb142d00 task.stack: a0fb83c34000 Apr 30 09:56:10 galago kernel: [97085.090499] RIP: 0010:__migration_entry_wait+0x16a/0x180 Apr 30 09:56:10 galago kernel: [97085.090512] RSP: :a0fb83c37d68 EFLAGS: 00010246 Apr 30 09:56:10 galago kernel: [97085.090525] RAX: 000fc0048078 RBX: faefcf1c69f0 RCX: faefcf1c69f0 Apr 30 09:56:10 galago kernel: [97085.090542] RDX: 0001 RSI: 9003c71a7800 RDI: faefc2f44000 Apr 30 09:56:10 galago kernel: [97085.090559] RBP: a0fb83c37d80 R08: 9003ebb38000 R09: 9003ebb38000 Apr 30 09:56:10 galago kernel: [97085.090576] R10: 7fc0379000c8 R11: 0206 R12: faefc2f44000 Apr 30 09:56:10 galago kernel: [97085.090592] R13: 3e0bd100 R14: a0fb83c37e30 R15: 9003d0b80898 Apr 30 09:56:10 galago kernel: [97085.090609] FS: 7fc01b3fa700() GS:90041fa0() knlGS: Apr 30 09:56:10 galago kernel: [97085.090628] CS: 0010 DS: ES: CR0: 80050033 Apr 30 09:56:10 galago kernel: [97085.090642] CR2: 7fc004700018 CR3: 0003c70f9000 CR4: 001406f0 Apr 30 09:56:10 galago kernel: [97085.090658] Call Trace: Apr 30 09:56:10 galago kernel: [97085.090667] migration_entry_wait+0x74/0x80 Apr 30 09:56:10 galago kernel: [97085.090680] do_swap_page+0x5b3/0x770 Apr 30 09:56:10 galago kernel: [97085.090691] handle_mm_fault+0x873/0x1360 Apr 30 09:56:10 galago kernel: [97085.090703] __do_page_fault+0x23e/0x4e0 Apr 30 09:56:10 galago kernel: [97085.090715] do_page_fault+0x22/0x30 Apr 30 09:56:10 galago kernel: [97085.090726] page_fault+0x28/0x30 Apr 30 09:56:10 galago kernel: [97085.090735] RIP: 0033:0x56520e6072b4 Apr 30 09:56:10 galago kernel: [97085.090744] RSP: 002b:7fc01b3f9c40 EFLAGS: 00010203 Apr 30 09:56:10 galago kernel: [97085.090757] RAX: RBX: 7fc037900040 RCX: 7fc037b7b137 Apr 30 09:56:10 galago kernel: [97085.090774] RDX: 0008 RSI: 2000 RDI: 7fc0047ba000 Apr 30 09:56:10 galago kernel: [97085.090790] RBP: 7fc00470 R08: 7fc0379000c8 R09: Apr 30 09:56:10 galago kernel: [97085.090807] R10: 7fc0379000c8 R11: 0206 R12: 0080 Apr 30 09:56:10 galago kernel: [97085.090824] R13: 7fc0379000c8 R14: 00b9 R15: 0002 Apr 30 09:56:10 galago kernel: [97085.090840] Code: ff ff ff 4c 89 e7 e8 96 a2 f8 ff e9 3c ff ff ff 85 d2 0f 84 2a ff ff ff 8d 4a 01 89 d0 f0 41 0f b1 4d 00 39 d0 74 81 89 c2 eb e5 <0f> 0b 4c 89 e7 e8 0c fb f9 ff eb b8 4c 8d 60 ff 4c 8d 68 1b eb Apr 30 09:56:10 galago kernel: [97085.090897] RIP: __migration_entry_wait+0x16a/0x180 RSP: a0fb83c37d68 Apr 30 09:56:10 galago kernel: [97085.096483] ---[ end trace 03fb86cf3cfe64f3 ]--- Apr 30 09:56:25 galago kernel: [97100.371032] iwlwif
[Kernel-packages] [Bug 1730550] Re: e1000e in 4.4.0-97-generic breaks 82574L under heavy load.
Hi, will these patches also land in the HWE kernel? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1730550 Title: e1000e in 4.4.0-97-generic breaks 82574L under heavy load. Status in linux package in Ubuntu: In Progress Status in linux source package in Xenial: In Progress Bug description: This issue was first reported on the netdev email list by Lennart Sorensen: https://www.mail-archive.com/netdev@vger.kernel.org/msg178170.html Commit 16ecba59bc333d6282ee057fb02339f77a880beb causes link drops on the 82574L under heavy load. "Unfortunately this commit changed the driver to assume that the Other Causes interrupt can only mean link state change and hence sets the flag that (unfortunately) means both link is down and link state should be checked. Since this now happens 3000 times per second, the chances of it happening while the watchdog_task is checking the link state becomes pretty high, and it if does happen to coincice, then the watchdog_task will reset the adapter, which causes a real loss of link." A fix for this issue was accepted into the net-next branch, along with other e1000e/igb patches: https://git.kernel.org/pub/scm/linux/kernel/git/davem/net- next.git/commit/?id=f44dea3421b47d355a835e9cfcc59ca7318575a9 The original reported experienced this issue on a Supermicro X7SPA- HF-D525 server board. We see this issue on many servers running X9DBL- 1F server boards. Both boards use the Intel 82574L for the network interfaces. We see messages like this under heavy load: [Nov 6 15:42] e1000e: eth0 NIC Link is Down [ +0.001670] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx [Nov 6 16:10] e1000e: eth0 NIC Link is Down [ +0.008505] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx [Nov 7 00:49] e1000e: eth0 NIC Link is Down [ +2.235111] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx We have confirmed that the connected switch sees the link drops also, to these are not false alarms from the e1000e driver. # lsb_release -rd Description: Ubuntu 16.04.2 LTS Release: 16.04 I could not cleanly apply the net-next patch to 4.4.0 so I tested with just the following cherry picked changes on the latest 4.4.0 kernel source package. https://patchwork.ozlabs.org/patch/823942/ https://patchwork.ozlabs.org/patch/823945/ https://patchwork.ozlabs.org/patch/823940/ https://patchwork.ozlabs.org/patch/823941/ https://patchwork.ozlabs.org/patch/823939/ Although it's my understanding the first two are the critical ones for the race condition. I have been running with the patches e1000e kernel driver, under network load for 7 days and I no longer see the network interface drops. Could we pull these changes into the Ubuntu 4.4.0 kernel ? Thanks --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Jul 19 07:34 seq crw-rw 1 root audio 116, 33 Jul 19 07:34 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.20.1-0ubuntu2.10 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 16.04 HibernationDevice: RESUME=UUID=49ca52b8-cf08-4485-b296-0dffb098e557 IwConfig: Error: [Errno 2] No such file or directory Lsusb: Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 003: ID 0557:2221 ATEN International Co., Ltd Winbond Hermon Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub MachineType: Supermicro X9DBL-3F/X9DBL-iF Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=xterm-256color PATH=(custom, no user) LANG=en_GB.UTF-8 SHELL=/bin/bash ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-83-generic root=UUID=957d7126-5452-4606-942d-1d58adbeb253 ro net.ifnames=0 biosdevname=0 quiet splash nomdmonddf nomdmonisw ProcVersionSignature: Ubuntu 4.4.0-83.106-generic 4.4.70 RelatedPackageVersions: linux-restricted-modules-4.4.0-83-generic N/A linux-backports-modules-4.4.0-83-generic N/A linux-firmware1.157.11 RfKill: Error: [Errno 2] No such file or directory Tags: xenial xenial Uname: Linux 4.4.0-83-generic x86_64 UnreportableReason: The report belongs to a package that is not installed. UpgradeStatus: Upgraded to xenial on 2016-12-05 (337 days ago) UserGroups: _MarkForUpload: False dmi.bios.date: 12/28/2012 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: 2.00 dmi.board.asset.tag: To be filled by O.E.M. dmi
[Kernel-packages] [Bug 1730550] Re: e1000e in 4.4.0-97-generic breaks 82574L under heavy load.
Thanks.
By the way, also confirming that the patch works here (have been running
the patched kernel for 3 days now). Running on hardware Super Micro
X9SCL with an Intel 82574L NIC.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1730550
Title:
e1000e in 4.4.0-97-generic breaks 82574L under heavy load.
Status in linux package in Ubuntu:
In Progress
Status in linux source package in Xenial:
In Progress
Status in linux source package in Zesty:
In Progress
Status in linux source package in Artful:
In Progress
Bug description:
== SRU Justification ==
This issue was first reported on the netdev email list by Lennart Sorensen:
https://www.mail-archive.com/netdev@vger.kernel.org/msg178170.html
Commit 16ecba59bc333d6282ee057fb02339f77a880beb causes link drops on
the 82574L under heavy load.
"Unfortunately this commit changed the driver to assume
that the Other Causes interrupt can only mean link state change and
hence sets the flag that (unfortunately) means both link is down and link
state should be checked. Since this now happens 3000 times per second,
the chances of it happening while the watchdog_task is checking the link
state becomes pretty high, and it if does happen to coincice, then the
watchdog_task will reset the adapter, which causes a real loss of link."
The original reported experienced this issue on a Supermicro X7SPA-HF-D525
server board.
However, the bug is now seen on many servers running X9DBL-1F server boards.
This bug is fixed by commits 19110cfbb34 and 4aea7a5c5e9, which were both
added
to mainline in v4.15-rc1.
The commit that introduced this bug,16ecba5, was added to mainlien in
v4.5-rc1. However,
Xenial recived this commit as well as commit 531ff577a. Bionic master-next
does not need
these commits, since it got them via bug 1735843 and the 4.14.3 updates.
== Fixes ==
19110cfbb34 ("e1000e: Separate signaling for link check/link up")
4aea7a5c5e9 ("e1000e: Avoid receiver overrun interrupt bursts")
== Regression Potential ==
These commits are specific to e1000.
== Test Case ==
A test kernel was built with these patches and tested by the original bug
reporter.
The bug reporter states the test kernel resolved the bug.
== Original Bug Descriptio ==
This issue was first reported on the netdev email list by Lennart Sorensen:
https://www.mail-archive.com/netdev@vger.kernel.org/msg178170.html
Commit 16ecba59bc333d6282ee057fb02339f77a880beb causes link drops on
the 82574L under heavy load.
"Unfortunately this commit changed the driver to assume
that the Other Causes interrupt can only mean link state change and
hence sets the flag that (unfortunately) means both link is down and link
state should be checked. Since this now happens 3000 times per second,
the chances of it happening while the watchdog_task is checking the link
state becomes pretty high, and it if does happen to coincice, then the
watchdog_task will reset the adapter, which causes a real loss of link."
A fix for this issue was accepted into the net-next branch, along with
other e1000e/igb patches:
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-
next.git/commit/?id=f44dea3421b47d355a835e9cfcc59ca7318575a9
The original reported experienced this issue on a Supermicro X7SPA-
HF-D525 server board. We see this issue on many servers running X9DBL-
1F server boards. Both boards use the Intel 82574L for the network
interfaces. We see messages like this under heavy load:
[Nov 6 15:42] e1000e: eth0 NIC Link is Down
[ +0.001670] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow
Control: Rx/Tx
[Nov 6 16:10] e1000e: eth0 NIC Link is Down
[ +0.008505] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow
Control: Rx/Tx
[Nov 7 00:49] e1000e: eth0 NIC Link is Down
[ +2.235111] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow
Control: Rx/Tx
We have confirmed that the connected switch sees the link drops also,
to these are not false alarms from the e1000e driver.
# lsb_release -rd
Description: Ubuntu 16.04.2 LTS
Release: 16.04
I could not cleanly apply the net-next patch to 4.4.0 so I tested with just
the following cherry picked changes on the latest 4.4.0 kernel source package.
https://patchwork.ozlabs.org/patch/823942/
https://patchwork.ozlabs.org/patch/823945/
https://patchwork.ozlabs.org/patch/823940/
https://patchwork.ozlabs.org/patch/823941/
https://patchwork.ozlabs.org/patch/823939/
Although it's my understanding the first two are the critical ones for
the race condition. I have been running with the patches e1000e kernel
driver, under network load for 7 days and I no longer see the network
interface drops.
Could we pull these changes into the Ubuntu 4.4.0 kernel ?
Thanks
---
AlsaDevices:
total 0
crw-rw 1
[Kernel-packages] [Bug 1666260] Re: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34
As original poster, if I didn't continue to post oom dumps, perhaps things started to peter out on 4.8.0-39 or later. What was particular about the load that triggered this bug was heavy IO putting cache pressure on ext4 on a system where there's zero locality of reference in anything read from or written to disk (ssd backed storage). In any case, by May these data storage servers that had been triggering this issue had been decommissioned and IO strategy had changed. Now writes are written to a raw block device before being flushed to filesystem periodically using O_DSYNC, taking ext4 disk cache out of the equation. The HWE kernel is now 4.10, and judging by the edge packages soon to be 4.13, so maybe its been fixed in that time. However I'm no longer able to confirm or deny that, as there's no possible way for me to reproduce it anyway. As per Rasmus' comment, its something that only happened on production workloads. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1666260 Title: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34 Status in linux package in Ubuntu: Confirmed Bug description: Same as #1655842 - Started seeing oom-killer on multiple servers upgraded to 4.4.0-59. Unlike #1655842, also seeing the same oom-killer on multiple servers updated to 4.8.0-34. First upgraded them all 4.8 servers 4.8.0-36, then downgraded a few to 4.4.0-63. I am seeing an even more pronounced change in the memory usage, so I can only assume that 4.4.0-63 is also bugged with the same problem as 4.4.0-59 and 4.8.0-34. Either because #1655842 is not fixed, or it is only fixed for certain kinds of workloads. These are the changes I'm seeing in our memory graphs between 4.4.0-59 and 4.4.0-63/4.8.0-34. The symptoms I'm seeing are: Upgrading 4.4.0-57 -> 4.4.0-59: - /proc/meminfo:Buffers: Up from 9GB to 15GB - /proc/meminfo:Cached: Up from 5GB to 10GB - /proc/meminfo:SReclaimable: Down from 15GB to 5GB - /proc/meminfo:SUnreclaim: Staying at 50MB Upgrading 4.4.0-57 -> 4.4.0-63: - /proc/meminfo:Buffers: Up from 9GB to 26GB - /proc/meminfo:Cached: Down from 5GB to 300MB - /proc/meminfo:SReclaimable: Down from 15GB to 2GB - /proc/meminfo:SUnreclaim: Down from 50MB to 30MB Upgrading 4.4.0-57 -> 4.8.0-34: - /proc/meminfo:Buffers: Up from 9GB to 14GB - /proc/meminfo:Cached: Down from 5GB to 2GB - /proc/meminfo:SReclaimable: Down from 15GB to 14GB - /proc/meminfo:SUnreclaim: Staying at 50MB Setting vm.vfs_cache_pressure = 300 seems to have a positive effect of not causing OOMs. Downgrading to 4.4.0-57 also works. Will also note that I haven't had a definitive OOM in 4.4.0-63. But the shift in memory usage is far too much from what I expect to be normal on these particular servers where I'm experiencing crashes. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-63-generic 4.4.0-63.84 ProcVersionSignature: Ubuntu 4.4.0-63.84-generic 4.4.44 Uname: Linux 4.4.0-63-generic x86_64 AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.5 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' Date: Mon Feb 20 16:15:56 2017 InstallationDate: Installed on 2012-06-04 (1721 days ago) InstallationMedia: IwConfig: lono wireless extensions. eth0 no wireless extensions. Lsusb: Error: [Errno 2] No such file or directory: 'lsusb' MachineType: System manufacturer System Product Name PciMultimedia: ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-63-generic root=UUID=b790930f-ad81-4b27-a353-a4b3d6a29007 ro nomodeset nomdmonddf nomdmonisw RelatedPackageVersions: linux-restricted-modules-4.4.0-63-generic N/A linux-backports-modules-4.4.0-63-generic N/A linux-firmware1.157.8 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: Upgraded to xenial on 2017-02-16 (4 days ago) dmi.bios.date: 10/17/2011 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: 1106 dmi.board.asset.tag: To be filled by O.E.M. dmi.board.name: P8H67-M PRO dmi.board.vendor: ASUSTeK Computer INC. dmi.board.version: Rev 1.xx dmi.chassis.asset.tag: Asset-1234567890 dmi.chassis.type: 3 dmi.chassis.vendor: Chassis Manufacture dmi.chassis.version: Chassis Version dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1106:bd10/17/2011:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKComputerINC.:rnP8H67-MPRO:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion: dmi.product.name: System Product Name dmi.product.version: System Version dmi.sys.
[Kernel-packages] [Bug 1666260] Re: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34
Servers are now running on 4.10 kernels, so I guess no one cares about 4.8 anymore. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1666260 Title: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34 Status in linux package in Ubuntu: Confirmed Bug description: Same as #1655842 - Started seeing oom-killer on multiple servers upgraded to 4.4.0-59. Unlike #1655842, also seeing the same oom-killer on multiple servers updated to 4.8.0-34. First upgraded them all 4.8 servers 4.8.0-36, then downgraded a few to 4.4.0-63. I am seeing an even more pronounced change in the memory usage, so I can only assume that 4.4.0-63 is also bugged with the same problem as 4.4.0-59 and 4.8.0-34. Either because #1655842 is not fixed, or it is only fixed for certain kinds of workloads. These are the changes I'm seeing in our memory graphs between 4.4.0-59 and 4.4.0-63/4.8.0-34. The symptoms I'm seeing are: Upgrading 4.4.0-57 -> 4.4.0-59: - /proc/meminfo:Buffers: Up from 9GB to 15GB - /proc/meminfo:Cached: Up from 5GB to 10GB - /proc/meminfo:SReclaimable: Down from 15GB to 5GB - /proc/meminfo:SUnreclaim: Staying at 50MB Upgrading 4.4.0-57 -> 4.4.0-63: - /proc/meminfo:Buffers: Up from 9GB to 26GB - /proc/meminfo:Cached: Down from 5GB to 300MB - /proc/meminfo:SReclaimable: Down from 15GB to 2GB - /proc/meminfo:SUnreclaim: Down from 50MB to 30MB Upgrading 4.4.0-57 -> 4.8.0-34: - /proc/meminfo:Buffers: Up from 9GB to 14GB - /proc/meminfo:Cached: Down from 5GB to 2GB - /proc/meminfo:SReclaimable: Down from 15GB to 14GB - /proc/meminfo:SUnreclaim: Staying at 50MB Setting vm.vfs_cache_pressure = 300 seems to have a positive effect of not causing OOMs. Downgrading to 4.4.0-57 also works. Will also note that I haven't had a definitive OOM in 4.4.0-63. But the shift in memory usage is far too much from what I expect to be normal on these particular servers where I'm experiencing crashes. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-63-generic 4.4.0-63.84 ProcVersionSignature: Ubuntu 4.4.0-63.84-generic 4.4.44 Uname: Linux 4.4.0-63-generic x86_64 AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.5 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' Date: Mon Feb 20 16:15:56 2017 InstallationDate: Installed on 2012-06-04 (1721 days ago) InstallationMedia: IwConfig: lono wireless extensions. eth0 no wireless extensions. Lsusb: Error: [Errno 2] No such file or directory: 'lsusb' MachineType: System manufacturer System Product Name PciMultimedia: ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-63-generic root=UUID=b790930f-ad81-4b27-a353-a4b3d6a29007 ro nomodeset nomdmonddf nomdmonisw RelatedPackageVersions: linux-restricted-modules-4.4.0-63-generic N/A linux-backports-modules-4.4.0-63-generic N/A linux-firmware1.157.8 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: Upgraded to xenial on 2017-02-16 (4 days ago) dmi.bios.date: 10/17/2011 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: 1106 dmi.board.asset.tag: To be filled by O.E.M. dmi.board.name: P8H67-M PRO dmi.board.vendor: ASUSTeK Computer INC. dmi.board.version: Rev 1.xx dmi.chassis.asset.tag: Asset-1234567890 dmi.chassis.type: 3 dmi.chassis.vendor: Chassis Manufacture dmi.chassis.version: Chassis Version dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1106:bd10/17/2011:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKComputerINC.:rnP8H67-MPRO:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion: dmi.product.name: System Product Name dmi.product.version: System Version dmi.sys.vendor: System manufacturer To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1666260] Re: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34
** Tags added: kernel-fixed-upstream -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1666260 Title: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34 Status in linux package in Ubuntu: Confirmed Bug description: Same as #1655842 - Started seeing oom-killer on multiple servers upgraded to 4.4.0-59. Unlike #1655842, also seeing the same oom-killer on multiple servers updated to 4.8.0-34. First upgraded them all 4.8 servers 4.8.0-36, then downgraded a few to 4.4.0-63. I am seeing an even more pronounced change in the memory usage, so I can only assume that 4.4.0-63 is also bugged with the same problem as 4.4.0-59 and 4.8.0-34. Either because #1655842 is not fixed, or it is only fixed for certain kinds of workloads. These are the changes I'm seeing in our memory graphs between 4.4.0-59 and 4.4.0-63/4.8.0-34. The symptoms I'm seeing are: Upgrading 4.4.0-57 -> 4.4.0-59: - /proc/meminfo:Buffers: Up from 9GB to 15GB - /proc/meminfo:Cached: Up from 5GB to 10GB - /proc/meminfo:SReclaimable: Down from 15GB to 5GB - /proc/meminfo:SUnreclaim: Staying at 50MB Upgrading 4.4.0-57 -> 4.4.0-63: - /proc/meminfo:Buffers: Up from 9GB to 26GB - /proc/meminfo:Cached: Down from 5GB to 300MB - /proc/meminfo:SReclaimable: Down from 15GB to 2GB - /proc/meminfo:SUnreclaim: Down from 50MB to 30MB Upgrading 4.4.0-57 -> 4.8.0-34: - /proc/meminfo:Buffers: Up from 9GB to 14GB - /proc/meminfo:Cached: Down from 5GB to 2GB - /proc/meminfo:SReclaimable: Down from 15GB to 14GB - /proc/meminfo:SUnreclaim: Staying at 50MB Setting vm.vfs_cache_pressure = 300 seems to have a positive effect of not causing OOMs. Downgrading to 4.4.0-57 also works. Will also note that I haven't had a definitive OOM in 4.4.0-63. But the shift in memory usage is far too much from what I expect to be normal on these particular servers where I'm experiencing crashes. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-63-generic 4.4.0-63.84 ProcVersionSignature: Ubuntu 4.4.0-63.84-generic 4.4.44 Uname: Linux 4.4.0-63-generic x86_64 AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.5 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' Date: Mon Feb 20 16:15:56 2017 InstallationDate: Installed on 2012-06-04 (1721 days ago) InstallationMedia: IwConfig: lono wireless extensions. eth0 no wireless extensions. Lsusb: Error: [Errno 2] No such file or directory: 'lsusb' MachineType: System manufacturer System Product Name PciMultimedia: ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-63-generic root=UUID=b790930f-ad81-4b27-a353-a4b3d6a29007 ro nomodeset nomdmonddf nomdmonisw RelatedPackageVersions: linux-restricted-modules-4.4.0-63-generic N/A linux-backports-modules-4.4.0-63-generic N/A linux-firmware1.157.8 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: Upgraded to xenial on 2017-02-16 (4 days ago) dmi.bios.date: 10/17/2011 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: 1106 dmi.board.asset.tag: To be filled by O.E.M. dmi.board.name: P8H67-M PRO dmi.board.vendor: ASUSTeK Computer INC. dmi.board.version: Rev 1.xx dmi.chassis.asset.tag: Asset-1234567890 dmi.chassis.type: 3 dmi.chassis.vendor: Chassis Manufacture dmi.chassis.version: Chassis Version dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1106:bd10/17/2011:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKComputerINC.:rnP8H67-MPRO:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion: dmi.product.name: System Product Name dmi.product.version: System Version dmi.sys.vendor: System manufacturer To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Bug is prevalent on 4.8 kernels too. --- [251529.693133] CPU: 3 PID: 1547 Comm: icinga2 Not tainted 4.8.0-34-generic #36~16.04.1-Ubuntu [251529.693134] Hardware name: MSI MS-7823/B85M-G43 (MS-7823), BIOS V3.14B3 06/23/2014 [251529.693135] 0286 e9fa7ede 95c3f774bb38 ab62d7b3 [251529.693137] 95c3f774bcc8 95c3f23bd700 95c3f774bba0 ab42e9bb [251529.693138] 95c3f774bb40 [251529.693140] Call Trace: [251529.693145] [] dump_stack+0x63/0x90 [251529.693148] [] dump_header+0x5c/0x1dc [251529.693151] [] oom_kill_process+0x226/0x3f0 [251529.693153] [] out_of_memory+0x35a/0x3f0 [251529.693155] [] __alloc_pages_slowpath+0x9fb/0xa20 [251529.693157] [] __alloc_pages_nodemask+0x2ba/0x300 [251529.693160] [] copy_process.part.30+0x146/0x1b50 [251529.693162] [] ? sock_recvmsg+0x3d/0x50 [251529.693163] [] ? SYSC_recvfrom+0xda/0x150 [251529.693164] [] _do_fork+0xe7/0x3f0 [251529.693166] [] ? __sys_recvmsg+0x51/0x90 [251529.693168] [] SyS_clone+0x19/0x20 [251529.693170] [] do_syscall_64+0x5e/0xc0 [251529.693174] [] entry_SYSCALL64_slow_path+0x25/0x25 [251529.693174] Mem-Info: [251529.693177] active_anon:339565 inactive_anon:133615 isolated_anon:0 active_file:3938458 inactive_file:328087 isolated_file:0 unevictable:8 dirty:200 writeback:37 unstable:0 slab_reclaimable:3365424 slab_unreclaimable:16102 mapped:9114 shmem:1459 pagetables:2462 bounce:0 free:49449 free_pcp:32 free_cma:0 --- Had 5 servers knocked out over the weekend. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 Status in linux package in Ubuntu: Fix Released Status in linux source package in Xenial: Fix Released Bug description: I recently replaced some Xenial servers, and started experiencing "Out of memory" problems with the default kernel. We bake Amazon AMIs based on an official Ubuntu-provided image (ami- e6b58e85, in ap-southeast-2, from https://cloud- images.ubuntu.com/locator/ec2/). Previous versions of our AMI included "4.4.0-57-generic", but the latest version picked up "4.4.0-59-generic" as part of a "dist-upgrade". Instances booted using the new AMI have been using more memory, and experiencing OOM issues - sometimes during boot, and sometimes a while afterwards. An example from the system log is: [ 130.113411] cloud-init[1560]: Cloud-init v. 0.7.8 running 'modules:final' at Wed, 11 Jan 2017 22:07:53 +. Up 29.28 seconds. [ 130.124219] cloud-init[1560]: Cloud-init v. 0.7.8 finished at Wed, 11 Jan 2017 22:09:35 +. Datasource DataSourceEc2. Up 130.09 seconds [29871.137128] Out of memory: Kill process 2920 (ruby) score 107 or sacrifice child [29871.140816] Killed process 2920 (ruby) total-vm:675048kB, anon-rss:51184kB, file-rss:2164kB [29871.449209] Out of memory: Kill process 3257 (splunkd) score 97 or sacrifice child [29871.453282] Killed process 3258 (splunkd) total-vm:66272kB, anon-rss:6676kB, file-rss:0kB [29871.677910] Out of memory: Kill process 2647 (fluentd) score 51 or sacrifice child [29871.681872] Killed process 2647 (fluentd) total-vm:117944kB, anon-rss:23956kB, file-rss:1356kB I have a hunch that this may be related to the fix for https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1647400, introduced in linux (4.4.0-58.79). ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-59-generic 4.4.0-59.80 ProcVersionSignature: User Name 4.4.0-59.80-generic 4.4.35 Uname: Linux 4.4.0-59-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Jan 12 06:29 seq crw-rw 1 root audio 116, 33 Jan 12 06:29 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.4 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: Date: Thu Jan 12 06:38:45 2017 Ec2AMI: ami-0f93966c Ec2AMIManifest: (unknown) Ec2AvailabilityZone: ap-southeast-2a Ec2InstanceType: t2.nano Ec2Kernel: unavailable Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lsusb: Error: command ['lsusb'] failed with exit code 1: MachineType: Xen HVM domU PciMultimedia: ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 cirrusdrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-59-generic root=UUID=fb0fef08-f3c5-40bf-9776-f7ba00fe72be ro console=tty1 console=ttyS0 RelatedPackageVersions: linux-restricted-modules-4.4.0-59-generic N/A linux-bac
[Kernel-packages] [Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
Yes they are, I'm seeing the same exorbitant memory usage that we had on 4.4.0-58 as am currently getting on 4.8.0-36. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 Status in linux package in Ubuntu: Fix Released Status in linux source package in Xenial: Fix Released Bug description: I recently replaced some Xenial servers, and started experiencing "Out of memory" problems with the default kernel. We bake Amazon AMIs based on an official Ubuntu-provided image (ami- e6b58e85, in ap-southeast-2, from https://cloud- images.ubuntu.com/locator/ec2/). Previous versions of our AMI included "4.4.0-57-generic", but the latest version picked up "4.4.0-59-generic" as part of a "dist-upgrade". Instances booted using the new AMI have been using more memory, and experiencing OOM issues - sometimes during boot, and sometimes a while afterwards. An example from the system log is: [ 130.113411] cloud-init[1560]: Cloud-init v. 0.7.8 running 'modules:final' at Wed, 11 Jan 2017 22:07:53 +. Up 29.28 seconds. [ 130.124219] cloud-init[1560]: Cloud-init v. 0.7.8 finished at Wed, 11 Jan 2017 22:09:35 +. Datasource DataSourceEc2. Up 130.09 seconds [29871.137128] Out of memory: Kill process 2920 (ruby) score 107 or sacrifice child [29871.140816] Killed process 2920 (ruby) total-vm:675048kB, anon-rss:51184kB, file-rss:2164kB [29871.449209] Out of memory: Kill process 3257 (splunkd) score 97 or sacrifice child [29871.453282] Killed process 3258 (splunkd) total-vm:66272kB, anon-rss:6676kB, file-rss:0kB [29871.677910] Out of memory: Kill process 2647 (fluentd) score 51 or sacrifice child [29871.681872] Killed process 2647 (fluentd) total-vm:117944kB, anon-rss:23956kB, file-rss:1356kB I have a hunch that this may be related to the fix for https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1647400, introduced in linux (4.4.0-58.79). ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-59-generic 4.4.0-59.80 ProcVersionSignature: User Name 4.4.0-59.80-generic 4.4.35 Uname: Linux 4.4.0-59-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Jan 12 06:29 seq crw-rw 1 root audio 116, 33 Jan 12 06:29 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.4 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: Date: Thu Jan 12 06:38:45 2017 Ec2AMI: ami-0f93966c Ec2AMIManifest: (unknown) Ec2AvailabilityZone: ap-southeast-2a Ec2InstanceType: t2.nano Ec2Kernel: unavailable Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lsusb: Error: command ['lsusb'] failed with exit code 1: MachineType: Xen HVM domU PciMultimedia: ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 cirrusdrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-59-generic root=UUID=fb0fef08-f3c5-40bf-9776-f7ba00fe72be ro console=tty1 console=ttyS0 RelatedPackageVersions: linux-restricted-modules-4.4.0-59-generic N/A linux-backports-modules-4.4.0-59-generic N/A linux-firmware1.157.6 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 12/09/2016 dmi.bios.vendor: Xen dmi.bios.version: 4.2.amazon dmi.chassis.type: 1 dmi.chassis.vendor: Xen dmi.modalias: dmi:bvnXen:bvr4.2.amazon:bd12/09/2016:svnXen:pnHVMdomU:pvr4.2.amazon:cvnXen:ct1:cvr: dmi.product.name: HVM domU dmi.product.version: 4.2.amazon dmi.sys.vendor: Xen To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
This is the reported /proc/meminfo Buffers usage for 4 different kernel versions. We got the same OOM call traces on both 4.4.0-58 and 4.8.0-34, I highly doubt that to be a coincidence. ** Attachment added: "meminfo-buffers.png" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+attachment/4822727/+files/meminfo-buffers.png -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 Status in linux package in Ubuntu: Fix Released Status in linux source package in Xenial: Fix Released Bug description: I recently replaced some Xenial servers, and started experiencing "Out of memory" problems with the default kernel. We bake Amazon AMIs based on an official Ubuntu-provided image (ami- e6b58e85, in ap-southeast-2, from https://cloud- images.ubuntu.com/locator/ec2/). Previous versions of our AMI included "4.4.0-57-generic", but the latest version picked up "4.4.0-59-generic" as part of a "dist-upgrade". Instances booted using the new AMI have been using more memory, and experiencing OOM issues - sometimes during boot, and sometimes a while afterwards. An example from the system log is: [ 130.113411] cloud-init[1560]: Cloud-init v. 0.7.8 running 'modules:final' at Wed, 11 Jan 2017 22:07:53 +. Up 29.28 seconds. [ 130.124219] cloud-init[1560]: Cloud-init v. 0.7.8 finished at Wed, 11 Jan 2017 22:09:35 +. Datasource DataSourceEc2. Up 130.09 seconds [29871.137128] Out of memory: Kill process 2920 (ruby) score 107 or sacrifice child [29871.140816] Killed process 2920 (ruby) total-vm:675048kB, anon-rss:51184kB, file-rss:2164kB [29871.449209] Out of memory: Kill process 3257 (splunkd) score 97 or sacrifice child [29871.453282] Killed process 3258 (splunkd) total-vm:66272kB, anon-rss:6676kB, file-rss:0kB [29871.677910] Out of memory: Kill process 2647 (fluentd) score 51 or sacrifice child [29871.681872] Killed process 2647 (fluentd) total-vm:117944kB, anon-rss:23956kB, file-rss:1356kB I have a hunch that this may be related to the fix for https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1647400, introduced in linux (4.4.0-58.79). ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-59-generic 4.4.0-59.80 ProcVersionSignature: User Name 4.4.0-59.80-generic 4.4.35 Uname: Linux 4.4.0-59-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Jan 12 06:29 seq crw-rw 1 root audio 116, 33 Jan 12 06:29 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.4 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: Date: Thu Jan 12 06:38:45 2017 Ec2AMI: ami-0f93966c Ec2AMIManifest: (unknown) Ec2AvailabilityZone: ap-southeast-2a Ec2InstanceType: t2.nano Ec2Kernel: unavailable Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lsusb: Error: command ['lsusb'] failed with exit code 1: MachineType: Xen HVM domU PciMultimedia: ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 cirrusdrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-59-generic root=UUID=fb0fef08-f3c5-40bf-9776-f7ba00fe72be ro console=tty1 console=ttyS0 RelatedPackageVersions: linux-restricted-modules-4.4.0-59-generic N/A linux-backports-modules-4.4.0-59-generic N/A linux-firmware1.157.6 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 12/09/2016 dmi.bios.vendor: Xen dmi.bios.version: 4.2.amazon dmi.chassis.type: 1 dmi.chassis.vendor: Xen dmi.modalias: dmi:bvnXen:bvr4.2.amazon:bd12/09/2016:svnXen:pnHVMdomU:pvr4.2.amazon:cvnXen:ct1:cvr: dmi.product.name: HVM domU dmi.product.version: 4.2.amazon dmi.sys.vendor: Xen To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1655842] Re: "Out of memory" errors after upgrade to 4.4.0-59
The OOM fixes were introduced in 4.4.0-58 according to the changelog, but sure. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1655842 Title: "Out of memory" errors after upgrade to 4.4.0-59 Status in linux package in Ubuntu: Fix Released Status in linux source package in Xenial: Fix Released Bug description: I recently replaced some Xenial servers, and started experiencing "Out of memory" problems with the default kernel. We bake Amazon AMIs based on an official Ubuntu-provided image (ami- e6b58e85, in ap-southeast-2, from https://cloud- images.ubuntu.com/locator/ec2/). Previous versions of our AMI included "4.4.0-57-generic", but the latest version picked up "4.4.0-59-generic" as part of a "dist-upgrade". Instances booted using the new AMI have been using more memory, and experiencing OOM issues - sometimes during boot, and sometimes a while afterwards. An example from the system log is: [ 130.113411] cloud-init[1560]: Cloud-init v. 0.7.8 running 'modules:final' at Wed, 11 Jan 2017 22:07:53 +. Up 29.28 seconds. [ 130.124219] cloud-init[1560]: Cloud-init v. 0.7.8 finished at Wed, 11 Jan 2017 22:09:35 +. Datasource DataSourceEc2. Up 130.09 seconds [29871.137128] Out of memory: Kill process 2920 (ruby) score 107 or sacrifice child [29871.140816] Killed process 2920 (ruby) total-vm:675048kB, anon-rss:51184kB, file-rss:2164kB [29871.449209] Out of memory: Kill process 3257 (splunkd) score 97 or sacrifice child [29871.453282] Killed process 3258 (splunkd) total-vm:66272kB, anon-rss:6676kB, file-rss:0kB [29871.677910] Out of memory: Kill process 2647 (fluentd) score 51 or sacrifice child [29871.681872] Killed process 2647 (fluentd) total-vm:117944kB, anon-rss:23956kB, file-rss:1356kB I have a hunch that this may be related to the fix for https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1647400, introduced in linux (4.4.0-58.79). ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-59-generic 4.4.0-59.80 ProcVersionSignature: User Name 4.4.0-59.80-generic 4.4.35 Uname: Linux 4.4.0-59-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Jan 12 06:29 seq crw-rw 1 root audio 116, 33 Jan 12 06:29 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.4 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: Date: Thu Jan 12 06:38:45 2017 Ec2AMI: ami-0f93966c Ec2AMIManifest: (unknown) Ec2AvailabilityZone: ap-southeast-2a Ec2InstanceType: t2.nano Ec2Kernel: unavailable Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lsusb: Error: command ['lsusb'] failed with exit code 1: MachineType: Xen HVM domU PciMultimedia: ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 cirrusdrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-59-generic root=UUID=fb0fef08-f3c5-40bf-9776-f7ba00fe72be ro console=tty1 console=ttyS0 RelatedPackageVersions: linux-restricted-modules-4.4.0-59-generic N/A linux-backports-modules-4.4.0-59-generic N/A linux-firmware1.157.6 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 12/09/2016 dmi.bios.vendor: Xen dmi.bios.version: 4.2.amazon dmi.chassis.type: 1 dmi.chassis.vendor: Xen dmi.modalias: dmi:bvnXen:bvr4.2.amazon:bd12/09/2016:svnXen:pnHVMdomU:pvr4.2.amazon:cvnXen:ct1:cvr: dmi.product.name: HVM domU dmi.product.version: 4.2.amazon dmi.sys.vendor: Xen To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1666260] [NEW] "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34
Public bug reported: Same as #1655842 - Started seeing oom-killer on multiple servers upgraded to 4.4.0-59. Unlike #1655842, also seeing the same oom-killer on multiple servers updated to 4.8.0-34. First upgraded them all 4.8 servers 4.8.0-36, then downgraded a few to 4.4.0-63. I am seeing no change in the memory usage, so I can only assume that 4.4.0-63 is also bugged with the same problem as 4.4.0-59 and 4.8.0-34. Either because #1655842 is not fixed, or it is only fixed for certain kinds of workloads. However, on closer inspection, there is definitely a notable difference between 4.4.0-59 and 4.4.0-63/4.8.0-34. The symptoms I'm seeing are: Upgrading 4.4.0-57 -> 4.4.0-59: - /proc/meminfo:Buffers: Up from 9GB to 15GB - /proc/meminfo:Cached: Up from 5GB to 10GB - /proc/meminfo:SReclaimable: Down from 15GB to 5GB. - /proc/meminfo:SUnreclaim: No notable change Upgrading 4.4.0-57 -> 4.8.0-34: - /proc/meminfo:Buffers: Up from 9GB to 14GB - /proc/meminfo:Cached: Down from 5GB to 2GB - /proc/meminfo:SReclaimable: Down from 15GB to 14GB. - /proc/meminfo:SUnreclaim: No notable change Downgrading 4.8.0-34 -> 4.4.0-63: (however uptime is only 1 hour) - /proc/meminfo:Buffers: No notable change - /proc/meminfo:Cached: No notable change - /proc/meminfo:SReclaimable: No notable change - /proc/meminfo:SUnreclaim: No notable change Setting vm.vfs_cache_pressure = 300 seems to have a positive effect of not causing OOMs. Downgrading to 4.4.0-57 also works. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-63-generic 4.4.0-63.84 ProcVersionSignature: Ubuntu 4.4.0-63.84-generic 4.4.44 Uname: Linux 4.4.0-63-generic x86_64 AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.5 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' Date: Mon Feb 20 16:15:56 2017 InstallationDate: Installed on 2012-06-04 (1721 days ago) InstallationMedia: IwConfig: lono wireless extensions. eth0 no wireless extensions. Lsusb: Error: [Errno 2] No such file or directory: 'lsusb' MachineType: System manufacturer System Product Name PciMultimedia: ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-63-generic root=UUID=b790930f-ad81-4b27-a353-a4b3d6a29007 ro nomodeset nomdmonddf nomdmonisw RelatedPackageVersions: linux-restricted-modules-4.4.0-63-generic N/A linux-backports-modules-4.4.0-63-generic N/A linux-firmware1.157.8 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: Upgraded to xenial on 2017-02-16 (4 days ago) dmi.bios.date: 10/17/2011 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: 1106 dmi.board.asset.tag: To be filled by O.E.M. dmi.board.name: P8H67-M PRO dmi.board.vendor: ASUSTeK Computer INC. dmi.board.version: Rev 1.xx dmi.chassis.asset.tag: Asset-1234567890 dmi.chassis.type: 3 dmi.chassis.vendor: Chassis Manufacture dmi.chassis.version: Chassis Version dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1106:bd10/17/2011:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKComputerINC.:rnP8H67-MPRO:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion: dmi.product.name: System Product Name dmi.product.version: System Version dmi.sys.vendor: System manufacturer ** Affects: linux (Ubuntu) Importance: Undecided Status: New ** Tags: amd64 apport-bug xenial -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1666260 Title: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34 Status in linux package in Ubuntu: New Bug description: Same as #1655842 - Started seeing oom-killer on multiple servers upgraded to 4.4.0-59. Unlike #1655842, also seeing the same oom-killer on multiple servers updated to 4.8.0-34. First upgraded them all 4.8 servers 4.8.0-36, then downgraded a few to 4.4.0-63. I am seeing no change in the memory usage, so I can only assume that 4.4.0-63 is also bugged with the same problem as 4.4.0-59 and 4.8.0-34. Either because #1655842 is not fixed, or it is only fixed for certain kinds of workloads. However, on closer inspection, there is definitely a notable difference between 4.4.0-59 and 4.4.0-63/4.8.0-34. The symptoms I'm seeing are: Upgrading 4.4.0-57 -> 4.4.0-59: - /proc/meminfo:Buffers: Up from 9GB to 15GB - /proc/meminfo:Cached: Up from 5GB to 10GB - /proc/meminfo:SReclaimable: Down from 15GB to 5GB. - /proc/meminfo:SUnreclaim: No notable change Upgrading 4.4.0-57 -> 4.8.0-34: - /proc/meminfo:Buffers: Up from 9GB to 14GB - /proc/meminfo:Cached: Down from 5GB to 2GB - /proc/meminfo:SReclaimable: Down from 15GB to 14GB. - /proc/meminfo:SUnreclaim: No notabl
[Kernel-packages] [Bug 1666260] Re: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34
Attaching the call trace for 4.4.0-59, this is from Jan 16th. ** Attachment added: "linux-44059.oom.txt" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+attachment/4822833/+files/linux-44059.oom.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1666260 Title: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34 Status in linux package in Ubuntu: Confirmed Bug description: Same as #1655842 - Started seeing oom-killer on multiple servers upgraded to 4.4.0-59. Unlike #1655842, also seeing the same oom-killer on multiple servers updated to 4.8.0-34. First upgraded them all 4.8 servers 4.8.0-36, then downgraded a few to 4.4.0-63. I am seeing no change in the memory usage, so I can only assume that 4.4.0-63 is also bugged with the same problem as 4.4.0-59 and 4.8.0-34. Either because #1655842 is not fixed, or it is only fixed for certain kinds of workloads. However, on closer inspection, there is definitely a notable difference between 4.4.0-59 and 4.4.0-63/4.8.0-34. The symptoms I'm seeing are: Upgrading 4.4.0-57 -> 4.4.0-59: - /proc/meminfo:Buffers: Up from 9GB to 15GB - /proc/meminfo:Cached: Up from 5GB to 10GB - /proc/meminfo:SReclaimable: Down from 15GB to 5GB. - /proc/meminfo:SUnreclaim: No notable change Upgrading 4.4.0-57 -> 4.8.0-34: - /proc/meminfo:Buffers: Up from 9GB to 14GB - /proc/meminfo:Cached: Down from 5GB to 2GB - /proc/meminfo:SReclaimable: Down from 15GB to 14GB. - /proc/meminfo:SUnreclaim: No notable change Downgrading 4.8.0-34 -> 4.4.0-63: (however uptime is only 1 hour) - /proc/meminfo:Buffers: No notable change - /proc/meminfo:Cached: No notable change - /proc/meminfo:SReclaimable: No notable change - /proc/meminfo:SUnreclaim: No notable change Setting vm.vfs_cache_pressure = 300 seems to have a positive effect of not causing OOMs. Downgrading to 4.4.0-57 also works. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-63-generic 4.4.0-63.84 ProcVersionSignature: Ubuntu 4.4.0-63.84-generic 4.4.44 Uname: Linux 4.4.0-63-generic x86_64 AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.5 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' Date: Mon Feb 20 16:15:56 2017 InstallationDate: Installed on 2012-06-04 (1721 days ago) InstallationMedia: IwConfig: lono wireless extensions. eth0 no wireless extensions. Lsusb: Error: [Errno 2] No such file or directory: 'lsusb' MachineType: System manufacturer System Product Name PciMultimedia: ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-63-generic root=UUID=b790930f-ad81-4b27-a353-a4b3d6a29007 ro nomodeset nomdmonddf nomdmonisw RelatedPackageVersions: linux-restricted-modules-4.4.0-63-generic N/A linux-backports-modules-4.4.0-63-generic N/A linux-firmware1.157.8 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: Upgraded to xenial on 2017-02-16 (4 days ago) dmi.bios.date: 10/17/2011 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: 1106 dmi.board.asset.tag: To be filled by O.E.M. dmi.board.name: P8H67-M PRO dmi.board.vendor: ASUSTeK Computer INC. dmi.board.version: Rev 1.xx dmi.chassis.asset.tag: Asset-1234567890 dmi.chassis.type: 3 dmi.chassis.vendor: Chassis Manufacture dmi.chassis.version: Chassis Version dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1106:bd10/17/2011:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKComputerINC.:rnP8H67-MPRO:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion: dmi.product.name: System Product Name dmi.product.version: System Version dmi.sys.vendor: System manufacturer To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1666260] Re: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34
Attaching the call trace for 4.8.0-34, this is from Feb 19th. ** Attachment added: "linux-48034.oom.txt" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+attachment/4822834/+files/linux-48034.oom.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1666260 Title: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34 Status in linux package in Ubuntu: Confirmed Bug description: Same as #1655842 - Started seeing oom-killer on multiple servers upgraded to 4.4.0-59. Unlike #1655842, also seeing the same oom-killer on multiple servers updated to 4.8.0-34. First upgraded them all 4.8 servers 4.8.0-36, then downgraded a few to 4.4.0-63. I am seeing no change in the memory usage, so I can only assume that 4.4.0-63 is also bugged with the same problem as 4.4.0-59 and 4.8.0-34. Either because #1655842 is not fixed, or it is only fixed for certain kinds of workloads. However, on closer inspection, there is definitely a notable difference between 4.4.0-59 and 4.4.0-63/4.8.0-34. The symptoms I'm seeing are: Upgrading 4.4.0-57 -> 4.4.0-59: - /proc/meminfo:Buffers: Up from 9GB to 15GB - /proc/meminfo:Cached: Up from 5GB to 10GB - /proc/meminfo:SReclaimable: Down from 15GB to 5GB. - /proc/meminfo:SUnreclaim: No notable change Upgrading 4.4.0-57 -> 4.8.0-34: - /proc/meminfo:Buffers: Up from 9GB to 14GB - /proc/meminfo:Cached: Down from 5GB to 2GB - /proc/meminfo:SReclaimable: Down from 15GB to 14GB. - /proc/meminfo:SUnreclaim: No notable change Downgrading 4.8.0-34 -> 4.4.0-63: (however uptime is only 1 hour) - /proc/meminfo:Buffers: No notable change - /proc/meminfo:Cached: No notable change - /proc/meminfo:SReclaimable: No notable change - /proc/meminfo:SUnreclaim: No notable change Setting vm.vfs_cache_pressure = 300 seems to have a positive effect of not causing OOMs. Downgrading to 4.4.0-57 also works. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-63-generic 4.4.0-63.84 ProcVersionSignature: Ubuntu 4.4.0-63.84-generic 4.4.44 Uname: Linux 4.4.0-63-generic x86_64 AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.5 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' Date: Mon Feb 20 16:15:56 2017 InstallationDate: Installed on 2012-06-04 (1721 days ago) InstallationMedia: IwConfig: lono wireless extensions. eth0 no wireless extensions. Lsusb: Error: [Errno 2] No such file or directory: 'lsusb' MachineType: System manufacturer System Product Name PciMultimedia: ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-63-generic root=UUID=b790930f-ad81-4b27-a353-a4b3d6a29007 ro nomodeset nomdmonddf nomdmonisw RelatedPackageVersions: linux-restricted-modules-4.4.0-63-generic N/A linux-backports-modules-4.4.0-63-generic N/A linux-firmware1.157.8 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: Upgraded to xenial on 2017-02-16 (4 days ago) dmi.bios.date: 10/17/2011 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: 1106 dmi.board.asset.tag: To be filled by O.E.M. dmi.board.name: P8H67-M PRO dmi.board.vendor: ASUSTeK Computer INC. dmi.board.version: Rev 1.xx dmi.chassis.asset.tag: Asset-1234567890 dmi.chassis.type: 3 dmi.chassis.vendor: Chassis Manufacture dmi.chassis.version: Chassis Version dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1106:bd10/17/2011:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKComputerINC.:rnP8H67-MPRO:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion: dmi.product.name: System Product Name dmi.product.version: System Version dmi.sys.vendor: System manufacturer To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1666260] Re: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34
I think this is currently in the middle of happening right now on one of the servers running 4.4.0-63. SReclaimable is down from 15GB to 5GB, and Buffers is has been slowly rising over the last hour from 12GB to approaching 25GB. --- # cat /proc/meminfo MemTotal: 32856000 kB MemFree: 1331808 kB MemAvailable: 30994136 kB Buffers:24951992 kB Cached: 388368 kB SwapCached: 188 kB Active: 22271936 kB Inactive:4383080 kB Active(anon): 786576 kB Inactive(anon): 534244 kB Active(file): 21485360 kB Inactive(file): 3848836 kB Unevictable: 32 kB Mlocked: 32 kB SwapTotal: 33553332 kB SwapFree: 33552460 kB Dirty: 2048 kB Writeback: 0 kB AnonPages: 1314500 kB Mapped:42820 kB Shmem: 6164 kB Slab:4765816 kB SReclaimable:4730460 kB SUnreclaim:35356 kB KernelStack:4512 kB PageTables:10164 kB NFS_Unstable: 0 kB Bounce:0 kB WritebackTmp: 0 kB CommitLimit:49981332 kB Committed_AS:1352800 kB VmallocTotal: 34359738367 kB VmallocUsed: 0 kB VmallocChunk: 0 kB HardwareCorrupted: 0 kB AnonHugePages:856064 kB CmaTotal: 0 kB CmaFree: 0 kB HugePages_Total: 0 HugePages_Free:0 HugePages_Rsvd:0 HugePages_Surp:0 Hugepagesize: 2048 kB DirectMap4k: 188008 kB DirectMap2M:33275904 kB --- -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1666260 Title: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34 Status in linux package in Ubuntu: Confirmed Bug description: Same as #1655842 - Started seeing oom-killer on multiple servers upgraded to 4.4.0-59. Unlike #1655842, also seeing the same oom-killer on multiple servers updated to 4.8.0-34. First upgraded them all 4.8 servers 4.8.0-36, then downgraded a few to 4.4.0-63. I am seeing no change in the memory usage, so I can only assume that 4.4.0-63 is also bugged with the same problem as 4.4.0-59 and 4.8.0-34. Either because #1655842 is not fixed, or it is only fixed for certain kinds of workloads. However, on closer inspection, there is definitely a notable difference between 4.4.0-59 and 4.4.0-63/4.8.0-34. The symptoms I'm seeing are: Upgrading 4.4.0-57 -> 4.4.0-59: - /proc/meminfo:Buffers: Up from 9GB to 15GB - /proc/meminfo:Cached: Up from 5GB to 10GB - /proc/meminfo:SReclaimable: Down from 15GB to 5GB. - /proc/meminfo:SUnreclaim: No notable change Upgrading 4.4.0-57 -> 4.8.0-34: - /proc/meminfo:Buffers: Up from 9GB to 14GB - /proc/meminfo:Cached: Down from 5GB to 2GB - /proc/meminfo:SReclaimable: Down from 15GB to 14GB. - /proc/meminfo:SUnreclaim: No notable change Downgrading 4.8.0-34 -> 4.4.0-63: (however uptime is only 1 hour) - /proc/meminfo:Buffers: No notable change - /proc/meminfo:Cached: No notable change - /proc/meminfo:SReclaimable: No notable change - /proc/meminfo:SUnreclaim: No notable change Setting vm.vfs_cache_pressure = 300 seems to have a positive effect of not causing OOMs. Downgrading to 4.4.0-57 also works. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-63-generic 4.4.0-63.84 ProcVersionSignature: Ubuntu 4.4.0-63.84-generic 4.4.44 Uname: Linux 4.4.0-63-generic x86_64 AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.5 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' Date: Mon Feb 20 16:15:56 2017 InstallationDate: Installed on 2012-06-04 (1721 days ago) InstallationMedia: IwConfig: lono wireless extensions. eth0 no wireless extensions. Lsusb: Error: [Errno 2] No such file or directory: 'lsusb' MachineType: System manufacturer System Product Name PciMultimedia: ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-63-generic root=UUID=b790930f-ad81-4b27-a353-a4b3d6a29007 ro nomodeset nomdmonddf nomdmonisw RelatedPackageVersions: linux-restricted-modules-4.4.0-63-generic N/A linux-backports-modules-4.4.0-63-generic N/A linux-firmware1.157.8 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: Upgraded to xenial on 2017-02-16 (4 days ago) dmi.bios.date: 10/17/2011 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: 1106 dmi.board.asset.tag: To be filled by O.E.M. dmi.board.name: P8H67-M PRO dmi.board.vendor: ASUSTeK Computer INC. dmi.board.version: Rev 1.xx dmi.chassis.asset.tag: Asset-1234567890 dmi.chassis.ty
[Kernel-packages] [Bug 1666260] Re: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34
SLAB is still dropping. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1666260 Title: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34 Status in linux package in Ubuntu: Confirmed Bug description: Same as #1655842 - Started seeing oom-killer on multiple servers upgraded to 4.4.0-59. Unlike #1655842, also seeing the same oom-killer on multiple servers updated to 4.8.0-34. First upgraded them all 4.8 servers 4.8.0-36, then downgraded a few to 4.4.0-63. I am seeing no change in the memory usage, so I can only assume that 4.4.0-63 is also bugged with the same problem as 4.4.0-59 and 4.8.0-34. Either because #1655842 is not fixed, or it is only fixed for certain kinds of workloads. However, on closer inspection, there is definitely a notable difference between 4.4.0-59 and 4.4.0-63/4.8.0-34. The symptoms I'm seeing are: Upgrading 4.4.0-57 -> 4.4.0-59: - /proc/meminfo:Buffers: Up from 9GB to 15GB - /proc/meminfo:Cached: Up from 5GB to 10GB - /proc/meminfo:SReclaimable: Down from 15GB to 5GB. - /proc/meminfo:SUnreclaim: No notable change Upgrading 4.4.0-57 -> 4.8.0-34: - /proc/meminfo:Buffers: Up from 9GB to 14GB - /proc/meminfo:Cached: Down from 5GB to 2GB - /proc/meminfo:SReclaimable: Down from 15GB to 14GB. - /proc/meminfo:SUnreclaim: No notable change Downgrading 4.8.0-34 -> 4.4.0-63: (however uptime is only 1 hour) - /proc/meminfo:Buffers: No notable change - /proc/meminfo:Cached: No notable change - /proc/meminfo:SReclaimable: No notable change - /proc/meminfo:SUnreclaim: No notable change Setting vm.vfs_cache_pressure = 300 seems to have a positive effect of not causing OOMs. Downgrading to 4.4.0-57 also works. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-63-generic 4.4.0-63.84 ProcVersionSignature: Ubuntu 4.4.0-63.84-generic 4.4.44 Uname: Linux 4.4.0-63-generic x86_64 AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.5 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' Date: Mon Feb 20 16:15:56 2017 InstallationDate: Installed on 2012-06-04 (1721 days ago) InstallationMedia: IwConfig: lono wireless extensions. eth0 no wireless extensions. Lsusb: Error: [Errno 2] No such file or directory: 'lsusb' MachineType: System manufacturer System Product Name PciMultimedia: ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-63-generic root=UUID=b790930f-ad81-4b27-a353-a4b3d6a29007 ro nomodeset nomdmonddf nomdmonisw RelatedPackageVersions: linux-restricted-modules-4.4.0-63-generic N/A linux-backports-modules-4.4.0-63-generic N/A linux-firmware1.157.8 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: Upgraded to xenial on 2017-02-16 (4 days ago) dmi.bios.date: 10/17/2011 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: 1106 dmi.board.asset.tag: To be filled by O.E.M. dmi.board.name: P8H67-M PRO dmi.board.vendor: ASUSTeK Computer INC. dmi.board.version: Rev 1.xx dmi.chassis.asset.tag: Asset-1234567890 dmi.chassis.type: 3 dmi.chassis.vendor: Chassis Manufacture dmi.chassis.version: Chassis Version dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1106:bd10/17/2011:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKComputerINC.:rnP8H67-MPRO:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion: dmi.product.name: System Product Name dmi.product.version: System Version dmi.sys.vendor: System manufacturer To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1666260] Re: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34
Attaching current slabtop --- # cat /proc/meminfo MemTotal: 32856000 kB MemFree: 381632 kB MemAvailable: 30990804 kB Buffers:28709688 kB Cached: 221664 kB SwapCached: 184 kB Active: 24854232 kB Inactive:5393552 kB Active(anon): 783780 kB Inactive(anon): 538772 kB Active(file): 24070452 kB Inactive(file): 4854780 kB Unevictable: 32 kB Mlocked: 32 kB SwapTotal: 33553332 kB SwapFree: 33552460 kB Dirty: 2472 kB Writeback: 0 kB AnonPages: 1316236 kB Mapped:42768 kB Shmem: 6164 kB Slab:2120976 kB SReclaimable:2086268 kB SUnreclaim:34708 kB KernelStack:4480 kB PageTables: 9944 kB NFS_Unstable: 0 kB Bounce:0 kB WritebackTmp: 0 kB CommitLimit:49981332 kB Committed_AS:1326540 kB VmallocTotal: 34359738367 kB VmallocUsed: 0 kB VmallocChunk: 0 kB HardwareCorrupted: 0 kB AnonHugePages:868352 kB CmaTotal: 0 kB CmaFree: 0 kB HugePages_Total: 0 HugePages_Free:0 HugePages_Rsvd:0 HugePages_Surp:0 Hugepagesize: 2048 kB DirectMap4k: 188008 kB DirectMap2M:33275904 kB ** Attachment added: "slabtop.txt" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+attachment/4822862/+files/slabtop.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1666260 Title: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34 Status in linux package in Ubuntu: Confirmed Bug description: Same as #1655842 - Started seeing oom-killer on multiple servers upgraded to 4.4.0-59. Unlike #1655842, also seeing the same oom-killer on multiple servers updated to 4.8.0-34. First upgraded them all 4.8 servers 4.8.0-36, then downgraded a few to 4.4.0-63. I am seeing no change in the memory usage, so I can only assume that 4.4.0-63 is also bugged with the same problem as 4.4.0-59 and 4.8.0-34. Either because #1655842 is not fixed, or it is only fixed for certain kinds of workloads. However, on closer inspection, there is definitely a notable difference between 4.4.0-59 and 4.4.0-63/4.8.0-34. The symptoms I'm seeing are: Upgrading 4.4.0-57 -> 4.4.0-59: - /proc/meminfo:Buffers: Up from 9GB to 15GB - /proc/meminfo:Cached: Up from 5GB to 10GB - /proc/meminfo:SReclaimable: Down from 15GB to 5GB. - /proc/meminfo:SUnreclaim: No notable change Upgrading 4.4.0-57 -> 4.8.0-34: - /proc/meminfo:Buffers: Up from 9GB to 14GB - /proc/meminfo:Cached: Down from 5GB to 2GB - /proc/meminfo:SReclaimable: Down from 15GB to 14GB. - /proc/meminfo:SUnreclaim: No notable change Downgrading 4.8.0-34 -> 4.4.0-63: (however uptime is only 1 hour) - /proc/meminfo:Buffers: No notable change - /proc/meminfo:Cached: No notable change - /proc/meminfo:SReclaimable: No notable change - /proc/meminfo:SUnreclaim: No notable change Setting vm.vfs_cache_pressure = 300 seems to have a positive effect of not causing OOMs. Downgrading to 4.4.0-57 also works. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-63-generic 4.4.0-63.84 ProcVersionSignature: Ubuntu 4.4.0-63.84-generic 4.4.44 Uname: Linux 4.4.0-63-generic x86_64 AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.5 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' Date: Mon Feb 20 16:15:56 2017 InstallationDate: Installed on 2012-06-04 (1721 days ago) InstallationMedia: IwConfig: lono wireless extensions. eth0 no wireless extensions. Lsusb: Error: [Errno 2] No such file or directory: 'lsusb' MachineType: System manufacturer System Product Name PciMultimedia: ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-63-generic root=UUID=b790930f-ad81-4b27-a353-a4b3d6a29007 ro nomodeset nomdmonddf nomdmonisw RelatedPackageVersions: linux-restricted-modules-4.4.0-63-generic N/A linux-backports-modules-4.4.0-63-generic N/A linux-firmware1.157.8 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: Upgraded to xenial on 2017-02-16 (4 days ago) dmi.bios.date: 10/17/2011 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: 1106 dmi.board.asset.tag: To be filled by O.E.M. dmi.board.name: P8H67-M PRO dmi.board.vendor: ASUSTeK Computer INC. dmi.board.version: Rev 1.xx dmi.chassis.asset.tag: Asset-1234567890 dmi.chassis.type: 3 dmi.chassis.vendor: Chassis Manufacture dmi.chassis.versio
[Kernel-packages] [Bug 1666260] Re: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34
And 4.8.0-36 is affected by bug also. ** Attachment added: "linux-48036.oom.txt" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+attachment/4823499/+files/linux-48036.oom.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1666260 Title: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34 Status in linux package in Ubuntu: Confirmed Bug description: Same as #1655842 - Started seeing oom-killer on multiple servers upgraded to 4.4.0-59. Unlike #1655842, also seeing the same oom-killer on multiple servers updated to 4.8.0-34. First upgraded them all 4.8 servers 4.8.0-36, then downgraded a few to 4.4.0-63. I am seeing no change in the memory usage, so I can only assume that 4.4.0-63 is also bugged with the same problem as 4.4.0-59 and 4.8.0-34. Either because #1655842 is not fixed, or it is only fixed for certain kinds of workloads. However, on closer inspection, there is definitely a notable difference between 4.4.0-59 and 4.4.0-63/4.8.0-34. The symptoms I'm seeing are: Upgrading 4.4.0-57 -> 4.4.0-59: - /proc/meminfo:Buffers: Up from 9GB to 15GB - /proc/meminfo:Cached: Up from 5GB to 10GB - /proc/meminfo:SReclaimable: Down from 15GB to 5GB. - /proc/meminfo:SUnreclaim: No notable change Upgrading 4.4.0-57 -> 4.8.0-34: - /proc/meminfo:Buffers: Up from 9GB to 14GB - /proc/meminfo:Cached: Down from 5GB to 2GB - /proc/meminfo:SReclaimable: Down from 15GB to 14GB. - /proc/meminfo:SUnreclaim: No notable change Downgrading 4.8.0-34 -> 4.4.0-63: (however uptime is only 1 hour) - /proc/meminfo:Buffers: No notable change - /proc/meminfo:Cached: No notable change - /proc/meminfo:SReclaimable: No notable change - /proc/meminfo:SUnreclaim: No notable change Setting vm.vfs_cache_pressure = 300 seems to have a positive effect of not causing OOMs. Downgrading to 4.4.0-57 also works. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-63-generic 4.4.0-63.84 ProcVersionSignature: Ubuntu 4.4.0-63.84-generic 4.4.44 Uname: Linux 4.4.0-63-generic x86_64 AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.5 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' Date: Mon Feb 20 16:15:56 2017 InstallationDate: Installed on 2012-06-04 (1721 days ago) InstallationMedia: IwConfig: lono wireless extensions. eth0 no wireless extensions. Lsusb: Error: [Errno 2] No such file or directory: 'lsusb' MachineType: System manufacturer System Product Name PciMultimedia: ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-63-generic root=UUID=b790930f-ad81-4b27-a353-a4b3d6a29007 ro nomodeset nomdmonddf nomdmonisw RelatedPackageVersions: linux-restricted-modules-4.4.0-63-generic N/A linux-backports-modules-4.4.0-63-generic N/A linux-firmware1.157.8 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: Upgraded to xenial on 2017-02-16 (4 days ago) dmi.bios.date: 10/17/2011 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: 1106 dmi.board.asset.tag: To be filled by O.E.M. dmi.board.name: P8H67-M PRO dmi.board.vendor: ASUSTeK Computer INC. dmi.board.version: Rev 1.xx dmi.chassis.asset.tag: Asset-1234567890 dmi.chassis.type: 3 dmi.chassis.vendor: Chassis Manufacture dmi.chassis.version: Chassis Version dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1106:bd10/17/2011:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKComputerINC.:rnP8H67-MPRO:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion: dmi.product.name: System Product Name dmi.product.version: System Version dmi.sys.vendor: System manufacturer To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1666260] Re: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34
No OOM on servers running 4.4.0-63 just yet, but the memory usage on them is weird to say the least. The Buffer/SLAB ratio is completely different from 4.4.0-57. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1666260 Title: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34 Status in linux package in Ubuntu: Confirmed Bug description: Same as #1655842 - Started seeing oom-killer on multiple servers upgraded to 4.4.0-59. Unlike #1655842, also seeing the same oom-killer on multiple servers updated to 4.8.0-34. First upgraded them all 4.8 servers 4.8.0-36, then downgraded a few to 4.4.0-63. I am seeing no change in the memory usage, so I can only assume that 4.4.0-63 is also bugged with the same problem as 4.4.0-59 and 4.8.0-34. Either because #1655842 is not fixed, or it is only fixed for certain kinds of workloads. However, on closer inspection, there is definitely a notable difference between 4.4.0-59 and 4.4.0-63/4.8.0-34. The symptoms I'm seeing are: Upgrading 4.4.0-57 -> 4.4.0-59: - /proc/meminfo:Buffers: Up from 9GB to 15GB - /proc/meminfo:Cached: Up from 5GB to 10GB - /proc/meminfo:SReclaimable: Down from 15GB to 5GB. - /proc/meminfo:SUnreclaim: No notable change Upgrading 4.4.0-57 -> 4.8.0-34: - /proc/meminfo:Buffers: Up from 9GB to 14GB - /proc/meminfo:Cached: Down from 5GB to 2GB - /proc/meminfo:SReclaimable: Down from 15GB to 14GB. - /proc/meminfo:SUnreclaim: No notable change Downgrading 4.8.0-34 -> 4.4.0-63: (however uptime is only 1 hour) - /proc/meminfo:Buffers: No notable change - /proc/meminfo:Cached: No notable change - /proc/meminfo:SReclaimable: No notable change - /proc/meminfo:SUnreclaim: No notable change Setting vm.vfs_cache_pressure = 300 seems to have a positive effect of not causing OOMs. Downgrading to 4.4.0-57 also works. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-63-generic 4.4.0-63.84 ProcVersionSignature: Ubuntu 4.4.0-63.84-generic 4.4.44 Uname: Linux 4.4.0-63-generic x86_64 AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.5 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' Date: Mon Feb 20 16:15:56 2017 InstallationDate: Installed on 2012-06-04 (1721 days ago) InstallationMedia: IwConfig: lono wireless extensions. eth0 no wireless extensions. Lsusb: Error: [Errno 2] No such file or directory: 'lsusb' MachineType: System manufacturer System Product Name PciMultimedia: ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-63-generic root=UUID=b790930f-ad81-4b27-a353-a4b3d6a29007 ro nomodeset nomdmonddf nomdmonisw RelatedPackageVersions: linux-restricted-modules-4.4.0-63-generic N/A linux-backports-modules-4.4.0-63-generic N/A linux-firmware1.157.8 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: Upgraded to xenial on 2017-02-16 (4 days ago) dmi.bios.date: 10/17/2011 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: 1106 dmi.board.asset.tag: To be filled by O.E.M. dmi.board.name: P8H67-M PRO dmi.board.vendor: ASUSTeK Computer INC. dmi.board.version: Rev 1.xx dmi.chassis.asset.tag: Asset-1234567890 dmi.chassis.type: 3 dmi.chassis.vendor: Chassis Manufacture dmi.chassis.version: Chassis Version dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1106:bd10/17/2011:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKComputerINC.:rnP8H67-MPRO:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion: dmi.product.name: System Product Name dmi.product.version: System Version dmi.sys.vendor: System manufacturer To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1666260] Re: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34
** Description changed: Same as #1655842 - Started seeing oom-killer on multiple servers upgraded to 4.4.0-59. Unlike #1655842, also seeing the same oom-killer on multiple servers updated to 4.8.0-34. First upgraded them all 4.8 servers 4.8.0-36, then downgraded a few to 4.4.0-63. I am seeing no change in the memory usage, so I can only assume that 4.4.0-63 is also bugged with the same problem as 4.4.0-59 and 4.8.0-34. Either because #1655842 is not fixed, or it is only fixed for certain kinds of workloads. However, on closer inspection, there is definitely a notable difference between 4.4.0-59 and 4.4.0-63/4.8.0-34. The symptoms I'm seeing are: Upgrading 4.4.0-57 -> 4.4.0-59: - /proc/meminfo:Buffers: Up from 9GB to 15GB - /proc/meminfo:Cached: Up from 5GB to 10GB - - /proc/meminfo:SReclaimable: Down from 15GB to 5GB. - - /proc/meminfo:SUnreclaim: No notable change + - /proc/meminfo:SReclaimable: Down from 15GB to 5GB + - /proc/meminfo:SUnreclaim: Staying at 50MB + + Upgrading 4.4.0-57 -> 4.4.0-63: + - /proc/meminfo:Buffers: Up from 9GB to 26GB + - /proc/meminfo:Cached: Down from 5GB to 300MB + - /proc/meminfo:SReclaimable: Down from 15GB to 2GB + - /proc/meminfo:SUnreclaim: Down from 50MB to 30MB Upgrading 4.4.0-57 -> 4.8.0-34: - /proc/meminfo:Buffers: Up from 9GB to 14GB - /proc/meminfo:Cached: Down from 5GB to 2GB - - /proc/meminfo:SReclaimable: Down from 15GB to 14GB. - - /proc/meminfo:SUnreclaim: No notable change - - Downgrading 4.8.0-34 -> 4.4.0-63: (however uptime is only 1 hour) - - /proc/meminfo:Buffers: No notable change - - /proc/meminfo:Cached: No notable change - - /proc/meminfo:SReclaimable: No notable change - - /proc/meminfo:SUnreclaim: No notable change + - /proc/meminfo:SReclaimable: Down from 15GB to 14GB + - /proc/meminfo:SUnreclaim: Staying at 50MB Setting vm.vfs_cache_pressure = 300 seems to have a positive effect of not causing OOMs. Downgrading to 4.4.0-57 also works. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-63-generic 4.4.0-63.84 ProcVersionSignature: Ubuntu 4.4.0-63.84-generic 4.4.44 Uname: Linux 4.4.0-63-generic x86_64 AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.5 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' Date: Mon Feb 20 16:15:56 2017 InstallationDate: Installed on 2012-06-04 (1721 days ago) InstallationMedia: - + IwConfig: - lono wireless extensions. - - eth0 no wireless extensions. + lono wireless extensions. + + eth0 no wireless extensions. Lsusb: Error: [Errno 2] No such file or directory: 'lsusb' MachineType: System manufacturer System Product Name PciMultimedia: - + ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-63-generic root=UUID=b790930f-ad81-4b27-a353-a4b3d6a29007 ro nomodeset nomdmonddf nomdmonisw RelatedPackageVersions: - linux-restricted-modules-4.4.0-63-generic N/A - linux-backports-modules-4.4.0-63-generic N/A - linux-firmware1.157.8 + linux-restricted-modules-4.4.0-63-generic N/A + linux-backports-modules-4.4.0-63-generic N/A + linux-firmware1.157.8 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: Upgraded to xenial on 2017-02-16 (4 days ago) dmi.bios.date: 10/17/2011 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: 1106 dmi.board.asset.tag: To be filled by O.E.M. dmi.board.name: P8H67-M PRO dmi.board.vendor: ASUSTeK Computer INC. dmi.board.version: Rev 1.xx dmi.chassis.asset.tag: Asset-1234567890 dmi.chassis.type: 3 dmi.chassis.vendor: Chassis Manufacture dmi.chassis.version: Chassis Version dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1106:bd10/17/2011:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKComputerINC.:rnP8H67-MPRO:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion: dmi.product.name: System Product Name dmi.product.version: System Version dmi.sys.vendor: System manufacturer ** Description changed: Same as #1655842 - Started seeing oom-killer on multiple servers upgraded to 4.4.0-59. Unlike #1655842, also seeing the same oom-killer on multiple servers updated to 4.8.0-34. First upgraded them all 4.8 servers 4.8.0-36, then downgraded a few to - 4.4.0-63. I am seeing no change in the memory usage, so I can only - assume that 4.4.0-63 is also bugged with the same problem as 4.4.0-59 - and 4.8.0-34. Either because #1655842 is not fixed, or it is only fixed - for certain kinds of workloads. + 4.4.0-63. I am seeing an even more pronounced change in the memory + usage, so I can only assume that 4.4.0-63 is also bugged w
[Kernel-packages] [Bug 1666260] Re: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34
And again on 4.8.0-36. Upgraded server to 4.8.0-39. ** Attachment added: "linux-48036.oom2.txt" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+attachment/4825131/+files/linux-48036.oom2.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1666260 Title: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34 Status in linux package in Ubuntu: Confirmed Bug description: Same as #1655842 - Started seeing oom-killer on multiple servers upgraded to 4.4.0-59. Unlike #1655842, also seeing the same oom-killer on multiple servers updated to 4.8.0-34. First upgraded them all 4.8 servers 4.8.0-36, then downgraded a few to 4.4.0-63. I am seeing an even more pronounced change in the memory usage, so I can only assume that 4.4.0-63 is also bugged with the same problem as 4.4.0-59 and 4.8.0-34. Either because #1655842 is not fixed, or it is only fixed for certain kinds of workloads. These are the changes I'm seeing in our memory graphs between 4.4.0-59 and 4.4.0-63/4.8.0-34. The symptoms I'm seeing are: Upgrading 4.4.0-57 -> 4.4.0-59: - /proc/meminfo:Buffers: Up from 9GB to 15GB - /proc/meminfo:Cached: Up from 5GB to 10GB - /proc/meminfo:SReclaimable: Down from 15GB to 5GB - /proc/meminfo:SUnreclaim: Staying at 50MB Upgrading 4.4.0-57 -> 4.4.0-63: - /proc/meminfo:Buffers: Up from 9GB to 26GB - /proc/meminfo:Cached: Down from 5GB to 300MB - /proc/meminfo:SReclaimable: Down from 15GB to 2GB - /proc/meminfo:SUnreclaim: Down from 50MB to 30MB Upgrading 4.4.0-57 -> 4.8.0-34: - /proc/meminfo:Buffers: Up from 9GB to 14GB - /proc/meminfo:Cached: Down from 5GB to 2GB - /proc/meminfo:SReclaimable: Down from 15GB to 14GB - /proc/meminfo:SUnreclaim: Staying at 50MB Setting vm.vfs_cache_pressure = 300 seems to have a positive effect of not causing OOMs. Downgrading to 4.4.0-57 also works. Will also note that I haven't had a definitive OOM in 4.4.0-63. But the shift in memory usage is far too much from what I expect to be normal on these particular servers where I'm experiencing crashes. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-63-generic 4.4.0-63.84 ProcVersionSignature: Ubuntu 4.4.0-63.84-generic 4.4.44 Uname: Linux 4.4.0-63-generic x86_64 AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.5 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' Date: Mon Feb 20 16:15:56 2017 InstallationDate: Installed on 2012-06-04 (1721 days ago) InstallationMedia: IwConfig: lono wireless extensions. eth0 no wireless extensions. Lsusb: Error: [Errno 2] No such file or directory: 'lsusb' MachineType: System manufacturer System Product Name PciMultimedia: ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-63-generic root=UUID=b790930f-ad81-4b27-a353-a4b3d6a29007 ro nomodeset nomdmonddf nomdmonisw RelatedPackageVersions: linux-restricted-modules-4.4.0-63-generic N/A linux-backports-modules-4.4.0-63-generic N/A linux-firmware1.157.8 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: Upgraded to xenial on 2017-02-16 (4 days ago) dmi.bios.date: 10/17/2011 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: 1106 dmi.board.asset.tag: To be filled by O.E.M. dmi.board.name: P8H67-M PRO dmi.board.vendor: ASUSTeK Computer INC. dmi.board.version: Rev 1.xx dmi.chassis.asset.tag: Asset-1234567890 dmi.chassis.type: 3 dmi.chassis.vendor: Chassis Manufacture dmi.chassis.version: Chassis Version dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1106:bd10/17/2011:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKComputerINC.:rnP8H67-MPRO:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion: dmi.product.name: System Product Name dmi.product.version: System Version dmi.sys.vendor: System manufacturer To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1666260] Re: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34
And again on 4.8.0-36. Upgraded server to 4.8.0-39. (How many times must I keep on doing this?) ** Attachment added: "linux-48036.oom3.txt" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+attachment/4825946/+files/linux-48036.oom3.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1666260 Title: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34 Status in linux package in Ubuntu: Confirmed Bug description: Same as #1655842 - Started seeing oom-killer on multiple servers upgraded to 4.4.0-59. Unlike #1655842, also seeing the same oom-killer on multiple servers updated to 4.8.0-34. First upgraded them all 4.8 servers 4.8.0-36, then downgraded a few to 4.4.0-63. I am seeing an even more pronounced change in the memory usage, so I can only assume that 4.4.0-63 is also bugged with the same problem as 4.4.0-59 and 4.8.0-34. Either because #1655842 is not fixed, or it is only fixed for certain kinds of workloads. These are the changes I'm seeing in our memory graphs between 4.4.0-59 and 4.4.0-63/4.8.0-34. The symptoms I'm seeing are: Upgrading 4.4.0-57 -> 4.4.0-59: - /proc/meminfo:Buffers: Up from 9GB to 15GB - /proc/meminfo:Cached: Up from 5GB to 10GB - /proc/meminfo:SReclaimable: Down from 15GB to 5GB - /proc/meminfo:SUnreclaim: Staying at 50MB Upgrading 4.4.0-57 -> 4.4.0-63: - /proc/meminfo:Buffers: Up from 9GB to 26GB - /proc/meminfo:Cached: Down from 5GB to 300MB - /proc/meminfo:SReclaimable: Down from 15GB to 2GB - /proc/meminfo:SUnreclaim: Down from 50MB to 30MB Upgrading 4.4.0-57 -> 4.8.0-34: - /proc/meminfo:Buffers: Up from 9GB to 14GB - /proc/meminfo:Cached: Down from 5GB to 2GB - /proc/meminfo:SReclaimable: Down from 15GB to 14GB - /proc/meminfo:SUnreclaim: Staying at 50MB Setting vm.vfs_cache_pressure = 300 seems to have a positive effect of not causing OOMs. Downgrading to 4.4.0-57 also works. Will also note that I haven't had a definitive OOM in 4.4.0-63. But the shift in memory usage is far too much from what I expect to be normal on these particular servers where I'm experiencing crashes. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-63-generic 4.4.0-63.84 ProcVersionSignature: Ubuntu 4.4.0-63.84-generic 4.4.44 Uname: Linux 4.4.0-63-generic x86_64 AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.5 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' Date: Mon Feb 20 16:15:56 2017 InstallationDate: Installed on 2012-06-04 (1721 days ago) InstallationMedia: IwConfig: lono wireless extensions. eth0 no wireless extensions. Lsusb: Error: [Errno 2] No such file or directory: 'lsusb' MachineType: System manufacturer System Product Name PciMultimedia: ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-63-generic root=UUID=b790930f-ad81-4b27-a353-a4b3d6a29007 ro nomodeset nomdmonddf nomdmonisw RelatedPackageVersions: linux-restricted-modules-4.4.0-63-generic N/A linux-backports-modules-4.4.0-63-generic N/A linux-firmware1.157.8 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: Upgraded to xenial on 2017-02-16 (4 days ago) dmi.bios.date: 10/17/2011 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: 1106 dmi.board.asset.tag: To be filled by O.E.M. dmi.board.name: P8H67-M PRO dmi.board.vendor: ASUSTeK Computer INC. dmi.board.version: Rev 1.xx dmi.chassis.asset.tag: Asset-1234567890 dmi.chassis.type: 3 dmi.chassis.vendor: Chassis Manufacture dmi.chassis.version: Chassis Version dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1106:bd10/17/2011:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKComputerINC.:rnP8H67-MPRO:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion: dmi.product.name: System Product Name dmi.product.version: System Version dmi.sys.vendor: System manufacturer To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1666260] Re: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34
Another 4 servers OOM'd on 4.8.0-36 over the weekend. Upgraded to 4.8.0-39 ** Attachment added: "linux-48036.oom4.txt" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+attachment/4827637/+files/linux-48036.oom4.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1666260 Title: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34 Status in linux package in Ubuntu: Confirmed Bug description: Same as #1655842 - Started seeing oom-killer on multiple servers upgraded to 4.4.0-59. Unlike #1655842, also seeing the same oom-killer on multiple servers updated to 4.8.0-34. First upgraded them all 4.8 servers 4.8.0-36, then downgraded a few to 4.4.0-63. I am seeing an even more pronounced change in the memory usage, so I can only assume that 4.4.0-63 is also bugged with the same problem as 4.4.0-59 and 4.8.0-34. Either because #1655842 is not fixed, or it is only fixed for certain kinds of workloads. These are the changes I'm seeing in our memory graphs between 4.4.0-59 and 4.4.0-63/4.8.0-34. The symptoms I'm seeing are: Upgrading 4.4.0-57 -> 4.4.0-59: - /proc/meminfo:Buffers: Up from 9GB to 15GB - /proc/meminfo:Cached: Up from 5GB to 10GB - /proc/meminfo:SReclaimable: Down from 15GB to 5GB - /proc/meminfo:SUnreclaim: Staying at 50MB Upgrading 4.4.0-57 -> 4.4.0-63: - /proc/meminfo:Buffers: Up from 9GB to 26GB - /proc/meminfo:Cached: Down from 5GB to 300MB - /proc/meminfo:SReclaimable: Down from 15GB to 2GB - /proc/meminfo:SUnreclaim: Down from 50MB to 30MB Upgrading 4.4.0-57 -> 4.8.0-34: - /proc/meminfo:Buffers: Up from 9GB to 14GB - /proc/meminfo:Cached: Down from 5GB to 2GB - /proc/meminfo:SReclaimable: Down from 15GB to 14GB - /proc/meminfo:SUnreclaim: Staying at 50MB Setting vm.vfs_cache_pressure = 300 seems to have a positive effect of not causing OOMs. Downgrading to 4.4.0-57 also works. Will also note that I haven't had a definitive OOM in 4.4.0-63. But the shift in memory usage is far too much from what I expect to be normal on these particular servers where I'm experiencing crashes. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-63-generic 4.4.0-63.84 ProcVersionSignature: Ubuntu 4.4.0-63.84-generic 4.4.44 Uname: Linux 4.4.0-63-generic x86_64 AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.5 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' Date: Mon Feb 20 16:15:56 2017 InstallationDate: Installed on 2012-06-04 (1721 days ago) InstallationMedia: IwConfig: lono wireless extensions. eth0 no wireless extensions. Lsusb: Error: [Errno 2] No such file or directory: 'lsusb' MachineType: System manufacturer System Product Name PciMultimedia: ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-63-generic root=UUID=b790930f-ad81-4b27-a353-a4b3d6a29007 ro nomodeset nomdmonddf nomdmonisw RelatedPackageVersions: linux-restricted-modules-4.4.0-63-generic N/A linux-backports-modules-4.4.0-63-generic N/A linux-firmware1.157.8 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: Upgraded to xenial on 2017-02-16 (4 days ago) dmi.bios.date: 10/17/2011 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: 1106 dmi.board.asset.tag: To be filled by O.E.M. dmi.board.name: P8H67-M PRO dmi.board.vendor: ASUSTeK Computer INC. dmi.board.version: Rev 1.xx dmi.chassis.asset.tag: Asset-1234567890 dmi.chassis.type: 3 dmi.chassis.vendor: Chassis Manufacture dmi.chassis.version: Chassis Version dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1106:bd10/17/2011:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKComputerINC.:rnP8H67-MPRO:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion: dmi.product.name: System Product Name dmi.product.version: System Version dmi.sys.vendor: System manufacturer To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1666260] Re: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34
** Attachment added: "linux-48036.oom7.txt" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+attachment/4827641/+files/linux-48036.oom7.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1666260 Title: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34 Status in linux package in Ubuntu: Confirmed Bug description: Same as #1655842 - Started seeing oom-killer on multiple servers upgraded to 4.4.0-59. Unlike #1655842, also seeing the same oom-killer on multiple servers updated to 4.8.0-34. First upgraded them all 4.8 servers 4.8.0-36, then downgraded a few to 4.4.0-63. I am seeing an even more pronounced change in the memory usage, so I can only assume that 4.4.0-63 is also bugged with the same problem as 4.4.0-59 and 4.8.0-34. Either because #1655842 is not fixed, or it is only fixed for certain kinds of workloads. These are the changes I'm seeing in our memory graphs between 4.4.0-59 and 4.4.0-63/4.8.0-34. The symptoms I'm seeing are: Upgrading 4.4.0-57 -> 4.4.0-59: - /proc/meminfo:Buffers: Up from 9GB to 15GB - /proc/meminfo:Cached: Up from 5GB to 10GB - /proc/meminfo:SReclaimable: Down from 15GB to 5GB - /proc/meminfo:SUnreclaim: Staying at 50MB Upgrading 4.4.0-57 -> 4.4.0-63: - /proc/meminfo:Buffers: Up from 9GB to 26GB - /proc/meminfo:Cached: Down from 5GB to 300MB - /proc/meminfo:SReclaimable: Down from 15GB to 2GB - /proc/meminfo:SUnreclaim: Down from 50MB to 30MB Upgrading 4.4.0-57 -> 4.8.0-34: - /proc/meminfo:Buffers: Up from 9GB to 14GB - /proc/meminfo:Cached: Down from 5GB to 2GB - /proc/meminfo:SReclaimable: Down from 15GB to 14GB - /proc/meminfo:SUnreclaim: Staying at 50MB Setting vm.vfs_cache_pressure = 300 seems to have a positive effect of not causing OOMs. Downgrading to 4.4.0-57 also works. Will also note that I haven't had a definitive OOM in 4.4.0-63. But the shift in memory usage is far too much from what I expect to be normal on these particular servers where I'm experiencing crashes. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-63-generic 4.4.0-63.84 ProcVersionSignature: Ubuntu 4.4.0-63.84-generic 4.4.44 Uname: Linux 4.4.0-63-generic x86_64 AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.5 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' Date: Mon Feb 20 16:15:56 2017 InstallationDate: Installed on 2012-06-04 (1721 days ago) InstallationMedia: IwConfig: lono wireless extensions. eth0 no wireless extensions. Lsusb: Error: [Errno 2] No such file or directory: 'lsusb' MachineType: System manufacturer System Product Name PciMultimedia: ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-63-generic root=UUID=b790930f-ad81-4b27-a353-a4b3d6a29007 ro nomodeset nomdmonddf nomdmonisw RelatedPackageVersions: linux-restricted-modules-4.4.0-63-generic N/A linux-backports-modules-4.4.0-63-generic N/A linux-firmware1.157.8 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: Upgraded to xenial on 2017-02-16 (4 days ago) dmi.bios.date: 10/17/2011 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: 1106 dmi.board.asset.tag: To be filled by O.E.M. dmi.board.name: P8H67-M PRO dmi.board.vendor: ASUSTeK Computer INC. dmi.board.version: Rev 1.xx dmi.chassis.asset.tag: Asset-1234567890 dmi.chassis.type: 3 dmi.chassis.vendor: Chassis Manufacture dmi.chassis.version: Chassis Version dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1106:bd10/17/2011:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKComputerINC.:rnP8H67-MPRO:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion: dmi.product.name: System Product Name dmi.product.version: System Version dmi.sys.vendor: System manufacturer To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1666260] Re: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34
** Attachment added: "linux-48036.oom5.txt" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+attachment/4827639/+files/linux-48036.oom5.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1666260 Title: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34 Status in linux package in Ubuntu: Confirmed Bug description: Same as #1655842 - Started seeing oom-killer on multiple servers upgraded to 4.4.0-59. Unlike #1655842, also seeing the same oom-killer on multiple servers updated to 4.8.0-34. First upgraded them all 4.8 servers 4.8.0-36, then downgraded a few to 4.4.0-63. I am seeing an even more pronounced change in the memory usage, so I can only assume that 4.4.0-63 is also bugged with the same problem as 4.4.0-59 and 4.8.0-34. Either because #1655842 is not fixed, or it is only fixed for certain kinds of workloads. These are the changes I'm seeing in our memory graphs between 4.4.0-59 and 4.4.0-63/4.8.0-34. The symptoms I'm seeing are: Upgrading 4.4.0-57 -> 4.4.0-59: - /proc/meminfo:Buffers: Up from 9GB to 15GB - /proc/meminfo:Cached: Up from 5GB to 10GB - /proc/meminfo:SReclaimable: Down from 15GB to 5GB - /proc/meminfo:SUnreclaim: Staying at 50MB Upgrading 4.4.0-57 -> 4.4.0-63: - /proc/meminfo:Buffers: Up from 9GB to 26GB - /proc/meminfo:Cached: Down from 5GB to 300MB - /proc/meminfo:SReclaimable: Down from 15GB to 2GB - /proc/meminfo:SUnreclaim: Down from 50MB to 30MB Upgrading 4.4.0-57 -> 4.8.0-34: - /proc/meminfo:Buffers: Up from 9GB to 14GB - /proc/meminfo:Cached: Down from 5GB to 2GB - /proc/meminfo:SReclaimable: Down from 15GB to 14GB - /proc/meminfo:SUnreclaim: Staying at 50MB Setting vm.vfs_cache_pressure = 300 seems to have a positive effect of not causing OOMs. Downgrading to 4.4.0-57 also works. Will also note that I haven't had a definitive OOM in 4.4.0-63. But the shift in memory usage is far too much from what I expect to be normal on these particular servers where I'm experiencing crashes. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-63-generic 4.4.0-63.84 ProcVersionSignature: Ubuntu 4.4.0-63.84-generic 4.4.44 Uname: Linux 4.4.0-63-generic x86_64 AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.5 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' Date: Mon Feb 20 16:15:56 2017 InstallationDate: Installed on 2012-06-04 (1721 days ago) InstallationMedia: IwConfig: lono wireless extensions. eth0 no wireless extensions. Lsusb: Error: [Errno 2] No such file or directory: 'lsusb' MachineType: System manufacturer System Product Name PciMultimedia: ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-63-generic root=UUID=b790930f-ad81-4b27-a353-a4b3d6a29007 ro nomodeset nomdmonddf nomdmonisw RelatedPackageVersions: linux-restricted-modules-4.4.0-63-generic N/A linux-backports-modules-4.4.0-63-generic N/A linux-firmware1.157.8 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: Upgraded to xenial on 2017-02-16 (4 days ago) dmi.bios.date: 10/17/2011 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: 1106 dmi.board.asset.tag: To be filled by O.E.M. dmi.board.name: P8H67-M PRO dmi.board.vendor: ASUSTeK Computer INC. dmi.board.version: Rev 1.xx dmi.chassis.asset.tag: Asset-1234567890 dmi.chassis.type: 3 dmi.chassis.vendor: Chassis Manufacture dmi.chassis.version: Chassis Version dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1106:bd10/17/2011:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKComputerINC.:rnP8H67-MPRO:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion: dmi.product.name: System Product Name dmi.product.version: System Version dmi.sys.vendor: System manufacturer To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1666260] Re: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34
** Attachment added: "linux-48036.oom6.txt" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+attachment/4827640/+files/linux-48036.oom6.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1666260 Title: "Out of memory" errors after upgrade to 4.4.0-59 + 4.8.0-34 Status in linux package in Ubuntu: Confirmed Bug description: Same as #1655842 - Started seeing oom-killer on multiple servers upgraded to 4.4.0-59. Unlike #1655842, also seeing the same oom-killer on multiple servers updated to 4.8.0-34. First upgraded them all 4.8 servers 4.8.0-36, then downgraded a few to 4.4.0-63. I am seeing an even more pronounced change in the memory usage, so I can only assume that 4.4.0-63 is also bugged with the same problem as 4.4.0-59 and 4.8.0-34. Either because #1655842 is not fixed, or it is only fixed for certain kinds of workloads. These are the changes I'm seeing in our memory graphs between 4.4.0-59 and 4.4.0-63/4.8.0-34. The symptoms I'm seeing are: Upgrading 4.4.0-57 -> 4.4.0-59: - /proc/meminfo:Buffers: Up from 9GB to 15GB - /proc/meminfo:Cached: Up from 5GB to 10GB - /proc/meminfo:SReclaimable: Down from 15GB to 5GB - /proc/meminfo:SUnreclaim: Staying at 50MB Upgrading 4.4.0-57 -> 4.4.0-63: - /proc/meminfo:Buffers: Up from 9GB to 26GB - /proc/meminfo:Cached: Down from 5GB to 300MB - /proc/meminfo:SReclaimable: Down from 15GB to 2GB - /proc/meminfo:SUnreclaim: Down from 50MB to 30MB Upgrading 4.4.0-57 -> 4.8.0-34: - /proc/meminfo:Buffers: Up from 9GB to 14GB - /proc/meminfo:Cached: Down from 5GB to 2GB - /proc/meminfo:SReclaimable: Down from 15GB to 14GB - /proc/meminfo:SUnreclaim: Staying at 50MB Setting vm.vfs_cache_pressure = 300 seems to have a positive effect of not causing OOMs. Downgrading to 4.4.0-57 also works. Will also note that I haven't had a definitive OOM in 4.4.0-63. But the shift in memory usage is far too much from what I expect to be normal on these particular servers where I'm experiencing crashes. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-63-generic 4.4.0-63.84 ProcVersionSignature: Ubuntu 4.4.0-63.84-generic 4.4.44 Uname: Linux 4.4.0-63-generic x86_64 AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.1-0ubuntu2.5 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' Date: Mon Feb 20 16:15:56 2017 InstallationDate: Installed on 2012-06-04 (1721 days ago) InstallationMedia: IwConfig: lono wireless extensions. eth0 no wireless extensions. Lsusb: Error: [Errno 2] No such file or directory: 'lsusb' MachineType: System manufacturer System Product Name PciMultimedia: ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-63-generic root=UUID=b790930f-ad81-4b27-a353-a4b3d6a29007 ro nomodeset nomdmonddf nomdmonisw RelatedPackageVersions: linux-restricted-modules-4.4.0-63-generic N/A linux-backports-modules-4.4.0-63-generic N/A linux-firmware1.157.8 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: Upgraded to xenial on 2017-02-16 (4 days ago) dmi.bios.date: 10/17/2011 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: 1106 dmi.board.asset.tag: To be filled by O.E.M. dmi.board.name: P8H67-M PRO dmi.board.vendor: ASUSTeK Computer INC. dmi.board.version: Rev 1.xx dmi.chassis.asset.tag: Asset-1234567890 dmi.chassis.type: 3 dmi.chassis.vendor: Chassis Manufacture dmi.chassis.version: Chassis Version dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1106:bd10/17/2011:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKComputerINC.:rnP8H67-MPRO:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion: dmi.product.name: System Product Name dmi.product.version: System Version dmi.sys.vendor: System manufacturer To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1956028] [NEW] Zvols get lost upon host startup and VM shut down
Public bug reported: Affects linux kernels 5.13+ (21.10 uses 5.13). See: https://github.com/openzfs/zfs/issues/12712 Patch to backport into linux-modules that should fix issue: https://github.com/openzfs/zfs/pull/12759 ProblemType: Bug DistroRelease: Ubuntu 21.10 Package: linux-modules-5.13.0-22-generic 5.13.0-22.22 ProcVersionSignature: Ubuntu 5.13.0-22.22-generic 5.13.19 Uname: Linux 5.13.0-22-generic x86_64 NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair ApportVersion: 2.20.11-0ubuntu71 Architecture: amd64 AudioDevicesInUse: USERPID ACCESS COMMAND /dev/snd/controlC1: ibuclaw5104 F pulseaudio /dev/snd/pcmC1D0p: ibuclaw5104 F...m pulseaudio /dev/snd/controlC0: ibuclaw5104 F pulseaudio CasperMD5CheckResult: pass Date: Thu Dec 30 10:31:05 2021 InstallationDate: Installed on 2021-12-10 (19 days ago) InstallationMedia: Ubuntu 21.10 "Impish Indri" - Release amd64 (20211012) MachineType: TUXEDO TUXEDO Pulse 15 Gen1 ProcFB: 0 amdgpudrmfb ProcKernelCmdLine: BOOT_IMAGE=/BOOT/ubuntu_2h043n@/vmlinuz-5.13.0-22-generic root=ZFS=rpool/ROOT/ubuntu_2h043n ro quiet splash vt.handoff=1 PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon. RelatedPackageVersions: linux-restricted-modules-5.13.0-22-generic N/A linux-backports-modules-5.13.0-22-generic N/A linux-firmware 1.201.3 SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 05/11/2021 dmi.bios.release: 5.16 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: N.1.07.A03 dmi.board.asset.tag: Standard dmi.board.name: PULSE1501 dmi.board.vendor: NB02 dmi.board.version: Standard dmi.chassis.asset.tag: Standard dmi.chassis.type: 10 dmi.chassis.vendor: TUXEDO dmi.chassis.version: Standard dmi.ec.firmware.release: 1.7 dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvrN.1.07.A03:bd05/11/2021:br5.16:efr1.7:svnTUXEDO:pnTUXEDOPulse15Gen1:pvrStandard:rvnNB02:rnPULSE1501:rvrStandard:cvnTUXEDO:ct10:cvrStandard:skuPULSE1501: dmi.product.family: TUXEDO Computers dmi.product.name: TUXEDO Pulse 15 Gen1 dmi.product.sku: PULSE1501 dmi.product.version: Standard dmi.sys.vendor: TUXEDO ** Affects: linux (Ubuntu) Importance: Undecided Status: New ** Tags: amd64 apport-bug impish ** Description changed: Affects linux kernels 5.13+ (21.10 uses 5.13). - See: https://github.com/openzfs/zfs/issues/1271 + See: https://github.com/openzfs/zfs/issues/12712 Patch to backport into linux-modules that should fix issue: https://github.com/openzfs/zfs/pull/12759 ProblemType: Bug DistroRelease: Ubuntu 21.10 Package: linux-modules-5.13.0-22-generic 5.13.0-22.22 ProcVersionSignature: Ubuntu 5.13.0-22.22-generic 5.13.19 Uname: Linux 5.13.0-22-generic x86_64 NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair ApportVersion: 2.20.11-0ubuntu71 Architecture: amd64 AudioDevicesInUse: - USERPID ACCESS COMMAND - /dev/snd/controlC1: ibuclaw5104 F pulseaudio - /dev/snd/pcmC1D0p: ibuclaw5104 F...m pulseaudio - /dev/snd/controlC0: ibuclaw5104 F pulseaudio + USERPID ACCESS COMMAND + /dev/snd/controlC1: ibuclaw5104 F pulseaudio + /dev/snd/pcmC1D0p: ibuclaw5104 F...m pulseaudio + /dev/snd/controlC0: ibuclaw5104 F pulseaudio CasperMD5CheckResult: pass Date: Thu Dec 30 10:31:05 2021 InstallationDate: Installed on 2021-12-10 (19 days ago) InstallationMedia: Ubuntu 21.10 "Impish Indri" - Release amd64 (20211012) MachineType: TUXEDO TUXEDO Pulse 15 Gen1 ProcFB: 0 amdgpudrmfb ProcKernelCmdLine: BOOT_IMAGE=/BOOT/ubuntu_2h043n@/vmlinuz-5.13.0-22-generic root=ZFS=rpool/ROOT/ubuntu_2h043n ro quiet splash vt.handoff=1 PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon. RelatedPackageVersions: - linux-restricted-modules-5.13.0-22-generic N/A - linux-backports-modules-5.13.0-22-generic N/A - linux-firmware 1.201.3 + linux-restricted-modules-5.13.0-22-generic N/A + linux-backports-modules-5.13.0-22-generic N/A + linux-firmware 1.201.3 SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 05/11/2021 dmi.bios.release: 5.16 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: N.1.07.A03 dmi.board.asset.tag: Standard dmi.board.name: PULSE1501 dmi.board.vendor: NB02 dmi.board.version: Standard dmi.chassis.asset.tag: Standard dmi.chassis.type: 10 dmi.chassis.vendor: TUXEDO dmi.chassis.version: Standard dmi.ec.firmware.release: 1.7 dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvrN.1.07.A03:bd05/11/2021:br5.16:efr1.7:svnTUXEDO:pnTUXEDOPulse15Gen1:pvrStandard:rvnNB02:rnPULSE1501:rvrStandard:cvnTUXEDO:ct10:cvrStandard:skuPULSE1501: dmi.product.family: TUXEDO Co
