Public bug reported: This bug closely resembles bug 1518457 and others, but is reported separately as the affected kernel versions do not match, see comments #45, #43 and #8 on 1518457.
The issue first occured on a production workload (JIRA running on Tomcat) on an EC2 t2.small instance after upgrading to 15.10. The issue happens when running out of memory, typicall during busy hours or when nightly Python backup script runs. To reproduce the issue, I use t2.micro (1GiB RAM) because the problem then arises just by starting the JIRA application. I do not see the problem on 14.04 or 15.05 on the same server. Symptoms: kswapd0 falls into a busy loop and spins on 100% CPU usage and swap not being used/barely being used. The system can be recovered with `echo 1 > /proc/sys/vm/drop_caches`, but with high traffic/little free memory, the problem returns immediately. Swappiness has not been changed (vm.swappiness = 60). I can reproduce this consitently on Ubuntu AMIs (https://cloud- images.ubuntu.com/locator/ec2/). I've used AMI ami-c4140aa8 (eu- central-1). I can provide SSH access to a machine on request and have public Puppet scripts to configure a new server. I've attached scripts and logs from an up to date 15.10 and 15.04 instance to compare results with and without the bug. Note that the CPU numbers in the vmstat output doesn't specify which process, but top has showed that the majority of user time is the java process and the majority of system time is the kswapd0 process. ProblemType: Bug DistroRelease: Ubuntu 15.10 Package: linux-image-4.2.0-23-generic 4.2.0-23.28 ProcVersionSignature: User Name 4.2.0-23.28-generic 4.2.6 Uname: Linux 4.2.0-23-generic x86_64 AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Jan 10 03:56 seq crw-rw---- 1 root audio 116, 33 Jan 10 03:56 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.19.1-0ubuntu5 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: N/A Date: Sun Jan 10 04:07:30 2016 Ec2AMI: ami-c4140aa8 Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-central-1b Ec2InstanceType: t2.micro Ec2Kernel: unavailable Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 MachineType: Xen HVM domU PciMultimedia: ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR=<set> LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 xen ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.2.0-23-generic root=UUID=56681ffc-93a4-47c9-b487-b67be12205d9 ro console=tty1 console=ttyS0 net.ifnames=0 RelatedPackageVersions: linux-restricted-modules-4.2.0-23-generic N/A linux-backports-modules-4.2.0-23-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UdevLog: Error: [Errno 2] No such file or directory: '/var/log/udev' UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 12/07/2015 dmi.bios.vendor: Xen dmi.bios.version: 4.2.amazon dmi.chassis.type: 1 dmi.chassis.vendor: Xen dmi.modalias: dmi:bvnXen:bvr4.2.amazon:bd12/07/2015:svnXen:pnHVMdomU:pvr4.2.amazon:cvnXen:ct1:cvr: dmi.product.name: HVM domU dmi.product.version: 4.2.amazon dmi.sys.vendor: Xen ** Affects: linux (Ubuntu) Importance: Undecided Status: New ** Tags: amd64 apport-bug ec2-images wily ** Attachment added: "Commands to reproduce on Ubuntu 15.10 and vmstat log" https://bugs.launchpad.net/bugs/1532513/+attachment/4547222/+files/commands-15.10.log -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1532513 Title: kswapd 100% CPU usage and running out of memory without swap being used Status in linux package in Ubuntu: New Bug description: This bug closely resembles bug 1518457 and others, but is reported separately as the affected kernel versions do not match, see comments #45, #43 and #8 on 1518457. The issue first occured on a production workload (JIRA running on Tomcat) on an EC2 t2.small instance after upgrading to 15.10. The issue happens when running out of memory, typicall during busy hours or when nightly Python backup script runs. To reproduce the issue, I use t2.micro (1GiB RAM) because the problem then arises just by starting the JIRA application. I do not see the problem on 14.04 or 15.05 on the same server. Symptoms: kswapd0 falls into a busy loop and spins on 100% CPU usage and swap not being used/barely being used. The system can be recovered with `echo 1 > /proc/sys/vm/drop_caches`, but with high traffic/little free memory, the problem returns immediately. Swappiness has not been changed (vm.swappiness = 60). I can reproduce this consitently on Ubuntu AMIs (https://cloud- images.ubuntu.com/locator/ec2/). I've used AMI ami-c4140aa8 (eu- central-1). I can provide SSH access to a machine on request and have public Puppet scripts to configure a new server. I've attached scripts and logs from an up to date 15.10 and 15.04 instance to compare results with and without the bug. Note that the CPU numbers in the vmstat output doesn't specify which process, but top has showed that the majority of user time is the java process and the majority of system time is the kswapd0 process. ProblemType: Bug DistroRelease: Ubuntu 15.10 Package: linux-image-4.2.0-23-generic 4.2.0-23.28 ProcVersionSignature: User Name 4.2.0-23.28-generic 4.2.6 Uname: Linux 4.2.0-23-generic x86_64 AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Jan 10 03:56 seq crw-rw---- 1 root audio 116, 33 Jan 10 03:56 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.19.1-0ubuntu5 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: N/A Date: Sun Jan 10 04:07:30 2016 Ec2AMI: ami-c4140aa8 Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-central-1b Ec2InstanceType: t2.micro Ec2Kernel: unavailable Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 MachineType: Xen HVM domU PciMultimedia: ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR=<set> LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 xen ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.2.0-23-generic root=UUID=56681ffc-93a4-47c9-b487-b67be12205d9 ro console=tty1 console=ttyS0 net.ifnames=0 RelatedPackageVersions: linux-restricted-modules-4.2.0-23-generic N/A linux-backports-modules-4.2.0-23-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UdevLog: Error: [Errno 2] No such file or directory: '/var/log/udev' UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 12/07/2015 dmi.bios.vendor: Xen dmi.bios.version: 4.2.amazon dmi.chassis.type: 1 dmi.chassis.vendor: Xen dmi.modalias: dmi:bvnXen:bvr4.2.amazon:bd12/07/2015:svnXen:pnHVMdomU:pvr4.2.amazon:cvnXen:ct1:cvr: dmi.product.name: HVM domU dmi.product.version: 4.2.amazon dmi.sys.vendor: Xen To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1532513/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
