Ok, finally it seems I found the problem. It seems Xen setup code is accidentally setting up the kernel page tables in a way that causes the last 2G of memory to be identically mapped (kernel mappings). This would just work normally because the page table that covers the first 1G would correctly be clean for the second 512M (which started the module space before). If modules ever reached more than 512M of memory this just would have happened with the old layout, too. But now that the kernel image is increased to 1G, we start to use the bad page table immediately.
Sent this patch upstream (right now only tested with the new layout) ** Patch added: "0001-x86-xen-Fix-setup-of-64bit-kernel-pagetables.patch" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1350522/+attachment/4189340/+files/0001-x86-xen-Fix-setup-of-64bit-kernel-pagetables.patch -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1350522 Title: EC2 kernel crash due to vmalloc Status in “linux” package in Ubuntu: Confirmed Status in “linux” source package in Utopic: Confirmed Bug description: During Alpha-2 automated testing, I saw the following in a log: 19:35:02 [ 2.475810] systemd-udevd[95]: starting version 204 19:35:02 [ 2.547049] ------------[ cut here ]------------ 19:35:02 [ 2.547065] WARNING: CPU: 0 PID: 97 at /build/buildd/linux-3.16.0/mm/vmalloc.c:128 vmap_page_range_noflush+0x2d1/0x370() 19:35:02 [ 2.547069] Modules linked in: 19:35:02 [ 2.547073] CPU: 0 PID: 97 Comm: systemd-udevd Not tainted 3.16.0-6-generic #11-Ubuntu 19:35:02 [ 2.547077] 0000000000000009 ffff880002defb98 ffffffff81755538 0000000000000000 19:35:02 [ 2.547082] ffff880002defbd0 ffffffff8106bb0d ffff88000400ec88 0000000000000001 19:35:02 [ 2.547086] ffff880002fcfb00 ffffffffc0391000 0000000000000000 ffff880002defbe0 19:35:02 [ 2.547090] Call Trace: 19:35:02 [ 2.547096] [<ffffffff81755538>] dump_stack+0x45/0x56 19:35:02 [ 2.547101] [<ffffffff8106bb0d>] warn_slowpath_common+0x7d/0xa0 19:35:02 [ 2.547104] [<ffffffff8106bbea>] warn_slowpath_null+0x1a/0x20 19:35:02 [ 2.547108] [<ffffffff81197c31>] vmap_page_range_noflush+0x2d1/0x370 19:35:02 [ 2.547112] [<ffffffff81197cfe>] map_vm_area+0x2e/0x40 19:35:02 [ 2.547115] [<ffffffff8119a058>] __vmalloc_node_range+0x188/0x280 19:35:02 [ 2.547120] [<ffffffff810e92b4>] ? module_alloc_update_bounds+0x14/0x70 19:35:02 [ 2.547124] [<ffffffff810e92b4>] ? module_alloc_update_bounds+0x14/0x70 19:35:02 [ 2.547129] [<ffffffff8104f294>] module_alloc+0x74/0xd0 19:35:02 [ 2.547132] [<ffffffff810e92b4>] ? module_alloc_update_bounds+0x14/0x70 19:35:02 [ 2.547135] [<ffffffff810e92b4>] module_alloc_update_bounds+0x14/0x70 19:35:02 [ 2.547146] [<ffffffff810e9a6c>] layout_and_allocate+0x74c/0xc70 19:35:02 [ 2.547149] [<ffffffff810ea063>] load_module+0xd3/0x1b70 19:35:02 [ 2.547154] [<ffffffff811cfeb1>] ? vfs_read+0xf1/0x170 19:35:02 [ 2.547157] [<ffffffff810e7aa1>] ? copy_module_from_fd.isra.46+0x121/0x180 19:35:02 [ 2.547161] [<ffffffff810ebc76>] SyS_finit_module+0x86/0xb0 19:35:02 [ 2.547167] [<ffffffff8175de7f>] tracesys+0xe1/0xe6 19:35:02 [ 2.547169] ---[ end trace 8a5de7fc66e75fe4 ]--- 19:35:02 [ 2.547172] vmalloc: allocation failure, allocated 20480 of 24576 bytes 19:35:02 [ 2.547175] systemd-udevd: page allocation failure: order:0, mode:0xd2 19:35:02 [ 2.547180] CPU: 0 PID: 97 Comm: systemd-udevd Tainted: G W 3.16.0-6-generic #11-Ubuntu 19:35:02 [ 2.547183] ffffffff81a88bc0 ffff880002defc08 ffffffff81755538 00000000000000d2 19:35:02 [ 2.547187] ffff880002defc90 ffffffff811642bf ffffffff81a88bc0 ffff880002defc28 19:35:02 [ 2.547191] 00003fff00000018 ffff880002defca0 ffff880002defc40 0000000000000163 19:35:02 [ 2.547195] Call Trace: 19:35:02 [ 2.547197] [<ffffffff81755538>] dump_stack+0x45/0x56 19:35:02 [ 2.547202] [<ffffffff811642bf>] warn_alloc_failed+0xdf/0x130 19:35:02 [ 2.547207] [<ffffffff8119a118>] __vmalloc_node_range+0x248/0x280 19:35:02 [ 2.547210] [<ffffffff810e92b4>] ? module_alloc_update_bounds+0x14/0x70 19:35:02 [ 2.547214] [<ffffffff810e92b4>] ? module_alloc_update_bounds+0x14/0x70 19:35:02 [ 2.547217] [<ffffffff8104f294>] module_alloc+0x74/0xd0 19:35:02 [ 2.547220] [<ffffffff810e92b4>] ? module_alloc_update_bounds+0x14/0x70 19:35:02 [ 2.547224] [<ffffffff810e92b4>] module_alloc_update_bounds+0x14/0x70 19:35:02 [ 2.547229] [<ffffffff810e9a6c>] layout_and_allocate+0x74c/0xc70 19:35:02 [ 2.547232] [<ffffffff810ea063>] load_module+0xd3/0x1b70 19:35:02 [ 2.547235] [<ffffffff811cfeb1>] ? vfs_read+0xf1/0x170 19:35:02 [ 2.547238] [<ffffffff810e7aa1>] ? copy_module_from_fd.isra.46+0x121/0x180 19:35:02 [ 2.547242] [<ffffffff810ebc76>] SyS_finit_module+0x86/0xb0 19:35:02 [ 2.547246] [<ffffffff8175de7f>] tracesys+0xe1/0xe6 19:35:02 [ 2.547248] Mem-Info: --- AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Jul 30 20:19 seq crw-rw---- 1 root audio 116, 33 Jul 30 20:19 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.4-0ubuntu2 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 42.901579] systemd-logind[1378]: New seat seat0. [ 42.932887] systemd-logind[1378]: New session 1 of user ubuntu. DistroRelease: Ubuntu 14.10 Ec2AMI: ami-c571d8d8 Ec2AMIManifest: (unknown) Ec2AvailabilityZone: sa-east-1b Ec2InstanceType: m3.medium Ec2Kernel: aki-5553f448 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux-image-virtual 3.16.0.5.6 PackageArchitecture: amd64 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 ProcModules: ProcVersionSignature: User Name 3.16.0-5.10-generic 3.16.0-rc6 RelatedPackageVersions: linux-restricted-modules-3.16.0-5-generic N/A linux-backports-modules-3.16.0-5-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: utopic ec2-images third-party-packages Uname: Linux 3.16.0-5-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video _MarkForUpload: True --- AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Jul 31 20:13 seq crw-rw---- 1 root audio 116, 33 Jul 31 20:13 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.5-0ubuntu1 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory CurrentDmesg: [ 17.975584] systemd-logind[1248]: New seat seat0. [ 17.993902] systemd-logind[1248]: New session 1 of user ubuntu. DistroRelease: Ubuntu 14.10 Ec2AMI: ami-b7ce1dc0 Ec2AMIManifest: (unknown) Ec2AvailabilityZone: eu-west-1c Ec2InstanceType: m3.medium Ec2Kernel: aki-52a34525 Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory Lspci: Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 Package: linux-image-virtual 3.16.0.6.7 PackageArchitecture: amd64 PciMultimedia: ProcFB: ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0 initcall_debug debug ignore_loglevel LOGLEVEL=8 ProcModules: ProcVersionSignature: Ubuntu 3.16.0-6.11-generic 3.16.0-rc7 RelatedPackageVersions: linux-restricted-modules-3.16.0-6-generic N/A linux-backports-modules-3.16.0-6-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: utopic ec2-images Uname: Linux 3.16.0-6-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1350522/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
