[Kernel-packages] [Bug 1534345] [NEW] Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking
Public bug reported: Lots of details and history of the problem here: https://askubuntu.com/questions/710747/after-upgrading- to-15-10-from-15-04-ec2-webservers-have-become-very-unstable 10 of my webservers have started crashing immediately following the 15.10 upgrade. As far as what exactly defines a "crash", Instance Status Checks fail, and I can no longer SSH to the machine. Background daemons running on the system stop responding, and nothing is written to the logs. After weeks of working with the AWS team, I finally fixed a netconsole issue via "echo 7 > /proc/sys/kernel/printk" and got netconsole working properly, and finally have a trace: [21410.260077] general protection fault: [#1] SMP [21410.261976] Modules linked in: isofs xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp bridge stp llc iptable_filter ip_tables x_tables ppdev intel_rapl iosf_mbi xen_fbfront fb_sys_fops input_leds serio_raw i2c_piix4 parport_pc 8250_fintek parport mac_hid netconsole configfs autofs4 crct10dif_pclmul crc32_pclmul cirrus syscopyarea sysfillrect sysimgblt aesni_intel ttm aes_x86_64 drm_kms_helper lrw gf128mul glue_helper ablk_helper cryptd psmouse drm ixgbevf pata_acpi floppy [21410.264054] CPU: 0 PID: 26957 Comm: apache2 Not tainted 4.2.0-23-generic #28-Ubuntu [21410.264054] Hardware name: Xen HVM domU, BIOS 4.2.amazon 12/07/2015 [21410.264054] task: 8803f9809b80 ti: 8803f999c000 task.ti: 8803f999c000 [21410.264054] RIP: 0010:[] [] run_timer_softirq+0x116/0x2d0 [21410.264054] RSP: :8803ff203e98 EFLAGS: 00010086 [21410.264054] RAX: dead00200200 RBX: 8803ff20e9c0 RCX: 8803ff203ec8 [21410.264054] RDX: 8803ff203ec8 RSI: 00011fc0 RDI: 8803ff20e9c0 [21410.264054] RBP: 8803ff203f08 R08: a77a R09: [21410.264054] R10: 0020 R11: 0004 R12: 007c [21410.264054] R13: 8172aaf0 R14: R15: 8803af955be0 [21410.264054] FS: 7fb0ce6e8780() GS:8803ff20() knlGS: [21410.264054] CS: 0010 DS: ES: CR0: 80050033 [21410.264054] CR2: 7fb0ce51e130 CR3: 0003fb233000 CR4: 001406f0 [21410.264054] Stack: [21410.264054] 8803ff203eb8 8803ff20f5f8 8803ff20f3f8 8803ff20f1f8 [21410.264054] 8803ff20e9f8 8803af955b58 dead00200200 f60fabc0 [21410.264054] 00011fc0 0001 81c0b0c8 0001 [21410.264054] Call Trace: [21410.264054] [21410.264054] [] __do_softirq+0xf6/0x250 [21410.264054] [] irq_exit+0xa3/0xb0 [21410.264054] [] xen_evtchn_do_upcall+0x39/0x50 [21410.264054] [] xen_hvm_callback_vector+0x6b/0x70 [21410.264054] [21410.264054] Code: 81 e6 00 00 20 00 48 85 d2 48 89 45 b8 0f 85 30 01 00 00 4c 89 7b 08 0f 1f 44 00 00 49 8b 07 49 8b 57 08 48 85 c0 48 89 02 74 04 <48> 89 50 08 41 f6 47 2a 10 48 b8 00 02 20 00 00 00 ad de 49 c7 [21410.264054] RIP [] run_timer_softirq+0x116/0x2d0 [21410.264054] RSP I don't have a vmcore at the moment, but I'm trying to get one from AWS and should have one in the next couple of days. This is happening frequently and repeatedly since I first upgraded to 15.10 on early December. ubuntu@xxx-web-xx:~$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description:Ubuntu 15.10 Release:15.10 Codename: wily ubuntu@xxx-web-xx:~$ uname -a Linux xxx-web-xx 4.2.0-23-generic #28-Ubuntu SMP Sun Dec 27 17:47:31 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux ubuntu@xxx-web-xx:~$ ProblemType: Bug DistroRelease: Ubuntu 15.10 Package: linux-image-4.2.0-23-generic 4.2.0-23.28 ProcVersionSignature: User Name 4.2.0-23.28-generic 4.2.6 Uname: Linux 4.2.0-23-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Jan 14 15:42 seq crw-rw 1 root audio 116, 33 Jan 14 15:42 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.19.1-0ubuntu5 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: Date: Thu Jan 14 21:31:14 2016 Ec2AMI: ami-d5e7adbf Ec2AMIManifest: (unknown) Ec2AvailabilityZone: us-east-1d Ec2InstanceType: m4.xlarge Ec2Kernel: unavailable Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99 MachineType: Xen HVM domU PciMultimedia: ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 cirrusdrmfb 1 xen ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.2.0-23-generic root=UUID=9bd55602-81dd-4868-8cfc-b7d63f8f8d7e ro console=tty1 console=ttyS0 crashkernel=256M@0M RelatedPackageVersions: linux-res
[Kernel-packages] [Bug 1534345] Re: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking
And now we've got a second trace from the same machine with a bit more info: And one more: [14032.676085] general protection fault: [#1] SMP [14032.678409] Modules linked in: isofs xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp bridge stp llc iptable_filter ip_tables x_tables ppdev xen_fbfront fb_sys_fops intel_rapl iosf_mbi input_leds i2c_piix4 parport_pc 8250_fintek serio_raw parport mac_hid netconsole configfs autofs4 cirrus syscopyarea sysfillrect sysimgblt ttm crct10dif_pclmul crc32_pclmul aesni_intel drm_kms_helper aes_x86_64 lrw gf128mul glue_helper ablk_helper drm cryptd ixgbevf psmouse pata_acpi floppy [14032.680061] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.2.0-23-generic #28-Ubuntu [14032.680061] Hardware name: Xen HVM domU, BIOS 4.2.amazon 12/07/2015 [14032.680061] task: 81c13500 ti: 81c0 task.ti: 81c0 [14032.680061] RIP: 0010:[] [] run_timer_softirq+0x116/0x2d0 [14032.680061] RSP: 0018:8803ff203e98 EFLAGS: 00010086 [14032.680061] RAX: dead00200200 RBX: 8803ff20e9c0 RCX: 000e1785 [14032.680061] RDX: 8803ff203ec8 RSI: 8803ff21be00 RDI: 8803ff20e9c0 [14032.680061] RBP: 8803ff203f08 R08: 0001be00 R09: [14032.680061] R10: ea00036dc680 R11: R12: 00d2 [14032.680061] R13: 8172aaf0 R14: R15: 8800db71a3d0 [14032.680061] FS: () GS:8803ff20() knlGS: [14032.680061] CS: 0010 DS: ES: CR0: 80050033 [14032.680061] CR2: 7f97cd08ec64 CR3: 0003f5834000 CR4: 001406f0 [14032.680061] Stack: [14032.680061] 8803ff203eb8 8803ff20f5f8 8803ff20f3f8 8803ff20f1f8 [14032.680061] 8803ff20e9f8 8800db71a348 dead00200200 08c9e9276e6f82e9 [14032.680061] 00011fc0 0001 81c0b0c8 0001 [14032.680061] Call Trace: [14032.680061] [14032.680061] [] __do_softirq+0xf6/0x250 [14032.680061] [] irq_exit+0xa3/0xb0 [14032.680061] [] xen_evtchn_do_upcall+0x39/0x50 [14032.680061] [] xen_hvm_callback_vector+0x6b/0x70 [14032.680061] [14032.680061] [] ? get_next_timer_interrupt+0xf2/0x240 [14032.680061] [] ? native_safe_halt+0x6/0x10 [14032.680061] [] default_idle+0x1e/0xa0 [14032.680061] [] arch_cpu_idle+0xf/0x20 [14032.680061] [] default_idle_call+0x2a/0x40 [14032.680061] [] cpu_startup_entry+0x2c9/0x320 [14032.680061] [] rest_init+0x7c/0x80 [14032.680061] [] start_kernel+0x48b/0x4ac [14032.680061] [] ? early_idt_handler_array+0x120/0x120 [14032.680061] [] x86_64_start_reservations+0x2a/0x2c [14032.680061] [] x86_64_start_kernel+0x14a/0x16d [14032.680061] Code: 81 e6 00 00 20 00 48 85 d2 48 89 45 b8 0f 85 30 01 00 00 4c 89 7b 08 0f 1f 44 00 00 49 8b 07 49 8b 57 08 48 85 c0 48 89 02 74 04 <48> 89 50 08 41 f6 47 2a 10 48 b8 00 02 20 00 00 00 ad de 49 c7 [14032.680061] RIP [] run_timer_softirq+0x116/0x2d0 [14032.680061] RSP -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1534345 Title: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking Status in linux package in Ubuntu: Confirmed Bug description: Lots of details and history of the problem here: https://askubuntu.com/questions/710747/after-upgrading- to-15-10-from-15-04-ec2-webservers-have-become-very-unstable 10 of my webservers have started crashing immediately following the 15.10 upgrade. As far as what exactly defines a "crash", Instance Status Checks fail, and I can no longer SSH to the machine. Background daemons running on the system stop responding, and nothing is written to the logs. After weeks of working with the AWS team, I finally fixed a netconsole issue via "echo 7 > /proc/sys/kernel/printk" and got netconsole working properly, and finally have a trace: [21410.260077] general protection fault: [#1] SMP [21410.261976] Modules linked in: isofs xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp bridge stp llc iptable_filter ip_tables x_tables ppdev intel_rapl iosf_mbi xen_fbfront fb_sys_fops input_leds serio_raw i2c_piix4 parport_pc 8250_fintek parport mac_hid netconsole configfs autofs4 crct10dif_pclmul crc32_pclmul cirrus syscopyarea sysfillrect sysimgblt aesni_intel ttm aes_x86_64 drm_kms_helper lrw gf128mul glue_helper ablk_helper cryptd psmouse drm ixgbevf pata_acpi floppy [21410.264054] CPU: 0 PID: 26957 Comm: apache2 Not tainted 4.2.0-23-generic #28-Ubuntu [21410.264054] Hardware name: Xen HVM domU, BIOS 4.2.amazon 12/07/2015 [21410.264054] task: 8803f9809b80 ti: 8803f999c000 task.ti: 8803f999c000 [21410.264054] RIP: 0010:[]
[Kernel-packages] [Bug 1534345] Re: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking
Do you guys need a vmcore? I'm working on getting one from AWS. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1534345 Title: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking Status in linux package in Ubuntu: Confirmed Bug description: Lots of details and history of the problem here: https://askubuntu.com/questions/710747/after-upgrading- to-15-10-from-15-04-ec2-webservers-have-become-very-unstable 10 of my webservers have started crashing immediately following the 15.10 upgrade. As far as what exactly defines a "crash", Instance Status Checks fail, and I can no longer SSH to the machine. Background daemons running on the system stop responding, and nothing is written to the logs. After weeks of working with the AWS team, I finally fixed a netconsole issue via "echo 7 > /proc/sys/kernel/printk" and got netconsole working properly, and finally have a trace: [21410.260077] general protection fault: [#1] SMP [21410.261976] Modules linked in: isofs xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp bridge stp llc iptable_filter ip_tables x_tables ppdev intel_rapl iosf_mbi xen_fbfront fb_sys_fops input_leds serio_raw i2c_piix4 parport_pc 8250_fintek parport mac_hid netconsole configfs autofs4 crct10dif_pclmul crc32_pclmul cirrus syscopyarea sysfillrect sysimgblt aesni_intel ttm aes_x86_64 drm_kms_helper lrw gf128mul glue_helper ablk_helper cryptd psmouse drm ixgbevf pata_acpi floppy [21410.264054] CPU: 0 PID: 26957 Comm: apache2 Not tainted 4.2.0-23-generic #28-Ubuntu [21410.264054] Hardware name: Xen HVM domU, BIOS 4.2.amazon 12/07/2015 [21410.264054] task: 8803f9809b80 ti: 8803f999c000 task.ti: 8803f999c000 [21410.264054] RIP: 0010:[] [] run_timer_softirq+0x116/0x2d0 [21410.264054] RSP: :8803ff203e98 EFLAGS: 00010086 [21410.264054] RAX: dead00200200 RBX: 8803ff20e9c0 RCX: 8803ff203ec8 [21410.264054] RDX: 8803ff203ec8 RSI: 00011fc0 RDI: 8803ff20e9c0 [21410.264054] RBP: 8803ff203f08 R08: a77a R09: [21410.264054] R10: 0020 R11: 0004 R12: 007c [21410.264054] R13: 8172aaf0 R14: R15: 8803af955be0 [21410.264054] FS: 7fb0ce6e8780() GS:8803ff20() knlGS: [21410.264054] CS: 0010 DS: ES: CR0: 80050033 [21410.264054] CR2: 7fb0ce51e130 CR3: 0003fb233000 CR4: 001406f0 [21410.264054] Stack: [21410.264054] 8803ff203eb8 8803ff20f5f8 8803ff20f3f8 8803ff20f1f8 [21410.264054] 8803ff20e9f8 8803af955b58 dead00200200 f60fabc0 [21410.264054] 00011fc0 0001 81c0b0c8 0001 [21410.264054] Call Trace: [21410.264054] [21410.264054] [] __do_softirq+0xf6/0x250 [21410.264054] [] irq_exit+0xa3/0xb0 [21410.264054] [] xen_evtchn_do_upcall+0x39/0x50 [21410.264054] [] xen_hvm_callback_vector+0x6b/0x70 [21410.264054] [21410.264054] Code: 81 e6 00 00 20 00 48 85 d2 48 89 45 b8 0f 85 30 01 00 00 4c 89 7b 08 0f 1f 44 00 00 49 8b 07 49 8b 57 08 48 85 c0 48 89 02 74 04 <48> 89 50 08 41 f6 47 2a 10 48 b8 00 02 20 00 00 00 ad de 49 c7 [21410.264054] RIP [] run_timer_softirq+0x116/0x2d0 [21410.264054] RSP I don't have a vmcore at the moment, but I'm trying to get one from AWS and should have one in the next couple of days. This is happening frequently and repeatedly since I first upgraded to 15.10 on early December. ubuntu@xxx-web-xx:~$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 15.10 Release: 15.10 Codename: wily ubuntu@xxx-web-xx:~$ uname -a Linux xxx-web-xx 4.2.0-23-generic #28-Ubuntu SMP Sun Dec 27 17:47:31 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux ubuntu@xxx-web-xx:~$ ProblemType: Bug DistroRelease: Ubuntu 15.10 Package: linux-image-4.2.0-23-generic 4.2.0-23.28 ProcVersionSignature: User Name 4.2.0-23.28-generic 4.2.6 Uname: Linux 4.2.0-23-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Jan 14 15:42 seq crw-rw 1 root audio 116, 33 Jan 14 15:42 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.19.1-0ubuntu5 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: Date: Thu Jan 14 21:31:14 2016 Ec2AMI: ami-d5e7adbf Ec2AMIManifest: (unknown) Ec2AvailabilityZone: us-east-1d Ec2InstanceType: m4.xlarge Ec2Kernel: unavailable Ec2Ramdisk: unavailable IwConfig: Error:
[Kernel-packages] [Bug 1534345] Re: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking
Thanks for looking into this Stefan! We were completely fine with 15.05 and 3.19. If it won't break anything terribly, I can try to put 3.19, 4.0, and 4.1 on these machines, but each one crashes every 24-48 hours, so it might take me several days. Which kernel would you recommend starting with, say, 4.0 or 4.4? Another thing that I didn't find relevant before, but seems to confirm what you're saying about the per-CPU timers--AWS told me the following after a crash where I disabled my auto-reboot-on-alarm triggers: I was able to successfully get a trace - most of the vCPU were just in a halted state, so nothing there, but one had some potentially useful information: VCPU 1 rip: 810c3ef5 __pv_queued_spin_lock_slowpath+0xc5 flags: 0206 i nz p rsp: 8803ff243e78 rax: 0a2a rcx: fffa rdx: 0003 rbx: 8803f7ef2e38 rsi: 8803ff243df8 rdi: 0003 rbp: 8803ff243ea8r8: r9: 8803fe80 r10: r11: 813ef2b0 r12: 8803ff2571c0 r13: 0008 r14: 88040ffa30c0 r15: 0001 cs: 0010 ss: ds: es: fs: @ 7fc1867b8700 gs: @ 8803ff24/ cr0: 80050033 cr2: 00a8 cr3: de15f000 cr4: 001406e0 dr0: dr1: dr2: dr3: dr6: 0ff0 dr7: 0400 Code (instr addr 810c3ef5) 41 bf 01 00 00 00 48 0f af c3 48 89 45 d0 b8 00 80 00 00 eb 0b 90 83 e8 01 0f 84 d4 00 00 00 Stack: 8c2fa8473f0f2e38 8803ff2577c0 8803f7ef2e10 8803f7ef2e10 000101155691 8803ff243eb8 817f0021 8803ff243f38 816e48f4 000101155693 0040012c 0024 8803ff243ee0 8803ff243ee0 8803ff243ef0 Call Trace: [] __pv_queued_spin_lock_slowpath+0xc5 <-- [] _raw_spin_lock+0x21 [] net_rx_action+0xe4 [] __do_softirq+0xf6 [] do_softirq_own_stack+0x1c -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1534345 Title: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking Status in linux package in Ubuntu: Triaged Bug description: Lots of details and history of the problem here: https://askubuntu.com/questions/710747/after-upgrading- to-15-10-from-15-04-ec2-webservers-have-become-very-unstable 10 of my webservers have started crashing immediately following the 15.10 upgrade. As far as what exactly defines a "crash", Instance Status Checks fail, and I can no longer SSH to the machine. Background daemons running on the system stop responding, and nothing is written to the logs. After weeks of working with the AWS team, I finally fixed a netconsole issue via "echo 7 > /proc/sys/kernel/printk" and got netconsole working properly, and finally have a trace: [21410.260077] general protection fault: [#1] SMP [21410.261976] Modules linked in: isofs xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp bridge stp llc iptable_filter ip_tables x_tables ppdev intel_rapl iosf_mbi xen_fbfront fb_sys_fops input_leds serio_raw i2c_piix4 parport_pc 8250_fintek parport mac_hid netconsole configfs autofs4 crct10dif_pclmul crc32_pclmul cirrus syscopyarea sysfillrect sysimgblt aesni_intel ttm aes_x86_64 drm_kms_helper lrw gf128mul glue_helper ablk_helper cryptd psmouse drm ixgbevf pata_acpi floppy [21410.264054] CPU: 0 PID: 26957 Comm: apache2 Not tainted 4.2.0-23-generic #28-Ubuntu [21410.264054] Hardware name: Xen HVM domU, BIOS 4.2.amazon 12/07/2015 [21410.264054] task: 8803f9809b80 ti: 8803f999c000 task.ti: 8803f999c000 [21410.264054] RIP: 0010:[] [] run_timer_softirq+0x116/0x2d0 [21410.264054] RSP: :8803ff203e98 EFLAGS: 00010086 [21410.264054] RAX: dead00200200 RBX: 8803ff20e9c0 RCX: 8803ff203ec8 [21410.264054] RDX: 8803ff203ec8 RSI: 00011fc0 RDI: 8803ff20e9c0 [21410.264054] RBP: 8803ff203f08 R08: a77a R09: [21410.264054] R10: 0020 R11: 0004 R12: 007c [21410.264054] R13: 8172aaf0 R14: R15: 8803af955be0 [21410.264054] FS: 7fb0ce6e8780() GS:8803ff20() knlGS: [21410.264054] CS: 0010 DS: ES: CR0: 80050033 [21410.264054] CR2: 7fb0ce51e130 CR3: 0003fb233000 CR4: 001406f0 [21410.264054] Stack: [21410.264054] 8803ff203eb8 8803ff20f5f8 8803ff20f3f8 8803ff20f1f8 [21410.264054] 8803ff20e9f8 8803af955b58 dead00200200 f60fabc0 [21410.264054] 00011fc0 0001 81c0b0c8 0001 [21410.264054] Call
[Kernel-packages] [Bug 1534345] Re: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking
Hey Stefan, Just wanted to update you that I've installed the Mainline 4.0.9 kernel (4.0.9-040009-generic #201507212131) on all affected machines, finishing about an hour ago, from https://wiki.ubuntu.com/Kernel/MainlineBuilds. I've still had two crashes in the hour since the downgrade. I don't know enough about kernel code to provide feedback on the detail above, but thanks for looking into this! I doubt that this could be related, but maybe it will help... We also appear to have one of our C utilities (developed in-house) crashing a few times a day: xxx.log:[ 4740.992084] traps: checkport[6624] general protection ip:4c3d07 sp:7f8441ce8e20 error:0 in checkport[40+ff000] xxx.log:[60087.824087] traps: checkport[2203] general protection ip:4c3d07 sp:7fcc7df61e20 error:0 in checkport[40+ff000] xxx.log:[ 5375.784101] traps: checkport[10795] general protection ip:4c3d07 sp:7f96ebcbde20 error:0 in checkport[40+ff000] in addition to numerous segfaults: xxx.log:[ 1354.456792] checkport[1644]: segfault at a8 ip 0049be3a sp 7fea90080da0 error 4 in checkport[40+ff000] xxx.log:[ 1393.894368] checkport[2754]: segfault at a8 ip 0049be3a sp 7f082e3c6da0 error 4 in checkport[40+ff000] xxx.log:[ 2474.260293] checkport[24996]: segfault at a8 ip 0049be3a sp 7f569f6dcda0 error 4 in checkport[40+ff000] xxx.log:[ 4170.613174] checkport[23278]: segfault at a8 ip 0049be3a sp 7fcf9b580da0 error 4 in checkport[40+ff000] The developers are looking into this on our end, but, we still had the same version of the tool and the same number of segfaults on our 3.19 kernel on 15.04 and it never triggered a kernel crash. Is it possible that invalid memory access in this binary could be triggering a kernel GPF? Or would that memory be protected? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1534345 Title: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking Status in linux package in Ubuntu: Triaged Bug description: Lots of details and history of the problem here: https://askubuntu.com/questions/710747/after-upgrading- to-15-10-from-15-04-ec2-webservers-have-become-very-unstable 10 of my webservers have started crashing immediately following the 15.10 upgrade. As far as what exactly defines a "crash", Instance Status Checks fail, and I can no longer SSH to the machine. Background daemons running on the system stop responding, and nothing is written to the logs. After weeks of working with the AWS team, I finally fixed a netconsole issue via "echo 7 > /proc/sys/kernel/printk" and got netconsole working properly, and finally have a trace: [21410.260077] general protection fault: [#1] SMP [21410.261976] Modules linked in: isofs xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp bridge stp llc iptable_filter ip_tables x_tables ppdev intel_rapl iosf_mbi xen_fbfront fb_sys_fops input_leds serio_raw i2c_piix4 parport_pc 8250_fintek parport mac_hid netconsole configfs autofs4 crct10dif_pclmul crc32_pclmul cirrus syscopyarea sysfillrect sysimgblt aesni_intel ttm aes_x86_64 drm_kms_helper lrw gf128mul glue_helper ablk_helper cryptd psmouse drm ixgbevf pata_acpi floppy [21410.264054] CPU: 0 PID: 26957 Comm: apache2 Not tainted 4.2.0-23-generic #28-Ubuntu [21410.264054] Hardware name: Xen HVM domU, BIOS 4.2.amazon 12/07/2015 [21410.264054] task: 8803f9809b80 ti: 8803f999c000 task.ti: 8803f999c000 [21410.264054] RIP: 0010:[] [] run_timer_softirq+0x116/0x2d0 [21410.264054] RSP: :8803ff203e98 EFLAGS: 00010086 [21410.264054] RAX: dead00200200 RBX: 8803ff20e9c0 RCX: 8803ff203ec8 [21410.264054] RDX: 8803ff203ec8 RSI: 00011fc0 RDI: 8803ff20e9c0 [21410.264054] RBP: 8803ff203f08 R08: a77a R09: [21410.264054] R10: 0020 R11: 0004 R12: 007c [21410.264054] R13: 8172aaf0 R14: R15: 8803af955be0 [21410.264054] FS: 7fb0ce6e8780() GS:8803ff20() knlGS: [21410.264054] CS: 0010 DS: ES: CR0: 80050033 [21410.264054] CR2: 7fb0ce51e130 CR3: 0003fb233000 CR4: 001406f0 [21410.264054] Stack: [21410.264054] 8803ff203eb8 8803ff20f5f8 8803ff20f3f8 8803ff20f1f8 [21410.264054] 8803ff20e9f8 8803af955b58 dead00200200 f60fabc0 [21410.264054] 00011fc0 0001 81c0b0c8 0001 [21410.264054] Call Trace: [21410.264054] [21410.264054] [] __do_softirq+0xf6/0x250 [21410.264054] [] irq_exit+0xa3/0xb0 [21410.264054] [] xen_evtchn_do_upcall+0x39/0x50 [21410.264054] [] xen_hvm_cal
[Kernel-packages] [Bug 1534345] Re: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking
And I should also point out that ifquery is crashing on boot most of the time on these systems: xxx.log:[3.909680] ifquery[378]: segfault at 1 ip 00403187 sp 7fff7078d8c0 error 4 in ifup[40+d000] xxx.log:[3.008003] ifquery[380]: segfault at 1 ip 00403187 sp 7ffdf6935bc0 error 4 in ifup[40+d000] xxx.log:[2.647084] ifquery[370]: segfault at 1 ip 00403187 sp 7ffda45292f0 error 4 in ifup[40+d000] xxx.log:[2.868947] ifquery[370]: segfault at 1 ip 00403187 sp 7ffcd9140230 error 4 in ifup[40+d000] xxx.log:[2.846450] ifquery[398]: segfault at 1 ip 00403187 sp 7ffc5da3d670 error 4 in ifup[40+d000] xxx.log:[3.070464] ifquery[372]: segfault at 1 ip 00403187 sp 7fff78691b90 error 4 in ifup[40+d000] But I've been unable to get a coredump and it doesn't save a .crash and I can't reproduce it once the system is online. And this "checkport" utility couldn't possibly be running so early in boot--it's triggered by a web request and represents a relatively low percentage of traffic. It's unlikely that it would be triggered at all in the first 30 seconds after boot. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1534345 Title: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking Status in linux package in Ubuntu: Triaged Bug description: Lots of details and history of the problem here: https://askubuntu.com/questions/710747/after-upgrading- to-15-10-from-15-04-ec2-webservers-have-become-very-unstable 10 of my webservers have started crashing immediately following the 15.10 upgrade. As far as what exactly defines a "crash", Instance Status Checks fail, and I can no longer SSH to the machine. Background daemons running on the system stop responding, and nothing is written to the logs. After weeks of working with the AWS team, I finally fixed a netconsole issue via "echo 7 > /proc/sys/kernel/printk" and got netconsole working properly, and finally have a trace: [21410.260077] general protection fault: [#1] SMP [21410.261976] Modules linked in: isofs xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp bridge stp llc iptable_filter ip_tables x_tables ppdev intel_rapl iosf_mbi xen_fbfront fb_sys_fops input_leds serio_raw i2c_piix4 parport_pc 8250_fintek parport mac_hid netconsole configfs autofs4 crct10dif_pclmul crc32_pclmul cirrus syscopyarea sysfillrect sysimgblt aesni_intel ttm aes_x86_64 drm_kms_helper lrw gf128mul glue_helper ablk_helper cryptd psmouse drm ixgbevf pata_acpi floppy [21410.264054] CPU: 0 PID: 26957 Comm: apache2 Not tainted 4.2.0-23-generic #28-Ubuntu [21410.264054] Hardware name: Xen HVM domU, BIOS 4.2.amazon 12/07/2015 [21410.264054] task: 8803f9809b80 ti: 8803f999c000 task.ti: 8803f999c000 [21410.264054] RIP: 0010:[] [] run_timer_softirq+0x116/0x2d0 [21410.264054] RSP: :8803ff203e98 EFLAGS: 00010086 [21410.264054] RAX: dead00200200 RBX: 8803ff20e9c0 RCX: 8803ff203ec8 [21410.264054] RDX: 8803ff203ec8 RSI: 00011fc0 RDI: 8803ff20e9c0 [21410.264054] RBP: 8803ff203f08 R08: a77a R09: [21410.264054] R10: 0020 R11: 0004 R12: 007c [21410.264054] R13: 8172aaf0 R14: R15: 8803af955be0 [21410.264054] FS: 7fb0ce6e8780() GS:8803ff20() knlGS: [21410.264054] CS: 0010 DS: ES: CR0: 80050033 [21410.264054] CR2: 7fb0ce51e130 CR3: 0003fb233000 CR4: 001406f0 [21410.264054] Stack: [21410.264054] 8803ff203eb8 8803ff20f5f8 8803ff20f3f8 8803ff20f1f8 [21410.264054] 8803ff20e9f8 8803af955b58 dead00200200 f60fabc0 [21410.264054] 00011fc0 0001 81c0b0c8 0001 [21410.264054] Call Trace: [21410.264054] [21410.264054] [] __do_softirq+0xf6/0x250 [21410.264054] [] irq_exit+0xa3/0xb0 [21410.264054] [] xen_evtchn_do_upcall+0x39/0x50 [21410.264054] [] xen_hvm_callback_vector+0x6b/0x70 [21410.264054] [21410.264054] Code: 81 e6 00 00 20 00 48 85 d2 48 89 45 b8 0f 85 30 01 00 00 4c 89 7b 08 0f 1f 44 00 00 49 8b 07 49 8b 57 08 48 85 c0 48 89 02 74 04 <48> 89 50 08 41 f6 47 2a 10 48 b8 00 02 20 00 00 00 ad de 49 c7 [21410.264054] RIP [] run_timer_softirq+0x116/0x2d0 [21410.264054] RSP I don't have a vmcore at the moment, but I'm trying to get one from AWS and should have one in the next couple of days. This is happening frequently and repeatedly since I first upgraded to 15.10 on early December. ubuntu@xxx-web-xx:~$ lsb_release -a No LSB modules are available.
[Kernel-packages] [Bug 1534345] Re: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking
I have now downgraded all of these systems to 3.19.8-031908-generic (Mainline). We'll know in a couple of days if this fixed it (or as soon as a few hours if it didn't fix it, possibly). I'll update when I know anything else; good so far! 4.0.9 definitely didn't fix it completely, but MAY have made it less frequent. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1534345 Title: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking Status in linux package in Ubuntu: Triaged Bug description: Lots of details and history of the problem here: https://askubuntu.com/questions/710747/after-upgrading- to-15-10-from-15-04-ec2-webservers-have-become-very-unstable 10 of my webservers have started crashing immediately following the 15.10 upgrade. As far as what exactly defines a "crash", Instance Status Checks fail, and I can no longer SSH to the machine. Background daemons running on the system stop responding, and nothing is written to the logs. After weeks of working with the AWS team, I finally fixed a netconsole issue via "echo 7 > /proc/sys/kernel/printk" and got netconsole working properly, and finally have a trace: [21410.260077] general protection fault: [#1] SMP [21410.261976] Modules linked in: isofs xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp bridge stp llc iptable_filter ip_tables x_tables ppdev intel_rapl iosf_mbi xen_fbfront fb_sys_fops input_leds serio_raw i2c_piix4 parport_pc 8250_fintek parport mac_hid netconsole configfs autofs4 crct10dif_pclmul crc32_pclmul cirrus syscopyarea sysfillrect sysimgblt aesni_intel ttm aes_x86_64 drm_kms_helper lrw gf128mul glue_helper ablk_helper cryptd psmouse drm ixgbevf pata_acpi floppy [21410.264054] CPU: 0 PID: 26957 Comm: apache2 Not tainted 4.2.0-23-generic #28-Ubuntu [21410.264054] Hardware name: Xen HVM domU, BIOS 4.2.amazon 12/07/2015 [21410.264054] task: 8803f9809b80 ti: 8803f999c000 task.ti: 8803f999c000 [21410.264054] RIP: 0010:[] [] run_timer_softirq+0x116/0x2d0 [21410.264054] RSP: :8803ff203e98 EFLAGS: 00010086 [21410.264054] RAX: dead00200200 RBX: 8803ff20e9c0 RCX: 8803ff203ec8 [21410.264054] RDX: 8803ff203ec8 RSI: 00011fc0 RDI: 8803ff20e9c0 [21410.264054] RBP: 8803ff203f08 R08: a77a R09: [21410.264054] R10: 0020 R11: 0004 R12: 007c [21410.264054] R13: 8172aaf0 R14: R15: 8803af955be0 [21410.264054] FS: 7fb0ce6e8780() GS:8803ff20() knlGS: [21410.264054] CS: 0010 DS: ES: CR0: 80050033 [21410.264054] CR2: 7fb0ce51e130 CR3: 0003fb233000 CR4: 001406f0 [21410.264054] Stack: [21410.264054] 8803ff203eb8 8803ff20f5f8 8803ff20f3f8 8803ff20f1f8 [21410.264054] 8803ff20e9f8 8803af955b58 dead00200200 f60fabc0 [21410.264054] 00011fc0 0001 81c0b0c8 0001 [21410.264054] Call Trace: [21410.264054] [21410.264054] [] __do_softirq+0xf6/0x250 [21410.264054] [] irq_exit+0xa3/0xb0 [21410.264054] [] xen_evtchn_do_upcall+0x39/0x50 [21410.264054] [] xen_hvm_callback_vector+0x6b/0x70 [21410.264054] [21410.264054] Code: 81 e6 00 00 20 00 48 85 d2 48 89 45 b8 0f 85 30 01 00 00 4c 89 7b 08 0f 1f 44 00 00 49 8b 07 49 8b 57 08 48 85 c0 48 89 02 74 04 <48> 89 50 08 41 f6 47 2a 10 48 b8 00 02 20 00 00 00 ad de 49 c7 [21410.264054] RIP [] run_timer_softirq+0x116/0x2d0 [21410.264054] RSP I don't have a vmcore at the moment, but I'm trying to get one from AWS and should have one in the next couple of days. This is happening frequently and repeatedly since I first upgraded to 15.10 on early December. ubuntu@xxx-web-xx:~$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 15.10 Release: 15.10 Codename: wily ubuntu@xxx-web-xx:~$ uname -a Linux xxx-web-xx 4.2.0-23-generic #28-Ubuntu SMP Sun Dec 27 17:47:31 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux ubuntu@xxx-web-xx:~$ ProblemType: Bug DistroRelease: Ubuntu 15.10 Package: linux-image-4.2.0-23-generic 4.2.0-23.28 ProcVersionSignature: User Name 4.2.0-23.28-generic 4.2.6 Uname: Linux 4.2.0-23-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Jan 14 15:42 seq crw-rw 1 root audio 116, 33 Jan 14 15:42 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.19.1-0ubuntu5 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/s
[Kernel-packages] [Bug 1534345] Re: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking
3.19 crashed as well, just now. This is surprising. I don't have a trace, as the kernel downgrades messed up netconsole somehow, but I'll try to get it working again. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1534345 Title: Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking Status in linux package in Ubuntu: Triaged Bug description: Lots of details and history of the problem here: https://askubuntu.com/questions/710747/after-upgrading- to-15-10-from-15-04-ec2-webservers-have-become-very-unstable 10 of my webservers have started crashing immediately following the 15.10 upgrade. As far as what exactly defines a "crash", Instance Status Checks fail, and I can no longer SSH to the machine. Background daemons running on the system stop responding, and nothing is written to the logs. After weeks of working with the AWS team, I finally fixed a netconsole issue via "echo 7 > /proc/sys/kernel/printk" and got netconsole working properly, and finally have a trace: [21410.260077] general protection fault: [#1] SMP [21410.261976] Modules linked in: isofs xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp bridge stp llc iptable_filter ip_tables x_tables ppdev intel_rapl iosf_mbi xen_fbfront fb_sys_fops input_leds serio_raw i2c_piix4 parport_pc 8250_fintek parport mac_hid netconsole configfs autofs4 crct10dif_pclmul crc32_pclmul cirrus syscopyarea sysfillrect sysimgblt aesni_intel ttm aes_x86_64 drm_kms_helper lrw gf128mul glue_helper ablk_helper cryptd psmouse drm ixgbevf pata_acpi floppy [21410.264054] CPU: 0 PID: 26957 Comm: apache2 Not tainted 4.2.0-23-generic #28-Ubuntu [21410.264054] Hardware name: Xen HVM domU, BIOS 4.2.amazon 12/07/2015 [21410.264054] task: 8803f9809b80 ti: 8803f999c000 task.ti: 8803f999c000 [21410.264054] RIP: 0010:[] [] run_timer_softirq+0x116/0x2d0 [21410.264054] RSP: :8803ff203e98 EFLAGS: 00010086 [21410.264054] RAX: dead00200200 RBX: 8803ff20e9c0 RCX: 8803ff203ec8 [21410.264054] RDX: 8803ff203ec8 RSI: 00011fc0 RDI: 8803ff20e9c0 [21410.264054] RBP: 8803ff203f08 R08: a77a R09: [21410.264054] R10: 0020 R11: 0004 R12: 007c [21410.264054] R13: 8172aaf0 R14: R15: 8803af955be0 [21410.264054] FS: 7fb0ce6e8780() GS:8803ff20() knlGS: [21410.264054] CS: 0010 DS: ES: CR0: 80050033 [21410.264054] CR2: 7fb0ce51e130 CR3: 0003fb233000 CR4: 001406f0 [21410.264054] Stack: [21410.264054] 8803ff203eb8 8803ff20f5f8 8803ff20f3f8 8803ff20f1f8 [21410.264054] 8803ff20e9f8 8803af955b58 dead00200200 f60fabc0 [21410.264054] 00011fc0 0001 81c0b0c8 0001 [21410.264054] Call Trace: [21410.264054] [21410.264054] [] __do_softirq+0xf6/0x250 [21410.264054] [] irq_exit+0xa3/0xb0 [21410.264054] [] xen_evtchn_do_upcall+0x39/0x50 [21410.264054] [] xen_hvm_callback_vector+0x6b/0x70 [21410.264054] [21410.264054] Code: 81 e6 00 00 20 00 48 85 d2 48 89 45 b8 0f 85 30 01 00 00 4c 89 7b 08 0f 1f 44 00 00 49 8b 07 49 8b 57 08 48 85 c0 48 89 02 74 04 <48> 89 50 08 41 f6 47 2a 10 48 b8 00 02 20 00 00 00 ad de 49 c7 [21410.264054] RIP [] run_timer_softirq+0x116/0x2d0 [21410.264054] RSP I don't have a vmcore at the moment, but I'm trying to get one from AWS and should have one in the next couple of days. This is happening frequently and repeatedly since I first upgraded to 15.10 on early December. ubuntu@xxx-web-xx:~$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 15.10 Release: 15.10 Codename: wily ubuntu@xxx-web-xx:~$ uname -a Linux xxx-web-xx 4.2.0-23-generic #28-Ubuntu SMP Sun Dec 27 17:47:31 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux ubuntu@xxx-web-xx:~$ ProblemType: Bug DistroRelease: Ubuntu 15.10 Package: linux-image-4.2.0-23-generic 4.2.0-23.28 ProcVersionSignature: User Name 4.2.0-23.28-generic 4.2.6 Uname: Linux 4.2.0-23-generic x86_64 AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Jan 14 15:42 seq crw-rw 1 root audio 116, 33 Jan 14 15:42 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.19.1-0ubuntu5 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: Date: Thu Jan 14 21:31:14 2016 Ec2AMI: ami-d5e7adbf Ec2AMIManifest: (unknown) Ec2AvailabilityZone: us-east-
[Kernel-packages] [Bug 1573231] Re: Kernel Panic on EC2 After Upgrading from 14.04 to 16.04 via do-release-upgrade -d
Sorry, I don't think there's any possible way to do it. The machine won't boot. I could try to reproduce the issue again on a different VM, but I still wouldn't be able to collect logs. ** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1573231 Title: Kernel Panic on EC2 After Upgrading from 14.04 to 16.04 via do- release-upgrade -d Status in linux package in Ubuntu: Confirmed Bug description: [0.00] Initializing cgroup subsys cpuset [0.00] Initializing cgroup subsys cpu [0.00] Initializing cgroup subsys cpuacct [0.00] Linux version 4.4.0-21-generic (buildd@lgw01-21) (gcc version 5.3.1 20160413 (Ubuntu 5.3.1-14ubuntu2) ) #37-Ubuntu SMP Mon Apr 18 18:33:37 UTC 2016 (Ubuntu 4.4.0-21.37-generic 4.4.6) [0.00] Command line: BOOT_IMAGE=/boot/vmlinuz-4.4.0-21-generic root=UUID=8ea401db-b84b-4cd6-a628-d72f30bbf1e5 ro console=tty1 console=ttyS0 [0.00] KERNEL supported cpus: [0.00] Intel GenuineIntel [0.00] AMD AuthenticAMD [0.00] Centaur CentaurHauls [0.00] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [0.00] x86/fpu: Supporting XSAVE feature 0x01: 'x87 floating point registers' [0.00] x86/fpu: Supporting XSAVE feature 0x02: 'SSE registers' [0.00] x86/fpu: Supporting XSAVE feature 0x04: 'AVX registers' [0.00] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [0.00] x86/fpu: Using 'eager' FPU context switches. [0.00] e820: BIOS-provided physical RAM map: [0.00] BIOS-e820: [mem 0x-0x0009dfff] usable [0.00] BIOS-e820: [mem 0x0009e000-0x0009] reserved [0.00] BIOS-e820: [mem 0x000e-0x000f] reserved [0.00] BIOS-e820: [mem 0x0010-0xefff] usable [0.00] BIOS-e820: [mem 0xfc00-0x] reserved [0.00] BIOS-e820: [mem 0x0001-0x0001efff] usable [0.00] NX (Execute Disable) protection: active [0.00] SMBIOS 2.4 present. [0.00] Hypervisor detected: Xen [0.00] Xen version 4.2. [0.00] Netfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated NICs. [0.00] Blkfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated disks. [0.00] You might have to change the root device [0.00] from /dev/hd[a-d] to /dev/xvd[a-d] [0.00] in your root= kernel command line option [0.00] e820: last_pfn = 0x1f max_arch_pfn = 0x4 [0.00] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WC UC- WT [0.00] e820: last_pfn = 0xf max_arch_pfn = 0x4 [0.00] found SMP MP-table at [mem 0x000fbba0-0x000fbbaf] mapped at [880fbba0] [0.00] Scanning 1 areas for low memory corruption [0.00] RAMDISK: [mem 0x3407e000-0x36036fff] [0.00] ACPI: Early table checksum verification disabled [0.00] ACPI: RSDP 0x000EA020 24 (v02 Xen ) [0.00] ACPI: XSDT 0xFC00F5A0 54 (v01 XenHVM HVML ) [0.00] ACPI: FACP 0xFC00F260 F4 (v04 XenHVM HVML ) [0.00] ACPI: DSDT 0xFC0035E0 00BBF6 (v02 XenHVM INTL 20090123) [0.00] ACPI: FACS 0xFC0035A0 40 [0.00] ACPI: FACS 0xFC0035A0 40 [0.00] ACPI: APIC 0xFC00F360 D8 (v02 XenHVM HVML ) [0.00] ACPI: HPET 0xFC00F4B0 38 (v01 XenHVM HVML ) [0.00] ACPI: WAET 0xFC00F4F0 28 (v01 XenHVM HVML ) [0.00] ACPI: SSDT 0xFC00F520 31 (v02 XenHVM INTL 20090123) [0.00] ACPI: SSDT 0xFC00F560 31 (v02 XenHVM INTL 20090123) [0.00] No NUMA configuration found [0.00] Faking a node at [mem 0x-0x0001efff] [0.00] NODE_DATA(0) allocated [mem 0x1efff7000-0x1efffbfff] [0.00] Zone ranges: [0.00] DMA [mem 0x1000-0x00ff] [0.00] DMA32[mem 0x0100-0x] [0.00] Normal [mem 0x0001-0x0001efff] [0.00] Device empty [0.00] Movable zone start for each node [0.00] Early memory node ranges [0.00] node 0: [mem 0x1000-0x0009dfff] [0.00] node 0: [mem 0x0010-0xef
