[Bug 705562] Re: ami-6836dc01 8.04 32 bit AMI kernel lock bug
Stefan, so far everything has been going great in our testing. We are rolling this out on a larger scale this week, I will let you know how things go. Thank you! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/705562 Title: ami-6836dc01 8.04 32 bit AMI kernel lock bug To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/705562/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 705562] Re: ami-6836dc01 8.04 32 bit AMI kernel lock bug
I was able to bundle an AMI for testing with this patched kernel. Unfortunately we are still experiencing the same behavior as before with the original kernel. Attached is the console output from one of the servers I reproduced the bug on. This was a c1.medium. ** Attachment added: "console.txt" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/705562/+attachment/2069723/+files/console.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/705562 Title: ami-6836dc01 8.04 32 bit AMI kernel lock bug -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 705562] Re: ami-6836dc01 8.04 32 bit AMI kernel lock bug
I can confirm that the proposed kernel does solve the problem. We have been running it for ~36 hours with no problems. In the past the problem has been triggered within 24 hours. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/705562 Title: ami-6836dc01 8.04 32 bit AMI kernel lock bug To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/705562/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 705562] Re: ami-6836dc01 8.04 32 bit AMI kernel lock bug
Stefan, thank you for keeping at this. I will test this and get back to you. Thanks, Joey -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/705562 Title: ami-6836dc01 8.04 32 bit AMI kernel lock bug To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/705562/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 705562] Re: ami-6836dc01 8.04 32 bit AMI kernel lock bug
We have been running this in production for ~2 weeks now and have begun rolling this out on the rest of our servers. It has resolved the problems that we had previously been seeing with random crashes, soft/hard lockups. Thank you! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/705562 Title: ami-6836dc01 8.04 32 bit AMI kernel lock bug To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/705562/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 705562] Re: ami-6836dc01 8.04 32 bit AMI kernel lock bug
I have been using the one from your people page. I will attempt to bundle up the proposed one soon and start testing. Will get back to you. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/705562 Title: ami-6836dc01 8.04 32 bit AMI kernel lock bug To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/705562/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 705562] Re: ami-6836dc01 8.04 32 bit AMI kernel lock bug
Just to keep things updated: I've bundled up a new AMI for testing using the proposed kernel (2.6.24-29.91). We have launched a few servers in production using this kernel and will update in a couple days with the status. (So far so good though.) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/705562 Title: ami-6836dc01 8.04 32 bit AMI kernel lock bug To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/705562/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 137472] Re: stunnel4 init.d script reports startup failures with pid=error message even when not so
This bug was reported in 2007, confirmed in 2008 and is still not fixed? I can confirm this is still affecting Ubuntu 8.04: ii stunnel4 3:4.21-1Universal SSL tunnel for network daemons Any plans to fix this? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/137472 Title: stunnel4 init.d script reports startup failures with pid=error message even when not so To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/stunnel4/+bug/137472/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 776642] Re: segfaults from 5.2.4-2ubuntu5.15
Is there an ETA of an official release of this fix? Within 24 hours? Within a week? We are trying to make plans as to whether or not to configure a temporary work around. Thanks for the help, Joey -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/776642 Title: segfaults from 5.2.4-2ubuntu5.15 -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 705562] Re: ami-6836dc01 8.04 32 bit AMI kernel lock bug
I am able to reproduce this error on a server that is not running GFS at all. Attached is the output of the server when it crashed. The server is running the following services: -puppet -mysql -php management scripts -nagios checks It seems that the commonalities between the two servers affected are: -puppet -php management scripts -nagios checks ** Attachment added: "server-39.kern.log" https://bugs.launchpad.net/ubuntu/hardy/+source/linux/+bug/705562/+attachment/2113245/+files/server-39.kern.log -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/705562 Title: ami-6836dc01 8.04 32 bit AMI kernel lock bug -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 705562] Re: ami-6836dc01 8.04 32 bit AMI kernel lock bug
It may be possible for us to setup an NFS export to test this, this will take a little bit of time so I will need to get back to you on this. In the meantime below is the output of lsmod on the instance I have been using to test this. We do use XFS for these file-systems. If you need additional information please let me know. ~# lsmod Module Size Used by dm_crypt 16772 0 crypto_blkcipher 20868 1 dm_crypt sha256_generic 16256 0 iptable_filter 6656 0 ip_tables 15504 1 iptable_filter x_tables 18692 1 ip_tables fbcon 43040 0 tileblit6656 1 fbcon font 12288 1 fbcon bitblit 9728 1 fbcon softcursor 5888 1 bitblit uvesafb30436 0 xfs 551336 0 binfmt_misc12552 0 dm_multipath 20364 0 dm_mod 58824 2 dm_crypt,dm_multipath scsi_dh10884 1 dm_multipath scsi_mod 149964 1 scsi_dh ipv6 256804 18 af_packet 21376 2 loop 19084 0 fuse 56604 3 ext3 127880 2 jbd53268 1 ext3 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/705562 Title: ami-6836dc01 8.04 32 bit AMI kernel lock bug -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 790532] Re: pam upgrade broke cron
We are experiencing the same issue while running these 8.04 AMIs on AWS (ami-6836dc01, ami-0c36dc65). After upgrading PAM, the cron service has stopped responding on all of our servers. Restarting the service seems to fix the issue. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/790532 Title: pam upgrade broke cron -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 705562] Re: ami-6836dc01 8.04 32 bit AMI kernel lock bug
Sorry for not getting back to you sooner on this, the past few days have been a little bumpy operationally. I was able to package a new ami to test your v2 kernel image with. I have only run one server so far with it, but it has taken a little longer to crash with output. Attached is the entire console output I receive on this server when it crashes. This is the same environment that I have been testing the other kernel images in. Just some more info about it in case it helps us debug this: - The server has apache running and is receiving traffic. It primarily serves PHP code which may query an external MySQL server. - There are PHP cron scripts that run to manage the server - Periodic Nagios checks are run that check the health/status of the server. These are perl/php/bash - Puppet runs periodically to check to ensure configuration consistency - SVN is frequently run checking repositories for updates - rsync is frequently run - A network filesystem is mounted using the Gluster client - No EBS volumes are attached There is a lot going on, so I'm sure its going to be a challenge trying to narrow this down. If there is anything else I can describe about the system that will help us let me know. ** Attachment added: "console-output-v2-image.txt" https://bugs.launchpad.net/ubuntu/hardy/+source/linux/+bug/705562/+attachment/2090759/+files/console-output-v2-image.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/705562 Title: ami-6836dc01 8.04 32 bit AMI kernel lock bug -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 705562] Re: ami-6836dc01 8.04 32 bit AMI kernel lock bug
I was able to get another server to produce some more output. Attached is the kern.log I was able to grab before it hard locked. ** Attachment added: "kern.log" https://bugs.launchpad.net/ubuntu/hardy/+source/linux/+bug/705562/+attachment/2090773/+files/kern.log -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/705562 Title: ami-6836dc01 8.04 32 bit AMI kernel lock bug -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 705562] Re: ami-6836dc01 8.04 32 bit AMI kernel lock bug
One other thing that I just thought of is that I am able to reproduce the bug more quickly when load is applied to the server. When I am running an idle server in development I noticed it took substantially longer to surface the bug. When I launch one in production and more load is applied to it, it seems to happen within an hour. I can send you some of our Gluster packages and configurations so you can get that setup easily. Please let me know a good place to share these with you. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/705562 Title: ami-6836dc01 8.04 32 bit AMI kernel lock bug -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 705562] Re: ami-6836dc01 8.04 32 bit AMI kernel lock bug
I have sent you the configuration files directly. One thing to note that is odd about this entire situation is that I have not seen this issue on other instance sizes in AWS. The other 32 bit type m1.small does not experience the same problem, and none of our 64 bit instances have this issue either, it is confined to the c1.medium type. I'm curious what is so different between the m1.small / c1.medium that would be triggering this issue. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/705562 Title: ami-6836dc01 8.04 32 bit AMI kernel lock bug -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 705562] Re: ami-6836dc01 8.04 32 bit AMI kernel lock bug
The output I attached cuts off mid-stream I believe because the server hard locks and stops logging anything to the console. I am currently trying to reproduce the output, but the servers keep locking up before spitting out anything useful. Currently I am able to reproduce the bug by launching one of our managed web servers in production on a c1.medium. A server I am reliably able to reproduce this on is: - apache2 - php5 - ruby 1.8 - puppet (If you'd like specific package versions I'd be happy to private message them to you, let me know) There are php/ruby/puppet management scripts running in cron in addition to apache. Within ~5-20 minutes the server will lock up and stop responding to any services. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/705562 Title: ami-6836dc01 8.04 32 bit AMI kernel lock bug -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
