[Bug 1078926] Re: raring instance failed to find EC2 datasource

2013-01-11 Thread Steve Langasek
There was a lot of visual inspection of the mountall --verbose output and upstart output, to examine the ordering / timing of the mounting and mounted events. I was heavily reliant on Scott's help to produce useful logs. -- You received this bug notification because you are a member of Ubuntu Bu

[Bug 1078926] Re: raring instance failed to find EC2 datasource

2013-01-11 Thread Margarita Manterola
Hi, I'm encountering either this problem or a very similar one (mountall gets stuck, even if all the fs are already mounted). The plymouth bug makes it very hard to debug this. How did you end up debugging the problem? -- You received this bug notification because you are a member of Ubuntu Bu

[Bug 1078926] Re: raring instance failed to find EC2 datasource

2012-12-11 Thread Launchpad Bug Tracker
This bug was fixed in the package mountall - 2.42ubuntu0.2 --- mountall (2.42ubuntu0.2) quantal-proposed; urgency=low * Fix a further remaining issue from 2.41: by fixing the missing 'mounted' events in 2.42ubuntu0.1, we have again introduced a case where mountall may block

[Bug 1078926] Re: raring instance failed to find EC2 datasource

2012-12-05 Thread Scott Moser
I've verified mountall in: quantal: 2.42ubuntu0.2 precise: 2.36.2 with respect to problem that originally opened this bug. At this point I've launched 20+ instances of each and have not seen similar failure failure. In raring, failure rates were ~50% in the same openstack cluster. -- You r

[Bug 1078926] Re: raring instance failed to find EC2 datasource

2012-12-05 Thread Steve Langasek
Marking verified based on comment #18. ** Tags removed: verification-needed ** Tags added: verification-done -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1078926 Title: raring instance failed to f

[Bug 1078926] Re: raring instance failed to find EC2 datasource

2012-12-05 Thread Steve Langasek
In practice since this bug was only present in precise-proposed, this is now fixed for precise. Further verification of the SRU will be handled on the other existing bug reports. ** Changed in: mountall (Ubuntu Precise) Status: Fix Committed => Fix Released -- You received this bug notif

[Bug 1078926] Re: raring instance failed to find EC2 datasource

2012-12-05 Thread Scott Moser
The output at [1] compared to the output at [2] pretty much verifies this in raring as "fix released". Yellow actually means "passed". [1] https://jenkins.qa.ubuntu.com/view/Raring/view/Smoke%20Testing/job/raring-server-ec2-daily/8/ [2] https://jenkins.qa.ubuntu.com/view/Raring/view/Smoke%20Te

[Bug 1078926] Re: raring instance failed to find EC2 datasource

2012-12-03 Thread Launchpad Bug Tracker
This bug was fixed in the package mountall - 2.46 --- mountall (2.46) unstable; urgency=low * Fix a further remaining issue from 2.41: by fixing the missing 'mounted' events in 2.43, we have again introduced a case where mountall may block waiting for 'mounted MOUNTPOINT=/'

[Bug 1078926] Re: raring instance failed to find EC2 datasource

2012-12-03 Thread Adam Conrad
Hello Scott, or anyone else affected, Accepted mountall into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/mountall/2.42ubuntu0.2 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://w

[Bug 1078926] Re: raring instance failed to find EC2 datasource

2012-12-03 Thread Scott Moser
Using 2.45 + this patch I've tested ~ 20 instances and gotten no failures. failure rates previously were ~ 50%. ** Patch added: "patch as per vorlon is 2.45->2.46" https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/1078926/+attachment/3451094/+files/mountall-more-asyncer.patch -- You re

[Bug 1078926] Re: raring instance failed to find EC2 datasource

2012-12-03 Thread Steve Langasek
** Description changed: + [Impact] + The previous SRU, while fixing the problem it was intended to fix, partially reintroduced the problem from before 2.41 where some filesystem events would end up blocking on one another when they shouldn't. This is particularly noticeable for filesystems tha

[Bug 1078926] Re: raring instance failed to find EC2 datasource

2012-12-03 Thread Launchpad Bug Tracker
** Branch linked: lp:ubuntu/mountall -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1078926 Title: raring instance failed to find EC2 datasource To manage notifications about this bug go to: https:/

[Bug 1078926] Re: raring instance failed to find EC2 datasource

2012-12-03 Thread Steve Langasek
Seems the problem is that / and /run have both been mounted in the first pass, and mountall is stuck waiting (in try_mounts()) for the 'mounted MOUNTPOINT=/' event to return before it considers the 'mounted MOUNTPOINT=/run' event, which has already triggered. Since the former *depends* on the latt

[Bug 1078926] Re: raring instance failed to find EC2 datasource

2012-12-03 Thread Steve Langasek
** Package changed: ubuntu => mountall (Ubuntu) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1078926 Title: raring instance failed to find EC2 datasource To manage notifications about this bug go

[Bug 1078926] Re: raring instance failed to find EC2 datasource

2012-12-03 Thread Scott Moser
I've split the plymouth / loss-of-console-output bug into bug 1086072. ** Description changed: This seems sporadic failure at best. I had seen it on openstack fail similarly. Today I launched 7 instances of us-east-1 t1.micro from each of :  ami-be70f5d7 ebs/ubuntu-raring-daily-amd64-se

[Bug 1078926] Re: raring instance failed to find EC2 datasource

2012-12-03 Thread Scott Moser
This seems to have been SRU'd to quantal under bug 1059471 see comment at https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/1059471/comments/14 ** Description changed: This seems sporadic failure at best. I had seen it on openstack fail similarly. Today I launched 7 instances of us-ea

[Bug 1078926] Re: raring instance failed to find EC2 datasource

2012-11-27 Thread Scott Moser
It seems to me that there are 3 issues at play here: a.) kernel: the device driver issue that stefan pointed to above. clearly a broken network device driver can result in failure to find a network resource b.) plymouth: many of our boot messages get "eaten". Robie's paste there shows the i

[Bug 1078926] Re: raring instance failed to find EC2 datasource

2012-11-23 Thread Robie Basak
Here's a failure I had that I think is the same issue: http://paste.ubuntu.com/1374464/. On a Canonical internal OpenStack deployment. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1078926 Title: ra

[Bug 1078926] Re: raring instance failed to find EC2 datasource

2012-11-21 Thread Launchpad Bug Tracker
** Branch linked: lp:ubuntu/raring-proposed/linux-lowlatency -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1078926 Title: raring instance failed to find EC2 datasource To manage notifications about

[Bug 1078926] Re: raring instance failed to find EC2 datasource

2012-11-21 Thread Launchpad Bug Tracker
** Branch linked: lp:ubuntu/raring-proposed/linux-ppc -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1078926 Title: raring instance failed to find EC2 datasource To manage notifications about this b

[Bug 1078926] Re: raring instance failed to find EC2 datasource

2012-11-20 Thread Ubuntu Foundations Team Bug Bot
** Tags added: patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1078926 Title: raring instance failed to find EC2 datasource To manage notifications about this bug go to: https://bugs.launchpad.

[Bug 1078926] Re: raring instance failed to find EC2 datasource

2012-11-20 Thread Stefan Bader
This may only be partial as the outcome of some discussion about addition error/safety handling is pending, but it seems to be ok enough as is to at least allow sensible networking. For the time being I have test kernel package at: http://people.canonical.com/~smb/lp1078926 ** Patch added: "Upstr

[Bug 1078926] Re: raring instance failed to find EC2 datasource

2012-11-20 Thread Stefan Bader
Found the following thread which starts off with something that affects the netback side but later on people have tx issues like reported here. There does not seem to be a solution for that, yet (http://markmail.org/message/kce5jybhfo2x5lxn). >From what I understand it is a problem where generic

[Bug 1078926] Re: raring instance failed to find EC2 datasource

2012-11-19 Thread Stefan Bader
There seems to be a generic issue in v3.7 on how much tx traffic out of the PVM can be handled. I actually can see something similar with Xen 4.2 with dom0 running a Quantal kernel (I know it is a daring combination). But I think that points either to net in general or to the netfront driver. I wou

[Bug 1078926] Re: raring instance failed to find EC2 datasource

2012-11-19 Thread Scott Moser
Ok, more interesting information... in /var/log/syslog. Note, 'failsafe' fired, *then* dhcp seemed to be happy to get an address. Strange. as if some ordering blocked ifup of eth0 until after failsafe. Nov 19 14:46:06 ubuntu kernel: [6.503401] EXT4-fs (xvda1): re-mounted. Opts: (null) Nov 1

[Bug 1078926] Re: raring instance failed to find EC2 datasource

2012-11-19 Thread Scott Moser
saw this on t1.micro on us-east-1 ami-3f70f756 canonical ebs/ubuntu-raring-daily-amd64-server-20121119 same symptoms * no console output after initramfs * could not ssh in as None datasource is selected Here, a reboot fixed the issue. I'll attach /var/log/boot.log which had some intere

[Bug 1078926] Re: raring instance failed to find EC2 datasource

2012-11-14 Thread Scott Moser
** Attachment added: "cloud-init.log of instance" https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1078926/+attachment/3433601/+files/cloud-init.log ** Package changed: cloud-init (Ubuntu) => ubuntu ** Changed in: ubuntu Status: New => Confirmed ** Changed in: ubuntu Impor

[Bug 1078926] Re: raring instance failed to find EC2 datasource

2012-11-14 Thread Scott Moser
** Attachment added: "console log of instance after stop-instances" https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1078926/+attachment/3433600/+files/console-ami-be70f5d7-t1.micro-stopped.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is su