[Bug 959221] Re: swift consumes over 100% of cpu during upload
Profile shows that eventlet/green/ssl.py is the culprit. Look at call numbers and cumulative time. ** Attachment added: "profile 1" https://bugs.launchpad.net/swift/+bug/959221/+attachment/3035280/+files/lp959221.prof1.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
It looks like the problem may be looping. If I do the following slew of hand, CPU burn dissipates: --- /usr/lib/python2.7/site-packages/eventlet/green/ssl.py.fcs 2012-04-09 19:32:04.516730732 -0600 +++ /usr/lib/python2.7/site-packages/eventlet/green/ssl.py 2012-04-09 21:49:38.823537699 -0600 @@ -128,7 +128,7 @@ amount = len(data) count = 0 while (count < amount): -v = self.send(data[count:]) +v = self.write(data[count:]) count += v return amount else: But I do not see a simple way to do anything about it. The client invokes the ssl.py indirectly, through httplib (there's also bufferedhttp, but it changes nothing in SSL case). The httplib hardcodes using of sendall() on the socket object, while only send() can be overrided easily by HTTPConnection. Since sendall just calls send repeatedly as seen above, one may be devious and try to turn send() into write(), but I cannot write my head around the possible implications of that. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
Roman, please try this and let us know if it works for you. ** Patch added: "patch 2, test workaround" https://bugs.launchpad.net/swift/+bug/959221/+attachment/3041102/+files/swift-lp959221-2.diff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
This works, too: [root@lembas green]# pwd /usr/lib/python2.7/site-packages/eventlet/green [root@lembas green]# diff -u ssl.py.fcs ssl.py --- ssl.py.fcs 2012-04-10 08:54:43.257885841 -0600 +++ ssl.py 2012-04-10 08:56:27.282585355 -0600 @@ -129,7 +129,10 @@ count = 0 while (count < amount): v = self.send(data[count:]) -count += v +if v: +count += v +else: +time.sleep(0.01) return amount else: while True: Still, I do not understand quite why this is a problem. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
This reproduces easily on my laptop, running Fedora 16, using git tip swift client (1.4.10). The problem definitely has something with SSL. I created a 2-node Swift+Keystone to be accessible through Pound, and configured Pound to allow both plain HTTP and SSL. Uploads with HTTP only use up a worrying amount of CPU - 25..30% but not everything. Uploads with HTTP+SSL peg at 105%. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
Fix releases in Havana release of OpenStack. ** Changed in: python-swiftclient Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/python-swiftclient/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
Just to clarify: my patch does not remove SSL at all. SSL is there and works as before. The only difference is that it's not trampolined into eventlet anymore. The root cause is the conflict between eventlet's green threads, which require faking of blocking sockets, and SSL, which requires looping (to dispatch opposing-read events and to send little chunks). There is nothing wrong with SSL or SSL performance, just the violation of assumptions about the way sockets behave. The whole problem is on the client. It has absolutely nothing with having SSL terminated on the proxy. In fact in my tests SSL was terminated by Pound. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
** Changed in: swift Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 920197] Re: [SRU] webob last stable version 1.1.1 response header bug
** Changed in: swift Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/920197 Title: [SRU] webob last stable version 1.1.1 response header bug To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/920197/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
see also https://review.openstack.org/22569 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
** Changed in: swift Assignee: Flavio Percoco (flaper87) => Pete Zaitcev (zaitcev) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
Fixed in: https://github.com/openstack/python-swiftclient/commit/3196daf9929eef25d69d47592beef4cd31573b80 For some reason, Jenkins and Gerrit did not close the bug. ** Changed in: swift Status: Confirmed => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 959221] Re: swift consumes over 100% of cpu during upload
** Patch added: "Fix as committed" https://bugs.launchpad.net/ubuntu/+source/swift/+bug/959221/+attachment/3675321/+files/python-swiftclient-1.4-lp959221-2.diff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/959221 Title: swift consumes over 100% of cpu during upload To manage notifications about this bug go to: https://bugs.launchpad.net/swift/+bug/959221/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 665667] Re: couldn't launch an AMI bundled and uploaded using euca2ools
I like David's fix with two subprocess.Popen and a loop that forks the stream. I would've preferred the same approach myself, but I did not have Python chops to do it. BTW, I have RT # now, for next time. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/665667 Title: couldn't launch an AMI bundled and uploaded using euca2ools -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
