Reviewed: https://review.openstack.org/517718 Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=5c564e53966b74d6d7589c3505dd33816b3c19f1 Submitter: Zuul Branch: feature/deep
commit 77a8a4455d4a2bd0a9b9146a01acfe0d93fe7550 Author: Tim Burke <[email protected]> Date: Tue Oct 3 00:11:07 2017 +0000 Let clients request heartbeats during SLO PUTs An SLO PUT requires that we HEAD every referenced object; as a result, it can be a very time-intensive operation. This makes it difficult as a client to differentiate between a proxy-server that's still doing work and one that's crashed but left the socket open. Now, clients can opt-in to receiving heartbeats during long-running PUTs by including the query parameter heartbeat=on With heartbeating turned on, the proxy will start its response immediately with 202 Accepted then send a single whitespace character periodically until the request completes. At that point, a final summary chunk will be sent which includes a "Response Status" key indicating success or failure and (if successful) an "Etag" key indicating the Etag of the resulting SLO. This mechanism is very similar to the way bulk extractions and deletions work, and even the way SLO behaves for ?multipart-manifest=delete requests. Note that this is opt-in: this prevents us from sending the 202 response to existing clients that may mis-interpret it as an immediate indication of success. Co-Authored-By: Alistair Coles <[email protected]> Related-Bug: 1718811 Change-Id: I65cee5f629c87364e188aa05a06d563c3849c8f3 commit 92705bb36b4a771a757977d11875e7b929c41741 Author: David Rabel <[email protected]> Date: Thu Nov 2 12:38:41 2017 +0100 Fix indent in overview_policies.rst Change-Id: I7f070956d8b996db798837392adfca4483067aea commit feee3998408e5ed03563c317ad9506ead92083a6 Author: Clay Gerrard <[email protected]> Date: Fri Sep 1 14:15:45 2017 -0700 Use check_drive consistently We added check_drive to the account/container servers to unify how all the storage wsgi servers treat device dirs/mounts. Thus pushes that unification down into the consistency engine. Drive-by: * use FakeLogger less * clean up some repeititon in probe utility for device re-"mounting" Related-Change-Id: I3362a6ebff423016bb367b4b6b322bb41ae08764 Change-Id: I941ffbc568ebfa5964d49964dc20c382a5e2ec2a commit 29e9ae1cc5b74dce205643c101601555c06b67dc Author: Tim Burke <[email protected]> Date: Thu Oct 19 18:53:04 2017 +0000 Make xml responses less insane Looking at bulk extractions: $ tar c *.py | curl http://saio:8090/v1/AUTH_test/c?extract-archive=tar \ -H accept:application/xml -T - <?xml version="1.0" encoding="UTF-8"?> <delete> <number_files_created>2</number_files_created> <response_body></response_body> <response_status>201 Created</response_status> <errors> </errors> </delete> Or SLO upload failures: $ curl http://saio:8090/v1/AUTH_test/c/slo?multipart-manifest=put -X PUT \ -d '[{"path": "not/found"}]' -H accept:application/xml <delete> <errors> <object><name>not/found</name><status>404 Not Found</status></object></errors> </delete> Why <delete>? Makes no sense. Drive-by: stop being so quadratic. Change-Id: I46b233864ba2815ac632d856b9f3c40cc9d0001a commit 449d83fb0c8668f444baf3c74cf15c26a70bb02e Author: Matthew Oliver <[email protected]> Date: Tue Oct 31 15:55:13 2017 +1100 Doc uses alias instead of aliases The overview_policies doc makes reference to an `alias` option when in fact the option is `aliases`. The sample storage policy snippet is correct, it's just incorrect when listing the possible options. This change changes the listed option to `aliases`. Change-Id: Iddf0f19f4d50819ff6abd46e6a1156dc8e4a451d commit 178d7f37cb80efa668bc81e79d1ee633204b8852 Author: HCLTech-SSW <[email protected]> Date: Wed Oct 25 03:23:47 2017 -0700 Added the man page for container-reconciler.conf Change-Id: Ic7fc6ddca2ab564b31156fa84b362bc9963825f1 Closes-Bug: #1607025 commit 83be309627e79c2f4dd66faf17b5459fdf5bfc25 Author: Kota Tsuyuzaki <[email protected]> Date: Thu Oct 26 18:41:56 2017 +0900 Fix a small typo That's redundant "to" in an inline comment. Change-Id: Idab1e11fbbd80a97b0e4ba1c8ab046be99e47d2d commit 8d882095375dc53acdafafac40aa2efc3bafbcd4 Author: Thiago da Silva <[email protected]> Date: Thu Sep 15 16:45:06 2016 -0400 Return 404 on a GET if tombstone is newer Currently the proxy keeps iterating through the connections in hope of finding a success even if it already has found a tombstone (404). This change changes the code a little bit to compare the timestamp of a 200 and a 404, if the tombstone is newer, then it should be returned, instead of returning a stale 200. Closes-Bug: #1560574 Co-Authored-By: Tim Burke <[email protected]> Change-Id: Ia81d6832709d18fe9a01ad247d75bf765e8a89f4 Signed-off-by: Thiago da Silva <[email protected]> commit e199192caefef068b5bf57da8b878e0bc82e3453 Author: Romain LE DISEZ <[email protected]> Date: Wed Oct 26 10:53:46 2016 +0200 Replace replication_one_per_device by custom count This commit replaces boolean replication_one_per_device by an integer replication_concurrency_per_device. The new configuration parameter is passed to utils.lock_path() which now accept as an argument a limit for the number of locks that can be acquired for a specific path. Instead of trying to lock path/.lock, utils.lock_path() now tries to lock files path/.lock-X, where X is in the range (0, N), N being the limit for the number of locks allowed for the path. The default value of limit is set to 1. Change-Id: I3c3193344c7a57a8a4fc7932d1b10e702efd3572 commit 9c97c80b2631bedebbd3196e87f0696a6c67b8be Author: Samuel Merritt <[email protected]> Date: Fri Oct 20 15:31:07 2017 -0700 Clean up a couple hand-rolled mocks. Change-Id: I6582985990e8b5e3a0c65bce5d3bb1e39d58dfb9 commit 9b2779ba131aa22893432b5c96fb19bf43656a4f Author: Samuel Merritt <[email protected]> Date: Fri Oct 20 14:48:31 2017 -0700 Clean up memcache tests This is mostly just using mock.patch instead of doing it by hand. One test was doing some weird exception counting; now it just uses assertRaises in a loop. Change-Id: I54f903f170226720405df2c5f6845124909ab830 commit 0bca0d4d2bf95a37e4a403457ee5c7db4c0b6290 Author: Pete Zaitcev <[email protected]> Date: Wed Oct 18 18:15:11 2017 -0500 Be more tolerant of exception messages from sqlite Parsing the text of error messages was always perilous, perhaps. Either way, this should fix the problem of test_get() failing. Change-Id: I929c2f6e60624241075a292d020c5707ea0ff1c3 commit e001c02ff938d9aea560970fa70f3d8884a8ea33 Author: Tim Burke <[email protected]> Date: Tue Oct 17 22:49:39 2017 +0000 Stop logging tracebacks on bad xLOs The error messages alone should provide plenty enough information. Plus, running functional tests really shouldn't cause tracebacks. Also, tighten up tests for log messages. Change-Id: I55136484d342af756fa153d971dcb9159a435f13 commit 356b110229ee5e5be93ce1ec60ed2de8f75090bf Author: Samuel Merritt <[email protected]> Date: Wed Oct 18 15:09:12 2017 -0700 Remove swift-temp-url man page. Commit 250da37a removed bin/swift-temp-url, but left the man page. Change-Id: Ic518f23678e3c3134f02a46a51a2bcb90d92bdc2 commit 1aecd1dfc61988f12d4cdfda7110dd648133e4cf Author: Alistair Coles <[email protected]> Date: Wed Oct 18 10:52:52 2017 +0100 tighten up drop_privileges unit tests add more assertions about args that are passed to os module functions Related-Change: Ida15e72ae4ecdc2d6ce0d37bd99c2d86bd4e5ddc Change-Id: Iee483274aff37fc9930cd54008533de2917157f4 commit e6cf9b4758612f38f7317c52423f317b9a11bbc9 Author: Samuel Merritt <[email protected]> Date: Tue Oct 17 15:16:43 2017 -0700 Fix some spelling in a docstring Change-Id: I6b32238da3381848ae56aed1c570d299be72473e commit 646f7507a1f5cf479b8d2023645009de0a28cd79 Author: Tim Burke <[email protected]> Date: Tue Oct 17 21:17:57 2017 +0000 Quiet test output when running test_utils.py in isolation Change-Id: I4cf85d3cd5a20424e9bbbdf0213120b3c3d4b837 commit 0f91c862e19bb8d81c20d03e65e6e6657f7265dc Author: Corey Bryant <[email protected]> Date: Tue Oct 17 15:04:45 2017 -0400 Drop group comparison from drop_privileges test Drop the group comparison from drop_privileges test as it isn't valid since os.setgroups() is mocked. Change-Id: Ida15e72ae4ecdc2d6ce0d37bd99c2d86bd4e5ddc Closes-Bug: #1724342 commit 5438fe7d9bcb7dc99118c12bdf1ccfd663390b08 Author: Samuel Merritt <[email protected]> Date: Fri Oct 13 17:03:18 2017 -0700 Clean up some leftover Python 2.6-isms. Change-Id: I24f0aa3bf7314a669c4cf44eadc46e874d6803f1 commit 250da37a7b279b05a1ac8954b506add1661f194d Author: Tim Burke <[email protected]> Date: Fri Oct 13 23:27:11 2017 +0000 Remove swift-temp-url script This has been deprecated since Swift 2.10.0 (Newton) including a message that it would go away. Let's actually remove it. Change-Id: I7d3659761c71119363ff2c0c750e37b4c6374a39 Related-Change: Ifa8bf636f20f82db4845b02d1b58699edaa39356 commit c118059719ded85381e9b134b091d04a7b5a3955 Author: Tim Burke <[email protected]> Date: Mon Sep 11 22:08:12 2017 +0000 Respond 400 Bad Request when Accept headers fail to parse Change-Id: I6eb4e4bca95e2ee4fecdb703394cb2419737922d Closes-Bug: 1716509 commit 03e8ab7171580f3fb93131e9ccfb69fd35364672 Author: Samuel Merritt <[email protected]> Date: Thu Oct 12 17:15:54 2017 -0700 functests: don't crash if no second account In test.functional.test_object.TestObject.setUp, we create a container in account 2. However, if we've only got one account, we don't skip this class, resulting in a TypeError down in requests somewhere and a stack trace. Since we're using account 2 in setup, we should skip the tests if account 2 is not configured. Change-Id: I569d98baf071d2dce7cf34a9538070f00afda388 commit eaea0c49332273085b17852e22582940ab059006 Author: Samuel Merritt <[email protected]> Date: Thu Oct 12 16:58:41 2017 -0700 Skip cross-account-copy functest if only one account This looks like a case of copy-paste-itis. The cross-account-copy functest is skipped if we have no test accounts configured, but not if we have only one. Change-Id: Ifbefdd9aeb98e3d02c536e9d29759f86ec9af6a1 commit 24188beb81d39790034fa0902246163a7bf54c91 Author: Samuel Merritt <[email protected]> Date: Thu Oct 12 16:13:25 2017 -0700 Remove some leftover threadpool cruft. Change-Id: I43a1a428bd96a2e18aac334c03743a9f94f7d3e1 commit 1d67485c0b935719e0c8999eb353dfd84713add6 Author: Samuel Merritt <[email protected]> Date: Fri Apr 15 12:43:44 2016 -0700 Move all monkey patching to one function Change-Id: I2db2e53c50bcfa17f08a136581cfd7ac4958ada2 commit dc8da5bb194d0a544fb8065aa9a4c7484f605715 Author: Samuel Merritt <[email protected]> Date: Thu Oct 12 10:45:12 2017 -0700 Use "poll" or "selects" Eventlet hub for all Swift daemons. Previously, Swift's WSGI servers, the object replicator, and the object reconstructor were setting Eventlet's hub to either "poll" or "selects", depending on availability. Other daemons were letting Eventlet use its default hub, which is "epoll". In any daemons that fork, we really don't want to use epoll. Epoll instances end up shared between the parent and all children, and you get some awful messes when file descriptors are shared. Here's an example where two processes are trying to wait on the same file descriptor using the same epoll instance, and everything goes wrong: [proc A] epoll_ctl(6, EPOLL_CTL_ADD, 3, ...) = 0 [proc B] epoll_ctl(6, EPOLL_CTL_ADD, 3, ...) = -1 EEXIST (File exists) [proc B] epoll_wait(6, ...) = 1 [proc B] epoll_ctl(6, EPOLL_CTL_DEL, 3, ...) = 0 [proc A] epoll_wait(6, ...) This primarily affects the container updater and object updater since they fork. I've decided to change the hub for all Swift daemons so that we don't add multiprocessing support to some other daemon someday and suffer through this same bug again. This problem was made more apparent by commit 6d16079, which made our logging mutex use file descriptors. However, it could have struck on any shared file descriptor on which a read or write returned EAGAIN. Change-Id: Ic2c1178ac918c88b0b901e581eb4fab3b2666cfe Closes-Bug: 1722951 commit 37f23b072ee7a655ab9072f1d0bece0919dec61b Author: Samuel Merritt <[email protected]> Date: Mon Oct 9 17:31:30 2017 -0700 Allow SLOs to have zero-byte last segments. Since we used to allow zero-byte last segments but now we don't, it can be difficult to deal with some old SLO manifests. Imagine you're writing some code to sync objects from Swift cluster A to Swift cluster B. You start off with just a GET from A piped into a PUT to B, and that works great until you hit a SLO manifest and B won't accept a 500GB object. So, you write some code to detect SLO manifests, sync their segments, then take the JSON manifest (?multipart-manifest=get) and sync *that* over. Now, life is good... until one day you get an exception notification that there's this manifest on cluster A that cluster B won't accept. Turns out that, back when Swift would take zero-byte final segments on SLOs (before commit 7f636a5), someone uploaded such a SLO to cluster A. Now, however, zero-byte final segments are invalid, so that SLO that exists over in cluster A can't just be copied to cluster B. A little coding later, your sync tool detects zero-byte final segments and removes them when copying a manifest. But now your ETags don't match between clusters, so you have to figure out some way to deal with that, and so you put it in metadata, but then you realize that your syncer might encounter a SLO which contains a sub-SLO which has a zero-byte final segment, and it's right about then that you start thinking about giving up on programming and getting a job as an elevator mechanic. This commit makes life easier for developers of such applications by allowing SLOs to have zero-byte segments again. Change-Id: Ia37880bbb435e269ec53b2963eb1b9121696d479 commit 348bd83b7eb638d3309ff6313d9a6501c540fa24 Author: Alistair Coles <[email protected]> Date: Wed Aug 30 15:29:14 2017 +0100 Respect co-builder partition moves when min_part_hours is zero Repeated calls to each co-builder's _update_last_part_moves() are unnecessary and have the unfortunate side effect of resetting the _last_part_moved bitmap. When a component builder has zero min_part_hours this results in it not preventing its co-builders from moving parts that it has already moved. This patch changes the CompositeRingBuilder to call each component builder _update_last_part_moves() *once* before rebalancing any component builder. CooperativeRingBuilder's no longer forward calls to their _update_last_part_moves() method. Each component's _last_part_moved bitmap is therefore preserved until for the duration of the composite rebalance. The initialisation of the RingBuilder _last_part_moves array is moved to the RingBuilder __init__ method, so that calls to _update_last_part_moves() are effective even when rebalance() has never been called on that builder. Otherwise, during a composite rebalance, a component that has not previously been rebalanced will not have its _last_part_moves_epoch updated during rebalance and as a result may report erroneous min_part_seconds_left after its first rebalance. Related-Change: I1b30cb3d776be441346a4131007d2487a5440a81 Closes-Bug: #1714274 Change-Id: Ib165cf974c865d47c2d9e8f7b3641971d2e9f404 commit a959d24bf5733650c6005412382fe24c65897c8b Author: Alistair Coles <[email protected]> Date: Tue Aug 15 12:37:33 2017 +0100 Document keystone role element in container ACL The use of a keystone role name in container ACLs is supported and tested. This patch adds documentation. [1] https://github.com/openstack/swift/blob/fb3d01a974fb7df8cfadc56ff15bdc04b3c90759/swift/common/middleware/keystoneauth.py#L491-L497 [2] test.unit.common.middleware.test_keystoneauth.TestAuthorize.test_authorize_succeeds_for_user_role_in_roles Change-Id: I77df27393a10f1d8c5a43161fdd4eb08be632566 Closes-Bug: #1705300 ** Tags added: in-feature-deep -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1724342 Title: test_drop_privileges AssertionError groups mismatch To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1724342/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
