This bug was fixed in the package python-oslo.messaging - 5.35.0-0ubuntu4~cloud0
---
python-oslo.messaging (5.35.0-0ubuntu4~cloud0) xenial-queens; urgency=medium
.
* New update for the Ubuntu Cloud Archive.
.
python-oslo.messaging (5.35.0-0ubuntu4) bionic; urgency=medium
.
[
The verification of the Stable Release Update for python-oslo.messaging
has completed successfully and the package has now been released to
-updates. In the event that you encounter a regression using the package
from -updates please report a new bug using ubuntu-bug and tag the bug
report regressi
This bug was fixed in the package python-oslo.messaging -
5.35.0-0ubuntu4
---
python-oslo.messaging (5.35.0-0ubuntu4) bionic; urgency=medium
[Seyeong Kim]
* Fix RabbitMQ fails to syncronize exchanges under high load (LP: #1789177)
- d/p/0001-Use-default-exchange-for-direct-mes
tested pkg in bionic
steps are below ( the same as queens above )
1. deploy bionic env
2. upgrade python-oslo.messaging in nova-compute/0
3. restart neutron-openvswitch-agent ( only )
4. check logs , no error
5. launch instance if it works, no error
ii python-oslo.messaging 5.
tested pkg in queens
test steps are below
1. deploy queens env
2. upgrade python-oslo.messaging in nova-compute/0
3. restart neutron-openvswitch-agent ( only )
4. check logs , no error
5. launch instance if it works, no error
ii python-oslo.messaging 5.35.0-0ubuntu4~cloud0
all
sorry for being late, I'll verify this soon
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1789177
Title:
RabbitMQ fails to synchronize exchanges under high load (Note for
ubuntu: stein, rocky, que
Hello Oleg, or anyone else affected,
Accepted python-oslo.messaging into bionic-proposed. The package will
build now and be available at https://launchpad.net/ubuntu/+source
/python-oslo.messaging/5.35.0-0ubuntu4 in a few hours, and then in the
-proposed repository.
Please help us by testing this
Fix proposed to branch: stable/queens
Review: https://review.opendev.org/c/openstack/oslo.messaging/+/787384
** Changed in: cloud-archive/queens
Status: Triaged => In Progress
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
htt
This bug was fixed in the package python-oslo.messaging -
8.1.0-0ubuntu1~cloud2.2
---
python-oslo.messaging (8.1.0-0ubuntu1~cloud2.2) bionic-rocky; urgency=medium
.
[Seyeong Kim]
* Fix RabbitMQ fails to syncronize exchanges under high load (LP: #1789177)
- d/p/0001-Use-de
@Łukasz, it's a little awkward. The single patch does not fix the
failure to synchronize exchanges under high load (based on Seyeong's
testing) however it does fix compatibility with releases that have been
fully patched. I've updated the description, hopefully that helps a bit
to clear this up.
*
** Description changed:
[Impact]
If there are many exchanges and queues, after failing over, rabbitmq-
server shows us error that exchanges are cannot be found.
Affected
Bionic (Queens)
Not affected
Focal
[Test Case]
1. deploy simple rabbitmq cluster
- https://pas
I have tested the rocky scenario that was failing for me. Trilio on
Train + OpenStack on Rocky. The Trilio functional test to snapshot a
server failed without the fix and passed once python3-oslo.messaging
8.1.0-0ubuntu1~cloud2.2 was installed and services restarted
** Tags removed: verification-r
Ok, reviewing the change. But question to be answered before this is
accepted - seeing comments #80 and #81, does this single patch actually
fix the failure to synchronize exchanges under high load? If not, then
we need to adjust the description. What will the current patch address
in its current s
Hello Corey
That makes sense to me as well.
Thanks
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1789177
Title:
RabbitMQ fails to synchronize exchanges under high load
To manage notifications abo
** Patch removed: "lp1789177_bionic.debdiff"
https://bugs.launchpad.net/oslo.messaging/+bug/1789177/+attachment/5466823/+files/lp1789177_bionic.debdiff
** Patch removed: "lp1789177_queens.debdiff"
https://bugs.launchpad.net/oslo.messaging/+bug/1789177/+attachment/5466822/+files/lp1789177_q
Hi Seyeong,
Thanks for testing. I was under the impression that the 3rd patch was
dependent on the 2nd patch since they both deal with the consumer side.
What do you think about moving forward with just patch 1? Unfortunately
it doesn't reduce the impact of the original bug reported here (based o
Testing with only 1st patch didn't work , I was able to see the same error as
the description in this LP
Testing with 1st and 3rd with manual configuration(enable_cancel_on_failover =
True ) showed me different error
( mentnioned above )
The different error is happening less time than I assume.
** Patch added: "lp1789177_bionic.debdiff"
https://bugs.launchpad.net/oslo.messaging/+bug/1789177/+attachment/5466823/+files/lp1789177_bionic.debdiff
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1
** Patch added: "lp1789177_queens.debdiff"
https://bugs.launchpad.net/oslo.messaging/+bug/1789177/+attachment/5466822/+files/lp1789177_queens.debdiff
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1
** Patch removed: "lp1789177_queens.debdiff"
https://bugs.launchpad.net/oslo.messaging/+bug/1789177/+attachment/5444721/+files/lp1789177_queens.debdiff
** Patch removed: "lp1789177_xenial.debdiff"
https://bugs.launchpad.net/oslo.messaging/+bug/1789177/+attachment/5444730/+files/lp1789177_x
** Patch removed: "lp1789177_bionic.debdiff"
https://bugs.launchpad.net/oslo.messaging/+bug/1789177/+attachment/5444720/+files/lp1789177_bionic.debdiff
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs
testing 1st, 3rd commit and manual configuration
enable_cancel_on_failover = True
I did the similar step above with Queens ( as I already made ppa for
this )
and in this case, I see different errors. restarting rabbitmq-server
solved error msgs
=ERROR REPORT 24-Feb-2021::08:07:46 ===
Channel
With 2nd try, I also faced the same error with patched component, not
even only openvswitch-agent.
I'm going to try to reproduce with 1st and 3rd commit with manual
configuration( enable_cancel_on_failover)
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is s
after restarting all rabbitmq-server, status are stable.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1789177
Title:
RabbitMQ fails to synchronize exchanges under high load
To manage notifications
1. deploy rocky
2. installed updated oslo.messaging pkg in below nodes
- neutron-api
- neutron-gateway
- nova-compute
- - restarted openvswitch-agent only
3. tried to reproduce with below config
- created 3000 test queue, exchange, bindings
- juju config rabbitmq-server min-cluster-size=1
- juju co
ah sorry corey you already uploaded it to bionic as well. thanks
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1789177
Title:
RabbitMQ fails to synchronize exchanges under high load
To manage notif
I've confirmed that only 1st patch is ok with below steps
1. deploy queens
2. patch nuetron node's oslo.messaging (1st patch only) except nova-compute
node's oslo.messaging
3. trying to create instance and delete
And I keep restarting cinder-scheduler while I blocked one rabbitmq-
server with ip
I've uploaded new package versions for this bug to the bionic unapproved
queue and the rocky-staging ppa. These package versions only include the
single patch to switch to the default exchange when publishing.
We have a scenario where queens and train need to communicate but can't
because train on
** Changed in: cloud-archive/rocky
Importance: Undecided => Medium
** Changed in: cloud-archive/rocky
Status: New => Triaged
** Changed in: cloud-archive/stein
Importance: High => Medium
** Changed in: cloud-archive/queens
Importance: Undecided => Medium
** Changed in: cloud-arc
** Changed in: cloud-archive/stein
Importance: Undecided => High
** Changed in: cloud-archive/stein
Status: New => Triaged
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1789177
Title:
Rab
I've updated the triage status back to New for releases that were
reverted.
** Changed in: cloud-archive/queens
Status: Fix Committed => New
** Changed in: cloud-archive/stein
Status: Fix Released => New
** Changed in: python-oslo.messaging (Ubuntu Bionic)
Status: Fix Releas
Seyeong, if the first patch helps fix this bug, we can still add it for
queens->stein. We were considering keeping that patch but had to revert
all of the patches in order to expedite the release with minimum
regression potential and testing needed.
--
You received this bug notification because y
I tested below
It was the same scenario I tested
0 deploy test env
- 5.35.0-0ubuntu1~cloud0
1. upgrading olso.messaging in n-ovs
- 5.35.0-0ubuntu2~cloud0 ( from queens-staging launchpad)
2. I got errors
3. upgrading it to new one
- 5.35.0-0ubuntu3~cloud0
it worked fine for me.
I'm trying to rep
This fix is being reverted fully in queens, rocky, and stein [1]. In
order to fast-track the revert it was decided with the SRU team to be
best to revert all patches. We can then consider adding patch 1 back as
a new SRU following the revert and give that more time for testing.
[1] https://bugs.la
@seyeongkim the problem here is that we cant make a change in a stable
release that requires a maintenance window to upgrade since there will
be environments that are not aware of this and e.g. unattended-upgrades
that will break when they upgrade. I think the safest action we can take
here is canc
I confirmed that upgrading olso.messaging in n-ovs causes rabbitmq issue
Right after restarted n-ovs-agent, I can see a lot of errors in rabbitmq log[1]
which is the same as the error when rabbitmq failover issue ( the original
issue of this LP )
Then after I upgraded oslo.messaging in neutron-
The same issue exists in bionic-updates. Since it's already in updates
you have to downgrade some neutron units (neutron-api and neutron-
gateway in my case) to python-oslo.messaging=5.35.0-0ubuntu1 and restart
neutron services in order to have the mismatched versions to recreate.
--
You received
@corey.bryant i agree, we shouldn't introduce that kind of behaviour for
a stable release
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1789177
Title:
RabbitMQ fails to synchronize exchanges under h
This really isn't a nice situation to introduce into a stable release. I
think we should have properly handled this similar to how upstream did,
which is to only introduce default exchange publishing in the N-1
release, and then introduce default exchange consumers in the N release.
That way, if up
To recreate just upgrade to the new python-oslo.messaging on all n-ovs
units, restart neutron-openvswitch-agent, and instance creation will
fail. The problem is the other neutron services on neutron-api and
neutron-gateway are using the old olso.messaging code. Once neutron-api
and neutron-gateway
Ed, I'm able to recreate the same issue, but need to narrow it down more
as we have new neutron and openvswitch packages in proposed as well.
Steps so far:
1) deploy xenial with cloud-archive:queens enabled
2) create instance successfully
3) enable cloud-archive:queens-proposed on all neutron-open
** Description changed:
[Impact]
If there are many exchanges and queues, after failing over, rabbitmq-
server shows us error that exchanges are cannot be found.
Affected
Bionic (Queens)
Not affected
Focal
-
[Test Case]
1. deploy simple rabbitmq cluster
- https://
** Tags removed: verification-queens-done
** Tags added: verification-queens-failed
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1789177
Title:
RabbitMQ fails to synchronize exchanges under high lo
e.g. 2021-02-02 12:07:53.930 27349 ERROR oslo.messaging._drivers.impl_rabbit
[-] Failed to process message ... skipping it.: DuplicateMessageError: Found
duplicate message(fc9335298407444ab0e7000d3fe2f4b7). Skipping it.
2021-02-02 12:07:53.930 27349 ERROR oslo.messaging._drivers.impl_rabbit
Trac
Upgrading to python-oslo.messaging 5.35.0-0ubuntu2~cloud0 from xenial-
queens/proposed has just broken my neutron-openvwitch-agent. After
upgrade I see a load of MessageTimeout and DuplicateMessage errors in
the logs. Downgrading back to 5.35.0-0ubuntu1~cloud0 fixed the problem.
--
You received t
I think UCA/Queens should be changed from "fix committed" to "fix
released"; we just applied this to a customer cloud today, however at
first we mistakenly thought it hadn't been released yet.
(By the way, it resolved the customer's issue, so there's another "works
on Bionic/Queens" datapoint.)
-
This bug was fixed in the package python-oslo.messaging -
5.35.0-0ubuntu2
---
python-oslo.messaging (5.35.0-0ubuntu2) bionic; urgency=medium
[ Corey Bryant ]
* d/gbp.conf: Create stable/queens branch.
[ Chris MacNaughton ]
* d/control: Update VCS paths for move to lp:~ubuntu-
autopkgtest failure analysis:
the only package failing is python-oslo.versionedobjects, and its
autopkgtest failures are due to bug 1912792 and can be ignored.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net
** Tags removed: verification-needed
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1789177
Title:
RabbitMQ fails to synchronize exchanges under high load
To manage notifications about this bug go t
This bug was fixed in the package python-oslo.messaging - 9.5.0-0ubuntu1~cloud1
---
python-oslo.messaging (9.5.0-0ubuntu1~cloud1) bionic-train; urgency=medium
.
[ Corey Bryant ]
* d/gbp.conf: Create stable/stein branch.
.
[ Chris MacNaughton ]
* d/control: Update VCS pat
This bug was fixed in the package python-oslo.messaging - 9.7.1-0ubuntu3~cloud1
---
python-oslo.messaging (9.7.1-0ubuntu3~cloud1) bionic-train; urgency=medium
.
[ Corey Bryant ]
* d/gbp.conf: Create stable/train branch.
.
[ Chris MacNaughton ]
* d/control: Update VCS pat
The autopkgtest failures [1] on this seem unrelated as they are failing
to importoslo.versionedobjects and there is no oslo found. This same
failure occurs on python-oslo.versionedobjects without using the new
python-oslo.messaging from proposed and a later (Groovy) version of
python-oslo.versioned
actually for bionic and queens, python-oslo.messaging is correct one ,
not python3-oslo.messaging
** Tags removed: verification-queens-needed
** Tags added: verification-queens-done
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
http
Verification done for Queens
ii python-oslo.messaging 5.35.0-0ubuntu2~cloud0
all oslo messaging library - Python 2.x
verification steps ( the same as above )
1. reproduce this issue
2. update all python3-oslo.messaging in test env
3. restart rabbitmq-server
all Channel
Verification done for Bionic
ii python-oslo.messaging 5.35.0-0ubuntu2
all oslo messaging library - Python 2.x
verification steps
1. reproduce this issue
2. update all python3-oslo.messaging in test env
3. restart rabbitmq-server
all Channel issue is gone.
** Tags remo
s/Robbie/Robie
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1789177
Title:
RabbitMQ fails to synchronize exchanges under high load
To manage notifications about this bug go to:
https://bugs.launch
Thanks Robbie for the acceptance on Bionic. Per some offline discussion,
we're planning to hold off on the xenial backport as it will result in
additional backports to the cloud archive as well for a bug that we
haven't had users hit on Xenial.
To clarify the last statement, the bug has been inten
I don't see anything in the queue for Xenial but I do see a debdiff
above, so I'm leaving ~ubuntu-sponsors subscribed (sorry I'm not
supposed to both sponsor and SRU review).
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs
Hello Oleg, or anyone else affected,
Accepted python-oslo.messaging into bionic-proposed. The package will
build now and be available at https://launchpad.net/ubuntu/+source
/python-oslo.messaging/5.35.0-0ubuntu2 in a few hours, and then in the
-proposed repository.
Please help us by testing this
Verification for Train is done.
ii python3-oslo.messaging 9.7.1-0ubuntu3~cloud1
all oslo messaging library - Python 3.x
verification steps
1. reproduce this issue
2. update all python3-oslo.messaging in test env
3. restart rabbitmq-server
all Channel issue is gone.
**
Verification for Stein is done.
ii python3-oslo.messaging 9.5.0-0ubuntu1~cloud1
verification steps
1. reproduce this issue
2. update all python3-oslo.messaging in test env
3. restart rabbitmq-server
all Channel issue is gone.
** Tags removed: verification-stein-needed
** Tags a
** Also affects: cloud-archive/stein
Importance: Undecided
Status: New
** Also affects: cloud-archive/train
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.n
** Changed in: python-oslo.messaging (Ubuntu Xenial)
Status: New => In Progress
** Changed in: python-oslo.messaging (Ubuntu Xenial)
Assignee: (unassigned) => Seyeong Kim (seyeongkim)
** Changed in: python-oslo.messaging (Ubuntu Bionic)
Status: New => In Progress
** Changed in
** Patch added: "lp1789177_train.debdiff"
https://bugs.launchpad.net/ubuntu/+source/python-oslo.messaging/+bug/1789177/+attachment/5444808/+files/lp1789177_train.debdiff
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.
** Patch added: "lp1789177_stein.debdiff"
https://bugs.launchpad.net/ubuntu/+source/python-oslo.messaging/+bug/1789177/+attachment/5444807/+files/lp1789177_stein.debdiff
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.
For Stein and Train
There is already commit 3a5de89dd686dbd9660f140f9c78b20e1632
But no 6fe1aec1c74f112db297cd727d2ea400a292b038
I think we need to fix this for both releases as well.
only one fix cannot be solve this issue.
and Train's functional test is already removed
but Stein's one
** Also affects: cloud-archive/mitaka
Importance: Undecided
Status: New
** Also affects: cloud-archive/queens
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad
** Also affects: python-oslo.messaging (Ubuntu Xenial)
Importance: Undecided
Status: New
** Changed in: python-oslo.messaging (Ubuntu Xenial)
Importance: Undecided => Medium
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
** Patch added: "lp1789177_mitaka.debdiff"
https://bugs.launchpad.net/ubuntu/+source/python-oslo.messaging/+bug/1789177/+attachment/5444740/+files/lp1789177_mitaka.debdiff
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bug
** Patch added: "lp1789177_xenial.debdiff"
https://bugs.launchpad.net/ubuntu/+source/python-oslo.messaging/+bug/1789177/+attachment/5444730/+files/lp1789177_xenial.debdiff
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bug
I can't reproduce this symptom in Focal though it is 12.1.0
it doesn't have commit 0a432c7fb107d04f7a41199fe9a8c4fbd344d009
I think xenial need fix as well, I can reproduce this in xenial,
I'm preparing debdiff for xenial as well
--
You received this bug notification because you are a member
** Changed in: python-oslo.messaging (Ubuntu)
Importance: Undecided => Medium
** Changed in: python-oslo.messaging (Ubuntu Bionic)
Importance: Undecided => Medium
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.laun
This was fixed in 12.3.0
---
python-oslo.messaging (12.3.0-0ubuntu1) groovy; urgency=medium
[ Chris MacNaughton ]
* New upstream release for OpenStack Victoria.
* d/control: Align (Build-)Depends with upstream.
* d/p/no-functional-test.patch: Refreshed.
[ Corey Bryant ]
* d/control:
** Patch added: "lp1789177_queens.debdiff"
https://bugs.launchpad.net/ubuntu/+source/python-oslo.messaging/+bug/1789177/+attachment/5444721/+files/lp1789177_queens.debdiff
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bug
** Patch added: "lp1789177_bionic.debdiff"
https://bugs.launchpad.net/ubuntu/+source/python-oslo.messaging/+bug/1789177/+attachment/5444720/+files/lp1789177_bionic.debdiff
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bug
** Patch removed: "lp1789177_bionic.debdiff"
https://bugs.launchpad.net/cloud-archive/+bug/1789177/+attachment/5444392/+files/lp1789177_bionic.debdiff
** Patch removed: "lp1789177_queens.debdiff"
https://bugs.launchpad.net/cloud-archive/+bug/1789177/+attachment/522/+files/lp1789177_que
** Description changed:
[Impact]
+ If there are many exchanges and queues, after failing over, rabbitmq-
+ server shows us error that exchanges are cannot be found.
Affected
- Bionic
+ Bionic (Queens)
Not affected
- Focal
+ Focal
+
[Test Case]
- TBD
+
+ 1. deploy simple rabb
** Also affects: cloud-archive
Importance: Undecided
Status: New
** Also affects: python-oslo.messaging (Ubuntu Bionic)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs
** Patch added: "lp1789177_queens.debdiff"
https://bugs.launchpad.net/ubuntu/+source/python-oslo.messaging/+bug/1789177/+attachment/522/+files/lp1789177_queens.debdiff
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bug
The attachment "lp1789177_bionic.debdiff" seems to be a debdiff. The
ubuntu-sponsors team has been subscribed to the bug report so that they
can review and hopefully sponsor the debdiff. If the attachment isn't a
patch, please remove the "patch" flag from the attachment, remove the
"patch" tag, a
** Patch added: "lp1789177_bionic.debdiff"
https://bugs.launchpad.net/ubuntu/+source/python-oslo.messaging/+bug/1789177/+attachment/5444392/+files/lp1789177_bionic.debdiff
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bug
** Tags added: sts
** Description changed:
+ [Impact]
+
+
+ Affected
+ Bionic
+ Not affected
+ Focal
+
+ [Test Case]
+ TBD
+
+
+ [Where problems could occur]
+ TBD
+
+ [Others]
+
+
+ // original description
+
Input:
- - OpenStack Pike cluster with ~500 nodes
- - DVR enabled in neut
** Also affects: python-oslo.messaging (Ubuntu)
Importance: Undecided
Status: New
** Changed in: python-oslo.messaging (Ubuntu)
Assignee: (unassigned) => Seyeong Kim (seyeongkim)
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to
83 matches
Mail list logo
