** Description changed: [Impact] Recent changes to systemd rlimit are resulting in memory exhaustion with ovs-vswitchd's use of mlockall(). mlockall() can be disabled via /etc/defaults/openvswitch-vswitch, however there is currently a bug in the shipped ovs-vswitchd systemd unit file that prevents it. The package will be fixed in this SRU. Additionally the neutron-openvswitch charm will be updated to enable disabling of mlockall() use in ovs-vswitchd via a config option. More details on the above summary can be found in the following comments: https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1906280/comments/16 https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1906280/comments/19 ==== Original bug details === Original bug title: Charm stuck waiting for ovsdb 'no key "ovn-remote" in Open_vSwitch record' Original bug details: As seen during this Focal Ussuri test run: https://solutions.qa.canonical.com/testruns/testRun/5f7ad510-f57e-40ce-beb7-5f39800fa5f0 Crashdump here: https://oil-jenkins.canonical.com/artifacts/5f7ad510-f57e-40ce-beb7-5f39800fa5f0/generated/generated/openstack/juju-crashdump-openstack-2020-11-28-03.40.36.tar.gz Full history of occurrences can be found here: https://solutions.qa.canonical.com/bugs/bugs/bug/1906280 Octavia's ovn-chassis units are stuck waiting: octavia/0 blocked idle 1/lxd/8 10.244.8.170 9876/tcp Awaiting leader to create required resources hacluster-octavia/1 active idle 10.244.8.170 Unit is ready and clustered logrotated/63 active idle 10.244.8.170 Unit is ready. octavia-ovn-chassis/1 waiting executing 10.244.8.170 'ovsdb' incomplete public-policy-routing/45 active idle 10.244.8.170 Unit is ready When the db is reporting healthy: ovn-central/0* active idle 1/lxd/9 10.246.64.225 6641/tcp,6642/tcp Unit is ready (leader: ovnnb_db, ovnsb_db) logrotated/19 active idle 10.246.64.225 Unit is ready. ovn-central/1 active idle 3/lxd/9 10.246.64.250 6641/tcp,6642/tcp Unit is ready (northd: active) logrotated/27 active idle 10.246.64.250 Unit is ready. ovn-central/2 active idle 5/lxd/9 10.246.65.21 6641/tcp,6642/tcp Unit is ready logrotated/52 active idle 10.246.65.21 Unit is ready. Warning in the juju unit logs indicates that the charm is blocking on a missing key in the ovsdb: 2020-11-27 23:36:57 INFO juju-log ovsdb:195: Invoking reactive handler: hooks/relations/ovsdb-subordinate/provides.py:97:joined:ovsdb-subordinate 2020-11-27 23:36:57 DEBUG jujuc server.go:211 running hook tool "relation-get" 2020-11-27 23:36:57 WARNING ovsdb-relation-changed ovs-vsctl: no key "ovn-remote" in Open_vSwitch record "." column external_ids 2020-11-27 23:36:57 DEBUG jujuc server.go:211 running hook tool "juju-log" 2020-11-27 23:36:57 INFO juju-log ovsdb:195: Invoking reactive handler: hooks/relations/ovsdb/requires.py:34:joined:ovsdb ============================== [Test Case] The easiest way to test this is to deploy openstack with the neutron-openvswitch charm, using the new charm updates. Once deployed, edit /usr/share/openvswitch/scripts/ovs-ctl with an echo to show what MLOCKALL is set to. Then toggle the charm config option [1] and look at journalctl -xe to find the echo output, which should correspond to the mlockall setting. [1] juju config neutron-openvswitch disable-mlockall=true juju config neutron-openvswitch disable-mlockall=false [Regression Potential] There's potential that this will break users who have come to depend on the incorrect EnvironmentFile setting and environment variable in the systemd unit file for ovs-vswitchd. If that is the case they must be running with modified systemd unit files anyway so it is probably a moot point. [Discussion] == Groovy == I have a query out to James and Christian about an undocumented commit that is getting picked up in the groovy upload. It is committed to the ubuntu/groovy branch of the package Vcs. See debian/ltmain-whole-archive.diff and debian/rules in the upload debdiff at http://launchpadlibrarian.net/511453613/openvswitch_2.13.1-0ubuntu1_2.13.1-0ubuntu1.1.diff.gz == Bionic == The bionic upload is paired with the following SRUs which will also require verification: https://bugs.launchpad.net/bugs/1823295 https://bugs.launchpad.net/bugs/1881077 + + == Package details == + New package versions are in progress and can be found at: + hirsute: https://launchpad.net/ubuntu/+source/openvswitch/2.14.0-0ubuntu2 + groovy: https://launchpad.net/ubuntu/groovy/+queue?queue_state=1&queue_text=openvswitch + focal: https://launchpad.net/ubuntu/focal/+queue?queue_state=1&queue_text=openvswitch + train: https://launchpad.net/~ubuntu-cloud-archive/+archive/ubuntu/train-staging/+packages?field.name_filter=openvswitch&field.status_filter=published&field.series_filter= + stein: https://launchpad.net/~ubuntu-cloud-archive/+archive/ubuntu/stein-staging/+packages?field.name_filter=openvswitch&field.status_filter=published&field.series_filter= + bionic: https://launchpad.net/ubuntu/bionic/+queue?queue_state=1&queue_text=openvswitch + + == Charm update == + https://review.opendev.org/c/openstack/charm-neutron-openvswitch/+/767212
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1906280 Title: [SRU] Add support for disabling mlockall() calls in ovs-vswitchd To manage notifications about this bug go to: https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1906280/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
