** Description changed: [ Impact ] Starting in OpenStack 2022.2 (Zed), compressing static assets with django-compressor fails around 5-10% of the time (see also the Debian bug [1]). This is done during the Horizon postinst and prevents the package from installing when the race occurs. This affects Noble & Oracular, and UCA Zed through Epoxy. Hitting a parse error in pyscss: ``` Looking for 'dashboard/scss/serial_console.scss' in storage Traceback (most recent call last): File "/usr/share/openstack-dashboard/manage.py", line 25, in <module> execute_from_command_line(sys.argv) File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 419, in execute_from_command_line utility.execute() File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 413, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 354, in run_from_argv self.execute(*args, **cmd_options) File "/usr/lib/python3/dist-packages/django/core/management/base.py", line 398, in execute output = self.handle(*args, **options) File "/usr/lib/python3/dist-packages/compressor/management/commands/compress.py", line 296, in handle self.handle_inner(**options) File "/usr/lib/python3/dist-packages/compressor/management/commands/compress.py", line 319, in handle_inner offline_manifest, block_count, results = self.compress(engine, extensions, verbosity, follow_links, log) File "/usr/lib/python3/dist-packages/compressor/management/commands/compress.py", line 227, in compress raise errors[0] django.core.management.base.CommandError: An error occurred during rendering serial_console.html: Syntax error: Found 'inline-blo' but expected one of ADD, ALPHA_FUNCTION, BANG_IMPORTANT, BAREWORD, COLOR, DOUBLE_QUOTE, FNCT, IF_FUNCTION, INTERP_START, LITERAL_FUNCTION, LPAR, NOT, NUM, SIGN, SINGLE_QUOTE, URL_FUNCTION, VAR ``` Debian had little luck in finding the root cause and uploaded a workaround that simply retries failed compressions 5 times, dramatically reducing the liklihood that the failure breaks installation [2]. django-compressor introduced threaded compression in 3.0 [3], which was released in UCA Zed and Ubuntu Noble. I opened [4] in django-compressor to troubleshoot. The investigation there points to pyscss failing with correct input from django-compressor, indicating that the root cause is likely in pyscss. Horizon (Flamingo+) has switched from pyscss to libsass [5]. This bug is not present in Ubuntu Plucky containing that commit. Since the bug is no longer present in devel, I'd like to SRU the Debian workaround to the affected Ubuntu & UCA packages without knowing the root cause of the race in pyscss. [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1069097 [2] https://salsa.debian.org/openstack-team/services/horizon/-/commit/9802ded2ab24cab767d0c91dfcabea117ec5fd9d [3] https://github.com/django-compressor/django-compressor/pull/912 [4] https://github.com/django-compressor/django-compressor/issues/1291 [5] https://opendev.org/openstack/horizon/commit/283219ab5ef4344e3808cd61b4cc0821886acffb [ Test Plan ] Install horizon and an affected version of python3-django-compressor (>=3.0). The Debian bug indicates that this fails between 5-10% of the time. Running `dpkg-reconfigure openstack-dashboard` in a loop on ~10 containers at once usually reproduces it in a couple of minutes, at most 30 minutes. The bug can be considered fixed if the loop runs for 1 hour with no occurances. I use the following scripts against my LXD cluster: ``` #!/bin/bash # # /usr/local/bin/horizon-reconfigure.sh in each test container set -ex while true; do dpkg-reconfigure openstack-dashboard done ``` Start and check if running: ``` lxc exec -t "${inst}" -- nohup bash -c "horizon-reconfigure.sh &>> /var/log/dpkg-horizon &" lxc exec "${inst}" -- ps -aux | grep horizon-reconfigure ``` For reference, compress can be run with the following to get more output and a traceback on failure: ``` python3 /usr/share/openstack-dashboard/manage.py compress -v 3 --traceback ``` [ Where problems could occur ] * The workaround does not fix the race; the package may still fail to install if the race occurs 5 times in a row. * If the change is wrong/broken, it risks regressions during openstack- dashboard install/reconfigure. This is covered by `dpkg-reconfigure` in the test plan. [ Other info ] * As this change affects upgrades between OpenStack versions via the UCA, I'm opening MPs against all affected versions, even those that are past EOL (Zed+). - * The patch I've proposed drops the scss change from the Debian commit + * The patch I've proposed drops the css change from the Debian commit as it is unrelated to the issue described here. [ Original Description ] We have seen that the puppet integration job with Ubuntu 22.04 and UCA Bobcat consistently fails. Looking into the error it seems something is wrong with installation of openstack-dashboard package. 2023-12-01 07:13:16 +0000 Puppet (err): Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install openstack-dashboard' returned 100: Reading package lists... Building dependency tree... Reading state information... The following additional packages will be installed: openstack-dashboard-common python3-asgiref python3-csscompressor python3-django python3-django-appconf python3-django-compressor python3-django-debreach python3-django-horizon python3-django-openstack-auth python3-django-pyscss python3-pint python3-pyscss python3-rcssmin python3-rjsmin Suggested packages: bpython3 geoip-database-contrib gettext ipython3 libgdal20 libsqlite3-mod-spatialite python-django-doc python3-flup python3-mysqldb python3-pil python3-selenium python3-sqlite python-django-appconf-doc python3-calmjs python-django-debreach-doc The following NEW packages will be installed: openstack-dashboard openstack-dashboard-common python3-asgiref python3-csscompressor python3-django python3-django-appconf python3-django-compressor python3-django-debreach python3-django-horizon python3-django-openstack-auth python3-django-pyscss python3-pint python3-pyscss python3-rcssmin python3-rjsmin 0 upgraded, 15 newly installed, 0 to remove and 0 not upgraded. Need to get 14.2 MB of archives. After this operation, 92.3 MB of additional disk space will be used. WARNING: The following packages cannot be authenticated! python3-rcssmin python3-rjsmin python3-django-compressor python3-pyscss openstack-dashboard-common python3-django-openstack-auth python3-django-horizon openstack-dashboard Authentication warning overridden. Get:1 http://mirror.gra1.ovh.opendev.org/ubuntu-cloud-archive jammy-updates/bobcat/main amd64 python3-rcssmin amd64 1:1.1.0-2~cloud0 [17.5 kB] Get:2 https://mirror.gra1.ovh.opendev.org/ubuntu jammy/main amd64 python3-asgiref all 3.5.0-1 [23.9 kB] Get:3 http://mirror.gra1.ovh.opendev.org/ubuntu-cloud-archive jammy-updates/bobcat/main amd64 python3-rjsmin amd64 1.2.0+dfsg1-2build2~cloud0 [16.6 kB] Get:4 https://mirror.gra1.ovh.opendev.org/ubuntu jammy/main amd64 python3-csscompressor all 0.9.5-2 [57.3 kB] Get:5 http://mirror.gra1.ovh.opendev.org/ubuntu-cloud-archive jammy-updates/bobcat/main amd64 python3-django-compressor all 4.0-2ubuntu1~cloud0 [71.8 kB] Get:6 https://mirror.gra1.ovh.opendev.org/ubuntu jammy-updates/main amd64 python3-django all 2:3.2.12-2ubuntu1.9 [3020 kB] Get:7 http://mirror.gra1.ovh.opendev.org/ubuntu-cloud-archive jammy-updates/bobcat/main amd64 python3-pyscss amd64 1.4.0-3~cloud0 [97.9 kB] Get:8 https://mirror.gra1.ovh.opendev.org/ubuntu jammy/main amd64 python3-django-appconf all 1.0.3-1 [7620 B] Get:9 http://mirror.gra1.ovh.opendev.org/ubuntu-cloud-archive jammy-updates/bobcat/main amd64 openstack-dashboard-common all 4:23.3.0-0ubuntu1.1~cloud0 [27.0 kB] Get:10 https://mirror.gra1.ovh.opendev.org/ubuntu jammy/main amd64 python3-django-debreach all 2.0.1-2 [5482 B] Get:11 http://mirror.gra1.ovh.opendev.org/ubuntu-cloud-archive jammy-updates/bobcat/main amd64 python3-django-openstack-auth all 4:23.3.0-0ubuntu1.1~cloud0 [54.9 kB] Get:12 https://mirror.gra1.ovh.opendev.org/ubuntu jammy/main amd64 python3-django-pyscss all 2.0.2-10 [9320 B] Get:13 http://mirror.gra1.ovh.opendev.org/ubuntu-cloud-archive jammy-updates/bobcat/main amd64 python3-django-horizon all 4:23.3.0-0ubuntu1.1~cloud0 [10.6 MB] Get:14 https://mirror.gra1.ovh.opendev.org/ubuntu jammy/main amd64 python3-pint all 0.18-1 [156 kB] Get:15 http://mirror.gra1.ovh.opendev.org/ubuntu-cloud-archive jammy-updates/bobcat/main amd64 openstack-dashboard all 4:23.3.0-0ubuntu1.1~cloud0 [20.2 kB] debconf: delaying package configuration, since apt-utils is not installed Fetched 14.2 MB in 7s (2004 kB/s) Selecting previously unselected package python3-asgiref. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 87302 files and directories currently installed.) Preparing to unpack .../00-python3-asgiref_3.5.0-1_all.deb ... Unpacking python3-asgiref (3.5.0-1) ... Selecting previously unselected package python3-csscompressor. Preparing to unpack .../01-python3-csscompressor_0.9.5-2_all.deb ... Unpacking python3-csscompressor (0.9.5-2) ... Selecting previously unselected package python3-django. Preparing to unpack .../02-python3-django_2%3a3.2.12-2ubuntu1.9_all.deb ... Unpacking python3-django (2:3.2.12-2ubuntu1.9) ... Selecting previously unselected package python3-django-appconf. Preparing to unpack .../03-python3-django-appconf_1.0.3-1_all.deb ... Unpacking python3-django-appconf (1.0.3-1) ... Selecting previously unselected package python3-rcssmin. Preparing to unpack .../04-python3-rcssmin_1%3a1.1.0-2~cloud0_amd64.deb ... Unpacking python3-rcssmin (1:1.1.0-2~cloud0) ... Selecting previously unselected package python3-rjsmin. Preparing to unpack .../05-python3-rjsmin_1.2.0+dfsg1-2build2~cloud0_amd64.deb ... Unpacking python3-rjsmin (1.2.0+dfsg1-2build2~cloud0) ... Selecting previously unselected package python3-django-compressor. Preparing to unpack .../06-python3-django-compressor_4.0-2ubuntu1~cloud0_all.deb ... Unpacking python3-django-compressor (4.0-2ubuntu1~cloud0) ... Selecting previously unselected package python3-django-debreach. Preparing to unpack .../07-python3-django-debreach_2.0.1-2_all.deb ... Unpacking python3-django-debreach (2.0.1-2) ... Selecting previously unselected package python3-pyscss. Preparing to unpack .../08-python3-pyscss_1.4.0-3~cloud0_amd64.deb ... Unpacking python3-pyscss (1.4.0-3~cloud0) ... Selecting previously unselected package python3-django-pyscss. Preparing to unpack .../09-python3-django-pyscss_2.0.2-10_all.deb ... Unpacking python3-django-pyscss (2.0.2-10) ... Selecting previously unselected package python3-pint. Preparing to unpack .../10-python3-pint_0.18-1_all.deb ... Unpacking python3-pint (0.18-1) ... Selecting previously unselected package openstack-dashboard-common. Preparing to unpack .../11-openstack-dashboard-common_4%3a23.3.0-0ubuntu1.1~cloud0_all.deb ... Unpacking openstack-dashboard-common (4:23.3.0-0ubuntu1.1~cloud0) ... Selecting previously unselected package python3-django-openstack-auth. Preparing to unpack .../12-python3-django-openstack-auth_4%3a23.3.0-0ubuntu1.1~cloud0_all.deb ... Unpacking python3-django-openstack-auth (4:23.3.0-0ubuntu1.1~cloud0) ... Selecting previously unselected package python3-django-horizon. Preparing to unpack .../13-python3-django-horizon_4%3a23.3.0-0ubuntu1.1~cloud0_all.deb ... Unpacking python3-django-horizon (4:23.3.0-0ubuntu1.1~cloud0) ... Selecting previously unselected package openstack-dashboard. Preparing to unpack .../14-openstack-dashboard_4%3a23.3.0-0ubuntu1.1~cloud0_all.deb ... Unpacking openstack-dashboard (4:23.3.0-0ubuntu1.1~cloud0) ... Setting up python3-rjsmin (1.2.0+dfsg1-2build2~cloud0) ... Setting up python3-pyscss (1.4.0-3~cloud0) ... Setting up python3-rcssmin (1:1.1.0-2~cloud0) ... Setting up python3-csscompressor (0.9.5-2) ... Setting up python3-pint (0.18-1) ... Setting up openstack-dashboard-common (4:23.3.0-0ubuntu1.1~cloud0) ... Setting up python3-asgiref (3.5.0-1) ... Setting up python3-django (2:3.2.12-2ubuntu1.9) ... Setting up python3-django-appconf (1.0.3-1) ... Setting up python3-django-debreach (2.0.1-2) ... Setting up python3-django-compressor (4.0-2ubuntu1~cloud0) ... Setting up python3-django-pyscss (2.0.2-10) ... Setting up python3-django-openstack-auth (4:23.3.0-0ubuntu1.1~cloud0) ... Setting up python3-django-horizon (4:23.3.0-0ubuntu1.1~cloud0) ... update-alternatives: using /usr/lib/python3/dist-packages/openstack_dashboard to provide /usr/share/openstack-dashboard/openstack_dashboard (openstack_dashboard) in auto mode Setting up openstack-dashboard (4:23.3.0-0ubuntu1.1~cloud0) ... Adding system user `horizon' (UID 133) ... Adding new user `horizon' (UID 133) with group `horizon' ... Not creating home directory `/var/lib/openstack-dashboard'. Collecting and compressing static assets... CommandError: An error occurred during rendering serial_console.html: Syntax error: Found 'inline-blo' but expected one of ADD, ALPHA_FUNCTION, BANG_IMPORTANT, BAREWORD, COLOR, DOUBLE_QUOTE, FNCT, IF_FUNCTION, INTERP_START, LITERAL_FUNCTION, LPAR, NOT, NUM, SIGN, SINGLE_QUOTE, URL_FUNCTION, VAR dpkg: error processing package openstack-dashboard (--configure): installed openstack-dashboard package post-installation script subprocess returned error exit status 1 Errors were encountered while processing: openstack-dashboard E: Sub-process /usr/bin/dpkg returned an error code (1) ...
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2045394 Title: Installation of openstack-dashboard fails during asset compression To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2045394/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
