Source: paperwork
Version: 2.1.2-1
Severity: important
User: debian...@lists.debian.org
Usertags: flaky
X-Debbugs-Cc: debian...@lists.debian.org
Control: affects -1 + src:gtk+3.0 src:gobject-introspection
The paperwork test suite seems to contain an unreliable ("flaky") test
which intermittently fails on at least armel:
397s ======================================================================
397s FAIL: test_import_duplicated_docx
(docimport.tests_converted.TestConvertedImport.test_import_duplicated_docx)
397s ----------------------------------------------------------------------
397s Traceback (most recent call last):
397s File
"/tmp/autopkgtest-lxc.dcawwldu/downtmp/autopkgtest_tmp/paperwork-backend/tests/docimport/tests_converted.py",
line 162, in test_import_duplicated_docx
397s self.assertEqual(len(self.add_docs), 1)
397s AssertionError: 2 != 1
This means that when packages that are depended on by the test suite,
such as gobject-introspection and gtk+3.0, are trying to migrate to
testing, the test will randomly pass or fail, causing those packages to
be detected as having caused a regression when in fact they have not.
For example see
https://ci.debian.net/packages/p/paperwork/testing/armel/41238454/
(while testing a new gtk+3.0) and
https://ci.debian.net/packages/p/paperwork/testing/armel/41380886/
(while testing a new gobject-introspection).
If this test or this feature cannot be made fully reliable, one option
is to skip it in debian/tests/unittests, and re-run it in a separate
autopkgtest script that is marked as "Restrictions: flaky". A few GNOME
packages use the convention that flaky tests are marked like this:
@unittest.skipIf('DEB_ALLOW_FLAKY_TESTS' not in os.environ,
'https://bugs.debian.org/123456')
def test_something_flaky(...):
...
so that they are normally skipped, but we can force them to be run
(to assess whether they are still a problem!) by using
"export DEB_ALLOW_FLAKY_TESTS=1".
smcv
