Hi all,
When Qt 4.8 was moved to gerrit, corners were cut to get it working with
the Qt 5 test infrastructure. This included the disabling of many
autotests, ostensibly as a temporary measure.
Unfortunately this temporary situation has, to date, not been followed
up on, and we still have greater than 100 autotests effectively disabled
(marked as insignificant) in Qt 4.8's CI system.
These are the relevant tests, in need of attention. Currently, _none_
of these tests are being used to block regressions:
tests/auto/bic
tests/auto/compilerwarnings
tests/auto/declarative/examples
tests/auto/declarative/qdeclarativeborderimage
tests/auto/declarative/qdeclarativedebugjs
tests/auto/declarative/qdeclarativeerror
tests/auto/declarative/qdeclarativeimage
tests/auto/declarative/qdeclarativetext
tests/auto/declarative/qdeclarativexmlhttprequest
tests/auto/declarative/qmlvisual
tests/auto/exceptionsafety_objects
tests/auto/guiapplauncher
tests/auto/headers
tests/auto/languagechange
tests/auto/linguist/lupdate
tests/auto/macgui
tests/auto/macnativeevents
tests/auto/mediaobject
tests/auto/q3actiongroup
tests/auto/q3process
tests/auto/q3socket
tests/auto/q3sqlcursor
tests/auto/q3urloperator
tests/auto/qaccessibility
tests/auto/qapplication
tests/auto/qaudioinput
tests/auto/qclipboard
tests/auto/qcolumnview
tests/auto/qdbusmarshall
tests/auto/qeventloop
tests/auto/qfilesystemmodel
tests/auto/qfilesystemwatcher
tests/auto/qfile
tests/auto/qfocusevent
tests/auto/qfont
tests/auto/qftp
tests/auto/qgl
tests/auto/qglthreads
tests/auto/qgraphicsanchorlayout1
tests/auto/qgraphicsanchorlayout
tests/auto/qgraphicseffect
tests/auto/qgraphicsitem
tests/auto/qgraphicsproxywidget
tests/auto/qgraphicsscene
tests/auto/qgraphicstransform
tests/auto/qgraphicsview
tests/auto/qgraphicswidget
tests/auto/qhostinfo
tests/auto/qhttp
tests/auto/qimagereader
tests/auto/qinputcontext
tests/auto/qinputdialog
tests/auto/qitemdelegate
tests/auto/qlibrary
tests/auto/qlistwidget
tests/auto/qlocale
tests/auto/qlocalsocket
tests/auto/qmacstyle
tests/auto/qmdiarea
tests/auto/qmdisubwindow
tests/auto/qmenu
tests/auto/qnetworkconfigurationmanager
tests/auto/qnetworkconfiguration
tests/auto/qnetworkreply
tests/auto/qnetworksession
tests/auto/qpainter
tests/auto/qpauseanimation
tests/auto/qpixmap
tests/auto/qpluginloader
tests/auto/qprinter
tests/auto/qscriptcontext
tests/auto/qscriptextensionplugin
tests/auto/qscriptv8testsuite
tests/auto/qscrollbar
tests/auto/qsettings
tests/auto/qsharedpointer_and_qwidget
tests/auto/qsharedpointer
tests/auto/qsocketnotifier
tests/auto/qsocks5socketengine
tests/auto/qsslsocket_onDemandCertificates_member
tests/auto/qsslsocket_onDemandCertificates_static
tests/auto/qsslsocket
tests/auto/qstylesheetstyle
tests/auto/qtableview
tests/auto/qtcpserver
tests/auto/qtcpsocket
tests/auto/qtextlayout
tests/auto/qtextscriptengine
tests/auto/qtextstream
tests/auto/qtimeline
tests/auto/qtipc/qsharedmemory
tests/auto/qtooltip
tests/auto/qtreeview
tests/auto/qudpsocket
tests/auto/quuid
tests/auto/qwebframe
tests/auto/qwebpage
tests/auto/qwidget
tests/auto/qwidget_window
tests/auto/qwizard
tests/auto/selftests
tests/auto/symbols
tests/auto/xmlpatterns
( found by: git grep insignificant_test -- tests/auto )
I need your help to improve the situation. For anyone interested in
ensuring the quality of Qt 4.8, this is a great way to help out.
Nokia's own QA staff is almost exclusively working on Qt 5 or related
projects and we won't be able to fix this ourselves.
The goal is to remove CONFIG+=insignificant_test from as many of these
autotests as possible.
How you can help:
- check the current status of these tests for each platform. This can
always be ascertained from these links:
http://testresults.qt-project.org/ci/Qt_4.8_Integration/latest-success/linux-g++-32_Ubuntu_10.04_x86/log.txt.gz
http://testresults.qt-project.org/ci/Qt_4.8_Integration/latest-success/linux-g++_developer-build_qtnamespace_qtlibinfix_Ubuntu_11.10_x64/log.txt.gz
http://testresults.qt-project.org/ci/Qt_4.8_Integration/latest-success/macx-g++_OSX_10.6/log.txt.gz
http://testresults.qt-project.org/ci/Qt_4.8_Integration/latest-success/macx-g++_OSX_10.7/log.txt.gz
http://testresults.qt-project.org/ci/Qt_4.8_Integration/latest-success/macx-g++_developer-build_OSX_10.7/log.txt.gz
http://testresults.qt-project.org/ci/Qt_4.8_Integration/latest-success/qws_linux-x86-g++/log.txt.gz
http://testresults.qt-project.org/ci/Qt_4.8_Integration/latest-success/win32-msvc2010_Windows_7/log.txt.gz
http://testresults.qt-project.org/ci/Qt_4.8_Integration/latest-success/win32-msvc2010_developer-build_Windows_7/log.txt.gz
- if the test is already passing on some platforms, you can amend the
insignificant_test condition to apply only to the platforms where it
is failing, at least allowing the test to detect regressions on some
platforms while the remaining failures are investigated.
It looks to me like many of the failing tests are only failing on
Windows, but incorrectly marked as insignificant for all platforms.
- if the test is failing consistently, you can replace the
insignificant_test marker with a QEXPECT_FAIL for the failing
portion of the test, and raise a bug for the test's failure. This
allows the rest of the test to be used to detect regressions, until
the whole test can be fixed.
- if the test failure looks easy to fix, you can of course fix it
directly.
I'd also like to apologize for not bringing this up sooner, as I've
known for some time the Qt 4.8 autotest situation is quite bad and
haven't attempted to make that more widely known.
_______________________________________________
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development
