I just created https://issues.apache.org/jira/browse/CASSANDRA-14134 which
includes tons of details (and a patch available for review) with my efforts to
migrate dtests from nosetest to pytest (which ultimately ended up also
including porting the ode from python 2.7 to python 3).
I'd love if people could pitch in in any way to help get this reviewed and
committed so we can reduce the natural drift that will occur with a huge patch
like this against the changes going into master. I apologize for sending this
so close to the holidays, but I really have been working non-stop trying to get
things into a completed and stable state.
The latest CircleCI runs I did took roughly 15 minutes to run all the dtests
with only 6 failures remaining (when run with vnodes) and 12 failures remaining
(when run without vnodes). For comparison the last ASF Jenkins Dtest job to
successfully complete took nearly 10 hours (9:51) and we had 36 test failures.
Of note, while I was working on this and trying to determine a baseline for the
existing tests I found that the ASF Jenkins jobs were incorrectly configured
due to a typo. The no-vnodes job is actually running with vnodes (meaning the
no-vnodes job is identical to the with-vnodes ASF Jenkins job). There are some
bootstrap tests that will 100% reliably hang both nosetest and pytest on test
cleanup, however this test only runs in the no-vnodes configuration. I've
debugged and fixed a lot of these cases across many test cases over the past
few weeks and I no longer know of any tests that can hang CI.
Thanks and I'm optimistic about making testing great for the project and most
importantly for the OSS C* community!
best,
kjellman
Some highlights that I quickly thought of (in no particular order): {also
included in the JIRA}
-Migrate dtests from executing using the nosetest framework to pytest
-Port the entire code base from Python 2.7 to Python 3.6
-Update run_dtests.py to work with pytest
-Add --dtest-print-tests-only option to run_dtests.py to get easily parsable
list of all available collected tests
-Update README.md for executing the dtests with pytest
-Add new debugging tips section to README.md to help with some basics of
debugging python3 and pytest
-Migrate all existing Enviornment Variable usage as a means to control dtest
operation modes to argparse command line options with documented help on each
toggles intended usage
-Migration of old unitTest and nose based test structure to modern pytest
fixture approach
-Automatic detection of physical system resources to automatically determine if
@pytest.mark.resource_intensive annotated tests should be collected and run on
the system where they are being executed
-new pytest fixture replacements for @since and @pytest.mark.upgrade_test
annotations
-Migration to python logging framework
-Upgrade thrift bindings to latest version with full python3 compatibility
-Remove deprecated cql and pycassa dependencies and migrate any remaining tests
to fully remove those dependencies
-Fixed dozens of tests that would hang the pytest framework forever when run in
CI enviornments
-Ran code nearly 300 times in CircleCI during the migration and to find,
identify, and fix any tests capable of hanging CI
-Upgrade Tests do not yet run in CI and still need additional migration work
(although all upgrade test classes compile successfully)
