Hello Kafka Devs, After years of working with Kafka system tests, I would like to propose moving them from Python/Ducktape to Java/Testcontainers, building on top of the @ClusterTest framework that contributors already know.
The motivation is making system tests more effective and accessible for everyone. Instead of learning a second language and a niche test runner, contributors would write system tests the same way they write integration tests just with a different execution mode. The benefits include a single language and framework with reduced maintenance. Some tests also run significantly faster (e.g., compression test dropped ~85s to ~12s), though that will not hold for every scenario. It is not without downsides. Ducktape can do things containers cannot (e.g., real VM isolation, Kibosh fault injection, multi-host setups). Version coverage is also narrower since the framework only targets KRaft 3.9+, so ZooKeeper-based versions cannot be tested. This is deliberate (i.e., KRaft has been production-ready since 3.6.0, and ZooKeeper mode was removed in 4.0, so investing in ZK-based test infrastructure isn't justified). Some tests will likely stay in Ducktape, and that is fine. If the community agrees this direction makes sense, I will follow up with the concrete details: 1. *which *tests can be migrated now, 2. *what *features @ClusterTest still needs, 3. *how *to wire up CI so both frameworks run from one place. Looking forward to hearing your thoughts. Cheers, Maros
