tl;dr the tree currently allows you to run tests via make targets and
mach. The mach commands are more intuitive and easier to support going
forward, so I'm proposing we remove support for using make targets to
run tests. Please take the survey at
http://www.surveymonkey.com/s/H5LB2GF to respond without clogging inboxes.
Long version.
The in-tree mechanism for running tests has always been a bit fragile.
It uses separate code paths from what is executed in release automation
(for a few reasons which I won't explain here).
For years, people have used make targets to run tests. They get the job
done, sure. But they are limited in what you can do and are awkward to
use e.g. specifying arguments or options to influence run-time behavior
involves setting environment variables (that's how make works). These
make targets are effectively shell scripts and are an abuse of make as a
dependency evaluation tool. As a build system maintainer, I don't want
these make targets polluting the core build system.
mach, by contrast, is easier to use. You can type |mach help
xpcshell-test| and get friendly output [1] telling you how to influence
test running. Even better - these docs are auto-generated from what's in
the tree, so MDN is never out of sync. It's easier to add features to
mach commands than make (Python is a better systems language than
shell/make). And, mach - being written in Python, the same language as
the test harnesses - integrates much more easily into the test
harnesses. You can perform deep, native integration into the test
harnesses. This facilitates interesting future possibilities, such as
mach commands that simultaneously invoke multiple test harnesses and
interleave their results - something not possible with just make. I've
even seen someone run an HTTP+WebSocket server from a mach command and
pipe test results to a web browser in real-time! Oh the possibilities!
Long term, the mach command for running tests and the code path for
running tests on release automation can converge. This will never happen
with make targets.
mach is the future for running tests. The make targets are effectively
deprecated and have become more burdensome to maintain. They have broken
many times in the last few months because people changing the test
harnesses are all using mach and didn't test the make targets. They are
baggage preserved only so developers aren't forced to change. That
baggage has become too costly to maintain. I propose we eliminate the
make targets for running tests.
Aside from obvious "I don't want to change" complaints, are there any
objections to this change? Is anyone not able to run the mach testing
commands today? Are there features missing from the mach commands?
Please file bugs!
Please take the survey at [2] lend your voice to this decision.
[1] https://gps.pastebin.mozilla.org/2942348
[2] http://www.surveymonkey.com/s/H5LB2GF
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
