Hi everyone, I'm on the tools and automation team. I wanted to talk a
bit about test automation for Servo, and specifically the use of Rust
for test harnesses.
I want to preface this by saying I'm not trying to tell anyone what to
do. I understand that Servo is a young project, and whatever helps you
guys be more efficient in the short term is a good thing. That being
said, I have a few concerns I'd like to voice over the handful of test
harnesses that have been implemented in Rust (I see reftest,
contenttest.. are there more?).
1) Test automation needs to be as rock solid as possible. This will be
very hard to do with a language that hasn't even been released yet. I'm
not knocking Rust, but it is a language that is currently in constant
flux. With test automation the goal is to eliminate as many variables as
possible, and you definitely don't want the language the harness is
written in to be a significant one.
2) The chicken and the egg. I might be a little off base with this one,
but my understanding is that Servo is like a proof of concept for Rust.
In this sense, Servo is almost one giant test for Rust itself. By using
Rust to implement the harnesses you are in a way using the system under
test to test itself.
3) Long term test automation. Right now Servo is a relatively small
project, so it's ok to take shortcuts that enable you to be efficient in
the short term. But eventually you'll want tests running on official
infrastructure, reporting to something like tbpl. I don't know when this
will be, but I do know that if the Servo team has its own ways of doing
things, implements harnesses in Rust, and generally doesn't take the
time to understand how the existing test infrastructure works, it will
take us (the tools and automation team) and releng a massive amount of
extra time to get things into production (think on the order of quarters
if not years).
I say this not to sound controlling or whiny, but out of a desire to
avoid many of the mistakes that were made with the B2G project. My goal
here is to raise awareness of the potential problems we will run into
down the road, and to create some bridges between our teams.
What does this mean for now? Not much. Go about your day as usual. But
if you do go to implement a new test harness, please consider doing so
in Python (2.7). It's what all our other harnesses, infrastructure,
monitoring tools, slaves and reporting depend on. If you aren't very
familiar with Python and need a harness implemented yesterday, by all
means use Rust. Just be aware it could cause you some problems down the
road. If you have concerns about using Python over Rust, I'd love to
hear why.
Finally, please feel free to contact me (ahal on irc) regarding any test
automation related questions, or join #ateam where there are many people
who can offer help.
Cheers,
Andrew
_______________________________________________
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo
