As an implementor of multiple passes in Relay, I had noticed that we have far
too little test case.
Most test_pass_*.py do not test for all program construct. For example,
test_pass_dead_code_elimination do not test about adt, or test_pass_fuse_ops do
not test about closure and reference. I had looked into each file, and almost
all of them are 'incomplete' in this sense.
The short and obvious answer ('require people to write more test!') do not
work, simply because relay will likely keep adding more and more language
construct (for example, Mutual Recursion as of #2468). So, even once complete
test case will also be incomplete.
I propose we create a file common.py in test, and put a list of test_case
(called test_list) in there.
Each compiler pass can now loop through the test_list, and see if the
transformed expression is semantically equal to the original one.
Additionally, I propose we also implement a function, semantic_equal, which
take two relay expression, and do some property testing to see if they are
semantically the same.
The reason is that a large portion of the test code is simply: creating an
executor, creating the right amount of input, in the right type, get their
output, deconstruct it to be multiple tensor, and testing that each of them is
correct.
It is difficult to write a generic helper function to do all of them, because
they have different input/output type. The check_eval function that sort of
does the above only handle the case for a single output tensor, and the user
still has to construct the input by hand.
If we implement semantic_equal, the test_list can have test of different type.
Additionally, it also mean we do test on more input, instead of the current
handwritten one.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/dmlc/tvm/issues/2884
