Glad to see this is being planned! I could help on this as much as I can.
One question/suggestion is that if we are going to have such formal
benchmarking approach, maybe we can make it MLPref friendly so that everyone
can use this TVM utility to run these models on the target platform and sub
It would also be great to consider output
https://tvm.apache.org/docs/dev/benchmark.html and iterate on a common log
format
---
[Visit
Topic](https://discuss.tvm.apache.org/t/rfc-building-a-new-reproducible-benchmark-for-tvm/8496/5)
to respond.
You are receiving this because you enabled
It is really nice to add the regression tests against a selected set of models,
since the down streams users usually have to spend quite amount of time to find
the root cause once there is a regression. Or they have to sync the upstream
codebase as frequent as possible and test regression loca
One question for the performance regression, how to judge the normal
fluctuation, especially CPU? Like resnet50 maybe 20.00ms, but becomes 20.88ms
after one pr?
---
[Visit
Topic](https://discuss.tvm.apache.org/t/rfc-building-a-new-reproducible-benchmark-for-tvm/8496/7)
to respond.
You
