[Apache TVM Discuss] [Development/RFC] [RFC] Building a new reproducible benchmark for TVM
Glad to see this is being planned! I could help on this as much as I can. One question/suggestion is that if we are going to have such formal benchmarking approach, maybe we can make it MLPref friendly so that everyone can use this TVM utility to run these models on the target platform and submit the results to MLPref. --- [Visit Topic](https://discuss.tvm.apache.org/t/rfc-building-a-new-reproducible-benchmark-for-tvm/8496/4) to respond. You are receiving this because you enabled mailing list mode. To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/baa10c86a6d4e24b3b4fcb457dba9fcf02e736c4666436ae66935c1062207d53).
[Apache TVM Discuss] [Development/RFC] [RFC] Building a new reproducible benchmark for TVM
It would also be great to consider output https://tvm.apache.org/docs/dev/benchmark.html and iterate on a common log format --- [Visit Topic](https://discuss.tvm.apache.org/t/rfc-building-a-new-reproducible-benchmark-for-tvm/8496/5) to respond. You are receiving this because you enabled mailing list mode. To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/4b4389061bec8e3a4092aadd35a1711769a54e60a0ac24213cd347c0553a97dd).
[Apache TVM Discuss] [Development/RFC] [RFC] Building a new reproducible benchmark for TVM
It is really nice to add the regression tests against a selected set of models, since the down streams users usually have to spend quite amount of time to find the root cause once there is a regression. Or they have to sync the upstream codebase as frequent as possible and test regression locally. cc @jroesch, you may have some comments about the output format or the UX of the test infra. --- [Visit Topic](https://discuss.tvm.apache.org/t/rfc-building-a-new-reproducible-benchmark-for-tvm/8496/6) to respond. You are receiving this because you enabled mailing list mode. To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/d9e2f4ed854594c4fba8c0dd3c906a06e136e1a5863fa2a416184349b9f66e50).
[Apache TVM Discuss] [Development/RFC] [RFC] Building a new reproducible benchmark for TVM
One question for the performance regression, how to judge the normal fluctuation, especially CPU? Like resnet50 maybe 20.00ms, but becomes 20.88ms after one pr? --- [Visit Topic](https://discuss.tvm.apache.org/t/rfc-building-a-new-reproducible-benchmark-for-tvm/8496/7) to respond. You are receiving this because you enabled mailing list mode. To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/89e901316f8e75b07e8a01c296b56e445eabe77c8e950d3bfce00fac7b762c47).
