# Workload record storage proposal
This is a migration and summation of the discussion
https://discuss.tvm.ai/t/rfc-benchmark-performance-log-format/4610/15. Please
head over there to put in your two cents. This is a WIP issue maintained for
visibility.
We propose a column-based data storage format that unifies how TVM results are
tracked. For now the only metric we suggest tracking is execution time, with a
focus on reproducibility. We foresees the need for a standard record format to
pinpoint the most crucial target areas for future development in TVM.
## Use cases and motivation
- Easier cross-reference to other frameworks/runs
- Experiment reproduction
- Building nightly perf regression
- Standardization will separate logging and visualization efforts
## Key points to consider
- Maintain reproducibility of experiments based solely on an output record
- Granularity of metrics -- e.g. per-layer vs end-to-end
- config/misc json objects -- enforce prescriptive keys or keep it flexible for
now?
- Extensibility of this schema towards desired features in the future
## Proposed format
|header|examples|category|notes/justification|
|---|---|---|---|
|timestamp|1572282699.6|metadata|Indicates when this record is logged|
|schema_version|0.1|metadata|Ensure reproducibility as we iterate on this
schema|
|additional_info|{ build_start:, host_env:, ... }|metadata|`docker_tag`,
`docker_hash` are required; `device_memory_usage`, `code_size`,
`validation_accuracy`, `compile_time` are optionally specified|
|workload|ai.onnx.resnet-18|workload|(1)
|workload_args|{{“input_shape”: (3, 224, 224)}, {“args_shape”:
[list_of_shape]}, {“data_layout”: NHCW}}|workload|
|engine|tvm / onnx-runtime|compiler|
|engine_version|0.5 / 996cf30e8d54b4bf58f0c9950475f47bba7e2c7e|compiler|Include
either version or SHA, or both in the format "0.5:996cf3..."
|engine_config|{“llvm”: “llvm-8”, “nvcc”: 10.1, "accelerator":
"MLAS"}|compiler|fields are optionally specified|
|target|llvm -mcpu=skylake-avx512|compiler|compilation target, used for single
platform|
|build_config|{"opt_level": 3, "layer_schedules": [...]|
|env_name|aws.a1.large / rapi3b / iphone11pro|hardware|Readable cloud vendor
category or hardware, mobile phone name|
|hw_model|Intel Xeon E5-2686 v4 / bcm2837/ apple-a13|hardware|Hardware type,
soc type|
|runtime_config|{“num_cpu_threads”: 3, “cudnn”: “cudnn-8”, “cuda_driver”:
“480.10.1”, “os”: linux}|hardware|Backend runtime arguments|
|execution_config|{“number”: 1, “repeat”: 10, “min_repeat_ms”,
0}|statistics|workload execution parameters|
|execution_elapsed_ms_mean|float32|statistics|
|execution_elapsed_ms_std|float32|statistics|
1. Namespacing: Use strings of the form “domain.name” where domain and name are
any valid string except for the domains of ‘topi’ and ‘relay’ which are
reserved to only refer to TOPI and Relay operator names respectively.\
For topi and relay namespaces, the operator name may be followed by an optional
“:\<SHA\>” which corresponds to a particular git SHA of the TVM repository.
## Possible future schema directions
- Add more metrics to execution statistics, with statistics collection
components enabled by some config object
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/apache/incubator-tvm/issues/4304
