In scenarios where multiple models are used back to back, with multiple inputs
and outputs, TVM doesn't produce helpful native libraries to connect them:
- `get_num_inputs()` returns all tensors instead of only the inputs of the model
- `get_output(id)` has no support for strings. And since output names are
mangled, it's unclear what an `id` corresponds to which output.
- as mentioned in the topic "multithreading and TVM runtime", there seems to be
an issue with the module factory shared between threads. In a multimode
scenario, each model runs under different threads and caching the module
factory doesn't work, forcing each thread to recreate it, which incurs some
performance hit.
- while a secondary goal, the names of operators in the graph can be many
characters long, where a simple integer would suffice.
- also a secondary goal, parameters saved in a library are uncompressed. When
saved separately and compressed with even a simple gzip, quite a lot of space
can be reclaimed.
What we need
- `get_num_inputs()` to return only inputs of the model,
- `get_num_params()` to return only parameters/weights,
- preserve output nodes names and so `get_output(name)` works,
- make sure 2 models running in their own thread can cache their module factory
at setup time and reuse PackedFuncs as fast as possible,
- replace parameter names with integers,
- provide an option to compress parameters' tensors, especially when stored in
the same library, even a default gz or LZ4 saves lots of space, and more
dedicated methods could be provided by users.
These would be extensions of the existing code as (most of) this information is
already available in graph runtime, for example. I'm not sure if there are
impacts on the rest of the codebase.
What do you think?
---
[Visit
Topic](https://discuss.tvm.apache.org/t/c-c-runtime-multimodel-support/8518/1)
to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click
here](https://discuss.tvm.apache.org/email/unsubscribe/b053d87b5dca7f4135f9674b99c70725f4ada8491cd7af2cf57a8d39f05b9f7b).
