Thanks everyone for the feedback. One thing that we seem to agree on is that there is a strong need to support symbolic shape use cases for TVM, as represented by many of the folks who chimed into this thread.
Hopefully, we all agree that there is a strong need to support robust and high-quality dynamic shape compilations, and that is not being supported in the current TVM. One of the questions is whether or not we can do that by separating the changes of C0, C1, C2 in [Rationale and Alternatives](https://github.com/YuchenJin/tvm-rfcs/blob/relax-upstream-rfc/rfcs/0089-relax-upstreaming.md#6-rationale-and-alternatives). In order to provide values to the user, we need **end-to-end** dynamic shape compilation for some specific environment of interest (for example it is less relevant at the moment for micro settings). C0, C1, and C2 are necessary to support a minimum but robust dynamic compilation: - We need to effectively lower the symbolic shape code to a sequence of calls into symbolic TensorIR functions and libraries (when we cannot generate TensorIR functions). This means in order to do so we need **C1: Add first-class support for interactions with TensorIR and TVM FFI.** - In order to further lower some of the shape handling (match_shape), we need to generate code that contains side effect (such that it allocates the buffer and writes into it). This naturally means we need **C2: Add first-class dataflow and side effect handling.** In summary, in order to build a minimum viable dynamic shape compilation in the most high quality, robust, and general fashion, the lowering flow would require stages that contain C1 and C2. We would also like to highlight that having C0, C1, and C2 helps us to achieve the goals **faster**. For example, running fusion on calls into TensorIR means we no longer need developers to annotate the fusion properties of each op, and we no longer need to handle weight layout rewrites in an ad-hoc way. This reduced engineering complexity thanks to better architecture helps us to bring robust high-quality support for dynamic shape faster to the community. The dynamic shape compilation, along with first-class C1 and C2 interaction are needs that we do not support well today in TVM. They offer clear values to our community, and differentiate from the cases that Relay already supports. One of the main topics raised was how Relax fits into TVM in general. We acknowledge that it is impossible to provide the full picture immediately. It is also not necessary to make all decisions in the beginning, as making uninformed decisions is worse. It is also common practices for OSS project to evolve and bringing in S1-conversations along the way. For example, when TorchFX got introduced, there was no prediction of TorchDynamo, which got introduced in the later part as things iterated through the codebase and became clearer. Nevertheless, we would try our best to address **how Relax fits into TVM**: C0, C1, C2 also highlight **how cohesive relax fits into TVM’s current codebase more than** many other modules in the codebase. In particular, the design deeply aligns with TensorIR, topi, symbolic shape and PackedFunc and many other modules. We also view having Relax and Relay co-exist in the codebase as a positive thing and better than the current state: - We get dynamic shape support for users who needs them, empowering the community members. - We get more developers thanks to the empowerment, offsetting the burden brought by the module. - The burden of establishing the module is minimized as it will cause no disruption to the existing modules/flows. - In many cases, having Relay and Relax work together is a positive thing: - We use Relay for importing and graph optimization and use relax for TIR-related co-optimization. It is a great combination and a better pipeline for some of the use cases. These positive parts already would bring the community a long way. We also acknowledge that we cannot pin down all the technical decisions around S1/S2 level in the very beginning. Please see the [Future Opportunities section](https://github.com/tqchen/tvm-rfcs/blob/main/rfcs/0091-establish-tvm-unity-connection.md#4-future-opportunities-after-unity-connection) in the TVM Unity connection RFC, which I encourage everyone to check out. The author also wants to come back to community empowerment. In this case, many in the community have voiced their need for empowerment with grounded technical reasonings. We would like to encourage us to take a step back, as most of the technical reasons are less grounded and we do not need to make all decisions in a single shot. We are looking at a module that is clearly isolated, makes no disruptive change to the code base, and is needed by the majority of community members. Having open-mindedness here would go a long way toward building a welcoming and inclusive community. -- Reply to this email directly or view it on GitHub: https://github.com/apache/tvm-rfcs/pull/89#issuecomment-1267729342 You are receiving this because you are subscribed to this thread. Message ID: <apache/tvm-rfcs/pull/89/c1267729...@github.com>
