The recently merged [CUTLASS BYOC](https://github.com/apache/tvm/pull/9261)
relies on C-codegen based BYOC infra to JIT generate and compile C++ template
classes.
Currently it doesn't support Constants embedded in an external function and
instead requires all weight and bias parameters etc to be passed in at runtime.
This caused a problem for me, when I apply CUTLASS BYOC to a real model: I need
to run constant folding to turn fp32 bias parameters into fp16 for pattern
matching purpose and sending fp16 tensors to CUTLASS. For that, I need to bind
parameters to the module by `bind_params_by_name`, which embeds constant to the
external functions like this, which is not supported by CUTLASS BYOC right now:
```
def @tvmgen_default_cutlass_main_267(%cutlass_267_i0: Tensor[(1024, 1024),
float16], %cutlass_267_i1: Tensor[(4096, 1024), float16], Inline=1,
Compiler="cutlass", global_symbol="tvmgen_default_cutlass_main_267",
Primitive=1) -> Tensor[(1024, 4096), float16] {
%9 = fn (%FunctionVar_8_0: Tensor[(1024, 1024), float16], %FunctionVar_8_1:
Tensor[(4096, 1024), float16], %FunctionVar_8_2: Tensor[(4096), float16],
PartitionedFromPattern="nn.dense_add_multiply_cast_erf_cast_multiply_add_multiply_",
Composite="cutlass.dense_bias_gelu_fp16") -> Tensor[(1024, 4096), float16] {
%1 = nn.dense(%FunctionVar_8_0, %FunctionVar_8_1, units=None,
out_dtype="float16") /* ty=Tensor[(1024, 4096), float16] */;
%2 = add(%1, %FunctionVar_8_2) /* ty=Tensor[(1024, 4096), float16] */;
%3 = multiply(%2, meta[relay.Constant][0] /* ty=float16 */) /*
ty=Tensor[(1024, 4096), float16] */;
%4 = cast(%3, dtype="float32") /* ty=Tensor[(1024, 4096), float32] */;
%5 = erf(%4) /* ty=Tensor[(1024, 4096), float32] */;
%6 = cast(%5, dtype="float16") /* ty=Tensor[(1024, 4096), float16] */;
%7 = multiply(%6, meta[relay.Constant][1] /* ty=float16 */) /*
ty=Tensor[(1024, 4096), float16] */;
%8 = add(%7, meta[relay.Constant][2] /* ty=float16 */) /* ty=Tensor[(1024,
4096), float16] */;
multiply(%8, %2) /* ty=Tensor[(1024, 4096), float16] */
};
// meta[relay.Constant][3] is the bias constant, not supported by CUTLASS
BYOC for now
%9(%cutlass_267_i0, %cutlass_267_i1, meta[relay.Constant][3] /*
ty=Tensor[(4096), float16] */) /* ty=Tensor[(1024, 4096), float16] */
}
```
So I now need to deal with Constants. I think embedding all constants into
C-source is infeasible for models like `BERT-large` which I'm working with.
Alternative I think of is to somehow "unbind" constants after constant folding.
But this requires modifying signatures of external functions and passing
additional parameters inside `main` module, for which I don't see an easy way
to achieve.
My questions:
* Is there a good way to deal with Constants in C-source codegen based BYOC?
Has there been any improvement since discussions from last year such as
https://discuss.tvm.apache.org/t/external-codegen-constant-tensors-in-c-codegen/5890
and https://github.com/apache/tvm/pull/5310 (also cc @lhutton1 @manupa-arm
@matt-arm)
* Should CUTLASS codegen switch to JSON runtime, which I believe has no issues
with constants? How can we compile generated C-source with JSON based BYOC? cc
@Laurawly @comaniac @zhiics
---
[Visit
Topic](https://discuss.tvm.apache.org/t/byoc-cutlass-dealing-with-constants-in-c-source-gen-based-byoc/11362/1)
to respond.
You are receiving this because you enabled mailing list mode.
To unsubscribe from these emails, [click
here](https://discuss.tvm.apache.org/email/unsubscribe/19c33f7efdc48881d2d968f7aeaa4a06e6ed321155a2859a15cc7f0473fa56f7).
