On Mon, Sep 13, 2021 at 12:51 AM Linus Walleij <linus.wall...@linaro.org> wrote: > On Sun, Sep 12, 2021 at 11:13 PM Dave Airlie <airl...@gmail.com> wrote: > > > For userspace components as well these communities of experts need to > > exist for each domain, and we need to encourage upstream first > > processes across the board for these split kernel/userspace stacks. > > > > The habanalabs compiler backend is an LLVM fork, I'd like to see the > > effort to upstream that LLVM backend into LLVM proper. > > I couldn't agree more. > > A big part of the problem with inference engines / NPU:s is that of no > standardized userspace. Several of the machine learning initiatives > from some years back now have stale git repositories and are > visibly unmaintained, c.f. Caffe https://github.com/BVLC/caffe > last commit 2 years ago.
Caffe as a standalone project was abandoned and merged into PyTorch, see https://caffe2.ai/. I think this is the kind of consolidation of those projects that you are looking for. > Habanalabs propose an LLVM fork as compiler, yet the Intel > logo is on the Apache TVM website, and no sign of integrating with > that project. They claim to support also TensorFlow. > > The way I perceive it is that there simply isn't any GCC/LLVM or > Gallium 3D of NPU:s, these people haven't yet decided that "here > is that userspace we are all going to use". Or have they? > > LLVM? TVM? TensorFlow? PyTorch? Some other one? > > What worries me is that I don't see one single developer being > able to say "this one definitely, and they will work with the kernel > community", and that is what we need to hear. I don't actually think this is a decision we can possibly wait for. The ones you listed all work on different levels, some build on top of others, and some may get replaced by new ones over time. For a generic kernel interface, we need something that can be supported as a back-end for multiple such libraries, and that works on more than just one hardware. Most likely we will need both higher-level and lower-level interfaces, so that a framework (or an application directly) may target one interface, but some hardware may not be able to implement this. One straightforward hardware independent low-level API would be the traditional BLAS GEMM call[1] for matrix multiplication and its variants (integer, float, bfloat16, ...). Most of the frameworks are able to use SGEMM to do the actual calculation since that has optimized versions for most CPUs and GPUs, and most hardware accelerators should be able to provide an implementation of this that doesn't completely suck. This can be used for both inferencing and training. On the kernel side, this could probably be done inside the existing crypto (async), media (mem2mem), or gpu/drm interfaces that all provide ways to offload computational functions on blocks of memory potentially backed by a dmabuf, but having a new top-level chardev interface may be a better fit. A completely different interface would something that lets you compile a model into a hardware specific blob in user space and then submit that blob into the kernel, using further commands to send and receive model specific data. As I understand it, this method is roughly what habanalabs and some of the other ones do for inferencing. The performance is almost certainly better here, but it requires a high degree of integration between model, framework, user space driver, compiler and kernel driver. We already do similar things in the gpu, fpga and remoteproc frameworks, all of which could be used here, or we add a more specialized interface. What the actual interfaces should be I have no clue, those two are just examples of what it could be, being completely ignorant of what drivers do today. As Dave said, this really needs a maintainer that understands both the kernel side and what kind of hardware and frameworks exist and what interfaces both sides actually require. Arnd [1] http://www.netlib.org/lapack/explore-html/db/dc9/group__single__blas__level3_gafe51bacb54592ff5de056acabd83c260.html --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@tvm.apache.org For additional commands, e-mail: dev-h...@tvm.apache.org
