Still a WIP, posting for people to take a look at. This should maybe become an RFC, happy to write one up. I just needed a `pyproject.toml` for a project I'm working on now so I'm posting it.
It's really hard to reproducibly use TVM outside the docker containers, and depending on what hardware you have at work/home (are they different now?), you can't always run the docker container you need to (I.e. ci-gpu). Clearly documenting our python dependencies will help. I think *just* checking this in as-is would be bad because: - I'm just copying dependencies from one place to another. There isn't anything to enforce that our docker images, and therefore our CI, actually uses the selected versions - That aside, it would be great to generate a constraints file or some other log *outside* the docker images, so that you can have a clear record of e.g. which version of `pylint` is used to lint without needing to pull the container. - In order to actually use `poetry`, I had to hack `setup.py` to hardcode the library location. Maybe we should just do this. Right now if you have `libtvm_runtime.so` in `PATH` then it just uses that one--very bad in development! The reason for this is that dependency tracking tools installing packages in editable mode really want to run something like `python setup.py egg_info` to determine the library version, and unless you've built tvm, poetry will just crash trying to do anything. that's a bad developer experience. - Making `pyproject.toml` the authoritative source of dependencies (I.e. rewriting the docker/install scripts to install only using `poetry`) would be bad because non-users of poetry would have to transcribe the dependencies into e.g. `requirements.txt`. Yes, tools exist to do this; we should do it for them. Here are some future things I'd like to do to address these issues: - [ ] Write a script that uses e.g. dephell to auto-generate `python/requirements.txt` and `python/requirements-<extra>.txt` for each extra section included. - [ ] Write a unit test that asserts that `python/requirements*.txt` is in sync with `pyproject.toml`. - [ ] Make it easy to install all extras in one command (likely needs output from previous script) - [ ] Rewrite `docker/install/ubuntu_install_<extra>.sh` to just run `poetry install -E <extra>`. - [ ] Add `poetry lock` to the docker build process and export the generated lockfile somehow... - [ ] Decide what to do about `setup.py` library location hack. Here's why I think we should use something other than requirements.txt as the authoritative source of dependencies: - In order to ensure a smooth user experience, we do have to pin our dependencies to *some* degree. Here, I've pinned to the package minor or major version where that made sense. - However, of course pinning is bad unless you update your pinned deps. so some dependency management tool provides that capability. I'm not at all attached to poetry as the tool to use for that, but it can facilitate it. You can view, comment on, or merge this pull request online at: https://github.com/apache/incubator-tvm/pull/6620 -- Commit Summary -- * obliterate USE_ANTLR from cmake.config * add poetry deps to pyproject.toml * initial attempt at setup.py + autodetect libtvm_runtime SO path * hack to hardcode in build * make pyproject lock -- File Changes -- M cmake/config.cmake (7) M pyproject.toml (89) M python/setup.py (87) M tests/scripts/task_config_build_cpu.sh (1) M tests/scripts/task_config_build_gpu.sh (1) M tests/scripts/task_config_build_wasm.sh (1) -- Patch Links -- https://github.com/apache/incubator-tvm/pull/6620.patch https://github.com/apache/incubator-tvm/pull/6620.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/apache/incubator-tvm/pull/6620
