mosya415 opened a new issue, #667: URL: https://github.com/apache/tvm-ffi/issues/667
### Summary `_build_impl` / `build_ninja` links the compiled `.so` in place and guards the build only with a `fcntl.flock` lockfile inside the (possibly shared) build directory. On a shared NFS cache with node-local flock semantics, concurrent builders on different hosts race and the link step fails with a Stale-file-handle (ESTALE) error. ### Why it happens When multiple processes on **different nodes** compile the same module into the same content-addressed directory on shared NFS (a common HPC setup: one shared `$HOME`/cache, `TVM_FFI_CACHE_DIR` pinned to NFS), the `fcntl.flock` advisory lock is node-local (NFS mounted `nolock` / `local_lock=all` / no `lockd`), so it does not serialize across nodes. Because outputs (`cuda_0.o`, `<name>.so`) are written in place rather than to a unique temp path and then atomically renamed, one node rewriting the object while another's `ld` reads it yields ESTALE and the link fails. ### Environment - tvm-ffi: 0.1.9 (mechanism unchanged through 0.1.12) - Shared NFS build/cache directory across >=2 nodes, node-local flock semantics - Cold cache (first compile) ### Actual (de-identified) ``` RuntimeError: ninja exited with status 1 ... /usr/bin/ld: final link failed: Stale file handle collect2: error: ld returned 1 exit status ninja: build stopped: subcommand failed. ... OSError: [Errno 116] Stale file handle ``` ### Expected Cold-start builds on a shared filesystem are safe under concurrent multi-host access: build outputs go to a unique per-process temp directory on the same filesystem and are atomically `os.replace`-d into the content-addressed path, and/or the lock does not rely on NFS flock semantics. ### Context Surfaced from sglang, which delegates JIT kernel compilation to tvm-ffi's `build_ninja`. Companion sglang report will link here. Reproduction requires >=2 nodes sharing an NFS cache with node-local flock and a cold cache. Related sglang report: https://github.com/sgl-project/sglang/issues/31347 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
