This bug is awaiting verification that the linux/6.14.0-32.32 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-plucky-linux' to 'verification-done-plucky-linux'. If the problem still exists, change the tag 'verification-needed-plucky- linux' to 'verification-failed-plucky-linux'.
If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you! ** Tags added: kernel-spammed-plucky-linux-v2 verification-needed-plucky-linux -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2120454 Title: Pytorch reports incorrect GPU memory causing "HIP Out of Memory" errors Status in linux package in Ubuntu: Fix Released Status in linux-oem-6.14 package in Ubuntu: Invalid Status in linux source package in Noble: Invalid Status in linux-oem-6.14 source package in Noble: In Progress Status in linux source package in Plucky: Fix Committed Status in linux-oem-6.14 source package in Plucky: Invalid Status in linux source package in Questing: Fix Released Status in linux-oem-6.14 source package in Questing: Invalid Bug description: [SRU Justification] [ Impact ] PyTorch running on an APU may report out of memory if 1/2 system memory is reserved for VRAM. This can be identified with `rocminfo` output: 1. for the default setup, 512MB for VRAM on a 64G RAM AMD Strix Halo development board: ``` $ sudo journalctl -b -1 | grep -B1 GTT Aug 15 18:47:46 test kernel: [drm] amdgpu: 512M of VRAM memory ready Aug 15 18:47:46 test kernel: [drm] amdgpu: 31822M of GTT memory ready. $ free -h total used free shared buff/cache available Mem: 62Gi 2.1Gi 3.5Gi 44Mi 57Gi 60Gi Swap: 0B 0B 0B ******* Agent 2 ******* Name: gfx1151 Uuid: GPU-XX Marketing Name: AMD Radeon Graphics ... Pool Info: Pool 1 Segment: GLOBAL; FLAGS: COARSE GRAINED Size: 32586284(0x1f13a2c) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Recommended Granule:2048KB Alloc Alignment: 4KB Accessible by all: FALSE ``` The pool 1 has 31822 MB = 32586284 KB. With dedicated VRAM set to 32GB: ``` $ sudo dmesg | grep -B1 GTT [ 3.640984] [drm] amdgpu: 32768M of VRAM memory ready [ 3.640986] [drm] amdgpu: 15970M of GTT memory ready. $ free -h total used free shared buff/cache available Mem: 31Gi 1.6Gi 28Gi 44Mi 1.3Gi 29Gi Swap: 0B 0B 0B ******* Agent 2 ******* Name: gfx1151 Uuid: GPU-XX Marketing Name: AMD Radeon Graphics ... Pool Info: Pool 1 Segment: GLOBAL; FLAGS: COARSE GRAINED Size: 16353840(0xf98a30) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Recommended Granule:2048KB Alloc Alignment: 4KB Accessible by all: FALSE ``` In this case, while we have 32768 MB = 33554432 KB VRAM, but the pool 1 still allocates 15970 MB = 16353840 KB from GTT. [ Test Plan ] 1. Follow https://rocm.docs.amd.com/projects/install-on- linux/en/latest/install/quick-start.html to install & setup necessary host environment for ROCm. 2. Follow https://rocm.docs.amd.com/projects/install-on- linux/en/latest/install/3rd-party/pytorch-install.html#using-docker- with-pytorch-pre-installed to use prebuilt PyTorch image for easy verification. Or, while we need only `rocminfo` to identify the behavior, one may install rocminfo snap instead to minimize the effort. 3. Assign more memory to VRAM. On AMD Strix Halo development board, it's in: Device Manager => AMD CBS => NBIO Common Options => GFX Configuration => Dedicated Graphics Memory. On the development board we have 64 GB RAM, and the options available are "High (32 GB)", "Medium (16 GB)", "Minimum (0.5 GB)". 4. Use `rocminfo` to identify if the allocation is now switched to VRAM: ``` ******* Agent 2 ******* Name: gfx1151 Uuid: GPU-XX Marketing Name: AMD Radeon Graphics ... Pool Info: Pool 1 Segment: GLOBAL; FLAGS: COARSE GRAINED Size: 33554432(0x2000000) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Recommended Granule:2048KB Alloc Alignment: 4KB Accessible by all: FALSE ``` With kernel patched, the allocated memory size is now 33554432 KB = 32768 MB. [ Where problems could occur ] This corrects the allocation source as expected. [ Other Info ] Nominate for Plucky for 6.14, and Noble for oem-6.14. ========== original bug report ========== When running PyTorch on an APU it reports wrong amount of memory and models can't run. torch.OutOfMemoryError: HIP out of memory. Tried to allocate 18.00 MiB. GPU 0 has a total capacity of 15.60 GiB of which 8.09 MiB is free. Of the allocated memory 15.10 GiB is allocated by PyTorch, and 195.37 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment- variables) These two commits need to be backported into amdkfd to fix it. commit 8b0d068e7dd1 ("drm/amdkfd: add a new flag to manage where VRAM allocations go") commit 759e764f7d58 ("drm/amdkfd: use GTT for VRAM on APUs only if GTT is larger") To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2120454/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : [email protected] Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
