Hi Timur,
Am 02.05.23 um 11:12 schrieb Timur Kristóf:
Hi Christian,
Christian König <[email protected]> ezt írta (időpont: 2023.
máj. 2., Ke 9:59):
Am 02.05.23 um 03:26 schrieb André Almeida:
> Em 01/05/2023 16:24, Alex Deucher escreveu:
>> On Mon, May 1, 2023 at 2:58 PM André Almeida
<[email protected]>
>> wrote:
>>>
>>> I know that devcoredump is also used for this kind of
information,
>>> but I believe
>>> that using an IOCTL is better for interfacing Mesa + Linux rather
>>> than parsing
>>> a file that its contents are subjected to be changed.
>>
>> Can you elaborate a bit on that? Isn't the whole point of
devcoredump
>> to store this sort of information?
>>
>
> I think that devcoredump is something that you could use to
submit to
> a bug report as it is, and then people can read/parse as they want,
> not as an interface to be read by Mesa... I'm not sure that it's
> something that I would call an API. But I might be wrong, if you
know
> something that uses that as an API please share.
>
> Anyway, relying on that for Mesa would mean that we would need to
> ensure stability for the file content and format, making it less
> flexible to modify in the future and probe to bugs, while the
IOCTL is
> well defined and extensible. Maybe the dump from Mesa + devcoredump
> could be complementary information to a bug report.
Neither using an IOCTL nor devcoredump is a good approach for this
since
the values read from the hw register are completely unreliable. They
could not be available because of GFXOFF or they could be
overwritten or
not even updated by the CP in the first place because of a hang
etc....
If you want to track progress inside an IB what you do instead is to
insert intermediate fence write commands into the IB. E.g. something
like write value X to location Y when this executes.
This way you can not only track how far the IB processed, but also in
which stages of processing we where when the hang occurred. E.g.
End of
Pipe, End of Shaders, specific shader stages etc...
Currently our biggest challenge in the userspace driver is debugging
"random" GPU hangs. We have many dozens of bug reports from users
which are like: "play the game for X hours and it will eventually hang
the GPU". With the currently available tools, it is impossible for us
to tackle these issues. André's proposal would be a step in improving
this situation.
We already do something like what you suggest, but there are multiple
problems with that approach:
1. we can only submit 1 command buffer at a time because we won't know
which IB hanged
2. we can't use chaining because we don't know where in the IB it hanged
3. it needs userspace to insert (a lot of) extra commands such as
extra synchronization and memory writes
4. It doesn't work when GPU recovery is enabled because the
information is already gone when we detect the hang
You can still submit multiple IBs and even chain them. All you need to
do is to insert into each IB commands which write to an extra memory
location with the IB executed and the position inside the IB.
The write data command allows to write as many dw as you want (up to
multiple kb). The only potential problem is when you submit the same IB
multiple times.
And yes that is of course quite some extra overhead, but I think that
should be manageable.
Consequences:
A. It has a huge perf impact, so we can't enable it always
B. Thanks to the extra synchronization, some issues can't be
reproduced when this kind of debugging is enabled
C. We have to ask users to disable GPU recovery to collect logs for us
In my opinion, the correct solution to those problems would be if the
kernel could give userspace the necessary information about a GPU hang
before a GPU reset.
The fundamental problem here is that the kernel doesn't have that
information either. We know which IB timed out and can potentially do a
devcoredump when that happens, but that's it.
The devcoredump can contain register values and memory locations, but
since the ASIC is hung reading back stuff like the CP or power
management state is dangerous. Alex already noted as well that we
potentially need to disable GFXOFF to be able to read back register values.
To avoid the massive peformance cost, it would be best if we could
know which IB hung and what were the commands being executed when it
hung (perhaps pointers to the VA of the commands), along with which
shaders were in flight (perhaps pointers to the VA of the shader
binaries).
If such an interface could be created, that would mean we could easily
query this information and create useful logs of GPU hangs without
much userspace overhead and without requiring the user to disable GPU
resets etc.
Disabling GPU reset is actually not something you can reliably do
either. If the core memory management runs OOM it can chose to wait for
the GPU reset to finish and when that timeout is infinite the system
just hard hangs.
That's also the reason why you can't wait for userspace to read back
state or something like that.
If it's not possible to do this, we'd appreciate some suggestions on
how to properly solve this without the massive performance cost and
without requiring the user to disable GPU recovery.
The most doable option I can see is to say instead of resetting the GPU
we tell the OS the GPU was hot unplugged and disable system memory
access for the ASIC to prevent random memory corruption.
This way you can investigate the GPU state with tools like umr and only
after that is done we do a hot add event and start over from scratch.
The game will obviously crash and needs to be restarted, but that is
still better than a full system crash.
Side note, it is also extremely difficult to even determine whether
the problem is in userspace or the kernel. While kernel developers
usually dismiss all GPU hangs as userspace problems, we've seen many
issues where the problem was in the kernel (eg. bugs where wrong
voltages were set, etc.) - any idea for tackling those kind of issues
is also welcome.
No idea either. If I had a better idea how to triage things my live
would be much simpler as well :)
Regards,
Christian.
Thanks & best regards,
Timur
