It seems that the issue has something to do with the fluidsynth package, when I remove it the issue goes away. So maybe this is a fluidsynth issue as it was updated along with pipewire.
- Bug#1100943: pipewire: Sporadic audio crackling/distortion... Alejandro Rosso
- Bug#1100943: Narrow down the problem Alejandro Rosso
