https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94740
--- Comment #4 from Peter Bergner <bergner at gcc dot gnu.org> --- Removing the bswap and recompiling, the non-bswap load pattern has no problem accepting an address like that, so this looks like a target issue with that pattern.
