I can confirm this bug. It happens with totem AND mplayer, but it's a xorg bug. This stupid bug keeps reappearing in al xorgs, while I never had this with xfree. The problem is that xorg is not smart enough to allocate memory required for xv framebuffers.
-- Totem receives BadAlloc when playing very large movies using Xv https://launchpad.net/bugs/4229 -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs