The relevant code was added in https://git.savannah.gnu.org/cgit/bash.git/commit/?id=da43077 with similar additions to both gmatch and extmatch, but I suspect the test on line 912 was not meant to be in extmatch:
> .a bash -O extglob -O dotglob -c ': ./!(.foo)' ERROR: AddressSanitizer: heap-buffer-overflow on address 0x000102e02daf READ of size 1 at 0x000102e02daf thread T0 #0 extmatch sm_loop.c:912 frame #5: bash`extmatch(xc=33, s=".a", se="", p="(.foo)", pe="", flags=161) at sm_loop.c:912:36 909 910 if (m1 == 0 && (flags & FNM_DOTDOT) && 911 (SDOT_OR_DOTDOT (s) || -> 912 ((flags & FNM_PATHNAME) && s[-1] == L('/') && PDOT_OR_DOTDOT(s)))) 913 return (FNM_NOMATCH); 914 915 /* if srest > s, we are not at start of string */