Found the culprit, it was missing header (libva_x11.h). If anybody interested to know, this caused the compiler to think the function will return a 32 bit pointer, so the compiler inserted a cltq statement into the assembly code which was flipping the upper 32 bits to all 1s.
On Wed, Oct 21, 2015 at 12:15 PM, Ratin <[email protected]> wrote: > # uname -a > Linux nuc1 3.16.0-51-generic #69~14.04.1-Ubuntu SMP Wed Oct 7 15:32:41 UTC > 2015 x86_64 x86_64 x86_64 GNU/Linux > > > > > On Wed, Oct 21, 2015 at 12:13 PM, Ratin <[email protected]> wrote: > >> Breakpoint 1, vaGetDisplay (native_dpy=0x7fffe4312870) at va_x11.c:154 >> 154 { >> (gdb) list >> 149 >> 150 >> 151 VADisplay vaGetDisplay ( >> 152 Display *native_dpy /* implementation specific */ >> 153 ) >> 154 { >> 155 VADisplay dpy = NULL; >> 156 VADisplayContextP pDisplayContext; >> 157 >> ;;; >> 179 pDisplayContext->opaque = NULL; >> 177 pDisplayContext->vaDestroy = va_DisplayContextDestroy; >> 178 pDisplayContext->vaGetDriverName = >> va_DisplayContextGetDriverName; >> 181 dpy = >> (VADisplay)pDisplayContext; >> 195 } >> (gdb) p dpy >> $2 = (VADisplay) 0x7fffe42d3160 >> (gdb) n <========================== stack unroll happens here >> (gdb) list >> 935 pVAAPICNTX->vaDisplay = (VADisplay) <==== upper level >> code vaGetDisplay(pVAAPICNTX->mX11Display); >> 936 pVAAPICNTX->voWindow = pRendererParams->mWindow; >> (gdb) p pVAAPICNTX->vaDisplay >> $3 = (void *) 0xffffffffe42d3160 >> >> >> pVAAPICNTX is malloced struct ptr and one of the element of it is the >> vaDisplay declated like this : >> >> typedef struct _VAAPICNTX >> { >> uint32_t VAContextID; >> Window voWindow; >> int voWidth; >> int voHeight; >> ;;; >> ;;; >> VADisplay vaDisplay; >> >> ;;; >> >> ;;; >> >> }VAAPICNTX, *PVAAPICNTX; >> >> >> As you can see, dpy above around line 195 in vaGetDisplay function is >> shown as 0x7fffe42d3160 but once the program counter comes out of that >> function and stack rolling happens, and context switches to upper level >> code, the assigned value has 0xffffffffe42d3160 (sign extended vs >> unsigned). What might cause this ? Next instance when I call vaInitialize >> with this vaDisplay, it craches: >> >> Program received signal SIGSEGV, Segmentation fault. >> vaDisplayIsValid (dpy=0xffffffffe42d3450) at va.c:102 >> 102 return pDisplayContext && (pDisplayContext->vadpy_magic == >> VA_DISPLAY_MAGIC) && pDisplayContext->vaIsValid(pDisplayContext); >> >> >> >> Any help/pointer would be appreciated. Thanks >> >> >> >> gcc --version >> gcc (Ubuntu 4.8.4-2ubuntu1~14.04) 4.8.4 >> Copyright (C) 2013 Free Software Foundation, Inc. >> This is free software; see the source for copying conditions. There is NO >> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR >> PURPOSE. >> >> ls -la libc-2.19.so >> -rwxr-xr-x 1 root root 1845024 Dec 4 2014 libc-2.19.so >> >> ldd --version >> ldd (Ubuntu EGLIBC 2.19-0ubuntu6.5) 2.19 >> >> >> Platform: >> >> Ratin >> >> >> >> >> >
_______________________________________________ Libva mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/libva
