Local Build ID Directory Lookup (DEBUGINFOD_LOCAL_PATH)

[Daniel Thornburgh via Elfutils-devel]

Hi elfutils-devel@,

I've been working on adding debuginfod support to various LLVM utilities,
while simultaneously setting up production use of debuginfod for a managed
developer environment.

One issue that keeps cropping up is that there's often quite a few places
to get debug files from locally, in addition to various remote backends.
For example, the output of local build systems, prebuilt packages and
tarballs that a developer is working on, etc.

One of the best things about debuginfod is that you can set up a service
once, then point an astonishingly wide array of tools at the server, with
one configuration.

This isn't the case for files available locally, but there is some prior
art: GDB's build ID directory layout (commonly
`/usr/lib/debug/.build-id//`). GDB has a flag to control this (`--with-separate-debug-dir`), but
that flag isn't shared by other tools. For example, binutils supports this
layout, but it simply hardcodes a list of directories to search for, with
no way to extend or override it. LLVM also supports this, but has a
differently named flag for it.

Local files can be served to localhost using a debuginfod server, but this
incurs the overhead of transferring the files over a local socket on first
access, and the much worse overhead of storing an additional copy of the
file on disk in the debuginfod cache. With sufficiently large projects,
this gets pretty rough.

It would be really nice if there were, a DEBUGINFOD_LOCAL_PATH environment
variable that provided a colon-separated list of directories in the GDB
`.build-id//` format to locally fetch files out of before
attempting remote servers. This would allow imbuing local build ID fetching
to tools that wouldn't otherwise support it, with one common configuration
language for how to do it.

I did definitely want to raise this on the mailing list before starting to
implement something like this though, since it increases the scope of
libdebuginfod to include local fetching out of more than just out of its
own cache. It would stretch debuginfod towards a more general "library for
obtaining debug info writ large", rather than a straightforward remote
client library.

Daniel Thornburgh | dth...@google.com

Re: Local Build ID Directory Lookup (DEBUGINFOD_LOCAL_PATH)

[Paul Smith]

On Wed, 2023-05-31 at 15:17 -0700, Daniel Thornburgh via Elfutils-devel
wrote:
> This isn't the case for files available locally, but there is some
> prior art: GDB's build ID directory layout (commonly
> `/usr/lib/debug/.build-id// build ID>`). GDB has a flag to control this (`--with-separate-debug-
> dir`), but that flag isn't shared by other tools. For example,
> binutils supports this layout, but it simply hardcodes a list of
> directories to search for, with no way to extend or override it. LLVM
> also supports this, but has a differently named flag for it.

Having content in /usr/lib is less good.  It's perfect for allowing
package managers to install debug info files for packages, but it's not
good for allowing users to either download or "publish" their own info
files.

Couldn't the standard XDG environment variables be used to generate a
search path or paths, rather than inventing new ones (or, the new
variable can be used to override the default which is computed based on
XDG), to find the user's personal build ID directories?  This location
can be updated without root, can easily be made available inside
containers (flatpak / snap), etc.

If everyone could agree on a naming convention so that all tools that
need debug info could write them to a common location after download
(or even on generation), and check that location for files first before
querying debuginfod, that would be a great "it just works" feature and
avoid a lot of duplication.

Or, maybe I'm just misunderstanding something :).

Re: Local Build ID Directory Lookup (DEBUGINFOD_LOCAL_PATH)

[Frank Ch. Eigler via Elfutils-devel]

Hi -

> [...]
> It would be really nice if there were, a DEBUGINFOD_LOCAL_PATH environment
> variable that provided a colon-separated list of directories in the GDB
> `.build-id//` [...]

Please note that $DEBUGINFOD_URLS can already include file:// URLs
that identify local filesystems whose directory structure mirrors the
webapi.  The old-school /usr/lib/debug/.build-id/xx/.debug
file hierarchy doesn't match that, and of course lacks source code
entirely.  

OTOH a shell script that creates debuginfod-style tree of symlinks
into the old-school hierarchy - for debuginfo/executables only - could
work, and work with current day unmodified debuginfod clients &
servers.  i.e.,:

  find /usr/lib/debug/.build-id -name '*.debug' | while read g; do
   buildid=`echo $g | cut -f6,7 -d/ | tr -d / | cut -f1 -d.`
   mkdir -p buildid/$buildid
   ln -s $g buildid/$buildid/debuginfo
  done

and similar for .../executable

Then the resulting tree is suitable for use as DEBUGINFOD_URLS=file://`pwd`
to a debuginfod server or a client.


- FChE

Re: Local Build ID Directory Lookup (DEBUGINFOD_LOCAL_PATH)

[Daniel Thornburgh via Elfutils-devel]

On Wed, May 31, 2023 at 3:35 PM Frank Ch. Eigler  wrote:

> > [...]
> > It would be really nice if there were, a DEBUGINFOD_LOCAL_PATH
> environment
> > variable that provided a colon-separated list of directories in the GDB
> > `.build-id//` [...]
>
> Please note that $DEBUGINFOD_URLS can already include file:// URLs
> that identify local filesystems whose directory structure mirrors the
> webapi.  The old-school /usr/lib/debug/.build-id/xx/.debug
> file hierarchy doesn't match that, and of course lacks source code
> entirely.
>

Ah, I had forgotten completely that file URLs worked out of the box
(libcurl, duh).
When we tried that way back when, the issue was just that it actually does
a libcurl fetch
out of that path and saves another copy of the file into the cache.

So I guess, sans the format, the feature request would just be that it
would have a shortcut
for file URLs to produce the path directly in response to e.g. a
debuginfod_find_debuginfo,
rather than making a copy of the file via libcurl.

Does that seem like a reasonable patch?

Daniel Thornburgh | dth...@google.com

Re: Local Build ID Directory Lookup (DEBUGINFOD_LOCAL_PATH)

[Frank Ch. Eigler via Elfutils-devel]

Hi -

> Ah, I had forgotten completely that file URLs worked out of the box
> (libcurl, duh).

Yeah.

> When we tried that way back when, the issue was just that it
> actually does a libcurl fetch out of that path and saves another
> copy of the file into the cache.

Yes, a copy is made.  This is "traditional" in the sense that the
client code ensures that even if an original upstream server goes
away, cached files stay accessible for a while.  I can also see how
this might be less valuable than just getting at the files with
minimum overhead.  Does anyone think we need to have both as options?

- FChE

Re: Local Build ID Directory Lookup (DEBUGINFOD_LOCAL_PATH)

[Roland McGrath via Elfutils-devel]

In elfutils, the libdwfl library is responsible for the "library for
obtaining debug info writ large" stuff. It provides  API parsers
for command-line arguments that mesh with its library stuff like
Dwfl_Callbacks that set the search path used equivalently to GDB's `set
debug-file-directory` path. eu-* tools as well as things like systemtap use
this to accept `--debuginfo-path=PATH`, but AFAIK none of them support
environment variables.  In the binutils world outside of GDB itself, I
think only objdump and readelf do .build-id/... lookup and those use common
binutils-private code (binutils/dwarf.c) that just has a hard-coded
directory list.  Some more uniformity here would be beneficial to everyone,
I think.

On Wed, May 31, 2023 at 3:17 PM Daniel Thornburgh  wrote:

> Hi elfutils-devel@,
>
> I've been working on adding debuginfod support to various LLVM utilities,
> while simultaneously setting up production use of debuginfod for a managed
> developer environment.
>
> One issue that keeps cropping up is that there's often quite a few places
> to get debug files from locally, in addition to various remote backends.
> For example, the output of local build systems, prebuilt packages and
> tarballs that a developer is working on, etc.
>
> One of the best things about debuginfod is that you can set up a service
> once, then point an astonishingly wide array of tools at the server, with
> one configuration.
>
> This isn't the case for files available locally, but there is some prior
> art: GDB's build ID directory layout (commonly
> `/usr/lib/debug/.build-id// ID>`). GDB has a flag to control this (`--with-separate-debug-dir`), but
> that flag isn't shared by other tools. For example, binutils supports this
> layout, but it simply hardcodes a list of directories to search for, with
> no way to extend or override it. LLVM also supports this, but has a
> differently named flag for it.
>
> Local files can be served to localhost using a debuginfod server, but this
> incurs the overhead of transferring the files over a local socket on first
> access, and the much worse overhead of storing an additional copy of the
> file on disk in the debuginfod cache. With sufficiently large projects,
> this gets pretty rough.
>
> It would be really nice if there were, a DEBUGINFOD_LOCAL_PATH environment
> variable that provided a colon-separated list of directories in the GDB
> `.build-id//` format to locally fetch files out of before
> attempting remote servers. This would allow imbuing local build ID fetching
> to tools that wouldn't otherwise support it, with one common configuration
> language for how to do it.
>
> I did definitely want to raise this on the mailing list before starting to
> implement something like this though, since it increases the scope of
> libdebuginfod to include local fetching out of more than just out of its
> own cache. It would stretch debuginfod towards a more general "library for
> obtaining debug info writ large", rather than a straightforward remote
> client library.
>
> Daniel Thornburgh | dth...@google.com
>
> --
> You received this message because you are subscribed to the Google Groups
> "(retired) tq-toolchain-team" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to tq-toolchain-team+unsubscr...@google.com.
> To view this discussion on the web visit
> https://groups.google.com/a/google.com/d/msgid/tq-toolchain-team/CAEdiOyeCSvpUijvZwJ%3DW2ncsGokefdm8z3yKVG%2Bu4jkAgXcumw%40mail.gmail.com
> 
> .
>
> --
> You received this message because you are subscribed to the Google Groups
> "rvos-toolchain-team" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to rvos-toolchain-team+unsubscr...@google.com.
> To view this discussion on the web visit
> https://groups.google.com/a/google.com/d/msgid/rvos-toolchain-team/CAEdiOyeCSvpUijvZwJ%3DW2ncsGokefdm8z3yKVG%2Bu4jkAgXcumw%40mail.gmail.com
> 
> .
>