[Bug debuginfod/25509] New: Break a cyclic dependency by core packages
https://sourceware.org/bugzilla/show_bug.cgi?id=25509 Bug ID: 25509 Summary: Break a cyclic dependency by core packages Product: elfutils Version: unspecified Status: UNCONFIRMED Severity: normal Priority: P2 Component: debuginfod Assignee: unassigned at sourceware dot org Reporter: marxin.liska at gmail dot com CC: elfutils-devel at sourceware dot org Target Milestone: --- In openSUSE, we do face a problem with cyclic dependencies. Many core packages like gcc, glibc, elfutils or binutils depend on each other and create a cycle. The cycle should contain a reasonable amount of packages. When debuginfod was added to elfutils, we would have a huge bunch of dependencies caused by libhttpmicro and libsqlite. These have very many transitive dependencies. So that I was forced to split elfutils into 2 packages: elfutils and elfutils-debuginfod. The later contains all the new packages and is not part of the boostrap cycle. What's more problematic is that there are (and will be) tools that want to utilize libdebuginfod such as Binutils. As mentioned, the tool is in the bootstrap cycle and so that can't depend on elfutils-debuginfod. So the question is how to unbreak all these dependencies for future core packages? -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/25509] Break a cyclic dependency by core packages
https://sourceware.org/bugzilla/show_bug.cgi?id=25509 --- Comment #1 from Martin Liška --- I have basically 2 possible solutions: - elfutils will provide a stub client library (libdebuginfod-stub.so) which will then do dlopen for the real libdebuginfod.so during run-rime - one can do execv of debuginfod-find which is a standalone tool that can download a debuginfo -- You are receiving this mail because: You are on the CC list for the bug.
contrasting elfutils debuginfod and clr-debug-info server
Hi - Through some comments on phoronix [0], just became (re?)aware of the neat clear-linux debuginfo server facility [1]. It's similar to the nixos one we have mentioned before in that it relies on a local fuse server expose remote debuginfo artifacts under /usr/src, /usr/lib/debug etc. to local tools. The server side relies on a web server [2] that has a carefully unpacked, unified-namespace tree of all the artifacts into individual .tar files. This has some pros/cons, naturally. Some contrasts in a tabular form: clear-linuxelfutils indexed by filenameby buildid client side sw fuse serverapprx. none server side sw plain file server debuginfod server side diskunpacked, unified-namespace layout orig pkg set multi-version maybe, if /usr/src/ filenames unique no problem multi-arch req. separate servers no problem multi-distroreq. separate servers one server ok Neat system. I wonder how interoperation would work. (cc:'ing a few clearlinux folks just for their information.) [0] https://www.phoronix.com/scan.php?page=news_item&px=Red-Hat-FOSDEM-debuginfod [1] https://docs.01.org/clearlinux/latest/guides/clear/debug.html [2] https://download.clearlinux.org/debuginfo/ - FChE
Re: contrasting elfutils debuginfod and clr-debug-info server
On 2/5/2020 9:32 AM, Frank Ch. Eigler wrote: Hi - Through some comments on phoronix [0], just became (re?)aware of the neat clear-linux debuginfo server facility [1]. It's similar to the nixos one we have mentioned before in that it relies on a local fuse server expose remote debuginfo artifacts under /usr/src, /usr/lib/debug etc. to local tools. The server side relies on a web server [2] that has a carefully unpacked, unified-namespace tree of all the artifacts into individual .tar files. This has some pros/cons, naturally. Some contrasts in a tabular form: clear-linuxelfutils indexed by filenameby buildid actually the clear side provides both build id and filename, the tools on the client (gdb and the like) try buildid first, and fall back to filename, and we just do both one of our requirements was that the server side is just "flat http files" so that no active components exist on the server (so that CDNs and mirrors scale this out with low friction)
[Bug general/25511] New: unknown program header entry type 0x6474e553
https://sourceware.org/bugzilla/show_bug.cgi?id=25511 Bug ID: 25511 Summary: unknown program header entry type 0x6474e553 Product: elfutils Version: unspecified Status: NEW Severity: normal Priority: P2 Component: general Assignee: unassigned at sourceware dot org Reporter: amerey at redhat dot com CC: elfutils-devel at sourceware dot org Target Milestone: --- On ubuntu 19.10 x86_64 run-strip-strmerge.sh, run-elflint-self.sh and run-reverse-sections-self.sh all with the same error: "program header entry 9: unknown program header entry type 0x6474e553". This is caused by the inclusion of a new ELF program header PT_GNU_PROPERTY in newer binutils. For more information see https://bugzilla.redhat.com/show_bug.cgi?id=1748802 Steps to reproduce on ubuntu 19.10 x86_64: `./configure; make; make check TESTS='run-strip-strmerge.sh run-elflint-self.sh run-reverse-sections-self.sh'` tests/test-suite.log: FAIL: run-strip-strmerge.sh === elflint /home/amerey/elfutils/tests/elfstrmerge program header entry 9: unknown program header entry type 0x6474e553 FAIL run-strip-strmerge.sh (exit status: 1) FAIL: run-elflint-self.sh = program header entry 9: unknown program header entry type 0x6474e553 *** failure in /home/amerey/elfutils/src/elflint --quiet --gnu-ld /home/amerey/elfutils/src/addr2line program header entry 9: unknown program header entry type 0x6474e553 *** failure in /home/amerey/elfutils/src/elflint --quiet --gnu-ld /home/amerey/elfutils/src/elfclassify program header entry 9: unknown program header entry type 0x6474e553 *** failure in /home/amerey/elfutils/src/elflint --quiet --gnu-ld /home/amerey/elfutils/src/stack program header entry 9: unknown program header entry type 0x6474e553 *** failure in /home/amerey/elfutils/src/elflint --quiet --gnu-ld /home/amerey/elfutils/src/unstrip program header entry 8: unknown program header entry type 0x6474e553 *** failure in /home/amerey/elfutils/src/elflint --quiet --gnu-ld /home/amerey/elfutils/libelf/libelf.so program header entry 8: unknown program header entry type 0x6474e553 *** failure in /home/amerey/elfutils/src/elflint --quiet --gnu-ld /home/amerey/elfutils/libasm/libasm.so FAIL run-elflint-self.sh (exit status: 1) FAIL: run-reverse-sections-self.sh == copy_elf: /home/amerey/elfutils/src/addr2line -> addr2line.rev (read,reverse) setshstrndx: 38 Swapping offsets of section 28 and 29 Swapping offsets of section 30 and 31 Swapping offsets of section 32 and 33 Swapping offsets of section 34 and 35 Swapping offsets of section 37 and 38 program header entry 9: unknown program header entry type 0x6474e553 FAIL run-reverse-sections-self.sh (exit status: 1) -- You are receiving this mail because: You are on the CC list for the bug.
Re: contrasting elfutils debuginfod and clr-debug-info server
Hi, Arjan - > > clear-linuxelfutils > > > > indexed by filenameby buildid > actually the clear side provides both build id and filename, the > tools on the client (gdb and the like) try buildid first, and fall > back to filename, and we just do both AIUI that works by using the "/usr/lib/debug/.build-id/HE/XCODE" as the key, i.e., the build-id as a file name. > one of our requirements was that the server side is just "flat http > files" so that no active components exist on the server (so that > CDNs and mirrors scale this out with low friction) Yup, that makes sense. In our case, aggressive caching with frontend CDNs should also be possible, but staging the data whole at a CDN is not so easy. - FChE
patch rfc: debuginfod -Z (generalized archive) support
Hi -
A little extension lets us process arch-linux archives. Awaiting
for some small test .pkg's from the arch folks for the elfutils
testsuite. However, hand-testing on severa larger files works!
commit b51ae89befeb81c8b51b15b7168c6e616255b486 (fche/pacman-Z)
Author: Frank Ch. Eigler
Date: Wed Feb 5 15:04:18 2020 -0500
debuginfod: generalized archive support
Add a '-Z EXT=CMD' option to debuginfod, which lets it scan any given
extension and run CMD on it to unwrap distro archives. For example,
for arch-linux pacman files, -Z '.tar.zst=zstdcat' lets debuginfod
grok debug and source content in split-debuginfo files.
diff --git a/debuginfod/ChangeLog b/debuginfod/ChangeLog
index 8c97fdcf7085..d812e6d71ff0 100644
--- a/debuginfod/ChangeLog
+++ b/debuginfod/ChangeLog
@@ -1,3 +1,9 @@
+2020-02-05 Frank Ch. Eigler
+
+ * debuginfod.cxx (argp options): Add -Z option.
+ (canonicalized_archive_entry_pathname): New function for
+ distro-agnostic file name matching/storage.
+
2020-01-22 Frank Ch. Eigler
* debuginfod.cxx (dwarf_extract_source_paths): Don't print
diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index 623dbc593c70..0de6bbaea0ee 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -333,9 +333,10 @@ ARGP_PROGRAM_BUG_ADDRESS_DEF = PACKAGE_BUGREPORT;
static const struct argp_option options[] =
{
{ NULL, 0, NULL, 0, "Scanners:", 1 },
- { "scan-file-dir", 'F', NULL, 0, "Enable ELF/DWARF file scanning threads.",
0 },
- { "scan-rpm-dir", 'R', NULL, 0, "Enable RPM scanning threads.", 0 },
- { "scan-deb-dir", 'U', NULL, 0, "Enable DEB scanning threads.", 0 },
+ { "scan-file-dir", 'F', NULL, 0, "Enable ELF/DWARF file scanning.", 0 },
+ { "scan-rpm-dir", 'R', NULL, 0, "Enable RPM scanning.", 0 },
+ { "scan-deb-dir", 'U', NULL, 0, "Enable DEB scanning.", 0 },
+ { "scan-archive", 'Z', "EXT=CMD", 0, "Enable arbitrary archive scanning.",
0 },
// "source-oci-imageregistry" ...
{ NULL, 0, NULL, 0, "Options:", 2 },
@@ -428,6 +429,15 @@ parse_opt (int key, char *arg,
scan_archives[".deb"]="dpkg-deb --fsys-tarfile";
scan_archives[".ddeb"]="dpkg-deb --fsys-tarfile";
break;
+case 'Z':
+ {
+char* extension = strchr(arg, '=');
+if (extension)
+ scan_archives[string(arg, (extension-arg))]=string(extension+1);
+else
+ argp_failure(state, 1, EINVAL, "bad EXT=CMD format");
+ }
+ break;
case 'L':
traverse_logical = true;
break;
@@ -1068,6 +1078,25 @@ class libarchive_fdcache
static libarchive_fdcache fdcache;
+// For security/portability reasons, many distro-package archives have
+// a "./" in front of path names; others have nothing, others have
+// "/". Canonicalize them all to a single leading "/", with the
+// assumption that this matches the dwarf-derived file names too.
+string canonicalized_archive_entry_pathname(struct archive_entry *e)
+{
+ string fn = archive_entry_pathname(e);
+ if (fn.size() == 0)
+return fn;
+ if (fn[0] == '/')
+return fn;
+ if (fn[0] == '.')
+return fn.substr(1);
+ else
+return string("/")+fn;
+}
+
+
+
static struct MHD_Response*
handle_buildid_r_match (int64_t b_mtime,
const string& b_source0,
@@ -1162,8 +1191,8 @@ handle_buildid_r_match (int64_t b_mtime,
if (! S_ISREG(archive_entry_mode (e))) // skip non-files completely
continue;
- string fn = archive_entry_pathname (e);
- if (fn != string(".")+b_source1)
+ string fn = canonicalized_archive_entry_pathname (e);
+ if (fn != b_source1)
continue;
// extract this file to a temporary file
@@ -2055,9 +2084,7 @@ archive_classify (const string& rps, string&
archive_extension,
if (! S_ISREG(archive_entry_mode (e))) // skip non-files completely
continue;
- string fn = archive_entry_pathname (e);
- if (fn.size() > 1 && fn[0] == '.')
-fn = fn.substr(1); // trim off the leading '.'
+ string fn = canonicalized_archive_entry_pathname (e);
if (verbose > 3)
obatched(clog) << "libarchive checking " << fn << endl;
@@ -2764,7 +2791,7 @@ main (int argc, char *argv[])
"unexpected argument: %s", argv[remaining]);
if (scan_archives.size()==0 && !scan_files && source_paths.size()>0)
-obatched(clog) << "warning: without -F -R -U, ignoring PATHs" << endl;
+obatched(clog) << "warning: without -F -R -U -Z, ignoring PATHs" << endl;
fdcache.limit(fdcache_fds, fdcache_mbs);
@@ -2894,7 +2921,7 @@ main (int argc, char *argv[])
obatched ob(clog);
auto& o = ob << "scanning archive types ";
for (auto&& arch : scan_archives)
- o << arch.first << " ";
+ o << arch.first << "(" << arch.second << ") ";
o << endl;
}
const char* du = getenv(DEBUGINFOD_URLS_ENV_VAR);
diff --git
Re: contrasting elfutils debuginfod and clr-debug-info server
On 2/5/2020 11:14 AM, Frank Ch. Eigler wrote: Hi, Arjan - clear-linuxelfutils indexed by filenameby buildid actually the clear side provides both build id and filename, the tools on the client (gdb and the like) try buildid first, and fall back to filename, and we just do both AIUI that works by using the "/usr/lib/debug/.build-id/HE/XCODE" as the key, i.e., the build-id as a file name. sure but it's not "/usr/bin/true" of which you might have different versions over the lifespan of a distro through its updates (e.g. you can connect gdb to a process that got updated on disk later and still get the right answers) the disinction of having the "/.build-id/" prefix versus just the code is not a huge difference (so all the things you can do with just the number you can do with the prefix added as well)
