[Bug debuginfod/25583] New: Use libarchive to extract packages?
https://sourceware.org/bugzilla/show_bug.cgi?id=25583 Bug ID: 25583 Summary: Use libarchive to extract packages? Product: elfutils Version: unspecified Status: UNCONFIRMED Severity: normal Priority: P2 Component: debuginfod Assignee: unassigned at sourceware dot org Reporter: ross at burtonini dot com CC: elfutils-devel at sourceware dot org Target Milestone: --- Using rpm2cpio to extract an RPM isn't ideal because the machine serving the RPMs may not actually have that tool on. However, libarchive which is already a dependency can extract RPMs directly as demonstrated with the libarchive example tool bsdtar: $ bsdtar -tvf /home/ross/Yocto/openbmc/build/tmp/deploy/rpm/noarch/os-release-1.0-r0.noarch.rpm drwxr-xr-x 1 0 0 0 Nov 8 16:41 ./etc lrwxrwxrwx 1 0 0 21 Nov 8 16:41 ./etc/os-release -> ../usr/lib/os-release drwxr-xr-x 1 0 0 0 Nov 8 16:41 ./usr drwxr-xr-x 1 0 0 0 Nov 8 16:41 ./usr/lib -rw-r--r-- 1 0 0 287 Nov 8 16:41 ./usr/lib/os-release Ditto for debs, dpkg-deb may not be present but libarchive can unpack the outer and give you access to the internals: $ bsdtar -tvf /var/cache/apt/archives/gdb_8.2.1-2+b1_amd64.deb -rw-r--r-- 0 0 0 4 Aug 28 16:18 debian-binary -rw-r--r-- 0 0 03024 Aug 28 16:18 control.tar.xz -rw-r--r-- 0 0 0 3131068 Aug 28 16:18 data.tar.xz -- You are receiving this mail because: You are on the CC list for the bug.
Re: Buildbot failure in Wildebeest Builder on whole buildset
On Mon, 2020-02-10 at 22:34 +, build...@builder.wildebeest.org
wrote:
> The Buildbot has detected a failed build on builder whole buildset
> while building elfutils.
> Full details are available at:
> https://builder.wildebeest.org/buildbot/#builders/15/builds/269
This is a somewhat older failure and newer ones did show all green.
I didn't see the fix being posted to the list, so just for the record
here it is:
commit 990955ef0a320a73ed583246f23d5f220e5916f8
Author: Frank Ch. Eigler
Date: Mon Feb 10 14:33:54 2020 -0500
debuginfod testing: SIGUSR2 vs "groom" metric synch
Previous code did not account for a groom job that was already
completed at startup, so the SIGUSR2-triggered one may not have
completed in time for the test. The shell won the race condition on
most buildbot VMs, but a debian builder showed the error of our ways.
Signed-off-by: Frank Ch. Eigler
diff --git a/tests/run-debuginfod-find.sh b/tests/run-debuginfod-find.sh
index 939826cd..1cc8f406 100755
--- a/tests/run-debuginfod-find.sh
+++ b/tests/run-debuginfod-find.sh
@@ -269,7 +269,8 @@ RPM_BUILDID=d44d42cbd7d915bc938c81333a21e355a6022fb7 # in
rhel6/ subdir, for a l
rm -r R/debuginfod-rpms/rhel6/*
kill -USR2 $PID1 # groom cycle
# Expect 3 rpms to be deleted by the groom
-wait_ready $PORT1 'thread_work_total{role="groom"}' 1
+# 1 groom already took place at/soon-after startup, so -USR2 makes 2
+wait_ready $PORT1 'thread_work_total{role="groom"}' 2
wait_ready $PORT1 'groom{statistic="file d/e"}' 3
rm -rf $DEBUGINFOD_CACHE_PATH # clean it from previous tests
[PATCH] tests: Explicitly unset DEBUGINFOD_URLS.
If DEBUGINFOD_URLS is set various tests will try to query the debuginfod server which can stall the tests a bit. If other evironment variables like DEBUGINFOD_PROGRESS are set it will make various tests fail because the expected output doesn't match. Tests should PASS without needing a debuginfod server, unless they test (and set) one themselves. Signed-off-by: Mark Wielaard --- tests/ChangeLog | 5 + tests/Makefile.am | 2 ++ 2 files changed, 7 insertions(+) diff --git a/tests/ChangeLog b/tests/ChangeLog index 01e4a083..32ca1ce0 100644 --- a/tests/ChangeLog +++ b/tests/ChangeLog @@ -1,3 +1,8 @@ +2020-02-21 Mark Wielaard + + * Makefile.am (TESTS_ENVIRONMENT): Explicitly unset DEBUGINFOD_URLS. + (installed_TESTS_ENVIRONMENT): Likewise. + 2020-02-19 Aaron Merey * run-debuginfod-find.sh: Test that files unrelated to debuginfod diff --git a/tests/Makefile.am b/tests/Makefile.am index 1db1150d..a83c3d67 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -495,6 +495,7 @@ installed_TESTS_ENVIRONMENT = libdir=$(DESTDIR)$(libdir); \ export abs_top_builddir; \ export libdir; export bindir; \ export LC_ALL; export LANG; export VALGRIND_CMD; \ + unset DEBUGINFOD_URLS; \ NM=$(NM); export NM; installed_LOG_COMPILER = $(abs_srcdir)/test-wrapper.sh \ installed $(tests_rpath) \ @@ -509,6 +510,7 @@ TESTS_ENVIRONMENT = LC_ALL=C; LANG=C; VALGRIND_CMD=$(valgrind_cmd); \ export abs_srcdir; export abs_builddir; \ export abs_top_builddir; \ export LC_ALL; export LANG; export VALGRIND_CMD; \ + unset DEBUGINFOD_URLS; \ NM=$(NM); export NM; LOG_COMPILER = $(abs_srcdir)/test-wrapper.sh \ $(abs_top_builddir)/libdw:$(abs_top_builddir)/backends:$(abs_top_builddir)/libelf:$(abs_top_builddir)/libasm:$(abs_top_builddir)/debuginfod -- 2.18.2
Re: [PATCH] debuginfod-client: default to XDG cache.
Hi Aaron,
On Wed, 2020-02-19 at 16:04 -0500, Aaron Merey wrote:
> From 8eb82eb58747078a3e914576a7c6ff3f7b2c7cb4 Mon Sep 17 00:00:00 2001
> From: Aaron Merey
> Date: Wed, 19 Feb 2020 12:30:14 -0500
> Subject: [PATCH] PR25502: debuginfod-client default to XDG cache.
>
> Location of client cache now defaults to $XDG_CACHE_HOME/debuginfod_client.
> If XDG_CACHE_HOME is not set then fallback to $HOME/.cache/debuginfod_client.
> Also maintain backwards compatibility with the previous default path-
> if $HOME/.debuginfod_client_cache exists, use that instead of the new
> defaults.
Another option might be to rename $HOME/.debuginfod_client_cache to the
new XDG cache debuginfod_client location. But that might be tricky if
it crosses file systems/mount points. So then you would probably need
to give up and fallback to what you do now if you get EXDEV from rename
(or actually any error). Might be too much work and a little bit too
tricky/clever...?
> diff --git a/debuginfod/ChangeLog b/debuginfod/ChangeLog
> index d812e6d7..fc076737 100644
> --- a/debuginfod/ChangeLog
> +++ b/debuginfod/ChangeLog
> @@ -1,3 +1,9 @@
> +2020-02-19 Aaron Merey
> + * debuginfod-client.c (xalloc_str): New macro. Call
> + asprintf with error checking.
> + (debuginfod_query_server): Use XDG_CACHE_HOME as a default
> + cache location if it is set. Replace snprintf with xalloc_str.
> +
> 2020-02-05 Frank Ch. Eigler
>
> * debuginfod.cxx (argp options): Add -Z option.
> diff --git a/debuginfod/debuginfod-client.c b/debuginfod/debuginfod-client.c
> index e5a2e824..5772367c 100644
> --- a/debuginfod/debuginfod-client.c
> +++ b/debuginfod/debuginfod-client.c
> @@ -98,6 +98,7 @@ static const time_t cache_default_max_unused_age_s =
> 604800; /* 1 week */
> /* Location of the cache of files downloaded from debuginfods.
> The default parent directory is $HOME, or '/' if $HOME doesn't exist. */
> static const char *cache_default_name = ".debuginfod_client_cache";
> +static const char *cache_xdg_name = "debuginfod_client";
> static const char *cache_path_envvar = DEBUGINFOD_CACHE_PATH_ENV_VAR;
>
> /* URLs of debuginfods, separated by url_delim. */
> @@ -359,6 +360,16 @@ add_extra_headers(CURL *handle)
> }
>
>
> +#define xalloc_str(p, fmt, args...)\
> + do \
> +{ \
> + if (asprintf (&p, fmt, args) < 0)\
> +{ \
> + p = NULL;\
> + rc = -ENOMEM;\
> + goto out;\
> +} \
> +} while (0)\
>
> /* Query each of the server URLs found in $DEBUGINFOD_URLS for the file
> with the specified build-id, type (debuginfo, executable or source)
> @@ -373,15 +384,15 @@ debuginfod_query_server (debuginfod_client *c,
> const char *filename,
> char **path)
> {
> - char *urls_envvar;
>char *server_urls;
> - char cache_path[PATH_MAX];
> - char maxage_path[PATH_MAX*3]; /* These *3 multipliers are to shut up gcc
> -Wformat-truncation */
> - char interval_path[PATH_MAX*4];
> - char target_cache_dir[PATH_MAX*2];
> - char target_cache_path[PATH_MAX*4];
> - char target_cache_tmppath[PATH_MAX*5];
> - char suffix[PATH_MAX*2];
> + char *urls_envvar;
> + char *cache_path = NULL;
> + char *maxage_path = NULL;
> + char *interval_path = NULL;
> + char *target_cache_dir = NULL;
> + char *target_cache_path = NULL;
> + char *target_cache_tmppath = NULL;
> + char suffix[PATH_MAX];
>char build_id_bytes[MAX_BUILD_ID_BYTES * 2 + 1];
>int rc;
>
> @@ -447,24 +458,65 @@ debuginfod_query_server (debuginfod_client *c,
> target_cache_path:
> $HOME/.debuginfod_cache/0123abcd/source#PATH#TO#SOURCE ?
>*/
>
> - if (getenv(cache_path_envvar))
> -strcpy(cache_path, getenv(cache_path_envvar));
> + /* Determine location of the cache. The path specified by the debuginfod
> + cache environment variable takes priority. */
> + char *cache_var = getenv(cache_path_envvar);
> + if (cache_var != NULL && strlen (cache_var) > 0)
> +xalloc_str (cache_path, "%s", cache_var);
>else
> {
> - if (getenv("HOME"))
> -sprintf(cache_path, "%s/%s", getenv("HOME"), cache_default_name);
> - else
> -sprintf(cache_path, "/%s", cache_default_name);
> + /* If a cache already exists in $HOME ('/' if $HOME isn't set), then
> use
> + that. Otherwise use the XDG cache directory naming format. */
> + xalloc_str (cache_path, "%s/%s", getenv ("HOME") ?: "/",
> cache_default_name);
> +
> + struct stat st;
> + if (stat (cache_path, &st) < 0)
> +{
> + char cachedir[PATH_MAX];
> + char *xdg = getenv ("XDG_CACHE_HOME");
> +
> + if (xdg != NULL && strlen (xdg) > 0)
> +snpr
Re: [PATCH] debuginfod-client: default to XDG cache.
On Fri, Feb 21, 2020 at 11:13 AM Mark Wielaard wrote:
> On Wed, 2020-02-19 at 16:04 -0500, Aaron Merey wrote:
> > Location of client cache now defaults to $XDG_CACHE_HOME/debuginfod_client.
> > If XDG_CACHE_HOME is not set then fallback to
> > $HOME/.cache/debuginfod_client.
> > Also maintain backwards compatibility with the previous default path-
> > if $HOME/.debuginfod_client_cache exists, use that instead of the new
> > defaults.
>
> Another option might be to rename $HOME/.debuginfod_client_cache to the
> new XDG cache debuginfod_client location. But that might be tricky if
> it crosses file systems/mount points. So then you would probably need
> to give up and fallback to what you do now if you get EXDEV from rename
> (or actually any error). Might be too much work and a little bit too
> tricky/clever...?
We can try a simple rename and fallback if there's any error. We could
mention this briefly in the man page.
> > + if (xdg != NULL && strlen (xdg) > 0)
> > +snprintf (cachedir, PATH_MAX, "%s", xdg);
> > + else
> > +snprintf (cachedir, PATH_MAX, "%s/.cache", getenv ("HOME" ?:
> > "/"));
>
> Are these brackets correct? Maybe ..., getenv ("HOME") ?: "/");
You're right it should be getenv ("HOME") ?: "/".
> > + /* Create XDG cache directory if it doesn't exist. */
> > + if (stat (cachedir, &st) == 0)
> > +{
> > + if (! S_ISDIR (st.st_mode))
> > +{
> > + rc = -EEXIST;
> > + goto out;
> > +}
> > +}
> > + else
> > +{
> > + char *cmd;
> > +
> > + xalloc_str (cmd, "mkdir -p \'%s\'", cachedir);
> > + rc = system (cmd);
>
> Please don't use system. Use rc = mkdir (cachedir, 0700);
Sure. My goal was to have any nonexisting parent directories created as
easily as possible and I wasn't aware of anything besides `mkdir -p`.
Should we not bother creating nonexisting parent directories?
> > + free (cmd);
> > + if (rc == 0)
> > +{
> > + free (cache_path);
> > + xalloc_str (cache_path, "%s/%s", cachedir,
> > cache_xdg_name);
> > +}
> > + else
> > +{
> > + rc = -EINVAL;
> > + goto out;
> > +}
>
> The creation of the directory might race with another client, so
> specifically test for EEXIST and then test S_ISDIR again before
> returning failure.
Ok.
> > .B $HOME/.debuginfod_client_cache
> > -Default cache directory.
> > +Default cache directory. If XDG_CACHE_HOME is not set then
> > +\fB$HOME/.cache/debuginfod_client\fP is used.
> > .PD
>
> Should we mention the old location might still be used?
> Maybe simply ignoring it is fine.
Here I could mention the possibility of a rename when using a client
cache at the old location.
> > .SH "SEE ALSO"
> > diff --git a/tests/ChangeLog b/tests/ChangeLog
> > index 1f55a291..ec081b3c 100644
> > --- a/tests/ChangeLog
> > +++ b/tests/ChangeLog
> > @@ -1,3 +1,7 @@
> > +2020-02-19 Aaron Merey
> > + * run-debuginfod-find.sh: Run tests for verifying default
> > + client cache locations.
>
> Probably too much work for this patch. But it feels run-debuginfod-
> find.sh is doing a lot of different testing. Might be good to put some
> tests in their own test file.
That's a good idea for a future patch, maybe separate client and server
tests into two files (to the extent that they can be separated).
> You need to use testrun to run "test" binaries (or at least make sure
> LD_LIBRARY_PATH is setup correctly) to make sure the test environment
> is setup correctly. The above doesn't work when using srcdir!=builddir
> and testrun will make sure that valgrind is ran (as is done with make
> distcheck) when you configure with --enable-valgrind (which would have
> caught the previous small memleak).
>
> diff --git a/tests/run-debuginfod-find.sh b/tests/run-debuginfod-find.sh
> index 317c687c..083ff59b 100755
> --- a/tests/run-debuginfod-find.sh
> +++ b/tests/run-debuginfod-find.sh
> @@ -152,14 +152,14 @@ cmp $filename ${PWD}/prog.c
> mkdir tmphome
>
> # $HOME/.cache should be created.
> -env HOME=$PWD/tmphome XDG_CACHE_HOME= DEBUGINFOD_CACHE_PATH=
> ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
> +testrun env HOME=$PWD/tmphome XDG_CACHE_HOME= DEBUGINFOD_CACHE_PATH=
> ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
> if [ ! -f $PWD/tmphome/.cache/debuginfod_client/$BUILDID/debuginfo ]; then
>echo "could not find cache in $PWD/tmphome/.cache"
>exit 1
> fi
>
> # $XDG_CACHE_HOME should take priority over $HOME.cache.
> -env HOME=$PWD/tmphome XDG_CACHE_HOME=$PWD/tmpxdg DEBUGINFOD_CACHE_PATH=
> ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo $BUILDID
> +testrun env HOME=$PWD/tmphome XDG_CACHE_HOME=$PWD/tmpxdg
> DEBUGINFOD_CA
Re: [PATCH] debuginfod-client: default to XDG cache.
Hi Aaron,
On Fri, 2020-02-21 at 12:21 -0500, Aaron Merey wrote:
> On Fri, Feb 21, 2020 at 11:13 AM Mark Wielaard wrote:
> > On Wed, 2020-02-19 at 16:04 -0500, Aaron Merey wrote:
> > > + /* Create XDG cache directory if it doesn't exist. */
> > > + if (stat (cachedir, &st) == 0)
> > > +{
> > > + if (! S_ISDIR (st.st_mode))
> > > +{
> > > + rc = -EEXIST;
> > > + goto out;
> > > +}
> > > +}
> > > + else
> > > +{
> > > + char *cmd;
> > > +
> > > + xalloc_str (cmd, "mkdir -p \'%s\'", cachedir);
> > > + rc = system (cmd);
> >
> > Please don't use system. Use rc = mkdir (cachedir, 0700);
>
> Sure. My goal was to have any nonexisting parent directories created as
> easily as possible and I wasn't aware of anything besides `mkdir -p`.
> Should we not bother creating nonexisting parent directories?
I think the spec is slightly ambiguous whether or not you need to
create all parents.
https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html
If, when attempting to write a file, the destination directory is
non-existant an attempt should be made to create it with permission
0700.
But I think that means we should only create the cachedir (either
$XDG_CACHE_HOME or $HOME/.cache). Which is what we are doing here. But
I don't think we are expected to create all parents. I think it is
reasonable to assume they already exist (it would almost always be
$HOME anyway).
Later in the code we create target_cache_path (the debuginfod_client
subdir), again just as one directory (we made sure its parent exists).
BTW. Looking at the code again I see this comment should be adjusted
too:
/* set paths needed to perform the query
example format
cache_path:$HOME/.debuginfod_cache
target_cache_dir: $HOME/.debuginfod_cache/0123abcd
target_cache_path: $HOME/.debuginfod_cache/0123abcd/debuginfo
target_cache_path:
$HOME/.debuginfod_cache/0123abcd/source#PATH#TO#SOURCE ?
*/
Cheers,
Mark
[Bug debuginfod/25583] Use libarchive to extract packages?
https://sourceware.org/bugzilla/show_bug.cgi?id=25583 Frank Ch. Eigler changed: What|Removed |Added Status|UNCONFIRMED |WAITING Last reconfirmed||2020-02-22 CC||fche at redhat dot com Ever confirmed|0 |1 --- Comment #1 from Frank Ch. Eigler --- Interesting idea. OTOH, rpm2cpio and dpkg binaries are not too hard to come by. One can experiment with the former already with git-master debuginfod with the "-Z .rpm" option instead of "-R". Performance seems to be roughly the same. The latter is less compelling in that it'd require hard-coding the inner data.tar.xz name and its processing ... meh. Maybe the status quo is good enough? -- You are receiving this mail because: You are on the CC list for the bug.
