Re: patch 0/2 debuginfod submission

2019-10-31 Thread Mark Wielaard
t wondering if you thought about that > > in the scope of the libdwfl calls. Currently there is no real good way > > to do any error reporting there. So it isn't a big concern. But if you > > have any ideas for improvement that would be nice. > > Will think about it. One idea is described here: https://lists.fedorahosted.org/pipermail/elfutils-devel/2015-May/thread.html#4841 Cheers, Mark

Re: [PATCH 3/5] libdwfl: add interface for attaching to/detaching from threads

2019-10-31 Thread Mark Wielaard
On Wed, 2019-10-30 at 16:49 -0700, Omar Sandoval wrote: > On Wed, Oct 30, 2019 at 01:47:52PM +0100, Mark Wielaard wrote: > > Also, if we would adopt dwfl_attach_thread then I think it should take > > a Dwfl_Thread object not a pid/tid as argument. > > In that case, we'

Re: [PATCH 4/5] libdwfl: cache Dwfl_Module and Dwarf_Frame for Dwfl_Frame

2019-10-31 Thread Mark Wielaard
On Wed, 2019-10-30 at 16:55 -0700, Omar Sandoval wrote: > On Wed, Oct 30, 2019 at 02:04:42PM +0100, Mark Wielaard wrote: > > Hi Omar, > > > > On Mon, 2019-10-07 at 02:05 -0700, Omar Sandoval wrote: > > > The next change will need to have the Dwarf_Frame readily a

Re: [PATCH 5/5] libdwfl: add interface for evaluating DWARF expressions in a frame

2019-10-31 Thread Mark Wielaard
On Wed, 2019-10-30 at 16:59 -0700, Omar Sandoval wrote: > On Wed, Oct 30, 2019 at 02:23:06PM +0100, Mark Wielaard wrote: > > Having some examples/testcases would also show how/where to get the > > DWARF expressions to use with this new function. > > Sounds good, I'll pu

Re: [PATCH] libdw: Don't free uninitialized Dwarf_Abbrev_Hash's of "fake" CUs.

2019-10-31 Thread Mark Wielaard
> - p->split->dbg->fake_addr_cu = NULL; > - INTUSE(dwarf_end) (p->split->dbg); > + Dwarf_Abbrev_Hash_free (&p->abbrev_hash); > + > + /* Free split dwarf one way (from skeleton to split). */ > + if (p->unit_type == DW_UT_skeleton > + && p->split != NULL && p->split != (void *)-1) > + { > + /* The fake_addr_cu might be shared, only release one. */ > + if (p->dbg->fake_addr_cu == p->split->dbg->fake_addr_cu) > + p->split->dbg->fake_addr_cu = NULL; > + INTUSE(dwarf_end) (p->split->dbg); > + } > } The Dwarf_Abbrev_Hash_free looks in the correct place. But I don't believe the Free split dwarf hunk should not be under the same if not-fake block. Thanks, Mark

Re: [PATCH] libdw: Don't free uninitialized Dwarf_Abbrev_Hash's of "fake" CUs.

2019-11-01 Thread Mark Wielaard
needed values. Your original approach however makes more sense if you like valgrind memcheck to catch more issues. Could you add a Signed-off-by and ChangeLog entry? Either for your original patch or this one. Whichever you like best.. Thanks, Mark

Re: [PATCH] libdw: Don't free uninitialized Dwarf_Abbrev_Hash's of "fake" CUs.

2019-11-01 Thread Mark Wielaard
Hi Jonathon, On Fri, 2019-11-01 at 08:14 -0500, Jonathon Anderson wrote: > How about a bit of both, I'll be happy if I don't initialize > `unit_type`. > Does this work? Looks perfect. Pushed to master. Thanks, Mark

Re: [PATCH 3/3] lib + libdw: Add and use a concurrent version of the dynamic-size hash table.

2019-11-04 Thread Mark Wielaard
I'll have to defer to Srdan on this one. Srdan, any feedback on this? Thanks, Mark From b70b350242d9752f41407c0ed7fe4683c8f31ce6 Mon Sep 17 00:00:00 2001 From: Mark Wielaard Date: Sat, 26 Oct 2019 01:54:43 +0200 Subject: [PATCH] no compare or val pass --- lib/dynamicsizehas

Re: [PATCH 3/3] lib + libdw: Add and use a concurrent version of the dynamic-size hash table.

2019-11-04 Thread Mark Wielaard
Hi, On Mon, 2019-10-28 at 21:12 +0100, Mark Wielaard wrote: > On Sun, 2019-10-27 at 17:13 +0100, Mark Wielaard wrote: > > On Fri, 2019-10-25 at 23:11 -0500, Jonathon Anderson wrote: > > > A quick grep -r shows that ITERATE and REVERSE are used for the > > > asm_s

Re: [PATCH] lib + libdw: Add and use a concurrent version of the dynamic-size hash table.

2019-11-07 Thread Mark Wielaard
were intended. Thanks, Mark

Re: [PATCH 1/2] Add configure options for Valgrind annotations.

2019-11-07 Thread Mark Wielaard
Hi Jonathon, On Tue, 2019-10-29 at 22:14 +0100, Mark Wielaard wrote: > +2019-08-25 Jonathon Anderson > + > + * configure.ac: Add new --enable-valgrind-annotations > + * configure.ac: Add new --with-valgrind (headers only) I think this looks fine. In theory we could also tr

Re: [PATCH 2/2] libdw: Rewrite the memory handler to be more robust.

2019-11-07 Thread Mark Wielaard
Hi Jonathan, On Tue, 2019-10-29 at 22:14 +0100, Mark Wielaard wrote: > Pthread's thread-local variables are highly limited, which makes > it difficult to use many Dwarfs. This replaces that with a > less efficient (or elegant) but more robust method. Thanks, it looks good (similar

Re: [PATCH] lib + libdw: Add and use a concurrent version of the dynamic-size hash table.

2019-11-08 Thread Mark Wielaard
Hi, On Thu, 2019-11-07 at 09:24 -0600, Jonathon Anderson wrote: > On Thu, Nov 7, 2019 at 12:07, Mark Wielaard wrote: > > Looking at the difference between the previous version and this one, > > it > > incorporates my simplification of FIND and lookup functions. And f

Re: [PATCH 2/2] libdw: Rewrite the memory handler to be more robust.

2019-11-08 Thread Mark Wielaard
er than the previous version, and compared to 0.177 things are maybe ~1% slower (so probably in the noise). A factor 1.5x-2.0x slower in parallel does seem significant. Is that in the case of many-threads that are colliding a lot or in general? Thanks, Mark

Re: [PATCH 2/2] libdw: Rewrite the memory handler to be more robust.

2019-11-08 Thread Mark Wielaard
Hi, On Thu, 2019-11-07 at 12:13 -0600, Jonathon Anderson wrote: > On Thu, Nov 7, 2019 at 18:20, Mark Wielaard wrote: > > Do we really need this? > > We already use __thread unconditionally in the rest of the code. > > The usage of threads.h seems to imply we actually want

Re: [PATCH] lib + libdw: Add and use a concurrent version of the dynamic-size hash table.

2019-11-10 Thread Mark Wielaard
> collision), in that case the new Abbrev would stick around until the > dwarf_end. Since the memblocks are per thread, it seems we could easily back out an allocation we don't need as long as the thread hasn't done any other allocation from the memblock. What do you think of the

Re: patch 1/2 debuginfod client

2019-11-12 Thread Mark Wielaard
1.2]) > +], [enable_debuginfod="no"]) > +AM_CONDITIONAL([DEBUGINFOD],[test "x$enable_debuginfod" = "xyes"]) Would it be better to enable by default with an error to disable if any dependencies (and CXX compiler?) aren't found? If not, then I think you should add it explicitly to the AM_DISTCHECK_CONFIGURE_FLAGS in Makefile.am. > AC_OUTPUT > > AC_MSG_NOTICE([ > @@ -669,6 +689,7 @@ AC_MSG_NOTICE([ >OTHER FEATURES > Deterministic archives by default : ${default_ar_deterministic} > Native language support: ${USE_NLS} > +Debuginfo server support : ${enable_debuginfod} client and server support? Cheers, Mark

Re: patch 1/2 debuginfod client

2019-11-12 Thread Mark Wielaard
, debuginfod expects source queries to prefix relative path > +names with the CU compilation-directory. > + > +If \fIpath\fP is not NULL and the query is successful, \fIpath\fP is set > +to the path of the file in the cache. The caller must \fBfree\fP() this > value. Ah, good, path is me

Re: [PATCH] lib + libdw: Add and use a concurrent version of the dynamic-size hash table.

2019-11-12 Thread Mark Wielaard
us, if you replace the `Abbrev_find == NULL` with `1`, you can make > everything collide. Which I did too, and it still works fine.) O nice trick. Just build and ran a distcheck with this to be sure things still work. Thanks for your review and testing. Pushed to master. Cheers, Mark

Re: patch 1/2 debuginfod client

2019-11-12 Thread Mark Wielaard
could keep it this way > (manually opt in) for this release to minimize build complications for > our non-rpm/intimate distro partners and then switch to opt-out the > later. I think it is better to enable it by default and error out when the dependencies aren't found. Just add an explicit notice that people can disable it when they don't want it. AC_MSG_NOTICE([checking debuginfod dependencies, disable to skip]) or something like that. Thanks, Mark

Re: patch 1/2 debuginfod client

2019-11-13 Thread Mark Wielaard
+ > + * dwfl_build_id_find_elf.c (dwfl_build_id_find_elf): Call debuginfod > + client functions via dlopen to look for elf/dwarf files as fallback. > + * find-debuginfo.c (dwfl_standard_find_debuginfo): Ditto. > + > 2019-08-12 Mark Wielaard > > * gzip.

Re: patch 2/2 debuginfod server etc.

2019-11-13 Thread Mark Wielaard
> +2019-10-28 Frank Ch. Eigler > + > + * eu.am (AM_CXXFLAGS): Clone & amend AM_CFLAGS for c++11 code. > + * debuginfod.service, debuginfod.sysconfig: New files: systemd. > + * Makefile.am: Install them. > + * elfutils.spec.in: Add debuginfod and debug

Re: patch 2/2 debuginfod server etc.

2019-11-13 Thread Mark Wielaard
t database for certificate verification with HTTPS? > +.SH "ENVIRONMENT VARIABLES" > + > +.TP 21 > +.B DEBUGINFOD_URLS > +This environment variable contains a list of URL prefixes for trusted > +debuginfod instances. Alternate URL prefixes are separated by space. > +Avoid referential loops that cause a server to contact itself, directly > +or indirectly - the results would be hilarious. > + > +.TP 21 > +.B DEBUGINFOD_TIMEOUT > +This environment variable governs the timeout for each debuginfod HTTP > +connection. A server that fails to respond within this many seconds > +is skipped. The default is 5. > + > +.TP 21 > +.B DEBUGINFOD_CACHE_PATH > +This environment variable governs the location of the cache where > +downloaded files are kept. It is cleaned periodically as this > +program is reexecuted. The default is $HOME/.debuginfod_client_cache. > +.\" XXX describe cache eviction policy In this case that is downloaded files from other, remote, debuginfod servers? > +.SH FILES > +.LP > +.PD .1v > +.TP 20 > +.B $HOME/.debuginfod.sqlite > +Default database file. > +.PD > + > +.TP 20 > +.B $HOME/.debuginfod_client_cache > +Default cache directory for content from upstream debuginfods. > +.PD > + > + > +.SH "SEE ALSO" > +.I "debuginfod-find(1)" > +.I "sqlite3(1)" Thanks so much for writing so much documentation! I haven't had time to read all the code yet. But I already feel pretty good about it because the documentation is so extensive. Cheers, Mark

Re: patch 2/2 debuginfod server etc.

2019-11-14 Thread Mark Wielaard
Hi, On Mon, 2019-10-28 at 15:07 -0400, Frank Ch. Eigler wrote: > Add the server to the debuginfod/ subdirectory. This is a highly > multithreaded c++11 program (still buildable on rhel7's gcc 4.8, > which is only partly c++11 compliant). Includes an initial suite > of tests, man pages, and a sam

Re: [PATCH] dwelf_elf_e_machine_string: Clear errno before calling strtol

2019-11-14 Thread Mark Wielaard
On Thu, Nov 14, 2019 at 02:54:58PM +0100, Andreas Schwab wrote: > Avoid spurious failure if errno is modified by any other library call in > the test. Looks correct. Pushed to master. Thanks, Mark

Re: [PATCH] run-large-elf-file.sh: skip if free memory information is not available

2019-11-14 Thread Mark Wielaard
Pushed to master. Thanks, Mark P.S. You forgot the Signed-off-by line.

Re: patch 2/2 debuginfod server etc.

2019-11-15 Thread Mark Wielaard
On Thu, 2019-11-14 at 21:44 +0100, Mark Wielaard wrote: > > + // validate buildid > > + if ((buildid.size() < 2) || // not empty > > + (buildid.size() % 2) || // even number > > + (buildid.find_first_not_of("0123456789abcdef") != > >

Re: patch 2/2 debuginfod server etc.

2019-11-15 Thread Mark Wielaard
* run-debuginfod-find.sh, debuginfod_build_id_find.c: New test. > + * testfile-debuginfod-*.rpm.bz2: New data files for test. > + * Makefile.am: Run it. > + > 2019-09-02 Mark Wielaard > > * run-readelf-s.sh: Add --dyn-syms case. > diff --git a/tests/Makefile.am b

Re: patch 3/3 debuginfod client interruptability

2019-11-15 Thread Mark Wielaard
th libdw and find-debuginfo directly. Cheers, Mark

Re: patch 4 debuginfod: symlink following mode

2019-11-15 Thread Mark Wielaard
debuginfod-find < -Usage: ../../debuginfod/debuginfod-find debuginfo BUILDID > - or: ../../debuginfod/debuginfod-find executable BUILDID > - or: ../../debuginfod/debuginfod-find source BUILDID /FILENAME > -EOF > - This wasn't a very useful test, but did you really intend to remove it? Cheers, Mark

Re: patch 5 debuginfod: prometheus metrics

2019-11-15 Thread Mark Wielaard
rics > +curl http://localhost:$PORT1/metrics | grep -q > 'http_responses_total.*result.*error' > +curl http://localhost:$PORT2/metrics | grep -q > 'http_responses_total.*result.*upstream' > +fi > > I think it is better to check with: if type curl >/dev/null 2>&1; then Which avoid executing which, which might not be installed... Cheers, Mark

Re: patch 3/3 debuginfod client interruptability

2019-11-15 Thread Mark Wielaard
Hi Aaron, On Fri, 2019-11-15 at 12:03 -0500, Aaron Merey wrote: > On Fri, Nov 15, 2019 at 11:16 AM Mark Wielaard wrote: > > On Mon, 2019-11-04 at 16:48 -0500, Frank Ch. Eigler wrote: > > > At the wise counsel of gdb folks such as and : > > > > > > deb

Re: patch 1/2 debuginfod client

2019-11-15 Thread Mark Wielaard
],[enable_debuginfod=no]) PKG_CHECK_MODULES([libarchive],[libarchive >= 3.1.2],[],[enable_debuginfod=no]) +if test "x$enable_debuginfod" = "xno"; then + AC_MSG_ERROR([C++ compiler or dependencies not found, use --disable-debuginfod to disable.]) +fi ]) AS_IF([test "x$enable_debuginfod" != "xno"],AC_DEFINE([ENABLE_DEBUGINFOD],[1],[Build debuginfod])) I can push that on top of the debuginfod-submit branch, unless that interferes with your work. I guess you want to squash and rebase it again? Thanks, Mark

Re: patch 1/2 debuginfod client

2019-11-15 Thread Mark Wielaard
certificates. But the client side might or might not verify the server side ssl certificate. Does it do that? And if so, which trusted roots does it use? And can that be disabled or changed? > > > + if (rc < 0) > > > +{ > > > + fprintf(stderr, "Server query failed: %s\n", strerror(-rc)); > > > + return 1; > > > +} > > > > Is there any way we can get/print the actual URL tried here? > > That would really help the user trying to figure out what happened. > > Hm, one problem here is that, while a subsequent version of > debuginfod-find does have a verbosity command line option, the code > itself does not know the URL. That's delegated to the -client solib. Maybe if we had a debuginfo-client request object it could store the URLs tried...? :) > > > +.TP 21 > > > +.B DEBUGINFOD_TIMEOUT > > > +This environment variable governs the timeout for each debuginfod HTTP > > > +connection. A server that fails to respond within this many seconds > > > +is skipped. The default is 5. > > > > zero isn't allowed? > > I suppose that wouldn't make any sense. Anything will take a nonzero > amount of time. :-) Maybe it could be a floating point number or > something; worth it? I was more thinking zero == infinity (no timeout). Cheers, Mark

Re: patch 1/2 debuginfod client

2019-11-15 Thread Mark Wielaard
imilar code in the other file and if you update one, then you probably want to update the other too. Cheers, Mark

Re: patch 2/2 debuginfod server etc.

2019-11-15 Thread Mark Wielaard
nfod client such that it was > dlopen'd into libdw(fl) because you didn't want all the > debuginfod-client (libcurl) required solibs to be loaded into the > program - or into the minimal elfutils installation footprint. This > would undo the latter. Lets do it. That I want it to be optional doesn't mean I don't want to see it everywhere possible :) I'll push it to debuginfod-submit. Cheers, Mark

Some debuginfod fixlets

2019-11-16 Thread Mark Wielaard
NU C 4.8.5 20150623 (Red Hat 4.8.5-39) -mtune=generic -march=x86-64 -g" language (data1) C89 (1) name (strp) "/home/mark/build/elfutils-obj/prog.c" low_pc (addr) 0x004004ed high_p

Re: Some debuginfod fixlets

2019-11-17 Thread Mark Wielaard
Hi, On Sat, 2019-11-16 at 17:42 +0100, Mark Wielaard wrote: > While testing Frank's new spec/rpms for the run-debuginfod-find.sh > testcase I found a couple of issues that I pushed to the debuginfod- > submit branch. > > Add tests/debuginfod-rpms to EXTRA_DISTS. > Fix

Re: patch 3/3 debuginfod client interruptability

2019-11-17 Thread Mark Wielaard
Hi, On Fri, 2019-11-15 at 18:14 +, Pedro Alves wrote: > On 11/15/19 5:35 PM, Mark Wielaard wrote: > > > IMHO it would be best to avoid any global state from the start. Since > > we haven't released this api yet we can make it so that it is easy to > > have state

Re: patch 2/2 debuginfod server etc.

2019-11-18 Thread Mark Wielaard
es include a > database-size report field, so ... I suppose we could quit when the > database gets large enough, but that wouldn't help anyone. Or could > stop indexing early ... but really the Proper solution is to rely on a > size-quota'd cachedir which is harmless if it hits ENOSPC, if it is > brought to the sysadmin's attention by our errors and by system > monitoring tools. Yeah. OK. I guess it isn't different from any other cache. > > Same question as with the client. Does it use the standard trust > > database for certificate verification with HTTPS? > > The text says that debuginfod does not provide https service at all, > so this is not relevant. A front-end https reverse-proxy would have > to deal with this stuff. I plan to cover this in a blog post later > on, and probably in the form of software baked into a container image. But debuginfod might use HTTPS services itself for fallback. I think it is important to describe how/if those https ssl/tls connections are authenticated. Cheers, Mark

Re: patch 2/2 debuginfod server etc.

2019-11-18 Thread Mark Wielaard
> Old debugedit truncated some strings by adding /// (to fill up the > > spaces till the '\0'...) Yes, terrible :{ > > It should just work(tm) if the debugger uses the documented convention > and preserves those extra slashes (just as if it preserved ../ and > such). See also the other email/review. So, we always add an '/' between comp_dir and the file. That should probably be explicitly documented because I wouldn't be surprised if some code doesn't when comp_dir already ends in a slash. I just double checked that dwarf_getsrclines always (unconditionally) adds a '/', so the use of dwarf_filesrc here does the right thing. Thanks, Mark

Re: patch 2/2 debuginfod server etc.

2019-11-18 Thread Mark Wielaard
> > > + > > > +testrun ${abs_builddir}/debuginfod_build_id_find -e F/prog 1 > > > + > > > +testrun ${abs_top_builddir}/debuginfod/debuginfod-find debuginfo > > > $BUILDID2 && false || true > > > > OK. But that means zero means never cache/always clean? > > I would have expected 0 to mean "forever". > > I see the man page doesn't specifically disclose the interpretation of > zero. A "no retention of prior results" purpose is useful, and is > consistent with 0 as per the text. A "retain forever" setting would > have to be a different value. Could you add the current interpretation of zero to the manual page? We could have some other setting (-1?) for "forever", but that doesn't seem urgent. Cheers, Mark

Re: patch 5 debuginfod: prometheus metrics

2019-11-18 Thread Mark Wielaard
> > > +#define gettid() syscall(SYS_gettid) > > > +#else > > > +#define gettid() pthread_self() > > > +#endif > > > > You might want to rename this since newer glibc might expose gettid(). > > OK. Note that the current code defines tid () as syscall(SYS_getpid). Should be SYS_gettid. Cheers, Mark

Re: patch 4 debuginfod: symlink following mode

2019-11-18 Thread Mark Wielaard
ption makes perfect sense for the case you show, but I think it is not really appropriate to mix it with the individual exec/debug and source files case. Cheers, Mark

Re: patch 1/2 debuginfod client

2019-11-18 Thread Mark Wielaard
hing will take a nonzero > > > amount of time. :-) Maybe it could be a floating point number or > > > something; worth it? > > > > I was more thinking zero == infinity (no timeout). > > An unset environment variable should do that. Are you sure? If DEBUGINFOD_TIMEOUT isn't set, then it seems it defaults to 5 seconds: /* Timeout for debuginfods, in seconds. This env var must be set for debuginfod-client to run. */ static const char *server_timeout_envvar = DEBUGINFOD_TIMEOUT_ENV_VAR; static int server_timeout = 5; [...] if (getenv(server_timeout_envvar)) server_timeout = atoi (getenv(server_timeout_envvar)); Cheers, Mark

Re: patch 3/3 debuginfod client interruptability

2019-11-19 Thread Mark Wielaard
w/elfutils/log/?h=debuginfod-client-context If you think those patches are ok I can squash them and add them to the debuginfod-submit branch. Cheers, Mark From 8006db933298ecca35a04207aad3991b242c21da Mon Sep 17 00:00:00 2001 From: Mark Wielaard Date: Tue, 19 Nov 2019 13:09:24 +0100 Subject: [P

Re: patch 2/2 debuginfod server etc.

2019-11-19 Thread Mark Wielaard
might use HTTPS services itself for fallback. I think it > > is important to describe how/if those https ssl/tls connections are > > authenticated. > > OK, will add a blurb. Thanks. So we don't do any authentication right now even when using https? Is that deliberate? What would it take to let it use the system certs for authentication? Cheers, Mark

Re: patch 1/2 debuginfod client

2019-11-19 Thread Mark Wielaard
buginfods, in seconds. > >This env var must be set for debuginfod-client to run. */ > > static const char *server_timeout_envvar = > > DEBUGINFOD_TIMEOUT_ENV_VAR; > > static int server_timeout = 5; > > [...] > > > > if (getenv(server_timeout_envvar)) > > server_timeout = atoi (getenv(server_timeout_envvar)); > > OK, hm, we could make an -empty- but set environment variable mean > 'infinity'. Then again, a user can also say =9. Yes. In this case even setting it to 600 probably feels like forever anyway since someone is waiting on the file... :) Cheers, Mark

Re: patch 5 debuginfod: prometheus metrics

2019-11-19 Thread Mark Wielaard
gs found in them. I still would like an option to turn the metrics off, but I don't think it needs to be on by default since the information exposed doesn't seem to really be that sensitive. So lets just mark this as future wishlist. Cheers, Mark

Re: patch 2/2 debuginfod server etc.

2019-11-19 Thread Mark Wielaard
iberate? What would it take to let it use the system certs > > for authentication? > > System certs do not serve to authenticate clients. Client > certificates are per-user things that come with their own management > headaches. Will think about authentication matters in the future. I thought ca-certificates.crt were normally used to authenticate remote servers. Cheers, Mark

Re: patch 1/2 debuginfod client

2019-11-19 Thread Mark Wielaard
ole, an application could be dlopen()ing libdw.so > itself. It could, but I think that would be unlikely. We can at least document that it is unsafe to use libdw.so with dlopen. But if it is done, could we detect it and not do the loading of libdebuginfod.so in that case? Thanks, Mark

Re: patch 2/2 debuginfod server etc.

2019-11-20 Thread Mark Wielaard
ally provided by the distro, so can't possibly serve as unique > *client* authentication. I think we are talking past each other here. I am not really interested in "client certificates". I am simply interested in knowing what is done for outgoing https connections to be authenticated. What would it take to use the trust roots for validating the server certificates? Thanks, Mark

Re: patch 1/2 debuginfod client

2019-11-20 Thread Mark Wielaard
Hi, On Tue, 2019-11-19 at 16:22 -0500, Frank Ch. Eigler wrote: > On Tue, Nov 19, 2019 at 09:15:20PM +0100, Mark Wielaard wrote: > > > That's what the doc says. What the code does, as far back as the year > > > 2001, is have a static flag/counter in curl_global_init() th

Re: patch 1/2 debuginfod client

2019-11-21 Thread Mark Wielaard
I assume you mean the second part. The attached is what I would > > propose > > for the first part. Do you think that is a bad idea? > > It's mostly harmless, so if you like it, go for it. I added it to the debuginfod-submit branch. Do note that you will want to rebase it to master again. The tests now need the fixes by Andreas (commit be1ed9). Cheers, Mark

Re: patch 2/2 debuginfod server etc.

2019-11-21 Thread Mark Wielaard
Hi, On Wed, 2019-11-20 at 12:53 +0100, Mark Wielaard wrote: > Sure, you could use that if you wanted to share your whole build/source > trees and don't mind serving any other files on some local network. I > just think it shouldn't be the default. If you go look for odd path

Re: patch 2/2 debuginfod server etc.

2019-11-21 Thread Mark Wielaard
On Thu, 2019-11-21 at 15:16 +0100, Mark Wielaard wrote: > Hi, > > On Wed, 2019-11-20 at 12:53 +0100, Mark Wielaard wrote: > > Sure, you could use that if you wanted to share your whole > > build/source > > trees and don't mind serving any other files on some local

Re: patch 2/2 debuginfod server etc.

2019-11-21 Thread Mark Wielaard
nd -F argument parsing. If that is the case, maybe just picking a default for how to interpret the extra arguments, as dirs for the file scanner or dirs for the rpm scanner or both, might make us both happy? Cheers, Mark

Re: patch 2/2 debuginfod server etc.

2019-11-21 Thread Mark Wielaard
ybe). > > So I think the real issue is the splitting of -R and -F argument > > parsing. If that is the case, maybe just picking a default for how to > > interpret the extra arguments, as dirs for the file scanner or dirs for > > the rpm scanner or both, might make us both happy? > > The branch code does "both", because it is simple. So if we make things simple again, but you do have to choose whether the arguments/dirs are for the rpms or files scanner, would that make you happy? Thanks, Mark

Re: patch 2/2 debuginfod server etc.

2019-11-22 Thread Mark Wielaard
Hi, On Thu, 2019-11-21 at 21:41 +0100, Mark Wielaard wrote: > I think what makes the discussion somewhat difficult is that there are > basically three cases: > > - Serving trees of rpms where only the contents of the rpms is shared. > - Serving of a build directory where it makes

Slow valgrind curl hostname lookups (Was: Buildbot failure in Wildebeest Builder on whole buildset)

2019-11-23 Thread Mark Wielaard
utes. I have committed the patch below and will monitor whether all buildbot workers will now succeed. Cheers, Mark From 1e651ad457f445ff933b4fdd7e2c82bd1fc0f2cd Mon Sep 17 00:00:00 2001 From: Mark Wielaard Date: Sat, 23 Nov 2019 14:09:42 +0100 Subject: [PATCH] tests: Use 127.0.0.1 in run-debugin

Add some metrics to debuginfod and use them to make test sleep less

2019-11-24 Thread Mark Wielaard
? It could just fail instead of timeing out. Then when we change the metrics we could just update the test simultaniously? Cheers, Mark

[PATCH 1/2] debuginfod: Add found_{executable,debuginfo,sourcerefs}_total metrics.

2019-11-24 Thread Mark Wielaard
Keeps metrics of how many executables, debuginfo and sourcerefs were found in total for file and rpm scanners. Signed-off-by: Mark Wielaard --- debuginfod/ChangeLog | 8 debuginfod/debuginfod.cxx | 24 2 files changed, 32 insertions(+) diff --git a

[PATCH 2/2] tests: Don't sleep in run-debuginfod-find.sh, but wait till ready.

2019-11-24 Thread Mark Wielaard
Wait for the debuginfod server to finish and use the metrics to see when a server is ready for the next test instead of sleeping. Also remove DEBUGINFOD_TEST_WEBAPI_SLEEP. Signed-off-by: Mark Wielaard --- debuginfod/ChangeLog | 6 +++ debuginfod/debuginfod.cxx| 8 --- tests

Re: Add some metrics to debuginfod and use them to make test sleep less

2019-11-25 Thread Mark Wielaard
Hi, On Sun, Nov 24, 2019 at 08:58:13PM +0100, Mark Wielaard wrote: > The following two patches add some metrics to debuginfod that can be > used ito see whether the scanners are still finding new executables, > debuginfo files or find new sources. They can also be used in the > ru

[COMMITTED] elfutils.spec: Add BuildRequires: curl

2019-11-25 Thread Mark Wielaard
As the buildbot just pointed out... The run-debuginfod-find.sh now relies on curl to fetch the metrics. Signed-off-by: Mark Wielaard --- config/ChangeLog| 4 config/elfutils.spec.in | 1 + 2 files changed, 5 insertions(+) diff --git a/config/ChangeLog b/config/ChangeLog index

[htdocs] [COMMITTED] update-coverage.sh: Add debuginfod to lcov coverage.

2019-11-26 Thread Mark Wielaard
Signed-off-by: Mark Wielaard --- update-coverage.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/update-coverage.sh b/update-coverage.sh index bff5461d..46f30ab2 100755 --- a/update-coverage.sh +++ b/update-coverage.sh @@ -30,7 +30,8 @@ ${tempdir}/elfutils/configure

[COMMITTED] Fix BUILD_STATIC build and enable gcov for debuginfod.

2019-11-26 Thread Mark Wielaard
libdw now always needs -lpthread and -ld even when BUILD_STATIC. BUILD_STATIC is only used when doing a gcov enabled build. Enable gcov coverage also for debuginfod. Signed-off-by: Mark Wielaard --- ChangeLog | 4 configure.ac | 1 + debuginfod/ChangeLog | 5

[COMMITTED] Prepare for 0.178

2019-11-26 Thread Mark Wielaard
Set version to 0.178. Update NEWS and elfutils.spec.in. Regenerate po/*.po files. Signed-off-by: Mark Wielaard --- ChangeLog | 5 + NEWS| 22 + config/elfutils.spec.in | 15 + configure.ac| 2 +- po/ChangeLog| 4 + po/de.po

[COMMITTED] elfutils.spec.in: Sync with fedora spec, remove rhel/fedora specifics.

2019-11-26 Thread Mark Wielaard
, add post/postun ldconfig for elfutils-libs and elfutils debuginfod-client subpackages, remove default defattr(-,root,root) for file lists and order binaries by name. Signed-off-by: Mark Wielaard --- config/ChangeLog| 9 +++ config/elfutils.spec.in | 145

elfutils 0.178 released

2019-11-26 Thread Mark Wielaard
be thread-safe. libdw: Don't free uninitialized Dwarf_Abbrev_Hash's of "fake" CUs. Add configure options for Valgrind annotations. libdw: Rewrite the memory handler to be more robust. Mark Wielaard (24): readelf: Add optional "SECTION" argument for --no

[COMMITTED] config: versioned libdebuginfod libraries should not be in -devel package.

2019-11-26 Thread Mark Wielaard
The versioned library (links) should be in the runtime package. The debuginfod-client-devel package should just contain the .so (symlink). This isn't fatal, just sloppy. ldconfig will correct the missing symlinks when it was missing. Signed-off-by: Mark Wielaard --- config/ChangeLog

[COMMITTED] config: Add an explicit Requires elfutils-debuginfod-client for debuginfod.

2019-11-28 Thread Mark Wielaard
elfutils-debuginfod depends indirectly already on the client library. Make that explicit so they are always updated in sync. Which isn't technically required, but makes things clearer and simpler on updates. Signed-off-by: Mark Wielaard --- config/ChangeLog| 5 + c

Re: Buildbot failure in Wildebeest Builder on whole buildset

2019-12-03 Thread Mark Wielaard
> Buildbot URL: https://builder.wildebeest.org/buildbot/ > > Worker for this Build: debian-i386 > > Build Reason: > Blamelist: Mark Wielaard > > BUILD FAILED: failed test (failure) Note that this was a build failure some (8) days ago. The ppc64 builder was offline and

[PATCH] debuginfod: Fix implicit conversion from 'CURLcode' to 'CURLMcode'

2019-12-03 Thread Mark Wielaard
, |^ libcurl has different error code enums. The "easy" interfaces return a CURLcode error. The "multi" interface functions return a CURLMcode. Signed-off-by: Mark Wielaard --- debuginfod/ChangeLog | 5 + debuginfod/debuginfod-client.c | 9 -

Re: RFCv2: debuginfod debian archive support

2019-12-05 Thread Mark Wielaard
How would one build a similar binary deb package (and the debuginfo subpackages) on a Debian/Ubuntu system? Thanks, Mark Summary: hello2 -- double hello, world rpm Name: hello2 Version: 1.0 Release: 2 Group: Utilities License: GPL Distribution: RPM ^W Elfutils test suite. Vendor: Red Hat Software Pa

[PATCH] tests: Run elfcompress under testrun in run-elfclassify.sh

2019-12-05 Thread Mark Wielaard
Otherwise elfcompress might run against the system libelf which might be too old or missing. Signed-off-by: Mark Wielaard --- tests/ChangeLog | 4 tests/run-elfclassify.sh | 4 ++-- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/tests/ChangeLog b/tests/ChangeLog

[PATCH] libdwfl: Find and handle compressed vmlinuz image.

2019-12-05 Thread Mark Wielaard
didn't up to now. Support for compressed ELF files was added in 2009 and the code was updated to to try to find the .gz, .bz2 or .xz extension variants in 2011. But not the vmlinuz named variant. Reported-by: Aaron Merey Tested-by: Frank Ch. Eigler Signed-off-by: Mark Wielaard --- li

[PATCH] config: Update elfutils.spec with explicit sub-package requires.

2019-12-05 Thread Mark Wielaard
on to match the current Fedora text. Signed-off-by: Mark Wielaard --- config/ChangeLog| 5 + config/elfutils.spec.in | 19 ++- 2 files changed, 15 insertions(+), 9 deletions(-) diff --git a/config/ChangeLog b/config/ChangeLog index d71fb39..4e28970 100644 --- a/c

[PATCH] tests: Add break to avoid implicit-fallthrough warning

2019-12-06 Thread Mark Wielaard
For some reason gcc might fail to recognize the assert (0) will never return and emit an implicit-fallthrough warning. Just add a break to silence it. Signed-off-by: Mark Wielaard --- tests/ChangeLog| 4 tests/backtrace-data.c | 1 + 2 files changed, 5 insertions(+) diff --git a

[PATCH] tests: Run strip under testrun in run-debuginfod-find.sh

2019-12-06 Thread Mark Wielaard
Otherwise strip might run against the system libelf which might be too old or missing. Signed-off-by: Mark Wielaard --- tests/ChangeLog | 4 tests/run-debuginfod-find.sh | 2 +- 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/tests/ChangeLog b/tests/ChangeLog

[PATCH] Make sure to always build with build-ids.

2019-12-06 Thread Mark Wielaard
We really need build-ids for various things. If the system compiler doesn't generate build-ids warn and generate them anyway for both the binaries and the tests. Signed-off-by: Mark Wielaard --- ChangeLog| 4 configure.ac | 9 + tests/Chan

Re: [PATCH] tests: Add break to avoid implicit-fallthrough warning

2019-12-06 Thread Mark Wielaard
On Fri, 2019-12-06 at 17:21 +0100, Florian Weimer wrote: > * Mark Wielaard: > > > For some reason gcc might fail to recognize the assert (0) will never > > return and emit an implicit-fallthrough warning. Just add a break to > > silence it. > > Is this with -D

Re: RFCv2: debuginfod debian archive support

2019-12-06 Thread Mark Wielaard
02 Frank Ch. Eigler > + > + * debuginfod.cxx (*_rpm_*): Rename to *_archive_* throughout. > + (scan_archives): New read-mostly global to identify archive > + file extensions and corresponding extractor commands. > + (parse_opt): Handle new -U flag. > + >

[PATCH] debuginfod: Check the DEBUGINFOD_URLS environment variable early in client.

2019-12-09 Thread Mark Wielaard
the library is used in a confined setup. Signed-off-by: Mark Wielaard --- debuginfod/ChangeLog | 5 + debuginfod/debuginfod-client.c | 20 +++- 2 files changed, 16 insertions(+), 9 deletions(-) diff --git a/debuginfod/ChangeLog b/debuginfod/ChangeLog index 10b6bfed..

Re: [PATCH] libdwfl: Find and handle compressed vmlinuz image.

2019-12-09 Thread Mark Wielaard
On Thu, 2019-12-05 at 15:21 +0100, Mark Wielaard wrote: > Both the dwfl_linux_kernel_find_elf callback and the > dwfl_linux_kernel_report_offline reporting function only handled > vmlinix images possibly compressed with .gz, .bz2 or .xz extension. > They did not find or handle the much

Re: [PATCH] debuginfod: Fix implicit conversion from 'CURLcode' to 'CURLMcode'

2019-12-09 Thread Mark Wielaard
On Wed, 2019-12-04 at 02:43 +0100, Mark Wielaard wrote: > GCC10 warns when converting the value of one enum type into another: > > debuginfod-client.c:530:24: error: implicit conversion from ‘CURLcode’ >to ‘CURLMcode’ [-Werror=enum-convers

Re: rfc/patch: debuginfod client $DEBUGINFOD_PROGRESS env var

2019-12-10 Thread Mark Wielaard
; + (debuginfod_query_server): Call progressfn -after- rather than > + before curl ops, to make it likely that a successful transfer > + results in final a=b call. Tweak cleanup sequence. > + * debuginfod.h: Document $DEBUGINFOD_PROGRESS name. > + > 2019-11-26 Ma

Re: [PATCH] tests: Run elfcompress under testrun in run-elfclassify.sh

2019-12-10 Thread Mark Wielaard
On Thu, Dec 05, 2019 at 02:23:56PM +0100, Mark Wielaard wrote: > Otherwise elfcompress might run against the system libelf which might > be too old or missing. I pushed this to master.

Re: [PATCH] config: Update elfutils.spec with explicit sub-package requires.

2019-12-10 Thread Mark Wielaard
On Thu, Dec 05, 2019 at 03:27:54PM +0100, Mark Wielaard wrote: > Consistently require version-release subpackages so that everything > is updated together. Technically this isn't always required (it is > for elfutils-libelf and elfutils-libs). But this makes things more > consis

Re: [PATCH] tests: Run strip under testrun in run-debuginfod-find.sh

2019-12-10 Thread Mark Wielaard
On Fri, Dec 06, 2019 at 03:49:14PM +0100, Mark Wielaard wrote: > Otherwise strip might run against the system libelf which might be too > old or missing. Pushed to master.

Re: [PATCH] Make sure to always build with build-ids.

2019-12-11 Thread Mark Wielaard
Hi, On Fri, Dec 06, 2019 at 05:22:41PM +0100, Mark Wielaard wrote: > We really need build-ids for various things. If the system compiler > doesn't generate build-ids warn and generate them anyway for both the > binaries and the tests. I pushed this to master with one change:

Re: RFCv2: debuginfod debian archive support

2019-12-11 Thread Mark Wielaard
ChangeLog > @@ -1,3 +1,8 @@ > +2019-12-06 Frank Ch. Eigler > + > + * elfutils.spec.in (debuginfod): Add BuildRequire dpkg > + for deb testing. (Available on Fedora & EPEL, not base RHEL.) > + > 2019-11-28 Mark Wielaard > > * elfutils.spec.in (debu

Re: RFCv2: debuginfod debian archive support

2019-12-11 Thread Mark Wielaard
t might make it simpler to... > > But it does feel like the errors, logs and metrics are a little > > generic (e.g. "cannot select all format"). > > The way in which specializing the format errors could help if > debuginfod were run against rpms that had a non-cpio payload, or debs > that had a non-tar payload. This means some sort of corruption, which > contravenes our "trustworthy data" assumption -- or upstream policy > change, which is nothing to worry about. > > If you think separate metrics for .deb vs .rpm archives might be > useful, can do. If it isn't too much work then I do think it would be useful/clearer if the logs/metrics reported debs and rpms separately. Thanks, Mark

Re: [PATCH] debuginfod: Check the DEBUGINFOD_URLS environment variable early in client.

2019-12-11 Thread Mark Wielaard
On Mon, 2019-12-09 at 19:49 +0100, Mark Wielaard wrote: > If the debuginfod-client isn't configured we should do as little > as possible. Simply return early with ENOSYS if no servers are > configured. This means we won't check > > This does change the behavior of

Re: [PATCH 0/2] elfutils: minor build system fixes

2019-12-12 Thread Mark Wielaard
er. Thanks. Do you still need -Wno-error? Cheers, Mark

Re: [PATCH 0/4] libdwfl: make dwfl_addrmodule work for Linux kernel modules

2019-12-18 Thread Mark Wielaard
t") does seem like a legitimate bug. It does indeed. How curious this code never worked as advertised. I agree that it is probably best to just document that IDENT should be NULL and is ignored. It comes with a testcase and removes various extra fields from Dwfl_Module. Applied to master. Thanks, Mark

Re: rfc/patch: debuginfod client $DEBUGINFOD_PROGRESS env var

2019-12-18 Thread Mark Wielaard
CALL_MULTI_PERFORM we > > continue/skip the progress function. > > If we have an error outcome, there is no more progress to report, so > that should be OK. Yeah, I guess if the first thing is an error, then no progress was ever really made, so we just never report it. It is just a tiny change in behavior. Although I doubt anybody would rely on the progress function being called at least once. Cheers, Mark

Re: rfc/patch: debuginfod client $DEBUGINFOD_PROGRESS env var

2019-12-18 Thread Mark Wielaard
l like it for the user). This might or might not be a separate timeout from the connect timeout, but I think they are kind of the same thing. Cheers, Mark

Re: RFCv2: debuginfod debian archive support

2019-12-22 Thread Mark Wielaard
atch added to fche/debuginfod-deb branch: I think this is ready to be squashed and put on master. I had hoped out Debian/Ubuntu friends would speak up. But they had their chance and will now have to live with the code as is :) Cheers, Mark

Re: rfc/patch: debuginfod client $DEBUGINFOD_PROGRESS env var

2019-12-22 Thread Mark Wielaard
because that seems twice the worse case time to decompress and that gives it about 45 seconds to provide ~10K/sec. But if you are seeing 60 seconds as worse case we could pick something like 120 seconds or something. But it should probably be a separate timeout from the connection timeout, and maybe from the total timeout (or maybe replace it?). What do you think? Cheers, Mark

<    11   12   13   14   15   16   17   18   19   20   >