Re: PR28514: [PATCH] debuginfod groom time-limit
Hi Frank, On Thu, 2021-11-04 at 13:21 -0400, Frank Ch. Eigler via Elfutils-devel wrote: > commit 8ce18d0f8a5d7c00dc897f99751d575c8deeae82 (HEAD -> master) > Author: Frank Ch. Eigler > Date: Thu Nov 4 13:08:35 2021 -0400 > > PR28514: debuginfod: limit groom operation times > > For large databases and many stale files, it was possible to starve > rescan operations by numerous groom "nuke" (database delete ops). > Under the theory that including new data is at least as important as > aging old, we now impose a rough deadline on groom queries. > > In the process, we discovered that we were commiting some > undefined-behaviour sqlite ops (deleting rows while iterating), which > may explain some previous heisenbug occurrences. So the groom nuke > operations are split into decision & action phases, with associated > progress-tracking metrics. > > Testing the timeout facility requires hand-testing beyond the > testsuite (since it requires LARGE databases to show measurable query > times). So confirmed this part by hand. Makes sense and looks good. Please push. Thanks, Mark
Re: Buildbot failure in Wildebeest Builder on whole buildset
Hi, On Thu, 2021-11-04 at 20:14 +, build...@builder.wildebeest.org wrote: > The Buildbot has detected a new failure on builder elfutils-centos- > x86_64 while building elfutils. > Full details are available at: > https://builder.wildebeest.org/buildbot/#builders/1/builds/851 > > Buildbot URL: https://builder.wildebeest.org/buildbot/ > > Worker for this Build: centos-x86_64 This was sadly a spurious test. In some cases on this older centos7 setup the biarch (32bit) multi-threaded backtrace from a core file is missing the "main" frame. But it is somehow timeing related. Maybe the core file itself is bad? It didn't repeat again and the builder is green again. > Build Reason: > Blamelist: Frank Ch. Eigler > > BUILD FAILED: failed test (failure) > > Sincerely, > -The BuildbotThe Buildbot has detected a new failure on builder > elfutils-fedora-x86_64 while building elfutils. > Full details are available at: > https://builder.wildebeest.org/buildbot/#builders/3/builds/844 > > Buildbot URL: https://builder.wildebeest.org/buildbot/ > > Worker for this Build: fedora-x86_64 This was funny. The fedora-x86_64 builder does a make distcheck. The distcheck noticed the top of NEWS (first 15 lines) didn't contain the version number anymore and assumed we forgot to update NEWS for the release. But it actually was all the new NEWS after the last release that made the last version number go "out of scope". Frank fixed it by adding the version number at the top of the NEWS file again. Cheers, Mark
Re: [PATCH] (v2) read inlining info in an NVIDIA extended line map
Hi, On Thu, 2021-11-04 at 16:41 -0500, John Mellor-Crummey via Elfutils- devel wrote: > [We would really like this patch in the forthcoming release] > > Attached is a new version of the patch for reading inlining > information encoded in an enhanced line map format used in NVIDIA GPU > binaries for CUDA 11.2+. It looks like the attachment is missing. Or the mailinglist removed it for some reason, but I also didn't see it here: https://sourceware.org/pipermail/elfutils-devel/2021q4/004307.html Could you resent it? Thanks, Mark > This is an updated version of a patch first submitted on Sept. 5. A > copy of the original submission email is quoted below this note. > > Here I describe just the improvements to that patch that address > Mark’s concerns: > > (1) all of the code for handling NVIDIA DWARF extensions is always > available; there is no special configuration switch needed. > (2) all changes are bracketed by comments that mark them NVIDIA > extensions > (3) the DWARF extended opcodes have been renamed with names that > include NVIDIA in them > (4) the two new API functions to surface the new information have > been improved to separate the interface result from the internal > representation (at Mark’s request) > (4a) the API for extracting the name of an inlined function in > a DWARF line now returns a const char * instead of a string table > index > (4b) the API for extracting an inline “context” now returns a > pointer to a DWARF line where the code is inlined rather than > returning an unsigned int (an index into the line table that one > could use to compute the pointer) > (5) there are test cases for readelf and libdw that use a binary > generated by NVIDIA’s compiler. the test cases include information > about how the binary was generated
[Bug debuginfod/28514] limit grooming time for many stale files
https://sourceware.org/bugzilla/show_bug.cgi?id=28514 Frank Ch. Eigler changed: What|Removed |Added Status|NEW |RESOLVED Resolution|--- |FIXED --- Comment #1 from Frank Ch. Eigler --- commit c1e8c8c6b25cb2b5c16553609f19a9ed5dd4e146 Author: Frank Ch. Eigler Date: Thu Nov 4 13:08:35 2021 -0400 PR28514: debuginfod: limit groom operation times -- You are receiving this mail because: You are on the CC list for the bug.
[PATCH] version 2 read extended nvidia linemap
the attachment was on the original message. i am resending the attachment. 0001-Read-inlining-info-in-NVIDIA-extended-line-map.patch Description: Binary data (sent from my phone)
PATCH: PR28430: debuginfod --passive mode
Hi -
I speculate this one will be handy for even larger debuginfod
installations, where one must pool multiple servers together, but
without duplicating indexing/grooming efforts.
(The diff is larger than it needs to be, by virtue of nesting previous
code into a conditional block.)
commit 8f2623a9ff8ac311c0ec829d982863a8dda4154a
Author: Frank Ch. Eigler
Date: Fri Nov 5 22:26:35 2021 -0400
PR28430: debuginfod: support --passive mode
Add support for a limited mode for debuginfod that uses a pure
read-only sqlite index. This mode is useful for load spreading based
on naively shared or replicated databases.
Signed-off-by: Frank Ch. Eigler
diff --git a/NEWS b/NEWS
index b812b7438967..709c4cd9e7f5 100644
--- a/NEWS
+++ b/NEWS
@@ -14,6 +14,7 @@ debuginfod: Supply extra HTTP response headers, describing
archive/file
Add -r option to use -I/-X regexes for grooming stale files.
Protect against wasted CPU from duplicate concurrent requests.
Limit the duration of groom ops roughly to rescan (-t) times.
+Add --passive mode for serving from read-only database.
Several other performance improvements & prometheus metrics.
Version 0.185
diff --git a/debuginfod/ChangeLog b/debuginfod/ChangeLog
index 15b2ba40fa0f..f06d3ee3ecf4 100644
--- a/debuginfod/ChangeLog
+++ b/debuginfod/ChangeLog
@@ -1,3 +1,11 @@
+2021-11-05 Frank Ch. Eigler
+
+ PR28430
+ * debuginfod.cxx (parse_opt): Add "--passive" flag. Complain
+ about inconsistent flags.
+ (main): In passive mode, suppress scan/groom/traverse threads and
+ other read-write database ops.
+
2021-11-04 Frank Ch. Eigler
PR28514
diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index 45981d8d4279..28217e6d92d4 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -376,6 +376,8 @@ static const struct argp_option options[] =
prefetch cache.", 0},
#define ARGP_KEY_FORWARDED_TTL_LIMIT 0x1007
{"forwarded-ttl-limit", ARGP_KEY_FORWARDED_TTL_LIMIT, "NUM", 0, "Limit of
X-Forwarded-For hops, default 8.", 0},
+#define ARGP_KEY_PASSIVE 0x1008
+ { "passive", ARGP_KEY_PASSIVE, NULL, 0, "Do not scan or groom, read-only
database.", 0 },
{ NULL, 0, NULL, 0, NULL, 0 },
};
@@ -425,6 +427,8 @@ static long fdcache_prefetch_mbs;
static long fdcache_prefetch_fds;
static unsigned forwarded_ttl_limit = 8;
static string tmpdir;
+static bool passive_p = false;
+
static void set_metric(const string& key, double value);
// static void inc_metric(const string& key);
@@ -524,36 +528,56 @@ parse_opt (int key, char *arg,
}
break;
case 'L':
+ if (passive_p)
+argp_failure(state, 1, EINVAL, "-L option inconsistent with passive
mode");
traverse_logical = true;
break;
-case 'D': extra_ddl.push_back(string(arg)); break;
+case 'D':
+ if (passive_p)
+argp_failure(state, 1, EINVAL, "-D option inconsistent with passive
mode");
+ extra_ddl.push_back(string(arg));
+ break;
case 't':
+ if (passive_p)
+argp_failure(state, 1, EINVAL, "-t option inconsistent with passive
mode");
rescan_s = (unsigned) atoi(arg);
break;
case 'g':
+ if (passive_p)
+argp_failure(state, 1, EINVAL, "-g option inconsistent with passive
mode");
groom_s = (unsigned) atoi(arg);
break;
case 'G':
+ if (passive_p)
+argp_failure(state, 1, EINVAL, "-G option inconsistent with passive
mode");
maxigroom = true;
break;
case 'c':
+ if (passive_p)
+argp_failure(state, 1, EINVAL, "-c option inconsistent with passive
mode");
concurrency = (unsigned) atoi(arg);
if (concurrency < 1) concurrency = 1;
break;
case 'I':
// NB: no problem with unconditional free here - an earlier failed
regcomp would exit program
+ if (passive_p)
+argp_failure(state, 1, EINVAL, "-I option inconsistent with passive
mode");
regfree (&file_include_regex);
rc = regcomp (&file_include_regex, arg, REG_EXTENDED|REG_NOSUB);
if (rc != 0)
argp_failure(state, 1, EINVAL, "regular expression");
break;
case 'X':
+ if (passive_p)
+argp_failure(state, 1, EINVAL, "-X option inconsistent with passive
mode");
regfree (&file_exclude_regex);
rc = regcomp (&file_exclude_regex, arg, REG_EXTENDED|REG_NOSUB);
if (rc != 0)
argp_failure(state, 1, EINVAL, "regular expression");
break;
case 'r':
+ if (passive_p)
+argp_failure(state, 1, EINVAL, "-r option inconsistent with passive
mode");
regex_groom = true;
break;
case ARGP_KEY_FDCACHE_FDS:
@@ -586,6 +610,15 @@ parse_opt (int key, char *arg,
if ( fdcache_prefetch_mbs < 0)
argp_failure(state, 1, EINVAL, "fdcache prefetch mbs");
break;
+case ARGP_KEY_PAS
