[Bug debuginfod/29982] New: sqlite errors when searching
https://sourceware.org/bugzilla/show_bug.cgi?id=29982 Bug ID: 29982 Summary: sqlite errors when searching Product: elfutils Version: unspecified Status: UNCONFIRMED Severity: normal Priority: P2 Component: debuginfod Assignee: unassigned at sourceware dot org Reporter: ross at burtonini dot com CC: elfutils-devel at sourceware dot org Target Milestone: --- When debuginfod services as request, I often get a slew of errors: [Tue Jan 10 10:42:51 2023] (2548256/2548514): fts traversed source paths in 0.232152s, scanned=6011, regex-skipped=0 [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548742): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548543): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548565): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548612): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548742): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548565): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548543): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548612): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548742): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548543): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548565): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548742): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548612): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548742): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2
[Bug debuginfod/29982] sqlite errors when searching
https://sourceware.org/bugzilla/show_bug.cgi?id=29982 --- Comment #1 from Ross Burton --- Obviously attaching a gdb and breaking on the error path magically fixes it for me... -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/29982] sqlite errors when searching
https://sourceware.org/bugzilla/show_bug.cgi?id=29982 Frank Ch. Eigler changed: What|Removed |Added CC||fche at redhat dot com Ever confirmed|0 |1 Last reconfirmed||2023-01-10 Status|UNCONFIRMED |WAITING --- Comment #2 from Frank Ch. Eigler --- Please supply a command line invocation for us to try to reproduce this, along with version numbers of ubuntu / debuginfod / sqlite3. This could happen if the database is actually a normal file instead of :memory:, and two separate debuginfod instances are trying to write to the same one. (debuginfod -d :memory: is already tested on debian etc. as a part of the testsuite. -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/29982] sqlite errors when searching
https://sourceware.org/bugzilla/show_bug.cgi?id=29982 --- Comment #3 from Ross Burton --- I can replicate with elfutils 0.188 and sqlite 3.40.1 on debian and ubuntu. Due to previously discussed concurrency issues my commandline is essentially -d:memory -c256 -C256. -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/29982] sqlite errors when searching
https://sourceware.org/bugzilla/show_bug.cgi?id=29982 --- Comment #4 from Ross Burton --- Just replicated with -c2 -C2. Potentially of interest is that the log shows two PIDs: [Tue Jan 10 11:32:20 2023] (2619858/2619864): sqlite3 error: sqlite3 step: database table is locked [repeated ~60 times] [Tue Jan 10 11:32:20 2023] (2619858/2619860): 192.168.7.2:54292 UA:elfutils/0.188,Linux/aarch64,/ XFF: GET /buildid/7fd4aae416bbcee8fd24cf5db0e805cd0c1e218e/debuginfo 200 3195072 0+122ms The errors come from a different thread to that which handled the request. -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/29982] sqlite errors when searching
https://sourceware.org/bugzilla/show_bug.cgi?id=29982 --- Comment #5 from Ross Burton --- Note that the errors are just logged, the lookup still works, so it's possible that your tests work but the logs are full of these messages. -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/29982] sqlite errors when searching
https://sourceware.org/bugzilla/show_bug.cgi?id=29982 --- Comment #6 from Frank Ch. Eigler --- Please confirm the complete command line. % debuginfod -d:memory creates a file named ":memory", not an in-memory sqlite database alias ":memory:") -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/29982] sqlite errors when searching
https://sourceware.org/bugzilla/show_bug.cgi?id=29982
--- Comment #7 from Ross Burton ---
cmd = [
os.path.join(native_sysroot, "usr", "bin", "debuginfod"),
"--verbose",
"--database=:memory:",
"--concurrency=2",
"--connection-pool=2",
get_bb_var("DEPLOY_DIR"),
]
--
You are receiving this mail because:
You are on the CC list for the bug.
[Bug debuginfod/29982] sqlite errors when searching
https://sourceware.org/bugzilla/show_bug.cgi?id=29982 --- Comment #8 from Ross Burton --- Replicated with -: [Tue Jan 10 11:43:35 2023] (2679838/2679844): searching for sources for cu=../sysdeps/aarch64/crtn.S comp_dir=/usr/src/debug/g libc/2.36-r0/csu #files=2 #dirs=2 [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/include/bits/types/FILE.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679844): /usr/src/debug/glibc/2.36-r0/csu/../sysdeps/aarch64/crtn.S new [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/include/c++/12.2.0/cstdlib dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/include/c++/12.2.0/bits/std_abs.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/include/sys/types.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/include/bits/types/time_t.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/include/stdlib.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/include/bits/stdlib-float.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/include/bits/stdlib-bsearch.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/include/bits/stdlib.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/include/c++/12.2.0/stdlib.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/src/debug/gcc/12.2.0-r0/include/hashtab.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/src/debug/gcc/12.2.0-r0/include/obstack.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/src/debug/gcc/12.2.0-r0/libcpp/include/symtab.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/src/debug/gcc/12.2.0-r0/libcpp/internal.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/src/debug/gcc/12.2.0-r0/libcpp/include/cpplib.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/include/iconv.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/src/debug/gcc/12.2.0-r0/libcpp/include/mkdeps.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/include/string.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/include/libintl.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/src/debug/gcc/12.2.0-r0/include/libiberty.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/src/debug/gcc/12.2.0-r0/libcpp/ dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/include/c++/12.2.0/aarch64-poky-linux/bits/c++config.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679844): rpm-buildid-intern bind 1=34c56faac74aa9ce0b7bb21f772b4aa35f0be2f8 [Tue Jan 10 11:43:35 2023] (2679838/2679844): rpm-buildid-intern step-ok-done(database table is locked) insert or ignore into buildids9_buildids VALUES (NULL, ?); Is the problem that it's still scanning the RPMs? -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/29982] sqlite errors when searching
https://sourceware.org/bugzilla/show_bug.cgi?id=29982 --- Comment #9 from Frank Ch. Eigler --- Perhaps a complete highly verbose trace (-vvv) would help identify the cause. Scanning threads should not be able to lock each other out for O(1sec) from sqlite operations in normal circumstances. OTOH, one can rest assured that in case of such sqlite problems, debuginfod just treats them as transient and retries the related scan operation later. Webapi service threads do not contend with the others at all, so clients don't see these transient errors. -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/29982] sqlite errors when searching
https://sourceware.org/bugzilla/show_bug.cgi?id=29982 --- Comment #10 from Ross Burton --- Created attachment 14569 --> https://sourceware.org/bugzilla/attachment.cgi?id=14569&action=edit Log Attached is a log (6MB zstd compressed, 171M uncompressed...) with the step locked errors, using -v to get the sqlite operation logging. -- You are receiving this mail because: You are on the CC list for the bug.
[PATCH] debuginfod-find.1: note on how to find a comp_dir
doc/ChangeLog: * debuginfod-find.1: add a note to DESCRIPTION section for the 'source' subcommand, clarifying where to find the CU compilation-directory. I'm looking at how to improve and document the workflow for using debuginfod-find to study the behaviour of packaged software on a system, e.g. in conjunction with a tracing tool like SystemTap. You can read the exact compiled source code with debuginfod-find source and use that to figure out which source code locations are interesting to trace. But that process has several non-obvious or inconvenient steps. This patch is a clarification to the debuginfod-find source man page pointing out the eu-readelf command that can show the comp_dir in downloaded debuginfo. Since debuginfod-find source could be picked up and used by a developer unfamiliar with DWARF terminology, I think such a clarification saves time for newbies figuring out what a CU compilation-directory is. Signed-off-by: Serhei Makarov --- doc/debuginfod-find.1 | 13 + 1 file changed, 13 insertions(+) diff --git a/doc/debuginfod-find.1 b/doc/debuginfod-find.1 index 957ec7e7..7d577bab 100644 --- a/doc/debuginfod-find.1 +++ b/doc/debuginfod-find.1 @@ -92,6 +92,19 @@ is made up of multiple CUs. Therefore, to disambiguate, debuginfod expects source queries to prefix relative path names with the CU compilation-directory, followed by a mandatory "/". +Note: for software packaged by distributions, the CU +compilation-directory may not be obvious. It can be found by +inspecting AT_comp_dir values in downloaded debuginfo. For example, +the comp_dir of the Fedora 37 version of /bin/ls can be found as +follows: + +.SAMPLE +% debuginfod-find debuginfo /bin/ls +~/.cache/debuginfod_client/03529d48345409576cd5c82a56ad08555088d353/ +% eu-readelf -w ~/.cache/debuginfod_client/03529d48345409576cd5c82a56ad08555088d353/debuginfo | grep comp_dir + comp_dir (line_strp) "/usr/src/debug/coreutils-9.1-6.fc37.x86_64/separate" +.ESAMPLE + Note: the caller may or may not elide \fB../\fP or \fB/./\fP or extraneous \fB///\fP sorts of path components in the directory names. debuginfod accepts both forms. Specifically, debuginfod canonicalizes path names -- 2.38.1
[PATCH] readelf: Check compression status of .debug section data
The various print_debug_*_section functions didn't get the section
data in the same way. Add a new get_debug_elf_data function that
gets the (possibly relocated) section data and that checks (and
warns) if the data might still be compressed in a way that we
cannot decompress.
Signed-off-by: Mark Wielaard
---
src/ChangeLog | 19 +
src/readelf.c | 213 ++
2 files changed, 130 insertions(+), 102 deletions(-)
diff --git a/src/ChangeLog b/src/ChangeLog
index db8a81ee..0490088e 100644
--- a/src/ChangeLog
+++ b/src/ChangeLog
@@ -1,3 +1,22 @@
+2023-01-10 Mark Wielaard
+
+ * readelf.c (get_debug_elf_data): New function.
+ (print_debug_abbrev_section): Use get_debug_elf_data.
+ (print_debug_addr_section): Likewise.
+ (print_debug_aranges_section): Likewise.
+ (print_debug_rnglists_section): Likewise.
+ (print_debug_ranges_section): Likewise.
+ (print_debug_frame_section): Likewise.
+ (print_debug_units): Likewise.
+ (print_debug_line_section): Likewise.
+ (print_debug_loclists_section): Likewise.
+ (print_debug_loc_section): Likewise.
+ (print_debug_macinfo_section): Likewise.
+ (print_debug_macro_section): Likewise.
+ (print_debug_str_section): Likewise.
+ (print_debug_str_offsets_section): Likewise.
+ (print_debug_pubnames_section): Likewise.
+
2022-12-21 Shahab Vahedi
* elflint.c (valid_e_machine): Add EM_ARCV2.
diff --git a/src/readelf.c b/src/readelf.c
index 451f8400..51b0e8b9 100644
--- a/src/readelf.c
+++ b/src/readelf.c
@@ -3857,6 +3857,39 @@ print_attributes (Ebl *ebl, const GElf_Ehdr *ehdr)
}
}
+/* Returns either the (relocated) data from the Dwarf, or tries to get
+ the "raw" (uncompressed) data from the Elf section. Produces a
+ warning if the data cannot be found (or decompressed). */
+static Elf_Data *
+get_debug_elf_data (Dwarf *dbg, Ebl *ebl, int idx, Elf_Scn *scn)
+{
+ /* We prefer to get the section data from the Dwarf because that
+ might have been relocated already. Note this is subtly wrong if
+ there are multiple sections with the same .debug name. */
+ if (dbg->sectiondata[idx] != NULL)
+return dbg->sectiondata[idx];
+
+ GElf_Shdr shdr_mem;
+ GElf_Shdr *shdr = gelf_getshdr (scn, &shdr_mem);
+ if (shdr != NULL && (shdr->sh_flags & SHF_COMPRESSED) != 0)
+{
+ if (elf_compress (scn, 0, 0) < 0)
+ {
+ error (0, 0, "%s [%zd] '%s'\n",
+_("Couldn't uncompress section"),
+elf_ndxscn (scn), section_name (ebl, shdr));
+ return NULL;
+ }
+}
+
+ Elf_Data *data = elf_getdata (scn, NULL);
+ if (data == NULL)
+error (0, 0, "%s [%zd] '%s': %s\n",
+ _("Couldn't get data from section"),
+ elf_ndxscn (scn), section_name (ebl, shdr), elf_errmsg (-1));
+
+ return elf_getdata (scn, NULL);
+}
void
print_dwarf_addr (Dwfl_Module *dwflmod,
@@ -5271,8 +5304,11 @@ print_debug_abbrev_section (Dwfl_Module *dwflmod
__attribute__ ((unused)),
Ebl *ebl, GElf_Ehdr *ehdr __attribute__ ((unused)),
Elf_Scn *scn, GElf_Shdr *shdr, Dwarf *dbg)
{
- const size_t sh_size = (dbg->sectiondata[IDX_debug_abbrev] ?
- dbg->sectiondata[IDX_debug_abbrev]->d_size : 0);
+ Elf_Data *elf_data = get_debug_elf_data (dbg, ebl, IDX_debug_abbrev, scn);
+ if (elf_data == NULL)
+return;
+
+ const size_t sh_size = elf_data->d_size;
printf (_("\nDWARF section [%2zu] '%s' at offset %#" PRIx64 ":\n"
" [ Code]\n"),
@@ -5344,6 +5380,10 @@ print_debug_addr_section (Dwfl_Module *dwflmod
__attribute__ ((unused)),
Ebl *ebl, GElf_Ehdr *ehdr,
Elf_Scn *scn, GElf_Shdr *shdr, Dwarf *dbg)
{
+ Elf_Data *data = get_debug_elf_data (dbg, ebl, IDX_debug_addr, scn);
+ if (data == NULL)
+return;
+
printf (_("\
\nDWARF section [%2zu] '%s' at offset %#" PRIx64 ":\n"),
elf_ndxscn (scn), section_name (ebl, shdr),
@@ -5352,16 +5392,6 @@ print_debug_addr_section (Dwfl_Module *dwflmod
__attribute__ ((unused)),
if (shdr->sh_size == 0)
return;
- /* We like to get the section from libdw to make sure they are relocated. */
- Elf_Data *data = (dbg->sectiondata[IDX_debug_addr]
- ?: elf_rawdata (scn, NULL));
- if (unlikely (data == NULL))
-{
- error (0, 0, _("cannot get .debug_addr section data: %s"),
-elf_errmsg (-1));
- return;
-}
-
size_t idx = 0;
sort_listptr (&known_addrbases, "addr_base");
@@ -5643,15 +5673,9 @@ print_debug_aranges_section (Dwfl_Module *dwflmod
__attribute__ ((unused)),
return;
}
- Elf_Data *data = (dbg->sectiondata[IDX_debug_aranges]
- ?: elf_rawdata (scn, NULL));
-
- if (unlikely (data == NULL))
-{
- error (0, 0, _("cannot get .debug_aranges content: %s"),
-elf_
[Bug debuginfod/29926] debuginfod using deprecated curl (since 7.55.0) API, fails to build with 7.87.0
https://sourceware.org/bugzilla/show_bug.cgi?id=29926 Mark Wielaard changed: What|Removed |Added Resolution|FIXED |--- CC||mark at klomp dot org Last reconfirmed||2023-01-10 Status|RESOLVED|REOPENED Ever confirmed|0 |1 --- Comment #4 from Mark Wielaard --- Does the following work for you? diff --git a/debuginfod/debuginfod-client.c b/debuginfod/debuginfod-client.c index a16165bd..1ce45632 100644 --- a/debuginfod/debuginfod-client.c +++ b/debuginfod/debuginfod-client.c @@ -1336,8 +1336,13 @@ debuginfod_query_server (debuginfod_client *c, /* Only allow http:// + https:// + file:// so we aren't being redirected to some unsupported protocol. */ +#if CURL_AT_LEAST_VERSION(7, 85, 0) + curl_easy_setopt_ck(data[i].handle, CURLOPT_PROTOCOLS_STR, + "http,https,file"); +#else curl_easy_setopt_ck(data[i].handle, CURLOPT_PROTOCOLS, (CURLPROTO_HTTP | CURLPROTO_HTTPS | CURLPROTO_FILE)); +#endif curl_easy_setopt_ck(data[i].handle, CURLOPT_URL, data[i].url); if (vfd >= 0) curl_easy_setopt_ck(data[i].handle, CURLOPT_ERRORBUFFER, https://code.wildebeest.org/git/user/mjw/elfutils/commit/?h=protocols_str -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/29976] webapi connection pool eats all file handles
https://sourceware.org/bugzilla/show_bug.cgi?id=29976 --- Comment #6 from Frank Ch. Eigler --- please check out commit 7399e3bd7eb72d045 on elfutils.git for a test patch -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/29975] Search concurrency doesn't respect CPU affinity
https://sourceware.org/bugzilla/show_bug.cgi?id=29975 --- Comment #3 from Frank Ch. Eigler --- please check out commit 7399e3bd7eb72d045 on elfutils.git for a test patch -- You are receiving this mail because: You are on the CC list for the bug.
