[Bug debuginfod/29926] debuginfod using deprecated curl (since 7.55.0) API, fails to build with 7.87.0
https://sourceware.org/bugzilla/show_bug.cgi?id=29926 Ross Burton changed: What|Removed |Added CC||ross at burtonini dot com --- Comment #3 from Ross Burton --- I can't reopen this but I'm still seeing errors: elfutils-0.188/debuginfod/debuginfod-client.c:1330:7: error: ‘CURLOPT_PROTOCOLS’ is deprecated: since 7.85.0. Use CURLOPT_PROTOCOLS_STR [-Werror=deprecated-declarations] 1330 | curl_easy_setopt_ck(data[i].handle, CURLOPT_PROTOCOLS, | ^~~ CURLOPT_PROTOCOLS was deprecated in 7.85.0 and the replacement CURLOPT_PROTOCOLS_STR introduced in 7.85.0. May I suggest building without fatal deprecation warnings by default? -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/29975] New: Search concurrency doesn't respect CPU affinity
https://sourceware.org/bugzilla/show_bug.cgi?id=29975 Bug ID: 29975 Summary: Search concurrency doesn't respect CPU affinity Product: elfutils Version: unspecified Status: UNCONFIRMED Severity: normal Priority: P2 Component: debuginfod Assignee: unassigned at sourceware dot org Reporter: ross at burtonini dot com CC: elfutils-devel at sourceware dot org Target Milestone: --- I've a machine with lots of cores (256) but debuginfod doesn't respect CPU affinity: $ taskset -c 2 debuginfod -v ... [Mon Jan 9 17:33:20 2023] (2351762/2351762): search concurrency 256 Instead of looking at the number of cores present, it should look at the number of cores available to it (sched_getaffinity()). -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/29976] New: webapi connection pool eats all file handles
https://sourceware.org/bugzilla/show_bug.cgi?id=29976
Bug ID: 29976
Summary: webapi connection pool eats all file handles
Product: elfutils
Version: unspecified
Status: UNCONFIRMED
Severity: normal
Priority: P2
Component: debuginfod
Assignee: unassigned at sourceware dot org
Reporter: ross at burtonini dot com
CC: elfutils-devel at sourceware dot org
Target Milestone: ---
If I start debuginfod without any concurrency limits:
[Mon Jan 9 17:40:14 2023] (2356243/2356243): libmicrohttpd error: Failed to
create worker inter-thread communication channel: Too many open files
My machine has 256 cores, and stracing debuginfod shows that it fails to open
more files after creating 510 epoll fds (twice):
epoll_create1(EPOLL_CLOEXEC)= 1021
epoll_ctl(1021, EPOLL_CTL_ADD, 3, {events=EPOLLIN, data={u32=4027013664,
u64=187651148175904}}) = 0
epoll_ctl(1021, EPOLL_CTL_ADD, 1020, {events=EPOLLIN, data={u32=2965961632,
u64=281473647704992}}) = 0
mmap(NULL, 8454144, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) =
0xfff6b97b
mprotect(0xfff6b97c, 8388608, PROT_READ|PROT_WRITE) = 0
rt_sigprocmask(SIG_BLOCK, ~[], [], 8) = 0
clone(child_stack=0xfff6b9fbea00,
flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID,
parent_tid=[2361982], tls=0xfff6b9fbf880, child_tidptr=0xfff6b9fbf210) =
2361982
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
eventfd2(0, EFD_CLOEXEC|EFD_NONBLOCK) = 1022
epoll_create1(EPOLL_CLOEXEC)= 1023
epoll_ctl(1023, EPOLL_CTL_ADD, 3, {events=EPOLLIN, data={u32=4027014456,
u64=187651148176696}}) = 0
epoll_ctl(1023, EPOLL_CTL_ADD, 1022, {events=EPOLLIN, data={u32=2965961632,
u64=281473647704992}}) = 0
mmap(NULL, 8454144, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) =
0xfff6b8fa
mprotect(0xfff6b8fb, 8388608, PROT_READ|PROT_WRITE) = 0
rt_sigprocmask(SIG_BLOCK, ~[], [], 8) = 0
clone(child_stack=0xfff6b97aea00,
flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID,
parent_tid=[2361983], tls=0xfff6b97af880, child_tidptr=0xfff6b97af210) =
2361983
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
eventfd2(0, EFD_CLOEXEC|EFD_NONBLOCK) = -1 EMFILE (Too many open files)
ulimit -n is 1024, do I really need more just to start debuginfod if I have 256
cores? As the web connections is 2xthreads and it appears to be using two fds
per connection, maybe I do.
Should the connection pool have a hard limit when using the default? I doubt
512 incoming connections would be usual, and if that is needed then the user
can specify -C.
--
You are receiving this mail because:
You are on the CC list for the bug.
[Bug debuginfod/29975] Search concurrency doesn't respect CPU affinity
https://sourceware.org/bugzilla/show_bug.cgi?id=29975 --- Comment #2 from Ross Burton --- No, but the C isn't that difficult: #define _GNU_SOURCE #include int ret, count; cpu_set_t mask; CPU_ZERO(&mask); ret = sched_getaffinity(0, sizeof(mask), &mask); // if non-zero, errno is set count = CPU_COUNT(&mask); -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/29976] webapi connection pool eats all file handles
https://sourceware.org/bugzilla/show_bug.cgi?id=29976 --- Comment #2 from Ross Burton --- Honestly, no idea. Appears to be the default on ubuntu. -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/29976] webapi connection pool eats all file handles
https://sourceware.org/bugzilla/show_bug.cgi?id=29976 --- Comment #3 from Ross Burton --- Yes, kernel defaults: 1024 soft, 4096 hard. I *can* change it to 4096 but there's still the point that: 1) debugging the failure case isn't trivial 2) cores*2 threads in the connection pool probably doesn't scale linearly -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/29976] webapi connection pool eats all file handles
https://sourceware.org/bugzilla/show_bug.cgi?id=29976 --- Comment #5 from Ross Burton --- Yes. My use case is a test that uses debuginfod, so it works everywhere and as it only has to service a few requests I'm just passing -C2 -c2. -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/29982] New: sqlite errors when searching
https://sourceware.org/bugzilla/show_bug.cgi?id=29982 Bug ID: 29982 Summary: sqlite errors when searching Product: elfutils Version: unspecified Status: UNCONFIRMED Severity: normal Priority: P2 Component: debuginfod Assignee: unassigned at sourceware dot org Reporter: ross at burtonini dot com CC: elfutils-devel at sourceware dot org Target Milestone: --- When debuginfod services as request, I often get a slew of errors: [Tue Jan 10 10:42:51 2023] (2548256/2548514): fts traversed source paths in 0.232152s, scanned=6011, regex-skipped=0 [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548742): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548543): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548565): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548612): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548742): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548565): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548543): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548612): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548742): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548543): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548565): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548742): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548612): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548517): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2548256/2548742): sqlite3 error: sqlite3 step: database table is locked [Tue Jan 10 10:43:06 2023] (2
[Bug debuginfod/29982] sqlite errors when searching
https://sourceware.org/bugzilla/show_bug.cgi?id=29982 --- Comment #1 from Ross Burton --- Obviously attaching a gdb and breaking on the error path magically fixes it for me... -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/29982] sqlite errors when searching
https://sourceware.org/bugzilla/show_bug.cgi?id=29982 --- Comment #3 from Ross Burton --- I can replicate with elfutils 0.188 and sqlite 3.40.1 on debian and ubuntu. Due to previously discussed concurrency issues my commandline is essentially -d:memory -c256 -C256. -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/29982] sqlite errors when searching
https://sourceware.org/bugzilla/show_bug.cgi?id=29982 --- Comment #4 from Ross Burton --- Just replicated with -c2 -C2. Potentially of interest is that the log shows two PIDs: [Tue Jan 10 11:32:20 2023] (2619858/2619864): sqlite3 error: sqlite3 step: database table is locked [repeated ~60 times] [Tue Jan 10 11:32:20 2023] (2619858/2619860): 192.168.7.2:54292 UA:elfutils/0.188,Linux/aarch64,/ XFF: GET /buildid/7fd4aae416bbcee8fd24cf5db0e805cd0c1e218e/debuginfo 200 3195072 0+122ms The errors come from a different thread to that which handled the request. -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/29982] sqlite errors when searching
https://sourceware.org/bugzilla/show_bug.cgi?id=29982 --- Comment #5 from Ross Burton --- Note that the errors are just logged, the lookup still works, so it's possible that your tests work but the logs are full of these messages. -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/29982] sqlite errors when searching
https://sourceware.org/bugzilla/show_bug.cgi?id=29982
--- Comment #7 from Ross Burton ---
cmd = [
os.path.join(native_sysroot, "usr", "bin", "debuginfod"),
"--verbose",
"--database=:memory:",
"--concurrency=2",
"--connection-pool=2",
get_bb_var("DEPLOY_DIR"),
]
--
You are receiving this mail because:
You are on the CC list for the bug.
[Bug debuginfod/29982] sqlite errors when searching
https://sourceware.org/bugzilla/show_bug.cgi?id=29982 --- Comment #8 from Ross Burton --- Replicated with -: [Tue Jan 10 11:43:35 2023] (2679838/2679844): searching for sources for cu=../sysdeps/aarch64/crtn.S comp_dir=/usr/src/debug/g libc/2.36-r0/csu #files=2 #dirs=2 [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/include/bits/types/FILE.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679844): /usr/src/debug/glibc/2.36-r0/csu/../sysdeps/aarch64/crtn.S new [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/include/c++/12.2.0/cstdlib dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/include/c++/12.2.0/bits/std_abs.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/include/sys/types.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/include/bits/types/time_t.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/include/stdlib.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/include/bits/stdlib-float.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/include/bits/stdlib-bsearch.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/include/bits/stdlib.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/include/c++/12.2.0/stdlib.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/src/debug/gcc/12.2.0-r0/include/hashtab.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/src/debug/gcc/12.2.0-r0/include/obstack.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/src/debug/gcc/12.2.0-r0/libcpp/include/symtab.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/src/debug/gcc/12.2.0-r0/libcpp/internal.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/src/debug/gcc/12.2.0-r0/libcpp/include/cpplib.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/include/iconv.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/src/debug/gcc/12.2.0-r0/libcpp/include/mkdeps.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/include/string.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/include/libintl.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/src/debug/gcc/12.2.0-r0/include/libiberty.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/src/debug/gcc/12.2.0-r0/libcpp/ dup [Tue Jan 10 11:43:35 2023] (2679838/2679845): /usr/include/c++/12.2.0/aarch64-poky-linux/bits/c++config.h dup [Tue Jan 10 11:43:35 2023] (2679838/2679844): rpm-buildid-intern bind 1=34c56faac74aa9ce0b7bb21f772b4aa35f0be2f8 [Tue Jan 10 11:43:35 2023] (2679838/2679844): rpm-buildid-intern step-ok-done(database table is locked) insert or ignore into buildids9_buildids VALUES (NULL, ?); Is the problem that it's still scanning the RPMs? -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/29982] sqlite errors when searching
https://sourceware.org/bugzilla/show_bug.cgi?id=29982 --- Comment #10 from Ross Burton --- Created attachment 14569 --> https://sourceware.org/bugzilla/attachment.cgi?id=14569&action=edit Log Attached is a log (6MB zstd compressed, 171M uncompressed...) with the step locked errors, using -v to get the sqlite operation logging. -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/29976] webapi connection pool eats all file handles
https://sourceware.org/bugzilla/show_bug.cgi?id=29976 --- Comment #7 from Ross Burton --- Looks good to me! -- You are receiving this mail because: You are on the CC list for the bug.
