[PATCH] PR28708 debuginfod testsuite

2022-04-03 Thread Frank Ch. Eigler via Elfutils-devel
Hi -

Planning to commit this shortly.


commit e646e363e72e06e0ed5574c929236d815ddcbbaf (HEAD -> master)
Author: Frank Ch. Eigler 
Date:   Sun Apr 3 12:47:17 2022 -0400

PR28708: debuginfod: use MHD_USE_EPOLL for microhttpd threads

Testing on s390x and other architectures indicates that this
configuration reduces thundering-herd wakeups and saturation of a
small number of threads.  The run-debuginfod-webapi-concurrency.sh
test appears solid now.

Signed-off-by: Frank Ch. Eigler 

diff --git a/debuginfod/ChangeLog b/debuginfod/ChangeLog
index dfb5d42ec8a5..38a389e7dfd3 100644
--- a/debuginfod/ChangeLog
+++ b/debuginfod/ChangeLog
@@ -1,3 +1,8 @@
+2022-04-03  Frank Ch. Eigler 
+
+   * debuginfod.cxx (main): Use MHD_USE_EPOLL for libmicrohttpd, to
+   encourage more round-robin dispatch of incoming connections.
+
 2021-12-09  Alexander Kanavin 
 
* debuginfod-client.c (cache_clean_default_interval_s): Change type to
diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index bb8e8e102896..99924d36f260 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -3880,6 +3880,9 @@ main (int argc, char *argv[])
  | MHD_USE_INTERNAL_POLLING_THREAD
 #else
  | MHD_USE_SELECT_INTERNALLY
+#endif
+#ifdef MHD_USE_EPOLL
+ | MHD_USE_EPOLL
 #endif
  | MHD_USE_DEBUG, /* report errors to 
stderr */
  http_port,
@@ -3894,6 +3897,9 @@ main (int argc, char *argv[])
  | MHD_USE_INTERNAL_POLLING_THREAD
 #else
  | MHD_USE_SELECT_INTERNALLY
+#endif
+#ifdef MHD_USE_EPOLL
+ | MHD_USE_EPOLL
 #endif
  | MHD_USE_IPv6
  | MHD_USE_DEBUG, /* report errors to 
stderr */



Re: [PATCH] PR28708 debuginfod testsuite

2022-04-03 Thread Mark Wielaard
Hi Frank,

On Sun, Apr 03, 2022 at 01:01:14PM -0400, Frank Ch. Eigler via Elfutils-devel 
wrote:
> Planning to commit this shortly.

Yes, please!

> commit e646e363e72e06e0ed5574c929236d815ddcbbaf (HEAD -> master)
> Author: Frank Ch. Eigler 
> Date:   Sun Apr 3 12:47:17 2022 -0400
> 
> PR28708: debuginfod: use MHD_USE_EPOLL for microhttpd threads
> 
> Testing on s390x and other architectures indicates that this
> configuration reduces thundering-herd wakeups and saturation of a
> small number of threads.  The run-debuginfod-webapi-concurrency.sh
> test appears solid now.

Thanks for digging into this and finding a solution. Nice that it is
just adding that flag. It was driving me nuts that I didn't understand
where the failures really came from. I assume otherwise libmicrohttpd
uses select and it runs into trouble because that can only support up
to 1024 connections simultaniously.

Cheers,

Mark


patch: debuginfod ipv4-ipv6 dual-stack

2022-04-03 Thread Frank Ch. Eigler via Elfutils-devel
Hi -

This little improvement reduces wasted memory from duplicated
worker threads for ipv4 vs ipv6 stacks.


commit 3a00f412b6554ba70f7b7e3d3aa6f67f278868af (HEAD -> master)
Author: Frank Ch. Eigler 
Date:   Sun Apr 3 19:42:48 2022 -0400

debuginfod: use single ipv4+ipv6 microhttpd daemon configuration

Use a single MHD_USE_DUAL_STACK mhd daemon.  This way, the thread
connection pool is not doubled, saving memory and better matching user
expectations.  A slight tweak to logging is required to pull IPv4
remote addresses back out, and also to allow IPv6 ::-laden address
forwarding through federation links.

Signed-off-by: Frank Ch. Eigler 

diff --git a/debuginfod/ChangeLog b/debuginfod/ChangeLog
index 38a389e7dfd3..578d951d0102 100644
--- a/debuginfod/ChangeLog
+++ b/debuginfod/ChangeLog
@@ -1,3 +1,12 @@
+2022-04-03  Frank Ch. Eigler 
+
+   * debuginfod.cxx (main): Use single dual-stack daemon setup,
+   rather than duplicate ipv4 and ipv6.
+   (conninfo, handle_buildid): Represent ipv4-mapped ipv6 addresses
+   in their native ipv4 form for logging and X-F-F: purposes.
+   * debuginfod-client.c (debuginfod_add_http_header): Tolerate
+   colons in http header values.
+
 2022-04-03  Frank Ch. Eigler 
 
* debuginfod.cxx (main): Use MHD_USE_EPOLL for libmicrohttpd, to
diff --git a/debuginfod/debuginfod-client.c b/debuginfod/debuginfod-client.c
index 024b09545d99..41ba88a56911 100644
--- a/debuginfod/debuginfod-client.c
+++ b/debuginfod/debuginfod-client.c
@@ -1548,13 +1548,13 @@ int debuginfod_find_source(debuginfod_client *client,
 int debuginfod_add_http_header (debuginfod_client *client, const char* header)
 {
   /* Sanity check header value is of the form Header: Value.
- It should contain exactly one colon that isn't the first or
+ It should contain at least one colon that isn't the first or
  last character.  */
-  char *colon = strchr (header, ':');
-  if (colon == NULL
-  || colon == header
-  || *(colon + 1) == '\0'
-  || strchr (colon + 1, ':') != NULL)
+  char *colon = strchr (header, ':'); /* first colon */
+  if (colon == NULL /* present */
+  || colon == header /* not at beginning - i.e., have a header name */
+  || *(colon + 1) == '\0') /* not at end - i.e., have a value */
+/* NB: but it's okay for a value to contain other colons! */
 return -EINVAL;
 
   struct curl_slist *temp = curl_slist_append (client->headers, header);
diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index 99924d36f260..48e0f37b33f0 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -1063,9 +1063,22 @@ conninfo (struct MHD_Connection * conn)
 sts = getnameinfo (so, sizeof (struct sockaddr_in), hostname, sizeof 
(hostname), servname,
sizeof (servname), NI_NUMERICHOST | NI_NUMERICSERV);
   } else if (so && so->sa_family == AF_INET6) {
-sts = getnameinfo (so, sizeof (struct sockaddr_in6), hostname, sizeof 
(hostname),
-   servname, sizeof (servname), NI_NUMERICHOST | 
NI_NUMERICSERV);
+struct sockaddr_in6* addr6 = (struct sockaddr_in6*) so;
+if (IN6_IS_ADDR_V4MAPPED(&addr6->sin6_addr)) {
+  struct sockaddr_in addr4;
+  memset (&addr4, 0, sizeof(addr4));
+  addr4.sin_family = AF_INET;
+  addr4.sin_port = addr6->sin6_port;
+  memcpy (&addr4.sin_addr.s_addr, addr6->sin6_addr.s6_addr+12, 
sizeof(addr4.sin_addr.s_addr));
+  sts = getnameinfo ((struct sockaddr*) &addr4, sizeof (addr4),
+ hostname, sizeof (hostname), servname, sizeof 
(servname),
+ NI_NUMERICHOST | NI_NUMERICSERV);
+} else {
+  sts = getnameinfo (so, sizeof (struct sockaddr_in6), hostname, sizeof 
(hostname), NULL, 0,
+ NI_NUMERICHOST);
+}
   }
+  
   if (sts != 0) {
 hostname[0] = servname[0] = '\0';
   }
@@ -2008,13 +2021,26 @@ and will not query the upstream servers");

MHD_CONNECTION_INFO_CLIENT_ADDRESS);
   struct sockaddr *so = u ? u->client_addr : 0;
   char hostname[256] = ""; // RFC1035
-  if (so && so->sa_family == AF_INET)
+  if (so && so->sa_family == AF_INET) {
 (void) getnameinfo (so, sizeof (struct sockaddr_in), hostname, 
sizeof (hostname), NULL, 0,
 NI_NUMERICHOST);
-  else if (so && so->sa_family == AF_INET6)
-(void) getnameinfo (so, sizeof (struct sockaddr_in6), hostname, 
sizeof (hostname), NULL, 0,
-NI_NUMERICHOST);
-
+  } else if (so && so->sa_family == AF_INET6) {
+struct sockaddr_in6* addr6 = (struct sockaddr_in6*) so;
+if (IN6_IS_ADDR_V4MAPPED(&addr6->sin6_addr)) {
+  struct sockaddr_in addr4;
+  memset (&addr4, 0, sizeof(addr4));
+  addr4.sin_family = AF_INET;

[Bug debuginfod/28708] run-debuginfod-webapi-concurrency.sh seems to be flaky

2022-04-03 Thread evvers at ya dot ru via Elfutils-devel
https://sourceware.org/bugzilla/show_bug.cgi?id=28708

--- Comment #12 from Evgeny Vereshchagin  ---
FWIW with
https://sourceware.org/git/?p=elfutils.git;a=commit;h=e646e363e72e06e0ed5574c929236d815ddcbbaf
applied the test appears to be flaky on Packit on s390x:
https://copr-be.cloud.fedoraproject.org/results/packit/evverx-elfutils-73/fedora-35-s390x/03942110-elfutils/builder-live.log.gz

-- 
You are receiving this mail because:
You are on the CC list for the bug.