Re: buildbot users try branch builders for elfutils
On 7/28/22 17:26, Mark Wielaard wrote: > Hi, > > I setup git users try branches for elfutils. If you have commit > access to elfutils.git you can now push to a users//try-xxx > branch and the buildbot will do builds and sent you (the commit > author) email about the results. The builds are also visible at: > https://builder.sourceware.org/buildbot/#/builders?tags=elfutils-try > Hi. I would like to try this great feature! > My workflow to use this looks like: > > - git checkout -b ar_size > - hack, hack, hack... OK, looks good to submit > - git commit -a -m "Now who got an ar_size?" > - git push origin ar_size:users/mark/try-ar_size > - ... wait for the emails to come in or watch buildbot logs ... But I get: git push origin PR29474-fix-debuginfod-concurrency:users/marxin/try-PR29474-fix-debuginfod-concurrency Enumerating objects: 15, done. Counting objects: 100% (15/15), done. Delta compression using up to 16 threads Compressing objects: 100% (9/9), done. Writing objects: 100% (9/9), 1.69 KiB | 1.69 MiB/s, done. Total 9 (delta 7), reused 0 (delta 0), pack-reused 0 error: remote unpack failed: unable to create temporary object directory To ssh://sourceware.org/git/elfutils.git ! [remote rejected] PR29474-fix-debuginfod-concurrency -> users/marxin/try-PR29474-fix-debuginfod-concurrency (unpacker error) error: failed to push some refs to 'ssh://sourceware.org/git/elfutils.git' Can you please take a look what happens? Thanks, Martin
[Bug debuginfod/29474] Server returns 404 for concurrent requests when leading to a same .rpm
https://sourceware.org/bugzilla/show_bug.cgi?id=29474 Martin Liska changed: What|Removed |Added Assignee|unassigned at sourceware dot org |mliska at suse dot cz -- You are receiving this mail because: You are on the CC list for the bug.
[PATCH] debuginfod: Fix concurrent request leading to a same .rpm file [PR29474]
As explained in detail in the PR, the problem happens when multiple
requests lead to a single RPM and one of the threads does prefetching.
In that case, the requested file is added to fdcache and thus skip
as interned.
---
debuginfod/debuginfod.cxx | 69 ---
1 file changed, 43 insertions(+), 26 deletions(-)
diff --git a/debuginfod/debuginfod.cxx b/debuginfod/debuginfod.cxx
index a089d0bd..e69a4bbc 100644
--- a/debuginfod/debuginfod.cxx
+++ b/debuginfod/debuginfod.cxx
@@ -1610,39 +1610,23 @@ string canonicalized_archive_entry_pathname(struct
archive_entry *e)
return string("/")+fn;
}
-
-
+// Try getting build_r match from fdcache.
static struct MHD_Response*
-handle_buildid_r_match (bool internal_req_p,
-int64_t b_mtime,
-const string& b_source0,
-const string& b_source1,
-int *result_fd)
+try_buildid_r_match_from_cache (const string& b_source0, const string&
b_source1, int *result_fd)
{
- struct stat fs;
- int rc = stat (b_source0.c_str(), &fs);
- if (rc != 0)
-throw libc_exception (errno, string("stat ") + b_source0);
-
- if ((int64_t) fs.st_mtime != b_mtime)
-{
- if (verbose)
-obatched(clog) << "mtime mismatch for " << b_source0 << endl;
- return 0;
-}
-
// check for a match in the fdcache first
+ struct stat fs;
int fd = fdcache.lookup(b_source0, b_source1);
- while (fd >= 0) // got one!; NB: this is really an if() with a possible
branch out to the end
+ if (fd >= 0) // got one!; NB: this is really an if() with a possible branch
out to the end
{
- rc = fstat(fd, &fs);
+ int rc = fstat(fd, &fs);
if (rc < 0) // disappeared?
{
if (verbose)
obatched(clog) << "cannot fstat fdcache " << b_source0 << endl;
close(fd);
fdcache.clear(b_source0, b_source1);
- break; // branch out of if "loop", to try new libarchive fetch
attempt
+ return 0;
}
struct MHD_Response* r = MHD_create_response_from_fd (fs.st_size, fd);
@@ -1651,7 +1635,7 @@ handle_buildid_r_match (bool internal_req_p,
if (verbose)
obatched(clog) << "cannot create fd-response for " << b_source0 <<
endl;
close(fd);
- break; // branch out of if "loop", to try new libarchive fetch
attempt
+ return 0;
}
inc_metric ("http_responses_total","result","archive fdcache");
@@ -1668,9 +1652,34 @@ handle_buildid_r_match (bool internal_req_p,
if (result_fd)
*result_fd = fd;
return r;
- // NB: see, we never go around the 'loop' more than once
}
+ return NULL;
+}
+
+static struct MHD_Response*
+handle_buildid_r_match (bool internal_req_p,
+int64_t b_mtime,
+const string& b_source0,
+const string& b_source1,
+int *result_fd)
+{
+ struct stat fs;
+ int rc = stat (b_source0.c_str(), &fs);
+ if (rc != 0)
+throw libc_exception (errno, string("stat ") + b_source0);
+
+ if ((int64_t) fs.st_mtime != b_mtime)
+{
+ if (verbose)
+obatched(clog) << "mtime mismatch for " << b_source0 << endl;
+ return 0;
+}
+
+ struct MHD_Response *r = try_buildid_r_match_from_cache (b_source0,
b_source1, result_fd);
+ if (r != 0)
+return r;
+
// no match ... grumble, must process the archive
string archive_decoder = "/dev/null";
string archive_extension = "";
@@ -1721,7 +1730,7 @@ handle_buildid_r_match (bool internal_req_p,
// 2) extract the matching entry name (set r = result)
// 3) extract some number of prefetched entries (just into fdcache)
// 4) abort any further processing
- struct MHD_Response* r = 0; // will set in stage 2
+ r = 0; // will set in stage 2
unsigned prefetch_count =
internal_req_p ? 0 : fdcache_prefetch;// will decrement in stage 3
@@ -1743,7 +1752,15 @@ handle_buildid_r_match (bool internal_req_p,
continue;
if (fdcache.probe (b_source0, fn)) // skip if already interned
+ {
+if (r == 0)
+{
+ r = try_buildid_r_match_from_cache (b_source0, b_source1, result_fd);
+ if (r != NULL)
+ return r;
+}
continue;
+ }
// extract this file to a temporary file
char* tmppath = NULL;
@@ -1751,7 +1768,7 @@ handle_buildid_r_match (bool internal_req_p,
if (rc < 0)
throw libc_exception (ENOMEM, "cannot allocate tmppath");
defer_dtor tmmpath_freer (tmppath, free);
- fd = mkstemp (tmppath);
+ int fd = mkstemp (tmppath);
if (fd < 0)
throw libc_exception (errno, "cannot create temporary file");
// NB: don't unlink (tmppath), as fdcache will take charge of it.
--
2.37.1
Re: buildbot users try branch builders for elfutils
Hi Martin, On Mon, Aug 15, 2022 at 09:55:13AM +0200, Martin Liška wrote: > On 7/28/22 17:26, Mark Wielaard wrote: > > I setup git users try branches for elfutils. If you have commit > > access to elfutils.git you can now [...] > > I would like to try this great feature! > [...] > But I get: > > git push origin > PR29474-fix-debuginfod-concurrency:users/marxin/try-PR29474-fix-debuginfod-concurrency > [...] > error: remote unpack failed: unable to create temporary object directory > To ssh://sourceware.org/git/elfutils.git > ! [remote rejected] PR29474-fix-debuginfod-concurrency -> > users/marxin/try-PR29474-fix-debuginfod-concurrency (unpacker error) > error: failed to push some refs to 'ssh://sourceware.org/git/elfutils.git' > > Can you please take a look what happens? That unhelpful error message tries to say you are not an elfutils developer. Lets fix that. I added you to the elfutils groups. Please remember, with great power comes great responsibility (aka, please reread the elfutils CONTRIBUTING file). You should now also be able to use the users try branches. So please try again. Cheers, Mark
Re: buildbot users try branch builders for elfutils
On 8/15/22 12:01, Mark Wielaard wrote: > Hi Martin, > > On Mon, Aug 15, 2022 at 09:55:13AM +0200, Martin Liška wrote: >> On 7/28/22 17:26, Mark Wielaard wrote: >>> I setup git users try branches for elfutils. If you have commit >>> access to elfutils.git you can now [...] >> >> I would like to try this great feature! >> [...] >> But I get: >> >> git push origin >> PR29474-fix-debuginfod-concurrency:users/marxin/try-PR29474-fix-debuginfod-concurrency >> [...] >> error: remote unpack failed: unable to create temporary object directory >> To ssh://sourceware.org/git/elfutils.git >> ! [remote rejected] PR29474-fix-debuginfod-concurrency -> >> users/marxin/try-PR29474-fix-debuginfod-concurrency (unpacker error) >> error: failed to push some refs to 'ssh://sourceware.org/git/elfutils.git' >> >> Can you please take a look what happens? > > That unhelpful error message tries to say you are not an elfutils > developer. > > Lets fix that. I added you to the elfutils groups. Please remember, > with great power comes great responsibility (aka, please reread the > elfutils CONTRIBUTING file). > > You should now also be able to use the users try branches. So please > try again. Thanks Mark, it works nice! Martin > > Cheers, > > Mark
[Bug debuginfod/29474] Server returns 404 for concurrent requests when leading to a same .rpm
https://sourceware.org/bugzilla/show_bug.cgi?id=29474 Frank Ch. Eigler changed: What|Removed |Added Status|NEW |RESOLVED Resolution|--- |FIXED --- Comment #5 from Frank Ch. Eigler --- elfutils-0.187-44-g31d6b1fe7 / debuginfod/ChangeLog debuginfod/debuginfod.cxx: PR29474: debuginfod -- You are receiving this mail because: You are on the CC list for the bug.
