[Bug debuginfod/29472] Support querying the debuginfod-server for metadata
https://sourceware.org/bugzilla/show_bug.cgi?id=29472 --- Comment #2 from Ryan Goldberg --- Created attachment 14289 --> https://sourceware.org/bugzilla/attachment.cgi?id=14289&action=edit Submit A Patch for 29472 Here is patch which is simpler than the above proposal. It allows for querying by source path and will return a JSON array of metadata (where the metadata for each file is its own JSON object). The patch was tested using the elfutils try-buildbots on branch users/rgoldber/try-metadata_query There is a need for this patch as currently, even with work like PR 28284, a user is unable to check what files are accessible with debuginfod, they can only at most try and query by buildid and see if they can get a result. This feature will be used in systemtap PR 27410 -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/29472] Support querying the debuginfod-server for metadata
https://sourceware.org/bugzilla/show_bug.cgi?id=29472 --- Comment #3 from Ryan Goldberg --- Created attachment 14306 --> https://sourceware.org/bugzilla/attachment.cgi?id=14306&action=edit Submit A Patch for 29472, bug fix Minor bug fix to be applied to the previous patch -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/29472] Support querying the debuginfod-server for metadata
https://sourceware.org/bugzilla/show_bug.cgi?id=29472 Ryan Goldberg changed: What|Removed |Added Attachment #14289|0 |1 is obsolete|| Attachment #14306|0 |1 is obsolete|| --- Comment #4 from Ryan Goldberg --- Created attachment 14310 --> https://sourceware.org/bugzilla/attachment.cgi?id=14310&action=edit Submit A Patch for 29472, revised I've revised the previous patch to include the suggestions from elfutils-devel [debuginfod metadata patch review] -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/29696] intermittent libmicrohttpd assertion failures related to socket fd closing
https://sourceware.org/bugzilla/show_bug.cgi?id=29696 Ryan Goldberg changed: What|Removed |Added Status|NEW |ASSIGNED Assignee|unassigned at sourceware dot org |rgoldber at redhat dot com CC||rgoldber at redhat dot com --- Comment #1 from Ryan Goldberg --- Created attachment 14933 --> https://sourceware.org/bugzilla/attachment.cgi?id=14933&action=edit Patch for 29696 The debuginfod cache config was using fdopen and then calling both fclose on the file stream & close on the original file descriptor. Since the fd is not dup'ed, this led to a race condition where if that fd was reused (by microhttpd) we'd end up prematurely closing their socket leading to the above issue. -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/29696] intermittent libmicrohttpd assertion failures related to socket fd closing
https://sourceware.org/bugzilla/show_bug.cgi?id=29696 --- Comment #3 from Ryan Goldberg --- I noticed that the issue was happening in run-debuginfod-federation-metrics.sh so to reproduce I was playing with sending lots of requests to a federation of servers. Only had the issue occur on the downstream so it was a client issue. This made it pretty quick to replicate, so I could go through debuginfod_query_server and see how far down I can put an early exit before seeing the race condition. That narrowed it down to debuginfod_config_cache. >From there noticed the double close and it was smooth sailing. I'm not sure about tooling around the double close but is it possible to know that something is a double close if the fd may just be reused? Since in this case for instance the close won't fail, we're just closing someone else's open, good to go fd. fwiw I looked in elfutils at least and we didn't use fdopen with a double close again. -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/28204] extend webapi / verification with forthcoming signed-contents archives
https://sourceware.org/bugzilla/show_bug.cgi?id=28204 Ryan Goldberg changed: What|Removed |Added Status|NEW |ASSIGNED --- Comment #10 from Ryan Goldberg --- Hi, it has been quite the journey, but the latest draft of this patch is ready for review It is sitting on the try-branch users/rgoldber/try-bz28204c (figure'd it was big enough that just looking at a patch might be hard to follow) Since the last review the major changes are as follows * I added a --koji-sigcache flag to the server which will enable koji specific mappings of rpm paths to get IMA signatures. * DEBUGINFOD_IMA_CERT_PATH can now include paths to dirs containing PEM and DER encoded certificates. And will be traversed looking for the first cert which has a skid matching the signature which we need to validate. * The verification certificates for RHEL and CentOS have been finalized and we have a green light to distribute copies of them alongside our source (since they have not been formally published to a known location yet). They are in debuginfod/ima-certs and will be installed to $(sysconfdir)/debuginfod/ima-certs. DEBUGINFOD_IMA_CERT_PATH will by default include this path. This dir also has copies of the current fedora verification certs (which are already public but not yet backported to f38 [fedora-repos commit 93b2c8a]) -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/28204] extend webapi / verification with forthcoming signed-contents archives
https://sourceware.org/bugzilla/show_bug.cgi?id=28204 --- Comment #16 from Ryan Goldberg --- (In reply to Mark Wielaard from comment #12) > In config/profile.csh.in and config/profile.sh.in the prefix variable is > explicitly set and no longer unset. Is that deliberate? Taking a look at both files they seem to still contain `unset prefix` as their last lines. Would that not do the trick? > In debuginfod_validate_imasig the file_data = malloc(data_len); depends on > the (externally) given file size. It is then read in one pread call. And the > whole buffer is then given to EVP_DigestUpdate. Note that this might create > a giant malloc buffer, which might trigger OOM. pread might succeed with > fewer bytes than given. It needs to be called in a loop. But it would be > better if we could read it and feed it to EVP_DigestUpdate in (small) chunks. Sure, just looked at the docs for EVP_DigestUpdate and seems like this change should be pretty straightforward. > Is EACCESS the right error code to return when the signature couldn't be > checked/is invalid? That is the same as when we get > CURLE_REMOTE_ACCESS_DENIED. It might be good if it was an unique error code > so users can know that the file was not trusted. How do you feel about EPERM? Not exactly its usage but we've used up a good number of the more applicable ones with the CURL errors (EACCESS, EINVAL) (In reply to Mark Wielaard from comment #13) > The configure checks might need to check whether the rpm development headers > define the needed constants. Sure (In reply to Mark Wielaard from comment #14) > I think it is the user/distro packager who should decide which ima-certs to > ship. I don't think elfutils should come with ima-certs itself. >From what I've seen finding the correct certificates for IMA verification has been pretty tricky, so distributing a copy of these (at least early certificates) might be a good idea. Eventually the intention is for these certs to be shipped in a known location which we can easily add to the search path, but until then having a copy seems like the best bet imho. > Why is there a "permissive" policy? What is the use case for this? It might be that a user wants to be aware of what's going on but does not need to be quite so strict on rejection. Permissive won't allow incorrect signatures but will allow say an unsigned file whereas an enforcing client will reject anything that does not come with a valid signature > Should the policy be per debuginfod url? So you can point to an official > distro debuginfod which must be in enforcing mode, but can add a local one > with an "ignore" policy. I was thinking of keeping things simple, but don't have anything against moving to a per URL sort of approach. What kind of format would you want for such a thing? For urls foo:bar:baz would we want something like ignore::enforcing where blanks are the default? Don't love this structure since it seems a little unwieldy. I'll think on a better format for it. (In reply to Mark Wielaard from comment #15) > But I must admit that I am a little lost in the rpm and koji mechanisms to > extract those signatures. How easy will it be to extend to other platforms > that might store such signatures in different ways? The koji flag seems like a necessary evil in my opinion. At least the way it works is that when the flag is enabled we try the koji storage method and then fall back to the rpm itself, so to add another signature storage format we could do the same method, so adding more shouldn't be too difficult. Might require a little logic tweak but wouldn't think its too complex a change -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/28204] extend webapi / verification with forthcoming signed-contents archives
https://sourceware.org/bugzilla/show_bug.cgi?id=28204 --- Comment #19 from Ryan Goldberg --- (In reply to Mark Wielaard from comment #17) > Maybe prefixing or postfixing URLS with + or adding the name of the cert? I'm leaning towards a combination of this idea with my original. We can use DEBUGINFOD_IMA_POLICY to set the default ima verification response policy (possibly renaming to DEBUGINFOD_IMA_DEFAULT_POLICY) and then we can use DEBUGINFOD_URLS="url1+enforcing url2 url3+ignore". This seems like a simple way the user can define policy at a granularity of their choice. I wrote up a quick test patch for this and it seems pretty straightforward. (In reply to Frank Ch. Eigler from comment #18) > > Doesn't that give a false sense of "security"? > > It still rejects some stuff, but doesn't really protect against "falsifying" > > files, all a server has to do is not provide an IMA > > Yes, but trusted servers won't just do that. It's up to the user to choose when to allow a permissive policy. Maybe not using it as the default will alleviate these concerns? A user would have to explicitly choose to 'let their guard down' when for instance using a trusted server > The difference between missing and invalid is that the latter is KNOWN bad. > An invalid signature is evidence that the file has a problem. I agree that the distinction is enough that having a 3rd mode seems like a good bet. When using https://debuginfod.fedoraproject.org/ it doesn't feel like an issue to me to use a file that doesn't happen to have a signature, but otoh I wouldn't want one that I know is incorrect -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/28204] extend webapi / verification with forthcoming signed-contents archives
https://sourceware.org/bugzilla/show_bug.cgi?id=28204 --- Comment #21 from Ryan Goldberg --- (In reply to Mark Wielaard from comment #20) > But isn't the idea of checking the IMA signatures that you don't have to > trust the server providing the debuginfo files as the distro intended them? But this will allow for the case of a trusted server which only has some of it's RPMs per-file signed. Take for instance a server which has the RPMs for f36,37,38. The f36 files don't have signatures so using enforcing here is too strict since we are ok just letting a client know that these ones are unverifiable, but we still want to be able to reject any of the invalid ones for f38 > So both are bad in some way. Which imho means that if we support some kind > of permissive mode, then it should explicitly warn for both kind of baddness. In the permissive mode you'll get: * "the signature is valid" for valid sigs * "ALERT: this download is being rejected since the IMA signature could not be verified" for invalid sigs * "the signature could not be verified" otherwise So we do warn for both kinds of bad, we just don't reject the 'unknown' bad -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/28204] extend webapi / verification with forthcoming signed-contents archives
https://sourceware.org/bugzilla/show_bug.cgi?id=28204
--- Comment #23 from Ryan Goldberg ---
(In reply to Mark Wielaard from comment #22)
> This still feels odd. Since you cannot distinguish between non-sig f36
> package (okish?) and non-sig f38 packages (bad?). I think you have to either
> trust or reject all non-sig packages from such a server.
I see where you're coming from but what policy would you use for
https://debuginfod.fedoraproject.org? We'd have to ignore then, which seems
like one of the primary users of such verification might just skip it all
together.
> I'll look at the code to see if I understand what this means in practice.
> Are those the messages presented to the user (in verbose mode? always?). And
> does this mean all three cases warn (or only the second and third)? And is
> it just a message or does it also mean actual rejection in some cases?
2008 if(NULL != url_ima_policies && ignore != url_ima_policies[committed_to])
2009 {
2010 int result = debuginfod_validate_imasig(c, target_cache_tmppath, fd);
2011 if(0 == result)
2012 {
2013 if (vfd >= 0) dprintf (vfd, "the signature is valid\n");
2014 }
2015 else if(EINVAL == result || enforcing ==
url_ima_policies[committed_to])
2016 {
2017 // All invalid signatures are rejected.
2018 // Additionally in enforcing mode any non-valid signature is
rejected, so by reaching
2019 // this case we do so since we know it is not valid. Note - this not
just invalid signatures
2020 // but also signatures that cannot be validated
2021 if (vfd >= 0) dprintf (vfd, "ALERT: this download is being rejected
since the IMA signature could not be verified\n");
2022 rc = -EPERM;
2023 goto out2;
2024 }
2025 else
2026 {
2027 // By default we are permissive, so since the signature isn't
invalid we
2028 // give it the benefit of the doubt
2029 if (vfd >= 0) dprintf (vfd, "the signature could not be
verified\n");
2030 }
2031 }
2032
Here is the relevant snippet
--
You are receiving this mail because:
You are on the CC list for the bug.
[Bug debuginfod/28204] extend webapi / verification with forthcoming signed-contents archives
https://sourceware.org/bugzilla/show_bug.cgi?id=28204
--- Comment #27 from Ryan Goldberg ---
(In reply to Mark Wielaard from comment #24)
> BTW. How does this interact with the "section" queries?
Since these aren't ima verifiable anyways wdyt of just skipping verification
all together (i.e treat that query in the same way as the ignore policy)
Tweaking the above to something like:
2008if(NULL != url_ima_policies && ignore != url_ima_policies[committed_to]
&& NULL == section) { ... }
(In reply to Mark Wielaard from comment #25)
> Includes an "undefined" policy?
Just used internally when parsing DEBUGINFOD_URLS
> Is the k += DATA_SIZE correct? Can't pread return an n < DATA_SIZE?
Fixed
> If the cert_paths = strdup (...) fails cert_paths gets assigned a static
> string?
Fixed
(In reply to Mark Wielaard from comment #26)
> If we have an permissive mode then I think it should work like the selinux
> permissive mode.
> That is, it should always check the signature, but instead of failing with
> EPERM, it should
> always produce a warning (whether or not we are in verbose mode or not).
I would be ok with this kind of permissive mode
--
You are receiving this mail because:
You are on the CC list for the bug.
[Bug debuginfod/28204] extend webapi / verification with forthcoming signed-contents archives
https://sourceware.org/bugzilla/show_bug.cgi?id=28204 Ryan Goldberg changed: What|Removed |Added CC||rgoldber at redhat dot com --- Comment #5 from Ryan Goldberg --- Created attachment 14208 --> https://sourceware.org/bugzilla/attachment.cgi?id=14208&action=edit Submit A Patch for 28204 At the moment there are no public x509 verification certificates available to validate the signatures generated for RHEL 9. There is discussion at https://issues.redhat.com/browse/SIGNSERVER-245 for the future location and in fedora 37 the cert will be available on the fedora security site, https://getfedora.org/security/ -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/28204] extend webapi / verification with forthcoming signed-contents archives
https://sourceware.org/bugzilla/show_bug.cgi?id=28204 Ryan Goldberg changed: What|Removed |Added Attachment #14208|0 |1 is obsolete|| --- Comment #7 from Ryan Goldberg --- Created attachment 14256 --> https://sourceware.org/bugzilla/attachment.cgi?id=14256&action=edit Submit A Patch for 28204, corrections made The following patch fixes the previously mentioned issues. It passes all the tests across the try buildbots (users/rgoldber/try-bz28204) -- You are receiving this mail because: You are on the CC list for the bug.
[Bug debuginfod/29472] New: Support querying the debuginfod-server for metadata
https://sourceware.org/bugzilla/show_bug.cgi?id=29472
Bug ID: 29472
Summary: Support querying the debuginfod-server for metadata
Product: elfutils
Version: unspecified
Status: NEW
Severity: normal
Priority: P2
Component: debuginfod
Assignee: unassigned at sourceware dot org
Reporter: rgoldber at redhat dot com
CC: elfutils-devel at sourceware dot org
Target Milestone: ---
It would be beneficial to be able to query a debuginfod-server for its metadata
matching certain conditions. This would allow for operations like seeing what
executable files are available without attempting to download them or seeing
which files are larger than a certain size, which could be used to
prefetch+cache large required files, before the user needs them.
The proposed format is as follows:
debuginfod-find [OPTION]... metadata CONDITION QUERYFMT
Query all of the metadata matching the given CONDITION and return it as
described by QUERYFMT
The supported debuginfod-fields are:
BUILDID, FILENAME, MODIFIED_TIME, SIZE, TYPE, SOURCE_TYPE, SOURCE
and should be referenced using {...}, noting that field names are case
insensitive.
CONDITION is a boolean expression composed of debuginfod-fields, constants
(strings and integers)
and the: `( ) = != < > <= >= && ||` operators, following the standard
conditional precedence order
QUERYFMT is a modified version of the standard printf(3) formatting.
The format is made up of static strings (which may include standard C character
escapes for new‐
lines, tabs, and other special characters (not including \0)) and printf(3)
type formatters. In
place of the type specifier in the format string, use the debuginfod-field you
wish to query.
For example:
debuginfod-find metadata "{FILENAME}=/usr/bin/grep && {TYPE}=E"
"{FILENAME}({BUILDID})\t{SOURCE_TYPE}\n"
Will query all executables with the name /usr/bin/grep and might return the
following
/usr/bin/grep(90e7d8894b94f47ad17722ff8658f833f329b035)R
/usr/bin/grep(e81e4e6e322030178260ae4f6055f781cd4997e1)F
debuginfod-find metadata "{FILENAME}=/bin/bash" "{FILENAME}-{TYPE}\n"
Might return
/bin/bash-E
/bin/bash-S
/bin/bash-D
--
You are receiving this mail because:
You are on the CC list for the bug.
