Systemd service and timer for refreshing archlinux-keyring keys via WKD
Hi all, I currently have a MR open against archlinux-keyring [1], that adds a systemd service and timer, which would automatically refresh valid and existing keys on user systems. For people without access to the repository's ticket and merge request features, it is possible to browse the commits in question on the respective branch [2]. To quote from the accompanying ticket [3]: ``` For situations in which we do have new signatures which have been released, we may want a service on user systems that update existing keys via WKD to have an updated set of keys before updating archlinux-keyring. Rationale: Keys are updated in WKD after a release of archlinux-keyring (currently this is still done in the https://gitlab.archlinux.org/archlinux/wkd repo). User systems may upgrade any time after that and pacman will pull new keys from WKD automatically, however it will not pull updates for existing keys. Packages that are signed with a key that still had marginal trust in release A (and therefore already existed on the user system since release A) and gained full trust in release B will not be updated before the user does a system upgrade. This leads to the requirement of installing archlinux-keyring before doing a system upgrade, as otherwise the key will still have marginal trust on the user system and the signatures of other updated packages using the key in question will fail to validate. To remedy this situation I think it would be helpful to have a service update existing keys from WKD twice daily on a timer. I am not 100% sure whether this has side-effects in regards to revocations, but as we only ever update WKD after issuing a release and not when updating the default branch, this should be fine(?). ``` In short: The service in question will update any valid keys in the pacman keyring on user systems, that use Arch Linux's relevant domains as UID to retrieve updated keys (e.g. extended expiration, additional signatures). Invalid keys and those of other domains are ignored. This feature is implemented to circumvent (some, but not all) cases of "update archlinux-keyring before doing a system upgrade". Cases not covered are (in no particular order): * *long* outdated system is updated * new main signing key is required for a packager key to gain full trust for the packages about to be updated * local system's archlinux-keyring does not yet offer this feature * local system's pacman does not yet offer the feature of downloading package signatures for new keys from WKD * packager key on local system does not yet have any main key signatures and would gain full trust with an update of archlinux-keyring Currently the timer which triggers this service is supposed to be vendor enabled (i.e. symlink in /usr/lib/systemd/system/timers.target.wants/) and run daily with a deviation of up to 12h. Members of the DevOps team have raised concerns about the interval as we do not really have a clear picture about how many Arch Linux installations there are world-wide to get numbers on expected median load for this. If you have ideas for improvement or general questions, please direct them at the merge request. Best, David [1] https://gitlab.archlinux.org/archlinux/archlinux-keyring/-/merge_requests/138 [2] https://gitlab.archlinux.org/archlinux/archlinux-keyring/-/tree/wkd_sync_service [3] https://gitlab.archlinux.org/archlinux/archlinux-keyring/-/issues/184 -- https://sleepmap.de signature.asc Description: PGP signature
Re: Systemd service and timer for refreshing archlinux-keyring keys via WKD
On 23.07.2022 18.38, David Runge wrote: Hi all, I currently have a MR open against archlinux-keyring [1], that adds a systemd service and timer, which would automatically refresh valid and existing keys on user systems. For people without access to the repository's ticket and merge request features, it is possible to browse the commits in question on the respective branch [2]. To quote from the accompanying ticket [3]: ``` For situations in which we do have new signatures which have been released, we may want a service on user systems that update existing keys via WKD to have an updated set of keys before updating archlinux-keyring. Rationale: Keys are updated in WKD after a release of archlinux-keyring (currently this is still done in the https://gitlab.archlinux.org/archlinux/wkd repo). User systems may upgrade any time after that and pacman will pull new keys from WKD automatically, however it will not pull updates for existing keys. Packages that are signed with a key that still had marginal trust in release A (and therefore already existed on the user system since release A) and gained full trust in release B will not be updated before the user does a system upgrade. This leads to the requirement of installing archlinux-keyring before doing a system upgrade, as otherwise the key will still have marginal trust on the user system and the signatures of other updated packages using the key in question will fail to validate. To remedy this situation I think it would be helpful to have a service update existing keys from WKD twice daily on a timer. I am not 100% sure whether this has side-effects in regards to revocations, but as we only ever update WKD after issuing a release and not when updating the default branch, this should be fine(?). ``` In short: The service in question will update any valid keys in the pacman keyring on user systems, that use Arch Linux's relevant domains as UID to retrieve updated keys (e.g. extended expiration, additional signatures). Invalid keys and those of other domains are ignored. This feature is implemented to circumvent (some, but not all) cases of "update archlinux-keyring before doing a system upgrade". Cases not covered are (in no particular order): * *long* outdated system is updated * new main signing key is required for a packager key to gain full trust for the packages about to be updated * local system's archlinux-keyring does not yet offer this feature * local system's pacman does not yet offer the feature of downloading package signatures for new keys from WKD * packager key on local system does not yet have any main key signatures and would gain full trust with an update of archlinux-keyring Currently the timer which triggers this service is supposed to be vendor enabled (i.e. symlink in /usr/lib/systemd/system/timers.target.wants/) and run daily with a deviation of up to 12h. Members of the DevOps team have raised concerns about the interval as we do not really have a clear picture about how many Arch Linux installations there are world-wide to get numbers on expected median load for this. The load aspect should be solvable, worst-case DevOps gets annoyed ;) My main concern is putting ourselves in a position, where we know every arch installation, yes it is just the IP addresses, but still. At the same time it becomes easier to detect that a computer is running arch, by just look at the network traffic (yes you can already do that today, by checking if the computer is connecting to a arch mirror). Maybe I'm blowing it out of proportion, but with our user base being more privacy aware than most people, I think it is worth mentioning. Cheers, Kristian If you have ideas for improvement or general questions, please direct them at the merge request. Best, David [1] https://gitlab.archlinux.org/archlinux/archlinux-keyring/-/merge_requests/138 [2] https://gitlab.archlinux.org/archlinux/archlinux-keyring/-/tree/wkd_sync_service [3] https://gitlab.archlinux.org/archlinux/archlinux-keyring/-/issues/184 OpenPGP_signature Description: OpenPGP digital signature
Re: Systemd service and timer for refreshing archlinux-keyring keys via WKD
On Sat, 23 Jul 2022 at 19:39, David Runge wrote: > Packages that are signed with a key that still had marginal trust in > release A (and therefore already existed on the user system since > release A) and gained full trust in release B will not be updated before > the user does a system upgrade. This leads to the requirement of > installing archlinux-keyring before doing a system upgrade, as otherwise > the key will still have marginal trust on the user system and the > signatures of other updated packages using the key in question will fail > to validate. This is solvable by not cutting a release with marginally trusted keys. Having all Arch Linux installations make 100-ish requests daily to cover such an edge case is a misutilization of resources (on both sides). I'm certain there are also privacy concerns about enabling this service by default. Furthermore, I doubt our users need to the have their systems babysat like this. In the rare situation where archlinux-keyring must be updated first, the user should be able to handle it by themselves. As you said, pacman will fetch new keys using WKD so, as long as marginally trusted keys are excluded from keyring releases, there's no issue with onboarding new people. tl;dr: I'm vibing way more with continuing to rely on the archlinux-keyring package exclusively; auto-updates are sus.
Re: Systemd service and timer for refreshing archlinux-keyring keys via WKD
On 2022-07-23 21:24:38 (+0200), Kristian Klausen wrote: > The load aspect should be solvable, worst-case DevOps gets annoyed ;) > > My main concern is putting ourselves in a position, where we know > every arch installation, yes it is just the IP addresses, but still. > At the same time it becomes easier to detect that a computer is > running arch, by just look at the network traffic (yes you can already > do that today, by checking if the computer is connecting to a arch > mirror). > > Maybe I'm blowing it out of proportion, but with our user base being > more privacy aware than most people, I think it is worth mentioning. Fair point, however, a similar concern was networkmanager doing a connectivity check [1]. It is up to us to configure our webserver accordingly and be as privacy conserving as possible. E.g. If we wanted to track users (which we don't), we could do so on a rudimentary basis now already by tracking downloads of packager keys. The Web Key Directory is a way more centralistic approach than e.g. SKS was, but we also have better control over how and what we provide there. Best, David [1] https://github.com/archlinux/svntogit-packages/blob/8bdf9488f2845cad527945898b5cb045a263a073/trunk/PKGBUILD#L121-L125 -- https://sleepmap.de signature.asc Description: PGP signature
Re: Systemd service and timer for refreshing archlinux-keyring keys via WKD
On 2022-07-23 23:49:09 (+0300), Evangelos Foutras wrote: > This is solvable by not cutting a release with marginally trusted > keys. Having all Arch Linux installations make 100-ish requests daily > to cover such an edge case is a misutilization of resources (on both > sides). Refreshing existing and valid keys not only has the upside of pulling in new signatures on marginally trusted keys, but also to update expiration dates. I do not see this as misutilization of the systems, as it solves an existing and continued issue for users and user experience by making the upgrade procedure less error-prone (and confusing). When it comes to utilization of our systems (because we would be on the giving end of this transaction), I do understand your concerns. The WKD is a very central and important resource to the distribution and we need to ensure (in any case), that we can serve many users at once and that this does not mess with our gitlab setup. If we could communicate more detailed numbers there and figure out a way to make the hosting of our WKD very stable and reliable, that would be fabulous! > I'm certain there are also privacy concerns about enabling this > service by default. Each upgrade of a user system may trigger a request towards our WKD (unrelated to the proposed service). As mentioned in my response to Kristian, I believe that it is up to us to configure our webserver infrastructure as privacy conserving as possible (for all of our services). > Furthermore, I doubt our users need to the have their systems babysat > like this. In the rare situation where archlinux-keyring must be > updated first, the user should be able to handle it by themselves. As > you said, pacman will fetch new keys using WKD so, as long as > marginally trusted keys are excluded from keyring releases, there's no > issue with onboarding new people. Marginally trusted keys may not only end up in the keyring by adding new packagers, but also by existing packagers changing their keys. As mentioned above, they may be (temporarily) expired as well. We are not really able to make the guarantees that you stated (e.g. "we will from now on only release fully trusted keys"). Scenarios such as emergency key revocations, or just plain oversight may still lead to marginally trusted keys ending up in the keyring. Establishing a workflow around the maintenance and release of archlinux-keyring is challenging in itself and changing it would not solve the problems with temporarily expiring keys either (unless we forbid expiring keys). Spending time on various fora to explain to users of all experience ranges to "first do a partial upgrade to upgrade the keyring" is not only tiring and creating friction, but also wasting a more valuable resource: Contributor time. The update procedure on Arch Linux should be as easy as possible and as resilient to failure as possible for as long as possible. Currently we have reports about this issue (for various reasons) every few weeks and those are only the users reporting these issues somewhere and are not those that want to upgrade their system after e.g. one year of downtime. > tl;dr: I'm vibing way more with continuing to rely on the > archlinux-keyring package exclusively; auto-updates are sus. I understand that. However, maintaining a keyring reliably over a longer period of time is non-trivial. If the implemented service is not wanted by users, it is always possible to disable the unit by masking it. Best, David -- https://sleepmap.de signature.asc Description: PGP signature
Re: Systemd service and timer for refreshing archlinux-keyring keys via WKD
On 24/7/22 06:49, Evangelos Foutras wrote: On Sat, 23 Jul 2022 at 19:39, David Runge wrote: Packages that are signed with a key that still had marginal trust in release A (and therefore already existed on the user system since release A) and gained full trust in release B will not be updated before the user does a system upgrade. This leads to the requirement of installing archlinux-keyring before doing a system upgrade, as otherwise the key will still have marginal trust on the user system and the signatures of other updated packages using the key in question will fail to validate. This is solvable by not cutting a release with marginally trusted keys. Having all Arch Linux installations make 100-ish requests daily to cover such an edge case is a misutilization of resources (on both sides). I'm certain there are also privacy concerns about enabling this service by default. Not shipping keys that are marginally trusted is ideal in principle... However we have seen it many times recently as main/master keys were cycled. Hopefully this is less of an issue moving forward. This is why the keyring was set up to have five main keys, to ensure packager keys were signed by at least 3 at all times. However, not everyone has all five main key signatures. Or even 4. In fact, according to the web page [1], we are screwed if anything happens to Pierre's of Florian's keys, and to a lesser extent with other keys. I am not sure the extent that page reflects current reality, but note Pierre's key is schedule for retirement... (Is it correct that Giancarlo has signed zero keys? I thought they were bought on to replace me almost a year ago!) [1] https://archlinux.org/master-keys/ We have also had marginal trust happen when a new packager (or packager with a new key) starts packaging having gotten all relevant signatures, but not having their key in WKD yet. A version of the key is attempted to be retrieved from a key server that may not have any signatures. I think this is less of an issue going forward given WKD is updated alongside new keyring changes now. Finally, this deals with keys that have expiry dates. We have hit a few cases where expiry and updating keyring happens in a period where a user has not updated their system in a while. We could monitor this and have packagers be reminded to extend their expiry date at an early enough point in time. I suggest an update needs to be at least 6 months in advance of expiry. My conclusion is the reality of the keyring is not perfect. If all packager keys were signed by all main key holders, and WKD was updated before packages were released using a key, and if expiry dates were extended at least 6 months in advance, and , then this would not be an issue. David has put a lot of effort into the keyring to allow us to expose these issues more easily, and has been pushing for main key holders to increase their signing coverage, but this has not progressed at any great rate. The service suggestion really works around limitations in key signing activity. It would not be needed if main key holders signed everything and packagers with keys that expire updated their keys with extensive notice. Onto the service. If the server load and privacy issues were not issues, this would "solve" the problems above. Daily is probably too often, even weekly/monthly would be fine. But I guess the concerns raised mean we should focus on fixing the keyring instead (as futile as that may seem to David currently...). Allan
