Re: [arch-dev-public] Starting x86_64_v3 port
On Sat, Jan 29, 2022 at 11:45:57AM +1000, Allan McRae via arch-dev-public wrote: > On 29/1/22 11:28, Morten Linderud via arch-dev-public wrote: > > On Sat, Jan 29, 2022 at 11:22:32AM +1000, Allan McRae via arch-dev-public > > wrote: > > > On 29/1/22 11:13, Morten Linderud via arch-dev-public wrote: > > > > On Sat, Jan 29, 2022 at 10:12:30AM +1000, Allan McRae via > > > > arch-dev-public wrote: > > > > > Is there any particular objection to requiring packagers upload both > > > > > architectures? > > > > > > > > I'm personally not really motivated doing the required builds. We have > > > > an > > > > underdeveloped infrastructure which hasn't changed since we abandoned > > > > i686 5 > > > > years ago. > > > > > > > > I'd personally like to see more work on our build tooling before we > > > > commit to > > > > new architectures. > > > > > > FYI, it is a single extra command. Either of these will work... > > > > > > offload-build --arch x86_64_v3 > > > extra-x86_64_v3-build > > > > > > Nothing else changes for the packager. > > > > > > Allan > > > > I'll spend twice as long waiting for a package to build which increases the > > time > > spent packaging. Which again requires me to spend more time watching stuff > > fly > > by. > > I have concerns if that is how you spend your time... :) > I have plenty time left over to ask for pacman backports :) DO NOT WORRY! > > This also assumes people are capable of using the buildserver which is not > > always the case either. > > > > This wasn't great with i686, and I'm not sure why we'd find this acceptable > > today? > > What was not great with i686? We managed two architectures for many, many > years. The reason for removing i686 was to do with outdated technology, not > to do with build times or infrastructure. > > Do you have objections beyond not wanting to package for both repos? i.e. do > you object to option C in my original email, where we have a team to keep > the repos in sync when package maintainers do not build both? I'm simply not sure where you are going to get the people for that and how you want to deal with it? A *lot* has changed since the x86_64 port. Bringing on people to do these rebuild implies they need access to infrastructure, keyring and so on. And we already have a staff shortage. I'd like to see some details on how you envision C should be working first. Generally my thoughts is that we shouldn't *need* to have a more manhours to deal with a x86_64_v3. We should instead strenghten our staff and work on the following: * Signing enclave * Better rebuilding tools * Build automation * Git migration It would make discussions like these completely obsolete. Do we want v2, v3, v4, v5, v90001? Enable it in a setting and we'd have the repos. It is a lot of work but it would modernize and make things a lot simpler for us. -- Morten Linderud PGP: 9C02FF419FECBE16 signature.asc Description: PGP signature
Re: [arch-dev-public] Starting x86_64_v3 port
On 29/1/22 22:11, Morten Linderud via arch-dev-public wrote: On Sat, Jan 29, 2022 at 11:45:57AM +1000, Allan McRae via arch-dev-public wrote: On 29/1/22 11:28, Morten Linderud via arch-dev-public wrote: On Sat, Jan 29, 2022 at 11:22:32AM +1000, Allan McRae via arch-dev-public wrote: On 29/1/22 11:13, Morten Linderud via arch-dev-public wrote: On Sat, Jan 29, 2022 at 10:12:30AM +1000, Allan McRae via arch-dev-public wrote: Is there any particular objection to requiring packagers upload both architectures? I'm personally not really motivated doing the required builds. We have an underdeveloped infrastructure which hasn't changed since we abandoned i686 5 years ago. I'd personally like to see more work on our build tooling before we commit to new architectures. FYI, it is a single extra command. Either of these will work... offload-build --arch x86_64_v3 extra-x86_64_v3-build Nothing else changes for the packager. Allan I'll spend twice as long waiting for a package to build which increases the time spent packaging. Which again requires me to spend more time watching stuff fly by. I have concerns if that is how you spend your time... :) I have plenty time left over to ask for pacman backports :) DO NOT WORRY! This also assumes people are capable of using the buildserver which is not always the case either. This wasn't great with i686, and I'm not sure why we'd find this acceptable today? What was not great with i686? We managed two architectures for many, many years. The reason for removing i686 was to do with outdated technology, not to do with build times or infrastructure. Do you have objections beyond not wanting to package for both repos? i.e. do you object to option C in my original email, where we have a team to keep the repos in sync when package maintainers do not build both? I'm simply not sure where you are going to get the people for that and how you want to deal with it? There is community demand for this port - it will provide benefit for the majority of our users. I'm sure I can find interested parties to join. A *lot* has changed since the x86_64 port. Bringing on people to do these rebuild implies they need access to infrastructure, keyring and so on. And we already have a staff shortage. I'd like to see some details on how you envision C should be working first. Exactly the same as it did in the i686/x86_64 days. Some packagers will upload both variants, some will not. There was a webpage that showed the package differences between the architectures and a group of people built and uploaded packages to keep x86_64 in sync. This was particularly important when many devs did not have x86_64 hardware yet. We may/will need to recruit some people to do these rebuilds. The number of people needed depends on how many packagers would package for both architectures. Generally my thoughts is that we shouldn't *need* to have a more manhours to deal with a x86_64_v3. We should instead strenghten our staff and work on the following: * Signing enclave * Better rebuilding tools * Build automation * Git migration It would make discussions like these completely obsolete. Do we want v2, v3, v4, v5, v90001? Enable it in a setting and we'd have the repos. It is a lot of work but it would modernize and make things a lot simpler for us. It has been over a decade since pacman allowed signing of repo databases, and we still have not managed to get those signed. And a git migration has been discussed almost as long. I don't think we should wait for this. Though, given there is several years between hardware releases for each x86_64 version so far, we may be in that situation for automated builds by v90001! But if developing this infrastructure does occur, everyone recruited for packaging x86_64_v3 will be able to help keep packages updated! Win-win. Allan
Re: [arch-dev-public] Starting x86_64_v3 port
On Sat, Jan 29, 2022 at 10:53:41PM +1000, Allan McRae via arch-dev-public wrote: > On 29/1/22 22:11, Morten Linderud via arch-dev-public wrote: > > On Sat, Jan 29, 2022 at 11:45:57AM +1000, Allan McRae via arch-dev-public > > wrote: > > > On 29/1/22 11:28, Morten Linderud via arch-dev-public wrote: > > > > On Sat, Jan 29, 2022 at 11:22:32AM +1000, Allan McRae via > > > > arch-dev-public wrote: > > > > > On 29/1/22 11:13, Morten Linderud via arch-dev-public wrote: > > > > > > On Sat, Jan 29, 2022 at 10:12:30AM +1000, Allan McRae via > > > > > > arch-dev-public wrote: > > > > > > > Is there any particular objection to requiring packagers upload > > > > > > > both > > > > > > > architectures? > > > > > > > > > > > > I'm personally not really motivated doing the required builds. We > > > > > > have an > > > > > > underdeveloped infrastructure which hasn't changed since we > > > > > > abandoned i686 5 > > > > > > years ago. > > > > > > > > > > > > I'd personally like to see more work on our build tooling before we > > > > > > commit to > > > > > > new architectures. > > > > > > > > > > FYI, it is a single extra command. Either of these will work... > > > > > > > > > > offload-build --arch x86_64_v3 > > > > > extra-x86_64_v3-build > > > > > > > > > > Nothing else changes for the packager. > > > > > > > > > > Allan > > > > > > > > I'll spend twice as long waiting for a package to build which increases > > > > the time > > > > spent packaging. Which again requires me to spend more time watching > > > > stuff fly > > > > by. > > > > > > I have concerns if that is how you spend your time... :) > > > > > > > I have plenty time left over to ask for pacman backports :) DO NOT WORRY! > > > > > > > > This also assumes people are capable of using the buildserver which is > > > > not > > > > always the case either. > > > > > > > > This wasn't great with i686, and I'm not sure why we'd find this > > > > acceptable > > > > today? > > > > > > What was not great with i686? We managed two architectures for many, many > > > years. The reason for removing i686 was to do with outdated technology, > > > not > > > to do with build times or infrastructure. > > > > > > Do you have objections beyond not wanting to package for both repos? i.e. > > > do > > > you object to option C in my original email, where we have a team to keep > > > the repos in sync when package maintainers do not build both? > > > > I'm simply not sure where you are going to get the people for that and how > > you > > want to deal with it? > > There is community demand for this port - it will provide benefit for the > majority of our users. I'm sure I can find interested parties to join. > > > A *lot* has changed since the x86_64 port. Bringing on people to do these > > rebuild implies they need access to infrastructure, keyring and so on. And > > we > > already have a staff shortage. > > > > I'd like to see some details on how you envision C should be working first. > > Exactly the same as it did in the i686/x86_64 days. Some packagers will > upload both variants, some will not. There was a webpage that showed the > package differences between the architectures and a group of people built > and uploaded packages to keep x86_64 in sync. This was particularly > important when many devs did not have x86_64 hardware yet. > > We may/will need to recruit some people to do these rebuilds. The number of > people needed depends on how many packagers would package for both > architectures. This doesn't explain what I wanted to know though. Are you expecting we find 5-10-20 people and then onboard them as developers or TUs? Are you envioning a seperate signing keyring, or are you planning on adding them to the archlinux-keyring? How are we dealing with access? Do they get full access to our package repos as devs, or are you planning on a seperate role which has access to the required repositories? This is relevant because of how dbscripts is deployed on gemini. All of this requires a fair bit of planning and thought before it's a feasable option to mention. -- Morten Linderud PGP: 9C02FF419FECBE16 signature.asc Description: PGP signature
Re: [arch-dev-public] Starting x86_64_v3 port
On 29/1/22 23:02, Morten Linderud wrote: On Sat, Jan 29, 2022 at 10:53:41PM +1000, Allan McRae via arch-dev-public wrote: Exactly the same as it did in the i686/x86_64 days. Some packagers will upload both variants, some will not. There was a webpage that showed the package differences between the architectures and a group of people built and uploaded packages to keep x86_64 in sync. This was particularly important when many devs did not have x86_64 hardware yet. We may/will need to recruit some people to do these rebuilds. The number of people needed depends on how many packagers would package for both architectures. This doesn't explain what I wanted to know though. Are you expecting we find 5-10-20 people and then onboard them as developers or TUs? Are you envioning a seperate signing keyring, or are you planning on adding them to the archlinux-keyring? How are we dealing with access? Do they get full access to our package repos as devs, or are you planning on a seperate role which has access to the required repositories? This is relevant because of how dbscripts is deployed on gemini. All of this requires a fair bit of planning and thought before it's a feasable option to mention. Assuming we need people to help the x86_64_v3 port, I would post a news item and have people apply. We have advertised developer positions in the past and received dozens of applications, and readily filled the available positions with quality applicants. They would be brought on as Package Maintainers (once approved on the staff list) with access to [extra] and [community], and have packaging privileges including being added to the keyring. While advertising for x86_64_v3 specific packagers, we should make a list of other packaging areas needing help and recruit for those too. Allan
Re: [arch-dev-public] Starting x86_64_v3 port
On Sat, Jan 29, 2022 at 2:49 PM Allan McRae via arch-dev-public wrote: > Assuming we need people to help the x86_64_v3 port, I would post a news > item and have people apply. We have advertised developer positions in > the past and received dozens of applications, and readily filled the > available positions with quality applicants. They would be brought on > as Package Maintainers (once approved on the staff list) with access to > [extra] and [community], and have packaging privileges including being > added to the keyring. > > While advertising for x86_64_v3 specific packagers, we should make a > list of other packaging areas needing help and recruit for those too. While I would have preferred to gradually have raised the CPU requirements of our main repo (e.g. v2 right now and v3 in a few years), maintaining two x86_64 variants for a transition period might work. Nonetheless we should in work on how to get Arch back to be bleeding edge regardless of this. One aspect might be to reduce the overall amount of packages and get rid of unmaintained software (either by us or upstream). As the vast majority of hardware is v3 already we should consider x86_64 (https://pierre-schmitz.com
Re: [arch-dev-public] Starting x86_64_v3 port
Shouldn't we rather recruit people to help with build automation, git migration and the other topics that keep biting us whenever we try adopting new stuff in a timely manner and/or with less manual stuff to do? I fully agree with Morten on this one and we should definitely get our infra and build automation sorted out first, before thinking about trying to "compete" with new features that other distros might offer. Admittedly I've been intrigued by the x86_64_v3/x86_64_v4 targets for a while but no matter how I look at it, having better build automation and tooling around it is an absolute blocker for this. Don't get me wrong, I'm totally in favour of this proposal in the hopefully not too distant future, just not right now until the aforementioned blockers have been resolved. On 29.01.22 23:49, Allan McRae via arch-dev-public wrote: > On 29/1/22 23:02, Morten Linderud wrote: > > On Sat, Jan 29, 2022 at 10:53:41PM +1000, Allan McRae via arch-dev-public > > wrote: > > > Exactly the same as it did in the i686/x86_64 days. Some packagers will > > > upload both variants, some will not. There was a webpage that showed the > > > package differences between the architectures and a group of people built > > > and uploaded packages to keep x86_64 in sync. This was particularly > > > important when many devs did not have x86_64 hardware yet. > > > > > > We may/will need to recruit some people to do these rebuilds. The number > > > of > > > people needed depends on how many packagers would package for both > > > architectures. > > > > This doesn't explain what I wanted to know though. > > > > Are you expecting we find 5-10-20 people and then onboard them as > > developers or TUs? > > > > Are you envioning a seperate signing keyring, or are you planning on adding > > them > > to the archlinux-keyring? > > > > How are we dealing with access? Do they get full access to our package > > repos as > > devs, or are you planning on a seperate role which has access to the > > required > > repositories? This is relevant because of how dbscripts is deployed on > > gemini. > > > > All of this requires a fair bit of planning and thought before it's a > > feasable > > option to mention. > > Assuming we need people to help the x86_64_v3 port, I would post a news item > and have people apply. We have advertised developer positions in the past > and received dozens of applications, and readily filled the available > positions with quality applicants. They would be brought on as Package > Maintainers (once approved on the staff list) with access to [extra] and > [community], and have packaging privileges including being added to the > keyring. > > While advertising for x86_64_v3 specific packagers, we should make a list of > other packaging areas needing help and recruit for those too. > > Allan Grüße, Thore -- Thore Bödecker GPG ID: 0xD622431AF8DB80F3 GPG FP: 0F96 559D 3556 24FC 2226 A864 D622 431A F8DB 80F3 signature.asc Description: PGP signature
[arch-dev-public] Signing enclave
Hi all The lack of package database signing was mentioned yet again and I think it is time to get the "Signing enclave" project rolling. A design was sketched two years ago[1], and based on that design I'm proposing a new design, without a HSM, which should be implementable today. The initial goal would be setting up the necessary infrastructure for us to be able to implement package database signing. Afterwards we can iterate and adapt the solution for more use-cases (ex: releng signing). Hosting: - Hosted on a Hetzner cloud VM as most of our infrastructure - Managed by the DevOps team Key management: - A master key is generated and stored encrypted in the infrastructure repository[2] - A subkey for signing is generated and stored encrypted in the infrastructure repository[2] and unencrypted on the signing server Signing: - SSHing to a restricted UNIX user with ForceCommand=signing-script - All signing operations are logged - Only signing requests from gemini's WireGuard IP address is allowed [1] https://gitlab.archlinux.org/archlinux/signstar [2] https://gitlab.archlinux.org/archlinux/infrastructure Best regards Kristian Klausen
Re: [arch-dev-public] Starting x86_64_v3 port
On 2022-01-29 15:16, Pierre Schmitz via arch-dev-public wrote: On Sat, Jan 29, 2022 at 2:49 PM Allan McRae via arch-dev-public wrote: Assuming we need people to help the x86_64_v3 port, I would post a news item and have people apply. We have advertised developer positions in the past and received dozens of applications, and readily filled the available positions with quality applicants. They would be brought on as Package Maintainers (once approved on the staff list) with access to [extra] and [community], and have packaging privileges including being added to the keyring. While advertising for x86_64_v3 specific packagers, we should make a list of other packaging areas needing help and recruit for those too. While I would have preferred to gradually have raised the CPU requirements of our main repo (e.g. v2 right now and v3 in a few years), maintaining two x86_64 variants for a transition period might work. Nonetheless we should in work on how to get Arch back to be bleeding edge regardless of this. One aspect might be to reduce the overall amount of packages and get rid of unmaintained software (either by us or upstream). As the vast majority of hardware is v3 already we should consider x86_64 ( Indeed, I would rather that we just move everything to v3 instead of supporting different versions. This seems the simplest way forward. signature.asc Description: PGP signature
Re: [arch-dev-public] Signing enclave
On Sat, Jan 29, 2022 at 06:22:29PM +0100, Kristian Klausen via arch-dev-public wrote: > Signing: > - SSHing to a restricted UNIX user with ForceCommand=signing-script > - All signing operations are logged > - Only signing requests from gemini's WireGuard IP address is allowed Some general thoughts about how we suppose to log these options. We should preferably use a Transparency Log for these as it would give us tamper evidence if our signing enclave gets compromised. For people unfamiliar with Transparency Logs; https://transparency.dev/verifiable-data-structures/ It's the same technology as Certificate Transparency; https://datatracker.ietf.org/doc/html/rfc6962 We have few options here: * Implement our own Trillian Log * Use an existing implementation * Use sigstore[1]. I'm a bit biased towards just using sigstore as it's essentially a continuation of stuff i wrote about in my master thesis. It's also fairly trivial to integrate towards and we don't need to host anything ourself. It's also funded by our own Santiago :) We would be using `rekor-cli` which would give us a lot of this for free.[2] The other option is hosting our own log. This is not super trivial as we want monitors and people replicating the log outside of our own organization to ensure nobody can tamper with the log. The LVFS has opted for such an implementation. I personally think this is an important part of our signing infrastructure. In the future we could have pacman ensure database signatures are present on the Transparency Log which would prevent most of the trivial compromises of our signing enclave. Does anyone have any opinions around this or have any questions? [1]: https://www.sigstore.dev/ [2]: https://docs.sigstore.dev/rekor/CLI -- Morten Linderud PGP: 9C02FF419FECBE16 signature.asc Description: PGP signature
Re: [arch-dev-public] Starting x86_64_v3 port
On Sat, Jan 29, 2022 at 09:35:07AM -0800, Brett Cornwall via arch-dev-public wrote: > On 2022-01-29 15:16, Pierre Schmitz via arch-dev-public wrote: > > On Sat, Jan 29, 2022 at 2:49 PM Allan McRae via arch-dev-public > > wrote: > > > Assuming we need people to help the x86_64_v3 port, I would post a news > > > item and have people apply. We have advertised developer positions in > > > the past and received dozens of applications, and readily filled the > > > available positions with quality applicants. They would be brought on > > > as Package Maintainers (once approved on the staff list) with access to > > > [extra] and [community], and have packaging privileges including being > > > added to the keyring. > > > > > > While advertising for x86_64_v3 specific packagers, we should make a > > > list of other packaging areas needing help and recruit for those too. > > > > While I would have preferred to gradually have raised the CPU > > requirements of our main repo (e.g. v2 right now and v3 in a few > > years), maintaining two x86_64 variants for a transition period might > > work. Nonetheless we should in work on how to get Arch back to be > > bleeding edge regardless of this. One aspect might be to reduce the > > overall amount of packages and get rid of unmaintained software > > (either by us or upstream). > > > > As the vast majority of hardware is v3 already we should consider > > x86_64 ( > when support for such CPUs will be dropped. Personally I would only > > use and test on v3 once it is available. While not all of you might > > agree right now, this is how it will end up eventually, like it did > > with i686. Long story short: we might be looking for people > > maintaining the x86_64 repos and not the v3 ones. > > Indeed, I would rather that we just move everything to v3 instead of > supporting different versions. > > This seems the simplest way forward. This was already discussed in our RFC and rejected. I'd prefer if we stick to what was agreed on there. Multiple points was raised which is worth reading up on. https://gitlab.archlinux.org/archlinux/rfcs/-/merge_requests/2 -- Morten Linderud PGP: 9C02FF419FECBE16 signature.asc Description: PGP signature
Re: [arch-dev-public] Starting x86_64_v3 port
On 2022-01-29 13:11:05 (+0100), Morten Linderud via arch-dev-public wrote: > * Signing enclave > * Better rebuilding tools > * Build automation > * Git migration > > It would make discussions like these completely obsolete. Do we want > v2, v3, v4, v5, v90001? Enable it in a setting and we'd have the > repos. It is a lot of work but it would modernize and make things a > lot simpler for us. FWIW: I fully agree with the above. As tempting as it may seem to "just do it": The time spent on the tooling that would allow us to handle this amount of packages in an automatable fashion lacks greatly. I do not see that "we have had feature X for Y years" can serve as reason for introducing a lot more work or for adding outsiders to the packaging context. The tooling - instead of improving - will further stagnate and be cemented this way. This is something we have to consider and work on. I am afraid that otherwise we are maneuvering ourselves into an unmanagable corner with this. To be more specific: I believe our current best way forward is to continuously propagate our pain point projects that are our main blockers to achieve adding more architectures easily (and in a manageable fashion, without burning out packagers). We need more contributors to improve and focus on these projects and this can and should be done by advertising them more specifically and more regularly. I am not against introducing one or many new architectures, but I am certain that it will take a negative toll on our current group of packagers (even with additional packagers) due to the current set of tooling at hand (as it diverts the attention from improving the tooling). Best, David -- https://sleepmap.de signature.asc Description: PGP signature
Re: [arch-dev-public] Starting x86_64_v3 port
On Sat, 29 Jan 2022 at 15:17, Pierre Schmitz via arch-dev-public < arch-dev-public@lists.archlinux.org> wrote: > On Sat, Jan 29, 2022 at 2:49 PM Allan McRae via arch-dev-public > wrote: > > Assuming we need people to help the x86_64_v3 port, I would post a news > > item and have people apply. We have advertised developer positions in > > the past and received dozens of applications, and readily filled the > > available positions with quality applicants. They would be brought on > > as Package Maintainers (once approved on the staff list) with access to > > [extra] and [community], and have packaging privileges including being > > added to the keyring. > > > > While advertising for x86_64_v3 specific packagers, we should make a > > list of other packaging areas needing help and recruit for those too. > > While I would have preferred to gradually have raised the CPU > requirements of our main repo (e.g. v2 right now and v3 in a few > years), maintaining two x86_64 variants for a transition period might > work. Nonetheless we should in work on how to get Arch back to be > bleeding edge regardless of this. One aspect might be to reduce the > overall amount of packages and get rid of unmaintained software > (either by us or upstream). > > As the vast majority of hardware is v3 already we should consider > x86_64 ( when support for such CPUs will be dropped. Personally I would only > use and test on v3 once it is available. While not all of you might > agree right now, this is how it will end up eventually, like it did > with i686. Long story short: we might be looking for people > maintaining the x86_64 repos and not the v3 ones. > > Greetings, > > Pierre > > -- > Pierre Schmitz, https://pierre-schmitz.com This wouldn't really be too much of an issue if we had proper automation. With automation, this exact problem solves itself to a degree. Surely there will still be specific breakages now and then but the bulk of the burden will go away. We'd even be able to support other targets with ease. However, I realize this will require a lot of upfront infra work before we're there and I'm not sure we should block this proposal on that work. If we don't eventually get good automation (and packages in git), this kinda problem will keep reoccurring. Sadly I don't really have time to work on this right now though I'd love to. Sven
Re: [arch-dev-public] Signing enclave
On 30/1/22 03:22, Kristian Klausen via arch-dev-public wrote: Hi all The lack of package database signing was mentioned yet again and I think it is time to get the "Signing enclave" project rolling. A design was sketched two years ago[1], and based on that design I'm proposing a new design, without a HSM, which should be implementable today. The initial goal would be setting up the necessary infrastructure for us to be able to implement package database signing. Afterwards we can iterate and adapt the solution for more use-cases (ex: releng signing). Hosting: - Hosted on a Hetzner cloud VM as most of our infrastructure - Managed by the DevOps team Key management: - A master key is generated and stored encrypted in the infrastructure repository[2] - A subkey for signing is generated and stored encrypted in the infrastructure repository[2] and unencrypted on the signing server Signing: - SSHing to a restricted UNIX user with ForceCommand=signing-script - All signing operations are logged - Only signing requests from gemini's WireGuard IP address is allowed [1] https://gitlab.archlinux.org/archlinux/signstar [2] https://gitlab.archlinux.org/archlinux/infrastructure Do it! If you get this done soon, I will write the dbscripts changes to automatically build for secondary archtiecture(s) for any package that is uploaded in the primary architecture only. I can not guarantee I will have time in a month... Allan
