Re: [arch-dev-public] Starting x86_64_v3 port
1) -1 for bringing new packagers onboard just due to the lack of automatic rebuilds. 2) For me 2 of my 3 systems are only x86-64-v2 so I won't be able to test anything seriously. So I'm out of the game if v3 is the new default. Both systems are fast enough for many more years. So there's no plan to replace them. 3) Anything that just needs simple rebuilds should be done in 2022 without any manpower. What's the plan with the new v3 port if I build only v1 packages? Will v3 stay at the old version or will the new v1 package also be pushed to v3 just lacking the better optimization until some new v3 rebuild brings it back? This way v3 would go forth and back all the time and should be avoided. But falling behind v1 for some time is also not desired. 4) Is anybody actually running ALHP or working on it? https://alhp.harting.dev/packages.html [5) Why don't we care about ARM and OpenRISC at all? Aren't we riding pretty dead horses here just for very minor improvements?] -Andy pgpcIJ1xweR5i.pgp Description: Digitale Signatur von OpenPGP
[arch-dev-public] Debug packages for Arch Linux
I'm very happy to announce that debug packages in Arch Linux has been deployed :) This work began after FOSDEM 2020 and was announced in November 2020. However because of time constraints it took quite a while before we managed to deploy it. Currently we have a debuginfod service which is capable of delivering source listings and debug symbols to users with gdb, delve and other debuggers. This can be enabled by installing the `debuginfod` package and setting the environment variable `DEBUGINFOD_URLS="https://debuginfod.archlinux.org"`. The `debuginfod` package is also going to be providing this variable :) The debug package repositories themself are not publicly accessible nor distributed to our mirrors. This is because there is a general concern around the repository size increase. The goal is to take a look at distributing them and/or making them accessible in the future. Packages are being rebuilt for debug symbols where deemed needed by the packager and I suspect we should have a discussion around enabling the debug option by default or not. However see the section below around creating debug packages as there are some caveats of the current approach. I have added some links to the work that has been done to accomplish all of this at the bottom of this email. For general information about the instance one can look at the service itself: https://debuginfod.archlinux.org/ Previous emails on this topic: * https://lists.archlinux.org/pipermail/arch-dev-public/2020-November/030222.html * https://lists.archlinux.org/pipermail/arch-dev-public/2021-December/030583.html # archlinux.org news announcement Suggestion for the news announcement on archweb. I'll post it in a day or two unless there are any objections. Subject: Debug packages and debuginfod We are very happy to announce that debug packages are now available in Arch Linux. Debug symbols and source listing are provided through our debuginfod instance which can be utilized by debuggers such as gdb and delve. In the future we are planning to make these repositories public on our mirrors. For more information on this please visit the [archwiki](https://wiki.archlinux.org/title/Debuginfod) article, Or the debuginfod service. https://debuginfod.archlinux.org/ # Creating debug packages Currently with pacman 6.0.1-3 debug package only works on C/C++ projects. This is mostly because pacman utilizes a fairly ugly awk hack to extract sources and if it encounters a binary from Go or Rust (as an example) it is simply unable to deal with them and produces invalid packages. This is fixed with a patch i wrote which replaces the AWK hack with debugedit. https://gitlab.archlinux.org/pacman/pacman/-/commit/ae2f506ddfd11d9becda7216033fe1b159536982 Until this patch is backported, or a pacman release is done, I would advise us to not build debug packages for anything that isn't using gcc/clangd. Saying that, I'm unsure how we should be documenting debug packages across our distro. They probably belong in package guidelines sections of each package, or we make a common wiki page for this. Generally Rust needs a environment variable and Golang needs a bit of work to disable decompressed DWARF headers. We can probably work something out as we go along, currently I think we should try enable debug packages on our [core] packages. When we have pacman with the above patch I can try compile a list of available methods to get debug packages from Rust, Go and other languages. Obviously I'd need some help with that :) Cheers! -- Morten Linderud PGP: 9C02FF419FECBE16 # infrastructure https://gitlab.archlinux.org/archlinux/infrastructure/-/merge_requests/168 # dbscripts https://gitlab.archlinux.org/archlinux/dbscripts/-/merge_requests/21 https://gitlab.archlinux.org/archlinux/dbscripts/-/merge_requests/33 # devtools https://gitlab.archlinux.org/archlinux/devtools/-/merge_requests/78 signature.asc Description: PGP signature
Re: [arch-dev-public] Starting x86_64_v3 port
On Sun, Jan 30, 2022 at 12:02:46PM +0100, Andreas Radke via arch-dev-public wrote: > > [5) Why don't we care about ARM and OpenRISC at all? Aren't we riding > pretty dead horses here just for very minor improvements?] ALARM has refused to join the project because of our ancient tooling. RISC is being worked on by Felix. https://github.com/felixonmars/archriscv-packages -- Morten Linderud PGP: 9C02FF419FECBE16 signature.asc Description: PGP signature
Re: [arch-dev-public] Starting x86_64_v3 port
On 2022-01-30 12:13:45 (+0100), Morten Linderud via arch-dev-public wrote: > ALARM has refused to join the project because of our ancient tooling. Case in point. :) -- https://sleepmap.de signature.asc Description: PGP signature
Re: [arch-dev-public] Debug packages for Arch Linux
On 30.01.2022 12.08, Morten Linderud via arch-dev-public wrote:
I'm very happy to announce that debug packages in Arch Linux has been deployed
:)
This work began after FOSDEM 2020 and was announced in November 2020. However
because of time constraints it took quite a while before we managed to deploy
it.
Currently we have a debuginfod service which is capable of delivering source
listings and debug symbols to users with gdb, delve and other debuggers. This
can be enabled by installing the `debuginfod` package and setting the
environment variable `DEBUGINFOD_URLS="https://debuginfod.archlinux.org"`. The
`debuginfod` package is also going to be providing this variable :)
The debug package repositories themself are not publicly accessible nor
distributed to our mirrors. This is because there is a general concern around
the repository size increase. The goal is to take a look at distributing them
and/or making them accessible in the future.
We have three sponsored machines, which are used as package mirrors and
archive mirrors[1].
At the time of writing they have 14TB available storage each, so we
could mirror the debug packages to them.
[1] {america,asia,europe}.{archive,mirror}.pkgbuild.com
- Kristian
Re: [arch-dev-public] openssl 3.0
So I just build the 437 packages (pkgbase) and let my computer compile for just 25 hours. The initial results can be seen here https://md.archlinux.org/s/t8HOyhNOi Currently there are 27 packages in [core]/[extra] and 92 in [community] that do not build. I did not check the logs for every package yet, but I guess there are these categories: * package does not build regardless of OpenSSL (e.g. unavailable sources, checksum mismatch, issues due to LTO, ...) * packages also links to a not yet update package that still uses openssl-1.1 * packages are actually incompatible with OpenSSL 3.0 I'll need some help with: * Document why a package fails (complete build logs are attached to the document linked above) * Create a todo list for packages that are broken for other reasons and fix them * Review the legacy openssl-1.1 package and check if this approach is valid. (last time we patched versiond symbols in 1.0 which I did not apply here) See https://github.com/archlinux/svntogit-packages/tree/packages/openssl-1.1/trunk and https://github.com/archlinux/svntogit-packages/tree/packages/openssl-1.0/trunk * Fix the incompatible packages and as a last resort link to 1.1 PS: if there is a tool that is able to rebuild and install packages in the correct order (not by explicit dependency but by so lib links), let me know. Greetings, Pierre On Fri, Jan 28, 2022 at 9:41 AM Maxime Gauduin via arch-dev-public wrote: > > On Thu, 2022-01-27 at 16:47 +0100, Christian Hesse via arch-dev-public > wrote: > > Pierre Schmitz via arch-dev-public > > on > > Sun, 2022/01/23 12:50: > > > Next steps: > > > 1) Let's agree on a time window where no other rebuild can take > > > place > > > within our staging repos. How about at least the first two weeks in > > > February? > > > > I guess the ffmpeg 5.0 will be blocking for some time... > > Not necessarily. There are too many packages that don't build, I will > maintain a temporary ffmpeg4.4 package to get this todo out quickly. > Should have time for this over the weekend. > > Cheers, > -- > Maxime -- Pierre Schmitz, https://pierre-schmitz.com
Re: [arch-dev-public] Starting x86_64_v3 port
On 30/1/22 21:13, Morten Linderud via arch-dev-public wrote: On Sun, Jan 30, 2022 at 12:02:46PM +0100, Andreas Radke via arch-dev-public wrote: [5) Why don't we care about ARM and OpenRISC at all? Aren't we riding pretty dead horses here just for very minor improvements?] ALARM has refused to join the project because of our ancient tooling. RISC is being worked on by Felix. Not 100% accurate. When I asked there was significant issues around sponsorship and ownership of the arm machines used as a buildfarm.
Re: [arch-dev-public] Starting x86_64_v3 port
On Sat, 2022-01-29 at 20:17 +0100, Sven-Hendrik Haase via arch-dev- public wrote: > On Sat, 29 Jan 2022 at 15:17, Pierre Schmitz via arch-dev-public < > arch-dev-public@lists.archlinux.org> wrote: > > > On Sat, Jan 29, 2022 at 2:49 PM Allan McRae via arch-dev-public > > wrote: > > > Assuming we need people to help the x86_64_v3 port, I would post > > > a news > > > item and have people apply. We have advertised developer > > > positions in > > > the past and received dozens of applications, and readily filled > > > the > > > available positions with quality applicants. They would be > > > brought on > > > as Package Maintainers (once approved on the staff list) with > > > access to > > > [extra] and [community], and have packaging privileges including > > > being > > > added to the keyring. > > > > > > While advertising for x86_64_v3 specific packagers, we should > > > make a > > > list of other packaging areas needing help and recruit for those > > > too. > > > > While I would have preferred to gradually have raised the CPU > > requirements of our main repo (e.g. v2 right now and v3 in a few > > years), maintaining two x86_64 variants for a transition period > > might > > work. Nonetheless we should in work on how to get Arch back to be > > bleeding edge regardless of this. One aspect might be to reduce the > > overall amount of packages and get rid of unmaintained software > > (either by us or upstream). > > > > As the vast majority of hardware is v3 already we should consider > > x86_64 ( > date > > when support for such CPUs will be dropped. Personally I would only > > use and test on v3 once it is available. While not all of you might > > agree right now, this is how it will end up eventually, like it did > > with i686. Long story short: we might be looking for people > > maintaining the x86_64 repos and not the v3 ones. > > > > Greetings, > > > > Pierre > > > > -- > > Pierre Schmitz, https://pierre-schmitz.com > > > This wouldn't really be too much of an issue if we had proper > automation. > With automation, this exact problem solves itself to a degree. Surely > there > will still be specific breakages now and then but the bulk of the > burden > will go away. We'd even be able to support other targets with ease. > > However, I realize this will require a lot of upfront infra work > before > we're there and I'm not sure we should block this proposal on that > work. > > If we don't eventually get good automation (and packages in git), > this > kinda problem will keep reoccurring. Sadly I don't really have time > to work > on this right now though I'd love to. > > Sven We already have 2 working automated build tools, that I know of, Evangelos' and mine [0]. I'm sure we can figure something out fairly quickly, unless we'd rather go with some Gitlab CI now that we have one. It would probably make more sense to go that route, but I've already fitted several Gitlab instances with a Buildbot CI, I find it more flexible and it also works wonders. [0] https://github.com/alucryd/archbuild Cheers, -- Maxime signature.asc Description: This is a digitally signed message part
Re: [arch-dev-public] Starting x86_64_v3 port
On Sun, Jan 30, 2022 at 02:00:51PM +0100, Maxime Gauduin via arch-dev-public wrote: > On Sat, 2022-01-29 at 20:17 +0100, Sven-Hendrik Haase via arch-dev- > > > > This wouldn't really be too much of an issue if we had proper automation. > > With automation, this exact problem solves itself to a degree. Surely there > > will still be specific breakages now and then but the bulk of the burden > > will go away. We'd even be able to support other targets with ease. > > > > However, I realize this will require a lot of upfront infra work before > > we're there and I'm not sure we should block this proposal on that work. > > > > If we don't eventually get good automation (and packages in git), this kinda > > problem will keep reoccurring. Sadly I don't really have time to work on > > this right now though I'd love to. > > > > Sven > > We already have 2 working automated build tools, that I know of, > Evangelos' and mine [0]. I'm sure we can figure something out fairly > quickly, unless we'd rather go with some Gitlab CI now that we have > one. It would probably make more sense to go that route, but I've > already fitted several Gitlab instances with a Buildbot CI, I find it > more flexible and it also works wonders. I think having buildbot would be more flexible for such an infrastructure then piggy back on the pipeline system for gitlab. It sometimes feels too restrictive being tied to git repositories and buildbot is a *lot* more flexible when it comes to it. It might also be worth a consideration that the build infrastructure would be completely seperate from our git forge. This would make it easier if we want to move VCS/forge in the future. -- Morten Linderud PGP: 9C02FF419FECBE16 signature.asc Description: PGP signature
Re: [arch-dev-public] Debug packages for Arch Linux
On Sun, Jan 30, 2022 at 08:49:29PM +0100, sL1pKn07 SpinFlo wrote: > hi > > thanks for the work! Thanks! > one question. This thing works, for example, if I build one of the packages > with debug package in the arch debuginfod site with my custom flag for > disable an option/feature. the debug package stored in the debuginfod site > still works? or i need to create a debug package locally? The buildid would be different so our debuginfo won't work with your packages. You can check with: readelf -n /bin/ls | grep -A4 build.id > If I need to build a local debug package and I want to use it > with debuginfod, is it possible add the link of the custom debuginfod site > in the DEBUGINFOD_URLS variable? how? is , separate? or ;? Yes, space is the seperator. See https://man.archlinux.org/man/debuginfod-find.1#ENVIRONMENT_VARIABLES You can also add a file under /etc/debuginfod/*.urls like we did with our `debuginfod` package. -- Morten Linderud PGP: 9C02FF419FECBE16 signature.asc Description: PGP signature
Re: [arch-dev-public] Debug packages for Arch Linux
On Sun, Jan 30, 2022 at 08:59:34PM +0100, sL1pKn07 SpinFlo wrote: > El dom, 30 ene 2022 a las 20:54, Morten Linderud () > escribió: > > > On Sun, Jan 30, 2022 at 08:49:29PM +0100, sL1pKn07 SpinFlo wrote: > > > hi > > > > > > thanks for the work! > > > > Thanks! > > > > > one question. This thing works, for example, if I build one of the > > packages > > > with debug package in the arch debuginfod site with my custom flag for > > > disable an option/feature. the debug package stored in the debuginfod > > site > > > still works? or i need to create a debug package locally? > > > > The buildid would be different so our debuginfo won't work with your > > packages. > > > > You can check with: readelf -n /bin/ls | grep -A4 build.id > > > > > If I need to build a local debug package and I want to use it > > > with debuginfod, is it possible add the link of the custom debuginfod > > site > > > in the DEBUGINFOD_URLS variable? how? is , separate? or ;? > > > > Yes, space is the seperator. See > > https://man.archlinux.org/man/debuginfod-find.1#ENVIRONMENT_VARIABLES > > > > You can also add a file under /etc/debuginfod/*.urls like we did with our > > `debuginfod` package. > > > > -- > > Morten Linderud > > PGP: 9C02FF419FECBE16 > > > > Hi > > thanks for the reply > > is here documented the file/directory structure of debuginfod "repo"? > > greetings I recommend reading the documentation frankly. debuginfod just recursively scans a set of directories which is polling our package repositories. The implementation is linked in the email. https://gitlab.archlinux.org/archlinux/infrastructure/-/merge_requests/168/diffs?commit_id=b7202bd6da295654171d890c4d139ddae96ace54 -- Morten Linderud PGP: 9C02FF419FECBE16 signature.asc Description: PGP signature
Re: [arch-dev-public] Debug packages for Arch Linux
(Sorry for the offlist spam. I'm too quick and forget to remove the CC :c) -- Morten Linderud PGP: 9C02FF419FECBE16
