On Apr 11, 2025, at 11:39, Mark Millard <mark...@yahoo.com> wrote: > On Apr 7, 2025, at 08:14, Baptiste Daroussin <b...@freebsd.org> wrote: > >> . . . >> the problem we have is the >> performance changes depending on what is happening in parallel on the >> machines. > > In separate list messages I've provided multiple examples > of the time-taking issue that do not depend on what is > running in parallel on the machines, no parallel builds > involved. > > Part of the issue is that there are thousands of examples of > "small build-step time" packages for which the build-depends, > lib-depends, run-depends combination, takes notable time, > given that the total time contribution across those thousands > of package builds is notable overall. > > As stands, mostly it is the early part of "bulk -c -a" avoids > the issue via building packages that have no or few > dependencies. Later "small build-step time" packages tend to > have various dependencies, greatly changing the time scale > for their builds. Few builds are of "large build-step > time" packages (relative to there being 30000+ packages). That > has implications for there being 30000+ packages to build for > "bulk -c -a" or other builds with large numbers of packages > to try to build. > >> which makes the performance issues invisible on local poudriere if you want >> to >> test it on port A or port B, > > I've provided counter examples to that that only involve the > one builder, after the prerequisites have already been built > (same or prior bulk run). > >> if we want to reduce the performance penalty we >> need to be able to make a reproducible case which can then be profiled, to >> know >> where to optimize if needed. > > I've provided examples of such . . . > (time intervals shown are from the aarch64 > Windows Dev Kit 2023 with just the one > builder active) > > www/rt50 > build-depends: 00:00:27->00:08:46 > > devel/py-inline-snapshot@py311 > build-depends: 00:00:01->00:00:55 > run-depends: 00:00:56->00:01:47 > > mail/mailest@nox > build-depends: 00:00:01->00:00:28 > run-depends: 00:00:30->00:00:59 > > devel/dwarves > build-depends: 00:00:05->00:02:23 > lib-depends: 00:02:23->00:02:42
net-mgmt/fastnetmon build-depends: 00:00:03->00:00:42 lib-depends: 00:00:42->00:01:29 (See later below.) > The timings are from the column next to > the Building/Status/Finished column from > using bulk -v , not from the column for > the overall bulk run. > >> I have tried to reproduce each individual case which happen in the ports tree >> and I am not able to reproduce them, so impossible to know where to look at >> exactly. > > Try some of the examples that I've provided? > > There are more examples that I could check > and report non-parallel timings on if you > want. I just picked to report on only a few > initially. > > An example that you might want is my > providing more examples of lib-depends > with non-parallel timings. I took a quick look and quickly ran into: (aarch64 Windows Dev Kit 2023 no-parallel-builders timing again, after having built the prerequisites that had not previously been built) [00:11:37] [01] [00:00:00] Building net-mgmt/fastnetmon | fastnetmon-1.2.8 [00:11:39] [01] [00:00:02] Status net-mgmt/fastnetmon | fastnetmon-1.2.8: check-sanity [00:11:39] [01] [00:00:02] Status net-mgmt/fastnetmon | fastnetmon-1.2.8: pkg-depends [00:11:40] [01] [00:00:03] Status net-mgmt/fastnetmon | fastnetmon-1.2.8: fetch-depends [00:11:40] [01] [00:00:03] Status net-mgmt/fastnetmon | fastnetmon-1.2.8: fetch [00:11:40] [01] [00:00:03] Status net-mgmt/fastnetmon | fastnetmon-1.2.8: checksum [00:11:40] [01] [00:00:03] Status net-mgmt/fastnetmon | fastnetmon-1.2.8: extract-depends [00:11:40] [01] [00:00:03] Status net-mgmt/fastnetmon | fastnetmon-1.2.8: extract [00:11:40] [01] [00:00:03] Status net-mgmt/fastnetmon | fastnetmon-1.2.8: patch-depends [00:11:40] [01] [00:00:03] Status net-mgmt/fastnetmon | fastnetmon-1.2.8: patch [00:11:40] [01] [00:00:03] Status net-mgmt/fastnetmon | fastnetmon-1.2.8: build-depends [00:12:19] [01] [00:00:42] Status net-mgmt/fastnetmon | fastnetmon-1.2.8: lib-depends [00:13:06] [01] [00:01:29] Status net-mgmt/fastnetmon | fastnetmon-1.2.8: configure [00:13:09] [01] [00:01:32] Status net-mgmt/fastnetmon | fastnetmon-1.2.8: build [00:14:20] [01] [00:02:43] Status net-mgmt/fastnetmon | fastnetmon-1.2.8: run-depends [00:14:20] [01] [00:02:43] Status net-mgmt/fastnetmon | fastnetmon-1.2.8: stage [00:14:20] [01] [00:02:43] Status net-mgmt/fastnetmon | fastnetmon-1.2.8: package [00:14:22] [01] [00:02:45] Finished net-mgmt/fastnetmon | fastnetmon-1.2.8: Success (I still have thousands of packages that have not built in the bulk -v -a build activity. The M4 MAX is in use for that.) >> I know what is new and what causes the performance penalty, but not >> which part is causing the super extra penalty on the cluster. > > Various examples reproduce the timing issues > outside the cluster and without the parallel > builds. === Mark Millard marklmi at yahoo.com
