On 2025-07-29 09:34, Mark Millard wrote:
On Jul 26, 2025, at 11:46, Mark Millard <[email protected]> wrote:

The other builders in the "bulk -Ca" run till are operating.

# poudriere version
poudriere-git-3.4.99.20250601

Extractions of appearently related information follow . . .

# poudriere status -b
=>> [main-ZNV4-bulk_a-alt] [2025-07-24_17h17m32s] [parallel_build] Time: 1D:14:22:35 Queued: 36640 Inspected: 0 Ignored: 361 Built: 31142 Failed: 141 Skipped: 619 Fetched: 0 Remaining: 4377
. . .
[03] 05:47:01 lang/dotnet8 | dotnet8-8.0.6_4 build 05:45:28 4.81 GiB 0% 0.5% [04] 06:39:07 lang/dotnet | dotnet-9.0.6 build_port_done 05:59:17 4.75 GiB
. . .

The poudriere-devel output shows:

. . .
[1D:07:43:30] [04] [00:00:00] Building   lang/dotnet | dotnet-9.0.6
. . .
[1D:08:23:20] [04] [00:39:50] Finished lang/dotnet | dotnet-9.0.6: Success
. . .
[1D:08:35:36] [03] [00:00:00] Building lang/dotnet8 | dotnet8-8.0.6_4
. . .

# ls -lodT /usr/local/poudriere/data/.m/main-ZNV4-bulk_a-alt/04/packages/All/dotnet-9.0.6.pkg -rw-r--r-- 1 root wheel uarch 758491155 Jul 26 01:40:52 2025 /usr/local/poudriere/data/.m/main-ZNV4-bulk_a-alt/04/packages/All/dotnet-9.0.6.pkg

# find /usr/local/poudriere/data/packages/main-ZNV4-bulk_a-alt/.building/.npkg/ -print
/usr/local/poudriere/data/packages/main-ZNV4-bulk_a-alt/.building/.npkg/
/usr/local/poudriere/data/packages/main-ZNV4-bulk_a-alt/.building/.npkg/dotnet-9.0.6
/usr/local/poudriere/data/packages/main-ZNV4-bulk_a-alt/.building/.npkg/dotnet-9.0.6/All

# find /usr/local/poudriere/data/packages/main-ZNV4-bulk_a-alt/ -name '*dotnet-9*' -print
/usr/local/poudriere/data/packages/main-ZNV4-bulk_a-alt/.building/All/dotnet-9.0.6.pkg
/usr/local/poudriere/data/packages/main-ZNV4-bulk_a-alt/.building/.npkg/dotnet-9.0.6

# ls -loaT /usr/local/poudriere/data/.m/main-ZNV4-bulk_a-alt/04/wrkdirs/
total 2
drwx------   2 root wheel uarch    2 Jul 26 01:40:59 2025 .
drwxr-xr-x  26 root wheel -     1984 Jul 26 01:33:37 2025 ..

# ls -loaT /usr/local/poudriere/data/.m/main-ZNV4-bulk_a-alt/04/
total 5082
drwxr-xr-x   26 root wheel -      1984 Jul 26 01:33:37 2025 .
drwx--x--x   15 root wheel -       832 Jul 24 17:19:30 2025 ..
-rw-r--r--    1 root wheel -         3 Jul 24 17:17:32 2025 .cpignore
-rw-r--r-- 1 root wheel - 0 Jul 26 01:01:03 2025 .need_rollback
drwxr-xr-x    3 root wheel uarch     3 Jul 26 01:33:37 2025 .npkg
-rw-r--r-- 1 root wheel - 0 Jul 26 01:33:37 2025 .npkg_mounted
drwxr-xr-x    4 root wheel -       192 Jul 24 17:19:31 2025 .p
-rw-r--r--    2 root wheel uarch   495 Aug 20 10:45:06 2023 .profile
-rw-r--r-- 1 root wheel - 68 Jul 26 01:01:03 2025 .tmpfs_blacklist_dir
-r--r--r--    1 root wheel uarch  6070 Jun 28 23:04:06 2025 COPYRIGHT
drwxr-xr-x    2 root wheel -      3008 Jun 28 23:03:57 2025 bin
drwxr-xr-x   14 root wheel -      4672 Jun 28 23:04:06 2025 boot
drwxr-xr-x    3 root wheel -        64 Jul 24 17:17:32 2025 compat
dr-xr-xr-x   11 root wheel -       512 Jul 26 01:01:03 2025 dev
drwxr-xr-x  485 root wheel uarch 32439 Jul 25 14:15:42 2025 distfiles
drwxr-xr-x   31 root wheel -      6784 Jul 26 01:01:57 2025 etc
lrwxr-xr-x 1 root wheel - 8 Jul 24 17:17:32 2025 home -> usr/home
drwxr-xr-x    4 root wheel -      4992 Jun 28 23:03:56 2025 lib
drwxr-xr-x    3 root wheel -       512 Jun 26 22:28:15 2025 libexec
drwxr-xr-x    2 root wheel -         0 Dec  4 14:52:10 2021 media
drwxr-xr-x    2 root wheel -         0 Dec  4 14:52:10 2021 mnt
drwxr-xr-x    2 root wheel -         0 Dec  4 14:52:10 2021 net
drwxr-xr-x    2 root wheel -         0 Jul 24 17:17:32 2025 overlays
drwxr-xr-x    5 root wheel uarch     8 Jul 24 17:19:23 2025 packages
drwxr-xr-x 3 root wheel - 64 Jul 26 01:01:03 2025 portdistfiles
dr-xr-xr-x    1 root wheel -         0 Jul 26 08:55:43 2025 proc
drwxr-xr-x    2 root wheel uarch   152 Jun 28 23:03:56 2025 rescue
drwxr-x---    3 root wheel -       384 Jul 24 17:17:32 2025 root
drwxr-xr-x    2 root wheel -      9536 Jun 28 23:03:58 2025 sbin
drwxrwxrwt   71 root wheel -      4480 Jul 26 01:48:28 2025 tmp
drwxr-xr-x   16 root wheel -       896 Jul 24 17:17:32 2025 usr
drwxr-xr-x   84 root wheel -      1408 Dec  4 14:52:10 2021 var
drwx------    2 root wheel uarch     2 Jul 26 01:40:59 2025 wrkdirs

# find /tmp/ -name '*lock-*' -exec ls -lodT {} \;
-rw-r--r-- 1 root wheel uarch 4 Jul 26 11:34:26 2025 /tmp/poudriere.LZmiUvjmEy/lock-main-ZNV4-bulk_a-alt-update_stats.pid -rw-r--r-- 1 root wheel uarch 0 Jul 24 17:17:34 2025 /tmp/poudriere.LZmiUvjmEy/lock-main-ZNV4-bulk_a-alt-update_stats.flock

# more /tmp/poudriere.LZmiUvjmEy/lock-main-ZNV4-bulk_a-alt-update_stats.pid
3615

From "ps -alxd":

. . .
0 3615 3158 15 59 0 16868 4152 wait S 1 22:19.56 | | | |-- sh: poudriere[main-ZNV4-bulk_a-alt]: html_json_main (sh) 0 56410 3615 5 89 0 14532 3728 - R 1 0:00.75 | | | | |-- /usr/bin/awk -v mini=yes -f /usr/bin/awk main-ZNV4-bulk_a-alt/2025-07-24_17h17m32s/.poudriere.builders main- 0 56411 3615 29 59 0 14532 2752 piperd S 1 0:00.00 | | | | |-- /usr/bin/awk ORS=""; {print} END {print "\\n"} 0 56412 3615 10 59 0 14108 2236 piperd S 1 0:00.00 | | | | |-- /usr/bin/sed -e s/,\\([]}]\\)/\\1/g 0 56413 3615 19 59 0 16868 4144 piperd S 1 0:00.00 | | | | `-- sh: poudriere[main-ZNV4-bulk_a-alt]: html_json_main (sh)
. . .

Doing "ps -alxd" does not show a process tree for dotnet-9.0.6 but does for dotnet8-8.0.6_4 :

UID PID PPID C PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND
. . .
0 28395 3158 18 59 0 23524 11608 select I 1 0:00.04 | | | |-- sh: poudriere[main-ZNV4-bulk_a-alt][03]: build_pkg (dotnet8-8.0.6_4) (sh) 0 55498 28395 25 59 0 23524 11604 wait I 1 0:00.00 | | | | `-- sh: poudriere[main-ZNV4-bulk_a-alt][03]: build_pkg (dotnet8-8.0.6_4) (sh) 0 55499 55498 21 59 0 14216 3784 wait IsJ - 0:00.02 | | | | `-- /usr/bin/make -C /usr/ports/lang/dotnet8 build 0 55524 55499 29 59 0 14668 3304 wait IJ - 0:00.00 | | | | `-- /bin/sh -e /tmp/makedskEnT 0 55525 55524 7 59 0 16204 4920 wait IJ - 0:00.01 | | | | `-- bash ./build.sh --release-manifest release.json --clean-while-building --with-sdk /wrkdirs/usr/ports 0 56836 55525 4 59 0 406855036 195184 uwait IJ - 0:30.89 | | | | `-- /wrkdirs/usr/ports/lang/dotnet8/work/bootstrap_sdk/dotnet msbuild /wrkdirs/usr/ports/lang/dotnet8/ 0 5126 56836 5 3 0 14748 3512 wait IJ - 0:00.00 | | | | `-- /bin/sh -c export LANG=en_US.UTF-8; export LC_ALL=en_US.UTF-8; . /tmp/MSBuildTemproot/tmpc8ecf97 0 5127 5126 4 11 0 16284 5032 wait IJ - 0:00.00 | | | | `-- bash /wrkdirs/usr/ports/lang/dotnet8/work/dotnet-8.0.6/src/xdt/build.sh --ci --configuration R 0 5130 5127 1 59 0 16284 5376 wait IJ - 0:00.02 | | | | `-- bash /wrkdirs/usr/ports/lang/dotnet8/work/dotnet-8.0.6/src/xdt/eng/common/build.sh --pack -- 0 5330 5130 0 59 0 406784004 105688 uwait IJ - 0:00.55 | | | | `-- /wrkdirs/usr/ports/lang/dotnet8/work/bootstrap_sdk//dotnet msbuild /m /nologo /clp:Summary 0 5473 5330 22 59 0 14748 3528 wait IJ - 0:00.00 | | | | `-- /bin/sh -c export LANG=en_US.UTF-8; export LC_ALL=en_US.UTF-8; . /tmp/MSBuildTemproot/tm 0 5474 5473 28 59 0 406867320 195692 uwait IJ - 0:06.62 | | | | `-- /wrkdirs/usr/ports/lang/dotnet8/work/bootstrap_sdk//dotnet msbuild /m /nologo /clp:Sum 0 5684 5474 2 9 0 406769428 112740 uwait IJ - 0:02.97 | | | | |-- /wrkdirs/usr/ports/lang/dotnet8/work/bootstrap_sdk/dotnet /wrkdirs/usr/ports/lang/do 0 5757 5474 9 59 0 406801372 139216 uwait IJ - 0:01.45 | | | | `-- /wrkdirs/usr/ports/lang/dotnet8/work/bootstrap_sdk/dotnet /wrkdirs/usr/ports/lang/do

A "ww"-added ps command output has:

UID PID PPID C PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAND
. . .
0 28395 3158 18 59 0 23524 11608 select I 1 0:00.04 | | | |-- sh: poudriere[main-ZNV4-bulk_a-alt][03]: build_pkg (dotnet8-8.0.6_4) (sh) 0 55498 28395 25 59 0 23524 11604 wait I 1 0:00.00 | | | | `-- sh: poudriere[main-ZNV4-bulk_a-alt][03]: build_pkg (dotnet8-8.0.6_4) (sh) 0 55499 55498 21 59 0 14216 3784 wait IsJ - 0:00.02 | | | | `-- /usr/bin/make -C /usr/ports/lang/dotnet8 build 0 55524 55499 29 59 0 14668 3304 wait IJ - 0:00.00 | | | | `-- /bin/sh -e /tmp/makedskEnT 0 55525 55524 7 59 0 16204 4920 wait IJ - 0:00.01 | | | | `-- bash ./build.sh --release-manifest release.json --clean-while-building --with-sdk /wrkdirs/usr/ports/lang/dotnet8/work/bootstrap_sdk --with-packages /wrkdirs/usr/ports/lang/dotnet8/work/packages -- -maxcpucount:3 0 56836 55525 4 59 0 406855036 195184 uwait IJ - 0:30.89 | | | | `-- /wrkdirs/usr/ports/lang/dotnet8/work/bootstrap_sdk/dotnet msbuild /wrkdirs/usr/ports/lang/dotnet8/work/dotnet-8.0.6/build.proj -bl:/wrkdirs/usr/ports/lang/dotnet8/work/dotnet-8.0.6/artifacts/log/Debug/Build_0726085441.binlog -flp:LogFile=/wrkdirs/usr/ports/lang/dotnet8/work/dotnet-8.0.6/artifacts/logs/Build_0726085441.log -flp:v=detailed -p:CleanWhileBuilding=true -p:CustomPrebuiltSourceBuiltPackagesPath=/wrkdirs/usr/ports/lang/dotnet8/work/packages -maxcpucount:3 0 5126 56836 5 3 0 14748 3512 wait IJ - 0:00.00 | | | | `-- /bin/sh -c export LANG=en_US.UTF-8; export LC_ALL=en_US.UTF-8; . /tmp/MSBuildTemproot/tmpc8ecf970b04a4faea5680a0118915aaa.exec.cmd 0 5127 5126 4 11 0 16284 5032 wait IJ - 0:00.00 | | | | `-- bash /wrkdirs/usr/ports/lang/dotnet8/work/dotnet-8.0.6/src/xdt/build.sh --ci --configuration Release --restore --build --pack --publish -bl /p:ArcadeBuildFromSource=true /p:CopyWipIntoInnerSourceBuildRepo=true /p:DotNetBuildOffline=true /p:CopySrcInsteadOfClone=true /p:DotNetPackageVersionPropsPath=/wrkdirs/usr/ports/lang/dotnet8/work/dotnet-8.0.6/artifacts/obj/x64/Release/PackageVersions.xdt.props /p:AdditionalSourceBuiltNupkgCacheDir=/wrkdirs/usr/ports/lang/dotnet8/work/dotnet-8.0.6/artifacts/obj/x64/Release/blob-feed/packages/ /p:ReferencePackageNupkgCacheDir=/wrkdirs/usr/ports/lang/dotnet8/work/dotnet-8.0.6/prereqs/packages/reference/ /p:PreviouslySourceBuiltNupkgCacheDir=/wrkdirs/usr/ports/lang/dotnet8/work/packages/ /p:SourceBuildUseMonoRuntime= --nodereuse false 0 5130 5127 1 59 0 16284 5376 wait IJ - 0:00.02 | | | | `-- bash /wrkdirs/usr/ports/lang/dotnet8/work/dotnet-8.0.6/src/xdt/eng/common/build.sh --pack --build --restore --test --ci --configuration Release --restore --build --pack --publish -bl /p:ArcadeBuildFromSource=true /p:CopyWipIntoInnerSourceBuildRepo=true /p:DotNetBuildOffline=true /p:CopySrcInsteadOfClone=true /p:DotNetPackageVersionPropsPath=/wrkdirs/usr/ports/lang/dotnet8/work/dotnet-8.0.6/artifacts/obj/x64/Release/PackageVersions.xdt.props /p:AdditionalSourceBuiltNupkgCacheDir=/wrkdirs/usr/ports/lang/dotnet8/work/dotnet-8.0.6/artifacts/obj/x64/Release/blob-feed/packages/ /p:ReferencePackageNupkgCacheDir=/wrkdirs/usr/ports/lang/dotnet8/work/dotnet-8.0.6/prereqs/packages/reference/ /p:PreviouslySourceBuiltNupkgCacheDir=/wrkdirs/usr/ports/lang/dotnet8/work/packages/ /p:SourceBuildUseMonoRuntime= --nodereuse false 0 5330 5130 0 59 0 406784004 105688 uwait IJ - 0:00.55 | | | | `-- /wrkdirs/usr/ports/lang/dotnet8/work/bootstrap_sdk//dotnet msbuild /m /nologo /clp:Summary /v:minimal /nr:false /warnaserror /p:TreatWarningsAsErrors=true /p:ContinuousIntegrationBuild=true /wrkdirs/usr/ports/lang/dotnet8/work/dotnet-8.0.6/artifacts/source-built-sdks/Microsoft.DotNet.Arcade.Sdk/tools/Build.proj /bl:"/wrkdirs/usr/ports/lang/dotnet8/work/dotnet-8.0.6/src/xdt/artifacts/log/Release/Build.binlog" /p:Configuration=Release /p:RepoRoot=/wrkdirs/usr/ports/lang/dotnet8/work/dotnet-8.0.6/src/xdt/ /p:Restore=true /p:Build=true /p:Rebuild=false /p:Test=true /p:Pack=true /p:IntegrationTest=false /p:PerformanceTest=false /p:Sign=false /p:Publish=true /p:ArcadeBuildFromSource=true /p:CopyWipIntoInnerSourceBuildRepo=true /p:DotNetBuildOffline=true /p:CopySrcInsteadOfClone=true /p:DotNetPackageVersionPropsPath=/wrkdirs/usr/ports/lang/dotnet8/work/dotnet-8.0.6/artifacts/obj/x64/Release/PackageVersions.xdt.props /p:AdditionalSourceBuiltNupkgCacheDir=/wrkdirs/usr/ports/lang/dotnet8/work/dotnet-8.0.6/artifacts/obj/x64/Release/blob-feed/packages/ /p:ReferencePackageNupkgCacheDir=/wrkdirs/usr/ports/lang/dotnet8/work/dotnet-8.0.6/prereqs/packages/reference/ /p:PreviouslySourceBuiltNupkgCacheDir=/wrkdirs/usr/ports/lang/dotnet8/work/packages/ /p:SourceBuildUseMonoRuntime= -logger:/wrkdirs/usr/ports/lang/dotnet8/work/dotnet-8.0.6/artifacts/source-built-sdks/Microsoft.DotNet.Arcade.Sdk/tools/net8.0/Microsoft.DotNet.ArcadeLogging.dll 0 5473 5330 22 59 0 14748 3528 wait IJ - 0:00.00 | | | | `-- /bin/sh -c export LANG=en_US.UTF-8; export LC_ALL=en_US.UTF-8; . /tmp/MSBuildTemproot/tmp5c13d207e7a4487188f01bcfbfeb74bc.exec.cmd 0 5474 5473 28 59 0 406867320 195692 uwait IJ - 0:06.62 | | | | `-- /wrkdirs/usr/ports/lang/dotnet8/work/bootstrap_sdk//dotnet msbuild /m /nologo /clp:Summary /v:minimal /nr:false /warnaserror /p:TreatWarningsAsErrors=true /p:ContinuousIntegrationBuild=true /wrkdirs/usr/ports/lang/dotnet8/work/dotnet-8.0.6/artifacts/source-built-sdks/Microsoft.DotNet.Arcade.Sdk/tools/Build.proj /bl:/wrkdirs/usr/ports/lang/dotnet8/work/dotnet-8.0.6/src/xdt/artifacts/log/Release/Build.binlog /p:Configuration=Release /p:RepoRoot=/wrkdirs/usr/ports/lang/dotnet8/work/dotnet-8.0.6/src/xdt/ /p:Restore=true /p:Build=true /p:Rebuild=false /p:Test=true /p:Pack=true /p:IntegrationTest=false /p:PerformanceTest=false /p:Sign=false /p:Publish=true /p:ArcadeBuildFromSource=true /p:CopyWipIntoInnerSourceBuildRepo=true /p:DotNetBuildOffline=true /p:CopySrcInsteadOfClone=true /p:DotNetPackageVersionPropsPath=/wrkdirs/usr/ports/lang/dotnet8/work/dotnet-8.0.6/artifacts/obj/x64/Release/PackageVersions.xdt.props /p:AdditionalSourceBuiltNupkgCacheDir=/wrkdirs/usr/ports/lang/dotnet8/work/dotnet-8.0.6/artifacts/obj/x64/Release/blob-feed/packages/ /p:ReferencePackageNupkgCacheDir=/wrkdirs/usr/ports/lang/dotnet8/work/dotnet-8.0.6/prereqs/packages/reference/ /p:PreviouslySourceBuiltNupkgCacheDir=/wrkdirs/usr/ports/lang/dotnet8/work/packages/ /p:SourceBuildUseMonoRuntime= -logger:/wrkdirs/usr/ports/lang/dotnet8/work/dotnet-8.0.6/artifacts/source-built-sdks/Microsoft.DotNet.Arcade.Sdk/tools/net8.0/Microsoft.DotNet.ArcadeLogging.dll /p:ArcadeInnerBuildFromSource=true /p:DotNetBuildFromSource=true /p:RepoRoot=/wrkdirs/usr/ports/lang/dotnet8/work/dotnet-8.0.6/src/xdt/artifacts/source-build/self/src/ /p:ArtifactsDir=/wrkdirs/usr/ports/lang/dotnet8/work/dotnet-8.0.6/src/xdt/artifacts/source-build/self/src/artifacts/ /bl:/wrkdirs/usr/ports/lang/dotnet8/work/dotnet-8.0.6/src/xdt/artifacts/source-build/self/src/artifacts/sourcebuild.binlog /p:ContinuousIntegrationBuild=true /p:SourceBuildOutputDir=/wrkdirs/usr/ports/lang/dotnet8/work/dotnet-8.0.6/src/xdt/artifacts/source-build/ /p:SourceBuiltBlob 0 5684 5474 2 9 0 406769428 112740 uwait IJ - 0:02.97 | | | | |-- /wrkdirs/usr/ports/lang/dotnet8/work/bootstrap_sdk/dotnet /wrkdirs/usr/ports/lang/dotnet8/work/bootstrap_sdk/sdk/8.0.100/MSBuild.dll /nologo /nodemode:1 /nodeReuse:false /low:false 0 5757 5474 9 59 0 406801372 139216 uwait IJ - 0:01.45 | | | | `-- /wrkdirs/usr/ports/lang/dotnet8/work/bootstrap_sdk/dotnet /wrkdirs/usr/ports/lang/dotnet8/work/bootstrap_sdk/sdk/8.0.100/MSBuild.dll /nologo /nodemode:1 /nodeReuse:false /low:false
. . .

Notes:

I eventually killed process 5757 to let [03] progress to the next package.
Such classfies dotnet8 as failed.

I still have [04] stuck at build_port_done for dotnet-9.0.6 .


In another bulk -Ca test, dotnet8 built fine but dotnet-9.0.6 seems
to be stuck in build_port_done again:

ID TOTAL ORIGIN PKGNAME PHASE TIME TMPFS CPU% MEM%
. . .
[25] 06:10:17 lang/dotnet | dotnet-9.0.6 build_port_done 04:40:03 4.75 GiB
. . .


===
Mark Millard
marklmi at yahoo.com

I'd like to point out that I too have been experiencing this, although only on my amd64 builders and never on my arm64 builder. One thing Mark did not point out, though, is that the log for the build ends with saying that the wrkdirs unmount failed, specifically I see this as the last line in my own logs:

umount: unmount of /usr/local/poudriere/data/.m/14amd64-default/01/wrkdirs failed: Device busy

And similarly to Mark, because it ends like this with absolutely no processes in the jail, there is nothing to kill and thus no way to get poudriere to finish off that job.

Thanks,
Naram "CyberBotX" Qashat

Reply via email to