On Thu, Sep 17, 2026, at 1:33 PM, Dan Langille wrote: > On Thu, Sep 17, 2026, at 12:42 PM, Mathieu Arnold wrote: >> On Thu, Sep 17, 2026 at 05:11:13PM +0200, Thierry Thomas wrote: >>> Le jeu. 17 sept. 26 à 17:00:55 +0200, Dan Langille <[email protected]> >>> écrivait : >>> >>> > The creation of INDEX takes time. By the time it is ready, more commits >>> > have >>> > occurred and INDEX is already out of date and some PORTVERSION values have >>> > changed. >>> >>> `make fetchindex` is fine. >> >> Well, it is faster, but it will not be what `make index` would output, >> as it's a previously generated INDEX that was geneated from another >> ports tree. > > That gave me an idea. I ran `make index` in a jail[1]. It took 747 > seconds (or 12.5 minutes). > > How does this sound for a strategy? > > * periodic/cronjob inserts a job into the FreshPorts work queue > * freshports daemon (FPD) processes the queue in order, so commit > processing is not going on during that job > * FPD find the job, notes the time, and launches a background job to > `make index` (MI) > * FPD continues on with the queue > * ... > * 12.5 minutes later, as the MI job completes, it inserts a reconcile > job into the queue > * when FPD gets to the reconcile job, it knows the start time > associated with the MI > * FPD compares the INDEX against the database and compiles a list of > ports which don't match INDEX > * any ports with commits after that MI start time are not included in > that list > * The ports on that list are refreshed from their Makefiles (because > more than just PORTVERSION can change) in the usual way that FP handles > a commit > * FPD continues on with the queue > * ... > > > 1 - [17:03 mydev dvl /usr/ports] % sudo time make index > Generating INDEX-15 - please wait..--- describe.accessibility --- > --- describe.arabic --- > --- describe.archivers --- > ... > --- describe.x11-toolkits --- > --- describe.x11-wm --- > Done. 747.38 real 2831.68 user 1706.13 sys
FYI, this solution is going to production soon. The scripts on the FreshPorts ingress (commit ingesting) nodes are now keeping a running list of ports which get PORTVERSION bumps from outside their Makefile. This is the compare script: https://github.com/FreshPorts/ingress-scripts/blob/main/compare-index.sh This script is where the running list is maintained: https://github.com/FreshPorts/ingress-scripts/blob/main/compare-index-daily.sh This is the list after few days. [12:06 dvl-ingress01 dvl ~freshports/cache/spooling] % sudo cat refresh-history.txt databases/py-gdbm databases/py-sqlite3 graphics/scantpaper lang/python-tools misc/eightctl multimedia/gstreamer1-plugins-all multimedia/gstreamer1-plugins-core security/py-blake3 security/x11appjail shells/meka sysutils/xxd www/github-release-monitor www/libdatachannel x11-toolkits/py-tkinter -- Dan Langille [email protected]
