On 2015-01-31 10:06, Niels Thykier wrote: > [...] > > I have started a different approach (see [1] for WIP code). It is > mostly a parallel track to your idea, so they can certainly co-exist. > > The goal of this approach is to: > > * Split harness into a "simple" coordinator > * Remove the Lab as a (primary) data store (it is too fragile) > * Harness state as datastore > [...]
I got a patch series to implement this (see also [BRANCH]). I have
also managed to do a few tests on lilburn.debian.org with no issues.
The commit messages are at the bare minimum - apologise for that.
Review/remarks welcome.
With the rewrite:
* harness now uses ~16MB (rather than 700ish MB).
* harness now supports processing in "throw-away" labs (e.g. temp
labs cleaned up by lintian at the end of the run).
- Using $USE_PERMANENT_LAB = 0 in config
- $LINTIAN_SCRATCH_SPACE (if defined) becomes TMPDIR of the lintian
processes (and also the directory of the temp-labs).
* it is now possible to run harness on lintian.d.o with up to date
live data without affecting the automated run[1].
* it is possible to terminate lintian (e.g. kill -TERM) during a run
and harness will still mark the completed packages as up to date.
- caveat: sending harness a signal (e.g. ^C'ing harness) will not.
- caveat: Not recommended for $USE_PERMANENT_LAB = 1 cases.
* the full run is now implemented as "reschedule-all (eventually)" and
then an incremental run to start off with. The "clean mode" works
similarly, except it throws out the existing state files first.
What is missing / known issues:
* A patch to reduce the memory usage of html_reports (but that is for a
different patch series).
Nice to have / scope + feature creep:
* Have harness stop lintian when the time is up, rather than letting
it run to the end of the current run (which can be several hours
away).
* The state cache should probably be gzip'ed as it takes 30+MB of disk
(compressed is at 5-6ish MB)
Bonus features:
* Add --schedule-limit-groups parameter to change the limit of groups
to process. The timeout remains hard-coded though.
* It /should/ migrate the "last-processed-by" from the previous state
cache (untested). While it will still reschedule all packages,
their relative priority is retained.
Remarks on the patches:
* The patches 0001-0003 are self-contained and can be cherry-picked to
master (in any order in fact).
* The patches 0004-0008 are *not* self-contained and must all be
applied. I decided to have a patch per tool being updated as I
was easier to write them that way.
- Nothing a "--no-ff" merge cannot solve though.
* The 0009-0011 are incremental patches on top of the 0004-0008 bundle.
If there are no reviews or objections, I intend merge this later this
week and push it to lilburn.d.o to make graphs work again.
Thanks,
~Niels
[1] Take a copy the config, logs/lintian.log and the harness-state.
Ensure $USE_PERMANENT_LAB is set to 0 and $LINTIAN_SCRATCH_SPACE is set
to a (for you) writable directory with enough capacity (probably *not*
/tmp). Furthermore change the rest of the config values to fit your
workspace and you are good to go.
You /may/ also want to pass --schedule-limit-group N unless you do not
mind waiting several hours at worst.
[BRANCH]:
http://anonscm.debian.org/cgit/users/nthykier/lintian.git/log/?h=reporting-rewrite
NB: Rebased regularly.
0001-Move-save_state_cache-to-L-Util.patch
Description: application/mbox
0002-L-Util-Add-untaint-subroutine.patch
Description: application/mbox
0003-lintian-Add-status-log-for-use-by-harness.patch
Description: application/mbox
0004-reporting-sync-state-New-internal-reporting-command.patch
Description: application/mbox
0005-Make-find_backlog-check-out-of-date-flag-in-the-stat.patch
Description: application/mbox
0006-Rewrite-harness-to-use-reporting-sync-state.patch
Description: application/mbox
0007-Rewrite-html_reports-to-use-harness-state-cache.patch
Description: application/mbox
0008-maintainer.tmpl-Use-out-of-date-marker.patch
Description: application/mbox
0009-harness-Move-backlog-time-out-handling-into-process_.patch
Description: application/mbox
0010-harness-Harness-lintian-in-a-subprocess.patch
Description: application/mbox
0011-r-harness-Add-schedule-limit-groups-cmd-line-argumen.patch
Description: application/mbox
