Hi Dave,
I tested it by running it blind over 50 release candidates from 40 podlings,
voted between January 2024 and September 2026. For each one, I read only the
opening vote email, wrote down the findings, then read the thread and compared.
It made 46 correct calls and one partial. It missed a real blocker twice and
raised one false blocker. It also found 13 problems no reviewer mentioned, so
it could be argued from those numbers that it does better than human review on
its own. Compared to the plain skill, which only got 39 of the 50 right.
The method and the list of podlings are in the README, and anyone can repeat
it. The Python checker, check_rc.py, is deterministic and needs no LLM, so that
is useful on its own.
On the roster point, the onboarding skill doesn't depend on the Whimsy roster
tool. If the roster tool is replaced, only the MCP underneath might need to
change, not the skill.
For the minimum LLM, only reviewing what the checker flags needs a model, and a
human could also review it, so no model would be needed. The SKILL does make a
few other checks by calling MCPs, but these are optional. I have not fully
tested it without the MCPs to see what difference they make, but it would make
a difference; for instance, it wouldn’t look at previous RCs/votes to see what
issues they contained. I've run the skill with Claude, Codex, and OpenCode. The
results with OpenCode (Zen) were not as good, but still helpful.
Here is what the checker output looks like before it gets parsed by an LLM:
# Release candidate check: 0.43.0-incubating-RC1
Deterministic checks only. `fail` items are facts. `candidate` items need a
person;
most will be fine and some will not. Nothing here is a vote.
## Artifacts
- apache-burr-0.43.0-incubating-sdist.tar.gz
- apache-burr-0.43.0-incubating-src.tar.gz
- apache_burr-0.43.0-py3-none-any.whl
## Counts
- files: 1152
- asf_header: 869
- no_header: 17
- spdx: 0
- third_party_header: 1
- asf_plus_foreign: 0
- binary: 0
- nested_archive: 0
- media: 90
- data: 0
- licence_signal: 1
## fail (1)
### naming: archive name does not contain "incubating" (1)
- `apache_burr-0.43.0-py3-none-any.whl`
## candidate (2)
### licence-signal: mentions GPL/LGPL/AGPL (1)
- `apache-burr-0.43.0-incubating-src.tar.gz/website/package-lock.json` |
2pRiaydKJ1G/rHZvnsrHAOcQALIi9sA5co5xenQdTugCvtb1cuf78Vf4g==", "cpu": [
"arm64" ], "license": "LGPL-3.0-or-later", "optional":
true, "os": [ "darwin" ], "funding": { "url":
"https://o
### header-third-party: third-party header (Apache-licensed or other
copyright); check LICENSE and NOTICE cover it (1)
By copyright holder: shadcn (1)
-
`apache-burr-0.43.0-incubating-src.tar.gz/website/src/components/ui/button.tsx`
| ["* Copyright (c) 2023 shadcn"]
## Images, fonts and documents (90)
Usually fine. Fonts and icon sets are the ones that tend to carry their own
licence.
-
`apache-burr-0.43.0-incubating-sdist.tar.gz/burr/tracking/server/s3/architecture.png`
-
`apache-burr-0.43.0-incubating-sdist.tar.gz/examples/hello-world-counter/statemachine.png`
-
`apache-burr-0.43.0-incubating-sdist.tar.gz/examples/deep-researcher/statemachine.png`
-
`apache-burr-0.43.0-incubating-sdist.tar.gz/examples/streaming-fastapi/statemachine.png`
-
`apache-burr-0.43.0-incubating-sdist.tar.gz/examples/multi-modal-chatbot/statemachine.png`
-
`apache-burr-0.43.0-incubating-sdist.tar.gz/examples/email-assistant/statemachine.png`
- `apache-burr-0.43.0-incubating-sdist.tar.gz/telemetry/ui/public/favicon.ico`
- `apache-burr-0.43.0-incubating-sdist.tar.gz/telemetry/ui/public/logo.png`
- `apache-burr-0.43.0-incubating-src.tar.gz/chatbot.gif`
- `apache-burr-0.43.0-incubating-src.tar.gz/burr_logo.svg`
- `apache-burr-0.43.0-incubating-src.tar.gz/burr_logo.png`
-
`apache-burr-0.43.0-incubating-src.tar.gz/burr/tracking/server/s3/architecture.png`
- `apache-burr-0.43.0-incubating-src.tar.gz/website/public/burr_logo.svg`
- `apache-burr-0.43.0-incubating-src.tar.gz/website/public/burr_logo.png`
- `apache-burr-0.43.0-incubating-src.tar.gz/website/public/logos/taskhuman.svg`
- ... 75 more in findings.json
## Code or documentation files with no licence header (17)
-
`apache-burr-0.43.0-incubating-sdist.tar.gz/examples/deep-researcher/prompts.py`
-
`apache-burr-0.43.0-incubating-sdist.tar.gz/examples/deep-researcher/deep_researcher_utils.py`
-
`apache-burr-0.43.0-incubating-src.tar.gz/website/src/components/ui/safari.tsx`
-
`apache-burr-0.43.0-incubating-src.tar.gz/website/src/components/ui/animated-beam.tsx`
-
`apache-burr-0.43.0-incubating-src.tar.gz/website/src/components/ui/shimmer-button.tsx`
-
`apache-burr-0.43.0-incubating-src.tar.gz/website/src/components/ui/icon-cloud.tsx`
-
`apache-burr-0.43.0-incubating-src.tar.gz/website/src/components/ui/border-beam.tsx`
-
`apache-burr-0.43.0-incubating-src.tar.gz/website/src/components/ui/dot-pattern.tsx`
-
`apache-burr-0.43.0-incubating-src.tar.gz/website/src/components/ui/marquee.tsx`
-
`apache-burr-0.43.0-incubating-src.tar.gz/website/src/components/ui/number-ticker.tsx`
-
`apache-burr-0.43.0-incubating-src.tar.gz/website/src/components/ui/magic-card.tsx`
-
`apache-burr-0.43.0-incubating-src.tar.gz/website/src/components/ui/animated-shiny-text.tsx`
-
`apache-burr-0.43.0-incubating-src.tar.gz/website/src/components/ui/blur-fade.tsx`
- `apache-burr-0.43.0-incubating-src.tar.gz/examples/deep-researcher/prompts.py`
-
`apache-burr-0.43.0-incubating-src.tar.gz/examples/deep-researcher/deep_researcher_utils.py`
- ... 2 more in findings.json
Kind regards,
Justin
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]