Hi everyone. Let me preface this by saying that we need a policy on this, and
we will probably struggle to reach a consensus. So to focus minds, I will
remind everyone that the project rules permit unilateral vetoes on
contributions, and in the absence of any policy on the matter that includes AI
generated contributions.
My proposal in brief is that LLM usage is
=====
Encouraged:
- Reviewing and otherwise validating human-authored patches before submission
- Debugging, diagnosing etc
Permitted:
- Generating or modifying tests, scripts, tooling or any other non-user facing
changes
- Minor changes to human-authored patches that are carefully reviewed by the
author
Restricted:
- Core code changes made by LLM may only be proposed by contributors with
demonstrated expertise
- Must have produced similar patches in size, scope and area unassisted and
with minimal third-party guidance
- Core code changes made by LLM require an additional reviewer
- LLM review is not a substitute for human review, and must be used only to
augment a complete and independent human understanding of the patch.
Prohibited:
- All public prose must be human authored. This includes inline comments, docs,
posts to Jira etc.
All LLM generated changes MUST be disclosed:
- Outlined to any reviewer;
- Summarised in the commit message;
- Large blocks or files must be individually marked with some agreed message
like "created by <some AI>"
=====
I will keep my argumentation brief.
LLM generated changes:
- Break the community and knowledge-building aspect of patch review, since
the contributor is not clearly learning from the feedback process
- Flips the asymmetry between author and reviewer/maintainer: it is now
cheaper to write than review or maintain
- Does not demonstrate expertise, commitment or that they can or will
maintain the patch
For these reasons, it should be expected that the person producing the patch
has already demonstrated their expertise and commitment by producing and
maintaining similar patches without the use of AI
Authoring a patch results in deeper understanding. To offset the reduced
collective knowledge of LLM generated changes, and improve our ability to
confidently maintain it, we should expect changes to be more widely socialised
through an additional review.
Finally, LLM comments are worse than no comment. An author must ensure their
reader's time is valued, by investing their own time crafting a message with a
proper understanding of its intended audience. LLMs may assist in preparation,
but must not produce the content itself.
I don't believe this fully accounts for all of the risks posed by AI usage in
the project, especially regarding maintainability of the codebase and
maintaining the community. But I hope this middle-ground policy will prove to
be acceptable to enough of the community.
