Glad to see some traction for the approach I proposed. If it would be helpful, since some folks have proposed edits, I can put the proposal into a PR similar to the previous proposal. I’m traveling today but I can likely find the time at some point.
@Josh I don’t personally object to your counterproposal on comments / docs. I would make the case that’s a project level standard and maybe the LLM guidelines doc then states something to the effect of “LLM generated docs and comments must meet [linked or insert guidelines]. As with code, quality of documentation and comments is the responsibility of the contributor”. I’m not sure all community members here will feel comfortable with that given the ongoing discussion but I’m happy to support it myself. Maybe I’m being pedantic but I’m trying to separate what are great project level standards (testing etc) from our LLM-use guiding principles which I’m hoping to keep short, direct, and as clear as we can. Personally, I’ve reduced my use of AI writing and had asked my team to do the same. But I know there are folks who are using AI to write effectively I just find it to be more rare these days. But as you correctly point out, this too will evolve. Jordan On Wed, Sep 30, 2026 at 06:13 Josh McKenzie <[email protected]> wrote: > * It is strongly preferred that documentation and comments are not LLM > generated. We have found these to be of low quality. However, if done, the > aforementioned accountability remains with the contributor. “The LLM wrote > it” is not an acceptable dismissal of a review comment in any context. It > is recommended that documentation and comments continue to be human written > and optional LLM reviewed or edited with human supervision. > > I'd prefer we move from discouraging the use of certain tooling in > documentation and comments to being more prescriptive about what quality > documentation and comments looks like. Since this tooling is evolving very > rapidly (and different peoples' local setups differs greatly in capability > and quality of output even today), this feels like a rule that's going to > rapidly age out of relevance. > > --- > > Taking a crack at an alternative: > > - Code should be clearly and concisely documented. Classes should > contain at least minimal javadoc headers explaining why they exist and > their role in the architecture, and any non-trivial methods should likewise > document their "why" and any non-obvious contextual assumptions at the time > of their authoring. Include {@link} pointers to other methods they have > behavioral dependencies on. Comments in the code should not be redundant > with what the code is doing; if the code is opaque enough that you have to > write comments to explain what it's doing that's probably a sign the code > should be refactored and simplified. > > > --- > The reason I'd advocate for this over the blanket discouragement of LLMs > is that we have a lot of really poorly documented code and a lack of > alignment on this front as a project is hurting us; these guidelines would > help set expectations for what we collectively think good comments look > like and leave it up to a contributor on how they want to get to that bar > of quality. > > Also: thanks for the measured and well thought out proposal Jordan. I > agree w/others that we really should just be more declarative about what > the bar of quality is and apply it evenly for code regardless of where it > comes from (human, LLM) and lift that bar up to where we largely all agree > it should be. If we have such agreement. :) > > On Tue, Sep 29, 2026, at 11:54 PM, C. Scott Andreas wrote: > > Dinesh, > > I follow the point that you are making, but your reply dismisses Jane’s > central concern without addressing it: the review burden. > > “Software testing, certification, and verification” are orthogonal > concerns to the burden on reviewers and the activity of review: ensuring > harmony with the overall codebase and architecture, replicating knowledge > of the implementation among maintainers, and refining software together. > > These are functions that aren’t and can’t be addressed by software. They > are the “community” part of code. > > – Scott > > On Sep 29, 2026, at 8:39 PM, Dinesh Joshi <[email protected]> wrote: > > > Jane, > > Everything that you just said is true about human generated code as well. > Believe it or not every engineer that hand writes code also believes that > their code is perfect and they understand it completely. Yet reviewers find > issues with it (and even they miss things that later get caught in testing > or production). > > The issue with AI is that we can produce a lot of code very quickly and > create a lot of work for reviewers. > > The problem we should be addressing is that of software testing, > verification and certification of the database. > > Thanks, > > Dinesh > > On Tue, Sep 29, 2026 at 4:58 PM Jane H <[email protected]> wrote: > > I'd support detailed and explicit policies, like Benedict's, Rust's, or > Caleb's, at least they can actually help the problem of AI assisted code > increasing reviewers' burden, though they might introduce other problems. > > Blake, why I think your 3 point simple rules won't work as you wish: > people have very different understandings in what's considered responsible > enough or comprehensive enough understanding. > > Everyone who has brought AI-coded PRs for me to review, new contributors > or committers, who costs around 4x time from me to review their PRs > compared to human-written ones (already more time than implementing on my > own), believes that they are fully responsible for their code and fully > understand their code. Every one of them believes so. > > I tried reminding them that they have to be fully responsible for the > verification of their code. Didn't help in my own experience. (I don't > blame them. We are all figuring out how to use this new tool. And I > voluntarily pick up the tasks of reviewing those PRs. Not their fault. But > that's why we need explicit guidance/policies.) > > I've already met people who believe one of the following is considered > responsible enough: > - Read every line of code and think it makes sense. > - Pass other AI's code reviews > - Pass all existing tests and AI-coded new unit tests and integration tests > > But I think none of them is enough. Therefore, when someone says they are > fully responsible for/fully understanding their code, it doesn't really > reduce my reviewer's load. Those detailed and explicit policies like > Benedict's or Rust's, are largely just defining what's considered > responsible enough. So they will help. > >
