jayzhan211 opened a new issue, #23839: URL: https://github.com/apache/datafusion/issues/23839
# Motivation It seems large PRs generated by AI is becoming a norm nowadays. SQL query engine is highly structured machine that the output is predictable from the given inputs. I think it is possible that we could build a testing framework that every AI generated code changes could run against the testing machine and we have highly confidence to ship the changes. https://github.com/apache/datafusion/pull/23738 is an example recently which has 4500+ LoC diffs from @Xuanwo. I believe such trends will continue, presenting an opportunity for us to review how we can conduct code reviews in agent era. # What we expect after running testing framework * Correctness * No Performance regression * Clean crate separation (logical plan stuffs doesn't go to physical layer and vice versa) Great crate design so it has clear responsibility and not ends up the mess. # What we have * Correctness - sqllogictest / [datafusion-fuzzer](https://github.com/datafusion-contrib/datafusion-fuzzer) * Performance - benchmark bot * Documentation to tell AI the purpose of each crate / component # What are the GAPs * Correctness coverage I think we need more test coverage for correctness that gives us high confidence to ship the code if the changes pass the test (e.g., catching silent semantic bugs in complex logic without relying solely on manual test writing). * Benchmarking for agents I'm thinking of more benchmark coverage that is easy to run and reproduce locally (assume most contributors run benchmark in their laptop) and also cost efficient for benchmark bot. * Architecture & System Design It should be carefully reviewed by reviewers (human). We might need tools to help us easily understand whether changes fit our system design or maybe we need spec in details to tell AI to follow the purpose of each components # Objective This discussion is to gather ideas for how code review could be like in agent era, and what gaps are left for us to achieve the goals. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
