Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

On Wed, 12 Jul 2023 at 15:05, Jacek Lewandowski wrote: > I believe some tools can determine which tests make sense to multiplex, > given that some exact lines of code were changed using code coverage > analysis. After the initial run, we should have data from the coverage > analysis, which would

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

> A strong +1 to getting to a single CI system. CircleCI definitely has some > niceties and I understand why it's currently used, but right now we get 2 > CI systems for twice the price. +1 on the proposed subsets. > That's not entirely true, it provides value in "double accounting" and that has

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

> Revert for only trunk patches right? > I’d say we need to completely stabilize the environment, no noise before we > go into that direction. Hm. Is the concern multi-branch reverts w/merge commits being awful? Because I hear that. Starting trunk-only would be reasonable enough I think, especia

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

Revert for only trunk patches right? I’d say we need to completely stabilize the environment, no noise before we go into that direction. On Wed, 12 Jul 2023 at 8:55, Jacek Lewandowski wrote: > Would it be re-opening the ticket or creating a new ticket with "revert of > fix" ? > > > > śr., 12 lip

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

> Would it be re-opening the ticket or creating a new ticket with "revert of > fix" ? I have a weak preference for re-opening the original ticket and tracking the revert + fix there. Keeps the workflow in one place. "Downside" is having multiple commits with "CASSANDRA-XX" in the message but

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

I believe some tools can determine which tests make sense to multiplex, given that some exact lines of code were changed using code coverage analysis. After the initial run, we should have data from the coverage analysis, which would tell us which test classes are tainted - that is, they cover the

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

Would it be re-opening the ticket or creating a new ticket with "revert of fix" ? śr., 12 lip 2023 o 14:51 Ekaterina Dimitrova napisał(a): > jenkins_jira_integration > > script > updat

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

jenkins_jira_integration script updating the JIRA ticket with test results if you cause a regression + us building a muscle around reverting your commit if they break tests.“ I am not sure

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

(This response ended up being a bit longer than intended; sorry about that) > What is more common though is packaging errors, > cdc/compression/system_ks_directory targeted fixes, CI w/wo > upgrade tests, being less responsive post-commit as you already > moved on *Two that ***should ***be resolve

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

“ On our 4.0 release I remember a number of such failures but not recently. ” Based on all the 5.0 work I’d say we need as a minimum to build and start a node with all JDK versions pre-commit. On Wed, 12 Jul 2023 at 7:29, Jacek Lewandowski wrote: > Isn't novnodes a special case of vnodes with

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

Isn't novnodes a special case of vnodes with n=1 ? We should rather select a subset of tests for which it makes sense to run with different configurations. The set of configurations against which we run the tests currently is still only the subset of all possible cases. I could ask - why don't ru

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

On our 4.0 release I remember a number of such failures but not recently. What is more common though is packaging errors, cdc/compression/system_ks_directory targeted fixes, CI w/wo upgrade tests, being less responsive post-commit as you already moved on,... Either the smoke pre-commit has appr

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

> 2: Pre-commit 'devBranch' full suite for high risk/disruptive merges: at > reviewer's discretion In general, maybe offering a dev the option of choosing either "pre-commit smoke" or "post-commit full" at their discretion for any work would be the right play. A follow-on thought: even with som

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

A strong +1 to getting to a single CI system. CircleCI definitely has some niceties and I understand why it's currently used, but right now we get 2 CI systems for twice the price. +1 on the proposed subsets. Derek On Mon, Jul 10, 2023 at 9:37 AM Josh McKenzie wrote: > I'm personally not thinki

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

Add a 'devBranch' jenkins job to that imo: The possibility to run the full suite + multiplex new tests before commit when you're about to release a Kraken into the codebase: Accord, TCM, TTL, SAI, Vector, JDK... So: 1: Pre-commit subset of tests (suites + matrices + env) runs. On green, merge

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

I'm personally not thinking about CircleCI at all; I'm envisioning a world where all of us have 1 CI *software* system (i.e. reproducible on any env) that we use for pre-commit validation, and then post-commit happens on reference ASF hardware. So: 1: Pre-commit subset of tests (suites + matric

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

+1 to Josh which is exactly my line of thought as well. But that is only valid if we have a solid Jenkins that will eventually run all test configs. So I think I lost track a bit here. Are you proposing: 1- CircleCI: Run pre-commit a single (the most common/meaningful, TBD) config of tests 2

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

I think 500 runs combining all configs could be reasonable, since it's unlikely to have config-specific flaky tests. As in five configs with 100 repetitions each. On Fri, 7 Jul 2023 at 16:14, Josh McKenzie wrote: > Maybe. Kind of depends on how long we write our tests to run doesn't it? :) > > B

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

Maybe. Kind of depends on how long we write our tests to run doesn't it? :) But point taken. Any non-trivial test would start to be something of a beast under this approach. On Fri, Jul 7, 2023, at 11:12 AM, Brandon Williams wrote: > On Fri, Jul 7, 2023 at 10:09 AM Josh McKenzie wrote: > > 3. M

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

On Fri, Jul 7, 2023 at 10:09 AM Josh McKenzie wrote: > 3. Multiplexed tests (changed, added) run against all JDK's and a broader > range of configs (no-vnode, vnode default, compression, etc) I think this is going to be too heavy...we're taking 500 iterations and multiplying that by like 4 or 5?

Re: Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

> I wouldn’t also advocate not running JDK17 tests until we enable all test > suites post-commit What about this: 1. Pre-commit suite runs against just 1 JDK (latest supported / most common / default) 2. Pre-commit suite runs against default config 3. *Multiplexed tests* (changed, added) run agai

Fwd: [DISCUSS] Formalizing requirements for pre-commit patches on new CI

“I'm curious what it triggers for you Brandon, Berenguer, Andres, Ekaterina, and Mick (when you're back from the mountains ;)). ” We already have pre-commit a minimum set being mandatory in CircleCI. People can manually trigger other tests if they feel they might have broken something. The only tes