Re: Status Update on CEP-7 Storage Attached Indexes (SAI)

[J. D. Jordan]

Thanks for all the work here! On Jul 26, 2023, at 1:57 PM, Caleb Rackliffe <calebrackli...@gmail.com> wrote:  Alright, the cep-7-sai branch is now merged to trunk! Now we move to addressing the most urgent items from "Phase 2" (CASSANDRA-18473) before (and in the case of some testing after) the 5.0 freeze... On Wed, Jul 26, 2023 at 6:07 AM Jeremy Hanna <jeremy.hanna1...@gmail.com> wrote: Thanks Caleb and Mike and Zhao and Andres and Piotr and everyone else involved with the SAI implementation! On Jul 25, 2023, at 3:01 PM, Caleb Rackliffe <calebrackli...@gmail.com> wrote:  Just a quick update... With CASSANDRA-18670 complete, and all remaining items in the category of performance optimizations and further testing, the process of merging to trunk will likely start today, beginning with a final rebase on the current trunk and J11 and J17 test runs. On Tue, Jul 18, 2023 at 3:47 PM Caleb Rackliffe <calebrackli...@gmail.com> wrote: Hello there! After much toil, the first phase of CEP-7 is nearing completion (see CASSANDRA-16052). There are presently two issues to resolve before we'd like to merge the cep-7-sai feature branch and all its goodness to trunk: CASSANDRA-18670 - Importer should build SSTable indexes successfully before making new SSTables readable (in review) CASSANDRA-18673 - Reduce size of per-SSTable index components (in progress) (We've been getting clean CircleCI runs for a while now, and have been using the multiplexer to sniff out as much flakiness as possible up front.) Once merged to trunk, the next steps are: 1.) Finish a Harry model that we can use to further fuzz test SAI before 5.0 releases (see CASSANDRA-18275). We've done a fair amount of fuzz/randomized testing at the component level, but I'd still consider Harry (at least around single-partition query use-cases) a critical item for us to have confidence before release. 2.) Start pursuing Phase 2 items as time and our needs allow. (see CASSANDRA-18473) A reminder, SAI is a secondary index, and therefore is by definition an opt-in feature, and has no explicit "feature flag". However, its availability to users is still subject to the secondary_indexes_enabled guardrail, which currently defaults to allowing creation. Any thoughts, questions, or comments on the pre-merge plan here?