CEP-21 - testing changing IP addresses

[Paul Chandler via dev]

Hi all,

We run some large clusters using Kubernetes and recently ran into an issue on 
4.0.x, so I decided to look at 5.1 and see if TCM fixes the issue.

However what we found was there are a number of new issues with TCM when a node 
is stopped then restarts with a new ip address, something that happens quite 
often with Kubernetes, and normally works fine in 4.0

So, I have a couple of questions, firstly, I am testing against trunk, is there 
a better branch to test TCM?
Secondly, is this stable enough that I should be raising Jira tickets for the 
issues, if so, should I attach them to the Epic CASSANDRA-19055 or create them 
standalone?

Thanks

Paul Chandler

Re: CEP-21 - testing changing IP addresses

[Marcus Eriksson]

Hi,

On Thu, Dec 21, 2023 at 09:45:23AM +, Paul Chandler via dev wrote:
> Hi all,
> 
> We run some large clusters using Kubernetes and recently ran into an issue on 
> 4.0.x, so I decided to look at 5.1 and see if TCM fixes the issue.
> 
> However what we found was there are a number of new issues with TCM when a 
> node is stopped then restarts with a new ip address, something that happens 
> quite often with Kubernetes, and normally works fine in 4.0
> 
> So, I have a couple of questions, firstly, I am testing against trunk, is 
> there a better branch to test TCM?

yes, trunk is the correct branch to test TCM

> Secondly, is this stable enough that I should be raising Jira tickets for the 
> issues, if so, should I attach them to the Epic CASSANDRA-19055 or create 
> them standalone?

yeah, please create jiras and attach to 19055

Thanks in advance!

/Marcus

Re: [DISCUSS] CEP-39: Cost Based Optimizer

[Benjamin Lerer]

Hi Scott,

Thanks for your feedback.

If I am not mistaken the main concern in your email is that without
features that will heavily benefit from the optimizer, this work will not
bring much. You are, therefore, under the impression that this work is one
or two years early.

In my perspective, we are already late. We have several features running in
production that we chose to not open source yet because implementing phase
1 of the CEP would have heavily simplify their designs. The cost of
developing them was much higher than what it would have been if the CEP had
already been implemented. We are also currently working on some SAI
features that need cost based optimization. This CEP will not be ready
before at least a year, so I imagine that it will be even more needed by
that time.

The proposal is a pretty generic framework. The approach has been around
for several decades and has been conceived with evolution in mind.

Regarding the CQL and SQL discussion, you seem to link relational algebra
to SQL but consider it separated from CQL. As SQL, CQL is a query language
used to perform queries on relations (a set of tuples) and as such
relational algebra can be use to express those. Using relational algebra
within the CQL layer does not change in anyway how CQL works today. On the
other end, it will allow to restructure the code making evolution much
easier. There are no reference to the SQL language in the proposal, even if
it refers in multiple places to optimization solutions implemented in
relational databases.

For Calcite, I took some time to download the source code, play with it and
dig into the documentation before discarding it. You can talk with Maxim, I
believe that he has a deeper knowledge of the tool than I have and could
provide a different perspective.

Let me know, if I misunderstood some of your comments or missed some
questions.







On Wed, Dec 20, 2023 at 5:15 PM C. Scott Andreas 
wrote:

> Thanks for this proposal and apologies for my delayed engagement during
> the Cassandra Summit last week. Benjamin, I appreciate your work on this
> and your engagement on this thread – I know it’s a lot of discussion to
> field.
>
> On ALLOW FILTERING:
>
> I share Chris Lohfink’s experience in operating clusters that have made
> heavy use of ALLOW FILTERING. It is a valuable guardrail for the database
> to require users specifically annotate queries that may cost 1000x+ that of
> a simple lookup for a primary key. For my own purposes, I’d actually like
> to go a step further and disable queries that require ALLOW FILTERING by
> default unless explicitly reviewed - but haven’t taken the step of adding
> such a guardrail yet.
>
> CBOs, CQL, and SQL:
>
> The CBO proposal cuts to the heart of one of the fundamental differences
> between SQL and CQL that I haven’t seen exercised yet.
>
> SQL allows users to define schemas that provide structure to data and to
> issue queries over them based on a relational algebra. SQL’s purpose is to
> decouple the on-disk representation of data from the query language used to
> access and aggregate it. This produces a very flexible query language that
> can be used to ask a database anything - but at a cost of execution that
> may be effectively infinite (think recursive subqueries).
>
> CQL is very different. While SQL is designed to decouple query language
> and on-disk representation, CQL is designed specifically to couple them. A
> CQL schema declares data placement, query routing, and disk serialization,
> and sorting to enable efficient retrieval. This is a very different design
> goal from a general-purpose query language. In time CQL may gain many
> SQL-like capabilities (and I hope it does!), but it will require careful
> work to do so without creating many footguns.
>
> Feature evolution:
>
> I agree that in the coming years, Cassandra is likely to gain
> semi-relational features via maturation of the byte-ordered partitioner
> (via range splitting, via TCM); the availability of SAI and its evolution
> (e.g., via new functionality enabled by Lucene libraries); potentially
> joins via BOP; and others. This is a really exciting future, and one that
> probably requires a planner and optimizer.
>
> My general inclination is that a planner + optimizer seem valuable for
> Cassandra, but that the proposal feels a year or two early. The database
> doesn’t yet have a plan of record to add support for some of the
> semirelational constructs we’ve talked about, and I’m not aware of active
> CEPs that propose designs for features like these yet.
>
> Like Jeff, I’d find this much easier to discuss in the context of a
> database gaining support for these features with specific designs available
> to discuss. The ALLOW FILTERING and constant folding examples are a little
> slim. Index selection is probably the best one I can think of right now -
> e.g., if we wanted to add the ability to issue partition-restricted queries
> over a base table with multiple indexes defined 

Re: [DISCUSS] CEP-39: Cost Based Optimizer

[Benjamin Lerer]

Hey German,

To clarify things, we intend to push cardinalities across nodes, not costs.
It will be up to the Cost Model to estimate cost based on those
cardinalities. We will implement some functionalities to collect costs on
query execution to be able to provide them as the output of EXPLAIN ANALYZE.

We will provide more details on how we will collect and distribute
cardinalities. We will probably not go into details on how we will estimate
costs before the patch for it is ready. The main reason being that there
are a lot of different parts that you need to account for and that it will
require significant testing and experimentation.

Regarding multi-tenancy, even if you use query cost, do not forget that you
will have to account also for background tasks such as compaction, repair,
backup, ... which is not included in this CEP.

Le jeu. 21 déc. 2023 à 00:18, German Eichberger via dev <
dev@cassandra.apache.org> a écrit :

> All,
>
> very much agree with Scott's reasoning.
>
> It seems expedient given the advent of ACCORD transactions to be more like
> the other distributed SQL databases and just support SQL. But just because
> it's expedient it isn't right and we should work out the relational
> features in more detail before we embark on tying us to some query planning
> design.
>
> The main problem in this space is pushing cost / across nodes based on
> data density. I understand that TCM will level out data density but the
> cost based optimizer proposal does a lot of hand waving when it comes to
> collecting/estimating costs for each node. I like to see more details on
> this since otherwise it will be fairly limiting.
>
> I am less tied to ALLOW FILTERING - many of my customers find allowing
> filtering beneficial for their workloads so I think removing it makes sense
> to me (and yes we try to discourage them 🙂)
>
> I am also intrigued by this proposal when I think about multi tenancy and
> resource governance: We have heard from several operator who run multiple
> internal teams on the same Cassandra cluster jut to optimize costs. Having
> a way to attribute those costs more fairly by adding up the costs the
> optimizer calculates might be hugely beneficial.  There could also be a way
> to have a "cost budget" on a keyspace to minimize the noisy neighbor
> problem and do more intelligent request throttling.
>
> In summary I support the proposal with the caveats raised above.
>
> Thanks,
> German
>
> --
> *From:* C. Scott Andreas 
> *Sent:* Wednesday, December 20, 2023 8:15 AM
> *To:* dev@cassandra.apache.org 
> *Cc:* dev@cassandra.apache.org 
> *Subject:* [EXTERNAL] Re: [DISCUSS] CEP-39: Cost Based Optimizer
>
> You don't often get email from sc...@paradoxica.net. Learn why this is
> important 
> Thanks for this proposal and apologies for my delayed engagement during
> the Cassandra Summit last week. Benjamin, I appreciate your work on this
> and your engagement on this thread – I know it’s a lot of discussion to
> field.
>
> On ALLOW FILTERING:
>
> I share Chris Lohfink’s experience in operating clusters that have made
> heavy use of ALLOW FILTERING. It is a valuable guardrail for the database
> to require users specifically annotate queries that may cost 1000x+ that of
> a simple lookup for a primary key. For my own purposes, I’d actually like
> to go a step further and disable queries that require ALLOW FILTERING by
> default unless explicitly reviewed - but haven’t taken the step of adding
> such a guardrail yet.
>
> CBOs, CQL, and SQL:
>
> The CBO proposal cuts to the heart of one of the fundamental differences
> between SQL and CQL that I haven’t seen exercised yet.
>
> SQL allows users to define schemas that provide structure to data and to
> issue queries over them based on a relational algebra. SQL’s purpose is to
> decouple the on-disk representation of data from the query language used to
> access and aggregate it. This produces a very flexible query language that
> can be used to ask a database anything - but at a cost of execution that
> may be effectively infinite (think recursive subqueries).
>
> CQL is very different. While SQL is designed to decouple query language
> and on-disk representation, CQL is designed specifically to couple them. A
> CQL schema declares data placement, query routing, and disk serialization,
> and sorting to enable efficient retrieval. This is a very different design
> goal from a general-purpose query language. In time CQL may gain many
> SQL-like capabilities (and I hope it does!), but it will require careful
> work to do so without creating many footguns.
>
> Feature evolution:
>
> I agree that in the coming years, Cassandra is likely to gain
> semi-relational features via maturation of the byte-ordered partitioner
> (via range splitting, via TCM); the availability of SAI and its evolution
> (e.g., via new functionality enabled by Lucene libraries); potentially
> joins via BOP;

Re: [DISCUSS] CEP-39: Cost Based Optimizer

[Josh McKenzie]

> we are already late. We have several features running in production that we 
> chose to not open source yet because implementing phase 1 of the CEP would 
> have heavily simplify their designs. The cost of developing them was much 
> higher than what it would have been if the CEP had already been implemented. 
> We are also currently working on some SAI features that need cost based 
> optimization.
Are there DISCUSS threads or CEP's for any of that work? For us to have a 
useful discussion about whether we're at a point in the project where a query 
optimizer is appropriate for the project this information would be vital.

On Thu, Dec 21, 2023, at 12:33 PM, Benjamin Lerer wrote:
> Hey German,
> 
> To clarify things, we intend to push cardinalities across nodes, not costs. 
> It will be up to the Cost Model to estimate cost based on those 
> cardinalities. We will implement some functionalities to collect costs on 
> query execution to be able to provide them as the output of EXPLAIN ANALYZE.
> 
> We will provide more details on how we will collect and distribute 
> cardinalities. We will probably not go into details on how we will estimate 
> costs before the patch for it is ready. The main reason being that there are 
> a lot of different parts that you need to account for and that it will 
> require significant testing and experimentation.
> 
> Regarding multi-tenancy, even if you use query cost, do not forget that you 
> will have to account also for background tasks such as compaction, repair, 
> backup, ... which is not included in this CEP.  
> 
> Le jeu. 21 déc. 2023 à 00:18, German Eichberger via dev 
>  a écrit :
>> All,
>> 
>> very much agree with Scott's reasoning. 
>> 
>> It seems expedient given the advent of ACCORD transactions to be more like 
>> the other distributed SQL databases and just support SQL. But just because 
>> it's expedient it isn't right and we should work out the relational features 
>> in more detail before we embark on tying us to some query planning design.
>> 
>> The main problem in this space is pushing cost / across nodes based on data 
>> density. I understand that TCM will level out data density but the cost 
>> based optimizer proposal does a lot of hand waving when it comes to 
>> collecting/estimating costs for each node. I like to see more details on 
>> this since otherwise it will be fairly limiting.
>> 
>> I am less tied to ALLOW FILTERING - many of my customers find allowing 
>> filtering beneficial for their workloads so I think removing it makes sense 
>> to me (and yes we try to discourage them 🙂)
>> 
>> I am also intrigued by this proposal when I think about multi tenancy and 
>> resource governance: We have heard from several operator who run multiple 
>> internal teams on the same Cassandra cluster jut to optimize costs. Having a 
>> way to attribute those costs more fairly by adding up the costs the 
>> optimizer calculates might be hugely beneficial.  There could also be a way 
>> to have a "cost budget" on a keyspace to minimize the noisy neighbor problem 
>> and do more intelligent request throttling. 
>> 
>> In summary I support the proposal with the caveats raised above.
>> 
>> Thanks,
>> German
>> 
>> 
>> *From:* C. Scott Andreas 
>> *Sent:* Wednesday, December 20, 2023 8:15 AM
>> *To:* dev@cassandra.apache.org 
>> *Cc:* dev@cassandra.apache.org 
>> *Subject:* [EXTERNAL] Re: [DISCUSS] CEP-39: Cost Based Optimizer
>>  
>> 
>> You don't often get email from sc...@paradoxica.net. Learn why this is 
>> important 
>> 
>> Thanks for this proposal and apologies for my delayed engagement during the 
>> Cassandra Summit last week. Benjamin, I appreciate your work on this and 
>> your engagement on this thread – I know it’s a lot of discussion to field.
>> 
>> On ALLOW FILTERING:
>> 
>> I share Chris Lohfink’s experience in operating clusters that have made 
>> heavy use of ALLOW FILTERING. It is a valuable guardrail for the database to 
>> require users specifically annotate queries that may cost 1000x+ that of a 
>> simple lookup for a primary key. For my own purposes, I’d actually like to 
>> go a step further and disable queries that require ALLOW FILTERING by 
>> default unless explicitly reviewed - but haven’t taken the step of adding 
>> such a guardrail yet.
>> 
>> CBOs, CQL, and SQL:
>> 
>> The CBO proposal cuts to the heart of one of the fundamental differences 
>> between SQL and CQL that I haven’t seen exercised yet.
>> 
>> SQL allows users to define schemas that provide structure to data and to 
>> issue queries over them based on a relational algebra. SQL’s purpose is to 
>> decouple the on-disk representation of data from the query language used to 
>> access and aggregate it. This produces a very flexible query language that 
>> can be used to ask a database anything - but at a cost of execution that may 
>> be effectively infinite (think recursive subqueries).
>> 
>> CQL is very differe

Harry in-tree (Forked from "Long tests, Burn tests, Simulator tests, Fuzz tests - can we clarify the diffs?")

[Alex Petrov]

Hey folks,

I am mostly done with a patch that brings Harry in-tree [1]. I will trigger one 
more CI run overnight, and my intention was to merge it some time soon, but I 
wanted to give a fair warning here, since this is a relatively large patch. 

Good news for everyone that it: 
  a) touches no production code whatsoever. Only test (in-jvm dtest namely) 
code that was using Harry already.
  b) the only tests that are changed are ones that used a duplicate version of 
placement simulator we had both for testing TCM, and in Harry
  c) in addition, I have converted 3 existing TCM tests to a new API to have 
some base for examples/usage.

Since we were effectively relying on this code for a while now, and the 
intention now is to converge to: 
  a) fewer different generators, and have a shareable version of generators for 
everyone to use accross the base
  b) a testing tool that can be useful for both trivial cases, and complex 
scenarios 
myself and many other Cassandra contributors have expressed an opinion that 
bringing Harry in-tree will be highly benefitial.

I strongly believe that bringing Harry in-tree will help to lower the barrier 
for fuzz test and simplify co-development of Cassandra and Harry. Previously, 
it has been rather difficult to debug edge cases because I had to either 
re-compile an in-jvm dtest jar and bring it to Harry, or re-compile a Harry jar 
and bring it to Cassandra, which is both tedious and time consuming. Moreover, 
I believe we have missed at very least one RT regression [2] because Harry was 
not in-tree, as its tests would've caught the issue even with the model that 
existed.

For other recently found issues, I think having Harry in-tree would have 
substantially lowered a turnaround time, and allowed me to share repros with 
developers of corresponding features much quicker.

I do expect a slight learning curve for Harry, but my intention is to build a 
web of simple tests (worked on some of them yesterday after conversation with 
David already), which can follow the in-jvm-dtest pattern of find-similar-test 
/ copy / modify. There's already copious documentation, so I do not believe not 
having docs for Harry was ever an issue, since there have been plenty. 

You all are aware of my dedication to testing and quality of Apache Cassandra, 
and I hope you also see the benefits of having a model checker in-tree.

Thank you and happy upcoming holidays,
--Alex

[1] https://issues.apache.org/jira/browse/CASSANDRA-19210
[2] https://issues.apache.org/jira/browse/CASSANDRA-18932

Re: Harry in-tree (Forked from "Long tests, Burn tests, Simulator tests, Fuzz tests - can we clarify the diffs?")

[Caleb Rackliffe]

+1

Agree w/ all the justifications mentioned above.

As a reviewer on CASSANDRA-19210
, my goals were to
a.) look at the directory, naming, and package structure of the ported
code, b.) make sure IDE integration was working, and c.) make sure any
modifications to existing code (rather than direct code movements from
cassandra-harry) were straightforward.

On Thu, Dec 21, 2023 at 3:23 PM Alex Petrov  wrote:

> Hey folks,
>
> I am mostly done with a patch that brings Harry in-tree [1]. I will
> trigger one more CI run overnight, and my intention was to merge it some
> time soon, but I wanted to give a fair warning here, since this is a
> relatively large patch.
>
> Good news for everyone that it:
>   a) touches no production code whatsoever. Only test (in-jvm dtest
> namely) code that was using Harry already.
>   b) the only tests that are changed are ones that used a duplicate
> version of placement simulator we had both for testing TCM, and in Harry
>   c) in addition, I have converted 3 existing TCM tests to a new API to
> have some base for examples/usage.
>
> Since we were effectively relying on this code for a while now, and the
> intention now is to converge to:
>   a) fewer different generators, and have a shareable version of
> generators for everyone to use accross the base
>   b) a testing tool that can be useful for both trivial cases, and complex
> scenarios
> myself and many other Cassandra contributors have expressed an opinion
> that bringing Harry in-tree will be highly benefitial.
>
> I strongly believe that bringing Harry in-tree will help to lower the
> barrier for fuzz test and simplify co-development of Cassandra and Harry.
> Previously, it has been rather difficult to debug edge cases because I had
> to either re-compile an in-jvm dtest jar and bring it to Harry, or
> re-compile a Harry jar and bring it to Cassandra, which is both tedious and
> time consuming. Moreover, I believe we have missed at very least one RT
> regression [2] because Harry was not in-tree, as its tests would've caught
> the issue even with the model that existed.
>
> For other recently found issues, I think having Harry in-tree would have
> substantially lowered a turnaround time, and allowed me to share repros
> with developers of corresponding features much quicker.
>
> I do expect a slight learning curve for Harry, but my intention is to
> build a web of simple tests (worked on some of them yesterday after
> conversation with David already), which can follow the in-jvm-dtest pattern
> of find-similar-test / copy / modify. There's already copious
> documentation, so I do not believe not having docs for Harry was ever an
> issue, since there have been plenty.
>
> You all are aware of my dedication to testing and quality of Apache
> Cassandra, and I hope you also see the benefits of having a model checker
> in-tree.
>
> Thank you and happy upcoming holidays,
> --Alex
>
> [1] https://issues.apache.org/jira/browse/CASSANDRA-19210
> [2] https://issues.apache.org/jira/browse/CASSANDRA-18932
>
>

Re: [DISCUSS] CEP-39: Cost Based Optimizer

[Caleb Rackliffe]

> We are also currently working on some SAI features that need cost based
optimization.

I don't even think we have to think about *new* SAI features to see where
it will benefit from further *local* optimization, and I'm sympathetic to
that happening in the context of a larger framework, as long as the
framework itself starts as thin as possible and grows over time.

For SAI, the main difficulties we're likely to have in the very short term
are a.) how to order/choose predicates during AND queries to minimize
intersection complexity, b.) how to make decisions about when to use an
index or simple filtering, and c.) combinations of those two, where we
might take different paths depending on how many predicates exist and the
cardinality of the term indexes those predicates touch.

ex. We have a system property called SAI_INTERSECTION_CLAUSE_LIMIT (in CRP)
that controls the maximum number of index query intersections that will
participate in an AND query, leaving the rest for post-filtering. Having
local cardinality estimation on the individual column indexes might make it
a lot easier to pick the two most selective predicates. (Numeric range
predicates, for example, can have matching posting lists of wildly varying
sizes.)

tl;dr I'd like to see us start by enumerating the specific scenarios where
query optimization will benefit SAI in conjunction w/ creating a template
for how a high-level CBO apparatus would work (which is sort of what we
have in this CEP, even if it doesn't go into extreme detail). Then, we
build from the bottom up to ship improvements as quickly as possible w/o
compromising the longer term CBO vision.

Re: [DISCUSS] CEP-39: Cost Based Optimizer

[Caleb Rackliffe]

What would the relationship between our present query tracing apparatus and
EXPLAIN ANALYZE look like?

On Thu, Dec 21, 2023 at 4:24 PM Caleb Rackliffe 
wrote:

> > We are also currently working on some SAI features that need cost based
> optimization.
>
> I don't even think we have to think about *new* SAI features to see where
> it will benefit from further *local* optimization, and I'm sympathetic to
> that happening in the context of a larger framework, as long as the
> framework itself starts as thin as possible and grows over time.
>
> For SAI, the main difficulties we're likely to have in the very short term
> are a.) how to order/choose predicates during AND queries to minimize
> intersection complexity, b.) how to make decisions about when to use an
> index or simple filtering, and c.) combinations of those two, where we
> might take different paths depending on how many predicates exist and the
> cardinality of the term indexes those predicates touch.
>
> ex. We have a system property called SAI_INTERSECTION_CLAUSE_LIMIT (in
> CRP) that controls the maximum number of index query intersections that
> will participate in an AND query, leaving the rest for post-filtering.
> Having local cardinality estimation on the individual column indexes might
> make it a lot easier to pick the two most selective predicates. (Numeric
> range predicates, for example, can have matching posting lists of wildly
> varying sizes.)
>
> tl;dr I'd like to see us start by enumerating the specific scenarios where
> query optimization will benefit SAI in conjunction w/ creating a template
> for how a high-level CBO apparatus would work (which is sort of what we
> have in this CEP, even if it doesn't go into extreme detail). Then, we
> build from the bottom up to ship improvements as quickly as possible w/o
> compromising the longer term CBO vision.
>

Re: [DISCUSS] CEP-39: Cost Based Optimizer

[Caleb Rackliffe]

I think I hinted at this in my first response, but just to clarify, I would
be interested to see this work broken up as much as possible into a.) the
set of things we can do without coordinator involvement (statistical
optimization for index and filtering queries) and b.) the set of things
where coordinator-level planning is harder to avoid (joins?)

Re: Harry in-tree (Forked from "Long tests, Burn tests, Simulator tests, Fuzz tests - can we clarify the diffs?")

[Joseph Lynch]

+1

Sounds like a great change that will help us unify around a common testing
paradigm, and even pave the path to in-tree load testing plus integrated
correctness checking which would be extremely valuable!

-Joey

On Thu, Dec 21, 2023 at 1:35 PM Caleb Rackliffe 
wrote:

> +1
>
> Agree w/ all the justifications mentioned above.
>
> As a reviewer on CASSANDRA-19210
> , my goals were to
> a.) look at the directory, naming, and package structure of the ported
> code, b.) make sure IDE integration was working, and c.) make sure any
> modifications to existing code (rather than direct code movements from
> cassandra-harry) were straightforward.
>
> On Thu, Dec 21, 2023 at 3:23 PM Alex Petrov  wrote:
>
>> Hey folks,
>>
>> I am mostly done with a patch that brings Harry in-tree [1]. I will
>> trigger one more CI run overnight, and my intention was to merge it some
>> time soon, but I wanted to give a fair warning here, since this is a
>> relatively large patch.
>>
>> Good news for everyone that it:
>>   a) touches no production code whatsoever. Only test (in-jvm dtest
>> namely) code that was using Harry already.
>>   b) the only tests that are changed are ones that used a duplicate
>> version of placement simulator we had both for testing TCM, and in Harry
>>   c) in addition, I have converted 3 existing TCM tests to a new API to
>> have some base for examples/usage.
>>
>> Since we were effectively relying on this code for a while now, and the
>> intention now is to converge to:
>>   a) fewer different generators, and have a shareable version of
>> generators for everyone to use accross the base
>>   b) a testing tool that can be useful for both trivial cases, and
>> complex scenarios
>> myself and many other Cassandra contributors have expressed an opinion
>> that bringing Harry in-tree will be highly benefitial.
>>
>> I strongly believe that bringing Harry in-tree will help to lower the
>> barrier for fuzz test and simplify co-development of Cassandra and Harry.
>> Previously, it has been rather difficult to debug edge cases because I had
>> to either re-compile an in-jvm dtest jar and bring it to Harry, or
>> re-compile a Harry jar and bring it to Cassandra, which is both tedious and
>> time consuming. Moreover, I believe we have missed at very least one RT
>> regression [2] because Harry was not in-tree, as its tests would've caught
>> the issue even with the model that existed.
>>
>> For other recently found issues, I think having Harry in-tree would have
>> substantially lowered a turnaround time, and allowed me to share repros
>> with developers of corresponding features much quicker.
>>
>> I do expect a slight learning curve for Harry, but my intention is to
>> build a web of simple tests (worked on some of them yesterday after
>> conversation with David already), which can follow the in-jvm-dtest pattern
>> of find-similar-test / copy / modify. There's already copious
>> documentation, so I do not believe not having docs for Harry was ever an
>> issue, since there have been plenty.
>>
>> You all are aware of my dedication to testing and quality of Apache
>> Cassandra, and I hope you also see the benefits of having a model checker
>> in-tree.
>>
>> Thank you and happy upcoming holidays,
>> --Alex
>>
>> [1] https://issues.apache.org/jira/browse/CASSANDRA-19210
>> [2] https://issues.apache.org/jira/browse/CASSANDRA-18932
>>
>>