Re: Evolving the client protocol

[sankalp kohli]

If you donate Thread per core to C*, I am sure someone can help you review
it and get it committed.

On Thu, Apr 19, 2018 at 11:15 AM, Ben Bromhead  wrote:

> Re #3:
>
> Yup I was thinking each shard/port would appear as a discrete server to the
> client.
>
> If the per port suggestion is unacceptable due to hardware requirements,
> remembering that Cassandra is built with the concept scaling *commodity*
> hardware horizontally, you'll have to spend your time and energy convincing
> the community to support a protocol feature it has no (current) use for or
> find another interim solution.
>
> Another way, would be to build support and consensus around a clear
> technical need in the Apache Cassandra project as it stands today.
>
> One way to build community support might be to contribute an Apache
> licensed thread per core implementation in Java that matches the protocol
> change and shard concept you are looking for ;P
>
>
> On Thu, Apr 19, 2018 at 1:43 PM Ariel Weisberg  wrote:
>
> > Hi,
> >
> > So at technical level I don't understand this yet.
> >
> > So you have a database consisting of single threaded shards and a socket
> > for accept that is generating TCP connections and in advance you don't
> know
> > which connection is going to send messages to which shard.
> >
> > What is the mechanism by which you get the packets for a given TCP
> > connection delivered to a specific core? I know that a given TCP
> connection
> > will normally have all of its packets delivered to the same queue from
> the
> > NIC because the tuple of source address + port and destination address +
> > port is typically hashed to pick one of the queues the NIC presents. I
> > might have the contents of the tuple slightly wrong, but it always
> includes
> > a component you don't get to control.
> >
> > Since it's hashing how do you manipulate which queue packets for a TCP
> > connection go to and how is it made worse by having an accept socket per
> > shard?
> >
> > You also mention 160 ports as bad, but it doesn't sound like a big number
> > resource wise. Is it an operational headache?
> >
> > RE tokens distributed amongst shards. The way that would work right now
> is
> > that each port number appears to be a discrete instance of the server. So
> > you could have shards be actual shards that are simply colocated on the
> > same box, run in the same process, and share resources. I know this
> pushes
> > more of the complexity into the server vs the driver as the server
> expects
> > all shards to share some client visible like system tables and certain
> > identifiers.
> >
> > Ariel
> > On Thu, Apr 19, 2018, at 12:59 PM, Avi Kivity wrote:
> > > Port-per-shard is likely the easiest option but it's too ugly to
> > > contemplate. We run on machines with 160 shards (IBM POWER 2s20c160t
> > > IIRC), it will be just horrible to have 160 open ports.
> > >
> > >
> > > It also doesn't fit will with the NICs ability to automatically
> > > distribute packets among cores using multiple queues, so the kernel
> > > would have to shuffle those packets around. Much better to have those
> > > packets delivered directly to the core that will service them.
> > >
> > >
> > > (also, some protocol changes are needed so the driver knows how tokens
> > > are distributed among shards)
> > >
> > > On 2018-04-19 19:46, Ben Bromhead wrote:
> > > > WRT to #3
> > > > To fit in the existing protocol, could you have each shard listen on
> a
> > > > different port? Drivers are likely going to support this due to
> > > > https://issues.apache.org/jira/browse/CASSANDRA-7544 (
> > > > https://issues.apache.org/jira/browse/CASSANDRA-11596).  I'm not
> super
> > > > familiar with the ticket so their might be something I'm missing but
> it
> > > > sounds like a potential approach.
> > > >
> > > > This would give you a path forward at least for the short term.
> > > >
> > > >
> > > > On Thu, Apr 19, 2018 at 12:10 PM Ariel Weisberg 
> > wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> I think that updating the protocol spec to Cassandra puts the onus
> on
> > the
> > > >> party changing the protocol specification to have an implementation
> > of the
> > > >> spec in Cassandra as well as the Java and Python driver (those are
> > both
> > > >> used in the Cassandra repo). Until it's implemented in Cassandra we
> > haven't
> > > >> fully evaluated the specification change. There is no substitute for
> > trying
> > > >> to make it work.
> > > >>
> > > >> There are also realities to consider as to what the maintainers of
> the
> > > >> drivers are willing to commit.
> > > >>
> > > >> RE #1,
> > > >>
> > > >> I am +1 on the fact that we shouldn't require an extra hop for range
> > scans.
> > > >>
> > > >> In JIRA Jeremiah made the point that you can still do this from the
> > client
> > > >> by breaking up the token ranges, but it's a leaky abstraction to
> have
> > a
> > > >> paging interface that isn't a vanilla ResultSet interface. Serial
> vs.
> > > >> parallel is kind 

Re: Evolving the client protocol

[Sankalp Kohli]

Is one of the “abuse” of Apache license is ScyllaDB which is using Cassandra 
but not contributing back?
Happy to be proved wrong as I am not a lawyer and don’t understand various 
licenses ..

> On Apr 23, 2018, at 16:55, Dor Laor  wrote:
> 
>> On Mon, Apr 23, 2018 at 4:13 PM, Jonathan Haddad  wrote:
>> 
>> From where I stand it looks like you've got only two options for any
>> feature that involves updating the protocol:
>> 
>> 1. Don't built the feature
>> 2. Built it in Cassanda & scylladb, update the drivers accordingly
>> 
>> I don't think you have a third option, which is built it only in ScyllaDB,
>> because that means you have to fork *all* the drivers and make it work,
>> then maintain them.  Your business model appears to be built on not doing
>> any of the driver work yourself, and you certainly aren't giving back to
>> the open source community via a permissive license on ScyllaDB itself, so
>> I'm a bit lost here.
>> 
> 
> It's totally not about business model.
> Scylla itself is 99% open source with AGPL license that prevents abuse and
> forces to be committed back to the project. We also have our core engine
> (seastar) licensed
> as Apache since it needs to be integrated with  the core application.
> Recently one of our community members even created a new Seastar based, C++
> driver.
> 
> Scylla chose to be compatible with the drivers in order to leverage the
> existing infrastructure
> and (let's be frank) in order to allow smooth migration.
> We would have loved to contribute more to the drivers but up to recently we:
> 1. Were busy on top of our heads with the server
> 2. Happy w/ the existing drivers
> 3. Developed extensions - GoCQLX - our own contribution
> 
> Finally we can contribute back to the same driver project, we want to do it
> the right way,
> without forking and without duplicated efforts.
> 
> Many times, having a private fork is way easier than proper open source
> work so from
> a pure business perspective, we don't select the shortest path.
> 
> 
>> 
>> To me it looks like you're asking a bunch of volunteers that work on
>> Cassandra to accommodate you.  What exactly do we get out of this
>> relationship?  What incentive do I or anyone else have to spend time
>> helping you instead of working on something that interests me?
>> 
> 
> Jon, this is certainty not the case.
> We genuinely wish to make true *open source* work on:
> a. Cassandra drivers
> b. Client protocol
> c. Scylla server side.
> d. Cassandra community related work: mailing list, Jira, design
> 
> But not
> e. Cassandra server side
> 
> While I wouldn't mind doing the Cassandra server work, we don't have the
> resources or
> the expertise. The Cassandra _developer_ community is welcome to decide
> whether
> we get to contribute a/b/c/d. Avi has enumerated the options of
> cooperation, passive cooperation
> and zero cooperation (below).
> 
> 1. The protocol change is developed using the Cassandra process in a JIRA
> ticket, culminating in a patch to doc/native_protocol*.spec when consensus
> is achieved.
> 2. The protocol change is developed outside the Cassandra process.
> 3. No cooperation.
> 
> Look, I can understand the hostility and suspicious, however, from the C*
> project POV, it makes no
> sense to ignore, otherwise we'll fork the drivers and you won't get
> anything back. There is another
> at least one vendor today with their server fork and driver fork and it
> makes sense to keep the protocol
> unified in an extensible way and to discuss new features _together_.
> 
> 
> 
>> 
>> Jon
>> 
>> 
>> On Mon, Apr 23, 2018 at 7:59 AM Ben Bromhead  wrote:
>> 
>> This doesn't work without additional changes, for RF>1. The token
>> ring
 could place two replicas of the same token range on the same physical
 server, even though those are two separate cores of the same server.
>> You
 could add another element to the hierarchy (cluster -> datacenter ->
>> rack
 -> node -> core/shard), but that generates unneeded range movements
>> when
>>> a
 node is added.
> I have seen rack awareness used/abused to solve this.
> 
 
 But then you lose real rack awareness. It's fine for a quick hack, but
 not a long-term solution.
 
 (it also creates a lot more tokens, something nobody needs)
 
>>> 
>>> I'm having trouble understanding how you loose "real" rack awareness, as
>>> these shards are in the same rack anyway, because the address and port
>> are
>>> on the same server in the same rack. So it behaves as expected. Could you
>>> explain a situation where the shards on a single server would be in
>>> different racks (or fault domains)?
>>> 
>>> If you wanted to support a situation where you have a single rack per DC
>>> for simple deployments, extending NetworkTopologyStrategy to behave the
>> way
>>> it did before https://issues.apache.org/jira/browse/CASSANDRA-7544 with
>>> respect to treating InetAddresses as servers rather than the address and
>>> port would 

Re: Evolving the client protocol

[Sankalp Kohli]

If you are so concerned about forking protocol, why did you fork the server? 
Please pick a side and not what is suitable for your Business. 




On Apr 23, 2018, at 17:28, Josh McKenzie  wrote:

>> it's not
>> reasonable to expect Scylla to contribute
>> such a huge effort to the C* server
> 
> But it's reasonable that a major portion of Scylla's business model is
> profiting off those huge efforts other companies have made?
> 
> Seems a little hypocritical to me.
> 
>> On Mon, Apr 23, 2018, 8:18 PM Dor Laor  wrote:
>> 
>> On Mon, Apr 23, 2018 at 5:03 PM, Sankalp Kohli 
>> wrote:
>> 
>>> Is one of the “abuse” of Apache license is ScyllaDB which is using
>>> Cassandra but not contributing back?
>>> 
>> 
>> It's not that we have a private version of Cassandra and we don't release
>> all of it or some of it back..
>> 
>> We didn't contribute because we have a different server base. We always
>> contribute where it makes sense.
>> I'll be happy to have several beers or emails about the cons and pros of
>> open source licensing but I don't think
>> this is the case. The discussion is about whether the community wish to
>> accept our contributions, we initiated it,
>> didn't we?
>> 
>> Let's be practical, I think it's not reasonable to commit C* protocol
>> changes that the community doesn't intend
>> to implement in C* in the short term (thread-per-core like), it's not
>> reasonable to expect Scylla to contribute
>> such a huge effort to the C* server. It is reasonable to collaborate around
>> protocol enhancements that are acceptable,
>> even without coding and make sure the protocol is enhanceable in a way that
>> forward compatible.
>> 
>> 
>> Happy to be proved wrong as I am not a lawyer and don’t understand various
>>> licenses ..
>>> 
>>>>> On Apr 23, 2018, at 16:55, Dor Laor  wrote:
>>>>> 
>>>>> On Mon, Apr 23, 2018 at 4:13 PM, Jonathan Haddad 
>>> wrote:
>>>>> 
>>>>> From where I stand it looks like you've got only two options for any
>>>>> feature that involves updating the protocol:
>>>>> 
>>>>> 1. Don't built the feature
>>>>> 2. Built it in Cassanda & scylladb, update the drivers accordingly
>>>>> 
>>>>> I don't think you have a third option, which is built it only in
>>> ScyllaDB,
>>>>> because that means you have to fork *all* the drivers and make it
>> work,
>>>>> then maintain them.  Your business model appears to be built on not
>>> doing
>>>>> any of the driver work yourself, and you certainly aren't giving back
>> to
>>>>> the open source community via a permissive license on ScyllaDB itself,
>>> so
>>>>> I'm a bit lost here.
>>>>> 
>>>> 
>>>> It's totally not about business model.
>>>> Scylla itself is 99% open source with AGPL license that prevents abuse
>>> and
>>>> forces to be committed back to the project. We also have our core
>> engine
>>>> (seastar) licensed
>>>> as Apache since it needs to be integrated with  the core application.
>>>> Recently one of our community members even created a new Seastar based,
>>> C++
>>>> driver.
>>>> 
>>>> Scylla chose to be compatible with the drivers in order to leverage the
>>>> existing infrastructure
>>>> and (let's be frank) in order to allow smooth migration.
>>>> We would have loved to contribute more to the drivers but up to
>> recently
>>> we:
>>>> 1. Were busy on top of our heads with the server
>>>> 2. Happy w/ the existing drivers
>>>> 3. Developed extensions - GoCQLX - our own contribution
>>>> 
>>>> Finally we can contribute back to the same driver project, we want to
>> do
>>> it
>>>> the right way,
>>>> without forking and without duplicated efforts.
>>>> 
>>>> Many times, having a private fork is way easier than proper open source
>>>> work so from
>>>> a pure business perspective, we don't select the shortest path.
>>>> 
>>>> 
>>>>> 
>>>>> To me it looks like you're asking a bunch of volunteers that work on
>>>>> Cassandra to accommodate you.  What exactly do we ge

Re: Improve the performance of CAS

[sankalp kohli]

Hi,
The idea of combining read with prepare sounds good. Regarding reducing
the commit round trip, it is possible today by giving a lower consistency
level for commit I think.

Regarding EPaxos, it is a large change and will take longer to land. I
think we should do this as it will help lower the latencies a lot.

Thanks,
Sankalp

On Wed, May 16, 2018 at 2:15 PM, Jeremy Hanna 
wrote:

> Hi Dikang,
>
> Have you seen Blake’s work on implementing egalitarian paxos or epaxos*?
> That might be helpful for the discussion.
>
> Jeremy
>
> * https://issues.apache.org/jira/browse/CASSANDRA-6246
>
> > On May 16, 2018, at 3:37 PM, Dikang Gu  wrote:
> >
> > Hello C* developers,
> >
> > I'm working on some performance improvements of the lightweight
> transitions
> > (compare and set), I'd like to hear your thoughts about it.
> >
> > As you know, current CAS requires 4 round trips to finish, which is not
> > efficient, especially in cross DC case.
> > 1) Prepare
> > 2) Quorum read current value
> > 3) Propose new value
> > 4) Commit
> >
> > I'm proposing the following improvements to reduce it to 2 round trips,
> > which is:
> > 1) Combine prepare and quorum read together, use only one round trip to
> > decide the ballot and also piggyback the current value in response.
> > 2) Propose new value, and then send out the commit request
> asynchronously,
> > so client will not wait for the ack of the commit. In case of commit
> > failures, we should still have chance to retry/repair it through hints or
> > following read/cas events.
> >
> > After the improvement, we should be able to finish the CAS operation
> using
> > 2 rounds trips. There can be following improvements as well, and this can
> > be a start point.
> >
> > What do you think? Did I miss anything?
> >
> > Thanks
> > Dikang
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: Tombstone passed GC period causes un-repairable inconsistent data

[sankalp kohli]

I agree with Stefan that we should use incremental repair and use patches
from Marcus to drop tombstones only from repaired data.
Regarding deep repair, you can bump the read repair and run the repair. The
issue will be that you will stream lot of data and also your blocking read
repair will go up when you bump the gc grace to higher value.

On Wed, Jun 20, 2018 at 1:10 AM Stefan Podkowinski  wrote:

> Sounds like an older issue that I tried to address two years ago:
> https://issues.apache.org/jira/browse/CASSANDRA-11427
>
> As you can see, the result hasn't been as expected and we got some
> unintended side effects based on the patch. I'm not sure I'd be willing
> to give this another try, considering the behaviour we like to fix in
> the first place is rather harmless and the read repairs shouldn't happen
> at all to any users who regularly run repairs within gc_grace.
>
> What I'd suggest is to think more into the direction of a
> post-full-repair-world and to fully embrace incremental repairs, as
> fixed by Blake in 4.0. In that case, we should stop doing read repairs
> at all for repaired data, as described in
> https://issues.apache.org/jira/browse/CASSANDRA-13912. RRs are certainly
> useful, but can be very risky if not very very carefully implemented. So
> I'm wondering if we shouldn't disable RRs for everything but unrepaired
> data. I'd btw also be interested to hear any opinions on this in context
> of transient replicas.
>
>
> On 20.06.2018 03:07, Jay Zhuang wrote:
> > Hi,
> >
> > We know that the deleted data may re-appear if repair is not run within
> > gc_grace_seconds. When the tombstone is not propagated to all nodes, the
> > data will re-appear. But it's also causing following 2 issues before the
> > tombstone is compacted away:
> > a. inconsistent query result
> >
> > With consistency level ONE or QUORUM, it may or may not return the value.
> > b. lots of read repairs, but doesn't repair anything
> >
> > With consistency level ALL, it always triggers a read repair.
> > With consistency level QUORUM, it also very likely (2/3) causes a read
> > repair. But it doesn't repair the data, so it's causing repair every
> time.
> >
> >
> > Here are the reproducing steps:
> >
> > 1. Create a 3 nodes cluster
> > 2. Create a table (with small gc_grace_seconds):
> >
> > CREATE KEYSPACE foo WITH replication = {'class': 'SimpleStrategy',
> > 'replication_factor': 3};
> > CREATE TABLE foo.bar (
> > id int PRIMARY KEY,
> > name text
> > ) WITH gc_grace_seconds=30;
> >
> > 3. Insert data with consistency all:
> >
> > INSERT INTO foo.bar (id, name) VALUES(1, 'cstar');
> >
> > 4. stop 1 node
> >
> > $ ccm node2 stop
> >
> > 5. Delete the data with consistency quorum:
> >
> > DELETE FROM foo.bar WHERE id=1;
> >
> > 6. Wait 30 seconds and then start node2:
> >
> > $ ccm node2 start
> >
> > Now the tombstone is on node1 and node3 but not on node2.
> >
> > With quorum read, it may or may not return value, and read repair will
> send
> > the data from node2 to node1 and node3, but it doesn't repair anything.
> >
> > I'd like to discuss a few potential solutions and workarounds:
> >
> > 1. Can hints replay sends GCed tombstone?
> >
> > 2. Can we have a "deep repair" which detects such issue and repair the
> GCed
> > tombstone? Or temperately increase the gc_grace_seconds for repair?
> >
> > What other suggestions you have if the user is having such issue?
> >
> >
> > Thanks,
> >
> > Jay
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Testing 4.0 Post-Freeze

[sankalp kohli]

Hi cassandra-dev@,

With the goal of making Cassandra's 4.0 the most stable major release to
date, we would like all committers of the project to consider joining us in
dedicating their time and attention to testing, running, and fixing issues
in 4.0 between the September freeze and the 4.0 beta release. This would
result in a freeze of new feature development on trunk or branches during
this period, instead focusing on writing, improving, and running tests or
fixing and reviewing bugs or performance regressions found in 4.0 or
earlier.

How would this work?

We propose that between the September freeze date and beta, a new branch
would not be created and trunk would only have bug fixes and performance
improvements committed to it. At the same time we do not want to discourage
community contributions. Not all contributors can be expected to be aware
of such a decision or may be new to the project. In cases where new
features are contributed during this time, the contributor can be informed
of the current status of the release process, be encouraged to contribute
to testing or bug fixing, and have their feature reviewed after the beta is
reached.


What happens when beta is reached?

Ideally, contributors who have made significant contributions to the
release will stick around to continue testing between beta and final
release. Any additional folks who continue this focus would also be greatly
appreciated.

What about before the freeze?

Testing new features is of course important. This isn't meant to discourage
development – only to enable us to focus on testing and hardening 4.0 to
deliver Cassandra's most stable major release. We would like to see
adoption of 4.0 happen much more quickly than its predecessor.

Thanks for considering this proposal,
Sankalp Kohli

Re: Testing 4.0 Post-Freeze

[sankalp kohli]

Having an explicit way to tell the community that we all will work on
testing is better than writing a patch which will sit without review for
months. I think not having your patch reviewed for months is more
discouraging than following the community and helping with stability of
4.0.



On Tue, Jul 3, 2018 at 3:02 PM Josh McKenzie  wrote:

> >
> > We propose that between the September freeze date and beta, a new branch
> > would not be created and trunk would only have bug fixes and performance
> > improvements committed to it.
>
>
> This is more of a call to action and announcement of intent than an attempt
> > to
> > enforce policy; we can and probably will branch off 4.0, and keep trunk
> > technically active.
>
> These are two very different statements. :) Which is it?
>
> On Tue, Jul 3, 2018 at 5:57 PM Aleksey Yeshchenko 
> wrote:
>
> > If we want to have a stable, usable 4.0.0 release out in the next 6-12
> > months, there needs to be a focused effort on getting it out - or else
> > it’ll just never happen.
> >
> > This is more of a call to action and announcement of intent than an
> > attempt to enforce policy; we can and probably will branch off 4.0, and
> > keep trunk technically active. But so long as there is a critical mass of
> > active contributors who are on board with only/mostly working on
> stability,
> > bug fixes, and validation - both as assignees and reviewers - we’ll have
> a
> > de-facto freeze.
> >
> > And I have a feeling that there is such a critical mass.
> >
> > —
> > AY
> >
> > On 3 July 2018 at 22:23:38, Jeff Jirsa (jji...@gmail.com) wrote:
> >
> > I think there's value in the psychological commitment that if someone has
> > time to contribute, their contributions should be focused on validating a
> > release, not pushing future features.
> >
> >
> > On Tue, Jul 3, 2018 at 1:03 PM, Jonathan Haddad 
> > wrote:
> >
> > > I agree with Josh. I don’t see how changing the convention around trunk
> > > will improve the process, seems like it’ll only introduce a handful of
> > > rollback commits when people forget.
> > >
> > > Other than that, it all makes sense to me.
> > >
> > > I’ve been working on a workload centric stress tool on and off for a
> > little
> > > bit in an effort to create something that will help with wider adoption
> > in
> > > stress testing. It differs from the stress we ship by including fully
> > > functional stress workloads as well as a validation process. The idea
> > being
> > > to be flexible enough to test both performance and correctness in LWT
> > and
> > > MVs as well as other arbitrary workloads.
> > >
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_thelastpickle_tlp-2Dstress&d=DwIFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=qK2RkRAsGtixYf0IgKlRBYLfTrXyOKED9OOTyMVvDf4&m=l_G2ByhfCyu3k9TzNVqiagdVQ8vOMJqHZvDq_JKvbiQ&s=f8gf_JCP6JRQIRkL_1R_zJOS_6gdAAsLleDr2PZHppE&e=
> >
> > >
> > > Jon
> > >
> > >
> > > On Tue, Jul 3, 2018 at 12:28 PM Josh McKenzie 
> > > wrote:
> > >
> > > > Why not just branch a 4.0-rel and bugfix there and merge up while
> > still
> > > > accepting new features or improvements on trunk?
> > > >
> > > > I don't think the potential extra engagement in testing will balance
> > out
> > > > the atrophy and discouraging contributions / community engagement
> > we'd
> > > get
> > > > by deferring all improvements and new features in an open-ended way.
> > > >
> > > > On Tue, Jul 3, 2018 at 1:33 PM sankalp kohli  >
> >
> > > > wrote:
> > > >
> > > > > Hi cassandra-dev@,
> > > > >
> > > > > With the goal of making Cassandra's 4.0 the most stable major
> > release
> > > to
> > > > > date, we would like all committers of the project to consider
> > joining
> > > us
> > > > in
> > > > > dedicating their time and attention to testing, running, and fixing
> > > > issues
> > > > > in 4.0 between the September freeze and the 4.0 beta release. This
> > > would
> > > > > result in a freeze of new feature development on trunk or branches
> > > during
> > > > > this period, instead focusing on writing, improving, and running
> > tests
> > > or
> > > > > fi

Re: Testing 4.0 Post-Freeze

[sankalp kohli]

Hi Kurt,
   Thank you for your feedback. I am reading your comment as +1 on
the idea of working on making a solid release but +0 on branching model. Is
that correct?
Thanks,
Sankalp

On Tue, Jul 3, 2018 at 10:09 PM kurt greaves  wrote:

> >
> > but by committing to reserving trunk for stabilizing changes until the
> > beta, we focus our effort on polishing and delivering the release.
>
> No, committing to testing, bug fixing, and validating focuses our effort on
> polishing and delivering the release.
> Committing to reserving trunk for stabilising changes doesn't actually do
> anything. We could reserve trunk and no one could ever commit a single
> patch to it, or we could reserve trunk and everyone continues working on
> features. Whatever we do to trunk doesn't have any impact on how our
> attention is directed.
>
> Granted, you could argue that there is some great beneficial psychological
> aspect and having no feature branch will make everyone suddenly start
> working on bug fixes, but to people this actually affects (committers),
> it's not going to mean much to them. They are either already going to be
> motivated and tasked with getting 4.0 out the door, or they are going to do
> their own thing.
>
> I'm fairly positive that better communication around the release, and
> progress towards the release would be a better motivator. And it would go
> further to getting new contributors (who definitely don't care about the
> branches) involved, because anyone on the ML could see progress getting
> made and be a part of any call to arms.
>
> But at the end of the day, bikeshedding about this is not important. This
> is one of those issues which someone should just stand up and say "We're
> doing it this way because I said so", because either way, it really doesn't
> matter and the impact is going to be minimal, so we may as well get on with
> our lives.
>
> On 4 July 2018 at 04:06, Scott Andreas  wrote:
>
> > Here’s why I really like this proposal:
> >
> > – It focuses our thought and effort on what it’ll take for each of us to
> > become confident in running Apache Cassandra 4.0 for our most critical
> > applications *prior* to cutting the dot-zero release.
> > – It marks a commitment we can make to our user community: delivering
> > 4.0’s featureset with safety and stability is our top priority.
> > – There are many different perspectives each contributor can bring:
> > correctness, performance, durability, automation, documentation,
> > operational safety, etc; and many ways to gain confidence in each – perf
> > tests, profiling, shadow writes/traffic mirroring, diff tests, replay
> > tests, fuzz tests, fault injection, chaos engineering, and so many
> others.
> > By bringing diverse methods to bear on the same class of problem
> (quality),
> > we’re more likely to identify issues that could impact our users and to
> > ship a better release.
> > – It’s disciplined and courageous!
> >
> > Here’s why I think this won’t discourage new contributors from joining –
> > in fact, the opposite:
> >
> > – It provides a very simple step that any contributor can take toward
> > shipping 4.0: running it.
> > – We raise spirits by delivering enhancements the community has been
> > working on for two years rather than fracturing our attention.
> > – If members of the community are interested in post-4.0 work, they’re
> not
> > blocked from making progress toward that – but it’s not the focus, and
> > unlikely that review bandwidth would be available given that focus. I
> agree
> > with Kurt’s note that nobody can be “forced" to do anything - but by
> > committing to reserving trunk for stabilizing changes until the beta, we
> > focus our effort on polishing and delivering the release.
> >
> > On the last question:
> > > On another note, we are still planning on accepting/reviewing bug fixes
> > for previous versions as well correct?
> >
> > I’d expect so – and to take it a step further, it’s likely that this
> rigor
> > will result in us identifying serious issues that were not regressions in
> > 4.0 warranting backports. As with any investment in testing / validation,
> > I’m hoping we do … but also hoping we don’t. :-)
> >
> >
> > > On Jul 3, 2018, at 7:21 PM, kurt greaves  wrote:
> > >
> > > tl;dr: +1 for committing to testing + bugfixing after feature freeze,
> > but I
> > > can't really see how changing the branching strategy is going to have
> any
> > > impact at all.
> > >
> > > I really don't think it's a big deal. People will work based on
> whatever
> > > priorities they've been assigned, regardless of what you do to the
> > > branching. That's essentially just red tape and adding unnecessary
> > > complexity to an already established process. Granted, you'll have to
> do
> > > one less merge, but it should be an effortless merge, and it saves
> there
> > > being any confusion about what people should do with features. If
> someone
> > > decided to commit a feature, they could, and we wouldn't have 

Re: Testing 4.0 Post-Freeze

[Sankalp Kohli]

Does anyone has any more feedback on this? 

> On Jul 4, 2018, at 05:36, Aleksey Yeshchenko  wrote:
> 
> For what it’s worth, I’m fine with both formal branching-level freeze and 
> informal ‘let people commit to trunk if they can find a reviewer’ one and 
> will support either.
> 
> So long as either/both are communicated to the contributors, the only 
> difference is in where new feature work gets accumulated: will stay a bit 
> longer in personal branches in the latter way.
> 
> —
> AY
> 
> On 4 July 2018 at 01:30:40, sankalp kohli (kohlisank...@gmail.com) wrote:
> 
> Having an explicit way to tell the community that we all will work on  
> testing is better than writing a patch which will sit without review for  
> months. I think not having your patch reviewed for months is more  
> discouraging than following the community and helping with stability of  
> 4.0.  
> 
> 
> 
> On Tue, Jul 3, 2018 at 3:02 PM Josh McKenzie  wrote:  
> 
>>> 
>>> We propose that between the September freeze date and beta, a new branch  
>>> would not be created and trunk would only have bug fixes and performance  
>>> improvements committed to it.  
>> 
>> 
>> This is more of a call to action and announcement of intent than an attempt  
>>> to  
>>> enforce policy; we can and probably will branch off 4.0, and keep trunk  
>>> technically active.  
>> 
>> These are two very different statements. :) Which is it?  
>> 
>> On Tue, Jul 3, 2018 at 5:57 PM Aleksey Yeshchenko   
>> wrote:  
>> 
>>> If we want to have a stable, usable 4.0.0 release out in the next 6-12  
>>> months, there needs to be a focused effort on getting it out - or else  
>>> it’ll just never happen.  
>>> 
>>> This is more of a call to action and announcement of intent than an  
>>> attempt to enforce policy; we can and probably will branch off 4.0, and  
>>> keep trunk technically active. But so long as there is a critical mass of  
>>> active contributors who are on board with only/mostly working on  
>> stability,  
>>> bug fixes, and validation - both as assignees and reviewers - we’ll have  
>> a  
>>> de-facto freeze.  
>>> 
>>> And I have a feeling that there is such a critical mass.  
>>> 
>>> —  
>>> AY  
>>> 
>>> On 3 July 2018 at 22:23:38, Jeff Jirsa (jji...@gmail.com) wrote:  
>>> 
>>> I think there's value in the psychological commitment that if someone has  
>>> time to contribute, their contributions should be focused on validating a  
>>> release, not pushing future features.  
>>> 
>>> 
>>> On Tue, Jul 3, 2018 at 1:03 PM, Jonathan Haddad   
>>> wrote:  
>>> 
>>>> I agree with Josh. I don’t see how changing the convention around trunk  
>>>> will improve the process, seems like it’ll only introduce a handful of  
>>>> rollback commits when people forget.  
>>>> 
>>>> Other than that, it all makes sense to me.  
>>>> 
>>>> I’ve been working on a workload centric stress tool on and off for a  
>>> little  
>>>> bit in an effort to create something that will help with wider adoption  
>>> in  
>>>> stress testing. It differs from the stress we ship by including fully  
>>>> functional stress workloads as well as a validation process. The idea  
>>> being  
>>>> to be flexible enough to test both performance and correctness in LWT  
>>> and  
>>>> MVs as well as other arbitrary workloads.  
>>>> 
>>>> 
>>> 
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_thelastpickle_tlp-2Dstress&d=DwIFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=qK2RkRAsGtixYf0IgKlRBYLfTrXyOKED9OOTyMVvDf4&m=l_G2ByhfCyu3k9TzNVqiagdVQ8vOMJqHZvDq_JKvbiQ&s=f8gf_JCP6JRQIRkL_1R_zJOS_6gdAAsLleDr2PZHppE&e=
>>   
>>> 
>>>> 
>>>> Jon  
>>>> 
>>>> 
>>>> On Tue, Jul 3, 2018 at 12:28 PM Josh McKenzie   
>>>> wrote:  
>>>> 
>>>>> Why not just branch a 4.0-rel and bugfix there and merge up while  
>>> still  
>>>>> accepting new features or improvements on trunk?  
>>>>> 
>>>>> I don't think the potential extra engagement in testing will balance  
>>> out  
>>>>> the atrophy and discouraging contributions / community engagement  
>>> we'd  
>>>> get  
>>>>> by deferring all impro

Re: Testing 4.0 Post-Freeze

[sankalp kohli]

I did not see a -1 but all +0s and a few +1s.

On Mon, Jul 9, 2018 at 5:49 AM Josh McKenzie  wrote:

> What did we land on? Sounds like we're pretty split without consensus on
> "change the old branching strategy and reject new things on trunk during
> stabilization" vs. "keep doing things the way we did but message strongly
> that we won't be reviewing new things until 4.0 is stable".
>
> On Sat, Jul 7, 2018 at 5:01 PM Sankalp Kohli 
> wrote:
>
> > Does anyone has any more feedback on this?
> >
> > > On Jul 4, 2018, at 05:36, Aleksey Yeshchenko 
> wrote:
> > >
> > > For what it’s worth, I’m fine with both formal branching-level freeze
> > and informal ‘let people commit to trunk if they can find a reviewer’ one
> > and will support either.
> > >
> > > So long as either/both are communicated to the contributors, the only
> > difference is in where new feature work gets accumulated: will stay a bit
> > longer in personal branches in the latter way.
> > >
> > > —
> > > AY
> > >
> > > On 4 July 2018 at 01:30:40, sankalp kohli (kohlisank...@gmail.com)
> > wrote:
> > >
> > > Having an explicit way to tell the community that we all will work on
> > > testing is better than writing a patch which will sit without review
> > for
> > > months. I think not having your patch reviewed for months is more
> > > discouraging than following the community and helping with stability of
> > > 4.0.
> > >
> > >
> > >
> > > On Tue, Jul 3, 2018 at 3:02 PM Josh McKenzie 
> > wrote:
> > >
> > >>>
> > >>> We propose that between the September freeze date and beta, a new
> > branch
> > >>> would not be created and trunk would only have bug fixes and
> > performance
> > >>> improvements committed to it.
> > >>
> > >>
> > >> This is more of a call to action and announcement of intent than an
> > attempt
> > >>> to
> > >>> enforce policy; we can and probably will branch off 4.0, and keep
> > trunk
> > >>> technically active.
> > >>
> > >> These are two very different statements. :) Which is it?
> > >>
> > >> On Tue, Jul 3, 2018 at 5:57 PM Aleksey Yeshchenko 
> > >> wrote:
> > >>
> > >>> If we want to have a stable, usable 4.0.0 release out in the next
> > 6-12
> > >>> months, there needs to be a focused effort on getting it out - or
> > else
> > >>> it’ll just never happen.
> > >>>
> > >>> This is more of a call to action and announcement of intent than an
> > >>> attempt to enforce policy; we can and probably will branch off 4.0,
> > and
> > >>> keep trunk technically active. But so long as there is a critical
> mass
> > of
> > >>> active contributors who are on board with only/mostly working on
> > >> stability,
> > >>> bug fixes, and validation - both as assignees and reviewers - we’ll
> > have
> > >> a
> > >>> de-facto freeze.
> > >>>
> > >>> And I have a feeling that there is such a critical mass.
> > >>>
> > >>> —
> > >>> AY
> > >>>
> > >>> On 3 July 2018 at 22:23:38, Jeff Jirsa (jji...@gmail.com) wrote:
> > >>>
> > >>> I think there's value in the psychological commitment that if someone
> > has
> > >>> time to contribute, their contributions should be focused on
> > validating a
> > >>> release, not pushing future features.
> > >>>
> > >>>
> > >>> On Tue, Jul 3, 2018 at 1:03 PM, Jonathan Haddad 
> > >>> wrote:
> > >>>
> > >>>> I agree with Josh. I don’t see how changing the convention around
> > trunk
> > >>>> will improve the process, seems like it’ll only introduce a handful
> > of
> > >>>> rollback commits when people forget.
> > >>>>
> > >>>> Other than that, it all makes sense to me.
> > >>>>
> > >>>> I’ve been working on a workload centric stress tool on and off for a
> > >>> little
> > >>>> bit in an effort to create something that will help with wider
> > adoption
> > >>> in
> > >>>> stress testing. It differs from the stress we s

Re: Testing 4.0 Post-Freeze

[sankalp kohli]

How is this preventing someone from working and collaborating on an Apache
Project? You can still collaborate on making 4.0 more stable.

Why will these committers not work on making 4.0 more stable and fix broken
tests? Considering  we cannot release without test passing, how will these
features be used?

On Mon, Jul 9, 2018 at 10:03 AM Jonathan Haddad  wrote:

> I don't see how changing the process and banning feature commits is
> going to be any help to the project. There may be a couple committers
> who are interested in committing new features.
>
> I'm a -1 on changing the branching strategy in a way that prevents
> people from working and collaborating on an Apache project.
>
> On Mon, Jul 9, 2018 at 9:56 AM sankalp kohli 
> wrote:
> >
> > I did not see a -1 but all +0s and a few +1s.
> >
> > On Mon, Jul 9, 2018 at 5:49 AM Josh McKenzie 
> wrote:
> >
> > > What did we land on? Sounds like we're pretty split without consensus
> on
> > > "change the old branching strategy and reject new things on trunk
> during
> > > stabilization" vs. "keep doing things the way we did but message
> strongly
> > > that we won't be reviewing new things until 4.0 is stable".
> > >
> > > On Sat, Jul 7, 2018 at 5:01 PM Sankalp Kohli 
> > > wrote:
> > >
> > > > Does anyone has any more feedback on this?
> > > >
> > > > > On Jul 4, 2018, at 05:36, Aleksey Yeshchenko 
> > > wrote:
> > > > >
> > > > > For what it’s worth, I’m fine with both formal branching-level
> freeze
> > > > and informal ‘let people commit to trunk if they can find a
> reviewer’ one
> > > > and will support either.
> > > > >
> > > > > So long as either/both are communicated to the contributors, the
> only
> > > > difference is in where new feature work gets accumulated: will stay
> a bit
> > > > longer in personal branches in the latter way.
> > > > >
> > > > > —
> > > > > AY
> > > > >
> > > > > On 4 July 2018 at 01:30:40, sankalp kohli (kohlisank...@gmail.com)
> > > > wrote:
> > > > >
> > > > > Having an explicit way to tell the community that we all will work
> on
> > > > > testing is better than writing a patch which will sit without
> review
> > > > for
> > > > > months. I think not having your patch reviewed for months is more
> > > > > discouraging than following the community and helping with
> stability of
> > > > > 4.0.
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Jul 3, 2018 at 3:02 PM Josh McKenzie  >
> > > > wrote:
> > > > >
> > > > >>>
> > > > >>> We propose that between the September freeze date and beta, a new
> > > > branch
> > > > >>> would not be created and trunk would only have bug fixes and
> > > > performance
> > > > >>> improvements committed to it.
> > > > >>
> > > > >>
> > > > >> This is more of a call to action and announcement of intent than
> an
> > > > attempt
> > > > >>> to
> > > > >>> enforce policy; we can and probably will branch off 4.0, and keep
> > > > trunk
> > > > >>> technically active.
> > > > >>
> > > > >> These are two very different statements. :) Which is it?
> > > > >>
> > > > >> On Tue, Jul 3, 2018 at 5:57 PM Aleksey Yeshchenko <
> alek...@apple.com>
> > > > >> wrote:
> > > > >>
> > > > >>> If we want to have a stable, usable 4.0.0 release out in the next
> > > > 6-12
> > > > >>> months, there needs to be a focused effort on getting it out - or
> > > > else
> > > > >>> it’ll just never happen.
> > > > >>>
> > > > >>> This is more of a call to action and announcement of intent than
> an
> > > > >>> attempt to enforce policy; we can and probably will branch off
> 4.0,
> > > > and
> > > > >>> keep trunk technically active. But so long as there is a critical
> > > mass
> > > > of
> > > > >>> active contributors who are on board with only/mostly working on
> > > > >> stability,

Re: Testing 4.0 Post-Freeze

[sankalp kohli]

If some committers want to bring in features without a path forward for
releasing(as tests are broken), is that an acceptable state for you? How do
we get out of this state.

Like I said in my original email, this is a "proposal" and I am trying to
see how we can improve things here. So if you are -1 on this, please help
us propose something else to get these tests pass?

And thanks for giving your feedback.

On Mon, Jul 9, 2018 at 10:50 AM Jonathan Haddad  wrote:

> Not everyone wants to work and collaborate the way you do.  It seems
> absurd to me to force everyone to hold off on merging into a single
> branch just because a lot of us want to prioritize testing 4.0.
>
> I think most committers are going to want to contribute to the 4.0
> testing process more than they want to commit new features to trunk,
> but I also think people shouldn't be banned from new bringing in new
> features if that's what they want to do.
>
> You're essentially giving a blanket -1 to all new features for a 3-6
> month period.
> On Mon, Jul 9, 2018 at 10:44 AM sankalp kohli 
> wrote:
> >
> > How is this preventing someone from working and collaborating on an
> Apache
> > Project? You can still collaborate on making 4.0 more stable.
> >
> > Why will these committers not work on making 4.0 more stable and fix
> broken
> > tests? Considering  we cannot release without test passing, how will
> these
> > features be used?
> >
> > On Mon, Jul 9, 2018 at 10:03 AM Jonathan Haddad 
> wrote:
> >
> > > I don't see how changing the process and banning feature commits is
> > > going to be any help to the project. There may be a couple committers
> > > who are interested in committing new features.
> > >
> > > I'm a -1 on changing the branching strategy in a way that prevents
> > > people from working and collaborating on an Apache project.
> > >
> > > On Mon, Jul 9, 2018 at 9:56 AM sankalp kohli 
> > > wrote:
> > > >
> > > > I did not see a -1 but all +0s and a few +1s.
> > > >
> > > > On Mon, Jul 9, 2018 at 5:49 AM Josh McKenzie 
> > > wrote:
> > > >
> > > > > What did we land on? Sounds like we're pretty split without
> consensus
> > > on
> > > > > "change the old branching strategy and reject new things on trunk
> > > during
> > > > > stabilization" vs. "keep doing things the way we did but message
> > > strongly
> > > > > that we won't be reviewing new things until 4.0 is stable".
> > > > >
> > > > > On Sat, Jul 7, 2018 at 5:01 PM Sankalp Kohli <
> kohlisank...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Does anyone has any more feedback on this?
> > > > > >
> > > > > > > On Jul 4, 2018, at 05:36, Aleksey Yeshchenko <
> alek...@apple.com>
> > > > > wrote:
> > > > > > >
> > > > > > > For what it’s worth, I’m fine with both formal branching-level
> > > freeze
> > > > > > and informal ‘let people commit to trunk if they can find a
> > > reviewer’ one
> > > > > > and will support either.
> > > > > > >
> > > > > > > So long as either/both are communicated to the contributors,
> the
> > > only
> > > > > > difference is in where new feature work gets accumulated: will
> stay
> > > a bit
> > > > > > longer in personal branches in the latter way.
> > > > > > >
> > > > > > > —
> > > > > > > AY
> > > > > > >
> > > > > > > On 4 July 2018 at 01:30:40, sankalp kohli (
> kohlisank...@gmail.com)
> > > > > > wrote:
> > > > > > >
> > > > > > > Having an explicit way to tell the community that we all will
> work
> > > on
> > > > > > > testing is better than writing a patch which will sit without
> > > review
> > > > > > for
> > > > > > > months. I think not having your patch reviewed for months is
> more
> > > > > > > discouraging than following the community and helping with
> > > stability of
> > > > > > > 4.0.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jul 3, 2018 at 3:02 PM Josh McKenzie &l

Re: Testing 4.0 Post-Freeze

[sankalp kohli]

We have done all this for previous releases and we know it has not worked
well. So how giving it one more try is going to help here. Can someone
outline what will change for 4.0 which will make it more successful?

Not branching is one idea but we should discuss what will change now than
iterating what has already happened.

On Mon, Jul 9, 2018 at 11:25 AM Jeremy Hanna 
wrote:

> I wholeheartedly agree with efforts to make releases more stable and the
> more contributors that add tests or test within the context of their own
> use cases, the better.  +1 non-binding to the idea of better, more complete
> testing for releases.
>
> When it comes to how to do this, I think that’s where there is
> disagreement.  I personally don’t think that branching detracts from the
> goal of stabilization.  There are a couple of personas involved here:
>
> * Longtime contributors/committers: I think it’s sufficient to generally
> ask that whatever time/effort they can contribute be towards stabilizing,
> testing, and especially testing their production scenarios to cover more
> surface area when it comes to usage edge cases.  That along with testing
> longer running things in those scenarios for other types of edge cases.
>
> * New contributors: They likely won’t know about the strategy.  They’re
> just poking around and looking at lhf tickets or tickets that they need
> done.  Those are typically low risk but at the same time we don’t want to
> compromise stability by merging those in.  Reviewing low risk stuff for
> inclusion doesn’t take a ton of time and gives them a sense that they can
> be of help and empowers them.  After they have that first experience, then
> a longer term contributor could share with them a blog post or tldr thread
> summary about the 4.0 stabilization efforts (would be nice to have one to
> point people too once we're done).  We could also start creating lhf
> tickets with stabilization and testing in mind.
>
> Unless we want to make a fundamental change to the project’s process
> (making trunk stable at all times going forward), then I don’t think
> there’s a reason why branching would detract from these efforts.  In other
> words if we’re just trying to say that we want to focus on stabilization
> for the alpha/beta/rc of 4.0, then I would prefer branching along with
> clear messaging.
>
> Jeremy
>
> > On Jul 9, 2018, at 12:43 PM, sankalp kohli 
> wrote:
> >
> > How is this preventing someone from working and collaborating on an
> Apache
> > Project? You can still collaborate on making 4.0 more stable.
> >
> > Why will these committers not work on making 4.0 more stable and fix
> broken
> > tests? Considering  we cannot release without test passing, how will
> these
> > features be used?
> >
> > On Mon, Jul 9, 2018 at 10:03 AM Jonathan Haddad 
> wrote:
> >
> >> I don't see how changing the process and banning feature commits is
> >> going to be any help to the project. There may be a couple committers
> >> who are interested in committing new features.
> >>
> >> I'm a -1 on changing the branching strategy in a way that prevents
> >> people from working and collaborating on an Apache project.
> >>
> >> On Mon, Jul 9, 2018 at 9:56 AM sankalp kohli 
> >> wrote:
> >>>
> >>> I did not see a -1 but all +0s and a few +1s.
> >>>
> >>> On Mon, Jul 9, 2018 at 5:49 AM Josh McKenzie 
> >> wrote:
> >>>
> >>>> What did we land on? Sounds like we're pretty split without consensus
> >> on
> >>>> "change the old branching strategy and reject new things on trunk
> >> during
> >>>> stabilization" vs. "keep doing things the way we did but message
> >> strongly
> >>>> that we won't be reviewing new things until 4.0 is stable".
> >>>>
> >>>> On Sat, Jul 7, 2018 at 5:01 PM Sankalp Kohli 
> >>>> wrote:
> >>>>
> >>>>> Does anyone has any more feedback on this?
> >>>>>
> >>>>>> On Jul 4, 2018, at 05:36, Aleksey Yeshchenko 
> >>>> wrote:
> >>>>>>
> >>>>>> For what it’s worth, I’m fine with both formal branching-level
> >> freeze
> >>>>> and informal ‘let people commit to trunk if they can find a
> >> reviewer’ one
> >>>>> and will support either.
> >>>>>>
> >>>>>> So long as either/both are communicated to the contributors, the
> >> only
> >>>>> difference is in where 

Re: Testing 4.0 Post-Freeze

[sankalp kohli]

Sure...thats an addition to the 3 states I was thinking(-1,+/-0 and +1) :)

On Mon, Jul 9, 2018 at 12:49 PM Josh McKenzie  wrote:

> >
> > I did not see a -1 but all +0s and a few +1s.
>
> It's more accurate to say you have quite a few -0's, some +0's, and a few
> +1's, probably some -0.5's if you're into that.
>
> On Mon, Jul 9, 2018 at 2:53 PM Jeremy Hanna 
> wrote:
>
> > What I’m saying is that for new contributors, not branching has a cost
> and
> > I think there are other ways to organize efforts.
> >
> > One of the suggestions was for each organization to test their own use
> > cases in the stabilization process.  I don’t think that’s been done very
> > effectively previously.  Most organizations only do any sort of
> significant
> > testing when they’re about to put their clusters on the line - i.e. once
> a
> > version has several post GA point releases.  That’s logical and no one
> > wants to be the first one to production.  However, if people were to
> agree
> > to do some form of testing pre-release, then I think that would be a step
> > in the right direction.  In the early days of the project I tried to do
> > this in a small way by testing the Hadoop support for every release
> (major
> > and minor) since it wasn’t on everyone’s radar but was important to me.
> > Then I would vote with a non-binding vote and described what was tested.
> >
> > > On Jul 9, 2018, at 1:30 PM, sankalp kohli 
> > wrote:
> > >
> > > We have done all this for previous releases and we know it has not
> worked
> > > well. So how giving it one more try is going to help here. Can someone
> > > outline what will change for 4.0 which will make it more successful?
> > >
> > > Not branching is one idea but we should discuss what will change now
> than
> > > iterating what has already happened.
> > >
> > > On Mon, Jul 9, 2018 at 11:25 AM Jeremy Hanna <
> jeremy.hanna1...@gmail.com
> > >
> > > wrote:
> > >
> > >> I wholeheartedly agree with efforts to make releases more stable and
> the
> > >> more contributors that add tests or test within the context of their
> own
> > >> use cases, the better.  +1 non-binding to the idea of better, more
> > complete
> > >> testing for releases.
> > >>
> > >> When it comes to how to do this, I think that’s where there is
> > >> disagreement.  I personally don’t think that branching detracts from
> the
> > >> goal of stabilization.  There are a couple of personas involved here:
> > >>
> > >> * Longtime contributors/committers: I think it’s sufficient to
> generally
> > >> ask that whatever time/effort they can contribute be towards
> > stabilizing,
> > >> testing, and especially testing their production scenarios to cover
> more
> > >> surface area when it comes to usage edge cases.  That along with
> testing
> > >> longer running things in those scenarios for other types of edge
> cases.
> > >>
> > >> * New contributors: They likely won’t know about the strategy.
> They’re
> > >> just poking around and looking at lhf tickets or tickets that they
> need
> > >> done.  Those are typically low risk but at the same time we don’t want
> > to
> > >> compromise stability by merging those in.  Reviewing low risk stuff
> for
> > >> inclusion doesn’t take a ton of time and gives them a sense that they
> > can
> > >> be of help and empowers them.  After they have that first experience,
> > then
> > >> a longer term contributor could share with them a blog post or tldr
> > thread
> > >> summary about the 4.0 stabilization efforts (would be nice to have one
> > to
> > >> point people too once we're done).  We could also start creating lhf
> > >> tickets with stabilization and testing in mind.
> > >>
> > >> Unless we want to make a fundamental change to the project’s process
> > >> (making trunk stable at all times going forward), then I don’t think
> > >> there’s a reason why branching would detract from these efforts.  In
> > other
> > >> words if we’re just trying to say that we want to focus on
> stabilization
> > >> for the alpha/beta/rc of 4.0, then I would prefer branching along with
> > >> clear messaging.
> > >>
> > >> Jeremy
> > >>
> > >>> On Jul 9, 2018, at 12:43 PM, sankal

[VOTE] Branching Change for 4.0 Freeze

[sankalp kohli]

Hi,
As discussed in the thread[1], we are proposing that we will not branch
on 1st September but will only allow following merges into trunk.

a. Bug and Perf fixes to 4.0.
b. Critical bugs in any version of C*.
c. Testing changes to help test 4.0

If someone has a change which does not fall under these three, we can
always discuss it and have an exception.

Vote will be open for 72 hours.

Thanks,
Sankalp

[1]
https://lists.apache.org/thread.html/494c3ced9e83ceeb53fa127e44eec6e2588a01b769896b25867fd59f@%3Cdev.cassandra.apache.org%3E

Re: [VOTE] Branching Change for 4.0 Freeze

[sankalp kohli]

We will be in this state till beta is reached.

On Wed, Jul 11, 2018 at 2:46 PM sankalp kohli 
wrote:

> Hi,
> As discussed in the thread[1], we are proposing that we will not
> branch on 1st September but will only allow following merges into trunk.
>
> a. Bug and Perf fixes to 4.0.
> b. Critical bugs in any version of C*.
> c. Testing changes to help test 4.0
>
> If someone has a change which does not fall under these three, we can
> always discuss it and have an exception.
>
> Vote will be open for 72 hours.
>
> Thanks,
> Sankalp
>
> [1]
> https://lists.apache.org/thread.html/494c3ced9e83ceeb53fa127e44eec6e2588a01b769896b25867fd59f@%3Cdev.cassandra.apache.org%3E
>

Re: [VOTE] Branching Change for 4.0 Freeze

[sankalp kohli]

merging non code will be allowed correct

On Thu, Jul 12, 2018 at 9:41 AM Stefan Podkowinski  wrote:

> +1
>
> (assuming merging patches on documentation will always be possible, as
> it's not effecting the code base)
>
>
> On 11.07.18 23:46, sankalp kohli wrote:
> > Hi,
> > As discussed in the thread[1], we are proposing that we will not
> branch
> > on 1st September but will only allow following merges into trunk.
> >
> > a. Bug and Perf fixes to 4.0.
> > b. Critical bugs in any version of C*.
> > c. Testing changes to help test 4.0
> >
> > If someone has a change which does not fall under these three, we can
> > always discuss it and have an exception.
> >
> > Vote will be open for 72 hours.
> >
> > Thanks,
> > Sankalp
> >
> > [1]
> >
> https://lists.apache.org/thread.html/494c3ced9e83ceeb53fa127e44eec6e2588a01b769896b25867fd59f@%3Cdev.cassandra.apache.org%3E
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: [VOTE] Branching Change for 4.0 Freeze

[sankalp kohli]

Vote passed with 15 +1s, 3 -0s and 1 +0

On Sat, Jul 14, 2018 at 8:48 PM Dinesh Joshi 
wrote:

> +1
>
> > On Jul 13, 2018, at 7:48 PM, Sumanth Pasupuleti 
> > 
> wrote:
> >
> > +1 nb
> > On Fri, Jul 13, 2018 at 6:58 PM Jaydeep Chovatia <
> chovatia.jayd...@gmail.com>
> > wrote:
> >
> >> +1
> >>
> >> On Wed, Jul 11, 2018 at 2:46 PM sankalp kohli 
> >> wrote:
> >>
> >>> Hi,
> >>>As discussed in the thread[1], we are proposing that we will not
> >> branch
> >>> on 1st September but will only allow following merges into trunk.
> >>>
> >>> a. Bug and Perf fixes to 4.0.
> >>> b. Critical bugs in any version of C*.
> >>> c. Testing changes to help test 4.0
> >>>
> >>> If someone has a change which does not fall under these three, we can
> >>> always discuss it and have an exception.
> >>>
> >>> Vote will be open for 72 hours.
> >>>
> >>> Thanks,
> >>> Sankalp
> >>>
> >>> [1]
> >>>
> >>>
> >>
> https://lists.apache.org/thread.html/494c3ced9e83ceeb53fa127e44eec6e2588a01b769896b25867fd59f@%3Cdev.cassandra.apache.org%3E
> >>>
> >>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

JIRAs in Review

[sankalp kohli]

Hi,
We are 7 weeks away from 4.0 freeze and there are ~150 JIRAs waiting
for review. It is hard to know which ones should be prioritized as some of
them could be not valid(fixes 2.0 bug), some of them will have the assignee
who no longer is active, etc.

If anyone is *not* getting traction on the JIRA to get it reviewed, please
use this thread to send your JIRA number and optionally why it is
important.

Thanks,
Sankalp

Apache Cassandra Blog is now live

[sankalp kohli]

Hi,
 Apache Cassandra Blog is now live. Check out the first blog post.

http://cassandra.apache.org/blog/2018/08/07/faster_streaming_in_cassandra.html

Thanks,
Sankalp

Re: Proposing an Apache Cassandra Management process

[Sankalp Kohli]

I am bumping this thread because patch has landed for this with repair 
functionality. 

I have a following proposal for this which I can put in the JIRA or doc 

1. We should see if we can keep this in a separate repo like Dtest. 
2. It should have its own release process.
3. It should have integration tests for different versions of Cassandra it will 
support. 

Does anyone has any ideas on this ? 

Thanks
Sankalp 

> On Apr 18, 2018, at 19:20, Dinesh Joshi  
> wrote:
> 
> Thank you all for the feedback. Nate made a Google doc with the proposal in 
> it and is linked off of the ticket. I will try to flesh it out as I gather 
> your input.
> Dinesh 
> 
>On Wednesday, April 18, 2018, 5:12:49 PM PDT, kurt greaves 
>  wrote:  
> 
> For anyone looking Dinesh made a ticket already: CASSANDRA-14395
> 
> 
>> On 18 April 2018 at 18:14, Vinay Chella  wrote:
>> 
>> This is a good initiative. We have advocated for and run a sidecar for the
>> past 5+ years, and we've learned and improved it a lot. We look forward to
>> moving features from Priam (such as backup, HTTP -> JMX, etc) incrementally
>> to this sidecar as they make sense.
>> 
>> 
>> Thanks,
>> Vinay Chella
>> 
>> On Fri, Apr 13, 2018 at 7:01 AM, Eric Evans 
>> wrote:
>> 
>>> On Thu, Apr 12, 2018 at 4:41 PM, Dinesh Joshi
>>>  wrote:
 Hey all -
 With the uptick in discussion around Cassandra operability and after
>>> discussing potential solutions with various members of the community, we
>>> would like to propose the addition of a management process/sub-project
>> into
>>> Apache Cassandra. The process would be responsible for common operational
>>> tasks like bulk execution of nodetool commands, backup/restore, and
>> health
>>> checks, among others. We feel we have a proposal that will garner some
>>> discussion and debate but is likely to reach consensus.
 While the community, in large part, agrees that these features should
>>> exist “in the database”, there is debate on how they should be
>> implemented.
>>> Primarily, whether or not to use an external process or build on
>>> CassandraDaemon. This is an important architectural decision but we feel
>>> the most critical aspect is not where the code runs but that the operator
>>> still interacts with the notion of a single database. Multi-process
>>> databases are as old as Postgres and continue to be common in newer
>> systems
>>> like Druid. As such, we propose a separate management process for the
>>> following reasons:
 
 - Resource isolation & Safety: Features in the management process
>>> will not affect C*'s read/write path which is critical for stability. An
>>> isolated process has several technical advantages including preventing
>> use
>>> of unnecessary dependencies in CassandraDaemon, separation of JVM
>> resources
>>> like thread pools and heap, and preventing bugs from adversely affecting
>>> the main process. In particular, GC tuning can be done separately for the
>>> two processes, hopefully helping to improve, or at least not adversely
>>> affect, tail latencies of the main process.
 
 - Health Checks & Recovery: Currently users implement health checks
>>> in their own sidecar process. Implementing them in the serving process
>> does
>>> not make sense because if the JVM running the CassandraDaemon goes south,
>>> the healthchecks and potentially any recovery code may not be able to
>> run.
>>> Having a management process running in isolation opens up the possibility
>>> to not only report the health of the C* process such as long GC pauses or
>>> stuck JVM but also to recover from it. Having a list of basic health
>> checks
>>> that are tested with every C* release and officially supported will help
>>> boost confidence in C* quality and make it easier to operate.
 
 - Reduced Risk: By having a separate Daemon we open the possibility
>>> to contribute features that otherwise would not have been considered
>> before
>>> eg. a UI. A library that started many background threads and is operated
>>> completely differently would likely be considered too risky for
>>> CassandraDaemon but is a good candidate for the management process.
>>> 
>>> Makes sense IMO.
>>> 
 What can go into the management process?
 - Features that are non-essential for serving reads & writes for eg.
>>> Backup/Restore or Running Health Checks against the CassandraDaemon, etc.
 
 - Features that do not make the management process critical for
>>> functioning of the serving process. In other words, if someone does not
>>> wish to use this management process, they are free to disable it.
 
 We would like to initially build minimal set of features such as health
>>> checks and bulk commands into the first iteration of the management
>>> process. We would use the same software stack that is used to build the
>>> current CassandraDaemon binary. This would be critical for sharing code
>>

Re: Proposing an Apache Cassandra Management process

[Sankalp Kohli]

The thread for side car is months old and no one has opposed to it and hence 
someone developed it. I am not sure how else you get consensus. 

Regarding separate repo, how do we get consensus? 

> On Aug 18, 2018, at 05:19, Stefan Podkowinski  wrote:
> 
> I don't see that we have reached sufficient consensus on this yet. We've
> had a brief discussion about the pros and cons of in-tree cassandra vs
> separate ASF repo here, but let's not frame it like it's either or. From
> my perspective, there hasn't been any final decision yet whether the
> proposed side-car solution should be further developed as part of the
> Cassandra project, or not.
> 
> 
>> On 18.08.18 03:12, Dinesh Joshi wrote:
>> Thanks, Nate. I’ll create this request.
>> 
>> Dinesh
>> 
>> On Aug 17, 2018, at 5:09 PM, Nate McCall  wrote:
>> 
 I'm not sure logistically how we get a new repo created and licensing and
 such, but if someone helps make it we can cut the patch against it
 
>>> This is pretty straight forward. For precedent, see:
>>> https://issues.apache.org/jira/browse/CASSANDRA-13634
>>> 
>>> We currently have three repositories:
>>> https://git-wip-us.apache.org/repos/asf
>>> 
>>> I'm +0 on what approach we take.
>>> 
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>> 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 
> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[Sankalp Kohli]

That’s a good point. Let’s review the patch and when it is ready to commit, we 
can create the repo then. 

> On Aug 18, 2018, at 14:04, Stefan Podkowinski  wrote:
> 
> I think we do have some consensus that 1) we should give it a try to
> have one or many side-car processes for non-essential features, and that
> 2) they should be developed in a separate repo. I'm also open to the
> idea of accepting the proposed implementation as a possible side-car
> solution and really appreciate effort. But my point is that creating a
> new repo, just for the patch, seems to imply that it's also going to
> become the de facto official side-car solution, which doesn't feel right
> to me, given that the proposed patch isn't even reviewed and hasn't
> received much feedback yet.
> 
> 
>> On 18.08.18 17:44, Sankalp Kohli wrote:
>> The thread for side car is months old and no one has opposed to it and hence 
>> someone developed it. I am not sure how else you get consensus. 
>> 
>> Regarding separate repo, how do we get consensus? 
>> 
>>> On Aug 18, 2018, at 05:19, Stefan Podkowinski  wrote:
>>> 
>>> I don't see that we have reached sufficient consensus on this yet. We've
>>> had a brief discussion about the pros and cons of in-tree cassandra vs
>>> separate ASF repo here, but let's not frame it like it's either or. From
>>> my perspective, there hasn't been any final decision yet whether the
>>> proposed side-car solution should be further developed as part of the
>>> Cassandra project, or not.
>>> 
>>> 
>>>> On 18.08.18 03:12, Dinesh Joshi wrote:
>>>> Thanks, Nate. I’ll create this request.
>>>> 
>>>> Dinesh
>>>> 
>>>> On Aug 17, 2018, at 5:09 PM, Nate McCall  wrote:
>>>> 
>>>>>> I'm not sure logistically how we get a new repo created and licensing and
>>>>>> such, but if someone helps make it we can cut the patch against it
>>>>>> 
>>>>> This is pretty straight forward. For precedent, see:
>>>>> https://issues.apache.org/jira/browse/CASSANDRA-13634
>>>>> 
>>>>> We currently have three repositories:
>>>>> https://git-wip-us.apache.org/repos/asf
>>>>> 
>>>>> I'm +0 on what approach we take.
>>>>> 
>>>>> -
>>>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>>>> 
>>>> -
>>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>>> 
>>> 
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 
> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[sankalp kohli]

Hi,
Here is how I think we can make progress
1. Consensus is reached that we need side car as this thread is months old
and I do not see any objections. I bumped it again and it is good to see it
being active.
2. There is no consensus on new repo vs not. I will start a thread on it
and lets discuss it.
3. We have 2 implementations now for running repair via side car. This is
actually awesome to see for the community. We should work on JIRA(like we
did for virtual table) to get the best out of both the implementation.

Thanks,
Sankalp



On Mon, Aug 20, 2018 at 4:23 AM Mick Semb Wever  wrote:

>
> On Fri, 17 Aug 2018, at 14:27, Sankalp Kohli wrote:
> > I am bumping this thread because patch has landed for this with repair
> > functionality.
>
>
> We are looking to contribute Reaper to the Cassandra project.
>
> Looking at the patch it's very similar in its base design already, but
> Reaper does has a lot more to offer. We have all been working hard to move
> it to also being a side-car so it can be contributed. This raises a number
> of relevant questions to this thread: would we then accept both works in
> the Cassandra project, and what burden would it put on the current PMC to
> maintain both works.
>
> I share Stefan's concern that consensus had not been met around a
> side-car, and that it was somehow default accepted before a patch landed.
> This seems at odds when we're already struggling to keep up with the
> incoming patches/contributions, and there could be other git repos in the
> project we will need to support in the future too. But I'm also curious
> about the whole "Community over Code" angle to this, how do we encourage
> multiple external works to collaborate together building value in both the
> technical and community.
>
> The Reaper project has worked hard in building both its user and
> contributor base. And I would have thought these, including having the
> contributor base overlap with the C* PMC, were prerequisites before moving
> a larger body of work into the project (separate git repo or not). I guess
> this isn't so much "Community over Code", but it illustrates a concern
> regarding abandoned code when there's no existing track record of
> maintaining it as OSS, as opposed to expecting an existing "show, don't
> tell" culture. Reaper for example has stronger indicators for ongoing
> support and an existing OSS user base: today C* committers having
> contributed to Reaper are Jon, Stefan, Nate, and myself, amongst the 40
> contributors in total. And we've been making steps to involve it more into
> the C* community (eg users ML), without being too presumptuous. On the
> technical side: Reaper supports (or can easily) all the concerns that the
> proposal here raises: distributed nodetool commands, centralising jmx
> interfacing, scheduling ops (repairs, snapshots, compactions, cleanups,
> etc), monitoring and diagnostics, etc etc. It's designed so that it can be
> a single instance, instance-per-datacenter, or side-car (per process). When
> there are multiple instances in a datacenter you get HA. You have a choice
> of different storage backends (memory, postgres, c*). You can ofc use a
> separate C* cluster as a backend so to separate infrastructure data from
> production data. And it's got an UI for C* Diagnostics already (which
> imposes a different jmx interface of polling for events rather than
> subscribing to jmx notifications which we know is problematic, thanks to
> Stefan). Anyway, that's my plug for Reaper :-)
>
> There's been little effort in evaluating these two bodies of work, one
> which is largely unknown to us, and my concern is how we would fairly
> support both going into the future?
>
> Another option would be that this side-car patch first exists as a github
> project for a period of time, on par to how Reaper has been. This will help
> evaluate its use and to first build up its contributors. This makes it
> easier for the C* PMC to choose which projects it would want to formally
> maintain, and to do so based on factors beyond merits of the technical. We
> may even see it converge (or collaborate more) with Reaper, a win for
> everyone.
>
> regards,
> Mick
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Side Car New Repo vs not

[sankalp kohli]

Hi,
I am starting a new thread to get consensus on where the side car
should be contributed.

Please send your responses with pro/cons of each approach or any other
approach. Please be clear which approach you will pick while still giving
pros/cons of both approaches.

Thanks.
Sankalp

Re: Side Car New Repo vs not

[sankalp kohli]

ld certainly be wrong), and we
> don't
> >> have to worry about testing matrices to assure that the sidecar works
> >> with
> >> various versions as the version of the sidecar that is released with
> that
> >> version of Cassandra is the only one we have to certify works. If
> people
> >> want to pull in new versions or maintain backports they can do that at
> >> their discretion/testing.
> >> * We can iterate and prove value before committing to a choice. Since
> it
> >> will be a separate artifact from the start we can always move the
> >> artifact
> >> to a separate repo later (but moving the other way is harder).
> >> * Users will get the sidecar "for free" when they install the daemon,
> >> they
> >> don't need to take affirmative action to e.g. be able to restart their
> >> cluster, run repair, or back their data up; it just comes out of the
> box
> >> for free.
> >>
> >> Unique pros of a separate repository sidecar:
> >> * We can use a more modern build system like gradle instead of ant
> >> * Merging changes is less "scary" I guess (I feel like if you're not
> >> touching the daemon this is already true but I could see this being
> less
> >> worrisome for some).
> >> * Releasing a separate artifact is somewhat easier from a separate
> repo
> >> (especially if we have gradle which makes e.g. building debs and rpms
> >> trivial).
> >> * We could backport to previous versions without getting into
> arguments
> >> about bug fixes vs features.
> >> * Committers could be different from the main repo, which ... may be a
> >> useful thing
> >>
> >> Non unique pros of a sidecar (could be achieved in the main repo or in
> a
> >> separate repo):
> >> * A separate build artifact .jar/.deb/.rpm that can be installed
> >> separately. It's slightly easier with a separate repo but certainly
> not
> >> out
> >> of reach within a single repo (indeed the current patch already creates
> a
> >> separate jar, and we could create a separate .deb reasonably easily).
> >> Personally I think having a separate .deb/.rpm is premature at this
> point
> >> (for companies that really want it they can build their own packages
> >> using
> >> the .jars), but I think it really is a distracting issue from where
> the
> >> patch should go as we can always choose to remove experimental .jar
> files
> >> that the main daemon doesn't touch.
> >> * A separate process lifecycle. No matter where the sidecar goes, we
> get
> >> the benefit of restarting it being less dangerous for availability
> than
> >> restarting the main daemon.
> >>
> >> That all being said, these are strong opinions weakly held and I would
> >> rather get something actually committed so that we can prove value one
> >> way
> >> or the other and am therefore, of course, happy to put sidecar patches
> >> wherever someone can review and commit it.
> >>
> >> -Joey
> >>
> >> On Mon, Aug 20, 2018 at 1:52 PM sankalp kohli 
>
> >> wrote:
> >>
> >>> Hi,
> >>> I am starting a new thread to get consensus on where the side car
> >>> should be contributed.
> >>>
> >>> Please send your responses with pro/cons of each approach or any
> other
> >>> approach. Please be clear which approach you will pick while still
> >> giving
> >>> pros/cons of both approaches.
> >>>
> >>> Thanks.
> >>> Sankalp
> >>>
> >>
> >
> >
> > --
> > Jon Haddad
> > http://www.rustyrazorblade.com
> > twitter: rustyrazorblade
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: Side Car New Repo vs not

[sankalp kohli]

Separate repo is in a majority so far. Please reply to this thread with
your responses.

On Tue, Aug 21, 2018 at 4:34 PM Rahul Singh 
wrote:

> +1 for separate repo. Especially on git. Maybe make it a submodule.
>
> Rahul
> On Aug 21, 2018, 3:33 PM -0500, Stefan Podkowinski ,
> wrote:
> > I'm also currently -1 on the in-tree option.
> >
> > Additionally to what Aleksey mentioned, I also don't see how we could
> > make this work with the current build and release process. Our scripts
> > [0] for creating releases (tarballs and native packages), would need
> > significant work to add support for an independent side-car. Our ant
> > based build process is also not a great start for adding new tasks, let
> > alone integrating other tool chains for web components for a potential
> UI.
> >
> > [0] https://git-wip-us.apache.org/repos/asf?p=cassandra-builds.git
> >
> >
> > On 21.08.18 19:20, Aleksey Yeshchenko wrote:
> > > Sure, allow me to elaborate - at least a little bit. But before I do,
> just let me note that this wasn’t a veto -1, just a shorthand for “I don’t
> like this option”.
> > >
> > > It would be nice to have sidecar and C* version and release cycles
> fully decoupled. I know it *can* be done when in-tree, but the way we vote
> on releases with tags off current branches would have to change somehow.
> Probably painfully. It would be nice to be able to easily enforce freezes,
> like the upcoming one, on the whole C* repo, while allowing feature
> development on the sidecar. It would be nice to not have sidecar commits in
> emails from commits@ mailing list. It would be nice to not have C* CI
> trigger necessarily on sidecar commits. Groups of people working on the two
> repos will mostly be different too, so what’s the point in sharing the repo?
> > >
> > > Having an extra repo with its own set of branches is cheap and easy -
> we already do that with dtests. I like cleanly separated things when
> coupling is avoidable. As such I would prefer the sidecar to live in a
> separate new repo, while still being part of the C* project.
> > >
> > > —
> > > AY
> > >
> > > On 21 August 2018 at 17:06:39, sankalp kohli (kohlisank...@gmail.com)
> wrote:
> > >
> > > Hi Aleksey,
> > > Can you please elaborate on the reasons for your -1? This
> > > way we can make progress towards any one approach.
> > > Thanks,
> > > Sankalp
> > >
> > > On Tue, Aug 21, 2018 at 8:39 AM Aleksey Yeshchenko 
> > > wrote:
> > >
> > > > FWIW I’m strongly -1 on in-tree approach, and would much prefer a
> separate
> > > > repo, dtest-style.
> > > >
> > > > —
> > > > AY
> > > >
> > > > On 21 August 2018 at 16:36:02, Jeremiah D Jordan (
> > > > jeremiah.jor...@gmail.com) wrote:
> > > >
> > > > I think the following is a very big plus of it being in tree:
> > > > > > * Faster iteration speed in general. For example when we need to
> add a
> > > > > > new
> > > > > > JMX endpoint that the sidecar needs, or change something from
> JMX to a
> > > > > > virtual table (e.g. for repair, or monitoring) we can do all
> changes
> > > > > > including tests as one commit within the main repository and
> don't
> > > > have
> > > > > > to
> > > > > > commit to main repo, sidecar repo,
> > > >
> > > > I also don’t see a reason why the sidecar being in tree means it
> would not
> > > > work in a mixed version cluster. The nodes themselves must work in a
> mixed
> > > > version cluster during a rolling upgrade, I would expect any
> management
> > > > side car to operate in the same manor, in tree or not.
> > > >
> > > > This tool will be pretty tightly coupled with the server, and as
> someone
> > > > with experience developing such tightly coupled tools, it is *much*
> easier
> > > > to make sure you don’t accidentally break them if they are in tree.
> How
> > > > many times has someone updated some JMX interface, updated nodetool,
> and
> > > > then moved on? Breaking all the external tools not in tree, without
> > > > realizing it. The above point about being able to modify interfaces
> and the
> > > > side car in the same commit is huge in terms of making sure someone
> doesn’t
> > > > inadvertently break the side car while fixing something else.
> > > &g

Re: Side Car New Repo vs not

[Sankalp Kohli]

Thanks everyone for the feedback. Looks like we will go with separate repo as 
that is what majority of people prefer. 

Also note that we can always change this approach later as we build the side 
car. 

> On Aug 24, 2018, at 07:00, Eric Evans  wrote:
> 
>> On Thu, Aug 23, 2018 at 3:01 PM sankalp kohli  wrote:
>> 
>> Separate repo is in a majority so far. Please reply to this thread with
>> your responses.
> 
> I think it makes sense for the code, project, and workflows to be
> (de|loosely)-coupled, so the repo should be as well.
> 
> +1 for a separate repository
> 
>> On Tue, Aug 21, 2018 at 4:34 PM Rahul Singh 
>> wrote:
>> 
>>> +1 for separate repo. Especially on git. Maybe make it a submodule.
>>> 
>>> Rahul
>>> On Aug 21, 2018, 3:33 PM -0500, Stefan Podkowinski ,
>>> wrote:
>>>> I'm also currently -1 on the in-tree option.
>>>> 
>>>> Additionally to what Aleksey mentioned, I also don't see how we could
>>>> make this work with the current build and release process. Our scripts
>>>> [0] for creating releases (tarballs and native packages), would need
>>>> significant work to add support for an independent side-car. Our ant
>>>> based build process is also not a great start for adding new tasks, let
>>>> alone integrating other tool chains for web components for a potential
>>> UI.
>>>> 
>>>> [0] https://git-wip-us.apache.org/repos/asf?p=cassandra-builds.git
>>>> 
>>>> 
>>>>> On 21.08.18 19:20, Aleksey Yeshchenko wrote:
>>>>> Sure, allow me to elaborate - at least a little bit. But before I do,
>>> just let me note that this wasn’t a veto -1, just a shorthand for “I don’t
>>> like this option”.
>>>>> 
>>>>> It would be nice to have sidecar and C* version and release cycles
>>> fully decoupled. I know it *can* be done when in-tree, but the way we vote
>>> on releases with tags off current branches would have to change somehow.
>>> Probably painfully. It would be nice to be able to easily enforce freezes,
>>> like the upcoming one, on the whole C* repo, while allowing feature
>>> development on the sidecar. It would be nice to not have sidecar commits in
>>> emails from commits@ mailing list. It would be nice to not have C* CI
>>> trigger necessarily on sidecar commits. Groups of people working on the two
>>> repos will mostly be different too, so what’s the point in sharing the repo?
>>>>> 
>>>>> Having an extra repo with its own set of branches is cheap and easy -
>>> we already do that with dtests. I like cleanly separated things when
>>> coupling is avoidable. As such I would prefer the sidecar to live in a
>>> separate new repo, while still being part of the C* project.
>>>>> 
>>>>> —
>>>>> AY
>>>>> 
>>>>> On 21 August 2018 at 17:06:39, sankalp kohli (kohlisank...@gmail.com)
>>> wrote:
>>>>> 
>>>>> Hi Aleksey,
>>>>> Can you please elaborate on the reasons for your -1? This
>>>>> way we can make progress towards any one approach.
>>>>> Thanks,
>>>>> Sankalp
>>>>> 
>>>>> On Tue, Aug 21, 2018 at 8:39 AM Aleksey Yeshchenko 
>>>>> wrote:
>>>>> 
>>>>>> FWIW I’m strongly -1 on in-tree approach, and would much prefer a
>>> separate
>>>>>> repo, dtest-style.
>>>>>> 
>>>>>> —
>>>>>> AY
>>>>>> 
>>>>>> On 21 August 2018 at 16:36:02, Jeremiah D Jordan (
>>>>>> jeremiah.jor...@gmail.com) wrote:
>>>>>> 
>>>>>> I think the following is a very big plus of it being in tree:
>>>>>>>> * Faster iteration speed in general. For example when we need to
>>> add a
>>>>>>>> new
>>>>>>>> JMX endpoint that the sidecar needs, or change something from
>>> JMX to a
>>>>>>>> virtual table (e.g. for repair, or monitoring) we can do all
>>> changes
>>>>>>>> including tests as one commit within the main repository and
>>> don't
>>>>>> have
>>>>>>>> to
>>>>>>>> commit to main repo, sidecar repo,
>>>>>> 
>>>>>> I also don’t see a reason why the sidecar being in tree means it
>>>

Re: Reaper as cassandra-admin

[Sankalp Kohli]

We can wait for a week post Freeze so everyone can participate however we need 
to decide after that so we can make progress. 
I am leaning towards piecemeal approach so we can review the code and pick best 
of all 3 options 

> On Aug 29, 2018, at 00:26, kurt greaves  wrote:
> 
> 2c: There's a lot to think about here, and as Blake already mentioned most
> people don't have time to dedicate a lot of thought to this at the moment.
> There appear to be a lot of voices missing from the discussion, and I think
> it's pretty clear this isn't super tied to the freeze, so maybe we should
> leave this discussion until next week when everyone can take part? This
> kind of goes for every sidecar related discussion going on at the moment
> IMO.
> 
> On 29 August 2018 at 16:44, Vinay Chella 
> wrote:
> 
>>> I haven’t settled on a position yet (will have more time think about
>> things after the 9/1 freeze), but I wanted to point out that the argument
>> that something new should be written because an existing project has tech
>> debt, and we'll do it the right way this time, is a pretty common software
>> engineering mistake. The thing you’re replacing usually needs to have some
>> really serious problems to make it worth replacing.
>> 
>> Agreed, Yes, I don’t think we should write everything from the scratch, but
>> carry forwarding tech debt (if any) and design decisions which makes new
>> features in future difficult to develop is something that we need to
>> consider. I second Dinesh’s thought on taking the best parts from available
>> projects to move forward with the right solution which works great and
>> easily pluggable.
>> 
>> -
>> Vinay Chella
>> 
>> 
>>> On Tue, Aug 28, 2018 at 10:03 PM Mick Semb Wever  wrote:
>>> 
>>> 
 the argument that something new should be written because an existing
>>> project has tech debt, and we'll do it the right way this time, is a
>> pretty
>>> common software engineering mistake. The thing you’re replacing usually
>>> needs to have some really serious problems to make it worth replacing.
>>> 
>>> 
>>> Thanks for writing this Blake. I'm no fan of writing from scratch.
>> Working
>>> with other people's code is the joy of open-source, imho.
>>> 
>>> Reaper is not a big project. None of its java files are large or
>>> complicated.
>>> This is not the C* codebase we're talking about.
>>> 
>>> It comes with strict code style in place (which the build enforces), unit
>>> and integration tests. The tech debt that I think of first is removing
>>> stuff that we would no longer want to support if it were inside the
>>> Cassandra project. A number of recent refactorings  have proved it's an
>>> easy codebase to work with.
>>> 
>>> It's also worth noting that Cassandra-4.x adoption is still some away, in
>>> which time Reaper will only continue to grow and gain users.
>>> 
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>> 
>>> 
>> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: NGCC 2018?

[sankalp kohli]

A good time for NGCC will be closer to 4.0 release where we can plan what
we can put it on 4.0-next. I am not sure doing it now is going to help when
we are months away from 4.0 release.

On Fri, Aug 31, 2018 at 7:42 AM Jay Zhuang  wrote:

> Are we going to have a dev event next month? Or anything this year? We may
> also be able to provide space in bay area and help to organize it. (Please
> let us know, so we could get final approval for that).
>
> On Fri, Jul 27, 2018 at 10:05 AM Jonathan Haddad 
> wrote:
>
> > My interpretation of Nate's statement was that since there would be a
> bunch
> > of us at Lynn's event, we might as well do NGCC at the same time.
> >
> > On Thu, Jul 26, 2018 at 9:03 PM Ben Bromhead 
> wrote:
> >
> > > It sounds like there may be an appetite for something, but the NGCC in
> > its
> > > current format is likely to not be that useful?
> > >
> > > Is a bay area event focused on C* developers something that is
> > interesting
> > > for the broader dev community? In whatever format that may be?
> > >
> > > On Tue, Jul 24, 2018 at 5:02 PM Nate McCall 
> wrote:
> > >
> > > > This was discussed amongst the PMC recently. We did not come to a
> > > > conclusion and there were not terribly strong feelings either way.
> > > >
> > > > I don't feel like we need to hustle to get "NGCC" in place,
> > > > particularly given our decided focus on 4.0. However, that should not
> > > > stop us from doing an additional 'c* developer' event in sept. to
> > > > coincide with distributed data summit.
> > > >
> > > > On Wed, Jul 25, 2018 at 5:03 AM, Patrick McFadin  >
> > > > wrote:
> > > > > Ben,
> > > > >
> > > > > Lynn Bender had offered a space the day before Distributed Data
> > Summit
> > > in
> > > > > September (http://distributeddatasummit.com/) since we are both
> > > platinum
> > > > > sponsors. I thought he and Nate had talked about that being a good
> > > place
> > > > > for NGCC since many of us will be in town already.
> > > > >
> > > > > Nate, now that I've spoken for you, you can clarify, :D
> > > > >
> > > > > Patrick
> > > > >
> > > >
> > > > -
> > > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > > > For additional commands, e-mail: dev-h...@cassandra.apache.org
> > > >
> > > > --
> > > Ben Bromhead
> > > CTO | Instaclustr 
> > > +1 650 284 9692
> > > Reliability at Scale
> > > Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer
> > >
> >
> >
> > --
> > Jon Haddad
> > http://www.rustyrazorblade.com
> > twitter: rustyrazorblade
> >
>

Re: NGCC 2018?

[sankalp kohli]

Another thing to  discuss will be how to improve testing further from the
learning of finding bugs in C* 4.0.

On Wed, Sep 5, 2018 at 9:57 AM Jason Brown  wrote:

> +1 to Jon's sentiment. Further, perhaps we should use that time after
> GA'ing 4.0 to poll our users what they need/want from the next major
> release of the database.
>
> On Wed, Sep 5, 2018 at 9:31 AM, Jonathan Haddad  wrote:
>
> > I'm thinking a month or two after 4.0 would give us time to unwind after
> > the release and start to give real thought to big changes coming in the
> > next release.  Let's focus on one thing at a time.
> >
> > On Wed, Sep 5, 2018 at 12:29 PM sankalp kohli 
> > wrote:
> >
> > > A good time for NGCC will be closer to 4.0 release where we can plan
> what
> > > we can put it on 4.0-next. I am not sure doing it now is going to help
> > when
> > > we are months away from 4.0 release.
> > >
> > > On Fri, Aug 31, 2018 at 7:42 AM Jay Zhuang 
> > wrote:
> > >
> > > > Are we going to have a dev event next month? Or anything this year?
> We
> > > may
> > > > also be able to provide space in bay area and help to organize it.
> > > (Please
> > > > let us know, so we could get final approval for that).
> > > >
> > > > On Fri, Jul 27, 2018 at 10:05 AM Jonathan Haddad 
> > > > wrote:
> > > >
> > > > > My interpretation of Nate's statement was that since there would
> be a
> > > > bunch
> > > > > of us at Lynn's event, we might as well do NGCC at the same time.
> > > > >
> > > > > On Thu, Jul 26, 2018 at 9:03 PM Ben Bromhead 
> > > > wrote:
> > > > >
> > > > > > It sounds like there may be an appetite for something, but the
> NGCC
> > > in
> > > > > its
> > > > > > current format is likely to not be that useful?
> > > > > >
> > > > > > Is a bay area event focused on C* developers something that is
> > > > > interesting
> > > > > > for the broader dev community? In whatever format that may be?
> > > > > >
> > > > > > On Tue, Jul 24, 2018 at 5:02 PM Nate McCall 
> > > > wrote:
> > > > > >
> > > > > > > This was discussed amongst the PMC recently. We did not come
> to a
> > > > > > > conclusion and there were not terribly strong feelings either
> > way.
> > > > > > >
> > > > > > > I don't feel like we need to hustle to get "NGCC" in place,
> > > > > > > particularly given our decided focus on 4.0. However, that
> should
> > > not
> > > > > > > stop us from doing an additional 'c* developer' event in sept.
> to
> > > > > > > coincide with distributed data summit.
> > > > > > >
> > > > > > > On Wed, Jul 25, 2018 at 5:03 AM, Patrick McFadin <
> > > pmcfa...@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > Ben,
> > > > > > > >
> > > > > > > > Lynn Bender had offered a space the day before Distributed
> Data
> > > > > Summit
> > > > > > in
> > > > > > > > September (http://distributeddatasummit.com/) since we are
> > both
> > > > > > platinum
> > > > > > > > sponsors. I thought he and Nate had talked about that being a
> > > good
> > > > > > place
> > > > > > > > for NGCC since many of us will be in town already.
> > > > > > > >
> > > > > > > > Nate, now that I've spoken for you, you can clarify, :D
> > > > > > > >
> > > > > > > > Patrick
> > > > > > > >
> > > > > > >
> > > > > > >
> > > -
> > > > > > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > > > > > > For additional commands, e-mail: dev-h...@cassandra.apache.org
> > > > > > >
> > > > > > > --
> > > > > > Ben Bromhead
> > > > > > CTO | Instaclustr <https://www.instaclustr.com/>
> > > > > > +1 650 284 9692
> > > > > > Reliability at Scale
> > > > > > Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Jon Haddad
> > > > > http://www.rustyrazorblade.com
> > > > > twitter: rustyrazorblade
> > > > >
> > > >
> > >
> >
> >
> > --
> > Jon Haddad
> > http://www.rustyrazorblade.com
> > twitter: rustyrazorblade
> >
>

Re: QA signup

[sankalp kohli]

Thanks for starting this Jon.
Instead of saying "I tested streaming", we should define what all was
tested like was all data transferred, what happened when stream failed,
etc.
Based on talking to a few users, looks like most testing is done by doing
an operation or running a load and seeing if it "worked" and no errors in
logs.

Another important thing will be to fix bugs asap ahead of testing,  as
fixes can lead to more bugs :)

On Thu, Sep 6, 2018 at 7:52 AM Jonathan Haddad  wrote:

> I was thinking along the same lines.  For this to be successful I think
> either weekly or bi-weekly summary reports back to the mailing list by the
> team lead for each subsection on what's been tested and how it's been
> tested will help keep things moving along.
>
> In my opinion the lead for each team should *not* be the contributor that
> wrote the feature, but someone who's very interested in it and can use the
> contributor as a resource.  I think it would be difficult for the
> contributor to poke holes in their own work - if they could do that it
> would have been done already.  This should be a verification process that's
> independent as possible from the original work.
>
> In addition to the QA process, it would be great if we could get a docs
> team together.  We've got quite a bit of undocumented features and nuance
> still, I think hammering that out would be a good idea.  Mick brought up
> updating the website docs in the thread on testing different JDK's [1], if
> we could figure that out in the process we'd be in a really great position
> from the user perspective.
>
> Jon
>
> [1]
>
> https://lists.apache.org/thread.html/5645178efb57939b96e73ab9c298e80ad8e76f11a563b4d250c1ae38@%3Cdev.cassandra.apache.org%3E
>
> On Thu, Sep 6, 2018 at 10:35 AM Jordan West  wrote:
>
> > Thanks for staring this thread Jon!
> >
> > On Thu, Sep 6, 2018 at 5:51 AM Jonathan Haddad 
> wrote:
> >
> > > For 4.0, I'm thinking it would be a good idea to put together a list of
> > the
> > > things that need testing and see if people are willing to help test /
> > break
> > > those things.  My goal here is to get as much coverage as possible, and
> > let
> > > folks focus on really hammering on specific things rather than just
> > firing
> > > up a cluster and rubber stamping it.  If we're going to be able to
> > > confidently deploy 4.0 quickly after it's release we're going to need a
> > > high attention to detail.
> > >
> > >
> > +1 to a more coordinated effort. I think we could use the Confluence that
> > was set up a little bit ago since it was setup for this purpose, at least
> > for finalized plans and results:
> > https://cwiki.apache.org/confluence/display/CASSANDRA.
> >
> >
> > > In addition to a signup sheet, I think providing some guidance on how
> to
> > QA
> > > each thing that's being tested would go a long way.  Throwing "hey
> please
> > > test sstable streaming" over the wall will only get quality feedback
> from
> > > folks that are already heavily involved in the development process.  It
> > > would be nice to bring some new faces into the project by providing a
> > > little guidance.
> > >
> >
> > > We could help facilitate this even further by considering the people
> > > signing up to test a particular feature as a team, with seasoned
> > Cassandra
> > > veterans acting as team leads.
> > >
> >
> > +1 to this as well. I am always a fan of folks learning about a
> > subsystem/project through testing. It can be challenging to get folks new
> > to a project excited about testing first but for those that do, or for
> > committers who want to learn another part of the db, its a great way to
> > learn.
> >
> > Another thing we can do here is make sure teams are writing about the
> > testing they are doing and their results. This will help share knowledge
> > about techniques and approaches that others can then apply. This
> knowledge
> > can be shared on the mailing list, a blog post, or in JIRA.
> >
> >  Jordan
> >
> >
> > > Any thoughts?  I'm happy to take the lead on this.
> > > --
> > > Jon Haddad
> > > http://www.rustyrazorblade.com
> > > twitter: rustyrazorblade
> > >
> >
>
>
> --
> Jon Haddad
> http://www.rustyrazorblade.com
> twitter: rustyrazorblade
>

Re: Proposing an Apache Cassandra Management process

[sankalp kohli]

Most of the discussions have been around whether we take some side car or
do a cherry pick approach where we add a feature at a time to side car and
use the best implementation.
We have been discussing this first in April and now for several days. I
think we all want some progress here. We will start a vote soon..may be
next week to decide which approach we will take. I don't see any other
option to make progress besides a vote!!

PS: I think these discussions are very good for the community and it shows
people care about Apache Cassandra and it is a sign of healthy community.
Several people offering to donate the side car or help build is showing the
interest everyone has in it.


On Sat, Sep 8, 2018 at 11:17 AM Joseph Lynch  wrote:

> On Fri, Sep 7, 2018 at 10:00 PM Blake Eggleston 
> wrote:
> >
> > Right, I understand the arguments for starting a new project. I’m not
> saying reaper is, technically speaking, the best place to start. The point
> I’m trying to make is that the non-technical advantages of using an
> existing project as a starting point may outweigh the technical benefits of
> a clean slate. Whether that’s the case or not, it’s not a strictly
> technical decision, and the non-technical advantages of starting with
> reaper need to be weighed.
> >
>
> Technical debt doesn't just refer to the technical solutions, having
> an existing user base means that a project has made promises in the
> past (in the case of Priam 5+ years ago) that the new project would
> have to keep if we make keeping users of those sidecars a goal (which
> for the record I don't think should be a goal, I think the goal is to
> make Cassandra the database work out of the box in the 4.x series).
>
> For example, Priam has to continue supporting the following as users
> actively use them (including Netflix):
> * Parallel token assignment and creation allowing parallel bootstrap
> and parallel doubling of hundred node clusters at once (as long as you
> use single tokens and run in AWS with asgs).
> * 3+ backup solutions, as well as assumptions about where in e.g. S3
> backups are present and what format they're present in.
> * Numerous configuration options and UI elements (mostly 5 year old JSON
> APIs)
> * Support for Cassandra 2.0, 2.1, 2.2, 3.0, 3.11 and soon 4.0
>
> Reaper has to continue supporting the following as users actively use them:
> * Postgres and h2 storage backends. These were the original storage
> engines and users may not have (probably haven't?) migrated to the
> relatively recently added Cassandra backend (which is probably the
> only one an official sidecar should support imo).
> * The three historical modes of running Reaper [1] all of which
> involve remote JMX (disallowed by many companies security policies
> including Netflix's) and none of which are a sidecar design (although
> Mick says we can add that back in, at which point it has to support
> four).
> * Numerous configuration options and UI elements (e.g. a UI around a
> single Reaper instance controlling many Cassandra clusters instead of
> each cluster having a separate UI more consistent with how Cassandra
> architecture works)
> * Support for Cassandra 2.2, 3.0, 3.11, and soon 4.0
>
> [1] http://cassandra-reaper.io/docs/usage/multi_dc/
> [2] https://github.com/hashbrowncipher/cassandra-mirror
>
> We can't "get the community" of these sidecars and drop support for
> 90+% of the supported configurations and features at the same time ...
> These projects have made promises to users, as per the name technical
> debt these choices made over years have explicit costs that we have to
> take into account if we accept a project as is with the goal of taking
> the community with us. If we don't have the goal of taking the
> existing community with us and are instead aiming to simply make
> Cassandra work out of the box without external tools, then we don't
> have to continue fulfilling these promises.
>
> Instead I think the organizations that made those promises (TLP and
> Netflix in these specific cases) should continue keeping them, and the
> Cassandra management process should be incrementally built by the
> Cassandra community with decisions made as a community and we only GA
> it when the PMC/community is happy with making a promise of support
> for the features that we've merged (and since we're starting from a
> fresh start if people have strong opinions about fundamental
> architecture we can try to take those into account like we did with
> the months of feedback on repair scheduling
> runtime/architecture/design). If we can't prove value over other
> community tools for running 4.x, which is definitely a possibility,
> then we don't mark the management process GA, we re-focus on
> individual community tools, and imo failure is a reasonable result of
> attempted innovation.
>
> -Joey
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional com

Re: Proposing an Apache Cassandra Management process

[sankalp kohli]

I agree with Jon and I think folks who are talking about tech debts in
Reaper should elaborate with examples about these tech debts. Can we be
more precise and list them down? I see it spread out over this long email
thread!!

On Sun, Sep 9, 2018 at 6:29 AM Elliott Sims  wrote:

> A big one to add to your list there, IMO as a user:
> * API for determining detailed repair state (and history?).  Essentially,
> something beyond just "Is some sort of repair running?" so that tools like
> Reaper can parallelize better.
>
> On Sun, Sep 9, 2018 at 8:30 AM, Stefan Podkowinski 
> wrote:
>
> > Does it have to be a single project with functionality provided by
> > multiple plugins? Designing a plugin API at this point seems to be a bit
> > early and comes with additional complexity around managing plugins in
> > general.
> >
> > I was more thinking into the direction of: "what can we do to enable
> > people to create any kind of side car or tooling solution?". Thinks like:
> >
> > Common cluster discovery and management API
> > * Detect local Cassandra processes
> > * Discover and receive events on cluster topology
> > * Get assigned tokens for nodes
> > * Read node configuration
> > * Health checks (as already proposed)
> >
> > Any side cars should be easy to install on nodes that already run
> Cassandra
> > * Scripts for packaging (tar, deb, rpm)
> > * Templates for systemd support, optionally with auto-startup dependency
> > on the Cassandra main process
> >
> > Integration testing
> > * Provide basic testing framework for mocking cluster state and messages
> >
> > Support for other languages / avoid having to use JMX
> > * JMX bridge (HTTP? gRPC?, already implemented in #14346?)
> >
> > Obviously the whole side car discussion is not moving into a direction
> > everyone's happy with. Would it be an option to take a step back and
> > start implementing such a tooling framework with scripts and libraries
> > for the features described above, as a small GitHub project, instead of
> > putting an existing side-car solution up for vote? If that would work
> > and we get people collaborating on code shared between existing
> > side-cars, then we could take the next step and think about either
> > revisit the "official Cassandra side-car" topic, or add the created
> > client tooling framework as official sub-project to the Cassandra
> > project (maybe via Apache incubator).
> >
> >
> > On 08.09.18 02:49, Joseph Lynch wrote:
> > > On Fri, Sep 7, 2018 at 5:03 PM Jonathan Haddad 
> > wrote:
> > >> We haven’t even defined any requirements for an admin tool. It’s hard
> to
> > >> make a case for anything without agreement on what we’re trying to
> > build.
> > >>
> > > We were/are trying to sketch out scope/requirements in the #14395 and
> > > #14346 tickets as well as their associated design documents. I think
> > > the general proposed direction is a distributed 1:1 management sidecar
> > > process similar in architecture to Netflix's Priam except explicitly
> > > built to be general and pluggable by anyone rather than tightly
> > > coupled to AWS.
> > >
> > > Dinesh, Vinay and I were aiming for low amounts of scope at first and
> > > take things in an iterative approach with just enough upfront design
> > > but not so much we are unable to make any progress at all. For example
> > > maybe something like:
> > >
> > > 1. Get a super simple and non controversial sidecar process that ships
> > > with Cassandra and exposes a lightweight HTTP interface to e.g. some
> > > basic JMX endpoints
> > > 2a. Add a pluggable execution engine for cron/oneshot/scheduled jobs
> > > with the basic interfaces and state store and such
> > > 2b. Start scoping and implementing the full HTTP interface, e.g.
> > > backup status, cluster health status, etc ...
> > > 3a. Start integrating implementations of the jobs from 2a such as
> > > snapshot, backup, cluster restart, daemon + sstable upgrade, repair,
> > > etc
> > > 3b. Start integrating UI components that pair with the HTTP interface
> > from 2b
> > > 4. ?? Perhaps start unlocking next generation operations like moving
> > > "background" activities like compaction, streaming, repair etc into
> > > one or more sidecar contained processes to ensure the main daemon only
> > > handles read+write requests
> > >
> > > There are going to be a lot of questions to answer, and I think trying
> > > to answer them all up front will mean that we get nowhere or make
> > > unfortunate compromises that cripple the project from the start. If
> > > people think we need to do more design and discussion than we have
> > > been doing then we can spend more time on the design, but personally
> > > I'd rather start iterating on code and prove value incrementally. If
> > > it doesn't work out we won't release it GA to the community ...
> > >
> > > -Joey
> > >
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > > For addition

Re: [Discuss] Accept GoCQL driver donation

[sankalp kohli]

Hi Nate,
Looks like we had a lot of discussion here and everyone seems
to be in favor. What is the next step? A vote?
Thanks,
Sankalp

On Fri, Aug 31, 2018 at 10:48 PM Alex Lourie  wrote:

> Same here. I've been working on this project for a bit now, and I'm
> planning to continue and contribute.
>
> I also like the idea of this project becoming an officially endorsed golang
> Cassandra driver. Makes a lot of sense too.
>
> Alex.
>
>
> On Sat., 1 Sep. 2018, 01:08 Chris Bannister, 
> wrote:
>
> > I intend to stay on and continue to contribute.
> >
> > On Fri, 31 Aug 2018 at 4:37 pm, Jonathan Ellis 
> wrote:
> >
> > > On Fri, Aug 31, 2018 at 9:14 AM Nate McCall 
> wrote:
> > >
> > > > Hi folks,
> > > > So I was recently talking with, Chris Bannister  the gocql [0]
> > > > maintainer, and he expressed an interest in donating the driver to
> the
> > > > ASF.
> > > >
> > >
> > > Is he looking to continue to maintain it or is he looking to give it a
> > good
> > > home when he moves on?
> > >
> > > We could accept this along the same lines as how we took in the dtest
> > > > donation - going through the incubator IP clearance process [1], but
> > > > in this case it's much simpler as an individual (Chris) owns the
> > > > copyright.
> > > >
> > >
> > > Is that actually the case?  Github says 128 contributors, and I don't
> see
> > > any mention of a CLA in
> > > https://github.com/gocql/gocql/blob/master/CONTRIBUTING.md.
> > >
> > > --
> > > Jonathan Ellis
> > > co-founder, http://www.datastax.com
> > > @spyced
> > >
> >
>

[VOTE] Development Approach for Apache Cassandra Management process

[sankalp kohli]

Hi,
Community has been discussing about Apache Cassandra Management process
since April and we had lot of discussion about which approach to take to
get started. Several contributors have been interested in doing this and we
need to make a decision of which approach to take.

The current approaches being evaluated are
a. Donate an existing project to Apache Cassandra like Reaper. If this
option is selected, we will evaluate various projects and see which one
fits best.
b. Take a piecemeal approach and use the features from different OSS
projects and build a new project.

Available options to vote
a. +1 to use existing project.
b. +1 to take piecemeal approach
c  -1 to both
d +0 I dont mind either option

You can also just type a,b,c,d as well to chose an option.

Dev threads with discussions

https://lists.apache.org/thread.html/4eace8cb258aab83fc3a220ff2203a281ea59f4d6557ebeb1af7b7f1@%3Cdev.cassandra.apache.org%3E

https://lists.apache.org/thread.html/4a7e608c46aa2256e8bcb696104a4e6d6aaa1f302834d211018ec96e@%3Cdev.cassandra.apache.org%3E

Re: [Discuss] Accept GoCQL driver donation

[sankalp kohli]

Great..please start the vote to get consensus.

On Wed, Sep 12, 2018 at 8:06 AM Nate McCall  wrote:

> Yep - that sounds like the best next step to me.
>
> (apologies for spotty comms folks - been/still on vacation).
> On Wed, Sep 12, 2018 at 8:03 AM sankalp kohli 
> wrote:
> >
> > Hi Nate,
> > Looks like we had a lot of discussion here and everyone seems
> > to be in favor. What is the next step? A vote?
> > Thanks,
> > Sankalp
> >
> > On Fri, Aug 31, 2018 at 10:48 PM Alex Lourie 
> wrote:
> >
> > > Same here. I've been working on this project for a bit now, and I'm
> > > planning to continue and contribute.
> > >
> > > I also like the idea of this project becoming an officially endorsed
> golang
> > > Cassandra driver. Makes a lot of sense too.
> > >
> > > Alex.
> > >
> > >
> > > On Sat., 1 Sep. 2018, 01:08 Chris Bannister, 
> > > wrote:
> > >
> > > > I intend to stay on and continue to contribute.
> > > >
> > > > On Fri, 31 Aug 2018 at 4:37 pm, Jonathan Ellis 
> > > wrote:
> > > >
> > > > > On Fri, Aug 31, 2018 at 9:14 AM Nate McCall 
> > > wrote:
> > > > >
> > > > > > Hi folks,
> > > > > > So I was recently talking with, Chris Bannister  the gocql [0]
> > > > > > maintainer, and he expressed an interest in donating the driver
> to
> > > the
> > > > > > ASF.
> > > > > >
> > > > >
> > > > > Is he looking to continue to maintain it or is he looking to give
> it a
> > > > good
> > > > > home when he moves on?
> > > > >
> > > > > We could accept this along the same lines as how we took in the
> dtest
> > > > > > donation - going through the incubator IP clearance process [1],
> but
> > > > > > in this case it's much simpler as an individual (Chris) owns the
> > > > > > copyright.
> > > > > >
> > > > >
> > > > > Is that actually the case?  Github says 128 contributors, and I
> don't
> > > see
> > > > > any mention of a CLA in
> > > > > https://github.com/gocql/gocql/blob/master/CONTRIBUTING.md.
> > > > >
> > > > > --
> > > > > Jonathan Ellis
> > > > > co-founder, http://www.datastax.com
> > > > > @spyced
> > > > >
> > > >
> > >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: [VOTE] Accept GoCQL driver donation and begin incubation process

[sankalp kohli]

+1

On Wed, Sep 12, 2018 at 8:12 AM Nate McCall  wrote:

> This will be the same process used for dtest. We will need to walk
> this through the incubator per the process outlined here:
>
> https://incubator.apache.org/guides/ip_clearance.html
>
> Pending the outcome of this vote, we will create the JIRA issues for
> tracking and after we go through the process, and discuss adding
> committers in a separate thread (we need to do this atomically anyway
> per general ASF committer adding processes).
>
> Thanks,
> -Nate
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: [VOTE] Development Approach for Apache Cassandra Management process

[sankalp kohli]

Hi Sylvain,
I would appreciate if we can give feedback on the
discussion threads and not wait for vote threads. I made it clear in the
discussion thread that we will start a vote!!
Thanks,
Sankalp

On Wed, Sep 12, 2018 at 12:47 PM Jeff Jirsa  wrote:

> On Wed, Sep 12, 2018 at 12:41 PM Sylvain Lebresne 
> wrote:
>
> > That's probably a stupid question, and excuse me if it is, but what does
> > those votes on the dev mailing list even mean?
> >
> > How do you count votes at the end? Just by counting all votes cast,
> > irregardless of whomever cast it? Or are we intending to only count PMC
> > members, or maybe committers votes?
> >
>
> I believe the intent is to try to see if there exists consensus.
> Ultimately, PMC is going to matter more than random email addresses from
> people nobody recognizes. This should be in public, though, not private, so
> seeing what feedback is beyond the PMC is useful (primarily because it will
> matter when it comes time to extend and maintain it - if people strongly
> prefer one or the other, then maintenance is going to be a problem).
>
> If there's 100 random non-contributor votes for one option and 20 pmc votes
> for another options, I think the real answer will be "we don't have
> consensus, and either we don't do it, or we do it the way the PMC thinks is
> best", for all of the reasons you describe in the paragraphs below.
>
>
> > If the former, that is a bit weird to me because we simply don't know who
> > votes. And I don't mean to be rude towards anyone, but 1) someone could
> > easily create 10 email addresses to vote 10 times (and sure, you could
> > invoke trust, and I'm not entirely against trust in general, but it's the
> > internet...) and 2) this kind of decision will have non-trivial
> > consequences for the project, particularly on those that maintain it, so
> I
> > admit I'm not entirely comfortable with "anyone's voice has the same
> > weight".
> > If the latter, then this makes more sense to me (why are we even
> bothering
> > voting PMC members in if it's not to handle these kinds of decisions,
> which
> > are very "project management" related), but we should be very clear about
> > this from the get go (we could still use the dev list for transparency
> > sake, that I don't mind)? We should probably also have some deadline to
> the
> > vote, one that isn't too short.
> >
>
> Like releases, I think PMC votes count
>
>
> >
> > Anyway, fwiw, my opinion on this vote is not far from the one on the
> golang
> > driver acceptance vote (for which my remark above also apply btw): no yet
> > 100% convinced adding more pieces and scope to the project is what the
> > project needs just right now, but not strongly opposed if people really
> > wants this (and this one makes more sense to me than the golang driver
> > actually). But if I'm to pick between a) and b), I'm leaning b).
> >
>
> FWIW, two of the main reasons I'm in favor is as a way to lower barrier to
> entry to both using the software AND contributing to the project, so I
> think your points are valid (both on gocql thread and on this note above),
> but I think that's also part of why we should be encouraging both.
>
> - Jeff
>

Re: [VOTE] Development Approach for Apache Cassandra Management process

[sankalp kohli]

The last email on the thread was 3 days ago and I made it clear days back
that we should vote on it to make progress. Without this vote, I am not
sure we will make progress.
Many people want to contribute on this and hence we are voting so we can
make progress.

My vote is d


On Wed, Sep 12, 2018 at 1:36 PM Jonathan Haddad  wrote:

> This voting process feels a bit rushed and frankly not well thought out.
> In addition to Sylvain's valid points, which you (Sankalp) didn't address
> at all, the discussion in the other threads seemed to be ongoing.  The last
> email you wrote on one of them was asking for additional feedback, that
> indicates the discussion is still open.
>
> Out of principal I vote for none of the options (inaction).  You're
> deliberately trying to ram *something* through, and that's not how this is
> supposed to work.
>
> For those of you unfamiliar with the process - please read
> https://www.apache.org/foundation/voting.html.
>
> I'd like to ask those of you that are +1'ing, are you willing to contribute
> or are you just voting we start an admin tool from scratch because you
> think it'll somehow produce a perfect codebase?
>
>
>
>
>
> On Wed, Sep 12, 2018 at 12:50 PM sankalp kohli 
> wrote:
>
> > Hi Sylvain,
> > I would appreciate if we can give feedback on the
> > discussion threads and not wait for vote threads. I made it clear in the
> > discussion thread that we will start a vote!!
> > Thanks,
> > Sankalp
> >
> > On Wed, Sep 12, 2018 at 12:47 PM Jeff Jirsa  wrote:
> >
> > > On Wed, Sep 12, 2018 at 12:41 PM Sylvain Lebresne 
> > > wrote:
> > >
> > > > That's probably a stupid question, and excuse me if it is, but what
> > does
> > > > those votes on the dev mailing list even mean?
> > > >
> > > > How do you count votes at the end? Just by counting all votes cast,
> > > > irregardless of whomever cast it? Or are we intending to only count
> PMC
> > > > members, or maybe committers votes?
> > > >
> > >
> > > I believe the intent is to try to see if there exists consensus.
> > > Ultimately, PMC is going to matter more than random email addresses
> from
> > > people nobody recognizes. This should be in public, though, not
> private,
> > so
> > > seeing what feedback is beyond the PMC is useful (primarily because it
> > will
> > > matter when it comes time to extend and maintain it - if people
> strongly
> > > prefer one or the other, then maintenance is going to be a problem).
> > >
> > > If there's 100 random non-contributor votes for one option and 20 pmc
> > votes
> > > for another options, I think the real answer will be "we don't have
> > > consensus, and either we don't do it, or we do it the way the PMC
> thinks
> > is
> > > best", for all of the reasons you describe in the paragraphs below.
> > >
> > >
> > > > If the former, that is a bit weird to me because we simply don't know
> > who
> > > > votes. And I don't mean to be rude towards anyone, but 1) someone
> could
> > > > easily create 10 email addresses to vote 10 times (and sure, you
> could
> > > > invoke trust, and I'm not entirely against trust in general, but it's
> > the
> > > > internet...) and 2) this kind of decision will have non-trivial
> > > > consequences for the project, particularly on those that maintain it,
> > so
> > > I
> > > > admit I'm not entirely comfortable with "anyone's voice has the same
> > > > weight".
> > > > If the latter, then this makes more sense to me (why are we even
> > > bothering
> > > > voting PMC members in if it's not to handle these kinds of decisions,
> > > which
> > > > are very "project management" related), but we should be very clear
> > about
> > > > this from the get go (we could still use the dev list for
> transparency
> > > > sake, that I don't mind)? We should probably also have some deadline
> to
> > > the
> > > > vote, one that isn't too short.
> > > >
> > >
> > > Like releases, I think PMC votes count
> > >
> > >
> > > >
> > > > Anyway, fwiw, my opinion on this vote is not far from the one on the
> > > golang
> > > > driver acceptance vote (for which my remark above also apply btw): no
> > yet
> > > > 100% convinced adding more pieces and scope to the project is what
> the
> > > > project needs just right now, but not strongly opposed if people
> really
> > > > wants this (and this one makes more sense to me than the golang
> driver
> > > > actually). But if I'm to pick between a) and b), I'm leaning b).
> > > >
> > >
> > > FWIW, two of the main reasons I'm in favor is as a way to lower barrier
> > to
> > > entry to both using the software AND contributing to the project, so I
> > > think your points are valid (both on gocql thread and on this note
> > above),
> > > but I think that's also part of why we should be encouraging both.
> > >
> > > - Jeff
> > >
> >
>
>
> --
> Jon Haddad
> http://www.rustyrazorblade.com
> twitter: rustyrazorblade
>

Re: [VOTE] Development Approach for Apache Cassandra Management process

[sankalp kohli]

Also my vote is same as Jeff. d but would slightly prefer b. It is
important we make progress as we have been discussing this since April!!

On Wed, Sep 12, 2018 at 1:52 PM sankalp kohli 
wrote:

> The last email on the thread was 3 days ago and I made it clear days back
> that we should vote on it to make progress. Without this vote, I am not
> sure we will make progress.
> Many people want to contribute on this and hence we are voting so we can
> make progress.
>
> My vote is d
>
>
> On Wed, Sep 12, 2018 at 1:36 PM Jonathan Haddad  wrote:
>
>> This voting process feels a bit rushed and frankly not well thought out.
>> In addition to Sylvain's valid points, which you (Sankalp) didn't address
>> at all, the discussion in the other threads seemed to be ongoing.  The
>> last
>> email you wrote on one of them was asking for additional feedback, that
>> indicates the discussion is still open.
>>
>> Out of principal I vote for none of the options (inaction).  You're
>> deliberately trying to ram *something* through, and that's not how this is
>> supposed to work.
>>
>> For those of you unfamiliar with the process - please read
>> https://www.apache.org/foundation/voting.html.
>>
>> I'd like to ask those of you that are +1'ing, are you willing to
>> contribute
>> or are you just voting we start an admin tool from scratch because you
>> think it'll somehow produce a perfect codebase?
>>
>>
>>
>>
>>
>> On Wed, Sep 12, 2018 at 12:50 PM sankalp kohli 
>> wrote:
>>
>> > Hi Sylvain,
>> > I would appreciate if we can give feedback on the
>> > discussion threads and not wait for vote threads. I made it clear in the
>> > discussion thread that we will start a vote!!
>> > Thanks,
>> > Sankalp
>> >
>> > On Wed, Sep 12, 2018 at 12:47 PM Jeff Jirsa  wrote:
>> >
>> > > On Wed, Sep 12, 2018 at 12:41 PM Sylvain Lebresne > >
>> > > wrote:
>> > >
>> > > > That's probably a stupid question, and excuse me if it is, but what
>> > does
>> > > > those votes on the dev mailing list even mean?
>> > > >
>> > > > How do you count votes at the end? Just by counting all votes cast,
>> > > > irregardless of whomever cast it? Or are we intending to only count
>> PMC
>> > > > members, or maybe committers votes?
>> > > >
>> > >
>> > > I believe the intent is to try to see if there exists consensus.
>> > > Ultimately, PMC is going to matter more than random email addresses
>> from
>> > > people nobody recognizes. This should be in public, though, not
>> private,
>> > so
>> > > seeing what feedback is beyond the PMC is useful (primarily because it
>> > will
>> > > matter when it comes time to extend and maintain it - if people
>> strongly
>> > > prefer one or the other, then maintenance is going to be a problem).
>> > >
>> > > If there's 100 random non-contributor votes for one option and 20 pmc
>> > votes
>> > > for another options, I think the real answer will be "we don't have
>> > > consensus, and either we don't do it, or we do it the way the PMC
>> thinks
>> > is
>> > > best", for all of the reasons you describe in the paragraphs below.
>> > >
>> > >
>> > > > If the former, that is a bit weird to me because we simply don't
>> know
>> > who
>> > > > votes. And I don't mean to be rude towards anyone, but 1) someone
>> could
>> > > > easily create 10 email addresses to vote 10 times (and sure, you
>> could
>> > > > invoke trust, and I'm not entirely against trust in general, but
>> it's
>> > the
>> > > > internet...) and 2) this kind of decision will have non-trivial
>> > > > consequences for the project, particularly on those that maintain
>> it,
>> > so
>> > > I
>> > > > admit I'm not entirely comfortable with "anyone's voice has the same
>> > > > weight".
>> > > > If the latter, then this makes more sense to me (why are we even
>> > > bothering
>> > > > voting PMC members in if it's not to handle these kinds of
>> decisions,
>> > > which
>> > > > are very "project management" related), but we should 

Re: [VOTE] Development Approach for Apache Cassandra Management process

[sankalp kohli]

If you think vote is being forced, why not reply to my email on another
thread when I said we should vote? Why was the thread dead for months and
someone comes back with a contribution and then people starts talking?

I would have happily waited for few more days!!



On Wed, Sep 12, 2018 at 2:09 PM Joshua McKenzie 
wrote:

> >
> >  It is important we make progress as we have been discussing this since
> > April!!
>
>
> The discussion was making progress. Just because you want things to happen
> faster is no reason to force an early vote.
>
> On Wed, Sep 12, 2018 at 5:04 PM sankalp kohli 
> wrote:
>
> > Also my vote is same as Jeff. d but would slightly prefer b. It is
> > important we make progress as we have been discussing this since April!!
> >
> > On Wed, Sep 12, 2018 at 1:52 PM sankalp kohli 
> > wrote:
> >
> > > The last email on the thread was 3 days ago and I made it clear days
> back
> > > that we should vote on it to make progress. Without this vote, I am not
> > > sure we will make progress.
> > > Many people want to contribute on this and hence we are voting so we
> can
> > > make progress.
> > >
> > > My vote is d
> > >
> > >
> > > On Wed, Sep 12, 2018 at 1:36 PM Jonathan Haddad 
> > wrote:
> > >
> > >> This voting process feels a bit rushed and frankly not well thought
> out.
> > >> In addition to Sylvain's valid points, which you (Sankalp) didn't
> > address
> > >> at all, the discussion in the other threads seemed to be ongoing.  The
> > >> last
> > >> email you wrote on one of them was asking for additional feedback,
> that
> > >> indicates the discussion is still open.
> > >>
> > >> Out of principal I vote for none of the options (inaction).  You're
> > >> deliberately trying to ram *something* through, and that's not how
> this
> > is
> > >> supposed to work.
> > >>
> > >> For those of you unfamiliar with the process - please read
> > >> https://www.apache.org/foundation/voting.html.
> > >>
> > >> I'd like to ask those of you that are +1'ing, are you willing to
> > >> contribute
> > >> or are you just voting we start an admin tool from scratch because you
> > >> think it'll somehow produce a perfect codebase?
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> On Wed, Sep 12, 2018 at 12:50 PM sankalp kohli <
> kohlisank...@gmail.com>
> > >> wrote:
> > >>
> > >> > Hi Sylvain,
> > >> > I would appreciate if we can give feedback on the
> > >> > discussion threads and not wait for vote threads. I made it clear in
> > the
> > >> > discussion thread that we will start a vote!!
> > >> > Thanks,
> > >> > Sankalp
> > >> >
> > >> > On Wed, Sep 12, 2018 at 12:47 PM Jeff Jirsa 
> wrote:
> > >> >
> > >> > > On Wed, Sep 12, 2018 at 12:41 PM Sylvain Lebresne <
> > lebre...@gmail.com
> > >> >
> > >> > > wrote:
> > >> > >
> > >> > > > That's probably a stupid question, and excuse me if it is, but
> > what
> > >> > does
> > >> > > > those votes on the dev mailing list even mean?
> > >> > > >
> > >> > > > How do you count votes at the end? Just by counting all votes
> > cast,
> > >> > > > irregardless of whomever cast it? Or are we intending to only
> > count
> > >> PMC
> > >> > > > members, or maybe committers votes?
> > >> > > >
> > >> > >
> > >> > > I believe the intent is to try to see if there exists consensus.
> > >> > > Ultimately, PMC is going to matter more than random email
> addresses
> > >> from
> > >> > > people nobody recognizes. This should be in public, though, not
> > >> private,
> > >> > so
> > >> > > seeing what feedback is beyond the PMC is useful (primarily
> because
> > it
> > >> > will
> > >> > > matter when it comes time to extend and maintain it - if people
> > >> strongly
> > >> > > prefer one or the other, then maintenance is going to be a
> problem).
> > >> > >
> > >>

Re: [VOTE] Development Approach for Apache Cassandra Management process

[sankalp kohli]

Congrats on the newborn. I am assuming others also have personal things
going on!!

Also discussion thread is ongoing from April and not last 4 days. If enough
people think it is rushed, we can always revote.

On Wed, Sep 12, 2018 at 2:24 PM Joshua McKenzie 
wrote:

> That was four days ago, and I have a newborn at home. Not a lot of time for
> people to respond that have other things going on in life. :)
>
> On Wed, Sep 12, 2018 at 5:13 PM sankalp kohli 
> wrote:
>
> > If you think vote is being forced, why not reply to my email on another
> > thread when I said we should vote? Why was the thread dead for months and
> > someone comes back with a contribution and then people starts talking?
> >
> > I would have happily waited for few more days!!
> >
> >
> >
> > On Wed, Sep 12, 2018 at 2:09 PM Joshua McKenzie 
> > wrote:
> >
> > > >
> > > >  It is important we make progress as we have been discussing this
> since
> > > > April!!
> > >
> > >
> > > The discussion was making progress. Just because you want things to
> > happen
> > > faster is no reason to force an early vote.
> > >
> > > On Wed, Sep 12, 2018 at 5:04 PM sankalp kohli 
> > > wrote:
> > >
> > > > Also my vote is same as Jeff. d but would slightly prefer b. It is
> > > > important we make progress as we have been discussing this since
> > April!!
> > > >
> > > > On Wed, Sep 12, 2018 at 1:52 PM sankalp kohli <
> kohlisank...@gmail.com>
> > > > wrote:
> > > >
> > > > > The last email on the thread was 3 days ago and I made it clear
> days
> > > back
> > > > > that we should vote on it to make progress. Without this vote, I am
> > not
> > > > > sure we will make progress.
> > > > > Many people want to contribute on this and hence we are voting so
> we
> > > can
> > > > > make progress.
> > > > >
> > > > > My vote is d
> > > > >
> > > > >
> > > > > On Wed, Sep 12, 2018 at 1:36 PM Jonathan Haddad  >
> > > > wrote:
> > > > >
> > > > >> This voting process feels a bit rushed and frankly not well
> thought
> > > out.
> > > > >> In addition to Sylvain's valid points, which you (Sankalp) didn't
> > > > address
> > > > >> at all, the discussion in the other threads seemed to be ongoing.
> > The
> > > > >> last
> > > > >> email you wrote on one of them was asking for additional feedback,
> > > that
> > > > >> indicates the discussion is still open.
> > > > >>
> > > > >> Out of principal I vote for none of the options (inaction).
> You're
> > > > >> deliberately trying to ram *something* through, and that's not how
> > > this
> > > > is
> > > > >> supposed to work.
> > > > >>
> > > > >> For those of you unfamiliar with the process - please read
> > > > >> https://www.apache.org/foundation/voting.html.
> > > > >>
> > > > >> I'd like to ask those of you that are +1'ing, are you willing to
> > > > >> contribute
> > > > >> or are you just voting we start an admin tool from scratch because
> > you
> > > > >> think it'll somehow produce a perfect codebase?
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Wed, Sep 12, 2018 at 12:50 PM sankalp kohli <
> > > kohlisank...@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > Hi Sylvain,
> > > > >> > I would appreciate if we can give feedback on
> the
> > > > >> > discussion threads and not wait for vote threads. I made it
> clear
> > in
> > > > the
> > > > >> > discussion thread that we will start a vote!!
> > > > >> > Thanks,
> > > > >> > Sankalp
> > > > >> >
> > > > >> > On Wed, Sep 12, 2018 at 12:47 PM Jeff Jirsa 
> > > wrote:
> > > > >> >
> > > > >> > > On Wed, Sep 12, 2018 at 12:41 PM Sylvain Lebresne <
> > > > lebre...@gmail.com
> > > > 

Re: [VOTE] Development Approach for Apache Cassandra Management process

[sankalp kohli]

Hi Joey,
 The intention of this vote is to do what you are saying. We
want to see movement on this and not drag it for months. I am happy to drag
it for few more weeks if thats is what we agree on.

Regarding evaluating different options, if we decide on option a, we can
always do that like I wrote in the first email.

Thanks,
Sankalp

On Wed, Sep 12, 2018 at 2:51 PM Joseph Lynch  wrote:

> > I'd like to ask those of you that are +1'ing, are you willing to
> contribute
> > or are you just voting we start an admin tool from scratch because you
> > think it'll somehow produce a perfect codebase?
>
> Roopa, Vinay, Sumanth and I are voting as community members (and a
> sizeable user) and our willingness to contribute should be clear from
> the close to six months of engineering work we've invested presenting,
> prototyping, deploying, refactoring, designing, more discussing, and
> producing the patch on CASSANDRA-14346 that then happened to revive
> the April maintenance process discussion as we needed something to put
> the scheduler in. Dinesh and other Apple contributors were the ones
> who floated the idea in the first place and we just tried to help that
> proposal actually happen. Clearly we have to re-work that patch as it
> seems we have turned the management process discussion into a
> discussion about repair which was not the point and we are already in
> the process of converting the patch into a pluggable maintenance
> execution engine for any scheduled task.
>
> I didn't understand this vote as a vote on on releasing the yet to
> exist maintenance process ... I thought we're trying to get consensus
> on if we should invest in a fresh repo and build to the design (when
> we have achieved the design there can be an actual release vote) or
> start from an existing project which has made existing choices and
> trying to refactor towards the scope/desires.
>
> -Joey
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: Proposing an Apache Cassandra Management process

[sankalp kohli]

(Using this thread and not the vote thread intentionally)
For folks talking about vote being rushed. I would use the email from
Joseph to show this is not rushed. There was no email on this thread for 4
months until I pinged.


Dec 2016: Vinay worked with Jon and Alex to try to collaborate on Reaper to
come up with design goals for a repair scheduler that could work at Netflix
scale.

~Feb 2017: Netflix believes that the fundamental design gaps prevented us
from using Reaper as it relies heavily on remote JMX connections and
central coordination.

Sep. 2017: Vinay gives a lightning talk at NGCC about a highly available
and distributed repair scheduling sidecar/tool. He is encouraged by
multiple committers to build repair scheduling into the daemon itself and
not as a sidecar so the database is truly eventually consistent.

~Jun. 2017 - Feb. 2018: Based on internal need and the positive feedback at
NGCC, Vinay and myself prototype the distributed repair scheduler within
Priam and roll it out at Netflix scale.

Mar. 2018: I open a Jira (CASSANDRA-14346) along with a detailed 20 page
design document for adding repair scheduling to the daemon itself and open
the design up for feedback from the community. We get feedback from Alex,
Blake, Nate, Stefan, and Mick. As far as I know there were zero proposals
to contribute Reaper at this point. We hear the consensus that the
community would prefer repair scheduling in a separate distributed sidecar
rather than in the daemon itself and we re-work the design to match this
consensus, re-aligning with our original proposal at NGCC.

Apr 2018: Blake brings the discussion of repair scheduling to the dev list (
https://lists.apache.org/thread.html/760fbef677f27aa5c2ab4c375c7efeb81304fea428deff986ba1c2eb@%3Cdev.cassandra.apache.org%3E
).
Many community members give positive feedback that we should solve it as
part of Cassandra and there is still no mention of contributing Reaper at
this point. The last message is my attempted summary giving context on how
we want to take the best of all the sidecars (OpsCenter, Priam, Reaper) and
ship them with Cassandra.

Apr. 2018: Dinesh opens CASSANDRA-14395 along with a public design document
for gathering feedback on a general management sidecar. Sankalp and Dinesh
encourage Vinay and myself to kickstart that sidecar using the repair
scheduler patch

Apr 2018: Dinesh reaches out to the dev list (
https://lists.apache.org/thread.html/a098341efd8f344494bcd2761dba5125e971b59b1dd54f282ffda253@%3Cdev.cassandra.apache.org%3E
)
about the general management process to gain further feedback. All feedback
remains positive as it is a potential place for multiple community members
to contribute their various sidecar functionality.

May-Jul 2017: Vinay and I work on creating a basic sidecar for running the
repair scheduler based on the feedback from the community in
CASSANDRA-14346 and CASSANDRA-14395

Jun 2018: I bump CASSANDRA-14346 indicating we're still working on this,
nobody objects

Jul 2018: Sankalp asks on the dev list if anyone has feature Jiras anyone
needs review for before 4.0, I mention again that we've nearly got the
basic sidecar and repair scheduling work done and will need help with
review. No one responds.

Aug 2018: We submit a patch that brings a basic distributed sidecar and
robust distributed repair to Cassandra itself. Dinesh mentions that he will
try to review. Now folks appear concerned about it being in tree and
instead maybe it should go in a different repo all together. I don't think
we have consensus on the repo choice yet.

On Sun, Sep 9, 2018 at 9:13 AM sankalp kohli  wrote:

> I agree with Jon and I think folks who are talking about tech debts in
> Reaper should elaborate with examples about these tech debts. Can we be
> more precise and list them down? I see it spread out over this long email
> thread!!
>
> On Sun, Sep 9, 2018 at 6:29 AM Elliott Sims  wrote:
>
>> A big one to add to your list there, IMO as a user:
>> * API for determining detailed repair state (and history?).  Essentially,
>> something beyond just "Is some sort of repair running?" so that tools like
>> Reaper can parallelize better.
>>
>> On Sun, Sep 9, 2018 at 8:30 AM, Stefan Podkowinski 
>> wrote:
>>
>> > Does it have to be a single project with functionality provided by
>> > multiple plugins? Designing a plugin API at this point seems to be a bit
>> > early and comes with additional complexity around managing plugins in
>> > general.
>> >
>> > I was more thinking into the direction of: "what can we do to enable
>> > people to create any kind of side car or tooling solution?". Thinks
>> like:
>> >
>> > Common cluster discovery and management API
>> > * Detect local Cassandra processes
>> > * Discover and receive events on cluster

Re: [VOTE] Development Approach for Apache Cassandra Management process

[sankalp kohli]

I am hoping all the folks who are saying we should not vote will drive the
other thread.  Also note that there is consensus about doing a side car but
no consensus on which approach to take. I hope not deciding which approach
is not a poison pill for side car!!



On Wed, Sep 12, 2018 at 4:11 PM Mick Semb Wever  wrote:

>
> > But I'd like to see a serious investigation of the options -- feature
> set,
> > maturity, maintainer availability, etc -- before making a decision.  This
> > will take some time, but this is a place where "measure twice, cut once"
> > seems like the right approach.
>
>
> This^ 100%.
>
> I dislike the idea of a vote altogether:
> https://www.apache.org/foundation/voting.html#reasons-for-votes
>
> Discussion should resolve things, accepting something into the community
> that divides (isn't built on consensus) seems an unhealthy thing to do to
> the community. Jonathan's suggestion gives us something productive to do to
> get to consensus.
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: [VOTE] Development Approach for Apache Cassandra Management process

[Sankalp Kohli]

The link to the document is available in the other thread. Comparisons are 
available in other thread as well. 

> On Sep 12, 2018, at 16:29, Mick Semb Wever  wrote:
> 
> 
>> I am hoping all the folks who are saying we should not vote will drive the
>> other thread.  Also note that there is consensus about doing a side car but
>> no consensus on which approach to take. I hope not deciding which approach
>> is not a poison pill for side car!!
> 
> 
> Call me pedantic, but I saw the consensus as favouring a side-car over 
> something in tree. That's not a consensus on doing a side-car, as that was 
> not an option on offer.
> 
> There's certainly interest in a side-car that warrants documenting further on 
> comparisons and investigations, IMHO.
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[sankalp kohli]

Here is a list of open discussion points from the voting thread. I think
some are already answered but I will still gather these questions here.

>From several people:
1. Vote is rushed and we need more time for discussion.

>From Sylvain
2. About the voting process...I think that was addressed by Jeff Jirsa and
deserves a separate thread as it is not directly related to this thread.
3. Does the project need a side car.

>From Jonathan Haddad
4. Are people doing +1 willing to contribute

>From Jonathan Ellis
5. List of feature set, maturity, maintainer availability from Reaper or
any other project being donated.

Mick Semb Wever
6. We should not vote on these things and instead build consensus.

Open Questions from this thread
7. What technical debts we are talking about in Reaper. Can someone give
concrete examples.
8. What is the timeline of donating Reaper to Apache Cassandra.

On Wed, Sep 12, 2018 at 3:49 PM sankalp kohli 
wrote:

> (Using this thread and not the vote thread intentionally)
> For folks talking about vote being rushed. I would use the email from
> Joseph to show this is not rushed. There was no email on this thread for 4
> months until I pinged.
>
>
> Dec 2016: Vinay worked with Jon and Alex to try to collaborate on Reaper to
> come up with design goals for a repair scheduler that could work at Netflix
> scale.
>
> ~Feb 2017: Netflix believes that the fundamental design gaps prevented us
> from using Reaper as it relies heavily on remote JMX connections and
> central coordination.
>
> Sep. 2017: Vinay gives a lightning talk at NGCC about a highly available
> and distributed repair scheduling sidecar/tool. He is encouraged by
> multiple committers to build repair scheduling into the daemon itself and
> not as a sidecar so the database is truly eventually consistent.
>
> ~Jun. 2017 - Feb. 2018: Based on internal need and the positive feedback at
> NGCC, Vinay and myself prototype the distributed repair scheduler within
> Priam and roll it out at Netflix scale.
>
> Mar. 2018: I open a Jira (CASSANDRA-14346) along with a detailed 20 page
> design document for adding repair scheduling to the daemon itself and open
> the design up for feedback from the community. We get feedback from Alex,
> Blake, Nate, Stefan, and Mick. As far as I know there were zero proposals
> to contribute Reaper at this point. We hear the consensus that the
> community would prefer repair scheduling in a separate distributed sidecar
> rather than in the daemon itself and we re-work the design to match this
> consensus, re-aligning with our original proposal at NGCC.
>
> Apr 2018: Blake brings the discussion of repair scheduling to the dev list
> (
>
> https://lists.apache.org/thread.html/760fbef677f27aa5c2ab4c375c7efeb81304fea428deff986ba1c2eb@%3Cdev.cassandra.apache.org%3E
> ).
> Many community members give positive feedback that we should solve it as
> part of Cassandra and there is still no mention of contributing Reaper at
> this point. The last message is my attempted summary giving context on how
> we want to take the best of all the sidecars (OpsCenter, Priam, Reaper) and
> ship them with Cassandra.
>
> Apr. 2018: Dinesh opens CASSANDRA-14395 along with a public design document
> for gathering feedback on a general management sidecar. Sankalp and Dinesh
> encourage Vinay and myself to kickstart that sidecar using the repair
> scheduler patch
>
> Apr 2018: Dinesh reaches out to the dev list (
>
> https://lists.apache.org/thread.html/a098341efd8f344494bcd2761dba5125e971b59b1dd54f282ffda253@%3Cdev.cassandra.apache.org%3E
> )
> about the general management process to gain further feedback. All feedback
> remains positive as it is a potential place for multiple community members
> to contribute their various sidecar functionality.
>
> May-Jul 2017: Vinay and I work on creating a basic sidecar for running the
> repair scheduler based on the feedback from the community in
> CASSANDRA-14346 and CASSANDRA-14395
>
> Jun 2018: I bump CASSANDRA-14346 indicating we're still working on this,
> nobody objects
>
> Jul 2018: Sankalp asks on the dev list if anyone has feature Jiras anyone
> needs review for before 4.0, I mention again that we've nearly got the
> basic sidecar and repair scheduling work done and will need help with
> review. No one responds.
>
> Aug 2018: We submit a patch that brings a basic distributed sidecar and
> robust distributed repair to Cassandra itself. Dinesh mentions that he will
> try to review. Now folks appear concerned about it being in tree and
> instead maybe it should go in a different repo all together. I don't think
> we have consensus on the repo choice yet.
>
> On Sun, Sep 9, 2018 at 9:13 AM sankalp kohli 
> wrote:
>
>> I agree

Re: Proposing an Apache Cassandra Management process

[sankalp kohli]

How about we start with a few basic features in side car. How about
starting with this
1. Bulk nodetool commands: User can curl any sidecar and be able to run a
nodetool command in bulk across the cluster.
:/bulk/nodetool/tablestats?arg0=keyspace_name.table_name&arg1=

And later
2: Health checks.

On Thu, Sep 13, 2018 at 11:34 AM dinesh.jo...@yahoo.com.INVALID
 wrote:

> I will update the document to add that point. The document did not mean to
> serve as a design or architectural document but rather something that would
> spark a discussion on the idea.
> Dinesh
>
> On Thursday, September 13, 2018, 10:59:34 AM PDT, Jonathan Haddad <
> j...@jonhaddad.com> wrote:
>
>  Most of the discussion and work was done off the mailing list - there's a
> big risk involved when folks disappear for months at a time and resurface
> with big pile of code plus an agenda that you failed to loop everyone in
> on. In addition, by your own words the design document didn't accurately
> describe what was being built.  I don't write this to try to argue about
> it, I just want to put some perspective for those of us that weren't part
> of this discussion on a weekly basis over the last several months.  Going
> forward let's keep things on the ML so we can avoid confusion and
> frustration for all parties.
>
> With that said - I think Blake made a really good point here and it's
> helped me understand the scope of what's being built better.  Looking at it
> from a different perspective it doesn't seem like there's as much overlap
> as I had initially thought.  There's the machinery that runs certain tasks
> (what Joey has been working on) and the user facing side of exposing that
> information in management tool.
>
> I do appreciate (and like) the idea of not trying to boil the ocean, and
> working on things incrementally.  Putting a thin layer on top of Cassandra
> that can perform cluster wide tasks does give us an opportunity to move in
> the direction of a general purpose user-facing admin tool without
> committing to trying to write the full stack all at once (or even make
> decisions on it now).  We do need a sensible way of doing rolling restarts
> / scrubs / scheduling and Reaper wasn't built for that, and even though we
> can add it I'm not sure if it's the best mechanism for the long term.
>
> So if your goal is to add maturity to the project by making cluster wide
> tasks easier by providing a framework to build on top of, I'm in favor of
> that and I don't see it as antithetical to what I had in mind with Reaper.
> Rather, the two are more complementary than I had originally realized.
>
> Jon
>
>
>
>
> On Thu, Sep 13, 2018 at 10:39 AM dinesh.jo...@yahoo.com.INVALID
>  wrote:
>
> > I have a few clarifications -
> > The scope of the management process is not to simply run repair
> > scheduling. Repair scheduling is one of the many features we could
> > implement or adopt from existing sources. So could we please split the
> > Management Process discussion and the repair scheduling?
> > After re-reading the management process proposal, I see we missed to
> > communicate a basic idea in the document. We wanted to take a pluggable
> > approach to various activities that the management process could perform.
> > This could accommodate different implementations of common activities
> such
> > as repair. The management process would provide the basic framework and
> it
> > would have default implementations for some of the basic activities. This
> > would allow for speedier iteration cycles and keep things extensible.
> > Turning to some questions that Jon and others have raised, when I +1, my
> > intention is to fully contribute and stay with this community. That said,
> > things feel rushed for some but for me it feels like analysis paralysis.
> > We're looking for actionable feedback and to discuss the management
> process
> > _not_ repair scheduling solutions.
> > Thanks,
> > Dinesh
> >
> >
> >
> > On Sep 12, 2018, at 6:24 PM, sankalp kohli 
> wrote:
> > Here is a list of open discussion points from the voting thread. I think
> > some are already answered but I will still gather these questions here.
> >
> > From several people:
> > 1. Vote is rushed and we need more time for discussion.
> >
> > From Sylvain
> > 2. About the voting process...I think that was addressed by Jeff Jirsa
> and
> > deserves a separate thread as it is not directly related to this thread.
> > 3. Does the project need a side car.
> >
> > From Jonathan Haddad
> > 4. Are people doing +1 wil

Re: [DISCUSS] changing default token behavior for 4.0

[sankalp kohli]

+1 to lowering it.
Thanks Jon for starting this.We should create a JIRA to find what other
defaults we need revisit. (Please keep this discussion for "default token"
only.  )

On Fri, Sep 21, 2018 at 8:26 PM Jeff Jirsa  wrote:

> Also agree it should be lowered, but definitely not to 1, and probably
> something closer to 32 than 4.
>
> --
> Jeff Jirsa
>
>
> > On Sep 21, 2018, at 8:24 PM, Jeremy Hanna 
> wrote:
> >
> > I agree that it should be lowered. What I’ve seen debated a bit in the
> past is the number but I don’t think anyone thinks that it should remain
> 256.
> >
> >> On Sep 21, 2018, at 7:05 PM, Jonathan Haddad  wrote:
> >>
> >> One thing that's really, really bothered me for a while is how we
> default
> >> to 256 tokens still.  There's no experienced operator that leaves it as
> is
> >> at this point, meaning the only people using 256 are the poor folks that
> >> just got started using C*.  I've worked with over a hundred clusters in
> the
> >> last couple years, and I think I only worked with one that had lowered
> it
> >> to something else.
> >>
> >> I think it's time we changed the default to 4 (or 8, up for debate).
> >>
> >> To improve the behavior, we need to change a couple other things.  The
> >> allocate_tokens_for_keyspace setting is... odd.  It requires you have a
> >> keyspace already created, which doesn't help on new clusters.  What I'd
> >> like to do is add a new setting, allocate_tokens_for_rf, and set it to
> 3 by
> >> default.
> >>
> >> To handle clusters that are already using 256 tokens, we could prevent
> the
> >> new node from joining unless a -D flag is set to explicitly allow
> >> imbalanced tokens.
> >>
> >> We've agreed to a trunk freeze, but I feel like this is important enough
> >> (and pretty trivial) to do now.  I'd also personally characterize this
> as a
> >> bug fix since 256 is horribly broken when the cluster gets to any
> >> reasonable size, but maybe I'm alone there.
> >>
> >> I honestly can't think of a use case where random tokens is a good
> choice
> >> anymore, so I'd be fine / ecstatic with removing it completely and
> >> requiring either allocate_tokens_for_keyspace (for existing clusters)
> >> or allocate_tokens_for_rf
> >> to be set.
> >>
> >> Thoughts?  Objections?
> >> --
> >> Jon Haddad
> >> http://www.rustyrazorblade.com
> >> twitter: rustyrazorblade
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: [DISCUSS] changing default token behavior for 4.0

[Sankalp Kohli]

Putting it on JIRA is to make sure someone is assigned to it and it is tracked. 
Changes should be discussed over ML like you are saying. 

On Sep 21, 2018, at 21:02, Jonathan Haddad  wrote:

>> We should create a JIRA to find what other defaults we need revisit.
> 
> Changing a default is a pretty big deal, I think we should discuss any
> changes to defaults here on the ML before moving it into JIRA.  It's nice
> to get a bit more discussion around the change than what happens in JIRA.
> 
> We (TLP) did some testing on 4 tokens and found it to work surprisingly
> well.   It wasn't particularly formal, but we verified the load stays
> pretty even with only 4 tokens as we added nodes to the cluster.  Higher
> token count hurts availability by increasing the number of nodes any given
> node is a neighbor with, meaning any 2 nodes that fail have an increased
> chance of downtime when using QUORUM.  In addition, with the recent
> streaming optimization it seems the token counts will give a greater chance
> of a node streaming entire sstables (with LCS), meaning we'll do a better
> job with node density out of the box.
> 
> Next week I can try to put together something a little more convincing.
> Weekend time.
> 
> Jon
> 
> 
> On Fri, Sep 21, 2018 at 8:45 PM sankalp kohli 
> wrote:
> 
>> +1 to lowering it.
>> Thanks Jon for starting this.We should create a JIRA to find what other
>> defaults we need revisit. (Please keep this discussion for "default token"
>> only.  )
>> 
>>> On Fri, Sep 21, 2018 at 8:26 PM Jeff Jirsa  wrote:
>>> 
>>> Also agree it should be lowered, but definitely not to 1, and probably
>>> something closer to 32 than 4.
>>> 
>>> --
>>> Jeff Jirsa
>>> 
>>> 
>>>> On Sep 21, 2018, at 8:24 PM, Jeremy Hanna 
>>> wrote:
>>>> 
>>>> I agree that it should be lowered. What I’ve seen debated a bit in the
>>> past is the number but I don’t think anyone thinks that it should remain
>>> 256.
>>>> 
>>>>> On Sep 21, 2018, at 7:05 PM, Jonathan Haddad 
>> wrote:
>>>>> 
>>>>> One thing that's really, really bothered me for a while is how we
>>> default
>>>>> to 256 tokens still.  There's no experienced operator that leaves it
>> as
>>> is
>>>>> at this point, meaning the only people using 256 are the poor folks
>> that
>>>>> just got started using C*.  I've worked with over a hundred clusters
>> in
>>> the
>>>>> last couple years, and I think I only worked with one that had lowered
>>> it
>>>>> to something else.
>>>>> 
>>>>> I think it's time we changed the default to 4 (or 8, up for debate).
>>>>> 
>>>>> To improve the behavior, we need to change a couple other things.  The
>>>>> allocate_tokens_for_keyspace setting is... odd.  It requires you have
>> a
>>>>> keyspace already created, which doesn't help on new clusters.  What
>> I'd
>>>>> like to do is add a new setting, allocate_tokens_for_rf, and set it to
>>> 3 by
>>>>> default.
>>>>> 
>>>>> To handle clusters that are already using 256 tokens, we could prevent
>>> the
>>>>> new node from joining unless a -D flag is set to explicitly allow
>>>>> imbalanced tokens.
>>>>> 
>>>>> We've agreed to a trunk freeze, but I feel like this is important
>> enough
>>>>> (and pretty trivial) to do now.  I'd also personally characterize this
>>> as a
>>>>> bug fix since 256 is horribly broken when the cluster gets to any
>>>>> reasonable size, but maybe I'm alone there.
>>>>> 
>>>>> I honestly can't think of a use case where random tokens is a good
>>> choice
>>>>> anymore, so I'd be fine / ecstatic with removing it completely and
>>>>> requiring either allocate_tokens_for_keyspace (for existing clusters)
>>>>> or allocate_tokens_for_rf
>>>>> to be set.
>>>>> 
>>>>> Thoughts?  Objections?
>>>>> --
>>>>> Jon Haddad
>>>>> http://www.rustyrazorblade.com
>>>>> twitter: rustyrazorblade
>>>> 
>>>> -
>>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>>> 
>>> 
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>> 
>>> 
>> 
> 
> 
> -- 
> Jon Haddad
> http://www.rustyrazorblade.com
> twitter: rustyrazorblade

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[Sankalp Kohli]

This is not part of core database and a separate repo and so my impression is 
that this can continue to make progress. Also we can always make progress and 
not merge it till freeze is lifted. 

Open to ideas/suggestions if someone thinks otherwise. 

> On Sep 22, 2018, at 03:13, kurt greaves  wrote:
> 
> Is this something we're moving ahead with despite the feature freeze?
> 
> On Sat, 22 Sep 2018 at 08:32, dinesh.jo...@yahoo.com.INVALID
>  wrote:
> 
>> I have created a sub-task - CASSANDRA-14783. Could we get some feedback
>> before we begin implementing anything?
>> 
>> Dinesh
>> 
>>On Thursday, September 20, 2018, 11:22:33 PM PDT, Dinesh Joshi <
>> dinesh.jo...@yahoo.com.INVALID> wrote:
>> 
>> I have updated the doc with a short paragraph providing the
>> clarification. Sankalp's suggestion is already part of the doc. If there
>> aren't further objections could we move this discussion over to the jira
>> (CASSANDRA-14395)?
>> 
>> Dinesh
>> 
>>> On Sep 18, 2018, at 10:31 AM, sankalp kohli 
>> wrote:
>>> 
>>> How about we start with a few basic features in side car. How about
>> starting with this
>>> 1. Bulk nodetool commands: User can curl any sidecar and be able to run
>> a nodetool command in bulk across the cluster.
>>> 
>> :/bulk/nodetool/tablestats?arg0=keyspace_name.table_name&arg1=> required>
>>> 
>>> And later
>>> 2: Health checks.
>>> 
>>> On Thu, Sep 13, 2018 at 11:34 AM dinesh.jo...@yahoo.com.INVALID <
>> dinesh.jo...@yahoo.com.invalid> wrote:
>>> I will update the document to add that point. The document did not mean
>> to serve as a design or architectural document but rather something that
>> would spark a discussion on the idea.
>>> Dinesh
>>> 
>>>   On Thursday, September 13, 2018, 10:59:34 AM PDT, Jonathan Haddad <
>> j...@jonhaddad.com <mailto:j...@jonhaddad.com>> wrote:
>>> 
>>> Most of the discussion and work was done off the mailing list - there's
>> a
>>> big risk involved when folks disappear for months at a time and resurface
>>> with big pile of code plus an agenda that you failed to loop everyone in
>>> on. In addition, by your own words the design document didn't accurately
>>> describe what was being built.  I don't write this to try to argue about
>>> it, I just want to put some perspective for those of us that weren't part
>>> of this discussion on a weekly basis over the last several months.  Going
>>> forward let's keep things on the ML so we can avoid confusion and
>>> frustration for all parties.
>>> 
>>> With that said - I think Blake made a really good point here and it's
>>> helped me understand the scope of what's being built better.  Looking at
>> it
>>> from a different perspective it doesn't seem like there's as much overlap
>>> as I had initially thought.  There's the machinery that runs certain
>> tasks
>>> (what Joey has been working on) and the user facing side of exposing that
>>> information in management tool.
>>> 
>>> I do appreciate (and like) the idea of not trying to boil the ocean, and
>>> working on things incrementally.  Putting a thin layer on top of
>> Cassandra
>>> that can perform cluster wide tasks does give us an opportunity to move
>> in
>>> the direction of a general purpose user-facing admin tool without
>>> committing to trying to write the full stack all at once (or even make
>>> decisions on it now).  We do need a sensible way of doing rolling
>> restarts
>>> / scrubs / scheduling and Reaper wasn't built for that, and even though
>> we
>>> can add it I'm not sure if it's the best mechanism for the long term.
>>> 
>>> So if your goal is to add maturity to the project by making cluster wide
>>> tasks easier by providing a framework to build on top of, I'm in favor of
>>> that and I don't see it as antithetical to what I had in mind with
>> Reaper.
>>> Rather, the two are more complementary than I had originally realized.
>>> 
>>> Jon
>>> 
>>> 
>>> 
>>> 
>>> On Thu, Sep 13, 2018 at 10:39 AM dinesh.jo...@yahoo.com.INVALID
>>> mailto:dinesh.jo...@yahoo.com>.invalid> wrote:
>>> 
>>>> I have a few clarifications -
>>>> The scope of the management process is not to simply run repair
>>>> sch

Re: Proposing an Apache Cassandra Management process

[sankalp kohli]

Thanks Dinesh for looking at the tools and thanks Mick for listing them
down.
Based on Dinesh response, is it accurate to say that for bulk
functionality, we have evaluated the tools listed by the community? If
anything is missed we should discuss it as we want to make sure we looked
at all aspects before implementation starts.

On Sat, Sep 29, 2018 at 1:19 PM Dinesh Joshi 
wrote:

> > On Sep 29, 2018, at 12:31 PM, Rahul Singh 
> wrote:
> >
> > All of Apache is a patchwork of tools. All of Open Source is a patchwork
> of tools. All of Linux is a patchwork of tools.
> >
> > What works, works.
>
> So there isn't a way to make it any better? You would prefer using an
> unsupported tool vs something that worked out of the box & was well tested?
>
> Dinesh
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: Proposing an Apache Cassandra Management process

[Sankalp Kohli]

Thank you for the feedback.  There are lot of advantages of doing this in 
Apache and you can read the thread to find more. The main one is to not divide 
the energy between different tools. 
Also we will not be reinventing the wheel as I think this project can still get 
donations along the way. 

> On Sep 30, 2018, at 05:31, Rahul Singh  wrote:
> 
> Perfection is the enemy of the good enough. All if not most informed open 
> source users understand that the tar they are downloading is “unsupported.”
> 
> Most of the blog posts people read or the documentation they have is in place 
> of tools. Open source software let alone Tooling even a nascent version comes 
> with a degree of uncertainty.
> 
> If the question is for me: I would rather use something that exists than 
> reinvent the wheel. The same way I’ll use “contrib” packages in any system 
> just to see if it works.
> 
> Example: While working on on a Solr / Kafka / Cassandra integration project 
> we must have used a few different “Kafka Connect” variations before settling 
> on a solution which was a combination of an existing sink connector and 
> created a source connector because what was out there “meh”.
> 
> If we had scoffed off “unsupported” packages we would have spent more time 
> making something than delivering value. Technology exists to serve business 
> and people’s lives not technology itself.
> 
> There is a level of discernment that comes with experience as to what works 
> and what doesn’t and what is good and what isn’t. The least we can do is help 
> the user community know the difference : as Apache does at a higher level.
> 
> 
> Rahul Singh
> Chief Executive Officer
> m 202.905.2818
> 
> Anant Corporation
> 1010 Wisconsin Ave NW, Suite 250
> Washington, D.C. 20007
> 
> We build and manage digital business technology platforms.
>> On Sep 29, 2018, 5:29 PM -0400, sankalp kohli , 
>> wrote:
>> Thanks Dinesh for looking at the tools and thanks Mick for listing them
>> down.
>> Based on Dinesh response, is it accurate to say that for bulk
>> functionality, we have evaluated the tools listed by the community? If
>> anything is missed we should discuss it as we want to make sure we looked
>> at all aspects before implementation starts.
>> 
>> On Sat, Sep 29, 2018 at 1:19 PM Dinesh Joshi 
>> wrote:
>> 
>>>> On Sep 29, 2018, at 12:31 PM, Rahul Singh 
>>> wrote:
>>>> 
>>>> All of Apache is a patchwork of tools. All of Open Source is a patchwork
>>> of tools. All of Linux is a patchwork of tools.
>>>> 
>>>> What works, works.
>>> 
>>> So there isn't a way to make it any better? You would prefer using an
>>> unsupported tool vs something that worked out of the box & was well tested?
>>> 
>>> Dinesh
>>> 
>>> 
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>> 
>>> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Proposing an Apache Cassandra Management process

[sankalp kohli]

Hi Mick,
Any other feedback?
Thanks,
Sankalp

On Sun, Sep 30, 2018 at 8:54 AM Sankalp Kohli 
wrote:

> Thank you for the feedback.  There are lot of advantages of doing this in
> Apache and you can read the thread to find more. The main one is to not
> divide the energy between different tools.
> Also we will not be reinventing the wheel as I think this project can
> still get donations along the way.
>
> > On Sep 30, 2018, at 05:31, Rahul Singh 
> wrote:
> >
> > Perfection is the enemy of the good enough. All if not most informed
> open source users understand that the tar they are downloading is
> “unsupported.”
> >
> > Most of the blog posts people read or the documentation they have is in
> place of tools. Open source software let alone Tooling even a nascent
> version comes with a degree of uncertainty.
> >
> > If the question is for me: I would rather use something that exists than
> reinvent the wheel. The same way I’ll use “contrib” packages in any system
> just to see if it works.
> >
> > Example: While working on on a Solr / Kafka / Cassandra integration
> project we must have used a few different “Kafka Connect” variations before
> settling on a solution which was a combination of an existing sink
> connector and created a source connector because what was out there “meh”.
> >
> > If we had scoffed off “unsupported” packages we would have spent more
> time making something than delivering value. Technology exists to serve
> business and people’s lives not technology itself.
> >
> > There is a level of discernment that comes with experience as to what
> works and what doesn’t and what is good and what isn’t. The least we can do
> is help the user community know the difference : as Apache does at a higher
> level.
> >
> >
> > Rahul Singh
> > Chief Executive Officer
> > m 202.905.2818
> >
> > Anant Corporation
> > 1010 Wisconsin Ave NW, Suite 250
> > Washington, D.C. 20007
> >
> > We build and manage digital business technology platforms.
> >> On Sep 29, 2018, 5:29 PM -0400, sankalp kohli ,
> wrote:
> >> Thanks Dinesh for looking at the tools and thanks Mick for listing them
> >> down.
> >> Based on Dinesh response, is it accurate to say that for bulk
> >> functionality, we have evaluated the tools listed by the community? If
> >> anything is missed we should discuss it as we want to make sure we
> looked
> >> at all aspects before implementation starts.
> >>
> >> On Sat, Sep 29, 2018 at 1:19 PM Dinesh Joshi  .invalid>
> >> wrote:
> >>
> >>>> On Sep 29, 2018, at 12:31 PM, Rahul Singh <
> rahul.xavier.si...@gmail.com>
> >>> wrote:
> >>>>
> >>>> All of Apache is a patchwork of tools. All of Open Source is a
> patchwork
> >>> of tools. All of Linux is a patchwork of tools.
> >>>>
> >>>> What works, works.
> >>>
> >>> So there isn't a way to make it any better? You would prefer using an
> >>> unsupported tool vs something that worked out of the box & was well
> tested?
> >>>
> >>> Dinesh
> >>>
> >>>
> >>> -
> >>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >>> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>>
> >>>
>

Re: Deprecating/removing PropertyFileSnitch?

[sankalp kohli]

Will GossipingPropertyFileSnitch not be vulnerable to Gossip bugs where we
lose hostId or some other fields when we restart C* for large
clusters(~1000 instances)?

On Tue, Oct 16, 2018 at 7:59 AM Jeff Jirsa  wrote:

> We should, but the 4.0 features that log/reject verbs to invalid replicas
> solves a lot of the concerns here
>
> --
> Jeff Jirsa
>
>
> > On Oct 16, 2018, at 4:10 PM, Jeremy Hanna 
> wrote:
> >
> > We have had PropertyFileSnitch for a long time even though
> GossipingPropertyFileSnitch is effectively a superset of what it offers and
> is much less error prone.  There are some unexpected behaviors when things
> aren’t configured correctly with PFS.  For example, if you replace nodes in
> one DC and add those nodes to that DCs property files and not the other DCs
> property files - the resulting problems aren’t very straightforward to
> troubleshoot.
> >
> > We could try to improve the resilience and fail fast error checking and
> error reporting of PFS, but honestly, why wouldn’t we deprecate and remove
> PropertyFileSnitch?  Are there reasons why GPFS wouldn’t be sufficient to
> replace it?
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: Proposing an Apache Cassandra Management process

[Sankalp Kohli]

Hi Mick,
 Can you share the link to cwiki if you have started it ?

Thanks
Sankalp 

> On Oct 4, 2018, at 5:20 PM, Mick Semb Wever  wrote:
> 
> Dinesh / Sankalp,
> 
> My suggestion was to document the landscape in hope and an attempt to better 
> understand the requirements possible to a side-car.  It wasn't a suggestion 
> to patchwork together everything. But rather as part of brainstorming, 
> designing, and exercising an inclusive process to see what's been done and 
> how it could have been done better. 
> 
> A well designed side-car could also be a valuable fundamental to some of 
> these third-party solutions, not just our own designs and ideals. Maybe, I 
> hope, that's already obvious.
> 
> It would be really fantastic to see more explorative documentation in 
> confluence. Part of that can be to list up all these external tools, listing 
> their goals, state, and how a side-car might help them. Reaching out to their 
> maintainers to be involved in the process would be awesome too. I can start 
> something in the cwiki (but i'm on vacation this week), I've also given you 
> write-access Dinesh.
> 
>> I also haven't seen a process to propose & discuss larger changes to 
>> Cassandra. The Cassandra contribution[1] guide possibly needs to be updated. 
>> Some communities have a process which facilitate things. See Kafka 
>> Improvement Process[2], Spark Improvement Process[3].
> 
> Bringing this up was gold, imho. I would love to see something like this 
> exist in the C* community (also in cwiki), and the side-car brainstorming and 
> design used to test and flesh it out.
> 
> regards,
> Mick 
> 
> 
> On Sun, 30 Sep 2018, at 05:19, Dinesh Joshi wrote:
>>> On Sep 27, 2018, at 7:35 PM, Mick Semb Wever  wrote:
>>> 
>>> Reaper,
>> 
>> I have looked at this already.
>> 
>>> Priam,
>> 
>> I have looked at this already.
>> 
>>> Marcus Olsson's offering,
>> 
>> This isn't OSS.
>> 
>>> CStar,
>> 
>> I have looked at this already.
>> 
>>> OpsCenter.
>> 
>> Latest release is only compatible with DSE and not Apache Cassandra[1]
>> 
>>> Then there's a host of command line tools like:
>>> ic-tools,
>>> ctop (was awesome, but is it maintained anymore?),
>>> tablesnap.
>> 
>> These are interesting tools and I don't think they do what we're 
>> interested in doing.
>> 
>>> And maybe it's worth including the diy approach people take… 
>>> pssh/dsh/clusterssh/mussh/fabric, etc
>> 
>> What's the point? You can definitely add this to the website as helpful 
>> documentation.
>> 
>> The proposal in the original thread was to create something that is 
>> supported by the Apache Cassandra project learning from the tooling 
>> we've all built over the years. The fact that everyone has a sidecar or 
>> their own internal tooling is an indicator that the project has room to 
>> grow. It will certainly help this project be more user friendly (at 
>> least for operators).
>> 
>> I, as a user and a developer, do not want to use a patchwork of 
>> disparate tools. Does anybody oppose this on technical grounds? If you 
>> do, please help me understand why would you prefer using a patchwork of 
>> tools vs something that is part of the Cassandra project?
>> 
>> Thanks,
>> 
>> Dinesh
>> 
>> [1] https://docs.datastax.com/en/opscenter/6.0/opsc/opscPolicyChanges.html
>> 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Deprecating/removing PropertyFileSnitch?

[sankalp kohli]

This is not the case during host replacement correct?

On Tue, Oct 16, 2018 at 10:04 AM Jeremiah D Jordan <
jeremiah.jor...@gmail.com> wrote:

> As long as we are correctly storing such things in the system tables and
> reading them out of the system tables when we do not have the information
> from gossip yet, it should not be a problem. (As far as I know GPFS does
> this, but I have not done extensive code diving or testing to make sure all
> edge cases are covered there)
>
> -Jeremiah
>
> > On Oct 16, 2018, at 11:56 AM, sankalp kohli 
> wrote:
> >
> > Will GossipingPropertyFileSnitch not be vulnerable to Gossip bugs where
> we
> > lose hostId or some other fields when we restart C* for large
> > clusters(~1000 instances)?
> >
> > On Tue, Oct 16, 2018 at 7:59 AM Jeff Jirsa  wrote:
> >
> >> We should, but the 4.0 features that log/reject verbs to invalid
> replicas
> >> solves a lot of the concerns here
> >>
> >> --
> >> Jeff Jirsa
> >>
> >>
> >>> On Oct 16, 2018, at 4:10 PM, Jeremy Hanna 
> >> wrote:
> >>>
> >>> We have had PropertyFileSnitch for a long time even though
> >> GossipingPropertyFileSnitch is effectively a superset of what it offers
> and
> >> is much less error prone.  There are some unexpected behaviors when
> things
> >> aren’t configured correctly with PFS.  For example, if you replace
> nodes in
> >> one DC and add those nodes to that DCs property files and not the other
> DCs
> >> property files - the resulting problems aren’t very straightforward to
> >> troubleshoot.
> >>>
> >>> We could try to improve the resilience and fail fast error checking and
> >> error reporting of PFS, but honestly, why wouldn’t we deprecate and
> remove
> >> PropertyFileSnitch?  Are there reasons why GPFS wouldn’t be sufficient
> to
> >> replace it?
> >>> -
> >>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >>> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>>
> >>
> >> -
> >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>
> >>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: CASSANDRA-13241 lower default chunk_length_in_kb

[Sankalp Kohli]

(We should definitely harden the definition for freeze in a separate thread)

My thinking is that this is the best time to do this change as we have not even 
cut alpha or beta. All the people involved in the test will definitely be 
testing it again when we have these releases. 

> On Oct 19, 2018, at 8:00 AM, Michael Shuler  wrote:
> 
>> On 10/19/18 9:16 AM, Joshua McKenzie wrote:
>> 
>> At the risk of hijacking this thread, when are we going to transition from
>> "no new features, change whatever else you want including refactoring and
>> changing years-old defaults" to "ok, we think we have something that's
>> stable, time to start testing"?
> 
> Creating a cassandra-4.0 branch would allow trunk to, for instance, get
> a default config value change commit and get more testing. We might
> forget again, from what I understand of Benedict's last comment :)
> 
> -- 
> Michael
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Deprecating/removing PropertyFileSnitch?

[Sankalp Kohli]

Say you restarted all instances in the cluster and status for some host goes 
missing. Now when you start a host replacement, the new host won’t learn about 
the host whose status is missing and the view of this host will be wrong.

PS: I will be happy to be proved wrong as I can also start using Gossip snitch 
:) 

> On Oct 19, 2018, at 2:41 PM, Jeremy Hanna  wrote:
> 
> Do you mean to say that during host replacement there may be a time when the 
> old->new host isn’t fully propagated and therefore wouldn’t yet be in all 
> system tables?
> 
>> On Oct 17, 2018, at 4:20 PM, sankalp kohli  wrote:
>> 
>> This is not the case during host replacement correct?
>> 
>> On Tue, Oct 16, 2018 at 10:04 AM Jeremiah D Jordan <
>> jeremiah.jor...@gmail.com> wrote:
>> 
>>> As long as we are correctly storing such things in the system tables and
>>> reading them out of the system tables when we do not have the information
>>> from gossip yet, it should not be a problem. (As far as I know GPFS does
>>> this, but I have not done extensive code diving or testing to make sure all
>>> edge cases are covered there)
>>> 
>>> -Jeremiah
>>> 
>>>> On Oct 16, 2018, at 11:56 AM, sankalp kohli 
>>> wrote:
>>>> 
>>>> Will GossipingPropertyFileSnitch not be vulnerable to Gossip bugs where
>>> we
>>>> lose hostId or some other fields when we restart C* for large
>>>> clusters(~1000 instances)?
>>>> 
>>>>> On Tue, Oct 16, 2018 at 7:59 AM Jeff Jirsa  wrote:
>>>>> 
>>>>> We should, but the 4.0 features that log/reject verbs to invalid
>>> replicas
>>>>> solves a lot of the concerns here
>>>>> 
>>>>> --
>>>>> Jeff Jirsa
>>>>> 
>>>>> 
>>>>>> On Oct 16, 2018, at 4:10 PM, Jeremy Hanna 
>>>>> wrote:
>>>>>> 
>>>>>> We have had PropertyFileSnitch for a long time even though
>>>>> GossipingPropertyFileSnitch is effectively a superset of what it offers
>>> and
>>>>> is much less error prone.  There are some unexpected behaviors when
>>> things
>>>>> aren’t configured correctly with PFS.  For example, if you replace
>>> nodes in
>>>>> one DC and add those nodes to that DCs property files and not the other
>>> DCs
>>>>> property files - the resulting problems aren’t very straightforward to
>>>>> troubleshoot.
>>>>>> 
>>>>>> We could try to improve the resilience and fail fast error checking and
>>>>> error reporting of PFS, but honestly, why wouldn’t we deprecate and
>>> remove
>>>>> PropertyFileSnitch?  Are there reasons why GPFS wouldn’t be sufficient
>>> to
>>>>> replace it?
>>>>>> -
>>>>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>>>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>>>>> 
>>>>> 
>>>>> -
>>>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>>>> 
>>>>> 
>>> 
>>> 
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>> 
>>> 
> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Deprecating/removing PropertyFileSnitch?

[sankalp kohli]

Yes it will happen. I am worried that same way DC or rack info can go
missing.

On Mon, Oct 22, 2018 at 12:52 PM Paulo Motta 
wrote:

> > the new host won’t learn about the host whose status is missing and the
> view of this host will be wrong.
>
> Won't this happen even with PropertyFileSnitch as the token(s) for this
> host will be missing from gossip/system.peers?
>
> Em sáb, 20 de out de 2018 às 00:34, Sankalp Kohli 
> escreveu:
>
> > Say you restarted all instances in the cluster and status for some host
> > goes missing. Now when you start a host replacement, the new host won’t
> > learn about the host whose status is missing and the view of this host
> will
> > be wrong.
> >
> > PS: I will be happy to be proved wrong as I can also start using Gossip
> > snitch :)
> >
> > > On Oct 19, 2018, at 2:41 PM, Jeremy Hanna 
> > wrote:
> > >
> > > Do you mean to say that during host replacement there may be a time
> when
> > the old->new host isn’t fully propagated and therefore wouldn’t yet be in
> > all system tables?
> > >
> > >> On Oct 17, 2018, at 4:20 PM, sankalp kohli 
> > wrote:
> > >>
> > >> This is not the case during host replacement correct?
> > >>
> > >> On Tue, Oct 16, 2018 at 10:04 AM Jeremiah D Jordan <
> > >> jeremiah.jor...@gmail.com> wrote:
> > >>
> > >>> As long as we are correctly storing such things in the system tables
> > and
> > >>> reading them out of the system tables when we do not have the
> > information
> > >>> from gossip yet, it should not be a problem. (As far as I know GPFS
> > does
> > >>> this, but I have not done extensive code diving or testing to make
> > sure all
> > >>> edge cases are covered there)
> > >>>
> > >>> -Jeremiah
> > >>>
> > >>>> On Oct 16, 2018, at 11:56 AM, sankalp kohli  >
> > >>> wrote:
> > >>>>
> > >>>> Will GossipingPropertyFileSnitch not be vulnerable to Gossip bugs
> > where
> > >>> we
> > >>>> lose hostId or some other fields when we restart C* for large
> > >>>> clusters(~1000 instances)?
> > >>>>
> > >>>>> On Tue, Oct 16, 2018 at 7:59 AM Jeff Jirsa 
> wrote:
> > >>>>>
> > >>>>> We should, but the 4.0 features that log/reject verbs to invalid
> > >>> replicas
> > >>>>> solves a lot of the concerns here
> > >>>>>
> > >>>>> --
> > >>>>> Jeff Jirsa
> > >>>>>
> > >>>>>
> > >>>>>> On Oct 16, 2018, at 4:10 PM, Jeremy Hanna <
> > jeremy.hanna1...@gmail.com>
> > >>>>> wrote:
> > >>>>>>
> > >>>>>> We have had PropertyFileSnitch for a long time even though
> > >>>>> GossipingPropertyFileSnitch is effectively a superset of what it
> > offers
> > >>> and
> > >>>>> is much less error prone.  There are some unexpected behaviors when
> > >>> things
> > >>>>> aren’t configured correctly with PFS.  For example, if you replace
> > >>> nodes in
> > >>>>> one DC and add those nodes to that DCs property files and not the
> > other
> > >>> DCs
> > >>>>> property files - the resulting problems aren’t very straightforward
> > to
> > >>>>> troubleshoot.
> > >>>>>>
> > >>>>>> We could try to improve the resilience and fail fast error
> checking
> > and
> > >>>>> error reporting of PFS, but honestly, why wouldn’t we deprecate and
> > >>> remove
> > >>>>> PropertyFileSnitch?  Are there reasons why GPFS wouldn’t be
> > sufficient
> > >>> to
> > >>>>> replace it?
> > >>>>>>
> > -
> > >>>>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > >>>>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
> > >>>>>>
> > >>>>>
> > >>>>>
> -
> > >>>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > >>>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
> > >>>>>
> > >>>>>
> > >>>
> > >>>
> > >>> -
> > >>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > >>> For additional commands, e-mail: dev-h...@cassandra.apache.org
> > >>>
> > >>>
> > >
> > >
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > > For additional commands, e-mail: dev-h...@cassandra.apache.org
> > >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >
>

Re: Deprecating/removing PropertyFileSnitch?

[sankalp kohli]

We will have similar issues with Gossip but this will create more issues as
more things will be relied on Gossip.

I agree PFS should be removed but I dont see how it can be with issues like
these or someone proves that it wont cause any issues.

On Mon, Oct 22, 2018 at 2:21 PM Paulo Motta 
wrote:

> I can understand keeping PFS for historical/compatibility reasons, but if
> gossip is broken I think you will have similar ring view problems during
> replace/bootstrap that would still occur with the use of PFS (such as
> missing tokens, since those are propagated via gossip), so that doesn't
> seem like a strong reason to keep it around.
>
> With PFS it's pretty easy to shoot yourself in the foot if you're not
> careful enough to have identical files across nodes and updating it when
> adding nodes/dcs, so it's seems to be less foolproof than other snitches.
> While the rejection of verbs to invalid replicas on trunk could address
> concerns raised by Jeremy, this would only happen after the new node joins
> the ring, so you would need to re-bootstrap the node and lose all the work
> done in the original bootstrap.
>
> Perhaps one good reason to use PFS is the ability to easily package it
> across multiple nodes, as pointed out by Sean Durity on CASSANDRA-10745
> (which is also it's Achilles' heel). To keep this ability, we could make
> GPFS compatible with the cassandra-topology.properties file, but reading
> only the dc/rack info about the local node.
>
> Em seg, 22 de out de 2018 às 16:58, sankalp kohli 
> escreveu:
>
> > Yes it will happen. I am worried that same way DC or rack info can go
> > missing.
> >
> > On Mon, Oct 22, 2018 at 12:52 PM Paulo Motta 
> > wrote:
> >
> > > > the new host won’t learn about the host whose status is missing and
> the
> > > view of this host will be wrong.
> > >
> > > Won't this happen even with PropertyFileSnitch as the token(s) for this
> > > host will be missing from gossip/system.peers?
> > >
> > > Em sáb, 20 de out de 2018 às 00:34, Sankalp Kohli <
> > kohlisank...@gmail.com>
> > > escreveu:
> > >
> > > > Say you restarted all instances in the cluster and status for some
> host
> > > > goes missing. Now when you start a host replacement, the new host
> won’t
> > > > learn about the host whose status is missing and the view of this
> host
> > > will
> > > > be wrong.
> > > >
> > > > PS: I will be happy to be proved wrong as I can also start using
> Gossip
> > > > snitch :)
> > > >
> > > > > On Oct 19, 2018, at 2:41 PM, Jeremy Hanna <
> > jeremy.hanna1...@gmail.com>
> > > > wrote:
> > > > >
> > > > > Do you mean to say that during host replacement there may be a time
> > > when
> > > > the old->new host isn’t fully propagated and therefore wouldn’t yet
> be
> > in
> > > > all system tables?
> > > > >
> > > > >> On Oct 17, 2018, at 4:20 PM, sankalp kohli <
> kohlisank...@gmail.com>
> > > > wrote:
> > > > >>
> > > > >> This is not the case during host replacement correct?
> > > > >>
> > > > >> On Tue, Oct 16, 2018 at 10:04 AM Jeremiah D Jordan <
> > > > >> jeremiah.jor...@gmail.com> wrote:
> > > > >>
> > > > >>> As long as we are correctly storing such things in the system
> > tables
> > > > and
> > > > >>> reading them out of the system tables when we do not have the
> > > > information
> > > > >>> from gossip yet, it should not be a problem. (As far as I know
> GPFS
> > > > does
> > > > >>> this, but I have not done extensive code diving or testing to
> make
> > > > sure all
> > > > >>> edge cases are covered there)
> > > > >>>
> > > > >>> -Jeremiah
> > > > >>>
> > > > >>>> On Oct 16, 2018, at 11:56 AM, sankalp kohli <
> > kohlisank...@gmail.com
> > > >
> > > > >>> wrote:
> > > > >>>>
> > > > >>>> Will GossipingPropertyFileSnitch not be vulnerable to Gossip
> bugs
> > > > where
> > > > >>> we
> > > > >>>> lose hostId or some other fields when we restart C* for large
> > > > >>>> clusters(~1000 instances)?
>

Re: Deprecating/removing PropertyFileSnitch?

[Sankalp Kohli]

I will send the JIRAs of the bug which we thought we have fixed but it still 
exists. 

Have you done any correctness testing after doing all these tests...have you 
done the tests for 1000 instance clusters? 

It is great you have done these tests and I am hoping the gossiping snitch is 
good. Also was there any Gossip bug fixed post 3.0? May be I am seeing the bug 
which is fixed. 

> On Oct 22, 2018, at 7:09 PM, J. D. Jordan  wrote:
> 
> Do you have a specific gossip bug that you have seen recently which caused a 
> problem that would make this happen?  Do you have a specific JIRA in mind?  
> “We can’t remove this because what if there is a bug” doesn’t seem like a 
> good enough reason to me. If that was a reason we would never make any 
> changes to anything.
> I think many people have seen PFS actually cause real problems, where with 
> GPFS the issue being talked about is predicated on some theoretical gossip 
> bug happening.
> In the past year at DataStax we have done a lot of testing on 3.0 and 3.11 
> around adding nodes, adding DC’s, replacing nodes, replacing racks, and 
> replacing DC’s, all while using GPFS, and as far as I know we have not seen 
> any “lost” rack/DC information during such testing.
> 
> -Jeremiah
> 
>> On Oct 22, 2018, at 5:46 PM, sankalp kohli  wrote:
>> 
>> We will have similar issues with Gossip but this will create more issues as
>> more things will be relied on Gossip.
>> 
>> I agree PFS should be removed but I dont see how it can be with issues like
>> these or someone proves that it wont cause any issues.
>> 
>> On Mon, Oct 22, 2018 at 2:21 PM Paulo Motta 
>> wrote:
>> 
>>> I can understand keeping PFS for historical/compatibility reasons, but if
>>> gossip is broken I think you will have similar ring view problems during
>>> replace/bootstrap that would still occur with the use of PFS (such as
>>> missing tokens, since those are propagated via gossip), so that doesn't
>>> seem like a strong reason to keep it around.
>>> 
>>> With PFS it's pretty easy to shoot yourself in the foot if you're not
>>> careful enough to have identical files across nodes and updating it when
>>> adding nodes/dcs, so it's seems to be less foolproof than other snitches.
>>> While the rejection of verbs to invalid replicas on trunk could address
>>> concerns raised by Jeremy, this would only happen after the new node joins
>>> the ring, so you would need to re-bootstrap the node and lose all the work
>>> done in the original bootstrap.
>>> 
>>> Perhaps one good reason to use PFS is the ability to easily package it
>>> across multiple nodes, as pointed out by Sean Durity on CASSANDRA-10745
>>> (which is also it's Achilles' heel). To keep this ability, we could make
>>> GPFS compatible with the cassandra-topology.properties file, but reading
>>> only the dc/rack info about the local node.
>>> 
>>> Em seg, 22 de out de 2018 às 16:58, sankalp kohli 
>>> escreveu:
>>> 
>>>> Yes it will happen. I am worried that same way DC or rack info can go
>>>> missing.
>>>> 
>>>> On Mon, Oct 22, 2018 at 12:52 PM Paulo Motta 
>>>> wrote:
>>>> 
>>>>>> the new host won’t learn about the host whose status is missing and
>>> the
>>>>> view of this host will be wrong.
>>>>> 
>>>>> Won't this happen even with PropertyFileSnitch as the token(s) for this
>>>>> host will be missing from gossip/system.peers?
>>>>> 
>>>>> Em sáb, 20 de out de 2018 às 00:34, Sankalp Kohli <
>>>> kohlisank...@gmail.com>
>>>>> escreveu:
>>>>> 
>>>>>> Say you restarted all instances in the cluster and status for some
>>> host
>>>>>> goes missing. Now when you start a host replacement, the new host
>>> won’t
>>>>>> learn about the host whose status is missing and the view of this
>>> host
>>>>> will
>>>>>> be wrong.
>>>>>> 
>>>>>> PS: I will be happy to be proved wrong as I can also start using
>>> Gossip
>>>>>> snitch :)
>>>>>> 
>>>>>>> On Oct 19, 2018, at 2:41 PM, Jeremy Hanna <
>>>> jeremy.hanna1...@gmail.com>
>>>>>> wrote:
>>>>>>> 
>>>>>>> Do you mean to say that during host replacement there may be a time
>>>>> when
>>>>>> th

Re: Deprecating/removing PropertyFileSnitch?

[sankalp kohli]

Here are some of the JIRAs which are fixed but actually did not fix the
issue. We have tried fixing this by several patches. May be it will be
fixed when Gossip is rewritten(CASSANDRA-12345). I should find or create a
new JIRA as this issue still exists.
https://issues.apache.org/jira/browse/CASSANDRA-10366
https://issues.apache.org/jira/browse/CASSANDRA-10089 (related to it)

Also the quote you are using was written as a follow on email. I have
already said what the bug I was referring to.

"Say you restarted all instances in the cluster and status for some host
goes missing. Now when you start a host replacement, the new host won’t
learn about the host whose status is missing and the view of this host will
be wrong."

   - CASSANDRA-10366


On Mon, Oct 22, 2018 at 7:22 PM Sankalp Kohli 
wrote:

> I will send the JIRAs of the bug which we thought we have fixed but it
> still exists.
>
> Have you done any correctness testing after doing all these tests...have
> you done the tests for 1000 instance clusters?
>
> It is great you have done these tests and I am hoping the gossiping snitch
> is good. Also was there any Gossip bug fixed post 3.0? May be I am seeing
> the bug which is fixed.
>
> > On Oct 22, 2018, at 7:09 PM, J. D. Jordan 
> wrote:
> >
> > Do you have a specific gossip bug that you have seen recently which
> caused a problem that would make this happen?  Do you have a specific JIRA
> in mind?  “We can’t remove this because what if there is a bug” doesn’t
> seem like a good enough reason to me. If that was a reason we would never
> make any changes to anything.
> > I think many people have seen PFS actually cause real problems, where
> with GPFS the issue being talked about is predicated on some theoretical
> gossip bug happening.
> > In the past year at DataStax we have done a lot of testing on 3.0 and
> 3.11 around adding nodes, adding DC’s, replacing nodes, replacing racks,
> and replacing DC’s, all while using GPFS, and as far as I know we have not
> seen any “lost” rack/DC information during such testing.
> >
> > -Jeremiah
> >
> >> On Oct 22, 2018, at 5:46 PM, sankalp kohli 
> wrote:
> >>
> >> We will have similar issues with Gossip but this will create more
> issues as
> >> more things will be relied on Gossip.
> >>
> >> I agree PFS should be removed but I dont see how it can be with issues
> like
> >> these or someone proves that it wont cause any issues.
> >>
> >> On Mon, Oct 22, 2018 at 2:21 PM Paulo Motta 
> >> wrote:
> >>
> >>> I can understand keeping PFS for historical/compatibility reasons, but
> if
> >>> gossip is broken I think you will have similar ring view problems
> during
> >>> replace/bootstrap that would still occur with the use of PFS (such as
> >>> missing tokens, since those are propagated via gossip), so that doesn't
> >>> seem like a strong reason to keep it around.
> >>>
> >>> With PFS it's pretty easy to shoot yourself in the foot if you're not
> >>> careful enough to have identical files across nodes and updating it
> when
> >>> adding nodes/dcs, so it's seems to be less foolproof than other
> snitches.
> >>> While the rejection of verbs to invalid replicas on trunk could address
> >>> concerns raised by Jeremy, this would only happen after the new node
> joins
> >>> the ring, so you would need to re-bootstrap the node and lose all the
> work
> >>> done in the original bootstrap.
> >>>
> >>> Perhaps one good reason to use PFS is the ability to easily package it
> >>> across multiple nodes, as pointed out by Sean Durity on CASSANDRA-10745
> >>> (which is also it's Achilles' heel). To keep this ability, we could
> make
> >>> GPFS compatible with the cassandra-topology.properties file, but
> reading
> >>> only the dc/rack info about the local node.
> >>>
> >>> Em seg, 22 de out de 2018 às 16:58, sankalp kohli <
> kohlisank...@gmail.com>
> >>> escreveu:
> >>>
> >>>> Yes it will happen. I am worried that same way DC or rack info can go
> >>>> missing.
> >>>>
> >>>> On Mon, Oct 22, 2018 at 12:52 PM Paulo Motta <
> pauloricard...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>>> the new host won’t learn about the host whose status is missing and
> >>> the
> >>>>> view of this host will be wrong.
> >>>>>
> >>>>> Won't this happen

Re: Deprecating/removing PropertyFileSnitch?

[Sankalp Kohli]

+1 to fallback and like I said before removing PFS is a good idea as long it is 
safe 

> On Oct 22, 2018, at 7:41 PM, Jeff Jirsa  wrote:
> 
> On Mon, Oct 22, 2018 at 7:09 PM J. D. Jordan 
> wrote:
> 
>> Do you have a specific gossip bug that you have seen recently which caused
>> a problem that would make this happen?  Do you have a specific JIRA in mind?
> 
> 
> Sankalp linked a few others, but also
> https://issues.apache.org/jira/browse/CASSANDRA-13700
> 
> 
>>  “We can’t remove this because what if there is a bug” doesn’t seem like
>> a good enough reason to me. If that was a reason we would never make any
>> changes to anything.
>> 
> 
> How about "we know that certain fields that are gossiped go missing even
> after all of the known races are fixed, so removing an existing
> low-maintenance feature and forcing users to rely on gossip for topology
> may be worth some discussion".
> 
> 
>> I think many people have seen PFS actually cause real problems, where with
>> GPFS the issue being talked about is predicated on some theoretical gossip
>> bug happening.
>> 
> 
> How many of those were actually caused by incorrect fallback from GPFS to
> PFS, rather than PFS itself?
> 
> 
>> In the past year at DataStax we have done a lot of testing on 3.0 and 3.11
>> around adding nodes, adding DC’s, replacing nodes, replacing racks, and
>> replacing DC’s, all while using GPFS, and as far as I know we have not seen
>> any “lost” rack/DC information during such testing.
>> 
> 
> I've also run very large GPFS clusters in the past without much gossip
> pain, and I'm in the "we should deprecate PFS" camp, but it is also true
> that PFS is low maintenance and mostly works. Perhaps the first step is
> breaking the GPFS->PFS fallback that people don't know about, maybe that'll
> help?

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Deprecating/removing PropertyFileSnitch?

[Sankalp Kohli]

No worries...I mentioned the issue not the JIRA number 

> On Oct 22, 2018, at 8:01 PM, Jeremiah D Jordan  wrote:
> 
> Sorry, maybe my spam filter got them or something, but I have never seen a 
> JIRA number mentioned in the thread before this one.  Just looked back 
> through again to make sure, and this is the first email I have with one.
> 
> -Jeremiah
> 
>> On Oct 22, 2018, at 9:37 PM, sankalp kohli  wrote:
>> 
>> Here are some of the JIRAs which are fixed but actually did not fix the
>> issue. We have tried fixing this by several patches. May be it will be
>> fixed when Gossip is rewritten(CASSANDRA-12345). I should find or create a
>> new JIRA as this issue still exists.
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_CASSANDRA-2D10366&d=DwIFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g&m=lI3KEen0YYUim6t3VWsvITHUZfFX8oYaczP_t3kk21o&s=W_HfejhgW1gmZ06L0CXOnp_EgBQ1oI5MLMoyz0OrvFw&e=
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_CASSANDRA-2D10089&d=DwIFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g&m=lI3KEen0YYUim6t3VWsvITHUZfFX8oYaczP_t3kk21o&s=qXzh1nq2yE27J8SvwYoRf9HPQE83m07cKdKVHXyOyAE&e=
>>  (related to it)
>> 
>> Also the quote you are using was written as a follow on email. I have
>> already said what the bug I was referring to.
>> 
>> "Say you restarted all instances in the cluster and status for some host
>> goes missing. Now when you start a host replacement, the new host won’t
>> learn about the host whose status is missing and the view of this host will
>> be wrong."
>> 
>>  - CASSANDRA-10366
>> 
>> 
>> On Mon, Oct 22, 2018 at 7:22 PM Sankalp Kohli 
>> wrote:
>> 
>>> I will send the JIRAs of the bug which we thought we have fixed but it
>>> still exists.
>>> 
>>> Have you done any correctness testing after doing all these tests...have
>>> you done the tests for 1000 instance clusters?
>>> 
>>> It is great you have done these tests and I am hoping the gossiping snitch
>>> is good. Also was there any Gossip bug fixed post 3.0? May be I am seeing
>>> the bug which is fixed.
>>> 
>>>> On Oct 22, 2018, at 7:09 PM, J. D. Jordan 
>>> wrote:
>>>> 
>>>> Do you have a specific gossip bug that you have seen recently which
>>> caused a problem that would make this happen?  Do you have a specific JIRA
>>> in mind?  “We can’t remove this because what if there is a bug” doesn’t
>>> seem like a good enough reason to me. If that was a reason we would never
>>> make any changes to anything.
>>>> I think many people have seen PFS actually cause real problems, where
>>> with GPFS the issue being talked about is predicated on some theoretical
>>> gossip bug happening.
>>>> In the past year at DataStax we have done a lot of testing on 3.0 and
>>> 3.11 around adding nodes, adding DC’s, replacing nodes, replacing racks,
>>> and replacing DC’s, all while using GPFS, and as far as I know we have not
>>> seen any “lost” rack/DC information during such testing.
>>>> 
>>>> -Jeremiah
>>>> 
>>>>> On Oct 22, 2018, at 5:46 PM, sankalp kohli 
>>> wrote:
>>>>> 
>>>>> We will have similar issues with Gossip but this will create more
>>> issues as
>>>>> more things will be relied on Gossip.
>>>>> 
>>>>> I agree PFS should be removed but I dont see how it can be with issues
>>> like
>>>>> these or someone proves that it wont cause any issues.
>>>>> 
>>>>> On Mon, Oct 22, 2018 at 2:21 PM Paulo Motta 
>>>>> wrote:
>>>>> 
>>>>>> I can understand keeping PFS for historical/compatibility reasons, but
>>> if
>>>>>> gossip is broken I think you will have similar ring view problems
>>> during
>>>>>> replace/bootstrap that would still occur with the use of PFS (such as
>>>>>> missing tokens, since those are propagated via gossip), so that doesn't
>>>>>> seem like a strong reason to keep it around.
>>>>>> 
>>>>>> With PFS it's pretty easy to shoot yourself in the foot if you're not
>>>>>> careful enough to have identical files across nodes and updating it
>>> when
>>>>>> adding nodes/dcs, so it's seems to be less foo

Re: 4.0 Testing Signup

[sankalp kohli]

This is good start and we should use this approach each component. Once we
have volunteers for each area, it will be important to also publish a
confluence page per component by the volunteer so we can know/discuss how
it was tested.

On Wed, Nov 7, 2018 at 12:14 PM Joseph Lynch  wrote:

> Following up on Jon's call
> <
> https://lists.apache.org/thread.html/17e57d1666393d961a15502a648a1174a1b145a4bf0a8e07fd8bb760@%3Cdev.cassandra.apache.org%3E
> >
> for QA, I put together the start of a confluence page
> for
> people to list out important components that they think should be tested
> before 4.0 releases and hopefully committers and contributors can signup
> and present their progress to the community. I've certainly missed a ton of
> components that need testing but I figured that it may be good to get the
> conversation started and moving forward.
>
> What do people think? Is there a more effective way to list these out or if
> people like this maybe folks can start contributing sections and
> volunteering to shepherd or test them?
>
> Let me know,
> -Joey
>

Re: Jepsen testing

[sankalp kohli]

Should we use confluence page to sign them up for this testing?

https://cwiki.apache.org/confluence/display/CASSANDRA/4.0+QA+Signup

On Thu, Nov 8, 2018 at 2:07 PM Nate McCall  wrote:

> [- cassandra-users]
> Hi Yuji,
> Thanks so much for working on this! Any fault injection testing is
> certainly worth the effort.
>
> Thanks,
> -Nate
> On Thu, Nov 8, 2018 at 1:36 PM Yuji Ito  wrote:
> >
> > Hi,
> >
> > We are working on Jepsen testing for Cassandra.
> > https://github.com/scalar-labs/jepsen/tree/cassandra/cassandra
> >
> > As you may know, Jepsen is a framework for distributed systems
> verification.
> > It can inject network failure and so on and check data consistency.
> > https://github.com/jepsen-io/jepsen
> >
> > Our tests are based on riptano's great work.
> > https://github.com/riptano/jepsen/tree/cassandra/cassandra
> >
> > I refined it for the latest Jepsen and removed some tests.
> > Next, I'll fix clock-drift tests.
> >
> > I would like to get your feedback.
> >
> > Thanks,
> > Yuji Ito
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: Proposing an Apache Cassandra Management process

[Sankalp Kohli]

Hi Mick,
  Any feedback? 
Also pinging this thread after a week for others to give feedback 

> On Nov 6, 2018, at 1:40 AM, Dinesh Joshi  
> wrote:
> 
> Hi all,
> 
> Joey, Vinay & I have fleshed out the Management process proposal as the very 
> first CIP document (with Jason’s inputs). It is available on the cwiki - 
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652224
> 
> Please comment on it and provide us any input that you may have. We want to 
> ideally time-box the period to 2 weeks so we avoid waiting indefinitely.
> 
> Thanks,
> 
> Dinesh
> 
>> On Oct 22, 2018, at 7:30 AM, "dinesh.jo...@yahoo.com.INVALID" 
>>  wrote:
>> 
>> Thanks for starting this, Mick. I will flesh it out.
>> Dinesh 
>> 
>>   On Sunday, October 21, 2018, 1:52:10 AM PDT, Mick Semb Wever 
>>  wrote:  
>> 
>> 
>>> But I'll try to put together a strawman proposal for the doc(s) over the 
>>> weekend. 
>> 
>> 
>> I've thrown something quickly together here:
>> - https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652201
>> - 
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CIP-1%3A+Proposing+an+Apache+Cassandra+Management+process
>> 
>> The former is a blatant rip-off from the Kafka and Spark design proposal 
>> pages that Dinesh previously mentioned. I'd hoped to do more of an analysis 
>> of the existing C* habits and precedence on design proposals (implicit in 
>> jira tickets), but in lei of that this is a strawman to start the discussion.
>> 
>> The latter still needs to be fleshed out. Dinesh, can you do this? I can add 
>> a subpage/section that describes the alternative/consuming third-party tools 
>> out there.
>> 
>> regards,
>> Mick
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Request to review feature-freeze proposed tickets

[Sankalp Kohli]

We already should be taking correctness and perf changes and I am +1 on taking 
operational tickets. Agree with Josh that the only exception will be if it 
causes disruption in testing.

I think as a project we should be more open to operational issues as having a 
fork is not ideal. 

Regarding time it is taking to review things, I think we should not start doing 
big features post 4.0 but instead review all possible JIRAs first. Once we are 
out of this debt, we should define a  process so that we don’t end up in this 
state. I think this item deserves a separate thread which we can start closer 
to 4.0 being cut. 

> On Nov 23, 2018, at 12:17 AM, Joshua McKenzie  wrote:
> 
> Strong +1 on prioritizing community engagement 1st and caution second,
> though still prioritizing it. I think the right metric for us to calibrate
> around is that "disrupt in-flight testing cycles", i.e. if changes cause
> significant rework for people that have already begun testing 4.0, probably
> ok to review and get it in but target 4.0.x.
> 
> On Thu, Nov 22, 2018 at 12:00 PM Benedict Elliott Smith 
> wrote:
> 
>>> I assume it's obvious to everyone that this should be taken on a
>>> case-by-case basis. There's at least 2 that were in that list (one of
>> which
>>> Marcus bumped off PA) that are potentially big and hairy changes that
>> would
>>> disrupt in-flight testing cycles.
>> 
>> Agreed.  I’d prefer we aim to be as permissive as practical, though.
>> 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 
>> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Inter-node messaging latency

[sankalp kohli]

Inter-node messaging is rewritten using Netty in 4.0. It will be better to
test it using that as potential changes will mostly land on top of that.

On Mon, Nov 26, 2018 at 7:39 AM Yuji Ito  wrote:

> Hi,
>
> I'm investigating LWT performance with C* 3.11.3.
> It looks that the performance is bounded by messaging latency when many
> requests are issued concurrently.
>
> According to the source code, the number of messaging threads per node is
> only 1 thread for incoming and 1 thread for outbound "small" message to
> another node.
>
> I guess these threads are frequently interrupted because many threads are
> executed when many requests are issued.
> Especially, I think it affects the LWT performance when many LWT requests
> which need lots of inter-node messaging are issued.
>
> I measured that latency. It took 2.5 ms in average to enqueue a message at
> a node and to receive the message at the **same** node with 96 concurrent
> LWT writes.
> Is it normal? I think it is too big latency, though a message was sent to
> the same node.
>
> Decreasing numbers of other threads like `concurrent_counter_writes`,
> `concurrent_materialized_view_writes` reduced a bit the latency.
> Can I change any other parameter to reduce the latency?
> I've tried using message coalescing, but they didn't reduce that.
>
> * Environment
> - 3 node cluster
> - Replication factor: 3
> - Node instance: AWS EC2 i3.xlarge
>
> * C* configuration
> - Apache Cassandra 3.11.3
> - commitlog_sync: batch
> - concurrent_reads: 32 (default)
> - concurrent_writes: 32 (default)
>
> Thanks,
> Yuji
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: JIRA Workflow Proposals

[Sankalp Kohli]

I have following initial comments on the proposal. 
1. Creating a JIRA should have few fields mandatory like platform, version, 
etc. We want to put less burden on someone creating a ticket but I feel these 
are required for opening bugs. 

2. What is the flow when a non committer does the first pass of review? 



> On Nov 26, 2018, at 7:46 PM, Joshua McKenzie  wrote:
> 
> 1) Removal of labels: one of the strengths of the current model is
> flexibility for groupings of concepts to arise from a user-perspective
> (lhf, etc). Calcifying the label concepts into components, categories, etc.
> is a strict loss of functionality for users to express and categorize their
> concerns with the project. This feels like an over-correction to our
> current lack of discipline cleaning up one-off labels. Counter-proposal:
> 
>   1. We beef up the categories and components as proposed and migrate
>   labels to those
>   2. We remove the one-off labels that aren't adding value, considering
>   aggregating similar labels
>   3. We leave the "labels" field intact, perhaps with a bit of guidance on
>   how to effectively use it
> 
> 2) Required fields on transition: assuming these are required upon *issue
> creation*, that's placing a significant burden on a user that's seen
> something and wants to open a ticket about it, but isn't sure if it's a
> "Semantic Failure" or a "Consistency Failure", for instance. If this is,
> instead, a field required for transition to other ticket states by the
> developer working on it, I think that makes sense.
> 
> 3) Priority Changes: to be blunt, this looks like shuffling chairs on the
> deck of the titanic on this one - in my experience, most users aren't going
> to set the priority on tickets when they open them (hence default == major
> and most tickets are opened as major), so this is something that will be
> either a) left alone so as not to offend those for whom the priority is
> *actually* major or consistent with the default, or b) changed by the dev
> anyway and the added signal from a new "High vs. Urgent" distinction and
> communication of developer intent to engage with a ticket is something
> that'll be lost on most users that are just reporting something. I don't
> have a meaningful counter-proposal at this point as the current problem is
> more about UX on this field than the signal / noise on the field itself IMO.
> 
> A meta question about the "What and Why" of what we're trying to accomplish
> here: it sounds like what you're looking at is:
> 
>   1. to capture more information
>   2. simplify the data entry
>   3. nudge people towards more complete and accurate data entry
>   4. our ability as a project to measure release quality over time and
>   identify when Cassandra is ready for (or due a) release.
> 
> The proposal in its current form makes certain strong inroads in all of
> those 4 metrics, but I think the major thing missing is the UX of what
> we're thinking about changing:
> 
>   1. Making it easy for / reduce friction for users to report bugs and
>   requests into the project JIRA (easy to do things right, hard to do things
>   wrong) (current proposal is a +1 on "do things right", though I'd argue
>   against it being *easy*)
>   2. Asking from the users what they can provide about their experience
>   and issues and no more
> 
> Philosophically, are we trying to make it easier for people that are paid
> FT to work on C*, are we trying to make things easier for *users* of C*,
> both, neither? Who are we targeting here w/these project changes and what
> of their issues / problems are we trying to improve?
> 
> And lastly: the TLC and management of the JIRA aspects of this project have
> *definitely* languished (not pointing any fingers here, I'm *at least* as
> guilty as anyone on this :) ) - so a big thanks to you and whomever you've
> collaborate with in getting this conversation started!
> 
> On Mon, Nov 26, 2018 at 8:39 AM Benedict Elliott Smith 
> wrote:
> 
>> We’ve concluded our efforts to produce a proposal for changes to the JIRA
>> workflow, and they can be found here:
>> https://cwiki.apache.org/confluence/display/CASSANDRA/JIRA+Workflow+Proposals
>> <
>> https://cwiki.apache.org/confluence/display/CASSANDRA/JIRA+Workflow+Proposals
>>> 
>> 
>> I hope there will be broad consensus, but I’m sure it won’t be plain
>> sailing.  It would be great to get a discussion going around the proposal,
>> so please take some time to read and respond if you think you’ll have a
>> strong opinion on this topic.
>> 
>> There remains an undecided question in our initial proposal, that is
>> highlighted in the wiki.  Specifically, there was no seemingly objective
>> winner for the suggested changes to Component (though I have a preference,
>> that I will express in the ensuing discussion).
>> 
>> Other contentious issues may be:
>> - The removal of ‘labels’ - which is very noisy, the vast majority of
>> which provide no value, and we expect can be superseded by other sug

Re: Proposing an Apache Cassandra Management process

[sankalp kohli]

If no one has more feedback, we should create JIRAs for all the subtasks
discussed in the proposal. I can see JIRA for Bulk command but not others.

On Mon, Nov 19, 2018 at 2:12 AM dinesh.jo...@yahoo.com.INVALID
 wrote:

> Thanks, Mick & Stefan for your inputs. What should we do as next steps to
> move this proposal forward?
> Dinesh
>
> On Sunday, November 18, 2018, 11:57:54 AM PST, Stefan Podkowinski <
> s...@apache.org> wrote:
>
>  My goal for a side car would be to enable more people to contribute to
> the project, by making it more accessible for anyone who’s not familiar
> with the Cassandra code base, or not familiar with Java development in
> general. Although most of the functionality described in the proposal
> sounds useful to have, I’d already be happy to have a solid REST API for
> the existing nodetool and JMX functionality. If an official side car,
> installed separately on each node, would provide that, I’m sure we’d see
> lots of new tools created by the community (web UIs, cli tools, ..)
> based on that. This would also be a good foundation for other existing
> tool to converge upon, e.g. by calling the REST APIs for repair
> scheduling and progress tracking instead of JMX, or by continually
> integrating and sharing useful helper calls. This would also give
> Cassandra devs more leeway to replace some of the existing tooling
> related code in Cassandra, e.g. by migrating to virtual tables, while at
> the same time keep providing a stable API through the side car.
>
> What I’d also like to point out here is that implementing such a project
> as an *official* side car, also implies to me having the same standards
> when it comes to release quality. I’d also really prefer having feature
> sets matching between Cassandra and the side car, e.g. authentication
> and SSL should also be supported in the side car from the beginning,
> ideally without any additional configuration.
>
>
> On 06.11.18 10:40, Dinesh Joshi wrote:
> > Hi all,
> >
> > Joey, Vinay & I have fleshed out the Management process proposal as the
> very first CIP document (with Jason’s inputs). It is available on the cwiki
> -
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652224
> >
> > Please comment on it and provide us any input that you may have. We want
> to ideally time-box the period to 2 weeks so we avoid waiting indefinitely.
> >
> > Thanks,
> >
> > Dinesh
> >
> >> On Oct 22, 2018, at 7:30 AM, "dinesh.jo...@yahoo.com.INVALID" <
> dinesh.jo...@yahoo.com.INVALID> wrote:
> >>
> >> Thanks for starting this, Mick. I will flesh it out.
> >> Dinesh
> >>
> >>On Sunday, October 21, 2018, 1:52:10 AM PDT, Mick Semb Wever <
> m...@apache.org> wrote:
> >>
> >>
> >>> But I'll try to put together a strawman proposal for the doc(s) over
> the
> >>> weekend.
> >>
> >> I've thrown something quickly together here:
> >> -
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652201
> >> -
> https://cwiki.apache.org/confluence/display/CASSANDRA/CIP-1%3A+Proposing+an+Apache+Cassandra+Management+process
> >>
> >> The former is a blatant rip-off from the Kafka and Spark design
> proposal pages that Dinesh previously mentioned. I'd hoped to do more of an
> analysis of the existing C* habits and precedence on design proposals
> (implicit in jira tickets), but in lei of that this is a strawman to start
> the discussion.
> >>
> >> The latter still needs to be fleshed out. Dinesh, can you do this? I
> can add a subpage/section that describes the alternative/consuming
> third-party tools out there.
> >>
> >> regards,
> >> Mick
> >>
> >> -
> >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: [VOTE] Change Jira Workflow

[sankalp kohli]

+1

On Tue, Dec 18, 2018 at 9:16 PM Ariel Weisberg  wrote:

> +1
>
> On Mon, Dec 17, 2018, at 10:19 AM, Benedict Elliott Smith wrote:
> > I propose these changes
> > <
> https://cwiki.apache.org/confluence/display/CASSANDRA/JIRA+Workflow+Proposals>*
>
> > to the Jira Workflow for the project.  The vote will be open for 72
> > hours**.
> >
> > I am, of course, +1.
> >
> > * With the addendum of the mailing list discussion
> > <
> https://lists.apache.org/thread.html/e4668093169aa4ef52f2bea779333f04a0afde8640c9a79a8c86ee74@%3Cdev.cassandra.apache.org%3E>;
>
> > in case of any conflict arising from a mistake on my part in the wiki,
> > the consensus reached by polling the mailing list will take precedence.
> > ** I won’t be around to close the vote, as I will be on vacation.
> > Everyone is welcome to ignore the result until I get back in a couple of
> > weeks, or if anybody is eager feel free to close the vote and take some
> > steps towards implementation.
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: Warn about SASI usage and allow to disable them

[sankalp kohli]

If we want to put a warning, we should list in a doc all the open issues it
has along with explanation of how it can impact. We have a few in the first
email of this thread but we should put it in a doc for people to know what
are the issues and if they want to take that risk.



On Wed, Jan 16, 2019 at 3:14 PM Brandon Williams  wrote:

> Which, if I'm not mistaken, is the goal here?
>
> On Wed, Jan 16, 2019 at 5:12 PM Jeff Jirsa  wrote:
>
> > The cost is in how many users you scare away
> >
> > --
> > Jeff Jirsa
> >
> >
> > > On Jan 16, 2019, at 2:34 PM, Brandon Williams 
> wrote:
> > >
> > > Also it costs us nothing to add it.
> > >
> > >> On Wed, Jan 16, 2019 at 4:29 PM Jonathan Haddad 
> > wrote:
> > >>
> > >> I'm +1 on the warning for two reasons.
> > >>
> > >>> A cqlsh warning only applies to those that create the sasi via cqlsh.
> > >>
> > >> 1. When people are creating their schemas in development, this is
> > usually
> > >> the first step.  You use the REPL to figure out what you need, then
> you
> > >> copy your schema somewhere else.  The warning here should prevent a
> lot
> > of
> > >> folks from making a serious mistake.
> > >>
> > >> 2. It's consistent with how we warn when people try to use
> materialized
> > >> views.
> > >>
> > >>
> > >>
> > >>
> > >>> On Wed, Jan 16, 2019 at 2:15 PM Mick Semb Wever 
> > wrote:
> > >>>
> > >>> Regarding the warning, we might add it at least in 3.11, since for
> that
> > >>> version the property to enable SASI is going to be present but not
> > >> disabled
> > >>> by default. WDYT?
> > >>>
> > >>>
> > >>> I'm  -0 on this.
> > >>>
> > >>> A single line warning in the logs on the sasi creation won't be
> noticed
> > >> by
> > >>> many users.
> > >>> A cqlsh warning only applies to those that create the sasi via cqlsh.
> > >>> And we're not talking about patching client drivers to generate a
> > warning
> > >>> there.
> > >>>
> > >>> So I'd be happy with a yaml comment on the config flag explaining
> that
> > >>> it's a beta feature and that users should check open tickets and
> > >> understand
> > >>> current limitations on sasi before using them.
> > >>>
> > >>> regards,
> > >>> Mick
> > >>>
> > >>> -
> > >>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > >>> For additional commands, e-mail: dev-h...@cassandra.apache.org
> > >>>
> > >>>
> > >>
> > >> --
> > >> Jon Haddad
> > >> http://www.rustyrazorblade.com
> > >> twitter: rustyrazorblade
> > >>
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >
>

Re: Warn about SASI usage and allow to disable them

[Sankalp Kohli]

Can we have a page or a JIRA label which users can see to know why it is 
experimental. Putting a warning without telling why is not good. But since 
warning is better than nothing, I am -0 on warn

> On Jan 26, 2019, at 9:37 AM, Andrés de la Peña  
> wrote:
> 
> I agree with Paulo's proposal. I think it will give us a very desirable
> homogeneity in how we deal with experimental features.
> 
> I'm +1 to warning, config property, and experimental features (SASI and MV)
> disabled by default in trunk.
> 
> These are the explicit votes for now, if I'm counting right:
> 
> - CQL native protocol warning on create SASI index: three +1s, one +0 and
> two -0s
> - Config property to disable new SASI creation: ten +1s
> - New SASI creation disabled by default in trunk: nine +1s and one -0
> - New MV creation disabled by default in trunk: three +1s
> 
> If there are no objections, I'll update the patch by the end of next week.
> 
>> On Mon, 21 Jan 2019 at 19:26, Paulo Motta  wrote:
>> 
>> +1 to enable_sasi_indexes flag
>> +1 to disabling experimental features by default on 4.0 (SASI and MVs,
>> transient replication already disabled)
>> 
>> Regarding the warning on creation of SASI indexes, I think that's a
>> user-level warning complimentary to the flag, which is targeted to admins,
>> so +1. If people are bothered by this, we could add another flag to disable
>> warnings on experimental features, which would be applied to both this and
>> MV creation warning (and any other future experimental feature).
>> 
>> I think the warning should be "SASI indexes are experimental and are not
>> recommended for production use.", similar to the MV warning added on
>> CASSANDRA-13959.
>> 
>> We should open a doc ticket to list limitations of experimental features
>> (MVs, SASI, transient replication), but this should probably be out of the
>> scope of CASSANDRA-14866. Once we have this doc, we can maybe amend the
>> warning to include a link to the doc.
>> 
>> Now that the number of experimental feature flags is growing we should
>> perhaps unify all flags in a "experimental features" section on
>> cassandra.yaml to allow easily locating them - and a pointer to the
>> limitations doc once we have it.
>> 
>> Em qua, 16 de jan de 2019 às 20:18, sankalp kohli 
>> escreveu:
>> 
>>> If we want to put a warning, we should list in a doc all the open issues
>> it
>>> has along with explanation of how it can impact. We have a few in the
>> first
>>> email of this thread but we should put it in a doc for people to know
>> what
>>> are the issues and if they want to take that risk.
>>> 
>>> 
>>> 
>>> On Wed, Jan 16, 2019 at 3:14 PM Brandon Williams 
>> wrote:
>>> 
>>>> Which, if I'm not mistaken, is the goal here?
>>>> 
>>>>> On Wed, Jan 16, 2019 at 5:12 PM Jeff Jirsa  wrote:
>>>>> 
>>>>> The cost is in how many users you scare away
>>>>> 
>>>>> --
>>>>> Jeff Jirsa
>>>>> 
>>>>> 
>>>>>> On Jan 16, 2019, at 2:34 PM, Brandon Williams 
>>>> wrote:
>>>>>> 
>>>>>> Also it costs us nothing to add it.
>>>>>> 
>>>>>>> On Wed, Jan 16, 2019 at 4:29 PM Jonathan Haddad <
>> j...@jonhaddad.com>
>>>>> wrote:
>>>>>>> 
>>>>>>> I'm +1 on the warning for two reasons.
>>>>>>> 
>>>>>>>> A cqlsh warning only applies to those that create the sasi via
>>> cqlsh.
>>>>>>> 
>>>>>>> 1. When people are creating their schemas in development, this is
>>>>> usually
>>>>>>> the first step.  You use the REPL to figure out what you need,
>> then
>>>> you
>>>>>>> copy your schema somewhere else.  The warning here should prevent
>> a
>>>> lot
>>>>> of
>>>>>>> folks from making a serious mistake.
>>>>>>> 
>>>>>>> 2. It's consistent with how we warn when people try to use
>>>> materialized
>>>>>>> views.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> On Wed, Jan 16, 2019 at 2:15 PM Mick Semb Wever 
>>>>> wrote:
>>>>>>>> 
>&g

Re: Stabilising Internode Messaging in 4.0

[Sankalp Kohli]

I think we should wait for testing doc on confluence to come up and discuss 
what all needs to be added there to gain confidence. 

If the work is more to split the patch into smaller parts and delays 4.0 even 
more, can we use time in adding more test coverage, documentation and design 
docs to this component?  Will that be a middle ground here ? 

Examples of large changes not going well is due to lack of testing, 
documentation and design docs not because they were big. Being big adds to the 
complexity and increased the total bug count but small changes can be equally 
worse in terms of impact. 


> On Apr 10, 2019, at 8:53 AM, Jordan West  wrote:
> 
> There is a lot of discuss here so I’ll try to keep my opinions brief:
> 
> 1. The bug fixes are a requirement in order to have a stable 4.0. Whether
> they come from this patch or the original I have less of an opinion. I do
> think its important to minimize code changes at this time in the
> development/freeze cycle — including large refactors which add risk despite
> how they are being discussed here. From that perspective, I would prefer to
> see more targeted fixes but since we don’t have them and we have this patch
> the decision is different.
> 
> 2. We should not be measuring complexity in LoC with the exception that all
> 20k lines *do need to be review* (not just the important parts and because
> code refactoring tools have bugs too) and more lines take more time.
> Otherwise, its a poor metric for how long this will take to review.
> Further, it seems odd that the authors are projecting how long it will take
> to review — this should be the charge of the reviewers and I would like to
> hear from them on this. Its clear this a complex patch — as risky as
> something like 8099 (and the original rewrite by Jason). We should treat it
> as such and not try to merge it in quickly for the sake of it, repeating
> past mistakes. The goal of reviewing the messaging service work was to do
> just that. It would be a disservice to rush in these changes now. If the
> goal is to get the patch in that should be the priority, not completing it
> “in two weeks”. Its clear several community members have pushed back on
> that and are not comfortable with the time frame.
> 
> 3. If we need to add special forks of Netty classes to the code because of
> “how we use Netty” that is a concern to me w.r.t to quality, stability, and
> maintenance. Is there a place that documents/justifies our non-traditional
> usage (I saw some in JavaDocs but found it lacking in *why* but had a lot
> of how/what was changed). Given folks in the community have decent
> relationships with the Netty team perhaps we should leverage that as well.
> Have we reached out to them?
> 
> 4. In principle, I agree with the technical improvements you mention
> (backpressure / checksumming / etc). These things should be there. Are they
> a hard requirement for 4.0? In my opinion no and Dinesh has done a good job
> of providing some reasons as to why so I won’t go into much detail here. In
> short, a bug and a missing safety mechanism are not the same thing. Its
> also important to consider how many users a change like that covers and for
> what risk — we found a bug in 13304 late in review, had it slipped through
> it would have subjected users to silent corruption they thought couldn’t
> occur anymore because we included the feature in a prod Cassandra release.
> 
> 5. The patch could seriously benefit from some commit hygiene that would
> make it easier for folks to review. Had this been done not only would
> review be easier but also the piecemeal breakup of features/fixes could
> have been done more easily. I don’t buy the premise that this wasn’t
> possible. If we had to add the feature/fix later it would have been
> possible. I’m sure there was a smart way we could have organized it, if it
> was a priority.
> 
> 6. Have any upgrade tests been done/added? I would also like to see some
> performance benchmarks before merging so that the patch is in a similar
> state as 14503 in terms of performance testing.
> 
> I’m sure I have more thoughts but these seem like the important ones for
> now.
> 
> Jordan
> 
> On Wed, Apr 10, 2019 at 8:21 AM Dinesh Joshi 
> wrote:
> 
>> Here's are my 2¢.
>> 
>> I think the general direction of this work is valuable but I have a few
>> concerns I’d like to address. More inline.
>> 
>>> On Apr 4, 2019, at 1:13 PM, Aleksey Yeschenko 
>> wrote:
>>> 
>>> I would like to propose CASSANDRA-15066 [1] - an important set of bug
>> fixes
>>> and stability improvements to internode messaging code that Benedict, I,
>>> and others have been working on for the past couple of months.
>>> 
>>> First, some context.   This work started off as a review of
>> CASSANDRA-14503
>>> (Internode connection management is race-prone [2]), CASSANDRA-13630
>>> (Support large internode messages with netty) [3], and a pre-4.0
>>> confirmatory review of such a major new feature.
>>> 
>>> However, as w

Re: "4.0: TBD" -> "4.0: Est. Q4 2019"?

[sankalp kohli]

Hi,
Is there a page where it is written what is expected from an alpha,
beta, rc and a 4.0 release?
Also how are we coming up with Q4 2019 timeline. Is this for alpha, beta,
rc or 4.0 release?

Thanks,
Sankalp

On Thu, May 23, 2019 at 11:27 AM Attila Wind  wrote:

> +1+1+1 I read a blog post was talking about last sept(?) to freeze
> features and start extensive testing. Maybe its really time to hit it! :-)
>
> Attila Wind
>
> http://www.linkedin.com/in/attilaw
> Mobile: +36 31 7811355
>
>
> On 2019. 05. 23. 19:30, ajs6f wrote:
> > +1 in the fullest degree. A date that needs to be changed is still
> enormously more attractive than no date at all.
> >
> > Adam Soroka
> >
> >> On May 23, 2019, at 12:01 PM, Sumanth Pasupuleti <
> spasupul...@netflix.com.INVALID> wrote:
> >>
> >> Having at least a ballpark target on the website will definitely help.
> +1
> >> on setting it to Q4 2019 for now.
> >>
> >> On Thu, May 23, 2019 at 8:52 AM Dinesh Joshi  wrote:
> >>
> >>> +1 on setting a date.
> >>>
> >>> Dinesh
> >>>
>  On May 23, 2019, at 11:07 AM, Michael Shuler 
> >>> wrote:
>  We've had 4.0 listed as TBD release date for a very long time.
> 
>  Yesterday, Alexander Dejanovski got a "when's 4.0 going to release?"
> >>> question after his repair talk and he suggested possibly Q4 2019. This
> >>> morning Nate McCall hinted at possibly being close by ApacheCon Las
> Vegas
> >>> in September. These got me thinking..
>  Think we can we shoot for having a 4.0 alpha/beta/rc ready to
> >>> announce/release at ApacheCon? At that time, we'll have been frozen
> for 1
> >>> year, and I think we can. We'll GA release when it's ready, but I
> think Q4
> >>> could be an realistic target.
>  With that said, I'd like to change "TBD" on the downloads page to
> "Est.
> >>> Q4 2019". We can always push or pull the estimate, but I think it's
> time to
> >>> have a goal line. This lines up with ApacheCon nicely for a preview
> release.
>  Any concerns or objections to editing the download page? Have some
> other
> >>> goal timeframe in mind?
>  --
>  Warm regards,
>  Michael
> 
>  -
>  To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>  For additional commands, e-mail: dev-h...@cassandra.apache.org
> 
> >>>
> >>> -
> >>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >>> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>>
> >>>
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
>

Re: "4.0: TBD" -> "4.0: Est. Q4 2019"?

[sankalp kohli]

Hi Jon,
   When you say 4.0 release, how do u match it with 3.0 minor
releases. The unofficial rule is to not upgrade to prod till .10 is cut.
Also due to heavy investment in testing, I dont think it will take as long
as 3.0 but want to know what is your thinking with this.

Thanks,
Sankalp

On Tue, May 28, 2019 at 9:40 AM Jon Haddad  wrote:

> Sept is a pretty long ways off.  I think the ideal case is we can announce
> 4.0 release at the summit.  I'm not putting this as a "do or die" date, and
> I don't think we need to announce it or make promises.  Sticking with "when
> it's ready" is the right approach, but we need a target, and this is imo a
> good one.
>
> This date also gives us a pretty good runway.  We could cut our first
> alphas in mid June / early July, betas in August and release in Sept.
>  There's a ton of work going into testing 4.0 already.
> Landing CASSANDRA-15066 will put us in a pretty good spot.  We've developed
> tooling at TLP that will make it a lot easier to spin up dev clusters in
> AWS as well as stress test them.  I've written about this a few times in
> the past, and I'll have a few blog posts coming up that will help show this
> in more details.
>
> There's some other quality of life things we should try to hammer out
> before then.  Updating our default JVM settings would be nice, for
> example.  Improving documentation (the data modeling section in
> particular), fixing the dynamic snitch issues [1], and some improvements to
> virtual tables like exposing the sstable metadata [2], and exposing table
> statistics [3] come to mind.  The dynamic snitch improvement will help
> performance in a big way, and the virtual tables will go a long way to
> helping with quality of life.  I showed a few folks virtual tables at the
> Accelerate conference last week and the missing table statistics was a big
> shock.  If we can get them in, it'll be a big help to operators.
>
> [1] https://issues.apache.org/jira/browse/CASSANDRA-14459
> [2] https://issues.apache.org/jira/browse/CASSANDRA-14630
> [3] https://issues.apache.org/jira/browse/CASSANDRA-14572
>
>
>
>
> On Mon, May 27, 2019 at 2:36 PM Nate McCall  wrote:
>
> > Hi Sumanth,
> > Thank you so much for taking the time to put this together.
> >
> > Cheers,
> > -Nate
> >
> > On Tue, May 28, 2019 at 3:27 AM Sumanth Pasupuleti <
> > sumanth.pasupuleti...@gmail.com> wrote:
> >
> > > I have taken an initial stab at documenting release types and exit
> > criteria
> > > in a google doc, to get us started, and to collaborate on.
> > >
> > >
> >
> https://docs.google.com/document/d/1bS6sr-HSrHFjZb0welife6Qx7u3ZDgRiAoENMLYlfz8/edit?usp=sharing
> > >
> > > Thanks,
> > > Sumanth
> > >
> > > On Thu, May 23, 2019 at 12:04 PM Dinesh Joshi 
> wrote:
> > >
> > > > Sankalp,
> > > >
> > > > Great point. This is the page created for testing.
> > > >
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/CASSANDRA/4.0+Quality%3A+Components+and+Test+Plans
> > > >
> > > > I think we need to define the various release types and the exit
> > criteria
> > > > for each type of release. Anybody want to take a stab at this or
> start
> > a
> > > > thread to discuss it?
> > > >
> > > > Thanks,
> > > >
> > > > Dinesh
> > > >
> > > >
> > > > > On May 23, 2019, at 11:57 AM, sankalp kohli <
> kohlisank...@gmail.com>
> > > > wrote:
> > > > >
> > > > > Hi,
> > > > >Is there a page where it is written what is expected from an
> > alpha,
> > > > > beta, rc and a 4.0 release?
> > > > > Also how are we coming up with Q4 2019 timeline. Is this for alpha,
> > > beta,
> > > > > rc or 4.0 release?
> > > > >
> > > > > Thanks,
> > > > > Sankalp
> > > > >
> > > > > On Thu, May 23, 2019 at 11:27 AM Attila Wind
>  > >
> > > > wrote:
> > > > >
> > > > >> +1+1+1 I read a blog post was talking about last sept(?) to freeze
> > > > >> features and start extensive testing. Maybe its really time to hit
> > it!
> > > > :-)
> > > > >>
> > > > >> Attila Wind
> > > > >>
> > > > >> http://www.linkedin.com/in/attilaw
> >

Apache Cassandra Virtual meetings

[sankalp kohli]

Hi All,
 There are projects (like k8s[1]) which do regular meetings using
video conferencing tools. We want to propose such a meeting for Apache
Cassandra once a quarter. Here are some of the initial details.

1. A two hour meeting once a quarter starting at 9am Pacific. We can later
move this to other times to make it easier for other timezones.
2. Agenda of the meeting will be due 2 days prior to the meeting. A sample
agenda for next one could cover updates on 4.0 testing, any major bugs
found and/or fixed, next steps for 4.0, etc.
3. Each agenda item will have a time duration and list of people to drive
that item.
4. We will have a moderator for each meeting which will rotate around the
community members.
5. We need to figure out which video conferencing tool to use for this.
Suggestions and donation of tools are welcome.
6. We will have meeting notes for each item discussed in the meeting.

Motivation for such a meeting
1. We currently have Slack, JIRA and emails however an agenda driven video
meeting can help facilitate alignment within the community.
2. This will give an opportunity to the community to summarize past
progress and talk about future tasks.
3. Agenda notes can serve as newsletters for the community.

Notes:
1. Does this violate any Apache rules? I could not find any rules but
someone can double check
2. Are there any other Apache projects which do something similar?

This is a proposal at this time and your feedback is greatly appreciated.
If anyone thinks this will not help then please provide a reason.

Thanks,
Sankalp
[1] https://github.com/kubernetes/community/tree/master/sig-storage

Re: Apache Cassandra Virtual meetings

[sankalp kohli]

@Dinesh/Nate: Yes we need to decide on the timing and we can always change
them as we go
@Joshua/Gary: We will publish notes on the mailing list. If we need to make
a decision, we will still need to get it voted on the ML. We should not
have a case where someone misses the boat because they could not attend one
of these. So ML is a big part of this.

Additional feedback welcome.

On Fri, Aug 9, 2019 at 6:28 AM Gary Dusbabek  wrote:

> Would publishing notes to the ML be sufficient? Apache board meetings work
> this way.
>
> Gary.
>
> On Wed, Aug 7, 2019 at 4:51 PM Nate McCall  wrote:
>
> > We can do the time mostly fair if we alternate back and forth between PST
> > morning and evening. This will at least let most folks attend every other
> > meeting.
> >
> > I agree with Josh's sentiment on the discussions. We can do it, we just
> > have to be aware of it and defer things to Jira and/or ML.
> >
> > On Thu, Aug 8, 2019 at 12:42 AM Joshua McKenzie 
> > wrote:
> >
> > > The one thing we need to keep in mind is the "If it didn't happen on a
> > > mailing list, it didn't happen <http://theapacheway.com/on-list/>"
> > > philosophy of apache projects. Shouldn't constrain us too much as the
> > > nuance is:
> > >
> > > *"Discussions and plan proposals often happen at events, in chats
> (Slack,
> > > IRC, IM, etc.) or other synchronous places. But all final decisions
> about
> > > executing on the plan, checking in the new code, or launching the
> website
> > > must be made by the community asyncrhonously on the mailing list."*
> > >
> > > So long as we keep that in mind (and maybe push it back to 8am PST
> since
> > > 9am can get pretty ugly for some of the more eastern european / asian
> > > countries), makes sense to me.
> > >
> > > On Tue, Aug 6, 2019 at 6:07 PM Dinesh Joshi  wrote:
> > >
> > > > Thanks for initiating this conversation Sankalp. On the ASF front, I
> > > think
> > > > we need to ensure that non-Pacific time participants can also
> > participate
> > > > in the discussions. So posting the notes and opening up discussions
> > after
> > > > the meet up to dev@ would be a great way of making sure everyone can
> > > > participate and gets visibility. Additionally, we should consider
> > > > scheduling this meetup in different timezones as far as logistics
> allow
> > > it.
> > > >
> > > > Dinesh
> > > >
> > > > > On Aug 6, 2019, at 2:58 PM, sankalp kohli 
> > > > wrote:
> > > > >
> > > > > Hi All,
> > > > > There are projects (like k8s[1]) which do regular meetings
> > > using
> > > > > video conferencing tools. We want to propose such a meeting for
> > Apache
> > > > > Cassandra once a quarter. Here are some of the initial details.
> > > > >
> > > > > 1. A two hour meeting once a quarter starting at 9am Pacific. We
> can
> > > > later
> > > > > move this to other times to make it easier for other timezones.
> > > > > 2. Agenda of the meeting will be due 2 days prior to the meeting. A
> > > > sample
> > > > > agenda for next one could cover updates on 4.0 testing, any major
> > bugs
> > > > > found and/or fixed, next steps for 4.0, etc.
> > > > > 3. Each agenda item will have a time duration and list of people to
> > > drive
> > > > > that item.
> > > > > 4. We will have a moderator for each meeting which will rotate
> around
> > > the
> > > > > community members.
> > > > > 5. We need to figure out which video conferencing tool to use for
> > this.
> > > > > Suggestions and donation of tools are welcome.
> > > > > 6. We will have meeting notes for each item discussed in the
> meeting.
> > > > >
> > > > > Motivation for such a meeting
> > > > > 1. We currently have Slack, JIRA and emails however an agenda
> driven
> > > > video
> > > > > meeting can help facilitate alignment within the community.
> > > > > 2. This will give an opportunity to the community to summarize past
> > > > > progress and talk about future tasks.
> > > > > 3. Agenda notes can serve as newsletters for the community.
> > > > >
> > > > > Notes:
> > > > > 1. Does this violate any Apache rules? I could not find any rules
> but
> > > > > someone can double check
> > > > > 2. Are there any other Apache projects which do something similar?
> > > > >
> > > > > This is a proposal at this time and your feedback is greatly
> > > appreciated.
> > > > > If anyone thinks this will not help then please provide a reason.
> > > > >
> > > > > Thanks,
> > > > > Sankalp
> > > > > [1]
> https://github.com/kubernetes/community/tree/master/sig-storage
> > > >
> > > >
> > > > -
> > > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > > > For additional commands, e-mail: dev-h...@cassandra.apache.org
> > > >
> > > >
> > >
> >
>

Re: Apache Cassandra Virtual meetings

[sankalp kohli]

Thanks Patrick for helping with logistics. We can certainly use your
resources unless someone objects

On Mon, Aug 12, 2019 at 10:42 AM Patrick McFadin  wrote:

> If it works for everyone, DataStax has some resources we could put to this
> effort. We do large scale conferences like this all the time and have the
> tools to pull it off. It would be a small group of people with full duplex
> audio and video with the ability for 100s of people to watch in streaming.
> As Scott mentioned, a recording could be made available after the fact for
> anyone that wanted to review and historical purposes.
>
> If we go this route, I would suggest using the ASF Cassandra slack for
> people posting questions. 1) It gets more people on slack 2) It's more or
> less a permanent, searchable record.
>
> Patrick
>
> On Sun, Aug 11, 2019 at 10:18 AM Rahul Xavier Singh <
> rahul.xavier.si...@gmail.com> wrote:
>
> > I think these meetings would be great.. if there is a specific structure.
> > We use a simple format that could help e.g.
> >
> > 1. Review long term vision/ roadmap.
> > 2. Review next release / features that are in progress.
> > 3. Discuss issues in general and make a game plan for the next quarter.
> >
> > Nothing too complicated, but at least some structure so that we are
> > timeboxed on what we are doing.
> >
> >
> >
> > On Wed, Aug 7, 2019 at 7:42 AM Joshua McKenzie 
> > wrote:
> >
> > > The one thing we need to keep in mind is the "If it didn't happen on a
> > > mailing list, it didn't happen <http://theapacheway.com/on-list/>"
> > > philosophy of apache projects. Shouldn't constrain us too much as the
> > > nuance is:
> > >
> > > *"Discussions and plan proposals often happen at events, in chats
> (Slack,
> > > IRC, IM, etc.) or other synchronous places. But all final decisions
> about
> > > executing on the plan, checking in the new code, or launching the
> website
> > > must be made by the community asyncrhonously on the mailing list."*
> > >
> > > So long as we keep that in mind (and maybe push it back to 8am PST
> since
> > > 9am can get pretty ugly for some of the more eastern european / asian
> > > countries), makes sense to me.
> > >
> > > On Tue, Aug 6, 2019 at 6:07 PM Dinesh Joshi  wrote:
> > >
> > > > Thanks for initiating this conversation Sankalp. On the ASF front, I
> > > think
> > > > we need to ensure that non-Pacific time participants can also
> > participate
> > > > in the discussions. So posting the notes and opening up discussions
> > after
> > > > the meet up to dev@ would be a great way of making sure everyone can
> > > > participate and gets visibility. Additionally, we should consider
> > > > scheduling this meetup in different timezones as far as logistics
> allow
> > > it.
> > > >
> > > > Dinesh
> > > >
> > > > > On Aug 6, 2019, at 2:58 PM, sankalp kohli 
> > > > wrote:
> > > > >
> > > > > Hi All,
> > > > > There are projects (like k8s[1]) which do regular meetings
> > > using
> > > > > video conferencing tools. We want to propose such a meeting for
> > Apache
> > > > > Cassandra once a quarter. Here are some of the initial details.
> > > > >
> > > > > 1. A two hour meeting once a quarter starting at 9am Pacific. We
> can
> > > > later
> > > > > move this to other times to make it easier for other timezones.
> > > > > 2. Agenda of the meeting will be due 2 days prior to the meeting. A
> > > > sample
> > > > > agenda for next one could cover updates on 4.0 testing, any major
> > bugs
> > > > > found and/or fixed, next steps for 4.0, etc.
> > > > > 3. Each agenda item will have a time duration and list of people to
> > > drive
> > > > > that item.
> > > > > 4. We will have a moderator for each meeting which will rotate
> around
> > > the
> > > > > community members.
> > > > > 5. We need to figure out which video conferencing tool to use for
> > this.
> > > > > Suggestions and donation of tools are welcome.
> > > > > 6. We will have meeting notes for each item discussed in the
> meeting.
> > > > >
> > > > > Motivation for such a

Re: Apache Cassandra Virtual meetings

[sankalp kohli]

Thank you all for the feedback. It has been a week since last last feedback
so we will move forward with these meeting. We plan to start them post
NGCC. If anyone feedback is not addressed, please let me know.

Also feedback on this will always be welcomed even when meetings are
ongoing as I am assuming shape of this meeting will keeping changing over
time based on feedback.

On Mon, Aug 12, 2019 at 4:33 PM sankalp kohli 
wrote:

> Thanks Patrick for helping with logistics. We can certainly use your
> resources unless someone objects
>
> On Mon, Aug 12, 2019 at 10:42 AM Patrick McFadin 
> wrote:
>
>> If it works for everyone, DataStax has some resources we could put to this
>> effort. We do large scale conferences like this all the time and have the
>> tools to pull it off. It would be a small group of people with full duplex
>> audio and video with the ability for 100s of people to watch in streaming.
>> As Scott mentioned, a recording could be made available after the fact for
>> anyone that wanted to review and historical purposes.
>>
>> If we go this route, I would suggest using the ASF Cassandra slack for
>> people posting questions. 1) It gets more people on slack 2) It's more or
>> less a permanent, searchable record.
>>
>> Patrick
>>
>> On Sun, Aug 11, 2019 at 10:18 AM Rahul Xavier Singh <
>> rahul.xavier.si...@gmail.com> wrote:
>>
>> > I think these meetings would be great.. if there is a specific
>> structure.
>> > We use a simple format that could help e.g.
>> >
>> > 1. Review long term vision/ roadmap.
>> > 2. Review next release / features that are in progress.
>> > 3. Discuss issues in general and make a game plan for the next quarter.
>> >
>> > Nothing too complicated, but at least some structure so that we are
>> > timeboxed on what we are doing.
>> >
>> >
>> >
>> > On Wed, Aug 7, 2019 at 7:42 AM Joshua McKenzie 
>> > wrote:
>> >
>> > > The one thing we need to keep in mind is the "If it didn't happen on a
>> > > mailing list, it didn't happen <http://theapacheway.com/on-list/>"
>> > > philosophy of apache projects. Shouldn't constrain us too much as the
>> > > nuance is:
>> > >
>> > > *"Discussions and plan proposals often happen at events, in chats
>> (Slack,
>> > > IRC, IM, etc.) or other synchronous places. But all final decisions
>> about
>> > > executing on the plan, checking in the new code, or launching the
>> website
>> > > must be made by the community asyncrhonously on the mailing list."*
>> > >
>> > > So long as we keep that in mind (and maybe push it back to 8am PST
>> since
>> > > 9am can get pretty ugly for some of the more eastern european / asian
>> > > countries), makes sense to me.
>> > >
>> > > On Tue, Aug 6, 2019 at 6:07 PM Dinesh Joshi 
>> wrote:
>> > >
>> > > > Thanks for initiating this conversation Sankalp. On the ASF front, I
>> > > think
>> > > > we need to ensure that non-Pacific time participants can also
>> > participate
>> > > > in the discussions. So posting the notes and opening up discussions
>> > after
>> > > > the meet up to dev@ would be a great way of making sure everyone
>> can
>> > > > participate and gets visibility. Additionally, we should consider
>> > > > scheduling this meetup in different timezones as far as logistics
>> allow
>> > > it.
>> > > >
>> > > > Dinesh
>> > > >
>> > > > > On Aug 6, 2019, at 2:58 PM, sankalp kohli > >
>> > > > wrote:
>> > > > >
>> > > > > Hi All,
>> > > > > There are projects (like k8s[1]) which do regular meetings
>> > > using
>> > > > > video conferencing tools. We want to propose such a meeting for
>> > Apache
>> > > > > Cassandra once a quarter. Here are some of the initial details.
>> > > > >
>> > > > > 1. A two hour meeting once a quarter starting at 9am Pacific. We
>> can
>> > > > later
>> > > > > move this to other times to make it easier for other timezones.
>> > > > > 2. Agenda of the meeting will be due 2 days prior to the meeting.
>> A
>> > > > sample
>> > > > > agenda for next one could cover updates on 4.0 testing

Re: Contributing cassandra-diff

[Sankalp Kohli]

A different repo will be better 

> On Aug 22, 2019, at 6:16 AM, Per Otterström  
> wrote:
> 
> Very powerful tool indeed, thanks for sharing!
> 
> I believe it is best to keep tools like this in different repos since 
> different tools will probably have different life cycles and tool chains. 
> Yes, that could be handled in a single repo, but with different repos we'd 
> get natural boundaries.
> 
> -Original Message-
> From: Sumanth Pasupuleti  
> Sent: den 22 augusti 2019 14:40
> To: dev@cassandra.apache.org
> Subject: Re: Contributing cassandra-diff
> 
> No hard preference on the repo, but just excited about this tool! Looking 
> forward to employing this for upgrade testing (very timely :))
> 
>> On Thu, Aug 22, 2019 at 3:38 AM Sam Tunnicliffe  wrote:
>> 
>> My own weak preference would be for a dedicated repo in the first 
>> instance. If/when additional tools are contributed we should look at 
>> co-locating common stuff, but rushing toward a monorepo would be a 
>> mistake IMO.
>> 
 On 22 Aug 2019, at 11:10, Jeff Jirsa  wrote:
>>> 
>>> I weakly prefer contrib.
>>> 
>>> 
>>> On Thu, Aug 22, 2019 at 12:09 PM Marcus Eriksson 
>>> 
>> wrote:
>>> 
 Hi, we are about to open source our tooling for comparing two 
 cassandra clusters and want to get some feedback where to push it. 
 I think the options are: (name bike-shedding welcome)
 
 1. create repos/asf/cassandra-diff.git 2. create a generic 
 repos/asf/cassandra-contrib.git where we can add
>> more
 contributed tools in the future
 
 Temporary location: 
 https://protect2.fireeye.com/url?k=e8982d07-b412e678-e8986d9c-86717
 581b0b5-292bc820a13b7138&q=1&u=https%3A%2F%2Fgithub.com%2Fkrummas%2
 Fcassandra-diff
 
 Cassandra-diff is a spark job that compares the data in two 
 clusters -
>> it
 pages through all partitions and reads all rows for those 
 partitions in both clusters to make sure they are identical. Based 
 on the
>> configuration
 variable “reverse_read_probability” the rows are either read 
 forward or
>> in
 reverse order.
 
 Our main use case for cassandra-diff has been to set up two 
 identical clusters, transfer a snapshot from the cluster we want to 
 test to these clusters and upgrade one side. When that is done we 
 run this tool to
>> make
 sure that 2.1 and 3.0 gives the same results. A few examples of the
>> bugs we
 have found using this tool:
 
 * CASSANDRA-14823: Legacy sstables with range tombstones spanning
>> multiple
 index blocks create invalid bound sequences on 3.0+
 * CASSANDRA-14803: Rows that cross index block boundaries can cause 
 incomplete reverse reads in some cases
 * CASSANDRA-15178: Skipping illegal legacy cells can break reverse 
 iteration of indexed partitions
 
 /Marcus
 
 ---
 -- To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
 For additional commands, e-mail: dev-h...@cassandra.apache.org
 
 
>> 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 
>> 
> B‹CB•È[œÝXœØÜšX™KK[XZ[ˆ]‹][œÝXœØÜšX™PØ\ÜØ[™˜K˜\XÚK›Ü™ÃB‘›ÜˆY][Û˜[ÛÛ[X[™ËK[XZ[ˆ]‹Z[Ø\ÜØ[™˜K˜\XÚK›Ü™ÃBƒB

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: 4.0 alpha before apachecon?

[sankalp kohli]

Will we cut alpha1 and later alpha releases from trunk or create a new
branch?
On Friday, August 30, 2019, Nate McCall  wrote:

> >
> >
> > I think the next decision is should we just cut 4.0-alpha1 now given
> > that Michael has some cycles regardless of the known issues and start
> > using the new fix versions for the 4.0-alpha2 release? I personally
> > feel we should cut 4.0-alpha1 with every imaginable "expect this
> > release to break" disclaimer and start working towards 4.0-alpha2.
> >
>
> Let's just pull the trigger on it. We know there are a couple of rough
> edges, but that is the point of an alpha.
>

Re: 4.0 alpha before apachecon?

[Sankalp Kohli]

I do not think we should branch and is -1 on it. The reason we have trunk 
frozen was for our focus to be on 4.0. I think we still need that focus till a 
few more releases like these. 

> On Aug 30, 2019, at 12:24 AM, Nate McCall  wrote:
> 
> On Fri, Aug 30, 2019 at 10:11 AM Benedict Elliott Smith 
> wrote:
> 
>> 
>>Seems to make sense to branch, right?
>> 
>> Feels like a good line in the sand. +1

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [VOTE] Release Apache Cassandra 4.0-alpha1 (24 hour vote)

[Sankalp Kohli]

Can we have a vote once the tests pass? I know we all are including me are 
excited about cutting this alpha but we cannot cut a release with test failing 
or not being run due to some Java home issue. 

If people have already started using the alpha artifacts, then I suggest we 
make test passing a blocker for next alpha 

> On Sep 6, 2019, at 5:44 AM, Michael Shuler  wrote:
> 
> Thanks a bunch for the test runs. Most of those upgrade tests show 
> "RuntimeError: You need to set JAVA8_HOME," so it does look like a systemic, 
> generic issue to resolve there somewhere in dtest. Thanks again!
> 
> Michael
> 
>> On 9/5/19 8:08 PM, Joseph Lynch wrote:
>> Following passed 100%: utest_compression, utests_stress, utests_long,
>> utests_fqltool, j8_dtests-no-vnodes
>> j8_unit_tests: 1 failure
>> - testAcquireReleaseOutbound - org.apache.cassandra.net.ConnectionTest
>> j11_unit_tests: 1 failure
>> - testIdleDisconnect - org.apache.cassandra.transport.IdleDisconnectTest
>> j8_dtests-with-vnodes + j11_dtests-with-vnodes: 1 failure
>> - test_remote_query - cql_test.TestCQLSlowQuery
>> j11_dtests-no-vnodes: 1 failure
>> - test_13595 - consistency_test.TestConsistency
>> j8_upgradetests-no-vnodes: 3923 failures
>> Other than the upgradetests (which afaik nobody has looked at for
>> 4.0), this looks somewhat reasonable to me. I'm launching a multi-dc
>> cluster to do some light load testing.
>> -Joey
>>> On Thu, Sep 5, 2019 at 4:03 PM Joseph Lynch  wrote:
>>> 
>>> Running all tests at 
>>> https://circleci.com/workflow-run/79918e2a-ea8e-48a6-a38d-96cf85de27ff
>>> 
>>> Will report back with results shortly,
>>> -Joey
>>> 
>>> On Thu, Sep 5, 2019 at 3:55 PM Jon Haddad  wrote:
 
 +1
 
 On Thu, Sep 5, 2019 at 3:44 PM Michael Shuler 
 wrote:
 
> I propose the following artifacts for release as 4.0-alpha1.
> 
> sha1: fc4381ca89ab39a82c9018e5171975285cc3bfe7
> Git:
> 
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/4.0-alpha1-tentative
> Artifacts:
> 
> https://repository.apache.org/content/repositories/orgapachecassandra-1177/org/apache/cassandra/apache-cassandra/4.0-alpha1/
> Staging repository:
> https://repository.apache.org/content/repositories/orgapachecassandra-1177/
> 
> The Debian and RPM packages are available here:
> http://people.apache.org/~mshuler
> 
> The vote will be open for 24 hours (longer if needed).
> 
> [1]: CHANGES.txt:
> 
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/4.0-alpha1-tentative
> [2]: NEWS.txt:
> 
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/4.0-alpha1-tentative
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 
> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [DISCUSS] Proposing an Cassandra Enhancement Proposal (CEP) process

[sankalp kohli]

Another thing which it should solve is someone proposing an alternate very
late into development which could be provided sooner. If someone has a good
feedback which could not have been given at the time of CEP then that is
good. We don't want situations where contributors have done the CEP and
then worked on implementation of it and then someone who has not read the
CEP comes in and starts giving feedback. This feedback should come at the
time of CEP if CEP has covered that area.
To be clear, I am not saying people should not give feedback later just
that they dont ignore the whole thing and wake up later in the process.
This causes huge productivity and morale loss to code contributors who are
in minority right now in the community.

On Tue, Sep 17, 2019 at 3:44 AM Benedict Elliott Smith 
wrote:

> Can we modify the document to make this really explicit then?  Right now,
> the language suggests the process is mandated, rather than encouraged and
> beneficial.
>
> It would be nice to frame it as a positive and incentivised undertaking by
> authors, and to list the intended advantages, as well as the potential
> disadvantages of not undertaking it, while making it clear it is left
> entirely to their own judgement whether or not to do so.
>
> To be really clear, I do not refer to the flexible definition of the
> process, but to whether participation in even a modest interpretation of
> the process is necessary at all.  This is a form of pre-registration for
> work, to achieve community buy-in.  If you want to go ahead and do
> something on your own, you only risk difficulty and delays when obtaining
> community buy-in after the fact.  Let's not dissuade hobbyists, part-timers
> or scratching an itch by suggesting their work will be discounted because
> it wasn't pre-registered.
>
>
> On 17/09/2019, 06:46, "Mick Semb Wever"  wrote:
>
>
>
> > I think we need to have a meta discussion about the goal for
> > introducing a new process.
>
>
> Indeed, and these were only two brief examples that came to me.
> Another, using the sidecar proposal as an example, is simply to ensure a
> little patience is taken during the initial brainstorming and navigation
> phase, to give more open collaboration a better chance. What's in the
> landscape, where's the value, who might be interested in getting involved
> in this, etc etc. I think the C* community has typically been pretty
> amazing at this, but it would be nice to see it formalised a bit better.
>
>
> > By not mandating it, we do not need to define where it is necessary;
> > the larger and more impactful the change, the greater the incentive
> to
> > the author.
>
>
> This is what Scott highlighted well.
> Sure, a CEP could be opened with nothing but a title to begin with.
> And where it goes from there is up to the working group that materialises.
> Just to have a landing space for new features that's not Jira, I believe
> would be of value.
>
> And in no way should the CEP be a return to waterfall. As you say,
> late discoveries and feedback (as annoying as it can be) is all part of the
> agile game.
>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>
>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: "4.0: TBD" -> "4.0: Est. Q4 2019"?

[sankalp kohli]

king for any additional
> >>>>> feedback
> >>>>>folks may have.
> >>>>>
> >>>>>
> >>>>
> >>>
> https://docs.google.com/document/d/1bS6sr-HSrHFjZb0welife6Qx7u3ZDgRiAoENMLYlfz8/edit#
> >>>>>
> >>>>>Thanks,
> >>>>>Sumanth
> >>>>>
> >>>>>On Tue, May 28, 2019 at 10:43 PM Scott Andreas <
> >>> sc...@paradoxica.net
> >>>>>
> >>>>> wrote:
> >>>>>
> >>>>>> Echoing Jon’s point here –
> >>>>>>
> >>>>>> JH: “My thinking is I'd like to be able to recommend 4.0.0 as a
> >>>>> production
> >>>>>> ready
> >>>>>> database for business critical cases”
> >>>>>>
> >>>>>> I feel that this is a standard that is both appropriate and
> >>>>> achievable,
> >>>>>> and one I’m legitimately excited about.
> >>>>>>
> >>>>>> Re: the current state of the test plan wiki in Confluence, I owe
> >>>>> another
> >>>>>> pass through. There has been a lot of progress here, but I’ve
> >>> let
> >>>>> perfect
> >>>>>> be the enemy of the good in getting updates out. I’ll complete
> >>> that
> >>>>> pass
> >>>>>> later this week.
> >>>>>>
> >>>>>> Cheers,
> >>>>>>
> >>>>>> — Scott
> >>>>>>
> >>>>>>> On May 28, 2019, at 10:48 AM, Dinesh Joshi  >>>>
> >>>>> wrote:
> >>>>>>>
> >>>>>>> +1. Wiki could be useful to document what the overall plan.
> >>> Jira
> >>>> to
> >>>>>> track progress.
> >>>>>>>
> >>>>>>> Dinesh
> >>>>>>>
> >>>>>>>>> On May 28, 2019, at 10:20 AM, Joshua McKenzie <
> >>>>> jmcken...@apache.org>
> >>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> The unofficial rule is to not upgrade to prod till .10 is
> >>> cut.
> >>>>>>>>
> >>>>>>>> FWIW, I believe it's historically .6. Which is still not a
> >>> great
> >>>>> look
> >>>>>> for
> >>>>>>>> the project.
> >>>>>>>>
> >>>>>>>> There's a ton of work going into testing 4.0 already.
> >>>>>>>>
> >>>>>>>> While I intuitively and anecdotally (from the people I've
> >>>>> backchanneled
> >>>>>>>> with) believe this to be true as well, the referenced wiki
> >>>>> page[1] and
> >>>>>>>> jql[2] doesn't look like it's an up to date reflection of the
> >>>>> testing
> >>>>>>>> efforts going on. Is there another place this information is
> >>>>> stored /
> >>>>>>>> queryable we can surface to people to keep us all
> >>> coordinated?
> >>>>>>>>
> >>>>>>>> [1]
> >>>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> https://cwiki.apache.org/confluence/display/CASSANDRA/4.0+Quality%3A+Components+and+Test+Plans
> >>>>>>>> [2]
> >>>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> https://issues.apache.org/jira/browse/CASSANDRA-14862?jql=project%20%3D%20CASSANDRA%20AND%20%20labels%20%3D%204.0-QA
> >>>>>>>>
> >>>>>>>> On Tue, May 28, 2019 at 12:57 PM sankalp kohli <
> >>>>> kohlisank...@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi Jon,
> >>>>>>>>> When you say 4.0 release, how do u match it with
> >>> 3.0
> >>>>> minor
> >>>>>>>>> releases. The unofficial rule is to not upgrade to prod till
> >>>&

Re: Cassandra GUI that supports cqlsh through Bastion

[sankalp kohli]

Please email to user list as I do not think this question is for dev list.

On Wed, Sep 18, 2019 at 10:17 AM Bhavesh Prajapati
 wrote:

> Hi,
>
> I am looking for Cassandra GUI that supports cqlsh connection to Cassandra
> node through bastion/jump host using ssh key.
>
> Thanks,
> Bhavesh
>

Re: "4.0: TBD" -> "4.0: Est. Q4 2019"?

[sankalp kohli]

Can we put it on vote(if required) if no one has more comments?

On Wed, Sep 18, 2019 at 5:44 PM Jonathan Koppenhofer 
wrote:

> Nice work... I like this and have no additions/comments at this time
>
> On Wed, Sep 18, 2019, 4:18 PM sankalp kohli 
> wrote:
>
> > We added and changed a lot of things to this doc during a discussion in
> > NGCC. Can everyone take a look at it and provide feedback.
> >
> > On Wed, Sep 11, 2019 at 10:51 PM Dinesh Joshi  wrote:
> >
> > > I have left some comments on the document. Apart from a few
> > clarifications
> > > and some minor changes, I feel its in a good shape. I think we should
> > move
> > > forward with it. We can refine the process, definitions & criteria as
> we
> > > learn.
> > >
> > > Dinesh
> > >
> > > > On Sep 11, 2019, at 11:15 AM, Sumanth Pasupuleti <
> > > sumanth.pasupuleti...@gmail.com> wrote:
> > > >
> > > > One more call for any additional comments/ feedback on the release
> > > > lifecycle document
> > > >
> > >
> >
> https://docs.google.com/document/d/1bS6sr-HSrHFjZb0welife6Qx7u3ZDgRiAoENMLYlfz8/edit#
> > > >
> > > > Thanks,
> > > > Sumanth
> > > >
> > > > On Sat, Jul 27, 2019 at 1:01 AM Sumanth Pasupuleti <
> > > > sumanth.pasupuleti...@gmail.com> wrote:
> > > >
> > > >> Submitted patch to add release lifecycle information to the website
> > > >> https://issues.apache.org/jira/browse/CASSANDRA-15249
> > > >>
> > > >> On Tue, Jun 25, 2019 at 6:57 AM Oleksandr Petrov <
> > > >> oleksandr.pet...@gmail.com> wrote:
> > > >>
> > > >>> Maybe a bit off-topic:
> > > >>>
> > > >>> Before we cut a release, we should make sure we take care of beta
> > > protocol
> > > >>> [1], include released driver versions [2] and remove compact
> storage
> > > >>> remainders [3]. Third one is optional, but I'd argue we should do
> it
> > > >>> sooner
> > > >>> rather than later.
> > > >>>
> > > >>> [1] https://issues.apache.org/jira/browse/CASSANDRA-14973
> > > >>> [2] https://issues.apache.org/jira/browse/CASSANDRA-13951
> > > >>> [3] https://issues.apache.org/jira/browse/CASSANDRA-13994
> > > >>>
> > > >>>
> > > >>>
> > > >>> On Sat, Jun 22, 2019 at 1:25 AM Sumanth Pasupuleti <
> > > >>> sumanth.pasupuleti...@gmail.com> wrote:
> > > >>>
> > > >>>> Thanks for the feedback Scott. I have incorporated all the
> > incremental
> > > >>>> feedback I have thus far.
> > > >>>>
> > > >>>> Looking for any additional feedback folks may have.
> > > >>>>
> > > >>>>
> > > >>>
> > >
> >
> https://docs.google.com/document/d/1bS6sr-HSrHFjZb0welife6Qx7u3ZDgRiAoENMLYlfz8/edit#
> > > >>>>
> > > >>>> On Tue, Jun 11, 2019 at 11:54 AM Scott Andreas <
> > sc...@paradoxica.net>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> Thanks for starting this discussion, Sumanth! Added a round of
> > > >>> comments
> > > >>>> as
> > > >>>>> well.
> > > >>>>>
> > > >>>>> Summarizing my non-binding feedback: I feel that many of the
> items
> > > >>> under
> > > >>>>> "Alpha" and "Beta" should be achieved prior to the release of an
> > > >>> alpha,
> > > >>>>> especially those related to correctness/safety, scope lock,
> feature
> > > >>>>> completeness, deprecation, and backwards compatibility.
> > Establishing
> > > a
> > > >>>>> higher standard for official project releases (even at the alpha
> > and
> > > >>> beta
> > > >>>>> stage) will help us really polish the final build together.
> > > >>>>>
> > > >>>>> Ideally, I feel that contributors should have completed extensive
> > > >>>>> testing/validation to ensure that no critical or severe bugs
> exist
> > > >>> prior
> > > >&

Re: "4.0: TBD" -> "4.0: Est. Q4 2019"?

[Sankalp Kohli]

Let’s put this to vote next week unless someone thinks it is not required 

> On Sep 25, 2019, at 10:56 AM, sankalp kohli  wrote:
> 
> 
> Can we put it on vote(if required) if no one has more comments? 
> 
>> On Wed, Sep 18, 2019 at 5:44 PM Jonathan Koppenhofer  
>> wrote:
>> Nice work... I like this and have no additions/comments at this time
>> 
>> On Wed, Sep 18, 2019, 4:18 PM sankalp kohli  wrote:
>> 
>> > We added and changed a lot of things to this doc during a discussion in
>> > NGCC. Can everyone take a look at it and provide feedback.
>> >
>> > On Wed, Sep 11, 2019 at 10:51 PM Dinesh Joshi  wrote:
>> >
>> > > I have left some comments on the document. Apart from a few
>> > clarifications
>> > > and some minor changes, I feel its in a good shape. I think we should
>> > move
>> > > forward with it. We can refine the process, definitions & criteria as we
>> > > learn.
>> > >
>> > > Dinesh
>> > >
>> > > > On Sep 11, 2019, at 11:15 AM, Sumanth Pasupuleti <
>> > > sumanth.pasupuleti...@gmail.com> wrote:
>> > > >
>> > > > One more call for any additional comments/ feedback on the release
>> > > > lifecycle document
>> > > >
>> > >
>> > https://docs.google.com/document/d/1bS6sr-HSrHFjZb0welife6Qx7u3ZDgRiAoENMLYlfz8/edit#
>> > > >
>> > > > Thanks,
>> > > > Sumanth
>> > > >
>> > > > On Sat, Jul 27, 2019 at 1:01 AM Sumanth Pasupuleti <
>> > > > sumanth.pasupuleti...@gmail.com> wrote:
>> > > >
>> > > >> Submitted patch to add release lifecycle information to the website
>> > > >> https://issues.apache.org/jira/browse/CASSANDRA-15249
>> > > >>
>> > > >> On Tue, Jun 25, 2019 at 6:57 AM Oleksandr Petrov <
>> > > >> oleksandr.pet...@gmail.com> wrote:
>> > > >>
>> > > >>> Maybe a bit off-topic:
>> > > >>>
>> > > >>> Before we cut a release, we should make sure we take care of beta
>> > > protocol
>> > > >>> [1], include released driver versions [2] and remove compact storage
>> > > >>> remainders [3]. Third one is optional, but I'd argue we should do it
>> > > >>> sooner
>> > > >>> rather than later.
>> > > >>>
>> > > >>> [1] https://issues.apache.org/jira/browse/CASSANDRA-14973
>> > > >>> [2] https://issues.apache.org/jira/browse/CASSANDRA-13951
>> > > >>> [3] https://issues.apache.org/jira/browse/CASSANDRA-13994
>> > > >>>
>> > > >>>
>> > > >>>
>> > > >>> On Sat, Jun 22, 2019 at 1:25 AM Sumanth Pasupuleti <
>> > > >>> sumanth.pasupuleti...@gmail.com> wrote:
>> > > >>>
>> > > >>>> Thanks for the feedback Scott. I have incorporated all the
>> > incremental
>> > > >>>> feedback I have thus far.
>> > > >>>>
>> > > >>>> Looking for any additional feedback folks may have.
>> > > >>>>
>> > > >>>>
>> > > >>>
>> > >
>> > https://docs.google.com/document/d/1bS6sr-HSrHFjZb0welife6Qx7u3ZDgRiAoENMLYlfz8/edit#
>> > > >>>>
>> > > >>>> On Tue, Jun 11, 2019 at 11:54 AM Scott Andreas <
>> > sc...@paradoxica.net>
>> > > >>>> wrote:
>> > > >>>>
>> > > >>>>> Thanks for starting this discussion, Sumanth! Added a round of
>> > > >>> comments
>> > > >>>> as
>> > > >>>>> well.
>> > > >>>>>
>> > > >>>>> Summarizing my non-binding feedback: I feel that many of the items
>> > > >>> under
>> > > >>>>> "Alpha" and "Beta" should be achieved prior to the release of an
>> > > >>> alpha,
>> > > >>>>> especially those related to correctness/safety, scope lock, feature
>> > > >>>>> completeness, deprecation, and backwards compatibility.
>> > Establishing
>> > > a
>> > > >>>>> higher standard for o

[VOTE] Apache Cassandra Release Lifecycle

[sankalp kohli]

Hi,
We have discussed in the email thread[1] about Apache Cassandra Release
Lifecycle. We came up with a doc[2] for it. Please vote on it if you agree
with the content of the doc[2].

Thanks,
Sankalp

[1]
https://lists.apache.org/thread.html/c610b23f9002978636b66d09f0e0481ed3de9b78895050da22c91c6f@%3Cdev.cassandra.apache.org%3E
[2]
https://docs.google.com/document/d/1bS6sr-HSrHFjZb0welife6Qx7u3ZDgRiAoENMLYlfz8/edit#heading=h.633eppni91tw

Re: "4.0: TBD" -> "4.0: Est. Q4 2019"?

[sankalp kohli]

zy consensus is a very important concept in
> the
> > Apache Way <
> https://community.apache.org/committers/lazyConsensus.html
> > >.
> >
> > Methinks if we got a little more comfortable w/lazy consensus and
> > majority
> > voting on process <https://www.apache.org/foundation/voting.html> we
> > might
> > see some quicker evolution on the project.
> >
> > Not to hijack the thread; just figured I'd point it out since it was
> > on my
> > mind and it may not be common knowledge.
> >
> > On Fri, Sep 27, 2019 at 12:20 PM Sankalp Kohli <
> kohlisank...@gmail.com
> > >
> > wrote:
> >
> > > Let’s put this to vote next week unless someone thinks it is not
> > required
> > >
> > > > On Sep 25, 2019, at 10:56 AM, sankalp kohli <
> > kohlisank...@gmail.com>
> > > wrote:
> > > >
> > > > 
> > > > Can we put it on vote(if required) if no one has more comments?
> > > >
> > > >> On Wed, Sep 18, 2019 at 5:44 PM Jonathan Koppenhofer <
> > > j...@koppedomain.com> wrote:
> > > >> Nice work... I like this and have no additions/comments at this
> > time
> > > >>
> > > >> On Wed, Sep 18, 2019, 4:18 PM sankalp kohli <
> > kohlisank...@gmail.com>
> > > wrote:
> > > >>
> > > >> > We added and changed a lot of things to this doc during a
> > discussion
> > > in
> > > >> > NGCC. Can everyone take a look at it and provide feedback.
> > > >> >
> > > >> > On Wed, Sep 11, 2019 at 10:51 PM Dinesh Joshi <
> > djo...@apache.org>
> > > wrote:
> > > >> >
> > > >> > > I have left some comments on the document. Apart from a few
> > > >> > clarifications
> > > >> > > and some minor changes, I feel its in a good shape. I think
> we
> > > should
> > > >> > move
> > > >> > > forward with it. We can refine the process, definitions &
> > criteria
> > > as we
> > > >> > > learn.
> > > >> > >
> > > >> > > Dinesh
> > > >> > >
> > > >> > > > On Sep 11, 2019, at 11:15 AM, Sumanth Pasupuleti <
> > > >> > > sumanth.pasupuleti...@gmail.com> wrote:
> > > >> > > >
> > > >> > > > One more call for any additional comments/ feedback on the
> > release
> > > >> > > > lifecycle document
> > > >> > > >
> > > >> > >
> > > >> >
> > >
> >
> https://docs.google.com/document/d/1bS6sr-HSrHFjZb0welife6Qx7u3ZDgRiAoENMLYlfz8/edit#
> > > >> > > >
> > > >> > > > Thanks,
> > > >> > > > Sumanth
> > > >> > > >
> > > >> > > > On Sat, Jul 27, 2019 at 1:01 AM Sumanth Pasupuleti <
> > > >> > > > sumanth.pasupuleti...@gmail.com> wrote:
> > > >> > > >
> > > >> > > >> Submitted patch to add release lifecycle information to
> the
> > > website
> > > >> > > >> https://issues.apache.org/jira/browse/CASSANDRA-15249
> > > >> > > >>
> > > >> > > >> On Tue, Jun 25, 2019 at 6:57 AM Oleksandr Petrov <
> > > >> > > >> oleksandr.pet...@gmail.com> wrote:
> > > >> > > >>
> > > >> > > >>> Maybe a bit off-topic:
> > > >> > > >>>
> > > >> > > >>> Before we cut a release, we should make sure we take
> care
> > of
> > > beta
> > > >> > > protocol
> > > >> > > >>> [1], include released driver versions [2] and remove
> > compact
> > > storage
> > > >> > > >>> remainders [3]. Third one is optional, but I'd argue we
> > should
> > > do it
> > > >> > > >>> sooner
> > > >> > &

Re: [VOTE] Apache Cassandra Release Lifecycle

[sankalp kohli]

Vote will be open for 72 hours.

On Mon, Sep 30, 2019 at 12:09 PM Sumanth Pasupuleti <
sumanth.pasupuleti...@gmail.com> wrote:

> +1
>
> On Mon, Sep 30, 2019 at 12:00 PM Nate McCall  wrote:
>
> > +1
> >
> > On Tue, Oct 1, 2019 at 7:52 AM sankalp kohli 
> > wrote:
> >
> > > Hi,
> > > We have discussed in the email thread[1] about Apache Cassandra
> > Release
> > > Lifecycle. We came up with a doc[2] for it. Please vote on it if you
> > agree
> > > with the content of the doc[2].
> > >
> > > Thanks,
> > > Sankalp
> > >
> > > [1]
> > >
> > >
> >
> https://lists.apache.org/thread.html/c610b23f9002978636b66d09f0e0481ed3de9b78895050da22c91c6f@%3Cdev.cassandra.apache.org%3E
> > > [2]
> > >
> > >
> >
> https://docs.google.com/document/d/1bS6sr-HSrHFjZb0welife6Qx7u3ZDgRiAoENMLYlfz8/edit#heading=h.633eppni91tw
> > >
> >
>

Re: [VOTE] Apache Cassandra Release Lifecycle

[sankalp kohli]

Sure..lets do that.

On Mon, Sep 30, 2019 at 3:21 PM Benedict Elliott Smith 
wrote:

> Perhaps we should move the proposed document to the cwiki for purposes of
> voting?  That way it's in a place owned by the project, with a permanent
> history.  Otherwise it's not entirely clear what was voted on.
>
>
> On 30/09/2019, 20:10, "Sumanth Pasupuleti" <
> sumanth.pasupuleti...@gmail.com> wrote:
>
> +1
>
> On Mon, Sep 30, 2019 at 12:00 PM Nate McCall 
> wrote:
>
>     > +1
> >
> > On Tue, Oct 1, 2019 at 7:52 AM sankalp kohli  >
> > wrote:
> >
> > > Hi,
> > > We have discussed in the email thread[1] about Apache Cassandra
> > Release
> > > Lifecycle. We came up with a doc[2] for it. Please vote on it if
> you
> > agree
> > > with the content of the doc[2].
> > >
> > > Thanks,
> > > Sankalp
> > >
> > > [1]
> > >
> > >
> >
> https://lists.apache.org/thread.html/c610b23f9002978636b66d09f0e0481ed3de9b78895050da22c91c6f@%3Cdev.cassandra.apache.org%3E
> > > [2]
> > >
> > >
> >
> https://docs.google.com/document/d/1bS6sr-HSrHFjZb0welife6Qx7u3ZDgRiAoENMLYlfz8/edit#heading=h.633eppni91tw
> > >
> >
>
>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

[VOTE-2] Apache Cassandra Release Lifecycle

[sankalp kohli]

Hi,
We have discussed in the email thread[1] about Apache Cassandra Release
Lifecycle. We came up with a doc[2] for it. We have finalized the doc
here[3] Please vote on it if you agree with the content of the doc [3].

We did not proceed with the previous vote as we want to use confluence for
it. Here is the link for that[4]. It also mentions why we are doing this
vote.

Vote will remain open for 72 hours.

Thanks,
Sankalp

[1]
https://lists.apache.org/thread.html/c610b23f9002978636b66d09f0e0481ed3de9b78895050da22c91c6f@%3Cdev.cassandra.apache.org%3E
[2]
https://docs.google.com/document/d/1bS6sr-HSrHFjZb0welife6Qx7u3ZDgRiAoENMLYlfz8/edit#heading=h.633eppni91tw
[3]https://cwiki.apache.org/confluence/display/CASSANDRA/Release+Lifecycle
Attachments area
[4]
https://lists.apache.org/thread.html/169b00f45dbad295e1aea1da70365fabc8452ef497f78ddfd28c311f@%3Cdev.cassandra.apache.org%3E

Re: [VOTE] Apache Cassandra Release Lifecycle

[sankalp kohli]

Thanks Sumanth. Let me start another vote

On Tue, Oct 8, 2019 at 7:26 AM Sumanth Pasupuleti <
sumanth.pasupuleti...@gmail.com> wrote:

> Thank you, Scott!
> I've moved the document content (along with outstanding comments) into
> cwiki at
> https://cwiki.apache.org/confluence/display/CASSANDRA/Release+Lifecycle.
> Made the GDoc deprecated and view-only, and left breadcrumbs to the cwiki
> in both the jira <https://issues.apache.org/jira/browse/CASSANDRA-15249>
> as
> well as the GDoc
> <
> https://docs.google.com/document/d/1bS6sr-HSrHFjZb0welife6Qx7u3ZDgRiAoENMLYlfz8/edit#
> >
> .
>
> On Mon, Oct 7, 2019 at 12:26 PM Jordan West  wrote:
>
> > +1 nb — to both the document and the benefits listed in Scott’s email.
> >
> > Jordan
> >
> > On Fri, Oct 4, 2019 at 2:26 PM Scott Andreas 
> wrote:
> >
> > > There are two main benefits to agreeing on this:
> > >
> > > 1. Providing clarity for contributors on release phases – i.e., what
> > types
> > > of changes are expected to land or be deferred during a particular
> point
> > in
> > > that cycle.
> > >
> > > 2. Providing semantic clarity to users of Cassandra in terms of what
> they
> > > can expect from a given release.
> > >
> > > The second one is more important. The document stands as a commitment
> > > between the Cassandra project and its users regarding what can be
> > expected
> > > from each type of release. It defines GA releases as "recommended for
> > > production deployment for all users," setting a standard of quality
> that
> > we
> > > aim to uphold together and that users can depend on. Affirming what it
> > > means for a release to be EOL, deprecated, or in maintenance signals
> > > importance of upgrading to a GA version.
> > >
> > > The prerelease phases set expectations for us as contributors to
> produce
> > a
> > > more stable release: what type of testing/validation is needed at what
> > > time, at which point interfaces/protocols solidify, when a release is
> > > considered feature complete, etc.
> > >
> > > Creating clarity for ourselves will help us build a better database,
> and
> > > creating clarity for our users will help give them the confidence to
> run
> > it.
> > >
> > > I want to thank Sumanth for his work on this document and to everyone
> > else
> > > who's contributed.
> > >
> > > I support it (+1 nb).
> > >
> > > – Scott
> > >
> > > 
> > > From: Stefan Podkowinski 
> > > Sent: Tuesday, October 1, 2019 1:43 PM
> > > To: dev@cassandra.apache.org
> > > Subject: Re: [VOTE] Apache Cassandra Release Lifecycle
> > >
> > > What exactly will be the implication of the outcome of this vote, in
> > > case the content is agreed upon? What's the proposal of the voting?
> > >
> > > The document seems to be informative rather then formal. It's verbose
> on
> > > definitions that should be commonly understood or can only broadly
> > > defined (what is alpha/beta/RC, GA for production, etc.), while at the
> > > same time is unclear and weasel-wordy on our actual commitment and
> rules.
> > >
> > >
> > > On 30.09.19 20:51, sankalp kohli wrote:
> > > > Hi,
> > > >  We have discussed in the email thread[1] about Apache Cassandra
> > > Release
> > > > Lifecycle. We came up with a doc[2] for it. Please vote on it if you
> > > agree
> > > > with the content of the doc[2].
> > > >
> > > > Thanks,
> > > > Sankalp
> > > >
> > > > [1]
> > > >
> > >
> >
> https://lists.apache.org/thread.html/c610b23f9002978636b66d09f0e0481ed3de9b78895050da22c91c6f@%3Cdev.cassandra.apache.org%3E
> > > > [2]
> > > >
> > >
> >
> https://docs.google.com/document/d/1bS6sr-HSrHFjZb0welife6Qx7u3ZDgRiAoENMLYlfz8/edit#heading=h.633eppni91tw
> > > >
> > >
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > > For additional commands, e-mail: dev-h...@cassandra.apache.org
> > >
> > >
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > > For additional commands, e-mail: dev-h...@cassandra.apache.org
> > >
> > >
> >
>

Re: [VOTE-2] Apache Cassandra Release Lifecycle

[Sankalp Kohli]

Can we please stick to the vote here. We can always change the documentation as 
it will evolve over time. 


> On Oct 8, 2019, at 2:57 PM, Joshua McKenzie  wrote:
> 
> It probably warrants a separate discussion about how we version.
> 
> Also, robust +1 to what you said here bes and haddad  ;)
> 
>> On Tue, Oct 8, 2019 at 5:31 PM Jon Haddad  wrote:
>> 
>> This has definitely been a confusing topic in the past, I completely agree
>> Benedict.  Glad you brought this up.
>> 
>> I'm 100% on board with 5.0 after 4.0.
>> 
>> On Tue, Oct 8, 2019 at 2:27 PM Benedict Elliott Smith >> 
>> wrote:
>> 
>>> As a brief side-step on the topic only of versioning (which no doubt will
>>> cause enough consternation), I personally endorse streamlining it.  We
>> have
>>> not had a consistently meaningful convention on this, at any point, and
>> we
>>> made it even worse in the 3.x line.  There's no real difference between
>>> 1.2->2.0, 2.0->2.1, or 3.0->3.11 and 3.11->4.0; let's admit this and go
>>> straight to 5.0 for our next feature release, with 4.1 our first patch
>>> release of the 4.x line.
>>> 
>>> 
>>>> On 08/10/2019, 21:36, "Scott Andreas"  wrote:
>>> 
>>>Re: "How can we decide that *all* new features are suppose to go into
>>> trunk only, if we don’t even have an idea about the upcoming release
>>> schedule?"
>>> 
>>>This is a great question. My understanding of the intent of the
>>> document is that new features are generally expected to land in trunk,
>> with
>>> an exception process defined for feature backports. I think that's a
>>> reasonable expectation to start with. But I also agree with you that it's
>>> important we evolve a way to discuss and agree up on release scope - this
>>> was the focus of my slides at NGCC. I would love to discuss this on a
>>> separate thread.
>>> 
>>>Re: “Bug fix releases have associated new minor version.”
>>>"Patchlevel version" might be more in keeping with our current
>>> convention.
>>> 
>>>Re: "We should give users a way to plan, by having EOL dates"
>>>Incorporating EOL dates into our release management / planning is a
>>> great idea.
>>> 
>>>Would you be willing to rephrase your comments in the form of
>> proposed
>>> edits to the document?
>>> 
>>>– Scott
>>> 
>>>
>>>From: Stefan Podkowinski 
>>>Sent: Tuesday, October 8, 2019 1:22 PM
>>>To: dev@cassandra.apache.org
>>>Subject: Re: [VOTE-2] Apache Cassandra Release Lifecycle
>>> 
>>> From the document:
>>> 
>>>General Availability (GA): “A new branch is created for the release
>>> with
>>>the new major version, limiting any new feature addition to the new
>>>release branch, with new feature development will continue to happen
>>>only on trunk.”
>>>Maintenance: “Missing features from newer generation releases are
>>>back-ported on per - PMC vote basis.”
>>> 
>>>We had a feature freeze before 4.0, which showed that people have
>>>different views on what actually qualifies as a feature. It doesn’t
>>> work
>>>without defining “feature” in more detail. Although I’d rather avoid
>> to
>>>have this in the document at all, since I don’t think this is getting
>>> us
>>>anywhere, without having a clearer picture on the bigger context in
>>>which release are going to happen in the future, starting with
>> release
>>>cadence and support periods. How can we decide that *all* new
>> features
>>>are suppose to go into trunk only, if we don’t even have an idea
>> about
>>>the upcoming release schedule?
>>> 
>>>“Bug fix releases have associated new minor version.”
>>> 
>>>So the next bug fix version will be 4.1? There will be no minor
>> feature
>>>releases like we did with 3.x.0/2.x.0?
>>> 
>>>Deprecated:
>>>"Through a dev community voting process, EOL date is determined for
>>> this
>>>    release.”
>>>“Users actively encouraged to move away from the offering.”
>>> 
>>>We should give users a way to plan, by having EOL dates that may be
>>>months

Re: [VOTE-2] Apache Cassandra Release Lifecycle

[sankalp kohli]

Vote passes with 11 +1 and no -1

On Thu, Oct 10, 2019 at 2:41 PM Jordan West  wrote:

> +1 nb
>
> On Wed, Oct 9, 2019 at 11:00 PM Per Otterström
>  wrote:
>
> > +1 nb
> >
> > -Original Message-
> > From: Mick Semb Wever 
> > Sent: den 10 oktober 2019 07:08
> > To: dev@cassandra.apache.org
> > Subject: Re: [VOTE-2] Apache Cassandra Release Lifecycle
> >
> >
> > > We have discussed in the email thread[1] about Apache Cassandra
> > > Release Lifecycle. We came up with a doc[2] for it. We have finalized
> > > the doc here[3] Please vote on it if you agree with the content of the
> > doc [3].
> >
> >
> > +1 nb
> >
> > the doc is good. it adds value for users, and does not need to be
> perfect.
> > thanks Sumanth for doing this.
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >
>