Re: Evolving the client protocol

[Eric Stevens]

Let met just say that as an observer to this conversation -- and someone
who believes that compatibility, extensibility, and frankly competition
bring out the best in products -- I'm fairly surprised and disappointed
with the apparent hostility many community members have shown toward a
sincere attempt by another open source product to find common ground here.

Yes, Scylla has a competing OSS project (albeit under a different
license).  They also have a business built around it.  It's hard for me to
see that as dramatically different than the DataStax relationship to this
community.  Though I would love to be shown why.

Re: secondary index table - tombstones surviving compactions

[Eric Stevens]

The answer to Question 3 is "yes."  One of the more subtle points about
tombstones is that Cassandra won't remove them during compaction if there
is a bloom filter on any SSTable on that replica indicating that it
contains the same partition (not primary) key.  Even if it is older than
gc_grace, and would otherwise be a candidate for cleanup.

If you're recycling partition keys, your tombstones may never be able to be
cleaned up, because in this scenario there is a high probability that an
SSTable not involved in that compaction also contains the same partition
key, and so compaction cannot have confidence that it's safe to remove the
tombstone (it would have to fully materialize every record in the
compaction, which is too expensive).

In general it is an antipattern in Cassandra to write to a given partition
indefinitely for this and other reasons.

On Fri, May 18, 2018 at 2:37 AM Roman Bielik <
roman.bie...@openmindnetworks.com> wrote:

> Hi,
>
> I have a Cassandra 3.11 table (with compact storage) and using secondary
> indices with rather unique data stored in the indexed columns. There are
> many inserts and deletes, so in order to avoid tombstones piling up I'm
> re-using primary keys from a pool (which works fine).
> I'm aware that this design pattern is not ideal, but for now I can not
> change it easily.
>
> The problem is, the size of 2nd index tables keeps growing (filled with
> tombstones) no matter what.
>
> I tried some aggressive configuration (just for testing) in order to
> expedite the tombstone removal but with little-to-zero effect:
> COMPACTION = { 'class':
> 'LeveledCompactionStrategy', 'unchecked_tombstone_compaction': 'true',
> 'tombstone_compaction_interval': 600 }
> gc_grace_seconds = 600
>
> I'm aware that perhaps Materialized views could provide a solution to this,
> but I'm bind to the Thrift interface, so can not use them.
>
> Questions:
> 1. Is there something I'm missing? How come compaction does not remove the
> obsolete indices/tombstones from 2nd index tables? Can I trigger the
> cleanup manually somehow?
> I have tried nodetool flush, compact, rebuild_index on both data table and
> internal Index table, but with no result.
>
> 2. When deleting a record I'm deleting the whole row at once - which would
> create one tombstone for the whole record if I'm correct. Would it help to
> delete the indexed columns separately creating extra tombstone for each
> cell?
> As I understand the underlying mechanism, the indexed column value must be
> read in order a proper tombstone for the index is created for it.
>
> 3. Could the fact that I'm reusing the primary key of a deleted record
> shortly for a new insert interact with the secondary index tombstone
> removal?
>
> Will be grateful for any advice.
>
> Regards,
> Roman
>
> --
>  
>  
> 
>   
>

TWCS - Disabling Tombstone Compactions for TWCS

[Eric Stevens]

I'm trying to understand the reasoning behind this stanza in Time Windowed
Compaction Strategy's init:
https://github.com/apache/cassandra/blob/cassandra-3.0.15/src/java/org/apache/cassandra/db/compaction/TimeWindowCompactionStrategy.java#L63-L69

Reposted (and slightly reformatted) here for convenience:
if (

!options.containsKey(AbstractCompactionStrategy.TOMBSTONE_COMPACTION_INTERVAL_OPTION)
&&

!options.containsKey(AbstractCompactionStrategy.TOMBSTONE_THRESHOLD_OPTION)
)
{
disableTombstoneCompactions = true;
logger.debug("Disabling tombstone compactions for TWCS");
}
else
logger.debug("Enabling tombstone compactions for TWCS");

If you have left both tombstone compaction interval, and tombstone
threshold at default values, you are *opting out* of tombstone compactions?

This was certainly a surprising side effect to us, as we expected unset
values to have the same exact effect as setting those values to their
defaults.

I'm wondering if someone can shed some more light on this, or if this
should be considered a bug (it seems really intentional).

Re: TWCS - Disabling Tombstone Compactions for TWCS

[Eric Stevens]

That's helpful background, thanks Lerh!  It does seem intentional.

I'd suggest that this violates principal of least surprise (unset defaults
should have the same effect as setting same to default values), and should
be worth calling out as a separate option for both DTCS and TWCS.

On Mon, Jun 4, 2018 at 6:02 PM Lerh Chuan Low  wrote:

> Hi Eric,
>
> I think it has something to do with the same reason it is disabled in DTCS:
> https://issues.apache.org/jira/browse/CASSANDRA-9234 can shed more light.
>
>
>
> On 5 June 2018 at 09:02, Eric Stevens  wrote:
>
> > I'm trying to understand the reasoning behind this stanza in Time
> Windowed
> > Compaction Strategy's init:
> > https://github.com/apache/cassandra/blob/cassandra-3.0.
> > 15/src/java/org/apache/cassandra/db/compaction/
> > TimeWindowCompactionStrategy.java#L63-L69
> >
> > Reposted (and slightly reformatted) here for convenience:
> > if (
> >
> > !options.containsKey(AbstractCompactionStrategy.
> > TOMBSTONE_COMPACTION_INTERVAL_OPTION)
> > &&
> >
> > !options.containsKey(AbstractCompactionStrategy.
> > TOMBSTONE_THRESHOLD_OPTION)
> > )
> > {
> > disableTombstoneCompactions = true;
> > logger.debug("Disabling tombstone compactions for TWCS");
> > }
> > else
> > logger.debug("Enabling tombstone compactions for TWCS");
> >
> > If you have left both tombstone compaction interval, and tombstone
> > threshold at default values, you are *opting out* of tombstone
> compactions?
> >
> > This was certainly a surprising side effect to us, as we expected unset
> > values to have the same exact effect as setting those values to their
> > defaults.
> >
> > I'm wondering if someone can shed some more light on this, or if this
> > should be considered a bug (it seems really intentional).
> >
>

Re: Compaction Filter in Cassandra

[Eric Stevens]

We have been working on filtering compaction for a month or so (though we
call it deleting compaction, its implementation is as a filtering
compaction strategy).  The feature is nearing completion, and we have used
it successfully in a limited production capacity against DSE 4.8 series.

Our use case is that our records are written anywhere between a month, up
to several years before they are scheduled for deletion.  Tombstones are
too expensive, as we have tables with hundreds of billions of rows.  In
addition, traditional TTLs don't work for us because our customers are
permitted to change their retention policy such that already-written
records should not be deleted if they increase their retention after the
record was written (or vice versa).

We can clean up data more cheaply and more quickly with filtered compaction
than with tombstones and traditional compaction.  Our implementation is a
wrapper compaction strategy for another underlying strategy, so that you
can have the characteristics of whichever strategy makes sense in terms of
managing your SSTables, while interceding and removing records during
compaction (including cleaning up secondary indexes) that otherwise would
have survived into the new SSTable.

We are hoping to contribute it back to the community, so if you'd be
interested in helping test it out, I'd love to hear from you.

On Sat, Mar 12, 2016 at 5:12 AM Marcus Eriksson  wrote:

> We don't have anything like that, do you have a specific use case in mind?
>
> Could you create a JIRA ticket and we can discuss there?
>
> /Marcus
>
> On Sat, Mar 12, 2016 at 7:05 AM, Dikang Gu  wrote:
>
>> Hello there,
>>
>> RocksDB has the feature called "Compaction Filter" to allow application
>> to modify/delete a key-value during the background compaction.
>> https://github.com/facebook/rocksdb/blob/v4.1/include/rocksdb/options.h#L201-L226
>>
>> I'm wondering is there a plan/value to add this into C* as well? Or is
>> there already a similar thing in C*?
>>
>> Thanks
>>
>> --
>> Dikang
>>
>>
>

Re: Cassandra Java Driver and DataStax

[Eric Stevens]

> A key portion of your software’s stack, a client driver to use it, exists
outside of Apache in separate communities. This is an inherent risk to the
project.

That's not at all obvious to me.  The driver you're concerned about is not
under ASF, but it is Apache licensed, if DataStax took it in a direction
unfavorable to the community, the community would be able to just fork it.
Your concern here seems mostly to be surrounding one project vs two.  At
worst this is a layout concern.

> I would ask you all this - has there been a PR or patch in the past year
or two that wasn’t singularly reviewed by DataStax committers and PMC?

There are numerous non-PMC non-DataStax committers.  Regardless, this is
the wrong question to ask.  The question is not how well reviewed the code
is and whether there are a tight set of gate keepers, but rather whether
there are contributions being rejected which would advance the product in a
material way, but are being rejected by reviewers due to, for example, a
conflict of interest.

> Does anyone in the community see this “controlling” behavior going on?
... Thanks for any help you can provide in rooting this out.
> ...
> Unfortunately it seems like there is a bit of control going on...There is
definitely some control going on

You asked the community for feedback on this, and it was pretty clear to me
that nobody here felt like that was the case.

If you're aware of *actual* examples of impropriety in this or another
sense, I think the community would like to hear about it.  Something more
substantial than vague assertions and hand waving.  However, claiming
repeatedly that you see control going on, but without something to
substantiate the accusation feels like an expedition fishing for drama.

On Mon, Jun 6, 2016 at 5:36 AM Jeremiah D Jordan 
wrote:

> The Apache Cassandra project has always left development of its drivers up
> to the community.  The DataStax Java Driver is not part of the Apache
> Cassandra project, it is an open source project created by DataStax.  You
> can find a large list of drivers for Cassandra here:
> https://wiki.apache.org/cassandra/ClientOptions some of them developed by
> DataStax, some developed by Netflix, and many others.
>
> -Jeremiah
>
> > On Jun 3, 2016, at 9:29 PM, Chris Mattmann  wrote:
> >
> > Hi All,
> >
> > I’m investigating something a few ASF members contacted
> > me about and pointed out, so I’m hoping you can help
> > guide me here as a community. I have heard that a company,
> > DataStax, whose marketing material mentions it as the only
> > Cassandra vendor, “controls” the Java Driver for Apache
> > Cassandra.
> >
> > Of course, no company “controls” our projects or its code,
> > so I told the folks that mentioned it to me that I’d investigate
> > with my board hat on.
> >
> > I’d like to hear the community’s thoughts here on this. Does
> > anyone in the community see this “controlling” behavior going
> > on? Please speak up, as I’d like to get to the bottom of it,
> > and I’ll be around on the lists, doing some homework and reading
> > up on the archives to see what’s up.
> >
> > Thanks for any help you can provide in rooting this out.
> >
> > Cheers,
> > Chris
> >
> >
> >
> >
>
>

Re: NewBie Question ~ Book for Cassandra

[Eric Stevens]

> Where else to discuss it? It should be constructive, yes. But we are all
> here to learn and grow with Cassandra.  The original question is a good
> one. The direction it took was poor.

Agreed, hijacking unrelated threads is not good netizenry.  There's
obviously a lot of strong opinions here, but we should keep discussions
about overall project health from bleeding into other conversations.

On Tue, Jun 14, 2016 at 2:42 AM Benjamin Lerer 
wrote:

> Hi Chris,
>
> Thanks for your advice.
>
> However, according to the Apache Way:
>
> " PMCs are expected to follow corporate policies in terms of licensing,
> branding, infrastructure and so on, and are expected to manage their
> projects independently using The Apache Way. PMCs are tasked with all other
> aspects of project management, especially technical direction. "
>
> and according to the Bylaws of The Apache Software Foundation:
>
> "Subject to the direction of the Board of Directors, the chairman of each
> Project Management Committee shall be primarily responsible for project(s)
> managed by such committee, and he or she shall establish rules and
> procedures for the day to day management of project(s) for which the
> committee is responsible."
>
> which appears to me to be exactly what Jonathan is doing.
>
> Concerning your question below:
>
> > However also see that besides the current documentation, there needs to
> be
> > a roadmap for making Apache Cassandra and *its* documentation (not
> *DataStax’s*)
> > up to par for a basic user to build, deploy and run Cassandra. I don’t
> think that’s
> > the current case, is it?
>
> The current documentation is indeed not complete but covers far more than
> the basics.
>
>
> On Mon, Jun 13, 2016 at 3:05 PM, Mattmann, Chris A (3980) <
> chris.a.mattm...@jpl.nasa.gov> wrote:
>
> > Hi Benjamin,
> >
> >
> >
> > On 6/13/16, 6:38 AM, "Benjamin Lerer" 
> wrote:
> >
> > >Hi Chris,
> > >
> > >Disclaimer: I am a Datastax employee
> > >
> > >It is clear to me that the current official documentation is far from
> > being
> > >enough. That's why I fully support the decision made by Jonathan to do
> our
> > >best to improve it.
> >
> > Just as a small piece of advice - it seems like Jonathan is the “boss” of
> > this
> > project. I’ve spoken with him here and there - he’s a great guy don’t get
> > me
> > wrong - but Apache projects don’t have bosses. He is the chair of the
> > project -
> > that earns him the great glory having to write a board report every month
> > after
> > the project is created, and quarterly thereafter. The chair is expected
> to
> > be
> > the eyes and ears of the project for the board. The project has a
> “Project
> > Management Committee (PMC) or PMC” responsible jointly for stewarding the
> > project. There is also a “Committer” role at the ASF. Some communities
> > define
> > PMC == C. The committer role does not have a binding VOTE on releases of
> > the
> > software and/or on additions of new personnel to the project.
> >
> > The reason I pointed this out and it may have just been me misreading but
> > it sounded like you suggested something like: Jonathan makes decision for
> > the project; you all jump. And I am just saying I hope that’s not the
> case.
> > You all should have equal decision making ability in the project
> especially
> > on the PMC.
> >
> > >
> > >As an Apache Cassandra Committer mostly working on the CQL layer, I know
> > >that we have done our best to keep the CQL documentation up to date
> > >(https://cassandra.apache.org/doc/cql3/CQL-3.0.html). Now, English not
> > >being the native language of some of us, and as we are not technical
> > >writers, I would not really be surprised if some external persons have
> > done
> > >a better job than us.
> > >
> > >I think our goal should be to provide an accurate and reliable
> > >documentation for the project.
> >
> > I would amend the above to add “for the project[at the ASF]”. That’s
> > the thing - as a *first* (and not *second*) though, the ASF project
> > should be getting careful attention and that includes the documentation.
> >
> >
> > > Nevertheless, it seems legitimate to me to
> > >also provide links to external documentations, when people are asking
> for
> > >it, if others did a better job than us.
> >
> > Sure, this happens in some projects from time to time. When there isn’t
> > a perception of control, it is possible to do this, especially if
> > coinciding
> > with the external links there is some roadmap or some plan for actually
> > keeping
> > the ASF documentation up to date. Real data point here - I wrote a book
> > about
> > Apache Tika, Tika in Action. This was done, with frequent updates on
> what’s
> > going on to d...@tika.apache.org. Over time, eventually we worked with
> > Manning
> > Publications to donate the code samples and examples from the book to the
> > Apache
> > Tika project. Much of the book inspiration and examples made it into
> > Apache Tika
> > in parallel to the goings-on outside.

Re: NewBie Question

[Eric Stevens]

The file format is SSTable:
http://wiki.apache.org/cassandra/ArchitectureSSTable

If you're getting into byte-level detail, I highly recommend you
familiarize yourself with the read and/or write path first, because that
deep in the bowels there are some non-obvious things going on where
Cassandra differs from other database storage formats you might already be
familiar with:
https://wiki.apache.org/cassandra/ReadPathForUsers
https://wiki.apache.org/cassandra/WritePathForUsers

On Wed, Jun 15, 2016 at 4:30 AM Jonathan Ellis  wrote:

> It's a little more involved than that.  I suggest inserting a single row in
> a test table, then looking at the sstabledump output as a first step, then
> compare with two rows in a single partition.  Then you can code dive to see
> what sstabledump is actually doing if you really need the byte-level
> detail.
>
> On Wed, Jun 15, 2016 at 9:30 AM, Deepak Goel  wrote:
>
> > Hey
> >
> > Namaskara~Nalama~Guten Tag~Bonjour
> >
> > I tried searching for the fileformat of how cassandra stores its data,
> but
> > I couldn't find any...
> >
> > Suppose I have a database structure of the following format:
> >
> > RowID: Name:Age
> > 1: Deepak : 33
> > 2: Deepak1:34
> > 3: Deepak2:35
> >
> > How would this data actually stored in the data file of Cassandra?
> >
> > Would it be something like this:
> > Deepak:1:33
> > Deepak1:2:34
> > Deepak2:3:35
> >
> > Or, would it be:
> > 1:Deepak :33
> > 2:Deepak1:34
> > 3:Deepak2:35
> >
> > Thanks
> > Deepak
> >
> >
> >
> >--
> > Keigu
> >
> > Deepak
> > 73500 12833
> > www.simtree.net, dee...@simtree.net
> > deic...@gmail.com
> >
> > LinkedIn: www.linkedin.com/in/deicool
> > Skype: thumsupdeicool
> > Google talk: deicool
> > Blog: http://loveandfearless.wordpress.com
> > Facebook: http://www.facebook.com/deicool
> >
> > "Contribute to the world, environment and more :
> > http://www.gridrepublic.org
> > "
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder, http://www.datastax.com
> @spyced
>

Re: Guidance

[Eric Stevens]

Hi Melinda, these questions are probably better suited for the users group
rather than the dev group (which concerns itself with database internals
and feature development).  I'd recommend reposting over there.

On Mon, Jul 4, 2016 at 6:16 AM Melinda Zoe 
wrote:

> Hello Group,
>
> I’ve been following the group for a while and I’m ready to take a stab at
> Cassandra. I have two questions as I plan everything out:
>
> 1.If I create the database on Mac OS, what is the process for
> migrating it to another environment like Windows? This concerns me because
> I’m not sure who will host me later. Is it as simple as copying and pasting
> the database and moving all of the files over or will the database need to
> be converted?
> 2.I want to use Cassandra for a website that will rely heavily on
> content that I provide the user. I plan on uploading content daily. Have
> any of you created a template for uploading records into Cassandra or have
> any of you created a script for loading multiple records at a time?
>
> Melinda

Re: Guidance

[Eric Stevens]

Apache mailing lists can be found at:
http://mail-archives.apache.org/mod_mbox/
Instructions on subscribing or unsubscribing to specific lists are
available at: http://apache.org/foundation/mailinglists.html

On Tue, Jul 5, 2016 at 10:26 AM Melinda 
wrote:

> Ok great; thanks Eric.
>
> Melinda
>
> Sent from the Best-est iPhone
>
> > On Jul 5, 2016, at 10:27 AM, Eric Stevens  wrote:
> >
> > Hi Melinda, these questions are probably better suited for the users
> group
> > rather than the dev group (which concerns itself with database internals
> > and feature development).  I'd recommend reposting over there.
> >
> > On Mon, Jul 4, 2016 at 6:16 AM Melinda Zoe  >
> > wrote:
> >
> >> Hello Group,
> >>
> >> I’ve been following the group for a while and I’m ready to take a stab
> at
> >> Cassandra. I have two questions as I plan everything out:
> >>
> >> 1.If I create the database on Mac OS, what is the process for
> >> migrating it to another environment like Windows? This concerns me
> because
> >> I’m not sure who will host me later. Is it as simple as copying and
> pasting
> >> the database and moving all of the files over or will the database need
> to
> >> be converted?
> >> 2.I want to use Cassandra for a website that will rely heavily on
> >> content that I provide the user. I plan on uploading content daily. Have
> >> any of you created a template for uploading records into Cassandra or
> have
> >> any of you created a script for loading multiple records at a time?
> >>
> >> Melinda
>
>

Re: A proposal to move away from Jira-centric development

[Eric Stevens]

There are a few strengths of discussion on the ticketing system over
mailing lists.  Mailing lists were fundamentally designed in the 1970's and
early 1980's, and the state of the art from a user experience perspective
has barely advanced since then.

* Mailing lists tend to end up with fragmented threads for large
discussions, subject changes, conversation restarts, topic forks, and
simple etiquette errors - all of which can make it very difficult to locate
the entire discussion related to a feature.   There is no single source
that an interested party can study thoroughly to understand the entire
conversation, rather it's more of a scavenger hunt with no way to be
certain you've covered all the territory.  8844 for example would have
ended up being numerous parallel threads as people forked the conversation
to have side discussions or offer alternatives, there's no way such a
ticket would ever have simply been a single massive email thread with no
forks.

* Mailing lists don't allow for selective subscription.  If I find a ticket
interesting, I can watch the ticket and follow along. Conversely and more
importantly if I find it uninteresting I don't have to wade through that
discussion as it progresses.  If I think I want to follow all tickets, that
should be possible too.  Likewise if I want to watch tickets that involve
certain components, certain milestones, certain labels, or even certain
contributors, I can create a subscription for such, and get emails
accordingly.  I can also subscribe to RSS feeds and add them to my news
reader if I prefer that approach better.  A tremendous amount of control is
given to the user over what they want to see, and how they want to see it.

* The concern that Chris voiced about having to open a web browser to
participate is actually not true unless Apache's Jira install is not well
configured.  If you reply to an email notification from Jira it should
appear as a comment on the ticket.  It shouldn't exclude anyone (even those
who want to participate but somehow can't be motivated to create an account
in the ticketing system, but who _could_ be bothered to figure out the
arcane mailing list subscription incantation).

* Permalinking conversations is an important capability.  It's possible
with a mailing list, but it's nontrivial, when you want to create that
permalink, you must first locate the discussion in the nonprimary interface
(the online archives), which involves a lot more effort.  Historically
we've also seen existing "permalinks" become invalidated with mailing list
archive software is switched or upgraded.  This leads to the next point:

* One of the simple but hugely valuable features of Jira is the short
memorable ticket numbers.  Several people in this thread have mentioned
8844.  Those who care about that conversation know that ID by heart.  And
in casual conversation if you want to bring someone's attention to an
issue, you can mention it by ID without having to try to remember what the
original thread subject was so the other participant can also hopefully
remember and maybe locate it later.  Write the number down on a napkin and
you _will_ find the issue, and know it's the right one, and not some
similar but unrelated conversation.

* Ticketing systems can maintain a summarized version of the conversation
in the ticket's description as a shortcut for those who want to know the
current state without having to read potentially months of back history to
catch up (the event log model).  Event logs are a great way to capture
changing state, but they're horridly inefficient if your only option is to
start from 0 and replay the entire log, particularly when a lot of the
contributors are as long winded as I am.

On Mon, Aug 15, 2016 at 5:29 PM Jonathan Ellis  wrote:

> ... but it's important to note that if we take this approach, we need to be
> careful not to just summarize the conclusion of the discussion, but also
> approaches that were examined and found to be unviable, and why.  Otherwise
> people looking at the ticket will have to cross reference back to a much
> harder-to-follow discussion on the list archives.
>
> On Mon, Aug 15, 2016 at 6:26 PM, Jonathan Ellis  wrote:
>
> > On Mon, Aug 15, 2016 at 6:18 PM, Josh McKenzie 
> > wrote:
> >
> >> 2: 8844 would have been a great candidate for being discussed on the
> >> mailing list rather than on JIRA. While I made it a point to front-load
> >> design, we still ran into some unforeseen consequences from the design
> >> that
> >> might have been prevented by more wide-spread discussion. In my opinion,
> >> it
> >> would have made sense to have the initial discussion(s) take place on
> the
> >> mailing list until a design had settled out, worked that design and the
> >> day-to-day back and forth on JIRA, then bringing it back to the mailing
> >> list when we ran into the problems with the design.
> >>
> >
> > This is a good example of what I had in mind here.
> >
> > --
> > Jonathan Ellis
> > Project Chair, 

Re: A proposal to move away from Jira-centric development

[Eric Stevens]

I agree with Benedict that we really shouldn't be getting into a legalese
debate on this subject, however "it didn't happen" has been brought up as a
hammer in this conversation multiple times, and I think it's important that
we put it to rest.  It's pretty clear cut that projects are free to
disregard this advice.  "It didn't happen" is a motto, not a rule.

Per ASF newbie FAQ referenced by someone else earlier [1]:

> The section on ASF Mottos is especially useful as a reminder of the way
things are in most ASF projects. This section includes such gems as:
> * Put community before code.
> * Let they that do the work make the decisions.
> * If it didn't happen on a mailing list, it didn't happen.
> * Don't feed the trolls.

This is presented as a general guideline and not a hard rule, and as
Benedict points out even this is preceded by a guideline suggesting that
developers are free to seek alternatives.

Now since this is just a reference to the Incubator code of conduct's list
of mottos (again, not ASF policy), which best source I could find [2],
mirrors the newbie FAQ, but provides the additional insight that the
objective of the motto is transparency.  The spirit of this motto is not
meant to dictate a technology choice, but merely to indicate that
discussions should happen in open spaces where all are able to
participate.  The motto was authored in a time when "the lists" was the
only real option.

Jira absolutely meets the design goal of that motto, if that's the
direction the community chooses, and it's clear from both sources that
individual communities (they that do the work) are free to find the path
here that's best for them.

[1]
https://community.apache.org/newbiefaq.html#NewbieFAQ-IsthereaCodeofConductforApacheprojects
?
[2] *https://wiki.apache.org/incubator/CodeOfConduct#ASF_Mottos
*

On Tue, Aug 16, 2016 at 5:57 AM James Carman 
wrote:

> On Mon, Aug 15, 2016 at 10:23 AM Jonathan Ellis  wrote:
>
> > A long time ago, I was a proponent of keeping most development
> discussions
> > on Jira, where tickets can be self contained and the threadless nature
> > helps keep discussions from getting sidetracked.
> >
> > But Cassandra was a lot smaller then, and as we've grown it has become
> > necessary to separate out the signal (discussions of new features and
> major
> > changes) from the noise of routine bug reports.
> >
> > I propose that we take advantage of the dev list to perform that
> > separation.  Major new features and architectural improvements should be
> > discussed first here, then when consensus on design is achieved, moved to
> > Jira for implementation and review.
> >
> >
> +1!  I think it's important to point out here that nobody is proposing that
> folks have to send an email like:
>
> "I was thinking of naming my variable 'foo' here, what do you guys think?"
>
> However, discussions and decisions that have an impact on Cassandra and its
> direction/architecture (not an all-inclusive list here and we should use
> reason to decide) should happen on the mailing list.  The idea here is
> inclusiveness.  We want everyone in the community to have a chance to
> contribute to these discussions.
>

Re: Batch read requests to one physical host?

[Eric Stevens]

We've had some luck with bulk known key reads with grouping by replica and
doing SELECT... WHERE key IN(...). Not compatible with all data models, but
it works well where we can get away with it.

As a more general purpose construct it makes sense to me. In our driver
layer we have abstracted batches to support read batches (under which the
above method is applied) even though Cassandra doesn't support it first
class.

On Tue, Oct 18, 2016, 5:00 PM Dikang Gu  wrote:

> Hi there,
>
> We have couple use cases that are doing fanout read for their data, means
> one single read request from client contains multiple keys which live on
> different physical hosts. (I know it's not recommended way to access C*).
>
> Right now, on the coordinator, it will issue separate read commands even
> though they will go to the same physical host, which I think is causing a
> lot of overheads.
>
> I'm wondering is it valuable to provide a new read command, that
> coordinator can batch the reads to one datanode, and send to it in one
> message, and datanode will return the results for all keys belong to it?
>
> Any similar ideas before?
>
>
> --
> Dikang
>

Re: Schema help needed

[Eric Stevens]

Hi Sagar, you'll probably do better to ask this question on the users group
rather than the dev group.  Dev groups are about internal development while
users groups are about using the software.

On Sat, Dec 17, 2016 at 5:08 AM Sagar Jambhulkar 
wrote:

Hi,
Needed a suggestion for a schema query. I want to build a reconciliation
using Cassandra. Basically two or more systems send message to a
reconciliation process. The reconciliation process first does a level one
match of id's and than does complete comparison of messages.

The best I could think of is a queue table with id's. My consumer thread
would, poll this table, create a pair and would have to delete from this
table. But this is a known anti pattern to not use Cassandra as a queue
causing tombstones etc.
But I could not think of any other way. Does anyone have any other
suggestion so as to not delete after a pair is crated. Is Cassandra not the
correct technology for a recon process?

Thanks,
Sagar

How does a node know it's fully joined to the ring?

[Eric Stevens]

I'm working on a bug report in my Deleting Compaction Strategy project.

Some of the provided deletion strategies allow for a series of deletion
rules to be added to a CQL table, before compaction begins, a snapshot of
the current rules are read out of the CQL table and a deletion plan is
produced that applies for the life of that compaction.

The bug report (at https://github.com/protectwise/cassandra-util/issues/2)
is that if compaction begins while a node is first starting up, before it
has joined the ring, the attempt to read the rules will fail with an
AssertionError (it's an attempt to call QueryProcessor.process at local
quorum before the node has joined the ring).

My question is, how can I detect when a node has fully joined the ring, and
it's safe to call QueryProcessor.process without just trying, failing, and
catching (and even then how would I know that it's because the node is
still starting up vs some other source of problems)?  I had thought that
something might set a flag during the startup process to indicate that the
node thinks it is fully started up, but as far as I see nothing like that
is tracked.

I took a stab at what this could look like here:
https://github.com/protectwise/cassandra-util/blob/93dfb02157446e2e37427f1d4846c6039cc97099/deleting-compaction-strategy/src/main/java/com/protectwise/cassandra/retrospect/deletion/QueryHelper.java#L31-L36

This approach seems cumbersome and fragile, and I feel like I'm missing
something.  How else could I tell whether it's safe to call out to
QueryProcessor.process yet?

Thanks in advance,
-e

Re: How does a node know it's fully joined to the ring?

[Eric Stevens]

Thanks for the quick response.

Looking at it, I'm not sure if it's quite the right thing for me.  Maybe I
used the wrong terminology; it actually doesn't matter if the node is still
bootstrapping, as long as it could query other nodes in the cluster.

On Thu, Jan 5, 2017 at 1:03 PM Brandon Williams  wrote:

> Probably want to do something like SystemKeyspace.bootstrapComplete()
>
> On Thu, Jan 5, 2017 at 1:58 PM, Eric Stevens  wrote:
>
> > I'm working on a bug report in my Deleting Compaction Strategy project.
> >
> > Some of the provided deletion strategies allow for a series of deletion
> > rules to be added to a CQL table, before compaction begins, a snapshot of
> > the current rules are read out of the CQL table and a deletion plan is
> > produced that applies for the life of that compaction.
> >
> > The bug report (at
> https://github.com/protectwise/cassandra-util/issues/2)
> > is that if compaction begins while a node is first starting up, before it
> > has joined the ring, the attempt to read the rules will fail with an
> > AssertionError (it's an attempt to call QueryProcessor.process at local
> > quorum before the node has joined the ring).
> >
> > My question is, how can I detect when a node has fully joined the ring,
> and
> > it's safe to call QueryProcessor.process without just trying, failing,
> and
> > catching (and even then how would I know that it's because the node is
> > still starting up vs some other source of problems)?  I had thought that
> > something might set a flag during the startup process to indicate that
> the
> > node thinks it is fully started up, but as far as I see nothing like that
> > is tracked.
> >
> > I took a stab at what this could look like here:
> > https://github.com/protectwise/cassandra-util/blob/
> > 93dfb02157446e2e37427f1d4846c6039cc97099/deleting-
> > compaction-strategy/src/main/java/com/protectwise/
> > cassandra/retrospect/deletion/QueryHelper.java#L31-L36
> >
> > This approach seems cumbersome and fragile, and I feel like I'm missing
> > something.  How else could I tell whether it's safe to call out to
> > QueryProcessor.process yet?
> >
> > Thanks in advance,
> > -e
> >
>

Re: How does a node know it's fully joined to the ring?

[Eric Stevens]

Perfect, thanks!

On Thu, Jan 5, 2017 at 2:26 PM Brandon Williams  wrote:

> We don't bind thrift/CQL until bootstrap is complete, so that's not
> possible.  But, if you want to be sure, you can check
> StorageService.isRPCServerRunning and isNativeTransportRunning.
>
> On Thu, Jan 5, 2017 at 3:17 PM, Eric Stevens  wrote:
>
> > Thanks for the quick response.
> >
> > Looking at it, I'm not sure if it's quite the right thing for me.  Maybe
> I
> > used the wrong terminology; it actually doesn't matter if the node is
> still
> > bootstrapping, as long as it could query other nodes in the cluster.
> >
> > On Thu, Jan 5, 2017 at 1:03 PM Brandon Williams 
> wrote:
> >
> > > Probably want to do something like SystemKeyspace.bootstrapComplete()
> > >
> > > On Thu, Jan 5, 2017 at 1:58 PM, Eric Stevens 
> wrote:
> > >
> > > > I'm working on a bug report in my Deleting Compaction Strategy
> project.
> > > >
> > > > Some of the provided deletion strategies allow for a series of
> deletion
> > > > rules to be added to a CQL table, before compaction begins, a
> snapshot
> > of
> > > > the current rules are read out of the CQL table and a deletion plan
> is
> > > > produced that applies for the life of that compaction.
> > > >
> > > > The bug report (at
> > > https://github.com/protectwise/cassandra-util/issues/2)
> > > > is that if compaction begins while a node is first starting up,
> before
> > it
> > > > has joined the ring, the attempt to read the rules will fail with an
> > > > AssertionError (it's an attempt to call QueryProcessor.process at
> local
> > > > quorum before the node has joined the ring).
> > > >
> > > > My question is, how can I detect when a node has fully joined the
> ring,
> > > and
> > > > it's safe to call QueryProcessor.process without just trying,
> failing,
> > > and
> > > > catching (and even then how would I know that it's because the node
> is
> > > > still starting up vs some other source of problems)?  I had thought
> > that
> > > > something might set a flag during the startup process to indicate
> that
> > > the
> > > > node thinks it is fully started up, but as far as I see nothing like
> > that
> > > > is tracked.
> > > >
> > > > I took a stab at what this could look like here:
> > > > https://github.com/protectwise/cassandra-util/blob/
> > > > 93dfb02157446e2e37427f1d4846c6039cc97099/deleting-
> > > > compaction-strategy/src/main/java/com/protectwise/
> > > > cassandra/retrospect/deletion/QueryHelper.java#L31-L36
> > > >
> > > > This approach seems cumbersome and fragile, and I feel like I'm
> missing
> > > > something.  How else could I tell whether it's safe to call out to
> > > > QueryProcessor.process yet?
> > > >
> > > > Thanks in advance,
> > > > -e
> > > >
> > >
> >
>

Re: Cassandra on RocksDB experiment result

[Eric Stevens]

> Right now all the compaction strategies share the assumption that the data
> structure and layout on disk is fixed. With pluggable storage engine, we
> need to special case each compaction strategy (or at least the Abstract
> class of compaction strategy) for each engine.

As Ben points out, compaction should be treated as _part_ of the storage
layer.  The need to engage in compaction is a consequence of immutable data
stores behind a mutable interface.  A different storage layer might require
a different approach or might not even require compaction in the same sense.

On Wed, Apr 19, 2017 at 4:01 PM Ben Bromhead  wrote:

> This looks super cool would love to see more details.
>
> On a general note, a pluggable storage layer allows other storage engines
> (and possibly datastores) to leverage Cassandras distributed primitives
> (dynamo, gossip, paxsos?, drivers, cql etc). This could allow Cassandra to
> fill similar use cases as Dynomite from Netflix.
>
> Also as Sankalp mentioned we get some other benefits including better
> testability.
>
> In my experience with pluggable storage engines (in the MySQL world), the
> > engine manages all storage that it "owns." The higher tiers in the
> > architecture don't need to get involved unless multiple storage engines
> > have to deal with compaction (or similar) issues over the entire
> database,
> > e.g., every storage engine has read/write access to every piece of data,
> > even if that data is owned by another storage engine.
> >
> > I don't know enough about Cassandra internals to have an opinion as to
> > whether or not the above scenario makes sense in the Cassandra context.
> But
> > "sharing" (processes or data) between storage engines gets pretty hairy,
> > easily deadlocky (!), even in something as relatively straightforward as
> > MySQL.
>
>
> This would be an implementation detail, but given that tables in Cassandra
> don't know about each other (no joins, foreign keys etc... ignore mv for
> the moment), but storage engine interactions probably wouldn't be an issue.
>
>
> > This was a long and old debate we had several times in the past. One of
> > the difficulty of pluggable storage engine is that we need to manage the
> > differences between the LSMT of native C* and RockDB engine for
> compaction,
> > repair, streaming etc...
> >
> > Right now all the compaction strategies share the assumption that the
> data
> > structure and layout on disk is fixed. With pluggable storage engine, we
> > need to special case each compaction strategy (or at least the Abstract
> > class of compaction strategy) for each engine.
>
>
> > The current approach is one storage engine, many compaction strategies
> for
> > different use-cases (TWCS for time series, LCS for heavy update...).
> >
> > With pluggable storage engine, we'll have a matrix of storage engine x
> > compaction strategies.
> >
>
> Compaction is part of the storage engine, and if I understand Dikangs
> design spec, it is bypassed?
>
> Cassandras currently storage engine is a log structured merge tree. RocksDB
> does it's own thing.
>
> Again this is an implementation detail about where the storage engine
> interface line is drawn, but from the above example compaction I think it
> is a non issue?
>
>
> > And not even mentioning the other operations to handle like streaming and
> > repair.
> >
>
> Streaming and repair would be the harder problem to solve than compaction
> imho.
> --
> Ben Bromhead
> CTO | Instaclustr 
> +1 650 284 9692 <(650)%20284-9692>
> Managed Cassandra / Spark on AWS, Azure and Softlayer
>

Re: Cassandra on RocksDB experiment result

[Eric Stevens]

In the spirit of what Eric mentions, as a community member, I'm
enthusiastically +1 on the idea.

On Fri, Apr 21, 2017 at 9:28 AM, Eric Evans 
wrote:

> On Fri, Apr 21, 2017 at 4:32 AM, benjamin roth  wrote:
> > I am not a PMC member or sth but just my 2 cents:
>
> Somewhat off-topic here, but I'd like to start discouraging people
> from prefacing remarks like this ("not a PMC member", "non-binding
> +1").  The exchange rate here is 1:1 IMO, your 2 cents are worth the
> same as any others! ;)
>
>
> --
> Eric Evans
> john.eric.ev...@gmail.com
>

Re: NGCC?

[Eric Stevens]

Trying to coordinate it around the eclipse seems neat on the surface, but
practically it's probably not a great idea as most major cities along the
eclipse line have largely sold out their hotel rooms for that event.

On Fri, Jun 2, 2017 at 3:13 PM Carlos Rolo  wrote:

> I might try to get some internal support for sponsorship if needed.
>
> Would love for a NGCC event to happen.
>
> Regards,
>
> Carlos Juzarte Rolo
> Cassandra Consultant / Datastax Certified Architect / Cassandra MVP
>
> Pythian - Love your data
>
> rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:
> *linkedin.com/in/carlosjuzarterolo
> *
> Mobile: +351 918 918 100 <+351%20918%20918%20100>
> www.pythian.com
>
> On Fri, Jun 2, 2017 at 9:28 PM, Ben Bromhead  wrote:
>
> > +1
> >
> > On Fri, 2 Jun 2017 at 13:17 Eric Evans 
> wrote:
> >
> > > On Fri, Jun 2, 2017 at 2:34 PM, Ben Bromhead 
> > wrote:
> > > > We are more than happy to donate some resources (both people and
> > > materials)
> > > > to putting on NGCC.
> > > >
> > > > I would suggest some sort of committee of folks who are willing to do
> > the
> > > > groundwork and who act as the executive so it gets done :)
> > >
> > > Gary Dusbabek and myself are both willing to shoulder the grunt work
> > > of organizing this.
> > >
> > > Since we're both in San Antonio, that's the location that would be
> > > most practical for us.  It's also easy to get to, centrally located,
> > > and the weather should be fantastic that time of year.
> > >
> > > We were thinking we'd put together some options for venue, and propose
> > > some dates, and circle back to the list in search of consensus.
> > >
> > > If this seems reasonable for now, then we'll get back to everyone with
> > > more info in a weeks time.
> > >
> > > --
> > > Eric Evans
> > > john.eric.ev...@gmail.com
> > >
> > --
> > Ben Bromhead
> > CTO | Instaclustr 
> > +1 650 284 9692 <(650)%20284-9692>
> > Managed Cassandra / Spark on AWS, Azure and Softlayer
> >
>
> --
>
>
> --
>
>
>
>