Re: How does the "batch" commit log sync works

That is the maximum length of time that queries may be batched together for, not the minimum. If there is a break in the flow of queries for the commit log, it will commit those outstanding immediately. It will anyway commit in clusters of commit log file size (default 32Mb). I know the documenta

Re: JVM safepoints, mmap, and slow disks

l going to read quite a bit extra. > > Ariel > > On Sun, Oct 9, 2016, at 05:39 AM, Benedict Elliott Smith wrote: > > The biggest problem with pread was the issue of over reading (reading 64k > where 4k would suffice), which was significantly improved in 2.2 iirc. I > don't

Re: JVM safepoints, mmap, and slow disks

The biggest problem with pread was the issue of over reading (reading 64k where 4k would suffice), which was significantly improved in 2.2 iirc. I don't think the penalty is very significant anymore, and if you are experiencing time to safe point issues it's very likely a worthwhile switch to flip.

Re: Cassandra data model right definition

so eager to deal with my pull request please review this one: > I would rather you review this pull request: https://issues. > apache.org/jira/browse/CASSANDRA-10825 > > > > > > On Mon, Oct 3, 2016 at 1:04 PM, Benedict Elliott Smith < > bened...@apache.org> wrote: > &

Re: Cassandra data model right definition

, and >>> semi-flexible collections doesn't disqualify it from calling it a "row >>> store" >>> >>> Postgres added flexible storage through hstore, I don't hear anyone >>> arguing that it needs to be renamed. >>> >>>

Re: Cassandra data model right definition

cribes a rowstore >>> >>> http://cs-www.cs.yale.edu/homes/dna/talks/abadi-sigmod08-slides.pdf >>> https://en.wikipedia.org/wiki/Column-oriented_DBMS#Row-oriented_systems >>> >>> Does it like this: >>> >>> 001:10,Smith,Joe,4; >>> 002:12,Jones,M

Re: Cassandra data model right definition

ide" > > The old sentence Cassandra statement was something to the effect of "with > the storage model of big table and the consistency model of dynamo". This > accurately described the system and gave reference to specific known > quantities (bigtable/dynamo) in which white

Re: Cassandra data model right definition

gt;>> <https://en.wikipedia.org/wiki/Row_(database)#cite_note-1> Each row in >>>> a table represents a set of related data, and every row in the table has >>>> the same structure. >>>> >>>> When you have static columns and rows with maps, a

Re: Cassandra data model right definition

ll use. > On Fri, Sep 30, 2016 at 1:53 PM DuyHai Doan wrote: > >> I used to present Cassandra as a NoSQL datastore with "distributed" >> table. This definition is closer to CQL and has some academic background >> (distributed hash table). >> >> >> On Fri

Re: Cassandra data model right definition

Cassandra is not a "wide column store" anymore. It has a schema. Only thrift users no longer think they have a schema (though they do), and thrift is being deprecated. I really wish everyone would kill the term "wide column store" with fire. It seems to have never meant anything beyond "schema-l

Re: External linking Was: Isolation in case of Single Partition Writes and Batching with LWT

Again: *dev list*. "Please Note: These requirements apply to Apache projects: that is, to *individual committer and PMC behaviors* and actions within the context of collaboratively building software products at The Apache Software Foundation. By definition here, "Apache project" means the collabor

Re: Partition size

4:16, Mark Thomas wrote: > On 12/09/2016 12:51, Benedict Elliott Smith wrote: > > Please tone down your language. There is no need for profanity. > > Now is probably a good time to remind everyone of the Apache Code of > Conduct: > http://www.apache.org/foundation/policies/conduct.

Re: Partition size

ir email content? On 12 September 2016 at 12:10, Mark Thomas wrote: > On 09/09/2016 21:11, Benedict Elliott Smith wrote: > > Come on. This kind of inconsistent 'policing' is not helpful. > > How is it inconsistent? Since I subscribed to the mailing list on 22 >

Re: Partition size

Come on. This kind of inconsistent 'policing' is not helpful. By all means, push the *committers* to improve the project docs as is happening, and to promote the internal resources over external ones. But Mark has absolutely no formal connection with the project, and his contributions have only b

Re: Isolation in case of Single Partition Writes and Batching with LWT

Yes, each partition modified by a batch has its modifications applied altogether, atomically (at the node level). On Friday, 9 September 2016, Bhuvan Rawal wrote: > As per this > doc > conditional batches can contain queries o

Re: Guidelines for configuring Thresholds for Cassandra metrics

t; point and it is often not the ideal once you go through more thorough > testing, all of which I said initially and I still think is a reasonable > statement. > > -regards, > > Ryan Svihla > > > > > > > On Sat, Aug 27, 2016 at 9:31 AM -0500, "Benedict

Re: Guidelines for configuring Thresholds for Cassandra metrics

e. I never intended that as a blanket recommendation, > just a starting one. > > _ > From: Benedict Elliott Smith > > Sent: Friday, August 26, 2016 9:40 AM > Subject: Re: Guidelines for configuring Thresholds for Cassandra metrics > To: > > > > The default when I wrote

Re: Guidelines for configuring Thresholds for Cassandra metrics

The default when I wrote it was 0.4 but it was found this did not saturate flush writers in JBOD configurations. Iirc it now defaults to 1/(1+#disks) which is not a terrible default, but obviously comes out much lower if you have many disks. This smaller value behaves better for peak performance,

Re: Flush activity and dropped messages

You should update from 2.0 to avoid this behaviour, is the simple answer. You are correct that when the commit log gets full the memtables are flushed to make room. 2.0 has several interrelated problems here though: There is a maximum flush queue length property (I cannot recall its name), and on

Re: how to force cassandra-stress to actually generate enough data

cassandra-stress has some (many) limitations - that I had planned to address now it's seeing wider adoption, but since I no longer work on the project for my day job I am unlikely to now... so, sorry but you'll have to tolerate them :) In particular, the problem you encounter here is that a given

Re: Cassandra Collections performance issue

If the overwrites are per map key there are no tombstones generated; only if the whole map is re-imaged are tombstones created, and prior to 3.0 this indeed can be major problem if done frequently. Prior to 3.0 collections also forbid certain optimisations to cell comparisons, and as a result can

Re: no change observed in read latency after switching from EBS to SSD storage

It is possible this is CPU bound. In 2.1 we have optimised the comparison of clustering columns (CASSANDRA-5417 ), but in 2.0 it is quite expensive. So for a large row with several million comparisons to perform (to merge, filter, etc.) it could

Re: Mutation Stage does not finish

Could you post the results of jstack on the process somewhere? On Thu, Sep 11, 2014 at 7:07 AM, Robert Coli wrote: > On Wed, Sep 10, 2014 at 1:53 PM, Eduardo Cusa < > eduardo.c...@usmediaconsulting.com> wrote: > >> No, is still running the Mutation Stage. >> > > If you're sure that it is not re

Re: Performance testing in Cassandra

With the official release of 2.1, I highly recommend using the new stress tool bundled with it - it is improved in many ways over the tool in 2.0, and is compatible with older clusters. It supports the same simple mode of operation as the old stress, with better command line interface and more acc

Re: When CHANGES and JIRA "Fix Versions" disagree which should we believe?

In this case, it seems more likely CHANGES.txt will be correct, since it is maintained *at time of commit*, whereas JIRA fix versions can be forgotten to be maintained. On Tue, Sep 9, 2014 at 7:07 AM, Robert Coli wrote: > On Mon, Sep 8, 2014 at 2:56 PM, Peter Haggerty > wrote: > >> When the CHA

Re: ETA for Cassandra 2.1 final release

It's up for vote right now, so should be a just few days unless something unexpected happens. On Mon, Sep 8, 2014 at 4:46 PM, Eugene Voytitsky wrote: > Hi all, > > is there preliminary date when Cassandra 2.1 be finally released (not > beta/rc)? > > -- > Best regards, > Eugene Voytitsky >

Re: Latest 2.1 and Datastax driver questions...

The "run versions >= x.x.6" is IMO an out of date trope. Not only does 2.1 have more than twice as many engineers working full time on it, and five times the QA engineers (which is cumulative with prior QA), we are also seeing many more users in the wild running release candidates and providing val

Re: cassandra-stress with clustering columns?

ng 2.0.9 > unfortunately. :/ I will keep that in mind for when we upgrade. > > On Tue, Aug 19, 2014 at 10:51 AM, Benedict Elliott Smith > wrote: > > The stress tool in 2.1 also now supports clustering columns: > > > http://www.datastax.com/dev/blog/improved-cassandra-2

Re: cassandra-stress with clustering columns?

The stress tool in 2.1 also now supports clustering columns: http://www.datastax.com/dev/blog/improved-cassandra-2-1-stress-tool-benchmark-any-schema There are however some features up for revision before release in order to help generate realistic workloads. See https://issues.apache.org/jira/bro

Re: moving older tables from SSD to HDD?

Hi Kevin, This is something we do plan to support, but don't right now. You can see the discussion around this and related issues here (although it may seem unrelated at first glance). On Mon, Aug 4, 2014 at 8:43 PM, Kevin Burton wrote:

Re: Why is the cassandra documentation such poor quality?

obby. If I wasn't serious about contributing to cassandra > community, I wouldn't have spent 2.5 months porting Hector to C# manually. > > Perhaps the real cause is that some committers can't "empathise" with > others in the community? > > > On Wed, Jul 2

Re: Why is the cassandra documentation such poor quality?

out docs, but > they don't always have time to do it. > > I know that given a choice between coding or writing docs, 90% of the time > I'll choose coding. What I've decided instead is to document stuff on one > of my blogs. If someone gets lucky, maybe google will retu

Re: Why is the cassandra documentation such poor quality?

It only takes a moment to ask to be added as a wiki contributor; if you email the dev list or ask on irc, somebody with privileges will ordinarily add you within a day. It may be a psychological barrier, but it isn't really a practical one. Still, if you feel the policy is incorrect, raise this on

Re: horizontal query scaling issues follow on

3332,"executionId":6214} >> >> >> Type2:e876d44d-246f-40c5-b5a3-4d0eb31db00d | >> HISTORY:2010-08-26T04:58:46.810-0400 | >> {"key":"e876d44d-246f-40c5-b5a3-4d0eb31db00d","keyType":" >> Type2",&quo

Re: horizontal query scaling issues follow on

How many columns are you inserting/querying per key? Could we see some example CQL statements for the insert/read workload? If you are maxing out at 10 clients, something fishy is going on. In general, though, if you find that adding nodes causes performance to degrade I would suspect that you are

Re: How to maintain the N-most-recent versions of a value?

If the versions can be guaranteed to be a adjacent (i.e. if the latest version is V, the prior version is V-1) you could issue a delete at the same time as an insert for V-N-(buffer) where buffer >= 0 In general guaranteeing that is probably hard, so this seems like something that would be nice to

Re: TTransportException (java.net.SocketException: Broken pipe)

July 2014 1:24 PM, Benedict Elliott Smith < > belliottsm...@datastax.com> wrote: > > > Are you still seeing the same exceptions about too many open files? > > > > > On Thu, Jul 17, 2014 at 6:28 AM, Bhaskar Singhal > wrote: > > Even after changing ulimits an

Re: TTransportException (java.net.SocketException: Broken pipe)

Are you still seeing the same exceptions about too many open files? On Thu, Jul 17, 2014 at 6:28 AM, Bhaskar Singhal wrote: > Even after changing ulimits and moving to the recommended production > settings, we are still seeing the same issue. > > root@lnx148-76:~# cat /proc/17663/limits > Lim

Re: C* 2.1-rc2 gets unstable after a 'DROP KEYSPACE' command ?

Also https://issues.apache.org/jira/browse/CASSANDRA-7437 and https://issues.apache.org/jira/browse/CASSANDRA-7465 for rc3, although the CounterCacheKey assertion looks like an independent (though comparatively benign) bug I will file a ticket for. Can you try this against rc3 to see if the proble

Re: EBS SSD <-> Cassandra ?

I would say this is worth benchmarking before jumping to conclusions. The network being a bottleneck (or latency causing) for EBS is, to my knowledge, supposition, and instances can be started with direct connections to EBS if this is a concern. The blog post below shows that even without SSDs the

Re: memtable mem usage off by 10?

:/usr/share/cassandra/lib/stress.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.3.jar > org.apache.cassandra.service.CassandraDaemon > > > > From: Benedict Elliott Smith > Reply-To: "user@cassandra.apache.org" > Date:

Re: memtable mem usage off by 10?

was assuming it was using the default. It does > not exist in my config file. Sorry for the confusion. > > > > From: Benedict Elliott Smith > Reply-To: "user@cassandra.apache.org" > Date: Wednesday 4 June 2014 16:36 > To: "user@cassandra.apache.org"

Re: memtable mem usage off by 10?

nse: > > > > Having memtable_total_size_in_mb set to 20480, memtables are flushed at a > reported value of ~2GB. > > > > With a constant overhead of ~10x, as suggested, this would mean that it > used 20GB, which is 2x the size of the heap. > > > > That shouldn

Re: memtable mem usage off by 10?

ad, this would mean that it used 20GB, which is 2x > the size of the heap, instead of 3/4 of the heap as it should be if > flush_largest_memtables_at was being respected. > > > This shouldn't be possible. > > > -- > *From:* Benedict Elliott Smith > *Sen

Re: memtable mem usage off by 10?

ive up I guess :) > > At least until 2.1.0 comes out and I can compare. > > > BR > > Johan > > > -- > *From:* Benedict Elliott Smith > *Sent:* Wednesday, June 4, 2014 12:33 PM > > *To:* user@cassandra.apache.org > *Subject:* Re: memtable mem u

Re: memtable mem usage off by 10?

0gb of memory) is showing ~10GB > free, with ~12GB used by cassandra, the rest in caches. > > > Reading 8k rows/s, writing 2k rows/s on a 3 node cluster. So it's not > idling. > > > BR > > Johan > > > -- &

Re: memtable mem usage off by 10?

If you are storing small values in your columns, the object overhead is very substantial. So what is 400Mb on disk may well be 4Gb in memtables, so if you are measuring the memtable size by the resulting sstable size, you are not getting an accurate picture. This overhead has been reduced by about

Re: Nectar client - New Cassandra Client for .Net

The native protocol specification has always been in the Apache Cassandra repository. The implementations are not. On 2 June 2014 13:25, Peter Lin wrote: > > There's nothing preventing support for native protocol going forward. It > was easier to go with thrift and I happen to like thirft. Nati

Re: : Read a negative frame size (-2113929216)!

Vivek, The error you are seeing is a thrift error, but you say you are using the Java driver which does not operate over thrift: are you perhaps trying to connect the datastax driver to the thrift protocol port? The two protocols are not compatible, you must connect to the native_transport_port (b

Re: any way to REALLY turn off commitlog?

Are you writing these records as batches (with begin batch, not begin unlogged)? As a logged batch ('begin batch' is logged) gets written to a system keyspace with CL before being applied to the underlying CF, and I'm not sure we want to go special casing that for non-durable writes, since you prob

Re: FileNotFoundException ...-Data.db

Note that drain stops gossip and thrift anyway, so they are redundant. On 25 March 2014 18:30, Robert Coli wrote: > On Tue, Mar 25, 2014 at 5:36 AM, Batranut Bogdan wrote: > >> I am running 2.0.6 and I use /etc/init.d/cassandra start / stop . Also >> before stopping I do : >> nodetool disablego

Re: Cassandra slow on some reads

Post the server logs and traces of one of the lengthy queries? On 15 Mar 2014 20:49, "Batranut Bogdan" wrote: > Hello, > > Yes for the first query the server can be slow but no matter what is > should not take 10 seconds to get one key from a column family. I can see > this happening if I have ti

Re: Cassandra slow on some reads

To add to this, if the 24G is the JVM limit, Cassandra will actually be using even more than this for bloom filters etc. that are managed off-heap. So the amount of page cache left is almost certainly inadequate. On 14 March 2014 13:01, Andras Szerdahelyi < andras.szerdahe...@ignitionone.com> wro

Re: Intermittent long application pauses on nodes

ay GC. On 27 February 2014 17:01, Frank Ng wrote: > We have swap disabled. Can death by paging still happen? > > > On Thu, Feb 27, 2014 at 11:32 AM, Benedict Elliott Smith < > belliottsm...@datastax.com> wrote: > >> That sounds a lot like death by paging. >>

Re: Intermittent long application pauses on nodes

t;> INFO [ScheduledTasks:1] 2014-02-20 13:51:14,542 GCInspector.java (line >>> 116) GC for ConcurrentMarkSweep: 3703 ms for 2 collections, 434394920 used; >>> max is 4253024256 >>> >>> Unfortunately it's a production cluster so I have no additional >>>

Re: Performance problem with large wide row inserts using CQL

> > Cassandra will throw an exception indicating the type is different than > the default type. If you want untyped data, store blobs. Or store in a different column (they're free when empty, after all). Type safety is considered a good thing by many. On 20 February 2014 17:26, Peter Lin wrote

Re: Intermittent long application pauses on nodes

we'd > thought it was a great idea. So I think we were hit by the problem > discussed in this thread, just the G1 report wasn't very clear, sorry. > > regards, > ondrej > > > > On Mon, Feb 17, 2014 at 11:45 AM, Benedict Elliott Smith < > belliottsm...@datastax.com&

Re: Intermittent long application pauses on nodes

t; >>> Subject: Re: Intermittent long application pauses on nodes >>> >>> We have had similar issues and upgrading C* to 2.0.x and Java to 1.7 >>> seems to have helped our issues. >>> >>> >>> 2014-02-13 Keith Wright : >>> >&

Re: impact of update operation to read operation

Also, a major compaction doesn't flush the memtable. If the memtable is still full, reads may take slightly longer as they may have to be merged with any on-disk data before being served. On 10 February 2014 21:18, Tupshin Harper wrote: > You don't mention disks and RAM, but I would assume tha

Re: Lots of deletions results in death by GC

g and fixed it. Now > I just need to figure out where to go from here: do I wait, use the dev > branch or work around. > > Robert > > From: Benedict Elliott Smith > Reply-To: > Date: Wednesday, February 5, 2014 at 8:32 AM > > To: > Subject: Re: Lots of deletions r

Re: Lots of deletions results in death by GC

il 2.0.6 On 5 February 2014 15:09, Robert Wille wrote: > Yes. It's kind of an unusual workload. An insertion phase followed by a > deletion phase, generally not overlapping. > > From: Benedict Elliott Smith > Reply-To: > Date: Tuesday, February 4, 2014 at 5:29 PM >

Re: Lots of deletions results in death by GC

Is it possible you are generating *exclusively* deletes for this table? On 5 February 2014 00:10, Robert Wille wrote: > I ran my test again, and Flush Writer's "All time blocked" increased to 2 > and then shortly thereafter GC went into its death spiral. I doubled > memtable_flush_writers (to 2

Re: Intermittent long application pauses on nodes

Statistics are printed after the Application Stop times, then > it seems that the Deoptimize was the cause of the pause. > In addition, I see a strange dependency failed error relating to JMX in > the JVM stdout log file. > > thanks > > > On Wed, Jan 29, 2014 at 4:44 PM,

Re: Weird GC

#x27;ve added those flags as well as some others I saw in another thread that > redirects stdout to a file. What information is it that you need? > > > 2014-01-29 Benedict Elliott Smith : > > It's possible the time attributed to GC is actually spent somewhere else; >>

Re: Intermittent long application pauses on nodes

to > startup cassandra and send all stdout to a log file and tell cassandra not > to close stdout? > > Also, we have swap turned off as recommended. > > thanks > > > On Wed, Jan 29, 2014 at 3:39 PM, Benedict Elliott Smith < > belliottsm...@datastax.com> wrote: > &g

Re: Intermittent long application pauses on nodes

Frank, The same advice for investigating holds: add the VM flags -XX:+PrintSafepointStatistics -XX:PrintSafepointStatisticsCount=1 (you could put something above 1 there, to reduce the amount of logging, since a pause of 52s will be pretty obvious even if aggregated with lots of other safe points

Re: Weird GC

It's possible the time attributed to GC is actually spent somewhere else; a multitude of tasks may occur during the same safepoint as a GC. We've seen some batch revoke of biased locks take a long time, for instance; *if* this is happening in your case, and we can track down which objects, I would

Re: java.lang.StackOverflowError with big IN list

It must be a very large IN clause, which is probably not advisable. But it shouldn't cause this error, and since it's an easy fix to prevent it, if you file a JIRA I'll post a patch. On 10 January 2014 13:08, Дмитрий Шохов wrote: > Hello I'm getting stack overflow when running prepared queries