Re: Unbalanced disk load

2015-07-18 Thread J. Ryan Earl
Even with https://issues.apache.org/jira/browse/CASSANDRA-7386 data balancing across JBOD setups is pretty horrible. Having used JBOD for about 2 years from 1.2.x and up, it is my opinion JBOD on Cassandra is nascent at best and far from mature. For a variety of reasons, JBOD should perform bette

Re: error='Cannot allocate memory' (errno=12)

2015-05-12 Thread J. Ryan Earl
ors limits, similar to how pipes, sockets, etc can count against file descriptor limits. On Tue, May 12, 2015 at 7:55 PM, J. Ryan Earl wrote: > What's your ulimit -a output? Did you adjust nproc and nofile ulimits > up? Do you have JNA installed? What about memlock ulimit and in > sysc

Re: error='Cannot allocate memory' (errno=12)

2015-05-12 Thread J. Ryan Earl
What's your ulimit -a output? Did you adjust nproc and nofile ulimits up? Do you have JNA installed? What about memlock ulimit and in sysctl.conf => kernel.shmmax? What's in cassandra.log? On Mon, May 11, 2015 at 7:24 AM, Rahul Bhardwaj < rahul.bhard...@indiamart.com> wrote: > Hi Robert, > > I

Re: OOM and high SSTables count

2015-03-04 Thread J. Ryan Earl
We think it is this bug: https://issues.apache.org/jira/browse/CASSANDRA-8860 We're rolling a patch to beta before rolling it into production. On Wed, Mar 4, 2015 at 4:12 PM, graham sanderson wrote: > We can confirm a problem on 2.1.3 (sadly our beta sstable state obviously > did not match our

Re: Setting up JNA on CentOS 6.6. with cassandra20-2.0.12 and Oracle Java 1.7.0_75

2015-02-25 Thread J. Ryan Earl
quot; > > and at the bottom > > "If you can't install using Yum or it provides a version of the JNA > earlier than 3.2.7, install as described in Installing the JNA from the JAR > file." > > Which version of OS and Cassandra are you running? > > thanks,

Re: Setting up JNA on CentOS 6.6. with cassandra20-2.0.12 and Oracle Java 1.7.0_75

2015-02-25 Thread J. Ryan Earl
We've been using jna-3.2.4-2.el6.x86_64 with the Sun/Oracle JDK for probably 2-years now, and it works just fine. Where are you seeing 3.2.7 required at? I searched the pages you link and that string isn't even in there. Regardless, I assure you the newest jna that ships in the EL6 repo works wi

Re: ava.lang.OutOfMemoryError: unable to create new native thread

2014-09-18 Thread J. Ryan Earl
What's the 'ulimit -a' output of the user cassandra runs as? From this and your previous OOM thread, is sounds like you skipped the requisite OS configuration. On Wed, Sep 17, 2014 at 9:43 AM, Yatong Zhang wrote: > Hi there, > > I am using leveled compaction strategy and have many sstable files

Re: hs_err_pid3013.log, out of memory?

2014-09-18 Thread J. Ryan Earl
On Wed, Sep 17, 2014 at 8:35 PM, Yatong Zhang wrote: > @Chris Lohfink I have 16G memory per node, all the other settings are > default > > @J. Ryan Earl I am not sure. I am using the default settings. > > But I've found out it might be because some settings in > 

Re: hs_err_pid3013.log, out of memory?

2014-09-16 Thread J. Ryan Earl
Are you using JNA? Did you adjust your memlock limit? On Tue, Sep 16, 2014 at 9:46 AM, Chris Lohfink wrote: > How much memory does your system have? How much memory is system utilizing > before starting Cassandra (use command free)? What are the heap setting it > tries to use? > > Chris > > On

Re: too many open files

2014-08-08 Thread J. Ryan Earl
Yes, definitely look how many open files are actual file handles versus networks sockets. We found a file handle leak in 2.0 but it was patched in 2.0.3 or .5 I think. A million open files is way too high. On Fri, Aug 8, 2014 at 5:19 PM, Andrey Ilinykh wrote: > You may have this problem if yo

Re: Write performance with 1.2.12

2013-12-12 Thread J. Ryan Earl
Why did you switch to RandomPartitioner away from Murmur3Partitioner? Have you tried with Murmur3? 1. # partitioner: org.apache.cassandra.dht.Murmur3Partitioner 2. partitioner: org.apache.cassandra.dht.RandomPartitioner On Fri, Dec 6, 2013 at 10:36 AM, srmore wrote: > > > > On Fri, De

Re: cassandra performance problems

2013-12-06 Thread J. Ryan Earl
On Thu, Dec 5, 2013 at 6:33 AM, Alexander Shutyaev wrote: > We've plugged it into our production environment as a cache in front of > postgres. Everything worked fine, we even stressed it by explicitly > propagating about 30G (10G/node) data from postgres to cassandra. > If you just want a cachin

Re: Getting into Too many open files issues

2013-11-20 Thread J. Ryan Earl
There was a bug introduced in 2.0.0-beta1 related to TTL, a patch just came available in: https://issues.apache.org/jira/browse/CASSANDRA-6275 On Thu, Nov 7, 2013 at 5:15 AM, Murthy Chelankuri wrote: > I have experimenting cassandra latest version for storing the huge the in > our application. >

Re: Cass 2.0.0: Extensive memory allocation when row_cache enabled

2013-11-14 Thread J. Ryan Earl
First off, I'm curious what hardware (system specs) you're running this on? Secondly, here are some observations: * You're not running the newest JDK7, I can tell by your stack-size. Consider getting the newest. * Cassandra 2.0.2 has a lot of improvements, consider upgrading. We noticed improve

Re: Multi-dc restart impact

2013-10-10 Thread J. Ryan Earl
Are you doing QUORUM reads instead of LOCAL_QUORUM reads? On Wed, Oct 9, 2013 at 7:41 PM, Chris Burroughs wrote: > I have not been able to do the test with the 2nd cluster, but have been > given a disturbing data point. We had a disk slowly fail causing a > significant performance degradation t