Re: Stack trace from a node during a repair

2013-12-03 Thread John Pyeatt
This is running the Amazon Linux OS which is essentially CentOS 6 I believe. java version "1.6.0_45" Java(TM) SE Runtime Environment (build 1.6.0_45-b06) Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode) Installed cassandra 1.2.9 from http://archive.apache.org/dist/cassandra/1.2.9/a

Re: Stack trace from a node during a repair

2013-12-03 Thread Robert Coli
On Tue, Dec 3, 2013 at 6:19 AM, John Pyeatt wrote: > Then my issue must be the 0.01% because > > 1) I'm running the repair as root. > Huh? Repair doesn't care what user your shell is. It is a process built into cassandra and has the permissions that cassandra does? > 2) The directory exists

Re: Stack trace from a node during a repair

2013-12-03 Thread John Pyeatt
Both cassandra and nodetool are running as root. also ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 59450 max locked memory

Re: Stack trace from a node during a repair

2013-12-03 Thread Hannu Kröger
Hi, Are you running nodetool or cassandra as root? I think it doesn't really matter what user is running the nodetool. Those directories should be writable by the user who is running the actual cassandra process. Hannu 2013/12/3 John Pyeatt > Then my issue must be the 0.01% because > > 1)

Re: Stack trace from a node during a repair

2013-12-03 Thread John Pyeatt
Then my issue must be the 0.01% because 1) I'm running the repair as root. 2) The directory exists and the permissions are appropriate. root:root 755 3) The three times it occurred during the repair it always complained about backups directories. But there are dozens other backups directories

Re: Stack trace from a node during a repair

2013-12-02 Thread Robert Coli
On Mon, Dec 2, 2013 at 2:59 PM, John Pyeatt wrote: > Caused by: java.io.IOException: Unable to create directory > /data-1/cassandra/data/SinglewireSupport/Binaries/backups > This is an exception directly from a core java method. The cause is 99.9% likely to be permissions. =Rob

Stack trace from a node during a repair

2013-12-02 Thread John Pyeatt
We are running a 6-node AWS EC2 (m1.large) cluster of cassandra 1.2.9 across three availability zones with Ec2Snitch and NetworkTopologyStrategy. One of our nodes was apparently sharing a physical box with another customer who was really hogging the IO. So we needed to bring the node up on a new e