The important part of that stack trace is "java.io.IOException: No space left on device", your disks are full (and it's not really a bug that Cassandra error out in that case).
-- Sylvain On Tue, Apr 29, 2014 at 11:09 AM, Yatong Zhang <bluefl...@gmail.com> wrote: > Hi there, > > Sorry if this is not the right place to report bugs. I am using 2.0.7 and I > have a 10 boxes clusters with about 200TB capacity. I just found I had 3 > boxes with error exceptions. With datastax opscenter I can see these three > nodes lost connections (no reponse), but after I sshed to these server, > cassandara were still running, and the 'system.log' still had logs. > > I think this might be a bug so any one would kindly help to investigate > into it? Thanks~ > > ERROR [CompactionExecutor:1] 2014-04-29 05:55:15,249 CassandraDaemon.java > > (line 198) Exception in thread Thread[CompactionExecutor:1,1,main] > > FSWriteError in > /data2/cass/mydb/images/mydb-images-tmp-jb-98219-Filter.db > > at > > > org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:475) > > at > > org.apache.cassandra.io.util.FileUtils.closeQuietly(FileUtils.java:212) > > at > > > org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:301) > > at > > > org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:209) > > at > > > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > > at > > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > > at > > > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) > > at > > > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) > > at > > > org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197) > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:744) > > Caused by: java.io.IOException: No space left on device > > at java.io.FileOutputStream.write(Native Method) > > at java.io.FileOutputStream.write(FileOutputStream.java:295) > > at java.io.DataOutputStream.writeInt(DataOutputStream.java:197) > > at > > > org.apache.cassandra.utils.BloomFilterSerializer.serialize(BloomFilterSerializer.java:34) > > at > > > org.apache.cassandra.utils.Murmur3BloomFilter$Murmur3BloomFilterSerializer.serialize(Murmur3BloomFilter.java:44) > > at > > org.apache.cassandra.utils.FilterFactory.serialize(FilterFactory.java:41) > > at > > > org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:468) > > ... 13 more > > ERROR [CompactionExecutor:1] 2014-04-29 05:55:15,406 StorageService.java > > (line 367) Stopping gossiper > > WARN [CompactionExecutor:1] 2014-04-29 05:55:15,406 StorageService.java > > (line 281) Stopping gossip by operator request > > INFO [CompactionExecutor:1] 2014-04-29 05:55:15,406 Gossiper.java (line > > 1271) Announcing shutdown > > ERROR [CompactionExecutor:1] 2014-04-29 05:55:17,406 StorageService.java > > (line 372) Stopping RPC server > > INFO [CompactionExecutor:1] 2014-04-29 05:55:17,406 ThriftServer.java > > (line 141) Stop listening to thrift clients > > ERROR [CompactionExecutor:1] 2014-04-29 05:55:17,417 StorageService.java > > (line 377) Stopping native transport > > INFO [CompactionExecutor:1] 2014-04-29 05:55:17,504 Server.java (line > > 181) Stop listening for CQL clients > > >
