I am pretty sure the disk has plenty of space, I am sure of that. I restarted cassandra and everything went fine again.
It's really wired On Tue, Apr 29, 2014 at 5:58 PM, Sylvain Lebresne <sylv...@datastax.com>wrote: > The important part of that stack trace is "java.io.IOException: No space > left on device", your disks are full (and it's not really a bug that > Cassandra error out in that case). > > -- > Sylvain > > > On Tue, Apr 29, 2014 at 11:09 AM, Yatong Zhang <bluefl...@gmail.com> > wrote: > > > Hi there, > > > > Sorry if this is not the right place to report bugs. I am using 2.0.7 > and I > > have a 10 boxes clusters with about 200TB capacity. I just found I had 3 > > boxes with error exceptions. With datastax opscenter I can see these > three > > nodes lost connections (no reponse), but after I sshed to these server, > > cassandara were still running, and the 'system.log' still had logs. > > > > I think this might be a bug so any one would kindly help to investigate > > into it? Thanks~ > > > > ERROR [CompactionExecutor:1] 2014-04-29 05:55:15,249 CassandraDaemon.java > > > (line 198) Exception in thread Thread[CompactionExecutor:1,1,main] > > > FSWriteError in > > /data2/cass/mydb/images/mydb-images-tmp-jb-98219-Filter.db > > > at > > > > > > org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:475) > > > at > > > org.apache.cassandra.io.util.FileUtils.closeQuietly(FileUtils.java:212) > > > at > > > > > > org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:301) > > > at > > > > > > org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:209) > > > at > > > > > > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > > > at > > > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > > > at > > > > > > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) > > > at > > > > > > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) > > > at > > > > > > org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197) > > > at > > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > > > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > > > at > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > > at > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > > at java.lang.Thread.run(Thread.java:744) > > > Caused by: java.io.IOException: No space left on device > > > at java.io.FileOutputStream.write(Native Method) > > > at java.io.FileOutputStream.write(FileOutputStream.java:295) > > > at java.io.DataOutputStream.writeInt(DataOutputStream.java:197) > > > at > > > > > > org.apache.cassandra.utils.BloomFilterSerializer.serialize(BloomFilterSerializer.java:34) > > > at > > > > > > org.apache.cassandra.utils.Murmur3BloomFilter$Murmur3BloomFilterSerializer.serialize(Murmur3BloomFilter.java:44) > > > at > > > > org.apache.cassandra.utils.FilterFactory.serialize(FilterFactory.java:41) > > > at > > > > > > org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:468) > > > ... 13 more > > > ERROR [CompactionExecutor:1] 2014-04-29 05:55:15,406 > StorageService.java > > > (line 367) Stopping gossiper > > > WARN [CompactionExecutor:1] 2014-04-29 05:55:15,406 > StorageService.java > > > (line 281) Stopping gossip by operator request > > > INFO [CompactionExecutor:1] 2014-04-29 05:55:15,406 Gossiper.java > (line > > > 1271) Announcing shutdown > > > ERROR [CompactionExecutor:1] 2014-04-29 05:55:17,406 > StorageService.java > > > (line 372) Stopping RPC server > > > INFO [CompactionExecutor:1] 2014-04-29 05:55:17,406 ThriftServer.java > > > (line 141) Stop listening to thrift clients > > > ERROR [CompactionExecutor:1] 2014-04-29 05:55:17,417 > StorageService.java > > > (line 377) Stopping native transport > > > INFO [CompactionExecutor:1] 2014-04-29 05:55:17,504 Server.java (line > > > 181) Stop listening for CQL clients > > > > > >
