Here is another type of exception, seems all are I/O related: INFO [SSTableBatchOpen:1] 2014-04-29 14:44:35,548 SSTableReader.java (line > 223) Opening > /data2/cass/system/compaction_history/system-compaction_history-jb-6956 > (447252 bytes) > INFO [SSTableBatchOpen:2] 2014-04-29 14:44:35,553 SSTableReader.java > (line 223) Opening > /data2/cass/system/compaction_history/system-compaction_history-jb-6958 > (257 bytes) > INFO [SSTableBatchOpen:3] 2014-04-29 14:44:35,554 SSTableReader.java > (line 223) Opening > /data2/cass/system/compaction_history/system-compaction_history-jb-6957 > (257 bytes) > INFO [main] 2014-04-29 14:44:35,592 ColumnFamilyStore.java (line 248) > Initializing system.batchlog > INFO [main] 2014-04-29 14:44:35,596 ColumnFamilyStore.java (line 248) > Initializing system.sstable_activity > INFO [SSTableBatchOpen:1] 2014-04-29 14:44:35,601 SSTableReader.java > (line 223) Opening > /data2/cass/system/sstable_activity/system-sstable_activity-jb-8084 (1562 > bytes) > INFO [SSTableBatchOpen:2] 2014-04-29 14:44:35,604 SSTableReader.java > (line 223) Opening > /data2/cass/system/sstable_activity/system-sstable_activity-jb-8083 (2075 > bytes) > INFO [SSTableBatchOpen:3] 2014-04-29 14:44:35,605 SSTableReader.java > (line 223) Opening > /data2/cass/system/sstable_activity/system-sstable_activity-jb-8085 (1555 > bytes) > INFO [main] 2014-04-29 14:44:35,687 AutoSavingCache.java (line 114) > reading saved cache > /data1/saved_caches/system-sstable_activity-KeyCache-b.db > INFO [main] 2014-04-29 14:44:35,696 ColumnFamilyStore.java (line 248) > Initializing system.peer_events > INFO [SSTableBatchOpen:1] 2014-04-29 14:44:35,697 SSTableReader.java > (line 223) Opening /data4/cass/system/peer_events/system-peer_events-jb-181 > (12342 bytes) > INFO [main] 2014-04-29 14:44:35,717 ColumnFamilyStore.java (line 248) > Initializing system.compactions_in_progress > INFO [SSTableBatchOpen:1] 2014-04-29 14:44:35,718 SSTableReader.java > (line 223) Opening > /data5/cass/system/compactions_in_progress/system-compactions_in_progress-jb-36448 > (167 bytes) > ERROR [SSTableBatchOpen:1] 2014-04-29 14:44:35,730 CassandraDaemon.java > (line 198) Exception in thread Thread[SSTableBatchOpen:1,5,main] > org.apache.cassandra.io.sstable.CorruptSSTableException: > java.io.EOFException > at > org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:110) > at > org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:64) > at > org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:42) > at > org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:458) > at > org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:422) > at > org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:203) > at > org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:184) > at > org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:264) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > Caused by: java.io.EOFException > at > java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340) > at java.io.DataInputStream.readUTF(DataInputStream.java:589) > at java.io.DataInputStream.readUTF(DataInputStream.java:564) > at > org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:85) > ... 12 more > INFO [main] 2014-04-29 14:44:35,733 ColumnFamilyStore.java (line 248) > Initializing system.hints > INFO [main] 2014-04-29 14:44:35,734 AutoSavingCache.java (line 114) > reading saved cache /data1/saved_caches/system-hints-KeyCache-b.db > INFO [main] 2014-04-29 14:44:35,737 ColumnFamilyStore.java (line 248) > Initializing system.schema_keyspaces >
On Tue, Apr 29, 2014 at 6:07 PM, Yatong Zhang <bluefl...@gmail.com> wrote: > I am pretty sure the disk has plenty of space, I am sure of that. I > restarted cassandra and everything went fine again. > > It's really wired > > > On Tue, Apr 29, 2014 at 5:58 PM, Sylvain Lebresne <sylv...@datastax.com>wrote: > >> The important part of that stack trace is "java.io.IOException: No space >> left on device", your disks are full (and it's not really a bug that >> Cassandra error out in that case). >> >> -- >> Sylvain >> >> >> On Tue, Apr 29, 2014 at 11:09 AM, Yatong Zhang <bluefl...@gmail.com> >> wrote: >> >> > Hi there, >> > >> > Sorry if this is not the right place to report bugs. I am using 2.0.7 >> and I >> > have a 10 boxes clusters with about 200TB capacity. I just found I had 3 >> > boxes with error exceptions. With datastax opscenter I can see these >> three >> > nodes lost connections (no reponse), but after I sshed to these server, >> > cassandara were still running, and the 'system.log' still had logs. >> > >> > I think this might be a bug so any one would kindly help to investigate >> > into it? Thanks~ >> > >> > ERROR [CompactionExecutor:1] 2014-04-29 05:55:15,249 >> CassandraDaemon.java >> > > (line 198) Exception in thread Thread[CompactionExecutor:1,1,main] >> > > FSWriteError in >> > /data2/cass/mydb/images/mydb-images-tmp-jb-98219-Filter.db >> > > at >> > > >> > >> org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:475) >> > > at >> > > >> org.apache.cassandra.io.util.FileUtils.closeQuietly(FileUtils.java:212) >> > > at >> > > >> > >> org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:301) >> > > at >> > > >> > >> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:209) >> > > at >> > > >> > >> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) >> > > at >> > > >> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) >> > > at >> > > >> > >> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) >> > > at >> > > >> > >> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) >> > > at >> > > >> > >> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197) >> > > at >> > > >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> > > at java.util.concurrent.FutureTask.run(FutureTask.java:262) >> > > at >> > > >> > >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> > > at >> > > >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> > > at java.lang.Thread.run(Thread.java:744) >> > > Caused by: java.io.IOException: No space left on device >> > > at java.io.FileOutputStream.write(Native Method) >> > > at java.io.FileOutputStream.write(FileOutputStream.java:295) >> > > at >> java.io.DataOutputStream.writeInt(DataOutputStream.java:197) >> > > at >> > > >> > >> org.apache.cassandra.utils.BloomFilterSerializer.serialize(BloomFilterSerializer.java:34) >> > > at >> > > >> > >> org.apache.cassandra.utils.Murmur3BloomFilter$Murmur3BloomFilterSerializer.serialize(Murmur3BloomFilter.java:44) >> > > at >> > > >> org.apache.cassandra.utils.FilterFactory.serialize(FilterFactory.java:41) >> > > at >> > > >> > >> org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:468) >> > > ... 13 more >> > > ERROR [CompactionExecutor:1] 2014-04-29 05:55:15,406 >> StorageService.java >> > > (line 367) Stopping gossiper >> > > WARN [CompactionExecutor:1] 2014-04-29 05:55:15,406 >> StorageService.java >> > > (line 281) Stopping gossip by operator request >> > > INFO [CompactionExecutor:1] 2014-04-29 05:55:15,406 Gossiper.java >> (line >> > > 1271) Announcing shutdown >> > > ERROR [CompactionExecutor:1] 2014-04-29 05:55:17,406 >> StorageService.java >> > > (line 372) Stopping RPC server >> > > INFO [CompactionExecutor:1] 2014-04-29 05:55:17,406 ThriftServer.java >> > > (line 141) Stop listening to thrift clients >> > > ERROR [CompactionExecutor:1] 2014-04-29 05:55:17,417 >> StorageService.java >> > > (line 377) Stopping native transport >> > > INFO [CompactionExecutor:1] 2014-04-29 05:55:17,504 Server.java (line >> > > 181) Stop listening for CQL clients >> > > >> > >> > >
