Re: bugs report for 2.0.7

Tue, 29 Apr 2014 18:28:26 -0700

Then you likely need to fix your I/O problem. The most recent error you posted is an EOFException - the file being read ended unexpectedly. Probably when you ran out of disk space. 

--
Michael

On 04/29/2014 07:48 PM, Yatong Zhang wrote:

Here is another type of exception, seems all are I/O related:

  INFO [SSTableBatchOpen:1] 2014-04-29 14:44:35,548 SSTableReader.java (line

223) Opening
/data2/cass/system/compaction_history/system-compaction_history-jb-6956
(447252 bytes)
  INFO [SSTableBatchOpen:2] 2014-04-29 14:44:35,553 SSTableReader.java
(line 223) Opening
/data2/cass/system/compaction_history/system-compaction_history-jb-6958
(257 bytes)
  INFO [SSTableBatchOpen:3] 2014-04-29 14:44:35,554 SSTableReader.java
(line 223) Opening
/data2/cass/system/compaction_history/system-compaction_history-jb-6957
(257 bytes)
  INFO [main] 2014-04-29 14:44:35,592 ColumnFamilyStore.java (line 248)
Initializing system.batchlog
  INFO [main] 2014-04-29 14:44:35,596 ColumnFamilyStore.java (line 248)
Initializing system.sstable_activity
  INFO [SSTableBatchOpen:1] 2014-04-29 14:44:35,601 SSTableReader.java
(line 223) Opening
/data2/cass/system/sstable_activity/system-sstable_activity-jb-8084 (1562
bytes)
  INFO [SSTableBatchOpen:2] 2014-04-29 14:44:35,604 SSTableReader.java
(line 223) Opening
/data2/cass/system/sstable_activity/system-sstable_activity-jb-8083 (2075
bytes)
  INFO [SSTableBatchOpen:3] 2014-04-29 14:44:35,605 SSTableReader.java
(line 223) Opening
/data2/cass/system/sstable_activity/system-sstable_activity-jb-8085 (1555
bytes)
  INFO [main] 2014-04-29 14:44:35,687 AutoSavingCache.java (line 114)
reading saved cache
/data1/saved_caches/system-sstable_activity-KeyCache-b.db
  INFO [main] 2014-04-29 14:44:35,696 ColumnFamilyStore.java (line 248)
Initializing system.peer_events
  INFO [SSTableBatchOpen:1] 2014-04-29 14:44:35,697 SSTableReader.java
(line 223) Opening /data4/cass/system/peer_events/system-peer_events-jb-181
(12342 bytes)
  INFO [main] 2014-04-29 14:44:35,717 ColumnFamilyStore.java (line 248)
Initializing system.compactions_in_progress
  INFO [SSTableBatchOpen:1] 2014-04-29 14:44:35,718 SSTableReader.java
(line 223) Opening
/data5/cass/system/compactions_in_progress/system-compactions_in_progress-jb-36448
(167 bytes)
ERROR [SSTableBatchOpen:1] 2014-04-29 14:44:35,730 CassandraDaemon.java
(line 198) Exception in thread Thread[SSTableBatchOpen:1,5,main]
org.apache.cassandra.io.sstable.CorruptSSTableException:
java.io.EOFException
         at
org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:110)
         at
org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:64)
         at
org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:42)
         at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:458)
         at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:422)
         at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:203)
         at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:184)
         at
org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:264)
         at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
         at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
         at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
         at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.EOFException
         at
java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
         at java.io.DataInputStream.readUTF(DataInputStream.java:589)
         at java.io.DataInputStream.readUTF(DataInputStream.java:564)
         at
org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:85)
         ... 12 more
  INFO [main] 2014-04-29 14:44:35,733 ColumnFamilyStore.java (line 248)
Initializing system.hints
  INFO [main] 2014-04-29 14:44:35,734 AutoSavingCache.java (line 114)
reading saved cache /data1/saved_caches/system-hints-KeyCache-b.db
  INFO [main] 2014-04-29 14:44:35,737 ColumnFamilyStore.java (line 248)
Initializing system.schema_keyspaces





On Tue, Apr 29, 2014 at 6:07 PM, Yatong Zhang <bluefl...@gmail.com> wrote:


I am pretty sure the disk has plenty of space, I am sure of that. I
restarted cassandra and everything went fine again.

It's really wired


On Tue, Apr 29, 2014 at 5:58 PM, Sylvain Lebresne <sylv...@datastax.com>wrote:


The important part of that stack trace is "java.io.IOException: No space
left on device", your disks are full (and it's not really a bug that
Cassandra error out in that case).

--
Sylvain


On Tue, Apr 29, 2014 at 11:09 AM, Yatong Zhang <bluefl...@gmail.com>
wrote:


Hi there,

Sorry if this is not the right place to report bugs. I am using 2.0.7

and I

have a 10 boxes clusters with about 200TB capacity. I just found I had 3
boxes with error exceptions. With datastax opscenter I can see these

three

nodes lost connections (no reponse), but after I sshed to these server,
cassandara were still running, and the 'system.log' still had logs.

I think this might be a bug so any one would kindly help to investigate
into it? Thanks~

ERROR [CompactionExecutor:1] 2014-04-29 05:55:15,249

CassandraDaemon.java

(line 198) Exception in thread Thread[CompactionExecutor:1,1,main]
FSWriteError in

/data2/cass/mydb/images/mydb-images-tmp-jb-98219-Filter.db

         at




org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:475)

         at


org.apache.cassandra.io.util.FileUtils.closeQuietly(FileUtils.java:212)

         at




org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:301)

         at




org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:209)

         at




org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)

         at


org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)

         at




org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)

         at




org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)

         at




org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197)

         at


java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
         at




java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

         at




java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

         at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.IOException: No space left on device
         at java.io.FileOutputStream.write(Native Method)
         at java.io.FileOutputStream.write(FileOutputStream.java:295)
         at

java.io.DataOutputStream.writeInt(DataOutputStream.java:197)

         at




org.apache.cassandra.utils.BloomFilterSerializer.serialize(BloomFilterSerializer.java:34)

         at




org.apache.cassandra.utils.Murmur3BloomFilter$Murmur3BloomFilterSerializer.serialize(Murmur3BloomFilter.java:44)

         at


org.apache.cassandra.utils.FilterFactory.serialize(FilterFactory.java:41)

         at




org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:468)

         ... 13 more
ERROR [CompactionExecutor:1] 2014-04-29 05:55:15,406

StorageService.java

(line 367) Stopping gossiper
  WARN [CompactionExecutor:1] 2014-04-29 05:55:15,406

StorageService.java

(line 281) Stopping gossip by operator request
  INFO [CompactionExecutor:1] 2014-04-29 05:55:15,406 Gossiper.java

(line

1271) Announcing shutdown
ERROR [CompactionExecutor:1] 2014-04-29 05:55:17,406

StorageService.java

(line 372) Stopping RPC server
  INFO [CompactionExecutor:1] 2014-04-29 05:55:17,406 ThriftServer.java
(line 141) Stop listening to thrift clients
ERROR [CompactionExecutor:1] 2014-04-29 05:55:17,417

StorageService.java

(line 377) Stopping native transport
  INFO [CompactionExecutor:1] 2014-04-29 05:55:17,504 Server.java (line
181) Stop listening for CQL clients

Reply via email to