How to deal with org.apache.cassandra.db.marshal.* types in client drivers?

[Nimi Wariboko]

[Reposting from client-dev, client-dev wasn't getting many responses, and
it seems my previous email is no longer allowed to post to the mailing
list?]

Hi,

After upgrading my cluster from 1.2.16 to 2.0.6, when using the Cassandra
community Go driver (http://github.com/gocql/gocql), I’ve found that
columns
that used to return the native TypeTimestamp (0x000D), now return
TypeCustom
(0x) with the name 'org.apache.cassandra.db.marshal.DataType’. As far
as I
can tell the serialization format is identical to TypeTimestamp, but it is
unclear why Cassandra made the change, or how to reproduce it (if I create
a
new table, Cassandra 2.0.6 will return TypeTimestamp). I originally thought
CustomTypes were used in a client-specific implementation but them see to
part of the driver spec.

After studying the other drivers its clear that there are special cases for
the
‘org.apache.cassandra.db.marshal.*’ custom types however I can’t seem to
find any
documentation on what types exist and how to deal with them (other than
what’s
in code). Can someone point me to a documentation and/or explain these
custom types and how to unmarshal them?

Thanks

BufferPool always allocates on heap?

[Nimi Wariboko Jr]

Hi,

I'm trying to debug an issue I'm having with Cassandra 3.6+ and Snappy
compressed tables - CASSANDRA-11993 (that may be related to CASSANDRA-5863).

I'm probably 3% familiar with Cassandra internals, so apologies if I'm
wrong here - I was looking at a change in CASSANDRA-5863 where a cache was
added to the SSTableIterator code path, which may have
included org.apache.cassandra.utils.memory.BufferPool into the read path.

My confusion comes from here - AFAICT, when the BufferPool needs to
allocate more memory, it calls allocate(int size, boolean onHeap) (
https://github.com/apache/cassandra/blob/30bb255ec9fb36ace2aab51474bd3bfb9bbd3bed/src/java/org/apache/cassandra/utils/memory/BufferPool.java#L82).
The value for whether onHeap is true is
constant ALLOCATE_ON_HEAP_WHEN_EXAHUSTED.

ALLOCATE_ON_HEAP_WHEN_EXAHUSTED is defined by the yaml
setting buffer_pool_use_heap_if_exhausted which is true by default (and was
so in my install).

Further still (
https://github.com/apache/cassandra/blob/30bb255ec9fb36ace2aab51474bd3bfb9bbd3bed/src/java/org/apache/cassandra/utils/memory/BufferPool.java),
I can't see any logic where BufferPool will actually allocate off heap
memory - it seems, due to the buffer_pool_use_heap_if_exhausted setting,
always allocate memory on heap. It never checks if the off heap memory is
exhausted (AFAICT from these 3 files).

Sure enough, if I set buffer_pool_use_heap_if_exhausted, my issue at
CASSANDRA-11993 goes away (I think? I'm having a number of issues lately
and I may have traded one exception for another).

Is this the correct behavior of buffer_pool_use_heap_if_exhausted?

Nimi

Re: BufferPool always allocates on heap?

[Nimi Wariboko Jr]

Hi Stefania,

Thanks for the explanation - I think it may be more likely I'm hitting
MEMORY_USAGE_THRESHOLD
(file_cache_size_in_mb, which is 512mb by default), as the table itself is
very large (a couple TB over 12 nodes). In that case, due to the changes
outlined CASSANDRA-9548/CASSANDRA-8709/CASSANDRA-5863, there should
probably be a warning in 3.6 that buffer_pool_use_heap_if_exhausted should
be set to false if you still have some tables on SnappyCompression.

Nimi

On Wed, Jun 22, 2016 at 12:14 AM, Stefania Alborghetti <
stefania.alborghe...@datastax.com> wrote:

> Hi Nimi
>
> I am not familiar with how CASSANDRA-5863 uses the buffer pool yet, but I
> do know how the buffer pool works
>
> It has a local pool, which is thread local, and takes chunks from a global
> pool. These are allocated off heap and aligned to the page cache
> boundaries, see here
> <
> https://github.com/apache/cassandra/blob/30bb255ec9fb36ace2aab51474bd3bfb9bbd3bed/src/java/org/apache/cassandra/utils/memory/BufferPool.java#L276
> >.
>
>
> The buffer pool will allocate according to
> buffer_pool_use_heap_if_exhausted
> only in two cases:
>
>- The global pool has run out of chunks, which happens when there is an
>OOM exception or when MEMORY_USAGE_THRESHOLD is exhausted, see here
><
> https://github.com/apache/cassandra/blob/30bb255ec9fb36ace2aab51474bd3bfb9bbd3bed/src/java/org/apache/cassandra/utils/memory/BufferPool.java#L259
> >
>.
>- The requested buffer size is larger than a chunk, currently
>CHUNK_SIZE, 64 kb.
>
> To understand this behaviour look at maybeTakeFromPool
> <
> https://github.com/apache/cassandra/blob/30bb255ec9fb36ace2aab51474bd3bfb9bbd3bed/src/java/org/apache/cassandra/utils/memory/BufferPool.java#L124
> >(),
> the line that takes from the local pool is the last one and follow from
> there.
>
> The original ticket for the buffer pool is CASSANDRA-8897
> <https://issues.apache.org/jira/browse/CASSANDRA-8897>if you need more
> details, and there is CASSANDRA-9468
> <https://issues.apache.org/jira/browse/CASSANDRA-9468> to extend it to
> allow chunks with bigger or smaller sizes (point 4).
>
> I've left a comment on CASSANDRA-11993.
>
> I hope this helps,
> Stefania
>
>
> On Wed, Jun 22, 2016 at 4:54 AM, Nimi Wariboko Jr 
> wrote:
>
> > Hi,
> >
> > I'm trying to debug an issue I'm having with Cassandra 3.6+ and Snappy
> > compressed tables - CASSANDRA-11993 (that may be related to
> > CASSANDRA-5863).
> >
> > I'm probably 3% familiar with Cassandra internals, so apologies if I'm
> > wrong here - I was looking at a change in CASSANDRA-5863 where a cache
> was
> > added to the SSTableIterator code path, which may have
> > included org.apache.cassandra.utils.memory.BufferPool into the read path.
> >
> > My confusion comes from here - AFAICT, when the BufferPool needs to
> > allocate more memory, it calls allocate(int size, boolean onHeap) (
> >
> >
> https://github.com/apache/cassandra/blob/30bb255ec9fb36ace2aab51474bd3bfb9bbd3bed/src/java/org/apache/cassandra/utils/memory/BufferPool.java#L82
> > ).
> > The value for whether onHeap is true is
> > constant ALLOCATE_ON_HEAP_WHEN_EXAHUSTED.
> >
> > ALLOCATE_ON_HEAP_WHEN_EXAHUSTED is defined by the yaml
> > setting buffer_pool_use_heap_if_exhausted which is true by default (and
> was
> > so in my install).
> >
> > Further still (
> >
> >
> https://github.com/apache/cassandra/blob/30bb255ec9fb36ace2aab51474bd3bfb9bbd3bed/src/java/org/apache/cassandra/utils/memory/BufferPool.java
> > ),
> > I can't see any logic where BufferPool will actually allocate off heap
> > memory - it seems, due to the buffer_pool_use_heap_if_exhausted setting,
> > always allocate memory on heap. It never checks if the off heap memory is
> > exhausted (AFAICT from these 3 files).
> >
> > Sure enough, if I set buffer_pool_use_heap_if_exhausted, my issue at
> > CASSANDRA-11993 goes away (I think? I'm having a number of issues lately
> > and I may have traded one exception for another).
> >
> > Is this the correct behavior of buffer_pool_use_heap_if_exhausted?
> >
> > Nimi
> >
>
>
>
> --
>
>
> [image: datastax_logo.png] <http://www.datastax.com/>
>
> Stefania Alborghetti
>
> Apache Cassandra Software Engineer
>
> |+852 6114 9265| stefania.alborghe...@datastax.com
>
>
> [image: cassandrasummit.org/Email_Signature]
> <http://cassandrasummit.org/Email_Signature>
>

Dealing with Corrupted/Missing LCS Metadata

[Nimi Wariboko Jr]

Hi,

I'm trying to solve an issue we are seeing on our production cluster, in
various LCS tables, with Cassandra 3.5. I believe it is related
to CASSANDRA-11373 (however CASSANDRA-11373 is marked as resolved in 3.5).

I'm not sure what causes it, but eventually `getCandidatesFor` in
LeveledManifest.java (
https://github.com/apache/cassandra/blob/cassandra-3.5/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java#L553)
gets stuck in an infinite loop on L0 (and eventually goes into a GC
thrashing loop). Investigating further, it looks like `getCandidatesFor`
returns a mix of sstables, some of which don't exist on disk (same symptom
as 11373).

What I'd like to know is what I can do here? For smaller tables we switched
to STS, but we have a very large LCS timeseries (~300GB per node) table,
that would take ages to switch over and really affect our performance on
STS (offline scrub doesn't work either).

Would it suffice to add a check here to ensure the file exists and evicting
the file if it doesn't? Or is there anyway I could force the rewrite of the
table's metadata?

Nimi