Re: JDK 17 official support -- coming in v5 (or sooner?)

2025-07-08 Thread Jon Haddad
I don't have the thread handy but I thought we decided to do this already and it would just be a doc change. Either way, I'm +1 on removing experimental flag. Jon On Tue, Jul 8, 2025 at 1:38 PM Geremy Cohen wrote: > Hi all, just curious as to when docs will be updated to signify official > JDK

Re: Request for Thoughts on Deployments on AWS EC2 vs. ECS

2025-06-12 Thread Jon Haddad
ries about there being >> additional hidden nuances to implementing the same level of functionality >> and reliability in ECS and even K8s. We agree that the K8ssandra operator >> would be the most advantageous and desired aspect of switching to a >> container-based solu

Re: Request for Thoughts on Deployments on AWS EC2 vs. ECS

2025-06-12 Thread Jon Haddad
I agree that managing Cassandra on Kubernetes can be challenging without prior experience, as understanding all the nuances of Kubernetes takes time. However, there are ways to address the rescheduling issues, node placement, and local disk concerns that were mentioned. You can pin pods to specifi

Re: Cassandra on JDK 17

2025-05-22 Thread Jon Haddad
I've got 1K nodes in various clusters running on Java 17. I've had great results with Shenandoah, 30GB heap, and off heap trie memtables. Pauses are as advertised, between 1-3ms. With smaller heaps or high throughput you might see the JVM apply pacing - to ensure it can keep up with allocations.

Re: [CFP] Community Over Code NA 2025

2025-04-18 Thread Jon Haddad
Just a reminder! The deadline to submit a proposal closes at 6:59 PM 21 Apr 2025 in Central Daylight Time (UTC-05:00) timezone. Jon On Mon, Mar 10, 2025 at 6:51 AM Paulo Motta wrote: > Hi, > > Please see message below with instructions on submitting talk proposals > for Community Over Code 202

Re: Graduating JDK17 support for Cassandra 5.x from experimental to production ready

2025-04-09 Thread Jon Haddad
Thanks for bringing this up. I've been meaning to for a while now. I'm at the early stages of doing some testing of Java 11 G1GC vs 17 w/ G1GC vs 17 w/ Shenandoah. I plan on sharing my results with the wider community, and if things look good I'll be proposing we remove the experimental label.

Re: [External]Cassandra 5.0: Any Official Tests Supporting 'Free Performance Gains'

2025-03-20 Thread Jon Haddad
I’m not sure if i shared this to the user list… I’m doing a massive series on C* 5.0 performance and how it relates to node density and cost. First post is up now. http://rustyrazorblade.com/post/2025/03-streaming/ The benefit any given feature depends on a lot of factors. Hardware and workload v

Re: Recommended Cassandra Version for POC & AWS EC2 Graviton Compatibility

2025-03-06 Thread Jon Haddad
There's several options for load testing. I'll admit that I'm massively biased as I wrote one of them. * easy-cass-stress offers a variety of customizable workloads out of the box and can stress many features without writing any code or learning new config. I wrote this with the goal of getting

Re: Cassandra Memory Spikes - Tuning Suggestions?

2025-02-26 Thread Jon Haddad
Can you explain a bit more what you mean by memory spikes? The defaults we ship use the same settings for min and max JVM heap size, so you should see all the memory allocated to the JVM at startup. Did you change anything here? I don't recommend doing so. If you're referring to files in the pa

Re: Cassandra 5 Upgrade - Storage Compatibility Modes

2024-12-18 Thread Jon Haddad
ch had very good automation for this sort of thing, I can >> still see this process taking 3 times as long to complete as a normal >> upgrade, and this does take up operators time. >> >> I can see the advantages of 3 stage process, and all things being equal I >> would r

Re: Cassandra 5 Upgrade - Storage Compatibility Modes

2024-12-18 Thread Jon Haddad
fficulty of upgrading in general? I'm all for improving it. It's just not what this thread is about. Jon On Wed, Dec 18, 2024 at 10:01 AM Eric Evans wrote: > > > On Wed, Dec 18, 2024 at 11:43 AM Jon Haddad > wrote: > >> > We (Wikimedia) have had more

Re: Cassandra 5 Upgrade - Storage Compatibility Modes

2024-12-18 Thread Jon Haddad
thing, I can >> still see this process taking 3 times as long to complete as a normal >> upgrade, and this does take up operators time. >> >> I can see the advantages of 3 stage process, and all things being equal I >> would recommend that process as being safe

Re: Cassandra 5 Upgrade - Storage Compatibility Modes

2024-12-17 Thread Jon Haddad
#x27;s question - clients (i.e. java driver, etc) should be able to > handle disconnects gracefully and route to other coordinators leaving the > application-facing symptom being a blip on latency. Are you seeing > something else more painful, or is it more just not having the built-in &

Re: Cassandra 5 Upgrade - Storage Compatibility Modes

2024-12-17 Thread Jon Haddad
age process, and all things being equal I > would recommend that process as being safer, however I am getting a lot of > push back whenever we discuss the upgrade process. > > Thanks > > Paul > > > On 17 Dec 2024, at 19:24, Jon Haddad wrote: > > > > Just cu

Re: Cassandra 5 Upgrade - Storage Compatibility Modes

2024-12-17 Thread Jon Haddad
Just curious, why is a rolling restart difficult? Is it a tooling issue, stability, just overall fear of messing with things? You *should* be able to do a rolling restart without it being an issue. I look at this as a fundamental workflow that every C* operator should have available, and you

Re: vector search question - 5.02

2024-11-14 Thread Jon Haddad
It sounds like enabling the JDK's vector preview api could significantly improve Vector search. I haven't verified this myself, but it might be worth trying Java 17 + this flag: --add-modules jdk.incubator.vector I'd love to hear how much of a difference this makes. Jon On Fri, Nov 8, 2024 at

Re: Unexplained stuck memtable flush

2024-11-10 Thread Jon Haddad
I think this is the correct explanation. It's very similar to CASSANDRA-19576, where compaction is unable to finish because we can't insert into compaction history. Really good analysis, Jaydeep. Jon On Sun, Nov 10, 2024 at 1:51 PM Jaydeep Chovatia wrote: > Upon further studying the thread d

Re: Unexplained stuck memtable flush

2024-11-05 Thread Jon Haddad
#x27;s a large number of snapshots. These can > have significant impact on file deletion performance on ZFS. Also worth > checking the disks, I have seen broken disks that stuck on some operations, > e.g. when a specific sector is being read, and this will certainly affect > the filesyste

Re: Unexplained stuck memtable flush

2024-11-05 Thread Jon Haddad
I ran into this a few months ago, and in my case I tracked it down to an issue with ZFS not unlinking commitlogs properly. https://issues.apache.org/jira/browse/CASSANDRA-19564 On Tue, Nov 5, 2024 at 6:05 AM Dmitry Konstantinov wrote: > I am speaking about a thread dump (stack traces for all th

Re: Tombstone Generation in Cassandra 4.1.3 Despite No Update/Delete Operations

2024-10-09 Thread Jon Haddad
Overwriting non collections does not generate tombstones on compaction. — Jon Haddad Rustyrazorblade Consulting rustyrazorblade.com On Wed, Oct 9, 2024 at 9:57 AM James Shaw wrote: > Hi, Naman: > How does the client side load large amounts of data ? Most > likely it ha

Re: Resources on Using Single Vnode in Cassandra

2024-10-09 Thread Jon Haddad
I've worked with a few hundred teams now, including the major ones that used single token (Apple, Netflix, Spotify), and pretty much all the rest used some form of vnodes. Jeff did a good job of summarizing the tradeoffs and I don't have anything to add. I would never, ever, recommend > 4 tokens.

Re: Tombstone Generation in Cassandra 4.1.3 Despite No Update/Delete Operations

2024-10-08 Thread Jon Haddad
Are you using collections? — Jon Haddad Rustyrazorblade Consulting rustyrazorblade.com On Tue, Oct 8, 2024 at 10:52 PM Naman kaushik wrote: > Hi Community, > > We are currently using Cassandra version 4.1.3 and have encountered an > issue related to tombstone generation. We hav

Re: Recommend Cassandra consultant

2024-09-27 Thread Jon Haddad
Thank you both for the recommendation! Jon On Fri, Sep 27, 2024 at 5:12 AM Aaron Ploetz wrote: > Casting a second vote for Jon Haddad. You can reach out to him on > LinkedIn: https://www.linkedin.com/in/rustyrazorblade/ > > Thanks, > > Aaron > > > On Fri, Sep 27,

Re: 5.0 webinar - Thursday August 22 9am PDT

2024-09-17 Thread Jon Haddad
Hey Matthias, Somehow this slipped by me. Here's the video: https://youtube.com/live/-i-ox-8q0nI Jon On Thu, Aug 29, 2024 at 3:31 AM Matthias Pfau via user < user@cassandra.apache.org> wrote: > Hey Jon, > just saw this and wondered if a recording is available? > > Thanks! > > Matthias > > > Au

Re: [RELEASE] Apache Cassandra 4.1.6 released

2024-08-19 Thread Jon Haddad
Riding on the coattails of the release... If folks are interested in trying this release but you don't have the tooling to spin it up, I've already released an update to easy-cass-lab [1] to support 4.1.6. It's my tooling that lets you get started with Apache Cassandra in AWS in under 15 minutes.

5.0 webinar - Thursday August 22 9am PDT

2024-08-13 Thread Jon Haddad
Hey everyone!! With 5.0 right around the corner I'm sure you're wondering if there's a good reason for you to upgrade, or if you should just ignore it and spend your time doing other things instead. ARE YOU KIDDING ME?!?!?! THIS RELEASE IS HUGE. Come join me as I take an hour to cover some of t

easy-cass-lab supports 5.0 RC1

2024-07-23 Thread Jon Haddad
Hey everyone! If you've already seen the news that 5.0 RC-1 is out, you might be wondering how you can kick the tires without investing a ton of time into changing your tooling around. I've got you covered :) I've just released an update to easy-cass-lab [1], my tooling to spin up Cassandra envi

Re: Java21 support for Cassandra

2024-06-17 Thread Jon Haddad
Yes, 5.0 already works with 17 and I’ve been running it almost exclusively in my tests. — Jon Haddad Rustyrazorblade Consulting rustyrazorblade.com On Mon, Jun 17, 2024 at 1:08 PM manish khandelwal < manishkhandelwa...@gmail.com> wrote: > Thans Erick, Jon and Stefan for the respons

Re: Java21 support for Cassandra

2024-06-17 Thread Jon Haddad
Erick, that blanket statement about it “never” going in 5.1 is incorrect. We could absolutely add Java 21 support in 5.1 if we wanted to. — Jon Haddad Rustyrazorblade Consulting rustyrazorblade.com On Mon, Jun 17, 2024 at 11:26 AM Erick Ramirez wrote: > Cassandra 5.0 will only work with J

Re: TWCS Log Warning

2024-05-23 Thread Jon Haddad
As an aside, if you're not putting a TTL on your data, it's a good idea to be proactive and use multiple tables. For example, one per month or year. This allows you the flexibility to delete your data by dropping old tables. Storing old data in Cassandra is expensive. Once you get to a certain p

Re: Replication factor, LOCAL_QUORUM write consistency and materialized views

2024-05-17 Thread Jon Haddad
I strongly suggest you don't use materialized views at all. There are edge cases that in my opinion make them unsuitable for production, both in terms of cluster stability as well as data integrity. Jon On Fri, May 17, 2024 at 8:58 AM Gábor Auth wrote: > Hi, > > I know, I know, the materialize

Re: Change num_tokens in a live cluster

2024-05-16 Thread Jon Haddad
Unless your cluster is very small, using the method of adding / removing nodes will eventually result in putting a much larger portion of your dataset on a very few number of nodes. I *highly* discourage this. The only correct, safe path is Bowen's suggestion of adding another DC and decommission

Re: storage engine series

2024-04-30 Thread Jon Haddad
Thanks Aaron! Just realized I made a mistake, the 4th week's URL is https://www.youtube.com/watch?v=MAxQ0QygcKk. Jon On Tue, Apr 30, 2024 at 4:58 AM Aaron Ploetz wrote: > Nice! This sounds awesome, Jon. > > On Mon, Apr 29, 2024 at 6:25 PM Jon Haddad wrote: > >> Hey ev

storage engine series

2024-04-29 Thread Jon Haddad
Hey everyone, I'm doing a 4 week YouTube series on the C* storage engine. My first video was last week where I gave an overview into some of the storage engine internals [1]. The next 3 weeks are looking at the new Trie indexes coming in 5.0 [2], running Cassandra on EBS [3], and finally looking

Trie Memtables

2024-04-09 Thread Jon Haddad
Hey all, Tomorrow at 10:30am PDT I'm taking a look at Trie Memtables tomorrow on my live stream. I'll do some performance comparisons between it and the legacy SkipListMemtable implementation and see what I can learn. https://www.youtube.com/live/Jp5R_-uXORQ?si=NnIoV3jqjHFoD8nF or if you prefer

Re: Datacenter decommissioning on Cassandra 4.1.4

2024-04-08 Thread Jon Haddad
You shouldn’t decom an entire DC before removing it from replication. — Jon Haddad Rustyrazorblade Consulting rustyrazorblade.com On Mon, Apr 8, 2024 at 6:26 AM Michalis Kotsiouros (EXT) via user < user@cassandra.apache.org> wrote: > Hello community, > > In our deployments, we

Re: Query on Performance Dip

2024-04-05 Thread Jon Haddad
Try changing the chunk length parameter on the compression settings to 4kb, and reduce read ahead to 16kb if you’re using EBS or 4KB if you’re using decent local ssd or nvme. Counters read before write. — Jon Haddad Rustyrazorblade Consulting rustyrazorblade.com On Fri, Apr 5, 2024 at 9:27 AM

Re: Query on Performance Dip

2024-03-30 Thread Jon Haddad
Hi, Unfortunately, the numbers you're posting have no meaning without context. The speculative retries could be the cause of a problem, or you could simply be executing enough queries and you have a fairly high variance in latency which triggers them often. It's unclear how many queries / second

Tomorrow 10AM PDT - Examining LWT perf in 5.0

2024-03-19 Thread Jon Haddad
Hey folks, I'm doing a working session tomorrow at 10am PDT, testing LWTs in C* 5.0. I'll be running benchmarks and doing some performance analysis. Come hang out and bring your questions! Jon YouTube: https://www.youtube.com/watch?v=IoWh647LRQ0 LinkedIn: https://www.linkedin.com/events/cassan

Streaming a working session with 5.0 - UCS

2024-03-05 Thread Jon Haddad
Hey everyone, Today starting at 10am PT I'm going to be streaming my session messing with 5.0, looking at UCS. I'm doing this with my easy-cass-lab and easy-cass-stress tools using a build of C* from last night. I'll also show some of the cool things you can do with my tools. I'll be running th

Re: Check out new features in K8ssandra and Mission Control

2024-02-27 Thread Jon Haddad
Hey Chris - this looks pretty interesting! It looks like there's a lot of functionality in here. * What aspects of Mission Control are dependent on using K8ssandra? * Can Mission Control work without K8ssandra? * Is mission control open source? * I'm not familiar with Vector - does it require an

stress testing & lab provisioning tools

2024-02-26 Thread Jon Haddad
Hey everyone, Over the last several months I've put a lot of work into 2 projects I started back at The Last Pickle, for stress testing Cassandra and for building labs in AWS. You may know them as tlp-stress and tlp-cluster. Since I haven't worked at TLP in almost half a decade, and am the prima

Re: Remove folders of deleted tables

2023-12-05 Thread Jon Haddad
I can't think of a reason to keep empty directories around, seems like a reasonable change, but I don't think you're butting up against a thing that most people would run into, as snapshots are enabled by default (auto_snapshot: true) and almost nobody changes it. The use case you described i

Re: Memory and caches

2023-11-27 Thread Jon Haddad
I haven't found chunk cache to be particularly useful. It's a fairly small cache that could only help when you're dealing with a small hot dataset. I wouldn't bother increasing memory for it. Key cache can be helpful, but it depends on the workload. I generally recommend optimizing for your

Re: Running Large Clusters in Production

2020-07-10 Thread Jon Haddad
I worked on a handful of large clusters (> 200 nodes) using vnodes, and there were some serious issues with both performance and availability. We had to put in a LOT of work to fix the problems. I agree with Jeff - it's way better to manage multiple clusters than a really large one. On Fri, Jul

Re: Upgrading cassandra cluster from 2.1 to 3.X when using custom TWCS

2020-07-09 Thread Jon Haddad
You could also pull TWCS out of the version of Cassandra you want to deploy, fix the imports and change the package name. Then you've got the same version as OSS, just under the name you're using in 2.1. Once you've moved to 3.11, you can switch to the OSS version. On Thu, Jul 9, 2020 at 9:09 AM

Re: Cassandra upgrade from 3.11.3 -> 3.11.6

2020-06-24 Thread Jon Haddad
Generally speaking, don't run mixed versions longer than you have to, and don't upgrade that way. Why? * We don't support it. * We don't even test it. * If you run into trouble and ask for help, the first thing people will tell you is to get all nodes on the same version. Anyone that's doing so

Re: Generating evenly distributed tokens for vnodes

2020-05-29 Thread Jon Haddad
I'm on mobile now so I might be mistaken, but I don't think nodetool move works with multiple tokens On Fri, May 29, 2020, 1:48 PM Kornel Pal wrote: > Hi Anthony, > > Thank you very much for looking into using the script for initial token > generation and for providing multiple detailed methods

Re: Performance of Data Types used for Primary keys

2020-03-06 Thread Jon Haddad
It's not going to matter at all. On Fri, Mar 6, 2020, 2:15 AM Hanauer, Arnulf, Vodacom South Africa (External) wrote: > Hi Cassandra folks, > > > > Is there any difference in performance of general operations if using a > TEXT based Primary key versus a BIGINT Primary key. > > > > Our use-case r

Re: Deleting data from future

2020-03-02 Thread Jon Haddad
You can issue a delete using a future timestamp. http://cassandra.apache.org/doc/latest/cql/dml.html#grammar-token-update-parameter Look for USING TIMESTAMP. Jon On Mon, Mar 2, 2020, 3:28 AM Furkan Cifci wrote: > Greetings, > In our C* cluster, one node lost time sync and it went to future(16

Re: Should we use Materialised Views or ditch them ?

2020-02-28 Thread Jon Haddad
I also recommend avoiding them. I've seen too many clusters fall over as a result of their usage. On Fri, Feb 28, 2020 at 9:52 AM Max C. wrote: > The general view of the community is that you should *NOT* use them in > production, due to multiple serious outstanding issues (see Jira). We used

Re: New seed node in the cluster immediately UN without passing for UJ state

2020-02-13 Thread Jon Haddad
Seeds don't bootstrap, don't list new nodes as seeds. On Thu, Feb 13, 2020 at 5:23 PM Sergio wrote: > Hi guys! > > I don't know how but this is the first time that I see such behavior. I > wanted to add a new node in the cluster and it looks to be working fine but > instead to wait for 2-3 hours

Re: [EXTERNAL] Cassandra 3.11.X upgrades

2020-02-12 Thread Jon Haddad
A while ago, on my first cluster, I decided to do an upgrade by adding nodes running 1.2 to an existing cluster running version 1.1. This was a bad decision, and at that point I decided to always play it safe and always stick to a single version, and never bootstrap in a node running different ver

Re: [RELEASE] Apache Cassandra 4.0-alpha3 released

2020-02-07 Thread Jon Haddad
Thanks for handling this, Mick! On Fri, Feb 7, 2020 at 12:02 PM Mick Semb Wever wrote: > > > The Cassandra team is pleased to announce the release of Apache Cassandra > version 4.0-alpha3. > > Apache Cassandra is a fully distributed database. It is the right choice > when you need scalability an

Re: Overload because of hint pressure + MVs

2020-02-07 Thread Jon Haddad
There's a few things you can do here that might help. First off, if you're using the default heap settings, that's a serious problem. If you've got the head room, my recommendation is to use 16GB heap with 12 GB new gen and pin your memtable heap space to 2GB. Set your max tenuring threshold to

Re: Question on large partition key

2019-12-31 Thread Jon Haddad
I suggest checking out Aaron Morton's post on the 3.0 storage engine. https://thelastpickle.com/blog/2016/03/04/introductiont-to-the-apache-cassandra-3-storage-engine.html On Tue, Dec 31, 2019 at 11:20 AM Subroto Barua wrote: > I have a table --- > > create Table mytable ( > > Id text, > > cdat

Re: Streaming Failed during bootstrap of a Replacement node

2019-12-20 Thread Jon Haddad
n, > > Yes we will upgrade it soon. But before we can upgrade shouldn’t we get > this lost node in the cluster to be replaced ? > > > > > > > > *From:* Jon Haddad > *Sent:* Friday, December 20, 2019 2:13 PM > *To:* user@cassandra.apache.org > *Subject:* Re:

Re: Streaming Failed during bootstrap of a Replacement node

2019-12-20 Thread Jon Haddad
You should upgrade to Cassandra 3.11.5 before doing anything else. You're running a pretty old and buggy version. There's been hundreds (maybe thousands) of bugs fixed between 3.3 and 3.11.5. On Fri, Dec 20, 2019 at 10:46 AM Nethi, Manoj wrote: > Hi, > > > > We are seeing the following error w

Re: execute is faster than execute_async?

2019-12-11 Thread Jon Haddad
I'm not sure how you're measuring this - could you share your benchmarking code? I ask because execute calls execute_async under the hood: https://github.com/datastax/python-driver/blob/master/cassandra/cluster.py#L2316 I tested the python driver a ways back and found some weird behavior due to t

Re: Connection Pooling in v4.x Java Driver

2019-12-10 Thread Jon Haddad
I'm not sure how closely the driver maintainers are following this list. You might want to ask on the Java Driver mailing list: https://groups.google.com/a/lists.datastax.com/forum/#!forum/java-driver-user On Tue, Dec 10, 2019 at 5:10 PM Caravaggio, Kevin < kevin.caravag...@lowes.com> wrote: >

Re: AWS ephemeral instances + backup

2019-12-05 Thread Jon Haddad
You can easily do this with bcache or LVM http://rustyrazorblade.com/post/2018/2018-04-24-intro-to-lvm/. Medusa might be a good route to go down if you want to do backups instead: https://thelastpickle.com/blog/2019/11/05/cassandra-medusa-backup-tool-is-open-source.html On Thu, Dec 5, 2019 at 1

Re: Cassandra 4 alpha/alpha2

2019-11-01 Thread Jon Haddad
A new thing like this would be much better served by the community through several iterations. For instance, over the last year I've developed a tool for spinning up lab clusters, it's here: https://thelastpickle.com/tlp-cluster/ I had to make a *lot* of tradeoffs here. Everything Jeff mentioned

Re: Cassandra 4 alpha/alpha2

2019-10-31 Thread Jon Haddad
What artifact did you use and what OS are you on? On Thu, Oct 31, 2019 at 12:40 PM Abdul Patel wrote: > Hey Everyone > > Did anyone was successfull to install either alpha or alpha2 version for > cassandra 4.0? > Found 2 issues : > 1> cassandra-env.sh: > JAVA_VERSION varianle is not defined. > J

Re: What is the status of counters? Should I use them?

2019-10-30 Thread Jon Haddad
table on a regular basis as any > other? > > > > ‐‐‐ Original Message ‐‐‐ > On Wednesday, 30 October 2019 16:26, Jon Haddad wrote: > > Counters are good for things like page views, bad for money. Yes they can > under or overcount in certain situations. If your cluster is

Re: Where to get old RPMs?

2019-10-30 Thread Jon Haddad
Archives are here: http://archive.apache.org/dist/cassandra/ For example, the RPM for 3.11.x you can find here: http://archive.apache.org/dist/cassandra/redhat/311x/ The old releases are removed by Apache automatically as part of their policy, it's not specific to Cassandra. On Wed, Oct 30, 201

Re: What is the status of counters? Should I use them?

2019-10-30 Thread Jon Haddad
Counters are good for things like page views, bad for money. Yes they can under or overcount in certain situations. If your cluster is stable, you'll see very little of it in practice. I've done quite a bit of tuning of counters. Here's the main takeaways: * They do a read before a write, so u

Re: TWCS and gc_grace_seconds

2019-10-26 Thread Jon Haddad
My coworker Radovan wrote up a post on the relationship between gc grace and hinted handoff: https://thelastpickle.com/blog/2018/03/21/hinted-handoff-gc-grace-demystified.html Jon On Sat, Oct 26, 2019 at 6:45 AM Hossein Ghiyasi Mehr wrote: > It needs to change gc_grace_seconds carefully because

Re: Repair Issues

2019-10-24 Thread Jon Haddad
There's some major warning signs for me with your environment. 4GB heap is too low, and Cassandra 3.7 isn't something I would put into production. Your surface area for problems is massive right now. Things I'd do: 1. Never use incremental repair. Seems like you've already stopped doing them,

Re: merge two cluster

2019-10-23 Thread Jon Haddad
Probably not beneficial, I wouldn't do it. Not a fan of multi-tenancy with Cassandra unless the use cases are so small that your noisy neighbor problem is not very noisy at all. For those cases I don't know what you get from Cassandra other than a cool resume. On Wed, Oct 23, 2019 at 12:41 PM Re

Re: Cassandra Rack - Datacenter Load Balancing relations

2019-10-23 Thread Jon Haddad
t-1a >- 4 write TWO us-east-1b 5 write TWO us-east-1b >- 6 write TWO us-east-1b > > > Here we have 2 DC read and write > One Rack per DC > One Availability Zone per DC > > Thanks, > > Sergio > > > On Wed, Oct 23, 2019, 1:11 PM Jon Haddad wr

Re: Cassandra Rack - Datacenter Load Balancing relations

2019-10-23 Thread Jon Haddad
Personally, I wouldn't ever do this. I recommend separate DCs if you want to keep workloads separate. On Wed, Oct 23, 2019 at 4:06 PM Sergio wrote: > I forgot to comment for > >OPTION C) >1. Node DC RACK AZ 1 read ONE us-east-1a 2 read ONE us-east-1b >2. 3 read ONE us-east

Re: Cassandra 2.1.18 - Question on stream/bootstrap throughput

2019-10-22 Thread Jon Haddad
CPU waiting on memory will look like CPU overhead. There's a good post on the topic by Brendan Gregg: http://www.brendangregg.com/blog/2017-05-09/cpu-utilization-is-wrong.html Regarding GC, I agree with Reid. You're probably not going to saturate your network card no matter what your settings,

Re: [EXTERNAL] Re: GC Tuning https://thelastpickle.com/blog/2018/04/11/gc-tuning.html

2019-10-21 Thread Jon Haddad
ospel truth that -XX:+UseNUMA is a good >> thing on AWS (or anything virtualized), you’d have to run your own tests >> and find out. >> >> >> >> R >> >> *From: *Jon Haddad >> *Reply-To: *"user@cassandra.apache.org" >> *Date: *Mo

Re: [EXTERNAL] Re: GC Tuning https://thelastpickle.com/blog/2018/04/11/gc-tuning.html

2019-10-21 Thread Jon Haddad
testing harnesses. It isn’t worth our time. As a previous > writer mentioned, there is usually better return on our time tuning the > schema (aka helping developers understand Cassandra’s strengths). > > > > We use 16 – 32 GB heaps, nothing smaller than that. > > >

Re: GC Tuning https://thelastpickle.com/blog/2018/04/11/gc-tuning.html

2019-10-21 Thread Jon Haddad
I still use ParNew + CMS over G1GC with Java 8. I haven't done a comparison with JDK 11 yet, so I'm not sure if it's any better. I've heard it is, but I like to verify first. The pause times with ParNew + CMS are generally lower than G1 when tuned right, but as Chris said it can be tricky. If y

Re: Elevated response times from all nodes in a data center at the same time.

2019-10-16 Thread Jon Haddad
It's possible the queries you're normally running are served out of page cache, and during the latency spike you're hitting your disks. If you're using read ahead you might be hitting a throughput limit on the disks. I've got some numbers and graphs I can share later when I'm not on my phone. Jon

Re: cluster rolling restart

2019-10-16 Thread Jon Haddad
I agree with Jeff here. Ideally you should be so comfortable with rolling restarts that they become second nature. Cassandra is designed to handle them and you should not be afraid to do them regularly. On Wed, Oct 16, 2019, 8:06 AM Jeff Jirsa wrote: > > Personally I encourage you to rolling res

Re: Update/where statement Adds Row

2019-09-11 Thread Jon Haddad
Probably not a great idea unless you're using it sparingly. Using LWTs without knowing all the caveats is likely to lead to terrible cluster performance. On Wed, Sep 11, 2019, 10:59 PM A wrote: > Is it ok if I do this? > > ... where email = em AND company_id = id IF EXISTS > > > > > > Sent fr

Re: Is it possible to build multi cloud cluster for Cassandra

2019-09-05 Thread Jon Haddad
Technically, not a problem. Use GossipingPropertyFileSnitch to keep things simple and you can go across whatever cloud providers you want without issue. The biggest issue you're going to have isn't going to be Cassandra, it's having the expertise in the different cloud providers to understand the

Re: New column

2019-08-22 Thread Jon Haddad
Just to close the loop on this, I did a release of tlp-stress last night, which now has this workload (AllowFiltering). You can grab a deb, rpm, tarball or docker image. Docs are here: http://thelastpickle.com/tlp-stress/ Jon On Mon, Aug 19, 2019 at 2:21 PM Jon Haddad wrote: > It'll

Re: Disk space utilization by from some Cassandra

2019-08-21 Thread Jon Haddad
This advice hasn't been valid for a long time now for most use cases. The only time you need to reserve 50% disk space is if you're going to be running major compactions against a table in your cluster that occupies 50% of its total disk space. Nowadays, that's far less common than it was when yo

Re: New column

2019-08-19 Thread Jon Haddad
gt; Jon, > > If we expect non of our partition key to have more than 100 records and > pass partition key in where clause we wouldnt see issues using new column > and allow filtering? Can you please point me to any doc how allow > filtering works. I was in assumption of it goes through

Re: New column

2019-08-18 Thread Jon Haddad
If you're giving the partition key you won't scan the whole table. The overhead will depend on the size or the partition. Would be an interesting workload for our tlp-stress tool, I'll code something up for the next release. On Sun, Aug 18, 2019, 12:58 PM Rahul Reddy wrote: > Hello, > > We have

Re: Datafile Corruption

2019-08-08 Thread Jon Haddad
Any chance you're using NVMe with an older Linux kernel? I've seen a *lot* filesystem errors from using older CentOS versions. You'll want to be using a version > 4.15. On Thu, Aug 8, 2019 at 9:31 AM Philip Ó Condúin wrote: > *@Jeff *- If it was hardware that would explain it all, but do you t

Re: Cassandra read requests not getting timeout

2019-08-05 Thread Jon Haddad
I think this might be because the timeout only applied to each request, and the driver is paginating in the background. Each page is a new request. On Mon, Aug 5, 2019, 12:08 AM Oleksandr Shulgin < oleksandr.shul...@zalando.de> wrote: > On Mon, Aug 5, 2019 at 8:50 AM nokia ceph > wrote: > >> Hi

Re: Cheat Sheet for Unix based OS, Performance troubleshooting

2019-07-28 Thread Jon Haddad
http://www.brendangregg.com/linuxperf.html On Sat, Jul 27, 2019 at 2:45 AM Paul Chandler wrote: > I have always found Amy's Cassandra 2.1 tuning guide great for the Linux > performance tuning: > https://tobert.github.io/pages/als-cassandra-21-tuning-guide.html > > Sent from my iPhone > > On 26 J

Re: Performance impact with ALLOW FILTERING clause.

2019-07-25 Thread Jon Haddad
If you're thinking about rewriting your data to be more performant when doing analytics, you might as well go the distance and put it in an analytics friendly format like Parquet. My 2 cents. On Thu, Jul 25, 2019 at 11:01 AM ZAIDI, ASAD A wrote: > Thank you all for your insights. > > > > When s

Re: Materialized View's additional PrimaryKey column

2019-07-25 Thread Jon Haddad
for using MV was avoiding updates (delete + > create) on primaryKey columns because we suppose that cassandra developers > can manage this unpreferred operation better then us. I'm really confused > now. > > > > On Wednesday, July 24, 2019, 11:30:15 PM GMT+3, Jon Haddad

Re: Materialized View's additional PrimaryKey column

2019-07-24 Thread Jon Haddad
I really, really advise against using MVs. I've had to help a number of teams move off them. Not sure what list of bugs you read, but if the list didn't include "will destabilize your cluster to the point of constant downtime" then the list was incomplete. Jon On Wed, Jul 24, 2019 at 6:32 AM me

Re: Compaction throughput

2019-07-19 Thread Jon Haddad
It's a limit on the total compaction throughput. On Fri, Jul 19, 2019 at 10:39 AM Vlad wrote: > Hi, > > is 'nodetool setcompactionthroughput' sets limit for all compactions on > the node, or is it per compaction thread? > > Thanks. >

Re: Running Node Repair After Changing RF or Replication Strategy for a Keyspace

2019-06-28 Thread Jon Haddad
Yep - not to mention the increased complexity and overhead of going from ONE to QUORUM, or the increased cost of QUORUM in RF=5 vs RF=3. If you're in a cloud provider, I've found you're almost always better off adding a new DC with a higher RF, assuming you're on NTS like Jeff mentioned. On Fri,

Re: Recover lost node from backup or evict/re-add?

2019-06-12 Thread Jon Haddad
100% agree with Sean. I would only use Cassandra backups in a case where you need to restore from full cluster loss. Example: An entire DC burns down, tornado, flooding. Your routine node replacement after a failure should be replace_address_first_boot. To ensure this goes smoothly, run regular

Re: Collecting Latency Metrics

2019-05-30 Thread Jon Haddad
Yep. I would *never* use mean when it comes to performance to make any sort of decisions. I prefer to graph all the p99 latencies as well as the max. Some good reading on the topic: https://bravenewgeek.com/everything-you-know-about-latency-is-wrong/ On Thu, May 30, 2019 at 7:35 AM Chris Lohfin

Re: Re: How to set up a cluster with allocate_tokens_for_keyspace?

2019-05-05 Thread Jon Haddad
. > > Sent using Zoho Mail > > > > ==== Forwarded message > From: Jon Haddad > To: > Date: Sat, 04 May 2019 22:10:39 +0430 > Subject: Re: How to set up a cluster with allocate_tokens_for_keyspace? > Forwarded message > > That line is only

Re: Priority in IN () cqlsh comand

2019-05-05 Thread Jon Haddad
Do separate queries for each partition you want. There's no benefit in using the IN() clause here, and performance is significantly worse with multi-partition IN(), especially if the partitions are small. On Sun, May 5, 2019 at 4:52 AM Soheil Pourbafrani wrote: > > Hi, > > I want to run cqlsh qu

Re: How to set up a cluster with allocate_tokens_for_keyspace?

2019-05-04 Thread Jon Haddad
That line is only relevant for when you're starting your cluster and you need to define your initial tokens in a non-random way. Random token distribution doesn't work very well when you only use 4 tokens. Once you get the cluster set up you don't need to specify tokens anymore, you can just use

Re: Increasing the size limits implications

2019-04-30 Thread Jon Haddad
Just curious - why are you using such large batches? Most of the time when someone asks this question, it's because they're using batches as they would in an RDBMS, because larger transactions improve performance. That doesn't apply with Cassandra. Batches are OK at keeping multiple tables in sy

Re: [EXTERNAL] multiple Cassandra instances per server, possible?

2019-04-18 Thread Jon Haddad
Agreed with Jeff here. The whole "community recommends no more than 1TB" has been around, and inaccurate, for a long time. The biggest issue with dense nodes is how long it takes to replace them. 4.0 should help with that under certain circumstances. On Thu, Apr 18, 2019 at 6:57 AM Jeff Jirsa

Re: 2.1.9 --> 2.2.13 upgrade node startup after upgrade very slow

2019-04-17 Thread Jon Haddad
Let me be more specific - run the async java profiler and generate a flame graph to determine where CPU time is spent. On Wed, Apr 17, 2019 at 11:36 AM Jon Haddad wrote: > > Run the async java profiler on the node to determine what it's doing: > https://github.com/jvm-profili

Re: 2.1.9 --> 2.2.13 upgrade node startup after upgrade very slow

2019-04-17 Thread Jon Haddad
Run the async java profiler on the node to determine what it's doing: https://github.com/jvm-profiling-tools/async-profiler On Wed, Apr 17, 2019 at 11:31 AM Carl Mueller wrote: > > No, we just did the package upgrade 2.1.9 --> 2.2.13 > > It definitely feels like some indexes are being recalculate

  1   2   3   >