Re: R/W timeouts VS number of tables in keyspace

2021-07-22 Thread Erick Ramirez
I wanted to add a word of warning that switching to G1 won't necessarily give you breathing space. In fact, I know it definitely won't. In your original post, it looked like the node had a very small heap (2GB). In my experience, you need to allocate at least 8GB of memory to the heap for producti

Re: R/W timeouts VS number of tables in keyspace

2021-07-22 Thread Scott Hirleman
I feel like that calls for an anti-pattern -> success blog post Luca 🤣 On Tue, Jul 20, 2021 at 9:17 AM Luca Rondanini wrote: > Thanks Sean, > > I'm switching to G1 in order to gain some time while refactoring. I should > be able to go down to 4 tables! Yes, the original design was that poor. > >

Re: R/W timeouts VS number of tables in keyspace

2021-07-20 Thread Luca Rondanini
Thanks Sean, I'm switching to G1 in order to gain some time while refactoring. I should be able to go down to 4 tables! Yes, the original design was that poor. Thanks again On Tue, Jul 20, 2021 at 6:41 AM Durity, Sean R wrote: > Each table in the cluster will have a memtable. This is why you d

RE: R/W timeouts VS number of tables in keyspace

2021-07-20 Thread Durity, Sean R
Each table in the cluster will have a memtable. This is why you do not want to fracture the memory into 900+ slices. The rule of thumb I have followed is to stay in the low hundreds (maybe 200) tables for the whole cluster. I would be requiring the hard refactoring (or moving tables to different

RE: Re: R/W timeouts VS number of tables in keyspace

2021-07-19 Thread MyWorld
Yes it seems like a GC issue. And since the default timeout for write/read is 2000ms and 5000ms, you might be experiencing timeout issues. But you need to check reason behind this GC pause. Heavy partition or high tombstones could be one reason. Check your table stats for the same. As suggested you

Re: R/W timeouts VS number of tables in keyspace

2021-07-19 Thread Yakir Gibraltar
In order to tune GC, you need gc.log or jvm metrics, you can check on https://gceasy.io/ and see the results before and after the change. On Mon, Jul 19, 2021 at 7:21 PM Luca Rondanini wrote: > Thanks Yakir, > > I can already experience slow repairs and startups but I'd like to > stabilize the s

Re: R/W timeouts VS number of tables in keyspace

2021-07-19 Thread Luca Rondanini
Thanks Yakir, I can already experience slow repairs and startups but I'd like to stabilize the system before jumping into refactoring (columns are not a problem, max 10/cols per table). Do you believe it's a GC problem to cause the timeouts and crashes? I'll give it a try and update this post. Th

Re: R/W timeouts VS number of tables in keyspace

2021-07-19 Thread Yakir Gibraltar
I recommend rethinking about this design, hard to maintain, slow startup and repair . About GC, try to replace CMS with G1 , see doc : https://docs.datastax.com/en/dse/6.0/dse-admin/datastax_enterprise/operations/opsTuningGcAbout.html BTW, also many columns may affect performance, see doc: https://