How do you I check if my cassandra cluster is performing at its optimum level
> > > Hi all, I am trying to push my sprak processed data to 3 node cluster of C*. I am pushing 200 million records to cassandra it is taking 2 hours Below it the my spark cluster configuration Nodes : 12 vCores Total : 112 Total memory : 1.5 TB. spark.cassandra.connection.port 9042 spark.cassandra.output.batch.grouping.buffer.size 1000 spark.cassandra.output.batch.size.bytes 2056 spark.cassandra.output.concurrent.writes 1500 spark.cassandra.output.throughput_mb_per_sec 5 spark.driver.cores 1 spark.driver.extraClassPath . spark.driver.extraJavaOptions Dlog4j.debug spark.driver.host nj11mhf0068 spark.driver.memory 2g spark.driver.memoryOverhead 512 spark.driver.port 54303 spark.dynamicAllocation.enabled TRUE spark.dynamicAllocation.executorIdleTimeout 180s spark.dynamicAllocation.maxExecutors 21 spark.dynamicAllocation.minExecutors 20 spark.executor.cores 2 spark.executor.extraJavaOptions Dlog4j.configuration=log4j.properties spark.executor.id driver spark.executor.instances 4 spark.executor.memory 4g spark.executor.memoryOverhead 1024 [image: image.png] I have made the spark data frame partitions 10 as below val df = df_raw.repartition(numOfPartitions) // numOfPartitions = 10 But still my application is very slow. Can you please help me what am I doing wrong here ? Regards, Shyam
Cassandra 3.11.x and Java 11
Hello, am I right that Java 11 support will only be in Cassandra 4.0 and not in future releases of the 3.11.x series? Just want to be sure. Thanks, Thomas The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria, Freist?dterstra?e 313
Re: Cassandra 3.11.x and Java 11
Correct. On Mon, Nov 19, 2018 at 1:02 PM Steinmaurer, Thomas < thomas.steinmau...@dynatrace.com> wrote: > Hello, > > am I right that Java 11 support will only be in Cassandra 4.0 and not in > future releases of the 3.11.x series? Just want to be sure. > > Thanks, > Thomas > > The contents of this e-mail are intended for the named addressee only. It > contains information that may be confidential. Unless you are the named > addressee or an authorized designee, you may not copy or use it, or > disclose it to anyone else. If you received it in error please notify us > immediately and then destroy it. Dynatrace Austria GmbH (registration > number FN 91482h) is a company registered in Linz whose registered office > is at 4040 Linz, Austria, Freist?dterstra?e 313 > -- Jon Haddad http://www.rustyrazorblade.com twitter: rustyrazorblade
I/O threads busy error
Dear all, I've got a 2 node Cassandra cluster(3.11.2). I'm using Datastax's c++ driver(ver 2.7) in my application to fetch/add data from the cluster(RHEL 6.9). Sometimes I'm getting the error "All connections on all I/O threads are busy" in my application. I'm unable to figure out why I am getting this error infrequently and not everytime. Has anyone faced such errors before? Please let me know any possible reasons behind the issue. Thanks and regards, Vishal Sharma P.S. I don't really know whether this is a Datastax driver's issue or Cassandra's issue that's why I'm posting this query here. "Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s). are confidential and may be privileged. If you are not the intended recipient. you are hereby notified that any review. re-transmission. conversion to hard copy. copying. circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient. please notify the sender immediately by return email. and delete this message and any attachments from your system. Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."
