Re: HELP with bulk loading

Thank you all! It turns out that the fastest ways are: https://github.com/brianmhess/ cassandra-loader and COPY FROM. So I decided to stick with COPY FROM as it built-in and easy-to-use. On Fri, Mar 10, 2017 at 2:22 PM, Ahmed Eljami wrote: > Hi, > > >3. sstableloader is slow too. Assuming that

Re: HELP with bulk loading

Hi, >3. sstableloader is slow too. Assuming that I have new empty C* cluster, how can I improve the upload speed? Maybe disable replication or some other settings while streaming and then turn it back? Maybe you can accelerate you load with the option -cph (connection per host): https://issues.ap

Re: HELP with bulk loading

When I tested cqlsh COPY FROM for CASSANDRA-11053 , I was able to import about 20 GB in under 4 minutes on a cluster with 8 nodes u

Re: HELP with bulk loading

I suggest using cassandra loader https://github.com/brianmhess/cassandra-loader On Mar 9, 2017 5:30 PM, "Artur R" wrote: > Hello all! > > There are ~500gb of CSV files and I am trying to find the way how to > upload them to C* table (new empty C* cluster of 3 nodes, replication > factor 2) with

HELP with bulk loading

Hello all! There are ~500gb of CSV files and I am trying to find the way how to upload them to C* table (new empty C* cluster of 3 nodes, replication factor 2) within reasonable time (say, 10 hours using 3-4 instance of c3.8xlarge EC2 nodes). My first impulse was to use CQLSSTableWriter, but it i

Re: Bulk loading performance

* Graham Sanderson , 2015-07-13 18:21:08 Mon: > > Is there a set of best practices for this kind of workload? We would > > like to avoid interfering with reads as much as possible. > Ironically in my experience the fastest ways to get data into C* are > considered “anti-patterns” by most (but I hav

Re: Bulk loading performance

Ironically in my experience the fastest ways to get data into C* are considered “anti-patterns” by most (but I have no problem saturating multiple gigabit network links if I really feel like inserting fast) It’s been a while since I tried some of the newer approaches though (my fast load code i

Bulk loading performance

Hi, I have a use case wherein I receive a daily batch of data; it's about 50M--100M records (a record is a list of integers, keyed by a UUID). The target is a 12-node cluster. Using a simple-minded approach (24 batched inserts in parallel, using the Ruby client), while the cluster is being read a

Re: Slow bulk loading

It sounds as though you could be having troubles with Garbage Collection. Check your cassandra system logs and search for "GC". If you see frequent garbage collections taking more than a second or two to complete, you're going to need to do some configuration tweaking. On 05/07/2015 04:44 AM,

Re: Slow bulk loading

> > > > When I upload I notice one core of the cassandra node is full CPU (all > other cores are idleing), > Take a look at the interrupt distribution (cat /proc/interrupts). You'll probably see disk and network interrupts mostly/all bound to CPU0. If that is the case, this article has an excellen

Slow bulk loading

Hi, I m streaming a big sstable using bulk loader of sstableloader but it's very slow (3 Mbytes/sec) : Summary statistics: Connections per host: : 1 Total files transferred: : 1 Total bytes transferred: : 10357947484 Total duration (ms): : 3280229 Average

RE: error in bulk loading

//g Rafael From: Tyler Hobbs [mailto:ty...@datastax.com] Sent: Tuesday, March 24, 2015 5:55 PM To: user@cassandra.apache.org Subject: Re: error in bulk loading On Tue, Mar 24, 2015 at 5:30 AM, Rahul Bhardwaj mailto:rahul.bhard...@indiamart.com>> wrote: I need to import a csv file to a

Re: error in bulk loading

On Tue, Mar 24, 2015 at 5:30 AM, Rahul Bhardwaj < rahul.bhard...@indiamart.com> wrote: > I need to import a csv file to a table using copy command, but file > contains carriage returns which causing me problem in doing so, Is there > any way in cassandra to solve this You can surround the field

error in bulk loading

Hi all, I need to import a csv file to a table using copy command, but file contains carriage returns which causing me problem in doing so, Is there any way in cassandra to solve this Regards: Rahul -- Follow IndiaMART.com for latest updates on this and more:

Re: error while bulk loading using copy command

Thanks Eric.. yes 18 lacs implies 1.8 million. On Thu, Jan 29, 2015 at 9:01 PM, Eric Stevens wrote: > As the error implies, you cannot insert into counters tables, you can only > update them as increments or decrements (updating a counter that doesn't > exist will create it with the initial del

Re: error while bulk loading using copy command

As the error implies, you cannot insert into counters tables, you can only update them as increments or decrements (updating a counter that doesn't exist will create it with the initial delta as if it had started at zero). I would recommend this documentation which describes how to update counters

error while bulk loading using copy command

Hi All, We need to upload 18 lacs rows into a table which consist columns with data type "counter". on uploading using copy command , we are getting below error: *Bad Request: INSERT statement are not allowed on counter tables, use UPDATE instead* we need counter data type because after loading

Re: Data tombstoned during bulk loading 1.2.10 -> 2.0.3

Seems good. I'll discus it with data owners and we choose the best method. Best regards, Aleksander 4 lut 2014 19:40 "Robert Coli" napisał(a): > On Tue, Feb 4, 2014 at 12:21 AM, olek.stas...@gmail.com < > olek.stas...@gmail.com> wrote: > >> I don't know what is the real cause of my problem. We ar

Re: Data tombstoned during bulk loading 1.2.10 -> 2.0.3

On Tue, Feb 4, 2014 at 12:21 AM, olek.stas...@gmail.com < olek.stas...@gmail.com> wrote: > I don't know what is the real cause of my problem. We are still guessing. > All operations I have done one cluster are described on timeline: > 1.1.7-> 1.2.10 -> upgradesstable -> 2.0.2 -> normal operations

Re: Data tombstoned during bulk loading 1.2.10 -> 2.0.3

I don't know what is the real cause of my problem. We are still guessing. All operations I have done one cluster are described on timeline: 1.1.7-> 1.2.10 -> upgradesstable -> 2.0.2 -> normal operations ->2.0.3 -> normal operations -> now normal operations means reads/writes/repairs. Could you plea

Re: Data tombstoned during bulk loading 1.2.10 -> 2.0.3

On Mon, Feb 3, 2014 at 2:17 PM, olek.stas...@gmail.com < olek.stas...@gmail.com> wrote: > No, i've done repair after upgrade sstables. In fact it was about 4 > weeks after, because of bug: > If you only did a repair after you upgraded SSTables, when did you have an opportunity to hit : https://i

Re: Data tombstoned during bulk loading 1.2.10 -> 2.0.3

2014-02-03 Robert Coli : > On Mon, Feb 3, 2014 at 1:02 PM, olek.stas...@gmail.com > wrote: >> >> Today I've noticed that oldest files with broken values appear during >> repair (we do repair once a week on each node). Maybe it's the repair >> operation, which caused data loss? > > > Yes, unless yo

Re: Data tombstoned during bulk loading 1.2.10 -> 2.0.3

On Mon, Feb 3, 2014 at 1:02 PM, olek.stas...@gmail.com < olek.stas...@gmail.com> wrote: > Today I've noticed that oldest files with broken values appear during > repair (we do repair once a week on each node). Maybe it's the repair > operation, which caused data loss? Yes, unless you added or re

Re: Data tombstoned during bulk loading 1.2.10 -> 2.0.3

Yes, I haven't run sstableloader. The data loss apperared somwhere on the line: 1.1.7-> 1.2.10 -> upgradesstable -> 2.0.2 -> normal operations ->2.0.3 normal operations -> now Today I've noticed that oldest files with broken values appear during repair (we do repair once a week on each node). Maybe

Re: Data tombstoned during bulk loading 1.2.10 -> 2.0.3

On Mon, Feb 3, 2014 at 12:51 AM, olek.stas...@gmail.com < olek.stas...@gmail.com> wrote: > We've faced very similar effect after upgrade from 1.1.7 to 2.0 (via > 1.2.10). Probably after upgradesstable (but it's only a guess, > because we noticed problem few weeks later), some rows became > tombst

Re: Data tombstoned during bulk loading 1.2.10 -> 2.0.3

Ok, but will upgrade "resurrect" my data? Or maybe I should perform additional action to bring my system to correct state? best regards Aleksander 3 lut 2014 17:08 "Yuki Morishita" napisał(a): > if you are using < 2.0.4, then you are hitting > https://issues.apache.org/jira/browse/CASSANDRA-652

Re: Data tombstoned during bulk loading 1.2.10 -> 2.0.3

if you are using < 2.0.4, then you are hitting https://issues.apache.org/jira/browse/CASSANDRA-6527 On Mon, Feb 3, 2014 at 2:51 AM, olek.stas...@gmail.com wrote: > Hi All, > We've faced very similar effect after upgrade from 1.1.7 to 2.0 (via > 1.2.10). Probably after upgradesstable (but it's o

Re: Data tombstoned during bulk loading 1.2.10 -> 2.0.3

Hi All, We've faced very similar effect after upgrade from 1.1.7 to 2.0 (via 1.2.10). Probably after upgradesstable (but it's only a guess, because we noticed problem few weeks later), some rows became tombstoned. They just disappear from results of queries. After inverstigation I've noticed, that

Re: Data tombstoned during bulk loading 1.2.10 -> 2.0.3

On Wed, Dec 11, 2013 at 6:27 AM, Mathijs Vogelzang wrote: > When I use sstable2json on the sstable on the destination cluster, it has > "metadata": {"deletionInfo": > {"markedForDeleteAt":1796952039620607,"localDeletionTime":0}}, whereas > it doesn't have that in the source sstable. > (Yes, this i

Data tombstoned during bulk loading 1.2.10 -> 2.0.3

Hi all, We're running into a weird problem trying to migrate our data from a 1.2.10 cluster to a 2.0.3 one. I've taken a snapshot on the old cluster, and for each host there, I'm running sstableloader -d KEYSPACE/COLUMNFAMILY (the sstableloader process from the 2.0.3 distribution, the one from 1

Re: The performance difference of online bulk insertion and the file-based bulk loading

On 10/15/2013 08:41 AM, José Elias Queiroga da Costa Araújo wrote: - is that is there a way that we can warm-up the cache, after the file-based bulk loading, so that we can allow the data to be cached first in the memory, and then afterwards, when we issue the bulk retrieval, the performance can

The performance difference of online bulk insertion and the file-based bulk loading

-based bulk loading. Alternatively, we used the Hector API to do the online batch-insertion (Mutator.addInsertion). Let’s call this second method online-bulk-insertion. After this online-bulk-insertion, we issued the bulk retrieval to get the data back using the Hector API’s multigetSliceQuery. We

Re: Bulk loading into CQL3 Composite Columns

"user@cassandra.apache.org<mailto:user@cassandra.apache.org>" mailto:user@cassandra.apache.org>> Date: Friday, May 31, 2013 9:01 AM To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" mailto:user@cassandra.apache.org>> Subject: Re: Bulk loading into

Re: Bulk loading into CQL3 Composite Columns

> ) > > You can see the source for CompositeSerializer here: > http://grepcode.com/file/repo1.maven.org/maven2/com.netflix.astyanax/astyanax/1.56.26/com/netflix/astyanax/serializers/CompositeSerializer.java > > Good luck! > > From: Daniel Morton > Reply-To: "user@cas

Re: Bulk loading into CQL3 Composite Columns

ache.org<mailto:user@cassandra.apache.org>" mailto:user@cassandra.apache.org>> Subject: Re: Bulk loading into CQL3 Composite Columns Hi Keith... Thanks for the help. I'm presently not importing the Hector library (Which is where classes like CompositeSerializer and Stri

Re: Bulk loading into CQL3 Composite Columns

umnComposite), null, >> System.currentTimeMillis() ); >> >> From: Keith Wright >> Date: Thursday, May 30, 2013 3:32 PM >> To: "user@cassandra.apache.org" >> Subject: Re: Bulk loading into CQL3 Composite Columns >> >> You do not want to repeat t

Re: Bulk loading into CQL3 Composite Columns

: Daniel Morton > Reply-To: "user@cassandra.apache.org" > Date: Thursday, May 30, 2013 1:06 PM > To: "user@cassandra.apache.org" > Subject: Bulk loading into CQL3 Composite Columns > > Hi All. I am trying to bulk load some data into a CQL3 table using the > sstableloa

Re: Bulk loading into CQL3 Composite Columns

> > ssTableWriter.addColumn( > CompositeSerializer.get().toByteBuffer(columnComposite), null, > System.currentTimeMillis() ); > > From: Keith Wright > Date: Thursday, May 30, 2013 3:32 PM > To: "user@cassandra.apache.org" > Subject: Re: Bulk loading into CQL3 Compo

Re: Bulk loading into CQL3 Composite Columns

ssandra.apache.org>> Subject: Re: Bulk loading into CQL3 Composite Columns You do not want to repeat the first item of your primary key again. If you recall, in CQL3 a primary key as defined below indicates that the row key is the first item (key) and then the column names are composites of

Re: Bulk loading into CQL3 Composite Columns

t;" mailto:user@cassandra.apache.org>> Date: Thursday, May 30, 2013 1:06 PM To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" mailto:user@cassandra.apache.org>> Subject: Bulk loading into CQL3 Composite Columns Hi All. I am trying to bulk load

Bulk loading into CQL3 Composite Columns

, val1 int, val2 int, primary key (key, val1, val2)); Loosely following the bulk loading example in the docs, I have constructed the following method to create my temporary SSTables. public static void main(String[] args) throws Exception { final List> compositeTypes = new ArrayL

Re: Bulk Loading-Unable to select from CQL3 tables with NO "COMPACT STORAGE" option after Bulk Loading - Cassandra version 1.2.1

Freelance Cassandra Developer New Zealand @aaronmorton http://www.thelastpickle.com On 26/02/2013, at 3:44 AM, praveen.akun...@wipro.com wrote: > Hi All, > > I am using the bulk loader program provided in Datastax website. > http://www.datastax.com/dev/blog/bulk-loading > >

Bulk Loading with Composite Column Slow?

efore adding components in each iteration but that didn't seem to help. thanks bri Code based on bulk loading tutorial: List> compositeList = new ArrayList>(); compositeList.add(UTF8Type.instance); compositeList.add(UTF8Type.instance); SSTableSimp

Re: Bulk loading and timestamps

Yes. See the example here http://www.datastax.com/dev/blog/bulk-loading Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 4/05/2012, at 2:49 AM, Oleg Proudnikov wrote: > Hello, group > > Will the bulk loader preserve origin

Re: SSTableWriter and Bulk Loading life cycle enhancement

/2012, at 11:40 PM, Benoit Perroud wrote: > Hi All, > > I'm bulk loading (a lot of) data from Hadoop into Cassandra 1.0.x. The > provided CFOutputFormat is not the best case here, I wanted to use the > bulk loading feature. I know 1.1 comes with a BulkOutputFormat but I &

Bulk loading and timestamps

Hello, group Will the bulk loader preserve original column timestamps? Thank you very much, Oleg

SSTableWriter and Bulk Loading life cycle enhancement

Hi All, I'm bulk loading (a lot of) data from Hadoop into Cassandra 1.0.x. The provided CFOutputFormat is not the best case here, I wanted to use the bulk loading feature. I know 1.1 comes with a BulkOutputFormat but I wanted to propose a simple enhancement to SSTableSimpleUnsortedWriter

Re: Bulk loading errors with 1.0.8

On Thu, Apr 5, 2012 at 10:58 AM, Benoit Perroud wrote: > ERROR [Thread-23] 2012-04-05 09:58:12,252 AbstractCassandraDaemon.java > (line 139) Fatal exception in thread Thread[Thread-23,5,main] > java.lang.RuntimeException: Insufficient disk space to flush > 7813594056494754913 bytes >        at >

Bulk loading errors with 1.0.8

Hi All, I'm experiencing the following errors while bulk loading data into a cluster ERROR [Thread-23] 2012-04-05 09:58:12,252 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[Thread-23,5,main] java.lang.RuntimeException: Insufficient disk space to

Re: Bulk Loading Recommendations: Files over 25GBs

At that scale of data, and the fact that it's a batch job, I would go with the bulk loading tool. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 19/10/2011, at 3:32 AM, Mike Rapuano wrote: > We are not currently live but test

Bulk Loading Recommendations: Files over 25GBs

uster. I was reading this article on DataStax: http://www.datastax.com/dev/blog/bulk-loading Is it most efficient to create the sstables and then use sstableloader or does anyone have other recommendations to "bulk load data"? We are new to Cassandra and trying to work within wh

Re: Bulk loading into live data

ill > get incremented ? > My use-case is that my bulk loading has data that's aggregated at a finer > granularity than the data in cassandra. > > Thanks >

Re: Bulk loading into live data

nter column multiple times for the same key/SC in different nextRow() calls and the counter will still get incremented ? My use-case is that my bulk loading has data that's aggregated at a finer granularity than the data in cassandra. Thanks

Re: Bulk loading into live data

m/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/CounterColumn.java#L138 Cheers - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 17/08/2011, at 6:57 PM, Philippe wrote: > http://www.datastax.com/dev/blog/bulk-loading

Bulk loading into live data

http://www.datastax.com/dev/blog/bulk-loading indicates that "it is perfectly reasonable to load data into a live, active cluster." So lets say my cluster has a single KS & CF and it contains a key "test" with a SC named "Cass" and a normal subcolumn named &quo

Re: bulk loading

Hello Torsten, I am working on Cassandra for last 4 weeks and am trying to load large amount of data.Data is in a csv file.I am trying to use the Bulk loading technique but am not clear with the process.Could you please explain the process for the bulk load? Thanks, Priyanka

Re: Bulk Loading

Hello All, I am trying to load huge amounts of data into Cassandra.I want you use bulk loading with hadoop. I look into bulkoloader utility in java.But I am not sure how to provide input to hadoop and then load to cassandra.Could some one please explain the process? Thank you. Regards

Re: bulk loading

I looked at the thrift service implementation and got it working. (Much faster import!) Thanks! On Mon, Jun 21, 2010 at 13:09, Oleg Anastasjev wrote: > Torsten Curdt vafer.org> writes: > >> >> First I tried with my one "cassandra -f" instance then I saw this >> requires a separate IP. (Why?) >

Re: bulk loading

Torsten Curdt vafer.org> writes: > > First I tried with my one "cassandra -f" instance then I saw this > requires a separate IP. (Why?) This is because your import program becomes a special member of cassandra cluster to be able to speak internal protocol. And each memboer of cassandra cluster

Re: bulk loading

> You should be using the thrift API, or a wrapper around the thrift API. It > looks like you're using internal cassandra classes. The goal is to get around using the overhead of the Thrift API for a bulk import. > There is a Java wrapper called Hector, and there was another talked about on > t

Re: bulk loading

http://wiki.apache.org/cassandra/Operations Aaron On 21 Jun 2010, at 05:50, Torsten Curdt wrote: > I am trying to get the bulk loading example to work for simple CF. > >List columnFamilies = new LinkedList(); > >while(...) { >String[] fields = ... > >

bulk loading

I am trying to get the bulk loading example to work for simple CF. List columnFamilies = new LinkedList(); while(...) { String[] fields = ... ColumnFamily columnFamily = ColumnFamily.create(keyspace, family); long now = System.currentTimeMillis(); for

Re: Cassandra cluster runs into OOM when bulk loading data

There are other threads linked to this issue. Most notable, I think we're hitting https://issues.apache.org/jira/browse/CASSANDRA-1014 here. 2010/4/27 Schubert Zhang > Seems: > > ROW-MUTATION-STAGE 32 3349 63897493 > is the clue, too many mutation requests are pending. > > > Yes,

Re: Cassandra cluster runs into OOM when bulk loading data

Seems: ROW-MUTATION-STAGE 32 3349 63897493 is the clue, too many mutation requests are pending. Yes, I also think cassandra should add a mechanism to avoid too many requests pending (in queue). When the queue is full, just reject the request from client. Seems https://issues.apache.

Re: Cassandra cluster runs into OOM when bulk loading data

I wrote a script to record the tpstats output every 5 seconds. Here is the output just before the jvm OOM: Pool NameActive Pending Completed FILEUTILS-DELETE-POOL 0 0280 STREAM-STAGE 0 0 0 RESPONSE

Re: Cassandra cluster runs into OOM when bulk loading data

I'll work on doing more tests around this. In 0.5 we used a different data structure that required polling. But this does seem problematic. -Chris On Apr 26, 2010, at 7:04 PM, Eric Yu wrote: > I have the same problem here, and I analysised the hprof file with mat, as > you said, LinkedBlockQu

Re: Cassandra cluster runs into OOM when bulk loading data

I have the same problem here, and I analysised the hprof file with mat, as you said, LinkedBlockQueue used 2.6GB. I think the ThreadPool of cassandra should limit the queue size. cassandra 0.6.1 java version $ java -version java version "1.6.0_20" Java(TM) SE Runtime Environment (build 1.6.0_20-b

Re: Cassandra cluster runs into OOM when bulk loading data

Thanks Chris 2010/4/26 Chris Goffinet > Upgrade to b20 of Sun's version of JVM. This OOM might be related to > LinkedBlockQueue issues that were fixed. > > -Chris > > > 2010/4/26 Roland Hänel > >> Cassandra Version 0.6.1 >> OpenJDK Server VM (build 14.0-b16, mixed mode) >> Import speed is about

Re: Cassandra cluster runs into OOM when bulk loading data

Upgrade to b20 of Sun's version of JVM. This OOM might be related to LinkedBlockQueue issues that were fixed. -Chris 2010/4/26 Roland Hänel > Cassandra Version 0.6.1 > OpenJDK Server VM (build 14.0-b16, mixed mode) > Import speed is about 10MB/s for the full cluster; if a compaction is going >

Re: Cassandra cluster runs into OOM when bulk loading data

Cassandra Version 0.6.1 OpenJDK Server VM (build 14.0-b16, mixed mode) Import speed is about 10MB/s for the full cluster; if a compaction is going on the individual node is I/O limited tpstats: caught me, didn't know this. I will set up a test and try to catch a node during the critical time. Than

Re: Cassandra cluster runs into OOM when bulk loading data

Which version of Cassandra? Which version of Java JVM are you using? What do your I/O stats look like when bulk importing? When you run `nodeprobe -host tpstats` is any thread pool backing up during the import? -Chris 2010/4/26 Roland Hänel > I have a cluster of 5 machines building a Cass

Cassandra cluster runs into OOM when bulk loading data

I have a cluster of 5 machines building a Cassandra datastore, and I load bulk data into this using the Java Thrift API. The first ~250GB runs fine, then, one of the nodes starts to throw OutOfMemory exceptions. I'm not using and row or index caches, and since I only have 5 CF's and some 2,5 GB of