Re: Heap sudden jump during import

On Wed, Apr 7, 2010 at 1:51 PM, Eric Evans wrote: > On Tue, 2010-04-06 at 10:55 -0700, Tatu Saloranta wrote: >> On Tue, Apr 6, 2010 at 12:15 AM, JKnight JKnight >> wrote: >> > When import, all data in json file will load in memory. So that, you >> can not >> > import large data. >> > You need to

Re: Heap sudden jump during import

On Tue, 2010-04-06 at 10:55 -0700, Tatu Saloranta wrote: > On Tue, Apr 6, 2010 at 12:15 AM, JKnight JKnight > wrote: > > When import, all data in json file will load in memory. So that, you > can not > > import large data. > > You need to export large sstable file to many small json files, and > r

Re: Heap sudden jump during import

On Tue, Apr 6, 2010 at 12:15 AM, JKnight JKnight wrote: > When import, all data in json file will load in memory. So that, you can not > import large data. > You need to export large sstable file to many small json files, and run > import. Why would you ever read the whole file in memory? JSON is

Re: Heap sudden jump during import

When import, all data in json file will load in memory. So that, you can not import large data. You need to export large sstable file to many small json files, and run import. On Mon, Apr 5, 2010 at 5:26 PM, Jonathan Ellis wrote: > Usually sudden heap jumps involve compacting large rows. > > 0.

Re: Heap sudden jump during import

Usually sudden heap jumps involve compacting large rows. 0.6 (since beta3) includes a warning log when it finishes compacts a row over 500MB by default, in the hopes that this will give you enough time to fix things before whatever is making large rows makes one too large to fit in memory. On Fri

Re: Heap sudden jump during import

Eclipse Memory Analyser rocks! Thanks a lot!! -Weijun On Sat, Apr 3, 2010 at 2:25 AM, Benoit Perroud wrote: > Have a look at either Eclipse Memory Analyser (they have a standalone > version of the memory analyser) or YourKit Java Profiler (commercial, > but with evaluation license). I successfu

Re: Heap sudden jump during import

Have a look at either Eclipse Memory Analyser (they have a standalone version of the memory analyser) or YourKit Java Profiler (commercial, but with evaluation license). I successfully load and browse heap bigger than the available memory on the system. Regards, Benoit 2010/4/3 Weijun Li : > Tha

Re: Heap sudden jump during import

Thank you Benoit. I did a search but couldn't find any that you mentioned. Both jhat and netbean load entire map file int memory. Do you know the name of the tools that requires less memory to view map file? Thanks, -Weijun On Sat, Apr 3, 2010 at 12:55 AM, Benoit Perroud wrote: > It exists othe

Re: Heap sudden jump during import

It exists other tools than jhat to browse a heap dump, which stream the heap dump instead of loading it full in memory like jhat do. Kind regards, Benoit. 2010/4/3 Weijun Li : > I'm running a test to write 30 million columns (700bytes each) to Cassandra: > the process ran smoothly for about 20mi

Heap sudden jump during import

I'm running a test to write 30 million columns (700bytes each) to Cassandra: the process ran smoothly for about 20mil then the heap usage suddenly jumped from 2GB to 3GB which is the up limit of JVM, --from this point Cassandra will freeze for long time (terrible latency, no response to nodetool th