Re: DataImport using SqlEntityProcessor running Out of Memory

2014-05-13 Thread Mikhail Khludnev
Hello O, It seems to me (but it's better to look at the heap histogram) that buffering sub-entities in SortedMapBackedCache blows heap off. I'm aware about two directions: - use file based cache instead. I don't know exactly how it works, you can start from https://issues.apache.org/jira/browse/SOL

Re: DataImport using SqlEntityProcessor running Out of Memory

2014-05-12 Thread Shawn Heisey
On 5/9/2014 9:16 AM, O. Olson wrote: > I have a Data Schema which is Hierarchical i.e. I have an Entity and a number > of attributes. For a small subset of the Data - about 300 MB, I can do the > import with 3 GB memory. Now with the entire 4 GB Dataset, I find I cannot > do the import with 9 GB of