Re: [Rd] Defragmentation of memory

2016-09-05 Thread luke-tierney
On Mon, 5 Sep 2016, Måns Magnusson wrote: Dear all developers, I'm working with a lot of textual data in R and need to handle this batch by batch. The problem is that I read in batches of 10 000 documents and do some calculations that results in objects that consume quite some memory (calculate

[Rd] Defragmentation of memory

2016-09-04 Thread Måns Magnusson
Dear all developers, I'm working with a lot of textual data in R and need to handle this batch by batch. The problem is that I read in batches of 10 000 documents and do some calculations that results in objects that consume quite some memory (calculate unigrams, 2-grams and 3-grams). In every ite