Merci beaucoup Milan, thank you very much Martin and Kjetil for your
responses.
I appreciate the caveat about virtual memory. I gather that besides
resident memory and swap space, it may also include memory mapped files,
which don't "cost" anything. Maybe by pure chance, in my case virtual
memory still seems mildly relevant. While there is RAM available, "res"
tracks "virt" closely, about till this point:
top - 13:32:28 up 208 days, 20:46, 3 users, load average: 1.68, 1.41,
1.17
Tasks: 1 total, 1 running, 0 sleeping, 0 stopped, 0 zombie
Cpu(s): 46.5%us, 6.5%sy, 0.0%ni, 6.0%id, 41.0%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 8063744k total, 8012976k used, 50768k free, 464k buffers
Swap: 19543064k total, 3445236k used, 16097828k free, 35096k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6210 brech 20 0 7486m 7.2g 7424 R 98 93.2 6:15.74 R
(That's 7.3g virtual memory.) After that, "res" stays the same, while
"virt" keeps growing. That's an issue because if it uses up all the swap
space (a bit beyond the state I showed in my original post), R starts
reporting problems, e.g.:
Error in system(command = command, intern = output) :
cannot popen 'whoami', probable reason 'Cannot allocate memory'
It sounds like a major reason for the discrepancy could be fragmentation,
possibly caused by repeated copying. It will take some work to profile
memory usage (thanks for you pointers to the tools) get a better picture,
and to create a minimal reproducible example.
I'm glad you pointed me to /proc/[pid]/maps and smaps; they have a wealth
of information. The most interesting entry is [heap]; it's growing rapidly
during the run of my code, and accounts for all of "res" and 98% of "virt".
The others are less exciting, mostly memory mapped files (e.g.,
lib/R/library/MASS/libs/x86_64/MASS.so), and change at most by a few k
Referenced, or moved from Rss to Swap.
So clearly my interest is in R's heap. Again many thanks for all your help!
/Christian
On Thu, Apr 18, 2013 at 5:17 PM, Kjetil Kjernsmo <kje...@ifi.uio.no> wrote:
> On Thursday 18. April 2013 12.18.03 Milan Bouchet-Valat wrote:
> > First, completely stop looking at virtual memory: it does not mean much,
> if
> > anything. What you care about is resident memory. See e.g.:
> >
> http://serverfault.com/questions/138427/top-what-does-virtual-memory-size-m
> > ean-linux-ubuntu
>
> I concur. I have lost track of R's internals long ago, but in a previous
> life
> analyzing the Apache HTTP server's actual memory use (something that
> focused
> on shared RAM, quite different from what you'd probably like to do), I
> found
> that if you really need to understand what's going on, you would need to
> look
> elsewhere.
>
> On Linux, you'll find the details in the /proc/[pid]/maps and
> /proc/[pid]/smaps
> pseudo-filesystem files, where [pid] is the process ID, im your example
> 6210.
> That's where you really see what's eating your RAM. :-)
>
> Cheers,
>
> KJetil
>
[[alternative HTML version deleted]]
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
