On Tue, 23 Nov 2010, derek eder wrote:
Hello,
I am facing the dreaded "Error: cannot allocate vector of size x Gb" and
don't understand
enough about R (or operating system) memory management to diagnose and solve
the problem
-- despite studying previous posts and relevant R help -- e.g.:
"Error messages beginning cannot allocate vector of size indicate a failure
to obtain memory,
either because the size exceeded the address-space limit for a process or,
more likely,
because the system was unable to provide the memory.
[...] On all builds of R, the maximum length (number of elements)
of a vector is 2^31 - 1 ~ 2*10^9, as lengths are stored as signed integers.
In addition, the storage space cannot exceed the address limit."
- from Memory-limits {Base}
Simple question: Given 64-bit R (AMD64 Linux) with a ulimit of "unlimited",
can the size of an R object exceed the amount of availlable RAM
memory?
Empirically my system with 4Gb RAM and ample Swap, is failing:
x <- integer(10^9)
object.size(x)
4000000040 bytes
gc()
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 121195 6.5 350000 18.7 350000 18.7
Vcells 500124024 3815.7 606849099 4629.9 550124408 4197.2
matrix(x, ncol=16)
Error: cannot allocate vector of size 3.7 Gb
I don't understand how this operation violates the limits detailed in the
Memory-limit help (above).
It doesn't, and that example works on my (32GB RAM) 64-bit system
gc()
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 134054 7.2 350000 18.7 350000 18.7
Vcells 1000131158 7630.4 1575382823 12019.3 1500131550 11445.1
So maybe you don't have 'ample swap' and/or have a VM limit set.
Note that
dim(x) <- c(length(x)/16, 16)
is much more efficient use of memory.
Thank you!
Derek Eder
-------------------------------------------------------------------------------------------------
version
_
platform x86_64-pc-linux-gnu
arch x86_64
os linux-gnu
system x86_64, linux-gnu
status
major 2
minor 11.1
year 2010
month 05
day 31
svn rev 52157
language R
version.string R version 2.11.1 (2010-05-31)
de...@papanca:~$ top
top - 09:10:18 up 51 min, 4 users, load average: 0.51, 0.51, 0.45
Tasks: 160 total, 2 running, 158 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 25.0%sy, 0.0%ni, 75.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 3796484k total, 3764852k used, 31632k free, 14204k buffers
Swap: 2929660k total, 834240k used, 2095420k free, 94800k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2854 derek 20 0 239m 9260 5448 S 6 0.2 0:05.53 gnome-terminal
1164 root 20 0 218m 31m 10m S 4 0.8 1:29.71 Xorg
3331 derek 20 0 19276 1324 944 R 1 0.0 0:00.6 top
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
--
Brian D. Ripley, rip...@stats.ox.ac.uk
Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/
University of Oxford, Tel: +44 1865 272861 (self)
1 South Parks Road, +44 1865 272866 (PA)
Oxford OX1 3TG, UK Fax: +44 1865 272595
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
