Let me begin stating that I read all help files and faq's on the subject
matter (there aren't more than about a dozen) but either did not find
solutions or found them not to work.
Here is the issue. I am trying to run a spatial regression on a
medium-sized dataset. Part of the functions in the spdep package I use
require me to allocate a vector of 1.1 Gb (mine is not a spatial SIG
question, I am using this just as an example). I have been trying my
luck on two different machines, each with 4 GB of real RAM, plus a
hundred GB of swap space. One machine runs XP, the other Windows 7.
I have tried to work with startup parameters --min-vsize, --min-nsize,
--max-vsize, max-nsize. The limit here seems to be 2048 M - I am getting
error messages if I try to allocate more. I can increase the general
memory allocation using memory.limit(3072), and gc() then reports that I
have 3 Gigs available, although I don't know how this is split into cons
and heap cells.
In any case, I still do not get to allocate my 1.1 GB, and I am at a
loss explaining why.
The help pages for memory() also do not explain what actually happens
with the command line options min-vsize and max-vsize, or at least did
not prepare me for the following odd observation. If I specify
min-vsize=2048M, then I do not get to read in more than 384 MB of data;
it is as if the allocation of a minimum of 2 Gig of RAM actually reduces
the amount available. If I specify max-vize=2048M, then I can at least
read my data, but still run out of memory trying to run my spatial
regression. In other words, the specification of a minimum renders my
nice machine unusable, whereas the specification of a maximum seems to
have not the desired effect of increasing the memory allocation for R.
So here are my questions:
1. What is the purpose of min-vsize and min-nsize? (please do not point
me to help pages, I read them)
2. Why do I get error messages for min-nsize > 2G even on machines whose
OS support more? (I tried the BCDEdit switch suggested by Kingsford
Jones but it did not have any visible effect)
3. How can I increase vsize within R rather than at startup
(memory.limit does not allow to specify types of memory)?
4. Why do I still get a "cannot allocate vector of size 1.1 Gb" even
when gc() tells me that I have used only 240 out of my 2048 Mb allocated?
5. At this point, I am actually using only a small test dataset to
develop my models. Once I have a functioning, well-tested model, I want
to apply it to a dataset 100 times larger and I know that I will have to
move onto a Linux platform. Given that I won't have 400 Gb of RAM
available there, will I run into similar problems there or will the OS
take care of memory allocation issues using swap space?
Cheers,
Jochen
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
