Saving and loading data is roughly doubling memory use. I'm trying to
understand and correct the problem.
R1 was an R process using just over 2G of memory.
I did save(r3b, r4, sflist, file="r4.rdata")
and then, in a new process R2,
load(file="r4.rdata")
R2 used just under 4G of memory, i.e., almost double the original
process. The r4.rdata file was just under 2G, which seemed like very
little compression.
r4 was created by
r4 <- sflist2stanfit(sflist)
I presume that r4 and sflist shared most of their memory.
The save() apparently lost the information that the memory was shared,
doubling memory use.
R 2.15.1, 64 bit on linux.
First, does my diagnosis sound right? The reports of memory use in R2
are quite a bit lower than the process footprint; is that normal?
> gc() # after loading data
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 1988691 106.3 3094291 165.3 2432643 130.0
Vcells 266976864 2036.9 282174979 2152.9 268661172 2049.8
> rm("r4")
> gc()
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 1949626 104.2 3094291 165.3 2432643 130.0
Vcells 190689777 1454.9 282174979 2152.9 268661172 2049.8
> r4 <- sflist2stanfit(sflist)
> gc()
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 1970497 105.3 3094291 165.3 2432643 130.0
Vcells 228827252 1745.9 296363727 2261.1 268661172 2049.8
>
Even weirder, R1 reports memory use well beyond the memory I show the
process using (2.2G)
> gc()
used (Mb) gc trigger (Mb) max used (Mb)
Ncells 3640941 194.5 5543382 296.1 5543382 296.1
Vcells 418720281 3194.6 553125025 4220.1 526708090 4018.5
Second, what can I do to avoid the problem?
I guess in this case I could not save r4 and recreate it, but is there a
more general solution?
If I did myboth <- list(r4, sflist) and
save(myboth, file="myfile")
would that be enough to keep the objects together? Judging from the
size of the file, it seems not.
Even if the myboth trick worked it seems like a kludge.
Ross Boylan
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
