Zerdna,
Please note that the CRAN version 3.12 is about
to be replaced by a new cluster of packages now on R-Forge; we consider the
new bigmemory >= 4.0 to be "stable" and recommend you start using it
immediately. Please see http://www.bigmemory.org.
In your case, two comments:
(1) Your for() loop will generate three identical copies of filebackings on
disk,
yes. Note that when the loop exists, the R object xx will reference only
the
3rd of these, so xx[1,1] <- 1 will modify only the third filebacking, not
the
first two. You'll need to use the separate descriptor files (probably
created
automatically for you, but we recommend naming them specifically using
descriptorfile=) to attach.big.matrix() whatever of these you really want to
be using.
(2) In the problem with "hanging" I believe you have exhausted the shared
resources on your system. This problem will no longer arise in the >= 4.0
problems, as we're handling mutexes separately rather than automatically.
These shared resource limits are mysterious, depending on the OS as well
as the hardware and other jobs or tasks in existence at any given point in
time.
But again, it shouldn't be a problem with the new version.
The CRAN update should take place early next week, along with some revised
documentation.
Regards,
Jay
---------------
Message: 125
Date: Fri, 23 Apr 2010 13:51:32 -0800 (PST)
From: zerdna <az...@yahoo.com>
To: r-help@r-project.org
Subject: [R] bigmemory package woes
Message-ID: <1272059492009-2062996.p...@n4.nabble.com>
Content-Type: text/plain; charset=us-ascii
I have pretty big data sizes, like matrices of .5 to 1.5GB so once i need to
juggle several of them i am in need of disk cache. I am trying to use
bigmemory package but getting problems that are hard to understand. I am
getting seg faults and machine just hanging. I work by the way on Red Hat
Linux, 64 bit R version 10.
Simplest problem is just saving matrices. When i do something like
r<-matrix(rnorm(100), nr=10); librarybigmemory)
for(i in 1:3) xx<-as.big.matrix(r, backingfile=paste("r",i, sep="",
collapse=""), backingpath=MyDirName)
it works just fine -- saves small matrices as three different matrices on
disc. However, when i try it with real size, like
with r<-matrix(normal(50000000), nr=1000)
I am either getting seg fault on saving the third big matrix, or hang
forever.
Am i doing something obviously wrong, or is it an unstable package at the
moment? Could anyone recommend something similar that is reliable in this
case?
--
John W. Emerson (Jay)
Associate Professor of Statistics
Department of Statistics
Yale University
http://www.stat.yale.edu/~jay
[[alternative HTML version deleted]]
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
