Re: [R] Reshaping an array - how does it work in R

2016-03-23 Thread Martin Maechler
> Dénes Tóth > on Tue, 22 Mar 2016 10:55:58 +0100 writes: > Hi Martin, > On 03/22/2016 10:20 AM, Martin Maechler wrote: >>> >Dénes Tóth >>> > on Fri, 18 Mar 2016 22:56:23 +0100 writes: >> > Hi Roy, >> > R (usually) makes a copy if the dime

Re: [R] Reshaping an array - how does it work in R

2016-03-22 Thread Roy Mendelssohn - NOAA Federal
Thanks all. This is interesting, and for what I am doing worthwhile and helpful. I have to be careful in each operation whether a copy is made or not, and knowing this allows me to test on small examples what any command will do before I use, Thanks again, I appreciate all the help. I will

Re: [R] Reshaping an array - how does it work in R

2016-03-22 Thread Dénes Tóth
Hi Martin, On 03/22/2016 10:20 AM, Martin Maechler wrote: >Dénes Tóth > on Fri, 18 Mar 2016 22:56:23 +0100 writes: > Hi Roy, > R (usually) makes a copy if the dimensionality of an array is modified, > even if you use this syntax: > x <- array(1:24, c(2, 3, 4))

Re: [R] Reshaping an array - how does it work in R

2016-03-22 Thread Martin Maechler
> Dénes Tóth > on Fri, 18 Mar 2016 22:56:23 +0100 writes: > Hi Roy, > R (usually) makes a copy if the dimensionality of an array is modified, > even if you use this syntax: > x <- array(1:24, c(2, 3, 4)) > dim(x) <- c(6, 4) > See also ?tracemem, ?data.table:

Re: [R] Reshaping an array - how does it work in R

2016-03-21 Thread Roy Mendelssohn - NOAA Federal
Thanks for the info, but I will stay with regular R. Work -arounds for what I want to do just took some thought and programming, I just didn’t know if R copied the array or just manipulated indices, and given the size of the array I am memory limited. This gets into the old thing of whether

[R] Reshaping an array - how does it work in R

2016-03-21 Thread Rodrigo Botafogo
Roy, I have implemented a Ruby Gem (SciCom) with exactly your use case in mind. SciCom is based on Renjin, an R interpreter for the JVM. So, this reply is about R, but not about GnuR. If this is not proper behavior, please let me know. I´ve looked at the posting guidelines and it seems to be ok

Re: [R] Reshaping an array - how does it work in R

2016-03-19 Thread Dénes Tóth
Hi Roy, R (usually) makes a copy if the dimensionality of an array is modified, even if you use this syntax: x <- array(1:24, c(2, 3, 4)) dim(x) <- c(6, 4) See also ?tracemem, ?data.table::address, ?pryr::address and other tools to trace if an internal copy is done. Workaround: use data.tab

Re: [R] Reshaping an array - how does it work in R

2016-03-19 Thread Roy Mendelssohn - NOAA Federal
> On Mar 19, 2016, at 8:18 AM, Henrik Bengtsson > wrote: > > On Fri, Mar 18, 2016 at 8:28 PM, Roy Mendelssohn - NOAA Federal > wrote: >> Hi Henrik: >> >> I want to do want in oceanography is called an EOF, which is just a PCA >> analysis. Unless I am missing something, in R I need to flatten

Re: [R] Reshaping an array - how does it work in R

2016-03-19 Thread Henrik Bengtsson
On Fri, Mar 18, 2016 at 8:28 PM, Roy Mendelssohn - NOAA Federal wrote: > Hi Henrik: > > I want to do want in oceanography is called an EOF, which is just a PCA > analysis. Unless I am missing something, in R I need to flatten my 3-D matrix > into a 2-D data matrix. I can fit the entire 30GB matr

[R] Reshaping an array - how does it work in R

2016-03-19 Thread Roy Mendelssohn - NOAA Federal
Hi All: I am working with a very large array. if noLat is the number of latitudes, noLon the number of longitudes and noTime the number of time periods, the array is of the form: myData[noLat, no Lon, noTime]. It is read in this way because that is how it is stored in a (series) of netcdf f

Re: [R] Reshaping an array - how does it work in R

2016-03-18 Thread Bert Gunter
arrays are vectors stored in column major order. So the answer is: reindexing. Does this make it clear: > v <- array(1:24,dim=2:4) > as.vector(v) [1] 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 > v , , 1 [,1] [,2] [,3] [1,]135 [2,]246

Re: [R] Reshaping an array - how does it work in R

2016-03-18 Thread Roy Mendelssohn - NOAA Federal
Hi Henrik: I want to do want in oceanography is called an EOF, which is just a PCA analysis. Unless I am missing something, in R I need to flatten my 3-D matrix into a 2-D data matrix. I can fit the entire 30GB matrix into memory, and I believe I have enough memory to do the PCA by constraining

Re: [R] Reshaping an array - how does it work in R

2016-03-18 Thread Roy Mendelssohn - NOAA Federal
> On Mar 18, 2016, at 2:56 PM, Bert Gunter wrote: > > However copying may occur anyway as part of R's semantics. Others will > have to help you on that, as the details here are beyond me. > > Cheers, > Bert Hi Bert: Thanks for your response. The only part I was concerned with is whether a co

Re: [R] Reshaping an array - how does it work in R

2016-03-18 Thread Roy Mendelssohn - NOAA Federal
Thanks. That is what I needed to know. I don’t want to play around with some of the other suggestions, as I don’t totally understand what they do, and don’t want to risk messing up something and not be aware of it. There is a way to read in the data chunks at a time and reshape it and put, it

Re: [R] Reshaping an array - how does it work in R

2016-03-18 Thread Henrik Bengtsson
On Fri, Mar 18, 2016 at 3:15 PM, Roy Mendelssohn - NOAA Federal wrote: > Thanks. That is what I needed to know. I don’t want to play around with > some of the other suggestions, as I don’t totally understand what they do, > and don’t want to risk messing up something and not be aware of it. >

Re: [R] Reshaping an array - how does it work in R

2016-03-18 Thread Jeff Newmiller
R always makes a copy for this kind of operation. There are some operations that don't make copies, but I don't think this one qualifies. -- Sent from my phone. Please excuse my brevity. On March 18, 2016 2:28:35 PM PDT, Roy Mendelssohn - NOAA Federal wrote: >Hi All: > >I am working with a v