[Rd] matrix bands
Dear R developers,
I was looking for a function analogous to base::diag() for getting and
setting bands of a matrix. The closest I could find was Matrix::band(),
but this was not exactly what I wanted for two reasons. Firstly,
Matrix::band() returns a matrix rather than just the specified band.
Secondly, Matrix::band() cannot be used for setting the values for a
matrix band.
Setting or getting a matrix band could be of interest for sufficiently
many users that you might consider including it in base R.
Alternatively, something like this could be included in the Matrix package.
I have included two versions of these functions, a simple and naive
version, and a more efficient version. The band index, n, is positive
for bands above the diagonal and negative for bands below the diagonal.
Regards,
Jeremy Silver
###
## less clear formulation - more efficient
band <- function(x,n = 0){
dx <- dim(x)
if(length(dx) != 2L)
stop("only matrix bands can be accessed")
max.dx <- max(dx)
n <- as.integer(n)
ij <- cbind(i = seq(1,max.dx) - n,
j = seq(1,max.dx))
ij <- ij[1 <= ij[,1] & ij[,1] <= dx[1] & 1 <= ij[,2] & ij[,2] <=
dx[2],,drop=FALSE]
if(nrow(ij) == 0)
stop('cannot access this matrix band')
x[ij]
}
'band<-' <- function(x,n = 0, value){
dx <- dim(x)
if(length(dx) != 2L)
stop("only matrix bands can be replaced")
max.dx <- max(dx)
n <- as.integer(n)
ij <- cbind(i = seq(1,max.dx) - n,
j = seq(1,max.dx))
ij <- ij[1 <= ij[,1] & ij[,1] <= dx[1] & 1 <= ij[,2] & ij[,2] <=
dx[2],,drop=FALSE]
if(nrow(ij) == 0)
stop('cannot replace this matrix band')
x[ij] <- value
x
}
## simple, clear formulation - not very efficient
band2 <- function(x, n = 0) {
x[col(x) - row(x) == as.integer(n)]
}
'band2<-' <- function(x, n = 0, value) {
x[which(col(x) - row(x) == as.integer(n))] <- value
x
}
## here are some examples to show that it works
## define a test matrix
> A <- matrix(rnorm(12),3,4)
> A
[,1] [,2] [,3] [,4]
[1,] -1.5560200 0.6452762 1.072565 0.1923451
[2,] 0.7940685 1.2441817 1.699486 -0.2998814
[3,] -0.7762252 -0.4824173 -0.981055 -0.9265627
## access some of the bands
> band(A,1)
[1] 0.6452762 1.6994858 -0.9265627
> band(A,-2)
[1] -0.7762252
> band(A,2)
[1] 1.0725649 -0.2998814
## set one of the bands
> band(A,2) <- 2:1
> A
[,1] [,2] [,3] [,4]
[1,] -1.5560200 0.6452762 2.00 0.1923451
[2,] 0.7940685 1.2441817 1.699486 1.000
[3,] -0.7762252 -0.4824173 -0.981055 -0.9265627
## another example - a single column
> A <- matrix(1:10)
> A
[,1]
[1,]1
[2,]2
[3,]3
[4,]4
[5,]5
[6,]6
[7,]7
[8,]8
[9,]9
[10,] 10
> band(A,0)
[1] 1
> band(A,1)
Error in band(A, 1) : cannot access this matrix band
> band(A,-1)
[1] 2
> band(A,-5)
[1] 6
## compare the results from the two versions of the function
> for(i in -2:3){print(band(A,i));print(band2(A,i))}
[1] -0.7762252
[1] -0.7762252
[1] 0.7940685 -0.4824173
[1] 0.7940685 -0.4824173
[1] -1.556020 1.244182 -0.981055
[1] -1.556020 1.244182 -0.981055
[1] 0.6452762 1.6994858 -0.9265627
[1] 0.6452762 1.6994858 -0.9265627
[1] 2 1
[1] 2 1
[1] 0.1923451
[1] 0.1923451
## show that the naive version is very slow for large matrices
> N <- 1e4
> M <- matrix(0,N,N)
> system.time(band(M,2))
user system elapsed
0.005 0.003 0.007
> system.time(band2(M,2))
user system elapsed
18.509 2.121 20.754
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] methods() not listing some S3 plot methods...?
Dear List,
This may be related to this email thread initiated by Ben Bolker last
month: https://stat.ethz.ch/pipermail/r-devel/2011-July/061630.html
In answering this Question on StackOverflow
http://stackoverflow.com/q/7195628/429846 I noticed that `methods()` was
not listing some S3 methods for `plot()` provided by the mgcv package.
At the time I wanted to check the development version of R as I recalled
Uwe mentioning that `plot.function` was listed by `methods()` there but
not in R2.13.x. I have now compiled the development version on two
Fedora installations and certain plot methods are still not being
listed. Details of the exact revision of R Devel are shown at the end of
this email.
As an example, consider:
> require(mgcv)
Loading required package: mgcv
This is mgcv 1.7-6. For overview type 'help("mgcv-package")'.
> methods("plot")
[1] plot.acf* plot.ACF* plot.augPred*
[4] plot.compareFits* plot.data.frame* plot.decomposed.ts*
[7] plot.default plot.dendrogram* plot.density
[10] plot.ecdf plot.factor* plot.formula*
[13] plot.function plot.gam plot.gls*
[16] plot.hclust* plot.histogram*plot.HoltWinters*
[19] plot.intervals.lmList* plot.isoreg* plot.lm
[22] plot.lme* plot.lmList* plot.medpolish*
[25] plot.mlm plot.nffGroupedData* plot.nfnGroupedData*
[28] plot.nls* plot.nmGroupedData*plot.pdMat*
[31] plot.ppr* plot.prcomp* plot.princomp*
[34] plot.profile.nls* plot.ranef.lme*plot.ranef.lmList*
[37] plot.shingle* plot.simulate.lme* plot.spec
[40] plot.stepfun plot.stl* plot.table*
[43] plot.trellis* plot.tsplot.tskernel*
[46] plot.TukeyHSD plot.Variogram*
Non-visible functions are asterisked
> pmeth <- methods("plot")
> grep("plot.mgcv.smooth", pmeth)
integer(0)
> getS3method("plot", "mgcv.smooth")
Error in getS3method("plot", "mgcv.smooth") :
S3 method 'plot.mgcv.smooth' not found
> pfun <- getAnywhere("plot.mgcv.smooth")
> str(pfun)
List of 5
$ name : chr "plot.mgcv.smooth"
$ objs :List of 1
..$ :function (x, P = NULL, data = NULL, label = "", se1.mult = 1,
se2.mult = 2, partial.resids = FALSE, rug = TRUE, se = TRUE,
scale = -1, n = 100, n2 = 40, pers = FALSE, theta = 30, phi = 30,
jit = FALSE, xlab = NULL, ylab = NULL, main = NULL, ylim = NULL,
xlim = NULL, too.far = 0.1, shade = FALSE, shade.col = "gray80",
shift = 0, trans = I, by.resids = FALSE, scheme = NULL, ...)
$ where : chr "namespace:mgcv"
$ visible: logi FALSE
$ dups : logi FALSE
- attr(*, "class")= chr "getAnywhere"
Both `methods()` and `getS3method()` don't list/find this method, but
the function exists in the mgcv name space and this method will be used
via R's S3 dispatch system in `plot.gam()`.
Shouldn't this method be returned by either `methods()` or
`getS3method()`?
TIA,
Gavin
> sessionInfo()
R Under development (unstable) (2011-08-26 r56801)
Platform: x86_64-unknown-linux-gnu (64-bit)
locale:
[1] LC_CTYPE=en_GB.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_GB.UTF-8LC_COLLATE=en_GB.UTF-8
[5] LC_MONETARY=en_GB.UTF-8LC_MESSAGES=en_GB.UTF-8
[7] LC_PAPER=C LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_GB.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods
base
other attached packages:
[1] mgcv_1.7-6
loaded via a namespace (and not attached):
[1] grid_2.14.0lattice_0.19-33Matrix_0.9996875-3
nlme_3.1-102
[5] tools_2.14.0
--
%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%
Dr. Gavin Simpson [t] +44 (0)20 7679 0522
ECRC, UCL Geography, [f] +44 (0)20 7679 0565
Pearson Building, [e] gavin.simpsonATNOSPAMucl.ac.uk
Gower Street, London [w] http://www.ucl.ac.uk/~ucfagls/
UK. WC1E 6BT. [w] http://www.freshwaters.org.uk
%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%~%
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] matrix bands
> Jeremy David Silver
> on Fri, 26 Aug 2011 13:23:43 +0200 writes:
> Dear R developers, I was looking for a function analogous
> to base::diag() for getting and setting bands of a
> matrix. The closest I could find was Matrix::band(), but
> this was not exactly what I wanted for two
> reasons. Firstly, Matrix::band() returns a matrix rather
> than just the specified band. Secondly, Matrix::band()
> cannot be used for setting the values for a matrix band.
Yes, but did you not look at help(band)
or not look carefully enough ?
--->
See Also:
‘bandSparse’ for the _construction_ of a banded sparse matrix
directly from its non-zero diagonals.
> Setting or getting a matrix band could be of interest for
> sufficiently many users that you might consider including
> it in base R. Alternatively, something like this could be
> included in the Matrix package.
well, see above and let us know if you see anything lacking in
bandSparse().
Till now we haven't got much feedback about it, and there may
well be room for improvement.
Martin Maechler, ETH Zurich and co- maintainer("Matrix")
> I have included two versions of these functions, a simple
> and naive version, and a more efficient version. The band
> index, n, is positive for bands above the diagonal and
> negative for bands below the diagonal.
> Regards, Jeremy Silver
> ###
## less clear formulation - more efficient
> band <- function(x,n = 0){ dx <- dim(x) if(length(dx) !=
> 2L) stop("only matrix bands can be accessed")
>max.dx <- max(dx) n <- as.integer(n)
>ij <- cbind(i = seq(1,max.dx) - n, j = seq(1,max.dx))
> ij <- ij[1 <= ij[,1] & ij[,1] <= dx[1] & 1 <= ij[,2] &
> ij[,2] <= dx[2],,drop=FALSE]
>if(nrow(ij) == 0) stop('cannot access this matrix
> band')
>x[ij] }
> 'band<-' <- function(x,n = 0, value){ dx <- dim(x)
> if(length(dx) != 2L) stop("only matrix bands can be
> replaced")
>max.dx <- max(dx) n <- as.integer(n)
>ij <- cbind(i = seq(1,max.dx) - n, j = seq(1,max.dx))
> ij <- ij[1 <= ij[,1] & ij[,1] <= dx[1] & 1 <= ij[,2] &
> ij[,2] <= dx[2],,drop=FALSE]
>if(nrow(ij) == 0) stop('cannot replace this matrix
> band')
>x[ij] <- value
>x }
> ## simple, clear formulation - not very efficient band2 <-
> function(x, n = 0) { x[col(x) - row(x) == as.integer(n)] }
> 'band2<-' <- function(x, n = 0, value) { x[which(col(x) -
> row(x) == as.integer(n))] <- value x }
> ## here are some examples to show that it works
> ## define a test matrix
>> A <- matrix(rnorm(12),3,4) A
> [,1] [,2] [,3] [,4] [1,] -1.5560200 0.6452762
> 1.072565 0.1923451 [2,] 0.7940685 1.2441817 1.699486
> -0.2998814 [3,] -0.7762252 -0.4824173 -0.981055 -0.9265627
> ## access some of the bands
>> band(A,1)
> [1] 0.6452762 1.6994858 -0.9265627
>> band(A,-2)
> [1] -0.7762252
>> band(A,2)
> [1] 1.0725649 -0.2998814
> ## set one of the bands
>> band(A,2) <- 2:1 A
> [,1] [,2] [,3] [,4] [1,] -1.5560200 0.6452762
> 2.00 0.1923451 [2,] 0.7940685 1.2441817 1.699486
> 1.000 [3,] -0.7762252 -0.4824173 -0.981055 -0.9265627
> ## another example - a single column
>> A <- matrix(1:10) A
>[,1] [1,] 1 [2,] 2 [3,] 3 [4,] 4 [5,] 5 [6,] 6 [7,]
> 7 [8,] 8 [9,] 9 [10,] 10
>> band(A,0)
> [1] 1
>> band(A,1)
> Error in band(A, 1) : cannot access this matrix band
>> band(A,-1)
> [1] 2
>> band(A,-5)
> [1] 6
> ## compare the results from the two versions of the
> function
>> for(i in -2:3){print(band(A,i));print(band2(A,i))}
> [1] -0.7762252 [1] -0.7762252 [1] 0.7940685 -0.4824173 [1]
> 0.7940685 -0.4824173 [1] -1.556020 1.244182 -0.981055 [1]
> -1.556020 1.244182 -0.981055 [1] 0.6452762 1.6994858
> -0.9265627 [1] 0.6452762 1.6994858 -0.9265627 [1] 2 1 [1]
> 2 1 [1] 0.1923451 [1] 0.1923451
> ## show that the naive version is very slow for large
> matrices
>> N <- 1e4 M <- matrix(0,N,N) system.time(band(M,2))
> user system elapsed 0.005 0.003 0.007
>> system.time(band2(M,2))
> user system elapsed 18.509 2.121 20.754
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] matrix bands
Thanks for the suggestion, Martin!
I looked at both Matrix::band and Matrix::bandSparse. Maybe I
misunderstand the help pages and the examples, but from what I can see
neither of them provides the functionality I was looking for.
For the getter version of the function I was looking for, I can't use
Matrix::band (by extracting the non-zero elements) because this would
require the assumption that the specified band is entirely non-zero, e.g.:
band3 <- function(x,n){
x <- Matrix::band(x,n,n)
x[x!=0] ## works for a dense matrix, but not if band n has zeroes
}
As for the setter version of the function I was looking for, it should
set the values of a specified band in an existing matrix to be the
vector of values provided. As far as I understand, Matrix::bandSparse
constructs a matrix rather than modifying the values of an existing matrix.
The functions included in my last post filled this gap. If Matrix::band
and Matrix::bandSparse can set/get bands in this way, I would like to
see how this can be done. If not, then perhaps they could be extended to
achieve this functionality.
On 2011-08-26 14:08, Martin Maechler wrote:
Jeremy David Silver
on Fri, 26 Aug 2011 13:23:43 +0200 writes:
> Dear R developers, I was looking for a function analogous
> to base::diag() for getting and setting bands of a
> matrix. The closest I could find was Matrix::band(), but
> this was not exactly what I wanted for two
> reasons. Firstly, Matrix::band() returns a matrix rather
> than just the specified band. Secondly, Matrix::band()
> cannot be used for setting the values for a matrix band.
Yes, but did you not look at help(band)
or not look carefully enough ?
--->
See Also:
‘bandSparse’ for the _construction_ of a banded sparse matrix
directly from its non-zero diagonals.
> Setting or getting a matrix band could be of interest for
> sufficiently many users that you might consider including
> it in base R. Alternatively, something like this could be
> included in the Matrix package.
well, see above and let us know if you see anything lacking in
bandSparse().
Till now we haven't got much feedback about it, and there may
well be room for improvement.
Martin Maechler, ETH Zurich and co- maintainer("Matrix")
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] methods() not listing some S3 plot methods...?
On 08/26/2011 04:44 AM, Gavin Simpson wrote:
Dear List,
This may be related to this email thread initiated by Ben Bolker last
month: https://stat.ethz.ch/pipermail/r-devel/2011-July/061630.html
In answering this Question on StackOverflow
http://stackoverflow.com/q/7195628/429846 I noticed that `methods()` was
not listing some S3 methods for `plot()` provided by the mgcv package.
Hi Gavin --
In the mgcv NAMESPACE, the methods is not registered with S3method
(which would have made it appear with a *) and is not export'ed; the
author of the package apparently intends that it be strictly internal to
the package. Dispatch works within the package name space, but not
outside, e.g., a=list; class(a) = "mgcv.smooth"; plot(a) ends up at
plot.default.
At the time I wanted to check the development version of R as I recalled
Uwe mentioning that `plot.function` was listed by `methods()` there but
not in R2.13.x. I have now compiled the development version on two
It looks like the cog that has changed between release and devel is the
addition of export(plot.function) and S3method(plot, "function") to the
NAMESPACE of graphics.
Martin
Fedora installations and certain plot methods are still not being
listed. Details of the exact revision of R Devel are shown at the end of
this email.
As an example, consider:
require(mgcv)
Loading required package: mgcv
This is mgcv 1.7-6. For overview type 'help("mgcv-package")'.
methods("plot")
[1] plot.acf* plot.ACF* plot.augPred*
[4] plot.compareFits* plot.data.frame* plot.decomposed.ts*
[7] plot.default plot.dendrogram* plot.density
[10] plot.ecdf plot.factor* plot.formula*
[13] plot.function plot.gam plot.gls*
[16] plot.hclust* plot.histogram*plot.HoltWinters*
[19] plot.intervals.lmList* plot.isoreg* plot.lm
[22] plot.lme* plot.lmList* plot.medpolish*
[25] plot.mlm plot.nffGroupedData* plot.nfnGroupedData*
[28] plot.nls* plot.nmGroupedData*plot.pdMat*
[31] plot.ppr* plot.prcomp* plot.princomp*
[34] plot.profile.nls* plot.ranef.lme*plot.ranef.lmList*
[37] plot.shingle* plot.simulate.lme* plot.spec
[40] plot.stepfun plot.stl* plot.table*
[43] plot.trellis* plot.tsplot.tskernel*
[46] plot.TukeyHSD plot.Variogram*
Non-visible functions are asterisked
pmeth<- methods("plot")
grep("plot.mgcv.smooth", pmeth)
integer(0)
getS3method("plot", "mgcv.smooth")
Error in getS3method("plot", "mgcv.smooth") :
S3 method 'plot.mgcv.smooth' not found
pfun<- getAnywhere("plot.mgcv.smooth")
str(pfun)
List of 5
$ name : chr "plot.mgcv.smooth"
$ objs :List of 1
..$ :function (x, P = NULL, data = NULL, label = "", se1.mult = 1,
se2.mult = 2, partial.resids = FALSE, rug = TRUE, se = TRUE,
scale = -1, n = 100, n2 = 40, pers = FALSE, theta = 30, phi = 30,
jit = FALSE, xlab = NULL, ylab = NULL, main = NULL, ylim = NULL,
xlim = NULL, too.far = 0.1, shade = FALSE, shade.col = "gray80",
shift = 0, trans = I, by.resids = FALSE, scheme = NULL, ...)
$ where : chr "namespace:mgcv"
$ visible: logi FALSE
$ dups : logi FALSE
- attr(*, "class")= chr "getAnywhere"
Both `methods()` and `getS3method()` don't list/find this method, but
the function exists in the mgcv name space and this method will be used
via R's S3 dispatch system in `plot.gam()`.
Shouldn't this method be returned by either `methods()` or
`getS3method()`?
TIA,
Gavin
sessionInfo()
R Under development (unstable) (2011-08-26 r56801)
Platform: x86_64-unknown-linux-gnu (64-bit)
locale:
[1] LC_CTYPE=en_GB.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_GB.UTF-8LC_COLLATE=en_GB.UTF-8
[5] LC_MONETARY=en_GB.UTF-8LC_MESSAGES=en_GB.UTF-8
[7] LC_PAPER=C LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_GB.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods
base
other attached packages:
[1] mgcv_1.7-6
loaded via a namespace (and not attached):
[1] grid_2.14.0lattice_0.19-33Matrix_0.9996875-3
nlme_3.1-102
[5] tools_2.14.0
--
Computational Biology
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N. PO Box 19024 Seattle, WA 98109
Location: M1-B861
Telephone: 206 667-2793
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] matrix bands
> Jeremy David Silver
> on Fri, 26 Aug 2011 15:08:03 +0200 writes:
> Thanks for the suggestion, Martin! I looked at both
> Matrix::band and Matrix::bandSparse. Maybe I misunderstand the
> help pages and the examples, but from what I can see neither of
> them provides the functionality I was looking for.
> For the getter version of the function I was looking for, I
> can't use Matrix::band (by extracting the non-zero elements)
> because this would require the assumption that the specified
> band is entirely non-zero, e.g.:
> band3 <- function(x,n){
>x <- Matrix::band(x,n,n)
>x[x!=0] ## works for a dense matrix, but not if band n has zeroes
> }
> As for the setter version of the function I was looking for, it should
> set the values of a specified band in an existing matrix to be the
> vector of values provided. As far as I understand, Matrix::bandSparse
> constructs a matrix rather than modifying the values of an existing matrix.
Yes, you are right. The current bandSparse() doesn't do that...
and while you could use constructions like
diag(A[,-1]) <- dd
there will probably better (more efficient) alternatives.
> The functions included in my last post filled this gap. If Matrix::band
> and Matrix::bandSparse can set/get bands in this way, I would like to
> see how this can be done. If not, then perhaps they could be extended to
> achieve this functionality.
That's a good suggestion, and I'll look at it,
further I'll look at your examples of "getting" ...
and at the moment I'm still a bit curious why, i.e., to what end
/ in what application you need them [ rather than band(*,.,.) ].
We may e-talk about this off-public if you want.
Martin
> On 2011-08-26 14:08, Martin Maechler wrote:
> >> Jeremy David Silver
> >> on Fri, 26 Aug 2011 13:23:43 +0200 writes:
> > > Dear R developers, I was looking for a function analogous
> > > to base::diag() for getting and setting bands of a
> > > matrix. The closest I could find was Matrix::band(), but
> > > this was not exactly what I wanted for two
> > > reasons. Firstly, Matrix::band() returns a matrix rather
> > > than just the specified band. Secondly, Matrix::band()
> > > cannot be used for setting the values for a matrix band.
> >
> > Yes, but did you not look at help(band)
> > or not look carefully enough ?
> >
> > --->
> >
> >See Also:
> >
> > ‘bandSparse’ for the _construction_ of a banded sparse matrix
> > directly from its non-zero diagonals.
> >
> >
> >
> > > Setting or getting a matrix band could be of interest for
> > > sufficiently many users that you might consider including
> > > it in base R. Alternatively, something like this could be
> > > included in the Matrix package.
> >
> > well, see above and let us know if you see anything lacking in
> > bandSparse().
> > Till now we haven't got much feedback about it, and there may
> > well be room for improvement.
> >
> > Martin Maechler, ETH Zurich and co- maintainer("Matrix")
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] read.table segfaults
> fil2s <- read.table("../Data/fil2_s.txt", header = FALSE, sep = "\t")
Program received signal SIGSEGV, Segmentation fault.
0x0041c2e1 in RunGenCollect (size_needed=8192000) at memory.c:1514
1514PROCESS_NODES();
(gdb)
> sessionInfo()
R version 2.13.1 Patched (2011-08-25 r56798)
Platform: x86_64-unknown-linux-gnu (64-bit)
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=C LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
>
The text file 'fil2_s.txt' is Huge, around 11 million records and 17
variables, but ...?
--
Göran Broström
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] read.table segfaults
Another one:
The 'death.RData' was created about a year ago, but ...? Same info as below.
Göran
> load("../Data/death.RData")
> summary(death)
*** caught segfault ***
address 0x4e04959, cause 'memory not mapped'
Traceback:
1: match(x, levels)
2: factor(a, levels = ll[!(ll %in% exclude)], exclude = if (useNA ==
"no") NA)
3: table(object)
4: summary.factor(X[[6L]], ...)
5: FUN(X[[6L]], ...)
6: lapply(as.list(object), summary, maxsum = maxsum, digits = 12, ...)
7: summary.data.frame(death)
8: summary(death)
Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace
Selection:
2011/8/26 Göran Broström :
> > fil2s <- read.table("../Data/fil2_s.txt", header = FALSE, sep = "\t")
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x0041c2e1 in RunGenCollect (size_needed=8192000) at memory.c:1514
> 1514 PROCESS_NODES();
> (gdb)
>
> > sessionInfo()
> R version 2.13.1 Patched (2011-08-25 r56798)
> Platform: x86_64-unknown-linux-gnu (64-bit)
>
> locale:
> [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
> [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
> [5] LC_MONETARY=C LC_MESSAGES=en_US.UTF-8
> [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
> [9] LC_ADDRESS=C LC_TELEPHONE=C
> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
>
> attached base packages:
> [1] stats graphics grDevices utils datasets methods base
> >
>
> The text file 'fil2_s.txt' is Huge, around 11 million records and 17
> variables, but ...?
>
>
>
> --
> Göran Broström
>
--
Göran Broström
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] read.table segfaults
One further note:
No problem with R version 2.13.0 (2011-04-13)
Göran
2011/8/26 Göran Broström :
> Another one:
>
> The 'death.RData' was created about a year ago, but ...? Same info as below.
>
> Göran
>
>> load("../Data/death.RData")
>> summary(death)
>
> *** caught segfault ***
> address 0x4e04959, cause 'memory not mapped'
>
> Traceback:
> 1: match(x, levels)
> 2: factor(a, levels = ll[!(ll %in% exclude)], exclude = if (useNA ==
> "no") NA)
> 3: table(object)
> 4: summary.factor(X[[6L]], ...)
> 5: FUN(X[[6L]], ...)
> 6: lapply(as.list(object), summary, maxsum = maxsum, digits = 12, ...)
> 7: summary.data.frame(death)
> 8: summary(death)
>
> Possible actions:
> 1: abort (with core dump, if enabled)
> 2: normal R exit
> 3: exit R without saving workspace
> 4: exit R saving workspace
> Selection:
>
>
> 2011/8/26 Göran Broström :
>> > fil2s <- read.table("../Data/fil2_s.txt", header = FALSE, sep = "\t")
>>
>> Program received signal SIGSEGV, Segmentation fault.
>> 0x0041c2e1 in RunGenCollect (size_needed=8192000) at memory.c:1514
>> 1514 PROCESS_NODES();
>> (gdb)
>>
>> > sessionInfo()
>> R version 2.13.1 Patched (2011-08-25 r56798)
>> Platform: x86_64-unknown-linux-gnu (64-bit)
>>
>> locale:
>> [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
>> [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
>> [5] LC_MONETARY=C LC_MESSAGES=en_US.UTF-8
>> [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
>> [9] LC_ADDRESS=C LC_TELEPHONE=C
>> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
>>
>> attached base packages:
>> [1] stats graphics grDevices utils datasets methods base
>> >
>>
>> The text file 'fil2_s.txt' is Huge, around 11 million records and 17
>> variables, but ...?
>>
>>
>>
>> --
>> Göran Broström
>>
>
>
>
> --
> Göran Broström
>
--
Göran Broström
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] read.table segfaults
It does look like you've got a memory issue. perhaps using as.is=TRUE, and/or stringsAsFactors=FALSE will help as optional arguments to read.table if you don't specify these sorts of things, R can have to look through the file and figure out which columns are characters/factors etc and so the larger files cause more of a headache for R I'm guess. Hopefully someone else can comment further on this? I'd true toggling TRUE/FALSE for as.is and stringsAsFactors. do you have other objects loaded in memory as well? this file by itself might not be the problem - but it's a cumulative issue. have you checked the file structure in any other manner? how large (Mb/kb) is the file that you're trying to read? if you just read in parts of the file, is it okay? read.table(filename,header=FALSE,sep="\t",nrows=100) read.table(filename,header=FALSE,sep="\t",skip=2,nrows=100) -- View this message in context: http://r.789695.n4.nabble.com/read-table-segfaults-tp3771793p3771817.html Sent from the R devel mailing list archive at Nabble.com. __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] Problem in calling R functions from Matlab
Hi,
I wish to use R (version 2.13.1) from within Matlab(ver R2009a) on windows XP
plaform (on both 64 bit and 32 bit OS) . For this I have installed
StatConnector
(http://rcom.univie.ac.at/download/current/statconnDCOM.latest.exe) for calling
R from within Matlab (R2009a) on Windows XP platform. I have added all the
files of MATLAB_RLINK folder (downloaded from
http://www.mathworks.com/matlabcentral/fileexchange/5051) into the
default Matlab working path (D:\User\MATLAB) and also added the folder
MATLAB_RLINK to the matlab path C:\Program Files\MATLAB\R2009a\toolbox . It
seems that Matlab is able to connect to R however I am facing a problem while
trying to call R functions from within Matlab. I have loaded all required
libraries. Shown below is the sequence of commands tried in blue, outcomes in
black and errors/unexpected/ undesired outcomes in red colour, and my comments
in black,italic after the command:
>> [a b c] = openR
a = 1
b = ''
c = COM.StatConnectorSrv_StatConnector
>> Rdemo
b = 1 4 9 16 25 36 49 64 81 100
c = 2 5 10 17 26 37 50 65 82 101
I then tried running the Rdemo available on
http://www.mathworks.com/matlabcentral/fx_files/5051/1/content/Rdemo.html
evalR('demo("persp")') works well. Also the arithmetic functions all work well:
a = 1:10;
putRdata('a',a)
b = evalR('a^2')
evalR('b <- a^2');
evalR('c <- b + 1');
c = getRdata('c')
However Now copy the volcano data into MATLAB section and all related
sessions therefore dont work.
>> volcano = getRdata('volcano')
volcano = []
>> size(volcano)
ans = 0 0
>> surf(volcano);
(see the attached figure:surf_volcano_output.jpg)
It seems Matlab is able to read/get data from R. Other subsequent commands
related to this section dont work.
I tried looking at the loaded libraries in R and in Matlab. In R everything
works well, but doesnt seem to be so in Matlab:
>> [a b c] = evalR('.libPaths()')
a = C:/Program Files/R/R-2.13.1/library
b = 1
c = ''
>> [a b c] = evalR('library()')
a = 'ActiveX VT_ERROR: '
'base'
'ActiveX VT_ERROR: '
b = 1
c = ''
>> [a b c] = evalR('library(rgdal)')
a = 'rscproxy'
'lattice'
'fields'
'spam'
'rgdal'
'sp'
'stats'
'graphics'
'grDevices'
'utils'
'datasets'
'methods'
'base'
b = 1
c = ''
As it shows that rgdal library is loaded, also the rscproxy library is loaded,
in addition to all others. I tried following:
imshow(imread('testimg_p.tif')) displays the image properly, however calling
this from R using rgdal from within Matlab gives error;
>> [a b c] = evalR('img <- readGDAL("testimg_p.tif")')
a = []
b = 0
c = Invoke Error, Dispatch Exception: Object is static; operation not allowed
The command works from R prompt:
> img <- readGDAL("testimg_p.tif")
testimg_p.tif has GDAL driver GTiff
and has 280 rows and 272 columns
It is giving same error while loading gstat library:
>> [a b c] = evalR('library(gstat) ')
a = []
b = 0
c = Invoke Error, Dispatch Exception: Object is static; operation not allowed
I guess the error is not related to readGDAL or gstat library but something
else which I am unable to trace. I have also tried to explore if this is
problem related to version of Matlab/windows 64/32 bit or R version but even
that does not seem to be an issue. (I have tried this on two differemt machines
one Workstation with WinXP 64 bit OS with R version 2.13.1and Matlab Version
7.8.0 R2009a and also on another portable workstation with WinXP 32 bit OS with
R version 2.12.1 and Matlab Version 7.8.0 R2009a. This indicates that version
of R or Matlab or win version 64 bit/32 bit does not seem to be an issue.
I also checked for some solution on page
http://www.mathworks.com/matlabcentral/fileexchange/5051 where others also
faced somewhat similar problem and it was suggested to check for setting R_HOME
and PATH variables in the environment variables for your system. I understand I
had followed all instructions carefully and hopefully all paths/R_HOME were set
correctly, as the command evalR('.libPaths()') recognized R home path
correctly.
I request R team to help.
Vandita Srivastava
Scientist/Engineer "SE",
Indian Institute of Remote Sensing(NRSC),
ISRO/Department of Space, Govt. of India
>>Please don't print this Email unless you really need to - this will preserve
>>trees
on planet earth.
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] S3 methods in default namespace
Hi guys, Are there any plans for figuring out potential S3 methods and declaring them with S3method() in the automatic default NAMESPACE? Thanks, Michael [[alternative HTML version deleted]] __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] read.table segfaults
Scott googlemail.com> writes:
>
> It does look like you've got a memory issue. perhaps using
> as.is=TRUE, and/or stringsAsFactors=FALSE will help as optional arguments
> to read.table
>
> if you don't specify these sorts of things, R can have to look through the
> file and figure out which columns are characters/factors etc and so the
> larger files cause more of a headache for R I'm guess. Hopefully someone
> else can comment further on this? I'd true toggling TRUE/FALSE for as.is and
> stringsAsFactors.
>
>do you have other objects loaded in memory as well? this file by itself
> might not be the problem - but it's a cumulative issue.
>have you checked the file structure in any other manner?
>how large (Mb/kb) is the file that you're trying to read?
>if you just read in parts of the file, is it okay?
> read.table(filename,header=FALSE,sep="\t",nrows=100)
> read.table(filename,header=FALSE,sep="\t",skip=2,nrows=100)
There seem to be two issues here:
1. what can the original poster (OP) do to work around this problem?
(e.g. get the data into a relational data base and import it from
there; use something from the High Performance task view such as
ff or data.table ...)
2. reporting a bug -- according to the R FAQ, any low-level
(segmentation-fault-type) crash of R when one is not messing
around with dynamically loaded code constitutes a bug. Unfortunately,
debugging problems like this is a huge pain in the butt.
Goran, can you randomly or systematically generate an
object of this size, write it to disk, read it back in, and
generate the same error? In other words, does something like
set.seed(1001)
d <- data.frame(label=rep(LETTERS[1:11],1e6),
values=matrix(rep(1.0,11*17*1e6),ncol=17)
write.table(d,file="big.txt")
read.table("big.txt")
do the same thing?
Reducing it to this kind of reproducible example will make
it possible for others to debug it without needing to gain
access to your huge file ...
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] methods() not listing some S3 plot methods...?
On Fri, 2011-08-26 at 07:06 -0700, Martin Morgan wrote:
> On 08/26/2011 04:44 AM, Gavin Simpson wrote:
> > Dear List,
> >
> > This may be related to this email thread initiated by Ben Bolker last
> > month: https://stat.ethz.ch/pipermail/r-devel/2011-July/061630.html
> >
> > In answering this Question on StackOverflow
> > http://stackoverflow.com/q/7195628/429846 I noticed that `methods()` was
> > not listing some S3 methods for `plot()` provided by the mgcv package.
>
> Hi Gavin --
>
> In the mgcv NAMESPACE, the methods is not registered with S3method
> (which would have made it appear with a *) and is not export'ed; the
> author of the package apparently intends that it be strictly internal to
> the package. Dispatch works within the package name space, but not
> outside, e.g., a=list; class(a) = "mgcv.smooth"; plot(a) ends up at
> plot.default.
Thanks for the explanation Martin. This strikes me as being somewhat
suboptimal and not conducive to studying and understanding code. Now
that I am aware of the distinction I won't be surprised when methods
don't show up.
Cheers
G
> > At the time I wanted to check the development version of R as I recalled
> > Uwe mentioning that `plot.function` was listed by `methods()` there but
> > not in R2.13.x. I have now compiled the development version on two
>
> It looks like the cog that has changed between release and devel is the
> addition of export(plot.function) and S3method(plot, "function") to the
> NAMESPACE of graphics.
>
> Martin
>
> > Fedora installations and certain plot methods are still not being
> > listed. Details of the exact revision of R Devel are shown at the end of
> > this email.
> >
> > As an example, consider:
> >
> >> require(mgcv)
> > Loading required package: mgcv
> > This is mgcv 1.7-6. For overview type 'help("mgcv-package")'.
> >> methods("plot")
> > [1] plot.acf* plot.ACF* plot.augPred*
> > [4] plot.compareFits* plot.data.frame* plot.decomposed.ts*
> > [7] plot.default plot.dendrogram* plot.density
> > [10] plot.ecdf plot.factor* plot.formula*
> > [13] plot.function plot.gam plot.gls*
> > [16] plot.hclust* plot.histogram*plot.HoltWinters*
> > [19] plot.intervals.lmList* plot.isoreg* plot.lm
> > [22] plot.lme* plot.lmList* plot.medpolish*
> > [25] plot.mlm plot.nffGroupedData* plot.nfnGroupedData*
> > [28] plot.nls* plot.nmGroupedData*plot.pdMat*
> > [31] plot.ppr* plot.prcomp* plot.princomp*
> > [34] plot.profile.nls* plot.ranef.lme*plot.ranef.lmList*
> > [37] plot.shingle* plot.simulate.lme* plot.spec
> > [40] plot.stepfun plot.stl* plot.table*
> > [43] plot.trellis* plot.tsplot.tskernel*
> > [46] plot.TukeyHSD plot.Variogram*
> >
> > Non-visible functions are asterisked
> >
> >> pmeth<- methods("plot")
> >> grep("plot.mgcv.smooth", pmeth)
> > integer(0)
> >> getS3method("plot", "mgcv.smooth")
> > Error in getS3method("plot", "mgcv.smooth") :
> >S3 method 'plot.mgcv.smooth' not found
> >> pfun<- getAnywhere("plot.mgcv.smooth")
> >> str(pfun)
> > List of 5
> > $ name : chr "plot.mgcv.smooth"
> > $ objs :List of 1
> >..$ :function (x, P = NULL, data = NULL, label = "", se1.mult = 1,
> > se2.mult = 2, partial.resids = FALSE, rug = TRUE, se = TRUE,
> > scale = -1, n = 100, n2 = 40, pers = FALSE, theta = 30, phi = 30,
> > jit = FALSE, xlab = NULL, ylab = NULL, main = NULL, ylim = NULL,
> > xlim = NULL, too.far = 0.1, shade = FALSE, shade.col = "gray80",
> > shift = 0, trans = I, by.resids = FALSE, scheme = NULL, ...)
> > $ where : chr "namespace:mgcv"
> > $ visible: logi FALSE
> > $ dups : logi FALSE
> > - attr(*, "class")= chr "getAnywhere"
> >
> > Both `methods()` and `getS3method()` don't list/find this method, but
> > the function exists in the mgcv name space and this method will be used
> > via R's S3 dispatch system in `plot.gam()`.
> >
> > Shouldn't this method be returned by either `methods()` or
> > `getS3method()`?
> >
> > TIA,
> >
> > Gavin
> >
> >> sessionInfo()
> > R Under development (unstable) (2011-08-26 r56801)
> > Platform: x86_64-unknown-linux-gnu (64-bit)
> >
> > locale:
> > [1] LC_CTYPE=en_GB.UTF-8 LC_NUMERIC=C
> > [3] LC_TIME=en_GB.UTF-8LC_COLLATE=en_GB.UTF-8
> > [5] LC_MONETARY=en_GB.UTF-8LC_MESSAGES=en_GB.UTF-8
> > [7] LC_PAPER=C LC_NAME=C
> > [9] LC_ADDRESS=C LC_TELEPHONE=C
> > [11] LC_MEASUREMENT=en_GB.UTF-8 LC_IDENTIFICATION=C
> >
> > attached base packages:
> > [1] stats graphics grDevices utils datasets methods
> > base
> >
> > other attached packages:
> > [1] mgcv_1.7-6
> >
> > loaded via a namespace (and not attached):
> > [1] grid_2.14.0lattice_0.19-33Matrix_0.9996875-3
>
