Re: [Rd] all.equal applied to function closures
> Bill Dunlap
> on Mon, 30 Nov 2020 13:41:54 -0800 writes:
> To make the comparison more complete, all.equal.environment could compare
> the parents of the target and current environments. That would have to be
> recursive but could stop at the first 'top level environment' (the global,
> empty, or a package-related environment generally) and use identical
> there. E.g.,
> > f1 <- function(x) (function(){ expx <- exp(x) ; function(y) y + expx})()
> > all.equal(f1(2), f1(3))
> [1] "Environments: Component “expx”: Mean relative difference: 1.718282"
>
> [2] "Environments: Component “x”: Mean relative difference:
> 0.5"
> This is from the following, where I avoided putting the existing
> non-recursive all.equal.environment into the body of this one.
> all.equal.environment <-
> function(target, current, ...)
> {
> .all.equal.environment <- base::all.equal.environment # temporary hack
> stopifnot(is.environment(target), is.environment(current))
> if (identical(target, current)) {
> TRUE
> } else {
> msg <- NULL # TODO: check attributes
> # deal with emptyenv now since parent.env(emptyenv()) gives error
> # and topenv(emptyenv()) gives GlobalEnv
> eTarget <- identical(target, emptyenv()) ||
> identical(target,topenv(target))
> eCurrent <- identical(current, emptyenv()) ||
> identical(current,topenv(current))
> if (eTarget || eCurrent) {
> msg <- c(msg, paste("target is", format(target), "and current
> is", format(current)))
> } else {
> thisComparison <- .all.equal.environment(target, current, ...)
> if (!isTRUE(thisComparison)) {
> msg <- c(msg, thisComparison)
> }
> parentComparison <- Recall(parent.env(target),
> parent.env(current), ...)
> if (!isTRUE(parentComparison)) {
> msg <- c(msg, paste("", parentComparison))
> }
> }
> if (is.null(msg) || isTRUE(msg)) TRUE else msg
> }
> }
Thank you, Duncan and Bill (and Kevin for bringing up the
topic).
I agree all.equal() should work better with functions,
and I think probably it would make sense to define all.equal.function()
rather than put this into all.equal.default()
However, it's not quite clear if it is always desirable to check the
environments as well notably as that *is* done recursively.
Bill, I'm sure you've noticed that we did write all.equal.environment()
to work recursively... Actually, I had worked quite a bit at
that, too long ago to remember details, but the relevant svn log
entry is
r66640 | maechler | 2014-09-18 22:10:20 +0200 (Thu, 18 Sep 2014) | 1 line
more sophisticated all.equal.environment(): no longer "simple" infinite
recursions
Are you sure that code with the internal recursive do1()
function should/could not be amended where needed?
Martin
> On Mon, Nov 30, 2020 at 10:42 AM Duncan Murdoch
> wrote:
>
> > On 30/11/2020 1:05 p.m., Kevin Van Horn via R-devel wrote:
> > > Consider the following code:
> > >
> > > f <- function(x)function(y){x+y}
> > > all.equal(f(5), f(0))
> > >
> > > This returns TRUE, when it should return FALSE; I think it’s hard to
> > make the case that f(5) and f(0) are “approximately equal” in any
> > meaningful sense. Digging into the code for all.equal(), I see that
> > all.equal(f(5), f(0)) results in a call to all.equal.language(f(5),
f(0)),
> > which only compares the function texts for equality.
> > >
> > > If it is decided to leave this behavior as-is, then at least it should
> > be documented. Currently I cannot find any documentation for all.equal
> > applied to functions.
> >
> > Clearly it should also compare the environments of the two functions,
> > then it would see a difference:
> >
> > > all.equal(environment(f(5)), environment(f(0)))
> > [1] "Component “x”: Mean relative difference: 1"
> >
> > Changing the first few lines from
> >
> > if (is.language(target) || is.function(target))
> > return(all.equal.language(target, current, ...))
> >
> > to
> >
> > if (is.function(target)) {
> > msg <- all.equal.language(target, current, ...)
> > if (isTRUE(msg)) {
> > msg <- all.equal.environment(environment(target),
> > environment(current), ...)
> > if (is.character(msg))
> >msg <- paste("Environments:", msg)
> > }
> > return(msg)
> > }
> > if (is.language(tar
Re: [Rd] all.equal applied to function closures
Probably all.equal.environment's do1() could be enhanced to do the
recursion (and look at the environments' attributes). I wrote a separate
function because it was easier to experiment that way (e.g., when to stop
recursing - it stops when one environment is a top-level environment or the
empty environment). I've been thinking about similar recursion options for
ls.str() - it would make it easier to debug refClass and R6 code where the
data is somewhere in a stack of environments.
> E1 <- list2env(list(X=1.1, Y=1.2), parent=list2env(list(p=1.3),
parent=baseenv()))
> E2 <- list2env(list(X=1.1, Y=1.2), parent=list2env(list(p=1.4),
parent=baseenv()))
> base::all.equal.environment(E1,E2)
[1] TRUE
> globalenv()$all.equal.environment(E1,E2)
[1] " Component “p”: Mean relative difference: 0.07692308"
>
> E3 <- list2env(list(X=1.1, Y=1.2), parent=list2env(list(p=1.5),
parent=new.env(parent=baseenv(
> base::all.equal.environment(E1,E3)
[1] TRUE
> globalenv()$all.equal.environment(E1,E3)
[1] " Component “p”: Mean relative difference: 0.1538462"
[2] " target is and current is
"
On Tue, Dec 1, 2020 at 1:31 AM Martin Maechler
wrote:
> > Bill Dunlap
> > on Mon, 30 Nov 2020 13:41:54 -0800 writes:
>
> > To make the comparison more complete, all.equal.environment could
> compare
> > the parents of the target and current environments. That would have
> to be
> > recursive but could stop at the first 'top level environment' (the
> global,
> > empty, or a package-related environment generally) and use identical
> > there. E.g.,
>
> > > f1 <- function(x) (function(){ expx <- exp(x) ; function(y) y +
> expx})()
> > > all.equal(f1(2), f1(3))
> > [1] "Environments: Component “expx”: Mean relative difference:
> 1.718282"
> >
> > [2] "Environments: Component “x”: Mean relative
> difference:
> > 0.5"
>
> > This is from the following, where I avoided putting the existing
> > non-recursive all.equal.environment into the body of this one.
>
> > all.equal.environment <-
> > function(target, current, ...)
> > {
> > .all.equal.environment <- base::all.equal.environment #
> temporary hack
> > stopifnot(is.environment(target), is.environment(current))
> > if (identical(target, current)) {
> > TRUE
> > } else {
> > msg <- NULL # TODO: check attributes
> > # deal with emptyenv now since parent.env(emptyenv()) gives
> error
> > # and topenv(emptyenv()) gives GlobalEnv
> > eTarget <- identical(target, emptyenv()) ||
> > identical(target,topenv(target))
> > eCurrent <- identical(current, emptyenv()) ||
> > identical(current,topenv(current))
> > if (eTarget || eCurrent) {
> > msg <- c(msg, paste("target is", format(target), "and
> current
> > is", format(current)))
> > } else {
> > thisComparison <- .all.equal.environment(target,
> current, ...)
> > if (!isTRUE(thisComparison)) {
> > msg <- c(msg, thisComparison)
> > }
> > parentComparison <- Recall(parent.env(target),
> > parent.env(current), ...)
> > if (!isTRUE(parentComparison)) {
> > msg <- c(msg, paste("",
> parentComparison))
> > }
> > }
> > if (is.null(msg) || isTRUE(msg)) TRUE else msg
> > }
> > }
>
> Thank you, Duncan and Bill (and Kevin for bringing up the
> topic).
>
> I agree all.equal() should work better with functions,
>
> and I think probably it would make sense to define all.equal.function()
> rather than put this into all.equal.default()
>
> However, it's not quite clear if it is always desirable to check the
> environments as well notably as that *is* done recursively.
>
> Bill, I'm sure you've noticed that we did write all.equal.environment()
> to work recursively... Actually, I had worked quite a bit at
> that, too long ago to remember details, but the relevant svn log
> entry is
>
> r66640 | maechler | 2014-09-18 22:10:20 +0200 (Thu, 18 Sep 2014) | 1 line
>
> more sophisticated all.equal.environment(): no longer "simple" infinite
> recursions
>
>
> Are you sure that code with the internal recursive do1()
> function should/could not be amended where needed?
>
> Martin
>
> > On Mon, Nov 30, 2020 at 10:42 AM Duncan Murdoch <
> murdoch.dun...@gmail.com>
> > wrote:
> >
> > > On 30/11/2020 1:05 p.m., Kevin Van Horn via R-devel wrote:
> > > > Consider the following code:
> > > >
> > > > f <- function(x)function(y){x+y}
> > > > all.equal(f(5), f(0))
> > > >
> > > > This returns TRUE, when it should return FALSE; I think it’s
> hard to
> > > make the case that f(5) and
Re: [Rd] all.equal applied to function closures
One more comment: it might be worthwhile reporting on a case where
identical(e1, e2) fails when those are the environments associated with
two functions (though I think not by default in all.equal.environment).
Functions can modify variables in their environment, so examples like
the open.account example in the R-intro manual would distinguish between
ross and robert even if the balances matched.
Duncan Murdoch
On 01/12/2020 10:37 a.m., Bill Dunlap wrote:
Probably all.equal.environment's do1() could be enhanced to do the
recursion (and look at the environments' attributes). I wrote a
separate function because it was easier to experiment that way (e.g.,
when to stop recursing - it stops when one environment is a top-level
environment or the empty environment). I've been thinking about similar
recursion options for ls.str() - it would make it easier to debug
refClass and R6 code where the data is somewhere in a stack of environments.
> E1 <- list2env(list(X=1.1, Y=1.2), parent=list2env(list(p=1.3),
parent=baseenv()))
> E2 <- list2env(list(X=1.1, Y=1.2), parent=list2env(list(p=1.4),
parent=baseenv()))
> base::all.equal.environment(E1,E2)
[1] TRUE
> globalenv()$all.equal.environment(E1,E2)
[1] " Component “p”: Mean relative difference: 0.07692308"
>
> E3 <- list2env(list(X=1.1, Y=1.2), parent=list2env(list(p=1.5),
parent=new.env(parent=baseenv(
> base::all.equal.environment(E1,E3)
[1] TRUE
> globalenv()$all.equal.environment(E1,E3)
[1] " Component “p”: Mean relative difference: 0.1538462"
[2] " target is and current
is "
On Tue, Dec 1, 2020 at 1:31 AM Martin Maechler
mailto:maech...@stat.math.ethz.ch>> wrote:
> Bill Dunlap
> on Mon, 30 Nov 2020 13:41:54 -0800 writes:
> To make the comparison more complete, all.equal.environment
could compare
> the parents of the target and current environments. That
would have to be
> recursive but could stop at the first 'top level environment'
(the global,
> empty, or a package-related environment generally) and use
identical
> there. E.g.,
> > f1 <- function(x) (function(){ expx <- exp(x) ; function(y)
y + expx})()
> > all.equal(f1(2), f1(3))
> [1] "Environments: Component “expx”: Mean relative
difference: 1.718282"
>
> [2] "Environments: Component “x”: Mean relative
difference:
> 0.5"
> This is from the following, where I avoided putting the existing
> non-recursive all.equal.environment into the body of this one.
> all.equal.environment <-
> function(target, current, ...)
> {
> .all.equal.environment <- base::all.equal.environment #
temporary hack
> stopifnot(is.environment(target), is.environment(current))
> if (identical(target, current)) {
> TRUE
> } else {
> msg <- NULL # TODO: check attributes
> # deal with emptyenv now since parent.env(emptyenv())
gives error
> # and topenv(emptyenv()) gives GlobalEnv
> eTarget <- identical(target, emptyenv()) ||
> identical(target,topenv(target))
> eCurrent <- identical(current, emptyenv()) ||
> identical(current,topenv(current))
> if (eTarget || eCurrent) {
> msg <- c(msg, paste("target is", format(target),
"and current
> is", format(current)))
> } else {
> thisComparison <- .all.equal.environment(target,
current, ...)
> if (!isTRUE(thisComparison)) {
> msg <- c(msg, thisComparison)
> }
> parentComparison <- Recall(parent.env(target),
> parent.env(current), ...)
> if (!isTRUE(parentComparison)) {
> msg <- c(msg, paste("",
parentComparison))
> }
> }
> if (is.null(msg) || isTRUE(msg)) TRUE else msg
> }
> }
Thank you, Duncan and Bill (and Kevin for bringing up the
topic).
I agree all.equal() should work better with functions,
and I think probably it would make sense to define all.equal.function()
rather than put this into all.equal.default()
However, it's not quite clear if it is always desirable to check the
environments as well notably as that *is* done recursively.
Bill, I'm sure you've noticed that we did write all.equal.environment()
to work recursively... Actually, I had worked quite a bit at
that, too long ago to remember details, but the relevant svn log
entry is
r66640 | maechler | 2014-09-18 22:10:20 +0200 (Thu, 18 Sep 2014) | 1
line
more sophisticated all
Re: [Rd] [External] Re: .Internal(quit(...)): system call failed: Cannot allocate memory
Thank you Luke, I tried your suggestion about R_MAX_VSIZE but I am not able to get the error you are getting. I tried recent R devel as I have seen you made a change to GC there. My machine is 128GB, free -h reports 125GB available. I tried to set 128, 125 and 100. In all cases the result is "Command terminated by signal 9". Each took around 6-6.5h. Details below, if it tells you anything how could I optimize it (or raise an exception early) please do let me know. R 4.0.3 unset R_MAX_VSIZE User time (seconds): 40447.92 System time (seconds): 4034.37 Percent of CPU this job got: 201% Elapsed (wall clock) time (h:mm:ss or m:ss): 6:07:59 Maximum resident set size (kbytes): 127261184 Major (requiring I/O) page faults: 72441 Minor (reclaiming a frame) page faults: 3315491751 Voluntary context switches: 381446 Involuntary context switches: 529554 File system inputs: 108339200 File system outputs: 120 R-devel 2020-11-27 r79522 unset R_MAX_VSIZE User time (seconds): 40713.52 System time (seconds): 4039.52 Percent of CPU this job got: 198% Elapsed (wall clock) time (h:mm:ss or m:ss): 6:15:52 Maximum resident set size (kbytes): 127254796 Major (requiring I/O) page faults: 72810 Minor (reclaiming a frame) page faults: 3433589848 Voluntary context switches: 384363 Involuntary context switches: 609024 File system inputs: 108467064 File system outputs: 112 R_MAX_VSIZE=128Gb User time (seconds): 40411.13 System time (seconds): 4227.99 Percent of CPU this job got: 198% Elapsed (wall clock) time (h:mm:ss or m:ss): 6:14:01 Maximum resident set size (kbytes): 127249316 Major (requiring I/O) page faults: 88500 Minor (reclaiming a frame) page faults: 3544520527 Voluntary context switches: 384117 Involuntary context switches: 545397 File system inputs: 111675896 File system outputs: 120 R_MAX_VSIZE=125Gb User time (seconds): 40246.83 System time (seconds): 4042.76 Percent of CPU this job got: 201% Elapsed (wall clock) time (h:mm:ss or m:ss): 6:06:56 Maximum resident set size (kbytes): 127254200 Major (requiring I/O) page faults: 63867 Minor (reclaiming a frame) page faults: 3449493803 Voluntary context switches: 370753 Involuntary context switches: 614607 File system inputs: 106322880 File system outputs: 112 R_MAX_VSIZE=100Gb User time (seconds): 41837.10 System time (seconds): 3979.57 Percent of CPU this job got: 192% Elapsed (wall clock) time (h:mm:ss or m:ss): 6:36:34 Maximum resident set size (kbytes): 127256940 Major (requiring I/O) page faults: 66829 Minor (reclaiming a frame) page faults: 3357778594 Voluntary context switches: 391149 Involuntary context switches: 646410 File system inputs: 106605648 File system outputs: 120 On Fri, Nov 27, 2020 at 10:18 PM wrote: > > On Thu, 26 Nov 2020, Jan Gorecki wrote: > > > Thank you Luke for looking into it. Your knowledge of gc is definitely > > helpful here. I put comments inline below. > > > > Best, > > Jan > > > > On Wed, Nov 25, 2020 at 10:38 PM wrote: > >> > >> On Tue, 24 Nov 2020, Jan Gorecki wrote: > >> > >>> As for other calls to system. I avoid calling system. In the past I > >>> had some (to get memory stats from OS), but they were failing with > >>> exactly the same issue. So yes, if I would add call to system before > >>> calling quit, I believe it would fail with the same error. > >>> At the same time I think (although I am not sure) that new allocations > >>> made in R are working fine. So R seems to reserve some memory and can > >>> continue to operate, while external call like system will fail. Maybe > >>> it is like this by design, don't know. > >> > >> Thanks for the report on quit(). We're exploring how to make the > >> cleanup on exit more robust to low memory situations like these. > >> > >>> > >>> Aside from this problem that is easy to report due to the warning > >>> message, I think that gc() is choking at the same time. I tried to > >>> make reproducible example for that, multiple times but couldn't, let > >>> me try one more time. > >>> It happens to manifest when there is 4e8+ unique characters/factors in > >>> an R session. I am able to reproduce it using data.table and dplyr > >>> (0.8.4 because 1.0.0+ fails even sooner), but using base R is not easy > >>> because of the size. I described briefly problem in: > >>> https://github.com/h2oai/db-benchmark/issues/110 > >> > >> Because of the design of R's character vectors, with each element > >> allocated separately, R is never going to be great at handling huge > >> numbers of distinct strings. But it can do an adequate job given > >> enough memory to work with.
Re: [Rd] [External] Re: .Internal(quit(...)): system call failed: Cannot allocate memory
The fact that your max resident size isn't affected looks odd. Are you setting the environment variable outside R? When I run env R_MAX_VSIZE=16Gb /usr/bin/time bin/Rscript jg.R 1e9 2e0 0 0 (your code in jg.R). I get a quick failure with 11785524maxresident)k Best, luke On Tue, 1 Dec 2020, Jan Gorecki wrote: Thank you Luke, I tried your suggestion about R_MAX_VSIZE but I am not able to get the error you are getting. I tried recent R devel as I have seen you made a change to GC there. My machine is 128GB, free -h reports 125GB available. I tried to set 128, 125 and 100. In all cases the result is "Command terminated by signal 9". Each took around 6-6.5h. Details below, if it tells you anything how could I optimize it (or raise an exception early) please do let me know. R 4.0.3 unset R_MAX_VSIZE User time (seconds): 40447.92 System time (seconds): 4034.37 Percent of CPU this job got: 201% Elapsed (wall clock) time (h:mm:ss or m:ss): 6:07:59 Maximum resident set size (kbytes): 127261184 Major (requiring I/O) page faults: 72441 Minor (reclaiming a frame) page faults: 3315491751 Voluntary context switches: 381446 Involuntary context switches: 529554 File system inputs: 108339200 File system outputs: 120 R-devel 2020-11-27 r79522 unset R_MAX_VSIZE User time (seconds): 40713.52 System time (seconds): 4039.52 Percent of CPU this job got: 198% Elapsed (wall clock) time (h:mm:ss or m:ss): 6:15:52 Maximum resident set size (kbytes): 127254796 Major (requiring I/O) page faults: 72810 Minor (reclaiming a frame) page faults: 3433589848 Voluntary context switches: 384363 Involuntary context switches: 609024 File system inputs: 108467064 File system outputs: 112 R_MAX_VSIZE=128Gb User time (seconds): 40411.13 System time (seconds): 4227.99 Percent of CPU this job got: 198% Elapsed (wall clock) time (h:mm:ss or m:ss): 6:14:01 Maximum resident set size (kbytes): 127249316 Major (requiring I/O) page faults: 88500 Minor (reclaiming a frame) page faults: 3544520527 Voluntary context switches: 384117 Involuntary context switches: 545397 File system inputs: 111675896 File system outputs: 120 R_MAX_VSIZE=125Gb User time (seconds): 40246.83 System time (seconds): 4042.76 Percent of CPU this job got: 201% Elapsed (wall clock) time (h:mm:ss or m:ss): 6:06:56 Maximum resident set size (kbytes): 127254200 Major (requiring I/O) page faults: 63867 Minor (reclaiming a frame) page faults: 3449493803 Voluntary context switches: 370753 Involuntary context switches: 614607 File system inputs: 106322880 File system outputs: 112 R_MAX_VSIZE=100Gb User time (seconds): 41837.10 System time (seconds): 3979.57 Percent of CPU this job got: 192% Elapsed (wall clock) time (h:mm:ss or m:ss): 6:36:34 Maximum resident set size (kbytes): 127256940 Major (requiring I/O) page faults: 66829 Minor (reclaiming a frame) page faults: 3357778594 Voluntary context switches: 391149 Involuntary context switches: 646410 File system inputs: 106605648 File system outputs: 120 On Fri, Nov 27, 2020 at 10:18 PM wrote: On Thu, 26 Nov 2020, Jan Gorecki wrote: Thank you Luke for looking into it. Your knowledge of gc is definitely helpful here. I put comments inline below. Best, Jan On Wed, Nov 25, 2020 at 10:38 PM wrote: On Tue, 24 Nov 2020, Jan Gorecki wrote: As for other calls to system. I avoid calling system. In the past I had some (to get memory stats from OS), but they were failing with exactly the same issue. So yes, if I would add call to system before calling quit, I believe it would fail with the same error. At the same time I think (although I am not sure) that new allocations made in R are working fine. So R seems to reserve some memory and can continue to operate, while external call like system will fail. Maybe it is like this by design, don't know. Thanks for the report on quit(). We're exploring how to make the cleanup on exit more robust to low memory situations like these. Aside from this problem that is easy to report due to the warning message, I think that gc() is choking at the same time. I tried to make reproducible example for that, multiple times but couldn't, let me try one more time. It happens to manifest when there is 4e8+ unique characters/factors in an R session. I am able to reproduce it using data.table and dplyr (0.8.4 because 1.0.0+ fails even sooner), but using base R is not easy because of the size. I described briefly problem in: https://github.com/h2oai/db-benchmark/issues/110 Because of the design of R's character vectors, with each element allocated separately, R is never going to be great at handling huge numbers o
Re: [Rd] [External] Re: .Internal(quit(...)): system call failed: Cannot allocate memory
Yes, I do set outside of R, in shell: R_MAX_VSIZE=100Gb SRC_DATANAME=G1_1e9_2e0_0_0 /usr/bin/time -v Rscript datatable/groupby-datatable.R I think it might be related to allocations made with malloc rather than R_alloc. Probably malloc allocation is not capped by setting this env var. If so, then I have to limit memory on OS/shell level. As you mentioned before. Best On Tue, Dec 1, 2020 at 6:54 PM wrote: > > The fact that your max resident size isn't affected looks odd. Are > you setting the environment variable outside R? When I run > > env R_MAX_VSIZE=16Gb /usr/bin/time bin/Rscript jg.R 1e9 2e0 0 0 > > (your code in jg.R). I get a quick failure with 11785524maxresident)k > > Best, > > luke > > On Tue, 1 Dec 2020, Jan Gorecki wrote: > > > Thank you Luke, > > > > I tried your suggestion about R_MAX_VSIZE but I am not able to get the > > error you are getting. > > I tried recent R devel as I have seen you made a change to GC there. > > My machine is 128GB, free -h reports 125GB available. I tried to set > > 128, 125 and 100. In all cases the result is "Command terminated by > > signal 9". Each took around 6-6.5h. > > Details below, if it tells you anything how could I optimize it (or > > raise an exception early) please do let me know. > > > > R 4.0.3 > > > > unset R_MAX_VSIZE > >User time (seconds): 40447.92 > >System time (seconds): 4034.37 > >Percent of CPU this job got: 201% > >Elapsed (wall clock) time (h:mm:ss or m:ss): 6:07:59 > >Maximum resident set size (kbytes): 127261184 > >Major (requiring I/O) page faults: 72441 > >Minor (reclaiming a frame) page faults: 3315491751 > >Voluntary context switches: 381446 > >Involuntary context switches: 529554 > >File system inputs: 108339200 > >File system outputs: 120 > > > > R-devel 2020-11-27 r79522 > > > > unset R_MAX_VSIZE > >User time (seconds): 40713.52 > >System time (seconds): 4039.52 > >Percent of CPU this job got: 198% > >Elapsed (wall clock) time (h:mm:ss or m:ss): 6:15:52 > >Maximum resident set size (kbytes): 127254796 > >Major (requiring I/O) page faults: 72810 > >Minor (reclaiming a frame) page faults: 3433589848 > >Voluntary context switches: 384363 > >Involuntary context switches: 609024 > >File system inputs: 108467064 > >File system outputs: 112 > > > > R_MAX_VSIZE=128Gb > >User time (seconds): 40411.13 > >System time (seconds): 4227.99 > >Percent of CPU this job got: 198% > >Elapsed (wall clock) time (h:mm:ss or m:ss): 6:14:01 > >Maximum resident set size (kbytes): 127249316 > >Major (requiring I/O) page faults: 88500 > >Minor (reclaiming a frame) page faults: 3544520527 > >Voluntary context switches: 384117 > >Involuntary context switches: 545397 > >File system inputs: 111675896 > >File system outputs: 120 > > > > R_MAX_VSIZE=125Gb > >User time (seconds): 40246.83 > >System time (seconds): 4042.76 > >Percent of CPU this job got: 201% > >Elapsed (wall clock) time (h:mm:ss or m:ss): 6:06:56 > >Maximum resident set size (kbytes): 127254200 > >Major (requiring I/O) page faults: 63867 > >Minor (reclaiming a frame) page faults: 3449493803 > >Voluntary context switches: 370753 > >Involuntary context switches: 614607 > >File system inputs: 106322880 > >File system outputs: 112 > > > > R_MAX_VSIZE=100Gb > >User time (seconds): 41837.10 > >System time (seconds): 3979.57 > >Percent of CPU this job got: 192% > >Elapsed (wall clock) time (h:mm:ss or m:ss): 6:36:34 > >Maximum resident set size (kbytes): 127256940 > >Major (requiring I/O) page faults: 66829 > >Minor (reclaiming a frame) page faults: 3357778594 > >Voluntary context switches: 391149 > >Involuntary context switches: 646410 > >File system inputs: 106605648 > >File system outputs: 120 > > > > On Fri, Nov 27, 2020 at 10:18 PM wrote: > >> > >> On Thu, 26 Nov 2020, Jan Gorecki wrote: > >> > >>> Thank you Luke for looking into it. Your knowledge of gc is definitely > >>> helpful here. I put comments inline below. > >>> > >>> Best, > >>> Jan > >>> > >>> On Wed, Nov 25, 2020 at 10:38 PM wrote: > > On Tue, 24 Nov 2020, Jan Gorecki wrote: > > > As for other calls to system. I avoid calling system. In the past I > > had some (to get memory stats from OS), but they were failing with > > exactly the same issue. So yes, if I would add call to system before > > calling quit, I believe it would fail with the same error. > > At the same time I think (although I am not sure) that new allocations > > made in R are working fine. So R seems to reserve some memory and can > > continue to operate, while external call like system
[Rd] order() and sort() on single row data.frames
Hi,
not sure whether this belongs here or has been reported/asked before.
In the current R devel the behavior of order() and sort() on data.frames
with a single row has changed.
Before (release):
> sort(data.frame("b", "a"))
X.a. X.b.
1 a b
Now (devel):
> sort(data.frame("b", "a"))
X.b.
1 b
I did not find any specific documentation in the daily news log on this.
Thanks,
Benjamin
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] order() and sort() on single row data.frames
> Benjamin Becker writes:
> Hi,
> not sure whether this belongs here or has been reported/asked before.
> In the current R devel the behavior of order() and sort() on data.frames
> with a single row has changed.
> Before (release):
>> sort(data.frame("b", "a"))
> X.a. X.b.
> 1 a b
> Now (devel):
>> sort(data.frame("b", "a"))
> X.b.
> 1 b
> I did not find any specific documentation in the daily news log on
> this.
Work in progress. Likely this will become an error: ? sort clearly says
it does not work on data frames.
Best
-k
> Thanks,
> Benjamin
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] translation domain is wrong in stopifnot()?
I don't know if this would be considered a bug in either stopifnot() or (n)gettext(), or not a bug at all, but sometimes the translation domain is not set properly for stopifnot() messages, so they won't be translated. E.g. Sys.setenv(LANGUAGE = "de") # This is good stopifnot(FALSE) #> Fehler: FALSE ist nicht TRUE f <- function() stopifnot(FALSE) # This is not f() #> Fehler in f() : FALSE is not TRUE Gabor __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] all.equal applied to function closures
> Bill, I'm sure you've noticed that we did write all.equal.environment() > to work recursively... Actually, I had worked quite a bit at > that, too long ago to remember details, but the relevant svn log > entry is > > r66640 | maechler | 2014-09-18 22:10:20 +0200 (Thu, 18 Sep 2014) | 1 line > > more sophisticated all.equal.environment(): no longer "simple" infinite > recursions > I haven't checked the above reference. But I would like to note the following behaviour: #e group e = new.env () e1 = new.env () e$e = e1 e1$e = e #f group f = new.env () f1 = new.env () f2 = new.env () f$e = f1 f1$e = f2 f2$e = f all.equal (e, f) I tried a number of examples with circular references. All worked correctly, except for "identical" environments nested an unequal number of times. I suspect there may be other special cases. __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
