[Rd] as.environment.list provides inconsistent results under torture
Hello,
Using R-devel (rev 53950), I get inconsistent results with
as.environment( VECSXP ) when gctorture is on.
Consider:
a <- list( aa = rnorm, bb = runif )
gctorture(TRUE)
as.environment( a )
The last line sometimes produces the correct environment, but sometimes
I get errors. Here are the three situations:
# good
> as.environment( a )
# not good
> as.environment( a )
Erreur dans length(x) : 'x' est manquant
# not good either
> as.environment( a )
Erreur dans list(NULL, list(aa = function (n, mean = 0, sd = 1) :
correspondance partielle de chaînes de caractères incorrecte
Is it because the call made by lang4 is not protected while evaluated in
this line :
case VECSXP: {
/* implement as.environment.list() {isObject(.) is false for a list} */
return(eval(lang4(install("list2env"), arg,
/*envir = */R_NilValue, /* parent = */R_EmptyEnv),
rho));
}
(BTW, this was detected in a long Rcpp-devel thread. See
http://comments.gmane.org/gmane.comp.lang.r.rcpp/1336)
Romain
--
Romain Francois
Professional R Enthusiast
+33(0) 6 28 91 30 30
http://romainfrancois.blog.free.fr
|- http://bit.ly/fT2rZM : highlight 0.2-5
|- http://bit.ly/gpCSpH : Evolution of Rcpp code size
`- http://bit.ly/hovakS : RcppGSL initial release
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] [Rcpp-devel] Loading a package using Rcpp Modules results in memory corruption
On Tue, Jan 11, 2011 at 2:41 PM, Romain Francois
wrote:
> Le 11/01/11 19:57, Romain Francois a écrit :
>
> Le 11/01/11 19:46, Douglas Bates a écrit :
>>
>>> On Tue, Jan 11, 2011 at 12:27 PM, Dominick
>>> Samperi wrote:
>>>
On Tue, Jan 11, 2011 at 1:20 PM, Romain
Francois
wrote:
>
> Le 11/01/11 19:12, Davor Cubranic a écrit :
>
>>
>> I think an important point from Doug has been lost in the subsequent
>> 20-odd messages of flamebombing:
>>
>> I do not see this as compatible with the C++ Design principle we use
whereby
protection / unprotection occurs relative to the end of scope -- and
not
after every single assignment or allocation.
In other words, gctorture() may well be a fine test for the C API
and
its
PROTECT and UNPROTECT at every step but possibly not quite as
much for
Rcpp.
>>>
>>> I don't think so. In the situation that Dominick is describing the C
>>> API is being used (calls to Rf_install, Rf_lang1, Rf_eval, ...) and
>>> you have to play by the rules of the C API. Essentially every time
>>> that you call a function in the C API that can cause a memory
>>> allocation you are open yourself to the possibility of having the
>>> garbage collector run and getting unprotected SEXPs blown away. And,
>>> naturally, Rf_eval can cause memory allocation.
>>>
>>> I understood Dominick to be saying that in the code related to
>>> Modules
>>> and the evaluator and all that which we have been trying to debug
>>> there are such cases of the use of the C API that are unsafe.
>>>
>>
>> Can anyone comment whether this is correct?
>>
>> Davor
>>
>
> Yep. Doug's analysis is right. Rcpp is implemented with the C R API,
> and
> apparently there were a few places where we were not careful enough.
> Most
> notably in calls to Rf_lcons and Rf_cons. This has been partially
> dealt with
> today.
>
Just for the record, Doug is summarizing my analysis, based on several
examples that I posted to this thread,
and that I am pleased to see have inspired some remedial action.
>>>
>>> Sorry for not responding earlier. I'm in the middle of teaching a
>>> short course.
>>>
>>> Dominick is correct that it was his analysis that picked up the
>>> failures to PROTECT/UNPROTECT in cases that were causing at least some
>>> of the problems with loading Modules. Credit where credit is due.
>>>
>>> As Romain has indicated, some of the problems have been fixed but
>>> apparently problems still remain. Debugging memory protection
>>> problems is often very difficult.
>>>
>>
>> It is. Not sure what is my next step here. Remaining problems seem not
>> directly related to Rcpp, but rather associated with lazy loading. See:
>>
>
> This now by not be related to Rcpp. See the thread on R-devel:
> http://comments.gmane.org/gmane.comp.lang.r.devel/26504
>
> Please whoever is interested in fixing this, send your remarks and patches
> to the R-devel mailing list.
>
This looks like the same problem that appeared in Rcpp, an unprotected SEXP
in
/src/main/envir.c, in the function do_as_environment(), case VECSXP of
the
switch. Here is modified code that seems to fix the problem, at least under
Linux:
case VECSXP: {
Rprintf("VECSXP as.environment\n");
/* implement as.environment.list() {isObject(.) is false for a list} */
SEXP sp = PROTECT(lang4(install("list2env"), arg,
/*envir = */R_NilValue, /* parent = */R_EmptyEnv));
SEXP res = eval(sp, rho);
UNPROTECT(1);
return res;
}
Dominick
> Romain
>
>
> --
> Romain Francois
> Professional R Enthusiast
> +33(0) 6 28 91 30 30
> http://romainfrancois.blog.free.fr
> |- http://bit.ly/fT2rZM : highlight 0.2-5
> |- http://bit.ly/gpCSpH : Evolution of Rcpp code size
> `- http://bit.ly/hovakS : RcppGSL initial release
>
>
> ___
> Rcpp-devel mailing list
> rcpp-de...@lists.r-forge.r-project.org
> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
>
[[alternative HTML version deleted]]
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] as.environment.list provides inconsistent results under torture
Interesting, I'd argue that the bug is in eval() not protecting its arguments
since the usual convention is for functions to protect its arguments...
On Jan 11, 2011, at 2:33 PM, Romain Francois wrote:
> Hello,
>
> Using R-devel (rev 53950), I get inconsistent results with as.environment(
> VECSXP ) when gctorture is on.
>
> Consider:
>
> a <- list( aa = rnorm, bb = runif )
> gctorture(TRUE)
> as.environment( a )
>
> The last line sometimes produces the correct environment, but sometimes I get
> errors. Here are the three situations:
>
> # good
> > as.environment( a )
>
>
> # not good
> > as.environment( a )
> Erreur dans length(x) : 'x' est manquant
>
> # not good either
> > as.environment( a )
> Erreur dans list(NULL, list(aa = function (n, mean = 0, sd = 1) :
> correspondance partielle de chaînes de caractères incorrecte
>
>
> Is it because the call made by lang4 is not protected while evaluated in this
> line :
>
>case VECSXP: {
> /* implement as.environment.list() {isObject(.) is false for a list} */
> return(eval(lang4(install("list2env"), arg,
> /*envir = */R_NilValue, /* parent = */R_EmptyEnv),
> rho));
>}
>
>
> (BTW, this was detected in a long Rcpp-devel thread. See
> http://comments.gmane.org/gmane.comp.lang.r.rcpp/1336)
>
> Romain
>
> --
> Romain Francois
> Professional R Enthusiast
> +33(0) 6 28 91 30 30
> http://romainfrancois.blog.free.fr
> |- http://bit.ly/fT2rZM : highlight 0.2-5
> |- http://bit.ly/gpCSpH : Evolution of Rcpp code size
> `- http://bit.ly/hovakS : RcppGSL initial release
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
>
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] as.environment.list provides inconsistent results under torture
No. Lots of internal functions expect their callers to protect their
arguments, for efficiency reasons. eval is called very often and
almost always with argument that are protected because they are in the
evaluation engine, so it would be wasteful and potentially very costly
if eval protected its arguments every time it is called. (I don't
tknow what the cost would be to do so in the current implementation
but it could be prohibitive if we moved to some different approaches,
so for now we whould continue to expect callers of eval to make sure
the argumetns are protected.)
Best,
luke
On Tue, 11 Jan 2011, Simon Urbanek wrote:
Interesting, I'd argue that the bug is in eval() not protecting its arguments
since the usual convention is for functions to protect its arguments...
On Jan 11, 2011, at 2:33 PM, Romain Francois wrote:
Hello,
Using R-devel (rev 53950), I get inconsistent results with as.environment(
VECSXP ) when gctorture is on.
Consider:
a <- list( aa = rnorm, bb = runif )
gctorture(TRUE)
as.environment( a )
The last line sometimes produces the correct environment, but sometimes I get
errors. Here are the three situations:
# good
as.environment( a )
# not good
as.environment( a )
Erreur dans length(x) : 'x' est manquant
# not good either
as.environment( a )
Erreur dans list(NULL, list(aa = function (n, mean = 0, sd = 1) :
correspondance partielle de chaînes de caractères incorrecte
Is it because the call made by lang4 is not protected while evaluated in this
line :
case VECSXP: {
/* implement as.environment.list() {isObject(.) is false for a list} */
return(eval(lang4(install("list2env"), arg,
/*envir = */R_NilValue, /* parent = */R_EmptyEnv),
rho));
}
(BTW, this was detected in a long Rcpp-devel thread. See
http://comments.gmane.org/gmane.comp.lang.r.rcpp/1336)
Romain
--
Romain Francois
Professional R Enthusiast
+33(0) 6 28 91 30 30
http://romainfrancois.blog.free.fr
|- http://bit.ly/fT2rZM : highlight 0.2-5
|- http://bit.ly/gpCSpH : Evolution of Rcpp code size
`- http://bit.ly/hovakS : RcppGSL initial release
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
--
Luke Tierney
Statistics and Actuarial Science
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa Phone: 319-335-3386
Department of Statistics andFax: 319-335-3017
Actuarial Science
241 Schaeffer Hall email: l...@stat.uiowa.edu
Iowa City, IA 52242 WWW: http://www.stat.uiowa.edu__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] as.environment.list provides inconsistent results under torture
On Jan 11, 2011, at 6:55 PM,
wrote:
> No. Lots of internal functions expect their callers to protect their
> arguments, for efficiency reasons. eval is called very often and almost
> always with argument that are protected because they are in the evaluation
> engine, so it would be wasteful and potentially very costly if eval protected
> its arguments every time it is called. (I don't tknow what the cost would be
> to do so in the current implementation but it could be prohibitive if we
> moved to some different approaches, so for now we whould continue to expect
> callers of eval to make sure the argumetns are protected.)
>
Fair enough. It would be nice if this was explicitly documented since eval() is
part of the API and I see several packages on CRAN using eval(LCONS(..),..) and
eval(listX(...),...) - and I don't blame them (partly because one of them is
mine ;)). Unfortunately all the examples in R-ext use implicitly protected
arguments (as function arguments or parts of larger already protected
constructs) so it's not obvious from that, either.
Thanks,
Simon
>
>
> On Tue, 11 Jan 2011, Simon Urbanek wrote:
>
>> Interesting, I'd argue that the bug is in eval() not protecting its
>> arguments since the usual convention is for functions to protect its
>> arguments...
>>
>> On Jan 11, 2011, at 2:33 PM, Romain Francois wrote:
>>
>>> Hello,
>>>
>>> Using R-devel (rev 53950), I get inconsistent results with as.environment(
>>> VECSXP ) when gctorture is on.
>>>
>>> Consider:
>>>
>>> a <- list( aa = rnorm, bb = runif )
>>> gctorture(TRUE)
>>> as.environment( a )
>>>
>>> The last line sometimes produces the correct environment, but sometimes I
>>> get errors. Here are the three situations:
>>>
>>> # good
as.environment( a )
>>>
>>>
>>> # not good
as.environment( a )
>>> Erreur dans length(x) : 'x' est manquant
>>>
>>> # not good either
as.environment( a )
>>> Erreur dans list(NULL, list(aa = function (n, mean = 0, sd = 1) :
>>> correspondance partielle de chaînes de caractères incorrecte
>>>
>>>
>>> Is it because the call made by lang4 is not protected while evaluated in
>>> this line :
>>>
>>> case VECSXP: {
>>> /* implement as.environment.list() {isObject(.) is false for a list} */
>>> return(eval(lang4(install("list2env"), arg,
>>> /*envir = */R_NilValue, /* parent = */R_EmptyEnv),
>>> rho));
>>> }
>>>
>>>
>>> (BTW, this was detected in a long Rcpp-devel thread. See
>>> http://comments.gmane.org/gmane.comp.lang.r.rcpp/1336)
>>>
>>> Romain
>>>
>>> --
>>> Romain Francois
>>> Professional R Enthusiast
>>> +33(0) 6 28 91 30 30
>>> http://romainfrancois.blog.free.fr
>>> |- http://bit.ly/fT2rZM : highlight 0.2-5
>>> |- http://bit.ly/gpCSpH : Evolution of Rcpp code size
>>> `- http://bit.ly/hovakS : RcppGSL initial release
>>>
>>> __
>>> R-devel@r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>>
>>>
>>
>> __
>> R-devel@r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel
>>
>
> --
> Luke Tierney
> Statistics and Actuarial Science
> Ralph E. Wareham Professor of Mathematical Sciences
> University of Iowa Phone: 319-335-3386
> Department of Statistics andFax: 319-335-3017
> Actuarial Science
> 241 Schaeffer Hall email: l...@stat.uiowa.edu
> Iowa City, IA 52242 WWW: http://www.stat.uiowa.edu
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
