Re: [Rd] stopping finalizers
It might help if you could be more specific about what the issue is -- if they are out of scope why does it matter whether the finalizers run? Generically two approaches I can think of: you keep track of whenit is safe to fully run your finalizers and have your finalizers put the objects on a linked list if it isn't safe to run the finalizer now and clear the list each time you make a new one keep track of your objects with a weak list andturn them into strong references before your calls, then drop the list after. I'm pretty sure we don't have a mechanism for temporarily suspending running the finalizers but it is probably fairly easy to add if that is the only option. I might be able to think of other options with more details on the issue. Best, luke On Tue, 12 Feb 2013, Thomas Lumley wrote: Is there some way to prevent finalizers running during a section of code? I have a package that includes R objects linked to database tables. To maintain the call-by-value semantics, tables are copied rather than modified, and the extra tables are removed by finalizers during garbage collection. However, if the garbage collection occurs in the middle of processing another SQL query (which is relatively likely, since that's where the memory allocations are) there are problems with the database interface. Since the guarantees for the finalizer are "at most once, not before the object is out of scope" it seems harmless to be able to prevent finalizers from running during a particular code block, but I can't see any way to do it. Suggestions? -thomas -- Luke Tierney Chair, Statistics and Actuarial Science Ralph E. Wareham Professor of Mathematical Sciences University of Iowa Phone: 319-335-3386 Department of Statistics andFax: 319-335-3017 Actuarial Science 241 Schaeffer Hall email: luke-tier...@uiowa.edu Iowa City, IA 52242 WWW: http://www.stat.uiowa.edu __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] stopping finalizers
I'm not sure I got your problem right, but you can keep a named copy
of your not-to-be-finalized object as long as it needs to be around,
so it doesn't go out of scope too early.
Something like:
local({
dontFinalizeMe <- obj
##
## code which creates copies, overwrites, and indirectly uses 'obj'
##
})
hth,
--
Antonio, Fabio Di Narzo,
Biostatistician
Mount Sinai School of Medicine, NY.
2013/2/12 Thomas Lumley :
> Is there some way to prevent finalizers running during a section of code?
>
> I have a package that includes R objects linked to database tables. To
> maintain the call-by-value semantics, tables are copied rather than
> modified, and the extra tables are removed by finalizers during garbage
> collection.
>
> However, if the garbage collection occurs in the middle of processing
> another SQL query (which is relatively likely, since that's where the
> memory allocations are) there are problems with the database interface.
>
> Since the guarantees for the finalizer are "at most once, not before the
> object is out of scope" it seems harmless to be able to prevent finalizers
> from running during a particular code block, but I can't see any way to do
> it.
>
> Suggestions?
>
> -thomas
>
>
> --
> Thomas Lumley
> Professor of Biostatistics
> University of Auckland
>
> [[alternative HTML version deleted]]
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] stopping finalizers
Luke, We're actually adopting the first of your generic approaches. As a more concrete description: There are R objects representing survey data sets, with the data stored in a database table. The subset() method, when applied to these objects, creates a new table indicating which rows of the data table are in the subset -- we don't modify the original table, because that breaks the call-by-value semantics. When the subset object in R goes out of scope, we need to delete the extra database table. I have been doing this with a finalizer on an environment that's part of the subset object in R. This all worked fine with JDBC, but the native database interface requires that all communications with the database come in send/receive pairs. Since R is single-threaded, this would normally not be any issue. However, since garbage collection can happen at any time, it is possible that the send part of the finalizer query "drop table _sbs_whatever" comes between the send and receive of some other query, and the database connection then falls over. So, I'm happy for the finalizer to run at any time except during a small critical section of R code. In this particular case the finalizer only issues "drop table" queries, and it doesn't need to know if they succeed, so we can keep a lock in the database connection and just store any "drop table" queries that arrive during a database operation for later execution. More generally, though, the fact that no R operation is atomic with respect to garbage collection seems to make it a bit difficult to use finalizers -- if you need a finalizer, it will often be in order to access and free some external resource, which is when the race conditions can matter. What I was envisaging was something like without_gc(expr) to evaluate expr with the memory manager set to allocate memory (or attempt to do so) without garbage collection. Even better would be if gc could run, but weak references were temporarily treated as strong so that garbage without finalizers would be collected but finalizers didn't get triggered. Using this facility would be inefficient, because it would allocate more memory than necessary and would also mess with the tuning of the garbage collector, but when communicating with other programs it seems it would be very useful to have some way of running an R code block and knowing that no other R code block would run during it (user interrupts are another issue, but they can be caught, and in any case I'm happy to fail when the user presses CTRL-C). -thomas On Fri, Feb 15, 2013 at 12:53 AM, wrote: > It might help if you could be more specific about what the issue is -- > if they are out of scope why does it matter whether the finalizers > run? > > Generically two approaches I can think of: > > you keep track of whenit is safe to fully run your finalizers and have > your finalizers put the objects on a linked list if it isn't safe to > run the finalizer now and clear the list each time you make a new one > > keep track of your objects with a weak list andturn them into strong > references before your calls, then drop the list after. > > I'm pretty sure we don't have a mechanism for temporarily suspending > running the finalizers but it is probably fairly easy to add if that > is the only option. > > I might be able to think of other options with more details on the > issue. > > Best, > > luke > > > On Tue, 12 Feb 2013, Thomas Lumley wrote: > > Is there some way to prevent finalizers running during a section of code? >> >> I have a package that includes R objects linked to database tables. To >> maintain the call-by-value semantics, tables are copied rather than >> modified, and the extra tables are removed by finalizers during garbage >> collection. >> >> However, if the garbage collection occurs in the middle of processing >> another SQL query (which is relatively likely, since that's where the >> memory allocations are) there are problems with the database interface. >> >> Since the guarantees for the finalizer are "at most once, not before the >> object is out of scope" it seems harmless to be able to prevent finalizers >> from running during a particular code block, but I can't see any way to do >> it. >> >> Suggestions? >> >>-thomas >> >> >> >> > -- > Luke Tierney > Chair, Statistics and Actuarial Science > Ralph E. Wareham Professor of Mathematical Sciences > University of Iowa Phone: 319-335-3386 > Department of Statistics andFax: 319-335-3017 >Actuarial Science > 241 Schaeffer Hall email: luke-tier...@uiowa.edu > Iowa City, IA 52242 WWW: http://www.stat.uiowa.edu > -- Thomas Lumley Professor of Biostatistics University of Auckland [[alternative HTML version deleted]] __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] stopping finalizers
I would argue that addressing this at a generic R level is the wrong place
(almost, see below) -- because this is about *specific* finalizers. You don't
mind running other people 's finalizers because they don't mess up your
connection. Therefore, I'd argue that primarily the approach should be in your
DB driver to synchronize the calls - which is what you did by queueing.
In a sense a more general approach won't be any different - you will need at
least a list of *specific* objects that should be deferred - it's either in
your driver or in R. I'd argue that the design is much easier in the driver
because you know which finalizers to register, whereas R has no concept of
finalizer "classes" to group them. You can also do this much more easily in the
driver since you know whether they need to be deferred and if they do, you can
easily process the deferred once when you get out of your critical section.
Because of this difference between specific finalizers and all GC any R-side
solution that doesn't register such finalizers in a special way will be
inherently wasteful - as you pointed out in the discussion below - and thus I'd
say it's more dangerous than helpful, because R can then lock itself out of
memory even if not necessary.
So IMHO the necessary practical way to solve this at R level (if someone wanted
to spend the time) would be to create "bags" of finalizers and use those when
defining critical regions, something like (pseudocode)
add_fin_bag("myDB", obj1)
// ...
add_fin_bag("myDB", obj2)
critical_fin_section_begin("myDB")
// ... here no finalizers for objects in the bag can be fired
critical_fin_section_end("myDB")
Technically, the simple solution would be to simply preserve the bag in the
critical region. However, this would not guarantee that the finalizers get
fired at the end of the section even if gc occurred. I suspect it would be
harder to guarantee that (other than running gc explicitly or performing
explicit de-allocation after the finalizer was detected to be scheduled but not
fired).
Cheers,
Simon
On Feb 14, 2013, at 3:12 PM, Thomas Lumley wrote:
> Luke,
>
> We're actually adopting the first of your generic approaches.
>
> As a more concrete description:
>
> There are R objects representing survey data sets, with the data stored in
> a database table. The subset() method, when applied to these objects,
> creates a new table indicating which rows of the data table are in the
> subset -- we don't modify the original table, because that breaks the
> call-by-value semantics. When the subset object in R goes out of scope, we
> need to delete the extra database table.
>
> I have been doing this with a finalizer on an environment that's part of
> the subset object in R. This all worked fine with JDBC, but the native
> database interface requires that all communications with the database come
> in send/receive pairs. Since R is single-threaded, this would normally not
> be any issue. However, since garbage collection can happen at any time, it
> is possible that the send part of the finalizer query "drop table
> _sbs_whatever" comes between the send and receive of some other query, and
> the database connection then falls over. So, I'm happy for the finalizer
> to run at any time except during a small critical section of R code.
>
> In this particular case the finalizer only issues "drop table" queries, and
> it doesn't need to know if they succeed, so we can keep a lock in the
> database connection and just store any "drop table" queries that arrive
> during a database operation for later execution. More generally, though,
> the fact that no R operation is atomic with respect to garbage collection
> seems to make it a bit difficult to use finalizers -- if you need a
> finalizer, it will often be in order to access and free some external
> resource, which is when the race conditions can matter.
>
> What I was envisaging was something like
>
> without_gc(expr)
>
> to evaluate expr with the memory manager set to allocate memory (or attempt
> to do so) without garbage collection. Even better would be if gc could
> run, but weak references were temporarily treated as strong so that garbage
> without finalizers would be collected but finalizers didn't get triggered.
> Using this facility would be inefficient, because it would allocate more
> memory than necessary and would also mess with the tuning of the garbage
> collector, but when communicating with other programs it seems it would be
> very useful to have some way of running an R code block and knowing that no
> other R code block would run during it (user interrupts are another issue,
> but they can be caught, and in any case I'm happy to fail when the user
> presses CTRL-C).
>
> -thomas
>
>
>
>
> On Fri, Feb 15, 2013 at 12:53 AM, wrote:
>
>> It might help if you could be more specific about what the issue is --
>> if they are out of scope why does it matter whether the finalizers
>> r
[Rd] Error: 'mcMap' is not an exported object from 'namespace:parallel'
> library(parallel) > mcMap(identity,1:5) Error: could not find function "mcMap" > parallel:::mcMap(identity,1:5) [[1]] [1] 1 [[2]] [1] 2 [[3]] [1] 3 [[4]] [1] 4 [[5]] [1] 5 > parallel::mcMap(identity,1:5) Error: 'mcMap' is not an exported object from 'namespace:parallel' > sessionInfo() R version 2.15.2 (2012-10-26) Platform: x86_64-unknown-linux-gnu (64-bit) locale: [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8LC_PAPER=C LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C attached base packages: [1] parallel stats graphics grDevices utils datasets methods base Nuff Said? Thanks for parallel! ~ malcolm_cook at stowers.org __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] stopping finalizers
> There are R objects representing survey data sets, with the data stored in > a database table. The subset() method, when applied to these objects, > creates a new table indicating which rows of the data table are in the > subset -- we don't modify the original table, because that breaks the > call-by-value semantics. When the subset object in R goes out of scope, we > need to delete the extra database table. Isn't subset slightly too early to do this? It would be slightly more efficient for subset to return an object that creates the table when you first attempt to modify it. Hadley -- Chief Scientist, RStudio http://had.co.nz/ __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] Suggestion: Custom filename patterns for non-Sweave vignettes
Hi,
as far as I understand it, the new R devel feature of processing
non-Sweave vignettes will (a) locate any "[.][RrSs](nw|tex)$" or
".Rmd" files, (b) check for a registered vignette engine, (c) process
the file using the registered "weave" function, (d) and possibly post
process the generated weave artifact (e.g. a *.tex file).
I'd like to propose to extend this non-Sweave mechanism to allow for
any filename patterns still using a very similar setup. Here is how
I'd like it to see it work with RSP vignettes (cf. the R.rsp package):
tools::vignetteEngine("rsp", weave=rspWeave, tangle=rspTangle,
patterns="[.]rsp$")
Argument 'patterns' could default to patterns=c("[.][RrSs](nw|tex)$",
"[.]Rmd$").
This is just a sketch/mock up and it may be that there are better
solutions. However, the idea is that when specify 'VignetteBuilder:
R.rsp' in DESCRIPTION of a package, R locates all engines registered
by the builder package. In this case it finds 'rsp'. (An alternative
to this lookup would be to use a DESCRIPTION field 'VignetteEngines:
R.rsp:rsp, knitr:knitr'.) It next looks for custom filename patterns
and use those to scan for vignette source files. With this approach,
the '%\VignetteEngine{knitr}' specifier would become optional. (I can
see how R now scans for Rnw and Rmd files, checks them for a
\VignetteEngine{} markup, and then looks up the corresponding engine).
Continuing, the above would make it possible to process RSP vignettes
that have filenames:
reportA.tex.rsp
reportB.html.rsp
reportC.md.rsp
reportD.Rnw.rsp
where rspWeave() will produce the following files:
reportA.tex
reportB.html
reportC.html
reportD.tex
I included the latter case just to illustrate a special case where
rspWeave() first generates a reportC.Rnw (Sweave or knitr) which is
the processed using the corresponding weaver to generate reportC.tex.
My point is that restricting vignette filenames to ".[RrSs](nw|tex)$"
or ".Rmd" is unnecessary and conceptually it would not be too hard to
extend it to handle any filename patterns.
I am aware that implementing this would require updates in several
place. If R core would approve on the above extended functionality, I
would be happy to dig into the source code and provide minimal and
backward compatible patches.
Finally, without knowing the details of all the other report
generating packages, my guess is that this extended feature would be
useful also for some of those packages, which in the long run
hopefully results in more packages having more vignettes (regardless
of the vignette format).
All the best,
Henrik
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
