> Does this happen in R-patched? I've seen similar errors in 2.10.0, but > not in a current build.
Yes, still there in R-patched. (Still haven't got to your code, this was in > mine. I'm reluctant to spend time on code that is messing with > internals, because you might be using things in a way not intended by > the author. Now, if you can show me some code that demonstrates the > problem without using internals directly, I'll follow up.) I did try, but it's not completely possible, because 'makeLazyLoadDB' is internal and there is no public alternative (a pity-- it's useful). However, the problem(s) can be demonstrated without directly calling 'parse_Rd', and with 'lazyLoad' (public) instead of 'fetchRdDB' (private), as per "pointer 1" below. If you have a look at 'tools:::.install_package_Rd_objects', you'll see that my use of 'makeLazyLoadDB' is quite standard. The problem is not easy to reproduce. It took 4-5 hours work to get the 3-line reproducible example that I posted, plus another couple since, so I'm also reluctant to spend more time... The examples in my previous post still apply-- the first one involves just 3 statements-- but here are some more pointers I've unearthed since: 1. Sometimes 'fetchRdDB' or 'lazyLoad' called directly from the prompt doesn't work, but public 'Rd_db' (which directly calls 'fetchRdDB') does. I've experimented with copying the installed 'tools' package into a new library "d:/temp/fakelib", then stuff like this: test> e <- new.env() test> lazyLoad( 'd:/temp/fakelib/tools/help/tools', e) # original files tools.rdx, tools.rdb test> e <- as.list( e) # force evaluation test> tools:::makeLazyLoadDB( e, 'd:/temp/fakelib/tools/help/tools') # modify tools.rd* test> e1 <- new.env() test> lazyLoad( 'd:/temp/fakelib/tools/help/tools', e1) test> as.list( e1) # try to force evaluation... Error in as.list.environment(e1) : cannot allocate memory block of size 2.7 Gb test> test> Rd_db( 'tools', 'd:/temp/fakelib') # no probs !? 2. Sometimes 'fetchRdDB' or 'lazyLoad' will fail in one R session, but will work in a fresh session on exactly the same files. For example, after restarting R, the previous commands involving 'e1' work fine. Mark > > Duncan Murdoch > > >>> I'm encountering problems when making lazy-loadable databases of the output >>> from 'parse_Rd'. The lazy- >>> load database is of seemingly limitless size when I try to reload it... >>> Admittedly I am using functions >>> that I'm not really supposed to use, which is why this isn't a bug report, >>> but there does seem to be >>> something strange going on; my code is very similar to code that lives >>> inside >>> 'tools:::.install_package_Rd_objects'. The problems occur with >>> just-released R2.10.0 on Windows. >> object.size() has problems when working on Rd objects, because it counts >> every environment separately, even though they may all be references to >> the same one. I haven't looked at your code, but that could be a problem. >> >> > > Mark > > ________________________________________ > From: Duncan Murdoch [murd...@stats.uwo.ca] > Sent: 31 October 2009 08:59 > To: Bravington, Mark (CMIS, Hobart) > Cc: r-devel@r-project.org > Subject: Re: [Rd] parse_Rd and/or lazyload problem > > On 30/10/2009 12:50 AM, mark.braving...@csiro.au wrote: >> I'm encountering problems when making lazy-loadable databases of the output >> from 'parse_Rd'. The lazy-load database is of seemingly limitless size when >> I try to reload it... Admittedly I am using functions that I'm not really >> supposed to use, which is why this isn't a bug report, but there does seem >> to be something strange going on; my code is very similar to code that lives >> inside 'tools:::.install_package_Rd_objects'. The problems occur with >> just-released R2.10.0 on Windows. > > object.size() has problems when working on Rd objects, because it counts > every environment separately, even though they may all be references to > the same one. I haven't looked at your code, but that could be a problem. > > Duncan Murdoch > >> The examples below use files which can be found at >> ftp://ftp.csiro.au/MarkBravington, but you'll obviously need to modify the >> paths. The file "scrunge.Rd" is just "Rdiff.Rd" from the 'tools' package. >> The file "fakepack.7z" should unzip to create a fake package with a >> DESCRIPTION file and a "man" directory that contains two Rd files. >> >> Transcript of first example: >> eglist <- list( scrunge=tools:::prepare_Rd( 'd:/temp/scrunge.Rd', >> defines=.Platform$OS.type, stages='install', warningCalls=FALSE)) >> tools:::makeLazyLoadDB( eglist, 'd:/temp/ll', compress=FALSE) >> tools:::fetchRdDB( 'd:/temp/ll') >> # Error: cannot allocate vector of size 1.4 Gb >> >> The second example triggers an error with >> 'tools:::.install_package_Rd_objects' itself. It doesn't happen the first >> time, but frequently does after the Rd files have been changed. Transcript: >> >> # Make sure d:/temp/help/ is empty, then >> test> tools:::.install_package_Rd_objects( 'd:/temp/fakepack', 'd:/temp') >> test> tools:::fetchRdDB( 'd:/temp/help/temp') >> # All good. Next, I *removed* one of the two Rd files in >> "d:/temp/fakepack/man", ... >> # ...deleted "d:/temp/help/temp*", and tried again >> test> tools:::.install_package_Rd_objects( 'd:/temp/fakepack', 'd:/temp') >> test> tools:::fetchRdDB( 'd:/temp/help/temp') >> Warning: Reached total allocation of 1535Mb: see help(memory.size) >> Warning: Reached total allocation of 1535Mb: see help(memory.size) >> Warning: Reached total allocation of 1535Mb: see help(memory.size) >> Warning: Reached total allocation of 1535Mb: see help(memory.size) >> Error: cannot allocate vector of size 1.9 Gb >> # Or on other occasions I get >> Error: internal error -3 in R_decompress1 >> >> Mark Bravington >> CSIRO CMIS >> Hobart >> Australia >> >> --please do not edit the information below-- >> Version: >> platform = i386-pc-mingw32 >> arch = i386 >> os = mingw32 >> system = i386, mingw32 >> status = >> major = 2 >> minor = 10.0 >> year = 2009 >> month = 10 >> day = 26 >> svn rev = 50208 >> language = R >> version.string = R version 2.10.0 (2009-10-26) >> Windows XP (build 2600) Service Pack 2 >> Locale: >> LC_COLLATE=English_Australia.1252;LC_CTYPE=English_Australia.1252;LC_MONETARY=English_Australia.1252;LC_NUMERIC=C;LC_TIME=English_Australia.1252 >> Search Path: >> .GlobalEnv, ROOT, package:grDevices, package:ad, package:chstuff, >> package:handy2, package:tweedie, package:statmod, package:handy, >> package:debug, package:mvbutils, mvb.session.info, package:tools, >> package:tcltk, package:stats, package:graphics, package:utils, >> package:methods, Autoloads, package:base >> ______________________________________________ >> R-devel@r-project.org mailing list >> https://stat.ethz.ch/mailman/listinfo/r-devel ______________________________________________ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
