On 05/04/2016 05:15 AM, Prof Brian Ripley wrote:
On 04/05/2016 08:44, Martin Maechler wrote:
Qin Zhu <qin...@outlook.com>
on Mon, 2 May 2016 16:19:44 -0400 writes:
> Hi,
> I’m working on a Shiny app for statistical analysis. I ran into
this "maximal number of DLLs reached" issue recently because my app
requires importing many other packages.
> I’ve posted my question on stackoverflow
(http://stackoverflow.com/questions/36974206/r-maximal-number-of-dlls-reached
<http://stackoverflow.com/questions/36974206/r-maximal-number-of-dlls-reached>).
> I’m just wondering is there any reason to set the maximal
number of DLLs to be 100, and is there any plan to increase it/not
hardcoding it in the future? It seems many people are also running
into this problem. I know I can work around this problem by modifying
the source, but since my package is going to be used by other people,
I don’t think this is a feasible solution.
> Any suggestions would be appreciated. Thanks!
> Qin
Increasing that number is of course "possible"... but it also
costs a bit (adding to the fixed memory footprint of R).
And not only that. At the time this was done (and it was once 50) the
main cost was searching DLLs for symbols. That is still an issue, and
few packages exclude their DLL from symbol search so if symbols have to
searched for a lot of DLLs will be searched. (Registering all the
symbols needed in a package avoids a search, and nowadays by default
searches from a namespace are restricted to that namespace.)
See
https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Registering-native-routines
for some further details about the search mechanism.
I did not set that limit, but I'm pretty sure it was also meant
as reminder for the useR to "clean up" a bit in her / his R
session, i.e., not load package namespaces unnecessarily. I
cannot yet imagine that you need > 100 packages | namespaces
loaded in your R session. OTOH, some packages nowadays have a
host of dependencies, so I agree that this at least may happen
accidentally more frequently than in the past.
I am not convinced that it is needed. The OP says he imports many
packages, and I doubt that more than a few are required at any one time.
Good practice is to load namespaces as required, using requireNamespace.
Extensive package dependencies in Bioconductor make it pretty easy to
end up with dozen of packages attached or loaded. For instance
library(GenomicFeatures)
library(DESeq2)
> length(loadedNamespaces())
[1] 63
> length(getLoadedDLLs())
[1] 41
Qin's use case is a shiny app, presumably trying to provide relatively
comprehensive access to a particular domain. Even if the app were to
load / requireNamespace() (this requires considerable programming
discipline to ensure that the namespace is available on all programming
paths where it is used), it doesn't seem at all improbable that the user
in an exploratory analysis would end up accessing dozens of packages
with orthogonal dependencies. This is also the use case with Karl
Forner's post
https://stat.ethz.ch/pipermail/r-devel/2015-May/071104.html (adding
library(crlmm) to the above gets us to 53 DLLs).
The real solution of course would be a code improvement that
starts with a relatively small number of "DLLinfo" structures
(say 32), and then allocates more batches (of size say 32) if
needed.
The problem of course is that such code will rarely be exercised, and
people have made errors on the boundaries (here multiples of 32) many
times in the past. (Note too that DLLs can be removed as well as added,
another point of coding errors.)
That argues for a simple increase in the maximum number of DLLs. This
would enable some people to have very bulky applications that pay a
performance cost (but the cost here is in small fractions of a
second...) in terms of symbol look-up (and collision?), but would have
no consequence for those of us with more sane use cases.
Martin Morgan
Patches to the R sources (development trunk in subversion at
https://svn.r-project.org/R/trunk/ ) are very welcome!
Martin Maechler
ETH Zurich & R Core Team
This email message may contain legally privileged and/or confidential
information. If you are not the intended recipient(s), or the employee or
agent responsible for the delivery of this message to the intended
recipient(s), you are hereby notified that any disclosure, copying,
distribution, or use of this email message is prohibited. If you have received
this message in error, please notify the sender immediately by e-mail and
delete this email message from your computer. Thank you.
______________________________________________
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
