Re: [Rd] Is it possible to increase MAX_NUM_DLLS in future R releases?

[Martin Maechler]

> Qin Zhu 
> on Mon, 2 May 2016 16:19:44 -0400 writes:

> Hi,
> I’m working on a Shiny app for statistical analysis. I ran into this 
"maximal number of DLLs reached" issue recently because my app requires 
importing many other packages.

> I’ve posted my question on stackoverflow 
(http://stackoverflow.com/questions/36974206/r-maximal-number-of-dlls-reached 
).
 

> I’m just wondering is there any reason to set the maximal number of DLLs 
to be 100, and is there any plan to increase it/not hardcoding it in the 
future? It seems many people are also running into this problem. I know I can 
work around this problem by modifying the source, but since my package is going 
to be used by other people, I don’t think this is a feasible solution. 

> Any suggestions would be appreciated. Thanks!
> Qin

Increasing that number is of course "possible"... but it also
costs a bit (adding to the fixed memory footprint of R). 

I did not set that limit, but I'm pretty sure it was also meant
as reminder for the useR to "clean up" a bit in her / his R
session, i.e., not load package namespaces unnecessarily. I
cannot yet imagine that you need > 100 packages | namespaces
loaded in your R session. OTOH, some packages nowadays have a
host of dependencies, so I agree that this at least may happen
accidentally more frequently than in the past. 

The real solution of course would be a code improvement that
starts with a relatively small number of "DLLinfo" structures
(say 32), and then allocates more batches (of size say 32) if
needed. 

Patches to the R sources (development trunk in subversion at
https://svn.r-project.org/R/trunk/ ) are very welcome! 

Martin Maechler
ETH Zurich  &  R Core Team

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Is it possible to increase MAX_NUM_DLLS in future R releases?

[Prof Brian Ripley]

On 04/05/2016 08:44, Martin Maechler wrote:

Qin Zhu 
 on Mon, 2 May 2016 16:19:44 -0400 writes:


 > Hi,
 > I’m working on a Shiny app for statistical analysis. I ran into this "maximal 
number of DLLs reached" issue recently because my app requires importing many other 
packages.

 > I’ve posted my question on stackoverflow 
(http://stackoverflow.com/questions/36974206/r-maximal-number-of-dlls-reached 
).

 > I’m just wondering is there any reason to set the maximal number of DLLs 
to be 100, and is there any plan to increase it/not hardcoding it in the future? 
It seems many people are also running into this problem. I know I can work around 
this problem by modifying the source, but since my package is going to be used by 
other people, I don’t think this is a feasible solution.

 > Any suggestions would be appreciated. Thanks!
 > Qin

Increasing that number is of course "possible"... but it also
costs a bit (adding to the fixed memory footprint of R).


And not only that.  At the time this was done (and it was once 50) the 
main cost was searching DLLs for symbols.  That is still an issue, and 
few packages exclude their DLL from symbol search so if symbols have to 
searched for a lot of DLLs will be searched.  (Registering all the 
symbols needed in a package avoids a search, and nowadays by default 
searches from a namespace are restricted to that namespace.)


See 
https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Registering-native-routines 
for some further details about the search mechanism.



I did not set that limit, but I'm pretty sure it was also meant
as reminder for the useR to "clean up" a bit in her / his R
session, i.e., not load package namespaces unnecessarily. I
cannot yet imagine that you need > 100 packages | namespaces
loaded in your R session. OTOH, some packages nowadays have a
host of dependencies, so I agree that this at least may happen
accidentally more frequently than in the past.


I am not convinced that it is needed.  The OP says he imports many 
packages, and I doubt that more than a few are required at any one time. 
 Good practice is to load namespaces as required, using requireNamespace.



The real solution of course would be a code improvement that
starts with a relatively small number of "DLLinfo" structures
(say 32), and then allocates more batches (of size say 32) if
needed.


The problem of course is that such code will rarely be exercised, and 
people have made errors on the boundaries (here multiples of 32) many 
times in the past.  (Note too that DLLs can be removed as well as added, 
another point of coding errors.)



Patches to the R sources (development trunk in subversion at
https://svn.r-project.org/R/trunk/ ) are very welcome!

Martin Maechler
ETH Zurich  &  R Core Team




--
Brian D. Ripley,  rip...@stats.ox.ac.uk
Emeritus Professor of Applied Statistics, University of Oxford

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] nlme: lme gives incorrect answers without warning with redundant data

[Jonathan Dushoff]

I think this issue will be of interest to the list. It seems like a
problem with nlme, and I was drawn into it because of a collaborator
who encountered this issue in a simple experiment and wound up
questioning the mixed model approach to his question.

The issue is that lme, when faced with certain numerically unstable
problems, seems to produce incorrect answers (understandable), but
without warnings (bad, in my opinion).

I have attached a reproducible example.

There is more information here:
https://htmlpreview.github.io/?https://github.com/dushoff/scratch/blob/master/nlme.html
and https://bugs.r-project.org/bugzilla3/show_bug.cgi?id=16864. The
first link shows my session information.

Jonathan
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Is it possible to increase MAX_NUM_DLLS in future R releases?

[Martin Morgan]

On 05/04/2016 05:15 AM, Prof Brian Ripley wrote:

On 04/05/2016 08:44, Martin Maechler wrote:

Qin Zhu 
 on Mon, 2 May 2016 16:19:44 -0400 writes:


 > Hi,
 > I’m working on a Shiny app for statistical analysis. I ran into
this "maximal number of DLLs reached" issue recently because my app
requires importing many other packages.

 > I’ve posted my question on stackoverflow
(http://stackoverflow.com/questions/36974206/r-maximal-number-of-dlls-reached
).


 > I’m just wondering is there any reason to set the maximal
number of DLLs to be 100, and is there any plan to increase it/not
hardcoding it in the future? It seems many people are also running
into this problem. I know I can work around this problem by modifying
the source, but since my package is going to be used by other people,
I don’t think this is a feasible solution.

 > Any suggestions would be appreciated. Thanks!
 > Qin

Increasing that number is of course "possible"... but it also
costs a bit (adding to the fixed memory footprint of R).


And not only that.  At the time this was done (and it was once 50) the
main cost was searching DLLs for symbols.  That is still an issue, and
few packages exclude their DLL from symbol search so if symbols have to
searched for a lot of DLLs will be searched.  (Registering all the
symbols needed in a package avoids a search, and nowadays by default
searches from a namespace are restricted to that namespace.)

See
https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Registering-native-routines
for some further details about the search mechanism.


I did not set that limit, but I'm pretty sure it was also meant
as reminder for the useR to "clean up" a bit in her / his R
session, i.e., not load package namespaces unnecessarily. I
cannot yet imagine that you need > 100 packages | namespaces
loaded in your R session. OTOH, some packages nowadays have a
host of dependencies, so I agree that this at least may happen
accidentally more frequently than in the past.


I am not convinced that it is needed.  The OP says he imports many
packages, and I doubt that more than a few are required at any one time.
  Good practice is to load namespaces as required, using requireNamespace.


Extensive package dependencies in Bioconductor make it pretty easy to 
end up with dozen of packages attached or loaded. For instance


  library(GenomicFeatures)
  library(DESeq2)

> length(loadedNamespaces())
[1] 63
> length(getLoadedDLLs())
[1] 41

Qin's use case is a shiny app, presumably trying to provide relatively 
comprehensive access to a particular domain. Even if the app were to 
load / requireNamespace() (this requires considerable programming 
discipline to ensure that the namespace is available on all programming 
paths where it is used), it doesn't seem at all improbable that the user 
in an exploratory analysis would end up accessing dozens of packages 
with orthogonal dependencies. This is also the use case with Karl 
Forner's post 
https://stat.ethz.ch/pipermail/r-devel/2015-May/071104.html (adding 
library(crlmm) to the above gets us to 53 DLLs).





The real solution of course would be a code improvement that
starts with a relatively small number of "DLLinfo" structures
(say 32), and then allocates more batches (of size say 32) if
needed.


The problem of course is that such code will rarely be exercised, and
people have made errors on the boundaries (here multiples of 32) many
times in the past.  (Note too that DLLs can be removed as well as added,
another point of coding errors.)


That argues for a simple increase in the maximum number of DLLs. This 
would enable some people to have very bulky applications that pay a 
performance cost (but the cost here is in small fractions of a 
second...) in terms of symbol look-up (and collision?), but would have 
no consequence for those of us with more sane use cases.


Martin Morgan




Patches to the R sources (development trunk in subversion at
https://svn.r-project.org/R/trunk/ ) are very welcome!

Martin Maechler
ETH Zurich  &  R Core Team







This email message may contain legally privileged and/or confidential 
information.  If you are not the intended recipient(s), or the employee or 
agent responsible for the delivery of this message to the intended 
recipient(s), you are hereby notified that any disclosure, copying, 
distribution, or use of this email message is prohibited.  If you have received 
this message in error, please notify the sender immediately by e-mail and 
delete this email message from your computer. Thank you.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] unix readline signal handling patches

[frederik]

Hello R Developers,

I posted some patches yesterday to your bug tracking system but I'm
not sure if I should have notified the mailing list as well. I haven't
gotten any responses yet.

https://bugs.r-project.org/bugzilla/show_bug.cgi?id=16603

https://bugs.r-project.org/bugzilla/show_bug.cgi?id=16604

Thank you,

Frederick

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] Is it possible to retrieve the last error? (not error *message*)

[Henrik Bengtsson]

Hi,

at the R prompt, is it possible to retrieve the last error (as in
condition object of class "error")?

I'm not asking for geterrmessage(), which only returns the error
message (as a character string).  I'm basically looking for a
.Last.error or .Last.condition, analogously to .Last.value for values,
which can be used when it is "too late" (not possible) to go back an
use try()/tryCatch().

Thanks,

Henrik

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Is it possible to retrieve the last error? (not error *message*)

[David Winsemius]

> On May 4, 2016, at 12:41 PM, Henrik Bengtsson  
> wrote:
> 
> Hi,
> 
> at the R prompt, is it possible to retrieve the last error (as in
> condition object of class "error")?
> 
> I'm not asking for geterrmessage(), which only returns the error
> message (as a character string).  I'm basically looking for a
> .Last.error or .Last.condition, analogously to .Last.value for values,
> which can be used when it is "too late" (not possible) to go back an
> use try()/tryCatch().

After looking at the code for the exposed `traceback`
>  I'm wondering if this delivers what you expect:

.traceback()[1]


> Thanks,
> 
> Henrik
> 
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

David Winsemius
Alameda, CA, USA

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Is it possible to retrieve the last error? (not error *message*)

[Henrik Bengtsson]

On Wed, May 4, 2016 at 1:27 PM, David Winsemius  wrote:
>
>> On May 4, 2016, at 12:41 PM, Henrik Bengtsson  
>> wrote:
>>
>> Hi,
>>
>> at the R prompt, is it possible to retrieve the last error (as in
>> condition object of class "error")?
>>
>> I'm not asking for geterrmessage(), which only returns the error
>> message (as a character string).  I'm basically looking for a
>> .Last.error or .Last.condition, analogously to .Last.value for values,
>> which can be used when it is "too late" (not possible) to go back an
>> use try()/tryCatch().
>
> After looking at the code for the exposed `traceback`
>>  I'm wondering if this delivers what you expect:
>
> .traceback()[1]

Thanks, but unfortunately not:

> stop("Hello")
Error: Hello
> ex <- .traceback()[1]
> str(ex)
List of 1
 $ : chr "stop(\"Hello\")"
> inherits(ex, "condition")
[1] FALSE


I'm looking for something that returns the object of class condition, cf.

> ex <- attr(try(stop("Hello")), "condition")
Error in try(stop("Hello")) : Hello
> str(ex)
List of 2
 $ message: chr "Hello"
 $ call   : language doTryCatch(return(expr), name, parentenv, handler)
 - attr(*, "class")= chr [1:3] "simpleError" "error" "condition"
> inherits(ex, "condition")
[1] TRUE


The reason is that this object may contain additional information that
is not available in the message nor the call.  For instance, a package
may define a richer error class that captures/carries more information
about an error (e.g. a time stamp, a remote session information, ...)
and which is therefore not available via neither geterrmessage() nor
traceback().

I am aware that I might be asking for something that is not supported
and requires that the default signal handlers be modified.

/Henrik

>
>
>> Thanks,
>>
>> Henrik
>>
>> __
>> R-devel@r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel
>
> David Winsemius
> Alameda, CA, USA
>

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] R process killed when allocating too large matrix (Mac OS X)

[Simon Urbanek]

On May 3, 2016, at 9:51 PM, Marius Hofert  wrote:

> Dear expeRts,
> 
> The following code leads to R being killed (under Mac OS X 10.11.4; R
> installed from source; also happened under a previous unstable
> version):
> 
> m <- matrix(0, 10, 10)
> 
> I expected an error that a vector of this size could not be allocated.
> 
> Besides the above (a bug?), how can I find out beforehand whether or
> not a square matrix can be allocated?
> 

Can you elaborate on "leads to R being killed"? You should tell to the killer 
not to do it again :).

Cheers,
Simon



> Cheers,
> Marius
> 
> 
> PS: Here is the sessionInfo() output:
> 
> R version 3.3.0 (2016-05-03)
> Platform: x86_64-apple-darwin15.4.0 (64-bit)
> Running under: OS X 10.11.4 (El Capitan)
> 
> locale:
> [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
> 
> attached base packages:
> [1] stats graphics  grDevices utils datasets  methods   base
> 
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
> 

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] R process killed when allocating too large matrix (Mac OS X)

[Marius Hofert]

> Can you elaborate on "leads to R being killed"? You should tell to the killer 
> not to do it again :).

Hi Simon!

Sure, but who do you tell it if you don't know the killer?
This is all the killer left me with, the 'crime scene' if you like :-)

> m <- matrix(0, 9, 10)
Killed: 9

My colleague Wayne Oldford also tried it on his Mac machine and
apparently the killer went further down the hallway to his office
now... so scary. Here is Wayne's sessionInfo():

> sessionInfo()
R version 3.2.4 (2016-03-10)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X 10.10.5 (Yosemite)

locale:
[1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8

attached base packages:
[1] stats graphics  grDevices utils datasets  methods   base

loaded via a namespace (and not attached):
[1] rsconnect_0.4.1.11 tools_3.2.4
>

Cheers,
M

>
> Cheers,
> Simon

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] R process killed when allocating too large matrix (Mac OS X)

[Simon Urbanek]

On May 4, 2016, at 6:14 PM, Marius Hofert  wrote:

>> Can you elaborate on "leads to R being killed"? You should tell to the 
>> killer not to do it again :).
> 
> Hi Simon!
> 
> Sure, but who do you tell it if you don't know the killer?
> This is all the killer left me with, the 'crime scene' if you like :-)
> 
>> m <- matrix(0, 9, 10)
> Killed: 9
> 
> My colleague Wayne Oldford also tried it on his Mac machine and
> apparently the killer went further down the hallway to his office
> now... so scary. Here is Wayne's sessionInfo():
> 

Yes, indeed, scary - since it means someone is killing R which means there is 
not much R itself can do about it. In fact from the syslog I see

May  4 20:48:11 ginaz kernel[0]: low swap: killing pid 56256 (R)

so it's the kernel's own defense mechanism. The bad thing is that R cannot do 
anything about it - the kernel just decides to snipe processes it thinks are 
dangerous to the health of the system, and it does so without a warning.

Cheers,
Simon


>> sessionInfo()
> R version 3.2.4 (2016-03-10)
> Platform: x86_64-apple-darwin13.4.0 (64-bit)
> Running under: OS X 10.10.5 (Yosemite)
> 
> locale:
> [1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8
> 
> attached base packages:
> [1] stats graphics  grDevices utils datasets  methods   base
> 
> loaded via a namespace (and not attached):
> [1] rsconnect_0.4.1.11 tools_3.2.4
>> 
> 
> Cheers,
> M
> 
>> 
>> Cheers,
>> Simon
> 

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] R process killed when allocating too large matrix (Mac OS X)

[Marius Hofert]

Hi Simon,

thanks for your quick reply.

1) ... so you can reproduce this?
2) Do you know a way how this can be 'foreseen'? We allocate larger
matrices in the copula package depending on the user's input
dimension. It would be good to tell her/him "Your dimension is quite
large. Be aware of killers in your neighborhood"... before the killer
attacks.

Thanks & cheers,
Marius


On Wed, May 4, 2016 at 8:54 PM, Simon Urbanek
 wrote:
>
> On May 4, 2016, at 6:14 PM, Marius Hofert  wrote:
>
>>> Can you elaborate on "leads to R being killed"? You should tell to the 
>>> killer not to do it again :).
>>
>> Hi Simon!
>>
>> Sure, but who do you tell it if you don't know the killer?
>> This is all the killer left me with, the 'crime scene' if you like :-)
>>
>>> m <- matrix(0, 9, 10)
>> Killed: 9
>>
>> My colleague Wayne Oldford also tried it on his Mac machine and
>> apparently the killer went further down the hallway to his office
>> now... so scary. Here is Wayne's sessionInfo():
>>
>
> Yes, indeed, scary - since it means someone is killing R which means there is 
> not much R itself can do about it. In fact from the syslog I see
>
> May  4 20:48:11 ginaz kernel[0]: low swap: killing pid 56256 (R)
>
> so it's the kernel's own defense mechanism. The bad thing is that R cannot do 
> anything about it - the kernel just decides to snipe processes it thinks are 
> dangerous to the health of the system, and it does so without a warning.
>
> Cheers,
> Simon
>

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] R process killed when allocating too large matrix (Mac OS X)

[Simon Urbanek]

On May 4, 2016, at 9:00 PM, Marius Hofert  wrote:

> Hi Simon,
> 
> thanks for your quick reply.
> 
> 1) ... so you can reproduce this?

Yes, I can on 10.11.4.


> 2) Do you know a way how this can be 'foreseen'? We allocate larger
> matrices in the copula package depending on the user's input
> dimension. It would be good to tell her/him "Your dimension is quite
> large. Be aware of killers in your neighborhood"... before the killer
> attacks.
> 

Not directly, because the system is happy to accommodate R's request for more 
memory but then it will strike out of the blue. Since that decision is made by 
the kernel and it doesn't expose its thinking about how it feels, it's hard to 
tell. I couldn't find any API on OS X akin to didReceiveMemoryWarning in iOS, 
unfortunately.

You could guess - e.g. by checking the total system memory

int mib [] = { CTL_HW, HW_MEMSIZE };
int64_t value = 0;
size_t length = sizeof(value);
sysctl(mib, 2, &value, &length, NULL, 0);

and comparing it to the sizes involved. However, even that is not foolproof, 
because it all depends on the other processes' memory usage, swap space size 
etc. There are slightly more involved ways to query the VM system as well, but 
I'm not sure I'd want to go so deep into the weeds, especially since this 
becomes quickly highly OS-specific.

Cheers,
Simon



> Thanks & cheers,
> Marius
> 
> 
> On Wed, May 4, 2016 at 8:54 PM, Simon Urbanek
>  wrote:
>> 
>> On May 4, 2016, at 6:14 PM, Marius Hofert  wrote:
>> 
 Can you elaborate on "leads to R being killed"? You should tell to the 
 killer not to do it again :).
>>> 
>>> Hi Simon!
>>> 
>>> Sure, but who do you tell it if you don't know the killer?
>>> This is all the killer left me with, the 'crime scene' if you like :-)
>>> 
 m <- matrix(0, 9, 10)
>>> Killed: 9
>>> 
>>> My colleague Wayne Oldford also tried it on his Mac machine and
>>> apparently the killer went further down the hallway to his office
>>> now... so scary. Here is Wayne's sessionInfo():
>>> 
>> 
>> Yes, indeed, scary - since it means someone is killing R which means there 
>> is not much R itself can do about it. In fact from the syslog I see
>> 
>> May  4 20:48:11 ginaz kernel[0]: low swap: killing pid 56256 (R)
>> 
>> so it's the kernel's own defense mechanism. The bad thing is that R cannot 
>> do anything about it - the kernel just decides to snipe processes it thinks 
>> are dangerous to the health of the system, and it does so without a warning.
>> 
>> Cheers,
>> Simon
>> 
> 

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] R process killed when allocating too large matrix (Mac OS X)

[Hervé Pagès]

Hi,

Interesting "feature" in 10.11.4. I wonder if the process is killed
before or after malloc() returns. If before, it seems very blunt:
"You're asking too much and I don't like it so I kill you now".
If after it doesn't look much better: "You're asking a lot and I
don't like it but I give it to you anyway. I'll kill you quickly
later".

Why wouldn't the kernel just refuse to give that memory instead
(i.e. malloc() returns NULL).

Just curious...

H.

On 05/04/2016 06:21 PM, Simon Urbanek wrote:


On May 4, 2016, at 9:00 PM, Marius Hofert  wrote:


Hi Simon,

thanks for your quick reply.

1) ... so you can reproduce this?


Yes, I can on 10.11.4.



2) Do you know a way how this can be 'foreseen'? We allocate larger
matrices in the copula package depending on the user's input
dimension. It would be good to tell her/him "Your dimension is quite
large. Be aware of killers in your neighborhood"... before the killer
attacks.



Not directly, because the system is happy to accommodate R's request for more 
memory but then it will strike out of the blue. Since that decision is made by 
the kernel and it doesn't expose its thinking about how it feels, it's hard to 
tell. I couldn't find any API on OS X akin to didReceiveMemoryWarning in iOS, 
unfortunately.

You could guess - e.g. by checking the total system memory

int mib [] = { CTL_HW, HW_MEMSIZE };
int64_t value = 0;
size_t length = sizeof(value);
sysctl(mib, 2, &value, &length, NULL, 0);

and comparing it to the sizes involved. However, even that is not foolproof, 
because it all depends on the other processes' memory usage, swap space size 
etc. There are slightly more involved ways to query the VM system as well, but 
I'm not sure I'd want to go so deep into the weeds, especially since this 
becomes quickly highly OS-specific.

Cheers,
Simon




Thanks & cheers,
Marius


On Wed, May 4, 2016 at 8:54 PM, Simon Urbanek
 wrote:


On May 4, 2016, at 6:14 PM, Marius Hofert  wrote:


Can you elaborate on "leads to R being killed"? You should tell to the killer 
not to do it again :).


Hi Simon!

Sure, but who do you tell it if you don't know the killer?
This is all the killer left me with, the 'crime scene' if you like :-)


m <- matrix(0, 9, 10)

Killed: 9

My colleague Wayne Oldford also tried it on his Mac machine and
apparently the killer went further down the hallway to his office
now... so scary. Here is Wayne's sessionInfo():



Yes, indeed, scary - since it means someone is killing R which means there is 
not much R itself can do about it. In fact from the syslog I see

May  4 20:48:11 ginaz kernel[0]: low swap: killing pid 56256 (R)

so it's the kernel's own defense mechanism. The bad thing is that R cannot do 
anything about it - the kernel just decides to snipe processes it thinks are 
dangerous to the health of the system, and it does so without a warning.

Cheers,
Simon





__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel



--
Hervé Pagès

Program in Computational Biology
Division of Public Health Sciences
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N, M1-B514
P.O. Box 19024
Seattle, WA 98109-1024

E-mail: hpa...@fredhutch.org
Phone:  (206) 667-5791
Fax:(206) 667-1319

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] R process killed when allocating too large matrix (Mac OS X)

[Simon Urbanek]

On May 4, 2016, at 9:49 PM, Hervé Pagès  wrote:

> Hi,
> 
> Interesting "feature" in 10.11.4. I wonder if the process is killed
> before or after malloc() returns. If before, it seems very blunt:
> "You're asking too much and I don't like it so I kill you now".
> If after it doesn't look much better: "You're asking a lot and I
> don't like it but I give it to you anyway. I'll kill you quickly
> later".
> 
> Why wouldn't the kernel just refuse to give that memory instead
> (i.e. malloc() returns NULL).
> 
> Just curious...
> 

You can google it, but this seems to be standard practice in modern OSes 
(including OS X and Linux) where the kernel is not actually allocating the 
physical memory when the application asks for it so malloc() will succeed and 
give only a pointer to a virtual space where it will be mapping physical memory 
into as needed (aka opportunistic allocation). However, when the OS figures out 
later that it is overcommitted, it has no good way to signal the app that it 
can't use what it was promised, so it will use SIGKILL. The rationale is that 
in most cases the kernel can scramble to fulfill the request so it gives it 
more flexibility in managing the memory (it can spread pages to RAM, swap, 
shared ,...), but when it cannot recover it has no choice but to kill the 
process.

I don't have the debugger open anymore, but if I recall correctly in my case 
the kill happened in copyVector() -> duplicate(), so it was not necessarily on 
the initial allocation but rather further down the line.

Cheers,
Simon



> H.
> 
> On 05/04/2016 06:21 PM, Simon Urbanek wrote:
>> 
>> On May 4, 2016, at 9:00 PM, Marius Hofert  wrote:
>> 
>>> Hi Simon,
>>> 
>>> thanks for your quick reply.
>>> 
>>> 1) ... so you can reproduce this?
>> 
>> Yes, I can on 10.11.4.
>> 
>> 
>>> 2) Do you know a way how this can be 'foreseen'? We allocate larger
>>> matrices in the copula package depending on the user's input
>>> dimension. It would be good to tell her/him "Your dimension is quite
>>> large. Be aware of killers in your neighborhood"... before the killer
>>> attacks.
>>> 
>> 
>> Not directly, because the system is happy to accommodate R's request for 
>> more memory but then it will strike out of the blue. Since that decision is 
>> made by the kernel and it doesn't expose its thinking about how it feels, 
>> it's hard to tell. I couldn't find any API on OS X akin to 
>> didReceiveMemoryWarning in iOS, unfortunately.
>> 
>> You could guess - e.g. by checking the total system memory
>> 
>> int mib [] = { CTL_HW, HW_MEMSIZE };
>> int64_t value = 0;
>> size_t length = sizeof(value);
>> sysctl(mib, 2, &value, &length, NULL, 0);
>> 
>> and comparing it to the sizes involved. However, even that is not foolproof, 
>> because it all depends on the other processes' memory usage, swap space size 
>> etc. There are slightly more involved ways to query the VM system as well, 
>> but I'm not sure I'd want to go so deep into the weeds, especially since 
>> this becomes quickly highly OS-specific.
>> 
>> Cheers,
>> Simon
>> 
>> 
>> 
>>> Thanks & cheers,
>>> Marius
>>> 
>>> 
>>> On Wed, May 4, 2016 at 8:54 PM, Simon Urbanek
>>>  wrote:
 
 On May 4, 2016, at 6:14 PM, Marius Hofert  
 wrote:
 
>> Can you elaborate on "leads to R being killed"? You should tell to the 
>> killer not to do it again :).
> 
> Hi Simon!
> 
> Sure, but who do you tell it if you don't know the killer?
> This is all the killer left me with, the 'crime scene' if you like :-)
> 
>> m <- matrix(0, 9, 10)
> Killed: 9
> 
> My colleague Wayne Oldford also tried it on his Mac machine and
> apparently the killer went further down the hallway to his office
> now... so scary. Here is Wayne's sessionInfo():
> 
 
 Yes, indeed, scary - since it means someone is killing R which means there 
 is not much R itself can do about it. In fact from the syslog I see
 
 May  4 20:48:11 ginaz kernel[0]: low swap: killing pid 56256 (R)
 
 so it's the kernel's own defense mechanism. The bad thing is that R cannot 
 do anything about it - the kernel just decides to snipe processes it 
 thinks are dangerous to the health of the system, and it does so without a 
 warning.
 
 Cheers,
 Simon
 
>>> 
>> 
>> __
>> R-devel@r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-devel
>> 
> 
> -- 
> Hervé Pagès
> 
> Program in Computational Biology
> Division of Public Health Sciences
> Fred Hutchinson Cancer Research Center
> 1100 Fairview Ave. N, M1-B514
> P.O. Box 19024
> Seattle, WA 98109-1024
> 
> E-mail: hpa...@fredhutch.org
> Phone:  (206) 667-5791
> Fax:(206) 667-1319
> 

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] R process killed when allocating too large matrix (Mac OS X)

[Marius Hofert]

Hi Simon,

... all interesting (but quite a bit above my head). I only read
'Linux' and want to throw in that this problem does not appear on
Linux (it seems). I talked about this with Martin Maechler and he
reported that the same example (on one of his machines; with NA_real_
instead of '0's in the matrix) gave:

  Error: cannot allocate vector of size 70.8 Gb
Timing stopped at: 144.79 41.619 202.019

... but no killer around...

Cheers,
Marius

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] Too many spaces in deparsed complex numbers with digits17 control option

[Richard Cotton]

If you set the "digits17" control option in deparse, you get a lot of
unnecessary space in the representation of complex numbers.

> deparse(0 + 0i)
[1] "0+0i"
> deparse(0 + 0i, control = "digits17")
[1] "0 + 0i"

As far as I can tell, the logic for this comes from this piece of
/src/main/deparse.c:

if (TYPEOF(vector) == CPLXSXP && (d->opts & DIGITS16)) {
  Rcomplex z =  COMPLEX(vector)[i];
  if (R_FINITE(z.r) && R_FINITE(z.i)) {
  snprintf(hex, 64, "%.17g + %17gi", z.r, z.i);
  strp = hex;
  } else
  strp = EncodeElement(vector, i, quote, '.');
}

I think this is a small bug, and that "%17gi" in the snprintf call
ought to be "%.17gi".

Also there shouldn't be any space around the plus sign for consistency
with the non-digits17 option.

Is this a real bug, or is it deliberate behaviour?

-- 
Regards,
Richie

Learning R
4dpiecharts.com

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel