Re: [Rd] Use of C++ in Packages

[Hugh Marera]

Some of us are learning about development in R and use R in our work data
analysis pipelines. What is the best way to identify packages that
currently have these C++ problems? I would like to be able to help fix the
bugs but more importantly not use these packages in critical work
pipelines. Any C++ R package bug squashing events out there?

Regards

Hugh

On Mon, Apr 1, 2019 at 6:23 PM Tomas Kalibera 
wrote:

> On 3/30/19 8:59 AM, Romain Francois wrote:
> > tl;dr: we need better C++ tools and documentation.
> >
> > We collectively know more now with the rise of tools like rchk and
> improved documentation such as Tomas’s post. That’s a start, but it appears
> that there still is a lot of knowledge that would deserve to be promoted to
> actual documentation of best practices.
> Well there is quite a bit of knowledge in Writing R Extensions and many
> problems could have been prevented had it been read more thoroughly by
> package developers. The problem that C++ runs some functions
> automatically (like destructors), should not be too hard to identify
> based on what WRE says about the need for protection against garbage
> collection.
>
>  From my experience, one can learn most about R internals from debugging
> and reading source code - when debugging PROTECT errors and other memory
> errors/memory corruption, common problems caused by bugs in native C/C++
> code - one needs to read and understand source code involved at all
> layers, one needs to understand the documentation covering code at
> different layers, and one has to think about these things, forming
> hypotheses, narrowing down to smaller examples, etc.
>
> My suggestion for package authors who write native code and want to
> learn more, and who want to be responsible (these kinds of bugs affect
> other packaged indirectly and can be woken up by inconsequential and
> correct code changes, even in R runtime): test and debug your code hard
> - look at UBSAN/ASAN/valgrind/rchk checks from CRAN and run these tools
> yourself if needed. Run with strict barrier checking and with gctorture.
> Write more tests to increase the coverage. Specifically now if you use
> C++ code, try to read all of your related code and check you do not have
> the problems I mentioned in my blog. Think of other related problems and
> if you find about them, tell others. Make sure you only use the API from
> Writing R Extensions (and R help system). If you really can't find
> anything wrong about your package, but still want to learn more, try to
> debug some bugs reported against R runtime or against your favorite
> packages you use (or their CRAN check reports from various tools). In
> addition to learning more about R internals, by spending much more time
> on debugging you may also get a different perspective on some of the
> things about C++ I pointed to. Finally, it would help us with the
> problem we have now - that many R packages in C++ have serious bugs.
>
> Tomas
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>

[[alternative HTML version deleted]]

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] R problems with lapack with gfortran

[Thomas König]

Hi,

I have tried to pinpoint potential problems which could lead to the
LAPACK issues that are currently seen in R.  I built the current R
trunk using

AR=gcc-ar RANLIB=gcc-ranlib ./configure --prefix=$HOME --enable-lto 
--enable-BLAS-shlib=no --without-recommended-packages


and used this to find problem areas.

There are quite a few warnings that were flagged, due to mismatches
in function types.

The prototypes that R has in its header files, for example BLAS.h,
are often not compatible with gfortran function declarations.  To take
one small example, in src/main/print.c, we have

void NORET F77_NAME(xerbla)(const char *srname, int *info)

so xerbla_ is defined with two arguments.

However, gfortran passes string lengths as hidden arguments.
You can see this by compiling the small example

$ cat xer.f
  SUBROUTINE FOO
  INTEGER INFO
  CALL XERBLA ('FOO', INFO)
  END
$ gfortran -c -fdump-tree-original xer.f
$ cat xer.f.004t.original
foo ()
{
  integer(kind=4) info;

  xerbla (&"FOO"[1]{lb: 1 sz: 1}, &info, 3);
}

so here we have three arguments. This mismatch is flagged
by -Wlto-type-mismatch, which, for example, yields

print.c:1120:12: note: type 'void' should match type 'long int'
../../src/extra/blas/blas.f:357:20: warning: type of 'xerbla' does not 
match original declaration [-Wlto-type-mismatch]

  357 |  CALL XERBLA( 'DGBMV ', INFO )


So, why can gcc's r268992 / r269349 matter? Before these patches,
gfortran used the variadic calling convention for calling procedures
outside the current file, and the non-variadic calling convention for
calling procedures found in the current file.

Because the procedures were all compiled as non-variadic, the caller and
the calle's signature did not match if they were not in the same
source file, which is an ABI violation.

This violation manifested itself in https://gcc.gnu.org/PR87689 ,
where the the problem resulted in crashes on a primary gcc platform,
POWER.

How can this potentially affect R?  After the fix for PR87689,
gfortran's calls to external procedures are no longer variadic.  It is
quite possible that, while this "works" most of the time, there
is a problem with a particular LAPACK routine, the call sequence
leading up to it or the procedures it calls.

How to fix this problem?  The only clear way I see is to fix this
on the R side, by adding the string lengths to the prototypes.
These are size_t (64 bit on 64-bit systems, 32 bit on 32-bit
systems).  You should then try to make --enable-lto pass
without any warnings.

Regarding LAPACK itself, the default build system for R builds
it as a shared library.  Offhand, I did not see any way to
build a *.a file instead, so I could not use LTO to check
for mismatched prototypes between R and LAPACK.

Of course, I cannot be sure that this is really the root cause
of the problem you are seeing,but it does seem to fit quite well.
I hope this analysis helps in resolving this.

Regards

Thomas

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Use of C++ in Packages

[Tomas Kalibera]

On 4/24/19 6:41 PM, Hugh Marera wrote:
> Some of us are learning about development in R and use R in our work 
> data analysis pipelines. What is the best way to identify packages 
> that currently have these C++ problems? I would like to be able to 
> help fix the bugs but more importantly not use these packages in 
> critical work pipelines. Any C++ R package bug squashing events out there?

I think the best way available now is manual inspection/review of the 
source code of the packages you are using for your critical work. Such 
review should cover more than just dangerous use of C++ - a lot of 
problems exist also in plain C code (using unexported API from R, 
violating value semantics of R, other kinds of PROTECT errors, memory 
leaks due to long jumps, etc). The review could be limited to the 
context of your pipeline, on how the package is used there and whether 
you have a reliable external process for validating the results.

Out of the problems I've mentioned in my blog, the worst for normal use 
of packages is probably a PROTECT error on the fast path due to 
allocation in a destructor or other function run automatically. Various 
memory leaks or correctness problems on error paths (long jumps) may not 
be a complete showstopper if you restart R often and if you have a 
reliable way of validating results, but such issues would still make it 
much harder to diagnose problems.

The simple steps may include looking at CRAN check results, if there 
were any errors, warnings, notes, reports from analyzers (valgrind, 
asan, ubsan, rchk). The analyzers _may_ be able to spot a PROTECT error 
due to allocation in a destructor if one is lucky (in the case I 
mentioned in the blog, there was an ASAN report), but I think manual 
inspection is needed, and it can also reveal other problems.

Tomas

>
> Regards
>
> Hugh
>
> On Mon, Apr 1, 2019 at 6:23 PM Tomas Kalibera 
> mailto:tomas.kalib...@gmail.com>> wrote:
>
> On 3/30/19 8:59 AM, Romain Francois wrote:
> > tl;dr: we need better C++ tools and documentation.
> >
> > We collectively know more now with the rise of tools like rchk
> and improved documentation such as Tomas’s post. That’s a start,
> but it appears that there still is a lot of knowledge that would
> deserve to be promoted to actual documentation of best practices.
> Well there is quite a bit of knowledge in Writing R Extensions and
> many
> problems could have been prevented had it been read more
> thoroughly by
> package developers. The problem that C++ runs some functions
> automatically (like destructors), should not be too hard to identify
> based on what WRE says about the need for protection against garbage
> collection.
>
>  From my experience, one can learn most about R internals from
> debugging
> and reading source code - when debugging PROTECT errors and other
> memory
> errors/memory corruption, common problems caused by bugs in native
> C/C++
> code - one needs to read and understand source code involved at all
> layers, one needs to understand the documentation covering code at
> different layers, and one has to think about these things, forming
> hypotheses, narrowing down to smaller examples, etc.
>
> My suggestion for package authors who write native code and want to
> learn more, and who want to be responsible (these kinds of bugs
> affect
> other packaged indirectly and can be woken up by inconsequential and
> correct code changes, even in R runtime): test and debug your code
> hard
> - look at UBSAN/ASAN/valgrind/rchk checks from CRAN and run these
> tools
> yourself if needed. Run with strict barrier checking and with
> gctorture.
> Write more tests to increase the coverage. Specifically now if you
> use
> C++ code, try to read all of your related code and check you do
> not have
> the problems I mentioned in my blog. Think of other related
> problems and
> if you find about them, tell others. Make sure you only use the
> API from
> Writing R Extensions (and R help system). If you really can't find
> anything wrong about your package, but still want to learn more,
> try to
> debug some bugs reported against R runtime or against your favorite
> packages you use (or their CRAN check reports from various tools). In
> addition to learning more about R internals, by spending much more
> time
> on debugging you may also get a different perspective on some of the
> things about C++ I pointed to. Finally, it would help us with the
> problem we have now - that many R packages in C++ have serious bugs.
>
> Tomas
>
> __
> R-devel@r-project.org  mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>


[[alternative HTML version deleted]]

_

Re: [Rd] Questions/suggestions about new staged installation

[Tomas Kalibera]

On 4/25/19 3:11 AM, Pages, Herve wrote:

Hi,

I was playing around with inotifywait (great tool!) to see the new
staged installation of source packages in action. In one terminal I'm
monitoring the create/delete/move events of the installation library with:

    inotifywait -m --timefmt '%F %T' --format '%T -- %w %e %f' -e create
-e delete -e move path/to/R/library/

While in another terminal I install CRAN package abc with:

    R CMD INSTALL abc_2.1.tar.gz

All deps are already installed.

## With R 3.5

When installing abc the first time with R 3.5 inotifywait reports the
following events:

2019-04-24 16:57:42 -- /home/hpages/R/R-3.5.r75051/library/ CREATE,ISDIR
00LOCK-abc
2019-04-24 16:57:42 -- /home/hpages/R/R-3.5.r75051/library/ CREATE,ISDIR abc
2019-04-24 16:57:45 -- /home/hpages/R/R-3.5.r75051/library/ DELETE,ISDIR
00LOCK-

Then on subsequent times:

2019-04-24 16:58:14 -- /home/hpages/R/R-3.5.r75051/library/ CREATE,ISDIR
00LOCK-abc
2019-04-24 16:58:14 -- /home/hpages/R/R-3.5.r75051/library/
MOVED_FROM,ISDIR abc
2019-04-24 16:58:14 -- /home/hpages/R/R-3.5.r75051/library/ CREATE,ISDIR abc
2019-04-24 16:58:17 -- /home/hpages/R/R-3.5.r75051/library/ DELETE,ISDIR
00LOCK-abc

IIUC the additional 'MOVED_FROM,ISDIR abc'  event on subsequent installs
is triggered by the backing up of the earlier installation in case the
new installation fails.

## With R 3.6

First installation:

2019-04-24 17:09:04 -- /home/hpages/R/R-3.6.r76385/library/ CREATE,ISDIR
00LOCK-abc
2019-04-24 17:09:04 -- /home/hpages/R/R-3.6.r76385/library/ CREATE,ISDIR abc
2019-04-24 17:09:08 -- /home/hpages/R/R-3.6.r76385/library/
MOVED_TO,ISDIR abc
2019-04-24 17:09:09 -- /home/hpages/R/R-3.6.r76385/library/ DELETE,ISDIR
00LOCK-abc

Subsequent installations:

2019-04-24 17:09:59 -- /home/hpages/R/R-3.6.r76385/library/ CREATE,ISDIR
00LOCK-abc
2019-04-24 17:09:59 -- /home/hpages/R/R-3.6.r76385/library/
MOVED_FROM,ISDIR abc
2019-04-24 17:09:59 -- /home/hpages/R/R-3.6.r76385/library/ CREATE,ISDIR abc
2019-04-24 17:10:02 -- /home/hpages/R/R-3.6.r76385/library/
MOVED_TO,ISDIR abc
2019-04-24 17:10:03 -- /home/hpages/R/R-3.6.r76385/library/ DELETE,ISDIR
00LOCK-abc

IIUC the new 'MOVED_TO,ISDIR abc' event is triggered by the package
installation folder being moved from the temporary location to the final
location.

However I wonder about the CREATE,ISDIR abc event. It looks like even
for a staged installation tools:::.install_packages() still creates the
empty abc dir. This should no longer be necessary because this empty
folder is replaced later by the 'MOVED_TO,ISDIR abc' event. Am I missing
somethin


Yes, the final directory location gets created before the installation. 
I did this to minimize the added cognitive complexity to the code. I 
could explore if this could be changed, if the installation could work 
without that directory (which would mean implementing and testing on all 
CRAN/BIOC, installing in different modes, so certainly some effort).  Is 
the current implementation causing any problem?



Another thing: the times indicate that the 'MOVED_TO,ISDIR abc' event
(move from temp to final location) happens about 3-4 seconds after the
'MOVED_FROM,ISDIR abc' event (backing up). In this interval of time, the
abc package is missing. Wouldn't it be safer to leave the earlier
installation where it is and to replace it when the new installation
folder is moved from the temporary location to the final location? Said
otherwise, the backing up step no longer seems necessary with staged
installs. So inotifywait would report something like this:

2019-04-24 17:09:59 -- /home/hpages/R/R-3.6.r76385/library/ CREATE,ISDIR
00LOCK-abc
2019-04-24 17:10:02 -- /home/hpages/R/R-3.6.r76385/library/
MOVED_TO,ISDIR abc
2019-04-24 17:10:03 -- /home/hpages/R/R-3.6.r76385/library/ DELETE,ISDIR
00LOCK-abc

That is, only 3 events. On first and subsequent installation.


I don't think that would be possible. The package is test-loaded also 
from the final installation directory, and if that fails, restored from 
a back-up. This test-loading is useful as it can detect some problems 
with hard-coded directories possibly not detected before on test-loading 
from the temporary installation directory.


Best,
Tomas



Cheers,

H.


  > sessionInfo()
R version 3.6.0 beta (2019-04-12 r76385)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.5 LTS

Matrix products: default
BLAS:   /home/hpages/R/R-3.6.r76385/lib/libRblas.so
LAPACK: /home/hpages/R/R-3.6.r76385/lib/libRlapack.so

locale:
   [1] LC_CTYPE=en_US.UTF-8   LC_NUMERIC=C
   [3] LC_TIME=en_US.UTF-8    LC_COLLATE=en_US.UTF-8
   [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
   [7] LC_PAPER=en_US.UTF-8   LC_NAME=C
   [9] LC_ADDRESS=C   LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats graphics  grDevices utils datasets  methods base

loaded via a namespace (and not attached):
[1] compiler_3.6.0

Re: [Rd] Questions/suggestions about new staged installation

[Pages, Herve]

On 4/25/19 04:57, Tomas Kalibera wrote:

> On 4/25/19 3:11 AM, Pages, Herve wrote:
>> Hi,
>>
>> I was playing around with inotifywait (great tool!) to see the new
>> staged installation of source packages in action. In one terminal I'm
>> monitoring the create/delete/move events of the installation library 
>> with:
>>
>>     inotifywait -m --timefmt '%F %T' --format '%T -- %w %e %f' -e create
>> -e delete -e move path/to/R/library/
>>
>> While in another terminal I install CRAN package abc with:
>>
>>     R CMD INSTALL abc_2.1.tar.gz
>>
>> All deps are already installed.
>>
>> ## With R 3.5
>>
>> When installing abc the first time with R 3.5 inotifywait reports the
>> following events:
>>
>> 2019-04-24 16:57:42 -- /home/hpages/R/R-3.5.r75051/library/ CREATE,ISDIR
>> 00LOCK-abc
>> 2019-04-24 16:57:42 -- /home/hpages/R/R-3.5.r75051/library/ 
>> CREATE,ISDIR abc
>> 2019-04-24 16:57:45 -- /home/hpages/R/R-3.5.r75051/library/ DELETE,ISDIR
>> 00LOCK-
>>
>> Then on subsequent times:
>>
>> 2019-04-24 16:58:14 -- /home/hpages/R/R-3.5.r75051/library/ CREATE,ISDIR
>> 00LOCK-abc
>> 2019-04-24 16:58:14 -- /home/hpages/R/R-3.5.r75051/library/
>> MOVED_FROM,ISDIR abc
>> 2019-04-24 16:58:14 -- /home/hpages/R/R-3.5.r75051/library/ 
>> CREATE,ISDIR abc
>> 2019-04-24 16:58:17 -- /home/hpages/R/R-3.5.r75051/library/ DELETE,ISDIR
>> 00LOCK-abc
>>
>> IIUC the additional 'MOVED_FROM,ISDIR abc'  event on subsequent installs
>> is triggered by the backing up of the earlier installation in case the
>> new installation fails.
>>
>> ## With R 3.6
>>
>> First installation:
>>
>> 2019-04-24 17:09:04 -- /home/hpages/R/R-3.6.r76385/library/ CREATE,ISDIR
>> 00LOCK-abc
>> 2019-04-24 17:09:04 -- /home/hpages/R/R-3.6.r76385/library/ 
>> CREATE,ISDIR abc
>> 2019-04-24 17:09:08 -- /home/hpages/R/R-3.6.r76385/library/
>> MOVED_TO,ISDIR abc
>> 2019-04-24 17:09:09 -- /home/hpages/R/R-3.6.r76385/library/ DELETE,ISDIR
>> 00LOCK-abc
>>
>> Subsequent installations:
>>
>> 2019-04-24 17:09:59 -- /home/hpages/R/R-3.6.r76385/library/ CREATE,ISDIR
>> 00LOCK-abc
>> 2019-04-24 17:09:59 -- /home/hpages/R/R-3.6.r76385/library/
>> MOVED_FROM,ISDIR abc
>> 2019-04-24 17:09:59 -- /home/hpages/R/R-3.6.r76385/library/ 
>> CREATE,ISDIR abc
>> 2019-04-24 17:10:02 -- /home/hpages/R/R-3.6.r76385/library/
>> MOVED_TO,ISDIR abc
>> 2019-04-24 17:10:03 -- /home/hpages/R/R-3.6.r76385/library/ DELETE,ISDIR
>> 00LOCK-abc
>>
>> IIUC the new 'MOVED_TO,ISDIR abc' event is triggered by the package
>> installation folder being moved from the temporary location to the final
>> location.
>>
>> However I wonder about the CREATE,ISDIR abc event. It looks like even
>> for a staged installation tools:::.install_packages() still creates the
>> empty abc dir. This should no longer be necessary because this empty
>> folder is replaced later by the 'MOVED_TO,ISDIR abc' event. Am I missing
>> somethin
>
> Yes, the final directory location gets created before the 
> installation. I did this to minimize the added cognitive complexity to 
> the code. I could explore if this could be changed, if the 
> installation could work without that directory (which would mean 
> implementing and testing on all CRAN/BIOC, installing in different 
> modes, so certainly some effort).  Is the current implementation 
> causing any problem?
>
>> Another thing: the times indicate that the 'MOVED_TO,ISDIR abc' event
>> (move from temp to final location) happens about 3-4 seconds after the
>> 'MOVED_FROM,ISDIR abc' event (backing up). In this interval of time, the
>> abc package is missing. Wouldn't it be safer to leave the earlier
>> installation where it is and to replace it when the new installation
>> folder is moved from the temporary location to the final location? Said
>> otherwise, the backing up step no longer seems necessary with staged
>> installs. So inotifywait would report something like this:
>>
>> 2019-04-24 17:09:59 -- /home/hpages/R/R-3.6.r76385/library/ CREATE,ISDIR
>> 00LOCK-abc
>> 2019-04-24 17:10:02 -- /home/hpages/R/R-3.6.r76385/library/
>> MOVED_TO,ISDIR abc
>> 2019-04-24 17:10:03 -- /home/hpages/R/R-3.6.r76385/library/ DELETE,ISDIR
>> 00LOCK-abc
>>
>> That is, only 3 events. On first and subsequent installation.
>
> I don't think that would be possible. The package is test-loaded also 
> from the final installation directory, and if that fails, restored 
> from a back-up. This test-loading is useful as it can detect some 
> problems with hard-coded directories possibly not detected before on 
> test-loading from the temporary installation directory.

Makes sense.

So backing up is still needed.

But couldn't the gap of 3-4 seconds where the abc package is missing be 
reduced by backing up right before moving the newly installed package 
from the temporary location to the final location?

In other words, instead of:

     back up, then install to temp loc, then move to final loc

do:

     install to temp loc, then back up, then move to final loc

This would reduce the gap to almost 0 seco

Re: [Rd] Questions/suggestions about new staged installation

[Tomas Kalibera]

On 4/25/19 2:20 PM, Pages, Herve wrote:

On 4/25/19 04:57, Tomas Kalibera wrote:


On 4/25/19 3:11 AM, Pages, Herve wrote:

Hi,

I was playing around with inotifywait (great tool!) to see the new
staged installation of source packages in action. In one terminal I'm
monitoring the create/delete/move events of the installation library
with:

     inotifywait -m --timefmt '%F %T' --format '%T -- %w %e %f' -e create
-e delete -e move path/to/R/library/

While in another terminal I install CRAN package abc with:

     R CMD INSTALL abc_2.1.tar.gz

All deps are already installed.

## With R 3.5

When installing abc the first time with R 3.5 inotifywait reports the
following events:

2019-04-24 16:57:42 -- /home/hpages/R/R-3.5.r75051/library/ CREATE,ISDIR
00LOCK-abc
2019-04-24 16:57:42 -- /home/hpages/R/R-3.5.r75051/library/
CREATE,ISDIR abc
2019-04-24 16:57:45 -- /home/hpages/R/R-3.5.r75051/library/ DELETE,ISDIR
00LOCK-

Then on subsequent times:

2019-04-24 16:58:14 -- /home/hpages/R/R-3.5.r75051/library/ CREATE,ISDIR
00LOCK-abc
2019-04-24 16:58:14 -- /home/hpages/R/R-3.5.r75051/library/
MOVED_FROM,ISDIR abc
2019-04-24 16:58:14 -- /home/hpages/R/R-3.5.r75051/library/
CREATE,ISDIR abc
2019-04-24 16:58:17 -- /home/hpages/R/R-3.5.r75051/library/ DELETE,ISDIR
00LOCK-abc

IIUC the additional 'MOVED_FROM,ISDIR abc'  event on subsequent installs
is triggered by the backing up of the earlier installation in case the
new installation fails.

## With R 3.6

First installation:

2019-04-24 17:09:04 -- /home/hpages/R/R-3.6.r76385/library/ CREATE,ISDIR
00LOCK-abc
2019-04-24 17:09:04 -- /home/hpages/R/R-3.6.r76385/library/
CREATE,ISDIR abc
2019-04-24 17:09:08 -- /home/hpages/R/R-3.6.r76385/library/
MOVED_TO,ISDIR abc
2019-04-24 17:09:09 -- /home/hpages/R/R-3.6.r76385/library/ DELETE,ISDIR
00LOCK-abc

Subsequent installations:

2019-04-24 17:09:59 -- /home/hpages/R/R-3.6.r76385/library/ CREATE,ISDIR
00LOCK-abc
2019-04-24 17:09:59 -- /home/hpages/R/R-3.6.r76385/library/
MOVED_FROM,ISDIR abc
2019-04-24 17:09:59 -- /home/hpages/R/R-3.6.r76385/library/
CREATE,ISDIR abc
2019-04-24 17:10:02 -- /home/hpages/R/R-3.6.r76385/library/
MOVED_TO,ISDIR abc
2019-04-24 17:10:03 -- /home/hpages/R/R-3.6.r76385/library/ DELETE,ISDIR
00LOCK-abc

IIUC the new 'MOVED_TO,ISDIR abc' event is triggered by the package
installation folder being moved from the temporary location to the final
location.

However I wonder about the CREATE,ISDIR abc event. It looks like even
for a staged installation tools:::.install_packages() still creates the
empty abc dir. This should no longer be necessary because this empty
folder is replaced later by the 'MOVED_TO,ISDIR abc' event. Am I missing
somethin

Yes, the final directory location gets created before the
installation. I did this to minimize the added cognitive complexity to
the code. I could explore if this could be changed, if the
installation could work without that directory (which would mean
implementing and testing on all CRAN/BIOC, installing in different
modes, so certainly some effort).  Is the current implementation
causing any problem?


Another thing: the times indicate that the 'MOVED_TO,ISDIR abc' event
(move from temp to final location) happens about 3-4 seconds after the
'MOVED_FROM,ISDIR abc' event (backing up). In this interval of time, the
abc package is missing. Wouldn't it be safer to leave the earlier
installation where it is and to replace it when the new installation
folder is moved from the temporary location to the final location? Said
otherwise, the backing up step no longer seems necessary with staged
installs. So inotifywait would report something like this:

2019-04-24 17:09:59 -- /home/hpages/R/R-3.6.r76385/library/ CREATE,ISDIR
00LOCK-abc
2019-04-24 17:10:02 -- /home/hpages/R/R-3.6.r76385/library/
MOVED_TO,ISDIR abc
2019-04-24 17:10:03 -- /home/hpages/R/R-3.6.r76385/library/ DELETE,ISDIR
00LOCK-abc

That is, only 3 events. On first and subsequent installation.

I don't think that would be possible. The package is test-loaded also
from the final installation directory, and if that fails, restored
from a back-up. This test-loading is useful as it can detect some
problems with hard-coded directories possibly not detected before on
test-loading from the temporary installation directory.

Makes sense.

So backing up is still needed.

But couldn't the gap of 3-4 seconds where the abc package is missing be
reduced by backing up right before moving the newly installed package
from the temporary location to the final location?

In other words, instead of:

      back up, then install to temp loc, then move to final loc

do:

      install to temp loc, then back up, then move to final loc

This would reduce the gap to almost 0 seconds on a Unix file system
where moving a folder is very cheap (even if the folder is big), at
least when moving doesn't cross partition borders.

Not a big deal for a light package like abc but some Bioconductor
packages take about 20 min. to compile!


It shoul

Re: [Rd] update.package() asking even when asked to not ask

[Tomas Kalibera]

On 4/20/19 10:39 PM, Pages, Herve wrote:

Hi,

I was trying to update packages today on one of our MacOS servers, and
got this:

    > update.packages(ask=FALSE)

      There is a binary version available but the source version is later:
      binary source needs_compilation
    gsl 1.9-10.3  2.1-6  TRUE

    Do you want to install from sources the package which needs
compilation? (Yes/no/cancel)

So it looks like 'ask=FALSE' only turns off one type of question. What
is not immediately obvious is that in order to also turn off the type of
question asked above one needs to set 'type' either to
"mac.binary.el-capitan" or "source" (by default 'type' is set to "both"
on Mac).

Some clarification in the man page about the scope of 'ask=FALSE' would
be welcome, with possibly some hint about how to turn off all questions.


From reading the documentation I think this behavior is intentional and 
as documented. See "Binary packages" section in ?install.packages. In 
particular there is option "install.packages.compile.from.source". 
?update.packages points to this option via "See Also:" 
‘install.packages’, "The options listed for ‘install.packages’ under 
‘options’.". The documentation for "ask" does not seem to suggest that 
this would control all options asked by update.packages(), but I've 
added a sentence to make that explicit.


Best
Tomas



Thanks,

H.

  > sessionInfo()
R version 3.6.0 beta (2019-04-16 r76403)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: OS X El Capitan 10.11.6

Matrix products: default
BLAS:
/Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRblas.0.dylib
LAPACK:
/Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats graphics  grDevices utils datasets  methods base

loaded via a namespace (and not attached):
[1] compiler_3.6.0 tools_3.6.0



__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] configure script issue with -flto with recent gcc and system ar/ranlib

[Tomas Kalibera]

On 4/23/19 2:59 PM, Thomas König wrote:

Hi,

there can be an issue with recent gcc where the system-installed "ar"
and "ranlib" commands cannot handle LTO binaries.  On compilation, this
manifests itself with error messages claiming that they need extra
plugins.

This can be fixed by using the command line

$ AR=gcc-ar RANLIB=gcc-ranlib ./configure --enable-lto

so it is not a big issue, but it would still be nicer if the configure
script tested the functionality of ar and ranlib itself and would
select the appropriate one accordingly.

This is with R version 3.5.3.


Thanks for the report. What was the version of binutils on the system 
with this problem? On my Ubuntu 18.04 I can use the binutils version of 
"ar" and "ranlib" with --enable-lto without problems.  I read that with 
recent binutils (2.25?), the LTO plugin should be loaded automatically, 
so one does not have to use the wrappers anymore.


Thanks
Tomas



Regards

Thomas

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel


__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] configure script issue with -flto with recent gcc and system ar/ranlib

[Thomas König]

Hi Tomas,


On 4/23/19 2:59 PM, Thomas König wrote:

Hi,

there can be an issue with recent gcc where the system-installed "ar"
and "ranlib" commands cannot handle LTO binaries.  On compilation, this
manifests itself with error messages claiming that they need extra
plugins.


Thanks for the report. What was the version of binutils on the system 
with this problem? On my Ubuntu 18.04 I can use the binutils version of 
"ar" and "ranlib" with --enable-lto without problems.  I read that with 
recent binutils (2.25?), the LTO plugin should be loaded automatically, 
so one does not have to use the wrappers anymore.


This was with, on x86_64-pc-linux-gnu,

GNU ar (GNU Binutils; openSUSE Leap 42.3) 2.31.1.20180828-19

and, on powerpc64le-unknown-linux-gnu,

GNU ar version 2.27-34.base.el7

both with a recent gcc 9.0.1 snapshot.

Regards

Thomas

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] update.package() asking even when asked to not ask

[Pages, Herve]

On 4/25/19 07:50, Tomas Kalibera wrote:

> On 4/20/19 10:39 PM, Pages, Herve wrote:
>> Hi,
>>
>> I was trying to update packages today on one of our MacOS servers, and
>> got this:
>>
>>     > update.packages(ask=FALSE)
>>
>>       There is a binary version available but the source version is 
>> later:
>>       binary source needs_compilation
>>     gsl 1.9-10.3  2.1-6  TRUE
>>
>>     Do you want to install from sources the package which needs
>> compilation? (Yes/no/cancel)
>>
>> So it looks like 'ask=FALSE' only turns off one type of question. What
>> is not immediately obvious is that in order to also turn off the type of
>> question asked above one needs to set 'type' either to
>> "mac.binary.el-capitan" or "source" (by default 'type' is set to "both"
>> on Mac).
>>
>> Some clarification in the man page about the scope of 'ask=FALSE' would
>> be welcome, with possibly some hint about how to turn off all questions.
>
> From reading the documentation I think this behavior is intentional 
> and as documented. See "Binary packages" section in ?install.packages. 
> In particular there is option "install.packages.compile.from.source". 
> ?update.packages points to this option via "See Also:" 
> ‘install.packages’, "The options listed for ‘install.packages’ under 
> ‘options’.". The documentation for "ask" does not seem to suggest that 
> this would control all options asked by update.packages(), but I've 
> added a sentence to make that explicit.

Thanks Tomas. This small addition to the description of the 'ask' 
argument helps a lot and the pointer to 
'install.packages.compile.from.source' is very useful.

Cheers,

H.


>
> Best
> Tomas
>
>
>> Thanks,
>>
>> H.
>>
>>   > sessionInfo()
>> R version 3.6.0 beta (2019-04-16 r76403)
>> Platform: x86_64-apple-darwin15.6.0 (64-bit)
>> Running under: OS X El Capitan 10.11.6
>>
>> Matrix products: default
>> BLAS:
>> /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRblas.0.dylib 
>>
>> LAPACK:
>> /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib 
>>
>>
>> locale:
>> [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
>>
>> attached base packages:
>> [1] stats graphics  grDevices utils datasets  methods base
>>
>> loaded via a namespace (and not attached):
>> [1] compiler_3.6.0 tools_3.6.0
>>
>
-- 
Hervé Pagès

Program in Computational Biology
Division of Public Health Sciences
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N, M1-B514
P.O. Box 19024
Seattle, WA 98109-1024

E-mail: hpa...@fredhutch.org
Phone:  (206) 667-5791
Fax:(206) 667-1319

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] [FORGED] src/modules/X11/devX11.c, can we remove "#if BUG" yet

[frederik]

Thanks Professor Dalgard.

If you have a different way to fix the bug then I'd be happy to test
it.

Or whatever. I understand that maybe some data was being referenced
before it had been initialized. I could also support moving the
R_ProcessEvents call in another place, but it seems one would also
like to generate some kind of warning message, at the location of the
bad reference, rather than segfaulting. Was it not possible to
identify this location? I'm guessing that Valgrind is a bit more
mature now than it was in 2001...?

Frederick

On Wed, Apr 24, 2019 at 03:12:55PM +0200, peter dalgaard wrote:

OK, so I did the archaeology anyway


This was the story, R-core November 29, 2001. Part of thread "X11 still 
segfaults".

>>
.
Gah. I've been too tired today. Why did that take me so long?

The culprit seems to be

R_ProcessEvents((void*) NULL)

in newX11DeviceDriver

This gets called *before* this stuff at the end of Rf_addX11Device

dd = GEcreateDevDesc(dev);
addDevice((DevDesc*) dd);
initDisplayList((DevDesc*) dd);

and it is that "dd" that gets called by Rf_playDisplayList. Removing
the offending line stops the segfaulting, seemingly with no ill
effects.

I'm not really sure what the use of that line ever was; it might be
necessary to make the call somewhere later, but it appears to have
been possible to race past it before receiving any events all the
time.

I also changed a couple of spots missing dd->newDevStruct=1

Will commit in a moment.
<<

And the following day, in "graphics saga part III", we had

->>
...

I can't make it happen in 1.3.1 but...

It is probably not unrelated to the R_ProcessEvents line that
I took out, but that was definitely wrong. However, one might reenable
it if one could change this bit of code

if (!(ptr_X11DeviceDriver)((DevDesc*)(dev), display, width, height, ps, 
gamma,
  colormodel, maxcubesize, canvascolor)) {
free(dev);
errorcall(gcall, "unable to start device %s", devname);
}
gsetVar(install(".Device"), mkString(devname), R_NilValue);
dd = GEcreateDevDesc(dev);
addDevice((DevDesc*) dd);
initDisplayList((DevDesc*) dd);


and put the if-clause last. A cursory clance through the three
functions that are being called didn't reveal anything that would rely
on having opened the device driver first.

Paul?

(I might try it locally, but I'm not sure I should commit anything.)

<<---

It seems that the suggestion was never followed up on?

-pd



On 24 Apr 2019, at 11:42 , peter dalgaard  wrote:

I don't recall exactly what I did 18 years ago eiher and I likely don't have 
the time to dig into the archives and reconstruct.

I can imagine that the issue had to do with the protocol around creating and 
mapping windows. Presumably the segfault comes from looking for events on a 
window that hasn't been created yet, or has already been destroyed, leading to 
a NULL reference somewhere. I have a vague recollection that the issue was 
window manager dependent (in 2001 probably not twm, more likely xvwm on RedHat 
if it was affecting me).

A proper fix should go via proper understanding of the X11 protocol - uncommenting a line 
is as bad as commenting it in the 1st place So more like "wait for window to 
exist THEN process events" -- but the 1st part may be WM specific, etc.

I recall docs being quite obtuse, and the X11 "mechanism not policy" credo 
doesn't help as WMs are not obliged to (say) send notifications, so you can end up 
stalling, waiting for events that never happen.

It is entirely possible that there is stuff in here that I didn't understand 
properly at the time, and still don't!

- pd


On 24 Apr 2019, at 02:30 , Paul Murrell  wrote:

Hi

Sorry, I can't offer an explanation for the commented-out line.
However, regarding your final question of avoiding the R-core bottleneck, you 
do have the option of creating a third-party graphics device package.  See, for 
example, the 'tikzDevice' and 'svglite' packages on CRAN.  Does that provide 
you with a way forward ?

Paul

On 20/04/2019 5:27 p.m., frede...@ofb.net wrote:

Dear R Devel,

I know that someone put this line in src/modules/X11/devX11.c:2824 for
a reason, because commenting it out causes R to miss an important
ConfigureNotify event in my window manager. The result is that plots
are initially drawn off the window borders, unreadable.

  R_ProcessX11Events((void*) NULL);

Unfortunately for me, this line is commented in the standard release
of R, it has "#if BUG ... #endif" around it.

I guess it is also unfortunate for anyone who uses the same window
manager as I do, namely i3, which I think is pretty popular among Unix
power users these days; not to mention other full-screen window
managers which probably exhibit the same bug in R.

Maybe everyone on the Core team uses twm as their window manager? Or
RStudio on Windows? Whic