Re: [Rd] Build failure on powerpc64

[Martin Maechler]

> Tom Callaway 
> on Thu, 12 Dec 2019 14:21:10 -0500 writes:

> Hi R folks,

> Went to build R 3.6.2 for Fedora/EPEL and got failures across the board.

> Disabling the test suite for all non-intel architectures resolves most of
> the failures, but powerpc64 dies in the compiler, specifically here:

> gcc -m64  -I../../src/extra/xdr -I. -I../../src/include 
-I../../src/include
> -I/usr/local/include -I../../src/nmath -DHAVE_CONFIG_H   -fopenmp -fPIC
> -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
> -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong
> -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
> -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mcpu=power8
> -mtune=power8 -fasynchronous-unwind-tables -fstack-clash-protection  -c
> arithmetic.c -o arithmetic.o
> arithmetic.c:180:26: error: initializer element is not constant
> 180 | static LDOUBLE q_1_eps = (1 / LDBL_EPSILON);
> |  ^
> make[3]: *** [../../Makeconf:124: arithmetic.o] Error 1

> Took me a bit to figure out why, but this is happening because on
> powerpc64, gcc is compiled with -mlong-double-128, and the long double
> format used on PPC is IBM's 128bit long double which is two doubles added
> together. As a result, gcc can't safely do const assignments to long
> doubles on ppc64, so it dies there.

> The fix is easy enough, do not try to assign a value to a static long
> double on ppc64.
> --- ./src/main/arithmetic.c.orig2019-12-12 18:30:12.416334062 
+
> +++ ./src/main/arithmetic.c 2019-12-12 18:30:44.966334062 +
> @@ -179,7 +179,10 @@ void attribute_hidden InitArithmetic()
> #endif
> }

> -#if HAVE_LONG_DOUBLE && (SIZEOF_LONG_DOUBLE > SIZEOF_DOUBLE)
> +/* PowerPC 64 (when gcc has -mlong-double-128) breaks here because
> + * of issues constant folding 128bit IBM long doubles.
> + */
> +#if HAVE_LONG_DOUBLE && (SIZEOF_LONG_DOUBLE > SIZEOF_DOUBLE) && 
!__PPC64__
> static LDOUBLE q_1_eps = 1 / LDBL_EPSILON;
> #else
> static double  q_1_eps = 1 / DBL_EPSILON;


> Hope that helps someone else.
> Tom

Thank you, Tom.  That is "interesting"  ...

The piece in question had been added by me,

r77193 | maechler | 2019-09-18 13:21:49 +0200 (Wed, 18 Sep 2019) | 1 line

 x %% +/- Inf  now typically return non-NaN; fix the "FIXME" in C

in order to use double precision in order to deal with finite cases
close to Inf, etc.

IIUC, your proposed patch is really a workaround a bug on PPC64 ?

But note the check on LONG_DOUBLE is not the only one in R's
sources: Via 'grep' in src/main/*.?  I also see

connections.c4285:#if HAVE_LONG_DOUBLE && (SIZEOF_LONG_DOUBLE > SIZEOF_DOUBLE)
connections.c4329:#if HAVE_LONG_DOUBLE
connections.c4379:#if HAVE_LONG_DOUBLE && (SIZEOF_LONG_DOUBLE > SIZEOF_DOUBLE)
connections.c4514:#if HAVE_LONG_DOUBLE && (SIZEOF_LONG_DOUBLE > SIZEOF_DOUBLE)
connections.c4592:#if HAVE_LONG_DOUBLE && (SIZEOF_LONG_DOUBLE > SIZEOF_DOUBLE)
format.c250:#if defined(HAVE_LONG_DOUBLE) && (SIZEOF_LONG_DOUBLE > 
SIZEOF_DOUBLE)
format.c285:#if defined(HAVE_LONG_DOUBLE) && (SIZEOF_LONG_DOUBLE > 
SIZEOF_DOUBLE)
format.c339:#if defined(HAVE_LONG_DOUBLE) && (SIZEOF_LONG_DOUBLE > 
SIZEOF_DOUBLE)

Do they also need protection against this PPC64 bug ?

Best,

Martin Maechler
ETH Zurich and R Core Team
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Build failure on powerpc64

[Tom Callaway]

An excellent question. It is important to remember two key facts:

1. With gcc on ppc64, long doubles exist, they can be used, just not safely
as constants (and maybe they still can be used safely under specific
conditions?).
2. I am not an expert in either PowerPC64 or gcc. :)

Looking at connections.c, we have (in order):
  * handling long double as a valid case in a switch statement checking size
  * adding long double as a field in the u union define
  * handling long double as a valid case in a switch statement checking size
  * handling long double as a valid case in a switch statement checking size
  * memcpy from the address of a long double

In format.c, we have (in order):
  * conditionally creating private_nearbyintl for R_nearbyintl
  * defining a static const long double tbl[]
  * use exact scaling factor in long double precision

For most of these, it seems safe to leave them as is for ppc64. I would
have thought that the gcc compiler might have had issue with:

connections.c:
  static long double ld1;
for (i = 0, j = 0; i < len; i++, j += size) {
ld1 = (long double) REAL(object)[i];

format.c:
   static const long double tbl[] =

... but it doesn't. Perhaps the original code at issue:

arithmetic.c:
   static LDOUBLE q_1_eps = 1 / LDBL_EPSILON;

only makes gcc unhappy because of the very large value trying to be stored
in the static long double, which would make it span the "folded double" on
that architecture.

*

It seems that the options are:

A) Patch the one place where the compiler determines it is not safe to use
a static long double on ppc64.
B) Treat PPC64 as a platform where it is never safe to use a static long
double

FWIW, I did run the test suite after applying my patch and all of the tests
pass on ppc64.

Tom


On Fri, Dec 13, 2019 at 5:44 AM Martin Maechler 
wrote:

> > Tom Callaway
> > on Thu, 12 Dec 2019 14:21:10 -0500 writes:
>
> > Hi R folks,
>
> > Went to build R 3.6.2 for Fedora/EPEL and got failures across the
> board.
>
> > Disabling the test suite for all non-intel architectures resolves
> most of
> > the failures, but powerpc64 dies in the compiler, specifically here:
>
> > gcc -m64  -I../../src/extra/xdr -I. -I../../src/include
> -I../../src/include
> > -I/usr/local/include -I../../src/nmath -DHAVE_CONFIG_H   -fopenmp
> -fPIC
> > -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
> > -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong
> > -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
> > -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mcpu=power8
> > -mtune=power8 -fasynchronous-unwind-tables -fstack-clash-protection
> -c
> > arithmetic.c -o arithmetic.o
> > arithmetic.c:180:26: error: initializer element is not constant
> > 180 | static LDOUBLE q_1_eps = (1 / LDBL_EPSILON);
> > |  ^
> > make[3]: *** [../../Makeconf:124: arithmetic.o] Error 1
>
> > Took me a bit to figure out why, but this is happening because on
> > powerpc64, gcc is compiled with -mlong-double-128, and the long
> double
> > format used on PPC is IBM's 128bit long double which is two doubles
> added
> > together. As a result, gcc can't safely do const assignments to long
> > doubles on ppc64, so it dies there.
>
> > The fix is easy enough, do not try to assign a value to a static long
> > double on ppc64.
> > --- ./src/main/arithmetic.c.orig2019-12-12
> 18:30:12.416334062 +
> > +++ ./src/main/arithmetic.c 2019-12-12 18:30:44.966334062 +
> > @@ -179,7 +179,10 @@ void attribute_hidden InitArithmetic()
> > #endif
> > }
>
> > -#if HAVE_LONG_DOUBLE && (SIZEOF_LONG_DOUBLE > SIZEOF_DOUBLE)
> > +/* PowerPC 64 (when gcc has -mlong-double-128) breaks here because
> > + * of issues constant folding 128bit IBM long doubles.
> > + */
> > +#if HAVE_LONG_DOUBLE && (SIZEOF_LONG_DOUBLE > SIZEOF_DOUBLE) &&
> !__PPC64__
> > static LDOUBLE q_1_eps = 1 / LDBL_EPSILON;
> > #else
> > static double  q_1_eps = 1 / DBL_EPSILON;
>
>
> > Hope that helps someone else.
> > Tom
>
> Thank you, Tom.  That is "interesting"  ...
>
> The piece in question had been added by me,
> 
> r77193 | maechler | 2019-09-18 13:21:49 +0200 (Wed, 18 Sep 2019) | 1 line
>
>  x %% +/- Inf  now typically return non-NaN; fix the "FIXME" in C
> 
> in order to use double precision in order to deal with finite cases
> close to Inf, etc.
>
> IIUC, your proposed patch is really a workaround a bug on PPC64 ?
>
> But note the check on LONG_DOUBLE is not the only one in R's
> sources: Via 'grep' in src/main/*.?  I also see
>
> connections.c 4285:#if HAVE_LONG_DOUBLE && (SIZEOF_LONG_DOUBLE >
> SIZEOF_DOUBLE)
>

Re: [Rd] Build failure on powerpc64

[Serguei Sokol]

Le 13/12/2019 à 17:06, Tom Callaway a écrit :

arithmetic.c:
static LDOUBLE q_1_eps = 1 / LDBL_EPSILON;
Just a thought: can it be that it's "1" which is at the origin of 
compiler complaint?

In this case, would the syntax "1.L" be sufficient to keep it calm?

Serguei.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] tempdir() containing spaces breaks installing source packages

[Ivan Krylov]

Hello everyone!

Temp paths are used in system2() calls without shQuote() because
they are assumed not to contain spaces. On Windows,
GetShortPathName() is used to try to ensure that.

Unfortunately, sometimes GetShortPathName() silently fails to return a
8.3 file path and gives a full path instead (8.3 file names might be
disabled on newer Windows 10 installations, or there may be another
reason). This has been spotted in the wild [*]. When %USERPROFILE%
contains spaces, this results in tempdir() also containing spaces and
prevents the user from being able to install source packages.

As of ,

 - src/library/utils/R/packages2.R line 839 contains an unquoted
   temporary file path (fil) passed to system2(), which results in it
   being split and R CMD INSTALL not being able to find the package
   file. In other invocations of R CMD INSTALL in the same file, the
   path is properly quoted.

 - src/library/tools/R/check.R line 125 contains an unquoted temporary
   file path passed to system2, which results in Rterm.exe -f not being
   able to find the RtmpXX\Rin file, causing the attempt to
   run tools:::makeLazyLoading(...) to fail.

I can report these two problems (thanks to Martin Maechler for the
Bugzilla account and the advice) and attach the patches required to fix
them, but there might be more. The bug report [**] is somewhat relevant
here (though changing the default behaviour of system2() is obviously
not the right solution as it would break existing code).

Is there anything I should consider before creating the PR as
described above?

-- 
Best regards,
Ivan

[*] https://stat.ethz.ch/pipermail/r-help/2019-December/465075.html

[**] https://bugs.r-project.org/bugzilla/show_bug.cgi?id=16127

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] tempdir() containing spaces breaks installing source packages

[William Dunlap via R-devel]

You might expand the scope of this a bit to include Windows usernames with
non-ASCII characters in them.  If I recall correctly, if you are logged
under a Cyrillic UTF-8 name then R will not even start.  We have seen this
in the wild.

Bill Dunlap
TIBCO Software
wdunlap tibco.com


On Fri, Dec 13, 2019 at 8:47 AM Ivan Krylov  wrote:

> Hello everyone!
>
> Temp paths are used in system2() calls without shQuote() because
> they are assumed not to contain spaces. On Windows,
> GetShortPathName() is used to try to ensure that.
>
> Unfortunately, sometimes GetShortPathName() silently fails to return a
> 8.3 file path and gives a full path instead (8.3 file names might be
> disabled on newer Windows 10 installations, or there may be another
> reason). This has been spotted in the wild [*]. When %USERPROFILE%
> contains spaces, this results in tempdir() also containing spaces and
> prevents the user from being able to install source packages.
>
> As of ,
>
>  - src/library/utils/R/packages2.R line 839 contains an unquoted
>temporary file path (fil) passed to system2(), which results in it
>being split and R CMD INSTALL not being able to find the package
>file. In other invocations of R CMD INSTALL in the same file, the
>path is properly quoted.
>
>  - src/library/tools/R/check.R line 125 contains an unquoted temporary
>file path passed to system2, which results in Rterm.exe -f not being
>able to find the RtmpXX\Rin file, causing the attempt to
>run tools:::makeLazyLoading(...) to fail.
>
> I can report these two problems (thanks to Martin Maechler for the
> Bugzilla account and the advice) and attach the patches required to fix
> them, but there might be more. The bug report [**] is somewhat relevant
> here (though changing the default behaviour of system2() is obviously
> not the right solution as it would break existing code).
>
> Is there anything I should consider before creating the PR as
> described above?
>
> --
> Best regards,
> Ivan
>
> [*] https://stat.ethz.ch/pipermail/r-help/2019-December/465075.html
>
> [**] https://bugs.r-project.org/bugzilla/show_bug.cgi?id=16127
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>

[[alternative HTML version deleted]]

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] running R with users home dirs on a shared filesystems

[lejeczek via R-devel]

Hi guys,

I want to ask devel for who knows better - having multiple
nodes serving users home dirs off the same shared network
filesystem : are there any precautions or must-dos &
must-donts in order to assure healthy and efficient parallel
Rs running simultaneously - and I don't mean obvious stuff,
I'm rather asking about R's internals & environment.

simple example: three nodes mount a NFS share and users on
all three nodes run R simultaneously.

many thanks, L.

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Build failure on powerpc64

[Tom Callaway]

No, that does not change the issue:

arithmetic.c:180:26: error: initializer element is not constant
  180 | static LDOUBLE q_1_eps = 1.L / LDBL_EPSILON;

Tom

On Fri, Dec 13, 2019 at 11:56 AM Serguei Sokol 
wrote:

> Le 13/12/2019 à 17:06, Tom Callaway a écrit :
> > arithmetic.c:
> > static LDOUBLE q_1_eps = 1 / LDBL_EPSILON;
> Just a thought: can it be that it's "1" which is at the origin of
> compiler complaint?
> In this case, would the syntax "1.L" be sufficient to keep it calm?
>
> Serguei.
>
>

[[alternative HTML version deleted]]

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] running R with users home dirs on a shared filesystems

[Simon Urbanek]

User home is not used by R directly, so it is really up to whatever 
package/code may be using user home. In our setup we have all machines using 
NFS mounted homes for years. From experience the only thing to watch for are 
packages that use their own cache directories in $HOME instead of tempdir() - 
it is technically against CRAN policies but we have seen it in the wild.

Cheers,
Simon



> On Dec 13, 2019, at 1:36 PM, lejeczek via R-devel  
> wrote:
> 
> Hi guys,
> 
> I want to ask devel for who knows better - having multiple
> nodes serving users home dirs off the same shared network
> filesystem : are there any precautions or must-dos &
> must-donts in order to assure healthy and efficient parallel
> Rs running simultaneously - and I don't mean obvious stuff,
> I'm rather asking about R's internals & environment.
> 
> simple example: three nodes mount a NFS share and users on
> all three nodes run R simultaneously.
> 
> many thanks, L.
> 
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
> 

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] running R with users home dirs on a shared filesystems

[Cook, Malcolm]

Another thing to avoid are having multiple processes simultaneously access 
single sqlite3 database stored on NFS mount.

From sqlite manual: “Your best defense is to not use SQLite for files on a 
network filesystem”

So, if you configuring RStudio Server, make sure to follow advice about RStudio 
Package Manager: “This 
location must exist on local storage”

And any package that uses sqlite “under the hood” will similarly want the db on 
local storage to avoid such issues stemming from multi-process access.

Cheers,
Malcolm

From: R-devel  On Behalf Of Simon Urbanek
Sent: Friday, December 13, 2019 12:52 PM
To: lejeczek 
Cc: r-devel 
Subject: Re: [Rd] running R with users home dirs on a shared filesystems

CAUTION: This email was received from an External Source


User home is not used by R directly, so it is really up to whatever 
package/code may be using user home. In our setup we have all machines using 
NFS mounted homes for years. From experience the only thing to watch for are 
packages that use their own cache directories in $HOME instead of tempdir() - 
it is technically against CRAN policies but we have seen it in the wild.

Cheers,
Simon



> On Dec 13, 2019, at 1:36 PM, lejeczek via R-devel 
> mailto:r-devel@r-project.org>> wrote:
>
> Hi guys,
>
> I want to ask devel for who knows better - having multiple
> nodes serving users home dirs off the same shared network
> filesystem : are there any precautions or must-dos &
> must-donts in order to assure healthy and efficient parallel
> Rs running simultaneously - and I don't mean obvious stuff,
> I'm rather asking about R's internals & environment.
>
> simple example: three nodes mount a NFS share and users on
> all three nodes run R simultaneously.
>
> many thanks, L.
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[[alternative HTML version deleted]]

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel