Re: [Rd] OpenBLAS in everyday R?

[Ista Zahn]

On Jan 10, 2018 8:24 PM, "Benjamin Tyner"  wrote:

Thanks Keith. We checked, and indeed libopenblas is not linked against
libomp nor libgomp. We suspect this is because we used conda to install R
and OpenBLAS. So I guess we should be barking up the conda tree instead?


What are you barking about? I don't understand what you are trying to
accomplish.

By the way, I also noticed on my home machine (Ubuntu),
/usr/lib/libopenblas.so.0 is also not linked against those, for what that's
worth.

Regards,
Ben


On 01/10/2018 12:04 AM, Keith O'Hara wrote:

> Check if libopenblas is linked against libomp or libgomp.
>
> I’d be curious to see any errors that arise when an OpenMP version of
> OpenBLAS is linked with R.
>
> Keith
>
>
> On Jan 9, 2018, at 11:01 PM, Benjamin Tyner  wrote:
>>
>> I didn't do the compile; is there a way to check whether that was used?
>> If not, I'll inquire with our sysadmin and report back.
>>
>> In any case, my suggestion was motivated by the fact that some parts of R
>> use OpenMP while others do not, in the hope that the former could have
>> their OpenBLAS omelet without breaking the OpenMP eggs, so to speak.
>>
>>
>> On 01/09/2018 06:41 PM, Keith O'Hara wrote:
>>
>>> Do those issues still arise when OpenBLAS is compiled with USE_OPENMP=1 ?
>>>
>>> Keith
>>>
>>> On Jan 9, 2018, at 6:03 PM, Benjamin Tyner  wrote:

 Please pardon my ignorance, but doesn't OpenBLAS still not always play
 nicely with multi-threaded OpenMP? (for example, don't race conditions
 sometimes crop up)? If so, it might be nice to have the ability to
 temporarily disable multi-threaded OpenMP (effectively:
 omp_set_num_threads(1)) for the duration of operations using OpenBLAS.

 Regards
 Ben

 Julia using OpenBLAS is *very* reassuring.
>
> I agree that having it included as an options(...) feature should be
> OK.
>
> On Sun, Dec 17, 2017, 3:22 PM Juan Telleria  https://stat.ethz.ch/mailman/listinfo/r-devel>> wrote:
>
> /Julia Programming Language uses also OpenBlas, and it is actively
>> />/maintained with bugs being fixed as I have checked it out: />//>/
>> http://www.openblas.net/Changelog.txt />//>/So I still see it ok to
>> be included as an options(...) feature (by default />/off, just for
>> safety), over other Blas libraries. />//>/R could not use Intel MKL for
>> legal reasons (I think), because as long />/that R ships with GPL
>> libraries, shipping R by default with Non-GPL is />/illegal. 
>> />//>/Cheers,
>> />/Juan />//>/El 17/12/2017 2:50 a. m., "Avraham Adler" > gmail.com >
>> />/escribió: />//>>/On Sat, Dec 16, 2017 at 7:41 PM, Kenny Bell > at gmail.com > wrote:
>> />>/> It seems like many of the multi-threaded BLASes have some sort of
>> />>/> fundamental problem preventing use in the way Juan suggests: />>/>
>> />>/> - Dirk's vignette states that ATLAS "fixes the number of cores used
>> at />>/> compile-time and cannot vary this setting at run-time", so any
>> />>/> user-friendly implementation for R would have to compile ATLAS for
>> 1-16 />>/> threads to allow the user to switch at run-time. This might
>> dramatically />>/> affect install times. />>/> />>/> - MKL seems like 
>> it's
>> been outright rejected in the past based on not />>/> being 
>> "free-enough".
>> />>/> />>/> - OpenBLAS causes crashes. />>/> />>/> Has anyone tried 
>> ExBLAS
>> for use with R? />>/> />>/> On Sun, Dec 17, 2017 at 1:03 PM, Peter
>> Langfelder < />>/> peter.langfelder at gmail.com <
>> https://stat.ethz.ch/mailman/listinfo/r-devel>> wrote: />>/> />>/>>
>> I would be very cautious about OpenBLAS in particular... from time to
>> />>/>> time I get complains from users that compiled code calculations in
>> my />>/>> WGCNA package crash or produce wrong answers with large data, 
>> and
>> they />>/>> all come from OpenBLAS users. I am yet to reproduce any of
>> their />>/>> crashes when using MKL and ATLAS BLAS implementations. 
>> />>/>>
>> />>/>> Just my 2 cents... />>/>> />>/>> Peter />>//>>/I've been building 
>> R
>> on Windows 64 bit with OpenBLAS for years and my />>/builds pass
>> check-devel. For a while in the past it failed one check />>/as the
>> tolerance was 5e-5 and with my build of OpenBLAS the error was />>/5.4e-5
>> or 5.7e-5, but that was changed around R 3.3, if I recall />>/correctly. 
>> I
>> provide descriptions here [1], but I haven't gone so far />>/as to post
>> compiled Rblas.dlls just yet. My personal build sets 4 />>/threads when
>> compiling OpenBLAS itself as I'm currently on a quad-core 
>> />>/SandyBridge.
>> In tests I ran a few years ago, both single and multi />>/threaded BLAS
>> compile and then can be compiled into R with no issues />

Re: [Rd] R CMD build then check fails on R-devel due to serialization version

[Jim Hester]

This change poses difficulties for automated build systems such as
travis-ci, which is widely used in the R community. In particular
because this is a WARNING and not a NOTE this causes all R-devel
builds with vignettes to fail, as the default settings fail the build
if R CMD check issues a WARNING.

The simplest change would be for R-core to change this message to be a
NOTE rather than a WARNING, the serialization could still be tested
and there would be a check against vignettes built with R-devel, but
it would not cause these builds to fail.

On Wed, Jan 10, 2018 at 3:52 PM, Duncan Murdoch
 wrote:
> On 10/01/2018 1:26 PM, Neal Richardson wrote:
>>
>> Hi,
>> Since yesterday I'm seeing `R CMD check --as-cran` failures on the
>> R-devel daily build (specifically, R Under development (unstable)
>> (2018-01-09 r74100)) for multiple packages:
>>
>> * checking serialized R objects in the sources ... WARNING
>> Found file(s) with version 3 serialization:
>> ‘build/vignette.rds’
>> Such files are only readable in R >= 3.5.0.
>> Recreate them with R < 3.5.0 or save(version = 2) or saveRDS(version =
>> 2) as appropriate
>>
>> As far as I can tell, revision 74099
>>
>> (https://github.com/wch/r-source/commit/d9530001046a582ff6a43ca834d6c3586abd0a97),
>> which changes the default serialization format to 3, clashes with
>> revision 73973
>> (https://github.com/wch/r-source/commit/885764eb74f2211a547b13727f2ecc5470c3dd00),
>> which checks that serialized R objects are _not_ version 3. It seems
>> that with the current development version of R, if you `R CMD build`
>> and then run `R CMD check --as-cran` on the built package, it will
>> fail.
>>
>
> I think the message basically says:  don't do that.  You should build with
> R-release for now.  You always need to check with R-devel, so life is
> complicated.
>
> If you build with R-devel without forcing the old format, nobody using
> R-release will be able to use your tarball.
>
> Eventually I guess the new format will be accepted by CRAN, but it will
> likely be a while:  nobody expects everyone to instantly upgrade to a new R
> release, let alone to an unreleased development version.
>
> Presumably that particular file (build/vignette.rds) could be automatically
> built in the old format for now, but the new format needs testing, so it
> makes sense to me to leave it as a default, even if it makes it more
> complicated to submit a package to CRAN.
>
> Duncan Murdoch
>
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] R CMD build then check fails on R-devel due to serialization version

[luke-tierney]

As things stand now, package tarballs with vignettes that are built
with R-devel will not install in R 3.4.x, so CRAN can't accept them
and someone running R CMD check --as-cran should be told that. A
WARNING is appropriate.

Most likely what will change soon is that build/version.rds will be
saved with serialization version = 2 and this warning will not be
triggered just by having a vignette. It will still be triggered by
data files serialized with R-devel's default version = 3.

Please do remember that the 'devel' in R-devel means exactly that:
things will at times be unstable. There are currently a lot of balls
flying around with changes in R-devel and also Biocontuctor, and the
CRAN maintainers are working hard to keep things all up in the
air. Please be patient.

Best,

luke

On Thu, 11 Jan 2018, Jim Hester wrote:


This change poses difficulties for automated build systems such as
travis-ci, which is widely used in the R community. In particular
because this is a WARNING and not a NOTE this causes all R-devel
builds with vignettes to fail, as the default settings fail the build
if R CMD check issues a WARNING.

The simplest change would be for R-core to change this message to be a
NOTE rather than a WARNING, the serialization could still be tested
and there would be a check against vignettes built with R-devel, but
it would not cause these builds to fail.

On Wed, Jan 10, 2018 at 3:52 PM, Duncan Murdoch
 wrote:

On 10/01/2018 1:26 PM, Neal Richardson wrote:


Hi,
Since yesterday I'm seeing `R CMD check --as-cran` failures on the
R-devel daily build (specifically, R Under development (unstable)
(2018-01-09 r74100)) for multiple packages:

* checking serialized R objects in the sources ... WARNING
Found file(s) with version 3 serialization:
‘build/vignette.rds’
Such files are only readable in R >= 3.5.0.
Recreate them with R < 3.5.0 or save(version = 2) or saveRDS(version =
2) as appropriate

As far as I can tell, revision 74099

(https://github.com/wch/r-source/commit/d9530001046a582ff6a43ca834d6c3586abd0a97),
which changes the default serialization format to 3, clashes with
revision 73973
(https://github.com/wch/r-source/commit/885764eb74f2211a547b13727f2ecc5470c3dd00),
which checks that serialized R objects are _not_ version 3. It seems
that with the current development version of R, if you `R CMD build`
and then run `R CMD check --as-cran` on the built package, it will
fail.



I think the message basically says:  don't do that.  You should build with
R-release for now.  You always need to check with R-devel, so life is
complicated.

If you build with R-devel without forcing the old format, nobody using
R-release will be able to use your tarball.

Eventually I guess the new format will be accepted by CRAN, but it will
likely be a while:  nobody expects everyone to instantly upgrade to a new R
release, let alone to an unreleased development version.

Presumably that particular file (build/vignette.rds) could be automatically
built in the old format for now, but the new format needs testing, so it
makes sense to me to leave it as a default, even if it makes it more
complicated to submit a package to CRAN.

Duncan Murdoch


__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel


__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel


--
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa  Phone: 319-335-3386
Department of Statistics andFax:   319-335-3017
   Actuarial Science
241 Schaeffer Hall  email:   luke-tier...@uiowa.edu
Iowa City, IA 52242 WWW:  http://www.stat.uiowa.edu
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] R-devel Digest, Vol 179, Issue 7

[Duncan Murdoch]

On 11/01/2018 7:17 AM, Therneau, Terry M., Ph.D. wrote:

This is not nice.  I have easy access to the "institutional" version of R on the
department servers, which do not track R-release all that fast (3-12 month 
delay, affects
300+ users, and at the back end of a formalized IT process), and a personal 
machine on
which I track R-devel, the latter at the behest of CRAN.  Now you are asking 
that I track
yet another R version just for the sake of the R CMD BUILD script?   There are 
other ways
to test the new serialization code than this single file in the tarball.


I'm not asking for anything.  I'm guessing at an explanation for what 
CRAN is asking.  I'm not part of CRAN or R Core.


But it seems like 2 versions should be sufficient:  build on the 
institutional version, test on your personal machine (or on one of the 
test services like WinBuilder, R-forge, R-hub, etc.).


Duncan Murdoch



Terry T.


On 01/11/2018 05:00 AM, r-devel-requ...@r-project.org wrote:

I think the message basically says:  don't do that.  You should build
with R-release for now.  You always need to check with R-devel, so life
is complicated.

If you build with R-devel without forcing the old format, nobody using
R-release will be able to use your tarball.

Eventually I guess the new format will be accepted by CRAN, but it will
likely be a while:  nobody expects everyone to instantly upgrade to a
new R release, let alone to an unreleased development version.

Presumably that particular file (build/vignette.rds) could be
automatically built in the old format for now, but the new format needs
testing, so it makes sense to me to leave it as a default, even if it
makes it more complicated to submit a package to CRAN.

Duncan Murdoch





__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] OpenBLAS in everyday R?

[Keith O'Hara]

I’m not really familiar with conda, but if they’re being packaged together then 
an omp build might be more appropriate. 

Perhaps another point for Juan’s list: whether OpenBLAS is the right choice to 
pair with. The library itself hasn’t produced optimized kernels for any of the 
Intel *Lake chips yet; might be worth considering its near- and long-term 
future (vs something else).

Keith

> On Jan 10, 2018, at 8:24 PM, Benjamin Tyner  wrote:
> 
> Thanks Keith. We checked, and indeed libopenblas is not linked against libomp 
> nor libgomp. We suspect this is because we used conda to install R and 
> OpenBLAS. So I guess we should be barking up the conda tree instead?
> 
> By the way, I also noticed on my home machine (Ubuntu), 
> /usr/lib/libopenblas.so.0 is also not linked against those, for what that's 
> worth.
> 
> Regards,
> Ben
> 
> On 01/10/2018 12:04 AM, Keith O'Hara wrote:
>> Check if libopenblas is linked against libomp or libgomp.
>> 
>> I’d be curious to see any errors that arise when an OpenMP version of 
>> OpenBLAS is linked with R.
>> 
>> Keith
>> 
>> 
>>> On Jan 9, 2018, at 11:01 PM, Benjamin Tyner  wrote:
>>> 
>>> I didn't do the compile; is there a way to check whether that was used? If 
>>> not, I'll inquire with our sysadmin and report back.
>>> 
>>> In any case, my suggestion was motivated by the fact that some parts of R 
>>> use OpenMP while others do not, in the hope that the former could have 
>>> their OpenBLAS omelet without breaking the OpenMP eggs, so to speak.
>>> 
>>> 
>>> On 01/09/2018 06:41 PM, Keith O'Hara wrote:
 Do those issues still arise when OpenBLAS is compiled with USE_OPENMP=1 ?
 
 Keith
 
> On Jan 9, 2018, at 6:03 PM, Benjamin Tyner  wrote:
> 
> Please pardon my ignorance, but doesn't OpenBLAS still not always play 
> nicely with multi-threaded OpenMP? (for example, don't race conditions 
> sometimes crop up)? If so, it might be nice to have the ability to 
> temporarily disable multi-threaded OpenMP (effectively: 
> omp_set_num_threads(1)) for the duration of operations using OpenBLAS.
> 
> Regards
> Ben
> 
>> Julia using OpenBLAS is *very* reassuring.
>> 
>> I agree that having it included as an options(...) feature should be OK.
>> 
>> On Sun, Dec 17, 2017, 3:22 PM Juan Telleria > > wrote:
>> 
>>> /Julia Programming Language uses also OpenBlas, and it is actively 
>>> />/maintained with bugs being fixed as I have checked it out: 
>>> />//>/http://www.openblas.net/Changelog.txt />//>/So I still see it ok 
>>> to be included as an options(...) feature (by default />/off, just for 
>>> safety), over other Blas libraries. />//>/R could not use Intel MKL for 
>>> legal reasons (I think), because as long />/that R ships with GPL 
>>> libraries, shipping R by default with Non-GPL is />/illegal. 
>>> />//>/Cheers, />/Juan />//>/El 17/12/2017 2:50 a. m., "Avraham Adler" 
>>> >> > />/escribió: />//>>/On 
>>> Sat, Dec 16, 2017 at 7:41 PM, Kenny Bell >> > wrote: />>/> It seems 
>>> like many of the multi-threaded BLASes have some sort of />>/> 
>>> fundamental problem preventing use in the way Juan suggests: />>/> 
>>> />>/> - Dirk's vignette states that ATLAS "fixes the number of cores 
>>> used at />>/> compile-time and cannot vary this setting at run-time", 
>>> so any />>/> user-friendly implementation for R would have to compile 
>>> ATLAS for 1-16 />>/> threads to allow the user to switch at run-time. 
>>> This might dramatically />>/> affect install times. />>/> />>/> - MKL 
>>> seems like it's been outright rejected in the past based on not />>/> 
>>> being "free-enough". />>/> />>/> - OpenBLAS causes crashes. />>/> />>/> 
>>> Has anyone tried ExBLAS for use with R? />>/> />>/> On Sun, Dec 17, 
>>> 2017 at 1:03 PM, Peter Langfelder < />>/> peter.langfelder at gmail.com 
>>> > wrote: />>/> />>/>> I 
>>> would be very cautious about OpenBLAS in particular... from time to 
>>> />>/>> time I get complains from users that compiled code calculations 
>>> in my />>/>> WGCNA package crash or produce wrong answers with large 
>>> data, and they />>/>> all come from OpenBLAS users. I am yet to 
>>> reproduce any of their />>/>> crashes when using MKL and ATLAS BLAS 
>>> implementations. />>/>> />>/>> Just my 2 cents... />>/>> />>/>> Peter 
>>> />>//>>/I've been building R on Windows 64 bit with OpenBLAS for years 
>>> and my />>/builds pass check-devel. For a while in the past it failed 
>>> one check />>/as the tolerance was 5e-5 and with my build of OpenBLAS 
>>> the error was />>/5.4e-5 or 5.7e-5, but that was changed around R 3.3, 
>>> if I recall />>/correctly

Re: [Rd] OpenBLAS in everyday R?

[Avraham Adler]

On Thu, Jan 11, 2018 at 10:38 AM, Keith O'Hara  wrote:

[snip]
>
> Perhaps another point for Juan’s list: whether OpenBLAS is the right choice 
> to pair with. The library itself hasn’t produced optimized kernels for any of 
> the Intel *Lake chips yet; might be worth considering its near- and long-term 
> future (vs something else).
>

Regarding this point, please see this thread on OpenBLAS-users [1], in
specific the third post by Jeff Hammond who says he is an employee of
Intel, where he says:

 "Skylake Xeon processors with AVX-512 are definitely going to require
code changes to perform optimally. However, the Core i[357] processors
up to and including Kaby Lake do not support AVX-512 and thus can
suffice with the existing AVX2 implementation that targets Haswell.
See https://ark.intel.com/#@Processors and look for "Instruction Set
Extensions" for full details on on specific SKUs."

I presume this holds for CoffeeLake as well, as it's pretty much a
hexacore SkyLake.

Thank you,

Avi

[1] https://groups.google.com/d/msg/openblas-users/XU6-9h-geVE/mwz2ewKrCQAJ

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] OpenBLAS in everyday R?

[Benjamin Tyner]

True or False: when USE_OPENMP=1 is not used, then race conditions are 
not unexpected.


If True, and we wish to avoid race conditions, then sources such as the 
conda channel and ubuntu would need to add this enhancement.


If False, then what is the next step (i.e. forum) for debugging the race 
condition?



On 01/11/2018 07:56 AM, Ista Zahn wrote:



On Jan 10, 2018 8:24 PM, "Benjamin Tyner" > wrote:


Thanks Keith. We checked, and indeed libopenblas is not linked
against libomp nor libgomp. We suspect this is because we used
conda to install R and OpenBLAS. So I guess we should be barking
up the conda tree instead?


What are you barking about? I don't understand what you are trying to 
accomplish.


By the way, I also noticed on my home machine (Ubuntu),
/usr/lib/libopenblas.so.0 is also not linked against those, for
what that's worth.

Regards,
Ben


On 01/10/2018 12:04 AM, Keith O'Hara wrote:

Check if libopenblas is linked against libomp or libgomp.

I’d be curious to see any errors that arise when an OpenMP
version of OpenBLAS is linked with R.

Keith


On Jan 9, 2018, at 11:01 PM, Benjamin Tyner
mailto:bty...@gmail.com>> wrote:

I didn't do the compile; is there a way to check whether
that was used? If not, I'll inquire with our sysadmin and
report back.

In any case, my suggestion was motivated by the fact that
some parts of R use OpenMP while others do not, in the
hope that the former could have their OpenBLAS omelet
without breaking the OpenMP eggs, so to speak.


On 01/09/2018 06:41 PM, Keith O'Hara wrote:

Do those issues still arise when OpenBLAS is compiled
with USE_OPENMP=1 ?

Keith

On Jan 9, 2018, at 6:03 PM, Benjamin Tyner
mailto:bty...@gmail.com>> wrote:

Please pardon my ignorance, but doesn't OpenBLAS
still not always play nicely with multi-threaded
OpenMP? (for example, don't race conditions
sometimes crop up)? If so, it might be nice to
have the ability to temporarily disable
multi-threaded OpenMP (effectively:
omp_set_num_threads(1)) for the duration of
operations using OpenBLAS.

Regards
Ben

Julia using OpenBLAS is *very* reassuring.

I agree that having it included as an
options(...) feature should be OK.

On Sun, Dec 17, 2017, 3:22 PM Juan Telleria
http://gmail.com>
>>
wrote:

/Julia Programming Language uses also
OpenBlas, and it is actively />/maintained
with bugs being fixed as I have checked it
out:
/>//>/http://www.openblas.net/Changelog.txt

/>//>/So I still see it ok to be included
as an options(...) feature (by default
/>/off, just for safety), over other Blas
libraries. />//>/R could not use Intel MKL
for legal reasons (I think), because as
long />/that R ships with GPL libraries,
shipping R by default with Non-GPL is
/>/illegal. />//>/Cheers, />/Juan />//>/El
17/12/2017 2:50 a. m., "Avraham Adler"
http://gmail.com>
>>
/>/escribió: />//>>/On Sat, Dec 16, 2017
at 7:41 PM, Kenny Bell http://gmail.com>
>>
wrote: />>/> It seems like many of the
multi-threaded BLASes have some sort of
/>>/> fundamental problem preventing use
in the way Juan suggests: />>/> />>/> -
Dirk's vignette states that ATLAS "fixes
the number of cores used at />>/>
compile-time and cannot vary this sett