[Rd] linux multi-threaded compilation is running only on one processor
Hello,
I compiled R-2.15.2 with linux intel compilers (see below).
when I execute some R code on a // 4 proc x 4 cores // server
(export MKL_NUM_THREADS = 16)
very often I have the situations where only one processor (4 cores) is
active instead of 4 (16 cores) !
do you know this fact depends on what ?
thank you !
P.S.
many functions of R are single-threaded,
is there a simple way to make them multi-threaded ?
=
CMP_LIB_PATH=/opt/intel/lib/intel64
MKL_LIB_PATH=/opt/intel/composerxe/mkl/lib/intel64
INC_LIB_PATH=/usr/local/lib
export CC="icc -std=c99"
export CFLAGS="-g -O3 -wd188 -ip"
export F77=ifort
export FFLAGS="-g -O3"
export CXX=icpc
export CXXFLAGS="-g -O3"
export FC=ifort
export FCFLAGS="-g -O3"
export SHLIB_CXXLD=icpc
export
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${MKL_LIB_PATH}:${INC_LIB_PATH}:${CMP_LIB_PATH}
export
LDFLAGS="-L${MKL_LIB_PATH},-L${INC_LIB_PATH},-L${CMP_LIB_PATH},-Bdirect,--hash-style=both,-Wl,-O1"
export SHLIB_LDFLAGS="-lpthread"
export SHLIB_CXXLDFLAGS="-lpthread"
export MAIN_LDFLAGS="-lpthread"
MKL="-L${MKL_LIB_PATH} -L${INC_LIB_PATH} -lmkl_lapack95_lp64
-lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread"
./configure --prefix=${INSTALL_DIR} --enable-R-shlib
--with-blas="$MKL" --with-lapack
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] linux multi-threaded compilation is running only on one processor
On Dec 28, 2012, at 5:23 AM, Vitaliy FEOKTISTOV wrote:
> Hello,
>
> I compiled R-2.15.2 with linux intel compilers (see below).
>
> when I execute some R code on a // 4 proc x 4 cores // server
> (export MKL_NUM_THREADS = 16)
> very often I have the situations where only one processor (4 cores) is
> active instead of 4 (16 cores) !
>
> do you know this fact depends on what ?
>
What code? Usually, this has nothing to do with R but the libraries you're
using and their capability to implicitly parallelize the calls used (presumably
MKL is your case?). You didn't provide any details on what you are actually
doing, but really you should be asking on the lists for the libraries you are
using.
> thank you !
>
> P.S.
> many functions of R are single-threaded, is there a simple way to make them
> multi-threaded ?
>
R is not thread-safe, so any parallelization has to be done at native code
level. So if "functions of R" is referring to R code then the answer is no
(except for the obvious use parallel/multicore where applicable), if it is
referring to native functions then the answer is possibly - Luke Tierney has
done some of that work -- did you try pnmath?
Cheers,
Simon
>
> =
>
> CMP_LIB_PATH=/opt/intel/lib/intel64
> MKL_LIB_PATH=/opt/intel/composerxe/mkl/lib/intel64
> INC_LIB_PATH=/usr/local/lib
>
> export CC="icc -std=c99"
> export CFLAGS="-g -O3 -wd188 -ip"
> export F77=ifort
> export FFLAGS="-g -O3"
> export CXX=icpc
> export CXXFLAGS="-g -O3"
> export FC=ifort
> export FCFLAGS="-g -O3"
> export SHLIB_CXXLD=icpc
>
> export
> LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${MKL_LIB_PATH}:${INC_LIB_PATH}:${CMP_LIB_PATH}
> export
> LDFLAGS="-L${MKL_LIB_PATH},-L${INC_LIB_PATH},-L${CMP_LIB_PATH},-Bdirect,--hash-style=both,-Wl,-O1"
>
> export SHLIB_LDFLAGS="-lpthread"
> export SHLIB_CXXLDFLAGS="-lpthread"
> export MAIN_LDFLAGS="-lpthread"
>
> MKL="-L${MKL_LIB_PATH} -L${INC_LIB_PATH} -lmkl_lapack95_lp64
> -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -liomp5 -lpthread"
> ./configure --prefix=${INSTALL_DIR} --enable-R-shlib
> --with-blas="$MKL" --with-lapack
>
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
>
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] How to ensure -O3 on Win64
On 28.12.2012 00:41, Simon Urbanek wrote: On Dec 27, 2012, at 6:08 PM, Matthew Dowle wrote: On 27.12.2012 17:53, Simon Urbanek wrote: On Dec 23, 2012, at 9:22 PM, Matthew Dowle wrote: Hi, Similar questions have come up before on the list and elsewhere but I haven't found a solution yet. winbuilder's install.out shows data.table's .c files compiled with -O3 on Win32 but -O2 on Win64. The same happens on R-Forge. I gather that some packages don't work with -O3 so the default is -O2. I've tried this in data.table's Makevars (entire contents) : MAKEFLAGS="CFLAGS=-O3"# added CFLAGS=-O3# added PKG_CFLAGS=-O3# added all: $(SHLIB) # no change mv $(SHLIB) datatable$(SHLIB_EXT) # no change but -O2 still appears in winbuilder's install.out (after -O3, and I believe the last -O is the one that counts) : gcc -m64 -I"D:/RCompile/recent/R-2.15.2/include" -DNDEBUG -I"d:/Rcompile/CRANpkg/extralibs215/local215/include" -O3 -O2 -Wall -std=gnu99 -mtune=core2 -c dogroups.c -o dogroups.o How can I ensure that data.table is compiled with -O3 on Win64? You can't - at least not in a way that doesn't circumvent the R build system. Also it's not portable so you don't want to mess with optimization flags and hard-code it in your package as it's user's choice how they setup R and its flags. You can certainly setup your R to compile with -O3, you just can't impose that on others. Cheers, Simon Thanks Simon. This makes complete sense where users compile packages on install (Unix and Mac, and I better check my settings then), but on Windows where it's more common for the user to install the pre-compiled .zip from CRAN is my concern. This came up because the new fread function in data.table wasn't showing as much of a speedup on Win64 as on Linux. I'm not 100% sure that non -O3 is the cause, but there are some function calls which get iterated a lot (e.g. isspace) and I'd seen that inlining was something -O3 did and not -O2. In general, why wouldn't a user of a package want the best performance from -O3? Because it doesn't work? I don't know, you said yourself that -O2 may be there since -O3 breaks - that was not the question, though. (If you are curious about that, ask on CRAN, I don't remember the answer -- note that Win64 compiler support is relatively recent). Indeed I had forgotten how recent that was. Ok, this is clicking now. By non portable do you mean the executable produced by winbuilder (or by CRAN) might not run on all Windows machines it's installed on (because -O3 (over) optimizes for the machine it's built on), or do you mean that -O3 itself might not be available on some compilers (and if so which compilers don't have -O3?). Non-portable as in -O3 may not be supported or may break (we have seen -O3 trigger bugs in gcc before). If you hard-code it, there is no way around it. The point is that you cannot make decisions for the user in advance, because you don't know the setup the user may use. I agree that Windows is a bit of a special-case in that there are very few choices so the risk of breaking things is lower, but if -O2 is really such a big deal, it is not just your problem and so you may want to investigate it further. Ok thanks a lot for info. I'll try a few more things and follow up off r-devel if need be. Matthew __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
