Re: [Rd] Lubuntu 13.04 raring ringtail: Problems installing pnmath_0.0-4

2013-04-27 Thread Prof Brian Ripley 
AFAIK pnmath is not a package on a public repository, and it seems it is 
using R internals not part of the API and not exposed in R 3.0.0.


I presume you got this from ~luke/R/experimental ... the clue is in the 
directory name.  My understanding was that this would eventually be 
merged into R rather than released as a package.


If you really want to use it you will need to modify the R sources to 
unhide PRIMFUN and SET_PRIMFUN and re-compile.



On 26/04/2013 19:39, Kjetil Halvorsen wrote:

Hola!

This is 64bit Lubuntu 13.04, with R-3.0.0 installed from
deb http://ppa.launchpad.net/marutter/rrutter/ubuntu raring  main
via synaptic.  gcc --version
gcc (Ubuntu/Linaro 4.7.3-1ubuntu1) 4.7.3


sessionInfo()

R version 3.0.0 (2013-04-03)
Platform: x86_64-pc-linux-gnu (64-bit)

locale:
  [1] LC_CTYPE=en_US.UTF-8   LC_NUMERIC=C
  [3] LC_TIME=en_US.UTF-8LC_COLLATE=en_US.UTF-8
  [5] LC_MONETARY=en_US.UTF-8LC_MESSAGES=en_US.UTF-8
  [7] LC_PAPER=C LC_NAME=C
  [9] LC_ADDRESS=C   LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats graphics  grDevices utils datasets  methods   base

other attached packages:
[1] MASS_7.3-26

loaded via a namespace (and not attached):
[1] compiler_3.0.0 fortunes_1.5-0 tools_3.0.0



I am trying to install pnmath from the commandline (outside R), this is
what happens:
kjetil@kjetil-HP-Pavilion-dv4-Notebook-PC:~/R/src$ sudo  R CMD INSTALL
pnmath_0.0-4.tar.gz
* installing to library ‘/usr/local/lib/R/site-library’
* installing *source* package ‘pnmath’ ...
** libs
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -fopenmp -fpic  -O2
-pipe -g  -c bd0.c -o bd0.o
...
...
...
installing to /usr/local/lib/R/site-library/pnmath/libs
** R
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
Error in dyn.load(file, DLLpath = DLLpath, ...) :
   unable to load shared object
'/usr/local/lib/R/site-library/pnmath/libs/pnmath.so':
   /usr/local/lib/R/site-library/pnmath/libs/pnmath.so: undefined symbol:
PRIMFUN
Error: loading failed
Execution halted
ERROR: loading failed
* removing ‘/usr/local/lib/R/site-library/pnmath’

I have installed sufficient libraries as to compile R itself from source
without problems!

Kjetil

[[alternative HTML version deleted]]



__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel




--
Brian D. Ripley,  rip...@stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford, Tel:  +44 1865 272861 (self)
1 South Parks Road, +44 1865 272866 (PA)
Oxford OX1 3TG, UKFax:  +44 1865 272595

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

[Rd] Speeding up build-from-source

2013-04-27 Thread Adam Seering 

Hi,
	I've been playing around with the R source code a little; mostly just 
trying to familiarize myself.  I have access to some computers on a 
reservation system; so I've been reserving a computer, downloading and 
compiling R, and going from there.


	I'm finding that R takes a long time to build, though.  (Well, ok, 
maybe 5 minutes -- I'm impatient :-) )  Most of that time, it's sitting 
there byte-compiling some internal package or another, which uses just 
one CPU core so leaves the system mostly idle.


I'm just curious if anyone has thought about parallelizing that process?

Thanks,
Adam

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Verbose output from R CMD check

2013-04-27 Thread dpleydell 
Following Simon's suggestion of replacing printf with Rprintf throughout 
the source code R CMD check now complains less than before, but it still 
complains...

I used the following two BASH lines to replace printf

find . -maxdepth 1 -type f -name '*.c' -exec sed -i 's\printf\Rprintf\' 
{} \;
find . -maxdepth 1 -type f -name '*.c' -exec sed -i 
's\fRprintf\fprintf\' {} \;

and using grep I have confirmed that only fprintf and Rprintf remain in 
the code.

the original warning (when printf was in the code) was

[QUOTE : ORIGINAL R CMD check WARNING]

R CMD check foobar_1.7.5.tar.gz
[LOTS OF '...OK' CHECK REPORTS REMOVED FOR BREVITY]
* checking compiled code ... NOTE
File '/home/david/foobar/package/foobar.Rcheck/foobar/libs/foobar.so':
  Found 'putchar', possibly from 'putchar' (C)
Object: 'foobar.o'
  Found 'puts', possibly from 'printf' (C), 'puts' (C)
Object: 'foobar.o'

Compiled code should not call functions which might terminate R nor write to 
stdout/stderr instead of to the console.

[end QUOTE : ORIGINAL R CMD check WARNING]


now printf is no longer in the code the warning message is

[QUOTE : ORIGINAL R CMD check WARNING]

R CMD check foobar_1.7.6.tar.gz
[LOTS OF '...OK' CHECK REPORTS REMOVED FOR BREVITY]
* checking compiled code ... NOTE
File '/home/david/foobar/package/foobar.Rcheck/foobar/libs/foobar.so':
  Found 'puts', possibly from 'printf' (C), 'puts' (C)
Object: 'foobar.o'

Compiled code should not call functions which might terminate R nor write to 
stdout/stderr instead of to the console.

[end QUOTE : ORIGINAL R CMD check WARNING]

Which is better than before, but still not clean.

Both the following lines return 0 hits
grep printf *.c|grep -v Rprintf|grep -v fprintf
grep puts *.c

Is there some way to detect what has caused this warning about puts? 
It's not a call to 'puts' or 'printf' because there are none.

many thanks
David


-- 
David Pleydell,
INRA,
UMR-1351 CMAEE,
Domaine Duclos,
Prise D'eau,
97122 Petit Bourg,
Guadeloupe
dpleyd...@antilles.inra.fr
pleyd...@cirad.fr
Tel +33 5 90 25 54 42
Fax +33 5 90 94 03 96


[[alternative HTML version deleted]]

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Speeding up build-from-source

2013-04-27 Thread Martin Morgan 

On 04/26/2013 07:50 AM, Adam Seering wrote:

Hi,
 I've been playing around with the R source code a little; mostly just
trying to familiarize myself.  I have access to some computers on a reservation
system; so I've been reserving a computer, downloading and compiling R, and
going from there.

 I'm finding that R takes a long time to build, though.  (Well, ok, maybe 5
minutes -- I'm impatient :-) )  Most of that time, it's sitting there
byte-compiling some internal package or another, which uses just one CPU core so
leaves the system mostly idle.

 I'm just curious if anyone has thought about parallelizing that process?


Hi Adam -- parallel builds are supported by adding the '-j' flag when you invoke 
make


  make -j

The packages are being built in parallel, in as much as this is possible by 
their dependency structure. Also, you can configure without byte compilation, 
see ~/src/R-devel/configure --help to make this part of the build go more 
quickly. And after an initial build subsets of R, e.g., just the 'main' source 
or a single package like 'stats', can be built with (assuming R's source, e.g., 
from svn, is in ~/src/R-devel, and you're building R in ~/bin/R-devel) with


  cd ~/bin/R-devel/src/main
  make -j
  cd ~/bin/R-devel/src/library/stats
  make -j

The definitive source for answers to questions like these is

  > RShowDoc("R-admin")

Martin




Thanks,
Adam

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel



--
Computational Biology / Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N.
PO Box 19024 Seattle, WA 98109

Location: Arnold Building M1 B861
Phone: (206) 667-2793

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Verbose output from R CMD check

2013-04-27 Thread peter dalgaard 

On Apr 25, 2013, at 22:01 , dpleydell wrote:

> Following Simon's suggestion of replacing printf with Rprintf throughout 
> the source code R CMD check now complains less than before, but it still 
> complains...
> 
> I used the following two BASH lines to replace printf
> 
> find . -maxdepth 1 -type f -name '*.c' -exec sed -i 's\printf\Rprintf\' 
> {} \;
> find . -maxdepth 1 -type f -name '*.c' -exec sed -i 
> 's\fRprintf\fprintf\' {} \;
> 
> and using grep I have confirmed that only fprintf and Rprintf remain in 
> the code.
> 
> the original warning (when printf was in the code) was
> 
> [QUOTE : ORIGINAL R CMD check WARNING]
> 
> R CMD check foobar_1.7.5.tar.gz
> [LOTS OF '...OK' CHECK REPORTS REMOVED FOR BREVITY]
> * checking compiled code ... NOTE
> File '/home/david/foobar/package/foobar.Rcheck/foobar/libs/foobar.so':
>  Found 'putchar', possibly from 'putchar' (C)
>Object: 'foobar.o'
>  Found 'puts', possibly from 'printf' (C), 'puts' (C)
>Object: 'foobar.o'
> 
> Compiled code should not call functions which might terminate R nor write to 
> stdout/stderr instead of to the console.
> 
> [end QUOTE : ORIGINAL R CMD check WARNING]
> 
> 
> now printf is no longer in the code the warning message is
> 
> [QUOTE : ORIGINAL R CMD check WARNING]
> 
> R CMD check foobar_1.7.6.tar.gz
> [LOTS OF '...OK' CHECK REPORTS REMOVED FOR BREVITY]
> * checking compiled code ... NOTE
> File '/home/david/foobar/package/foobar.Rcheck/foobar/libs/foobar.so':
>  Found 'puts', possibly from 'printf' (C), 'puts' (C)
>Object: 'foobar.o'
> 
> Compiled code should not call functions which might terminate R nor write to 
> stdout/stderr instead of to the console.
> 
> [end QUOTE : ORIGINAL R CMD check WARNING]
> 
> Which is better than before, but still not clean.
> 
> Both the following lines return 0 hits
> grep printf *.c|grep -v Rprintf|grep -v fprintf
> grep puts *.c
> 
> Is there some way to detect what has caused this warning about puts? 
> It's not a call to 'puts' or 'printf' because there are none.

Longshot: 

grep '[^Rf]printf' *.c

might turn up cases not found by the pipeline above.

Apart from that, trying out the nm command on your .o and .so files could be 
enlightening.  If push comes to shove, you might compile to assembler code (gcc 
-S, as far as I remember) and look for the offending puts symbol in the output.

-- 
Peter Dalgaard, Professor,
Center for Statistics, Copenhagen Business School
Solbjerg Plads 3, 2000 Frederiksberg, Denmark
Phone: (+45)38153501
Email: pd@cbs.dk  Priv: pda...@gmail.com

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Lubuntu 13.04 raring ringtail: Problems installing pnmath_0.0-4

2013-04-27 Thread luke-tierney 

On Sat, 27 Apr 2013, Prof Brian Ripley wrote:

AFAIK pnmath is not a package on a public repository, and it seems it is 
using R internals not part of the API and not exposed in R 3.0.0.


I presume you got this from ~luke/R/experimental ... the clue is in the 
directory name.  My understanding was that this would eventually be merged 
into R rather than released as a package.


The intent is t merge it eventually; until that happens the intent is
to keep it working as an experimental package.  It is experimental,
and it's math code has not been updated to reflect improvements in the
R library, so should be used with caution.



If you really want to use it you will need to modify the R sources to unhide 
PRIMFUN and SET_PRIMFUN and re-compile.


These are now no longer hidden in R-devel and R-patched, so should be
available in the next R release.

Best,

luke




On 26/04/2013 19:39, Kjetil Halvorsen wrote:

Hola!

This is 64bit Lubuntu 13.04, with R-3.0.0 installed from
deb http://ppa.launchpad.net/marutter/rrutter/ubuntu raring  main
via synaptic.  gcc --version
gcc (Ubuntu/Linaro 4.7.3-1ubuntu1) 4.7.3


sessionInfo()

R version 3.0.0 (2013-04-03)
Platform: x86_64-pc-linux-gnu (64-bit)

locale:
  [1] LC_CTYPE=en_US.UTF-8   LC_NUMERIC=C
  [3] LC_TIME=en_US.UTF-8LC_COLLATE=en_US.UTF-8
  [5] LC_MONETARY=en_US.UTF-8LC_MESSAGES=en_US.UTF-8
  [7] LC_PAPER=C LC_NAME=C
  [9] LC_ADDRESS=C   LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats graphics  grDevices utils datasets  methods   base

other attached packages:
[1] MASS_7.3-26

loaded via a namespace (and not attached):
[1] compiler_3.0.0 fortunes_1.5-0 tools_3.0.0



I am trying to install pnmath from the commandline (outside R), this is
what happens:
kjetil@kjetil-HP-Pavilion-dv4-Notebook-PC:~/R/src$ sudo  R CMD INSTALL
pnmath_0.0-4.tar.gz
* installing to library ‘/usr/local/lib/R/site-library’
* installing *source* package ‘pnmath’ ...
** libs
gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -fopenmp -fpic  -O2
-pipe -g  -c bd0.c -o bd0.o
...
...
...
installing to /usr/local/lib/R/site-library/pnmath/libs
** R
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
Error in dyn.load(file, DLLpath = DLLpath, ...) :
   unable to load shared object
'/usr/local/lib/R/site-library/pnmath/libs/pnmath.so':
   /usr/local/lib/R/site-library/pnmath/libs/pnmath.so: undefined symbol:
PRIMFUN
Error: loading failed
Execution halted
ERROR: loading failed
* removing ‘/usr/local/lib/R/site-library/pnmath’

I have installed sufficient libraries as to compile R itself from source
without problems!

Kjetil

[[alternative HTML version deleted]]



__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel







--
Luke Tierney
Chair, Statistics and Actuarial Science
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa  Phone: 319-335-3386
Department of Statistics andFax:   319-335-3017
   Actuarial Science
241 Schaeffer Hall  email:   luke-tier...@uiowa.edu
Iowa City, IA 52242 WWW:  http://www.stat.uiowa.edu__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Speeding up build-from-source

2013-04-27 Thread Adam Seering 



On 04/27/2013 09:10 AM, Martin Morgan wrote:

On 04/26/2013 07:50 AM, Adam Seering wrote:

Hi,
 I've been playing around with the R source code a little; mostly
just
trying to familiarize myself.  I have access to some computers on a
reservation
system; so I've been reserving a computer, downloading and compiling
R, and
going from there.

 I'm finding that R takes a long time to build, though.  (Well,
ok, maybe 5
minutes -- I'm impatient :-) )  Most of that time, it's sitting there
byte-compiling some internal package or another, which uses just one
CPU core so
leaves the system mostly idle.

 I'm just curious if anyone has thought about parallelizing that
process?


Hi Adam -- parallel builds are supported by adding the '-j' flag when
you invoke make

   make -j

The packages are being built in parallel, in as much as this is possible
by their dependency structure. Also, you can configure without byte
compilation, see ~/src/R-devel/configure --help to make this part of the
build go more quickly. And after an initial build subsets of R, e.g.,
just the 'main' source or a single package like 'stats', can be built
with (assuming R's source, e.g., from svn, is in ~/src/R-devel, and
you're building R in ~/bin/R-devel) with

   cd ~/bin/R-devel/src/main
   make -j
   cd ~/bin/R-devel/src/library/stats
   make -j

The definitive source for answers to questions like these is

   > RShowDoc("R-admin")

Martin


Hi Martin,
	Thanks for the reply -- but I'm afraid the question you've answered 
isn't the question that I intended to ask.


	Based on your response, I think the answer to my question is likely 
"no."  But let me try rephrasing anyway, just in case:


	I'm certainly quite aware of "-j" as a make argument; if I weren't, the 
bottleneck would not be the byte-compilation, and the build would take 
rather more than 5 minutes :-)  That was the very first thing I tried. 
I don't believe that parallel make is as parallel as it theoretically 
could be.  (In fact, I see almost no parallelism between libraries on my 
system; individual .c files are parallelized nicely but only one library 
at a time.  This mostly matters at the compiling-bytecode step, since 
that's the biggest serial operation per library.)  My question is, has 
anyone thought about what it would take to parallelize the build further?


	I'm not sure that this can be done with just the makefiles.  But the 
following comment makes me at least a little suspicious:


""" src/library/Makefile
## FIXME: do some of this in parallel?
"""

	Surely some of the 'for' loops there could be unwound into proper make 
targets with dependency information?  I'm not sure if the dependency 
information would effectively force a serial compilation anyway, though?...


	Another approach, if the above is hard for some reason:  What I'm 
seeing is that the byte compilation is largely serial; but as you note, 
byte-compilation is optional.  Could the makefiles just defer it?; skip 
it up front and then do all the byte-compilations for all of the 
packages concurrently?  From a very cursory read of the code, it looks 
like the relevant code is in src/library/tools/R/makeLazyLoad.R?; and 
that file doesn't immediately look like it's doing anything that 
fundamentally couldn't be parallelized?  (ie., running multiple R 
processes at once, one per library; at a glance the logic looks nicely 
per-library.)


	A third approach could be to try to parallelize the logic in 
makeLazyLoad.R.  I would expect that to be at best much more difficult, 
though.


	Anyway, there are lots of things that look like they could in theory be 
done here.  And I know just enough at this point to be dangerous; not 
enough to contribute :-)  Hence my asking, has anyone thought about 
this?  If not, I assume the best thing for me to do would be to poke at 
it; try to figure out own my own how this works and what's most 
feasible.  But if anyone has any pointers, that would likely save me a 
bunch of time.  And if this is something that you prefer to keep serial 
for some reason, that would be good to know too, so I don't spend time 
on it.


Thanks,
Adam







Thanks,
Adam

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel





__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Lubuntu 13.04 raring ringtail: Problems installing pnmath_0.0-4

2013-04-27 Thread Kjetil Halvorsen 
see inline.



On Sat, Apr 27, 2013 at 2:38 PM, Prof Brian Ripley wrote:

> AFAIK pnmath is not a package on a public repository, and it seems it is
> using R internals not part of the API and not exposed in R 3.0.0.
>
> I presume you got this from ~luke/R/experimental ... the clue is in the
> directory name.


Ye, that is correct!

thanks, kjetil


> My understanding was that this would eventually be merged into R rather
> than released as a package.
>
> If you really want to use it you will need to modify the R sources to
> unhide PRIMFUN and SET_PRIMFUN and re-compile.
>
>
>
> On 26/04/2013 19:39, Kjetil Halvorsen wrote:
>
>> Hola!
>>
>> This is 64bit Lubuntu 13.04, with R-3.0.0 installed from
>> deb 
>> http://ppa.launchpad.net/**marutter/rrutter/ubunturaring
>>   main
>> via synaptic.  gcc --version
>> gcc (Ubuntu/Linaro 4.7.3-1ubuntu1) 4.7.3
>>
>>  sessionInfo()
>>>
>> R version 3.0.0 (2013-04-03)
>> Platform: x86_64-pc-linux-gnu (64-bit)
>>
>> locale:
>>   [1] LC_CTYPE=en_US.UTF-8   LC_NUMERIC=C
>>   [3] LC_TIME=en_US.UTF-8LC_COLLATE=en_US.UTF-8
>>   [5] LC_MONETARY=en_US.UTF-8LC_MESSAGES=en_US.UTF-8
>>   [7] LC_PAPER=C LC_NAME=C
>>   [9] LC_ADDRESS=C   LC_TELEPHONE=C
>> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
>>
>> attached base packages:
>> [1] stats graphics  grDevices utils datasets  methods   base
>>
>> other attached packages:
>> [1] MASS_7.3-26
>>
>> loaded via a namespace (and not attached):
>> [1] compiler_3.0.0 fortunes_1.5-0 tools_3.0.0
>>
>>>
>>>  I am trying to install pnmath from the commandline (outside R), this is
>> what happens:
>> kjetil@kjetil-HP-Pavilion-dv4-**Notebook-PC:~/R/src$ sudo  R CMD INSTALL
>> pnmath_0.0-4.tar.gz
>> * installing to library ‘/usr/local/lib/R/site-**library’
>> * installing *source* package ‘pnmath’ ...
>>
>> ** libs
>> gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -fopenmp -fpic  -O2
>> -pipe -g  -c bd0.c -o bd0.o
>> ...
>> ...
>> ...
>> installing to /usr/local/lib/R/site-library/**pnmath/libs
>> ** R
>> ** byte-compile and prepare package for lazy loading
>> ** help
>> *** installing help indices
>> ** building package indices
>> ** testing if installed package can be loaded
>> Error in dyn.load(file, DLLpath = DLLpath, ...) :
>>unable to load shared object
>> '/usr/local/lib/R/site-**library/pnmath/libs/pnmath.so'**:
>>/usr/local/lib/R/site-library/**pnmath/libs/pnmath.so: undefined
>> symbol:
>> PRIMFUN
>> Error: loading failed
>> Execution halted
>> ERROR: loading failed
>> * removing ‘/usr/local/lib/R/site-**library/pnmath’
>>
>>
>> I have installed sufficient libraries as to compile R itself from source
>> without problems!
>>
>> Kjetil
>>
>> [[alternative HTML version deleted]]
>>
>>
>>
>> __**
>> R-devel@r-project.org mailing list
>> https://stat.ethz.ch/mailman/**listinfo/r-devel
>>
>>
>
> --
> Brian D. Ripley,  rip...@stats.ox.ac.uk
> Professor of Applied Statistics,  
> http://www.stats.ox.ac.uk/~**ripley/
> University of Oxford, Tel:  +44 1865 272861 (self)
> 1 South Parks Road, +44 1865 272866 (PA)
> Oxford OX1 3TG, UKFax:  +44 1865 272595
>

[[alternative HTML version deleted]]

__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel

Re: [Rd] Speeding up build-from-source

2013-04-27 Thread Simon Urbanek 
On Apr 27, 2013, at 11:34 AM, Adam Seering wrote:

> 
> 
> On 04/27/2013 09:10 AM, Martin Morgan wrote:
>> On 04/26/2013 07:50 AM, Adam Seering wrote:
>>> Hi,
>>> I've been playing around with the R source code a little; mostly
>>> just
>>> trying to familiarize myself.  I have access to some computers on a
>>> reservation
>>> system; so I've been reserving a computer, downloading and compiling
>>> R, and
>>> going from there.
>>> 
>>> I'm finding that R takes a long time to build, though.  (Well,
>>> ok, maybe 5
>>> minutes -- I'm impatient :-) )  Most of that time, it's sitting there
>>> byte-compiling some internal package or another, which uses just one
>>> CPU core so
>>> leaves the system mostly idle.
>>> 
>>> I'm just curious if anyone has thought about parallelizing that
>>> process?
>> 
>> Hi Adam -- parallel builds are supported by adding the '-j' flag when
>> you invoke make
>> 
>>   make -j
>> 
>> The packages are being built in parallel, in as much as this is possible
>> by their dependency structure. Also, you can configure without byte
>> compilation, see ~/src/R-devel/configure --help to make this part of the
>> build go more quickly. And after an initial build subsets of R, e.g.,
>> just the 'main' source or a single package like 'stats', can be built
>> with (assuming R's source, e.g., from svn, is in ~/src/R-devel, and
>> you're building R in ~/bin/R-devel) with
>> 
>>   cd ~/bin/R-devel/src/main
>>   make -j
>>   cd ~/bin/R-devel/src/library/stats
>>   make -j
>> 
>> The definitive source for answers to questions like these is
>> 
>>   > RShowDoc("R-admin")
>> 
>> Martin
> 
> Hi Martin,
>   Thanks for the reply -- but I'm afraid the question you've answered 
> isn't the question that I intended to ask.
> 
>   Based on your response, I think the answer to my question is likely 
> "no."  But let me try rephrasing anyway, just in case:
> 
>   I'm certainly quite aware of "-j" as a make argument; if I weren't, the 
> bottleneck would not be the byte-compilation, and the build would take rather 
> more than 5 minutes :-)  That was the very first thing I tried. I don't 
> believe that parallel make is as parallel as it theoretically could be.  (In 
> fact, I see almost no parallelism between libraries on my system; individual 
> .c files are parallelized nicely but only one library at a time.  This mostly 
> matters at the compiling-bytecode step, since that's the biggest serial 
> operation per library.)  My question is, has anyone thought about what it 
> would take to parallelize the build further?
> 

I think you may have failed to notice that installation of packages *is* 
parallelized. The *output* is shown only en-bloc and to avoid mixing outputs of 
the parallel installations. But there are dependencies among packages, so those 
that require most of the others have to be built last -- nonetheless, in the 
current R you can install 9 recommended packages in parallel.


>   I'm not sure that this can be done with just the makefiles.  But the 
> following comment makes me at least a little suspicious:
> 
> """ src/library/Makefile
> ## FIXME: do some of this in parallel?
> """
> 
>   Surely some of the 'for' loops there could be unwound into proper make 
> targets with dependency information?  I'm not sure if the dependency 
> information would effectively force a serial compilation anyway, though?...
> 
>   Another approach, if the above is hard for some reason:  What I'm 
> seeing is that the byte compilation is largely serial; but as you note, 
> byte-compilation is optional.  Could the makefiles just defer it?; skip it up 
> front and then do all the byte-compilations for all of the packages 
> concurrently?

The problem is, again, dependencies - you cannot defer the compilation since it 
would change the package *after* is has already been used by another package 
which can cause inconsistencies (note that lazy loading is a red herring - it's 
used regardless of compilation). That said, you won't save significant amount 
of time anyway (did you actually profile the time or are you relying on your 
eyes to deceive you? ;)), so it's not worth the bother (try enabling LTO ;)).

Personally, I simply disable package compilation for all developments builds. 
You won't notice the difference for testing anyway. Moreover, you'll be barely 
doing a full build repeatedly, so the 4 minutes it takes are certainly nothing 
compared to other projects of such size... It becomes more fun when you start 
building all CRAN packages ;).

Cheers,
Simon


>  From a very cursory read of the code, it looks like the relevant code is in 
> src/library/tools/R/makeLazyLoad.R?; and that file doesn't immediately look 
> like it's doing anything that fundamentally couldn't be parallelized?  (ie., 
> running multiple R processes at once, one per library; at a glance the logic 
> looks nicely per-library.)
> 
>   A third approach could be to try to parallelize the logic in