[Rd] Including multiple third party libraries in an extension
Hi, I've got a C extension structured roughly like: package/ src/ Makevars foo.c some-lib/... some-other-lib/.. where foo.c and Makevars define dependencies on some-lib and some-other-lib. Currently I'm having Makevars configure;make install some-lib and some-other-lib into a local build directory, which produces shard libraries that ultimately I reference for foo.o in PKG_LIBS. I'm concerned about distribution. I've setup the appropriate magic with rpath for the packages .so (meaning that when the final .so is produced the dynamic libraries dependencies on some-lib and some-other-lib will prefer the location built in src/some-lib/... and src/some-other-lib/... But does this preclude me from being able to distribute a binary package? If I do want to build a binary distribution, is there a way I can package up everything needed, not just the resulting .so? Or, are there better ways to bundle extension-specific third party dependencies? ;) I'd rather not have my users have to install obscure libraries globally on their systems. Thanks! Tyler [[alternative HTML version deleted]] __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] use() might take a list of packages
I agree that it would be nice if library took an argument that is a vector of packages. I don't think one should need to use sapply here. -- View this message in context: http://r.789695.n4.nabble.com/use-might-take-a-list-of-packages-tp4031097p4032622.html Sent from the R devel mailing list archive at Nabble.com. __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Including multiple third party libraries in an extension
Tyler, On Nov 11, 2011, at 7:55 PM, Tyler Pirtle wrote: > Hi, > > I've got a C extension structured roughly like: > > package/ > src/ >Makevars >foo.c >some-lib/... >some-other-lib/.. > > where foo.c and Makevars define dependencies on some-lib and > some-other-lib. Currently I'm having > Makevars configure;make install some-lib and some-other-lib into a local > build directory, which produces > shard libraries that ultimately I reference for foo.o in PKG_LIBS. > > I'm concerned about distribution. I've setup the appropriate magic with > rpath for the packages .so That is certainly non-portable and won't work for a vast majority of users. > (meaning > that when the final .so is produced the dynamic libraries dependencies on > some-lib and some-other-lib > will prefer the location built in src/some-lib/... and > src/some-other-lib/... But does this preclude me from > being able to distribute a binary package? Yes. And I doubt the package will work the way you described it at all, because the "deep" .so won't be even installed. Also there are potential issues in multi-arch R (please consider testing that as well). > If I do want to build a binary > distribution, is there a way I can > package up everything needed, not just the resulting .so? > > Or, are there better ways to bundle extension-specific third party > dependencies? ;) I'd rather not have > my users have to install obscure libraries globally on their systems. > Typically the best solution is to compile the dependencies as --disable-shared --enable-static --with-pic (in autoconf speak - you don't need to actually use autoconf). That way your .so has all its dependencies inside and you avoid all run-time hassle. Note that it is very unlikely that you can take advantage of the dynamic nature of the dependencies (since no one else knows about them anyway) so there is not real point to build them dynamically. Also note that typically you want to use the package-level configure to run subconfigures, and *not* Makevars. (There may be reasons for an exception to that convention, but you need to be aware of the differences in multi-arch builds since Makevars builds all architectures at once from separate copies of the src directories whereas the presence of configure allows you to treat your package as one architecture at a time and you can pass-though parameters). Cheers, Simon __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] cygwin R-2.14.0 build fail
I tried to build R-2.14.0 on cygwin using the commands: ./configure --with-x=no make I started to get a whole lot of errors starting with: /cygdrive/c/Users/mcarter/src/R-2.14.0/src/modules/internet/Rhttpd.c:275: undefined reference to `_R_InputHandlers' which I have pasted at http://pastebin.com/GFb2pq92 I'm aware that there is a cygwinports ports, but it seems outdated, and when I tried installing it, it was very lengthy and seemed more trouble that it was worth. I abandoned the installation attempt. Any tips on a way forward? __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] cygwin R-2.14.0 build fail
On Nov 12, 2011, at 15:25 , Mark Carter wrote: > I tried to build R-2.14.0 on cygwin using the commands: > ./configure --with-x=no > make > > I started to get a whole lot of errors starting with: > /cygdrive/c/Users/mcarter/src/R-2.14.0/src/modules/internet/Rhttpd.c:275: > undefined reference to `_R_InputHandlers' > which I have pasted at > > http://pastebin.com/GFb2pq92 > > I'm aware that there is a cygwinports ports, but it seems outdated, and when > I tried installing it, it was very lengthy and seemed more trouble that it > was worth. I abandoned the installation attempt. > > > Any tips on a way forward? > As far as I can tell, what is happening to you is that cygwin needs special configuration to build .dll libraries and ./configure does not know how to set that up. R is not officially supported under Cygwin. Basically, we - er, Brian, mostly, I think - tried it several years ago, and it broke so many times that patience ran out. Others seem to be having better luck with recent versions (Google finds something that looks like ports of 2.13.2, so you might just need a little patience to get 2.14.0), but I think you're altogether better off over on https://lists.sourceforge.net/lists/listinfo/cygwin-ports-general > __ > R-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel -- Peter Dalgaard, Professor, Center for Statistics, Copenhagen Business School Solbjerg Plads 3, 2000 Frederiksberg, Denmark Phone: (+45)38153501 Email: pd@cbs.dk Priv: pda...@gmail.com __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] cygwin R-2.14.0 build fail
On 12.11.2011 21:28, peter dalgaard wrote: On Nov 12, 2011, at 15:25 , Mark Carter wrote: I tried to build R-2.14.0 on cygwin using the commands: ./configure --with-x=no make I started to get a whole lot of errors starting with: /cygdrive/c/Users/mcarter/src/R-2.14.0/src/modules/internet/Rhttpd.c:275: undefined reference to `_R_InputHandlers' which I have pasted at http://pastebin.com/GFb2pq92 I'm aware that there is a cygwinports ports, but it seems outdated, and when I tried installing it, it was very lengthy and seemed more trouble that it was worth. I abandoned the installation attempt. Any tips on a way forward? As far as I can tell, what is happening to you is that cygwin needs special configuration to build .dll libraries and ./configure does not know how to set that up. R is not officially supported under Cygwin. Basically, we - er, Brian, mostly, I think - tried it several years ago, and it broke so many times that patience ran out. Others seem to be having better luck with recent versions (Google finds something that looks like ports of 2.13.2, so you might just need a little patience to get 2.14.0), but I think you're altogether better off over on https://lists.sourceforge.net/lists/listinfo/cygwin-ports-general Or much simpler: use the native version. Uwe Ligges __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Including multiple third party libraries in an extension
Thanks Simon, a few replies...
On Sat, Nov 12, 2011 at 6:14 AM, Simon Urbanek
wrote:
> Tyler,
>
> On Nov 11, 2011, at 7:55 PM, Tyler Pirtle wrote:
>
> > Hi,
> >
> > I've got a C extension structured roughly like:
> >
> > package/
> > src/
> >Makevars
> >foo.c
> >some-lib/...
> >some-other-lib/..
> >
> > where foo.c and Makevars define dependencies on some-lib and
> > some-other-lib. Currently I'm having
> > Makevars configure;make install some-lib and some-other-lib into a local
> > build directory, which produces
> > shard libraries that ultimately I reference for foo.o in PKG_LIBS.
> >
> > I'm concerned about distribution. I've setup the appropriate magic with
> > rpath for the packages .so
>
> That is certainly non-portable and won't work for a vast majority of users.
Yea I figured, but apparently I have other, more pressing problems.. ;)
> > (meaning
> > that when the final .so is produced the dynamic libraries dependencies on
> > some-lib and some-other-lib
> > will prefer the location built in src/some-lib/... and
> > src/some-other-lib/... But does this preclude me from
> > being able to distribute a binary package?
>
> Yes. And I doubt the package will work the way you described it at all,
> because the "deep" .so won't be even installed. Also there are potential
> issues in multi-arch R (please consider testing that as well).
>
>
Understood. I wasn't a fan of any of the potential solutions I'd seen (one
of wich included source-only availability).
I've seen some other folks using the inst/ or data/ dirs for purposes like
this, but I agree it's ugly and has
issues. You raise a great point, too, about multi-arch R. I have potential
users that are definitely on
heterogeneous architectures, I noticed that when I R CMD INSTALL --build .
to check my current build,
I end up with a src-${ARCH} for both x86_64 and i386 - is there more
explicit multiarch testing I should be
doing?
>
> > If I do want to build a binary
> > distribution, is there a way I can
> > package up everything needed, not just the resulting .so?
> >
> > Or, are there better ways to bundle extension-specific third party
> > dependencies? ;) I'd rather not have
> > my users have to install obscure libraries globally on their systems.
> >
>
> Typically the best solution is to compile the dependencies as
> --disable-shared --enable-static --with-pic (in autoconf speak - you don't
> need to actually use autoconf). That way your .so has all its dependencies
> inside and you avoid all run-time hassle. Note that it is very unlikely
> that you can take advantage of the dynamic nature of the dependencies
> (since no one else knows about them anyway) so there is not real point to
> build them dynamically.
>
>
That is a much better solution and the one I've been looking for! I was
afraid I'd have to manually specific all the dependency objects but if I
just disable
shared than that makes much more sense, I can let the compiler and linker
do the work for me.
> Also note that typically you want to use the package-level configure to
> run subconfigures, and *not* Makevars. (There may be reasons for an
> exception to that convention, but you need to be aware of the differences
> in multi-arch builds since Makevars builds all architectures at once from
> separate copies of the src directories whereas the presence of configure
> allows you to treat your package as one architecture at a time and you can
> pass-though parameters).
>
>
Understood. Is src/ still the appropriate directory then for my third party
packages? Also, do you happen to know of any packages off-hand that I can
use
as a reference?
Thanks Simon! Your insights here are invaluable. I really appreciate it.
Tyler
> Cheers,
> Simon
>
>
>
[[alternative HTML version deleted]]
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Including multiple third party libraries in an extension
On Sat, Nov 12, 2011 at 8:08 PM, Tyler Pirtle wrote:
> Thanks Simon, a few replies...
>
> On Sat, Nov 12, 2011 at 6:14 AM, Simon Urbanek <
> simon.urba...@r-project.org> wrote:
>
>> Tyler,
>>
>> On Nov 11, 2011, at 7:55 PM, Tyler Pirtle wrote:
>>
>> > Hi,
>> >
>> > I've got a C extension structured roughly like:
>> >
>> > package/
>> > src/
>> >Makevars
>> >foo.c
>> >some-lib/...
>> >some-other-lib/..
>> >
>> > where foo.c and Makevars define dependencies on some-lib and
>> > some-other-lib. Currently I'm having
>> > Makevars configure;make install some-lib and some-other-lib into a local
>> > build directory, which produces
>> > shard libraries that ultimately I reference for foo.o in PKG_LIBS.
>> >
>> > I'm concerned about distribution. I've setup the appropriate magic with
>> > rpath for the packages .so
>>
>> That is certainly non-portable and won't work for a vast majority of
>> users.
>
>
> Yea I figured, but apparently I have other, more pressing problems.. ;)
>
>
>
>> > (meaning
>> > that when the final .so is produced the dynamic libraries dependencies
>> on
>> > some-lib and some-other-lib
>> > will prefer the location built in src/some-lib/... and
>> > src/some-other-lib/... But does this preclude me from
>> > being able to distribute a binary package?
>>
>> Yes. And I doubt the package will work the way you described it at all,
>> because the "deep" .so won't be even installed. Also there are potential
>> issues in multi-arch R (please consider testing that as well).
>>
>>
> Understood. I wasn't a fan of any of the potential solutions I'd seen (one
> of wich included source-only availability).
> I've seen some other folks using the inst/ or data/ dirs for purposes like
> this, but I agree it's ugly and has
> issues. You raise a great point, too, about multi-arch R. I have potential
> users that are definitely on
> heterogeneous architectures, I noticed that when I R CMD INSTALL --build .
> to check my current build,
> I end up with a src-${ARCH} for both x86_64 and i386 - is there more
> explicit multiarch testing I should be
> doing?
>
>
>>
>> > If I do want to build a binary
>> > distribution, is there a way I can
>> > package up everything needed, not just the resulting .so?
>> >
>> > Or, are there better ways to bundle extension-specific third party
>> > dependencies? ;) I'd rather not have
>> > my users have to install obscure libraries globally on their systems.
>> >
>>
>> Typically the best solution is to compile the dependencies as
>> --disable-shared --enable-static --with-pic (in autoconf speak - you don't
>> need to actually use autoconf). That way your .so has all its dependencies
>> inside and you avoid all run-time hassle. Note that it is very unlikely
>> that you can take advantage of the dynamic nature of the dependencies
>> (since no one else knows about them anyway) so there is not real point to
>> build them dynamically.
>>
>>
> That is a much better solution and the one I've been looking for! I was
> afraid I'd have to manually specific all the dependency objects but if I
> just disable
> shared than that makes much more sense, I can let the compiler and linker
> do the work for me.
>
>
>> Also note that typically you want to use the package-level configure to
>> run subconfigures, and *not* Makevars. (There may be reasons for an
>> exception to that convention, but you need to be aware of the differences
>> in multi-arch builds since Makevars builds all architectures at once from
>> separate copies of the src directories whereas the presence of configure
>> allows you to treat your package as one architecture at a time and you can
>> pass-though parameters).
>>
>>
> Understood. Is src/ still the appropriate directory then for my third
> party packages? Also, do you happen to know of any packages off-hand that I
> can use
> as a reference?
>
> Thanks Simon! Your insights here are invaluable. I really appreciate it.
>
>
>
> Tyler
>
>
Ah, also a few more questions...
I don't really understand the flow for developing multi-arch extensions.
Does configure run only once? Once per arch? What is the state of
src-${ARCH} by the time the src/Makevars or Makefile is executed? Is any of
this actually in the manual and am I just missing it? ;)
And why does R_ARCH start with a '/'? ;)
thanks again!
Tyler
>
>
>
>> Cheers,
>> Simon
>>
>>
>>
>
[[alternative HTML version deleted]]
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] cygwin R-2.14.0 build fail
On Sat, 12 Nov 2011, peter dalgaard wrote: On Nov 12, 2011, at 15:25 , Mark Carter wrote: I tried to build R-2.14.0 on cygwin using the commands: ./configure --with-x=no make I started to get a whole lot of errors starting with: /cygdrive/c/Users/mcarter/src/R-2.14.0/src/modules/internet/Rhttpd.c:275: undefined reference to `_R_InputHandlers' which I have pasted at http://pastebin.com/GFb2pq92 I'm aware that there is a cygwinports ports, but it seems outdated, and when I tried installing it, it was very lengthy and seemed more trouble that it was worth. I abandoned the installation attempt. Any tips on a way forward? As far as I can tell, what is happening to you is that cygwin needs special configuration to build .dll libraries and ./configure does not know how to set that up. It does. Currently R can be built under Cygwin, and how to do so is documented in the R-admin manual. But that was not the case a few weeks ago, so you need to look in R-patched/R-devel (and Cygwin might change again ...). R is not officially supported under Cygwin. Basically, we - er, Brian, mostly, I think - tried it several years ago, and it broke so many times that patience ran out. It was broken from most of 2011 too, for three separate reasons. Even when it 'works', it is far slower than the native Windows port and it fails 'make check' (not handling CR-delimited files, problems with wide-characters as Cygwin does not have proper locales). Others seem to be having better luck with recent versions (Google finds something that looks like ports of 2.13.2, so you might just need a little patience to get 2.14.0), but I think you're altogether better off over on https://lists.sourceforge.net/lists/listinfo/cygwin-ports-general __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel -- Peter Dalgaard, Professor, Center for Statistics, Copenhagen Business School Solbjerg Plads 3, 2000 Frederiksberg, Denmark Phone: (+45)38153501 Email: pd@cbs.dk Priv: pda...@gmail.com __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel -- Brian D. Ripley, rip...@stats.ox.ac.uk Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/ University of Oxford, Tel: +44 1865 272861 (self) 1 South Parks Road, +44 1865 272866 (PA) Oxford OX1 3TG, UKFax: +44 1865 272595 __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
