[Rd] Minimal build of R ...
Hi, I'm trying to cross-compile R to javascript so that it can run in a web-browser. Take as long as you need to stop laughing. So, as I was saying - I want to try and get a build of R running in the browser. [If you're not familiar with it already, you might enjoy looking at emscripten.org. It's a remarkably capable tool for translating LLVM bitcode to javascript. Check out some of the demos!] I'm trying to start out with the most minimal build of R possible. I can turn off various options in the configure script, but I'm wondering about the bundled R packages (base, stats etc). I'm guessing that the native code portions of these packages are dynamically loaded at runtime, which will probably need patching. To start off, I'd like to not build these packages if possible. So, is there a way to configure which packages in the library get built or is it just a case of editing the makefile? And is there a minimal set of them that would still allow R to run (not be useful - that can come later - just run)? Thanks in advance for any help anyone can provide :-) Jony -- Centre for Cold Matter, The Blackett Laboratory, Imperial College London, London SW7 2BW T: +44 (0)207 5947741 http://www.imperial.ac.uk/people/jony.hudson http://www.imperial.ac.uk/ccm/research/edm http://www.monkeycruncher.org http://j-star.org/ -- __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Minimal build of R ...
Hi All, thanks for the replies. Very helpful to know that it will run with just base. Looks like the best bet, at least to get started, is to not use the usual build-process, but to come up with a simple build-script for just the core. Ultimately, the build script has to be different anyway, as compiling the Fortran code to JS requires a few more steps than the native compile. For a bit of context, the reason I'm toying with this is I've been experimenting recently with analysis-in-the-browser. The kernel of the idea is that if you could do real analysis, without installing anything, and share it on the web then it would be a Good Thing, and could make it easier to engage people with data. I've got a proof-of-concept version running here http://www.monkeycruncher.org that let's you write javascript analysis code in notebook-style documents. It's neat, but it's a bit hamstrung by the lack of javascript libraries to actually do any useful analysis! If you could have R running in there though, that would be a much better proposition ... I'll let you know if I make any progress! Jony -- Centre for Cold Matter, The Blackett Laboratory, Imperial College London, London SW7 2BW T: +44 (0)207 5947741 http://www.imperial.ac.uk/people/jony.hudson http://www.imperial.ac.uk/ccm/research/edm http://www.monkeycruncher.org http://j-star.org/ -- On 3 May 2013, at 01:31, Simon Urbanek wrote: > On May 2, 2013, at 6:18 PM, Gabriel Becker wrote: > >> Jony, >> >> I'm currently writing up the paper for something with a similar result but >> very different implementation. The RBrowserPlugin package/browser plugin >> (joint with my advisor Duncan Temple Lang) embeds R within the web browser >> as an NPAPI plugin. >> >> This approach allows full bi-directional communication between R and the >> javascript engine (including direct function calling and references to >> native objects in both directions) using a user's existing local R >> installation (including packages). >> > > Minor detail: it requires you to have R *and* a special plugin which makes it > pretty much non-deployable. It's completely unrelated to what Jony is > proposing - which doesn't require any dependencies and is actually pretty > cool and would be useful if feasible. FWIW: There are many ways to run R from > a browser that already exist - without the need for plugins or other > client-side hacks - that's the beauty of modern browsers :). > > > To get this back on the actual topic: I have been toying with cross-compiling > R when I was porting it on the iPhone and it's possible, however, you can't > use the build process as-is. It does build core R properly, but the problem > is that you need to bootstrap R to build any packages. I worked around the > problem at the time by building packages on another platform and re-using > those files (things like lazy-loaded DBs, compiled RD files etc.). > > I can imagine that you'll need some equivalent to dynamic linking, but > conceptually it's nothing else but calling functions, so I think you should > be able to compile each package separately and just replace the dynload code > by code that loads another JavaScript. The nice thing is that packages will > simply be just another JS libraries. That's all in theory, I didn't actually > try that part. I suspect you'll have a lot of work, e.g. you'll need to map > all the I/O operations that load compiled/stored R code, documentation, data > from somewhere etc. Good luck! > If all fails, you can always compile R for JS/Linux ;). > > Cheers, > Simon > > > >> Devel source at https://github.com/gmbecker/RFirefox, release, (hopefully) >> officially cross-platform version to coincide with the paper going off for >> review. >> >> I had toyed with the idea of the emscripten approach, but I think putting R >> in the browser once is enough for me at the moment so I will happily keep >> an eye on your project instead of attacking that myself :). >> >> As for your actual question I can't really say, other than that I suspect >> you will not be able to dispense with base and methods, but that I would >> conjecture that stats is "optional". >> >> ~G >> >> >> On Thu, May 2, 2013 at 9:12 AM, Jony Hudson >> wrote: >> >>> Hi, >>> >>> I'm trying to cross-compile R to javascript so that it can run in a >>> web-browser. Take as long as you need to stop laughing. So, as I was saying >>> - I want to try and get a build of R running in the browser. [If you're not >>> familiar with it alr
Re: [Rd] Minimal build of R ...
Hi Gabriel, yes, packages obviously contain all the good stuff, but need to start somewhere! The ipython notebook project is very impressive, and I've been keeping a close eye on it, although I started out on monkeycruncher long before I was aware of it (I make slow progress). I guess I think of my thing as an experiment in just how much can be done purely in the web client. There are some advantages to pure client-side (rich interactivity, no need for a server, ubiquity) which make it interesting, but it might be a bit "too soon" to be useful! Jony -- Centre for Cold Matter, The Blackett Laboratory, Imperial College London, London SW7 2BW T: +44 (0)207 5947741 http://www.imperial.ac.uk/people/jony.hudson http://www.imperial.ac.uk/ccm/research/edm http://www.monkeycruncher.org http://j-star.org/ -- On 3 May 2013, at 16:46, Gabriel Becker wrote: > Jony, > > I would caution that while R will run with just base, you won't be able to do > much of anything with it. All the statistical analysis and graphing functions > reside in additional packages. So practically speaking you'll need the > workarounds Simon mentioned involving an alternative to dyn.load so you can > attach additional packages. Your project looks pretty cool though! > > Also, as an aside have you seen the ipython notebook ( > http://ipython.org/ipython-doc/dev/interactive/htmlnotebook.html )? It > doesn't meet your requirement of not installing anything, and currently has a > slightly different focus, but the appearance of the documents is very similar > to what you are doing here (other than being server based), and it already > understands multiple languages, including python, R, matlab, octave, etc and > is being actively developed and supported. > > ~G > > > On Fri, May 3, 2013 at 8:21 AM, Jony Hudson > wrote: > Hi All, > > thanks for the replies. Very helpful to know that it will run with just > base. Looks like the best bet, at least to get started, is to not use the > usual build-process, but to come up with a simple build-script for just the > core. Ultimately, the build script has to be different anyway, as compiling > the Fortran code to JS requires a few more steps than the native compile. > > For a bit of context, the reason I'm toying with this is I've been > experimenting recently with analysis-in-the-browser. The kernel of the idea > is that if you could do real analysis, without installing anything, and share > it on the web then it would be a Good Thing, and could make it easier to > engage people with data. I've got a proof-of-concept version running here > http://www.monkeycruncher.org that let's you write javascript analysis code > in notebook-style documents. It's neat, but it's a bit hamstrung by the lack > of javascript libraries to actually do any useful analysis! If you could have > R running in there though, that would be a much better proposition ... > > I'll let you know if I make any progress! > > > Jony > > > -- > Centre for Cold Matter, The Blackett Laboratory, > Imperial College London, London SW7 2BW > T: +44 (0)207 5947741 > http://www.imperial.ac.uk/people/jony.hudson > http://www.imperial.ac.uk/ccm/research/edm > http://www.monkeycruncher.org > http://j-star.org/ > -- > > On 3 May 2013, at 01:31, Simon Urbanek wrote: > > > On May 2, 2013, at 6:18 PM, Gabriel Becker wrote: > > > >> Jony, > >> > >> I'm currently writing up the paper for something with a similar result but > >> very different implementation. The RBrowserPlugin package/browser plugin > >> (joint with my advisor Duncan Temple Lang) embeds R within the web browser > >> as an NPAPI plugin. > >> > >> This approach allows full bi-directional communication between R and the > >> javascript engine (including direct function calling and references to > >> native objects in both directions) using a user's existing local R > >> installation (including packages). > >> > > > > Minor detail: it requires you to have R *and* a special plugin which makes > > it pretty much non-deployable. It's completely unrelated to what Jony is > > proposing - which doesn't require any dependencies and is actually pretty > > cool and would be useful if feasible. FWIW: There are many ways to run R > > from a browser that already exist - without the need for plugins or other > > client-side hacks - that's the beauty of modern browsers :). > > > > > > To get this back on the actual topic: I have been toying with > > cross-compiling R when I was porting it on the iPhone and it&#x
Re: [Rd] Minimal build of R ...
Hi, I'm making some progress with this, but have hit a sticking point and am looking for a hint. Most of the compilation seems to be going ok, after some liberal use of f2c, but I'm getting compile errors in src/main/connections.c : connections.c:926:43: error: use of undeclared identifier 'SSIZE_MAX' if ((double) size * (double) nitems > SSIZE_MAX) ^ connections.c:937:43: error: use of undeclared identifier 'SSIZE_MAX' if ((double) size * (double) nitems > SSIZE_MAX) ^ connections.c:3354:21: warning: implicit conversion from 'long long' to 'R_xlen_t' (aka 'int') changes value from 4503599627370496 to 0 [-Wconstant-conversion] nnn = (n < 0) ? R_XLEN_T_MAX : n; ~ ^~~~ ../../src/include/Rinternals.h:65:23: note: expanded from macro 'R_XLEN_T_MAX' # define R_XLEN_T_MAX 4503599627370496 ^~~~ connections.c:3662:11: error: duplicate case value '4' case sizeof(long): ^ connections.c:3660:11: note: previous case defined here case sizeof(int): ^ connections.c:3680:11: error: duplicate case value '4' case sizeof(long): ^ connections.c:3678:11: note: previous case defined here case sizeof(int): ^ connections.c:3912:11: error: duplicate case value '4' case sizeof(long): ^ connections.c:3910:11: note: previous case defined here case sizeof(int): ^ connections.c:3956:11: error: duplicate case value '4' case sizeof(long): ^ connections.c:3952:11: note: previous case defined here case sizeof(int): Recall that I'm compiling with emscripten, which uses clang to generate LLVM bitcode, which is then converted to javascript. I'm currently using the existing autotools build scripts, which emscripten tries to twist in to doing something sensible. It's quite possible that it's ending up mis-"./configure"d though. I appreciate this is fairly off-topic, but if anyone has any pointers where to start looking, they would be greatly appreciated :-) Thanks, Jony -- Centre for Cold Matter, The Blackett Laboratory, Imperial College London, London SW7 2BW T: +44 (0)207 5947741 http://www.imperial.ac.uk/people/jony.hudson http://www.imperial.ac.uk/ccm/research/edm http://www.monkeycruncher.org http://j-star.org/ -- On 2 May 2013, at 17:12, Jony Hudson wrote: > Hi, > > I'm trying to cross-compile R to javascript so that it can run in a > web-browser. Take as long as you need to stop laughing. So, as I was saying - > I want to try and get a build of R running in the browser. [If you're not > familiar with it already, you might enjoy looking at emscripten.org. It's a > remarkably capable tool for translating LLVM bitcode to javascript. Check out > some of the demos!] > > I'm trying to start out with the most minimal build of R possible. I can turn > off various options in the configure script, but I'm wondering about the > bundled R packages (base, stats etc). I'm guessing that the native code > portions of these packages are dynamically loaded at runtime, which will > probably need patching. To start off, I'd like to not build these packages if > possible. > > So, is there a way to configure which packages in the library get built or is > it just a case of editing the makefile? And is there a minimal set of them > that would still allow R to run (not be useful - that can come later - just > run)? > > Thanks in advance for any help anyone can provide :-) > > > Jony > > -- > Centre for Cold Matter, The Blackett Laboratory, > Imperial College London, London SW7 2BW > T: +44 (0)207 5947741 > http://www.imperial.ac.uk/people/jony.hudson > http://www.imperial.ac.uk/ccm/research/edm > http://www.monkeycruncher.org > http://j-star.org/ > -- > > __ > R-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] R in the browser ...
Hi all, I hope you'll forgive me - I don't plan to start using this list as my blog - but given the discussion following my last post I thought people on here might be interested to see some progress. This is a minimal build of R, cross-compiled from C/Fortran to javascript with emscripten - to be clear, nothing is running server-side, this is all running in the browser's JS engine. The user experience is rather lacking at the minute, much is missing (see below), and there are no compiler optimisations applied (also below) but still, it kind of works. Have a play here: http://r-in-the-browser.herokuapp.com/ (WARNING: 20MB HTML file, 3.5MB gzipped + 7MB data file). It goes without saying that you'll want to use a modern web browser to look at it! It works in the latest chrome, firefox, and safari (although I can't see the session output until after I q() in my safari :-() It also seems to run on my iPad, after a very long start-up wait, but it shares the problem of desktop safari that you have to work blind. It also runs on my Android phone, although it takes a _very_ long time to start up - something like 15 minutes! Picture of a session running on an iPad: http://imgur.com/jzYL5wf Picture of a session running on a Nexus 4 phone: http://imgur.com/bjDA95j So far, it's only the core of R - the only package it is loading is "base". The other default packages build mostly, I think, but I haven't yet figured out how to patch the dynamic loading to pull the "native" code in at the right time. Also, nothing using LAPACK works as R is trying to dynamically load that too. Getting it to build was fiddly, but that probably has a lot to do with my lack of experience building anything unix-y, and this being the first time I've used autotools (I read in the manual that their aim was not to make it user-friendly for maintainers, but rather for users. Well, chapeau to them as they've certainly achieved their aims). In rough outline (and from memory): - I used f2c to convert the Fortran sources in appl, main and extra/blas to c, and modified the Makefile.in's accordingly - also adding instruction to link in a version of libf2c that I'd pre-compiled to LLVM bitcode. - I had to manually hack on the configure.h file to remove references to some functions that don't seem to exist in emscripten: ccosh, cexp etc. - I also added code to set SSIZE_MAX and R_XLEN_T_MAX to sensible values (thanks Peter). - I hacked connections.c as there were some duplicate case statements. I think this was because the ./configure was getting confused over the size of some basic types. I suspect all of above hacks could be avoided if I understood autotools better. - I tweaked configure.h again by hand to disable ARPA_INET. I think there's a problem in emscripten's inet header files, but am not sure yet. - I had to hack xdr_mem.c to force it to use an appropriate ntohl and htonl. Again, probably an autotools problem that I'm not understanding. - This was enough to run make and have it build LLVM-bitcode for the main source tree (except, bizarrely mkdtemp.c which I had to compile by hand). The make errors out before the end, but it gets far enough. I could then link all of the generated bitcode together and convert to JS. The "virtual filesystem" for the JS code was populated with the contents of /usr/local/lib/R which I trimmed down a bit to get rid of stuff that wasn't going to work. - The emscripten libc implementation is incomplete so I had to stub out some functions that are probably quite important - glob() and globfree(). That really needs to be fixed! I also stubbed out __locale_mb_cur_max to return some value or other, as I wasn't sure where it was supposed to come from, and I was getting tired. At the moment all compiler optimisations are turned off, which makes a big difference to code size and performance. The problem is that it appears that some of the R code uses unsafe function pointer casting, which causes trouble with the way that emscripten optimises code (I think it's that it depends on each function having a well-defined type that the JS interpreter can be sure of). I haven't looked into where these casts are, and how difficult it would be to make them safe, but hopefully they're not too pervasive. Anyway, it's just a start, but I'm pretty pleased with it :-) Jony -- Centre for Cold Matter, The Blackett Laboratory, Imperial College London, London SW7 2BW T: +44 (0)207 5947741 http://www.imperial.ac.uk/people/jony.hudson http://www.imperial.ac.uk/ccm/research/edm http://www.monkeycruncher.org http://j-star.org/ -- __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
