Re: [R-pkg-devel] libarrow
I'll also add that you can write R level code that processes files lazily chunk-by-chunk using arrow::open_dataset() |> arrow::map_batches(), and that you can write C level code that processes batches chunk-by-chunk using arrow::open_dataset() |> nanoarrow::as_nanoarrow_array_stream(), which will give you a pointer to an ABI stable ArrowArrayStream that you can pass between packages (helpers exist in the Arrow rust implementation and the nanoarrow C/C++ library to help consume that, and I'm happy to work up an example if you are interested). Cheers, -dewey On 2024-11-26 15:45, Bryce Mecum wrote: The work in progress to add libarrow to MXE is located at [1]. I don't think it's complete but I'm sharing it here in case it's a useful starting point for someone. Continuing this work would be welcome. Thang: Feel free to file an issue on the Arrow issue tracker [2] or email the user@ mailing list [3] if you get stuck. [1] https://github.com/assignUser/cran-mxe [2] https://github.com/apache/arrow/issues [3] https://arrow.apache.org/community/ On Tue, Nov 26, 2024 at 3:11 AM Tomas Kalibera wrote: On 11/26/24 11:18, Pham, T.V. via R-package-devel wrote: > Dear all, > > I am planning to use libarrow for reading parquet files, chunk by chunk. It seems that by installing "arrow", one gets the libarrow library too (at least on Windows). On Windows, to properly use Apache Arrow with R, it would have to be added to Rtools (and hence ideally first upstream to MXE). I've been in touch with two arrow developers who started working on adding libarrow to MXE, but then ran out of time. If there were any volunteers to finish this, perhaps its best to check first with the arrow team where they got, rather than starting from scratch. Best Tomas > > I wonder if the C++ header files are already in some package like Rcpp. Also, how should the link to the libarrow look like? > > Thank you & best regards, > Thang > > __ > AmsterdamUMC disclaimer : www.amsterdamumc.org/nl/disclaimers.htm > > __ > R-package-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-package-devel __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] libarrow
It is clear. Thank you very much. Thang -Original Message- From: R-package-devel On Behalf Of r-package-devel-requ...@r-project.org Sent: Wednesday, November 27, 2024 12:00 PM To: r-package-devel@r-project.org Subject: R-package-devel Digest, Vol 115, Issue 19 Send R-package-devel mailing list submissions to r-package-devel@r-project.org To subscribe or unsubscribe via the World Wide Web, visit https://stat.ethz.ch/mailman/listinfo/r-package-devel or, via email, send a message with subject or body 'help' to r-package-devel-requ...@r-project.org You can reach the person managing the list at r-package-devel-ow...@r-project.org When replying, please edit your Subject line so it is more specific than "Re: Contents of R-package-devel digest..." Today's Topics: 1. Re: libarrow (Tomas Kalibera) -- Message: 1 Date: Tue, 26 Nov 2024 12:10:47 +0100 From: Tomas Kalibera To: r-package-devel@r-project.org Subject: Re: [R-pkg-devel] libarrow Message-ID: <1989b439-df81-4f7e-aee8-718124441...@gmail.com> Content-Type: text/plain; charset="utf-8"; Format="flowed" On 11/26/24 11:18, Pham, T.V. via R-package-devel wrote: > Dear all, > > I am planning to use libarrow for reading parquet files, chunk by chunk. It > seems that by installing "arrow", one gets the libarrow library too (at least > on Windows). On Windows, to properly use Apache Arrow with R, it would have to be added to Rtools (and hence ideally first upstream to MXE). I've been in touch with two arrow developers who started working on adding libarrow to MXE, but then ran out of time. If there were any volunteers to finish this, perhaps its best to check first with the arrow team where they got, rather than starting from scratch. Best Tomas > > I wonder if the C++ header files are already in some package like Rcpp. Also, > how should the link to the libarrow look like? > > Thank you & best regards, > Thang > > __ > AmsterdamUMC disclaimer : http://www.amsterdamumc.org/nl/disclaimers.htm > > __ > R-package-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-package-devel -- Subject: Digest Footer ___ R-package-devel mailing list R-package-devel@r-project.org https://stat.ethz.ch/mailman/listinfo/r-package-devel -- End of R-package-devel Digest, Vol 115, Issue 19 __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
Re: [R-pkg-devel] libarrow
The work in progress to add libarrow to MXE is located at [1]. I don't think it's complete but I'm sharing it here in case it's a useful starting point for someone. Continuing this work would be welcome. Thang: Feel free to file an issue on the Arrow issue tracker [2] or email the user@ mailing list [3] if you get stuck. [1] https://github.com/assignUser/cran-mxe [2] https://github.com/apache/arrow/issues [3] https://arrow.apache.org/community/ On Tue, Nov 26, 2024 at 3:11 AM Tomas Kalibera wrote: > > On 11/26/24 11:18, Pham, T.V. via R-package-devel wrote: > > Dear all, > > > > I am planning to use libarrow for reading parquet files, chunk by chunk. It > > seems that by installing "arrow", one gets the libarrow library too (at > > least on Windows). > > On Windows, to properly use Apache Arrow with R, it would have to be > added to Rtools (and hence ideally first upstream to MXE). I've been in > touch with two arrow developers who started working on adding libarrow > to MXE, but then ran out of time. If there were any volunteers to finish > this, perhaps its best to check first with the arrow team where they > got, rather than starting from scratch. > > Best > Tomas > > > > > I wonder if the C++ header files are already in some package like Rcpp. > > Also, how should the link to the libarrow look like? > > > > Thank you & best regards, > > Thang > > > > __ > > AmsterdamUMC disclaimer : www.amsterdamumc.org/nl/disclaimers.htm > > > > __ > > R-package-devel@r-project.org mailing list > > https://stat.ethz.ch/mailman/listinfo/r-package-devel > > __ > R-package-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-package-devel __ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
