[Rd] R extension memory leak detection question
I am writing C++ program based on R extensions and also try to test the program with google address sanitizer. I thought if I don't protect the variable from the allocation API such as Rf_allocVector, there will be a memory leak. However, the address sanitizer didn't report it. Is my understanding correct? Or I will see the memory leak only if I compile R source code with the address sanitizer. Please help! Thanks, Xiaoyan [[alternative HTML version deleted]] __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] R extension memory leak detection question
Thank you all for your help. We embedded R in our program and found the memory in the process accumulated while our expectation is that the memory will go down after each R evaluation. I started to write a test program with only a few lines of R embedded codes and found the memory never went down even after R library is unloaded. Please find more details in the readme and test program at https://github.com/xiaoyanyuvt/RMemTest Thanks, Xiaoyan On Fri, Mar 19, 2021 at 2:21 PM Lionel Henry wrote: > > Still, memory leaks are possible if the program forgets about a > > pointer to some piece of memory no longer needed, and keeps that > > pointer in say some global structure. Such memory leaks would not be > > found using address sanitizer. > > We had a few cases of this in the past. Given the difficulty of > tracing the leaking references, we wrote this package for taking > snapshots of the R heap and finding dominators and shortest paths > between nodes: > > Repo: https://github.com/r-lib/memtools > Vignette: https://memtools.r-lib.org/articles/memtools.html > > One issue that complicates taking snapshots is that R doesn't expose > the GC roots. In practice, only the precious list is needed I think. > Would you consider a patch that allows retrieving the precious list > for debugging purposes via a `.Internal()` call? > > Best, > Lionel > > > On 3/15/21, Tomas Kalibera wrote: > > On 3/12/21 7:43 PM, xiaoyan yu wrote: > >> I am writing C++ program based on R extensions and also try to test the > >> program with google address sanitizer. > >> > >> I thought if I don't protect the variable from the allocation API such > as > >> Rf_allocVector, there will be a memory leak. However, the address > >> sanitizer > >> didn't report it. Is my understanding correct? Or I will see the memory > >> leak only if I compile R source code with the address sanitizer. > > > > Yes, you should use special options for compilation and linking to use > > address sanitizer. See Writing R Extensions, section 4.3.3. > > > > If you allocate an R object using Rf_allocVector(), but don't protect > > it, it means this object is available for the garbage collector to > > reclaim. So it is not a memory leak. > > > > Memory leaks with a garbage collector are much less common than without, > > because if the program loses a pointer to some piece of memory, that > > piece will automatically be reclaimed (not leaked). Still, memory leaks > > are possible if the program forgets about a pointer to some piece of > > memory no longer needed, and keeps that pointer in say some global > > structure. Such memory leaks would not be found using address sanitizer. > > > > Address sanitizer/Undefined behavior sanitizer can sometimes find errors > > caused by that the program forgets to protect an R object, but this is > > relatively rare, as they don't understand R heap specifically, so you > > cannot assume that if you create such example, the error will always be > > found. > > > > Best > > Tomas > > > >> > >> Please help! > >> > >> Thanks, > >> Xiaoyan > >> > >> [[alternative HTML version deleted]] > >> > >> __ > >> R-devel@r-project.org mailing list > >> https://stat.ethz.ch/mailman/listinfo/r-devel > > > > __ > > R-devel@r-project.org mailing list > > https://stat.ethz.ch/mailman/listinfo/r-devel > > > [[alternative HTML version deleted]] __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] R extension memory leak detection question
Thanks for your quick response. It is also surprising for us to notice the memory accumulation when running our program since it has been years since we developed our program. Here is the memory status I observed from 15384k to 234208k to 242024k without decreasing when running the test program. [Before the first ENTER]$ps -aux | grep foo xy 16985 0.0 0.0 *15384* 1312 pts/0S+ 00:09 0:00 ./foo [After the first ENTER ]$ps -aux | grep foo xy 16985 0.4 0.2 *234208* 42104 pts/0S+ 00:09 0:00 ./foo [After the second ENTER and also before the program exit]$ps -aux | grep foo xy16985 0.1 0.2 *242024* 42244 pts/0S+ 00:09 0:00 ./foo The test program is just a small simplified portion of our program. We observed even more memory in use when running our program. We would like to try to understand more of the memory life cycle of the embedded R. Thanks, Xiaoyan On Mon, Apr 5, 2021 at 6:53 PM Dirk Eddelbuettel wrote: > > On 5 April 2021 at 18:27, xiaoyan yu wrote: > | Thank you all for your help. > | We embedded R in our program and found the memory in the process > | accumulated while our expectation is that the memory will go down after > | each R evaluation. > | I started to write a test program with only a few lines of R embedded > codes > | and found the memory never went down even after R library is unloaded. > | Please find more details in the readme and test program at > | https://github.com/xiaoyanyuvt/RMemTest > > You may find the projects RInside (for easily embedding R inside C++ > programs) and littler (also embedding R, but using C only, for use in > lightweight cmdline applications) useful. Those have existed for, give or > take, 10 and 15 years and have not proven to show memory leaks so I feel > the > burden of proof is still on you. > > Also I got your program to compile (after making the 'makefile' a bit more > general, and fixing two things upsetting current C++ compilers) but I am > not > sure we really see memory consumption: > >edd@rob:~$ ps -fv $(pgrep -x foo) >PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND >1456192 pts/9S+ 0:00 0 1 5890 1768 0.0 ./foo >edd@rob:~$ ps -fv $(pgrep -x foo) >PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND >1456192 pts/9Sl+0:00 0 1 1617174 9896 0.0 ./foo >edd@rob:~$ > > Dirk > > -- > https://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org > [[alternative HTML version deleted]] __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] How to get utf8 string using R externals
I have a R Script Predict.R:
set.seed(42)
C <- seq(1:1000)
A <- rep(seq(1:200),5)
E <- (seq(1:1000) * (0.8 + (0.4*runif(50, 0, 1
L <- ifelse(runif(1000)>.5,1,0)
df <- data.frame(cbind(C, A, E, L))
load("C:/Temp/tree.RData")# load the model for scoring
P <- as.character(predict(tree_model_1,df,type='class'))
Then in a C++ program
I call eval to evaluate the script and then findVar the P variable.
After get each class label from P using string_elt and then
Rf_translateChar, the characters are unicodes () instead of
utf8 encoding of the korean characters 부실.
Can I know how to get UTF8 by using R externals?
I also found the same script giving utf8 characters in RGui but unicode in
Rterm.
I tried to attach a screenshot but got message "The message's content type
was not explicitly allowed"
In RGui, I saw the output 부실, while in Rterm, .
Please help.
[[alternative HTML version deleted]]
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] How to get utf8 string using R externals
I am using gmail. Not sure of the configuration of plain text.
The memory pointed by the char * as the output of Rf_translateChar() is
actually the string "".
On Wed, Jun 2, 2021 at 6:09 PM David Winsemius
wrote:
> First; you should configure yopu mail client to send plain text.
>
> Can you explain what is meant by:
>
> the characters are unicodes () instead of
> utf8 encoding of the korean characters 부실.
>
> As far as I can tell those two unicodes _are_ the utf8 encodings of 부실.
>
> You may need to consult a couple of R help pages. I suggest:
>
> ?Quotes
> ?points # has examples of changing fonts used for display on console.
>
> Sorry if I've misunderstood. I'm not on a Windows device, so posting the
> C++ program won't be helpful, but maybe it would for other prospective
> respondents.
>
> --
> David.
>
> On 6/2/21 1:33 PM, xiaoyan yu wrote:
> > I have a R Script Predict.R:
> > set.seed(42)
> > C <- seq(1:1000)
> > A <- rep(seq(1:200),5)
> > E <- (seq(1:1000) * (0.8 + (0.4*runif(50, 0, 1
> > L <- ifelse(runif(1000)>.5,1,0)
> > df <- data.frame(cbind(C, A, E, L))
> > load("C:/Temp/tree.RData")# load the model for scoring
> >
> >P <- as.character(predict(tree_model_1,df,type='class'))
> >
> > Then in a C++ program
> > I call eval to evaluate the script and then findVar the P variable.
> > After get each class label from P using string_elt and then
> > Rf_translateChar, the characters are unicodes () instead
> of
> > utf8 encoding of the korean characters 부실.
> > Can I know how to get UTF8 by using R externals?
> >
> > I also found the same script giving utf8 characters in RGui but unicode
> in
> > Rterm.
> > I tried to attach a screenshot but got message "The message's content
> type
> > was not explicitly allowed"
> > In RGui, I saw the output 부실, while in Rterm, .
> >
> > Please help.
> >
> > [[alternative HTML version deleted]]
> >
> > __
> > R-devel@r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-devel
>
[[alternative HTML version deleted]]
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] How to get utf8 string using R externals
Thanks! I tried my C++ program based on R externals and the same R script
and found the results shown are the desired glyph.
Hence this is R windows specific problem.
On Wed, Jun 2, 2021 at 9:08 PM brodie gaslam
wrote:
>
> > On Wednesday, June 2, 2021, 7:58:54 PM EDT, xiaoyan yu <
> xiaoyan...@gmail.com> wrote:
> >
> > I am using gmail. Not sure of the configuration of plain text.
> > The memory pointed by the char * as the output of Rf_translateChar() is
> > actually the string "".
>
> Hi Xiaoyan,
>
> Unfortunately I'm not super familiar with R on Windows, but I think
> I can provide a simpler reproducible example. In Rgui, if I type "\UBD80"
> at the prompt and hit enter, I see the desired glyph. In Rterm I see the
> unicode escape.
>
> IIRC the capabilities of Rterm and Rgui are different, and UTF8 support
> in windows is limited. Tomas Kalibera discusses this in some detail:
>
>
> https://developer.r-project.org/Blog/public/2020/05/02/utf-8-support-on-windows/index.html
>
> In terms of `Rf_translateChar()`, presumably the `Riconv` call is failing
> on Rterm, but not on Rgui:
>
> https://github.com/r-devel/r-svn/blob/master/src/main/sysutils.c#L924
>
> I'm guessing, but that would explain why the C level string is in that
> format. I don't know why the string would translate in Rgui though. My
> guess is that it did not as even in Rgui the following:
>
> enc2native("\uBD80")
>
> Produces the escaped version of the string.
>
> As others have suggested you could try the experimental UCRT Windows
> release:
>
>
> https://developer.r-project.org/Blog/public/2021/03/12/windows/utf-8-toolchain-and-cran-package-checks/index.html
>
> Install instructions (focus on Binary installer):
>
>
> https://svn.r-project.org/R-dev-web/trunk/WindowsBuilds/winutf8/ucrt3/howto.html
>
> If I try UCRT on my system this no longer produces the escape:
>
> enc2native("\uBD80")
>
> Although all I see is a question mark. My guess is that my code page or
> something similar is not set right. Examining with `charToRaw` reveals
> the string remains in UTF-8 encoding.
>
> Aside: it's not clear to me that you need to translate the string if your
> intent is for it to remain UTF-8. You just don't seem to be set-up to
> interpret UTF-8 strings currently.
>
> Best,
>
> B
>
> > On Wed, Jun 2, 2021 at 6:09 PM David Winsemius
> > wrote:
> >
> >> First; you should configure yopu mail client to send plain text.
> >>
> >> Can you explain what is meant by:
> >>
> >> the characters are unicodes () instead of
> >> utf8 encoding of the korean characters 부실.
> >>
> >> As far as I can tell those two unicodes _are_ the utf8 encodings of 부실.
> >>
> >> You may need to consult a couple of R help pages. I suggest:
> >>
> >> ?Quotes
> >> ?points # has examples of changing fonts used for display on console.
> >>
> >> Sorry if I've misunderstood. I'm not on a Windows device, so posting
> the
> >> C++ program won't be helpful, but maybe it would for other prospective
> >> respondents.
> >>
> >> --
> >> David.
> >>
> >> On 6/2/21 1:33 PM, xiaoyan yu wrote:
> >> > I have a R Script Predict.R:
> >> > set.seed(42)
> >> > C <- seq(1:1000)
> >> > A <- rep(seq(1:200),5)
> >> > E <- (seq(1:1000) * (0.8 + (0.4*runif(50, 0, 1
> >> > L <- ifelse(runif(1000)>.5,1,0)
> >> > df <- data.frame(cbind(C, A, E, L))
> >> > load("C:/Temp/tree.RData")# load the model for
> scoring
> >> >
> >> >P <- as.character(predict(tree_model_1,df,type='class'))
> >> >
> >> > Then in a C++ program
> >> > I call eval to evaluate the script and then findVar the P variable.
> >> > After get each class label from P using string_elt and then
> >> > Rf_translateChar, the characters are unicodes ()
> instead
> >> of
> >> > utf8 encoding of the korean characters 부실.
> >> > Can I know how to get UTF8 by using R externals?
> >> >
> >> > I also found the same script giving utf8 characters in RGui but
> unicode
> >> in
> >> > Rterm.
> >> > I tried to attach a screenshot but got message "The message's content
> >> type
> >> > was not explicitly allowed"
> >> > In RGui, I saw the output 부실, while in Rterm, .
> >> >
> >> > Please help.
> >> >
> >> > [[alternative HTML version deleted]]
> >> >
> >> > __
> >> > R-devel@r-project.org mailing list
> >> > https://stat.ethz.ch/mailman/listinfo/r-devel
> >
> >>
> >
> > [[alternative HTML version deleted]]
> >
> > __
> > R-devel@r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-devel
> >
> >
>
[[alternative HTML version deleted]]
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
