[Rd] custom allocators, Valgrind and uninitialized memory
Hi, In my package bettermc, I use a custom allocator, which hands out already defined/initialized memory (mmap of a POSIX shared memory object). If my code is run in R which was configured --with-valgrind-instrumentation > 0, Valgrind will (correctly) complain about uninitialized memory being used, e.g. ==813836== Conditional jump or move depends on uninitialised value(s) ==813836==at 0x4F0A9D: getvar (svn/R-devel/src/main/eval.c:5171) ==813836==by 0x4D9B38: bcEval (svn/R-devel/src/main/eval.c:6867) ==813836==by 0x4F0077: Rf_eval (svn/R-devel/src/main/eval.c:727) ==813836==by 0x4F09AF: forcePromise (svn/R-devel/src/main/eval.c:555) ==813836==by 0x4F0C57: FORCE_PROMISE (svn/R-devel/src/main/eval.c:5136) ==813836==by 0x4F0C57: getvar (svn/R-devel/src/main/eval.c:5177) ==813836==by 0x4D9B38: bcEval (svn/R-devel/src/main/eval.c:6867) ==813836==by 0x4F0077: Rf_eval (svn/R-devel/src/main/eval.c:727) ==813836==by 0x4F1A8D: R_execClosure (svn/R-devel/src/main/eval.c:1897) ==813836==by 0x4F2783: Rf_applyClosure (svn/R-devel/src/main/eval.c:1823) ==813836==by 0x4DF61D: bcEval (svn/R-devel/src/main/eval.c:7083) ==813836==by 0x4F0077: Rf_eval (svn/R-devel/src/main/eval.c:727) ==813836==by 0x4F1A8D: R_execClosure (svn/R-devel/src/main/eval.c:1897) ==813836== Uninitialised value was created by a client request ==813836==at 0x52D5CF: Rf_allocVector3 (svn/R-devel/src/main/memory.c:2892) ==813836==by 0x16B415EA: allocate_from_shm (packages/tests-vg/bettermc/src/copy2shm.c:289) ==813836==by 0x49D123: R_doDotCall (svn/R-devel/src/main/dotcode.c:614) ==813836==by 0x4DA36D: bcEval (svn/R-devel/src/main/eval.c:7671) ==813836==by 0x4F0077: Rf_eval (svn/R-devel/src/main/eval.c:727) ==813836==by 0x4F1A8D: R_execClosure (svn/R-devel/src/main/eval.c:1897) ==813836==by 0x4F2783: Rf_applyClosure (svn/R-devel/src/main/eval.c:1823) ==813836==by 0x4F0243: Rf_eval (svn/R-devel/src/main/eval.c:850) ==813836==by 0x49B68F: do_External (svn/R-devel/src/main/dotcode.c:573) ==813836==by 0x4D3566: bcEval (svn/R-devel/src/main/eval.c:7115) ==813836==by 0x4F0077: Rf_eval (svn/R-devel/src/main/eval.c:727) ==813836==by 0x4F1A8D: R_execClosure (svn/R-devel/src/main/eval.c:1897) (allocate_from_shm() is my function calling allocVector3() with a custom allocator.) Valgrind is correct, because allocVector3() explicitly calls VALGRIND_MAKE_MEM_UNDEFINED() on the memory my custom allocator returns. - Sould allocVector3() call VALGRIND_MAKE_MEM_UNDEFINED() also if a custom allocator is used? For some custom allocators this is correct, for others not. - Or should the code using a custom allocator call VALGRIND_MAKE_MEM_DEFINED() on the DATAPTR() returned by allocVector3()? E.g. ... ret = PROTECT(allocVector3(asInteger(type), asReal(length), &allocator)); VALGRIND_MAKE_MEM_DEFINED(DATAPTR(ret), size); ... For the latter to work also on systems without Valgrind installed, I need to include both valgrind.h and memcheck.h in the src of my package and include these (rather than the system headers), correct? Should I best take these headers directly from R (src/include/vg)? Thanks! Regards, Andreas __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] [Solved] Possible x11 window manager window aggregation under one icon?
> Duncan Murdoch
> on Thu, 25 Mar 2021 10:41:46 -0400 writes:
> On 25/03/2021 9:18 a.m., Dirk Eddelbuettel wrote:
>>
>> On 24 March 2021 at 10:30, Martin Maechler wrote:
>> | For this reason I've committed to R (the trunk, i.e., R-devel,
>> | for R 4.1.0 in a month or so) in svn rev 80110.
>>
>> I just saw that via the (still extremely helpful) RSS feed of SVN
changes and
>> then pulled.
>>
>> You may have missed that Ivan concluded, and I followed, that the _patch
is
>> not needed_. All one needs is to adjust the .desktop file. I posted my
full
>> changelog from the Debian package (of which I currently run a test build
on
>> two Ubuntu machines using the binaries from Launchpad I pointed to).
>>
>> So in that sense I think r80110 may wants to be reverted.
> I'm not sure either if Martin saw your conclusion.
I saw it, but as there were two patches of Ivan, I understood
that the 2nd one (which would even group X11 windows of
unrelated R instances) was unneeded.
I concluded I liked the first one because it would achieve
what's considered "uniformly better" in the sense that it makes
R graphics behave like "all other" desktop applications *and* it
would do so for all possible window manager scheme without any
need of some desktop setting (which a typical user would not
know about, nor know that s?he should/could change).
> I haven't tested R-devel with r80110 yet, but I did make the equivalent
> change in rgl, and have been working with that.
> In Ubuntu, it makes no difference if the .desktop file is changed as you
> describe,
a big "iff" at least conceptually, when in the present case,
Dirk as Debian maintainer of the 'R debian pkg' can make it happen.
What about Redhat/Fedora etc, what about the next cool window manager
on Linux distribution Z? They may change to do what .desktop
does in a different way, etc, or more typically not package R
that way and hence not have a *.desktop equivalent.
> but I think it's an improvement if you don't make that change
> for the usual case. You don't get a ton of icons, you get one rgl icon
> per process.
> In macOS, it does affect the behaviour of windows. During rgl testing,
> I sometimes create 100+ windows. Before the change, the window manager
> put them all over the screen, trying to make the newest one visible.
> After the change (now it knows they're all in the same group), it just
> cascades them down the screen until it hits the bottom, then keeps
> creating tiny windows crammed against the bottom of the screen. I think
> this is negative (the usual reason I create them all is to hope to spot
> bad changes).
> So for a reasonable number of windows the change is an improvement: the
> windows appear grouped. For a very large number of
> windows it's a negative..
> Duncan Murdoch
which probably also depends on your screen size and the
configuration of several tuning parameters of your window
manager etc..
This all started with Dirk saying the R behaves differently than
"all" other applications in this respect, and Ivan found compact
way to change that .. window-manager -- independently which I
still think is a pro.
Given Duncan's use case, may this should become an argument for
x11() and X11.options(), say grouping = c("process", "none", "all")
with match.arg(grouping) used, so the default was "process"
i.e. group things together that belong to the same "process"
(current R-devel), "none" would correspond to the previous
default and "all" would correspond to what the 2nd patch of Ivan
aimed for.
?
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] custom allocators, Valgrind and uninitialized memory
Another idea for the second option. Instead of including the Valgrind headers,
the following could be enough:
#if __has_include()
#include
#else
#define VALGRIND_MAKE_MEM_DEFINED(_qzz_addr,_qzz_len) \
do { \
(_qzz_addr); \
(_qzz_len);\
} while (0)
#endif
I guess the packages are build on the same CRAN machine which also runs the
tests under Valgrind, i.e. valgrind/memcheck.h is available during build of the
package!?
Not sure though if Oracle Developer Studio on Solaris supports __has_include()
...
2021-03-26 08:40 GMT+01:00 "Andreas Kersting" :
> Hi,
>
> In my package bettermc, I use a custom allocator, which hands out already
> defined/initialized memory (mmap of a POSIX shared memory object).
>
> If my code is run in R which was configured --with-valgrind-instrumentation >
> 0, Valgrind will (correctly) complain about uninitialized memory being used,
> e.g.
>
> ==813836== Conditional jump or move depends on uninitialised value(s)
> ==813836==at 0x4F0A9D: getvar (svn/R-devel/src/main/eval.c:5171)
> ==813836==by 0x4D9B38: bcEval (svn/R-devel/src/main/eval.c:6867)
> ==813836==by 0x4F0077: Rf_eval (svn/R-devel/src/main/eval.c:727)
> ==813836==by 0x4F09AF: forcePromise (svn/R-devel/src/main/eval.c:555)
> ==813836==by 0x4F0C57: FORCE_PROMISE (svn/R-devel/src/main/eval.c:5136)
> ==813836==by 0x4F0C57: getvar (svn/R-devel/src/main/eval.c:5177)
> ==813836==by 0x4D9B38: bcEval (svn/R-devel/src/main/eval.c:6867)
> ==813836==by 0x4F0077: Rf_eval (svn/R-devel/src/main/eval.c:727)
> ==813836==by 0x4F1A8D: R_execClosure (svn/R-devel/src/main/eval.c:1897)
> ==813836==by 0x4F2783: Rf_applyClosure (svn/R-devel/src/main/eval.c:1823)
> ==813836==by 0x4DF61D: bcEval (svn/R-devel/src/main/eval.c:7083)
> ==813836==by 0x4F0077: Rf_eval (svn/R-devel/src/main/eval.c:727)
> ==813836==by 0x4F1A8D: R_execClosure (svn/R-devel/src/main/eval.c:1897)
> ==813836== Uninitialised value was created by a client request
> ==813836==at 0x52D5CF: Rf_allocVector3
> (svn/R-devel/src/main/memory.c:2892)
> ==813836==by 0x16B415EA: allocate_from_shm
> (packages/tests-vg/bettermc/src/copy2shm.c:289)
> ==813836==by 0x49D123: R_doDotCall (svn/R-devel/src/main/dotcode.c:614)
> ==813836==by 0x4DA36D: bcEval (svn/R-devel/src/main/eval.c:7671)
> ==813836==by 0x4F0077: Rf_eval (svn/R-devel/src/main/eval.c:727)
> ==813836==by 0x4F1A8D: R_execClosure (svn/R-devel/src/main/eval.c:1897)
> ==813836==by 0x4F2783: Rf_applyClosure (svn/R-devel/src/main/eval.c:1823)
> ==813836==by 0x4F0243: Rf_eval (svn/R-devel/src/main/eval.c:850)
> ==813836==by 0x49B68F: do_External (svn/R-devel/src/main/dotcode.c:573)
> ==813836==by 0x4D3566: bcEval (svn/R-devel/src/main/eval.c:7115)
> ==813836==by 0x4F0077: Rf_eval (svn/R-devel/src/main/eval.c:727)
> ==813836==by 0x4F1A8D: R_execClosure (svn/R-devel/src/main/eval.c:1897)
>
> (allocate_from_shm() is my function calling allocVector3() with a custom
> allocator.) Valgrind is correct, because allocVector3() explicitly calls
> VALGRIND_MAKE_MEM_UNDEFINED() on the memory my custom allocator returns.
>
> - Sould allocVector3() call VALGRIND_MAKE_MEM_UNDEFINED() also if a custom
> allocator is used? For some custom allocators this is correct, for others not.
>
> - Or should the code using a custom allocator call
> VALGRIND_MAKE_MEM_DEFINED() on the DATAPTR() returned by allocVector3()? E.g.
>
> ...
> ret = PROTECT(allocVector3(asInteger(type), asReal(length), &allocator));
> VALGRIND_MAKE_MEM_DEFINED(DATAPTR(ret), size);
> ...
>
> For the latter to work also on systems without Valgrind installed, I need to
> include both valgrind.h and memcheck.h in the src of my package and include
> these (rather than the system headers), correct? Should I best take these
> headers directly from R (src/include/vg)?
>
> Thanks! Regards,
> Andreas
> __
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>
__
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] custom allocators, Valgrind and uninitialized memory
Andreas, Can you briefly describe what it is you are trying to do? In general, no R package would use valgrind directly; it is an optional debugger. Also note _Writing R Extensions_ has a few things to say about how memory destined for R object can and cannot be allocated -- I presume your custom allocator is only for objects you managed and do not return to R? Best, Dirk -- https://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] custom allocators, Valgrind and uninitialized memory
Hi Dirk, Sure, let me try to explain: CRAN ran the tests of my package using R which was configured --with-valgrind-instrumentation > 0. Valgrind reported many errors related to the use of supposedly uninitialized memory and the CRAN team asked me to tackle these. These errors are false positives, because I pass a custom allocator to allocVector3() which hands out memory which is already initialized. However, this memory is explicitly marked for Valgrind as uninitialized by allocVector3(), and I do not initialize it subsequently, so Valgrind complains. Now I am asking if it is correct that allocVector3() marks memory as uninitialized/undefined, even if it comes from a custom allocator. This is because allocVector3() cannot know if the memory might already by initialized. If this is the intended behavior of allocVector3(), then I am looking for the proper way to get rid of these false Valgrind errors. So to be able to more easily spot the true ones ... Which section of _Writing R Extensions_ do you have in mind? I cannot find anything on custom allocators there, but maybe I am using the wrong search terms. No, these object are returned to R and I am not aware that this is a problem / not allowed. Regards, Andreas 2021-03-26 19:51 GMT+01:00 "Dirk Eddelbuettel" : > > Andreas, > > Can you briefly describe what it is you are trying to do? > > In general, no R package would use valgrind directly; it is an optional > debugger. Also note _Writing R Extensions_ has a few things to say about how > memory destined for R object can and cannot be allocated -- I presume your > custom allocator is only for objects you managed and do not return to R? > > Best, Dirk > > -- > https://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org > __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
