On 2/19/20 3:55 AM, Stefan Schreiber wrote:
I have posted this question on R-help where it was suggested to me
that I might get a better response on R-devel. So far I have gotten no
response. The post I am talking about is here:
https://stat.ethz.ch/pipermail/r-help/2020-February/465700.html
My apologies for cross-posting, which I am aware is impolite and I
should have posted on R-devel in the first place - but I wasn't sure.
Here is my question again:
I am currently working through Advanced R by H. Wickham and came
across the `lobstr::obj_size` function which appears to calculate the
size of an object by taking into account whether the same object has
been referenced multiple times, e.g.
x <- runif(1e6)
y <- list(x, x, x)
lobstr::obj_size(y)
# 8,000,128 B
# versus:
object.size(y)
# 24000224 bytes
Reading through `?object.size` in the "Details" it reads: [...] but
does not detect if elements of a list are shared [...].
My questions are:
(1) is the result of `obj_size()` the "correct" one when it comes to
actual size used in memory?
(2) And if yes, why wouldn't `object.size()` be updated to reflect the
more precise calculation of an object in question similar to
`obj_size()`?
Please keep in mind that "actual size used in memory" is an elusive
concept, particularly in managed languages such as R. Even in native
languages, you have on-demand paging (not all data in physical memory,
some may be imputed (all zeros), some may be swapped out, some may be
stored in files (code), etc). Also you have internal and external
fragmentation caused by the "C library" memory allocator, overhead of
object headers and allocator meta-data. On top of that you have the
managed heap: more of internal and external fragmentation, more headers.
Moreover, memory representation may change invisibly and sometimes in
surprising ways (in R it is copy-on-write, so the sharing, but also
compact objects via ALTREP, e.g. sequences). R has the symbol table,
string cache (strings are interned, as in some other language runtimes,
so the price is paid only once for each string). In principle, managed
runtimes could do much more, including say compression of objects with
adaptive decompression, some systems internally split representation of
large objects depending on their size with additional overheads, systems
could have some transparent de-duplication (not only for strings), some
choices could be adaptive based on memory pressure. Then in R, packages
often can maintain memory related to specific R objects, linked say via
external pointers, and again there may be no meaningful way to map that
usage to individual objects.
Not only that what is a size of an object tree is not easy to define.
That information is in addition not very useful, either, because
innocuous changes may change it in arbitrary ways out of control of the
user: there is no good intuition how much that size will change from
intended application-level modifications of the tree. Users of the
system could hardly create a reliable mental model of the memory usage,
because it depends on internal design of the virtual machine, which in
addition can change over time.
As the concept is elusive, the best advice would be don't ask for the
object size, find some other solutions to your problem. In some cases,
it makes sense to ask for object size in some application-specific way,
and then implement object size methods for specific application classes
(e.g. structures holding strings would sum up number of characters in
the strings, etc). Such application-specific way may be inspired by some
particular (perhaps trivial) serialization format.
I've used object.size() myself only for profiling when quickly
identifying objects that are probably very large from objects of trivial
size, where these nuances did not matter, but for that I knew roughly
what the objects were (e.g. that they were not hiding things in
environments).
Intuitively, the choices made by object.size() in R are conservative,
they provide an over-approximation that somewhat intuitively makes sense
at user level, and they reduce surprises of significant size expansion
due to minimal updates. The choices and their limitations are
documented. I think this at least no worse than than say taking into
account sharing, looking at current "size" of compact objects, etc. One
could provide more options to object.size(), but I don't think that it
would be useful.
Best,
Tomas
There are probably valid reasons for this and any insight would be
greatly appreciated.
______________________________________________
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
______________________________________________
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
