Re: [dev-servo] CSS parser architecture: data structures vs. callbacks

[Nicholas Nethercote]

On Mon, Feb 11, 2013 at 10:37 PM, Robert O'Callahan
 wrote:
>
> Couldn't you just make it a pointer into the text of the style sheet? Then
> an auxiliary data structure could be used to quickly extract a line number
> given such a pointer. (E.g. for each 256 byte boundary in the style sheet,
> store the line number at that boundary.)

That's fine so long as you don't have the equivalent of C's #line
defines.  I don't know if CSS has them but JavaScript does (in source
maps) which makes things much more difficult.

Nick
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Minutes of the discussion regarding strings

[Nicholas Nethercote]

On Thu, Jun 6, 2013 at 11:09 AM, Robert O'Callahan  wrote:
> Immutable strings sound good to me.
>
> How hard would it be to add UTF-8 strings to Spidermonkey? JSString already
> has a lot of "modes", perhaps one more wouldn't hurt :-). I'm imagining
> that for anything that required UTF-16 (charAt etc) you'd convert the
> string internals to UTF-16, and for passing into WebIDL we'd convert string
> internals to UTF-8.

I asked this a while ago and Waldo said it would be very difficult,
but I can't remember why.  I've CC'd him... (and Terrence, who has
been doing some string-related changes).

Nick
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Unanswered questions from a Q&A about Servo

[Nicholas Nethercote]

> The pwn2own and Pwnium results demonstrate, at least to me, that memory
> safety is still valuable even in the presence of intricate sandboxing. We
> need defense in depth, and Rust's type system provides a strong layer of
> defense for new code.

Yeah.  IMO, memory safety is a BFD.

Nick
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Rust and Servo internships with the GNOME Outreach Program for Women

[Nicholas Nethercote]

On Sat, Oct 19, 2013 at 8:14 AM, Brian Anderson  wrote:
>
> This year, the Rust and Servo projects are participating in the GNOME
> Outreach Program for Women (OPW).

Nice to hear.

Nick
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] meeting notes (UTF8)

[Nicholas Nethercote]

On Mon, Jul 7, 2014 at 7:51 PM, Cameron Zwarich  wrote:
> Are UTF8-backed (as opposed to Latin1-backed) JS strings with random access 
> going to be a real possibility in SpiderMonkey? It’s obviously possible to 
> make random access work with an appropriate indexing data structure, but 
> popular JS benchmarks are pretty sensitive to string performance.

Jan de Mooij (a.k.a. jandem) is implementing them. My understanding is
that he considered utf8 as well, and concluded that latin1 would be
simpler and give the same performance benefits. But he also said that
if utf8 is needed in the future, the latin1 work should pave the way
substantially.

So I wouldn't count on utf8 soon, but it might happen at some point.

Nick
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Meeting notes (Q3)

[Nicholas Nethercote]

On Mon, Jul 7, 2014 at 8:03 PM, Robert O'Callahan  wrote:
>
> If you want to brainstom about getting to absurdly low memory targets like
> 32MB, I suggest talking to Nick Nethercote. This is where he lives.

I've already been asked about whether SpiderMonkey could be made to
work in that context. My response was "not likely". JS is a
high-level, dynamically-typed, GC'd language, and SpiderMonkey is
designed primarily for high performance, not low memory usage. If you
implemented a new JS engine designed from the ground up for low memory
usage, and you set your expectations really low, then you might be
able to meet those expectations.

Nick
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

[dev-servo] Servo memory usage

[Nicholas Nethercote]

Hi,

With https://github.com/mozilla/servo/pull/2803 applied, the -m option
now gives some measurements from jemalloc. Here's example output after
starting up and viewing about:mozilla on my Linux box.

_category_  : _size (MiB)_
vsize   :  1414.54
resident:88.50
heap-allocated  : 8.53
heap-active : 8.76
heap-mapped :80.00

The heap-mapped number isn't so important -- it appears to map a big
chunk but not actually touch most of it, so it's only using address
space. The heap-allocated and heap-active numbers are more important,
and indicate that the jemalloc heap -- which is used by the Rust code
-- is small.

So there must be a decent chunk of memory being used by third-party
libraries, which don't (AFAIK) use jemalloc, but instead use the
system malloc. I ran Massif (a Valgrind-based memory profiler) to
measure this usage, and I've attached the results. In short, there's
about 30 MiB in the profile, and over 2% of it is the GL driver
allocating memory for GL contexts under glfwCreateWindow(). It's hard
to get good measurements of these allocations without invasive
instrumentation techniques, alas. (I've hit exactly the same issue
recently with Firefox, sigh:
https://blog.mozilla.org/nnethercote/2014/07/07/measuring-memory-used-by-third-party-code/)

There is also some Servo stuff in that output: some font stuff and
some SleeperThread stuff. I wasn't expecting these in this output,
since they are jemalloc allocations that Massif shouldn't be able to
see, but it turns out that jemalloc has some Valgrind annotations that
tell Massif about the allocations. So this profile is actually a mix
of jemalloc and the system malloc. I'm not sure how good the
annotations are, and thus whether the jemalloc heap coverage is
complete or not.

That still leaves a decent chunk of the 88.5 MiB of resident
unaccounted for. So I did some grovelling through /proc//smaps.
As well as the expected stuff (various sizeable anonymous mmaps, and
various libraries, most of which were shared with multiple other
processes), here were the most notable measurements:

- /home/njn/moz/servo/build/servo 6260 kB
- [stack:28150] 1136 kB
- [stack:28145] 6144 kB
- [stack:28144] 2048 kB
- [stack:28141] 8192 kB
- [stack:28139] 7944 kB
- [stack:28136] 5120 kB

The |servo| process isn't surprising, but those are some big stacks.
Much bigger than I'm used to seeing with C or C++ programs. (There
were a few more stacks in the 100s of kBs that I didn't bother
listing.)

So, overall, of that 88.5 MiB resident, the rough breakdown is this:

- jemalloc (Rust) heap: 9 MiB
- gl driver: 23 MiB
- stacks: 32 MiB
- servo executable: 6 MiB
- various libs: ~5 MiB
- not sure: ~12 MiB

Does Rust have any stack-measuring capabilities?

Nick
49.60% (15,745,040B)
- 0x12E135E1: ??? (in /usr/lib/x86_64-linux-gnu/dri/i965_dri.so)
- 0x12F6F938: ??? (in /usr/lib/x86_64-linux-gnu/dri/i965_dri.so)
- 0x12F0B302: ??? (in /usr/lib/x86_64-linux-gnu/dri/i965_dri.so)
- 0x6CA2903: ??? (in /usr/lib/x86_64-linux-gnu/mesa/libGL.so.1.2.0)
- 0x6C7AA40: ??? (in /usr/lib/x86_64-linux-gnu/mesa/libGL.so.1.2.0)
- 0x10D569D: _glfwCreateContext (in /home/njn/moz/servo/build/servo)
- 0x10DBD1B: _glfwPlatformCreateWindow (in /home/njn/moz/servo/build/servo)
- 0x10D37AC: glfwCreateWindow (in /home/njn/moz/servo/build/servo)

08.58% (2,722,848B)
- 0x12D471F4: ??? (in /usr/lib/x86_64-linux-gnu/dri/i965_dri.so)
- 0x12E03086: ??? (in /usr/lib/x86_64-linux-gnu/dri/i965_dri.so)
- 0x12E556F6: ??? (in /usr/lib/x86_64-linux-gnu/dri/i965_dri.so)
- 0x12F6F96B: ??? (in /usr/lib/x86_64-linux-gnu/dri/i965_dri.so)
- 0x12F0B302: ??? (in /usr/lib/x86_64-linux-gnu/dri/i965_dri.so)
- 0x6CA2903: ??? (in /usr/lib/x86_64-linux-gnu/mesa/libGL.so.1.2.0)
- 0x6C7AA40: ??? (in /usr/lib/x86_64-linux-gnu/mesa/libGL.so.1.2.0)
- 0x10D569D: _glfwCreateContext (in /home/njn/moz/servo/build/servo)
- 0x10DBD1B: _glfwPlatformCreateWindow (in /home/njn/moz/servo/build/servo)
- 0x10D37AC: glfwCreateWindow (in /home/njn/moz/servo/build/servo)

06.75% (2,144,284B)
- in 489 places, all below massif's threshold (01.00%)

05.94% (1,884,729B)
- 0x12F0975C: ??? (in /usr/lib/x86_64-linux-gnu/dri/i965_dri.so)
- 0x12E7D63B: ??? (in /usr/lib/x86_64-linux-gnu/dri/i965_dri.so)
- 0x12F9D36A: ??? (in /usr/lib/x86_64-linux-gnu/dri/i965_dri.so)
- 0x12F9D7DC: ??? (in /usr/lib/x86_64-linux-gnu/dri/i965_dri.so)
- 0x12F7010E: ??? (in /usr/lib/x86_64-linux-gnu/dri/i965_dri.so)
- 0x12F0B302: ??? (in /usr/lib/x86_64-linux-gnu/dri/i965_dri.so)
- 0x6CA2903: ??? (in /usr/lib/x86_64-linux-gnu/mesa/libGL.so.1.2.0)
- 0x6C7AA40: ??? (in /usr/lib/x86_64-linux-gnu/mesa/libGL.so.1.2.0)
- 0x10D569D: _glfwCreateContext (in /home/njn/moz/servo/build/servo)
- 0x10DBD1B: _glfwPlatformCreateWindow (in /home/njn/moz/servo/build/servo)
- 0x10D37AC: glfwCreateWindow (in /home/njn/moz/servo/build/servo)

03.30% (1,048,576B)
- 0xC5B099: je_rallocx (in /home/njn/moz/servo/build/servo)
- 0x6B3039: vec::alloc_or_reallo

Re: [dev-servo] Servo memory usage

[Nicholas Nethercote]

On Thu, Jul 10, 2014 at 10:34 PM, Patrick Walton  wrote:
>
> Historically, Rust's large stacks were due to the fact that at the time
> stack growth was removed, rustc had worse codegen than it did today,
> resulting in frames with very large activation records. Coupled with the
> fact that rustc itself uses recursion heavily (probably too heavily), the
> easiest thing to do was to give tasks large stacks by default. Perhaps this
> decision should be revisited, as I think that it may well be the case that
> rustc is going to be the exception, not the rule.
>
> You can tune the size of stack growth when spawning a Rust task. Servo could
> definitely use a stack size audit; I bet a ton of MBs can be shaved off with
> some tuning.

"Large default stack size" doesn't seem to explain it.

I'd guess that the default is 2 MiB, because there are quite a few
entries in smaps like this:

7f54091fe000-7f54093fe000 rw-p  00:00 0
  [stack:28143]
Size:   2048 kB
Rss:  12 kB
Pss:  12 kB

2 MiB has been allocated, but only a tiny amount has been touched and
is thus in RSS.

Here are the two biggest stacks:

7f540a70-7f540b80 rw-p  00:00 0
  [stack:28139]
Size:  17408 kB
Rss:7944 kB
Pss:7944 kB

7f540960-7f540a40 rw-p  00:00 0
  [stack:28141]
Size:  14336 kB
Rss:8192 kB
Pss:8192 kB

For both, |Rss| is big and |Size| (equivalent to vsize) is even bigger.

Nick
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Servo memory usage

[Nicholas Nethercote]

On Thu, Jul 10, 2014 at 10:34 PM, Patrick Walton  wrote:
>>
>> Does Rust have any stack-measuring capabilities?
>
> It doesn't; let's definitely file a Rust issue on that.

I filed https://github.com/rust-lang/rust/issues/15600. I have no idea
how it would be implemented.

Nick
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] WTF-8 encoding for DOM strings and HTML parsing

[Nicholas Nethercote]

On Thu, Oct 9, 2014 at 9:21 PM, Henri Sivonen  wrote:
> On Wed, Oct 8, 2014 at 4:13 PM, Jan de Mooij  wrote:
>
> Has SpiderMonkey ever been instrumented to find out if most strings
> are even just ASCII?

There are some measurements in
https://blog.mozilla.org/javascript/2014/07/21/slimmer-and-faster-javascript-strings-in-firefox/.

But even better, you can visit about:memory and see for yourself. Look
for entries like this:

│   ├──26.43 MB (08.36%) -- strings
│   │  ├──13.98 MB (04.43%) -- malloc-heap
│   │  │  ├──10.84 MB (03.43%) ── latin1
│   │  │  └───3.14 MB (00.99%) ── two-byte
│   │  └──12.45 MB (03.94%) -- gc-heap
│   │ ├───9.05 MB (02.86%) ── latin1
│   │ └───3.40 MB (01.08%) ── two-byte

You can see these stats on a per-zone basis in the "explicit" tree, or
for the entire main runtime under the "js-main-runtime" tree.

"gc-heap" refers to the JSString objects store on the GC heap, some of
which hold the entire string's chars. "malloc-heap" refers to
separately-stored chars for longer strings.

Nick
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] November 2014 work week minutes

[Nicholas Nethercote]

On Tue, Nov 11, 2014 at 8:47 AM, Josh Matthews  wrote:
>
> - arewefastyet/areweslimyet style recording/display of servo data over time
> (mbrubeck)

Relatedly: I still hope to work on about:memory-style measurements for
Servo. It'll hopefully end up on my Q1 goal list. Let me know if it's
not still needed (though I bet it is!)

Thanks.

Nick
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Servo memory usage

[Nicholas Nethercote]

Last year I claimed that Servo had a number of huge stacks, as
reported by /proc//smaps. It turns out that these measurements
are inaccurate -- if a non-stack mapping gets placed next to a stack
mapping, the kernel merges them and the merged segment is marked as a
stack. I've confirmed this happens by looking at the output of strace.
It's also something I should have known since it came up in Firefox in
bug 827691.

Nick

On Fri, Jul 11, 2014 at 12:04 PM, Brian Anderson  wrote:
>
> On 07/10/2014 11:11 PM, Nicholas Nethercote wrote:
>>
>> On Thu, Jul 10, 2014 at 10:34 PM, Patrick Walton 
>> wrote:
>>>
>>> Historically, Rust's large stacks were due to the fact that at the time
>>> stack growth was removed, rustc had worse codegen than it did today,
>>> resulting in frames with very large activation records. Coupled with the
>>> fact that rustc itself uses recursion heavily (probably too heavily), the
>>> easiest thing to do was to give tasks large stacks by default. Perhaps
>>> this
>>> decision should be revisited, as I think that it may well be the case
>>> that
>>> rustc is going to be the exception, not the rule.
>>>
>>> You can tune the size of stack growth when spawning a Rust task. Servo
>>> could
>>> definitely use a stack size audit; I bet a ton of MBs can be shaved off
>>> with
>>> some tuning.
>>
>> "Large default stack size" doesn't seem to explain it.
>>
>> I'd guess that the default is 2 MiB, because there are quite a few
>> entries in smaps like this:
>>
>> 7f54091fe000-7f54093fe000 rw-p  00:00 0
>>[stack:28143]
>> Size:   2048 kB
>> Rss:  12 kB
>> Pss:  12 kB
>>
>> 2 MiB has been allocated, but only a tiny amount has been touched and
>> is thus in RSS.
>>
>> Here are the two biggest stacks:
>>
>> 7f540a70-7f540b80 rw-p  00:00 0
>>[stack:28139]
>> Size:  17408 kB
>> Rss:7944 kB
>> Pss:7944 kB
>>
>> 7f540960-7f540a40 rw-p  00:00 0
>>[stack:28141]
>> Size:  14336 kB
>> Rss:8192 kB
>> Pss:8192 kB
>>
>> For both, |Rss| is big and |Size| (equivalent to vsize) is even bigger.
>>
>
> Rust does have 2MB stacks by default. These giant numbers are surprising.
>
>
> ___
> dev-servo mailing list
> dev-servo@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-servo
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Question about regex crate in util

[Nicholas Nethercote]

On Fri, Feb 27, 2015 at 12:21 PM, Gilles Leblanc
 wrote:
> Hello, I have a question.
>
> The crate components/util uses regular expressions. In it's lib.rs file it
> has the following statements:
>
> #[cfg(target_os="linux")]
> extern crate regex;
>
> Why is the extern crate regex only used under Linux?

Because the only function that uses functions from the `regex` crate
is get_resident_segments(), which is Linux-only. See
https://github.com/servo/servo/blob/master/components/util/memory.rs#L286.

Nick
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

[dev-servo] Draft blog post about memory measurement

[Nicholas Nethercote]

Hi there,

I've written a draft of a blog post about memory profiling in Firefox
vs. Servo, which focuses on the Rust features that make the task more
pleasant in Servo. There's a draft copy here:
http://njn.valgrind.org/measuring.html

I'd appreciate any feedback you all might have, esp. if I've gotten
anything wrong in the Servo/Rust part.

Thank you.

Nick
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Draft blog post about memory measurement

[Nicholas Nethercote]

Thanks for the comments so far. I will address them in the final version.

BTW, some eager beaver posted my link to Hacker News where it made the
front page. I knew this post would be perfect linkbait for HN but this
was a little premature. Whoops.

Anyway, I edited to the page to remove all the WordPress gunk and to
make it clear that it's a draft. Full details here:
https://news.ycombinator.com/item?id=9642833. I'll post the final
version on my real blog in the next day or two.

Nick

On Sun, May 31, 2015 at 8:39 PM, Nicholas Nethercote
 wrote:
> Hi there,
>
> I've written a draft of a blog post about memory profiling in Firefox
> vs. Servo, which focuses on the Rust features that make the task more
> pleasant in Servo. There's a draft copy here:
> http://njn.valgrind.org/measuring.html
>
> I'd appreciate any feedback you all might have, esp. if I've gotten
> anything wrong in the Servo/Rust part.
>
> Thank you.
>
> Nick
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Draft blog post about memory measurement

[Nicholas Nethercote]

On Mon, Jun 1, 2015 at 8:46 AM, Andrew McCreight  wrote:
>
> This is probably too esoteric for the blog post, but it seems like the
> malloc_size_of thing isn't passed in for the Rust version. (For those who
> don't know, in Firefox, this is used for the Dark Matter Detector (DMD): a
> method can be passed in to record all blocks that are measured via the
> memory reporters, to then report the allocation stacks for blocks that
> aren't reported, to find places that are missing reporters.) I assume that
> this is just because there's no DMD set up yet?

Yep, exactly.

> I also assume that
> basically you can just modify the derive generator code, which would be
> really great, compared to the hideous giant patches that are required when
> modifying the reporter boiler plate in Firefox. I don't know if it is worth
> calling out that global changes can be made locally, or maybe that's just
> implicit in the "less boilerplate" part of things.

Yeah, the plugin code is pretty short and is in the Servo repo (or
will be when it lands).

Nick
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Draft blog post about memory measurement

[Nicholas Nethercote]

On Mon, Jun 1, 2015 at 8:59 AM, Jack Moffitt  wrote:
>
> One question I still had at the end was whether things like Arc
> (non-owning references) are accounted for somewhere?

If there's a clear owner, then it makes sense to measure the struct
from there. Otherwise, we'll have to do some kind of "divide by the
refcount", but Firefox doesn't even have that yet because the amount
of memory that falls into that case has never been very high. Servo
may be different; I don't yet know.

Nick
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Draft blog post about memory measurement

[Nicholas Nethercote]

On Mon, Jun 1, 2015 at 4:07 PM, Azita Rashed  wrote:
> This is very nice.  One small note in the beginning of your post, you called 
> Servo an experimental browser engine.  We should take out the word 
> experimental and just call it a new or next generation browser engine.  The 
> project isn’t in the experimental phase anymore and we don’t want to give 
> that impression to the readers.

Will do!

Nick
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

[dev-servo] Best way to work on multiple branches concurrently?

[Nicholas Nethercote]

Hi,

I often like to work on multiple branches concurrently, to work around
the fact that Servo builds are slow. The obvious way to do this is to
just have multiple clones. That's what I do with Firefox (using
Mercurial).

I've read that git 2.5 will have a `git checkout --to=path` command
which will support multiple working branches from a single repo
directly. (See 
http://stackoverflow.com/questions/6270193/multiple-working-directories-with-git/30185564#30185564.)
So that's nice.

But for Servo there's the added complication of having a rustc and a
.cargo/ directory. It'd be nice to avoid having to download rustc
twice if both clones require the same version. It'd also be nice to
avoid building all the code in the .cargo/ directory twice, at least
(again) when the versions match up. IIRC there is a way to specify
that the .cargo/ directory should go somewhere else (and indeed it
used to go in $HOME by default), though I don't know what it is.

Anyway, I'm just wondering how other people deal with this. Thanks.

Nick
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Multiprocess safety

[Nicholas Nethercote]

The memory profiler relies heavily on passing channels over channels.
The memory profiler makes measurement requests of threads and passes
them a channel through which they can return the measurements.

One example from paint_task.rs:

impl Reporter for PaintChan {
// Just injects an appropriate event into the paint task's queue.
fn collect_reports(&self, reports_chan: ReportsChan) -> bool {
let PaintChan(ref c) = *self;
c.send(Msg::CollectReports(reports_chan)).is_ok()// pass a
channel over a channel!
}
}

In multi-process Servo it would make sense to have one memory profiler
per process, so I'm not sure if this is a showstopper or not.

Nick

On Thu, Jun 18, 2015 at 4:40 PM, Patrick Walton  wrote:
> Hi everyone,
>
> I'm rebasing my multiprocess patch and found that many of the new features
> that have landed were implemented in a way that was incompatible with
> multiprocess Servo. In particular:
>
> * WebDriver passes channels over channels between script and the chrome
> process.
>
> * Canvas relies on passing channels over channels between code that can
> touch the GPU and script.
>
> * The new messages for pipeline/iframe control rely on passing channels
> over channels between the chrome process and the script task.
>
> To be sure, it's my fault that I wasn't keeping the multiprocess branch
> current. I didn't anticipate so much breakage, though, and in order to fix
> these issues I will have to make severe modifications to all of these
> features. (I'm two days into rebasing and large crates like compositing and
> script aren't even building yet, and I've had to disable lots of features
> such as canvas.)
>
> In an effort to keep this sort of thing from happening again, I'd like to
> suggest that all new code that spawns threads and passes channels or boxed
> objects over channels not be allowed to pass review until audited for
> multiprocess safety up until the multiprocess branch lands. Obviously, all
> other browser engines are cautionary tales for the amount of technical debt
> that can accrue if this is allowed to continue to happen unchecked. Even
> though we've been good about keeping things in separate threads, the
> passing-channels-over-channels pattern does not work among separate
> processes, and the more we rely on it the harder it will be to harden Servo.
>
> Thoughts?
> Patrick
> ___
> dev-servo mailing list
> dev-servo@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-servo
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Multiprocess safety

[Nicholas Nethercote]

On Thu, Jun 18, 2015 at 5:45 PM, Patrick Walton  wrote:
> It's fine to pass channels over channels as long as those channels don't
> cross process boundaries. In that case, the paint task lives in the chrome
> process (since it has access to the GPU), so you're fine. The problematic
> cases are essentially {script,layout} <-> (anything else).

There's a similar case for the layout task, in
components/script/layout_interface.rs

impl Reporter for LayoutChan {
// Just injects an appropriate event into the layout task's queue.
fn collect_reports(&self, reports_chan: ReportsChan) -> bool {
let LayoutChan(ref c) = *self;
c.send(Msg::CollectReports(reports_chan)).is_ok()
}
}

And the script task will presumably be instrumented eventually.

If there is one memory profiler per process then it might require some
arrangement, but it'll probably work out ok.

Nick
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] The rules of @bors-servo: retry

[Nicholas Nethercote]

On Tue, Aug 4, 2015 at 9:14 AM, James Graham  wrote:
>
> 2) Start posting the results of test runs to treeherder, which will surface
> the failures in a clearer way than the buildbot waterfall and will hopefully
> in the near future get a measure of auto-matching test results against known
> intermittents (this will probably be internal rather than using any
> bugtracker as a backend, so it should be possible to also link specific
> intermittents to github issues if you want that feature).

For those who aren't familiar with Firefox development, when you get
an intermittent failure in Firefox tests (which is very frequent; it's
rare to get a full test run that doesn't have at least one) treeherder
will auto-suggest existing bugs that match the failure. This is
*enormously* helpful for working out whether it's a known
intermittent. It also requires that people are assiduous about filing
bugs for intermittents. For Firefox the sheriffs do a great job of
this.

More generally, intermittent test failures are the thing that really
stresses the Not Rocket Science Rule of Software Engineering
(http://graydon.livejournal.com/186550.html). If you're writing a
batch program like a compiler, it tends to work great. If you're
writing a complex interactive app with lots of time-sensitive aspects,
it gets harder.

Nick
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] PSA: import blocks are now sorted

[Nicholas Nethercote]

On Thu, Aug 20, 2015 at 7:54 AM, Ms2ger  wrote:
>
> Consecutive lines of imports are now sorted, thanks to Johann Tuffe.
> test-tidy will complain about any violations.

You're talking about `use` items, right? It looks like they're sorted
within each group, but you can still have separate groups. (Where the
boundary between a group is a blank line.) Is that right?

Nick
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] PSA: import blocks are now sorted

[Nicholas Nethercote]

On Fri, Aug 21, 2015 at 4:40 AM, Ms2ger  wrote:
>>
>> You're talking about `use` items, right? It looks like they're
>> sorted within each group, but you can still have separate groups.
>> (Where the boundary between a group is a blank line.) Is that
>> right?
>
> That's correct.

Unfortunately the group ordering is inconsistent and sometimes wrong.

For example, in some files it goes:
- imports from this crate
- imports from other Servo crates
- imports from non-Servo crates
- imports from built-in crates

Which is easy to get wrong, especially for part-time contributors
(like myself) who don't have a clear idea which crates are in which
group. For example, the very first file I looked in was
components/script/script_task.rs, and it has *five* import groups,
with the fifth one containing some |hyper| imports that should be with
the other |hyper| imports in the third group.

The second file I looked at was components/layout/layout_task.rs which
only has two groups. I *think* the grouping is as follows:
- imports from this crate
- all other imports

But I can't tell for sure.

Rust as a language avoids so many of these silly, arbitrary decision
points; it's a shame that Servo has found a place to introduce one.
Especially given that import lists are modified frequently, so the
question of "where does this new import get inserted" comes up
frequently. I'd argue for "all one group" unless test-tidy can be made
to check more complex rules.

Nick
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] PSA: Review delegation enabled for homu

[Nicholas Nethercote]

On Wed, Nov 11, 2015 at 6:31 PM, Bobby Holley  wrote:
>
> From a project standpoint, I am trying to blaze the trail of the
> committer-but-not-reviewer role

IME having to re-get r+ every time you have a trivial merge conflict
or minor CI failure (e.g. a "tidy" failure due to trailing whitespace)
is a huge PITA. I remember one bug where I had to ask gw four times
over the course of an entire day to re-give r+ for such things. It's
infantilising, and I'm 95% sure he didn't look at my code changes --
he didn't review the original patch.

On the other hand, I'm more optimistic than bholley about the per-bug
delegation. Given that it's been implemented I'd suggest using it and
if reviewers do tend to forget it then consider implementing a broader
mechanism.

Nick
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Hello and intro!

[Nicholas Nethercote]

On Tue, Feb 9, 2016 at 2:20 AM, Simon Sapin  wrote:

Feel free to ask questions on this list or on the #servo IRC channel:
> https://wiki.mozilla.org/IRC
>

I second that. The #rust channel is also really good. Both channels are
full of friendly and helpful people. Don't be afraid to ask questions, even
for small and/or simple things.

Nick
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Weekly status updates

[Nicholas Nethercote]

I'm not a Servo person, but...

I think hand-written updates are *much* better than auto-generated
updates, because you can put interesting context and background into
hand-written updates. (In my long experience at Mozilla I enjoy
reading hand-written updates of various kinds, but auto-generated ones
I invariable ignore.) They're a great way to communicate what you've
been doing to other people. They're also a good way to be aware of
your own progress and self-motivate, e.g. "that was a really
productive week" or "that was a crappy week, I'll try to do better
next week".

I also don't think a weekly update is onerous. I have a calendar
reminder which means I don't forget it. I have a text file in which I
write down things when I land them. I have a Bugzilla search that
finds patches I've landed in the past 7 days which I use to check if
I've forgotten to add anything to the text file. So when I go to
submit my update it's already 90% complete. The whole thing accounts
for maybe 10 minutes of effort a week. Maybe I'm a paragon of
organization and efficiency, but it really doesn't seem that hard. For
employees in particular, "keep your colleagues informed" is a basic
responsibility and this is an easy and effective way to do it.

As for moving to a different system -- are there any features that
Status Updates is missing? It's not a complicated thing. Does it need
more than a textbox and a "send" button?

Nick

On Fri, Sep 23, 2016 at 2:07 AM, Lars Bergstrom  wrote:
> I've noticed that while some people are still submitting their weekly
> status updates on http://statusupdates.dev.mozaws.net/ (thank you!),
> it's both not everybody and we're missing a bunch of the more active
> community members who might like to highlight their work.
>
> I'd love to either get that moving again or move to something that
> people like more (e.g., edunham suggested http://standu.ps/ ).
>
> I find this really useful on the team so that we all know what each
> other are working on, especially since we cancelled the weekly
> meeting. It's also included externally in "This Week in Servo" so
> others get a sense for what's in progress but not yet landing.
> Finally, internally it's helpful for people funding the project to
> have a general sense for what everybody's doing without having to ask
> myself or Jack :-)
>
> Opinions? I'm also open to pushback here that status reporting is too
> much work and larsberg should just be scraping scraping harder on
> GitHub.
>
> Thanks,
> - Lars
> ___
> dev-servo mailing list
> dev-servo@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-servo
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Incremental compilation and other compile time tricks

[Nicholas Nethercote]

`mach cargo check` is worth listing in Servo's README.md, IMO!

Nick

On Sat, Apr 1, 2017 at 4:43 AM, Simon Sapin  wrote:

> If you build Servo a lot you may have noticed that compilation times have
> improved over the last few months. This is combination of multiple factors.
>
>
> # Compiler optimization
>
> Plain old optimization of the compiler’s code. Sometimes a rustup makes
> things a bit faster without us changing anything else. For example:
>
> https://blog.mozilla.org/nnethercote/2016/10/14/how-to-speed
> -up-the-rust-compiler/
> https://blog.mozilla.org/nnethercote/2016/11/23/how-to-speed
> -up-the-rust-compiler-some-more/
>
>
> # Limited parallelism
>
> Cargo can compile multiple crates at the same time, but a crate can only
> start after its dependencies are done. And in the default configuration,
> compiling one crate only uses one CPU core/thread. Servo’s dependency graph
> tends to be narrow near the end, in particular with the script crate that
> takes a lot of time to compile. This limits the overall available
> parallelism.
>
>
> # Hardware
>
> CPUs with many cores tend to make each core slower, to manage heat.
> Counter-intuitively this can make then worse for compiling Servo. If you’re
> spending (your employer’s) money on hardware, I recommend the highest
> single-thread performance you can find even if this means fewer cores.
> Currently, this is probably Intel’s i7-7700K. (4 cores, 8 threads, 4.2 GHz,
> easy to overclock to 4.6 GHz.)
>
>
> # Codegen units
>
> Compilation of a crate can be roughly split into two phases: analysis and
> code generation. The former includes parsing, type checking, borrow
> checking, etc. The latter is mostly LLVM.
>
> rustc added a feature to get parallelism in code generation. With `rustc
> -C codegen-units=4` for example, the code to generate is split into four
> parts that are given separately to LLVM in threads that run in parallel.
>
> This makes compilation faster but has a runtime cost: LLVM’s optimizer
> only sees one unit at a time, which reduces opportunities for inlining and
> other compile-time optimizations.
>
> Also, only code generation is parallelized, the analysis phase still runs
> on a single thread. So Amdahl’s law limits how much speed improvement we
> can get with this.
>
> In Servo we’ve enabled codegen-units=4 by default in debug mode, but not
> in release mode.
>
> https://github.com/servo/servo/pull/14995
>
>
> # LLVM assertions
>
> When compiling LLVM (and so when compiling rustc) we can choose to enable
> LLVM assertions. There assertions can help find bugs (in LLVM or in rustc),
> but they have a runtime cost. (LLVM’s runtime, which is Servo’s compile
> time.) This cost can be significant, especially in release mode (where
> LLVM’s optimizer does a lot of work).
>
> Official Rust builds have LLVM assertions enabled in Rust Nightly (to help
> catch bug), but not in the beta or stable release channels.
>
> At our request, the Rust team started making alternative Rust Nightly
> builds without LLVM assertions, for some platforms. We’re now using them by
> default, except for some of our Continuous Integration builders (in order
> to preserve some of the coverage).
>
> https://github.com/rust-lang/rust/pull/39754
> https://github.com/servo/servo/pull/15559
> https://github.com/servo/servo/pull/15564
>
>
> # cargo check
>
> Remember how I said a crate needs its dependencies to be compiled before
> it can start compiling? It actually only needs metadata (the results of the
> analysis phase), not the generated code.
>
> Cargo added a `cargo check` command that skips code generation entirely
> (except for build scripts and build-dependencies). This saves a lot of time
> (I’ve measured up to 5× speed from `./mach clean`), though obviously you
> don’t get an executable at the end. Still, it can help for example during a
> refactoring: run `./mach cargo check` many times until compiler error are
> resolved, and only then run `./mach build` and/or `./mach test-unit`.
>
> https://github.com/rust-lang/cargo/pull/3296
> https://github.com/servo/servo/pull/14594
>
> Follow-up work wanted: add a mach sub-command or option to do this with
> geckolib.
>
> (It would also be nice for rustc to have a mode to write both metadata and
> full build, and signal somehow as soon as the former is ready; and have
> Cargo start building dependents on that signal while codegen for the
> dependencies is still running, increasing overall parallelism.)
>
>
> # Preserving compilation output between CI runs
>
> Previously, Servo’s Continuous Integration was configured to run `git
> clean` at the start of each build, with options such that anything not
> checked into the repository was removed. This included the `target`
> directory, so each pull request was recompiled from scratch.
>
> We’ve changed the configuration to preserve files in .gitignore, which
> includes the target directory, so that some of the compilation output for
> previous builds can be reused. T

Re: [dev-servo] Pre-alpha of libservo available

[Nicholas Nethercote]

What is the backwards compatibility story? Are the APIs stable?

I ask because, as I understand it, API instability has always been the
problem with embedding Firefox.

Nick

On Thu, Sep 21, 2017 at 2:03 AM, Till Schneidereit <
t...@tillschneidereit.net> wrote:

> Hello friends of Servo!
>
> We're happy to announce the availability of a pre-alpha version of
> libservo, a Rust embedding API for Servo.
>
> You can find documentation for the API and a tutorial for using it at the
> following locations:
> https://doc.servo.org/servo/struct.Servo.html
> http://github.com/paulrouget/servo-embedding-example
>
> Binaries for demo applications for the three desktop platforms can be found
> here:
> https://github.com/paulrouget/servoshell/releases
>
>
> Please take a look, play around with the API and the demo applications, and
> let us know what you think.
>
> With the TL;DR out of the way, let's take a look at the details.
>
>
> ## What is this announcement about?
>
> To some extent, Servo was always embeddable. Over the past year a lot of
> effort has gone into the design and implementation of a clean low-level API
> - libservo. The API is designed for maximal flexibility, with the goal
> being to cover all use cases for Servo - ranging from simple webview-like
> uses to fully-fledged browsers to server-side uses as a headless utility.
>
> We do not anticipate most embedders to use this API directly: to address
> the use cases of the majority of embedders, we intend to build higher-level
> APIs on top of this. Those will range from higher-level Rust APIs to an
> implementation of CEF[1] to drop-in replacements for the Android WebView
> component[2].
>
> Because most embedders won’t be using this low-level API directly, we are
> choosing flexibility over usability when adding functionality.We want the
> API to be as usable as possible - but not more so. The API requires
> embedders to provide a GL rendering context and manually drive the event
> loop to ensure rendering and input processing. It also allows fine-grained
> influence on behavior such as navigation. Going forward, it'll grow more
> capabilities along those lines, such as rendering to buffers or output of
> raw display lists to be consumed by compositing systems such as a
> customized WebRender, or delegating all I/O to the embedder.
>
>
>
> [1] https://en.wikipedia.org/wiki/Chromium_Embedded_Framework
> [2] https://developer.android.com/reference/android/webkit/WebView.html
>
>
> ## What's the current status?
>
> At this point libservo is far from complete, but ready for early
> experimentation. Crate-level documentation is here:
> https://doc.servo.org/servo/struct.Servo.html
>
> A How To guide for setting up a project to use libservo can be found here:
> 
> http://github.com/paulrouget/servo-embedding-example
>
> In addition to this example application Paul has built two additional demo
> apps: a cross-platform minimalist browser with as reduced a UI as is
> possible and a more fully-fledged macOS version with a native tabbed UI and
> a proper location bar. These are both implemented as ports of the
> ServoShell project:
> https://github.com/paulrouget/servoshell
>
> Releases for the three desktop platforms are available as github releases:
> https://github.com/paulrouget/servoshell/releases
>
> We're happy with the overall shape of the API, but expect at least cosmetic
> changes to most of it. Additionally, there is a lot of functionality
> missing, so you should see this as the kernel of an API to be fleshed out.
>
> Additionally, there are a number of issues making it harder to get a
> project up and running than it should be. From a build system and
> dependency management perspective, embedding Servo should be as simple as
> adding it as a dependency to a crate's Cargo.toml. For now Paul's
> above-mentioned embedding example has a detailed explanation of the
> required steps.
>
>
> ## What's next?
>
> Our highest priority right now is to streamline the process. That means
> reducing the build system complexity and doing the above-mentioned cosmetic
> changes.
>
> We're also working on more examples and demos. Specifically, Paul is
> experimenting with a simple C API and with an integration into a Xamarin
> Forms-based C# application.
>
> We'll share more information about those experiments and about our plans
> for more high-level ways of embedding Servo, such as bindings for
> high-level languages like Java and JavaScript.
>
> For now, please download the demo applications, take a look at the
> documentation, play around with the example code, and let us know what you
> think: ping Paul or me in #servo on IRC, file issues in the servo/servo or
> repository or Paul's ServoShell or Servo Embedding Example repositories, or
> simply reply to this email.
>
> We'd be particularly interested in experiments around integrating Servo
> into platform-native toolkits. E.g., a Gtk-based Linux equivalent

[dev-servo] Memory reporting in Servo: the next step

[Nicholas Nethercote]

Hi,

Currently we have two similar but distinct approaches to measuring memory
usage
in Servo.

- Servo uses the heapsize and heapsize_derive crates, from crates.io.

- Gecko uses the malloc_size_of and malloc_size_of_derive crates, which are
in
  the tree.

Because of this, you see this pattern quite a bit in style code:

> #[cfg_attr(feature = "gecko", derive(MallocSizeOf))]
> #[cfg_attr(feature = "servo", derive(HeapSizeOf))]
> struct Foo {
> ...
> }

Why the difference? heapsize is the original approach. It sorta works, but
has
some big flaws. malloc_size_of is a redesign that addresses these flaws.

- heapsize assumes you only want a single number for the size of any data
  structure, when sometimes you want to break it down into different
buckets.

  malloc_size_of provides both "shallow" and "deep" measurements, which give
  greater flexibilty, which helps with the multi-bucket case.

- heapsize assumes jemalloc is the allocator. This causes build problems in
  some configurations, e.g. https://github.com/servo/heapsize/issues/80.
  It also means it doesn't integrate with DMD, which is the tool we use to
  identify heap-unclassified memory in Firefox.

  malloc_size_of doesn't assume a particular allocator. You pass in
functions
  that measure heap allocations. This avoids the build problems and also
allows
  integration with DMD.

- heapsize doesn't measure HashMap/HashSet properly -- it computes an
estimate
  of the size, instead of getting the true size from the allocator. This
  estimate can be (and in practice often will be) an underestimate.

  malloc_size_of does measure HashMap/HashSet properly. However, this
requires
  that the allocator provide a function that can measure the size of an
  allocation from an interior pointer. (Unlike Vec, HashMap/HashSet don't
  provide a function that gives the raw pointer to the storage.) I had to
add
  support for this to mozjemalloc, and vanilla jemalloc doesn't support it.
(I
  guess we could fall back to computing the size when the allocator doesn't
  support this, e.g. for Servo, which uses vanilla jemalloc.)

- heapsize doesn't measure Rc/Arc properly -- currently it just defaults to
  measuring through the Rc/Arc, which can lead to double-counting.
Especially
  when you use derive, where it's easy to overlook that Rc/Arc typically
need
  special handling. (This is https://github.com/servo/heapsize/issues/37.)

  malloc_size_of does measure Rc/Arc properly. It lets you provide a table
that
  tracks which pointers have already been measured, which is used to prevent
  double-counting. malloc_size_of also doesn't implement the standard
  MallocSizeOf trait for Rc/Arc, which means they can't be unintentionally
  measured via derive. (You can still use derive if you explicitly choose to
  ignore all Rc/Arc fields, however.)

Basically, malloc_size_of is heapsize done right, and using both is silly. I
went with this dual-track approach while adding memory reporting to Stylo
because time was tight and the exact design choices required to handle all
the
necessary cases weren't clear. But now that things have settled down I'd
like
to pay back this technical debt by removing the duplication.

I see two options.

- Overwrite the heapsize crate on crates.io with the malloc_size_of code. So
  the crate name wouldn't change, but the API would change significantly,
and
  it would still be on crates.io. Then switch Servo over to using heapsize
  everywhere.

- Switch Servo over to using malloc_size_of everywhere. (This leaves open
the
  question of what should happen to the heapsize crate.)

I personally prefer the second option, mostly because I view all of this
code
as basically unstable -- much like the allocator APIs in Rust itself -- and
publishing it on crates.io makes me uneasy. Also, keeping the code in the
tree
makes it easier to modify.

Thoughts?

Nick
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Memory reporting in Servo: the next step

[Nicholas Nethercote]

On Fri, Oct 6, 2017 at 2:33 PM, Xidorn Quan  wrote:

>
> There are multiple Servo dependencies on crates.io depend on heapsize.
> [1] How are they handled at the moment? And what would we do to them in
> the future with the second option?
>
> [1] https://crates.io/crates/heapsize/reverse_dependencies
>

Good question.

There are two ways for the HeapSizeOf trait to be implemented for a type
Foo.

(1) Foo's crate depends on heapsize, in which case Foo's crate can
implement HeapSizeOf itself. The crates at that link above (app_units,
euclid, url) are examples. The advantage of this option is that the
HeapSizeOf trait can access non-public fields of types.

(2) Foo's crate doesn't depend on heapsize, in which case a different crate
must implement HeapSizeOf for Foo. This could be the heapsize crate itself
-- as is done for all the standard types like Vec and HashMap -- or it
could be a different crate. The advantage of this option is that you don't
depend on heapsize.

If we went with the second option from my original mail (i.e. stop using
heapsize) then it would probably make sense to deprecate heapsize entirely,
stop those crates from depending on it, and implement the MallocSizeOf
trait for those crates' types within Servo.

Nick



Nick
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Memory reporting in Servo: the next step

[Nicholas Nethercote]

On Fri, Oct 6, 2017 at 4:07 PM, Manish Goregaokar 
wrote:
> I kinda feel like upgrading HeapSizeOf to have a generic second
trait-bound
> argument that we substitute MallocSizeOfOps as would be good.

What problem would that solve? I don't want to add unnecessary abstraction.

Nick
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Memory reporting in Servo: the next step

[Nicholas Nethercote]

On Fri, Oct 6, 2017 at 4:15 PM, Manish Goregaokar 
wrote:

> If we do impls in the mallocsizeof crate is we can't make use of the custom
> derive functionality and have to manually write out impls (which in many
> cases won't be possible).


Not being able to use derive doesn't worry me, because it's just a
convenience. I just looked through all the external crates that implement
HeapSizeOf and hardly any are using derive.

Lack of access to internal fields is more of a concern, because that can
make a type impossible to measure. However... a crate can either implement
HeapSizeOf, or it can provide functions that give sufficient internal
access such that HeapSizeOf can be implemented externally. (E.g. in
smallbitvec mbrubeck added a heap_ptr() method on my request:
https://docs.rs/smallbitvec/1.0.7/smallbitvec/struct.SmallBitVec.html#method.heap_ptr.)
But in either case we probably need to control or have influence over the
crate authorship, so to some degree it doesn't really matter.

Nick
___
dev-servo mailing list
dev-servo@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-servo

Re: [dev-servo] Memory reporting in Servo: the next step

[Nicholas Nethercote]

Memory reporting as done for about:memory is quite different to the memory
profiling done for devtools.

Here's how memory reporting works in SpiderMonkey: we iterate over every
cell in the GC heap, and measure any malloc'd things hanging of those
cells. That's it. It's not at all a standard GC type tracing algorithm
because it treats reachable and unreachable cells identically. It's simple
and works well and I don't want to change it.

Nick

On Sat, Oct 7, 2017 at 4:35 AM, Nick Fitzgerald 
wrote:

> Hi Nick!
>
> The combination of deriving traits and Rust's ownership model is great for
> memory reporting, and this stuff is _so_ much nicer to define than the
> equivalent about:memory measurements in Firefox.
>
> But it doesn't play out as well in the presence of a GC: ownership is
> muddied. The Arc/Rc issues you mention are the tip of the iceberg, as
> ownership is relaxed and heap edges proliferate. In the general case of
> arbitrary edges that GCs support, one needs to analyze the heap graph to
> get equivalent information about why something is alive and how much heap
> space it is responsible for. Things like computing dominator trees and the
> shortest path to some object from GC roots. For everything in Servo relying
> on SpiderMonkey's GC, I think it makes sense to use the `JS::ubi::Node`
> infrastructure[0] from SpiderMonkey, which gives a graph API to the GC
> heap, and has various analyses (dominator trees, shortest paths, stack at
> allocation time, etc) out of the box. This is what we use for the Firefox
> DevTools' memory tool.
>
> Of course, `JS::ubi::Node` is fairly heavily templated and has many inline
> functions, so exposing this infrastructure to Rust will take some care and
> effort.
>
> Backing up a bit: there is a large benefit in separating the heap graph
> traversal from the analyses run on the heap graph.
>
> * With a separation, we can traverse the heap graph at some particular
> point in time, serialize it to a core dump file, and then perform as much
> post processing and analyzing as we like, without needing to reproduce that
> exact heap graph over and over. This is similar to how rr separates
> reproducing a bug from debugging it.
>
> * With a separation, the bucketing of various types and their collective
> heap size is just *one* analysis of the heap graph. We could have many
> others, like the ones I've mentioned above: dominator trees, shortest path
> to GC roots (I guess to a stack variable or global in general Rust),
> dynamic leak detection[1], etc. The possibilities are open ended.
>
> So maybe instead of using `JS::ubi::Node`, what do you think of creating
> `#[derive(HeapGraph)]` and implementing `MallocSizeOf`, etc on top of it?
>
> If `#[derive(HeapGraph)]` implemented `petgraph`'s graph traits, we would
> even get shortest paths and dominators[2] for free.
>
> Interested in your thoughts on the subject!
>
> Cheers,
>
> Nick
>
> [0] http://searchfox.org/mozilla-central/source/js/public/UbiNode.h#32-164
> [1] http://www.cs.utexas.edu/ftp/techreports/tr06-07.pdf
> [2] https://github.com/bluss/petgraph/blob/master/src/algo/dominators.rs
>
> On Thu, Oct 5, 2017 at 6:27 PM, Nicholas Nethercote <
> n.netherc...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Currently we have two similar but distinct approaches to measuring memory
> > usage
> > in Servo.
> >
> > - Servo uses the heapsize and heapsize_derive crates, from crates.io.
> >
> > - Gecko uses the malloc_size_of and malloc_size_of_derive crates, which
> are
> > in
> >   the tree.
> >
> > Because of this, you see this pattern quite a bit in style code:
> >
> > > #[cfg_attr(feature = "gecko", derive(MallocSizeOf))]
> > > #[cfg_attr(feature = "servo", derive(HeapSizeOf))]
> > > struct Foo {
> > > ...
> > > }
> >
> > Why the difference? heapsize is the original approach. It sorta works,
> but
> > has
> > some big flaws. malloc_size_of is a redesign that addresses these flaws.
> >
> > - heapsize assumes you only want a single number for the size of any data
> >   structure, when sometimes you want to break it down into different
> > buckets.
> >
> >   malloc_size_of provides both "shallow" and "deep" measurements, which
> > give
> >   greater flexibilty, which helps with the multi-bucket case.
> >
> > - heapsize assumes jemalloc is the allocator. This causes build problems
> in
> >   some configurations, e.g. https://github.com/servo/heapsize/issues/80.
> >   It also means it doesn't integrate with DMD, which i