Re: We live in a memory-constrained world

2014-03-04 Thread Nicholas Nethercote
On Mon, Mar 3, 2014 at 11:48 PM, Henri Sivonen wrote: > > Are static atoms and the HTML parser's pre-interned element name and > attribute name objects that are on the heap shared between processes > under Nuwa already? I.e. is the heap cloned with copy-on-write > sharing? On the memory page granu

Re: We live in a memory-constrained world

2014-03-04 Thread Trevor Saunders
On Tue, Mar 04, 2014 at 09:48:33AM +0200, Henri Sivonen wrote: > On Fri, Feb 28, 2014 at 8:09 PM, L. David Baron wrote: > > In other words, whenever you have a pointer in a static data > > structure pointing to some other data, that pointer needs to get > > fixed up when the library loads, which m

Re: We live in a memory-constrained world

2014-03-03 Thread Henri Sivonen
On Fri, Feb 28, 2014 at 8:09 PM, L. David Baron wrote: > In other words, whenever you have a pointer in a static data > structure pointing to some other data, that pointer needs to get > fixed up when the library loads, which makes the memory that pointer > is in less likely to be shared across pr

Re: We live in a memory-constrained world

2014-02-28 Thread L. David Baron
On Friday 2014-02-28 11:04 +, Neil wrote: > Henri Sivonen wrote: > >Any chance static atoms could reside as plain old static C data structures > >in the data segment of libxul.so instead of being heap-allocated? > > > At least under MSVC, they have vtables, so they need to be > constructed, so

Re: We live in a memory-constrained world

2014-02-28 Thread Boris Zbarsky
On 2/28/14 12:26 PM, Neil wrote: Perhaps we could probably use string buffers directly as atoms. Atoms have various info string buffers don't (like the hashcode). -Boris ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozil

Re: We live in a memory-constrained world

2014-02-28 Thread Neil
Neil wrote: Henri Sivonen wrote: Any chance static atoms could reside as plain old static C data structures in the data segment of libxul.so instead of being heap-allocated? At least under MSVC, they have vtables, so they need to be constructed, so they're not static. Note that their strin

Re: We live in a memory-constrained world

2014-02-28 Thread Trevor Saunders
On Fri, Feb 28, 2014 at 05:59:13AM -0800, Nathan Froyd wrote: > - Original Message - > > http://en.wikipedia.org/wiki/Plain_Old_Data_Structures > > > > > > confirms that POD can't have a vptr :-) > > > class Interface { > public: > constexpr Interface() {} > virtual int f() = 0; > }

Re: We live in a memory-constrained world

2014-02-28 Thread Nathan Froyd
- Original Message - > http://en.wikipedia.org/wiki/Plain_Old_Data_Structures > > > confirms that POD can't have a vptr :-) Just because it's not POD in the language doesn't mean that it necessarily requires a static initializer in all implementations: class Interface { public: const

Re: We live in a memory-constrained world

2014-02-28 Thread Benoit Jacob
http://en.wikipedia.org/wiki/Plain_Old_Data_Structures confirms that POD can't have a vptr :-) Benoit 2014-02-28 7:39 GMT-05:00 Henri Sivonen : > On Fri, Feb 28, 2014 at 1:04 PM, Neil wrote: > > At least under MSVC, they have vtables, so they need to be constructed, > so > > they're not stat

Re: We live in a memory-constrained world

2014-02-28 Thread Henri Sivonen
On Fri, Feb 28, 2014 at 1:04 PM, Neil wrote: > At least under MSVC, they have vtables, so they need to be constructed, so > they're not static. So structs that inherit at least one virtual method can't be plain old C data? That surprises me. And we still don't want to give the dynamic linker init

Re: We live in a memory-constrained world

2014-02-28 Thread Neil
Henri Sivonen wrote: Any chance static atoms could reside as plain old static C data structures in the data segment of libxul.so instead of being heap-allocated? At least under MSVC, they have vtables, so they need to be constructed, so they're not static. Note that their string storage is

Re: We live in a memory-constrained world

2014-02-27 Thread Henri Sivonen
On Fri, Feb 21, 2014 at 11:38 PM, Nicholas Nethercote wrote: > We now live in a memory-constrained world. By "we", I mean anyone > working on Mozilla platform code. I realize that .so size matters much less than per-process heap-allocated stuff, but still: On the i18n front, there are binary siz

Re: We live in a memory-constrained world

2014-02-26 Thread Jonathan Griffin
Splitting the valgrind tests up and running them separately as test jobs in TBPL is definitely something the A*Team can help with. I've filed bug 977240 for this. Jonathan On 2/25/14 7:25 PM, Nicholas Nethercote wrote: On Tue, Feb 25, 2014 at 2:32 PM, Mike Hommey wrote: I never understood

Re: We live in a memory-constrained world

2014-02-26 Thread Ehsan Akhgari
On 2014-02-26, 8:44 AM, Nicholas Nethercote wrote: On Tue, Feb 25, 2014 at 8:18 PM, Mike Hommey wrote: I never understood why we need those jobs to be builds. Why not turn --enable-valgrind on m-c builds, and run valgrind as a test job? --disable-jemalloc is needed as well. That shouldn't

Re: We live in a memory-constrained world

2014-02-26 Thread Nicholas Nethercote
On Tue, Feb 25, 2014 at 8:18 PM, Mike Hommey wrote: >> > >> > I never understood why we need those jobs to be builds. Why not turn >> > --enable-valgrind on m-c builds, and run valgrind as a test job? >> >> --disable-jemalloc is needed as well. > > That shouldn't be needed anymore with --soname-sy

Re: We live in a memory-constrained world

2014-02-25 Thread Mike Hommey
On Tue, Feb 25, 2014 at 07:25:56PM -0800, Nicholas Nethercote wrote: > On Tue, Feb 25, 2014 at 2:32 PM, Mike Hommey wrote: > > > > I never understood why we need those jobs to be builds. Why not turn > > --enable-valgrind on m-c builds, and run valgrind as a test job? > > --disable-jemalloc is ne

Re: We live in a memory-constrained world

2014-02-25 Thread Nicholas Nethercote
On Tue, Feb 25, 2014 at 2:32 PM, Mike Hommey wrote: > > I never understood why we need those jobs to be builds. Why not turn > --enable-valgrind on m-c builds, and run valgrind as a test job? --disable-jemalloc is needed as well. As for the structure... I just took what already existed and got i

Re: We live in a memory-constrained world

2014-02-25 Thread Nicholas Nethercote
On Tue, Feb 25, 2014 at 1:10 PM, Ehsan Akhgari wrote: > > How much slower is a valgrind run of let's say mochitest-1 on a test slave? > I don't think that we can run many interesting tests on valgrind on the > build machine (and the PGO test set is *really* old and doesn't cover a lot > of the pla

Re: We live in a memory-constrained world

2014-02-25 Thread Mike Hommey
On Tue, Feb 25, 2014 at 12:53:12PM -0800, Nicholas Nethercote wrote: > On Tue, Feb 25, 2014 at 12:48 PM, Ehsan Akhgari > wrote: > >> > >> The Valgrind test job does leak checking, and it's recently caught > >> some leaks and caused the offending patches to be backed out. However, > >> the test co

Re: We live in a memory-constrained world

2014-02-25 Thread Ehsan Akhgari
On 2014-02-25, 3:53 PM, Nicholas Nethercote wrote: On Tue, Feb 25, 2014 at 12:48 PM, Ehsan Akhgari wrote: The Valgrind test job does leak checking, and it's recently caught some leaks and caused the offending patches to be backed out. However, the test coverage is pretty meagre, and it's deskt

Re: We live in a memory-constrained world

2014-02-25 Thread Nicholas Nethercote
On Tue, Feb 25, 2014 at 12:48 PM, Ehsan Akhgari wrote: >> >> The Valgrind test job does leak checking, and it's recently caught >> some leaks and caused the offending patches to be backed out. However, >> the test coverage is pretty meagre, and it's desktop only so can't >> detect leaks in IPC cod

Re: We live in a memory-constrained world

2014-02-25 Thread Ehsan Akhgari
On 2/24/2014, 10:08 PM, Nicholas Nethercote wrote: On Mon, Feb 24, 2014 at 6:50 PM, Ehsan Akhgari wrote: Nick, do we run any kind of automated leak checking through valgrind? I found bug 976363 today, it's something that any automated leak detection tool should be able to catch. The Valgrind

Re: We live in a memory-constrained world

2014-02-25 Thread Kyle Machulis
--- > From: "Nicholas Nethercote" > To: "Ehsan Akhgari" > Cc: "dev-platform" > Sent: Monday, February 24, 2014 7:08:31 PM > Subject: Re: We live in a memory-constrained world > > On Mon, Feb 24, 2014 at 6:50 PM, Ehsan Akhgari > wrote: > >

Re: We live in a memory-constrained world

2014-02-24 Thread Andrew McCreight
- Original Message - > The Valgrind test job does leak checking, and it's recently caught > some leaks and caused the offending patches to be backed out. However, > the test coverage is pretty meagre, and it's desktop only so can't > detect leaks in IPC code. > > I believe the ASan builds

Re: We live in a memory-constrained world

2014-02-24 Thread Nicholas Nethercote
On Mon, Feb 24, 2014 at 6:50 PM, Ehsan Akhgari wrote: > Nick, do we run any kind of automated leak checking through valgrind? I > found bug 976363 today, it's something that any automated leak detection > tool should be able to catch. The Valgrind test job does leak checking, and it's recently c

Re: We live in a memory-constrained world

2014-02-24 Thread Ehsan Akhgari
Nick, do we run any kind of automated leak checking through valgrind? I found bug 976363 today, it's something that any automated leak detection tool should be able to catch. Thanks! Ehsan On 2/21/2014, 4:38 PM, Nicholas Nethercote wrote: Greetings, We now live in a memory-constrained world

Re: We live in a memory-constrained world

2014-02-24 Thread John Schoenick
On 02/21/2014 02:57 PM, Nicholas Nethercote wrote: On Sat, Feb 22, 2014 at 9:40 AM, Brian Smith wrote: How worthwhile is it to cut 100KiB from the parent process? We don't have a fixed memory budget per se. If it's easy and doesn't have bad side-effects, do it. Otherwise... use your judgment.

Re: We live in a memory-constrained world

2014-02-24 Thread Ben Kelly
On 2/21/2014 5:40 PM, Brian Smith wrote: On Fri, Feb 21, 2014 at 1:38 PM, Nicholas Nethercote wrote: Optimizations that wouldn't have been worthwhile in the desktop-only days are now worthwhile. For example, an optimization that saves 100 KiB of memory per process is pretty worthwhile for Firef

Re: We live in a memory-constrained world

2014-02-24 Thread Chris Peterson
On 2/22/14, 1:39 PM, Nicholas Nethercote wrote: On Sat, Feb 22, 2014 at 11:22 PM, Andreas Gal wrote: So, I'm wondering how much effort we should put in reducing the number of ChromeWorkers. We should continue to use JS in Chrome where it makes sense. Its often easier and faster to write so

Re: We live in a memory-constrained world

2014-02-23 Thread Nicholas Nethercote
On Fri, Feb 21, 2014 at 9:56 PM, Nicholas Nethercote wrote: >> >>> If your data is read-only after the point at which Nuwa forks the >>> process (which is once things are mostly loaded but before we load the >>> app) then generally yes. It's copy-on-write at page granularity of >>> course, so if

Re: We live in a memory-constrained world

2014-02-22 Thread Nicholas Nethercote
On Sat, Feb 22, 2014 at 11:22 PM, Andreas Gal wrote: > >> So, I'm wondering how much effort we should put in reducing the number >> of ChromeWorkers. > > We should continue to use JS in Chrome where it makes sense. Its often easier > and faster to write some functionality in JS (and sometimes als

Re: We live in a memory-constrained world

2014-02-22 Thread Andreas Gal
On Feb 22, 2014, at 1:12 PM, David Rajchenbach-Teller wrote: > So, I'm wondering how much effort we should put in reducing the number > of ChromeWorkers. On Desktop, we have a PageThumbsWorker and a > SessionWorker, which are both useful but not strictly necessary. I seem > to remember that the

Re: We live in a memory-constrained world

2014-02-22 Thread David Rajchenbach-Teller
So, I'm wondering how much effort we should put in reducing the number of ChromeWorkers. On Desktop, we have a PageThumbsWorker and a SessionWorker, which are both useful but not strictly necessary. I seem to remember that they each take 1Mb+ of RAM. On all platforms, we have the OS.File ChromeWor

Re: We live in a memory-constrained world

2014-02-21 Thread Nicholas Nethercote
On Sat, Feb 22, 2014 at 1:00 PM, Till Schneidereit wrote: > >> If your data is read-only after the point at which Nuwa forks the >> process (which is once things are mostly loaded but before we load the >> app) then generally yes. It's copy-on-write at page granularity of >> course, so if you hav

Re: We live in a memory-constrained world

2014-02-21 Thread Till Schneidereit
On Sat, Feb 22, 2014 at 2:31 PM, Kyle Huey wrote: > On Fri, Feb 21, 2014 at 5:28 PM, Till Schneidereit > wrote: > > Does that mean that bug 964057[1] will automatically cause all > self-hosting > > state to be shared amongst content processes, too? > > > > [1]: https://bugzilla.mozilla.org/show_

Re: We live in a memory-constrained world

2014-02-21 Thread Kyle Huey
On Fri, Feb 21, 2014 at 5:28 PM, Till Schneidereit wrote: > Does that mean that bug 964057[1] will automatically cause all self-hosting > state to be shared amongst content processes, too? > > [1]: https://bugzilla.mozilla.org/show_bug.cgi?id=964057 > > > On Sat, Feb 22, 2014 at 2:12 PM, Kyle Huey

Re: We live in a memory-constrained world

2014-02-21 Thread Till Schneidereit
Does that mean that bug 964057[1] will automatically cause all self-hosting state to be shared amongst content processes, too? [1]: https://bugzilla.mozilla.org/show_bug.cgi?id=964057 On Sat, Feb 22, 2014 at 2:12 PM, Kyle Huey wrote: > On Fri, Feb 21, 2014 at 5:07 PM, Jason Duell > wrote: > >

Re: We live in a memory-constrained world

2014-02-21 Thread Kyle Huey
On Fri, Feb 21, 2014 at 5:07 PM, Jason Duell wrote: > On 02/21/2014 01:38 PM, Nicholas Nethercote wrote: >> >> Greetings, >> >> We now live in a memory-constrained world. By "we", I mean anyone >> working on Mozilla platform code. When desktop Firefox was our only >> product, this wasn't especiall

Re: We live in a memory-constrained world

2014-02-21 Thread Jason Duell
On 02/21/2014 01:38 PM, Nicholas Nethercote wrote: Greetings, We now live in a memory-constrained world. By "we", I mean anyone working on Mozilla platform code. When desktop Firefox was our only product, this wasn't especially true -- bad leaks and the like were a problem, sure, but ordinary us

Re: We live in a memory-constrained world

2014-02-21 Thread Nicholas Nethercote
On Sat, Feb 22, 2014 at 9:40 AM, Brian Smith wrote: > > How worthwhile is it to cut 100KiB from the parent process? We don't have a fixed memory budget per se. If it's easy and doesn't have bad side-effects, do it. Otherwise... use your judgment. Having a Firefox OS device to test on helps with e

Re: We live in a memory-constrained world

2014-02-21 Thread Nicholas Nethercote
On Sat, Feb 22, 2014 at 9:30 AM, David Rajchenbach-Teller wrote: > Does this memory-constrained policy apply only to release builds or also > to debug builds? Debug builds are definitely less important. But it's also nice to not have big differences between debug and opt builds. E.g. in https://b

Re: We live in a memory-constrained world

2014-02-21 Thread Andreas Gal
Caches are fine in the child process as long they are reasonably sized, and they purge themselves in case of a low-memory notification. On FFOS the parent process never dies, so leaks or long-lived memory allocation there tends to hurt. Responding to low-memory notifications is really important

Re: We live in a memory-constrained world

2014-02-21 Thread Brian Smith
On Fri, Feb 21, 2014 at 1:38 PM, Nicholas Nethercote wrote: > Optimizations that wouldn't have been worthwhile in the desktop-only > days are now worthwhile. For example, an optimization that saves 100 > KiB of memory per process is pretty worthwhile for Firefox OS. Do you mean 100KiB per child p

Re: We live in a memory-constrained world

2014-02-21 Thread David Rajchenbach-Teller
Does this memory-constrained policy apply only to release builds or also to debug builds? Cheers, David On 2/21/14 10:38 PM, Nicholas Nethercote wrote: > Greetings, > > We now live in a memory-constrained world. By "we", I mean anyone > working on Mozilla platform code. When desktop Firefox was