Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-06-02 Thread Milan Sreckovic
For example, searching for “graphics critical error” containing 0x8007000e (Windows error code for “out of memory”) leads you to a lot of crashes/bugs that are not tagged as OOM (which is OK), but clearly have some aspect of “running out of stuff” in them. In general, it can come in handy when

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-06-02 Thread Milan Sreckovic
If you want a treasure trove :), when it comes to graphics crashes, the “graphics critical error” field is it. It’s a recent history of barely avoidable crashes and error states just before the crash actually happens. Very often the real cause of the crash happens prior to the line of code ca

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-06-02 Thread Gabriele Svelto
On 01/06/2016 15:20, Jonathan Kew wrote: Does this suggest that we're not sufficiently proactive about firing memory-pressure notifications, so that we'll free up memory from various caches, etc? It looks like we regard 128MB of available VM as "low" (see [1]) on Windows 32-bit, but apparently we

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-06-01 Thread Nicholas Nethercote
I apologize for the sarcasm. I was frustrated with this comment: > I looked at the query from the first post, but it's just noise to me. If it > included the file that it > crashed from, it would suddenly be very useful, since it'd then be trivial to > see if there's > something relevant to me.

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-06-01 Thread Ted Mielczarek
On Tue, May 31, 2016, at 09:26 PM, Jeff Gilbert wrote: > On Tue, May 31, 2016 at 4:39 PM, Nicholas Nethercote > wrote: > > On Wed, Jun 1, 2016 at 1:05 AM, Benjamin Smedberg > > wrote: > >> You shouldn't need to annotate the file/line separately, because that is > >> (or at least should be!) the

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-06-01 Thread Jeff Gilbert
Awesome, this sounds like what I was after. (though actual components isn't really necessary. If that part is a pain point, I would prefer to have a tool without it, than to have no tool) On Wed, Jun 1, 2016 at 12:23 PM, Ted Mielczarek wrote: > On Tue, May 31, 2016, at 09:26 PM, Jeff Gilbert wrot

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-06-01 Thread Jeff Gilbert
It would be useful to have a dashboard that collates this information better. PS: Sarcasm is unhelpful. On Tue, May 31, 2016 at 7:14 PM, Nicholas Nethercote wrote: > On Wed, Jun 1, 2016 at 11:26 AM, Jeff Gilbert wrote: >> >> Perhaps this isn't meant for me then? I looked at the query from the >

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-06-01 Thread Milan Sreckovic
Not sure what you mean - the stack information is there, right? This is my workflow: Search for all the MOZ_CRASHes (e.g., moz crash reason “exists” (sometimes I just search for the graphics reasons, but we don’t have the GFX prefix on all the crashes, so that doesn’t quite work all the time)

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-06-01 Thread Milan Sreckovic
Cairo graphics reports “out of memory” error condition when the author didn’t have time to figure out what went wrong. We caught a few problems that were being reported as out of memory (we would pick up the Cairo library error as out of memory, and dutifully propagate it up the chain), when th

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-06-01 Thread Andrew McCreight
On Tue, May 31, 2016 at 7:14 PM, Nicholas Nethercote wrote: > If you use your favourite source code search tool to look for > "Shutdown too long", you'll find that this crash is occurring in > toolkit/components/terminator/nsTerminator.cpp. For example, here's a > DXR link: > Sure, you can indiv

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-06-01 Thread Andrew McCreight
On Mon, May 30, 2016 at 11:22 PM, Nicholas Nethercote < n.netherc...@gmail.com> wrote: > 6 MOZ_RELEASE_ASSERT(pi->mInternalRefs < pi->mRefCount) (Cycle > collector found more references to an object than its refcount)509 > 0.04 % > That's bug 1266882. 15 MOZ_RELEASE_ASSERT(aRefCount !=

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-06-01 Thread Boris Zbarsky
On 6/1/16 2:32 AM, Nicholas Nethercote wrote: 4 MOZ_CRASH(Unexpected error during FakeBlack creation.) 16790.13 % This is https://bugzilla.mozilla.org/show_bug.cgi?id=1247977 (credit to Milan for noticing that). 49 MOZ_CRASH(Accessing the Subject Principal without an AutoJSAPI on the

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-06-01 Thread Jonathan Kew
On 1/6/16 00:51, Nicholas Nethercote wrote: On Wed, Jun 1, 2016 at 2:37 AM, Jonathan Kew wrote: I took a quick look at a random one of these OOMs[1], and what strikes me about it is that according to the crash report: Total Virtual Memory2147352576 Available Virtual Mem

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-06-01 Thread Josh Matthews
On 2016-06-01 2:32 AM, Nicholas Nethercote wrote: 37 MOZ_RELEASE_ASSERT(aInAndOutListener) (can not perform CORS checks without a listener) 23 0.00 % bug 1276943 ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-06-01 Thread Nicholas Nethercote
Here's an update. This one is bug 1235183: > 7 MOZ_RELEASE_ASSERT(!mDoingStableStates) 466 0.04 % This one is covered by bug 616421 (the signature includes a combination of MOZ_CRASHes and other kinds of crashes): > 8 MOZ_CRASH(Bogus tree op)459 0.04 % Below are all th

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-05-31 Thread Nicholas Nethercote
On Wed, Jun 1, 2016 at 11:26 AM, Jeff Gilbert wrote: > > Perhaps this isn't meant for me then? I looked at the query from the > first post, but it's just noise to me. If it included the file that it > crashed from, it would suddenly be very useful, since it'd then be > trivial to see if there's so

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-05-31 Thread Jeff Gilbert
On Tue, May 31, 2016 at 4:39 PM, Nicholas Nethercote wrote: > On Wed, Jun 1, 2016 at 1:05 AM, Benjamin Smedberg > wrote: >> You shouldn't need to annotate the file/line separately, because that is >> (or at least should be!) the top of the stack. > > Yes. Don't get hung up on the lack of annotat

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-05-31 Thread Nicholas Nethercote
On Wed, Jun 1, 2016 at 9:39 AM, Nicholas Nethercote wrote: > > Yes. Don't get hung up on the lack of annotations. It isn't much of a > problem; you can click through easily enough. I have filed bug 1277104 > to fix the handful of instances that are showing up in practice, but > it'll only be a min

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-05-31 Thread Mike Hommey
On Tue, May 31, 2016 at 05:01:34PM -0700, L. David Baron wrote: > On Tuesday 2016-05-31 12:50 -0400, Benjamin Smedberg wrote: > > It's likely that this particular report is running out of VM, yes. jemalloc > > allocates new memory chunks in large blocks (1MB?), and with only 122MB of > > VM it's li

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-05-31 Thread L. David Baron
On Tuesday 2016-05-31 12:50 -0400, Benjamin Smedberg wrote: > It's likely that this particular report is running out of VM, yes. jemalloc > allocates new memory chunks in large blocks (1MB?), and with only 122MB of > VM it's likely that a lot of that is inaccessible, either because of > fragmentati

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-05-31 Thread Nicholas Nethercote
On Wed, Jun 1, 2016 at 2:37 AM, Jonathan Kew wrote: > > I took a quick look at a random one of these OOMs[1], and what strikes me > about it is that according to the crash report: > > Total Virtual Memory2147352576 > > Available Virtual Memory122331136 > > Syste

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-05-31 Thread Nicholas Nethercote
On Wed, Jun 1, 2016 at 1:05 AM, Benjamin Smedberg wrote: > You shouldn't need to annotate the file/line separately, because that is > (or at least should be!) the top of the stack. Yes. Don't get hung up on the lack of annotations. It isn't much of a problem; you can click through easily enough.

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-05-31 Thread Eric Rescorla
Also, perhaps function name (__func__) or one of the pretty versions. -Ekr On Tue, May 31, 2016 at 1:20 PM, Jeff Gilbert wrote: > On Tue, May 31, 2016 at 6:18 AM, Gabriele Svelto > wrote: > > On 31/05/2016 13:26, Gijs Kruitbosch wrote: > >> We could do a find/replace of no-arg calls to a new

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-05-31 Thread Jeff Gilbert
On Tue, May 31, 2016 at 6:18 AM, Gabriele Svelto wrote: > On 31/05/2016 13:26, Gijs Kruitbosch wrote: >> We could do a find/replace of no-arg calls to a new macro that uses >> MOZ_CRASH with a boilerplate message, and make the argument non-optional >> for new uses of MOZ_CRASH? That would avoid th

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-05-31 Thread Ralph Giles
A few of these are in MediaFormatReader. On Mon, May 30, 2016 at 11:22 PM, Nicholas Nethercote wrote: > 11 MOZ_RELEASE_ASSERT(!mSkipRequest.Exists()) (called mid-skipping) > 222 0.02 % Fixed in bug 1068151. > 16 MOZ_RELEASE_ASSERT(!mAudio.HasPromise()) (No duplicate sample > requests)

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-05-31 Thread Milan Sreckovic
Agreed that all startup crashes are important. The reason I focused on the MOZ_CRASHes is that this is where somebody explicitly said “this is as bad as it gets, we must crash now”, and I’d like to look at those once in a while, and see if that’s really the case. Because I certainly ran into c

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-05-31 Thread Benjamin Smedberg
It's likely that this particular report is running out of VM, yes. jemalloc allocates new memory chunks in large blocks (1MB?), and with only 122MB of VM it's likely that a lot of that is inaccessible, either because of fragmentation or because sites are allocating VM blocks of less than 64k, which

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-05-31 Thread Jonathan Kew
On May 31, 2016, at 2:22 , Nicholas Nethercote wrote: #2 is unannotated MOZ_CRASH() calls, i.e. there is no string argument given. These are mostly OOMs, though there are a few others in there. These ones should be annotated so they show up separately. I took a quick look at a random one of

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-05-31 Thread Benjamin Smedberg
You're assuming that this happens every time, instead of randomly. If you add the time since last crash to your column list, you can see that this is true in some cases and not others. I changed your link a little: * remove "moz crash reason exists" - any startup crash is a problem * excluded con

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-05-31 Thread Milan Sreckovic
By the way, this is the kiss of death query. MOZ_CRASH, start up, in safe mode. We’re basically forcing these people away. There is nothing they can do even if they really want to run Firefox (assuming this is a persistent start up crash, of course.) The numbers aren’t high, and majority of

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-05-31 Thread Markus Stange
On 2016-05-31 2:22 AM, Nicholas Nethercote wrote: 9 MOZ_RELEASE_ASSERT(sAliveDisplayItemDatas && sAliveDisplayItemDatas->Contains(aData)) 263 0.02 % This one is https://bugzilla.mozilla.org/show_bug.cgi?id=1141089 . -Markus ___ dev-platform ma

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-05-31 Thread Ben Kelly
On Tue, May 31, 2016 at 10:28 AM, Milan Sreckovic wrote: > On a side note, the one that stood out for me was a “TODO” one. A crash > seems to be a wrong way to tag TODOs :) > FWIW, this appears to have been fixed last week: https://hg.mozilla.org/mozilla-central/rev/a61e4c04aadb __

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-05-31 Thread Lawrence Mandel
We do have contacts. The more information we pass along about this crash including how to avoid crashing if we know the better our chances of success. What would you recommend that we tell IBM Rapport? Sylvestre - Can you please pass along information about the crash? Thanks, Lawrence On Tue, M

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-05-31 Thread Benjamin Smedberg
You shouldn't need to annotate the file/line separately, because that is (or at least should be!) the top of the stack. FWIW, we are currently working on changing the signature for crashes with an AbortMessage (those using NS_RUNTIMEABORT) so that the abort message is part of the signature. After

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-05-31 Thread Milan Sreckovic
We considered that for the graphics ones, but the line number doesn’t persist between versions, so we weren’t sure how the search would find all the ones that are the same, but on different line numbers in different versions. In the end we settled on using unique strings (for the interesting on

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-05-31 Thread Milan Sreckovic
I search for and track the ones that start with “GFX” (which is why we added those prefixes, to make it easier to find them all.) Here’s a comment on the top few of those: #3 is bug 1254400, filed in March, fixed in May, uplifted to 47. #12 is a collection of different crashes, with the same m

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-05-31 Thread Josh Matthews
On 2016-05-31 9:27 AM, Gijs Kruitbosch wrote: On 31/05/2016 07:22, Nicholas Nethercote wrote: 10 MOZ_CRASH(Using observer service off the main thread!) 223 0.02 % This looked interesting to me, but it seems almost all of them are caused by IBM Rapport which hooks into the process and call

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-05-31 Thread Gijs Kruitbosch
On 31/05/2016 07:22, Nicholas Nethercote wrote: 10 MOZ_CRASH(Using observer service off the main thread!) 223 0.02 % This looked interesting to me, but it seems almost all of them are caused by IBM Rapport which hooks into the process and calls the observer service off-main-thread. If so

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-05-31 Thread Gabriele Svelto
On 31/05/2016 13:26, Gijs Kruitbosch wrote: > We could do a find/replace of no-arg calls to a new macro that uses > MOZ_CRASH with a boilerplate message, and make the argument non-optional > for new uses of MOZ_CRASH? That would avoid the problem for new > MOZ_CRASH() additions, which seems like it

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-05-31 Thread Gijs Kruitbosch
On 31/05/2016 09:40, Nicholas Nethercote wrote: On Tue, May 31, 2016 at 5:06 PM, Chris Peterson wrote: #2 is unannotated MOZ_CRASH() calls, i.e. there is no string argument given. These are mostly OOMs, though there are a few others in there. These ones should be annotated so they show up sepa

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-05-31 Thread Nicholas Nethercote
On Tue, May 31, 2016 at 5:06 PM, Chris Peterson wrote: >> >> #2 is unannotated MOZ_CRASH() calls, i.e. there is no string argument >> given. These are mostly OOMs, though there are a few others in there. >> These ones should be annotated so they show up separately. > > MOZ_CRASH()'s explanation st

Re: Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-05-31 Thread Chris Peterson
On 5/30/16 11:22 PM, Nicholas Nethercote wrote: #2 is unannotated MOZ_CRASH() calls, i.e. there is no string argument given. These are mostly OOMs, though there are a few others in there. These ones should be annotated so they show up separately. MOZ_CRASH()'s explanation string parameter is op

Common crashes due to MOZ_CRASH and MOZ_RELEASE_ASSERT

2016-05-30 Thread Nicholas Nethercote
Hi, Here is a crash-stats search that shows all the crash reports in the past 7 days that have a "MozCrashReason" field -- which means they were triggered by MOZ_CRASH or MOZ_RELEASE_ASSERT -- faceted (i.e. aggregated) by that field: https://crash-stats.mozilla.com/search/?product=Firefox&_facets