On Wed, Apr 6, 2016 at 8:48 PM, Rob Miller <[email protected]> wrote: > I'm not sure exactly what is going on here, but I can provide you w/ some > info that might help you debug further: > > * Any packs that are in a decoder's input channel can only have come from > the input pool, i.e. inputRecycleChan. I see ~40 packs tied up there. > > * Any packs that are in a match channel or input channel for a filter or > output plugin could have come from either the input pool or the inject pool, > although they can only have come from the inject pool if they were injected > by some other filter plugin first. If a plugin's matcher only matches > messages that you know came directly from an input plugin, then they have to > have come from the input pool. Most of these are empty, but there are 60 > packs sitting in the queues for http_metrics_filter. > > * Heka doesn't freeze the world while generating the report data, so it's > possible that the data you're seeing doesn't represent a single point in > time, which can cause the math to be weird. However, if Heka is truly wedged > and no traffic is flowing, this is moot, since the state probably isn't > changing at all during the entire reporting process. > > * It's possible for a filter or output to hold an arbitrary number of packs > in its internal memory. These won't show up in the reporting at all, but > neither will they be recycled. Without seeing the code that's running in > your filters I can't say whether or not that's happening. > > I have to run into a meeting right now, hopefully this will help at least a > bit, or at least help you come up with more specific questions.
Ok, I have done new tests with fewer plugins. I can reproduce the "idle packs" issue but the diagnostic makes more sense, matching the explanation you provided in the old thread (with Bruno Binet). See <https://gist.github.com/elemoine/1f393f36630df01fac82778eb6f900e2> for the new diagnostic trace. This time, the downstream filter (influxdb_accumulator_filter) is the one that is deadlocked. The recycle inject channel is empty, and this filter has 60 packs in his queues (30 in the match queue and 30 in the input queue). There are also 30 packs in the router's input queue, which may be inject packs. So that makes for a maximum of 90 packs. I'm still missing 10 packs! Any idea where they could be? Note that the 6 packs in the http_metrics_filter plugin are input packs coming from keystone_7_0_logstreamer input, they cannot be inject packs because of the matcher used for that filter. So I get a diagnostic that makes more sense, but I'm still missing 10 inject packs! Sorry for insisting. Understanting this would help me troubleshoot problems in the future. PS: I am indeed not able to reproduce the problem with poolsize 200, but I'd like to make sure the problem will not come back in the future. _______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
