On Wed, Apr 6, 2016 at 8:48 PM, Rob Miller <[email protected]> wrote:
> I'm not sure exactly what is going on here, but I can provide you w/ some
> info that might help you debug further:
>
> * Any packs that are in a decoder's input channel can only have come from
> the input pool, i.e. inputRecycleChan. I see ~40 packs tied up there.
>
> * Any packs that are in a match channel or input channel for a filter or
> output plugin could have come from either the input pool or the inject pool,
> although they can only have come from the inject pool if they were injected
> by some other filter plugin first. If a plugin's matcher only matches
> messages that you know came directly from an input plugin, then they have to
> have come from the input pool. Most of these are empty, but there are 60
> packs sitting in the queues for http_metrics_filter.
>
> * Heka doesn't freeze the world while generating the report data, so it's
> possible that the data you're seeing doesn't represent a single point in
> time, which can cause the math to be weird. However, if Heka is truly wedged
> and no traffic is flowing, this is moot, since the state probably isn't
> changing at all during the entire reporting process.
>
> * It's possible for a filter or output to hold an arbitrary number of packs
> in its internal memory. These won't show up in the reporting at all, but
> neither will they be recycled. Without seeing the code that's running in
> your filters I can't say whether or not that's happening.
>
> I have to run into a meeting right now, hopefully this will help at least a
> bit, or at least help you come up with more specific questions.
Thanks Rob.
I'll start with one very specific question.
In the diagnostic I have this for the idle inject packs:
2016/03/31 11:56:20 Diagnostics: 30 packs have been idle more than 120 seconds.
2016/03/31 11:56:20 Diagnostics: (inject) Plugin names and quantities
found on idle packs:
2016/03/31 11:56:20 Diagnostics: influxdb_accumulator_filter: 30
(and no other plugins with idle inject packs!)
In the same diagnostic I have this:
influxdb_accumulator_filter:
InChanCapacity: 30
InChanLength: 0
MatchChanCapacity: 30
MatchChanLength: 0
My question: where can the 30 idle inject packs attributed to
influxdb_accumulator_filter be?
InChanLength is 0, so they're not in the influxdb_accumulator_filter
plugin's input channel. influxdb_accumulator_filter is a sandbox, and
looking at sandbox_filter.go code [*], I do not see how the filter
could be holding these packs.
At some point I thought that these packs could sit in another
filter/output's input channel, or be hold by another filter/channel.
But in that case the diagnostic should attribute the idle inject packs
to that other plugin as well. This is not case, the 30 idle packs are
only attributed to influxdb_accumulator_filter.
[*]
<https://github.com/mozilla-services/heka/blob/dev/sandbox/plugins/sandbox_filter.go#L297>
I hope that my question is clear.
Thanks again!
PS: I hope to come up with more specific information about this problem.
_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka
