Heka can get wedged when there's a deadlock in the pipeline, usually related to pack exhaustion. For example, a filter might block because it's waiting for an empty pack with which to inject a new message, but there are no packs available because they're all tied up in the input channels for the filter. Tweaking the channel and pool sizes is usually helpful here.
There are no verbosity levels to Heka's logging, sorry. When Heka is wedged, you can get a dump to stdout of the current internal state of the pipeline by sending SIGUSR1 to the Heka process. You can get the dump, followed by serializing all of the sandboxes and exiting the process by sending SIGUSR2. You can monitor for wedgedness by setting up a filter to emit a message every N seconds, and then setting up an output to catch those messages. You could write them to a file and monitor that the file is growing, or send them to a downstream listener that will notice if the heartbeats stop coming. -r On 03/30/2016 11:13 AM, Ramin Ali Dousti wrote:
Hi Timur, 1) wedged: In my case, my hekad gets wedged for no reason (or I think i cannot pinpoint as to why) 2) logging Yes, I could do a LogOutput (which I'm doing when I'm troubleshooting) but I was looking for some kind of application logging with different severity so that i could up or down the severity and get to see less or more log entries about what was going on. I'm not talking about the actual messages that are passing through the system. For example when a TcpInput drops a connection you will see a line indicating that, I'd like to see a similar thing about "important events" within the system. For example "I've just become wedged because ...". Thanks for the answers though. Best, Ramin On Wed, Mar 30, 2016 at 7:34 AM, Timur Batyrshin <[email protected] <mailto:[email protected]>> wrote: Hi Ramin, When heka is running it emits HeapAlloc, HeapSys and few other metrics which you can graph or alert based on them. I've seen heka freezing only when I send TERM signal to it and some Lua plugin fails to stop correctly or something like that. May be someone else will give you insights on other conditions. For checking how heka is processing I know 3 ways: * dashboard (http://hekad.readthedocs.org/en/v0.10.0/config/outputs/dashboard.html) -- it doesn't display individual messages but displays number of messages processed, queues, lags, etc * if you send messages over TcpOutput you can enable buffering and use "heka-cat" command to browse/tail protobuf logs in /var/cache/heka * the most usable for me way is creating additional output like the following: [debug_encoder] type="RstEncoder" [LogOutput] encoder="debug_encoder" message_matcher="TRUE" This makes heka dump all messages to stdout. If you store that in logs hope you rotate them based on size reached. Best regards, Timur On Wed, Mar 30, 2016 at 2:22 AM, Ramin Ali Dousti <[email protected] <mailto:[email protected]>> wrote: Hi, I have a nagging question. I read in the documentations and I also experienced it myself that hekad might become "wedged" and there would be no activity within the system while the process seems up from the outside. My question is what are the conditions that this might happen? The reason I ask is that my hekad instances might be running for weeks with no problem but they could end up "wedged" for no obvious reason. Knowing what gets them into this mode, might help me prevent that situation. Also, hekad seems a very quiet process log-wise. How can I have it log "important" information about its doing? I really appreciate any insight. -- Ramin _______________________________________________ Heka mailing list [email protected] <mailto:[email protected]> https://mail.mozilla.org/listinfo/heka -- Ramin _______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
