Hi Mathieu, I've meant: https://github.com/mozilla-services/heka/issues/1484
Unfortunately i'm a little busy and i can't verify source of this bug. I'll try to debug it later, but probably @trink and @rafrombrc will be faster. On Thu, May 7, 2015 at 3:03 PM, Mathieu Chouquet-Stringer <[email protected] > wrote: > Hello, > > On Wed, May 06, 2015 at 04:26:36PM +0200, Jarek Śmiejczak wrote: > > > Now I'll have to look at why my ElasticSearch output is buffering > > instead of sending stuff... > > > > Could you post your config? There was an issue with blocking queue > which > > was resolved by Rob in 0.9.2. > > By blocking, do you mean another issue than this one "Changed default > plugin_chansize setting from 50 to 30 (#1242)."? > https://github.com/mozilla-services/heka/issues/1242 > > Please find below my config: I've tried tweaking so many things without > much results I went back to the default settings. > > Using the web dashboard I see around 4k messages filtered by second (as > in 20k every 5 seconds) and that's without decoders (just using > ProtobufDecoder). With decoders it's much slower (around 2k or 3k/s) but > it's expected as I should really rewrite my dual multidecoders to LPEG: > - a protobuf followed by the multidecoder below using the "all" cascade > strategy > - a multidecoder with all my regexps in a "first-wins" mode > > If I look at CPU/disk usage, there's nothing out of the ordinary: I'm > using 30% of a core and that's about it. > > Note that there seems to be an issue with the dashboard where the output > switches from say (this is valid): > > Router Router inputRecycleChan > injectRecycleChan > 503,271 30/30 0/100 100/100 > Messages Processed Messages in Channel Messages Available Messages > Available > > to (this is invalid and the 626 million thing is most likely the number > of records in my Kafka): > > Router Router inputRecycleChan > injectRecycleChan > 626,788,390 0/30 300/300 300/300 > Messages Processed Messages in Channel Messages Available Messages > Available > > When it happens, the decoder section becomes empty. Note this behavior > is not depend on having decoders enabled. > > [Dashboard] > type = "DashboardOutput" > address = ":4352" > ticker_interval = 5 > > [hekad] > maxprocs = 2 > base_dir = "/home/heka/heka/heka1" > share_dir = "/home/heka/heka/share/heka" > > [KafkaInput1] > type = "KafkaInput" > topic = "logs" > addrs = ["10.100.100.37:9092"] > splitter = "KafkaSplitter" > decoder = "ProtobufDecoder"" > partition = 0 > offset_method = "Oldest" > > [KafkaSplitter] > type = "NullSplitter" > use_message_bytes = true > > [ESJsonEncoder] > index = "heka-%{2006.01.02}" > es_index_from_timestamp = true > > [RstEncoder] > > [ElasticsearchOutput] > type = "ElasticSearchOutput" > message_matcher = "TRUE" > server = "http://10.100.100.30:9200"; > encoder = "ESJsonEncoder" > use_buffering = false > queue_full_action = "block" > > -- > Mathieu Chouquet-Stringer [email protected] > The sun itself sees not till heaven clears. > -- William Shakespeare -- > -- jarek@reijutsu:~$ fortune You have Egyptian flu: you're going to be a mummy. jarek@reijutsu:~$ fortune You now have Asian Flu.
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
