Number of messages in the pool is controlled by the poolsize setting.
-r On 01/19/2016 12:15 PM, Timur Batyrshin wrote:
That is configured by which param? plugin_chansize or which? Hopefully I will not hit the case as there will be no more than 10 messages emitted by a single decoder at the same time. But if there are several inputs with the same decoder running it becomes not so unlikely.. Timur On 19 Jan 2016 at 23:07:20, Rob Miller ([email protected] <mailto:[email protected]>) wrote: > Yes, decoders do, but keep in mind that none of the messages will be > injected until they're *all* injected. This isn't usually a problem if > you're only emitting a few output messages per input message, but if > you're emitting a lot of messages you can easily exhaust the message > pool. > > Filters don't have this problem. > > -r > > > > On 01/19/2016 12:04 PM, Timur Batyrshin wrote: > > Ok, thanks for clarification! > > > > I think I’ll have to split the messages in decode then. > > Do decoders support injecting several messages? > > > > Timur > > > > On 19 Jan 2016 at 23:00:53, Rob Miller ([email protected] > > <mailto:[email protected]>) wrote: > > > > > No, I mean that SandboxEncoders don't at this point support multiple > > > inject calls for each input message. You can only call inject once. If > > > you want smaller input, you'll have to do the work in a filter. > > > > > > -r > > > > > > > > > On 01/19/2016 11:53 AM, Timur Batyrshin wrote: > > > > My point was not accumulating strings in output buffer but injecting > > > > several messages instead to make them smaller. > > > > Or do you mean that inject_message() run several times in a single run > > > > should work? > > > > I’ll try that — haven’t done that yet. > > > > > > > > > > > > Timur > > > > > > > > On 19 Jan 2016 at 22:50:28, Rob Miller ([email protected] > > > > <mailto:[email protected]>) wrote: > > > > > > > > > Yes, that's right. A SandboxEncoder can only make a single inject > > > > > call. If you call `inject_message`, then you will be emitting a > > > > > protobuf encoded Heka message. If you call `inject_payload`, then you > > > > > will be emitting a UTF8 encoded set of bytes. If you want to > > > > > accumulate multiple string values in the output buffer, you can use > > > > > `add_to_payload` before a final `inject_payload` call. > > > > > > > > > > -r > > > > > > > > > > > > > > > On 01/16/2016 12:08 PM, Timur Batyrshin wrote: > > > > > > An encoder I was using for reference: > > > > > > > > > > > > function process_message() > > > > > > inject_payload("msg1", "", "msg1") > > > > > > inject_payload("msg2", "", "msg2") > > > > > > return 0 > > > > > > end > > > > > > > > > > > > This produces only “msg2”. > > > > > > > > > > > > > > > > > > Timur > > > > > > > > > > > > > > > > > > On 16 Jan 2016 at 22:09:35, Timur Batyrshin ([email protected] > > > > > > <mailto:[email protected]>) wrote: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > I’ve been trying to do some buffering and splitting of messages in an > > > > > > > encoder and I’ve run in a following issue. > > > > > > > > > > > > > > Looks like calling inject_payload() several times in a single run of > > > > > > > encoder’s process_message() doesn’t work and only the last message is > > > > > > > injected. > > > > > > > Is it so? > > > > > > > > > > > > > > If yes, any ideas how can I solve the following task? > > > > > > > * I have incoming messages consisting of up to several hundreds of > > > > > > > custom fields > > > > > > > * I’m encoding the fields into JSON array like [{“metric": > > > > > > > “foobar.count", "value": 1000}, {“metric": “bazqux.rate”, “value”: > > > > > > > 10}] and sending them out via HTTP to a different service > > > > > > > * I’d like to keep outgoing HTTP messages relatively small (no more > > > > > > > than a few dozens of fields) and for that reason I’m going to split > > > > > > > messages with too many fields into several smaller ones > > > > > > > > > > > > > > For that I was going to do the splitting inside encoder attached to > > > > > > > HTTPOutput and calling inject_payload() several times but it seems > > > > > > > that only the last message is sent out. > > > > > > > I’ve tried setting max_process_inject = 1000 in [hekad] config section > > > > > > > but this didn’t help either. > > > > > > > > > > > > > > I could try using a filter to split messages and reinject them back to > > > > > > > Heka (haven’t tested yet if it is going to work in this scenario) but > > > > > > > this looks like a bit overweight to me. > > > > > > > > > > > > > > Any other ideas how to handle the task? > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > Timur > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > Heka mailing list > > > > > > [email protected] > > > > > > https://mail.mozilla.org/listinfo/heka > > > > > > > > > > > > > > >
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
