Hi Satish,
You may be able to achieve what you are asking by using the ScribbleDecoder.
http://hekad.readthedocs.io/en/latest/config/decoders/scribble.html
In combination with your SomethingLogDecoder both within a MultiDecoder.
i.e.
[SomethingLogInput]
type = "LogstreamerInput"
log_directory = "/var/log/xxx"
file_match = '(?P<something>\s+)/program\.log\.?(?P<Seq>\d*)'
differentiator = ["something", "program","Seq"]
decoder = "SomethingLogDecoder"
[SomethingLogMultiDecoder]type = "MultiDecoder"subs =
["SomethingLogDecoder", "SomethingLogScribbler"]cascade_strategy =
"all"log_sub_errors = true
[SomethingLogDecoder]
type = "PayloadRegexDecoder"
match_regex = '(?P<comp>[^(]+\w+)\@(?P<host>\D+[^)]).*node[(]+.*\.'
[SomethingLogDecoder.message_fields]
Type = "erlang"
comp = '%comp%'
host = '%host%'
seq = '%Seq%'
logger = '%Component%'
[SomethingLogScribbler]type = "ScribbleDecoder"
[SomethingLogScribbler.message_fields]
Something = "something"
On Thu, Aug 4, 2016 at 7:08 PM, Satish Anupindi Rao <
[email protected]> wrote:
>
>
> Hi,
>
>
>
> Please allow me to rephrase my question. It may not have been clear the
> first time.
>
> I have many folders as below with files as shown.
>
> /var/log/xxx/jack/program.log.1
>
> /var/log/xxx/jack/program.log.2
>
> /var/log/xxx/john/program.log.1
>
> /var/log/xxx/john/program.log.3
>
> /var/log/xxx/sam/program.log.1
>
> ...
>
> The LogStreamerInput is shown below and I have tested it – it works just
> fine.
>
>
>
> [SomethingLogInput]
>
> type = "LogstreamerInput"
>
> log_directory = "/var/log/xxx"
>
> file_match = '(?P<something>\s+)/program\.log\.?(?P<Seq>\d*)'
>
> differentiator = ["something", "program","Seq"]
>
> decoder = "SomethingLogDecoder"
>
>
>
> The SomethingLogDecoder is below and I have verified that it works for the
> fields which are in the match_regex i.e comp and host
>
>
>
> [SomethingLogDecoder]
>
> type = "PayloadRegexDecoder"
>
> match_regex = '(?P<comp>[^(]+\w+)\@(?P<host>
> \D+[^)]).*node[(]+.*\.'
>
>
>
> [SomethingLogDecoder.message_fields]
>
> Type = "erlang"
>
> comp = '%comp%'
>
> host = '%host%'
>
> seq = '%Seq%'
>
> logger = '%Component%'
>
>
>
> I would like the ElasticSearchJSONEncoder to have access to the
> “something” field which was used as a differentiator in the Input phase. I
> need it since it gives me information if it was the “jack” or “john” or
> “sam” folder the messages came from. But I have not been able to figure out
> a way to do this i.e. have the encoder include fields from the input phase.
> Would anyone be able to help me out here?
>
>
>
> Thanks!
>
> *From:* Heka [mailto:[email protected]] *On Behalf Of *Satish
> Anupindi Rao
> *Sent:* Thursday, August 04, 2016 1:28 PM
> *To:* [email protected]
> *Subject:* [heka] Heka input differentiator fields to decoder
>
>
>
> Hi!
>
>
>
> I have many folders with files as follows :
>
> /var/log/xxx/jack/program.log.1
>
> /var/log/xxx/jack/program.log.2
>
> /var/log/xxx/john/program.log.1
>
> /var/log/xxx/john/program.log.3
>
> /var/log/xxx/sam/program.log.1
>
> ...
>
>
>
> I would like to pass the "something" field to the decoder , so that the
> message fields of the decoder also have access to the "something" fields.
> Is there a way for me to do this ?
>
> I post these fields via a REST api (using an elasticsearchoutput) and
> store them in a mongo DB. I would like the "something" field also stored.
>
>
>
> [SomethingLogInput]
>
> type = "LogstreamerInput"
>
> log_directory = "/var/log/xxx"
>
> file_match = '(?P<something>\s+)/program\.log\.?(?P<Seq>\d*)'
>
> differentiator = ["something", "erlang","Seq"]
>
> decoder = "SomethingLogDecoder"
>
>
>
> Thanks!
>
> _______________________________________________
> Heka mailing list
> [email protected]
> https://mail.mozilla.org/listinfo/heka
>
>
_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka
