Yeah, the apache decoder currently expects the data in the message payload. Since your local Heka instance is sending the logs protobuf encoded to the remote Heka (with the full log line in the message payload) this can be achieved by using a multi decoder with a protobuf decoder first or by modifying the apache decoder to read the raw message, decode it, and then process the payload as normal.
Some other options: - apache decode the logs on the local Heka before sending it to the remote Heka - change your local Heka to stream the log as text and split/decode it on the remote side Trink On Mon, Apr 6, 2015 at 2:33 PM, J. Christopher Giordano <[email protected]> wrote: > Heka community: > > I’m new to Heka and am having some difficulties setting up delivery of > Apache log messages from a local running Heka agent to a remote Heka > instance via TCP Output/Input plugins. The issue is directly coupled to > using the Apache Access Log Decoder configured on at the remote Heka > instance. When this is configured on my TCPInput there is no message > Payload available to the [lua] decoder. I have been able to create a > working configuration that does not use the Apache Access Log Decoder but > would like to request assistance on how to troubleshoot this issue further. > > What I have done thus far is to modify the function process_message() in > the file: lua_decoders/apache_access.lua I’ve added a field called tcplog > containing the Payload to verify there is no message available to parse. > The modification I’ve made is described following the configurations below > of my working configuration and non-working configuration for comparison. > > I must add that I am able to use the Apache Access Log Decoder with the > LogstreamerInput to process local files. So, this issue is specifically > related to the TCPInput/Apache Access Log Decoder combination. > > My installation is Heka 0.9.1 on Ubuntu 14.04 > > # hekad -version > 0.9.1 > > # lsb_release -a > No LSB modules are available. > Distributor ID: Ubuntu > Description: Ubuntu 14.04.2 LTS > Release: 14.04 > Codename: trusty > > I didn’t see any outstanding bugs related to my issues. Any advice would > be greatly appreciated. > > Thanks, > > Chris > > Local running Heka agent used to ship Apache logs to remote Heka instance: > > > **************************************************************************************** > [test_com] > type = "LogstreamerInput" > log_directory = "/export/test/apache2/test_com" > file_match = '/(?P<Year>)\d+/(?P<Month>\d+)_(?P<Day>\d+)_access\.log' > priority = ["Year", "Month", "Day"] > > [aggregator_output] > type = "TcpOutput" > address = "10.10.10.1:5565" > message_matcher = “TRUE” > > Remote Heka instance - Working config > > > **************************************************************************************** > [TcpInput] > address = ":5565" > > [Influxdb] > type = "SandboxEncoder" > filename = "lua_encoders/schema_influx.lua" > > [Influxdb.config] > series = "%{logger}" > skip_fields = "Pid EnvVersion" > > [FileOutput] > message_matcher = "TRUE" > path = "/home/giordano/heka/output.log" > perm = "775" > flush_count = 100 > flush_operator = "OR" > encoder = "Influxdb" > > Remote Heka instance - Configuration not working config > > > **************************************************************************************** > [TcpInput] > address = ":5565" > decoder = "CombinedLogDecoder" > > [CombinedLogDecoder] > type = "SandboxDecoder" > filename = "lua_decoders/apache_access.lua" > > [CombinedLogDecoder.config] > type = "combinedutrack" > user_agent_transform = false > payload_keep = true > > # combinedutrack log format > log_format = "%v %h %l %u %t \"%r\" %s %b \"%{Referer}i\" > \"%{User-Agent}i\" \"%{Cookie}i\"" > > [Influxdb] > type = "SandboxEncoder" > filename = "lua_encoders/schema_influx.lua" > > [Influxdb.config] > series = "%{logger}" > skip_fields = "Pid EnvVersion" > > [FileOutput] > message_matcher = "TRUE" > path = "/home/giordano/heka/output.log" > perm = "775" > flush_count = 100 > flush_operator = "OR" > encoder = “Influxdb" > > Additions to the apache access log decoder process_message() function > > > **************************************************************************************** > > function process_message () > local log = read_message("Payload") > local fields = grammar:match(log) > — if not fields then return -1 end > if not fields then fields = {} end > fields.tcplog = log > > msg.Timestamp = fields.time > fields.time = nil > … > > Sample output from non working configuration > > > **************************************************************************************** > > > [{"points":[[1427985263000,"combinedutrack","","","",7,""]],"name":"%{logger}",”columns":["time","Type","Payload","Hostname","Logger","Severity",”tcplog”]}] > > > > > _______________________________________________ > Heka mailing list > [email protected] > https://mail.mozilla.org/listinfo/heka > >
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
