Thanks for the input Trink. I was able to get this working by using a multi-decoder as you've described.
Chris On Apr 21, 2015, at 10:31 AM, Michael Trinkala wrote: Yeah, the apache decoder currently expects the data in the message payload. Since your local Heka instance is sending the logs protobuf encoded to the remote Heka (with the full log line in the message payload) this can be achieved by using a multi decoder with a protobuf decoder first or by modifying the apache decoder to read the raw message, decode it, and then process the payload as normal. Some other options: - apache decode the logs on the local Heka before sending it to the remote Heka - change your local Heka to stream the log as text and split/decode it on the remote side Trink On Mon, Apr 6, 2015 at 2:33 PM, J. Christopher Giordano <[email protected]<mailto:[email protected]>> wrote: Heka community: I’m new to Heka and am having some difficulties setting up delivery of Apache log messages from a local running Heka agent to a remote Heka instance via TCP Output/Input plugins. The issue is directly coupled to using the Apache Access Log Decoder configured on at the remote Heka instance. When this is configured on my TCPInput there is no message Payload available to the [lua] decoder. I have been able to create a working configuration that does not use the Apache Access Log Decoder but would like to request assistance on how to troubleshoot this issue further. What I have done thus far is to modify the function process_message() in the file: lua_decoders/apache_access.lua I’ve added a field called tcplog containing the Payload to verify there is no message available to parse. The modification I’ve made is described following the configurations below of my working configuration and non-working configuration for comparison. I must add that I am able to use the Apache Access Log Decoder with the LogstreamerInput to process local files. So, this issue is specifically related to the TCPInput/Apache Access Log Decoder combination. My installation is Heka 0.9.1 on Ubuntu 14.04 # hekad -version 0.9.1 # lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 14.04.2 LTS Release: 14.04 Codename: trusty I didn’t see any outstanding bugs related to my issues. Any advice would be greatly appreciated. Thanks, Chris Local running Heka agent used to ship Apache logs to remote Heka instance: **************************************************************************************** [test_com] type = "LogstreamerInput" log_directory = "/export/test/apache2/test_com" file_match = '/(?P<Year>)\d+/(?P<Month>\d+)_(?P<Day>\d+)_access\.log' priority = ["Year", "Month", "Day"] [aggregator_output] type = "TcpOutput" address = "10.10.10.1:5565<http://10.10.10.1:5565/>" message_matcher = “TRUE” Remote Heka instance - Working config **************************************************************************************** [TcpInput] address = ":5565" [Influxdb] type = "SandboxEncoder" filename = "lua_encoders/schema_influx.lua" [Influxdb.config] series = "%{logger}" skip_fields = "Pid EnvVersion" [FileOutput] message_matcher = "TRUE" path = "/home/giordano/heka/output.log" perm = "775" flush_count = 100 flush_operator = "OR" encoder = "Influxdb" Remote Heka instance - Configuration not working config **************************************************************************************** [TcpInput] address = ":5565" decoder = "CombinedLogDecoder" [CombinedLogDecoder] type = "SandboxDecoder" filename = "lua_decoders/apache_access.lua" [CombinedLogDecoder.config] type = "combinedutrack" user_agent_transform = false payload_keep = true # combinedutrack log format log_format = "%v %h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-Agent}i\" \"%{Cookie}i\"" [Influxdb] type = "SandboxEncoder" filename = "lua_encoders/schema_influx.lua" [Influxdb.config] series = "%{logger}" skip_fields = "Pid EnvVersion" [FileOutput] message_matcher = "TRUE" path = "/home/giordano/heka/output.log" perm = "775" flush_count = 100 flush_operator = "OR" encoder = “Influxdb" Additions to the apache access log decoder process_message() function **************************************************************************************** function process_message () local log = read_message("Payload") local fields = grammar:match(log) — if not fields then return -1 end if not fields then fields = {} end fields.tcplog = log msg.Timestamp = fields.time fields.time = nil … Sample output from non working configuration **************************************************************************************** [{"points":[[1427985263000,"combinedutrack","","","",7,""]],"name":"%{logger}",”columns":["time","Type","Payload","Hostname","Logger","Severity",”tcplog”]}] _______________________________________________ Heka mailing list [email protected]<mailto:[email protected]> https://mail.mozilla.org/listinfo/heka
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
