Hi Robert Point 1 is exactly what I am after but to be honest I have no idea if I am using LogstreamerInput correctly or not, I am fairly new to Unix as a whole and even newer yet t any form of coding. My current issue is how to figure out what each of these unique 'Logger' values are (or who to set them) so I can flag each of these inputs as different streams, I work better on working examples but currently I am struggling to find much information on Heka outside of the heka readme documentation and this email system.
Here is what I currently have in its entirely, it is not in a working state and it's because of the message matcher line causing a syntax error, I am hoping its ok to just show you my current setup in this way. (This setup is really just what I have made to work from multiple attempts and retries, so it will be slighty messy I am sorry about that) --------------------------------------------------------------------- On the client server (sending logs to the central server): --------------------------------------------------------------------- [hekad] maxprocs = 2 [TcpInput] address = "127.0.0.1:5565" [StatsdInput] [StatAccumInput] ticker_interval = 1000 emit_in_fields = true [gardens_wellington] type = "LogstreamerInput" log_directory = "/var/log/apache2" file_match = 'gardens(?P<FileName>[^/]+).log' [wordpress_wellington] type = "LogstreamerInput" log_directory = "/var/log/apache2" file_match = 'wordpress(?P<FileName>[^/]+).log' [PayloadEncoder] append_newlines = false [LogOutput] message_matcher = "TRUE" encoder = "PayloadEncoder" [aggregator_output] type = "TcpOutput" address = "CentralServer-IP:5565" local_address = "LocalServer-IP:5565" message_matcher = "TRUE" --------------------------------------------------------------------- On the Central Server: --------------------------------------------------------------------- [hekad] maxprocs = 2 [LogstreamerInput] log_directory = "/var/log" file_match = 'auth\.log' [StatsdInput] [StatAccumInput] ticker_interval = 1000 emit_in_fields = true [TcpInput] address = ":5565" [PayloadEncoder] append_newlines = false [LogOutput] message_matcher = "TRUE" encoder = "PayloadEncoder" [gardens_wellington] type = "FileOutput" #message_matcher = "Hostname == 'wellington' || Hostname == '147.109.11.247'" message_matcher = "(Hostname == 'wellington' || Hostname == '147.109.11.247') && Logger = 'TRUE'" path = "/logs/production/apache2/wellington-gardens-%Y%m%d-apache2.log" perm = "666" encoder = "PayloadEncoder" rotation_interval = 24 [wordpress_wellington] type = "FileOutput" #message_matcher = "Hostname == 'wellington' || Hostname == '147.109.11.247'" message_matcher = "(Hostname == 'wellington' || Hostname == '147.109.11.247') && Logger = 'TRUE'" path = "/logs/production/apache2/wellington-wordpress-%Y%m%d-apache2.log" perm = "666" encoder = "PayloadEncoder" rotation_interval = 24 ------------------------------------------------------------------------------------------------------------------------ The hashed out section in the above configuration (on the central server) is the filter I used before I attempted to split the output into multiple files. As I said above I know this configuration is wrong and it isn't working, but I have no idea what to set the values of Logger and how to make the FileOutput tell the difference between the two different outputs. Thank you again for any help on this, I'm sure this will end up being something stupid and easy but as I mentioned above my experience with this sort of work is not great. -----Original Message----- From: Rob Miller [mailto:[email protected]] Sent: Tuesday, 24 May 2016 3:35 AM To: Watson, Chris (DPIPWE) Cc: [email protected] Subject: Re: [heka] Heka "TCPOutput" to remote server to 'FileOutput' matching question At this point you have two options: 1. Create a different LogOutput on the central server for each output file that you want to generate, and give each of them a message_matcher such that they only catch the entries that are specific to the file(s) they should be catching. LogstreamerInput will set the `Logger` value uniquely for each separate input plugin (or for each separate Logstream, if a single input plugin is generating multiple streams) so matching on the Logger value is probably what you'd want. Note that this assumes that you're using LogstreamerInput correctly in that you're not interleaving unrelated log files into a single stream. 2. Alternatively, you could use a SandboxOutput, which would let you write a bit of Lua code to do whatever you want. This would give you a lot more flexibility. You'd only need a single output, which would look at each message coming in and make a decision re: where to write it out to, even generating new files on the fly if new message types start coming in. Hope this helps, -r On 05/22/2016 07:45 PM, Watson, Chris (DPIPWE) wrote: > HI all > > I have been using heka as a proof of concept on remote central logging > at my office, everything is working great so far, but I am a little > confused on the next step. > > Currently I have been using TCPOutput to write to a remote server any > and all logs in directories to a single file, for each server we have EG: > > On output box: > [Accesslogs_apache] > > type = "LogstreamerInput" > > log_directory = "/var/log/apache2" > > file_match = '(?P<FileName>[^/]+).log' > > [LogOutput] > > message_matcher = "TRUE" > > encoder = "PayloadEncoder" > > [server1_output] > > type = "TcpOutput" > > address = "central-server-IP:5565" > > local_address = "Local-IP:5565" > > message_matcher = "TRUE" > > Then on central-server: > > [apache_server1] > > type = "FileOutput" > > message_matcher = "Hostname == 'server1' || Hostname == 'server1-IP'" > > path = "/logs/production/apache2/server1-%Y%m%d.log" > > perm = "666" > > encoder = "PayloadEncoder" > > rotation_interval = 24 > > that all works fine. > > My issue is I now need to produce multiple log files from the same > server and have them output to different files as dumping multiple > logs from server1 into one large file on the central server is no > longer practical, but I am having a really hard time getting this working EG: > > Server1 has two different logs files: ‘foo.log’ and ‘bar.log’ > > I want both these logs to be passed to central-server and both of them > to create different FileOutput’s to separate outputs so they appear > on the Central server: > > /logs/production/apache2/foo.log > > /logs/production/apache2/bar.log > > Currently they all being dumped to a single file > > "/logs/production/apache2/server1-%Y%m%d.log" > > Where do I need to make the definition that these are two separate > files, on the central server or on server1. > > > ---------------------------------------------------------------------- > -- > > CONFIDENTIALITY NOTICE AND DISCLAIMER > The information in this transmission may be confidential and/or > protected by legal professional privilege, and is intended only for > the person or persons to whom it is addressed. If you are not such a > person, you are warned that any disclosure, copying or dissemination > of the information is unauthorised. If you have received the > transmission in error, please immediately contact this office by > telephone, fax or email, to inform us of the error and to enable > arrangements to be made for the destruction of the transmission, or > its return at our cost. No liability is accepted for any unauthorised > use of the information contained in this transmission. > > > _______________________________________________ > Heka mailing list > [email protected] > https://mail.mozilla.org/listinfo/heka > ________________________________ CONFIDENTIALITY NOTICE AND DISCLAIMER The information in this transmission may be confidential and/or protected by legal professional privilege, and is intended only for the person or persons to whom it is addressed. If you are not such a person, you are warned that any disclosure, copying or dissemination of the information is unauthorised. If you have received the transmission in error, please immediately contact this office by telephone, fax or email, to inform us of the error and to enable arrangements to be made for the destruction of the transmission, or its return at our cost. No liability is accepted for any unauthorised use of the information contained in this transmission. _______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
