Hello Duncan, thank you for doing this!
Could you transform this post to http://www.freeipa.org/page/HowTos#Working_with_FreeIPA article, please? I think that other people could use that too. Thank you very much. Petr^2 Spacek On 19.12.2014 17:35, Innes, Duncan wrote: > Earlier this year I said I'd feed back how my IPA to Rsyslog to Logstash > experiments went. > > They went badly. And I didn't get much time. Today, however, I managed > to get over my imaginary finishing line: > > All systems are RHEL 6.6. > > Rsyslog (rsyslog7-7.4.10) is configured to import logs from some dirsrv > files: > > # cat /etc/rsyslog.d/dirsrv.conf > module(load="imfile" PollingInterval="2") > > input(type="imfile" > File="/var/log/dirsrv/slapd-EXAMPLE-COM/access" > Tag="dirsrv" > StateFile="statedirsrv" > Facility="local0") > > input(type="imfile" > File="/var/log/dirsrv/slapd-EXAMPLE-COM/errors" > Tag="dirsrv" > StateFile="statedirsrverr" > Severity="error" > Facility="local0") > > # > > This pulls in those log entries on a regular basis. Rsyslog8 allows you > to use inotify for file changes, but that's not available to me. > > Rsyslog is then also configured to push all logs to my Logstash servers: > > # cat /etc/rsyslog.d/logstash.conf > template(name="ls_json" type="list" option.json="on") > { constant(value="{") > constant(value="\"@timestamp\":\"") property(name="timegenerated" > dateFormat="rfc3339") > constant(value="\",\"@version\":\"1") > constant(value="\",\"message\":\"") property(name="msg") > constant(value="\",\"host\":\"") property(name="hostname") > constant(value="\",\"my_environment\":\"dev") > constant(value="\",\"my_project\":\"Infrastructure") > constant(value="\",\"my_use\":\"IPA") > constant(value="\",\"logsource\":\"") property(name="fromhost") > constant(value="\",\"severity_label\":\"") > property(name="syslogseverity-text") > constant(value="\",\"severity\":\"") property(name="syslogseverity") > constant(value="\",\"facility_label\":\"") > property(name="syslogfacility-text") > constant(value="\",\"facility\":\"") property(name="syslogfacility") > constant(value="\",\"program\":\"") property(name="programname") > constant(value="\",\"pid\":\"") property(name="procid") > constant(value="\",\"rawmsg\":\"") property(name="rawmsg") > constant(value="\",\"syslogtag\":\"") property(name="syslogtag") > constant(value="\"}\n") > } > > *.* @@logstash01.example.com:5500;ls_json > $ActionExecOnlyWhenPreviousIsSuspended on > & @@logstash02.example.com:5500;ls_json > & /var/log/localbuffer > $ActionExecOnlyWhenPreviousIsSuspended off > > [root@lvdlvldap02 ~]# > > Which pushes all logs to my logstash servers in JSON format. Failover > is built in by using 2 logstash servers. > The client needs to have SELinux managed to allow rsyslog to write to > port 5500: > > # semanage port -a -t syslogd_port_t -p tcp 5500 > # semanage port -l | grep 5500 > > The Logstash servers are then configured to listen on this port and do > some simple groking, before sending everything to the ElasticSearch > cluster: > > # cat /etc/logstash/conf.d/syslog.conf > input { > tcp { > type => syslogjson > port => 5500 > codec => "json" > } > } > > filter { > # This replaces the host field (UDP source) with the host that > generated the message (sysloghost) > if [sysloghost] { > mutate { > replace => [ "host", "%{sysloghost}" ] > remove_field => "sysloghost" # prune the field after successfully > replacing "host" > } > } > if [type] == "syslogjson" { > grok { > patterns_dir => "/opt/logstash/patterns" > match => { "message" => "%{VIRGINFW}" } > match => { "message" => "%{AUDITAVC}" } > match => { "message" => "%{COMMONAPACHELOG}" } > tag_on_failure => [] > } > } > > # This filter populates the @timestamp field with the timestamp that's > in the actual message > # dirsrv logs are currently pulled in every 2 minutes, so @timestamp > is wrong > if [syslogtag] == "dirsrv" { > mutate { > remove_field => [ 'rawmsg' ] > } > grok { > match => [ "message", "%{HTTPDATE:log_timestamp}" ] > } > date { > match => [ "log_timestamp", "dd/MMM/YYY:HH:mm:ss Z"] > locale => "en" > remove_field => [ "log_timestamp" ] > } > } > } > > output { > elasticsearch { > protocol => node > node_name => "Indexer01" > } > } > # > > It works well for the most part. I'm not performing any groking of the > actual message line as yet to pull out various bits of data into their > own separate fields, but at least I'm managing to log the access and > errors from multiple IPA servers. > > The @timestamp field ends up with the timestamp from the actual message > line, so it's only down to second accuracy. This means that multiple > log lines on the same second lose their ordering when viewed in the > Logstash/Kibana interface. But the important thing at this point is > that they're now held centrally. > > Is it feasible to alter the timestamp resolution that dirsrv uses? This > would help separate log lines properly. > > Cheers & Merry Festive Holiday thing > > Duncan > > This message has been checked for viruses and spam by the Virgin Money email > scanning system powered by Messagelabs. > > This e-mail is intended to be confidential to the recipient. If you receive a > copy in error, please inform the sender and then delete this message. > > Virgin Money plc - Registered in England and Wales (Company no. 6952311). > Registered office - Jubilee House, Gosforth, Newcastle upon Tyne NE3 4PL. > Virgin Money plc is authorised by the Prudential Regulation Authority and > regulated by the Financial Conduct Authority and the Prudential Regulation > Authority. > > The following companies also trade as Virgin Money. They are both authorised > and regulated by the Financial Conduct Authority, are registered in England > and Wales and have their registered office at Jubilee House, Gosforth, > Newcastle upon Tyne NE3 4PL: Virgin Money Personal Financial Service Limited > (Company no. 3072766) and Virgin Money Unit Trust Managers Limited (Company > no. 3000482). > > For further details of Virgin Money group companies please visit our website > at virginmoney.com > -- Petr^2 Spacek -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go To http://freeipa.org for more info on the project
