Thanks all for your comments. I think my question is now answered and I accept that analog could ot be updated to do the writing out log records thing.
If anyone knows of a good log filtering application that is really fast and has the simplicity and flexibility for defining the kind of includes/excludes, thresholds, log file format rules etc. that are found in analog (without having to get into writing perl scripts or similar) then I would be very interested. Regards, Danny Freedman BBCi Stats Manager :-----Original Message----- :From: [EMAIL PROTECTED] [mailto:owner-analog- :[EMAIL PROTECTED] On Behalf Of Jeremy Wadsack :Sent: 26 February 2004 19:52 :To: [EMAIL PROTECTED] :Subject: Re: [analog-help] Can analog be used to filter log files : : : :>>:Analog processes log file lines in a stream, sequentially. As each :>>:line is encountered it's checked against the data filters to determine :>>:if it will be included and then the data is added to the hashes that :>>:hold the results for the reports. (This is simplified of course.) So :>>:there is no place where Analog really could write out a log file of :>>:the lines that are just in the Request Report. In fact the cache files :>>:are fairly close to a serialized version of a memory dump just before :>>:reports are written. : :>>[DF] this gets to the heart of my question. Could the log line not :>>therefore be written out at the point at which it is determined it will :>>be included? : :> This seems a little distant from Analog's stated purpose. [DF] I don't think that is a reason in itself not to repurpose some very fine code. : :Agreed. Also, the original questions was for Analog to write out all :the lines that corresponded to entries in the Request Report. Perhaps :that's just different interpretations of the same thing, but the lines :that are filtered with the date-level *INCLUDE/EXCLUDE commands also :include lines that end up in the Redirected Requests Report, Failed :Requests Report, etc. And, of course, the Request Report is also :filtered by REQINCLUDE/REQEXCLUDE which is applied to the lines in the :reports AFTER all log file lines have been processed (i.e. to the data :that's similar to the cache files). [DF] The request report was an example - but probably the most useful. My idea could only work if the selections are made as each log line is read in. If, as you say, this happens only after all the data is collated then it wouldn't work. : :However, I also recognize the advantage of only having to manage :configurations in a single place (your Analog configuration files) and :the value of not having to run multiple passes through the log files. :One "solution" (probably not optimal) is to use grep/perl to :pre-process all the log files Analog reads, rather than filter them :with any of HOSTINCLUDE/HOSTEXCLUDE, FILEINCLUDE/FILEEXCLUDE, :BROWINCLUDE/BROWEXCLUDE, REFINCLUDE/REFEXCLUDE, :USERINCLUDE/USEREXCLUDE, VHOSTINCLUDE/VHOSTEXCLUDE, or :STATUSINCLUDE/STATUSEXCLUDE in your Analog configurations. [DF] We already pre-process our log files with a C program to get rid of large amounts of data (e.g. graphics, known robots) but we still have to do some serious filtering on what is left, and analog is far more flexible when it comes to defining include/exclude parameters. : :As a trick/hack, you can actually use the UNCOMPRESS command to :trigger any pre-processor on any file types. So if you did have a :script called "filter-logfiles" you could do something like :"UNCOMPRESS filter-logifles *". Of course, this means you can't have :any other UNCOMPRESS commands, but with the latest 5.91beta1 you :probably won't need any anymore. [DF] Yes, we have experimented with this approach. But it still doesn't give access to the analog include/exclude rules. : : : :-- : :Jeremy Wadsack :Wadsack-Allen Digital Group : :+---------------------------------------------------------------------- -- :| TO UNSUBSCRIBE from this list: :| http://lists.isite.net/listgate/analog-help/unsubscribe.html :| :| Digest version: http://lists.isite.net/listgate/analog-help-digest/ :| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general :| List archives: http://www.analog.cx/docs/mailing.html#listarchives :+---------------------------------------------------------------------- -- BBCi at http://www.bbc.co.uk/ This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated. If you have received it in error, please delete it from your system. Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately. Please note that the BBC monitors e-mails sent or received. Further communication will signify your consent to this. +------------------------------------------------------------------------ | TO UNSUBSCRIBE from this list: | http://lists.isite.net/listgate/analog-help/unsubscribe.html | | Digest version: http://lists.isite.net/listgate/analog-help-digest/ | Usenet version: news://news.gmane.org/gmane.comp.web.analog.general | List archives: http://www.analog.cx/docs/mailing.html#listarchives +------------------------------------------------------------------------
