Hi Tim,
On 18.12.2009 03:26, Tim Whittington wrote:
Hi all
We're experiencing issues with the Tomcat Connector log in some IIS production
sites where the log file grows to a very large size (8GB on one site).
This is almost entirely due to connection errors between the front end and back
end produced when the back-end Tomcat is restarted after crashes or for
maintenance.
(we have some very high request volume sites - some with hundreds of concurrent
requests -, so you can get thousands of these errors on each restart).
I'm thinking about implementing log file rotation in the IIS connector as a
solution to this, and wanted to canvas opinions on alternatives and
implementation first.
Good thing, that was open for a long time for the isapi redirector.
What I intend implementing is:
* Daily log file rolling (not configurable) as a clone of the log_to_file
function (check rotation, close and reopen log file, continue)
* Very simple configuration for a daily rolling file appender - basically
'on/off' and 'number of days to keep', possibly a single rotateable=bool/int
parameter.
* The log file name suffix will be appended/inserted to the name of the file
specified in the connector config with a static pattern e.g. YYYY_MM_dd
* The log method(s) will be guarded with a mutex to protect the rollover
operation
* I'll only be doing this for the ISAPI redirector at present (so won't need to
refactor jk_mt.h to make the locking primitives available on prefork builds)
* On rollover, the file will be closed and a new one opened. Older files will
be pruned.
* Rollover checking can be a simple mod operation on the timestamp (since I'm
not intending to handle log formats)
Questions I have:
* Short of silencing the logging, is there an alternative to rolling the file?
(perhaps rolling up of repeated connection errors?)
* Are there any performance concerns about using a mutex on each log? (log4c
uses this approach, so I presume it's not prohibitive)
* Does anyone have any other input into config/behaviour?
Mutex
=====
- log rotation for mod_jk with Apache is done using an external process
and logging through a pipe. The external process often used with Apache
is rotatelogs, that comes with apache and thus is also available under
ASL 2.0
- log path for Apache should not change seriously, e.g. no additional
mutex when running mod_jk inside Apache.
- for the redirector I'd say adding a mutex is OK, if it is acquired
after the log line has been formatted and immediately before the line
gets written to the file. So you'll only lock the access to the file.
In apache-2.0/mod_jk.c you will find for example jk_log_to_file(), which
does
rv = apr_global_mutex_lock(jk_log_lock);
if (rv != APR_SUCCESS) {
... log error via Apache ...
}
rv = apr_file_write(p->jklogfp, what, &wrote);
if (rv != APR_SUCCESS) {
... log error via Apache ...
}
rv = apr_global_mutex_unlock(jk_log_lock);
if (rv != APR_SUCCESS) {
... log error via Apache ...
}
Rolling
=======
You could also think about doing the rolling externally like we do with
Apache and pipes in case the user configured a pipe as the log file. I'm
not sure about ISAPI and the IIS process management, whether this is
feasable, e.g. all IIS processes sharing the file descriptor to the pipe
etc.
Apart from that I think having a simple daily solution, which can be
activated or deactivated would already be a very useful start. We can
always add more features, once the basics are there. Obvious possible
extensions are more fine grained rolling intervals, alternatively basing
the rolling on file size, flexible file name pattern, etc.
Pruning
=======
I'm not a big fan of pruning the old files directly from the server
process. I'd say pruning is part of some external log file management,
which usually does compression, transferral to a log server and removing
old files once they are old enough or have been relocated.
Automatic pruning is nice for people to get a quick start withour caring
about those details, but it should at least be possible to turn it off.
Log message dedup
=================
Reducing the log volume is not easy to do. Of course we could keep track
of state and not log problems we already noted, but then you would also
like to log when the state changes to something better. That means
comparing state also in case something worked correctly and will make
our cluttered code even more difficult to understand.
I remember Mladen once mentioning, that he also has some ideas or
possibly code for log rotation in the isapi redirector, so he might want
to chime in as well.
Thanks for taking the initiative!
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
