[issue33506] [logging] template for filename timestamps
New submission from Steve R. Hastings : When a logged file has a timed rotation, it is renamed with an archival filename that includes a timestamp. The logging library has a small set of predefined filename templates and no way is provided to override them. Also, the current rule is that while the file is in active use for logging, it will not have a timestamp in the filename; the filename will be something like: foo.log Then, at file rotation time, foo.log is closed and then renamed to an archival name that includes a timestamp (something like: foo-2018-05-14.log), and a new active log file is opened with the name foo.log again. Proposed enhancement: it should be possible to provide template codes that specify the format of the timestamp, and there should be an option that when the file is in active use its filename will include the timestamp. (Then at rotation time, the file is simply closed, with no need to rename it; and a new file with a new timestamp is opened.) For example, specifying a log filename of "foo-%Y%m%d-%H%M%S" would specify a filename like: foo-20180514-16.log Use case: the company that employs me had a logging system requiring both of the above features. The timestamp in the filename had to be in a format different than the one format built-in to the logging module, and the timestamp needed to be in the file at all times. The logging system included a daemon that did the equivalent of tail -f on the logfile and collected log events as they were written. Note: I have written code that implements the features described above. Additional note: some template formats could cause a problem with deleting the oldest backup files. When the option backupCount is set to a number >= 1, then at rotation time the handler will check whether the number of backup files exceeds the specified count, and will delete the oldest files to reduce the number of backup files to exactly the backupCount number. The oldest files are found by sorting the filenames; the presumption is that the archival filenames will sort with the oldest files first. All the built-in templates produce filenames where this presumption is correct. A foolish user could specify a timestamp format where month appears before date, or hour appears before month, or any other template that does not follow this ordering: year/month/date/hour/minutes/seconds We could also add a feature where, when the template is specified and backupCount is specified, the handler will check that the template follows the ordering so that the oldest files do sort first, and raise an exception if the template doesn't follow the ordering. Alternatively, we can add a note in the documentation warning of this issue. -- components: Library (Lib) messages: 316573 nosy: steveha, vinay.sajip priority: normal severity: normal status: open title: [logging] template for filename timestamps type: enhancement versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue33506> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33508] [logging] allow %p code to put PID into log filename
New submission from Steve R. Hastings : Python logging is not multi-process safe. When a Python program has multiple processes, one way to log safely would be to put the process ID number into the filename of the log file, giving each process its own log file. It would be convenient if a %p format code would be expanded to the process ID number when the logging system opens the log file. -- components: Library (Lib) messages: 316576 nosy: steveha, vinay.sajip priority: normal severity: normal status: open title: [logging] allow %p code to put PID into log filename type: enhancement versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue33508> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33510] [logging] add JSON log formatter
New submission from Steve R. Hastings :
Many logging aggregation and indexing systems require that log data must be
written in an easy-to-parse format such as JSON. A "batteries included" JSON
log formatter would be a useful enhancement.
I have written code with the following features:
Log events are translated into JSON. The JSON is formatted without
indentation, so that one log event will produce one line in the output file.
Each line is a single JSON object. Because log files contain many lines and
JSON logging is usually not read by humans, the JSON separator punctuation are
set to simply ':' and ',' (no leading or trailing spaces).
The standard fields of a log event turn into key/value pairs in the JSON
object, using the standard field name as the output JSON key name. For
example, a log statement like this: log.info("foo initialized") would result in
JSON that included: "message": "foo initialized", "levelname": "INFO"
By default, the JSON output has the timestamp as the first key/value pair;
therefore if the file is sorted, or multiple log files are concatenated and
then sorted, the logged events will be sorted into elapsed order. By default,
the timestamp format is RFC3339: https://tools.ietf.org/html/rfc3339 Also, by
default, the seconds field will have a fractional part with three digits (i.e.
millisecond resolution on log event timestamps).
The user can provide a dictionary that will translate key names. If the user's
log collection system requires the logging level to have a key name of
"priority", then by specifying that "levelname" should be mapped to "priority"
the log event will be output with "priority": "INFO" This dictionary also
provides a way to control which logging event fields are written to the logfile
and in what order.
Taking advantage of a feature already present in the Python logging code: The
user can provide a dictionary as the only argument after the logging message,
and it will be handled specially. For the JSON formatter, the special handling
is that the key/value pairs from the dictionary will be included in the JSON
output. Alternatively, the user can provide a message string that includes %
templating codes, and specify arguments to be templated in, as is common
practice in logging. (This is to be backwards-compatible with code written to
the current formatters.)
The Python logging code feature mentioned above can be found in
LogRecord.__init__() The first thing that init code does is check whether
there is exactly one argument and it is a non-empty Mapping.
In the case where there are multiple arguments provided and templated into the
message string, one extra key/value pair is written in the JSON object: the key
is "args" and the value is a JSON array with the values of the arguments.
Because logging is fundamental, if anything goes wrong it's important to log as
much as possible. If any exception is raised during the call to json.dumps()
that creates an output line, there is fallback code that converts the arguments
dictionary to a string and logs that string under an identifiable key name such
as "args_text" So if someone attempts to log arguments that json.dumps() can't
handle, something will be written to the logfile even if it's just something
like "<__main__.Foo object at 0x7f8fe7621da0>"
Here is a concrete example. The JSON logging formatter is initialized with the
following field name translation table:
TRANSLATE_FIELD_NAMES = {
"asctime": "@timestamp",
"levelname": "priority",
"message": None,
"process": "pid",
}
The above dictionary means that asctime, levelname, message, and process will
be written to the output file, in that order. The output JSON object will have
key names of, respectively: "@timestamp", priority, message (no change), and
pid.
Then the following logging call occurs:
log.info("connecting to server", {"url": server_url, "port": sever_port})
The output written to the log file would be similar to this:
{"@timestamp":"2018-05-14T17:28:04.112-04:00","priority":"INFO","message":"connecting
to server","pid":32073,"url":"http://127.0.0.1","port":8043}
As noted above, I have already written this code, and it has been in production
for months at the company where I work. However, the code can be simplified
and cleaned up a bit, since it contains Python 2.7 backward compatibility code.
--
components: Library (Lib)
messages: 316597
nosy: steveha, vinay.sajip
priority: normal
severity: normal
status: open
title: [logging] add JSON log formatter
type: enhancement
versions: Python 3.8
___
Python tracker
<https://bugs.python.org/issue33510>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33506] [logging] template for filename timestamps
Steve R. Hastings added the comment: > What about [ BaseRotatingHandler.namer ] which allows you to specify your own > naming scheme? I confess that I overlooked that; it was added later than the version of Python in which I wrote my original code. The current Python code still has a small set of predefined file templates, but now has a way to specify a callable that can completely override the filename. I agree that this can be used to implement any desired filename system, although it would be more work than simply specifying the desired template codes. Are you planning to remove the predefined filename templates and produce a more functional version of the code where the .namer() function is always used to produce a filename, with a set of predefined functions that produce equivalent filenames to the built-in templates? -- ___ Python tracker <https://bugs.python.org/issue33506> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33510] [logging] add JSON log formatter
Steve R. Hastings added the comment: The solution recommended in the logging cookbook does not do the same thing as I have proposed. I have proposed a logging handler that emits JSON suitable for ingestion by a log collection system. With the JSON formatter, all log events, no matter their source, are written completely in JSON. This includes log events from already-written library functions that have no expectation of JSON output. Also the JSON formatter writes timestamps in the now-universally-understood RFC3339 format, the easiest date format to parse. The logging cookbook solution is that the user must explicitly write JSON data into the message line of each call to the logging code. This would make some or all of the message field into JSON but does not solve the problem of logging in JSON; the other fields of the log file would remain non-JSON. A company that is using a log collection system might need the logs to be pure JSON. The proposed JSON formatter solves that problem. P.S. Here is a blog posting on why it is useful to log in pure JSON: https://journal.paul.querna.org/articles/2011/12/26/log-for-machines-in-json/ -- ___ Python tracker <https://bugs.python.org/issue33510> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33506] [logging] template for filename timestamps
Steve R. Hastings added the comment:
I just reviewed the Python 3.6 logging code. Unless I am mistaken, it still
has the limitation that when a logging file is active for writing log events,
that file always has the same filename (self.baseFilename). Did I overlook
anything?
At rotation time, the self.namer() function is called, and it could implement
any desired naming scheme. However, its only argument is the already templated
filename.
So if I needed to implement logging with a filename of: foo-mmdd.log at all
times, how would I get the current file (the open file to which log events are
being written) to have that pattern of name?
And if I needed to write a self.namer() function to rename to that standard, is
this the recommended approach?
# for daily rotations, the built-in template is: "%Y-%m-%d"
def namer(self, filename):
# filename will look like: foo--mm-dd.log
year = filename[-14:-10]
month = filename[-9:-7]
day = filename[-6:-4]
base = filename[:-14]
new_filename = base + year + month + day + ".log"
return new_filename # will look like: foo-mmdd.log
The logging event that triggered the rollover is not passed to the namer()
function, just the templated default filename.
In my opinion, it is more convenient for the user to simply specify the desired
format using template codes, such as:
handler = TimedRotatingFileHandler("foo", suffix="-%Y%m%d", when="D")
Also, unless I have overlooked something, using the self.namer() function to
implement a custom naming scheme completely disables the code that
automatically deletes the oldest backup files when backupCount is specified.
The getBackupFilesToDelete() function uses the self.extMatch value (a
pre-compiled regular expression) to find files to delete; with a custom naming
scheme, the files will not match the pattern. Therefore users with a custom
naming scheme would have to implement some external solution to deleting old
files after log rotation.
The code I am proposing to donate to Python automatically builds a suitable
regular expression for the self.extMatch member variable, based on the user's
specified format string. I believe this is an advantage for the code I propose
to donate.
--
___
Python tracker
<https://bugs.python.org/issue33506>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23958] compile warnings in libffi
New submission from Steve R. Hastings: The normal build should not have any warnings. The source for libffi included some comparisons between signed int and unsigned int, which caused warnings. The patch changes the signed int variables to unsigned int. -- components: Build files: libffi_warning_patch.txt messages: 241011 nosy: steveha priority: normal severity: normal status: open title: compile warnings in libffi type: behavior versions: Python 3.5 Added file: http://bugs.python.org/file39017/libffi_warning_patch.txt ___ Python tracker <http://bugs.python.org/issue23958> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
