må. den 19. 06. 2023 klokka 13.51 (-0700) skreiv David Lang: > one common mistake that's made when using name-value pairs in log messages is > using the same key multiple times. Please try to add some protection against > that > > Personally, I don't mind JSON, and the ability to have multi-level structures > can signficiantly clarify logs, but defining a way to encode a hierarcy in > the > key field (for example, use . as a separator, so source.name source.ip > source.interface are related and clearly different thatn dest.name dest.ip > dest.interface) >
two points: we are using Vector to inject log files in ElasticSearch, and it has built-in support for turning k=v pairs into JSON documents for indexing and storage. it would be nice if the quoting technique used was compatible with this (I think it is, but haven't checked closely.) Elasticsearch has tried to come up with standards for field names for easier analysis across products, Elastic Common Schema. take a look at https://www.elastic.co/guide/en/ecs/current/ecs-reference.html David's examples are very similar to how ECS organise the namespace :) > On Mon, 19 Jun 2023, Ricardo Signes via Devel wrote: > > > > For about three years, Cyrus has been moving to use the new xsyslog > > function for syslogging, meant to have a more uniform format for easier > > parsing or scanning. Meanwhile, inside Fastmail, we've begun moving to > > logfmt <https://brandur.org/logfmt>, which is similar but not the same. > > This format is pretty easy for a human to read, most of the time, but also > > easy for a machine to read. I think Cyrus should start moving from the > > existing xsyslog format to logfmt. > > > > We (Fastmail) are primarily using our own Log::Fmt Perl library > > <https://metacpan.org/dist/Log-Dispatchouli/source/lib/Log/Fmt.pm>. You > > can read the logfmt "spec <https://pkg.go.dev/github.com/kr/logfmt>", from > > the Go source. Our behavior is not exactly the same, but I suspect the > > difference is largely theoretical as regards Cyrus. Our behavior is: > > • every logfmt line is a sequence of k=v pairs separated by spaces > > • every key must only contain octets in [\x21\x23-\x3C\x3E-\x7E] that was a pretty strange list: ! # < > and ?…~ this excludes "." as a hierarchy delimiter, which ECS uses. > > • any value that does not match the same rule as a key must be quoted and > > escaped > > • values are quoted by being placed inside "..." > > • escaping… > > • replaces \ with \\ > > • replaces " with \" > > • replaces \x0a and \x0d with \n and \r respectively > > • replaces any control character or vertical whitespace with \x{...} > > where ... is the hex value of the character > > I'm sending this email after some discussion about using logfmt on our > > weekly call that ended with me saying, "I'll send notes on just what we do > > in the Perl code." -- venleg helsing, Kjetil T. ------------------------------------------ Cyrus: Devel Permalink: https://cyrus.topicbox.com/groups/devel/T4f31bdf24bafdf26-M37be8938792d3ab1e701230d Delivery options: https://cyrus.topicbox.com/groups/devel/subscription
