I am often asked to take a look at one to many Solr log files that are hundreds of megabytes to gigabytes in size. "Peaking" at this amount of logs is a bit time consuming. Anybody that does this often enough has to build a log parsing tool eventually. One off greps can only get you so far. The last log reader I started putting together was after a hurricane a year or two back while I was without power for a week. I have a 6 core machine and a lot of RAM and I wanted to be able to blow through hundreds of megabytes of log files in a few seconds (why not). So I started this multi threaded single file Solr log reader that uses MappedByteBuffers. As I have had to do a little debugging here or there over the months I have added a bit to the program. It's my current go to tool when faced with a dozen gigabyte log files. And it's pretty darn fast if you have some cores to throw at it.
At this point, to avoid losing the program and someday starting yet again, I've shared it on GitHub and invite anyone else looking to have a really good standard log analyzer available to help complete it or offer logs for testing. It's still early. It's still basically in personal, hack project phase. But as I have further needs I will continue to add to it and improve it. It already does a bit more than the current sample output in the README. My goal is to make it generic enough to read a variety of formats – either by being flexible internally or through user configuration. With a little effort, there is a lot of great information and summarization that can be pulled out of Solr logs. https://github.com/markrmiller/SolrLogReader -- - Mark http://about.me/markrmiller
