On 9/12/2016 3:48 PM, Aaron Greenspan wrote: > I have been on this list for some time because I know that any time I > try to do anything related to Solr I’m going to have to spend hours on > it, wondering why everything has to be so awful, and I just want > somewhere to provide feedback with the dim hope that the product might > improve one day. (So far, for my purposes, it hasn’t.) Sure enough, I > still absolutely hate using Solr, and I have more feedback.
First, let me thank you for mentioning your experiences. It's harsh feedback, but I still welcome it. I'm going to say some things that may sound to you like excuses ... and you aren't really wrong to think that, but we *do* take your comments seriously, and in many cases, we already know that there are problems we need to solve. As others have said, and as I'm sure you probably know, open source is created by people trying to solve a problem for themselves, and is done on a volunteer basis. If a project is lucky, it attracts interested volunteers and truly magical things happen for the project users. I think Solr is a good project, with an awesome community. Beginner documentation is one of the hardest things to write. It's difficult for people who live and breathe the software to view the system from the perspective of someone who has never touched it at all, and to write something that explains to that novice exactly how to make it work. > I started with a confusing error on the web console, which I still > can’t figure out how to password protect without going through an > insanely process involving "ZooKeeper," which I don’t know anything > about, or have, to the best of my knowledge: Problem accessing /solr/. > Reason: Forbidden This particular part of your message involves an old fight in the Solr project: Security. Those of us who have been in the industry forever have learned that many of the security features that people expect for Internet-facing services are not at all helpful for the security of internal systems like Solr. The best thing you can do to prevent problems is to place Solr in a location where it cannot be reached by people who cannot be trusted. If you can trust those who have access, then there's no need for intrinsic security features. Any security that you layer on top of Solr (encryption, authentication, etc) is useless if somebody compromises the system that talks to Solr, which already has all the keys/passwords/etc required to get right in. As evidenced by the fact that authentication has come to Solr, we *are* listening to our users that demand security features. The authentication feature that you are trying to use, which involves basic username/password authentication of the API calls that the admin UI makes (*not* the admin UI itself), was originally developed for SolrCloud -- which utilizes Zookeeper as a central configuration database. Work is underway right now to bring basic authentication to standalone Solr, but it is not going to be available until at least 6.3, and may take even longer to finish. It will also require separate configuration on every host, which is not required for SolrCloud. > According to logs, this apparently meant that a MySQL query had failed > due to a field name change. Since I would have to change my XML > configuration files, I decided to use the opportunity to upgrade from > Solr 5.1.4 to 6.2.0. It broke everything. For almost ANY software, but especially for an open source package, upgrading to a new major version is a good way to cause problems, not usually a good way to solve them. Solr does maintain compatibility with configs that are completely current for the later versions in the previous major release ... but there are a LOT of configs out in the world (even in the latest versions of their software!) that were originally designed for Solr 3.x, 4.x, or *early* 5.x. Solr makes zero guarantees about configs designed for software that old. For an example of a similar situation in a different software package: If you try to copy configs for the Apache webserver (httpd) from a 2.2 install to a 2.4 install, chances are excellent that you're going to have to change those configs before Apache will even start, much less operate as expected. Upgrading Apache from 2.2 to 2.4 is technically a "minor" version upgrade, but in terms of capability and configuration, is similar to a major Solr version upgrade. > First I was getting errors about "Unsupported major.minor version > 52.0", so I needed to install the Linux x64 JRE 1.8.0, which I managed > on CentOS 6 with... yum install openjdk-1.8.0 ...going to Oracle’s web > site, downloading the latest JRE 1.8 build, and then running... yum > localinstall jre-8u101-linux-x64.rpm So far so good. But I didn’t have > JAVA_HOME set properly apparently, so I needed to do the > not-exactly-intuitive… export > JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.101-3.b13.el6_8.x86_64/jre/ Others have already covered this topic. Managing Java installations is required before Solr will work, and there are so many ways to handle it that Solr cannot cover the topic in its documentation. > As usual, I manually moved over my mysql-connector-java-5.1.38-bin.jar > file from the dist/ folder in the old version to the new one. Then > after stopping the old process (with kill -9, since there seems to be > no graceful way to shut down Solr—README.txt doesn’t mention bin/solr > stop) I moved over my two core folders from the old server/solr/ > folder. I tried to start it up with bin/solr start, and watched the > errors roll in. A standard "kill" would have stopped Solr gracefully, with no need for the -9 signal. It can take a while to happen, though. Using "kill -9" can cause large problems on restart that are difficult to track down. If you change the the extracted download or the installed program root (normally /opt/solr on Linux), and type "bin/solr" without options, it gives you enough information to track down the stop command. If you do use the installer script for Linux with default options, you can typically type "service solr stop" because the installer handles an init script. > There was some kind of problem with StopFilterFactory and the > text_general field type. Thanks to Stack Overflow I was able to > determine that the apparent problem was that there was a parameter, > previously fine, which was no longer fine. So I removed all instances > of enablePositionIncrements="true". That helped, but then I ran into a > broader error: "Plugin Initializing failure for [schema.xml] > fieldType". It didn’t say which field type. Buried in the logs I found > a reference in the Java stack trace—which *disappears* (and distorts > the viewing window horribly) after a few seconds when you try to view > it in the web log UI—to the string "units="degrees"". Sure enough, > this string appeared in my schema.xml for a class called > "solr.SpatialRecursivePrefixTreeFieldType" that I’m pretty sure I > never use. I removed that parameter, and moved on to the next set of > errors. The actual logfile (solr.log) is generally MUCH more useful for tracking down problems than the Logging tab in the admin UI. This is another example of something that is not immediately obvious to a novice. > Apparently there is some aspect of the Thai text field type that Solr > 6.2.0 doesn’t like. So I disabled it. I don’t use Thai text. Now Solr > was complaining about "Error loading class > 'solr.admin.AdminHandlers'". So I found the reference to > solr.admin.AdminHandlers in solrconfig.xml for each of my cores and > commented it out. Only then did Solr work again. This was not a smooth > process. It took about two hours. The user interface is still as buggy > as an early alpha of most products, the errors are difficult to > understand when they don’t actually specify what’s wrong (and they > almost never do), and there should have been an automatic process to > highlight and fix problems in old (pre-6) configuration files. Never > mind the fact that the XML-based configuration process is an > antiquated nightmare when the rest of the world has long since moved > onto databases. I've already mentioned how configs designed for very old versions won't work. It sounds like your configs were designed for 4.x, which means they will usually work in 5.x, but won't work if you go higher. Zookeeper is a configuration database. If it is properly set up, SolrCloud uses zookeeper to build a true cluster of nodes with no single point of failure and very little config stored on individual servers. It's true that the config files that you add to the zookeeper database are still in XML format. XML is a well-defined standard that is particularly good at holding structured configuration information. Switching to another format like JSON is possible, but that has a strong potential to cause *additional* upgrade headaches, and I think you've already encountered enough of those! Solr's documentation is not where we want it to be. Detailed step-by-step instructions for novices (and tools to match) is the biggest omission. It would also be useful if we had more automated tools for upgrading. Error messages are another area for improvement. Writing all these things is not easy. Good documentation is HARD. Writing tools that work right takes time. It's a struggle to provide error messages that are helpful for *developers* ... error messages that are helpful for *users* are even harder. The change in Solr 5.0 where we took control of the servlet container away from the user was one of the first steps in making the end-user experience more consistent and easier to document. It was only an initial step, though. There is lots of work to do. If you're able to help, we would appreciate it. https://wiki.apache.org/solr/WhyNoWar Thanks, Shawn
