Re: Modify the log directory for dih
Hi,
I installed solr as a service onwindows using nssm.
The log4j2.xml file resides in solr\server\resources.
I managed to direct logging of the class "LogUpdateProcessorFactory" to a
new location (a sub-directory in logs).
My problem now is:
- I need to create a new log file for each dih request I send to solr.
- I need more information about indexing process: the file beeing indexed,
whether it succeded or not, the cause af failures if exists...
This is what I've added to the xml configuration file log4j2.xml:
In appenders:
[%p] %c{1.} %m%n
And In Loggers:
Now, a dih file named by date is created & the messages of the class
LogUpdateProcessorFactory are logged into it, but if tow requests are sent
in a very short interval of time, they both are logged in the same file.
I appreciate any help.
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Does SolrJ support JSON DSL?
On Sat, Oct 6, 2018 at 1:04 AM Chris Hostetter wrote: > Which is to say: there are no explicit convenience methods for it, but you > can absolutely use the JSON DSL and JSON facets via SolrJ and the > QueryRequest -- just add the param key=value that you want, where the > value is the JSON syntax... > For more convenience with relatively complex JSON facet queries, I like having some data classes and enums that make building a valid JSON facet data structure faster and safer. In the end, just before sending the query, I just use Jackson to turn the object graph into JSON.
Re: Solr AutoSuggest Configuration Issue
Context filtering, at least using the suggest.cfq parameter, was not introduced before Solr 6 to my knowledge. As Edwin, I highly recommend updating. On Mon, Oct 8, 2018 at 2:20 PM Manu Nair wrote: > Hi, > > I am using Solr 5.1 for my application. > I am trying to use the autoSuggest feature of Solr. > I want to do context filtering on the results returned by Solr suggest. > > Please help me know if this feature is supported in the version that I am > using(5.1). > Also if it works with multivalued field. I tried multiple times but it is > not working. > > I am referring the following link for details : > https://lucene.apache.org/solr/guide/6_6/suggester.html > > Please find the configuration in my solrconfig.xml as below > > > mySuggester > AnalyzingInfixLookupFactory > DocumentDictionaryFactory > name > price > text_en > false > countries > > > > Thanks alot for your help in advance. > > Regards, > Manu Nair. >
Re: checksum failed (hardware problem?)
Exactly. I have a node with checksum issue and it is alive which is good
for us since if it goes down one of the shard would be down and thus
outage.
Yes, i agree that we don't get to know if there is node having checksum
issue and thats where we are putting log monitoring which will alert if
"corrupt" or "checksum" keyword is found in the logs.
Thnx
On Mon, Oct 8, 2018 at 5:41 PM Stephen Bianamara
wrote:
> Hi Susheel,
>
> Yes, I believe you are correct on fixing a node in place. My org actually
> just cycles instances rather than repair broken ones.
>
> It's too bad that there's nothing conclusive we can look for to help
> investigate the scope. We'd love to pin this down so that we could take
> something concrete to investigate to AWS if it's a hardware failure (e.g.,
> we found a log indicating ). I haven't been able to find anything which
> might clarify that matter outside of SOLR either. Perhaps it's just not
> realistic at this time.
>
> I'm also curious about another aspect, which is that the nodes don't report
> as unhealthy. Currently a node with a bad checksum will just stay in the
> collection forever. Shouldn't the node go to "down" if it has an
> irreparable checksum?
>
> On Fri, Oct 5, 2018 at 5:25 AM Susheel Kumar
> wrote:
>
> > My understanding is once the index is corrupt, the only way to fix is
> using
> > checkindex utility which will remove some bad segments and then only we
> can
> > use it.
> >
> > This is bit scary that you see similar error on 6.6.2 though in our case
> we
> > know we are going thru some hardware problem which likely would have
> caused
> > the corruption but there is no concrete evidence which can be used to
> > confirm if it is hardware or Solr/Lucene. Are you able to use another
> AWS
> > instance similar to Simon's case.
> >
> > Thanks,
> > Susheel
> >
> > On Thu, Oct 4, 2018 at 7:11 PM Stephen Bianamara >
> > wrote:
> >
> > > To be more concrete: Is the definitive test of whether or not a core's
> > > index is corrupt to copy it onto a new set of hardware and attempt to
> > write
> > > to it? If this is a definitive test, we can run the experiment and
> update
> > > the report so you have a sense of how often this happens.
> > >
> > > Since this is a SOLR cloud node, which is already removed but whose
> data
> > > dir was preserved, I believe I can just copy the data directory to a
> > fresh
> > > machine and start a regular non-cloud solr node hosting this core. Can
> > you
> > > please confirm that this will be a definitive test, or whether there is
> > > some aspect needed to make it definitive?
> > >
> > > Thanks!
> > >
> > > On Wed, Oct 3, 2018 at 2:10 AM Stephen Bianamara <
> sbianam...@panopto.com
> > >
> > > wrote:
> > >
> > > > Hello All --
> > > >
> > > > As it would happen, we've seen this error on version 6.6.2 very
> > recently.
> > > > This is also on an AWS instance, like Simon's report. The drive
> doesn't
> > > > show any sign of being unhealthy, either from cursory investigation.
> > > FWIW,
> > > > this occurred during a collection backup.
> > > >
> > > > Erick, is there some diagnostic data we can find to help pin this
> down?
> > > >
> > > > Thanks!
> > > > Stephen
> > > >
> > > > On Sun, Sep 30, 2018 at 12:48 PM Susheel Kumar <
> susheel2...@gmail.com>
> > > > wrote:
> > > >
> > > >> Thank you, Simon. Which basically points that something related to
> env
> > > and
> > > >> was causing the checksum failures than any lucene/solr issue.
> > > >>
> > > >> Eric - I did check with hardware folks and they are aware of some
> > VMware
> > > >> issue where the VM hosted in HCI environment is coming into some
> halt
> > > >> state
> > > >> for minute or so and may be loosing connections to disk/network. So
> > > that
> > > >> probably may be the reason of index corruption though they have not
> > been
> > > >> able to find anything specific from logs during the time Solr run
> into
> > > >> issue
> > > >>
> > > >> Also I had again issue where Solr is loosing the connection with
> > > zookeeper
> > > >> (Client session timed out, have not heard from server in 8367ms for
> > > >> sessionid 0x0) Does that points to similar hardware issue, Any
> > > >> suggestions?
> > > >>
> > > >> Thanks,
> > > >> Susheel
> > > >>
> > > >> 2018-09-29 17:30:44.070 INFO
> > > >> (searcherExecutor-7-thread-1-processing-n:server54:8080_solr
> > > >> x:COLL_shard4_replica2 s:shard4 c:COLL r:core_node8) [c:COLL
> s:shard4
> > > >> r:core_node8 x:COLL_shard4_replica2] o.a.s.c.SolrCore
> > > >> [COLL_shard4_replica2] Registered new searcher
> > > >> Searcher@7a4465b1[COLL_shard4_replica2]
> > > >>
> > > >>
> > >
> >
> main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_7x3f(6.6.2):C826923/317917:delGen=2523)
> > > >> Uninverting(_83pb(6.6.2):C805451/172968:delGen=2957)
> > > >> Uninverting(_3ywj(6.6.2):C727978/334529:delGen=2962)
> > > >> Uninverting(_7vsw(6.6.2):C872110/385178:delGen=2020)
> > > >> Uninverting(_8n89(6.6.2):C741293/109260:delG
Re: Realtime get not always returning existing data
H. I wonder if a version conflict or perhaps other failure can
somehow cause this. It shouldn't be very hard to add that to my test
setup, just randomly add n _version_ field value.
Erick
On Mon, Oct 1, 2018 at 8:20 AM Erick Erickson wrote:
>
> Thanks. I'll be away for the rest of the week, so won't be able to try
> anything more
> On Mon, Oct 1, 2018 at 5:10 AM Chris Ulicny wrote:
> >
> > In our case, we are heavily indexing in the collection while the /get
> > requests are happening which is what we assumed was causing this very rare
> > behavior. However, we have experienced the problem for a collection where
> > the following happens in sequence with minutes in between them.
> >
> > 1. Document id=1 is indexed
> > 2. Document successfully retrieved with /get?id=1
> > 3. Document failed to be retrieved with /get?id=1
> > 4. Document successfully retrieved with /get?id=1
> >
> > We've haven't looked at the issue in a while, so I don't have the exact
> > timing of that sequence on hand right now. I'll try to find an actual
> > example, although I'm relatively certain it was multiple minutes in between
> > each of those requests. However our autocommit (and soft commit) times are
> > 60s for both collections.
> >
> > I think the following two are probably the biggest differences for our
> > setup, besides the version difference (v6.3.0):
> >
> > > index to this collection, perhaps not at a high rate
> > > separate the machines running solr from the one doing any querying or
> > indexing
> >
> > The clients are on 3 hosts separate from the solr instances. The total
> > number of threads that are making updates and making /get requests is
> > around 120-150. About 40-50 per host. Each of our two collections gets an
> > average of 500 requests per second constantly for ~5 minutes, and then the
> > number slowly tapers off to essentially 0 after ~15 minutes.
> >
> > Every thread attempts to make the same series of requests.
> >
> > -- Update with "_version_=-1". If successful, no other requests are made.
> > -- On 409 Conflict failure, it makes a /get request for the id
> > -- On doc:null failure, the client handles the error and moves on
> >
> > Combining this with the previous series of /get requests, we end up with
> > situations where an update fails as expected, but the subsequent /get
> > request fails to retrieve the existing document:
> >
> > 1. Thread 1 updates id=1 successfully
> > 2. Thread 2 tries to update id=1, fails (409)
> > 3. Thread 2 tries to get id=1 succeeds.
> >
> > ...Minutes later...
> >
> > 4. Thread 3 tries to update id=1, fails (409)
> > 5. Thread 3 tries to get id=1, fails (doc:null)
> >
> > ...Minutes later...
> >
> > 6. Thread 4 tries to update id=1, fails (409)
> > 7. Thread 4 tries to get id=1 succeeds.
> >
> > As Steven mentioned, it happens very, very rarely. We tried to recreate it
> > in a more controlled environment, but ran into the same issue that you are,
> > Erick. Every simplified situation we ran produced no problems. Since it's
> > not a large issue for us and happens very rarely, we stopped trying to
> > recreate it.
> >
> >
> > On Sun, Sep 30, 2018 at 9:16 PM Erick Erickson
> > wrote:
> >
> > > 57 million queries later, with constant indexing going on and 9 dummy
> > > collections in the mix and the main collection I'm querying having 2
> > > shards, 2 replicas each, I have no errors.
> > >
> > > So unless the code doesn't look like it exercises any similar path,
> > > I'm not sure what more I can test. "It works on my machine" ;)
> > >
> > > Here's my querying code, does it look like it what you're seeing?
> > >
> > > while (Main.allStop.get() == false) {
> > > try (SolrClient client = new HttpSolrClient.Builder()
> > > //("http://my-solr-server:8981/solr/eoe_shard1_replica_n4";)) {
> > > .withBaseSolrUrl("http://localhost:8981/solr/eoe";).build()) {
> > >
> > > //SolrQuery query = new SolrQuery();
> > > String lower = Integer.toString(rand.nextInt(1_000_000));
> > > SolrDocument rsp = client.getById(lower);
> > > if (rsp == null) {
> > > System.out.println("Got a null response!");
> > > Main.allStop.set(true);
> > > }
> > >
> > > rsp = client.getById(lower);
> > >
> > > if (rsp.get("id").equals(lower) == false) {
> > > System.out.println("Got an invalid response, looking for "
> > > + lower + " got: " + rsp.get("id"));
> > > Main.allStop.set(true);
> > > }
> > > long queries = Main.eoeCounter.incrementAndGet();
> > > if ((queries % 100_000) == 0) {
> > > long seconds = (System.currentTimeMillis() - Main.start) /
> > > 1000;
> > > System.out.println("Query count: " +
> > > numFormatter.format(queries) + ", rate is " +
> > > numFormatter.format(queries / seconds) + " QPS");
> > > }
> > > } catch (Exception cle) {
> > > cle.printSta
Does providing Zookeeper FQDN with port number solve the multiple connections problem
Hi Team, We have a scenario where we give the Zookeeper FQDN to our Solr Application. When we use this we are seeing that zookeeper is accepting connections from all Solr nodes every minute. When we provide the Zookeeper FQDN with the port number, we are seeing that there is a single connection created with Solr. Is this something that has been already seen earlier on. I have tested it out on my local cluster and identified this issue. Has anyone seen this issue previously? If yes, please let me know. I have tested on a cluster with *3 Solr nodes* and *3 Zookeeper nodes* Logs for one of the Zookeeper nodes *Column Description:[Date, time interval(hourly), number of connection from Solr, Number of connections from Zookeeper]* Oct 09 16-17 180 connections from solr nodes.240 connections from zk nodes Oct 09 17-18 180 connections from solr nodes.240 connections from zk nodes Oct 09 18-19 69 connections from solr nodes.93 connections from zk nodes Oct 08 10-11 120 connections from solr nodes.174 connections from zk nodes Oct 08 11-12 180 connections from solr nodes.240 connections from zk nodes Oct 08 12-13 180 connections from solr nodes.240 connections from zk nodes Oct 08 13-14 180 connections from solr nodes.240 connections from zk nodes Oct 08 14-15 180 connections from solr nodes.240 connections from zk nodes Oct 08 15-16 180 connections from solr nodes.240 connections from zk nodes Oct 08 16-17 180 connections from solr nodes.240 connections from zk nodes Oct 08 17-18 180 connections from solr nodes.240 connections from zk nodes Oct 08 18-19 180 connections from solr nodes.240 connections from zk nodes Oct 08 19-20 180 connections from solr nodes.240 connections from zk nodes Oct 08 20-21 180 connections from solr nodes.240 connections from zk nodes Oct 08 21-22 180 connections from solr nodes.240 connections from zk nodes Oct 08 22-23 180 connections from solr nodes.240 connections from zk nodes Oct 08 23-24 180 connections from solr nodes.240 connections from zk nodes Thank you, Karthik
