Re: Flexible search field analyser/tokenizer configuration

[Jack Krupansky]

What exactly do you think that filter query is doing? Explain it in plain 
English.


My guess is that it eliminates all your document matches.

-- Jack Krupansky

-Original Message- 
From: PeterKerk

Sent: Saturday, October 4, 2014 12:34 AM
To: solr-user@lucene.apache.org
Subject: Re: Flexible search field analyser/tokenizer configuration

Ok, that field now totally works, thanks again!

I've removed the wildcard to benefit from ranking and boosting and am now
trying to combine this field with another, but I have some difficulties
figuring out the right query.

I want to search on the occurence of the keyword in the title field
(title_search_global) of a document OR in the description field
(description_search)
and if it occurs in the title field give that the largest boost, over a
minor boost in the description_search field.

Here's what I have now on query "Ballonnenboog"

http://localhost:8983/solr/tt-shop/select?q=(title_search_global%3A(Ballonnenboog)+OR+title_search_global%3A%22Ballonnenboog%22%5E100)+OR+description_search%3A(Ballonnenboog)&fq=title_search_global%5E10.0%2Bdescription_search%5E0.3&fl=id%2Ctitle&wt=xml&indent=true

But it returns 0 results, even though there are results that have
"Ballonnenboog" in the title_search_global field.

What am I missing?



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Flexible-search-field-analyser-tokenizer-configuration-tp4161624p4162638.html
Sent from the Solr - User mailing list archive at Nabble.com. 

Re: Master-Slave setup using SolrCloud

[Sachin Kale]

Apparently, there is a bug in Solr 4.10.0 which was causing the
NullPointerExceptions. SOLR-6501

We have updated our production SOLR to 4.10.1


On Thu, Oct 2, 2014 at 8:13 PM, Sachin Kale  wrote:

> If I look into the logs, many times I get only following line without any
> stacktrace:
>
> *ERROR - 2014-10-02 19:35:25.516; org.apache.solr.common.SolrException;
> java.lang.NullPointerException*
>
> These exceptions are not coming continuously. Once in every 10-15 minutes.
> But once it starts, there are continuous 800-1000 such exceptions one after
> another. Is it related to cache warmup?
>
> I can provide following information regarding the setup:
> We are now on using Solr 4.10.0
> Memory allocated to each SOLR instance is 7GB. I guess it is more than
> sufficient for 1 GB index, right?
> Indexes are stored as normal, local filesystem.
> I am using three caches:
> Query Cache: Size 4096, autoWarmCount 2048
> Filter cache: size 8192, autoWarmCount 4096
> Document cache: size 4096
>
> I am experimenting with commitMaxTime for both soft and hard commits
>
> After referring following:
>
> http://lucidworks.com/blog/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>
> Hence, I set following:
>
> 
> ${solr.autoCommit.maxTime:6}
> false
> 
>
> 
> ${solr.autoSoftCommit.maxTime:90}
> 
>
> Also, we are getting following warnings many times:
>
> *java.lang.NumberFormatException: For input string: "5193.0"*
>
> Earlier we were on SOLR 4.4.0 and when we are upgraded to 4.10.0, we
> pointed it to the same index we were using for 4.4.0
>
> On Thu, Oct 2, 2014 at 7:11 PM, Shawn Heisey  wrote:
>
>> On 10/2/2014 6:58 AM, Sachin Kale wrote:
>> > We are trying to move our traditional master-slave Solr configuration to
>> > SolrCloud. As our index size is very small (around 1 GB), we are having
>> > only one shard.
>> > So basically, we are having same master-slave configuration with one
>> leader
>> > and 6 replicas.
>> > We are experimenting with maxTime of both AutoCommit and AutoSoftCommit.
>> > Currently, autoCommit maxTime is 15 minutes and autoSoftCommit is 1
>> minute
>> > (Let me know if these values does not make sense).
>> >
>> > Caches are set such that warmup time is at most 20 seconds.
>> >
>> > We are having continuous indexing requests mostly for updating the
>> existing
>> > documents. Few requests are for deleting/adding the documents.
>> >
>> > The problem we are facing is that we are getting very frequent
>> > NullPointerExceptions.
>> > We get continuous 200-300 such exceptions within a period of 30 seconds
>> and
>> > for next few minutes, it works fine.
>> >
>> > Stacktrace of NullPointerException:
>> >
>> > *ERROR - 2014-10-02 18:09:38.464; org.apache.solr.common.SolrException;
>> > null:java.lang.NullPointerException*
>> > *at
>> >
>> org.apache.solr.handler.component.QueryComponent.returnFields(QueryComponent.java:1257)*
>> > *at
>> >
>> org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:720)*
>> > *at
>> >
>> org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:695)*
>> >
>> > ​
>> > I am not sure what would be causing it. My guess, whenever, it is
>> trying to
>> > replay tlog, we are getting these exceptions. Is anything wrong in my
>> > configuration?
>>
>> Your automatic commit settings are fine.  If you had tried to use a very
>> small maxTime like 1000 (1 second), I would tell you that it's probably
>> too short.
>>
>> The tlogs only get replayed when a core is first started or reloaded.
>> These appear to be errors during queries, having nothing at all to do
>> with indexing.
>>
>> I can't be sure with the available information (no Solr version,
>> incomplete stacktrace, no info about what request caused and received
>> the error), but if I had to guess, I'd say you probably changed your
>> schema so that certain fields are now required that weren't required
>> before, and didn't reindex, so those fields are not present on every
>> document.  Or it might be that you added a uniqueKey and didn't reindex,
>> and that field is not present on every document.
>>
>> http://wiki.apache.org/solr/HowToReindex
>>
>> Thanks,
>> Shawn
>>
>>
>

Re: Flexible search field analyser/tokenizer configuration

[PeterKerk]

In Engish, I think this part:
(title_search_global:(Ballonnenboog) OR
title_search_global:"Ballonnenboog"^100)
is looking for a match on "Ballonenboog" in the title and give a boost if it
occurs exactly as this.

The second part does the same but then for the description_search field, and
with an OR operator (so I would think it would not eliminate all matches:

(description_search:(Ballonnenboog) OR
description_search:"Ballonnenboog"^100)

And finally this part:

title_search_global^10.0+description_search^0.3

Gives a higher boost to the occurrence of the query in title_search_global
field than description_search field.

But something must be wrong with my analysis :)



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Flexible-search-field-analyser-tokenizer-configuration-tp4161624p4162660.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Flexible search field analyser/tokenizer configuration

[Jack Krupansky]

Thanks for the clarification. Now... "fq" is simply another query, with 
normal query syntax. You wrote two field names as if they were query terms, 
but that's not meaningful query syntax. Sorry, but there is no such feature 
in Solr.


Although the qf parameter of dismax and edismax can be used to apply a boost 
to all un-fielded terms for a field, you otherwise need to apply any boost 
on a term, not a field.


-- Jack Krupansky

-Original Message- 
From: PeterKerk

Sent: Saturday, October 4, 2014 10:43 AM
To: solr-user@lucene.apache.org
Subject: Re: Flexible search field analyser/tokenizer configuration

In Engish, I think this part:
(title_search_global:(Ballonnenboog) OR
title_search_global:"Ballonnenboog"^100)
is looking for a match on "Ballonenboog" in the title and give a boost if it
occurs exactly as this.

The second part does the same but then for the description_search field, and
with an OR operator (so I would think it would not eliminate all matches:

(description_search:(Ballonnenboog) OR
description_search:"Ballonnenboog"^100)

And finally this part:

title_search_global^10.0+description_search^0.3

Gives a higher boost to the occurrence of the query in title_search_global
field than description_search field.

But something must be wrong with my analysis :)



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Flexible-search-field-analyser-tokenizer-configuration-tp4161624p4162660.html
Sent from the Solr - User mailing list archive at Nabble.com. 

Re: Flexible search field analyser/tokenizer configuration

[PeterKerk]

Thanks, removing the fq parameters helped :)



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Flexible-search-field-analyser-tokenizer-configuration-tp4161624p4162667.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Export feature issue in Solr 4.10

[Ahmed Adel]

Thanks Joel, I changed enableLazyFieldLoading to false and it worked just
fine.

However, for some reason, I was expecting it to return
"content-disposition: attachment" in the response. Maybe because response
of this request would most probably be huge and if returned to the browser,
it makes sense to be downloaded as browser won't be able to handle it
efficiently, at least by supplying a request parameter. What do you think?

A. Adel
On Oct 2, 2014 11:06 PM, "Joel Bernstein"  wrote:

> There is bug in how the export handler is working when you have very few
> documents in the index and the solrconfig.xml is configured to enable lazy
> document loading:
>
> true
>
> The tests didn't catch this because lazy loading was set to the default
> which is false in the tests. The manual testing I did, didn't catch this
> because I tested with a large number of documents in the index.
>
> Your example will work if you change:
>
> false
>
> And if you load a typical index with lots of documents you should have no
> problems running with lazy loading enabled.
>
> I'll create jira to fix this issue.
>
>
>
>
>
>
>
>
> Joel Bernstein
> Search Engineer at Heliosearch
>
> On Thu, Oct 2, 2014 at 4:10 PM, Joel Bernstein  wrote:
>
> > Yep getting the same error. Investigating...
> >
> > Joel Bernstein
> > Search Engineer at Heliosearch
> >
> > On Thu, Oct 2, 2014 at 12:59 PM, Ahmed Adel 
> wrote:
> >
> >> Hi All,
> >>
> >> I'm trying to use Solr 4.10 export feature, but I'm getting an error.
> >> Maybe
> >> I missed something.
> >>
> >> Here's the scenario:
> >>
> >>
> >>1. Download Solr 4.10.0
> >>2. Use collection1 schema out of the box
> >>3. Add docValues="true" to price and pages fields in schema.xml
> >>4. Index books.json using command line:
> >>curl http://localhost:8984/solr/collection1/update -H
> >> "Content-Type: text/json" --data-binary
> >> @example/exampledocs/books.json
> >>5. Try running this query:
> >>
> >>
> http://localhost:8984/solr/collection1/export?q=*:*&sort=price%20asc&fl=price
> >>6. Here's the error I get:
> >>
> >>java.lang.IllegalArgumentException: docID must be >= 0 and <
> >> maxDoc=4 (got docID=4)
> >> at
> >>
> org.apache.lucene.index.BaseCompositeReader.readerIndex(BaseCompositeReader.java:182)
> >> at
> >>
> org.apache.lucene.index.BaseCompositeReader.document(BaseCompositeReader.java:109)
> >> at
> >> org.apache.solr.search.SolrIndexSearcher.doc(SolrIndexSearcher.java:700)
> >> at
> >>
> org.apache.solr.util.SolrPluginUtils.optimizePreFetchDocs(SolrPluginUtils.java:213)
> >> at
> >>
> org.apache.solr.handler.component.QueryComponent.doPrefetch(QueryComponent.java:623)
> >> at
> >>
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:507)
> >> at
> >>
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:218)
> >> at
> >>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> >> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1967)
> >> ...
> >>
> >>
> >> Any ideas what could be missing?
> >>
> >> Thanks,
> >> A. Adel
> >>
> >
> >
>

Re: Export feature issue in Solr 4.10

[Alexandre Rafalovitch]

You don't have to call the URL from the browser. "curl" is what's
usually used for this kinds of things and will save directly to disk
with the right flags. The attachment makes sense if you have some sort
of UI triggering this, but that's not what's happening here.

Regards,
   Alex.
Personal: http://www.outerthoughts.com/ and @arafalov
Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
Solr popularizers community: https://www.linkedin.com/groups?gid=6713853


On 4 October 2014 15:18, Ahmed Adel  wrote:
> Thanks Joel, I changed enableLazyFieldLoading to false and it worked just
> fine.
>
> However, for some reason, I was expecting it to return
> "content-disposition: attachment" in the response. Maybe because response
> of this request would most probably be huge and if returned to the browser,
> it makes sense to be downloaded as browser won't be able to handle it
> efficiently, at least by supplying a request parameter. What do you think?
>
> A. Adel
> On Oct 2, 2014 11:06 PM, "Joel Bernstein"  wrote:
>
>> There is bug in how the export handler is working when you have very few
>> documents in the index and the solrconfig.xml is configured to enable lazy
>> document loading:
>>
>> true
>>
>> The tests didn't catch this because lazy loading was set to the default
>> which is false in the tests. The manual testing I did, didn't catch this
>> because I tested with a large number of documents in the index.
>>
>> Your example will work if you change:
>>
>> false
>>
>> And if you load a typical index with lots of documents you should have no
>> problems running with lazy loading enabled.
>>
>> I'll create jira to fix this issue.
>>
>>
>>
>>
>>
>>
>>
>>
>> Joel Bernstein
>> Search Engineer at Heliosearch
>>
>> On Thu, Oct 2, 2014 at 4:10 PM, Joel Bernstein  wrote:
>>
>> > Yep getting the same error. Investigating...
>> >
>> > Joel Bernstein
>> > Search Engineer at Heliosearch
>> >
>> > On Thu, Oct 2, 2014 at 12:59 PM, Ahmed Adel 
>> wrote:
>> >
>> >> Hi All,
>> >>
>> >> I'm trying to use Solr 4.10 export feature, but I'm getting an error.
>> >> Maybe
>> >> I missed something.
>> >>
>> >> Here's the scenario:
>> >>
>> >>
>> >>1. Download Solr 4.10.0
>> >>2. Use collection1 schema out of the box
>> >>3. Add docValues="true" to price and pages fields in schema.xml
>> >>4. Index books.json using command line:
>> >>curl http://localhost:8984/solr/collection1/update -H
>> >> "Content-Type: text/json" --data-binary
>> >> @example/exampledocs/books.json
>> >>5. Try running this query:
>> >>
>> >>
>> http://localhost:8984/solr/collection1/export?q=*:*&sort=price%20asc&fl=price
>> >>6. Here's the error I get:
>> >>
>> >>java.lang.IllegalArgumentException: docID must be >= 0 and <
>> >> maxDoc=4 (got docID=4)
>> >> at
>> >>
>> org.apache.lucene.index.BaseCompositeReader.readerIndex(BaseCompositeReader.java:182)
>> >> at
>> >>
>> org.apache.lucene.index.BaseCompositeReader.document(BaseCompositeReader.java:109)
>> >> at
>> >> org.apache.solr.search.SolrIndexSearcher.doc(SolrIndexSearcher.java:700)
>> >> at
>> >>
>> org.apache.solr.util.SolrPluginUtils.optimizePreFetchDocs(SolrPluginUtils.java:213)
>> >> at
>> >>
>> org.apache.solr.handler.component.QueryComponent.doPrefetch(QueryComponent.java:623)
>> >> at
>> >>
>> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:507)
>> >> at
>> >>
>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:218)
>> >> at
>> >>
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
>> >> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1967)
>> >> ...
>> >>
>> >>
>> >> Any ideas what could be missing?
>> >>
>> >> Thanks,
>> >> A. Adel
>> >>
>> >
>> >
>>