Re: DataImportHandler in Solr 4.0

[Rob]

Not a java pro, and the documentation hasn't been updated to include these
instructions (at least that I could find). What do I need to do to perform
the steps that Alexandre is talking about?

--
View this message in context: 
http://lucene.472066.n3.nabble.com/DataImportHandler-in-Solr-4-0-tp2563053p3667942.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Field Collapsing - disable cache

[rob]

my
solconfig can be seen at http://www.intelcompute.com/solrconfig.xml
[1]
On Tue 22/12/09 10:51 , r...@intelcompute.com wrote:Is
it possible to disable the field collapsing cache?  I'm trying to
perform some speed tests, and have managed to comment out the filter,
queryResult, and document caches successfully.

on 1.5
...
...
collapse

facet

tvComponent
...
-
Message sent via Atmail Open - http://atmail.org/ [2]"
target="_blank">http://atmail.org/
-
Message sent via Atmail Open - http://atmail.org/

Links:
--
[1] http://www.intelcompute.com/solrconfig.xml
[2] http://mail.intelcompute.com/parse.php?redirect=

Field Collapsing - disable cache

[rob]

Is
it possible to disable the field collapsing cache?  I'm trying to
perform some speed tests, and have managed to comment out the filter,
queryResult, and document caches successfully.

on 1.5
...
...
collapse

facet

tvComponent
...
-
Message sent via Atmail Open - http://atmail.org/

Re: Field Collapsing - disable cache

[rob]

That's what I assumed, but I'm getting the following error with it
commented out
MESSAGE null java.lang.NullPointerException at
org.apache.solr.search.fieldcollapse.AbstractDocumentCollapser.createDocumentCollapseResult(AbstractDocumentCollapser.java:276)
 at
org.apache.solr.search.fieldcollapse.AbstractDocumentCollapser.executeCollapse(AbstractDocumentCollapser.java:249)
 at
org.apache.solr.search.fieldcollapse.AbstractDocumentCollapser.collapse(AbstractDocumentCollapser.java:172)
 at
org.apache.solr.handler.component.CollapseComponent.doProcess(CollapseComponent.java:173)
 at
org.apache.solr.handler.component.CollapseComponent.process(CollapseComponent.java:127)
 at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
 at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
 at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:336)
 at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:239)
 at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
 at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
 at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210)
 at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
 at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
 at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
 at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
 at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
 at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:870)
 at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
 at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
 at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
 at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)
 at java.lang.Thread.run(Thread.java:636) 
On Tue 22/12/09 11:02 , Toby Cole  wrote:If you take out the
fieldCollapsing/fieldCollapseCache element in your  
config the fieldcollapse component will not use a cache.
 From http://wiki.apache.org/solr/FieldCollapsing%23line-63 [1]"
target="_blank">http://wiki.apache.org/solr/FieldCollapsing#line-63
"If the field collapse cache is not configured then the field  
collapse logic will not be cached."

Regards, Toby.

On 22 Dec 2009, at 10:56,  wrote:

> my
> solconfig can be seen at http://www.intelcompute.com/solrconfig.xml
[3]" target="_blank">http://www.intelcompute.com/solrconfig.xml
> [1]
> On Tue 22/12/09 10:51 ,  wrote:Is
> it possible to disable the field collapsing cache?  I'm trying to
> perform some speed tests, and have managed to comment out the
filter,
> queryResult, and document caches successfully.
>
> on 1.5
> ...
> ...
> collapse
>
> facet
>
> tvComponent
> ...
> -
> Message sent via Atmail Open - http://atmail.org/ [5]"
target="_blank">http://atmail.org/ [2]"
> target="_blank">http://atmail.org/ [6]"
target="_blank">http://atmail.org/
> -
> Message sent via Atmail Open - http://atmail.org/ [7]"
target="_blank">http://atmail.org/
>
> Links:
> --
> [1] http://www.intelcompute.com/solrconfig.xml [8]"
target="_blank">http://www.intelcompute.com/solrconfig.xml
> [2] http://mail.intelcompute.com/parse.php%3Fredirect%3D%26lt%3Ba
[9]"
target="_blank">http://mail.intelcompute.com/parse.php?redirect=http://blogs.semantico.com/discovery-blog/
-
Message sent via Atmail Open - http://atmail.org/

Links:
--
[1] http://mail.intelcompute.com/parse.php?redirect=http://mail.intelcompute.com/parse.php?redirect=http://mail.intelcompute.com/parse.php?redirect=http://mail.intelcompute.com/parse.php?redirect=http://mail.intelcompute.com/parse.php?redirect=http://mail.intelcompute.com/parse.php?redirect=http://mail.intelcompute.com/parse.php?redirect=http://mail.intelcompute.com/parse.php?redirect=

Re: Solr - Group search result

[rob]

see
http://blog.jteam.nl/2009/10/20/result-grouping-field-collapsing-with-solr/
[1] for a good little howto.
and also http://wiki.apache.org/solr/FieldCollapsing [2]
On Tue 22/12/09 11:03 , JKnight JKnight  wrote:Dear all,

I have document doc.xml
  3
  Love you love me
  2
  Love story
  3
  How deep is your love?
  4
  I love you
  5
  How deep is your love?
  6
  Love story
and in schema.xml, I add two line
After post doc.xml to solr server and run query
http://localhost:8983/solr/select/%3Fwt%3Djson%26amp%3Bindent%3Don%26amp%3Bq%3Dlove%2520you%26amp%3Bfl%3Dname%26amp%3Bfacet%3Dtrue%26amp%3Bfacet.field%3Dname_exact
[3]"
target="_blank">http://localhost:8983/solr/select/?wt=json&indent=on&q=love%20you&fl=name&facet=true&facet.field=name_exact

I see the response:
{
 "responseHeader":{
  "status":0,
  "QTime":4,
  "params":{
"facet":"true",
"fl":"name",
"indent":"on",
"q":"love you",
"facet.field":"name_exact",
"wt":"json"}},
 "response":{"numFound":5,"start":0,"docs":[
{
 "name":"I love you"},
{
 "name":"Love story"},
{
 "name":"Love story"},
{
 "name":"How deep is your love?"},
{
 "name":"How deep is your love?"}]
 },
 "facet_counts":{
  "facet_queries":{},
  "facet_fields":{
"name_exact":[
 "How deep is your love?",2,
 "Love story",2,
 "I love you",1,
 "Love you love me",0,
 "Quyen day ne",0,
 "day ne",0]},
  "facet_dates":{}}}

And I have a question. How can I group search result? With my data, I
want
to group by "name" field and see only 3 record
{
 "name":"I love you"},
{
 "name":"Love story"},
{
 "name":"How deep is your love?"}

Could you help me?

Thank a lot for support.

-- 
Best regards,
JKnight
-
Message sent via Atmail Open - http://atmail.org/

Links:
--
[1]
http://blog.jteam.nl/2009/10/20/result-grouping-field-collapsing-with-solr/
[2] http://wiki.apache.org/solr/FieldCollapsing
[3] http://mail.intelcompute.com/parse.php?redirect=

Re: Field Collapsing - disable cache

[rob]

I've tried both, the whole fieldCollapsing tag, and just the
fieldCollapseCache tag inside it.
both cause error.
I guess I can just set size, initialSize, and autowarmCount to 0 ??
On Tue 22/12/09 11:17 , Toby Cole  wrote:Which elements did you
comment out? It could be the case that you need  
to get rid of the entire fieldCollapsing element, not just the  
fieldCollapsingCache element.
(Disclaimer: I've not used field collapsing in anger before :)
Toby.

On 22 Dec 2009, at 11:09,  wrote:

> That's what I assumed, but I'm getting the following error with it
> commented out
> MESSAGE null java.lang.NullPointerException at
> org 
> .apache 
> .solr 
> .search 
> .fieldcollapse 
> .AbstractDocumentCollapser 
> .createDocumentCollapseResult(AbstractDocumentCollapser.java:276)
> at
> org 
> .apache 
> .solr 
> .search 
> .fieldcollapse 
> .AbstractDocumentCollapser 
> .executeCollapse(AbstractDocumentCollapser.java:249)
> at
> org 
> .apache 
> .solr 
> .search 
> .fieldcollapse 
> .AbstractDocumentCollapser.collapse(AbstractDocumentCollapser.java:

> 172)
> at
> org 
> .apache 
> .solr 
> .handler 
> .component.CollapseComponent.doProcess(CollapseComponent.java:173)
> at
> org 
> .apache 
> .solr 
>
.handler.component.CollapseComponent.process(CollapseComponent.java: 
> 127)
> at
> org 
> .apache 
> .solr 
> .handler 
> .component.SearchHandler.handleRequestBody(SearchHandler.java:195)
> at
> org 
> .apache 
> .solr 
>
.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) at
> org 
> .apache 
>
.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:336)
> at
> org 
> .apache 
>
.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:239)
> at
> org 
> .apache 
> .catalina 
> .core 
>
.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:

> 215)
> at
> org 
> .apache 
> .catalina 
>
.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
> at
> org 
> .apache 
>
.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:

> 210)
> at
> org 
> .apache 
>
.catalina.core.StandardContextValve.invoke(StandardContextValve.java:

> 172)
> at
> org 
> .apache 
> .catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> at
> org 
> .apache 
> .catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
> at
> org 
> .apache 
> .catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:

> 108)
> at
> org 
>
.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java: 
> 151)
> at
> org 
> .apache.coyote.http11.Http11Processor.process(Http11Processor.java:

> 870)
> at
> org.apache.coyote.http11.Http11BaseProtocol 
> $Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:

> 665)
> at
> org 
> .apache 
>
.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java: 
> 528)
> at
> org 
> .apache 
> .tomcat 
> .util 
> .net 
>
.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
> at
> org.apache.tomcat.util.threads.ThreadPool 
> $ControlRunnable.run(ThreadPool.java:685)
> at java.lang.Thread.run(Thread.java:636)
> On Tue 22/12/09 11:02 , Toby Cole  wrote:If you take out the
> fieldCollapsing/fieldCollapseCache element in your
> config the fieldcollapse component will not use a cache.
> From http://wiki.apache.org/solr/FieldCollapsing%2523line-63 [2]"
target="_blank">http://wiki.apache.org/solr/FieldCollapsing%23line-63
[1]"
>
target="_blank">http://wiki.apache.org/solr/FieldCollapsing%23line-63
[3]"
target="_blank">http://wiki.apache.org/solr/FieldCollapsing#line-63
> "If the field collapse cache is not configured then the field
> collapse logic will not be cached."
>
> Regards, Toby.
>
> On 22 Dec 2009, at 10:56,  wrote:
>
>> my
>> solconfig can be seen at
http://www.intelcompute.com/solrconfig.xml [4]"
target="_blank">http://www.intelcompute.com/solrconfig.xml
> [3]" target="_blank">http://www.intelcompute.com/solrconfig.xml
[5]" target="_blank">http://www.intelcompute.com/solrconfig.xml
>> [1]
>> On Tue 22/12/09 10:51 ,  wrote:Is
>> it possible to disable the field collapsing cache?  I'm trying to
>> perform some speed tests, and have managed to comment out the
> filter,
>> queryResult, and document caches successfully.
>>
>> on 1.5
>> ...
>> ...
>> collapse
>>
>> facet
>>
>> tvComponent
>> ...
>> -
>> Message sent via Atmail Open - http://atmail.org/ [6]"
target="_blank">http://atmail.org/ [5]"
> target="_blank">http://atmail.org/ [7]"
target="_blank">http://atmail.org/ [2]"
>> target="_blank">http://atmail.org/ [8]"
target="_blank">http://atmail.org/ [6]"
> target="_blank">http://atmail.org/ [9]"
target="_blank">http://atmail.org/
>> -
>> Message sent via Atmail Open - http://atmail.org/ [10]"
target="_blank">http://atmail.org/ [7]"
> target="_blank">http://atmail.org/ [11]"
target="_blank">http://atmail.org/
>>
>> Links:
>> --
>> [1] htt

Re: Field Collapsing - disable cache

[rob]

Yup, that's the one, with a copy of trunk from last week.



On Tue 22/12/09 12:28 , Martijn v Groningen  wrote:

> Hi Rob,
> What patch are you actually using from SOLR-236?
> Martijn
> 2009/12/22  :
> > I've tried both, the whole fieldCollapsing tag, and just the
> > fieldCollapseCache tag inside it.
> >both cause error.
> >I guess I can just set size, initialSize, and autowarmCount
> to 0 ??
> > On Tue 22/12/09 11:17 , Toby Cole  wrote:Which elements did you
> > comment out? It could be the case that you need
> > to get rid of the entire fieldCollapsing element, not just the
> > fieldCollapsingCache element.
> > (Disclaimer: I've not used field collapsing in anger before :)
> > Toby.
> >
> > On 22 Dec 2009, at 11:09,  wrote:
> >
> >> That's what I assumed, but I'm getting the following error with
> it
> >> commented out
> >> MESSAGE null java.lang.NullPointerException at
> >> org
> >> .apache
> >> .solr
> >> .search
> >> .fieldcollapse
> >> .AbstractDocumentCollapser
> >> .createDocumentCollapseResult(AbstractDocumentCollapser.java:276)
> >> at
> >> org
> >> .apache
> >> .solr
> >> .search
> >> .fieldcollapse
> >> .AbstractDocumentCollapser
> >> .executeCollapse(AbstractDocumentCollapser.java:249)
> >> at
> >> org
> >> .apache
> >> .solr
> >> .search
> >> .fieldcollapse
> >>
> .AbstractDocumentCollapser.collapse(AbstractDocumentCollapser.java:
> >
> >> 172)
> >> at
> >> org
> >> .apache
> >> .solr
> >> .handler
> >>
> .component.CollapseComponent.doProcess(CollapseComponent.java:173)
> >> at
> >> org
> >> .apache
> >> .solr
> >>
> >
> .handler.component.CollapseComponent.process(CollapseComponent.java:
> >> 127)
> >> at
> >> org
> >> .apache
> >> .solr
> >> .handler
> >>
> .component.SearchHandler.handleRequestBody(SearchHandler.java:195)
> >> at
> >> org
> >> .apache
> >> .solr
> >>
> >
> .handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
> >> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) at
> >> org
> >> .apache
> >>
> >
> .solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:336)
> >> at
> >> org
> >> .apache
> >>
> >
> .solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:239)
> >> at
> >> org
> >> .apache
> >> .catalina
> >> .core
> >>
> >
> .ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:
> >
> >> 215)
> >> at
> >> org
> >> .apache
> >> .catalina
> >>
> >
> .core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
> >> at
> >> org
> >> .apache
> >>
> >
> .catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:
> >
> >> 210)
> >> at
> >> org
> >> .apache
> >>
> >
> .catalina.core.StandardContextValve.invoke(StandardContextValve.java:
> >
> >> 172)
> >> at
> >> org
> >> .apache
> >>
> .catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> >> at
> >> org
> >> .apache
> >>
> .catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
> >> at
> >> org
> >> .apache
> >>
> .catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:
> >
> >> 108)
> >> at
> >> org
> >>
> >
> .apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:
> >> 151)
> >> at
> >> org
> >>
> .apache.coyote.http11.Http11Processor.process(Http11Processor.java:
> >
> >> 870)
> >> at
> >> org.apache.coyote.http11.Http11BaseProtocol
> >>
> $Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:
> >
> >> 665)
> >> at
> >> org
> >> .apache
> >>
> >
> .tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:
> >> 528)
> >> at
> >> org
> >> .apache
> >> .tomcat
> >> .util
> >> .net
> >>
> >
> .LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThr

Re: Field Collapsing - disable cache

[rob]

actually i got field-collapse-5.patch, what's the diff?




On Tue 22/12/09 12:28 , Martijn v Groningen  wrote:

> Hi Rob,
> What patch are you actually using from SOLR-236?
> Martijn
> 2009/12/22  :
> > I've tried both, the whole fieldCollapsing tag, and just the
> > fieldCollapseCache tag inside it.
> >both cause error.
> >I guess I can just set size, initialSize, and autowarmCount
> to 0 ??
> > On Tue 22/12/09 11:17 , Toby Cole  wrote:Which elements did you
> > comment out? It could be the case that you need
> > to get rid of the entire fieldCollapsing element, not just the
> > fieldCollapsingCache element.
> > (Disclaimer: I've not used field collapsing in anger before :)
> > Toby.
> >
> > On 22 Dec 2009, at 11:09,  wrote:
> >
> >> That's what I assumed, but I'm getting the following error with
> it
> >> commented out
> >> MESSAGE null java.lang.NullPointerException at
> >> org
> >> .apache
> >> .solr
> >> .search
> >> .fieldcollapse
> >> .AbstractDocumentCollapser
> >> .createDocumentCollapseResult(AbstractDocumentCollapser.java:276)
> >> at
> >> org
> >> .apache
> >> .solr
> >> .search
> >> .fieldcollapse
> >> .AbstractDocumentCollapser
> >> .executeCollapse(AbstractDocumentCollapser.java:249)
> >> at
> >> org
> >> .apache
> >> .solr
> >> .search
> >> .fieldcollapse
> >>
> .AbstractDocumentCollapser.collapse(AbstractDocumentCollapser.java:
> >
> >> 172)
> >> at
> >> org
> >> .apache
> >> .solr
> >> .handler
> >>
> .component.CollapseComponent.doProcess(CollapseComponent.java:173)
> >> at
> >> org
> >> .apache
> >> .solr
> >>
> >
> .handler.component.CollapseComponent.process(CollapseComponent.java:
> >> 127)
> >> at
> >> org
> >> .apache
> >> .solr
> >> .handler
> >>
> .component.SearchHandler.handleRequestBody(SearchHandler.java:195)
> >> at
> >> org
> >> .apache
> >> .solr
> >>
> >
> .handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
> >> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) at
> >> org
> >> .apache
> >>
> >
> .solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:336)
> >> at
> >> org
> >> .apache
> >>
> >
> .solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:239)
> >> at
> >> org
> >> .apache
> >> .catalina
> >> .core
> >>
> >
> .ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:
> >
> >> 215)
> >> at
> >> org
> >> .apache
> >> .catalina
> >>
> >
> .core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
> >> at
> >> org
> >> .apache
> >>
> >
> .catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:
> >
> >> 210)
> >> at
> >> org
> >> .apache
> >>
> >
> .catalina.core.StandardContextValve.invoke(StandardContextValve.java:
> >
> >> 172)
> >> at
> >> org
> >> .apache
> >>
> .catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> >> at
> >> org
> >> .apache
> >>
> .catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
> >> at
> >> org
> >> .apache
> >>
> .catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:
> >
> >> 108)
> >> at
> >> org
> >>
> >
> .apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:
> >> 151)
> >> at
> >> org
> >>
> .apache.coyote.http11.Http11Processor.process(Http11Processor.java:
> >
> >> 870)
> >> at
> >> org.apache.coyote.http11.Http11BaseProtocol
> >>
> $Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:
> >
> >> 665)
> >> at
> >> org
> >> .apache
> >>
> >
> .tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:
> >> 528)
> >> at
> >> org
> >> .apache
> >> .tomcat
> >> .util
> >> .net
> >>
> >
> .LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThr

Re: Field Collapsing - disable cache

[rob]

On Tue 22/12/09 12:28 , Martijn v Groningen  wrote:

> Hi Rob,
> What patch are you actually using from SOLR-236?
> Martijn
> 2009/12/22  :
> > I've tried both, the whole fieldCollapsing tag, and just the
> > fieldCollapseCache tag inside it.
> >both cause error.
> >I guess I can just set size, initialSize, and autowarmCount
> to 0 ??
> > On Tue 22/12/09 11:17 , Toby Cole  wrote:Which elements did you
> > comment out? It could be the case that you need
> > to get rid of the entire fieldCollapsing element, not just the
> > fieldCollapsingCache element.
> > (Disclaimer: I've not used field collapsing in anger before :)
> > Toby.
> >
> > On 22 Dec 2009, at 11:09,  wrote:
> >
> >> That's what I assumed, but I'm getting the following error with
> it
> >> commented out
> >> MESSAGE null java.lang.NullPointerException at
> >> org
> >> .apache
> >> .solr
> >> .search
> >> .fieldcollapse
> >> .AbstractDocumentCollapser
> >> .createDocumentCollapseResult(AbstractDocumentCollapser.java:276)
> >> at
> >> org
> >> .apache
> >> .solr
> >> .search
> >> .fieldcollapse
> >> .AbstractDocumentCollapser
> >> .executeCollapse(AbstractDocumentCollapser.java:249)
> >> at
> >> org
> >> .apache
> >> .solr
> >> .search
> >> .fieldcollapse
> >>
> .AbstractDocumentCollapser.collapse(AbstractDocumentCollapser.java:
> >
> >> 172)
> >> at
> >> org
> >> .apache
> >> .solr
> >> .handler
> >>
> .component.CollapseComponent.doProcess(CollapseComponent.java:173)
> >> at
> >> org
> >> .apache
> >> .solr
> >>
> >
> .handler.component.CollapseComponent.process(CollapseComponent.java:
> >> 127)
> >> at
> >> org
> >> .apache
> >> .solr
> >> .handler
> >>
> .component.SearchHandler.handleRequestBody(SearchHandler.java:195)
> >> at
> >> org
> >> .apache
> >> .solr
> >>
> >
> .handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
> >> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) at
> >> org
> >> .apache
> >>
> >
> .solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:336)
> >> at
> >> org
> >> .apache
> >>
> >
> .solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:239)
> >> at
> >> org
> >> .apache
> >> .catalina
> >> .core
> >>
> >
> .ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:
> >
> >> 215)
> >> at
> >> org
> >> .apache
> >> .catalina
> >>
> >
> .core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
> >> at
> >> org
> >> .apache
> >>
> >
> .catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:
> >
> >> 210)
> >> at
> >> org
> >> .apache
> >>
> >
> .catalina.core.StandardContextValve.invoke(StandardContextValve.java:
> >
> >> 172)
> >> at
> >> org
> >> .apache
> >>
> .catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> >> at
> >> org
> >> .apache
> >>
> .catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
> >> at
> >> org
> >> .apache
> >>
> .catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:
> >
> >> 108)
> >> at
> >> org
> >>
> >
> .apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:
> >> 151)
> >> at
> >> org
> >>
> .apache.coyote.http11.Http11Processor.process(Http11Processor.java:
> >
> >> 870)
> >> at
> >> org.apache.coyote.http11.Http11BaseProtocol
> >>
> $Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:
> >
> >> 665)
> >> at
> >> org
> >> .apache
> >>
> >
> .tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:
> >> 528)
> >> at
> >> org
> >> .apache
> >> .tomcat
> >> .util
> >> .net
> >>
> >
> .LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
> >> at
> >> org.apache.to

Re: Field Collapsing - disable cache

[rob]

I'm currently trying to patch aginst trunk, using SOLR-236.patch from 
18/12/2009 but getting the following error...


[...@intelcompute solr]$ patch -p0 < SOLR-236.patch
patching file src/test/test-files/solr/conf/solrconfig-fieldcollapse.xml
patching file src/test/test-files/solr/conf/schema-fieldcollapse.xml
patching file src/test/test-files/solr/conf/solrconfig.xml
patching file src/test/test-files/fieldcollapse/testResponse.xml
can't find file to patch at input line 787
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--
|
|Property changes on: src/test/test-files/fieldcollapse/testResponse.xml
|___
|Added: svn:keywords
|   + Date Author Id Revision HeadURL
|Added: svn:eol-style
|   + native
|
|Index: src/test/org/apache/solr/BaseDistributedSearchTestCase.java
|===
|--- src/test/org/apache/solr/BaseDistributedSearchTestCase.java
(revision 891214)
|+++ src/test/org/apache/solr/BaseDistributedSearchTestCase.java
(working copy)
--
File to patch:



any suggestions, or should i checkout the 1.4 branch instead?

can't remember what i did last time to get field-collapse-5.patch working 
successfully.






On Tue 22/12/09 22:43 , Lance Norskog  wrote:

> To avoid this possible bug, you could change the cache to only have a
> few entries.
> On Tue, Dec 22, 2009 at 6:34 AM, Martijn v Groningen
> wrote:
> > In the latest patch some changes where made on the configuration
> side,
> > but if you add the CollapseComponent to the conf no field collapse
> > cache should be enabled. If not let me know.
> >
> > Martijn
> >
> > 2009/12/22  :
> >>
> >>
> >>
> >>
> >>
> >> On Tue 22/12/09 12:28 , Martijn v Groningen  wrote:
> >>
> >>> Hi Rob,
> >>> What patch are you actually using from SOLR-236?
> >>> Martijn
> >>> 2009/12/22  :
> >>> > I've tried both, the whole fieldCollapsing tag, and just the
> >>> > fieldCollapseCache tag inside it.
> >>> >both cause error.
> >>> >I guess I can just set size, initialSize, and
> autowarmCount
> >>> to 0 ??
> >>> > On Tue 22/12/09 11:17 , Toby Cole  wrote:Which elements did
> you
> >>> > comment out? It could be the case that you need
> >>> > to get rid of the entire fieldCollapsing element, not just the
> >>> > fieldCollapsingCache element.
> >>> > (Disclaimer: I've not used field collapsing in anger before :)
> >>> > Toby.
> >>> >
> >>> > On 22 Dec 2009, at 11:09,  wrote:
> >>> >
> >>> >> That's what I assumed, but I'm getting the following error
> with
> >>> it
> >>> >> commented out
> >>> >> MESSAGE null java.lang.NullPointerException at
> >>> >> org
> >>> >> .apache
> >>> >> .solr
> >>> >> .search
> >>> >> .fieldcollapse
> >>> >> .AbstractDocumentCollapser
> >>> >>
> .createDocumentCollapseResult(AbstractDocumentCollapser.java:276)
> >>> >> at
> >>> >> org
> >>> >> .apache
> >>> >> .solr
> >>> >> .search
> >>> >> .fieldcollapse
> >>> >> .AbstractDocumentCollapser
> >>> >> .executeCollapse(AbstractDocumentCollapser.java:249)
> >>> >> at
> >>> >> org
> >>> >> .apache
> >>> >> .solr
> >>> >> .search
> >>> >> .fieldcollapse
> >>> >>
> >>>
> .AbstractDocumentCollapser.collapse(AbstractDocumentCollapser.java:
> >>> >
> >>> >> 172)
> >>> >> at
> >>> >> org
> >>> >> .apache
> >>> >> .solr
> >>> >> .handler
> >>> >>
> >>>
> .component.CollapseComponent.doProcess(CollapseComponent.java:173)
> >>> >> at
> >>> >> org
> >>> >> .apache
> >>> >> .solr
> >>> >>
> >>> >
> >>>
> .handler.component.CollapseComponent.process(CollapseComponent.java:
> >>> >> 127)
> >>> >> at
> >>> >> org
> >>> >> .apache
> >>> >&g

Re: Field Collapsing - disable cache

[rob]

Sorry, that was when trying to patch the 1.4 branch

attempting to patch the trunk gives...

patching file src/test/test-files/fieldcollapse/testResponse.xml
patching file src/test/org/apache/solr/BaseDistributedSearchTestCase.java
Hunk #2 FAILED at 502.
1 out of 2 hunks FAILED -- saving rejects to file 
src/test/org/apache/solr/BaseDistributedSearchTestCase.java.rej
patching file 
src/test/org/apache/solr/search/fieldcollapse/FieldCollapsingIntegrationTest.java


btw, when is trunk actually updated?





On Wed 23/12/09 11:53 , r...@intelcompute.com wrote:

> I'm currently trying to patch aginst trunk, using SOLR-236.patch
> from 18/12/2009 but getting the following error...
> [ solr]$ patch -p0 < SOLR-236.patch
> patching file
> src/test/test-files/solr/conf/solrconfig-fieldcollapse.xml
> patching file src/test/test-files/solr/conf/schema-fieldcollapse.xml
> patching file src/test/test-files/solr/conf/solrconfig.xml
> patching file src/test/test-files/fieldcollapse/testResponse.xml
> can't find file to patch at input line 787
> Perhaps you used the wrong -p or --strip option?
> The text leading up to this was:
> --
> |
> |Property changes on:
> src/test/test-files/fieldcollapse/testResponse.xml
> |___
> |Added: svn:keywords
> |   + Date Author Id Revision HeadURL
> |Added: svn:eol-style
> |   + native
> |
> |Index: src/test/org/apache/solr/BaseDistributedSearchTestCase.java
> |===
> |--- src/test/org/apache/solr/BaseDistributedSearchTestCase.java
> (revision 891214)
> |+++ src/test/org/apache/solr/BaseDistributedSearchTestCase.java
> (working copy)
> --
> File to patch:
> any suggestions, or should i checkout the 1.4 branch instead?
> can't remember what i did last time to get field-collapse-5.patch
> working successfully.
> On Tue 22/12/09 22:43 , Lance Norskog  wrote:
> > To avoid this possible bug, you could change the cache to only
> have a
> > few entries.
> > On Tue, Dec 22, 2009 at 6:34 AM, Martijn v Groningen
> > wrote:
> > > In the latest patch some changes where made on the configuration
> > side,
> > > but if you add the CollapseComponent to the conf no field
> collapse
> > > cache should be enabled. If not let me know.
> > >
> > > Martijn
> > >
> > > 2009/12/22  :
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> On Tue 22/12/09 12:28 , Martijn v Groningen  wrote:
> > >>
> > >>> Hi Rob,
> > >>> What patch are you actually using from SOLR-236?
> > >>> Martijn
> > >>> 2009/12/22  :
> > >>> > I've tried both, the whole fieldCollapsing tag, and just the
> > >>> > fieldCollapseCache tag inside it.
> > >>> >both cause error.
> > >>> >I guess I can just set size, initialSize, and
> > autowarmCount
> > >>> to 0 ??
> > >>> > On Tue 22/12/09 11:17 , Toby Cole  wrote:Which elements did
> > you
> > >>> > comment out? It could be the case that you need
> > >>> > to get rid of the entire fieldCollapsing element, not just
> the
> > >>> > fieldCollapsingCache element.
> > >>> > (Disclaimer: I've not used field collapsing in anger before
> :)
> > >>> > Toby.
> > >>> >
> > >>> > On 22 Dec 2009, at 11:09,  wrote:
> > >>> >
> > >>> >> That's what I assumed, but I'm getting the following error
> > with
> > >>> it
> > >>> >> commented out
> > >>> >> MESSAGE null java.lang.NullPointerException at
> > >>> >> org
> > >>> >> .apache
> > >>> >> .solr
> > >>> >> .search
> > >>> >> .fieldcollapse
> > >>> >> .AbstractDocumentCollapser
> > >>> >>
> > .createDocumentCollapseResult(AbstractDocumentCollapser.java:276)
> > >>> >> at
> > >>> >> org
> > >>> >> .apache
> > >>> >> .solr
> > >>> >> .search
> > >>> >> .fieldcollapse
> > >>> >> .AbstractDocumentCollapser
> > >>> >> .executeCollapse(AbstractDocumentCollapser.java:249)
> > >>> >> at
> > >>> >> org
&g

Re: Field Collapsing - disable cache

[rob]

Thanks, that latest update to the patch works fine now.



On Wed 23/12/09 13:13 , Martijn v Groningen  wrote:

> Latest SOLR-236.patch is for the trunk, if have updated the latest
> patch so it should patch now without conflicts. If I remember
> correctly the latest field-collapse-5.patch should work for 1.4, but
> it doesn't for the trunk.
> 2009/12/23  :
> >
> > Sorry, that was when trying to patch the 1.4 branch
> >
> > attempting to patch the trunk gives...
> >
> > patching file src/test/test-files/fieldcollapse/testResponse.xml
> > patching file
> src/test/org/apache/solr/BaseDistributedSearchTestCase.java
> > Hunk #2 FAILED at 502.
> > 1 out of 2 hunks FAILED -- saving rejects to file
> src/test/org/apache/solr/BaseDistributedSearchTestCase.java.rej
> > patching file
> src/test/org/apache/solr/search/fieldcollapse/FieldCollapsingIntegrationTes
> t.java>
> >
> > btw, when is trunk actually updated?
> >
> >
> >
> >
> >
> > On Wed 23/12/09 11:53 ,  wrote:
> >
> >> I'm currently trying to patch aginst trunk, using SOLR-236.patch
> >> from 18/12/2009 but getting the following error...
> >> [ solr]$ patch -p0 < SOLR-236.patch
> >> patching file
> >> src/test/test-files/solr/conf/solrconfig-fieldcollapse.xml
> >> patching file
> src/test/test-files/solr/conf/schema-fieldcollapse.xml
> >> patching file src/test/test-files/solr/conf/solrconfig.xml
> >> patching file src/test/test-files/fieldcollapse/testResponse.xml
> >> can't find file to patch at input line 787
> >> Perhaps you used the wrong -p or --strip option?
> >> The text leading up to this was:
> >> --
> >> |
> >> |Property changes on:
> >> src/test/test-files/fieldcollapse/testResponse.xml
> >>
> |___
> >> |Added: svn:keywords
> >> |   + Date Author Id Revision HeadURL
> >> |Added: svn:eol-style
> >> |   + native
> >> |
> >> |Index:
> src/test/org/apache/solr/BaseDistributedSearchTestCase.java
> >>
> |===
> >> |--- src/test/org/apache/solr/BaseDistributedSearchTestCase.java
> >> (revision 891214)
> >> |+++ src/test/org/apache/solr/BaseDistributedSearchTestCase.java
> >> (working copy)
> >> --
> >> File to patch:
> >> any suggestions, or should i checkout the 1.4 branch instead?
> >> can't remember what i did last time to get field-collapse-5.patch
> >> working successfully.
> >> On Tue 22/12/09 22:43 , Lance Norskog  wrote:
> >> > To avoid this possible bug, you could change the cache to only
> >> have a
> >> > few entries.
> >> > On Tue, Dec 22, 2009 at 6:34 AM, Martijn v Groningen
> >> > wrote:
> >> > > In the latest patch some changes where made on the
> configuration
> >> > side,
> >> > > but if you add the CollapseComponent to the conf no field
> >> collapse
> >> > > cache should be enabled. If not let me know.
> >> > >
> >> > > Martijn
> >> > >
> >> > > 2009/12/22  :
> >> > >>
> >> > >>
> >> > >>
> >> > >>
> >> > >>
> >> > >> On Tue 22/12/09 12:28 , Martijn v Groningen  wrote:
> >> > >>
> >> > >>> Hi Rob,
> >> > >>> What patch are you actually using from SOLR-236?
> >> > >>> Martijn
> >> > >>> 2009/12/22  :
> >> > >>> > I've tried both, the whole fieldCollapsing tag, and just
> the
> >> > >>> > fieldCollapseCache tag inside it.
> >> > >>> >both cause error.
> >> > >>> >I guess I can just set size, initialSize, and
> >> > autowarmCount
> >> > >>> to 0 ??
> >> > >>> > On Tue 22/12/09 11:17 , Toby Cole  wrote:Which elements
> did
> >> > you
> >> > >>> > comment out? It could be the case that you need
> >> > >>> > to get rid of the entire fieldCollapsing element, not
> just
> >> the
> >> > >>> > fieldCollapsingCache element.
> >> > >>> > (Disclaimer: I've not used field colla

Re: Field Collapsing - disable cache

[rob]

gt; |   + native
> > >> |
> > >> |Index:
> > src/test/org/apache/solr/BaseDistributedSearchTestCase.java
> > >>
> >
> |===
> > >> |---
> src/test/org/apache/solr/BaseDistributedSearchTestCase.java
> > >> (revision 891214)
> > >> |+++
> src/test/org/apache/solr/BaseDistributedSearchTestCase.java
> > >> (working copy)
> > >> --
> > >> File to patch:
> > >> any suggestions, or should i checkout the 1.4 branch instead?
> > >> can't remember what i did last time to get
> field-collapse-5.patch
> > >> working successfully.
> > >> On Tue 22/12/09 22:43 , Lance Norskog  wrote:
> > >> > To avoid this possible bug, you could change the cache to
> only
> > >> have a
> > >> > few entries.
> > >> > On Tue, Dec 22, 2009 at 6:34 AM, Martijn v Groningen
> > >> > wrote:
> > >> > > In the latest patch some changes where made on the
> > configuration
> > >> > side,
> > >> > > but if you add the CollapseComponent to the conf no field
> > >> collapse
> > >> > > cache should be enabled. If not let me know.
> > >> > >
> > >> > > Martijn
> > >> > >
> > >> > > 2009/12/22  :
> > >> > >>
> > >> > >>
> > >> > >>
> > >> > >>
> > >> > >>
> > >> > >> On Tue 22/12/09 12:28 , Martijn v Groningen  wrote:
> > >> > >>
> > >> > >>> Hi Rob,
> > >> > >>> What patch are you actually using from SOLR-236?
> > >> > >>> Martijn
> > >> > >>> 2009/12/22  :
> > >> > >>> > I've tried both, the whole fieldCollapsing tag, and
> just
> > the
> > >> > >>> > fieldCollapseCache tag inside it.
> > >> > >>> >both cause error.
> > >> > >>> >I guess I can just set size, initialSize, and
> > >> > autowarmCount
> > >> > >>> to 0 ??
> > >> > >>> > On Tue 22/12/09 11:17 , Toby Cole  wrote:Which elements
> > did
> > >> > you
> > >> > >>> > comment out? It could be the case that you need
> > >> > >>> > to get rid of the entire fieldCollapsing element, not
> > just
> > >> the
> > >> > >>> > fieldCollapsingCache element.
> > >> > >>> > (Disclaimer: I've not used field collapsing in anger
> > before
> > >> :)
> > >> > >>> > Toby.
> > >> > >>> >
> > >> > >>> > On 22 Dec 2009, at 11:09,  wrote:
> > >> > >>> >
> > >> > >>> >> That's what I assumed, but I'm getting the following
> > error
> > >> > with
> > >> > >>> it
> > >> > >>> >> commented out
> > >> > >>> >> MESSAGE null java.lang.NullPointerException at
> > >> > >>> >> org
> > >> > >>> >> .apache
> > >> > >>> >> .solr
> > >> > >>> >> .search
> > >> > >>> >> .fieldcollapse
> > >> > >>> >> .AbstractDocumentCollapser
> > >> > >>> >>
> > >> >
> > .createDocumentCollapseResult(AbstractDocumentCollapser.java:276)
> > >> > >>> >> at
> > >> > >>> >> org
> > >> > >>> >> .apache
> > >> > >>> >> .solr
> > >> > >>> >> .search
> > >> > >>> >> .fieldcollapse
> > >> > >>> >> .AbstractDocumentCollapser
> > >> > >>> >> .executeCollapse(AbstractDocumentCollapser.java:249)
> > >> > >>> >> at
> > >> > >>> >> org
> > >> > >>> >> .apache
> > >> > >>> >> .solr
> > >> > >>> >> .search
> > >> > >>> >> .fieldcollapse
> > >> > >&g

Re: Field Collapsing - disable cache

[rob]

> > >
> >> > > btw, when is trunk actually updated?
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > On Wed 23/12/09 11:53 ,  wrote:
> >> > >
> >> > >> I'm currently trying to patch aginst trunk, using
> >> SOLR-236.patch
> >> > >> from 18/12/2009 but getting the following error...
> >> > >> [ solr]$ patch -p0 < SOLR-236.patch
> >> > >> patching file
> >> > >> src/test/test-files/solr/conf/solrconfig-fieldcollapse.xml
> >> > >> patching file
> >> > src/test/test-files/solr/conf/schema-fieldcollapse.xml
> >> > >> patching file src/test/test-files/solr/conf/solrconfig.xml
> >> > >> patching file
> >> src/test/test-files/fieldcollapse/testResponse.xml
> >> > >> can't find file to patch at input line 787
> >> > >> Perhaps you used the wrong -p or --strip option?
> >> > >> The text leading up to this was:
> >> > >> --
> >> > >> |
> >> > >> |Property changes on:
> >> > >> src/test/test-files/fieldcollapse/testResponse.xml
> >> > >>
> >> >
> >>
> |___
> >> > >> |Added: svn:keywords
> >> > >> |   + Date Author Id Revision HeadURL
> >> > >> |Added: svn:eol-style
> >> > >> |   + native
> >> > >> |
> >> > >> |Index:
> >> > src/test/org/apache/solr/BaseDistributedSearchTestCase.java
> >> > >>
> >> >
> >>
> |=======
> >> > >> |---
> >> src/test/org/apache/solr/BaseDistributedSearchTestCase.java
> >> > >> (revision 891214)
> >> > >> |+++
> >> src/test/org/apache/solr/BaseDistributedSearchTestCase.java
> >> > >> (working copy)
> >> > >> --
> >> > >> File to patch:
> >> > >> any suggestions, or should i checkout the 1.4 branch
> instead?
> >> > >> can't remember what i did last time to get
> >> field-collapse-5.patch
> >> > >> working successfully.
> >> > >> On Tue 22/12/09 22:43 , Lance Norskog  wrote:
> >> > >> > To avoid this possible bug, you could change the cache to
> >> only
> >> > >> have a
> >> > >> > few entries.
> >> > >> > On Tue, Dec 22, 2009 at 6:34 AM, Martijn v Groningen
> >> > >> > wrote:
> >> > >> > > In the latest patch some changes where made on the
> >> > configuration
> >> > >> > side,
> >> > >> > > but if you add the CollapseComponent to the conf no
> field
> >> > >> collapse
> >> > >> > > cache should be enabled. If not let me know.
> >> > >> > >
> >> > >> > > Martijn
> >> > >> > >
> >> > >> > > 2009/12/22  :
> >> > >> > >>
> >> > >> > >>
> >> > >> > >>
> >> > >> > >>
> >> > >> > >>
> >> > >> > >> On Tue 22/12/09 12:28 , Martijn v Groningen  wrote:
> >> > >> > >>
> >> > >> > >>> Hi Rob,
> >> > >> > >>> What patch are you actually using from SOLR-236?
> >> > >> > >>> Martijn
> >> > >> > >>> 2009/12/22  :
> >> > >> > >>> > I've tried both, the whole fieldCollapsing tag, and
> >> just
> >> > the
> >> > >> > >>> > fieldCollapseCache tag inside it.
> >> > >> > >>> >both cause error.
> >> > >> > >>> >I guess I can just set size, initialSize, and
> >> > >> > autowarmCount
> >> > >> > >>> to 0 ??
> >> > >> > >>> > On Tue 22/12/09 11:17 , Toby Cole  wrote:Which
> elements
> >> > did
> >> > >> > you
> >> > >> >

Re: Delete, commit, optimize doesn't reduce index file size

[rob]

Hi Mark,

I can't help with reducing filesizes, but I'm curious...

What sort of documents were you storing, number of fields, average document 
size, many dynamic fields or mainly all static?

It would be good to hear about a real-world large-scale index in terms of 
response times, did the server have enough RAM to store it all in memory?

Cheers,
Rob




On Tue 29/12/09 18:23 , markwaddle  wrote:

> I have an index that used to have ~38M docs at 17.2GB. I deleted all
> but 13K
> docs using a delete by query, commit and then optimize. A "*:*"
> query now
> returns 13K docs. The problem is that the files on disk are still
> 17.1GB in
> size. I expected the optimize to shrink the files. Is there a way I
> can
> shrink them now that the index only has 13K docs?
> Mark
> -- 
> View this message in context:
> http://old.nabble.com/Delete%2C-commit%2C-optimize-doesn%27t-reduce-index-f
> ile-size-tp26958067p26958067.htmlSent from the Solr - User mailing list 
> archive at Nabble.com.
> 
> 
Message sent via Atmail Open - http://atmail.org/

Re: High Availability

[rob]

Have you looked into a basic floating IP setup?

Have the master also replicate to another hot-spare master.

Any downtime during an outage of the 'live' master would be minimal as the 
hot-spare takes up the floating IP.




On Mon 04/01/10 16:13 , Matthew Inger  wrote:

> I'm kind of stuck and looking for suggestions for high availability
> options.  I've figured out without much trouble how to get the
> master-slave replication working.  This eliminates any single points
> of failure in the application in terms of the application's searching
> capability.
> I would setup a master which would create the index and several
> slaves to act as the search servers, and put them behind a load
> balancer to distribute the requests.  This would ensure that if a
> slave node goes down, requests would continue to get serviced by the
> other nodes that are still up.
> The problem I have is that my particular application also has the
> capability to trigger index updates from the user interface.  This
> means that the master now becomes a single point of failure for the
> user interface.  
> The basic idea of the app is that there are multiple oracle
> instances contributing to a single document.  The volume and
> organization of the data (database links, normalization, etc...)
> prevents any sort of fast querying via SQL to do querying of the
> documents.  The solution is to build a lucene index (via solr), and
> use that for searching.  When updates are made in the UI, we will
> also send the updates directly to the solr server as well (we don't
> want to wait some arbitrary interval for a delta query to run).  
> So you can see the problem here is that if the master is down, the
> sending of the updates to the master solr server will fail, thus
> causing an application exception.
> I have tried configuring multiple solr servers which are both setup
> as masters and slaves to each other, but they keep clobber each
> other's index updates and rolling back each other's delta updates. 
> It seems that the replication doesn't take the generation # into
> account and check that the generation it's fetching is > the
> generation it already has before it applies it.
> I thought of maybe introducing a JMS queue to send my updates to and
> having the JMS message listener set to manually acknowledge the
> messages only after a succesfull application of the solrj api calls,
> but that seems kind of contrived, and is only a band-aid.
> Does anyone have any suggestions?
> 
> "Once you start down the dark path, forever will it
> dominate your destiny.  Consume you it will " - Yoda
> 
> 
Message sent via Atmail Open - http://atmail.org/

Re: High Availability

[rob]

Even when Master 1 is alive again, it shouldn't get the floating IP until 
Master 2 actually fails.

So you'd ideally want them replicating to eachother, but since one will only be 
updated/Live at a time, it shouldn't cause an issue with cobbling data (?).

Just a suggestion tho, not done it myself on Solr, only with DB servers.




On Mon 04/01/10 16:28 , Matthew Inger  wrote:

> So, when the masters switch back, does that mean, we have to force a
> full delta update, correct?
> 
> "Once you start down the dark path, forever will it
> dominate your destiny.  Consume you it will " - Yoda
> - Original Message 
> From: "" 
> To: 
> Sent: Mon, January 4, 2010 11:17:40 AM
> Subject: Re: High Availability
> Have you looked into a basic floating IP setup?
> Have the master also replicate to another hot-spare master.
> Any downtime during an outage of the 'live' master would be minimal
> as the hot-spare takes up the floating IP.
> On Mon 04/01/10 16:13 , Matthew Inger  wrote:
> > I'm kind of stuck and looking for suggestions for high
> availability
> > options.  I've figured out without much trouble how to get the
> > master-slave replication working.  This eliminates any single
> points
> > of failure in the application in terms of the application's
> searching
> > capability.
> > I would setup a master which would create the index and several
> > slaves to act as the search servers, and put them behind a load
> > balancer to distribute the requests.  This would ensure that if a
> > slave node goes down, requests would continue to get serviced by
> the
> > other nodes that are still up.
> > The problem I have is that my particular application also has the
> > capability to trigger index updates from the user interface.  This
> > means that the master now becomes a single point of failure for
> the
> > user interface.  
> > The basic idea of the app is that there are multiple oracle
> > instances contributing to a single document.  The volume and
> > organization of the data (database links, normalization, etc...)
> > prevents any sort of fast querying via SQL to do querying of the
> > documents.  The solution is to build a lucene index (via solr),
> and
> > use that for searching.  When updates are made in the UI, we will
> > also send the updates directly to the solr server as well (we
> don't
> > want to wait some arbitrary interval for a delta query to run).  
> > So you can see the problem here is that if the master is down, the
> > sending of the updates to the master solr server will fail, thus
> > causing an application exception.
> > I have tried configuring multiple solr servers which are both
> setup
> > as masters and slaves to each other, but they keep clobber each
> > other's index updates and rolling back each other's delta updates.
> 
> > It seems that the replication doesn't take the generation # into
> > account and check that the generation it's fetching is > the
> > generation it already has before it applies it.
> > I thought of maybe introducing a JMS queue to send my updates to
> and
> > having the JMS message listener set to manually acknowledge the
> > messages only after a succesfull application of the solrj api
> calls,
> > but that seems kind of contrived, and is only a band-aid.
> > Does anyone have any suggestions?
> > 
> > "Once you start down the dark path, forever will it
> > dominate your destiny.  Consume you it will " - Yoda
> > 
> > 
> Message sent via Atmail Open - http://atmail.org/
> 
> 
Message sent via Atmail Open - http://atmail.org/

Re: High Availability

[rob]

I'm also not sure what hooks you could put in upon the IP floating to the other 
machine, to start/stop replication - if it IS an issue anyway.




On Mon 04/01/10 16:28 , Matthew Inger  wrote:

> So, when the masters switch back, does that mean, we have to force a
> full delta update, correct?
> 
> "Once you start down the dark path, forever will it
> dominate your destiny.  Consume you it will " - Yoda
> - Original Message 
> From: "" 
> To: 
> Sent: Mon, January 4, 2010 11:17:40 AM
> Subject: Re: High Availability
> Have you looked into a basic floating IP setup?
> Have the master also replicate to another hot-spare master.
> Any downtime during an outage of the 'live' master would be minimal
> as the hot-spare takes up the floating IP.
> On Mon 04/01/10 16:13 , Matthew Inger  wrote:
> > I'm kind of stuck and looking for suggestions for high
> availability
> > options.  I've figured out without much trouble how to get the
> > master-slave replication working.  This eliminates any single
> points
> > of failure in the application in terms of the application's
> searching
> > capability.
> > I would setup a master which would create the index and several
> > slaves to act as the search servers, and put them behind a load
> > balancer to distribute the requests.  This would ensure that if a
> > slave node goes down, requests would continue to get serviced by
> the
> > other nodes that are still up.
> > The problem I have is that my particular application also has the
> > capability to trigger index updates from the user interface.  This
> > means that the master now becomes a single point of failure for
> the
> > user interface.  
> > The basic idea of the app is that there are multiple oracle
> > instances contributing to a single document.  The volume and
> > organization of the data (database links, normalization, etc...)
> > prevents any sort of fast querying via SQL to do querying of the
> > documents.  The solution is to build a lucene index (via solr),
> and
> > use that for searching.  When updates are made in the UI, we will
> > also send the updates directly to the solr server as well (we
> don't
> > want to wait some arbitrary interval for a delta query to run).  
> > So you can see the problem here is that if the master is down, the
> > sending of the updates to the master solr server will fail, thus
> > causing an application exception.
> > I have tried configuring multiple solr servers which are both
> setup
> > as masters and slaves to each other, but they keep clobber each
> > other's index updates and rolling back each other's delta updates.
> 
> > It seems that the replication doesn't take the generation # into
> > account and check that the generation it's fetching is > the
> > generation it already has before it applies it.
> > I thought of maybe introducing a JMS queue to send my updates to
> and
> > having the JMS message listener set to manually acknowledge the
> > messages only after a succesfull application of the solrj api
> calls,
> > but that seems kind of contrived, and is only a band-aid.
> > Does anyone have any suggestions?
> > 
> > "Once you start down the dark path, forever will it
> > dominate your destiny.  Consume you it will " - Yoda
> > 
> > 
> Message sent via Atmail Open - http://atmail.org/
> 
> 
Message sent via Atmail Open - http://atmail.org/

Re: London Search Social - this Tuesday, 12th January

[rob]

On Sun 10/01/10 20:24 , Richard Marr  wrote:

> Hi all,
> Apologies for the cross-post. If you're near London on Tuesday the
> 12th Jan (i.e. this Tuesday) please come along and geek with us over
> a
> beer or two. All experience levels welcome, don't be scared. Details
> on the Meetup page below... (please sign up on there if you're
> interested in subsequent events).
> http://www.meetup.com/london-search-social/
> Cheers,
> Richard Marr
> 
> 
Message sent via Atmail Open - http://atmail.org/

Multi-core support for indexing multiple servers

[Rob Veliz]

Trying to find specific information to support the following scenario:

- I have one site running on one server with marketing content, blog, etc.
I want to index.
- I have another site running on Magento on a different server with
ecommerce content (products).
- Both servers live in completely different environments.
- I would like to create one single search index between both sites and
make that index searchable from both sites.

I think I can/should use the multi-core approach and spin off a new server
to host Solr but can anyone verify this is the best/most appropriate
approach?  Are there any other details I need to consider?  Can anyone
provide a step by step for making this happen to validate my own technical
plan?  Any help appreciated...was initially thinking I needed SolrCloud but
that seems like overkill for my primary use case.

Re: Multi-core support for indexing multiple servers

[Rob Veliz]

Great feedback, thanks.  So the multi-core structure I have then is a
single Solr server set up, essentially hosted by one domain owner (but to
be used by both).  My question is how does that Solr server connect to the
2 Web applications to create the 1 master index (to be used when searching
on either Web app)?  It feels like I just reference the Solr server from
within the Web app search templates (e.g. PHP files).  That is logical in
terms of pulling the data into the Web apps, but it's still not clear to me
how the data from those 2 Web apps actually gets into the Solr server if
Solr server doesn't live on the same server as the Web app(s).  Any
thoughts?


On Wed, Nov 6, 2013 at 10:57 PM, Shawn Heisey  wrote:

> On 11/6/2013 11:38 PM, Rob Veliz wrote:
> > Trying to find specific information to support the following scenario:
> >
> > - I have one site running on one server with marketing content, blog,
> etc.
> > I want to index.
> > - I have another site running on Magento on a different server with
> > ecommerce content (products).
> > - Both servers live in completely different environments.
> > - I would like to create one single search index between both sites and
> > make that index searchable from both sites.
> >
> > I think I can/should use the multi-core approach and spin off a new
> server
> > to host Solr but can anyone verify this is the best/most appropriate
> > approach?  Are there any other details I need to consider?  Can anyone
> > provide a step by step for making this happen to validate my own
> technical
> > plan?  Any help appreciated...was initially thinking I needed SolrCloud
> but
> > that seems like overkill for my primary use case.
>
> SolrCloud makes for *easy* redundancy.  There is a three-server minimum
> if you want it to be fault-tolerant for both Solr and Zookeeper.  The
> third server would only run zookeeper and could be an extremely
> inexpensive machine.  The other two servers would run both Solr and
> Zookeeper.  Redundancy without cloud is possible, it's just not as
> automated, and can be done with two servers.
>
> It is highly recommended that redundant servers are not separated
> geographically.  This is especially important with SolrCloud, as
> Zookeeper redundancy requires that a majority of the servers be
> operational.  That can be extremely difficult to guarantee in a
> multi-datacenter model, if one assumes that an entire datacenter can
> disappear from the network.
>
> If you don't care about redundancy, then you'd just run a single server,
> and SolrCloud wouldn't provide much benefit.
>
> Multiple cores is a good way to go -- the two indexes would be logically
> separate, but you'd be able to use either one.  With SolrCloud, it would
> be multiple collections.
>
> Thanks,
> Shawn
>
>


-- 
*Rob Veliz*, Founder | *Mavenbridge* | rob...@mavenbridge.com | M: +1 (206)
909 - 3490

Follow us at: http://twitter.com/mavenbridge

Re: Multi-core support for indexing multiple servers

[Rob Veliz]

I've been reading about Solarium--definitely useful.  Could you elaborate
here:

If you are planning a single master index, that's not multicore.  Having
more than one document type in a single index is possible, they just
have to overlap on at least one field - whatever field is the uniqueKey
for the index.

What I'm trying to do is index marketing pages from one server AND index
product pages from a different ecommerce server and then combine those
results into a single index, so when I search for "foo" from either site, I
get the exact same results for "foo".  If that's not multi-core, what's the
right approach to accomplish this?


On Wed, Nov 6, 2013 at 11:29 PM, Shawn Heisey  wrote:

> On 11/7/2013 12:07 AM, Rob Veliz wrote:
> > Great feedback, thanks.  So the multi-core structure I have then is a
> > single Solr server set up, essentially hosted by one domain owner (but to
> > be used by both).  My question is how does that Solr server connect to
> the
> > 2 Web applications to create the 1 master index (to be used when
> searching
> > on either Web app)?  It feels like I just reference the Solr server from
> > within the Web app search templates (e.g. PHP files).  That is logical in
> > terms of pulling the data into the Web apps, but it's still not clear to
> me
> > how the data from those 2 Web apps actually gets into the Solr server if
> > Solr server doesn't live on the same server as the Web app(s).  Any
> > thoughts?
>
> Solr uses HTTP calls.  It is REST-like, though there has been some
> recent work to make parts of it actually use true REST, that paradigm
> might later be extended to the entire interface.
>
> There are a number of Solr API packages for PHP that give you an
> obect-oriented interface to Solr that won't require learning Solr's HTTP
> interface - you write PHP code to access Solr.  These are two of them
> that I have heard about.  I've not actually used these, as I have little
> personal experience with writing PHP:
>
> http://pecl.php.net/package/solr
> http://www.solarium-project.org/
>
> If you are planning a single master index, that's not multicore.  Having
> more than one document type in a single index is possible, they just
> have to overlap on at least one field - whatever field is the uniqueKey
> for the index.
>
> Thanks,
> Shawn
>
>


-- 
*Rob Veliz*, Founder | *Mavenbridge* | rob...@mavenbridge.com | M: +1 (206)
909 - 3490

Follow us at: http://twitter.com/mavenbridge

Querying for results

[Rob Veliz]

Hello,

I am running Solr from Magento and using DIH to import/index data from 1
other source (external).  I am trying to query for results...two questions:

1. The query I'm using runs against "fulltext_1_en" which is a specific
shard created by the Magento deployment in solrconfig.xml.  Should I be
using/querying from another field/store (e.g. not fulltext_1*) to get
results from both Magento and the other data source?  How would I add the
data from my DIH indexing to that specific shard so it was all in the same
place?

2. OR do I need to add another shard to correspond to the DIH data elements?

3. OR is there something else I'm missing in trying to query for data from
2 sources?

Thanks!

Re: Querying for results

[Rob Veliz]

Follow-up: Would anyone very familiar with DIH be willing to jump on a side
thread with me and my developer to help troubleshoot some issues we're
having?  Please little r me at: robert [at] mavenbridge.com.  Thanks!




On Wed, Dec 4, 2013 at 1:14 PM, Rob Veliz  wrote:

> Hello,
>
> I am running Solr from Magento and using DIH to import/index data from 1
> other source (external).  I am trying to query for results...two questions:
>
> 1. The query I'm using runs against "fulltext_1_en" which is a specific
> shard created by the Magento deployment in solrconfig.xml.  Should I be
> using/querying from another field/store (e.g. not fulltext_1*) to get
> results from both Magento and the other data source?  How would I add the
> data from my DIH indexing to that specific shard so it was all in the same
> place?
>
> 2. OR do I need to add another shard to correspond to the DIH data
> elements?
>
> 3. OR is there something else I'm missing in trying to query for data from
> 2 sources?
>
> Thanks!
>
>
>


-- 
*Rob Veliz*, Founder | *Mavenbridge* | rob...@mavenbridge.com | M: +1 (206)
909 - 3490

Follow us at: http://twitter.com/mavenbridge

Re: Solr warming when using master/slave replication

[Rob Casson]

it's always been my understanding that the caches are discarded, then
rebuilt/warmed:

 
http://wiki.apache.org/solr/SolrCaching#Caching_and_Distribution.2BAC8-Replication

hth,
rob

On Mon, Aug 29, 2011 at 5:30 PM, Mike Austin  wrote:
> How does warming work when a collection is being distributed to a slave.  I
> understand that a temp directory is created and it is eventually copied to
> the live folder, but what happens to the cache that was built in with the
> old index?  Does the cache get rebuilt, can we warm it before it becomes
> live, or can we keep the old cache?
>
> Thanks,
> Mike
>

Re: synonyms.txt: different results on admin and on site..

[Rob Casson]

you should probably post your schema.xml and some parts of your
synonyms.txt.  it could be differences between your index and query
analysis chains, synonym expansion errors, etc, but folks will likely
need more details to help you out.

cheers,
rob

On Wed, Sep 7, 2011 at 9:46 PM, deniz  wrote:
> could it be related with analysis issue about synonyms once again?
>
>
>
> -
> Zeki ama calismiyor... Calissa yapar...
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/synonyms-txt-different-results-on-admin-and-on-site-tp3318338p3318464.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: Two unrelated questions

[Rob Casson]

for #1, i don't use DIH, but is there any possibility of that column
having duplicate keys, with subsequent docs replacing existing ones?

and for #2, for some cases you could use a negative filterquery:

 
http://wiki.apache.org/solr/SimpleFacetParameters#Retrieve_docs_with_facets_missing

so instead of that "fq=-facetField:[* TO *]", something like
"fq=-car_make:Taurus".  picking "negatives" might even make the UI a
bit easier.

anyway, just some thoughts.  cheers,
rob

On Wed, Sep 21, 2011 at 5:17 PM, Olson, Ron  wrote:
> Thanks for the reply. As far as #1, my table that I'm indexing via DIH has a 
> PK field, generated by a sequence, so there are records with ID of 1, 2, 3, 
> etc. That same id is the one I use in my unique id field in the document 
> (ID).
>
> I've noticed that the table has, say, 10 rows. My index only has 8. I don't 
> know why that is, but I'd like to figure out which records are missing and 
> add them (and hopefully understand why they weren't added in the first 
> place). I was just wondering if there was some way to compare the two as part 
> of a sql query, but on reflection, it does seem like an absurd request, so I 
> apologize; I think what I'll have to do is write a solrj program that gets 
> every ID in the table, then does a search on that ID in the index, and add 
> the ones that are missing.
>
> Regarding the second item, yes, it's crazy but I'm not sure what to do; there 
> really are that many options and some searches will be extremely specific, 
> yet broad enough in terms for this to be a problem.
>
> -Original Message-
> From: Erick Erickson [mailto:erickerick...@gmail.com]
> Sent: Wednesday, September 21, 2011 3:55 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Two unrelated questions
>
> for <1> I don't quite get what you're driving at. Your DIH
> query assigns the uniqueKey, it's not like it's something
> auto-generated. Perhaps a concrete example would
> help.
>
> <2> There's a limit you can adjust that defaults to
> 1024 (maxBooleanClauses in solrconfig.xml). You can
>  bump this very high, but you're right, if anyone actually
> does something absurd it'll slow *that* query down. But
> just bumping this query higher won't change performance
> absent someone actually putting a ton of items in it...
>
> Best
> Erick
>
> On Mon, Sep 19, 2011 at 9:12 AM, Olson, Ron  wrote:
>> Hi all-
>>
>> I'm not sure if I should break this out into two separate questions to the 
>> list for searching purposes, or if one is more acceptable (don't want to 
>> flood).
>>
>> I have two (hopefully) straightforward questions:
>>
>> 1. Is it possible to expose the unique ID of a document to a DIH query? The 
>> reason I want to do this is because I use the unique ID of the row in the 
>> table as the unique ID of the Lucene document, but I've noticed that the 
>> counts of documents doesn't match the count in the table; I'd like to add 
>> these rows and was hoping to avoid writing a custom SolrJ app to do it.
>>
>> 2. Is there any limit to the number of conditions in a Boolean search? We're 
>> working on a new project where the user can choose either, for example, 
>> "Ford Vehicles", in which case I can simply search for "Ford", but if the 
>> user chooses specific makes and models, then I have to say something like 
>> "Crown Vic OR Focus OR Taurus OR F-150", etc., where they could 
>> theoretically choose every model of Ford ever made except one. This could 
>> lead to a *very* large query, and was worried both that it was even 
>> possible, but also the impact on performance.
>>
>>
>> Thanks, and I apologize if this really should be two separate messages.
>>
>> Ron
>>
>> DISCLAIMER: This electronic message, including any attachments, files or 
>> documents, is intended only for the addressee and may contain CONFIDENTIAL, 
>> PROPRIETARY or LEGALLY PRIVILEGED information.  If you are not the intended 
>> recipient, you are hereby notified that any use, disclosure, copying or 
>> distribution of this message or any of the information included in or with 
>> it is  unauthorized and strictly prohibited.  If you have received this 
>> message in error, please notify the sender immediately by reply e-mail and 
>> permanently delete and destroy this message and its attachments, along with 
>> any copies thereof. This message does not create any contractual obligation 
>> on behalf of the sender or Law Bulletin Publishing Company.
>>

RE: what is the recommended way to store locations?

[Rob Brown]

Just to throw this out there, we use UK postal data for locations, 9m
records and do location lookups/searches by grouping on the first part
of the post code.

Works really well, but apologies for going even more off-topic  :)



-Original Message-
From: Jaeger, Jay - DOT 
Reply-to: solr-user@lucene.apache.org
To: solr-user@lucene.apache.org 
Subject: RE: what is the recommended way to store locations?
Date: Thu, 6 Oct 2011 16:13:06 -0500

We do much the same (along with name, address, postal code, etc.).

However, we use AND when we search: the more data someone can provide, the 
fewer and more applicable their search results.

JRJ

-Original Message-
From: Jason Toy [mailto:jason...@gmail.com] 
Sent: Thursday, October 06, 2011 10:28 AM
To: solr-user@lucene.apache.org
Subject: what is the recommended way to store locations?

In our current system ,we have 3 fields for location,  city, state, and
country.People in our system search for one of those 3 strings.
So a user can search for "San Francisco" or "California".  In solr I store
those 3 fields as strings and when a search happens I search with an OR
statement across those 3 fields.

Is there a more efficient way to store this data storage wise and/or speed
wise?  We don't currently plan to use any spacial features like "3 miles
near SF".

Re: negative boosts for docs with common field value

[Rob Brown]

The setup for this question was to simplify the actual environment,
we're not actually demoting popular authors.

Perhaps index-time (negative) boosts are indeed the only way.


-- 

IntelCompute
Web Design and Online Marketing

http://www.intelcompute.com


-Original Message-
From: Chris Hostetter 
Reply-to: solr-user@lucene.apache.org
To: solr-user@lucene.apache.org
Subject: Re: negative boosts for docs with common field value
Date: Tue, 11 Oct 2011 15:37:03 -0700 (PDT)

: Some searches will obviously be saturated by docs from any given author if
: they've simply written more.
: 
: I'd like to give a negative boost to these matches, there-by making sure that
: 1 Author doesn't saturate the results just because they've written 500
: documents, compared to others who may have only written 2-3 documents.
: 
: The actual author value doesn't matter, I just want to bring down the score of
: docs by any common author to give more varied results.
: 
: What's the easiest approach for this, and is it even possible at query time?
: I could do this at index time but would prefer a Solr solution.

w/o a custom plugin, the only way i know of to do something like this 
would be to index a numeric "author_prolificness" field in each doc and 
use that as the basis of a function query.

but honestly: i *really* don't think you want to do this - not if you are 
dealing with real user queries (maybe if this is for some syntheticly 
generated "related documents" or "interesting documents" query)

Imagine a user is searching for a *very* specific title (ie: "Nightfall") 
by a very prolific author ("Isaac Asimov).  What your'e describing would 
penalize the desired match just because the author is prolific -- even if 
the user types in the exact title of a document, so that some much more 
esoteric document with the same title by an author who has written nothing 
else ("Stephen Leather") would likely score higher.


I mean: if someone types in "Romeo and Juliet" do you really want to score 
documents by "Shakespeare" lower then documents by "Stanley W. Wells" just 
because Wells has written fewer total books?



-Hoss

Multi CPU Cores

[Rob Brown]

Hi,

I'm running Solr on a machine with 16 CPU cores, yet watching "top" shows that 
java is only apparently using 1 and maxing it out.

Is there anything that can be done to take advantage of more CPU cores?

Solr 3.4 under Tomcat

[root@solr01 ~]# java -version
java version "1.6.0_20"
OpenJDK Runtime Environment (IcedTea6 1.9.8) (rhel-1.22.1.9.8.el5_6-x86_64)
OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)


top - 14:36:18 up 22 days, 21:54,  4 users,  load average: 1.89, 1.24, 1.08
Tasks: 317 total,   1 running, 315 sleeping,   0 stopped,   1 zombie
Cpu0  :  0.0%us,  0.0%sy,  0.0%ni, 99.6%id,  0.4%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu4  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu5  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu6  : 99.6%us,  0.4%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu7  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu8  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu9  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu10 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu11 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu12 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu13 :  0.7%us,  0.0%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu14 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu15 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  132088928k total, 23760584k used, 108328344k free,   318228k buffers
Swap: 25920868k total,0k used, 25920868k free, 18371128k cached

  PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+  COMMAND


 4466 tomcat20   0 31.2g 4.0g 171m S 101.0  3.2   2909:38 java  


 6495 root  15   0 42416 3892 1740 S  0.4  0.0   9:34.71 openvpn


11456 root  16   0 12892 1312  836 R  0.4  0.0   0:00.08 top


1 root  15   0 10368  632  536 S  0.0  0.0   0:04.69 init   
  

Multi CPU Cores

[Rob Brown]

Hi,

I'm running Solr on a machine with 16 CPU cores, yet watching "top"
shows that java is only apparently using 1 and maxing it out.

Is there anything that can be done to take advantage of more CPU cores?

Solr 3.4 under Tomcat

[root@solr01 ~]# java -version
java version "1.6.0_20"
OpenJDK Runtime Environment (IcedTea6 1.9.8)
(rhel-1.22.1.9.8.el5_6-x86_64)
OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)


top - 14:36:18 up 22 days, 21:54,  4 users,  load average: 1.89, 1.24,
1.08
Tasks: 317 total,   1 running, 315 sleeping,   0 stopped,   1 zombie
Cpu0  :  0.0%us,  0.0%sy,  0.0%ni, 99.6%id,  0.4%wa,  0.0%hi,  0.0%si,
0.0%st
Cpu1  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
0.0%st
Cpu2  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
0.0%st
Cpu3  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
0.0%st
Cpu4  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
0.0%st
Cpu5  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
0.0%st
Cpu6  : 99.6%us,  0.4%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,
0.0%st
Cpu7  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
0.0%st
Cpu8  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
0.0%st
Cpu9  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
0.0%st
Cpu10 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
0.0%st
Cpu11 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
0.0%st
Cpu12 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
0.0%st
Cpu13 :  0.7%us,  0.0%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,
0.0%st
Cpu14 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
0.0%st
Cpu15 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
0.0%st
Mem:  132088928k total, 23760584k used, 108328344k free,   318228k
buffers
Swap: 25920868k total,0k used, 25920868k free, 18371128k cached

  PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+
COMMAND 

   
 4466 tomcat20   0 31.2g 4.0g 171m S 101.0  3.2   2909:38
java

  
 6495 root  15   0 42416 3892 1740 S  0.4  0.0   9:34.71
openvpn 

   
11456 root  16   0 12892 1312  836 R  0.4  0.0   0:00.08
top 

   
1 root  15   0 10368  632  536 S  0.0  0.0   0:04.69
init 

Re: Multi CPU Cores

[Rob Brown]

Looks like I checked the load during a quiet period, ab -n 1 -c 1000
saw a decent 40% load on each core.

Still a little confused as to why 1 core stays at 100% constantly - even
during the quiet periods?


-- 

IntelCompute
Web Design and Online Marketing

http://www.intelcompute.com


-Original Message-
From: Johannes Goll 
Reply-to: solr-user@lucene.apache.org
To: solr-user@lucene.apache.org 
Subject: Re: Multi CPU Cores
Date: Sat, 15 Oct 2011 21:30:11 -0400

Did you try to submit multiple search requests in parallel? The apache ab tool 
is great tool to simulate simultaneous load using (-n and -c).
Johannes

On Oct 15, 2011, at 7:32 PM, Rob Brown  wrote:

> Hi,
> 
> I'm running Solr on a machine with 16 CPU cores, yet watching "top"
> shows that java is only apparently using 1 and maxing it out.
> 
> Is there anything that can be done to take advantage of more CPU cores?
> 
> Solr 3.4 under Tomcat
> 
> [root@solr01 ~]# java -version
> java version "1.6.0_20"
> OpenJDK Runtime Environment (IcedTea6 1.9.8)
> (rhel-1.22.1.9.8.el5_6-x86_64)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
> 
> 
> top - 14:36:18 up 22 days, 21:54,  4 users,  load average: 1.89, 1.24,
> 1.08
> Tasks: 317 total,   1 running, 315 sleeping,   0 stopped,   1 zombie
> Cpu0  :  0.0%us,  0.0%sy,  0.0%ni, 99.6%id,  0.4%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Cpu1  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Cpu2  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Cpu3  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Cpu4  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Cpu5  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Cpu6  : 99.6%us,  0.4%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Cpu7  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Cpu8  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Cpu9  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Cpu10 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Cpu11 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Cpu12 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Cpu13 :  0.7%us,  0.0%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Cpu14 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Cpu15 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Mem:  132088928k total, 23760584k used, 108328344k free,   318228k
> buffers
> Swap: 25920868k total,0k used, 25920868k free, 18371128k cached
> 
>  PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+
> COMMAND   
>   
>
> 4466 tomcat20   0 31.2g 4.0g 171m S 101.0  3.2   2909:38
> java  
>   
>   
> 6495 root  15   0 42416 3892 1740 S  0.4  0.0   9:34.71
> openvpn   
>   
>
> 11456 root  16   0 12892 1312  836 R  0.4  0.0   0:00.08
> top   
>   
>
>1 root  15   0 10368  632  536 S  0.0  0.0   0:04.69
> init 
> 
> 
> 

Re: Multi CPU Cores

[Rob Brown]

Thanks, Java is completely new to me (Perl/C background), so a little
guidance would be great with config options like this, while I get to
grips with Java...

Or pointing to a useful resource to start filling in these gaps too.



-Original Message-
From: Johannes Goll 
Reply-to: solr-user@lucene.apache.org
To: solr-user@lucene.apache.org 
Subject: Re: Multi CPU Cores
Date: Sun, 16 Oct 2011 08:18:47 -0400

Try using -useParallelGc as vm option. 

Johannes

On Oct 16, 2011, at 7:51 AM, Ken Krugler  wrote:

> 
> On Oct 16, 2011, at 1:44pm, Rob Brown wrote:
> 
>> Looks like I checked the load during a quiet period, ab -n 1 -c 1000
>> saw a decent 40% load on each core.
>> 
>> Still a little confused as to why 1 core stays at 100% constantly - even
>> during the quiet periods?
> 
> Could be background GC, depending on what you've got your JVM configured to 
> use.
> 
> Though that shouldn't stay at 100% for very long.
> 
> -- Ken
> 
> 
>> -Original Message-
>> From: Johannes Goll 
>> Reply-to: solr-user@lucene.apache.org
>> To: solr-user@lucene.apache.org 
>> Subject: Re: Multi CPU Cores
>> Date: Sat, 15 Oct 2011 21:30:11 -0400
>> 
>> Did you try to submit multiple search requests in parallel? The apache ab 
>> tool is great tool to simulate simultaneous load using (-n and -c).
>> Johannes
>> 
>> On Oct 15, 2011, at 7:32 PM, Rob Brown  wrote:
>> 
>>> Hi,
>>> 
>>> I'm running Solr on a machine with 16 CPU cores, yet watching "top"
>>> shows that java is only apparently using 1 and maxing it out.
>>> 
>>> Is there anything that can be done to take advantage of more CPU cores?
>>> 
>>> Solr 3.4 under Tomcat
>>> 
>>> [root@solr01 ~]# java -version
>>> java version "1.6.0_20"
>>> OpenJDK Runtime Environment (IcedTea6 1.9.8)
>>> (rhel-1.22.1.9.8.el5_6-x86_64)
>>> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
>>> 
>>> 
>>> top - 14:36:18 up 22 days, 21:54,  4 users,  load average: 1.89, 1.24,
>>> 1.08
>>> Tasks: 317 total,   1 running, 315 sleeping,   0 stopped,   1 zombie
>>> Cpu0  :  0.0%us,  0.0%sy,  0.0%ni, 99.6%id,  0.4%wa,  0.0%hi,  0.0%si,
>>> 0.0%st
>>> Cpu1  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
>>> 0.0%st
>>> Cpu2  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
>>> 0.0%st
>>> Cpu3  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
>>> 0.0%st
>>> Cpu4  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
>>> 0.0%st
>>> Cpu5  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
>>> 0.0%st
>>> Cpu6  : 99.6%us,  0.4%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,
>>> 0.0%st
>>> Cpu7  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
>>> 0.0%st
>>> Cpu8  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
>>> 0.0%st
>>> Cpu9  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
>>> 0.0%st
>>> Cpu10 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
>>> 0.0%st
>>> Cpu11 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
>>> 0.0%st
>>> Cpu12 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
>>> 0.0%st
>>> Cpu13 :  0.7%us,  0.0%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,
>>> 0.0%st
>>> Cpu14 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
>>> 0.0%st
>>> Cpu15 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
>>> 0.0%st
>>> Mem:  132088928k total, 23760584k used, 108328344k free,   318228k
>>> buffers
>>> Swap: 25920868k total,0k used, 25920868k free, 18371128k cached
>>> 
>>> PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEMTIME+
>>> COMMAND 
>>> 
>>>
>>> 4466 tomcat20   0 31.2g 4.0g 171m S 101.0  3.2   2909:38
>>> java
>>> 
>>>   
>>> 6495 root  15   0 42416 3892 1740 S  0.4  0.0   9:34.71
>>> openvpn 
>>> 
>>>
>>> 11456 root  16   0 12892 1312  836 R  0.4  0.0   0:00.08
>>> top 
>>> 
>>>
>>>  1 root  15   0 10368  632  536 S  0.0  0.0   0:04.69
>>> init 
>>> 
>>> 
>>> 
>> 
> 
> --
> Ken Krugler
> +1 530-210-6378
> http://bixolabs.com
> custom big data solutions & training
> Hadoop, Cascading, Mahout & Solr
> 
> 
> 

Replication issues with multiple Slaves

[Rob Nicholls]

Hey all,

We have a Master (1 server) and 2 Slaves (2 servers) setup and running 
replication across multiple cores.

However, the replication appears to behave sporadically and often fails when 
left to replicate automatically via poll. More often than not a replicate will 
fail after the slave has finished pulling down the segment files, because it 
cannot find a particular file, giving errors such as:

Oct 25, 2011 10:00:17 AM org.apache.solr.handler.SnapPuller copyAFile
SEVERE: Unable to move index file from: 
D:\web\solr\collection\data\index.2011102510\_3u.tii to: 
D:\web\solr\Collection\data\index\_3u.tiiTrying to do a copy

SEVERE: Unable to copy index file from: 
D:\web\solr\collection\data\index.2011102510\_3s.fdt to: 
D:\web\solr\Collection\data\index\_3s.fdt
java.io.FileNotFoundException: 
D:\web\solr\collection\data\index.2011102510\_3s.fdt (The system cannot 
find the file specified)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.(Unknown Source)
at org.apache.solr.common.util.FileUtils.copyFile(FileUtils.java:47)
at org.apache.solr.handler.SnapPuller.copyAFile(SnapPuller.java:585)
at org.apache.solr.handler.SnapPuller.copyIndexFiles(SnapPuller.java:621)
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:317)
at 
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:267)
at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(Unknown Source)
at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(Unknown
 Source)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(Unknown
 Source)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
 Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

For these files, I checked the master, and they did indeed exist.

Both slave machines are configured the same, with the same replication settings 
and a 60 minutes poll interval. Using Solr 3.1

Is it perhaps because both slave machines are trying to pull down files at the 
same time? (and the other has a lock on the file, thus it gets skipped maybe?)

Note: If I manually force replication on each slave, one at a time, the 
replication always seems to work fine.




Is there any obvious explanation or oddities I should be aware of that may 
cause this?

Thanks,
Rob




  

Replication issues with multiple Slaves

[Rob Nicholls]

Hey guys,

We have a Master (1 server) and 2 Slaves (2 servers) setup and running 
replication across multiple cores.

However, the replication appears to behave sporadically and often fails when 
left to replicate automatically via poll. More often than not a replicate will 
fail after the slave has finished pulling down the segment files, because it 
cannot find a particular file, giving errors such as:

Oct 25, 2011 10:00:17 AM org.apache.solr.handler.SnapPuller copyAFile
SEVERE: Unable to move index file from: 
D:\web\solr\collection\data\index.2011102510\_3u.tii to: 
D:\web\solr\Collection\data\index\_3u.tiiTrying to do a copy

SEVERE: Unable to copy index file from: 
D:\web\solr\collection\data\index.2011102510\_3s.fdt to: 
D:\web\solr\Collection\data\index\_3s.fdt
java.io.FileNotFoundException: 
D:\web\solr\collection\data\index.2011102510\_3s.fdt (The system cannot 
find the file specified)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.(Unknown Source)
at org.apache.solr.common.util.FileUtils.copyFile(FileUtils.java:47)
at org.apache.solr.handler.SnapPuller.copyAFile(SnapPuller.java:585)
at org.apache.solr.handler.SnapPuller.copyIndexFiles(SnapPuller.java:621)
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:317)
at 
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:267)
at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(Unknown Source)
at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(Unknown
 Source)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(Unknown
 Source)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
 Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

For these files, I checked the master, and they did indeed exist.

Both slave machines are configured the same, with the same replication settings 
and a 60 minutes poll interval.

Is it perhaps because both slave machines are trying to pull down files at the 
same time? (and the other has a lock on the file, thus it gets skipped maybe?)

Note: If I manually force replication on each slave, one at a time, the 
replication always seems to work fine.



Is there any obvious explanation or oddities I should be aware of that may 
cause this?

Thanks,
Rob



  

RE: Replication issues with multiple Slaves

[Rob Nicholls]

1) Hmm, maybe, didn't notice that... but I'd be very confused why it works
occasionally, and manual replication (through Solr Admin) always works ok in
that case?
2) This was my initial thought, it was happening on one core (multiple
commits while replication in progress), but I noticed it happening on
another core (the one mentioned below) which only had 1 commit and a single
generation (11 > 12) change to replicate. 


I too hoped and presumed that the Master is being Locked while replication
is copying files... can anyone confirm this? We are using the native Lock
type on a Windows/Tomcat server.

Is anyone aware of any reason why the replication skips files, or fails to
copy/find files other than because of presumably a commit or optimize
re-chunking the segments and deleting them on the Master?

-Original Message-
From: Jaeger, Jay - DOT [mailto:jay.jae...@dot.wi.gov] 
Sent: 25 October 2011 20:48
To: solr-user@lucene.apache.org
Subject: RE: Replication issues with multiple Slaves

I noted that in these messages the left hand side is lower case collection,
but the right hand side is upper case Collection.  Assuming you did a
cut/paste, could you have a core name mismatch between a master and a slave
somehow?

Otherwise (shudder):  could you be doing a commit while the replication is
in progress, causing files to shift about on it?  I'd have expected (perhaps
naively) solr to have some sort of lock to prevent such a problem.  But if
there is no internal lock, that would be a serious matter (and could happen
to us, too, down the road).

JRJ

-Original Message-
From: Rob Nicholls [mailto:robst...@hotmail.com] 
Sent: Tuesday, October 25, 2011 10:32 AM
To: solr-user@lucene.apache.org
Subject: Replication issues with multiple Slaves


Hey guys,

We have a Master (1 server) and 2 Slaves (2 servers) setup and running
replication across multiple cores.

However, the replication appears to behave sporadically and often fails when
left to replicate automatically via poll. More often than not a replicate
will fail after the slave has finished pulling down the segment files,
because it cannot find a particular file, giving errors such as:

Oct 25, 2011 10:00:17 AM org.apache.solr.handler.SnapPuller copyAFile
SEVERE: Unable to move index file from:
D:\web\solr\collection\data\index.2011102510\_3u.tii to:
D:\web\solr\Collection\data\index\_3u.tiiTrying to do a copy

SEVERE: Unable to copy index file from:
D:\web\solr\collection\data\index.2011102510\_3s.fdt to:
D:\web\solr\Collection\data\index\_3s.fdt
java.io.FileNotFoundException:
D:\web\solr\collection\data\index.2011102510\_3s.fdt (The system cannot
find the file specified)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.(Unknown Source)
at org.apache.solr.common.util.FileUtils.copyFile(FileUtils.java:47)
at org.apache.solr.handler.SnapPuller.copyAFile(SnapPuller.java:585)
at
org.apache.solr.handler.SnapPuller.copyIndexFiles(SnapPuller.java:621)
at
org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:317)
at
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:2
67)
at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(Unknown Source)
at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$
101(Unknown Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeri
odic(Unknown Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unk
nown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

For these files, I checked the master, and they did indeed exist.

Both slave machines are configured the same, with the same replication
settings and a 60 minutes poll interval.

Is it perhaps because both slave machines are trying to pull down files at
the same time? (and the other has a lock on the file, thus it gets skipped
maybe?)

Note: If I manually force replication on each slave, one at a time, the
replication always seems to work fine.



Is there any obvious explanation or oddities I should be aware of that may
cause this?

Thanks,
Rob



  

RE: Replication issues with multiple Slaves

[Rob Nicholls]

Thanks... Yes, and no. 

The main thing is, after the replicate failed below, I checked the master
and the files that it complains about below (and several others) did
exist... which is where I'm stumped about what is causing the issue (I have
added the maxCommits setting you mention below already).

I'll retest to confirm that there is only a single commit happening in this
scenario, and it's not some weird oddity to do with Windows just being an
arrse with file and path capitalization.


-Original Message-
From: Markus Jelsma [mailto:markus.jel...@openindex.io] 
Sent: 25 October 2011 20:51
To: solr-user@lucene.apache.org
Cc: Jaeger, Jay - DOT
Subject: Re: Replication issues with multiple Slaves

Are you frequently adding and deleting documents and committing those
mutations? Then it might try to download a file that doesnt exist anymore.
If that is the case try increasing :



> I noted that in these messages the left hand side is lower case 
> collection, but the right hand side is upper case Collection.  
> Assuming you did a cut/paste, could you have a core name mismatch 
> between a master and a slave somehow?
> 
> Otherwise (shudder):  could you be doing a commit while the 
> replication is in progress, causing files to shift about on it?  I'd 
> have expected (perhaps naively) solr to have some sort of lock to 
> prevent such a problem.  But if there is no internal lock, that would 
> be a serious matter (and could happen to us, too, down the road).
> 
> JRJ
> 
> -Original Message-
> From: Rob Nicholls [mailto:robst...@hotmail.com]
> Sent: Tuesday, October 25, 2011 10:32 AM
> To: solr-user@lucene.apache.org
> Subject: Replication issues with multiple Slaves
> 
> 
> Hey guys,
> 
> We have a Master (1 server) and 2 Slaves (2 servers) setup and running 
> replication across multiple cores.
> 
> However, the replication appears to behave sporadically and often 
> fails when left to replicate automatically via poll. More often than 
> not a replicate will fail after the slave has finished pulling down 
> the segment files, because it cannot find a particular file, giving errors
such as:
> 
> Oct 25, 2011 10:00:17 AM org.apache.solr.handler.SnapPuller copyAFile
> SEVERE: Unable to move index file from:
> D:\web\solr\collection\data\index.2011102510\_3u.tii to:
> D:\web\solr\Collection\data\index\_3u.tiiTrying to do a copy
> 
> SEVERE: Unable to copy index file from:
> D:\web\solr\collection\data\index.2011102510\_3s.fdt to:
> D:\web\solr\Collection\data\index\_3s.fdt java.io.FileNotFoundException:
> D:\web\solr\collection\data\index.2011102510\_3s.fdt (The system 
> cannot find the file specified) at java.io.FileInputStream.open(Native
> Method)
> at java.io.FileInputStream.(Unknown Source)
> at org.apache.solr.common.util.FileUtils.copyFile(FileUtils.java:47)
> at org.apache.solr.handler.SnapPuller.copyAFile(SnapPuller.java:585)
> at
> org.apache.solr.handler.SnapPuller.copyIndexFiles(SnapPuller.java:621) 
> at
> org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:31
> 7)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.
> java
> :267) at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159) 
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) 
> at java.util.concurrent.FutureTask$Sync.innerRunAndReset(Unknown 
> Source) at java.util.concurrent.FutureTask.runAndReset(Unknown Source) 
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.a
> cces
> s$101(Unknown Source) at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.r
> unPe
> riodic(Unknown Source) at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.r
> un(U
> nknown Source) at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) 
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 
> at java.lang.Thread.run(Unknown Source)
> 
> For these files, I checked the master, and they did indeed exist.
> 
> Both slave machines are configured the same, with the same replication 
> settings and a 60 minutes poll interval.
> 
> Is it perhaps because both slave machines are trying to pull down 
> files at the same time? (and the other has a lock on the file, thus it 
> gets skipped
> maybe?)
> 
> Note: If I manually force replication on each slave, one at a time, 
> the replication always seems to work fine.
> 
> 
> 
> Is there any obvious explanation or oddities I should be aware of that 
> may cause this?
> 
> Thanks,
> Rob

Re: Don't snowball depending on terms

[Rob Brown]

Yes, it looks like I'll have to do some pre-processing outside of Solr.

I don't mind giving users the option to query a differently indexed
field, ie, same content, but not stemmed, although this would apply to
all keywords they enter, so they couldn't allow stemming on one keyword,
but not another.

ie, "manage" and exec = manage and (exec or executive)

My current config is using the example "text" fieldtype, so stemmed at
index time.



-- 

IntelCompute
Web Design and Online Marketing

http://www.intelcompute.com


-Original Message-
From: François Schiettecatte 
Reply-to: solr-user@lucene.apache.org
To: solr-user@lucene.apache.org
Subject: Re: Don't snowball depending on terms
Date: Tue, 29 Nov 2011 13:53:44 -0500

It won't and depending on how your analyzer is set up the terms are most likely 
stemmed at index time.

You could create a separate field for unstemmed terms though, or use a less 
aggressive stemmer such as EnglishMinimalStemFilterFactory.

François

On Nov 29, 2011, at 12:33 PM, Robert Brown wrote:

> Is it possible to search a field but not be affected by the snowball filter?
> 
> ie, searching for "manage" is matching "management", but a user may want to 
> restrict results to only containing "manage".
> 
> I was hoping that simply quoting the term would do this, but it doesn't 
> appear to make any difference.
> 
> 
> 
> 
> --
> 
> IntelCompute
> Web Design & Local Online Marketing
> 
> http://www.intelcompute.com
> 

Re: Don't snowball depending on terms

[Rob Brown]

I guess I could do a bit of pre-processing, look for any words that are
quoted, and search in a diff field for those

How is a query like this formulated?

q=unstemmed:perl or java&q=stemmed:manager


-- 

IntelCompute
Web Design and Online Marketing

http://www.intelcompute.com


-Original Message-
From: Tomas Zerolo 
Reply-to: solr-user@lucene.apache.org
To: solr-user@lucene.apache.org
Subject: Re: Don't snowball depending on terms
Date: Wed, 30 Nov 2011 08:49:37 +0100

On Tue, Nov 29, 2011 at 01:53:44PM -0500, François Schiettecatte wrote:
> It won't and depending on how your analyzer is set up the terms are most 
> likely stemmed at index time.
> 
> You could create a separate field for unstemmed terms though, or use a less 
> aggressive stemmer such as EnglishMinimalStemFilterFactory.

This is surprising to me. Snowball introduces new homonyms, meaning it
will lump e.g. "management" and "manage" into one index entry. Thus,
I'd expect a handful of "false positives" (but usually not too many).

That's a "lossy index" (loosely speaking) and could be fixed by
post-filtering (instead of introducing another index, which in
most cases would seem a waste of resurces).

Is there no way in SOLR of filtering the results *after* the index
scan? I'd be disappointed!

Regards
-- tomás

Re: spatial search or null

[Rob Brown]

Recently had this myself...

http://wiki.apache.org/solr/SpatialSearch#How_to_combine_with_a_sub-query_to_expand_results



-- 

IntelCompute
Web Design and Online Marketing

http://www.intelcompute.com


-Original Message-
From: dan whelan 
Reply-to: solr-user@lucene.apache.org
To: solr-user@lucene.apache.org
Subject: spatial search or null
Date: Thu, 01 Dec 2011 10:22:38 -0800

Hi,

how would I go about constructing a solr 3.2 spatial query that would 
return documents that are in a specified radius OR documents that have 
no location information.

The query would have a similar result as this:   q=City:"San Diego" OR 
-City:['' TO *]


Thanks

Re: Search Results optimization

[Rob Casson]

you might find these helpful...similar question came up last week:

 http://ln-s.net/7WpX
 
http://robotlibrarian.billdueber.com/solr-forcing-items-with-all-query-terms-to-the-top-of-a-solr-search/

not exactly the same, as this case wanted to boost if *every* term
matched, but a similar tactic may workhope that helps,
rob

On Thu, Aug 26, 2010 at 4:02 PM, Hasnain  wrote:
>
> perhaps i wasnt clear in my earlier post
>
> if user searches for "swingline red stapler hammer hand rigid", then
> documents that matches max number of words written in query should come
> first
> e.g a document with name field as "swingline stapler" should come later than
> the document with "swingline red stapler"
>
> any suggestions how to achieve this functionality?
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/Search-Results-optimization-tp1129374p1359916.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: command line to check if Solr is up running

[Rob Casson]

you could look at the ping stuff:

 http://wiki.apache.org/solr/SolrConfigXml#The_Admin.2BAC8-GUI_Section

cheers,
rob

On Mon, Oct 25, 2010 at 3:56 PM, Xin Li  wrote:
> As we know we can use browser to check if Solr is running by going to 
> http://$hostName:$portNumber/$masterName/admin, say 
> http://localhost:8080/solr1/admin. My questions is: are there any ways to 
> check it using command line? I used "curl http://localhost:8080"; to check my 
> Tomcat, it worked fine. However, no response if I try "curl 
> http://localhost:8080/solr1/admin"; (even when my Solr is running). Does 
> anyone know any command line alternatives?
>
> Thanks,
> Xin
> This electronic mail message contains information that (a) is or
> may be CONFIDENTIAL, PROPRIETARY IN NATURE, OR OTHERWISE
> PROTECTED
> BY LAW FROM DISCLOSURE, and (b) is intended only for the use of
> the addressee(s) named herein.  If you are not an intended
> recipient, please contact the sender immediately and take the
> steps necessary to delete the message completely from your
> computer system.
>
> Not Intended as a Substitute for a Writing: Notwithstanding the
> Uniform Electronic Transaction Act or any other law of similar
> effect, absent an express statement to the contrary, this e-mail
> message, its contents, and any attachments hereto are not
> intended
> to represent an offer or acceptance to enter into a contract and
> are not otherwise intended to bind this sender,
> barnesandnoble.com
> llc, barnesandnoble.com inc. or any other person or entity.
>

Re: Which Tokeniser (and/or filter)

[Rob Brown]

Apologies if things were a little vague.

Given the example snippet to index (numbered to show searches needed to
match)...

1: i am a sales-manager in here
2: using asp.net and .net daily
3: working in design.
4: using something called sage 200. and i'm fluent
5: german sausages.
6: busy A&E dept earning £10,000 annually


... all with newlines in place.

able to match...

1. sales
1. "sales manager"
1. sales-manager
1. "sales-manager"
2. .net
2. asp.net
3. design
4. sage 200
6. A&E
6. £10,000

But do NOT match "fluent german" from 4 + 5 since there's a newline
between them when indexed, but not when searched.


Do the filters (wdf in this case) not create multiple tokens, so if
splitting on period in "asp.net" would create tokens for all of "asp",
"asp.", "asp.net", ".net", "net".


Cheers,
Rob

-- 

IntelCompute
Web Design and Online Marketing

http://www.intelcompute.com


-Original Message-
From: Chris Hostetter 
Reply-to: solr-user@lucene.apache.org
To: solr-user@lucene.apache.org
Subject: Re: Which Tokeniser (and/or filter)
Date: Tue, 7 Feb 2012 15:02:36 -0800 (PST)

: This all seems a bit too much work for such a real-world scenario?

You haven't really told us what your scenerio is.

You said you want to split tokens on whitespace, full-stop (aka: 
period) and comma only, but then in response to some suggestions you added 
comments other things that you never mentioned previously...

1) evidently you don't want the "." in foo.net to cause a split in tokens?
2) evidently you not only want token splits on newlines, but also 
positition gaps to prevent phrases matching across newlines.

...these are kind of important details that affect suggestions people 
might give you.

can you please provide some concrete examples of hte types of data you 
have, the types of queries you want them to match, and the types of 
queries you *don't* want to match?


-Hoss

Re: Copying the index from one solr instance to another

[Rob Casson]

just making sure that you're aware of the built-in replication:

 http://wiki.apache.org/solr/SolrReplication

can pull the indexes, along with config files.

cheers,
rob

2010/12/15 Robert Gründler :
> Hi again,
>
> let's say you have 2 solr Instances, which have both exactly the same 
> configuration (schema, solrconfig, etc).
>
> Could it cause any troubles if we import an index from a SQL database on solr 
> instance A, and copy the whole
> index to the datadir of solr instance B (both solr instances run on different 
> servers) ?.
>
> As far as i can tell, this should work and solr instance B should have the 
> exact same index as solr instance A after the copy-process.
>
> Do we miss something, or is this workflow safe to go with?
>
> -robert

Re: Facet Query Question

[Rob Casson]

if i'm understanding your question, it sounds like
localparams/tagging/exclusion might be what you want:

 
http://wiki.apache.org/solr/SimpleFacetParameters#Multi-Select_Faceting_and_LocalParams

hth,
rob

On Sun, Feb 27, 2011 at 6:41 PM, Tim Christensen
 wrote:
> Hi,
>
> I am trying to do the following:
>
> Where a query might return:
>
> Facet 1
> A
> B
> C
>
> Facet 2
> X
> Y
> Z
>
> User selects Facet 1 option A. Normally this paradigm would contract the 
> results as in a refining paradigm. That would be fine and the obvious UI 
> change. But by doing so, Facet 2 option X is no longer available -- again 
> because of the refining. Let's say I still wanted Facet 2 option X to be 
> available to instead of refining, expands the results.
>
> Normally, my query might look like:
>
> q=query&fq=Facet 1:A (for the first part of my question. What I have done is 
> return two sets of facet results, one for the main query and one for the 
> refined query. That way I can still offer option X. What I don't know how to 
> do is query beyond that. I have tried some ORs and ANDs  in my unit tests, 
> but don't think this is the right way.
>
> My question is whether there is a way in a single query to bring back all the 
> original facets regardless of any facet refining. If not, give that I return 
> two sets of facets - a refined set and the 'original' querys' facet set, how 
> would I fashion this query?
>
> My apologies if this is rookie, I have a few years of Solr under my belt, but 
> can't think outside the refining and then expanding the result set with a 
> facet query that was available in the original query results.
>
> Thank you,
>
> Tim Christensen
>
>
>
>
>
>
>

Re: Result order when score is the same

[Rob Casson]

you could just explicitly send multiple sorts...from the tutorial:

 &sort=inStock asc, price desc

cheers.

On Wed, Apr 13, 2011 at 2:59 PM, kenf_nc  wrote:
> Is sort order when 'score' is the same a Lucene thing? Should I ask on the
> Lucene forum?
>
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/Result-order-when-score-is-the-same-tp2816127p2817330.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: querying in Java

[Rob Casson]

copyField should do the trick:

 http://wiki.apache.org/solr/SchemaXml#Copy_Fields

"A common requirement is to copy or merge all input fields into a
single solr field. This can be done as follows:-

 "

hth,
rob

On Fri, Apr 29, 2011 at 2:06 PM, Saler, Jeff  wrote:
> Thanks for the reply.  What I want is for the query to search all fields
> for the specified value.
>
> -Original Message-
> From: Anuj Kumar [mailto:anujs...@gmail.com]
> Sent: Friday, April 29, 2011 1:51 PM
> To: solr-user@lucene.apache.org
> Subject: Re: querying in Java
>
> Hi Jeff,
>
> In that case, it will query w.r.t default field. What is your default
> search
> field in the schema?
>
> Regards,
> Anuj
>
> On Fri, Apr 29, 2011 at 11:10 PM, Saler, Jeff  wrote:
>
>> Is there any way to query for data that is in any field, i.e. not
> using
>> a specific field name?
>>
>>
>>
>> For example, when I use the following statements:
>>
>>
>>
>>                SolrQuery  query = new SolrQuery();
>>
>>                Query.setQuery("ANALYST:John Schummers");
>>
>>      QueryResponse  rsp = server.query(query);
>>
>>
>>
>>
>>
>> I get the documents I'm looking for.
>>
>>
>>
>> But I would like to get the same set of documents without using the
>> specific ANALYST field name.
>>
>> I have tried using just "Schummers" as the query, but no documents are
>> returned.
>>
>> The ANALYST field is an indexed field.
>>
>>
>>
>>
>>
>>
>> This message and any enclosures are intended only for the addressee.
>>  Please
>> notify the sender by email if you are not the intended recipient.  If
> you
>> are
>> not the intended recipient, you may not use, copy, disclose, or
> distribute
>> this
>> message or its contents or enclosures to any other person and any such
>> actions
>> may be unlawful.  Ball reserves the right to monitor and review all
>> messages
>> and enclosures sent to or from this email address.
>
>
>
> This message and any enclosures are intended only for the addressee.  Please
> notify the sender by email if you are not the intended recipient.  If you are
> not the intended recipient, you may not use, copy, disclose, or distribute 
> this
> message or its contents or enclosures to any other person and any such actions
> may be unlawful.  Ball reserves the right to monitor and review all messages
> and enclosures sent to or from this email address.
>

Re: *:* query with dismax

[Rob Casson]

it does seem a little weird, but q.alt will get what you want:

  http://wiki.apache.org/solr/DisMaxQParserPlugin#q.alt

hth,
rc

On Fri, May 6, 2011 at 7:41 PM, Jason Chaffee  wrote:
> Can you shed some light on what you did to configure it to handle *:*?
> I have the same issue that I need it to work for faceting, but I do need
> the dismax abilities as well.
>
> -Original Message-
> From: Mark Mandel [mailto:mark.man...@gmail.com]
> Sent: Friday, May 06, 2011 4:30 PM
> To: solr-user@lucene.apache.org
> Subject: Re: *:* query with dismax
>
> This is exactly what should be happening, as the dismax parser doesn't
> understand regular query syntax (and for good reason too). This tripped
> me
> up as well when I first started using dismax.
>
> Solution for me was to comfigure the handler to use *:* when the query
> is
> empty, so that you can still get back a full result set if you need it,
> say
> for faceting.
>
> HTH
>
> Mark
> On May 7, 2011 9:22 AM, "Jason Chaffee"  wrote:
>> I am using dismax and trying to use q=*:* to return all indexed
>> documents. However, it is always returning 0 found.
>>
>>
>>
>> If I used the default select (not dismax) handler and try q=*:* then
> it
>> returns all documents.
>>
>>
>>
>> There is nothing in the logs to indicate why this happening.
>>
>>
>>
>> Does anyone have any clues?
>>
>>
>>
>> Thanks,
>>
>>
>>
>> Jason
>>
>

Re: aliasing?

[Rob Casson]

a lot of this stuff is covered in the tutorial, and expanded in the
wiki. still the best places to start in figuring out the fundamentals:

 http://lucene.apache.org/solr/tutorial.html
 
http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.SynonymFilterFactory

hth,
rc

On Mon, May 9, 2011 at 9:09 PM, deniz  wrote:
> well... if i knew what to do about aliasing, i wouldnt post my question here
> Grijesh :) My idea is this: for some search queries, I need to provide some
> synonyms...
> But it is just an idea...
>
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/aliasing-tp2917733p2921305.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: indexing numbers

[Rob Casson]

the default schema.xml provided in the Solr distribution is
well-documented, and a good place to get started (including numeric
fieldTypes):

 http://wiki.apache.org/solr/SchemaXml

Lucid Imagination also provides a nice reference guide:

 
http://www.lucidimagination.com/Downloads/LucidWorks-for-Solr/Reference-Guide

hope that helps,
rob

On Wed, May 25, 2011 at 6:20 PM, antoniosi  wrote:
> Hi,
>
> How does solr index a numeric value? Does it index it as a string or does it
> keep it as a numeric value?
>
> Thanks.
>
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/indexing-numbers-tp2986424p2986424.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: [Handling] empty fields

[Rob Casson]

i also thought of the lengthFilter stuff, provided it's a
text/KeywordTokenizer field:

 
http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.LengthFilterFactory

cheers,
rob

On Wed, Jun 15, 2011 at 12:00 PM, Erick Erickson
 wrote:
> Have you tried setting 'facet.missing="false" '?
>
> See:
> http://wiki.apache.org/solr/SimpleFacetParameters#facet.missing
>
> Best
> Erick
>
> On Wed, Jun 15, 2011 at 11:52 AM, Adam Estrada
>  wrote:
>> All,
>>
>> I have a field "foo" with several thousand blank or non-existing records in
>> it. This is also my faceting field. My question is, how can I deal with this
>> field so that I don't get a blank facet at query time?
>>
>> 5000
>> vs.
>> 1000
>>
>> Adam
>>
>

Re: Can I invert the inverted index?

[Rob Casson]

sounds like the Luke request handler will get what you're after:

 http://wiki.apache.org/solr/LukeRequestHandler
 http://wiki.apache.org/solr/LukeRequestHandler#id

cheers,
rob

On Tue, Jul 5, 2011 at 3:59 PM, Gabriele Kahlout
 wrote:
> Hello,
>
> With an inverted index the term is the key, and the documents are the
> values. Is it still however possible that given a document id I get the
> terms indexed for that document?
>
> --
> Regards,
> K. Gabriele
>
> --- unchanged since 20/9/10 ---
> P.S. If the subject contains "[LON]" or the addressee acknowledges the
> receipt within 48 hours then I don't resend the email.
> subject(this) ∈ L(LON*) ∨ ∃x. (x ∈ MyInbox ∧ Acknowledges(x, this) ∧ time(x)
> < Now + 48h) ⇒ ¬resend(I, this).
>
> If an email is sent by a sender that is not a trusted contact or the email
> does not contain a valid code then the email is not received. A valid code
> starts with a hyphen and ends with "X".
> ∀x. x ∈ MyInbox ⇒ from(x) ∈ MySafeSenderList ∨ (∃y. y ∈ subject(x) ∧ y ∈
> L(-[a-z]+[0-9]X)).
>

Re: Searching for strings

[Rob Casson]

chip,

gonna need more information about your particular analysis chain,
content, and example searches to give a better answer, but phrase
queries (using quotes) are supported in both the standard and dismax
query parsers

that being said, lots of things may not match a person's idea of an
exact string...stopwords, synonyms, slop, etc.

cheers,
rob

On Mon, Jul 18, 2011 at 5:25 PM, Chip Calhoun  wrote:
> Is there a way to search for a specific string using Solr, either by putting 
> it in quotes or by some other means?  I haven't been able to do this, but I 
> may be missing something.
>
> Thanks,
> Chip
>

Re: Exact matching on names?

[Rob Casson]

"exact" can mean a lot of things (do diacritics count?, etc), but in
this case, it sounds like you just need to turn off the stemmer you
have on this fieldtype (or create a new one that doesn't include the
stemmer).

hth,
rob

On Tue, Aug 16, 2011 at 11:20 AM, Olson, Ron  wrote:
> Hi all-
>
> I'm missing something fundamental yet I've been unable to find the definitive 
> answer for exact name matching. I'm indexing names using the standard "text" 
> field type and my search is for the name "clarke". My results include 
> "clark", which is incorrect, it needs to match clarke exactly (case 
> insensitive).
>
> I tried textType but that doesn't work because I believe it needs to be 
> *really* exact, whereas I'm looking for things like "clark oil", "bob, frank, 
> and clark", etc.
>
> Thanks for any help,
>
> Ron
>
> DISCLAIMER: This electronic message, including any attachments, files or 
> documents, is intended only for the addressee and may contain CONFIDENTIAL, 
> PROPRIETARY or LEGALLY PRIVILEGED information.  If you are not the intended 
> recipient, you are hereby notified that any use, disclosure, copying or 
> distribution of this message or any of the information included in or with it 
> is  unauthorized and strictly prohibited.  If you have received this message 
> in error, please notify the sender immediately by reply e-mail and 
> permanently delete and destroy this message and its attachments, along with 
> any copies thereof. This message does not create any contractual obligation 
> on behalf of the sender or Law Bulletin Publishing Company.
> Thank you.
>

Re: solr 1872

[Rob Casson]

looks like it might actually be a zip file.  try renaming/unzipping it.

cheers,
rob

On Mon, Jul 30, 2012 at 2:50 PM, Sujatha Arun  wrote:
> I am uable to use the rar file from the site
> https://issues.apache.org/jira/browse/SOLR-1872.
>
> When I try to open it,I get the message 'SolrACLSecurity.rar is not RAR
> archive.
>
> Is the file there at this link?
>
> Regards
> Sujatha

Problem with loading dictionary for Hunspell

[Rob Koeling]

I'm trying to employ the HunspellStemFilterFactory, but have trouble
loading a dictionary.

I  downloaded the .dic and .aff file for en_GB, en_US and nl_NL from
the OpenOffice site, but they all give me the same error message.

When I use them AS IS, I get the error message:

Oct 26, 2012 2:39:37 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.RuntimeException: Unable to load hunspell data!
[dictionary=en_GB.dic,affix=en_GB.aff]
at 
org.apache.solr.analysis.HunspellStemFilterFactory.inform(HunspellStemFilterFactory.java:87)
at 
org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:551)

Caused by: java.text.ParseException: The first non-comment line in the
affix file must be a 'SET charset', was: 'FLAG num'
at 
org.apache.lucene.analysis.hunspell.HunspellDictionary.getDictionaryEncoding(HunspellDictionary.java:280)
at 
org.apache.lucene.analysis.hunspell.HunspellDictionary.(HunspellDictionary.java:112)
at 
org.apache.solr.analysis.HunspellStemFilterFactory.inform(HunspellStemFilterFactory.java:85)
... 32 more


When I add the following first line to both the .dic and the .aff file:
 SET UTF-8

The error message changes into:

Oct 26, 2012 10:16:42 AM org.apache.solr.common.SolrException log
SEVERE: java.lang.RuntimeException: Unable to load hunspell data!
[dictionary=en_GB.dic,affix=en_GB.aff]
at org.apache.solr.analysis.HunspellStemFilterFactory.infoOSX
10.7.5rm(HunspellStemFilterFactory.java:87)
at 
org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:551)

Caused by: java.nio.charset.IllegalCharsetNameException: 'UTF-8'
at java.nio.charset.Charset.checkName(Charset.java:284)
at java.nio.charset.Charset.lookup2(Charset.java:458)
at java.nio.charset.Charset.lookup(Charset.java:437)
at java.nio.charset.Charset.forName(Charset.java:502)
at 
org.apache.lucene.analysis.hunspell.HunspellDictionary.getJavaEncoding(HunspellDictionary.java:293)
at 
org.apache.lucene.analysis.hunspell.HunspellDictionary.(HunspellDictionary.java:113)
at 
org.apache.solr.analysis.HunspellStemFilterFactory.inform(HunspellStemFilterFactory.java:85)
... 32 more



I am aware of a similar issue that was raised on this list in 12-2011,
which was escalated to the Jiria list
(https://issues.apache.org/jira/browse/SOLR-2934), but am not sure if
that was ever resolved. Or am I just missing something? In either
case, could anyone who has working dictionary files share them with me
(any old language; as long as it works!)

I am using Solr 3.6.1 on a Mac running OSX 10.7.5

  - Rob

Has anyone HunspellStemFilterFactory working?

[Rob Koeling]

If so, would you be willing to share the .dic and .aff files with me?
When I try to load a dictionary file, Solr is complaining that:

java.lang.RuntimeException: java.io.IOException: Unable to load hunspell
data! [dictionary=en_GB.dic,affix=en_GB.aff]
at org.apache.solr.schema.IndexSchema.(IndexSchema.java:116)
...
Caused by: java.text.ParseException: The first non-comment line in the
affix file must be a 'SET charset', was: 'FLAG num'
at
org.apache.lucene.analysis.hunspell.HunspellDictionary.getDictionaryEncoding(HunspellDictionary.java:306)
at
org.apache.lucene.analysis.hunspell.HunspellDictionary.(HunspellDictionary.java:130)
at
org.apache.lucene.analysis.hunspell.HunspellStemFilterFactory.inform(HunspellStemFilterFactory.java:103)
... 46 more

When I change the first line to 'SET charset' it is still not happy. I got
the dictionary files from the OpenOffice website.

I'm using Solr 4.0 (but had the same problem with 3.6)

  - Rob

Re: Has anyone HunspellStemFilterFactory working?

[Rob Koeling]

Thanks for your reply, Sergey!

Well, I was a bit puzzled. I tried adding a line to set the character set
before, but then it complained about that as well.
I installed the Russian dictionary and Solr was happy to load that. I
noticed that the character-set was only set in the affix file for Russian.
So, when I added the line 'SET UTF-8' only to the affix file for en_UK, all
was well. I must have added that same line to the .dic file as well before
and I suppose that was what Solr was complaining about.

I just checked that, and that seems to be the case. The character-set
should only be set on the first line of the .aff file, the .dic file should
be left alone.

Thanks again Sergey, that was very useful.

Best,

   - Rob




On Wed, Nov 14, 2012 at 11:08 AM, Сергей Бирюков  wrote:

> Rob, as regards your "problem"
>
>> 'SET charset'
>>
> 'charset' word must be replaced with a name-of-character-set (i.e.
> encoding)
> For exampe,  you can write 'SET UTF-8'
>
> BUT...
>
> 
>
> Be careful!
> At least for russian language morthology HunspellStemFilterFactory has
> bug(s) in its algorythm.
>
> Simple comparison with original hunspell library shown huge difference.
>
>
> For example on  Linux x86_64 Ubuntu 12.10
>
> 1) INSTALL:
> # sudo apt-get install hunspell hunspell-ru
>
>
> 2) TEST with string "мама мыла раму мелом"
> (it has a meaning: "mom was_washing frame (with) chalk" ):
>
> 2.1) OS hunspell library
> # echo "мама мыла раму мелом" | hunspell -d ru_RU -D -m
> gives results:
> ...
> LOADED DICTIONARY:
> /usr/share/hunspell/ru_RU.aff
> /usr/share/hunspell/ru_RU.dic
>
> мама  -> мама
> мыла  -> мыло | мыть <<< as noun | as verb
> раму  -> рама
> мелом -> мел
>
> 2.2) solr's HunspellStemFilterFactory
> config fieldType
>  positionIncrementGap="100">
>   
> 
> 
>  dictionary="ru_RU.dic" affix="ru_RU.aff" ignoreCase="true" />
>   
> 
>
> gives results:
> мама -> мама | мама : FAILED:  duplicate words
> мыла -> мыть | мыло : SUSSECC: all OK
> раму -> рама | расти  : FAILED: second word is wrong and excess
> мелом -> мести | метить | месть | мел  :  FAILED: only last word is
> correct, other ones are excess
>
> --
>
> That's why I use a JNA (v3.2.7) binding on original (system)
> libhunspell.so for a long time :)
>
> 
> Best regards
>   Sergey Biryukov
>   Moscow, Russian Federation
>
>
>
> 14.11.2012 04:18, Rob Koeling wrote:
>
>> If so, would you be willing to share the .dic and .aff files with me?
>> When I try to load a dictionary file, Solr is complaining that:
>>
>> java.lang.RuntimeException: java.io.IOException: Unable to load hunspell
>> data! [dictionary=en_GB.dic,affix=**en_GB.aff]
>>  at org.apache.solr.schema.**IndexSchema.(**
>> IndexSchema.java:116)
>> ...
>> Caused by: java.text.ParseException: The first non-comment line in the
>> affix file must be a 'SET charset', was: 'FLAG num'
>>  at
>> org.apache.lucene.analysis.**hunspell.HunspellDictionary.**
>> getDictionaryEncoding(**HunspellDictionary.java:306)
>>  at
>> org.apache.lucene.analysis.**hunspell.HunspellDictionary.<**
>> init>(HunspellDictionary.java:**130)
>>  at
>> org.apache.lucene.analysis.**hunspell.**HunspellStemFilterFactory.**
>> inform(**HunspellStemFilterFactory.**java:103)
>>  ... 46 more
>>
>> When I change the first line to 'SET charset' it is still not happy. I got
>> the dictionary files from the OpenOffice website.
>>
>> I'm using Solr 4.0 (but had the same problem with 3.6)
>>
>>- Rob
>>
>>
>

Re: Solr 1.4 Release Date

[Rob Ganly]

i was wondering the same thing since:

'*NOTE: THE CURRENT GOAL IS TO START THE SOLR 1.4 RELEASE PROCESS
APPROXIMATELY ONE TO TWO WEEKS AFTER LUCENE JAVA 2.9 HAS BEEN RELEASED.* '

http://wiki.apache.org/solr/Solr1.4

patiently,

rob

2009/10/16 Michael R. 

>
> Any news on this? Lucene 2.9 is out for some weeks now.
>
>
>
> markrmiller wrote:
> >
> > Agreed! We are pushing towards it - one of the holds up is that Lucene
> 2.9
> > is about to release, so we are waiting for that. We really need to prune
> > down the JIRA list though. A few have been tackling it, but many of the
> > issues are still up in the air. I think once Lucene 2.9 releases though,
> > Solr 1.4 will shortly follow one way or another.
> > Lucene 2.9 is right on the verge - only a handful of pretty much finished
> > issues to resolve.
> >
> --
> View this message in context:
> http://www.nabble.com/Solr-1.4-Release-Date-tp23260381p25922568.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>

Re: How to get Solr 1.4 to replicate spellcheck directories as well?

[Rob Casson]

i don't think that's currently supported, but sure others will correct
me if i'm wrong:


http://www.lucidimagination.com/search/document/ac8cf41bdb761069/solr_replication_and_spellcheck_data

cheers,
rob

On Wed, Dec 16, 2009 at 10:08 AM, michael8  wrote:
>
> I'm currently using Solr 1.4 with its built-in solr.ReplicationHandler
> enabled in solrconfig.xml for a master and slave as follows:
>
>  
>    
>      ${enable.master:false}
>      commit
>      startup
>       name="confFiles">schema.xml,protwords.txt,spellings.txt,stopwords.txt,synonyms.txt
>    
>    
>      ${enable.slave:false}
>       name="masterUrl">http://searchhost:8983/solr/items/replication
>      00:00:60
>    
>  
>
> Everything in the index is replicated perfectly except that my spellcheck
> directories are not being replicated.  Here is my spellcheck config in
> solrconfig.xml:
>
>  
>    textSpell
>    
>      default
>      spell
>      ./spellchecker1
>      false
>
>    
>    
>      jarowinkler
>      spell
>      
>       name="distanceMeasure">org.apache.lucene.search.spell.JaroWinklerDistance
>      ./spellchecker2
>      false
>
>    
>
>    
>      solr.FileBasedSpellChecker
>      file
>      spellings.txt
>      UTF-8
>      ./spellcheckerFile
>      false
>    
>  
>
> I have set the buildOnCommit to 'false', but instead have a separate cron to
> build my spellcheck dictionaries on a nightly basis.
>
> Is there a way to tell Solr to also replicate the spellcheck files too?  Is
> my setting 'buildOnCommit' to 'false' causing my spellcheck files to not
> replicate?  I would think after the nightly build is triggered and done (via
> cron) that the spellcheck files would be replicated by that is not the case.
>
> Thanks for any help or info.
>
> Michael
>
> --
> View this message in context: 
> http://old.nabble.com/How-to-get-Solr-1.4-to-replicate-spellcheck-directories-as-well--tp26812569p26812569.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>

Re: query ( dynamicField ) in solr

[Rob Casson]

sounds like a job for copyField:

 http://wiki.apache.org/solr/FAQ#How_do_I_use_copyField_with_wildcards.3F

add sku_defaultPriceAll to your  and then:



...query on sku_defaultPriceAll.

hth,
rob

On Tue, Dec 22, 2009 at 3:36 PM, Gustavo  wrote:
> Hello All,
>
>  I have been trying to make a query at a dynamicField.
> for example:
> i have a product that can have many sku's. So at my xsd i have
>  stored="true" />
>  stored="true" />
>                                                    .
>                                                    .
>                                                    .
> when a make a query:
> http://localhost:8983/solr/productIdx/select/?q=sku_defaultPrice_2:valueOfThePrice
> or
> http://localhost:8983/solr/productIdx/select/?q=sku_defaultPrice_1:valueOfThePriceor
> sku_defaultPrice_2:valueOfThePrice or
> sku_defaultPrice_3:valueOfThePrice
> it works !!
>
> but i can have many sku_defaultPrice_*
> but i have no idea in how to make the query with this kind of field
> thanks for helping
>
> --
> --
> Atenciosamente,
>
> Gustavo de Araujo Lima Machado
> Telefone: +55 +21 ***
> Celular: +55+21 99034401
> E-Mail: g.macha...@gmail.com.br
>
> Residência
> :
>

Re: Simple Searching Question

[Rob Casson]

you're likely not copyField-ing *_facet to text, and we'd need to see
what type of field it is to see how it will be analyzed at both
search/index time.

the default schema.xml file is pretty well documented, so you might
want to spend some time looking thru it, and reading the
commentslots of good info in there.

cheers,
rob

On Thu, Aug 14, 2008 at 7:17 PM, Jake Conk <[EMAIL PROTECTED]> wrote:
> Hi Shalin,
>
> "foobar_facet" is a dynamic field. Its defined in my schema like this:
>
> 
>
> I have the default search field set to text. Can I use more than one
> default search field?
>
> text
>
> Thanks,
> - Jake
>
>
> On Thu, Aug 14, 2008 at 2:48 PM, Shalin Shekhar Mangar
> <[EMAIL PROTECTED]> wrote:
>> Hi Jake,
>>
>> What is the type of the foobar_facet field in your schema.xml ?
>> Did you add foobar_facet as the default search field?
>>
>> On Fri, Aug 15, 2008 at 3:13 AM, Jake Conk <[EMAIL PROTECTED]> wrote:
>>
>>> Hello,
>>>
>>> I inserted the following documents into Solr:
>>>
>>>
>>> ---
>>>
>>> 
>>> 
>>>  124
>>>  Jake Conk
>>> 
>>> 
>>>  125
>>>  Jake Conk
>>> 
>>> 
>>>
>>>
>>> ---
>>>
>>> id is the only required integer field.
>>> foobar_facet is a dynamic string field.
>>>
>>> When I try to search for anything with the word Jake in it the
>>> following ways I get no results.
>>>
>>>
>>> select?q=Jake
>>> select?q=Jake*
>>>
>>>
>>> I thought one of those two should work but the only way I got it to
>>> work was by specifying which field "Jake" is in along with a wild
>>> card.
>>>
>>>
>>> select?q=foobar_facet:Jake*
>>>
>>>
>>> 1) Does this mean for each field I would like to search if Jake exists
>>> I would have to add each field like I did above to the query?
>>>
>>> 2) How would I search if I want to find the name Jake anywhere in the
>>> string? The documentation
>>> (http://lucene.apache.org/java/docs/queryparsersyntax.html) states
>>> that I cannot use a wildcard as the first character such as *Jake*
>>>
>>> Thanks,
>>> - Jake
>>>
>>
>>
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
>

Re: [VOTE] Community Logo Preferences

[Rob Casson]

https://issues.apache.org/jira/secure/attachment/12394282/solr2_maho_impression.png
https://issues.apache.org/jira/secure/attachment/12394266/apache_solr_b_red.jpg

thanks to everyone who contributed,
rob

On Sun, Nov 23, 2008 at 11:59 AM, Ryan McKinley <[EMAIL PROTECTED]> wrote:
> Please submit your preferences for the solr logo.
>
> For full voting details, see:
>  http://wiki.apache.org/solr/LogoContest#Voting
>
> The eligible logos are:
>  http://people.apache.org/~ryan/solr-logo-options.html
>
> Any and all members of the Solr community are encouraged to reply to this
> thread and list (up to) 5 ranked choices by listing the Jira attachment
> URLs. Votes will be assigned a point value based on rank. For each vote, 1st
> choice has a point value of 5, 5th place has a point value of 1, and all
> others follow a similar pattern.
>
> https://issues.apache.org/jira/secure/attachment/12345/yourfrstchoice.jpg
> https://issues.apache.org/jira/secure/attachment/34567/yoursecondchoice.jpg
> ...
>
> This poll will be open until Wednesday November 26th, 2008 @ 11:59PM GMT
>
> When the poll is complete, the solr committers will tally the community
> preferences and take a final vote on the logo.
>
> A big thanks to everyone would submitted possible logos -- its great to see
> so many good options.

Re: Compiling Solr 1.3.0 + KStem

[Rob Casson]

i've experimented with the KStem stuff in the past, and just pulled a
fresh copy of solr from trunk

it looks like Hoss' suggestion #1 does the trick, by simply commenting
out the super.init call...loaded the example data, tested some
analysis, and it seems to work as before.

just a confirmation, and thanks,
rob

On Fri, Nov 28, 2008 at 6:18 PM, Chris Hostetter
<[EMAIL PROTECTED]> wrote:
>
> : /usr/local/build/apache-solr-1.3.0/src/java/org/apache/solr/analysis/
> : KStemFilterFactory.java:63:
> : cannot find symbol
> : [javac] symbol  : method
> : init(org.apache
> : .solr.core.SolrConfig,java.util.Map)
> : [javac] location: class org.apache.solr.analysis.BaseTokenFilterFactory
> : [javac] super.init(solrConfig, args);
> : [javac]  ^
>
> that KStemFilterFactory seems to be trying to use a method that existed
> for a while on the trunk, but was never released.
>
> i'm not familiary with KStemFilterFactory to know why/if it needs a
> SolrConfig, but a few things you can try...
>
> 1) if there are no references to solrConfig anywhere except the init
> method (and the super.init method it calls) just remove the refrences to
> it (so the methods just deal with the Map)
>
> 2) if there are other refrences to the solrConfig, they *may* just be to
> take advantage of ResourceLoader methods, so after making the changes
> above, make KStemFilterFactory "implements ResourceLoaderAware" and then
> add a method like this...
>
>  public void inform(ResourceLoader loader) {
>// code that used solrConfig should go here, but use loader
>  }
>
> ...it will get called after the init(Map) method and let
> KStemmFilterFactory get access to files on disk.
>
> 3) if that doesn't work ... i don't know what else to try (i'd need to get
> a lot more familiar with KStem to guess)
>
>
>
> -Hoss
>
>

Re: new faceting algorithm

[Rob Casson]

very similar situation to those already reported.  2.9M bilbiographic
records, with authors being the (previous) bottleneck, and the one
we're starting to test with the new algorithm.

so far, no load tests, but just in single requests i'm seeing the same
improvements...phenomenal improvements, btw, with most example queries
taking less than 1/100th of the time

always very impressed with this project/product, and just thought i'd
add a "me-too" to the list...cheers, and have a great weekend,

rob

On Mon, Nov 24, 2008 at 11:12 PM, Yonik Seeley <[EMAIL PROTECTED]> wrote:
> A new faceting algorithm has been committed to the development version
> of Solr, and should be available in the next nightly test build (will
> be dated 11-25).  This change should generally improve field faceting
> where the field has many unique values but relatively few values per
> document.  This new algorithm is now the default for multi-valued
> fields (including tokenized fields) so you shouldn't have to do
> anything to enable it.  We'd love some feedback on how it works to
> ensure that it actually is a win for the majority and should be the
> default.
>
> -Yonik
>

Re: Filter query with wildcard, fq=a*

[Rob Casson]

it sounds to me like the field you're using (artistText) is tokenized
and lowercased.  it might be good to go over  the wiki pages again:

 http://wiki.apache.org/solr/SolrFacetingOverview

if you keep having problems, post your schema...cheers,
rob

On Thu, Apr 30, 2009 at 10:21 AM, Andrew McCombe  wrote:
> Hi
>
> It has now introduced a new issue.  I am now getting results for my
> query but they are not what I wanted.  I'm using a Dismax handler and
> am trying to filter a result set with a filter query:
>
> http://localhost:8180/solr/select?indent=on&version=2.2&q=i+love&start=0&rows=30&fl=*%2Cscore&qt=dismax&wt=standard&fq=artistText:e*
>
> This is bringing results with 'e' or 'E' as the first letter and with
> 'e' or 'E' as the first letter after whitespace.  e.g, Eric Prydz,
> Marc Et Claude.
>
> Looking at the solr wiki it seems that I am using the fq param to
> boost the results and not filter them.  Is there a way to achieve the
> required result, i.e, filtering the result set for all artistText
> beginning with 'E'?
>
> Regards
> Andrew
>
>
>
> 2009/4/30 Andrew McCombe 
>>
>> Hi
>>
>> I've sorted it.  Wildcard term must be lowercase to get results.
>>
>> Thanks
>> Andrew
>>
>> 2009/4/30 Andrew McCombe 
>>>
>>> Hi
>>>
>>> I have half a million records indexed and need to filter results on a term 
>>> by the first letter.  For Example the search term is 'I love' and returns a 
>>> few thousand records.  I need to filter these results for all artists 
>>> beginning with 'A'.
>>>
>>> I've tried:
>>> 'fq=artistText:A*'
>>>
>>> But then get no results.
>>>
>>> Can someone point me in the right direction please?
>>>
>>> Regards
>>> Andrew
>>
>

Re: Upgrading from 1.2.0 to 1.3.0

[Rob Casson]

this isn't advice on how to upgrade, but if you/your-project have a
bit of time to wait, 1.4 sounds like it's getting close to an official
releasefyi.

cheers,
rob


On Tue, May 5, 2009 at 1:05 PM, Francis Yakin  wrote:
>
> What's the best way to upgrade solr from 1.2.0 to 1.3.0 ?
>
> We have the current index that our users search running on 1.2.0 Solr version.
>
> We would like to upgrade it to 1.3.0?
>
> We have Master/Slaves env.
>
> What's the best way to upgrade it without affecting the search? Do we need to 
> do it on master or slaves first?
>
>
>
> Thanks
>
> Francis
>
>
>

Re: case-insensitive string type

[Rob Casson]

from http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters

"On wildcard and fuzzy searches, no text analysis is performed on
the search word."

i'd just lowercase the wildcard-ed search term in your client code,
before you send it to solr.

hth,
rob

On Wed, Jan 13, 2010 at 2:18 PM, Harsch, Timothy J. (ARC-TI)[PEROT
SYSTEMS]  wrote:
> Hi I have a field:
>
>  multiValued="true" />
>
> With type definition:
>                
>                                        sortMissingLast="true" omitNorms="true">
>                        
>                                 class="solr.KeywordTokenizerFactory"/>
>                                
>                        
>                        
>                                 class="solr.KeywordTokenizerFactory"/>
>                                
>                        
>                
>
> When searching that field I can't get a case-insensitive match.  It works as 
> if it is a regular string, for instance I can do a prefix query and so long 
> as the prefix matches the case of the value it works, but if I change the 
> prefix case it doesn't
>
> Essentially I am trying to get case-insensitive matching that supports wild 
> cards...
>
> Tim Harsch
> Sr. Software Engineer
> Dell Perot Systems
> (650) 604-0374
>
>

Re: filter querying working on dynamic int fields but not dynamic string fields?

[Rob Casson]

> http://localhost:8983/solr/select?indent=on&version=2.2&q=climate&fq=awardinstrument_s:Continuing+grant
> Continuing grant 

everything that erik already mentioned, but looks like you also have a
trailing space in the document, so even quoting it would require that
last space.

Re: how to do partial word searches?

[Rob Ganly]

hi all,

i was having the same problem, i needed to be able to search a substring
anywhere within a word for a specific field. i used the
NGramTokenizerFactory factory in my index analyzer and it seems to work
well.  (
http://lucene.apache.org/solr/api/org/apache/solr/analysis/NGramTokenizerFactory.html
).

i created a new field type based on this definition:
http://coderrr.wordpress.com/category/solr/#ngram_schema_xml

apparently it will increased the size of your index and perhaps indexing
time but is working fine at the moment (although i'm currently only using a
testbed of 20'000 records). i will report back if i discover any painful
issues with scaling up!

rob ganly

On 3 December 2009 18:21, Joel Nylund  wrote:

> Just for an update on this, I tried text_rev and it seems to work great.
>
> So in summary, if you want partial word matches within a url or small
> sentence (title), here is what I did and it seems to work pretty well:
>
> - create an extra field that is all lower case , I used mysql lcase in the
> query for DIH
> - make that field use text_rev type in schema.xml
> - make the query be "sulli OR *sulli*"(the *sulli* doesnt seem to match
> sulli if its at the end of the field)
>
> thanks
> Joel
>
>
>
>
> On Nov 25, 2009, at 9:21 AM, Robert Muir wrote:
>
>  Hi, if you are using Solr 1.4 I think you might want to try type text_rev
>> (look in the example schema.xml)
>>
>> unless i am mistaken:
>>
>> this will enable leading wildcard support for that field.
>> this doesn't do any stemming, which I think might be making your wildcards
>> behave wierd.
>> it also enables reverse wildcard support, so some of your substring
>> matches
>> will be faster.
>>
>> On Tue, Nov 24, 2009 at 7:51 PM, Joel Nylund  wrote:
>>
>>  Hi, I saw some older postings on this, but didnt see a resolution.
>>>
>>> I have a field called title, I would like to be able to find partial word
>>> matches within the title.
>>>
>>> For example:
>>>
>>> http://localhost:8983/solr/select?q=textTitle:%22*sulli*%22
>>>
>>> I would expect it to find:
>>> the daily dish | by andrew sullivan
>>>
>>> but it doesnt, it does find sully (which is fine with me also as a
>>> bonus),
>>> but doesnt seem to get any of the partial word stuff. Oddly enough before
>>> I
>>> lowercased the title, the wildcard matching seemed to work a bit better,
>>> it
>>> just didnt deal with the case sensitive query.
>>>
>>> At first I had mixed case titles and I read that the wildcard doesn't
>>> work
>>> with mixed case, so I created another field that is a lowered version of
>>> the
>>> title called "textTitle", it is of type text.
>>>
>>> Is it possible with solr to achieve what I am trying to do, if so how? If
>>> not, anything closer than what I have?
>>>
>>> thanks
>>> Joel
>>>
>>>
>>>
>>
>> --
>> Robert Muir
>> rcm...@gmail.com
>>
>
>

RE: Field Collapsing SOLR-236

[Rob Z]

Thanks Ill give field-collapse-5 a try although I heard it has some bad memory 
bugs in it.

> From: martijn.is.h...@gmail.com
> Date: Thu, 25 Mar 2010 13:29:30 +0100
> Subject: Re: Field Collapsing SOLR-236
> To: solr-user@lucene.apache.org
> 
> Hi Blargy,
> 
> The latest path is not compatible with 1.4, I believe that the latest
> field-collapse-5.patch file is compatible with 1.4. The file should at
> least compile with 1.4 trunk. I'm not sure how the performance is.
> 
> Martijn
> 
> On 25 March 2010 01:49, Dennis Gearon  wrote:
> > Boy, I hope that field collapsing works! I'm planning on using it heavily.
> > Dennis Gearon
> >
> > Signature Warning
> > 
> > EARTH has a Right To Life,
> >  otherwise we all die.
> >
> > Read 'Hot, Flat, and Crowded'
> > Laugh at http://www.yert.com/film.php
> >
> >
> > --- On Wed, 3/24/10, blargy  wrote:
> >
> >> From: blargy 
> >> Subject: Field Collapsing SOLR-236
> >> To: solr-user@lucene.apache.org
> >> Date: Wednesday, March 24, 2010, 12:17 PM
> >>
> >> Has anyone had any luck with the field collapsing patch
> >> (SOLR-236) with Solr
> >> 1.4? I tried patching my version of 1.4 with no such luck.
> >>
> >> Thanks
> >> --
> >> View this message in context: 
> >> http://old.nabble.com/Field-Collapsing-SOLR-236-tp28019949p28019949.html
> >> Sent from the Solr - User mailing list archive at
> >> Nabble.com.
> >>
> >>
> >
> 
> 
> 
> -- 
> Met vriendelijke groet,
> 
> Martijn van Groningen
  
_
Hotmail: Trusted email with powerful SPAM protection.
http://clk.atdmt.com/GBL/go/210850553/direct/01/

RE: Field Collapsing SOLR-236

[Rob Z]

What do you mean you had to revert to Trunk 1.5. Do you mean upgrade? Which 
version were you using before hand? 

Can you please list the exact version of 1.5 and the patch # you used. I 
downloaded the latest nightly build and tried patching using the 2/1 patch. 
Everything went ok but I am getting 1 failing test.

Would you recommend using the latest nightly 1.5 build or 1.4 for production 
use? I really need this feature so I don't think I have much of a choice. Can 
you also explain the performance implications you are seeing AND what 
configuration tweaks you've used that helped. 

Thanks!

> From: mark.robe...@red-gate.com
> To: solr-user@lucene.apache.org
> Date: Thu, 25 Mar 2010 15:21:54 +
> Subject: RE: Field Collapsing SOLR-236
> 
> Yeah got it working fine - but I needed to revert to Trunk (1.5) to get the 
> patch to apply.
> 
> It does certainly have some performance implications, but tweaking 
> configuration can help here. 
> 
> Overall the benefits very much outweigh the costs for us :)
> 
> Mark.
> 
> 
> -Original Message-
> From: Dennis Gearon [mailto:gear...@sbcglobal.net] 
> Sent: 25 March 2010 00:49
> To: solr-user@lucene.apache.org
> Subject: Re: Field Collapsing SOLR-236
> 
> Boy, I hope that field collapsing works! I'm planning on using it heavily.
> Dennis Gearon
> 
> Signature Warning
> 
> EARTH has a Right To Life,
>   otherwise we all die.
> 
> Read 'Hot, Flat, and Crowded'
> Laugh at http://www.yert.com/film.php
> 
> 
> --- On Wed, 3/24/10, blargy  wrote:
> 
> > From: blargy 
> > Subject: Field Collapsing SOLR-236
> > To: solr-user@lucene.apache.org
> > Date: Wednesday, March 24, 2010, 12:17 PM
> > 
> > Has anyone had any luck with the field collapsing patch
> > (SOLR-236) with Solr
> > 1.4? I tried patching my version of 1.4 with no such luck.
> > 
> > Thanks
> > -- 
> > View this message in context: 
> > http://old.nabble.com/Field-Collapsing-SOLR-236-tp28019949p28019949.html
> > Sent from the Solr - User mailing list archive at
> > Nabble.com.
> > 
> > 
  
_
Hotmail: Trusted email with Microsoft’s powerful SPAM protection.
http://clk.atdmt.com/GBL/go/210850552/direct/01/

spellcheck suggestion ordering and distance measures

[Rob Casson]

poking around at the spellcheck component, and have a couple questions:

1) is there a way to return the distance measure with
spellcheck.extendedResults?  haven't poked too closely at the source,
but it might be useful.

2) i'm not entirely clear on the order in which suggestions are
returned. for a search of "porgrams" against a subset of my data, i
get these suggestions...i've added my own levenshtein calculations,
provided they're correct:

 program: 3
 pulgram: 3
 porras: 2
 portrait: 3

relevant section of my solrconfig.xml:

text


  default
  spell
  ./spellchecker



granted, i like that 'program' is at the top, and it has the highest
frequency in my corpus, but just want to make sure i can reliably
interpret these results.

cheers, and apologies if i'm just being dense,
rob

Re: spellcheck suggestion ordering and distance measures

[Rob Casson]

> 2) i'm not entirely clear on the order in which suggestions are
> returned. for a search of "porgrams" against a subset of my data, i
> get these suggestions...i've added my own levenshtein calculations,
> provided they're correct:
>
> program: 3
> pulgram: 3
> porras: 2
> portrait: 3

i should have included the relevant section of the response:


-

-

5
0
8
0
-

-

program
271

-

pulgram
1

-

porras
1

-

portrait
74

-

nomogram
1



false




erikhatcher might be pulling-my-head-out-of-my-ass off-list, fwiw ;)

cheers, and thanks all,
rc

Re: Update Index : Updating Specific Fields

[Rob Ganly]

>>may be this is one very imp feature to be considered for next releases of
SOLR.

i agree, is glaringly important. i'm quite suprised that this feature isn't
already available.  can anyone elaborate on the reason that it isn't
available? (out of interest).

>>sometimes these kind of cases would come up.

very often these type of cases come up! i need to use it now... !

hopefully it'll be available soon.

rob ganly

2010/3/4 Kranti™ K K Parisa 

> may be this is one very imp feature to be considered for next releases of
> SOLR. sometimes these kind of cases would come up.
>
> Best Regards,
> Kranti K K Parisa
>
>
>
> On Thu, Mar 4, 2010 at 3:01 PM, Andrzej Bialecki  wrote:
>
> > On 2010-03-04 07:41, Walter Underwood wrote:
> >
> >> No. --wunder
> >>
> >
> > Or perhaps "not yet" ...
> >
> > http://portal.acm.org/ft_gateway.cfm?id=1458171
> >
> >
> >
> >> On Mar 3, 2010, at 10:40 PM, Kranti™ K K Parisa wrote:
> >>
> >>  Hi,
> >>>
> >>> Is there any way to update the index for only the specific fields?
> >>>
> >>> Eg:
> >>> Index has ONE document consists of 4 fields,  F1, F2, F3, F4
> >>> Now I want to update the value of field F2, so if I send the update xml
> >>> to
> >>> SOLR, can it keep the old field values for F1,F3,F4 and update the new
> >>> value
> >>> specified for F2?
> >>>
> >>> Best Regards,
> >>> Kranti K K Parisa
> >>>
> >>
> >>
> >>
> >>
> >
> >
> > --
> > Best regards,
> > Andrzej Bialecki <><
> >  ___. ___ ___ ___ _ _   __
> > [__ || __|__/|__||\/|  Information Retrieval, Semantic Web
> > ___|||__||  \|  ||  |  Embedded Unix, System Integration
> > http://www.sigram.com  Contact: info at sigram dot com
> >
> >
>

Re: Anyone using Solr spatial from trunk?

[Rob Ganly]

i used the 1.5 build a few weeks ago, implemented the geospatial
functionality and it worked really well.

however due to the unknown quantity in terms of stability (and the uncertain
future of 1.5) etc. we decided not to use it in production.

rob ganly

On 8 June 2010 03:50, Darren Govoni  wrote:

> I've been experimenting with it, but haven't quite gotten it to work as
> yet.
>
> On Mon, 2010-06-07 at 17:47 -0700, Jay Hill wrote:
>
> > I was wondering about the production readiness of the new-in-trunk
> spatial
> > functionality. Is anyone using this in a production environment?
> >
> > -Jay
>
>
>

Re: Anyone using Solr spatial from trunk?

[Rob Ganly]

>>... but decided not to use it anyway?

that's pretty much correct.  the huge commercial scale of the project
dictates that we need as much system stability as possible from the outset;
thus the tools we are use must be established, community-tested and trusted
versions.  we also noticed that some of the regular non-geospatial queries
seemed to run slightly slower than on 1.4, with only a fraction of the total
amount of records we'd be searching in production (but that wasn't the main
reason for our decision).

i would perhaps use it for a much smaller [private] project where speed,
scaling and reliability weren't such critical issues.

future proofing was also a consideration:

*"With all the changes currently occurring with Solr, I would go so far as
to say that users should continue to use Solr 1.4. However, if you need
access to one of the many new features introduced in Solr 1.5+ or Lucene
3.x, then given Solr 3.1 a shot, and report back your experiences."  *(from
http://blog.jteam.nl/2010/04/14/state-of-solr/*).*

On 8 June 2010 21:09, Darren Govoni  wrote:

> So let me understand what you said. You went through the trouble to
> implement a geospatial
> solution using Solr 1.5, it worked really well. You saw no signs of
> instability, but decided not to use it anyway?
>
> Did you put it through a routine of tests and witness some stability
> problem? Or just guessing it had them?
>
> I'm just curious the reasoning behind your comment.
>
> On Tue, 2010-06-08 at 09:05 +0100, Rob Ganly wrote:
>
> > i used the 1.5 build a few weeks ago, implemented the geospatial
> > functionality and it worked really well.
> >
> > however due to the unknown quantity in terms of stability (and the
> uncertain
> > future of 1.5) etc. we decided not to use it in production.
> >
> > rob ganly
> >
> > On 8 June 2010 03:50, Darren Govoni  wrote:
> >
> > > I've been experimenting with it, but haven't quite gotten it to work as
> > > yet.
> > >
> > > On Mon, 2010-06-07 at 17:47 -0700, Jay Hill wrote:
> > >
> > > > I was wondering about the production readiness of the new-in-trunk
> > > spatial
> > > > functionality. Is anyone using this in a production environment?
> > > >
> > > > -Jay
> > >
> > >
> > >
>
>
>

delete by negative query

[Rob Casson]

i'm having no luck deleting by a negative query

indexing the example docs from 1.2, these steps work:

curl http://localhost:8983/solr/update --data-binary
'solr' -H 'Content-type:text/xml;
charset=utf-8'

curl http://localhost:8983/solr/update --data-binary '' -H
'Content-type:text/xml; charset=utf-8'

but if i reindex, and change the delete query to a negative, the
non-'solr' docs don't get deleted:

curl http://localhost:8983/solr/update --data-binary
'-solr' -H 'Content-type:text/xml;
charset=utf-8'

curl http://localhost:8983/solr/update --data-binary '' -H
'Content-type:text/xml; charset=utf-8'

good chance i'm missing something obvious

tia,
r

Re: delete by negative query

[Rob Casson]

piete,

thanks for the quick reply.

> You need to explicitly define the field you are referring to in order to
> achieve this, otherwise the query parser will assume that the minus
> character is part of the query and interpret it as field:"-solr" (where
> "field" is the name of the default field set in your schema).  Try:
>
> curl http://localhost:8983/solr/update --data-binary
> '-field:solr' -H 'Content-type:text/xml;
> charset=utf-8'

does this work for you?:

curl http://localhost:8983/solr/update --data-binary
'-name:solr' -H
'Content-type:text/xml; charset=utf-8'

curl http://localhost:8983/solr/update --data-binary '' -H
'Content-type:text/xml; charset=utf-8'

then, 
http://localhost:8983/solr/select/?q=-name:solr&version=2.2&start=0&rows=10&indent=on

i still get all of those documents returned

> parser will assume that the minus
> character is part of the query and interpret it as field:"-solr" (where
> "field" is the name of the default field set in your schema)

my true use case is to deleteByQuery against my default field, but
even in this case, with the field specified, and using the
exampledocs, i'm not seeing the document get deleted.

thanks again...maybe i should just grab a beer, and go to bed...cheers :)
r

Re: delete by negative query

[Rob Casson]

> the work arround is to include *:* in yoru query ...
>*:* -solr
> ... if/when this is fixed
> in Solr that's esentally what solr will do under the covers.
>
> (would you mind opening a bug to track this and mention the work arround
> for other people who encounter it)

will do.

thanks again...adding the *:* to any query (qualified or not) seems to
be working as expected

solr++,
r

LowerCaseFilterFactory and spellchecker

[Rob Casson]

think i'm just doing something wrong...

was experimenting with the spellcheck handler with the nightly
checkout from 11-28; seems my spellchecking is case-sensitive, even
tho i think i'm adding the LowerCaseFilterFactory to both the index
and query analyzers.

here's a brief rundown of my testing steps.

from schema.xml:























from solrconfig.xml:



1
0.5

spell
spelling




adding the doc:

curl http://localhost:8983/solr/update -H "Content-Type: text/xml"
--data-binary 'Thorne'
curl http://localhost:8983/solr/update -H "Content-Type: text/xml"
--data-binary ''



building the spellchecker:

http://localhost:8983/solr/select/?q=Thorne&qt=spellchecker&cmd=rebuild



querying the spellchecker:

results from http://localhost:8983/solr/select/?q=Thorne&qt=spellchecker




0
1

Thorne
false

thorne



results from http://localhost:8983/solr/select/?q=thorne&qt=spellchecker




0
2

thorne
true




any pointers as to what i'm doing wrong, misinterpreting?  i suspect
i'm just doing something bone-headed in the analyzer sections...

thanks as always,

rob casson
miami university libraries

Re: LowerCaseFilterFactory and spellchecker

[Rob Casson]

lance,

thanks for the quick replylooks like 'thorne' is getting added to
the dictionary, as it comes up as a suggestion for 'Thorne'

i could certainly just lowercase in my client, but just confirming
that i'm not just screwing it up in the firstplace :)

thanks again,
rc

On Nov 28, 2007 8:11 PM, Norskog, Lance <[EMAIL PROTECTED]> wrote:
> There are a few parameters for limiting what words are added to the
> dictionary.  You might be trimming out 'thorne'. See this page:
>
> http://wiki.apache.org/solr/SpellCheckerRequestHandler
>
>
> -Original Message-
> From: Rob Casson [mailto:[EMAIL PROTECTED]
> Sent: Wednesday, November 28, 2007 4:25 PM
> To: solr-user@lucene.apache.org
> Subject: LowerCaseFilterFactory and spellchecker
>
> think i'm just doing something wrong...
>
> was experimenting with the spellcheck handler with the nightly checkout
> from 11-28; seems my spellchecking is case-sensitive, even tho i think
> i'm adding the LowerCaseFilterFactory to both the index and query
> analyzers.
>
> here's a brief rundown of my testing steps.
>
> from schema.xml:
>
>  positionIncrementGap="100">
> 
> 
> 
>  class="solr.RemoveDuplicatesTokenFilterFactory"/>
> 
> 
> 
> 
> 
>  class="solr.RemoveDuplicatesTokenFilterFactory"/>
> 
> 
> 
>
>  multiValued="true"/>
>  multiValued="true"/>
>
> 
>
> 
>
> from solrconfig.xml:
>
>  class="solr.SpellCheckerRequestHandler" startup="lazy">
> 
> 1
> 0.5
> 
> spell
> spelling
> 
>
> 
>
> adding the doc:
>
> curl http://localhost:8983/solr/update -H "Content-Type: text/xml"
> --data-binary ' name="title">Thorne'
> curl http://localhost:8983/solr/update -H "Content-Type: text/xml"
> --data-binary ''
>
> 
>
> building the spellchecker:
>
> http://localhost:8983/solr/select/?q=Thorne&qt=spellchecker&cmd=rebuild
>
> 
>
> querying the spellchecker:
>
> results from http://localhost:8983/solr/select/?q=Thorne&qt=spellchecker
>
> 
> 
> 
> 0
> 1
> 
> Thorne
> false
> 
> thorne
> 
> 
>
> results from http://localhost:8983/solr/select/?q=thorne&qt=spellchecker
>
> 
> 
> 
> 0
> 2
> 
> thorne
> true
> 
> 
>
>
> any pointers as to what i'm doing wrong, misinterpreting?  i suspect i'm
> just doing something bone-headed in the analyzer sections...
>
> thanks as always,
>
> rob casson
> miami university libraries
>

Re: How to delete records that don't contain a field?

[Rob Casson]

i'm using this:

*:* -[* TO *]

which is what lance suggested..works just fine.

fyi: https://issues.apache.org/jira/browse/SOLR-381

On Dec 3, 2007 8:09 PM, Norskog, Lance <[EMAIL PROTECTED]> wrote:
> Wouldn't this be: *:* AND "negative query"
>
>
> -Original Message-
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Yonik
> Seeley
> Sent: Monday, December 03, 2007 2:23 PM
> To: solr-user@lucene.apache.org
> Subject: Re: How to delete records that don't contain a field?
>
> On Dec 3, 2007 5:22 PM, Jeff Leedy <[EMAIL PROTECTED]> wrote:
>
> > I was wondering if there was a way to post a delete query using curl
> > to delete all records that do not contain a certain field--something
> > like
> > this:
> >
> > curl http://localhost:8080/solr/update --data-binary
> > '-_title:[* TO *]' -H
> > 'Content-type:text/xml; charset=utf-8'
> >
> > The minus syntax seems to return the correct list of ids (that is, all
>
> > records that do not contain the "_title" field) when I use the Solr
> > administrative console to do the above query, so I'm wondering if Solr
>
> > just doesn't support this type of delete.
>
>
> Not yet... it makes sense to support this in the future though.
>
> -Yonik
>

Re: schema for "literal string, case insensitive"

[Rob Casson]

bram,

you'll want to look at the KeywordTokenizerFactory (which doesn't
actually tokenize), and then use the LowerCaseFilterFactory.  the
schema in the example has a fieldType called 'alphaOnlySort' that
should get you started.

cheers,
rob

On Thu, May 29, 2008 at 6:21 AM, Bram de Jong <[EMAIL PROTECTED]> wrote:
> hello all,
>
> a while ago I was looking into making the schema for my (rather rich)
> data set, and I wanted to have a field for username.
> I would need a case insensitive string for that, but literal (no
> tokenizing, ...).
>
> how can I do this within the Solr schema definition?
>
>  - bram
>
> --
> http://freesound.iua.upf.edu
> http://www.smartelectronix.com
> http://www.musicdsp.org
>

Problem with importing tab-delimited csv file

[Rob Koeling Ai]

I'm having trouble importing a tab-delimited file with the csv update handler.My data file looks like this:"id"	"question"	"answer"	"url""q99"	"Who?"	"You!"	"none"When I send this data to Solr using Curl:curl 'http://localhost:8181/solr/development/update/csv?commit=true&separator=%09' --data @sample.tmpAll seems to be well:0221But when I query the development core, there is no data. I must be overlooking something trivial. I would appreciate if anyone could spot what!  - Rob

Problem with importing tab-delimited csv file

[Rob Koeling Ai]

Thanks for the reply, Jack.

It only looks like spaces, because I did a cut-and-paste. The file in question 
does contain tabs instead of spaces, i.e.:

"id""question"  "answer""url"
"q99"   "Who?"  "You!"  "none"


Another question I means to ask, is whether this sort of activity is logged 
anywhere. I mean, after adding or deleting data, is there somewhere a record of 
that action?
The 'logging' tab on the Dashboard page only reports errors as far as I can see.

Thanks,

   - Rob



Your data file appears to use spaces rather than tabs.

-- Jack Krupansky

From: Rob Koeling Ai 
Sent: Friday, August 23, 2013 6:38 AM
To: solr-user@lucene.apache.org 
Subject: Problem with importing tab-delimited csv file


I'm having trouble importing a tab-delimited file with the csv update handler.

My data file looks like this:

"id" "question" "answer" "url"
"q99" "Who?" "You!" "none"

When I send this data to Solr using Curl:

curl 
'http://localhost:8181/solr/development/update/csv?commit=true&separator=%09' 
--data
@sample.tmp

All seems to be well:



0221



But when I query the development core, there is no data. I must be overlooking 
something trivial.
I would appreciate if anyone could spot what!

  - Rob