General hardware requirements?
Our index is about 10 gigs in size with about 3 million documents. The documents range in size from dozens to hundreds of kilobytes. Per week, we only get about 50k queries. Currently, we use lucene and have one box for our indexer that has 32 gigs of memory and an 8 core CPU. We have a pair of search boxes that have about 16 gigs of ram a piece and 8 core CPUs. They hardly break a sweat. We're looking to adopt Solr. Should we consider changing our configuration at all? Are there any other hardware considerations for adopting Solr? Thanks,Nick
RE: General hardware requirements?
I think the indexing will be fine. We are looking to use multi-select faceting, spelling suggestions, and highlighting to name a few. On the front end (and on separate machines) are .NET web applications that issue queries via HTTP requests to our searchers. I can't think of anything else that will require extra processing. Thanks for bringing those considerations to my attention. Is there anything there that significantly impacts the hardware needs? -Original Message- From: Gora Mohanty [mailto:g...@mimirtech.com] Sent: Thursday, September 30, 2010 10:47 AM To: solr-user@lucene.apache.org Subject: Re: General hardware requirements? On Thu, Sep 30, 2010 at 8:09 PM, Nicholas Swarr wrote: > > Our index is about 10 gigs in size with about 3 million documents. The documents range in size from dozens to hundreds of kilobytes. Per week, we only get about 50k queries. > Currently, we use lucene and have one box for our indexer that has 32 gigs of memory and an 8 core CPU. We have a pair of search boxes that have about 16 gigs of ram a piece and 8 core CPUs. They hardly break a sweat. > We're looking to adopt Solr. Should we consider changing our configuration at all? Are there any other hardware considerations for adopting Solr? [...] On the face of it, your machines should easily be adequate for the the search volume you are looking at. However, there are other things that you should consider: * How are you indexing? What are acceptable times for this? * Are there any new Solr-specific features that you are considering using, e.g., faceting? What performance benchmarks are you looking to achieve? * What is your front-end for the search? Where is it running? Regards, Gora
solr.xml isn't loaded from classpath?
(may have double posted...apologies if it is) It seems like when "solr home" is absent, Solr makes an attempt to look a few other places to load its configuration. It will try to look for solrconfig.xml on the classpath as well. It doesn't seem like it makes any attempt to find solr.xml though. Why is that? Read below for the larger narrative... The gory details: Having this configuration discovery makes things really convenient for creating custom Solr web applications where you can throw all of Solr's config in your resources, create a war, deploy it to Tomcat and it happily loads. No setting of environment variables or setup required. Something like this, /someapp/src/main/resources |-solrconfig.xml |-schema.xml |-etc. The same approach is outlined here: http://netsuke.wordpress.com/2010/06/24/launching-solr-from-maven-for-rapid-development/ We're creating a multicore installation and have created a folder structure which no longer has a solrconfig.xml at the top level of the resources. /someapp/src/main/resources |-solr.xml |-core1 |-solrconfig.xml |-schema.xml |-etc. |-core2 |-solrconfig.xml |-schema.xml |-etc. And when you try to run this, Solr can't find what it needs to start up. To fix this, we manually deployed the configuration on the web server and set the solr/home environment variable on the web app's config within Tomcat. Not ideal and it makes automation awkward. Ultimately, I want a completely packaged war for a multicore instance I can drop anywhere without additional setup. Is this possible? Am I approaching this wrong? This e-mail message, and any attachments, is intended only for the use of the individual or entity identified in the alias address of this message and may contain information that is confidential, privileged and subject to legal restrictions and penalties regarding its unauthorized disclosure and use. Any unauthorized review, copying, disclosure, use or distribution is strictly prohibited. If you have received this e-mail message in error, please notify the sender immediately by reply e-mail and delete this message, and any attachments, from your system. Thank you.
Hierarchical faceting and filter query exclusions
We're using Solr 4.0 Beta, testing the hierarchical faceting support to see if it's a good fit to facet on taxonomies. One issue we've encountered is that we can't apply filter exclusions to the hierarchical facets so as to preserve facet count with multi-select. I haven't been able to locate or otherwise determine if there's documentation that would outline how this is done. We've tried a few things with local params but it appears those aren't parsed with the facet.pivot argument. I found this ticket related to that: https://issues.apache.org/jira/browse/SOLR-2255 Could anyone offer some insight or guidance on this? Thanks, Nick This e-mail message, and any attachments, is intended only for the use of the individual or entity identified in the alias address of this message and may contain information that is confidential, privileged and subject to legal restrictions and penalties regarding its unauthorized disclosure and use. Any unauthorized review, copying, disclosure, use or distribution is strictly prohibited. If you have received this e-mail message in error, please notify the sender immediately by reply e-mail and delete this message, and any attachments, from your system. Thank you.
Re: Hierarchical faceting and filter query exclusions
Thanks, Erick. I have looked at the documentation at length and done a number of tests. I don't see a way to do exclusions with hierarchical facets. Thanks, Nick On Aug 29, 2012, at 9:50 PM, "Erick Erickson" wrote: > See "Tagging and excluding filters" here: > http://lucidworks.lucidimagination.com/display/solr/Faceting > > > Best > Erick > > > On Wed, Aug 29, 2012 at 11:44 AM, Nicholas Swarr wrote: >> We're using Solr 4.0 Beta, testing the hierarchical faceting support to see >> if it's a good fit to facet on taxonomies. One issue we've encountered is >> that we can't apply filter exclusions to the hierarchical facets so as to >> preserve facet count with multi-select. I haven't been able to locate or >> otherwise determine if there's documentation that would outline how this is >> done. We've tried a few things with local params but it appears those >> aren't parsed with the facet.pivot argument. I found this ticket related to >> that: >> >> https://issues.apache.org/jira/browse/SOLR-2255 >> >> Could anyone offer some insight or guidance on this? >> >> Thanks, >> Nick >> This e-mail message, and any attachments, is intended only for the use of >> the individual or entity identified in the alias address of this message and >> may contain information that is confidential, privileged and subject to >> legal restrictions and penalties regarding its unauthorized disclosure and >> use. Any unauthorized review, copying, disclosure, use or distribution is >> strictly prohibited. If you have received this e-mail message in error, >> please notify the sender immediately by reply e-mail and delete this >> message, and any attachments, from your system. Thank you. >> >> > This e-mail message, and any attachments, is intended only for the use of the individual or entity identified in the alias address of this message and may contain information that is confidential, privileged and subject to legal restrictions and penalties regarding its unauthorized disclosure and use. Any unauthorized review, copying, disclosure, use or distribution is strictly prohibited. If you have received this e-mail message in error, please notify the sender immediately by reply e-mail and delete this message, and any attachments, from your system. Thank you.
Re: Hierarchical faceting and filter query exclusions
Tanguy, thanks for the confirmation! We found the same issue with
facet.missing parameter as well.
Sent from my iPad
On Aug 30, 2012, at 10:18 AM, "Tanguy Moal" wrote:
> You are correct, it doesn't work :
>
> Queries like :
> http://localhost:8983/solr/collection1/select?q=*:*&facet=on&facet.pivot={!ex=a_tag}field1,field2&facet.limit=5&rows=0&fq={!tag=a_tag}field3:"filter";
> result in the following response :
>
>
>
>
> 400
> 1
>
> on
> *:*
> 5
> {!ex=a_tag}field1,field2
> {!tag=a_tag}field3:"filter"
> 0
>
>
>
>
> undefined field: "{!ex=a_tag}field1"
> 400
>
>
>
> Seems like that face.pivot didn't expect the {!ex } local params and died
> horribly (not even fully repeated: the ',field2' part is missing in the
> error message)
>
> Don't know if that's supposed to be supported.
>
> I've got another funny one : facet pivoting doesn't work when missing
> values are faceted :
> Queries like :
> http://localhost:8983/solr/collection1/select?q=*:*&facet=on&facet.pivot=field1,field2&facet.limit=5&facet.missing=on&rows=0result
> in the following error :
>
>
>
>
> 500
> 3
>
> on
> on
> on
> *:*
> 5
> field1,field2
> 0
>
>
>
>
> java.lang.NullPointerException at
> org.apache.solr.schema.FieldType.readableToIndexed(FieldType.java:376) at
> org.apache.solr.handler.component.PivotFacetHelper.doPivots(PivotFacetHelper.java:109)
> at
> org.apache.solr.handler.component.PivotFacetHelper.doPivots(PivotFacetHelper.java:126)
> at
> org.apache.solr.handler.component.PivotFacetHelper.process(PivotFacetHelper.java:85)
> at
> org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:90)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:206)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
> at
> org.apache.solr.core.SolrCore.execute(SolrCore.java:1656) at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:454)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:275)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1331)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:477)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:521)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
> at
> org.eclipse.jetty.server.Server.handle(Server.java:349) at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:449)
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:910)
> at
> org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:634) at
> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230) at
> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76)
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
> at
> java.lang.Thread.run(Thread.java:722)
>
> 500
>
>
>
> Here also, I don't know if that's supposed to be supported, but it
> obviously isn't.
> Worst, it results in an horrible 500 response with a stack instead of a
> clean 400 + message : "sorry, facet.missing can't be turned on with
> facet.pivot. Search again better :)
