Re: Limiting duplicate field occurrences to specified number
Again, thanks for pointing me to the patches. Admittedly, I am not all that well informed in the patching world. I know how to apply them and all that. But, I am trying to track down exactly which patches I need to apply. I currently have the source for Solr 1.2. The patches are for 1.1, 1.3 and the current dev (a total of 8 patches in there). But, there is no tag or anything for a 1.3 The 1.3 patch does seem to successfully apply to the 1.2 code base and compiles (though, not sure if that will work anyway). But, there is a: " This issue depends on: SOLR-281 Search Components (plugins)" heading under it. Then that link shows 14 patches. So, if you have a moment, could you list the patches that I need to get to test this? If not, I would be more than happy to read any documentation how to understand the patching process. On Feb 4, 2008 11:52 AM, Briggs <[EMAIL PROTECTED]> wrote: > Cool, thanks! > > > On Feb 4, 2008 11:36 AM, Ryan McKinley <[EMAIL PROTECTED]> wrote: > > perhaps: > > https://issues.apache.org/jira/browse/SOLR-236 > > > > > > > > Briggs wrote: > > > Is it possible to limit the number of duplicate field values are > > > returned in a search result? > > > > > > To give a use case, > > > > > > I have a set of products. Each product belongs to a single vendor. > > > When I query, I would like only n-number of results per vendor > > > returned. > > > > > > Thanks! > > > > > > > > > > > > > > > > > > -- > "Conscious decisions by conscious minds are what make reality real" > -- "Conscious decisions by conscious minds are what make reality real"
Limiting duplicate field occurrences to specified number
Is it possible to limit the number of duplicate field values are returned in a search result? To give a use case, I have a set of products. Each product belongs to a single vendor. When I query, I would like only n-number of results per vendor returned. Thanks! -- "Conscious decisions by conscious minds are what make reality real"
Re: Limiting duplicate field occurrences to specified number
Cool, thanks! On Feb 4, 2008 11:36 AM, Ryan McKinley <[EMAIL PROTECTED]> wrote: > perhaps: > https://issues.apache.org/jira/browse/SOLR-236 > > > > Briggs wrote: > > Is it possible to limit the number of duplicate field values are > > returned in a search result? > > > > To give a use case, > > > > I have a set of products. Each product belongs to a single vendor. > > When I query, I would like only n-number of results per vendor > > returned. > > > > Thanks! > > > > > > > > -- "Conscious decisions by conscious minds are what make reality real"
Re: what is searcher
Searcher is the main search abstraction in Lucene. It defines the methods used for querying an underlying index(es). See: http://lucene.apache.org/java/2_3_0/api/org/apache/lucene/search/Searcher.html On Feb 12, 2008 10:33 PM, Mochamad bahri nurhabbibi <[EMAIL PROTECTED]> wrote: > hello all.. > > I am learning SOLR since 2 days ago. > > I have to make training/presentation aboutSOLR to rest of my fellow in my > company. > > my question is: what is searcher ? > > this term seems to be found everywhere. but > there's no exact definition of this term either in google nor SOLR wiki. > > anyone please help me.. > > thank you > > regards > > - habibi- > -- "Conscious decisions by conscious minds are what make reality real"
DataImportHandler w/ multivalued fields
Hello Solr Community! I am implementing a data connection to Solr through the Data Import Handler and non-multivalued fields are working correctly, but multivalued fields are not getting indexed properly. I am new to DataImportHandler, but from what I could find, the entity is the way to go for multivalued field. The weird thing is that data is being indexed for one row, meaning first raw_tag gets populated. Anyone have any ideas? Thanks, Briggs This is the relevant part of the schema: And the relevant part of data-import.xml:
Re: DataImportHandler w/ multivalued fields
In addition, I tried a query like below and changed the column definition
to
and still no luck. It is indexing the full content now but not multivalued.
It seems like the "splitBy" ins't working properly.
select group_concat(freetags.raw_tag separator ', ') as raw_tag, site.*
from site
left outer join
(freetags inner join freetagged_objects)
on (freetags.id = freetagged_objects.tag_id
and site.siteId = freetagged_objects.object_id)
group by site.siteId
Am I doing something wrong?
Thanks,
Briggs Thompson
On Thu, Dec 1, 2011 at 11:46 AM, Briggs Thompson <
w.briggs.thomp...@gmail.com> wrote:
> Hello Solr Community!
>
> I am implementing a data connection to Solr through the Data Import
> Handler and non-multivalued fields are working correctly, but multivalued
> fields are not getting indexed properly.
>
> I am new to DataImportHandler, but from what I could find, the entity is
> the way to go for multivalued field. The weird thing is that data is being
> indexed for one row, meaning first raw_tag gets populated.
>
>
> Anyone have any ideas?
> Thanks,
> Briggs
>
> This is the relevant part of the schema:
>
> stored="false" multivalued="true"/>
> stored="true" multivalued="true"/>
>
>
> And the relevant part of data-import.xml:
>
>
>query="select * from site ">
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> query="select raw_tag, freetags.id,
> freetagged_objects.object_id as siteId
>from freetags
>inner join freetagged_objects
>on freetags.id=freetagged_objects.tag_id
> where freetagged_objects.object_id='${site.siteId}'">
>
>
>
>
>
Re: DataImportHandler w/ multivalued fields
Hey Rahul,
Thanks for the response. I actually just figured it thankfully :). To
answer your question, the raw_tag is indexed and not stored (tokenized),
and then there is a copyField for raw_tag to "raw_tag_string" which would
be used for facets. That *should have* been displayed in the results.
The silly mistake I made was not camel casing "multiValued", which is
clearly the source of the problem.
The second email I sent changing the query and using the split for the
multivalued field had an error in it in the form of a missing line:
transformer="RegexTransformer"
in the entity declaration.
Anyhow, thanks for the quick response!
Briggs
On Thu, Dec 1, 2011 at 12:57 PM, Rahul Warawdekar <
rahul.warawde...@gmail.com> wrote:
> Hi Briggs,
>
> By saying "multivalued fields are not getting indexed prperly", do you mean
> to say that you are not able to search on those fields ?
> Have you tried actually searching your Solr index for those multivalued
> terms and make sure if it returns the search results ?
>
> One possibility could be that the multivalued fields are getting indexed
> correctly and are searchable.
> However, since your schema.xml has a "raw_tag" field whose "stored"
> attribute is set to false, you may not be able to see those fields.
>
>
>
> On Thu, Dec 1, 2011 at 1:43 PM, Briggs Thompson <
> w.briggs.thomp...@gmail.com
> > wrote:
>
> > In addition, I tried a query like below and changed the column definition
> > to
> >
> > and still no luck. It is indexing the full content now but not
> multivalued.
> > It seems like the "splitBy" ins't working properly.
> >
> >select group_concat(freetags.raw_tag separator ', ') as raw_tag,
> site.*
> > from site
> > left outer join
> > (freetags inner join freetagged_objects)
> > on (freetags.id = freetagged_objects.tag_id
> > and site.siteId = freetagged_objects.object_id)
> > group by site.siteId
> >
> > Am I doing something wrong?
> > Thanks,
> > Briggs Thompson
> >
> > On Thu, Dec 1, 2011 at 11:46 AM, Briggs Thompson <
> > w.briggs.thomp...@gmail.com> wrote:
> >
> > > Hello Solr Community!
> > >
> > > I am implementing a data connection to Solr through the Data Import
> > > Handler and non-multivalued fields are working correctly, but
> multivalued
> > > fields are not getting indexed properly.
> > >
> > > I am new to DataImportHandler, but from what I could find, the entity
> is
> > > the way to go for multivalued field. The weird thing is that data is
> > being
> > > indexed for one row, meaning first raw_tag gets populated.
> > >
> > >
> > > Anyone have any ideas?
> > > Thanks,
> > > Briggs
> > >
> > > This is the relevant part of the schema:
> > >
> > > > > stored="false" multivalued="true"/>
> > > > > stored="true" multivalued="true"/>
> > >
> > >
> > > And the relevant part of data-import.xml:
> > >
> > >
> > > > > query="select * from site ">
> > >
> > >
> > >
> > >
> > >
> > >
> > > name="requiresModeration"
> > />
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > name="enableSplitTesting"
> > />
> > > />
> > >
> > >
> > > > > query="select raw_tag, freetags.id,
> > > freetagged_objects.object_id as siteId
> > >from freetags
> > >inner join freetagged_objects
> > >on freetags.id=freetagged_objects.tag_id
> > > where freetagged_objects.object_id='${site.siteId}'">
> > >
> > >
> > >
> > >
> > >
> >
>
>
>
> --
> Thanks and Regards
> Rahul A. Warawdekar
>
Dynamic Fields vs. Multicore
Hi All, I was searching around for documentation of the performance differences of having a sharded, single schema, dynamic field set up vs. a multi-core, static multi-schema setup (which I currently have), but I have not had much luck finding what I am looking for. I understand commits and optimizes will be more intensive in a single core since there is more data (though I would offset by sharding heavily), but I am particularly curious about the search performance implications. I am interested in moving to the dynamic field setup in order to implement a better global search, but I want to make sure I understood the drawbacks of hitting those datasets individually and globally after they are merged (NOTE: I would have a global field signifying the dataset type, which could then be added to the filter query in order to create the subset for individual dataset queries). Some background about the data: it is extremely variable. Some documents contain only 2 or 3 sentences, and some are 20 page extracted PDFs. There would probably only be about 100-150 unique fields. Any input is greatly appreciated! Thanks, Briggs Thompson
Hit Rate
Hello all, Is there a good way to get the hit count of a search? Example query: textField:solr AND documentId:1000 Say document with Id = 1000 has "solr" 13 times in the document. Any way to extract that number [13] in the response? I know we can return the score which is loosely related to hit counts via tf-idf, but for this case I need the actually hit counts. I believe you can get this information from the logs, but that is less useful if the use case is on the presentation layer. I tried faceting on the query but it seems like that returns the number of documents that query matches rather than the hit count. http://localhost:8080/solr/ExampleCore/select/?q=textField%3Asolr+AND+documentId%3A1246727&version=2.2&start=0&rows=10&indent=on&&facet=true&face.field=textField:solr&facet.query=<http://cobra:8080/solr/BusinessDescriptionCore/select/?q=businessDescription%3Afacebook+AND+businessDescriptionId%3A1246727&version=2.2&start=0&rows=10&indent=on&&facet=true&face.field=businessDescriptionQuoted:facebook&facet.query=businessDescriptionQuoted:facebook> textField:solr<http://cobra:8080/solr/BusinessDescriptionCore/select/?q=businessDescription%3Afacebook+AND+businessDescriptionId%3A1246727&version=2.2&start=0&rows=10&indent=on&&facet=true&face.field=businessDescriptionQuoted:facebook&facet.query=businessDescriptionQuoted:facebook> I was thinking that highlighting essentially returns the hit count if you supply unlimited amount of snippets, but I imagine there must be a more elegant solution. Thanks in advance, Briggs
Re: Hit Rate
Yes indeed, that is what I was missing. Thanks Ahmet! On Tue, Jul 5, 2011 at 12:48 PM, Ahmet Arslan wrote: > > > Is there a good way to get the hit count of a search? > > > > Example query: > > textField:solr AND documentId:1000 > > > > Say document with Id = 1000 has "solr" 13 times in the > > document. Any way to > > extract that number [13] in the response? > > Looks like you are looking for term frequency info: > > Two separate solutions: > http://wiki.apache.org/solr/TermVectorComponent > http://wiki.apache.org/solr/FunctionQuery#tf > > >
Re: Need help with troublesome wildcard query
Hello Christopher, Can you provide the exact query sent to Solr for the one word query and also the two word query? The field type definition for your title field would be useful too. >From what I understand, Solr should be able to handle your use case. I am guessing it is a problem with how the field is defined assuming the query is correct. Briggs Thompson On Thu, Jul 7, 2011 at 12:22 PM, Christopher Cato < christopher.c...@minimedia.se> wrote: > Hi, I'm running Solr 3.2 with edismax under Tomcat 6 via Drupal. > > I'm having some problems writing a query that matches a specific field on > several words. I have implemented an AJAX search that basically takes > whatever is in a form field and attempts to match documents. I'm not having > much luck though. First word always matches correctly but as soon as I enter > the second word I'm loosing matches, the third word doesn't give any matches > at all. > > The title field that I'm searching contains a product name that may or may > not have several words. > > The requirement is that the search should be progressive i.e. as the user > inputs words I should always return results that contain all of the words > entered. I also have to correct bad input like an erraneous space in the > product name ex. "product name" instead of "productname". > > I'm wondering if there isn't an easier way to query Solr? Ideally I'd want > to say "give me all docs that have the following text in it's titles" Is > that possible? > > > I'd really appreciate any help! > > > Regards, > Christopher Cato
Re: Need help with troublesome wildcard query
Hey Chris,
Removing the ORs in each query might help narrow down the problem, but I
suggest you run this through the query analyzer in order to see where it is
dropping out. It is a great tool for troubleshooting issues like these.
I see a few things here.
- for leading wildcard queries, you should include the
reverseWildcardFilterFactory. Check out the documentation here:
http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.ReversedWildcardFilterFactory
- Your result might get dropped out because you are trying to do wildcard
searches on a stemmed field. Wildcard searches on a stemmed field is
counter-intuitive because if you index "computers", it may stem to "comput",
in which wildcard query of "computer*" would not match.
- If you want to support stemming and wildcard searches, I suggest
creating a copy field with an un-stemmed field type definition.
Don't forget if you modify your field type definition, you need to
re-index.
In response to your question about text_ws, this is just a different field
type definition that essentially splits on whiteSpaces. You should use that
if that is what the desired search logic is, but it probably isn't. Check
out the documentation on each of the tokenizers and filter factories in your
"text" field type and see what you need and what you don't to satisfy your
use cases.
Hope that helps,
Briggs Thompson
On Fri, Jul 8, 2011 at 9:03 AM, Christopher Cato <
christopher.c...@minimedia.se> wrote:
> Hi Briggs. Thanks for taking the time. I have the query nearly working now,
> currently this is how it looks when it matches on the title "Super
> Technocrane 30" and others with similar names:
>
> INFO: [] webapp=/solr path=/select/
> params={qf=title^40.0&hl.fl=title&wt=json&rows=10&fl=*,score&start=0&q=(title:*super*+AND+*technocran*)+OR+(title:*super*+AND+*technocran)&qt=standard&fq=type:product+AND+language:sv}
> hits=3 status=0 QTime=1
>
> Adding another letter stops it matching:
>
> INFO: [] webapp=/solr path=/select/
> params={qf=title^40.0&hl.fl=title&wt=json&rows=10&fl=*,score&start=0&q=(title:*super*+AND+*technocrane*)+OR+(title:*super*+AND+*technocrane)&qt=standard&fq=type:product+AND+language:sv}
> hits=0 status=0 QTime=0
>
> The field type definitions are as follows:
>
> termVectors="true" omitNorms="true"/>
>
> positionIncrementGap="100">
>
> mapping="mapping-ISOLatin1Accent.txt"/>
>
>
>
>ignoreCase="true"
>words="stopwords.txt"
>enablePositionIncrements="true"
>/>
>generateWordParts="1"
>generateNumberParts="1"
>catenateWords="1"
>catenateNumbers="1"
>catenateAll="0"
>splitOnCaseChange="1"
>preserveOriginal="1"/>
>
> protected="protwords.txt"/>
>
>
>
> mapping="mapping-ISOLatin1Accent.txt"/>
>
> ignoreCase="true" expand="true"/>
>ignoreCase="true"
>words="stopwords.txt"
>enablePositionIncrements="true"
>/>
>generateWordParts="1"
>generateNumberParts="1"
>catenateWords="0"
>catenateNumbers="0"
> catenateAll="0"
>splitOnCaseChange="1"
>preserveOriginal="1"/>
>
> protected="protwords.txt"/>
>
>
>
>
>
> There is also a type definition that is called text_ws, should I use that
> instead and change text to text_ws in the field definition for title?
>
>
> positionIncrementGap="100">
>
>
>
>
>
>
>
>
> Mvh
>
> Christopher Cato
> Teknikchef
> ---
> MiniMedia
> Phone: +46761927603
> www.minimedia.se
>
> 7 jul 2011 kl. 23.16 skrev Briggs Thompson:
>
> > Hello Christopher,
> >
> > Can you provide the exact query sent to Solr for the one word query and
> also
> > the two word query? The field type definition for your title field would
> be
> > useful too.
> >
> > From what I underst
Re: difference between shard and core in solr
I think everything you said is correct for static schemas, but a single core does not necessarily have a unique schema since you can have dynamic fields. With dynamic fields, you can have multiple types of documents in the same index (core), and a multiple types of indexed fields specific to individual document types - all in the same core. Briggs Thompson On Mon, Jul 18, 2011 at 2:22 AM, pravesh wrote: > >a single core is an index with same schema , is this wat core really is ? > > YES. A single core is a independent index with its own unique schema. You > go with a new core for cases where your schema/analysis/search requirements > are completely different from your existing core(s). > > >can a single core contain two separate indexes with different schema in it > ? > > NO (for same reason as explained above). > > >Is a shard refers to a collection of index in a single physical machine > >?can a single core be presented in different shards ? > > You can think of a Shard as a big index distributed across a cluster of > machines. So all shards belonging to a single core share same > schema/analysis/search requirements. You go with sharding when index is not > scalable on a single machine, or, when your index grows really big in size. > > > Thanx > Pravesh > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/difference-between-shard-and-core-in-solr-tp3178214p3178249.html > Sent from the Solr - User mailing list archive at Nabble.com. >
Re: Trunk error in Tomcat
Thanks Erik. If anyone else has any ideas about the NoSuchFieldError issue please let me know. Thanks! -Briggs On Mon, Jul 2, 2012 at 6:27 PM, Erik Hatcher wrote: > Interestingly, I just logged the issue of it not showing the right error > in the UI here: <https://issues.apache.org/jira/browse/SOLR-3591> > > As for your specific issue, not sure, but the error should at least also > show in the admin view. > > Erik > > > On Jul 2, 2012, at 18:59 , Briggs Thompson wrote: > > > Hi All, > > > > I just grabbed the latest version of trunk and am having a hard time > > getting it running properly in tomcat. It does work fine in Jetty. The > > admin screen gives the following error: > > This interface requires that you activate the admin request handlers, add > > the following configuration to your Solrconfig.xml > > > > I am pretty certain the front end error has nothing to do with the actual > > error. I have seen some other folks on the distro with the same problem, > > but none of the threads have a solution (that I could find). Below is the > > stack trace. I also tried with different versions of Lucene but none > > worked. Note: my index is EMPTY and I am not migrating over an index > build > > with a previous version of lucene. I think I ran into this a while ago > with > > an earlier version of trunk, but I don't recall doing anything to fix it. > > Anyhow, if anyone has an idea with this one, please let me know. > > > > Thanks! > > Briggs Thompson > > > > SEVERE: null:java.lang.NoSuchFieldError: LUCENE_50 > > at > > > org.apache.solr.analysis.SynonymFilterFactory$1.createComponents(SynonymFilterFactory.java:83) > > at org.apache.lucene.analysis.Analyzer.tokenStream(Analyzer.java:83) > > at > > > org.apache.lucene.analysis.synonym.SynonymMap$Builder.analyze(SynonymMap.java:120) > > at > > > org.apache.lucene.analysis.synonym.SolrSynonymParser.addInternal(SolrSynonymParser.java:99) > > at > > > org.apache.lucene.analysis.synonym.SolrSynonymParser.add(SolrSynonymParser.java:70) > > at > > > org.apache.solr.analysis.SynonymFilterFactory.loadSolrSynonyms(SynonymFilterFactory.java:131) > > at > > > org.apache.solr.analysis.SynonymFilterFactory.inform(SynonymFilterFactory.java:93) > > at > > > org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:584) > > at org.apache.solr.schema.IndexSchema.(IndexSchema.java:112) > > at org.apache.solr.core.CoreContainer.create(CoreContainer.java:812) > > at org.apache.solr.core.CoreContainer.load(CoreContainer.java:510) > > at org.apache.solr.core.CoreContainer.load(CoreContainer.java:333) > > at > > > org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:282) > > at > > > org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:101) > > at > > > org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:277) > > at > > > org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:258) > > at > > > org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:382) > > at > > > org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:103) > > at > > > org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4649) > > at > > > org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5305) > > at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) > > at > > > org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) > > at > org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875) > > at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:618) > > at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:963) > > at > > > org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1600) > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:680) > >
Re: Trunk error in Tomcat
Also, I forgot to include this before, but there is a client side error which is a failed 404 request to the below URL. http://localhost:8983/solr/null/admin/system?wt=json On Tue, Jul 3, 2012 at 8:45 AM, Briggs Thompson wrote: > Thanks Erik. If anyone else has any ideas about the NoSuchFieldError issue > please let me know. Thanks! > > -Briggs > > > On Mon, Jul 2, 2012 at 6:27 PM, Erik Hatcher wrote: > >> Interestingly, I just logged the issue of it not showing the right error >> in the UI here: <https://issues.apache.org/jira/browse/SOLR-3591> >> >> As for your specific issue, not sure, but the error should at least also >> show in the admin view. >> >> Erik >> >> >> On Jul 2, 2012, at 18:59 , Briggs Thompson wrote: >> >> > Hi All, >> > >> > I just grabbed the latest version of trunk and am having a hard time >> > getting it running properly in tomcat. It does work fine in Jetty. The >> > admin screen gives the following error: >> > This interface requires that you activate the admin request handlers, >> add >> > the following configuration to your Solrconfig.xml >> > >> > I am pretty certain the front end error has nothing to do with the >> actual >> > error. I have seen some other folks on the distro with the same problem, >> > but none of the threads have a solution (that I could find). Below is >> the >> > stack trace. I also tried with different versions of Lucene but none >> > worked. Note: my index is EMPTY and I am not migrating over an index >> build >> > with a previous version of lucene. I think I ran into this a while ago >> with >> > an earlier version of trunk, but I don't recall doing anything to fix >> it. >> > Anyhow, if anyone has an idea with this one, please let me know. >> > >> > Thanks! >> > Briggs Thompson >> > >> > SEVERE: null:java.lang.NoSuchFieldError: LUCENE_50 >> > at >> > >> org.apache.solr.analysis.SynonymFilterFactory$1.createComponents(SynonymFilterFactory.java:83) >> > at org.apache.lucene.analysis.Analyzer.tokenStream(Analyzer.java:83) >> > at >> > >> org.apache.lucene.analysis.synonym.SynonymMap$Builder.analyze(SynonymMap.java:120) >> > at >> > >> org.apache.lucene.analysis.synonym.SolrSynonymParser.addInternal(SolrSynonymParser.java:99) >> > at >> > >> org.apache.lucene.analysis.synonym.SolrSynonymParser.add(SolrSynonymParser.java:70) >> > at >> > >> org.apache.solr.analysis.SynonymFilterFactory.loadSolrSynonyms(SynonymFilterFactory.java:131) >> > at >> > >> org.apache.solr.analysis.SynonymFilterFactory.inform(SynonymFilterFactory.java:93) >> > at >> > >> org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:584) >> > at org.apache.solr.schema.IndexSchema.(IndexSchema.java:112) >> > at org.apache.solr.core.CoreContainer.create(CoreContainer.java:812) >> > at org.apache.solr.core.CoreContainer.load(CoreContainer.java:510) >> > at org.apache.solr.core.CoreContainer.load(CoreContainer.java:333) >> > at >> > >> org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:282) >> > at >> > >> org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:101) >> > at >> > >> org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:277) >> > at >> > >> org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:258) >> > at >> > >> org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:382) >> > at >> > >> org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:103) >> > at >> > >> org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4649) >> > at >> > >> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5305) >> > at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) >> > at >> > >> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899) >> > at >> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875) >> > at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:618) >> > at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:963) >> > at >> > >> org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1600) >> > at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) >> > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >> > at java.util.concurrent.FutureTask.run(FutureTask.java:138) >> > at >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> > at >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> > at java.lang.Thread.run(Thread.java:680) >> >> >
Re: Trunk error in Tomcat
Wow! I didn't know 4.0 alpha was released today. I think I will just get that going. Woo!! On Tue, Jul 3, 2012 at 9:00 AM, Vadim Kisselmann wrote: > same problem here: > > > https://mail.google.com/mail/u/0/?ui=2&view=btop&ver=18zqbez0n5t35&q=tomcat%20v.kisselmann&qs=true&search=query&th=13615cfb9a5064bd&qt=kisselmann.1.tomcat.1.tomcat's.1.v.1&cvid=3 > > > > https://issues.apache.org/jira/browse/SOLR-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13230056#comment-13230056 > > i use an older solr-trunk version from february/march, it works. with > newer versions from trunk i get the same error: "This interface > requires that you activate the admin request handlers..." > > regards > vadim > > > > 2012/7/3 Briggs Thompson : > > Also, I forgot to include this before, but there is a client side error > > which is a failed 404 request to the below URL. > > > > http://localhost:8983/solr/null/admin/system?wt=json > > > > On Tue, Jul 3, 2012 at 8:45 AM, Briggs Thompson < > w.briggs.thomp...@gmail.com > >> wrote: > > > >> Thanks Erik. If anyone else has any ideas about the NoSuchFieldError > issue > >> please let me know. Thanks! > >> > >> -Briggs > >> > >> > >> On Mon, Jul 2, 2012 at 6:27 PM, Erik Hatcher >wrote: > >> > >>> Interestingly, I just logged the issue of it not showing the right > error > >>> in the UI here: <https://issues.apache.org/jira/browse/SOLR-3591> > >>> > >>> As for your specific issue, not sure, but the error should at least > also > >>> show in the admin view. > >>> > >>> Erik > >>> > >>> > >>> On Jul 2, 2012, at 18:59 , Briggs Thompson wrote: > >>> > >>> > Hi All, > >>> > > >>> > I just grabbed the latest version of trunk and am having a hard time > >>> > getting it running properly in tomcat. It does work fine in Jetty. > The > >>> > admin screen gives the following error: > >>> > This interface requires that you activate the admin request handlers, > >>> add > >>> > the following configuration to your Solrconfig.xml > >>> > > >>> > I am pretty certain the front end error has nothing to do with the > >>> actual > >>> > error. I have seen some other folks on the distro with the same > problem, > >>> > but none of the threads have a solution (that I could find). Below is > >>> the > >>> > stack trace. I also tried with different versions of Lucene but none > >>> > worked. Note: my index is EMPTY and I am not migrating over an index > >>> build > >>> > with a previous version of lucene. I think I ran into this a while > ago > >>> with > >>> > an earlier version of trunk, but I don't recall doing anything to fix > >>> it. > >>> > Anyhow, if anyone has an idea with this one, please let me know. > >>> > > >>> > Thanks! > >>> > Briggs Thompson > >>> > > >>> > SEVERE: null:java.lang.NoSuchFieldError: LUCENE_50 > >>> > at > >>> > > >>> > org.apache.solr.analysis.SynonymFilterFactory$1.createComponents(SynonymFilterFactory.java:83) > >>> > at org.apache.lucene.analysis.Analyzer.tokenStream(Analyzer.java:83) > >>> > at > >>> > > >>> > org.apache.lucene.analysis.synonym.SynonymMap$Builder.analyze(SynonymMap.java:120) > >>> > at > >>> > > >>> > org.apache.lucene.analysis.synonym.SolrSynonymParser.addInternal(SolrSynonymParser.java:99) > >>> > at > >>> > > >>> > org.apache.lucene.analysis.synonym.SolrSynonymParser.add(SolrSynonymParser.java:70) > >>> > at > >>> > > >>> > org.apache.solr.analysis.SynonymFilterFactory.loadSolrSynonyms(SynonymFilterFactory.java:131) > >>> > at > >>> > > >>> > org.apache.solr.analysis.SynonymFilterFactory.inform(SynonymFilterFactory.java:93) > >>> > at > >>> > > >>> > org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:584) > >>> > at org.apache.solr.schema.IndexSchema.(IndexSchema.java:112) > >>> > at org.apache.solr.core.CoreContainer.create(CoreContainer.jav
Re: Solr 4 Alpha SolrJ Indexing Issue
I have realized this is not specific to SolrJ but to my instance of Solr.
Using curl to delete by query is not working either.
Running
curl http://localhost:8983/solr/coupon/update -H "Content-Type: text/xml"
--data-binary '*:*'
Yields this in the logs:
INFO: [coupon] webapp=/solr path=/update
params={stream.body=*:*}
{deleteByQuery=*:*} 0 0
But the corpus of documents in the core do not change.
My solrconfig is pretty barebones at this point, but I attached it in case
anyone sees something strange. Anyone have any idea why documents aren't
getting deleted?
Thanks in advance,
Briggs Thompson
On Wed, Jul 18, 2012 at 12:54 PM, Briggs Thompson <
w.briggs.thomp...@gmail.com> wrote:
> Hello All,
>
> I am using 4.0 Alpha and running into an issue with indexing using
> HttpSolrServer (SolrJ).
>
> Relevant java code:
> HttpSolrServer solrServer = new HttpSolrServer(MY_SERVER);
> solrServer.setRequestWriter(new BinaryRequestWriter());
>
> Relevant Solrconfig.xml content:
>
>
>
>class="solr.BinaryUpdateRequestHandler" />
>
> Indexing documents works perfectly fine (using addBeans()), however, when
> trying to do deletes I am seeing issues. I tried to do
> a solrServer.deleteByQuery("*:*") followed by a commit and optimize, and
> nothing is deleted.
>
> The response from delete request is a "success", and even in the solr logs
> I see the following:
>
> INFO: [coupon] webapp=/solr path=/update/javabin
> params={wt=javabin&version=2} {deleteByQuery=*:*} 0 1
> Jul 18, 2012 11:15:34 AM org.apache.solr.update.DirectUpdateHandler2 commit
> INFO: start
> commit{flags=0,version=0,optimize=true,openSearcher=true,waitSearcher=false,expungeDeletes=false,softCommit=false}
>
>
>
> I tried removing the binaryRequestWriter and have the request send out in
> default format, and I get the following error.
>
> SEVERE: org.apache.solr.common.SolrException: Unsupported ContentType:
> application/octet-stream Not in: [application/xml, text/csv, text/json,
> application/csv, application/javabin, text/xml, application/json]
>
> at
> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:86)
> at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1561)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:442)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:263)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:225)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1001)
> at
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:579)
> at
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
>
>
> I thought that an optimize does the same thing as expungeDeletes, but in
> the log I see expungeDeletes=false. Is there a way to force that using
> SolrJ?
>
> Thanks in advance,
> Briggs
>
>
LUCENE_40
${solr.data.dir:}
${autoCommitTime:0}
false
${autoSoftCommitTime:0}
${solr.data.dir:}
true
solr-data-config.xml
${masterEnabled:false}
commit
startup
optimize
schema.xml,stopwords.txt,synonyms.txt
${slaveEnabled:false}
http://${masterServer:localhost}:${port:8983}/solr/${masterCoreName:}/replication
${pollInterval:00:00:15}
solrpingquery
all
solr
Re: Solr 4 Alpha SolrJ Indexing Issue
Yury, Thank you so much! That was it. Man, I spent a good long while trouble shooting this. Probably would have spent quite a bit more time. I appreciate your help!! -Briggs On Wed, Jul 18, 2012 at 9:35 PM, Yury Kats wrote: > On 7/18/2012 7:11 PM, Briggs Thompson wrote: > > I have realized this is not specific to SolrJ but to my instance of > Solr. Using curl to delete by query is not working either. > > Can be this: https://issues.apache.org/jira/browse/SOLR-3432 >
Re: Solr 4 Alpha SolrJ Indexing Issue
This is unrelated for the most part, but the javabin update request handler does not seem to be working properly when calling solrj method*HttpSolrServer.deleteById(List ids) *. A single Id gets deleted from the index as opposed to the full list. It appears properly in the logs - shows delete of all Ids sent, although all but one remain in the index. I confirmed that the default update request handler deletes the list properly, so this appears to be a problem with the BinaryUpdateRequestHandler. Not an issue for me, just spreading the word. Thanks, Briggs On Thu, Jul 19, 2012 at 9:00 AM, Mark Miller wrote: > we really need to resolve that issue soon... > > On Jul 19, 2012, at 12:08 AM, Briggs Thompson wrote: > > > Yury, > > > > Thank you so much! That was it. Man, I spent a good long while trouble > > shooting this. Probably would have spent quite a bit more time. I > > appreciate your help!! > > > > -Briggs > > > > On Wed, Jul 18, 2012 at 9:35 PM, Yury Kats wrote: > > > >> On 7/18/2012 7:11 PM, Briggs Thompson wrote: > >>> I have realized this is not specific to SolrJ but to my instance of > >> Solr. Using curl to delete by query is not working either. > >> > >> Can be this: https://issues.apache.org/jira/browse/SOLR-3432 > >> > > - Mark Miller > lucidimagination.com > > > > > > > > > > > >
Re: Solr 4 Alpha SolrJ Indexing Issue
Thanks Mark! On Thu, Jul 19, 2012 at 4:07 PM, Mark Miller wrote: > https://issues.apache.org/jira/browse/SOLR-3649 > > On Thu, Jul 19, 2012 at 3:34 PM, Briggs Thompson < > w.briggs.thomp...@gmail.com> wrote: > > > This is unrelated for the most part, but the javabin update request > handler > > does not seem to be working properly when calling solrj > > method*HttpSolrServer.deleteById(List ids) > > *. A single Id gets deleted from the index as opposed to the full list. > It > > appears properly in the logs - shows delete of all Ids sent, although all > > but one remain in the index. > > > > I confirmed that the default update request handler deletes the list > > properly, so this appears to be a problem with > > the BinaryUpdateRequestHandler. > > > > Not an issue for me, just spreading the word. > > > > Thanks, > > Briggs > > > > On Thu, Jul 19, 2012 at 9:00 AM, Mark Miller > > wrote: > > > > > we really need to resolve that issue soon... > > > > > > On Jul 19, 2012, at 12:08 AM, Briggs Thompson wrote: > > > > > > > Yury, > > > > > > > > Thank you so much! That was it. Man, I spent a good long while > trouble > > > > shooting this. Probably would have spent quite a bit more time. I > > > > appreciate your help!! > > > > > > > > -Briggs > > > > > > > > On Wed, Jul 18, 2012 at 9:35 PM, Yury Kats > wrote: > > > > > > > >> On 7/18/2012 7:11 PM, Briggs Thompson wrote: > > > >>> I have realized this is not specific to SolrJ but to my instance of > > > >> Solr. Using curl to delete by query is not working either. > > > >> > > > >> Can be this: https://issues.apache.org/jira/browse/SOLR-3432 > > > >> > > > > > > - Mark Miller > > > lucidimagination.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > - Mark > > http://www.lucidimagination.com >
4.0 Strange Commit/Replication Issue
Hello all, I am running 4.0 alpha and have encountered something I am unable to explain. I am indexing content to a master server, and the data is replicating to a slave. The odd part is that when searching through the UI, no documents show up on master with a standard *:* query. All cache types are set to zero. I know indexing is working because I am watching the logs and I can see documents getting added, not to mention the data is written to the filesystem. I have autocommit set to 6 (1 minute) so it isn't a commit issue. The very strange part is that the slave is correctly replicating the data, and it is searchable in the UI on the slave (but not master). I don't understand how/why the data is visible on the slave and not visible on the master. Does anyone have any thoughts on this or seen it before? Thanks in advance! Briggs
Re: 4.0 Strange Commit/Replication Issue
That is the problem. I wasn't aware of that new feature in 4.0. Thanks for the quick response Tomás. -Briggs On Wed, Aug 1, 2012 at 3:08 PM, Tomás Fernández Löbbe wrote: > Could your autocommit in the master be using "openSearcher=false"? If you > go to the Master admin, do you see that the searcher has all the segments > that you see in the filesystem? > > > > On Wed, Aug 1, 2012 at 4:24 PM, Briggs Thompson < > w.briggs.thomp...@gmail.com > > wrote: > > > Hello all, > > > > I am running 4.0 alpha and have encountered something I am unable to > > explain. I am indexing content to a master server, and the data is > > replicating to a slave. The odd part is that when searching through the > UI, > > no documents show up on master with a standard *:* query. All cache types > > are set to zero. I know indexing is working because I am watching the > logs > > and I can see documents getting added, not to mention the data is written > > to the filesystem. I have autocommit set to 6 (1 minute) so it isn't > a > > commit issue. > > > > The very strange part is that the slave is correctly replicating the > data, > > and it is searchable in the UI on the slave (but not master). I don't > > understand how/why the data is visible on the slave and not visible on > the > > master. Does anyone have any thoughts on this or seen it before? > > > > Thanks in advance! > > Briggs > > >
Re: SolrJ - IOException
I have also just ran into this a few times over the weekend in a newly
deployed system. We are running Solr 4.0 Beta (not using SolrCloud) and it
is hosted via AWS.
I have a RabbitMQ consumer that reads updates from a queue and posts
updates to Solr via SolrJ. There is quite a bit of error handling around
the indexing request, and even if Solr is not live the consumer application
successfully logs the exception and attempts to move along in the queue.
There are two consumer applications running at once, and at times processes
400 requests per minute. The high volume times is not necessarily when this
problem occurs, though.
This exception is causing the entire application to hang - which is
surprising considering all SolrJ logic is wrapped with try/catches. Has
anyone found out more information regarding the possible keep alive bug?
Any insight is much appreciated.
Thanks,
Briggs Thompson
Oct 8, 2012 7:25:48 AM org.apache.http.impl.client.DefaultRequestDirector
tryExecute
INFO: I/O exception (java.net.SocketException) caught when processing
request: Broken pipe
Oct 8, 2012 7:25:48 AM org.apache.http.impl.client.DefaultRequestDirector
tryExecute
INFO: Retrying request
Oct 8, 2012 7:25:48 AM com.<>.rabbitmq.worker.SolrWriter work
SEVERE: {"id":4049703,"datetime":"2012-10-08 07:22:05"}
IOException occured when talking to server at:
<<http://ec2-50-18-73-42.us-west-1.compute.amazonaws.com:8983/solr/coupon>
server>
org.apache.solr.client.solrj.SolrServerException: IOException occured when
talking to server at:
<<http://ec2-50-18-73-42.us-west-1.compute.amazonaws.com:8983/solr/coupon>
server>
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:362)
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
at
org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:69)
at org.apache.solr.client.solrj.SolrServer.addBeans(SolrServer.java:96)
at org.apache.solr.client.solrj.SolrServer.addBeans(SolrServer.java:79)
at com.<>.solr.SolrIndexService.IndexCoupon(SolrIndexService.java:57)
at com.<>.solr.SolrIndexService.Index(SolrIndexService.java:36)
at com.<>.rabbitmq.worker.SolrWriter.work(SolrWriter.java:47)
at com.<>.rabbitmq.job.Runner.run(Runner.java:84)
at com.<>.rabbitmq.job.SolrConsumer.main(SolrConsumer.java:10)
Caused by: org.apache.http.client.ClientProtocolException
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:909)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:306)
... 10 more
Caused by: org.apache.http.client.NonRepeatableRequestException: Cannot
retry request with a non-repeatable request entity. The cause lists the
reason the original request failed.
at
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:686)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:517)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
... 13 more
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at
org.apache.http.impl.io.AbstractSessionOutputBuffer.flushBuffer(AbstractSessionOutputBuffer.java:147)
at
org.apache.http.impl.io.AbstractSessionOutputBuffer.flush(AbstractSessionOutputBuffer.java:154)
at
org.apache.http.impl.conn.LoggingSessionOutputBuffer.flush(LoggingSessionOutputBuffer.java:95)
at
org.apache.http.impl.io.ChunkedOutputStream.flush(ChunkedOutputStream.java:178)
at
org.apache.http.entity.mime.content.InputStreamBody.writeTo(InputStreamBody.java:72)
at
org.apache.http.entity.mime.HttpMultipart.doWriteTo(HttpMultipart.java:206)
at org.apache.http.entity.mime.HttpMultipart.writeTo(HttpMultipart.java:224)
at
org.apache.http.entity.mime.MultipartEntity.writeTo(MultipartEntity.java:183)
at
org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:98)
at
org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:108)
at
org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:122)
at
org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:271)
at
org.apache.http.impl.conn.AbstractClientConnAdapter.sendRequestEntity(AbstractClientConnAdapter.java:227)
at
org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:257
Re: SolrJ - IOException
Also note there were no exceptions in the actual Solr log, only on the
SolrJ side.
Thanks,
Briggs
On Mon, Oct 8, 2012 at 10:45 AM, Briggs Thompson <
w.briggs.thomp...@gmail.com> wrote:
> I have also just ran into this a few times over the weekend in a newly
> deployed system. We are running Solr 4.0 Beta (not using SolrCloud) and it
> is hosted via AWS.
>
> I have a RabbitMQ consumer that reads updates from a queue and posts
> updates to Solr via SolrJ. There is quite a bit of error handling around
> the indexing request, and even if Solr is not live the consumer application
> successfully logs the exception and attempts to move along in the queue.
> There are two consumer applications running at once, and at times processes
> 400 requests per minute. The high volume times is not necessarily when this
> problem occurs, though.
>
> This exception is causing the entire application to hang - which is
> surprising considering all SolrJ logic is wrapped with try/catches. Has
> anyone found out more information regarding the possible keep alive bug?
> Any insight is much appreciated.
>
> Thanks,
> Briggs Thompson
>
>
> Oct 8, 2012 7:25:48 AM org.apache.http.impl.client.DefaultRequestDirector
> tryExecute
> INFO: I/O exception (java.net.SocketException) caught when processing
> request: Broken pipe
> Oct 8, 2012 7:25:48 AM org.apache.http.impl.client.DefaultRequestDirector
> tryExecute
> INFO: Retrying request
> Oct 8, 2012 7:25:48 AM com.<>.rabbitmq.worker.SolrWriter work
> SEVERE: {"id":4049703,"datetime":"2012-10-08 07:22:05"}
> IOException occured when talking to server at:
> <<http://ec2-50-18-73-42.us-west-1.compute.amazonaws.com:8983/solr/coupon>
> server>
> org.apache.solr.client.solrj.SolrServerException: IOException occured when
> talking to server at:
> <<http://ec2-50-18-73-42.us-west-1.compute.amazonaws.com:8983/solr/coupon>
> server>
> at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:362)
> at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
> at
> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105)
> at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:69)
> at org.apache.solr.client.solrj.SolrServer.addBeans(SolrServer.java:96)
> at org.apache.solr.client.solrj.SolrServer.addBeans(SolrServer.java:79)
> at com.<>.solr.SolrIndexService.IndexCoupon(SolrIndexService.java:57)
> at com.<>.solr.SolrIndexService.Index(SolrIndexService.java:36)
> at com.<>.rabbitmq.worker.SolrWriter.work(SolrWriter.java:47)
> at com.<>.rabbitmq.job.Runner.run(Runner.java:84)
> at com.<>.rabbitmq.job.SolrConsumer.main(SolrConsumer.java:10)
> Caused by: org.apache.http.client.ClientProtocolException
> at
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:909)
> at
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
> at
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
> at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:306)
> ... 10 more
> Caused by: org.apache.http.client.NonRepeatableRequestException: Cannot
> retry request with a non-repeatable request entity. The cause lists the
> reason the original request failed.
> at
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:686)
> at
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:517)
> at
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
> ... 13 more
> Caused by: java.net.SocketException: Broken pipe
> at java.net.SocketOutputStream.socketWrite0(Native Method)
> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
> at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
> at
> org.apache.http.impl.io.AbstractSessionOutputBuffer.flushBuffer(AbstractSessionOutputBuffer.java:147)
> at
> org.apache.http.impl.io.AbstractSessionOutputBuffer.flush(AbstractSessionOutputBuffer.java:154)
> at
> org.apache.http.impl.conn.LoggingSessionOutputBuffer.flush(LoggingSessionOutputBuffer.java:95)
> at
> org.apache.http.impl.io.ChunkedOutputStream.flush(ChunkedOutputStream.java:178)
> at
> org.apache.http.entity.mime.content.InputStreamBody.writeTo(InputStreamBody.java:72)
> at
> org.apache.http.entity.mime.HttpMultipart.doWriteTo(HttpMultipart.java:206)
> at
> org.apache.http.entity.mime.HttpMultipart.writeTo(HttpMultipart.java:224)
> at
> org.apache.http.entity.mime.MultipartEntity.writeTo(MultipartEntity.java:183)
>
Re: SolrJ 4.0 Beta maxConnectionsPerHost
Thanks all for your responses. For some reason the emails were getting filtered out of my inbox. *Otis* - jstack is a great suggestion, thanks! The problem didn't happen this morning but next time it does I will certainly get the dump to see exactly where the app is swimming around. I haven't used StreamingUpdateSolrServer but I will see if that makes a difference. Are there any major drawbacks of going this route? *Sami* - if you are referring to config:maxConnections=200&maxConnectionsPerHost=8, it showed up up in the Solr logs, not the SolrJ logs. The client IS instantiated only once and not for every request. I was curious if this was part of the problem. Do I need to re-instantiate the object for each request made? I figured there would be more overhead if I am re-creating the connection several times when I never really need to shut it down, but at this point the overhead would be minimal though so I will try that. *Hoss* - The reason it seemed the client was creating the log was because the indexer (solr *server*) was more or less dormant for several hours, then I booted up my indexing *client* and the maxConnectionsPerHost tidbit was spit out right away. I was looking for something in the solrconfig and online but didn't find anything. I didn't look for very long so will check it out again. Some very good suggestions here. I appreciate everyones feedback. I will follow up after some experimentation. Thanks, Briggs Thompson On Tue, Oct 9, 2012 at 11:19 AM, Chris Hostetter wrote: > > : I did some digging and experimentation and found something interesting. > : When starting up the application, I see the following in Solr logs: > : Creating new http client, > config:maxConnections=200&maxConnectionsPerHost=8 > ... > : It seems as though the maxConnections and maxConnectionsPerHost are not > : actually getting set. Anyone seen this problem or have an idea how to > : resolve? > > To elaborate on sami's comment... > > If you are seeing this in the logs from your solr *server*, it is unlikey > that it has anything to do with the settings you are making on your solr > *client* this is probably related to the http client created inside > solr for communicating with other solr nodes (ie: replication, solr cloud > distributed updates, solr cloud peersync, etc...). Which is different > from the properties you set on the http client in your solr client > application. > > I believe there is a way to configure the defaults for the internal used > http clients via solrconfig.xml, but off the top of my head i don't > remember what that is. > > > > -Hoss >
Re: SolrJ 4.0 Beta maxConnectionsPerHost
There are other updates that happen on the server that do not fail, so the answer to your question is yes. On Wed, Oct 10, 2012 at 8:12 AM, Sami Siren wrote: > On Wed, Oct 10, 2012 at 12:02 AM, Briggs Thompson > wrote: > > *Sami* > > The client IS > > instantiated only once and not for every request. I was curious if this > was > > part of the problem. Do I need to re-instantiate the object for each > > request made? > > No, it is expensive if you instantiate the client every time. > > When the client seems to be hanging, can you still access the Solr > instance normally and execute updates/searches from other clients? > > -- > Sami Siren >
Re: SolrJ 4.0 Beta maxConnectionsPerHost
They are both SolrJ. What is happening is I have a "batch" indexer application that does a full re-index once per day. I also have an "incremental" indexer that takes items off a queue when they are updated. The problem only happens when both are running at the same time - they also run from the same machine. I am going to dig into this today and see what I find - I didn't get around to it yesterday. Question: I don't seem to see a StreamingUpdateSolrServer object on the 4.0 beta. I did see the ConcurrentUpdateSolrServer - this seems like a similar choice. Is this correct? On Wed, Oct 10, 2012 at 9:43 AM, Sami Siren wrote: > On Wed, Oct 10, 2012 at 5:36 PM, Briggs Thompson > wrote: > > There are other updates that happen on the server that do not fail, so > the > > answer to your question is yes. > > The other updates are using solrj or something else? > > It would be helpful if you could prepare a simple java program that > uses solrj to demonstrate the problem. Based on the available > information it is really difficult try to guess what's happening. > > -- > Sami Siren >
Re: SolrJ 4.0 Beta maxConnectionsPerHost
Thanks for the heads up. I just tested this and you are right. I am making a call to "addBeans" and it succeeds without any issue even when the server is down. That sucks. A big part of this process is reliant on knowing exactly what has made it into the index and what has not, so this a difficult problem to solve when you can't catch exceptions. I was thinking I could execute a ping request first to determine if the Solr server is still operational, but that doesn't help if the updateRequestHandler fails. On Wed, Oct 10, 2012 at 1:48 PM, Shawn Heisey wrote: > On 10/9/2012 3:02 PM, Briggs Thompson wrote: > >> *Otis* - jstack is a great suggestion, thanks! The problem didn't happen >> >> this morning but next time it does I will certainly get the dump to see >> exactly where the app is swimming around. I haven't used >> StreamingUpdateSolrServer >> but I will see if that makes a difference. Are there any major drawbacks >> of >> going this route? >> > > One caveat -- when using the Streaming/Concurrent object, your application > will not be notified when there is a problem indexing. I've been told there > is a way to override a method in the object to allow trapping errors, but I > have not seen sample code and haven't figured out how to do it. I've filed > an issue and a patch to fix this. It's received some comments, but so far > nobody has decided to commit it. > > https://issues.apache.org/**jira/browse/SOLR-3284<https://issues.apache.org/jira/browse/SOLR-3284> > > Thanks, > Shawn > >
