Phrase Query Issue
Hi, I am implementing phrase search query using my application. I am trying to search "Glorious Revolution", I am expecting all the records which contains this as whole string in any of its field should display. Below is my query: ?q=fieldname:"Glorious Revolution"&qt=dismaxrequest I am passing this to my solr search engine using httpclient. But it is thrwoing me Invaild Query exception. If I append double quotes with backslash, it passes the query but it doesnt shows the right values. Instead it creates the query like: isbn13_product_s:\"Glorious Revolution\" Please suggest how I should tackle this. If I give the phrase search in my solr console, it returns me the correct results. Thanks, Amit Garg -- View this message in context: http://www.nabble.com/Phrase-Query-Issue-tp22863529p22863529.html Sent from the Solr - User mailing list archive at Nabble.com.
filter query question
Consider, I have following 3 fields I want to query all documents where name:somevalue and actionuser value is not equal to creationuser value. Can we do this??? -- View this message in context: http://www.nabble.com/filter-query-question-tp22863789p22863789.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Additive filter queries
>I have a design question for all of those who might be willing to provide an >answer. > >We are looking for a way to do a type of additive filters. Our documents >are comprised of a single item of a specified color. We will use shoes as >an example. Each document contains a multivalued ³size² field with all >sizes and a multivalued ³width² field for all widths available for a given >color. Our issue is that the values are not linked to each other. This >issue can be seen when a user chooses a size (e.g. 7) and we filter the >options down to only size 7. When the width facet is displayed it will have >all widths available for all documents that match on size 7 even though most >don¹t come in a wide width. We are looking for strategies to filter facets >based on other facets in separate queries. > >-- >Jeff Newburn >Software Engineer, Zappos.com >jnewb...@zappos.com - 702-943-7562 Ditto! As best I understand, you somehow need to arrange for each different combination of colour, size and width to be indexed as a separate sol document. -- === Fergus McMenemie Email:fer...@twig.me.uk Techmore Ltd Phone:(UK) 07721 376021 Unix/Mac/Intranets Analyst Programmer ===
Re: filter query question
On Fri, Apr 3, 2009 at 1:32 PM, Ashish P wrote: > > I want to query all documents where name:somevalue and actionuser value is > not equal to creationuser value. > > Can we do this??? Nope. But you can create a new field which holds true if actionuser != creationuser and filter on that. -- Regards, Shalin Shekhar Mangar.
Re: Phrase Query Issue
On Fri, Apr 3, 2009 at 1:11 PM, dabboo wrote: > > I am passing this to my solr search engine using httpclient. But it is > thrwoing me Invaild Query exception. > I suggest that you use the Solrj client. It will save a you a lot of effort. http://wiki.apache.org/solr/Solrj -- Regards, Shalin Shekhar Mangar.
Re: filter query question
Thanks Shalin. Another question what is the meaning of this syntax [* TO *] Thanks, Ashish Shalin Shekhar Mangar wrote: > > On Fri, Apr 3, 2009 at 1:32 PM, Ashish P wrote: > >> >> I want to query all documents where name:somevalue and actionuser value >> is >> not equal to creationuser value. >> >> Can we do this??? > > > Nope. But you can create a new field which holds true if actionuser != > creationuser and filter on that. > > -- > Regards, > Shalin Shekhar Mangar. > > -- View this message in context: http://www.nabble.com/filter-query-question-tp22863789p22864255.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: filter query question
On Fri, Apr 3, 2009 at 2:04 PM, Ashish P wrote: > > Another question what is the meaning of this syntax > [* TO *] > > It means match all tokens from the beginning to the end. -- Regards, Shalin Shekhar Mangar.
Re: Oracle Clob column with DIH does not turn to String
Noble, I put in a few 'System.out.println' statements in the ClobTransformer.java file & remade the war. But I see none of these prints coming up in my 'catalina.out' file. Is that the right file to be looking at? As an aside, is 'catalina.out' the ONLY log file for SOLR? I turned on the logging to 'FINE' for everything. Also, these settings seem to go away when Tomcat is restarted. - ashok Noble Paul നോബിള് नोब्ळ् wrote: > > yeah, ant dist will give you the .war file you may need . just drop it > in and you are set to go. or if you can hook up a debugger to a > running Solr that is the easiest > --Noble > > On Fri, Apr 3, 2009 at 9:35 AM, ashokc wrote: >> >> That would require me to recompile (with ant/maven scripts?) the source >> and >> replace the jar for DIH, right? I can try - for the first time. >> - ashok >> >> Noble Paul നോബിള് नोब्ळ् wrote: >>> >>> This looks strange. Apparently the Transformer did not get applied. Is >>> it possible for you to debug ClobTransformer adding(System.out.println >>> into ClobTransformer may help) >>> >>> On Fri, Apr 3, 2009 at 6:04 AM, ashokc wrote: Correcting my earlier post. It lost some lines some how. Hi, I have set up to import some oracle clob columns with DIH. I am using the latest nightly release. My config says, >>> ... But it does not seem to turn this clob into a String. The search results show: 1.8670129 oracle.sql.c...@aed3a5 4486 Any pointers on why I do not get the 'string' out of the clob for indexing? Is the nightly war NOT the right one to use? Thanks for your help. - ashok ashokc wrote: > > Hi, > > I have set up to import some oracle clob columns with DIH. I am using > the > latest nightly release. My config says, > > column="description" clob="true" /> > > > > > But it does not seem to turn this clob into a String. The search > results > show: > > > 1.8670129 > oracle.sql.c...@aed3a5 > 4486 > > > Any pointers on why I do not get the 'string' out of the clob for > indexing? Is the nightly war NOT the right one to use? > > Thanks for your help. > > - ashok > > > -- View this message in context: http://www.nabble.com/Oracle-Clob-column-with-DIH-does-not-turn-to-String-tp22859837p22859865.html Sent from the Solr - User mailing list archive at Nabble.com. >>> >>> >>> >>> -- >>> --Noble Paul >>> >>> >> >> -- >> View this message in context: >> http://www.nabble.com/Oracle-Clob-column-with-DIH-does-not-turn-to-String-tp22859837p22861630.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > > > -- > --Noble Paul > > -- View this message in context: http://www.nabble.com/Oracle-Clob-column-with-DIH-does-not-turn-to-String-tp22859837p22867161.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: Remote Access To Schema Data
Thanks, Jeff. Luke will work , but it's a GUI - unless they expose an API that I can call. What I need is a way to get the informatiom as a client call to a SolrJ SolrServer object. We want to be able to get this info whether we are running Solr embedded or as a Web service. I guess I could write a request handler to return this info, but I like to see what's the "right" way to do this. Clay -Original Message- From: Jeff Newburn [mailto:jnewb...@zappos.com] Sent: Thursday, April 02, 2009 6:14 PM To: solr-user@lucene.apache.org Subject: Re: Remote Access To Schema Data Fastest way I know of to get the schema is using the luke browser. http://localhost/solr/admin/luke It returns in xml and has tons of info you probably aren't interested it. However, it does contain information like fields and type. -- Jeff Newburn Software Engineer, Zappos.com jnewb...@zappos.com - 702-943-7562 > From: "Fink, Clayton R." > Reply-To: > Date: Thu, 2 Apr 2009 17:29:38 -0400 > To: "solr-user@lucene.apache.org" > Subject: Remote Access To Schema Data > > Hi: > > I want to get a list of the fields and field types for an index deployed on a > Solr server (over HTTP or embedded). I can't see any obvious way to do this as > a client. > > This is part of the use case for an app we are working on where all field > information for an index is available and we can programmatically format > updates and queries based on the available fields. > > Thanks, > > Clay Fink >
Re: Remote Access To Schema Data
On Fri, Apr 3, 2009 at 6:11 PM, Fink, Clayton R. wrote: > Thanks, Jeff. > > Luke will work , but it's a GUI - unless they expose an API that I can > call. What I need is a way to get the informatiom as a client call to a > SolrJ SolrServer object. We want to be able to get this info whether we are > running Solr embedded or as a Web service. I guess I could write a request > handler to return this info, but I like to see what's the "right" way to do > this. > Hit /admin/luke to get luke's output as an XML which you can parse. You can also hit /admin/file/?file=schema.xml. If you are using Solrj, you'd need to set qt=/admin/luke or qt=/admin/file for this to work. Note that the luke handler gives out a lot of information like term frequency and therefore takes a longer time to execute. -- Regards, Shalin Shekhar Mangar.
Re: Phrase Query Issue
Maybe you can have to escape quotes with the % folowed by the exa code of the quote like is done by the php urlencode fonction. "Returns a string in which all non-alphanumeric characters except /-_./ have been replaced with a percent (/%/) sign followed by two hex digits and spaces encoded as plus (/+/) signs." Nicolas. dabboo a écrit : Hi, I am implementing phrase search query using my application. I am trying to search "Glorious Revolution", I am expecting all the records which contains this as whole string in any of its field should display. Below is my query: ?q=fieldname:"Glorious Revolution"&qt=dismaxrequest I am passing this to my solr search engine using httpclient. But it is thrwoing me Invaild Query exception. If I append double quotes with backslash, it passes the query but it doesnt shows the right values. Instead it creates the query like: isbn13_product_s:\"Glorious Revolution\" Please suggest how I should tackle this. If I give the phrase search in my solr console, it returns me the correct results. Thanks, Amit Garg
Re: Facets drill down
On Apr 2, 2009, at 3:07 PM, revas wrote: Hi, I typically issue a facetdrill down query thus q=somequery and Facetfield:facetval . Is there any issues with the above approach as opposed to &fq=facetfield:value in terms of memory consumption and the use of cache. Yes... using fq allows those queries to be cached independently into the filter cache. Combining clauses like you show avoids the cache, and also the Facetfield:facetval clause is used in scoring which is typically unnecessary/undesirable. Erik
Re: Phrase Query Issue
Also note that the dismax parser does NOT support fielded clauses, so fieldname:"some phrase" is not parsed as you might expect with dismax. Erik On Apr 3, 2009, at 3:41 AM, dabboo wrote: Hi, I am implementing phrase search query using my application. I am trying to search "Glorious Revolution", I am expecting all the records which contains this as whole string in any of its field should display. Below is my query: ?q=fieldname:"Glorious Revolution"&qt=dismaxrequest I am passing this to my solr search engine using httpclient. But it is thrwoing me Invaild Query exception. If I append double quotes with backslash, it passes the query but it doesnt shows the right values. Instead it creates the query like: isbn13_product_s:\"Glorious Revolution\" Please suggest how I should tackle this. If I give the phrase search in my solr console, it returns me the correct results. Thanks, Amit Garg -- View this message in context: http://www.nabble.com/Phrase-Query-Issue-tp22863529p22863529.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Remote Access To Schema Data
On Apr 3, 2009, at 9:26 AM, Shalin Shekhar Mangar wrote: Note that the luke handler gives out a lot of information like term frequency and therefore takes a longer time to execute. It's fast if you say &numTerms=0 though, which is good enough to get field/type info. Erik
Re: Remote Access To Schema Data
On 4/3/09, Erik Hatcher wrote: > > On Apr 3, 2009, at 9:26 AM, Shalin Shekhar Mangar wrote: >> Note that the luke handler gives out a lot of information like term >> frequency and therefore takes a longer time to execute. > > It's fast if you say &numTerms=0 though, which is good enough to get > field/type info. Nice. I didn't know that. Thanks Erik. -- Regards, Shalin Shekhar Mangar.
Re: crazy parentheses
On Fri, Apr 3, 2009 at 9:56 AM, Dean Missikowski (Consultant), CLSA < dean.missikow...@clsa.com> wrote: > I've got a problem that's driving me crazy with parentheses. > > I'm using a recent nightly Solr 1.4 > > My index includes these three docs. > > doc #1 has title: "saints & sinners" > > doc #2 has title: "(saints and sinners)" > > doc #3 has title: "( saints & sinners )" > > doc #4 has title: "(saints & sinners)" > > > when I try any of these searches: > > title:saints & sinners > > title:"saints & sinners" > > title:saints and sinners > > > > Only docs #1-3 are found, but doc #4 should match too? > > > > The analyzer shows that the tokenizer and filters should find a match. > > I'm guessing this might be a bug with WordDelimiterFactory? > > I just tried indexing "(saints & sinners)" into a field and tried searching by "saints & sinners" and I got a match. The type definition in schema.xml that I used for testing was: -- Regards, Shalin Shekhar Mangar.
Re: DIH Date conversion from a source column skews time
Okay, I will give that a try.
I could resolve this any other day by being able to execute the same XPATH
retrieval twice. Why does the following not work:
When I do this, only the second_date_s will make it into the index. I know
first_date_d instruction is valid but, it just disappears.
Any thoughts?
On 4/1/09 11:59 PM, "Noble Paul നോബിള് नोब्ळ्"
wrote:
> I guess dateFormat does the job properly but the returned value is
> changed according to timezone.
>
> can y try this out add an extra field which converts the date to toString()
>
> template="${.original_air_date_d}"/>
> this would add an extra field as string to the index
>
>
>
> On Wed, Apr 1, 2009 at 11:31 PM, Wesley Small
> wrote:
>> Was there any follow up to this issue I found? Is this a legitimate bug
>> with the time of day changing?
>>
>> I could try to solve this by executing same xpath statement twice.
>>
>> > xpath="/add/doc/fie...@name='original_air_date_d']" />
>>
>> > xpath="/add/doc/fie...@name='original_air_date_d']" />
>>
>> However, when I do that, the first field original_air_date_d does not make
>> it into the index. Is seems that you cannot have two identical xpath
>> statements in the data input config file. Is this by design?
>>
>>
>> On 4/1/09 7:45 AM, "Small, Wesley" wrote:
>>
>>> I have noticed that setting a dynamic date field from source column changes
>>> the time within the date. Can anyone confirm this?
>>>
>>> For example, the document I import has the following xml field.
>>>
>>> 2002-12-18T00:00:00Z
>>>
>>>
>>> In my data-inport-config file I define the following instructions:
>>>
>>> >> xpath="/add/doc/fie...@name='original_air_date_d']" />
>>>
>>> >> sourceColName="temp_original_air_date_s" regex="([0-9][0-9][0-9][0-9])[-
>>> /.][0-9][0-9][- /.][0-
>>> 9][0-9][T][0-9][0-9][:][0-9][0-9][:][0-9][0-9][Z]" replaceWith="$1" />
>>>
>>> >> dateTimeFormat="-MM-dd'T'HH:mm:ss'Z'"/>
>>>
>>>
>>> What is set in my index is is the following:
>>>
>>>
>>> 2002-12-18T00:00:00Z
>>>
>>>
>>>
>>> 2002
>>>
>>>
>>>
>>> 2002-12-18T05:00:00Z
>>>
>>>
>>> You'll notice that the hour (HH) in original_air_date_d changes is set to
>>> 05. It should still be 00. I have noticed that it changes to either 04 or
>>> 05 in all cases within my index.
>>>
>>> In my schema the dynamic field "*_d"
>>>
>>>
>>> Thanks,
>>> Wesley.
>>>
>>>
>>
>>
>
>
>
> --
> --Noble Paul
>
Re: Additive filter queries
We could do that by going down one level in our inventory, but then we have other problems.. for example: Right now a document looks like this: 1598548 12545 Adidas 1, 2, 3, 4, 5, 6, 7 AA, A, B, W, W, Brown If we went down a level, it could look like.. 1598548 12545 654641654684 Adidas 1 AA Brown The question now is this: - At the stock level, we don't want a search for "brown shoes" to return with al the various size/width combos as separate results - each productId / styleId combo should be a single result - At the stock level, if you filter by "Size: 7" and then "Width: B" you're assured to only get things that are width B and size 7 - At the style level, we can't tell for sure which size / width combos are in stock, since this data is not exposed to solr This seems like a problem that isn't unique to us. Any store that has size/width or anything like that will have the same issue. How might it be solved? Thanks for your time! Matthew Runo Software Engineer, Zappos.com mr...@zappos.com - 702-943-7833 On Apr 3, 2009, at 1:13 AM, Fergus McMenemie wrote: I have a design question for all of those who might be willing to provide an answer. We are looking for a way to do a type of additive filters. Our documents are comprised of a single item of a specified color. We will use shoes as an example. Each document contains a multivalued ³size² field with all sizes and a multivalued ³width² field for all widths available for a given color. Our issue is that the values are not linked to each other. This issue can be seen when a user chooses a size (e.g. 7) and we filter the options down to only size 7. When the width facet is displayed it will have all widths available for all documents that match on size 7 even though most don¹t come in a wide width. We are looking for strategies to filter facets based on other facets in separate queries. -- Jeff Newburn Software Engineer, Zappos.com jnewb...@zappos.com - 702-943-7562 Ditto! As best I understand, you somehow need to arrange for each different combination of colour, size and width to be indexed as a separate sol document. -- === Fergus McMenemie Email:fer...@twig.me.uk Techmore Ltd Phone:(UK) 07721 376021 Unix/Mac/Intranets Analyst Programmer ===
Re: DIH Date conversion from a source column skews time
I same XPATH does not work . but what is stopping you from copying
one field to another using a Template
On Fri, Apr 3, 2009 at 8:39 PM, Wesley Small wrote:
> Okay, I will give that a try.
>
> I could resolve this any other day by being able to execute the same XPATH
> retrieval twice. Why does the following not work:
>
> xpath="/add/doc/fie...@name='original_air_date_d']" />
> xpath="/add/doc/fie...@name='original_air_date_d']" />
>
> When I do this, only the second_date_s will make it into the index. I know
> first_date_d instruction is valid but, it just disappears.
>
> Any thoughts?
>
> On 4/1/09 11:59 PM, "Noble Paul നോബിള് नोब्ळ्"
> wrote:
>
>> I guess dateFormat does the job properly but the returned value is
>> changed according to timezone.
>>
>> can y try this out add an extra field which converts the date to toString()
>>
>> > template="${.original_air_date_d}"/>
>> this would add an extra field as string to the index
>>
>>
>>
>> On Wed, Apr 1, 2009 at 11:31 PM, Wesley Small
>> wrote:
>>> Was there any follow up to this issue I found? Is this a legitimate bug
>>> with the time of day changing?
>>>
>>> I could try to solve this by executing same xpath statement twice.
>>>
>>> >> xpath="/add/doc/fie...@name='original_air_date_d']" />
>>>
>>> >> xpath="/add/doc/fie...@name='original_air_date_d']" />
>>>
>>> However, when I do that, the first field original_air_date_d does not make
>>> it into the index. Is seems that you cannot have two identical xpath
>>> statements in the data input config file. Is this by design?
>>>
>>>
>>> On 4/1/09 7:45 AM, "Small, Wesley" wrote:
>>>
I have noticed that setting a dynamic date field from source column changes
the time within the date. Can anyone confirm this?
For example, the document I import has the following xml field.
2002-12-18T00:00:00Z
In my data-inport-config file I define the following instructions:
>>> xpath="/add/doc/fie...@name='original_air_date_d']" />
>>> sourceColName="temp_original_air_date_s" regex="([0-9][0-9][0-9][0-9])[-
/.][0-9][0-9][- /.][0-
9][0-9][T][0-9][0-9][:][0-9][0-9][:][0-9][0-9][Z]" replaceWith="$1" />
>>> sourceColName="temp_original_air_date_s"
dateTimeFormat="-MM-dd'T'HH:mm:ss'Z'"/>
What is set in my index is is the following:
2002-12-18T00:00:00Z
2002
2002-12-18T05:00:00Z
You'll notice that the hour (HH) in original_air_date_d changes is set to
05. It should still be 00. I have noticed that it changes to either 04 or
05 in all cases within my index.
In my schema the dynamic field "*_d"
Thanks,
Wesley.
>>>
>>>
>>
>>
>>
>> --
>> --Noble Paul
>>
>
>
--
--Noble Paul
Re: Oracle Clob column with DIH does not turn to String
There is something else wrong with your setup. can you just paste the whole data-config.xml --Noble On Fri, Apr 3, 2009 at 5:39 PM, ashokc wrote: > > Noble, > I put in a few 'System.out.println' statements in the ClobTransformer.java > file & remade the war. But I see none of these prints coming up in my > 'catalina.out' file. Is that the right file to be looking at? > > As an aside, is 'catalina.out' the ONLY log file for SOLR? I turned on the > logging to 'FINE' for everything. Also, these settings seem to go away when > Tomcat is restarted. > - ashok > > Noble Paul നോബിള് नोब्ळ् wrote: >> >> yeah, ant dist will give you the .war file you may need . just drop it >> in and you are set to go. or if you can hook up a debugger to a >> running Solr that is the easiest >> --Noble >> >> On Fri, Apr 3, 2009 at 9:35 AM, ashokc wrote: >>> >>> That would require me to recompile (with ant/maven scripts?) the source >>> and >>> replace the jar for DIH, right? I can try - for the first time. >>> - ashok >>> >>> Noble Paul നോബിള് नोब्ळ् wrote: This looks strange. Apparently the Transformer did not get applied. Is it possible for you to debug ClobTransformer adding(System.out.println into ClobTransformer may help) On Fri, Apr 3, 2009 at 6:04 AM, ashokc wrote: > > Correcting my earlier post. It lost some lines some how. > > Hi, > > I have set up to import some oracle clob columns with DIH. I am using > the > latest nightly release. My config says, > > > ... > > > > > > > But it does not seem to turn this clob into a String. The search > results > show: > > > 1.8670129 > oracle.sql.c...@aed3a5 > 4486 > > > Any pointers on why I do not get the 'string' out of the clob for > indexing? > Is the nightly war NOT the right one to use? > > Thanks for your help. > > - ashok > > > > ashokc wrote: >> >> Hi, >> >> I have set up to import some oracle clob columns with DIH. I am using >> the >> latest nightly release. My config says, >> >> > column="description" clob="true" /> >> >> >> >> >> But it does not seem to turn this clob into a String. The search >> results >> show: >> >> >> 1.8670129 >> oracle.sql.c...@aed3a5 >> 4486 >> >> >> Any pointers on why I do not get the 'string' out of the clob for >> indexing? Is the nightly war NOT the right one to use? >> >> Thanks for your help. >> >> - ashok >> >> >> > > -- > View this message in context: > http://www.nabble.com/Oracle-Clob-column-with-DIH-does-not-turn-to-String-tp22859837p22859865.html > Sent from the Solr - User mailing list archive at Nabble.com. > > -- --Noble Paul >>> >>> -- >>> View this message in context: >>> http://www.nabble.com/Oracle-Clob-column-with-DIH-does-not-turn-to-String-tp22859837p22861630.html >>> Sent from the Solr - User mailing list archive at Nabble.com. >>> >>> >> >> >> >> -- >> --Noble Paul >> >> > > -- > View this message in context: > http://www.nabble.com/Oracle-Clob-column-with-DIH-does-not-turn-to-String-tp22859837p22867161.html > Sent from the Solr - User mailing list archive at Nabble.com. > > -- --Noble Paul
Re: Oracle Clob column with DIH does not turn to String
Sure: data-config Xml === === A search result on the field short_desc: -- 1.8670129 oracle.sql.c...@155e3ab 4486 Develop Rating functionality for QIN 2009-04-03T11:47:32.635Z Noble Paul നോബിള് नोब्ळ् wrote: > > There is something else wrong with your setup. > > can you just paste the whole data-config.xml > > --Noble > > On Fri, Apr 3, 2009 at 5:39 PM, ashokc wrote: >> >> Noble, >> I put in a few 'System.out.println' statements in the >> ClobTransformer.java >> file & remade the war. But I see none of these prints coming up in my >> 'catalina.out' file. Is that the right file to be looking at? >> >> As an aside, is 'catalina.out' the ONLY log file for SOLR? I turned on >> the >> logging to 'FINE' for everything. Also, these settings seem to go away >> when >> Tomcat is restarted. >> - ashok >> >> Noble Paul നോബിള് नोब्ळ् wrote: >>> >>> yeah, ant dist will give you the .war file you may need . just drop it >>> in and you are set to go. or if you can hook up a debugger to a >>> running Solr that is the easiest >>> --Noble >>> >>> On Fri, Apr 3, 2009 at 9:35 AM, ashokc wrote: That would require me to recompile (with ant/maven scripts?) the source and replace the jar for DIH, right? I can try - for the first time. - ashok Noble Paul നോബിള് नोब्ळ् wrote: > > This looks strange. Apparently the Transformer did not get applied. Is > it possible for you to debug ClobTransformer adding(System.out.println > into ClobTransformer may help) > > On Fri, Apr 3, 2009 at 6:04 AM, ashokc wrote: >> >> Correcting my earlier post. It lost some lines some how. >> >> Hi, >> >> I have set up to import some oracle clob columns with DIH. I am using >> the >> latest nightly release. My config says, >> >> >> > ... >> >> >> >> >> >> >> But it does not seem to turn this clob into a String. The search >> results >> show: >> >> >> 1.8670129 >> oracle.sql.c...@aed3a5 >> 4486 >> >> >> Any pointers on why I do not get the 'string' out of the clob for >> indexing? >> Is the nightly war NOT the right one to use? >> >> Thanks for your help. >> >> - ashok >> >> >> >> ashokc wrote: >>> >>> Hi, >>> >>> I have set up to import some oracle clob columns with DIH. I am >>> using >>> the >>> latest nightly release. My config says, >>> >>> >> column="description" clob="true" /> >>> >>> >>> >>> >>> But it does not seem to turn this clob into a String. The search >>> results >>> show: >>> >>> >>> 1.8670129 >>> oracle.sql.c...@aed3a5 >>> 4486 >>> >>> >>> Any pointers on why I do not get the 'string' out of the clob for >>> indexing? Is the nightly war NOT the right one to use? >>> >>> Thanks for your help. >>> >>> - ashok >>> >>> >>> >> >> -- >> View this message in context: >> http://www.nabble.com/Oracle-Clob-column-with-DIH-does-not-turn-to-String-tp22859837p22859865.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > > > -- > --Noble Paul > > -- View this message in context: http://www.nabble.com/Oracle-Clob-column-with-DIH-does-not-turn-to-String-tp22859837p22861630.html Sent from the Solr - User mailing list archive at Nabble.com. >>> >>> >>> >>> -- >>> --Noble Paul >>> >>> >> >> -- >> View this message in context: >> http://www.nabble.com/Oracle-Clob-column-with-DIH-does-not-turn-to-String-tp22859837p22867161.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > > > -- > --Noble Paul > > -- View this message in context: http://www.nabble.com/Oracle-Clob-column-with-DIH-does-not-turn-to-String-tp22859837p22872184.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Oracle Clob column with DIH does not turn to String
and which version of Solr are u using? On Fri, Apr 3, 2009 at 10:09 PM, ashokc wrote: > > Sure: > > data-config Xml > === > > > url="jdbc:oracle:thin:@x" user="remedy" password="y"/> > > > > > > > > > > === > > A search result on the field short_desc: > -- > > > 1.8670129 > oracle.sql.c...@155e3ab > 4486 > Develop Rating functionality for QIN > 2009-04-03T11:47:32.635Z > > > > > > Noble Paul നോബിള് नोब्ळ् wrote: >> >> There is something else wrong with your setup. >> >> can you just paste the whole data-config.xml >> >> --Noble >> >> On Fri, Apr 3, 2009 at 5:39 PM, ashokc wrote: >>> >>> Noble, >>> I put in a few 'System.out.println' statements in the >>> ClobTransformer.java >>> file & remade the war. But I see none of these prints coming up in my >>> 'catalina.out' file. Is that the right file to be looking at? >>> >>> As an aside, is 'catalina.out' the ONLY log file for SOLR? I turned on >>> the >>> logging to 'FINE' for everything. Also, these settings seem to go away >>> when >>> Tomcat is restarted. >>> - ashok >>> >>> Noble Paul നോബിള് नोब्ळ् wrote: yeah, ant dist will give you the .war file you may need . just drop it in and you are set to go. or if you can hook up a debugger to a running Solr that is the easiest --Noble On Fri, Apr 3, 2009 at 9:35 AM, ashokc wrote: > > That would require me to recompile (with ant/maven scripts?) the source > and > replace the jar for DIH, right? I can try - for the first time. > - ashok > > Noble Paul നോബിള് नोब्ळ् wrote: >> >> This looks strange. Apparently the Transformer did not get applied. Is >> it possible for you to debug ClobTransformer adding(System.out.println >> into ClobTransformer may help) >> >> On Fri, Apr 3, 2009 at 6:04 AM, ashokc wrote: >>> >>> Correcting my earlier post. It lost some lines some how. >>> >>> Hi, >>> >>> I have set up to import some oracle clob columns with DIH. I am using >>> the >>> latest nightly release. My config says, >>> >>> >>> >> ... >>> >>> >>> >>> >>> >>> >>> But it does not seem to turn this clob into a String. The search >>> results >>> show: >>> >>> >>> 1.8670129 >>> oracle.sql.c...@aed3a5 >>> 4486 >>> >>> >>> Any pointers on why I do not get the 'string' out of the clob for >>> indexing? >>> Is the nightly war NOT the right one to use? >>> >>> Thanks for your help. >>> >>> - ashok >>> >>> >>> >>> ashokc wrote: Hi, I have set up to import some oracle clob columns with DIH. I am using the latest nightly release. My config says, >>> column="description" clob="true" /> But it does not seem to turn this clob into a String. The search results show: 1.8670129 oracle.sql.c...@aed3a5 4486 Any pointers on why I do not get the 'string' out of the clob for indexing? Is the nightly war NOT the right one to use? Thanks for your help. - ashok >>> >>> -- >>> View this message in context: >>> http://www.nabble.com/Oracle-Clob-column-with-DIH-does-not-turn-to-String-tp22859837p22859865.html >>> Sent from the Solr - User mailing list archive at Nabble.com. >>> >>> >> >> >> >> -- >> --Noble Paul >> >> > > -- > View this message in context: > http://www.nabble.com/Oracle-Clob-column-with-DIH-does-not-turn-to-String-tp22859837p22861630.html > Sent from the Solr - User mailing list archive at Nabble.com. > > -- --Noble Paul >>> >>> -- >>> View this message in context: >>> http://www.nabble.com/Oracle-Clob-column-with-DIH-does-not-turn-to-String-tp22859837p22867161.html >>> Sent from the Solr - User mailing list archive at Nabble.com. >>> >>> >> >> >> >> -- >> --Noble Paul >> >> > > -- > View this message in context: > http://www.nabble.com/Oracle-Clob-column-with-DIH-does-not-turn-to-String-tp22859837p22872184.html > Sent from the Solr - User mailing list archive at Nabble.com. > > -- --Noble Paul
Re: Oracle Clob column with DIH does not turn to String
I downloaded the nightly build yesterday (2nd April), modified the ClobTransformer.java file with some prints, compiled it all (ant dist). It produced a war file, "apache-solr-1.4-dev.war". That is what I am using. My modification & compilation has not affected the results. I was getting the same behavior with the 'war' that download came with. Thanks Noble. Noble Paul നോബിള് नोब्ळ् wrote: > > and which version of Solr are u using? > > On Fri, Apr 3, 2009 at 10:09 PM, ashokc wrote: >> >> Sure: >> >> data-config Xml >> === >> >> >> > url="jdbc:oracle:thin:@x" user="remedy" password="y"/> >> >> >> >> >> >> >> >> >> >> === >> >> A search result on the field short_desc: >> -- >> >> >> 1.8670129 >> oracle.sql.c...@155e3ab >> 4486 >> Develop Rating functionality for QIN >> 2009-04-03T11:47:32.635Z >> >> >> >> >> >> Noble Paul നോബിള് नोब्ळ् wrote: >>> >>> There is something else wrong with your setup. >>> >>> can you just paste the whole data-config.xml >>> >>> --Noble >>> >>> On Fri, Apr 3, 2009 at 5:39 PM, ashokc wrote: Noble, I put in a few 'System.out.println' statements in the ClobTransformer.java file & remade the war. But I see none of these prints coming up in my 'catalina.out' file. Is that the right file to be looking at? As an aside, is 'catalina.out' the ONLY log file for SOLR? I turned on the logging to 'FINE' for everything. Also, these settings seem to go away when Tomcat is restarted. - ashok Noble Paul നോബിള് नोब्ळ् wrote: > > yeah, ant dist will give you the .war file you may need . just drop it > in and you are set to go. or if you can hook up a debugger to a > running Solr that is the easiest > --Noble > > On Fri, Apr 3, 2009 at 9:35 AM, ashokc wrote: >> >> That would require me to recompile (with ant/maven scripts?) the >> source >> and >> replace the jar for DIH, right? I can try - for the first time. >> - ashok >> >> Noble Paul നോബിള് नोब्ळ् wrote: >>> >>> This looks strange. Apparently the Transformer did not get applied. >>> Is >>> it possible for you to debug ClobTransformer >>> adding(System.out.println >>> into ClobTransformer may help) >>> >>> On Fri, Apr 3, 2009 at 6:04 AM, ashokc wrote: Correcting my earlier post. It lost some lines some how. Hi, I have set up to import some oracle clob columns with DIH. I am using the latest nightly release. My config says, >>> ... But it does not seem to turn this clob into a String. The search results show: 1.8670129 oracle.sql.c...@aed3a5 4486 Any pointers on why I do not get the 'string' out of the clob for indexing? Is the nightly war NOT the right one to use? Thanks for your help. - ashok ashokc wrote: > > Hi, > > I have set up to import some oracle clob columns with DIH. I am > using > the > latest nightly release. My config says, > > column="description" clob="true" /> > > > > > But it does not seem to turn this clob into a String. The search > results > show: > > > 1.8670129 > oracle.sql.c...@aed3a5 > 4486 > > > Any pointers on why I do not get the 'string' out of the clob for > indexing? Is the nightly war NOT the right one to use? > > Thanks for your help. > > - ashok > > > -- View this message in context: http://www.nabble.com/Oracle-Clob-column-with-DIH-does-not-turn-to-String-tp22859837p22859865.html Sent from the Solr - User mailing list archive at Nabble.com. >>> >>> >>> >>> -- >>> --Noble Paul >>> >>> >> >> -- >> View this message in context: >> http://www.nabble.com/Oracle-Clob-column-with-DIH-does-not-turn-to-String-tp22859837p22861630.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > > > -- > --Noble Paul > > -- View this message in context: http://www.nabble.com/Oracle-Clob-column-with-DIH-does-not-turn-to-String-tp22859837p22867161.html Sent from the Solr - User mailing list archive at Nabble.com. >>> >>> >>> >>> -- >>> --Noble Pau
Re: Oracle Clob column with DIH does not turn to String
I guess u can write a custom transformer which gets a String out of the oracle.sql.CLOB. I am just out of clue, why this may happen. I even wrote a testcase and it seems to work fine --Noble On Fri, Apr 3, 2009 at 10:23 PM, ashokc wrote: > > I downloaded the nightly build yesterday (2nd April), modified the > ClobTransformer.java file with some prints, compiled it all (ant dist). It > produced a war file, "apache-solr-1.4-dev.war". That is what I am using. My > modification & compilation has not affected the results. I was getting the > same behavior with the 'war' that download came with. Thanks Noble. > > Noble Paul നോബിള് नोब्ळ् wrote: >> >> and which version of Solr are u using? >> >> On Fri, Apr 3, 2009 at 10:09 PM, ashokc wrote: >>> >>> Sure: >>> >>> data-config Xml >>> === >>> >>> >>> >> url="jdbc:oracle:thin:@x" user="remedy" password="y"/> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> === >>> >>> A search result on the field short_desc: >>> -- >>> >>> >>> 1.8670129 >>> oracle.sql.c...@155e3ab >>> 4486 >>> Develop Rating functionality for QIN >>> 2009-04-03T11:47:32.635Z >>> >>> >>> >>> >>> >>> Noble Paul നോബിള് नोब्ळ् wrote: There is something else wrong with your setup. can you just paste the whole data-config.xml --Noble On Fri, Apr 3, 2009 at 5:39 PM, ashokc wrote: > > Noble, > I put in a few 'System.out.println' statements in the > ClobTransformer.java > file & remade the war. But I see none of these prints coming up in my > 'catalina.out' file. Is that the right file to be looking at? > > As an aside, is 'catalina.out' the ONLY log file for SOLR? I turned on > the > logging to 'FINE' for everything. Also, these settings seem to go away > when > Tomcat is restarted. > - ashok > > Noble Paul നോബിള് नोब्ळ् wrote: >> >> yeah, ant dist will give you the .war file you may need . just drop it >> in and you are set to go. or if you can hook up a debugger to a >> running Solr that is the easiest >> --Noble >> >> On Fri, Apr 3, 2009 at 9:35 AM, ashokc wrote: >>> >>> That would require me to recompile (with ant/maven scripts?) the >>> source >>> and >>> replace the jar for DIH, right? I can try - for the first time. >>> - ashok >>> >>> Noble Paul നോബിള് नोब्ळ् wrote: This looks strange. Apparently the Transformer did not get applied. Is it possible for you to debug ClobTransformer adding(System.out.println into ClobTransformer may help) On Fri, Apr 3, 2009 at 6:04 AM, ashokc wrote: > > Correcting my earlier post. It lost some lines some how. > > Hi, > > I have set up to import some oracle clob columns with DIH. I am > using > the > latest nightly release. My config says, > > > ... > > > > > > > But it does not seem to turn this clob into a String. The search > results > show: > > > 1.8670129 > oracle.sql.c...@aed3a5 > 4486 > > > Any pointers on why I do not get the 'string' out of the clob for > indexing? > Is the nightly war NOT the right one to use? > > Thanks for your help. > > - ashok > > > > ashokc wrote: >> >> Hi, >> >> I have set up to import some oracle clob columns with DIH. I am >> using >> the >> latest nightly release. My config says, >> >> > > column="description" clob="true" /> >> >> >> >> >> But it does not seem to turn this clob into a String. The search >> results >> show: >> >> >> 1.8670129 >> oracle.sql.c...@aed3a5 >> 4486 >> >> >> Any pointers on why I do not get the 'string' out of the clob for >> indexing? Is the nightly war NOT the right one to use? >> >> Thanks for your help. >> >> - ashok >> >> >> > > -- > View this message in context: > http://www.nabble.com/Oracle-Clob-column-with-DIH-does-not-turn-to-String-tp22859837p22859865.html > Sent from the Solr - User mailing list archive at Nabble.com. > > -- --Noble Paul >>> >>> -- >>> View this message in context: >>> http://www.nabble.com/Oracle-Clob-column-with-DIH-does-not-turn-to-String-
Re: Spelling Component
Otis, Thanks for getting back...I did have to restart for the schema/config changes to take effect which I'd guessed. : ) I am using the SpellCheckComponent and following the directions on the wiki. It says to do a set it up using the component AND the RequestHandler. > http://wiki.apache.org/solr/SpellCheckComponent I still don't get any suggestions when I do /spellCheckCompRH?q=helultrashar&spellcheck=true&spellcheck.collate=true What I do get is: 0 30 I see the current time stamp when I ask to do a build so something is happening...just not sure why I'm not getting the proper result. When I add &spellcheck.extendedResults=true to the URL I get a response: 0 2 true Any ideas folks? ~ Otis Gospodnetic wrote: > Hi, > > The $SOLR_URL/solr/biblio/ looks suspicious. Should that be $SOLR_URL/solr/ ? > Also note that you are using the old SpellCheck RequestHandler, and we not > have a SpellCheckComponent - http://wiki.apache.org/solr/SpellCheckComponent > > Otis > -- > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > > > > - Original Message >> From: Anoop Atre >> To: "solr-user@lucene.apache.org" >> Sent: Monday, March 30, 2009 2:58:58 PM >> Subject: Spelling Component >> >> Hi all, I'm new to Solr and have been reading up as much as possible, >> especially about implementing the spellcheck feature. >> >> The instructions I'm looking at are on the following links: >>> http://wiki.apache.org/solr/SpellCheckComponent >>> http://snipurl.com/elxmr [vufind tech list] >> Error I get when I try to build the index using >> $SOLR_URL/solr/biblio/spellCheckCompRH?q=*:*&spellcheck.q=hell&spellcheck=true&spellcheck.build=true >> >> NOT_FOUND >> RequestURI=/solr/biblio/spellCheckCompRH >> Powered by Jetty://
Re: Spelling Component
On Sat, Apr 4, 2009 at 12:01 AM, Anoop Atre wrote: > I still don't get any suggestions when I do > /spellCheckCompRH?q=helultrashar&spellcheck=true&spellcheck.collate=true > > Did you build the spellcheck index? Try specifying a correct word which you know is in the index. See if spellchecker returns it. If it does, then it might be that no suggestions are available or there are no suggestions above the configured threshold. -- Regards, Shalin Shekhar Mangar.
Re: Oracle Clob column with DIH does not turn to String
Happy to report that it is working. Looks like we have to use UPPER CASE for all the column names. When I examined the map 'aRow', it had the column names in upper case, where as my config had lower case. No match was found so nothing happened. Changed my config and it works now. Thanks for your help. Perhaps this transformer can be modified to be case-insensitive for the column names. If you had written it perhaps it is a quick change for you? Noble Paul നോബിള് नोब्ळ् wrote: > > I guess u can write a custom transformer which gets a String out of > the oracle.sql.CLOB. I am just out of clue, why this may happen. I > even wrote a testcase and it seems to work fine > --Noble > > On Fri, Apr 3, 2009 at 10:23 PM, ashokc wrote: >> >> I downloaded the nightly build yesterday (2nd April), modified the >> ClobTransformer.java file with some prints, compiled it all (ant dist). >> It >> produced a war file, "apache-solr-1.4-dev.war". That is what I am using. >> My >> modification & compilation has not affected the results. I was getting >> the >> same behavior with the 'war' that download came with. Thanks Noble. >> >> Noble Paul നോബിള് नोब्ळ् wrote: >>> >>> and which version of Solr are u using? >>> >>> On Fri, Apr 3, 2009 at 10:09 PM, ashokc wrote: Sure: data-config Xml === >>> url="jdbc:oracle:thin:@x" user="remedy" password="y"/> >>> query="SELECT mylog_ato, name_char, dsc FROM log_tbl"> === A search result on the field short_desc: -- 1.8670129 oracle.sql.c...@155e3ab 4486 Develop Rating functionality for QIN 2009-04-03T11:47:32.635Z Noble Paul നോബിള് नोब्ळ् wrote: > > There is something else wrong with your setup. > > can you just paste the whole data-config.xml > > --Noble > > On Fri, Apr 3, 2009 at 5:39 PM, ashokc wrote: >> >> Noble, >> I put in a few 'System.out.println' statements in the >> ClobTransformer.java >> file & remade the war. But I see none of these prints coming up in my >> 'catalina.out' file. Is that the right file to be looking at? >> >> As an aside, is 'catalina.out' the ONLY log file for SOLR? I turned >> on >> the >> logging to 'FINE' for everything. Also, these settings seem to go >> away >> when >> Tomcat is restarted. >> - ashok >> >> Noble Paul നോബിള് नोब्ळ् wrote: >>> >>> yeah, ant dist will give you the .war file you may need . just drop >>> it >>> in and you are set to go. or if you can hook up a debugger to a >>> running Solr that is the easiest >>> --Noble >>> >>> On Fri, Apr 3, 2009 at 9:35 AM, ashokc wrote: That would require me to recompile (with ant/maven scripts?) the source and replace the jar for DIH, right? I can try - for the first time. - ashok Noble Paul നോബിള് नोब्ळ् wrote: > > This looks strange. Apparently the Transformer did not get > applied. > Is > it possible for you to debug ClobTransformer > adding(System.out.println > into ClobTransformer may help) > > On Fri, Apr 3, 2009 at 6:04 AM, ashokc > wrote: >> >> Correcting my earlier post. It lost some lines some how. >> >> Hi, >> >> I have set up to import some oracle clob columns with DIH. I am >> using >> the >> latest nightly release. My config says, >> >> >> > ... >> >> >> >> >> >> >> But it does not seem to turn this clob into a String. The search >> results >> show: >> >> >> 1.8670129 >> oracle.sql.c...@aed3a5 >> 4486 >> >> >> Any pointers on why I do not get the 'string' out of the clob for >> indexing? >> Is the nightly war NOT the right one to use? >> >> Thanks for your help. >> >> - ashok >> >> >> >> ashokc wrote: >>> >>> Hi, >>> >>> I have set up to import some oracle clob columns with DIH. I am >>> using >>> the >>> latest nightly release. My config says, >>> >>> >> >> column="description" clob="true" /> >>> >>> >>> >>> >>> But it does not seem to turn this clob into a String. The search >>> results >>> show: >>> >>> >>> 1.8670129 >>> oracle.sql.c...@aed3a5
Re: Spelling Component
Shalin, I "think" I did build the spellcheck index, I made the changes to solrconfig and schema, restarted, passed a &spellcheck.build=true which created the index. ls -ltr ./spellchecker -rw-r--r-- 1 XXX users 20 2009-04-03 13:23 segments.gen -rw-r--r-- 1 XXX users 28 2009-04-03 13:23 segments_f Hmm...how would I know if the word is in the index? As for the threshold do you mean reduce the 0.7 entry in solrconfig? Thanks! Shalin Shekhar Mangar wrote: > On Sat, Apr 4, 2009 at 12:01 AM, Anoop Atre wrote: > >> I still don't get any suggestions when I do >> /spellCheckCompRH?q=helultrashar&spellcheck=true&spellcheck.collate=true >> >> > Did you build the spellcheck index? Try specifying a correct word which you > know is in the index. See if spellchecker returns it. If it does, then it > might be that no suggestions are available or there are no suggestions above > the configured threshold.
Searching on mulit-core Solr
Hi, I've a multi-core system (one core per day), so there would be around 30 cores in a month on a box running one Solr instance. We have two boxes running the Solr instance and input data is feeded to them in round-robin fashion. Each box can have up to 30 cores in a month. Here are questions, 1) How would I search for a term in multiple cores on same box? Single core I'm able to search like, http://localhost:8080/solr/20090402/select?q=*:* 2) How would I search for a term in multiple cores on both boxes at the same time? 3) Is it possible to have two Solr instances on one box with one doing the indexing and other perform only searches on that index? The idea is have two JVMs with each doing its own task - I'm not sure whether the indexer process needs to know about searcher process - like do they need to have the same solr.xml (for multicore etc). We don't want to replicate the indexes also (we got very light search traffic, but very high indexing traffic) so they need to use the same index. Thanks, -vivek
Solr 1.4 (nightly build) seem hung under load
Hi, I'm using Solr 1.4 (nightly build - 03/29/09). I'm stress testing my application with Solr. My app uses Solrj to write to remote Solr (on same box, but different JVM). The stress test sends over 2 million records (1 record = 500 bytes, with each record having 10 fields) within 5 minutes. All was working fine (with 2 million records processed - 2G index size) and all the sudden Solr stopped responding - I call server.addBeans(...) passing 15K object and don't get any response for over an hour (usually it returns in 5 sec). I've 3 threads writing to the same index at the same time - not sure if that could cause any problem. I was told by Otis that it should be ok to have multiple threads write to same index - so I'm assuming it's ok, though from thread dump I do see couple of "update" threads waiting on ReadWriteLock and another thread (pool-6-thread-1) have a lock on SolrWriter. Attached is the thread dump of the Tomcat process where Solr is running. Any ideas? Thanks, -vivek
Re: Solr 1.4 (nightly build) seem hung under load
Just an update on this issue, the Solr did come back after 80 min - so not sure where was it stuck. I do use RAMBuffer of 64MB and have heap size of 6G. There is no error is Solr log and I'd it running under WARNING level so missed the INFO if there was any during that period. I'm also not running any "optimize" command. What could cause Solr to hang for 80 min? Thanks, -vivek On Fri, Apr 3, 2009 at 1:55 PM, vivek sar wrote: > Hi, > > I'm using Solr 1.4 (nightly build - 03/29/09). I'm stress testing my > application with Solr. My app uses Solrj to write to remote Solr (on > same box, but different JVM). The stress test sends over 2 million > records (1 record = 500 bytes, with each record having 10 fields) > within 5 minutes. All was working fine (with 2 million records > processed - 2G index size) and all the sudden Solr stopped responding > - I call server.addBeans(...) passing 15K object and don't get any > response for over an hour (usually it returns in 5 sec). > > I've 3 threads writing to the same index at the same time - not sure > if that could cause any problem. I was told by Otis that it should be > ok to have multiple threads write to same index - so I'm assuming it's > ok, though from thread dump I do see couple of "update" threads > waiting on ReadWriteLock and another thread (pool-6-thread-1) have a > lock on SolrWriter. > > Attached is the thread dump of the Tomcat process where Solr is > running. Any ideas? > > Thanks, > -vivek >
Re: caching
I don't really understand your question. what do you mean by a "default query" ? as long as you have the caches that existing the example configs, then solr with cache queries and filters for you. and as long as you have a non-zero autowarm count for those caches, solr will use that number of keys from the "old" cache to pre-populate the "new" cache whenever there is a commit. : If i don't explicity set any default query in the solrconfig.xml for : caching and make use of the default config file,does solr do the caching : automatically based on the query? -Hoss
Re: Solr 1.4 (nightly build) seem hung under load
Hi,
more update. It happened again and this time I'd INFO logged in the Solr log,
INFO: {add=[330274716, 330274717, 330274718, 330274719, 330274720,
330274721, 330274722, 330274723, ...(14992 more)]} 0 6041
Apr 3, 2009 10:38:01 PM org.apache.solr.core.SolrCore execute
INFO: [20090403] webapp=/solr path=/update params={wt=javabin}
status=0 QTime=6041
Apr 3, 2009 10:38:11 PM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: start commit(optimize=false,waitFlush=true,waitSearcher=true)
It's still hung at commit even after 30 min. So, looks like it takes
a long time to commit the records. I'm committing the records myself,
but have the auto-commit turned on in the solrconfig.xml,
1000
100
In 15 min time period I'm getting approximately 6 million
documents/records. Earlier I've read in the mailing list that we
shouldn't be committing very often and now it seems not committing on
time makes the commit process take forever. I want the records
searchable every 30 min basically. So, 30 min old data is ok for
searching, but indexing shouldn't slow down.
1) So, what's the good commit strategy?
2) How often (on how many records) should I do this?
3) Should I do it programmatically or can I have it in the solrconfig.xml?
Thanks,
-vivek
On Fri, Apr 3, 2009 at 2:27 PM, vivek sar wrote:
> Just an update on this issue, the Solr did come back after 80 min - so
> not sure where was it stuck. I do use RAMBuffer of 64MB and have heap
> size of 6G.
>
> There is no error is Solr log and I'd it running under WARNING level
> so missed the INFO if there was any during that period. I'm also not
> running any "optimize" command. What could cause Solr to hang for 80
> min?
>
> Thanks,
> -vivek
>
> On Fri, Apr 3, 2009 at 1:55 PM, vivek sar wrote:
>> Hi,
>>
>> I'm using Solr 1.4 (nightly build - 03/29/09). I'm stress testing my
>> application with Solr. My app uses Solrj to write to remote Solr (on
>> same box, but different JVM). The stress test sends over 2 million
>> records (1 record = 500 bytes, with each record having 10 fields)
>> within 5 minutes. All was working fine (with 2 million records
>> processed - 2G index size) and all the sudden Solr stopped responding
>> - I call server.addBeans(...) passing 15K object and don't get any
>> response for over an hour (usually it returns in 5 sec).
>>
>> I've 3 threads writing to the same index at the same time - not sure
>> if that could cause any problem. I was told by Otis that it should be
>> ok to have multiple threads write to same index - so I'm assuming it's
>> ok, though from thread dump I do see couple of "update" threads
>> waiting on ReadWriteLock and another thread (pool-6-thread-1) have a
>> lock on SolrWriter.
>>
>> Attached is the thread dump of the Tomcat process where Solr is
>> running. Any ideas?
>>
>> Thanks,
>> -vivek
>>
>
Re: Dynamic range Facets
: my documents (products) have a price field, and I want to have : a "dynamically" calculated range facet for that in the response. ... : So the question is how to get the dynamic facets response from solr. : : This is same question as previously posted back in 2007. But still waits an : answer?? : Is there any solution on this?? I'm not sure what you mean by "still waits an answer" ... the email thread you cut/pasted your example from gots lots of answers. My comments today are exactly the same as they were then (and in the 2006 threads i linked to back in 2007)... http://www.nabble.com/Dynamically-calculated-range-facet-to11314725.html#a11314725 http://www.nabble.com/faceted-browsing-to3655780.html#a3753053 in summary... * this is a hard problem to implement in a generic manner * it's much easier to do when you have some domain knowledge (ie: field is in dollars) What's changed since then is... 1) we have SearchComponents now 2) we have a stats component which does the hard work of finding min/max/avg/etc... It would be fairly easy to write a custom component that expected to run after the stats component to then get the counts for whatever ranges it thought were appropriate -- but knowing when to use ranges of "20" vs ranges of "100" is a very subejctive thing. -Hoss
Multi-valued fields with DIH
Hi, I need to assign multiple values to a field, with each value coming from a different column of the sql query. My data config snippet has lines like where 'project_area' & 'project_version' are output by the sql query to the datasource. The 'verbose-output' from dataimport.jsp does show that these columns have values returned by the query === − − + x 0:0:0.142 --- row #1- MySource/Area/Admin MySource/Version/06.02 2008-10-21T07:00:00Z . == But the resulting index has no data in the field 'projects'. Is it NOT possible to create multi-valued fields with DIH? Thanks -- View this message in context: http://www.nabble.com/Multi-valued-fields-with-DIH-tp22877509p22877509.html Sent from the Solr - User mailing list archive at Nabble.com.
solr index size
Hi, We built a Solr index on a set of documents a few times. Each time, we did an optimize to reduce the index to a single segment. The index sizes are slightly different across different runs. Even though the documents are not inserted in the same order across runs, it seems to me that the final optimized index should be identical. Running CheckIndex showed that the number of docs and fields are the same, but the number of terms are slightly different. Does anyone know how to explain this? Thanks, Jun IBM Almaden Research Center K55/B1, 650 Harry Road, San Jose, CA 95120-6099 jun...@almaden.ibm.com
Re: Dynamic range Facets
I think another challenge here is in distributed mode, how would you merge the ranges from subsearchers? -John On Fri, Apr 3, 2009 at 3:03 PM, Chris Hostetter wrote: > > : my documents (products) have a price field, and I want to have > : a "dynamically" calculated range facet for that in the response. > ... > : So the question is how to get the dynamic facets response from solr. > : > : This is same question as previously posted back in 2007. But still waits > an > : answer?? > : Is there any solution on this?? > > I'm not sure what you mean by "still waits an answer" ... the email thread > you cut/pasted your example from gots lots of answers. My comments today > are exactly the same as they were then (and in the 2006 threads i > linked to back in 2007)... > > > http://www.nabble.com/Dynamically-calculated-range-facet-to11314725.html#a11314725 > http://www.nabble.com/faceted-browsing-to3655780.html#a3753053 > > in summary... > * this is a hard problem to implement in a generic manner > * it's much easier to do when you have some domain knowledge (ie: field is > in dollars) > > What's changed since then is... > 1) we have SearchComponents now > 2) we have a stats component which does the hard work of finding > min/max/avg/etc... > > It would be fairly easy to write a custom component that expected to run > after the stats component to then get the counts for whatever ranges it > thought were appropriate -- but knowing when to use ranges of "20" vs > ranges of "100" is a very subejctive thing. > > > > > > > > -Hoss > >
Re: solr index size
Slightly different index sizes (even optimized) are normal - a same document may get different internal docids in different runs. I don't know why the number of terms are slight different. On Fri, Apr 3, 2009 at 7:21 PM, Jun Rao wrote: > > > Hi, > > We built a Solr index on a set of documents a few times. Each time, we did > an optimize to reduce the index to a single segment. The index sizes are > slightly different across different runs. Even though the documents are not > inserted in the same order across runs, it seems to me that the final > optimized index should be identical. Running CheckIndex showed that the > number of docs and fields are the same, but the number of terms are > slightly different. Does anyone know how to explain this? Thanks, > > Jun > IBM Almaden Research Center > K55/B1, 650 Harry Road, San Jose, CA 95120-6099 > > jun...@almaden.ibm.com
ExtractingRequestHandler Question
Hi, I am using ExtractingRequestHandler to index rich text documents. The way I am doing it is I get some data related to the document from database and then post an xml (containing only this data ) to solr. Then I make another call to solr, which sends the actual document to be indexed. But while doing so I am loosing out all the other data that is related to the document. Is this the right way to do handle it or am I missing out on something. TIA
Re: getting started
: http://lucene.apache.org/solr/tutorial.html#Getting+Started : : link - lucene QueryParser syntax fixed in svn, ... the site should update in about 30 minutes. thanks for pointing this out. -Hoss
How to send a parsed Query to shards?
I want to use distributed search with some search components that I would like to execute only on the main server, not on the shards, because they reference some large in-memory lookup tables. After the search components get done processing the orignal query, the query may contain SpanNearQueries and DisjunctionMaxQueries. I'd like to send that query to the shards, not the original query. I've come up with the following idea for doing this. Would people please comment on this idea or suggest a better alternative? * Subclass QueryComponent to base64 encode the serialized form of the query and send that in place of the original query. * set the queryParser on the shard servers to a custom class that unencodes and deserializes the encoded query and returns it. Thoughts on this approach, or is there a better one? Thanks, -dallan
Re: Multi-valued fields with DIH
the column names are case sensitive try this On Sat, Apr 4, 2009 at 3:58 AM, ashokc wrote: > > Hi, > I need to assign multiple values to a field, with each value coming from a > different column of the sql query. > > My data config snippet has lines like > > > > > where 'project_area' & 'project_version' are output by the sql query to the > datasource. The 'verbose-output' from dataimport.jsp does show that these > columns have values returned by the query > > === > > > - > > - > > + > > x > > 0:0:0.142 > --- row #1- > MySource/Area/Admin > MySource/Version/06.02 > 2008-10-21T07:00:00Z > . > > == > > But the resulting index has no data in the field 'projects'. Is it NOT > possible to create multi-valued fields with DIH? > > Thanks > -- > View this message in context: > http://www.nabble.com/Multi-valued-fields-with-DIH-tp22877509p22877509.html > Sent from the Solr - User mailing list archive at Nabble.com. > > -- --Noble Paul
Re: Oracle Clob column with DIH does not turn to String
it is very expensive to do a case insensitive lookup. It must first convert all the keys to lower case and try looking up there. because it may not be always in uppercase it can be in mixed case as well On Sat, Apr 4, 2009 at 12:58 AM, ashokc wrote: > > Happy to report that it is working. Looks like we have to use UPPER CASE for > all the column names. When I examined the map 'aRow', it had the column > names in upper case, where as my config had lower case. No match was found > so nothing happened. Changed my config and it works now. Thanks for your > help. Perhaps this transformer can be modified to be case-insensitive for > the column names. If you had written it perhaps it is a quick change for > you? > > Noble Paul നോബിള് नोब्ळ् wrote: >> >> I guess u can write a custom transformer which gets a String out of >> the oracle.sql.CLOB. I am just out of clue, why this may happen. I >> even wrote a testcase and it seems to work fine >> --Noble >> >> On Fri, Apr 3, 2009 at 10:23 PM, ashokc wrote: >>> >>> I downloaded the nightly build yesterday (2nd April), modified the >>> ClobTransformer.java file with some prints, compiled it all (ant dist). >>> It >>> produced a war file, "apache-solr-1.4-dev.war". That is what I am using. >>> My >>> modification & compilation has not affected the results. I was getting >>> the >>> same behavior with the 'war' that download came with. Thanks Noble. >>> >>> Noble Paul നോബിള് नोब्ळ् wrote: and which version of Solr are u using? On Fri, Apr 3, 2009 at 10:09 PM, ashokc wrote: > > Sure: > > data-config Xml > === > > > url="jdbc:oracle:thin:@x" user="remedy" password="y"/> > > query="SELECT > mylog_ato, name_char, dsc FROM log_tbl"> > > > > > > > > === > > A search result on the field short_desc: > -- > > > 1.8670129 > oracle.sql.c...@155e3ab > 4486 > Develop Rating functionality for QIN > 2009-04-03T11:47:32.635Z > > > > > > Noble Paul നോബിള് नोब्ळ् wrote: >> >> There is something else wrong with your setup. >> >> can you just paste the whole data-config.xml >> >> --Noble >> >> On Fri, Apr 3, 2009 at 5:39 PM, ashokc wrote: >>> >>> Noble, >>> I put in a few 'System.out.println' statements in the >>> ClobTransformer.java >>> file & remade the war. But I see none of these prints coming up in my >>> 'catalina.out' file. Is that the right file to be looking at? >>> >>> As an aside, is 'catalina.out' the ONLY log file for SOLR? I turned >>> on >>> the >>> logging to 'FINE' for everything. Also, these settings seem to go >>> away >>> when >>> Tomcat is restarted. >>> - ashok >>> >>> Noble Paul നോബിള് नोब्ळ् wrote: yeah, ant dist will give you the .war file you may need . just drop it in and you are set to go. or if you can hook up a debugger to a running Solr that is the easiest --Noble On Fri, Apr 3, 2009 at 9:35 AM, ashokc wrote: > > That would require me to recompile (with ant/maven scripts?) the > source > and > replace the jar for DIH, right? I can try - for the first time. > - ashok > > Noble Paul നോബിള് नोब्ळ् wrote: >> >> This looks strange. Apparently the Transformer did not get >> applied. >> Is >> it possible for you to debug ClobTransformer >> adding(System.out.println >> into ClobTransformer may help) >> >> On Fri, Apr 3, 2009 at 6:04 AM, ashokc >> wrote: >>> >>> Correcting my earlier post. It lost some lines some how. >>> >>> Hi, >>> >>> I have set up to import some oracle clob columns with DIH. I am >>> using >>> the >>> latest nightly release. My config says, >>> >>> >>> >> ... >>> >>> >>> >>> >>> >>> >>> But it does not seem to turn this clob into a String. The search >>> results >>> show: >>> >>> >>> 1.8670129 >>> oracle.sql.c...@aed3a5 >>> 4486 >>> >>> >>> Any pointers on why I do not get the 'string' out of the clob for >>> indexing? >>> Is the nightly war NOT the right one to use? >>> >>> Thanks for your help. >>> >>> - ashok >>> >>> >>> >>> ashokc wrote: Hi, I have set up to import some oracle clob columns with DIH. I am
Re: Phrase Query Issue
Erik, Thanks a lot for your reply. I have made some changes in the solr code and now field clauses are working fine with dismax request. Not only this, wildcard characters are also working with dismax and q query parameter. If you want I can share modified code with you. Thanks, Amit Garg Erik Hatcher wrote: > > Also note that the dismax parser does NOT support fielded clauses, so > fieldname:"some phrase" is not parsed as you might expect with dismax. > > Erik > > On Apr 3, 2009, at 3:41 AM, dabboo wrote: > >> >> Hi, >> >> I am implementing phrase search query using my application. I am >> trying to >> search "Glorious Revolution", I am expecting all the records which >> contains >> this as whole string in any of its field should display. >> >> Below is my query: >> >> ?q=fieldname:"Glorious Revolution"&qt=dismaxrequest >> >> I am passing this to my solr search engine using httpclient. But it is >> thrwoing me Invaild Query exception. If I append double quotes with >> backslash, it passes the query but it doesnt shows the right values. >> >> Instead it creates the query like: >> >> isbn13_product_s:\"Glorious Revolution\" >> >> Please suggest how I should tackle this. If I give the phrase search >> in my >> solr console, it returns me the correct results. >> >> Thanks, >> Amit Garg >> >> >> -- >> View this message in context: >> http://www.nabble.com/Phrase-Query-Issue-tp22863529p22863529.html >> Sent from the Solr - User mailing list archive at Nabble.com. > > > -- View this message in context: http://www.nabble.com/Phrase-Query-Issue-tp22863529p22880202.html Sent from the Solr - User mailing list archive at Nabble.com.
