Re: fieldCache problem OOM exception

[Bernd Fehling]

The current status of my installation is that with some tweeking of
JAVA I get a runtime of about 2 weeks until OldGen (14GB) is filled
to 100 percent and won't free anything even with FullGC.
The part of fieldCache in a HeapDump to that time is over 80 percent
from the whole heap (20GB). And that is what eats up all OldGen
until OOM.
Next week I will start with tomcat 6.x to see how that one behaves, but
there isn't any hope. It is just a different container which wouldn't
change anything about how Lucene eats up memory with fieldCache.

After digging through all the code, logging and debugging I can say that it
seams to be not a memory leak.

Solr is using the fieldCache from Lucene under the hood of the servlet 
container.
The fieldCache grows until everything cachable is in memory or OOM
is reached, what ever comes first.
The description says: "Provides introspection of the Lucene FieldCache,
this is **NOT** a cache that is managed by Solr."
So it seams to be a Lucene problem.

As a matter of fact and due to this limitation solr can't be used
with a single huge index. I don't know how other applications which are
using Lucene and it's fieldCache (and there are a lot of them) are
handling this and how they take care of the size of the fieldCache.
And, I currently don't know how to calculate the limit.
Say for example: the size of *.tii and *.tis file in the index should be
 the -Xmx size of your JAVA to be save with fieldCache and OOM.

May be an expert can give more detailed info about fieldCache and its
possible maximum size.

Some data about our index:
-rw-r--r-- 1 solr users 84448291214 19. Jul 10:43 _12jl.fdt
-rw-r--r-- 1 solr users   236458468 19. Jul 10:43 _12jl.fdx
-rw-r--r-- 1 solr users1208 19. Jul 10:30 _12jl.fnm
-rw-r--r-- 1 solr users 19950615826 19. Jul 11:20 _12jl.frq
-rw-r--r-- 1 solr users   532031548 19. Jul 11:20 _12jl.nrm
-rw-r--r-- 1 solr users 20616887682 19. Jul 11:20 _12jl.prx
-rw-r--r-- 1 solr users   291149087 19. Jul 11:20 _12jl.tii
-rw-r--r-- 1 solr users 30850743727 19. Jul 11:20 _12jl.tis
-rw-r--r-- 1 solr users  20  9. Jun 11:11 segments.gen
-rw-r--r-- 1 solr users 274 19. Jul 11:20 segments_pl
Size: 146,15 GB
Docs: 29.557.308


Regards,
Bernd


Am 22.07.2011 00:10, schrieb Santiago Bazerque:

Hello Erick,

I have a 1.7MM documents, 3.6GB index. I also hava an unusual amount of
dynamic fields, that I use for sorting. My FieldCache currently has about
13.000 entries, even though my index only has 1-3 queries per second. Each
query sorts by two dynamic fields, and facets on 3-4 fields that are fixed.
These latter fields are always in the field cache, what I find suspicious is
the other ~13.000 that are sitting there.

I am using a 32GB heap, and I am seeing periodical OOM errors (I didn't spot
a regular pattern as Bernd did, but haven't increased RAM as methodically as
he has).

If you need any more info, I'll be glad to post it to the list.

Best,
Santiago

On Fri, Jun 17, 2011 at 9:13 AM, Erick Ericksonwrote:


Sorry, it was late last night when I typed that...

Basically, if you sort and facet on #all# the fields you mentioned, it
should populate
the cache in one go. If the problem is that you just have too many unique
terms
for all those operations, then it should go bOOM.

But, frankly, that's unlikely, I'm just suggesting that to be sure the
easy case isn't
the problem. Take a memory snapshot at that point just to see, it should be
a
high-water mark.

The fact that you increase the heap and can then run for longer is
extremely
suspicious, and really smells like a memory issue, so we'd like to pursue
it.

I'd be really interested if anyone else is seeing anything similar,
these are the
scary ones...

Best
Erick

On Fri, Jun 17, 2011 at 3:09 AM, Bernd Fehling
  wrote:

Hi Erik,
I will take some memory snapshots during the next week,
but how can it be to get OOMs with one query?

- I started with 6g for JVM -->  1 day until OOM.
- increased to 8 g -->  2 days until OOM
- increased to 10g -->  3.5 days until OOM
- increased to 16g -->  5 days until OOM
- currently 20g -->  about 7 days until OOM

Starting the system takes about 3.5g and goes up to about 4g after a

while.


The only dirty workaround so far is to restart the whole system after 5
days.
Not really nice.

The problem seams to be fieldCache which is under the hood of jetty.
Do you know of any sizing features for fieldCache to limit the memory
consumption?

Regards
Bernd

Am 17.06.2011 03:37, schrieb Erick Erickson:


Well, if my theory is right, you should be able to generate OOMs at will
by
sorting and faceting on all your fields in one query.

But Lucene's cache should be garbage collected, can you take some memory
snapshots during the week? It should hit a point and stay steady there.

How much memory are you giving your JVM? It looks like a lot given your
memory snapshot.

Best
Erick

On Thu, Jun 16, 2011 at 3:01 AM, Bernd Fehling
   wrote:


Hi Erik,

yes I'm sortin

RE: What is the different?

[Pierre GOSSE]

Hi,

Have you check the queries by using the debugQuery=true parameter ? This could 
give some hints of what is searched in both cases.

Pierre

-Message d'origine-
De : cnyee [mailto:yeec...@gmail.com] 
Envoyé : vendredi 22 juillet 2011 05:14
À : solr-user@lucene.apache.org
Objet : What is the different?

Hi,

I have two queries:

(1) q = (change management)
(2) q = (change management) AND domain_ids:(0^1.3 OR 1)

The purpose of the (2) is to boost the records with domain_ids=0.
In my database all records has domain_ids = 0 or 1, so domains_ids:(0 or 1)
will always returns the full database.

Now my questions is - query (2) returns 5000+ results, but query (1) returns
700+ results.

Can somebody enlighten me on what is the reasons behind such a vast
different in number of results?

Many thanks in advance.

Yee

--
View this message in context: 
http://lucene.472066.n3.nabble.com/What-is-the-different-tp3190278p3190278.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Geospatial queries in Solr

[David Smiley (@MITRE.org)]

Jamie,
  You are using the field named "point" which is based on PointFieldType. 
Keep in mind that just because this field type is named this way, does *not*
mean at all that other fields don't hold points, or that this one is
especially suited to it.  Arguably this one is named poorly.  This field
type is loosely based on Solr's LatLonType but it doesn't yet fully
implement capabilities like crossing the dateline and shapes other than a
bounding box.  I've enhanced the code to log a warning if the query shape is
expanded to the bounding box. Any way, it's a currently a 2nd class citizen
in LSP.
  I recommend you use the field named "geohash" which uses a field type name
starting with "Recursive...".  Note that the same query you did with this
field returned the polygon of Russia.  There is an outstanding box in which
we don't properly handle polygons crossing the dateline; other shapes are
fine.

~ David

-
 Author: https://www.packtpub.com/solr-1-4-enterprise-search-server/book
--
View this message in context: 
http://lucene.472066.n3.nabble.com/Geospatial-queries-in-Solr-tp3183374p3190762.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: embeded solrj doesn't refresh index

[Marc Sturlese]

Are u indexing with full import? In case yes and the resultant index has
similar num of docs (that the one you had before) try setting reopenReaders
to false in solrconfig.xml
* You have to send the comit, of course.

--
View this message in context: 
http://lucene.472066.n3.nabble.com/embeded-solrj-doesn-t-refresh-index-tp3184321p3190892.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: Culr Tika not working with blanks into literal.field

[Peralta Gutiérrez del Álamo]

Hi.
Is it possible setting fields with blaks values using extract update ?
Thanks

From: pacopera...@hotmail.com
To: solr-user@lucene.apache.org
Subject: Culr Tika not working with blanks into literal.field
Date: Wed, 20 Jul 2011 12:53:18 +
























Hi. 
I'm trying to index binary documents with curl and Tika for extracting text. 

The problem  is that when I set the value of a field with spaces blanks using 
the input parameter: literal.=, the document is not indexed. 

The sentence I send is the follow: 

curl 
http://localhost:8983/solr/update/extract?literal.id="doc1"\&literal.url="/mnt/windows/Ofertas/2006
 Portal 
Intranet/DOCUMENTACION/datos.doc"\&uprefix=attr_\&fmap.content=text\&commit=true
 -F myfile=\@"/mnt/windows/Ofertas/DOCUMENTACION/datos.doc" 


That is literal.url="value with blanks" apparently is not working 
  

  

RE: Solr 3.3: Exception in thread "Lucene Merge Thread #1"

[mdz-munich]

mdz-munich wrote:
> 
> Yeah, indeed.
> 
> But since the VM is equipped with plenty of RAM (22GB) and it works so far
> (Solr 3.2) very well with this setup, I AM slightly confused, am I?
> 
> Maybe we should LOWER the dedicated Physical Memory? The remaining 10GB
> are used for a second tomcat (8GB) and the OS (Suse). As far as I
> understand NIO (mostly un-far), this package "can directly use the most
> efficient operations of the underlying platform".
> 

After three days of banging my head against the wall: Problem solved! 

It seems there was not enough Memory left for NIO. 

--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-3-3-Exception-in-thread-Lucene-Merge-Thread-1-tp3185248p3190916.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: commit time and lock

[Pierre GOSSE]

Solr still respond to search queries during commit, only new indexations 
requests will have to wait (until end of commit?). So I don't think your users 
will experience increased response time during commits (unless your server is 
much undersized).

Pierre

-Message d'origine-
De : Jonty Rhods [mailto:jonty.rh...@gmail.com] 
Envoyé : jeudi 21 juillet 2011 20:27
À : solr-user@lucene.apache.org
Objet : Re: commit time and lock

Actually i m worried about the response time. i k commiting around 500
docs in every 5 minutes. as i know,correct me if i m wrong; at the
time of commiting solr server stop responding. my concern is how to
minimize the response time so user not need to wait. or any other
logic will require for my case. please suggest.

regards
jonty

On Tuesday, June 21, 2011, Erick Erickson  wrote:
> What is it you want help with? You haven't told us what the
> problem you're trying to solve is. Are you asking how to
> speed up indexing? What have you tried? Have you
> looked at: http://wiki.apache.org/solr/FAQ#Performance?
>
> Best
> Erick
>
> On Tue, Jun 21, 2011 at 2:16 AM, Jonty Rhods  wrote:
>> I am using solrj to index the data. I have around 5 docs indexed. As at
>> the time of commit due to lock server stop giving response so I was
>> calculating commit time:
>>
>> double starttemp = System.currentTimeMillis();
>> server.add(docs);
>> server.commit();
>> System.out.println("total time in commit = " + (System.currentTimeMillis() -
>> starttemp)/1000);
>>
>> It taking around 9 second to commit the 5000 docs with 15 fields. However I
>> am not confirm the lock time of index whether it is start
>> since server.add(docs); time or server.commit(); time only.
>>
>> If I am changing from above to following
>>
>> server.add(docs);
>> double starttemp = System.currentTimeMillis();
>> server.commit();
>> System.out.println("total time in commit = " + (System.currentTimeMillis() -
>> starttemp)/1000);
>>
>> then commit time becomes less then 1 second. I am not sure which one is
>> right.
>>
>> please help.
>>
>> regards
>> Jonty
>>
>

RE: Solr 3.3: Exception in thread "Lucene Merge Thread #1"

[mdz-munich]

I was wrong. 

After rebooting tomcat we discovered a new sweetness: 

/SEVERE: REFCOUNT ERROR: unreferenced org.apache.solr.core.SolrCore@3c753c75
(core.name) has a reference count of 1
22.07.2011 11:52:07 org.apache.solr.common.SolrException log
SEVERE: java.lang.RuntimeException: java.io.IOException: Map failed
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1099)
at org.apache.solr.core.SolrCore.(SolrCore.java:585)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:463)
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:316)
at org.apache.solr.core.CoreContainer.load(CoreContainer.java:207)
at
org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:130)
at
org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:94)
at
org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:273)
at
org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:254)
at
org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:372)
at
org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:98)
at
org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4584)
at
org.apache.catalina.core.StandardContext$2.call(StandardContext.java:5262)
at
org.apache.catalina.core.StandardContext$2.call(StandardContext.java:5257)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314)
at java.util.concurrent.FutureTask.run(FutureTask.java:149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
at java.lang.Thread.run(Thread.java:736)
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:782)
at
org.apache.lucene.store.MMapDirectory$MMapIndexInput.(MMapDirectory.java:264)
at 
org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:216)
at org.apache.lucene.index.FieldsReader.(FieldsReader.java:129)
at
org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:244)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:92)
at 
org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:113)
at
org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:29)
at
org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:81)
at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:750)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:75)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:428)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:371)
at
org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:38)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1088)
... 18 more
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:779)
... 33 more/

Any ideas and/or suggestions? 

Best regards & thank you,

Sebastian 

--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-3-3-Exception-in-thread-Lucene-Merge-Thread-1-tp3185248p3190976.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: commit time and lock

[Jonty Rhods]

Thanks for clarity.

One more thing I want to know about optimization.

Right now I am planning to optimize the server in 24 hour. Optimization is
also time taking ( last time took around 13 minutes), so I want to know that
:

1. when optimization is under process that time will solr server response or
not?
2. if server will not response then how to do optimization of server fast or
other way to do optimization so our user will not have to wait to finished
optimization process.

regards
Jonty



On Fri, Jul 22, 2011 at 2:44 PM, Pierre GOSSE wrote:

> Solr still respond to search queries during commit, only new indexations
> requests will have to wait (until end of commit?). So I don't think your
> users will experience increased response time during commits (unless your
> server is much undersized).
>
> Pierre
>
> -Message d'origine-
> De : Jonty Rhods [mailto:jonty.rh...@gmail.com]
> Envoyé : jeudi 21 juillet 2011 20:27
> À : solr-user@lucene.apache.org
> Objet : Re: commit time and lock
>
> Actually i m worried about the response time. i k commiting around 500
> docs in every 5 minutes. as i know,correct me if i m wrong; at the
> time of commiting solr server stop responding. my concern is how to
> minimize the response time so user not need to wait. or any other
> logic will require for my case. please suggest.
>
> regards
> jonty
>
> On Tuesday, June 21, 2011, Erick Erickson  wrote:
> > What is it you want help with? You haven't told us what the
> > problem you're trying to solve is. Are you asking how to
> > speed up indexing? What have you tried? Have you
> > looked at: http://wiki.apache.org/solr/FAQ#Performance?
> >
> > Best
> > Erick
> >
> > On Tue, Jun 21, 2011 at 2:16 AM, Jonty Rhods 
> wrote:
> >> I am using solrj to index the data. I have around 5 docs indexed. As
> at
> >> the time of commit due to lock server stop giving response so I was
> >> calculating commit time:
> >>
> >> double starttemp = System.currentTimeMillis();
> >> server.add(docs);
> >> server.commit();
> >> System.out.println("total time in commit = " +
> (System.currentTimeMillis() -
> >> starttemp)/1000);
> >>
> >> It taking around 9 second to commit the 5000 docs with 15 fields.
> However I
> >> am not confirm the lock time of index whether it is start
> >> since server.add(docs); time or server.commit(); time only.
> >>
> >> If I am changing from above to following
> >>
> >> server.add(docs);
> >> double starttemp = System.currentTimeMillis();
> >> server.commit();
> >> System.out.println("total time in commit = " +
> (System.currentTimeMillis() -
> >> starttemp)/1000);
> >>
> >> then commit time becomes less then 1 second. I am not sure which one is
> >> right.
> >>
> >> please help.
> >>
> >> regards
> >> Jonty
> >>
> >
>

convert date format at indexing time

[Peralta Gutiérrez del Álamo]

Hi.I'm indexing binary documents as Word, pdf,... from a file system
I'm  extracting the  attribute attr_creation_date for these documents but the 
format I'm getting is Wed Jan 14 12:13:00 CET 2004 instead 2004-01-14T12:12:00Z
Is It possible to convert date format at indexing time?
Thanks Best regards
  

RE: commit time and lock

[Pierre GOSSE]

Solr will response for search during optimization, but commits will have to 
wait the end of the optimization process.

During optimization a new index is generated on disk by merging every single 
file of the current index into one big file, so you're server will be busy, 
especially regarding disk access. This may alter your response time and has 
very negative effect on the replication of index if you have a master/slave 
architecture.

I've read here that optimization is not always a requirement to have an 
efficient index, due to some low level changes in lucene 3.xx, so maybe you 
don't really need optimization. What version of solr are you using ? Maybe 
someone can point toward a relevant link about optimization other than solr 
wiki 
http://wiki.apache.org/solr/SolrPerformanceFactors#Optimization_Considerations

Pierre


-Message d'origine-
De : Jonty Rhods [mailto:jonty.rh...@gmail.com] 
Envoyé : vendredi 22 juillet 2011 12:45
À : solr-user@lucene.apache.org
Objet : Re: commit time and lock

Thanks for clarity.

One more thing I want to know about optimization.

Right now I am planning to optimize the server in 24 hour. Optimization is
also time taking ( last time took around 13 minutes), so I want to know that
:

1. when optimization is under process that time will solr server response or
not?
2. if server will not response then how to do optimization of server fast or
other way to do optimization so our user will not have to wait to finished
optimization process.

regards
Jonty



On Fri, Jul 22, 2011 at 2:44 PM, Pierre GOSSE wrote:

> Solr still respond to search queries during commit, only new indexations
> requests will have to wait (until end of commit?). So I don't think your
> users will experience increased response time during commits (unless your
> server is much undersized).
>
> Pierre
>
> -Message d'origine-
> De : Jonty Rhods [mailto:jonty.rh...@gmail.com]
> Envoyé : jeudi 21 juillet 2011 20:27
> À : solr-user@lucene.apache.org
> Objet : Re: commit time and lock
>
> Actually i m worried about the response time. i k commiting around 500
> docs in every 5 minutes. as i know,correct me if i m wrong; at the
> time of commiting solr server stop responding. my concern is how to
> minimize the response time so user not need to wait. or any other
> logic will require for my case. please suggest.
>
> regards
> jonty
>
> On Tuesday, June 21, 2011, Erick Erickson  wrote:
> > What is it you want help with? You haven't told us what the
> > problem you're trying to solve is. Are you asking how to
> > speed up indexing? What have you tried? Have you
> > looked at: http://wiki.apache.org/solr/FAQ#Performance?
> >
> > Best
> > Erick
> >
> > On Tue, Jun 21, 2011 at 2:16 AM, Jonty Rhods 
> wrote:
> >> I am using solrj to index the data. I have around 5 docs indexed. As
> at
> >> the time of commit due to lock server stop giving response so I was
> >> calculating commit time:
> >>
> >> double starttemp = System.currentTimeMillis();
> >> server.add(docs);
> >> server.commit();
> >> System.out.println("total time in commit = " +
> (System.currentTimeMillis() -
> >> starttemp)/1000);
> >>
> >> It taking around 9 second to commit the 5000 docs with 15 fields.
> However I
> >> am not confirm the lock time of index whether it is start
> >> since server.add(docs); time or server.commit(); time only.
> >>
> >> If I am changing from above to following
> >>
> >> server.add(docs);
> >> double starttemp = System.currentTimeMillis();
> >> server.commit();
> >> System.out.println("total time in commit = " +
> (System.currentTimeMillis() -
> >> starttemp)/1000);
> >>
> >> then commit time becomes less then 1 second. I am not sure which one is
> >> right.
> >>
> >> please help.
> >>
> >> regards
> >> Jonty
> >>
> >
>

Re: Geospatial queries in Solr

[Jamie Johnson]

Ah, my mistake then.  I will switch to using the geohash field.  When
doing my query I did run it against geohash but when I got Russia that
was more incorrect than point so I stopped using it.

Is there a timeline by which you expect the dateline issue to be
addressed?  I don't believe that will be an issue for me since I don't
believe we have any information which will be from that area of the
world but I'll have to verify.


On Fri, Jul 22, 2011 at 3:35 AM, David Smiley (@MITRE.org)
 wrote:
> Jamie,
>  You are using the field named "point" which is based on PointFieldType.
> Keep in mind that just because this field type is named this way, does *not*
> mean at all that other fields don't hold points, or that this one is
> especially suited to it.  Arguably this one is named poorly.  This field
> type is loosely based on Solr's LatLonType but it doesn't yet fully
> implement capabilities like crossing the dateline and shapes other than a
> bounding box.  I've enhanced the code to log a warning if the query shape is
> expanded to the bounding box. Any way, it's a currently a 2nd class citizen
> in LSP.
>  I recommend you use the field named "geohash" which uses a field type name
> starting with "Recursive...".  Note that the same query you did with this
> field returned the polygon of Russia.  There is an outstanding box in which
> we don't properly handle polygons crossing the dateline; other shapes are
> fine.
>
> ~ David
>
> -
>  Author: https://www.packtpub.com/solr-1-4-enterprise-search-server/book
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/Geospatial-queries-in-Solr-tp3183374p3190762.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: Logically equivalent queries but vastly different no of results?

[cnyee]

I think I know what it is. The second query has higher scores than the first.

The additional condition "domain_ids:(0^1.3 OR 1)" which evaluates to true
always - pushed up the scores and allows a LOT more records to pass.

Is there a better way of doing this?

Regards,
Yee

--
View this message in context: 
http://lucene.472066.n3.nabble.com/Logically-equivalent-queries-but-vastly-different-no-of-results-tp3190278p3191211.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Logically equivalent queries but vastly different no of results?

[Michael Kuhlmann]

Am 22.07.2011 14:27, schrieb cnyee:
> I think I know what it is. The second query has higher scores than the first.
> 
> The additional condition "domain_ids:(0^1.3 OR 1)" which evaluates to true
> always - pushed up the scores and allows a LOT more records to pass.

This can't be, because the score doesn't affect the document count. Only
relative score values are relevant, not absolute ones.

Another guess: What's your query handler? If it's dismax, than the
reason is quite obvious.

-Kuli

Re: Solr 3.3: Exception in thread "Lucene Merge Thread #1"

[Yonik Seeley]

> IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux amd64-64

I'm confused why MMapDirectory is getting used with the IBM JVM... I
had thought it would default to NIOFSDirectory on Linux w/ a non
Oracle JVM.
Are you specifically selecting MMapDirectory in solrconfig.xml?

Can you try the Oracle JVM to see if that changes things?

-Yonik
http://www.lucidimagination.com



On Fri, Jul 22, 2011 at 5:58 AM, mdz-munich
 wrote:
> I was wrong.
>
> After rebooting tomcat we discovered a new sweetness:
>
> /SEVERE: REFCOUNT ERROR: unreferenced org.apache.solr.core.SolrCore@3c753c75
> (core.name) has a reference count of 1
> 22.07.2011 11:52:07 org.apache.solr.common.SolrException log
> SEVERE: java.lang.RuntimeException: java.io.IOException: Map failed
>        at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1099)
>        at org.apache.solr.core.SolrCore.(SolrCore.java:585)
>        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:463)
>        at org.apache.solr.core.CoreContainer.load(CoreContainer.java:316)
>        at org.apache.solr.core.CoreContainer.load(CoreContainer.java:207)
>        at
> org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:130)
>        at
> org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:94)
>        at
> org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:273)
>        at
> org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:254)
>        at
> org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:372)
>        at
> org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:98)
>        at
> org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4584)
>        at
> org.apache.catalina.core.StandardContext$2.call(StandardContext.java:5262)
>        at
> org.apache.catalina.core.StandardContext$2.call(StandardContext.java:5257)
>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:149)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
>        at java.lang.Thread.run(Thread.java:736)
> Caused by: java.io.IOException: Map failed
>        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:782)
>        at
> org.apache.lucene.store.MMapDirectory$MMapIndexInput.(MMapDirectory.java:264)
>        at 
> org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:216)
>        at org.apache.lucene.index.FieldsReader.(FieldsReader.java:129)
>        at
> org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:244)
>        at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
>        at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:92)
>        at 
> org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:113)
>        at
> org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:29)
>        at
> org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:81)
>        at
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:750)
>        at 
> org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:75)
>        at org.apache.lucene.index.IndexReader.open(IndexReader.java:428)
>        at org.apache.lucene.index.IndexReader.open(IndexReader.java:371)
>        at
> org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:38)
>        at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1088)
>        ... 18 more
> Caused by: java.lang.OutOfMemoryError: Map failed
>        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:779)
>        ... 33 more/
>
> Any ideas and/or suggestions?
>
> Best regards & thank you,
>
> Sebastian
>
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/Solr-3-3-Exception-in-thread-Lucene-Merge-Thread-1-tp3185248p3190976.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: commit time and lock

[Marc SCHNEIDER]

Hello,

Pierre, can you tell us where you read that?
"I've read here that optimization is not always a requirement to have an
efficient index, due to some low level changes in lucene 3.xx"

Marc.

On Fri, Jul 22, 2011 at 2:10 PM, Pierre GOSSE wrote:

> Solr will response for search during optimization, but commits will have to
> wait the end of the optimization process.
>
> During optimization a new index is generated on disk by merging every
> single file of the current index into one big file, so you're server will be
> busy, especially regarding disk access. This may alter your response time
> and has very negative effect on the replication of index if you have a
> master/slave architecture.
>
> I've read here that optimization is not always a requirement to have an
> efficient index, due to some low level changes in lucene 3.xx, so maybe you
> don't really need optimization. What version of solr are you using ? Maybe
> someone can point toward a relevant link about optimization other than solr
> wiki
> http://wiki.apache.org/solr/SolrPerformanceFactors#Optimization_Considerations
>
> Pierre
>
>
> -Message d'origine-
> De : Jonty Rhods [mailto:jonty.rh...@gmail.com]
> Envoyé : vendredi 22 juillet 2011 12:45
> À : solr-user@lucene.apache.org
> Objet : Re: commit time and lock
>
> Thanks for clarity.
>
> One more thing I want to know about optimization.
>
> Right now I am planning to optimize the server in 24 hour. Optimization is
> also time taking ( last time took around 13 minutes), so I want to know
> that
> :
>
> 1. when optimization is under process that time will solr server response
> or
> not?
> 2. if server will not response then how to do optimization of server fast
> or
> other way to do optimization so our user will not have to wait to finished
> optimization process.
>
> regards
> Jonty
>
>
>
> On Fri, Jul 22, 2011 at 2:44 PM, Pierre GOSSE  >wrote:
>
> > Solr still respond to search queries during commit, only new indexations
> > requests will have to wait (until end of commit?). So I don't think your
> > users will experience increased response time during commits (unless your
> > server is much undersized).
> >
> > Pierre
> >
> > -Message d'origine-
> > De : Jonty Rhods [mailto:jonty.rh...@gmail.com]
> > Envoyé : jeudi 21 juillet 2011 20:27
> > À : solr-user@lucene.apache.org
> > Objet : Re: commit time and lock
> >
> > Actually i m worried about the response time. i k commiting around 500
> > docs in every 5 minutes. as i know,correct me if i m wrong; at the
> > time of commiting solr server stop responding. my concern is how to
> > minimize the response time so user not need to wait. or any other
> > logic will require for my case. please suggest.
> >
> > regards
> > jonty
> >
> > On Tuesday, June 21, 2011, Erick Erickson 
> wrote:
> > > What is it you want help with? You haven't told us what the
> > > problem you're trying to solve is. Are you asking how to
> > > speed up indexing? What have you tried? Have you
> > > looked at: http://wiki.apache.org/solr/FAQ#Performance?
> > >
> > > Best
> > > Erick
> > >
> > > On Tue, Jun 21, 2011 at 2:16 AM, Jonty Rhods 
> > wrote:
> > >> I am using solrj to index the data. I have around 5 docs indexed.
> As
> > at
> > >> the time of commit due to lock server stop giving response so I was
> > >> calculating commit time:
> > >>
> > >> double starttemp = System.currentTimeMillis();
> > >> server.add(docs);
> > >> server.commit();
> > >> System.out.println("total time in commit = " +
> > (System.currentTimeMillis() -
> > >> starttemp)/1000);
> > >>
> > >> It taking around 9 second to commit the 5000 docs with 15 fields.
> > However I
> > >> am not confirm the lock time of index whether it is start
> > >> since server.add(docs); time or server.commit(); time only.
> > >>
> > >> If I am changing from above to following
> > >>
> > >> server.add(docs);
> > >> double starttemp = System.currentTimeMillis();
> > >> server.commit();
> > >> System.out.println("total time in commit = " +
> > (System.currentTimeMillis() -
> > >> starttemp)/1000);
> > >>
> > >> then commit time becomes less then 1 second. I am not sure which one
> is
> > >> right.
> > >>
> > >> please help.
> > >>
> > >> regards
> > >> Jonty
> > >>
> > >
> >
>

Re: Updating fields in an existing document

[Marc SCHNEIDER]

Yes that's it if you add twice the same document (ie with the same id) it
will replace it.

On Thu, Jul 21, 2011 at 7:46 PM, Benson Margulies wrote:

> A followup. The wiki has a whole discussion of the 'update' XML
> message. But solrj has nothing like it. Does that really exist? Is
> there a reason to use it? If I just 'add' the document a second time,
> it will replace?
>
> On Wed, Jul 20, 2011 at 7:08 PM, Jonathan Rochkind 
> wrote:
> > Nope, you're not missing anything, there's no way to alter a document in
> an index but reindexing the whole document. Solr's architecture would make
> it difficult (although never say impossible) to do otherwise. But you're
> right it would be convenient for people other than you.
> >
> > Reindexing a single document ought not to be slow, although if you have
> many of them at once it could be, or if you end up needing to very
> frequently commit to an index it can indeed cause problems.
> > 
> > From: Benson Margulies [bimargul...@gmail.com]
> > Sent: Wednesday, July 20, 2011 6:05 PM
> > To: solr-user
> > Subject: Updating fields in an existing document
> >
> > We find ourselves in the following quandry:
> >
> > At initial index time, we store a value in a field, and we use it for
> > facetting. So it, seemingly, has to be there as a field.
> >
> > However, from time to time, something happens that causes us to want
> > to change this value. As far as we know, this requires us to
> > completely re-index the document, which is slow.
> >
> > It struck me that we can't be the only people to go down this road, so
> > I write to inquire if we are missing something.
> >
>

Re: Solr 3.3: Exception in thread "Lucene Merge Thread #1"

[Yonik Seeley]

On Fri, Jul 22, 2011 at 9:44 AM, Yonik Seeley
 wrote:
>> IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 Linux amd64-64
>
> I'm confused why MMapDirectory is getting used with the IBM JVM... I
> had thought it would default to NIOFSDirectory on Linux w/ a non
> Oracle JVM.

I verified that the MMapDirectory is selected by default with the IBM
JVM (it must also contain the right Sun internal classes).

Anyone else have experience with MMapDirectory w/ IBM's JVM?

-Yonik
http://www.lucidimagination.com


> Are you specifically selecting MMapDirectory in solrconfig.xml?
>
> Can you try the Oracle JVM to see if that changes things?
>
> -Yonik
> http://www.lucidimagination.com
>
>
>
> On Fri, Jul 22, 2011 at 5:58 AM, mdz-munich
>  wrote:
>> I was wrong.
>>
>> After rebooting tomcat we discovered a new sweetness:
>>
>> /SEVERE: REFCOUNT ERROR: unreferenced org.apache.solr.core.SolrCore@3c753c75
>> (core.name) has a reference count of 1
>> 22.07.2011 11:52:07 org.apache.solr.common.SolrException log
>> SEVERE: java.lang.RuntimeException: java.io.IOException: Map failed
>>        at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1099)
>>        at org.apache.solr.core.SolrCore.(SolrCore.java:585)
>>        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:463)
>>        at org.apache.solr.core.CoreContainer.load(CoreContainer.java:316)
>>        at org.apache.solr.core.CoreContainer.load(CoreContainer.java:207)
>>        at
>> org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:130)
>>        at
>> org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:94)
>>        at
>> org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:273)
>>        at
>> org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:254)
>>        at
>> org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:372)
>>        at
>> org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:98)
>>        at
>> org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4584)
>>        at
>> org.apache.catalina.core.StandardContext$2.call(StandardContext.java:5262)
>>        at
>> org.apache.catalina.core.StandardContext$2.call(StandardContext.java:5257)
>>        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314)
>>        at java.util.concurrent.FutureTask.run(FutureTask.java:149)
>>        at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
>>        at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
>>        at java.lang.Thread.run(Thread.java:736)
>> Caused by: java.io.IOException: Map failed
>>        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:782)
>>        at
>> org.apache.lucene.store.MMapDirectory$MMapIndexInput.(MMapDirectory.java:264)
>>        at 
>> org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:216)
>>        at org.apache.lucene.index.FieldsReader.(FieldsReader.java:129)
>>        at
>> org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:244)
>>        at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
>>        at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:92)
>>        at 
>> org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:113)
>>        at
>> org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:29)
>>        at
>> org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:81)
>>        at
>> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:750)
>>        at 
>> org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:75)
>>        at org.apache.lucene.index.IndexReader.open(IndexReader.java:428)
>>        at org.apache.lucene.index.IndexReader.open(IndexReader.java:371)
>>        at
>> org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:38)
>>        at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1088)
>>        ... 18 more
>> Caused by: java.lang.OutOfMemoryError: Map failed
>>        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:779)
>>        ... 33 more/
>>
>> Any ideas and/or suggestions?
>>
>> Best regards & thank you,
>>
>> Sebastian
>>
>> --
>> View this message in context: 
>> http://lucene.472066.n3.nabble.com/Solr-3-3-Exception-in-thread-Lucene-Merge-Thread-1-tp3185248p3190976.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>

Re: Geospatial queries in Solr

[Smiley, David W.]

Wrapping the dateline or being able to encircle one of the poles (but not 
necessarily both) are polygon query features that I feel need to be addressed 
before this module is first released (whenever that is), definitely.  And 
arguably before benchmarking, which we're looking to focus on soon.  So "Soon". 
 W/i 2 months?  Just a guess.

~ David
 
On Jul 22, 2011, at 8:18 AM, Jamie Johnson wrote:

> Ah, my mistake then.  I will switch to using the geohash field.  When
> doing my query I did run it against geohash but when I got Russia that
> was more incorrect than point so I stopped using it.
> 
> Is there a timeline by which you expect the dateline issue to be
> addressed?  I don't believe that will be an issue for me since I don't
> believe we have any information which will be from that area of the
> world but I'll have to verify.
> 
> 
> On Fri, Jul 22, 2011 at 3:35 AM, David Smiley (@MITRE.org)
>  wrote:
>> Jamie,
>>  You are using the field named "point" which is based on PointFieldType.
>> Keep in mind that just because this field type is named this way, does *not*
>> mean at all that other fields don't hold points, or that this one is
>> especially suited to it.  Arguably this one is named poorly.  This field
>> type is loosely based on Solr's LatLonType but it doesn't yet fully
>> implement capabilities like crossing the dateline and shapes other than a
>> bounding box.  I've enhanced the code to log a warning if the query shape is
>> expanded to the bounding box. Any way, it's a currently a 2nd class citizen
>> in LSP.
>>  I recommend you use the field named "geohash" which uses a field type name
>> starting with "Recursive...".  Note that the same query you did with this
>> field returned the polygon of Russia.  There is an outstanding box in which
>> we don't properly handle polygons crossing the dateline; other shapes are
>> fine.
>> 
>> ~ David
>> 
>> -
>>  Author: https://www.packtpub.com/solr-1-4-enterprise-search-server/book
>> --
>> View this message in context: 
>> http://lucene.472066.n3.nabble.com/Geospatial-queries-in-Solr-tp3183374p3190762.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>> 

Re: Geospatial queries in Solr

[Jamie Johnson]

Thanks David.  I'm going to continue to play with this, as an FYI you
were spot on, changing to use a geohash field worked with the previous
test.  Again I appreciate all of the information, and awesome work.


On Fri, Jul 22, 2011 at 10:05 AM, Smiley, David W.  wrote:
> Wrapping the dateline or being able to encircle one of the poles (but not 
> necessarily both) are polygon query features that I feel need to be addressed 
> before this module is first released (whenever that is), definitely.  And 
> arguably before benchmarking, which we're looking to focus on soon.  So 
> "Soon".  W/i 2 months?  Just a guess.
>
> ~ David
>
> On Jul 22, 2011, at 8:18 AM, Jamie Johnson wrote:
>
>> Ah, my mistake then.  I will switch to using the geohash field.  When
>> doing my query I did run it against geohash but when I got Russia that
>> was more incorrect than point so I stopped using it.
>>
>> Is there a timeline by which you expect the dateline issue to be
>> addressed?  I don't believe that will be an issue for me since I don't
>> believe we have any information which will be from that area of the
>> world but I'll have to verify.
>>
>>
>> On Fri, Jul 22, 2011 at 3:35 AM, David Smiley (@MITRE.org)
>>  wrote:
>>> Jamie,
>>>  You are using the field named "point" which is based on PointFieldType.
>>> Keep in mind that just because this field type is named this way, does *not*
>>> mean at all that other fields don't hold points, or that this one is
>>> especially suited to it.  Arguably this one is named poorly.  This field
>>> type is loosely based on Solr's LatLonType but it doesn't yet fully
>>> implement capabilities like crossing the dateline and shapes other than a
>>> bounding box.  I've enhanced the code to log a warning if the query shape is
>>> expanded to the bounding box. Any way, it's a currently a 2nd class citizen
>>> in LSP.
>>>  I recommend you use the field named "geohash" which uses a field type name
>>> starting with "Recursive...".  Note that the same query you did with this
>>> field returned the polygon of Russia.  There is an outstanding box in which
>>> we don't properly handle polygons crossing the dateline; other shapes are
>>> fine.
>>>
>>> ~ David
>>>
>>> -
>>>  Author: https://www.packtpub.com/solr-1-4-enterprise-search-server/book
>>> --
>>> View this message in context: 
>>> http://lucene.472066.n3.nabble.com/Geospatial-queries-in-Solr-tp3183374p3190762.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>
>

RE: commit time and lock

[Pierre GOSSE]

Hi Mark

I've read that in a thread title " Weird optimize performance degradation", 
where Erick Erickson states that "Older versions of Lucene would search faster 
on an optimized index, but this is no longer necessary.", and more recently in 
a thread you initiated a month ago "Question about optimization".

I'll also be very interested if anyone had a more precise idea/datas of 
benefits and tradeoff of optimize vs merge ...

Pierre


-Message d'origine-
De : Marc SCHNEIDER [mailto:marc.schneide...@gmail.com] 
Envoyé : vendredi 22 juillet 2011 15:45
À : solr-user@lucene.apache.org
Objet : Re: commit time and lock

Hello,

Pierre, can you tell us where you read that?
"I've read here that optimization is not always a requirement to have an
efficient index, due to some low level changes in lucene 3.xx"

Marc.

On Fri, Jul 22, 2011 at 2:10 PM, Pierre GOSSE wrote:

> Solr will response for search during optimization, but commits will have to
> wait the end of the optimization process.
>
> During optimization a new index is generated on disk by merging every
> single file of the current index into one big file, so you're server will be
> busy, especially regarding disk access. This may alter your response time
> and has very negative effect on the replication of index if you have a
> master/slave architecture.
>
> I've read here that optimization is not always a requirement to have an
> efficient index, due to some low level changes in lucene 3.xx, so maybe you
> don't really need optimization. What version of solr are you using ? Maybe
> someone can point toward a relevant link about optimization other than solr
> wiki
> http://wiki.apache.org/solr/SolrPerformanceFactors#Optimization_Considerations
>
> Pierre
>
>
> -Message d'origine-
> De : Jonty Rhods [mailto:jonty.rh...@gmail.com]
> Envoyé : vendredi 22 juillet 2011 12:45
> À : solr-user@lucene.apache.org
> Objet : Re: commit time and lock
>
> Thanks for clarity.
>
> One more thing I want to know about optimization.
>
> Right now I am planning to optimize the server in 24 hour. Optimization is
> also time taking ( last time took around 13 minutes), so I want to know
> that
> :
>
> 1. when optimization is under process that time will solr server response
> or
> not?
> 2. if server will not response then how to do optimization of server fast
> or
> other way to do optimization so our user will not have to wait to finished
> optimization process.
>
> regards
> Jonty
>
>
>
> On Fri, Jul 22, 2011 at 2:44 PM, Pierre GOSSE  >wrote:
>
> > Solr still respond to search queries during commit, only new indexations
> > requests will have to wait (until end of commit?). So I don't think your
> > users will experience increased response time during commits (unless your
> > server is much undersized).
> >
> > Pierre
> >
> > -Message d'origine-
> > De : Jonty Rhods [mailto:jonty.rh...@gmail.com]
> > Envoyé : jeudi 21 juillet 2011 20:27
> > À : solr-user@lucene.apache.org
> > Objet : Re: commit time and lock
> >
> > Actually i m worried about the response time. i k commiting around 500
> > docs in every 5 minutes. as i know,correct me if i m wrong; at the
> > time of commiting solr server stop responding. my concern is how to
> > minimize the response time so user not need to wait. or any other
> > logic will require for my case. please suggest.
> >
> > regards
> > jonty
> >
> > On Tuesday, June 21, 2011, Erick Erickson 
> wrote:
> > > What is it you want help with? You haven't told us what the
> > > problem you're trying to solve is. Are you asking how to
> > > speed up indexing? What have you tried? Have you
> > > looked at: http://wiki.apache.org/solr/FAQ#Performance?
> > >
> > > Best
> > > Erick
> > >
> > > On Tue, Jun 21, 2011 at 2:16 AM, Jonty Rhods 
> > wrote:
> > >> I am using solrj to index the data. I have around 5 docs indexed.
> As
> > at
> > >> the time of commit due to lock server stop giving response so I was
> > >> calculating commit time:
> > >>
> > >> double starttemp = System.currentTimeMillis();
> > >> server.add(docs);
> > >> server.commit();
> > >> System.out.println("total time in commit = " +
> > (System.currentTimeMillis() -
> > >> starttemp)/1000);
> > >>
> > >> It taking around 9 second to commit the 5000 docs with 15 fields.
> > However I
> > >> am not confirm the lock time of index whether it is start
> > >> since server.add(docs); time or server.commit(); time only.
> > >>
> > >> If I am changing from above to following
> > >>
> > >> server.add(docs);
> > >> double starttemp = System.currentTimeMillis();
> > >> server.commit();
> > >> System.out.println("total time in commit = " +
> > (System.currentTimeMillis() -
> > >> starttemp)/1000);
> > >>
> > >> then commit time becomes less then 1 second. I am not sure which one
> is
> > >> right.
> > >>
> > >> please help.
> > >>
> > >> regards
> > >> Jonty
> > >>
> > >
> >
>

problem searching on non standard characters

[Jason Toy]

How does one search for words with characters like # and +.   I have tried
searching solr with "#test" and "\#test" but all my results always come up
with "test" and not "#test". Is this some kind of configuration option I
need to set in solr?

-- 
- sent from my mobile
6176064373

Re: commit time and lock

[Shawn Heisey]

On 7/22/2011 8:23 AM, Pierre GOSSE wrote:

I've read that in a thread title " Weird optimize performance degradation", where Erick Erickson 
states that "Older versions of Lucene would search faster on an optimized index, but this is no longer 
necessary.", and more recently in a thread you initiated a month ago "Question about 
optimization".

I'll also be very interested if anyone had a more precise idea/datas of 
benefits and tradeoff of optimize vs merge ...


My most recent testing has been with Solr 3.2.0.  I have noticed some 
speedup after optimizing an index, but the gain is not 
earth-shattering.  My index consists of 7 shards.  One of them is small, 
and receives all new documents every two minutes.  The others are large, 
and aside from deletes, are mostly static.  Once a day, the oldest data 
is distributed from the small shard to its proper place in the other six 
shards.


The small shard is optimized once an hour, and usually takes less than a 
minute.  I optimize one large shard every day, so each one gets 
optimized once every six days.  That optimize takes 10-15 minutes.  The 
only reason that I optimize is to remove deleted documents, whatever 
speedup I get is just icing on the cake.  Deleted documents take up 
space and continue to influence the relevance scoring of queries, so I 
want to remove them.


Thanks,
Shawn

Re: problem searching on non standard characters

[François Schiettecatte]

Check your analyzers to make sure that these characters are not getting 
stripped out in the tokenization process, the url for 3.3 is somewhere along 
the lines of:

http://localhost/solr/admin/analysis.jsp?highlight=on

And you should be indeed be searching on "\#test".

François

On Jul 22, 2011, at 10:34 AM, Jason Toy wrote:

> How does one search for words with characters like # and +.   I have tried
> searching solr with "#test" and "\#test" but all my results always come up
> with "test" and not "#test". Is this some kind of configuration option I
> need to set in solr?
> 
> -- 
> - sent from my mobile
> 6176064373

Re: problem searching on non standard characters

[Shawn Heisey]

On 7/22/2011 8:34 AM, Jason Toy wrote:

How does one search for words with characters like # and +.   I have tried
searching solr with "#test" and "\#test" but all my results always come up
with "test" and not "#test". Is this some kind of configuration option I
need to set in solr?


I would guess that your analysis chain (in schema.xml) includes 
something that removes and/or splits terms at non-alphanumeric 
characters.  There are a several components that do this, but 
WordDelimiterFilter is the one that comes to mind most readily.  I've 
never used the StandardTokenizer, but I believe it might do something 
similar.


Thanks,
Shawn

Re: problem searching on non standard characters

[François Schiettecatte]

Adding to my previous reply, I just did a quick check on the 'text_en' and 
'text_en_splitting' field types and they both strip leading '#'.

Cheers

François

On Jul 22, 2011, at 10:49 AM, Shawn Heisey wrote:

> On 7/22/2011 8:34 AM, Jason Toy wrote:
>> How does one search for words with characters like # and +.   I have tried
>> searching solr with "#test" and "\#test" but all my results always come up
>> with "test" and not "#test". Is this some kind of configuration option I
>> need to set in solr?
> 
> I would guess that your analysis chain (in schema.xml) includes something 
> that removes and/or splits terms at non-alphanumeric characters.  There are a 
> several components that do this, but WordDelimiterFilter is the one that 
> comes to mind most readily.  I've never used the StandardTokenizer, but I 
> believe it might do something similar.
> 
> Thanks,
> Shawn
> 

RE: Re: previous and next rows of current record

[Jonathan Rochkind]

 > Yes exactly same problem i m facing. Is there any way to resolve this issue..

I am not sure what you mean, "any way to resolve this issue." Did you read and 
understand what I wrote below? I have nothing more to add.  What is it you're 
looking for?

The way to provide that kind of next/previous is to write application code to 
do it. Although it's not easy to do cleanly a web app because of the 
sessionless architecture of the web. What are you using for your client 
application?  But honestly I probably have nothing more to say on the topic.


>From : Jonathan Rochkind
To : solr-user@lucene.apache.org;
Subject : Re: previous and next rows of current record


I think maybe I know what you mean.

You have a result set generated by a query. You have an item detail page
in your web app -- on that item detail page, you want to give
next/previous buttons for "current search results".

If that's it, read on (although news isn't good), if that's not it,
ignore me.

There is no good way to do it. Although it's not really so much a solr
problem.  As far as Solr is concerned, if you know the query, and you
know the current row into the query i, then just ask Solr for
rows=1&start=$(i=1) to get previous, or i+1 to get next. (You can't send
$(i-1) or $(i+1) to Solr that's just short hand, your app would have to
calculate em and send the literals).

The problem is architecting a web app so when you are on an item detail
page, the app knows what the "current" Solr query was, and what the "i"
index into it was.

The app I work on wants to provide this feature too, but I am so unhappy
with what it currently does (it is both ugly AND does not actually work
at all right on several very common cases), that I am definitely not
going to provide it as an example.  But if you are willing to have your
web app send the "current" search and the index in the URL to the item
detail page, that'd certainly make it easier.

It's not so much a Solr problem -- the answer in Solr is pretty clear.
Keep track of what index into your results you are on, and then just ask
for one previous or more.  But there's no great way to make a web app
taht actually does that without horrid urls.  There's nothing built into
Solr to help you. Solr is pretty much sessionless/stateless, it's got no
idea what the 'current' search for your particular session is.



On 7/21/2011 2:38 PM, Bob Sandiford wrote:
> But - what is it that makes '9' the next id after '5'?  why not '6'?  Or 
> '91238412'? or '4'?
>
> i.e. you still haven't answered the question about what 'next' and 'previous' 
> really means...
>
> But - if you already know that '9' is the next page, why not just do another 
> query with id '9' to get the next record?
>
> Bob Sandiford | Lead Software Engineer | SirsiDynix
> P: 800.288.8020 X6943 | bob.sandif...@sirsidynix.com
> www.sirsidynix.com
>
>
>> -Original Message-
>> From: Jonty Rhods [mailto:jonty.rh...@gmail.com]
>> Sent: Thursday, July 21, 2011 2:33 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: previous and next rows of current record
>>
>> Hi
>>
>> in my case there is no id sequence. id is generated sequence wise for
>> all category. but when we filter by category then same id become
>> random. If i m on detail page which have id 5 and nrxt id is 9 so on
>> same page my requirment is to get next id is 9.
>>
>> On Thursday, July 21, 2011, Bob Sandiford
>>   wrote:
>>> Well, it sort of depends on what you mean by the 'previous' and the
>> 'next' record.
>>> Do you have some type of sequencing built into your concept of your
>> solr / lucene indexes?  Do you have sequential id's?
>>> i.e. What's the use case, and what's the data available to support
>> your use case?
>>> Bob Sandiford | Lead Software Engineer | SirsiDynix
>>> P: 800.288.8020 X6943 | bob.sandif...@sirsidynix.com
>>> www.sirsidynix.com
>>>
 -Original Message-
 From: Jonty Rhods [mailto:jonty.rh...@gmail.com]
 Sent: Thursday, July 21, 2011 2:18 PM
 To: solr-user@lucene.apache.org
 Subject: Re: previous and next rows of current record

 Pls help..

 On Thursday, July 21, 2011, Jonty Rhods
>> wrote:
> Hi,
>
> Is there any special query in solr to get the previous and next
 record of current record.I am getting single record detail using id
 from solr server. I need to get  next and previous on detail page.
> regardsJonty
>
>>>
>>>
>
>

RE: commit time and lock

[Pierre GOSSE]

Merging does not happen often enough to keep deleted documents to a low enough 
count ?

Maybe there's a need to have "partial" optimization available in solr, meaning 
that segment with too much deleted document could be copied to a new file 
without unnecessary datas. That way cleaning deleted datas could be compatible 
with having light replications.

I'm worried by this idea of deleted documents influencing relevance scores, any 
pointer to how important this influence may be ?

Pierre

-Message d'origine-
De : Shawn Heisey [mailto:s...@elyograg.org] 
Envoyé : vendredi 22 juillet 2011 16:42
À : solr-user@lucene.apache.org
Objet : Re: commit time and lock

On 7/22/2011 8:23 AM, Pierre GOSSE wrote:
> I've read that in a thread title " Weird optimize performance degradation", 
> where Erick Erickson states that "Older versions of Lucene would search 
> faster on an optimized index, but this is no longer necessary.", and more 
> recently in a thread you initiated a month ago "Question about optimization".
>
> I'll also be very interested if anyone had a more precise idea/datas of 
> benefits and tradeoff of optimize vs merge ...

My most recent testing has been with Solr 3.2.0.  I have noticed some 
speedup after optimizing an index, but the gain is not 
earth-shattering.  My index consists of 7 shards.  One of them is small, 
and receives all new documents every two minutes.  The others are large, 
and aside from deletes, are mostly static.  Once a day, the oldest data 
is distributed from the small shard to its proper place in the other six 
shards.

The small shard is optimized once an hour, and usually takes less than a 
minute.  I optimize one large shard every day, so each one gets 
optimized once every six days.  That optimize takes 10-15 minutes.  The 
only reason that I optimize is to remove deleted documents, whatever 
speedup I get is just icing on the cake.  Deleted documents take up 
space and continue to influence the relevance scoring of queries, so I 
want to remove them.

Thanks,
Shawn

RE: commit time and lock

[Jonathan Rochkind]

How old is 'older'?  I'm pretty sure I'm still getting much faster performance 
on an optimized index in Solr 1.4. 

This could be due to the nature of my index and queries (which include some 
medium sized stored fields, and extensive facetting -- facetting on up to a 
dozen fields in every request, where each field can include millions of unique 
values. Amazing I can do this with good performance at all!). 

It's also possible i'm wrong about that faster performance, i haven't done 
robustly valid benchmarking on a clone of my production index yet. But it 
really looks like that way to me, from what investigation I have done. 

If the answer is that optimization is believed no longer neccesary on versions 
LATER than 1.4, that might be the simplest explanation. 

From: Pierre GOSSE [pierre.go...@arisem.com]
Sent: Friday, July 22, 2011 10:23 AM
To: solr-user@lucene.apache.org
Subject: RE: commit time and lock

Hi Mark

I've read that in a thread title " Weird optimize performance degradation", 
where Erick Erickson states that "Older versions of Lucene would search faster 
on an optimized index, but this is no longer necessary.", and more recently in 
a thread you initiated a month ago "Question about optimization".

I'll also be very interested if anyone had a more precise idea/datas of 
benefits and tradeoff of optimize vs merge ...

Pierre


-Message d'origine-
De : Marc SCHNEIDER [mailto:marc.schneide...@gmail.com]
Envoyé : vendredi 22 juillet 2011 15:45
À : solr-user@lucene.apache.org
Objet : Re: commit time and lock

Hello,

Pierre, can you tell us where you read that?
"I've read here that optimization is not always a requirement to have an
efficient index, due to some low level changes in lucene 3.xx"

Marc.

On Fri, Jul 22, 2011 at 2:10 PM, Pierre GOSSE wrote:

> Solr will response for search during optimization, but commits will have to
> wait the end of the optimization process.
>
> During optimization a new index is generated on disk by merging every
> single file of the current index into one big file, so you're server will be
> busy, especially regarding disk access. This may alter your response time
> and has very negative effect on the replication of index if you have a
> master/slave architecture.
>
> I've read here that optimization is not always a requirement to have an
> efficient index, due to some low level changes in lucene 3.xx, so maybe you
> don't really need optimization. What version of solr are you using ? Maybe
> someone can point toward a relevant link about optimization other than solr
> wiki
> http://wiki.apache.org/solr/SolrPerformanceFactors#Optimization_Considerations
>
> Pierre
>
>
> -Message d'origine-
> De : Jonty Rhods [mailto:jonty.rh...@gmail.com]
> Envoyé : vendredi 22 juillet 2011 12:45
> À : solr-user@lucene.apache.org
> Objet : Re: commit time and lock
>
> Thanks for clarity.
>
> One more thing I want to know about optimization.
>
> Right now I am planning to optimize the server in 24 hour. Optimization is
> also time taking ( last time took around 13 minutes), so I want to know
> that
> :
>
> 1. when optimization is under process that time will solr server response
> or
> not?
> 2. if server will not response then how to do optimization of server fast
> or
> other way to do optimization so our user will not have to wait to finished
> optimization process.
>
> regards
> Jonty
>
>
>
> On Fri, Jul 22, 2011 at 2:44 PM, Pierre GOSSE  >wrote:
>
> > Solr still respond to search queries during commit, only new indexations
> > requests will have to wait (until end of commit?). So I don't think your
> > users will experience increased response time during commits (unless your
> > server is much undersized).
> >
> > Pierre
> >
> > -Message d'origine-
> > De : Jonty Rhods [mailto:jonty.rh...@gmail.com]
> > Envoyé : jeudi 21 juillet 2011 20:27
> > À : solr-user@lucene.apache.org
> > Objet : Re: commit time and lock
> >
> > Actually i m worried about the response time. i k commiting around 500
> > docs in every 5 minutes. as i know,correct me if i m wrong; at the
> > time of commiting solr server stop responding. my concern is how to
> > minimize the response time so user not need to wait. or any other
> > logic will require for my case. please suggest.
> >
> > regards
> > jonty
> >
> > On Tuesday, June 21, 2011, Erick Erickson 
> wrote:
> > > What is it you want help with? You haven't told us what the
> > > problem you're trying to solve is. Are you asking how to
> > > speed up indexing? What have you tried? Have you
> > > looked at: http://wiki.apache.org/solr/FAQ#Performance?
> > >
> > > Best
> > > Erick
> > >
> > > On Tue, Jun 21, 2011 at 2:16 AM, Jonty Rhods 
> > wrote:
> > >> I am using solrj to index the data. I have around 5 docs indexed.
> As
> > at
> > >> the time of commit due to lock server stop giving response so I was
> > >> calculating commit time:
> > >>
> > >> double starttemp = System.cur

Re: commit time and lock

[Shawn Heisey]

On 7/22/2011 9:32 AM, Pierre GOSSE wrote:

Merging does not happen often enough to keep deleted documents to a low enough 
count ?

Maybe there's a need to have "partial" optimization available in solr, meaning 
that segment with too much deleted document could be copied to a new file without 
unnecessary datas. That way cleaning deleted datas could be compatible with having light 
replications.

I'm worried by this idea of deleted documents influencing relevance scores, any 
pointer to how important this influence may be ?


I've got a pretty high mergeFactor, for fast indexing. Also, I want to 
know for sure and control when merges happen, so I am not leaving it up 
to Lucene/Solr.


Right now the largest number of deleted documents on any shard at this 
moment is 45347.  The shard (17.65GB) contains 9663271 documents, in six 
segments.  That will be one HUGE segment (from the last optimize) and 
five very very tiny segments, each with only a few thousand documents in 
them.  Tonight when the document distribution process runs, that index 
will be optimized again.  Tomorrow night a different shard will be 
optimized.


Deleted documents can (and do) happen anywhere in the index, so even if 
I had a lot of largish segments rather than one huge segment, it's very 
likely that just expunging deletes would still result in the entire 
index being merged, so I am not losing anything by doing a full 
optimize, and I am gaining a small bit of performance.


The 45000 deletes mentioned above represent less than half a percent of 
the shard, so the influence on relevance is *probably* not large ... but 
that's not something I can say definitively.  I think it all depends on 
what people are searching for and how common the terms in the deleted 
documents are.


Thanks,
Shawn

Re: Solr 3.3: Exception in thread "Lucene Merge Thread #1"

[mdz-munich]

Hi Yonik,

thanks for your reply! 

> Are you specifically selecting MMapDirectory in solrconfig.xml? 

Nope.

We installed Oracle's Runtime from 

http://java.com/de/download/linux_manual.jsp?locale=de

/java.runtime.name = Java(TM) SE Runtime Environment
sun.boot.library.path = /usr/java/jdk1.6.0_26/jre/lib/amd64
java.vm.version = 20.1-b02
shared.loader = 
java.vm.vendor = Sun Microsystems Inc.
enable.master = true
java.vendor.url = http://java.sun.com/
path.separator = :
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
tomcat.util.buf.StringCache.byte.enabled = true
file.encoding.pkg = sun.io
java.util.logging.config.file =
/local/master01_tomcat7x_solr33x/conf/logging.properties
user.country = DE
sun.java.launcher = SUN_STANDARD
sun.os.patch.level = unknown
java.vm.specification.name = Java Virtual Machine Specification
user.dir = /local/master01_tomcat7x_solr33x/logs
solr.abortOnConfigurationError = true
java.runtime.version = 1.6.0_26-b03
java.awt.graphicsenv = sun.awt.X11GraphicsEnvironment
java.endorsed.dirs = /local/master01_tomcat7x_solr33x/endorsed
os.arch = amd64
java.io.tmpdir = /local/master01_tomcat7x_solr33x/temp
line.separator =  /

But no success with 1000 docs/batch, this was thrown during optimize: 

/
22.07.2011 18:44:05 org.apache.solr.core.SolrCore execute
INFO: [core.digi20] webapp=/solr path=/update params={} status=500
QTime=87540 
22.07.2011 18:44:05 org.apache.solr.common.SolrException log
SEVERE: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
at
org.apache.lucene.store.MMapDirectory$MMapIndexInput.(MMapDirectory.java:303)
at 
org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:217)
at org.apache.lucene.index.FieldsReader.(FieldsReader.java:129)
at
org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:245)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:117)
at 
org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:703)
at 
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4196)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3863)
at
org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37)
at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2715)
at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2525)
at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2462)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:410)
at
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
at
org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:154)
at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:177)
at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:77)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:67)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:240)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:164)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:462)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:164)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:563)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:403)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:301)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:162)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:309)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.ni

Re: Solr 3.3: Exception in thread "Lucene Merge Thread #1"

[Yonik Seeley]

OK, best guess is that you're going over some per-process address space limit.

Try seeing what "ulimit -a" says.

-Yonik
http://www.lucidimagination.com

On Fri, Jul 22, 2011 at 12:51 PM, mdz-munich
 wrote:
> Hi Yonik,
>
> thanks for your reply!
>
>> Are you specifically selecting MMapDirectory in solrconfig.xml?
>
> Nope.
>
> We installed Oracle's Runtime from
>
> http://java.com/de/download/linux_manual.jsp?locale=de
>
> /java.runtime.name = Java(TM) SE Runtime Environment
> sun.boot.library.path = /usr/java/jdk1.6.0_26/jre/lib/amd64
> java.vm.version = 20.1-b02
> shared.loader =
> java.vm.vendor = Sun Microsystems Inc.
> enable.master = true
> java.vendor.url = http://java.sun.com/
> path.separator = :
> java.vm.name = Java HotSpot(TM) 64-Bit Server VM
> tomcat.util.buf.StringCache.byte.enabled = true
> file.encoding.pkg = sun.io
> java.util.logging.config.file =
> /local/master01_tomcat7x_solr33x/conf/logging.properties
> user.country = DE
> sun.java.launcher = SUN_STANDARD
> sun.os.patch.level = unknown
> java.vm.specification.name = Java Virtual Machine Specification
> user.dir = /local/master01_tomcat7x_solr33x/logs
> solr.abortOnConfigurationError = true
> java.runtime.version = 1.6.0_26-b03
> java.awt.graphicsenv = sun.awt.X11GraphicsEnvironment
> java.endorsed.dirs = /local/master01_tomcat7x_solr33x/endorsed
> os.arch = amd64
> java.io.tmpdir = /local/master01_tomcat7x_solr33x/temp
> line.separator =  /
>
> But no success with 1000 docs/batch, this was thrown during optimize:
>
> /
> 22.07.2011 18:44:05 org.apache.solr.core.SolrCore execute
> INFO: [core.digi20] webapp=/solr path=/update params={} status=500
> QTime=87540
> 22.07.2011 18:44:05 org.apache.solr.common.SolrException log
> SEVERE: java.io.IOException: Map failed
>        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
>        at
> org.apache.lucene.store.MMapDirectory$MMapIndexInput.(MMapDirectory.java:303)
>        at 
> org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:217)
>        at org.apache.lucene.index.FieldsReader.(FieldsReader.java:129)
>        at
> org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:245)
>        at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:117)
>        at 
> org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:703)
>        at 
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4196)
>        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3863)
>        at
> org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37)
>        at 
> org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2715)
>        at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2525)
>        at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2462)
>        at
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:410)
>        at
> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
>        at
> org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:154)
>        at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:177)
>        at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:77)
>        at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:67)
>        at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
>        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
>        at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
>        at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
>        at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>        at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>        at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:240)
>        at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:164)
>        at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:462)
>        at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:164)
>        at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
>        at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:563)
>        at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>        at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:403)
>        at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:301)
>        at
> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:162)
>        at
> org.apa

Re: Solr 3.3: Exception in thread "Lucene Merge Thread #1"

[mdz-munich]

It says:

/core file size  (blocks, -c) 0
data seg size   (kbytes, -d) unlimited
scheduling priority (-e) 0
file size   (blocks, -f) unlimited
pending signals (-i) 257869
max locked memory   (kbytes, -l) 64
max memory size (kbytes, -m) 28063940
open files  (-n) 8192
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority  (-r) 0
stack size  (kbytes, -s) 8192
cpu time   (seconds, -t) unlimited
max user processes  (-u) 257869
virtual memory  (kbytes, -v) 27216080
file locks  (-x) unlimited/


Best regards,

Sebastian



Yonik Seeley-2-2 wrote:
> 
> OK, best guess is that you're going over some per-process address space
> limit.
> 
> Try seeing what "ulimit -a" says.
> 
> -Yonik
> http://www.lucidimagination.com
> 
> On Fri, Jul 22, 2011 at 12:51 PM, mdz-munich
>  wrote:
>> Hi Yonik,
>>
>> thanks for your reply!
>>
>>> Are you specifically selecting MMapDirectory in solrconfig.xml?
>>
>> Nope.
>>
>> We installed Oracle's Runtime from
>>
>> http://java.com/de/download/linux_manual.jsp?locale=de
>>
>> /java.runtime.name = Java(TM) SE Runtime Environment
>> sun.boot.library.path = /usr/java/jdk1.6.0_26/jre/lib/amd64
>> java.vm.version = 20.1-b02
>> shared.loader =
>> java.vm.vendor = Sun Microsystems Inc.
>> enable.master = true
>> java.vendor.url = http://java.sun.com/
>> path.separator = :
>> java.vm.name = Java HotSpot(TM) 64-Bit Server VM
>> tomcat.util.buf.StringCache.byte.enabled = true
>> file.encoding.pkg = sun.io
>> java.util.logging.config.file =
>> /local/master01_tomcat7x_solr33x/conf/logging.properties
>> user.country = DE
>> sun.java.launcher = SUN_STANDARD
>> sun.os.patch.level = unknown
>> java.vm.specification.name = Java Virtual Machine Specification
>> user.dir = /local/master01_tomcat7x_solr33x/logs
>> solr.abortOnConfigurationError = true
>> java.runtime.version = 1.6.0_26-b03
>> java.awt.graphicsenv = sun.awt.X11GraphicsEnvironment
>> java.endorsed.dirs = /local/master01_tomcat7x_solr33x/endorsed
>> os.arch = amd64
>> java.io.tmpdir = /local/master01_tomcat7x_solr33x/temp
>> line.separator =  /
>>
>> But no success with 1000 docs/batch, this was thrown during optimize:
>>
>> /
>> 22.07.2011 18:44:05 org.apache.solr.core.SolrCore execute
>> INFO: [core.digi20] webapp=/solr path=/update params={} status=500
>> QTime=87540
>> 22.07.2011 18:44:05 org.apache.solr.common.SolrException log
>> SEVERE: java.io.IOException: Map failed
>>        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
>>        at
>> org.apache.lucene.store.MMapDirectory$MMapIndexInput.(MMapDirectory.java:303)
>>        at
>> org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:217)
>>        at
>> org.apache.lucene.index.FieldsReader.(FieldsReader.java:129)
>>        at
>> org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:245)
>>        at
>> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:117)
>>        at
>> org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:703)
>>        at
>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4196)
>>        at
>> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3863)
>>        at
>> org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37)
>>        at
>> org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2715)
>>        at
>> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2525)
>>        at
>> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2462)
>>        at
>> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:410)
>>        at
>> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
>>        at
>> org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:154)
>>        at
>> org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:177)
>>        at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:77)
>>        at
>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:67)
>>        at
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
>>        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
>>        at
>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
>>        at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
>>        at
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>>        at
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>        at
>> org.apache.catalina.core.StandardWrapperValve

Re: Solr 3.3: Exception in thread "Lucene Merge Thread #1"

[Yonik Seeley]

Yep, there ya go... your OS configuration is limiting you to 27G of
virtual address space per process.  Consider setting that to
unlimited.

-Yonik
http://www.lucidimagination.com



On Fri, Jul 22, 2011 at 1:05 PM, mdz-munich
 wrote:
> It says:
>
> /core file size          (blocks, -c) 0
> data seg size           (kbytes, -d) unlimited
> scheduling priority             (-e) 0
> file size               (blocks, -f) unlimited
> pending signals                 (-i) 257869
> max locked memory       (kbytes, -l) 64
> max memory size         (kbytes, -m) 28063940
> open files                      (-n) 8192
> pipe size            (512 bytes, -p) 8
> POSIX message queues     (bytes, -q) 819200
> real-time priority              (-r) 0
> stack size              (kbytes, -s) 8192
> cpu time               (seconds, -t) unlimited
> max user processes              (-u) 257869
> virtual memory          (kbytes, -v) 27216080
> file locks                      (-x) unlimited/
>
>
> Best regards,
>
> Sebastian
>
>
>
> Yonik Seeley-2-2 wrote:
>>
>> OK, best guess is that you're going over some per-process address space
>> limit.
>>
>> Try seeing what "ulimit -a" says.
>>
>> -Yonik
>> http://www.lucidimagination.com
>>
>> On Fri, Jul 22, 2011 at 12:51 PM, mdz-munich
>>  wrote:
>>> Hi Yonik,
>>>
>>> thanks for your reply!
>>>
 Are you specifically selecting MMapDirectory in solrconfig.xml?
>>>
>>> Nope.
>>>
>>> We installed Oracle's Runtime from
>>>
>>> http://java.com/de/download/linux_manual.jsp?locale=de
>>>
>>> /java.runtime.name = Java(TM) SE Runtime Environment
>>> sun.boot.library.path = /usr/java/jdk1.6.0_26/jre/lib/amd64
>>> java.vm.version = 20.1-b02
>>> shared.loader =
>>> java.vm.vendor = Sun Microsystems Inc.
>>> enable.master = true
>>> java.vendor.url = http://java.sun.com/
>>> path.separator = :
>>> java.vm.name = Java HotSpot(TM) 64-Bit Server VM
>>> tomcat.util.buf.StringCache.byte.enabled = true
>>> file.encoding.pkg = sun.io
>>> java.util.logging.config.file =
>>> /local/master01_tomcat7x_solr33x/conf/logging.properties
>>> user.country = DE
>>> sun.java.launcher = SUN_STANDARD
>>> sun.os.patch.level = unknown
>>> java.vm.specification.name = Java Virtual Machine Specification
>>> user.dir = /local/master01_tomcat7x_solr33x/logs
>>> solr.abortOnConfigurationError = true
>>> java.runtime.version = 1.6.0_26-b03
>>> java.awt.graphicsenv = sun.awt.X11GraphicsEnvironment
>>> java.endorsed.dirs = /local/master01_tomcat7x_solr33x/endorsed
>>> os.arch = amd64
>>> java.io.tmpdir = /local/master01_tomcat7x_solr33x/temp
>>> line.separator =  /
>>>
>>> But no success with 1000 docs/batch, this was thrown during optimize:
>>>
>>> /
>>> 22.07.2011 18:44:05 org.apache.solr.core.SolrCore execute
>>> INFO: [core.digi20] webapp=/solr path=/update params={} status=500
>>> QTime=87540
>>> 22.07.2011 18:44:05 org.apache.solr.common.SolrException log
>>> SEVERE: java.io.IOException: Map failed
>>>        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748)
>>>        at
>>> org.apache.lucene.store.MMapDirectory$MMapIndexInput.(MMapDirectory.java:303)
>>>        at
>>> org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:217)
>>>        at
>>> org.apache.lucene.index.FieldsReader.(FieldsReader.java:129)
>>>        at
>>> org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:245)
>>>        at
>>> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:117)
>>>        at
>>> org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:703)
>>>        at
>>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4196)
>>>        at
>>> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3863)
>>>        at
>>> org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:37)
>>>        at
>>> org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2715)
>>>        at
>>> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2525)
>>>        at
>>> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2462)
>>>        at
>>> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:410)
>>>        at
>>> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
>>>        at
>>> org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:154)
>>>        at
>>> org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:177)
>>>        at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:77)
>>>        at
>>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:67)
>>>        at
>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
>>>        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
>>>        at
>>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
>>

Re: Solr 3.3: Exception in thread "Lucene Merge Thread #1"

[mdz-munich]

Maybe it's important:

- The OS (Open Suse 10) is virtualized on VMWare
- Network Attached Storage

Best regards

Sebastian


--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-3-3-Exception-in-thread-Lucene-Merge-Thread-1-tp3185248p3191986.html
Sent from the Solr - User mailing list archive at Nabble.com.

Rounding errors in solr

[Brian Lamb]

Hi all,

I've noticed some peculiar scoring issues going on in my application. For
example, I have a field that is multivalued and has several records that
have the same value. For example,


  National Society of Animal Lovers
  Nat. Soc. of Ani. Lov.


I have about 300 records with that exact value.

Now, when I do a search for references:(national society animal lovers), I
get the following results:

252
159
82
452
105

When I do a search for references:(nat soc ani lov), I get the results
ordered differently:

510
122
501
82
252

When I load all the records that match, I notice that at some point, the
scores aren't the same but differ by only a little:

1.471928 in one and the one before it was 1.471929

I turned on debugQuery=on and the scores for each of those two records are
exactly the same. Therefore, I think there is some kind of rounding error
going on.

Is there a way I can fix this?

Alternatively, can I sort by a rounded version of the score? I tried
sort=round(score,5) but I get the following message:

Can't determine Sort Order: 'round(score,5) ', pos=5

I also tried sort=sum(score,1) just to see if I was using round
incorrectly but I get an error message there too saying score is not a
recognized field.

Please help!

Thanks,

Brian Lamb

Re: Rounding errors in solr

[Yonik Seeley]

On Fri, Jul 22, 2011 at 4:11 PM, Brian Lamb
 wrote:
> I've noticed some peculiar scoring issues going on in my application. For
> example, I have a field that is multivalued and has several records that
> have the same value. For example,
>
> 
>  National Society of Animal Lovers
>  Nat. Soc. of Ani. Lov.
> 
>
> I have about 300 records with that exact value.
>
> Now, when I do a search for references:(national society animal lovers), I
> get the following results:
>
> 252
> 159
> 82
> 452
> 105
>
> When I do a search for references:(nat soc ani lov), I get the results
> ordered differently:
>
> 510
> 122
> 501
> 82
> 252
>
> When I load all the records that match, I notice that at some point, the
> scores aren't the same but differ by only a little:
>
> 1.471928 in one and the one before it was 1.471929

32 bit floats only have 7 decimal digits of precision, and in floating
point land (a+b+c) can be slightly different than (c+b+a)

-Yonik
http://www.lucidimagination.com

saving timestamps in trunk broken?

[Jason Toy]

In Solr 1.3.1 I am able to store timestamps in my docs so that I query them.

In trunk when I try to store a doc with a timestamp I get a sever error, is
there a different way I should store this data or is this a bug?


Jul 22, 2011 7:20:14 PM org.apache.solr.update.processor.LogUpdateProcessor
finish
INFO: {add=[Post 75004824785129473]} 0 0
Jul 22, 2011 7:20:14 PM org.apache.solr.common.SolrException log
SEVERE: org.apache.solr.common.SolrException: ERROR: [doc=Post
75004824785129473] Error adding field 'at_d'='2011-05-30T01:05:18Z'
at
org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:300)
at
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:60)
at
org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:115)
at
org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:127)
at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:77)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:67)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1401)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:353)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:248)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.lang.NumberFormatException: For input string:
"2011-05-30T01:05:18Z"
at
sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1222)
at java.lang.Double.parseDouble(Double.java:510)
at org.apache.solr.schema.TrieField.createField(TrieField.java:509)
at
org.apache.solr.schema.SchemaField.createField(SchemaField.java:107)
at
org.apache.solr.update.DocumentBuilder.addField(DocumentBuilder.java:202)
at
org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:261)

-- 
- sent from my mobile
6176064373

Re: saving timestamps in trunk broken?

[Chris Hostetter]

: In Solr 1.3.1 I am able to store timestamps in my docs so that I query them.
: 
: In trunk when I try to store a doc with a timestamp I get a sever error, is
: there a different way I should store this data or is this a bug?

are you sure your schema has that field defined as a (Trie)DateField ?

Based on the error it looks like it's configured as a "TrieDoubleField" 
...

: 75004824785129473] Error adding field 'at_d'='2011-05-30T01:05:18Z'
...
: org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
: Caused by: java.lang.NumberFormatException: For input string:
: "2011-05-30T01:05:18Z"
: at
: sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1222)
: at java.lang.Double.parseDouble(Double.java:510)
: at org.apache.solr.schema.TrieField.createField(TrieField.java:509)


-Hoss

Re: saving timestamps in trunk broken?

[Jason Toy]

I haven't modified my schema in the older solr or trunk solr,is it required
to modify my schema to support timestamps?

On Fri, Jul 22, 2011 at 4:45 PM, Chris Hostetter
wrote:

> : In Solr 1.3.1 I am able to store timestamps in my docs so that I query
> them.
> :
> : In trunk when I try to store a doc with a timestamp I get a sever error,
> is
> : there a different way I should store this data or is this a bug?
>
> are you sure your schema has that field defined as a (Trie)DateField ?
>
> Based on the error it looks like it's configured as a "TrieDoubleField"
> ...
>
> : 75004824785129473] Error adding field 'at_d'='2011-05-30T01:05:18Z'
> ...
> :
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> : Caused by: java.lang.NumberFormatException: For input string:
> : "2011-05-30T01:05:18Z"
> : at
> : sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1222)
> : at java.lang.Double.parseDouble(Double.java:510)
> : at
> org.apache.solr.schema.TrieField.createField(TrieField.java:509)
>
>
> -Hoss
>



-- 
- sent from my mobile
6176064373

Spellcheck compounded words

[O. Klein]

How do I get spellchecker to suggest compounded words?

Like. q=sail booat

and suggestion/collate is "sailboat" and "sail boat"

--
View this message in context: 
http://lucene.472066.n3.nabble.com/Spellcheck-compounded-words-tp3192748p3192748.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: saving timestamps in trunk broken?

[Jason Toy]

This is the document I am posting:
Post
75004824785129473Post2011-05-30T01:05:18ZNew
YorkUnited Stateshello world!


In my schema.xml file I have these date fields, do I need more?

 
 
 





On Fri, Jul 22, 2011 at 5:00 PM, Jason Toy  wrote:

> I haven't modified my schema in the older solr or trunk solr,is it required
> to modify my schema to support timestamps?
>
>
> On Fri, Jul 22, 2011 at 4:45 PM, Chris Hostetter  > wrote:
>
>> : In Solr 1.3.1 I am able to store timestamps in my docs so that I query
>> them.
>> :
>> : In trunk when I try to store a doc with a timestamp I get a sever error,
>> is
>> : there a different way I should store this data or is this a bug?
>>
>> are you sure your schema has that field defined as a (Trie)DateField ?
>>
>> Based on the error it looks like it's configured as a "TrieDoubleField"
>> ...
>>
>> : 75004824785129473] Error adding field 'at_d'='2011-05-30T01:05:18Z'
>> ...
>> :
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>> : Caused by: java.lang.NumberFormatException: For input string:
>> : "2011-05-30T01:05:18Z"
>> : at
>> : sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1222)
>> : at java.lang.Double.parseDouble(Double.java:510)
>> : at
>> org.apache.solr.schema.TrieField.createField(TrieField.java:509)
>>
>>
>> -Hoss
>>
>
>
>
> --
> - sent from my mobile
> 6176064373
>
>


-- 
- sent from my mobile
6176064373

Re: saving timestamps in trunk broken?

[Jason Toy]

Hi Chris, you were correct, the filed was getting set as a double. Thanks
for the help.



On Fri, Jul 22, 2011 at 7:03 PM, Jason Toy  wrote:

> This is the document I am posting:
> Post
> 75004824785129473Post name="at_d">2011-05-30T01:05:18ZNew
> YorkUnited States name="data_text">hello world!
>
>
> In my schema.xml file I have these date fields, do I need more?
>
>   precisionStep="0" positionIncrementGap="0"/>
>   precisionStep="6" positionIncrementGap="0"/>
>   omitNorms="true"/>
>
>
>
>
>
> On Fri, Jul 22, 2011 at 5:00 PM, Jason Toy  wrote:
>
>> I haven't modified my schema in the older solr or trunk solr,is it
>> required to modify my schema to support timestamps?
>>
>>
>> On Fri, Jul 22, 2011 at 4:45 PM, Chris Hostetter <
>> hossman_luc...@fucit.org> wrote:
>>
>>> : In Solr 1.3.1 I am able to store timestamps in my docs so that I query
>>> them.
>>> :
>>> : In trunk when I try to store a doc with a timestamp I get a sever
>>> error, is
>>> : there a different way I should store this data or is this a bug?
>>>
>>> are you sure your schema has that field defined as a (Trie)DateField ?
>>>
>>> Based on the error it looks like it's configured as a "TrieDoubleField"
>>> ...
>>>
>>> : 75004824785129473] Error adding field 'at_d'='2011-05-30T01:05:18Z'
>>> ...
>>> :
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> : Caused by: java.lang.NumberFormatException: For input string:
>>> : "2011-05-30T01:05:18Z"
>>> : at
>>> :
>>> sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1222)
>>> : at java.lang.Double.parseDouble(Double.java:510)
>>> : at
>>> org.apache.solr.schema.TrieField.createField(TrieField.java:509)
>>>
>>>
>>> -Hoss
>>>
>>
>>
>>
>> --
>> - sent from my mobile
>> 6176064373
>>
>>
>
>
> --
> - sent from my mobile
> 6176064373
>
>


-- 
- sent from my mobile
6176064373

Problem in execution of error call in ajax request if solr server is not running

[Romi]

*$.ajax({
url:
solrURL+"/solr/db/select/?qt=dismax&wt=json&&start="+start+"&rows="+end+"&q="+query+"&json.wrf=?",
async:false,
dataType: 'json',
success: function(){
   
getSolrResponse(sort,order,itemPerPage,showPage,query,solrURL);  
}
,error:function(){

alert("there is an error");
}

})*

I have this ajax request, but even if solr url is not correct, than also
error code is not executing, why so. please tell me the solution. i have
tried with both async:false and async:true. in both cases error not called
if server is shutdown

-
Thanks & Regards
Romi
--
View this message in context: 
http://lucene.472066.n3.nabble.com/Problem-in-execution-of-error-call-in-ajax-request-if-solr-server-is-not-running-tp3192962p3192962.html
Sent from the Solr - User mailing list archive at Nabble.com.

Error with custom search component which adds filter

[Jamie Johnson]

I have a custom search component which does the following in process

 SolrQueryRequest req = rb.req;
 SolrParams params = req.getParams();


 QueryWrapperFilter qwf = new QueryWrapperFilter(rb.getQuery());

 Filter filter = new TestFilter(qwf);
 ConstantScoreQuery fq = new ConstantScoreQuery(filter);
 rb.setQuery(fq);

this essentially filters the result set using TestFilter which does
this in getDocIdSet

return new FilteredDocSet(startingFilter.getDocIdSet(readerCtx));

where FilteredDocSet always returns true from match.  When a query is
executed that returns 0 results an IllegalArgumentException is thrown
from org.apache.lucene.search.FilteredDocIdSetIterator constructor

to fix this in FilteredDocSet I added the following:

@Override
public DocIdSetIterator iterator() throws IOException {
try {
return super.iterator();
} catch(IllegalArgumentException e){
return null;
}
}

What is the correct way to deal with this?  Am I getting this error
because I'm misusing something?