without the indexed property is set to true by default?

[joo]

Indexed properties in a constant field current to the field, I did not give
the search.
indexed attribute is set to true by default, does not turn you on?



--
View this message in context: 
http://lucene.472066.n3.nabble.com/without-the-indexed-property-is-set-to-true-by-default-tp4060973.html
Sent from the Solr - User mailing list archive at Nabble.com.

Questions about the performance of Solr

[joo]

Search speed at which data is loaded is more than 7 ten millon current will
be reduced too.
About 50 seconds it will take, but the number is often just this, it is not
possible to know whether such.
Will there is a problem with the Query I use it to know the Query Optimizing
Solr and fall.
The Query, for example I use,
time: [time to time] AND category: (1,2) AND (message1: message OR message2:
message)
I try to this.
As long as there is no this problem, you need advice please do take a look
at which part.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Questions-about-the-performance-of-Solr-tp4060988.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Questions about the performance of Solr

[joo]

Thank you.
However, fq is already in use.
In my opinion, it is to think that it might be slow data of 70 million
reviews is contained in the core of one, but do you have examples of
performance of a certain number or more may decrease maybe?



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Questions-about-the-performance-of-Solr-tp4060988p4061461.html
Sent from the Solr - User mailing list archive at Nabble.com.

statsCache issue

[Jae Joo]

Solr Version: 5.2.1

Container: Tomcat (still).

in SolrConfig.xml:




However, I see the class is not plugged in.

in log file:

org.apache.solr.core.SolrCore; Using default statsCache cache:
org.apache.solr.search.stats.LocalStatsCache


Any reason why?


Thanks,


Jae

Re: statsCache issue

[Jae Joo]

Thanks for your tip. Let me test in 5.3.



On Wed, Sep 9, 2015 at 4:23 PM, Markus Jelsma 
wrote:

> Hello - there are several issues with StatsCache < 5.3. If it  is loaded,
> it won't work reliably. We are using it properly on 5.3. Statistics may be
> a bit off if you are using BM25 though. You should upgrade to 5.3.
>
> Markus
>
> -Original message-
> > From:Jae Joo 
> > Sent: Wednesday 9th September 2015 21:23
> > To: solr-user@lucene.apache.org
> > Subject: statsCache issue
> >
> > Solr Version: 5.2.1
> >
> > Container: Tomcat (still).
> >
> > in SolrConfig.xml:
> >
> > 
> >
> >
> > However, I see the class is not plugged in.
> >
> > in log file:
> >
> > org.apache.solr.core.SolrCore; Using default statsCache cache:
> > org.apache.solr.search.stats.LocalStatsCache
> >
> >
> > Any reason why?
> >
> >
> > Thanks,
> >
> >
> > Jae
> >
>

SolrCloud - Replica is showen as "Recovery-Failed"

[Jae Joo]

Solr Version " 5.3

I just built the SoleCloud with 5 shards and 3 replicationfactor in 15
nodes. It means that I have shards and replicas running in it's own servers.

When I see the Cloud page, I see that the status of replica is
"recovery-failed".
For testing, I downed the leader, but a replica couldn't be a leader
because it's status was not active.

NFO  - 2015-10-19 16:46:16.297;
org.apache.solr.cloud.ShardLeaderElectionContext; My last published State
was recovery_failed, I won't be the leader.

There is no document indexed..

Any help?

Jae

Re: SolrCloud - Replica is showen as "Recovery-Failed"

[Jae Joo]

Found the root cause. I disabled the transaction log.

Thanks,

On Mon, Oct 19, 2015 at 1:07 PM, Jae Joo  wrote:

> Solr Version " 5.3
>
> I just built the SoleCloud with 5 shards and 3 replicationfactor in 15
> nodes. It means that I have shards and replicas running in it's own servers.
>
> When I see the Cloud page, I see that the status of replica is
> "recovery-failed".
> For testing, I downed the leader, but a replica couldn't be a leader
> because it's status was not active.
>
> NFO  - 2015-10-19 16:46:16.297;
> org.apache.solr.cloud.ShardLeaderElectionContext; My last published State
> was recovery_failed, I won't be the leader.
>
> There is no document indexed..
>
> Any help?
>
> Jae
>

PatternReplaceCharFilterfactor and Position

[Jae Joo]

I am having some issue regarding "start" and "End" position of token.
Here is the CharFilterFactory.




Then the input data is

1

In the Analysis page,
textraw_bytesstartendpositionLengthtypeposition
1[31]21311word1

Should the "end" position "22"? It breaks the Highlighting...
HTMLStripCharFilterFactory is working properly

Any help?


Jae

Synonyms - 20th and 20

[Jae Joo]

I have a synonyms.txt file which has
20th,twentieth

Once I apply the synonym, I see "20th", "twentieth" and "20" for "20th".
Does anyone know where "20" comes from? How can I have only "20th" and
"twentieth"?

Thanks,

Jae

Range query and Highlighting

[Jae Joo]

If I use a combined query - range query and others (term query), all terms
in field matched is highlighted. Any way to highlight only the term(s) in
term query?
Here is example.

+date:{20031231 TO *] +(title:red)

It highlight all terms except stopword.


using fq would not be an option because there may be multiple term queries
and boolean queries combined.


Any idea?


Jae

Synonym - multiple words and position

[Jae Joo]

In the synonym file,
antigravity, anti gravity

In the analysis, I see the position of anti is 1 and gravity is 2.
Is there any way to keep  postions of "anti" and "gravity" to 1?
And any ways to configure or define  to have synonym "anti gravity" rather
than "anti" and "gravity" for "antigravity"

Thanks,

Jae

field specified edismax

[Jae Joo]

Any way to apply different edismax parameter to field by field?
For ex.
q=keywords:(lung cancer) AND title:chemotherapy

I would like to apply different qf for fields, keywords and title.
f.keywords.qf=keywords^40 subkeywords^20
f.title.qf=title^80 subtitle^20

I know it can be done by field aliasing, but doesn't like to use field
aliasing.

Thanks,

Jae

WordDelimiterFilterFactory and PatternReplaceCharFilterFactory

[Jae Joo]

Hi,

Once I apply PatternReplaceCharFilterFactory to the input string, the
position of token is changed.
Here is an example.



In the analysis page,
p-xylene and p-xylene (without xml tags) have
different positions.

for p-xylene,
p-xylene --> 1
xylene --> 2
p --> 2
pxylene -->

However, for the term (without tags) p-xylene,
p-xylene --> 1
p --> 1
xylene --> 2
pxylene --> 3

Only difference I can see is the start and end position because of xml tag.

Does any one know why?

Thanks,

Jae Joo

Spatial search - Solr 4.0

[Jae Joo]

Hi,

I am implementing spatial search and found some odd things. As I know that
the returning distance is still being implemented, so I have implement
algorithm to calculate the actual distance based on lat and long returned.
when I do it, I have found the sort is not working properly. Any thing I
missed?

Jae

Solr Sharding and idf

[Jae Joo]

Is there still issue regarding distributed idf in sharding environment in
Solr 1.4 or 4.0?
If yes, any suggestions to resolve it?

Thanks,

Jae

Re: Solr Sharding and idf

[Jae Joo]

Yes, I knew that the ticket is still open. This is why I am looking for the
solutions now.

2011/3/2 Tomás Fernández Löbbe 

> Hi Jae, this is the Jira created for the problem of IDF on distributed
> search:
>
> https://issues.apache.org/jira/browse/SOLR-1632
>
> It's still open
>
> On Wed, Mar 2, 2011 at 1:48 PM, Upayavira  wrote:
>
> > As I understand it there is, and the best you can do is keep the same
> > number of docs per shard, and keep your documents randomised across
> > shards. That way you'll minimise the chances of suffering from
> > distributed IDF issues.
> >
> > Upayavira
> >
> > On Wed, 02 Mar 2011 10:10 -0500, "Jae Joo"  wrote:
> > > Is there still issue regarding distributed idf in sharding environment
> in
> > > Solr 1.4 or 4.0?
> > > If yes, any suggestions to resolve it?
> > >
> > > Thanks,
> > >
> > > Jae
> > >
> > ---
> > Enterprise Search Consultant at Sourcesense UK,
> > Making Sense of Open Source
> >
> >
>

NRT in Solr

[Jae Joo]

Hi,
Is NRT in Solr 4.0 from trunk? I have checkouted from Trunk, but could not
find the configuration for NRT.

Regards

Jae

Retrieving Ranking (Position)

[Jae Joo]

Hi,

I am looking for the way to retrieve a ranking (or position) of  the
document matched  in the result set.

I can get the data, then parse it to find the position of the document
matched, but am looking for the way if there is a feature.

Thanks,

Jae

multi words synonyms

[Jae Joo]

Hi,

I would like to make the synonym for internal medicine to physician or
doctor. but it is not working properly. Anyone help me?

synonym.index.txt
internal medicine  => physician

synonyms.query.txt
physician, internal medicine  => physician, doctor

In the Analysis tool, I can see clearly that internal medicine is converted
to physician and doctor in index and querying times, but when actual query,
it is not converted (with debugQuery=true paprameter).


internal medicine
internal medicine
job:intern job:medicin
job:intern job:medicin

It returns

1.3963256
874878_INTERNATIONAL CONSULTANTS


Here is what I have in schema.xml
   
  
  

   
  
  

solr.RemoveDuplicatesTokenFilterFactory

[Jae Joo]

Hi,

Here is the string to be indexed without duplication.

Kitchen Cabinet Utah Kitchen Remodeling Utah

Is RemoveDuplicatesTokenFilterFactory for this solution? or for something
else?

Jae

dismax and standard query types

[Jae Joo]

If there any way to mix dismax and standard query type?

Here is the example..

(wendys columbus) AND state:Georgia

wendys columbus --> qt=dismax

Thanks,

Jae Joo

Re: Out of Memory Errors

[Jae Joo]

Here is what I am doing to check the memory statues.
1. Run the Servelt and Solr application.
2. On command prompt, jstat -gc  5s (5s means that getting data every 5
seconds.)
3. Watch it or pipe to the file.
4. Analyze the data gathered.

Jae

On Tue, Oct 21, 2008 at 9:48 PM, Willie Wong <[EMAIL PROTECTED]>wrote:

> Hello,
>
> I've been having issues with out of memory errors on searches in Solr. I
> was wondering if I'm hitting a limit with solr or if I've configured
> something seriously wrong.
>
> Solr Setup
> - 3 cores
> - 3163615 documents each
> - 10 GB size
> - approx 10 fields
> - document sizes vary from a few kb to a few MB
> - no faceting is used however the search query can be fairly complex with
> 8 or more fields being searched on at once
>
> Environment:
> - windows 2003
> - 2.8 GHz zeon processor
> - 1.5 GB memory assigned to solr
> - Jetty 6 server
>
> Once we get to around a few  concurrent users OOM start occuring and Jetty
> restarts.  Would this just be a case of more memory or are there certain
> configuration settings that need to be set?  We're using an out of the box
> Solr 1.3 beta version.
>
> A few of the things we considered that might help:
> - Removing sorts on the result sets (result sets are approx 40,000 +
> documents)
> - Reducing cache sizes such as the queryResultMaxDocsCached setting,
> document cache, queryResultCache, filterCache, etc
>
> Am I missing anything else that should be looked at, or is it time to
> simply increase the memory/start looking at distributing the indexes?  Any
> help would be much appreciated.
>
>
> Regards,
>
> WW
>

Facet Query (fq) and Query (q)

[Jae Joo]

I am having some trouble to utilize the facet Query. As I know that the
facet Query has better performance that simple query (q).
Here is the example.

http://localhost:8080/test_solr/select?q=*:*&facet=true&fq=state:CA&facet.mincount=1&facet.field=city&facet.field=sector&facet.limit=-1&sort=score+desc

--> facet by sector and city for state of CA.
Any idea how to optimize this query to avoid "q=*:*"?

Thanks,

Jae

Facet Query and Query

[Jae Joo]

>
> I am having some trouble to utilize the facet Query. As I know that the
> facet Query has better performance that simple query (q).
> Here is the example.
>
>
> http://localhost:8080/test_solr/select?q=*:*&facet=true&fq=state:CA&facet.mincount=1&facet.field=city&facet.field=sector&facet.limit=-1&sort=score+desc
>
> --> facet by sector and city for state of CA.
> Any idea how to optimize this query to avoid "q=*:*"?
>
> Thanks,
>
> Jae
>
>
>

DataImport Hadnler - new bee question

[Jae Joo]

Hey,

I am trying to connect the Oracle database and index the values into solr,
but I ma getting the
"Document [null] missing required field: id".

Here is the debug output.
1
2
0
2008-12-02 13:49:35
−

Indexing completed. Added/Updated: 0 documents. Deleted 0 documents.


schema.xml

   

 
 id


data-config.xml












Database Schema
id  is the pk.
There are only 2 rows in the table solr_test.

Will anyone help me what I am wrong?

Jae

Re: DataImport Hadnler - new bee question

[Jae Joo]

I actually found the problem. Oracle returns the field name as "Capital".

On Tue, Dec 2, 2008 at 1:57 PM, Jae Joo <[EMAIL PROTECTED]> wrote:

> Hey,
>
> I am trying to connect the Oracle database and index the values into solr,
> but I ma getting the
> "Document [null] missing required field: id".
>
> Here is the debug output.
> 1
> 2
> 0
> 2008-12-02 13:49:35
> −
> 
> Indexing completed. Added/Updated: 0 documents. Deleted 0 documents.
> 
>
> schema.xml
>  />
> omitNorms="true"/>
>
>  
>  id
>
>
> data-config.xml
>
> 
>  url="jdbc:oracle:thin:@x.x.x.x:" user="..."  password="..."/>
> 
> 
> 
> 
>
> 
> 
> 
>
> Database Schema
> id  is the pk.
> There are only 2 rows in the table solr_test.
>
> Will anyone help me what I am wrong?
>
> Jae
>
>

DataImportHandler - time stamp format in

[Jae Joo]

In the dataimport.properties file, there is the timespamp.

#Thu Dec 04 15:36:22 EST 2008
last_index_time=2008-12-04 15\:36\:20

I am using the Oracle (10g) and would like to know which format of timestamp
I have to use in Oracle.

Thanks,

Jae

Re: Solr on Solaris

[Jae Joo]

I do have same experience.
What is the CPU in the Solaris box? it is not depending on the operating
system (linux or Solaris). It is depenong on the CPU (Intel ro SPARC).
Don't know why, but based on my performance test, SPARC machine requires
MORE memory for java application.

Jae

On Thu, Dec 4, 2008 at 10:40 PM, Kashyap, Raghu <[EMAIL PROTECTED]>wrote:

> We are running solr on a solaris box with 4 CPU's(8 cores) and  3GB Ram.
> When we try to index sometimes the HTTP Connection just hangs and the
> client which is posting documents to solr doesn't get any response back.
> We since then have added timeouts to our http requests from the clients.
>
>
>
> I then get this error.
>
>
>
> java.lang.OutOfMemoryError: requested 239848 bytes for Chunk::new. Out
> of swap space?
>
> java.lang.OutOfMemoryError: unable to create new native thread
>
> Exception in thread "JmxRmiRegistryConnectionPoller"
> java.lang.OutOfMemoryError: unable to create new native thread
>
>
>
> We are running JDK 1.6_10 on the solaris box. . The weird thing is we
> are running the same application on linux box with JDK 1.6 and we
> haven't seen any problem like this.
>
>
>
> Any suggestions?
>
>
>
> -Raghu
>
>

spellCheckComponent and dismax query type

[Jae Joo]

I would like to use spell check with dismax, but it is not working. This
query searchs only default search field which is defined in schema.xml.

http://localhost:8080/ibegin_mb3/spellCheckCompRH?q=pluming%20heaing&qt=dismax&spellcheck.q=pluming%20heaing&spellcheck.count=10&spellcheck=true&spellcheck.collate=true

Can any one help me?

Thanks,

Jae Joo

prefetching question

[Jae Joo]

Hi,

We do have 16 millions of company name and would like to find the way for
"prefetching" by using Solr.

Does anyone have experience and/or suggestions?

Thanks,

Jae Joo

multiple indexes

[Jae Joo]

Hi,

I would like to know how it can be implemented.

Index1 has fields id,1,2,3 and index2 has fields id,5,6,7.
The ID in both indexes are unique id.

Can I use "a kind of " distributed search and/or multicore to search, sort,
and facet through 2 indexes (index1 and index2)?

Thanks,

Jae joo

Joining Solr Indexes

[Jae Joo]

Hi,

Is there any way to join multiple indexes in Solr?

Thanks,

Jae

Solr 1.2 and 1.3 - different Stamming

[Jae Joo]

I have found that the stamming in solr 1.2 and 1.3 is different for
"communication". We have index built in Solr 1.2 and the index is being
queried by 1.3. Is there any way to adjust it?

Jae joo

Sharding and Index Update

[Jae Joo]

All,

I have two indices - one has 23 M document and the other has less than 1000.
The small index is for real time update.

Does updating small index (with commit) hurt the overall performance?
(We can not update realtime for 23M big index because of heavy traffic and
size).

Thanks,

Jae Joo

Java Heap Space

[Jae Joo]

I am new in Solr and try to use Jitty and example with 13 million records.
During running it, I have the error -
*HTTP ERROR: 500*

Java heap space



java.lang.OutOfMemoryError: Java heap space


Any recommendation? We have a million transactions, so would it be better to
use Tomcat?

Thanks,

Jae

example solr configurartion file

[Jae Joo]

does anyone know what exactly is?
10
1
2147483647
1
1000
1

If we change the value, do I have to reindex it?

Thanks,

Jae

facet error - example

[Jae Joo]

Hi,

I have updated the example and have an error if I empty one of the facet
field.

Any know what I missed and/or wrong,


Here is exception list

org.apache.solr.core.SolrException: undefined field
at org.apache.solr.schema.IndexSchema.getField(IndexSchema.java:829)
at org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java
:140)
at org.apache.solr.request.SimpleFacets.getFacetFieldCounts(
SimpleFacets.java:170)
at org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java
:87)
at org.apache.solr.request.StandardRequestHandler.getFacetInfo(
StandardRequestHandler.java:172)
at org.apache.solr.request.StandardRequestHandler.handleRequestBody(
StandardRequestHandler.java:120)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(
RequestHandlerBase.java:77)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:658)
at org.apache.solr.servlet.SolrDispatchFilter.execute(
SolrDispatchFilter.java:191)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
SolrDispatchFilter.java:159)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
ApplicationFilterChain.java:215)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(
ApplicationFilterChain.java:188)
at org.apache.catalina.core.StandardWrapperValve.invoke(
StandardWrapperValve.java:210)
at org.apache.catalina.core.StandardContextValve.invoke(
StandardContextValve.java:174)
at org.apache.catalina.core.StandardHostValve.invoke(
StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(
ErrorReportValve.java:117)
at org.apache.catalina.core.StandardEngineValve.invoke(
StandardEngineValve.java:108)
at org.apache.catalina.connector.CoyoteAdapter.service(
CoyoteAdapter.java:151)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java
:870)
at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection
(Http11BaseProtocol.java:665)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(
PoolTcpEndpoint.java:528)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(
LeaderFollowerWorkerThread.java:81)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(
ThreadPool.java:685)
at java.lang.Thread.run(Thread.java:595)

Solr Sort

[Jae Joo]

Will anyone helps me how to sort the "text string" which length is less than
50.

Thanks,

Jae Joo

Optimize index

[Jae Joo]

Does anyone know how to optimize the index and what the difference between
compound format and stand format?

Thanks,

Jae Joo

RE: Optimize index

[Jae Joo]

So, is compound index faster at query time?
Jae

-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Yonik
Seeley
Sent: Wednesday, August 08, 2007 4:32 PM
To: solr-user@lucene.apache.org
Subject: Re: Optimize index

On 8/8/07, Jae Joo <[EMAIL PROTECTED]> wrote:
> Does anyone know how to optimize the index and what the difference
between
> compound format and stand format?

Compound index format squishes almost all the files of a segment into
a single file.  It's slower at index time.

-Yonik

RE: Optimize index

[Jae Joo]

How about standformat optimizion?
Jae

-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Yonik
Seeley
Sent: Wednesday, August 08, 2007 5:07 PM
To: solr-user@lucene.apache.org
Subject: Re: Optimize index

On 8/8/07, Jae Joo <[EMAIL PROTECTED]> wrote:
> So, is compound index faster at query time?

Slower (but very slightly).  A little less concurrency under heavy load.

-Yonik

Re: Optimize index

[Jae Joo]

No, I am merging any index at all. Just want to know how much faster than
non optimized index.
And how to do that.
Also, want to know exactly what the compound format optimization and
standard format one.
Here is the part of solrconfig.xml file which mentions Optimization.


 From: Mike Klaas <[EMAIL PROTECTED]>
> Sent: Wed, August 8, 2007 5:16 pm
> To: solr-user@lucene.apache.org
> Subject: Re: Optimize index
>
> On 8-Aug-07, at 2:09 PM, Jae Joo wrote:
>
> > How about standformat optimizion?
> > Jae
>
> Optimized indexes are always faster at query time that their non-
> optimized counterparts.  Sometimes significantly so.
>
> -Mike
>

RE: Optimize index

[Jae Joo]

The subject is the "Company name". If you are going to our web site
www.manta.com/mb we do need the "FAST" search for company name.

Thanks,

Jae

-Original Message-
From: Stu Hood [mailto:[EMAIL PROTECTED] 
Sent: Wednesday, August 08, 2007 6:08 PM
To: solr-user@lucene.apache.org
Subject: Re: Optimize index



While we're on the subject of optimizing: Are there any benefits to
optimizing an index before merging it into another index?

Thanks,
Stu


-Original Message-
From: Mike Klaas <[EMAIL PROTECTED]>
Sent: Wed, August 8, 2007 5:16 pm
To: solr-user@lucene.apache.org
Subject: Re: Optimize index

On 8-Aug-07, at 2:09 PM, Jae Joo wrote:

> How about standformat optimizion?
> Jae

Optimized indexes are always faster at query time that their non- 
optimized counterparts.  Sometimes significantly so.

-Mike

tomcat and solr multiple instances

[Jae Joo]

Hi,

 

I have built 2 solr instance - one is "example" and the other is
"ca_companies".

 

The "ca_companies" solr instance is working find, but "example is not
working...

 

In the admin page, "/solr/admin", for "example" instance, it shows that

 

Cwd=/rpt/src/apache-solr-1.2.0/ca_companies/solr/conf  

--> this should be 

Cwd=/rpt/src/apache-solr-1.2.0/example/solr/conf  

 

SolrHome=/rpt/src/apache-solr-1.2.0/example/solr/

 

Any one knows why?

 

If I run Jetty for instance "example", it is working well...

 

Thanks,

 

Jae Joo

RE: tomcat and solr multiple instances

[Jae Joo]

Here are the Catalina/localhost/ files
For "example" instance






For ca_companies instance







Urls
http://host:8080/solr/admin --> pointint "example" instance (Problem...)
http://host:8080/solr_ca/admin --> pointing "ca-companies" instance (it
is working)

-----Original Message-
From: Jae Joo [mailto:[EMAIL PROTECTED] 
Sent: Thursday, August 09, 2007 5:45 PM
To: solr-user@lucene.apache.org
Subject: tomcat and solr multiple instances

Hi,

 

I have built 2 solr instance - one is "example" and the other is
"ca_companies".

 

The "ca_companies" solr instance is working find, but "example is not
working...

 

In the admin page, "/solr/admin", for "example" instance, it shows that

 

Cwd=/rpt/src/apache-solr-1.2.0/ca_companies/solr/conf  

--> this should be 

Cwd=/rpt/src/apache-solr-1.2.0/example/solr/conf  

 

SolrHome=/rpt/src/apache-solr-1.2.0/example/solr/

 

Any one knows why?

 

If I run Jetty for instance "example", it is working well...

 

Thanks,

 

Jae Joo

Solr and XML format

[Jae Joo]

I would like to find out how to build the index for general format of XML
file which does not have  tags.

Do I have to update all my xml file to Solr specific XML which have
… format? Or any way to build index?



Thanks,



Jae Joo

facet case sensitive

[Jae Joo]

Hi,

The XML file to be indexed has Case Sensitive
Ex.
Computer Software

I would like to have "facet by field name field1"  CASE SENSITIVE and search
by field1 with CASE INSENSITIVE.
If I add solr.LowerCaseFilterFactory in the analyzer in both index and
query, the query is working correctly - case insensitive,
but in the facets, I got "computer software" - All lower cases which is NOT
what I want. I need

can anyone help me?

Thanks,

Jae

breadcrumb in Solr

[Jae Joo]

Hi,

I am looking for the way to have "breadcrumb"
Is there any way to get thoes kind of information from Solr search result..


Thanks,

Jae Joo

sub facets

[Jae Joo]

Hi,

Can anyone help me how to do sub faces?
Thanks,

Jae Joo

Multiple indexes

[Jae Joo]

Is there any solution to handle 13 millions document shown as below?
Each document is not big, but the number of ones is 13 million.
Any way to utilize the multiple indexes?

Thanks,
Jae Joo



Unlimi-Tech Software Inc
ON

Single Location

Canada
K1C 4R1
Ottawa
Ontario
G2_Computer Software
1447a Youville Dr
G_Technology & Communications
203439018

Carleton

30
G2173_Computer Programming Services
G217308D_Computer software development

Re: Multiple indexes

[Jae Joo]

The potential problem is performance. I have tested with 1.3 Millions and it
returns fast enough.

Jae

On 8/23/07, Walter Underwood <[EMAIL PROTECTED]> wrote:
>
> It should work fine to index them and search them. 13 million docs is
> not even close to the limits for Lucene and Solr. Have you had problems?
>
> wunder
>
> On 8/23/07 7:30 AM, "Jae Joo" <[EMAIL PROTECTED]> wrote:
>
> > Is there any solution to handle 13 millions document shown as below?
> > Each document is not big, but the number of ones is 13 million.
> > Any way to utilize the multiple indexes?
> >
> > Thanks,
> > Jae Joo
> >
> >
> > 
> > Unlimi-Tech Software Inc
> > ON
> > 
> > Single Location
> > 
> > Canada
> > K1C 4R1
> > Ottawa
> > Ontario
> > G2_Computer Software
> > 1447a Youville Dr
> > G_Technology & Communications
> > 203439018
> > 
> > Carleton
> > 
> > 30
> > G2173_Computer Programming Services
> > G217308D_Computer software development
> > 
>
>

Solr and JBOSS Integration

[Jae Joo]

Hi,

I have Solr 1.2 and JBoss Application Server 4.2.1 installed in Windows
machine and would like to find how to integrate it together.

can anyone help me?

Thanks,

Jae Joo

Re: Solr and JBOSS Integration

[Jae Joo]

sort of, but need more details.
How about solr.war file? do I have to copy it into jboss deploy directory?
Otherwise, how I let jboss kow where Solr is?
Any idea how to handle *.jar files for Solr?

Thanks,

Jae

On 8/26/07, Grant Ingersoll <[EMAIL PROTECTED]> wrote:
>
> Is this what you are looking for: http://wiki.apache.org/solr/SolrJBoss
>
> On Aug 25, 2007, at 10:18 AM, Jae Joo wrote:
>
> > Hi,
> >
> > I have Solr 1.2 and JBoss Application Server 4.2.1 installed in
> > Windows
> > machine and would like to find how to integrate it together.
> >
> > can anyone help me?
> >
> > Thanks,
> >
> > Jae Joo
>
> --
> Grant Ingersoll
> http://lucene.grantingersoll.com
>
> Lucene Helpful Hints:
> http://wiki.apache.org/lucene-java/BasicsOfPerformance
> http://wiki.apache.org/lucene-java/LuceneFAQ
>
>
>

range index

[Jae Joo]

Is there any way to catagorize by price range?

I would like to do facet by price range. (ex. 100-200, 201-500, 501-1000,
...)

Thanks,

Jae Joo

Re: range index

[Jae Joo]

That works. But I am looking how to do that at INDEXING TIME, but at query
time.

Any way for that?

Thanks,

Jae

On 8/27/07, Erik Hatcher <[EMAIL PROTECTED]> wrote:
>
>
> On Aug 27, 2007, at 9:32 AM, Jae Joo wrote:
> > Is there any way to catagorize by price range?
> >
> > I would like to do facet by price range. (ex. 100-200, 201-500,
> > 501-1000,
> > ...)
>
> Yes, look at using facet queries using range queries.  There is an
> example of this very thing here:
>
> <http://wiki.apache.org/solr/
> SimpleFacetParameters#head-1da3ab3995bc4abcdce8e0f04be7355ba19e9b2c>
>
>Erik
>
>

Re: range index

[Jae Joo]

I could build index with Sales Vol ranges using PatternReplaceFilterFactory



   

 

Thanks,

Jae
On 8/27/07, Erik Hatcher <[EMAIL PROTECTED]> wrote:
>
>
> On Aug 27, 2007, at 9:48 AM, Jae Joo wrote:
> > That works. But I am looking how to do that at INDEXING TIME, but
> > at query
> > time.
> >
> > Any way for that?
>
> I'm not sure I understand the question.   The example provided works
> at query time.  If you want to bucket things at indexing time you
> could do that, but no real reason to with Solr's caching making the
> range buckets fast at query time.
>
> Could you elaborate on what you are trying to do?
>
> Erik
>
>
>
> >
> > Thanks,
> >
> > Jae
> >
> > On 8/27/07, Erik Hatcher <[EMAIL PROTECTED]> wrote:
> >>
> >>
> >> On Aug 27, 2007, at 9:32 AM, Jae Joo wrote:
> >>> Is there any way to catagorize by price range?
> >>>
> >>> I would like to do facet by price range. (ex. 100-200, 201-500,
> >>> 501-1000,
> >>> ...)
> >>
> >> Yes, look at using facet queries using range queries.  There is an
> >> example of this very thing here:
> >>
> >> <http://wiki.apache.org/solr/
> >> SimpleFacetParameters#head-1da3ab3995bc4abcdce8e0f04be7355ba19e9b2c>
> >>
> >>Erik
> >>
> >>
>
>

Re: range index

[Jae Joo]

Any sample code and howto write Analyzer and Tockenizer available?

Jae

On 8/27/07, Jonathan Woods <[EMAIL PROTECTED]> wrote:
>
> Or you could write your own Analyzer and Tokenizer to produce single
> values
> corresponding, say, to the start of each range.
>
> Jon
>
> > -Original Message-
> > From: Jae Joo [mailto:[EMAIL PROTECTED]
> > Sent: 27 August 2007 16:46
> > To: solr-user@lucene.apache.org
> > Subject: Re: range index
> >
> > I could build index with Sales Vol ranges using
> > PatternReplaceFilterFactory
> >
> >
> >  > pattern="(^000[1-4].*)" replacement="10M - 50M"
> > replace="all"
> > />
> > > pattern="(^000[5-9].*)" replacement="50M - 100M"
> > replace="all"
> > />
> >  > pattern="(^00[1-9].*)" replacement="100M -
> > 1B" replace="all"
> >         />
> >   > pattern="(^0[1-9].*)" replacement="\>1B" replace="all"
> > />
> >
> > Thanks,
> >
> > Jae
> > On 8/27/07, Erik Hatcher <[EMAIL PROTECTED]> wrote:
> > >
> > >
> > > On Aug 27, 2007, at 9:48 AM, Jae Joo wrote:
> > > > That works. But I am looking how to do that at INDEXING
> > TIME, but at
> > > > query time.
> > > >
> > > > Any way for that?
> > >
> > > I'm not sure I understand the question.   The example provided works
> > > at query time.  If you want to bucket things at indexing time you
> > > could do that, but no real reason to with Solr's caching making the
> > > range buckets fast at query time.
> > >
> > > Could you elaborate on what you are trying to do?
> > >
> > > Erik
> > >
> > >
> > >
> > > >
> > > > Thanks,
> > > >
> > > > Jae
> > > >
> > > > On 8/27/07, Erik Hatcher <[EMAIL PROTECTED]> wrote:
> > > >>
> > > >>
> > > >> On Aug 27, 2007, at 9:32 AM, Jae Joo wrote:
> > > >>> Is there any way to catagorize by price range?
> > > >>>
> > > >>> I would like to do facet by price range. (ex. 100-200, 201-500,
> > > >>> 501-1000,
> > > >>> ...)
> > > >>
> > > >> Yes, look at using facet queries using range queries.
> > There is an
> > > >> example of this very thing here:
> > > >>
> > > >> <http://wiki.apache.org/solr/
> > > >>
> > SimpleFacetParameters#head-1da3ab3995bc4abcdce8e0f04be7355ba19e9b2c
> > > >> >
> > > >>
> > > >>Erik
> > > >>
> > > >>
> > >
> > >
> >
>
>

Heap size error during indexing

[Jae Joo]

Hi,

I have a Java Heap size problem during indexing for 13 millions doc. under
linux using post.sh (optimized).
each document size is about 2k.

Is there any way to set java heap size in post.sh under tomcat?

Thanks,

Jae Joo

Re: Trouble with Windows / Tomcat install

[Jae Joo]

did you build solr.xml in $CATALINA_HOME/conf//Catalina/localhost ?
it yes, please double check the directory information.
And did you copy the apache-solr-1.2.0.war to solr.war in dist directory?

Jae

On 9/1/07, Robin Bonin <[EMAIL PROTECTED]> wrote:
>
> Hi all, I followed the instructions in the wiki here,
> http://wiki.apache.org/solr/SolrTomcat
> I know Tomcat is running, but when I pull up my solr admin page, I get
> the following error.
>
>
> description The server encountered an internal error () that prevented
> it from fulfilling this request.
>
> exception org.apache.jasper.JasperException
> org.apache.jasper.servlet.JspServletWrapper.handleJspException(
> JspServletWrapper.java:476)
> org.apache.jasper.servlet.JspServletWrapper.service(
> JspServletWrapper.java:371)
> org.apache.jasper.servlet.JspServlet.serviceJspFile(
> JspServlet.java:315)
> org.apache.jasper.servlet.JspServlet.service(JspServlet.java:265)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>
> root cause javax.servlet.ServletException
> org.apache.jasper.runtime.PageContextImpl.doHandlePageException(
> PageContextImpl.java:846)
> org.apache.jasper.runtime.PageContextImpl.handlePageException(
> PageContextImpl.java:779)
> org.apache.jsp.admin.index_jsp._jspService(index_jsp.java:313)
> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:98)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> org.apache.jasper.servlet.JspServletWrapper.service(
> JspServletWrapper.java:328)
> org.apache.jasper.servlet.JspServlet.serviceJspFile(
> JspServlet.java:315)
> org.apache.jasper.servlet.JspServlet.service(JspServlet.java:265)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>
> root cause java.lang.NoClassDefFoundError
> org.apache.jsp.admin.index_jsp._jspService(index_jsp.java:80)
> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:98)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> org.apache.jasper.servlet.JspServletWrapper.service(
> JspServletWrapper.java:328)
> org.apache.jasper.servlet.JspServlet.serviceJspFile(
> JspServlet.java:315)
> org.apache.jasper.servlet.JspServlet.service(JspServlet.java:265)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>
> Apache Tomcat/5.5.23
>

Re: Trouble with Windows / Tomcat install

[Jae Joo]

Solr and tomcat connection does not require any copies and moves of jar
file. All of the jar files are in solr.war file.

Can you send your solr.xml file?
If you use "\" instead of "/", you have to your "\\" to point the solr
instance in solr.xml conf.

Jae

On 9/1/07, Robin Bonin <[EMAIL PROTECTED]> wrote:
>
> I tried both solr-1.1 and 1.2, I was having more trouble with 1.2, so
> i went back to 1.1.
> I did copy the war from dist, and renamed to just solr, but I have no
> xml file for solr in conf.
> I was using the java 'option' -Dsolr.solr.home=C:\Solr\
>
> I just removed the solr war and folder from web apps and moved to
> tomcat\shared\lib and created a solr.xml file under locahost with the
> correct path to the war,and solr folder, and I get the same error.
>
> I tried changing the paths in the XML to the wrong ones to watch how
> the message changed and I found the problem (mid email)...
>
> the step 'Copy the contents of the example directory
> c:\temp\solrZip\example\solr\ to c:\web\solr\'
>
> I had copied everything from the example directory, not example\solr.
> so the path was a directory off
>
> Thanks for your help.
>
>
> On 9/1/07, Jae Joo <[EMAIL PROTECTED]> wrote:
> > did you build solr.xml in $CATALINA_HOME/conf//Catalina/localhost ?
> > it yes, please double check the directory information.
> > And did you copy the apache-solr-1.2.0.war to solr.war in dist
> directory?
> >
> > Jae
> >
> > On 9/1/07, Robin Bonin <[EMAIL PROTECTED]> wrote:
> > >
> > > Hi all, I followed the instructions in the wiki here,
> > > http://wiki.apache.org/solr/SolrTomcat
> > > I know Tomcat is running, but when I pull up my solr admin page, I get
> > > the following error.
> > >
> > >
> > > description The server encountered an internal error () that prevented
> > > it from fulfilling this request.
> > >
> > > exception org.apache.jasper.JasperException
> > > org.apache.jasper.servlet.JspServletWrapper.handleJspException
> (
> > > JspServletWrapper.java:476)
> > > org.apache.jasper.servlet.JspServletWrapper.service(
> > > JspServletWrapper.java:371)
> > > org.apache.jasper.servlet.JspServlet.serviceJspFile(
> > > JspServlet.java:315)
> > > org.apache.jasper.servlet.JspServlet.service(JspServlet.java
> :265)
> > > javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> > >
> > > root cause javax.servlet.ServletException
> > >
> org.apache.jasper.runtime.PageContextImpl.doHandlePageException(
> > > PageContextImpl.java:846)
> > > org.apache.jasper.runtime.PageContextImpl.handlePageException(
> > > PageContextImpl.java:779)
> > > org.apache.jsp.admin.index_jsp._jspService(index_jsp.java:313)
> > > org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java
> :98)
> > > javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> > > org.apache.jasper.servlet.JspServletWrapper.service(
> > > JspServletWrapper.java:328)
> > > org.apache.jasper.servlet.JspServlet.serviceJspFile(
> > > JspServlet.java:315)
> > > org.apache.jasper.servlet.JspServlet.service(JspServlet.java
> :265)
> > > javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> > >
> > > root cause java.lang.NoClassDefFoundError
> > > org.apache.jsp.admin.index_jsp._jspService(index_jsp.java:80)
> > > org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java
> :98)
> > > javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> > > org.apache.jasper.servlet.JspServletWrapper.service(
> > > JspServletWrapper.java:328)
> > > org.apache.jasper.servlet.JspServlet.serviceJspFile(
> > > JspServlet.java:315)
> > > org.apache.jasper.servlet.JspServlet.service(JspServlet.java
> :265)
> > > javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> > >
> > > Apache Tomcat/5.5.23
> > >
> >
>

caching query result

[Jae Joo]

HI,

I am wondering that is there any way for CACHING FACETS SEARCH Result?

I have 13 millions and have facets by states (50). If there is a mechasim to
chche, I may get faster result back.

Thanks,

Jae

Re: caching query result

[Jae Joo]

Here is the response XML faceted by multiple fields including state.

−

0
1782
−

-1

10
0
score desc
true
1
−

duns_number,company_name,phys_state, phys_city, score

phys_country:"United States"

2.2


−

sales_range
total_emp_range
company_type
phys_state
sic1

on



On 9/6/07, Yonik Seeley <[EMAIL PROTECTED]> wrote:
>
> On 9/6/07, Jae Joo <[EMAIL PROTECTED]> wrote:
> > I have 13 millions and have facets by states (50). If there is a
> mechasim to
> > chche, I may get faster result back.
>
> How fast are you getting results back with standard field faceting
> (facet.field=state)?
>

RAMDirectory

[Jae Joo]

HI,

Does any know how to use RAM disk for index?

Thanks,

Jae Joo

LockObtainFailedException

[Jae Joo]

will anyone help me why and how?


org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out:
SimpleFSLock@/usr/local/se
archengine/apache-solr-1.2.0/fr_companies/solr/data/index/write.lock
at org.apache.lucene.store.Lock.obtain(Lock.java:70)
at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:579)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java
:341)
at org.apache.solr.update.SolrIndexWriter.<init>(
SolrIndexWriter.java:65)
at org.apache.solr.update.UpdateHandler.createMainIndexWriter(
UpdateHandler.java:120)
at org.apache.solr.update.DirectUpdateHandler2.openWriter(
DirectUpdateHandler2.java:181)
at org.apache.solr.update.DirectUpdateHandler2.addDoc(
DirectUpdateHandler2.java:259)
at org.apache.solr.handler.XmlUpdateRequestHandler.update(
XmlUpdateRequestHandler.java:166)
at org.apache.solr.handler.XmlUpdateRequestHandler.handleRequestBody
(XmlUpdateRequestHandler
.java:84)

Thanks,

Jae Joo

moving index

[Jae Joo]

Hi,

I do need to move the index files, but have a concerns any potential problem
including performance?
Do I have to keep the original document for querying?

Thanks,

Jae Joo

Re: LockObtainFailedException

[Jae Joo]

In solrconfig.xml,
false
10
25000
1400
500
1000
1

Does writeLockTimeout too small?

Thanks,

Jae
On 9/27/07, matt davies <[EMAIL PROTECTED]> wrote:
>
> quick fix
>
> look for a lucene lock file in your tmp directory and delete it, then
> restart solr, should start
>
> I am an idiot though, so be careful, in fact, I'm worse than an
> idiot, I know a little
>
> :-)
>
> you got a lock file somewhere though, deleting that will help you
> out, for me it was in my /tmp directory
>
> On 27 Sep 2007, at 14:10, Jae Joo wrote:
>
> > will anyone help me why and how?
> >
> >
> > org.apache.lucene.store.LockObtainFailedException: Lock obtain
> > timed out:
> > SimpleFSLock@/usr/local/se
> > archengine/apache-solr-1.2.0/fr_companies/solr/data/index/write.lock
> > at org.apache.lucene.store.Lock.obtain(Lock.java:70)
> > at org.apache.lucene.index.IndexWriter.init
> > (IndexWriter.java:579)
> > at org.apache.lucene.index.IndexWriter.<init>
> > (IndexWriter.java
> > :341)
> > at org.apache.solr.update.SolrIndexWriter.<init>(
> > SolrIndexWriter.java:65)
> > at org.apache.solr.update.UpdateHandler.createMainIndexWriter(
> > UpdateHandler.java:120)
> > at org.apache.solr.update.DirectUpdateHandler2.openWriter(
> > DirectUpdateHandler2.java:181)
> > at org.apache.solr.update.DirectUpdateHandler2.addDoc(
> > DirectUpdateHandler2.java:259)
> > at org.apache.solr.handler.XmlUpdateRequestHandler.update(
> > XmlUpdateRequestHandler.java:166)
> > at
> > org.apache.solr.handler.XmlUpdateRequestHandler.handleRequestBody
> > (XmlUpdateRequestHandler
> > .java:84)
> >
> > Thanks,
> >
> > Jae Joo
>
>

Indexing without application server

[Jae Joo]

Hi,

I have a multi millions document to be indexed and looking for the way to
index it without j2ee application server.
It is not incremental indexing, this is a kind of "Index once, use forever"
- all batch mode.

I can guess if there is a way to index it without J2EE, it may be much
faster...

Thanks,

Jae Joo

Solr - Lucene Query

[Jae Joo]

In the schema.xml, this fiend is defined by 



Is there any way to find the document by querying - The Appraisal Station?


Thanks,
Jae

Merging Fields

[Jae Joo]

Is there any way to merge fields  during indexing time.

I have field1 and field2 and would like to combine these fields and make
field3.
In the document, there are field1 and field2, and I may build field3 using
CopyField.

Thanks,

Jae

Re: Syntax for newSearcher query

[Jae Joo]

Do I have to define the  name/values as exactly same as acturl query
(order...)?

Here is actual query

indent=on&version=2.2&facet=true&facet.mincount=1
&facet.field=phys_state&facet.field=sic1&facet.limit=-1
&sort=sales_volume_us+desc&q=%28phys_country%3A%22United+States%22%29
&start=0&rows=20&fl=duns_number%2Ccompany_name%2Cphys_address%2C+
phys_state%2C+phys_city%2C+phys_zip%2C+ticker_symbol%2C+status_id_descr%2Cscore&
qt=&wt=&explainOther=&hl.fl=


In the newSearch event, I defined as
   
  

true
1
phys_state
sic1
sales_volume_us desc
phys_country:"United States"
0
20
duns_number, company_name, phys_address, phys_state,
phys_city, phys_
zip, ticker_symbol, status_id_descr, score

  


But, I am not sure this is working or not (may be not!).

Is there anything else I missed in configuration?

Thanks,

Jae




On 10/10/07, BrendanD <[EMAIL PROTECTED]> wrote:
>
>
> Awesome! Thanks!
>
>
> hossman wrote:
> >
> >
> > : looking queries that I'm not quite sure how to specify in my
> > solrconfig.xml
> > : file in the newSearcher section.
> >
> > :
> >
> rows=20&start=0&facet.query=attribute_id:1003278&facet.query=attribute_id:1003928&sort=merchant_count+desc&facet=true&facet.field=min_price_cad_rounded_to_tens&facet.field=manufacturer_id&facet.field=merchant_id&facet.field=has_coupon&facet.field=has_bundle&facet.field=has_sale_price&facet.field=has_promo&fq=product_is_active:true&fq=product_status_code:complete&fq=category_id:"1001143"&qt=sti_dismax_en&f.min_price_cad_rounded_to_tens.facet.limit=-1
> >
> > all you have to do is put each key=val pair as a  name="key">val
> >
> > it doesn't matter what the param is, or if it's a param that has
> multiple
> > values, just list each of them the same way...
> >
> > 
> >   
> >  
> >   20
> >   0
> >   attribute_id:1003278
> >   attribute_id:1003928
> > ...
> > 
> >  
> >   ...
> >
> >
> > -Hoss
> >
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Syntax-for-newSearcher-query-tf4604487.html#a13148914
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>

Delete index and "commit or optimize"

[Jae Joo]

Hi,

I have 9g index and try to delete a couple of document. The actual deletion
is working fine.

Here is my question.
Do I have to OPTIMIZE the index after deleting? or just COMMIT it? The
original index already optimized.

Thanks,

Jae Joo

Solr Index update - specific field only

[Jae Joo]

Hi,

I have index which has the field  NOT stored and would like update some
field which is indexed and stored.
Updating index requires all fields same as original (before updating) with
updated field.
Is there any way to post "JUST  UPDATED FIELD ONLY"?
Here is an example.
field  indexed  stored
-
item_id  yes yes
searchable yes yes
price yes yes
title  yes yes
description yes no

The way I know to update  the "Searchable" field from Y to N for item_it
"12345".


12345
Y
6699
title sample
This is the detail description of item




and I am looking the way to update the specific field by



12345
Y


  --> it may keep the unchanged field.

Thanks,

Jae Joo

Remote access - Solr index for deleting

[Jae Joo]

Hi,

I am trying to delete the document remotly through curl command, but got the
internal server error - Permission Denied.
Anyone knows how to solve this problem?

Thanks,

Jae

Solr and Lucene Indexing Performance

[Jae Joo]

Hi,

I have 6 millions article to be indexed by Solr and do need your
recommendation.

I do need to parse and generate the Solr based xml file to post it. How
about to use Lucene directly?
I have short testing, it looks like Sola based indexing is faster than
direct indexing through Lucene.

Am I did something wrong and/or does Solr use multiple threading or
something else to get the good indexing performance?

Thanks

Jae Joo

Multiple indexes

[Jae Joo]

Hi,

I am looking for the way to utilize the multiple indexes for signle sole
instance.
I saw that there is the patch 215  available  and would like to ask someone
who knows how to use multiple indexes.

Thanks,

Jae Joo

Query and heap Size

[Jae Joo]

In my system, the heap size (old generation) keeps growing up caused by
heavy traffic.
I have adjusted the size of young generation, but it does not work well.

Does anyone have any recommendation regarding this issue? - Solr
configuration and/or web.xml ...etc...

Thanks,

Jae

Re: Multiple indexes

[Jae Joo]

Here is my situation.

I have 6 millions articles indexed and adding about 10k articles everyday.
If I maintain only one index, whenever the daily feeding is running, it
consumes the heap area and causes FGC.
I am thinking the way to have multiple indexes - one is for ongoing querying
service and one is for update. Once update is done, switch the index by
automatically and/or my application.

Thanks,

Jae joo


On Nov 12, 2007 8:48 AM, Ryan McKinley <[EMAIL PROTECTED]> wrote:

> The advantages of a multi-core setup are configuration flexibility and
> dynamically changing available options (without a full restart).
>
> For high-performance production solr servers, I don't think there is
> much reason for it.  You may want to split the two indexes on to two
> machines.  You may want to run each index in a separate JVM (so if one
> crashes, the other does not)
>
> Maintaining 2 indexes is pretty easy, if that was a larger number or you
> need to create indexes for each user in a system then it would be worth
> investigating the multi-core setup (it is still in development)
>
> ryan
>
>
> Pierre-Yves LANDRON wrote:
> > Hello,
> >
> > Until now, i've used two instance of solr, one for each of my
> collections ; it works fine, but i wonder
> > if there is an advantage to use multiple indexes in one instance over
> several instances with one index each ?
> > Note that the two indexes have different schema.xml.
> >
> > Thanks.
> > PL
> >
> >> Date: Thu, 8 Nov 2007 18:05:43 -0500
> >> From: [EMAIL PROTECTED]
> >> To: solr-user@lucene.apache.org
> >> Subject: Multiple indexes
> >>
> >> Hi,
> >>
> >> I am looking for the way to utilize the multiple indexes for signle
> sole
> >> instance.
> >> I saw that there is the patch 215  available  and would like to ask
> someone
> >> who knows how to use multiple indexes.
> >>
> >> Thanks,
> >>
> >> Jae Joo
> >
> > _
> > Discover the new Windows Vista
> > http://search.msn.com/results.aspx?q=windows+vista&mkt=en-US&form=QBRE
>
>

Re: Multiple indexes

[Jae Joo]

I have built the master solr instance and indexed some files. Once I run
snapshotter, i complains the error..  - snapshooter -d data/index (in
solr/bin directory)
Did I missed something?

++ date '+%Y/%m/%d %H:%M:%S'
+ echo 2007/11/12 12:38:40 taking snapshot
/solr/master/solr/data/index/snapshot.20071112123840
+ [[ -n '' ]]
+ mv 
/solr/master/solr/data/index/temp-snapshot.20071112123840/solr/master/solr/data/index/snapshot.20071112123840
mv: cannot access /solr/master/solr/data/index/temp-snapshot.20071112123840
Jae

On Nov 12, 2007 9:09 AM, Ryan McKinley <[EMAIL PROTECTED]> wrote:

>
> just use the standard collection distribution stuff.  That is what it is
> made for! http://wiki.apache.org/solr/CollectionDistribution
>
> Alternatively, open up two indexes using the same config/dir -- do your
> indexing on one and the searching on the other.  when indexing is done
> (or finishes a big chunk) send  to the 'searching' one and it
> will see the new stuff.
>
> ryan
>
>
>
> Jae Joo wrote:
> > Here is my situation.
> >
> > I have 6 millions articles indexed and adding about 10k articles
> everyday.
> > If I maintain only one index, whenever the daily feeding is running, it
> > consumes the heap area and causes FGC.
> > I am thinking the way to have multiple indexes - one is for ongoing
> querying
> > service and one is for update. Once update is done, switch the index by
> > automatically and/or my application.
> >
> > Thanks,
> >
> > Jae joo
> >
> >
> > On Nov 12, 2007 8:48 AM, Ryan McKinley <[EMAIL PROTECTED]> wrote:
> >
> >> The advantages of a multi-core setup are configuration flexibility and
> >> dynamically changing available options (without a full restart).
> >>
> >> For high-performance production solr servers, I don't think there is
> >> much reason for it.  You may want to split the two indexes on to two
> >> machines.  You may want to run each index in a separate JVM (so if one
> >> crashes, the other does not)
> >>
> >> Maintaining 2 indexes is pretty easy, if that was a larger number or
> you
> >> need to create indexes for each user in a system then it would be worth
> >> investigating the multi-core setup (it is still in development)
> >>
> >> ryan
> >>
> >>
> >> Pierre-Yves LANDRON wrote:
> >>> Hello,
> >>>
> >>> Until now, i've used two instance of solr, one for each of my
> >> collections ; it works fine, but i wonder
> >>> if there is an advantage to use multiple indexes in one instance over
> >> several instances with one index each ?
> >>> Note that the two indexes have different schema.xml.
> >>>
> >>> Thanks.
> >>> PL
> >>>
> >>>> Date: Thu, 8 Nov 2007 18:05:43 -0500
> >>>> From: [EMAIL PROTECTED]
> >>>> To: solr-user@lucene.apache.org
> >>>> Subject: Multiple indexes
> >>>>
> >>>> Hi,
> >>>>
> >>>> I am looking for the way to utilize the multiple indexes for signle
> >> sole
> >>>> instance.
> >>>> I saw that there is the patch 215  available  and would like to ask
> >> someone
> >>>> who knows how to use multiple indexes.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Jae Joo
> >>> _
> >>> Discover the new Windows Vista
> >>> http://search.msn.com/results.aspx?q=windows+vista&mkt=en-US&form=QBRE
> >>
> >
>
>

Solr/bin script - Solaris bash version?

[Jae Joo]

Hi,

Is there Solaris bash based script available? The couple of command is not
working, and wondering any available scripts I can use before I update it.

For ex. snapshooter, snappuller, snapinstaller



Thanks,

Jae

snapshot files

[Jae Joo]

Hi,

I have successfully built generated the snalshot files but have a question.
Does each snapshot file has all files in the index directory?

Here is the file list in the index
_0.fdt_0.fnm_0.nrm_0.tii_1.fdt
_1.fnm_1.nrm_1.tiisegments.gen
_0.fdx_0.frq_0.prx_0.tis_1.fdx
_1.frq_1.prx_1.tissegments_3

And here is the file list of 2 snapshot files.

 snapshot.20071113094936
_0.fdt_0.fdx_0.fnm_0.frq_0.nrm
_0.prx_0.tii_0.tissegments.gen  segments_2


 snapshot.20071113095508
_0.fdt_0.fnm_0.nrm_0.tii_1.fdt
_1.fnm_1.nrm_1.tiisegments.gen
_0.fdx_0.frq_0.prx_0.tis_1.fdx
_1.frq_1.prx_1.tissegments_3

The later one have all files same as index directory.

I have changed the snapshooter script because the bash in solaris do not
have cp -l option.

#cp -lr ${data_dir}/index ${temp} --> original
mkdir ${temp}
ln  ${data_dir}/index/* ${temp}



Thanks,

Jae Joo

two solr instances - index and commit

[Jae Joo]

Hi,

I have two solr instance running under different tomcat environment.
One solr instance is for indexing and would like to commit to the other solr
instance.

This is what I tried, but failed.
using post.sh (without commit), the docs are  indexed in solr-1 instance.
After indexed,
call commit command with the attribute of solr-2.

Can any help me?

Jae

Re: two solr instances - index and commit

[Jae Joo]

Solr-1 - for indexing instance - localhost:8880/solr_1
solr-2 - for search - localhost:8080/solr_2 (different port number than
solr-1)
Here is the list of steps what I tried.
1. in the Solr-1, index and commit it. I could see all indexes in the
data/index directory.

How can I commit the index  on solr-2 which is already committed in solr1?

Thanks

Jae





On Nov 13, 2007 6:48 PM, Kasi Sankaralingam <[EMAIL PROTECTED]> wrote:

> This works, the only thing you need to be aware of is the NFS problem if
> you are
> running in a distributed environment sharing a NFS partition.
>
> a) Index and commit on instance (Typically partitioned as an index server)
>
> b) Issue a commit on the search server (like a read only mode)
>
> Things to watch out for, you will get stale NFS problem, I replaced lucene
> core
> that is shipped with solr to the latest one and it works.
>
> -Original Message-
> From: Jae Joo [mailto:[EMAIL PROTECTED]
> Sent: Tuesday, November 13, 2007 9:06 AM
> To: solr-user
> Subject: two solr instances - index and commit
>
> Hi,
>
> I have two solr instance running under different tomcat environment.
> One solr instance is for indexing and would like to commit to the other
> solr
> instance.
>
> This is what I tried, but failed.
> using post.sh (without commit), the docs are  indexed in solr-1 instance.
> After indexed,
> call commit command with the attribute of solr-2.
>
> Can any help me?
>
> Jae
>

snappuller rsync parameter error? - "solr" hardcoded

[Jae Joo]

In the snappuller, the "solr" is hardcoded. Should it be
"${master_data_dir}?

# rsync over files that have changed
rsync -Wa${verbose}${compress} --delete ${sizeonly} \
${stats} rsync://${master_host}:${rsyncd_port}/solr/${name}/
${data_dir}/${name}-wip

Thanks,

Jae

score customization

[Jae Joo]

Hi,

I am looking for the way to get the score - only hundredth - ex.
4.09something like that.
Currently, it has 7 decimal digits. 1.8032384

Thanks,

Jae

facet - associated fields

[Jae Joo]

Hi,
Can anyone help me how to facet and/or search for associated fields? -


 
  1234
  Baseball hall of Fame opens Jackie Robinson
exhibit
  Description about the new JR hall of fame
exhibit.
  20071114
  200711
  0
  press

  
  Sports
  Baseball
  Major League Baseball

  
  Arts and Culture
  Culture
  Heritage Sites
 


Thanks,

Jae

Local Disk and SAN

[Jae Joo]

Hi,

I have about 20G bytes of index with 1 Million transactions per day.
I am considering the disk system between local disk and SAN based system
(not NFS).
Is there any performance difference to run solr instance with 20 G index on
local disk and on SAN based disk which is connected with fiber channel?

Thanks,

Jae

Max. number of Error messages

[Jae Joo]

Is there any parameter to set the max. number of error messages..
The Solr system was killed after a couple of error messages which caused by
WRONG QUERY

Thanks,

Jae

Re: Issues with postOptimize

[Jae Joo]

try it.

 /search/replication_test/0/index/solr/bin/snapshooter
 .
 true


Jae

On Dec 19, 2007 9:10 AM, Bill Au <[EMAIL PROTECTED]> wrote:

> Just changing the permission on the script is not enough.  The id
> executing
> the script needs to have write permission to create the snapshot.
>
> Bill
>
> On Dec 18, 2007 6:26 PM, Sunny Bassan <[EMAIL PROTECTED]> wrote:
>
> > I've set the permissions on the script to execute for all users. And it
> > does seem like the user who is running SOLR has the permissions to run
> > the script. I've come to the conclusion - Linux permissions are
> > annoying, lol. I've also tried setting selinux to permissive mode and
> > added the user to the sudoers file, but this has not fixed the issue.
> > The only thing that does work is croning the script to run after the
> > optimize script.
> >
> > Sunny
> >
>

Duplicated Keyword

[Jae Joo]

Hi,

Is there any way to dedup the keyword cross the document?

Ex.

"china" keyword is in doc1 and doc2. Will Solr index have only 1 "china"
keyword for both document?

Thanks,

Jae Joo

Re: Duplicated Keyword

[Jae Joo]

title of Document 1 - "This is document 1 regarding china" - fieldtype =
text
title of Document 2 - "This is document 2 regarding china"  fieldtype=text

Once it is indexed, will index hold  2 "china"  text fields  or just 1 china
word which is pointing document1 and document2?

Jae

On Jan 4, 2008 10:54 AM, Robert Young <[EMAIL PROTECTED]> wrote:

> I don't quite understand what you're getting at. What is the problem
> you're encountering or what are you trying to achieve?
>
> Cheers
> Rob
>
> On Jan 4, 2008 3:26 PM, Jae Joo <[EMAIL PROTECTED]> wrote:
> > Hi,
> >
> > Is there any way to dedup the keyword cross the document?
> >
> > Ex.
> >
> > "china" keyword is in doc1 and doc2. Will Solr index have only 1 "china"
> > keyword for both document?
> >
> > Thanks,
> >
> > Jae Joo
> >
>

Tomcat and Solr - out of memory

[Jae Joo]

Hi,

What happens if Solr application hit the max. memory of heap assigned?

Will be die or just slow down?

Jae

Query - multiple

[Jae Joo]

If the number of results > 2500 then sort by company_name
otherwise, sort by revenue;

Do I have to access 2 times? One is to get the number of results and the
other one is for sort.
The second query should be accessed by necessary.

Any efficient way?

Thanks,

Jae

Solr Multicore

[Jae Joo]

I have set multicores - core0 and core1, core0 is default.


  
  


Once I update the index by http://localhost:8983/solr/update, it updates
core1 not core0.

Also, I tried to set the deault core using SETASDEFAULT, but it is "unknown
action command".

Can any one help me?

Thanks,

Jae

Multicore request

[Jae Joo]

I have built two cores - core0 and core1.
each core has different set of index.

I can access core0 and core 1 by
http://localhost:8983/solr/core[01]/admin/form.jsp.

Is there any way to access multiple indexes with single query?

Thanks,

Jae

Tomcat and JBOss

[Jae Joo]

I have a problem - memory and performance issues for more than 10 request
(solr Search and Facet) per second.
On tomcat, it requires 4 to 5 G Bytes, but still not enough.
Does anyone have any experience regarding high volume and performance issue
on Tomcat and JBOss and resolutions share with me?

Thanks,

Jae

Solr Warm up on Tomcat

[Jae Joo]

Hi,

Does anyone have experience or solution how to warm up the solr instance on
the tomcat automatically?

I am using Apache 2 for load balancer and 3 Tomcat machines running Solr.
If one of tomcat is needed to shutdown and startup again, the solr should be
warm up before serving the request.

Thanks,

Jae joo

auto Warming and Special Character

[Jae Joo]

In the firstsearch listner, I need to use special character "&" in the q
string, but it complains "Error - filterStart"



  

company_desc:"Advertising & Marketing"
0
20
company_name, score

   


Thanks,

Jae Joo

RE: Multiple Search in Solr

[Jae Joo]

I have downloaded version 1.3 and built multiple indices.

I could not find any way for multiple indices search at Solr level, I
have written the Lucene application. It is working well.

Jae Joo

-Original Message-
From: Niveen Nagy [mailto:[EMAIL PROTECTED] 
Sent: Monday, February 04, 2008 8:55 AM
To: solr-user@lucene.apache.org
Subject: Multiple Search in Solr

Hello ,

 

I have a question concerning solr multiple indices. We have 4 solr
indices in our system and we want to use distributed search (Multiple
search) that searches in the four indices in parallel. We downloaded the
latest code from svn and we applied the patch distributed.patch but we
need more detailed description on how to use this patch and what changes
should be applied to solr schema, and how these indices should be
located. Another question here is could the steps be applied to our
indices that was built using a version before applying the distributed
patch.

 

 Thanks in advance.

   

Best Regards,

 

Niveen Nagy

 

RE: Commit preformance problem

[Jae Joo]

Or, if you have multiple files to be updated, please make sure "Index
Multiple Files" and commit "Once" at the end of Indexing..

Jae

-Original Message-
From: Jae Joo [mailto:[EMAIL PROTECTED] 
Sent: Tuesday, February 12, 2008 10:50 AM
To: solr-user@lucene.apache.org
Subject: RE: Commit preformance problem

I have same experience.. I do have 6.5G Index and update it daily.
Have you ever check that the updated file does not have any document and
tried "commit"? I don't know why, but it takes so long - more than 10
minutes.

Jae Joo

-Original Message-
From: Ken Krugler [mailto:[EMAIL PROTECTED] 
Sent: Tuesday, February 12, 2008 10:34 AM
To: solr-user@lucene.apache.org
Subject: Re: Commit preformance problem

>I have a large solr index that is currently about 6 GB and is suffering
of
>severe performance problems during updates. A commit can take over 10
>minutes to complete. I have tried to increase max memory to the JVM to
over
>6 GB, but without any improvement. I have also tried to turn off
>waitSearcher and waitFlush, which do significantly improve the commit
speed.
>However, the max number of searchers is then quickly reached.

If you have a large index, then I'd recommend having a separate Solr 
installation that you use to update/commit changes, after which you 
use snappuller or equivalent to swap it in to the live (search) 
system.

>Would a switch to another container (currently using Jetty) make any
>difference?

Very unlikely.

>Does anyone have any other tip for improving the performance?

Switch to Lucene 2.3, and tune the new parameters that control memory 
usage during updating.

-- Ken
-- 
Ken Krugler
Krugle, Inc.
+1 530-210-6378
"If you can't find it, you can't fix it"

RE: Commit preformance problem

[Jae Joo]

I have same experience.. I do have 6.5G Index and update it daily.
Have you ever check that the updated file does not have any document and
tried "commit"? I don't know why, but it takes so long - more than 10
minutes.

Jae Joo

-Original Message-
From: Ken Krugler [mailto:[EMAIL PROTECTED] 
Sent: Tuesday, February 12, 2008 10:34 AM
To: solr-user@lucene.apache.org
Subject: Re: Commit preformance problem

>I have a large solr index that is currently about 6 GB and is suffering
of
>severe performance problems during updates. A commit can take over 10
>minutes to complete. I have tried to increase max memory to the JVM to
over
>6 GB, but without any improvement. I have also tried to turn off
>waitSearcher and waitFlush, which do significantly improve the commit
speed.
>However, the max number of searchers is then quickly reached.

If you have a large index, then I'd recommend having a separate Solr 
installation that you use to update/commit changes, after which you 
use snappuller or equivalent to swap it in to the live (search) 
system.

>Would a switch to another container (currently using Jetty) make any
>difference?

Very unlikely.

>Does anyone have any other tip for improving the performance?

Switch to Lucene 2.3, and tune the new parameters that control memory 
usage during updating.

-- Ken
-- 
Ken Krugler
Krugle, Inc.
+1 530-210-6378
"If you can't find it, you can't fix it"