date:20110816

OOM due to JRE Issue (LUCENE-1566)

2011-08-16 Thread Pranav Prakash

Hi,

This might probably have been discussed long time back, but I got this error
recently in one of my production slaves.

SEVERE: java.lang.OutOfMemoryError: OutOfMemoryError likely caused by the
Sun VM Bug described in https://issues.apache.org/jira/browse/LUCENE-1566;
try calling FSDirectory.setReadChunkSize with a a value smaller than the
current chunk size (2147483647)

I am currently using Solr1.4. Going through JIRA Issue comments, I found
that this patch applies to 2.9 or above. We are also planning an upgrade to
Solr 3.3. Is this patch included in 3.3 so as to I don't have to manually
apply the patch?

What are the other workarounds of the problem?

Thanks in adv.

*Pranav Prakash*

"temet nosce"

Twitter  | Blog  |
Google

Filtering results based on a set of values for a field

2011-08-16 Thread tomas.zerolo

Hello, Solrs

we are trying to filter out documents written by (one or more of) the authors 
from
a mediumish list (~2K). The document set itself is in the millions.

Apart from the obvious approach of building a huge OR-list and appending it
to the query, it seems that writing a Lucene[1] filter (or a SolrFilter[2]) 
seems
to suggest itself. In fact [3] seems to strongly encourage this approach.

Basically, as we understand it, the filter's method getDocIdSet gets called and 
is
fed with index segments, "one spoonful at a time". It then decides which docs
of the segment will be accepted, setting the corresponding bits in the result 
(in
our case, e.g. look up the document's author's name in a HashMap or something
like it).

Our first question is: how does it all fit together? Would be enough to write 
such a
class? How do I reference that in the SOLR configuration? In the query? A Lucene
Filter or a SolrFilter?

The problem is, we are experiencing very slow response times, in the order of
12 seconds for a query (the OR alternative, which we tested on a smallish author
list of aboug a couple of hundred is nearly-instantaneous).

Our second question is: are we on track with this? Intuition would say, of 
course,
that sifting sequentially through the index, checking each document for its 
author
*will* take its time. So may be the approach is doomed? Are there other, better
approaches?

Thanks for any pointers

--

[1] 

[2] 
[1] 

-- tomás

Re: how to add automatic keyword filters in solr?

2011-08-16 Thread nagarjuna

Thank u very much for ur reply iorixxx
 i already know about the field type "suggest" and i am able to get
the related keywords in json response format my problem is i
developed one jsp page and integrated to solr if i type "test" in the jsp
page i will get the response whichever having the test keyword...everything
is fine ...but,when ever i type "t" i need to get the full keywords like
"test","Test","TEST" etc...in the jsp page search field ..i am unable to do
this

so,please help me to get this in jsp page searchfield..

--
View this message in context: 
http://lucene.472066.n3.nabble.com/how-to-add-automatic-keyword-filters-in-solr-tp3257708p3258058.html
Sent from the Solr - User mailing list archive at Nabble.com.

how to use solr search keywords url in jsp page?

2011-08-16 Thread nagarjuna

hi everybody
   i developed one response format which gives the following response when
ever i enter "t" in my solr search field...

   {
  "responseHeader":{
"status":0,
"QTime":0,
"params":{
  "fl":"keywords",
  "indent":"on",
  "start":"0",
  "q":"t\r\n",
  "version":"2.2",
  "rows":"30"}},
  "response":{"numFound":19,"start":0,"docs":[
  {
"keywords":"test"},
  {
"keywords":"test"},
  {
"keywords":"test"},
  {
"keywords":"Test"},
  {
"keywords":"test"},
  {
"keywords":"test"},
  {
"keywords":"test"},
  {
"keywords":"test"},
  {
"keywords":"test"},
  {
"keywords":"Test"},
  {
"keywords":"Test"},
  {
"keywords":"Test"},
  {
"keywords":"Test"},
  {
"keywords":"Test"},
  {
"keywords":"Test"},
  {
"keywords":"test"},
  {
"keywords":"testing"},
  {
"keywords":"testing"},
  {
"keywords":"test iphone android"}]
  }}
now i dont know how to implement this in my jsp pageactually i have one
jsp page with search field when ever i enter the word "t" in my search field
i need to get all the above results as auto keywords as like in google...if
we enter "a" the related words "ant","axe"etc...like this i need to
display
please help me 

thank u in advance

--
View this message in context: 
http://lucene.472066.n3.nabble.com/how-to-use-solr-search-keywords-url-in-jsp-page-tp3258100p3258100.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: OOM due to JRE Issue (LUCENE-1566)

2011-08-16 Thread Simon Willnauer

hey,

On Tue, Aug 16, 2011 at 9:34 AM, Pranav Prakash  wrote:
> Hi,
>
> This might probably have been discussed long time back, but I got this error
> recently in one of my production slaves.
>
> SEVERE: java.lang.OutOfMemoryError: OutOfMemoryError likely caused by the
> Sun VM Bug described in https://issues.apache.org/jira/browse/LUCENE-1566;
> try calling FSDirectory.setReadChunkSize with a a value smaller than the
> current chunk size (2147483647)
>
> I am currently using Solr1.4. Going through JIRA Issue comments, I found
> that this patch applies to 2.9 or above. We are also planning an upgrade to
> Solr 3.3. Is this patch included in 3.3 so as to I don't have to manually
> apply the patch?
AFAIK, solr 1.4 is on Lucene 2.9.1 so this patch is already applied to
the version you are using.
maybe you can provide the stacktrace and more deatails about your
problem and report back?

simon

>
> What are the other workarounds of the problem?
>
> Thanks in adv.
>
> *Pranav Prakash*
>
> "temet nosce"
>
> Twitter  | Blog  |
> Google 
>

SOlr upgrade: Invalid version (expected 2, but 1) error when using shards

2011-08-16 Thread olivier sallou

Hi,
I just migrated to solr 3.3 from 1.4.1.
My index is still in 1.4.1 format (will be migrated soon).

I have an error when I use sharding with the new version:

org.apache.solr.common.SolrException: java.lang.RuntimeException: Invalid
version (expected 2, but 1) or the data in not in 'javabin' format

However, if I request each shard independently (/request), answer is
correct. So the error is triggered only with the shard mechanism.

While I foresee to upgrade my indexes, I'd like to understand the issue,
e.g. is it an "upgrade" issue or don't shards support using an "old" format.

Thanks

Olivier

Re: OOM due to JRE Issue (LUCENE-1566)

2011-08-16 Thread Pranav Prakash

>
>
> AFAIK, solr 1.4 is on Lucene 2.9.1 so this patch is already applied to
> the version you are using.
> maybe you can provide the stacktrace and more deatails about your
> problem and report back?
>

Unfortunately, I have only this much information with me. However following
is my speficiations, if they are any helpful :-

/usr/bin/java -d64 -Xms5000M -Xmx5000M -XX:+UseParallelGC -verbose:gc
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:$GC_LOGFILE
-XX:+CMSPermGenSweepingEnabled -Dsolr.solr.home=multicore
 -Denable.slave=true -jar start.jar

32GiB RAM


Any thoughts? Will a switch to ConcurrentGC help in any means?

How to deal with "java.net.SocketTimeoutException: Read timed out" on commit?

2011-08-16 Thread Geir Gullestad Pettersen

Hi,

We're getting a lot of these timeouts during bulk feeding or a large
document set. We're sending batches of 1000 documents and commiting every 15
minutes or for every 10.000 docs, whichever happens first. We find that the
first few commits (after 10'/20'/30' docs) go through without exceptions
(although the "CommonsHttpSolrServer.soTimeout" is set quite high -
12ms). The time for a commit to complete seems to increase for every
subsequent commit request, and eventually the timeout value is reached
causing Solrj to throw the java.net.SocketTimeoutException. Increasing the
soTimeout value only delays the problem and it will allow us to send a few
more commits before seeing the exception.

I've noticed that even though we get these exceptions on commit, documents
seem to be searchable after some time. Does that mean that the commit
process in solr continues even though the socket times out?

We're also setting "waitFlush=false" and  "waitSearcher=false" when
committing, which I would expect to enable a non-blocking commit request to
solr, but on n+1 commits this doesn't aleays happen and this when we're
getting the timeouts.

I suspect that solr.commit(false,false) is a non-blocking call only if a all
previous commits are complete. Correct?

I'm aware that commiting often is not adviced as it is a resouce intensive
process for solr, however it is a requirement that documents are searchable
before the entire document set has been added to solr. Our workaround is to
limit the amount of commits by time and document count "triggers" in our
code. We cannot use the autocommit feature in solrconfig.xml because we also
need to be able to issue immediate commits for incremental feeding.

So, my main question is regarding the severity of
"java.net.SocketTimeoutException: Read timed out" on commit. Should I worry
and try to re-issue commit when this happens, or is it safe to ignore it?


Thanks for any input!

Cheers,
Geir

Re: How to deal with "java.net.SocketTimeoutException: Read timed out" on commit?

2011-08-16 Thread Manish Bafna

We too were getting same issue.
We solved it by ensuring that when commit is in progress, no one access the
index.

Though SOLR's UpdateRequest does it, we will still read timeout issues
because of CommonsHttpSolrServer.

If we have another layer which doesnt send the request itself, then you wont
get this error.

On Tue, Aug 16, 2011 at 5:34 PM, Geir Gullestad Pettersen
wrote:

> Hi,
>
> We're getting a lot of these timeouts during bulk feeding or a large
> document set. We're sending batches of 1000 documents and commiting every
> 15
> minutes or for every 10.000 docs, whichever happens first. We find that the
> first few commits (after 10'/20'/30' docs) go through without exceptions
> (although the "CommonsHttpSolrServer.soTimeout" is set quite high -
> 12ms). The time for a commit to complete seems to increase for every
> subsequent commit request, and eventually the timeout value is reached
> causing Solrj to throw the java.net.SocketTimeoutException. Increasing the
> soTimeout value only delays the problem and it will allow us to send a few
> more commits before seeing the exception.
>
> I've noticed that even though we get these exceptions on commit, documents
> seem to be searchable after some time. Does that mean that the commit
> process in solr continues even though the socket times out?
>
> We're also setting "waitFlush=false" and  "waitSearcher=false" when
> committing, which I would expect to enable a non-blocking commit request to
> solr, but on n+1 commits this doesn't aleays happen and this when we're
> getting the timeouts.
>
> I suspect that solr.commit(false,false) is a non-blocking call only if a
> all
> previous commits are complete. Correct?
>
> I'm aware that commiting often is not adviced as it is a resouce intensive
> process for solr, however it is a requirement that documents are searchable
> before the entire document set has been added to solr. Our workaround is to
> limit the amount of commits by time and document count "triggers" in our
> code. We cannot use the autocommit feature in solrconfig.xml because we
> also
> need to be able to issue immediate commits for incremental feeding.
>
> So, my main question is regarding the severity of
> "java.net.SocketTimeoutException: Read timed out" on commit. Should I worry
> and try to re-issue commit when this happens, or is it safe to ignore it?
>
>
> Thanks for any input!
>
> Cheers,
> Geir
>

RE: ideas for indexing large amount of pdf docs

2011-08-16 Thread Rode González

Hi Jay, thanks. great idea, in next days we'll try to do something like
you'd exposed. 

best,
rode.

---
Rode González
Libnova, SL
Paseo de la Castellana, 153-Madrid
[t]91 449 08 94  [f]91 141 21 21
www.libnova.es

> -Mensaje original-
> De: Jaeger, Jay - DOT [mailto:jay.jae...@dot.wi.gov]
> Enviado el: lunes, 15 de agosto de 2011 14:54
> Para: solr-user@lucene.apache.org
> Asunto: RE: ideas for indexing large amount of pdf docs
> 
> Note on i:  Solr replication provides pretty good clustering support
> out-of-the-box, including replication of multiple cores.  Read the Wiki
> on replication (Google +solr +replication if you don't know where it
> is).
> 
> In my experience, the problem with indexing PDFs is it takes a lot of
> CPU on the document parsing side (client), not on the Solr server side.
> So make sure you do that part on the client and not the server.
> 
> Avoiding iii:
> 
> 
> Suggest that you write yourself a multi-threaded performance test so
> that you aren't guessing what your performance will be.
> 
> We wrote one in Perl.  It handles an individual thread (we were testing
> inquiry), and we wrote a little batch file / shell script to start up
> the desired number of threads.
> 
> The main statement in our batch file (the rest just set the variables).
> A  Shell script would be even easier.
> 
> for /L %%i in (1,1,%THREADS%) DO start /B perl solrtest.pl -h
> %SOLRHOST%
> -c %COUNT% -u %1 -p %2 -r %SOLRREALM% -f %SOLRLOC%\firstsynonyms.txt -l
> %SOLRLOC%\lastsynonyms.txt -z %FUZZ%
> 
> The perl
> 
> 
> #!/usr/bin/perl
> 
> #
> # Perl program to run a thread of solr testing
> #
> 
> use Getopt::Std;  # For options processing
> use POSIX;# For time formatting
> use XML::Simple;  # For processing of XML config file
> use Data::Dumper; # For debugging XML config file
> use HTTP::Request::Common;# For HTTP request to Solr
> use HTTP::Response;
> use LWP::UserAgent;   # For HTTP request to Solr
> 
> $host = "YOURHOST:8983";
> $realm = "YOUR AUTHENTICATION REALM";
> $firstlist = "firstsynonyms.txt";
> $lastlist = "lastsynonyms.txt";
> $fuzzy = "";
> 
> $me = $0;
> 
> sub usage() {
>   print "perl $me -c iterations [-d] [-h host:port ] [-u user [-p
> password]] \n";
>   print "\t\t[-f firstnamefile] [-l lastnamefile] [-z fuzzy] [-r
> realm]\n";
>   exit(8);
> }
> 
> 
> #
> # Process the command line options, and open the output file.
> #
> 
> getopts('dc:u:p:f:l:h:r:z:') || usage();
> 
> if(!$opt_c) {
>   usage();
> }
> 
> $count = $opt_c;
> 
> if($opt_u) {
>   $user = $opt_u;
> }
> 
> if($opt_p) {
>   $password = $opt_p;
> }
> 
> if($opt_h) {
>   $host = $opt_h;
> }
> 
> if($opt_f) {
>   $firstlist = $opt_f;
> }
> 
> if($opt_l) {
>   $lastlist = $opt_l;
> }
> 
> if($opt_r) {
>   $realm = $opt_r;
> }
> 
> if($opt_z) {
>   $fuzzy = "~" . $opt_z;
> }
> 
> $debug = $opt_d;
> 
> 
> #
> # If the host string does not include a :, add :80
> #
> 
> if($host !~ /:/) {
>   $host = $host . ":80";
> }
> 
> #
> # Read the lists of first and last names
> #
> 
> open(SYNFILE,"<$firstlist") || die "Can't open first name list
> $firstlist\n";
> while() {
>   @newwords = split /,/;
>   for($i=0; $i <= $#newwords; ++$i) {
>   $newwords[$i] =~ s/^\s+//;
>   $newwords[$i] =~ s/\s+$//;
>   $newwords[$i] = lc($newwords[$i]);
>   }
>   push @firstnames, @newwords;
> }
> close(SYNFILE);
> 
> open(SYNFILE,"<$lastlist") || die "Can't open last name list
> $lastlist\n";
> while() {
>   @newwords = split /,/;
>   for($i=0; $i <= $#newwords; ++$i) {
>   $newwords[$i] =~ s/^\s+//;
>   $newwords[$i] =~ s/\s+$//;
>   $newwords[$i] = lc($newwords[$i]);
>   }
>   push @lastnames, @newwords;
> }
> close(SYNFILE);
> 
> 
> print "$#firstnames First Names, $#lastnames Last Names\n";
> print "User: $user\n";
> 
> 
> my $userAgent = LWP::UserAgent->new(agent => 'solrtest.pl');
> $userAgent->credentials("$host",$realm,$user,$password);
> 
> $uri = "http://$host/solr/select";;
> 
> $starttime = time();
> 
> for($c=0; $c < $count; ++$c) {
>   $fname = $firstnames[rand $#firstnames];
>   $lname = $lastnames[rand $#lastnames];
>   $response = $userAgent->request(
>   POST $uri,
>   [
>   q => "lnamesyn:$lname AND fnamesyn:$fname$fuzzy",
>   rows => "25"
>   ]);
> 
>   if($debug) {
>   print "Query: lnamesyn:$lname AND fnamesyn:$fname$fuzzy";
>   print $response->content();
>   }
>   print "POST for $fname $lname completed, HTTP status=" .
> $response->code . "\n";
> }
> 
> $elapsed = time() - $starttime;
> $average = $elapsed / $count;
> 
> print "Time: $elapsed s ($average/request)\n";
> 
> 
> -Original Message-
> From: Rode Go

Re: Some questions about SolrJ

2011-08-16 Thread Erick Erickson

About updating the Wiki, just create your login and have at it. Anything
people think is wrong, they can edit 

Best
Erick

On Sun, Aug 14, 2011 at 3:39 PM, Shawn Heisey  wrote:
> On 8/13/2011 9:59 AM, Michael Sokolov wrote:
>>
>>> Shawn, my experience with SolrJ in that configuration (no autoCommit) is
>>> that you have control over commits: if you don't issue an explicit commit,
>>> it won't happen.  Re lifecycle: we don't use a static instance; rather our
>>> app maintains a small pool of CommonsHttpSolrServer instances that we re-use
>>> across requests.  I think that will be preferable since I don't think the
>>> underlying HttpClient is thread safe?
>>
>> Hmm, I just checked and actually CommonsHttpSolrServer uses
>> MultiThreadedHttpConnectionManager so it should be thread-safe, and OK to
>> use a static instance as per documentation.  Sorry for the misinformation.
>
> Thanks for the help!
>
> I've been able to muddle my way through part of my implementation on my own.
>  There doesn't seem to be any way to point to the base /solr/ url and then
> ask SolrJ to add a core when creating requests.  I did see that you can set
> the URL for the server object after it's created, but if I ever make this
> thing multithreaded, I fear doing so will cause problems.  I'm going with
> one server object (solrServer) for CoreAdmin and another object (solrCore)
> for requests against the core.
>
> This new build system has an object representing one complete index, which
> uses a container of seven objects representing each of the shards.  Each of
> the shard objects has two objects representing a build core and a live core.
>  Each of the core objects contains the solrServer and solrCore already
> mentioned.  Since I have two complete indexes, this means that the final
> product will initialize 56 server objects.
>
> I couldn't use static server objects as recommended by the docs, because I
> have so many instances that all need different URLs.  They are private class
> members that get created only once, so I think it will be OK.  A static
> object would be a good idea for a search application, because it likely only
> needs to deal with one URL.  Our webapp developers told me that they will be
> putting the server object into a bean in the application context.
>
> When I've got everything done and debugged, I will use what I've learned to
> augment the SolrJ wiki page.  Who is the best community person to coordinate
> with on that to make sure I put up good information?
>
> Thanks,
> Shawn
>
>

RE: Product data schema question

2011-08-16 Thread Jaeger, Jay - DOT

On the surface, you could simply add some more fields to your schema.  But as 
far as I can tell, you would have to have a separate Solr "document" for each 
SKU/size combination,  and store the rest of the information (brand, model, 
color, SKU) redundantly and make the unique key a combination of the SKU and 
the size (presumably by having an additional field called sku_size - as far as 
I can tell Solr can't make a key up out of multiple fields).

But, perhaps you shouldn't store that dynamic inventory information in Solr.   
Instead store a key that gets you to an inventory database organized by SKU and 
size.  In such a schema, Size could be added as a multi-valued field to your 
existing schema, so you would know what sizes existed, so you could at least 
facet on that, maybe.  That way Solr doesn't have to be updated every time the 
inventory changes.  Of course, that won't help with the faceting  on variant 
inventory.
 
-Original Message-
From: Steve Cerny [mailto:sjce...@gmail.com] 
Sent: Monday, August 15, 2011 6:29 PM
To: solr-user@lucene.apache.org
Subject: Product data schema question

I'm working on an online eCommerce project and am having difficulties
building the core / index schema.  Here is the way we organize our product
information in a normalized database.

A product model has many SKUs (called colorways)
A SKU has many sizes (called variants)
A SKU size has associated inventory (called variant inventory)

When we setup our product core we have the following field information

Doc
* brand
* model name
* SKU
* color name

Sample records are as follows

* Haynes, Undershirt, 1234, white
* Haynes, Undershirt, 1235, grey
* Fruit of the Loom, Undershirt, 1236, white
* Fruit of the Loom, Underwear, 1237, grey

The issue I'm having is I want to add inventory to each size of each SKU for
faceting.  Example,

SKU 1234 has sizes small, medium, large.  Size small has 5 in stock, size
medium 10, and size large 25.

In a normalized data table I would have a separate table just for inventory
and related it back to the SKU with a foreign key.  How do I store size and
inventory information effectively with Solr?

-- 
Steve

Re: Can Master push data to slave

2011-08-16 Thread Erick Erickson

Why do you care about the lock file on the slave? It shouldn't matter,
so I'm wondering if this is an XY problem:

>From Hossman's Apache page:
Your question appears to be an "XY Problem" ... that is: you are dealing
with "X", you are assuming "Y" will help you, and you are asking about "Y"
without giving more details about the "X" so that we can understand the
full issue.  Perhaps the best solution doesn't involve "Y" at all?
See Also: http://www.perlmonks.org/index.pl?node_id=542341

Best
Erick

On Mon, Aug 15, 2011 at 1:59 AM, Pawan Darira  wrote:
> Regarding point b, i mean that when Slave server does a replication from
> Master, it creates a lock-file in it's index directory. How to avoid that?
>
>
> On Tue, Aug 9, 2011 at 2:56 AM, Markus Jelsma 
> wrote:
>
>> Hi,
>>
>> > Hi
>> >
>> > I am using Solr 1.4. and doing a replication process where my slave is
>> > pulling data from Master. I have 2 questions
>> >
>> > a. Can Master push data to slave
>>
>> Not in current versions. Not sure about exotic patches for this.
>>
>> > b. How to make sure that lock file is not created while replication
>>
>> What do you mean?
>>
>> >
>> > Please help
>> >
>> > thanks
>> > Pawan
>>
>

Re: Minimum score filter

2011-08-16 Thread Erick Erickson

Right, so you're using edismax? This is expected. You can do
a number of things:

1> change the parameters of edismax
2> have your app filter out returns that dive beneath some threshold
 that is relative to the score of the first doc in the list.

But I don't see why, given your example, it makes a difference. Because
you got the document you were looking for in this instance says nothing
abou the next time when you search on something different

Best
Erick

On Mon, Aug 15, 2011 at 2:22 PM, Donald J. Organ IV
 wrote:
> OK I am doing a search using the following fields     name^2.0 code^1.8 
> cat_search^1.5 description^0.8
>
> I am searching for:   free range dog nips
>
> I am getting back 2 documents the first is the document I am looking for, and 
> contains those works in the name field, as the name field is "Free Range Dog 
> Nip Chicken Breast Wraps"
>
>
> The second looks like its matching because those words are contained within 
> the description.
>
>
>
> - Original Message -
> From: "simon" 
> To: solr-user@lucene.apache.org
> Sent: Monday, August 15, 2011 1:59:17 PM
> Subject: Re: Minimum score filter
>
> The absolute value of a relevance score doesn't have a lot of meaning and
> the range of scores can vary a lot depending on any boost you may apply.
> Even if you normalize them (say on a 1-100 scale where 100 is the max
> relevance) you can't really draw any valid conclusions from those values.
>
> It would help if you described exactly what problem you're trying to solve.
>
> -Simon
>
> On Mon, Aug 15, 2011 at 1:02 PM, Donald J. Organ IV
>

Re: Indexing from a database via SolrJ

2011-08-16 Thread Erick Erickson

What have you tried and what doesn't it do that you want it to do?

This works, instantiating the StreamingUpdateSolrServer (server) and
the JDBC connection/SQL statement are left as exercises for the
reader .:

while (rs.next()) {
  SolrInputDocument doc = new SolrInputDocument();

  String id = rs.getString("id");
  String title = rs.getString("title");
  String text = rs.getString("text");

  doc.addField("id", id);
  doc.addField("title", title);
  doc.addField("text", text);

  docs.add(doc);
  ++counter;
  ++total;
  if (counter > 100) { // Completely arbitrary, just batch up more
than one document for throughput!
server.add(docs);
docs.clear();
counter = 0;
  }
}

Best
Erick

On Mon, Aug 15, 2011 at 6:25 PM, Shawn Heisey  wrote:
> Is there a simple way to get all the fields from a jdbc resultset into a
> bunch of SolrJ documents, which I will then send to be indexed in Solr?  I
> would like to avoid the looping required to copy the data one field at a
> time.  Copying it one document at a time would be acceptable, but it would
> be nice if there was a way to copy them all at once.
>
> Another idea that occurred to me is to add the dataimporter jar to my
> project and leverage it to do the heavy lifting, but I will need some
> pointers about what objects and methods to research.  Is that a reasonable
> idea, or is it too integrated into the server code to be used with SolrJ?
>
> Can anyone point me in the right direction?
>
> Thanks,
> Shawn
>
>

Re: Score

2011-08-16 Thread Yonik Seeley

On Mon, Aug 15, 2011 at 8:13 PM, Bill Bell  wrote:
> How do I change the score to scale it between 0 and 100 irregardless of the 
> score?
>
> q.alt=*:*&bq=lang:Spanish&defType=dismax

Doing this for a single query is easy:  when you retrieve scores, the
maxScore is also reported. So just do
score/maxScore*100

-Yonik
http://www.lucidimagination.com

Re: OOM due to JRE Issue (LUCENE-1566)

2011-08-16 Thread Bill Bell

Send gc log and force dump if you can when it happens.

Bill Bell
Sent from mobile


On Aug 16, 2011, at 5:27 AM, Pranav Prakash  wrote:

>> 
>> 
>> AFAIK, solr 1.4 is on Lucene 2.9.1 so this patch is already applied to
>> the version you are using.
>> maybe you can provide the stacktrace and more deatails about your
>> problem and report back?
>> 
> 
> Unfortunately, I have only this much information with me. However following
> is my speficiations, if they are any helpful :-
> 
> /usr/bin/java -d64 -Xms5000M -Xmx5000M -XX:+UseParallelGC -verbose:gc
> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:$GC_LOGFILE
> -XX:+CMSPermGenSweepingEnabled -Dsolr.solr.home=multicore
> -Denable.slave=true -jar start.jar
> 
> 32GiB RAM
> 
> 
> Any thoughts? Will a switch to ConcurrentGC help in any means?

Re: Migration from Autonomy IDOL to SOLR

2011-08-16 Thread karsten-solr

Hi Arcadius,

currently we have a migration project from verity k2 search server to solr.
I do not know IDOL, but autonomy bought verity before IDOL was released, so 
possible it is comparable?
verity k2 works directly on xml-Files, in result the query syntax is a little 
bit like xpath e.g. with "text1  zone2  zone1" instead of 
contains(//zone1/zone2,'text1').

About verity query syntax:
http://gregconely.getmyip.com/dl/OTG%20Software/5.30.087%20Suite%20%28SP3%29/Disc%204%20-%20Verity/Verity%20K2%20Server%205.5/doc/docs/pdf/VerityQueryLanguage.pdf

Does IDOL work the same way?


Best regards
  Karsten

P.S. in Context:
http://lucene.472066.n3.nabble.com/Migration-from-Autonomy-IDOL-to-SOLR-td3255377.html

 Original-Nachricht 
> Datum: Mon, 15 Aug 2011 11:11:36 +0100
> Von: Arcadius Ahouansou 
> An: solr-user@lucene.apache.org
> Betreff: Migration from Autonomy IDOL to SOLR

> Hello.
> 
> We have a couple of application running on half a dozen Autonomy IDOL
> servers.
> Currently, all feature we need are supported by Solr.
> 
> We have done some internal testing and realized that SOLR would do a
> better
> job.
> 
> So, we are investigation all possibilities for a smooth migration from
> IDOL
> to SOLR.
> 
> I am looking for advice from people who went through something similar.
> 
> Ideally, we would like to keep most of our legacy code unchanged and have
> a
> kind of query-translation-layer plugged into our app if possible.
> 
> -Is there lib available?
> 
> -Any thought?
> 
> Thanks.
> 
> Arcadius.

Exact matching on names?

2011-08-16 Thread Olson, Ron

Hi all-

I'm missing something fundamental yet I've been unable to find the definitive 
answer for exact name matching. I'm indexing names using the standard "text" 
field type and my search is for the name "clarke". My results include "clark", 
which is incorrect, it needs to match clarke exactly (case insensitive).

I tried textType but that doesn't work because I believe it needs to be 
*really* exact, whereas I'm looking for things like "clark oil", "bob, frank, 
and clark", etc.

Thanks for any help,

Ron

DISCLAIMER: This electronic message, including any attachments, files or 
documents, is intended only for the addressee and may contain CONFIDENTIAL, 
PROPRIETARY or LEGALLY PRIVILEGED information.  If you are not the intended 
recipient, you are hereby notified that any use, disclosure, copying or 
distribution of this message or any of the information included in or with it 
is  unauthorized and strictly prohibited.  If you have received this message in 
error, please notify the sender immediately by reply e-mail and permanently 
delete and destroy this message and its attachments, along with any copies 
thereof. This message does not create any contractual obligation on behalf of 
the sender or Law Bulletin Publishing Company.
Thank you.

Re: SOlr upgrade: Invalid version (expected 2, but 1) error when using shards

2011-08-16 Thread Shawn Heisey


On 8/16/2011 4:16 AM, olivier sallou wrote:

I just migrated to solr 3.3 from 1.4.1.
My index is still in 1.4.1 format (will be migrated soon).

I have an error when I use sharding with the new version:

org.apache.solr.common.SolrException: java.lang.RuntimeException: Invalid
version (expected 2, but 1) or the data in not in 'javabin' format

However, if I request each shard independently (/request), answer is
correct. So the error is triggered only with the shard mechanism.


In version 3.1, the javabin format was changed, it's completely 
incompatible with 1.4.1 and earlier at the application level, which 
should have nothing to do with the index format.  Javabin is used in 
distributed search and by SolrJ.


It sounds like you either have an old SolrJ version talking to the new 
Solr version, or you are requesting data from your shards using an old 
version of Solr.  If it's the latter, just upgrade the Solr instance 
that serves as the broker/middleman between your application and your 
shards.


If it's SolrJ, you have two choices.  You can either upgrade your SolrJ 
jar and the jars found in solrj-lib, or keep using the old version and 
add the following line to your code right after your server object is 
created.  This tells SolrJ to use the XML format, which works on any 
version:


server.setParser(new XMLResponseParser());

You'll also need to import 
org.apache.solr.client.solrj.impl.XMLResponseParser for the above line 
to work.


Thanks,
Shawn

Re: Indexing from a database via SolrJ

2011-08-16 Thread Shawn Heisey


On 8/16/2011 7:14 AM, Erick Erickson wrote:

What have you tried and what doesn't it do that you want it to do?

This works, instantiating the StreamingUpdateSolrServer (server) and
the JDBC connection/SQL statement are left as exercises for the
reader.:

 while (rs.next()) {
   SolrInputDocument doc = new SolrInputDocument();

   String id = rs.getString("id");
   String title = rs.getString("title");
   String text = rs.getString("text");

   doc.addField("id", id);
   doc.addField("title", title);
   doc.addField("text", text);

   docs.add(doc);
   ++counter;
   ++total;
   if (counter>  100) { // Completely arbitrary, just batch up more
than one document for throughput!
 server.add(docs);
 docs.clear();
 counter = 0;
   }
 }


I've implemented a basic loop with the structure you've demonstrated, 
but it currently doesn't do anything yet with SolrInputDocument or 
SolrDocumentList.  I figured there would be a way to avoid going through 
the field list one by one, but what you've written suggests that the 
field-by-field method is required.  I can live with that.


It does look like addField just takes an Object, so hopefully I can 
create a loop that determines the type of each field from the JDBC 
metadata, retrieves the correct Java type from the ResultSet, and 
inserts it.  I imagine that everything still works if you happen to 
insert a field that doesn't exist in the index.  This must be how the 
DIH does it, so I was hoping that the DIH might expose a method that 
takes a ResultSet and produces a SolrDocumentList.  I still have to take 
a deeper look at the source and documentation.


Thanks for the help so far, I can get a little more implemented now.

Shawn

Re: Exact matching on names?

2011-08-16 Thread Rob Casson

"exact" can mean a lot of things (do diacritics count?, etc), but in
this case, it sounds like you just need to turn off the stemmer you
have on this fieldtype (or create a new one that doesn't include the
stemmer).

hth,
rob

On Tue, Aug 16, 2011 at 11:20 AM, Olson, Ron  wrote:
> Hi all-
>
> I'm missing something fundamental yet I've been unable to find the definitive 
> answer for exact name matching. I'm indexing names using the standard "text" 
> field type and my search is for the name "clarke". My results include 
> "clark", which is incorrect, it needs to match clarke exactly (case 
> insensitive).
>
> I tried textType but that doesn't work because I believe it needs to be 
> *really* exact, whereas I'm looking for things like "clark oil", "bob, frank, 
> and clark", etc.
>
> Thanks for any help,
>
> Ron
>
> DISCLAIMER: This electronic message, including any attachments, files or 
> documents, is intended only for the addressee and may contain CONFIDENTIAL, 
> PROPRIETARY or LEGALLY PRIVILEGED information.  If you are not the intended 
> recipient, you are hereby notified that any use, disclosure, copying or 
> distribution of this message or any of the information included in or with it 
> is  unauthorized and strictly prohibited.  If you have received this message 
> in error, please notify the sender immediately by reply e-mail and 
> permanently delete and destroy this message and its attachments, along with 
> any copies thereof. This message does not create any contractual obligation 
> on behalf of the sender or Law Bulletin Publishing Company.
> Thank you.
>

Re: Product data schema question

2011-08-16 Thread Steve Cerny

Jay, this is great information.

I don't know enough about Solr whether this is possible...Can we setup two
indexes in the same core, one for product_catalog and the other for
inventory?  Then using a Solr query we could join the indexed content
together.

In Sql it would look like this

select
 p.brand
 , p.model_name
 , p.sku
 , p.color_name
 , i.variant_id
 , i.variant_count
from product_catalog p
join inventory i on (p.sku = i.sku)

On Tue, Aug 16, 2011 at 8:00 AM, Jaeger, Jay - DOT wrote:

> On the surface, you could simply add some more fields to your schema.  But
> as far as I can tell, you would have to have a separate Solr "document" for
> each SKU/size combination,  and store the rest of the information (brand,
> model, color, SKU) redundantly and make the unique key a combination of the
> SKU and the size (presumably by having an additional field called sku_size -
> as far as I can tell Solr can't make a key up out of multiple fields).
>
> But, perhaps you shouldn't store that dynamic inventory information in
> Solr.   Instead store a key that gets you to an inventory database organized
> by SKU and size.  In such a schema, Size could be added as a multi-valued
> field to your existing schema, so you would know what sizes existed, so you
> could at least facet on that, maybe.  That way Solr doesn't have to be
> updated every time the inventory changes.  Of course, that won't help with
> the faceting  on variant inventory.
>
> -Original Message-
> From: Steve Cerny [mailto:sjce...@gmail.com]
> Sent: Monday, August 15, 2011 6:29 PM
> To: solr-user@lucene.apache.org
> Subject: Product data schema question
>
> I'm working on an online eCommerce project and am having difficulties
> building the core / index schema.  Here is the way we organize our product
> information in a normalized database.
>
> A product model has many SKUs (called colorways)
> A SKU has many sizes (called variants)
> A SKU size has associated inventory (called variant inventory)
>
> When we setup our product core we have the following field information
>
> Doc
> * brand
> * model name
> * SKU
> * color name
>
> Sample records are as follows
>
> * Haynes, Undershirt, 1234, white
> * Haynes, Undershirt, 1235, grey
> * Fruit of the Loom, Undershirt, 1236, white
> * Fruit of the Loom, Underwear, 1237, grey
>
> The issue I'm having is I want to add inventory to each size of each SKU
> for
> faceting.  Example,
>
> SKU 1234 has sizes small, medium, large.  Size small has 5 in stock, size
> medium 10, and size large 25.
>
> In a normalized data table I would have a separate table just for inventory
> and related it back to the SKU with a foreign key.  How do I store size and
> inventory information effectively with Solr?
>
> --
> Steve
>



-- 
Steve

RE: Product data schema question

2011-08-16 Thread Jaeger, Jay - DOT

No, I don't think so.  A given core can only use one configuration and 
therefore only one schema, as far as I know, and a schema can only have one key.

You could use two cores with two configurations (but that presumably wouldn't 
be much help).

Solr is not a DBMS.  It is an index.  

-Original Message-
From: Steve Cerny [mailto:sjce...@gmail.com] 
Sent: Tuesday, August 16, 2011 11:37 AM
To: solr-user@lucene.apache.org
Subject: Re: Product data schema question

Jay, this is great information.

I don't know enough about Solr whether this is possible...Can we setup two
indexes in the same core, one for product_catalog and the other for
inventory?  Then using a Solr query we could join the indexed content
together.

In Sql it would look like this

select
 p.brand
 , p.model_name
 , p.sku
 , p.color_name
 , i.variant_id
 , i.variant_count
from product_catalog p
join inventory i on (p.sku = i.sku)

On Tue, Aug 16, 2011 at 8:00 AM, Jaeger, Jay - DOT wrote:

> On the surface, you could simply add some more fields to your schema.  But
> as far as I can tell, you would have to have a separate Solr "document" for
> each SKU/size combination,  and store the rest of the information (brand,
> model, color, SKU) redundantly and make the unique key a combination of the
> SKU and the size (presumably by having an additional field called sku_size -
> as far as I can tell Solr can't make a key up out of multiple fields).
>
> But, perhaps you shouldn't store that dynamic inventory information in
> Solr.   Instead store a key that gets you to an inventory database organized
> by SKU and size.  In such a schema, Size could be added as a multi-valued
> field to your existing schema, so you would know what sizes existed, so you
> could at least facet on that, maybe.  That way Solr doesn't have to be
> updated every time the inventory changes.  Of course, that won't help with
> the faceting  on variant inventory.
>
> -Original Message-
> From: Steve Cerny [mailto:sjce...@gmail.com]
> Sent: Monday, August 15, 2011 6:29 PM
> To: solr-user@lucene.apache.org
> Subject: Product data schema question
>
> I'm working on an online eCommerce project and am having difficulties
> building the core / index schema.  Here is the way we organize our product
> information in a normalized database.
>
> A product model has many SKUs (called colorways)
> A SKU has many sizes (called variants)
> A SKU size has associated inventory (called variant inventory)
>
> When we setup our product core we have the following field information
>
> Doc
> * brand
> * model name
> * SKU
> * color name
>
> Sample records are as follows
>
> * Haynes, Undershirt, 1234, white
> * Haynes, Undershirt, 1235, grey
> * Fruit of the Loom, Undershirt, 1236, white
> * Fruit of the Loom, Underwear, 1237, grey
>
> The issue I'm having is I want to add inventory to each size of each SKU
> for
> faceting.  Example,
>
> SKU 1234 has sizes small, medium, large.  Size small has 5 in stock, size
> medium 10, and size large 25.
>
> In a normalized data table I would have a separate table just for inventory
> and related it back to the SKU with a foreign key.  How do I store size and
> inventory information effectively with Solr?
>
> --
> Steve
>



-- 
Steve

Re: Product data schema question

2011-08-16 Thread Steve Cerny

Thanks Jay, if we come to a reasonable solution are you interested in the
details?

On Tue, Aug 16, 2011 at 11:44 AM, Jaeger, Jay - DOT
wrote:

> No, I don't think so.  A given core can only use one configuration and
> therefore only one schema, as far as I know, and a schema can only have one
> key.
>
> You could use two cores with two configurations (but that presumably
> wouldn't be much help).
>
> Solr is not a DBMS.  It is an index.
>
> -Original Message-
> From: Steve Cerny [mailto:sjce...@gmail.com]
> Sent: Tuesday, August 16, 2011 11:37 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Product data schema question
>
> Jay, this is great information.
>
> I don't know enough about Solr whether this is possible...Can we setup two
> indexes in the same core, one for product_catalog and the other for
> inventory?  Then using a Solr query we could join the indexed content
> together.
>
> In Sql it would look like this
>
> select
>  p.brand
>  , p.model_name
>  , p.sku
>  , p.color_name
>  , i.variant_id
>  , i.variant_count
> from product_catalog p
> join inventory i on (p.sku = i.sku)
>
> On Tue, Aug 16, 2011 at 8:00 AM, Jaeger, Jay - DOT  >wrote:
>
> > On the surface, you could simply add some more fields to your schema.
>  But
> > as far as I can tell, you would have to have a separate Solr "document"
> for
> > each SKU/size combination,  and store the rest of the information (brand,
> > model, color, SKU) redundantly and make the unique key a combination of
> the
> > SKU and the size (presumably by having an additional field called
> sku_size -
> > as far as I can tell Solr can't make a key up out of multiple fields).
> >
> > But, perhaps you shouldn't store that dynamic inventory information in
> > Solr.   Instead store a key that gets you to an inventory database
> organized
> > by SKU and size.  In such a schema, Size could be added as a multi-valued
> > field to your existing schema, so you would know what sizes existed, so
> you
> > could at least facet on that, maybe.  That way Solr doesn't have to be
> > updated every time the inventory changes.  Of course, that won't help
> with
> > the faceting  on variant inventory.
> >
> > -Original Message-
> > From: Steve Cerny [mailto:sjce...@gmail.com]
> > Sent: Monday, August 15, 2011 6:29 PM
> > To: solr-user@lucene.apache.org
> > Subject: Product data schema question
> >
> > I'm working on an online eCommerce project and am having difficulties
> > building the core / index schema.  Here is the way we organize our
> product
> > information in a normalized database.
> >
> > A product model has many SKUs (called colorways)
> > A SKU has many sizes (called variants)
> > A SKU size has associated inventory (called variant inventory)
> >
> > When we setup our product core we have the following field information
> >
> > Doc
> > * brand
> > * model name
> > * SKU
> > * color name
> >
> > Sample records are as follows
> >
> > * Haynes, Undershirt, 1234, white
> > * Haynes, Undershirt, 1235, grey
> > * Fruit of the Loom, Undershirt, 1236, white
> > * Fruit of the Loom, Underwear, 1237, grey
> >
> > The issue I'm having is I want to add inventory to each size of each SKU
> > for
> > faceting.  Example,
> >
> > SKU 1234 has sizes small, medium, large.  Size small has 5 in stock, size
> > medium 10, and size large 25.
> >
> > In a normalized data table I would have a separate table just for
> inventory
> > and related it back to the SKU with a foreign key.  How do I store size
> and
> > inventory information effectively with Solr?
> >
> > --
> > Steve
> >
>
>
>
> --
> Steve
>



-- 
Steve Cerny
sjce...@gmail.com
715-302-0639

Re: Exact matching on names?

2011-08-16 Thread Sujit Pal

Hi Ron,

There was a discussion about this some time back, which I implemented
(with great success btw) in my own code...basically you store both the
analyzed and non-analyzed versions (use string type) in the index, then
send in a query like this:

+name:clarke name_s:"clarke"^100

The name field is text so it will analyze down "clarke" to "clark" but
it will match both "clark" and "clarke" and the second clause would
boost the entry with "clarke" up to the top, which you then select with
rows=1.

-sujit

On Tue, 2011-08-16 at 10:20 -0500, Olson, Ron wrote:
> Hi all-
> 
> I'm missing something fundamental yet I've been unable to find the definitive 
> answer for exact name matching. I'm indexing names using the standard "text" 
> field type and my search is for the name "clarke". My results include 
> "clark", which is incorrect, it needs to match clarke exactly (case 
> insensitive).
> 
> I tried textType but that doesn't work because I believe it needs to be 
> *really* exact, whereas I'm looking for things like "clark oil", "bob, frank, 
> and clark", etc.
> 
> Thanks for any help,
> 
> Ron
> 
> DISCLAIMER: This electronic message, including any attachments, files or 
> documents, is intended only for the addressee and may contain CONFIDENTIAL, 
> PROPRIETARY or LEGALLY PRIVILEGED information.  If you are not the intended 
> recipient, you are hereby notified that any use, disclosure, copying or 
> distribution of this message or any of the information included in or with it 
> is  unauthorized and strictly prohibited.  If you have received this message 
> in error, please notify the sender immediately by reply e-mail and 
> permanently delete and destroy this message and its attachments, along with 
> any copies thereof. This message does not create any contractual obligation 
> on behalf of the sender or Law Bulletin Publishing Company.
> Thank you.

Re: Indexing from a database via SolrJ

2011-08-16 Thread Erick Erickson

The problem with anything "automatic" is that I don't see how it could know
which fields in the document to map DB columns to. Unless you had
fields that exactly matched column names, it would be iffy...

I assume DIH actually does something like this, but don't know any way
of having SolrJ automagically do this.

At root these kinds of things don't generalize well, but that doesn't mean
that there's not a good case for doing this.

Best
Erick

On Tue, Aug 16, 2011 at 11:26 AM, Shawn Heisey  wrote:
> On 8/16/2011 7:14 AM, Erick Erickson wrote:
>>
>> What have you tried and what doesn't it do that you want it to do?
>>
>> This works, instantiating the StreamingUpdateSolrServer (server) and
>> the JDBC connection/SQL statement are left as exercises for the
>> reader.:
>>
>>     while (rs.next()) {
>>       SolrInputDocument doc = new SolrInputDocument();
>>
>>       String id = rs.getString("id");
>>       String title = rs.getString("title");
>>       String text = rs.getString("text");
>>
>>       doc.addField("id", id);
>>       doc.addField("title", title);
>>       doc.addField("text", text);
>>
>>       docs.add(doc);
>>       ++counter;
>>       ++total;
>>       if (counter>  100) { // Completely arbitrary, just batch up more
>> than one document for throughput!
>>         server.add(docs);
>>         docs.clear();
>>         counter = 0;
>>       }
>>     }
>
> I've implemented a basic loop with the structure you've demonstrated, but it
> currently doesn't do anything yet with SolrInputDocument or
> SolrDocumentList.  I figured there would be a way to avoid going through the
> field list one by one, but what you've written suggests that the
> field-by-field method is required.  I can live with that.
>
> It does look like addField just takes an Object, so hopefully I can create a
> loop that determines the type of each field from the JDBC metadata,
> retrieves the correct Java type from the ResultSet, and inserts it.  I
> imagine that everything still works if you happen to insert a field that
> doesn't exist in the index.  This must be how the DIH does it, so I was
> hoping that the DIH might expose a method that takes a ResultSet and
> produces a SolrDocumentList.  I still have to take a deeper look at the
> source and documentation.
>
> Thanks for the help so far, I can get a little more implemented now.
>
> Shawn
>
>

RE: Product data schema question

2011-08-16 Thread Jaeger, Jay - DOT

Not particularly.  Just trying to do my part to answer some questions on the 
list.

-Original Message-
From: Steve Cerny [mailto:sjce...@gmail.com] 
Sent: Tuesday, August 16, 2011 11:49 AM
To: solr-user@lucene.apache.org
Subject: Re: Product data schema question

Thanks Jay, if we come to a reasonable solution are you interested in the
details?

On Tue, Aug 16, 2011 at 11:44 AM, Jaeger, Jay - DOT
wrote:

> No, I don't think so.  A given core can only use one configuration and
> therefore only one schema, as far as I know, and a schema can only have one
> key.
>
> You could use two cores with two configurations (but that presumably
> wouldn't be much help).
>
> Solr is not a DBMS.  It is an index.
>
> -Original Message-
> From: Steve Cerny [mailto:sjce...@gmail.com]
> Sent: Tuesday, August 16, 2011 11:37 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Product data schema question
>
> Jay, this is great information.
>
> I don't know enough about Solr whether this is possible...Can we setup two
> indexes in the same core, one for product_catalog and the other for
> inventory?  Then using a Solr query we could join the indexed content
> together.
>
> In Sql it would look like this
>
> select
>  p.brand
>  , p.model_name
>  , p.sku
>  , p.color_name
>  , i.variant_id
>  , i.variant_count
> from product_catalog p
> join inventory i on (p.sku = i.sku)
>
> On Tue, Aug 16, 2011 at 8:00 AM, Jaeger, Jay - DOT  >wrote:
>
> > On the surface, you could simply add some more fields to your schema.
>  But
> > as far as I can tell, you would have to have a separate Solr "document"
> for
> > each SKU/size combination,  and store the rest of the information (brand,
> > model, color, SKU) redundantly and make the unique key a combination of
> the
> > SKU and the size (presumably by having an additional field called
> sku_size -
> > as far as I can tell Solr can't make a key up out of multiple fields).
> >
> > But, perhaps you shouldn't store that dynamic inventory information in
> > Solr.   Instead store a key that gets you to an inventory database
> organized
> > by SKU and size.  In such a schema, Size could be added as a multi-valued
> > field to your existing schema, so you would know what sizes existed, so
> you
> > could at least facet on that, maybe.  That way Solr doesn't have to be
> > updated every time the inventory changes.  Of course, that won't help
> with
> > the faceting  on variant inventory.
> >
> > -Original Message-
> > From: Steve Cerny [mailto:sjce...@gmail.com]
> > Sent: Monday, August 15, 2011 6:29 PM
> > To: solr-user@lucene.apache.org
> > Subject: Product data schema question
> >
> > I'm working on an online eCommerce project and am having difficulties
> > building the core / index schema.  Here is the way we organize our
> product
> > information in a normalized database.
> >
> > A product model has many SKUs (called colorways)
> > A SKU has many sizes (called variants)
> > A SKU size has associated inventory (called variant inventory)
> >
> > When we setup our product core we have the following field information
> >
> > Doc
> > * brand
> > * model name
> > * SKU
> > * color name
> >
> > Sample records are as follows
> >
> > * Haynes, Undershirt, 1234, white
> > * Haynes, Undershirt, 1235, grey
> > * Fruit of the Loom, Undershirt, 1236, white
> > * Fruit of the Loom, Underwear, 1237, grey
> >
> > The issue I'm having is I want to add inventory to each size of each SKU
> > for
> > faceting.  Example,
> >
> > SKU 1234 has sizes small, medium, large.  Size small has 5 in stock, size
> > medium 10, and size large 25.
> >
> > In a normalized data table I would have a separate table just for
> inventory
> > and related it back to the SKU with a foreign key.  How do I store size
> and
> > inventory information effectively with Solr?
> >
> > --
> > Steve
> >
>
>
>
> --
> Steve
>



-- 
Steve Cerny
sjce...@gmail.com
715-302-0639

Re: Exception DirectSolrSpellChecker when using spellcheck.q

2011-08-16 Thread O. Klein

Thanx. I was using a build of the day you fixed the bug :)

Keep up the good work.

--
View this message in context: 
http://lucene.472066.n3.nabble.com/Exception-DirectSolrSpellChecker-when-using-spellcheck-q-tp3249565p3259372.html
Sent from the Solr - User mailing list archive at Nabble.com.

Random + Boost?

2011-08-16 Thread Steven Ou

Hey guys,

This might seem odd, but is it possible to use boost with random ordering?
That is, documents that get boosted are more likely to appear towards the
top of the ordering (I only display page 1, say 30 documents). Does that
make sense? I'm assuming that random ordering is, well, really random - so
then it's not possible. But I figured I'd ask.

My problem is that I want to display a random assortment of documents, but
unfortunately certain types of documents far outnumber other types. So a
"random" assortment ends up with 50% type A, 50% type B, C, D, E, F. So, I
was thinking I would essentially "boost" types B, C, D, E, F until all types
are approximately evenly represented in the random assortment. (Or
alternatively, if the user has an affinity for type B documents, further
boost type B documents so that they're more likely to be represented than
other types).

Anyone know if there's a way to do something like this in Solr?

Much appreciated. Thanks.
--
Steven Ou | 歐偉凡

*ravn.com* | Chief Technology Officer
steve...@gmail.com | +1 909-569-9880

How to connect Solr with external Zookeeper ensemble

2011-08-16 Thread Sharath Jagannath

Hey,

How could I connect my solr server with external zookeeper?

Thanks,
Sharath

Faceted Search Patent Lawsuit - Please Read

2011-08-16 Thread LaMaze Johnson

Just wanted to make people aware of a company called Kolera that is sending
threatening letters via their law firm, manatt, phelps, phillips in regard
to violation of their patent U.S. Patent No. 6,275,821 titled "Method and
System for Executing a Guided Parametric Search.

Apparently, they believe that this patent gives them the rights to faceted
search technology.  Consequently, anyone who uses this technology without
paying licensing fees is potentially a target of litigation.  They have
outstanding suits with companies such as Microsoft and Ebay.  Additionally,
Adobe filed a counter-suit on August 10, 2011.Unfortunately, I received
a letter this afternoon and we are determining how to proceed.

Just thought I would make others aware of this.  I'd appreciate any insight
others might have regarding the issue.

--
View this message in context: 
http://lucene.472066.n3.nabble.com/Faceted-Search-Patent-Lawsuit-Please-Read-tp3259475p3259475.html
Sent from the Solr - User mailing list archive at Nabble.com.

Search by range in multivalued fields

2011-08-16 Thread marotosg

Hi.

I have a solr core with job records and one guy can work in different
companies in
a specific range of "dateini" to "dateend".

  
  
IBM10012005
APPLE10012005

  
IBM10012005
APPLE10012005
  
   

Is possible to make a range query on a multivalue field over text fields.
For instance something like that.
companyinimultivaluefield['IBM10012005' TO *] AND 
companyendmultivaluefield['IBM10012005' TO *]




--
View this message in context: 
http://lucene.472066.n3.nabble.com/Search-by-range-in-multivalued-fields-tp3259343p3259343.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: SolrJ & Spatial Search

2011-08-16 Thread federico.wachs

I'm sorry, i'm trying to do the same as he is.
I've read your reply many times now, but i still don't know how to do this.

Would somebody help me with this? Thanks a lot

--
View this message in context: 
http://lucene.472066.n3.nabble.com/Spatial-search-with-SolrJ-3-1-How-to-tp2961136p3259456.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Random + Boost?

2011-08-16 Thread Ahmet Arslan

> This might seem odd, but is it possible to use boost with
> random ordering?
> That is, documents that get boosted are more likely to
> appear towards the
> top of the ordering (I only display page 1, say 30
> documents). Does that
> make sense? I'm assuming that random ordering is, well,
> really random - so
> then it's not possible. But I figured I'd ask.
> 
> My problem is that I want to display a random assortment of
> documents, but
> unfortunately certain types of documents far outnumber
> other types. So a
> "random" assortment ends up with 50% type A, 50% type B, C,
> D, E, F. So, I
> was thinking I would essentially "boost" types B, C, D, E,
> F until all types
> are approximately evenly represented in the random
> assortment. (Or
> alternatively, if the user has an affinity for type B
> documents, further
> boost type B documents so that they're more likely to be
> represented than
> other types).
> 
> Anyone know if there's a way to do something like this in
> Solr?

Sounds like you want to achieve diversity of results.

Consider using http://wiki.apache.org/solr/FieldCollapsing

Alternatively you can make use of RandomSortField with function queries.
http://lucene.apache.org/solr/api/org/apache/solr/schema/RandomSortField.html

Re: Faceted Search Patent Lawsuit - Please Read

2011-08-16 Thread Gora Mohanty

On Wed, Aug 17, 2011 at 12:03 AM, LaMaze Johnson  wrote:
[...]
> Just thought I would make others aware of this.  I'd appreciate any insight
> others might have regarding the issue.
[...]

If you will permit me a moment of levity, from the perspective of
someone in India, I would say, move to a non-third-world country
that does not believe in Irrational Property Rights (IPR). Apply
smileys as needed.

Regards,
Gora

Re: Random + Boost?

2011-08-16 Thread Alexei Martchenko

To make random results i'd use something related to dates and milliseconds,
not boosting. lemme think about this...

2011/8/16 Ahmet Arslan 

> > This might seem odd, but is it possible to use boost with
> > random ordering?
> > That is, documents that get boosted are more likely to
> > appear towards the
> > top of the ordering (I only display page 1, say 30
> > documents). Does that
> > make sense? I'm assuming that random ordering is, well,
> > really random - so
> > then it's not possible. But I figured I'd ask.
> >
> > My problem is that I want to display a random assortment of
> > documents, but
> > unfortunately certain types of documents far outnumber
> > other types. So a
> > "random" assortment ends up with 50% type A, 50% type B, C,
> > D, E, F. So, I
> > was thinking I would essentially "boost" types B, C, D, E,
> > F until all types
> > are approximately evenly represented in the random
> > assortment. (Or
> > alternatively, if the user has an affinity for type B
> > documents, further
> > boost type B documents so that they're more likely to be
> > represented than
> > other types).
> >
> > Anyone know if there's a way to do something like this in
> > Solr?
>
> Sounds like you want to achieve diversity of results.
>
> Consider using http://wiki.apache.org/solr/FieldCollapsing
>
> Alternatively you can make use of RandomSortField with function queries.
>
> http://lucene.apache.org/solr/api/org/apache/solr/schema/RandomSortField.html
>



-- 

*Alexei Martchenko* | *CEO* | Superdownloads
ale...@superdownloads.com.br | ale...@martchenko.com.br | (11)
5083.1018/5080.3535/5080.3533

Re: Indexing from a database via SolrJ

2011-08-16 Thread Shawn Heisey


On 8/16/2011 11:23 AM, Erick Erickson wrote:

The problem with anything "automatic" is that I don't see how it could know
which fields in the document to map DB columns to. Unless you had
fields that exactly matched column names, it would be iffy...

I assume DIH actually does something like this, but don't know any way
of having SolrJ automagically do this.

At root these kinds of things don't generalize well, but that doesn't mean
that there's not a good case for doing this.


In my case, the Solr field names are in perfect sync with the database 
field names.  My DIH config doesn't mention any fields by name, it just 
passes them as-is and lets the schema handle everything.  I'm perfectly 
OK with handling everything myself in my code, but if someone had 
already invented the wheel, no sense in designing a new one. :)


Thanks for all your help, Erick.

Shawn

custom filterquery

2011-08-16 Thread Jon Wagoner

Hello,

I am writing software for an e-commerce site.  Different customers can have
different selections of product depending on what is priced out for them, so
to get the faceting counts correct I need to filter the values based on the
pricing.  I have written a functionquery to get the pricing, which works
fine as part of the search query, but doesn't seem to be doing anything when
I try to use it in a filter query.  I wrote my pricing function query based
on
http://www.supermind.org/blog/756/how-to-write-a-custom-solr-functionquery,
and I can see the parser part getting logged from the filter query, but
nothing ever calls getValues on my ValueSource.  If I use my function query
as part of the main query, getValues is getting called.  Can anyone point me
in the right direction to get this working in the filter query?

Jon Wagoner

Re: Random + Boost?

2011-08-16 Thread Steven Ou

Thank you for the response! I'm learning much about Solr... So I think
FieldCollapsing might do the trick... So if I understand correctly, I should
be able to group by type A, B, C, D, E, F, sort groups randomly, sort within
groups randomly, display simple format, and get an evenly distributed set of
results across A, B, C, D, E, F...

But I'm not sure it's exactly what I need:

   - If I have, in this case, 6 document types, how do I return 25
   documents? From what I understand, I need to set how many groups (let's say
   all 6) * how many documents per group (closest would be 4) = 24 documents.
   Would I simply need to return more documents and just process the first 25?
   - Also, this appears to me to return *exactly* evenly distributed
   results. How do I:
  - Make certain types a little more likely to appear, or
  - If, suppose, I have only 2 documents in type F and I query for all 6
  groups, at 5 documents per group (for a total of 30 documents), will the
  fact that type F only has 2 documents result in a query that
only returns 27
  documents? If so, I would want to select the missing 3 documents
evenly from
  the other 5 types.

As for RandomSortField + function queries... I'm not sure I understand how I
can use that to achieve what I need :-/
--
Steven Ou | 歐偉凡

*ravn.com* | Chief Technology Officer
steve...@gmail.com | +1 909-569-9880


On Tue, Aug 16, 2011 at 12:00 PM, Ahmet Arslan  wrote:

> > This might seem odd, but is it possible to use boost with
> > random ordering?
> > That is, documents that get boosted are more likely to
> > appear towards the
> > top of the ordering (I only display page 1, say 30
> > documents). Does that
> > make sense? I'm assuming that random ordering is, well,
> > really random - so
> > then it's not possible. But I figured I'd ask.
> >
> > My problem is that I want to display a random assortment of
> > documents, but
> > unfortunately certain types of documents far outnumber
> > other types. So a
> > "random" assortment ends up with 50% type A, 50% type B, C,
> > D, E, F. So, I
> > was thinking I would essentially "boost" types B, C, D, E,
> > F until all types
> > are approximately evenly represented in the random
> > assortment. (Or
> > alternatively, if the user has an affinity for type B
> > documents, further
> > boost type B documents so that they're more likely to be
> > represented than
> > other types).
> >
> > Anyone know if there's a way to do something like this in
> > Solr?
>
> Sounds like you want to achieve diversity of results.
>
> Consider using http://wiki.apache.org/solr/FieldCollapsing
>
> Alternatively you can make use of RandomSortField with function queries.
>
> http://lucene.apache.org/solr/api/org/apache/solr/schema/RandomSortField.html
>

Re: Faceted Search Patent Lawsuit - Please Read

2011-08-16 Thread LaMaze Johnson


Gora Mohanty-3 wrote:
> 
> On Wed, Aug 17, 2011 at 12:03 AM, LaMaze Johnson 
> wrote:
> [...]
>> Just thought I would make others aware of this.  I'd appreciate any
>> insight
>> others might have regarding the issue.
> [...]
> 
> If you will permit me a moment of levity, from the perspective of
> someone in India, I would say, move to a non-third-world country
> that does not believe in Irrational Property Rights (IPR). Apply
> smileys as needed.
> 
> Regards,
> Gora
> 

Permission granted.  I'm interested to know how all of this will play out in
the courts.  Faceted search seems to be a component of every e-commerce site
on the web.

--
View this message in context: 
http://lucene.472066.n3.nabble.com/Faceted-Search-Patent-Lawsuit-Please-Read-tp3259475p3259672.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Faceted Search Patent Lawsuit - Please Read

2011-08-16 Thread Grant Ingersoll

I know you mean well and are probably wondering what to do next, but such a 
discussion is really beyond the scope of this mailing list.  Most of us aren't 
lawyers (I wonder if anyone here is?) and if we were, we wouldn't likely 
speculate in public on something that can only be decided in the courts.

-Grant

On Aug 16, 2011, at 2:33 PM, LaMaze Johnson wrote:

> Just wanted to make people aware of a company called Kolera that is sending
> threatening letters via their law firm, manatt, phelps, phillips in regard
> to violation of their patent U.S. Patent No. 6,275,821 titled "Method and
> System for Executing a Guided Parametric Search.
> 
> Apparently, they believe that this patent gives them the rights to faceted
> search technology.  Consequently, anyone who uses this technology without
> paying licensing fees is potentially a target of litigation.  They have
> outstanding suits with companies such as Microsoft and Ebay.  Additionally,
> Adobe filed a counter-suit on August 10, 2011.Unfortunately, I received
> a letter this afternoon and we are determining how to proceed.
> 
> Just thought I would make others aware of this.  I'd appreciate any insight
> others might have regarding the issue.
> 
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/Faceted-Search-Patent-Lawsuit-Please-Read-tp3259475p3259475.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Unable to get multicore working

2011-08-16 Thread David Sauve

I've been trying (unsuccessfully) to get multicore working for about a day and 
a half now I'm nearly at wits end and unsure what to do anymore. **Any** help 
would be appreciated.

I've installed Solr using the solr-jetty packages on Ubuntu 10.04. The default 
Solr install seems to work fine.

Now, I want to add three cores: live, staging, preview to be used for the 
various states of the site.

I've created a `solr.xml` file as follows and symlinked it in to 
/usr/share/solr: 










Now, when I try to view any cores, I get a 404 - Not found. In fact, I can't 
even view /solr/admin/ anymore after installing that `solr.xml` file.

Also, /solr/admin/cores returns an XML file, but it looks to me like there's no 
cores listed. The output:



0
0









Finally, looking through the logs produced by Jetty doesn't seem to reveal any 
clues about what is wrong. There doesn't seem to be any errors in there, except 
the 404s.

Long story short. I'm stuck. Any suggestions on where to go with this?

David

Re: Faceted Search Patent Lawsuit - Please Read

2011-08-16 Thread LaMaze Johnson


Grant Ingersoll-2 wrote:
> 
> I know you mean well and are probably wondering what to do next, but such
> a discussion is really beyond the scope of this mailing list.  Most of us
> aren't lawyers (I wonder if anyone here is?) and if we were, we wouldn't
> likely speculate in public on something that can only be decided in the
> courts.
> 
> -Grant
> 

We will definitely be in touch with our legal counsel.  This was more a
"heads up" in hopes that others aren't blindsided by this.  


--
View this message in context: 
http://lucene.472066.n3.nabble.com/Faceted-Search-Patent-Lawsuit-Please-Read-tp3259475p3259733.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: SolrJ & Spatial Search

2011-08-16 Thread David Smiley (@MITRE.org)

solrQuery.setQuery("*:*");
solrQuery.addFilterQuery("{!func}geodist()");
solrQuery.set("sfield", "store");
solrQuery.set("pt", lat + "," + lon);
solrQuery.set("sort", "geodist() asc");

//disclaimer: I haven't run this

-
 Author: https://www.packtpub.com/solr-1-4-enterprise-search-server/book
--
View this message in context: 
http://lucene.472066.n3.nabble.com/Spatial-search-with-SolrJ-3-1-How-to-tp2961136p3259743.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: Unable to get multicore working

2011-08-16 Thread Jaeger, Jay - DOT

Perhaps your admin doesn’t work because you don't have 
defaultCoreName="whatever-core-you-want-by-default" in your  tag?  E.g.:



Perhaps this was enough to prevent it starting any cores -- I'd expect a 
default to be required.

Also, from experience, if you add cores, and you have security turned on, you 
probably need to modify web.xml to teach it about them, so it maps the URLs to 
the right place with the right security.

Finally, check your logs to make sure that Solr isn't complaining about 
something else (like not having a defaultCoreName, for instance)


-Original Message-
From: David Sauve [mailto:dnsa...@gmail.com] 
Sent: Tuesday, August 16, 2011 3:02 PM
To: solr-user@lucene.apache.org
Subject: Unable to get multicore working

I've been trying (unsuccessfully) to get multicore working for about a day and 
a half now I'm nearly at wits end and unsure what to do anymore. **Any** help 
would be appreciated.

I've installed Solr using the solr-jetty packages on Ubuntu 10.04. The default 
Solr install seems to work fine.

Now, I want to add three cores: live, staging, preview to be used for the 
various states of the site.

I've created a `solr.xml` file as follows and symlinked it in to 
/usr/share/solr: 










Now, when I try to view any cores, I get a 404 - Not found. In fact, I can't 
even view /solr/admin/ anymore after installing that `solr.xml` file.

Also, /solr/admin/cores returns an XML file, but it looks to me like there's no 
cores listed. The output:



0
0









Finally, looking through the logs produced by Jetty doesn't seem to reveal any 
clues about what is wrong. There doesn't seem to be any errors in there, except 
the 404s.

Long story short. I'm stuck. Any suggestions on where to go with this?

David

Re: Unable to get multicore working

2011-08-16 Thread Alexei Martchenko

Lets try something simplier.
My start.jar is on \apache-solr-3.3.0\example\
Here's my local config placed in \apache-solr-3.3.0\example\solr\





  


Create \apache-solr-3.3.0\example\solr\softwares01\conf\
and \apache-solr-3.3.0\example\solr\softwares01\data\

http://localhost:8983/solr/ should work and so is
http://localhost:8983/solr/softwares01/admin/



2011/8/16 David Sauve 

> I've been trying (unsuccessfully) to get multicore working for about a day
> and a half now I'm nearly at wits end and unsure what to do anymore. **Any**
> help would be appreciated.
>
> I've installed Solr using the solr-jetty packages on Ubuntu 10.04. The
> default Solr install seems to work fine.
>
> Now, I want to add three cores: live, staging, preview to be used for the
> various states of the site.
>
> I've created a `solr.xml` file as follows and symlinked it in to
> /usr/share/solr:
>
> 
> 
> 
>  dataDir="/home/webteam/preview/data" />
>  dataDir="/home/webteam/staging/data" />
>  dataDir="/home/webteam/live/data" />
> 
> 
>
> Now, when I try to view any cores, I get a 404 - Not found. In fact, I
> can't even view /solr/admin/ anymore after installing that `solr.xml` file.
>
> Also, /solr/admin/cores returns an XML file, but it looks to me like
> there's no cores listed. The output:
>
> 
> 
> 0
> 0
>
> 
>
>
> 
>
> 
>
>
> Finally, looking through the logs produced by Jetty doesn't seem to reveal
> any clues about what is wrong. There doesn't seem to be any errors in there,
> except the 404s.
>
> Long story short. I'm stuck. Any suggestions on where to go with this?
>
> David
>
>


-- 

*Alexei Martchenko* | *CEO* | Superdownloads
ale...@superdownloads.com.br | ale...@martchenko.com.br | (11)
5083.1018/5080.3535/5080.3533

Re: Unable to get multicore working

2011-08-16 Thread David Sauve

 I've installed using aptitude so I don't have an example folder (that I can 
find). 

/solr/ does work (but lists no cores)
/solr/live/admin/ does not -- 404


On Tuesday, 16 August, 2011 at 1:13 PM, Alexei Martchenko wrote:

> Lets try something simplier.
> My start.jar is on \apache-solr-3.3.0\example\
> Here's my local config placed in \apache-solr-3.3.0\example\solr\
> 
> 
> 
> 
> 
> 
> 
> 
> Create \apache-solr-3.3.0\example\solr\softwares01\conf\
> and \apache-solr-3.3.0\example\solr\softwares01\data\
> 
> http://localhost:8983/solr/ should work and so is
> http://localhost:8983/solr/softwares01/admin/
> 
> 
> 
> 2011/8/16 David Sauve mailto:dnsa...@gmail.com)>
> 
> > I've been trying (unsuccessfully) to get multicore working for about a day
> > and a half now I'm nearly at wits end and unsure what to do anymore. **Any**
> > help would be appreciated.
> > 
> > I've installed Solr using the solr-jetty packages on Ubuntu 10.04. The
> > default Solr install seems to work fine.
> > 
> > Now, I want to add three cores: live, staging, preview to be used for the
> > various states of the site.
> > 
> > I've created a `solr.xml` file as follows and symlinked it in to
> > /usr/share/solr:
> > 
> > 
> > 
> > 
> >  > dataDir="/home/webteam/preview/data" />
> >  > dataDir="/home/webteam/staging/data" />
> >  > dataDir="/home/webteam/live/data" />
> > 
> > 
> > 
> > Now, when I try to view any cores, I get a 404 - Not found. In fact, I
> > can't even view /solr/admin/ anymore after installing that `solr.xml` file.
> > 
> > Also, /solr/admin/cores returns an XML file, but it looks to me like
> > there's no cores listed. The output:
> > 
> > 
> > 
> > 0
> > 0
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > Finally, looking through the logs produced by Jetty doesn't seem to reveal
> > any clues about what is wrong. There doesn't seem to be any errors in there,
> > except the 404s.
> > 
> > Long story short. I'm stuck. Any suggestions on where to go with this?
> > 
> > David
> 
> 
> -- 
> 
> *Alexei Martchenko* | *CEO* | Superdownloads
> ale...@superdownloads.com.br (mailto:ale...@superdownloads.com.br) | 
> ale...@martchenko.com.br (mailto:ale...@martchenko.com.br) | (11)
> 5083.1018/5080.3535/5080.3533

Re: Indexing from a database via SolrJ

2011-08-16 Thread Shawn Heisey


On 8/16/2011 1:12 PM, Shawn Heisey wrote:

On 8/16/2011 11:23 AM, Erick Erickson wrote:
The problem with anything "automatic" is that I don't see how it 
could know

which fields in the document to map DB columns to. Unless you had
fields that exactly matched column names, it would be iffy...

I assume DIH actually does something like this, but don't know any way
of having SolrJ automagically do this.

At root these kinds of things don't generalize well, but that doesn't 
mean

that there's not a good case for doing this.


In my case, the Solr field names are in perfect sync with the database 
field names.  My DIH config doesn't mention any fields by name, it 
just passes them as-is and lets the schema handle everything.  I'm 
perfectly OK with handling everything myself in my code, but if 
someone had already invented the wheel, no sense in designing a new 
one. :)


Thanks for all your help, Erick.


Here's what I've ended up with in my method that takes a ResultSet and 
puts the data into Solr.  I have to get a testbed set up before I can 
actually test this code, which will take me a while.  I'm inviting 
comment now, knowing it might have bugs.  Eclipse is happy with it, but 
that doesn't mean it works. :)


/**
 * Takes an SQL ResultSet and adds the documents to solr. Does it 
in batches

 * of fetchSize.
 *
 * @param rs
 * @throws SQLException
 * @throws IOException
 * @throws SolrServerException
 */
private long addResultSet(ResultSet rs) throws SQLException,
SolrServerException, IOException
{
long count = 0;
int innerCount = 0;
Collection docs = new 
ArrayList();

ResultSetMetaData rsm = rs.getMetaData();
int numColumns = rsm.getColumnCount();
String[] colNames = new String[numColumns + 1];

for (int i = 1; i < (numColumns + 1); i++)
{
colNames[i] = rsm.getColumnName(i);
}

while (rs.next())
{
count++;
innerCount++;

SolrInputDocument doc = new SolrInputDocument();
for (int j = 1; j < (numColumns + 1); j++)
{
Object f;
switch (rsm.getColumnType(j))
{
case Types.BIGINT:
{
f = rs.getLong(j);
break;
}
case Types.INTEGER:
{
f = rs.getInt(j);
break;
}
case Types.DATE:
{
f = rs.getDate(j);
break;
}
case Types.FLOAT:
{
f = rs.getFloat(j);
break;
}
case Types.DOUBLE:
{
f = rs.getDouble(j);
break;
}
case Types.TIME:
{
f = rs.getDate(j);
break;
}
case Types.BOOLEAN:
{
f = rs.getBoolean(j);
break;
}
default:
{
f = rs.getString(j);
}
}
doc.addField(colNames[j], f);
}
docs.add(doc);

/**
 * When we reach fetchSize, index the documents and reset 
the inner

 * counter.
 */
if (innerCount == IdxStatic.fetchSize)
{
solrCore.add(docs);
docs.clear();
innerCount = 0;
}
}

/**
 * If the outer loop ended before the inner loop reset, index the
 * remaining documents.
 */
if (innerCount != 0)
{
solrCore.add(docs);
}
return count;
}

Re: Unable to get multicore working

2011-08-16 Thread David Sauve

I tried setting `defaultCoreName="admin"` and that didn't seem to change 
anything.

I also tried adding an `env-entry` for "solr/home" pointing to 
"/home/webteam/config" but that didn't seem to help either.

The logs don't have any errors in them, besides 404 errors.  


On Tuesday, 16 August, 2011 at 1:10 PM, Jaeger, Jay - DOT wrote:

> Perhaps your admin doesn’t work because you don't have 
> defaultCoreName="whatever-core-you-want-by-default" in your  tag? E.g.:
>  
> 
>  
> Perhaps this was enough to prevent it starting any cores -- I'd expect a 
> default to be required.
>  
> Also, from experience, if you add cores, and you have security turned on, you 
> probably need to modify web.xml (http://web.xml) to teach it about them, so 
> it maps the URLs to the right place with the right security.
>  
> Finally, check your logs to make sure that Solr isn't complaining about 
> something else (like not having a defaultCoreName, for instance)
>  
>  
> -Original Message-
> From: David Sauve [mailto:dnsa...@gmail.com]  
> Sent: Tuesday, August 16, 2011 3:02 PM
> To: solr-user@lucene.apache.org (mailto:solr-user@lucene.apache.org)
> Subject: Unable to get multicore working
>  
> I've been trying (unsuccessfully) to get multicore working for about a day 
> and a half now I'm nearly at wits end and unsure what to do anymore. **Any** 
> help would be appreciated.
>  
> I've installed Solr using the solr-jetty packages on Ubuntu 10.04. The 
> default Solr install seems to work fine.
>  
> Now, I want to add three cores: live, staging, preview to be used for the 
> various states of the site.
>  
> I've created a `solr.xml` file as follows and symlinked it in to 
> /usr/share/solr:  
>  
> 
> 
> 
>  dataDir="/home/webteam/preview/data" />
>  dataDir="/home/webteam/staging/data" />
>  dataDir="/home/webteam/live/data" />
> 
> 
>  
> Now, when I try to view any cores, I get a 404 - Not found. In fact, I can't 
> even view /solr/admin/ anymore after installing that `solr.xml` file.
>  
> Also, /solr/admin/cores returns an XML file, but it looks to me like there's 
> no cores listed. The output:
>  
> 
> 
> 0
> 0
>  
> 
>  
>  
> 
>  
> 
>  
>  
> Finally, looking through the logs produced by Jetty doesn't seem to reveal 
> any clues about what is wrong. There doesn't seem to be any errors in there, 
> except the 404s.
>  
> Long story short. I'm stuck. Any suggestions on where to go with this?
>  
> David

Re: Unable to get multicore working

2011-08-16 Thread Alexei Martchenko

AFAIK you're still seeing singlecore version

where is your start.jar?

search for solr.xml, see how many u've got plz.

2011/8/16 David Sauve 

>  I've installed using aptitude so I don't have an example folder (that I
> can find).
>
> /solr/ does work (but lists no cores)
> /solr/live/admin/ does not -- 404
>
>
> On Tuesday, 16 August, 2011 at 1:13 PM, Alexei Martchenko wrote:
>
> > Lets try something simplier.
> > My start.jar is on \apache-solr-3.3.0\example\
> > Here's my local config placed in \apache-solr-3.3.0\example\solr\
> >
> > 
> > 
> > 
> > 
> > 
> > 
> >
> > Create \apache-solr-3.3.0\example\solr\softwares01\conf\
> > and \apache-solr-3.3.0\example\solr\softwares01\data\
> >
> > http://localhost:8983/solr/ should work and so is
> > http://localhost:8983/solr/softwares01/admin/
> >
> >
> >
> > 2011/8/16 David Sauve mailto:dnsa...@gmail.com)>
> >
> > > I've been trying (unsuccessfully) to get multicore working for about a
> day
> > > and a half now I'm nearly at wits end and unsure what to do anymore.
> **Any**
> > > help would be appreciated.
> > >
> > > I've installed Solr using the solr-jetty packages on Ubuntu 10.04. The
> > > default Solr install seems to work fine.
> > >
> > > Now, I want to add three cores: live, staging, preview to be used for
> the
> > > various states of the site.
> > >
> > > I've created a `solr.xml` file as follows and symlinked it in to
> > > /usr/share/solr:
> > >
> > > 
> > > 
> > > 
> > >  > > dataDir="/home/webteam/preview/data" />
> > >  > > dataDir="/home/webteam/staging/data" />
> > >  > > dataDir="/home/webteam/live/data" />
> > > 
> > > 
> > >
> > > Now, when I try to view any cores, I get a 404 - Not found. In fact, I
> > > can't even view /solr/admin/ anymore after installing that `solr.xml`
> file.
> > >
> > > Also, /solr/admin/cores returns an XML file, but it looks to me like
> > > there's no cores listed. The output:
> > >
> > > 
> > > 
> > > 0
> > > 0
> > >
> > > 
> > >
> > >
> > > 
> > >
> > > 
> > >
> > >
> > > Finally, looking through the logs produced by Jetty doesn't seem to
> reveal
> > > any clues about what is wrong. There doesn't seem to be any errors in
> there,
> > > except the 404s.
> > >
> > > Long story short. I'm stuck. Any suggestions on where to go with this?
> > >
> > > David
> >
> >
> > --
> >
> > *Alexei Martchenko* | *CEO* | Superdownloads
> > ale...@superdownloads.com.br (mailto:ale...@superdownloads.com.br) |
> ale...@martchenko.com.br (mailto:ale...@martchenko.com.br) | (11)
> > 5083.1018/5080.3535/5080.3533
>
>


-- 

*Alexei Martchenko* | *CEO* | Superdownloads
ale...@superdownloads.com.br | ale...@martchenko.com.br | (11)
5083.1018/5080.3535/5080.3533

Re: Unable to get multicore working

2011-08-16 Thread David Sauve

Just the one `solr.xml`. The one I added (well, symlinked form my config folder 
-- I like to keep my configurations files organized so they can be managed by 
git)

`start.jar` is in `usr/share/jetty/start.jar`. 


On Tuesday, 16 August, 2011 at 1:33 PM, Alexei Martchenko wrote:

> AFAIK you're still seeing singlecore version
> 
> where is your start.jar?
> 
> search for solr.xml, see how many u've got plz.
> 
> 2011/8/16 David Sauve mailto:dnsa...@gmail.com)>
> 
> >  I've installed using aptitude so I don't have an example folder (that I
> > can find).
> > 
> > /solr/ does work (but lists no cores)
> > /solr/live/admin/ does not -- 404
> > 
> > 
> > On Tuesday, 16 August, 2011 at 1:13 PM, Alexei Martchenko wrote:
> > 
> > > Lets try something simplier.
> > > My start.jar is on \apache-solr-3.3.0\example\
> > > Here's my local config placed in \apache-solr-3.3.0\example\solr\
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > Create \apache-solr-3.3.0\example\solr\softwares01\conf\
> > > and \apache-solr-3.3.0\example\solr\softwares01\data\
> > > 
> > > http://localhost:8983/solr/ should work and so is
> > > http://localhost:8983/solr/softwares01/admin/
> > > 
> > > 
> > > 
> > > 2011/8/16 David Sauve mailto:dnsa...@gmail.com)>
> > > 
> > > > I've been trying (unsuccessfully) to get multicore working for about a
> > day
> > > > and a half now I'm nearly at wits end and unsure what to do anymore.
> > **Any**
> > > > help would be appreciated.
> > > > 
> > > > I've installed Solr using the solr-jetty packages on Ubuntu 10.04. The
> > > > default Solr install seems to work fine.
> > > > 
> > > > Now, I want to add three cores: live, staging, preview to be used for
> > the
> > > > various states of the site.
> > > > 
> > > > I've created a `solr.xml` file as follows and symlinked it in to
> > > > /usr/share/solr:
> > > > 
> > > > 
> > > > 
> > > > 
> > > >  > > > dataDir="/home/webteam/preview/data" />
> > > >  > > > dataDir="/home/webteam/staging/data" />
> > > >  > > > dataDir="/home/webteam/live/data" />
> > > > 
> > > > 
> > > > 
> > > > Now, when I try to view any cores, I get a 404 - Not found. In fact, I
> > > > can't even view /solr/admin/ anymore after installing that `solr.xml`
> > file.
> > > > 
> > > > Also, /solr/admin/cores returns an XML file, but it looks to me like
> > > > there's no cores listed. The output:
> > > > 
> > > > 
> > > > 
> > > > 0
> > > > 0
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > Finally, looking through the logs produced by Jetty doesn't seem to
> > reveal
> > > > any clues about what is wrong. There doesn't seem to be any errors in
> > there,
> > > > except the 404s.
> > > > 
> > > > Long story short. I'm stuck. Any suggestions on where to go with this?
> > > > 
> > > > David
> > > 
> > > 
> > > --
> > > 
> > > *Alexei Martchenko* | *CEO* | Superdownloads
> > > ale...@superdownloads.com.br (mailto:ale...@superdownloads.com.br) |
> > ale...@martchenko.com.br (mailto:ale...@martchenko.com.br) | (11)
> > > 5083.1018/5080.3535/5080.3533
> 
> 
> -- 
> 
> *Alexei Martchenko* | *CEO* | Superdownloads
> ale...@superdownloads.com.br (mailto:ale...@superdownloads.com.br) | 
> ale...@martchenko.com.br (mailto:ale...@martchenko.com.br) | (11)
> 5083.1018/5080.3535/5080.3533

Re: Unable to get multicore working

2011-08-16 Thread Alexei Martchenko

Is your solr.xml in usr/share/jetty/solr/solr.xml?

lets try this xml instead







  


Can you see the logs? You should see something like this

16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader 
*INFO: Solr home set to 'solr/'*
16/08/2011 17:30:55 org.apache.solr.servlet.SolrDispatchFilter init
INFO: SolrDispatchFilter.init()
16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader locateSolrHome
INFO: JNDI not configured for solr (NoInitialContextEx)
16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader locateSolrHome
*INFO: solr home defaulted to 'solr/' (could not find system property or
JNDI)*
16/08/2011 17:30:55 org.apache.solr.core.CoreContainer$Initializer
initialize
*INFO: looking for solr.xml: usr/share/jetty/solr/solr.xml*
16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader locateSolrHome
INFO: JNDI not configured for solr (NoInitialContextEx)
16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader locateSolrHome
*INFO: solr home defaulted to 'solr/' (could not find system property or
JNDI)*
16/08/2011 17:30:55 org.apache.solr.core.CoreContainer 
*INFO: New CoreContainer: solrHome=solr/ instance=21357269*
16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader 
*INFO: Solr home set to 'solr/'*
16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader 
*INFO: Solr home set to 'solr\core01\'*

2011/8/16 David Sauve 

> Just the one `solr.xml`. The one I added (well, symlinked form my config
> folder -- I like to keep my configurations files organized so they can be
> managed by git)
>
> `start.jar` is in `usr/share/jetty/start.jar`.
>
>
> On Tuesday, 16 August, 2011 at 1:33 PM, Alexei Martchenko wrote:
>
> > AFAIK you're still seeing singlecore version
> >
> > where is your start.jar?
> >
> > search for solr.xml, see how many u've got plz.
> >
> > 2011/8/16 David Sauve mailto:dnsa...@gmail.com)>
> >
> > >  I've installed using aptitude so I don't have an example folder (that
> I
> > > can find).
> > >
> > > /solr/ does work (but lists no cores)
> > > /solr/live/admin/ does not -- 404
> > >
> > >
> > > On Tuesday, 16 August, 2011 at 1:13 PM, Alexei Martchenko wrote:
> > >
> > > > Lets try something simplier.
> > > > My start.jar is on \apache-solr-3.3.0\example\
> > > > Here's my local config placed in \apache-solr-3.3.0\example\solr\
> > > >
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > >
> > > > Create \apache-solr-3.3.0\example\solr\softwares01\conf\
> > > > and \apache-solr-3.3.0\example\solr\softwares01\data\
> > > >
> > > > http://localhost:8983/solr/ should work and so is
> > > > http://localhost:8983/solr/softwares01/admin/
> > > >
> > > >
> > > >
> > > > 2011/8/16 David Sauve mailto:dnsa...@gmail.com)>
> > > >
> > > > > I've been trying (unsuccessfully) to get multicore working for
> about a
> > > day
> > > > > and a half now I'm nearly at wits end and unsure what to do
> anymore.
> > > **Any**
> > > > > help would be appreciated.
> > > > >
> > > > > I've installed Solr using the solr-jetty packages on Ubuntu 10.04.
> The
> > > > > default Solr install seems to work fine.
> > > > >
> > > > > Now, I want to add three cores: live, staging, preview to be used
> for
> > > the
> > > > > various states of the site.
> > > > >
> > > > > I've created a `solr.xml` file as follows and symlinked it in to
> > > > > /usr/share/solr:
> > > > >
> > > > > 
> > > > > 
> > > > > 
> > > > >  instanceDir="/home/webteam/config/search/preview"
> > > > > dataDir="/home/webteam/preview/data" />
> > > > >  instanceDir="/home/webteam/config/search/staging"
> > > > > dataDir="/home/webteam/staging/data" />
> > > > >  > > > > dataDir="/home/webteam/live/data" />
> > > > > 
> > > > > 
> > > > >
> > > > > Now, when I try to view any cores, I get a 404 - Not found. In
> fact, I
> > > > > can't even view /solr/admin/ anymore after installing that
> `solr.xml`
> > > file.
> > > > >
> > > > > Also, /solr/admin/cores returns an XML file, but it looks to me
> like
> > > > > there's no cores listed. The output:
> > > > >
> > > > > 
> > > > > 
> > > > > 0
> > > > > 0
> > > > >
> > > > > 
> > > > >
> > > > >
> > > > > 
> > > > >
> > > > > 
> > > > >
> > > > >
> > > > > Finally, looking through the logs produced by Jetty doesn't seem to
> > > reveal
> > > > > any clues about what is wrong. There doesn't seem to be any errors
> in
> > > there,
> > > > > except the 404s.
> > > > >
> > > > > Long story short. I'm stuck. Any suggestions on where to go with
> this?
> > > > >
> > > > > David
> > > >
> > > >
> > > > --
> > > >
> > > > *Alexei Martchenko* | *CEO* | Superdownloads
> > > > ale...@superdownloads.com.br (mailto:ale...@superdownloads.com.br) |
> > > ale...@martchenko.com.br (mailto:ale...@martchenko.com.br) | (11)
> > > > 5083.1018/5080.3535/5080.3533
> >
> >
> > --
> >
> > *Alexei Martchenko* | *CEO* | Superdownloads
> > ale...@superdownloads.com.br (mailto:ale...@superdownloads.com.br) |
> ale...@martchenko.com.br (mailto:ale...@martchenko.com.br) | (11)
> > 5083.101

RE: Unable to get multicore working

2011-08-16 Thread Jaeger, Jay - DOT

That won't work -- it would have to identify one of the three cores in your 
cores list (say, "live").

-Original Message-
From: David Sauve [mailto:dnsa...@gmail.com] 
Sent: Tuesday, August 16, 2011 3:29 PM
To: solr-user@lucene.apache.org
Subject: Re: Unable to get multicore working

I tried setting `defaultCoreName="admin"` and that didn't seem to change 
anything.

I also tried adding an `env-entry` for "solr/home" pointing to 
"/home/webteam/config" but that didn't seem to help either.

The logs don't have any errors in them, besides 404 errors.  


On Tuesday, 16 August, 2011 at 1:10 PM, Jaeger, Jay - DOT wrote:

> Perhaps your admin doesn’t work because you don't have 
> defaultCoreName="whatever-core-you-want-by-default" in your  tag? E.g.:
>  
> 
>  
> Perhaps this was enough to prevent it starting any cores -- I'd expect a 
> default to be required.
>  
> Also, from experience, if you add cores, and you have security turned on, you 
> probably need to modify web.xml (http://web.xml) to teach it about them, so 
> it maps the URLs to the right place with the right security.
>  
> Finally, check your logs to make sure that Solr isn't complaining about 
> something else (like not having a defaultCoreName, for instance)
>  
>  
> -Original Message-
> From: David Sauve [mailto:dnsa...@gmail.com]  
> Sent: Tuesday, August 16, 2011 3:02 PM
> To: solr-user@lucene.apache.org (mailto:solr-user@lucene.apache.org)
> Subject: Unable to get multicore working
>  
> I've been trying (unsuccessfully) to get multicore working for about a day 
> and a half now I'm nearly at wits end and unsure what to do anymore. **Any** 
> help would be appreciated.
>  
> I've installed Solr using the solr-jetty packages on Ubuntu 10.04. The 
> default Solr install seems to work fine.
>  
> Now, I want to add three cores: live, staging, preview to be used for the 
> various states of the site.
>  
> I've created a `solr.xml` file as follows and symlinked it in to 
> /usr/share/solr:  
>  
> 
> 
> 
>  dataDir="/home/webteam/preview/data" />
>  dataDir="/home/webteam/staging/data" />
>  dataDir="/home/webteam/live/data" />
> 
> 
>  
> Now, when I try to view any cores, I get a 404 - Not found. In fact, I can't 
> even view /solr/admin/ anymore after installing that `solr.xml` file.
>  
> Also, /solr/admin/cores returns an XML file, but it looks to me like there's 
> no cores listed. The output:
>  
> 
> 
> 0
> 0
>  
> 
>  
>  
> 
>  
> 
>  
>  
> Finally, looking through the logs produced by Jetty doesn't seem to reveal 
> any clues about what is wrong. There doesn't seem to be any errors in there, 
> except the 404s.
>  
> Long story short. I'm stuck. Any suggestions on where to go with this?
>  
> David

Re: Unable to get multicore working

2011-08-16 Thread David Sauve

Nope. Only thing in the log:

1 [main] INFO org.mortbay.log - Logging to 
org.slf4j.impl.SimpleLogger(org.mortbay.log) via org.mortbay.log.Slf4jLog
173 [main] INFO org.mortbay.log - Redirecting stderr/stdout to 
/var/log/jetty/2011_08_16.stderrout.log




On Tuesday, 16 August, 2011 at 1:45 PM, Alexei Martchenko wrote:

> Is your solr.xml in usr/share/jetty/solr/solr.xml?
> 
> lets try this xml instead
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Can you see the logs? You should see something like this
> 
> 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader 
> *INFO: Solr home set to 'solr/'*
> 16/08/2011 17:30:55 org.apache.solr.servlet.SolrDispatchFilter init
> INFO: SolrDispatchFilter.init()
> 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader locateSolrHome
> INFO: JNDI not configured for solr (NoInitialContextEx)
> 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader locateSolrHome
> *INFO: solr home defaulted to 'solr/' (could not find system property or
> JNDI)*
> 16/08/2011 17:30:55 org.apache.solr.core.CoreContainer$Initializer
> initialize
> *INFO: looking for solr.xml: usr/share/jetty/solr/solr.xml*
> 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader locateSolrHome
> INFO: JNDI not configured for solr (NoInitialContextEx)
> 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader locateSolrHome
> *INFO: solr home defaulted to 'solr/' (could not find system property or
> JNDI)*
> 16/08/2011 17:30:55 org.apache.solr.core.CoreContainer 
> *INFO: New CoreContainer: solrHome=solr/ instance=21357269*
> 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader 
> *INFO: Solr home set to 'solr/'*
> 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader 
> *INFO: Solr home set to 'solr\core01\'*
> 
> 2011/8/16 David Sauve mailto:dnsa...@gmail.com)>
> 
> > Just the one `solr.xml`. The one I added (well, symlinked form my config
> > folder -- I like to keep my configurations files organized so they can be
> > managed by git)
> > 
> > `start.jar` is in `usr/share/jetty/start.jar`.
> > 
> > 
> > On Tuesday, 16 August, 2011 at 1:33 PM, Alexei Martchenko wrote:
> > 
> > > AFAIK you're still seeing singlecore version
> > > 
> > > where is your start.jar?
> > > 
> > > search for solr.xml, see how many u've got plz.
> > > 
> > > 2011/8/16 David Sauve mailto:dnsa...@gmail.com)>
> > > 
> > > >  I've installed using aptitude so I don't have an example folder (that
> > I
> > > > can find).
> > > > 
> > > > /solr/ does work (but lists no cores)
> > > > /solr/live/admin/ does not -- 404
> > > > 
> > > > 
> > > > On Tuesday, 16 August, 2011 at 1:13 PM, Alexei Martchenko wrote:
> > > > 
> > > > > Lets try something simplier.
> > > > > My start.jar is on \apache-solr-3.3.0\example\
> > > > > Here's my local config placed in \apache-solr-3.3.0\example\solr\
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > Create \apache-solr-3.3.0\example\solr\softwares01\conf\
> > > > > and \apache-solr-3.3.0\example\solr\softwares01\data\
> > > > > 
> > > > > http://localhost:8983/solr/ should work and so is
> > > > > http://localhost:8983/solr/softwares01/admin/
> > > > > 
> > > > > 
> > > > > 
> > > > > 2011/8/16 David Sauve mailto:dnsa...@gmail.com)>
> > > > > 
> > > > > > I've been trying (unsuccessfully) to get multicore working for
> > about a
> > > > day
> > > > > > and a half now I'm nearly at wits end and unsure what to do
> > anymore.
> > > > **Any**
> > > > > > help would be appreciated.
> > > > > > 
> > > > > > I've installed Solr using the solr-jetty packages on Ubuntu 10.04.
> > The
> > > > > > default Solr install seems to work fine.
> > > > > > 
> > > > > > Now, I want to add three cores: live, staging, preview to be used
> > for
> > > > the
> > > > > > various states of the site.
> > > > > > 
> > > > > > I've created a `solr.xml` file as follows and symlinked it in to
> > > > > > /usr/share/solr:
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > >  > instanceDir="/home/webteam/config/search/preview"
> > > > > > dataDir="/home/webteam/preview/data" />
> > > > > >  > instanceDir="/home/webteam/config/search/staging"
> > > > > > dataDir="/home/webteam/staging/data" />
> > > > > >  > > > > > dataDir="/home/webteam/live/data" />
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > Now, when I try to view any cores, I get a 404 - Not found. In
> > fact, I
> > > > > > can't even view /solr/admin/ anymore after installing that
> > `solr.xml`
> > > > file.
> > > > > > 
> > > > > > Also, /solr/admin/cores returns an XML file, but it looks to me
> > like
> > > > > > there's no cores listed. The output:
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 0
> > > > > > 0
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > Finally, looking through the logs produced by Jetty doesn't seem to
> > > > reveal
> > > > > > any clues about what is wrong. There doesn't seem t

Re: Faceted Search Patent Lawsuit - Please Read

2011-08-16 Thread Paul Libbrecht

While I agree with Grant we shouldn't engage on a legal discussion, it may be 
worth that this thread shares a few dates of when faceted search was used "in 
the old times"...

paul


Le 16 août 2011 à 22:02, LaMaze Johnson a écrit :

> 
> Grant Ingersoll-2 wrote:
>> 
>> I know you mean well and are probably wondering what to do next, but such
>> a discussion is really beyond the scope of this mailing list.  Most of us
>> aren't lawyers (I wonder if anyone here is?) and if we were, we wouldn't
>> likely speculate in public on something that can only be decided in the
>> courts.
>> 
>> -Grant
>> 
> 
> We will definitely be in touch with our legal counsel.  This was more a
> "heads up" in hopes that others aren't blindsided by this.  
> 
> 
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/Faceted-Search-Patent-Lawsuit-Please-Read-tp3259475p3259733.html
> Sent from the Solr - User mailing list archive at Nabble.com.

RE: Unable to get multicore working

2011-08-16 Thread Jaeger, Jay - DOT

I tried on my own test environment -- pulling out the default core parameter 
out, under Solr 3.1  

I got exactly your symptom: an error 404. 

HTTP ERROR 404
Problem accessing /solr/admin/index.jsp. Reason: 

missing core name in path

The log showed:

2011-08-16 16:00:12.469:WARN::/solr/admin/
java.lang.IllegalStateException: STREAM
at org.mortbay.jetty.Response.getWriter(Response.java:616)
at 
org.apache.jasper.runtime.JspWriterImpl.initOut(JspWriterImpl.java:187)
at 
org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:180)
at 
org.apache.jasper.runtime.PageContextImpl.release(PageContextImpl.java:237)
at 
org.apache.jasper.runtime.JspFactoryImpl.internalReleasePageContext(JspFactoryImpl.java:173)
at 
org.apache.jasper.runtime.JspFactoryImpl.releasePageContext(JspFactoryImpl.java:124)

(etc.)

Adding the defaultCoreName fixed it.

I expect this is indeed your problem.

-Original Message-
From: David Sauve [mailto:dnsa...@gmail.com] 
Sent: Tuesday, August 16, 2011 3:50 PM
To: solr-user@lucene.apache.org
Subject: Re: Unable to get multicore working

Nope. Only thing in the log:

1 [main] INFO org.mortbay.log - Logging to 
org.slf4j.impl.SimpleLogger(org.mortbay.log) via org.mortbay.log.Slf4jLog
173 [main] INFO org.mortbay.log - Redirecting stderr/stdout to 
/var/log/jetty/2011_08_16.stderrout.log




On Tuesday, 16 August, 2011 at 1:45 PM, Alexei Martchenko wrote:

> Is your solr.xml in usr/share/jetty/solr/solr.xml?
> 
> lets try this xml instead
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Can you see the logs? You should see something like this
> 
> 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader 
> *INFO: Solr home set to 'solr/'*
> 16/08/2011 17:30:55 org.apache.solr.servlet.SolrDispatchFilter init
> INFO: SolrDispatchFilter.init()
> 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader locateSolrHome
> INFO: JNDI not configured for solr (NoInitialContextEx)
> 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader locateSolrHome
> *INFO: solr home defaulted to 'solr/' (could not find system property or
> JNDI)*
> 16/08/2011 17:30:55 org.apache.solr.core.CoreContainer$Initializer
> initialize
> *INFO: looking for solr.xml: usr/share/jetty/solr/solr.xml*
> 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader locateSolrHome
> INFO: JNDI not configured for solr (NoInitialContextEx)
> 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader locateSolrHome
> *INFO: solr home defaulted to 'solr/' (could not find system property or
> JNDI)*
> 16/08/2011 17:30:55 org.apache.solr.core.CoreContainer 
> *INFO: New CoreContainer: solrHome=solr/ instance=21357269*
> 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader 
> *INFO: Solr home set to 'solr/'*
> 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader 
> *INFO: Solr home set to 'solr\core01\'*
> 
> 2011/8/16 David Sauve mailto:dnsa...@gmail.com)>
> 
> > Just the one `solr.xml`. The one I added (well, symlinked form my config
> > folder -- I like to keep my configurations files organized so they can be
> > managed by git)
> > 
> > `start.jar` is in `usr/share/jetty/start.jar`.
> > 
> > 
> > On Tuesday, 16 August, 2011 at 1:33 PM, Alexei Martchenko wrote:
> > 
> > > AFAIK you're still seeing singlecore version
> > > 
> > > where is your start.jar?
> > > 
> > > search for solr.xml, see how many u've got plz.
> > > 
> > > 2011/8/16 David Sauve mailto:dnsa...@gmail.com)>
> > > 
> > > >  I've installed using aptitude so I don't have an example folder (that
> > I
> > > > can find).
> > > > 
> > > > /solr/ does work (but lists no cores)
> > > > /solr/live/admin/ does not -- 404
> > > > 
> > > > 
> > > > On Tuesday, 16 August, 2011 at 1:13 PM, Alexei Martchenko wrote:
> > > > 
> > > > > Lets try something simplier.
> > > > > My start.jar is on \apache-solr-3.3.0\example\
> > > > > Here's my local config placed in \apache-solr-3.3.0\example\solr\
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > Create \apache-solr-3.3.0\example\solr\softwares01\conf\
> > > > > and \apache-solr-3.3.0\example\solr\softwares01\data\
> > > > > 
> > > > > http://localhost:8983/solr/ should work and so is
> > > > > http://localhost:8983/solr/softwares01/admin/
> > > > > 
> > > > > 
> > > > > 
> > > > > 2011/8/16 David Sauve mailto:dnsa...@gmail.com)>
> > > > > 
> > > > > > I've been trying (unsuccessfully) to get multicore working for
> > about a
> > > > day
> > > > > > and a half now I'm nearly at wits end and unsure what to do
> > anymore.
> > > > **Any**
> > > > > > help would be appreciated.
> > > > > > 
> > > > > > I've installed Solr using the solr-jetty packages on Ubuntu 10.04.
> > The
> > > > > > default Solr install seems to work fine.
> > > > > > 
> > > > > > Now, I want to add three cores: live, staging, preview to be used
> > for
> > > > the
> > > > > > various states of the site.
>

RE: Unable to get multicore working

2011-08-16 Thread Jaeger, Jay - DOT

Whoops:  That was Solr 4.0 (which pre-dates 3.1).

I doubt very much that the release matters, though: I expect the behavior would 
be the same.

-Original Message-
From: Jaeger, Jay - DOT [mailto:jay.jae...@dot.wi.gov] 
Sent: Tuesday, August 16, 2011 4:04 PM
To: solr-user@lucene.apache.org
Subject: RE: Unable to get multicore working

I tried on my own test environment -- pulling out the default core parameter 
out, under Solr 3.1  

I got exactly your symptom: an error 404. 

HTTP ERROR 404
Problem accessing /solr/admin/index.jsp. Reason: 

missing core name in path

The log showed:

2011-08-16 16:00:12.469:WARN::/solr/admin/
java.lang.IllegalStateException: STREAM
at org.mortbay.jetty.Response.getWriter(Response.java:616)
at 
org.apache.jasper.runtime.JspWriterImpl.initOut(JspWriterImpl.java:187)
at 
org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:180)
at 
org.apache.jasper.runtime.PageContextImpl.release(PageContextImpl.java:237)
at 
org.apache.jasper.runtime.JspFactoryImpl.internalReleasePageContext(JspFactoryImpl.java:173)
at 
org.apache.jasper.runtime.JspFactoryImpl.releasePageContext(JspFactoryImpl.java:124)

(etc.)

Adding the defaultCoreName fixed it.

I expect this is indeed your problem.

-Original Message-
From: David Sauve [mailto:dnsa...@gmail.com] 
Sent: Tuesday, August 16, 2011 3:50 PM
To: solr-user@lucene.apache.org
Subject: Re: Unable to get multicore working

Nope. Only thing in the log:

1 [main] INFO org.mortbay.log - Logging to 
org.slf4j.impl.SimpleLogger(org.mortbay.log) via org.mortbay.log.Slf4jLog
173 [main] INFO org.mortbay.log - Redirecting stderr/stdout to 
/var/log/jetty/2011_08_16.stderrout.log




On Tuesday, 16 August, 2011 at 1:45 PM, Alexei Martchenko wrote:

> Is your solr.xml in usr/share/jetty/solr/solr.xml?
> 
> lets try this xml instead
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Can you see the logs? You should see something like this
> 
> 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader 
> *INFO: Solr home set to 'solr/'*
> 16/08/2011 17:30:55 org.apache.solr.servlet.SolrDispatchFilter init
> INFO: SolrDispatchFilter.init()
> 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader locateSolrHome
> INFO: JNDI not configured for solr (NoInitialContextEx)
> 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader locateSolrHome
> *INFO: solr home defaulted to 'solr/' (could not find system property or
> JNDI)*
> 16/08/2011 17:30:55 org.apache.solr.core.CoreContainer$Initializer
> initialize
> *INFO: looking for solr.xml: usr/share/jetty/solr/solr.xml*
> 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader locateSolrHome
> INFO: JNDI not configured for solr (NoInitialContextEx)
> 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader locateSolrHome
> *INFO: solr home defaulted to 'solr/' (could not find system property or
> JNDI)*
> 16/08/2011 17:30:55 org.apache.solr.core.CoreContainer 
> *INFO: New CoreContainer: solrHome=solr/ instance=21357269*
> 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader 
> *INFO: Solr home set to 'solr/'*
> 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader 
> *INFO: Solr home set to 'solr\core01\'*
> 
> 2011/8/16 David Sauve mailto:dnsa...@gmail.com)>
> 
> > Just the one `solr.xml`. The one I added (well, symlinked form my config
> > folder -- I like to keep my configurations files organized so they can be
> > managed by git)
> > 
> > `start.jar` is in `usr/share/jetty/start.jar`.
> > 
> > 
> > On Tuesday, 16 August, 2011 at 1:33 PM, Alexei Martchenko wrote:
> > 
> > > AFAIK you're still seeing singlecore version
> > > 
> > > where is your start.jar?
> > > 
> > > search for solr.xml, see how many u've got plz.
> > > 
> > > 2011/8/16 David Sauve mailto:dnsa...@gmail.com)>
> > > 
> > > >  I've installed using aptitude so I don't have an example folder (that
> > I
> > > > can find).
> > > > 
> > > > /solr/ does work (but lists no cores)
> > > > /solr/live/admin/ does not -- 404
> > > > 
> > > > 
> > > > On Tuesday, 16 August, 2011 at 1:13 PM, Alexei Martchenko wrote:
> > > > 
> > > > > Lets try something simplier.
> > > > > My start.jar is on \apache-solr-3.3.0\example\
> > > > > Here's my local config placed in \apache-solr-3.3.0\example\solr\
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > Create \apache-solr-3.3.0\example\solr\softwares01\conf\
> > > > > and \apache-solr-3.3.0\example\solr\softwares01\data\
> > > > > 
> > > > > http://localhost:8983/solr/ should work and so is
> > > > > http://localhost:8983/solr/softwares01/admin/
> > > > > 
> > > > > 
> > > > > 
> > > > > 2011/8/16 David Sauve mailto:dnsa...@gmail.com)>
> > > > > 
> > > > > > I've been trying (unsuccessfully) to get multicore working for
> > about a
> > > > day
> > > > > > and a half now I'm nearly at wits end and unsure what to do
> > anymore.
> > > > **Any*

Re: How to deal with "java.net.SocketTimeoutException: Read timed out" on commit?

2011-08-16 Thread lboutros

We had this type of error too.
Now we are using the StreamingUpdateSolrServer with a quite big queue and
2-4 threads depending on data type:

http://lucene.apache.org/solr/api/org/apache/solr/client/solrj/impl/StreamingUpdateSolrServer.html

And we do not do any intermediate commit. We send only one commit at the end
of  the process and now everything is fine.

Ludovic.

-
Jouve
France.
--
View this message in context: 
http://lucene.472066.n3.nabble.com/How-to-deal-with-java-net-SocketTimeoutException-Read-timed-out-on-commit-tp3258430p3259890.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Unable to get multicore working

2011-08-16 Thread David Sauve

I updated my `solr.xml` as follow:











and I'm still seeing the same 404 when I true to view /solr/admin/ or 
/solr/live/admin/

That said, the logs are showing a different error now. Excellent! The site 
schemas are loading!

Looks like the site schemas have an issue:

"SEVERE: org.apache.solr.common.SolrException: Unknown fieldtype 'long' 
specified on field area_id"

Errr. Why would `long` be an invalid type? 


On Tuesday, 16 August, 2011 at 2:06 PM, Jaeger, Jay - DOT wrote:

> Whoops: That was Solr 4.0 (which pre-dates 3.1).
> 
> I doubt very much that the release matters, though: I expect the behavior 
> would be the same.
> 
> -Original Message-
> From: Jaeger, Jay - DOT [mailto:jay.jae...@dot.wi.gov] 
> Sent: Tuesday, August 16, 2011 4:04 PM
> To: solr-user@lucene.apache.org (mailto:solr-user@lucene.apache.org)
> Subject: RE: Unable to get multicore working
> 
> I tried on my own test environment -- pulling out the default core parameter 
> out, under Solr 3.1 
> 
> I got exactly your symptom: an error 404. 
> 
>  HTTP ERROR 404
>  Problem accessing /solr/admin/index.jsp. Reason: 
> 
>  missing core name in path
> 
> The log showed:
> 
> 2011-08-16 16:00:12.469:WARN::/solr/admin/
> java.lang.IllegalStateException: STREAM
>  at org.mortbay.jetty.Response.getWriter(Response.java:616)
>  at org.apache.jasper.runtime.JspWriterImpl.initOut(JspWriterImpl.java:187)
>  at 
> org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:180)
>  at 
> org.apache.jasper.runtime.PageContextImpl.release(PageContextImpl.java:237)
>  at 
> org.apache.jasper.runtime.JspFactoryImpl.internalReleasePageContext(JspFactoryImpl.java:173)
>  at 
> org.apache.jasper.runtime.JspFactoryImpl.releasePageContext(JspFactoryImpl.java:124)
> 
> (etc.)
> 
> Adding the defaultCoreName fixed it.
> 
> I expect this is indeed your problem.
> 
> -Original Message-
> From: David Sauve [mailto:dnsa...@gmail.com] 
> Sent: Tuesday, August 16, 2011 3:50 PM
> To: solr-user@lucene.apache.org (mailto:solr-user@lucene.apache.org)
> Subject: Re: Unable to get multicore working
> 
> Nope. Only thing in the log:
> 
> 1 [main] INFO org.mortbay.log - Logging to 
> org.slf4j.impl.SimpleLogger(org.mortbay.log) via org.mortbay.log.Slf4jLog
> 173 [main] INFO org.mortbay.log - Redirecting stderr/stdout to 
> /var/log/jetty/2011_08_16.stderrout.log
> 
> 
> 
> 
> On Tuesday, 16 August, 2011 at 1:45 PM, Alexei Martchenko wrote:
> 
> > Is your solr.xml in usr/share/jetty/solr/solr.xml?
> > 
> > lets try this xml instead
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > Can you see the logs? You should see something like this
> > 
> > 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader 
> > *INFO: Solr home set to 'solr/'*
> > 16/08/2011 17:30:55 org.apache.solr.servlet.SolrDispatchFilter init
> > INFO: SolrDispatchFilter.init()
> > 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader locateSolrHome
> > INFO: JNDI not configured for solr (NoInitialContextEx)
> > 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader locateSolrHome
> > *INFO: solr home defaulted to 'solr/' (could not find system property or
> > JNDI)*
> > 16/08/2011 17:30:55 org.apache.solr.core.CoreContainer$Initializer
> > initialize
> > *INFO: looking for solr.xml: usr/share/jetty/solr/solr.xml*
> > 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader locateSolrHome
> > INFO: JNDI not configured for solr (NoInitialContextEx)
> > 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader locateSolrHome
> > *INFO: solr home defaulted to 'solr/' (could not find system property or
> > JNDI)*
> > 16/08/2011 17:30:55 org.apache.solr.core.CoreContainer 
> > *INFO: New CoreContainer: solrHome=solr/ instance=21357269*
> > 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader 
> > *INFO: Solr home set to 'solr/'*
> > 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader 
> > *INFO: Solr home set to 'solr\core01\'*
> > 
> > 2011/8/16 David Sauve mailto:dnsa...@gmail.com)>
> > 
> > > Just the one `solr.xml`. The one I added (well, symlinked form my config
> > > folder -- I like to keep my configurations files organized so they can be
> > > managed by git)
> > > 
> > > `start.jar` is in `usr/share/jetty/start.jar`.
> > > 
> > > 
> > > On Tuesday, 16 August, 2011 at 1:33 PM, Alexei Martchenko wrote:
> > > 
> > > > AFAIK you're still seeing singlecore version
> > > > 
> > > > where is your start.jar?
> > > > 
> > > > search for solr.xml, see how many u've got plz.
> > > > 
> > > > 2011/8/16 David Sauve mailto:dnsa...@gmail.com)>
> > > > 
> > > > >  I've installed using aptitude so I don't have an example folder (that
> > > I
> > > > > can find).
> > > > > 
> > > > > /solr/ does work (but lists no cores)
> > > > > /solr/live/admin/ does not -- 404
> > > > > 
> > > > > 
> > > > > On Tuesday, 16 August, 2011 at 1:13 PM, Alexei Martchenko wrote:
> > > > > 
> > > > > > Lets try something simplier.
> > > > > > My start.jar is on \ap

Re: Unable to get multicore working

2011-08-16 Thread Donald Organ

When you go to /solr  what do you see?

On Tue, Aug 16, 2011 at 5:23 PM, David Sauve  wrote:

> I updated my `solr.xml` as follow:
>
> 
> 
> 
>  dataDir="/home/webteam/preview/data" />
>  dataDir="/home/webteam/staging/data" />
>  dataDir="/home/webteam/live/data" />
> 
> 
>
>
> and I'm still seeing the same 404 when I true to view /solr/admin/ or
> /solr/live/admin/
>
> That said, the logs are showing a different error now. Excellent! The site
> schemas are loading!
>
> Looks like the site schemas have an issue:
>
> "SEVERE: org.apache.solr.common.SolrException: Unknown fieldtype 'long'
> specified on field area_id"
>
> Errr. Why would `long` be an invalid type?
>
>
> On Tuesday, 16 August, 2011 at 2:06 PM, Jaeger, Jay - DOT wrote:
>
> > Whoops: That was Solr 4.0 (which pre-dates 3.1).
> >
> > I doubt very much that the release matters, though: I expect the behavior
> would be the same.
> >
> > -Original Message-
> > From: Jaeger, Jay - DOT [mailto:jay.jae...@dot.wi.gov]
> > Sent: Tuesday, August 16, 2011 4:04 PM
> > To: solr-user@lucene.apache.org (mailto:solr-user@lucene.apache.org)
> > Subject: RE: Unable to get multicore working
> >
> > I tried on my own test environment -- pulling out the default core
> parameter out, under Solr 3.1
> >
> > I got exactly your symptom: an error 404.
> >
> >  HTTP ERROR 404
> >  Problem accessing /solr/admin/index.jsp. Reason:
> >
> >  missing core name in path
> >
> > The log showed:
> >
> > 2011-08-16 16:00:12.469:WARN::/solr/admin/
> > java.lang.IllegalStateException: STREAM
> >  at org.mortbay.jetty.Response.getWriter(Response.java:616)
> >  at
> org.apache.jasper.runtime.JspWriterImpl.initOut(JspWriterImpl.java:187)
> >  at
> org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:180)
> >  at
> org.apache.jasper.runtime.PageContextImpl.release(PageContextImpl.java:237)
> >  at
> org.apache.jasper.runtime.JspFactoryImpl.internalReleasePageContext(JspFactoryImpl.java:173)
> >  at
> org.apache.jasper.runtime.JspFactoryImpl.releasePageContext(JspFactoryImpl.java:124)
> >
> > (etc.)
> >
> > Adding the defaultCoreName fixed it.
> >
> > I expect this is indeed your problem.
> >
> > -Original Message-
> > From: David Sauve [mailto:dnsa...@gmail.com]
> > Sent: Tuesday, August 16, 2011 3:50 PM
> > To: solr-user@lucene.apache.org (mailto:solr-user@lucene.apache.org)
> > Subject: Re: Unable to get multicore working
> >
> > Nope. Only thing in the log:
> >
> > 1 [main] INFO org.mortbay.log - Logging to
> org.slf4j.impl.SimpleLogger(org.mortbay.log) via org.mortbay.log.Slf4jLog
> > 173 [main] INFO org.mortbay.log - Redirecting stderr/stdout to
> /var/log/jetty/2011_08_16.stderrout.log
> >
> >
> >
> >
> > On Tuesday, 16 August, 2011 at 1:45 PM, Alexei Martchenko wrote:
> >
> > > Is your solr.xml in usr/share/jetty/solr/solr.xml?
> > >
> > > lets try this xml instead
> > >
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > >
> > > Can you see the logs? You should see something like this
> > >
> > > 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader 
> > > *INFO: Solr home set to 'solr/'*
> > > 16/08/2011 17:30:55 org.apache.solr.servlet.SolrDispatchFilter init
> > > INFO: SolrDispatchFilter.init()
> > > 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader
> locateSolrHome
> > > INFO: JNDI not configured for solr (NoInitialContextEx)
> > > 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader
> locateSolrHome
> > > *INFO: solr home defaulted to 'solr/' (could not find system property
> or
> > > JNDI)*
> > > 16/08/2011 17:30:55 org.apache.solr.core.CoreContainer$Initializer
> > > initialize
> > > *INFO: looking for solr.xml: usr/share/jetty/solr/solr.xml*
> > > 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader
> locateSolrHome
> > > INFO: JNDI not configured for solr (NoInitialContextEx)
> > > 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader
> locateSolrHome
> > > *INFO: solr home defaulted to 'solr/' (could not find system property
> or
> > > JNDI)*
> > > 16/08/2011 17:30:55 org.apache.solr.core.CoreContainer 
> > > *INFO: New CoreContainer: solrHome=solr/ instance=21357269*
> > > 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader 
> > > *INFO: Solr home set to 'solr/'*
> > > 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader 
> > > *INFO: Solr home set to 'solr\core01\'*
> > >
> > > 2011/8/16 David Sauve mailto:dnsa...@gmail.com)>
> > >
> > > > Just the one `solr.xml`. The one I added (well, symlinked form my
> config
> > > > folder -- I like to keep my configurations files organized so they
> can be
> > > > managed by git)
> > > >
> > > > `start.jar` is in `usr/share/jetty/start.jar`.
> > > >
> > > >
> > > > On Tuesday, 16 August, 2011 at 1:33 PM, Alexei Martchenko wrote:
> > > >
> > > > > AFAIK you're still seeing singlecore version
> > > > >
> > > > > where is your start.jar?
> > > > >
> > > > > search for solr.xml, see how many u've got plz.
> > > > >
> > > > > 2011/8/16

Re: Unable to get multicore working

2011-08-16 Thread David Sauve

"Welcome to Solr" with a link to "Admin". The link returns a 404.

On Tuesday, 16 August, 2011 at 2:30 PM, Donald Organ wrote:

> When you go to /solr what do you see?
> 
> On Tue, Aug 16, 2011 at 5:23 PM, David Sauve  (mailto:dnsa...@gmail.com)> wrote:
> 
> > I updated my `solr.xml` as follow:
> > 
> > 
> > 
> > 
> >  > dataDir="/home/webteam/preview/data" />
> >  > dataDir="/home/webteam/staging/data" />
> >  > dataDir="/home/webteam/live/data" />
> > 
> > 
> > 
> > 
> > and I'm still seeing the same 404 when I true to view /solr/admin/ or
> > /solr/live/admin/
> > 
> > That said, the logs are showing a different error now. Excellent! The site
> > schemas are loading!
> > 
> > Looks like the site schemas have an issue:
> > 
> > "SEVERE: org.apache.solr.common.SolrException: Unknown fieldtype 'long'
> > specified on field area_id"
> > 
> > Errr. Why would `long` be an invalid type?
> > 
> > 
> > On Tuesday, 16 August, 2011 at 2:06 PM, Jaeger, Jay - DOT wrote:
> > 
> > > Whoops: That was Solr 4.0 (which pre-dates 3.1).
> > > 
> > > I doubt very much that the release matters, though: I expect the behavior
> > would be the same.
> > > 
> > > -Original Message-
> > > From: Jaeger, Jay - DOT [mailto:jay.jae...@dot.wi.gov]
> > > Sent: Tuesday, August 16, 2011 4:04 PM
> > > To: solr-user@lucene.apache.org (mailto:solr-user@lucene.apache.org)
> > > Subject: RE: Unable to get multicore working
> > > 
> > > I tried on my own test environment -- pulling out the default core
> > parameter out, under Solr 3.1
> > > 
> > > I got exactly your symptom: an error 404.
> > > 
> > >  HTTP ERROR 404
> > >  Problem accessing /solr/admin/index.jsp. Reason:
> > > 
> > >  missing core name in path
> > > 
> > > The log showed:
> > > 
> > > 2011-08-16 16:00:12.469:WARN::/solr/admin/
> > > java.lang.IllegalStateException: STREAM
> > >  at org.mortbay.jetty.Response.getWriter(Response.java:616)
> > >  at
> > org.apache.jasper.runtime.JspWriterImpl.initOut(JspWriterImpl.java:187)
> > >  at
> > org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:180)
> > >  at
> > org.apache.jasper.runtime.PageContextImpl.release(PageContextImpl.java:237)
> > >  at
> > org.apache.jasper.runtime.JspFactoryImpl.internalReleasePageContext(JspFactoryImpl.java:173)
> > >  at
> > org.apache.jasper.runtime.JspFactoryImpl.releasePageContext(JspFactoryImpl.java:124)
> > > 
> > > (etc.)
> > > 
> > > Adding the defaultCoreName fixed it.
> > > 
> > > I expect this is indeed your problem.
> > > 
> > > -Original Message-
> > > From: David Sauve [mailto:dnsa...@gmail.com]
> > > Sent: Tuesday, August 16, 2011 3:50 PM
> > > To: solr-user@lucene.apache.org (mailto:solr-user@lucene.apache.org)
> > > Subject: Re: Unable to get multicore working
> > > 
> > > Nope. Only thing in the log:
> > > 
> > > 1 [main] INFO org.mortbay.log - Logging to
> > org.slf4j.impl.SimpleLogger(org.mortbay.log) via org.mortbay.log.Slf4jLog
> > > 173 [main] INFO org.mortbay.log - Redirecting stderr/stdout to
> > /var/log/jetty/2011_08_16.stderrout.log
> > > 
> > > 
> > > 
> > > 
> > > On Tuesday, 16 August, 2011 at 1:45 PM, Alexei Martchenko wrote:
> > > 
> > > > Is your solr.xml in usr/share/jetty/solr/solr.xml?
> > > > 
> > > > lets try this xml instead
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > Can you see the logs? You should see something like this
> > > > 
> > > > 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader 
> > > > *INFO: Solr home set to 'solr/'*
> > > > 16/08/2011 17:30:55 org.apache.solr.servlet.SolrDispatchFilter init
> > > > INFO: SolrDispatchFilter.init()
> > > > 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader
> > locateSolrHome
> > > > INFO: JNDI not configured for solr (NoInitialContextEx)
> > > > 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader
> > locateSolrHome
> > > > *INFO: solr home defaulted to 'solr/' (could not find system property
> > or
> > > > JNDI)*
> > > > 16/08/2011 17:30:55 org.apache.solr.core.CoreContainer$Initializer
> > > > initialize
> > > > *INFO: looking for solr.xml: usr/share/jetty/solr/solr.xml*
> > > > 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader
> > locateSolrHome
> > > > INFO: JNDI not configured for solr (NoInitialContextEx)
> > > > 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader
> > locateSolrHome
> > > > *INFO: solr home defaulted to 'solr/' (could not find system property
> > or
> > > > JNDI)*
> > > > 16/08/2011 17:30:55 org.apache.solr.core.CoreContainer 
> > > > *INFO: New CoreContainer: solrHome=solr/ instance=21357269*
> > > > 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader 
> > > > *INFO: Solr home set to 'solr/'*
> > > > 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader 
> > > > *INFO: Solr home set to 'solr\core01\'*
> > > > 
> > > > 2011/8/16 David Sauve mailto:dnsa...@gmail.com)>
> > > > 
> > > > > Just the one `solr.xml`. The one I added (well, symlinked form my
>

RE: Unable to get multicore working

2011-08-16 Thread Jaeger, Jay - DOT

That said, the logs are showing a different error now. Excellent! The 
site schemas are loading!

Great!

"SEVERE: org.apache.solr.common.SolrException: Unknown fieldtype 'long' 
specified on field area_id"

Go have a look at your conf/schema.xml.  

Is the following line present??  Does your field definition for area_id follow 
it?



Look at the file with an XML editor.  Perhaps an edit to some earlier portion 
of the schema is messing up this part of the schema?


-Original Message-
From: David Sauve [mailto:dnsa...@gmail.com] 
Sent: Tuesday, August 16, 2011 4:24 PM
To: solr-user@lucene.apache.org
Subject: Re: Unable to get multicore working

I updated my `solr.xml` as follow:











and I'm still seeing the same 404 when I true to view /solr/admin/ or 
/solr/live/admin/

That said, the logs are showing a different error now. Excellent! The site 
schemas are loading!

Looks like the site schemas have an issue:

"SEVERE: org.apache.solr.common.SolrException: Unknown fieldtype 'long' 
specified on field area_id"

Errr. Why would `long` be an invalid type? 


On Tuesday, 16 August, 2011 at 2:06 PM, Jaeger, Jay - DOT wrote:

> Whoops: That was Solr 4.0 (which pre-dates 3.1).
> 
> I doubt very much that the release matters, though: I expect the behavior 
> would be the same.
> 
> -Original Message-
> From: Jaeger, Jay - DOT [mailto:jay.jae...@dot.wi.gov] 
> Sent: Tuesday, August 16, 2011 4:04 PM
> To: solr-user@lucene.apache.org (mailto:solr-user@lucene.apache.org)
> Subject: RE: Unable to get multicore working
> 
> I tried on my own test environment -- pulling out the default core parameter 
> out, under Solr 3.1 
> 
> I got exactly your symptom: an error 404. 
> 
>  HTTP ERROR 404
>  Problem accessing /solr/admin/index.jsp. Reason: 
> 
>  missing core name in path
> 
> The log showed:
> 
> 2011-08-16 16:00:12.469:WARN::/solr/admin/
> java.lang.IllegalStateException: STREAM
>  at org.mortbay.jetty.Response.getWriter(Response.java:616)
>  at org.apache.jasper.runtime.JspWriterImpl.initOut(JspWriterImpl.java:187)
>  at 
> org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:180)
>  at 
> org.apache.jasper.runtime.PageContextImpl.release(PageContextImpl.java:237)
>  at 
> org.apache.jasper.runtime.JspFactoryImpl.internalReleasePageContext(JspFactoryImpl.java:173)
>  at 
> org.apache.jasper.runtime.JspFactoryImpl.releasePageContext(JspFactoryImpl.java:124)
> 
> (etc.)
> 
> Adding the defaultCoreName fixed it.
> 
> I expect this is indeed your problem.
> 
> -Original Message-
> From: David Sauve [mailto:dnsa...@gmail.com] 
> Sent: Tuesday, August 16, 2011 3:50 PM
> To: solr-user@lucene.apache.org (mailto:solr-user@lucene.apache.org)
> Subject: Re: Unable to get multicore working
> 
> Nope. Only thing in the log:
> 
> 1 [main] INFO org.mortbay.log - Logging to 
> org.slf4j.impl.SimpleLogger(org.mortbay.log) via org.mortbay.log.Slf4jLog
> 173 [main] INFO org.mortbay.log - Redirecting stderr/stdout to 
> /var/log/jetty/2011_08_16.stderrout.log
> 
> 
> 
> 
> On Tuesday, 16 August, 2011 at 1:45 PM, Alexei Martchenko wrote:
> 
> > Is your solr.xml in usr/share/jetty/solr/solr.xml?
> > 
> > lets try this xml instead
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > Can you see the logs? You should see something like this
> > 
> > 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader 
> > *INFO: Solr home set to 'solr/'*
> > 16/08/2011 17:30:55 org.apache.solr.servlet.SolrDispatchFilter init
> > INFO: SolrDispatchFilter.init()
> > 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader locateSolrHome
> > INFO: JNDI not configured for solr (NoInitialContextEx)
> > 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader locateSolrHome
> > *INFO: solr home defaulted to 'solr/' (could not find system property or
> > JNDI)*
> > 16/08/2011 17:30:55 org.apache.solr.core.CoreContainer$Initializer
> > initialize
> > *INFO: looking for solr.xml: usr/share/jetty/solr/solr.xml*
> > 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader locateSolrHome
> > INFO: JNDI not configured for solr (NoInitialContextEx)
> > 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader locateSolrHome
> > *INFO: solr home defaulted to 'solr/' (could not find system property or
> > JNDI)*
> > 16/08/2011 17:30:55 org.apache.solr.core.CoreContainer 
> > *INFO: New CoreContainer: solrHome=solr/ instance=21357269*
> > 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader 
> > *INFO: Solr home set to 'solr/'*
> > 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader 
> > *INFO: Solr home set to 'solr\core01\'*
> > 
> > 2011/8/16 David Sauve mailto:dnsa...@gmail.com)>
> > 
> > > Just the one `solr.xml`. The one I added (well, symlinked form my config
> > > folder -- I like to keep my configurations files organized so they can be
> > > managed by git)
> > > 
> > > `start.jar` is in `usr/share/jetty/start.jar`.
> > > 
> > > 
> > > On Tuesday, 16 August, 20

Re: Unable to get multicore working

2011-08-16 Thread David Sauve

Ok. Fixed that too, now. The schema didn't define "long".

Looks like everything is a-okay, now. Thanks for the help. You guys saved me 
from the insane asylum. 

On Tuesday, 16 August, 2011 at 2:32 PM, Jaeger, Jay - DOT wrote:

>  That said, the logs are showing a different error now. Excellent! The site 
> schemas are loading!
> 
> Great!
> 
>  "SEVERE: org.apache.solr.common.SolrException: Unknown fieldtype 'long' 
> specified on field area_id"
> 
> Go have a look at your conf/schema.xml. 
> 
> Is the following line present?? Does your field definition for area_id follow 
> it?
> 
>  omitNorms="true" positionIncrementGap="0"/>
> 
> Look at the file with an XML editor. Perhaps an edit to some earlier portion 
> of the schema is messing up this part of the schema?
> 
> 
> -Original Message-
> From: David Sauve [mailto:dnsa...@gmail.com] 
> Sent: Tuesday, August 16, 2011 4:24 PM
> To: solr-user@lucene.apache.org (mailto:solr-user@lucene.apache.org)
> Subject: Re: Unable to get multicore working
> 
> I updated my `solr.xml` as follow:
> 
> 
> 
> 
>  dataDir="/home/webteam/preview/data" />
>  dataDir="/home/webteam/staging/data" />
>  dataDir="/home/webteam/live/data" />
> 
> 
> 
> 
> and I'm still seeing the same 404 when I true to view /solr/admin/ or 
> /solr/live/admin/
> 
> That said, the logs are showing a different error now. Excellent! The site 
> schemas are loading!
> 
> Looks like the site schemas have an issue:
> 
> "SEVERE: org.apache.solr.common.SolrException: Unknown fieldtype 'long' 
> specified on field area_id"
> 
> Errr. Why would `long` be an invalid type? 
> 
> 
> On Tuesday, 16 August, 2011 at 2:06 PM, Jaeger, Jay - DOT wrote:
> 
> > Whoops: That was Solr 4.0 (which pre-dates 3.1).
> > 
> > I doubt very much that the release matters, though: I expect the behavior 
> > would be the same.
> > 
> > -Original Message-
> > From: Jaeger, Jay - DOT [mailto:jay.jae...@dot.wi.gov] 
> > Sent: Tuesday, August 16, 2011 4:04 PM
> > To: solr-user@lucene.apache.org (mailto:solr-user@lucene.apache.org) 
> > (mailto:solr-user@lucene.apache.org)
> > Subject: RE: Unable to get multicore working
> > 
> > I tried on my own test environment -- pulling out the default core 
> > parameter out, under Solr 3.1 
> > 
> > I got exactly your symptom: an error 404. 
> > 
> >  HTTP ERROR 404
> >  Problem accessing /solr/admin/index.jsp. Reason: 
> > 
> >  missing core name in path
> > 
> > The log showed:
> > 
> > 2011-08-16 16:00:12.469:WARN::/solr/admin/
> > java.lang.IllegalStateException: STREAM
> >  at org.mortbay.jetty.Response.getWriter(Response.java:616)
> >  at org.apache.jasper.runtime.JspWriterImpl.initOut(JspWriterImpl.java:187)
> >  at 
> > org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:180)
> >  at 
> > org.apache.jasper.runtime.PageContextImpl.release(PageContextImpl.java:237)
> >  at 
> > org.apache.jasper.runtime.JspFactoryImpl.internalReleasePageContext(JspFactoryImpl.java:173)
> >  at 
> > org.apache.jasper.runtime.JspFactoryImpl.releasePageContext(JspFactoryImpl.java:124)
> > 
> > (etc.)
> > 
> > Adding the defaultCoreName fixed it.
> > 
> > I expect this is indeed your problem.
> > 
> > -Original Message-
> > From: David Sauve [mailto:dnsa...@gmail.com] 
> > Sent: Tuesday, August 16, 2011 3:50 PM
> > To: solr-user@lucene.apache.org (mailto:solr-user@lucene.apache.org) 
> > (mailto:solr-user@lucene.apache.org)
> > Subject: Re: Unable to get multicore working
> > 
> > Nope. Only thing in the log:
> > 
> > 1 [main] INFO org.mortbay.log - Logging to 
> > org.slf4j.impl.SimpleLogger(org.mortbay.log) via org.mortbay.log.Slf4jLog
> > 173 [main] INFO org.mortbay.log - Redirecting stderr/stdout to 
> > /var/log/jetty/2011_08_16.stderrout.log
> > 
> > 
> > 
> > 
> > On Tuesday, 16 August, 2011 at 1:45 PM, Alexei Martchenko wrote:
> > 
> > > Is your solr.xml in usr/share/jetty/solr/solr.xml?
> > > 
> > > lets try this xml instead
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > Can you see the logs? You should see something like this
> > > 
> > > 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader 
> > > *INFO: Solr home set to 'solr/'*
> > > 16/08/2011 17:30:55 org.apache.solr.servlet.SolrDispatchFilter init
> > > INFO: SolrDispatchFilter.init()
> > > 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader locateSolrHome
> > > INFO: JNDI not configured for solr (NoInitialContextEx)
> > > 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader locateSolrHome
> > > *INFO: solr home defaulted to 'solr/' (could not find system property or
> > > JNDI)*
> > > 16/08/2011 17:30:55 org.apache.solr.core.CoreContainer$Initializer
> > > initialize
> > > *INFO: looking for solr.xml: usr/share/jetty/solr/solr.xml*
> > > 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader locateSolrHome
> > > INFO: JNDI not configured for solr (NoInitialContextEx)
> > > 16/08/2011 17:30:55 org.apache.solr.core.SolrResourceLoader locateSolrH

Re: Migration from Autonomy IDOL to SOLR

2011-08-16 Thread Arcadius Ahouansou

Hello Karsten.
>From the doc you provided, it seems the two are totally different products.

I thought a bit about it and it seems that the best aproach would be to:

1-refactor our app and add an abstraction layer that will call the IDOL ACI
API.
Make sure we have good tests in place.

2-implement a "solr-dialect" for the abstraction layer that we can switch on
or off in config.


Thanks.

Arcadius.

On Tue, Aug 16, 2011 at 4:09 PM,  wrote:

> Hi Arcadius,
>
> currently we have a migration project from verity k2 search server to solr.
> I do not know IDOL, but autonomy bought verity before IDOL was released, so
> possible it is comparable?
> verity k2 works directly on xml-Files, in result the query syntax is a
> little bit like xpath e.g. with "text1  zone2  zone1" instead of
> contains(//zone1/zone2,'text1').
>
> About verity query syntax:
>
> http://gregconely.getmyip.com/dl/OTG%20Software/5.30.087%20Suite%20%28SP3%29/Disc%204%20-%20Verity/Verity%20K2%20Server%205.5/doc/docs/pdf/VerityQueryLanguage.pdf
>
> Does IDOL work the same way?
>
>
> Best regards
>  Karsten
>
> P.S. in Context:
>
> http://lucene.472066.n3.nabble.com/Migration-from-Autonomy-IDOL-to-SOLR-td3255377.html
>
>  Original-Nachricht 
> > Datum: Mon, 15 Aug 2011 11:11:36 +0100
> > Von: Arcadius Ahouansou 
> > An: solr-user@lucene.apache.org
> > Betreff: Migration from Autonomy IDOL to SOLR
>
> > Hello.
> >
> > We have a couple of application running on half a dozen Autonomy IDOL
> > servers.
> > Currently, all feature we need are supported by Solr.
> >
> > We have done some internal testing and realized that SOLR would do a
> > better
> > job.
> >
> > So, we are investigation all possibilities for a smooth migration from
> > IDOL
> > to SOLR.
> >
> > I am looking for advice from people who went through something similar.
> >
> > Ideally, we would like to keep most of our legacy code unchanged and have
> > a
> > kind of query-translation-layer plugged into our app if possible.
> >
> > -Is there lib available?
> >
> > -Any thought?
> >
> > Thanks.
> >
> > Arcadius.
>

Problems generating war distribution using ant

2011-08-16 Thread arian487

So the way I generate war files now is by running an 'ant dist' in the solr
folder.  It generates the war fine and I get a build success, and then I
deploy it to tomcat and once again the logs show it was successful (from the
looks of it).  However, when I go to 'myip:8080/solr/admin' I get an HTTP
status 404.

However, it works when I take a war from the nightly build, expand it, drop
some new class files in there that I need, and close it up again.  The solr
I have checked out seems fine though and I can't find any differences
between the war I'm generating and the one that has been generated.

--
View this message in context: 
http://lucene.472066.n3.nabble.com/Problems-generating-war-distribution-using-ant-tp3260070p3260070.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Problems generating war distribution using ant

2011-08-16 Thread Sujit Pal

FWIW, we have some custom classes on top of solr as well. The way we do
it is using the following ant target:

  





  
  
  ...





Seems to work fine...basically automates what you have described in your
second paragraph, but allows us to keep our own code separately from
solr code under source control.

-sujit

On Tue, 2011-08-16 at 16:09 -0700, arian487 wrote:
> So the way I generate war files now is by running an 'ant dist' in the solr
> folder.  It generates the war fine and I get a build success, and then I
> deploy it to tomcat and once again the logs show it was successful (from the
> looks of it).  However, when I go to 'myip:8080/solr/admin' I get an HTTP
> status 404.
> 
> However, it works when I take a war from the nightly build, expand it, drop
> some new class files in there that I need, and close it up again.  The solr
> I have checked out seems fine though and I can't find any differences
> between the war I'm generating and the one that has been generated.
> 
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/Problems-generating-war-distribution-using-ant-tp3260070p3260070.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Problems generating war distribution using ant

2011-08-16 Thread arian487

Interesting.  I can use this as an option and create a custom 'war' target if
need be but I'd like to avoid this.  I'd rather do a full build from the
source code I have checked out from the SVN.  Any reason why 'ant dist'
doesn't produce a good war file?

--
View this message in context: 
http://lucene.472066.n3.nabble.com/Problems-generating-war-distribution-using-ant-tp3260070p3260122.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Problems generating war distribution using ant

2011-08-16 Thread arian487

Interesting.  I can use this as an option and create a custom 'war' target if
need be but I'd like to avoid this.  I'd rather do a full build from the
source code I have checked out from the SVN.  Any reason why 'ant dist'
doesn't produce a good war file?

--
View this message in context: 
http://lucene.472066.n3.nabble.com/Problems-generating-war-distribution-using-ant-tp3260070p3260126.html
Sent from the Solr - User mailing list archive at Nabble.com.

Solr UIMA integration problem

2011-08-16 Thread solr nps

Hello,

I am using Solr 3.3. I have been following instructions at
https://svn.apache.org/repos/asf/lucene/dev/tags/lucene_solr_3_3/solr/contrib/uima/README.txt

My setup looks like the following.

solr lib directory contains the following jars

apache-solr-uima-3.3.0.jar
commons-digester-2.0.jar
uima-an-alchemy-2.3.1-SNAPSHOT-r1062868.jar
uima-an-calais-2.3.1-SNAPSHOT-r1062868.jar
uima-an-tagger-2.3.1-SNAPSHOT-r1062868.jar
uima-an-wst-2.3.1-SNAPSHOT-r1076132.jar
uimaj-core-2.3.1.jar


solr_config.xml has the following changes.

 

  

  MY_KEY
  MY_KEY
  MY_KEY
  MY_KEY
  MY_KEY
  MY_SECOND_KEY

/org/apache/uima/desc/OverridingParamsExtServicesAE.xml
false
title

  false
*  *
*title*
*  *


  
org.apache.uima.alchemy.ts.concept.ConceptFS

  *title*
*  concept*

  
  
org.apache.uima.SentenceAnnotation

 * title*
*  sentence*

  

  


and



  uima

 

I am trying to index a simple document which looks like the following



1456780001
Canon powershow camera 9000




I am using curl to post this document on the /update end point and I am
getting the following error

*org.apache.solr.common.SolrException: processing error: null.* title=Canon
powershow camera 9000,  text="Canon powershow camera 9000..."
at
org.apache.solr.uima.processor.UIMAUpdateRequestProcessor.processAdd(UIMAUpdateRequestProcessor.java:107)
at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:147)
at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:77)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:67)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:164)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:929)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:405)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:279)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:515)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:300)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: org.apache.solr.uima.processor.exception.FieldMappingException
*at
org.apache.solr.uima.processor.UIMAToSolrMapper.map(UIMAToSolrMapper.java:83)
*
* at
org.apache.solr.uima.processor.UIMAUpdateRequestProcessor.processAdd(UIMAUpdateRequestProcessor.java:85)
*
* ... 23 more*

What could be the problem?

Thanks for your time

Re: How to connect Solr with external Zookeeper ensemble

2011-08-16 Thread Erick Erickson

What have you tried already? In particular, have you looked at
http://wiki.apache.org/solr/SolrCloud

Best
Erick

On Tue, Aug 16, 2011 at 2:22 PM, Sharath Jagannath
 wrote:
> Hey,
>
> How could I connect my solr server with external zookeeper?
>
> Thanks,
> Sharath
>

Re: exceeded limit of maxWarmingSearchers ERROR

2011-08-16 Thread Nagendra Nagarajayya


Naveen:

See below:

*NRT with Apache Solr 3.3 and RankingAlgorithm does need a commit for a
document to become searchable*. Any document that you add through update
becomes  immediately searchable. So no need to commit from within your
update client code.  Since there is no commit, the cache does not have to be
cleared or the old searchers closed or  new searchers opened, and warmed
(error that you are facing).


Looking at the link which you mentioned is clearly what we wanted. But the
real thing is that you have "RA does need a commit for  a document to become
searchable" (please take a look at bold sentence) .



Yes, as said earlier you do not need a commit. A document becomes 
searchable as soon as you add it. Below is an example of adding a 
document with curl (this from the wiki at 
http://solr-ra.tgels.com/wiki/en/Near_Real_Time_Search_ver_3.x):


curl 
"http://localhost:8983/solr/update/csv?stream.file=/tmp/x1.csv&encapsulator=%1f";


There is no commit included. The contents of the document become 
immediately searchable.



In future, for more loads, can it cater to Master Slave (Replication) and
etc to scale and perform better? If yes, we would like to go for NRT and
looking at the performance described in the article is acceptable. We were
expecting the same real time performance for a single user.



There are no changes to Master/Slave (replication) process. So any 
changes you have currently will work as before or if you enable 
replication later, it should still work as without NRT.



What about multiple users, should we wait for 1-2 secs before calling the
curl request to make SOLR perform better. Or internally it will handle with
multiple request (multithreaded and etc).


Again for updating documents, you do not have to change your current 
process or code. Everything remains the same, except that if you were 
including commit, you do not include commit in your update statements. 
There is no change to the existing update process so internally it will 
not queue or multi-thread updates. It is as in existing Solr 
functionality, there no changes to the existing setup.


Regarding perform better, in the Wiki paper  every update through curl 
adds (streams) 500 documents. So you could take this approach. (this was 
something that I chose randomly to test the performance but seems to be 
good)



What would be doc size (10,000 docs) to allow JVM perform better? Have you
done any kind of benchmarking in terms of multi threaded and multi user for
NRT and also JVM tuning in terms of SOLR sever performance. Any kind of
performance analysis would help us to decide quickly to switch over to NRT.



The performance discussed in the wiki paper uses the MBArtists index. 
The MBArtists index is the index used as one of the examples in the 
book, Solr 1.4 Enterprise Search Server. You can download and build this 
index if you have the book or can also download the contents from 
musicbrainz.org.  Each doc maybe about 100 bytes and has about 7 fields. 
Performance with wikipedia's xml dump, commenting out skipdoc field 
(include redirects) in the dataconfig.xml [ dataimport handler ], the 
update performance is about 15000 docs / sec (100 million docs), with 
the skipdoc enabled (does not skip redirects), the performance is about 
1350 docs / sec [ time spent mostly converting validating/xml  than 
actual update ] (about 11 million docs ).  Documents in wikipedia can be 
quite big, at least avg size of about 2500-5000 bytes or more.


I would suggest that you download and give NRT with Apache Solr 3.3 and 
RankingAlgorithm a try and get a feel of it as this would be the best 
way to see how your config works with it.



Questions in terms for switching over to NRT,


1.Should we upgrade to SOLR 4.x ?

2. Any benchmarking (10,000 docs/secs).  The question here is more specific

the detail of individual doc (fields, number of fields, fields size,
parameters affecting performance with faceting or w/o faceting)


Please see the MBArtists index as discussed above.



3. What about multiple users ?

A user in real time might be having an large doc size of .1 million. How to
break and analyze which one is better (though it is our task to do). But
still any kind of break up will help us. Imagine a user inbox.



You maybe able to stream the documents in a set as in the example in the 
wiki. The example streams 500 documents at a time. The wiki paper has an 
example of a document that was used. You could copy/paste that to try it 
out.



4. JVM tuning and performance result based on Multithreaded environment.

5. Machine Details (RAM, CPU, and settings from SOLR perspective).



Default Solr settings with the shipped jetty container. The startup 
script used is available when you download Solr 3.3 with 
RankingAlgorithm. It has mx set to 2Gb and uses the default collector 
with parallel collection enabled for the young generation.  The system 
is a x86_64 Linux (2.6 kernel), 2 core (2.5Ghz) and uses interna

Re: Product data schema question

2011-08-16 Thread Alexander Ramos Jardim

Why don't you use fields for each size? You can update our inventory only in
the event of a size becoming avaiable or unavaviable. That would remove a
lot of the load in inventory update.

Anothe way is to treat each sku/inventory pair as a document.

2011/8/16 Jaeger, Jay - DOT 

> Not particularly.  Just trying to do my part to answer some questions on
> the list.
>
> -Original Message-
> From: Steve Cerny [mailto:sjce...@gmail.com]
> Sent: Tuesday, August 16, 2011 11:49 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Product data schema question
>
> Thanks Jay, if we come to a reasonable solution are you interested in the
> details?
>
> On Tue, Aug 16, 2011 at 11:44 AM, Jaeger, Jay - DOT
> wrote:
>
> > No, I don't think so.  A given core can only use one configuration and
> > therefore only one schema, as far as I know, and a schema can only have
> one
> > key.
> >
> > You could use two cores with two configurations (but that presumably
> > wouldn't be much help).
> >
> > Solr is not a DBMS.  It is an index.
> >
> > -Original Message-
> > From: Steve Cerny [mailto:sjce...@gmail.com]
> > Sent: Tuesday, August 16, 2011 11:37 AM
> > To: solr-user@lucene.apache.org
> > Subject: Re: Product data schema question
> >
> > Jay, this is great information.
> >
> > I don't know enough about Solr whether this is possible...Can we setup
> two
> > indexes in the same core, one for product_catalog and the other for
> > inventory?  Then using a Solr query we could join the indexed content
> > together.
> >
> > In Sql it would look like this
> >
> > select
> >  p.brand
> >  , p.model_name
> >  , p.sku
> >  , p.color_name
> >  , i.variant_id
> >  , i.variant_count
> > from product_catalog p
> > join inventory i on (p.sku = i.sku)
> >
> > On Tue, Aug 16, 2011 at 8:00 AM, Jaeger, Jay - DOT <
> jay.jae...@dot.wi.gov
> > >wrote:
> >
> > > On the surface, you could simply add some more fields to your schema.
> >  But
> > > as far as I can tell, you would have to have a separate Solr "document"
> > for
> > > each SKU/size combination,  and store the rest of the information
> (brand,
> > > model, color, SKU) redundantly and make the unique key a combination of
> > the
> > > SKU and the size (presumably by having an additional field called
> > sku_size -
> > > as far as I can tell Solr can't make a key up out of multiple fields).
> > >
> > > But, perhaps you shouldn't store that dynamic inventory information in
> > > Solr.   Instead store a key that gets you to an inventory database
> > organized
> > > by SKU and size.  In such a schema, Size could be added as a
> multi-valued
> > > field to your existing schema, so you would know what sizes existed, so
> > you
> > > could at least facet on that, maybe.  That way Solr doesn't have to be
> > > updated every time the inventory changes.  Of course, that won't help
> > with
> > > the faceting  on variant inventory.
> > >
> > > -Original Message-
> > > From: Steve Cerny [mailto:sjce...@gmail.com]
> > > Sent: Monday, August 15, 2011 6:29 PM
> > > To: solr-user@lucene.apache.org
> > > Subject: Product data schema question
> > >
> > > I'm working on an online eCommerce project and am having difficulties
> > > building the core / index schema.  Here is the way we organize our
> > product
> > > information in a normalized database.
> > >
> > > A product model has many SKUs (called colorways)
> > > A SKU has many sizes (called variants)
> > > A SKU size has associated inventory (called variant inventory)
> > >
> > > When we setup our product core we have the following field information
> > >
> > > Doc
> > > * brand
> > > * model name
> > > * SKU
> > > * color name
> > >
> > > Sample records are as follows
> > >
> > > * Haynes, Undershirt, 1234, white
> > > * Haynes, Undershirt, 1235, grey
> > > * Fruit of the Loom, Undershirt, 1236, white
> > > * Fruit of the Loom, Underwear, 1237, grey
> > >
> > > The issue I'm having is I want to add inventory to each size of each
> SKU
> > > for
> > > faceting.  Example,
> > >
> > > SKU 1234 has sizes small, medium, large.  Size small has 5 in stock,
> size
> > > medium 10, and size large 25.
> > >
> > > In a normalized data table I would have a separate table just for
> > inventory
> > > and related it back to the SKU with a foreign key.  How do I store size
> > and
> > > inventory information effectively with Solr?
> > >
> > > --
> > > Steve
> > >
> >
> >
> >
> > --
> > Steve
> >
>
>
>
> --
> Steve Cerny
> sjce...@gmail.com
> 715-302-0639
>



-- 
Alexander Ramos Jardim

Periodic search in date field

2011-08-16 Thread slaava

Hi,
I have this problem: We have about 100k persons with date of birth indexed
by solr. Now we need find persons with birth anniversary for any input date.
Exactly 1, 5 and 10 year is neccessary.

For example:
Input date: 17.8.2011

Required output:
1 year: persons with date of birth 17.8.2010, 17.8.2009, ..., 17.8.879, ...
5 year: ... date of birth 17.8.2006, 17.8.2001, ..., 17.8.1006, ...
10 year: 17.8.2001, 17.8.1991, ..., 17.8.891, ...

My first idea was to index date in mmdd format (20110817) and use
modulo: 
1 year: indexed_date % 1 = 0817
5 year: indexed_date % 5 = 10817
10 year: indexed_date % 10 = 10817

but I didn't found something like modulo function in solr...

Is there any simple solution for this problem? I'm just beginner in solr, so
maybe I missed something.
Thanks for any help!

P.S: I'm sorry for my bad english.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Periodic-search-in-date-field-tp3260793p3260793.html
Sent from the Solr - User mailing list archive at Nabble.com.

Using Solr for indexing and searching files in a directory

2011-08-16 Thread Jagdish Kumar


Hi All
 
I have this requirement of indexing and searching files (txt, doc,pdf) on my 
disk using Solr Search which I have installed.
I am unable to find a relevant tutorial for the same, I would be thankfull if 
anyone of you can actually help me out with the specific steps required.
 
Thanks and regards
Jagdish

75 matches

Mail list logo