If you send a dummy document using a curl command, without the commit
option, does it auto-commit and become visible in 1 minute?
-- Jack Krupansky
-----Original Message-----
From: Jonatan Fournier
Sent: Tuesday, August 14, 2012 11:03 AM
To: solr-user@lucene.apache.org ; erickerick...@gmail.com
Subject: Re: Index not loading
Hi Erick,
On Tue, Aug 14, 2012 at 10:25 AM, Erick Erickson
<erickerick...@gmail.com> wrote:
This is quite odd, it really sounds like you're not
actually committing. So, some questions.
1> What happens if you search before you shut
down your tomcat? Do you see docs then? If so,
somehow you're doing soft commits and never
doing a hard commit.
No I'm not seeing any documents if I do search for anything. Like
mentioned above, Num and Max docs are 0.
Like I mentioned below, my index files are not deleted when I
start/restart tomcat, but when within tomcat I send a commit/optimize
command.
On thing I noticed that was different in the log output from the
embedded server was that when I use the solrconfig.xml autoCommit,
after the delay I see some stdout message about commiting to the
index. But when relying on the commitWithin, I never see the solr
server output freeze for a moment while commiting, I only see all my
add document stdout message. Should the behavior be the same? Or the
commit messages pass by so fast I don't see them?
It must be trying to do some kind of commit/merge, because when I was
monitoring the memory I could see periodic memory increase (when I
assumed it was merging) then memory decreased until the next delay...
2> What happens if, as the last statement in your SolrJ
program you do a commit()?
Let me try that and come back to you, for now here's the commands I
was using in the 3 test scenarios:
SolrInputDocument doc = new SolrInputDocument();
solrDoc.addField("id", someId);
...
server.add(doc); // In the case I have either autoCommit
<maxTime>60000</maxTime> enabled in the solrconfig.xml or
<autoSoftCommit>
// Both scenarios works, in those 2 cases when I shutdown my
embeddedserver and restart tomcat I have all my data indexed/commited
or
server.add(doc, 60000) // In the case I don't have autoCommit enabled,
try to rely on commitWithin param.
3> While you're indexing, what do you see in your index
directory? You should see multiple segments being
created, and possibly merged so the number of
files should go up and down. If you only have a single
set of files, you're somehow not doing a commit.
No I do see a bunch of files being created/merged, at the end I had a
bout 89G in many many files.
Another thing I was playing around when trying to use the commitWithin
is to change the <useCompoundFile>true</useCompoundFile> and
<mergeFactor>10</mergeFactor> to reduce the number of files created.
Could it impact things?
4> Is there something really silly going on like your
restart scripts delete the index directory? Or you're
using a VM that restores a blank image?
No VM, no scripts, no replication.
5> When you do restart, are there any files at all
in your index directory?
When I restart tomcat I do see all the same 89G files that was created
using the embedded server, they only vanish when I force a commit or
optimize, then it's like if my data directory didn't exist and the 2
initial segment files are being created and all the rest deleted.
I really suspect you've got some configuration problem
here....
Maybe, but other than playing with the compound file thingy I don't
have any fancy config changes.
Cheers,
/jonathan
Best
Erick
On Mon, Aug 13, 2012 at 9:11 AM, Jonatan Fournier
<jonatan.fourn...@gmail.com> wrote:
Hi,
I'm using Solr 4.0.0-ALPHA and the EmbeddedSolrServer.
Within my SolrJ application, the documents are added to the server
using the commitWithin parameter (in my case 60s). After 1 day my 125
millions document are all added to the server and I can see 89G of
index data files. I stop my SolrJ application and reload my Solr
instance in Tomcat.
From the Solr admin panel related to my Core (collection1) I see this
info:
Last Modified:
Num Docs:0
Max Doc:0
Version:1
Segment Count:0
Optimized: (green check)
Current: (green check)
Master:
Version: 0
Gen: 1
Size: 88.14 GB
From the general Core Admin panel I see:
lastModified:
version:1
numDocs:0
maxDoc:0
optimized: (red circle)
current: (green check)
hasDeletions: (red circle)
If I query my index for *:* I get 0 result. If I trigger optimize it
wipes ALL my data inside the index and reset to empty. I've played
around my EmbeddedServer initially using autoCommit/softCommit and it
was working fine. Now that I've switched to commitWithin the document
add query, it always do that! I'm never able to reload my index within
Tomcat/Solr.
Any idea?
Cheers,
/jonathan
