Hi Alexandre,
OK some good progress was made based on this advice. Thanks! I think we're
in the home stretch with the data import. Not there yet. But hopefully
close.
> Two problems:
> 1) You have <span> (span) elements in your solrconfig.xml. They just
> do not belong there. The original tutorial screwed up. Your element
> should be on the same level as the other elements in that example.
> 2) You also seem to have another random piece of data configuration in
> the solrconfig.xml. Also in the spans, so they are being ignored. But
> still very very wrong. Take those out all together.
> You should just have 3 things tying together:
> 1) jars loaded in the lib statement in solrconfig.xml
> 2) handler definition that points at your data-config file
> 3) data-config file itself.
OK so here I'm loading the libs:
* <lib dir="/opt/solr/lib/" regex="mysql-connector-java-.*\.jar" />*
* <lib dir="/opt/solr/dist/" regex="solr-dataimporthandler-.*\.jar" />*
<lib dir="/opt/solr/contrib/extraction/lib" regex=".*\.jar" />
<lib dir="/opt/solr/dist/" regex="solr-cell-\d.*\.jar" />
<lib dir="/opt/solr/contrib/clustering/lib/" regex=".*\.jar" />
<lib dir="/opt/solr/dist/" regex="solr-clustering-\d.*\.jar" />
<lib dir="/opt/solr/contrib/langid/lib/" regex=".*\.jar" />
<lib dir="/opt/solr/dist/" regex="solr-langid-\d.*\.jar" />
<lib dir="/opt/solr/contrib/velocity/lib" regex=".*\.jar" />
<lib dir="/opt/solr/dist/" regex="solr-velocity-\d.*\.jar" />
Verified the files are there.
[root@solr1:/opt/solr/collection1/conf] #ls -l /opt/solr/lib/ | grep mysql
-rw-r--r--. 1 root root 959987 Nov 3 19:17
mysql-connector-java-5.1.33-bin.jar
[root@solr1:/opt/solr/collection1/conf] #ls -l /opt/solr/lib/ | grep mysql
-rw-r--r--. 1 root root 959987 Nov 3 19:17
mysql-connector-java-5.1.33-bin.jar
[root@solr1:/opt/solr/collection1/conf] #ls -l /opt/solr/dist/ | grep
dataimport
-rw-r--r--. 1 tomcat tomcat 219261 Sep 24 06:07
solr-dataimporthandler-4.10.1.jar
-rw-r--r--. 1 tomcat tomcat 37443 Sep 24 06:07
solr-dataimporthandler-extras-4.10.1.jar
Added this entry to solrconfig.xml (without the spans):
<requestHandler name="/dataimport" class="solr.DataImportHandler">
<lst name="defaults">
<str name="config">db-data-config.xml</str>
</lst>
</requestHandler>
Then added this db-data-config.xml file in the same directory as the
solrconfig.xml
<dataConfig>
<dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
url="jdbc:mysql://web1.mydomain.com:3306/jokefire" user="admin"
password="secret" batchSize="1" />
<document name="players">
<entity name="jokefire_users" query="SELECT * FROM wp_users">
<field column="ID" name="ID" />
<field column="user_login" name="user_login" />
<field column="user_nicename" name="user_nicename" />
<field column="user_email" name="user_email" />
<field column="user_url" name="user_url" />
<field column="user_registered" name="user_registered" />
<field column="user_activation_key" name="user_activation_key"
/>
<field column="user_status" name="user_status" />
<field column="display_name" name="display_name" />
</entity>
</document>
</dataConfig>
Verified I could connect to the DB with the info supplied in the data
config file:
[root@solr1:/opt/solr/collection1/conf] #mysql -uadmin -p -h
web1.mydomain.com jokefire
Enter password:
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is 8628551
Server version: 5.5.39 MySQL Community Server (GPL) by Remi
Copyright (c) 2000, 2014, Oracle, Monty Program Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input
statement.
MySQL [jokefire]>
Bounced tomcat, and there it was!! I now had a web interface for the data
import feature! Thank you for helping to get me this far!
[image: Inline image 1]
However at this stage the import, even tho it says it's been started, just
kind of sits there. And no records are actually imported.
I took a look at the logs and found these entries:
11/3/2014, 7:21:03 PMWARNSimplePropertiesWriterUnable to read:
dataimport.properties11/3/2014, 7:21:04 PMWARNSimplePropertiesWriterUnable
to read: dataimport.properties
It looks as if something is still failing. But I googled that error and
found that the answer to that was to make 'the conf directory writable'.
I'll experiment with tightening up permissions, but at that point I just
wanted to see if that would solve this. So I made it world writable with
chmod 777. And lo and behold an import happened!!
Last Update: 19:40:26
*Indexing completed. Added/Updated: 4 documents. Deleted 0 documents.
(Duration: 01s)*
Requests: 1 (1/s), Fetched: 4 (4/s), Skipped: 0, Processed: 4 (4/s)
Started: 2 minutes ago
Very cool. Finally I can start to use solr with some real data. Not much in
this database yet. But that's ok I'll add some data and have a look.
Hopefully this will be the database for a live app someday, making this
little exercise in solr indexing useful!
Thanks again!
Tim
> If you are still having troubles, I strongly recommend getting the
> shipped example to work and then adding your own stuff until you get
> that working. Then, try to create a standalone configuration.
> Sometimes, this is an easier approach for the first time user.
> Regards,
> Alex.
> P.s. I also cover that in my Solr book. A relevant example is here:
> https://github.com/arafalov/solr-indexing-book/tree/master/published/dihdb
> Personal: http://www.outerthoughts.com/ and @arafalov
> Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
> Solr popularizers community: https://www.linkedin.com/groups?gid=6713853
On Mon, Nov 3, 2014 at 1:50 PM, Alexandre Rafalovitch <arafa...@gmail.com>
wrote:
> Two problems:
> 1) You have <span> (span) elements in your solrconfig.xml. They just
> do not belong there. The original tutorial screwed up. Your element
> should be on the same level as the other elements in that example.
> 2) You also seem to have another random piece of data configuration in
> the solrconfig.xml. Also in the spans, so they are being ignored. But
> still very very wrong. Take those out all together.
>
> You should just have 3 things tying together:
> 1) jars loaded in the lib statement in solrconfig.xml
> 2) handler definition that points at your data-config file
> 3) data-config file itself.
>
> If you are still having troubles, I strongly recommend getting the
> shipped example to work and then adding your own stuff until you get
> that working. Then, try to create a standalone configuration.
> Sometimes, this is an easier approach for the first time user.
>
> Regards,
> Alex.
> P.s. I also cover that in my Solr book. A relevant example is here:
> https://github.com/arafalov/solr-indexing-book/tree/master/published/dihdb
> Personal: http://www.outerthoughts.com/ and @arafalov
> Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
> Solr popularizers community: https://www.linkedin.com/groups?gid=6713853
>
>
> On 3 November 2014 13:40, Tim Dunphy <bluethu...@gmail.com> wrote:
> > Hi Alexandre,
> >
> > Thanks so much for your input and examples! Ok so here's what I've done
> so
> > far with no luck as of yet unfortunately.
> >
> > Inside of solrconfig.xml I put the following:
> >
> > <lib dir="/opt/solr/dist/" regex="solr-dataimporthandler-.*\.jar" />
> > <lib dir="/opt/solr/contrib/extraction/lib" regex=".*\.jar" />
> > <lib dir="/opt/solr/dist/" regex="solr-cell-\d.*\.jar" />
> >
> > <lib dir="/opt/solr/contrib/clustering/lib/" regex=".*\.jar" />
> > <lib dir="/opt/solr/dist/" regex="solr-clustering-\d.*\.jar" />
> > <lib dir="/opt/solr/contrib/langid/lib/" regex=".*\.jar" />
> > <lib dir="/opt/solr/dist/" regex="solr-langid-\d.*\.jar" />
> >
> > <lib dir="/opt/solr/contrib/velocity/lib" regex=".*\.jar" />
> > <lib dir="/opt/solr/dist/" regex="solr-velocity-\d.*\.jar" />
> >
> > As you can see, I've replaced the relative paths with absolute ones. So
> as
> > of now, my solr 4 server is no longer complaining about not being able to
> > find directories and modules. So we're off to a good start! And now I can
> > list the 'dist' directory and in my case find the jar files I'm looking
> for.
> >
> >
> > [root@solr1:/opt/solr/collection1/conf] #ls /opt/solr/dist/ | grep
> > dataimporthandler
> > solr-dataimporthandler-4.10.1.jar
> > solr-dataimporthandler-extras-4.10.1.jar
> >
> > So far so good.
> >
> > I next tried this db-data-config file in the same directory as
> > solrconfig.xml
> >
> > [root@solr1:/opt/solr/collection1/conf] #cat db-data-config.xml.bak
> > <span>
> >
> > <dataConfig>
> >
> > <dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
> > url="jdbc:mysql://web1.mydomain.com:3306/jokefire" user="admin"
> > password="secret" batchSize="1" />
> > <document name="players">
> >
> > <entity name="jokefire_users" query="SELECT * FROM wp_users">
> > <field column="ID" name="ID" />
> > <field column="user_login" name="user_login" />
> > <field column="user_nicename" name="user_nicename" />
> > <field column="user_email" name="user_email" />
> > <field column="user_url" name="user_url" />
> > <field column="user_registered" name="user_registered" />
> > <field column="user_activation_key"
> name="user_activation_key"
> > />
> > <field column="user_status" name="user_status" />
> > <field column="display_name" name="display_name" />
> > </entity>
> >
> > </document>
> > </dataConfig>
> > </span>
> >
> > Restarted tomcat, and with this setup I wasn't getting any errors in the
> > browser or logs and the web interface was still working. Always a good
> sign!
> >
> > So then I went down to Core Selector -> collection1 -> data import. And
> it
> > was quite frustrating, cuz I was getting the same error as before!
> >
> > sorry, no dataimport-handler defined!
> >
> > So then I tried the exact db-data-config.xml file from your example.
> >
> > <dataConfig>
> > <dataSource driver="org.hsqldb.jdbcDriver"
> > url="jdbc:hsqldb:./example-DIH/hsqldb/ex" user="sa" />
> > <document>
> > <entity name="item" query="select * from item"
> > deltaQuery="select id from item where last_modified >
> > '${dataimporter.last_index_time}'">
> > <field column="NAME" name="name" />
> >
> > <entity name="feature"
> > query="select DESCRIPTION from FEATURE where
> > ITEM_ID='${item.ID}'"
> > deltaQuery="select ITEM_ID from FEATURE where
> > last_modified > '${dataimporter.last_index_time}'"
> > parentDeltaQuery="select ID from item where
> > ID=${feature.ITEM_ID}">
> > <field name="features" column="DESCRIPTION" />
> > </entity>
> >
> > <entity name="item_category"
> > query="select CATEGORY_ID from item_category where
> > ITEM_ID='${item.ID}'"
> > deltaQuery="select ITEM_ID, CATEGORY_ID from
> > item_category where last_modified > '${dataimporter.last_index_time}'"
> > parentDeltaQuery="select ID from item where
> > ID=${item_category.ITEM_ID}">
> > <entity name="category"
> > query="select DESCRIPTION from category where ID
> =
> > '${item_category.CATEGORY_ID}'"
> > deltaQuery="select ID from category where
> > last_modified > '${dataimporter.last_index_time}'"
> > parentDeltaQuery="select ITEM_ID, CATEGORY_ID
> from
> > item_category where CATEGORY_ID=${category.ID}">
> > <field column="DESCRIPTION" name="cat" />
> > </entity>
> > </entity>
> > </entity>
> > </document>
> > </dataConfig>
> >
> > Knowing full well it wouldn't actually work, as I"m using a remote mysql
> > database instead of a local hsqldb database.But at this point, my only
> goal
> > was to get the data import to show up as an option. I'd tweak the
> > db-data-config.xml file at a later point if this in fact worked!
> >
> > But alas, I was still getting the same result...
> >
> > sorry, no dataimport-handler defined!
> >
> > Grrrr.. so annoying after all that work. Anyway, I really do appreciate
> your
> > kindness and help. :) I'm enclosing my solrconfig.xml and both versions
> of
> > my db-data-config.xml in hopes that we can make some progress here!
> >
> >
> > Thank
> >
> >
> > Tim
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Sun, Nov 2, 2014 at 9:50 PM, Alexandre Rafalovitch <
> arafa...@gmail.com>
> > wrote:
> >>
> >> That tutorial seems to be somewhat dodgy. You need at least one more
> >> step of adding DIH library in solrconfig.xml:
> >>
> >>
> https://github.com/apache/lucene-solr/blob/lucene_solr_4_10_2/solr/example/example-DIH/solr/db/conf/solrconfig.xml#L75
> >> (I recommend using absolute path though).
> >>
> >> Also, you should not need to spell the full class out. See lower down
> >> in the same class:
> >>
> >>
> https://github.com/apache/lucene-solr/blob/lucene_solr_4_10_2/solr/example/example-DIH/solr/db/conf/solrconfig.xml#L823
> >>
> >> Finally, in the config file, I don't remember document element having
> >> a name. Again, the working example can be found in the same directory:
> >>
> >>
> https://github.com/apache/lucene-solr/blob/lucene_solr_4_10_2/solr/example/example-DIH/solr/db/conf/db-data-config.xml#L3
> >>
> >> Solr ships with a bunch of examples. If you are using/download
> >> standard distribution, you could start from those until you understand
> >> how it all hangs together.
> >>
> >> Regards,
> >> Alex.
> >>
> >> Personal: http://www.outerthoughts.com/ and @arafalov
> >> Solr resources and newsletter: http://www.solr-start.com/ and
> @solrstart
> >> Solr popularizers community:
> https://www.linkedin.com/groups?gid=6713853
> >>
> >>
> >> On 2 November 2014 21:26, Tim Dunphy <bluethu...@gmail.com> wrote:
> >> > Hi Alex,
> >> >
> >> >
> >> >> I thought the "<span style="font-family: Trebuchet MS, sans-serif;">"
> >> >> and the ending span were broken email thing but they seem to be in
> the
> >> >> solrconfig.xml file as well. I would start from removing those and
> >> >> leaving just the actual definition.
> >> >
> >> >
> >> > Thanks for your response!
> >> >
> >> > OK so I tried your suggestion of removing those span tags like so:
> >> >
> >
> >
> >
> >
> > --
> > GPG me!!
> >
> > gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
> >
>
--
GPG me!!
gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
