Search and ye shall find! http://www.lucidimagination.com/search/link?url=http://wiki.apache.org/solr/DataImportHandlerFaq
On Fri, Jun 1, 2012 at 8:15 AM, Dyer, James <james.d...@ingrambook.com> wrote: > Try setting it to 0 or -1. Or check the Mysql JDBC driver documentation > about valid values for "Statement.setFetchSize()" I think someone else > recently asked on this same list about problems with the latest Mysql driver > and fetch sizes, so this driver may be particularly finicky. > > James Dyer > E-Commerce Systems > Ingram Content Group > (615) 213-4311 > > > -----Original Message----- > From: wangjing [mailto:ppm10...@gmail.com] > Sent: Friday, June 01, 2012 10:00 AM > To: solr-user@lucene.apache.org > Subject: Re: why DIH works in normal mode,error in debug mode > > In my datasource config file: > <dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver" > > url="jdbc:mysql://127.0.0.1:3306/MYSOLR?useUnicode=true&characterEncoding=UTF-8" > user="root" password="qwertyuiop" batchSize="500" /> > > i have done it,set batchSize="500" > > > > On Fri, Jun 1, 2012 at 10:38 PM, Dyer, James <james.d...@ingrambook.com> > wrote: >> I see this in your stacktrace: java.sql.SQLException: Illegal value for >> setFetchSize(). >> >> It must be that your JDBC driver doesn't like the default value (300) that >> is used. In your datasource tag, try adding a "batchSize" attribute of >> either 0 or -1 (if using -1, DIH automatically changes it to >> "Integer.MIN_VALUE". According to the wiki this is to fix this error.) The >> value of "batchSize" is used on the java.sql.Statement objects with >> "setFetchSize(batchSize)". >> >> example: >> <dataSource ... batchSize="0" /> >> >> see: >> http://wiki.apache.org/solr/DataImportHandler#Configuring_JdbcDataSource >> >> James Dyer >> E-Commerce Systems >> Ingram Content Group >> (615) 213-4311 >> >> >> -----Original Message----- >> From: wangjing [mailto:ppm10...@gmail.com] >> Sent: Friday, June 01, 2012 9:18 AM >> To: solr-user@lucene.apache.org >> Subject: why DIH works in normal mode,error in debug mode >> >> why DIH works in normal mode,use >> http://localhost:8080/apache-solr-3.6.0/cn/admin/ >> query *:* it can find all >> >> 0 35 0 2.2 *:* on 10 测试类型1 >> 台灯是人们生活中用来照明的一种家用电器。它一般分为两种，一种是立柱式的，一种是有夹子的。它的工作原理主要是把灯光集中在一小块区域内，便于工作和学习。一般台灯用的灯泡是白炽灯或者节能灯泡。 >> 有的台灯还有应急功能，用于停电时无电照明已用来应急。 包含灯泡啥的 1 2012-06-01T06:15:58Z 台灯 法国台灯A234 3 >> 100.2 100.2,USD 包含灯泡啥的法国台灯A234台灯 >> 台灯是人们生活中用来照明的一种家用电器。它一般分为两种，一种是立柱式的，一种是有夹子的。它的工作原理主要是把灯光集中在一小块区域内，便于工作和学习。一般台灯用的灯泡是白炽灯或者节能灯泡。 >> 有的台灯还有应急功能，用于停电时无电照明已用来应急。 10.0 >> >> >> BUT ERROR in debug mode,it boring me few days :_( >> >> the detail exception statck is >> >> 2012-6-1 21:51:51 org.apache.solr.common.SolrException log >> ????: Full Import failed:java.lang.RuntimeException: >> java.lang.RuntimeException: >> org.apache.solr.handler.dataimport.DataImportHandlerException: Unable >> to execute query: select * from ITEM; Processing Document # 1 >> at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:264) >> at >> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:375) >> at >> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:445) >> at >> org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:205) >> at >> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) >> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1376) >> at >> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:365) >> at >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:260) >> at >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) >> at >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) >> at >> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:225) >> at >> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169) >> at >> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) >> at >> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168) >> at >> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98) >> at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927) >> at >> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) >> at >> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407) >> at >> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:999) >> at >> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:565) >> at >> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:309) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:680) >> Caused by: java.lang.RuntimeException: >> org.apache.solr.handler.dataimport.DataImportHandlerException: Unable >> to execute query: select * from ITEM; Processing Document # 1 >> at >> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:621) >> at >> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:327) >> at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:225) >> ... 23 more >> Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException: >> Unable to execute query: select * from ITEM; Processing Document # 1 >> at >> org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:72) >> at >> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:253) >> at >> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:210) >> at >> org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:39) >> at >> org.apache.solr.handler.dataimport.DebugLogger$2.getData(DebugLogger.java:188) >> at >> org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:59) >> at >> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73) >> at >> org.apache.solr.handler.dataimport.EntityProcessorWrapper.pullRow(EntityProcessorWrapper.java:330) >> at >> org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:296) >> at >> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:683) >> at >> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:619) >> ... 25 more >> Caused by: java.sql.SQLException: Illegal value for setFetchSize(). >> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1055) >> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956) >> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:926) >> at com.mysql.jdbc.StatementImpl.setFetchSize(StatementImpl.java:2461) >> at >> org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:242) >> ... 34 more >> >> 2012-6-1 21:51:51 org.apache.solr.update.DirectUpdateHandler2 rollback >> ??Ϣ: start rollback >> 2012-6-1 21:51:51 org.apache.solr.update.DirectUpdateHandler2 rollback >> ??Ϣ: end_rollback >> 2012-6-1 21:51:51 org.apache.solr.core.SolrCore execute >> ??Ϣ: [cn] webapp=/apache-solr-3.6.0 path=/dataimport >> params={start=0&dataConfig=<dataConfig>%0d%0a%09<dataSource+type%3D"JdbcDataSource"+driver%3D"com.mysql.jdbc.Driver"%0d%0a%09%09url%3D"jdbc:mysql://127.0.0.1:3306/MYSOLR?useUnicode%3Dtrue%26amp;characterEncoding%3DUTF-8"%0d%0a%09%09user%3D"root"+password%3D"qwertyuiop"+batchSize%3D"500"+/>%0d%0a%0d%0a%09<document+datasource%3D"JdbcDataSource"+name%3D"item_document">%0d%0a%09%09<entity++name%3D"item"+pk%3D"ITEM_ID"+query%3D"select+*+from+ITEM;"%0d%0a%09%09%09deltaQuery%3D"select+ITEM_ID+from+ITEM+where+LAST_MODIFIED+>+'${dataimporter.last_index_time}';">%0d%0a%09%09%09<field+column%3D"ITEM_ID"+name%3D"item_id"+/>%0d%0a%09%09%09<field+column%3D"NAME"+name%3D"name"+/>%0d%0a%09%09%09<field+column%3D"NAME"+name%3D"nameSort"+/>%0d%0a%09%09%09<field+column%3D"NAME"+name%3D"alphaNameSort"+/>%0d%0a%0d%0a%09%09%09<entity+name%3D"feature"%0d%0a%09%09%09%09query%3D"select+*+from+FEATURE+where+ITEM_ID%3D'${item.ITEM_ID}';"%0d%0a%09%09%09%09deltaQuery%3D"select+ITEM_ID+from+FEATURE+where+LAST_MODIFIED+>+'${dataimporter.last_index_time}';"%0d%0a%09%09%09%09parentDeltaQuery%3D"select+ITEM_ID+from+ITEM+where+ITEM_ID%3D'${feature.ITEM_ID}';"+pk%3D"FEATURE_ID">%0d%0a%09%09%09%09<field+name%3D"features"+column%3D"DESCRIPTION"+/>%0d%0a%09%09%09</entity>%0d%0a%0d%0a%09%09%09<entity+name%3D"item_category"%0d%0a%09%09%09%09query%3D"select+CATEGORY_ID+from+ITEM_CATEGORY+where+ITEM_ID%3D'${item.ITEM_ID}';"%0d%0a%09%09%09%09deltaQuery%3D"select+ITEM_ID,+CATEGORY_ID+from+ITEM_CATEGORY+where+LAST_MODIFIED+>+'${dataimporter.last_index_time}';"%0d%0a%09%09%09%09parentDeltaQuery%3D"select+ITEM_ID+from+ITEM+where+ITEM_ID%3D$'{item_category.ITEM_ID}';"+pk%3D"ITEM_CATEGORY_ID">%0d%0a%09%09%09%09<entity+name%3D"category"%0d%0a%09%09%09%09%09query%3D"select+DESCRIPTION+from+CATEGORY+where+CATEGORY_ID+%3D+'${item_category.CATEGORY_ID}';"%0d%0a%09%09%09%09%09deltaQuery%3D"select+CATEGORY_ID+from+CATEGORY+where+LAST_MODIFIED+>+'${dataimporter.last_index_time}';"%0d%0a%09%09%09%09%09parentDeltaQuery%3D"select+ITEM_ID,+CATEGORY_ID+from+ITEM_CATEGORY+where+CATEGORY_ID%3D'${category.CATEGORY_ID}';"+pk%3D"CATEGORY_ID">%0d%0a%09%09%09%09%09<field+column%3D"description"+name%3D"cat"+/>%0d%0a%09%09%09%09</entity>%0d%0a%09%09%09</entity>%0d%0a%09%09</entity>%0d%0a%09</document>%0d%0a</dataConfig>&verbose=on&command=full-import&debug=on&rows=10} >> status=0 QTime=18 >> >> >> /* >> Navicat MySQL Data Transfer >> >> Source Server : 172.16.197.2 >> Source Server Version : 50162 >> Source Host : 172.16.197.2:3306 >> Source Database : MYSOLR >> >> Target Server Type : MYSQL >> Target Server Version : 50162 >> File Encoding : 65001 >> >> Date: 2012-06-01 14:18:03 >> */ >> >> SET FOREIGN_KEY_CHECKS=0; >> -- ---------------------------- >> -- Table structure for `CATEGORY` >> -- ---------------------------- >> DROP TABLE IF EXISTS `CATEGORY`; >> CREATE TABLE `CATEGORY` ( >> `CATEGORY_ID` int(11) NOT NULL, >> `DESCRIPTION` varchar(100) DEFAULT NULL, >> `LAST_MODIFIED` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON >> UPDATE CURRENT_TIMESTAMP, >> PRIMARY KEY (`CATEGORY_ID`) >> ) ENGINE=MyISAM DEFAULT CHARSET=utf8; >> >> -- ---------------------------- >> -- Records of CATEGORY >> -- ---------------------------- >> INSERT INTO CATEGORY VALUES ('1', '测试类型1', '2012-06-01 14:14:33'); >> >> -- ---------------------------- >> -- Table structure for `FEATURE` >> -- ---------------------------- >> DROP TABLE IF EXISTS `FEATURE`; >> CREATE TABLE `FEATURE` ( >> `FEATURE_ID` char(8) NOT NULL, >> `ITEM_ID` char(8) DEFAULT NULL, >> `DESCRIPTION` varchar(500) DEFAULT NULL, >> `LAST_MODIFIED` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON >> UPDATE CURRENT_TIMESTAMP, >> PRIMARY KEY (`FEATURE_ID`), >> KEY `FK_Reference_1` (`ITEM_ID`) >> ) ENGINE=MyISAM DEFAULT CHARSET=utf8; >> >> -- ---------------------------- >> -- Records of FEATURE >> -- ---------------------------- >> INSERT INTO FEATURE VALUES ('1', '1', >> '台灯是人们生活中用来照明的一种家用电器。它一般分为两种，一种是立柱式的，一种是有夹子的。它的工作原理主要是把灯光集中在一小块区域内，便于工作和学习。一般台灯用的灯泡是白炽灯或者节能灯泡。 >> 有的台灯还有应急功能，用于停电时无电照明已用来应急。 ', '2012-06-01 14:17:54'); >> >> -- ---------------------------- >> -- Table structure for `ITEM` >> -- ---------------------------- >> DROP TABLE IF EXISTS `ITEM`; >> CREATE TABLE `ITEM` ( >> `ITEM_ID` char(8) NOT NULL, >> `NAME` varchar(100) DEFAULT NULL, >> `MANU` varchar(100) DEFAULT NULL, >> `WEIGHT` double DEFAULT NULL, >> `PRICE` double DEFAULT NULL, >> `POPULARITY` int(11) DEFAULT NULL, >> `INCLUDES` varchar(100) DEFAULT NULL, >> `LAST_MODIFIED` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON >> UPDATE CURRENT_TIMESTAMP, >> PRIMARY KEY (`ITEM_ID`) >> ) ENGINE=MyISAM DEFAULT CHARSET=utf8; >> >> -- ---------------------------- >> -- Records of ITEM >> -- ---------------------------- >> INSERT INTO ITEM VALUES ('1', '法国台灯A234', '台灯', '10', '100.2', '3', >> '包含灯泡啥的', '2012-06-01 14:15:58'); >> >> -- ---------------------------- >> -- Table structure for `ITEM_CATEGORY` >> -- ---------------------------- >> DROP TABLE IF EXISTS `ITEM_CATEGORY`; >> CREATE TABLE `ITEM_CATEGORY` ( >> `ITEM_CATEGORY_ID` char(8) DEFAULT NULL, >> `ITEM_ID` char(8) DEFAULT NULL, >> `CATEGORY_ID` int(11) DEFAULT NULL, >> `LAST_MODIFIED` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON >> UPDATE CURRENT_TIMESTAMP, >> KEY `FK_Reference_2` (`ITEM_ID`), >> KEY `FK_Reference_3` (`CATEGORY_ID`) >> ) ENGINE=MyISAM DEFAULT CHARSET=utf8; >> >> -- ---------------------------- >> -- Records of ITEM_CATEGORY >> -- ---------------------------- >> INSERT INTO ITEM_CATEGORY VALUES ('1', '1', '1', '2012-06-01 14:16:08'); >> >> >> <dataConfig> >> <dataSource type="JdbcDataSource" driver="com.mysql.jdbc.Driver" >> >> url="jdbc:mysql://127.0.0.1:3306/MYSOLR?useUnicode=true&characterEncoding=UTF-8" >> user="root" password="qwertyuiop" batchSize="500" /> >> >> <document datasource="JdbcDataSource" name="item_document"> >> <entity name="item" pk="ITEM_ID" query="select * from ITEM;" >> deltaQuery="select ITEM_ID from ITEM where >> LAST_MODIFIED > >> '${dataimporter.last_index_time}';"> >> <field column="ITEM_ID" name="item_id" /> >> <field column="NAME" name="name" /> >> <field column="NAME" name="nameSort" /> >> <field column="NAME" name="alphaNameSort" /> >> >> <entity name="feature" >> query="select * from FEATURE where >> ITEM_ID='${item.ITEM_ID}';" >> deltaQuery="select ITEM_ID from FEATURE where >> LAST_MODIFIED > >> '${dataimporter.last_index_time}';" >> parentDeltaQuery="select ITEM_ID from ITEM >> where >> ITEM_ID='${feature.ITEM_ID}';" pk="FEATURE_ID"> >> <field name="features" column="DESCRIPTION" /> >> </entity> >> >> <entity name="item_category" >> query="select CATEGORY_ID from ITEM_CATEGORY >> where >> ITEM_ID='${item.ITEM_ID}';" >> deltaQuery="select ITEM_ID, CATEGORY_ID from >> ITEM_CATEGORY where >> LAST_MODIFIED > '${dataimporter.last_index_time}';" >> parentDeltaQuery="select ITEM_ID from ITEM >> where >> ITEM_ID=$'{item_category.ITEM_ID}';" pk="ITEM_CATEGORY_ID"> >> <entity name="category" >> query="select DESCRIPTION from >> CATEGORY where CATEGORY_ID = >> '${item_category.CATEGORY_ID}';" >> deltaQuery="select CATEGORY_ID from >> CATEGORY where LAST_MODIFIED >>> '${dataimporter.last_index_time}';" >> parentDeltaQuery="select ITEM_ID, >> CATEGORY_ID from ITEM_CATEGORY >> where CATEGORY_ID='${category.CATEGORY_ID}';" pk="CATEGORY_ID"> >> <field column="description" >> name="cat" /> >> </entity> >> </entity> >> </entity> >> </document> >> </dataConfig> >> >> >> <requestHandler name="/dataimport" >> class="org.apache.solr.handler.dataimport.DataImportHandler"> >> <lst name="defaults"> >> <str >> name="config">/Volumes/Date/apache-tomcat-7.0.27/webapps/apache-solr-3.6.0/home/cores/cn/conf/mysql-data-config.xml</str> >> </lst> >> </requestHandler> >> >> <fieldType name="text_cn" class="solr.TextField"> >> <analyzer type="index"> >> <tokenizer >> class="org.wltea.analyzer.solr.IKTokenizerFactory" >> useSmart="false" /> >> </analyzer> >> <analyzer type="query"> >> <tokenizer >> class="org.wltea.analyzer.solr.IKTokenizerFactory" >> useSmart="false" /> >> </analyzer> >> </fieldType> >> >> <field name="text_cn" type="text_cn" indexed="true" stored="true" >> multiValued="true"/> >> >> <field name="item_id" type="string" indexed="true" stored="true" >> required="true" /> >> <field name="sku" type="text_en_splitting_tight" indexed="true" >> stored="true" omitNorms="true"/> >> <field name="name" type="text_cn" indexed="true" stored="true"/> >> <field name="alphaNameSort" type="alphaOnlySort" indexed="true" >> stored="false"/> >> <field name="manu" type="text_cn" indexed="true" stored="true" >> omitNorms="true"/> >> <field name="cat" type="string" indexed="true" stored="true" >> multiValued="true"/> >> <field name="features" type="text_cn" indexed="true" stored="true" >> multiValued="true" /> >> <field name="includes" type="text_cn" indexed="true" stored="true" >> termVectors="true" termPositions="true" termOffsets="true" /> >> >> <field name="weight" type="float" indexed="true" stored="true"/> >> <field name="price" type="float" indexed="true" stored="true"/> >> <field name="popularity" type="int" indexed="true" stored="true" /> >> <field name="inStock" type="boolean" indexed="true" stored="true" /> >> ...... -- Lance Norskog goks...@gmail.com
