This looks like a problem with your select statement returning too many rows. I doubt it has to do with the multiValued field, I don’t think DIH is getting to the point where it even tries to create a SolrInputDocument.
Depending on the driver, there are ways to limit the number of rows returned at a time, depends on the individual driver you’re using. I know of at least one driver that _defaults_ to returning all the rows in a single batch. Check the docs for the particular driver you’re using. If this is the problem, the ram buffer size is irrelevant. You could try it and specify a much smaller subset of docs to be returned from your SQL statement as a sanity test. Best, Erick > On Jun 26, 2019, at 7:19 PM, Robert Dadzie <[email protected]> wrote: > > Hi All, > > > I'm trygin to use DIH to import about 150k documents to Solr. One of the > multi-valued fields I need to import stores about 1500 unique ID per record. > I tried increasing the 'ramBufferSizeMB' setting but that didn't help. I get > this ArrayIndexOutOfBoundsException error and I can't make any sense of it. > An extract of the error log is below, any assistance would be greatly > appreciated. > > > o.a.s.h.d.DocBuilder Exception while processing: matter document : > SolrInputDocument(fields: > []):org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to > execute query: SELECT * FROM myView WITH (NOLOCK) ORDER BY viewId Processing > Document # 1 > at > org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:69) > at > org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:327) > at > org.apache.solr.handler.dataimport.JdbcDataSource.createResultSetIterator(JdbcDataSource.java:288) > at > org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:283) > at > org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.java:52) > at > org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityProcessor.java:59) > at > org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73) > at > org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:267) > at > org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:476) > at > org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:415) > at > org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:330) > at > org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:233) > at > org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:424) > at > org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:483) > at > org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:466) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.ArrayIndexOutOfBoundsException > at java.net.SocketInputStream.socketRead0(Native Method) > at > java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > at java.net.SocketInputStream.read(SocketInputStream.java:170) > at java.net.SocketInputStream.read(SocketInputStream.java:141) > at com.jnetdirect.jsql.DBComms.receive(DBComms.java:777) > at com.jnetdirect.jsql.IOBuffer.sendCommand(IOBuffer.java:248) > at > com.jnetdirect.jsql.JSQLStatement.sendExecute(JSQLStatement.java:2478) > at > com.jnetdirect.jsql.JSQLStatement.doExecute(JSQLStatement.java:2447) > at > com.jnetdirect.jsql.JSQLStatement.execute(JSQLStatement.java:2433) > at > org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.executeStatement(JdbcDataSource.java:349) > at > org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(JdbcDataSource.java:321) > ... 14 more > > Thanks, > Robert Dadzie > > > ________________________________ > > If you wish to view the CPA Global group email disclaimer, please click > here<https://www.cpaglobal.com/Privacy/CPAGlobalEmailDisclaimer/> > ________________________________
