On 4/13/2016 4:34 AM, Bastien Latard - MDPI AG wrote: > Thank you all again for your good and detailed answer. > I will combine all of them to try to build a better environment. > > *Just a last question...* > /I don't remember exactly when I needed to increase the java heap.../ > /but is it possible that this was for the DataImport.../ > > *Would the DIH work if it cannot "load" the temporary index into the > java heap in the full-index mode?* > I thought that's why I needed to increase this value...but I might be > confused!
The default behavior on many JDBC drivers is to load the *entire* SQL result into memory *before* sending those results to the requesting application. This is the way the MySQL driver behaves by default, and the way that older versions of the Microsoft driver for SQL Server behave by default. There should be a way to tell the JDBC driver to stream the results back instead of loading them into memory. For MySQL, you just have to set the batchSize parameter in the DIH config to -1, which causes the underlying code to do "setFetchSize(Integer.MIN_VALUE)". For SQL Server, you need a recent version of the driver, where they changed the default behavior. For other databases, you may need a JDBC url parameter. Thanks, Shawn
