> > Hello all, > > Greetings!!, > > Environment: > I have been working on a solr instance with requirement of ~ 40 columns and > 50000 records of data.i am using data import handler to configure the data > load and indexing..also I use regex transformer to read the data file(csv) > and apply regex to parse individual columns.note that all the columns are set > with index option as true .configuration is given below > > Issue: > While loading data with 15 columns and 50000 records hardly takes around 2 > mins, increasing the number of columns increase the data load time very > high..for e.g) 24 columns with 50000 records takes nearly 30 mins.i don't see > any increased memory consumption with solr admin dashboard .on the other > hand,using the update csv handler takes hardly around one min to load the > complete data > > Please provide Any suggestions/alternatives to improve the performance > (.please note I couldn't use csv handler since with csv handler, there's not > an query able option to know the status and the need to delete the index > every time before data load ) > > Configuration details > > Solr index size 14 MB( when used update csv handler) > Data file size 5 MB > > > Data- config.xml > > Datasource type - Filedatasource encoding UTF8 > > Transformer - > regextransformer > > > Regex ="^(.*),(.*)............" > > groupNames = "field1,field2,......field40" > > > > Regards > Ibrahim > >
