On Sun, Apr 19, 2009 at 7:30 AM, Gustavo A. Lopes <galo...@mediacapital.pt>wrote:
> I'm indexing around 1 million documents of one type that requires 4 > additional queries for each document + 0,5 M documents that only require 1 > query for all. > > I'm using the data import handler from contrib with SolrWriter modified > with allowDups = true (doesn't seem to have made any difference). > > This doesn't seem to be a that many documents, however, after 21 hours, I > have only ~700 k documents of the first type indexed. The size of index is > currently 2.1 GB > > I'm noticing the initial import rate is relatively high, such as all the > documents of first type would be indexed in less than 6 hours if it were > maintained. When the number of documents already imported rises, the import > rate falls significatively. > > Does anyone have any suggestions on how to speed up full imports? What is > the bottleneck? I will probably have to make some changes to schema over the > next days that will require new imports. > Further to Otis's suggestions -- Do you have autoCommit+autowarming turned on? Maybe that is the cause of the slowdown as the import progresses? -- Regards, Shalin Shekhar Mangar.
