hi Shawn, first of all, thanks for your answer.
How you import simultaneously these 6 shards? 2018-04-13 19:30 GMT+02:00 Shawn Heisey <apa...@elyograg.org>: > On 4/13/2018 11:03 AM, Jesus Olivan wrote: > > thanks for your answer. It happens that when we launch full import > process > > didn't finished (we wait for more than 60 hours last time, and we > cancelled > > it, because this is not an acceptable time for us) There weren't any > errors > > in solr logfile simply because it was working fine. The problem is that > it > > lasted eternally and didn't finish. We tried it on Aurora cluster under > > AWS, and after 20 hours of work, it failed due to lack of space in Aurora > > tmp folder. > > 375 million documents importing from MySQL with one DIH import is going > to take quite a while. > > The last full rebuild I did of my main index took 21.61 hours. This is > an index where six large shards build simultaneously, using DIH, each > one having more than 30 million documents. If I were to build it as a > single 180 million document import, it would probably take 5 days, maybe > longer. > > We had another index (since retired) that had more than 400 million > total documents, built similarly with multiple shards at the same time. > The last rebuild I can remember on that index took about two days. > > Thanks, > Shawn > >
