Gora, Our environment, currently under development, is very nearly the exact same thing as yours. My DB is currently only about 10GB, but likely to grow. We also use Solr as primary repository (store all fields there), but use the DB as a back up when Full Import is needed. Delta imports aren't that bad, except when one of our larger data feeds comes in once a month. That is a very large delta import and hits some of the same issues as a full import.
I'm still trying out different architectures to deal with this. I've tried doing a Bulk Copy from the DB to some flat files and importing from there. File handles seem to be more stable than database connections. But it brings it's own issues to the party. I'm also currently looking at using queuing (either MSMQ or Amazons Simple Queue service) so the database piece isn't used for 20 hours, but gets it's part over fairly quickly. I haven't done this using DataImportHandler however, not sure yet how, so I'm writing my own Import manager. I know this isn't a solve, but maybe some other ideas you can consider. As to the GData handler and response writer. I would be very interested in OData versions, which wouldn't be too much of a stretch from GData to deal with. Would you be moving in that direction later? Or if you put your contrib out there could someone else (maybe me if time allows) be able to take it there? That would be a great edition for our work in a few months. Good luck, and I'd love to keep in touch about your solutions, I'm sure I could get some great ideas from them for our own work. Ken -- View this message in context: http://lucene.472066.n3.nabble.com/Database-connections-during-data-import-tp956325p958071.html Sent from the Solr - User mailing list archive at Nabble.com.
