Sending 32K docs at a time is a bit of overkill, I usually stay around 1,000.
I agree with Michael, it's rarely a Good Thing to do the commit in
line, except (perhaps) at the very end. Just let the autocommit
settings take care of it for you.
Your log shows numbers of "overlapping ondeck searcher
Two possibly unrelated things:
1. Don't commit until the end.
2. Consider not optimizing at all.
You might want to look at your autocommit settings in your solrconfig.xml.
You probably want soft commits set at something north of 10 seconds, and
hard commits set to openSearcher=false with a maxTi
