May be related to https://issues.apache.org/jira/browse/LUCENE-3418 which does ensure things are really written; if you do commit very often, you may see this sort of performance loss (at least me did in my junit test where i do commit very often and 3.3 switch to 3.4 really hurts here at test time - but its ok for tests to take longer because the real app does use batch commits).
You can try to use solr 3.3.x and see how it works against 3.4.0 (which does include the fix for 3418) if you want to find out if this is related to #3418. HTH Torsten
smime.p7s
Description: S/MIME cryptographic signature