Park, I think there is no way initial queries will be the same IF: * your index in ramfs is really in RAM * your index in regular FS is not already in RAM due to being previously cached (you *did* flush OS cache before the test, right?)
Having said that, if you update your index infrequently and make use of warm up queries and cache warming, you are likely to be very fine with the index on disk. For example, we have a customer right now that we helped a bit with performance. They also have lots of RAM, 10M docs in the index, and replicate the whole optimized index nightly. They have 2 servers, each handling about 1000 requests per minute and their average response time is under 20 ms with pre-1.4.1 Solr and lots of facets and fqs (they use Solr not only for search, but also navigation). No ramfs involved, but they have zero disk reads because the whole index is cached in memory, so things are fast. Otis ---- Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/ ----- Original Message ---- > From: Parker Johnson <parker_john...@gap.com> > To: "solr-user@lucene.apache.org" <solr-user@lucene.apache.org> > Sent: Thu, June 2, 2011 9:20:55 PM > Subject: Re: Indexes in ramdisk don't show performance improvement? > > > That¹s just the thing. Even the initial queries have similar response > times as the later ones. WEIRD! > > I was considering running from /dev/shm in production, but for slaves only > (master remains on disk). At this point though, I'm not seeing a benefit > to ramdisk so I think I'm going back to traditional disk so the indexes > stay intact after a power cycle. > > Has anyone else seen that indexes served from disk perform similarly as > indexes served from ramdisk? > > -Park > > On 6/2/11 4:15 PM, "Erick Erickson" <erickerick...@gmail.com> wrote: > > >What I expect is happening is that the Solr caches are effectively making > >the > >two tests identical, using memory to hold the vital parts of the code in > >both > >cases (after disk warming on the instance using the local disk). I > >suspect if > >you measured the first few queries (assuming no auto-warming) you'd see > >the > >local disk version be slower. > > > >Were you running these tests for curiosity or is running from /dev/shm > >something > >you're considering for production? > > > >Best > >Erick > > > >On Thu, Jun 2, 2011 at 5:47 PM, Parker Johnson <parker_john...@gap.com> > >wrote: > >> > >> Hey everyone. > >> > >> Been doing some load testing over the past few days. I've been throwing > >>a > >> good bit of load at an instance of solr and have been measuring response > >> time. We're running a variety of different keyword searches to keep > >> solr's cache on its toes. > >> > >> I'm running two exact same load testing scenarios: one with indexes > >> residing in /dev/shm and another from local disk. The indexes are about > >> 4.5GB in size. > >> > >> On both tests the response times are the same. I wasn't expecting that. > >> I do see the java heap size grow when indexes are served from disk > >>(which > >> is expected). When the indexes are served out of /dev/shm, the java > >>heap > >> stays small. > >> > >> So in general is this consistent behavior? I don't really see the > >> advantage of serving indexes from /dev/shm. When the indexes are being > >> served out of ramdisk, is the linux kernel or the memory mapper doing > >> something tricky behind the scenes to use ramdisk in lieu of the java > >>heap? > >> > >> For what it is worth, we are running x_64 rh5.4 on a 12 core 2.27Ghz > >>Xeon > >> system with 48GB ram. > >> > >> Thoughts? > >> > >> -Park > >> > >> > >> > > > > >
