Tushar: Is it necessary to do the optimize on each iteration? When you run an optimize, the entire index is rewritten. Thus each index file can have at most one hard link and each snapshot will consume the full amount of space on your disk.
Asir On Thu, Jul 9, 2009 at 3:26 AM, tushar kapoor < tushar_kapoor...@rediffmail.com> wrote: > > What I gather from this discussion is - > > 1. Snapshots are always hard links and not actual files so they cannot > possibly consume the same amount of space. > 2. Snapshots contain hard links to existing docs + delta docs. > > We are facing a situation wherein the snapshot occupies the same space as > the actual indexes thus violating the first point. > We have a batch processing scheme for refreshing indexes. the steps we > follow are - > > 1. Delete 200 documents in one go. > 2. Do an optimize. > 3. Create the 200 documents deleted earlier. > 4. Do a commit. > > This process continues for around 160,000 documents i.e. 800 times and by > the end of it we have 800 snapshots. > > The size of actual indexes is 200 Mb and remarkably all the 800 snapshots > are of size around 200 Mb each. In effect this process consumes around 160 > Gb space on our disks. This is causing a lot of pain right now. > > My concern are - Is our understanding of the snapshooter correct ? Should > this massive space consumption be happening at all ? Are we missing > something critical ? > > Regards, > Tushar. > > Shalin Shekhar Mangar wrote: > > > > On Sat, Apr 18, 2009 at 1:06 PM, Koushik Mitra > > <koushik_mi...@infosys.com>wrote: > > > >> Ok.... > >> > >> If these are hard links, then where does the index data get stored? > Those > >> must be getting stored somewhere in the file system. > >> > > > > Yes, of course they are stored on disk. The hard links are created from > > the > > actual files inside the index directory. When those older files are > > deleted > > by Solr, they are still left on the disk if at least one hard link to > that > > file exists. If you are looking for how to clean old snapshots, you could > > use the snapcleaner script. > > > > Is that what you wanted to do? > > > > -- > > Regards, > > Shalin Shekhar Mangar. > > > > > > -- > View this message in context: > http://www.nabble.com/Create-incremental-snapshot-tp23109877p24405434.html > Sent from the Solr - User mailing list archive at Nabble.com. > > -- Asif Rahman Lead Engineer - NewsCred a...@newscred.com http://platform.newscred.com
