Hi, It seems that both of you simply don't understand what's happening in your operating system kernel. Please read the blog post again!
> It happens in 3.6, for this reasons I thought of moving to solandra. > If I do a commit, the all documents are persisted with out any issues. > There is no issues in terms of any functionality, but only this happens is > increase in physical RAM, goes higher and higher and stop at maximum and it > never comes down. This is perfectly fine in Windows and Linux (and any other operating system). If an operating system would not use *all* available physical memory it would waste costly hardware resources. Why not use resources that are unused otherwise? As said before: O/S kernel uses *all* available physical RAM for caching file system accesses. The memory used for that is always reported as not free, because it is used (very simple, right?). But if some other application wants to use it, its free for malloc(), so it is not permanently occupied. That's always that case, using MMapDirectory or not (same for SimpleFSDirectory or NIOFSDirectory). Of course, when you freshly booted your kernel, it reports free memory, but definitely not on a server running 24/7 since weeks. For all people who don't want to understand that, here is the easy explanation page: http://www.linuxatemyram.com/ > > > all my physical memory say its 100 percentage used(windows). On deep > > > investigation found that mmap is not releasing os files handles. Do > > > you find this behaviour? One comment: The file handles are not freed as long as the index is open. Used file handles have nothing to do with memory mapping, it's completely unrelated to each other. Uwe > On Sun, Jul 22, 2012 at 3:38 AM, Lance Norskog <goks...@gmail.com> wrote: > > > Interesting. Which version of Solr is this? What happens if you do a > > commit? > > > > On Sat, Jul 21, 2012 at 8:01 AM, geetha anjali > > <anjaliprabh...@gmail.com> > > wrote: > > > Hi uwe, > > > Great to know. We have files indexing 10000/min. After 30 mins I see > > > all my physical memory say its 100 percentage used(windows). On deep > > > investigation found that mmap is not releasing os files handles. Do > > > you find this behaviour? > > > > > > Thanks > > > > > > On 20 Jul 2012 14:04, "Uwe Schindler" <u...@thetaphi.de> wrote: > > > > > > Hi Bill, > > > > > > MMapDirectory uses the file system cache of your operating system, > > > which > > has > > > following consequences: In Linux, top & free should normally report > > > only > > > *few* free memory, because the O/S uses all memory not allocated by > > > applications to cache disk I/O (and shows it as allocated, so having > > > 0% > > free > > > memory is just normal on Linux and also Windows). If you have other > > > applications or Lucene/Solr itself that allocate lot's of heap space > > > or > > > malloc() a lot, then you are reducing free physical memory, so > > > reducing > > fs > > > cache. This depends also on your swappiness parameter (if swappiness > > > is higher, inactive processes are swapped out easier, default is 60% > > > on > > linux - > > > freeing more space for FS cache - the backside is of course that > > > maybe in-memory structures of Lucene and other applications get pages > out). > > > > > > You will only see no paging at all if all memory allocated all > > applications > > > + all mmapped files fit into memory. But paging in/out the mmapped > > > + Lucene > > > index is muuuuuch cheaper than using SimpleFSDirectory or > > NIOFSDirectory. If > > > you use SimpleFS or NIO and your index is not in FS cache, it will > > > also > > read > > > it from physical disk again, so where is the difference. Paging is > > actually > > > cheaper as no syscalls are involved. > > > > > > If you want as much as possible of your index in physical RAM, copy > > > it to /dev/null regularily and buy more RUM :-) > > > > > > > > > ----- > > > Uwe Schindler > > > H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de > > > eMail: uwe@thetaphi... > > > > > >> From: Bill Bell [mailto:billnb...@gmail.com] > > >> Sent: Friday, July 20, 2012 5:17 AM > > >> Subject: Re: ... > > >> s=op using it? The least used memory will be removed from the OS > > >> automaticall=? Isee some paging. Wouldn't paging slow down the > querying? > > > > > >> > > >> My index is 10gb and every 8 hours we get most of it in shared memory. > > The > > >> m=mory is 99 percent used, and that does not leave any room for > > >> other > > > apps. = > > > > > >> Other implications? > > >> > > >> Sent from my mobile device > > >> 720-256-8076 > > >> > > >> On Jul 19, 2012, at 9:49 A... > > >> H=ap space or free system RAM: > > > > > >> > > > >> > > > http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.htm > > >> > l > > >> > > > >> > Uwe > > >> >... > > >> >> use i= since you might run out of memory on large indexes right? > > > > > >> >> > > >> >> Here is how I got iSimpleFSDirectoryFactory to work. Just set - > > >> >> Dsolr.directoryFactor... > > >> >> set it=all up with a helper in solrconfig.xml... > > > > > >> >> > > >> >> if (Constants.WINDOWS) { > > >> >> if (MMapDirectory.UNMAP_SUPPORTED && Constants.JRE_IS_64... > > > > > > > > -- > > Lance Norskog > > goks...@gmail.com > >
