On 9/12/13 3:28 PM, Toke Eskildsen wrote:
On Thu, 2013-09-12 at 14:48 +0200, Per Steffensen wrote:
Actually some months back I made PoC of a FieldCache that could expand
beyond the heap. Basically imagine a FieldCache with room for
"unlimited" data-arrays, that just behind the scenes goes to
memory-mapped files when there is no more room on heap.
That sounds a lot like disk-based DocValues.
He he
But that solution will also have the "running out of swap space"-problems.
Not really. Memory mapping works like the disk cache: There is no
requirement that a certain amount of physical memory needs to be
available, it just takes what it can get. If there are not a lot of
physical memory, it will require a lot of storage access, but it will
not over-allocate swap space.
That was also my impression, but during the work, I experienced some
problems around swap space, but I do not remember exactly what I saw,
and therefore how I concluded that everything in mm-files actually have
to fit in physical mem + swap. I might very well have been wrong in that
conclusion
It seems that different setups vary quite a lot in this area and some
systems are prone to aggressive use of the swap file, which can severely
harm responsiveness of applications with out-swapped data.
However, this should still not result in any OOM's, as the system can
always discard some of the memory mapped data if it needs more physical
memory.
I saw no OOMs
- Toke Eskildsen, State and University Library, Denmark
