Comment 704 for bug 500069

(In reply to comment #543)
> And can we find a solution that would apply in the case where the system is
> running out of free RAM and starts swapping out everything? I often
> experienced
> total unresponsiveness of both X and the consoles when a program tries to use
> more RAM than is available, and I wasn't even even able to kill the process
> manually (forced reboot). Maybe that should be considered as a pathological
> case requiring just the OOM killer to be more aggressive - I don't know.

If you have the Magic SysRq key enabled in your kernel, you could do AltGr+SysRq+F to invoke the OOM killer manually.

I do agree in principle, though, that the offending process should be denied the allocation of any additional memory before any frequently used memory-mapped pages start getting evicted from RAM.

One possible solution might be to set a threshold for the minimum number of memory-mapped pages that the kernel must allow to remain in RAM. As an example, setting such a knob to 100000 would mean that the kernel would not evict any memory-mapped pages if fewer than 100000 memory-mapped pages were resident in RAM. Assuming that the kernel uses a least-recently-used eviction policy, this would prevent the debilitating thrashing scenario that occurs when essentially all memory-mapped pages have been and continue to be evicted.