On Mon, 26 Apr 2010 11:16:32 +0100 Gavin Wraith <[email protected]> wrote:
> I am not au fait with RISC OS's memory-management, but I would like > to know more about it (an article in Archive one day?). I take the > point that till now most tasks have been insufficiently complex to > show up its defects. The memory-management system I am frequently > using is that of Lua, which only handles the memory that the RISC OS > task- manager assigns to it. I have often wondered, perhaps naively, > if there were a way of exploiting it more widely in RISC OS. Perhaps > the RISC OS task-manager's memory-management is bound too tightly > to the hardware for something as generic as Lua's system to be > exploited more widely? Hi Gavin, I've CCed the netsurf-dev list with this, as it might be interesting for the non-RISC OS developers, as an insight into restrictions we have to keep in mind if we want to continue supporting RISC OS. I hope you don't mind. The issue is that RISC OS's memory management system is both complex, and old-fashioned. It makes mistakes that were solved in UNIX in the 1970s. It has three main "arenas" in which memory can be allocated; the WIMP slot, the Relocatable Module Area, and the Dynamic Area area. (UNIX has two; it traditionally chops the memory map in to [commonly 3GB/1GB]. You have user process [3GB], and kernel memory [1GB].) The main issue is that memory allocations under RISC OS are always contiguous. The secondary issue is that if you want to store more than 28MB of data (and/or have good task swapping performance), you need to store your data in Dynamic Areas that use a single global pool of virtual address space, and that space for the DA must be reserved when it created. The first problem means that should you have, say, 4MB of allocated data, and you free the middle 2MB, that 2MB of memory cannot be returned to the operating system. The memory literally becomes fragmented. While NetSurf may have DAs amounting up to 100MB, it may actually only have 20MB of data in them. But it has no way of returning that free memory back to the OS. Historically, RISC OS applications have used "sliding heaps", where you can move blocks of data around to free up space at the end, allowing you to shrink the DA. Unfortunately, this means that all data you store must be accessed via an "anchor", that never moves. (ie, when the memory heap is reshuffled, it updates all the anchors.). While this solves one problem, it causes several others: All data access has to be indirected, meaning more reads from memory for the same job, as well as reducing the effectiveness of the memory cache. Secondly, it greatly complicates the application, making it more difficult to debug. The second problem causes the "NetSurf is running out of memory" problem, when there is clearly free memory. The problem isn't that it's running out of memory, but it's running out of memory address space. When you create a DA, you tell the OS how large it is permitted to grow to. RISC OS keeps all DAs in a fixed-size region of memory map. This means that if the memory map is completely allocated, new DAs cannot be created, because there is nowhere they would fit. There are solutions to these problems (short of rewriting RISC OS to have a modern memory system). The first problem could be significantly mitigated by using a feature available in RISC OS 3.7 onwards (I believe) called "Sparse Dynamic Areas". These allow you to punch holes in DAs such that RISC OS can use the pages of RAM that represent the hole elsewhere. This means you could have a huge DA with only a tiny amount of data in them. You'll still have problem two, which is running out of address space, but it does mean the memory would be available for other applications to use. This would require massive re-engineering of UnixLib's memory allocator, and is not a trivial job by any stretch of the imagination. The solution to the second problem has already been defined by RISC OS's reference manuals, but has never been implemented. There is a bit you can set in the flags field you pass when creating a dynamic area that means "This DA is local to this WIMP task". It allows the DAs to be swapped out at the same time as the WIMP task they are associated with. This means that the pages of the memory map they consume can be used by other WIMP tasks too, and the memory is not visible to the other WIMP tasks. Unfortunately, as I said, RISC OS ignores this flag, and hardly any applications set it. Additionally, for such a scheme to work anyway, some of the memory map would have to be reserved for such DAs, meaning that its implementation would actually make matters worse for existing applications. This is just two of the problems; there are others. Hope this clears things up a bit. B.
