Jason Matthews wrote: > > So I have determined the trigger event for the "resets" in the response > times. It appears that there is some sort of memory leak in the kernel. When > memory utilization gets very high, I am not sure how high but it is around a > few hundred bytes on the freelists with zfs data near zero, the reset cycles > start on the vnics.
If it really is a memory leak (as opposed to just a lot of memory legitimately in use), then it should be possible (albeit slightly painful) to find out where it's going. Putting "set kmem_flags=0xf" in /etc/system will cause the kernel memory allocator to keep track of allocations. You can then panic the system (as in "reboot -d"), get the kernel dump up in mdb, and use "::findleaks" to discover what's leaking memory. It's possible that this is related to zero-copy issues that cause dblks to get held down for a long time. I think there are tunables that disable the zero-copy optimizations, but I don't remember what they are. At least with some of the bugs I've seen historically in this area, the toxic behavior is to have some application that has a socket or DLPI stream open, but that doesn't bother reading from the socket. A large number of buffers can accumulate on an unread stream, and if enough of them get stuck there, you can end up with a pretty sick system. It's also possible that the causation runs the other way -- that whatever causes the problem to happen (a packet storm of some sort?) also happens to cause a large amount of kernel memory to be allocated. > I suspect excessive memory allocation and eventual starvation have a lot to > do with the weird symptoms. I wonder if the memory leak is an independent > process from the network issue or if they are directly related in some way. It's possible that they're related. It sounds to me like this is a pretty good clue. -- James Carlson 42.703N 71.076W <[email protected]> _______________________________________________ OpenIndiana-discuss mailing list [email protected] http://openindiana.org/mailman/listinfo/openindiana-discuss
