On Mon, Oct 15, 2012 at 5:52 PM, Jason Matthews <[email protected]> wrote:
> > > -----Original Message----- > From: Brendan Gregg [mailto:[email protected]] > > > > A full "echo ::kmastat | mdb -k" before reboot may show where the > > memory is or isn't. > > I am not a point where I would normally reboot it, but we are pretty well > deep into memory utilization for the size of the apps that run... > > On the proxy tier, it seems that the DCE cache is way out of out of line. > What are the tunables here? > This sounds like the DCE cache cleanup issue: http://smartos.org/2012/02/28/using-flamegraph-to-solve-ip-scaling-issue-dce/ A "netstat -dn | wc -l" should show how many entries are in the cache. Brendan > > > Thanks, > j. > > > Here is a dump from a box on the proxy tier... > > root@www003:~# echo ::kmastat |mdb -k |sort -nrk 5 |head > kmem_va_4096 4096 2964721 2964736 12143558656B 3023570 > 0 > dce_cache 152 73389856 73389862 11561725952B 73389938 > 0 > zio_data_buf_131072 131072 43616 43644 5720506368B 893451 0 > kmem_va_16384 16384 51737 51744 847773696B 51969 0 > zio_buf_16384 16384 51386 51475 843366400B 5324046 0 > tcp_conn_cache 1808 17352 88512 181272576B 1597914226 0 > kmem_va_8192 8192 22097 22112 181141504B 37356 0 > streams_dblk_1040 1152 42 106253 124346368B 1582704292 0 > kmem_bufctl_cache 24 3527424 3527541 86519808B 3750227 > 0 > arc_buf_hdr_t 176 345516 345576 64339968B 1199529 0 > root@www003:~# echo ::memstat |mdb -k > Page Summary Pages MB %Tot > ------------ ---------------- ---------------- ---- > Kernel 3721822 14538 44% > ZFS File Data 1448069 5656 17% > Anon 84512 330 1% > Exec and libs 5085 19 0% > Page cache 15805 61 0% > Free (cachelist) 22861 89 0% > Free (freelist) 3072717 12002 37% > > Total 8370871 32698 > Physical 8370870 32698 > > -- Brendan Gregg, Joyent http://dtrace.org/blogs/brendan _______________________________________________ OpenIndiana-discuss mailing list [email protected] http://openindiana.org/mailman/listinfo/openindiana-discuss
