Package: icecast2 Version: 2.3.2-2 Severity: important
We are using icecast2 as an on-demand streaming server offering about 120 different mp3 streams. Streams are changing very frequently. Ices is used as streaming source client. Additionally apache2 and mysql5-server are installed and running, providing a HTTP based API to request on-demand streams. Hardware: Intel i7 CPU, 4 phys + 4 virt cores, 8 GB RAM + 4 GB swap After some hours in production the server became unresponsive for several minutes, showing a system load of about 100. All memory including the whole swap space was in use. /var/log/syslog: Jan 21 09:23:12 istream1 kernel: [168616.389704] icecast2 invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0 Jan 21 09:23:12 istream1 kernel: [168616.389750] Pid: 6192, comm: icecast2 Not tainted 2.6.26-2-amd64 #1 Jan 21 09:23:12 istream1 kernel: [168616.389775] Jan 21 09:23:12 istream1 kernel: [168616.389775] Call Trace: Jan 21 09:23:12 istream1 kernel: [168616.389826] [<ffffffff802738a0>] oom_kill_process+0x57/0x1dc Jan 21 09:23:12 istream1 kernel: [168616.389855] [<ffffffff8023b531>] __capable+0x9/0x1c Jan 21 09:23:12 istream1 kernel: [168616.389880] [<ffffffff80273bcb>] badness+0x188/0x1c7 Jan 21 09:23:12 istream1 kernel: [168616.389906] [<ffffffff80273dff>] out_of_memory+0x1f5/0x28e Jan 21 09:23:12 istream1 kernel: [168616.389935] [<ffffffff80276b50>] __alloc_pages_internal+0x31d/0x3bf Jan 21 09:23:12 istream1 kernel: [168616.389967] [<ffffffff80278806>] __do_page_cache_readahead+0x79/0x183 Jan 21 09:23:12 istream1 kernel: [168616.389998] [<ffffffff802730b5>] filemap_fault+0x15d/0x33c Jan 21 09:23:12 istream1 kernel: [168616.390027] [<ffffffff8027e530>] __do_fault+0x50/0x3e6 Jan 21 09:23:12 istream1 kernel: [168616.390054] [<ffffffff8024fb3f>] futex_wake+0x74/0xfa Jan 21 09:23:12 istream1 kernel: [168616.390081] [<ffffffff80281893>] handle_mm_fault+0x3f4/0x867 Jan 21 09:23:12 istream1 kernel: [168616.390107] [<ffffffff80250734>] do_futex+0xa6/0x777 Jan 21 09:23:12 istream1 kernel: [168616.390133] [<ffffffff803b14af>] sys_recvfrom+0xff/0x119 Jan 21 09:23:12 istream1 kernel: [168616.390160] [<ffffffff8024612b>] bit_waitqueue+0x10/0x97 Jan 21 09:23:12 istream1 kernel: [168616.390188] [<ffffffff80221fbc>] do_page_fault+0x5d8/0x9c8 Jan 21 09:23:12 istream1 kernel: [168616.390219] [<ffffffff8042a6a9>] error_exit+0x0/0x60 Jan 21 09:23:12 istream1 kernel: [168616.390249] Jan 21 09:23:12 istream1 kernel: [168616.390266] Mem-info: Jan 21 09:23:12 istream1 kernel: [168616.390285] Node 0 DMA per-cpu: Jan 21 09:23:12 istream1 kernel: [168616.390306] CPU 0: hi: 0, btch: 1 usd: 0 Jan 21 09:23:12 istream1 kernel: [168616.390329] CPU 1: hi: 0, btch: 1 usd: 0 Jan 21 09:23:12 istream1 kernel: [168616.390352] CPU 2: hi: 0, btch: 1 usd: 0 Jan 21 09:23:12 istream1 kernel: [168616.390375] CPU 3: hi: 0, btch: 1 usd: 0 Jan 21 09:23:12 istream1 kernel: [168616.390398] CPU 4: hi: 0, btch: 1 usd: 0 Jan 21 09:23:12 istream1 kernel: [168616.390421] CPU 5: hi: 0, btch: 1 usd: 0 Jan 21 09:23:12 istream1 kernel: [168616.393041] CPU 6: hi: 0, btch: 1 usd: 0 Jan 21 09:23:12 istream1 kernel: [168616.393064] CPU 7: hi: 0, btch: 1 usd: 0 Jan 21 09:23:12 istream1 kernel: [168616.393086] Node 0 DMA32 per-cpu: Jan 21 09:23:12 istream1 kernel: [168616.393108] CPU 0: hi: 186, btch: 31 usd: 91 Jan 21 09:23:12 istream1 kernel: [168616.393131] CPU 1: hi: 186, btch: 31 usd: 27 Jan 21 09:23:12 istream1 kernel: [168616.393154] CPU 2: hi: 186, btch: 31 usd: 48 Jan 21 09:23:12 istream1 kernel: [168616.393177] CPU 3: hi: 186, btch: 31 usd: 105 Jan 21 09:23:12 istream1 kernel: [168616.393200] CPU 4: hi: 186, btch: 31 usd: 148 Jan 21 09:23:12 istream1 kernel: [168616.393223] CPU 5: hi: 186, btch: 31 usd: 80 Jan 21 09:23:12 istream1 kernel: [168616.393246] CPU 6: hi: 186, btch: 31 usd: 181 Jan 21 09:23:12 istream1 kernel: [168616.393312] CPU 7: hi: 186, btch: 31 usd: 70 Jan 21 09:23:12 istream1 kernel: [168616.393335] Node 0 Normal per-cpu: Jan 21 09:23:12 istream1 kernel: [168616.393335] Node 0 Normal per-cpu: Jan 21 09:23:12 istream1 kernel: [168616.393356] CPU 0: hi: 186, btch: 31 usd: 99 Jan 21 09:23:12 istream1 kernel: [168616.393379] CPU 1: hi: 186, btch: 31 usd: 108 Jan 21 09:23:12 istream1 kernel: [168616.393402] CPU 2: hi: 186, btch: 31 usd: 159 Jan 21 09:23:12 istream1 kernel: [168616.393425] CPU 3: hi: 186, btch: 31 usd: 185 Jan 21 09:23:12 istream1 kernel: [168616.393447] CPU 4: hi: 186, btch: 31 usd: 178 Jan 21 09:23:12 istream1 kernel: [168616.393470] CPU 5: hi: 186, btch: 31 usd: 106 Jan 21 09:23:12 istream1 kernel: [168616.393493] CPU 6: hi: 186, btch: 31 usd: 169 Jan 21 09:23:12 istream1 kernel: [168616.393516] CPU 7: hi: 186, btch: 31 usd: 183 Jan 21 09:23:12 istream1 kernel: [168616.395622] Active:1041731 inactive:971416 dirty:0 writeback:0 unstable:0 Jan 21 09:23:12 istream1 kernel: [168616.395623] free:10818 slab:4404 mapped:18 pagetables:8080 bounce:0 Jan 21 09:23:13 istream1 kernel: [168616.395673] Node 0 DMA free:11680kB min:12kB low:12kB high:16kB active:0kB inactive:0kB present:10756kB pages_scanned:0 all_unreclaimable? yes Jan 21 09:23:13 istream1 kernel: [168616.395724] lowmem_reserve[]: 0 2991 8041 8041 Jan 21 09:23:13 istream1 kernel: [168616.395754] Node 0 DMA32 free:24264kB min:4264kB low:5328kB high:6396kB active:1450652kB inactive:1447976kB present:3063520kB pages_scanned:5038855 all_unreclaimable? yes Jan 21 09:23:13 istream1 kernel: [168616.395822] lowmem_reserve[]: 0 0 5050 5050 Jan 21 09:23:13 istream1 kernel: [168616.395850] Node 0 Normal free:7328kB min:7200kB low:9000kB high:10800kB active:2716272kB inactive:2437688kB present:5171200kB pages_scanned:8942966 all_unreclaimable? yes Jan 21 09:23:13 istream1 kernel: [168616.395919] lowmem_reserve[]: 0 0 0 0 Jan 21 09:23:13 istream1 kernel: [168616.395946] Node 0 DMA: 6*4kB 7*8kB 5*16kB 6*32kB 5*64kB 2*128kB 2*256kB 2*512kB 1*1024kB 0*2048kB 2*4096kB = 11680kB Jan 21 09:23:13 istream1 kernel: [168616.396010] Node 0 DMA32: 236*4kB 157*8kB 311*16kB 50*32kB 34*64kB 24*128kB 12*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 24264kB Jan 21 09:23:13 istream1 kernel: [168616.396076] Node 0 Normal: 148*4kB 4*8kB 5*16kB 1*32kB 3*64kB 2*128kB 2*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 7328kB Jan 21 09:23:13 istream1 kernel: [168616.396141] 118 total pagecache pages Jan 21 09:23:13 istream1 kernel: [168616.396162] Swap cache: add 3615167, delete 3615164, find 932862/944525 Jan 21 09:23:13 istream1 kernel: [168616.396188] Free swap = 0kB Jan 21 09:23:13 istream1 kernel: [168616.396207] Total swap = 4200888kB Jan 21 09:23:13 istream1 kernel: [168616.421112] 2097152 pages of RAM Jan 21 09:23:13 istream1 kernel: [168616.421112] 48651 reserved pages Jan 21 09:23:13 istream1 kernel: [168616.421112] 3168 pages shared Jan 21 09:23:13 istream1 kernel: [168616.421112] 3 pages swap cached Jan 21 09:23:13 istream1 kernel: [168616.421112] Out of memory: kill process 5232 (icecast2) score 3087713 or a child Jan 21 09:23:13 istream1 kernel: [168616.421112] Killed process 5232 (icecast2) This happened several times. After googling i found this thread "http://icecast.imux.net/viewtopic.php?t=7001"; which describes the same problem. I started then to reproduce the scenario on an identical server using some test scripts to trigger a high load. "top" showed the icecast2 process slowly eating up the mem. I'm sure that the very frequent mounting and unmounting of streaming sources (as we do in our on-demand setup) accelerates the allocation and not deallocation of memory by icecast2, but the forum post mentioned above uses a somewhat lowlevel setup with only a few "static" streams, resulting in OOM too. Icecast2 compiled from source (SVN, trunk version, http://svn.xiph.org/icecast/trunk/icecast) at the same machine (all required libs, headers etc. installed from lenny packages) runs rock solid, utilising just about 320M while serving 120 streams. Cheers Harald -- System Information: Debian Release: 5.0.3 APT prefers stable APT policy: (500, 'stable') Architecture: amd64 (x86_64) Kernel: Linux 2.6.26-2-amd64 (SMP w/8 CPU cores) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/bash -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
