David Nghiem <[email protected]> wrote:
> On Tue, Mar 22, 2011 at 2:39 PM, Eric Wong <[email protected]> wrote:
> > No Errno::EBADF errors at all?
>
> I didn't check before or after the block of Errno::IOError errors,
> there were enough IOError logs to fill up 40gb of disk. I'll capture
> the entire deleted log next time.
Yikes. The first few lines of error messages should be enough.
> >> ruby 15209 zendesk 1w REG 252,0 0
> >> 3015360 /data/zendesk/log/unicorn.stdout.log
> >> ruby 15209 zendesk 2w REG 252,0 24585
> >> 3015203 /data/zendesk/log/unicorn.stderr.log-20110110 (deleted)
> >> ruby 15209 zendesk 3w REG 252,0 61617590073
> >> 3015097 /data/zendesk/log/unicorn.log-20110110 (deleted)
> >
> > Permissions issue during log rotation? Are you doing "killall -USR1
> > unicorn" by any chance or sending a bunch of USR1 signals faster than
> > unicorn can process them?
> >
> > Try just sending one USR1 to the master process and letting the master
> > signal the workers.
>
> Are the IOError exceptions and failure to rotate logs related? My
> wild guess was this specific worker's unix socket got corrupted
> somehow.
There's a good possibility. The USR1 signal closes a pipe we normally
select() on. The closed pipe triggers a wakeup from select(), which
triggers the log reopening (and reopening of the pipe).
Do you see: "worker=$NUM done reopening logs" messages in your
logs?
Another explanation is somehow you have a library that does something
along the lines of:
ObjectSpace.each_object(IO) { |io| io.close unless io.closed? }
--
Eric Wong
_______________________________________________
Unicorn mailing list - [email protected]
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
