On 2/27/15 12:10 PM, Dave Anderson wrote:
>
> This issue was first reported with respect to the crash utility,
> which is an interactive program that uses the readline library.
>
> The problem occurs only if the crash utility is run from within
> an executable bash script, i.e., like so:
>
> $ cat doit
> crash
> $
>
> If crash is invoked as above, the crash utility does its initialization
> and eventually calls readline(). Then, if CTRL-z is entered, the parent
> bash shell itself is blocked, but the crash utility spins at 100% cpu usage.
> Debugging it shows that the crash utility is stuck spinning in the readline
> libary's _set_tty_settings() function, where the tcsetattr() call repeatedly
> fails with an EINTR, where _rl_caught_signal contains SIGTTOU.
>
> But taking the crash utility out of the picture, I can reproduce it with
> readline-6.3.tar.gz, where I simply build it with "configure; make", then
> go into the examples subdirectory, and enter "make". If I then put the
> simple "rl" command in script file, and do the same thing, this happens:
>
> $ cat doit
> ./rl
> $ ./doit
> readline$ ^Z
> [1]+ Stopped ./doit
> $
> $ top
> top - 12:02:33 up 23:12, 5 users, load average: 0.37, 0.09, 0.04
> Tasks: 159 total, 2 running, 154 sleeping, 3 stopped, 0 zombie
> Cpu(s): 3.4%us, 21.6%sy, 0.0%ni, 75.0%id, 0.0%wa, 0.0%hi, 0.0%si,
> 0.0%st
> Mem: 3917056k total, 3709052k used, 208004k free, 88732k buffers
> Swap: 4063228k total, 0k used, 4063228k free, 3049316k cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
> 12336 root 20 0 100m 1016 788 R 100.0 0.0 1:12.13 rl
> 1 root 20 0 19356 1532 1216 S 0.0 0.0 0:02.89 init
> 2 root 20 0 0 0 0 S 0.0 0.0 0:00.02 kthreadd
> 3 root RT 0 0 0 0 S 0.0 0.0 0:00.19 migration/0
> ...
>
>
> If I attach gdb to the rl process above, it shows the same ultimate trace as
> the spinning crash utility does:
>
> # gdb -p 12336
> GNU gdb (GDB) Red Hat Enterprise Linux (7.2-75.el6)
> Copyright (C) 2010 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>.
> Attaching to process 12336
> Reading symbols from /root/readline-6.3/examples/rl...done.
> Reading symbols from /lib64/libtinfo.so.5...Reading symbols from
> /usr/lib/debug/lib64/libtinfo.so.5.7.debug...done.
> done.
> Loaded symbols for /lib64/libtinfo.so.5
> Reading symbols from /lib64/libc.so.6...Reading symbols from
> /usr/lib/debug/lib64/libc-2.12.so.debug...done.
> done.
> Loaded symbols for /lib64/libc.so.6
> Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from
> /usr/lib/debug/lib64/ld-2.12.so.debug...done.
> done.
> Loaded symbols for /lib64/ld-linux-x86-64.so.2
> 0x00000033f52dff48 in tcsetattr (fd=0, optional_actions=<value optimized
> out>, termios_p=0x62cb60)
> at ../sysdeps/unix/sysv/linux/tcsetattr.c:84
> 84 retval = INLINE_SYSCALL (ioctl, 3, fd, cmd, &k_termios);
> (gdb) bt
> #0 0x00000033f52dff48 in tcsetattr (fd=0, optional_actions=<value
> optimized out>, termios_p=0x62cb60)
> at ../sysdeps/unix/sysv/linux/tcsetattr.c:84
> #1 0x0000000000406f8d in _set_tty_settings (tty=0, tiop=0x62cb60) at
> rltty.c:476
> #2 0x0000000000406fe3 in set_tty_settings (tty=<value optimized out>,
> tiop=<value optimized out>) at rltty.c:490
> #3 0x00000000004072d0 in rl_deprep_terminal () at rltty.c:688
> #4 0x0000000000413352 in rl_cleanup_after_signal () at signals.c:536
> #5 0x0000000000413731 in _rl_handle_signal (sig=20) at signals.c:232
> #6 0x00000000004137e5 in _rl_signal_handler (sig=<value optimized out>) at
> signals.c:155
> #7 0x0000000000415575 in rl_getc (stream=0x33f558e6c0) at input.c:480
> #8 0x0000000000415a60 in rl_read_key () at input.c:462
> #9 0x000000000040340d in readline_internal_char () at readline.c:564
> #10 0x00000000004037d3 in readline_internal_charloop (prompt=<value
> optimized out>) at readline.c:629
> #11 readline_internal (prompt=<value optimized out>) at readline.c:643
> #12 readline (prompt=<value optimized out>) at readline.c:369
> #13 0x00000000004025b6 in main (argc=1, argv=0x7fff0af285d8) at rl.c:149
> (gdb)
> (gdb) c
> Continuing.
>
> Program received signal SIGTTOU, Stopped (tty output).
> 0x00000033f52dff48 in tcsetattr (fd=0, optional_actions=<value optimized
> out>, termios_p=0x62cb60)
> at ../sysdeps/unix/sysv/linux/tcsetattr.c:84
> 84 retval = INLINE_SYSCALL (ioctl, 3, fd, cmd, &k_termios);
> (gdb) c
> Continuing.
>
> Program received signal SIGTTOU, Stopped (tty output).
> 0x00000033f52dff48 in tcsetattr (fd=0, optional_actions=<value optimized
> out>, termios_p=0x62cb60)
> at ../sysdeps/unix/sysv/linux/tcsetattr.c:84
> 84 retval = INLINE_SYSCALL (ioctl, 3, fd, cmd, &k_termios);
> (gdb) c
> Continuing.
>
> Program received signal SIGTTOU, Stopped (tty output).
> 0x00000033f52dff48 in tcsetattr (fd=0, optional_actions=<value optimized
> out>, termios_p=0x62cb60)
> at ../sysdeps/unix/sysv/linux/tcsetattr.c:84
> 84 retval = INLINE_SYSCALL (ioctl, 3, fd, cmd, &k_termios);
> (gdb)
>
> I have tried this with several different kernel versions, RHEL6, Fedora,
> RHEL7,
> all with the same result. The examples above are on a RHEL6
> 2.6.32-537.el6.x86_64
> kernel.
Here's what's happening: the application calling readline (crash) gets the
SIGTSTP while readline has control. The readline signal handler catches
it and tries to clean up readline's state, including restoring the terminal
attributes. Unfortunately, by this time, the kernel has marked the entire
process group as a background pgrp, and disallows writing to the terminal
(the TOSTOP setting doesn't matter in this case). The attempt to restore
the terminal settings generates the SIGTTOU you see. The SIGTTOU causes
readline to follow the signal handling code path, and the same thing
happens again and again.
There are a couple of things that can be done here. The first is removing
the shell from the equation. Changing `rl' to `exec rl' seems to eliminate
the problem behavior. (The shell doesn't matter: running it from dash
does the same thing with and without the `exec'.) That makes me wonder
whether the difference is whether or not the process using readline is the
process group leader, but I can't figure out why that would make the
difference.
Obviously, preventing readline from trying to restore the terminal settings
will solve this problem, but that's a little drastic: any program using
readline will then leave the terminal settings modified on SIGTSTP, which
will cause havoc for users of shells who don't restore the terminal
settings when a process stops or terminates due to a signal.
I will have to look at some other things. Any ideas are welcome.
Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRU c...@case.edu http://cnswww.cns.cwru.edu/~chet/
