Mac OS X Unicode NFD filesystems and bash completion
Configuration Information [Automatically generated, do not change]: Machine: powerpc OS: darwin8.11.0 Compiler: gcc Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='powerpc' - DCONF_OSTYPE='darwin8.11.0' -DCONF_MACHTYPE='powerpc-apple- darwin8.11.0' -DCONF_VENDOR='apple' -DLOCALEDIR='/usr/local/ bash-3.2.9/share/locale' -DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H - DMACOSX -I. -I. -I./include -I./lib -g -O2 uname output: Darwin asterix.local 8.11.0 Darwin Kernel Version 8.11.0: Wed Oct 10 18:26:00 PDT 2007; root:xnu-792.24.17~1/ RELEASE_PPC Power Macintosh powerpc Machine Type: powerpc-apple-darwin8.11.0 Bash Version: 3.2 Patch Level: 9 Release Status: release Description: See: http://www.mail-archive.com/bug-bash@gnu.org/msg04073.html Mac OS X, when listing directories, reports filenames in UTF-8 NFD. With an UTF-8 configured input, bash receives strings in UTF-8 NFC. Thus its filename completion cannot match input strings with filesystem entities when accents occur. Repeat-By: cd /tmp mkdir Réseau ls Ré (Réseau is not proposed as possible completion) Fix: I have come up with a fix that works great on my system, using iconv (bundled with Mac OS X, anyway). I'm wondering though: - Does it work well on "alternate" filesystems (is NFD done on OS level, or is it driver dependent?). If someone on the list has NTFS-3G-mounted disks or SSHFS, can he/she test that at least there is no loss in functionality? - Is it Mac OS X specific? I think Darwin has the same VFS layer. In this case, is there a more general replacement for #ifdef MACOSX? - iconv could certainly be detected in a cleaner way as I did (forcefully adding it to link libs, and implying it from MACOSX preprocessing define). Tested with Terminal (UTF-8: works; non-UTF-8: does not complete, so not better than before but that's normal) and xterm. I have put the patch here (diff -u format): http://ks31107.kimsufi.com/gui/bash-mac-filename-completion.patch I have made it for bash 3.2.9, it runs unmodified on a 3.2.49, and I now use it on a 4.0.24 (patch utility detected a 83-lines offset and ran happily)
parallel several jobs in bash
Hi, I wonder how to in a bash script(or other languages if more convenient and/or fast) manage independent jobs(executables with command line arguments) and make them run in parallel? Is there a way to create several sub-process without waiting them to finish? Which one is faster: multi-threading or multi-process? Thanks and regards!
Re: parallel several jobs in bash
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
[please consider making your mailer wrap long lines]
According to Tim on 7/14/2009 12:02 PM:
> Hi,
>
> I wonder how to in a bash script(or other languages if more convenient
and/or fast) manage independent jobs(executables with command line
arguments) and make them run in parallel? Is there a way to create several
sub-process without waiting them to finish?
Short answer - yes, bash can manage multiple parallel worker tasks.
Long answer - you may want to read up on how current autoconf.git
implements parallel testsuites. Doing it portably in shell is a nightmare
(as so many shells out there have bugs in their trap handlers), but bash
at least shines in this area.
>
> Which one is faster: multi-threading or multi-process?
Multi-threading is inherently faster than multi-process, as fewer
resources must be managed when swapping between workers. But how much
faster depends on the particular OS. Bash can only do multi-process.
Here's the core driver loop from a testsuite created by current
autoconf.git (take it with a grain of salt - this is a relatively new
feature of autotest, and portability improvements are welcome):
if (set -m && set +m) >/dev/null 2>&1; then
at_job_control_on='set -m' at_job_control_off='set +m' at_job_group=-
else
at_job_control_on=: at_job_control_off=: at_job_group=
fi
for at_signal in 1 2 15; do
trap 'set +x; set +e
$at_job_control_off
at_signal='"$at_signal"'
echo stop > "$at_stop_file"
trap "" $at_signal
at_pgids=
for at_pgid in `jobs -p 2>/dev/null`; do
at_pgids="$at_pgids $at_job_group$at_pgid"
done
test -z "$at_pgids" || kill -$at_signal $at_pgids 2>/dev/null
wait
if test "$at_jobs" -eq 1 || test -z "$at_verbose"; then
echo >&2
fi
at_signame=`kill -l $at_signal 2>&1 || echo $at_signal`
set x $at_signame
test 0 -gt 2 && at_signame=$at_signal
{ $as_echo "$as_me:${as_lineno-$LINENO}: WARNING: caught signal
$at_signame, bailing out" >&5
$as_echo "$as_me: WARNING: caught signal $at_signame, bailing out" >&2;}
as_fn_arith 128 + $at_signal && exit_status=$as_val
as_fn_exit $exit_status' $at_signal
done
rm -f "$at_stop_file"
at_first=:
if test $at_jobs -ne 1 &&
rm -f "$at_job_fifo" &&
test -n "$at_job_group" &&
( mkfifo "$at_job_fifo" && trap 'exit 1' PIPE STOP TSTP ) 2>/dev/null
then
# FIFO job dispatcher.
trap 'at_pids=
for at_pid in `jobs -p`; do
at_pids="$at_pids $at_job_group$at_pid"
done
if test -n "$at_pids"; then
at_sig=TSTP
test "${TMOUT+set}" = set && at_sig=STOP
kill -$at_sig $at_pids 2>/dev/null
fi
kill -STOP $$
test -z "$at_pids" || kill -CONT $at_pids 2>/dev/null' TSTP
echo
# Turn jobs into a list of numbers, starting from 1.
at_joblist=`$as_echo " $at_groups_all " | \
sed 's/\( '$at_jobs'\) .*/\1/'`
set X $at_joblist
shift
for at_group in $at_groups; do
$at_job_control_on 2>/dev/null
(
# Start one test group.
$at_job_control_off
exec 6>"$at_job_fifo"
trap 'set +x; set +e
trap "" PIPE
echo stop > "$at_stop_file"
echo token >&6
as_fn_exit 141' PIPE
at_fn_group_prepare
if cd "$at_group_dir" &&
at_fn_test $at_group &&
. "$at_test_source" # AT_JOB_FIFO_FD>&-
then :; else
{ $as_echo "$as_me:${as_lineno-$LINENO}: WARNING: unable to parse
test group: $at_group" >&5
$as_echo "$as_me: WARNING: unable to parse test group: $at_group" >&2;}
at_failed=:
fi
at_fn_group_postprocess
echo token >&6
) &
$at_job_control_off
if $at_first; then
at_first=false
exec 6<"$at_job_fifo"
fi
shift # Consume one token.
if test $# -gt 0; then :; else
read at_token <&6 || break
set x $*
fi
test -f "$at_stop_file" && break
done
# Read back the remaining ($at_jobs - 1) tokens.
set X $at_joblist
shift
if test $# -gt 0; then
shift
for at_job
do
read at_token
done <&6
fi
exec 6<&-
wait
else
# Run serially, avoid forks and other potential surprises.
for at_group in $at_groups; do
at_fn_group_prepare
if cd "$at_group_dir" &&
at_fn_test $at_group &&
. "$at_test_source"; then :; else
{ $as_echo "$as_me:${as_lineno-$LINENO}: WARNING: unable to parse
test group: $at_group" >&5
$as_echo "$as_me: WARNING: unable to parse test group: $at_group" >&2;}
at_failed=:
fi
at_fn_group_postprocess
test -f "$at_stop_file" && break
at_first=false
done
fi
- --
Don't work too hard, make some time for fun as well!
Eric Blake e...@byu.net
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (Cygwin)
Comment: Public key at home.comcast.net/~ericblake/eblake.gpg
Comment: Using GnuPG with
Pressing esc twice then left arrow deletes last character
Configuration Information [Automatically generated, do not change]: Machine: x86_64 OS: linux-gnu Compiler: gcc Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64' -DCONF_OSTYPE='li nux-gnu' -DCONF_MACHTYPE='x86_64-redhat-linux-gnu' -DCONF_VENDOR='redhat' -DLOCA LEDIR='/usr/share/locale' -DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H -I. -I. -I ./include -I./lib -D_GNU_SOURCE -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fe xceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic uname output: Linux tigerMtn.cgtown 2.6.27.5-41.fc9.x86_64 #1 SMP Thu Nov 13 20: 29:07 EST 2008 x86_64 x86_64 x86_64 GNU/Linux Machine Type: x86_64-redhat-linux-gnu Bash Version: 3.2 Patch Level: 33 Release Status: release Description: There are several ways to get this bug to appear, but they all involve being in vi-command-mode (set -o vi) after entering some text to the command line, then pressing escape twice, then using a left or right arrow key to move around at which point characters are deleted from the point of the cursor to the end of the line. Repeat-By: - At the prompt type 0123456789 - hit escape twice - press the left-arrow - the '9' will be deleted - press escape again - press left-arrow again - the '8' will be deleted - etc...
Prompt length calculation fails when UTF-8 is used within \[ \]
Steps to reproduce: (using UTF-8 locales) $ export PS1="\[\e]2;test Ä and Ö here\a\]prompt>" prompt>abcdefgh# Enter some alphabets and press Home ^ Cursor goes here, instead of ^ here where it should go. Add more UTF-8 letters as non-printable characters in PS1 and the offset from the correct position gets bigger. Here it was two characters because of two "extra" bytes in the UTF-8 put in there. I noticed that you have to type a right amount of characters to make Home key use the calculated prompt length. abcdefgh seems to work in this case, but if the prompt is modified, you may need to type more or less than that to see the effect. Tested with versions: GNU bash, version 4.0.0(2)-release (x86_64-unknown-linux-gnu) GNU bash, version 3.2.48(1)-release (x86_64-pc-linux-gnu)
