Question about inline code / incompatibility between FreeBSD sh and bash
Hi,
Sorry for emailing this list, but neither the GNU indexed
homepage nor the actual project page listed any other email address.
I'm trying to determine whether or not this is an implicit bug or
a feature of BSD bourne shell (in particular FreeBSD's sh):
When I try and execute `portsnap fetch update' [1], things go
through as the author intended. However, using `bash portsnap fetch
update' I get the following error message:
/usr/sbin/portsnap: portsnap: line 882: syntax error near unexpected
token `newline'
/usr/sbin/portsnap: portsnap: line 882: ` if !'
The incompatible section of code (at least the one that bash
barfs on -- I say that because there are a few similar sections IIRC)
is:
if !
if ! [ -z "${EXTRACTPATH}" ]; then
grep "^${EXTRACTPATH}" ${WORKDIR}/INDEX
elif ! [ -z "${REFUSE}" ]; then
grep -vE "${REFUSE}" ${WORKDIR}/INDEX
else
cat ${WORKDIR}/INDEX
fi | tr '|' ' ' | while read FILE HASH; do
echo ${PORTSDIR}/${FILE}
if ! [ -r "${WORKDIR}/files/${HASH}.gz" ]; then
echo "files/${HASH}.gz not found -- snapshot corrupt."
return 1
fi
case ${FILE} in
*/)
rm -rf ${PORTSDIR}/${FILE%/}
mkdir -p ${PORTSDIR}/${FILE}
tar -xzf ${WORKDIR}/files/${HASH}.gz\
-C ${PORTSDIR}/${FILE}
;;
*)
rm -f ${PORTSDIR}/${FILE}
tar -xzf ${WORKDIR}/files/${HASH}.gz\
-C ${PORTSDIR} ${FILE}
;;
esac
done; then
I'm pretty sure that the issue is because the code spans multiple
lines and could be (typically) modularized out into a function in
bash.
So the question is, can anyone verify that either bash or
FreeBSD's sh is correct, and/or verify whether or not there is a
possible incompatibility with the POSIX spec for bourne shells in
either camp?
Thanks,
-Garrett
PS Please CC me as I'm not subscribed to [EMAIL PROTECTED]
[1] A copy of portsnap can be found here:
http://www.freebsd.org/cgi/cvsweb.cgi/src/usr.sbin/portsnap/portsnap/portsnap.sh?rev=1.28;content-type=text%2Fplain
Re: Question about inline code / incompatibility between FreeBSD sh and bash
On Thu, Jun 19, 2008 at 2:12 AM, Garrett Cooper <[EMAIL PROTECTED]> wrote:
> Hi,
> Sorry for emailing this list, but neither the GNU indexed
> homepage nor the actual project page listed any other email address.
> I'm trying to determine whether or not this is an implicit bug or
> a feature of BSD bourne shell (in particular FreeBSD's sh):
> When I try and execute `portsnap fetch update' [1], things go
> through as the author intended. However, using `bash portsnap fetch
> update' I get the following error message:
>
> /usr/sbin/portsnap: portsnap: line 882: syntax error near unexpected
> token `newline'
> /usr/sbin/portsnap: portsnap: line 882: ` if !'
>
> The incompatible section of code (at least the one that bash
> barfs on -- I say that because there are a few similar sections IIRC)
> is:
>
>if !
>if ! [ -z "${EXTRACTPATH}" ]; then
>grep "^${EXTRACTPATH}" ${WORKDIR}/INDEX
>elif ! [ -z "${REFUSE}" ]; then
>grep -vE "${REFUSE}" ${WORKDIR}/INDEX
>else
>cat ${WORKDIR}/INDEX
>fi | tr '|' ' ' | while read FILE HASH; do
>echo ${PORTSDIR}/${FILE}
>if ! [ -r "${WORKDIR}/files/${HASH}.gz" ]; then
>echo "files/${HASH}.gz not found -- snapshot corrupt."
>return 1
>fi
>case ${FILE} in
>*/)
>rm -rf ${PORTSDIR}/${FILE%/}
>mkdir -p ${PORTSDIR}/${FILE}
>tar -xzf ${WORKDIR}/files/${HASH}.gz\
>-C ${PORTSDIR}/${FILE}
>;;
>*)
>rm -f ${PORTSDIR}/${FILE}
>tar -xzf ${WORKDIR}/files/${HASH}.gz\
>-C ${PORTSDIR} ${FILE}
>;;
>esac
>done; then
>
>I'm pretty sure that the issue is because the code spans multiple
> lines and could be (typically) modularized out into a function in
> bash.
>So the question is, can anyone verify that either bash or
> FreeBSD's sh is correct, and/or verify whether or not there is a
> possible incompatibility with the POSIX spec for bourne shells in
> either camp?
> Thanks,
> -Garrett
>
> PS Please CC me as I'm not subscribed to [EMAIL PROTECTED]
>
> [1] A copy of portsnap can be found here:
> http://www.freebsd.org/cgi/cvsweb.cgi/src/usr.sbin/portsnap/portsnap/portsnap.sh?rev=1.28;content-type=text%2Fplain
When I take the above code, combine remove the newline and make it into ...
if ! if ! [ -z "${EXTRACTPATH}" ]; then
... as well as line 951, the script is parsed properly.
The "problem" is reported at parse.y:3071, but I'm unsure how to trace
back the issue through the yacc code.
-Garrett
Re: Question about inline code / incompatibility between FreeBSD sh and bash
On Thu, Jun 19, 2008 at 2:29 AM, Garrett Cooper <[EMAIL PROTECTED]> wrote:
> On Thu, Jun 19, 2008 at 2:12 AM, Garrett Cooper <[EMAIL PROTECTED]> wrote:
>>
>> /usr/sbin/portsnap: portsnap: line 882: syntax error near unexpected
>> token `newline'
>> /usr/sbin/portsnap: portsnap: line 882: ` if !'
>>
>> The incompatible section of code (at least the one that bash
>> barfs on -- I say that because there are a few similar sections IIRC)
>> is:
>>
>>if !
>>if ! [ -z "${EXTRACTPATH}" ]; then
>>grep "^${EXTRACTPATH}" ${WORKDIR}/INDEX
>>elif ! [ -z "${REFUSE}" ]; then
>>grep -vE "${REFUSE}" ${WORKDIR}/INDEX
>>else
>>cat ${WORKDIR}/INDEX
>>fi | tr '|' ' ' | while read FILE HASH; do
>>echo ${PORTSDIR}/${FILE}
>>if ! [ -r "${WORKDIR}/files/${HASH}.gz" ]; then
>>echo "files/${HASH}.gz not found -- snapshot corrupt."
>>return 1
>>fi
>>case ${FILE} in
>>*/)
>>rm -rf ${PORTSDIR}/${FILE%/}
>>mkdir -p ${PORTSDIR}/${FILE}
>>tar -xzf ${WORKDIR}/files/${HASH}.gz\
>>-C ${PORTSDIR}/${FILE}
>>;;
>>*)
>>rm -f ${PORTSDIR}/${FILE}
>>tar -xzf ${WORKDIR}/files/${HASH}.gz\
>>-C ${PORTSDIR} ${FILE}
>>;;
>>esac
>>done; then
>>
>
> When I take the above code, combine remove the newline and make it into ...
>
>if ! if ! [ -z "${EXTRACTPATH}" ]; then
>
> ... as well as line 951, the script is parsed properly.
>
> The "problem" is reported at parse.y:3071, but I'm unsure how to trace
> back the issue through the yacc code.
Correction: report_syntax_error in parse.y.
-Garrett
access(2) shouldn't be used with test(1)
According to the POSIX spec, using access(2) is implementation dependent when running as superuser [1]. FreeBSD intentionally returns true whenever euid/uid = 0 [2]. FreeBSD's /bin/sh doesn't have this `issue' with test(1). Example: $ ls -l typescript -rw-r--r-- 1 gcooper gcooper 37875 Jul 12 22:19 typescript $ sudo sh -c 'test -x typescript; echo $?' 1 $ sudo bash -c 'test -x typescript; echo $?' 0 Code should be added to detect the mode via stat(2), instead of access(2) (the FreeBSD manpage also notes security issues with race conditions when using access(2), so access(2) use is discouraged). If I can get the details for grabbing bash from cvs/svn/whatever, I'll whip up a patch for this. Thanks, -Garrett [1] http://www.opengroup.org/onlinepubs/95399/functions/access.html - see RATIONALE. [2] http://www.freebsd.org/cgi/man.cgi?query=access
Re: access(2) shouldn't be used with test(1)
On Tue, Jul 20, 2010 at 11:00 AM, Garrett Cooper wrote: > According to the POSIX spec, using access(2) is implementation > dependent when running as superuser [1]. FreeBSD intentionally returns > true whenever euid/uid = 0 [2]. FreeBSD's /bin/sh doesn't have this > `issue' with test(1). Example: > > $ ls -l typescript > -rw-r--r-- 1 gcooper gcooper 37875 Jul 12 22:19 typescript > $ sudo sh -c 'test -x typescript; echo $?' > 1 > $ sudo bash -c 'test -x typescript; echo $?' > 0 > > Code should be added to detect the mode via stat(2), instead of > access(2) (the FreeBSD manpage also notes security issues with race > conditions when using access(2), so access(2) use is discouraged). > If I can get the details for grabbing bash from cvs/svn/whatever, > I'll whip up a patch for this. > > [1] http://www.opengroup.org/onlinepubs/95399/functions/access.html > - see RATIONALE. > [2] http://www.freebsd.org/cgi/man.cgi?query=access Oh, and BTW... here's the version of bash I'm using: $ bash --version GNU bash, version 4.1.7(0)-release (amd64-portbld-freebsd9.0) Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software; you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Thanks, -Garrett
Re: access(2) shouldn't be used with test(1)
On Tue, Jul 20, 2010 at 11:28 AM, Eric Blake wrote: > On 07/20/2010 12:00 PM, Garrett Cooper wrote: >> According to the POSIX spec, using access(2) is implementation >> dependent when running as superuser [1]. > > But as long as the answer is correct, then the access(2) family of calls > is the right thing to use. > >> FreeBSD intentionally returns >> true whenever euid/uid = 0 [2]. > > Remember, POSIX allows an OS where the superuser CAN read/write/execute > any file it wants, regardless of whether there are explicit permission > bits set in the stat() results. If FreeBSD is one of the OS's where the > superuser can execute a file that doesn't have any x bits set in stat(), > then that's one of the privileges of being the superuser, and their > implementation of access(X_OK) always returning true is correct. On the > other hand, POSIX suggests that a better implementation is that the > superuser should only be able to arbitrarily execute files if the stat() > and ACL bits allow execute permissions to at least one entity (whether > or not that entity is also the superuser), in which case, blindly > returning true for access(X_OK) for uid 0 on a file with no other > execute permissions would be a bug in access(). Yeah, the manpage gives access(2) the `license to lie': Even if a process's real or effective user has appropriate privileges and indicates success for X_OK, the file may not actually have execute per- mission bits set. Likewise for R_OK and W_OK. > Also, remember that access(2) takes into account ACLs, but stat() does > not. POSIX requires that 'test -x file' succeed for a non-superuser uid > that does not own a file with 700 stat() bits, but where the file also > has an ACL granting execute rights to that uid. Hmmm... >> FreeBSD's /bin/sh doesn't have this >> `issue' with test(1). Example: > > I argue that either FreeBSD's /bin/sh or their access(2) is buggy; the > question now is which one. Technically it's test(1) that's reporting the mode info, and what they do is use eaccess(2) instead of access(2). >> $ ls -l typescript >> -rw-r--r-- 1 gcooper gcooper 37875 Jul 12 22:19 typescript >> $ sudo sh -c 'test -x typescript; echo $?' >> 1 >> $ sudo bash -c 'test -x typescript; echo $?' >> 0 > > Try: > > echo echo hi > foo > chmod 644 foo > sudo sh -c './foo' > > If it prints hi, then bash is correct as-is, because FreeBSD falls into > the category of OSs that allow arbitrary superuser execution regardless > of whether any other entity has execute rights, which is allowed but not > recommended by POSIX. On the other hand, if the superuser fails to > execute ./foo but access(2) says it can, then FreeBSD's access(2) is > broken, at which point bash should be working around the broken > access(). But I still argue that bash is correct for using the > access(2) family (actually, bash should be using faccessat(,AT_EACCESS) > [required by POSIX 2008] or the non-portable alternatives eaccess() or > euidaccess() in preference to access(2), because you want to know if the > file is executable for the effective id). Yeah, it fails: $ echo echo hi > foo $ chmod 644 foo $ sudo sh -c './foo' ./foo: Permission denied >> Code should be added to detect the mode via stat(2), instead of >> access(2) (the FreeBSD manpage also notes security issues with race >> conditions when using access(2), so access(2) use is discouraged). >> If I can get the details for grabbing bash from cvs/svn/whatever, > > Bash (unfortunately) is not available in a public repository. Provide > any patch against the 4.1 sources instead; that's the latest publicly > available source. But be aware that bash already has fallback code to > use stat() on systems with known-buggy access(2), and that your solution > must NOT interfere with correct test behavior on systems with ACLs (that > is, relying _solely_ on stat() is almost guaranteed to be wrong on any > platform with alternate access controls). > > Meanwhile, if bash ever uses access() instead of faccessat(,AT_EACCESS) > on a POSIX 2008 platform, then that would be a bug worth fixing. I see the quandary now (thanks for the clarification). FreeBSD supports POSIX ACLs and MAC, so yes, it does appear that access(2) (or rather the underlying callers for it) should be fixed. I'll have to check faccessat(2) to see whether or not it's buggy by writing unittests for it (*sigh*). At least I'll integrate them into the open_posix_testsuite in ltp, which means they'll be available for Linux as well... Thanks, -Garrett
Re: access(2) shouldn't be used with test(1)
On Tue, Jul 20, 2010 at 2:18 PM, Linda Walsh wrote: > What are you saying it should return, because I get correct results > on linux: > > sudo sh -c 'test -x start_trans; echo $?' > 1 > sudo bash -c 'test -x start_trans; echo $?' 1 > > Are you saying that access on BSD, when > passed (filename "X_OK") returns "true" for superuser even > when the file isn't executable? Correct. > Sounds like a bug to me if a file isn't executable, I don't care > how "super" you are -- permission won't make a text file executable! > ;-) Yes, but POSIX says that it's allowed to be broken based on past behavior, and FreeBSD just says "don't use access, act on the file", which means that access on FreeBSD is only really useful for F_OK, but I can get that from open/fstat/whatever. Not having correct data via test(1) renders the utility useless in many scenarios, which isn't desirable... I've found other issues with other implementations of access that need to be resolved as well. This is just one of the problem applications / languages that I'm addressing that I care about. Thanks, -Garrett > Garrett Cooper wrote: >> >> According to the POSIX spec, using access(2) is implementation >> dependent when running as superuser [1]. FreeBSD intentionally returns >> true whenever euid/uid = 0 [2]. FreeBSD's /bin/sh doesn't have this >> `issue' with test(1). Example: >> >> $ ls -l typescript >> -rw-r--r-- 1 gcooper gcooper 37875 Jul 12 22:19 typescript >> $ sudo sh -c 'test -x typescript; echo $?' >> 1 >> $ sudo bash -c 'test -x typescript; echo $?' >> 0 >> >> Code should be added to detect the mode via stat(2), instead of >> access(2) (the FreeBSD manpage also notes security issues with race >> conditions when using access(2), so access(2) use is discouraged). >> If I can get the details for grabbing bash from cvs/svn/whatever, >> I'll whip up a patch for this. >> Thanks, >> -Garrett >> >> [1] http://www.opengroup.org/onlinepubs/95399/functions/access.html >> - see RATIONALE. >> [2] http://www.freebsd.org/cgi/man.cgi?query=access >> >
Re: access(2) shouldn't be used with test(1)
On Tue, Jul 20, 2010 at 5:16 PM, Chet Ramey wrote: > On 7/20/10 2:00 PM, Garrett Cooper wrote: >> According to the POSIX spec, using access(2) is implementation >> dependent when running as superuser [1]. FreeBSD intentionally returns >> true whenever euid/uid = 0 [2]. FreeBSD's /bin/sh doesn't have this >> `issue' with test(1). Example: > > Bash-4.1 doesn't use access for `test -x' (or -e, -r, or -w, for that > matter) on FreeBSD. If eaccess is available and configure detects its > presence, bash uses that, otherwise it either uses access or checks the > permissions returned by stat. > > Bash-4.2 will prefer the use of faccessat if available, falling back to > eaccess and then access/stat. On FreeBSD, bash-4.2 will use stat to > verify X_OK when euid == 0 even if eaccess returns true, since eaccess > lies also (the FreeBSD test does the same thing). > > The relevant code is in lib/sh/eaccess.c:sh_eaccess(). This was > discussed extensively back in March. I see what you mean.. I need to do more digging because according to the config logs, eaccess(2) is used, and when I put an #error in the relevant block, it pops up with: findcmd.c:98:2: error: #error "!eaccess!" *** Error code 1 Stop in /usr/ports/shells/bash/work/bash-4.1. *** Error code 1 :(... Thanks, -Garrett
