Re: Some fuzzer workarounds
Hi, On Fri, Mar 18, 2022 at 10:26:16AM +0300, Evgeny Vereshchagin wrote: > I think before looking at those reports it would make sense to > figure out what they are supposed to test and how they were tested > to make sure they don't produce false positives. If they weren't > actually tested I think it would make sense to revert them to avoid > getting auto-generated CVEs until they're in more or less good shape > at least. So I took the fuzz-libelf.c and fuzz-libdwfl.c files from the oss-fuzz repo, tweaked them so they have a normal main that takes one file argument to try to replicate the reports. That found some "real" issues I submitted patches for. Then I ran afl-fuzz on them locally during the weekend and found another issue for which I also submitted a patch. There are several duplicates though and all the MSAN reported issues seem bogus. > > There are also some other > > misaligned type access checks reported by ubsan, but I don't know if > > that is because of ALLOW_UNALIGNED is still defined or not (when > > configuring with --enable-analyze-undefined ALLOW_UNALIGNED is not > > defined, otherwise it is for some arches, including x86_64). > > Looking at > https://github.com/google/oss-fuzz/commit/8747524f04b1b906d4a21a6ade87f7803b3f9b8c, > I think I turned ALLOW_UNALIGNED off with UBSan there (and tested it > in https://sourceware.org/bugzilla/show_bug.cgi?id=28720). Yes, you are right, all the unaligned type access issues were "real" (they occur with or without ALLOW_UNALIGNED on). It is not really a big issue, but could potentially cause the compiler to do some surprising optimisations. So I did fix them all. Cheers, Mark
Re: Some fuzzer workarounds
Hi, On Thu, Mar 17, 2022 at 02:30:49PM +0100, Mark Wielaard wrote: > The following fixes should fix reading of some broken ar archives and > misaligned access of the section zero Shdr for mmaped ELF files where > the start of the Elf image is at some offset from the start of the > map. > > [PATCH 1/2] libelf: Take map offset into account for Shdr alignment > [PATCH 2/2] libelf: Make sure ar_size starts with a digit before > > https://code.wildebeest.org/git/user/mjw/elfutils/log/?h=fuzz > > I haven't been able to replicate any other issues locally. So I did eventually replicate some other issues and ran afl-fuzz locally over the new fuzz targets during the weekend to look for any others. I pushed the above two patches, plus the other fixes I posted: libelf: Check alignment of Verdef, Verdaux, Verneed and Vernaux offsets libdwfl: Close ar members when they cannot be processed. libdwfl: Use memcpy to assign image header field values libelf: Don't overflow offsets in elf_cvt_Verneed and elf_cvt_Verdef That should hopefully shutup the monorail reports. Except for those using MSAN, which look bogus to me. Cheers, Mark
Re: Some fuzzer workarounds
Hi Mark, > So I took the fuzz-libelf.c and fuzz-libdwfl.c files from the oss-fuzz > repo, tweaked them so they have a normal main that takes one file > argument to try to replicate the reports. That found some "real" > issues I submitted patches for. Then I ran afl-fuzz on them locally > during the weekend and found another issue for which I also submitted > a patch. FWIW to test the "fuzz" branch I integrated the new fuzz targets into the elfutils build system by analogy with https://sourceware.org/pipermail/elfutils-devel/2021q4/004615.html and there they are linked with the main function automatically and it's also possible to pass --enable-afl to ./configure to automatically run it with AFL. It still needs polishing but I wonder if it makes sense to send that to the mailing list? I still think the fuzz targets should be kept and reviewed upstream and that patch would make it possible. It's being tested in https://github.com/evverx/elfutils/pull/72. I'll report back once I figure out why the unit tests are failing on Fedora Rawhide: https://copr-be.cloud.fedoraproject.org/results/packit/evverx-elfutils-72/fedora-rawhide-x86_64/03799633-elfutils/builder-live.log.gz > There are several duplicates though and all the MSAN reported > issues seem bogus. I'm not sure all of them are bogus but I would ignore them for now. Once the new fuzz targets are linked with zlib built with MSan bogus reports will be closed and I'll take a look at what's left there. Thanks, Evgeny Vereshchagin
Re: Some fuzzer workarounds
Hi Mark, > I'll report back once I figure > out why the unit tests are failing on Fedora Rawhide: > https://copr-be.cloud.fedoraproject.org/results/packit/evverx-elfutils-72/fedora-rawhide-x86_64/03799633-elfutils/builder-live.log.gz > I tested the fuzz branch and I can confirm that all the issues reported by OSS-Fuzz found with ASan+UBSan are gone. I kind of lost track of them at some point but the following issues can no longer be triggered: fuzz-libdwfl-crashes/oss-fuzz-45629 \ fuzz-libdwfl-crashes/oss-fuzz-45634 \ fuzz-libdwfl-crashes/oss-fuzz-45635 \ fuzz-libdwfl-crashes/oss-fuzz-45636 \ fuzz-libdwfl-crashes/oss-fuzz-45646 \ fuzz-libelf-crashes/oss-fuzz-45637 \ fuzz-libelf-crashes/oss-fuzz-45682 The unit tests have nothing to do with the fuzz branch because once I pointed the tests to the master branch they also failed. Looking at "phdr[8]: unknown object file note type 3405650558 with owner name 'FDO' at offset 200" it seems it's caused by the toolchain used there. On Fedora 35 the tests pass. Thanks, Evgeny Vereshchagin
Issue 45705 in oss-fuzz: elfutils:fuzz-libdwfl: Indirect-leak in __libelf_next_arhdr_wrlock
Comment #4 on issue 45705 by da...@adalogics.com: elfutils:fuzz-libdwfl: Indirect-leak in __libelf_next_arhdr_wrlock https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=45705#c4 Yes -- I did this because I asked in an internal email with Mark if it would be appreciated (the answer was yet). But will stop this as you're taking care of it by way of the mailing list. -- You received this message because: 1. You were specifically CC'd on the issue You may adjust your notification preferences at: https://bugs.chromium.org/hosting/settings Reply to this email to add a comment.
Issue 45705 in oss-fuzz: elfutils:fuzz-libdwfl: Indirect-leak in __libelf_next_arhdr_wrlock
Comment #5 on issue 45705 by evv...@gmail.com: elfutils:fuzz-libdwfl: Indirect-leak in __libelf_next_arhdr_wrlock https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=45705#c5 > I did this because I asked in an internal email with Mark if it would be > appreciated (the answer was yet). Sorry. I didn't know that. If it was decided testcases should be attached as well I think it should be OK. It's just that I receive a lot of emails from Monorail and it's hard to keep track of them. I'll just set up a filter to only show the "new owner" and "verified" messages here going forward or something like that. Thanks! -- You received this message because: 1. You were specifically CC'd on the issue You may adjust your notification preferences at: https://bugs.chromium.org/hosting/settings Reply to this email to add a comment.
Re: Some fuzzer workarounds
Hi Evgeny, On Mon, 2022-03-21 at 17:33 +0300, Evgeny Vereshchagin wrote: > I tested the fuzz branch and I can confirm that all the issues > reported by OSS-Fuzz found with ASan+UBSan are gone. > I kind of lost track of them at some point but the following issues > can no longer be triggered: > > fuzz-libdwfl-crashes/oss-fuzz-45629 \ > fuzz-libdwfl-crashes/oss-fuzz-45634 \ > fuzz-libdwfl-crashes/oss-fuzz-45635 \ > fuzz-libdwfl-crashes/oss-fuzz-45636 \ > fuzz-libdwfl-crashes/oss-fuzz-45646 \ > fuzz-libelf-crashes/oss-fuzz-45637 \ > fuzz-libelf-crashes/oss-fuzz-45682 Great. Thanks for testing. All patches from the fuzz branch are now merged. My local fuzzer also hasn't found any new issues for almost 24 hours now. > The unit tests have nothing to do with the fuzz branch because once I > pointed the tests to the master branch they also > failed. Looking at "phdr[8]: unknown object file note type 3405650558 > with owner name 'FDO' at offset 200" it seems > it's caused by the toolchain used there. On Fedora 35 the tests pass. Ah, oops. Yeah that is: https://systemd.io/COREDUMP_PACKAGE_METADATA/ https://fedoraproject.org/wiki/Changes/Package_information_on_ELF_objects There are actually patches to properly recognize those. https://sourceware.org/pipermail/elfutils-devel/2021q4/thread.html#4375 I'll integrate those asap. Cheers, Mark
Re: Some fuzzer workarounds
Hi Mark, > Great. Thanks for testing. All patches from the fuzz branch are now > merged. My local fuzzer also hasn't found any new issues for almost 24 > hours now. Thanks! I synced my fork with the elfutils repository and tonight it will be sent to Coverity. If anything pops up I'll report it. > Yeah that is: > https://systemd.io/COREDUMP_PACKAGE_METADATA/ > https://fedoraproject.org/wiki/Changes/Package_information_on_ELF_objects That looks familiar. As far as I can remember fuzz-dwfl-core was introduced to fuzz that part of systemd. Thanks, Evgeny Vereshchagin
[PATCH] tests: integrate fuzz-targets into the test suite
[v5]
1) `fuzz-libdwfl` and `fuzz-libelf` were moved from OSS-Fuzz.
2) The regression testsuite was extended.
3) The OSS-Fuzz build script was removed. It should probably be
kept on OSS-Fuzz at this point.
4) The honggfuzz kludges were removed because
https://github.com/google/honggfuzz/issues/431 was fixed.
[v4]
1) `--enable-afl` was added to make it possible to build and run
the fuzz target with afl-gcc/afl-g++/afl-fuzz. It's compatible
with both AFL and AFL++ and can be tested on Ubuntu by running
the following commands:
```
apt-get install afl++
autoreconf -i -f
./configure --enable-maintainer-mode --enable-afl
make -j$(nproc) V=1
make check V=1 VERBOSE=1 TESTS=run-fuzz-dwfl-core.sh FUZZ_TIME=600
```
It's compatible with ASan/UBsan as well so something like
`--enable-sanitize-address` and `--enable-sanitize-undefined` can
additionally be passed to run the fuzzer under ASan/UBsan.
It was tested on Fedora with AFL (https://github.com/google/AFL) and
on Ubuntu with AFL++ (https://github.com/AFLplusplus/AFLplusplus/).
2) Both `use_afl` and `use_honggfuzz` are now shown among the additional
test features printed by `./configure` to make it easier to figure out
how exactly elfutils is tested.
[v3]
The test handles infinite loops much better now.
In https://sourceware.org/bugzilla/show_bug.cgi?id=28715#c4 it took
it about 5 hours on Packit to discover an infinite loop on 32 bit
platforms because it didn't enforce any timeouts. It was fixed
by passing --tmout_sigvtalrm to honggfuzz (which treats timeouts
as normal crashes) and by additionally running the fuzz target
with timeout -s9.
[v2]
1) At https://sourceware.org/pipermail/elfutils-devel/2021q4/004541.html
it was pointed out that build-fuzzers.sh is too tied to OSS-Fuzz
and while it was kind of decoupled from it as much as possible
in the sense that it was enough to install clang and run the script to build
the fuzz target with libFuzzer it's true that it can't be integrated smoothly
into buildbot for example where gcc is used and various configure options
control what exactly is testsed. To address that, `--enable-honggfuzz` is
introduced. It looks for hfuzz-gcc, hfuzz-g++ and honggfuzz and if
they exist elfutils is built with those wrappers and the fuzz target
is additionally run for a few minutes under honggfuzz to make
regression testing more effective. It was tested on Fedora 35 and
in https://github.com/evverx/elfutils/pull/53 on Ubuntu Focal with both gcc and
clang with and without sanitizers/Valgrind
and with two versions of honggfuzz (including the latest stable version).
To make it work on Ubuntu the following commands should be run
```
apt-get install libbfd-dev libunwind8-dev
git clone https://github.com/google/honggfuzz
cd honggfuzz
git checkout 2.4
make
make PREFIX=/usr install
cd PATH/TO/ELFUTILS
autoreconf -i -f
./configure --enable-maintainer-mode --enable-honggfuzz
make check V=1 VERBOSE=1 # FUZZ_TIME can be optionally passed
```
If hongfuzz is installed elsewhere it's possible to point
configure to it with CC, CXX and HONGGFUZZ
```
./configure CC=path-to-hfuzz-gcc CXX=path-to-hfuzz-g++
HONGGFUZZ=path-to-honggfuzz
```
I decided to use honggfuzz instead of AFL because AFL doesn't seem
to be maintained actively anymore. Other than that I can't seem to
make it work with various combinations of compilers, sanitizers and
so on. But thanks to the way the fuzz target is written it should
be possible to add it eventually by analogy with honggfuzz.
2) fuzz-dwfl-core-corpus was renamed to fuzz-dwfl-core-crashes to
make it more clear that it isn't exaclty a seed corpus.
3) run-strip-g.sh and run-strip-nothing.sh started to compile test programs
using temporary files instead of gcc -xc -. It should address
https://github.com/google/honggfuzz/issues/431 but more generally
autoconf uses temporary files to make sure compiler works so
it seems in general it's safer to rely on compiler features that
are known to work.
4) A comment was added where I tried to expand on why the fuzz target
is written that way.
[v1]
The fuzz target was integrated into OSS-Fuzz in
https://github.com/google/oss-fuzz/pull/6944 and since then it
has been running there continously (uncovering various issues
along the way). It's all well and good but since OSS-Fuzz
is far from the elfutils repository it's unnecessarily hard
to build the fuzz target locally, verify patches and more generally
test new code to make sure that it doesn't introduce new issues (
or reintroduce regressions). This patch aims to address all those
issues by moving the fuzz target into the elfutils repository,
integrating it into the testsuite and also providing a script
that can be used to build full-fledged fuzzers utilizing libFuzzer.
With this patch applied `make check` can be used to make sure
that files kept in tests/fuzz-dwfl-core-corpus don't crash the
code on various architecures.
`--enable-sanitize-{address,undefined}` and/or `--enable-valgrind`
can additionally be used to uncove
