[PATCH] readelf: Add optional "SECTION" argument for --notes.
There are multiple sections that can contain ELF Notes. It is sometimes nice to just list the notes from a specific section. -n, --notes[=SECTION] Display the ELF notes Signed-off-by: Mark Wielaard --- src/ChangeLog | 7 +++ src/readelf.c | 13 - 2 files changed, 19 insertions(+), 1 deletion(-) diff --git a/src/ChangeLog b/src/ChangeLog index c2102fcd..ad673c11 100644 --- a/src/ChangeLog +++ b/src/ChangeLog @@ -1,3 +1,10 @@ +2019-08-26 Mark Wielaard + + * readelf (options): Add OPTION_ARG_OPTIONAL "SECTION" for notes. + (notes_section): New global variable. + (parse_opt): Set notes_section. + (handle_notes): Check if notes_section is set. + 2019-07-26 Florian Weimer Mark Wielaard diff --git a/src/readelf.c b/src/readelf.c index 2084fb1f..24be7a9a 100644 --- a/src/readelf.c +++ b/src/readelf.c @@ -114,7 +114,7 @@ static const struct argp_option options[] = { "symbols", 's', "SECTION", OPTION_ARG_OPTIONAL, N_("Display the symbol table sections"), 0 }, { "version-info", 'V', NULL, 0, N_("Display versioning information"), 0 }, - { "notes", 'n', NULL, 0, N_("Display the ELF notes"), 0 }, + { "notes", 'n', "SECTION", OPTION_ARG_OPTIONAL, N_("Display the ELF notes"), 0 }, { "arch-specific", 'A', NULL, 0, N_("Display architecture specific information, if any"), 0 }, { "exception", 'e', NULL, 0, @@ -190,6 +190,9 @@ static bool print_symbol_table; /* A specific section name, or NULL to print all symbol tables. */ static char *symbol_table_section; +/* A specific section name, or NULL to print all ELF notes. */ +static char *notes_section; + /* True if the version information should be printed. */ static bool print_version_info; @@ -439,6 +442,7 @@ parse_opt (int key, char *arg, case 'n': print_notes = true; any_control_option = true; + notes_section = arg; break; case 'r': print_relocations = true; @@ -12408,6 +12412,13 @@ handle_notes (Ebl *ebl, GElf_Ehdr *ehdr) /* Not what we are looking for. */ continue; + if (notes_section != NULL) + { + char *sname = elf_strptr (ebl->elf, shstrndx, shdr->sh_name); + if (sname == NULL || strcmp (sname, notes_section) != 0) + continue; + } + printf (gettext ("\ \nNote section [%2zu] '%s' of %" PRIu64 " bytes at offset %#0" PRIx64 ":\n"), elf_ndxscn (scn), -- 2.18.1
Re: Accepting GNU Free Documentation Licensed content
Hi Ben, On Tue, 2019-08-20 at 12:18 -0700, Ben Coyote Woodard wrote: > What do you guys think of accepting derived works based upon GNU Free > Documentation Licensed content? https://www.gnu.org/licenses/fdl-1.3.en.html > > As far as I can tell, allowing a project like elfutils the freedom to > fork and derive specialized content is what this license was intended to > do and elfutils use of the various GPL licenses expresses an overall > intention to not restrict user's freedom. The GFDL is in line with that > intention. As long as it has "with no Invariant Sections, with no Front-Cover Texts, and with no Back-Cover Texts" I am fine with GFDL. Otherwise it is a pain to share and an argument could be made (as e.g. Debian does) that it isn't really a Free license. The DWARF Standard Document is also published under the GFDL 1.3, so we could then also freely use some standard texts. That said, I actually like the documentation being under the same license as the code (GPL) since that makes it easy to share documentation and code. e.g. We could generate an initial set of man pages for the tools with https://www.gnu.org/software/help2man/ and then copy/paste changes between the man pages and the source code. Which isn't possible if we use different incompatible licenses between documentation and code. The counter argument of course is that a manual describes something different (better) than the simple code docs/help does. So sharing code and documentation might not happen all that much in the first place. > e.g. there is readelf and eu-readelf is designed to take the same > options and work the same way but there may be a couple of differences. > Why don't we just fork the readelf man page and describe any differences > that there may be. How are those manuals generated? Do you want to keep sharing/updating them from binutils? Or just use them as initial templates? It would be good to see which differences there are in options. We try to not be deliberately incompatible, but I believe there are some (accidental) incompatibilities anyway. Cheers, Mark
Re: [PATCH] libdw: add thread-safety to dwarf_getabbrev()
Hi Jonathon and Srđan, On Fri, 2019-08-16 at 14:24 -0500, Jonathon Anderson wrote: > For parallel applications that need the information in the DIEs, the > Dwarf_Abbrev hash table et al. become a massive data race. This fixes > that by: > > 1. Adding atomics & locks to the hash table to manage concurrency >(lib/dynamicsizehash_concurrent.{c,h}) > 2. Adding a lock & array structure to the memory manager (pseudo-TLS) >(libdwP.h, libdw_alloc.c) > 3. Adding extra configure options for Helgrind/DRD annotations >(configure.ac) > 4. Including "stdatomic.h" from FreeBSD, to support C11-style atomics. >(lib/stdatomic.h) This looks like really nice work. Thanks! I am splitting review in some smaller parts if you don't mind. Simply because it is large and I cannot keep everything in my head at once :) But here some initial overall comments. > Notes: > - GCC >= 4.9 provides natively; for those versions >lib/stdatomic.h could be removed or disabled. We can also rewrite the >file if the copyright becomes an issue. If the compiler provides stdatomic.h then I think it would be good to use that instead of our own implementation. The copyright isn't a problem. But do you have a reference/URL to the upstream version? I like to add that somewhere, so we can sync with it in the future. I see various commented out parts. Was that already upstream? Should we just remove those parts? > - Currently the concurrent hash table is always enabled, >performance-wise there is no known difference between it >and the non-concurrent version. >This can be changed to toggle with --enable-thread-safety >if preferred. I would prefer it always enabled, unless there is a massive slowdown of the single-threaded case. The problem with --enable-thread-safety is that it is a) known broken (sigh) and b) it basically introduces two separate libraries that behave subtly differently. I would very much like to get rid of --enable-thread-safety by fixing the broken locking and simply making it the default. > - Another implementation of #2 above might use dynamic TLS >(pthread_key_*), >we chose this implementation to reduce the overall complexity. Are there any other trade-offs? Thanks, Mark
Re: [PATCH] libdw: add thread-safety to dwarf_getabbrev()
First message failed to send, hopefully this one works... On Wed, Aug 21, 2019 at 6:16 AM, Mark Wielaard wrote: Hi Jonathon and Srđan, On Fri, 2019-08-16 at 14:24 -0500, Jonathon Anderson wrote: For parallel applications that need the information in the DIEs, the Dwarf_Abbrev hash table et al. become a massive data race. This fixes that by: 1. Adding atomics & locks to the hash table to manage concurrency (lib/dynamicsizehash_concurrent.{c,h}) 2. Adding a lock & array structure to the memory manager (pseudo-TLS) (libdwP.h, libdw_alloc.c) 3. Adding extra configure options for Helgrind/DRD annotations (configure.ac) 4. Including "stdatomic.h" from FreeBSD, to support C11-style atomics. (lib/stdatomic.h) This looks like really nice work. Thanks! I am splitting review in some smaller parts if you don't mind. Simply because it is large and I cannot keep everything in my head at once :) But here some initial overall comments. Thank you for looking over it! Comments on comments below. Notes: - GCC >= 4.9 provides natively; for those versions lib/stdatomic.h could be removed or disabled. We can also rewrite the file if the copyright becomes an issue. If the compiler provides stdatomic.h then I think it would be good to use that instead of our own implementation. The copyright isn't a problem. But do you have a reference/URL to the upstream version? I like to add that somewhere, so we can sync with it in the future. I see various commented out parts. Was that already upstream? Should we just remove those parts? It would definitely be preferable to use the compiler's implementation if possible, we used this in case GCC 4.7 and 4.8 (RHEL7) compatibility was needed. If those versions are old enough the file can be removed entirely. The upstream is at https://github.com/freebsd/freebsd/blob/master/sys/sys/stdatomic.h, although the version here appears to be slightly modified (we used the version that HPCToolkit ships). The components we use don't seem affected, so a resync shouldn't make a difference. - Currently the concurrent hash table is always enabled, performance-wise there is no known difference between it and the non-concurrent version. This can be changed to toggle with --enable-thread-safety if preferred. I would prefer it always enabled, unless there is a massive slowdown of the single-threaded case. The problem with --enable-thread-safety is that it is a) known broken (sigh) and b) it basically introduces two separate libraries that behave subtly differently. I would very much like to get rid of --enable-thread-safety by fixing the broken locking and simply making it the default. I haven't noticed any slowdown in the single-threaded case, although I haven't stressed it hard enough to find out for certain. From a theoretical standpoint it shouldn't, atomics (with the proper memory orders) are usually (on x86 at least) as cheap as normal accesses when used by a single thread, and the algorithm is otherwise effectively the same as the original hash table. How difficult would it be to fix the locking (or rather, what's "broken")? We would definitely benefit from having thread-safety at least for getters, which would only need locks around the internal management. - Another implementation of #2 above might use dynamic TLS (pthread_key_*), we chose this implementation to reduce the overall complexity. Are there any other trade-offs? If the application spawns N threads that all use a Dwarf object (same or different) enough to cause an allocation, and then joins them all, any Dwarf objects allocated after will allocate N unusable slots in the mem_tails array. Thus long-running applications (that don't use thread pools) would start experiencing effects similar to a memory leak, of 1 pointer's worth (8 bytes on 64-bit) per dead thread. The alternative is to use dynamic TLS so that only threads that are currently active use the extra memory, assuming libpthread is sufficiently proactive about reclaiming unused key values. I think if we assume `dwarf_end` happens-after any memory management (which would make sense for a destructor), there should be a simple atomic pattern to handle the freeing, but I would need to sit down for a while to work out a sufficient proof. I was also under the impression that dynamic TLS was particularly expensive performance-wise, although I haven't experimented with it myself enough to know. The compiler can be a little smarter about static TLS, and the result is easier to reason about, which is why we chose this implementation for initial patch. If the alternative would be preferable we can change it. Thanks, Mark
Re: Accepting GNU Free Documentation Licensed content
On 8/21/19 3:50 AM, Mark Wielaard wrote: Hi Ben, On Tue, 2019-08-20 at 12:18 -0700, Ben Coyote Woodard wrote: What do you guys think of accepting derived works based upon GNU Free Documentation Licensed content? https://www.gnu.org/licenses/fdl-1.3.en.html As far as I can tell, allowing a project like elfutils the freedom to fork and derive specialized content is what this license was intended to do and elfutils use of the various GPL licenses expresses an overall intention to not restrict user's freedom. The GFDL is in line with that intention. As long as it has "with no Invariant Sections, with no Front-Cover Texts, and with no Back-Cover Texts" I am fine with GFDL. Otherwise it is a pain to share and an argument could be made (as e.g. Debian does) that it isn't really a Free license. Yeah the man pages simply reference the main GDFL. The DWARF Standard Document is also published under the GFDL 1.3, so we could then also freely use some standard texts. That said, I actually like the documentation being under the same license as the code (GPL) since that makes it easy to share documentation and code. e.g. We could generate an initial set of man pages for the tools with https://www.gnu.org/software/help2man/ and then copy/paste changes between the man pages and the source code. Which isn't possible if we use different incompatible licenses between documentation and code. The counter argument of course is that a manual describes something different (better) than the simple code docs/help does. So sharing code and documentation might not happen all that much in the first place. I find this argument much more convincing. There are formatting issues such as the length of the text and the supported man page sections which are not part of the usage text. I therefore think that the man pages should be independent from the usage text. e.g. there is readelf and eu-readelf is designed to take the same options and work the same way but there may be a couple of differences. Why don't we just fork the readelf man page and describe any differences that there may be. How are those manuals generated? They are using perl's pod format and POD::MAN, -- yet another markup format. I know that you ultimately would prefer sphinx. Given the excessive number of markup formats, I'm disinclined to use any of them. They are all johnny come latelys and hopefully most of them will die off and be merged into each other. Using any particular one just perpetuates its existence by increasing its installed base. *roff on the other hand has been around for generations. It will only die when man is extended to directly read some more modern format. (Why hasn't someone done this yet? ...) Do you want to keep sharing/updating them from binutils? Or just use them as initial templates? My thought was to create a handful of stubs which just point to analogous binutils pages and then compare the binutils pages to what the elfutils actually do. If I find any differences, then I was planning on forking the binutils version of the man page with just the minimal modifications so that it has an obvious fork point, which is diff-able and evolutionary history. This would allow us to effectively cherry pick changes when it suits us. It would be good to see which differences there are in options. We try to not be deliberately incompatible, but I believe there are some (accidental) incompatibilities anyway. Right. I think carefully going through the options and the man page should help surface these. I think not having documentation kind of put you in a straight jacket. Since anything beyond --help was implicitly assumed to be documented by binutils's man pages, you were required to stay in sync. Having your own documentation will allow Elfutils to kick off the dusty shackles of binutils and intelligently move forward to support a broader range of use cases in this modern era. Cheers, Mark
Re: [PATCH] libdw: add thread-safety to dwarf_getabbrev()
On Fri, 2019-08-16 at 14:24 -0500, Jonathon Anderson wrote: > diff --git a/ChangeLog b/ChangeLog > index bed3999f..93907ddd 100644 > --- a/ChangeLog > +++ b/ChangeLog > @@ -1,3 +1,8 @@ > +2019-08-15 Jonathon Anderson > + > + * configure.ac: Add new --enable-valgrind-annotations > + * configure.ac: Add new --with-valgrind (headers only) > + > 2019-08-13 Mark Wielaard > > * configure.ac: Set version to 0.177. > diff --git a/configure.ac b/configure.ac > index c443fa3b..c5406b44 100644 > --- a/configure.ac > +++ b/configure.ac > @@ -323,6 +323,35 @@ if test "$use_valgrind" = yes; then > fi > AM_CONDITIONAL(USE_VALGRIND, test "$use_valgrind" = yes) > > +AC_ARG_WITH([valgrind], > +AS_HELP_STRING([--with-valgrind],[include directory for Valgrind > headers]), > +[with_valgrind_headers=$withval], [with_valgrind_headers=no]) > +if test "x$with_valgrind_headers" != xno; then > +save_CFLAGS="$CFLAGS" > +CFLAGS="$CFLAGS -I$with_valgrind_headers" > +AC_COMPILE_IFELSE([AC_LANG_SOURCE([[ > + #include > + int main() { return 0; } > +]])], [ HAVE_VALGRIND_HEADERS="yes" > +CFLAGS="$save_CFLAGS -I$with_valgrind_headers" ], > + [ AC_MSG_ERROR([invalid valgrind include directory: > $with_valgrind_headers]) ]) > +fi > + > +AC_ARG_ENABLE([valgrind-annotations], > +AS_HELP_STRING([--enable-valgrind-annotations],[insert extra > annotations for better valgrind support]), > +[use_vg_annotations=$enableval], [use_vg_annotations=no]) > +if test "$use_vg_annotations" = yes; then > +if test "x$HAVE_VALGRIND_HEADERS" != "xyes"; then > + AC_MSG_CHECKING([whether Valgrind headers are available]) > + AC_COMPILE_IFELSE([AC_LANG_SOURCE([[ > +#include > +int main() { return 0; } > + ]])], [ AC_MSG_RESULT([yes]) ], > +[ AC_MSG_ERROR([valgrind annotations requested but no > headers are available]) ]) > +fi > +fi > +AM_CONDITIONAL(USE_VG_ANNOTATIONS, test "$use_vg_annotations" = yes) > + > AC_ARG_ENABLE([install-elfh], > AS_HELP_STRING([--enable-install-elfh],[install elf.h in include dir]), > [install_elfh=$enableval], [install_elfh=no]) > @@ -668,6 +697,7 @@ AC_MSG_NOTICE([ >OTHER FEATURES > Deterministic archives by default : ${default_ar_deterministic} > Native language support: ${USE_NLS} > +Extra Valgrind annotations : ${use_vg_annotations} > >EXTRA TEST FEATURES (used with make check) > have bunzip2 installed (required) : ${HAVE_BUNZIP2} This part sets up things so we can include extra valgrind annotations, but then doesn't seem to be used. It sounds useful though, because valgrind/helgrind won't know about any of the atomics. Is this something you added, but then removed? Thanks, Mark P.S. It looks like something decided to add some line breaks in the patch so that it doesn't easily apply. It isn't hard to fixup, but you might want to consider submitting using git send-email or attaching the result of git format-patch instead of putting the patch in the message body.
Re: [PATCH] libdw: add thread-safety to dwarf_getabbrev()
On Wed, 2019-08-21 at 23:50 +0200, Mark Wielaard wrote: > On Fri, 2019-08-16 at 14:24 -0500, Jonathon Anderson wrote: > > @@ -668,6 +697,7 @@ AC_MSG_NOTICE([ > >OTHER FEATURES > > Deterministic archives by default : > > ${default_ar_deterministic} > > Native language support: ${USE_NLS} > > +Extra Valgrind annotations : ${use_vg_annotations} > > > >EXTRA TEST FEATURES (used with make check) > > have bunzip2 installed (required) : ${HAVE_BUNZIP2} > > This part sets up things so we can include extra valgrind > annotations, > but then doesn't seem to be used. It sounds useful though, because > valgrind/helgrind won't know about any of the atomics. Is this > something you added, but then removed? Ah, I should have read to the end, sorry. I see you then use them in libdw/libdw_alloc.c as happens before/after annotations. Cheers, Mark
Re: [PATCH] libdw: add thread-safety to dwarf_getabbrev()
On Wed, Aug 21, 2019 at 4:50 PM, Mark Wielaard wrote: On Fri, 2019-08-16 at 14:24 -0500, Jonathon Anderson wrote: diff --git a/ChangeLog b/ChangeLog index bed3999f..93907ddd 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,8 @@ +2019-08-15 Jonathon Anderson + + * configure.ac: Add new --enable-valgrind-annotations + * configure.ac: Add new --with-valgrind (headers only) + 2019-08-13 Mark Wielaard * configure.ac: Set version to 0.177. diff --git a/configure.ac b/configure.ac index c443fa3b..c5406b44 100644 --- a/configure.ac +++ b/configure.ac @@ -323,6 +323,35 @@ if test "$use_valgrind" = yes; then fi AM_CONDITIONAL(USE_VALGRIND, test "$use_valgrind" = yes) +AC_ARG_WITH([valgrind], +AS_HELP_STRING([--with-valgrind],[include directory for Valgrind headers]), +[with_valgrind_headers=$withval], [with_valgrind_headers=no]) +if test "x$with_valgrind_headers" != xno; then +save_CFLAGS="$CFLAGS" +CFLAGS="$CFLAGS -I$with_valgrind_headers" +AC_COMPILE_IFELSE([AC_LANG_SOURCE([[ + #include + int main() { return 0; } +]])], [ HAVE_VALGRIND_HEADERS="yes" +CFLAGS="$save_CFLAGS -I$with_valgrind_headers" ], + [ AC_MSG_ERROR([invalid valgrind include directory: $with_valgrind_headers]) ]) +fi + +AC_ARG_ENABLE([valgrind-annotations], +AS_HELP_STRING([--enable-valgrind-annotations],[insert extra annotations for better valgrind support]), +[use_vg_annotations=$enableval], [use_vg_annotations=no]) +if test "$use_vg_annotations" = yes; then +if test "x$HAVE_VALGRIND_HEADERS" != "xyes"; then + AC_MSG_CHECKING([whether Valgrind headers are available]) + AC_COMPILE_IFELSE([AC_LANG_SOURCE([[ +#include +int main() { return 0; } + ]])], [ AC_MSG_RESULT([yes]) ], +[ AC_MSG_ERROR([valgrind annotations requested but no headers are available]) ]) +fi +fi +AM_CONDITIONAL(USE_VG_ANNOTATIONS, test "$use_vg_annotations" = yes) + AC_ARG_ENABLE([install-elfh], AS_HELP_STRING([--enable-install-elfh],[install elf.h in include dir]), [install_elfh=$enableval], [install_elfh=no]) @@ -668,6 +697,7 @@ AC_MSG_NOTICE([ OTHER FEATURES Deterministic archives by default : ${default_ar_deterministic} Native language support: ${USE_NLS} +Extra Valgrind annotations : ${use_vg_annotations} EXTRA TEST FEATURES (used with make check) have bunzip2 installed (required) : ${HAVE_BUNZIP2} This part sets up things so we can include extra valgrind annotations, but then doesn't seem to be used. It sounds useful though, because valgrind/helgrind won't know about any of the atomics. Is this something you added, but then removed? Thanks, Mark P.S. It looks like something decided to add some line breaks in the patch so that it doesn't easily apply. It isn't hard to fixup, but you might want to consider submitting using git send-email or attaching the result of git format-patch instead of putting the patch in the message body. Originally I had some issues with git send-mail, I usually do PRs purely in git so the email side is still a little new. I've attached the original patch from git format-patch, sorry for the mess. -Jonathon From 9077c0df713e9adfdee7fe1f66005453316842de Mon Sep 17 00:00:00 2001 From: Jonathon Anderson Date: Thu, 8 Aug 2019 09:01:56 -0500 Subject: [PATCH] libdw: add thread-safety to dwarf_getabbrev() MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit For applications that need the information in the DIEs, the Dwarf_Abbrev hash table et al. becomes a massive data race. This fixes that by: 1. Adding atomics & locks to the hash table to manage concurrency (lib/dynamicsizehash_concurrent.{c,h}) 2. Adding a lock & array structure the the memory manager (pseudo-TLS) (libdwP.h, libdw_alloc.c) 3. Adding extra configure options for Helgrind/DRD annotations (configure.ac) 4. Including "stdatomic.h" from FreeBSD, to support C11-style atomics. (lib/stdatomic.h) Signed-off-by: Jonathon Anderson Signed-off-by: SrÄan MilakoviÄ --- ChangeLog| 5 + configure.ac | 30 ++ lib/ChangeLog| 6 + lib/Makefile.am | 5 +- lib/dynamicsizehash_concurrent.c | 522 +++ lib/dynamicsizehash_concurrent.h | 118 +++ lib/stdatomic.h | 442 ++ libdw/ChangeLog | 9 + libdw/Makefile.am| 2 +- libdw/dwarf_abbrev_hash.c| 2 +- libdw/dwarf_abbrev_hash.h| 2 +- libdw/dwarf_begin_elf.c | 13 +- libdw/dwarf_end.c| 24 +- libdw/libdwP.h | 13 +- libdw/libdw_alloc.c | 70 - 15 files changed, 1240 insertions(+), 23 deletions(-) create mode 100644 lib/dynamicsizehash_concurrent.c creat