Re: [PATCH v5 0/7] debuginfod: speed up extraction from kernel debuginfo packages by 200x

2024-07-24 Thread Omar Sandoval
On Wed, Jul 24, 2024 at 06:20:21PM -0400, Aaron Merey wrote: > On Tue, Jul 23, 2024 at 6:40 PM Omar Sandoval wrote: > > > > From: Omar Sandoval > > > > This is v4 of my patch series optimizing debuginfod for kernel > > debuginfo. v1 is here [1], v2 is here [2],

[PATCH v5 5/7] debuginfod: optimize extraction from seekable xz archives

2024-07-23 Thread Omar Sandoval
From: Omar Sandoval The kernel debuginfo packages on Fedora, Debian, and Ubuntu, and many of their downstreams, are all compressed with xz in multi-threaded mode, which allows random access. We can use this to bypass the full archive extraction and dramatically speed up kernel debuginfo

[PATCH v5 4/7] debugifod: add new table and views for seekable archives

2024-07-23 Thread Omar Sandoval
From: Omar Sandoval In order to extract a file from a seekable archive, we need to know where in the uncompressed archive the file data starts and its size. Additionally, in order to populate the response headers, we need the file modification time (since we won't be able to get it fro

[PATCH v5 6/7] debuginfod: populate _r_seekable on scan

2024-07-23 Thread Omar Sandoval
From: Omar Sandoval Whenever a new archive is scanned, check if it is seekable with a little liblzma magic, and populate _r_seekable if so. With this, newly scanned seekable archives will used the optimized extraction path added in the previous commit. Also add a test case using some

[PATCH v5 3/7] debuginfod: factor out common code for responding from an archive

2024-07-23 Thread Omar Sandoval
From: Omar Sandoval handle_buildid_r_match has two very similar branches where it optionally extracts a section and then creates a microhttpd response. In preparation for adding a third one, factor it out into a function. Signed-off-by: Omar Sandoval --- debuginfod/debuginfod.cxx | 213

[PATCH v5 7/7] debuginfod: populate _r_seekable on request

2024-07-23 Thread Omar Sandoval
From: Omar Sandoval Since the schema change adding _r_seekable was done in a backward compatible way, seekable archives that were previously scanned will not be in _r_seekable. Whenever an archive is going to be extracted to satisfy a request, check if it is seekable. If so, populate

[PATCH v5 2/7] tests/run-debuginfod-fd-prefetch-caches.sh: disable fdcache limit check

2024-07-23 Thread Omar Sandoval
From: Omar Sandoval Since commit acd9525e93d7 ("PR31265 - rework debuginfod archive-extract fdcache"), the fdcache limit is only applied when a new file is interned and it has been at least 10 seconds since the limit was last applied. This means that the fdcache can go over the limit t

[PATCH v5 1/7] debuginfod: fix skipping source file

2024-07-23 Thread Omar Sandoval
From: Omar Sandoval dwarf_extract_source_paths explicitly skips source files that equal "", but dwarf_filesrc may return a path like "dir/". Check for and skip that case, too. In particular, the test debuginfod RPMs have paths like this. However, the test cases didn'

[PATCH v5 0/7] debuginfod: speed up extraction from kernel debuginfo packages by 200x

2024-07-23 Thread Omar Sandoval
From: Omar Sandoval This is v4 of my patch series optimizing debuginfod for kernel debuginfo. v1 is here [1], v2 is here [2], v3 is here [3], v4 is here [4]. The only change from v4 in this version is adding --fdcache-mbs and --fdcache-mintmp to the new test to fix some sporadic test failures

Re: [PATCH v4 0/7] debuginfod: speed up extraction from kernel debuginfo packages by 200x

2024-07-23 Thread Omar Sandoval
On Tue, Jul 23, 2024 at 05:47:50PM -0400, Aaron Merey wrote: > Hi Omar, > > On Fri, Jul 19, 2024 at 2:24 PM Omar Sandoval wrote: > > > > From: Omar Sandoval > > > > This is v4 of my patch series optimizing debuginfod for kernel > > debuginfo. v1 is

[PATCH v4 6/7] debuginfod: populate _r_seekable on scan

2024-07-19 Thread Omar Sandoval
From: Omar Sandoval Whenever a new archive is scanned, check if it is seekable with a little liblzma magic, and populate _r_seekable if so. With this, newly scanned seekable archives will used the optimized extraction path added in the previous commit. Also add a test case using some

[PATCH v4 7/7] debuginfod: populate _r_seekable on request

2024-07-19 Thread Omar Sandoval
From: Omar Sandoval Since the schema change adding _r_seekable was done in a backward compatible way, seekable archives that were previously scanned will not be in _r_seekable. Whenever an archive is going to be extracted to satisfy a request, check if it is seekable. If so, populate

[PATCH v4 5/7] debuginfod: optimize extraction from seekable xz archives

2024-07-19 Thread Omar Sandoval
From: Omar Sandoval The kernel debuginfo packages on Fedora, Debian, and Ubuntu, and many of their downstreams, are all compressed with xz in multi-threaded mode, which allows random access. We can use this to bypass the full archive extraction and dramatically speed up kernel debuginfo

[PATCH v4 3/7] debuginfod: factor out common code for responding from an archive

2024-07-19 Thread Omar Sandoval
From: Omar Sandoval handle_buildid_r_match has two very similar branches where it optionally extracts a section and then creates a microhttpd response. In preparation for adding a third one, factor it out into a function. Signed-off-by: Omar Sandoval --- debuginfod/debuginfod.cxx | 213

[PATCH v4 4/7] debugifod: add new table and views for seekable archives

2024-07-19 Thread Omar Sandoval
From: Omar Sandoval In order to extract a file from a seekable archive, we need to know where in the uncompressed archive the file data starts and its size. Additionally, in order to populate the response headers, we need the file modification time (since we won't be able to get it fro

[PATCH v4 1/7] debuginfod: fix skipping source file

2024-07-19 Thread Omar Sandoval
From: Omar Sandoval dwarf_extract_source_paths explicitly skips source files that equal "", but dwarf_filesrc may return a path like "dir/". Check for and skip that case, too. In particular, the test debuginfod RPMs have paths like this. However, the test cases didn'

[PATCH v4 2/7] tests/run-debuginfod-fd-prefetch-caches.sh: disable fdcache limit check

2024-07-19 Thread Omar Sandoval
From: Omar Sandoval Since commit acd9525e93d7 ("PR31265 - rework debuginfod archive-extract fdcache"), the fdcache limit is only applied when a new file is interned and it has been at least 10 seconds since the limit was last applied. This means that the fdcache can go over the limit t

[PATCH v4 0/7] debuginfod: speed up extraction from kernel debuginfo packages by 200x

2024-07-19 Thread Omar Sandoval
From: Omar Sandoval This is v4 of my patch series optimizing debuginfod for kernel debuginfo. v1 is here [1], v2 is here [2], v3 is here [3]. The only changes from v3 in this version are fixing a bogus maybe-uninitialized error on the Debian build and adding the new test files to EXTRA_DIST so

Re: [PATCH v3 0/7] debuginfod: speed up extraction from kernel debuginfo packages by 200x

2024-07-19 Thread Omar Sandoval
On Fri, Jul 19, 2024 at 01:34:48PM -0400, Frank Ch. Eigler wrote: > Hi - > > > This is v3 of my patch series optimizing debuginfod for kernel > > debuginfo. v1 is here [7], v2 is here [8]. This version fixes a couple > > of minor bugs and adds test cases. [...] > > Thanks, LGTM, running through

[PATCH v3 2/7] tests/run-debuginfod-fd-prefetch-caches.sh: disable fdcache limit check

2024-07-19 Thread Omar Sandoval
From: Omar Sandoval Since commit acd9525e93d7 ("PR31265 - rework debuginfod archive-extract fdcache"), the fdcache limit is only applied when a new file is interned and it has been at least 10 seconds since the limit was last applied. This means that the fdcache can go over the limit t

[PATCH v3 3/7] debuginfod: factor out common code for responding from an archive

2024-07-19 Thread Omar Sandoval
From: Omar Sandoval handle_buildid_r_match has two very similar branches where it optionally extracts a section and then creates a microhttpd response. In preparation for adding a third one, factor it out into a function. Signed-off-by: Omar Sandoval --- debuginfod/debuginfod.cxx | 213

[PATCH v3 1/7] debuginfod: fix skipping source file

2024-07-19 Thread Omar Sandoval
From: Omar Sandoval dwarf_extract_source_paths explicitly skips source files that equal "", but dwarf_filesrc may return a path like "dir/". Check for and skip that case, too. In particular, the test debuginfod RPMs have paths like this. However, the test cases didn'

[PATCH v3 6/7] debuginfod: populate _r_seekable on scan

2024-07-19 Thread Omar Sandoval
From: Omar Sandoval Whenever a new archive is scanned, check if it is seekable with a little liblzma magic, and populate _r_seekable if so. With this, newly scanned seekable archives will used the optimized extraction path added in the previous commit. Also add a test case using some

[PATCH v3 5/7] debuginfod: optimize extraction from seekable xz archives

2024-07-19 Thread Omar Sandoval
From: Omar Sandoval The kernel debuginfo packages on Fedora, Debian, and Ubuntu, and many of their downstreams, are all compressed with xz in multi-threaded mode, which allows random access. We can use this to bypass the full archive extraction and dramatically speed up kernel debuginfo

[PATCH v3 7/7] debuginfod: populate _r_seekable on request

2024-07-19 Thread Omar Sandoval
From: Omar Sandoval Since the schema change adding _r_seekable was done in a backward compatible way, seekable archives that were previously scanned will not be in _r_seekable. Whenever an archive is going to be extracted to satisfy a request, check if it is seekable. If so, populate

[PATCH v3 4/7] debugifod: add new table and views for seekable archives

2024-07-19 Thread Omar Sandoval
From: Omar Sandoval In order to extract a file from a seekable archive, we need to know where in the uncompressed archive the file data starts and its size. Additionally, in order to populate the response headers, we need the file modification time (since we won't be able to get it fro

[PATCH v3 0/7] debuginfod: speed up extraction from kernel debuginfo packages by 200x

2024-07-19 Thread Omar Sandoval
From: Omar Sandoval This is v3 of my patch series optimizing debuginfod for kernel debuginfo. v1 is here [7], v2 is here [8]. This version fixes a couple of minor bugs and adds test cases. Changes from v2 to v3: - Added a test case with seekable rpm and deb files. - Added a couple of

Re: [PATCH v2 0/5] debuginfod: speed up extraction from kernel debuginfo packages by 200x

2024-07-16 Thread Omar Sandoval
On Tue, Jul 16, 2024 at 06:15:16PM -0400, Frank Ch. Eigler wrote: > Hi - > > > [...] I'll send it through the testsuite > > trybots here. [...] > > There was some success and there was some failure. :-) > > all 11 runs: > > https://builder.sourceware.org/testruns/?commitishes=&has_expfile_glob

Re: [PATCH v2 0/5] debuginfod: speed up extraction from kernel debuginfo packages by 200x

2024-07-16 Thread Omar Sandoval
On Tue, Jul 16, 2024 at 04:16:01PM -0400, Frank Ch. Eigler wrote: > Hi - > > > This is v2 of my patch series optimizing debuginfod for kernel > > debuginfo. v1 is here [7]. > > This generally looks great to me. I'll send it through the testsuite > trybots here. Great, thank you! > But there i

[PATCH v2 5/5] debuginfod: populate _r_seekable on request

2024-07-15 Thread Omar Sandoval
From: Omar Sandoval Since the schema change adding _r_seekable was done in a backward compatible way, seekable archives that were previously scanned will not be in _r_seekable. Whenever an archive is going to be extracted to satisfy a request, check if it is seekable. If so, populate

[PATCH v2 4/5] debuginfod: populate _r_seekable on scan

2024-07-15 Thread Omar Sandoval
From: Omar Sandoval Whenever a new archive is scanned, check if it is seekable with a little liblzma magic, and populate _r_seekable if so. With this, newly scanned seekable archives will used the optimized extraction path added in the previous commit. Signed-off-by: Omar Sandoval

[PATCH v2 3/5] debuginfod: optimize extraction from seekable xz archives

2024-07-15 Thread Omar Sandoval
From: Omar Sandoval The kernel debuginfo packages on Fedora, Debian, and Ubuntu, and many of their downstreams, are all compressed with xz in multi-threaded mode, which allows random access. We can use this to bypass the full archive extraction and dramatically speed up kernel debuginfo

[PATCH v2 1/5] debuginfod: factor out common code for responding from an archive

2024-07-15 Thread Omar Sandoval
From: Omar Sandoval handle_buildid_r_match has two very similar branches where it optionally extracts a section and then creates a microhttpd response. In preparation for adding a third one, factor it out into a function. Signed-off-by: Omar Sandoval --- debuginfod/debuginfod.cxx | 213

[PATCH v2 0/5] debuginfod: speed up extraction from kernel debuginfo packages by 200x

2024-07-15 Thread Omar Sandoval
From: Omar Sandoval This is v2 of my patch series optimizing debuginfod for kernel debuginfo. v1 is here [7]. The main change from v1 is reworking the database changes to be backward compatible and therefore not require reindexing. Patch 1 is a preparatory refactor. Patch 2 makes the schema

[PATCH v2 2/5] debugifod: add new table and views for seekable archives

2024-07-15 Thread Omar Sandoval
From: Omar Sandoval In order to extract a file from a seekable archive, we need to know where in the uncompressed archive the file data starts and its size. Additionally, in order to populate the response headers, we need the file modification time (since we won't be able to get it fro

Re: [PATCH 0/3] debuginfod: speed up extraction from kernel debuginfo packages by 200x

2024-07-11 Thread Omar Sandoval
On Thu, Jul 11, 2024 at 04:16:25PM -0400, Frank Ch. Eigler wrote: > Hi, Omar - > > Thanks. I wish this sort of amazing kludge weren't necessary, but > given that it helps, so be it. > > I'd like to commend you on the effort needed to match your code up > with the stylistic idiosyncracies of the

[PATCH 3/3] debuginfod: optimize extraction from seekable xz archives

2024-07-10 Thread Omar Sandoval
From: Omar Sandoval The kernel debuginfo packages on Fedora, Debian, and Ubuntu, and many of their downstreams, are all compressed with xz in multi-threaded mode, which allows random access. We can use this to bypass the full archive extraction and dramatically speed up kernel debuginfo

[PATCH 2/3] debuginfod: add archive entry size, mtime, and uncompressed offset to database

2024-07-10 Thread Omar Sandoval
From: Omar Sandoval In order to extract a file from a seekable archive, we need to know where in the uncompressed archive the file data starts and its size. Additionally, in order to populate the response headers, we need the file modification time (since we won't be able to get it fro

[PATCH 1/3] debuginfod: factor out common code for responding from an archive

2024-07-10 Thread Omar Sandoval
From: Omar Sandoval handle_buildid_r_match has two very similar branches where it optionally extracts a section and then creates a microhttpd response. In preparation for adding a third one, factor it out into a function. Signed-off-by: Omar Sandoval --- debuginfod/debuginfod.cxx | 213

[PATCH 0/3] debuginfod: speed up extraction from kernel debuginfo packages by 200x

2024-07-10 Thread Omar Sandoval
From: Omar Sandoval drgn [1] currently uses debuginfod with great success for debugging userspace processes. However, for debugging the Linux kernel (drgn's main use case), we have had some performance issues with debuginfod, so we intentionally avoid using it. Specifically, it sometimes

Re: [PATCH v3 4/4] libdw: Handle overflowed DW_SECT_INFO offsets in DWARF package file indexes

2024-03-01 Thread Omar Sandoval
On Fri, Mar 01, 2024 at 03:59:22PM +0100, Mark Wielaard wrote: > Hi Omar, > > On Mon, 2024-02-26 at 11:32 -0800, Omar Sandoval wrote: > > Meta uses DWARF package files for our large, statically-linked C++ > > applications. Some of our largest applications

[PATCH v3 2/4] libdw: Refactor dwarf_next_lines and fix skipped CU

2024-02-26 Thread Omar Sandoval
From: Omar Sandoval dwarf_next_lines has two loops over CUs: one from the CU after the given CU to the end, and one from the first CU up to _but not including_ the given CU. This means that the given CU is never checked. This is unlikely to matter in practice since CUs usually correspond 1:1

[PATCH v3 4/4] libdw: Handle overflowed DW_SECT_INFO offsets in DWARF package file indexes

2024-02-26 Thread Omar Sandoval
From: Omar Sandoval Meta uses DWARF package files for our large, statically-linked C++ applications. Some of our largest applications have more than 4GB in .debug_info.dwo, but the section offsets in .debug_cu_index and .debug_tu_index are 32 bits; see the discussion here [1]. We implemented a

[PATCH v3 3/4] libdw: Apply DWARF package file section offsets where appropriate

2024-02-26 Thread Omar Sandoval
From: Omar Sandoval The final piece of DWARF package file support is that offsets have to be interpreted relative to the section offset from the package index. .debug_abbrev.dwo is already covered, so sprinkle around calls to dwarf_cu_dwp_section_info for the remaining sections: .debug_line.dwo

[PATCH v3 1/4] libdw: Handle split DWARF in dwarf_decl_file

2024-02-26 Thread Omar Sandoval
From: Omar Sandoval Calling dwarf_decl_file on a split DWARF DIE fails this assertion: dwarf_decl_file.c:72: dwarf_decl_file: Assertion `cu->files != NULL && cu->files != (void *) -1l' failed. This is because dwarf_decl_file calls dwarf_getsrclines to populate cu->fi

[PATCH v3 0/4] elfutils: DWARF package (.dwp) file support

2024-02-26 Thread Omar Sandoval
From: Omar Sandoval Hi, This is version 3 of my patch series adding support for DWARF package files to libdw and the elfutils tools. Version 2 is here [1], and version 1 is [here]. This version fixes handling of line number information. Patches 1 and 2 are new in this version. Patch 1 fixes

Re: [PATCH v2 3/4] libdw: Apply DWARF package file section offsets where appropriate

2024-02-26 Thread Omar Sandoval
On Sat, Feb 24, 2024 at 03:00:04PM +0100, Mark Wielaard wrote: > Hi Omar, > > On Thu, Feb 22, 2024 at 05:03:44PM -0800, Omar Sandoval wrote: > > On Thu, Feb 22, 2024 at 04:53:19PM -0800, Omar Sandoval wrote: > > > On Fri, Feb 16, 2024 at 04:00:47PM +0100, Mark Wielaard

Re: [PATCH v2 3/4] libdw: Apply DWARF package file section offsets where appropriate

2024-02-22 Thread Omar Sandoval
On Thu, Feb 22, 2024 at 04:53:19PM -0800, Omar Sandoval wrote: > On Fri, Feb 16, 2024 at 04:00:47PM +0100, Mark Wielaard wrote: > > Hi Omar, > > > > On Wed, 2023-12-06 at 01:22 -0800, Omar Sandoval wrote: > > > The final piece of DWARF package file supp

Re: [PATCH v2 3/4] libdw: Apply DWARF package file section offsets where appropriate

2024-02-22 Thread Omar Sandoval
On Fri, Feb 16, 2024 at 04:00:47PM +0100, Mark Wielaard wrote: > Hi Omar, > > On Wed, 2023-12-06 at 01:22 -0800, Omar Sandoval wrote: > > The final piece of DWARF package file support is that offsets have to be > > interpreted relative to the section offset f

Re: [PATCH] libdw: Update dwarf_cu_dwp_section_info documentation

2024-02-22 Thread Omar Sandoval
On Fri, Feb 16, 2024 at 02:34:18PM +0100, Mark Wielaard wrote: > Update the documentation of dwarf_cu_dwp_section_info to make clear > that the function only returns an error if the DWARF package file data > couldn't be read or an unknown section constant is provided. Missing > DWP information for

Re: [PATCH v2 0/4] elfutils: DWARF package (.dwp) file support

2024-01-03 Thread Omar Sandoval
On Wed, Dec 06, 2023 at 01:22:15AM -0800, Omar Sandoval wrote: > From: Omar Sandoval > > Hi, > > This is version 2 of my patch series adding support for DWARF package > files to libdw and the elfutils tools. Version 1 is here [1]. > > Patches 1-3 add the main implemen

[PATCH v2 2/4] libdw: Try .dwp file in __libdw_find_split_unit()

2023-12-06 Thread Omar Sandoval
From: Omar Sandoval Try opening the file in the location suggested by the standard (the skeleton file name + ".dwp") and looking up the unit in the package index. The rest is similar to .dwo files, with slightly different cleanup since a single Dwarf handle is shared.

[PATCH v2 4/4] libdw: Handle overflowed DW_SECT_INFO offsets in DWARF package file indexes

2023-12-06 Thread Omar Sandoval
From: Omar Sandoval Meta uses DWARF package files for our large, statically-linked C++ applications. Some of our largest applications have more than 4GB in .debug_info.dwo, but the section offsets in .debug_cu_index and .debug_tu_index are 32 bits; see the discussion here [1]. We implemented a

[PATCH v2 3/4] libdw: Apply DWARF package file section offsets where appropriate

2023-12-06 Thread Omar Sandoval
From: Omar Sandoval The final piece of DWARF package file support is that offsets have to be interpreted relative to the section offset from the package index. .debug_abbrev.dwo is already covered, so sprinkle around calls to dwarf_cu_dwp_section_info for the remaining sections: .debug_line.dwo

[PATCH v2 1/4] libdw: Parse DWARF package file index sections

2023-12-06 Thread Omar Sandoval
From: Omar Sandoval The .debug_cu_index and .debug_tu_index sections in DWARF package files are basically hash tables mapping a unit's 8 byte signature to an offset and size in each section used by that unit [1]. Add support for parsing and doing lookups in the index sections. We look

[PATCH v2 0/4] elfutils: DWARF package (.dwp) file support

2023-12-06 Thread Omar Sandoval
From: Omar Sandoval Hi, This is version 2 of my patch series adding support for DWARF package files to libdw and the elfutils tools. Version 1 is here [1]. Patches 1-3 add the main implementation and tests for dwp files. Most of this support is internal to libdw, but patch 1 adds a new public

Re: [PATCH 00/14] elfutils: DWARF package (.dwp) file support

2023-11-07 Thread Omar Sandoval
On Fri, Nov 03, 2023 at 12:05:57AM +0100, Mark Wielaard wrote: > Hi Omar, > > On Wed, Sep 27, 2023 at 11:20:49AM -0700, Omar Sandoval wrote: > > This patch series adds support for DWARF package files to libdw and the > > elfutils tools. It supports the GNU DebugFission

Re: [PATCH 13/14] libdw: Apply DWARF package file section offsets where appropriate

2023-11-07 Thread Omar Sandoval
On Thu, Nov 02, 2023 at 11:20:33PM +0100, Mark Wielaard wrote: > Hi Omar, > > On Wed, Sep 27, 2023 at 11:21:02AM -0700, Omar Sandoval wrote: > > The final piece of DWARF package file support is that offsets have to be > > interpreted relative to the section offset f

Re: [PATCH 10/14] libdw: Try .dwp file in __libdw_find_split_unit()

2023-11-06 Thread Omar Sandoval
On Thu, Nov 02, 2023 at 08:56:14PM +0100, Mark Wielaard wrote: > Hi Omar, > > On Wed, Sep 27, 2023 at 11:20:59AM -0700, Omar Sandoval wrote: > > Try opening the file in the location suggested by the standard (the > > skeleton file name + ".dwp") and looking up th

Re: [PATCH 09/14] libdw, libdwfl: Save original path of ELF file

2023-11-06 Thread Omar Sandoval
On Thu, Nov 02, 2023 at 06:04:27PM +0100, Mark Wielaard wrote: > Hi Omar, > > On Wed, 2023-09-27 at 11:20 -0700, Omar Sandoval wrote: > > libdw and libdwfl currently save the path of the directory containing > > the ELF file to use when searching for alt and dwo files.

Re: [PATCH 08/14] libdw: Parse DWARF package file index sections

2023-11-06 Thread Omar Sandoval
On Thu, Nov 02, 2023 at 12:07:04AM +0100, Mark Wielaard wrote: > Hi Omar, > > On Wed, Sep 27, 2023 at 11:20:57AM -0700, Omar Sandoval wrote: > > The .debug_cu_index and .debug_tu_index sections in DWARF package files > > are basically hash tables mapping a unit's 8 b

Re: [PATCH 07/14] libdw: Recognize .debug_[ct]u_index sections in dwarf_elf_begin

2023-11-01 Thread Omar Sandoval
On Wed, Nov 01, 2023 at 03:03:57PM +0100, Mark Wielaard wrote: > Hi Omar, > > On Wed, 2023-09-27 at 11:20 -0700, Omar Sandoval wrote: > > From: Omar Sandoval > > > > DWARF package (.dwp) files have a .debug_cu_index section and, > > optionally, a .debug_tu_ind

Re: [PATCH 00/14] elfutils: DWARF package (.dwp) file support

2023-09-27 Thread Omar Sandoval
On Wed, Sep 27, 2023 at 03:20:40PM -0400, Frank Ch. Eigler wrote: > Hi - > > > This patch series adds support for DWARF package files to libdw and the > > elfutils tools. It supports the GNU DebugFission format for DWARF 4 [1] > > and the format standardized in DWARF 5 (section 7.3.5 "DWARF Packag

[PATCH 13/14] libdw: Apply DWARF package file section offsets where appropriate

2023-09-27 Thread Omar Sandoval
From: Omar Sandoval The final piece of DWARF package file support is that offsets have to be interpreted relative to the section offset from the package index. .debug_abbrev.dwo is already covered, so sprinkle around calls to dwarf_cu_dwp_section_info for the remaining sections: .debug_line.dwo

[PATCH 08/14] libdw: Parse DWARF package file index sections

2023-09-27 Thread Omar Sandoval
From: Omar Sandoval The .debug_cu_index and .debug_tu_index sections in DWARF package files are basically hash tables mapping a unit's 8 byte signature to an offset and size in each section used by that unit [1]. Add support for parsing and doing lookups in the index sections. We look

[PATCH 12/14] tests: Optionally dump all units in dwarf-getmacros

2023-09-27 Thread Omar Sandoval
From: Omar Sandoval If instead of a CU offset an empty string is given as the second argument, dump all units. Signed-off-by: Omar Sandoval --- tests/ChangeLog | 3 +++ tests/dwarf-getmacros.c | 51 + 2 files changed, 44 insertions(+), 10

[PATCH 11/14] tests: Handle DW_MACRO_{define, undef}_{strx, sup} in dwarf-getmacros

2023-09-27 Thread Omar Sandoval
From: Omar Sandoval Signed-off-by: Omar Sandoval --- tests/ChangeLog | 3 +++ tests/dwarf-getmacros.c | 4 2 files changed, 7 insertions(+) diff --git a/tests/ChangeLog b/tests/ChangeLog index 687a9f32..4380c57f 100644 --- a/tests/ChangeLog +++ b/tests/ChangeLog @@ -14,6 +14,9

[PATCH 14/14] libdw: Handle overflowed DW_SECT_INFO offsets in DWARF package file indexes

2023-09-27 Thread Omar Sandoval
From: Omar Sandoval Meta uses DWARF package files for our large, statically-linked C++ applications. Some of our largest applications have more than 4GB in .debug_info.dwo, but the section offsets in .debug_cu_index and .debug_tu_index are 32 bits; see the discussion here [1]. We implemented a

[PATCH 10/14] libdw: Try .dwp file in __libdw_find_split_unit()

2023-09-27 Thread Omar Sandoval
From: Omar Sandoval Try opening the file in the location suggested by the standard (the skeleton file name + ".dwp") and looking up the unit in the package index. The rest is similar to .dwo files, with slightly different cleanup since a single Dwarf handle is shared. Signed-of

[PATCH 09/14] libdw, libdwfl: Save original path of ELF file

2023-09-27 Thread Omar Sandoval
From: Omar Sandoval libdw and libdwfl currently save the path of the directory containing the ELF file to use when searching for alt and dwo files. To search for dwp files, we need the file name too. Add an elfpath field to Dwarf, and set the debugdir field from it. Also update libdwfl to set

[PATCH 04/14] libdw: Handle other string forms in dwarf_macro_param2

2023-09-27 Thread Omar Sandoval
From: Omar Sandoval dwarf_getmacros handles the additional macro string forms added by DWARF 5, but dwarf_macro_param2 doesn't. Update it with the list of all string forms allowed in .debug_macro. In particular, GCC and Clang generate DW_MACRO_define_strx and DW_MACRO_undef_strx,

[PATCH 07/14] libdw: Recognize .debug_[ct]u_index sections in dwarf_elf_begin

2023-09-27 Thread Omar Sandoval
From: Omar Sandoval DWARF package (.dwp) files have a .debug_cu_index section and, optionally, a .debug_tu_index section. Add them to the list of DWARF sections. Unfortunately, it's not that simple: the other debug sections in a dwp file have names ending with .dwo, which confuses the c

[PATCH 03/14] libdw: Handle DW_AT_ranges in split DWARF 5 skeleton in dwarf_ranges

2023-09-27 Thread Omar Sandoval
From: Omar Sandoval When commit 879f3a4f99df ("libdw: Handle .debug_rnglists in dwarf_ranges.") added support for split DWARF 5 in 2018, GCC put all range lists for split DWARF in the .debug_rnglists section of the skeleton file (similarly to GNU DebugFission, which puts all rang

[PATCH 05/14] libdw: Fix dwarf_macro_getsrcfiles for DWARF 5

2023-09-27 Thread Omar Sandoval
From: Omar Sandoval Dwarf_Macro_Op_Table::is_64bit conflates the address size and the offset size: for .debug_macinfo, it is initialized based on the compilation unit's address size, but for .debug_macro, it is initialized based on the macro unit's offset size. is_64bit is used to det

[PATCH 06/14] libdw: Handle split DWARF in dwarf_macro_getsrcfiles

2023-09-27 Thread Omar Sandoval
From: Omar Sandoval Macro information references file names from the line number information table, which is tricky in split DWARF for a couple of reasons. First, the line number information for a macro unit comes from the .debug_line.dwo section in the split file, not the .debug_line section

[PATCH 02/14] libdw: Handle split DWARF in dwarf_entrypc

2023-09-27 Thread Omar Sandoval
From: Omar Sandoval If a DIE has no DW_AT_entry_pc attribute, dwarf_entrypc looks for DW_AT_low_pc in that DIE. But for a split compilation unit DIE, DW_AT_low_pc is in the corresponding skeleton DIE, so this fails. dwarf_lowpc already handles this fallback, so use it instead. Signed-off-by

[PATCH 00/14] elfutils: DWARF package (.dwp) file support

2023-09-27 Thread Omar Sandoval
From: Omar Sandoval Hi, This patch series adds support for DWARF package files to libdw and the elfutils tools. It supports the GNU DebugFission format for DWARF 4 [1] and the format standardized in DWARF 5 (section 7.3.5 "DWARF Package Files"). It supports both automatically openin

[PATCH 01/14] libdw: Make try_split_file static

2023-09-27 Thread Omar Sandoval
From: Omar Sandoval It's only used in libdw_find_split_unit.c. Signed-off-by: Omar Sandoval --- libdw/ChangeLog | 4 libdw/libdw_find_split_unit.c | 2 +- 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/libdw/ChangeLog b/libdw/ChangeLog index 406

Re: [PATCH] libelf, readelf, elflint: Add RELR support

2023-08-24 Thread Omar Sandoval
On Thu, Aug 24, 2023 at 09:51:54PM +0200, Mark Wielaard wrote: > Hi Omar, > > On Thu, Aug 24, 2023 at 12:40:19PM -0700, Omar Sandoval wrote: > > On Thu, Aug 24, 2023 at 12:02:11AM +0200, Mark Wielaard wrote: > > > > > > * libelf/gelf.h (Gelf_

Re: [PATCH] libelf, readelf, elflint: Add RELR support

2023-08-24 Thread Omar Sandoval
On Thu, Aug 24, 2023 at 12:02:11AM +0200, Mark Wielaard wrote: > Handle RELR as defined here: > https://groups.google.com/g/generic-abi/c/bX460iggiKg/m/YT2RrjpMAwAJ > > Introduce new ELF_T_RELR Elf_Type and handle it for SHT_RELR. Check > various properties in elflint. Print RELR relocations in

Re: [PATCH] libdwfl: fix crash when reading link map

2021-06-09 Thread Omar Sandoval
On Wed, Jun 09, 2021 at 09:59:06PM -0300, Érico Nogueira wrote: > Hope you don't mind my nit, then :) Thanks, that's what I get for sending patches out right before dinner... > On Wed Jun 9, 2021 at 9:51 PM -03, Omar Sandoval wrote: > > On Wed, Jun 09, 2021 at 05:45:57P

Re: [PATCH] libdwfl: fix crash when reading link map

2021-06-09 Thread Omar Sandoval
On Wed, Jun 09, 2021 at 05:45:57PM -0700, Omar Sandoval wrote: > From: Omar Sandoval > > When read_addrs() was converted was converted from a nested function to > a normal function, there was a mistake in converting "buffer" from a > closure variable to a parameter: we

[PATCH] libdwfl: fix crash when reading link map

2021-06-09 Thread Omar Sandoval
From: Omar Sandoval When read_addrs() was converted was converted from a nested function to a normal function, there was a mistake in converting "buffer" from a closure variable to a parameter: we are checking whether the pointer argument is NULL, not whether the buffer itself is

Re: [PATCH] libdw: handle DW_FORM_indirect when reading attributes

2021-05-04 Thread Omar Sandoval
On Sat, May 01, 2021 at 06:03:37PM +0200, Mark Wielaard wrote: > Hi, > > On Sat, 2021-05-01 at 17:59 +0200, Mark Wielaard wrote: > > There is __libdw_form_val_compute_len which already handles > > DW_FORM_indirect: > > > > case DW_FORM_indirect: > > get_uleb128 (u128, valp, endp); > >

Re: [PATCH] libdw: handle DW_FORM_indirect when reading attributes

2021-05-04 Thread Omar Sandoval
On Sat, May 01, 2021 at 05:59:31PM +0200, Mark Wielaard wrote: > Hi Omar, > > On Fri, 2021-04-23 at 16:36 -0700, Omar Sandoval wrote: > > Whenever we encounter an attribute with DW_FORM_indirect, we need to > > read its true form from the DIE data. Then, we can continue no

[PATCH] libdw: handle DW_FORM_indirect when reading attributes

2021-04-23 Thread Omar Sandoval
From: Omar Sandoval Whenever we encounter an attribute with DW_FORM_indirect, we need to read its true form from the DIE data. Then, we can continue normally. This adds support to the most obvious places: __libdw_find_attr() and dwarf_getattrs(). There may be more places that need to be updated

Re: [PATCH] libelf: handle PN_XNUM in elf_getphdrnum before shdr 0 is cached

2020-03-23 Thread Omar Sandoval
On Sun, Mar 22, 2020 at 11:40:34PM +0100, Mark Wielaard wrote: > Hi Omar, > > On Sat, 2020-03-21 at 11:21 -0700, Omar Sandoval wrote: > > I encountered this in drgn on a vmcore for a large server created by > > makedumpfile, > > That makes sense since [vm]cor

Re: [PATCH] libelf: handle PN_XNUM in elf_getphdrnum before shdr 0 is cached

2020-03-21 Thread Omar Sandoval
On Sat, Mar 21, 2020 at 01:30:55AM +0100, Mark Wielaard wrote: > Hi Omar, > > On Wed, Mar 18, 2020 at 01:18:51PM -0700, Omar Sandoval wrote: > > __elf_getphdrnum_rdlock() handles PN_XNUM by getting sh_info from > > elf->state.elf{32,64}.scns.data[0].shdr.e{32,64}. How

[PATCH] libelf: handle PN_XNUM in elf_getphdrnum before shdr 0 is cached

2020-03-18 Thread Omar Sandoval
From: Omar Sandoval __elf_getphdrnum_rdlock() handles PN_XNUM by getting sh_info from elf->state.elf{32,64}.scns.data[0].shdr.e{32,64}. However, that is only a cache that may or may not have been populated by elf_begin() or elf{32,64}_getshdr(); if it hasn't been cached yet, elf_ge

Re: [PATCH 0/4] libdwfl: make dwfl_addrmodule work for Linux kernel modules

2019-12-12 Thread Omar Sandoval
On Wed, Dec 11, 2019 at 05:29:42PM -0800, Omar Sandoval wrote: > From: Omar Sandoval > > Hello, > > I recently encountered a bug that dwfl_addrmodule doesn't work correctly > for Linux kernel modules. This is because each section of a kernel > module is allocated indep

Re: [PATCH 0/2] elfutils: minor build system fixes

2019-12-12 Thread Omar Sandoval
On Thu, Dec 12, 2019 at 12:17:23PM +0100, Mark Wielaard wrote: > On Wed, 2019-12-11 at 16:23 -0800, Omar Sandoval wrote: > > This small series fixes a couple of issues I encountered when building > > elfutils. Patch 1 fixes an issue when compiling with CFLAGS='-Wno-error &g

[PATCH 3/4] libdwfl: store module lookup table separately from segments

2019-12-11 Thread Omar Sandoval
From: Omar Sandoval Currently, the address ranges for segments reported with dwfl_report_segment and modules are stored in the same sorted array. This requires complicated logic in reify_segments for splitting up existing segments and inserting into the table, which can have quadratic runtime in

[PATCH 4/4] libdwfl: use sections of relocatable files for dwfl_addrmodule

2019-12-11 Thread Omar Sandoval
From: Omar Sandoval dwfl_addrmodule matches a module if the address lies within low_addr and high_addr. This is incorrect for relocatable files, particularly kernel modules: sections of different modules can be intermingled within the same range of addresses. Instead, we should index each

[PATCH 1/4] libdwfl: return error from __libdwfl_relocate_value for unloaded sections

2019-12-11 Thread Omar Sandoval
From: Omar Sandoval Currently, __libdwfl_relocate_value doesn't distinguish between unloaded sections and sections loaded at address zero. This has a few consequences: * relocate.c attempts relocation on unloaded sections when we don't have anything meaningful to reloca

[PATCH 2/4] libdwfl: remove broken coalescing logic in dwfl_report_segment

2019-12-11 Thread Omar Sandoval
From: Omar Sandoval dwfl_report_segment has some logic that detects when a segment is contiguous with the previously reported segment, in which case it's supposed to coalesce them. However, in this case, it actually returns without updating the segment array at all. As far as I can tell,

[PATCH 0/4] libdwfl: make dwfl_addrmodule work for Linux kernel modules

2019-12-11 Thread Omar Sandoval
From: Omar Sandoval Hello, I recently encountered a bug that dwfl_addrmodule doesn't work correctly for Linux kernel modules. This is because each section of a kernel module is allocated independently, so sections from different kernel modules may be intermixed. For example: # cd /sys/mo

Re: [PATCH] libdwfl: remove broken coalescing logic in dwfl_report_segment()

2019-12-11 Thread Omar Sandoval
On Tue, Dec 10, 2019 at 05:28:19PM -0800, Omar Sandoval wrote: > From: Omar Sandoval > > dwfl_report_segment() has some logic that detects when a segment is > contiguous with the previously reported segment, in which case it's > supposed to coalesce them. However, in th

[PATCH 1/2] configure: Fix -D_FORTIFY_SOURCE=2 check when CFLAGS contains -Wno-error

2019-12-11 Thread Omar Sandoval
From: Omar Sandoval If CFLAGS contains -Wno-error, then the check for -D_FORTIFY_SOURCE=2 won't fail when appropriate. E.g., compiling with: ./configure CFLAGS='-Wno-error -O0' && Results in a flood of "_FORTIFY_SOURCE requires compiling with optimization (-O)&

[PATCH 2/2] libcpu: Compile i386_lex.c with -Wno-implicit-fallthrough

2019-12-11 Thread Omar Sandoval
From: Omar Sandoval elfutils is compiled with -Wimplicit-fallthrough=5, so the fallthrough comment in i386_lex.c (generated by flex) doesn't prevent the implicit fallthrough warning. Add -Wno-implicit-fallthrough to i386_lex_CFLAGS. Signed-off-by: Omar Sandoval --- libcpu/ChangeLog

[PATCH 0/2] elfutils: minor build system fixes

2019-12-11 Thread Omar Sandoval
From: Omar Sandoval Hello, This small series fixes a couple of issues I encountered when building elfutils. Patch 1 fixes an issue when compiling with CFLAGS='-Wno-error -O0'. Patch 2 gets rid of a warning. Thanks! Omar Sandoval (2): configure: Fix -D_FORTIFY_SOURCE=2 check w

  1   2   >