location list
Hi all, I am trying to parse a location list given as an sec_offset. How do I get this offset value that points to .debug_loc so I can call dwarf_getlocations()? Should I pass this offset as the second parameter of this call? Sasha
Re: Range lists, zero-length functions, linker gc
Hi, On Mon, 2020-06-01 at 13:18 -0700, David Blaikie wrote: > On Mon, Jun 1, 2020 at 2:31 AM Mark Wielaard wrote: > > Each skeleton compilation unit has a DW_AT_dwo_name attribute which > > indicates the .dwo file where the split unit sections can be found. It > > actually seems seems easier to generate a different one for each > > skeleton compilation unit than trying to combine them for all the > > different skeleton compilation units you produce. > > > > > Certainly Bazel (& the internal Google version used to build most > > > Google software) can't handle an unbounded/unknown number of output > > > files from a build action. > > > > Yes, in principle .dwo files seems troublesome for build systems in > > general. > > They're pretty practical when they're generated right next to the .o > file & that's guaranteed by the compiler. "if you generate x.o, there > will be x.dwo next to it" - that's certainly how Bazel deals with > this. It doesn't parse the DWARF at all - knowing where the .dwo files > are along with the .o files. The DWARF spec makes it clear that a DWO is per CU, not per object file. So when an object file contains multiple CUs, it might also be associated with multiple .dwo files (as is also the case with a linked executable or shared library). The spec makes says the DW_AT_dwo_name can contain both a (relative) file or a path to the associated DWO file. Which means that relying on a one-to-one mapping from .o to .dwo is fragile and is likely to break when tools start using multiple CUs or different naming heuristics. > > Because of that I am > > actually a fan of the SHF_EXCLUDED hack that simply places the split > > .dwo sections in the same object file. For the above that would mean, > > just place them in the same section group. > > This was a newer feature added during standardization of Split DWARF, > which is handy for some users Although it is used in practice by some producers, it is not standardize (yet). Also because SHF_EXCLUDED isn't standardized (although it is used consistently for those arches that support it). > - but doesn't address the needs of the > original design of Split DWARF (for Google) - a distributed build > system that is trying to avoid moving more bytes than it must to one > machine to run the link step. So not having to ship all the DWARF > bytes to one machine for interactive debugging (pulling down from a > distributed file system only the needed .dwo files during debugging - > not all of them) - or at least being able to ship all the .dwo files > to one machine to make a .dwp, and ship all the .o files to another > machine for the link. I think that is not what most people would use split-dwarf for. The Google setup seems somewhat unique. Most people probably do compiling, linking and debugging on the same machine. The main use case (for me) is to speed up the edit-compile-debug cycle. Making sure that the linker doesn't have to deal with (most of) the .debug sections and can just leave them behind (either in the .o file, or a separate .dwo file) is the main attraction of split-dwarf IMHO. When actually producing production builds with debug you still pay the price anyway, because instead of the linker, you now need to build your dwp packages which does most of the same work the linker would have done anyway (combining the data, merging the string indexes, deduplicating debug types, etc.) > > > Multiple CUs in a single .dwo file is not really supported, which > > > would be another challenge (we had to compromise debug info quality a > > > little because of this limitation when doing ThinLTO - unable to emit > > > multiple CUs into each thin-linked .o file) - at which point maybe the > > > compiler'd need to produce an intermediate .dwp file of sorts... > > > > Are you sure? > > Fairly sure - I worked in depth on the implementation of ThinLTO & > considered a variety of options trying to support Split DWARF in that > situation. > > > Each CU would have a separate dwo_id field to > > distinquish them. At least that is how elfutils figures out which CU > > in a dwo file matches a given skeleton DIE. This should work the same > > as for type units, you can have multiple type untis in the same file > > and distinquish which one you need by matching the signature. > > One of the complications is that it increased the complexity of making > a .dwp file - Split DWARF is spec'd to ensure that the linking process > is as lightweight as possible. Not having the size overhead of > relocations (though trading off more indirection through the cu_index, > debug_str_offsets, etc). Oh right... that was the critical issue: > There was no way I could think of to do cross-CU references in Split > DWARF (cross-CU references being critical to LTO - inlining from one > CU into another, etc). Because there was no relocation processing in > dwp generation. Arguably maybe one could use a sec_offset that's > resolved relative to a local range within the contributi
Re: location list
Hi, On Tue, 2020-06-02 at 14:18 +, Sasha Da Rocha Pinheiro wrote: > I am trying to parse a location list given as an sec_offset. > How do I get this offset value that points to .debug_loc so I can > call dwarf_getlocations()? > Should I pass this offset as the second parameter of this call? Normally an offset isn't enough information to resolve an DIE attribute reference. So if at all possible you should try to use an Dwarf_Attribute you got from some DIE. It isn't really supported, but you could try creating a "fake" attribute that carries all information. e.g. Dwarf_Attribute loc_attr; loc_attr.code = DW_AT_location; loc_attr.form = DW_FORM_data4; /* Assuming 32bit DWARF. */ loc_attr.valp = &offset; /* Your offset should be a 32bit type. */ loc_attr.cu = cu; dwarf_getlocations (&loc_attr, offset, ...); Note that the CU needs to be version 3 or lower for the above to work. If the CU is > 3 then the only correct form to use is DW_FORM_sec_offset, and your valp should point to a uleb128 encoded offset value. But in general I would not recommend this approach. It isn't really supported. And some code might do sanity checks on the valp pointer and decide it looks bogus and just error out. Also you have to have a valid Dwarf_CU pointer around because you cannot create a valid fake CU easily. So try to keep a reference (or copy) around of the Dwarf_Attribute from which you got this offset and use that for your dwarf_getlocations call. Cheers, Mark
Re: Range lists, zero-length functions, linker gc
On Tue, Jun 2, 2020 at 9:50 AM Mark Wielaard wrote: > > Hi, > > On Mon, 2020-06-01 at 13:18 -0700, David Blaikie wrote: > > On Mon, Jun 1, 2020 at 2:31 AM Mark Wielaard wrote: > > > Each skeleton compilation unit has a DW_AT_dwo_name attribute which > > > indicates the .dwo file where the split unit sections can be found. It > > > actually seems seems easier to generate a different one for each > > > skeleton compilation unit than trying to combine them for all the > > > different skeleton compilation units you produce. > > > > > > > Certainly Bazel (& the internal Google version used to build most > > > > Google software) can't handle an unbounded/unknown number of output > > > > files from a build action. > > > > > > Yes, in principle .dwo files seems troublesome for build systems in > > > general. > > > > They're pretty practical when they're generated right next to the .o > > file & that's guaranteed by the compiler. "if you generate x.o, there > > will be x.dwo next to it" - that's certainly how Bazel deals with > > this. It doesn't parse the DWARF at all - knowing where the .dwo files > > are along with the .o files. > > The DWARF spec makes it clear that a DWO is per CU, not per object > file. So when an object file contains multiple CUs, it might also be > associated with multiple .dwo files (as is also the case with a linked > executable or shared library). The spec makes says the DW_AT_dwo_name > can contain both a (relative) file or a path to the associated DWO > file. Which means that relying on a one-to-one mapping from .o to .dwo > is fragile and is likely to break when tools start using multiple CUs > or different naming heuristics. Yep, agreed - in the most general form there's no guarantee that one compilation would produce one .dwo and you'd have to parse the .o to find all the associated .dwos. Practically speaking that's not the reality right now (build systems rely on stronger/narrower guarantees by the compiler about how many/where the .dwo files are). > > > Because of that I am > > > actually a fan of the SHF_EXCLUDED hack that simply places the split > > > .dwo sections in the same object file. For the above that would mean, > > > just place them in the same section group. > > > > This was a newer feature added during standardization of Split DWARF, > > which is handy for some users > > Although it is used in practice by some producers, it is not > standardize (yet). Also because SHF_EXCLUDED isn't standardized > (although it is used consistently for those arches that support it). Ah, sorry, I didn't mean the specific implementation strategy of using SHF_EXCLUDED, I meant the general concept of having a .o file be its own .dwo file is standardized "The sections that do not require relocation, however, can be written to the relocatable object (.o) file but ignored by the linker, or they can be written to a separate DWARF object (.dwo) file that need not be accessed by the linker." > > - but doesn't address the needs of the > > original design of Split DWARF (for Google) - a distributed build > > system that is trying to avoid moving more bytes than it must to one > > machine to run the link step. So not having to ship all the DWARF > > bytes to one machine for interactive debugging (pulling down from a > > distributed file system only the needed .dwo files during debugging - > > not all of them) - or at least being able to ship all the .dwo files > > to one machine to make a .dwp, and ship all the .o files to another > > machine for the link. > > I think that is not what most people would use split-dwarf for. Probably not - but it's the use case I care about/need to support. > The > Google setup seems somewhat unique. Most people probably do compiling, > linking and debugging on the same machine. The main use case (for me) > is to speed up the edit-compile-debug cycle. Making sure that the > linker doesn't have to deal with (most of) the .debug sections and can > just leave them behind (either in the .o file, or a separate .dwo file) > is the main attraction of split-dwarf IMHO. When actually producing > production builds with debug you still pay the price anyway, because > instead of the linker, you now need to build your dwp packages which > does most of the same work the linker would have done anyway (combining > the data, merging the string indexes, deduplicating debug types, etc.) It's still a price you can parallelize, rather than having to serialize (somewhat - lld is multithreaded for instance). And the dwp support for linking other dwp files together means you can do it iteratively (rather than taking all the .dwo files and doing noe big link step - you can take a few dwos, link them into an intermediate dwp (removing duplicate type information and strings) then link again with other intermediate dwps, etc - with some distribution/parallelism benefits). > > > > Multiple CUs in a single .dwo file is not really supported, which > > > > would be another challenge (w
Re: location list
Well, I have been trying to use the Dwarf_Attribute.
If you see the snippet below, locationAttribute is acquired by doing
dwarf_attr(&e, DW_AT_location, &locationAttribute);
and &e is the DW_TAG_variable DIE.
Dwarf_Op * exprs = NULL;
size_t exprlen = 0;
std::vector locDescs;
ptrdiff_t offset = 0;
Dwarf_Addr basep, start, end;
do {
offset = dwarf_getlocations(&locationAttribute, offset, &basep,
&start, &end, &exprs, &exprlen);
if(offset==-1) return false;
if(offset==0) break;
LocDesc ld;
ld.ld_lopc = start;
ld.ld_hipc = end;
ld.dwarfOp = exprs;
ld.opLen = exprlen;
locDescs.push_back(ld);
}while(offset > 0);
But what happens here is I always get the very first entry in .debug_loc. Where
clearly for this variable, the location list (sec_offset) is at [4a] of
that section.
Maybe I am using the offset or the basep wrongly?
Sasha
From: Mark Wielaard
Sent: Tuesday, June 2, 2020 12:19 PM
To: Sasha Da Rocha Pinheiro ;
elfutils-devel@sourceware.org
Subject: Re: location list
Hi,
On Tue, 2020-06-02 at 14:18 +, Sasha Da Rocha Pinheiro wrote:
> I am trying to parse a location list given as an sec_offset.
> How do I get this offset value that points to .debug_loc so I can
> call dwarf_getlocations()?
> Should I pass this offset as the second parameter of this call?
Normally an offset isn't enough information to resolve an DIE attribute
reference. So if at all possible you should try to use an
Dwarf_Attribute you got from some DIE.
It isn't really supported, but you could try creating a "fake"
attribute that carries all information. e.g.
Dwarf_Attribute loc_attr;
loc_attr.code = DW_AT_location;
loc_attr.form = DW_FORM_data4; /* Assuming 32bit DWARF. */
loc_attr.valp = &offset; /* Your offset should be a 32bit type. */
loc_attr.cu = cu;
dwarf_getlocations (&loc_attr, offset, ...);
Note that the CU needs to be version 3 or lower for the above to work.
If the CU is > 3 then the only correct form to use is
DW_FORM_sec_offset, and your valp should point to a uleb128 encoded
offset value.
But in general I would not recommend this approach. It isn't really
supported. And some code might do sanity checks on the valp pointer and
decide it looks bogus and just error out. Also you have to have a valid
Dwarf_CU pointer around because you cannot create a valid fake CU
easily.
So try to keep a reference (or copy) around of the Dwarf_Attribute from
which you got this offset and use that for your dwarf_getlocations
call.
Cheers,
Mark
Re: Range lists, zero-length functions, linker gc
On Tue, Jun 02, 2020 at 11:06:10AM -0700, David Blaikie via Binutils wrote: > On Tue, Jun 2, 2020 at 9:50 AM Mark Wielaard wrote: > > where I > > would argue the compiler simply needs to make sure that if it generates > > code in separate sections it also should create the DWARF separate > > section (groups). > > I don't think that's practical - the overhead, I believe, is too high. > Headers for each section contribution (ELF headers but DWARF headers > moreso - having a separate .debug_addr, .debug_line, etc section for > each function would be very expensive) would make for very large > object files. With a little linker magic I don't see the neccesity of duplicating the DWARF headers. Taking .debug_line as an example, a compiler could emit the header, opcode, directory and file tables to a .debug_line section with line statements for function foo emitted to .debug_line.foo and for bar to .debug_line.bar, trusting that the linker will combine these sections in order to create an output .debug_line section. If foo code is excluded then .debug_line.foo info will also be dropped if section groups are used. -- Alan Modra Australia Development Lab, IBM
Re: Range lists, zero-length functions, linker gc
On 2020-06-03, Alan Modra wrote: On Tue, Jun 02, 2020 at 11:06:10AM -0700, David Blaikie via Binutils wrote: On Tue, Jun 2, 2020 at 9:50 AM Mark Wielaard wrote: > where I > would argue the compiler simply needs to make sure that if it generates > code in separate sections it also should create the DWARF separate > section (groups). I don't think that's practical - the overhead, I believe, is too high. Headers for each section contribution (ELF headers but DWARF headers moreso - having a separate .debug_addr, .debug_line, etc section for each function would be very expensive) would make for very large object files. With a little linker magic I don't see the neccesity of duplicating the DWARF headers. Taking .debug_line as an example, a compiler could emit the header, opcode, directory and file tables to a .debug_line section with line statements for function foo emitted to .debug_line.foo and for bar to .debug_line.bar, trusting that the linker will combine these sections in order to create an output .debug_line section. If foo code is excluded then .debug_line.foo info will also be dropped if section groups are used. -- Alan Modra Australia Development Lab, IBM sizeof(Elf64_Shdr) = 64. If we create a .debug_line fragment and a .debug_info fragment for a function, we waste 128 bytes. https://sourceware.org/pipermail/binutils/2020-May/111361.html .debug_line.bar We should use the unique linkage feature https://sourceware.org/bugzilla/show_bug.cgi?id=25380 otherwise we also waste lots of bytes for the .debug_*.* section names.
