> On 7/14/22 8:09 AM, Jose E. Marchesi wrote: >> Hi Yonghong. >> >>> On 7/7/22 1:24 PM, Jose E. Marchesi wrote: >>>> Hi Yonghong. >>>> >>>>> On 6/21/22 9:12 AM, Jose E. Marchesi wrote: >>>>>> >>>>>>> On 6/17/22 10:18 AM, Jose E. Marchesi wrote: >>>>>>>> Hi Yonghong. >>>>>>>> >>>>>>>>> On 6/15/22 1:57 PM, David Faust wrote: >>>>>>>>>> >>>>>>>>>> On 6/14/22 22:53, Yonghong Song wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 6/7/22 2:43 PM, David Faust wrote: >>>>>>>>>>>> Hello, >>>>>>>>>>>> >>>>>>>>>>>> This patch series adds support for: >>>>>>>>>>>> >>>>>>>>>>>> - Two new C-language-level attributes that allow to associate (to >>>>>>>>>>>> "annotate" or >>>>>>>>>>>> to "tag") particular declarations and types with arbitrary >>>>>>>>>>>> strings. As >>>>>>>>>>>> explained below, this is intended to be used to, for >>>>>>>>>>>> example, characterize >>>>>>>>>>>> certain pointer types. >>>>>>>>>>>> >>>>>>>>>>>> - The conveyance of that information in the DWARF output in the >>>>>>>>>>>> form of a new >>>>>>>>>>>> DIE: DW_TAG_GNU_annotation. >>>>>>>>>>>> >>>>>>>>>>>> - The conveyance of that information in the BTF output in the form >>>>>>>>>>>> of two new >>>>>>>>>>>> kinds of BTF objects: BTF_KIND_DECL_TAG and >>>>>>>>>>>> BTF_KIND_TYPE_TAG. >>>>>>>>>>>> >>>>>>>>>>>> All of these facilities are being added to the eBPF ecosystem, and >>>>>>>>>>>> support for >>>>>>>>>>>> them exists in some form in LLVM. >>>>>>>>>>>> >>>>>>>>>>>> Purpose >>>>>>>>>>>> ======= >>>>>>>>>>>> >>>>>>>>>>>> 1) Addition of C-family language constructs (attributes) to >>>>>>>>>>>> specify free-text >>>>>>>>>>>> tags on certain language elements, such as struct fields. >>>>>>>>>>>> >>>>>>>>>>>> The purpose of these annotations is to provide >>>>>>>>>>>> additional information about >>>>>>>>>>>> types, variables, and function parameters of interest to >>>>>>>>>>>> the kernel. A >>>>>>>>>>>> driving use case is to tag pointer types within the >>>>>>>>>>>> linux kernel and eBPF >>>>>>>>>>>> programs with additional semantic information, such as >>>>>>>>>>>> '__user' or '__rcu'. >>>>>>>>>>>> >>>>>>>>>>>> For example, consider the linux kernel function >>>>>>>>>>>> do_execve with the >>>>>>>>>>>> following declaration: >>>>>>>>>>>> >>>>>>>>>>>> static int do_execve(struct filename *filename, >>>>>>>>>>>> const char __user *const __user *__argv, >>>>>>>>>>>> const char __user *const __user *__envp); >>>>>>>>>>>> >>>>>>>>>>>> Here, __user could be defined with these annotations to >>>>>>>>>>>> record semantic >>>>>>>>>>>> information about the pointer parameters (e.g., they are >>>>>>>>>>>> user-provided) in >>>>>>>>>>>> DWARF and BTF information. Other kernel facilites such >>>>>>>>>>>> as the eBPF verifier >>>>>>>>>>>> can read the tags and make use of the information. >>>>>>>>>>>> >>>>>>>>>>>> 2) Conveying the tags in the generated DWARF debug info. >>>>>>>>>>>> >>>>>>>>>>>> The main motivation for emitting the tags in DWARF is >>>>>>>>>>>> that the Linux kernel >>>>>>>>>>>> generates its BTF information via pahole, using DWARF as >>>>>>>>>>>> a source: >>>>>>>>>>>> >>>>>>>>>>>> +--------+ BTF BTF +----------+ >>>>>>>>>>>> | pahole |-------> vmlinux.btf ------->| verifier | >>>>>>>>>>>> +--------+ +----------+ >>>>>>>>>>>> ^ ^ >>>>>>>>>>>> | | >>>>>>>>>>>> DWARF | BTF | >>>>>>>>>>>> | | >>>>>>>>>>>> vmlinux +-------------+ >>>>>>>>>>>> module1.ko | BPF program | >>>>>>>>>>>> module2.ko +-------------+ >>>>>>>>>>>> ... >>>>>>>>>>>> >>>>>>>>>>>> This is because: >>>>>>>>>>>> >>>>>>>>>>>> a) Unlike GCC, LLVM will only generate BTF for BPF >>>>>>>>>>>> programs. >>>>>>>>>>>> >>>>>>>>>>>> b) GCC can generate BTF for whatever target with -gbtf, >>>>>>>>>>>> but there is no >>>>>>>>>>>> support for linking/deduplicating BTF in the linker. >>>>>>>>>>>> >>>>>>>>>>>> In the scenario above, the verifier needs access to the >>>>>>>>>>>> pointer tags of >>>>>>>>>>>> both the kernel types/declarations (conveyed in the >>>>>>>>>>>> DWARF and translated >>>>>>>>>>>> to BTF by pahole) and those of the BPF program >>>>>>>>>>>> (available directly in BTF). >>>>>>>>>>>> >>>>>>>>>>>> Another motivation for having the tag information in >>>>>>>>>>>> DWARF, unrelated to >>>>>>>>>>>> BPF and BTF, is that the drgn project (another DWARF >>>>>>>>>>>> consumer) also wants >>>>>>>>>>>> to benefit from these tags in order to differentiate >>>>>>>>>>>> between different >>>>>>>>>>>> kinds of pointers in the kernel. >>>>>>>>>>>> >>>>>>>>>>>> 3) Conveying the tags in the generated BTF debug info. >>>>>>>>>>>> >>>>>>>>>>>> This is easy: the main purpose of having this info in >>>>>>>>>>>> BTF is for the >>>>>>>>>>>> compiled eBPF programs. The kernel verifier can then >>>>>>>>>>>> access the tags >>>>>>>>>>>> of pointers used by the eBPF programs. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> For more information about these tags and the motivation behind >>>>>>>>>>>> them, please >>>>>>>>>>>> refer to the following linux kernel discussions: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> https://lore.kernel.org/bpf/20210914223004.244411-1-...@fb.com/ >>>>>>>>>>>> >>>>>>>>>>>> https://lore.kernel.org/bpf/20211012164838.3345699-1-...@fb.com/ >>>>>>>>>>>> >>>>>>>>>>>> https://lore.kernel.org/bpf/20211112012604.1504583-1-...@fb.com/ >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Implementation Overview >>>>>>>>>>>> ======================= >>>>>>>>>>>> >>>>>>>>>>>> To enable these annotations, two new C language attributes are >>>>>>>>>>>> added: >>>>>>>>>>>> __attribute__((debug_annotate_decl("foo"))) and >>>>>>>>>>>> __attribute__((debug_annotate_type("bar"))). Both attributes >>>>>>>>>>>> accept a single >>>>>>>>>>>> arbitrary string constant argument, which will be recorded in the >>>>>>>>>>>> generated >>>>>>>>>>>> DWARF and/or BTF debug information. They have no effect on code >>>>>>>>>>>> generation. >>>>>>>>>>>> >>>>>>>>>>>> Note that we are not using the same attribute names as LLVM >>>>>>>>>>>> (btf_decl_tag and >>>>>>>>>>>> btf_type_tag, respectively). While these attributes are >>>>>>>>>>>> functionally very >>>>>>>>>>>> similar, they have grown beyond purely BTF-specific uses, so >>>>>>>>>>>> inclusion of "btf" >>>>>>>>>>>> in the attribute name seems misleading. >>>>>>>>>>>> >>>>>>>>>>>> DWARF support is enabled via a new DW_TAG_GNU_annotation. When >>>>>>>>>>>> generating DWARF, >>>>>>>>>>>> declarations and types will be checked for the corresponding >>>>>>>>>>>> attributes. If >>>>>>>>>>>> present, a DW_TAG_GNU_annotation DIE will be created as a child of >>>>>>>>>>>> the DIE for >>>>>>>>>>>> the annotated type or declaration, one for each tag. These DIEs >>>>>>>>>>>> link the >>>>>>>>>>>> arbitrary tag value to the item they annotate. >>>>>>>>>>>> >>>>>>>>>>>> For example, the following variable declaration: >>>>>>>>>>>> >>>>>>>>>>>> #define __typetag1 __attribute__((debug_annotate_type >>>>>>>>>>>> ("typetag1"))) >>>>>>>>>>>> >>>>>>>>>>>> #define __decltag1 __attribute__((debug_annotate_decl >>>>>>>>>>>> ("decltag1"))) >>>>>>>>>>>> #define __decltag2 __attribute__((debug_annotate_decl >>>>>>>>>>>> ("decltag2"))) >>>>>>>>>>>> >>>>>>>>>>>> int * __typetag1 x __decltag1 __decltag2; >>>>>>>>>>> >>>>>>>>>>> Based on the above example >>>>>>>>>>> static int do_execve(struct filename *filename, >>>>>>>>>>> const char __user *const __user *__argv, >>>>>>>>>>> const char __user *const __user *__envp); >>>>>>>>>>> >>>>>>>>>>> Should the above example should be the below? >>>>>>>>>>> int __typetag1 * x __decltag1 __decltag2 >>>>>>>>>>> >>>>>>>>>> This example is not related to the one above. It is just meant to >>>>>>>>>> show the behavior of both attributes. My apologies for not making >>>>>>>>>> that clear. >>>>>>>>> >>>>>>>>> Okay, it should be fine if the dwarf debug_info is shown. >>>>>>>>> >>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Produces the following DWARF information: >>>>>>>>>>>> >>>>>>>>>>>> <1><1e>: Abbrev Number: 3 (DW_TAG_variable) >>>>>>>>>>>> <1f> DW_AT_name : x >>>>>>>>>>>> <21> DW_AT_decl_file : 1 >>>>>>>>>>>> <22> DW_AT_decl_line : 7 >>>>>>>>>>>> <23> DW_AT_decl_column : 18 >>>>>>>>>>>> <24> DW_AT_type : <0x49> >>>>>>>>>>>> <28> DW_AT_external : 1 >>>>>>>>>>>> <28> DW_AT_location : 9 byte block: 3 0 0 0 0 0 0 0 >>>>>>>>>>>> 0 (DW_OP_addr: 0) >>>>>>>>>>>> <32> DW_AT_sibling : <0x49> >>>>>>>>>>>> <2><36>: Abbrev Number: 1 (User TAG value: 0x6000) >>>>>>>>>>>> <37> DW_AT_name : (indirect string, offset: 0xd6): >>> debug_annotate_decl >>>>>>>>>>>> <3b> DW_AT_const_value : (indirect string, offset: >>>>>>>>>>>> 0xcd): decltag2 >>>>>>>>>>>> <2><3f>: Abbrev Number: 1 (User TAG value: 0x6000) >>>>>>>>>>>> <40> DW_AT_name : (indirect string, offset: 0xd6): >>> debug_annotate_decl >>>>>>>>>>>> <44> DW_AT_const_value : (indirect string, offset: >>>>>>>>>>>> 0x0): decltag1 >>>>>>>>>>>> <2><48>: Abbrev Number: 0 >>>>>>>>>>>> <1><49>: Abbrev Number: 4 (DW_TAG_pointer_type) >>>>>>>>>>>> <4a> DW_AT_byte_size : 8 >>>>>>>>>>>> <4b> DW_AT_type : <0x5d> >>>>>>>>>>>> <4f> DW_AT_sibling : <0x5d> >>>>>>>>>>>> <2><53>: Abbrev Number: 1 (User TAG value: 0x6000) >>>>>>>>>>>> <54> DW_AT_name : (indirect string, offset: 0x9): >>> debug_annotate_type >>>>>>>>>>>> <58> DW_AT_const_value : (indirect string, offset: >>>>>>>>>>>> 0x1d): typetag1 >>>>>>>>>>>> <2><5c>: Abbrev Number: 0 >>>>>>>>>>>> <1><5d>: Abbrev Number: 5 (DW_TAG_base_type) >>>>>>>>>>>> <5e> DW_AT_byte_size : 4 >>>>>>>>>>>> <5f> DW_AT_encoding : 5 (signed) >>>>>>>>>>>> <60> DW_AT_name : int >>>>>>>>>>>> <1><64>: Abbrev Number: 0 >>>>>>>>> >>>>>>>>> This shows the info in .debug_abbrev. What I mean is to >>>>>>>>> show the related info in .debug_info section which seems more useful >>>>>>>>> to >>>>>>>>> understand the relationships between different tags. Maybe this is due >>>>>>>>> to that I am not fully understanding what <1>/<2> means in <1><49> and >>>>>>>>> <2><53> etc. >>>>>>>> I think that dump actually shows .debug_info, with the abbrevs >>>>>>>> expanded... >>>>>>>> Anyway, it seems to us that the root of this problem is the fact the >>>>>>>> kernel sparse annotations, such as address_space(__user), are: >>>>>>>> 1) To be processed by an external kernel-specific tool ( >>>>>>>> https://sparse.docs.kernel.org/en/latest/annotations.html) and >>>>>>>> not a >>>>>>>> C compiler, and therefore, >>>>>>>> 2) Not quite the same than compiler attributes (despite the way they >>>>>>>> look.) In particular, they seem to assume an ordering >>>>>>>> different than >>>>>>>> of GNU attributes: in some cases given the same written order, >>>>>>>> they >>>>>>>> refer to different things!. Which is quite unfortunate :( >>>>>>> >>>>>>> Yes, currently __user/__kernel macros (implemented with address_space >>>>>>> attribute) are processed by macros. >>>>>>> >>>>>>>> Now, if I understood properly, you plan to change the definition of >>>>>>>> __user and __kernel in the kernel sources in order to generate the tag >>>>>>>> compiler attributes, correct? >>>>>>> >>>>>>> Right. The original __user definition likes: >>>>>>> # define __user __attribute__((noderef, >>>>>>> address_space(__user))) >>>>>>> >>>>>>> The new attribute looks like >>>>>>> # define BTF_TYPE_TAG(value) __attribute__((btf_type_tag(#value))) >>>>>>> # define __user BTF_TYPE_TAG(user) >>>>>> Ok I see. So the kernel will stop using sparse attributes to >>>>>> implement >>>>>> __user and __kernel and start using compiler attributes for tags >>>>>> instead. >>>>>> >>>>>>>> Is that the reason why LLVM implements what we assume to be the >>>>>>>> sparse >>>>>>>> ordering, and not the correct GNU attributes ordering, for the tag >>>>>>>> attributes? >>>>>>> >>>>>>> Note that __user attributes apply to pointee's and not pointers. >>>>>>> Just like >>>>>>> const int *p; >>>>>>> the 'const' is not applied to pointer 'p', but the pointee of 'p'. >>>>>>> >>>>>>> What current llvm dwarf generation with >>>>>>> pointer >>>>>>> <--- btf_type_tag >>>>>>> is just ONE implementation. As I said earlier, I am okay to >>>>>>> have dwarf implementation like >>>>>>> p->btf_type_tag->const->int. >>>>>>> If you can propose an implementation like this in dwarf. I can propose >>>>>>> to change implementation in llvm. >>>>>> I think we are miscommunicating. >>>>>> Looks like there is a divergence on what attributes apply to what >>>>>> language entities between the sparse compiler and GCC/LLVM. How to >>>>>> represent that in DWARF is a different matter. >>>>>> For this example: >>>>>> int __typetag1 * __typetag2 __typetag3 * g; >>>>>> a) GCC associates __typetag1 with the pointer-to-pointer-to-int. >>>>>> b) LLVM associates __typetag1 to pointer-to-int. >>>>>> Where: >>>>>> a) Is the expected behavior of a compiler attributes, as documented >>>>>> in >>>>>> the GCC manual. >>>>>> b) Is presumably what the sparse compiler expects, but _not_ the >>>>>> ordering expected for a compiler GNU attribute. >>>>>> So, if the kernel source __user and __kernel annotations (which >>>>>> currently expand to sparse attributes) follow the sparse ordering, and >>>>>> you want to implement __user and __kernel in terms of compiler >>>>>> attributes instead (the annotation attributes) then you will have to: >>>>>> 1) Fix LLVM to implement the usual ordering for these attributes and >>>>>> 2) fix the kernel sources to use that ordering >>>>>> [Incidentally, the same applies to another "ex-sparse" attribute you >>>>>> have in the kernel and also implemented in LLVM with a weird >>>>>> ordering: >>>>>> the address_space attribute.] >>>>>> For 2), it may be possible to write a coccinnelle script to generate >>>>>> the >>>>>> patch... >>>>> >>>>> I don't think (2) (to change kernel source for different attr ordering) >>>>> will work. So the only thing we can do is in compiler/pahole except >>>>> macro replacement in kernel. >>>> I looked at sparse and its parser. Wanted to be sure the ordering >>>> it >>>> uses to interpret sparse annotations (such as address_space, alignment, >>>> etc) is definitely _not_ the same ordering used by __attribute__ in C >>>> compilers. >>>> It is very different indeed and the same can be said about how >>>> sparse >>>> interprets other modifiers like `const': in sparse both `int const *foo' >>>> and `int *const foo' parse to a constant pointer to int, for example. >>>> I am not to judge how sparse handles its annotations. It may be >>>> very >>>> well and pertinent for its particular purpose. >>>> But I am not sure if it is reasonable to expect C compilers to >>>> implement >>>> certain type __attributes__ to parse differently, just because it >>>> happens these attributes are reused from sparse annotations in a >>>> particular program (in this case the kernel.) The debug_annotate_decl >>>> and debug_annotate_type attributes are not even intended to be >>>> kernel-specific. >>>> So, if changing the kernel sources is not an option (why btw, other >>>> than >>>> being a PITA?) at this point I really don't know what else to suggest :/ >>>> Any suggestion from the front-end people? >>> >>> Just want to understand the overall picture. So gcc can still emit >>> BTF properly with btf_type_tag right? The issue we are talking about >>> here is about the dwarf, right? >> If by "properly" you mean how sparse handles its annotations, then >> not >> really. >> The issue we are talking about is rather a language-level one: to >> what >> entity/type the compiler attribute applies. >> So, for: >> int __attribute__((debug_annotate_decl("user"))) *foo; >> GCC will apply the attribute to the int type, following the rules >> for >> type attributes (sparse would apply the annotation to the *int type >> instead). The emitted debug info (be it DWARF or BTF) will reflect >> that, no more no less :/ > > I don't know what does this 'apply the attribute to the int' mean. > In current clang implementation it means the following dwarf chains > from right to left > variable 'foo' > type: ptr > base type: attr_type: attr > underlying type: int > > So the type chain is foo -> ptr -> attr -> int
Urgh sorry Yonghong, that was a bad example where there is no divergence. At this point I find myself confused regarding the sparse, clang and GCC attribute issue (I have so many dumps around from all three tools in several formats) so I better recap on this before creating further confusion. Will be back to you soon. >>> If this is the case, we might have >>> a partial solution here. >>> - gcc emits BTF for vmlinux >> Note that for emitting BTF for vmlinux we would need support in the >> linker to merge and deduplicate BTF, which at the moment we don't have. > > This should be okay. pahole will merge and deduplicate btf. In pahole > '-j' mode, each thread will convert each .o file dwarf to btf, and > then pahole will merge and deduplicate btf. Thats nice. If LLVM supported generating BTF for any target (I believe you got patches for that) you could even skip the dwarf->BTF translation step alltogether with both LLVM and GCC kernel builds :) > >> >>> - gcc emits dwarf for vmlinux ignoring btf_type_tag >>> - in pahole, vmlinux BTF is amended with some additional misc things. >>> Although there are some use cases to have btf_type_tag in dwarf, but >>> that can be workarouned with BTF + dwarf both of which are generated >>> by the compiler. Not elegent, but probably works. >>>> >>>>>> Does this make sense? >>>>>> >>>>>>>> If that is so, we have quite a problem here: I don't think we can >>>>>>>> change >>>>>>>> the way GCC handles GNU-like attributes just because the kernel sources >>>>>>>> want to hook on these __user/__kernel sparse annotations to generate >>>>>>>> the >>>>>>>> compiler tags, even if we could mayhaps get GCC to handle >>>>>>>> debug_annotate_type and debug_annotate_decl differently. Some would >>>>>>>> say >>>>>>>> doing so would perpetuate the mistake instead of fixing it... >>>>>>>> Is my understanding correct? >>>>>>> >>>>>>> Let us just say that the btf_type_tag attribute applies to pointees. >>>>>>> Does this help? >>>>>>> >>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Maybe you can also show what dwarf debug_info looks like >>>>>>>>>> I am not sure what you mean. This is the .debug_info section as >>>>>>>>>> output >>>>>>>>>> by readelf -w. I did trim some information not relevant to the >>>>>>>>>> discussion >>>>>>>>>> such as the DW_TAG_compile_unit DIE, for brevity. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> In the case of BTF, the annotations are recorded in two type kinds >>>>>>>>>>>> recently >>>>>>>>>>>> added to the BTF specification: BTF_KIND_DECL_TAG and >>>>>>>>>>>> BTF_KIND_TYPE_TAG. >>>>>>>>>>>> The above example declaration prodcues the following BTF >>>>>>>>>>>> information: >>>>>>>>>>>> >>>>>>>>>>>> [1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED >>>>>>>>>>>> [2] PTR '(anon)' type_id=3 >>>>>>>>>>>> [3] TYPE_TAG 'typetag1' type_id=1 >>>>>>>>>>>> [4] DECL_TAG 'decltag1' type_id=6 component_idx=-1 >>>>>>>>>>>> [5] DECL_TAG 'decltag2' type_id=6 component_idx=-1 >>>>>>>>>>>> [6] VAR 'x' type_id=2, linkage=global >>>>>>>>>>>> [7] DATASEC '.bss' size=0 vlen=1 >>>>>>>>>>>> type_id=6 offset=0 size=8 (VAR 'x') >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> [...]
