Re: Handling pgoff in perf elf mmap/mmap2 elf info
On Donnerstag, 11. Oktober 2018 20:14:43 CEST Milian Wolff wrote: > On Donnerstag, 11. Oktober 2018 19:37:07 CEST Mark Wielaard wrote: > > Hi, > > > > My apologies for not having looked deeper at this. > > It is a bit tricky and I just didnt have enough time to > > really sit down and think it all through yet. > > > > On Thu, Oct 11, 2018 at 05:02:18PM +, Ulf Hermann wrote: > > > is there any pattern in how the loader maps the ELF sections into > > > memory? What sections does it actually map and which of those do we need > > > for unwinding? > > > > Yes, it would be helpful to have some examples of mmap events plus > > the associated segment header (eu-readelf -l) of the ELF file. > > > > Note that the kernel and dynamic loader will use the (PT_LOAD) segments, > > not the sections, to map things into memory. Each segment might contain > > multiple sections. > > > > libdwfl then tries to associate the correct sections (and address bias) > > with how the ELF file was mapped into memory. > > > > > I hope that only one of those MMAPs per ELF is actually meaningful and > > > we can simply add that one's pgoff as an extra member to Dwfl_Module and > > > use it whenever we poke the underlying file. > > > > One "trick" might be to just substract the pgoff from the load address. > > And so report as if the ELF file was being mapped from the start. This > > isn't really correct, but it might be interesting to see if that makes > > libdwfl able to just associate the whole ELF file with the correct > > address map. > > I'll try to come up with some minimal code examples we can use to test all > of this. But from what I remember, neither of the above suggestions will be > sufficient as we can still run into overlapping module errors from elfutils > when we always load everything. I.e. I believe we've seen mappings that > eventually become partially obsoleted by a future mmap event. At that > point, we somehow need to be able to only map parts of a file, not all of > it. So just subtracting or honoring pgoff is not enough, I believe we also > need to be able to explicitly say how much of a file to map. > > But to make this discussion easier to follow for others, I'll create some > standalone cpp code that takes a `perf script --show-mmap-events | grep > PERF_RECORD_MMAP` input file and then runs this through elfutils API to > reproduce the issues we are facing. > > I'll get back to you all once this is done. Hey all, here's one example of mmap events recorded by perf: 0x7fac5ec0b000 to 0x7fac5ed9a000, len = 0x18f000, offset =0 r--p/usr/lib/libstdc++.so.6.0.25 0x7fac5ec94000 to 0x7fac5ed8a000, len =0xf6000, offset = 0x89000 ---p/usr/lib/libstdc++.so.6.0.25 0x7fac5ec94000 to 0x7fac5ed4c000, len =0xb8000, offset = 0x89000 r-xp/usr/lib/libstdc++.so.6.0.25 0x7fac5ed4c000 to 0x7fac5ed89000, len =0x3d000, offset = 0x141000 r--p/usr/lib/libstdc++.so.6.0.25 0x7fac5ed8a000 to 0x7fac5ed97000, len = 0xd000, offset = 0x17e000 rw-p/usr/lib/libstdc++.so.6.0.25 this is noteworthy in multiple ways: - the first mapping we receive is for pgoff = 0 for the full file size aligned to the page boundary - the first mapping isn't executable yet - the last mappings have a huge offset which actually lies beyond the initially mmaped region?! And to make things worse, when we report the file at address 0x7fac5ec0b000 via dwfl, we get: reported module /usr/lib/libstdc++.so.6.0.25 expected: 0x7fac5ec0b000 to 0x7fac5ed9a000 (0x18f000) actual: 0x7fac5ec0b000 to 0x7fac5ed99640 (0x18e640) So now dwfl won't ever be able to map any addresses into this module when they come after 0x7fac5ed99640, but the mmap events above seem to indicate that this could be possible? I'll now upload my code to enable you all to play around with this yourself. Bye -- Milian Wolff m...@milianw.de http://milianw.de signature.asc Description: This is a digitally signed message part.
Re: Handling pgoff in perf elf mmap/mmap2 elf info
On Donnerstag, 11. Oktober 2018 20:14:43 CEST Milian Wolff wrote: > On Donnerstag, 11. Oktober 2018 19:37:07 CEST Mark Wielaard wrote: > > Hi, > > > > My apologies for not having looked deeper at this. > > It is a bit tricky and I just didnt have enough time to > > really sit down and think it all through yet. > > > > On Thu, Oct 11, 2018 at 05:02:18PM +, Ulf Hermann wrote: > > > is there any pattern in how the loader maps the ELF sections into > > > memory? What sections does it actually map and which of those do we need > > > for unwinding? > > > > Yes, it would be helpful to have some examples of mmap events plus > > the associated segment header (eu-readelf -l) of the ELF file. > > > > Note that the kernel and dynamic loader will use the (PT_LOAD) segments, > > not the sections, to map things into memory. Each segment might contain > > multiple sections. > > > > libdwfl then tries to associate the correct sections (and address bias) > > with how the ELF file was mapped into memory. > > > > > I hope that only one of those MMAPs per ELF is actually meaningful and > > > we can simply add that one's pgoff as an extra member to Dwfl_Module and > > > use it whenever we poke the underlying file. > > > > One "trick" might be to just substract the pgoff from the load address. > > And so report as if the ELF file was being mapped from the start. This > > isn't really correct, but it might be interesting to see if that makes > > libdwfl able to just associate the whole ELF file with the correct > > address map. > > I'll try to come up with some minimal code examples we can use to test all > of this. But from what I remember, neither of the above suggestions will be > sufficient as we can still run into overlapping module errors from elfutils > when we always load everything. I.e. I believe we've seen mappings that > eventually become partially obsoleted by a future mmap event. At that > point, we somehow need to be able to only map parts of a file, not all of > it. So just subtracting or honoring pgoff is not enough, I believe we also > need to be able to explicitly say how much of a file to map. > > But to make this discussion easier to follow for others, I'll create some > standalone cpp code that takes a `perf script --show-mmap-events | grep > PERF_RECORD_MMAP` input file and then runs this through elfutils API to > reproduce the issues we are facing. > > I'll get back to you all once this is done. I've pushed a preliminary POC for a reproducer: https://github.com/milianw/perf_mmaps_to_elfutils Note that it's not really exhibiting any dwfl errors as-is. We would need to feed it also all the instruction pointer addresses that perf encounters, then try to find the matching module via libdwfl. This isn't easily done, and I hope the current output already exemplifies some of the issues with the current libdwfl API. Thanks -- Milian Wolff m...@milianw.de http://milianw.de signature.asc Description: This is a digitally signed message part.
Re: Handling pgoff in perf elf mmap/mmap2 elf info
Hi Milian, On Mon, 2018-10-15 at 22:38 +0200, Milian Wolff wrote: > here's one example of mmap events recorded by perf: > > 0x7fac5ec0b000 to 0x7fac5ed9a000, len = 0x18f000, offset =0 > > r--p/usr/lib/libstdc++.so.6.0.25 > 0x7fac5ec94000 to 0x7fac5ed8a000, len =0xf6000, offset = 0x89000 > > ---p/usr/lib/libstdc++.so.6.0.25 > 0x7fac5ec94000 to 0x7fac5ed4c000, len =0xb8000, offset = 0x89000 > > r-xp/usr/lib/libstdc++.so.6.0.25 > 0x7fac5ed4c000 to 0x7fac5ed89000, len =0x3d000, offset = 0x141000 > > r--p/usr/lib/libstdc++.so.6.0.25 > 0x7fac5ed8a000 to 0x7fac5ed97000, len = 0xd000, offset = 0x17e000 > > rw-p/usr/lib/libstdc++.so.6.0.25 Could you also post the matching phdr output for the file? eu-readelf -l /usr/lib/libstdc++.so.6.0.25 should show it. That way we can see how the PT_LOAD segments map to the mmap events. Thanks, Mark
Re: Handling pgoff in perf elf mmap/mmap2 elf info
On Montag, 15. Oktober 2018 23:04:52 CEST Mark Wielaard wrote: > Hi Milian, > > On Mon, 2018-10-15 at 22:38 +0200, Milian Wolff wrote: > > here's one example of mmap events recorded by perf: > > > > 0x7fac5ec0b000 to 0x7fac5ed9a000, len = 0x18f000, offset > > =0r--p/usr/lib/libstdc++.so.6.0.25 > > 0x7fac5ec94000 to 0x7fac5ed8a000, len =0xf6000, offset > > = 0x89000---p/usr/lib/libstdc++.so.6.0.25 > > 0x7fac5ec94000 to 0x7fac5ed4c000, len =0xb8000, offset > > = 0x89000r-xp/usr/lib/libstdc++.so.6.0.25 > > 0x7fac5ed4c000 to 0x7fac5ed89000, len =0x3d000, offset > > = 0x141000r--p/usr/lib/libstdc++.so.6.0.25 > > 0x7fac5ed8a000 to 0x7fac5ed97000, len = 0xd000, offset > > = 0x17e000rw-p/usr/lib/libstdc++.so.6.0.25 > > Could you also post the matching phdr output for the file? > eu-readelf -l /usr/lib/libstdc++.so.6.0.25 should show it. > That way we can see how the PT_LOAD segments map to the mmap events. Sure: $ eu-readelf -l /usr/lib/libstdc++.so.6.0.25 Program Headers: Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align LOAD 0x00 0x 0x 0x088fa8 0x088fa8 R 0x1000 LOAD 0x089000 0x00089000 0x00089000 0x0b7ae1 0x0b7ae1 R E 0x1000 LOAD 0x141000 0x00141000 0x00141000 0x03cfe0 0x03cfe0 R 0x1000 LOAD 0x17e8e0 0x0017f8e0 0x0017f8e0 0x00b8b8 0x00ed60 RW 0x1000 DYNAMIC0x1873a8 0x001883a8 0x001883a8 0x0001e0 0x0001e0 RW 0x8 NOTE 0x0002a8 0x02a8 0x02a8 0x24 0x24 R 0x4 NOTE 0x17dfc0 0x0017dfc0 0x0017dfc0 0x20 0x20 R 0x8 TLS0x17e8e0 0x0017f8e0 0x0017f8e0 0x00 0x20 R 0x8 GNU_EH_FRAME 0x149558 0x00149558 0x00149558 0x007f04 0x007f04 R 0x4 GNU_STACK 0x00 0x 0x 0x00 0x00 RW 0x10 GNU_RELRO 0x17e8e0 0x0017f8e0 0x0017f8e0 0x00b720 0x00b720 R 0x1 Section to Segment mapping: Segment Sections... 00 [RO: .note.gnu.build-id .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_d .gnu.version_r .rela.dyn] 01 [RO: .init .text .fini] 02 [RO: .rodata .eh_frame_hdr .eh_frame .gcc_except_table .note.gnu.property] 03 [RELRO: .tbss .init_array .fini_array .data.rel.ro .dynamic .got] .got.plt .data .bss 04 [RELRO: .dynamic] 05 [RO: .note.gnu.build-id] 06 [RO: .note.gnu.property] 07 [RELRO: .tbss] 08 [RO: .eh_frame_hdr] 09 10 [RELRO: .tbss .init_array .fini_array .data.rel.ro .dynamic .got] -- Milian Wolff m...@milianw.de http://milianw.de signature.asc Description: This is a digitally signed message part.
[Bug tools/23673] TEST ./tests/backtrace-dwarf fails on s390x in at least 0.173
https://sourceware.org/bugzilla/show_bug.cgi?id=23673 Michael Hudson-Doyle changed: What|Removed |Added CC||michael.hudson at canonical dot co ||m --- Comment #19 from Michael Hudson-Doyle --- I see a similar looking failure on arm64 on Ubuntu 18.10: https://launchpadlibrarian.net/391377304/buildlog_ubuntu-cosmic-arm64.elfutils_0.170-0.5_BUILDING.txt.gz I've gdb-ed this to the point that the key difference between a working system (Ubuntu 18.04) and the failing one is that libc.so.6 has a lot more entries in .eh_frame_hdr in the failing system. On 18.04 it fails to find a fde for abort() (or raise, I think) and unwinds using .debug_frame and that succeeds. On 18.10 it finds a fde for both raise and abort but fails to successfully unwind past abort using it. I don't know either why the newer libc.so.6 has a bigger eh_frame_hdr (it is glibc 2.28 vs 2.27 but also built with newer gcc and binutils) or why unwinding using eh_frame info fails. -- You are receiving this mail because: You are on the CC list for the bug.
