[lldb-dev] Symbol Server for everyone.

[Taras Tsugrii via lldb-dev]

Hello lldb developers,


In one of the older posts 
(http://blog.llvm.org/2015/01/lldb-is-coming-to-windows.html), symbol server 
support was mentioned. Most likely it was meant for Windows, but at FB we have 
our own symbol server implementation for Linux (technically it's completely 
platform agnostic), which we would like to integrate with LLDB and eventually 
open source along with the server. As such I thought I'd ask LLDB gurus like 
you, if anyone is already working on symbol server support and if not, I'd 
appreciate your thoughts on a desired architecture.


General idea.

Based on current LLDB implementation and the fact that symbol server feature is 
a cross-cutting concern, the natural place to put this logic would be inside 
SymbolVendor plugin, which on Mac is used to resolve separate dSYM bundles. In 
theory symbol server logic is completely platform-agnostic, as all we need to 
know is some sort of binary ID (could either be a real .build-id or UUID or 
some custom logic to compute a stable binary hash) and binary name. This info 
can be used to make a network request to check whether corresponding binary 
exists and if so, download it to a temporary location and call 
symbol_vendor->AddSymbolFileRepresentation with FileSpec pointing at that 
temporary location.


Implementation details.


Logic placement.

Even though symbol resolution is platform agnostic, the process of 
extracting/computing binary ID is. As such it seems like SymbolServerResolver 
can either be a part of LLDB core, or a separate directory in 
Plugins/SymbolVendor, which will then be used by SymbolVendorELF and 
SymbolVendorMacOSX. First both symbol vendors will try to resolve the symbols 
the way they currently do and only if they cannot find anything, will they try 
to use SymbolVendorSymbolServer.

Alternatively symbol server resolution logic can be placed into its own 
SymbolVendorSymbolServer, and modify SymbolVendor FindPlugin's logic such that 
it does not return the first found SymbolVendor instance and instead returns 
either the first SymbolVendor instance that managed to successfully resolve 
symbols or just last one.

Yet another alternative would be to use a delegation chain, such that any 
SymbolVendor could be wrapped into a SymbolVendorSymbolServer, which would 
first try to invoke the delegate and if it cannot find symbols, will try to 
perform its magic. This approach seems nice, but does not play nice with 
current implementation based on static factory method.


Symbol server communication.

Network communication can either be implemented natively for different 
platforms or it can be delegated to a python script invoked by 
ScriptInterpreter. Using Python seems an easier option in order to make this 
cross-platform, but it adds a dependency on Python and will require propagating 
ScriptInterpreter to SymbolVendor creation factory.


Thoughts, suggestions and comments are very welcome.


Thank you,

Taras

LLVM Project Blog: LLDB is Coming to 
Windows
blog.llvm.org
This preliminary bootstraping work is mostly complete, and you can use LLDB to 
debug simple executables generated with Clang on Windows today.


___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Symbol Server for everyone.

[Taras Tsugrii via lldb-dev]

Zachary, I agree that adding a Python dependency might not be a good idea, so 
I'll take a closer look at the network code available in lldb. Symbol format we 
are currently using is pretty simple - every artifact is identified by a type 
(elf, src, etc), an id (build id for binary or hash for source) and a path. I'm 
not sure what you mean by platform agnostic, but with this approach every 
SymbolVendor will just have to pass the appropriate type, build id and a path 
to a ArtifactManager, which will download or locate a locally cached artifact 
and return a path to it.


From: Zachary Turner 
Sent: Friday, August 26, 2016 8:18:54 PM
To: Taras Tsugrii; lldb-dev@lists.llvm.org
Cc: Kevin Frei
Subject: Re: [lldb-dev] Symbol Server for everyone.

Making the SymbolVendor dependent on Python is probably a non starter, and it 
would also make debugging more difficult.

We have network code for various platforms in Host already.

It would be nice to have a symbol server format that is platform agnostic. On 
the other hand, Microsoft tools already understand their own symbol server 
format , so if i ever reprioritize this, we will probably want the standard 
Microsoft symbol server format on Windows for interoperability.


On Fri, Aug 26, 2016 at 8:00 PM Taras Tsugrii via lldb-dev 
mailto:lldb-dev@lists.llvm.org>> wrote:

Hello lldb developers,


In one of the older posts 
(http://blog.llvm.org/2015/01/lldb-is-coming-to-windows.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__blog.llvm.org_2015_01_lldb-2Dis-2Dcoming-2Dto-2Dwindows.html&d=DQMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=jqaYv5aDYHR_MGTz1rkWPg&m=cGDliRKtgFrgnkWwFqxPh4RyRVqdfaAmOCGP-zL_LbA&s=0aQ3fWYzXJkFu8RXsNn-tueEzkRgtHCC53MVc6Mbbqw&e=>),
 symbol server support was mentioned. Most likely it was meant for Windows, but 
at FB we have our own symbol server implementation for Linux (technically it's 
completely platform agnostic), which we would like to integrate with LLDB and 
eventually open source along with the server. As such I thought I'd ask LLDB 
gurus like you, if anyone is already working on symbol server support and if 
not, I'd appreciate your thoughts on a desired architecture.


General idea.

Based on current LLDB implementation and the fact that symbol server feature is 
a cross-cutting concern, the natural place to put this logic would be inside 
SymbolVendor plugin, which on Mac is used to resolve separate dSYM bundles. In 
theory symbol server logic is completely platform-agnostic, as all we need to 
know is some sort of binary ID (could either be a real .build-id or UUID or 
some custom logic to compute a stable binary hash) and binary name. This info 
can be used to make a network request to check whether corresponding binary 
exists and if so, download it to a temporary location and call 
symbol_vendor->AddSymbolFileRepresentation with FileSpec pointing at that 
temporary location.


Implementation details.


Logic placement.

Even though symbol resolution is platform agnostic, the process of 
extracting/computing binary ID is. As such it seems like SymbolServerResolver 
can either be a part of LLDB core, or a separate directory in 
Plugins/SymbolVendor, which will then be used by SymbolVendorELF and 
SymbolVendorMacOSX. First both symbol vendors will try to resolve the symbols 
the way they currently do and only if they cannot find anything, will they try 
to use SymbolVendorSymbolServer.

Alternatively symbol server resolution logic can be placed into its own 
SymbolVendorSymbolServer, and modify SymbolVendor FindPlugin's logic such that 
it does not return the first found SymbolVendor instance and instead returns 
either the first SymbolVendor instance that managed to successfully resolve 
symbols or just last one.

Yet another alternative would be to use a delegation chain, such that any 
SymbolVendor could be wrapped into a SymbolVendorSymbolServer, which would 
first try to invoke the delegate and if it cannot find symbols, will try to 
perform its magic. This approach seems nice, but does not play nice with 
current implementation based on static factory method.


Symbol server communication.

Network communication can either be implemented natively for different 
platforms or it can be delegated to a python script invoked by 
ScriptInterpreter. Using Python seems an easier option in order to make this 
cross-platform, but it adds a dependency on Python and will require propagating 
ScriptInterpreter to SymbolVendor creation factory.


Thoughts, suggestions and comments are very welcome.


Thank you,

Taras

LLVM Project Blog: LLDB is Coming to 
Windows<https://urldefense.proofpoint.com/v2/url?u=http-3A__blog.llvm.org_2015_01_lldb-2Dis-2Dcoming-2Dto-2Dwindows.html&d=DQMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=jqaYv5aDYHR_MGTz1rkWPg&m=cGDliRKtgFrgnkWwFqxPh4RyRVqdfaAmOCGP-zL_LbA&s=0aQ3fWYzXJkFu8RXsNn-tueEzkRgtHCC53MVc

Re: [lldb-dev] Symbol Server for everyone.

[Taras Tsugrii via lldb-dev]

Yes, Zachary, design does not have any platform constraints, as long as we have 
a reliable way to identify a binary, which can always be arranged.


From: Zachary Turner 
Sent: Friday, August 26, 2016 10:01:30 PM
To: Taras Tsugrii; lldb-dev@lists.llvm.org
Cc: Kevin Frei
Subject: Re: [lldb-dev] Symbol Server for everyone.

By platform agnostic i mean having a single symbol server that works across 
multiple platforms is very nice. It sounds like in addition to being a symbol 
server this can also serve source code, and should work with embedded debug 
info, split dwo, or even pdb?
On Fri, Aug 26, 2016 at 9:54 PM Taras Tsugrii 
mailto:ttsug...@fb.com>> wrote:

Zachary, I agree that adding a Python dependency might not be a good idea, so 
I'll take a closer look at the network code available in lldb. Symbol format we 
are currently using is pretty simple - every artifact is identified by a type 
(elf, src, etc), an id (build id for binary or hash for source) and a path. I'm 
not sure what you mean by platform agnostic, but with this approach every 
SymbolVendor will just have to pass the appropriate type, build id and a path 
to a ArtifactManager, which will download or locate a locally cached artifact 
and return a path to it.


From: Zachary Turner mailto:ztur...@google.com>>
Sent: Friday, August 26, 2016 8:18:54 PM
To: Taras Tsugrii; lldb-dev@lists.llvm.org<mailto:lldb-dev@lists.llvm.org>
Cc: Kevin Frei
Subject: Re: [lldb-dev] Symbol Server for everyone.

Making the SymbolVendor dependent on Python is probably a non starter, and it 
would also make debugging more difficult.

We have network code for various platforms in Host already.

It would be nice to have a symbol server format that is platform agnostic. On 
the other hand, Microsoft tools already understand their own symbol server 
format , so if i ever reprioritize this, we will probably want the standard 
Microsoft symbol server format on Windows for interoperability.


On Fri, Aug 26, 2016 at 8:00 PM Taras Tsugrii via lldb-dev 
mailto:lldb-dev@lists.llvm.org>> wrote:

Hello lldb developers,


In one of the older posts 
(http://blog.llvm.org/2015/01/lldb-is-coming-to-windows.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__blog.llvm.org_2015_01_lldb-2Dis-2Dcoming-2Dto-2Dwindows.html&d=DQMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=jqaYv5aDYHR_MGTz1rkWPg&m=cGDliRKtgFrgnkWwFqxPh4RyRVqdfaAmOCGP-zL_LbA&s=0aQ3fWYzXJkFu8RXsNn-tueEzkRgtHCC53MVc6Mbbqw&e=>),
 symbol server support was mentioned. Most likely it was meant for Windows, but 
at FB we have our own symbol server implementation for Linux (technically it's 
completely platform agnostic), which we would like to integrate with LLDB and 
eventually open source along with the server. As such I thought I'd ask LLDB 
gurus like you, if anyone is already working on symbol server support and if 
not, I'd appreciate your thoughts on a desired architecture.


General idea.

Based on current LLDB implementation and the fact that symbol server feature is 
a cross-cutting concern, the natural place to put this logic would be inside 
SymbolVendor plugin, which on Mac is used to resolve separate dSYM bundles. In 
theory symbol server logic is completely platform-agnostic, as all we need to 
know is some sort of binary ID (could either be a real .build-id or UUID or 
some custom logic to compute a stable binary hash) and binary name. This info 
can be used to make a network request to check whether corresponding binary 
exists and if so, download it to a temporary location and call 
symbol_vendor->AddSymbolFileRepresentation with FileSpec pointing at that 
temporary location.


Implementation details.


Logic placement.

Even though symbol resolution is platform agnostic, the process of 
extracting/computing binary ID is. As such it seems like SymbolServerResolver 
can either be a part of LLDB core, or a separate directory in 
Plugins/SymbolVendor, which will then be used by SymbolVendorELF and 
SymbolVendorMacOSX. First both symbol vendors will try to resolve the symbols 
the way they currently do and only if they cannot find anything, will they try 
to use SymbolVendorSymbolServer.

Alternatively symbol server resolution logic can be placed into its own 
SymbolVendorSymbolServer, and modify SymbolVendor FindPlugin's logic such that 
it does not return the first found SymbolVendor instance and instead returns 
either the first SymbolVendor instance that managed to successfully resolve 
symbols or just last one.

Yet another alternative would be to use a delegation chain, such that any 
SymbolVendor could be wrapped into a SymbolVendorSymbolServer, which would 
first try to invoke the delegate and if it cannot find symbols, will try to 
perform its magic. This approach seems nice, but does not play nice with 
current implementation based on static factory method.


Sym

Re: [lldb-dev] Symbol Server for everyone.

[Taras Tsugrii via lldb-dev]

 debug info isn't already in the object file
2 - check for the symbols in proximity to the executable (same directory, at 
the bundle level, and a few other places)
3 - check for the symbol file locally in one of our dsymForUUID cache locations
4 - check in common symbol directories (~/Library/Symbols, /Library/Symbols, 
user specified directories)
5 - if enabled, run the dsymForUUID shell script to possibly go out and fetch 
the symbols and cache them locally so that step #3 above can find the symbol 
file the next time without having to run an external shell script tool

Steps 3 through 5 happen in DebugSymbols.framework for us. The nice thing about 
an external tool is it allows the symbol locator to be updated separately from 
LLDB itself and makes it easy to update servers with a new version of a tool 
without having to update the LLDB on the server.

One thing that is key for this to work well is we build UUIDs into each binary. 
The same binary built with the same compiler with the same sources will produce 
the same UUID even if the two binaries were built in different directories. 
This allows our database to not have to store a new path for each build as many 
builds of a binary will share the same dSYM file. This also keeps the size of 
our caches on symbolication servers from filling up with every new 
symbolication request as many stack logs contain the same symbols.

The other way to do this would be to allow each Platform subclass to roll their 
own SymbolServer, or to allow multiple symbol server/symbol cache plugins to 
coexist. Right now Apple just uses the stuff from Symbols.h, but the Apple 
implementation could easily be plug-in-ified. If each platform, like say linux 
or windows, want to roll their own symbol servers, this could just be done by 
extending the Platform.h virtual functions to include functions that allow each 
platform to locate symbols in their own way. This might be easier that trying 
to make a whole new SymbolServer type of plug-in.

Se we have quite a bit of experience with this at Apple. Let us know if you 
have any questions.

Greg Clayton


> On Aug 4, 2016, at 11:34 AM, Taras Tsugrii via lldb-dev 
>  wrote:
>
> Hello lldb developers,
>
> In one of the older posts 
> (https://urldefense.proofpoint.com/v2/url?u=http-3A__blog.llvm.org_2015_01_lldb-2Dis-2Dcoming-2Dto-2Dwindows.html&d=DQIFAw&c=5VD0RTtNlTh3ycd41b3MUw&r=jqaYv5aDYHR_MGTz1rkWPg&m=-5Rr7M5kMrCwSi6BQuqsUTJAykHNh1lFc8VoQ-o04Lo&s=mlWf4Pb7glTpusBnOL457TxuiSHJZo1zYuSxXbRZ4T4&e=
>  ), symbol server support was mentioned. Most likely it was meant for 
> Windows, but at FB we have our own symbol server implementation for Linux 
> (technically it's completely platform agnostic), which we would like to 
> integrate with LLDB and eventually open source along with the server. As such 
> I thought I'd ask LLDB gurus like you, if anyone is already working on symbol 
> server support and if not, I'd appreciate your thoughts on a desired 
> architecture.
>
> General idea.
> Based on current LLDB implementation and the fact that symbol server feature 
> is a cross-cutting concern, the natural place to put this logic would be 
> inside SymbolVendor plugin, which on Mac is used to resolve separate dSYM 
> bundles. In theory symbol server logic is completely platform-agnostic, as 
> all we need to know is some sort of binary ID (could either be a real 
> .build-id or UUID or some custom logic to compute a stable binary hash) and 
> binary name. This info can be used to make a network request to check whether 
> corresponding binary exists and if so, download it to a temporary location 
> and call symbol_vendor->AddSymbolFileRepresentation with FileSpec pointing at 
> that temporary location.
>
> Implementation details.
>
> Logic placement.
> Even though symbol resolution is platform agnostic, the process of 
> extracting/computing binary ID is. As such it seems like SymbolServerResolver 
> can either be a part of LLDB core, or a separate directory in 
> Plugins/SymbolVendor, which will then be used by SymbolVendorELF and 
> SymbolVendorMacOSX. First both symbol vendors will try to resolve the symbols 
> the way they currently do and only if they cannot find anything, will they 
> try to use SymbolVendorSymbolServer.
> Alternatively symbol server resolution logic can be placed into its own 
> SymbolVendorSymbolServer, and modify SymbolVendor FindPlugin's logic such 
> that it does not return the first found SymbolVendor instance and instead 
> returns either the first SymbolVendor instance that managed to successfully 
> resolve symbols or just last one.
> Yet another alternative would be to use a delegation chain, such that any 
> SymbolVendor could be wrapped into a SymbolVendorSymbolServer, which would 
> first try to invoke the delegate and 

Re: [lldb-dev] Symbol Server for everyone.

[Taras Tsugrii via lldb-dev]

Thank you Jim! Sounds like this should work!

On 8/30/16, 8:45 AM, "jing...@apple.com on behalf of Jim Ingham" 
 wrote:

The "dsymForUUID" tool doesn't handle copying source files around - we tend 
to just remote mount them.  But we do include these keys in the return plist so 
that lldb can automatically remap the source files from where they were at 
build time to where they are at debug time.  So if your symbol server copies 
files locally and they aren't in the same location as at build time, you might 
want to play a similar trick on your end.

Jim

> On Aug 30, 2016, at 7:18 AM, Taras Tsugrii via lldb-dev 
 wrote:
> 
> DBGBuildSourcePath
> /path/to/build/sources
> DBGSourcePath
> /path/to/actual/sources



___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev