[PATCH] D101793: [clang][AST] Improve AST Reader/Writer memory footprint

yaxunl added a comment. In D101793#2774191 , @weiwang wrote: > In D101793#2772717 , @yaxunl wrote: > >> In D101793#2772461 , @weiwang >> wrote: >> >>> Thanks for the appr

[PATCH] D101793: [clang][AST] Improve AST Reader/Writer memory footprint

weiwang added a comment. In D101793#2772717 , @yaxunl wrote: > In D101793#2772461 , @weiwang wrote: > >> Thanks for the approval! >> >> Just want to understand the list of "decls to check for deferred >> diagnost

[PATCH] D101793: [clang][AST] Improve AST Reader/Writer memory footprint

yaxunl added a comment. In D101793#2772461 , @weiwang wrote: > Thanks for the approval! > > Just want to understand the list of "decls to check for deferred diagnostics" > better, where are these decls coming from? And why do they need to be checked > f

[PATCH] D101793: [clang][AST] Improve AST Reader/Writer memory footprint

weiwang added a comment. Thanks for the approval! Just want to understand the list of "decls to check for deferred diagnostics" better, where are these decls coming from? And why do they need to be checked for warnings? I see decls from libc are in the list, but I have no idea why are they sel

[PATCH] D101793: [clang][AST] Improve AST Reader/Writer memory footprint

This revision was landed with ongoing or failed builds. This revision was automatically updated to reflect the committed changes. Closed by commit rGe6b8320c0a63: [clang][AST] Improve AST Reader/Writer memory footprint (authored by weiwang). Repository: rG LLVM Github Monorepo CHANGES SINCE LA

[PATCH] D101793: [clang][AST] Improve AST Reader/Writer memory footprint

yaxunl accepted this revision. yaxunl added a comment. This revision is now accepted and ready to land. LGTM. Thanks. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D101793/new/ https://reviews.llvm.org/D101793 __

[PATCH] D101793: [clang][AST] Improve AST Reader/Writer memory footprint

weiwang updated this revision to Diff 346866. weiwang added a comment. make both ASTReader::DeclsToCheckForDeferredDiags and Sema::DeclsToCheckForDeferredDiags SmallSetVector Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D101793/new/ https://review

[PATCH] D101793: [clang][AST] Improve AST Reader/Writer memory footprint

yaxunl added a comment. In D101793#2772033 , @weiwang wrote: > In D101793#2772021 , @yaxunl wrote: > >> In D101793#2769297 , @weiwang >> wrote: >> >>> Tried to make `Sema

[PATCH] D101793: [clang][AST] Improve AST Reader/Writer memory footprint

weiwang added a comment. In D101793#2772021 , @yaxunl wrote: > In D101793#2769297 , @weiwang wrote: > >> Tried to make `Sema::DeclsToCheckForDeferredDiags` `llvm::SmallSetVector`. >> The heap RSS did drop signifi

[PATCH] D101793: [clang][AST] Improve AST Reader/Writer memory footprint

yaxunl added a comment. In D101793#2769297 , @weiwang wrote: > Tried to make `Sema::DeclsToCheckForDeferredDiags` `llvm::SmallSetVector`. > The heap RSS did drop significantly (from peak 100GB to 59GB) , but not as > good as the current fix (peak 26GB),

[PATCH] D101793: [clang][AST] Improve AST Reader/Writer memory footprint

weiwang added a comment. Tried to make `Sema::DeclsToCheckForDeferredDiags` `llvm::SmallSetVector`. The heap RSS did drop significantly (from peak 100GB to 59GB) , but not as good as the current fix (peak 26GB), which makes `ASTReader::DeclsToCheckForDeferredDiags` `llvm::SmallSetVector`. I th

[PATCH] D101793: [clang][AST] Improve AST Reader/Writer memory footprint

weiwang added a comment. In D101793#2760639 , @yaxunl wrote: > I think the root cause might be duplicated decls are added to > Sema::DeclsToCheckForDeferredDiags defined in > > https://github.com/llvm/llvm-project/blob/main/clang/include/clang/Sema/Sema.

[PATCH] D101793: [clang][AST] Improve AST Reader/Writer memory footprint

yaxunl added a comment. I think the root cause might be duplicated decls are added to Sema::DeclsToCheckForDeferredDiags defined in https://github.com/llvm/llvm-project/blob/main/clang/include/clang/Sema/Sema.h#L1789 When compiling source codes, a decl is added only once. However if modules are

[PATCH] D101793: [clang][AST] Improve AST Reader/Writer memory footprint

weiwang added a comment. Finally dealt with the other issues I need to take care. Let's resume the discussion. I printed out the decls that are duplicated. The list of very long, but the top ones all seem from glibc headers. For example (the number after 'c:' is the duplication count), stat

[PATCH] D101793: [clang][AST] Improve AST Reader/Writer memory footprint

weiwang added a comment. In D101793#2735336 , @yaxunl wrote: > Decls in Sema::DeclsToCheckForDeferredDiags is supposed to be unique. > Therefore the fact that '1,734,387,685 out of 1,734,404,000 elements are the > same' is surprising. Did this happen wh

[PATCH] D101793: [clang][AST] Improve AST Reader/Writer memory footprint

yaxunl added a comment. Decls in Sema::DeclsToCheckForDeferredDiags is supposed to be unique. Therefore the fact that '1,734,387,685 out of 1,734,404,000 elements are the same' is surprising. Did this happen when you compile the source code and write AST? What language was the source code? C++,

[PATCH] D101793: [clang][AST] Improve AST Reader/Writer memory footprint

weiwang added a comment. We've seen a huge memory footprint from AST Reader/Writer in a single CU with module enabled. Upon further analysis, the content of vector `DeclsToCheckForDeferredDiags` seems mostly redundant. In one case, 1,734,387,685 out of 1,734,404,000 elements are the same. While

[PATCH] D101793: [clang][AST] Improve AST Reader/Writer memory footprint

weiwang created this revision. Herald added subscribers: hoy, wenlei. weiwang requested review of this revision. Herald added a project: clang. Herald added a subscriber: cfe-commits. Reduce memory footprint of AST Reader/Writer: 1. Adjust internal data containers' element type. 2. Switch to set