[PATCH] D153769: [clangd] Implement the 'Organize Imports' source action. Fix include-cleaner findings in batch.
VitaNuo updated this revision to Diff 557771. VitaNuo marked 10 inline comments as done. VitaNuo added a comment. Address review comments. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D153769/new/ https://reviews.llvm.org/D153769 Files: clang-tools-extra/clangd/ClangdLSPServer.cpp clang-tools-extra/clangd/ClangdServer.cpp clang-tools-extra/clangd/ClangdServer.h clang-tools-extra/clangd/ParsedAST.cpp clang-tools-extra/clangd/ParsedAST.h clang-tools-extra/clangd/Protocol.cpp clang-tools-extra/clangd/Protocol.h clang-tools-extra/clangd/refactor/Tweak.cpp clang-tools-extra/clangd/refactor/Tweak.h clang-tools-extra/clangd/refactor/tweaks/CMakeLists.txt clang-tools-extra/clangd/refactor/tweaks/OrganizeImports.cpp clang-tools-extra/clangd/test/code-action-request.test clang-tools-extra/clangd/test/include-cleaner-batch-fix.test clang-tools-extra/clangd/test/request-reply.test clang-tools-extra/clangd/test/tweaks-format.test clang-tools-extra/clangd/tool/Check.cpp clang-tools-extra/clangd/unittests/CMakeLists.txt clang-tools-extra/clangd/unittests/ClangdTests.cpp clang-tools-extra/clangd/unittests/FeatureModulesTests.cpp clang-tools-extra/clangd/unittests/tweaks/OrganizeImportsTests.cpp clang-tools-extra/clangd/unittests/tweaks/TweakTesting.cpp clang-tools-extra/clangd/unittests/tweaks/TweakTesting.h Index: clang-tools-extra/clangd/unittests/tweaks/TweakTesting.h === --- clang-tools-extra/clangd/unittests/tweaks/TweakTesting.h +++ clang-tools-extra/clangd/unittests/tweaks/TweakTesting.h @@ -18,6 +18,7 @@ #include "gtest/gtest.h" #include #include +#include namespace clang { namespace clangd { @@ -88,13 +89,17 @@ // - if the tweak produces a message, returns "message:\n" // - if prepare() returns false, returns "unavailable" // - if apply() returns an error, returns "fail: " - std::string apply(llvm::StringRef MarkedCode, -llvm::StringMap *EditedFiles = nullptr) const; + std::string + apply(llvm::StringRef MarkedCode, +llvm::StringMap *EditedFiles = nullptr, +const std::vector &RequestedActionKinds = {}) const; // Helpers for EXPECT_AVAILABLE/EXPECT_UNAVAILABLE macros. using WrappedAST = std::pair; WrappedAST build(llvm::StringRef) const; - bool isAvailable(WrappedAST &, llvm::Annotations::Range) const; + bool + isAvailable(WrappedAST &, llvm::Annotations::Range, + const std::vector &RequestedActionKinds = {}) const; // Return code re-decorated with a single point/range. static std::string decorate(llvm::StringRef, unsigned); static std::string decorate(llvm::StringRef, llvm::Annotations::Range); @@ -116,9 +121,10 @@ auto AST = build(A.code());\ assert(!A.points().empty() || !A.ranges().empty());\ for (const auto &P : A.points()) \ - EXPECT_EQ(Available, isAvailable(AST, {P, P})) << decorate(A.code(), P); \ + EXPECT_EQ(Available, isAvailable(AST, {P, P}, {})) \ + << decorate(A.code(), P);\ for (const auto &R : A.ranges()) \ - EXPECT_EQ(Available, isAvailable(AST, R)) << decorate(A.code(), R); \ + EXPECT_EQ(Available, isAvailable(AST, R, {})) << decorate(A.code(), R); \ } while (0) #define EXPECT_AVAILABLE(MarkedCode) EXPECT_AVAILABLE_(MarkedCode, true) #define EXPECT_UNAVAILABLE(MarkedCode) EXPECT_AVAILABLE_(MarkedCode, false) Index: clang-tools-extra/clangd/unittests/tweaks/TweakTesting.cpp === --- clang-tools-extra/clangd/unittests/tweaks/TweakTesting.cpp +++ clang-tools-extra/clangd/unittests/tweaks/TweakTesting.cpp @@ -8,6 +8,7 @@ #include "TweakTesting.h" +#include "Protocol.h" #include "SourceCode.h" #include "TestTU.h" #include "refactor/Tweak.h" @@ -16,6 +17,7 @@ #include "gtest/gtest.h" #include #include +#include namespace clang { namespace clangd { @@ -63,12 +65,14 @@ // Returns std::nullopt if and only if prepare() failed. std::optional> applyTweak(ParsedAST &AST, llvm::Annotations::Range Range, StringRef TweakID, - const SymbolIndex *Index, llvm::vfs::FileSystem *FS) { + const SymbolIndex *Index, llvm::vfs::FileSystem *FS, + const std::vector &RequestedActionKinds) { std::optional> Result; SelectionTree::createEach(AST.getASTContext(), AST.getTokens(), Range.Begin, Range.End, [&](SelectionTree ST) { Tweak::Selection S(Index, AST, Range.Begin, - Range.End, std::move(ST), FS); + Range.End, std::move(ST)
[PATCH] D153769: [clangd] Implement the 'Organize Imports' source action. Fix include-cleaner findings in batch.
VitaNuo added a comment. Thanks! Comment at: clang-tools-extra/clangd/unittests/tweaks/OrganizeImportsTests.cpp:35-41 + { + R"cpp( +#include "Te^stTU.h" +)cpp", + true, + {}}, + {"void foo(^) {}", false, {}}}; kadircet wrote: > nit: you can use EXPECT_AVAILABLE and EXPECT_UNAVAILABLE directly for these > two cases I think it's the other way around. I can use the macros directly when I don't need to pass specific requested action kinds, which is the last (third) case. So it's probably not worth it. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D153769/new/ https://reviews.llvm.org/D153769 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D139409: [include-cleaner] Handle dependent type members in AST
VitaNuo updated this revision to Diff 483881. VitaNuo marked 6 inline comments as done. VitaNuo added a comment. Address review comments. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D139409/new/ https://reviews.llvm.org/D139409 Files: clang-tools-extra/include-cleaner/lib/WalkAST.cpp clang-tools-extra/include-cleaner/unittests/WalkASTTest.cpp Index: clang-tools-extra/include-cleaner/unittests/WalkASTTest.cpp === --- clang-tools-extra/include-cleaner/unittests/WalkASTTest.cpp +++ clang-tools-extra/include-cleaner/unittests/WalkASTTest.cpp @@ -229,6 +229,13 @@ namespace ns { template struct Foo { int a; }; } using ns::$explicit^Foo;)cpp", "void k(Foo b) { b.^a; }"); + // Test the dependent-type case (CXXDependentScopeMemberExpr) + testWalk("template struct $explicit^Base { void method(); };", + "template void k(Base t) { t.^method(); }"); + testWalk("template struct $explicit^Base { void method(); };", + "template void k(Base& t) { t.^method(); }"); + testWalk("template struct $explicit^Base { void method(); };", + "template void k(Base* t) { t->^method(); }"); } TEST(WalkAST, ConstructExprs) { Index: clang-tools-extra/include-cleaner/lib/WalkAST.cpp === --- clang-tools-extra/include-cleaner/lib/WalkAST.cpp +++ clang-tools-extra/include-cleaner/lib/WalkAST.cpp @@ -11,6 +11,7 @@ #include "clang/AST/DeclCXX.h" #include "clang/AST/Expr.h" #include "clang/AST/ExprCXX.h" +#include "clang/AST/NestedNameSpecifier.h" #include "clang/AST/RecursiveASTVisitor.h" #include "clang/AST/TemplateName.h" #include "clang/AST/Type.h" @@ -84,6 +85,10 @@ report(E->getMemberLoc(), getMemberProvider(Type)); return true; } + bool VisitCXXDependentScopeMemberExpr(CXXDependentScopeMemberExpr *E) { +report(E->getMemberLoc(), getMemberProvider(E->getBaseType())); +return true; + } bool VisitCXXConstructExpr(CXXConstructExpr *E) { report(E->getLocation(), E->getConstructor(), Index: clang-tools-extra/include-cleaner/unittests/WalkASTTest.cpp === --- clang-tools-extra/include-cleaner/unittests/WalkASTTest.cpp +++ clang-tools-extra/include-cleaner/unittests/WalkASTTest.cpp @@ -229,6 +229,13 @@ namespace ns { template struct Foo { int a; }; } using ns::$explicit^Foo;)cpp", "void k(Foo b) { b.^a; }"); + // Test the dependent-type case (CXXDependentScopeMemberExpr) + testWalk("template struct $explicit^Base { void method(); };", + "template void k(Base t) { t.^method(); }"); + testWalk("template struct $explicit^Base { void method(); };", + "template void k(Base& t) { t.^method(); }"); + testWalk("template struct $explicit^Base { void method(); };", + "template void k(Base* t) { t->^method(); }"); } TEST(WalkAST, ConstructExprs) { Index: clang-tools-extra/include-cleaner/lib/WalkAST.cpp === --- clang-tools-extra/include-cleaner/lib/WalkAST.cpp +++ clang-tools-extra/include-cleaner/lib/WalkAST.cpp @@ -11,6 +11,7 @@ #include "clang/AST/DeclCXX.h" #include "clang/AST/Expr.h" #include "clang/AST/ExprCXX.h" +#include "clang/AST/NestedNameSpecifier.h" #include "clang/AST/RecursiveASTVisitor.h" #include "clang/AST/TemplateName.h" #include "clang/AST/Type.h" @@ -84,6 +85,10 @@ report(E->getMemberLoc(), getMemberProvider(Type)); return true; } + bool VisitCXXDependentScopeMemberExpr(CXXDependentScopeMemberExpr *E) { +report(E->getMemberLoc(), getMemberProvider(E->getBaseType())); +return true; + } bool VisitCXXConstructExpr(CXXConstructExpr *E) { report(E->getLocation(), E->getConstructor(), ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D139409: [include-cleaner] Handle dependent type members in AST
VitaNuo added a comment. Thanks for the comments! Comment at: clang-tools-extra/include-cleaner/lib/WalkAST.cpp:51 Type = Type->getPointeeType(); +if (const TemplateSpecializationType *TST = +Type->getAs()) { hokein wrote: > nit: we can use `const auto*` here as the type is explicitly spelled in the > `getAs` Seems to be obsolete with the above patch. Comment at: clang-tools-extra/include-cleaner/lib/WalkAST.cpp:52 +if (const TemplateSpecializationType *TST = +Type->getAs()) { + return TST->getTemplateName().getAsTemplateDecl(); hokein wrote: > nit: remove the {} around the if statement Seems to be obsolete with the above patch. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D139409/new/ https://reviews.llvm.org/D139409 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D139409: [include-cleaner] Handle dependent type members in AST
VitaNuo updated this revision to Diff 483882. VitaNuo added a comment. Remove unusued include. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D139409/new/ https://reviews.llvm.org/D139409 Files: clang-tools-extra/include-cleaner/lib/WalkAST.cpp clang-tools-extra/include-cleaner/unittests/WalkASTTest.cpp Index: clang-tools-extra/include-cleaner/unittests/WalkASTTest.cpp === --- clang-tools-extra/include-cleaner/unittests/WalkASTTest.cpp +++ clang-tools-extra/include-cleaner/unittests/WalkASTTest.cpp @@ -229,6 +229,13 @@ namespace ns { template struct Foo { int a; }; } using ns::$explicit^Foo;)cpp", "void k(Foo b) { b.^a; }"); + // Test the dependent-type case (CXXDependentScopeMemberExpr) + testWalk("template struct $explicit^Base { void method(); };", + "template void k(Base t) { t.^method(); }"); + testWalk("template struct $explicit^Base { void method(); };", + "template void k(Base& t) { t.^method(); }"); + testWalk("template struct $explicit^Base { void method(); };", + "template void k(Base* t) { t->^method(); }"); } TEST(WalkAST, ConstructExprs) { Index: clang-tools-extra/include-cleaner/lib/WalkAST.cpp === --- clang-tools-extra/include-cleaner/lib/WalkAST.cpp +++ clang-tools-extra/include-cleaner/lib/WalkAST.cpp @@ -84,6 +84,10 @@ report(E->getMemberLoc(), getMemberProvider(Type)); return true; } + bool VisitCXXDependentScopeMemberExpr(CXXDependentScopeMemberExpr *E) { +report(E->getMemberLoc(), getMemberProvider(E->getBaseType())); +return true; + } bool VisitCXXConstructExpr(CXXConstructExpr *E) { report(E->getLocation(), E->getConstructor(), Index: clang-tools-extra/include-cleaner/unittests/WalkASTTest.cpp === --- clang-tools-extra/include-cleaner/unittests/WalkASTTest.cpp +++ clang-tools-extra/include-cleaner/unittests/WalkASTTest.cpp @@ -229,6 +229,13 @@ namespace ns { template struct Foo { int a; }; } using ns::$explicit^Foo;)cpp", "void k(Foo b) { b.^a; }"); + // Test the dependent-type case (CXXDependentScopeMemberExpr) + testWalk("template struct $explicit^Base { void method(); };", + "template void k(Base t) { t.^method(); }"); + testWalk("template struct $explicit^Base { void method(); };", + "template void k(Base& t) { t.^method(); }"); + testWalk("template struct $explicit^Base { void method(); };", + "template void k(Base* t) { t->^method(); }"); } TEST(WalkAST, ConstructExprs) { Index: clang-tools-extra/include-cleaner/lib/WalkAST.cpp === --- clang-tools-extra/include-cleaner/lib/WalkAST.cpp +++ clang-tools-extra/include-cleaner/lib/WalkAST.cpp @@ -84,6 +84,10 @@ report(E->getMemberLoc(), getMemberProvider(Type)); return true; } + bool VisitCXXDependentScopeMemberExpr(CXXDependentScopeMemberExpr *E) { +report(E->getMemberLoc(), getMemberProvider(E->getBaseType())); +return true; + } bool VisitCXXConstructExpr(CXXConstructExpr *E) { report(E->getLocation(), E->getConstructor(), ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D139716: [include-cleaner] Use expansion locations for macros.
VitaNuo updated this revision to Diff 483910. VitaNuo added a comment. Address review comments. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D139716/new/ https://reviews.llvm.org/D139716 Files: clang-tools-extra/include-cleaner/lib/FindHeaders.cpp clang-tools-extra/include-cleaner/unittests/FindHeadersTest.cpp Index: clang-tools-extra/include-cleaner/unittests/FindHeadersTest.cpp === --- clang-tools-extra/include-cleaner/unittests/FindHeadersTest.cpp +++ clang-tools-extra/include-cleaner/unittests/FindHeadersTest.cpp @@ -7,11 +7,17 @@ //===--===// #include "AnalysisInternal.h" +#include "clang-include-cleaner/Analysis.h" #include "clang-include-cleaner/Record.h" +#include "clang-include-cleaner/Types.h" +#include "clang/AST/RecursiveASTVisitor.h" #include "clang/Basic/FileEntry.h" #include "clang/Basic/FileManager.h" #include "clang/Frontend/FrontendActions.h" #include "clang/Testing/TestAST.h" +#include "llvm/ADT/ArrayRef.h" +#include "llvm/Support/raw_ostream.h" +#include "llvm/Testing/Support/Annotations.h" #include "gmock/gmock.h" #include "gtest/gtest.h" #include @@ -31,7 +37,7 @@ std::unique_ptr AST; FindHeadersTest() { Inputs.MakeAction = [this] { - struct Hook : public PreprocessOnlyAction { + struct Hook : public SyntaxOnlyAction { public: Hook(PragmaIncludes *Out) : Out(Out) {} bool BeginSourceFileAction(clang::CompilerInstance &CI) override { @@ -153,5 +159,89 @@ physicalHeader("exporter.h"))); } +struct CustomVisitor : RecursiveASTVisitor { + const Decl *Out = nullptr; + + bool VisitCXXRecordDecl(const CXXRecordDecl *ND) { +if (ND->getName() == "Foo") { + EXPECT_TRUE(Out == nullptr); + Out = ND; +} +return true; + } + + bool VisitNamedDecl(const NamedDecl *ND) { +if (ND->getName() == "FLAG_foo") { + EXPECT_TRUE(Out == nullptr); + Out = ND; +} +return true; + } +}; + +TEST_F(FindHeadersTest, TargetIsExpandedFromMacroInHeader) { + struct { +llvm::StringRef Main; +llvm::StringRef DeclareHeader; +llvm::StringRef MacroHeader; + } TestCases[] = {{ + R"cpp( +#include "declare.h" +Foo foo; + )cpp", + R"cpp( +#include "macro.h" +DEFINE_CLASS(Foo) + )cpp", + R"cpp( +#define DEFINE_CLASS(name) class name {}; + )cpp"}, + { + R"cpp( +#include "declare.h" +void test(Foo foo); + )cpp", + R"cpp( +#include "macro.h" +DEFINE_Foo + )cpp", + R"cpp( +#define DEFINE_Foo class Foo {}; + )cpp"}, + { + R"cpp( +#include "declare.h" +void foo() { + FLAG_foo; +} + )cpp", + R"cpp( +#include "macro.h" +DECLARE_FLAGS(foo); + )cpp", + R"cpp( +#define DECLARE_FLAGS(name) extern int FLAG_##name + )cpp"}}; + + for (const auto &T : TestCases) { +llvm::Annotations MainFile(T.Main); +Inputs.Code = MainFile.code(); +Inputs.ExtraFiles["declare.h"] = T.DeclareHeader; +Inputs.ExtraFiles["macro.h"] = T.MacroHeader; +buildAST(); + +CustomVisitor Visitor; +for (Decl *D : AST->context().getTranslationUnitDecl()->decls()) { + Visitor.TraverseDecl(D); +} + +const Decl *Foo = Visitor.Out; +llvm::SmallVector Headers = clang::include_cleaner::findHeaders( +Foo->getLocation(), AST->sourceManager(), nullptr); +EXPECT_EQ(Headers.size(), 1u); +EXPECT_THAT(Headers, UnorderedElementsAre(physicalHeader("declare.h"))); + } +} + } // namespace } // namespace clang::include_cleaner Index: clang-tools-extra/include-cleaner/lib/FindHeaders.cpp === --- clang-tools-extra/include-cleaner/lib/FindHeaders.cpp +++ clang-tools-extra/include-cleaner/lib/FindHeaders.cpp @@ -18,8 +18,7 @@ llvm::SmallVector Results; switch (Loc.kind()) { case SymbolLocation::Physical: { -// FIXME: Handle macro locations. -FileID FID = SM.getFileID(Loc.physical()); +FileID FID = SM.getFileID(SM.getExpansionLoc(Loc.physical())); const FileEntry *FE = SM.getFileEntryForID(FID); if (!PI) { return FE ? llvm::SmallVector{Header(FE)} ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D139716: [include-cleaner] Use expansion locations for macros.
VitaNuo updated this revision to Diff 483911. VitaNuo added a comment. Add guard calls to headers. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D139716/new/ https://reviews.llvm.org/D139716 Files: clang-tools-extra/include-cleaner/lib/FindHeaders.cpp clang-tools-extra/include-cleaner/unittests/FindHeadersTest.cpp Index: clang-tools-extra/include-cleaner/unittests/FindHeadersTest.cpp === --- clang-tools-extra/include-cleaner/unittests/FindHeadersTest.cpp +++ clang-tools-extra/include-cleaner/unittests/FindHeadersTest.cpp @@ -7,11 +7,17 @@ //===--===// #include "AnalysisInternal.h" +#include "clang-include-cleaner/Analysis.h" #include "clang-include-cleaner/Record.h" +#include "clang-include-cleaner/Types.h" +#include "clang/AST/RecursiveASTVisitor.h" #include "clang/Basic/FileEntry.h" #include "clang/Basic/FileManager.h" #include "clang/Frontend/FrontendActions.h" #include "clang/Testing/TestAST.h" +#include "llvm/ADT/ArrayRef.h" +#include "llvm/Support/raw_ostream.h" +#include "llvm/Testing/Support/Annotations.h" #include "gmock/gmock.h" #include "gtest/gtest.h" #include @@ -31,7 +37,7 @@ std::unique_ptr AST; FindHeadersTest() { Inputs.MakeAction = [this] { - struct Hook : public PreprocessOnlyAction { + struct Hook : public SyntaxOnlyAction { public: Hook(PragmaIncludes *Out) : Out(Out) {} bool BeginSourceFileAction(clang::CompilerInstance &CI) override { @@ -153,5 +159,89 @@ physicalHeader("exporter.h"))); } +struct CustomVisitor : RecursiveASTVisitor { + const Decl *Out = nullptr; + + bool VisitCXXRecordDecl(const CXXRecordDecl *ND) { +if (ND->getName() == "Foo") { + EXPECT_TRUE(Out == nullptr); + Out = ND; +} +return true; + } + + bool VisitNamedDecl(const NamedDecl *ND) { +if (ND->getName() == "FLAG_foo") { + EXPECT_TRUE(Out == nullptr); + Out = ND; +} +return true; + } +}; + +TEST_F(FindHeadersTest, TargetIsExpandedFromMacroInHeader) { + struct { +llvm::StringRef Main; +llvm::StringRef DeclareHeader; +llvm::StringRef MacroHeader; + } TestCases[] = {{ + R"cpp( +#include "declare.h" +Foo foo; + )cpp", + R"cpp( +#include "macro.h" +DEFINE_CLASS(Foo) + )cpp", + R"cpp( +#define DEFINE_CLASS(name) class name {}; + )cpp"}, + { + R"cpp( +#include "declare.h" +void test(Foo foo); + )cpp", + R"cpp( +#include "macro.h" +DEFINE_Foo + )cpp", + R"cpp( +#define DEFINE_Foo class Foo {}; + )cpp"}, + { + R"cpp( +#include "declare.h" +void foo() { + FLAG_foo; +} + )cpp", + R"cpp( +#include "macro.h" +DECLARE_FLAGS(foo); + )cpp", + R"cpp( +#define DECLARE_FLAGS(name) extern int FLAG_##name + )cpp"}}; + + for (const auto &T : TestCases) { +llvm::Annotations MainFile(T.Main); +Inputs.Code = MainFile.code(); +Inputs.ExtraFiles["declare.h"] = guard(T.DeclareHeader); +Inputs.ExtraFiles["macro.h"] = guard(T.MacroHeader); +buildAST(); + +CustomVisitor Visitor; +for (Decl *D : AST->context().getTranslationUnitDecl()->decls()) { + Visitor.TraverseDecl(D); +} + +const Decl *Foo = Visitor.Out; +llvm::SmallVector Headers = clang::include_cleaner::findHeaders( +Foo->getLocation(), AST->sourceManager(), nullptr); +EXPECT_EQ(Headers.size(), 1u); +EXPECT_THAT(Headers, UnorderedElementsAre(physicalHeader("declare.h"))); + } +} + } // namespace } // namespace clang::include_cleaner Index: clang-tools-extra/include-cleaner/lib/FindHeaders.cpp === --- clang-tools-extra/include-cleaner/lib/FindHeaders.cpp +++ clang-tools-extra/include-cleaner/lib/FindHeaders.cpp @@ -18,8 +18,7 @@ llvm::SmallVector Results; switch (Loc.kind()) { case SymbolLocation::Physical: { -// FIXME: Handle macro locations. -FileID FID = SM.getFileID(Loc.physical()); +FileID FID = SM.getFileID(SM.getExpansionLoc(Loc.physical())); const FileEntry *FE = SM.getFileEntryForID(FID); if (!PI) { return FE ? llvm::SmallVector{Header(FE)} ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D139716: [include-cleaner] Use expansion locations for macros.
VitaNuo marked an inline comment as done. VitaNuo added inline comments. Comment at: clang-tools-extra/include-cleaner/unittests/FindHeadersTest.cpp:161 +TEST_F(FindHeadersTest, TargetIsArgumentExpandedFromMacroInHeader) { + llvm::Annotations MainFile(R"cpp( hokein wrote: > these 3 newly-added tests are similar, instead of duplicating them, we can > use a table-gen style test, see an > [example](https://github.com/llvm/llvm-project/blob/main/clang-tools-extra/include-cleaner/unittests/AnalysisTest.cpp#L265) Great idea, thanks! Comment at: clang-tools-extra/include-cleaner/unittests/FindHeadersTest.cpp:186 + std::vector Headers; + walkUsed( + TopLevelDecls, {}, nullptr, AST->sourceManager(), hokein wrote: > The intention of the test is to test the function `findHeaders`, we're > testing it in an implicit way (`findHeaders` is hidden inside the > `walkUsed`), there is no code here explicitly calling this function. > > We can make it clearer (no need to use walkUsed): > 1. we get a `Foo` Decl AST node from the testcase. > 2. with the AST node from step1, we cam call the `findHeaders` directly in > the test code. > > for 1, we can just write a simple RecursiveASTVisitor, and capture a > NamedDecl which called Foo, something like: > > ``` > struct Visitor : RecursiveASTVisitor { > const NamedDecl *Out = nullptr; > bool VisitNamedDecl(const NamedDecl *ND) { > if (ND->getName() == "Foo") { > EXPECT_TRUE(Out == nullptr); > Out = ND; > } > return true; > } > }; > ``` Ok, I've attempted to follow your advice now, please have a look. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D139716/new/ https://reviews.llvm.org/D139716 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D139409: [include-cleaner] Handle dependent type members in AST
This revision was automatically updated to reflect the committed changes. Closed by commit rG0e545816a9e5: [include-cleaner] Handle dependent type members in AST. (authored by VitaNuo). Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D139409/new/ https://reviews.llvm.org/D139409 Files: clang-tools-extra/include-cleaner/lib/WalkAST.cpp clang-tools-extra/include-cleaner/unittests/WalkASTTest.cpp Index: clang-tools-extra/include-cleaner/unittests/WalkASTTest.cpp === --- clang-tools-extra/include-cleaner/unittests/WalkASTTest.cpp +++ clang-tools-extra/include-cleaner/unittests/WalkASTTest.cpp @@ -229,6 +229,13 @@ namespace ns { template struct Foo { int a; }; } using ns::$explicit^Foo;)cpp", "void k(Foo b) { b.^a; }"); + // Test the dependent-type case (CXXDependentScopeMemberExpr) + testWalk("template struct $explicit^Base { void method(); };", + "template void k(Base t) { t.^method(); }"); + testWalk("template struct $explicit^Base { void method(); };", + "template void k(Base& t) { t.^method(); }"); + testWalk("template struct $explicit^Base { void method(); };", + "template void k(Base* t) { t->^method(); }"); } TEST(WalkAST, ConstructExprs) { Index: clang-tools-extra/include-cleaner/lib/WalkAST.cpp === --- clang-tools-extra/include-cleaner/lib/WalkAST.cpp +++ clang-tools-extra/include-cleaner/lib/WalkAST.cpp @@ -84,6 +84,10 @@ report(E->getMemberLoc(), getMemberProvider(Type)); return true; } + bool VisitCXXDependentScopeMemberExpr(CXXDependentScopeMemberExpr *E) { +report(E->getMemberLoc(), getMemberProvider(E->getBaseType())); +return true; + } bool VisitCXXConstructExpr(CXXConstructExpr *E) { report(E->getLocation(), E->getConstructor(), Index: clang-tools-extra/include-cleaner/unittests/WalkASTTest.cpp === --- clang-tools-extra/include-cleaner/unittests/WalkASTTest.cpp +++ clang-tools-extra/include-cleaner/unittests/WalkASTTest.cpp @@ -229,6 +229,13 @@ namespace ns { template struct Foo { int a; }; } using ns::$explicit^Foo;)cpp", "void k(Foo b) { b.^a; }"); + // Test the dependent-type case (CXXDependentScopeMemberExpr) + testWalk("template struct $explicit^Base { void method(); };", + "template void k(Base t) { t.^method(); }"); + testWalk("template struct $explicit^Base { void method(); };", + "template void k(Base& t) { t.^method(); }"); + testWalk("template struct $explicit^Base { void method(); };", + "template void k(Base* t) { t->^method(); }"); } TEST(WalkAST, ConstructExprs) { Index: clang-tools-extra/include-cleaner/lib/WalkAST.cpp === --- clang-tools-extra/include-cleaner/lib/WalkAST.cpp +++ clang-tools-extra/include-cleaner/lib/WalkAST.cpp @@ -84,6 +84,10 @@ report(E->getMemberLoc(), getMemberProvider(Type)); return true; } + bool VisitCXXDependentScopeMemberExpr(CXXDependentScopeMemberExpr *E) { +report(E->getMemberLoc(), getMemberProvider(E->getBaseType())); +return true; + } bool VisitCXXConstructExpr(CXXConstructExpr *E) { report(E->getLocation(), E->getConstructor(), ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D139716: [include-cleaner] Use expansion locations for macros.
VitaNuo updated this revision to Diff 483926. VitaNuo marked 7 inline comments as done. VitaNuo added a comment. Address most recent review comments. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D139716/new/ https://reviews.llvm.org/D139716 Files: clang-tools-extra/include-cleaner/lib/FindHeaders.cpp clang-tools-extra/include-cleaner/unittests/FindHeadersTest.cpp Index: clang-tools-extra/include-cleaner/unittests/FindHeadersTest.cpp === --- clang-tools-extra/include-cleaner/unittests/FindHeadersTest.cpp +++ clang-tools-extra/include-cleaner/unittests/FindHeadersTest.cpp @@ -7,11 +7,17 @@ //===--===// #include "AnalysisInternal.h" +#include "clang-include-cleaner/Analysis.h" #include "clang-include-cleaner/Record.h" +#include "clang-include-cleaner/Types.h" +#include "clang/AST/RecursiveASTVisitor.h" #include "clang/Basic/FileEntry.h" #include "clang/Basic/FileManager.h" #include "clang/Frontend/FrontendActions.h" #include "clang/Testing/TestAST.h" +#include "llvm/ADT/ArrayRef.h" +#include "llvm/Support/raw_ostream.h" +#include "llvm/Testing/Support/Annotations.h" #include "gmock/gmock.h" #include "gtest/gtest.h" #include @@ -31,7 +37,7 @@ std::unique_ptr AST; FindHeadersTest() { Inputs.MakeAction = [this] { - struct Hook : public PreprocessOnlyAction { + struct Hook : public SyntaxOnlyAction { public: Hook(PragmaIncludes *Out) : Out(Out) {} bool BeginSourceFileAction(clang::CompilerInstance &CI) override { @@ -153,5 +159,61 @@ physicalHeader("exporter.h"))); } +TEST_F(FindHeadersTest, TargetIsExpandedFromMacroInHeader) { + struct CustomVisitor : RecursiveASTVisitor { +const Decl *Out = nullptr; +bool VisitNamedDecl(const NamedDecl *ND) { + if (ND->getName() == "FLAG_foo" || ND->getName() == "Foo") { +EXPECT_TRUE(Out == nullptr); +Out = ND; + } + return true; +} + }; + + struct { +llvm::StringRef DeclareHeader; +llvm::StringRef MacroHeader; + } TestCases[] = {{ + R"cpp( +#include "macro.h" +DEFINE_CLASS(Foo) + )cpp", + R"cpp( +#define DEFINE_CLASS(name) class name {}; + )cpp"}, + { + R"cpp( +#include "macro.h" +DEFINE_Foo + )cpp", + R"cpp( +#define DEFINE_Foo class Foo {}; + )cpp"}, + { + R"cpp( +#include "macro.h" +DECLARE_FLAGS(foo); + )cpp", + R"cpp( +#define DECLARE_FLAGS(name) extern int FLAG_##name + )cpp"}}; + + for (const auto &T : TestCases) { +Inputs.Code = R"cpp(#include "declare.h")cpp"; +Inputs.ExtraFiles["declare.h"] = guard(T.DeclareHeader); +Inputs.ExtraFiles["macro.h"] = guard(T.MacroHeader); +buildAST(); + +CustomVisitor Visitor; +Visitor.TraverseDecl(AST->context().getTranslationUnitDecl()); + +const Decl *Foo = Visitor.Out; +llvm::SmallVector Headers = clang::include_cleaner::findHeaders( +Foo->getLocation(), AST->sourceManager(), nullptr); +EXPECT_THAT(Headers, UnorderedElementsAre(physicalHeader("declare.h"))); + } +} + } // namespace } // namespace clang::include_cleaner Index: clang-tools-extra/include-cleaner/lib/FindHeaders.cpp === --- clang-tools-extra/include-cleaner/lib/FindHeaders.cpp +++ clang-tools-extra/include-cleaner/lib/FindHeaders.cpp @@ -18,8 +18,7 @@ llvm::SmallVector Results; switch (Loc.kind()) { case SymbolLocation::Physical: { -// FIXME: Handle macro locations. -FileID FID = SM.getFileID(Loc.physical()); +FileID FID = SM.getFileID(SM.getExpansionLoc(Loc.physical())); const FileEntry *FE = SM.getFileEntryForID(FID); if (!PI) { return FE ? llvm::SmallVector{Header(FE)} ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D139716: [include-cleaner] Use expansion locations for macros.
VitaNuo added a comment. Thanks for the review! Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D139716/new/ https://reviews.llvm.org/D139716 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D139716: [include-cleaner] Use expansion locations for macros.
VitaNuo updated this revision to Diff 483927. VitaNuo added a comment. Remove extra variable. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D139716/new/ https://reviews.llvm.org/D139716 Files: clang-tools-extra/include-cleaner/lib/FindHeaders.cpp clang-tools-extra/include-cleaner/unittests/FindHeadersTest.cpp Index: clang-tools-extra/include-cleaner/unittests/FindHeadersTest.cpp === --- clang-tools-extra/include-cleaner/unittests/FindHeadersTest.cpp +++ clang-tools-extra/include-cleaner/unittests/FindHeadersTest.cpp @@ -7,11 +7,17 @@ //===--===// #include "AnalysisInternal.h" +#include "clang-include-cleaner/Analysis.h" #include "clang-include-cleaner/Record.h" +#include "clang-include-cleaner/Types.h" +#include "clang/AST/RecursiveASTVisitor.h" #include "clang/Basic/FileEntry.h" #include "clang/Basic/FileManager.h" #include "clang/Frontend/FrontendActions.h" #include "clang/Testing/TestAST.h" +#include "llvm/ADT/ArrayRef.h" +#include "llvm/Support/raw_ostream.h" +#include "llvm/Testing/Support/Annotations.h" #include "gmock/gmock.h" #include "gtest/gtest.h" #include @@ -31,7 +37,7 @@ std::unique_ptr AST; FindHeadersTest() { Inputs.MakeAction = [this] { - struct Hook : public PreprocessOnlyAction { + struct Hook : public SyntaxOnlyAction { public: Hook(PragmaIncludes *Out) : Out(Out) {} bool BeginSourceFileAction(clang::CompilerInstance &CI) override { @@ -153,5 +159,60 @@ physicalHeader("exporter.h"))); } +TEST_F(FindHeadersTest, TargetIsExpandedFromMacroInHeader) { + struct CustomVisitor : RecursiveASTVisitor { +const Decl *Out = nullptr; +bool VisitNamedDecl(const NamedDecl *ND) { + if (ND->getName() == "FLAG_foo" || ND->getName() == "Foo") { +EXPECT_TRUE(Out == nullptr); +Out = ND; + } + return true; +} + }; + + struct { +llvm::StringRef DeclareHeader; +llvm::StringRef MacroHeader; + } TestCases[] = {{ + R"cpp( +#include "macro.h" +DEFINE_CLASS(Foo) + )cpp", + R"cpp( +#define DEFINE_CLASS(name) class name {}; + )cpp"}, + { + R"cpp( +#include "macro.h" +DEFINE_Foo + )cpp", + R"cpp( +#define DEFINE_Foo class Foo {}; + )cpp"}, + { + R"cpp( +#include "macro.h" +DECLARE_FLAGS(foo); + )cpp", + R"cpp( +#define DECLARE_FLAGS(name) extern int FLAG_##name + )cpp"}}; + + for (const auto &T : TestCases) { +Inputs.Code = R"cpp(#include "declare.h")cpp"; +Inputs.ExtraFiles["declare.h"] = guard(T.DeclareHeader); +Inputs.ExtraFiles["macro.h"] = guard(T.MacroHeader); +buildAST(); + +CustomVisitor Visitor; +Visitor.TraverseDecl(AST->context().getTranslationUnitDecl()); + +llvm::SmallVector Headers = clang::include_cleaner::findHeaders( +Visitor.Out->getLocation(), AST->sourceManager(), nullptr); +EXPECT_THAT(Headers, UnorderedElementsAre(physicalHeader("declare.h"))); + } +} + } // namespace } // namespace clang::include_cleaner Index: clang-tools-extra/include-cleaner/lib/FindHeaders.cpp === --- clang-tools-extra/include-cleaner/lib/FindHeaders.cpp +++ clang-tools-extra/include-cleaner/lib/FindHeaders.cpp @@ -18,8 +18,7 @@ llvm::SmallVector Results; switch (Loc.kind()) { case SymbolLocation::Physical: { -// FIXME: Handle macro locations. -FileID FID = SM.getFileID(Loc.physical()); +FileID FID = SM.getFileID(SM.getExpansionLoc(Loc.physical())); const FileEntry *FE = SM.getFileEntryForID(FID); if (!PI) { return FE ? llvm::SmallVector{Header(FE)} ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D139716: [include-cleaner] Use expansion locations for macros.
VitaNuo updated this revision to Diff 483941. VitaNuo marked 4 inline comments as done. VitaNuo added a comment. Address review comments. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D139716/new/ https://reviews.llvm.org/D139716 Files: clang-tools-extra/include-cleaner/lib/FindHeaders.cpp clang-tools-extra/include-cleaner/unittests/FindHeadersTest.cpp Index: clang-tools-extra/include-cleaner/unittests/FindHeadersTest.cpp === --- clang-tools-extra/include-cleaner/unittests/FindHeadersTest.cpp +++ clang-tools-extra/include-cleaner/unittests/FindHeadersTest.cpp @@ -7,11 +7,17 @@ //===--===// #include "AnalysisInternal.h" +#include "clang-include-cleaner/Analysis.h" #include "clang-include-cleaner/Record.h" +#include "clang-include-cleaner/Types.h" +#include "clang/AST/RecursiveASTVisitor.h" #include "clang/Basic/FileEntry.h" #include "clang/Basic/FileManager.h" #include "clang/Frontend/FrontendActions.h" #include "clang/Testing/TestAST.h" +#include "llvm/ADT/ArrayRef.h" +#include "llvm/Support/raw_ostream.h" +#include "llvm/Testing/Support/Annotations.h" #include "gmock/gmock.h" #include "gtest/gtest.h" #include @@ -31,7 +37,7 @@ std::unique_ptr AST; FindHeadersTest() { Inputs.MakeAction = [this] { - struct Hook : public PreprocessOnlyAction { + struct Hook : public SyntaxOnlyAction { public: Hook(PragmaIncludes *Out) : Out(Out) {} bool BeginSourceFileAction(clang::CompilerInstance &CI) override { @@ -153,5 +159,60 @@ physicalHeader("exporter.h"))); } +TEST_F(FindHeadersTest, TargetIsExpandedFromMacroInHeader) { + struct CustomVisitor : RecursiveASTVisitor { +const Decl *Out = nullptr; +bool VisitNamedDecl(const NamedDecl *ND) { + if (ND->getName() == "FLAG_foo" || ND->getName() == "Foo") { +EXPECT_TRUE(Out == nullptr); +Out = ND; + } + return true; +} + }; + + struct { +llvm::StringRef MacroHeader; +llvm::StringRef DeclareHeader; + } TestCases[] = { + {/*MacroHeader=*/R"cpp( +#define DEFINE_CLASS(name) class name {}; + )cpp", + /*DeclareHeader=*/R"cpp( +#include "macro.h" +DEFINE_CLASS(Foo) + )cpp"}, + {/*MacroHeader=*/R"cpp( +#define DEFINE_Foo class Foo {}; + )cpp", + /*DeclareHeader=*/R"cpp( +#include "macro.h" +DEFINE_Foo + )cpp"}, + {/*MacroHeader=*/R"cpp( +#define DECLARE_FLAGS(name) extern int FLAG_##name + )cpp", + /*DeclareHeader=*/R"cpp( +#include "macro.h" +DECLARE_FLAGS(foo); + )cpp"}, + }; + + for (const auto &T : TestCases) { +Inputs.Code = R"cpp(#include "declare.h")cpp"; +Inputs.ExtraFiles["declare.h"] = guard(T.DeclareHeader); +Inputs.ExtraFiles["macro.h"] = guard(T.MacroHeader); +buildAST(); + +CustomVisitor Visitor; +Visitor.TraverseDecl(AST->context().getTranslationUnitDecl()); + +llvm::SmallVector Headers = clang::include_cleaner::findHeaders( +Visitor.Out->getLocation(), AST->sourceManager(), +/*PragmaIncludes=*/nullptr); +EXPECT_THAT(Headers, UnorderedElementsAre(physicalHeader("declare.h"))); + } +} + } // namespace } // namespace clang::include_cleaner Index: clang-tools-extra/include-cleaner/lib/FindHeaders.cpp === --- clang-tools-extra/include-cleaner/lib/FindHeaders.cpp +++ clang-tools-extra/include-cleaner/lib/FindHeaders.cpp @@ -18,8 +18,7 @@ llvm::SmallVector Results; switch (Loc.kind()) { case SymbolLocation::Physical: { -// FIXME: Handle macro locations. -FileID FID = SM.getFileID(Loc.physical()); +FileID FID = SM.getFileID(SM.getExpansionLoc(Loc.physical())); const FileEntry *FE = SM.getFileEntryForID(FID); if (!PI) { return FE ? llvm::SmallVector{Header(FE)} ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D139716: [include-cleaner] Use expansion locations for macros.
VitaNuo added a comment. Thanks! Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D139716/new/ https://reviews.llvm.org/D139716 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D139716: [include-cleaner] Use expansion locations for macros.
This revision was landed with ongoing or failed builds. This revision was automatically updated to reflect the committed changes. Closed by commit rG81c3739156fe: [include-cleaner] Use expansion locations for macros. (authored by VitaNuo). Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D139716/new/ https://reviews.llvm.org/D139716 Files: clang-tools-extra/include-cleaner/lib/FindHeaders.cpp clang-tools-extra/include-cleaner/unittests/FindHeadersTest.cpp Index: clang-tools-extra/include-cleaner/unittests/FindHeadersTest.cpp === --- clang-tools-extra/include-cleaner/unittests/FindHeadersTest.cpp +++ clang-tools-extra/include-cleaner/unittests/FindHeadersTest.cpp @@ -7,11 +7,17 @@ //===--===// #include "AnalysisInternal.h" +#include "clang-include-cleaner/Analysis.h" #include "clang-include-cleaner/Record.h" +#include "clang-include-cleaner/Types.h" +#include "clang/AST/RecursiveASTVisitor.h" #include "clang/Basic/FileEntry.h" #include "clang/Basic/FileManager.h" #include "clang/Frontend/FrontendActions.h" #include "clang/Testing/TestAST.h" +#include "llvm/ADT/ArrayRef.h" +#include "llvm/Support/raw_ostream.h" +#include "llvm/Testing/Support/Annotations.h" #include "gmock/gmock.h" #include "gtest/gtest.h" #include @@ -31,7 +37,7 @@ std::unique_ptr AST; FindHeadersTest() { Inputs.MakeAction = [this] { - struct Hook : public PreprocessOnlyAction { + struct Hook : public SyntaxOnlyAction { public: Hook(PragmaIncludes *Out) : Out(Out) {} bool BeginSourceFileAction(clang::CompilerInstance &CI) override { @@ -153,5 +159,60 @@ physicalHeader("exporter.h"))); } +TEST_F(FindHeadersTest, TargetIsExpandedFromMacroInHeader) { + struct CustomVisitor : RecursiveASTVisitor { +const Decl *Out = nullptr; +bool VisitNamedDecl(const NamedDecl *ND) { + if (ND->getName() == "FLAG_foo" || ND->getName() == "Foo") { +EXPECT_TRUE(Out == nullptr); +Out = ND; + } + return true; +} + }; + + struct { +llvm::StringRef MacroHeader; +llvm::StringRef DeclareHeader; + } TestCases[] = { + {/*MacroHeader=*/R"cpp( +#define DEFINE_CLASS(name) class name {}; + )cpp", + /*DeclareHeader=*/R"cpp( +#include "macro.h" +DEFINE_CLASS(Foo) + )cpp"}, + {/*MacroHeader=*/R"cpp( +#define DEFINE_Foo class Foo {}; + )cpp", + /*DeclareHeader=*/R"cpp( +#include "macro.h" +DEFINE_Foo + )cpp"}, + {/*MacroHeader=*/R"cpp( +#define DECLARE_FLAGS(name) extern int FLAG_##name + )cpp", + /*DeclareHeader=*/R"cpp( +#include "macro.h" +DECLARE_FLAGS(foo); + )cpp"}, + }; + + for (const auto &T : TestCases) { +Inputs.Code = R"cpp(#include "declare.h")cpp"; +Inputs.ExtraFiles["declare.h"] = guard(T.DeclareHeader); +Inputs.ExtraFiles["macro.h"] = guard(T.MacroHeader); +buildAST(); + +CustomVisitor Visitor; +Visitor.TraverseDecl(AST->context().getTranslationUnitDecl()); + +llvm::SmallVector Headers = clang::include_cleaner::findHeaders( +Visitor.Out->getLocation(), AST->sourceManager(), +/*PragmaIncludes=*/nullptr); +EXPECT_THAT(Headers, UnorderedElementsAre(physicalHeader("declare.h"))); + } +} + } // namespace } // namespace clang::include_cleaner Index: clang-tools-extra/include-cleaner/lib/FindHeaders.cpp === --- clang-tools-extra/include-cleaner/lib/FindHeaders.cpp +++ clang-tools-extra/include-cleaner/lib/FindHeaders.cpp @@ -18,8 +18,7 @@ llvm::SmallVector Results; switch (Loc.kind()) { case SymbolLocation::Physical: { -// FIXME: Handle macro locations. -FileID FID = SM.getFileID(Loc.physical()); +FileID FID = SM.getFileID(SM.getExpansionLoc(Loc.physical())); const FileEntry *FE = SM.getFileEntryForID(FID); if (!PI) { return FE ? llvm::SmallVector{Header(FE)} ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D141271: [include-cleaner] Filter template instantiations from AST roots.
VitaNuo created this revision. Herald added a subscriber: kadircet. Herald added a project: All. VitaNuo requested review of this revision. Herald added a project: clang-tools-extra. Herald added a subscriber: cfe-commits. Repository: rG LLVM Github Monorepo https://reviews.llvm.org/D141271 Files: clang-tools-extra/include-cleaner/lib/Record.cpp clang-tools-extra/include-cleaner/unittests/RecordTest.cpp Index: clang-tools-extra/include-cleaner/unittests/RecordTest.cpp === --- clang-tools-extra/include-cleaner/unittests/RecordTest.cpp +++ clang-tools-extra/include-cleaner/unittests/RecordTest.cpp @@ -101,6 +101,28 @@ EXPECT_THAT(Recorded.Roots, testing::ElementsAre(named("x"))); } +// Decl from template instantiation is filtered out from roots. +TEST_F(RecordASTTest, TemplateInstantiations) { + Inputs.ExtraFiles["dispatch.h"] = R"cpp( + struct A { +static constexpr int value = 1; + }; + template + int dispatch() { +return Getter::template get(); + } + )cpp"; + Inputs.Code = R"cpp( + #include "dispatch.h" + struct MyGetter { +template static int get() { return T::value; } + }; + int v = dispatch(); + )cpp"; + auto AST = build(); + EXPECT_THAT(Recorded.Roots, testing::Not(testing::ElementsAre(named("get"; +} + class RecordPPTest : public ::testing::Test { protected: TestInputs Inputs; Index: clang-tools-extra/include-cleaner/lib/Record.cpp === --- clang-tools-extra/include-cleaner/lib/Record.cpp +++ clang-tools-extra/include-cleaner/lib/Record.cpp @@ -12,6 +12,7 @@ #include "clang/AST/ASTContext.h" #include "clang/AST/DeclGroup.h" #include "clang/Basic/SourceManager.h" +#include "clang/Basic/Specifiers.h" #include "clang/Frontend/CompilerInstance.h" #include "clang/Lex/MacroInfo.h" #include "clang/Lex/PPCallbacks.h" @@ -352,6 +353,15 @@ return IWYUPublic.find(FE->getUniqueID()) != IWYUPublic.end(); } +namespace { +template +bool isImplicitTemplateSpecialization(const NamedDecl *D) { + if (const auto *TD = dyn_cast(D)) +return TD->getTemplateSpecializationKind() == TSK_ImplicitInstantiation; + return false; +} +} // namespace + std::unique_ptr RecordedAST::record() { class Recorder : public ASTConsumer { RecordedAST *Out; @@ -364,11 +374,21 @@ for (Decl *D : DG) { if (!SM.isWrittenInMainFile(SM.getExpansionLoc(D->getLocation( continue; -// FIXME: Filter out certain Obj-C and template-related decls. +if (const NamedDecl *ND = dyn_cast(D)) + if (isImplicitTemplateInstantiation(ND)) +continue; +// FIXME: Filter out certain Obj-C as well. Out->Roots.push_back(D); } return ASTConsumer::HandleTopLevelDecl(DG); } + + private: + bool isImplicitTemplateInstantiation(const NamedDecl *D) { +return isImplicitTemplateSpecialization(D) || + isImplicitTemplateSpecialization(D) || + isImplicitTemplateSpecialization(D); + } }; return std::make_unique(this); Index: clang-tools-extra/include-cleaner/unittests/RecordTest.cpp === --- clang-tools-extra/include-cleaner/unittests/RecordTest.cpp +++ clang-tools-extra/include-cleaner/unittests/RecordTest.cpp @@ -101,6 +101,28 @@ EXPECT_THAT(Recorded.Roots, testing::ElementsAre(named("x"))); } +// Decl from template instantiation is filtered out from roots. +TEST_F(RecordASTTest, TemplateInstantiations) { + Inputs.ExtraFiles["dispatch.h"] = R"cpp( + struct A { +static constexpr int value = 1; + }; + template + int dispatch() { +return Getter::template get(); + } + )cpp"; + Inputs.Code = R"cpp( + #include "dispatch.h" + struct MyGetter { +template static int get() { return T::value; } + }; + int v = dispatch(); + )cpp"; + auto AST = build(); + EXPECT_THAT(Recorded.Roots, testing::Not(testing::ElementsAre(named("get"; +} + class RecordPPTest : public ::testing::Test { protected: TestInputs Inputs; Index: clang-tools-extra/include-cleaner/lib/Record.cpp === --- clang-tools-extra/include-cleaner/lib/Record.cpp +++ clang-tools-extra/include-cleaner/lib/Record.cpp @@ -12,6 +12,7 @@ #include "clang/AST/ASTContext.h" #include "clang/AST/DeclGroup.h" #include "clang/Basic/SourceManager.h" +#include "clang/Basic/Specifiers.h" #include "clang/Frontend/CompilerInstance.h" #include "clang/Lex/MacroInfo.h" #include "clang/Lex/PPCallbacks.h" @@ -352,6 +353,15 @@ return IWYUPublic.find(FE->getUniqueID()) != IWYUPublic.end(); } +namespace { +template +bool isImplicitTemplateSpecialization(const NamedDecl *D) { + if (const auto *TD = dyn_cast(D)) +return TD->getTemplateSpecializationKind() == TSK_ImplicitInstantiation; + return false; +} +} // namespace + std::unique
[PATCH] D141271: [include-cleaner] Filter template instantiations from AST roots.
VitaNuo updated this revision to Diff 487385. VitaNuo added a comment. Address review comments. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D141271/new/ https://reviews.llvm.org/D141271 Files: clang-tools-extra/include-cleaner/lib/Record.cpp clang-tools-extra/include-cleaner/unittests/RecordTest.cpp Index: clang-tools-extra/include-cleaner/unittests/RecordTest.cpp === --- clang-tools-extra/include-cleaner/unittests/RecordTest.cpp +++ clang-tools-extra/include-cleaner/unittests/RecordTest.cpp @@ -101,6 +101,29 @@ EXPECT_THAT(Recorded.Roots, testing::ElementsAre(named("x"))); } +// Decl from template instantiation is filtered out from roots. +TEST_F(RecordASTTest, ImplicitTemplates) { + Inputs.ExtraFiles["dispatch.h"] = R"cpp( + struct A { +static constexpr int value = 1; + }; + template + int dispatch() { +return Getter::template get(); + } + )cpp"; + Inputs.Code = R"cpp( + #include "dispatch.h" + struct MyGetter { +template static int get() { return T::value; } + }; + int v = dispatch(); + )cpp"; + auto AST = build(); + EXPECT_THAT(Recorded.Roots, + testing::ElementsAre(named("MyGetter"), named("v"))); +} + class RecordPPTest : public ::testing::Test { protected: TestInputs Inputs; Index: clang-tools-extra/include-cleaner/lib/Record.cpp === --- clang-tools-extra/include-cleaner/lib/Record.cpp +++ clang-tools-extra/include-cleaner/lib/Record.cpp @@ -12,6 +12,7 @@ #include "clang/AST/ASTContext.h" #include "clang/AST/DeclGroup.h" #include "clang/Basic/SourceManager.h" +#include "clang/Basic/Specifiers.h" #include "clang/Frontend/CompilerInstance.h" #include "clang/Lex/MacroInfo.h" #include "clang/Lex/PPCallbacks.h" @@ -352,6 +353,14 @@ return IWYUPublic.find(FE->getUniqueID()) != IWYUPublic.end(); } +namespace { +template bool isImplicitTemplateSpecialization(const Decl *D) { + if (const auto *TD = dyn_cast(D)) +return TD->getTemplateSpecializationKind() == TSK_ImplicitInstantiation; + return false; +} +} // namespace + std::unique_ptr RecordedAST::record() { class Recorder : public ASTConsumer { RecordedAST *Out; @@ -364,7 +373,11 @@ for (Decl *D : DG) { if (!SM.isWrittenInMainFile(SM.getExpansionLoc(D->getLocation( continue; -// FIXME: Filter out certain Obj-C and template-related decls. +if (isImplicitTemplateSpecialization(D) || +isImplicitTemplateSpecialization(D) || +isImplicitTemplateSpecialization(D)) + continue; +// FIXME: Filter out certain Obj-C as well. Out->Roots.push_back(D); } return ASTConsumer::HandleTopLevelDecl(DG); Index: clang-tools-extra/include-cleaner/unittests/RecordTest.cpp === --- clang-tools-extra/include-cleaner/unittests/RecordTest.cpp +++ clang-tools-extra/include-cleaner/unittests/RecordTest.cpp @@ -101,6 +101,29 @@ EXPECT_THAT(Recorded.Roots, testing::ElementsAre(named("x"))); } +// Decl from template instantiation is filtered out from roots. +TEST_F(RecordASTTest, ImplicitTemplates) { + Inputs.ExtraFiles["dispatch.h"] = R"cpp( + struct A { +static constexpr int value = 1; + }; + template + int dispatch() { +return Getter::template get(); + } + )cpp"; + Inputs.Code = R"cpp( + #include "dispatch.h" + struct MyGetter { +template static int get() { return T::value; } + }; + int v = dispatch(); + )cpp"; + auto AST = build(); + EXPECT_THAT(Recorded.Roots, + testing::ElementsAre(named("MyGetter"), named("v"))); +} + class RecordPPTest : public ::testing::Test { protected: TestInputs Inputs; Index: clang-tools-extra/include-cleaner/lib/Record.cpp === --- clang-tools-extra/include-cleaner/lib/Record.cpp +++ clang-tools-extra/include-cleaner/lib/Record.cpp @@ -12,6 +12,7 @@ #include "clang/AST/ASTContext.h" #include "clang/AST/DeclGroup.h" #include "clang/Basic/SourceManager.h" +#include "clang/Basic/Specifiers.h" #include "clang/Frontend/CompilerInstance.h" #include "clang/Lex/MacroInfo.h" #include "clang/Lex/PPCallbacks.h" @@ -352,6 +353,14 @@ return IWYUPublic.find(FE->getUniqueID()) != IWYUPublic.end(); } +namespace { +template bool isImplicitTemplateSpecialization(const Decl *D) { + if (const auto *TD = dyn_cast(D)) +return TD->getTemplateSpecializationKind() == TSK_ImplicitInstantiation; + return false; +} +} // namespace + std::unique_ptr RecordedAST::record() { class Recorder : public ASTConsumer { RecordedAST *Out; @@ -364,7 +373,11 @@ for (Decl *D : DG) { if (!SM.isWrittenInMainFile(SM.getExpansionLoc(D->getLocation( continue; -// FIXME: Filter out certa
[PATCH] D141271: [include-cleaner] Filter template instantiations from AST roots.
VitaNuo added inline comments. Comment at: clang-tools-extra/include-cleaner/lib/Record.cpp:377 continue; -// FIXME: Filter out certain Obj-C and template-related decls. +if (const NamedDecl *ND = dyn_cast(D)) + if (isImplicitTemplateInstantiation(ND)) hokein wrote: > as we already do a dyn_cast inside the `isImplicitTemplateInstantiation`, we > can get rid of this NamedDecl cast by changing the > `isImplicitTemplateInstantiation` to accept a `Decl*`. Oh right, I didn't notice. Thanks. Comment at: clang-tools-extra/include-cleaner/lib/Record.cpp:387 + private: + bool isImplicitTemplateInstantiation(const NamedDecl *D) { +return isImplicitTemplateSpecialization(D) || hokein wrote: > we can move it to the above anonymous namespace as well (it is fine to have > two functions with same name because of C++ overloads). In this case, I'd > probably inline it in `HandleTopLevelDecl`. I am not sure I fully understand this comment, since I cannot both move this method to the anonymous namespace above and inline it at the same time. I have inlined it now, hopefully it looks fine to you. Comment at: clang-tools-extra/include-cleaner/unittests/RecordTest.cpp:105 +// Decl from template instantiation is filtered out from roots. +TEST_F(RecordASTTest, TemplateInstantiations) { + Inputs.ExtraFiles["dispatch.h"] = R"cpp( hokein wrote: > maybe DropImplicitTemplateTopDecl? Not sure, since the rest of tests above only use nouns, e.g., "Macros", "Inclusion", "Namespace", etc. I've called it "ImplicitTemplates" now, hopefully it's fine. Comment at: clang-tools-extra/include-cleaner/unittests/RecordTest.cpp:120 + }; + int v = dispatch(); + )cpp"; hokein wrote: > this example is a nice example to show the original bug (where we inserted > unexpected additional headers) in include-cleaner. > > The scope of this patch is to filter out the implicit template instantiation, > a simple testcase like below could work. (the current testcase is fine as > well, up to you) > > ``` > template > int func(T) { return 0; } > auto s = func(1); > ``` You're right, but IMHO I would rather keep the current example. The current example, although simplified, still highly resembles the pattern we've found it real code. The snippet above does not anymore. Since it's a bug fix for a real pattern, I'd rather keep it as is. Let me know if you disagree. Comment at: clang-tools-extra/include-cleaner/unittests/RecordTest.cpp:123 + auto AST = build(); + EXPECT_THAT(Recorded.Roots, testing::Not(testing::ElementsAre(named("get"; +} hokein wrote: > nit: instead doing a negative verification, I'd check the captured root decl, > which is (a `MyGetter` CXXRecordDecl, and a `v` var decl), and with a comment > saying the implicitly-instantiated method decl is filtered out. Sure, I have no preference. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D141271/new/ https://reviews.llvm.org/D141271 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D141271: [include-cleaner] Filter template instantiations from AST roots.
This revision was landed with ongoing or failed builds. This revision was automatically updated to reflect the committed changes. Closed by commit rGde81dc8fdf27: [include-cleaner] Filter template instantiations from AST roots. (authored by VitaNuo). Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D141271/new/ https://reviews.llvm.org/D141271 Files: clang-tools-extra/include-cleaner/lib/Record.cpp clang-tools-extra/include-cleaner/unittests/RecordTest.cpp Index: clang-tools-extra/include-cleaner/unittests/RecordTest.cpp === --- clang-tools-extra/include-cleaner/unittests/RecordTest.cpp +++ clang-tools-extra/include-cleaner/unittests/RecordTest.cpp @@ -101,6 +101,29 @@ EXPECT_THAT(Recorded.Roots, testing::ElementsAre(named("x"))); } +// Decl from template instantiation is filtered out from roots. +TEST_F(RecordASTTest, ImplicitTemplates) { + Inputs.ExtraFiles["dispatch.h"] = R"cpp( + struct A { +static constexpr int value = 1; + }; + template + int dispatch() { +return Getter::template get(); + } + )cpp"; + Inputs.Code = R"cpp( + #include "dispatch.h" + struct MyGetter { +template static int get() { return T::value; } + }; + int v = dispatch(); + )cpp"; + auto AST = build(); + EXPECT_THAT(Recorded.Roots, + testing::ElementsAre(named("MyGetter"), named("v"))); +} + class RecordPPTest : public ::testing::Test { protected: TestInputs Inputs; Index: clang-tools-extra/include-cleaner/lib/Record.cpp === --- clang-tools-extra/include-cleaner/lib/Record.cpp +++ clang-tools-extra/include-cleaner/lib/Record.cpp @@ -12,6 +12,7 @@ #include "clang/AST/ASTContext.h" #include "clang/AST/DeclGroup.h" #include "clang/Basic/SourceManager.h" +#include "clang/Basic/Specifiers.h" #include "clang/Frontend/CompilerInstance.h" #include "clang/Lex/MacroInfo.h" #include "clang/Lex/PPCallbacks.h" @@ -352,6 +353,14 @@ return IWYUPublic.find(FE->getUniqueID()) != IWYUPublic.end(); } +namespace { +template bool isImplicitTemplateSpecialization(const Decl *D) { + if (const auto *TD = dyn_cast(D)) +return TD->getTemplateSpecializationKind() == TSK_ImplicitInstantiation; + return false; +} +} // namespace + std::unique_ptr RecordedAST::record() { class Recorder : public ASTConsumer { RecordedAST *Out; @@ -364,7 +373,11 @@ for (Decl *D : DG) { if (!SM.isWrittenInMainFile(SM.getExpansionLoc(D->getLocation( continue; -// FIXME: Filter out certain Obj-C and template-related decls. +if (isImplicitTemplateSpecialization(D) || +isImplicitTemplateSpecialization(D) || +isImplicitTemplateSpecialization(D)) + continue; +// FIXME: Filter out certain Obj-C as well. Out->Roots.push_back(D); } return ASTConsumer::HandleTopLevelDecl(DG); Index: clang-tools-extra/include-cleaner/unittests/RecordTest.cpp === --- clang-tools-extra/include-cleaner/unittests/RecordTest.cpp +++ clang-tools-extra/include-cleaner/unittests/RecordTest.cpp @@ -101,6 +101,29 @@ EXPECT_THAT(Recorded.Roots, testing::ElementsAre(named("x"))); } +// Decl from template instantiation is filtered out from roots. +TEST_F(RecordASTTest, ImplicitTemplates) { + Inputs.ExtraFiles["dispatch.h"] = R"cpp( + struct A { +static constexpr int value = 1; + }; + template + int dispatch() { +return Getter::template get(); + } + )cpp"; + Inputs.Code = R"cpp( + #include "dispatch.h" + struct MyGetter { +template static int get() { return T::value; } + }; + int v = dispatch(); + )cpp"; + auto AST = build(); + EXPECT_THAT(Recorded.Roots, + testing::ElementsAre(named("MyGetter"), named("v"))); +} + class RecordPPTest : public ::testing::Test { protected: TestInputs Inputs; Index: clang-tools-extra/include-cleaner/lib/Record.cpp === --- clang-tools-extra/include-cleaner/lib/Record.cpp +++ clang-tools-extra/include-cleaner/lib/Record.cpp @@ -12,6 +12,7 @@ #include "clang/AST/ASTContext.h" #include "clang/AST/DeclGroup.h" #include "clang/Basic/SourceManager.h" +#include "clang/Basic/Specifiers.h" #include "clang/Frontend/CompilerInstance.h" #include "clang/Lex/MacroInfo.h" #include "clang/Lex/PPCallbacks.h" @@ -352,6 +353,14 @@ return IWYUPublic.find(FE->getUniqueID()) != IWYUPublic.end(); } +namespace { +template bool isImplicitTemplateSpecialization(const Decl *D) { + if (const auto *TD = dyn_cast(D)) +return TD->getTemplateSpecializationKind() == TSK_ImplicitInstantiation; + return false; +} +} // namespace + std::unique_ptr RecordedAST::record() { class Recorder : public ASTConsumer { RecordedAST *Out; @@ -364,7 +373,11 @@
[PATCH] D141477: [include-mapping] Fix the instructions for running stdlib recognizer. Mention python command explicitly. Remove angle brackets.
VitaNuo created this revision. Herald added a project: All. VitaNuo requested review of this revision. Herald added a project: clang. Herald added a subscriber: cfe-commits. Repository: rG LLVM Github Monorepo https://reviews.llvm.org/D141477 Files: clang/tools/include-mapping/gen_std.py Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -24,13 +24,13 @@ https://www.crummy.com/software/BeautifulSoup/bs4/doc/#installing-beautiful-soup 2. Download cppreference offline HTML files (e.g. html_book_20181028.zip) at https://en.cppreference.com/w/Cppreference:Archives - 3. Unzip the zip file from step 2 to directory , you should - get a "reference" directory in + 3. Unzip the zip file from step 2 (e.g., to a "cppreference" directory). You should + get a "cppreference/reference" directory. 4. Run the command: // Generate C++ symbols - gen_std.py -cppreference -language=cpp > StdSymbolMap.inc + python3 gen_std.py -cppreference cppreference/reference -language=cpp > StdSymbolMap.inc // Generate C symbols - gen_std.py -cppreference -language=c > CSymbolMap.inc + python3 gen_std.py -cppreference cppreference/reference -language=c > CSymbolMap.inc """ Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -24,13 +24,13 @@ https://www.crummy.com/software/BeautifulSoup/bs4/doc/#installing-beautiful-soup 2. Download cppreference offline HTML files (e.g. html_book_20181028.zip) at https://en.cppreference.com/w/Cppreference:Archives - 3. Unzip the zip file from step 2 to directory , you should - get a "reference" directory in + 3. Unzip the zip file from step 2 (e.g., to a "cppreference" directory). You should + get a "cppreference/reference" directory. 4. Run the command: // Generate C++ symbols - gen_std.py -cppreference -language=cpp > StdSymbolMap.inc + python3 gen_std.py -cppreference cppreference/reference -language=cpp > StdSymbolMap.inc // Generate C symbols - gen_std.py -cppreference -language=c > CSymbolMap.inc + python3 gen_std.py -cppreference cppreference/reference -language=c > CSymbolMap.inc """ ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D141477: [include-mapping] Fix the instructions for running stdlib recognizer. Mention python command explicitly. Remove angle brackets.
This revision was landed with ongoing or failed builds. This revision was automatically updated to reflect the committed changes. Closed by commit rG83c5040f6f32: [include-mapping] Fix the instructions for running stdlib recognizer. Mention… (authored by VitaNuo). Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D141477/new/ https://reviews.llvm.org/D141477 Files: clang/tools/include-mapping/gen_std.py Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -24,13 +24,13 @@ https://www.crummy.com/software/BeautifulSoup/bs4/doc/#installing-beautiful-soup 2. Download cppreference offline HTML files (e.g. html_book_20181028.zip) at https://en.cppreference.com/w/Cppreference:Archives - 3. Unzip the zip file from step 2 to directory , you should - get a "reference" directory in + 3. Unzip the zip file from step 2 (e.g., to a "cppreference" directory). You should + get a "cppreference/reference" directory. 4. Run the command: // Generate C++ symbols - gen_std.py -cppreference -language=cpp > StdSymbolMap.inc + python3 gen_std.py -cppreference cppreference/reference -language=cpp > StdSymbolMap.inc // Generate C symbols - gen_std.py -cppreference -language=c > CSymbolMap.inc + python3 gen_std.py -cppreference cppreference/reference -language=c > CSymbolMap.inc """ Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -24,13 +24,13 @@ https://www.crummy.com/software/BeautifulSoup/bs4/doc/#installing-beautiful-soup 2. Download cppreference offline HTML files (e.g. html_book_20181028.zip) at https://en.cppreference.com/w/Cppreference:Archives - 3. Unzip the zip file from step 2 to directory , you should - get a "reference" directory in + 3. Unzip the zip file from step 2 (e.g., to a "cppreference" directory). You should + get a "cppreference/reference" directory. 4. Run the command: // Generate C++ symbols - gen_std.py -cppreference -language=cpp > StdSymbolMap.inc + python3 gen_std.py -cppreference cppreference/reference -language=cpp > StdSymbolMap.inc // Generate C symbols - gen_std.py -cppreference -language=c > CSymbolMap.inc + python3 gen_std.py -cppreference cppreference/reference -language=c > CSymbolMap.inc """ ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D141509: [include-mapping] Fix parsing of html_book_20190607.zip (https://en.cppreference.com/w/File:html_book_20190607.zip). Skip entries that have been added to the index (C++20 symbols), bu
VitaNuo created this revision. VitaNuo added a reviewer: hokein. Herald added a subscriber: arphaman. Herald added a project: All. VitaNuo requested review of this revision. Herald added a project: clang. Herald added a subscriber: cfe-commits. Repository: rG LLVM Github Monorepo https://reviews.llvm.org/D141509 Files: clang/tools/include-mapping/cppreference_parser.py Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -114,8 +114,17 @@ def _ReadSymbolPage(path, name): - with open(path) as f: -return _ParseSymbolPage(f.read(), name) + try: +f = open(path) + except FileNotFoundError as e: +# Some entries exist in the index, but their corresponding pages have not +# been written yet (i.e., they are "red links"). When trying to open such +# a page, it redirects to a generic error page with "redlink=1" in the URL. +if "redlink" in str(e): + return [] +raise + with f: +return _ParseSymbolPage(f.read(), name) def _GetSymbols(pool, root_dir, index_page_name, namespace, variants_to_accept): Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -114,8 +114,17 @@ def _ReadSymbolPage(path, name): - with open(path) as f: -return _ParseSymbolPage(f.read(), name) + try: +f = open(path) + except FileNotFoundError as e: +# Some entries exist in the index, but their corresponding pages have not +# been written yet (i.e., they are "red links"). When trying to open such +# a page, it redirects to a generic error page with "redlink=1" in the URL. +if "redlink" in str(e): + return [] +raise + with f: +return _ParseSymbolPage(f.read(), name) def _GetSymbols(pool, root_dir, index_page_name, namespace, variants_to_accept): ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo added inline comments. Comment at: clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc:100 SYMBOL(atoll, std::, ) +SYMBOL(atomic, std::, ) +SYMBOL(atomic, std::, ) VitaNuo wrote: > kadircet wrote: > > hokein wrote: > > > Conceptually, this (and other `atomic_*` symbols) doesn't feel correct: > > > - `` provides the generic template `template struct > > > atomic;` > > > - `` provides partial template specializations for > > > `std::shared_ptr` and `std::weak_ptr` > > > > > > They are variant symbols (ideally, they should be treat as the > > > `std::move()`). The issue here is that unlike `std::move` which has two > > > individual entries in the index page, we only have one entry for > > > `std::atomic` (extending the cppreference_parser.py script to perfectly > > > distinguish these two cases in the > > > [page](https://en.cppreference.com/w/cpp/atomic/atomic) seems > > > non-trivial). Some options: > > > > > > 1) treat them as multiple-header symbols (like what this patch does now) > > > 2) special-case these symbols like `std::move()` > > > 3) always prefer the header providing generic templates > > > > > > @kadircet, what do you think? > > right, i believe this patch is definitely adding keeping multiple headers > > for a symbol around, but mixing it with keeping headers for variants (e.g. > > overloads provided by different headers, or specializations as mentioned > > here). > > > > we definitely need some more changes in parser to make sure we're only > > preserving alternatives for **the same symbol** and not for any other > > variants (overloads, specializations). IIRC there are currently 2 places > > these variants could come into play: > > - first is the symbol index page itself, symbols that have ` (foo)` next to > > them have variants and should still be ignored (e.g. std::remove variant of > > cstdio shouldn't be preserved in the scope of this patch) > > - second is inside the detail pages for symbols, e.g. > > [std::atomic](http://go/std::atomic), we can have headers for different > > declarations, they're clearly different variants so we shouldn't add such > > symbols into the mapping `_ParseSymbolPage` already does some detection of > > declarations in between headers. > > > > in the scope of this patch, we should keep ignoring both. > I suggest to special-case the overloads for now, just not to solve all the > problems in the same patch. The first group are already ignored. We need to take a lot at how to ignore the second one. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo updated this revision to Diff 491796. VitaNuo added a comment. Add early return for found symbols. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 Files: clang/include/clang/Tooling/Inclusions/CSymbolMap.inc clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp clang/tools/include-mapping/cppreference_parser.py clang/tools/include-mapping/gen_std.py Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -14,10 +14,7 @@ The generated files are located in clang/include/Tooling/Inclusions. Caveats and FIXMEs: - - only symbols directly in "std" namespace are added, we should also add std's -subnamespace symbols (e.g. chrono). - - symbols with multiple variants or defined in multiple headers aren't added, -e.g. std::move, std::swap + - symbols with multiple variants aren't added, e.g., std::move Usage: 1. Install BeautifulSoup dependency, see instruction: @@ -41,6 +38,7 @@ import datetime import os import sys +import re CODE_PREFIX = """\ //===-- gen_std.py generated file ---*- C++ -*-===// @@ -109,18 +107,16 @@ cppreference_modified_date = datetime.datetime.fromtimestamp( os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') print(CODE_PREFIX % (args.symbols.upper(), cppreference_modified_date)) + # FIXME: take care of overloads for symbol in symbols: -if len(symbol.headers) == 1: - # SYMBOL(unqualified_name, namespace, header) - print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, -symbol.headers[0])) -elif len(symbol.headers) == 0: +if re.match("^remove$|^swap$", symbol.name) and symbol.namespace == "std::": + continue +if len(symbol.headers) == 0: sys.stderr.write("No header found for symbol %s\n" % symbol.name) else: - # FIXME: support symbols with multiple headers (e.g. std::move). - sys.stderr.write("Ambiguous header for symbol %s: %s\n" % ( - symbol.name, ', '.join(symbol.headers))) - - + for header in symbol.headers: +# SYMBOL(unqualified_name, namespace, header) +print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, + header)) if __name__ == '__main__': main() Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -47,8 +47,7 @@ Returns a list of headers. """ - headers = set() - all_headers = set() + all_headers = [] soup = BeautifulSoup(symbol_page_html, "html.parser") # Rows in table are like: @@ -67,8 +66,11 @@ found_symbols = row.find('td').stripped_strings if not symbol_name in found_symbols: continue -headers.update(current_headers) - elif _HasClass(row, 't-dsc-header'): +# If the symbol was never named, consider the first of named headers. +if all_headers: + all_headers = [all_headers[0]] +return current_headers or all_headers + if _HasClass(row, 't-dsc-header'): # If we saw a decl since the last header, this is a new block of headers # for a new block of decls. if was_decl: @@ -80,10 +82,8 @@ # The interesting header content (e.g. ) is wrapped in . for header_code in row.find_all("code"): current_headers.append(header_code.text) - all_headers.add(header_code.text) - # If the symbol was never named, consider all named headers. - return headers or all_headers - + all_headers.append(header_code.text) + return all_headers def _ParseIndexPage(index_page_html): """Parse index page. @@ -118,7 +118,7 @@ return _ParseSymbolPage(f.read(), name) -def _GetSymbols(pool, root_dir, index_page_name, namespace, variants_to_accept): +def _GetSymbols(pool, root_dir, index_page_name, namespace): """Get all symbols listed in the index page. All symbols should be in the given namespace. @@ -135,11 +135,7 @@ # Read each symbol page in parallel. results = [] # (symbol_name, promise of [header...]) for symbol_name, symbol_page_path, variant in _ParseIndexPage(f.read()): - # Variant symbols (e.g. the std::locale version of isalpha) add ambiguity. - # FIXME: use these as a fallback rather than ignoring entirely. - variants_for_symbol = variants_to_accept.get( - (namespace or "") + symbol_name, ()) - if variant and variant not in variants_for_symbol: + if variant: continue path = os.path.join(root_dir, symbol_page_path)
[PATCH] D142555: Refactor symbol page parsing.
VitaNuo created this revision. Herald added a project: All. VitaNuo requested review of this revision. Herald added a project: clang. Herald added a subscriber: cfe-commits. Repository: rG LLVM Github Monorepo https://reviews.llvm.org/D142555 Files: clang/tools/include-mapping/cppreference_parser.py Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -47,7 +47,7 @@ Returns a list of headers. """ - headers = set() + symbol_headers = set() all_headers = set() soup = BeautifulSoup(symbol_page_html, "html.parser") @@ -58,31 +58,39 @@ # Defined in header .t-dsc-header # decl2.t-dcl for table in soup.select('table.t-dcl-begin, table.t-dsc-begin'): -current_headers = [] -was_decl = False -for row in table.select('tr'): - if _HasClass(row, 't-dcl', 't-dsc'): -was_decl = True -# Symbols are in the first cell. -found_symbols = row.find('td').stripped_strings -if not symbol_name in found_symbols: - continue -headers.update(current_headers) - elif _HasClass(row, 't-dsc-header'): -# If we saw a decl since the last header, this is a new block of headers -# for a new block of decls. -if was_decl: - current_headers = [] -was_decl = False +rows = table.select('tr') +i = 0 +while i < len(rows): + start = i + current_headers = set() + while i < len(rows) and _HasClass(rows[i], 't-dsc-header'): +row = rows[i] # There are also .t-dsc-header for "defined in namespace". if not "Defined in header " in row.text: + i = i + 1 continue # The interesting header content (e.g. ) is wrapped in . for header_code in row.find_all("code"): - current_headers.append(header_code.text) all_headers.add(header_code.text) - # If the symbol was never named, consider all named headers. - return headers or all_headers + current_headers.add(header_code.text) +i = i + 1 + # some tables have header rows, skip them + while i < len(rows) and _HasClass(rows[i], 't-dsc-hitem'): +i = i + 1 + while i < len(rows) and (_HasClass(rows[i], 't-dcl', 't-dsc') or not rows[i].has_attr("class")): +row = rows[i] +# Symbols are in the first cell. +found_symbols = row.find('td').stripped_strings +if symbol_name in found_symbols: + for header in current_headers: +symbol_headers.add(header) +i = i + 1 + # no headers or symbols in this block + if i == start: +i = i + 1 + + # If the symbol was never named, consider all named headers. + return symbol_headers or all_headers def _ParseIndexPage(index_page_html): Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -47,7 +47,7 @@ Returns a list of headers. """ - headers = set() + symbol_headers = set() all_headers = set() soup = BeautifulSoup(symbol_page_html, "html.parser") @@ -58,31 +58,39 @@ # Defined in header .t-dsc-header # decl2.t-dcl for table in soup.select('table.t-dcl-begin, table.t-dsc-begin'): -current_headers = [] -was_decl = False -for row in table.select('tr'): - if _HasClass(row, 't-dcl', 't-dsc'): -was_decl = True -# Symbols are in the first cell. -found_symbols = row.find('td').stripped_strings -if not symbol_name in found_symbols: - continue -headers.update(current_headers) - elif _HasClass(row, 't-dsc-header'): -# If we saw a decl since the last header, this is a new block of headers -# for a new block of decls. -if was_decl: - current_headers = [] -was_decl = False +rows = table.select('tr') +i = 0 +while i < len(rows): + start = i + current_headers = set() + while i < len(rows) and _HasClass(rows[i], 't-dsc-header'): +row = rows[i] # There are also .t-dsc-header for "defined in namespace". if not "Defined in header " in row.text: + i = i + 1 continue # The interesting header content (e.g. ) is wrapped in . for header_code in row.find_all("code"): - current_headers.append(header_code.text) all_headers.add(header_code.text) - # If the symbol was never named, consider all named headers. - return headers or all_headers + current_headers.add(header_code.text) +i = i + 1 + # some tables have header rows, skip them +
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo updated this revision to Diff 492130. VitaNuo added a comment. Refactor the symbol parsing loop. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 Files: clang/include/clang/Tooling/Inclusions/CSymbolMap.inc clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp clang/tools/include-mapping/cppreference_parser.py clang/tools/include-mapping/gen_std.py Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -14,10 +14,7 @@ The generated files are located in clang/include/Tooling/Inclusions. Caveats and FIXMEs: - - only symbols directly in "std" namespace are added, we should also add std's -subnamespace symbols (e.g. chrono). - - symbols with multiple variants or defined in multiple headers aren't added, -e.g. std::move, std::swap + - symbols with multiple variants aren't added, e.g., std::move Usage: 1. Install BeautifulSoup dependency, see instruction: @@ -41,6 +38,7 @@ import datetime import os import sys +import re CODE_PREFIX = """\ //===-- gen_std.py generated file ---*- C++ -*-===// @@ -109,18 +107,16 @@ cppreference_modified_date = datetime.datetime.fromtimestamp( os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') print(CODE_PREFIX % (args.symbols.upper(), cppreference_modified_date)) + # FIXME: take care of overloads for symbol in symbols: -if len(symbol.headers) == 1: - # SYMBOL(unqualified_name, namespace, header) - print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, -symbol.headers[0])) -elif len(symbol.headers) == 0: +if re.match("^remove$|^swap$", symbol.name) and symbol.namespace == "std::": + continue +if len(symbol.headers) == 0: sys.stderr.write("No header found for symbol %s\n" % symbol.name) else: - # FIXME: support symbols with multiple headers (e.g. std::move). - sys.stderr.write("Ambiguous header for symbol %s: %s\n" % ( - symbol.name, ', '.join(symbol.headers))) - - + for header in symbol.headers: +# SYMBOL(unqualified_name, namespace, header) +print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, + header)) if __name__ == '__main__': main() Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -47,7 +47,7 @@ Returns a list of headers. """ - headers = set() + symbol_headers = set() all_headers = set() soup = BeautifulSoup(symbol_page_html, "html.parser") @@ -58,32 +58,39 @@ # Defined in header .t-dsc-header # decl2.t-dcl for table in soup.select('table.t-dcl-begin, table.t-dsc-begin'): -current_headers = [] -was_decl = False -for row in table.select('tr'): - if _HasClass(row, 't-dcl', 't-dsc'): -was_decl = True -# Symbols are in the first cell. -found_symbols = row.find('td').stripped_strings -if not symbol_name in found_symbols: - continue -headers.update(current_headers) - elif _HasClass(row, 't-dsc-header'): -# If we saw a decl since the last header, this is a new block of headers -# for a new block of decls. -if was_decl: - current_headers = [] -was_decl = False +rows = table.select('tr') +i = 0 +while i < len(rows): + start = i + current_headers = set() + while i < len(rows) and _HasClass(rows[i], 't-dsc-header'): +row = rows[i] # There are also .t-dsc-header for "defined in namespace". if not "Defined in header " in row.text: + i = i + 1 continue # The interesting header content (e.g. ) is wrapped in . for header_code in row.find_all("code"): - current_headers.append(header_code.text) all_headers.add(header_code.text) - # If the symbol was never named, consider all named headers. - return headers or all_headers - + current_headers.add(header_code.text) +i = i + 1 + # some tables have header rows, skip them + while i < len(rows) and _HasClass(rows[i], 't-dsc-hitem'): +i = i + 1 + while i < len(rows) and (_HasClass(rows[i], 't-dcl', 't-dsc') or not rows[i].has_attr("class")): +row = rows[i] +# Symbols are in the first cell. +found_symbols = row.find('td').stripped_strings +if symbol_name in found_symbols: + for header in current_headers: +symbol_headers.add(header) +
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo updated this revision to Diff 492160. VitaNuo added a comment. Refine C parsing. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 Files: clang/include/clang/Tooling/Inclusions/CSymbolMap.inc clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp clang/tools/include-mapping/cppreference_parser.py clang/tools/include-mapping/gen_std.py Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -14,10 +14,7 @@ The generated files are located in clang/include/Tooling/Inclusions. Caveats and FIXMEs: - - only symbols directly in "std" namespace are added, we should also add std's -subnamespace symbols (e.g. chrono). - - symbols with multiple variants or defined in multiple headers aren't added, -e.g. std::move, std::swap + - symbols with multiple variants aren't added, e.g., std::move Usage: 1. Install BeautifulSoup dependency, see instruction: @@ -41,6 +38,7 @@ import datetime import os import sys +import re CODE_PREFIX = """\ //===-- gen_std.py generated file ---*- C++ -*-===// @@ -109,18 +107,17 @@ cppreference_modified_date = datetime.datetime.fromtimestamp( os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') print(CODE_PREFIX % (args.symbols.upper(), cppreference_modified_date)) + # FIXME: take care of overloads for symbol in symbols: -if len(symbol.headers) == 1: - # SYMBOL(unqualified_name, namespace, header) - print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, -symbol.headers[0])) -elif len(symbol.headers) == 0: +if re.match("^remove$|^swap$", symbol.name) and symbol.namespace == "std::": + continue +if len(symbol.headers) == 0: sys.stderr.write("No header found for symbol %s\n" % symbol.name) else: - # FIXME: support symbols with multiple headers (e.g. std::move). - sys.stderr.write("Ambiguous header for symbol %s: %s\n" % ( - symbol.name, ', '.join(symbol.headers))) - - + symbol.headers = sorted(symbol.headers) + for header in symbol.headers: +# SYMBOL(unqualified_name, namespace, header) +print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, + header)) if __name__ == '__main__': main() Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -47,7 +47,7 @@ Returns a list of headers. """ - headers = set() + symbol_headers = set() all_headers = set() soup = BeautifulSoup(symbol_page_html, "html.parser") @@ -58,32 +58,45 @@ # Defined in header .t-dsc-header # decl2.t-dcl for table in soup.select('table.t-dcl-begin, table.t-dsc-begin'): -current_headers = [] -was_decl = False -for row in table.select('tr'): - if _HasClass(row, 't-dcl', 't-dsc'): -was_decl = True -# Symbols are in the first cell. -found_symbols = row.find('td').stripped_strings -if not symbol_name in found_symbols: - continue -headers.update(current_headers) - elif _HasClass(row, 't-dsc-header'): -# If we saw a decl since the last header, this is a new block of headers -# for a new block of decls. -if was_decl: - current_headers = [] -was_decl = False +rows = table.select('tr') +i = 0 +while i < len(rows): + start = i + current_headers = set() + while i < len(rows) and _HasClass(rows[i], 't-dsc-header'): +row = rows[i] # There are also .t-dsc-header for "defined in namespace". if not "Defined in header " in row.text: + i = i + 1 continue # The interesting header content (e.g. ) is wrapped in . for header_code in row.find_all("code"): - current_headers.append(header_code.text) all_headers.add(header_code.text) + current_headers.add(header_code.text) +i = i + 1 + # some tables have header rows, skip them + while i < len(rows) and _HasClass(rows[i], 't-dsc-hitem'): +i = i + 1 + while i < len(rows) and (_HasClass(rows[i], 't-dcl', 't-dsc') or not rows[i].has_attr("class")): +row = rows[i] +# Symbols are in the first cell. +raw_symbols = row.find('td').stripped_strings +found_symbols = [] +# Finding all text in is not enough, some strings contain more than one symbol (e.g., "div_t div" is parsed as single symbol). +for raw_symbol in r
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo updated this revision to Diff 492352. VitaNuo added a comment. Sort the C symbol map. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 Files: clang/include/clang/Tooling/Inclusions/CSymbolMap.inc clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp clang/tools/include-mapping/cppreference_parser.py clang/tools/include-mapping/gen_std.py Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -14,10 +14,7 @@ The generated files are located in clang/include/Tooling/Inclusions. Caveats and FIXMEs: - - only symbols directly in "std" namespace are added, we should also add std's -subnamespace symbols (e.g. chrono). - - symbols with multiple variants or defined in multiple headers aren't added, -e.g. std::move, std::swap + - symbols with multiple variants aren't added, e.g., std::move Usage: 1. Install BeautifulSoup dependency, see instruction: @@ -41,6 +38,7 @@ import datetime import os import sys +import re CODE_PREFIX = """\ //===-- gen_std.py generated file ---*- C++ -*-===// @@ -109,18 +107,17 @@ cppreference_modified_date = datetime.datetime.fromtimestamp( os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') print(CODE_PREFIX % (args.symbols.upper(), cppreference_modified_date)) + # FIXME: take care of overloads for symbol in symbols: -if len(symbol.headers) == 1: - # SYMBOL(unqualified_name, namespace, header) - print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, -symbol.headers[0])) -elif len(symbol.headers) == 0: +if re.match("^remove$|^swap$", symbol.name) and symbol.namespace == "std::": + continue +if len(symbol.headers) == 0: sys.stderr.write("No header found for symbol %s\n" % symbol.name) else: - # FIXME: support symbols with multiple headers (e.g. std::move). - sys.stderr.write("Ambiguous header for symbol %s: %s\n" % ( - symbol.name, ', '.join(symbol.headers))) - - + symbol.headers = sorted(symbol.headers) + for header in symbol.headers: +# SYMBOL(unqualified_name, namespace, header) +print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, + header)) if __name__ == '__main__': main() Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -47,7 +47,7 @@ Returns a list of headers. """ - headers = set() + symbol_headers = set() all_headers = set() soup = BeautifulSoup(symbol_page_html, "html.parser") @@ -58,32 +58,45 @@ # Defined in header .t-dsc-header # decl2.t-dcl for table in soup.select('table.t-dcl-begin, table.t-dsc-begin'): -current_headers = [] -was_decl = False -for row in table.select('tr'): - if _HasClass(row, 't-dcl', 't-dsc'): -was_decl = True -# Symbols are in the first cell. -found_symbols = row.find('td').stripped_strings -if not symbol_name in found_symbols: - continue -headers.update(current_headers) - elif _HasClass(row, 't-dsc-header'): -# If we saw a decl since the last header, this is a new block of headers -# for a new block of decls. -if was_decl: - current_headers = [] -was_decl = False +rows = table.select('tr') +i = 0 +while i < len(rows): + start = i + current_headers = set() + while i < len(rows) and _HasClass(rows[i], 't-dsc-header'): +row = rows[i] # There are also .t-dsc-header for "defined in namespace". if not "Defined in header " in row.text: + i = i + 1 continue # The interesting header content (e.g. ) is wrapped in . for header_code in row.find_all("code"): - current_headers.append(header_code.text) all_headers.add(header_code.text) + current_headers.add(header_code.text) +i = i + 1 + # some tables have header rows, skip them + while i < len(rows) and _HasClass(rows[i], 't-dsc-hitem'): +i = i + 1 + while i < len(rows) and (_HasClass(rows[i], 't-dcl', 't-dsc') or not rows[i].has_attr("class")): +row = rows[i] +# Symbols are in the first cell. +raw_symbols = row.find('td').stripped_strings +found_symbols = [] +# Finding all text in is not enough, some strings contain more than one symbol (e.g., "div_t div" is parsed as single symbol). +for raw_symbol
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo updated this revision to Diff 492366. VitaNuo added a comment. Fix "atomic" and "div" overloads programmatically. Add a FIXME for future proper handling of overloads. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 Files: clang/include/clang/Tooling/Inclusions/CSymbolMap.inc clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp clang/tools/include-mapping/cppreference_parser.py clang/tools/include-mapping/gen_std.py Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -14,10 +14,7 @@ The generated files are located in clang/include/Tooling/Inclusions. Caveats and FIXMEs: - - only symbols directly in "std" namespace are added, we should also add std's -subnamespace symbols (e.g. chrono). - - symbols with multiple variants or defined in multiple headers aren't added, -e.g. std::move, std::swap + - symbols with multiple variants aren't added, e.g., std::move Usage: 1. Install BeautifulSoup dependency, see instruction: @@ -41,6 +38,7 @@ import datetime import os import sys +import re CODE_PREFIX = """\ //===-- gen_std.py generated file ---*- C++ -*-===// @@ -109,18 +107,17 @@ cppreference_modified_date = datetime.datetime.fromtimestamp( os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') print(CODE_PREFIX % (args.symbols.upper(), cppreference_modified_date)) + # FIXME: take care of overloads for symbol in symbols: -if len(symbol.headers) == 1: - # SYMBOL(unqualified_name, namespace, header) - print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, -symbol.headers[0])) -elif len(symbol.headers) == 0: +if re.match("^remove$|^swap$", symbol.name) and symbol.namespace == "std::": + continue +if len(symbol.headers) == 0: sys.stderr.write("No header found for symbol %s\n" % symbol.name) else: - # FIXME: support symbols with multiple headers (e.g. std::move). - sys.stderr.write("Ambiguous header for symbol %s: %s\n" % ( - symbol.name, ', '.join(symbol.headers))) - - + symbol.headers = sorted(symbol.headers) + for header in symbol.headers: +# SYMBOL(unqualified_name, namespace, header) +print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, + header)) if __name__ == '__main__': main() Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -47,7 +47,7 @@ Returns a list of headers. """ - headers = set() + symbol_headers = set() all_headers = set() soup = BeautifulSoup(symbol_page_html, "html.parser") @@ -58,32 +58,54 @@ # Defined in header .t-dsc-header # decl2.t-dcl for table in soup.select('table.t-dcl-begin, table.t-dsc-begin'): -current_headers = [] -was_decl = False -for row in table.select('tr'): - if _HasClass(row, 't-dcl', 't-dsc'): -was_decl = True -# Symbols are in the first cell. -found_symbols = row.find('td').stripped_strings -if not symbol_name in found_symbols: - continue -headers.update(current_headers) - elif _HasClass(row, 't-dsc-header'): -# If we saw a decl since the last header, this is a new block of headers -# for a new block of decls. -if was_decl: - current_headers = [] -was_decl = False +rows = table.select('tr') +i = 0 +while i < len(rows): + start = i + current_headers = set() + while i < len(rows) and _HasClass(rows[i], 't-dsc-header'): +row = rows[i] # There are also .t-dsc-header for "defined in namespace". if not "Defined in header " in row.text: + i = i + 1 continue # The interesting header content (e.g. ) is wrapped in . for header_code in row.find_all("code"): - current_headers.append(header_code.text) all_headers.add(header_code.text) - # If the symbol was never named, consider all named headers. - return headers or all_headers - + current_headers.add(header_code.text) +i = i + 1 + # Some tables have header rows, skip them. + while i < len(rows) and _HasClass(rows[i], 't-dsc-hitem'): +i = i + 1 + while i < len(rows) and (_HasClass(rows[i], 't-dcl', 't-dsc') + or not rows[i].has_attr("class")): +row = rows[i] +# Symbols are in the first cell. +raw_symbols = row.find('td').stripped_strings +
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo marked 2 inline comments as done. VitaNuo added inline comments. Comment at: clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc:106 -SYMBOL(atomic_exchange_explicit, std::, ) -SYMBOL(atomic_fetch_add, std::, ) -SYMBOL(atomic_fetch_add_explicit, std::, ) hokein wrote: > Looks like the regex filters too many symbols -- e.g. atomic_fetch_add is not > a variant symbol, we should keep it instead. The same to other following > symbols as well. They're all back now. Comment at: clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc:100 SYMBOL(atoll, std::, ) +SYMBOL(atomic, std::, ) +SYMBOL(atomic, std::, ) kadircet wrote: > hokein wrote: > > VitaNuo wrote: > > > VitaNuo wrote: > > > > kadircet wrote: > > > > > hokein wrote: > > > > > > Conceptually, this (and other `atomic_*` symbols) doesn't feel > > > > > > correct: > > > > > > - `` provides the generic template `template > > > > > > struct atomic;` > > > > > > - `` provides partial template specializations for > > > > > > `std::shared_ptr` and `std::weak_ptr` > > > > > > > > > > > > They are variant symbols (ideally, they should be treat as the > > > > > > `std::move()`). The issue here is that unlike `std::move` which has > > > > > > two individual entries in the index page, we only have one entry > > > > > > for `std::atomic` (extending the cppreference_parser.py script to > > > > > > perfectly distinguish these two cases in the > > > > > > [page](https://en.cppreference.com/w/cpp/atomic/atomic) seems > > > > > > non-trivial). Some options: > > > > > > > > > > > > 1) treat them as multiple-header symbols (like what this patch does > > > > > > now) > > > > > > 2) special-case these symbols like `std::move()` > > > > > > 3) always prefer the header providing generic templates > > > > > > > > > > > > @kadircet, what do you think? > > > > > right, i believe this patch is definitely adding keeping multiple > > > > > headers for a symbol around, but mixing it with keeping headers for > > > > > variants (e.g. overloads provided by different headers, or > > > > > specializations as mentioned here). > > > > > > > > > > we definitely need some more changes in parser to make sure we're > > > > > only preserving alternatives for **the same symbol** and not for any > > > > > other variants (overloads, specializations). IIRC there are currently > > > > > 2 places these variants could come into play: > > > > > - first is the symbol index page itself, symbols that have ` (foo)` > > > > > next to them have variants and should still be ignored (e.g. > > > > > std::remove variant of cstdio shouldn't be preserved in the scope of > > > > > this patch) > > > > > - second is inside the detail pages for symbols, e.g. > > > > > [std::atomic](http://go/std::atomic), we can have headers for > > > > > different declarations, they're clearly different variants so we > > > > > shouldn't add such symbols into the mapping `_ParseSymbolPage` > > > > > already does some detection of declarations in between headers. > > > > > > > > > > in the scope of this patch, we should keep ignoring both. > > > > I suggest to special-case the overloads for now, just not to solve all > > > > the problems in the same patch. > > > The first group are already ignored. We need to take a lot at how to > > > ignore the second one. > > Ideally, we should tweak the `_ParseSymbolPage` to handle this case, but I > > don't see a easy way to do it (the `atomic` > > [case](https://en.cppreference.com/w/cpp/atomic/atomic) is quite tricky > > where the symbol name is identical, only the template argument is > > different, and the simple text-match heuristic in `_ParseSymbolPage` fails > > in this case). > > > > so specialcasing these (there are not too many of them) looks fine to me. > I don't follow why we can't perform this detection directly, haven't been > taking a look at the details but the page looks like: > ``` > Defined in header > template< class T > struct atomic; > template< class U > struct atomic; > Defined in header > template< class U > struct atomic>; > template< class U > struct atomic>; > Defined in header > #define _Atomic(T) /* see below */ > ``` > > So `_ParseSymbolPage`, first sees then a bunch of decls, then > then a bunch more decls and so on, and in the end it returns all the > headers it has found. > > Why can't we change the logic here to return `nothing` when there are > different decls && headers? i.e. return empty if we see a new header after > seeing some declarations. > does this result in undesired behaviour elsewhere? Ok, I've fixed (or hacked :) this programmatically. The parsing of constants in https://en.cppreference.com/w/cpp/locale/codecvt_mode is not complete now, everything is attributed to the "codecvt" header, the "locale" header is ignored. Hope it's OK. If not, then I'm afraid I can only special-case the "atomic.*" symbols. ===
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo updated this revision to Diff 492396. VitaNuo marked an inline comment as done. VitaNuo added a comment. Remove special casing from gen_std.py. Re-introduce special handling for std::remove. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 Files: clang/include/clang/Tooling/Inclusions/CSymbolMap.inc clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp clang/tools/include-mapping/cppreference_parser.py clang/tools/include-mapping/gen_std.py Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -14,10 +14,7 @@ The generated files are located in clang/include/Tooling/Inclusions. Caveats and FIXMEs: - - only symbols directly in "std" namespace are added, we should also add std's -subnamespace symbols (e.g. chrono). - - symbols with multiple variants or defined in multiple headers aren't added, -e.g. std::move, std::swap + - symbols with multiple variants aren't added, e.g., std::move Usage: 1. Install BeautifulSoup dependency, see instruction: @@ -41,6 +38,7 @@ import datetime import os import sys +import re CODE_PREFIX = """\ //===-- gen_std.py generated file ---*- C++ -*-===// @@ -109,18 +107,15 @@ cppreference_modified_date = datetime.datetime.fromtimestamp( os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') print(CODE_PREFIX % (args.symbols.upper(), cppreference_modified_date)) + # FIXME: take care of overloads for symbol in symbols: -if len(symbol.headers) == 1: - # SYMBOL(unqualified_name, namespace, header) - print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, -symbol.headers[0])) -elif len(symbol.headers) == 0: +if len(symbol.headers) == 0: sys.stderr.write("No header found for symbol %s\n" % symbol.name) else: - # FIXME: support symbols with multiple headers (e.g. std::move). - sys.stderr.write("Ambiguous header for symbol %s: %s\n" % ( - symbol.name, ', '.join(symbol.headers))) - - + symbol.headers = sorted(symbol.headers) + for header in symbol.headers: +# SYMBOL(unqualified_name, namespace, header) +print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, + header)) if __name__ == '__main__': main() Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -36,7 +36,7 @@ return False -def _ParseSymbolPage(symbol_page_html, symbol_name): +def _ParseSymbolPage(symbol_page_html, symbol_name, header_to_accept): """Parse symbol page and retrieve the include header defined in this page. The symbol page provides header for the symbol, specifically in "Defined in header " section. An example: @@ -47,7 +47,7 @@ Returns a list of headers. """ - headers = set() + symbol_headers = set() all_headers = set() soup = BeautifulSoup(symbol_page_html, "html.parser") @@ -58,32 +58,57 @@ # Defined in header .t-dsc-header # decl2.t-dcl for table in soup.select('table.t-dcl-begin, table.t-dsc-begin'): -current_headers = [] -was_decl = False -for row in table.select('tr'): - if _HasClass(row, 't-dcl', 't-dsc'): -was_decl = True -# Symbols are in the first cell. -found_symbols = row.find('td').stripped_strings -if not symbol_name in found_symbols: - continue -headers.update(current_headers) - elif _HasClass(row, 't-dsc-header'): -# If we saw a decl since the last header, this is a new block of headers -# for a new block of decls. -if was_decl: - current_headers = [] -was_decl = False +rows = table.select('tr') +i = 0 +while i < len(rows): + start = i + current_headers = set() + while i < len(rows) and _HasClass(rows[i], 't-dsc-header'): +row = rows[i] # There are also .t-dsc-header for "defined in namespace". if not "Defined in header " in row.text: + i = i + 1 continue # The interesting header content (e.g. ) is wrapped in . for header_code in row.find_all("code"): - current_headers.append(header_code.text) + # FIXME: replace with proper handling of overloads + if header_to_accept != "" and header_code.text.strip("<>") != header_to_accept: +continue all_headers.add(header_code.text) - # If the symbol was never named, consider all named headers. - return headers o
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo updated this revision to Diff 492398. VitaNuo added a comment. Formatting. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 Files: clang/include/clang/Tooling/Inclusions/CSymbolMap.inc clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp clang/tools/include-mapping/cppreference_parser.py clang/tools/include-mapping/gen_std.py Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -14,10 +14,7 @@ The generated files are located in clang/include/Tooling/Inclusions. Caveats and FIXMEs: - - only symbols directly in "std" namespace are added, we should also add std's -subnamespace symbols (e.g. chrono). - - symbols with multiple variants or defined in multiple headers aren't added, -e.g. std::move, std::swap + - symbols with multiple variants aren't added, e.g., std::move Usage: 1. Install BeautifulSoup dependency, see instruction: @@ -41,6 +38,7 @@ import datetime import os import sys +import re CODE_PREFIX = """\ //===-- gen_std.py generated file ---*- C++ -*-===// @@ -109,18 +107,15 @@ cppreference_modified_date = datetime.datetime.fromtimestamp( os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') print(CODE_PREFIX % (args.symbols.upper(), cppreference_modified_date)) + # FIXME: take care of overloads for symbol in symbols: -if len(symbol.headers) == 1: - # SYMBOL(unqualified_name, namespace, header) - print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, -symbol.headers[0])) -elif len(symbol.headers) == 0: +if len(symbol.headers) == 0: sys.stderr.write("No header found for symbol %s\n" % symbol.name) else: - # FIXME: support symbols with multiple headers (e.g. std::move). - sys.stderr.write("Ambiguous header for symbol %s: %s\n" % ( - symbol.name, ', '.join(symbol.headers))) - - + symbol.headers = sorted(symbol.headers) + for header in symbol.headers: +# SYMBOL(unqualified_name, namespace, header) +print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, + header)) if __name__ == '__main__': main() Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -36,7 +36,7 @@ return False -def _ParseSymbolPage(symbol_page_html, symbol_name): +def _ParseSymbolPage(symbol_page_html, symbol_name, header_to_accept): """Parse symbol page and retrieve the include header defined in this page. The symbol page provides header for the symbol, specifically in "Defined in header " section. An example: @@ -47,7 +47,7 @@ Returns a list of headers. """ - headers = set() + symbol_headers = set() all_headers = set() soup = BeautifulSoup(symbol_page_html, "html.parser") @@ -58,32 +58,57 @@ # Defined in header .t-dsc-header # decl2.t-dcl for table in soup.select('table.t-dcl-begin, table.t-dsc-begin'): -current_headers = [] -was_decl = False -for row in table.select('tr'): - if _HasClass(row, 't-dcl', 't-dsc'): -was_decl = True -# Symbols are in the first cell. -found_symbols = row.find('td').stripped_strings -if not symbol_name in found_symbols: - continue -headers.update(current_headers) - elif _HasClass(row, 't-dsc-header'): -# If we saw a decl since the last header, this is a new block of headers -# for a new block of decls. -if was_decl: - current_headers = [] -was_decl = False +rows = table.select('tr') +i = 0 +while i < len(rows): + start = i + current_headers = set() + while i < len(rows) and _HasClass(rows[i], 't-dsc-header'): +row = rows[i] # There are also .t-dsc-header for "defined in namespace". if not "Defined in header " in row.text: + i = i + 1 continue # The interesting header content (e.g. ) is wrapped in . for header_code in row.find_all("code"): - current_headers.append(header_code.text) + # FIXME: replace with proper handling of overloads + if header_to_accept != "" and header_code.text.strip("<>") != header_to_accept: +continue all_headers.add(header_code.text) - # If the symbol was never named, consider all named headers. - return headers or all_headers - + current_headers.add(header_code.text) +i = i + 1 + # Some tables have header
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo updated this revision to Diff 492400. VitaNuo added a comment. Formatting. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 Files: clang/include/clang/Tooling/Inclusions/CSymbolMap.inc clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp clang/tools/include-mapping/cppreference_parser.py clang/tools/include-mapping/gen_std.py Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -14,10 +14,7 @@ The generated files are located in clang/include/Tooling/Inclusions. Caveats and FIXMEs: - - only symbols directly in "std" namespace are added, we should also add std's -subnamespace symbols (e.g. chrono). - - symbols with multiple variants or defined in multiple headers aren't added, -e.g. std::move, std::swap + - symbols with multiple variants aren't added, e.g., std::move Usage: 1. Install BeautifulSoup dependency, see instruction: @@ -41,6 +38,7 @@ import datetime import os import sys +import re CODE_PREFIX = """\ //===-- gen_std.py generated file ---*- C++ -*-===// @@ -109,18 +107,15 @@ cppreference_modified_date = datetime.datetime.fromtimestamp( os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') print(CODE_PREFIX % (args.symbols.upper(), cppreference_modified_date)) + # FIXME: take care of overloads for symbol in symbols: -if len(symbol.headers) == 1: - # SYMBOL(unqualified_name, namespace, header) - print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, -symbol.headers[0])) -elif len(symbol.headers) == 0: +if len(symbol.headers) == 0: sys.stderr.write("No header found for symbol %s\n" % symbol.name) else: - # FIXME: support symbols with multiple headers (e.g. std::move). - sys.stderr.write("Ambiguous header for symbol %s: %s\n" % ( - symbol.name, ', '.join(symbol.headers))) - - + symbol.headers = sorted(symbol.headers) + for header in symbol.headers: +# SYMBOL(unqualified_name, namespace, header) +print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, + header)) if __name__ == '__main__': main() Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -36,7 +36,7 @@ return False -def _ParseSymbolPage(symbol_page_html, symbol_name): +def _ParseSymbolPage(symbol_page_html, symbol_name, header_to_accept): """Parse symbol page and retrieve the include header defined in this page. The symbol page provides header for the symbol, specifically in "Defined in header " section. An example: @@ -47,7 +47,7 @@ Returns a list of headers. """ - headers = set() + symbol_headers = set() all_headers = set() soup = BeautifulSoup(symbol_page_html, "html.parser") @@ -58,32 +58,57 @@ # Defined in header .t-dsc-header # decl2.t-dcl for table in soup.select('table.t-dcl-begin, table.t-dsc-begin'): -current_headers = [] -was_decl = False -for row in table.select('tr'): - if _HasClass(row, 't-dcl', 't-dsc'): -was_decl = True -# Symbols are in the first cell. -found_symbols = row.find('td').stripped_strings -if not symbol_name in found_symbols: - continue -headers.update(current_headers) - elif _HasClass(row, 't-dsc-header'): -# If we saw a decl since the last header, this is a new block of headers -# for a new block of decls. -if was_decl: - current_headers = [] -was_decl = False +rows = table.select('tr') +i = 0 +while i < len(rows): + start = i + current_headers = set() + while i < len(rows) and _HasClass(rows[i], 't-dsc-header'): +row = rows[i] # There are also .t-dsc-header for "defined in namespace". if not "Defined in header " in row.text: + i = i + 1 continue # The interesting header content (e.g. ) is wrapped in . for header_code in row.find_all("code"): - current_headers.append(header_code.text) + # FIXME: replace with proper handling of overloads + if header_to_accept != "" and header_code.text.strip("<>") != header_to_accept: +continue all_headers.add(header_code.text) - # If the symbol was never named, consider all named headers. - return headers or all_headers - + current_headers.add(header_code.text) +i = i + 1 + # Some tables have header
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo updated this revision to Diff 492408. VitaNuo marked an inline comment as done. VitaNuo added a comment. Rely on alphabetic ordering to determine whether the symbol is new. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 Files: clang/include/clang/Tooling/Inclusions/CSymbolMap.inc clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp clang/tools/include-mapping/cppreference_parser.py clang/tools/include-mapping/gen_std.py Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -14,10 +14,7 @@ The generated files are located in clang/include/Tooling/Inclusions. Caveats and FIXMEs: - - only symbols directly in "std" namespace are added, we should also add std's -subnamespace symbols (e.g. chrono). - - symbols with multiple variants or defined in multiple headers aren't added, -e.g. std::move, std::swap + - symbols with multiple variants aren't added, e.g., std::move Usage: 1. Install BeautifulSoup dependency, see instruction: @@ -41,6 +38,7 @@ import datetime import os import sys +import re CODE_PREFIX = """\ //===-- gen_std.py generated file ---*- C++ -*-===// @@ -109,18 +107,15 @@ cppreference_modified_date = datetime.datetime.fromtimestamp( os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') print(CODE_PREFIX % (args.symbols.upper(), cppreference_modified_date)) + # FIXME: take care of overloads for symbol in symbols: -if len(symbol.headers) == 1: - # SYMBOL(unqualified_name, namespace, header) - print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, -symbol.headers[0])) -elif len(symbol.headers) == 0: +if len(symbol.headers) == 0: sys.stderr.write("No header found for symbol %s\n" % symbol.name) else: - # FIXME: support symbols with multiple headers (e.g. std::move). - sys.stderr.write("Ambiguous header for symbol %s: %s\n" % ( - symbol.name, ', '.join(symbol.headers))) - - + symbol.headers = sorted(symbol.headers) + for header in symbol.headers: +# SYMBOL(unqualified_name, namespace, header) +print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, + header)) if __name__ == '__main__': main() Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -36,7 +36,7 @@ return False -def _ParseSymbolPage(symbol_page_html, symbol_name): +def _ParseSymbolPage(symbol_page_html, symbol_name, header_to_accept): """Parse symbol page and retrieve the include header defined in this page. The symbol page provides header for the symbol, specifically in "Defined in header " section. An example: @@ -47,7 +47,7 @@ Returns a list of headers. """ - headers = set() + symbol_headers = set() all_headers = set() soup = BeautifulSoup(symbol_page_html, "html.parser") @@ -58,32 +58,57 @@ # Defined in header .t-dsc-header # decl2.t-dcl for table in soup.select('table.t-dcl-begin, table.t-dsc-begin'): -current_headers = [] -was_decl = False -for row in table.select('tr'): - if _HasClass(row, 't-dcl', 't-dsc'): -was_decl = True -# Symbols are in the first cell. -found_symbols = row.find('td').stripped_strings -if not symbol_name in found_symbols: - continue -headers.update(current_headers) - elif _HasClass(row, 't-dsc-header'): -# If we saw a decl since the last header, this is a new block of headers -# for a new block of decls. -if was_decl: - current_headers = [] -was_decl = False +rows = table.select('tr') +i = 0 +while i < len(rows): + start = i + current_headers = set() + while i < len(rows) and _HasClass(rows[i], 't-dsc-header'): +row = rows[i] # There are also .t-dsc-header for "defined in namespace". if not "Defined in header " in row.text: + i = i + 1 continue # The interesting header content (e.g. ) is wrapped in . for header_code in row.find_all("code"): - current_headers.append(header_code.text) + # FIXME: replace with proper handling of overloads + if header_to_accept != "" and header_code.text.strip("<>") != header_to_accept: +continue all_headers.add(header_code.text) - # If the symbol was never named, consider all named headers. - return headers or all_headers - +
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo added a comment. Thank you all for comments! The patch should be ready for the next review now. Comment at: clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc:1053 SYMBOL(remainder, std::, ) +SYMBOL(remove, std::, ) SYMBOL(remove_all_extents, std::, ) VitaNuo wrote: > hokein wrote: > > I think `std::remove` should not be in the scope of the patch, it has two > > variants: > > - std::remove from `` > > - and std::remove from ``. > Yes, agreed. Let's take care of the overloads going forward. The latest version keeps the `std::remove` overload from ``. Comment at: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp:58 +int SymIndex = NextAvailSymIndex; +if (NSSymbolMap *NSSymbols = NamespaceSymbols->lookup(NS)) { + auto It = NSSymbols->find(Name); hokein wrote: > VitaNuo wrote: > > hokein wrote: > > > Given the fact that multiple-header symbols are grouped in the .inc file, > > > we could simplify the code of checking a new symbol by looking at the > > > last available SymIndex: > > > > > > ``` > > > if (NextAvailSymIndex > 0 && SymbolNames[NextAvailSymIndex-1].first == NS > > > && SymbolNames[NextAvailSymIndex-1].second == Name) { > > >// this is not a new symbol. > > > } > > > ``` > > I know this is easier in terms of code, but this heavily relies on the > > order in the generated files. I would prefer to keep the library and the > > generator as decoupled as possible, even if it means slightly more complex > > code here. Overall, it seems more future-proof in case of unrelated > > generator changes (bugs?) that might change the order. > > but this heavily relies on the order in the generated files. > > Yeah, this is true. I actually think we should make it as an invariant > (multiple-header symbols are grouped) of the generated .inc files. This > invariant is important and useful, it is much easier for human to read and > justify. We can probably guarantee it in the generator side. Ok, sure. Sorted the symbols in the generator now and also applied the snippet from above. I'm not 100% sure the code became much simpler but this version seems fine too :) Comment at: clang/tools/include-mapping/cppreference_parser.py:174 - # std::remove<> has variant algorithm. - "std::remove": ("algorithm"), - } kadircet wrote: > VitaNuo wrote: > > VitaNuo wrote: > > > kadircet wrote: > > > > VitaNuo wrote: > > > > > kadircet wrote: > > > > > > this is actually checking for something else (sorry for the > > > > > > confusing naming). > > > > > > > > > > > > the `variant` here refers to library name mentioned in parentheses > > > > > > (this is same problem as `std::move`) on the std symbol index page > > > > > > https://en.cppreference.com/w/cpp/symbol_index (e.g. `remove<>() > > > > > > (algorithm)`). by getting rid of this we're introducing a > > > > > > regression, as previously `std::remove` wouldn't be recognized by > > > > > > the library, but now it'll be recognized and we'll keep suggesting > > > > > > `` for it. > > > > > > > > > > > > so we should actually keep this around. > > > > > Ok, I can keep this out of this patch, but we'll have to remove this > > > > > logic evetually when we deal with overloads. > > > > > > > > > > I have a slight suspicion that this code might be buggy, because it > > > > > suggests that one _of_ the variants should be accepted. What is does > > > > > in reality, though, is it keeps `algorithm` in the list of headers > > > > > suitable for `std::remove` alongside `cstdio`, and then in the last > > > > > step `std::remove` is ignored by the generator because of being > > > > > defined in two headers. > > > > > > > > > > With this patch, the result will be both `{cstdio, algorithm}`. Is > > > > > this (more) satisfactory for now compared to skipping `algorithm` due > > > > > to being an overload? > > > > > > > > > > Ok, I can keep this out of this patch, but we'll have to remove this > > > > > logic evetually when we deal with overloads. > > > > > > > > Surely, I wasn't saying this should stay here forever, i am just saying > > > > that what's done in the scope of this patch doesn't really address the > > > > issues "worked around" by this piece. > > > > > > > > > I have a slight suspicion that this code might be buggy, because it > > > > > suggests that one _of_ the variants should be accepted. What is does > > > > > in reality, though, is it keeps algorithm in the list of headers > > > > > suitable for std::remove alongside cstdio, and then in the last step > > > > > std::remove is ignored by the generator because of being defined in > > > > > two headers. > > > > > > > > right, it's because we have logic to prefer "non-variant" versions of > > > > symbols when available (i.e. in the absence of this logic, we'd prefer > > > > std::remove from cstdio). this logic enables us to preserve ce
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo updated this revision to Diff 492413. VitaNuo added a comment. Remove extra import. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 Files: clang/include/clang/Tooling/Inclusions/CSymbolMap.inc clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp clang/tools/include-mapping/cppreference_parser.py clang/tools/include-mapping/gen_std.py Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -14,10 +14,7 @@ The generated files are located in clang/include/Tooling/Inclusions. Caveats and FIXMEs: - - only symbols directly in "std" namespace are added, we should also add std's -subnamespace symbols (e.g. chrono). - - symbols with multiple variants or defined in multiple headers aren't added, -e.g. std::move, std::swap + - symbols with multiple variants aren't added, e.g., std::move Usage: 1. Install BeautifulSoup dependency, see instruction: @@ -109,18 +106,15 @@ cppreference_modified_date = datetime.datetime.fromtimestamp( os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') print(CODE_PREFIX % (args.symbols.upper(), cppreference_modified_date)) + # FIXME: take care of overloads for symbol in symbols: -if len(symbol.headers) == 1: - # SYMBOL(unqualified_name, namespace, header) - print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, -symbol.headers[0])) -elif len(symbol.headers) == 0: +if len(symbol.headers) == 0: sys.stderr.write("No header found for symbol %s\n" % symbol.name) else: - # FIXME: support symbols with multiple headers (e.g. std::move). - sys.stderr.write("Ambiguous header for symbol %s: %s\n" % ( - symbol.name, ', '.join(symbol.headers))) - - + symbol.headers = sorted(symbol.headers) + for header in symbol.headers: +# SYMBOL(unqualified_name, namespace, header) +print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, + header)) if __name__ == '__main__': main() Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -36,7 +36,7 @@ return False -def _ParseSymbolPage(symbol_page_html, symbol_name): +def _ParseSymbolPage(symbol_page_html, symbol_name, header_to_accept): """Parse symbol page and retrieve the include header defined in this page. The symbol page provides header for the symbol, specifically in "Defined in header " section. An example: @@ -47,7 +47,7 @@ Returns a list of headers. """ - headers = set() + symbol_headers = set() all_headers = set() soup = BeautifulSoup(symbol_page_html, "html.parser") @@ -58,32 +58,57 @@ # Defined in header .t-dsc-header # decl2.t-dcl for table in soup.select('table.t-dcl-begin, table.t-dsc-begin'): -current_headers = [] -was_decl = False -for row in table.select('tr'): - if _HasClass(row, 't-dcl', 't-dsc'): -was_decl = True -# Symbols are in the first cell. -found_symbols = row.find('td').stripped_strings -if not symbol_name in found_symbols: - continue -headers.update(current_headers) - elif _HasClass(row, 't-dsc-header'): -# If we saw a decl since the last header, this is a new block of headers -# for a new block of decls. -if was_decl: - current_headers = [] -was_decl = False +rows = table.select('tr') +i = 0 +while i < len(rows): + start = i + current_headers = set() + while i < len(rows) and _HasClass(rows[i], 't-dsc-header'): +row = rows[i] # There are also .t-dsc-header for "defined in namespace". if not "Defined in header " in row.text: + i = i + 1 continue # The interesting header content (e.g. ) is wrapped in . for header_code in row.find_all("code"): - current_headers.append(header_code.text) + # FIXME: replace with proper handling of overloads + if header_to_accept != "" and header_code.text.strip("<>") != header_to_accept: +continue all_headers.add(header_code.text) - # If the symbol was never named, consider all named headers. - return headers or all_headers - + current_headers.add(header_code.text) +i = i + 1 + # Some tables have header rows, skip them. + while i < len(rows) and _HasClass(rows[i], 't-dsc-hitem'): +i = i + 1 + while i < len(rows) and (_HasClass(rows[i], 't-dcl',
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo updated this revision to Diff 492774. VitaNuo marked 6 inline comments as done. VitaNuo added a comment. Address review comments. No new tests in this version. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 Files: clang/include/clang/Tooling/Inclusions/CSymbolMap.inc clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp clang/tools/include-mapping/cppreference_parser.py clang/tools/include-mapping/gen_std.py Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -14,10 +14,7 @@ The generated files are located in clang/include/Tooling/Inclusions. Caveats and FIXMEs: - - only symbols directly in "std" namespace are added, we should also add std's -subnamespace symbols (e.g. chrono). - - symbols with multiple variants or defined in multiple headers aren't added, -e.g. std::move, std::swap + - symbols with multiple variants aren't added, e.g., std::move Usage: 1. Install BeautifulSoup dependency, see instruction: @@ -109,18 +106,15 @@ cppreference_modified_date = datetime.datetime.fromtimestamp( os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') print(CODE_PREFIX % (args.symbols.upper(), cppreference_modified_date)) + # FIXME: take care of overloads for symbol in symbols: -if len(symbol.headers) == 1: - # SYMBOL(unqualified_name, namespace, header) - print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, -symbol.headers[0])) -elif len(symbol.headers) == 0: +if len(symbol.headers) == 0: sys.stderr.write("No header found for symbol %s\n" % symbol.name) else: - # FIXME: support symbols with multiple headers (e.g. std::move). - sys.stderr.write("Ambiguous header for symbol %s: %s\n" % ( - symbol.name, ', '.join(symbol.headers))) - - + symbol.headers = sorted(symbol.headers) + for header in symbol.headers: +# SYMBOL(unqualified_name, namespace, header) +print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, + header)) if __name__ == '__main__': main() Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -36,7 +36,7 @@ return False -def _ParseSymbolPage(symbol_page_html, symbol_name): +def _ParseSymbolPage(symbol_page_html, symbol_name, variant_to_accept): """Parse symbol page and retrieve the include header defined in this page. The symbol page provides header for the symbol, specifically in "Defined in header " section. An example: @@ -47,7 +47,7 @@ Returns a list of headers. """ - headers = set() + symbol_headers = set() all_headers = set() soup = BeautifulSoup(symbol_page_html, "html.parser") @@ -58,32 +58,51 @@ # Defined in header .t-dsc-header # decl2.t-dcl for table in soup.select('table.t-dcl-begin, table.t-dsc-begin'): -current_headers = [] -was_decl = False -for row in table.select('tr'): - if _HasClass(row, 't-dcl', 't-dsc'): -was_decl = True -# Symbols are in the first cell. -found_symbols = row.find('td').stripped_strings -if not symbol_name in found_symbols: - continue -headers.update(current_headers) - elif _HasClass(row, 't-dsc-header'): -# If we saw a decl since the last header, this is a new block of headers -# for a new block of decls. -if was_decl: - current_headers = [] -was_decl = False +rows = table.select('tr') +i = 0 +while i < len(rows): + start = i + current_headers = set() + while i < len(rows) and _HasClass(rows[i], 't-dsc-header'): +row = rows[i] # There are also .t-dsc-header for "defined in namespace". if not "Defined in header " in row.text: + i = i + 1 continue # The interesting header content (e.g. ) is wrapped in . for header_code in row.find_all("code"): - current_headers.append(header_code.text) + # FIXME: replace with proper handling of overloads + if variant_to_accept != "" and header_code.text.strip("<>") != variant_to_accept: +continue all_headers.add(header_code.text) - # If the symbol was never named, consider all named headers. - return headers or all_headers - + current_headers.add(header_code.text) +i = i + 1 + # Some tables have header rows, skip them. + while i < len(rows) and not _HasClass(rows[i], 't-dsc', 't-
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo updated this revision to Diff 492775. VitaNuo added a comment. Fix tests. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 Files: clang/include/clang/Tooling/Inclusions/CSymbolMap.inc clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp clang/tools/include-mapping/cppreference_parser.py clang/tools/include-mapping/gen_std.py clang/tools/include-mapping/test.py Index: clang/tools/include-mapping/test.py === --- clang/tools/include-mapping/test.py +++ clang/tools/include-mapping/test.py @@ -53,7 +53,7 @@ """ -self.assertEqual(_ParseSymbolPage(html, 'foo'), set([''])) +self.assertEqual(_ParseSymbolPage(html, 'foo', ""), set([''])) def testParseSymbolPage_MulHeaders(self): @@ -94,7 +94,7 @@ """ -self.assertEqual(_ParseSymbolPage(html, "foo"), +self.assertEqual(_ParseSymbolPage(html, "foo", ""), set(['', ''])) @@ -121,7 +121,7 @@ """ -self.assertEqual(_ParseSymbolPage(html, "foo"), +self.assertEqual(_ParseSymbolPage(html, "foo", ""), set(['', ''])) def testParseSymbolPage_MulSymbolsInSameTd(self): @@ -145,9 +145,9 @@ """ -self.assertEqual(_ParseSymbolPage(html, "int8_t"), +self.assertEqual(_ParseSymbolPage(html, "int8_t", ""), set([''])) -self.assertEqual(_ParseSymbolPage(html, "int16_t"), +self.assertEqual(_ParseSymbolPage(html, "int16_t", ""), set([''])) Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -14,10 +14,7 @@ The generated files are located in clang/include/Tooling/Inclusions. Caveats and FIXMEs: - - only symbols directly in "std" namespace are added, we should also add std's -subnamespace symbols (e.g. chrono). - - symbols with multiple variants or defined in multiple headers aren't added, -e.g. std::move, std::swap + - symbols with multiple variants aren't added, e.g., std::move Usage: 1. Install BeautifulSoup dependency, see instruction: @@ -109,18 +106,15 @@ cppreference_modified_date = datetime.datetime.fromtimestamp( os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') print(CODE_PREFIX % (args.symbols.upper(), cppreference_modified_date)) + # FIXME: take care of overloads for symbol in symbols: -if len(symbol.headers) == 1: - # SYMBOL(unqualified_name, namespace, header) - print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, -symbol.headers[0])) -elif len(symbol.headers) == 0: +if len(symbol.headers) == 0: sys.stderr.write("No header found for symbol %s\n" % symbol.name) else: - # FIXME: support symbols with multiple headers (e.g. std::move). - sys.stderr.write("Ambiguous header for symbol %s: %s\n" % ( - symbol.name, ', '.join(symbol.headers))) - - + symbol.headers = sorted(symbol.headers) + for header in symbol.headers: +# SYMBOL(unqualified_name, namespace, header) +print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, + header)) if __name__ == '__main__': main() Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -36,7 +36,7 @@ return False -def _ParseSymbolPage(symbol_page_html, symbol_name): +def _ParseSymbolPage(symbol_page_html, symbol_name, variant_to_accept): """Parse symbol page and retrieve the include header defined in this page. The symbol page provides header for the symbol, specifically in "Defined in header " section. An example: @@ -47,7 +47,7 @@ Returns a list of headers. """ - headers = set() + symbol_headers = set() all_headers = set() soup = BeautifulSoup(symbol_page_html, "html.parser") @@ -58,32 +58,51 @@ # Defined in header .t-dsc-header # decl2.t-dcl for table in soup.select('table.t-dcl-begin, table.t-dsc-begin'): -current_headers = [] -was_decl = False -for row in table.select('tr'): - if _HasClass(row, 't-dcl', 't-dsc'): -was_decl = True -# Symbols are in the first cell. -found_symbols = row.find('td').stripped_strings -if not symbol_name in found_symbols: - continue -headers.update(current_headers) - elif _HasClass(row, 't-dsc-header'): -# If we saw a decl since the last header, this is a new block of headers -# for a new block of decls. -if was_decl: -
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo marked an inline comment as done. VitaNuo added a comment. Thanks for the comments! AFAICS I've addressed all of them. Re tests, thanks for the reminder @hokein! I've fixed them now, everything is green. Comment at: clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc:205 SYMBOL(basic_syncbuf, std::, ) SYMBOL(begin, std::, ) SYMBOL(bernoulli_distribution, std::, ) kadircet wrote: > i think we should have other providers here, > https://en.cppreference.com/w/cpp/iterator/begin. do we know why they're > dropped? Yes, this list is the regeneration of the 2018 html book. It's explicitly _not_ generated from the 2022 book in order to have a clear diff related to the topic of this patch. In the 2018 book, the only provider for `std::begin` is `iterator`. Comment at: clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc:1162 SYMBOL(subtract_with_carry_engine, std::, ) +SYMBOL(swap, std::, ) SYMBOL(swap_ranges, std::, ) hokein wrote: > Is this intended? it seems to me the cppparser doesn't handle this case > correctly, in the swap symbol page, we have two headers in a single > `t-dsc-header` tr, the parser should consider both (like the above `size_t`). > > ``` > Defined in header > Defined in header > ``` You're right. There's another random class name (`t-dcl-rev-aux`) that needs to be skipped. I've changed the condition in line 81 of cppreference_parser.py (the loop that skips unneeded rows between the header block and the symbol block) to negation in order to make it more robust and avoid listing all these unneeded classes unnecessarily. Comment at: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp:57 +// ordered alpahbetically in the map. +unsigned SymIndex = NextAvailSymIndex; +if (NextAvailSymIndex > 0 && hokein wrote: > We can avoid a local variable by > > ``` > auto Add = [.. SymIndex(-1)] () { > if (SymIndex >=0 && > SymbolNames[SymIndex].first == NS && ...) { > } else { > // the first symbol, or new symbol. > ++SymIndex; > } > > SymbolNames[SymIndex] = {NS, Name}; > > } > ``` Ok, sure. Comment at: clang/tools/include-mapping/cppreference_parser.py:39 -def _ParseSymbolPage(symbol_page_html, symbol_name): +def _ParseSymbolPage(symbol_page_html, symbol_name, header_to_accept): """Parse symbol page and retrieve the include header defined in this page. hokein wrote: > If the `header_to_accept` is set, I think we can just return > {header_to_accept} directly. We'd need to add `<>` then. It needs a couple of extra lines here or there anyways, I don't think it makes a difference. Comment at: clang/tools/include-mapping/cppreference_parser.py:96 +if symbol_name in found_symbols: + # FIXME: only update symbol headers on first symbol discovery, assume + # same symbol occurence in a subsequent header block is an overload. hokein wrote: > IIUC, this is the major and intended change of the function, this is not a > FIXME. I guess it depends on how you view it: I meant it as FIXME in the context of overloads, i.e., we might want to remove this and add infrastructure to differentiate between overloads explicitly. I can remove the FIXME part, no problem. Comment at: clang/tools/include-mapping/cppreference_parser.py:105 + + # If the symbol was never named, consider all named headers. + # FIXME: only consider first symbol, assume further symbols are overloads hokein wrote: > The comment doesn't match the current behavior. IIUC, iof the symbol was > never named, we only consider the first header now. I've removed all the hacking with `all_headers` now, so the comment is valid again. Comment at: clang/tools/include-mapping/cppreference_parser.py:107 + # FIXME: only consider first symbol, assume further symbols are overloads + all_headers = sorted(list(all_headers)) + if len(all_headers) > 0: hokein wrote: > why sort the headers? This was a hack to make the generator return `atomic` rather than `memory` for `std::atomic.*` symbols. But I've noticed now that this adds some trouble to the C symbol map. I think now that cutting `all_headers` to the first element is wrong. I will special-case the `std::atomic.*` symbols instead, because so far I don't see a way to solve this programmatically without other collateral effects. Note that the problem is not `std::atomic` itself (for which Kadir extensively described the solution in some of the other comments), but rather `atomic_bool`, `atomic_char` etc. mentioned further in the page. The headers for them come from `all_headers` rather than `symbol_headers`, and there seems to be no way to limit them to `` without other collateral effects. Comment at: clang/tools/include-mapping/cpp
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo updated this revision to Diff 493564. VitaNuo marked 2 inline comments as done. VitaNuo added a comment. Remove all special-casing, skip all variant pages. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 Files: clang/include/clang/Tooling/Inclusions/CSymbolMap.inc clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp clang/tools/include-mapping/cppreference_parser.py clang/tools/include-mapping/gen_std.py clang/tools/include-mapping/test.py Index: clang/tools/include-mapping/test.py === --- clang/tools/include-mapping/test.py +++ clang/tools/include-mapping/test.py @@ -53,7 +53,7 @@ """ -self.assertEqual(_ParseSymbolPage(html, 'foo'), set([''])) +self.assertEqual(_ParseSymbolPage(html, 'foo', ""), set([''])) def testParseSymbolPage_MulHeaders(self): @@ -94,7 +94,7 @@ """ -self.assertEqual(_ParseSymbolPage(html, "foo"), +self.assertEqual(_ParseSymbolPage(html, "foo", ""), set(['', ''])) @@ -121,7 +121,7 @@ """ -self.assertEqual(_ParseSymbolPage(html, "foo"), +self.assertEqual(_ParseSymbolPage(html, "foo", ""), set(['', ''])) def testParseSymbolPage_MulSymbolsInSameTd(self): @@ -145,9 +145,9 @@ """ -self.assertEqual(_ParseSymbolPage(html, "int8_t"), +self.assertEqual(_ParseSymbolPage(html, "int8_t", ""), set([''])) -self.assertEqual(_ParseSymbolPage(html, "int16_t"), +self.assertEqual(_ParseSymbolPage(html, "int16_t", ""), set([''])) Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -14,10 +14,7 @@ The generated files are located in clang/include/Tooling/Inclusions. Caveats and FIXMEs: - - only symbols directly in "std" namespace are added, we should also add std's -subnamespace symbols (e.g. chrono). - - symbols with multiple variants or defined in multiple headers aren't added, -e.g. std::move, std::swap + - symbols with multiple variants aren't added, e.g., std::move Usage: 1. Install BeautifulSoup dependency, see instruction: @@ -109,18 +106,15 @@ cppreference_modified_date = datetime.datetime.fromtimestamp( os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') print(CODE_PREFIX % (args.symbols.upper(), cppreference_modified_date)) + # FIXME: take care of overloads for symbol in symbols: -if len(symbol.headers) == 1: - # SYMBOL(unqualified_name, namespace, header) - print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, -symbol.headers[0])) -elif len(symbol.headers) == 0: +if len(symbol.headers) == 0: sys.stderr.write("No header found for symbol %s\n" % symbol.name) else: - # FIXME: support symbols with multiple headers (e.g. std::move). - sys.stderr.write("Ambiguous header for symbol %s: %s\n" % ( - symbol.name, ', '.join(symbol.headers))) - - + symbol.headers = sorted(symbol.headers) + for header in symbol.headers: +# SYMBOL(unqualified_name, namespace, header) +print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, + header)) if __name__ == '__main__': main() Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -47,7 +47,7 @@ Returns a list of headers. """ - headers = set() + symbol_headers = set() all_headers = set() soup = BeautifulSoup(symbol_page_html, "html.parser") @@ -58,32 +58,48 @@ # Defined in header .t-dsc-header # decl2.t-dcl for table in soup.select('table.t-dcl-begin, table.t-dsc-begin'): -current_headers = [] -was_decl = False -for row in table.select('tr'): - if _HasClass(row, 't-dcl', 't-dsc'): -was_decl = True -# Symbols are in the first cell. -found_symbols = row.find('td').stripped_strings -if not symbol_name in found_symbols: - continue -headers.update(current_headers) - elif _HasClass(row, 't-dsc-header'): -# If we saw a decl since the last header, this is a new block of headers -# for a new block of decls. -if was_decl: - current_headers = [] -was_decl = False +rows = table.select('tr') +i = 0 +while i < len(rows): + start = i + current_headers = set() + while i < len(rows) and _HasClass(rows[i], 't-dsc-header'): +row = rows[i] #
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo added a comment. Thanks for the comments! Comment at: clang/tools/include-mapping/cppreference_parser.py:196 + "std::remove$": "algorithm", + "std::atomic.*": "atomic", + kadircet wrote: > there's no variant of "std::atomic.*" called "atomic", in > https://en.cppreference.com/w/cpp/symbol_index. is it something in 2018 > version of the html book? otherwise there's only the unnamed variant and > std::shared_ptr variant, and we should be preferring unnamed variant. This is obsolete. As discussed offline, I've removed all the special-casing logic. All the variants are skipped now without exceptions. Comment at: clang/tools/include-mapping/cppreference_parser.py:107 + # FIXME: only consider first symbol, assume further symbols are overloads + all_headers = sorted(list(all_headers)) + if len(all_headers) > 0: kadircet wrote: > VitaNuo wrote: > > hokein wrote: > > > why sort the headers? > > This was a hack to make the generator return `atomic` rather than `memory` > > for `std::atomic.*` symbols. But I've noticed now that this adds some > > trouble to the C symbol map. I think now that cutting `all_headers` to the > > first element is wrong. > > I will special-case the `std::atomic.*` symbols instead, because so far I > > don't see a way to solve this programmatically without other collateral > > effects. > > > > Note that the problem is not `std::atomic` itself (for which Kadir > > extensively described the solution in some of the other comments), but > > rather `atomic_bool`, `atomic_char` etc. mentioned further in the page. The > > headers for them come from `all_headers` rather than `symbol_headers`, and > > there seems to be no way to limit them to `` without other > > collateral effects. > > I think now that cutting all_headers to the first element is wrong. > > What's exactly breaking once you do that? (I guess by `first element` you > still mean "all the headers we've seen until the first declaration in the > page") > > > but rather atomic_bool, atomic_char etc. mentioned further in the page. > > The headers for them come from all_headers rather than symbol_headers, and > > there seems to be no way to limit them to without other collateral > > effects. > > I guess this is happening because we're trying to find headers that are for > the declaration block containing the interesting symbol name. Any idea if > that logic is winning us anything? e.g. if we dropped the `if > symbol_name in found_symbols:` what does change? > I guess by first element you still mean "all the headers we've seen until the > first declaration in the page" No, I mean the first element of the `all_headers` collection. `all_headers` is a collection that collects _all_ the headers in the page and, in case a symbol is not defined in a table with an individual header block, `all_headers` in the page are returned as an approximation. > What's exactly breaking once you do that? It is essentially arbitrary to limit `all_headers` to the first element (at the moment it would be the first element alphabetically). E.g., for the `INT8_C` and other constants [here](https://en.cppreference.com/w/cpp/types/integer), the result would be `inttypes.h` and `stdint.h` would be cut, so this result doesn't make sense. > if we dropped the `if symbol_name in found_symbols` what does change? Then all the headers in the table will be attributed to all the symbols in the table. Consider [this page](https://en.cppreference.com/w/cpp/numeric/math/div): `std::imaxdiv` would be reported as defined both in `cstdlib` and `cinttypes`, which is wrong. > I guess this is happening because we're trying to find headers that are for > the declaration block containing the interesting symbol name. Not really. Headers for `atomic_bool`, `atomic_char` etc. come from `all_headers` rather than `symbol_headers`. The reason for that is they are defined in a table which doesn't have its own header block, so we are trying to approximate the headers by naming all the headers in the file. Comment at: clang/tools/include-mapping/cppreference_parser.py:165 # FIXME: use these as a fallback rather than ignoring entirely. - variants_for_symbol = variants_to_accept.get( - (namespace or "") + symbol_name, ()) - if variant and variant not in variants_for_symbol: + header_to_accept = variants_to_accept.get( + (namespace or "") + symbol_name, "") kadircet wrote: > VitaNuo wrote: > > kadircet wrote: > > > `variant` is not necessarily the `header`, eg: > > > ``` > > > acos() > > > acos<>() (std::complex) (since C++11) > > > acos<>() (std::valarray) > > > ``` > > > > > > in this case variants are `std::complex` and `std::valarray`. > > > > > > hence we're not trying to "infer" the header we want to preserve but > > > rather decide on which symbol page we want to par
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo updated this revision to Diff 493565. VitaNuo added a comment. Remove a FIXME. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 Files: clang/include/clang/Tooling/Inclusions/CSymbolMap.inc clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp clang/tools/include-mapping/cppreference_parser.py clang/tools/include-mapping/gen_std.py clang/tools/include-mapping/test.py Index: clang/tools/include-mapping/test.py === --- clang/tools/include-mapping/test.py +++ clang/tools/include-mapping/test.py @@ -53,7 +53,7 @@ """ -self.assertEqual(_ParseSymbolPage(html, 'foo'), set([''])) +self.assertEqual(_ParseSymbolPage(html, 'foo', ""), set([''])) def testParseSymbolPage_MulHeaders(self): @@ -94,7 +94,7 @@ """ -self.assertEqual(_ParseSymbolPage(html, "foo"), +self.assertEqual(_ParseSymbolPage(html, "foo", ""), set(['', ''])) @@ -121,7 +121,7 @@ """ -self.assertEqual(_ParseSymbolPage(html, "foo"), +self.assertEqual(_ParseSymbolPage(html, "foo", ""), set(['', ''])) def testParseSymbolPage_MulSymbolsInSameTd(self): @@ -145,9 +145,9 @@ """ -self.assertEqual(_ParseSymbolPage(html, "int8_t"), +self.assertEqual(_ParseSymbolPage(html, "int8_t", ""), set([''])) -self.assertEqual(_ParseSymbolPage(html, "int16_t"), +self.assertEqual(_ParseSymbolPage(html, "int16_t", ""), set([''])) Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -14,10 +14,7 @@ The generated files are located in clang/include/Tooling/Inclusions. Caveats and FIXMEs: - - only symbols directly in "std" namespace are added, we should also add std's -subnamespace symbols (e.g. chrono). - - symbols with multiple variants or defined in multiple headers aren't added, -e.g. std::move, std::swap + - symbols with multiple variants aren't added, e.g., std::move Usage: 1. Install BeautifulSoup dependency, see instruction: @@ -110,17 +107,13 @@ os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') print(CODE_PREFIX % (args.symbols.upper(), cppreference_modified_date)) for symbol in symbols: -if len(symbol.headers) == 1: - # SYMBOL(unqualified_name, namespace, header) - print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, -symbol.headers[0])) -elif len(symbol.headers) == 0: +if len(symbol.headers) == 0: sys.stderr.write("No header found for symbol %s\n" % symbol.name) else: - # FIXME: support symbols with multiple headers (e.g. std::move). - sys.stderr.write("Ambiguous header for symbol %s: %s\n" % ( - symbol.name, ', '.join(symbol.headers))) - - + symbol.headers = sorted(symbol.headers) + for header in symbol.headers: +# SYMBOL(unqualified_name, namespace, header) +print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, + header)) if __name__ == '__main__': main() Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -47,7 +47,7 @@ Returns a list of headers. """ - headers = set() + symbol_headers = set() all_headers = set() soup = BeautifulSoup(symbol_page_html, "html.parser") @@ -58,32 +58,48 @@ # Defined in header .t-dsc-header # decl2.t-dcl for table in soup.select('table.t-dcl-begin, table.t-dsc-begin'): -current_headers = [] -was_decl = False -for row in table.select('tr'): - if _HasClass(row, 't-dcl', 't-dsc'): -was_decl = True -# Symbols are in the first cell. -found_symbols = row.find('td').stripped_strings -if not symbol_name in found_symbols: - continue -headers.update(current_headers) - elif _HasClass(row, 't-dsc-header'): -# If we saw a decl since the last header, this is a new block of headers -# for a new block of decls. -if was_decl: - current_headers = [] -was_decl = False +rows = table.select('tr') +i = 0 +while i < len(rows): + start = i + current_headers = set() + while i < len(rows) and _HasClass(rows[i], 't-dsc-header'): +row = rows[i] # There are also .t-dsc-header for "defined in namespace". if not "Defined in header " in row.text: + i = i + 1 continue # The interesting h
[PATCH] D142992: [include-mapping] Implement language separation in stdlib recognizer library
VitaNuo created this revision. Herald added a project: All. VitaNuo requested review of this revision. Herald added a project: clang. Herald added a subscriber: cfe-commits. Repository: rG LLVM Github Monorepo https://reviews.llvm.org/D142992 Files: clang/include/clang/Tooling/Inclusions/StandardLibrary.h clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp Index: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp === --- clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp +++ clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp @@ -8,6 +8,8 @@ #include "clang/Tooling/Inclusions/StandardLibrary.h" #include "clang/AST/Decl.h" +#include "clang/Basic/LangOptions.h" +#include "clang/Basic/LangStandard.h" #include "llvm/ADT/StringRef.h" #include "llvm/Support/Casting.h" @@ -15,35 +17,58 @@ namespace tooling { namespace stdlib { -static llvm::StringRef *HeaderNames; -static std::pair *SymbolNames; -static unsigned *SymbolHeaderIDs; -static llvm::DenseMap *HeaderIDs; // Maps symbol name -> Symbol::ID, within a namespace. using NSSymbolMap = llvm::DenseMap; -static llvm::DenseMap *NamespaceSymbols; -static int initialize() { +struct SymbolHeaderMapping { + llvm::StringRef *HeaderNames; + std::pair *SymbolNames; + unsigned *SymbolHeaderIDs; + llvm::DenseMap HeaderIDs; + llvm::DenseMap NamespaceSymbols; +}; + +static llvm::DenseMap LanguageMappings; + +static int countSymbols(Lang Language) { unsigned SymCount = 0; #define SYMBOL(Name, NS, Header) ++SymCount; + switch (Language) { + case Lang::C: #include "clang/Tooling/Inclusions/CSymbolMap.inc" +break; + case Lang::Cpp: #include "clang/Tooling/Inclusions/StdSymbolMap.inc" +break; + } #undef SYMBOL - SymbolNames = new std::remove_reference_t[SymCount]; - SymbolHeaderIDs = - new std::remove_reference_t[SymCount]; - NamespaceSymbols = new std::remove_reference_t; - HeaderIDs = new std::remove_reference_t; + return SymCount; +} + +static void initialize(Lang Language) { + SymbolHeaderMapping *Mapping = new SymbolHeaderMapping(); + LanguageMappings.try_emplace(Language, Mapping); + + unsigned SymCount = countSymbols(Language); + Mapping->SymbolNames = + new std::remove_reference_tSymbolNames)>[SymCount]; + Mapping->SymbolHeaderIDs = new std::remove_reference_t< + decltype(*Mapping->SymbolHeaderIDs)>[SymCount]; + // Mapping->NamespaceSymbols = + // new std::remove_reference_tNamespaceSymbols)>; + // Mapping->HeaderIDs = + // new std::remove_reference_tHeaderIDs)>; auto AddNS = [&](llvm::StringRef NS) -> NSSymbolMap & { -auto R = NamespaceSymbols->try_emplace(NS, nullptr); +auto R = Mapping->NamespaceSymbols.try_emplace(NS, nullptr); if (R.second) R.first->second = new NSSymbolMap(); return *R.first->second; }; auto AddHeader = [&](llvm::StringRef Header) -> unsigned { -return HeaderIDs->try_emplace(Header, HeaderIDs->size()).first->second; +return Mapping->HeaderIDs.try_emplace(Header, Mapping->HeaderIDs.size()) +.first->second; }; auto Add = [&, SymIndex(0)](llvm::StringRef Name, llvm::StringRef NS, @@ -51,8 +76,8 @@ if (NS == "None") NS = ""; -SymbolNames[SymIndex] = {NS, Name}; -SymbolHeaderIDs[SymIndex] = AddHeader(HeaderName); +Mapping->SymbolNames[SymIndex] = {NS, Name}; +Mapping->SymbolHeaderIDs[SymIndex] = AddHeader(HeaderName); NSSymbolMap &NSSymbols = AddNS(NS); NSSymbols.try_emplace(Name, SymIndex); @@ -60,14 +85,24 @@ ++SymIndex; }; #define SYMBOL(Name, NS, Header) Add(#Name, #NS, #Header); + switch (Language) { + case Lang::C: #include "clang/Tooling/Inclusions/CSymbolMap.inc" +break; + case Lang::Cpp: #include "clang/Tooling/Inclusions/StdSymbolMap.inc" +break; + } #undef SYMBOL - HeaderNames = new llvm::StringRef[HeaderIDs->size()]; - for (const auto &E : *HeaderIDs) -HeaderNames[E.second] = E.first; + Mapping->HeaderNames = new llvm::StringRef[Mapping->HeaderIDs.size()]; + for (const auto &E : Mapping->HeaderIDs) +Mapping->HeaderNames[E.second] = E.first; +} +static int initialize() { + for (Lang Language : AllLangs) +initialize(Language); return 0; } @@ -76,38 +111,67 @@ (void)Dummy; } -std::optional Header::named(llvm::StringRef Name) { +static SymbolHeaderMapping *GetMapping(Lang Language) { + auto MapIt = LanguageMappings.find(Language); + SymbolHeaderMapping *Mapping = MapIt->getSecond(); + return Mapping; +} + +std::optional Header::named(llvm::StringRef Name, Lang Language) { ensureInitialized(); - auto It = HeaderIDs->find(Name); - if (It == HeaderIDs->end()) + SymbolHeaderMapping *Mapping = GetMapping(Language); + auto It = Mapping->HeaderIDs.find(Name); + if (It == Mapping->HeaderIDs.end()) return std::nullopt; - return Header(It->second); + return Header(It->second, Language); } -llvm::StringRef Header::
[PATCH] D142992: [include-mapping] Implement language separation in stdlib recognizer library
VitaNuo updated this revision to Diff 493651. VitaNuo added a comment. Remove comments. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142992/new/ https://reviews.llvm.org/D142992 Files: clang/include/clang/Tooling/Inclusions/StandardLibrary.h clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp Index: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp === --- clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp +++ clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp @@ -8,6 +8,8 @@ #include "clang/Tooling/Inclusions/StandardLibrary.h" #include "clang/AST/Decl.h" +#include "clang/Basic/LangOptions.h" +#include "clang/Basic/LangStandard.h" #include "llvm/ADT/StringRef.h" #include "llvm/Support/Casting.h" @@ -15,35 +17,54 @@ namespace tooling { namespace stdlib { -static llvm::StringRef *HeaderNames; -static std::pair *SymbolNames; -static unsigned *SymbolHeaderIDs; -static llvm::DenseMap *HeaderIDs; // Maps symbol name -> Symbol::ID, within a namespace. using NSSymbolMap = llvm::DenseMap; -static llvm::DenseMap *NamespaceSymbols; -static int initialize() { +struct SymbolHeaderMapping { + llvm::StringRef *HeaderNames; + std::pair *SymbolNames; + unsigned *SymbolHeaderIDs; + llvm::DenseMap HeaderIDs; + llvm::DenseMap NamespaceSymbols; +}; + +static llvm::DenseMap LanguageMappings; + +static int countSymbols(Lang Language) { unsigned SymCount = 0; #define SYMBOL(Name, NS, Header) ++SymCount; + switch (Language) { + case Lang::C: #include "clang/Tooling/Inclusions/CSymbolMap.inc" +break; + case Lang::Cpp: #include "clang/Tooling/Inclusions/StdSymbolMap.inc" +break; + } #undef SYMBOL - SymbolNames = new std::remove_reference_t[SymCount]; - SymbolHeaderIDs = - new std::remove_reference_t[SymCount]; - NamespaceSymbols = new std::remove_reference_t; - HeaderIDs = new std::remove_reference_t; + return SymCount; +} + +static void initialize(Lang Language) { + SymbolHeaderMapping *Mapping = new SymbolHeaderMapping(); + LanguageMappings.try_emplace(Language, Mapping); + + unsigned SymCount = countSymbols(Language); + Mapping->SymbolNames = + new std::remove_reference_tSymbolNames)>[SymCount]; + Mapping->SymbolHeaderIDs = new std::remove_reference_t< + decltype(*Mapping->SymbolHeaderIDs)>[SymCount]; auto AddNS = [&](llvm::StringRef NS) -> NSSymbolMap & { -auto R = NamespaceSymbols->try_emplace(NS, nullptr); +auto R = Mapping->NamespaceSymbols.try_emplace(NS, nullptr); if (R.second) R.first->second = new NSSymbolMap(); return *R.first->second; }; auto AddHeader = [&](llvm::StringRef Header) -> unsigned { -return HeaderIDs->try_emplace(Header, HeaderIDs->size()).first->second; +return Mapping->HeaderIDs.try_emplace(Header, Mapping->HeaderIDs.size()) +.first->second; }; auto Add = [&, SymIndex(0)](llvm::StringRef Name, llvm::StringRef NS, @@ -51,8 +72,8 @@ if (NS == "None") NS = ""; -SymbolNames[SymIndex] = {NS, Name}; -SymbolHeaderIDs[SymIndex] = AddHeader(HeaderName); +Mapping->SymbolNames[SymIndex] = {NS, Name}; +Mapping->SymbolHeaderIDs[SymIndex] = AddHeader(HeaderName); NSSymbolMap &NSSymbols = AddNS(NS); NSSymbols.try_emplace(Name, SymIndex); @@ -60,14 +81,24 @@ ++SymIndex; }; #define SYMBOL(Name, NS, Header) Add(#Name, #NS, #Header); + switch (Language) { + case Lang::C: #include "clang/Tooling/Inclusions/CSymbolMap.inc" +break; + case Lang::Cpp: #include "clang/Tooling/Inclusions/StdSymbolMap.inc" +break; + } #undef SYMBOL - HeaderNames = new llvm::StringRef[HeaderIDs->size()]; - for (const auto &E : *HeaderIDs) -HeaderNames[E.second] = E.first; + Mapping->HeaderNames = new llvm::StringRef[Mapping->HeaderIDs.size()]; + for (const auto &E : Mapping->HeaderIDs) +Mapping->HeaderNames[E.second] = E.first; +} +static int initialize() { + for (Lang Language : AllLangs) +initialize(Language); return 0; } @@ -76,38 +107,67 @@ (void)Dummy; } -std::optional Header::named(llvm::StringRef Name) { +static SymbolHeaderMapping *GetMapping(Lang Language) { + auto MapIt = LanguageMappings.find(Language); + SymbolHeaderMapping *Mapping = MapIt->getSecond(); + return Mapping; +} + +std::optional Header::named(llvm::StringRef Name, Lang Language) { ensureInitialized(); - auto It = HeaderIDs->find(Name); - if (It == HeaderIDs->end()) + SymbolHeaderMapping *Mapping = GetMapping(Language); + auto It = Mapping->HeaderIDs.find(Name); + if (It == Mapping->HeaderIDs.end()) return std::nullopt; - return Header(It->second); + return Header(It->second, Language); } -llvm::StringRef Header::name() const { return HeaderNames[ID]; } -llvm::StringRef Symbol::scope() const { return SymbolNames[ID].first; } -llvm::StringRef Symbol::name() const { return SymbolNames[ID].second;
[PATCH] D142992: [include-mapping] Implement language separation in stdlib recognizer library
VitaNuo updated this revision to Diff 493863. VitaNuo added a comment. Minor improvements. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142992/new/ https://reviews.llvm.org/D142992 Files: clang/include/clang/Tooling/Inclusions/StandardLibrary.h clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp Index: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp === --- clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp +++ clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp @@ -8,6 +8,8 @@ #include "clang/Tooling/Inclusions/StandardLibrary.h" #include "clang/AST/Decl.h" +#include "clang/Basic/LangOptions.h" +#include "clang/Basic/LangStandard.h" #include "llvm/ADT/StringRef.h" #include "llvm/Support/Casting.h" @@ -15,35 +17,54 @@ namespace tooling { namespace stdlib { -static llvm::StringRef *HeaderNames; -static std::pair *SymbolNames; -static unsigned *SymbolHeaderIDs; -static llvm::DenseMap *HeaderIDs; // Maps symbol name -> Symbol::ID, within a namespace. using NSSymbolMap = llvm::DenseMap; -static llvm::DenseMap *NamespaceSymbols; -static int initialize() { +struct SymbolHeaderMapping { + llvm::StringRef *HeaderNames; + std::pair *SymbolNames; + unsigned *SymbolHeaderIDs; + llvm::DenseMap HeaderIDs; + llvm::DenseMap NamespaceSymbols; +}; + +static llvm::DenseMap LanguageMappings; + +static int countSymbols(Lang Language) { unsigned SymCount = 0; #define SYMBOL(Name, NS, Header) ++SymCount; + switch (Language) { + case Lang::C: #include "clang/Tooling/Inclusions/CSymbolMap.inc" +break; + case Lang::CXX: #include "clang/Tooling/Inclusions/StdSymbolMap.inc" +break; + } #undef SYMBOL - SymbolNames = new std::remove_reference_t[SymCount]; - SymbolHeaderIDs = - new std::remove_reference_t[SymCount]; - NamespaceSymbols = new std::remove_reference_t; - HeaderIDs = new std::remove_reference_t; + return SymCount; +} + +static void initialize(Lang Language) { + SymbolHeaderMapping *Mapping = new SymbolHeaderMapping(); + LanguageMappings.try_emplace(Language, Mapping); + + unsigned SymCount = countSymbols(Language); + Mapping->SymbolNames = + new std::remove_reference_tSymbolNames)>[SymCount]; + Mapping->SymbolHeaderIDs = new std::remove_reference_t< + decltype(*Mapping->SymbolHeaderIDs)>[SymCount]; auto AddNS = [&](llvm::StringRef NS) -> NSSymbolMap & { -auto R = NamespaceSymbols->try_emplace(NS, nullptr); +auto R = Mapping->NamespaceSymbols.try_emplace(NS, nullptr); if (R.second) R.first->second = new NSSymbolMap(); return *R.first->second; }; auto AddHeader = [&](llvm::StringRef Header) -> unsigned { -return HeaderIDs->try_emplace(Header, HeaderIDs->size()).first->second; +return Mapping->HeaderIDs.try_emplace(Header, Mapping->HeaderIDs.size()) +.first->second; }; auto Add = [&, SymIndex(0)](llvm::StringRef Name, llvm::StringRef NS, @@ -51,8 +72,8 @@ if (NS == "None") NS = ""; -SymbolNames[SymIndex] = {NS, Name}; -SymbolHeaderIDs[SymIndex] = AddHeader(HeaderName); +Mapping->SymbolNames[SymIndex] = {NS, Name}; +Mapping->SymbolHeaderIDs[SymIndex] = AddHeader(HeaderName); NSSymbolMap &NSSymbols = AddNS(NS); NSSymbols.try_emplace(Name, SymIndex); @@ -60,14 +81,24 @@ ++SymIndex; }; #define SYMBOL(Name, NS, Header) Add(#Name, #NS, #Header); + switch (Language) { + case Lang::C: #include "clang/Tooling/Inclusions/CSymbolMap.inc" +break; + case Lang::CXX: #include "clang/Tooling/Inclusions/StdSymbolMap.inc" +break; + } #undef SYMBOL - HeaderNames = new llvm::StringRef[HeaderIDs->size()]; - for (const auto &E : *HeaderIDs) -HeaderNames[E.second] = E.first; + Mapping->HeaderNames = new llvm::StringRef[Mapping->HeaderIDs.size()]; + for (const auto &E : Mapping->HeaderIDs) +Mapping->HeaderNames[E.second] = E.first; +} +static int initialize() { + for (Lang Language : AllLangs) +initialize(Language); return 0; } @@ -76,38 +107,63 @@ (void)Dummy; } -std::optional Header::named(llvm::StringRef Name) { +static SymbolHeaderMapping *GetMapping(Lang Language) { + auto MapIt = LanguageMappings.find(Language); + SymbolHeaderMapping *Mapping = MapIt->getSecond(); + return Mapping; +} + +std::optional Header::named(llvm::StringRef Name, Lang Language) { ensureInitialized(); - auto It = HeaderIDs->find(Name); - if (It == HeaderIDs->end()) + SymbolHeaderMapping *Mapping = GetMapping(Language); + auto It = Mapping->HeaderIDs.find(Name); + if (It == Mapping->HeaderIDs.end()) return std::nullopt; - return Header(It->second); + return Header(It->second, Language); +} + +llvm::StringRef Header::name() const { + SymbolHeaderMapping *Mapping = GetMapping(Language); + return Mapping->HeaderNames[ID]; +} +llvm::StringRef Symbol::scope() const { + SymbolHeaderMapping *Mappi
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo updated this revision to Diff 493869. VitaNuo marked an inline comment as done. VitaNuo added a comment. Add comment. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 Files: clang/include/clang/Tooling/Inclusions/CSymbolMap.inc clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp clang/tools/include-mapping/cppreference_parser.py clang/tools/include-mapping/gen_std.py clang/tools/include-mapping/test.py Index: clang/tools/include-mapping/test.py === --- clang/tools/include-mapping/test.py +++ clang/tools/include-mapping/test.py @@ -53,7 +53,7 @@ """ -self.assertEqual(_ParseSymbolPage(html, 'foo'), set([''])) +self.assertEqual(_ParseSymbolPage(html, 'foo', ""), set([''])) def testParseSymbolPage_MulHeaders(self): @@ -94,7 +94,7 @@ """ -self.assertEqual(_ParseSymbolPage(html, "foo"), +self.assertEqual(_ParseSymbolPage(html, "foo", ""), set(['', ''])) @@ -121,7 +121,7 @@ """ -self.assertEqual(_ParseSymbolPage(html, "foo"), +self.assertEqual(_ParseSymbolPage(html, "foo", ""), set(['', ''])) def testParseSymbolPage_MulSymbolsInSameTd(self): @@ -145,9 +145,9 @@ """ -self.assertEqual(_ParseSymbolPage(html, "int8_t"), +self.assertEqual(_ParseSymbolPage(html, "int8_t", ""), set([''])) -self.assertEqual(_ParseSymbolPage(html, "int16_t"), +self.assertEqual(_ParseSymbolPage(html, "int16_t", ""), set([''])) Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -14,10 +14,7 @@ The generated files are located in clang/include/Tooling/Inclusions. Caveats and FIXMEs: - - only symbols directly in "std" namespace are added, we should also add std's -subnamespace symbols (e.g. chrono). - - symbols with multiple variants or defined in multiple headers aren't added, -e.g. std::move, std::swap + - symbols with multiple variants aren't added, e.g., std::move Usage: 1. Install BeautifulSoup dependency, see instruction: @@ -110,17 +107,13 @@ os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') print(CODE_PREFIX % (args.symbols.upper(), cppreference_modified_date)) for symbol in symbols: -if len(symbol.headers) == 1: - # SYMBOL(unqualified_name, namespace, header) - print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, -symbol.headers[0])) -elif len(symbol.headers) == 0: +if len(symbol.headers) == 0: sys.stderr.write("No header found for symbol %s\n" % symbol.name) else: - # FIXME: support symbols with multiple headers (e.g. std::move). - sys.stderr.write("Ambiguous header for symbol %s: %s\n" % ( - symbol.name, ', '.join(symbol.headers))) - - + symbol.headers = sorted(symbol.headers) + for header in symbol.headers: +# SYMBOL(unqualified_name, namespace, header) +print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, + header)) if __name__ == '__main__': main() Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -47,7 +47,7 @@ Returns a list of headers. """ - headers = set() + symbol_headers = set() all_headers = set() soup = BeautifulSoup(symbol_page_html, "html.parser") @@ -58,32 +58,48 @@ # Defined in header .t-dsc-header # decl2.t-dcl for table in soup.select('table.t-dcl-begin, table.t-dsc-begin'): -current_headers = [] -was_decl = False -for row in table.select('tr'): - if _HasClass(row, 't-dcl', 't-dsc'): -was_decl = True -# Symbols are in the first cell. -found_symbols = row.find('td').stripped_strings -if not symbol_name in found_symbols: - continue -headers.update(current_headers) - elif _HasClass(row, 't-dsc-header'): -# If we saw a decl since the last header, this is a new block of headers -# for a new block of decls. -if was_decl: - current_headers = [] -was_decl = False +rows = table.select('tr') +i = 0 +while i < len(rows): + start = i + current_headers = set() + while i < len(rows) and _HasClass(rows[i], 't-dsc-header'): +row = rows[i] # There are also .t-dsc-header for "defined in namespace". if not "Defined in header " in row.text: + i = i + 1
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo updated this revision to Diff 493870. VitaNuo added a comment. Revert all changes to generator. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 Files: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp clang/tools/include-mapping/test.py Index: clang/tools/include-mapping/test.py === --- clang/tools/include-mapping/test.py +++ clang/tools/include-mapping/test.py @@ -53,7 +53,7 @@ """ -self.assertEqual(_ParseSymbolPage(html, 'foo'), set([''])) +self.assertEqual(_ParseSymbolPage(html, 'foo', ""), set([''])) def testParseSymbolPage_MulHeaders(self): @@ -94,7 +94,7 @@ """ -self.assertEqual(_ParseSymbolPage(html, "foo"), +self.assertEqual(_ParseSymbolPage(html, "foo", ""), set(['', ''])) @@ -121,7 +121,7 @@ """ -self.assertEqual(_ParseSymbolPage(html, "foo"), +self.assertEqual(_ParseSymbolPage(html, "foo", ""), set(['', ''])) def testParseSymbolPage_MulSymbolsInSameTd(self): @@ -145,9 +145,9 @@ """ -self.assertEqual(_ParseSymbolPage(html, "int8_t"), +self.assertEqual(_ParseSymbolPage(html, "int8_t", ""), set([''])) -self.assertEqual(_ParseSymbolPage(html, "int16_t"), +self.assertEqual(_ParseSymbolPage(html, "int16_t", ""), set([''])) Index: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp === --- clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp +++ clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp @@ -8,6 +8,7 @@ #include "clang/Tooling/Inclusions/StandardLibrary.h" #include "clang/AST/Decl.h" +#include "llvm/ADT/DenseMap.h" #include "llvm/ADT/StringRef.h" #include "llvm/Support/Casting.h" @@ -17,7 +18,7 @@ static llvm::StringRef *HeaderNames; static std::pair *SymbolNames; -static unsigned *SymbolHeaderIDs; +static llvm::SmallVector *SymbolHeaderIDs; static llvm::DenseMap *HeaderIDs; // Maps symbol name -> Symbol::ID, within a namespace. using NSSymbolMap = llvm::DenseMap; @@ -29,6 +30,8 @@ #include "clang/Tooling/Inclusions/CSymbolMap.inc" #include "clang/Tooling/Inclusions/StdSymbolMap.inc" #undef SYMBOL + // Allocates more space than necessary since multiple SYMBOLs might correspond + // to the same qualified name. SymbolNames = new std::remove_reference_t[SymCount]; SymbolHeaderIDs = new std::remove_reference_t[SymCount]; @@ -46,18 +49,26 @@ return HeaderIDs->try_emplace(Header, HeaderIDs->size()).first->second; }; - auto Add = [&, SymIndex(0)](llvm::StringRef Name, llvm::StringRef NS, - llvm::StringRef HeaderName) mutable { + auto Add = [&, SymIndex(-1)](llvm::StringRef Name, llvm::StringRef NS, + llvm::StringRef HeaderName) mutable { if (NS == "None") NS = ""; +// Determine whether the symbol is new. This relies on symbols being +// ordered alpahbetically in the map. +if (SymIndex >= 0 && SymbolNames[SymIndex].first == NS && +SymbolNames[SymIndex].second == Name) { + // Not a new symbol, use the same index. +} else { + // First symbol or new symbol, increment next available index. + ++SymIndex; +} + SymbolNames[SymIndex] = {NS, Name}; -SymbolHeaderIDs[SymIndex] = AddHeader(HeaderName); +SymbolHeaderIDs[SymIndex].emplace_back(AddHeader(HeaderName)); NSSymbolMap &NSSymbols = AddNS(NS); NSSymbols.try_emplace(Name, SymIndex); - -++SymIndex; }; #define SYMBOL(Name, NS, Header) Add(#Name, #NS, #Header); #include "clang/Tooling/Inclusions/CSymbolMap.inc" @@ -87,7 +98,7 @@ llvm::StringRef Symbol::scope() const { return SymbolNames[ID].first; } llvm::StringRef Symbol::name() const { return SymbolNames[ID].second; } std::optional Symbol::named(llvm::StringRef Scope, - llvm::StringRef Name) { +llvm::StringRef Name) { ensureInitialized(); if (NSSymbolMap *NSSymbols = NamespaceSymbols->lookup(Scope)) { auto It = NSSymbols->find(Name); @@ -96,9 +107,14 @@ } return std::nullopt; } -Header Symbol::header() const { return Header(SymbolHeaderIDs[ID]); } + +Header Symbol::header() const { return Header(SymbolHeaderIDs[ID][0]); } llvm::SmallVector Symbol::headers() const { - return {header()}; // FIXME: multiple in case of ambiguity + llvm::SmallVector result; + for (auto HeaderID : SymbolHeaderIDs[ID]) { +result.emplace_back(Header(HeaderID)); + } + return result; } Recognizer::Recognizer() { ensureInitialized(); } ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo updated this revision to Diff 493871. VitaNuo added a comment. Revert more. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 Files: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp Index: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp === --- clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp +++ clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp @@ -8,6 +8,7 @@ #include "clang/Tooling/Inclusions/StandardLibrary.h" #include "clang/AST/Decl.h" +#include "llvm/ADT/DenseMap.h" #include "llvm/ADT/StringRef.h" #include "llvm/Support/Casting.h" @@ -17,7 +18,7 @@ static llvm::StringRef *HeaderNames; static std::pair *SymbolNames; -static unsigned *SymbolHeaderIDs; +static llvm::SmallVector *SymbolHeaderIDs; static llvm::DenseMap *HeaderIDs; // Maps symbol name -> Symbol::ID, within a namespace. using NSSymbolMap = llvm::DenseMap; @@ -29,6 +30,8 @@ #include "clang/Tooling/Inclusions/CSymbolMap.inc" #include "clang/Tooling/Inclusions/StdSymbolMap.inc" #undef SYMBOL + // Allocates more space than necessary since multiple SYMBOLs might correspond + // to the same qualified name. SymbolNames = new std::remove_reference_t[SymCount]; SymbolHeaderIDs = new std::remove_reference_t[SymCount]; @@ -46,18 +49,26 @@ return HeaderIDs->try_emplace(Header, HeaderIDs->size()).first->second; }; - auto Add = [&, SymIndex(0)](llvm::StringRef Name, llvm::StringRef NS, - llvm::StringRef HeaderName) mutable { + auto Add = [&, SymIndex(-1)](llvm::StringRef Name, llvm::StringRef NS, + llvm::StringRef HeaderName) mutable { if (NS == "None") NS = ""; +// Determine whether the symbol is new. This relies on symbols being +// ordered alpahbetically in the map. +if (SymIndex >= 0 && SymbolNames[SymIndex].first == NS && +SymbolNames[SymIndex].second == Name) { + // Not a new symbol, use the same index. +} else { + // First symbol or new symbol, increment next available index. + ++SymIndex; +} + SymbolNames[SymIndex] = {NS, Name}; -SymbolHeaderIDs[SymIndex] = AddHeader(HeaderName); +SymbolHeaderIDs[SymIndex].emplace_back(AddHeader(HeaderName)); NSSymbolMap &NSSymbols = AddNS(NS); NSSymbols.try_emplace(Name, SymIndex); - -++SymIndex; }; #define SYMBOL(Name, NS, Header) Add(#Name, #NS, #Header); #include "clang/Tooling/Inclusions/CSymbolMap.inc" @@ -87,7 +98,7 @@ llvm::StringRef Symbol::scope() const { return SymbolNames[ID].first; } llvm::StringRef Symbol::name() const { return SymbolNames[ID].second; } std::optional Symbol::named(llvm::StringRef Scope, - llvm::StringRef Name) { +llvm::StringRef Name) { ensureInitialized(); if (NSSymbolMap *NSSymbols = NamespaceSymbols->lookup(Scope)) { auto It = NSSymbols->find(Name); @@ -96,9 +107,14 @@ } return std::nullopt; } -Header Symbol::header() const { return Header(SymbolHeaderIDs[ID]); } + +Header Symbol::header() const { return Header(SymbolHeaderIDs[ID][0]); } llvm::SmallVector Symbol::headers() const { - return {header()}; // FIXME: multiple in case of ambiguity + llvm::SmallVector result; + for (auto HeaderID : SymbolHeaderIDs[ID]) { +result.emplace_back(Header(HeaderID)); + } + return result; } Recognizer::Recognizer() { ensureInitialized(); } Index: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp === --- clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp +++ clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp @@ -8,6 +8,7 @@ #include "clang/Tooling/Inclusions/StandardLibrary.h" #include "clang/AST/Decl.h" +#include "llvm/ADT/DenseMap.h" #include "llvm/ADT/StringRef.h" #include "llvm/Support/Casting.h" @@ -17,7 +18,7 @@ static llvm::StringRef *HeaderNames; static std::pair *SymbolNames; -static unsigned *SymbolHeaderIDs; +static llvm::SmallVector *SymbolHeaderIDs; static llvm::DenseMap *HeaderIDs; // Maps symbol name -> Symbol::ID, within a namespace. using NSSymbolMap = llvm::DenseMap; @@ -29,6 +30,8 @@ #include "clang/Tooling/Inclusions/CSymbolMap.inc" #include "clang/Tooling/Inclusions/StdSymbolMap.inc" #undef SYMBOL + // Allocates more space than necessary since multiple SYMBOLs might correspond + // to the same qualified name. SymbolNames = new std::remove_reference_t[SymCount]; SymbolHeaderIDs = new std::remove_reference_t[SymCount]; @@ -46,18 +49,26 @@ return HeaderIDs->try_emplace(Header, HeaderIDs->size()).first->second; }; - auto Add = [&, SymIndex(0)](llvm::StringRef Name, llvm::StringRef NS, - llvm::StringRef HeaderName) mutable { + auto Add
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo added a comment. Reverted everything apart from library support. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D143054: [include-mapping] Regenerate the mappings from the 20220730 html book.
VitaNuo created this revision. Herald added a project: All. VitaNuo requested review of this revision. Herald added a reviewer: jdoerfert. Herald added subscribers: cfe-commits, sstefan1. Herald added a project: clang. Repository: rG LLVM Github Monorepo https://reviews.llvm.org/D143054 Files: clang/include/clang/Tooling/Inclusions/CSymbolMap.inc clang/include/clang/Tooling/Inclusions/StdRemovedSymbolMap.inc clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc Index: clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc === --- clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc +++ clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc @@ -6,45 +6,18 @@ // This file was generated automatically by // clang/tools/include-mapping/gen_std.py, DO NOT EDIT! // -// Generated from cppreference offline HTML book (modified on 2018-10-28). +// Generated from cppreference offline HTML book (modified on 2022-07-30). //===--===// -SYMBOL(Assignable, std::, ) -SYMBOL(Boolean, std::, ) -SYMBOL(Common, std::, ) -SYMBOL(CommonReference, std::, ) -SYMBOL(Constructible, std::, ) -SYMBOL(ConvertibleTo, std::, ) -SYMBOL(CopyConstructible, std::, ) -SYMBOL(Copyable, std::, ) -SYMBOL(DefaultConstructible, std::, ) -SYMBOL(DerivedFrom, std::, ) -SYMBOL(Destructible, std::, ) -SYMBOL(EqualityComparable, std::, ) -SYMBOL(EqualityComparableWith, std::, ) SYMBOL(FILE, std::, ) -SYMBOL(Integral, std::, ) -SYMBOL(Invocable, std::, ) -SYMBOL(Movable, std::, ) -SYMBOL(MoveConstructible, std::, ) -SYMBOL(Predicate, std::, ) -SYMBOL(Regular, std::, ) -SYMBOL(RegularInvocable, std::, ) -SYMBOL(Relation, std::, ) -SYMBOL(Same, std::, ) -SYMBOL(Semiregular, std::, ) -SYMBOL(SignedIntegral, std::, ) -SYMBOL(StrictTotallyOrdered, std::, ) -SYMBOL(StrictTotallyOrderedWith, std::, ) -SYMBOL(StrictWeakOrder, std::, ) -SYMBOL(Swappable, std::, ) -SYMBOL(SwappableWith, std::, ) -SYMBOL(UniformRandomBitGenerator, std::, ) -SYMBOL(UnsignedIntegral, std::, ) SYMBOL(_Exit, std::, ) SYMBOL(accumulate, std::, ) SYMBOL(acos, std::, ) +SYMBOL(acosf, std::, ) SYMBOL(acosh, std::, ) +SYMBOL(acoshf, std::, ) +SYMBOL(acoshl, std::, ) +SYMBOL(acosl, std::, ) SYMBOL(add_const, std::, ) SYMBOL(add_const_t, std::, ) SYMBOL(add_cv, std::, ) @@ -73,7 +46,10 @@ SYMBOL(alignment_of, std::, ) SYMBOL(alignment_of_v, std::, ) SYMBOL(all_of, std::, ) +SYMBOL(allocate_at_least, std::, ) SYMBOL(allocate_shared, std::, ) +SYMBOL(allocate_shared_for_overwrite, std::, ) +SYMBOL(allocation_result, std::, ) SYMBOL(allocator, std::, ) SYMBOL(allocator_arg, std::, ) SYMBOL(allocator_arg_t, std::, ) @@ -83,15 +59,35 @@ SYMBOL(apply, std::, ) SYMBOL(arg, std::, ) SYMBOL(array, std::, ) +SYMBOL(as_bytes, std::, ) SYMBOL(as_const, std::, ) +SYMBOL(as_writable_bytes, std::, ) SYMBOL(asctime, std::, ) SYMBOL(asin, std::, ) +SYMBOL(asinf, std::, ) SYMBOL(asinh, std::, ) +SYMBOL(asinhf, std::, ) +SYMBOL(asinhl, std::, ) +SYMBOL(asinl, std::, ) +SYMBOL(assignable_from, std::, ) +SYMBOL(assoc_laguerre, std::, ) +SYMBOL(assoc_laguerref, std::, ) +SYMBOL(assoc_laguerrel, std::, ) +SYMBOL(assoc_legendre, std::, ) +SYMBOL(assoc_legendref, std::, ) +SYMBOL(assoc_legendrel, std::, ) +SYMBOL(assume_aligned, std::, ) SYMBOL(async, std::, ) SYMBOL(at_quick_exit, std::, ) SYMBOL(atan, std::, ) SYMBOL(atan2, std::, ) +SYMBOL(atan2f, std::, ) +SYMBOL(atan2l, std::, ) +SYMBOL(atanf, std::, ) SYMBOL(atanh, std::, ) +SYMBOL(atanhf, std::, ) +SYMBOL(atanhl, std::, ) +SYMBOL(atanl, std::, ) SYMBOL(atexit, std::, ) SYMBOL(atof, std::, ) SYMBOL(atoi, std::, ) @@ -116,19 +112,28 @@ SYMBOL(atomic_flag, std::, ) SYMBOL(atomic_flag_clear, std::, ) SYMBOL(atomic_flag_clear_explicit, std::, ) +SYMBOL(atomic_flag_notify_all, std::, ) +SYMBOL(atomic_flag_notify_one, std::, ) +SYMBOL(atomic_flag_test, std::, ) SYMBOL(atomic_flag_test_and_set, std::, ) SYMBOL(atomic_flag_test_and_set_explicit, std::, ) +SYMBOL(atomic_flag_test_explicit, std::, ) +SYMBOL(atomic_flag_wait, std::, ) +SYMBOL(atomic_flag_wait_explicit, std::, ) SYMBOL(atomic_init, std::, ) -SYMBOL(atomic_is_lockfree, std::, ) +SYMBOL(atomic_is_lock_free, std::, ) SYMBOL(atomic_load, std::, ) SYMBOL(atomic_load_explicit, std::, ) +SYMBOL(atomic_notify_all, std::, ) +SYMBOL(atomic_notify_one, std::, ) SYMBOL(atomic_ref, std::, ) SYMBOL(atomic_signal_fence, std::, ) SYMBOL(atomic_store, std::, ) SYMBOL(atomic_store_explicit, std::, ) SYMBOL(atomic_thread_fence, std::, ) +SYMBOL(atomic_wait, std::, ) +SYMBOL(atomic_wait_explicit, std::, ) SYMBOL(atto, std::, ) -SYMBOL(auto_ptr, std::, ) SYMBOL(back_insert_iterator, std::, ) SYMBOL(back_inserter, std::, ) SYMBOL(bad_alloc, std::, ) @@ -141,35 +146,54 @@ SYMBOL(bad_typeid, std::, ) SYMBOL(bad_variant_access, std::, ) SYMBOL(bad_weak_ptr, std::, ) +SYMBOL(barrier, std::, ) SYMBOL(basic_common_reference, std:
[PATCH] D142992: [include-mapping] Implement language separation in stdlib recognizer library
VitaNuo updated this revision to Diff 493886. VitaNuo added a comment. Address review comments. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142992/new/ https://reviews.llvm.org/D142992 Files: clang/include/clang/Tooling/Inclusions/StandardLibrary.h clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp Index: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp === --- clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp +++ clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp @@ -8,6 +8,8 @@ #include "clang/Tooling/Inclusions/StandardLibrary.h" #include "clang/AST/Decl.h" +#include "clang/Basic/LangOptions.h" +#include "clang/Basic/LangStandard.h" #include "llvm/ADT/StringRef.h" #include "llvm/Support/Casting.h" @@ -15,35 +17,54 @@ namespace tooling { namespace stdlib { -static llvm::StringRef *HeaderNames; -static std::pair *SymbolNames; -static unsigned *SymbolHeaderIDs; -static llvm::DenseMap *HeaderIDs; // Maps symbol name -> Symbol::ID, within a namespace. using NSSymbolMap = llvm::DenseMap; -static llvm::DenseMap *NamespaceSymbols; -static int initialize() { +struct SymbolHeaderMapping { + llvm::StringRef *HeaderNames; + std::pair *SymbolNames; + unsigned *SymbolHeaderIDs; + llvm::DenseMap HeaderIDs; + llvm::DenseMap NamespaceSymbols; +}; + +static llvm::DenseMap LanguageMappings; + +static int countSymbols(Lang Language) { unsigned SymCount = 0; #define SYMBOL(Name, NS, Header) ++SymCount; + switch (Language) { + case Lang::C: #include "clang/Tooling/Inclusions/CSymbolMap.inc" +break; + case Lang::CXX: #include "clang/Tooling/Inclusions/StdSymbolMap.inc" +break; + } #undef SYMBOL - SymbolNames = new std::remove_reference_t[SymCount]; - SymbolHeaderIDs = - new std::remove_reference_t[SymCount]; - NamespaceSymbols = new std::remove_reference_t; - HeaderIDs = new std::remove_reference_t; + return SymCount; +} + +static void initialize(Lang Language) { + SymbolHeaderMapping *Mapping = new SymbolHeaderMapping(); + LanguageMappings.try_emplace(Language, Mapping); + + unsigned SymCount = countSymbols(Language); + Mapping->SymbolNames = + new std::remove_reference_tSymbolNames)>[SymCount]; + Mapping->SymbolHeaderIDs = new std::remove_reference_t< + decltype(*Mapping->SymbolHeaderIDs)>[SymCount]; auto AddNS = [&](llvm::StringRef NS) -> NSSymbolMap & { -auto R = NamespaceSymbols->try_emplace(NS, nullptr); +auto R = Mapping->NamespaceSymbols.try_emplace(NS, nullptr); if (R.second) R.first->second = new NSSymbolMap(); return *R.first->second; }; auto AddHeader = [&](llvm::StringRef Header) -> unsigned { -return HeaderIDs->try_emplace(Header, HeaderIDs->size()).first->second; +return Mapping->HeaderIDs.try_emplace(Header, Mapping->HeaderIDs.size()) +.first->second; }; auto Add = [&, SymIndex(0)](llvm::StringRef Name, llvm::StringRef NS, @@ -51,8 +72,8 @@ if (NS == "None") NS = ""; -SymbolNames[SymIndex] = {NS, Name}; -SymbolHeaderIDs[SymIndex] = AddHeader(HeaderName); +Mapping->SymbolNames[SymIndex] = {NS, Name}; +Mapping->SymbolHeaderIDs[SymIndex] = AddHeader(HeaderName); NSSymbolMap &NSSymbols = AddNS(NS); NSSymbols.try_emplace(Name, SymIndex); @@ -60,14 +81,24 @@ ++SymIndex; }; #define SYMBOL(Name, NS, Header) Add(#Name, #NS, #Header); + switch (Language) { + case Lang::C: #include "clang/Tooling/Inclusions/CSymbolMap.inc" +break; + case Lang::CXX: #include "clang/Tooling/Inclusions/StdSymbolMap.inc" +break; + } #undef SYMBOL - HeaderNames = new llvm::StringRef[HeaderIDs->size()]; - for (const auto &E : *HeaderIDs) -HeaderNames[E.second] = E.first; + Mapping->HeaderNames = new llvm::StringRef[Mapping->HeaderIDs.size()]; + for (const auto &E : Mapping->HeaderIDs) +Mapping->HeaderNames[E.second] = E.first; +} +static int initialize() { + for (Lang Language : AllLangs) +initialize(Language); return 0; } @@ -76,38 +107,62 @@ (void)Dummy; } -std::optional Header::named(llvm::StringRef Name) { +static SymbolHeaderMapping *GetMapping(Lang Language) { + auto MapIt = LanguageMappings.find(Language); + SymbolHeaderMapping *Mapping = MapIt->getSecond(); + return Mapping; +} + +std::optional Header::named(llvm::StringRef Name, Lang Language) { ensureInitialized(); - auto It = HeaderIDs->find(Name); - if (It == HeaderIDs->end()) + SymbolHeaderMapping *Mapping = GetMapping(Language); + auto It = Mapping->HeaderIDs.find(Name); + if (It == Mapping->HeaderIDs.end()) return std::nullopt; - return Header(It->second); + return Header(It->second, Language); +} + +llvm::StringRef Header::name() const { + SymbolHeaderMapping *Mapping = GetMapping(Language); + return Mapping->HeaderNames[ID]; +} +llvm::StringRef Symbol::scope() const { + SymbolHeaderMapping *
[PATCH] D142992: [include-mapping] Implement language separation in stdlib recognizer library
VitaNuo added a comment. Thanks for the comments! Comment at: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp:31 + +static llvm::DenseMap LanguageMappings; + hokein wrote: > using a map here seems like an overkill, we have just 2 elements, I'd just > use two separate variables (`CMapping`, and `CXXMapping`) what about the design idea that we might potentially want to extend this to multiple standards etc.? The idea is that it's extensible to `ObjC`, `OpenCL`... and so on and so forth, as has been discussed offline. Comment at: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp:100 +static int initialize() { + for (Lang Language : AllLangs) +initialize(Language); hokein wrote: > nit: just `for (Lang Language: {Lang::C, Lang::CXX})` or two statements > `initilize(Lang::C);` and `initialize(Lang::CXX);`. yes, same argument as above. I remember extensive discussions about the idea that we might want to extend this to multiple language versions etc. in the future. Comment at: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp:164 + ensureInitialized(); + SymbolHeaderMapping *Mapping = GetMapping(Language); hokein wrote: > Do we need the `ensureInitialized()` here? looks like no needed, we have > called it in the Recognizer constructor, you're right, not needed. Comment at: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp:188 + Lang RecognizerLang = Lang::CXX; + if (Language == clang::Language::C) { +RecognizerLang = Lang::C; hokein wrote: > nit: just use `LangOpts.CPlusPlus` to check the language. There's `LangStandard::isCPlusPlus` method that I've just discovered. That's probably even better. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142992/new/ https://reviews.llvm.org/D142992 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D142992: [include-mapping] Implement language separation in stdlib recognizer library
VitaNuo updated this revision to Diff 493918. VitaNuo added a comment. Add a couple more test cases to StandardLibraryTest.cpp. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142992/new/ https://reviews.llvm.org/D142992 Files: clang/include/clang/Tooling/Inclusions/StandardLibrary.h clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp clang/unittests/Tooling/StandardLibraryTest.cpp Index: clang/unittests/Tooling/StandardLibraryTest.cpp === --- clang/unittests/Tooling/StandardLibraryTest.cpp +++ clang/unittests/Tooling/StandardLibraryTest.cpp @@ -10,6 +10,7 @@ #include "clang/AST/ASTContext.h" #include "clang/AST/Decl.h" #include "clang/AST/DeclarationName.h" +#include "clang/Testing/CommandLineArgs.h" #include "clang/Testing/TestAST.h" #include "llvm/ADT/StringRef.h" #include "llvm/Support/Casting.h" @@ -17,6 +18,7 @@ #include "gmock/gmock.h" #include "gtest/gtest.h" +#include using ::testing::ElementsAre; @@ -103,6 +105,56 @@ EXPECT_EQ(Recognizer(Sec), std::nullopt); } +TEST(StdlibTest, LanguageSeparationForSymbols) { + EXPECT_NE(std::nullopt, stdlib::Symbol::named("std::", "vector")); + EXPECT_NE(std::nullopt, +stdlib::Symbol::named("std::", "vector", stdlib::Lang::CXX)); + EXPECT_EQ(std::nullopt, +stdlib::Symbol::named("std::", "vector", stdlib::Lang::C)); + + EXPECT_EQ(std::nullopt, stdlib::Symbol::named("", "int16_t")); + EXPECT_EQ(std::nullopt, +stdlib::Symbol::named("", "int16_t", stdlib::Lang::CXX)); + EXPECT_NE(std::nullopt, +stdlib::Symbol::named("", "int16_t", stdlib::Lang::C)); +} + +TEST(StdlibTest, LanguageSeparationForHeaders) { + EXPECT_NE(std::nullopt, stdlib::Header::named("vector")); + EXPECT_NE(std::nullopt, stdlib::Header::named("vector", stdlib::Lang::CXX)); + EXPECT_EQ(std::nullopt, stdlib::Header::named("vector", stdlib::Lang::C)); + + EXPECT_EQ(std::nullopt, stdlib::Header::named("stdint.h")); + EXPECT_EQ(std::nullopt, stdlib::Header::named("stdint.h", stdlib::Lang::CXX)); + EXPECT_NE(std::nullopt, stdlib::Header::named("stdint.h", stdlib::Lang::C)); +} + +TEST(StdlibTest, LanguageSeparationForRecognizer) { + llvm::StringRef Code = R"cpp( + std::vector vec; + )cpp"; + TestInputs Inputs(Code); + Inputs.Language = TestLanguage::Lang_CXX11; + TestAST AST(Inputs); + auto *Vec = cast(lookup(AST, "vec")).getType()->getAsCXXRecordDecl(); + + stdlib::Recognizer Recognizer; + EXPECT_EQ(Recognizer(Vec), stdlib::Symbol::named("std::", "vector")); + EXPECT_EQ(Recognizer(Vec), +stdlib::Symbol::named("std::", "vector", stdlib::Lang::CXX)); + + Code = R"c( + int16_t var = 42; + )c"; + Inputs = TestInputs(Code); + Inputs.Language = TestLanguage::Lang_C99; + AST = TestAST(Inputs); + + auto *Var = cast(lookup(AST, "var")).getType()->getAsRecordDecl(); + EXPECT_EQ(Recognizer(Var), +stdlib::Symbol::named("", "int16_t", stdlib::Lang::C)); +} + } // namespace } // namespace tooling } // namespace clang Index: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp === --- clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp +++ clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp @@ -8,6 +8,8 @@ #include "clang/Tooling/Inclusions/StandardLibrary.h" #include "clang/AST/Decl.h" +#include "clang/Basic/LangOptions.h" +#include "clang/Basic/LangStandard.h" #include "llvm/ADT/StringRef.h" #include "llvm/Support/Casting.h" @@ -15,35 +17,54 @@ namespace tooling { namespace stdlib { -static llvm::StringRef *HeaderNames; -static std::pair *SymbolNames; -static unsigned *SymbolHeaderIDs; -static llvm::DenseMap *HeaderIDs; // Maps symbol name -> Symbol::ID, within a namespace. using NSSymbolMap = llvm::DenseMap; -static llvm::DenseMap *NamespaceSymbols; -static int initialize() { +struct SymbolHeaderMapping { + llvm::StringRef *HeaderNames; + std::pair *SymbolNames; + unsigned *SymbolHeaderIDs; + llvm::DenseMap HeaderIDs; + llvm::DenseMap NamespaceSymbols; +}; + +static llvm::DenseMap LanguageMappings; + +static int countSymbols(Lang Language) { unsigned SymCount = 0; #define SYMBOL(Name, NS, Header) ++SymCount; + switch (Language) { + case Lang::C: #include "clang/Tooling/Inclusions/CSymbolMap.inc" +break; + case Lang::CXX: #include "clang/Tooling/Inclusions/StdSymbolMap.inc" +break; + } #undef SYMBOL - SymbolNames = new std::remove_reference_t[SymCount]; - SymbolHeaderIDs = - new std::remove_reference_t[SymCount]; - NamespaceSymbols = new std::remove_reference_t; - HeaderIDs = new std::remove_reference_t; + return SymCount; +} + +static void initialize(Lang Language) { + SymbolHeaderMapping *Mapping = new SymbolHeaderMapping(); + LanguageMappings.try_emplace(Language, Mapping); + + unsigned SymCount = countSymbols(Language); + Mapping->Sy
[PATCH] D142992: [include-mapping] Implement language separation in stdlib recognizer library
VitaNuo added a comment. Added a couple of test cases. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142992/new/ https://reviews.llvm.org/D142992 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D143054: [include-mapping] Regenerate the mappings from the 20220730 html book.
VitaNuo updated this revision to Diff 494250. VitaNuo added a comment. Remove the C symbol map from commit. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D143054/new/ https://reviews.llvm.org/D143054 Files: clang/include/clang/Tooling/Inclusions/StdRemovedSymbolMap.inc clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc Index: clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc === --- clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc +++ clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc @@ -6,45 +6,18 @@ // This file was generated automatically by // clang/tools/include-mapping/gen_std.py, DO NOT EDIT! // -// Generated from cppreference offline HTML book (modified on 2018-10-28). +// Generated from cppreference offline HTML book (modified on 2022-07-30). //===--===// -SYMBOL(Assignable, std::, ) -SYMBOL(Boolean, std::, ) -SYMBOL(Common, std::, ) -SYMBOL(CommonReference, std::, ) -SYMBOL(Constructible, std::, ) -SYMBOL(ConvertibleTo, std::, ) -SYMBOL(CopyConstructible, std::, ) -SYMBOL(Copyable, std::, ) -SYMBOL(DefaultConstructible, std::, ) -SYMBOL(DerivedFrom, std::, ) -SYMBOL(Destructible, std::, ) -SYMBOL(EqualityComparable, std::, ) -SYMBOL(EqualityComparableWith, std::, ) SYMBOL(FILE, std::, ) -SYMBOL(Integral, std::, ) -SYMBOL(Invocable, std::, ) -SYMBOL(Movable, std::, ) -SYMBOL(MoveConstructible, std::, ) -SYMBOL(Predicate, std::, ) -SYMBOL(Regular, std::, ) -SYMBOL(RegularInvocable, std::, ) -SYMBOL(Relation, std::, ) -SYMBOL(Same, std::, ) -SYMBOL(Semiregular, std::, ) -SYMBOL(SignedIntegral, std::, ) -SYMBOL(StrictTotallyOrdered, std::, ) -SYMBOL(StrictTotallyOrderedWith, std::, ) -SYMBOL(StrictWeakOrder, std::, ) -SYMBOL(Swappable, std::, ) -SYMBOL(SwappableWith, std::, ) -SYMBOL(UniformRandomBitGenerator, std::, ) -SYMBOL(UnsignedIntegral, std::, ) SYMBOL(_Exit, std::, ) SYMBOL(accumulate, std::, ) SYMBOL(acos, std::, ) +SYMBOL(acosf, std::, ) SYMBOL(acosh, std::, ) +SYMBOL(acoshf, std::, ) +SYMBOL(acoshl, std::, ) +SYMBOL(acosl, std::, ) SYMBOL(add_const, std::, ) SYMBOL(add_const_t, std::, ) SYMBOL(add_cv, std::, ) @@ -73,7 +46,10 @@ SYMBOL(alignment_of, std::, ) SYMBOL(alignment_of_v, std::, ) SYMBOL(all_of, std::, ) +SYMBOL(allocate_at_least, std::, ) SYMBOL(allocate_shared, std::, ) +SYMBOL(allocate_shared_for_overwrite, std::, ) +SYMBOL(allocation_result, std::, ) SYMBOL(allocator, std::, ) SYMBOL(allocator_arg, std::, ) SYMBOL(allocator_arg_t, std::, ) @@ -83,15 +59,35 @@ SYMBOL(apply, std::, ) SYMBOL(arg, std::, ) SYMBOL(array, std::, ) +SYMBOL(as_bytes, std::, ) SYMBOL(as_const, std::, ) +SYMBOL(as_writable_bytes, std::, ) SYMBOL(asctime, std::, ) SYMBOL(asin, std::, ) +SYMBOL(asinf, std::, ) SYMBOL(asinh, std::, ) +SYMBOL(asinhf, std::, ) +SYMBOL(asinhl, std::, ) +SYMBOL(asinl, std::, ) +SYMBOL(assignable_from, std::, ) +SYMBOL(assoc_laguerre, std::, ) +SYMBOL(assoc_laguerref, std::, ) +SYMBOL(assoc_laguerrel, std::, ) +SYMBOL(assoc_legendre, std::, ) +SYMBOL(assoc_legendref, std::, ) +SYMBOL(assoc_legendrel, std::, ) +SYMBOL(assume_aligned, std::, ) SYMBOL(async, std::, ) SYMBOL(at_quick_exit, std::, ) SYMBOL(atan, std::, ) SYMBOL(atan2, std::, ) +SYMBOL(atan2f, std::, ) +SYMBOL(atan2l, std::, ) +SYMBOL(atanf, std::, ) SYMBOL(atanh, std::, ) +SYMBOL(atanhf, std::, ) +SYMBOL(atanhl, std::, ) +SYMBOL(atanl, std::, ) SYMBOL(atexit, std::, ) SYMBOL(atof, std::, ) SYMBOL(atoi, std::, ) @@ -116,19 +112,28 @@ SYMBOL(atomic_flag, std::, ) SYMBOL(atomic_flag_clear, std::, ) SYMBOL(atomic_flag_clear_explicit, std::, ) +SYMBOL(atomic_flag_notify_all, std::, ) +SYMBOL(atomic_flag_notify_one, std::, ) +SYMBOL(atomic_flag_test, std::, ) SYMBOL(atomic_flag_test_and_set, std::, ) SYMBOL(atomic_flag_test_and_set_explicit, std::, ) +SYMBOL(atomic_flag_test_explicit, std::, ) +SYMBOL(atomic_flag_wait, std::, ) +SYMBOL(atomic_flag_wait_explicit, std::, ) SYMBOL(atomic_init, std::, ) -SYMBOL(atomic_is_lockfree, std::, ) +SYMBOL(atomic_is_lock_free, std::, ) SYMBOL(atomic_load, std::, ) SYMBOL(atomic_load_explicit, std::, ) +SYMBOL(atomic_notify_all, std::, ) +SYMBOL(atomic_notify_one, std::, ) SYMBOL(atomic_ref, std::, ) SYMBOL(atomic_signal_fence, std::, ) SYMBOL(atomic_store, std::, ) SYMBOL(atomic_store_explicit, std::, ) SYMBOL(atomic_thread_fence, std::, ) +SYMBOL(atomic_wait, std::, ) +SYMBOL(atomic_wait_explicit, std::, ) SYMBOL(atto, std::, ) -SYMBOL(auto_ptr, std::, ) SYMBOL(back_insert_iterator, std::, ) SYMBOL(back_inserter, std::, ) SYMBOL(bad_alloc, std::, ) @@ -141,35 +146,54 @@ SYMBOL(bad_typeid, std::, ) SYMBOL(bad_variant_access, std::, ) SYMBOL(bad_weak_ptr, std::, ) +SYMBOL(barrier, std::, ) SYMBOL(basic_common_reference, std::, ) SYMBOL(basic_filebuf, std::, ) +SYMBOL(basic_format_arg, std::, ) +SYMBOL(basic_format_args,
[PATCH] D143054: [include-mapping] Regenerate the mappings from the 20220730 html book.
VitaNuo added a comment. Sure, removed the C symbol mapping from this patch. Thanks. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D143054/new/ https://reviews.llvm.org/D143054 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D142992: [include-mapping] Implement language separation in stdlib recognizer library
VitaNuo updated this revision to Diff 494329. VitaNuo added a comment. Address review comments. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142992/new/ https://reviews.llvm.org/D142992 Files: clang/include/clang/Tooling/Inclusions/StandardLibrary.h clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp clang/unittests/Tooling/StandardLibraryTest.cpp Index: clang/unittests/Tooling/StandardLibraryTest.cpp === --- clang/unittests/Tooling/StandardLibraryTest.cpp +++ clang/unittests/Tooling/StandardLibraryTest.cpp @@ -38,14 +38,28 @@ EXPECT_EQ(llvm::to_string(*VectorH), ""); EXPECT_FALSE(stdlib::Header::named("HeadersTests.cpp")); + EXPECT_TRUE(stdlib::Header::named("", stdlib::Lang::CXX)); + EXPECT_FALSE(stdlib::Header::named("", stdlib::Lang::C)); + auto Vector = stdlib::Symbol::named("std::", "vector"); EXPECT_TRUE(Vector); EXPECT_EQ(llvm::to_string(*Vector), "std::vector"); EXPECT_FALSE(stdlib::Symbol::named("std::", "dongle")); EXPECT_FALSE(stdlib::Symbol::named("clang::", "ASTContext")); + EXPECT_TRUE(stdlib::Symbol::named("std::", "vector", stdlib::Lang::CXX)); + EXPECT_FALSE(stdlib::Symbol::named("std::", "vector", stdlib::Lang::C)); + EXPECT_EQ(Vector->header(), *VectorH); EXPECT_THAT(Vector->headers(), ElementsAre(*VectorH)); + + EXPECT_FALSE(stdlib::Header::named("")); + EXPECT_FALSE(stdlib::Header::named("", stdlib::Lang::CXX)); + EXPECT_TRUE(stdlib::Header::named("", stdlib::Lang::C)); + + EXPECT_FALSE(stdlib::Symbol::named("", "int16_t")); + EXPECT_FALSE(stdlib::Symbol::named("", "int16_t", stdlib::Lang::CXX)); + EXPECT_TRUE(stdlib::Symbol::named("", "int16_t", stdlib::Lang::C)); } TEST(StdlibTest, Recognizer) { @@ -96,10 +110,14 @@ EXPECT_EQ(Recognizer(&VectorNonstd), std::nullopt); EXPECT_EQ(Recognizer(Vec), stdlib::Symbol::named("std::", "vector")); + EXPECT_EQ(Recognizer(Vec), +stdlib::Symbol::named("std::", "vector", stdlib::Lang::CXX)); EXPECT_EQ(Recognizer(Nest), stdlib::Symbol::named("std::", "vector")); EXPECT_EQ(Recognizer(Clock), stdlib::Symbol::named("std::chrono::", "system_clock")); EXPECT_EQ(Recognizer(CDivT), stdlib::Symbol::named("", "div_t")); + EXPECT_EQ(Recognizer(CDivT), +stdlib::Symbol::named("", "div_t", stdlib::Lang::C)); EXPECT_EQ(Recognizer(Sec), std::nullopt); } Index: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp === --- clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp +++ clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp @@ -8,42 +8,69 @@ #include "clang/Tooling/Inclusions/StandardLibrary.h" #include "clang/AST/Decl.h" +#include "clang/Basic/LangOptions.h" #include "llvm/ADT/StringRef.h" #include "llvm/Support/Casting.h" +#include "llvm/Support/raw_ostream.h" namespace clang { namespace tooling { namespace stdlib { -static llvm::StringRef *HeaderNames; -static std::pair *SymbolNames; -static unsigned *SymbolHeaderIDs; -static llvm::DenseMap *HeaderIDs; +Lang &operator++(Lang &Language) { + Language = static_cast(static_cast(Language) + 1); + return Language; +} + // Maps symbol name -> Symbol::ID, within a namespace. using NSSymbolMap = llvm::DenseMap; -static llvm::DenseMap *NamespaceSymbols; -static int initialize() { +struct SymbolHeaderMapping { + llvm::StringRef *HeaderNames; + std::pair *SymbolNames; + unsigned *SymbolHeaderIDs; + llvm::DenseMap HeaderIDs; + llvm::DenseMap NamespaceSymbols; +}; + +static SymbolHeaderMapping +*LanguageMappings[static_cast(Lang::LastValue) + 1]; + +static int countSymbols(Lang Language) { unsigned SymCount = 0; #define SYMBOL(Name, NS, Header) ++SymCount; + switch (Language) { + case Lang::C: #include "clang/Tooling/Inclusions/CSymbolMap.inc" +break; + case Lang::CXX: #include "clang/Tooling/Inclusions/StdSymbolMap.inc" +break; + } #undef SYMBOL - SymbolNames = new std::remove_reference_t[SymCount]; - SymbolHeaderIDs = - new std::remove_reference_t[SymCount]; - NamespaceSymbols = new std::remove_reference_t; - HeaderIDs = new std::remove_reference_t; + return SymCount; +} + +static void initialize(Lang Language) { + SymbolHeaderMapping *Mapping = new SymbolHeaderMapping(); + LanguageMappings[static_cast(Language)] = Mapping; + + unsigned SymCount = countSymbols(Language); + Mapping->SymbolNames = + new std::remove_reference_tSymbolNames)>[SymCount]; + Mapping->SymbolHeaderIDs = new std::remove_reference_t< + decltype(*Mapping->SymbolHeaderIDs)>[SymCount]; auto AddNS = [&](llvm::StringRef NS) -> NSSymbolMap & { -auto R = NamespaceSymbols->try_emplace(NS, nullptr); +auto R = Mapping->NamespaceSymbols.try_emplace(NS, nullptr); if (R.second) R.first->second = new NSSymbolMap(); return *R.first->second; }; auto Add
[PATCH] D142992: [include-mapping] Implement language separation in stdlib recognizer library
VitaNuo updated this revision to Diff 494332. VitaNuo added a comment. Remove extra include. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142992/new/ https://reviews.llvm.org/D142992 Files: clang/include/clang/Tooling/Inclusions/StandardLibrary.h clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp clang/unittests/Tooling/StandardLibraryTest.cpp Index: clang/unittests/Tooling/StandardLibraryTest.cpp === --- clang/unittests/Tooling/StandardLibraryTest.cpp +++ clang/unittests/Tooling/StandardLibraryTest.cpp @@ -38,14 +38,28 @@ EXPECT_EQ(llvm::to_string(*VectorH), ""); EXPECT_FALSE(stdlib::Header::named("HeadersTests.cpp")); + EXPECT_TRUE(stdlib::Header::named("", stdlib::Lang::CXX)); + EXPECT_FALSE(stdlib::Header::named("", stdlib::Lang::C)); + auto Vector = stdlib::Symbol::named("std::", "vector"); EXPECT_TRUE(Vector); EXPECT_EQ(llvm::to_string(*Vector), "std::vector"); EXPECT_FALSE(stdlib::Symbol::named("std::", "dongle")); EXPECT_FALSE(stdlib::Symbol::named("clang::", "ASTContext")); + EXPECT_TRUE(stdlib::Symbol::named("std::", "vector", stdlib::Lang::CXX)); + EXPECT_FALSE(stdlib::Symbol::named("std::", "vector", stdlib::Lang::C)); + EXPECT_EQ(Vector->header(), *VectorH); EXPECT_THAT(Vector->headers(), ElementsAre(*VectorH)); + + EXPECT_FALSE(stdlib::Header::named("")); + EXPECT_FALSE(stdlib::Header::named("", stdlib::Lang::CXX)); + EXPECT_TRUE(stdlib::Header::named("", stdlib::Lang::C)); + + EXPECT_FALSE(stdlib::Symbol::named("", "int16_t")); + EXPECT_FALSE(stdlib::Symbol::named("", "int16_t", stdlib::Lang::CXX)); + EXPECT_TRUE(stdlib::Symbol::named("", "int16_t", stdlib::Lang::C)); } TEST(StdlibTest, Recognizer) { @@ -96,10 +110,14 @@ EXPECT_EQ(Recognizer(&VectorNonstd), std::nullopt); EXPECT_EQ(Recognizer(Vec), stdlib::Symbol::named("std::", "vector")); + EXPECT_EQ(Recognizer(Vec), +stdlib::Symbol::named("std::", "vector", stdlib::Lang::CXX)); EXPECT_EQ(Recognizer(Nest), stdlib::Symbol::named("std::", "vector")); EXPECT_EQ(Recognizer(Clock), stdlib::Symbol::named("std::chrono::", "system_clock")); EXPECT_EQ(Recognizer(CDivT), stdlib::Symbol::named("", "div_t")); + EXPECT_EQ(Recognizer(CDivT), +stdlib::Symbol::named("", "div_t", stdlib::Lang::C)); EXPECT_EQ(Recognizer(Sec), std::nullopt); } Index: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp === --- clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp +++ clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp @@ -8,6 +8,7 @@ #include "clang/Tooling/Inclusions/StandardLibrary.h" #include "clang/AST/Decl.h" +#include "clang/Basic/LangOptions.h" #include "llvm/ADT/StringRef.h" #include "llvm/Support/Casting.h" @@ -15,35 +16,60 @@ namespace tooling { namespace stdlib { -static llvm::StringRef *HeaderNames; -static std::pair *SymbolNames; -static unsigned *SymbolHeaderIDs; -static llvm::DenseMap *HeaderIDs; +Lang &operator++(Lang &Language) { + Language = static_cast(static_cast(Language) + 1); + return Language; +} + // Maps symbol name -> Symbol::ID, within a namespace. using NSSymbolMap = llvm::DenseMap; -static llvm::DenseMap *NamespaceSymbols; -static int initialize() { +struct SymbolHeaderMapping { + llvm::StringRef *HeaderNames; + std::pair *SymbolNames; + unsigned *SymbolHeaderIDs; + llvm::DenseMap HeaderIDs; + llvm::DenseMap NamespaceSymbols; +}; + +static SymbolHeaderMapping +*LanguageMappings[static_cast(Lang::LastValue) + 1]; + +static int countSymbols(Lang Language) { unsigned SymCount = 0; #define SYMBOL(Name, NS, Header) ++SymCount; + switch (Language) { + case Lang::C: #include "clang/Tooling/Inclusions/CSymbolMap.inc" +break; + case Lang::CXX: #include "clang/Tooling/Inclusions/StdSymbolMap.inc" +break; + } #undef SYMBOL - SymbolNames = new std::remove_reference_t[SymCount]; - SymbolHeaderIDs = - new std::remove_reference_t[SymCount]; - NamespaceSymbols = new std::remove_reference_t; - HeaderIDs = new std::remove_reference_t; + return SymCount; +} + +static void initialize(Lang Language) { + SymbolHeaderMapping *Mapping = new SymbolHeaderMapping(); + LanguageMappings[static_cast(Language)] = Mapping; + + unsigned SymCount = countSymbols(Language); + Mapping->SymbolNames = + new std::remove_reference_tSymbolNames)>[SymCount]; + Mapping->SymbolHeaderIDs = new std::remove_reference_t< + decltype(*Mapping->SymbolHeaderIDs)>[SymCount]; auto AddNS = [&](llvm::StringRef NS) -> NSSymbolMap & { -auto R = NamespaceSymbols->try_emplace(NS, nullptr); +auto R = Mapping->NamespaceSymbols.try_emplace(NS, nullptr); if (R.second) R.first->second = new NSSymbolMap(); return *R.first->second; }; auto AddHeader = [&](llvm::StringRef Header) -> uns
[PATCH] D142992: [include-mapping] Implement language separation in stdlib recognizer library
VitaNuo added a comment. Thanks for the comments! Comment at: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp:31 + +static llvm::DenseMap LanguageMappings; + hokein wrote: > VitaNuo wrote: > > hokein wrote: > > > using a map here seems like an overkill, we have just 2 elements, I'd > > > just use two separate variables (`CMapping`, and `CXXMapping`) > > what about the design idea that we might potentially want to extend this to > > multiple standards etc.? The idea is that it's extensible to `ObjC`, > > `OpenCL`... and so on and so forth, as has been discussed offline. > I think having a generic `Lang` enum structure is sufficient for the future > extension, and I don't think we're going to add other languages in the > foreseeable future (that's why I value the simple implementation at the > beginning). > > But you're right, getting the implementation right is probably a good idea. > I'd like to remove the DenseMap, just use a raw array, something like below > should work > > ``` > enum Lang { >C = 0, >CXX, > >LastValue = CXX, > }; > > // access by e.g. LanguageMappings[static_cast(Lang::C)]. > static SymbolHeaderMapping* > LanguageMappings[static_cast(Lang::LastValue) + 1]; > > ``` Ok, this will need some casting, but I don't have a strong opinion. Comment at: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp:188 + Lang RecognizerLang = Lang::CXX; + if (Language == clang::Language::C) { +RecognizerLang = Lang::C; hokein wrote: > VitaNuo wrote: > > hokein wrote: > > > nit: just use `LangOpts.CPlusPlus` to check the language. > > There's `LangStandard::isCPlusPlus` method that I've just discovered. > > That's probably even better. > sorry if I was not clearer -- there is a bit `CPlusPlus` in the `LangOptions` > which already does the equivalent thing (see > https://github.com/llvm/llvm-project/blob/main/clang/lib/Basic/LangOptions.cpp#L107). > > So we can just use `if (D->getLangOpts().CPlusPlus)`. Ah ok, thanks. Comment at: clang/unittests/Tooling/StandardLibraryTest.cpp:115 + + EXPECT_EQ(std::nullopt, stdlib::Symbol::named("", "int16_t")); + EXPECT_EQ(std::nullopt, hokein wrote: > nit: just `EXPECT_FALSE(stdlib::Symbol::named("", "int16_t")))`, following > the existing pattern L46. Right, thanks. Comment at: clang/unittests/Tooling/StandardLibraryTest.cpp:122 + +TEST(StdlibTest, LanguageSeparationForHeaders) { + EXPECT_NE(std::nullopt, stdlib::Header::named("vector")); hokein wrote: > The existing `TEST(StdlibTest, All)` test seems like a good umbrella for the > this test and the above test, how about moving them to the All test? Ok, I've moved everything to existing test cases. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142992/new/ https://reviews.llvm.org/D142992 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D141509: [include-mapping] Fix parsing of html_book_20190607.zip (https://en.cppreference.com/w/File:html_book_20190607.zip). Skip entries that have been added to the index (C++20 symbols), bu
VitaNuo updated this revision to Diff 488539. VitaNuo added a comment. Address review comments. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D141509/new/ https://reviews.llvm.org/D141509 Files: clang/tools/include-mapping/cppreference_parser.py Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -142,7 +142,8 @@ if variant and variant not in variants_for_symbol: continue path = os.path.join(root_dir, symbol_page_path) - results.append((symbol_name, + if os.path.isfile(path): +results.append((symbol_name, pool.apply_async(_ReadSymbolPage, (path, symbol_name # Build map from symbol name to a set of headers. Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -142,7 +142,8 @@ if variant and variant not in variants_for_symbol: continue path = os.path.join(root_dir, symbol_page_path) - results.append((symbol_name, + if os.path.isfile(path): +results.append((symbol_name, pool.apply_async(_ReadSymbolPage, (path, symbol_name # Build map from symbol name to a set of headers. ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D141509: [include-mapping] Fix parsing of html_book_20190607.zip (https://en.cppreference.com/w/File:html_book_20190607.zip). Skip entries that have been added to the index (C++20 symbols), bu
VitaNuo marked an inline comment as done. VitaNuo added a comment. Thanks for the review! Comment at: clang/tools/include-mapping/cppreference_parser.py:117 def _ReadSymbolPage(path, name): - with open(path) as f: -return _ParseSymbolPage(f.read(), name) + try: +f = open(path) hokein wrote: > so the input path can be invalid, pointing to a non-existing file, I'd just > check it in the caller side, see my other comment. good idea! Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D141509/new/ https://reviews.llvm.org/D141509 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D141509: [include-mapping] Fix parsing of html_book_20190607.zip (https://en.cppreference.com/w/File:html_book_20190607.zip). Skip entries that have been added to the index (C++20 symbols), bu
This revision was landed with ongoing or failed builds. This revision was automatically updated to reflect the committed changes. Closed by commit rG301123c7a8b5: [include-mapping] Fix parsing of html_book_20190607.zip (https://en. (authored by VitaNuo). Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D141509/new/ https://reviews.llvm.org/D141509 Files: clang/tools/include-mapping/cppreference_parser.py Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -142,7 +142,8 @@ if variant and variant not in variants_for_symbol: continue path = os.path.join(root_dir, symbol_page_path) - results.append((symbol_name, + if os.path.isfile(path): +results.append((symbol_name, pool.apply_async(_ReadSymbolPage, (path, symbol_name # Build map from symbol name to a set of headers. Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -142,7 +142,8 @@ if variant and variant not in variants_for_symbol: continue path = os.path.join(root_dir, symbol_page_path) - results.append((symbol_name, + if os.path.isfile(path): +results.append((symbol_name, pool.apply_async(_ReadSymbolPage, (path, symbol_name # Build map from symbol name to a set of headers. ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D141586: [include-mapping] Replace the stdlib symbol maps with the newest official version from https://en.cppreference.com/w/File:html_book_20190607.zip.
VitaNuo created this revision. VitaNuo added a reviewer: hokein. Herald added a project: All. VitaNuo requested review of this revision. Herald added a project: clang. Herald added a subscriber: cfe-commits. Repository: rG LLVM Github Monorepo https://reviews.llvm.org/D141586 Files: clang/include/clang/Tooling/Inclusions/CSymbolMap.inc clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc Index: clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc === --- clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc +++ clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc @@ -6,7 +6,7 @@ // This file was generated automatically by // clang/tools/include-mapping/gen_std.py, DO NOT EDIT! // -// Generated from cppreference offline HTML book (modified on 2018-10-28). +// Generated from cppreference offline HTML book (modified on 2019-06-07). //===--===// SYMBOL(Assignable, std::, ) @@ -44,7 +44,11 @@ SYMBOL(_Exit, std::, ) SYMBOL(accumulate, std::, ) SYMBOL(acos, std::, ) +SYMBOL(acosf, std::, ) SYMBOL(acosh, std::, ) +SYMBOL(acoshf, std::, ) +SYMBOL(acoshl, std::, ) +SYMBOL(acosl, std::, ) SYMBOL(add_const, std::, ) SYMBOL(add_const_t, std::, ) SYMBOL(add_cv, std::, ) @@ -74,6 +78,7 @@ SYMBOL(alignment_of_v, std::, ) SYMBOL(all_of, std::, ) SYMBOL(allocate_shared, std::, ) +SYMBOL(allocate_shared_default_init, std::, ) SYMBOL(allocator, std::, ) SYMBOL(allocator_arg, std::, ) SYMBOL(allocator_arg_t, std::, ) @@ -86,12 +91,23 @@ SYMBOL(as_const, std::, ) SYMBOL(asctime, std::, ) SYMBOL(asin, std::, ) +SYMBOL(asinf, std::, ) SYMBOL(asinh, std::, ) +SYMBOL(asinhf, std::, ) +SYMBOL(asinhl, std::, ) +SYMBOL(asinl, std::, ) +SYMBOL(assume_aligned, std::, ) SYMBOL(async, std::, ) SYMBOL(at_quick_exit, std::, ) SYMBOL(atan, std::, ) SYMBOL(atan2, std::, ) +SYMBOL(atan2f, std::, ) +SYMBOL(atan2l, std::, ) +SYMBOL(atanf, std::, ) SYMBOL(atanh, std::, ) +SYMBOL(atanhf, std::, ) +SYMBOL(atanhl, std::, ) +SYMBOL(atanl, std::, ) SYMBOL(atexit, std::, ) SYMBOL(atof, std::, ) SYMBOL(atoi, std::, ) @@ -128,7 +144,6 @@ SYMBOL(atomic_store_explicit, std::, ) SYMBOL(atomic_thread_fence, std::, ) SYMBOL(atto, std::, ) -SYMBOL(auto_ptr, std::, ) SYMBOL(back_insert_iterator, std::, ) SYMBOL(back_inserter, std::, ) SYMBOL(bad_alloc, std::, ) @@ -165,6 +180,7 @@ SYMBOL(bidirectional_iterator_tag, std::, ) SYMBOL(binary_search, std::, ) SYMBOL(bind, std::, ) +SYMBOL(bind_front, std::, ) SYMBOL(binomial_distribution, std::, ) SYMBOL(bit_and, std::, ) SYMBOL(bit_cast, std::, ) @@ -181,13 +197,18 @@ SYMBOL(byte, std::, ) SYMBOL(c16rtomb, std::, ) SYMBOL(c32rtomb, std::, ) +SYMBOL(c8rtomb, std::, ) SYMBOL(call_once, std::, ) SYMBOL(calloc, std::, ) SYMBOL(cauchy_distribution, std::, ) SYMBOL(cbegin, std::, ) SYMBOL(cbrt, std::, ) +SYMBOL(cbrtf, std::, ) +SYMBOL(cbrtl, std::, ) SYMBOL(ceil, std::, ) SYMBOL(ceil2, std::, ) +SYMBOL(ceilf, std::, ) +SYMBOL(ceill, std::, ) SYMBOL(cend, std::, ) SYMBOL(centi, std::, ) SYMBOL(cerr, std::, ) @@ -226,14 +247,21 @@ SYMBOL(conjunction, std::, ) SYMBOL(conjunction_v, std::, ) SYMBOL(const_pointer_cast, std::, ) +SYMBOL(contiguous_iterator_tag, std::, ) SYMBOL(contract_violation, std::, ) SYMBOL(copy, std::, ) SYMBOL(copy_backward, std::, ) SYMBOL(copy_if, std::, ) SYMBOL(copy_n, std::, ) SYMBOL(copysign, std::, ) +SYMBOL(copysignf, std::, ) +SYMBOL(copysignl, std::, ) SYMBOL(cos, std::, ) +SYMBOL(cosf, std::, ) SYMBOL(cosh, std::, ) +SYMBOL(coshf, std::, ) +SYMBOL(coshl, std::, ) +SYMBOL(cosl, std::, ) SYMBOL(count, std::, ) SYMBOL(count_if, std::, ) SYMBOL(cout, std::, ) @@ -297,8 +325,14 @@ SYMBOL(equal, std::, ) SYMBOL(equal_range, std::, ) SYMBOL(equal_to, std::, ) +SYMBOL(erase, std::, ) +SYMBOL(erase_if, std::, ) SYMBOL(erf, std::, ) SYMBOL(erfc, std::, ) +SYMBOL(erfcf, std::, ) +SYMBOL(erfcl, std::, ) +SYMBOL(erff, std::, ) +SYMBOL(erfl, std::, ) SYMBOL(errc, std::, ) SYMBOL(error_category, std::, ) SYMBOL(error_code, std::, ) @@ -311,14 +345,25 @@ SYMBOL(exit, std::, ) SYMBOL(exp, std::, ) SYMBOL(exp2, std::, ) +SYMBOL(exp2f, std::, ) +SYMBOL(exp2l, std::, ) +SYMBOL(expf, std::, ) +SYMBOL(expl, std::, ) SYMBOL(expm1, std::, ) +SYMBOL(expm1f, std::, ) +SYMBOL(expm1l, std::, ) SYMBOL(exponential_distribution, std::, ) SYMBOL(extent, std::, ) SYMBOL(extent_v, std::, ) SYMBOL(extreme_value_distribution, std::, ) +SYMBOL(fabs, std::, ) +SYMBOL(fabsf, std::, ) +SYMBOL(fabsl, std::, ) SYMBOL(false_type, std::, ) SYMBOL(fclose, std::, ) SYMBOL(fdim, std::, ) +SYMBOL(fdimf, std::, ) +SYMBOL(fdiml, std::, ) SYMBOL(feclearexcept, std::, ) SYMBOL(fegetenv, std::, ) SYMBOL(fegetexceptflag, std::, ) @@ -356,12 +401,22 @@ SYMBOL(float_t, std::, ) SYMBOL(floor, std::, ) SYMBOL(floor2, std::, ) +SYMBOL(floorf, std::, ) +SYMBOL(floorl, std::, ) SYMBOL(flush, std::, ) SYMBO
[PATCH] D141611: [include-mapping] Print an error message in case the symbol index points to a non-existent page.
VitaNuo created this revision. Herald added a subscriber: arphaman. Herald added a project: All. VitaNuo requested review of this revision. Herald added a project: clang. Herald added a subscriber: cfe-commits. Repository: rG LLVM Github Monorepo https://reviews.llvm.org/D141611 Files: clang/tools/include-mapping/cppreference_parser.py Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -145,6 +145,8 @@ if os.path.isfile(path): results.append((symbol_name, pool.apply_async(_ReadSymbolPage, (path, symbol_name + else: +print("Could not parse page:", path, ". Page does not exist.", file=sys.stderr) # Build map from symbol name to a set of headers. symbol_headers = collections.defaultdict(set) Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -145,6 +145,8 @@ if os.path.isfile(path): results.append((symbol_name, pool.apply_async(_ReadSymbolPage, (path, symbol_name + else: +print("Could not parse page:", path, ". Page does not exist.", file=sys.stderr) # Build map from symbol name to a set of headers. symbol_headers = collections.defaultdict(set) ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D141611: [include-mapping] Print an error message in case the symbol index points to a non-existent page.
VitaNuo updated this revision to Diff 488653. VitaNuo added a comment. Change implementation from using "print" to using "sys.stderr.write". Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D141611/new/ https://reviews.llvm.org/D141611 Files: clang/tools/include-mapping/cppreference_parser.py Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -145,6 +145,8 @@ if os.path.isfile(path): results.append((symbol_name, pool.apply_async(_ReadSymbolPage, (path, symbol_name + else: +sys.stderr.write("Could not parse page: %s. Page does not exist.\n" % path) # Build map from symbol name to a set of headers. symbol_headers = collections.defaultdict(set) Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -145,6 +145,8 @@ if os.path.isfile(path): results.append((symbol_name, pool.apply_async(_ReadSymbolPage, (path, symbol_name + else: +sys.stderr.write("Could not parse page: %s. Page does not exist.\n" % path) # Build map from symbol name to a set of headers. symbol_headers = collections.defaultdict(set) ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D141509: [include-mapping] Fix parsing of html_book_20190607.zip (https://en.cppreference.com/w/File:html_book_20190607.zip). Skip entries that have been added to the index (C++20 symbols), bu
VitaNuo added inline comments. Comment at: clang/tools/include-mapping/cppreference_parser.py:145 path = os.path.join(root_dir, symbol_page_path) - results.append((symbol_name, + if os.path.isfile(path): +results.append((symbol_name, kadircet wrote: > sorry for being late to the party, but can you also print an error in the > case we're dropping the symbol? https://reviews.llvm.org/D141611 Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D141509/new/ https://reviews.llvm.org/D141509 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D141611: [include-mapping] Print an error message in case the symbol index points to a non-existent page.
VitaNuo updated this revision to Diff 488659. VitaNuo added a comment. Mention symbol name in the error message. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D141611/new/ https://reviews.llvm.org/D141611 Files: clang/tools/include-mapping/cppreference_parser.py Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -145,6 +145,9 @@ if os.path.isfile(path): results.append((symbol_name, pool.apply_async(_ReadSymbolPage, (path, symbol_name + else: +sys.stderr.write("Discarding information for symbol: %s. Page %s does not exist.\n" + % (symbol_name, path)) # Build map from symbol name to a set of headers. symbol_headers = collections.defaultdict(set) Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -145,6 +145,9 @@ if os.path.isfile(path): results.append((symbol_name, pool.apply_async(_ReadSymbolPage, (path, symbol_name + else: +sys.stderr.write("Discarding information for symbol: %s. Page %s does not exist.\n" + % (symbol_name, path)) # Build map from symbol name to a set of headers. symbol_headers = collections.defaultdict(set) ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D141611: [include-mapping] Print an error message in case the symbol index points to a non-existent page.
This revision was landed with ongoing or failed builds. This revision was automatically updated to reflect the committed changes. Closed by commit rGed001018a02b: [include-mapping] Print an error message in case the symbol index points to a… (authored by VitaNuo). Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D141611/new/ https://reviews.llvm.org/D141611 Files: clang/tools/include-mapping/cppreference_parser.py Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -145,6 +145,9 @@ if os.path.isfile(path): results.append((symbol_name, pool.apply_async(_ReadSymbolPage, (path, symbol_name + else: +sys.stderr.write("Discarding information for symbol: %s. Page %s does not exist.\n" + % (symbol_name, path)) # Build map from symbol name to a set of headers. symbol_headers = collections.defaultdict(set) Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -145,6 +145,9 @@ if os.path.isfile(path): results.append((symbol_name, pool.apply_async(_ReadSymbolPage, (path, symbol_name + else: +sys.stderr.write("Discarding information for symbol: %s. Page %s does not exist.\n" + % (symbol_name, path)) # Build map from symbol name to a set of headers. symbol_headers = collections.defaultdict(set) ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D141855: [include-mapping] Parse zombie_names.html into a removed symbols map.
VitaNuo created this revision. VitaNuo added a reviewer: hokein. Herald added a project: All. VitaNuo requested review of this revision. Herald added a project: clang. Herald added a subscriber: cfe-commits. Repository: rG LLVM Github Monorepo https://reviews.llvm.org/D141855 Files: clang/tools/include-mapping/gen_std.py Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -64,6 +64,14 @@ default='cpp', help='Generate c or cpp symbols', required=True) + parser.add_argument('-output', + default='SymbolMap.inc', + help='path to the output file for symbol map', + required=True) + parser.add_argument('-removedoutput', + default='RemovedSymbolMap.inc', + help='path to the output file for removed symbols map', + required=False) return parser.parse_args() @@ -87,6 +95,9 @@ (symbol_index_root, "regex_constants.html", "std::regex_constants::"), (symbol_index_root, "this_thread.html", "std::this_thread::"), ] +removed_symbols = cppreference_parser.GetSymbols( + [(symbol_index_root, "zombie_names.html", "std::")]) +WriteSymbols(args.language, removed_symbols, args.removedoutput, page_root) elif args.language == 'c': page_root = os.path.join(args.cppreference, "en", "c") symbol_index_root = page_root @@ -96,24 +107,28 @@ exit("Path %s doesn't exist!" % symbol_index_root) symbols = cppreference_parser.GetSymbols(parse_pages) + WriteSymbols(args.language, symbols, args.output, page_root) +def WriteSymbols(language, symbols, path, page_root): # We don't have version information from the unzipped offline HTML files. # so we use the modified time of the symbol_index.html as the version. index_page_path = os.path.join(page_root, "index.html") cppreference_modified_date = datetime.datetime.fromtimestamp( os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') - print(CODE_PREFIX % (args.language.upper(), cppreference_modified_date)) - for symbol in symbols: -if len(symbol.headers) == 1: - # SYMBOL(unqualified_name, namespace, header) - print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, -symbol.headers[0])) -elif len(symbol.headers) == 0: - sys.stderr.write("No header found for symbol %s\n" % symbol.name) -else: - # FIXME: support symbols with multiple headers (e.g. std::move). - sys.stderr.write("Ambiguous header for symbol %s: %s\n" % ( - symbol.name, ', '.join(symbol.headers))) + with open(path, "w") as f: +f.write(CODE_PREFIX % (language.upper(), cppreference_modified_date)) +f.write("\n") +for symbol in symbols: + if len(symbol.headers) == 1: +# SYMBOL(unqualified_name, namespace, header) +f.write("SYMBOL(%s, %s, %s)\n" % (symbol.name, symbol.namespace, + symbol.headers[0])) + elif len(symbol.headers) == 0: +sys.stderr.write("No header found for symbol %s\n" % symbol.name) + else: +# FIXME: support symbols with multiple headers (e.g. std::move). +sys.stderr.write("Ambiguous header for symbol %s: %s\n" % ( +symbol.name, ', '.join(symbol.headers))) if __name__ == '__main__': Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -64,6 +64,14 @@ default='cpp', help='Generate c or cpp symbols', required=True) + parser.add_argument('-output', + default='SymbolMap.inc', + help='path to the output file for symbol map', + required=True) + parser.add_argument('-removedoutput', + default='RemovedSymbolMap.inc', + help='path to the output file for removed symbols map', + required=False) return parser.parse_args() @@ -87,6 +95,9 @@ (symbol_index_root, "regex_constants.html", "std::regex_constants::"), (symbol_index_root, "this_thread.html", "std::this_thread::"), ] +removed_symbols = cppreference_parser.GetSymbols( + [(symbol_index_root, "zombie_names.html", "std::")]) +WriteSymbols(args.language, removed_symbols, args.removedoutput, page_root) elif args.language == 'c': page_root = os.path.join(args.cppreference, "en", "c") symbol_index_root = page_root @@ -96,24 +107,28 @@ exit("Path %s doesn't exist!" % symbol_index_root) symbols = cppreference_p
[PATCH] D141586: [include-mapping] Replace the stdlib symbol maps with the newest official version from https://en.cppreference.com/w/File:html_book_20190607.zip.
VitaNuo abandoned this revision. VitaNuo added a comment. Abandoning since we will eventually parse a newer version of the archive. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D141586/new/ https://reviews.llvm.org/D141586 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D141855: [include-mapping] Parse zombie_names.html into a removed symbols map.
VitaNuo updated this revision to Diff 489824. VitaNuo added a comment. Address review comments. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D141855/new/ https://reviews.llvm.org/D141855 Files: clang/tools/include-mapping/gen_std.py Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -60,16 +60,16 @@ help='path to the cppreference offline HTML directory', required=True ) - parser.add_argument('-language', - default='cpp', - help='Generate c or cpp symbols', - required=True) + parser.add_argument('-symbols', + default='cpp_symbols', + help='Generate c or cpp (removed) symbols. One of {cpp_symbols, c_symbols, cpp_removed_symbols}.', + required=True) return parser.parse_args() def main(): args = ParseArg() - if args.language == 'cpp': + if args.symbols == 'cpp': page_root = os.path.join(args.cppreference, "en", "cpp") symbol_index_root = os.path.join(page_root, "symbol_index") parse_pages = [ @@ -87,22 +87,26 @@ (symbol_index_root, "regex_constants.html", "std::regex_constants::"), (symbol_index_root, "this_thread.html", "std::this_thread::"), ] - elif args.language == 'c': + elif args.symbols == 'c': page_root = os.path.join(args.cppreference, "en", "c") symbol_index_root = page_root parse_pages = [(page_root, "index.html", None)] - + elif args.symbols == 'cpp_removed_symbols': +page_root = os.path.join(args.cppreference, "en", "cpp") +symbol_index_root = os.path.join(page_root, "symbol_index") +parse_pages = [(symbol_index_root, "zombie_names.html", "std::")] + if not os.path.exists(symbol_index_root): exit("Path %s doesn't exist!" % symbol_index_root) symbols = cppreference_parser.GetSymbols(parse_pages) - + # We don't have version information from the unzipped offline HTML files. # so we use the modified time of the symbol_index.html as the version. index_page_path = os.path.join(page_root, "index.html") cppreference_modified_date = datetime.datetime.fromtimestamp( -os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') - print(CODE_PREFIX % (args.language.upper(), cppreference_modified_date)) +os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') + print(CODE_PREFIX % (args.symbols.upper(), cppreference_modified_date)) for symbol in symbols: if len(symbol.headers) == 1: # SYMBOL(unqualified_name, namespace, header) Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -60,16 +60,16 @@ help='path to the cppreference offline HTML directory', required=True ) - parser.add_argument('-language', - default='cpp', - help='Generate c or cpp symbols', - required=True) + parser.add_argument('-symbols', + default='cpp_symbols', + help='Generate c or cpp (removed) symbols. One of {cpp_symbols, c_symbols, cpp_removed_symbols}.', + required=True) return parser.parse_args() def main(): args = ParseArg() - if args.language == 'cpp': + if args.symbols == 'cpp': page_root = os.path.join(args.cppreference, "en", "cpp") symbol_index_root = os.path.join(page_root, "symbol_index") parse_pages = [ @@ -87,22 +87,26 @@ (symbol_index_root, "regex_constants.html", "std::regex_constants::"), (symbol_index_root, "this_thread.html", "std::this_thread::"), ] - elif args.language == 'c': + elif args.symbols == 'c': page_root = os.path.join(args.cppreference, "en", "c") symbol_index_root = page_root parse_pages = [(page_root, "index.html", None)] - + elif args.symbols == 'cpp_removed_symbols': +page_root = os.path.join(args.cppreference, "en", "cpp") +symbol_index_root = os.path.join(page_root, "symbol_index") +parse_pages = [(symbol_index_root, "zombie_names.html", "std::")] + if not os.path.exists(symbol_index_root): exit("Path %s doesn't exist!" % symbol_index_root) symbols = cppreference_parser.GetSymbols(parse_pages) - + # We don't have version information from the unzipped offline HTML files. # so we use the modified time of the symbol_index.html as the version. index_page_path = os.path.join(page_root, "index.html") cppreference_modified_date = datetime.datetime.fromtimestamp( -os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') - print
[PATCH] D141855: [include-mapping] Parse zombie_names.html into a removed symbols map.
VitaNuo updated this revision to Diff 489827. VitaNuo added a comment. Remove extra whitespace. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D141855/new/ https://reviews.llvm.org/D141855 Files: clang/tools/include-mapping/gen_std.py Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -60,16 +60,16 @@ help='path to the cppreference offline HTML directory', required=True ) - parser.add_argument('-language', - default='cpp', - help='Generate c or cpp symbols', - required=True) + parser.add_argument('-symbols', + default='cpp_symbols', + help='Generate c or cpp (removed) symbols. One of {cpp_symbols, c_symbols, cpp_removed_symbols}.', + required=True) return parser.parse_args() def main(): args = ParseArg() - if args.language == 'cpp': + if args.symbols == 'cpp': page_root = os.path.join(args.cppreference, "en", "cpp") symbol_index_root = os.path.join(page_root, "symbol_index") parse_pages = [ @@ -87,22 +87,26 @@ (symbol_index_root, "regex_constants.html", "std::regex_constants::"), (symbol_index_root, "this_thread.html", "std::this_thread::"), ] - elif args.language == 'c': + elif args.symbols == 'c': page_root = os.path.join(args.cppreference, "en", "c") symbol_index_root = page_root parse_pages = [(page_root, "index.html", None)] - + elif args.symbols == 'cpp_removed_symbols': +page_root = os.path.join(args.cppreference, "en", "cpp") +symbol_index_root = os.path.join(page_root, "symbol_index") +parse_pages = [(symbol_index_root, "zombie_names.html", "std::")] + if not os.path.exists(symbol_index_root): exit("Path %s doesn't exist!" % symbol_index_root) symbols = cppreference_parser.GetSymbols(parse_pages) - + # We don't have version information from the unzipped offline HTML files. # so we use the modified time of the symbol_index.html as the version. index_page_path = os.path.join(page_root, "index.html") cppreference_modified_date = datetime.datetime.fromtimestamp( os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') - print(CODE_PREFIX % (args.language.upper(), cppreference_modified_date)) + print(CODE_PREFIX % (args.symbols.upper(), cppreference_modified_date)) for symbol in symbols: if len(symbol.headers) == 1: # SYMBOL(unqualified_name, namespace, header) Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -60,16 +60,16 @@ help='path to the cppreference offline HTML directory', required=True ) - parser.add_argument('-language', - default='cpp', - help='Generate c or cpp symbols', - required=True) + parser.add_argument('-symbols', + default='cpp_symbols', + help='Generate c or cpp (removed) symbols. One of {cpp_symbols, c_symbols, cpp_removed_symbols}.', + required=True) return parser.parse_args() def main(): args = ParseArg() - if args.language == 'cpp': + if args.symbols == 'cpp': page_root = os.path.join(args.cppreference, "en", "cpp") symbol_index_root = os.path.join(page_root, "symbol_index") parse_pages = [ @@ -87,22 +87,26 @@ (symbol_index_root, "regex_constants.html", "std::regex_constants::"), (symbol_index_root, "this_thread.html", "std::this_thread::"), ] - elif args.language == 'c': + elif args.symbols == 'c': page_root = os.path.join(args.cppreference, "en", "c") symbol_index_root = page_root parse_pages = [(page_root, "index.html", None)] - + elif args.symbols == 'cpp_removed_symbols': +page_root = os.path.join(args.cppreference, "en", "cpp") +symbol_index_root = os.path.join(page_root, "symbol_index") +parse_pages = [(symbol_index_root, "zombie_names.html", "std::")] + if not os.path.exists(symbol_index_root): exit("Path %s doesn't exist!" % symbol_index_root) symbols = cppreference_parser.GetSymbols(parse_pages) - + # We don't have version information from the unzipped offline HTML files. # so we use the modified time of the symbol_index.html as the version. index_page_path = os.path.join(page_root, "index.html") cppreference_modified_date = datetime.datetime.fromtimestamp( os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') - print(CODE_PREFIX % (args.language.upper(), cppreference_modified_date))
[PATCH] D141855: [include-mapping] Parse zombie_names.html into a removed symbols map.
VitaNuo updated this revision to Diff 489828. VitaNuo added a comment. Adjust argument value set. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D141855/new/ https://reviews.llvm.org/D141855 Files: clang/tools/include-mapping/gen_std.py Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -60,16 +60,16 @@ help='path to the cppreference offline HTML directory', required=True ) - parser.add_argument('-language', + parser.add_argument('-symbols', default='cpp', - help='Generate c or cpp symbols', - required=True) + help='Generate c or cpp (removed) symbols. One of {cpp, c, cppremoved}.', + required=True) return parser.parse_args() def main(): args = ParseArg() - if args.language == 'cpp': + if args.symbols == 'cpp': page_root = os.path.join(args.cppreference, "en", "cpp") symbol_index_root = os.path.join(page_root, "symbol_index") parse_pages = [ @@ -87,22 +87,26 @@ (symbol_index_root, "regex_constants.html", "std::regex_constants::"), (symbol_index_root, "this_thread.html", "std::this_thread::"), ] - elif args.language == 'c': + elif args.symbols == 'c': page_root = os.path.join(args.cppreference, "en", "c") symbol_index_root = page_root parse_pages = [(page_root, "index.html", None)] - + elif args.symbols == 'cppremoved': +page_root = os.path.join(args.cppreference, "en", "cpp") +symbol_index_root = os.path.join(page_root, "symbol_index") +parse_pages = [(symbol_index_root, "zombie_names.html", "std::")] + if not os.path.exists(symbol_index_root): exit("Path %s doesn't exist!" % symbol_index_root) symbols = cppreference_parser.GetSymbols(parse_pages) - + # We don't have version information from the unzipped offline HTML files. # so we use the modified time of the symbol_index.html as the version. index_page_path = os.path.join(page_root, "index.html") cppreference_modified_date = datetime.datetime.fromtimestamp( os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') - print(CODE_PREFIX % (args.language.upper(), cppreference_modified_date)) + print(CODE_PREFIX % (args.symbols.upper(), cppreference_modified_date)) for symbol in symbols: if len(symbol.headers) == 1: # SYMBOL(unqualified_name, namespace, header) Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -60,16 +60,16 @@ help='path to the cppreference offline HTML directory', required=True ) - parser.add_argument('-language', + parser.add_argument('-symbols', default='cpp', - help='Generate c or cpp symbols', - required=True) + help='Generate c or cpp (removed) symbols. One of {cpp, c, cppremoved}.', + required=True) return parser.parse_args() def main(): args = ParseArg() - if args.language == 'cpp': + if args.symbols == 'cpp': page_root = os.path.join(args.cppreference, "en", "cpp") symbol_index_root = os.path.join(page_root, "symbol_index") parse_pages = [ @@ -87,22 +87,26 @@ (symbol_index_root, "regex_constants.html", "std::regex_constants::"), (symbol_index_root, "this_thread.html", "std::this_thread::"), ] - elif args.language == 'c': + elif args.symbols == 'c': page_root = os.path.join(args.cppreference, "en", "c") symbol_index_root = page_root parse_pages = [(page_root, "index.html", None)] - + elif args.symbols == 'cppremoved': +page_root = os.path.join(args.cppreference, "en", "cpp") +symbol_index_root = os.path.join(page_root, "symbol_index") +parse_pages = [(symbol_index_root, "zombie_names.html", "std::")] + if not os.path.exists(symbol_index_root): exit("Path %s doesn't exist!" % symbol_index_root) symbols = cppreference_parser.GetSymbols(parse_pages) - + # We don't have version information from the unzipped offline HTML files. # so we use the modified time of the symbol_index.html as the version. index_page_path = os.path.join(page_root, "index.html") cppreference_modified_date = datetime.datetime.fromtimestamp( os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') - print(CODE_PREFIX % (args.language.upper(), cppreference_modified_date)) + print(CODE_PREFIX % (args.symbols.upper(), cppreference_modified_date)) for symbol in symbols: if len(symbol.headers) == 1: # SYMBOL(unqual
[PATCH] D141855: [include-mapping] Parse zombie_names.html into a removed symbols map.
VitaNuo marked an inline comment as done. VitaNuo added a comment. Yes, ofc I've run it, I wouldn't send a patch otherwise :) Comment at: clang/tools/include-mapping/gen_std.py:67 required=True) + parser.add_argument('-output', + default='SymbolMap.inc', hokein wrote: > instead adding two CLI flags, I'd suggest extending the existing `language` > flag to something like `-symbols = {cpp_symbols, c_symbols, > cpp_removed_symbols}`, it is easy to extend in the future (e.g. > `c_removed_symbols`). > Sure, good idea! Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D141855/new/ https://reviews.llvm.org/D141855 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D141944: [include-mapping] Fix gen_std.py test
VitaNuo created this revision. VitaNuo added a reviewer: hokein. Herald added a project: All. VitaNuo requested review of this revision. Herald added a project: clang. Herald added a subscriber: cfe-commits. Repository: rG LLVM Github Monorepo https://reviews.llvm.org/D141944 Files: clang/tools/include-mapping/test.py Index: clang/tools/include-mapping/test.py === --- clang/tools/include-mapping/test.py +++ clang/tools/include-mapping/test.py @@ -24,11 +24,11 @@ actual = _ParseIndexPage(html) expected = [ - ("abs", "abs.html", True), - ("abs", "complex/abs.html", True), - ("acos", "acos.html", False), - ("acosh", "acosh.html", False), - ("as_bytes", "as_bytes.html", False), + ("abs", "abs.html", 'int'), + ("abs", "complex/abs.html", 'std::complex'), + ("acos", "acos.html", None), + ("acosh", "acosh.html", None), + ("as_bytes", "as_bytes.html", None), ] self.assertEqual(len(actual), len(expected)) for i in range(0, len(actual)): Index: clang/tools/include-mapping/test.py === --- clang/tools/include-mapping/test.py +++ clang/tools/include-mapping/test.py @@ -24,11 +24,11 @@ actual = _ParseIndexPage(html) expected = [ - ("abs", "abs.html", True), - ("abs", "complex/abs.html", True), - ("acos", "acos.html", False), - ("acosh", "acosh.html", False), - ("as_bytes", "as_bytes.html", False), + ("abs", "abs.html", 'int'), + ("abs", "complex/abs.html", 'std::complex'), + ("acos", "acos.html", None), + ("acosh", "acosh.html", None), + ("as_bytes", "as_bytes.html", None), ] self.assertEqual(len(actual), len(expected)) for i in range(0, len(actual)): ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D141944: [include-mapping] Fix gen_std.py test
This revision was landed with ongoing or failed builds. This revision was automatically updated to reflect the committed changes. Closed by commit rG0aaeb25525ec: [include-mapping] Fix gen_std.py test (authored by VitaNuo). Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D141944/new/ https://reviews.llvm.org/D141944 Files: clang/tools/include-mapping/test.py Index: clang/tools/include-mapping/test.py === --- clang/tools/include-mapping/test.py +++ clang/tools/include-mapping/test.py @@ -24,11 +24,11 @@ actual = _ParseIndexPage(html) expected = [ - ("abs", "abs.html", True), - ("abs", "complex/abs.html", True), - ("acos", "acos.html", False), - ("acosh", "acosh.html", False), - ("as_bytes", "as_bytes.html", False), + ("abs", "abs.html", 'int'), + ("abs", "complex/abs.html", 'std::complex'), + ("acos", "acos.html", None), + ("acosh", "acosh.html", None), + ("as_bytes", "as_bytes.html", None), ] self.assertEqual(len(actual), len(expected)) for i in range(0, len(actual)): Index: clang/tools/include-mapping/test.py === --- clang/tools/include-mapping/test.py +++ clang/tools/include-mapping/test.py @@ -24,11 +24,11 @@ actual = _ParseIndexPage(html) expected = [ - ("abs", "abs.html", True), - ("abs", "complex/abs.html", True), - ("acos", "acos.html", False), - ("acosh", "acosh.html", False), - ("as_bytes", "as_bytes.html", False), + ("abs", "abs.html", 'int'), + ("abs", "complex/abs.html", 'std::complex'), + ("acos", "acos.html", None), + ("acosh", "acosh.html", None), + ("as_bytes", "as_bytes.html", None), ] self.assertEqual(len(actual), len(expected)) for i in range(0, len(actual)): ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D141855: [include-mapping] Parse zombie_names.html into a removed symbols map.
VitaNuo updated this revision to Diff 490117. VitaNuo marked an inline comment as done. VitaNuo added a comment. Minor adjustments in accordance with review comments. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D141855/new/ https://reviews.llvm.org/D141855 Files: clang/tools/include-mapping/gen_std.py Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -28,9 +28,11 @@ get a "cppreference/reference" directory. 4. Run the command: // Generate C++ symbols - python3 gen_std.py -cppreference cppreference/reference -language=cpp > StdSymbolMap.inc + python3 gen_std.py -cppreference cppreference/reference -symbols=cpp > StdSymbolMap.inc + // Generate C++ removed symbols + python3 gen_std.py -cppreference cppreference/reference -symbols=cpp_removed > RemovedSymbolMap.inc // Generate C symbols - python3 gen_std.py -cppreference cppreference/reference -language=c > CSymbolMap.inc + python3 gen_std.py -cppreference cppreference/reference -symbols=c > CSymbolMap.inc """ @@ -60,16 +62,16 @@ help='path to the cppreference offline HTML directory', required=True ) - parser.add_argument('-language', + parser.add_argument('-symbols', default='cpp', - help='Generate c or cpp symbols', - required=True) + help='Generate c or cpp (removed) symbols. One of {cpp, c, cpp_removed}.', + required=True) return parser.parse_args() def main(): args = ParseArg() - if args.language == 'cpp': + if args.symbols == 'cpp': page_root = os.path.join(args.cppreference, "en", "cpp") symbol_index_root = os.path.join(page_root, "symbol_index") parse_pages = [ @@ -87,22 +89,26 @@ (symbol_index_root, "regex_constants.html", "std::regex_constants::"), (symbol_index_root, "this_thread.html", "std::this_thread::"), ] - elif args.language == 'c': + elif args.symbols == 'cpp_removed': +page_root = os.path.join(args.cppreference, "en", "cpp") +symbol_index_root = os.path.join(page_root, "symbol_index") +parse_pages = [(symbol_index_root, "zombie_names.html", "std::")] + elif args.symbols == 'c': page_root = os.path.join(args.cppreference, "en", "c") symbol_index_root = page_root -parse_pages = [(page_root, "index.html", None)] - +parse_pages = [(page_root, "index.html", None)] + if not os.path.exists(symbol_index_root): exit("Path %s doesn't exist!" % symbol_index_root) symbols = cppreference_parser.GetSymbols(parse_pages) - + # We don't have version information from the unzipped offline HTML files. # so we use the modified time of the symbol_index.html as the version. index_page_path = os.path.join(page_root, "index.html") cppreference_modified_date = datetime.datetime.fromtimestamp( os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') - print(CODE_PREFIX % (args.language.upper(), cppreference_modified_date)) + print(CODE_PREFIX % (args.symbols.upper(), cppreference_modified_date)) for symbol in symbols: if len(symbol.headers) == 1: # SYMBOL(unqualified_name, namespace, header) Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -28,9 +28,11 @@ get a "cppreference/reference" directory. 4. Run the command: // Generate C++ symbols - python3 gen_std.py -cppreference cppreference/reference -language=cpp > StdSymbolMap.inc + python3 gen_std.py -cppreference cppreference/reference -symbols=cpp > StdSymbolMap.inc + // Generate C++ removed symbols + python3 gen_std.py -cppreference cppreference/reference -symbols=cpp_removed > RemovedSymbolMap.inc // Generate C symbols - python3 gen_std.py -cppreference cppreference/reference -language=c > CSymbolMap.inc + python3 gen_std.py -cppreference cppreference/reference -symbols=c > CSymbolMap.inc """ @@ -60,16 +62,16 @@ help='path to the cppreference offline HTML directory', required=True ) - parser.add_argument('-language', + parser.add_argument('-symbols', default='cpp', - help='Generate c or cpp symbols', - required=True) + help='Generate c or cpp (removed) symbols. One of {cpp, c, cpp_removed}.', + required=True) return parser.parse_args() def main(): args = ParseArg() - if args.language == 'cpp': + if args.symbols == 'cpp':
[PATCH] D141855: [include-mapping] Parse zombie_names.html into a removed symbols map.
VitaNuo added a comment. Thanks for the comments! Comment at: clang/tools/include-mapping/gen_std.py:31 // Generate C++ symbols python3 gen_std.py -cppreference cppreference/reference -language=cpp > StdSymbolMap.inc // Generate C symbols hokein wrote: > nit: update the doc. Ah thanks for the reminder! Comment at: clang/tools/include-mapping/gen_std.py:94 parse_pages = [(page_root, "index.html", None)] - + elif args.symbols == 'cppremoved': +page_root = os.path.join(args.cppreference, "en", "cpp") hokein wrote: > nit: can we add a `_` between `cpp` and `removed`. And maybe move it after > the if `args.symbols == 'cpp'` branch as they are `cpp` related. Ok, sure. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D141855/new/ https://reviews.llvm.org/D141855 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo created this revision. Herald added a project: All. VitaNuo requested review of this revision. Herald added a project: clang. Herald added a subscriber: cfe-commits. Repository: rG LLVM Github Monorepo https://reviews.llvm.org/D142092 Files: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp clang/tools/include-mapping/gen_std.py Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -104,17 +104,12 @@ os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') print(CODE_PREFIX % (args.language.upper(), cppreference_modified_date)) for symbol in symbols: -if len(symbol.headers) == 1: - # SYMBOL(unqualified_name, namespace, header) - print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, -symbol.headers[0])) -elif len(symbol.headers) == 0: +if len(symbol.headers) == 0: sys.stderr.write("No header found for symbol %s\n" % symbol.name) else: - # FIXME: support symbols with multiple headers (e.g. std::move). - sys.stderr.write("Ambiguous header for symbol %s: %s\n" % ( - symbol.name, ', '.join(symbol.headers))) - - + for header in symbol.headers: +# SYMBOL(unqualified_name, namespace, header) +print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, + header)) if __name__ == '__main__': main() Index: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp === --- clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp +++ clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp @@ -8,6 +8,7 @@ #include "clang/Tooling/Inclusions/StandardLibrary.h" #include "clang/AST/Decl.h" +#include "llvm/ADT/DenseMap.h" #include "llvm/ADT/StringRef.h" #include "llvm/Support/Casting.h" @@ -16,8 +17,9 @@ namespace stdlib { static llvm::StringRef *HeaderNames; -static std::pair *SymbolNames; -static unsigned *SymbolHeaderIDs; +static llvm::DenseMap> +*SymbolNames; +static llvm::SmallVector *SymbolHeaderIDs; static llvm::DenseMap *HeaderIDs; // Maps symbol name -> Symbol::ID, within a namespace. using NSSymbolMap = llvm::DenseMap; @@ -29,7 +31,7 @@ #include "clang/Tooling/Inclusions/CSymbolMap.inc" #include "clang/Tooling/Inclusions/StdSymbolMap.inc" #undef SYMBOL - SymbolNames = new std::remove_reference_t[SymCount]; + SymbolNames = new std::remove_reference_t; SymbolHeaderIDs = new std::remove_reference_t[SymCount]; NamespaceSymbols = new std::remove_reference_t; @@ -46,18 +48,31 @@ return HeaderIDs->try_emplace(Header, HeaderIDs->size()).first->second; }; - auto Add = [&, SymIndex(0)](llvm::StringRef Name, llvm::StringRef NS, - llvm::StringRef HeaderName) mutable { + auto Add = [&, NextAvailSymIndex(0)](llvm::StringRef Name, llvm::StringRef NS, + llvm::StringRef HeaderName) mutable { if (NS == "None") NS = ""; -SymbolNames[SymIndex] = {NS, Name}; -SymbolHeaderIDs[SymIndex] = AddHeader(HeaderName); +bool IsNewSymbol = true; +int SymIndex = NextAvailSymIndex; +if (NSSymbolMap *NSSymbols = NamespaceSymbols->lookup(NS)) { + auto It = NSSymbols->find(Name); + if (It != NSSymbols->end()) { +SymIndex = It->getSecond(); +IsNewSymbol = false; + } +} + +SymbolNames->try_emplace(SymIndex, std::make_pair(NS, Name)); + +unsigned HeaderID = AddHeader(HeaderName); +SymbolHeaderIDs[SymIndex].emplace_back(HeaderID); NSSymbolMap &NSSymbols = AddNS(NS); NSSymbols.try_emplace(Name, SymIndex); -++SymIndex; +if (IsNewSymbol) + ++NextAvailSymIndex; }; #define SYMBOL(Name, NS, Header) Add(#Name, #NS, #Header); #include "clang/Tooling/Inclusions/CSymbolMap.inc" @@ -84,8 +99,20 @@ return Header(It->second); } llvm::StringRef Header::name() const { return HeaderNames[ID]; } -llvm::StringRef Symbol::scope() const { return SymbolNames[ID].first; } -llvm::StringRef Symbol::name() const { return SymbolNames[ID].second; } +llvm::StringRef Symbol::scope() const { + auto It = SymbolNames->find(ID); + if (It != SymbolNames->end()) { +return It->second.first; + } + return ""; +} +llvm::StringRef Symbol::name() const { + auto It = SymbolNames->find(ID); + if (It != SymbolNames->end()) { +return It->second.second; + } + return ""; +} std::optional Symbol::named(llvm::StringRef Scope, llvm::StringRef Name) { ensureInitialized(); @@ -96,9 +123,14 @@ } return std::nullopt; } -Header Symbol::header() const { return Header(SymbolHeaderIDs[ID]); } + +Header Symbol::header() const { return Header(SymbolHeaderIDs[ID][0]); } llvm::SmallVector Symbol::headers() const { - return {hea
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo updated this revision to Diff 490468. VitaNuo added a comment. Remove variant logic in generator. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 Files: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp clang/tools/include-mapping/cppreference_parser.py clang/tools/include-mapping/gen_std.py Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -13,12 +13,6 @@ The generated files are located in clang/include/Tooling/Inclusions. -Caveats and FIXMEs: - - only symbols directly in "std" namespace are added, we should also add std's -subnamespace symbols (e.g. chrono). - - symbols with multiple variants or defined in multiple headers aren't added, -e.g. std::move, std::swap - Usage: 1. Install BeautifulSoup dependency, see instruction: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#installing-beautiful-soup @@ -104,17 +98,12 @@ os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') print(CODE_PREFIX % (args.language.upper(), cppreference_modified_date)) for symbol in symbols: -if len(symbol.headers) == 1: - # SYMBOL(unqualified_name, namespace, header) - print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, -symbol.headers[0])) -elif len(symbol.headers) == 0: +if len(symbol.headers) == 0: sys.stderr.write("No header found for symbol %s\n" % symbol.name) else: - # FIXME: support symbols with multiple headers (e.g. std::move). - sys.stderr.write("Ambiguous header for symbol %s: %s\n" % ( - symbol.name, ', '.join(symbol.headers))) - - + for header in symbol.headers: +# SYMBOL(unqualified_name, namespace, header) +print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, + header)) if __name__ == '__main__': main() Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -93,7 +93,7 @@ abs() (int) acos() - Returns a list of tuple (symbol_name, relative_path_to_symbol_page, variant). + Returns a list of tuple (symbol_name, relative_path_to_symbol_page). """ symbols = [] soup = BeautifulSoup(index_page_html, "html.parser") @@ -103,13 +103,10 @@ # This accidentally accepts begin/end despite the (iterator) caption: the # (since C++11) note is first. They are good symbols, so the bug is unfixed. caption = symbol_href.next_sibling -variant = None -if isinstance(caption, NavigableString) and "(" in caption: - variant = caption.text.strip(" ()") symbol_tt = symbol_href.find("tt") if symbol_tt: symbols.append((symbol_tt.text.rstrip("<>()"), # strip any trailing <>() - symbol_href["href"], variant)) + symbol_href["href"])) return symbols @@ -118,7 +115,7 @@ return _ParseSymbolPage(f.read(), name) -def _GetSymbols(pool, root_dir, index_page_name, namespace, variants_to_accept): +def _GetSymbols(pool, root_dir, index_page_name, namespace): """Get all symbols listed in the index page. All symbols should be in the given namespace. @@ -134,13 +131,7 @@ with open(index_page_path, "r") as f: # Read each symbol page in parallel. results = [] # (symbol_name, promise of [header...]) -for symbol_name, symbol_page_path, variant in _ParseIndexPage(f.read()): - # Variant symbols (e.g. the std::locale version of isalpha) add ambiguity. - # FIXME: use these as a fallback rather than ignoring entirely. - variants_for_symbol = variants_to_accept.get( - (namespace or "") + symbol_name, ()) - if variant and variant not in variants_for_symbol: -continue +for symbol_name, symbol_page_path in _ParseIndexPage(f.read()): path = os.path.join(root_dir, symbol_page_path) if os.path.isfile(path): results.append((symbol_name, @@ -166,13 +157,6 @@ Args: parse_pages: a list of tuples (page_root_dir, index_page_name, namespace) """ - # By default we prefer the non-variant versions, as they're more common. But - # there are some symbols, whose variant is more common. This list describes - # those symbols. - variants_to_accept = { - # std::remove<> has variant algorithm. - "std::remove": ("algorithm"), - } symbols = [] # Run many workers to process individual symbol pages under the symbol index. # Don't allow workers to capture Ctrl-C. @@ -180,8 +164,7 @@ initializer=lambda: signal.signal(signal.SIGINT, signal.SIG_IGN)) try: for root_dir, page_name, namespace
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo updated this revision to Diff 490476. VitaNuo added a comment. Restore variant logic for overloads. Remove broken logic for accepted variants. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 Files: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp clang/tools/include-mapping/cppreference_parser.py clang/tools/include-mapping/gen_std.py Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -14,10 +14,7 @@ The generated files are located in clang/include/Tooling/Inclusions. Caveats and FIXMEs: - - only symbols directly in "std" namespace are added, we should also add std's -subnamespace symbols (e.g. chrono). - - symbols with multiple variants or defined in multiple headers aren't added, -e.g. std::move, std::swap + - symbols with multiple variants aren't added, e.g., std::move, std::swap Usage: 1. Install BeautifulSoup dependency, see instruction: @@ -69,7 +66,7 @@ def main(): args = ParseArg() - if args.language == 'cpp': + if args.symbols == 'cpp': page_root = os.path.join(args.cppreference, "en", "cpp") symbol_index_root = os.path.join(page_root, "symbol_index") parse_pages = [ @@ -104,17 +101,12 @@ os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') print(CODE_PREFIX % (args.language.upper(), cppreference_modified_date)) for symbol in symbols: -if len(symbol.headers) == 1: - # SYMBOL(unqualified_name, namespace, header) - print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, -symbol.headers[0])) -elif len(symbol.headers) == 0: +if len(symbol.headers) == 0: sys.stderr.write("No header found for symbol %s\n" % symbol.name) else: - # FIXME: support symbols with multiple headers (e.g. std::move). - sys.stderr.write("Ambiguous header for symbol %s: %s\n" % ( - symbol.name, ', '.join(symbol.headers))) - - + for header in symbol.headers: +# SYMBOL(unqualified_name, namespace, header) +print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, + header)) if __name__ == '__main__': main() Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -118,7 +118,7 @@ return _ParseSymbolPage(f.read(), name) -def _GetSymbols(pool, root_dir, index_page_name, namespace, variants_to_accept): +def _GetSymbols(pool, root_dir, index_page_name, namespace): """Get all symbols listed in the index page. All symbols should be in the given namespace. @@ -137,9 +137,7 @@ for symbol_name, symbol_page_path, variant in _ParseIndexPage(f.read()): # Variant symbols (e.g. the std::locale version of isalpha) add ambiguity. # FIXME: use these as a fallback rather than ignoring entirely. - variants_for_symbol = variants_to_accept.get( - (namespace or "") + symbol_name, ()) - if variant and variant not in variants_for_symbol: + if variant: continue path = os.path.join(root_dir, symbol_page_path) if os.path.isfile(path): @@ -166,13 +164,6 @@ Args: parse_pages: a list of tuples (page_root_dir, index_page_name, namespace) """ - # By default we prefer the non-variant versions, as they're more common. But - # there are some symbols, whose variant is more common. This list describes - # those symbols. - variants_to_accept = { - # std::remove<> has variant algorithm. - "std::remove": ("algorithm"), - } symbols = [] # Run many workers to process individual symbol pages under the symbol index. # Don't allow workers to capture Ctrl-C. @@ -180,8 +171,7 @@ initializer=lambda: signal.signal(signal.SIGINT, signal.SIG_IGN)) try: for root_dir, page_name, namespace in parse_pages: - symbols.extend(_GetSymbols(pool, root_dir, page_name, namespace, - variants_to_accept)) + symbols.extend(_GetSymbols(pool, root_dir, page_name, namespace)) finally: pool.terminate() pool.join() Index: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp === --- clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp +++ clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp @@ -8,6 +8,7 @@ #include "clang/Tooling/Inclusions/StandardLibrary.h" #include "clang/AST/Decl.h" +#include "llvm/ADT/DenseMap.h" #include "llvm/ADT/StringRef.h" #include "llvm/Support/Casting.h" @@ -16,8 +17,9 @@ namespace stdlib { static llvm::StringRef *HeaderNames; -static std::pair *Symbol
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo updated this revision to Diff 490478. VitaNuo added a comment. Fix typo. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 Files: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp clang/tools/include-mapping/cppreference_parser.py clang/tools/include-mapping/gen_std.py Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -14,10 +14,7 @@ The generated files are located in clang/include/Tooling/Inclusions. Caveats and FIXMEs: - - only symbols directly in "std" namespace are added, we should also add std's -subnamespace symbols (e.g. chrono). - - symbols with multiple variants or defined in multiple headers aren't added, -e.g. std::move, std::swap + - symbols with multiple variants aren't added, e.g., std::move, std::swap Usage: 1. Install BeautifulSoup dependency, see instruction: @@ -104,17 +101,12 @@ os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') print(CODE_PREFIX % (args.language.upper(), cppreference_modified_date)) for symbol in symbols: -if len(symbol.headers) == 1: - # SYMBOL(unqualified_name, namespace, header) - print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, -symbol.headers[0])) -elif len(symbol.headers) == 0: +if len(symbol.headers) == 0: sys.stderr.write("No header found for symbol %s\n" % symbol.name) else: - # FIXME: support symbols with multiple headers (e.g. std::move). - sys.stderr.write("Ambiguous header for symbol %s: %s\n" % ( - symbol.name, ', '.join(symbol.headers))) - - + for header in symbol.headers: +# SYMBOL(unqualified_name, namespace, header) +print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, + header)) if __name__ == '__main__': main() Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -118,7 +118,7 @@ return _ParseSymbolPage(f.read(), name) -def _GetSymbols(pool, root_dir, index_page_name, namespace, variants_to_accept): +def _GetSymbols(pool, root_dir, index_page_name, namespace): """Get all symbols listed in the index page. All symbols should be in the given namespace. @@ -137,9 +137,7 @@ for symbol_name, symbol_page_path, variant in _ParseIndexPage(f.read()): # Variant symbols (e.g. the std::locale version of isalpha) add ambiguity. # FIXME: use these as a fallback rather than ignoring entirely. - variants_for_symbol = variants_to_accept.get( - (namespace or "") + symbol_name, ()) - if variant and variant not in variants_for_symbol: + if variant: continue path = os.path.join(root_dir, symbol_page_path) if os.path.isfile(path): @@ -166,13 +164,6 @@ Args: parse_pages: a list of tuples (page_root_dir, index_page_name, namespace) """ - # By default we prefer the non-variant versions, as they're more common. But - # there are some symbols, whose variant is more common. This list describes - # those symbols. - variants_to_accept = { - # std::remove<> has variant algorithm. - "std::remove": ("algorithm"), - } symbols = [] # Run many workers to process individual symbol pages under the symbol index. # Don't allow workers to capture Ctrl-C. @@ -180,8 +171,7 @@ initializer=lambda: signal.signal(signal.SIGINT, signal.SIG_IGN)) try: for root_dir, page_name, namespace in parse_pages: - symbols.extend(_GetSymbols(pool, root_dir, page_name, namespace, - variants_to_accept)) + symbols.extend(_GetSymbols(pool, root_dir, page_name, namespace)) finally: pool.terminate() pool.join() Index: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp === --- clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp +++ clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp @@ -8,6 +8,7 @@ #include "clang/Tooling/Inclusions/StandardLibrary.h" #include "clang/AST/Decl.h" +#include "llvm/ADT/DenseMap.h" #include "llvm/ADT/StringRef.h" #include "llvm/Support/Casting.h" @@ -16,8 +17,9 @@ namespace stdlib { static llvm::StringRef *HeaderNames; -static std::pair *SymbolNames; -static unsigned *SymbolHeaderIDs; +static llvm::DenseMap> +*SymbolNames; +static llvm::SmallVector *SymbolHeaderIDs; static llvm::DenseMap *HeaderIDs; // Maps symbol name -> Symbol::ID, within a namespace. using NSSymbolMap = llvm::DenseMap; @@ -29,7 +31,7 @@ #include "clang/Tooling/Inclusions/CSymbolMap.inc" #inc
[PATCH] D141855: [include-mapping] Parse zombie_names.html into a removed symbols map.
This revision was landed with ongoing or failed builds. This revision was automatically updated to reflect the committed changes. Closed by commit rGe74f9e788507: [include-mapping] Parse zombie_names.html into a removed symbols map. (authored by VitaNuo). Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D141855/new/ https://reviews.llvm.org/D141855 Files: clang/tools/include-mapping/gen_std.py Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -28,9 +28,11 @@ get a "cppreference/reference" directory. 4. Run the command: // Generate C++ symbols - python3 gen_std.py -cppreference cppreference/reference -language=cpp > StdSymbolMap.inc + python3 gen_std.py -cppreference cppreference/reference -symbols=cpp > StdSymbolMap.inc + // Generate C++ removed symbols + python3 gen_std.py -cppreference cppreference/reference -symbols=cpp_removed > RemovedSymbolMap.inc // Generate C symbols - python3 gen_std.py -cppreference cppreference/reference -language=c > CSymbolMap.inc + python3 gen_std.py -cppreference cppreference/reference -symbols=c > CSymbolMap.inc """ @@ -60,16 +62,16 @@ help='path to the cppreference offline HTML directory', required=True ) - parser.add_argument('-language', + parser.add_argument('-symbols', default='cpp', - help='Generate c or cpp symbols', - required=True) + help='Generate c or cpp (removed) symbols. One of {cpp, c, cpp_removed}.', + required=True) return parser.parse_args() def main(): args = ParseArg() - if args.language == 'cpp': + if args.symbols == 'cpp': page_root = os.path.join(args.cppreference, "en", "cpp") symbol_index_root = os.path.join(page_root, "symbol_index") parse_pages = [ @@ -87,22 +89,26 @@ (symbol_index_root, "regex_constants.html", "std::regex_constants::"), (symbol_index_root, "this_thread.html", "std::this_thread::"), ] - elif args.language == 'c': + elif args.symbols == 'cpp_removed': +page_root = os.path.join(args.cppreference, "en", "cpp") +symbol_index_root = os.path.join(page_root, "symbol_index") +parse_pages = [(symbol_index_root, "zombie_names.html", "std::")] + elif args.symbols == 'c': page_root = os.path.join(args.cppreference, "en", "c") symbol_index_root = page_root -parse_pages = [(page_root, "index.html", None)] - +parse_pages = [(page_root, "index.html", None)] + if not os.path.exists(symbol_index_root): exit("Path %s doesn't exist!" % symbol_index_root) symbols = cppreference_parser.GetSymbols(parse_pages) - + # We don't have version information from the unzipped offline HTML files. # so we use the modified time of the symbol_index.html as the version. index_page_path = os.path.join(page_root, "index.html") cppreference_modified_date = datetime.datetime.fromtimestamp( os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') - print(CODE_PREFIX % (args.language.upper(), cppreference_modified_date)) + print(CODE_PREFIX % (args.symbols.upper(), cppreference_modified_date)) for symbol in symbols: if len(symbol.headers) == 1: # SYMBOL(unqualified_name, namespace, header) Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -28,9 +28,11 @@ get a "cppreference/reference" directory. 4. Run the command: // Generate C++ symbols - python3 gen_std.py -cppreference cppreference/reference -language=cpp > StdSymbolMap.inc + python3 gen_std.py -cppreference cppreference/reference -symbols=cpp > StdSymbolMap.inc + // Generate C++ removed symbols + python3 gen_std.py -cppreference cppreference/reference -symbols=cpp_removed > RemovedSymbolMap.inc // Generate C symbols - python3 gen_std.py -cppreference cppreference/reference -language=c > CSymbolMap.inc + python3 gen_std.py -cppreference cppreference/reference -symbols=c > CSymbolMap.inc """ @@ -60,16 +62,16 @@ help='path to the cppreference offline HTML directory', required=True ) - parser.add_argument('-language', + parser.add_argument('-symbols', default='cpp', - help='Generate c or cpp symbols', - required=True) + help='Generate c or cpp (removed) symbols. One of {cpp, c, cpp_removed}.', + required=True) return parser.parse_args() def main
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo updated this revision to Diff 490824. VitaNuo added a comment. Herald added a reviewer: jdoerfert. Herald added a subscriber: sstefan1. Add re-generated symbol maps. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 Files: clang/include/clang/Tooling/Inclusions/CSymbolMap.inc clang/include/clang/Tooling/Inclusions/RemovedStdSymbolMap.inc clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp clang/tools/include-mapping/cppreference_parser.py clang/tools/include-mapping/gen_std.py Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -14,10 +14,7 @@ The generated files are located in clang/include/Tooling/Inclusions. Caveats and FIXMEs: - - only symbols directly in "std" namespace are added, we should also add std's -subnamespace symbols (e.g. chrono). - - symbols with multiple variants or defined in multiple headers aren't added, -e.g. std::move, std::swap + - symbols with multiple variants aren't added, e.g., std::move, std::swap Usage: 1. Install BeautifulSoup dependency, see instruction: @@ -110,17 +107,12 @@ os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') print(CODE_PREFIX % (args.symbols.upper(), cppreference_modified_date)) for symbol in symbols: -if len(symbol.headers) == 1: - # SYMBOL(unqualified_name, namespace, header) - print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, -symbol.headers[0])) -elif len(symbol.headers) == 0: +if len(symbol.headers) == 0: sys.stderr.write("No header found for symbol %s\n" % symbol.name) else: - # FIXME: support symbols with multiple headers (e.g. std::move). - sys.stderr.write("Ambiguous header for symbol %s: %s\n" % ( - symbol.name, ', '.join(symbol.headers))) - - + for header in symbol.headers: +# SYMBOL(unqualified_name, namespace, header) +print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, + header)) if __name__ == '__main__': main() Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -118,7 +118,7 @@ return _ParseSymbolPage(f.read(), name) -def _GetSymbols(pool, root_dir, index_page_name, namespace, variants_to_accept): +def _GetSymbols(pool, root_dir, index_page_name, namespace): """Get all symbols listed in the index page. All symbols should be in the given namespace. @@ -137,9 +137,7 @@ for symbol_name, symbol_page_path, variant in _ParseIndexPage(f.read()): # Variant symbols (e.g. the std::locale version of isalpha) add ambiguity. # FIXME: use these as a fallback rather than ignoring entirely. - variants_for_symbol = variants_to_accept.get( - (namespace or "") + symbol_name, ()) - if variant and variant not in variants_for_symbol: + if variant: continue path = os.path.join(root_dir, symbol_page_path) if os.path.isfile(path): @@ -166,13 +164,6 @@ Args: parse_pages: a list of tuples (page_root_dir, index_page_name, namespace) """ - # By default we prefer the non-variant versions, as they're more common. But - # there are some symbols, whose variant is more common. This list describes - # those symbols. - variants_to_accept = { - # std::remove<> has variant algorithm. - "std::remove": ("algorithm"), - } symbols = [] # Run many workers to process individual symbol pages under the symbol index. # Don't allow workers to capture Ctrl-C. @@ -180,8 +171,7 @@ initializer=lambda: signal.signal(signal.SIGINT, signal.SIG_IGN)) try: for root_dir, page_name, namespace in parse_pages: - symbols.extend(_GetSymbols(pool, root_dir, page_name, namespace, - variants_to_accept)) + symbols.extend(_GetSymbols(pool, root_dir, page_name, namespace)) finally: pool.terminate() pool.join() Index: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp === --- clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp +++ clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp @@ -8,6 +8,7 @@ #include "clang/Tooling/Inclusions/StandardLibrary.h" #include "clang/AST/Decl.h" +#include "llvm/ADT/DenseMap.h" #include "llvm/ADT/StringRef.h" #include "llvm/Support/Casting.h" @@ -16,8 +17,9 @@ namespace stdlib { static llvm::StringRef *HeaderNames; -static std::pair *SymbolNames; -static unsigned *SymbolHeaderIDs; +static llvm::Dense
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo added a comment. Added the re-generated symbol maps to this patch as per @hokein's request. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo added a comment. Oh I misunderstood your comment regarding which maps need to be regenerated as part of this patch. Regenerated the 2018 ones now. As expected, there are only added symbols now. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo updated this revision to Diff 491260. VitaNuo marked an inline comment as done. VitaNuo added a comment. Address review comment. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 Files: clang/include/clang/Tooling/Inclusions/CSymbolMap.inc clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp clang/tools/include-mapping/cppreference_parser.py clang/tools/include-mapping/gen_std.py Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -14,10 +14,7 @@ The generated files are located in clang/include/Tooling/Inclusions. Caveats and FIXMEs: - - only symbols directly in "std" namespace are added, we should also add std's -subnamespace symbols (e.g. chrono). - - symbols with multiple variants or defined in multiple headers aren't added, -e.g. std::move, std::swap + - symbols with multiple variants aren't added, e.g., std::move Usage: 1. Install BeautifulSoup dependency, see instruction: @@ -110,17 +107,12 @@ os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') print(CODE_PREFIX % (args.symbols.upper(), cppreference_modified_date)) for symbol in symbols: -if len(symbol.headers) == 1: - # SYMBOL(unqualified_name, namespace, header) - print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, -symbol.headers[0])) -elif len(symbol.headers) == 0: +if len(symbol.headers) == 0: sys.stderr.write("No header found for symbol %s\n" % symbol.name) else: - # FIXME: support symbols with multiple headers (e.g. std::move). - sys.stderr.write("Ambiguous header for symbol %s: %s\n" % ( - symbol.name, ', '.join(symbol.headers))) - - + for header in symbol.headers: +# SYMBOL(unqualified_name, namespace, header) +print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, + header)) if __name__ == '__main__': main() Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -118,7 +118,7 @@ return _ParseSymbolPage(f.read(), name) -def _GetSymbols(pool, root_dir, index_page_name, namespace, variants_to_accept): +def _GetSymbols(pool, root_dir, index_page_name, namespace): """Get all symbols listed in the index page. All symbols should be in the given namespace. @@ -137,9 +137,7 @@ for symbol_name, symbol_page_path, variant in _ParseIndexPage(f.read()): # Variant symbols (e.g. the std::locale version of isalpha) add ambiguity. # FIXME: use these as a fallback rather than ignoring entirely. - variants_for_symbol = variants_to_accept.get( - (namespace or "") + symbol_name, ()) - if variant and variant not in variants_for_symbol: + if variant: continue path = os.path.join(root_dir, symbol_page_path) if os.path.isfile(path): @@ -166,13 +164,6 @@ Args: parse_pages: a list of tuples (page_root_dir, index_page_name, namespace) """ - # By default we prefer the non-variant versions, as they're more common. But - # there are some symbols, whose variant is more common. This list describes - # those symbols. - variants_to_accept = { - # std::remove<> has variant algorithm. - "std::remove": ("algorithm"), - } symbols = [] # Run many workers to process individual symbol pages under the symbol index. # Don't allow workers to capture Ctrl-C. @@ -180,8 +171,7 @@ initializer=lambda: signal.signal(signal.SIGINT, signal.SIG_IGN)) try: for root_dir, page_name, namespace in parse_pages: - symbols.extend(_GetSymbols(pool, root_dir, page_name, namespace, - variants_to_accept)) + symbols.extend(_GetSymbols(pool, root_dir, page_name, namespace)) finally: pool.terminate() pool.join() Index: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp === --- clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp +++ clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp @@ -8,6 +8,7 @@ #include "clang/Tooling/Inclusions/StandardLibrary.h" #include "clang/AST/Decl.h" +#include "llvm/ADT/DenseMap.h" #include "llvm/ADT/StringRef.h" #include "llvm/Support/Casting.h" @@ -16,8 +17,9 @@ namespace stdlib { static llvm::StringRef *HeaderNames; -static std::pair *SymbolNames; -static unsigned *SymbolHeaderIDs; +static llvm::DenseMap> +*SymbolNames; +static llvm::SmallVector *SymbolHeaderIDs; static llvm::DenseMap *HeaderIDs; // Maps s
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo updated this revision to Diff 491258. VitaNuo added a comment. Update the 20181028 archive maps. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 Files: clang/include/clang/Tooling/Inclusions/CSymbolMap.inc clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp clang/tools/include-mapping/cppreference_parser.py clang/tools/include-mapping/gen_std.py Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -14,10 +14,7 @@ The generated files are located in clang/include/Tooling/Inclusions. Caveats and FIXMEs: - - only symbols directly in "std" namespace are added, we should also add std's -subnamespace symbols (e.g. chrono). - - symbols with multiple variants or defined in multiple headers aren't added, -e.g. std::move, std::swap + - symbols with multiple variants aren't added, e.g., std::move, std::swap Usage: 1. Install BeautifulSoup dependency, see instruction: @@ -110,17 +107,12 @@ os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') print(CODE_PREFIX % (args.symbols.upper(), cppreference_modified_date)) for symbol in symbols: -if len(symbol.headers) == 1: - # SYMBOL(unqualified_name, namespace, header) - print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, -symbol.headers[0])) -elif len(symbol.headers) == 0: +if len(symbol.headers) == 0: sys.stderr.write("No header found for symbol %s\n" % symbol.name) else: - # FIXME: support symbols with multiple headers (e.g. std::move). - sys.stderr.write("Ambiguous header for symbol %s: %s\n" % ( - symbol.name, ', '.join(symbol.headers))) - - + for header in symbol.headers: +# SYMBOL(unqualified_name, namespace, header) +print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, + header)) if __name__ == '__main__': main() Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -118,7 +118,7 @@ return _ParseSymbolPage(f.read(), name) -def _GetSymbols(pool, root_dir, index_page_name, namespace, variants_to_accept): +def _GetSymbols(pool, root_dir, index_page_name, namespace): """Get all symbols listed in the index page. All symbols should be in the given namespace. @@ -137,9 +137,7 @@ for symbol_name, symbol_page_path, variant in _ParseIndexPage(f.read()): # Variant symbols (e.g. the std::locale version of isalpha) add ambiguity. # FIXME: use these as a fallback rather than ignoring entirely. - variants_for_symbol = variants_to_accept.get( - (namespace or "") + symbol_name, ()) - if variant and variant not in variants_for_symbol: + if variant: continue path = os.path.join(root_dir, symbol_page_path) if os.path.isfile(path): @@ -166,13 +164,6 @@ Args: parse_pages: a list of tuples (page_root_dir, index_page_name, namespace) """ - # By default we prefer the non-variant versions, as they're more common. But - # there are some symbols, whose variant is more common. This list describes - # those symbols. - variants_to_accept = { - # std::remove<> has variant algorithm. - "std::remove": ("algorithm"), - } symbols = [] # Run many workers to process individual symbol pages under the symbol index. # Don't allow workers to capture Ctrl-C. @@ -180,8 +171,7 @@ initializer=lambda: signal.signal(signal.SIGINT, signal.SIG_IGN)) try: for root_dir, page_name, namespace in parse_pages: - symbols.extend(_GetSymbols(pool, root_dir, page_name, namespace, - variants_to_accept)) + symbols.extend(_GetSymbols(pool, root_dir, page_name, namespace)) finally: pool.terminate() pool.join() Index: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp === --- clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp +++ clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp @@ -8,6 +8,7 @@ #include "clang/Tooling/Inclusions/StandardLibrary.h" #include "clang/AST/Decl.h" +#include "llvm/ADT/DenseMap.h" #include "llvm/ADT/StringRef.h" #include "llvm/Support/Casting.h" @@ -16,8 +17,9 @@ namespace stdlib { static llvm::StringRef *HeaderNames; -static std::pair *SymbolNames; -static unsigned *SymbolHeaderIDs; +static llvm::DenseMap> +*SymbolNames; +static llvm::SmallVector *SymbolHeaderIDs; static llvm::DenseMap *HeaderIDs; // Maps symbol name -> Symbol:
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo updated this revision to Diff 491367. VitaNuo added a comment. Re-introduce the special std::remove handling for now. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 Files: clang/include/clang/Tooling/Inclusions/CSymbolMap.inc clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp clang/tools/include-mapping/cppreference_parser.py clang/tools/include-mapping/gen_std.py Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -14,10 +14,7 @@ The generated files are located in clang/include/Tooling/Inclusions. Caveats and FIXMEs: - - only symbols directly in "std" namespace are added, we should also add std's -subnamespace symbols (e.g. chrono). - - symbols with multiple variants or defined in multiple headers aren't added, -e.g. std::move, std::swap + - symbols with multiple variants aren't added, e.g., std::move Usage: 1. Install BeautifulSoup dependency, see instruction: @@ -110,17 +107,12 @@ os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') print(CODE_PREFIX % (args.symbols.upper(), cppreference_modified_date)) for symbol in symbols: -if len(symbol.headers) == 1: - # SYMBOL(unqualified_name, namespace, header) - print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, -symbol.headers[0])) -elif len(symbol.headers) == 0: +if len(symbol.headers) == 0: sys.stderr.write("No header found for symbol %s\n" % symbol.name) else: - # FIXME: support symbols with multiple headers (e.g. std::move). - sys.stderr.write("Ambiguous header for symbol %s: %s\n" % ( - symbol.name, ', '.join(symbol.headers))) - - + for header in symbol.headers: +# SYMBOL(unqualified_name, namespace, header) +print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, + header)) if __name__ == '__main__': main() Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -180,8 +180,7 @@ initializer=lambda: signal.signal(signal.SIGINT, signal.SIG_IGN)) try: for root_dir, page_name, namespace in parse_pages: - symbols.extend(_GetSymbols(pool, root_dir, page_name, namespace, - variants_to_accept)) + symbols.extend(_GetSymbols(pool, root_dir, page_name, namespace, variants_to_accept)) finally: pool.terminate() pool.join() Index: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp === --- clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp +++ clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp @@ -8,6 +8,7 @@ #include "clang/Tooling/Inclusions/StandardLibrary.h" #include "clang/AST/Decl.h" +#include "llvm/ADT/DenseMap.h" #include "llvm/ADT/StringRef.h" #include "llvm/Support/Casting.h" @@ -16,8 +17,9 @@ namespace stdlib { static llvm::StringRef *HeaderNames; -static std::pair *SymbolNames; -static unsigned *SymbolHeaderIDs; +static llvm::DenseMap> +*SymbolNames; +static llvm::SmallVector *SymbolHeaderIDs; static llvm::DenseMap *HeaderIDs; // Maps symbol name -> Symbol::ID, within a namespace. using NSSymbolMap = llvm::DenseMap; @@ -29,7 +31,7 @@ #include "clang/Tooling/Inclusions/CSymbolMap.inc" #include "clang/Tooling/Inclusions/StdSymbolMap.inc" #undef SYMBOL - SymbolNames = new std::remove_reference_t[SymCount]; + SymbolNames = new std::remove_reference_t; SymbolHeaderIDs = new std::remove_reference_t[SymCount]; NamespaceSymbols = new std::remove_reference_t; @@ -46,18 +48,31 @@ return HeaderIDs->try_emplace(Header, HeaderIDs->size()).first->second; }; - auto Add = [&, SymIndex(0)](llvm::StringRef Name, llvm::StringRef NS, - llvm::StringRef HeaderName) mutable { + auto Add = [&, NextAvailSymIndex(0)](llvm::StringRef Name, llvm::StringRef NS, + llvm::StringRef HeaderName) mutable { if (NS == "None") NS = ""; -SymbolNames[SymIndex] = {NS, Name}; -SymbolHeaderIDs[SymIndex] = AddHeader(HeaderName); +bool IsNewSymbol = true; +int SymIndex = NextAvailSymIndex; +if (NSSymbolMap *NSSymbols = NamespaceSymbols->lookup(NS)) { + auto It = NSSymbols->find(Name); + if (It != NSSymbols->end()) { +SymIndex = It->getSecond(); +IsNewSymbol = false; + } +} + +SymbolNames->try_emplace(SymIndex, std::make_pair(NS, Name)); + +unsigned HeaderID = AddHeader
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo added inline comments. Comment at: clang/tools/include-mapping/cppreference_parser.py:174 - # std::remove<> has variant algorithm. - "std::remove": ("algorithm"), - } kadircet wrote: > this is actually checking for something else (sorry for the confusing naming). > > the `variant` here refers to library name mentioned in parentheses (this is > same problem as `std::move`) on the std symbol index page > https://en.cppreference.com/w/cpp/symbol_index (e.g. `remove<>() > (algorithm)`). by getting rid of this we're introducing a regression, as > previously `std::remove` wouldn't be recognized by the library, but now it'll > be recognized and we'll keep suggesting `` for it. > > so we should actually keep this around. Ok, I can keep this out of this patch, but we'll have to remove this logic evetually when we deal with overloads. I have a slight suspicion that this code might be buggy, because it suggests that one _of_ the variants should be accepted. What is does in reality, though, is it keeps `algorithm` in the list of headers suitable for `std::remove` alongside `cstdio`, and then in the last step `std::remove` is ignored by the generator because of being defined in two headers. With this patch, the result will be both `{cstdio, algorithm}`. Is this (more) satisfactory for now compared to skipping `algorithm` due to being an overload? Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo updated this revision to Diff 491380. VitaNuo marked 2 inline comments as done. VitaNuo added a comment. Address review comments regarding data structure use and style. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 Files: clang/include/clang/Tooling/Inclusions/CSymbolMap.inc clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp clang/tools/include-mapping/cppreference_parser.py clang/tools/include-mapping/gen_std.py Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -14,10 +14,7 @@ The generated files are located in clang/include/Tooling/Inclusions. Caveats and FIXMEs: - - only symbols directly in "std" namespace are added, we should also add std's -subnamespace symbols (e.g. chrono). - - symbols with multiple variants or defined in multiple headers aren't added, -e.g. std::move, std::swap + - symbols with multiple variants aren't added, e.g., std::move Usage: 1. Install BeautifulSoup dependency, see instruction: @@ -110,17 +107,12 @@ os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') print(CODE_PREFIX % (args.symbols.upper(), cppreference_modified_date)) for symbol in symbols: -if len(symbol.headers) == 1: - # SYMBOL(unqualified_name, namespace, header) - print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, -symbol.headers[0])) -elif len(symbol.headers) == 0: +if len(symbol.headers) == 0: sys.stderr.write("No header found for symbol %s\n" % symbol.name) else: - # FIXME: support symbols with multiple headers (e.g. std::move). - sys.stderr.write("Ambiguous header for symbol %s: %s\n" % ( - symbol.name, ', '.join(symbol.headers))) - - + for header in symbol.headers: +# SYMBOL(unqualified_name, namespace, header) +print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, + header)) if __name__ == '__main__': main() Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -180,8 +180,7 @@ initializer=lambda: signal.signal(signal.SIGINT, signal.SIG_IGN)) try: for root_dir, page_name, namespace in parse_pages: - symbols.extend(_GetSymbols(pool, root_dir, page_name, namespace, - variants_to_accept)) + symbols.extend(_GetSymbols(pool, root_dir, page_name, namespace, variants_to_accept)) finally: pool.terminate() pool.join() Index: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp === --- clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp +++ clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp @@ -8,6 +8,7 @@ #include "clang/Tooling/Inclusions/StandardLibrary.h" #include "clang/AST/Decl.h" +#include "llvm/ADT/DenseMap.h" #include "llvm/ADT/StringRef.h" #include "llvm/Support/Casting.h" @@ -17,7 +18,7 @@ static llvm::StringRef *HeaderNames; static std::pair *SymbolNames; -static unsigned *SymbolHeaderIDs; +static llvm::SmallVector *SymbolHeaderIDs; static llvm::DenseMap *HeaderIDs; // Maps symbol name -> Symbol::ID, within a namespace. using NSSymbolMap = llvm::DenseMap; @@ -46,18 +47,29 @@ return HeaderIDs->try_emplace(Header, HeaderIDs->size()).first->second; }; - auto Add = [&, SymIndex(0)](llvm::StringRef Name, llvm::StringRef NS, - llvm::StringRef HeaderName) mutable { + auto Add = [&, NextAvailSymIndex(0)](llvm::StringRef Name, llvm::StringRef NS, + llvm::StringRef HeaderName) mutable { if (NS == "None") NS = ""; +bool IsNewSymbol = true; +unsigned SymIndex = NextAvailSymIndex; +if (NSSymbolMap *NSSymbols = NamespaceSymbols->lookup(NS)) { + auto It = NSSymbols->find(Name); + if (It != NSSymbols->end()) { +SymIndex = It->getSecond(); +IsNewSymbol = false; + } +} + SymbolNames[SymIndex] = {NS, Name}; -SymbolHeaderIDs[SymIndex] = AddHeader(HeaderName); +SymbolHeaderIDs[SymIndex].emplace_back(AddHeader(HeaderName)); NSSymbolMap &NSSymbols = AddNS(NS); NSSymbols.try_emplace(Name, SymIndex); -++SymIndex; +if (IsNewSymbol) + ++NextAvailSymIndex; }; #define SYMBOL(Name, NS, Header) Add(#Name, #NS, #Header); #include "clang/Tooling/Inclusions/CSymbolMap.inc" @@ -96,9 +108,14 @@ } return std::nullopt; } -Header Symbol::header() const { return Header(SymbolHeaderIDs[ID]); } + +H
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo updated this revision to Diff 491382. VitaNuo added a comment. Fix formatting. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 Files: clang/include/clang/Tooling/Inclusions/CSymbolMap.inc clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp clang/tools/include-mapping/cppreference_parser.py clang/tools/include-mapping/gen_std.py Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -14,10 +14,7 @@ The generated files are located in clang/include/Tooling/Inclusions. Caveats and FIXMEs: - - only symbols directly in "std" namespace are added, we should also add std's -subnamespace symbols (e.g. chrono). - - symbols with multiple variants or defined in multiple headers aren't added, -e.g. std::move, std::swap + - symbols with multiple variants aren't added, e.g., std::move Usage: 1. Install BeautifulSoup dependency, see instruction: @@ -110,17 +107,12 @@ os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') print(CODE_PREFIX % (args.symbols.upper(), cppreference_modified_date)) for symbol in symbols: -if len(symbol.headers) == 1: - # SYMBOL(unqualified_name, namespace, header) - print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, -symbol.headers[0])) -elif len(symbol.headers) == 0: +if len(symbol.headers) == 0: sys.stderr.write("No header found for symbol %s\n" % symbol.name) else: - # FIXME: support symbols with multiple headers (e.g. std::move). - sys.stderr.write("Ambiguous header for symbol %s: %s\n" % ( - symbol.name, ', '.join(symbol.headers))) - - + for header in symbol.headers: +# SYMBOL(unqualified_name, namespace, header) +print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, + header)) if __name__ == '__main__': main() Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -180,8 +180,7 @@ initializer=lambda: signal.signal(signal.SIGINT, signal.SIG_IGN)) try: for root_dir, page_name, namespace in parse_pages: - symbols.extend(_GetSymbols(pool, root_dir, page_name, namespace, - variants_to_accept)) + symbols.extend(_GetSymbols(pool, root_dir, page_name, namespace, variants_to_accept)) finally: pool.terminate() pool.join() Index: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp === --- clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp +++ clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp @@ -8,6 +8,7 @@ #include "clang/Tooling/Inclusions/StandardLibrary.h" #include "clang/AST/Decl.h" +#include "llvm/ADT/DenseMap.h" #include "llvm/ADT/StringRef.h" #include "llvm/Support/Casting.h" @@ -17,7 +18,7 @@ static llvm::StringRef *HeaderNames; static std::pair *SymbolNames; -static unsigned *SymbolHeaderIDs; +static llvm::SmallVector *SymbolHeaderIDs; static llvm::DenseMap *HeaderIDs; // Maps symbol name -> Symbol::ID, within a namespace. using NSSymbolMap = llvm::DenseMap; @@ -46,18 +47,29 @@ return HeaderIDs->try_emplace(Header, HeaderIDs->size()).first->second; }; - auto Add = [&, SymIndex(0)](llvm::StringRef Name, llvm::StringRef NS, - llvm::StringRef HeaderName) mutable { + auto Add = [&, NextAvailSymIndex(0)](llvm::StringRef Name, llvm::StringRef NS, + llvm::StringRef HeaderName) mutable { if (NS == "None") NS = ""; +bool IsNewSymbol = true; +unsigned SymIndex = NextAvailSymIndex; +if (NSSymbolMap *NSSymbols = NamespaceSymbols->lookup(NS)) { + auto It = NSSymbols->find(Name); + if (It != NSSymbols->end()) { +SymIndex = It->getSecond(); +IsNewSymbol = false; + } +} + SymbolNames[SymIndex] = {NS, Name}; -SymbolHeaderIDs[SymIndex] = AddHeader(HeaderName); +SymbolHeaderIDs[SymIndex].emplace_back(AddHeader(HeaderName)); NSSymbolMap &NSSymbols = AddNS(NS); NSSymbols.try_emplace(Name, SymIndex); -++SymIndex; +if (IsNewSymbol) + ++NextAvailSymIndex; }; #define SYMBOL(Name, NS, Header) Add(#Name, #NS, #Header); #include "clang/Tooling/Inclusions/CSymbolMap.inc" @@ -87,7 +99,7 @@ llvm::StringRef Symbol::scope() const { return SymbolNames[ID].first; } llvm::StringRef Symbol::name() const { return SymbolNames[ID].second; } std::optional Symbol::named(llvm::StringRef Scope, -
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo added inline comments. Comment at: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp:20 static llvm::StringRef *HeaderNames; -static std::pair *SymbolNames; -static unsigned *SymbolHeaderIDs; +static llvm::DenseMap> +*SymbolNames; hokein wrote: > int => unsigned. > > Any reason to change it to a map? My understanding is that SymbolNames is a > mapping from the SymbolID => {Scope, Name}, it should not be affected if we > add multiple-header symbols. Right. I am not sure what I was thinking. Possibly I was thinking that symbol IDs won't be continuous anymore after this change, but they actually are. Thanks! Comment at: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp:58 +int SymIndex = NextAvailSymIndex; +if (NSSymbolMap *NSSymbols = NamespaceSymbols->lookup(NS)) { + auto It = NSSymbols->find(Name); hokein wrote: > Given the fact that multiple-header symbols are grouped in the .inc file, we > could simplify the code of checking a new symbol by looking at the last > available SymIndex: > > ``` > if (NextAvailSymIndex > 0 && SymbolNames[NextAvailSymIndex-1].first == NS && > SymbolNames[NextAvailSymIndex-1].second == Name) { >// this is not a new symbol. > } > ``` I know this is easier in terms of code, but this heavily relies on the order in the generated files. I would prefer to keep the library and the generator as decoupled as possible, even if it means slightly more complex code here. Overall, it seems more future-proof in case of unrelated generator changes (bugs?) that might change the order. Comment at: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp:103 +llvm::StringRef Symbol::scope() const { + auto It = SymbolNames->find(ID); + if (It != SymbolNames->end()) { hokein wrote: > The `ID` field is guaranteed to be valid, so no need to do the sanity check. This is all gone now, since the code is back to the array version of SymbolNames. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo updated this revision to Diff 491407. VitaNuo added a comment. Special case the overloads for now. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 Files: clang/include/clang/Tooling/Inclusions/CSymbolMap.inc clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp clang/tools/include-mapping/cppreference_parser.py clang/tools/include-mapping/gen_std.py Index: clang/tools/include-mapping/gen_std.py === --- clang/tools/include-mapping/gen_std.py +++ clang/tools/include-mapping/gen_std.py @@ -14,10 +14,7 @@ The generated files are located in clang/include/Tooling/Inclusions. Caveats and FIXMEs: - - only symbols directly in "std" namespace are added, we should also add std's -subnamespace symbols (e.g. chrono). - - symbols with multiple variants or defined in multiple headers aren't added, -e.g. std::move, std::swap + - symbols with multiple variants aren't added, e.g., std::move Usage: 1. Install BeautifulSoup dependency, see instruction: @@ -41,6 +38,7 @@ import datetime import os import sys +import re CODE_PREFIX = """\ //===-- gen_std.py generated file ---*- C++ -*-===// @@ -109,18 +107,16 @@ cppreference_modified_date = datetime.datetime.fromtimestamp( os.stat(index_page_path).st_mtime).strftime('%Y-%m-%d') print(CODE_PREFIX % (args.symbols.upper(), cppreference_modified_date)) + # FIXME: take care of overloads for symbol in symbols: -if len(symbol.headers) == 1: - # SYMBOL(unqualified_name, namespace, header) - print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, -symbol.headers[0])) -elif len(symbol.headers) == 0: +if re.match("^div$|^remove$|atomic.*", symbol.name) and symbol.namespace == "std::": + continue +if len(symbol.headers) == 0: sys.stderr.write("No header found for symbol %s\n" % symbol.name) else: - # FIXME: support symbols with multiple headers (e.g. std::move). - sys.stderr.write("Ambiguous header for symbol %s: %s\n" % ( - symbol.name, ', '.join(symbol.headers))) - - + for header in symbol.headers: +# SYMBOL(unqualified_name, namespace, header) +print("SYMBOL(%s, %s, %s)" % (symbol.name, symbol.namespace, + header)) if __name__ == '__main__': main() Index: clang/tools/include-mapping/cppreference_parser.py === --- clang/tools/include-mapping/cppreference_parser.py +++ clang/tools/include-mapping/cppreference_parser.py @@ -118,7 +118,7 @@ return _ParseSymbolPage(f.read(), name) -def _GetSymbols(pool, root_dir, index_page_name, namespace, variants_to_accept): +def _GetSymbols(pool, root_dir, index_page_name, namespace): """Get all symbols listed in the index page. All symbols should be in the given namespace. @@ -135,11 +135,7 @@ # Read each symbol page in parallel. results = [] # (symbol_name, promise of [header...]) for symbol_name, symbol_page_path, variant in _ParseIndexPage(f.read()): - # Variant symbols (e.g. the std::locale version of isalpha) add ambiguity. - # FIXME: use these as a fallback rather than ignoring entirely. - variants_for_symbol = variants_to_accept.get( - (namespace or "") + symbol_name, ()) - if variant and variant not in variants_for_symbol: + if variant: continue path = os.path.join(root_dir, symbol_page_path) if os.path.isfile(path): @@ -166,13 +162,6 @@ Args: parse_pages: a list of tuples (page_root_dir, index_page_name, namespace) """ - # By default we prefer the non-variant versions, as they're more common. But - # there are some symbols, whose variant is more common. This list describes - # those symbols. - variants_to_accept = { - # std::remove<> has variant algorithm. - "std::remove": ("algorithm"), - } symbols = [] # Run many workers to process individual symbol pages under the symbol index. # Don't allow workers to capture Ctrl-C. @@ -180,8 +169,7 @@ initializer=lambda: signal.signal(signal.SIGINT, signal.SIG_IGN)) try: for root_dir, page_name, namespace in parse_pages: - symbols.extend(_GetSymbols(pool, root_dir, page_name, namespace, - variants_to_accept)) + symbols.extend(_GetSymbols(pool, root_dir, page_name, namespace)) finally: pool.terminate() pool.join() Index: clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp === --- clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp +++ clang/lib/Tooling/Inclusions/Stdlib/StandardLibrary.cpp @@ -8,6 +8,7 @@ #include "cl
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo added inline comments. Comment at: clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc:100 SYMBOL(atoll, std::, ) +SYMBOL(atomic, std::, ) +SYMBOL(atomic, std::, ) kadircet wrote: > hokein wrote: > > Conceptually, this (and other `atomic_*` symbols) doesn't feel correct: > > - `` provides the generic template `template struct > > atomic;` > > - `` provides partial template specializations for > > `std::shared_ptr` and `std::weak_ptr` > > > > They are variant symbols (ideally, they should be treat as the > > `std::move()`). The issue here is that unlike `std::move` which has two > > individual entries in the index page, we only have one entry for > > `std::atomic` (extending the cppreference_parser.py script to perfectly > > distinguish these two cases in the > > [page](https://en.cppreference.com/w/cpp/atomic/atomic) seems non-trivial). > > Some options: > > > > 1) treat them as multiple-header symbols (like what this patch does now) > > 2) special-case these symbols like `std::move()` > > 3) always prefer the header providing generic templates > > > > @kadircet, what do you think? > right, i believe this patch is definitely adding keeping multiple headers for > a symbol around, but mixing it with keeping headers for variants (e.g. > overloads provided by different headers, or specializations as mentioned > here). > > we definitely need some more changes in parser to make sure we're only > preserving alternatives for **the same symbol** and not for any other > variants (overloads, specializations). IIRC there are currently 2 places > these variants could come into play: > - first is the symbol index page itself, symbols that have ` (foo)` next to > them have variants and should still be ignored (e.g. std::remove variant of > cstdio shouldn't be preserved in the scope of this patch) > - second is inside the detail pages for symbols, e.g. > [std::atomic](http://go/std::atomic), we can have headers for different > declarations, they're clearly different variants so we shouldn't add such > symbols into the mapping `_ParseSymbolPage` already does some detection of > declarations in between headers. > > in the scope of this patch, we should keep ignoring both. I suggest to special-case the overloads for now, just not to solve all the problems in the same patch. Comment at: clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc:369 +SYMBOL(div, std::, ) +SYMBOL(div, std::, ) SYMBOL(div_t, std::, ) hokein wrote: > this one as well, both headers provide different overloads. I suggest to special-case the overloads for now, just not to solve all the problems in the same patch. Comment at: clang/tools/include-mapping/cppreference_parser.py:174 - # std::remove<> has variant algorithm. - "std::remove": ("algorithm"), - } kadircet wrote: > VitaNuo wrote: > > kadircet wrote: > > > this is actually checking for something else (sorry for the confusing > > > naming). > > > > > > the `variant` here refers to library name mentioned in parentheses (this > > > is same problem as `std::move`) on the std symbol index page > > > https://en.cppreference.com/w/cpp/symbol_index (e.g. `remove<>() > > > (algorithm)`). by getting rid of this we're introducing a regression, as > > > previously `std::remove` wouldn't be recognized by the library, but now > > > it'll be recognized and we'll keep suggesting `` for it. > > > > > > so we should actually keep this around. > > Ok, I can keep this out of this patch, but we'll have to remove this logic > > evetually when we deal with overloads. > > > > I have a slight suspicion that this code might be buggy, because it > > suggests that one _of_ the variants should be accepted. What is does in > > reality, though, is it keeps `algorithm` in the list of headers suitable > > for `std::remove` alongside `cstdio`, and then in the last step > > `std::remove` is ignored by the generator because of being defined in two > > headers. > > > > With this patch, the result will be both `{cstdio, algorithm}`. Is this > > (more) satisfactory for now compared to skipping `algorithm` due to being > > an overload? > > > > Ok, I can keep this out of this patch, but we'll have to remove this logic > > evetually when we deal with overloads. > > Surely, I wasn't saying this should stay here forever, i am just saying that > what's done in the scope of this patch doesn't really address the issues > "worked around" by this piece. > > > I have a slight suspicion that this code might be buggy, because it > > suggests that one _of_ the variants should be accepted. What is does in > > reality, though, is it keeps algorithm in the list of headers suitable for > > std::remove alongside cstdio, and then in the last step std::remove is > > ignored by the generator because of being defined in two headers. > > right, it's because we have log
[PATCH] D142092: [include-mapping] Allow multiple headers for the same symbol. Choose the first header of available ones.
VitaNuo added inline comments. Comment at: clang/include/clang/Tooling/Inclusions/StdSymbolMap.inc:1053 SYMBOL(remainder, std::, ) +SYMBOL(remove, std::, ) SYMBOL(remove_all_extents, std::, ) hokein wrote: > I think `std::remove` should not be in the scope of the patch, it has two > variants: > - std::remove from `` > - and std::remove from ``. Yes, agreed. Let's take care of the overloads going forward. Comment at: clang/tools/include-mapping/cppreference_parser.py:174 - # std::remove<> has variant algorithm. - "std::remove": ("algorithm"), - } VitaNuo wrote: > kadircet wrote: > > VitaNuo wrote: > > > kadircet wrote: > > > > this is actually checking for something else (sorry for the confusing > > > > naming). > > > > > > > > the `variant` here refers to library name mentioned in parentheses > > > > (this is same problem as `std::move`) on the std symbol index page > > > > https://en.cppreference.com/w/cpp/symbol_index (e.g. `remove<>() > > > > (algorithm)`). by getting rid of this we're introducing a regression, > > > > as previously `std::remove` wouldn't be recognized by the library, but > > > > now it'll be recognized and we'll keep suggesting `` for it. > > > > > > > > so we should actually keep this around. > > > Ok, I can keep this out of this patch, but we'll have to remove this > > > logic evetually when we deal with overloads. > > > > > > I have a slight suspicion that this code might be buggy, because it > > > suggests that one _of_ the variants should be accepted. What is does in > > > reality, though, is it keeps `algorithm` in the list of headers suitable > > > for `std::remove` alongside `cstdio`, and then in the last step > > > `std::remove` is ignored by the generator because of being defined in two > > > headers. > > > > > > With this patch, the result will be both `{cstdio, algorithm}`. Is this > > > (more) satisfactory for now compared to skipping `algorithm` due to being > > > an overload? > > > > > > Ok, I can keep this out of this patch, but we'll have to remove this > > > logic evetually when we deal with overloads. > > > > Surely, I wasn't saying this should stay here forever, i am just saying > > that what's done in the scope of this patch doesn't really address the > > issues "worked around" by this piece. > > > > > I have a slight suspicion that this code might be buggy, because it > > > suggests that one _of_ the variants should be accepted. What is does in > > > reality, though, is it keeps algorithm in the list of headers suitable > > > for std::remove alongside cstdio, and then in the last step std::remove > > > is ignored by the generator because of being defined in two headers. > > > > right, it's because we have logic to prefer "non-variant" versions of > > symbols when available (i.e. in the absence of this logic, we'd prefer > > std::remove from cstdio). this logic enables us to preserve certain > > variants (in addition to non-variants). that way we treat std::remove as > > ambigious rather than always resolving to , hence it's marked as > > "missing", similar to `std::move`. > > > > > With this patch, the result will be both {cstdio, algorithm}. Is this > > > (more) satisfactory for now compared to skipping algorithm due to being > > > an overload? > > > > in the end this should probably look like {algorithm, cstdio}, but as > > mentioned elsewhere, this is not the same problem as "same symbol being > > provided by multiple header" but rather "different symbols having same name > > and different headers". so treatment of it shouldn't change by this patch. > On second thought, I think we'd rather special-case the overloads for now > (until we get to them). See the newest patch version. > right, it's because we have logic to prefer "non-variant" versions of symbols > when available (i.e. in the absence of this logic, we'd prefer std::remove > from cstdio). Where is this logic? AFAICS the generator in the current state doesn't generate anything for std::remove. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D142092/new/ https://reviews.llvm.org/D142092 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D155614: [clangd] Make an include always refer to itself. Background: clang-review expects all referents to have definition, declaration or reference(s).
VitaNuo created this revision. Herald added subscribers: kadircet, arphaman. Herald added a project: All. VitaNuo requested review of this revision. Herald added subscribers: cfe-commits, MaskRay, ilya-biryukov. Herald added a project: clang-tools-extra. Repository: rG LLVM Github Monorepo https://reviews.llvm.org/D155614 Files: clang-tools-extra/clangd/XRefs.cpp Index: clang-tools-extra/clangd/XRefs.cpp === --- clang-tools-extra/clangd/XRefs.cpp +++ clang-tools-extra/clangd/XRefs.cpp @@ -1358,18 +1358,13 @@ Result.Loc.uri = URIMainFile; Results.References.push_back(std::move(Result)); }); - if (Results.References.empty()) -return std::nullopt; - + // Add the #include line to the references list. ReferencesResult::Reference Result; Result.Loc.range = rangeTillEOL(SM.getBufferData(SM.getMainFileID()), Inc.HashOffset); Result.Loc.uri = URIMainFile; Results.References.push_back(std::move(Result)); - - if (Results.References.empty()) -return std::nullopt; return Results; } } // namespace Index: clang-tools-extra/clangd/XRefs.cpp === --- clang-tools-extra/clangd/XRefs.cpp +++ clang-tools-extra/clangd/XRefs.cpp @@ -1358,18 +1358,13 @@ Result.Loc.uri = URIMainFile; Results.References.push_back(std::move(Result)); }); - if (Results.References.empty()) -return std::nullopt; - + // Add the #include line to the references list. ReferencesResult::Reference Result; Result.Loc.range = rangeTillEOL(SM.getBufferData(SM.getMainFileID()), Inc.HashOffset); Result.Loc.uri = URIMainFile; Results.References.push_back(std::move(Result)); - - if (Results.References.empty()) -return std::nullopt; return Results; } } // namespace ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D155614: [clangd] Remove redundant emptiness check in cross ref calculation for includes.
VitaNuo updated this revision to Diff 541882. VitaNuo marked 2 inline comments as done. VitaNuo added a comment. Remove only one of the checks. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D155614/new/ https://reviews.llvm.org/D155614 Files: clang-tools-extra/clangd/XRefs.cpp Index: clang-tools-extra/clangd/XRefs.cpp === --- clang-tools-extra/clangd/XRefs.cpp +++ clang-tools-extra/clangd/XRefs.cpp @@ -1367,9 +1367,6 @@ rangeTillEOL(SM.getBufferData(SM.getMainFileID()), Inc.HashOffset); Result.Loc.uri = URIMainFile; Results.References.push_back(std::move(Result)); - - if (Results.References.empty()) -return std::nullopt; return Results; } } // namespace Index: clang-tools-extra/clangd/XRefs.cpp === --- clang-tools-extra/clangd/XRefs.cpp +++ clang-tools-extra/clangd/XRefs.cpp @@ -1367,9 +1367,6 @@ rangeTillEOL(SM.getBufferData(SM.getMainFileID()), Inc.HashOffset); Result.Loc.uri = URIMainFile; Results.References.push_back(std::move(Result)); - - if (Results.References.empty()) -return std::nullopt; return Results; } } // namespace ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D155614: [clangd] Remove redundant emptiness check in cross ref calculation for includes.
VitaNuo accepted this revision. VitaNuo added inline comments. This revision is now accepted and ready to land. Comment at: clang-tools-extra/clangd/XRefs.cpp:1361-1362 }); - if (Results.References.empty()) -return std::nullopt; - usaxena95 wrote: > Sorry for the confusion. This looks intentional and somewhat valuable for > unused headers. We could fix this on clang-review's end as discussed on the > internal patch. We should also add a comment here explaining the rationale > behind returning no refs in such cases. > > (The check below certainly looks redundant though and could be removed). Yeah you're right the second check is redundant (probably an artefact of some unfinished refactoring). I will land this patch with the redundant check removed. Comment at: clang-tools-extra/clangd/XRefs.cpp:1365 // Add the #include line to the references list. ReferencesResult::Reference Result; Result.Loc.range = kadircet wrote: > i guess this patch is not needed anymore, but one comment about this > behaviour in clangd as well (no action needed in this patch) > > why are we providing the reference to the include line itself? in other > interactions we do that for the sake of completeness (e.g. if you trigger > xrefs on a reference, seeing the declaration/definition location is useful) > but usually the result under the cursor is not so interesting, especially in > this case. the include line itself is drastically different than the nature > of other references, the user can only see the include, if they triggered the > xrefs on the include line itself and not when they trigger it on the symbol. > also that ref doesn't count if there's no symbols used? > > FWIW I am not sure if that's a useful interaction. What was the rationale > here exactly? It might be useful to always emit this reference, but also > append provider to the refs list in the other direction as well (that way the > results will be coherent again) I think the rationale was that it provides more clarity in the VS Code UI. If you look back at the prototype image in the internal design doc, the references list always looks like a dropdown list with the containing file (in this case main file only) as title. In case of normal references, it shows the same symbol in different contexts, and it makes sense intuitively. In case of includes, it looks like a forest of unconnected symbols and might look like a bug to a user. IIRC that's the reason we initially agreed to add clarity by adding the include line itself. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D155614/new/ https://reviews.llvm.org/D155614 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[PATCH] D155614: [clangd] Remove redundant emptiness check in cross ref calculation for includes.
This revision was landed with ongoing or failed builds. This revision was automatically updated to reflect the committed changes. Closed by commit rG22605f5f1bf8: [clangd] Make an include always refer to itself. Background: clang-review… (authored by VitaNuo). Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D155614/new/ https://reviews.llvm.org/D155614 Files: clang-tools-extra/clangd/XRefs.cpp Index: clang-tools-extra/clangd/XRefs.cpp === --- clang-tools-extra/clangd/XRefs.cpp +++ clang-tools-extra/clangd/XRefs.cpp @@ -1367,9 +1367,6 @@ rangeTillEOL(SM.getBufferData(SM.getMainFileID()), Inc.HashOffset); Result.Loc.uri = URIMainFile; Results.References.push_back(std::move(Result)); - - if (Results.References.empty()) -return std::nullopt; return Results; } } // namespace Index: clang-tools-extra/clangd/XRefs.cpp === --- clang-tools-extra/clangd/XRefs.cpp +++ clang-tools-extra/clangd/XRefs.cpp @@ -1367,9 +1367,6 @@ rangeTillEOL(SM.getBufferData(SM.getMainFileID()), Inc.HashOffset); Result.Loc.uri = URIMainFile; Results.References.push_back(std::move(Result)); - - if (Results.References.empty()) -return std::nullopt; return Results; } } // namespace ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits