[PATCH] D123896: [clang-format] fix nested angle brackets parse inside concept definition

[Sergey Semushin via Phabricator via cfe-commits]

predelnik created this revision.
Herald added a project: All.
predelnik requested review of this revision.
Herald added a project: clang.
Herald added a subscriber: cfe-commits.

Due to how parseBracedList always stopped on the first closing angle
bracket and was used in parsing angle bracketed expression inside concept
definition, nested brackets inside concepts were parsed incorrectly.

nextToken() call before calling parseBracedList is required because
we were processing opening angle bracket inside parseBracedList second
time leading to incorrect logic after my fix.


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D123896

Files:
  clang/lib/Format/UnwrappedLineParser.cpp
  clang/unittests/Format/FormatTest.cpp


Index: clang/unittests/Format/FormatTest.cpp
===
--- clang/unittests/Format/FormatTest.cpp
+++ clang/unittests/Format/FormatTest.cpp
@@ -24040,6 +24040,12 @@
   verifyFormat("template \n"
"concept Same = __is_same_as;");
 
+  verifyFormat(
+  "template \n"
+  "concept _Can_reread_dest =\n"
+  "std::forward_iterator<_OutIt> &&\n"
+  "std::same_as, 
std::iter_value_t<_OutIt>>;");
+
   auto Style = getLLVMStyle();
   Style.BreakBeforeConceptDeclarations = FormatStyle::BBCDS_Allowed;
 
Index: clang/lib/Format/UnwrappedLineParser.cpp
===
--- clang/lib/Format/UnwrappedLineParser.cpp
+++ clang/lib/Format/UnwrappedLineParser.cpp
@@ -2174,7 +2174,8 @@
   parseBracedList();
   break;
 case tok::less:
-  if (Style.Language == FormatStyle::LK_Proto) {
+  if (Style.Language == FormatStyle::LK_Proto ||
+  ClosingBraceKind == tok::greater) {
 nextToken();
 parseBracedList(/*ContinueOnSemicolons=*/false, /*IsEnum=*/false,
 /*ClosingBraceKind=*/tok::greater);
@@ -3218,6 +3219,7 @@
   if (!FormatTok->is(tok::less))
 return;
 
+  nextToken();
   parseBracedList(/*ContinueOnSemicolons=*/false, /*IsEnum=*/false,
   /*ClosingBraceKind=*/tok::greater);
   break;
@@ -3258,9 +3260,11 @@
 
   // Read identifier with optional template declaration.
   nextToken();
-  if (FormatTok->is(tok::less))
+  if (FormatTok->is(tok::less)) {
+nextToken();
 parseBracedList(/*ContinueOnSemicolons=*/false, /*IsEnum=*/false,
 /*ClosingBraceKind=*/tok::greater);
+  }
   break;
 }
   } while (!eof());


Index: clang/unittests/Format/FormatTest.cpp
===
--- clang/unittests/Format/FormatTest.cpp
+++ clang/unittests/Format/FormatTest.cpp
@@ -24040,6 +24040,12 @@
   verifyFormat("template \n"
"concept Same = __is_same_as;");
 
+  verifyFormat(
+  "template \n"
+  "concept _Can_reread_dest =\n"
+  "std::forward_iterator<_OutIt> &&\n"
+  "std::same_as, std::iter_value_t<_OutIt>>;");
+
   auto Style = getLLVMStyle();
   Style.BreakBeforeConceptDeclarations = FormatStyle::BBCDS_Allowed;
 
Index: clang/lib/Format/UnwrappedLineParser.cpp
===
--- clang/lib/Format/UnwrappedLineParser.cpp
+++ clang/lib/Format/UnwrappedLineParser.cpp
@@ -2174,7 +2174,8 @@
   parseBracedList();
   break;
 case tok::less:
-  if (Style.Language == FormatStyle::LK_Proto) {
+  if (Style.Language == FormatStyle::LK_Proto ||
+  ClosingBraceKind == tok::greater) {
 nextToken();
 parseBracedList(/*ContinueOnSemicolons=*/false, /*IsEnum=*/false,
 /*ClosingBraceKind=*/tok::greater);
@@ -3218,6 +3219,7 @@
   if (!FormatTok->is(tok::less))
 return;
 
+  nextToken();
   parseBracedList(/*ContinueOnSemicolons=*/false, /*IsEnum=*/false,
   /*ClosingBraceKind=*/tok::greater);
   break;
@@ -3258,9 +3260,11 @@
 
   // Read identifier with optional template declaration.
   nextToken();
-  if (FormatTok->is(tok::less))
+  if (FormatTok->is(tok::less)) {
+nextToken();
 parseBracedList(/*ContinueOnSemicolons=*/false, /*IsEnum=*/false,
 /*ClosingBraceKind=*/tok::greater);
+  }
   break;
 }
   } while (!eof());
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D123896: [clang-format] fix nested angle brackets parse inside concept definition

[Sergey Semushin via Phabricator via cfe-commits]

predelnik added a comment.

In D123896#3455299 , @curdeius wrote:

> Is there an issue report for this bug already?

I could create an issue if that is required.

In D123896#3455301 , @curdeius wrote:

> Could you please add an annotator test as well?

Will look into that, thank you for reviewing.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D123896/new/

https://reviews.llvm.org/D123896

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D123896: [clang-format] fix nested angle brackets parse inside concept definition

[Sergey Semushin via Phabricator via cfe-commits]

predelnik updated this revision to Diff 423231.
predelnik added a comment.

Added annotator test, checked that it fails without a patch


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D123896/new/

https://reviews.llvm.org/D123896

Files:
  clang/lib/Format/UnwrappedLineParser.cpp
  clang/unittests/Format/FormatTest.cpp
  clang/unittests/Format/TokenAnnotatorTest.cpp


Index: clang/unittests/Format/TokenAnnotatorTest.cpp
===
--- clang/unittests/Format/TokenAnnotatorTest.cpp
+++ clang/unittests/Format/TokenAnnotatorTest.cpp
@@ -294,6 +294,18 @@
   EXPECT_TOKEN(Tokens[21], tok::r_brace, TT_Unknown);
   EXPECT_EQ(Tokens[21]->MatchingParen, Tokens[15]);
   EXPECT_TRUE(Tokens[21]->ClosesRequiresClause);
+
+  Tokens =
+  annotate("template  concept C ="
+   "std::same_as, std::iter_value_t>;");
+  ASSERT_EQ(Tokens.size(), 31u) << Tokens;
+  EXPECT_TOKEN(Tokens[8], tok::kw_concept, TT_Unknown);
+  EXPECT_TOKEN(Tokens[14], tok::less, TT_TemplateOpener);
+  EXPECT_TOKEN(Tokens[18], tok::less, TT_TemplateOpener);
+  EXPECT_TOKEN(Tokens[20], tok::greater, TT_TemplateCloser);
+  EXPECT_TOKEN(Tokens[25], tok::less, TT_TemplateOpener);
+  EXPECT_TOKEN(Tokens[27], tok::greater, TT_TemplateCloser);
+  EXPECT_TOKEN(Tokens[28], tok::greater, TT_TemplateCloser);
 }
 
 TEST_F(TokenAnnotatorTest, UnderstandsRequiresExpressions) {
Index: clang/unittests/Format/FormatTest.cpp
===
--- clang/unittests/Format/FormatTest.cpp
+++ clang/unittests/Format/FormatTest.cpp
@@ -24040,6 +24040,12 @@
   verifyFormat("template \n"
"concept Same = __is_same_as;");
 
+  verifyFormat(
+  "template \n"
+  "concept _Can_reread_dest =\n"
+  "std::forward_iterator<_OutIt> &&\n"
+  "std::same_as, 
std::iter_value_t<_OutIt>>;");
+
   auto Style = getLLVMStyle();
   Style.BreakBeforeConceptDeclarations = FormatStyle::BBCDS_Allowed;
 
Index: clang/lib/Format/UnwrappedLineParser.cpp
===
--- clang/lib/Format/UnwrappedLineParser.cpp
+++ clang/lib/Format/UnwrappedLineParser.cpp
@@ -2174,7 +2174,8 @@
   parseBracedList();
   break;
 case tok::less:
-  if (Style.Language == FormatStyle::LK_Proto) {
+  if (Style.Language == FormatStyle::LK_Proto ||
+  ClosingBraceKind == tok::greater) {
 nextToken();
 parseBracedList(/*ContinueOnSemicolons=*/false, /*IsEnum=*/false,
 /*ClosingBraceKind=*/tok::greater);
@@ -3218,6 +3219,7 @@
   if (!FormatTok->is(tok::less))
 return;
 
+  nextToken();
   parseBracedList(/*ContinueOnSemicolons=*/false, /*IsEnum=*/false,
   /*ClosingBraceKind=*/tok::greater);
   break;
@@ -3258,9 +3260,11 @@
 
   // Read identifier with optional template declaration.
   nextToken();
-  if (FormatTok->is(tok::less))
+  if (FormatTok->is(tok::less)) {
+nextToken();
 parseBracedList(/*ContinueOnSemicolons=*/false, /*IsEnum=*/false,
 /*ClosingBraceKind=*/tok::greater);
+  }
   break;
 }
   } while (!eof());


Index: clang/unittests/Format/TokenAnnotatorTest.cpp
===
--- clang/unittests/Format/TokenAnnotatorTest.cpp
+++ clang/unittests/Format/TokenAnnotatorTest.cpp
@@ -294,6 +294,18 @@
   EXPECT_TOKEN(Tokens[21], tok::r_brace, TT_Unknown);
   EXPECT_EQ(Tokens[21]->MatchingParen, Tokens[15]);
   EXPECT_TRUE(Tokens[21]->ClosesRequiresClause);
+
+  Tokens =
+  annotate("template  concept C ="
+   "std::same_as, std::iter_value_t>;");
+  ASSERT_EQ(Tokens.size(), 31u) << Tokens;
+  EXPECT_TOKEN(Tokens[8], tok::kw_concept, TT_Unknown);
+  EXPECT_TOKEN(Tokens[14], tok::less, TT_TemplateOpener);
+  EXPECT_TOKEN(Tokens[18], tok::less, TT_TemplateOpener);
+  EXPECT_TOKEN(Tokens[20], tok::greater, TT_TemplateCloser);
+  EXPECT_TOKEN(Tokens[25], tok::less, TT_TemplateOpener);
+  EXPECT_TOKEN(Tokens[27], tok::greater, TT_TemplateCloser);
+  EXPECT_TOKEN(Tokens[28], tok::greater, TT_TemplateCloser);
 }
 
 TEST_F(TokenAnnotatorTest, UnderstandsRequiresExpressions) {
Index: clang/unittests/Format/FormatTest.cpp
===
--- clang/unittests/Format/FormatTest.cpp
+++ clang/unittests/Format/FormatTest.cpp
@@ -24040,6 +24040,12 @@
   verifyFormat("template \n"
"concept Same = __is_same_as;");
 
+  verifyFormat(
+  "template \n"
+  "concept _Can_reread_dest =\n"
+  "std::forward_iterator<_OutIt> &&\n"
+  "std::same_as, std::iter_value_t<_OutIt>>;");
+
   auto Style = getLLVMStyle();
   Style.BreakBeforeConceptDeclarations = FormatStyle::BBCDS_Allowed;
 
Index: clang/lib/Format/UnwrappedLineParser.cpp
=

[PATCH] D123896: [clang-format] fix nested angle brackets parse inside concept definition

[Sergey Semushin via Phabricator via cfe-commits]

predelnik added a comment.

I've created the issue here:
https://github.com/llvm/llvm-project/issues/54943
It also seems to fix the other issue I reported which was partially fixed 
before this change in master:
https://github.com/llvm/llvm-project/issues/54837


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D123896/new/

https://reviews.llvm.org/D123896

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D123896: [clang-format] fix nested angle brackets parse inside concept definition

[Sergey Semushin via Phabricator via cfe-commits]

predelnik updated this revision to Diff 423234.
predelnik edited the summary of this revision.
predelnik added a comment.

Added issue links to commit message


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D123896/new/

https://reviews.llvm.org/D123896

Files:
  clang/lib/Format/UnwrappedLineParser.cpp
  clang/unittests/Format/FormatTest.cpp
  clang/unittests/Format/TokenAnnotatorTest.cpp


Index: clang/unittests/Format/TokenAnnotatorTest.cpp
===
--- clang/unittests/Format/TokenAnnotatorTest.cpp
+++ clang/unittests/Format/TokenAnnotatorTest.cpp
@@ -294,6 +294,18 @@
   EXPECT_TOKEN(Tokens[21], tok::r_brace, TT_Unknown);
   EXPECT_EQ(Tokens[21]->MatchingParen, Tokens[15]);
   EXPECT_TRUE(Tokens[21]->ClosesRequiresClause);
+
+  Tokens =
+  annotate("template  concept C ="
+   "std::same_as, std::iter_value_t>;");
+  ASSERT_EQ(Tokens.size(), 31u) << Tokens;
+  EXPECT_TOKEN(Tokens[8], tok::kw_concept, TT_Unknown);
+  EXPECT_TOKEN(Tokens[14], tok::less, TT_TemplateOpener);
+  EXPECT_TOKEN(Tokens[18], tok::less, TT_TemplateOpener);
+  EXPECT_TOKEN(Tokens[20], tok::greater, TT_TemplateCloser);
+  EXPECT_TOKEN(Tokens[25], tok::less, TT_TemplateOpener);
+  EXPECT_TOKEN(Tokens[27], tok::greater, TT_TemplateCloser);
+  EXPECT_TOKEN(Tokens[28], tok::greater, TT_TemplateCloser);
 }
 
 TEST_F(TokenAnnotatorTest, UnderstandsRequiresExpressions) {
Index: clang/unittests/Format/FormatTest.cpp
===
--- clang/unittests/Format/FormatTest.cpp
+++ clang/unittests/Format/FormatTest.cpp
@@ -24040,6 +24040,12 @@
   verifyFormat("template \n"
"concept Same = __is_same_as;");
 
+  verifyFormat(
+  "template \n"
+  "concept _Can_reread_dest =\n"
+  "std::forward_iterator<_OutIt> &&\n"
+  "std::same_as, 
std::iter_value_t<_OutIt>>;");
+
   auto Style = getLLVMStyle();
   Style.BreakBeforeConceptDeclarations = FormatStyle::BBCDS_Allowed;
 
Index: clang/lib/Format/UnwrappedLineParser.cpp
===
--- clang/lib/Format/UnwrappedLineParser.cpp
+++ clang/lib/Format/UnwrappedLineParser.cpp
@@ -2174,7 +2174,8 @@
   parseBracedList();
   break;
 case tok::less:
-  if (Style.Language == FormatStyle::LK_Proto) {
+  if (Style.Language == FormatStyle::LK_Proto ||
+  ClosingBraceKind == tok::greater) {
 nextToken();
 parseBracedList(/*ContinueOnSemicolons=*/false, /*IsEnum=*/false,
 /*ClosingBraceKind=*/tok::greater);
@@ -3218,6 +3219,7 @@
   if (!FormatTok->is(tok::less))
 return;
 
+  nextToken();
   parseBracedList(/*ContinueOnSemicolons=*/false, /*IsEnum=*/false,
   /*ClosingBraceKind=*/tok::greater);
   break;
@@ -3258,9 +3260,11 @@
 
   // Read identifier with optional template declaration.
   nextToken();
-  if (FormatTok->is(tok::less))
+  if (FormatTok->is(tok::less)) {
+nextToken();
 parseBracedList(/*ContinueOnSemicolons=*/false, /*IsEnum=*/false,
 /*ClosingBraceKind=*/tok::greater);
+  }
   break;
 }
   } while (!eof());


Index: clang/unittests/Format/TokenAnnotatorTest.cpp
===
--- clang/unittests/Format/TokenAnnotatorTest.cpp
+++ clang/unittests/Format/TokenAnnotatorTest.cpp
@@ -294,6 +294,18 @@
   EXPECT_TOKEN(Tokens[21], tok::r_brace, TT_Unknown);
   EXPECT_EQ(Tokens[21]->MatchingParen, Tokens[15]);
   EXPECT_TRUE(Tokens[21]->ClosesRequiresClause);
+
+  Tokens =
+  annotate("template  concept C ="
+   "std::same_as, std::iter_value_t>;");
+  ASSERT_EQ(Tokens.size(), 31u) << Tokens;
+  EXPECT_TOKEN(Tokens[8], tok::kw_concept, TT_Unknown);
+  EXPECT_TOKEN(Tokens[14], tok::less, TT_TemplateOpener);
+  EXPECT_TOKEN(Tokens[18], tok::less, TT_TemplateOpener);
+  EXPECT_TOKEN(Tokens[20], tok::greater, TT_TemplateCloser);
+  EXPECT_TOKEN(Tokens[25], tok::less, TT_TemplateOpener);
+  EXPECT_TOKEN(Tokens[27], tok::greater, TT_TemplateCloser);
+  EXPECT_TOKEN(Tokens[28], tok::greater, TT_TemplateCloser);
 }
 
 TEST_F(TokenAnnotatorTest, UnderstandsRequiresExpressions) {
Index: clang/unittests/Format/FormatTest.cpp
===
--- clang/unittests/Format/FormatTest.cpp
+++ clang/unittests/Format/FormatTest.cpp
@@ -24040,6 +24040,12 @@
   verifyFormat("template \n"
"concept Same = __is_same_as;");
 
+  verifyFormat(
+  "template \n"
+  "concept _Can_reread_dest =\n"
+  "std::forward_iterator<_OutIt> &&\n"
+  "std::same_as, std::iter_value_t<_OutIt>>;");
+
   auto Style = getLLVMStyle();
   Style.BreakBeforeConceptDeclarations = FormatStyle::BBCDS_Allowed;
 
Index: clang/lib/Format/UnwrappedLineParser.cpp
==

[PATCH] D123896: [clang-format] fix nested angle brackets parse inside concept definition

[Sergey Semushin via Phabricator via cfe-commits]

predelnik added a comment.

In D123896#3464171 , 
@HazardyKnusperkeks wrote:

> As this is still on my todo list: Have you considered
>
>   concept X = Bar<5 < 4, 5 > 8>;
>   concept X = Bar<5 < 4, false>;
>
> Or stuff like that?

I think it would not work but would not make things much worse than before, 
have to check that though.
Looks like `parseBracedList` probably would not be enough for correct parsing 
of such expressions, is there any better approach?


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D123896/new/

https://reviews.llvm.org/D123896

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D123896: [clang-format] fix nested angle brackets parse inside concept definition

[Sergey Semushin via Phabricator via cfe-commits]

predelnik added a comment.

I see now, would certainly like to help but it sounds like I might not be 
competent enough for this. I guess this review could be closed unless it's an 
acceptable half-measure, not sure.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D123896/new/

https://reviews.llvm.org/D123896

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D123896: [clang-format] fix nested angle brackets parse inside concept definition

[Sergey Semushin via Phabricator via cfe-commits]

predelnik added a comment.

Probably, yes. I didn't know I have to do it myself, sorry.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D123896/new/

https://reviews.llvm.org/D123896

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D123896: [clang-format] fix nested angle brackets parse inside concept definition

[Sergey Semushin via Phabricator via cfe-commits]

predelnik added a comment.

Ok I've found it on a guide here at last 
https://llvm.org/docs/MyFirstTypoFix.html
Please land a patch for me, @curdeius, my author data is “Sergey Semushin 
predel...@gmail.com”
Thank you.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D123896/new/

https://reviews.llvm.org/D123896

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits