[PATCH] D122549: [VFS] RedirectingFileSystem only replace path if not already mapped

[Ben Barham via Phabricator via cfe-commits]

bnbarham created this revision.
bnbarham added reviewers: dexonsmith, keith, JDevlieghere, vsapsai, sammccall.
Herald added a subscriber: hiraditya.
Herald added a project: All.
bnbarham requested review of this revision.
Herald added projects: clang, LLVM.
Herald added subscribers: llvm-commits, cfe-commits.

If the `ExternalFS` has already remapped a path then the
`RedirectingFileSystem` should not change it to the originally provided
path. This fixes the original path always being used if multiple VFS
overlays were provided and the path wasn't found in the highest (ie.
first in the chain).

This also changes `IsVFSMapped` to mean the returned path is a *mapped*
path, rather than just from a virtual filesystem. But it was only being used
by a hack in `FileManager` for module searching, where `external-names: true`
is always used.

Resolves rdar://90578880 and llvm-project#53306.


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D122549

Files:
  clang/lib/Basic/FileManager.cpp
  clang/test/VFS/external-names-multi-overlay.c
  llvm/include/llvm/Support/VirtualFileSystem.h
  llvm/lib/Support/VirtualFileSystem.cpp
  llvm/unittests/Support/VirtualFileSystemTest.cpp

Index: llvm/unittests/Support/VirtualFileSystemTest.cpp
===
--- llvm/unittests/Support/VirtualFileSystemTest.cpp
+++ llvm/unittests/Support/VirtualFileSystemTest.cpp
@@ -1478,16 +1478,16 @@
   // file in remapped directory
   S = O->status("//root/mappeddir/a");
   ASSERT_FALSE(S.getError());
-  ASSERT_FALSE(S->isDirectory());
-  ASSERT_TRUE(S->IsVFSMapped);
-  ASSERT_EQ("//root/foo/bar/a", S->getName());
+  EXPECT_FALSE(S->isDirectory());
+  EXPECT_TRUE(S->IsVFSMapped);
+  EXPECT_EQ("//root/foo/bar/a", S->getName());
 
   // file in remapped directory, with use-external-name=false
   S = O->status("//root/mappeddir2/a");
   ASSERT_FALSE(S.getError());
-  ASSERT_FALSE(S->isDirectory());
-  ASSERT_TRUE(S->IsVFSMapped);
-  ASSERT_EQ("//root/mappeddir2/a", S->getName());
+  EXPECT_FALSE(S->isDirectory());
+  EXPECT_FALSE(S->IsVFSMapped);
+  EXPECT_EQ("//root/mappeddir2/a", S->getName());
 
   // file contents in remapped directory
   OpenedF = O->openFileForRead("//root/mappeddir/a");
@@ -1503,7 +1503,7 @@
   OpenedS = (*OpenedF)->status();
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("//root/mappeddir2/a", OpenedS->getName());
-  EXPECT_TRUE(OpenedS->IsVFSMapped);
+  EXPECT_FALSE(OpenedS->IsVFSMapped);
 
   // broken mapping
   EXPECT_EQ(O->status("//root/file2").getError(),
@@ -1776,12 +1776,12 @@
   auto OpenedS = (*OpenedF)->status();
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("vfsname", OpenedS->getName());
-  EXPECT_TRUE(OpenedS->IsVFSMapped);
+  EXPECT_FALSE(OpenedS->IsVFSMapped);
 
   auto DirectS = FS->status("vfsname");
   ASSERT_FALSE(DirectS.getError());
   EXPECT_EQ("vfsname", DirectS->getName());
-  EXPECT_TRUE(DirectS->IsVFSMapped);
+  EXPECT_FALSE(DirectS->IsVFSMapped);
 
   EXPECT_EQ(0, NumDiagnostics);
 }
Index: llvm/lib/Support/VirtualFileSystem.cpp
===
--- llvm/lib/Support/VirtualFileSystem.cpp
+++ llvm/lib/Support/VirtualFileSystem.cpp
@@ -2163,10 +2163,16 @@
 static Status getRedirectedFileStatus(const Twine &OriginalPath,
   bool UseExternalNames,
   Status ExternalStatus) {
+  // The path has been mapped by some nested VFS, don't override it with the
+  // original path.
+  if (ExternalStatus.IsVFSMapped)
+return ExternalStatus;
+
   Status S = ExternalStatus;
   if (!UseExternalNames)
 S = Status::copyWithNewName(S, OriginalPath);
-  S.IsVFSMapped = true;
+  else
+S.IsVFSMapped = true;
   return S;
 }
 
@@ -2268,7 +2274,9 @@
 
 ErrorOr>
 File::getWithPath(ErrorOr> Result, const Twine &P) {
-  if (!Result)
+  // See \c getRedirectedFileStatus - don't update path if it's already been
+  // mapped.
+  if (!Result || (*Result)->status()->IsVFSMapped)
 return Result;
 
   ErrorOr> F = std::move(*Result);
Index: llvm/include/llvm/Support/VirtualFileSystem.h
===
--- llvm/include/llvm/Support/VirtualFileSystem.h
+++ llvm/include/llvm/Support/VirtualFileSystem.h
@@ -55,8 +55,12 @@
   llvm::sys::fs::perms Perms;
 
 public:
-  // FIXME: remove when files support multiple names
+  /// Whether the path in this status has been mapped by a VFS to another path.
   bool IsVFSMapped = false;
+  // Note: Currently used by a hack in \c FileManager and internally in
+  // \c RedirectingFileSystem. We should change this to "HasVFSMapping" and
+  // *always* return the virtual path, only providing the mapped/external path
+  // when requested.
 
   Status() = default;
   Status(const llvm::sys::fs::file_status &Status);
Index: clang/test/VFS/external-names-multi-overlay.c
===
--- /d

[PATCH] D122549: [VFS] RedirectingFileSystem only replace path if not already mapped

[Ben Barham via Phabricator via cfe-commits]

bnbarham added a comment.

`clang-apply-replacements/relative-paths.cpp` is failing, I haven't looked into 
it but my guess would be that it's from the `Status.getName() == Filename` -> 
`!Status.IsVFSMapped` change. That seems very odd to me.




Comment at: clang/lib/Basic/FileManager.cpp:287-289
 // name-as-accessed on the \a FileEntryRef. Maybe the returned \a
 // FileEntryRef::getName() could return the accessed name unmodified, but
 // make the external name available via a separate API.

dexonsmith wrote:
> I'd suggest updating this text to match the new FIXME next to `IsVFSMapped` 
> (instead of "maybe", list the plan).
Will do



Comment at: clang/lib/Basic/FileManager.cpp:316-318
+// case above is removed. That one can be removed once we always return
+// virtual paths and anywhere that needs external paths specifically
+// requests them.

dexonsmith wrote:
> It's not obvious to me that the redirection case above depends on this logic 
> sticking around.
> - If that's what you mean, can you explain in more detail why the redirection 
> above depends on this `UFE.Dir` logic?
> - If you're just talking about `IsVFSMapped` having to stick around, I think 
> that part should be covered by the comment for the redirection.
I actually meant the reverse - the UFE.Dir logic being removed depends on the 
redirection logic above being removed. That's because for this to be removed, 
anywhere that cares about the requested path needs to be changed to use Refs. 
But even if they change to use Refs, the redirection logic above would itself 
replace the path with the external one. So it needs to be removed as well.



Comment at: llvm/include/llvm/Support/VirtualFileSystem.h:58-63
+  /// Whether the path in this status has been mapped by a VFS to another path.
   bool IsVFSMapped = false;
+  // Note: Currently used by a hack in \c FileManager and internally in
+  // \c RedirectingFileSystem. We should change this to "HasVFSMapping" and
+  // *always* return the virtual path, only providing the mapped/external path
+  // when requested.

dexonsmith wrote:
> I think "mapped to another path" has a broader definition in my mind than 
> you're using here. IMO, all things in a redirecting filesystem map from one 
> path to another. The question is whether the mapping is leaked/available due 
> to use-external-names.
> 
> I think this comment be more precise, and the note should be a FIXME that's 
> part of the main comment, not something that comes later.
> 
> I also think it good to change the field name ~now (or in an immediately 
> following NFC patch) to reflect the new meaning. I like "exposes" or "leaks" 
> because that is clear that this indicates whether information about the 
> mapping is available / exposed in the abstraction, whereas "mapped" and 
> "mapping" sound to me like they're just talking about whether something was 
> redirected. And I like the word "external" because it ties back to 
> use-external-names.
> 
> Suggested text (feel free to change entirely, but I think it covers a couple 
> of important points):
> ```
> /// Whether this entity has an external path different from the virtual path, 
> and
> /// the external path is exposed by leaking it through the abstraction. For 
> example,
> /// a RedirectingFileSystem will set this for paths where UseExternalName is 
> true.
> ///
> /// FIXME: Currently the external path is exposed by replacing the virtual 
> path
> /// in this Status object. Instead, we should leave the path in the Status 
> intact
> /// (matching the requested virtual path), and just use this flag as hint 
> that a
> /// new API, FileSystem::getExternalVFSPath(), will not return `None`. 
> Clients can
> /// then request the external path only where needed (e.g., for diagnostics, 
> or to
> /// report discovered dependencies to external tools).
> bool ExposesExternalVFSPath = false;
> ```
> 
> 
That's fair. I suppose I was thinking that without `use-external-names: true` 
then nothing should be aware that it is "mapped" and thus "mapped" == 
"external". But I can see how that could be confusing.

I also thought that renaming now would be a little odd since eg. 
"HasExternalPath" seems a little different to "the actual path is currently the 
external path" (which is what it is right now). But with the "exposes" naming I 
think this could still make sense (especially with the FIXME you have). So I'll 
rename 👍 


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D122549/new/

https://reviews.llvm.org/D122549

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D122549: [VFS] RedirectingFileSystem only replace path if not already mapped

[Ben Barham via Phabricator via cfe-commits]

bnbarham added a comment.

In D122549#3412064 , @dexonsmith 
wrote:

> In D122549#3412021 , @bnbarham 
> wrote:
>
>> `clang-apply-replacements/relative-paths.cpp` is failing, I haven't looked 
>> into it but my guess would be that it's from the `Status.getName() == 
>> Filename` -> `!Status.IsVFSMapped` change. That seems very odd to me.
>
> Is it just failing on Windows? I wonder (rather speculatively...) whether 
> https://reviews.llvm.org/D121733 would help.

No, also debian. Not sure why it isn't saying debian failed in the review.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D122549/new/

https://reviews.llvm.org/D122549

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D122549: [VFS] RedirectingFileSystem only replace path if not already mapped

[Ben Barham via Phabricator via cfe-commits]

bnbarham updated this revision to Diff 418721.
bnbarham edited the summary of this revision.
bnbarham added a comment.

Rename `IsVFSMapped` as suggested by Duncan. Update comments.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D122549/new/

https://reviews.llvm.org/D122549

Files:
  clang/lib/Basic/FileManager.cpp
  clang/test/VFS/external-names-multi-overlay.c
  llvm/include/llvm/Support/VirtualFileSystem.h
  llvm/lib/Support/VirtualFileSystem.cpp
  llvm/unittests/Support/VirtualFileSystemTest.cpp

Index: llvm/unittests/Support/VirtualFileSystemTest.cpp
===
--- llvm/unittests/Support/VirtualFileSystemTest.cpp
+++ llvm/unittests/Support/VirtualFileSystemTest.cpp
@@ -1442,12 +1442,12 @@
   ErrorOr S = O->status("//root/file1");
   ASSERT_FALSE(S.getError());
   EXPECT_EQ("//root/foo/bar/a", S->getName());
-  EXPECT_TRUE(S->IsVFSMapped);
+  EXPECT_TRUE(S->ExposesExternalVFSPath);
 
   ErrorOr SLower = O->status("//root/foo/bar/a");
   EXPECT_EQ("//root/foo/bar/a", SLower->getName());
   EXPECT_TRUE(S->equivalent(*SLower));
-  EXPECT_FALSE(SLower->IsVFSMapped);
+  EXPECT_FALSE(SLower->ExposesExternalVFSPath);
 
   // file after opening
   auto OpenedF = O->openFileForRead("//root/file1");
@@ -1455,7 +1455,7 @@
   auto OpenedS = (*OpenedF)->status();
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("//root/foo/bar/a", OpenedS->getName());
-  EXPECT_TRUE(OpenedS->IsVFSMapped);
+  EXPECT_TRUE(OpenedS->ExposesExternalVFSPath);
 
   // directory
   S = O->status("//root/");
@@ -1467,27 +1467,27 @@
   S = O->status("//root/mappeddir");
   ASSERT_FALSE(S.getError());
   EXPECT_TRUE(S->isDirectory());
-  EXPECT_TRUE(S->IsVFSMapped);
+  EXPECT_TRUE(S->ExposesExternalVFSPath);
   EXPECT_TRUE(S->equivalent(*O->status("//root/foo/bar")));
 
   SLower = O->status("//root/foo/bar");
   EXPECT_EQ("//root/foo/bar", SLower->getName());
   EXPECT_TRUE(S->equivalent(*SLower));
-  EXPECT_FALSE(SLower->IsVFSMapped);
+  EXPECT_FALSE(SLower->ExposesExternalVFSPath);
 
   // file in remapped directory
   S = O->status("//root/mappeddir/a");
   ASSERT_FALSE(S.getError());
-  ASSERT_FALSE(S->isDirectory());
-  ASSERT_TRUE(S->IsVFSMapped);
-  ASSERT_EQ("//root/foo/bar/a", S->getName());
+  EXPECT_FALSE(S->isDirectory());
+  EXPECT_TRUE(S->ExposesExternalVFSPath);
+  EXPECT_EQ("//root/foo/bar/a", S->getName());
 
   // file in remapped directory, with use-external-name=false
   S = O->status("//root/mappeddir2/a");
   ASSERT_FALSE(S.getError());
-  ASSERT_FALSE(S->isDirectory());
-  ASSERT_TRUE(S->IsVFSMapped);
-  ASSERT_EQ("//root/mappeddir2/a", S->getName());
+  EXPECT_FALSE(S->isDirectory());
+  EXPECT_FALSE(S->ExposesExternalVFSPath);
+  EXPECT_EQ("//root/mappeddir2/a", S->getName());
 
   // file contents in remapped directory
   OpenedF = O->openFileForRead("//root/mappeddir/a");
@@ -1495,7 +1495,7 @@
   OpenedS = (*OpenedF)->status();
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("//root/foo/bar/a", OpenedS->getName());
-  EXPECT_TRUE(OpenedS->IsVFSMapped);
+  EXPECT_TRUE(OpenedS->ExposesExternalVFSPath);
 
   // file contents in remapped directory, with use-external-name=false
   OpenedF = O->openFileForRead("//root/mappeddir2/a");
@@ -1503,7 +1503,7 @@
   OpenedS = (*OpenedF)->status();
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("//root/mappeddir2/a", OpenedS->getName());
-  EXPECT_TRUE(OpenedS->IsVFSMapped);
+  EXPECT_FALSE(OpenedS->ExposesExternalVFSPath);
 
   // broken mapping
   EXPECT_EQ(O->status("//root/file2").getError(),
@@ -1535,12 +1535,12 @@
   ErrorOr S = O->status("//mappedroot/a");
   ASSERT_FALSE(S.getError());
   EXPECT_EQ("//root/foo/bar/a", S->getName());
-  EXPECT_TRUE(S->IsVFSMapped);
+  EXPECT_TRUE(S->ExposesExternalVFSPath);
 
   ErrorOr SLower = O->status("//root/foo/bar/a");
   EXPECT_EQ("//root/foo/bar/a", SLower->getName());
   EXPECT_TRUE(S->equivalent(*SLower));
-  EXPECT_FALSE(SLower->IsVFSMapped);
+  EXPECT_FALSE(SLower->ExposesExternalVFSPath);
 
   // file after opening
   auto OpenedF = O->openFileForRead("//mappedroot/a");
@@ -1548,7 +1548,7 @@
   auto OpenedS = (*OpenedF)->status();
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("//root/foo/bar/a", OpenedS->getName());
-  EXPECT_TRUE(OpenedS->IsVFSMapped);
+  EXPECT_TRUE(OpenedS->ExposesExternalVFSPath);
 
   EXPECT_EQ(0, NumDiagnostics);
 }
@@ -1696,12 +1696,12 @@
   auto OpenedS = (*OpenedF)->status();
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("a", OpenedS->getName());
-  EXPECT_FALSE(OpenedS->IsVFSMapped);
+  EXPECT_FALSE(OpenedS->ExposesExternalVFSPath);
 
   auto DirectS = RemappedFS->status("a");
   ASSERT_FALSE(DirectS.getError());
   EXPECT_EQ("a", DirectS->getName());
-  EXPECT_FALSE(DirectS->IsVFSMapped);
+  EXPECT_FALSE(DirectS->ExposesExternalVFSPath);
 
   EXPECT_EQ(0, NumDiagnostics);
 }
@@ -1736,12 +1736,12 @@
   auto OpenedS = (*OpenedF)->status();
   ASSERT_FALSE(OpenedS.getError());
   EXPE

[PATCH] D122549: [VFS] RedirectingFileSystem only replace path if not already mapped

[Ben Barham via Phabricator via cfe-commits]

bnbarham marked 4 inline comments as done.
bnbarham added inline comments.



Comment at: clang/lib/Basic/FileManager.cpp:273
 
-  if (Status.getName() == Filename) {
+  if (!Status.IsVFSMapped) {
 // The name matches. Set the FileEntry.

dexonsmith wrote:
> An incremental patch you could try would be:
> ```
> lang=c++
> if (Status.getName() == Filename || !Status.IsVFSMapped)
> ```
> ... dropping all the other changes.
> 
> This limits the redirection hack to only apply when a RedirectingFS is 
> involved (leaving until later the fine-tuning of when `IsVFSMapped` gets 
> set). If this smaller change still causes a test failure, it might be easier 
> to reason about why / how to fix it / be sure that the fix is sound.
> 
> Eventually we might want something like:
> ```
> lang=c++
> if (!Status.IsVFSMapped) {
>   assert(Status.getName() == Filename);
> ```
> I imagine that's not going to succeed yet due to the CWD behaviour in the 
> VFSes, but as a speculative patch it might help track down whatever the 
> problem is.
That assertion currently won't succeed for any relative path, since 
`getStatValue` does the absolute pathing for relative paths. So 
`Status.getName()` is guaranteed to be different to `Filename` in that case.

Possibly even more odd is that the failing test passes locally on MacOSX. I'll 
try the suggestion above and see if they fail again.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D122549/new/

https://reviews.llvm.org/D122549

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D122549: [VFS] RedirectingFileSystem only replace path if not already mapped

[Ben Barham via Phabricator via cfe-commits]

bnbarham added inline comments.



Comment at: clang/lib/Basic/FileManager.cpp:273
 
-  if (Status.getName() == Filename) {
+  if (!Status.IsVFSMapped) {
 // The name matches. Set the FileEntry.

bnbarham wrote:
> dexonsmith wrote:
> > An incremental patch you could try would be:
> > ```
> > lang=c++
> > if (Status.getName() == Filename || !Status.IsVFSMapped)
> > ```
> > ... dropping all the other changes.
> > 
> > This limits the redirection hack to only apply when a RedirectingFS is 
> > involved (leaving until later the fine-tuning of when `IsVFSMapped` gets 
> > set). If this smaller change still causes a test failure, it might be 
> > easier to reason about why / how to fix it / be sure that the fix is sound.
> > 
> > Eventually we might want something like:
> > ```
> > lang=c++
> > if (!Status.IsVFSMapped) {
> >   assert(Status.getName() == Filename);
> > ```
> > I imagine that's not going to succeed yet due to the CWD behaviour in the 
> > VFSes, but as a speculative patch it might help track down whatever the 
> > problem is.
> That assertion currently won't succeed for any relative path, since 
> `getStatValue` does the absolute pathing for relative paths. So 
> `Status.getName()` is guaranteed to be different to `Filename` in that case.
> 
> Possibly even more odd is that the failing test passes locally on MacOSX. 
> I'll try the suggestion above and see if they fail again.
Actually, thinking about this a bit more - the failing test doesn't use an 
overlay at all, so changing to the above wouldn't help at all.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D122549/new/

https://reviews.llvm.org/D122549

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D122549: [VFS] RedirectingFileSystem only replace path if not already mapped

[Ben Barham via Phabricator via cfe-commits]

bnbarham marked an inline comment as not done.
bnbarham added inline comments.



Comment at: clang/lib/Basic/FileManager.cpp:273
 
-  if (Status.getName() == Filename) {
+  if (!Status.IsVFSMapped) {
 // The name matches. Set the FileEntry.

dexonsmith wrote:
> bnbarham wrote:
> > bnbarham wrote:
> > > dexonsmith wrote:
> > > > An incremental patch you could try would be:
> > > > ```
> > > > lang=c++
> > > > if (Status.getName() == Filename || !Status.IsVFSMapped)
> > > > ```
> > > > ... dropping all the other changes.
> > > > 
> > > > This limits the redirection hack to only apply when a RedirectingFS is 
> > > > involved (leaving until later the fine-tuning of when `IsVFSMapped` 
> > > > gets set). If this smaller change still causes a test failure, it might 
> > > > be easier to reason about why / how to fix it / be sure that the fix is 
> > > > sound.
> > > > 
> > > > Eventually we might want something like:
> > > > ```
> > > > lang=c++
> > > > if (!Status.IsVFSMapped) {
> > > >   assert(Status.getName() == Filename);
> > > > ```
> > > > I imagine that's not going to succeed yet due to the CWD behaviour in 
> > > > the VFSes, but as a speculative patch it might help track down whatever 
> > > > the problem is.
> > > That assertion currently won't succeed for any relative path, since 
> > > `getStatValue` does the absolute pathing for relative paths. So 
> > > `Status.getName()` is guaranteed to be different to `Filename` in that 
> > > case.
> > > 
> > > Possibly even more odd is that the failing test passes locally on MacOSX. 
> > > I'll try the suggestion above and see if they fail again.
> > Actually, thinking about this a bit more - the failing test doesn't use an 
> > overlay at all, so changing to the above wouldn't help at all.
> The test was added in 8188484daa4195a2c8b5253765036fa2c6da7263 / 
> https://reviews.llvm.org/D112647, with a change to do:
> ```
> +auto PrevWorkingDir = SM.getFileManager().getFileSystemOpts().WorkingDir;
> +if (BuildDir)
> +  SM.getFileManager().getFileSystemOpts().WorkingDir = 
> std::move(*BuildDir);
>  if (auto Entry = SM.getFileManager().getFile(R.getFilePath())) {
>if (SourceTU) {
>  auto &Replaces = DiagReplacements[*Entry];
> @@ -170,18 +175,19 @@ groupReplacements(const TUReplacements &TUs, const 
> TUDiagnostics &TUDs,
>errs() << "Described file '" << R.getFilePath()
>   << "' doesn't exist. Ignoring...\n";
>  }
> +SM.getFileManager().getFileSystemOpts().WorkingDir = PrevWorkingDir;
> ```
> 
> This looks incredibly suspicious. Changing the working directory mid-stream 
> on the FileManager isn't sound. Consider:
> ```
> /dir1/file1.h
> /dir2/file1.h
> /dir2/file2.h
> ```
> if you do:
> ```
> FM.WorkingDir = "/dir1";
> FM.getFile("file1.h"); // returns /dir1/file1.h
> FM.WorkingDir = "/dir2";
> FM.getFile("file1.h"); // returns /dir1/file1.h !!!
> FM.getFile("file2.h"); // returns /dir2/file2.h
> ```
> 
> That doesn't explain why it's not reproducing locally for you, though.
I ended up building in a docker container and tracked it down.

The underlying problem is two fold:
1. What you've said (shouldn't change WorkingDir)
2. clang-apply-replacements stores the FileEntry and uses that to read the file 
later. Because relative paths are absolute pathed inside `getStatValue`, the 
path in the FileEntryRef remains relative (without the redirection hack).

The reason I'm not seeing this test fail on MacOS is down to the iteration 
order of files. On MacOS we process `file2.yaml` and then `file1.yaml`. On 
others it looks like we're processing `file1.yaml` and then `file2.yaml`.

If `file1.yaml` is processed last, its last lookup is an absolute path and thus 
everything works fine. But if `file2.yaml` is processed last, its last lookup 
is relative and then the file can't be found.

This would normally be fine since the relative path would be resolved using the 
working directory, but because that's continually changed... 💥 

Another quirk here is that if `getBufferForFile(FileEntry)` was used and 
`getFile` originally passed `openFile=true` then this would work, because it 
would be cached. And wouldn't work if `openFile=false` was passed, which is the 
default case.

TLDR: I don't think we should modify the redirection-hack condition in this 
patch.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D122549/new/

https://reviews.llvm.org/D122549

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D122549: [VFS] RedirectingFileSystem only replace path if not already mapped

[Ben Barham via Phabricator via cfe-commits]

bnbarham updated this revision to Diff 418959.
bnbarham added a comment.

Keep using the redirection hack for the time being


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D122549/new/

https://reviews.llvm.org/D122549

Files:
  clang/lib/Basic/FileManager.cpp
  clang/test/VFS/external-names-multi-overlay.c
  llvm/include/llvm/Support/VirtualFileSystem.h
  llvm/lib/Support/VirtualFileSystem.cpp
  llvm/unittests/Support/VirtualFileSystemTest.cpp

Index: llvm/unittests/Support/VirtualFileSystemTest.cpp
===
--- llvm/unittests/Support/VirtualFileSystemTest.cpp
+++ llvm/unittests/Support/VirtualFileSystemTest.cpp
@@ -1442,12 +1442,12 @@
   ErrorOr S = O->status("//root/file1");
   ASSERT_FALSE(S.getError());
   EXPECT_EQ("//root/foo/bar/a", S->getName());
-  EXPECT_TRUE(S->IsVFSMapped);
+  EXPECT_TRUE(S->ExposesExternalVFSPath);
 
   ErrorOr SLower = O->status("//root/foo/bar/a");
   EXPECT_EQ("//root/foo/bar/a", SLower->getName());
   EXPECT_TRUE(S->equivalent(*SLower));
-  EXPECT_FALSE(SLower->IsVFSMapped);
+  EXPECT_FALSE(SLower->ExposesExternalVFSPath);
 
   // file after opening
   auto OpenedF = O->openFileForRead("//root/file1");
@@ -1455,7 +1455,7 @@
   auto OpenedS = (*OpenedF)->status();
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("//root/foo/bar/a", OpenedS->getName());
-  EXPECT_TRUE(OpenedS->IsVFSMapped);
+  EXPECT_TRUE(OpenedS->ExposesExternalVFSPath);
 
   // directory
   S = O->status("//root/");
@@ -1467,27 +1467,27 @@
   S = O->status("//root/mappeddir");
   ASSERT_FALSE(S.getError());
   EXPECT_TRUE(S->isDirectory());
-  EXPECT_TRUE(S->IsVFSMapped);
+  EXPECT_TRUE(S->ExposesExternalVFSPath);
   EXPECT_TRUE(S->equivalent(*O->status("//root/foo/bar")));
 
   SLower = O->status("//root/foo/bar");
   EXPECT_EQ("//root/foo/bar", SLower->getName());
   EXPECT_TRUE(S->equivalent(*SLower));
-  EXPECT_FALSE(SLower->IsVFSMapped);
+  EXPECT_FALSE(SLower->ExposesExternalVFSPath);
 
   // file in remapped directory
   S = O->status("//root/mappeddir/a");
   ASSERT_FALSE(S.getError());
-  ASSERT_FALSE(S->isDirectory());
-  ASSERT_TRUE(S->IsVFSMapped);
-  ASSERT_EQ("//root/foo/bar/a", S->getName());
+  EXPECT_FALSE(S->isDirectory());
+  EXPECT_TRUE(S->ExposesExternalVFSPath);
+  EXPECT_EQ("//root/foo/bar/a", S->getName());
 
   // file in remapped directory, with use-external-name=false
   S = O->status("//root/mappeddir2/a");
   ASSERT_FALSE(S.getError());
-  ASSERT_FALSE(S->isDirectory());
-  ASSERT_TRUE(S->IsVFSMapped);
-  ASSERT_EQ("//root/mappeddir2/a", S->getName());
+  EXPECT_FALSE(S->isDirectory());
+  EXPECT_FALSE(S->ExposesExternalVFSPath);
+  EXPECT_EQ("//root/mappeddir2/a", S->getName());
 
   // file contents in remapped directory
   OpenedF = O->openFileForRead("//root/mappeddir/a");
@@ -1495,7 +1495,7 @@
   OpenedS = (*OpenedF)->status();
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("//root/foo/bar/a", OpenedS->getName());
-  EXPECT_TRUE(OpenedS->IsVFSMapped);
+  EXPECT_TRUE(OpenedS->ExposesExternalVFSPath);
 
   // file contents in remapped directory, with use-external-name=false
   OpenedF = O->openFileForRead("//root/mappeddir2/a");
@@ -1503,7 +1503,7 @@
   OpenedS = (*OpenedF)->status();
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("//root/mappeddir2/a", OpenedS->getName());
-  EXPECT_TRUE(OpenedS->IsVFSMapped);
+  EXPECT_FALSE(OpenedS->ExposesExternalVFSPath);
 
   // broken mapping
   EXPECT_EQ(O->status("//root/file2").getError(),
@@ -1535,12 +1535,12 @@
   ErrorOr S = O->status("//mappedroot/a");
   ASSERT_FALSE(S.getError());
   EXPECT_EQ("//root/foo/bar/a", S->getName());
-  EXPECT_TRUE(S->IsVFSMapped);
+  EXPECT_TRUE(S->ExposesExternalVFSPath);
 
   ErrorOr SLower = O->status("//root/foo/bar/a");
   EXPECT_EQ("//root/foo/bar/a", SLower->getName());
   EXPECT_TRUE(S->equivalent(*SLower));
-  EXPECT_FALSE(SLower->IsVFSMapped);
+  EXPECT_FALSE(SLower->ExposesExternalVFSPath);
 
   // file after opening
   auto OpenedF = O->openFileForRead("//mappedroot/a");
@@ -1548,7 +1548,7 @@
   auto OpenedS = (*OpenedF)->status();
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("//root/foo/bar/a", OpenedS->getName());
-  EXPECT_TRUE(OpenedS->IsVFSMapped);
+  EXPECT_TRUE(OpenedS->ExposesExternalVFSPath);
 
   EXPECT_EQ(0, NumDiagnostics);
 }
@@ -1696,12 +1696,12 @@
   auto OpenedS = (*OpenedF)->status();
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("a", OpenedS->getName());
-  EXPECT_FALSE(OpenedS->IsVFSMapped);
+  EXPECT_FALSE(OpenedS->ExposesExternalVFSPath);
 
   auto DirectS = RemappedFS->status("a");
   ASSERT_FALSE(DirectS.getError());
   EXPECT_EQ("a", DirectS->getName());
-  EXPECT_FALSE(DirectS->IsVFSMapped);
+  EXPECT_FALSE(DirectS->ExposesExternalVFSPath);
 
   EXPECT_EQ(0, NumDiagnostics);
 }
@@ -1736,12 +1736,12 @@
   auto OpenedS = (*OpenedF)->status();
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("realname", OpenedS->getName());
-  EXPECT_TRUE(Ope

[PATCH] D122549: [VFS] RedirectingFileSystem only replace path if not already mapped

[Ben Barham via Phabricator via cfe-commits]

This revision was landed with ongoing or failed builds.
This revision was automatically updated to reflect the committed changes.
Closed by commit rG3fda0edc51fd: [VFS] RedirectingFileSystem only replace path 
if not already mapped (authored by bnbarham).

Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D122549/new/

https://reviews.llvm.org/D122549

Files:
  clang/lib/Basic/FileManager.cpp
  clang/test/VFS/external-names-multi-overlay.c
  llvm/include/llvm/Support/VirtualFileSystem.h
  llvm/lib/Support/VirtualFileSystem.cpp
  llvm/unittests/Support/VirtualFileSystemTest.cpp

Index: llvm/unittests/Support/VirtualFileSystemTest.cpp
===
--- llvm/unittests/Support/VirtualFileSystemTest.cpp
+++ llvm/unittests/Support/VirtualFileSystemTest.cpp
@@ -1442,12 +1442,12 @@
   ErrorOr S = O->status("//root/file1");
   ASSERT_FALSE(S.getError());
   EXPECT_EQ("//root/foo/bar/a", S->getName());
-  EXPECT_TRUE(S->IsVFSMapped);
+  EXPECT_TRUE(S->ExposesExternalVFSPath);
 
   ErrorOr SLower = O->status("//root/foo/bar/a");
   EXPECT_EQ("//root/foo/bar/a", SLower->getName());
   EXPECT_TRUE(S->equivalent(*SLower));
-  EXPECT_FALSE(SLower->IsVFSMapped);
+  EXPECT_FALSE(SLower->ExposesExternalVFSPath);
 
   // file after opening
   auto OpenedF = O->openFileForRead("//root/file1");
@@ -1455,7 +1455,7 @@
   auto OpenedS = (*OpenedF)->status();
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("//root/foo/bar/a", OpenedS->getName());
-  EXPECT_TRUE(OpenedS->IsVFSMapped);
+  EXPECT_TRUE(OpenedS->ExposesExternalVFSPath);
 
   // directory
   S = O->status("//root/");
@@ -1467,27 +1467,27 @@
   S = O->status("//root/mappeddir");
   ASSERT_FALSE(S.getError());
   EXPECT_TRUE(S->isDirectory());
-  EXPECT_TRUE(S->IsVFSMapped);
+  EXPECT_TRUE(S->ExposesExternalVFSPath);
   EXPECT_TRUE(S->equivalent(*O->status("//root/foo/bar")));
 
   SLower = O->status("//root/foo/bar");
   EXPECT_EQ("//root/foo/bar", SLower->getName());
   EXPECT_TRUE(S->equivalent(*SLower));
-  EXPECT_FALSE(SLower->IsVFSMapped);
+  EXPECT_FALSE(SLower->ExposesExternalVFSPath);
 
   // file in remapped directory
   S = O->status("//root/mappeddir/a");
   ASSERT_FALSE(S.getError());
-  ASSERT_FALSE(S->isDirectory());
-  ASSERT_TRUE(S->IsVFSMapped);
-  ASSERT_EQ("//root/foo/bar/a", S->getName());
+  EXPECT_FALSE(S->isDirectory());
+  EXPECT_TRUE(S->ExposesExternalVFSPath);
+  EXPECT_EQ("//root/foo/bar/a", S->getName());
 
   // file in remapped directory, with use-external-name=false
   S = O->status("//root/mappeddir2/a");
   ASSERT_FALSE(S.getError());
-  ASSERT_FALSE(S->isDirectory());
-  ASSERT_TRUE(S->IsVFSMapped);
-  ASSERT_EQ("//root/mappeddir2/a", S->getName());
+  EXPECT_FALSE(S->isDirectory());
+  EXPECT_FALSE(S->ExposesExternalVFSPath);
+  EXPECT_EQ("//root/mappeddir2/a", S->getName());
 
   // file contents in remapped directory
   OpenedF = O->openFileForRead("//root/mappeddir/a");
@@ -1495,7 +1495,7 @@
   OpenedS = (*OpenedF)->status();
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("//root/foo/bar/a", OpenedS->getName());
-  EXPECT_TRUE(OpenedS->IsVFSMapped);
+  EXPECT_TRUE(OpenedS->ExposesExternalVFSPath);
 
   // file contents in remapped directory, with use-external-name=false
   OpenedF = O->openFileForRead("//root/mappeddir2/a");
@@ -1503,7 +1503,7 @@
   OpenedS = (*OpenedF)->status();
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("//root/mappeddir2/a", OpenedS->getName());
-  EXPECT_TRUE(OpenedS->IsVFSMapped);
+  EXPECT_FALSE(OpenedS->ExposesExternalVFSPath);
 
   // broken mapping
   EXPECT_EQ(O->status("//root/file2").getError(),
@@ -1535,12 +1535,12 @@
   ErrorOr S = O->status("//mappedroot/a");
   ASSERT_FALSE(S.getError());
   EXPECT_EQ("//root/foo/bar/a", S->getName());
-  EXPECT_TRUE(S->IsVFSMapped);
+  EXPECT_TRUE(S->ExposesExternalVFSPath);
 
   ErrorOr SLower = O->status("//root/foo/bar/a");
   EXPECT_EQ("//root/foo/bar/a", SLower->getName());
   EXPECT_TRUE(S->equivalent(*SLower));
-  EXPECT_FALSE(SLower->IsVFSMapped);
+  EXPECT_FALSE(SLower->ExposesExternalVFSPath);
 
   // file after opening
   auto OpenedF = O->openFileForRead("//mappedroot/a");
@@ -1548,7 +1548,7 @@
   auto OpenedS = (*OpenedF)->status();
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("//root/foo/bar/a", OpenedS->getName());
-  EXPECT_TRUE(OpenedS->IsVFSMapped);
+  EXPECT_TRUE(OpenedS->ExposesExternalVFSPath);
 
   EXPECT_EQ(0, NumDiagnostics);
 }
@@ -1696,12 +1696,12 @@
   auto OpenedS = (*OpenedF)->status();
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("a", OpenedS->getName());
-  EXPECT_FALSE(OpenedS->IsVFSMapped);
+  EXPECT_FALSE(OpenedS->ExposesExternalVFSPath);
 
   auto DirectS = RemappedFS->status("a");
   ASSERT_FALSE(DirectS.getError());
   EXPECT_EQ("a", DirectS->getName());
-  EXPECT_FALSE(DirectS->IsVFSMapped);
+  EXPECT_FALSE(DirectS->ExposesExternalVFSPath);
 
   EXPECT_EQ(0, NumDiagnostics);
 }
@@ -1736,12 +1736,12 @@
   auto Ope

[PATCH] D123103: Revert "[VFS] RedirectingFileSystem only replace path if not already mapped"

[Ben Barham via Phabricator via cfe-commits]

bnbarham created this revision.
bnbarham added a reviewer: dexonsmith.
Herald added a subscriber: hiraditya.
Herald added a project: All.
bnbarham requested review of this revision.
Herald added projects: clang, LLVM.
Herald added subscribers: llvm-commits, cfe-commits.

This reverts commit 3fda0edc51fd68192a30e302d45db081bb02d7f9 
.


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D123103

Files:
  clang/lib/Basic/FileManager.cpp
  clang/test/VFS/external-names-multi-overlay.c
  llvm/include/llvm/Support/VirtualFileSystem.h
  llvm/lib/Support/VirtualFileSystem.cpp
  llvm/unittests/Support/VirtualFileSystemTest.cpp

STAMPS
actor(@bnbarham) application(Differential) author(@bnbarham) herald(H224) 
herald(H225) herald(H423) herald(H576) herald(H615) herald(H688) herald(H864) 
monogram(D123103) object-type(DREV) phid(PHID-DREV-tt3oqfaxmuyojrbgy6z7) 
reviewer(@dexonsmith) revision-repository(rG) revision-status(needs-review) 
subscriber(@cfe-commits) subscriber(@hiraditya) subscriber(@llvm-commits) 
tag(#all) tag(#clang) tag(#llvm) via(conduit)

Index: llvm/unittests/Support/VirtualFileSystemTest.cpp
===
--- llvm/unittests/Support/VirtualFileSystemTest.cpp
+++ llvm/unittests/Support/VirtualFileSystemTest.cpp
@@ -1442,12 +1442,12 @@
   ErrorOr S = O->status("//root/file1");
   ASSERT_FALSE(S.getError());
   EXPECT_EQ("//root/foo/bar/a", S->getName());
-  EXPECT_TRUE(S->ExposesExternalVFSPath);
+  EXPECT_TRUE(S->IsVFSMapped);
 
   ErrorOr SLower = O->status("//root/foo/bar/a");
   EXPECT_EQ("//root/foo/bar/a", SLower->getName());
   EXPECT_TRUE(S->equivalent(*SLower));
-  EXPECT_FALSE(SLower->ExposesExternalVFSPath);
+  EXPECT_FALSE(SLower->IsVFSMapped);
 
   // file after opening
   auto OpenedF = O->openFileForRead("//root/file1");
@@ -1455,7 +1455,7 @@
   auto OpenedS = (*OpenedF)->status();
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("//root/foo/bar/a", OpenedS->getName());
-  EXPECT_TRUE(OpenedS->ExposesExternalVFSPath);
+  EXPECT_TRUE(OpenedS->IsVFSMapped);
 
   // directory
   S = O->status("//root/");
@@ -1467,27 +1467,27 @@
   S = O->status("//root/mappeddir");
   ASSERT_FALSE(S.getError());
   EXPECT_TRUE(S->isDirectory());
-  EXPECT_TRUE(S->ExposesExternalVFSPath);
+  EXPECT_TRUE(S->IsVFSMapped);
   EXPECT_TRUE(S->equivalent(*O->status("//root/foo/bar")));
 
   SLower = O->status("//root/foo/bar");
   EXPECT_EQ("//root/foo/bar", SLower->getName());
   EXPECT_TRUE(S->equivalent(*SLower));
-  EXPECT_FALSE(SLower->ExposesExternalVFSPath);
+  EXPECT_FALSE(SLower->IsVFSMapped);
 
   // file in remapped directory
   S = O->status("//root/mappeddir/a");
   ASSERT_FALSE(S.getError());
-  EXPECT_FALSE(S->isDirectory());
-  EXPECT_TRUE(S->ExposesExternalVFSPath);
-  EXPECT_EQ("//root/foo/bar/a", S->getName());
+  ASSERT_FALSE(S->isDirectory());
+  ASSERT_TRUE(S->IsVFSMapped);
+  ASSERT_EQ("//root/foo/bar/a", S->getName());
 
   // file in remapped directory, with use-external-name=false
   S = O->status("//root/mappeddir2/a");
   ASSERT_FALSE(S.getError());
-  EXPECT_FALSE(S->isDirectory());
-  EXPECT_FALSE(S->ExposesExternalVFSPath);
-  EXPECT_EQ("//root/mappeddir2/a", S->getName());
+  ASSERT_FALSE(S->isDirectory());
+  ASSERT_TRUE(S->IsVFSMapped);
+  ASSERT_EQ("//root/mappeddir2/a", S->getName());
 
   // file contents in remapped directory
   OpenedF = O->openFileForRead("//root/mappeddir/a");
@@ -1495,7 +1495,7 @@
   OpenedS = (*OpenedF)->status();
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("//root/foo/bar/a", OpenedS->getName());
-  EXPECT_TRUE(OpenedS->ExposesExternalVFSPath);
+  EXPECT_TRUE(OpenedS->IsVFSMapped);
 
   // file contents in remapped directory, with use-external-name=false
   OpenedF = O->openFileForRead("//root/mappeddir2/a");
@@ -1503,7 +1503,7 @@
   OpenedS = (*OpenedF)->status();
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("//root/mappeddir2/a", OpenedS->getName());
-  EXPECT_FALSE(OpenedS->ExposesExternalVFSPath);
+  EXPECT_TRUE(OpenedS->IsVFSMapped);
 
   // broken mapping
   EXPECT_EQ(O->status("//root/file2").getError(),
@@ -1535,12 +1535,12 @@
   ErrorOr S = O->status("//mappedroot/a");
   ASSERT_FALSE(S.getError());
   EXPECT_EQ("//root/foo/bar/a", S->getName());
-  EXPECT_TRUE(S->ExposesExternalVFSPath);
+  EXPECT_TRUE(S->IsVFSMapped);
 
   ErrorOr SLower = O->status("//root/foo/bar/a");
   EXPECT_EQ("//root/foo/bar/a", SLower->getName());
   EXPECT_TRUE(S->equivalent(*SLower));
-  EXPECT_FALSE(SLower->ExposesExternalVFSPath);
+  EXPECT_FALSE(SLower->IsVFSMapped);
 
   // file after opening
   auto OpenedF = O->openFileForRead("//mappedroot/a");
@@ -1548,7 +1548,7 @@
   auto OpenedS = (*OpenedF)->status();
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("//root/foo/bar/a", OpenedS->getName());
-  EXPECT_TRUE(OpenedS->ExposesExternalVFSPath);
+  EXPECT_TRUE(OpenedS->IsVFSMapped);
 
   EXPECT_EQ(0, NumDiagnostics)

[PATCH] D123104: [Modules] Use looked-up filename when looking for module maps

[Ben Barham via Phabricator via cfe-commits]

bnbarham created this revision.
bnbarham added a reviewer: dexonsmith.
Herald added a project: All.
bnbarham requested review of this revision.
Herald added a project: clang.
Herald added a subscriber: cfe-commits.

Prevent possible modulemap collisions by making sure to always use the
looked-up filename, regardless of any possible overlays.

Upstreamed from apple#llvm-project
72cf785051fb1b3ef22eee4dd33366e41a275981. As well as preventing the
collisions as above, it also highlighted an issue with the recent change
to narrow the FileManager hack of setting the `DirectoryEntry` in
`FileEntry` to the most recently looked up directory
(3fda0edc51fd68192a30e302d45db081bb02d7f9 
). 
Specifically, it would cause
the incorrect path to be used in `DoFrameworkLookup` in a crash
reproducer:

- Crash reproducers have `use-external-names` set to false
- `File->getFileEntry().getDir()->getName()` would then be the *cached* path, 
not the just looked up one
- `crash-vfs-umbrella-frameworks.m` fails with this change since the correct 
path is now looked up and causes B to be loaded twice


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D123104

Files:
  clang/include/clang/Lex/HeaderSearch.h
  clang/lib/Lex/HeaderSearch.cpp
  clang/test/Modules/Inputs/all-product-headers.yaml
  clang/test/Modules/modulemap-collision.m

STAMPS
actor(@bnbarham) application(Differential) author(@bnbarham) herald(H423) 
herald(H576) herald(H864) monogram(D123104) object-type(DREV) 
phid(PHID-DREV-h3pufxsxbgppzpajqza7) reviewer(@dexonsmith) 
revision-repository(rG) revision-status(needs-review) subscriber(@cfe-commits) 
tag(#all) tag(#clang) via(conduit)

Index: clang/test/Modules/modulemap-collision.m
===
--- /dev/null
+++ clang/test/Modules/modulemap-collision.m
@@ -0,0 +1,15 @@
+// REQUIRES: shell
+
+// RUN: rm -rf %t
+// RUN: mkdir -p %t/sources %t/build
+// RUN: echo "// A.h" > %t/sources/A.h
+// RUN: echo "framework module A {}" > %t/sources/module.modulemap
+// RUN: echo "framework module A.Private { umbrella header \"A.h\" }" > %t/sources/module.private.modulemap
+// RUN: cp %t/sources/module.modulemap %t/build/module.modulemap
+// RUN: cp %t/sources/module.private.modulemap %t/build/module.private.modulemap
+
+// RUN: sed -e "s:DUMMY_DIR:%t:g" %S/Inputs/all-product-headers.yaml > %t/build/all-product-headers.yaml
+// RUN: %clang_cc1 -fsyntax-only -ivfsoverlay %t/build/all-product-headers.yaml -F%t/build -fmodules -fimplicit-module-maps -fmodules-cache-path=tmp -x objective-c %s -verify
+
+// expected-no-diagnostics
+#import 
Index: clang/test/Modules/Inputs/all-product-headers.yaml
===
--- /dev/null
+++ clang/test/Modules/Inputs/all-product-headers.yaml
@@ -0,0 +1,33 @@
+{
+  'version': 0,
+  'case-sensitive': 'false',
+  'roots': [
+{
+  'type': 'directory',
+  'name': "DUMMY_DIR/build/A.framework/PrivateHeaders"
+  'contents': [
+{
+  'type': 'file',
+  'name': "A.h",
+  'external-contents': "DUMMY_DIR/sources/A.h"
+}
+  ]
+},
+{
+  'type': 'directory',
+  'name': "DUMMY_DIR/build/A.framework/Modules"
+  'contents': [
+{
+  'type': 'file',
+  'name': "module.modulemap",
+  'external-contents': "DUMMY_DIR/build/module.modulemap"
+},
+{
+  'type': 'file',
+  'name': "module.private.modulemap",
+  'external-contents': "DUMMY_DIR/build/module.private.modulemap"
+}
+  ]
+}
+  ]
+}
Index: clang/lib/Lex/HeaderSearch.cpp
===
--- clang/lib/Lex/HeaderSearch.cpp
+++ clang/lib/Lex/HeaderSearch.cpp
@@ -415,7 +415,7 @@
   // If there is a module that corresponds to this header, suggest it.
   if (!findUsableModuleForHeader(
   &File->getFileEntry(), Dir ? Dir : File->getFileEntry().getDir(),
-  RequestingModule, SuggestedModule, IsSystemHeaderDir))
+  RequestingModule, SuggestedModule, IsSystemHeaderDir, FileName))
 return None;
 
   return *File;
@@ -1563,10 +1563,18 @@
 
 bool HeaderSearch::findUsableModuleForHeader(
 const FileEntry *File, const DirectoryEntry *Root, Module *RequestingModule,
-ModuleMap::KnownHeader *SuggestedModule, bool IsSystemHeaderDir) {
+ModuleMap::KnownHeader *SuggestedModule, bool IsSystemHeaderDir,
+StringRef FileName) {
   if (File && needModuleLookup(RequestingModule, SuggestedModule)) {
 // If there is a module that corresponds to this header, suggest it.
-hasModuleMap(File->getName(), Root, IsSystemHeaderDir);
+
+// FIXME: File->getName() *should* be the same as FileName, but because
+// of the VFS and various hacks in FileManager, that's not necessarily the
+// case. We should update this to use FileEntryRef ins

[PATCH] D123103: Revert "[VFS] RedirectingFileSystem only replace path if not already mapped"

[Ben Barham via Phabricator via cfe-commits]

bnbarham added a comment.

This broke crash reproducers in very specific circumstances, see 
https://reviews.llvm.org/D123104. I'll re-commit with adding 
`ExposesExternalVFSPath` instead of replacing `IsVFSMapped`.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D123103/new/

https://reviews.llvm.org/D123103

STAMPS
actor(@bnbarham) application(Differential) author(@bnbarham) herald(H224) 
herald(H225) herald(H423) herald(H576) herald(H615) herald(H688) herald(H864) 
monogram(D123103) object-type(DREV) phid(PHID-DREV-tt3oqfaxmuyojrbgy6z7) 
reviewer(@dexonsmith) revision-repository(rG) revision-status(needs-review) 
subscriber(@cfe-commits) subscriber(@hiraditya) subscriber(@llvm-commits) 
tag(#all) tag(#clang) tag(#llvm) via(web)

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D123103: Revert "[VFS] RedirectingFileSystem only replace path if not already mapped"

[Ben Barham via Phabricator via cfe-commits]

This revision was landed with ongoing or failed builds.
This revision was automatically updated to reflect the committed changes.
Closed by commit rGf65b0b5dcfeb: Revert "[VFS] RedirectingFileSystem only 
replace path if not already mapped" (authored by bnbarham).

Changed prior to commit:
  https://reviews.llvm.org/D123103?vs=420354&id=420634#toc

Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D123103/new/

https://reviews.llvm.org/D123103

Files:
  clang/lib/Basic/FileManager.cpp
  clang/test/VFS/external-names-multi-overlay.c
  llvm/include/llvm/Support/VirtualFileSystem.h
  llvm/lib/Support/VirtualFileSystem.cpp
  llvm/unittests/Support/VirtualFileSystemTest.cpp

Index: llvm/unittests/Support/VirtualFileSystemTest.cpp
===
--- llvm/unittests/Support/VirtualFileSystemTest.cpp
+++ llvm/unittests/Support/VirtualFileSystemTest.cpp
@@ -1442,12 +1442,12 @@
   ErrorOr S = O->status("//root/file1");
   ASSERT_FALSE(S.getError());
   EXPECT_EQ("//root/foo/bar/a", S->getName());
-  EXPECT_TRUE(S->ExposesExternalVFSPath);
+  EXPECT_TRUE(S->IsVFSMapped);
 
   ErrorOr SLower = O->status("//root/foo/bar/a");
   EXPECT_EQ("//root/foo/bar/a", SLower->getName());
   EXPECT_TRUE(S->equivalent(*SLower));
-  EXPECT_FALSE(SLower->ExposesExternalVFSPath);
+  EXPECT_FALSE(SLower->IsVFSMapped);
 
   // file after opening
   auto OpenedF = O->openFileForRead("//root/file1");
@@ -1455,7 +1455,7 @@
   auto OpenedS = (*OpenedF)->status();
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("//root/foo/bar/a", OpenedS->getName());
-  EXPECT_TRUE(OpenedS->ExposesExternalVFSPath);
+  EXPECT_TRUE(OpenedS->IsVFSMapped);
 
   // directory
   S = O->status("//root/");
@@ -1467,27 +1467,27 @@
   S = O->status("//root/mappeddir");
   ASSERT_FALSE(S.getError());
   EXPECT_TRUE(S->isDirectory());
-  EXPECT_TRUE(S->ExposesExternalVFSPath);
+  EXPECT_TRUE(S->IsVFSMapped);
   EXPECT_TRUE(S->equivalent(*O->status("//root/foo/bar")));
 
   SLower = O->status("//root/foo/bar");
   EXPECT_EQ("//root/foo/bar", SLower->getName());
   EXPECT_TRUE(S->equivalent(*SLower));
-  EXPECT_FALSE(SLower->ExposesExternalVFSPath);
+  EXPECT_FALSE(SLower->IsVFSMapped);
 
   // file in remapped directory
   S = O->status("//root/mappeddir/a");
   ASSERT_FALSE(S.getError());
-  EXPECT_FALSE(S->isDirectory());
-  EXPECT_TRUE(S->ExposesExternalVFSPath);
-  EXPECT_EQ("//root/foo/bar/a", S->getName());
+  ASSERT_FALSE(S->isDirectory());
+  ASSERT_TRUE(S->IsVFSMapped);
+  ASSERT_EQ("//root/foo/bar/a", S->getName());
 
   // file in remapped directory, with use-external-name=false
   S = O->status("//root/mappeddir2/a");
   ASSERT_FALSE(S.getError());
-  EXPECT_FALSE(S->isDirectory());
-  EXPECT_FALSE(S->ExposesExternalVFSPath);
-  EXPECT_EQ("//root/mappeddir2/a", S->getName());
+  ASSERT_FALSE(S->isDirectory());
+  ASSERT_TRUE(S->IsVFSMapped);
+  ASSERT_EQ("//root/mappeddir2/a", S->getName());
 
   // file contents in remapped directory
   OpenedF = O->openFileForRead("//root/mappeddir/a");
@@ -1495,7 +1495,7 @@
   OpenedS = (*OpenedF)->status();
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("//root/foo/bar/a", OpenedS->getName());
-  EXPECT_TRUE(OpenedS->ExposesExternalVFSPath);
+  EXPECT_TRUE(OpenedS->IsVFSMapped);
 
   // file contents in remapped directory, with use-external-name=false
   OpenedF = O->openFileForRead("//root/mappeddir2/a");
@@ -1503,7 +1503,7 @@
   OpenedS = (*OpenedF)->status();
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("//root/mappeddir2/a", OpenedS->getName());
-  EXPECT_FALSE(OpenedS->ExposesExternalVFSPath);
+  EXPECT_TRUE(OpenedS->IsVFSMapped);
 
   // broken mapping
   EXPECT_EQ(O->status("//root/file2").getError(),
@@ -1535,12 +1535,12 @@
   ErrorOr S = O->status("//mappedroot/a");
   ASSERT_FALSE(S.getError());
   EXPECT_EQ("//root/foo/bar/a", S->getName());
-  EXPECT_TRUE(S->ExposesExternalVFSPath);
+  EXPECT_TRUE(S->IsVFSMapped);
 
   ErrorOr SLower = O->status("//root/foo/bar/a");
   EXPECT_EQ("//root/foo/bar/a", SLower->getName());
   EXPECT_TRUE(S->equivalent(*SLower));
-  EXPECT_FALSE(SLower->ExposesExternalVFSPath);
+  EXPECT_FALSE(SLower->IsVFSMapped);
 
   // file after opening
   auto OpenedF = O->openFileForRead("//mappedroot/a");
@@ -1548,7 +1548,7 @@
   auto OpenedS = (*OpenedF)->status();
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("//root/foo/bar/a", OpenedS->getName());
-  EXPECT_TRUE(OpenedS->ExposesExternalVFSPath);
+  EXPECT_TRUE(OpenedS->IsVFSMapped);
 
   EXPECT_EQ(0, NumDiagnostics);
 }
@@ -1696,12 +1696,12 @@
   auto OpenedS = (*OpenedF)->status();
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("a", OpenedS->getName());
-  EXPECT_FALSE(OpenedS->ExposesExternalVFSPath);
+  EXPECT_FALSE(OpenedS->IsVFSMapped);
 
   auto DirectS = RemappedFS->status("a");
   ASSERT_FALSE(DirectS.getError());
   EXPECT_EQ("a", DirectS->getName());
-  EXPECT_FALSE(DirectS->ExposesExternalVFSPath);
+  EXPECT_FALSE(Di

[PATCH] D123104: [Modules] Use looked-up filename when looking for module maps

[Ben Barham via Phabricator via cfe-commits]

bnbarham added inline comments.



Comment at: clang/include/clang/Lex/HeaderSearch.h:758
+ bool IsSystemHeaderDir,
+ StringRef FileName = "");
 

dexonsmith wrote:
> benlangmuir wrote:
> > This parameter could use a comment, even if it just points you to read the 
> > FIXME in the implementation.
> Good point; maybe it could have a better name/type, too, something like:
> ```
> lang=c++
> Optional OverrideModuleMapFilename = None
> ```
I believe this is actually the header name, but `Optional` + `OverrideFilename` 
seems worthwhile to me.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D123104/new/

https://reviews.llvm.org/D123104

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D123104: [Modules] Use looked-up filename when looking for module maps

[Ben Barham via Phabricator via cfe-commits]

bnbarham updated this revision to Diff 420678.
bnbarham added a comment.

Added a potential plan to remove the FileManager hacks.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D123104/new/

https://reviews.llvm.org/D123104

Files:
  clang/include/clang/Lex/HeaderSearch.h
  clang/lib/Basic/FileManager.cpp
  clang/lib/Lex/HeaderSearch.cpp
  clang/test/Modules/Inputs/all-product-headers.yaml
  clang/test/Modules/modulemap-collision.m

Index: clang/test/Modules/modulemap-collision.m
===
--- /dev/null
+++ clang/test/Modules/modulemap-collision.m
@@ -0,0 +1,15 @@
+// REQUIRES: shell
+
+// RUN: rm -rf %t
+// RUN: mkdir -p %t/sources %t/build
+// RUN: echo "// A.h" > %t/sources/A.h
+// RUN: echo "framework module A {}" > %t/sources/module.modulemap
+// RUN: echo "framework module A.Private { umbrella header \"A.h\" }" > %t/sources/module.private.modulemap
+// RUN: cp %t/sources/module.modulemap %t/build/module.modulemap
+// RUN: cp %t/sources/module.private.modulemap %t/build/module.private.modulemap
+
+// RUN: sed -e "s:DUMMY_DIR:%t:g" %S/Inputs/all-product-headers.yaml > %t/build/all-product-headers.yaml
+// RUN: %clang_cc1 -fsyntax-only -ivfsoverlay %t/build/all-product-headers.yaml -F%t/build -fmodules -fimplicit-module-maps -fmodules-cache-path=tmp -x objective-c %s -verify
+
+// expected-no-diagnostics
+#import 
Index: clang/test/Modules/Inputs/all-product-headers.yaml
===
--- /dev/null
+++ clang/test/Modules/Inputs/all-product-headers.yaml
@@ -0,0 +1,33 @@
+{
+  'version': 0,
+  'case-sensitive': 'false',
+  'roots': [
+{
+  'type': 'directory',
+  'name': "DUMMY_DIR/build/A.framework/PrivateHeaders"
+  'contents': [
+{
+  'type': 'file',
+  'name': "A.h",
+  'external-contents': "DUMMY_DIR/sources/A.h"
+}
+  ]
+},
+{
+  'type': 'directory',
+  'name': "DUMMY_DIR/build/A.framework/Modules"
+  'contents': [
+{
+  'type': 'file',
+  'name': "module.modulemap",
+  'external-contents': "DUMMY_DIR/build/module.modulemap"
+},
+{
+  'type': 'file',
+  'name': "module.private.modulemap",
+  'external-contents': "DUMMY_DIR/build/module.private.modulemap"
+}
+  ]
+}
+  ]
+}
Index: clang/lib/Lex/HeaderSearch.cpp
===
--- clang/lib/Lex/HeaderSearch.cpp
+++ clang/lib/Lex/HeaderSearch.cpp
@@ -415,7 +415,7 @@
   // If there is a module that corresponds to this header, suggest it.
   if (!findUsableModuleForHeader(
   &File->getFileEntry(), Dir ? Dir : File->getFileEntry().getDir(),
-  RequestingModule, SuggestedModule, IsSystemHeaderDir))
+  RequestingModule, SuggestedModule, IsSystemHeaderDir, FileName))
 return None;
 
   return *File;
@@ -1563,10 +1563,20 @@
 
 bool HeaderSearch::findUsableModuleForHeader(
 const FileEntry *File, const DirectoryEntry *Root, Module *RequestingModule,
-ModuleMap::KnownHeader *SuggestedModule, bool IsSystemHeaderDir) {
+ModuleMap::KnownHeader *SuggestedModule, bool IsSystemHeaderDir,
+Optional OverrideFilename) {
   if (File && needModuleLookup(RequestingModule, SuggestedModule)) {
 // If there is a module that corresponds to this header, suggest it.
-hasModuleMap(File->getName(), Root, IsSystemHeaderDir);
+//
+// FIXME: File->getName() *should* be the same as FileName, but because
+// of the VFS and various hacks in FileManager, that's not necessarily the
+// case. See the FIXME in `FileManager::getFileRef` for a plan to remove
+// this case. The eventual goal is to be able to use
+// `FileEntryRef::getName`, though it might take a few steps to get there.
+StringRef Filename = File->getName();
+if (OverrideFilename)
+  Filename = *OverrideFilename;
+hasModuleMap(Filename, Root, IsSystemHeaderDir);
 return suggestModule(*this, File, RequestingModule, SuggestedModule);
   }
   return true;
Index: clang/lib/Basic/FileManager.cpp
===
--- clang/lib/Basic/FileManager.cpp
+++ clang/lib/Basic/FileManager.cpp
@@ -287,11 +287,48 @@
 // name to users (in diagnostics) and to tools that don't have access to
 // the VFS (in debug info and dependency '.d' files).
 //
-// FIXME: This is pretty complicated. It's also inconsistent with how
-// "real" filesystems behave and confuses parts of clang expect to see the
-// name-as-accessed on the \a FileEntryRef. Maybe the returned \a
-// FileEntryRef::getName() could return the accessed name unmodified, but
-// make the external name available via a separate API.
+// FIXME: This is pretty complex and has some very complicated interactions
+// with the rest of clan

[PATCH] D130934: [clang] Update code that assumes FileEntry::getName is absolute NFC

[Ben Barham via Phabricator via cfe-commits]

bnbarham accepted this revision.
bnbarham added a comment.
This revision is now accepted and ready to land.

LGTM, thanks for doing this 🙇


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D130934/new/

https://reviews.llvm.org/D130934

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D130935: [clang] Only modify FileEntryRef names that are externally remapped

[Ben Barham via Phabricator via cfe-commits]

bnbarham accepted this revision.
bnbarham added inline comments.
This revision is now accepted and ready to land.



Comment at: clang/lib/Basic/FileManager.cpp:277-278
 
-  if (Status.getName() == Filename) {
-// The name matches. Set the FileEntry.
+  if (Status.getName() == Filename || !Status.ExposesExternalVFSPath) {
+// Use the requested name. Set the FileEntry.
 NamedFileEnt->second = FileEntryRef::MapValue(*UFE, DirInfo);

Is it possible for an `ExposesExternalVFSPath` to have the same filename as the 
one that was requested? Just in the case of mapping `A -> A` or something 
equally pointless? Could check for that in the VFS, but probably doesn't matter 
in the long run.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D130935/new/

https://reviews.llvm.org/D130935

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D131004: [clang] Add FileEntryRef::getNameAsRequested()

[Ben Barham via Phabricator via cfe-commits]

bnbarham accepted this revision.
bnbarham added a comment.
This revision is now accepted and ready to land.

Thanks 🙇




Comment at: clang/unittests/Basic/FileManagerTest.cpp:381
   EXPECT_TRUE(F1->isSameRef(*F1Again));
-  EXPECT_TRUE(F1->isSameRef(*F1Redirect));
+  EXPECT_FALSE(F1->isSameRef(*F1Redirect));
   EXPECT_FALSE(F1->isSameRef(*F1Also));

Bit weird that this test was checking for true 🤔. Seems like it isn't used 
outside of tests except for `isEqual` in `DenseMapInfo` though.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D131004/new/

https://reviews.llvm.org/D131004

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D131076: [clang][modules] Don't depend on sharing FileManager during module build

[Ben Barham via Phabricator via cfe-commits]

bnbarham accepted this revision.
bnbarham added inline comments.
This revision is now accepted and ready to land.



Comment at: clang/test/ClangScanDeps/modulemap-via-vfs.m:44
   "case-sensitive": "false",
+  'use-external-names': true,
   "roots": [

Nitpick: `'` vs `"` in the rest of the file, doesn't really matter though.



Comment at: clang/test/Modules/submodule-in-private-mmap-vfs.m:38
+@import Foo_Private;
+// expected-no-diagnostics

Nitpick: no new line


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D131076/new/

https://reviews.llvm.org/D131076

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D105328: [Frontend] Only compile modules if not already finalized

[Ben Barham via Phabricator via cfe-commits]

bnbarham updated this revision to Diff 358798.

Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D105328/new/

https://reviews.llvm.org/D105328

Files:
  clang/include/clang/Basic/DiagnosticCommonKinds.td
  clang/include/clang/Basic/DiagnosticSerializationKinds.td
  clang/include/clang/Serialization/ASTReader.h
  clang/lib/Frontend/CompilerInstance.cpp
  clang/lib/Serialization/ASTReader.cpp
  clang/unittests/Serialization/CMakeLists.txt
  clang/unittests/Serialization/ModuleCacheTest.cpp

Index: clang/unittests/Serialization/ModuleCacheTest.cpp
===
--- /dev/null
+++ clang/unittests/Serialization/ModuleCacheTest.cpp
@@ -0,0 +1,179 @@
+//===- unittests/Serialization/ModuleCacheTest.cpp - CI tests -===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Basic/FileManager.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Frontend/CompilerInvocation.h"
+#include "clang/Frontend/FrontendActions.h"
+#include "clang/Lex/HeaderSearch.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/raw_ostream.h"
+
+#include "gtest/gtest.h"
+
+using namespace llvm;
+using namespace clang;
+
+namespace {
+
+class ModuleCacheTest : public ::testing::Test {
+  void SetUp() override {
+ASSERT_FALSE(sys::fs::createUniqueDirectory("modulecache-test", TestDir));
+
+ModuleCachePath = SmallString<256>(TestDir);
+sys::path::append(ModuleCachePath, "mcp");
+ASSERT_FALSE(sys::fs::create_directories(ModuleCachePath));
+  }
+
+  void TearDown() override { sys::fs::remove_directories(TestDir); }
+
+public:
+  SmallString<256> TestDir;
+  SmallString<256> ModuleCachePath;
+
+  void addFile(StringRef Path, StringRef Contents) {
+ASSERT_FALSE(sys::path::is_absolute(Path));
+
+SmallString<256> AbsPath(TestDir);
+sys::path::append(AbsPath, Path);
+
+std::error_code EC;
+ASSERT_FALSE(
+sys::fs::create_directories(llvm::sys::path::parent_path(AbsPath)));
+llvm::raw_fd_ostream OS(AbsPath, EC);
+ASSERT_FALSE(EC);
+OS << Contents;
+  }
+
+  void addDuplicateFrameworks() {
+addFile("test.m", R"cpp(
+@import Top;
+)cpp");
+
+addFile("frameworks/Top.framework/Headers/top.h", R"cpp(
+@import M;
+)cpp");
+addFile("frameworks/Top.framework/Modules/module.modulemap", R"cpp(
+framework module Top [system] {
+  header "top.h"
+  export *
+}
+)cpp");
+
+addFile("frameworks/M.framework/Headers/m.h", R"cpp(
+void foo();
+)cpp");
+addFile("frameworks/M.framework/Modules/module.modulemap", R"cpp(
+framework module M [system] {
+  header "m.h"
+  export *
+}
+)cpp");
+
+addFile("frameworks2/M.framework/Headers/m.h", R"cpp(
+void foo();
+)cpp");
+addFile("frameworks2/M.framework/Modules/module.modulemap", R"cpp(
+framework module M [system] {
+  header "m.h"
+  export *
+}
+)cpp");
+  }
+};
+
+TEST_F(ModuleCacheTest, CachedModuleNewPath) {
+  addDuplicateFrameworks();
+
+  SmallString<256> MCPArg("-fmodules-cache-path=");
+  MCPArg.append(ModuleCachePath);
+  IntrusiveRefCntPtr Diags =
+  CompilerInstance::createDiagnostics(new DiagnosticOptions());
+
+  // First run should pass with no errors
+  const char *Args[] = {"clang","-fmodules",  "-Fframeworks",
+MCPArg.c_str(), "-working-directory", TestDir.c_str(),
+"test.m"};
+  std::shared_ptr Invocation =
+  createInvocationFromCommandLine(Args, Diags);
+  ASSERT_TRUE(Invocation);
+  CompilerInstance Instance;
+  Instance.setDiagnostics(Diags.get());
+  Instance.setInvocation(Invocation);
+  SyntaxOnlyAction Action;
+  ASSERT_TRUE(Instance.ExecuteAction(Action));
+  ASSERT_FALSE(Diags->hasErrorOccurred());
+
+  // Now add `frameworks2` to the search path. `Top.pcm` will have a reference
+  // to the `M` from `frameworks`, but a search will find the `M` from
+  // `frameworks2` - causing a mismatch and it to be considered out of date.
+  //
+  // Normally this would be fine - `M` and the modules it depends on would be
+  // rebuilt. However, since we have a shared module cache and thus an already
+  // finalized `Top`, recompiling `Top` will cause the existing module to be
+  // removed from the cache, causing possible crashed if it is ever used.
+  //
+  // Make sure that an error occurs instead.
+  const char *Args2[] = {"clang", "-fmodules","-Fframeworks2",
+ "-Fframeworks",  MCPArg.c_str(), "-working-directory",
+ 

[PATCH] D105328: [Frontend] Only compile modules if not already finalized

[Ben Barham via Phabricator via cfe-commits]

bnbarham marked 2 inline comments as done.
bnbarham added inline comments.



Comment at: clang/lib/Frontend/CompilerInstance.cpp:1063
+<< ModuleName;
+return ImportingInstance.getFrontendOpts().AllowPCMWithCompilerErrors;
+  }

vsapsai wrote:
> bnbarham wrote:
> > bnbarham wrote:
> > > vsapsai wrote:
> > > > Can we get in infinite loop with `AllowPCMWithCompilerErrors = true`? 
> > > > Specifically, I'm thinking about the scenario
> > > > 
> > > > 1. `compileModuleAndReadAST` obtains a file lock and calls 
> > > > `compileModule`
> > > > 2. `compileModule` calls `compileModuleImpl`
> > > > 3. Module is finalized but `AllowPCMWithCompilerErrors` is true, so 
> > > > `compileModuleImpl` returns true
> > > > 4. `compileModule` returns true
> > > > 5. `compileModuleAndReadAST` tries to read AST because compilation was 
> > > > successful
> > > > 6. AST is out of date, so `compileModuleAndReadAST` decides to try 
> > > > again, goto 1
> > > > 
> > > > Haven't tried to reproduce it locally but even if this scenario is 
> > > > impossible, a corresponding test case can be useful.
> > > Nice catch, that does seem likely - I'll see if I can add a test for this.
> > It doesn't end up causing an infinite recursion as `false` will end up 
> > being returned from `compileModuleAndReadAST`, but in that case there's no 
> > point returning `true` from `compileModuleImpl` in the first place so I've 
> > changed it anyway. Have also added that as a test case, just to make sure.
> Thanks for investigating it and adding a test case.
> 
> And appreciate that the error message mentions the module name. It is so hard 
> to work with errors like "cannot rebuild this module".
Note that this error is really just a fallback - it will be silenced by the 
previous fatal errors in ASTReader.



Comment at: clang/lib/Serialization/ASTReader.cpp:2859-2860
   << F.FileName << !F.ModuleName.empty() << F.ModuleName;
+if (recompileFinalized)
+  Diag(diag::note_module_file_conflict);
 

The failing test was because this note was being output when if `OutOfDate` 
wasn't in `Capabilities`. It should only get output when `OutOfDate` can be 
handled (ie. it'd be recompiled) *and* the module is already finalized.



Comment at: clang/lib/Serialization/ASTReader.cpp:5929
 
+bool ASTReader::diagnoseOutOfDate(StringRef ModuleFileName,
+  unsigned int ClientLoadCapabilities) {

vsapsai wrote:
> bnbarham wrote:
> > bnbarham wrote:
> > > vsapsai wrote:
> > > > Based on the rest of the code in clang, the expectation for 
> > > > `diagnose...` methods is to emit diagnostic in some cases. Personally, 
> > > > I'm often confused what true/false means for these methods, so I'm 
> > > > thinking about renaming the method to something like 
> > > > isRecoverableOutOfDateModule, canRecoverOutOfDateModule or some such. 
> > > > Feel free to pick a name you believe is appropriate, mine are just 
> > > > examples.
> > > Fair enough, `canRecoverOutOfDateModule` sounds reasonable to me. Or 
> > > maybe `canRecoverFromOutOfDate`?
> > I went with the latter.
> I think with the current naming we need to flip the conditions to opposite. 
> Currently,
> 
> ```lang=c++
>   return !(ClientLoadCapabilities & ARR_OutOfDate) ||
>  getModuleManager().getModuleCache().isPCMFinal(ModuleFileName);
> ```
> 
> corresponds to `cannotRecoverFromOutOfDate`. But to avoid double negations it 
> is better to have `canRecoverFromOutOfDate` (with `(ClientLoadCapabilities & 
> ARR_OutOfDate) && !isPCMFinal(ModuleFileName)`) and call it as 
> `!canRecoverFromOutOfDate` when necessary.
Argh, sorry - just did a find + replace without thinking. Have flipped that 
function and its calls.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D105328/new/

https://reviews.llvm.org/D105328

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D105328: [Frontend] Only compile modules if not already finalized

[Ben Barham via Phabricator via cfe-commits]

bnbarham updated this revision to Diff 358826.
bnbarham marked 2 inline comments as done.

Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D105328/new/

https://reviews.llvm.org/D105328

Files:
  clang/include/clang/Basic/DiagnosticCommonKinds.td
  clang/include/clang/Basic/DiagnosticSerializationKinds.td
  clang/include/clang/Serialization/ASTReader.h
  clang/lib/Frontend/CompilerInstance.cpp
  clang/lib/Serialization/ASTReader.cpp
  clang/unittests/Serialization/CMakeLists.txt
  clang/unittests/Serialization/ModuleCacheTest.cpp

Index: clang/unittests/Serialization/ModuleCacheTest.cpp
===
--- /dev/null
+++ clang/unittests/Serialization/ModuleCacheTest.cpp
@@ -0,0 +1,179 @@
+//===- unittests/Serialization/ModuleCacheTest.cpp - CI tests -===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Basic/FileManager.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Frontend/CompilerInvocation.h"
+#include "clang/Frontend/FrontendActions.h"
+#include "clang/Lex/HeaderSearch.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/raw_ostream.h"
+
+#include "gtest/gtest.h"
+
+using namespace llvm;
+using namespace clang;
+
+namespace {
+
+class ModuleCacheTest : public ::testing::Test {
+  void SetUp() override {
+ASSERT_FALSE(sys::fs::createUniqueDirectory("modulecache-test", TestDir));
+
+ModuleCachePath = SmallString<256>(TestDir);
+sys::path::append(ModuleCachePath, "mcp");
+ASSERT_FALSE(sys::fs::create_directories(ModuleCachePath));
+  }
+
+  void TearDown() override { sys::fs::remove_directories(TestDir); }
+
+public:
+  SmallString<256> TestDir;
+  SmallString<256> ModuleCachePath;
+
+  void addFile(StringRef Path, StringRef Contents) {
+ASSERT_FALSE(sys::path::is_absolute(Path));
+
+SmallString<256> AbsPath(TestDir);
+sys::path::append(AbsPath, Path);
+
+std::error_code EC;
+ASSERT_FALSE(
+sys::fs::create_directories(llvm::sys::path::parent_path(AbsPath)));
+llvm::raw_fd_ostream OS(AbsPath, EC);
+ASSERT_FALSE(EC);
+OS << Contents;
+  }
+
+  void addDuplicateFrameworks() {
+addFile("test.m", R"cpp(
+@import Top;
+)cpp");
+
+addFile("frameworks/Top.framework/Headers/top.h", R"cpp(
+@import M;
+)cpp");
+addFile("frameworks/Top.framework/Modules/module.modulemap", R"cpp(
+framework module Top [system] {
+  header "top.h"
+  export *
+}
+)cpp");
+
+addFile("frameworks/M.framework/Headers/m.h", R"cpp(
+void foo();
+)cpp");
+addFile("frameworks/M.framework/Modules/module.modulemap", R"cpp(
+framework module M [system] {
+  header "m.h"
+  export *
+}
+)cpp");
+
+addFile("frameworks2/M.framework/Headers/m.h", R"cpp(
+void foo();
+)cpp");
+addFile("frameworks2/M.framework/Modules/module.modulemap", R"cpp(
+framework module M [system] {
+  header "m.h"
+  export *
+}
+)cpp");
+  }
+};
+
+TEST_F(ModuleCacheTest, CachedModuleNewPath) {
+  addDuplicateFrameworks();
+
+  SmallString<256> MCPArg("-fmodules-cache-path=");
+  MCPArg.append(ModuleCachePath);
+  IntrusiveRefCntPtr Diags =
+  CompilerInstance::createDiagnostics(new DiagnosticOptions());
+
+  // First run should pass with no errors
+  const char *Args[] = {"clang","-fmodules",  "-Fframeworks",
+MCPArg.c_str(), "-working-directory", TestDir.c_str(),
+"test.m"};
+  std::shared_ptr Invocation =
+  createInvocationFromCommandLine(Args, Diags);
+  ASSERT_TRUE(Invocation);
+  CompilerInstance Instance;
+  Instance.setDiagnostics(Diags.get());
+  Instance.setInvocation(Invocation);
+  SyntaxOnlyAction Action;
+  ASSERT_TRUE(Instance.ExecuteAction(Action));
+  ASSERT_FALSE(Diags->hasErrorOccurred());
+
+  // Now add `frameworks2` to the search path. `Top.pcm` will have a reference
+  // to the `M` from `frameworks`, but a search will find the `M` from
+  // `frameworks2` - causing a mismatch and it to be considered out of date.
+  //
+  // Normally this would be fine - `M` and the modules it depends on would be
+  // rebuilt. However, since we have a shared module cache and thus an already
+  // finalized `Top`, recompiling `Top` will cause the existing module to be
+  // removed from the cache, causing possible crashed if it is ever used.
+  //
+  // Make sure that an error occurs instead.
+  const char *Args2[] = {"clang", "-fmodules","-Fframeworks2",
+ "-Fframeworks",  MCPArg.c_str(), "-wor

[PATCH] D105328: [Frontend] Only compile modules if not already finalized

[Ben Barham via Phabricator via cfe-commits]

bnbarham marked an inline comment as done.
bnbarham added inline comments.



Comment at: clang/lib/Serialization/ASTReader.cpp:2854
+bool recompileFinalized =
+Result == OutOfDate && Capabilities & ARR_OutOfDate &&
+getModuleManager().getModuleCache().isPCMFinal(F.FileName);

vsapsai wrote:
> I don't remember recommended LLVM style or if clang-tidy would complain about 
> it but `... & ... && ...` can be unclear regarding the priority of 
> operations. Personally, I would do `(... & ...) && ...` but I don't know what 
> is the rule, so not insisting.
I don't feel particularly strongly about it either way, so have just updated to 
add the parentheses :).

Also renamed `recompileFinalized` to `recompilingFinalized` as I thought that 
made more sense after coming back to it.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D105328/new/

https://reviews.llvm.org/D105328

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D105328: [Frontend] Only compile modules if not already finalized

[Ben Barham via Phabricator via cfe-commits]

bnbarham marked an inline comment as done.
bnbarham added a comment.

I don't have commit access. @vsapsai or @akyrtzi would you mind committing 
this? The test failures seem unrelated to me.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D105328/new/

https://reviews.llvm.org/D105328

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D95159: [ASTReader] Allow controlling separately whether validation should be disabled for a PCH vs a module file

[Ben Barham via Phabricator via cfe-commits]

bnbarham added a comment.

In D95159#2897941 , @mibintc wrote:

> Can you suggest how to find the problem?  Thanks a lot.

@mibintc the test is reusing the module cache directory and that's causing some 
issues between runs, sorry about that. You could add a `rm -rf %t` or `rm -rf 
%t/mcp` to this test to fix that. Looks like the same issue was run into in 
https://reviews.llvm.org/D96816#2572589. I'd prefer 
`clang/test/Index/preamble-reparse-changed-module.m` add the `rm` and another 
test added for the `LLVM_APPEND_VC_REV=NO` if we want that.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D95159/new/

https://reviews.llvm.org/D95159

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D95159: [ASTReader] Allow controlling separately whether validation should be disabled for a PCH vs a module file

[Ben Barham via Phabricator via cfe-commits]

bnbarham added a comment.

Yep, that would be perfect :). Thanks @mibintc


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D95159/new/

https://reviews.llvm.org/D95159

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D121426: [VFS] Update uses of getVFSFromYAML to use the new getVFSFromYAMLs API

[Ben Barham via Phabricator via cfe-commits]

bnbarham created this revision.
bnbarham added reviewers: keith, dexonsmith, JDevlieghere, vsapsai.
Herald added a subscriber: carlosgalvezp.
Herald added a project: All.
bnbarham requested review of this revision.
Herald added projects: clang, LLDB, LLVM, clang-tools-extra.
Herald added subscribers: cfe-commits, llvm-commits, lldb-commits.

Includes two test fixes (since chained mappings are no longer allowed)
and adds a new test for multiple overlays.

Uses other than `CompilerInvocation.cpp` are simple 1:1 mappings, but
without the need to read into a buffer first.

Depends on D121421  and D121423 
 and D121424 
 and D121425 



Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D121426

Files:
  clang-tools-extra/clang-tidy/tool/ClangTidyMain.cpp
  clang/lib/Frontend/CompilerInvocation.cpp
  clang/test/VFS/Inputs/vfsroot.yaml
  clang/test/VFS/directory.c
  clang/test/VFS/multiple-overlays.c
  clang/test/VFS/vfsroot-with-overlay.c
  lldb/source/Commands/CommandObjectReproducer.cpp
  llvm/include/llvm/Support/Error.h
  llvm/tools/dsymutil/Reproducer.cpp
  llvm/tools/dsymutil/Reproducer.h

Index: llvm/tools/dsymutil/Reproducer.h
===
--- llvm/tools/dsymutil/Reproducer.h
+++ llvm/tools/dsymutil/Reproducer.h
@@ -59,8 +59,7 @@
 };
 
 /// Reproducer instance used to use an existing reproducer. The VFS returned by
-/// this instance is a RedirectingFileSystem that remaps paths to their
-/// counterpart in the reproducer.
+/// remaps paths to their counterpart in the reproducer.
 class ReproducerUse : public Reproducer {
 public:
   ReproducerUse(StringRef Root, std::error_code &EC);
Index: llvm/tools/dsymutil/Reproducer.cpp
===
--- llvm/tools/dsymutil/Reproducer.cpp
+++ llvm/tools/dsymutil/Reproducer.cpp
@@ -48,15 +48,15 @@
 ReproducerUse::ReproducerUse(StringRef Root, std::error_code &EC) {
   SmallString<128> Mapping(Root);
   sys::path::append(Mapping, "mapping.yaml");
-  ErrorOr> Buffer =
-  vfs::getRealFileSystem()->getBufferForFile(Mapping.str());
 
-  if (!Buffer) {
-EC = Buffer.getError();
+  auto OverlayFS = llvm::vfs::getVFSFromYAMLs(Mapping.str());
+  if (auto Err = OverlayFS.takeError()) {
+llvm::handleAllErrors(std::move(Err), [&](const llvm::ErrorInfoBase &E) {
+  EC = E.convertToErrorCode();
+});
 return;
   }
-
-  VFS = llvm::vfs::getVFSFromYAML(std::move(Buffer.get()), nullptr, Mapping);
+  VFS = std::move(*OverlayFS);
 }
 
 llvm::Expected>
Index: llvm/include/llvm/Support/Error.h
===
--- llvm/include/llvm/Support/Error.h
+++ llvm/include/llvm/Support/Error.h
@@ -1281,7 +1281,7 @@
 return OS.str();
   }
 
-  StringRef getFileName() { return FileName; }
+  StringRef getFileName() const { return FileName; }
 
   Error takeError() { return Error(std::move(Err)); }
 
Index: lldb/source/Commands/CommandObjectReproducer.cpp
===
--- lldb/source/Commands/CommandObjectReproducer.cpp
+++ lldb/source/Commands/CommandObjectReproducer.cpp
@@ -409,23 +409,19 @@
 switch (m_options.provider) {
 case eReproducerProviderFiles: {
   FileSpec vfs_mapping = loader->GetFile();
+  std::string overlay_path = vfs_mapping.GetPath();
 
-  // Read the VFS mapping.
-  ErrorOr> buffer =
-  vfs::getRealFileSystem()->getBufferForFile(vfs_mapping.GetPath());
-  if (!buffer) {
-SetError(result, errorCodeToError(buffer.getError()));
+  Expected> vfs =
+  vfs::getVFSFromYAMLs(StringRef(overlay_path));
+  if (auto err = vfs.takeError()) {
+SetError(result, std::move(err));
 return false;
   }
 
-  // Initialize a VFS from the given mapping.
-  IntrusiveRefCntPtr vfs = vfs::getVFSFromYAML(
-  std::move(buffer.get()), nullptr, vfs_mapping.GetPath());
-
   // Dump the VFS to a buffer.
   std::string str;
   raw_string_ostream os(str);
-  static_cast(*vfs).dump(os);
+  (*vfs)->dump(os);
   os.flush();
 
   // Return the string.
Index: clang/test/VFS/vfsroot-with-overlay.c
===
--- clang/test/VFS/vfsroot-with-overlay.c
+++ clang/test/VFS/vfsroot-with-overlay.c
@@ -1,7 +1,7 @@
 // RUN: rm -rf %t
 // RUN: mkdir -p %t
 // RUN: sed -e "s@TEST_DIR@%{/S:regex_replacement}@g" -e "s@OUT_DIR@%{/t:regex_replacement}@g" %S/Inputs/vfsroot.yaml > %t.yaml
-// RUN: sed -e "s@INPUT_DIR@/indirect-vfs-root-files@g" -e "s@OUT_DIR@/overlay-dir@g" %S/Inputs/vfsoverlay.yaml > %t/vfsoverlay.yaml
+// RUN: sed -e "s@INPUT_DIR@%{/S:regex_replacement}/Inputs@g" -e "s@OUT_DIR@/overlay-dir@g" %S/Inputs/vfsoverlay.yaml > %t/vfsoverlay.yaml
 // RUN: %clang_c

[PATCH] D121426: [VFS] Update uses of getVFSFromYAML to use the new getVFSFromYAMLs API

[Ben Barham via Phabricator via cfe-commits]

bnbarham added inline comments.



Comment at: llvm/include/llvm/Support/Error.h:1284
 
-  StringRef getFileName() { return FileName; }
+  StringRef getFileName() const { return FileName; }
 

Should this be in a change all by itself?


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D121426/new/

https://reviews.llvm.org/D121426

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D117730: [DNM][VFS] Do not overwrite the path when nesting RedirectingFileSystems

[Ben Barham via Phabricator via cfe-commits]

bnbarham abandoned this revision.
bnbarham added a comment.
Herald added a project: All.

Rather than trying to fix nested RedirectingFileSystems, I've instead put up a 
bunch of patches to simplify RedirectingFileSystem. This does mean we can't 
"chain" remappings any more, but that seems like a good change to me. See:
https://reviews.llvm.org/D121421
https://reviews.llvm.org/D121423
https://reviews.llvm.org/D121424
https://reviews.llvm.org/D121425
https://reviews.llvm.org/D121426


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D117730/new/

https://reviews.llvm.org/D117730

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D121426: [VFS] Update uses of getVFSFromYAML to use the new getVFSFromYAMLs API

[Ben Barham via Phabricator via cfe-commits]

bnbarham updated this revision to Diff 414718.
bnbarham added a comment.

Handle empty overlay file in clang tidy


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D121426/new/

https://reviews.llvm.org/D121426

Files:
  clang-tools-extra/clang-tidy/tool/ClangTidyMain.cpp
  clang/lib/Frontend/CompilerInvocation.cpp
  clang/test/VFS/Inputs/vfsroot.yaml
  clang/test/VFS/directory.c
  clang/test/VFS/multiple-overlays.c
  clang/test/VFS/vfsroot-with-overlay.c
  lldb/source/Commands/CommandObjectReproducer.cpp
  llvm/include/llvm/Support/Error.h
  llvm/tools/dsymutil/Reproducer.cpp
  llvm/tools/dsymutil/Reproducer.h

Index: llvm/tools/dsymutil/Reproducer.h
===
--- llvm/tools/dsymutil/Reproducer.h
+++ llvm/tools/dsymutil/Reproducer.h
@@ -59,8 +59,7 @@
 };
 
 /// Reproducer instance used to use an existing reproducer. The VFS returned by
-/// this instance is a RedirectingFileSystem that remaps paths to their
-/// counterpart in the reproducer.
+/// this instance remaps paths to their counterpart in the reproducer.
 class ReproducerUse : public Reproducer {
 public:
   ReproducerUse(StringRef Root, std::error_code &EC);
Index: llvm/tools/dsymutil/Reproducer.cpp
===
--- llvm/tools/dsymutil/Reproducer.cpp
+++ llvm/tools/dsymutil/Reproducer.cpp
@@ -48,15 +48,15 @@
 ReproducerUse::ReproducerUse(StringRef Root, std::error_code &EC) {
   SmallString<128> Mapping(Root);
   sys::path::append(Mapping, "mapping.yaml");
-  ErrorOr> Buffer =
-  vfs::getRealFileSystem()->getBufferForFile(Mapping.str());
 
-  if (!Buffer) {
-EC = Buffer.getError();
+  auto OverlayFS = llvm::vfs::getVFSFromYAMLs(Mapping.str());
+  if (auto Err = OverlayFS.takeError()) {
+llvm::handleAllErrors(std::move(Err), [&](const llvm::ErrorInfoBase &E) {
+  EC = E.convertToErrorCode();
+});
 return;
   }
-
-  VFS = llvm::vfs::getVFSFromYAML(std::move(Buffer.get()), nullptr, Mapping);
+  VFS = std::move(*OverlayFS);
 }
 
 llvm::Expected>
Index: llvm/include/llvm/Support/Error.h
===
--- llvm/include/llvm/Support/Error.h
+++ llvm/include/llvm/Support/Error.h
@@ -1281,7 +1281,7 @@
 return OS.str();
   }
 
-  StringRef getFileName() { return FileName; }
+  StringRef getFileName() const { return FileName; }
 
   Error takeError() { return Error(std::move(Err)); }
 
Index: lldb/source/Commands/CommandObjectReproducer.cpp
===
--- lldb/source/Commands/CommandObjectReproducer.cpp
+++ lldb/source/Commands/CommandObjectReproducer.cpp
@@ -409,23 +409,19 @@
 switch (m_options.provider) {
 case eReproducerProviderFiles: {
   FileSpec vfs_mapping = loader->GetFile();
+  std::string overlay_path = vfs_mapping.GetPath();
 
-  // Read the VFS mapping.
-  ErrorOr> buffer =
-  vfs::getRealFileSystem()->getBufferForFile(vfs_mapping.GetPath());
-  if (!buffer) {
-SetError(result, errorCodeToError(buffer.getError()));
+  Expected> vfs =
+  vfs::getVFSFromYAMLs(StringRef(overlay_path));
+  if (auto err = vfs.takeError()) {
+SetError(result, std::move(err));
 return false;
   }
 
-  // Initialize a VFS from the given mapping.
-  IntrusiveRefCntPtr vfs = vfs::getVFSFromYAML(
-  std::move(buffer.get()), nullptr, vfs_mapping.GetPath());
-
   // Dump the VFS to a buffer.
   std::string str;
   raw_string_ostream os(str);
-  static_cast(*vfs).dump(os);
+  (*vfs)->dump(os);
   os.flush();
 
   // Return the string.
Index: clang/test/VFS/vfsroot-with-overlay.c
===
--- clang/test/VFS/vfsroot-with-overlay.c
+++ clang/test/VFS/vfsroot-with-overlay.c
@@ -1,7 +1,7 @@
 // RUN: rm -rf %t
 // RUN: mkdir -p %t
 // RUN: sed -e "s@TEST_DIR@%{/S:regex_replacement}@g" -e "s@OUT_DIR@%{/t:regex_replacement}@g" %S/Inputs/vfsroot.yaml > %t.yaml
-// RUN: sed -e "s@INPUT_DIR@/indirect-vfs-root-files@g" -e "s@OUT_DIR@/overlay-dir@g" %S/Inputs/vfsoverlay.yaml > %t/vfsoverlay.yaml
+// RUN: sed -e "s@INPUT_DIR@%{/S:regex_replacement}/Inputs@g" -e "s@OUT_DIR@/overlay-dir@g" %S/Inputs/vfsoverlay.yaml > %t/vfsoverlay.yaml
 // RUN: %clang_cc1 -Werror -ivfsoverlay %t.yaml -ivfsoverlay /direct-vfs-root-files/vfsoverlay.yaml -I /overlay-dir -fsyntax-only /tests/vfsroot-with-overlay.c
 
 #include "not_real.h"
Index: clang/test/VFS/multiple-overlays.c
===
--- /dev/null
+++ clang/test/VFS/multiple-overlays.c
@@ -0,0 +1,39 @@
+// RUN: rm -rf %t
+// RUN: split-file %s %t
+// RUN: sed -e "s@EXTERNAL_DIR@%{/t:regex_replacement}/B@g" -e "s@NAME_DIR@%{/t:regex_replacement}/A@g" %t/vfs/base.yaml > %t/vfs/a-b.yaml
+// RUN: sed -e "s@EXTERNAL_DIR@%{/t:regex_re

[PATCH] D121426: [VFS] Update uses of getVFSFromYAML to use the new getVFSFromYAMLs API

[Ben Barham via Phabricator via cfe-commits]

bnbarham updated this revision to Diff 414751.
bnbarham edited the summary of this revision.
bnbarham added a comment.

Update to single review dependency


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D121426/new/

https://reviews.llvm.org/D121426

Files:
  clang-tools-extra/clang-tidy/tool/ClangTidyMain.cpp
  clang/lib/Frontend/CompilerInvocation.cpp
  clang/test/VFS/Inputs/vfsroot.yaml
  clang/test/VFS/directory.c
  clang/test/VFS/multiple-overlays.c
  clang/test/VFS/vfsroot-with-overlay.c
  lldb/source/Commands/CommandObjectReproducer.cpp
  llvm/include/llvm/Support/Error.h
  llvm/tools/dsymutil/Reproducer.cpp
  llvm/tools/dsymutil/Reproducer.h

Index: llvm/tools/dsymutil/Reproducer.h
===
--- llvm/tools/dsymutil/Reproducer.h
+++ llvm/tools/dsymutil/Reproducer.h
@@ -59,8 +59,7 @@
 };
 
 /// Reproducer instance used to use an existing reproducer. The VFS returned by
-/// this instance is a RedirectingFileSystem that remaps paths to their
-/// counterpart in the reproducer.
+/// this instance remaps paths to their counterpart in the reproducer.
 class ReproducerUse : public Reproducer {
 public:
   ReproducerUse(StringRef Root, std::error_code &EC);
Index: llvm/tools/dsymutil/Reproducer.cpp
===
--- llvm/tools/dsymutil/Reproducer.cpp
+++ llvm/tools/dsymutil/Reproducer.cpp
@@ -48,15 +48,15 @@
 ReproducerUse::ReproducerUse(StringRef Root, std::error_code &EC) {
   SmallString<128> Mapping(Root);
   sys::path::append(Mapping, "mapping.yaml");
-  ErrorOr> Buffer =
-  vfs::getRealFileSystem()->getBufferForFile(Mapping.str());
 
-  if (!Buffer) {
-EC = Buffer.getError();
+  auto OverlayFS = llvm::vfs::getVFSFromYAMLs(Mapping.str());
+  if (auto Err = OverlayFS.takeError()) {
+llvm::handleAllErrors(std::move(Err), [&](const llvm::ErrorInfoBase &E) {
+  EC = E.convertToErrorCode();
+});
 return;
   }
-
-  VFS = llvm::vfs::getVFSFromYAML(std::move(Buffer.get()), nullptr, Mapping);
+  VFS = std::move(*OverlayFS);
 }
 
 llvm::Expected>
Index: llvm/include/llvm/Support/Error.h
===
--- llvm/include/llvm/Support/Error.h
+++ llvm/include/llvm/Support/Error.h
@@ -1281,7 +1281,7 @@
 return OS.str();
   }
 
-  StringRef getFileName() { return FileName; }
+  StringRef getFileName() const { return FileName; }
 
   Error takeError() { return Error(std::move(Err)); }
 
Index: lldb/source/Commands/CommandObjectReproducer.cpp
===
--- lldb/source/Commands/CommandObjectReproducer.cpp
+++ lldb/source/Commands/CommandObjectReproducer.cpp
@@ -409,23 +409,19 @@
 switch (m_options.provider) {
 case eReproducerProviderFiles: {
   FileSpec vfs_mapping = loader->GetFile();
+  std::string overlay_path = vfs_mapping.GetPath();
 
-  // Read the VFS mapping.
-  ErrorOr> buffer =
-  vfs::getRealFileSystem()->getBufferForFile(vfs_mapping.GetPath());
-  if (!buffer) {
-SetError(result, errorCodeToError(buffer.getError()));
+  Expected> vfs =
+  vfs::getVFSFromYAMLs(StringRef(overlay_path));
+  if (auto err = vfs.takeError()) {
+SetError(result, std::move(err));
 return false;
   }
 
-  // Initialize a VFS from the given mapping.
-  IntrusiveRefCntPtr vfs = vfs::getVFSFromYAML(
-  std::move(buffer.get()), nullptr, vfs_mapping.GetPath());
-
   // Dump the VFS to a buffer.
   std::string str;
   raw_string_ostream os(str);
-  static_cast(*vfs).dump(os);
+  (*vfs)->dump(os);
   os.flush();
 
   // Return the string.
Index: clang/test/VFS/vfsroot-with-overlay.c
===
--- clang/test/VFS/vfsroot-with-overlay.c
+++ clang/test/VFS/vfsroot-with-overlay.c
@@ -1,7 +1,7 @@
 // RUN: rm -rf %t
 // RUN: mkdir -p %t
 // RUN: sed -e "s@TEST_DIR@%{/S:regex_replacement}@g" -e "s@OUT_DIR@%{/t:regex_replacement}@g" %S/Inputs/vfsroot.yaml > %t.yaml
-// RUN: sed -e "s@INPUT_DIR@/indirect-vfs-root-files@g" -e "s@OUT_DIR@/overlay-dir@g" %S/Inputs/vfsoverlay.yaml > %t/vfsoverlay.yaml
+// RUN: sed -e "s@INPUT_DIR@%{/S:regex_replacement}/Inputs@g" -e "s@OUT_DIR@/overlay-dir@g" %S/Inputs/vfsoverlay.yaml > %t/vfsoverlay.yaml
 // RUN: %clang_cc1 -Werror -ivfsoverlay %t.yaml -ivfsoverlay /direct-vfs-root-files/vfsoverlay.yaml -I /overlay-dir -fsyntax-only /tests/vfsroot-with-overlay.c
 
 #include "not_real.h"
Index: clang/test/VFS/multiple-overlays.c
===
--- /dev/null
+++ clang/test/VFS/multiple-overlays.c
@@ -0,0 +1,39 @@
+// RUN: rm -rf %t
+// RUN: split-file %s %t
+// RUN: sed -e "s@EXTERNAL_DIR@%{/t:regex_replacement}/B@g" -e "s@NAME_DIR@%{/t:regex_replacement}/A@g" %t/vfs/base.yaml > %t/vfs/a-b.yaml
+// 

[PATCH] D121426: [VFS] Update uses of getVFSFromYAML to use the new getVFSFromYAMLs API

[Ben Barham via Phabricator via cfe-commits]

bnbarham added a comment.

In D121426#3376442 , @dexonsmith 
wrote:

>> Includes two test fixes (since chained mappings are no longer allowed)
>> and adds a new test for multiple overlays.
>
> Are we sure no one needs to support chained mappings? Has there been a 
> ~~clang-dev~~ discourse discussion about it already? Just concerned that some 
> vendor might rely on being able to support this.

I'm not *positive*, no, but I would be fairly surprised. You could just add the 
`A -> C` mapping if you really do want it. But I can start up that conversation 
if you think it needs having.

I actually didn't initially realise that there was a test for this case - 
`vfsroot-with-overlay.c` did test "indirection", but I completely missed it 
when I was looking through. I *thought* the only case was the one Nathan added 
in `directory.c` (and in that case what we really wanted was was what's now 
`fallback`).

@vsapsai do you know any clients of the chaining/nesting/indirection?


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D121426/new/

https://reviews.llvm.org/D121426

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D121426: [VFS] Update uses of getVFSFromYAML to use the new getVFSFromYAMLs API

[Ben Barham via Phabricator via cfe-commits]

bnbarham marked an inline comment as done.
bnbarham added inline comments.



Comment at: llvm/include/llvm/Support/Error.h:1284
 
-  StringRef getFileName() { return FileName; }
+  StringRef getFileName() const { return FileName; }
 

dexonsmith wrote:
> bnbarham wrote:
> > Should this be in a change all by itself?
> Yes, but I also know you already split this out so I guess you just need to 
> rebase :).
Heh, when you suggested I pull the rename out I basically thought "welll, that 
answer this question"

So yep, it's out and I'll rebase this one :). Thanks!


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D121426/new/

https://reviews.llvm.org/D121426

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D121425: [VFS] Add a new getVFSFromYAMLs API

[Ben Barham via Phabricator via cfe-commits]

bnbarham updated this revision to Diff 415615.
bnbarham edited the summary of this revision.
bnbarham added a comment.
Herald added subscribers: cfe-commits, lldb-commits, carlosgalvezp.
Herald added projects: clang, LLDB, clang-tools-extra.

Re-order to be before D121424  and merge with 
D121426 


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D121425/new/

https://reviews.llvm.org/D121425

Files:
  clang-tools-extra/clang-tidy/tool/ClangTidyMain.cpp
  clang/lib/Frontend/CompilerInvocation.cpp
  clang/test/VFS/Inputs/vfsroot.yaml
  clang/test/VFS/directory.c
  clang/test/VFS/multiple-overlays.c
  clang/test/VFS/vfsroot-with-overlay.c
  lldb/source/Commands/CommandObjectReproducer.cpp
  llvm/include/llvm/Support/VirtualFileSystem.h
  llvm/lib/Support/VirtualFileSystem.cpp
  llvm/tools/dsymutil/Reproducer.cpp
  llvm/tools/dsymutil/Reproducer.h
  llvm/unittests/Support/VirtualFileSystemTest.cpp

Index: llvm/unittests/Support/VirtualFileSystemTest.cpp
===
--- llvm/unittests/Support/VirtualFileSystemTest.cpp
+++ llvm/unittests/Support/VirtualFileSystemTest.cpp
@@ -14,6 +14,7 @@
 #include "llvm/Support/MemoryBuffer.h"
 #include "llvm/Support/Path.h"
 #include "llvm/Support/SourceMgr.h"
+#include "llvm/Testing/Support/Error.h"
 #include "llvm/Testing/Support/SupportHelpers.h"
 #include "gmock/gmock.h"
 #include "gtest/gtest.h"
@@ -2571,7 +2572,7 @@
 }
 
 TEST_F(VFSFromYAMLTest, WorkingDirectory) {
-  IntrusiveRefCntPtr Lower(new DummyFileSystem());
+  IntrusiveRefCntPtr Lower(new ErrorDummyFileSystem());
   Lower->addDirectory("//root/");
   Lower->addDirectory("//root/foo");
   Lower->addRegularFile("//root/foo/a");
@@ -2593,6 +2594,7 @@
   "}",
   Lower);
   ASSERT_NE(FS.get(), nullptr);
+
   std::error_code EC = FS->setCurrentWorkingDirectory("//root/bar");
   ASSERT_FALSE(EC);
 
@@ -2621,6 +2623,14 @@
   ASSERT_TRUE(WorkingDir);
   EXPECT_EQ(*WorkingDir, "//root/");
 
+  Status = FS->status("bar/a");
+  ASSERT_FALSE(Status.getError());
+  EXPECT_TRUE(Status->exists());
+
+  Status = FS->status("foo/a");
+  ASSERT_FALSE(Status.getError());
+  EXPECT_TRUE(Status->exists());
+
   EC = FS->setCurrentWorkingDirectory("bar");
   ASSERT_FALSE(EC);
   WorkingDir = FS->getCurrentWorkingDirectory();
@@ -2710,43 +2720,6 @@
   EXPECT_TRUE(Status->exists());
 }
 
-TEST_F(VFSFromYAMLTest, WorkingDirectoryFallthroughInvalid) {
-  IntrusiveRefCntPtr Lower(new ErrorDummyFileSystem());
-  Lower->addDirectory("//root/");
-  Lower->addDirectory("//root/foo");
-  Lower->addRegularFile("//root/foo/a");
-  Lower->addRegularFile("//root/foo/b");
-  Lower->addRegularFile("//root/c");
-  IntrusiveRefCntPtr FS = getFromYAMLString(
-  "{ 'use-external-names': false,\n"
-  "  'roots': [\n"
-  "{\n"
-  "  'type': 'directory',\n"
-  "  'name': '//root/bar',\n"
-  "  'contents': [ {\n"
-  "  'type': 'file',\n"
-  "  'name': 'a',\n"
-  "  'external-contents': '//root/foo/a'\n"
-  "}\n"
-  "  ]\n"
-  "}\n"
-  "]\n"
-  "}",
-  Lower);
-  ASSERT_NE(FS.get(), nullptr);
-  std::error_code EC = FS->setCurrentWorkingDirectory("//root/");
-  ASSERT_FALSE(EC);
-  ASSERT_NE(FS.get(), nullptr);
-
-  llvm::ErrorOr Status = FS->status("bar/a");
-  ASSERT_FALSE(Status.getError());
-  EXPECT_TRUE(Status->exists());
-
-  Status = FS->status("foo/a");
-  ASSERT_FALSE(Status.getError());
-  EXPECT_TRUE(Status->exists());
-}
-
 TEST_F(VFSFromYAMLTest, VirtualWorkingDirectory) {
   IntrusiveRefCntPtr Lower(new ErrorDummyFileSystem());
   Lower->addDirectory("//root/");
@@ -3207,3 +3180,168 @@
 "  DummyFileSystem (RecursiveContents)\n",
 Output);
 }
+
+static std::unique_ptr
+getVFSOrNull(ArrayRef YAMLOverlays,
+ IntrusiveRefCntPtr ExternalFS) {
+  SmallVector OverlayRefs;
+  for (const auto &Overlay : YAMLOverlays) {
+OverlayRefs.emplace_back(Overlay, "");
+  }
+
+  auto ExpectedFS = vfs::getVFSFromYAMLs(OverlayRefs, ExternalFS);
+  if (auto Err = ExpectedFS.takeError()) {
+consumeError(std::move(Err));
+return nullptr;
+  }
+  return std::move(*ExpectedFS);
+}
+
+static std::string createSimpleOverlay(StringRef RedirectKind, StringRef From,
+   StringRef To) {
+  return ("{\n"
+  "  'version': 0,\n"
+  "  'redirecting-with': '" +
+  RedirectKind +
+  "'\n"
+  "  'roots': [\n"
+  " {\n"
+  "   'type': 'directory-remap',\n"
+  "   'name': '" +
+  From +
+  "',\n"
+  "   'external-contents': '" +
+  To +
+  "',\n"
+  "   }]\n"
+  " }"
+  "  ]")
+  .str();
+}
+
+// Make sure that overlays are not transitive. Giv

[PATCH] D121426: [VFS] Update uses of getVFSFromYAML to use the new getVFSFromYAMLs API

[Ben Barham via Phabricator via cfe-commits]

bnbarham marked an inline comment as done.
bnbarham added a comment.

Merged into D121425  instead.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D121426/new/

https://reviews.llvm.org/D121426

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D121425: [VFS] Add a new getVFSFromYAMLs API

[Ben Barham via Phabricator via cfe-commits]

bnbarham added a comment.

There's two failing tests with this change:

- VFSFromYAMLTest.ReturnsExternalPathVFSHit
- VFSFromYAMLTest.ReturnsInternalPathVFSHit

Apparently we allow relative paths in external-contents *without* specifying 
`overlay-relative: true`. In this case the relative paths are resolved based on 
the CWD at the time of the operation. Do we really want that? I believe this 
wouldn't have worked prior to https://reviews.llvm.org/D109128 as there was no 
canonicalising before then as well. @keith was that change intentional? If we 
want it then it will *require* setting CWD on each FS in OverlayFS, which I was 
really hoping to avoid.

To give the concrete case:

  OverlayFileSystem
RedirectingFileSystem
  /a/foo -> bar
RealFileSystem
  /a/bar
  /b/bar
  
  cd /a
  cat foo # /a/bar
  
  cd /b # would fail previously since it doesn't exist in RedirectingFS but 
would still setCWD for RealFS
  cat foo # and thus return /a/bar since RedirectingFS is still in /a

This now fails completely because OverlayFS doesn't setCWD on the underlying 
filesystems. So RedirectingFS has the CWD of whatever ExternalFS was at 
creation. In D121424  it will fail because 
there's no longer any canonicalise after mapping the path in RedirectingFS (I 
assumed this was unintentional and missed the test case).

If we want to allow this then I think we'll need to go with the previous plan 
for OverlayFS, ie. keep track of filesystems that setCWD failed on and don't 
call them at all until they succeed again. That then doesn't allow certain 
cases that the current method allows.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D121425/new/

https://reviews.llvm.org/D121425

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D121425: [VFS] Add a new getVFSFromYAMLs API

[Ben Barham via Phabricator via cfe-commits]

bnbarham added a comment.

In D121425#3384188 , @bnbarham wrote:

> There's two failing tests with this change:
>
> - VFSFromYAMLTest.ReturnsExternalPathVFSHit
> - VFSFromYAMLTest.ReturnsInternalPathVFSHit
>
> Apparently we allow relative paths in external-contents *without* specifying 
> `overlay-relative: true`. In this case the relative paths are resolved based 
> on the CWD at the time of the operation. Do we really want that? I believe 
> this wouldn't have worked prior to https://reviews.llvm.org/D109128 as there 
> was no canonicalising before then as well. @keith was that change 
> intentional? If we want it then it will *require* setting CWD on each FS in 
> OverlayFS, which I was really hoping to avoid.

I spoke to Keith offline. This has always worked - it previously worked by 
`RedirectingFileSystem` setting CWD on `ExternalFS` when `setCWD` was called on 
it. It's also important to keep supporting as it's used in Bazel 
(https://github.com/bazelbuild/rules_swift/blob/c1d7d1df6969c2675c7826ecf1202d78016b1753/swift/internal/vfsoverlay.bzl#L41-L55).

I'm hoping I can fix this by resolving the paths when the overlay is created. 
I'll see if that works (it'll depend on when -working-directory is actually 
used by the frontend).


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D121425/new/

https://reviews.llvm.org/D121425

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D121425: [VFS] Add a new getVFSFromYAMLs API

[Ben Barham via Phabricator via cfe-commits]

bnbarham added a comment.

In D121425#3384492 , @dexonsmith 
wrote:

> Can you be more detailed about the overall state at the time of `cat`, which 
> causes it to fail?

Sure. I think there's also some confusion with names here but I'm generally 
trying to use the names as they are used in the overlay YAML or the name of the 
members in RedirectingFS.

"external-contents" is just the path that we're mapping to, so in my example 
above (/a/foo -> bar), "bar" is what I was referring to by "external-contents". 
So let's use that again -

  OverlayFileSystem
RedirectingFileSystem
  /a/foo -> bar
  
RealFileSystem
  /a/bar
  /b/bar

After D121423 , `cd /a` will result in the 
following -

- OverlayFS: CWD set to `/a`
- RedirectingFS and RealFS: CWD unchanged, so let's assume they were `/` at the 
time of creation and that's what they are now

`cat foo` -

- OverlayFS canonicalises to `/a/foo` and passes that to `openFileForRead` in 
RedirectingFS
- RedirectingFS has a `/a/foo` -> `bar` mapping, so it now runs 
`openFileForRead` on its ExternalFS (which is the RealFS)
- RealFS is at `/` and there is no `/bar` and thus fails
- RedirectingFS also fails
- Back in OverlayFS now, and because we failed we now try RealFS with `/a/foo`
- RealFS has no `/a/foo`, it fails
- Back in OverlayFS, all FS have now failed, return a failure

How this used to work really depends on when we're talking about 😆. Prior to 
D121423  the FS would have looked like:

  RedirectingFileSystem
/a/foo -> bar
  ExternalFS
RealFileSystem
  /a/bar
  /b/bar

Prior to Jonas' change, `setCWD` on RedirectingFS would run `setCWD` on 
`ExternalFS`. So `cd /a` would set CWD to `/a` for both RedirectingFS and 
RealFileSystem. Then `cat /a/foo` would give `bar` and `openFileForRead` on 
`ExternalFS` (RealFS) (which has `/a` CWD) would open `/a/bar`.

After Jonas' *and* Keith's change, `setCWD` no longer runs on `ExternalFS` but 
instead the path was canonicalised before sending it along. So `/a/foo` maps to 
`bar` as before, but then it would be canonicalised using `RedirectingFS` CWD 
and thus do an open for `/a/bar` on `ExternalFS`.

In D121425#3384499 , @dexonsmith 
wrote:

> (Would this mean resolving everything in the `.yaml` file eagerly at launch? 
> That sounds a bit scary...)

Yes, depending on what you mean by "resolving". All I mean is "prefix relative 
paths with CWD". The main issue with doing this is that it prohibits 
canonicalising paths to a virtual CWD, since you wouldn't be able to set CWD to 
a virtual path before making the overlay. Perhaps that's what you mean by "a 
bit scary" though.

We currently prefix relative paths with `ExternalContentsPrefixDir` if 
`overlay-relative: true` is set. That path is a combination of CWD + the 
directory to the overlay. This is to handle overlays that have absolute paths 
that we want to remap (specifically for the crash dump use case I believe). But 
if `overlay-relative: false` is set then we just canonicalise the path on 
open/etc.

I just ran a few tests with the frontend though:

- By default, paths come in relative and will become absolute from the 
process-wide CWD
- If `-working-directory` *is* set then paths come in absolute from 
`FileManager` - `setCWD` is never run

I'm not sure if there's a good reason it's done this way or it's just that CWD 
was only added to FileSystem relatively recently, but it does mean that that 
we're actually canonicalising the paths with the process-wide CWD regardless at 
the moment (at least from the frontend). This isn't the case if something was 
to use the API directly, but in that case I'm not sure that using the current 
CWD makes any more sense than using the CWD when the overlay is created. The 
semantics would be less confusing if that was the case (IMO), but again would 
prohibit canonicalising to a virtual CWD.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D121425/new/

https://reviews.llvm.org/D121425

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D121425: [VFS] Add a new getVFSFromYAMLs API

[Ben Barham via Phabricator via cfe-commits]

bnbarham planned changes to this revision.
bnbarham added a comment.

Blocked on the dependent


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D121425/new/

https://reviews.llvm.org/D121425

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D121733: Clean pathnames in FileManager.

[Ben Barham via Phabricator via cfe-commits]

bnbarham added a comment.

In D121733#3392931 , @dexonsmith 
wrote:

> However, FileManager changes sometimes have odd side effects... and it's 
> possible that somewhere in clang relies on having `FileManager::getFileRef()` 
> return precisely the same path that was requested. Tagging a few other people 
> that have some context... please share your opinions!

Wouldn't that already not be the case though (ie. given `RedirectingFileSystem` 
and `use-external-names` is currently a thing)? I know we're wanting to change 
that, but I don't *know* of anywhere that depends on this currently.

> @ppluzhnikov, can you give more context on how this interacts with 
> https://reviews.llvm.org/D121658? I had a quick look but it wasn't 
> immediately obvious.

If I understand correctly, the failing tests in that patch are failing because 
they're always expecting "/" and since `sys::append` is used, it's now "\\" on 
Windows. The remove dots change doesn't fix those, since the tests would still 
need to be updated to remove the "./" (which is most of the tests in this 
patch). But there's also some others where I wouldn't expect them to be failing 
in this patch, eg. the ones from `/` -> `{{[/\\]}}`.

Are there tests that we can't just fix to expect either `/` or `\\`? Why do we 
need to change the underlying behaviour here at all?


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D121733/new/

https://reviews.llvm.org/D121733

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D121733: Clean pathnames in FileManager.

[Ben Barham via Phabricator via cfe-commits]

bnbarham added a comment.

In D121733#3392968 , @ppluzhnikov 
wrote:

>> There's also some others where I wouldn't expect them to be failing in this 
>> patch, eg. the ones from `/` -> `{{[/\\]}}`.
>
> These are failing because `remove_dots` (un-intuitively) also changes 
> "foo/bar\\baz" to "foo\\bar\\baz"
>
>> Are there tests that we can't just fix to expect either `/` or `\\`?
>
> That's what this change is doing (but there is more Winx64 cleanup pending; 
> sorry, I should have sent this as a draft, but I am still learning the ropes 
> here).

Ah, I didn't realise that `remove_dots` also changed slashes.

>> Why do we need to change the underlying behaviour here at all?
>
> Sorry, I didn't understand that comment.

I was referring to the other patch here. I was under the impression that you 
started this patch to fix the failures in D121658 
, but perhaps that isn't the case and you 
just thought this was a generally nice cleanup.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D121733/new/

https://reviews.llvm.org/D121733

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D121733: Clean pathnames in FileManager.

[Ben Barham via Phabricator via cfe-commits]

bnbarham added a comment.

In D121733#3393546 , @rnk wrote:

> I've been somewhat afraid to touch this code because of symlinks. Is that 
> something we need to worry about? Consider this path: root/symlink/../foo.h. 
> remove_dots will turn this into root/foo.h, but if symlink points to some 
> unrelated directory, the .. directory entry points to something other than 
> root. I can't imagine that people rely on this behavior often, but I could be 
> wrong.

`remove_dots` doesn't remove `..` by default, it needs to be passed `true` to 
do that. This is only removing `.`.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D121733/new/

https://reviews.llvm.org/D121733

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D123398: [VFS] RedirectingFileSystem only replace path if not already mapped

[Ben Barham via Phabricator via cfe-commits]

bnbarham created this revision.
bnbarham added reviewers: dexonsmith, keith, JDevlieghere, vsapsai, sammccall.
Herald added a subscriber: hiraditya.
Herald added a project: All.
bnbarham requested review of this revision.
Herald added projects: clang, LLVM.
Herald added subscribers: llvm-commits, cfe-commits.

If the `ExternalFS` has already remapped to an external path then
`RedirectingFileSystem` should not change it to the originally provided
path. This fixes the original path always being used if multiple VFS
overlays were provided and the path wasn't found in the highest (ie.
first in the chain).

For now this is accomplished through the use of a new
`ExposesExternalVFSPath` field on `vfs::Status`. This flag is true when
the `Status` has an external path that's different from its virtual
path, ie. the contained path is the external path. See the plan in
`FileManager::getFileRef` for where this is going - eventually we won't
need `IsVFSMapped` any more and all returned paths should be virtual.

Resolves rdar://90578880 and llvm-project#53306.


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D123398

Files:
  clang/lib/Basic/FileManager.cpp
  clang/test/VFS/external-names-multi-overlay.c
  llvm/include/llvm/Support/VirtualFileSystem.h
  llvm/lib/Support/VirtualFileSystem.cpp
  llvm/unittests/Support/VirtualFileSystemTest.cpp

Index: llvm/unittests/Support/VirtualFileSystemTest.cpp
===
--- llvm/unittests/Support/VirtualFileSystemTest.cpp
+++ llvm/unittests/Support/VirtualFileSystemTest.cpp
@@ -1443,11 +1443,13 @@
   ASSERT_FALSE(S.getError());
   EXPECT_EQ("//root/foo/bar/a", S->getName());
   EXPECT_TRUE(S->IsVFSMapped);
+  EXPECT_TRUE(S->ExposesExternalVFSPath);
 
   ErrorOr SLower = O->status("//root/foo/bar/a");
   EXPECT_EQ("//root/foo/bar/a", SLower->getName());
   EXPECT_TRUE(S->equivalent(*SLower));
   EXPECT_FALSE(SLower->IsVFSMapped);
+  EXPECT_FALSE(SLower->ExposesExternalVFSPath);
 
   // file after opening
   auto OpenedF = O->openFileForRead("//root/file1");
@@ -1456,6 +1458,7 @@
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("//root/foo/bar/a", OpenedS->getName());
   EXPECT_TRUE(OpenedS->IsVFSMapped);
+  EXPECT_TRUE(OpenedS->ExposesExternalVFSPath);
 
   // directory
   S = O->status("//root/");
@@ -1468,26 +1471,30 @@
   ASSERT_FALSE(S.getError());
   EXPECT_TRUE(S->isDirectory());
   EXPECT_TRUE(S->IsVFSMapped);
+  EXPECT_TRUE(S->ExposesExternalVFSPath);
   EXPECT_TRUE(S->equivalent(*O->status("//root/foo/bar")));
 
   SLower = O->status("//root/foo/bar");
   EXPECT_EQ("//root/foo/bar", SLower->getName());
   EXPECT_TRUE(S->equivalent(*SLower));
   EXPECT_FALSE(SLower->IsVFSMapped);
+  EXPECT_FALSE(SLower->ExposesExternalVFSPath);
 
   // file in remapped directory
   S = O->status("//root/mappeddir/a");
   ASSERT_FALSE(S.getError());
-  ASSERT_FALSE(S->isDirectory());
-  ASSERT_TRUE(S->IsVFSMapped);
-  ASSERT_EQ("//root/foo/bar/a", S->getName());
+  EXPECT_FALSE(S->isDirectory());
+  EXPECT_TRUE(S->IsVFSMapped);
+  EXPECT_TRUE(S->ExposesExternalVFSPath);
+  EXPECT_EQ("//root/foo/bar/a", S->getName());
 
   // file in remapped directory, with use-external-name=false
   S = O->status("//root/mappeddir2/a");
   ASSERT_FALSE(S.getError());
-  ASSERT_FALSE(S->isDirectory());
-  ASSERT_TRUE(S->IsVFSMapped);
-  ASSERT_EQ("//root/mappeddir2/a", S->getName());
+  EXPECT_FALSE(S->isDirectory());
+  EXPECT_TRUE(S->IsVFSMapped);
+  EXPECT_FALSE(S->ExposesExternalVFSPath);
+  EXPECT_EQ("//root/mappeddir2/a", S->getName());
 
   // file contents in remapped directory
   OpenedF = O->openFileForRead("//root/mappeddir/a");
@@ -1496,6 +1503,7 @@
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("//root/foo/bar/a", OpenedS->getName());
   EXPECT_TRUE(OpenedS->IsVFSMapped);
+  EXPECT_TRUE(OpenedS->ExposesExternalVFSPath);
 
   // file contents in remapped directory, with use-external-name=false
   OpenedF = O->openFileForRead("//root/mappeddir2/a");
@@ -1504,6 +1512,7 @@
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("//root/mappeddir2/a", OpenedS->getName());
   EXPECT_TRUE(OpenedS->IsVFSMapped);
+  EXPECT_FALSE(OpenedS->ExposesExternalVFSPath);
 
   // broken mapping
   EXPECT_EQ(O->status("//root/file2").getError(),
@@ -1536,11 +1545,13 @@
   ASSERT_FALSE(S.getError());
   EXPECT_EQ("//root/foo/bar/a", S->getName());
   EXPECT_TRUE(S->IsVFSMapped);
+  EXPECT_TRUE(S->ExposesExternalVFSPath);
 
   ErrorOr SLower = O->status("//root/foo/bar/a");
   EXPECT_EQ("//root/foo/bar/a", SLower->getName());
   EXPECT_TRUE(S->equivalent(*SLower));
   EXPECT_FALSE(SLower->IsVFSMapped);
+  EXPECT_FALSE(SLower->ExposesExternalVFSPath);
 
   // file after opening
   auto OpenedF = O->openFileForRead("//mappedroot/a");
@@ -1549,6 +1560,7 @@
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("//root/foo/bar/a", OpenedS->getName());
   EXPECT_TRUE(OpenedS->IsVFSMapped);
+  EXPECT_TRUE(OpenedS->ExposesExternalVFSPath);
 
   EXPECT_EQ(0, N

[PATCH] D123104: [Modules] Use looked-up filename when looking for module maps

[Ben Barham via Phabricator via cfe-commits]

bnbarham abandoned this revision.
bnbarham added a comment.

Looks like there's more changes required for modulemap-collision.m to actually 
pass. I'll try figure those out when I have the time.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D123104/new/

https://reviews.llvm.org/D123104

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D123398: [VFS] RedirectingFileSystem only replace path if not already mapped

[Ben Barham via Phabricator via cfe-commits]

bnbarham added a comment.

Does `clang/test/VFS/external-names-multi-overlay.c` need to be formatted or is 
it fine? It uses split-file so I'd really like to avoid the format here (makes 
it pretty silly).


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D123398/new/

https://reviews.llvm.org/D123398

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D123398: [VFS] RedirectingFileSystem only replace path if not already mapped

[Ben Barham via Phabricator via cfe-commits]

This revision was landed with ongoing or failed builds.
This revision was automatically updated to reflect the committed changes.
Closed by commit rGfe2478d44e4f: [VFS] RedirectingFileSystem only replace path 
if not already mapped (authored by bnbarham).

Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D123398/new/

https://reviews.llvm.org/D123398

Files:
  clang/lib/Basic/FileManager.cpp
  clang/test/VFS/external-names-multi-overlay.c
  llvm/include/llvm/Support/VirtualFileSystem.h
  llvm/lib/Support/VirtualFileSystem.cpp
  llvm/unittests/Support/VirtualFileSystemTest.cpp

Index: llvm/unittests/Support/VirtualFileSystemTest.cpp
===
--- llvm/unittests/Support/VirtualFileSystemTest.cpp
+++ llvm/unittests/Support/VirtualFileSystemTest.cpp
@@ -1443,11 +1443,13 @@
   ASSERT_FALSE(S.getError());
   EXPECT_EQ("//root/foo/bar/a", S->getName());
   EXPECT_TRUE(S->IsVFSMapped);
+  EXPECT_TRUE(S->ExposesExternalVFSPath);
 
   ErrorOr SLower = O->status("//root/foo/bar/a");
   EXPECT_EQ("//root/foo/bar/a", SLower->getName());
   EXPECT_TRUE(S->equivalent(*SLower));
   EXPECT_FALSE(SLower->IsVFSMapped);
+  EXPECT_FALSE(SLower->ExposesExternalVFSPath);
 
   // file after opening
   auto OpenedF = O->openFileForRead("//root/file1");
@@ -1456,6 +1458,7 @@
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("//root/foo/bar/a", OpenedS->getName());
   EXPECT_TRUE(OpenedS->IsVFSMapped);
+  EXPECT_TRUE(OpenedS->ExposesExternalVFSPath);
 
   // directory
   S = O->status("//root/");
@@ -1468,26 +1471,30 @@
   ASSERT_FALSE(S.getError());
   EXPECT_TRUE(S->isDirectory());
   EXPECT_TRUE(S->IsVFSMapped);
+  EXPECT_TRUE(S->ExposesExternalVFSPath);
   EXPECT_TRUE(S->equivalent(*O->status("//root/foo/bar")));
 
   SLower = O->status("//root/foo/bar");
   EXPECT_EQ("//root/foo/bar", SLower->getName());
   EXPECT_TRUE(S->equivalent(*SLower));
   EXPECT_FALSE(SLower->IsVFSMapped);
+  EXPECT_FALSE(SLower->ExposesExternalVFSPath);
 
   // file in remapped directory
   S = O->status("//root/mappeddir/a");
   ASSERT_FALSE(S.getError());
-  ASSERT_FALSE(S->isDirectory());
-  ASSERT_TRUE(S->IsVFSMapped);
-  ASSERT_EQ("//root/foo/bar/a", S->getName());
+  EXPECT_FALSE(S->isDirectory());
+  EXPECT_TRUE(S->IsVFSMapped);
+  EXPECT_TRUE(S->ExposesExternalVFSPath);
+  EXPECT_EQ("//root/foo/bar/a", S->getName());
 
   // file in remapped directory, with use-external-name=false
   S = O->status("//root/mappeddir2/a");
   ASSERT_FALSE(S.getError());
-  ASSERT_FALSE(S->isDirectory());
-  ASSERT_TRUE(S->IsVFSMapped);
-  ASSERT_EQ("//root/mappeddir2/a", S->getName());
+  EXPECT_FALSE(S->isDirectory());
+  EXPECT_TRUE(S->IsVFSMapped);
+  EXPECT_FALSE(S->ExposesExternalVFSPath);
+  EXPECT_EQ("//root/mappeddir2/a", S->getName());
 
   // file contents in remapped directory
   OpenedF = O->openFileForRead("//root/mappeddir/a");
@@ -1496,6 +1503,7 @@
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("//root/foo/bar/a", OpenedS->getName());
   EXPECT_TRUE(OpenedS->IsVFSMapped);
+  EXPECT_TRUE(OpenedS->ExposesExternalVFSPath);
 
   // file contents in remapped directory, with use-external-name=false
   OpenedF = O->openFileForRead("//root/mappeddir2/a");
@@ -1504,6 +1512,7 @@
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("//root/mappeddir2/a", OpenedS->getName());
   EXPECT_TRUE(OpenedS->IsVFSMapped);
+  EXPECT_FALSE(OpenedS->ExposesExternalVFSPath);
 
   // broken mapping
   EXPECT_EQ(O->status("//root/file2").getError(),
@@ -1536,11 +1545,13 @@
   ASSERT_FALSE(S.getError());
   EXPECT_EQ("//root/foo/bar/a", S->getName());
   EXPECT_TRUE(S->IsVFSMapped);
+  EXPECT_TRUE(S->ExposesExternalVFSPath);
 
   ErrorOr SLower = O->status("//root/foo/bar/a");
   EXPECT_EQ("//root/foo/bar/a", SLower->getName());
   EXPECT_TRUE(S->equivalent(*SLower));
   EXPECT_FALSE(SLower->IsVFSMapped);
+  EXPECT_FALSE(SLower->ExposesExternalVFSPath);
 
   // file after opening
   auto OpenedF = O->openFileForRead("//mappedroot/a");
@@ -1549,6 +1560,7 @@
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("//root/foo/bar/a", OpenedS->getName());
   EXPECT_TRUE(OpenedS->IsVFSMapped);
+  EXPECT_TRUE(OpenedS->ExposesExternalVFSPath);
 
   EXPECT_EQ(0, NumDiagnostics);
 }
@@ -1697,11 +1709,13 @@
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("a", OpenedS->getName());
   EXPECT_FALSE(OpenedS->IsVFSMapped);
+  EXPECT_FALSE(OpenedS->ExposesExternalVFSPath);
 
   auto DirectS = RemappedFS->status("a");
   ASSERT_FALSE(DirectS.getError());
   EXPECT_EQ("a", DirectS->getName());
   EXPECT_FALSE(DirectS->IsVFSMapped);
+  EXPECT_FALSE(DirectS->ExposesExternalVFSPath);
 
   EXPECT_EQ(0, NumDiagnostics);
 }
@@ -1737,11 +1751,13 @@
   ASSERT_FALSE(OpenedS.getError());
   EXPECT_EQ("realname", OpenedS->getName());
   EXPECT_TRUE(OpenedS->IsVFSMapped);
+  EXPECT_TRUE(OpenedS->ExposesExternalVFSPath);
 
   auto DirectS = FS->status("vfsname");
   ASSERT_FALSE(DirectS.getError());
   EXPECT_E

[PATCH] D123574: [clang] NFCI: Use FileEntryRef in PPCallbacks::InclusionDirective

[Ben Barham via Phabricator via cfe-commits]

bnbarham accepted this revision.
bnbarham added a comment.

Thanks Jan :)!


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D123574/new/

https://reviews.llvm.org/D123574

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D123767: [clang][parse] NFCI: Use FileEntryRef in Parser::ParseModuleImport()

[Ben Barham via Phabricator via cfe-commits]

bnbarham accepted this revision.
bnbarham added a comment.
This revision is now accepted and ready to land.

Thanks for doing all these Jan!


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D123767/new/

https://reviews.llvm.org/D123767

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D123771: [clang][lex] NFCI: Use DirectoryEntryRef in HeaderSearch::load*()

[Ben Barham via Phabricator via cfe-commits]

bnbarham added inline comments.



Comment at: clang/lib/Lex/HeaderSearch.cpp:339
 // Search for a module map file in this directory.
-if (loadModuleMapFile(Dir.getDir(), IsSystem,
+if (loadModuleMapFile(*Dir.getDirRef(), IsSystem,
   /*IsFramework*/false) == LMM_NewlyLoaded) {

I'd prefer the following since I had to specifically go and check `getDirRef`, 
though I suppose I probably would have done so even with a comment 😅 :
```
// Only returns None if not a normal directory, which we just checked
DirectoryEntryRef NormalDir = *Dir.getDirRef();
...loadModuleMapFile(NormalDir, ...
```



Comment at: clang/lib/Lex/HeaderSearch.cpp:1583
+::getTopFrameworkDir(FileMgr, FrameworkName, SubmodulePath);
+assert(TopFrameworkDir && "Could not find the top-most framework dir");
 

Can we just return false in this case, or is it definitely always a logic error?



Comment at: clang/lib/Lex/HeaderSearch.cpp:1630
+  llvm::sys::path::parent_path(OriginalModuleMapFile))) {
+Dir = *MaybeDir;
   } else {

Shouldn't need the * should it? Also doesn't seem necessary to actually check 
here since `Dir` hasn't been set yet. Just `Dir = 
FileMgr.getOptionalDirectoryRef...` should be fine.



Comment at: clang/lib/Lex/HeaderSearch.cpp:1636
 } else {
-  Dir = File->getDir();
+  Dir = File->getLastRef().getDir();
 }

I assume this is the place you mentioned in the message? (ie. to prevent patch 
from growing even more)



Comment at: clang/lib/Lex/HeaderSearch.cpp:1639
 
 StringRef DirName(Dir->getName());
 if (llvm::sys::path::filename(DirName) == "Modules") {

Can't `Dir` still be `None` here? It *shouldn't* be since the directory for 
`OriginalModuleMapFile` should exist, but I'd feel more comfortable with either 
an early return or an assert here.

Same thing below at the switch as well.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D123771/new/

https://reviews.llvm.org/D123771

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D123771: [clang][lex] NFCI: Use DirectoryEntryRef in HeaderSearch::load*()

[Ben Barham via Phabricator via cfe-commits]

bnbarham accepted this revision.
bnbarham added inline comments.
This revision is now accepted and ready to land.



Comment at: clang/lib/Lex/HeaderSearch.cpp:1583
+::getTopFrameworkDir(FileMgr, FrameworkName, SubmodulePath);
+assert(TopFrameworkDir && "Could not find the top-most framework dir");
 

jansvoboda11 wrote:
> bnbarham wrote:
> > Can we just return false in this case, or is it definitely always a logic 
> > error?
> I agree returning `false` here would be better, but I wanted to keep this 
> patch as NFC as possible. How about I make that change in a follow-up patch?
Sure. It would have been a segfault before, so I assume it must never happen 
anyway.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D123771/new/

https://reviews.llvm.org/D123771

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D123854: [clang][lex] NFCI: Use DirectoryEntryRef in FrameworkCacheEntry

[Ben Barham via Phabricator via cfe-commits]

bnbarham accepted this revision.
bnbarham added inline comments.
This revision is now accepted and ready to land.



Comment at: clang/lib/Lex/HeaderSearch.cpp:584
   // If it is known and in some other directory, fail.
-  if (CacheEntry.Directory && CacheEntry.Directory != getFrameworkDir())
+  if (CacheEntry.Directory && *CacheEntry.Directory != getFrameworkDir())
 return None;





Comment at: clang/lib/Lex/HeaderSearch.cpp:607
 // If the framework dir doesn't exist, we fail.
 auto Dir = FileMgr.getDirectory(FrameworkName);
 if (!Dir)

Is the plan to do these ones in later commits? Ie. this just does what's needed 
for `FrameworkCacheEntry`?



Comment at: clang/lib/Lex/HeaderSearch.cpp:1192
 // framework.
 CacheLookup.second.Directory = *Dir;
   }

Don't need the `*`


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D123854/new/

https://reviews.llvm.org/D123854

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D123856: [clang][lex] NFCI: Use FileEntryRef in ModuleMap::diagnoseHeaderInclusion()

[Ben Barham via Phabricator via cfe-commits]

bnbarham accepted this revision.
bnbarham added a comment.
This revision is now accepted and ready to land.

Shadowing of `FE` almost tripped me up there 😅


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D123856/new/

https://reviews.llvm.org/D123856

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D124288: [Index] Add a USR and symbol kind for UnresolvedUsingIfExists

[Ben Barham via Phabricator via cfe-commits]

bnbarham created this revision.
bnbarham added reviewers: benlangmuir, arphaman, jansvoboda11.
Herald added a project: All.
bnbarham requested review of this revision.
Herald added a project: clang.
Herald added a subscriber: cfe-commits.

`UnresolvedUsingIfExists` has existed for a long time now, but it never
had a USR or symbol kind added. Add those now.


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D124288

Files:
  clang/include/clang/Index/IndexSymbol.h
  clang/lib/Index/IndexSymbol.cpp
  clang/lib/Index/USRGeneration.cpp
  clang/test/Index/using_if_exists.cpp


Index: clang/test/Index/using_if_exists.cpp
===
--- /dev/null
+++ clang/test/Index/using_if_exists.cpp
@@ -0,0 +1,9 @@
+// RUN: c-index-test core -print-source-symbols -- %s -target 
x86_64-unknown-unknown 2>&1 | FileCheck %s
+
+namespace ns {
+//  void foo();
+}
+
+using ns::foo __attribute__((using_if_exists));
+// CHECK: [[@LINE-1]]:11 | using/C++ | foo | c:@UD@foo |  | Decl | 
rel: 0
+// CHECK: [[@LINE-2]]:11 | using/using-unresolved/C++ | foo | 
c:using_if_exists.cpp@UUIE@foo |  | Ref | rel: 0
Index: clang/lib/Index/USRGeneration.cpp
===
--- clang/lib/Index/USRGeneration.cpp
+++ clang/lib/Index/USRGeneration.cpp
@@ -103,6 +103,7 @@
   void VisitTemplateTemplateParmDecl(const TemplateTemplateParmDecl *D);
   void VisitUnresolvedUsingValueDecl(const UnresolvedUsingValueDecl *D);
   void VisitUnresolvedUsingTypenameDecl(const UnresolvedUsingTypenameDecl *D);
+  void VisitUnresolvedUsingIfExistsDecl(const UnresolvedUsingIfExistsDecl *D);
 
   void VisitLinkageSpecDecl(const LinkageSpecDecl *D) {
 IgnoreResults = true; // No USRs for linkage specs themselves.
@@ -1007,7 +1008,14 @@
   Out << D->getName(); // Simple name.
 }
 
-
+void USRGenerator::VisitUnresolvedUsingIfExistsDecl(
+const UnresolvedUsingIfExistsDecl *D) {
+  if (ShouldGenerateLocation(D) && GenLoc(D, /*IncludeOffset=*/isLocal(D)))
+return;
+  VisitDeclContext(D->getDeclContext());
+  Out << "@UUIE@";
+  Out << D->getName();
+}
 
 
//===--===//
 // USR generation functions.
Index: clang/lib/Index/IndexSymbol.cpp
===
--- clang/lib/Index/IndexSymbol.cpp
+++ clang/lib/Index/IndexSymbol.cpp
@@ -334,6 +334,11 @@
   Info.Lang = SymbolLanguage::CXX;
   Info.SubKind = SymbolSubKind::UsingEnum;
   break;
+case Decl::UnresolvedUsingIfExists:
+  Info.Kind = SymbolKind::Using;
+  Info.SubKind = SymbolSubKind::UnresolvedUsing;
+  Info.Lang = SymbolLanguage::CXX;
+  break;
 case Decl::Binding:
   Info.Kind = SymbolKind::Variable;
   Info.Lang = SymbolLanguage::CXX;
@@ -549,6 +554,8 @@
   case SymbolSubKind::UsingValue: return "using-value";
   case SymbolSubKind::UsingEnum:
 return "using-enum";
+  case SymbolSubKind::UnresolvedUsing:
+return "using-unresolved";
   }
   llvm_unreachable("invalid symbol subkind");
 }
Index: clang/include/clang/Index/IndexSymbol.h
===
--- clang/include/clang/Index/IndexSymbol.h
+++ clang/include/clang/Index/IndexSymbol.h
@@ -76,6 +76,7 @@
   UsingTypename,
   UsingValue,
   UsingEnum,
+  UnresolvedUsing,
 };
 
 typedef uint16_t SymbolPropertySet;


Index: clang/test/Index/using_if_exists.cpp
===
--- /dev/null
+++ clang/test/Index/using_if_exists.cpp
@@ -0,0 +1,9 @@
+// RUN: c-index-test core -print-source-symbols -- %s -target x86_64-unknown-unknown 2>&1 | FileCheck %s
+
+namespace ns {
+//  void foo();
+}
+
+using ns::foo __attribute__((using_if_exists));
+// CHECK: [[@LINE-1]]:11 | using/C++ | foo | c:@UD@foo |  | Decl | rel: 0
+// CHECK: [[@LINE-2]]:11 | using/using-unresolved/C++ | foo | c:using_if_exists.cpp@UUIE@foo |  | Ref | rel: 0
Index: clang/lib/Index/USRGeneration.cpp
===
--- clang/lib/Index/USRGeneration.cpp
+++ clang/lib/Index/USRGeneration.cpp
@@ -103,6 +103,7 @@
   void VisitTemplateTemplateParmDecl(const TemplateTemplateParmDecl *D);
   void VisitUnresolvedUsingValueDecl(const UnresolvedUsingValueDecl *D);
   void VisitUnresolvedUsingTypenameDecl(const UnresolvedUsingTypenameDecl *D);
+  void VisitUnresolvedUsingIfExistsDecl(const UnresolvedUsingIfExistsDecl *D);
 
   void VisitLinkageSpecDecl(const LinkageSpecDecl *D) {
 IgnoreResults = true; // No USRs for linkage specs themselves.
@@ -1007,7 +1008,14 @@
   Out << D->getName(); // Simple name.
 }
 
-
+void USRGenerator::VisitUnresolvedUsingIfExistsDecl(
+const UnresolvedUsingIfExistsDecl *D) {
+  if (ShouldGenerateLocation(D) && GenLoc(D, /*IncludeOffset=*/isLocal(D)))
+return;
+  VisitDeclContext(D->getDeclContext());
+  Out << "@UUIE@";
+  Out << D->get

[PATCH] D124288: [Index] Add a USR and symbol kind for UnresolvedUsingIfExists

[Ben Barham via Phabricator via cfe-commits]

bnbarham added inline comments.



Comment at: clang/test/Index/using_if_exists.cpp:9
+// CHECK: [[@LINE-1]]:11 | using/C++ | foo | c:@UD@foo |  | Decl | 
rel: 0
+// CHECK: [[@LINE-2]]:11 | using/using-unresolved/C++ | foo | 
c:using_if_exists.cpp@UUIE@foo |  | Ref | rel: 0

The AST here is:
UsingDecl -> UsingShadowDecl -> UnresolvedUsingIfExistsDecl

So the `UnresolvedUsingIfExistsDecl` is a reference, which is a bit weird since 
there is no decl. But... there is no decl ("foo" doesn't exist), so this seems 
somewhat reasonable. Any objections?


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D124288/new/

https://reviews.llvm.org/D124288

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D124288: [Index] Add a USR and symbol kind for UnresolvedUsingIfExists

[Ben Barham via Phabricator via cfe-commits]

bnbarham updated this revision to Diff 424621.
bnbarham added a comment.

After speaking with Ben, we decided it makes more sense to just remove the 
reference entirely.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D124288/new/

https://reviews.llvm.org/D124288

Files:
  clang/lib/Index/IndexDecl.cpp
  clang/test/Index/using_if_exists.cpp


Index: clang/test/Index/using_if_exists.cpp
===
--- /dev/null
+++ clang/test/Index/using_if_exists.cpp
@@ -0,0 +1,9 @@
+// RUN: c-index-test core -print-source-symbols -- %s -target 
x86_64-unknown-unknown 2>&1 | FileCheck %s
+
+namespace ns {
+//  void foo();
+}
+
+using ns::foo __attribute__((using_if_exists));
+// CHECK: [[@LINE-1]]:11 | using/C++ | foo | c:@UD@foo |  | Decl | 
rel: 0
+// CHECK-NOT: [[@LINE-2]]:11 | 
Index: clang/lib/Index/IndexDecl.cpp
===
--- clang/lib/Index/IndexDecl.cpp
+++ clang/lib/Index/IndexDecl.cpp
@@ -605,9 +605,16 @@
 const NamedDecl *Parent = dyn_cast(DC);
 IndexCtx.indexNestedNameSpecifierLoc(D->getQualifierLoc(), Parent,
  D->getLexicalDeclContext());
-for (const auto *I : D->shadows())
+for (const auto *I : D->shadows()) {
+  // Skip unresolved using decls - we already have a decl for the using
+  // itself, so there's not much point adding another decl or reference to
+  // refer to the same location.
+  if (isa(I->getUnderlyingDecl()))
+continue;
+
   IndexCtx.handleReference(I->getUnderlyingDecl(), D->getLocation(), 
Parent,
D->getLexicalDeclContext(), SymbolRoleSet());
+}
 return true;
   }
 


Index: clang/test/Index/using_if_exists.cpp
===
--- /dev/null
+++ clang/test/Index/using_if_exists.cpp
@@ -0,0 +1,9 @@
+// RUN: c-index-test core -print-source-symbols -- %s -target x86_64-unknown-unknown 2>&1 | FileCheck %s
+
+namespace ns {
+//  void foo();
+}
+
+using ns::foo __attribute__((using_if_exists));
+// CHECK: [[@LINE-1]]:11 | using/C++ | foo | c:@UD@foo |  | Decl | rel: 0
+// CHECK-NOT: [[@LINE-2]]:11 | 
Index: clang/lib/Index/IndexDecl.cpp
===
--- clang/lib/Index/IndexDecl.cpp
+++ clang/lib/Index/IndexDecl.cpp
@@ -605,9 +605,16 @@
 const NamedDecl *Parent = dyn_cast(DC);
 IndexCtx.indexNestedNameSpecifierLoc(D->getQualifierLoc(), Parent,
  D->getLexicalDeclContext());
-for (const auto *I : D->shadows())
+for (const auto *I : D->shadows()) {
+  // Skip unresolved using decls - we already have a decl for the using
+  // itself, so there's not much point adding another decl or reference to
+  // refer to the same location.
+  if (isa(I->getUnderlyingDecl()))
+continue;
+
   IndexCtx.handleReference(I->getUnderlyingDecl(), D->getLocation(), Parent,
D->getLexicalDeclContext(), SymbolRoleSet());
+}
 return true;
   }
 
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D124288: [Index] Remove reference to `UnresolvedUsingIfExists`

[Ben Barham via Phabricator via cfe-commits]

bnbarham updated this revision to Diff 424622.
bnbarham retitled this revision from "[Index] Add a USR and symbol kind for 
UnresolvedUsingIfExists" to "[Index] Remove reference to 
`UnresolvedUsingIfExists`".
bnbarham edited the summary of this revision.
bnbarham added a comment.

Update title/message


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D124288/new/

https://reviews.llvm.org/D124288

Files:
  clang/lib/Index/IndexDecl.cpp
  clang/test/Index/using_if_exists.cpp


Index: clang/test/Index/using_if_exists.cpp
===
--- /dev/null
+++ clang/test/Index/using_if_exists.cpp
@@ -0,0 +1,9 @@
+// RUN: c-index-test core -print-source-symbols -- %s -target 
x86_64-unknown-unknown 2>&1 | FileCheck %s
+
+namespace ns {
+//  void foo();
+}
+
+using ns::foo __attribute__((using_if_exists));
+// CHECK: [[@LINE-1]]:11 | using/C++ | foo | c:@UD@foo |  | Decl | 
rel: 0
+// CHECK-NOT: [[@LINE-2]]:11 | 
Index: clang/lib/Index/IndexDecl.cpp
===
--- clang/lib/Index/IndexDecl.cpp
+++ clang/lib/Index/IndexDecl.cpp
@@ -605,9 +605,16 @@
 const NamedDecl *Parent = dyn_cast(DC);
 IndexCtx.indexNestedNameSpecifierLoc(D->getQualifierLoc(), Parent,
  D->getLexicalDeclContext());
-for (const auto *I : D->shadows())
+for (const auto *I : D->shadows()) {
+  // Skip unresolved using decls - we already have a decl for the using
+  // itself, so there's not much point adding another decl or reference to
+  // refer to the same location.
+  if (isa(I->getUnderlyingDecl()))
+continue;
+
   IndexCtx.handleReference(I->getUnderlyingDecl(), D->getLocation(), 
Parent,
D->getLexicalDeclContext(), SymbolRoleSet());
+}
 return true;
   }
 


Index: clang/test/Index/using_if_exists.cpp
===
--- /dev/null
+++ clang/test/Index/using_if_exists.cpp
@@ -0,0 +1,9 @@
+// RUN: c-index-test core -print-source-symbols -- %s -target x86_64-unknown-unknown 2>&1 | FileCheck %s
+
+namespace ns {
+//  void foo();
+}
+
+using ns::foo __attribute__((using_if_exists));
+// CHECK: [[@LINE-1]]:11 | using/C++ | foo | c:@UD@foo |  | Decl | rel: 0
+// CHECK-NOT: [[@LINE-2]]:11 | 
Index: clang/lib/Index/IndexDecl.cpp
===
--- clang/lib/Index/IndexDecl.cpp
+++ clang/lib/Index/IndexDecl.cpp
@@ -605,9 +605,16 @@
 const NamedDecl *Parent = dyn_cast(DC);
 IndexCtx.indexNestedNameSpecifierLoc(D->getQualifierLoc(), Parent,
  D->getLexicalDeclContext());
-for (const auto *I : D->shadows())
+for (const auto *I : D->shadows()) {
+  // Skip unresolved using decls - we already have a decl for the using
+  // itself, so there's not much point adding another decl or reference to
+  // refer to the same location.
+  if (isa(I->getUnderlyingDecl()))
+continue;
+
   IndexCtx.handleReference(I->getUnderlyingDecl(), D->getLocation(), Parent,
D->getLexicalDeclContext(), SymbolRoleSet());
+}
 return true;
   }
 
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D124288: [Index] Remove reference to `UnresolvedUsingIfExists`

[Ben Barham via Phabricator via cfe-commits]

bnbarham updated this revision to Diff 424631.
bnbarham added a comment.

Remove line for  check


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D124288/new/

https://reviews.llvm.org/D124288

Files:
  clang/lib/Index/IndexDecl.cpp
  clang/test/Index/using_if_exists.cpp


Index: clang/test/Index/using_if_exists.cpp
===
--- /dev/null
+++ clang/test/Index/using_if_exists.cpp
@@ -0,0 +1,9 @@
+// RUN: c-index-test core -print-source-symbols -- %s -target 
x86_64-unknown-unknown 2>&1 | FileCheck %s
+
+namespace ns {
+//  void foo();
+}
+
+using ns::foo __attribute__((using_if_exists));
+// CHECK: [[@LINE-1]]:11 | using/C++ | foo | c:@UD@foo |  | Decl | 
rel: 0
+// CHECK-NOT: 
Index: clang/lib/Index/IndexDecl.cpp
===
--- clang/lib/Index/IndexDecl.cpp
+++ clang/lib/Index/IndexDecl.cpp
@@ -605,9 +605,16 @@
 const NamedDecl *Parent = dyn_cast(DC);
 IndexCtx.indexNestedNameSpecifierLoc(D->getQualifierLoc(), Parent,
  D->getLexicalDeclContext());
-for (const auto *I : D->shadows())
+for (const auto *I : D->shadows()) {
+  // Skip unresolved using decls - we already have a decl for the using
+  // itself, so there's not much point adding another decl or reference to
+  // refer to the same location.
+  if (isa(I->getUnderlyingDecl()))
+continue;
+
   IndexCtx.handleReference(I->getUnderlyingDecl(), D->getLocation(), 
Parent,
D->getLexicalDeclContext(), SymbolRoleSet());
+}
 return true;
   }
 


Index: clang/test/Index/using_if_exists.cpp
===
--- /dev/null
+++ clang/test/Index/using_if_exists.cpp
@@ -0,0 +1,9 @@
+// RUN: c-index-test core -print-source-symbols -- %s -target x86_64-unknown-unknown 2>&1 | FileCheck %s
+
+namespace ns {
+//  void foo();
+}
+
+using ns::foo __attribute__((using_if_exists));
+// CHECK: [[@LINE-1]]:11 | using/C++ | foo | c:@UD@foo |  | Decl | rel: 0
+// CHECK-NOT: 
Index: clang/lib/Index/IndexDecl.cpp
===
--- clang/lib/Index/IndexDecl.cpp
+++ clang/lib/Index/IndexDecl.cpp
@@ -605,9 +605,16 @@
 const NamedDecl *Parent = dyn_cast(DC);
 IndexCtx.indexNestedNameSpecifierLoc(D->getQualifierLoc(), Parent,
  D->getLexicalDeclContext());
-for (const auto *I : D->shadows())
+for (const auto *I : D->shadows()) {
+  // Skip unresolved using decls - we already have a decl for the using
+  // itself, so there's not much point adding another decl or reference to
+  // refer to the same location.
+  if (isa(I->getUnderlyingDecl()))
+continue;
+
   IndexCtx.handleReference(I->getUnderlyingDecl(), D->getLocation(), Parent,
D->getLexicalDeclContext(), SymbolRoleSet());
+}
 return true;
   }
 
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D124288: [Index] Remove reference to `UnresolvedUsingIfExists`

[Ben Barham via Phabricator via cfe-commits]

This revision was automatically updated to reflect the committed changes.
Closed by commit rG089b6efefc3d: [Index] Remove reference to 
`UnresolvedUsingIfExists` (authored by bnbarham).

Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D124288/new/

https://reviews.llvm.org/D124288

Files:
  clang/lib/Index/IndexDecl.cpp
  clang/test/Index/using_if_exists.cpp


Index: clang/test/Index/using_if_exists.cpp
===
--- /dev/null
+++ clang/test/Index/using_if_exists.cpp
@@ -0,0 +1,9 @@
+// RUN: c-index-test core -print-source-symbols -- %s -target 
x86_64-unknown-unknown 2>&1 | FileCheck %s
+
+namespace ns {
+//  void foo();
+}
+
+using ns::foo __attribute__((using_if_exists));
+// CHECK: [[@LINE-1]]:11 | using/C++ | foo | c:@UD@foo |  | Decl | 
rel: 0
+// CHECK-NOT: 
Index: clang/lib/Index/IndexDecl.cpp
===
--- clang/lib/Index/IndexDecl.cpp
+++ clang/lib/Index/IndexDecl.cpp
@@ -605,9 +605,16 @@
 const NamedDecl *Parent = dyn_cast(DC);
 IndexCtx.indexNestedNameSpecifierLoc(D->getQualifierLoc(), Parent,
  D->getLexicalDeclContext());
-for (const auto *I : D->shadows())
+for (const auto *I : D->shadows()) {
+  // Skip unresolved using decls - we already have a decl for the using
+  // itself, so there's not much point adding another decl or reference to
+  // refer to the same location.
+  if (isa(I->getUnderlyingDecl()))
+continue;
+
   IndexCtx.handleReference(I->getUnderlyingDecl(), D->getLocation(), 
Parent,
D->getLexicalDeclContext(), SymbolRoleSet());
+}
 return true;
   }
 


Index: clang/test/Index/using_if_exists.cpp
===
--- /dev/null
+++ clang/test/Index/using_if_exists.cpp
@@ -0,0 +1,9 @@
+// RUN: c-index-test core -print-source-symbols -- %s -target x86_64-unknown-unknown 2>&1 | FileCheck %s
+
+namespace ns {
+//  void foo();
+}
+
+using ns::foo __attribute__((using_if_exists));
+// CHECK: [[@LINE-1]]:11 | using/C++ | foo | c:@UD@foo |  | Decl | rel: 0
+// CHECK-NOT: 
Index: clang/lib/Index/IndexDecl.cpp
===
--- clang/lib/Index/IndexDecl.cpp
+++ clang/lib/Index/IndexDecl.cpp
@@ -605,9 +605,16 @@
 const NamedDecl *Parent = dyn_cast(DC);
 IndexCtx.indexNestedNameSpecifierLoc(D->getQualifierLoc(), Parent,
  D->getLexicalDeclContext());
-for (const auto *I : D->shadows())
+for (const auto *I : D->shadows()) {
+  // Skip unresolved using decls - we already have a decl for the using
+  // itself, so there's not much point adding another decl or reference to
+  // refer to the same location.
+  if (isa(I->getUnderlyingDecl()))
+continue;
+
   IndexCtx.handleReference(I->getUnderlyingDecl(), D->getLocation(), Parent,
D->getLexicalDeclContext(), SymbolRoleSet());
+}
 return true;
   }
 
___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D91580: [Frontend] Add flag to allow PCM generation despite compiler errors

[Ben Barham via Phabricator via cfe-commits]

bnbarham created this revision.
bnbarham added a reviewer: akyrtzi.
Herald added subscribers: cfe-commits, dang, arphaman.
Herald added a project: clang.
bnbarham requested review of this revision.

As with precompiled headers, it's useful for indexers to be able to
continue through compiler errors in dependent modules.

Resolves rdar://69816264


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D91580

Files:
  clang/include/clang/Driver/Options.td
  clang/include/clang/Frontend/ASTUnit.h
  clang/include/clang/Frontend/FrontendActions.h
  clang/include/clang/Frontend/FrontendOptions.h
  clang/lib/Frontend/ASTUnit.cpp
  clang/lib/Frontend/CompilerInstance.cpp
  clang/lib/Frontend/CompilerInvocation.cpp
  clang/lib/Frontend/FrontendActions.cpp
  clang/test/Modules/Inputs/error.h
  clang/test/Modules/Inputs/module.map
  clang/test/Modules/load-module-with-errors.m
  clang/tools/c-index-test/core_main.cpp
  clang/tools/libclang/CIndex.cpp

Index: clang/tools/libclang/CIndex.cpp
===
--- clang/tools/libclang/CIndex.cpp
+++ clang/tools/libclang/CIndex.cpp
@@ -3476,7 +3476,7 @@
   ast_filename, CXXIdx->getPCHContainerOperations()->getRawReader(),
   ASTUnit::LoadEverything, Diags, FileSystemOpts, /*UseDebugInfo=*/false,
   CXXIdx->getOnlyLocalDecls(), None, CaptureDiagsKind::All,
-  /*AllowPCHWithCompilerErrors=*/true,
+  /*AllowASTWithCompilerErrors=*/true,
   /*UserFilesAreVolatile=*/true);
   *out_TU = MakeCXTranslationUnit(CXXIdx, std::move(AU));
   return *out_TU ? CXError_Success : CXError_Failure;
Index: clang/tools/c-index-test/core_main.cpp
===
--- clang/tools/c-index-test/core_main.cpp
+++ clang/tools/c-index-test/core_main.cpp
@@ -263,7 +263,7 @@
   std::string(modulePath), *pchRdr, ASTUnit::LoadASTOnly, Diags,
   FileSystemOpts, /*UseDebugInfo=*/false,
   /*OnlyLocalDecls=*/true, None, CaptureDiagsKind::None,
-  /*AllowPCHWithCompilerErrors=*/true,
+  /*AllowASTWithCompilerErrors=*/true,
   /*UserFilesAreVolatile=*/false);
   if (!AU) {
 errs() << "failed to create TU for: " << modulePath << '\n';
Index: clang/test/Modules/load-module-with-errors.m
===
--- /dev/null
+++ clang/test/Modules/load-module-with-errors.m
@@ -0,0 +1,14 @@
+// RUN: rm -rf %t
+
+// RUN: %clang_cc1 -fmodules -fdisable-module-hash -fallow-pcm-with-compiler-errors -fmodules-cache-path=%t -emit-module -x objective-c -fmodule-name=error %S/Inputs/module.map
+// RUN: %clang_cc1 -fmodules -fdisable-module-hash -fimplicit-module-maps -fallow-pcm-with-compiler-errors -ast-print -x objective-c -fmodules-cache-path=%t -I %S/Inputs %s | FileCheck -check-prefix=CHECK-PRINT %s
+
+@import error;
+
+void test(id x) {
+  [x method];
+}
+
+// CHECK-PRINT: @interface Error
+// CHECK-PRINT-NEXT: - (int)method;
+// CHECK-PRINT: void test(id x)
Index: clang/test/Modules/Inputs/module.map
===
--- clang/test/Modules/Inputs/module.map
+++ clang/test/Modules/Inputs/module.map
@@ -483,3 +483,4 @@
   header "template-nontrivial1.h"
   export *
 }
+module error { header "error.h" }
Index: clang/test/Modules/Inputs/error.h
===
--- /dev/null
+++ clang/test/Modules/Inputs/error.h
@@ -0,0 +1,8 @@
+@import undefined
+
+@interface Error
+- (int)method;
+undefined
+@end
+
+undefined
Index: clang/lib/Frontend/FrontendActions.cpp
===
--- clang/lib/Frontend/FrontendActions.cpp
+++ clang/lib/Frontend/FrontendActions.cpp
@@ -177,7 +177,8 @@
   Consumers.push_back(std::make_unique(
   CI.getPreprocessor(), CI.getModuleCache(), OutputFile, Sysroot, Buffer,
   CI.getFrontendOpts().ModuleFileExtensions,
-  /*AllowASTWithErrors=*/false,
+  /*AllowASTWithErrors=*/
+  +CI.getFrontendOpts().AllowPCMWithCompilerErrors,
   /*IncludeTimestamps=*/
   +CI.getFrontendOpts().BuildingImplicitModule,
   /*ShouldCacheASTInMemory=*/
@@ -187,6 +188,11 @@
   return std::make_unique(std::move(Consumers));
 }
 
+bool GenerateModuleAction::shouldEraseOutputFiles() {
+  return !getCompilerInstance().getFrontendOpts().AllowPCMWithCompilerErrors &&
+ ASTFrontendAction::shouldEraseOutputFiles();
+}
+
 bool GenerateModuleFromModuleMapAction::BeginSourceFileAction(
 CompilerInstance &CI) {
   if (!CI.getLangOpts().Modules) {
@@ -339,7 +345,7 @@
   CI.getPCHContainerReader(), CI.getFrontendOpts().ModuleFileExtensions,
   Sysroot.empty() ? "" : Sysroot.c_str(),
   /*DisableValidation*/ false,
-  /*AllowPCHWithCompilerErrors*/ false,
+  /*AllowASTWithCompilerErrors*/ false,
   /*AllowConfigurationMismatch*/ true,
   /*ValidateSystemInputs*/ true));
 
Index: clang/lib/Frontend/CompilerInv

[PATCH] D91580: [Frontend] Add flag to allow PCM generation despite compiler errors

[Ben Barham via Phabricator via cfe-commits]

bnbarham updated this revision to Diff 305622.
bnbarham added a comment.

Noticed I had left in the `-fdisable-module-hash` flags in the test, removed 
now.


CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D91580/new/

https://reviews.llvm.org/D91580

Files:
  clang/include/clang/Driver/Options.td
  clang/include/clang/Frontend/ASTUnit.h
  clang/include/clang/Frontend/FrontendActions.h
  clang/include/clang/Frontend/FrontendOptions.h
  clang/lib/Frontend/ASTUnit.cpp
  clang/lib/Frontend/CompilerInstance.cpp
  clang/lib/Frontend/CompilerInvocation.cpp
  clang/lib/Frontend/FrontendActions.cpp
  clang/test/Modules/Inputs/error.h
  clang/test/Modules/Inputs/module.map
  clang/test/Modules/load-module-with-errors.m
  clang/tools/c-index-test/core_main.cpp
  clang/tools/libclang/CIndex.cpp

Index: clang/tools/libclang/CIndex.cpp
===
--- clang/tools/libclang/CIndex.cpp
+++ clang/tools/libclang/CIndex.cpp
@@ -3476,7 +3476,7 @@
   ast_filename, CXXIdx->getPCHContainerOperations()->getRawReader(),
   ASTUnit::LoadEverything, Diags, FileSystemOpts, /*UseDebugInfo=*/false,
   CXXIdx->getOnlyLocalDecls(), None, CaptureDiagsKind::All,
-  /*AllowPCHWithCompilerErrors=*/true,
+  /*AllowASTWithCompilerErrors=*/true,
   /*UserFilesAreVolatile=*/true);
   *out_TU = MakeCXTranslationUnit(CXXIdx, std::move(AU));
   return *out_TU ? CXError_Success : CXError_Failure;
Index: clang/tools/c-index-test/core_main.cpp
===
--- clang/tools/c-index-test/core_main.cpp
+++ clang/tools/c-index-test/core_main.cpp
@@ -263,7 +263,7 @@
   std::string(modulePath), *pchRdr, ASTUnit::LoadASTOnly, Diags,
   FileSystemOpts, /*UseDebugInfo=*/false,
   /*OnlyLocalDecls=*/true, None, CaptureDiagsKind::None,
-  /*AllowPCHWithCompilerErrors=*/true,
+  /*AllowASTWithCompilerErrors=*/true,
   /*UserFilesAreVolatile=*/false);
   if (!AU) {
 errs() << "failed to create TU for: " << modulePath << '\n';
Index: clang/test/Modules/load-module-with-errors.m
===
--- /dev/null
+++ clang/test/Modules/load-module-with-errors.m
@@ -0,0 +1,14 @@
+// RUN: rm -rf %t
+
+// RUN: %clang_cc1 -fmodules -fallow-pcm-with-compiler-errors -fmodules-cache-path=%t -emit-module -x objective-c -fmodule-name=error %S/Inputs/module.map
+// RUN: %clang_cc1 -fmodules -fimplicit-module-maps -fallow-pcm-with-compiler-errors -ast-print -x objective-c -fmodules-cache-path=%t -I %S/Inputs %s | FileCheck -check-prefix=CHECK-PRINT %s
+
+@import error;
+
+void test(id x) {
+  [x method];
+}
+
+// CHECK-PRINT: @interface Error
+// CHECK-PRINT-NEXT: - (int)method;
+// CHECK-PRINT: void test(id x)
Index: clang/test/Modules/Inputs/module.map
===
--- clang/test/Modules/Inputs/module.map
+++ clang/test/Modules/Inputs/module.map
@@ -483,3 +483,4 @@
   header "template-nontrivial1.h"
   export *
 }
+module error { header "error.h" }
Index: clang/test/Modules/Inputs/error.h
===
--- /dev/null
+++ clang/test/Modules/Inputs/error.h
@@ -0,0 +1,8 @@
+@import undefined
+
+@interface Error
+- (int)method;
+undefined
+@end
+
+undefined
Index: clang/lib/Frontend/FrontendActions.cpp
===
--- clang/lib/Frontend/FrontendActions.cpp
+++ clang/lib/Frontend/FrontendActions.cpp
@@ -177,7 +177,8 @@
   Consumers.push_back(std::make_unique(
   CI.getPreprocessor(), CI.getModuleCache(), OutputFile, Sysroot, Buffer,
   CI.getFrontendOpts().ModuleFileExtensions,
-  /*AllowASTWithErrors=*/false,
+  /*AllowASTWithErrors=*/
+  +CI.getFrontendOpts().AllowPCMWithCompilerErrors,
   /*IncludeTimestamps=*/
   +CI.getFrontendOpts().BuildingImplicitModule,
   /*ShouldCacheASTInMemory=*/
@@ -187,6 +188,11 @@
   return std::make_unique(std::move(Consumers));
 }
 
+bool GenerateModuleAction::shouldEraseOutputFiles() {
+  return !getCompilerInstance().getFrontendOpts().AllowPCMWithCompilerErrors &&
+ ASTFrontendAction::shouldEraseOutputFiles();
+}
+
 bool GenerateModuleFromModuleMapAction::BeginSourceFileAction(
 CompilerInstance &CI) {
   if (!CI.getLangOpts().Modules) {
@@ -339,7 +345,7 @@
   CI.getPCHContainerReader(), CI.getFrontendOpts().ModuleFileExtensions,
   Sysroot.empty() ? "" : Sysroot.c_str(),
   /*DisableValidation*/ false,
-  /*AllowPCHWithCompilerErrors*/ false,
+  /*AllowASTWithCompilerErrors*/ false,
   /*AllowConfigurationMismatch*/ true,
   /*ValidateSystemInputs*/ true));
 
Index: clang/lib/Frontend/CompilerInvocation.cpp
===
--- clang/lib/Frontend/CompilerInvocation.cpp
+++ clang/lib/Frontend/CompilerInvocation.cpp
@@ -2027,6 +2027,7 @@
   Opts.Includ

[PATCH] D91580: [Frontend] Add flag to allow PCM generation despite compiler errors

[Ben Barham via Phabricator via cfe-commits]

bnbarham updated this revision to Diff 305914.
bnbarham added a comment.

Have allow-pcm also set allow-pch + test to make sure that works.


CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D91580/new/

https://reviews.llvm.org/D91580

Files:
  clang/include/clang/Driver/Options.td
  clang/include/clang/Frontend/ASTUnit.h
  clang/include/clang/Frontend/FrontendActions.h
  clang/include/clang/Frontend/FrontendOptions.h
  clang/lib/Frontend/ASTUnit.cpp
  clang/lib/Frontend/CompilerInstance.cpp
  clang/lib/Frontend/CompilerInvocation.cpp
  clang/lib/Frontend/FrontendActions.cpp
  clang/test/Modules/Inputs/error.h
  clang/test/Modules/Inputs/module.map
  clang/test/Modules/load-module-with-errors.m
  clang/tools/c-index-test/core_main.cpp
  clang/tools/libclang/CIndex.cpp

Index: clang/tools/libclang/CIndex.cpp
===
--- clang/tools/libclang/CIndex.cpp
+++ clang/tools/libclang/CIndex.cpp
@@ -3476,7 +3476,7 @@
   ast_filename, CXXIdx->getPCHContainerOperations()->getRawReader(),
   ASTUnit::LoadEverything, Diags, FileSystemOpts, /*UseDebugInfo=*/false,
   CXXIdx->getOnlyLocalDecls(), None, CaptureDiagsKind::All,
-  /*AllowPCHWithCompilerErrors=*/true,
+  /*AllowASTWithCompilerErrors=*/true,
   /*UserFilesAreVolatile=*/true);
   *out_TU = MakeCXTranslationUnit(CXXIdx, std::move(AU));
   return *out_TU ? CXError_Success : CXError_Failure;
Index: clang/tools/c-index-test/core_main.cpp
===
--- clang/tools/c-index-test/core_main.cpp
+++ clang/tools/c-index-test/core_main.cpp
@@ -263,7 +263,7 @@
   std::string(modulePath), *pchRdr, ASTUnit::LoadASTOnly, Diags,
   FileSystemOpts, /*UseDebugInfo=*/false,
   /*OnlyLocalDecls=*/true, None, CaptureDiagsKind::None,
-  /*AllowPCHWithCompilerErrors=*/true,
+  /*AllowASTWithCompilerErrors=*/true,
   /*UserFilesAreVolatile=*/false);
   if (!AU) {
 errs() << "failed to create TU for: " << modulePath << '\n';
Index: clang/test/Modules/load-module-with-errors.m
===
--- /dev/null
+++ clang/test/Modules/load-module-with-errors.m
@@ -0,0 +1,25 @@
+// RUN: rm -rf %t
+// RUN: mkdir %t
+
+// Write out a module with errors make sure it can be read
+// RUN: %clang_cc1 -fmodules -fallow-pcm-with-compiler-errors \
+// RUN:   -fmodules-cache-path=%t -x objective-c -emit-module \
+// RUN:   -fmodule-name=error %S/Inputs/module.map
+// RUN: %clang_cc1 -fmodules -fallow-pcm-with-compiler-errors \
+// RUN:   -fmodules-cache-path=%t -x objective-c -I %S/Inputs \
+// RUN:   -fimplicit-module-maps -ast-print %s | FileCheck %s
+
+// allow-pcm-with-compiler-errors should also allow errors in PCH
+// RUN: %clang_cc1 -fallow-pcm-with-compiler-errors -x c++ -emit-pch \
+// RUN:   -o %t/check.pch %S/Inputs/error.h
+
+@import error;
+
+void test(id x) {
+  [x method];
+}
+
+// CHECK: @interface Error
+// CHECK-NEXT: - (int)method;
+// CHECK-NEXT: @end
+// CHECK: void test(id x)
Index: clang/test/Modules/Inputs/module.map
===
--- clang/test/Modules/Inputs/module.map
+++ clang/test/Modules/Inputs/module.map
@@ -483,3 +483,4 @@
   header "template-nontrivial1.h"
   export *
 }
+module error { header "error.h" }
Index: clang/test/Modules/Inputs/error.h
===
--- /dev/null
+++ clang/test/Modules/Inputs/error.h
@@ -0,0 +1,8 @@
+@import undefined
+
+@interface Error
+- (int)method;
+undefined
+@end
+
+undefined
Index: clang/lib/Frontend/FrontendActions.cpp
===
--- clang/lib/Frontend/FrontendActions.cpp
+++ clang/lib/Frontend/FrontendActions.cpp
@@ -177,7 +177,8 @@
   Consumers.push_back(std::make_unique(
   CI.getPreprocessor(), CI.getModuleCache(), OutputFile, Sysroot, Buffer,
   CI.getFrontendOpts().ModuleFileExtensions,
-  /*AllowASTWithErrors=*/false,
+  /*AllowASTWithErrors=*/
+  +CI.getFrontendOpts().AllowPCMWithCompilerErrors,
   /*IncludeTimestamps=*/
   +CI.getFrontendOpts().BuildingImplicitModule,
   /*ShouldCacheASTInMemory=*/
@@ -187,6 +188,11 @@
   return std::make_unique(std::move(Consumers));
 }
 
+bool GenerateModuleAction::shouldEraseOutputFiles() {
+  return !getCompilerInstance().getFrontendOpts().AllowPCMWithCompilerErrors &&
+ ASTFrontendAction::shouldEraseOutputFiles();
+}
+
 bool GenerateModuleFromModuleMapAction::BeginSourceFileAction(
 CompilerInstance &CI) {
   if (!CI.getLangOpts().Modules) {
@@ -339,7 +345,7 @@
   CI.getPCHContainerReader(), CI.getFrontendOpts().ModuleFileExtensions,
   Sysroot.empty() ? "" : Sysroot.c_str(),
   /*DisableValidation*/ false,
-  /*AllowPCHWithCompilerErrors*/ false,
+  /*AllowASTWithCompilerErrors*/ false,
   /*AllowConfigurationMismatch*/ true,
   /*Va

[PATCH] D89024: [AST] Fix crashes caused by redeclarations in hidden prototypes

[Ben Barham via Phabricator via cfe-commits]

bnbarham created this revision.
bnbarham added a reviewer: akyrtzi.
Herald added subscribers: cfe-commits, arphaman, dexonsmith.
Herald added a project: clang.
bnbarham requested review of this revision.

ObjCContainerDecl.getMethod returns a nullptr by default when the
container is a hidden prototype. Callsites where the method is being
looked up on the redeclaration's own container should skip this check
since they (rightly) expect a valid method to be found.

Resolves rdar://69444243


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D89024

Files:
  clang/lib/AST/DeclObjC.cpp
  clang/test/Index/Inputs/hidden-redecls-sub.h
  clang/test/Index/Inputs/hidden-redecls.h
  clang/test/Index/Inputs/module.map
  clang/test/Index/hidden-redecls.m


Index: clang/test/Index/hidden-redecls.m
===
--- /dev/null
+++ clang/test/Index/hidden-redecls.m
@@ -0,0 +1,12 @@
+@import hidden_redecls;
+
+@interface Foo (Top)
+- (void)top_method;
+@end
+
+// p1_method in protocol P1 is hidden since module_redecls.sub hasn't been
+// imported yet. Check it is still indexed.
+
+// RUN: c-index-test -index-file-full %s -isystem %S/Inputs -fmodules | 
FileCheck %s
+// CHECK: [indexDeclaration]: kind: objc-instance-method | name: p1_method | 
{{.*}} | loc: {{.*}}/hidden-redecls-sub.h:2:9 | {{.*}} | isRedecl: 0
+// CHECK: [indexDeclaration]: kind: objc-instance-method | name: p1_method | 
{{.*}} | loc: {{.*}}/hidden-redecls-sub.h:3:9 | {{.*}} | isRedecl: 1
Index: clang/test/Index/Inputs/module.map
===
--- clang/test/Index/Inputs/module.map
+++ clang/test/Index/Inputs/module.map
@@ -20,3 +20,11 @@
 export *
   }
 }
+
+module hidden_redecls {
+  header "hidden-redecls.h"
+
+  explicit module sub {
+header "hidden-redecls-sub.h"
+  }
+}
Index: clang/test/Index/Inputs/hidden-redecls.h
===
--- /dev/null
+++ clang/test/Index/Inputs/hidden-redecls.h
@@ -0,0 +1,3 @@
+@interface Foo
+- (void)parent_method;
+@end
Index: clang/test/Index/Inputs/hidden-redecls-sub.h
===
--- /dev/null
+++ clang/test/Index/Inputs/hidden-redecls-sub.h
@@ -0,0 +1,7 @@
+@protocol P1
+- (void)p1_method;
+- (void)p1_method;
+@end
+
+@interface Foo (SubP1) 
+@end
Index: clang/lib/AST/DeclObjC.cpp
===
--- clang/lib/AST/DeclObjC.cpp
+++ clang/lib/AST/DeclObjC.cpp
@@ -950,7 +950,8 @@
   if (!Redecl && isRedeclaration()) {
 // This is the last redeclaration, go back to the first method.
 return cast(CtxD)->getMethod(getSelector(),
-isInstanceMethod());
+isInstanceMethod(),
+/*AllowHidden=*/true);
   }
 
   return Redecl ? Redecl : this;
@@ -983,7 +984,8 @@
   if (isRedeclaration()) {
 // It is possible that we have not done deserializing the ObjCMethod yet.
 ObjCMethodDecl *MD =
-cast(CtxD)->getMethod(Sel, isInstanceMethod());
+  cast(CtxD)->getMethod(Sel, isInstanceMethod(),
+   /*AllowHidden=*/true);
 return MD ? MD : this;
   }
 
@@ -1308,8 +1310,9 @@
   const ObjCMethodDecl *Method = this;
 
   if (Method->isRedeclaration()) {
-Method = cast(Method->getDeclContext())->
-   getMethod(Method->getSelector(), 
Method->isInstanceMethod());
+Method = cast(Method->getDeclContext())
+ ->getMethod(Method->getSelector(), Method->isInstanceMethod(),
+ /*AllowHidden=*/true);
   }
 
   if (Method->isOverriding()) {


Index: clang/test/Index/hidden-redecls.m
===
--- /dev/null
+++ clang/test/Index/hidden-redecls.m
@@ -0,0 +1,12 @@
+@import hidden_redecls;
+
+@interface Foo (Top)
+- (void)top_method;
+@end
+
+// p1_method in protocol P1 is hidden since module_redecls.sub hasn't been
+// imported yet. Check it is still indexed.
+
+// RUN: c-index-test -index-file-full %s -isystem %S/Inputs -fmodules | FileCheck %s
+// CHECK: [indexDeclaration]: kind: objc-instance-method | name: p1_method | {{.*}} | loc: {{.*}}/hidden-redecls-sub.h:2:9 | {{.*}} | isRedecl: 0
+// CHECK: [indexDeclaration]: kind: objc-instance-method | name: p1_method | {{.*}} | loc: {{.*}}/hidden-redecls-sub.h:3:9 | {{.*}} | isRedecl: 1
Index: clang/test/Index/Inputs/module.map
===
--- clang/test/Index/Inputs/module.map
+++ clang/test/Index/Inputs/module.map
@@ -20,3 +20,11 @@
 export *
   }
 }
+
+module hidden_redecls {
+  header "hidden-redecls.h"
+
+  explicit module sub {
+header "hidden-redecls-sub.h"
+  }
+}
Index: clang/test/Index/Inputs/hidden-redecls.

[PATCH] D89024: [AST] Fix crashes caused by redeclarations in hidden prototypes

[Ben Barham via Phabricator via cfe-commits]

bnbarham updated this revision to Diff 296859.
bnbarham added a comment.

Ran clang-format again


CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D89024/new/

https://reviews.llvm.org/D89024

Files:
  clang/lib/AST/DeclObjC.cpp
  clang/test/Index/Inputs/hidden-redecls-sub.h
  clang/test/Index/Inputs/hidden-redecls.h
  clang/test/Index/Inputs/module.map
  clang/test/Index/hidden-redecls.m


Index: clang/test/Index/hidden-redecls.m
===
--- /dev/null
+++ clang/test/Index/hidden-redecls.m
@@ -0,0 +1,12 @@
+@import hidden_redecls;
+
+@interface Foo (Top)
+- (void)top_method;
+@end
+
+// p1_method in protocol P1 is hidden since module_redecls.sub hasn't been
+// imported yet. Check it is still indexed.
+
+// RUN: c-index-test -index-file-full %s -isystem %S/Inputs -fmodules | 
FileCheck %s
+// CHECK: [indexDeclaration]: kind: objc-instance-method | name: p1_method | 
{{.*}} | loc: {{.*}}/hidden-redecls-sub.h:2:9 | {{.*}} | isRedecl: 0
+// CHECK: [indexDeclaration]: kind: objc-instance-method | name: p1_method | 
{{.*}} | loc: {{.*}}/hidden-redecls-sub.h:3:9 | {{.*}} | isRedecl: 1
Index: clang/test/Index/Inputs/module.map
===
--- clang/test/Index/Inputs/module.map
+++ clang/test/Index/Inputs/module.map
@@ -20,3 +20,11 @@
 export *
   }
 }
+
+module hidden_redecls {
+  header "hidden-redecls.h"
+
+  explicit module sub {
+header "hidden-redecls-sub.h"
+  }
+}
Index: clang/test/Index/Inputs/hidden-redecls.h
===
--- /dev/null
+++ clang/test/Index/Inputs/hidden-redecls.h
@@ -0,0 +1,3 @@
+@interface Foo
+- (void)parent_method;
+@end
Index: clang/test/Index/Inputs/hidden-redecls-sub.h
===
--- /dev/null
+++ clang/test/Index/Inputs/hidden-redecls-sub.h
@@ -0,0 +1,7 @@
+@protocol P1
+- (void)p1_method;
+- (void)p1_method;
+@end
+
+@interface Foo (SubP1) 
+@end
Index: clang/lib/AST/DeclObjC.cpp
===
--- clang/lib/AST/DeclObjC.cpp
+++ clang/lib/AST/DeclObjC.cpp
@@ -950,7 +950,8 @@
   if (!Redecl && isRedeclaration()) {
 // This is the last redeclaration, go back to the first method.
 return cast(CtxD)->getMethod(getSelector(),
-isInstanceMethod());
+isInstanceMethod(),
+/*AllowHidden=*/true);
   }
 
   return Redecl ? Redecl : this;
@@ -983,7 +984,8 @@
   if (isRedeclaration()) {
 // It is possible that we have not done deserializing the ObjCMethod yet.
 ObjCMethodDecl *MD =
-cast(CtxD)->getMethod(Sel, isInstanceMethod());
+cast(CtxD)->getMethod(Sel, isInstanceMethod(),
+ /*AllowHidden=*/true);
 return MD ? MD : this;
   }
 
@@ -1308,8 +1310,9 @@
   const ObjCMethodDecl *Method = this;
 
   if (Method->isRedeclaration()) {
-Method = cast(Method->getDeclContext())->
-   getMethod(Method->getSelector(), 
Method->isInstanceMethod());
+Method = cast(Method->getDeclContext())
+ ->getMethod(Method->getSelector(), Method->isInstanceMethod(),
+ /*AllowHidden=*/true);
   }
 
   if (Method->isOverriding()) {


Index: clang/test/Index/hidden-redecls.m
===
--- /dev/null
+++ clang/test/Index/hidden-redecls.m
@@ -0,0 +1,12 @@
+@import hidden_redecls;
+
+@interface Foo (Top)
+- (void)top_method;
+@end
+
+// p1_method in protocol P1 is hidden since module_redecls.sub hasn't been
+// imported yet. Check it is still indexed.
+
+// RUN: c-index-test -index-file-full %s -isystem %S/Inputs -fmodules | FileCheck %s
+// CHECK: [indexDeclaration]: kind: objc-instance-method | name: p1_method | {{.*}} | loc: {{.*}}/hidden-redecls-sub.h:2:9 | {{.*}} | isRedecl: 0
+// CHECK: [indexDeclaration]: kind: objc-instance-method | name: p1_method | {{.*}} | loc: {{.*}}/hidden-redecls-sub.h:3:9 | {{.*}} | isRedecl: 1
Index: clang/test/Index/Inputs/module.map
===
--- clang/test/Index/Inputs/module.map
+++ clang/test/Index/Inputs/module.map
@@ -20,3 +20,11 @@
 export *
   }
 }
+
+module hidden_redecls {
+  header "hidden-redecls.h"
+
+  explicit module sub {
+header "hidden-redecls-sub.h"
+  }
+}
Index: clang/test/Index/Inputs/hidden-redecls.h
===
--- /dev/null
+++ clang/test/Index/Inputs/hidden-redecls.h
@@ -0,0 +1,3 @@
+@interface Foo
+- (void)parent_method;
+@end
Index: clang/test/Index/Inputs/hidden-redecls-sub.h
===
--- /dev/null
+++ clang/test/Index/Inputs/hidden-rede

[PATCH] D89024: [AST] Fix crashes caused by redeclarations in hidden prototypes

[Ben Barham via Phabricator via cfe-commits]

bnbarham updated this revision to Diff 296861.
bnbarham added a comment.

Update test to include explicit OSX target triple


CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D89024/new/

https://reviews.llvm.org/D89024

Files:
  clang/lib/AST/DeclObjC.cpp
  clang/test/Index/Inputs/hidden-redecls-sub.h
  clang/test/Index/Inputs/hidden-redecls.h
  clang/test/Index/Inputs/module.map
  clang/test/Index/hidden-redecls.m


Index: clang/test/Index/hidden-redecls.m
===
--- /dev/null
+++ clang/test/Index/hidden-redecls.m
@@ -0,0 +1,12 @@
+@import hidden_redecls;
+
+@interface Foo (Top)
+- (void)top_method;
+@end
+
+// p1_method in protocol P1 is hidden since module_redecls.sub hasn't been
+// imported yet. Check it is still indexed.
+
+// RUN: c-index-test -index-file-full %s -isystem %S/Inputs -fmodules -target 
x86_64-apple-macosx10.7 | FileCheck %s
+// CHECK: [indexDeclaration]: kind: objc-instance-method | name: p1_method | 
{{.*}} | loc: {{.*}}/hidden-redecls-sub.h:2:9 | {{.*}} | isRedecl: 0
+// CHECK: [indexDeclaration]: kind: objc-instance-method | name: p1_method | 
{{.*}} | loc: {{.*}}/hidden-redecls-sub.h:3:9 | {{.*}} | isRedecl: 1
Index: clang/test/Index/Inputs/module.map
===
--- clang/test/Index/Inputs/module.map
+++ clang/test/Index/Inputs/module.map
@@ -20,3 +20,11 @@
 export *
   }
 }
+
+module hidden_redecls {
+  header "hidden-redecls.h"
+
+  explicit module sub {
+header "hidden-redecls-sub.h"
+  }
+}
Index: clang/test/Index/Inputs/hidden-redecls.h
===
--- /dev/null
+++ clang/test/Index/Inputs/hidden-redecls.h
@@ -0,0 +1,3 @@
+@interface Foo
+- (void)parent_method;
+@end
Index: clang/test/Index/Inputs/hidden-redecls-sub.h
===
--- /dev/null
+++ clang/test/Index/Inputs/hidden-redecls-sub.h
@@ -0,0 +1,7 @@
+@protocol P1
+- (void)p1_method;
+- (void)p1_method;
+@end
+
+@interface Foo (SubP1) 
+@end
Index: clang/lib/AST/DeclObjC.cpp
===
--- clang/lib/AST/DeclObjC.cpp
+++ clang/lib/AST/DeclObjC.cpp
@@ -950,7 +950,8 @@
   if (!Redecl && isRedeclaration()) {
 // This is the last redeclaration, go back to the first method.
 return cast(CtxD)->getMethod(getSelector(),
-isInstanceMethod());
+isInstanceMethod(),
+/*AllowHidden=*/true);
   }
 
   return Redecl ? Redecl : this;
@@ -983,7 +984,8 @@
   if (isRedeclaration()) {
 // It is possible that we have not done deserializing the ObjCMethod yet.
 ObjCMethodDecl *MD =
-cast(CtxD)->getMethod(Sel, isInstanceMethod());
+cast(CtxD)->getMethod(Sel, isInstanceMethod(),
+ /*AllowHidden=*/true);
 return MD ? MD : this;
   }
 
@@ -1308,8 +1310,9 @@
   const ObjCMethodDecl *Method = this;
 
   if (Method->isRedeclaration()) {
-Method = cast(Method->getDeclContext())->
-   getMethod(Method->getSelector(), 
Method->isInstanceMethod());
+Method = cast(Method->getDeclContext())
+ ->getMethod(Method->getSelector(), Method->isInstanceMethod(),
+ /*AllowHidden=*/true);
   }
 
   if (Method->isOverriding()) {


Index: clang/test/Index/hidden-redecls.m
===
--- /dev/null
+++ clang/test/Index/hidden-redecls.m
@@ -0,0 +1,12 @@
+@import hidden_redecls;
+
+@interface Foo (Top)
+- (void)top_method;
+@end
+
+// p1_method in protocol P1 is hidden since module_redecls.sub hasn't been
+// imported yet. Check it is still indexed.
+
+// RUN: c-index-test -index-file-full %s -isystem %S/Inputs -fmodules -target x86_64-apple-macosx10.7 | FileCheck %s
+// CHECK: [indexDeclaration]: kind: objc-instance-method | name: p1_method | {{.*}} | loc: {{.*}}/hidden-redecls-sub.h:2:9 | {{.*}} | isRedecl: 0
+// CHECK: [indexDeclaration]: kind: objc-instance-method | name: p1_method | {{.*}} | loc: {{.*}}/hidden-redecls-sub.h:3:9 | {{.*}} | isRedecl: 1
Index: clang/test/Index/Inputs/module.map
===
--- clang/test/Index/Inputs/module.map
+++ clang/test/Index/Inputs/module.map
@@ -20,3 +20,11 @@
 export *
   }
 }
+
+module hidden_redecls {
+  header "hidden-redecls.h"
+
+  explicit module sub {
+header "hidden-redecls-sub.h"
+  }
+}
Index: clang/test/Index/Inputs/hidden-redecls.h
===
--- /dev/null
+++ clang/test/Index/Inputs/hidden-redecls.h
@@ -0,0 +1,3 @@
+@interface Foo
+- (void)parent_method;
+@end
Index: clang/test/Index/Inputs/hidden-redecls-sub.h
==

[PATCH] D89024: [AST] Fix crashes caused by redeclarations in hidden prototypes

[Ben Barham via Phabricator via cfe-commits]

bnbarham updated this revision to Diff 297098.
bnbarham added a comment.

Removed platform-specific path separator


CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D89024/new/

https://reviews.llvm.org/D89024

Files:
  clang/lib/AST/DeclObjC.cpp
  clang/test/Index/Inputs/hidden-redecls-sub.h
  clang/test/Index/Inputs/hidden-redecls.h
  clang/test/Index/Inputs/module.map
  clang/test/Index/hidden-redecls.m


Index: clang/test/Index/hidden-redecls.m
===
--- /dev/null
+++ clang/test/Index/hidden-redecls.m
@@ -0,0 +1,12 @@
+@import hidden_redecls;
+
+@interface Foo (Top)
+- (void)top_method;
+@end
+
+// p1_method in protocol P1 is hidden since module_redecls.sub hasn't been
+// imported yet. Check it is still indexed.
+
+// RUN: c-index-test -index-file-full %s -isystem %S/Inputs -fmodules -target 
x86_64-apple-macosx10.7 | FileCheck %s
+// CHECK: [indexDeclaration]: kind: objc-instance-method | name: p1_method | 
{{.*}} | loc: {{.*}}hidden-redecls-sub.h:2:9 | {{.*}} | isRedecl: 0
+// CHECK: [indexDeclaration]: kind: objc-instance-method | name: p1_method | 
{{.*}} | loc: {{.*}}hidden-redecls-sub.h:3:9 | {{.*}} | isRedecl: 1
Index: clang/test/Index/Inputs/module.map
===
--- clang/test/Index/Inputs/module.map
+++ clang/test/Index/Inputs/module.map
@@ -20,3 +20,11 @@
 export *
   }
 }
+
+module hidden_redecls {
+  header "hidden-redecls.h"
+
+  explicit module sub {
+header "hidden-redecls-sub.h"
+  }
+}
Index: clang/test/Index/Inputs/hidden-redecls.h
===
--- /dev/null
+++ clang/test/Index/Inputs/hidden-redecls.h
@@ -0,0 +1,3 @@
+@interface Foo
+- (void)parent_method;
+@end
Index: clang/test/Index/Inputs/hidden-redecls-sub.h
===
--- /dev/null
+++ clang/test/Index/Inputs/hidden-redecls-sub.h
@@ -0,0 +1,7 @@
+@protocol P1
+- (void)p1_method;
+- (void)p1_method;
+@end
+
+@interface Foo (SubP1) 
+@end
Index: clang/lib/AST/DeclObjC.cpp
===
--- clang/lib/AST/DeclObjC.cpp
+++ clang/lib/AST/DeclObjC.cpp
@@ -950,7 +950,8 @@
   if (!Redecl && isRedeclaration()) {
 // This is the last redeclaration, go back to the first method.
 return cast(CtxD)->getMethod(getSelector(),
-isInstanceMethod());
+isInstanceMethod(),
+/*AllowHidden=*/true);
   }
 
   return Redecl ? Redecl : this;
@@ -983,7 +984,8 @@
   if (isRedeclaration()) {
 // It is possible that we have not done deserializing the ObjCMethod yet.
 ObjCMethodDecl *MD =
-cast(CtxD)->getMethod(Sel, isInstanceMethod());
+cast(CtxD)->getMethod(Sel, isInstanceMethod(),
+ /*AllowHidden=*/true);
 return MD ? MD : this;
   }
 
@@ -1308,8 +1310,9 @@
   const ObjCMethodDecl *Method = this;
 
   if (Method->isRedeclaration()) {
-Method = cast(Method->getDeclContext())->
-   getMethod(Method->getSelector(), 
Method->isInstanceMethod());
+Method = cast(Method->getDeclContext())
+ ->getMethod(Method->getSelector(), Method->isInstanceMethod(),
+ /*AllowHidden=*/true);
   }
 
   if (Method->isOverriding()) {


Index: clang/test/Index/hidden-redecls.m
===
--- /dev/null
+++ clang/test/Index/hidden-redecls.m
@@ -0,0 +1,12 @@
+@import hidden_redecls;
+
+@interface Foo (Top)
+- (void)top_method;
+@end
+
+// p1_method in protocol P1 is hidden since module_redecls.sub hasn't been
+// imported yet. Check it is still indexed.
+
+// RUN: c-index-test -index-file-full %s -isystem %S/Inputs -fmodules -target x86_64-apple-macosx10.7 | FileCheck %s
+// CHECK: [indexDeclaration]: kind: objc-instance-method | name: p1_method | {{.*}} | loc: {{.*}}hidden-redecls-sub.h:2:9 | {{.*}} | isRedecl: 0
+// CHECK: [indexDeclaration]: kind: objc-instance-method | name: p1_method | {{.*}} | loc: {{.*}}hidden-redecls-sub.h:3:9 | {{.*}} | isRedecl: 1
Index: clang/test/Index/Inputs/module.map
===
--- clang/test/Index/Inputs/module.map
+++ clang/test/Index/Inputs/module.map
@@ -20,3 +20,11 @@
 export *
   }
 }
+
+module hidden_redecls {
+  header "hidden-redecls.h"
+
+  explicit module sub {
+header "hidden-redecls-sub.h"
+  }
+}
Index: clang/test/Index/Inputs/hidden-redecls.h
===
--- /dev/null
+++ clang/test/Index/Inputs/hidden-redecls.h
@@ -0,0 +1,3 @@
+@interface Foo
+- (void)parent_method;
+@end
Index: clang/test/Index/Inputs/hidden-redecls-sub.h
===

[PATCH] D95159: [ASTReader] Allow controlling separately whether validation should be disabled for a PCH vs a module file

[Ben Barham via Phabricator via cfe-commits]

bnbarham accepted this revision.
bnbarham added a comment.
This revision is now accepted and ready to land.

LGTM, just the one minor comment




Comment at: clang/lib/Serialization/ASTReader.cpp:2221-
+  // validation for the PCH and the modules it loads.
+  ModuleKind K = CurrentDeserializingModuleKind.hasValue() ?
+ CurrentDeserializingModuleKind.getValue() : M.Kind;
+

There's a `getValueOr` method that you could use instead (ie. 
`CurrentDeserializingModuleKind.getValueOr(M.Kind)`).



Comment at: clang/test/Index/preamble-reparse-changed-module.m:8
+
+// RUN: env CINDEXTEST_EDITING=1 CINDEXTEST_EXECUTE_COMMAND="cp 
%S/Inputs/preamble-reparse-changed-module/new-head.h %t/mod/head.h" 
CINDEXTEST_EXECUTE_AFTER_TRIAL=1 \
+// RUN: c-index-test -test-load-source-reparse 3 local %s -I %t -I %t/mod 
-fmodules -fmodules-cache-path=%t/mcp 2>&1 | FileCheck %s

For some reason I was expecting the trial's to be 1 based, not 0. I'm not sure 
why though.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D95159/new/

https://reviews.llvm.org/D95159

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D95989: [ASTReader] Always rebuild a cached module that has errors

[Ben Barham via Phabricator via cfe-commits]

bnbarham created this revision.
bnbarham added a reviewer: akyrtzi.
bnbarham requested review of this revision.
Herald added a project: clang.
Herald added a subscriber: cfe-commits.

A module in the cache with an error should just be a cache miss. If
allowing errors (with -fallow-pcm-with-compiler-errors), a rebuild is
needed so that the appropriate diagnostics are output and in case search
paths have changed. If not allowing errors, the module was built
*allowing* errors and thus should be rebuilt regardless.


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D95989

Files:
  clang/lib/Serialization/ASTReader.cpp
  clang/test/Modules/Inputs/error.h
  clang/test/Modules/Inputs/error/error.h
  clang/test/Modules/Inputs/error/module.modulemap
  clang/test/Modules/Inputs/module.map
  clang/test/Modules/load-module-with-errors.m

Index: clang/test/Modules/load-module-with-errors.m
===
--- clang/test/Modules/load-module-with-errors.m
+++ clang/test/Modules/load-module-with-errors.m
@@ -1,25 +1,68 @@
 // RUN: rm -rf %t
 // RUN: mkdir %t
+// RUN: mkdir %t/prebuilt
 
-// Write out a module with errors make sure it can be read
 // RUN: %clang_cc1 -fmodules -fallow-pcm-with-compiler-errors \
-// RUN:   -fmodules-cache-path=%t -x objective-c -emit-module \
-// RUN:   -fmodule-name=error %S/Inputs/module.map
-// RUN: %clang_cc1 -fmodules -fallow-pcm-with-compiler-errors \
-// RUN:   -fmodules-cache-path=%t -x objective-c -I %S/Inputs \
-// RUN:   -fimplicit-module-maps -ast-print %s | FileCheck %s
+// RUN:   -fmodule-name=error -o %t/prebuilt/error.pcm \
+// RUN:   -x objective-c -emit-module %S/Inputs/error/module.modulemap
+
+// RUN: %clang_cc1 -fsyntax-only -fmodules -fallow-pcm-with-compiler-errors \
+// RUN:   -fprebuilt-module-path=%t/prebuilt -fmodules-cache-path=%t \
+// RUN:   -ast-print %s | FileCheck %s
+// RUN: %clang_cc1 -fsyntax-only -fmodules \
+// RUN:   -fprebuilt-module-path=%t/prebuilt -fmodules-cache-path=%t \
+// RUN:   -verify=pcherror %s
+
+// RUN: %clang_cc1 -fsyntax-only -fmodules -fallow-pcm-with-compiler-errors \
+// RUN:   -fmodule-file=error=%t/prebuilt/error.pcm -fmodules-cache-path=%t \
+// RUN:   -ast-print %s | FileCheck %s
+// RUN: %clang_cc1 -fsyntax-only -fmodules \
+// RUN:   -fmodule-file=error=%t/prebuilt/error.pcm -fmodules-cache-path=%t \
+// RUN:   -verify=pcherror %s
+
+// RUN: %clang_cc1 -fsyntax-only -fmodules -fallow-pcm-with-compiler-errors \
+// RUN:   -fmodule-file=%t/prebuilt/error.pcm -fmodules-cache-path=%t \
+// RUN:   -ast-print %s | FileCheck %s
+// RUN: not %clang_cc1 -fsyntax-only -fmodules \
+// RUN:   -fmodule-file=%t/prebuilt/error.pcm -fmodules-cache-path=%t \
+// RUN:   -verify=pcherror %s
+
+// Shouldn't build the cached module (that has errors) when not allowing errors
+// RUN: not %clang_cc1 -fsyntax-only -fmodules \
+// RUN:   -fmodules-cache-path=%t -fimplicit-module-maps -I %S/Inputs/error \
+// RUN:   -x objective-c %s
+// RUN: find %t -name "error-*.pcm" | not grep error
+
+// Should build the cached module when allowing errors
+// RUN: %clang_cc1 -fsyntax-only -fmodules -fallow-pcm-with-compiler-errors \
+// RUN:   -fmodules-cache-path=%t -fimplicit-module-maps -I %S/Inputs/error \
+// RUN:   -x objective-c -verify %s
+// RUN: find %t -name "error-*.pcm" | grep error
+
+// Make sure there is still an error after the module is already in the cache
+// RUN: %clang_cc1 -fsyntax-only -fmodules -fallow-pcm-with-compiler-errors \
+// RUN:   -fmodules-cache-path=%t -fimplicit-module-maps -I %S/Inputs/error \
+// RUN:   -x objective-c -verify %s
+
+// Should rebuild the cached module if it had an error (if it wasn't rebuilt
+// the verify would fail as it would be the PCH error instead)
+// RUN: %clang_cc1 -fsyntax-only -fmodules \
+// RUN:   -fmodules-cache-path=%t -fimplicit-module-maps -I %S/Inputs/error \
+// RUN:   -x objective-c -verify %s
 
 // allow-pcm-with-compiler-errors should also allow errors in PCH
-// RUN: %clang_cc1 -fallow-pcm-with-compiler-errors -x c++ -emit-pch \
-// RUN:   -o %t/check.pch %S/Inputs/error.h
+// RUN: %clang_cc1 -fallow-pcm-with-compiler-errors -x objective-c \
+// RUN:   -o %t/check.pch -emit-pch %S/Inputs/error/error.h
 
-@import error;
+// pcherror-error@* {{PCH file contains compiler errors}}
+@import error; // expected-error {{could not build module 'error'}}
 
-void test(id x) {
+void test(Error *x) {
   [x method];
 }
 
 // CHECK: @interface Error
 // CHECK-NEXT: - (int)method;
+// CHECK-NEXT: - (id)method2;
 // CHECK-NEXT: @end
-// CHECK: void test(id x)
+// CHECK: void test(Error *x)
Index: clang/test/Modules/Inputs/module.map
===
--- clang/test/Modules/Inputs/module.map
+++ clang/test/Modules/Inputs/module.map
@@ -483,4 +483,3 @@
   header "template-nontrivial1.h"
   export *
 }
-module error { header "error.h" }
Index: clang/test/Modules/Inputs/error/module.modulemap
=

[PATCH] D95989: [ASTReader] Always rebuild a cached module that has errors

[Ben Barham via Phabricator via cfe-commits]

bnbarham updated this revision to Diff 321305.
bnbarham added a comment.

Added a couple semi-colons to the test file


CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D95989/new/

https://reviews.llvm.org/D95989

Files:
  clang/lib/Serialization/ASTReader.cpp
  clang/test/Modules/Inputs/error.h
  clang/test/Modules/Inputs/error/error.h
  clang/test/Modules/Inputs/error/module.modulemap
  clang/test/Modules/Inputs/module.map
  clang/test/Modules/load-module-with-errors.m

Index: clang/test/Modules/load-module-with-errors.m
===
--- clang/test/Modules/load-module-with-errors.m
+++ clang/test/Modules/load-module-with-errors.m
@@ -1,25 +1,68 @@
 // RUN: rm -rf %t
 // RUN: mkdir %t
+// RUN: mkdir %t/prebuilt
 
-// Write out a module with errors make sure it can be read
 // RUN: %clang_cc1 -fmodules -fallow-pcm-with-compiler-errors \
-// RUN:   -fmodules-cache-path=%t -x objective-c -emit-module \
-// RUN:   -fmodule-name=error %S/Inputs/module.map
-// RUN: %clang_cc1 -fmodules -fallow-pcm-with-compiler-errors \
-// RUN:   -fmodules-cache-path=%t -x objective-c -I %S/Inputs \
-// RUN:   -fimplicit-module-maps -ast-print %s | FileCheck %s
+// RUN:   -fmodule-name=error -o %t/prebuilt/error.pcm \
+// RUN:   -x objective-c -emit-module %S/Inputs/error/module.modulemap
+
+// RUN: %clang_cc1 -fsyntax-only -fmodules -fallow-pcm-with-compiler-errors \
+// RUN:   -fprebuilt-module-path=%t/prebuilt -fmodules-cache-path=%t \
+// RUN:   -ast-print %s | FileCheck %s
+// RUN: %clang_cc1 -fsyntax-only -fmodules \
+// RUN:   -fprebuilt-module-path=%t/prebuilt -fmodules-cache-path=%t \
+// RUN:   -verify=pcherror %s
+
+// RUN: %clang_cc1 -fsyntax-only -fmodules -fallow-pcm-with-compiler-errors \
+// RUN:   -fmodule-file=error=%t/prebuilt/error.pcm -fmodules-cache-path=%t \
+// RUN:   -ast-print %s | FileCheck %s
+// RUN: %clang_cc1 -fsyntax-only -fmodules \
+// RUN:   -fmodule-file=error=%t/prebuilt/error.pcm -fmodules-cache-path=%t \
+// RUN:   -verify=pcherror %s
+
+// RUN: %clang_cc1 -fsyntax-only -fmodules -fallow-pcm-with-compiler-errors \
+// RUN:   -fmodule-file=%t/prebuilt/error.pcm -fmodules-cache-path=%t \
+// RUN:   -ast-print %s | FileCheck %s
+// RUN: not %clang_cc1 -fsyntax-only -fmodules \
+// RUN:   -fmodule-file=%t/prebuilt/error.pcm -fmodules-cache-path=%t \
+// RUN:   -verify=pcherror %s
+
+// Shouldn't build the cached module (that has errors) when not allowing errors
+// RUN: not %clang_cc1 -fsyntax-only -fmodules \
+// RUN:   -fmodules-cache-path=%t -fimplicit-module-maps -I %S/Inputs/error \
+// RUN:   -x objective-c %s
+// RUN: find %t -name "error-*.pcm" | not grep error
+
+// Should build the cached module when allowing errors
+// RUN: %clang_cc1 -fsyntax-only -fmodules -fallow-pcm-with-compiler-errors \
+// RUN:   -fmodules-cache-path=%t -fimplicit-module-maps -I %S/Inputs/error \
+// RUN:   -x objective-c -verify %s
+// RUN: find %t -name "error-*.pcm" | grep error
+
+// Make sure there is still an error after the module is already in the cache
+// RUN: %clang_cc1 -fsyntax-only -fmodules -fallow-pcm-with-compiler-errors \
+// RUN:   -fmodules-cache-path=%t -fimplicit-module-maps -I %S/Inputs/error \
+// RUN:   -x objective-c -verify %s
+
+// Should rebuild the cached module if it had an error (if it wasn't rebuilt
+// the verify would fail as it would be the PCH error instead)
+// RUN: %clang_cc1 -fsyntax-only -fmodules \
+// RUN:   -fmodules-cache-path=%t -fimplicit-module-maps -I %S/Inputs/error \
+// RUN:   -x objective-c -verify %s
 
 // allow-pcm-with-compiler-errors should also allow errors in PCH
-// RUN: %clang_cc1 -fallow-pcm-with-compiler-errors -x c++ -emit-pch \
-// RUN:   -o %t/check.pch %S/Inputs/error.h
+// RUN: %clang_cc1 -fallow-pcm-with-compiler-errors -x objective-c \
+// RUN:   -o %t/check.pch -emit-pch %S/Inputs/error/error.h
 
-@import error;
+// pcherror-error@* {{PCH file contains compiler errors}}
+@import error; // expected-error {{could not build module 'error'}}
 
-void test(id x) {
+void test(Error *x) {
   [x method];
 }
 
 // CHECK: @interface Error
 // CHECK-NEXT: - (int)method;
+// CHECK-NEXT: - (id)method2;
 // CHECK-NEXT: @end
-// CHECK: void test(id x)
+// CHECK: void test(Error *x)
Index: clang/test/Modules/Inputs/module.map
===
--- clang/test/Modules/Inputs/module.map
+++ clang/test/Modules/Inputs/module.map
@@ -483,4 +483,3 @@
   header "template-nontrivial1.h"
   export *
 }
-module error { header "error.h" }
Index: clang/test/Modules/Inputs/error/module.modulemap
===
--- /dev/null
+++ clang/test/Modules/Inputs/error/module.modulemap
@@ -0,0 +1,3 @@
+module error {
+  header "error.h"
+}
Index: clang/test/Modules/Inputs/error/error.h
===
--- /dev/null
+++ clang/test/Modules/Inputs/error/error.h
@@ -0,0 +1

[PATCH] D95989: [ASTReader] Always rebuild a cached module that has errors

[Ben Barham via Phabricator via cfe-commits]

bnbarham updated this revision to Diff 321308.
bnbarham set the repository for this revision to rG LLVM Github Monorepo.

Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D95989/new/

https://reviews.llvm.org/D95989

Files:
  clang/lib/Serialization/ASTReader.cpp
  clang/test/Modules/Inputs/error.h
  clang/test/Modules/Inputs/error/error.h
  clang/test/Modules/Inputs/error/module.modulemap
  clang/test/Modules/Inputs/module.map
  clang/test/Modules/load-module-with-errors.m

Index: clang/test/Modules/load-module-with-errors.m
===
--- clang/test/Modules/load-module-with-errors.m
+++ clang/test/Modules/load-module-with-errors.m
@@ -1,25 +1,68 @@
 // RUN: rm -rf %t
 // RUN: mkdir %t
+// RUN: mkdir %t/prebuilt
 
-// Write out a module with errors make sure it can be read
 // RUN: %clang_cc1 -fmodules -fallow-pcm-with-compiler-errors \
-// RUN:   -fmodules-cache-path=%t -x objective-c -emit-module \
-// RUN:   -fmodule-name=error %S/Inputs/module.map
-// RUN: %clang_cc1 -fmodules -fallow-pcm-with-compiler-errors \
-// RUN:   -fmodules-cache-path=%t -x objective-c -I %S/Inputs \
-// RUN:   -fimplicit-module-maps -ast-print %s | FileCheck %s
+// RUN:   -fmodule-name=error -o %t/prebuilt/error.pcm \
+// RUN:   -x objective-c -emit-module %S/Inputs/error/module.modulemap
+
+// RUN: %clang_cc1 -fsyntax-only -fmodules -fallow-pcm-with-compiler-errors \
+// RUN:   -fprebuilt-module-path=%t/prebuilt -fmodules-cache-path=%t \
+// RUN:   -ast-print %s | FileCheck %s
+// RUN: %clang_cc1 -fsyntax-only -fmodules \
+// RUN:   -fprebuilt-module-path=%t/prebuilt -fmodules-cache-path=%t \
+// RUN:   -verify=pcherror %s
+
+// RUN: %clang_cc1 -fsyntax-only -fmodules -fallow-pcm-with-compiler-errors \
+// RUN:   -fmodule-file=error=%t/prebuilt/error.pcm -fmodules-cache-path=%t \
+// RUN:   -ast-print %s | FileCheck %s
+// RUN: %clang_cc1 -fsyntax-only -fmodules \
+// RUN:   -fmodule-file=error=%t/prebuilt/error.pcm -fmodules-cache-path=%t \
+// RUN:   -verify=pcherror %s
+
+// RUN: %clang_cc1 -fsyntax-only -fmodules -fallow-pcm-with-compiler-errors \
+// RUN:   -fmodule-file=%t/prebuilt/error.pcm -fmodules-cache-path=%t \
+// RUN:   -ast-print %s | FileCheck %s
+// RUN: not %clang_cc1 -fsyntax-only -fmodules \
+// RUN:   -fmodule-file=%t/prebuilt/error.pcm -fmodules-cache-path=%t \
+// RUN:   -verify=pcherror %s
+
+// Shouldn't build the cached module (that has errors) when not allowing errors
+// RUN: not %clang_cc1 -fsyntax-only -fmodules \
+// RUN:   -fmodules-cache-path=%t -fimplicit-module-maps -I %S/Inputs/error \
+// RUN:   -x objective-c %s
+// RUN: find %t -name "error-*.pcm" | not grep error
+
+// Should build the cached module when allowing errors
+// RUN: %clang_cc1 -fsyntax-only -fmodules -fallow-pcm-with-compiler-errors \
+// RUN:   -fmodules-cache-path=%t -fimplicit-module-maps -I %S/Inputs/error \
+// RUN:   -x objective-c -verify %s
+// RUN: find %t -name "error-*.pcm" | grep error
+
+// Make sure there is still an error after the module is already in the cache
+// RUN: %clang_cc1 -fsyntax-only -fmodules -fallow-pcm-with-compiler-errors \
+// RUN:   -fmodules-cache-path=%t -fimplicit-module-maps -I %S/Inputs/error \
+// RUN:   -x objective-c -verify %s
+
+// Should rebuild the cached module if it had an error (if it wasn't rebuilt
+// the verify would fail as it would be the PCH error instead)
+// RUN: %clang_cc1 -fsyntax-only -fmodules \
+// RUN:   -fmodules-cache-path=%t -fimplicit-module-maps -I %S/Inputs/error \
+// RUN:   -x objective-c -verify %s
 
 // allow-pcm-with-compiler-errors should also allow errors in PCH
-// RUN: %clang_cc1 -fallow-pcm-with-compiler-errors -x c++ -emit-pch \
-// RUN:   -o %t/check.pch %S/Inputs/error.h
+// RUN: %clang_cc1 -fallow-pcm-with-compiler-errors -x objective-c \
+// RUN:   -o %t/check.pch -emit-pch %S/Inputs/error/error.h
 
-@import error;
+// pcherror-error@* {{PCH file contains compiler errors}}
+@import error; // expected-error {{could not build module 'error'}}
 
-void test(id x) {
+void test(Error *x) {
   [x method];
 }
 
 // CHECK: @interface Error
 // CHECK-NEXT: - (int)method;
+// CHECK-NEXT: - (id)method2;
 // CHECK-NEXT: @end
-// CHECK: void test(id x)
+// CHECK: void test(Error *x)
Index: clang/test/Modules/Inputs/module.map
===
--- clang/test/Modules/Inputs/module.map
+++ clang/test/Modules/Inputs/module.map
@@ -483,4 +483,3 @@
   header "template-nontrivial1.h"
   export *
 }
-module error { header "error.h" }
Index: clang/test/Modules/Inputs/error/module.modulemap
===
--- /dev/null
+++ clang/test/Modules/Inputs/error/module.modulemap
@@ -0,0 +1,3 @@
+module error {
+  header "error.h"
+}
Index: clang/test/Modules/Inputs/error/error.h
===
--- /dev/null
+++ clang/tes

[PATCH] D96246: Make sure a module file with errors produced via '-fallow-pcm-with-compiler-errors' can be loaded when using implicit modules

[Ben Barham via Phabricator via cfe-commits]

bnbarham accepted this revision.
bnbarham added a comment.
This revision is now accepted and ready to land.

LGTM


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D96246/new/

https://reviews.llvm.org/D96246

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D107690: [Modules] Do not remove failed modules after the control block phase

[Ben Barham via Phabricator via cfe-commits]

bnbarham created this revision.
bnbarham added reviewers: vsapsai, dexonsmith.
Herald added a subscriber: kristof.beyls.
bnbarham requested review of this revision.
Herald added a project: clang.
Herald added a subscriber: cfe-commits.

Reading modules first reads each control block in the chain and then all
AST blocks.

The first phase is intended to find recoverable errors, eg. an out of
date or missing module. If any error occurs during this phase, it is
safe to remove all modules in the chain as no references to them will
exist.

While reading the AST blocks, however, various fields in ASTReader are
updated with references to the module. Removing modules at this point
can cause dangling pointers which can be accessed later. These would be
otherwise harmless, eg. a binary search over `GlobalSLocEntryMap` may
access a failed module that could error, but shouldn't crash.  Do not
remove modules in this phase, regardless of failures.

Since this is the case, it also doesn't make sense to return OutOfDate
during this phase, so remove the two cases where this happens.

When they were originally added these checks would return OutOfDate when
the serialized and current path didn't match up. This is not the same as
the existing check, which is now different since only the name as
written is serialized. This means the OutOfDate condition is really only
checking for an umbrella with a different *name* (assuming it exists in
the base directory). It is also skipped completely for frameworks since
these don't include `Headers/` in the name, causing no file entry to be
found.

Given all this, it seems safe to ignore this case entirely for now.
This makes the handling of an umbrella header/directory the same as
regular headers, which also don't check for differences in the path
caused by VFS.

Resolves rdar://79329355


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D107690

Files:
  clang/lib/Serialization/ASTReader.cpp
  clang/test/VFS/module-header-mismatches.m
  clang/test/VFS/umbrella-mismatch.m

Index: clang/test/VFS/umbrella-mismatch.m
===
--- clang/test/VFS/umbrella-mismatch.m
+++ /dev/null
@@ -1,7 +0,0 @@
-// RUN: rm -rf %t
-// RUN: sed -e "s;INPUT_DIR;%/S/Inputs;g" -e "s;OUT_DIR;%/S/Inputs;g" %S/Inputs/vfsoverlay.yaml > %t.yaml
-
-// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t -ivfsoverlay %t.yaml -F %S/Inputs -fsyntax-only %s -Wno-atimport-in-framework-header -verify
-// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t -F %S/Inputs -fsyntax-only %s -Wno-atimport-in-framework-header -verify
-// expected-no-diagnostics
-@import UsesFoo;
Index: clang/test/VFS/module-header-mismatches.m
===
--- /dev/null
+++ clang/test/VFS/module-header-mismatches.m
@@ -0,0 +1,86 @@
+// RUN: rm -rf %t
+// RUN: split-file %s %t
+// RUN: sed -e "s;TEST_DIR;%/t;g" %t/sed-overlay.yaml > %t/overlay.yaml
+
+// These tests first build with an overlay such that the header is resolved
+// to %t/other/Mismatch.h. They then build again with the header resolved
+// to the one in their directory.
+//
+// This should cause a rebuild if the contents is different (and thus multiple
+// PCMs), but this currently isn't the case. We should at least not error,
+// since this does happen in real projects (with a different copy of the same
+// file).
+
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/hf-mcp -ivfsoverlay %t/overlay.yaml -F %t/header-frameworks -fsyntax-only -verify %t/use.m
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/hf-mcp -F %t/header-frameworks -fsyntax-only -verify %t/use.m
+// RUN: find %t/hf-mcp -name "Mismatch-*.pcm" | count 1
+
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/df-mcp -ivfsoverlay %t/overlay.yaml -F %t/dir-frameworks -fsyntax-only -verify %t/use.m
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/hf-mcp -F %t/dir-frameworks -fsyntax-only -verify %t/use.m
+// RUN: find %t/df-mcp -name "Mismatch-*.pcm" | count 1
+
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/nf-mcp -ivfsoverlay %t/overlay.yaml -F %t/norm-frameworks -fsyntax-only -verify %t/use.m
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/nf-mcp -F %t/norm-frameworks -fsyntax-only -verify %t/use.m
+// RUN: find %t/nf-mcp -name "Mismatch-*.pcm" | count 1
+
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/m-mcp -ivfsoverlay %t/overlay.yaml -I %t/mod -fsyntax-only -verify %t/use.m
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/m-mcp -I %t/mod -fsyntax-only -verify %t/use.m
+// RUN: find %t/m-mcp -name "Mismatch-*.pcm" | count 1
+
+//--- use.m
+// expected-no-d

[PATCH] D107690: [Modules] Do not remove failed modules after the control block phase

[Ben Barham via Phabricator via cfe-commits]

bnbarham added a comment.

I have another change to update the post-control-block functions to llvm::Error 
instead as well to hopefully make the distinction more clear as well. Let me 
know if you think it belongs in this PR, otherwise I'll open another once this 
one is in.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D107690/new/

https://reviews.llvm.org/D107690

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D107690: [Modules] Do not remove failed modules after the control block phase

[Ben Barham via Phabricator via cfe-commits]

bnbarham added inline comments.



Comment at: clang/test/VFS/umbrella-mismatch.m:4
-
-// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps 
-fmodules-cache-path=%t -ivfsoverlay %t.yaml -F %S/Inputs -fsyntax-only %s 
-Wno-atimport-in-framework-header -verify
-// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps 
-fmodules-cache-path=%t -F %S/Inputs -fsyntax-only %s 
-Wno-atimport-in-framework-header -verify

vsapsai wrote:
> Are you deleting this test case because it is subsumed by 
> module-header-mismatches.m?
> 
> Also if it makes sense to delete this test, need to do more cleanup because 
> UsesFoo.framework and Foo.framework seem to be used only from this test. 
> Though I'm not 100% sure.
Yep, the check in this one is the same as the `header-frameworks`.

`Foo.framework` is used by `subframework-symlink.m`. But `UsesFoo.framework` 
isn't used by anything else, thanks.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D107690/new/

https://reviews.llvm.org/D107690

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D107690: [Modules] Do not remove failed modules after the control block phase

[Ben Barham via Phabricator via cfe-commits]

bnbarham updated this revision to Diff 365619.
bnbarham added a comment.

Removed the now unused UsesFoo.framework in the tests


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D107690/new/

https://reviews.llvm.org/D107690

Files:
  clang/lib/Serialization/ASTReader.cpp
  clang/test/VFS/Inputs/UsesFoo.framework/Headers/UsesFoo.h
  clang/test/VFS/Inputs/UsesFoo.framework/Modules/module.modulemap
  clang/test/VFS/module-header-mismatches.m
  clang/test/VFS/umbrella-mismatch.m

Index: clang/test/VFS/umbrella-mismatch.m
===
--- clang/test/VFS/umbrella-mismatch.m
+++ /dev/null
@@ -1,7 +0,0 @@
-// RUN: rm -rf %t
-// RUN: sed -e "s;INPUT_DIR;%/S/Inputs;g" -e "s;OUT_DIR;%/S/Inputs;g" %S/Inputs/vfsoverlay.yaml > %t.yaml
-
-// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t -ivfsoverlay %t.yaml -F %S/Inputs -fsyntax-only %s -Wno-atimport-in-framework-header -verify
-// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t -F %S/Inputs -fsyntax-only %s -Wno-atimport-in-framework-header -verify
-// expected-no-diagnostics
-@import UsesFoo;
Index: clang/test/VFS/module-header-mismatches.m
===
--- /dev/null
+++ clang/test/VFS/module-header-mismatches.m
@@ -0,0 +1,86 @@
+// RUN: rm -rf %t
+// RUN: split-file %s %t
+// RUN: sed -e "s;TEST_DIR;%/t;g" %t/sed-overlay.yaml > %t/overlay.yaml
+
+// These tests first build with an overlay such that the header is resolved
+// to %t/other/Mismatch.h. They then build again with the header resolved
+// to the one in their directory.
+//
+// This should cause a rebuild if the contents is different (and thus multiple
+// PCMs), but this currently isn't the case. We should at least not error,
+// since this does happen in real projects (with a different copy of the same
+// file).
+
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/hf-mcp -ivfsoverlay %t/overlay.yaml -F %t/header-frameworks -fsyntax-only -verify %t/use.m
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/hf-mcp -F %t/header-frameworks -fsyntax-only -verify %t/use.m
+// RUN: find %t/hf-mcp -name "Mismatch-*.pcm" | count 1
+
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/df-mcp -ivfsoverlay %t/overlay.yaml -F %t/dir-frameworks -fsyntax-only -verify %t/use.m
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/hf-mcp -F %t/dir-frameworks -fsyntax-only -verify %t/use.m
+// RUN: find %t/df-mcp -name "Mismatch-*.pcm" | count 1
+
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/nf-mcp -ivfsoverlay %t/overlay.yaml -F %t/norm-frameworks -fsyntax-only -verify %t/use.m
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/nf-mcp -F %t/norm-frameworks -fsyntax-only -verify %t/use.m
+// RUN: find %t/nf-mcp -name "Mismatch-*.pcm" | count 1
+
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/m-mcp -ivfsoverlay %t/overlay.yaml -I %t/mod -fsyntax-only -verify %t/use.m
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/m-mcp -I %t/mod -fsyntax-only -verify %t/use.m
+// RUN: find %t/m-mcp -name "Mismatch-*.pcm" | count 1
+
+//--- use.m
+// expected-no-diagnostics
+@import Mismatch;
+
+//--- header-frameworks/Mismatch.framework/Modules/module.modulemap
+framework module Mismatch {
+  umbrella header "Mismatch.h"
+}
+//--- header-frameworks/Mismatch.framework/Headers/Mismatch.h
+
+//--- dir-frameworks/Mismatch.framework/Modules/module.modulemap
+framework module Mismatch {
+  umbrella "someheaders"
+}
+//--- dir-frameworks/Mismatch.framework/someheaders/Mismatch.h
+
+//--- norm-frameworks/Mismatch.framework/Modules/module.modulemap
+framework module Mismatch {
+  header "Mismatch.h"
+}
+//--- norm-frameworks/Mismatch.framework/Headers/Mismatch.h
+
+//--- mod/module.modulemap
+module Mismatch {
+  umbrella header "Mismatch.h"
+}
+//--- mod/Mismatch.h
+
+//--- other/Mismatch.h
+
+//--- sed-overlay.yaml
+{
+  'version': 0,
+  'roots': [
+{ 'name': 'TEST_DIR', 'type': 'directory',
+  'contents': [
+{ 'name': 'header-frameworks/Mismatch.framework/Headers/Mismatch.h',
+  'type': 'file',
+  'external-contents': 'TEST_DIR/other/Mismatch.h'
+},
+{ 'name': 'dir-frameworks/Mismatch.framework/someheaders',
+  'type': 'directory',
+  'external-contents': 'TEST_DIR/others'
+},
+{ 'name': 'norm-frameworks/Mismatch.framework/Headers/Mismatch.h',
+  'type': 'file',
+  'external-contents': 'TEST_DIR/other/Mismatch.h'
+},
+{ 'name': 'mod/Mismatch.h',
+  'type': 'file',
+  'external-contents': 'TEST_DIR/other/Mismatch.h'
+}
+ 

[PATCH] D107690: [Modules] Do not remove failed modules after the control block phase

[Ben Barham via Phabricator via cfe-commits]

bnbarham updated this revision to Diff 366152.
bnbarham marked an inline comment as done.
bnbarham edited the summary of this revision.
bnbarham added a comment.

Changed to keep setting the umbrella header/directory with a FIXME that it 
currently doesn't work for framework modules.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D107690/new/

https://reviews.llvm.org/D107690

Files:
  clang/lib/Serialization/ASTReader.cpp
  clang/test/VFS/Inputs/UsesFoo.framework/Headers/UsesFoo.h
  clang/test/VFS/Inputs/UsesFoo.framework/Modules/module.modulemap
  clang/test/VFS/module-header-mismatches.m
  clang/test/VFS/umbrella-mismatch.m

Index: clang/test/VFS/umbrella-mismatch.m
===
--- clang/test/VFS/umbrella-mismatch.m
+++ /dev/null
@@ -1,7 +0,0 @@
-// RUN: rm -rf %t
-// RUN: sed -e "s;INPUT_DIR;%/S/Inputs;g" -e "s;OUT_DIR;%/S/Inputs;g" %S/Inputs/vfsoverlay.yaml > %t.yaml
-
-// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t -ivfsoverlay %t.yaml -F %S/Inputs -fsyntax-only %s -Wno-atimport-in-framework-header -verify
-// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t -F %S/Inputs -fsyntax-only %s -Wno-atimport-in-framework-header -verify
-// expected-no-diagnostics
-@import UsesFoo;
Index: clang/test/VFS/module-header-mismatches.m
===
--- /dev/null
+++ clang/test/VFS/module-header-mismatches.m
@@ -0,0 +1,86 @@
+// RUN: rm -rf %t
+// RUN: split-file %s %t
+// RUN: sed -e "s;TEST_DIR;%/t;g" %t/sed-overlay.yaml > %t/overlay.yaml
+
+// These tests first build with an overlay such that the header is resolved
+// to %t/other/Mismatch.h. They then build again with the header resolved
+// to the one in their directory.
+//
+// This should cause a rebuild if the contents is different (and thus multiple
+// PCMs), but this currently isn't the case. We should at least not error,
+// since this does happen in real projects (with a different copy of the same
+// file).
+
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/hf-mcp -ivfsoverlay %t/overlay.yaml -F %t/header-frameworks -fsyntax-only -verify %t/use.m
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/hf-mcp -F %t/header-frameworks -fsyntax-only -verify %t/use.m
+// RUN: find %t/hf-mcp -name "Mismatch-*.pcm" | count 1
+
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/df-mcp -ivfsoverlay %t/overlay.yaml -F %t/dir-frameworks -fsyntax-only -verify %t/use.m
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/hf-mcp -F %t/dir-frameworks -fsyntax-only -verify %t/use.m
+// RUN: find %t/df-mcp -name "Mismatch-*.pcm" | count 1
+
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/nf-mcp -ivfsoverlay %t/overlay.yaml -F %t/norm-frameworks -fsyntax-only -verify %t/use.m
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/nf-mcp -F %t/norm-frameworks -fsyntax-only -verify %t/use.m
+// RUN: find %t/nf-mcp -name "Mismatch-*.pcm" | count 1
+
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/m-mcp -ivfsoverlay %t/overlay.yaml -I %t/mod -fsyntax-only -verify %t/use.m
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/m-mcp -I %t/mod -fsyntax-only -verify %t/use.m
+// RUN: find %t/m-mcp -name "Mismatch-*.pcm" | count 1
+
+//--- use.m
+// expected-no-diagnostics
+@import Mismatch;
+
+//--- header-frameworks/Mismatch.framework/Modules/module.modulemap
+framework module Mismatch {
+  umbrella header "Mismatch.h"
+}
+//--- header-frameworks/Mismatch.framework/Headers/Mismatch.h
+
+//--- dir-frameworks/Mismatch.framework/Modules/module.modulemap
+framework module Mismatch {
+  umbrella "someheaders"
+}
+//--- dir-frameworks/Mismatch.framework/someheaders/Mismatch.h
+
+//--- norm-frameworks/Mismatch.framework/Modules/module.modulemap
+framework module Mismatch {
+  header "Mismatch.h"
+}
+//--- norm-frameworks/Mismatch.framework/Headers/Mismatch.h
+
+//--- mod/module.modulemap
+module Mismatch {
+  umbrella header "Mismatch.h"
+}
+//--- mod/Mismatch.h
+
+//--- other/Mismatch.h
+
+//--- sed-overlay.yaml
+{
+  'version': 0,
+  'roots': [
+{ 'name': 'TEST_DIR', 'type': 'directory',
+  'contents': [
+{ 'name': 'header-frameworks/Mismatch.framework/Headers/Mismatch.h',
+  'type': 'file',
+  'external-contents': 'TEST_DIR/other/Mismatch.h'
+},
+{ 'name': 'dir-frameworks/Mismatch.framework/someheaders',
+  'type': 'directory',
+  'external-contents': 'TEST_DIR/others'
+},
+{ 'name': 'norm-frameworks/Mismatch.framework/Headers/Mismatch.h',
+  'type': 'file',
+  'external-contents': 'TEST_DIR/other/Mism

[PATCH] D107690: [Modules] Do not remove failed modules after the control block phase

[Ben Barham via Phabricator via cfe-commits]

bnbarham updated this revision to Diff 366154.
bnbarham added a comment.

Forgot to add the braces back in for the case statements


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D107690/new/

https://reviews.llvm.org/D107690

Files:
  clang/lib/Serialization/ASTReader.cpp
  clang/test/VFS/Inputs/UsesFoo.framework/Headers/UsesFoo.h
  clang/test/VFS/Inputs/UsesFoo.framework/Modules/module.modulemap
  clang/test/VFS/module-header-mismatches.m
  clang/test/VFS/umbrella-mismatch.m

Index: clang/test/VFS/umbrella-mismatch.m
===
--- clang/test/VFS/umbrella-mismatch.m
+++ /dev/null
@@ -1,7 +0,0 @@
-// RUN: rm -rf %t
-// RUN: sed -e "s;INPUT_DIR;%/S/Inputs;g" -e "s;OUT_DIR;%/S/Inputs;g" %S/Inputs/vfsoverlay.yaml > %t.yaml
-
-// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t -ivfsoverlay %t.yaml -F %S/Inputs -fsyntax-only %s -Wno-atimport-in-framework-header -verify
-// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t -F %S/Inputs -fsyntax-only %s -Wno-atimport-in-framework-header -verify
-// expected-no-diagnostics
-@import UsesFoo;
Index: clang/test/VFS/module-header-mismatches.m
===
--- /dev/null
+++ clang/test/VFS/module-header-mismatches.m
@@ -0,0 +1,86 @@
+// RUN: rm -rf %t
+// RUN: split-file %s %t
+// RUN: sed -e "s;TEST_DIR;%/t;g" %t/sed-overlay.yaml > %t/overlay.yaml
+
+// These tests first build with an overlay such that the header is resolved
+// to %t/other/Mismatch.h. They then build again with the header resolved
+// to the one in their directory.
+//
+// This should cause a rebuild if the contents is different (and thus multiple
+// PCMs), but this currently isn't the case. We should at least not error,
+// since this does happen in real projects (with a different copy of the same
+// file).
+
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/hf-mcp -ivfsoverlay %t/overlay.yaml -F %t/header-frameworks -fsyntax-only -verify %t/use.m
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/hf-mcp -F %t/header-frameworks -fsyntax-only -verify %t/use.m
+// RUN: find %t/hf-mcp -name "Mismatch-*.pcm" | count 1
+
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/df-mcp -ivfsoverlay %t/overlay.yaml -F %t/dir-frameworks -fsyntax-only -verify %t/use.m
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/hf-mcp -F %t/dir-frameworks -fsyntax-only -verify %t/use.m
+// RUN: find %t/df-mcp -name "Mismatch-*.pcm" | count 1
+
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/nf-mcp -ivfsoverlay %t/overlay.yaml -F %t/norm-frameworks -fsyntax-only -verify %t/use.m
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/nf-mcp -F %t/norm-frameworks -fsyntax-only -verify %t/use.m
+// RUN: find %t/nf-mcp -name "Mismatch-*.pcm" | count 1
+
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/m-mcp -ivfsoverlay %t/overlay.yaml -I %t/mod -fsyntax-only -verify %t/use.m
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/m-mcp -I %t/mod -fsyntax-only -verify %t/use.m
+// RUN: find %t/m-mcp -name "Mismatch-*.pcm" | count 1
+
+//--- use.m
+// expected-no-diagnostics
+@import Mismatch;
+
+//--- header-frameworks/Mismatch.framework/Modules/module.modulemap
+framework module Mismatch {
+  umbrella header "Mismatch.h"
+}
+//--- header-frameworks/Mismatch.framework/Headers/Mismatch.h
+
+//--- dir-frameworks/Mismatch.framework/Modules/module.modulemap
+framework module Mismatch {
+  umbrella "someheaders"
+}
+//--- dir-frameworks/Mismatch.framework/someheaders/Mismatch.h
+
+//--- norm-frameworks/Mismatch.framework/Modules/module.modulemap
+framework module Mismatch {
+  header "Mismatch.h"
+}
+//--- norm-frameworks/Mismatch.framework/Headers/Mismatch.h
+
+//--- mod/module.modulemap
+module Mismatch {
+  umbrella header "Mismatch.h"
+}
+//--- mod/Mismatch.h
+
+//--- other/Mismatch.h
+
+//--- sed-overlay.yaml
+{
+  'version': 0,
+  'roots': [
+{ 'name': 'TEST_DIR', 'type': 'directory',
+  'contents': [
+{ 'name': 'header-frameworks/Mismatch.framework/Headers/Mismatch.h',
+  'type': 'file',
+  'external-contents': 'TEST_DIR/other/Mismatch.h'
+},
+{ 'name': 'dir-frameworks/Mismatch.framework/someheaders',
+  'type': 'directory',
+  'external-contents': 'TEST_DIR/others'
+},
+{ 'name': 'norm-frameworks/Mismatch.framework/Headers/Mismatch.h',
+  'type': 'file',
+  'external-contents': 'TEST_DIR/other/Mismatch.h'
+},
+{ 'name': 'mod/Mismatch.h',
+  'type': 'file',
+  'external-contents': 'TEST_DIR/other/Mismatch.h'
+}

[PATCH] D107690: [Modules] Do not remove failed modules after the control block phase

[Ben Barham via Phabricator via cfe-commits]

bnbarham updated this revision to Diff 366378.

Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D107690/new/

https://reviews.llvm.org/D107690

Files:
  clang/lib/Serialization/ASTReader.cpp
  clang/test/VFS/Inputs/UsesFoo.framework/Headers/UsesFoo.h
  clang/test/VFS/Inputs/UsesFoo.framework/Modules/module.modulemap
  clang/test/VFS/module-header-mismatches.m
  clang/test/VFS/umbrella-mismatch.m

Index: clang/test/VFS/umbrella-mismatch.m
===
--- clang/test/VFS/umbrella-mismatch.m
+++ /dev/null
@@ -1,7 +0,0 @@
-// RUN: rm -rf %t
-// RUN: sed -e "s;INPUT_DIR;%/S/Inputs;g" -e "s;OUT_DIR;%/S/Inputs;g" %S/Inputs/vfsoverlay.yaml > %t.yaml
-
-// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t -ivfsoverlay %t.yaml -F %S/Inputs -fsyntax-only %s -Wno-atimport-in-framework-header -verify
-// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t -F %S/Inputs -fsyntax-only %s -Wno-atimport-in-framework-header -verify
-// expected-no-diagnostics
-@import UsesFoo;
Index: clang/test/VFS/module-header-mismatches.m
===
--- /dev/null
+++ clang/test/VFS/module-header-mismatches.m
@@ -0,0 +1,86 @@
+// RUN: rm -rf %t
+// RUN: split-file %s %t
+// RUN: sed -e "s;TEST_DIR;%/t;g" %t/sed-overlay.yaml > %t/overlay.yaml
+
+// These tests first build with an overlay such that the header is resolved
+// to %t/other/Mismatch.h. They then build again with the header resolved
+// to the one in their directory.
+//
+// This should cause a rebuild if the contents is different (and thus multiple
+// PCMs), but this currently isn't the case. We should at least not error,
+// since this does happen in real projects (with a different copy of the same
+// file).
+
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/hf-mcp -ivfsoverlay %t/overlay.yaml -F %t/header-frameworks -fsyntax-only -verify %t/use.m
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/hf-mcp -F %t/header-frameworks -fsyntax-only -verify %t/use.m
+// RUN: find %t/hf-mcp -name "Mismatch-*.pcm" | count 1
+
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/df-mcp -ivfsoverlay %t/overlay.yaml -F %t/dir-frameworks -fsyntax-only -verify %t/use.m
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/hf-mcp -F %t/dir-frameworks -fsyntax-only -verify %t/use.m
+// RUN: find %t/df-mcp -name "Mismatch-*.pcm" | count 1
+
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/nf-mcp -ivfsoverlay %t/overlay.yaml -F %t/norm-frameworks -fsyntax-only -verify %t/use.m
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/nf-mcp -F %t/norm-frameworks -fsyntax-only -verify %t/use.m
+// RUN: find %t/nf-mcp -name "Mismatch-*.pcm" | count 1
+
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/m-mcp -ivfsoverlay %t/overlay.yaml -I %t/mod -fsyntax-only -verify %t/use.m
+// RUN: %clang_cc1 -Werror -fmodules -fimplicit-module-maps -fmodules-cache-path=%t/m-mcp -I %t/mod -fsyntax-only -verify %t/use.m
+// RUN: find %t/m-mcp -name "Mismatch-*.pcm" | count 1
+
+//--- use.m
+// expected-no-diagnostics
+@import Mismatch;
+
+//--- header-frameworks/Mismatch.framework/Modules/module.modulemap
+framework module Mismatch {
+  umbrella header "Mismatch.h"
+}
+//--- header-frameworks/Mismatch.framework/Headers/Mismatch.h
+
+//--- dir-frameworks/Mismatch.framework/Modules/module.modulemap
+framework module Mismatch {
+  umbrella "someheaders"
+}
+//--- dir-frameworks/Mismatch.framework/someheaders/Mismatch.h
+
+//--- norm-frameworks/Mismatch.framework/Modules/module.modulemap
+framework module Mismatch {
+  header "Mismatch.h"
+}
+//--- norm-frameworks/Mismatch.framework/Headers/Mismatch.h
+
+//--- mod/module.modulemap
+module Mismatch {
+  umbrella header "Mismatch.h"
+}
+//--- mod/Mismatch.h
+
+//--- other/Mismatch.h
+
+//--- sed-overlay.yaml
+{
+  'version': 0,
+  'roots': [
+{ 'name': 'TEST_DIR', 'type': 'directory',
+  'contents': [
+{ 'name': 'header-frameworks/Mismatch.framework/Headers/Mismatch.h',
+  'type': 'file',
+  'external-contents': 'TEST_DIR/other/Mismatch.h'
+},
+{ 'name': 'dir-frameworks/Mismatch.framework/someheaders',
+  'type': 'directory',
+  'external-contents': 'TEST_DIR/others'
+},
+{ 'name': 'norm-frameworks/Mismatch.framework/Headers/Mismatch.h',
+  'type': 'file',
+  'external-contents': 'TEST_DIR/other/Mismatch.h'
+},
+{ 'name': 'mod/Mismatch.h',
+  'type': 'file',
+  'external-contents': 'TEST_DIR/other/Mismatch.h'
+}
+  ]
+}
+  ]
+}
+
Index: clang/test/VFS/Inputs/UsesFoo.framework/Modules/mod

[PATCH] D108268: [Modules] Change result of reading AST block to llvm::Error instead

[Ben Barham via Phabricator via cfe-commits]

bnbarham created this revision.
bnbarham added reviewers: vsapsai, dexonsmith.
bnbarham requested review of this revision.
Herald added a project: clang.
Herald added a subscriber: cfe-commits.

Reading the AST block can never fail with a recoverable error as modules
cannot be removed during this phase. Change the return type of these
functions to return an llvm::Error instead, ie. either success or
failure.

NFC other than the wording of some of the errors.


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D108268

Files:
  clang/include/clang/Serialization/ASTReader.h
  clang/lib/Serialization/ASTReader.cpp

Index: clang/lib/Serialization/ASTReader.cpp
===
--- clang/lib/Serialization/ASTReader.cpp
+++ clang/lib/Serialization/ASTReader.cpp
@@ -815,6 +815,23 @@
 // AST reader implementation
 //===--===//
 
+namespace {
+/// An already-diagnosed error
+class DiagnosedError : public llvm::ErrorInfo {
+public:
+  static char ID;
+
+  DiagnosedError() {}
+
+  void log(llvm::raw_ostream &OS) const override {}
+
+  std::error_code convertToErrorCode() const override {
+return llvm::inconvertibleErrorCode();
+  }
+};
+char DiagnosedError::ID;
+} // namespace
+
 static uint64_t readULEB(const unsigned char *&P) {
   unsigned Length = 0;
   const char *Error = nullptr;
@@ -1263,7 +1280,14 @@
 }
 
 void ASTReader::Error(llvm::Error &&Err) const {
-  Error(toString(std::move(Err)));
+  SmallVector Errors;
+  handleAllErrors(
+  std::move(Err), [](const DiagnosedError &E) {},
+  [&Errors](const llvm::ErrorInfoBase &EI) {
+Errors.push_back(EI.message());
+  });
+  if (!Errors.empty())
+Error(llvm::join(Errors.begin(), Errors.end(), "\n"));
 }
 
 //===--===//
@@ -1271,9 +1295,7 @@
 //===--===//
 
 /// Read the line table in the source manager block.
-/// \returns true if there was an error.
-bool ASTReader::ParseLineTable(ModuleFile &F,
-   const RecordData &Record) {
+void ASTReader::ParseLineTable(ModuleFile &F, const RecordData &Record) {
   unsigned Idx = 0;
   LineTableInfo &LineTable = SourceMgr.getLineTable();
 
@@ -1312,12 +1334,10 @@
 }
 LineTable.AddEntry(FileID::get(FID), Entries);
   }
-
-  return false;
 }
 
 /// Read a source manager block
-bool ASTReader::ReadSourceManagerBlock(ModuleFile &F) {
+llvm::Error ASTReader::ReadSourceManagerBlock(ModuleFile &F) {
   using namespace SrcMgr;
 
   BitstreamCursor &SLocEntryCursor = F.SLocEntryCursor;
@@ -1329,36 +1349,29 @@
   SLocEntryCursor = F.Stream;
 
   // The stream itself is going to skip over the source manager block.
-  if (llvm::Error Err = F.Stream.SkipBlock()) {
-Error(std::move(Err));
-return true;
-  }
+  if (llvm::Error Err = F.Stream.SkipBlock())
+return Err;
 
   // Enter the source manager block.
-  if (llvm::Error Err =
-  SLocEntryCursor.EnterSubBlock(SOURCE_MANAGER_BLOCK_ID)) {
-Error(std::move(Err));
-return true;
-  }
+  if (llvm::Error Err = SLocEntryCursor.EnterSubBlock(SOURCE_MANAGER_BLOCK_ID))
+return Err;
   F.SourceManagerBlockStartOffset = SLocEntryCursor.GetCurrentBitNo();
 
   RecordData Record;
   while (true) {
 Expected MaybeE =
 SLocEntryCursor.advanceSkippingSubblocks();
-if (!MaybeE) {
-  Error(MaybeE.takeError());
-  return true;
-}
+if (!MaybeE)
+  return MaybeE.takeError();
 llvm::BitstreamEntry E = MaybeE.get();
 
 switch (E.Kind) {
 case llvm::BitstreamEntry::SubBlock: // Handled for us already.
 case llvm::BitstreamEntry::Error:
-  Error("malformed block record in AST file");
-  return true;
+  return llvm::createStringError(std::errc::illegal_byte_sequence,
+ "malformed block record in AST file");
 case llvm::BitstreamEntry::EndBlock:
-  return false;
+  return llvm::Error::success();
 case llvm::BitstreamEntry::Record:
   // The interesting case.
   break;
@@ -1369,10 +1382,8 @@
 StringRef Blob;
 Expected MaybeRecord =
 SLocEntryCursor.readRecord(E.ID, Record, &Blob);
-if (!MaybeRecord) {
-  Error(MaybeRecord.takeError());
-  return true;
-}
+if (!MaybeRecord)
+  return MaybeRecord.takeError();
 switch (MaybeRecord.get()) {
 default:  // Default behavior: ignore.
   break;
@@ -1381,7 +1392,7 @@
 case SM_SLOC_BUFFER_ENTRY:
 case SM_SLOC_EXPANSION_ENTRY:
   // Once we hit one of the source location entries, we're done.
-  return false;
+  return llvm::Error::success();
 }
   }
 }
@@ -1632,13 +1643,11 @@
 /// Enter a subblock of the specified BlockID with the specified cursor. Read
 /// the abbreviations that are at the top of the block and then leave 

[PATCH] D105328: [Frontend] Only compile modules if not already finalized

[Ben Barham via Phabricator via cfe-commits]

bnbarham created this revision.
bnbarham added reviewers: akyrtzi, arphaman.
Herald added a subscriber: mgorny.
bnbarham requested review of this revision.
Herald added a project: clang.
Herald added a subscriber: cfe-commits.

It was possible to re-add a module to a shared in-memory module cache
when search paths are changed.  This can eventually cause a crash if the
original module is referenced after this occurs.

1. Module A depends on B
2. B exists in two paths C and D
3. First run only has C on the search path, finds A and B and loads them
4. Second run adds D to the front of the search path. A is loaded and contains 
a reference to the already compiled module from C. But searching finds the 
module from D instead, causing a mismatch
5. B is considered out of date and is now rebuilt
6. The recompiled module is added to the in-memory cache, freeing the 
previously inserted one

This can never occur from a regular clang process, but is very easy to
do through the API. Update the compilation to return early if a module
is already finalized so that the pre-condition in the in-memory module
cache holds.


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D105328

Files:
  clang/include/clang/Basic/DiagnosticCommonKinds.td
  clang/include/clang/Serialization/ASTReader.h
  clang/lib/Frontend/CompilerInstance.cpp
  clang/lib/Serialization/ASTReader.cpp
  clang/unittests/Serialization/CMakeLists.txt
  clang/unittests/Serialization/ModuleCacheTest.cpp

Index: clang/unittests/Serialization/ModuleCacheTest.cpp
===
--- /dev/null
+++ clang/unittests/Serialization/ModuleCacheTest.cpp
@@ -0,0 +1,135 @@
+//===- unittests/Serialization/ModuleCacheTest.cpp - CI tests -===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Basic/FileManager.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Frontend/CompilerInvocation.h"
+#include "clang/Frontend/FrontendActions.h"
+#include "clang/Lex/HeaderSearch.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/raw_ostream.h"
+
+#include "gtest/gtest.h"
+
+using namespace llvm;
+using namespace clang;
+
+namespace {
+
+class ModuleCacheTest : public ::testing::Test {
+  void SetUp() override {
+ASSERT_FALSE(sys::fs::createUniqueDirectory("modulecache-test", TestDir));
+
+ModuleCachePath = SmallString<256>(TestDir);
+sys::path::append(ModuleCachePath, "mcp");
+ASSERT_FALSE(sys::fs::create_directories(ModuleCachePath));
+  }
+
+  void TearDown() override { sys::fs::remove_directories(TestDir); }
+
+public:
+  SmallString<256> TestDir;
+  SmallString<256> ModuleCachePath;
+
+  void AddFile(StringRef Path, StringRef Contents) {
+ASSERT_FALSE(sys::path::is_absolute(Path));
+
+SmallString<256> AbsPath(TestDir);
+sys::path::append(AbsPath, Path);
+
+std::error_code EC;
+ASSERT_FALSE(
+sys::fs::create_directories(llvm::sys::path::parent_path(AbsPath)));
+llvm::raw_fd_ostream OS(AbsPath, EC);
+ASSERT_FALSE(EC);
+OS << Contents;
+  }
+};
+
+TEST_F(ModuleCacheTest, CachedModuleNewPath) {
+  AddFile("test.m", R"cpp(
+  @import Top;
+  )cpp");
+
+  AddFile("frameworks/Top.framework/Headers/top.h", R"cpp(
+  @import M;
+  )cpp");
+  AddFile("frameworks/Top.framework/Modules/module.modulemap", R"cpp(
+  framework module Top [system] {
+header "top.h"
+export *
+  }
+  )cpp");
+
+  AddFile("frameworks/M.framework/Headers/m.h", R"cpp(
+  void foo();
+  )cpp");
+  AddFile("frameworks/M.framework/Modules/module.modulemap", R"cpp(
+  framework module M [system] {
+header "m.h"
+export *
+  }
+  )cpp");
+
+  AddFile("frameworks2/M.framework/Headers/m.h", R"cpp(
+  void foo();
+  )cpp");
+  AddFile("frameworks2/M.framework/Modules/module.modulemap", R"cpp(
+  framework module M [system] {
+header "m.h"
+export *
+  }
+  )cpp");
+
+  SmallString<256> MCPArg("-fmodules-cache-path=");
+  MCPArg.append(ModuleCachePath);
+  IntrusiveRefCntPtr Diags =
+  CompilerInstance::createDiagnostics(new DiagnosticOptions());
+
+  // First run should pass with no errors
+  const char *Args[] = {"clang","-fmodules",  "-Fframeworks",
+MCPArg.c_str(), "-working-directory", TestDir.c_str(),
+"test.m"};
+  std::shared_ptr Invocation =
+  createInvocationFromCommandLine(Args, Diags);
+  ASSERT_TRUE(Invocation);
+  CompilerInstance Instance;
+  Instance.setDiagnostics(Diags.get());
+  Instance.setInvocation(Invocation);
+  SyntaxOnlyAction Action;
+  ASSERT_TRUE(Instance.ExecuteAction(Action));
+  ASSERT_F

[PATCH] D105328: [Frontend] Only compile modules if not already finalized

[Ben Barham via Phabricator via cfe-commits]

bnbarham added a comment.

In D105328#2861028 , @vsapsai wrote:

> This problem shouldn't happen based on module-level name hashing but for some 
> reason it is happening. If I'm not mistaken, InMemoryModuleCache uses .pcm 
> file name as the key. But that file name should contain a hash part that is 
> based on the modulemap path. But frameworks M in "frameworks" and 
> "frameworks2" should have different paths for modulemaps and should have 
> different .pcm names, therefore no conflicts in InMemoryModuleCache.

Ah sorry, looks like I had my modules slightly confused. The underlying problem 
is that `B` is considered out of date due to the check on line 2920, ie. the 
one that creates the `err_imported_module_relocated` diagnostic. That causes 
`Top` to be re-compiled as a module it depends on is out of date. The location 
of `Top` hasn't changed, so it keeps the same .pcm path.

`B` is also re-compiled, but as you point out that's perfectly fine since it'll 
get a different path in the cache.

I'll update the comments in the test + the review summary. Sorry for the 
confusion.

> I see the value in not crashing but looks like there might be a problem 
> somewhere else.

For what it's worth, even if there was another cause I'd still like to get this 
change in - writing over a module in the cache causes all sorts of different 
crashes that are hard to diagnose. I'd be *very* surprised if there weren't 
more issues like this, so at least we'll get an actual error rather than random 
crashes.

> Also it is more of my personal pet peeve. InMemoryModuleCache should simplify 
> memory management for modules but in practice it simplifies creating dangling 
> references. I'm not asking to address that, so if you feel like my requests 
> get too close to that direction, it's good to push back.

Heh, I find this a nice summary of how I feel about it after trying to find the 
root cause of these crashes!


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D105328/new/

https://reviews.llvm.org/D105328

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D105328: [Frontend] Only compile modules if not already finalized

[Ben Barham via Phabricator via cfe-commits]

bnbarham updated this revision to Diff 356861.
bnbarham added a comment.

Updated the test comment with the actual cause (ie. `Top` being recompiled and 
inserted, not `M`).


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D105328/new/

https://reviews.llvm.org/D105328

Files:
  clang/include/clang/Basic/DiagnosticCommonKinds.td
  clang/include/clang/Serialization/ASTReader.h
  clang/lib/Frontend/CompilerInstance.cpp
  clang/lib/Serialization/ASTReader.cpp
  clang/unittests/Serialization/CMakeLists.txt
  clang/unittests/Serialization/ModuleCacheTest.cpp

Index: clang/unittests/Serialization/ModuleCacheTest.cpp
===
--- /dev/null
+++ clang/unittests/Serialization/ModuleCacheTest.cpp
@@ -0,0 +1,134 @@
+//===- unittests/Serialization/ModuleCacheTest.cpp - CI tests -===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Basic/FileManager.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Frontend/CompilerInvocation.h"
+#include "clang/Frontend/FrontendActions.h"
+#include "clang/Lex/HeaderSearch.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/raw_ostream.h"
+
+#include "gtest/gtest.h"
+
+using namespace llvm;
+using namespace clang;
+
+namespace {
+
+class ModuleCacheTest : public ::testing::Test {
+  void SetUp() override {
+ASSERT_FALSE(sys::fs::createUniqueDirectory("modulecache-test", TestDir));
+
+ModuleCachePath = SmallString<256>(TestDir);
+sys::path::append(ModuleCachePath, "mcp");
+ASSERT_FALSE(sys::fs::create_directories(ModuleCachePath));
+  }
+
+  void TearDown() override { sys::fs::remove_directories(TestDir); }
+
+public:
+  SmallString<256> TestDir;
+  SmallString<256> ModuleCachePath;
+
+  void AddFile(StringRef Path, StringRef Contents) {
+ASSERT_FALSE(sys::path::is_absolute(Path));
+
+SmallString<256> AbsPath(TestDir);
+sys::path::append(AbsPath, Path);
+
+std::error_code EC;
+ASSERT_FALSE(
+sys::fs::create_directories(llvm::sys::path::parent_path(AbsPath)));
+llvm::raw_fd_ostream OS(AbsPath, EC);
+ASSERT_FALSE(EC);
+OS << Contents;
+  }
+};
+
+TEST_F(ModuleCacheTest, CachedModuleNewPath) {
+  AddFile("test.m", R"cpp(
+  @import Top;
+  )cpp");
+
+  AddFile("frameworks/Top.framework/Headers/top.h", R"cpp(
+  @import M;
+  )cpp");
+  AddFile("frameworks/Top.framework/Modules/module.modulemap", R"cpp(
+  framework module Top [system] {
+header "top.h"
+export *
+  }
+  )cpp");
+
+  AddFile("frameworks/M.framework/Headers/m.h", R"cpp(
+  void foo();
+  )cpp");
+  AddFile("frameworks/M.framework/Modules/module.modulemap", R"cpp(
+  framework module M [system] {
+header "m.h"
+export *
+  }
+  )cpp");
+
+  AddFile("frameworks2/M.framework/Headers/m.h", R"cpp(
+  void foo();
+  )cpp");
+  AddFile("frameworks2/M.framework/Modules/module.modulemap", R"cpp(
+  framework module M [system] {
+header "m.h"
+export *
+  }
+  )cpp");
+
+  SmallString<256> MCPArg("-fmodules-cache-path=");
+  MCPArg.append(ModuleCachePath);
+  IntrusiveRefCntPtr Diags =
+  CompilerInstance::createDiagnostics(new DiagnosticOptions());
+
+  // First run should pass with no errors
+  const char *Args[] = {"clang","-fmodules",  "-Fframeworks",
+MCPArg.c_str(), "-working-directory", TestDir.c_str(),
+"test.m"};
+  std::shared_ptr Invocation =
+  createInvocationFromCommandLine(Args, Diags);
+  ASSERT_TRUE(Invocation);
+  CompilerInstance Instance;
+  Instance.setDiagnostics(Diags.get());
+  Instance.setInvocation(Invocation);
+  SyntaxOnlyAction Action;
+  ASSERT_TRUE(Instance.ExecuteAction(Action));
+  ASSERT_FALSE(Diags->hasErrorOccurred());
+
+  // Now add `frameworks2` to the search path. `Top.pcm` will have a reference
+  // to the `M` from `frameworks`, but a search will find the `M` from
+  // `frameworks2` - causing a mismatch and it to be considered out of date.
+  //
+  // Normally this would be fine - `M` and the modules it depends on would be
+  // rebuilt. However, since we have a shared module cache and thus an already
+  // finalized `Top`, recompiling `Top` will cause the existing module to be
+  // removed from the cache, causing possible crashed if it is ever used.
+  //
+  // Make sure that an error occurs instead.
+  const char *Args2[] = {"clang", "-fmodules","-Fframeworks2",
+ "-Fframeworks",  MCPArg.c_str(), "-working-directory",
+ TestDir.c_str(), "test.m"};
+  std::shared_ptr Invocation2 =
+ 

[PATCH] D105328: [Frontend] Only compile modules if not already finalized

[Ben Barham via Phabricator via cfe-commits]

bnbarham added inline comments.



Comment at: clang/unittests/Serialization/ModuleCacheTest.cpp:125-126
+  ASSERT_TRUE(Invocation2);
+  CompilerInstance Instance2(Instance.getPCHContainerOperations(),
+ &Instance.getModuleCache());
+  Instance2.setDiagnostics(Diags.get());

vsapsai wrote:
> Haven't rechecked the code more carefully but had an idea is that if we want 
> to allow InMemoryModuleCache reuse between multiple CompilerInstance, safer 
> API would be to transfer ModuleCache ownership to the new CompilerInstance 
> and maybe make all records in the cache purgeable. But that's applicable only 
> if ModuleCache reuse is important.
Every module compilation runs in a separate thread with a separate 
CompilerInstance (but shared cache). So ownership would have to be transferred 
back to the original instance once complete. I might be misunderstanding what 
you mean here though.

Another point to note is that eg. Swift doesn't actually create another 
CompilerInstance, it's just re-used across module loading and 
`CompilerInstance::loadModule` is called directly. I went with this way in the 
test as otherwise we'd need to inline most of `ExecuteAction` and have a custom 
`FrontendAction`.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D105328/new/

https://reviews.llvm.org/D105328

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D105328: [Frontend] Only compile modules if not already finalized

[Ben Barham via Phabricator via cfe-commits]

bnbarham added inline comments.



Comment at: clang/lib/Frontend/CompilerInstance.cpp:1063
+<< ModuleName;
+return ImportingInstance.getFrontendOpts().AllowPCMWithCompilerErrors;
+  }

vsapsai wrote:
> Can we get in infinite loop with `AllowPCMWithCompilerErrors = true`? 
> Specifically, I'm thinking about the scenario
> 
> 1. `compileModuleAndReadAST` obtains a file lock and calls `compileModule`
> 2. `compileModule` calls `compileModuleImpl`
> 3. Module is finalized but `AllowPCMWithCompilerErrors` is true, so 
> `compileModuleImpl` returns true
> 4. `compileModule` returns true
> 5. `compileModuleAndReadAST` tries to read AST because compilation was 
> successful
> 6. AST is out of date, so `compileModuleAndReadAST` decides to try again, 
> goto 1
> 
> Haven't tried to reproduce it locally but even if this scenario is 
> impossible, a corresponding test case can be useful.
Nice catch, that does seem likely - I'll see if I can add a test for this.



Comment at: clang/lib/Serialization/ASTReader.cpp:2923
   if (!BuildDir || *BuildDir != M->Directory) {
-if ((ClientLoadCapabilities & ARR_OutOfDate) == 0)
+if (diagnoseOutOfDate(F.FileName, ClientLoadCapabilities))
   Diag(diag::err_imported_module_relocated)

vsapsai wrote:
> I'm thinking if in case of finalized modules diagnostic messages are good 
> enough. One concern is it won't be clear why a module wasn't rebuilt. It can 
> be already confusing for precompiled headers and I'm afraid we won't be able 
> to detect `isPCMFinal` code path without a debugger. Though I don't know how 
> bad that would be in practice.
> 
> Another concern is that retrying a compilation should succeed as for a new 
> process we have a new InMemoryModuleCache and `isPCMFinal` should return 
> false. So we might have non-deterministic behavior and some of the valid 
> error messages can seem to be non-deterministic and not reliable. I was 
> thinking about adding a note in case we are dealing with `isPCMFinal` to 
> distinguish these cases but not sure it is a good idea.
> I'm thinking if in case of finalized modules diagnostic messages are good 
> enough. One concern is it won't be clear why a module wasn't rebuilt. It can 
> be already confusing for precompiled headers and I'm afraid we won't be able 
> to detect `isPCMFinal` code path without a debugger. Though I don't know how 
> bad that would be in practice.

The error messages will mention a module in the module cache, which would be 
the main way to tell. We could add a note here as you suggest below, but I'm 
not quite sure what it would say... something about there being two modules 
with the same name?

> Another concern is that retrying a compilation should succeed as for a new 
> process we have a new InMemoryModuleCache and `isPCMFinal` should return 
> false. So we might have non-deterministic behavior and some of the valid 
> error messages can seem to be non-deterministic and not reliable. I was 
> thinking about adding a note in case we are dealing with `isPCMFinal` to 
> distinguish these cases but not sure it is a good idea.

The errors should be deterministic I believe. If one process has the issue then 
a new one will have the issue as well. For what it's worth, I don't think these 
crashes are possible from the clang frontend. They require messing around with 
search paths such that between two compilations in the same process, different 
modules are found.



Comment at: clang/lib/Serialization/ASTReader.cpp:5929
 
+bool ASTReader::diagnoseOutOfDate(StringRef ModuleFileName,
+  unsigned int ClientLoadCapabilities) {

vsapsai wrote:
> Based on the rest of the code in clang, the expectation for `diagnose...` 
> methods is to emit diagnostic in some cases. Personally, I'm often confused 
> what true/false means for these methods, so I'm thinking about renaming the 
> method to something like isRecoverableOutOfDateModule, 
> canRecoverOutOfDateModule or some such. Feel free to pick a name you believe 
> is appropriate, mine are just examples.
Fair enough, `canRecoverOutOfDateModule` sounds reasonable to me. Or maybe 
`canRecoverFromOutOfDate`?


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D105328/new/

https://reviews.llvm.org/D105328

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D105328: [Frontend] Only compile modules if not already finalized

[Ben Barham via Phabricator via cfe-commits]

bnbarham marked 5 inline comments as done.
bnbarham added inline comments.



Comment at: clang/lib/Frontend/CompilerInstance.cpp:1063
+<< ModuleName;
+return ImportingInstance.getFrontendOpts().AllowPCMWithCompilerErrors;
+  }

bnbarham wrote:
> vsapsai wrote:
> > Can we get in infinite loop with `AllowPCMWithCompilerErrors = true`? 
> > Specifically, I'm thinking about the scenario
> > 
> > 1. `compileModuleAndReadAST` obtains a file lock and calls `compileModule`
> > 2. `compileModule` calls `compileModuleImpl`
> > 3. Module is finalized but `AllowPCMWithCompilerErrors` is true, so 
> > `compileModuleImpl` returns true
> > 4. `compileModule` returns true
> > 5. `compileModuleAndReadAST` tries to read AST because compilation was 
> > successful
> > 6. AST is out of date, so `compileModuleAndReadAST` decides to try again, 
> > goto 1
> > 
> > Haven't tried to reproduce it locally but even if this scenario is 
> > impossible, a corresponding test case can be useful.
> Nice catch, that does seem likely - I'll see if I can add a test for this.
It doesn't end up causing an infinite recursion as `false` will end up being 
returned from `compileModuleAndReadAST`, but in that case there's no point 
returning `true` from `compileModuleImpl` in the first place so I've changed it 
anyway. Have also added that as a test case, just to make sure.



Comment at: clang/lib/Serialization/ASTReader.cpp:2923
   if (!BuildDir || *BuildDir != M->Directory) {
-if ((ClientLoadCapabilities & ARR_OutOfDate) == 0)
+if (diagnoseOutOfDate(F.FileName, ClientLoadCapabilities))
   Diag(diag::err_imported_module_relocated)

bnbarham wrote:
> vsapsai wrote:
> > I'm thinking if in case of finalized modules diagnostic messages are good 
> > enough. One concern is it won't be clear why a module wasn't rebuilt. It 
> > can be already confusing for precompiled headers and I'm afraid we won't be 
> > able to detect `isPCMFinal` code path without a debugger. Though I don't 
> > know how bad that would be in practice.
> > 
> > Another concern is that retrying a compilation should succeed as for a new 
> > process we have a new InMemoryModuleCache and `isPCMFinal` should return 
> > false. So we might have non-deterministic behavior and some of the valid 
> > error messages can seem to be non-deterministic and not reliable. I was 
> > thinking about adding a note in case we are dealing with `isPCMFinal` to 
> > distinguish these cases but not sure it is a good idea.
> > I'm thinking if in case of finalized modules diagnostic messages are good 
> > enough. One concern is it won't be clear why a module wasn't rebuilt. It 
> > can be already confusing for precompiled headers and I'm afraid we won't be 
> > able to detect `isPCMFinal` code path without a debugger. Though I don't 
> > know how bad that would be in practice.
> 
> The error messages will mention a module in the module cache, which would be 
> the main way to tell. We could add a note here as you suggest below, but I'm 
> not quite sure what it would say... something about there being two modules 
> with the same name?
> 
> > Another concern is that retrying a compilation should succeed as for a new 
> > process we have a new InMemoryModuleCache and `isPCMFinal` should return 
> > false. So we might have non-deterministic behavior and some of the valid 
> > error messages can seem to be non-deterministic and not reliable. I was 
> > thinking about adding a note in case we are dealing with `isPCMFinal` to 
> > distinguish these cases but not sure it is a good idea.
> 
> The errors should be deterministic I believe. If one process has the issue 
> then a new one will have the issue as well. For what it's worth, I don't 
> think these crashes are possible from the clang frontend. They require 
> messing around with search paths such that between two compilations in the 
> same process, different modules are found.
I've added an extra note "this is generally caused by modules with the same 
name found in multiple paths". So the diagnostics would now be:
```
/.../test.m:2:10: fatal error: module 'M' was built in directory 
'/.../frameworks/M.framework' but now resides in directory 
'frameworks2/M.framework'
@import Top;
 ^
/.../test.m:2:10: note: imported by module 'Top' in 
'/path/to/module/cache/LGSY9KSYAKN1/Top-1MZE9QJ1AHENQ.pcm
/.../test.m:2:10: note: this is generally caused by modules with the same name 
found in multiple paths
```

I was thinking of mentioning the finalized module, but that's really just a 
side effect of the error that should have already been output (ie. `module 'M' 
was built in...` in this case). So the existence of this note should help point 
users to the problem + make which path caused the error obvious when looking at 
the code.



Comment at: clang/lib/Serialization/ASTReader.cpp:5929
 
+bool ASTReader::diagn

[PATCH] D105328: [Frontend] Only compile modules if not already finalized

[Ben Barham via Phabricator via cfe-commits]

bnbarham updated this revision to Diff 358465.
bnbarham edited the summary of this revision.

Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D105328/new/

https://reviews.llvm.org/D105328

Files:
  clang/include/clang/Basic/DiagnosticCommonKinds.td
  clang/include/clang/Basic/DiagnosticSerializationKinds.td
  clang/include/clang/Serialization/ASTReader.h
  clang/lib/Frontend/CompilerInstance.cpp
  clang/lib/Serialization/ASTReader.cpp
  clang/unittests/Serialization/CMakeLists.txt
  clang/unittests/Serialization/ModuleCacheTest.cpp

Index: clang/unittests/Serialization/ModuleCacheTest.cpp
===
--- /dev/null
+++ clang/unittests/Serialization/ModuleCacheTest.cpp
@@ -0,0 +1,179 @@
+//===- unittests/Serialization/ModuleCacheTest.cpp - CI tests -===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+#include "clang/Basic/FileManager.h"
+#include "clang/Frontend/CompilerInstance.h"
+#include "clang/Frontend/CompilerInvocation.h"
+#include "clang/Frontend/FrontendActions.h"
+#include "clang/Lex/HeaderSearch.h"
+#include "llvm/ADT/SmallString.h"
+#include "llvm/Support/FileSystem.h"
+#include "llvm/Support/raw_ostream.h"
+
+#include "gtest/gtest.h"
+
+using namespace llvm;
+using namespace clang;
+
+namespace {
+
+class ModuleCacheTest : public ::testing::Test {
+  void SetUp() override {
+ASSERT_FALSE(sys::fs::createUniqueDirectory("modulecache-test", TestDir));
+
+ModuleCachePath = SmallString<256>(TestDir);
+sys::path::append(ModuleCachePath, "mcp");
+ASSERT_FALSE(sys::fs::create_directories(ModuleCachePath));
+  }
+
+  void TearDown() override { sys::fs::remove_directories(TestDir); }
+
+public:
+  SmallString<256> TestDir;
+  SmallString<256> ModuleCachePath;
+
+  void addFile(StringRef Path, StringRef Contents) {
+ASSERT_FALSE(sys::path::is_absolute(Path));
+
+SmallString<256> AbsPath(TestDir);
+sys::path::append(AbsPath, Path);
+
+std::error_code EC;
+ASSERT_FALSE(
+sys::fs::create_directories(llvm::sys::path::parent_path(AbsPath)));
+llvm::raw_fd_ostream OS(AbsPath, EC);
+ASSERT_FALSE(EC);
+OS << Contents;
+  }
+
+  void addDuplicateFrameworks() {
+addFile("test.m", R"cpp(
+@import Top;
+)cpp");
+
+addFile("frameworks/Top.framework/Headers/top.h", R"cpp(
+@import M;
+)cpp");
+addFile("frameworks/Top.framework/Modules/module.modulemap", R"cpp(
+framework module Top [system] {
+  header "top.h"
+  export *
+}
+)cpp");
+
+addFile("frameworks/M.framework/Headers/m.h", R"cpp(
+void foo();
+)cpp");
+addFile("frameworks/M.framework/Modules/module.modulemap", R"cpp(
+framework module M [system] {
+  header "m.h"
+  export *
+}
+)cpp");
+
+addFile("frameworks2/M.framework/Headers/m.h", R"cpp(
+void foo();
+)cpp");
+addFile("frameworks2/M.framework/Modules/module.modulemap", R"cpp(
+framework module M [system] {
+  header "m.h"
+  export *
+}
+)cpp");
+  }
+};
+
+TEST_F(ModuleCacheTest, CachedModuleNewPath) {
+  addDuplicateFrameworks();
+
+  SmallString<256> MCPArg("-fmodules-cache-path=");
+  MCPArg.append(ModuleCachePath);
+  IntrusiveRefCntPtr Diags =
+  CompilerInstance::createDiagnostics(new DiagnosticOptions());
+
+  // First run should pass with no errors
+  const char *Args[] = {"clang","-fmodules",  "-Fframeworks",
+MCPArg.c_str(), "-working-directory", TestDir.c_str(),
+"test.m"};
+  std::shared_ptr Invocation =
+  createInvocationFromCommandLine(Args, Diags);
+  ASSERT_TRUE(Invocation);
+  CompilerInstance Instance;
+  Instance.setDiagnostics(Diags.get());
+  Instance.setInvocation(Invocation);
+  SyntaxOnlyAction Action;
+  ASSERT_TRUE(Instance.ExecuteAction(Action));
+  ASSERT_FALSE(Diags->hasErrorOccurred());
+
+  // Now add `frameworks2` to the search path. `Top.pcm` will have a reference
+  // to the `M` from `frameworks`, but a search will find the `M` from
+  // `frameworks2` - causing a mismatch and it to be considered out of date.
+  //
+  // Normally this would be fine - `M` and the modules it depends on would be
+  // rebuilt. However, since we have a shared module cache and thus an already
+  // finalized `Top`, recompiling `Top` will cause the existing module to be
+  // removed from the cache, causing possible crashed if it is ever used.
+  //
+  // Make sure that an error occurs instead.
+  const char *Args2[] = {"clang", "-fmodules","-Fframeworks2",
+ "-Fframeworks",  MCPArg.c_str(), "-

[PATCH] D113473: [clang][deps] Don't emit `-fmodule-map-file=`

[Ben Barham via Phabricator via cfe-commits]

bnbarham accepted this revision.
bnbarham added a comment.
This revision is now accepted and ready to land.

That was quick! Seems reasonable to me -hopefully there's not some strange case 
where this wouldn't work, will be nice to avoid the parsing time (which 
actually isn't insignificant).


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D113473/new/

https://reviews.llvm.org/D113473

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D135634: [clang][modules] Serialize VFS overlay paths into PCMs

[Ben Barham via Phabricator via cfe-commits]

bnbarham added a comment.

In D135634#3866353 , @benlangmuir 
wrote:

> I think we should deduplicate the vfs overlays if the same ivfsoverlay is 
> specified in both the pcm and the command-line.
>
> @bnbarham any concern about overlay vs chaining behaviour here? I remember 
> you looking at that a while ago.

It still chains right now since I never ended up finishing those patches. This 
does explicitly *chain* with the command-line specified FS, not sure if one 
makes sense over the other here. Should we be passing in the real FS to the 
overlay creation here and then creating an actual overlay FS with both the 
original FileManager VFS + the PCM VFS (or building the overlay from scratch 
with the original overlays as well, though that still chains so...)?

Do reproducers read PCMs at all or do they rebuild from the headers?


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D135634/new/

https://reviews.llvm.org/D135634

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D135634: [clang][modules] Serialize VFS overlay paths into PCMs

[Ben Barham via Phabricator via cfe-commits]

bnbarham added a comment.

In D135634#3866704 , @jansvoboda11 
wrote:

> In D135634#3866353 , @benlangmuir 
> wrote:
>
>> I think we should deduplicate the vfs overlays if the same ivfsoverlay is 
>> specified in both the pcm and the command-line.
>
> My understanding is that `ASTUnit` never uses command-line and always adopts 
> whatever is in the PCM.

It's now using the FS from `FileManager` as the base FS and I imagine was 
already just using the `FileManager` FS, so we are (and were) implicitly using 
the command-line overlays.

> In D135634#3866528 , @bnbarham 
> wrote:
>
>> Do reproducers read PCMs at all or do they rebuild from the headers?
>
> I don't know. How are they related to this patch?

It was more related to the chaining discussion. If reproducers do use PCMs then 
that would imply we *have* to chain, since we'd need to map from the path in 
the PCM -> original path (with PCM overlay) -> reproducer path (with reproducer 
overlay).

I suspect the answer is here is that they don't as it'd be easier to just 
include all the headers (I think?).


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D135634/new/

https://reviews.llvm.org/D135634

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D136651: [Clang] Give Clang the ability to use a shared stat cache

[Ben Barham via Phabricator via cfe-commits]

bnbarham added a comment.

Mostly just skimmed so far, will hopefully have some time to look at this more 
tomorrow. Out of interest, do you have any performance numbers using this 
change? I assume this mainly impacts implicit modules (since I suspect we'd 
also be opening the file as well anyway), is that true?




Comment at: clang/lib/Frontend/CompilerInvocation.cpp:5115-5130
+  for (const auto &File : CI.getHeaderSearchOpts().VFSStatCacheFiles) {
+llvm::ErrorOr> Buffer =
+llvm::MemoryBuffer::getFile(File);
+if (!Buffer) {
+  Diags.Report(diag::err_missing_vfs_stat_cache_file) << File;
+  continue;
+}

IMO VFS overlays being modeled like this is a mistake and makes 
reasoning/testing about them fairly difficult. I have a PR up 
https://reviews.llvm.org/D121423 to change `OverlayFileSystem` to make more 
sense and be a real overlay rather than what it is now. If I finish that one 
off, how would you feel about changing the behavior of `StatCacheFileSystem` to 
just immediately error if it doesn't contain the file, rather than proxy (and 
thus chain) as it does now?

So for multiple files we'd then have:
  - OverlayFileSystem
- StatCacheFileSystem
- StatCacheFileSystem
- RealFileSystem

Then any non-stat or exists call would return 
`llvm::errc::no_such_file_or_directory` and then the next FS would be used 
instead.

I don't think this *really* matters for `StatCacheFileSystem`, so I'm fine if 
you'd rather not wait for me to change `OverlayFileSystem`. I can make the 
changes myself after getting my PR in.



Comment at: clang/tools/clang-stat-cache/clang-stat-cache.cpp:143
+// There is no cross-platform way to implement a validity check. If this
+// platofrm doesn't support it, just consider the cache constents always
+// valid. When that's the case, the tool running cache generation needs





Comment at: llvm/lib/Support/VirtualFileSystem.cpp:2958
+// cache prefix. It could be that the file doesn't exist, or the spelling
+// the pathis different. The canonicalization that the call to 
remove_dots()
+// does leaves only '..' with symlinks as a source of confusion. If the 
path





Comment at: llvm/lib/Support/VirtualFileSystem.cpp:2959-2960
+// the pathis different. The canonicalization that the call to 
remove_dots()
+// does leaves only '..' with symlinks as a source of confusion. If the 
path
+// does not contain '..' we can safely say it doesn't exist.
+if (std::find(sys::path::begin(SuffixPath), sys::path::end(SuffixPath),

This sentence is a little confusing to me. `remove_dots` just leaves `..` 
unless you pass `remove_dot_dot` (but it doesn't do any checking). IMO just the 
`If the path does not contain '..' we can safely say it doesn't exist.` is 
enough.



Comment at: llvm/lib/Support/VirtualFileSystem.cpp:2961
+// does not contain '..' we can safely say it doesn't exist.
+if (std::find(sys::path::begin(SuffixPath), sys::path::end(SuffixPath),
+  "..") == sys::path::end(SuffixPath)) {

FWIW `StringRef` has a `contains`


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D136651/new/

https://reviews.llvm.org/D136651

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D136651: [Clang] Give Clang the ability to use a shared stat cache

[Ben Barham via Phabricator via cfe-commits]

bnbarham added a comment.

> You're correct that this overhead has been measured on implicit module 
> builds. As I mentioned in the commit message this saves over 20% of the 
> overall built time in some cases. This time is split between module 
> validation (which could be skipped) and HeaderSearch (which cannot be 
> skipped).

Heh, I read the commit as saying we've seen a 20% overhead, but not that this 
completely removed that. Not sure why I didn't make that connection in my head, 
it's a reasonable implication. Very nice!




Comment at: clang/lib/Frontend/CompilerInvocation.cpp:5115-5130
+  for (const auto &File : CI.getHeaderSearchOpts().VFSStatCacheFiles) {
+llvm::ErrorOr> Buffer =
+llvm::MemoryBuffer::getFile(File);
+if (!Buffer) {
+  Diags.Report(diag::err_missing_vfs_stat_cache_file) << File;
+  continue;
+}

friss wrote:
> bnbarham wrote:
> > IMO VFS overlays being modeled like this is a mistake and makes 
> > reasoning/testing about them fairly difficult. I have a PR up 
> > https://reviews.llvm.org/D121423 to change `OverlayFileSystem` to make more 
> > sense and be a real overlay rather than what it is now. If I finish that 
> > one off, how would you feel about changing the behavior of 
> > `StatCacheFileSystem` to just immediately error if it doesn't contain the 
> > file, rather than proxy (and thus chain) as it does now?
> > 
> > So for multiple files we'd then have:
> >   - OverlayFileSystem
> > - StatCacheFileSystem
> > - StatCacheFileSystem
> > - RealFileSystem
> > 
> > Then any non-stat or exists call would return 
> > `llvm::errc::no_such_file_or_directory` and then the next FS would be used 
> > instead.
> > 
> > I don't think this *really* matters for `StatCacheFileSystem`, so I'm fine 
> > if you'd rather not wait for me to change `OverlayFileSystem`. I can make 
> > the changes myself after getting my PR in.
> I don't think that's really doable if you want to keep the ability to cache 
> negative hits. If a miss always results in a query to the real filesystem, 
> then you're not saving the `stat` call. A lot of the time is spent in 
> HeaderSearch which queries a similar number of non-existing and existing 
> files.
> But I'm not dead set on this. I also haven't spent a lot of time thinking 
> about your proposal.
That's a great point. It'd be easy enough to avoid by eg. passing down an error 
to use for either "this is a path I handle and it doesn't exist" or "this is 
not a path I handle" and handling that in `OverlayFileSystem`, which could 
default to `llvm:errc::no_such_file_or_directory`. Generally I just think this 
is easier to reason about, but it's more  problem for `RedirectingFileSystem` 
then it is for this (since this isn't *really* an overlay at all).

This is definitely fine as is either way though. If I find 
`RedirectingFileSystem` needs the same sort of idea then we can revisit it then.



Comment at: llvm/lib/Support/VirtualFileSystem.cpp:2959-2960
+// the pathis different. The canonicalization that the call to 
remove_dots()
+// does leaves only '..' with symlinks as a source of confusion. If the 
path
+// does not contain '..' we can safely say it doesn't exist.
+if (std::find(sys::path::begin(SuffixPath), sys::path::end(SuffixPath),

friss wrote:
> bnbarham wrote:
> > This sentence is a little confusing to me. `remove_dots` just leaves `..` 
> > unless you pass `remove_dot_dot` (but it doesn't do any checking). IMO just 
> > the `If the path does not contain '..' we can safely say it doesn't exist.` 
> > is enough.
> `remove_dots` does more than remove dots. It is confusing, but it also 
> removes excess separators (see the `Canonical` unit test). This means that 
> the cache will work for /path/to/cache/file/a as well as 
> /path/to/cache/file///a and /path/to/cache/file/././a. There are basically 
> infinite spellings of a path just by adding `.` and additional separators.
> `..` is interesting because it's not semantically preserving to remove them 
> in the presence of symlinks.
> I'm fine with simplifying the description, but that is the nuance I tried to 
> convey.
I suppose it could be confusing if you didn't know what `remove_dots` did, but 
in that case I'd prefer something like:
> `remove_dots` canonicalizes the path by removing `.` and excess separators, 
> but leaves `..` since it isn't semantically preserving to remove them.



Comment at: llvm/lib/Support/VirtualFileSystem.cpp:2961
+// does not contain '..' we can safely say it doesn't exist.
+if (std::find(sys::path::begin(SuffixPath), sys::path::end(SuffixPath),
+  "..") == sys::path::end(SuffixPath)) {

friss wrote:
> bnbarham wrote:
> > FWIW `StringRef` has a `contains`
> But that wouldn't be correct. Here we are looking for a path component which 
> is `..`. A simple text search would fire on a filename containi

[PATCH] D136844: [libclang] Expose completion result kind in `CXCompletionResult`

[Ben Barham via Phabricator via cfe-commits]

bnbarham accepted this revision.
bnbarham added a comment.
This revision is now accepted and ready to land.

Thanks Egor!


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D136844/new/

https://reviews.llvm.org/D136844

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D134249: [modules] Fix error "malformed or corrupted AST file: 'SourceLocation remap refers to unknown module...'".

[Ben Barham via Phabricator via cfe-commits]

bnbarham accepted this revision.
bnbarham added a comment.

LGTM.

> When we try to load such outdated .pcm file, we shouldn't change any already 
> loaded and processed modules.

Or to put another way - we can't remove these modules because they may already 
have references... anywhere.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D134249/new/

https://reviews.llvm.org/D134249

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D135220: [clang] Update ModuleMap::getModuleMapFile* to use FileEntryRef

[Ben Barham via Phabricator via cfe-commits]

bnbarham accepted this revision.
bnbarham added inline comments.



Comment at: clang/include/clang/Basic/SourceManager.h:146-155
   /// References the file which the contents were actually loaded from.
   ///
   /// Can be different from 'Entry' if we overridden the contents of one file
   /// with the contents of another file.
   const FileEntry *ContentsEntry;
 
   /// The filename that is used to access OrigEntry.

I haven't checked all uses, but do we still need all of 
`OrigEntry`/`ContentsEntry`/`Filename`? Seems like a single `FileEntryRef` 
should encapsulate all the information we need there.



Comment at: clang/include/clang/Basic/SourceManager.h:1062
+Optional F = sloc.getFile().getContentCache().OrigEntry;
+return F ? &F->getFileEntry() : nullptr;
   }

jansvoboda11 wrote:
> Could you wrap `F` in `OptionalFileEntryRefDegradesToFileEntryPtr` to handle 
> the conversion for you?
`OrigEntry` is a `OptionalFileEntryRefDegradesToFileEntryPtr`, so this should 
just be able to be `return slot.getFile().getContentCache().OrigEntry`?



Comment at: clang/lib/Lex/ModuleMap.cpp:1030
+Optional ModuleMapRef = getModuleMapFileForUniquing(Parent);
+ModuleMapFile = ModuleMapRef ? *ModuleMapRef : nullptr;
+  }

jansvoboda11 wrote:
> Nit: Use `OptionalFileEntryRefDegradesToFileEntryPtr`? Maybe we should return 
> that from `getModuleMapFileForUniquing` in the first place.
I like the idea of returning `OptionalFileEntryRefDegradesToFileEntryPtr` until 
we move all this to use `FileEntryRef` as well.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D135220/new/

https://reviews.llvm.org/D135220

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D115232: [clangd] Indexing of standard library

[Ben Barham via Phabricator via cfe-commits]

bnbarham added inline comments.



Comment at: clang-tools-extra/clangd/ClangdServer.cpp:1008-1010
+  // TUScheduler is the only thing that starts background indexing work.
+  if (IndexTasks && !IndexTasks->wait(timeoutSeconds(TimeoutSeconds)))
+return false;

@sammccall shouldn't we also be waiting for this to finish when `ClangdServer` 
is destroyed? IIUC right now the both `FileIndex` itself (stored in 
`ClangdServer`) and the actual `UpdateIndexCallbacks` (stored in `TUScheduler`) 
can be freed while `indexStdlib` is running asynchronously, resulting in a 
use-after-free on eg. `FIndex->updatePreamble(std::move(IF))`. I was confused 
as to why this wasn't happening in the tests, but these lines would explain it 
😅 

Adding a `IndexTasks->wait()` to `~ClangdServer` fixes the crash I'm seeing in 
the sourcekit-lsp tests (see https://github.com/apple/llvm-project/pull/5837), 
though ideally we (sourcekit-lsp) wouldn't be running any indexing at all. As 
far as I can tell there's no way to turn off dynamic indexing now though, 
except for `StandardLibrary` indexing through the config file (but not from 
clangd args)?


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D115232/new/

https://reviews.llvm.org/D115232

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D115232: [clangd] Indexing of standard library

[Ben Barham via Phabricator via cfe-commits]

bnbarham added inline comments.



Comment at: clang-tools-extra/clangd/ClangdServer.cpp:1008-1010
+  // TUScheduler is the only thing that starts background indexing work.
+  if (IndexTasks && !IndexTasks->wait(timeoutSeconds(TimeoutSeconds)))
+return false;

sammccall wrote:
> bnbarham wrote:
> > @sammccall shouldn't we also be waiting for this to finish when 
> > `ClangdServer` is destroyed? IIUC right now the both `FileIndex` itself 
> > (stored in `ClangdServer`) and the actual `UpdateIndexCallbacks` (stored in 
> > `TUScheduler`) can be freed while `indexStdlib` is running asynchronously, 
> > resulting in a use-after-free on eg. 
> > `FIndex->updatePreamble(std::move(IF))`. I was confused as to why this 
> > wasn't happening in the tests, but these lines would explain it 😅 
> > 
> > Adding a `IndexTasks->wait()` to `~ClangdServer` fixes the crash I'm seeing 
> > in the sourcekit-lsp tests (see 
> > https://github.com/apple/llvm-project/pull/5837), though ideally we 
> > (sourcekit-lsp) wouldn't be running any indexing at all. As far as I can 
> > tell there's no way to turn off dynamic indexing now though, except for 
> > `StandardLibrary` indexing through the config file (but not from clangd 
> > args)?
> Thanks for flagging this!
> 
> We *almost* have the sequencing we need in ~ClangdServer:
>  - when we fall off the end of ~ClangdServer it destroys all its members
>  - `ClangdServer::IndexTasks` is declared after `FIndex`, so is destroyed 
> first
>  - ~AsyncTaskRunner calls `wait()`
> 
> But the task we schedule on `IndexTasks` captures a ref to 
> `UpdateIndexCallbacks`, which is owned by the `TUScheduler`, which we 
> explicitly destroy at the beginning of `~ClangdServer`.
> 
> However I think your patch is *also* not quite correct: we can wait for the 
> tasks to be empty, but then the TUScheduler could fill it up again before we 
> destroy TUScheduler.
> 
> Options include adding an explicit stop() to TUScheduler, changing 
> TUScheduler to not (exclusively) own UpdateIndexCallbacks, or have the task 
> not capture the callbacks by reference.
> I'll try the latter first, which seems least invasive.
> 
> ---
> 
> > ideally we (sourcekit-lsp) wouldn't be running any indexing at all. As far 
> > as I can tell there's no way to turn off dynamic indexing now though, 
> > except for StandardLibrary indexing through the config file (but not from 
> > clangd args)?
> 
> Clangd won't provide any top-level/namespace-level completions at all without 
> dynamic index (index of preambles), and various other features won't work 
> (docs on hover, include-fixer, type/call-hierarchy). We dropped support for 
> disabling this at some point, as it didn't really seem usable and made 
> features more complex if we tried to accommodate it. At a technical level it 
> would be possible to disable I think, but I'd be really surprised if 
> completion worked well, or if a language server without completion was useful.
> 
> > `StandardLibrary` indexing through the config file (but not from clangd 
> > args)
> 
> We've tried to move away from flags for options that are interesting to 
> users, as config files are more flexible, more forgiving on errors, and allow 
> different settings per-project in a consistent way. (We don't own the 
> editors, so cross-editor consistency is important to being able to support 
> users at all...)
> 
> I can see how requiring config to be materialized on disk could be 
> inconvenient for IDEs though. I think we could add a general-purpose 
> `--config-inline=` flag, and/or the ability to set 
> config over LSP (this can be dynamic, accordingly bigger design space that 
> might be hard to get right).
Ah, I didn't actually check `AsyncTaskRunner`. Makes sense it would wait though 
:). Thanks for looking into this in detail!

> or have the task not capture the callbacks by reference. I'll try the latter 
> first, which seems least invasive.

This + moving `FIndex` after `IndexTasks` seems reasonable to me.

> Clangd won't provide any top-level/namespace-level completions at all without 
> dynamic index (index of preambles), and various other features won't work 
> (docs on hover, include-fixer, type/call-hierarchy).

That's good to know - I assume this extends to indexing the stdlib as well, ie. 
the stdlib would be missing from top/namespace level completion if not indexed? 
Does the dynamic index grow with every opened file, or is it just the currently 
opened file? If disabling breaks everything it's not something we'd want to do, 
we just don't need it for find refs/etc.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D115232/new/

https://reviews.llvm.org/D115232

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D115232: [clangd] Indexing of standard library

[Ben Barham via Phabricator via cfe-commits]

bnbarham added inline comments.



Comment at: clang-tools-extra/clangd/ClangdServer.cpp:1008-1010
+  // TUScheduler is the only thing that starts background indexing work.
+  if (IndexTasks && !IndexTasks->wait(timeoutSeconds(TimeoutSeconds)))
+return false;

sammccall wrote:
> bnbarham wrote:
> > sammccall wrote:
> > > bnbarham wrote:
> > > > @sammccall shouldn't we also be waiting for this to finish when 
> > > > `ClangdServer` is destroyed? IIUC right now the both `FileIndex` itself 
> > > > (stored in `ClangdServer`) and the actual `UpdateIndexCallbacks` 
> > > > (stored in `TUScheduler`) can be freed while `indexStdlib` is running 
> > > > asynchronously, resulting in a use-after-free on eg. 
> > > > `FIndex->updatePreamble(std::move(IF))`. I was confused as to why this 
> > > > wasn't happening in the tests, but these lines would explain it 😅 
> > > > 
> > > > Adding a `IndexTasks->wait()` to `~ClangdServer` fixes the crash I'm 
> > > > seeing in the sourcekit-lsp tests (see 
> > > > https://github.com/apple/llvm-project/pull/5837), though ideally we 
> > > > (sourcekit-lsp) wouldn't be running any indexing at all. As far as I 
> > > > can tell there's no way to turn off dynamic indexing now though, except 
> > > > for `StandardLibrary` indexing through the config file (but not from 
> > > > clangd args)?
> > > Thanks for flagging this!
> > > 
> > > We *almost* have the sequencing we need in ~ClangdServer:
> > >  - when we fall off the end of ~ClangdServer it destroys all its members
> > >  - `ClangdServer::IndexTasks` is declared after `FIndex`, so is destroyed 
> > > first
> > >  - ~AsyncTaskRunner calls `wait()`
> > > 
> > > But the task we schedule on `IndexTasks` captures a ref to 
> > > `UpdateIndexCallbacks`, which is owned by the `TUScheduler`, which we 
> > > explicitly destroy at the beginning of `~ClangdServer`.
> > > 
> > > However I think your patch is *also* not quite correct: we can wait for 
> > > the tasks to be empty, but then the TUScheduler could fill it up again 
> > > before we destroy TUScheduler.
> > > 
> > > Options include adding an explicit stop() to TUScheduler, changing 
> > > TUScheduler to not (exclusively) own UpdateIndexCallbacks, or have the 
> > > task not capture the callbacks by reference.
> > > I'll try the latter first, which seems least invasive.
> > > 
> > > ---
> > > 
> > > > ideally we (sourcekit-lsp) wouldn't be running any indexing at all. As 
> > > > far as I can tell there's no way to turn off dynamic indexing now 
> > > > though, except for StandardLibrary indexing through the config file 
> > > > (but not from clangd args)?
> > > 
> > > Clangd won't provide any top-level/namespace-level completions at all 
> > > without dynamic index (index of preambles), and various other features 
> > > won't work (docs on hover, include-fixer, type/call-hierarchy). We 
> > > dropped support for disabling this at some point, as it didn't really 
> > > seem usable and made features more complex if we tried to accommodate it. 
> > > At a technical level it would be possible to disable I think, but I'd be 
> > > really surprised if completion worked well, or if a language server 
> > > without completion was useful.
> > > 
> > > > `StandardLibrary` indexing through the config file (but not from clangd 
> > > > args)
> > > 
> > > We've tried to move away from flags for options that are interesting to 
> > > users, as config files are more flexible, more forgiving on errors, and 
> > > allow different settings per-project in a consistent way. (We don't own 
> > > the editors, so cross-editor consistency is important to being able to 
> > > support users at all...)
> > > 
> > > I can see how requiring config to be materialized on disk could be 
> > > inconvenient for IDEs though. I think we could add a general-purpose 
> > > `--config-inline=` flag, and/or the ability to set 
> > > config over LSP (this can be dynamic, accordingly bigger design space 
> > > that might be hard to get right).
> > Ah, I didn't actually check `AsyncTaskRunner`. Makes sense it would wait 
> > though :). Thanks for looking into this in detail!
> > 
> > > or have the task not capture the callbacks by reference. I'll try the 
> > > latter first, which seems least invasive.
> > 
> > This + moving `FIndex` after `IndexTasks` seems reasonable to me.
> > 
> > > Clangd won't provide any top-level/namespace-level completions at all 
> > > without dynamic index (index of preambles), and various other features 
> > > won't work (docs on hover, include-fixer, type/call-hierarchy).
> > 
> > That's good to know - I assume this extends to indexing the stdlib as well, 
> > ie. the stdlib would be missing from top/namespace level completion if not 
> > indexed? Does the dynamic index grow with every opened file, or is it just 
> > the currently opened file? If disabling breaks everything it's not 
> > something we'd want to do, we just don't need it for find refs/etc.
> > Th

[PATCH] D141324: [clang] extend external_source_symbol attribute with the USR clause

[Ben Barham via Phabricator via cfe-commits]

bnbarham accepted this revision.
bnbarham added a comment.
This revision is now accepted and ready to land.

LGTM, thanks Alex!


CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D141324/new/

https://reviews.llvm.org/D141324

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D137304: [clang] Store filename per include instead of mutating filename

[Ben Barham via Phabricator via cfe-commits]

bnbarham added inline comments.



Comment at: clang/include/clang/Basic/SourceManager.h:147-163
   /// Indicates whether the buffer itself was provided to override
   /// the actual file contents.
   ///
   /// When true, the original entry may be a virtual file that does not
   /// exist.
   unsigned BufferOverridden : 1;
 

benlangmuir wrote:
> dexonsmith wrote:
> > Have you thought about whether it makes sense for these fields to be shared 
> > for all `FileEntryRef`s to the same `FileEntry`?  (Maybe it does! just 
> > checking)
> Yes, I think this split actually makes the behaviour clearer. The remaining 
> fields are related to the contents, while the extracted fields are for the 
> name.  You only need one SourceLineCache, etc. for one set of file contents.
Makes sense to me. Might be worth adding a comment for that when you're done + 
fixing up the comment on `ContentsEntry` since it still references `Entry` 
(which I assume is what `OrigEntry` used to be called?).



Comment at: clang/include/clang/Basic/SourceManager.h:314
   /// Return a FileInfo object.
-  static FileInfo get(SourceLocation IL, ContentCache &Con,
-  CharacteristicKind FileCharacter, StringRef Filename) {
+  static FileInfo get(SourceLocation IL, NamedContentCache &Con,
+  CharacteristicKind FileCharacter) {

I believe `NamedContentCache` can be `const` here now that we're not setting 
its filename.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D137304/new/

https://reviews.llvm.org/D137304

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D137473: [vfs] Allow root paths relative to the directory of the vfsoverlay YAML file

[Ben Barham via Phabricator via cfe-commits]

bnbarham added a subscriber: dexonsmith.
bnbarham added a comment.

This seems reasonable to me in general. @dexonsmith in case you have any 
thoughts.




Comment at: clang/test/VFS/Inputs/root-relative-overlay.yaml:4
+  'case-sensitive': false,
+  'overlay-relative': true,
+  'root-relative': 'yaml-dir',

I'd prefer a test without `overlay-relative` set to make it clear they don't 
depend on each other.



Comment at: llvm/include/llvm/Support/VirtualFileSystem.h:660
 ///   'use-external-names': 
+///   'root-relative': 
 ///   'overlay-relative': 

phosek wrote:
> Could we make this just a boolean akin to `overlay-relative` since there are 
> only two options (default to `false`)?
I personally prefer being explicit here, `overlay-relative` is fairly confusing 
as it is.

`overlay-relative` isn't about allowing relative paths, but instead means that 
*all* external paths should be prefixed with the directory of the overlay. To 
put another way, external paths can be relative whether this is true/false, 
`overlay-relative` just *always* prepends the overlay path.

Could you add a comment to make it clear that this has no interaction with 
`overlay-relative`? If you want to add a comment to `overlay-relative` with 
something like the above that would also be appreciated :)



Comment at: llvm/include/llvm/Support/VirtualFileSystem.h:752
+  enum class RootRelativeKind {
+/// The roots are relative to the current working directory.
+CWD,

`to the current working directory when the overlay is created.` is maybe a 
little clearer to me



Comment at: llvm/include/llvm/Support/VirtualFileSystem.h:755
+/// The roots are relative to the directory where the YAML file locates.
+YAMLDir
+  };

Any thoughts on something like `OverlayDir` instead?



Comment at: llvm/include/llvm/Support/VirtualFileSystem.h:926
+  /// is set. This will also be prefixed to each 'roots->name' if RootRelative
+  /// is set to RootRelativeKind::YAMLDir.
+  std::string YAMLFileDir;

*and the path is relative*



Comment at: llvm/lib/Support/VirtualFileSystem.cpp:1903
+assert(!FullPath.empty() && "YAML file directory must exist");
+sys::fs::make_absolute(FS->getYAMLFileDir(), Name);
+Name = canonicalize(Name);

IMO both this and CWD should be using the base FS instead. VFS didn't have a 
CWD previously, but now that it does it doesn't really make sense to use the 
process wide CWD. Especially since `-working-directory` doesn't change it.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D137473/new/

https://reviews.llvm.org/D137473

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D137473: [vfs] Allow root paths relative to the directory of the vfsoverlay YAML file

[Ben Barham via Phabricator via cfe-commits]

bnbarham added inline comments.



Comment at: clang/test/VFS/Inputs/root-relative-overlay.yaml:4
+  'case-sensitive': false,
+  'overlay-relative': true,
+  'root-relative': 'yaml-dir',

bnbarham wrote:
> I'd prefer a test without `overlay-relative` set to make it clear they don't 
> depend on each other.
There's also unit tests in `llvm/unittests/Support/VirtualFileSystemTest.cpp` 
that you could add to.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D137473/new/

https://reviews.llvm.org/D137473

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits