marcrasi created this revision.
Herald added subscribers: cfe-commits, jfb, mgorny.
Herald added a project: clang.
marcrasi edited the summary of this revision.
This allows clients to invoke clang without writing anything to disk.
I'm not sure if this is the right design, so I'm sending this patch to
start a discussion about the right way to do this. I'll be happy to make
changes and add tests after talking about the right way to do this!
Motivation:
At Google, we run some Swift source tooling on diskless servers. The
Swift source tooling calls Clang through the ClangImporter
(https://github.com/apple/swift/blob/master/lib/ClangImporter/ClangImporter.cpp).
Clang compiles module to an on-disk module cache, which does not work on
the diskless servers. This patch lets clients ask Clang to write the
module cache to memory.
Current design:
- Adds `class InMemoryOutputFileSystem : public llvm::vfs::FileSystem` that
supports write operations backed by memory. After a file is written, its
contents are accessible through the `llvm::vfs::FileSystem` interface.
- Adds an `InMemoryOutputFileSystem` field to `CompilerInstance`. When this
field is set, the `CompilerInstance` writes to the `InMemoryOutputFileSystem`
instead of the real file system.
Repository:
rG LLVM Github Monorepo
https://reviews.llvm.org/D78058
Files:
clang/include/clang/Basic/InMemoryOutputFileSystem.h
clang/include/clang/Frontend/CompilerInstance.h
clang/lib/Basic/CMakeLists.txt
clang/lib/Basic/InMemoryOutputFileSystem.cpp
clang/lib/Frontend/CompilerInstance.cpp
Index: clang/lib/Frontend/CompilerInstance.cpp
===
--- clang/lib/Frontend/CompilerInstance.cpp
+++ clang/lib/Frontend/CompilerInstance.cpp
@@ -642,6 +642,18 @@
void CompilerInstance::clearOutputFiles(bool EraseFiles) {
for (OutputFile &OF : OutputFiles) {
+if (InMemoryOutputFileSystem) {
+ assert(!OF.TempFilename.empty() &&
+ "InMemoryOutputFileSystem requires using temporary files");
+ if (EraseFiles) {
+InMemoryOutputFileSystem->DeleteTemporaryBuffer(OF.TempFilename);
+ } else {
+InMemoryOutputFileSystem->FinalizeTemporaryBuffer(OF.Filename,
+ OF.TempFilename);
+ }
+ continue;
+}
+
if (!OF.TempFilename.empty()) {
if (EraseFiles) {
llvm::sys::fs::remove(OF.TempFilename);
@@ -728,6 +740,18 @@
OutFile = "-";
}
+ if (InMemoryOutputFileSystem) {
+assert(UseTemporary &&
+ "InMemoryOutputFileSystem requires using temporary files");
+auto stream =
+InMemoryOutputFileSystem->CreateTemporaryBuffer(OutFile, &TempFile);
+if (ResultPathName)
+ *ResultPathName = OutFile;
+if (TempPathName)
+ *TempPathName = TempFile;
+return stream;
+ }
+
std::unique_ptr OS;
std::string OSFile;
@@ -1126,6 +1150,9 @@
ImportingInstance.getDiagnosticClient()),
/*ShouldOwnClient=*/true);
+ Instance.setInMemoryOutputFileSystem(
+ ImportingInstance.getInMemoryOutputFileSystem());
+
// Note that this module is part of the module build stack, so that we
// can detect cycles in the module graph.
Instance.setFileManager(&ImportingInstance.getFileManager());
@@ -1271,6 +1298,32 @@
<< Module->Name << SourceRange(ImportLoc, ModuleNameLoc);
};
+ // If we're writing to an InMemoryOutputFileSystem, then immediately compile
+ // and read the module, rather than doing all the lockfile based locking logic
+ // below, because the InMemoryOutputFileSystem doesn't support lockfiles. This
+ // is okay because the locks are only necessary for performance, not
+ // correctness.
+ if (ImportingInstance.getInMemoryOutputFileSystem()) {
+if (!compileModule(ImportingInstance, ModuleNameLoc, Module,
+ ModuleFileName)) {
+ diagnoseBuildFailure();
+ return false;
+}
+
+// Try to read the module file, now that we've compiled it.
+ASTReader::ASTReadResult ReadResult =
+ImportingInstance.getASTReader()->ReadAST(
+ModuleFileName, serialization::MK_ImplicitModule, ImportLoc,
+ASTReader::ARR_None);
+
+if (ReadResult != ASTReader::Success) {
+ diagnoseBuildFailure();
+ return false;
+}
+
+return true;
+ }
+
// FIXME: have LockFileManager return an error_code so that we can
// avoid the mkdir when the directory already exists.
StringRef Dir = llvm::sys::path::parent_path(ModuleFileName);
Index: clang/lib/Basic/InMemoryOutputFileSystem.cpp
===
--- /dev/null
+++ clang/lib/Basic/InMemoryOutputFileSystem.cpp
@@ -0,0 +1,56 @@
+//=== InMemoryOutputFileSystem.cpp - Collects outputs in memory -*- C++ -*-===//
+//
+// The LLVM Compiler Infrastructure
+//
+// This file is dist
