zturner created this revision.
zturner added a reviewer: cfe-commits.
zturner added subscribers: klimek, alexfh, djasper.
Allow explicit specification of a compilation database file and source root.
While trying to create a compilation database test for D23455, I ran into the
problem that compilation database tests require a command shell, evidenced by
the fact that at the top of `clang-tidy-run-with-database-cpp` contains the
line `REQUIRES: SHELL`. This makes it impossible to write the test that was
requested in D23455 (not to mention it means the existing test is not running
on Windows either).
I solve this by introducing a new command line parameter and slightly enhancing
the functionality of an existing one, detailed below:
1. There was no way to specify an arbitrary compilation database. At best you
could specify the directory of a compilation database, but it would still be
forced to look for `compile_commands.json`. This is an inflexible design when
you want to have multiple tests which each use their own compilation database.
You could put them in different folders and call each one
`compile_commands.json`, but that leads to many different directories with only
one file, which is kind of silly.
The `-p` option let you specify a build path, which would be the folder where a
compilation database existed, but it did not let you specify a *specific file*
to use as a compilation database. I improved expanded this behavior of this
option to check if the specified path is a regular file or a directory. If it
is a regular file it doesn't search for a compilation database, and instead
just uses the file you specify. The nice thing about this is that now we can
put many compilation databases for various tests in the same directory, and
have the run line of the test reference the exact compilation database needed
for the test.
2. Instead of a "real" compilation database, the test consisted of a template.
And the test used shell in order to invoke `sed` to modify the template so that
the files and directories would be correct. So the test would output the
modified template into the CMake output directory, and then it had to use more
shell commands to copy the source files over to the CMake output directory so
that they would be in the right hierarchy in order to be found by the relative
paths written to the compilation database.
In order to remove the dependency on shell, we need a way to reference the
source files in their original location in the source tree, and not require
them to be copied to the output tree. To do this I introduced a new command
line option `-r` that lets you specify the root at which relative paths are
resolved against.
These two things combined eliminate the need to modify the compilation database
with `sed` and write it to the output folder because we can write *actual*
source code on disk and check it in for the tests, and the compilation database
can refer to these files using actual relative paths. Then because of #1, the
run line can point directly to the compilation database in question.
I re-wrote the compilation database test to use this new method, and it now no
longer depends on shell. I will upload the re-written test in a separate patch
since I can't do cross-repo diffs.
https://reviews.llvm.org/D23532
Files:
include/clang/Tooling/CompilationDatabase.h
include/clang/Tooling/JSONCompilationDatabase.h
lib/Tooling/CommonOptionsParser.cpp
lib/Tooling/CompilationDatabase.cpp
lib/Tooling/JSONCompilationDatabase.cpp
lib/Tooling/Tooling.cpp
Index: lib/Tooling/Tooling.cpp
===================================================================
--- lib/Tooling/Tooling.cpp
+++ lib/Tooling/Tooling.cpp
@@ -407,10 +407,12 @@
// difference for example on network filesystems, where symlinks might be
// switched during runtime of the tool. Fixing this depends on having a
// file system abstraction that allows openat() style interactions.
- if (OverlayFileSystem->setCurrentWorkingDirectory(
- CompileCommand.Directory))
- llvm::report_fatal_error("Cannot chdir into \"" +
- Twine(CompileCommand.Directory) + "\n!");
+ if (!CompileCommand.Directory.empty()) {
+ if (OverlayFileSystem->setCurrentWorkingDirectory(
+ CompileCommand.Directory))
+ llvm::report_fatal_error("Cannot chdir into \"" +
+ Twine(CompileCommand.Directory) + "\n!");
+ }
// Now fill the in-memory VFS with the relative file mappings so it will
// have the correct relative paths. We never remove mappings but that
Index: lib/Tooling/JSONCompilationDatabase.cpp
===================================================================
--- lib/Tooling/JSONCompilationDatabase.cpp
+++ lib/Tooling/JSONCompilationDatabase.cpp
@@ -16,10 +16,12 @@
#include "clang/Tooling/CompilationDatabasePluginRegistry.h"
#include "clang/Tooling/Tooling.h"
#include "llvm/ADT/SmallString.h"
+#include "llvm/ADT/Triple.h"
#include "llvm/Support/Allocator.h"
#include "llvm/Support/CommandLine.h"
#include "llvm/Support/Path.h"
#include "llvm/Support/StringSaver.h"
+#include <direct.h>
#include <system_error>
namespace clang {
@@ -116,26 +118,34 @@
std::vector<std::string> unescapeCommandLine(
StringRef EscapedCommandLine) {
-#if defined(LLVM_ON_WIN32)
- llvm::BumpPtrAllocator Alloc;
- llvm::StringSaver Saver(Alloc);
- llvm::SmallVector<const char *, 64> T;
- llvm::cl::TokenizeWindowsCommandLine(EscapedCommandLine, Saver, T);
- std::vector<std::string> Result(T.begin(), T.end());
- return Result;
-#else
- CommandLineArgumentParser parser(EscapedCommandLine);
- return parser.parse();
-#endif
+ llvm::Triple Triple(llvm::sys::getDefaultTargetTriple());
+ if (Triple.getOS() == llvm::Triple::OSType::Win32) {
+ llvm::BumpPtrAllocator Alloc;
+ llvm::StringSaver Saver(Alloc);
+ llvm::SmallVector<const char *, 64> T;
+ llvm::cl::TokenizeWindowsCommandLine(EscapedCommandLine, Saver, T);
+ std::vector<std::string> Result(T.begin(), T.end());
+ return Result;
+ } else {
+ CommandLineArgumentParser parser(EscapedCommandLine);
+ return parser.parse();
+ }
}
class JSONCompilationDatabasePlugin : public CompilationDatabasePlugin {
std::unique_ptr<CompilationDatabase>
- loadFromDirectory(StringRef Directory, std::string &ErrorMessage) override {
+ loadFromDirectory(StringRef Directory, StringRef SourceRoot,
+ std::string &ErrorMessage) override {
SmallString<1024> JSONDatabasePath(Directory);
llvm::sys::path::append(JSONDatabasePath, "compile_commands.json");
+ return loadFromFile(JSONDatabasePath, SourceRoot, ErrorMessage);
+ }
+
+ std::unique_ptr<CompilationDatabase>
+ loadFromFile(StringRef File, StringRef SourceRoot,
+ std::string &ErrorMessage) override {
std::unique_ptr<CompilationDatabase> Database(
- JSONCompilationDatabase::loadFromFile(JSONDatabasePath, ErrorMessage));
+ JSONCompilationDatabase::loadFromFile(File, SourceRoot, ErrorMessage));
if (!Database)
return nullptr;
return Database;
@@ -154,16 +164,16 @@
volatile int JSONAnchorSource = 0;
std::unique_ptr<JSONCompilationDatabase>
-JSONCompilationDatabase::loadFromFile(StringRef FilePath,
+JSONCompilationDatabase::loadFromFile(StringRef FilePath, StringRef SourceRoot,
std::string &ErrorMessage) {
llvm::ErrorOr<std::unique_ptr<llvm::MemoryBuffer>> DatabaseBuffer =
llvm::MemoryBuffer::getFile(FilePath);
if (std::error_code Result = DatabaseBuffer.getError()) {
ErrorMessage = "Error while opening JSON database: " + Result.message();
return nullptr;
}
std::unique_ptr<JSONCompilationDatabase> Database(
- new JSONCompilationDatabase(std::move(*DatabaseBuffer)));
+ new JSONCompilationDatabase(SourceRoot, std::move(*DatabaseBuffer)));
if (!Database->parse(ErrorMessage))
return nullptr;
return Database;
@@ -175,7 +185,7 @@
std::unique_ptr<llvm::MemoryBuffer> DatabaseBuffer(
llvm::MemoryBuffer::getMemBuffer(DatabaseString));
std::unique_ptr<JSONCompilationDatabase> Database(
- new JSONCompilationDatabase(std::move(DatabaseBuffer)));
+ new JSONCompilationDatabase("", std::move(DatabaseBuffer)));
if (!Database->parse(ErrorMessage))
return nullptr;
return Database;
@@ -239,12 +249,13 @@
ArrayRef<CompileCommandRef> CommandsRef,
std::vector<CompileCommand> &Commands) const {
for (int I = 0, E = CommandsRef.size(); I != E; ++I) {
- SmallString<8> DirectoryStorage;
+ SmallString<128> DirectoryStorage;
SmallString<32> FilenameStorage;
+ getDirectoryForCommand(*std::get<0>(CommandsRef[I]), DirectoryStorage);
Commands.emplace_back(
- std::get<0>(CommandsRef[I])->getValue(DirectoryStorage),
- std::get<1>(CommandsRef[I])->getValue(FilenameStorage),
- nodeToCommandLine(std::get<2>(CommandsRef[I])));
+ DirectoryStorage,
+ std::get<1>(CommandsRef[I])->getValue(FilenameStorage),
+ nodeToCommandLine(std::get<2>(CommandsRef[I])));
}
}
@@ -337,21 +348,35 @@
StringRef FileName = File->getValue(FileStorage);
SmallString<128> NativeFilePath;
if (llvm::sys::path::is_relative(FileName)) {
- SmallString<8> DirectoryStorage;
- SmallString<128> AbsolutePath(
- Directory->getValue(DirectoryStorage));
- llvm::sys::path::append(AbsolutePath, FileName);
- llvm::sys::path::native(AbsolutePath, NativeFilePath);
- } else {
- llvm::sys::path::native(FileName, NativeFilePath);
+ SmallString<128> TempStorage(Directory->getValue(NativeFilePath));
+ getDirectoryForCommand(*Directory, TempStorage);
+ llvm::sys::path::append(TempStorage, FileName);
+ FileStorage.swap(TempStorage);
}
+
+ llvm::sys::path::native(FileStorage, NativeFilePath);
auto Cmd = CompileCommandRef(Directory, File, *Command);
IndexByFile[NativeFilePath].push_back(Cmd);
AllCommands.push_back(Cmd);
MatchTrie.insert(NativeFilePath);
}
return true;
}
+void JSONCompilationDatabase::getDirectoryForCommand(
+ llvm::yaml::ScalarNode &JsonDir, llvm::SmallVectorImpl<char> &Dir) const {
+ SmallString<128> DirectoryStorage;
+ Dir.clear();
+ StringRef JsonVal = JsonDir.getValue(DirectoryStorage);
+ Dir.append(JsonVal.begin(), JsonVal.end());
+ if (llvm::sys::path::is_relative(Dir)) {
+ SmallString<128> TempStorage(SourceRoot);
+ llvm::sys::path::append(TempStorage, Dir);
+ Dir.swap(TempStorage);
+ }
+ llvm::sys::fs::make_absolute(Dir);
+ llvm::sys::path::native(Dir);
+}
+
} // end namespace tooling
} // end namespace clang
Index: lib/Tooling/CompilationDatabase.cpp
===================================================================
--- lib/Tooling/CompilationDatabase.cpp
+++ lib/Tooling/CompilationDatabase.cpp
@@ -36,35 +36,54 @@
CompilationDatabase::~CompilationDatabase() {}
-std::unique_ptr<CompilationDatabase>
-CompilationDatabase::loadFromDirectory(StringRef BuildDirectory,
- std::string &ErrorMessage) {
+std::unique_ptr<CompilationDatabase> CompilationDatabase::loadFromDirectory(
+ StringRef BuildDirectory, StringRef SourceRoot, std::string &ErrorMessage) {
std::stringstream ErrorStream;
for (CompilationDatabasePluginRegistry::iterator
It = CompilationDatabasePluginRegistry::begin(),
Ie = CompilationDatabasePluginRegistry::end();
It != Ie; ++It) {
std::string DatabaseErrorMessage;
std::unique_ptr<CompilationDatabasePlugin> Plugin(It->instantiate());
+ if (std::unique_ptr<CompilationDatabase> DB = Plugin->loadFromDirectory(
+ BuildDirectory, SourceRoot, DatabaseErrorMessage))
+ return DB;
+ ErrorStream << It->getName() << ": " << DatabaseErrorMessage << "\n";
+ }
+ ErrorMessage = ErrorStream.str();
+ return nullptr;
+}
+
+std::unique_ptr<CompilationDatabase>
+CompilationDatabase::loadFromFile(StringRef File, StringRef SourceRoot,
+ std::string &ErrorMessage) {
+ std::stringstream ErrorStream;
+ for (CompilationDatabasePluginRegistry::iterator
+ It = CompilationDatabasePluginRegistry::begin(),
+ Ie = CompilationDatabasePluginRegistry::end();
+ It != Ie; ++It) {
+ std::string DatabaseErrorMessage;
+ std::unique_ptr<CompilationDatabasePlugin> Plugin(It->instantiate());
if (std::unique_ptr<CompilationDatabase> DB =
- Plugin->loadFromDirectory(BuildDirectory, DatabaseErrorMessage))
+ Plugin->loadFromFile(File, SourceRoot, DatabaseErrorMessage))
return DB;
ErrorStream << It->getName() << ": " << DatabaseErrorMessage << "\n";
}
ErrorMessage = ErrorStream.str();
return nullptr;
}
static std::unique_ptr<CompilationDatabase>
-findCompilationDatabaseFromDirectory(StringRef Directory,
+findCompilationDatabaseFromDirectory(StringRef Directory, StringRef SourceRoot,
std::string &ErrorMessage) {
std::stringstream ErrorStream;
bool HasErrorMessage = false;
while (!Directory.empty()) {
std::string LoadErrorMessage;
if (std::unique_ptr<CompilationDatabase> DB =
- CompilationDatabase::loadFromDirectory(Directory, LoadErrorMessage))
+ CompilationDatabase::loadFromDirectory(Directory, SourceRoot,
+ LoadErrorMessage))
return DB;
if (!HasErrorMessage) {
@@ -86,21 +105,24 @@
StringRef Directory = llvm::sys::path::parent_path(AbsolutePath);
std::unique_ptr<CompilationDatabase> DB =
- findCompilationDatabaseFromDirectory(Directory, ErrorMessage);
+ findCompilationDatabaseFromDirectory(Directory, "", ErrorMessage);
if (!DB)
ErrorMessage = ("Could not auto-detect compilation database for file \"" +
- SourceFile + "\"\n" + ErrorMessage).str();
+ SourceFile + "\"\n" + ErrorMessage)
+ .str();
return DB;
}
std::unique_ptr<CompilationDatabase>
CompilationDatabase::autoDetectFromDirectory(StringRef SourceDir,
+ StringRef SourceRoot,
std::string &ErrorMessage) {
SmallString<1024> AbsolutePath(getAbsolutePath(SourceDir));
std::unique_ptr<CompilationDatabase> DB =
- findCompilationDatabaseFromDirectory(AbsolutePath, ErrorMessage);
+ findCompilationDatabaseFromDirectory(AbsolutePath, SourceRoot,
+ ErrorMessage);
if (!DB)
ErrorMessage = ("Could not auto-detect compilation database from directory \"" +
Index: lib/Tooling/CommonOptionsParser.cpp
===================================================================
--- lib/Tooling/CommonOptionsParser.cpp
+++ lib/Tooling/CommonOptionsParser.cpp
@@ -36,14 +36,21 @@
"\n"
"-p <build-path> is used to read a compile command database.\n"
"\n"
- "\tFor example, it can be a CMake build directory in which a file named\n"
- "\tcompile_commands.json exists (use -DCMAKE_EXPORT_COMPILE_COMMANDS=ON\n"
- "\tCMake option to get this output). When no build path is specified,\n"
+ "\t<build-path> can be either a file or a directory. If it is a file,\n"
+ "\tit will be treated as a JSON compile command database. If it is a\n"
+ "\tdirectory, the directory will be searched for a file named\n"
+ "\tcompile_commands.json (CMake, for example, generates such a file in\n"
+ "\tthe build output directory when used with the special option\n"
+ "\t-DCMAKE_EXPORT_COMPILE_COMMANDS=ON. When this option is not present,\n"
"\ta search for compile_commands.json will be attempted through all\n"
"\tparent paths of the first input file . See:\n"
"\thttp://clang.llvm.org/docs/HowToSetupToolingForLLVM.html for an\n"
"\texample of setting up Clang Tooling on a source tree.\n"
"\n"
+ "-r <source-root> is used to set the base folder from which to resolve\n"
+ "\trelative paths in the compile command database. By default relative\n"
+ "\tpaths are expected to be found relative to the current working\n"
+ "\tdirectory, as described below."
"<source0> ... specify the paths of source files. These paths are\n"
"\tlooked up in the compile command database. If the path of a file is\n"
"\tabsolute, it needs to point into CMake's source tree. If the path is\n"
@@ -99,7 +106,8 @@
static cl::opt<std::string> BuildPath("p", cl::desc("Build path"),
cl::Optional, cl::cat(Category));
-
+ static cl::opt<std::string> SourceRoot("r", cl::desc("Source Root"),
+ cl::Optional, cl::cat(Category));
static cl::list<std::string> SourcePaths(
cl::Positional, cl::desc("<source0> [... <sourceN>]"), OccurrencesFlag,
cl::cat(Category));
@@ -127,8 +135,16 @@
if (!Compilations) {
std::string ErrorMessage;
if (!BuildPath.empty()) {
- Compilations =
- CompilationDatabase::autoDetectFromDirectory(BuildPath, ErrorMessage);
+ if (llvm::sys::fs::is_regular_file(BuildPath)) {
+ Compilations = CompilationDatabase::loadFromFile(BuildPath, SourceRoot,
+ ErrorMessage);
+ if (!Compilations)
+ ErrorMessage = "Could not load compilation database \"" + BuildPath +
+ "\"\n" + ErrorMessage;
+ } else {
+ Compilations = CompilationDatabase::autoDetectFromDirectory(
+ BuildPath, SourceRoot, ErrorMessage);
+ }
} else {
Compilations = CompilationDatabase::autoDetectFromSource(SourcePaths[0],
ErrorMessage);
Index: include/clang/Tooling/JSONCompilationDatabase.h
===================================================================
--- include/clang/Tooling/JSONCompilationDatabase.h
+++ include/clang/Tooling/JSONCompilationDatabase.h
@@ -62,7 +62,8 @@
/// Returns NULL and sets ErrorMessage if the database could not be
/// loaded from the given file.
static std::unique_ptr<JSONCompilationDatabase>
- loadFromFile(StringRef FilePath, std::string &ErrorMessage);
+ loadFromFile(StringRef FilePath, StringRef SourceRoot,
+ std::string &ErrorMessage);
/// \brief Loads a JSON compilation database from a data buffer.
///
@@ -89,8 +90,10 @@
private:
/// \brief Constructs a JSON compilation database on a memory buffer.
- JSONCompilationDatabase(std::unique_ptr<llvm::MemoryBuffer> Database)
- : Database(std::move(Database)),
+ /// If a source root is given, it is used to resolve relative paths.
+ JSONCompilationDatabase(StringRef SourceRoot,
+ std::unique_ptr<llvm::MemoryBuffer> Database)
+ : Database(std::move(Database)), SourceRoot(SourceRoot),
YAMLStream(this->Database->getBuffer(), SM) {}
/// \brief Parses the database file and creates the index.
@@ -113,6 +116,9 @@
void getCommands(ArrayRef<CompileCommandRef> CommandsRef,
std::vector<CompileCommand> &Commands) const;
+ void getDirectoryForCommand(llvm::yaml::ScalarNode &JsonDir,
+ llvm::SmallVectorImpl<char> &Dir) const;
+
// Maps file paths to the compile command lines for that file.
llvm::StringMap<std::vector<CompileCommandRef>> IndexByFile;
@@ -123,6 +129,7 @@
FileMatchTrie MatchTrie;
std::unique_ptr<llvm::MemoryBuffer> Database;
+ StringRef SourceRoot;
llvm::SourceMgr SM;
llvm::yaml::Stream YAMLStream;
};
Index: include/clang/Tooling/CompilationDatabase.h
===================================================================
--- include/clang/Tooling/CompilationDatabase.h
+++ include/clang/Tooling/CompilationDatabase.h
@@ -91,7 +91,17 @@
/// are named 'compile_commands.json' in the given directory. Extend this
/// for other build types (like ninja build files).
static std::unique_ptr<CompilationDatabase>
- loadFromDirectory(StringRef BuildDirectory, std::string &ErrorMessage);
+ loadFromDirectory(StringRef BuildDirectory, StringRef SourceRoot,
+ std::string &ErrorMessage);
+
+ /// \brief Attempts to load a compilation database from the specified file.
+ ///
+ /// Returns NULL and sets ErrorMessage if the specified compilation database
+ /// does not exist or is in an unrecognized format. Relative paths will
+ /// be resolved starting from 'SourceRoot'. If 'SourceRoot' is empty
+ /// they will be resolved from the current working directory.
+ static std::unique_ptr<CompilationDatabase>
+ loadFromFile(StringRef File, StringRef SourceRoot, std::string &ErrorMessage);
/// \brief Tries to detect a compilation database location and load it.
///
@@ -102,10 +112,13 @@
/// \brief Tries to detect a compilation database location and load it.
///
- /// Looks for a compilation database in directory 'SourceDir' and all
- /// its parent paths by calling loadFromDirectory.
+ /// Looks for a compilation database in directory 'DatabaseDir' and all
+ /// its parent paths by calling loadFromDirectory. Relative paths will
+ /// be resolved starting from 'SourceRoot'. If 'SourceRoot' is empty
+ /// they will be resolved from the current working directory.
static std::unique_ptr<CompilationDatabase>
- autoDetectFromDirectory(StringRef SourceDir, std::string &ErrorMessage);
+ autoDetectFromDirectory(StringRef DatabaseDir, StringRef SourceRoot,
+ std::string &ErrorMessage);
/// \brief Returns all compile commands in which the specified file was
/// compiled.
@@ -149,7 +162,15 @@
///
/// \see CompilationDatabase::loadFromDirectory().
virtual std::unique_ptr<CompilationDatabase>
- loadFromDirectory(StringRef Directory, std::string &ErrorMessage) = 0;
+ loadFromDirectory(StringRef Directory, StringRef SourceRoot,
+ std::string &ErrorMessage) = 0;
+
+ /// \brief Loads a compilation database from the specified file.
+ ///
+ /// \see CompilationDatabase::loadFromFile().
+ virtual std::unique_ptr<CompilationDatabase>
+ loadFromFile(StringRef File, StringRef SourceRoot,
+ std::string &ErrorMessage) = 0;
};
/// \brief A compilation database that returns a single compile command line.
_______________________________________________
cfe-commits mailing list
cfe-commits@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
