[PATCH] D131618: [WIP][Do NOT review] LLD related changes for -ffat-lto-objects support

[Arda Unal via Phabricator via cfe-commits]

arda marked an inline comment as done.
arda added inline comments.



Comment at: llvm/lib/Bitcode/Writer/EmbedBitcodePass.cpp:26
+
+PreservedAnalyses EmbedBitcodePass::run(Module &M, ModuleAnalysisManager &AM) {
+  if (M.getGlobalVariable("llvm.embedded.module", true))

arda wrote:
> sfertile wrote:
> > From the discourse discussion:
> > 1) it was suggested that we remove the existing -fembed-bitcode 
> > functionality as Apple has stop supporting it.
> > 2) mentioned that MLGO uses the option to embed the bitcode at various 
> > points in the pipeline depending on if its using LTO our not.
> > 
> > Do we want the pass to be a bit more generic and be able to specify the 
> > global to use for embedding, and the section name to use as arguments? That 
> > way MLGO can keep using the section name it uses now . It also helps 
> > consuming tools to disambiguate between bitcode embedded for lto purpose 
> > from bitcode embedded for other purposes.
> I should have an answer in a couple of days. I will keep you updated.
We have talked to folks at MLGO and reached a decision that their use case is 
sufficiently different from ours that it should be handled separately. More 
detailed discussion here:

[[ https://discourse.llvm.org/t/rfc-ffat-lto-objects-support/63977/15?u=arda | 
[RFC] -ffat-lto-objects support on Discours ]]


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D131618/new/

https://reviews.llvm.org/D131618

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D131618: [WIP][Do NOT review] LLD related changes for -ffat-lto-objects support

[Arda Unal via Phabricator via cfe-commits]

arda updated this revision to Diff 453345.
arda added a comment.
Herald added subscribers: cfe-commits, aheejin.
Herald added a project: clang.

Add clang and llvm related changes


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D131618/new/

https://reviews.llvm.org/D131618

Files:
  clang/include/clang/Basic/CodeGenOptions.def
  clang/include/clang/CodeGen/BackendUtil.h
  clang/include/clang/Driver/Options.td
  clang/include/clang/Driver/ToolChain.h
  clang/lib/CodeGen/BackendUtil.cpp
  clang/lib/CodeGen/CodeGenAction.cpp
  clang/lib/Driver/Driver.cpp
  clang/lib/Driver/ToolChains/Clang.cpp
  lld/ELF/Config.h
  lld/ELF/Driver.cpp
  lld/ELF/InputFiles.cpp
  lld/ELF/Options.td
  lld/test/ELF/fatlto/Inputs/a-fatLTO.o
  lld/test/ELF/fatlto/Inputs/a-thinLTO.bc
  lld/test/ELF/fatlto/Inputs/a.c
  lld/test/ELF/fatlto/Inputs/a.h
  lld/test/ELF/fatlto/Inputs/a.o
  lld/test/ELF/fatlto/Inputs/main-fatLTO.o
  lld/test/ELF/fatlto/Inputs/main-thinLTO.bc
  lld/test/ELF/fatlto/Inputs/main.c
  lld/test/ELF/fatlto/Inputs/main.o
  lld/test/ELF/fatlto/fatlto.test
  llvm/include/llvm/Bitcode/BitcodeWriter.h
  llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
  llvm/lib/Object/ObjectFile.cpp

Index: llvm/lib/Object/ObjectFile.cpp
===
--- llvm/lib/Object/ObjectFile.cpp
+++ llvm/lib/Object/ObjectFile.cpp
@@ -79,7 +79,7 @@
 bool ObjectFile::isSectionBitcode(DataRefImpl Sec) const {
   Expected NameOrErr = getSectionName(Sec);
   if (NameOrErr)
-return *NameOrErr == ".llvmbc";
+return *NameOrErr == ".llvmbc" || *NameOrErr == ".fatlto";
   consumeError(NameOrErr.takeError());
   return false;
 }
Index: llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
===
--- llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
+++ llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
@@ -4953,6 +4953,37 @@
   llvm_unreachable("Unimplemented ObjectFormatType");
 }
 
+static const char *getSectionNameForBitcodeForFatLTO(const Triple &T) {
+  switch (T.getObjectFormat()) {
+  case Triple::MachO:
+llvm::report_fatal_error("MachO is not yet implemented for FatLTO");
+break;
+  case Triple::COFF:
+llvm::report_fatal_error("COFF is not yet implemented for FatLTO");
+break;
+  case Triple::ELF:
+return ".fatlto";
+  case Triple::Wasm:
+llvm::report_fatal_error("Wasm is not yet implemented for FatLTO");
+break;
+  case Triple::UnknownObjectFormat:
+return ".fatlto";
+  case Triple::GOFF:
+llvm::report_fatal_error("GOFF is not yet implemented for FatLTO");
+break;
+  case Triple::SPIRV:
+llvm::report_fatal_error("SPIRV is not yet implemented for FatLTO");
+break;
+  case Triple::XCOFF:
+llvm::report_fatal_error("XCOFF is not yet implemented for FatLTO");
+break;
+  case Triple::DXContainer:
+llvm::report_fatal_error("DXContainer is not yet implemented for FatLTO");
+break;
+  }
+  llvm_unreachable("Unimplemented ObjectFormatType");
+}
+
 void llvm::embedBitcodeInModule(llvm::Module &M, llvm::MemoryBufferRef Buf,
 bool EmbedBitcode, bool EmbedCmdline,
 const std::vector &CmdArgs) {
@@ -5045,3 +5076,68 @@
   llvm::ConstantArray::get(ATy, UsedArray), "llvm.compiler.used");
   NewUsed->setSection("llvm.metadata");
 }
+
+void llvm::embedBitcodeInFatObject(llvm::Module &M, llvm::MemoryBufferRef Buf) {
+  // Save llvm.compiler.used and remove it.
+  SmallVector UsedArray;
+  SmallVector UsedGlobals;
+  Type *UsedElementType = Type::getInt8Ty(M.getContext())->getPointerTo(0);
+  GlobalVariable *Used = collectUsedGlobalVariables(M, UsedGlobals, true);
+  for (auto *GV : UsedGlobals) {
+if (GV->getName() != "llvm.embedded.module" &&
+GV->getName() != "llvm.cmdline")
+  UsedArray.push_back(
+  ConstantExpr::getPointerBitCastOrAddrSpaceCast(GV, UsedElementType));
+  }
+  if (Used)
+Used->eraseFromParent();
+
+  // Embed the bitcode for the llvm module.
+  std::string Data;
+  ArrayRef ModuleData;
+  Triple T(M.getTargetTriple());
+
+  if (Buf.getBufferSize() == 0 ||
+  !isBitcode((const unsigned char *)Buf.getBufferStart(),
+ (const unsigned char *)Buf.getBufferEnd())) {
+// If the input is LLVM Assembly, bitcode is produced by serializing
+// the module. Use-lists order need to be preserved in this case.
+llvm::raw_string_ostream OS(Data);
+llvm::WriteBitcodeToFile(M, OS, /* ShouldPreserveUseListOrder */ true);
+ModuleData =
+ArrayRef((const uint8_t *)OS.str().data(), OS.str().size());
+  } else
+// If the input is LLVM bitcode, write the input byte stream directly.
+ModuleData = ArrayRef((const uint8_t *)Buf.getBufferStart(),
+   Buf.getBufferSize());
+  llvm::Constant *ModuleConstant =
+  llvm::ConstantDataArray::get(M.getContext(), ModuleData);
+  llvm::GlobalVariable *GV = new 

[PATCH] D131618: [WIP][Do NOT review] LLD related changes for -ffat-lto-objects support

[Arda Unal via Phabricator via cfe-commits]

arda updated this revision to Diff 453386.
arda added a comment.

Replace section name from .fatlto to .llvm.lto


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D131618/new/

https://reviews.llvm.org/D131618

Files:
  clang/include/clang/Basic/CodeGenOptions.def
  clang/include/clang/CodeGen/BackendUtil.h
  clang/include/clang/Driver/Options.td
  clang/include/clang/Driver/ToolChain.h
  clang/lib/CodeGen/BackendUtil.cpp
  clang/lib/CodeGen/CodeGenAction.cpp
  clang/lib/Driver/Driver.cpp
  clang/lib/Driver/ToolChains/Clang.cpp
  lld/ELF/Config.h
  lld/ELF/Driver.cpp
  lld/ELF/InputFiles.cpp
  lld/ELF/Options.td
  lld/test/ELF/fatlto/Inputs/a.c
  lld/test/ELF/fatlto/Inputs/a.h
  lld/test/ELF/fatlto/Inputs/main.c
  lld/test/ELF/fatlto/fatlto.test
  llvm/include/llvm/Bitcode/BitcodeWriter.h
  llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
  llvm/lib/Object/ObjectFile.cpp

Index: llvm/lib/Object/ObjectFile.cpp
===
--- llvm/lib/Object/ObjectFile.cpp
+++ llvm/lib/Object/ObjectFile.cpp
@@ -79,7 +79,7 @@
 bool ObjectFile::isSectionBitcode(DataRefImpl Sec) const {
   Expected NameOrErr = getSectionName(Sec);
   if (NameOrErr)
-return *NameOrErr == ".llvmbc";
+return *NameOrErr == ".llvmbc" || *NameOrErr == ".llvm.lto";
   consumeError(NameOrErr.takeError());
   return false;
 }
Index: llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
===
--- llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
+++ llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
@@ -4953,6 +4953,37 @@
   llvm_unreachable("Unimplemented ObjectFormatType");
 }
 
+static const char *getSectionNameForBitcodeForFatLTO(const Triple &T) {
+  switch (T.getObjectFormat()) {
+  case Triple::MachO:
+llvm::report_fatal_error("MachO is not yet implemented for FatLTO");
+break;
+  case Triple::COFF:
+llvm::report_fatal_error("COFF is not yet implemented for FatLTO");
+break;
+  case Triple::ELF:
+return ".llvm.lto";
+  case Triple::Wasm:
+llvm::report_fatal_error("Wasm is not yet implemented for FatLTO");
+break;
+  case Triple::UnknownObjectFormat:
+return ".llvm.lto";
+  case Triple::GOFF:
+llvm::report_fatal_error("GOFF is not yet implemented for FatLTO");
+break;
+  case Triple::SPIRV:
+llvm::report_fatal_error("SPIRV is not yet implemented for FatLTO");
+break;
+  case Triple::XCOFF:
+llvm::report_fatal_error("XCOFF is not yet implemented for FatLTO");
+break;
+  case Triple::DXContainer:
+llvm::report_fatal_error("DXContainer is not yet implemented for FatLTO");
+break;
+  }
+  llvm_unreachable("Unimplemented ObjectFormatType");
+}
+
 void llvm::embedBitcodeInModule(llvm::Module &M, llvm::MemoryBufferRef Buf,
 bool EmbedBitcode, bool EmbedCmdline,
 const std::vector &CmdArgs) {
@@ -5045,3 +5076,68 @@
   llvm::ConstantArray::get(ATy, UsedArray), "llvm.compiler.used");
   NewUsed->setSection("llvm.metadata");
 }
+
+void llvm::embedBitcodeInFatObject(llvm::Module &M, llvm::MemoryBufferRef Buf) {
+  // Save llvm.compiler.used and remove it.
+  SmallVector UsedArray;
+  SmallVector UsedGlobals;
+  Type *UsedElementType = Type::getInt8Ty(M.getContext())->getPointerTo(0);
+  GlobalVariable *Used = collectUsedGlobalVariables(M, UsedGlobals, true);
+  for (auto *GV : UsedGlobals) {
+if (GV->getName() != "llvm.embedded.module" &&
+GV->getName() != "llvm.cmdline")
+  UsedArray.push_back(
+  ConstantExpr::getPointerBitCastOrAddrSpaceCast(GV, UsedElementType));
+  }
+  if (Used)
+Used->eraseFromParent();
+
+  // Embed the bitcode for the llvm module.
+  std::string Data;
+  ArrayRef ModuleData;
+  Triple T(M.getTargetTriple());
+
+  if (Buf.getBufferSize() == 0 ||
+  !isBitcode((const unsigned char *)Buf.getBufferStart(),
+ (const unsigned char *)Buf.getBufferEnd())) {
+// If the input is LLVM Assembly, bitcode is produced by serializing
+// the module. Use-lists order need to be preserved in this case.
+llvm::raw_string_ostream OS(Data);
+llvm::WriteBitcodeToFile(M, OS, /* ShouldPreserveUseListOrder */ true);
+ModuleData =
+ArrayRef((const uint8_t *)OS.str().data(), OS.str().size());
+  } else
+// If the input is LLVM bitcode, write the input byte stream directly.
+ModuleData = ArrayRef((const uint8_t *)Buf.getBufferStart(),
+   Buf.getBufferSize());
+  llvm::Constant *ModuleConstant =
+  llvm::ConstantDataArray::get(M.getContext(), ModuleData);
+  llvm::GlobalVariable *GV = new llvm::GlobalVariable(
+  M, ModuleConstant->getType(), true, llvm::GlobalValue::PrivateLinkage,
+  ModuleConstant);
+  GV->setSection(getSectionNameForBitcodeForFatLTO(T));
+  // Set alignment to 1 to prevent padding between two contributions from input
+  // sections after linking.
+  GV->s

[PATCH] D131618: [WIP][Do NOT review] LLD related changes for -ffat-lto-objects support

[Arda Unal via Phabricator via cfe-commits]

arda updated this revision to Diff 453550.
arda added a comment.

Use obj2yaml and yaml2obj to avoid unreadable object files


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D131618/new/

https://reviews.llvm.org/D131618

Files:
  clang/include/clang/Basic/CodeGenOptions.def
  clang/include/clang/CodeGen/BackendUtil.h
  clang/include/clang/Driver/Options.td
  clang/include/clang/Driver/ToolChain.h
  clang/lib/CodeGen/BackendUtil.cpp
  clang/lib/CodeGen/CodeGenAction.cpp
  clang/lib/Driver/Driver.cpp
  clang/lib/Driver/ToolChains/Clang.cpp
  lld/ELF/Config.h
  lld/ELF/Driver.cpp
  lld/ELF/InputFiles.cpp
  lld/ELF/Options.td
  lld/test/ELF/fatlto/Inputs/a-fatLTO.yaml
  lld/test/ELF/fatlto/Inputs/a-thinLTO.ll
  lld/test/ELF/fatlto/Inputs/a.c
  lld/test/ELF/fatlto/Inputs/a.h
  lld/test/ELF/fatlto/Inputs/a.yaml
  lld/test/ELF/fatlto/Inputs/main-fatLTO.yaml
  lld/test/ELF/fatlto/Inputs/main-thinLTO.ll
  lld/test/ELF/fatlto/Inputs/main.c
  lld/test/ELF/fatlto/Inputs/main.yaml
  lld/test/ELF/fatlto/fatlto.test
  llvm/include/llvm/Bitcode/BitcodeWriter.h
  llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
  llvm/lib/Object/ObjectFile.cpp

Index: llvm/lib/Object/ObjectFile.cpp
===
--- llvm/lib/Object/ObjectFile.cpp
+++ llvm/lib/Object/ObjectFile.cpp
@@ -79,7 +79,7 @@
 bool ObjectFile::isSectionBitcode(DataRefImpl Sec) const {
   Expected NameOrErr = getSectionName(Sec);
   if (NameOrErr)
-return *NameOrErr == ".llvmbc";
+return *NameOrErr == ".llvmbc" || *NameOrErr == ".llvm.lto";
   consumeError(NameOrErr.takeError());
   return false;
 }
Index: llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
===
--- llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
+++ llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
@@ -4953,6 +4953,37 @@
   llvm_unreachable("Unimplemented ObjectFormatType");
 }
 
+static const char *getSectionNameForBitcodeForFatLTO(const Triple &T) {
+  switch (T.getObjectFormat()) {
+  case Triple::MachO:
+llvm::report_fatal_error("MachO is not yet implemented for FatLTO");
+break;
+  case Triple::COFF:
+llvm::report_fatal_error("COFF is not yet implemented for FatLTO");
+break;
+  case Triple::ELF:
+return ".llvm.lto";
+  case Triple::Wasm:
+llvm::report_fatal_error("Wasm is not yet implemented for FatLTO");
+break;
+  case Triple::UnknownObjectFormat:
+return ".llvm.lto";
+  case Triple::GOFF:
+llvm::report_fatal_error("GOFF is not yet implemented for FatLTO");
+break;
+  case Triple::SPIRV:
+llvm::report_fatal_error("SPIRV is not yet implemented for FatLTO");
+break;
+  case Triple::XCOFF:
+llvm::report_fatal_error("XCOFF is not yet implemented for FatLTO");
+break;
+  case Triple::DXContainer:
+llvm::report_fatal_error("DXContainer is not yet implemented for FatLTO");
+break;
+  }
+  llvm_unreachable("Unimplemented ObjectFormatType");
+}
+
 void llvm::embedBitcodeInModule(llvm::Module &M, llvm::MemoryBufferRef Buf,
 bool EmbedBitcode, bool EmbedCmdline,
 const std::vector &CmdArgs) {
@@ -5045,3 +5076,68 @@
   llvm::ConstantArray::get(ATy, UsedArray), "llvm.compiler.used");
   NewUsed->setSection("llvm.metadata");
 }
+
+void llvm::embedBitcodeInFatObject(llvm::Module &M, llvm::MemoryBufferRef Buf) {
+  // Save llvm.compiler.used and remove it.
+  SmallVector UsedArray;
+  SmallVector UsedGlobals;
+  Type *UsedElementType = Type::getInt8Ty(M.getContext())->getPointerTo(0);
+  GlobalVariable *Used = collectUsedGlobalVariables(M, UsedGlobals, true);
+  for (auto *GV : UsedGlobals) {
+if (GV->getName() != "llvm.embedded.module" &&
+GV->getName() != "llvm.cmdline")
+  UsedArray.push_back(
+  ConstantExpr::getPointerBitCastOrAddrSpaceCast(GV, UsedElementType));
+  }
+  if (Used)
+Used->eraseFromParent();
+
+  // Embed the bitcode for the llvm module.
+  std::string Data;
+  ArrayRef ModuleData;
+  Triple T(M.getTargetTriple());
+
+  if (Buf.getBufferSize() == 0 ||
+  !isBitcode((const unsigned char *)Buf.getBufferStart(),
+ (const unsigned char *)Buf.getBufferEnd())) {
+// If the input is LLVM Assembly, bitcode is produced by serializing
+// the module. Use-lists order need to be preserved in this case.
+llvm::raw_string_ostream OS(Data);
+llvm::WriteBitcodeToFile(M, OS, /* ShouldPreserveUseListOrder */ true);
+ModuleData =
+ArrayRef((const uint8_t *)OS.str().data(), OS.str().size());
+  } else
+// If the input is LLVM bitcode, write the input byte stream directly.
+ModuleData = ArrayRef((const uint8_t *)Buf.getBufferStart(),
+   Buf.getBufferSize());
+  llvm::Constant *ModuleConstant =
+  llvm::ConstantDataArray::get(M.getContext(), ModuleData);
+  llvm::GlobalVariable *GV = new llvm::GlobalVariable(
+  M, Modul

[PATCH] D131618: [WIP][Do NOT review] LLD related changes for -ffat-lto-objects support

[Arda Unal via Phabricator via cfe-commits]

arda updated this revision to Diff 453751.
arda added a comment.

Add a clang Driver test


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D131618/new/

https://reviews.llvm.org/D131618

Files:
  clang/include/clang/Basic/CodeGenOptions.def
  clang/include/clang/CodeGen/BackendUtil.h
  clang/include/clang/Driver/Options.td
  clang/include/clang/Driver/ToolChain.h
  clang/lib/CodeGen/BackendUtil.cpp
  clang/lib/CodeGen/CodeGenAction.cpp
  clang/lib/Driver/Driver.cpp
  clang/lib/Driver/ToolChains/Clang.cpp
  clang/test/Driver/fat-lto-objects.c
  lld/ELF/Config.h
  lld/ELF/Driver.cpp
  lld/ELF/InputFiles.cpp
  lld/ELF/Options.td
  lld/test/ELF/fatlto/Inputs/a-fatLTO.yaml
  lld/test/ELF/fatlto/Inputs/a-thinLTO.ll
  lld/test/ELF/fatlto/Inputs/a.c
  lld/test/ELF/fatlto/Inputs/a.h
  lld/test/ELF/fatlto/Inputs/a.yaml
  lld/test/ELF/fatlto/Inputs/main-fatLTO.yaml
  lld/test/ELF/fatlto/Inputs/main-thinLTO.ll
  lld/test/ELF/fatlto/Inputs/main.c
  lld/test/ELF/fatlto/Inputs/main.yaml
  lld/test/ELF/fatlto/fatlto.test
  llvm/include/llvm/Bitcode/BitcodeWriter.h
  llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
  llvm/lib/Object/ObjectFile.cpp

Index: llvm/lib/Object/ObjectFile.cpp
===
--- llvm/lib/Object/ObjectFile.cpp
+++ llvm/lib/Object/ObjectFile.cpp
@@ -79,7 +79,7 @@
 bool ObjectFile::isSectionBitcode(DataRefImpl Sec) const {
   Expected NameOrErr = getSectionName(Sec);
   if (NameOrErr)
-return *NameOrErr == ".llvmbc";
+return *NameOrErr == ".llvmbc" || *NameOrErr == ".llvm.lto";
   consumeError(NameOrErr.takeError());
   return false;
 }
Index: llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
===
--- llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
+++ llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
@@ -4953,6 +4953,37 @@
   llvm_unreachable("Unimplemented ObjectFormatType");
 }
 
+static const char *getSectionNameForBitcodeForFatLTO(const Triple &T) {
+  switch (T.getObjectFormat()) {
+  case Triple::MachO:
+llvm::report_fatal_error("MachO is not yet implemented for FatLTO");
+break;
+  case Triple::COFF:
+llvm::report_fatal_error("COFF is not yet implemented for FatLTO");
+break;
+  case Triple::ELF:
+return ".llvm.lto";
+  case Triple::Wasm:
+llvm::report_fatal_error("Wasm is not yet implemented for FatLTO");
+break;
+  case Triple::UnknownObjectFormat:
+return ".llvm.lto";
+  case Triple::GOFF:
+llvm::report_fatal_error("GOFF is not yet implemented for FatLTO");
+break;
+  case Triple::SPIRV:
+llvm::report_fatal_error("SPIRV is not yet implemented for FatLTO");
+break;
+  case Triple::XCOFF:
+llvm::report_fatal_error("XCOFF is not yet implemented for FatLTO");
+break;
+  case Triple::DXContainer:
+llvm::report_fatal_error("DXContainer is not yet implemented for FatLTO");
+break;
+  }
+  llvm_unreachable("Unimplemented ObjectFormatType");
+}
+
 void llvm::embedBitcodeInModule(llvm::Module &M, llvm::MemoryBufferRef Buf,
 bool EmbedBitcode, bool EmbedCmdline,
 const std::vector &CmdArgs) {
@@ -5045,3 +5076,68 @@
   llvm::ConstantArray::get(ATy, UsedArray), "llvm.compiler.used");
   NewUsed->setSection("llvm.metadata");
 }
+
+void llvm::embedBitcodeInFatObject(llvm::Module &M, llvm::MemoryBufferRef Buf) {
+  // Save llvm.compiler.used and remove it.
+  SmallVector UsedArray;
+  SmallVector UsedGlobals;
+  Type *UsedElementType = Type::getInt8Ty(M.getContext())->getPointerTo(0);
+  GlobalVariable *Used = collectUsedGlobalVariables(M, UsedGlobals, true);
+  for (auto *GV : UsedGlobals) {
+if (GV->getName() != "llvm.embedded.module" &&
+GV->getName() != "llvm.cmdline")
+  UsedArray.push_back(
+  ConstantExpr::getPointerBitCastOrAddrSpaceCast(GV, UsedElementType));
+  }
+  if (Used)
+Used->eraseFromParent();
+
+  // Embed the bitcode for the llvm module.
+  std::string Data;
+  ArrayRef ModuleData;
+  Triple T(M.getTargetTriple());
+
+  if (Buf.getBufferSize() == 0 ||
+  !isBitcode((const unsigned char *)Buf.getBufferStart(),
+ (const unsigned char *)Buf.getBufferEnd())) {
+// If the input is LLVM Assembly, bitcode is produced by serializing
+// the module. Use-lists order need to be preserved in this case.
+llvm::raw_string_ostream OS(Data);
+llvm::WriteBitcodeToFile(M, OS, /* ShouldPreserveUseListOrder */ true);
+ModuleData =
+ArrayRef((const uint8_t *)OS.str().data(), OS.str().size());
+  } else
+// If the input is LLVM bitcode, write the input byte stream directly.
+ModuleData = ArrayRef((const uint8_t *)Buf.getBufferStart(),
+   Buf.getBufferSize());
+  llvm::Constant *ModuleConstant =
+  llvm::ConstantDataArray::get(M.getContext(), ModuleData);
+  llvm::GlobalVariable *GV = new llvm::GlobalVariable(
+  M, Mo

[PATCH] D131618: [WIP][Do NOT review] LLD related changes for -ffat-lto-objects support

[Arda Unal via Phabricator via cfe-commits]

arda updated this revision to Diff 454048.
arda added a comment.

Invoke -flto=thin in the first stage of -ffat-lto-objects


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D131618/new/

https://reviews.llvm.org/D131618

Files:
  clang/include/clang/Basic/CodeGenOptions.def
  clang/include/clang/CodeGen/BackendUtil.h
  clang/include/clang/Driver/Options.td
  clang/include/clang/Driver/ToolChain.h
  clang/lib/CodeGen/BackendUtil.cpp
  clang/lib/CodeGen/CodeGenAction.cpp
  clang/lib/Driver/Driver.cpp
  clang/lib/Driver/ToolChains/Clang.cpp
  clang/test/Driver/fat-lto-objects.c
  lld/ELF/Config.h
  lld/ELF/Driver.cpp
  lld/ELF/InputFiles.cpp
  lld/ELF/Options.td
  lld/test/ELF/fatlto/Inputs/a-fatLTO.yaml
  lld/test/ELF/fatlto/Inputs/a-thinLTO.ll
  lld/test/ELF/fatlto/Inputs/a.c
  lld/test/ELF/fatlto/Inputs/a.h
  lld/test/ELF/fatlto/Inputs/a.yaml
  lld/test/ELF/fatlto/Inputs/main-fatLTO.yaml
  lld/test/ELF/fatlto/Inputs/main-thinLTO.ll
  lld/test/ELF/fatlto/Inputs/main.c
  lld/test/ELF/fatlto/Inputs/main.yaml
  lld/test/ELF/fatlto/fatlto.test
  llvm/include/llvm/Bitcode/BitcodeWriter.h
  llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
  llvm/lib/Object/ObjectFile.cpp

Index: llvm/lib/Object/ObjectFile.cpp
===
--- llvm/lib/Object/ObjectFile.cpp
+++ llvm/lib/Object/ObjectFile.cpp
@@ -79,7 +79,7 @@
 bool ObjectFile::isSectionBitcode(DataRefImpl Sec) const {
   Expected NameOrErr = getSectionName(Sec);
   if (NameOrErr)
-return *NameOrErr == ".llvmbc";
+return *NameOrErr == ".llvmbc" || *NameOrErr == ".llvm.lto";
   consumeError(NameOrErr.takeError());
   return false;
 }
Index: llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
===
--- llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
+++ llvm/lib/Bitcode/Writer/BitcodeWriter.cpp
@@ -4953,6 +4953,37 @@
   llvm_unreachable("Unimplemented ObjectFormatType");
 }
 
+static const char *getSectionNameForBitcodeForFatLTO(const Triple &T) {
+  switch (T.getObjectFormat()) {
+  case Triple::MachO:
+llvm::report_fatal_error("MachO is not yet implemented for FatLTO");
+break;
+  case Triple::COFF:
+llvm::report_fatal_error("COFF is not yet implemented for FatLTO");
+break;
+  case Triple::ELF:
+return ".llvm.lto";
+  case Triple::Wasm:
+llvm::report_fatal_error("Wasm is not yet implemented for FatLTO");
+break;
+  case Triple::UnknownObjectFormat:
+return ".llvm.lto";
+  case Triple::GOFF:
+llvm::report_fatal_error("GOFF is not yet implemented for FatLTO");
+break;
+  case Triple::SPIRV:
+llvm::report_fatal_error("SPIRV is not yet implemented for FatLTO");
+break;
+  case Triple::XCOFF:
+llvm::report_fatal_error("XCOFF is not yet implemented for FatLTO");
+break;
+  case Triple::DXContainer:
+llvm::report_fatal_error("DXContainer is not yet implemented for FatLTO");
+break;
+  }
+  llvm_unreachable("Unimplemented ObjectFormatType");
+}
+
 void llvm::embedBitcodeInModule(llvm::Module &M, llvm::MemoryBufferRef Buf,
 bool EmbedBitcode, bool EmbedCmdline,
 const std::vector &CmdArgs) {
@@ -5045,3 +5076,68 @@
   llvm::ConstantArray::get(ATy, UsedArray), "llvm.compiler.used");
   NewUsed->setSection("llvm.metadata");
 }
+
+void llvm::embedBitcodeInFatObject(llvm::Module &M, llvm::MemoryBufferRef Buf) {
+  // Save llvm.compiler.used and remove it.
+  SmallVector UsedArray;
+  SmallVector UsedGlobals;
+  Type *UsedElementType = Type::getInt8Ty(M.getContext())->getPointerTo(0);
+  GlobalVariable *Used = collectUsedGlobalVariables(M, UsedGlobals, true);
+  for (auto *GV : UsedGlobals) {
+if (GV->getName() != "llvm.embedded.module" &&
+GV->getName() != "llvm.cmdline")
+  UsedArray.push_back(
+  ConstantExpr::getPointerBitCastOrAddrSpaceCast(GV, UsedElementType));
+  }
+  if (Used)
+Used->eraseFromParent();
+
+  // Embed the bitcode for the llvm module.
+  std::string Data;
+  ArrayRef ModuleData;
+  Triple T(M.getTargetTriple());
+
+  if (Buf.getBufferSize() == 0 ||
+  !isBitcode((const unsigned char *)Buf.getBufferStart(),
+ (const unsigned char *)Buf.getBufferEnd())) {
+// If the input is LLVM Assembly, bitcode is produced by serializing
+// the module. Use-lists order need to be preserved in this case.
+llvm::raw_string_ostream OS(Data);
+llvm::WriteBitcodeToFile(M, OS, /* ShouldPreserveUseListOrder */ true);
+ModuleData =
+ArrayRef((const uint8_t *)OS.str().data(), OS.str().size());
+  } else
+// If the input is LLVM bitcode, write the input byte stream directly.
+ModuleData = ArrayRef((const uint8_t *)Buf.getBufferStart(),
+   Buf.getBufferSize());
+  llvm::Constant *ModuleConstant =
+  llvm::ConstantDataArray::get(M.getContext(), ModuleData);
+  llvm::GlobalVariable *GV = new 

[PATCH] D131618: [WIP][Do NOT review] LLD related changes for -ffat-lto-objects support

[Arda Unal via Phabricator via cfe-commits]

arda updated this revision to Diff 454585.
arda marked 9 inline comments as done.
arda added a comment.
Herald added a subscriber: mgorny.

Reimplement embed-bitcode part as a separate pass


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D131618/new/

https://reviews.llvm.org/D131618

Files:
  clang/include/clang/Basic/CodeGenOptions.def
  clang/include/clang/Driver/Options.td
  clang/lib/CodeGen/BackendUtil.cpp
  clang/lib/Driver/Driver.cpp
  clang/lib/Driver/ToolChains/Clang.cpp
  clang/test/CodeGen/embed-lto-fatlto.c
  clang/test/Driver/fat-lto-objects.c
  lld/ELF/Config.h
  lld/ELF/Driver.cpp
  lld/ELF/InputFiles.cpp
  lld/ELF/Options.td
  lld/test/ELF/fatlto/Inputs/a-LTO.ll
  lld/test/ELF/fatlto/Inputs/a-fatLTO.yaml
  lld/test/ELF/fatlto/Inputs/a.c
  lld/test/ELF/fatlto/Inputs/a.h
  lld/test/ELF/fatlto/Inputs/a.yaml
  lld/test/ELF/fatlto/Inputs/main-LTO.ll
  lld/test/ELF/fatlto/Inputs/main-fatLTO.yaml
  lld/test/ELF/fatlto/Inputs/main.c
  lld/test/ELF/fatlto/Inputs/main.yaml
  lld/test/ELF/fatlto/fatlto.test
  llvm/include/llvm/Bitcode/EmbedBitcodePass.h
  llvm/lib/Bitcode/Writer/CMakeLists.txt
  llvm/lib/Bitcode/Writer/EmbedBitcodePass.cpp
  llvm/lib/Object/ObjectFile.cpp
  llvm/lib/Passes/PassBuilder.cpp
  llvm/lib/Passes/PassRegistry.def
  llvm/test/Bitcode/embed-multiple.ll
  llvm/test/Bitcode/embed-unsupported-object-format.ll
  llvm/test/Bitcode/embed.ll

Index: llvm/test/Bitcode/embed.ll
===
--- /dev/null
+++ llvm/test/Bitcode/embed.ll
@@ -0,0 +1,7 @@
+; RUN: opt --mtriple x86_64-unknown-linux-gnu < %s -passes=embed-bitcode -S | FileCheck %s
+
+@a = global i32 1
+
+; CHECK: @a = global i32 1
+; CHECK: @llvm.embedded.module = private constant [1252 x i8] c"{{.*}}", section ".llvm.lto", align 1
+; CHECK: @llvm.compiler.used = appending global [1 x ptr] [ptr @llvm.embedded.module], section "llvm.metadata"
Index: llvm/test/Bitcode/embed-unsupported-object-format.ll
===
--- /dev/null
+++ llvm/test/Bitcode/embed-unsupported-object-format.ll
@@ -0,0 +1,5 @@
+; RUN: not --crash opt --mtriple powerpc64-unknown-aix < %s -passes=embed-bitcode -S 2>&1 | FileCheck %s
+
+@a = global i32 1
+
+; CHECK: LLVM ERROR: Embed bitcode pass currently only supports ELF object format.
Index: llvm/test/Bitcode/embed-multiple.ll
===
--- /dev/null
+++ llvm/test/Bitcode/embed-multiple.ll
@@ -0,0 +1,6 @@
+; RUN: not --crash opt --mtriple x86_64-unknown-linux-gnu < %s -passes=embed-bitcode -S 2>&1 | FileCheck %s
+
+@a = global i32 1
+@llvm.embedded.module = private constant [4 x i8] c"BC\C0\DE"
+
+; CHECK: LLVM ERROR: Can only embed the module once.
Index: llvm/lib/Passes/PassRegistry.def
===
--- llvm/lib/Passes/PassRegistry.def
+++ llvm/lib/Passes/PassRegistry.def
@@ -57,6 +57,7 @@
 MODULE_PASS("dot-callgraph", CallGraphDOTPrinterPass())
 MODULE_PASS("elim-avail-extern", EliminateAvailableExternallyPass())
 MODULE_PASS("extract-blocks", BlockExtractorPass())
+MODULE_PASS("embed-bitcode", EmbedBitcodePass())
 MODULE_PASS("forceattrs", ForceFunctionAttrsPass())
 MODULE_PASS("function-import", FunctionImportPass())
 MODULE_PASS("function-specialization", FunctionSpecializationPass())
Index: llvm/lib/Passes/PassBuilder.cpp
===
--- llvm/lib/Passes/PassBuilder.cpp
+++ llvm/lib/Passes/PassBuilder.cpp
@@ -72,6 +72,7 @@
 #include "llvm/Analysis/TargetLibraryInfo.h"
 #include "llvm/Analysis/TargetTransformInfo.h"
 #include "llvm/Analysis/TypeBasedAliasAnalysis.h"
+#include "llvm/Bitcode/EmbedBitcodePass.h"
 #include "llvm/IR/Dominators.h"
 #include "llvm/IR/IRPrintingPasses.h"
 #include "llvm/IR/PassManager.h"
Index: llvm/lib/Object/ObjectFile.cpp
===
--- llvm/lib/Object/ObjectFile.cpp
+++ llvm/lib/Object/ObjectFile.cpp
@@ -79,7 +79,7 @@
 bool ObjectFile::isSectionBitcode(DataRefImpl Sec) const {
   Expected NameOrErr = getSectionName(Sec);
   if (NameOrErr)
-return *NameOrErr == ".llvmbc";
+return *NameOrErr == ".llvmbc" || *NameOrErr == ".llvm.lto";
   consumeError(NameOrErr.takeError());
   return false;
 }
Index: llvm/lib/Bitcode/Writer/EmbedBitcodePass.cpp
===
--- /dev/null
+++ llvm/lib/Bitcode/Writer/EmbedBitcodePass.cpp
@@ -0,0 +1,52 @@
+//===- EmbedBitcodePass.cpp - Pass that embeds the bitcode into a global---===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// EmbedBitcodePass imple

[PATCH] D131618: [WIP][Do NOT review] LLD related changes for -ffat-lto-objects support

[Arda Unal via Phabricator via cfe-commits]

arda added inline comments.



Comment at: lld/test/ELF/fatlto/fatlto.test:4-6
+; clang -c -ffat-lto-objects -o a-fatLTO.o a.c
+; clang -c -ffat-lto-objects -o main-fatLTO.o main.c
+; clang -c a.c main.c

phosek wrote:
> LLD tests cannot depend on `clang`, that's a layering violation. It also 
> doesn't seem like the output is being used anywhere so this could be safely 
> removed.
Sorry for the confusion. These ones are just comments and do not run. I will 
remove them.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D131618/new/

https://reviews.llvm.org/D131618

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D131618: [WIP][Do NOT review] LLD related changes for -ffat-lto-objects support

[Arda Unal via Phabricator via cfe-commits]

arda marked 2 inline comments as done.
arda added inline comments.



Comment at: lld/ELF/InputFiles.cpp:1721
+Expected fatLTOData = 
IRObjectFile::findBitcodeInMemBuffer(mb);
+if (!errorToBool(fatLTOData.takeError()))
+  return make(*fatLTOData, archiveName, offsetInArchive, 
/*lazy=*/false);

paulkirth wrote:
> Do we just keep going if there's an error? This is only checking if there's 
> **not** an error, right? or does `takeError` do more than I remember?
We just keep going if there is an error. So we end up using the assembly and 
not the embedded LTO module when generating the object file if there is no 
embedded bitcode in the input files.



Comment at: lld/test/ELF/fatlto/fatlto.test:28-30
+; RUN: obj2yaml %tfoo-fatLTO > %tfoo-fatLTO.yaml
+; RUN: obj2yaml %tfoo-LTO > %tfoo-LTO.yaml
+; RUN: diff %tfoo-fatLTO.yaml %tfoo-LTO.yaml

paulkirth wrote:
> Can this test fail given how you generate the linker's inputs and what you're 
> comparing? 
> 
> So, does the diff fail if you don't pass the `-fat-lto-objects` flag on Line 
> 13? 
Yes, it fails.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D131618/new/

https://reviews.llvm.org/D131618

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits

[PATCH] D131618: [WIP][Do NOT review] LLD related changes for -ffat-lto-objects support

[Arda Unal via Phabricator via cfe-commits]

arda marked 2 inline comments as done.
arda added inline comments.



Comment at: llvm/lib/Bitcode/Writer/EmbedBitcodePass.cpp:26
+
+PreservedAnalyses EmbedBitcodePass::run(Module &M, ModuleAnalysisManager &AM) {
+  if (M.getGlobalVariable("llvm.embedded.module", true))

sfertile wrote:
> From the discourse discussion:
> 1) it was suggested that we remove the existing -fembed-bitcode functionality 
> as Apple has stop supporting it.
> 2) mentioned that MLGO uses the option to embed the bitcode at various points 
> in the pipeline depending on if its using LTO our not.
> 
> Do we want the pass to be a bit more generic and be able to specify the 
> global to use for embedding, and the section name to use as arguments? That 
> way MLGO can keep using the section name it uses now . It also helps 
> consuming tools to disambiguate between bitcode embedded for lto purpose from 
> bitcode embedded for other purposes.
I should have an answer in a couple of days. I will keep you updated.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D131618/new/

https://reviews.llvm.org/D131618

___
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits