Re: [Request for Comments] Using Rust libraries in the Rust frontend
Hi Richard, On 1/23/24 08:23, Richard Biener wrote: On Mon, Jan 22, 2024 at 7:51 PM Arthur Cohen wrote: Hi everyone, In order to increase the development speed of Rust features, we are seeking feedback on reusing some Rust components directly within our front-end. As mentioned in other conferences, the most important component we would like to integrate into the front-end is the polonius borrow-checker. Another component we've become interested in is the `rustc_format_parser` library, responsible for handling parsing and handling of format arguments as described in the documentation for `std::fmt` [1]. However, since these libraries are written in Rust, GCC in itself is not yet able to compile them. They all depend on the Rust standard library, which we cannot yet compile and link properly. This obviously raises a question - how to actually compile, integrate and distribute these components? We do have the option to rewrite them from scratch, but we feel that spending time on these existing, correct, battle-tested and easily integrable components instead of focusing on other aspects of the language would be a mistake. Spending this time instead on Rust features that we are missing for compiling these components would, in our opinion, yield more results, as it would also help in compiling other Rust programs. We could either distribute these components as compiled libraries, or look at integrating the official Rust compiler to our build system as a temporary measure. I am aware that this would mean restricting the Rust GCC front-end to platforms where the official Rust compiler is also available, which is less than ideal. But that's only for the host part - you can still cross compile to another target and possibly, once the GCC frontend can compile these libraries itself, also use them to bootstrap a hosted version on that target - speaking of .. However, this would only be temporary - as soon as the Rust front-end is able to compile these components, we would simply reuse them and compile them with gccrs as part of our bootstrapping process. .. gccrs would then need to be able to build itself without those modules, at least during stage1 so that the stage1 compiler can then be used to build them. Or you go like m2 and build a "mini-rust" that's just capable of building the modules. Right, that makes a lot of sense. We should definitely be able to build the format string parser without a format string parser, as it does not use format strings for error handling or anything. And even if it did, it would be pretty easy to remove that and do the formatting by hand. Similarly, the borrow checker is not "needed" for compilation and we do plan on building stage1 without it, while making it mandatory for stage2/3 builds. I think re-using parts already available is very sensible at this point. Note that while we might temporarily require a host rust compiler to boostrap gccrs I'd rather not have the build system download something from the internet - so at least the sources of those dependences need to be in the GCC repository, possibly in a new toplevel directory. Okay, that makes a lot of sense. I was thinking of adding a basic check for the Rust compiler to be present when compiling these components - and error out if that isn't the case. Are you suggesting we embed a full copy of rustc in GCC and build it from source when compiling the Rust frontend? Or am I misunderstanding? The documentation for `std::fmt` [1] describes all of the features available in Rust format strings. It also contains a grammar for the format-string parser, which we would need to re-implement on top of supporting all the formatting features. As a prototype, I wrote an FFI interface to the `rustc_format_parser` library and integrated it to our macro expansion system, which took me less than a couple hours. In less than an afternoon, we had bindings for all of the exported types and functions in the library and had access to a compliant and performant Rust format string parser. But re-implementing a correct `format_args!()` parser - with the same performance as the Rust one, and the same amount of features - would probably take days, if not weeks. And we are talking about one of the simplest components we aim to reuse. Something like a fully-compliant trait solver for the Rust programming language would take months if not years to implement properly from scratch. I would like to stress once again that relying on distributing compiled libraries or using `rustc` in our build system would be temporary, and that these measures would be removed as soon as gccrs is able to compile these components from source. I am looking for comments on this decision as well as the best practices we could adopt. Have there been any similar decisions for other self-hosted front-ends? Any previous email threads/commits that you could point me to would be greatly appreciated. Not in this very same way but the D fronten
Re: [Request for Comments] Using Rust libraries in the Rust frontend
Note that it may not make sense to include a source copy of rustc, as that will itself require an (earlier) stage of rustc to build. mrustc can offer a bootstrap for 1.54, but depending on the versions required, you may need upwards of 10 additional rustc sources. On Thu, 25 Jan 2024 at 10:04, Arthur Cohen wrote: > Hi Richard, > > On 1/23/24 08:23, Richard Biener wrote: > > On Mon, Jan 22, 2024 at 7:51 PM Arthur Cohen > wrote: > >> > >> Hi everyone, > >> > >> In order to increase the development speed of Rust features, we are > >> seeking feedback on reusing some Rust components directly within our > >> front-end. As mentioned in other conferences, the most important > >> component we would like to integrate into the front-end is the polonius > >> borrow-checker. Another component we've become interested in is the > >> `rustc_format_parser` library, responsible for handling parsing and > >> handling of format arguments as described in the documentation for > >> `std::fmt` [1]. > >> > >> However, since these libraries are written in Rust, GCC in itself is not > >> yet able to compile them. They all depend on the Rust standard library, > >> which we cannot yet compile and link properly. This obviously raises a > >> question - how to actually compile, integrate and distribute these > >> components? > >> > >> We do have the option to rewrite them from scratch, but we feel that > >> spending time on these existing, correct, battle-tested and easily > >> integrable components instead of focusing on other aspects of the > >> language would be a mistake. Spending this time instead on Rust features > >> that we are missing for compiling these components would, in our > >> opinion, yield more results, as it would also help in compiling other > >> Rust programs. > >> > >> We could either distribute these components as compiled libraries, or > >> look at integrating the official Rust compiler to our build system as a > >> temporary measure. I am aware that this would mean restricting the Rust > >> GCC front-end to platforms where the official Rust compiler is also > >> available, which is less than ideal. > > > > But that's only for the host part - you can still cross compile to > another > > target and possibly, once the GCC frontend can compile these libraries > > itself, also use them to bootstrap a hosted version on that target - > > speaking of .. > > > >> However, this would only be > >> temporary - as soon as the Rust front-end is able to compile these > >> components, we would simply reuse them and compile them with gccrs as > >> part of our bootstrapping process. > > > > .. gccrs would then need to be able to build itself without those > modules, > > at least during stage1 so that the stage1 compiler can then be used to > > build them. Or you go like m2 and build a "mini-rust" that's just > capable > > of building the modules. > > Right, that makes a lot of sense. We should definitely be able to build > the format string parser without a format string parser, as it does not > use format strings for error handling or anything. And even if it did, > it would be pretty easy to remove that and do the formatting by hand. > > Similarly, the borrow checker is not "needed" for compilation and we do > plan on building stage1 without it, while making it mandatory for > stage2/3 builds. > > > I think re-using parts already available is very sensible at this > point. Note > > that while we might temporarily require a host rust compiler to boostrap > > gccrs I'd rather not have the build system download something from the > > internet - so at least the sources of those dependences need to be in the > > GCC repository, possibly in a new toplevel directory. > > Okay, that makes a lot of sense. I was thinking of adding a basic check > for the Rust compiler to be present when compiling these components - > and error out if that isn't the case. Are you suggesting we embed a full > copy of rustc in GCC and build it from source when compiling the Rust > frontend? Or am I misunderstanding? > > >> The documentation for `std::fmt` [1] describes all of the features > >> available in Rust format strings. It also contains a grammar for the > >> format-string parser, which we would need to re-implement on top of > >> supporting all the formatting features. As a prototype, I wrote an FFI > >> interface to the `rustc_format_parser` library and integrated it to our > >> macro expansion system, which took me less than a couple hours. In less > >> than an afternoon, we had bindings for all of the exported types and > >> functions in the library and had access to a compliant and performant > >> Rust format string parser. But re-implementing a correct > >> `format_args!()` parser - with the same performance as the Rust one, and > >> the same amount of features - would probably take days, if not weeks. > >> And we are talking about one of the simplest components we aim to reuse. > >> Something like a fully-compliant trait solver for the
Re: [Request for Comments] Using Rust libraries in the Rust frontend
Hi Arthur, > On 25 Jan 2024, at 15:03, Arthur Cohen wrote: > On 1/23/24 08:23, Richard Biener wrote: >> On Mon, Jan 22, 2024 at 7:51 PM Arthur Cohen >> wrote: >>> >>> Hi everyone, >>> >>> In order to increase the development speed of Rust features, we are >>> seeking feedback on reusing some Rust components directly within our >>> front-end. As mentioned in other conferences, the most important >>> component we would like to integrate into the front-end is the polonius >>> borrow-checker. Another component we've become interested in is the >>> `rustc_format_parser` library, responsible for handling parsing and >>> handling of format arguments as described in the documentation for >>> `std::fmt` [1]. >>> >>> However, since these libraries are written in Rust, GCC in itself is not >>> yet able to compile them. They all depend on the Rust standard library, >>> which we cannot yet compile and link properly. This obviously raises a >>> question - how to actually compile, integrate and distribute these >>> components? >>> >>> We do have the option to rewrite them from scratch, but we feel that >>> spending time on these existing, correct, battle-tested and easily >>> integrable components instead of focusing on other aspects of the >>> language would be a mistake. Spending this time instead on Rust features >>> that we are missing for compiling these components would, in our >>> opinion, yield more results, as it would also help in compiling other >>> Rust programs. >>> >>> We could either distribute these components as compiled libraries, or >>> look at integrating the official Rust compiler to our build system as a >>> temporary measure. I am aware that this would mean restricting the Rust >>> GCC front-end to platforms where the official Rust compiler is also >>> available, which is less than ideal. >> But that's only for the host part - you can still cross compile to another >> target and possibly, once the GCC frontend can compile these libraries >> itself, also use them to bootstrap a hosted version on that target - >> speaking of .. >>> However, this would only be >>> temporary - as soon as the Rust front-end is able to compile these >>> components, we would simply reuse them and compile them with gccrs as >>> part of our bootstrapping process. >> .. gccrs would then need to be able to build itself without those modules, >> at least during stage1 so that the stage1 compiler can then be used to >> build them. Or you go like m2 and build a "mini-rust" that's just capable >> of building the modules. > > Right, that makes a lot of sense. We should definitely be able to build the > format string parser without a format string parser, as it does not use > format strings for error handling or anything. And even if it did, it would > be pretty easy to remove that and do the formatting by hand. > > Similarly, the borrow checker is not "needed" for compilation and we do plan > on building stage1 without it, while making it mandatory for stage2/3 builds. As long as you leave a route available for porting it to either new (or old) systems which do not have rustc (or any other rust compiler available). E.g. with Ada it is possible to port to a new platform by first building a cross-compiler and then to use that cross-compiler to build a “native cross” (build != host == target) to provide an initial compiler on the target platform. Those cross compilers are not bootstrapped (it is a single stage) - but it is reasonable to make it a condition that the $build compiler is the same (even exact) version as the sources you are trying to port to the new platform. thanks Iain >> I think re-using parts already available is very sensible at this point. >> Note >> that while we might temporarily require a host rust compiler to boostrap >> gccrs I'd rather not have the build system download something from the >> internet - so at least the sources of those dependences need to be in the >> GCC repository, possibly in a new toplevel directory. > > Okay, that makes a lot of sense. I was thinking of adding a basic check for > the Rust compiler to be present when compiling these components - and error > out if that isn't the case. Are you suggesting we embed a full copy of rustc > in GCC and build it from source when compiling the Rust frontend? Or am I > misunderstanding? > >>> The documentation for `std::fmt` [1] describes all of the features >>> available in Rust format strings. It also contains a grammar for the >>> format-string parser, which we would need to re-implement on top of >>> supporting all the formatting features. As a prototype, I wrote an FFI >>> interface to the `rustc_format_parser` library and integrated it to our >>> macro expansion system, which took me less than a couple hours. In less >>> than an afternoon, we had bindings for all of the exported types and >>> functions in the library and had access to a compliant and performant >>> Rust format string parser. But re-implementi
Re: [Request for Comments] Using Rust libraries in the Rust frontend
> Am 25.01.2024 um 16:03 schrieb Arthur Cohen : > > Hi Richard, > >> On 1/23/24 08:23, Richard Biener wrote: >>> On Mon, Jan 22, 2024 at 7:51 PM Arthur Cohen >>> wrote: >>> >>> Hi everyone, >>> >>> In order to increase the development speed of Rust features, we are >>> seeking feedback on reusing some Rust components directly within our >>> front-end. As mentioned in other conferences, the most important >>> component we would like to integrate into the front-end is the polonius >>> borrow-checker. Another component we've become interested in is the >>> `rustc_format_parser` library, responsible for handling parsing and >>> handling of format arguments as described in the documentation for >>> `std::fmt` [1]. >>> >>> However, since these libraries are written in Rust, GCC in itself is not >>> yet able to compile them. They all depend on the Rust standard library, >>> which we cannot yet compile and link properly. This obviously raises a >>> question - how to actually compile, integrate and distribute these >>> components? >>> >>> We do have the option to rewrite them from scratch, but we feel that >>> spending time on these existing, correct, battle-tested and easily >>> integrable components instead of focusing on other aspects of the >>> language would be a mistake. Spending this time instead on Rust features >>> that we are missing for compiling these components would, in our >>> opinion, yield more results, as it would also help in compiling other >>> Rust programs. >>> >>> We could either distribute these components as compiled libraries, or >>> look at integrating the official Rust compiler to our build system as a >>> temporary measure. I am aware that this would mean restricting the Rust >>> GCC front-end to platforms where the official Rust compiler is also >>> available, which is less than ideal. >> But that's only for the host part - you can still cross compile to another >> target and possibly, once the GCC frontend can compile these libraries >> itself, also use them to bootstrap a hosted version on that target - >> speaking of .. >>> However, this would only be >>> temporary - as soon as the Rust front-end is able to compile these >>> components, we would simply reuse them and compile them with gccrs as >>> part of our bootstrapping process. >> .. gccrs would then need to be able to build itself without those modules, >> at least during stage1 so that the stage1 compiler can then be used to >> build them. Or you go like m2 and build a "mini-rust" that's just capable >> of building the modules. > > Right, that makes a lot of sense. We should definitely be able to build the > format string parser without a format string parser, as it does not use > format strings for error handling or anything. And even if it did, it would > be pretty easy to remove that and do the formatting by hand. > > Similarly, the borrow checker is not "needed" for compilation and we do plan > on building stage1 without it, while making it mandatory for stage2/3 builds. > >> I think re-using parts already available is very sensible at this point. >> Note >> that while we might temporarily require a host rust compiler to boostrap >> gccrs I'd rather not have the build system download something from the >> internet - so at least the sources of those dependences need to be in the >> GCC repository, possibly in a new toplevel directory. > > Okay, that makes a lot of sense. I was thinking of adding a basic check for > the Rust compiler to be present when compiling these components - and error > out if that isn't the case. Are you suggesting we embed a full copy of rustc > in GCC and build it from source when compiling the Rust frontend? Or am I > misunderstanding? No, you’d have the format parser and all it’s dependencies in the source tree but rely on an installed rustc to build that. Maybe that doesn’t make sense - I’m not too familiar with the rust compilation model. Richard >>> The documentation for `std::fmt` [1] describes all of the features >>> available in Rust format strings. It also contains a grammar for the >>> format-string parser, which we would need to re-implement on top of >>> supporting all the formatting features. As a prototype, I wrote an FFI >>> interface to the `rustc_format_parser` library and integrated it to our >>> macro expansion system, which took me less than a couple hours. In less >>> than an afternoon, we had bindings for all of the exported types and >>> functions in the library and had access to a compliant and performant >>> Rust format string parser. But re-implementing a correct >>> `format_args!()` parser - with the same performance as the Rust one, and >>> the same amount of features - would probably take days, if not weeks. >>> And we are talking about one of the simplest components we aim to reuse. >>> Something like a fully-compliant trait solver for the Rust programming >>> language would take months if not years to implement properly from scratch.
Re: [PATCH 5/4] libbacktrace: improve getting debug information for loaded dlls
Am 23.01.2024 um 23:37 schrieb Ian Lance Taylor: On Thu, Jan 4, 2024 at 2:33 PM Björn Schäpers wrote: Am 03.01.2024 um 00:12 schrieb Björn Schäpers: Am 30.11.2023 um 20:53 schrieb Ian Lance Taylor: On Fri, Jan 20, 2023 at 2:55 AM Björn Schäpers wrote: From: Björn Schäpers Fixes https://github.com/ianlancetaylor/libbacktrace/issues/53, except that libraries loaded after the backtrace_initialize are not handled. But as far as I can see that's the same for elf. Thanks, but I don't want a patch that loops using goto statements. Please rewrite to avoid that. It may be simpler to call a function. Also starting with a module count of 1000 seems like a lot. Do typical Windows programs load that many modules? Ian Rewritten using a function. If that is commited, could you attribute that commit to me (--author="Björn Schäpers ")? Thanks and kind regards, Björn. I noticed that under 64 bit libraries loaded with LoadLibrary were missing. EnumProcessModules stated the correct number of modules, but did not fill the the HMODULEs, but set them to 0. While trying to investigate I noticed if I do the very same thing from main (in C++) I even got fewer module HMODULEs. So I went a different way. This detects all libraries correctly, in 32 and 64 bit. The question is, if it should be a patch on top of the previous, or should they be merged, or even only this solution and drop the EnumProcessModules variant? Is there any reason to use both patches? Seems simpler to just use this one if it works. Thanks. Ian This one needs the tlhelp32 header and its functions, which are (accoridng to the microsoft documentation) are only available since Windows XP rsp. Windows Server 2003. If that's no problem, and in my opinion it shouldn't be, then I can basically drop patch 4 and rebase this one. Kind regards, Björn.
Re: [PATCH 5/4] libbacktrace: improve getting debug information for loaded dlls
On Thu, Jan 25, 2024 at 11:53 AM Björn Schäpers wrote: > > Am 23.01.2024 um 23:37 schrieb Ian Lance Taylor: > > On Thu, Jan 4, 2024 at 2:33 PM Björn Schäpers wrote: > >> > >> Am 03.01.2024 um 00:12 schrieb Björn Schäpers: > >>> Am 30.11.2023 um 20:53 schrieb Ian Lance Taylor: > On Fri, Jan 20, 2023 at 2:55 AM Björn Schäpers wrote: > > > > From: Björn Schäpers > > > > Fixes https://github.com/ianlancetaylor/libbacktrace/issues/53, except > > that libraries loaded after the backtrace_initialize are not handled. > > But as far as I can see that's the same for elf. > > Thanks, but I don't want a patch that loops using goto statements. > Please rewrite to avoid that. It may be simpler to call a function. > > Also starting with a module count of 1000 seems like a lot. Do > typical Windows programs load that many modules? > > Ian > > > >>> > >>> Rewritten using a function. > >>> > >>> If that is commited, could you attribute that commit to me > >>> (--author="Björn > >>> Schäpers ")? > >>> > >>> Thanks and kind regards, > >>> Björn. > >> > >> I noticed that under 64 bit libraries loaded with LoadLibrary were missing. > >> EnumProcessModules stated the correct number of modules, but did not fill > >> the > >> the HMODULEs, but set them to 0. While trying to investigate I noticed if > >> I do > >> the very same thing from main (in C++) I even got fewer module HMODULEs. > >> > >> So I went a different way. This detects all libraries correctly, in 32 and > >> 64 > >> bit. The question is, if it should be a patch on top of the previous, or > >> should > >> they be merged, or even only this solution and drop the EnumProcessModules > >> variant? > > > > Is there any reason to use both patches? Seems simpler to just use > > this one if it works. Thanks. > > > > Ian > > This one needs the tlhelp32 header and its functions, which are (accoridng to > the microsoft documentation) are only available since Windows XP rsp. Windows > Server 2003. > > If that's no problem, and in my opinion it shouldn't be, then I can basically > drop patch 4 and rebase this one. I don't see that as a problem. It seems like the patch will fall back cleanly on ancient Windows and simply fail to pick up other loaded DLLs. I think that is fine. I'll look for an updated patch. Thanks. Ian
gcc-11-20240125 is now available
Snapshot gcc-11-20240125 is now available on https://gcc.gnu.org/pub/gcc/snapshots/11-20240125/ and on various mirrors, see https://gcc.gnu.org/mirrors.html for details. This snapshot has been generated from the GCC 11 git branch with the following options: git://gcc.gnu.org/git/gcc.git branch releases/gcc-11 revision 2d99c820f271b24203d0dae21f1f49db104e22c9 You'll find: gcc-11-20240125.tar.xz Complete GCC SHA256=e91456a8719f55a69a488f7f7d669cdea143b0f0e0121656ba1d79cbda3edaaf SHA1=b4850673dbffcaa3fd49a527b6cbb88bc34a3782 Diffs from 11-20240118 are available in the diffs/ subdirectory. When a particular snapshot is ready for public consumption the LATEST-11 link is updated and a message is sent to the gcc list. Please do not use a snapshot before it has been announced that way.
