[dev-servo] holiday meetings canceled
I'm canceling the meetings today and next week for the holidays. Many of the team is out (especially next week). Please use the meeting time to review some PRs if you are still around :) Happy holidays everyone! jack. ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] Using Cargo in Gecko
Developing against external packages (including ones shared with Servo) and pulling in changes as needed can be an easier process than you expect, and the alternatives (forking or syncing) have much more serious consequences. The basic idea is that you work with shared dependencies from crates.io or github under normal circumstances. If you find you need to make a quick change to a dependency to keep moving, you can fork that dependency on Github and use your local changes instead. At the same time, you submit the changes upstream as a pull request, and revert back to using the shared dependencies once they have been accepted. Since Gecko intends to, at minimum, share dependencies with Servo, this strategy should be the most seamless. That said, Servo has blazed a trail as one of the first Rust projects with a huge tree of dependencies, and uncovered some weaknesses in Cargo as a result. At Mozlando, we had a great conversation with the Servo team about how to help them manage upstream dependencies better. --- The main improvement that we plan to make to Cargo as a result of that conversation is *temporary overrides*. **Temporary overrides are an important part of the design of a package manager like Cargo** (Bundler, the package manager I worked on in the Ruby ecosystem, has support for temporary overrides). Let me outline how I envision that working in Cargo. --- Workflow (using Servo as an example) When a project discovers that it needs to make a temporary patch to an upstream dependency, it forks the project on Github. Next, it goes into the top-level `Cargo.toml` for their project and enters the override: ``` [[overrides.simd]] version = "*" git = "https://github.com/servo/simd"; ``` This means that any version of the `simd` crate used by Servo (directly or indirectly) will be overridden by the version supplied by `servo/simd`. This applies to crates from crates.io as well as other git crates. Next, Servo would submit a pull request to the original git repository that they have forked. Once the original owner has accepted the pull request, Servo can remove the override. Alternately, if the crate in question is owned by Servo (`servo/rust-freetype` for example), Servo can create a branch in `servo/rust-freetype` and use those changes using the same override mechanism: ``` [[overrides.rust-freetype]] version = "*" branch = "bugfix-3561" ``` I would also like to make this workflow nice from the command line, with `cargo override` or something similar. -- Yehuda > On Thu, Dec 17, 2015 at 3:04 PM, Valentin Gosu > wrote: > > > Hi folks, > > > > I just wanted to share some notes on using Cargo in Gecko. One of the > > requirements was vendoring packages in tree. It turns out I was able to do > > that with a .cargo/config file in the scope of the rust directory, which > > defines the paths for dependencies, and [http] timeout = 0, which prevents > > downloading any resources from crates.io. [1] > > Better support for this use case would be welcome, such as making sure no > > network connections are made (the index still gets refreshed), but this is > > good enough for what we need at the moment. > > > > I'm not sure if we've agreed to have a top level directory for all of the > > rust crates, but I think it makes sense. It allows for easier sharing of > > crates (between necko and media for example). > > From what I understand servo's mach has a command for two-way sync between > > web-platform-tests and upstream. This would be a nice feature for cargo, as > > most of our dependencies would come from crates.io. So identifying the > > upstream repo, and doing a diff would be super awesome - probably for other > > projects as well. > > > > I'll probably have more to share once/if I manage to use cargo withing > > Gecko's build system. > > > > Thanks! > > > > [1] https://github.com/rust-lang/cargo/issues/1926 > > ___ > > dev-servo mailing list > > dev-servo@lists.mozilla.org > > https://lists.mozilla.org/listinfo/dev-servo > > ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] Using Cargo in Gecko
On Mon, Dec 21, 2015 at 1:52 PM, wrote: > Developing against external packages (including ones shared with Servo) > and pulling in changes as needed can be an easier process than you expect, > and the alternatives (forking or syncing) have much more serious > consequences. > > The basic idea is that you work with shared dependencies from crates.io > or github under normal circumstances. If you find you need to make a quick > change to a dependency to keep moving, you can fork that dependency on > Github and use your local changes instead. At the same time, you submit > the changes upstream as a pull request, and revert back to using the shared > dependencies once they have been accepted. > > Since Gecko intends to, at minimum, share dependencies with Servo, this > strategy should be the most seamless. > I don't think this is going to fly in Gecko, unfortunately. Gecko is a monolithic repo, and everything needs to be vendored in-tree (a non-negotiable requirement from the build peers). This means that we'll already need an automated process for pulling in updates to the shared code, committing them, and running them through CI. The only remaining question, then, is whether we allow Gecko hackers to commit directly (with appropriate review) to the vendored code, or whether we require them hack on a local clone of the upstream. Gecko hackers expect to be able to easily make atomic changesets that span modules. They want to make a series of changsets that is, say, 15-deep and spanning multiple programming languages, interactively reorder those commits, bisect them, etc. They may grudgingly put up with barriers to that workflow for something self-contained and infrequently-touched (like the url parser), but requiring them to make a github fork to hack on the style system will likely meet insurmountable opposition. So of the aforementioned options, I'm pretty sure that only the first is workable. The two ways to implement that are with tooling for automatic upstreaming, or forking Servo. So I sure hope that we can figure out the tooling. ;-) bholley > That said, Servo has blazed a trail as one of the first Rust projects with > a huge tree of dependencies, and uncovered some weaknesses in Cargo as a > result. At Mozlando, we had a great conversation with the Servo team about > how to help them manage upstream dependencies better. > > --- > > The main improvement that we plan to make to Cargo as a result of that > conversation is *temporary overrides*. > > **Temporary overrides are an important part of the design of a package > manager like Cargo** (Bundler, the package manager I worked on in the Ruby > ecosystem, has support for temporary overrides). > > Let me outline how I envision that working in Cargo. > > --- > > Workflow (using Servo as an example) > > When a project discovers that it needs to make a temporary patch to an > upstream dependency, it forks the project on Github. > > Next, it goes into the top-level `Cargo.toml` for their project and enters > the override: > > ``` > [[overrides.simd]] > > version = "*" > git = "https://github.com/servo/simd"; > ``` > > This means that any version of the `simd` crate used by Servo (directly or > indirectly) will be overridden by the version supplied by `servo/simd`. > This applies to crates from crates.io as well as other git crates. > > Next, Servo would submit a pull request to the original git repository > that they have forked. Once the original owner has accepted the pull > request, Servo can remove the override. > > Alternately, if the crate in question is owned by Servo > (`servo/rust-freetype` for example), Servo can create a branch in > `servo/rust-freetype` and use those changes using the same override > mechanism: > > ``` > [[overrides.rust-freetype]] > > version = "*" > branch = "bugfix-3561" > ``` > > I would also like to make this workflow nice from the command line, with > `cargo override` or something similar. > > -- Yehuda > > > On Thu, Dec 17, 2015 at 3:04 PM, Valentin Gosu > > wrote: > > > > > Hi folks, > > > > > > I just wanted to share some notes on using Cargo in Gecko. One of the > > > requirements was vendoring packages in tree. It turns out I was able > to do > > > that with a .cargo/config file in the scope of the rust directory, > which > > > defines the paths for dependencies, and [http] timeout = 0, which > prevents > > > downloading any resources from crates.io. [1] > > > Better support for this use case would be welcome, such as making sure > no > > > network connections are made (the index still gets refreshed), but > this is > > > good enough for what we need at the moment. > > > > > > I'm not sure if we've agreed to have a top level directory for all of > the > > > rust crates, but I think it makes sense. It allows for easier sharing > of > > > crates (between necko and media for example). > > > From what I understand servo's mach has a command for two-way sync > between > > > web-platform-tests and upstream. This wo
Re: [dev-servo] Using Cargo in Gecko
On Mon, Dec 21, 2015 at 2:14 PM, Bobby Holley wrote: > I don't think this is going to fly in Gecko, unfortunately. > > Gecko is a monolithic repo, and everything needs to be vendored in-tree (a > non-negotiable requirement from the build peers). This means that we'll > already need an automated process for pulling in updates to the shared > code, committing them, and running them through CI. > Can you say a bit more about what the requirements are here? Is the reason for including the code in-tree that you need to be 100% confident that everyone is talking about the same code? Or is it more than that? ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] Using Cargo in Gecko
On 2015-12-21 5:21 PM, Yehuda Katz wrote: On Mon, Dec 21, 2015 at 2:14 PM, Bobby Holley wrote: I don't think this is going to fly in Gecko, unfortunately. Gecko is a monolithic repo, and everything needs to be vendored in-tree (a non-negotiable requirement from the build peers). This means that we'll already need an automated process for pulling in updates to the shared code, committing them, and running them through CI. Can you say a bit more about what the requirements are here? Is the reason for including the code in-tree that you need to be 100% confident that everyone is talking about the same code? Or is it more than that? The "non-negotiable requirement from the build peers" is that the machines that build Firefox do not have internet access. ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] Using Cargo in Gecko
On Mon, Dec 21, 2015 at 2:21 PM, Yehuda Katz wrote: > On Mon, Dec 21, 2015 at 2:14 PM, Bobby Holley > wrote: > > > I don't think this is going to fly in Gecko, unfortunately. > > > > Gecko is a monolithic repo, and everything needs to be vendored in-tree > (a > > non-negotiable requirement from the build peers). This means that we'll > > already need an automated process for pulling in updates to the shared > > code, committing them, and running them through CI. > > > > Can you say a bit more about what the requirements are here? Is the reason > for including the code in-tree that you need to be 100% confident that > everyone is talking about the same code? Or is it more than that? > The reason I've heard are: (1) Gecko/Firefox has a very complicated releng setup, and the builders are heavily firewalled from the outside, and not allowed to hit the network. So adding network dependencies to the build step would require a lot of operations work. (2) Gecko exists on a pretty long timescale, and we want to make sure that we can still build Firefox 50 ten years from now, even if Caro has long migrated to some other setup. (3) A general unease about depending on any third-party service without a contract and SLA in order to build and ship Firefox. There may be other reasons, or I may be getting some of these wrong. This all comes from gps, ted, etc, so you're probably better off discussing with them directly. ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] Using Cargo in Gecko
Yehuda Katz (ph) 718.877.1325 On Mon, Dec 21, 2015 at 2:28 PM, Josh Matthews wrote: > On 2015-12-21 5:21 PM, Yehuda Katz wrote: > >> On Mon, Dec 21, 2015 at 2:14 PM, Bobby Holley >> wrote: >> >> I don't think this is going to fly in Gecko, unfortunately. >>> >>> Gecko is a monolithic repo, and everything needs to be vendored in-tree >>> (a >>> non-negotiable requirement from the build peers). This means that we'll >>> already need an automated process for pulling in updates to the shared >>> code, committing them, and running them through CI. >>> >>> >> Can you say a bit more about what the requirements are here? Is the reason >> for including the code in-tree that you need to be 100% confident that >> everyone is talking about the same code? Or is it more than that? >> >> > The "non-negotiable requirement from the build peers" is that the machines > that build Firefox do not have internet access. That's a totally reasonable requirement and one that I am very comfortable saying that Cargo should support. To support this requirement, we plan to add a new command that packages up an application into a single tarball (or equivalent), complete with all dependencies, and another command that can build such a package without network connectivity. Bundler has equivalent facilities (bundle package and bundle install --deployment) that I consider indispensable for deployment scenarios, and which I championed early on in the bundler process. (Heroku uses `bundle install --deployment` in their buildpacks). -- Yehuda ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] Using Cargo in Gecko
On 21/12/15 22:45, Yehuda Katz wrote: Yehuda Katz (ph) 718.877.1325 On Mon, Dec 21, 2015 at 2:28 PM, Josh Matthews wrote: On 2015-12-21 5:21 PM, Yehuda Katz wrote: On Mon, Dec 21, 2015 at 2:14 PM, Bobby Holley wrote: I don't think this is going to fly in Gecko, unfortunately. Gecko is a monolithic repo, and everything needs to be vendored in-tree (a non-negotiable requirement from the build peers). This means that we'll already need an automated process for pulling in updates to the shared code, committing them, and running them through CI. Can you say a bit more about what the requirements are here? Is the reason for including the code in-tree that you need to be 100% confident that everyone is talking about the same code? Or is it more than that? The "non-negotiable requirement from the build peers" is that the machines that build Firefox do not have internet access. That's a totally reasonable requirement and one that I am very comfortable saying that Cargo should support. To support this requirement, we plan to add a new command that packages up an application into a single tarball (or equivalent), complete with all dependencies, and another command that can build such a package without network connectivity. It is not immediately apparent to me if that is sufficient to meet the Gecko requirements, but again, having this conversation on a list without gps, ted, glandium and other build peers seems rather counterproductive since they know both the requirements that we have from our current build system, and the improvements that are in the pipeline for next year (which I understand to be numerous and considered a priority). Without this information it's impossible to tell if any proposed change to cargo is solving the right problem. ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] Using Cargo in Gecko
On Mon, Dec 21, 2015 at 4:17 PM, James Graham wrote: > > It is not immediately apparent to me if that is sufficient to meet the > Gecko requirements, but again, having this conversation on a list without > gps, ted, glandium and other build peers seems rather counterproductive > since they know both the requirements that we have from our current build > system, and the improvements that are in the pipeline for next year (which > I understand to be numerous and considered a priority). Without this > information it's impossible to tell if any proposed change to cargo is > solving the right problem. Can we get them here? I'm very interested in working through this with someone who knows all of the constraints deeply. At minimum, I would like Cargo to be a suitable tool for people in similar situations as much as possible. -- Yehuda > > > ___ > dev-servo mailing list > dev-servo@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-servo > ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] Using Cargo in Gecko
I CCed them upthread, but then the subsequent reply dropped them. ;-) I would suggest starting a new, explicit thread with a summary of what's been discussed thus far (rather than asking them to jump into this one), and CCing dev-servo. That being said, my guess is that it's going to be a mostly academic discussion, because the workflow issues mentioned above would likely lead us to vendoring even without the requirements from the build peers. On Mon, Dec 21, 2015 at 4:21 PM, Yehuda Katz wrote: > On Mon, Dec 21, 2015 at 4:17 PM, James Graham > wrote: > > > > > It is not immediately apparent to me if that is sufficient to meet the > > Gecko requirements, but again, having this conversation on a list without > > gps, ted, glandium and other build peers seems rather counterproductive > > since they know both the requirements that we have from our current build > > system, and the improvements that are in the pipeline for next year > (which > > I understand to be numerous and considered a priority). Without this > > information it's impossible to tell if any proposed change to cargo is > > solving the right problem. > > > Can we get them here? I'm very interested in working through this with > someone who knows all of the constraints deeply. At minimum, I would like > Cargo to be a suitable tool for people in similar situations as much as > possible. > > -- Yehuda > > > > > > > > ___ > > dev-servo mailing list > > dev-servo@lists.mozilla.org > > https://lists.mozilla.org/listinfo/dev-servo > > > ___ > dev-servo mailing list > dev-servo@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-servo > ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] Using Cargo in Gecko
On Mon, Dec 21, 2015 at 4:31 PM, Bobby Holley wrote: > I CCed them upthread, but then the subsequent reply dropped them. ;-) > > I would suggest starting a new, explicit thread with a summary of what's > been discussed thus far (rather than asking them to jump into this one), > and CCing dev-servo. > Happy to. > > That being said, my guess is that it's going to be a mostly academic > discussion, because the workflow issues mentioned above would likely lead > us to vendoring even without the requirements from the build peers. > We assume that those workflow issues can be addressed relatively quickly (Q1 2016), so it's more than just academic to me :) Also, I'd like to make Cargo's workflow appropriate for other, similar scenarios, so understanding the constraints more deeply is useful regardless of what Gecko does in the short term. -- Yehuda > > On Mon, Dec 21, 2015 at 4:21 PM, Yehuda Katz wrote: > > > On Mon, Dec 21, 2015 at 4:17 PM, James Graham > > wrote: > > > > > > > > It is not immediately apparent to me if that is sufficient to meet the > > > Gecko requirements, but again, having this conversation on a list > without > > > gps, ted, glandium and other build peers seems rather counterproductive > > > since they know both the requirements that we have from our current > build > > > system, and the improvements that are in the pipeline for next year > > (which > > > I understand to be numerous and considered a priority). Without this > > > information it's impossible to tell if any proposed change to cargo is > > > solving the right problem. > > > > > > Can we get them here? I'm very interested in working through this with > > someone who knows all of the constraints deeply. At minimum, I would like > > Cargo to be a suitable tool for people in similar situations as much as > > possible. > > > > -- Yehuda > > > > > > > > > > > > > ___ > > > dev-servo mailing list > > > dev-servo@lists.mozilla.org > > > https://lists.mozilla.org/listinfo/dev-servo > > > > > ___ > > dev-servo mailing list > > dev-servo@lists.mozilla.org > > https://lists.mozilla.org/listinfo/dev-servo > > > ___ > dev-servo mailing list > dev-servo@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-servo > ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] Using Cargo in Gecko
On Mon, Dec 21, 2015 at 4:56 PM, Yehuda Katz wrote: > > On Mon, Dec 21, 2015 at 4:31 PM, Bobby Holley > wrote: > >> I CCed them upthread, but then the subsequent reply dropped them. ;-) >> >> I would suggest starting a new, explicit thread with a summary of what's >> been discussed thus far (rather than asking them to jump into this one), >> and CCing dev-servo. >> > > Happy to. > Who are the right people (with emails) to include? Thanks! -- Yehuda > > >> >> On Mon, Dec 21, 2015 at 4:21 PM, Yehuda Katz wrote: >> >> > On Mon, Dec 21, 2015 at 4:17 PM, James Graham >> > wrote: >> > >> > > >> > > It is not immediately apparent to me if that is sufficient to meet the >> > > Gecko requirements, but again, having this conversation on a list >> without >> > > gps, ted, glandium and other build peers seems rather >> counterproductive >> > > since they know both the requirements that we have from our current >> build >> > > system, and the improvements that are in the pipeline for next year >> > (which >> > > I understand to be numerous and considered a priority). Without this >> > > information it's impossible to tell if any proposed change to cargo is >> > > solving the right problem. >> > >> > >> > Can we get them here? I'm very interested in working through this with >> > someone who knows all of the constraints deeply. At minimum, I would >> like >> > Cargo to be a suitable tool for people in similar situations as much as >> > possible. >> > >> > -- Yehuda >> > >> > >> > > >> > > >> > > ___ >> > > dev-servo mailing list >> > > dev-servo@lists.mozilla.org >> > > https://lists.mozilla.org/listinfo/dev-servo >> > > >> > ___ >> > dev-servo mailing list >> > dev-servo@lists.mozilla.org >> > https://lists.mozilla.org/listinfo/dev-servo >> > >> ___ >> dev-servo mailing list >> dev-servo@lists.mozilla.org >> https://lists.mozilla.org/listinfo/dev-servo >> > > ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] Using Cargo in Gecko
On Mon, Dec 21, 2015 at 2:29 PM, Bobby Holley wrote: > On Mon, Dec 21, 2015 at 2:21 PM, Yehuda Katz wrote: > >> On Mon, Dec 21, 2015 at 2:14 PM, Bobby Holley >> wrote: >> >> > I don't think this is going to fly in Gecko, unfortunately. >> > >> > Gecko is a monolithic repo, and everything needs to be vendored in-tree >> (a >> > non-negotiable requirement from the build peers). This means that we'll >> > already need an automated process for pulling in updates to the shared >> > code, committing them, and running them through CI. >> > >> >> Can you say a bit more about what the requirements are here? Is the reason >> for including the code in-tree that you need to be 100% confident that >> everyone is talking about the same code? Or is it more than that? >> > > The reason I've heard are: > (1) Gecko/Firefox has a very complicated releng setup, and the builders > are heavily firewalled from the outside, and not allowed to hit the > network. So adding network dependencies to the build step would require a > lot of operations work. > (2) Gecko exists on a pretty long timescale, and we want to make sure that > we can still build Firefox 50 ten years from now, even if Caro has long > migrated to some other setup. > (3) A general unease about depending on any third-party service without a > contract and SLA in order to build and ship Firefox. > > There may be other reasons, or I may be getting some of these wrong. This > all comes from gps, ted, etc, so you're probably better off discussing with > them directly. > This is the gist of it. There are also implications for downstream packagers. The more complicated our build mechanism is, the more work it is for them. Having everything vendored makes it self contained and more manageable. There is also a general trend towards reproducible builds. Those are a bit harder to attain when you are trying to cobble together pieces from multiple repositories. Related to this are security and integrity concerns. Could a malicious actor insert a vulnerability in Firefox by compromising a 3rd party repository/project? Would we necessarily have the audit trail in place to detect this if things weren't vendored? (Yes, we have exposure to this today.) Also, #3 is more important than #1. To add some perspective, we can't have parts of automation clone from github.com because we've found GitHub to be too unreliable. I'm not talking about the China-based DDoS from a few months back - this has been a longstanding problem. In general, we don't want to have a Firefox chemspill delayed because some random 3rd party server isn't available. ___ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
Re: [dev-servo] Using Cargo in Gecko
Thanks. This is great! Responses inline. Yehuda Katz (ph) 718.877.1325 On Mon, Dec 21, 2015 at 6:19 PM, Gregory Szorc wrote: > On Mon, Dec 21, 2015 at 2:29 PM, Bobby Holley > wrote: > > > On Mon, Dec 21, 2015 at 2:21 PM, Yehuda Katz wrote: > > > >> On Mon, Dec 21, 2015 at 2:14 PM, Bobby Holley > >> wrote: > >> > >> > I don't think this is going to fly in Gecko, unfortunately. > >> > > >> > Gecko is a monolithic repo, and everything needs to be vendored > in-tree > >> (a > >> > non-negotiable requirement from the build peers). This means that > we'll > >> > already need an automated process for pulling in updates to the shared > >> > code, committing them, and running them through CI. > >> > > >> > >> Can you say a bit more about what the requirements are here? Is the > reason > >> for including the code in-tree that you need to be 100% confident that > >> everyone is talking about the same code? Or is it more than that? > >> > > > > The reason I've heard are: > > (1) Gecko/Firefox has a very complicated releng setup, and the builders > > are heavily firewalled from the outside, and not allowed to hit the > > network. So adding network dependencies to the build step would require a > > lot of operations work. > > (2) Gecko exists on a pretty long timescale, and we want to make sure > that > > we can still build Firefox 50 ten years from now, even if Caro has long > > migrated to some other setup. > > (3) A general unease about depending on any third-party service without a > > contract and SLA in order to build and ship Firefox. > > > > There may be other reasons, or I may be getting some of these wrong. This > > all comes from gps, ted, etc, so you're probably better off discussing > with > > them directly. > > > > This is the gist of it. There are also implications for downstream > packagers. The more complicated our build mechanism is, the more work it is > for them. Having everything vendored makes it self contained and more > manageable. > > There is also a general trend towards reproducible builds. Those are a bit > harder to attain when you are trying to cobble together pieces from > multiple repositories. Related to this are security and integrity concerns. > Could a malicious actor insert a vulnerability in Firefox by compromising a > 3rd party repository/project? Would we necessarily have the audit trail in > place to detect this if things weren't vendored? (Yes, we have exposure to > this today.) > Cargo also has reproducible builds as a hard requirement. For Cargo, this means that the source code used to build a project on one machine must be byte-for-byte equivalent to the source code used on another machine. This is true about direct dependencies as a well as indirect (transitive) dependencies, and is non-negotiable. Today, we achieve this goal by serializing the "precise version" of each dependency into Cargo.lock. "Precise versions" have these requirements: - For a given source, the precise version is sufficient to uniquely identify the source code of the dependency. - The code referenced by a precise version is immutable; it should be impossible for the precise dependency to point at different byte-for-byte source code over time. For crates.io, this is achieved by using the name and version of the dependency as the precise version, and making the registry immutable. It is impossible to change the code pointed to by a version after it has been published. For git dependencies, this is achieved by recording the precise rev of the code that was used when the lock file was produced. Because both of these approaches have hypothetical compromises (the crates.io server could be compromised, and git's use of SHA1 for revisions is insufficiently secure), we plan to add an additional layer of protection: we will record a SHA256 digest of the exact source code used in the Cargo.lock, which we will use to verify that the source code is byte-for-byte compatible. Running `cargo build` against a repository with a `Cargo.lock` will reliably use exactly the versions of the source code referenced by the `Cargo.lock`, which makes Cargo builds fairly reliable by design. > Also, #3 is more important than #1. To add some perspective, we can't have > parts of automation clone from github.com because we've found GitHub to be > too unreliable. I'm not talking about the China-based DDoS from a few > months back - this has been a longstanding problem. In general, we don't > want to have a Firefox chemspill delayed because some random 3rd party > server isn't available. > In my opinion, #3 is best addressed with support for automated mirroring of both crates.io and git dependencies. Mega-corps like LinkedIn use mirroring services for Rubygems, npm, and maven to ensure high-availability. This maintains the normal developer workflow and ensures that developers on the project do not rely on custom infrastructure for updates, while still ensuring that the deployment process isn't dependent
