I've seen a few follow-up questions in IRC and bug reports and would like to clarify a few points. Content inline and below.
On Thu, Apr 6, 2017 at 4:13 PM, Gregory Szorc <g...@mozilla.com> wrote: > This message applies to anyone who clones a Firefox repository from > hg.mozilla.org. *Failure to read this message may result in degraded > Mercurial performance.* If you use git-cinnabar or use the mozilla-unified > repository exclusively, you are likely shielded from degraded performance > and can stop reading. > > Some changes were made to the hosting of high-volume Firefox repositories > on hg.mozilla.org in the past week. The full list of repositories > impacted so far is available at https://hg.cdn.mozilla.net/. Client-side > operations against repositories cloned from hg.mozilla.org may be slower > until a one-time operation is performed. Once complete, operations against > hg.mozilla.org should be *faster* than they were before. > > tldr > ---- > > To ensure your Firefox repository clone runs at peak performance: > > 1) Upgrade to the latest Mercurial release (currently 4.1.2) - > instructions at [1]. > > 2a) Run `hg debugupgraderepo --optimize redeltaparent --optimize > redeltamultibase --run`. Learn more at [2]. This command will likely take a > few hours to run. But your existing repo will function just as it did > before, just faster. The command keeps a backup of your existing data and > doesn't modify your repo until the last few milliseconds of execution. > If you just run `hg debugupgraderepo`, it tells you which changes it would perform. If the list is empty and your .hg/store/00manifest.d file is <300MB, the repo is likely optimally stored already. > > 2b) Create a fresh clone from hg.mozilla.org using Mercurial 4.1 then > push/pull data from your existing clone to it. See [3] for specific > instructions. > You only need to do either 2a or 2b. Doing a clone will be faster than an in-place upgrade assuming it doesn't take you >1hr to download ~1 GB from the Internet. If you do the clone method, do read [3] and note the command to adjust the phases/publishing setting when pulling changesets from your old repo into the new one. If you make a mistake [3] also says how to correct it. > > Technical Changes > ----------------- > > The main Firefox repositories on hg.mozilla.org were using a legacy > storage format with known deficiencies. The repositories have been upgraded > to use a modern storage format (generaldelta). This storage format results > in smaller repositories (300-500 MB on-disk reduction is common) and faster > operations against optimally encoded repositories. > > The main Firefox repositories on hg.mozilla.org have not only had their > storage formats upgraded, they have also been optimized so data is stored > within as efficiently as possible. This is effectively the Mercurial > equivalent of `git repack -a -f`. > > While your local clone may be using generaldelta (run `hg > debugupgraderepo` to see if it is already present), the deltas may not be > optimal and this can result in repo bloat and slow operations. A re-clone > or one-time data upgrade with certain optimizations (using the `hg > debugupgraderepo` described in the tldr section) will align your local > clone with hg.mozilla.org and result in optimal performance. > > Performance Implications > ------------------------ > > If 2 Mercurial peers (say your local clone and hg.mozilla.org) are using > different repository storage formats, one side has to do work to normalize > exchanged data so it can be applied. For 95% of repositories, this data > normalization is fast and nobody likely notices. But because the Firefox > repository is so large, it can add noticeable to extreme overhead to > operations. > > With the Firefox repositories on hg.mozilla.org upgraded to generaldelta, > the following can occur: > > * A Mercurial 3.7+ client with a generaldelta repo will essentially stream > data from the server to local disk with minimal processing. (Read: this is > ideal and fastest.) > > * A Mercurial 3.5+ client with a non-generaldelta repo will have to > heavily process incoming data so it is compatible with local storage. This > will slow down operations like `hg pull` in proportion to the amount of > incoming data. The slowdown will be less pronounced on Mercurial 4.1 due to > performance improvements in Mercurial. > > * A Mercurial 3.4 or older client isn't capable of receiving the server's > native storage format over the wire protocol so the *server* has to > normalize things to a format the client can understand. This slows down `hg > clone` and `hg pull` operations drastically (45+ minute clone times > regardless of network speed) and can create load problems on the server. > (This is why we waited a while before performing this server-side change > and is why we'll have to lock out legacy clients if this load becomes a > problem.) > The increased clone times on legacy clients has already broken SeaMonkey automation. It is likely other automated consumers will also break if they have timeouts in play. Normally I'd be very sympathetic to this. However, Mercurial 3.7.3 (released ~1 year ago) has fixes for a number of CVEs, including one that could lead to code execution when pulling/updating to maliciously crafted data. While hg.mozilla.org should not serve such malicious data, the security issue is severe enough that all clients should have upgraded already. To be blunt, it is difficult to sympathize with your position if you are running insecure software. So, consider this degraded performance a wake-up call that you should upgrade to secure software that also happens to not exhibit performance problems. Now, there are is a small set of people running older Mercurial versions via distro packages that have backported the security fixes from 3.7.3. You have a legitimate reason to complain. While I would highly prefer you upgrade to a modern Mercurial to fix the performance regressions, you can work around the slow clone times by running `hg clone --uncompressed`. If any part of Mozilla automation does this, assume someone will eventually hunt you down because you should be running a modern Mercurial version for general performance reasons. Failure to do so means automation is running slower than it could and you are depriving developers and end-users of faster results. > > In addition, a Mercurial 4.1+ client will use zstandard instead of zlib > for compression, resulting in faster compression/decompression and fewer > bytes transferred. This translates to faster operations against > hg.mozilla.org. > > Why We Made This Change > ----------------------- > > Many of you may be asking: "since this change slows clients down, why make > it?" Good question. The viable choices were between doing nothing and not > allowing peak performance or making a 1-time change that enabled faster > operations while temporarily inconveniencing (to various extents) existing > clones and their users (namely Firefox developers). In other words, it was > about letting the past and legacy software continue to hinder us or to > break free and move to the future. > > With Firefox and Mozilla, we have to play the long game. And, we need to > enable developers to move fast. This includes things like Firefox's > automation. This change will enable VCS interactions in automation to > complete faster which will reduce end-to-end times - something many > complain about. > > We could have made this change over a year ago. But we waited for better > client-side adoption so the impact of this transition would be mitigated. > We rolled out repositories like mozilla-unified (that have been encoded in > the optimal format since day 0) and encouraged their use. > > There would never be an optimal time to make this transition: someone > would always lose. In my mind at least, we had postponed allowing clients > to achieve optimal performance for long enough (over a year) and it was > time to make the transition. So we did. > > Additional Work and Related Bugs > -------------------------------- > > Bug 1351859 is the bug tracking upgrading all repos to the latest storage > format. If you notice any fallout from this work, please track against it. > > There may be issues with the pre-generated zstd bundles and 32-bit > Python/Mercurial processes [on Windows] running out of memory. That's bug > 1344790. If you run into a memory error, run `hg --config > ui.clonebundleprefers=COMPRESSION=gzip clone ...` as a workaround. > > [1] https://mozilla-version-control-tools.readthedocs.io/en/ > latest/hgmozilla/installing.html > <https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmozilla/common.html#upgrading-repository-storage> > [2] https://mozilla-version-control-tools.readthedocs.io/en/late > st/hgmozilla/common.html#upgrading-storage-via-debugupgraderepo > [3] https://mozilla-version-control-tools.readthedocs.io/en/late > st/hgmozilla/common.html#upgrading-repository-storage > > > <https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmozilla/common.html#upgrading-repository-storage> > _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
