Cloning from Pre-Generated Bundles¶
hg.mozilla.org supports offloading clone requests to pre-generated
bundle files stored in a CDN and Amazon S3. This results in drastically
reduced server load (which helps prevent outages due to accidental,
excessive load) and frequently results in faster clone times.
How It Works¶
When a Mercurial client clones a repository, it looks to see if the
server is advertising a list of available, pre-generated bundle files.
If it is, it looks at the list, finds the most appropriate entry,
downloads and applies that bundle, then does the equivalent of an
pull against the original Mercurial server to fetch new data since the
time the bundle file was produced. The end result is a faster clone with
drastically reduced load on the Mercurial server.
If you are running Mercurial 3.7 or newer, support for cloning from pre-generated bundles is built-in to Mercurial itself and enabled by default.
If you are running Mercurial 3.6, support is built-in but requires enabling a config option:
[experimental] clonebundles = true
If you are running a Mercurial older than 3.6, upgrade to leverage the clone bundles feature.
Mercurial 4.1 is required to support zstd bundles, which are smaller and faster than bundles supported by earlier versions.
hg.mozilla.org will advertise multiple bundles/URLs for each repository. Each listing varies by:
- Bundle type
- Server location
By default, Mercurial uses the first entry in the server-advertised bundles list that the client supports.
The clone bundles feature allows the client to define preferences of which bundles to fetch. The way this works is the client defines some key-value pairs in its config and bundles having these attributes will be upweighted.
Bundle Attributes on hg.mozilla.org¶
hg.mozilla.org, following attributes are defined in the manifest:
This defines the type of bundle.
We currently generate bundles with the following specifications:
- Indicates whether the URL requires SNI (a TLS extension). This is set
truefor URLs where multiple certificates are installed on the same IP and SNI is required. It is undefined if SNI is not required.
- The EC2 region the bundle file should be served from. We support
eu-central-. You should prefer the region that is closest to you.
- The GCE region the bundle file should be served from. We only support
us-central1at this time.
- Indicates whether the URL is on a CDN. Value is
trueto indicate the URL is a CDN. All other values or undefined values are to be interpretted as not a CDN.
Here is an example clone bundles manifest:
https://hg.cdn.mozilla.net/mozilla-unified/82c75fd3a2de796351296592c459ab4aa4cd0baf.zstd-max.hg BUNDLESPEC=zstd-v2 REQUIRESNI=true cdn=true https://s3-us-west-2.amazonaws.com/moz-hg-bundles-us-west-2/mozilla-unified/82c75fd3a2de796351296592c459ab4aa4cd0baf.zstd-max.hg BUNDLESPEC=zstd-v2 ec2region=us-west-2 https://s3-us-west-1.amazonaws.com/moz-hg-bundles-us-west-1/mozilla-unified/82c75fd3a2de796351296592c459ab4aa4cd0baf.zstd-max.hg BUNDLESPEC=zstd-v2 ec2region=us-west-1 https://s3-external-1.amazonaws.com/moz-hg-bundles-us-east-1/mozilla-unified/82c75fd3a2de796351296592c459ab4aa4cd0baf.zstd-max.hg BUNDLESPEC=zstd-v2 ec2region=us-east-1 https://s3-eu-central-1.amazonaws.com/moz-hg-bundles-eu-central-1/mozilla-unified/82c75fd3a2de796351296592c459ab4aa4cd0baf.zstd-max.hg BUNDLESPEC=zstd-v2 ec2region=eu-central-1 https://hg.cdn.mozilla.net/mozilla-unified/82c75fd3a2de796351296592c459ab4aa4cd0baf.gzip-v2.hg BUNDLESPEC=gzip-v2 REQUIRESNI=true cdn=true https://s3-us-west-2.amazonaws.com/moz-hg-bundles-us-west-2/mozilla-unified/82c75fd3a2de796351296592c459ab4aa4cd0baf.gzip-v2.hg BUNDLESPEC=gzip-v2 ec2region=us-west-2 https://s3-us-west-1.amazonaws.com/moz-hg-bundles-us-west-1/mozilla-unified/82c75fd3a2de796351296592c459ab4aa4cd0baf.gzip-v2.hg BUNDLESPEC=gzip-v2 ec2region=us-west-1 https://s3-external-1.amazonaws.com/moz-hg-bundles-us-east-1/mozilla-unified/82c75fd3a2de796351296592c459ab4aa4cd0baf.gzip-v2.hg BUNDLESPEC=gzip-v2 ec2region=us-east-1 https://s3-eu-central-1.amazonaws.com/moz-hg-bundles-eu-central-1/mozilla-unified/82c75fd3a2de796351296592c459ab4aa4cd0baf.gzip-v2.hg BUNDLESPEC=gzip-v2 ec2region=eu-central-1 https://hg.cdn.mozilla.net/mozilla-unified/82c75fd3a2de796351296592c459ab4aa4cd0baf.packed1-gd.hg BUNDLESPEC=none-packed1;requirements%3Dgeneraldelta%2Crevlogv1 REQUIRESNI=true cdn=true https://s3-us-west-2.amazonaws.com/moz-hg-bundles-us-west-2/mozilla-unified/82c75fd3a2de796351296592c459ab4aa4cd0baf.packed1-gd.hg BUNDLESPEC=none-packed1;requirements%3Dgeneraldelta%2Crevlogv1 ec2region=us-west-2 https://s3-us-west-1.amazonaws.com/moz-hg-bundles-us-west-1/mozilla-unified/82c75fd3a2de796351296592c459ab4aa4cd0baf.packed1-gd.hg BUNDLESPEC=none-packed1;requirements%3Dgeneraldelta%2Crevlogv1 ec2region=us-west-1 https://s3-external-1.amazonaws.com/moz-hg-bundles-us-east-1/mozilla-unified/82c75fd3a2de796351296592c459ab4aa4cd0baf.packed1-gd.hg BUNDLESPEC=none-packed1;requirements%3Dgeneraldelta%2Crevlogv1 ec2region=us-east-1 https://s3-eu-central-1.amazonaws.com/moz-hg-bundles-eu-central-1/mozilla-unified/82c75fd3a2de796351296592c459ab4aa4cd0baf.packed1-gd.hg BUNDLESPEC=none-packed1;requirements%3Dgeneraldelta%2Crevlogv1 ec2region=eu-central-1
As you can see, listed bundle URLs vary by bundle type (compression and format) and location. For each repository we generate bundles for, we generate:
- A zstd bundle (either default compression or maximum compression depending on repo utilization)
- A gzip bundle (the default compression format)
- A streaming bundle file (larger but faster)
For each of these bundles, we upload them to the following locations:
- CloudFront CDN
- S3 in us-west-2 region
- S3 in us-west-1 region
- S3 in us-east-1 region
- S3 in eu-central-1 region
Which Bundles to Prefer¶
The zstd bundle hosted on CloudFront is the first entry and is thus preferred by clients by default.
zstd bundles are the smallest bundles and for most people they are the ideal bundle to use.
Mercurial 4.1 is required to use zstd bundles. If an older Mercurial client is used, larger, non-zstd bundles will be used.
If you have a super fast internet connection, you can prefer the packed/streaming bundles. This will transfer 30-40% more data on average, but will require almost no CPU to apply. If you can fetch from S3 or CloudFront at 1 Gbps speeds, you should be able to clone Firefox in under 60s.:
# HG 3.7+ [ui] clonebundleprefers = VERSION=packed1 # HG 3.6 [experimental] clonebundleprefers = VERSION=packed1
Manifest Advertisement to AWS Clients¶
If a client in Amazon Web Services (e.g. EC2) is requesting a bundle manifest and that client is in an AWS region where bundles are hosted in S3, the advertised manifest will only show S3 URLs for the same AWS region. In addition, stream clone bundles are the highest priority bundle.
This behavior ensures that AWS transfer are intra-region (which means
they are fast and don’t result in a billable AWS event) and that
clone completes as fast as possible (stream clone bundles are faster
than gzip bundles).
If you have machinery in an AWS region where we don’t host bundles, please let us know. There’s a good chance that establishing bundles in your region is cheaper than paying the cross-region transfer costs (intra-region transfer is free).
Manifest Advertisements to Mozilla Offices¶
If the client request appears to originate from a Mozilla office network, we make the assumption that the network speed and bandwidth are sufficient to always prefer the high-speed streamed clone bundles.
Which Repositories Have Bundles Available¶
Bundles are automatically generated for repositories that are high volume (in terms of repository size and clone frequency) or have a need for bundles.
If you think bundles should be made available for a particular
repository, let a server operator know by filing a
Developer Services :: hg.mozilla.org bug or by asking in #vcs