moz.build Metadata

hg.mozilla.org has the ability to extract and render metadata from moz.build files.

Web Command

On repositories that have this feature enabled, a new web command is available: mozbuildinfo. Requests to this web command have the form:

/<repo>/json-mozbuildinfo[/<rev>][?p=<path1>[&p=<path2>]]

By default, requests return moz.build info for all files changed by the current tip changeset. To specify which changeset should be returned, the hash of that changeset can be specified as a URL component. To change which files have their metadata resolved, pass the p query string argument 1 or more times with the filename(s) you are interested in.

The response from this web command is JSON. It looks something like:

{
  "aggregate": {
    "bug_component_counts": [
      [
        [
          "Core",
          "Build Config"
        ],
        1
      ]
    ],
    "recommended_bug_component": [
      "Core",
      "Build Config"
    ]
  },
  "files": {
    "Makefile.in": {
      "bug_component": [
        "Core",
        "Build Config"
      ]
    }
  }
}

The JSON object typically contains a files object containing per-file metadata and an aggregate object containing, well, aggregate data from all files.

If an error occurs, the object will have an error property containing a string error message.

The data inside the JSON is generated by the mozbuild Python package, whose canonical home lives in the python/mozbuild directory of mozilla-central. The mozbuild.frontend.context module is likely of the most interest.

Security of moz.build Evaluation

moz.build files are Python files. By evaluating moz.build files checked into version control, we are essentially enabling remote code execution. The security of the moz.build evaluation implementation is thus critically important.

It is essentially impossible to properly sandbox Python code from within (C)Python itself. Many have tried. They have all failed. So while moz.build files themselves are executed within a limited sandbox, this sandbox only reduces functionality initially available to moz.build files and doesn’t comprise a security sandbox.

When the mozbuildinfo web command is requested, the following occurs on the server:

  1. A WSGI process running as the hg user receives the request.
  2. The mozbuildinfo web command handler is invoked
  3. We verify that moz.build evaluation is enabled for the repository (it isn’t enabled by default).
  4. We verify that a wrapper script is installed (we don’t allow executing moz.build files in the same process as the WSGI handler).
  5. We verify additional request parameters.
  6. We invoke the mozbuild-eval executable using sudo, passing JSON describing the request to it. The executable initially runs as root.
  7. mozbuild-eval calls the clone() system call to create a new process with new namespaces for most Linux primitives (IPC, network, mount, pid, UTC).
  8. The new process is moved to a special control group which has resource and device access limits in place.
  9. The mounts in the new process are reset and all mounts besides a read-only loopback mount containing repository contents are removed. This includes procfs and /dev.
  10. We chroot() into a new directory, which contains only the Python code needed to evaluate moz.build files.
  11. The real, effective, and saved GID and UID of the process are changed to a low-privileged user.
  12. We fork a new Python process to evaluate the moz.build files metadata.

The process evaluating moz.build files:

  • Is running as a regular user/group
  • That doesn’t have write access to repositories via normal filesystem permissions
  • That doesn’t have write access to repositories via a read-only bind mount of the repositories
  • Is executing inside a chroot with minimal files available
  • Is executing in a control group that doesn’t allow access to any devices except urandom.
  • Is executing in a control group that limits CPU, memory, and I/O usage
  • Doesn’t have access to procfs
  • Doesn’t have a handle on any mounts from the host except the bind mount