Mozilla collects and republishes GitHub Webhooks for a number of Mozilla’s organizations and projects.
GitHub Webhooks are configured at the organizational or project level
application/json payloads to
These HTTP requests are delivered to an Amazon API Gateway service operated by the Developer Productivity team. Each webhook request invokes an AWS Lambda function which does the following:
- Publishes the record to an AWS Kinesis Firehose
- Publishes the record to an all AWS SNS topic and optionally a a public AWS SNS topic if the event is non-private.
Data published to the Kinesis Firehose is flushed to Amazon S3 for long-term storage and to facilitate analytics.
Additional AWS Lambda functions consume the public SNS topic and republish events to other channels, such as Pulse.
While the ingestion server often receives all events for an organization or repository, not all events are republished in public channels.
The following events are excluded from the public:
- Any event belonging to a private repository
- Team membership changes (
- Transition of repository from private to public (
- Repository creation, deletion, or public/private transitions (
- Any new events GitHub adds that aren’t in a list of allowed events
Pulse is a RabbitMQ exchange operated by Mozilla that serves as a nexus of event publishing for various systems.
GitHub Webhook events are republished to the exchange/github-webhooks/v1 exchange.
The routing key for each message is of the form
<repository> is the GitHub account/organization + repository
<event> is the GitHub event name. e.g.
The JSON message published to pulse has the following relevant keys:
- GitHub event name. e.g.
- UUID uniquely identifying this message. The ID is generated by GitHub.
- The payload of the GitHub event. The formats are documented at https://developer.github.com/v3/activity/events/types/.
Delivery of GitHub events to Pulse is best effort. If Pulse is down, data may fail to publish.
Non-private GitHub events are published to the
arn:aws:sns:us-west-2:699292812394:github-webhooks-public AWS SNS topic.
Kinesis Firehose and S3 Access¶
Access to the streaming GitHub data in Kinesis Firehose and the historical
data retained in S3 can be granted on a per-case basis. If interested,