Integrating with PagerDuty
Our PagerDuty integration enables various functions to improve the efficiency of your incident management. You can:
- Automatically create FireHydrant incidents from PagerDuty alerts
- Notify different channels based on incoming alert content
- Page out to teams and service in PagerDuty from FireHydrant
- Automatically pull in on-call responders from PagerDuty's escalation policies to incident channels
First, you need a PagerDuty API key.
- Go to your PagerDuty dashboard and navigate to the page where you can set up API keys. (This will be something like Integrations > API Access Keys ).
- On the API Access page, click Create New API Key and provide a description for the API key.
- Click Create Key.
- Copy the API Key that PagerDuty generates. Then go to FireHydrant's integrations page.
- Click the PagerDuty integration card, then click Setup PagerDuty.
- Enter the PagerDuty API key that you copied into the API Token field.
- For Domain, we default to showing
https://api.pagerduty.com, which is the domain for PagerDuty's US service region. If you are a company based out of the EU and using PagerDuty's EU service region, then you will need to modify that value to
https://api.eu.pagerduty.com. For more information, visit PagerDuty's documentation.
- Click Authorize Application.
Note: If you plan to link or import services from PagerDuty, do not configure this outgoing webhook and skip this section. When you link or import PagerDuty Services to FireHydrant services, we automatically create service-scoped hooks in PagerDuty.
After configuring the connection, you will lastly need to configure an outgoing webhook from PagerDuty so FireHydrant can both sync incident states as well as receive webhooks on alerts.
- In the integration settings page for PagerDuty on FireHydrant, copy the webhook address provided.
- In PagerDuty, Head to Integrations > Generic Webhooks and click "+ New Webhook."
- In Webhook URL paste in the value you copied from FireHydrant.
- For Scope Type choose "Account".
- For Event Subscription, you can uncheck all the
service.*events, leaving only the incident updates.
- Click "Add Webhook".
To test this, go ahead and create an incident in FireHydrant. This will, with default rules in FireHydrant, log an alert processing message.
After PagerDuty is installed, you also have the capability to link and/or import PagerDuty services into FireHydrant's services. This enables multiple capabilities, including:
- From Slack, directly paging out to teams or finding who's on call for a particular service
- Automatically paging owners of a particular service when it's marked impacted in a FireHydrant incident
- Head over to Service Catalog > Services and click the Add services dropdown and Import from third party.
- Click the Import button next to PagerDuty.
- From here, you can import all of your services, or you can load all of the available services in PagerDuty and selectively pick which ones you'd like to import or link.
- Tick the checkbox next to services you want to import/link, and then in the dropdown box, you can specify two choices:
- Create New - Will create a new entry in the FireHydrant service catalog with the same name as the PagerDuty service. This will also automatically link the new FireHydrant service and the PagerDuty Service.
- Link to Existing - If you already have services in FireHydrant, you can choose which one you'd like to link this PagerDuty service to.
When you link services between FireHydrant and PagerDuty, FireHydrant automatically creates Webhooks in PagerDuty for each linked service.
When using the
/fh page command in Slack, you can send pages to any user or esclation policy in your Pager Duty account. For more information, see Page & Lookup On-Call.
Via Slack, FireHydrant gives you the capability to directly page the teams that respond to a service through PagerDuty. To make use of this, there are two prerequisites:
- You must complete the section/steps above to link PagerDuty services to FireHydrant services - FireHydrant relies on this link to find the correct on-call/escalation schedule for a particular service.
- You must fill in the Default User Email field in the integration settings. This is because PagerDuty requires a user as part of each request's header to log who the request is on behalf of.
We recommend using a service account in PagerDuty that you create to represent FireHydrant as a whole.
Once you complete the above steps, you should be able to run
/fh page service [NAME] within Slack, or pull up a list of services to page with just
/fh page service.
For more information, see Page & Lookup On-Call.
Now that you've configured the PagerDuty integration, you can take the next steps to get the full power of FireHydrant + PagerDuty:
- Configure teams to pull in users from on-call schedules.
- Configuring Alert Routing to automatically open incidents or post notifications from inbound alerts.
- Automatically create PagerDuty incidents via Runbook step
Once your PagerDuty instance is configured, you can setup Alert Routes to take action on your alerts based on the data included in the alert. You can automatically open new incidents, send alerts to any Slack channel, log an alert in FireHydrant, or simply ignore it. To learn more, read about Alert Routes.
Here is the table of routable parameters on FireHydrant and the corresponding key/value from the inbound PagerDuty webhook(s). The
$ refers to the webhook body content as a JSON object.
An explanation of PagerDuty's Webhook content can be found in their docs here.
|PagerDuty Webhook Body
|PagerDuty: Event Type
|The type of event sent by PagerDuty
|PagerDuty: Occurred At
|The datetime of when the event occurred according to PagerDuty
|PagerDuty: Alert ID
|The ID of the alert in PagerDuty
|PagerDuty: Incident ID
|The ID of the incident in PagerDuty
|PagerDuty: Incident Summary
|Short summary of the PagerDuty incident
|PagerDuty: Data Type
|The type of data, e.g., 'incident' or 'incident_note', present in this message
|PagerDuty: Alert Web URL
|URL to alert page in PagerDuty
|PagerDuty: Incident Web URL
|URL to incident page in PagerDuty
|PagerDuty: Incident Priority
|Incident priority in PagerDuty. Must enable priorities in PagerDuty to use this.
|PagerDuty: Incident Status
|The status of the incident in PagerDuty. Is one of
|PagerDuty: Incident Note Content
|The content of a note posted to an incident in PagerDuty. Only shows up for
|PagerDuty: Incident Title
|Name of the incident. Essentially the same as Incident Summary.
The following table is a mapping of our overall Alert Routing mapping object - these parameters are common across all Alert Routing/Monitoring integrations.
|PagerDuty Webhook Body
|The same as PagerDuty: Incident Summary above.
|Maps to nothing for PagerDuty. PagerDuty has a Description field in their alerts, but oddly they do not surface it in the webhook, so FireHydrant has no way of accessing it.
|The same as PagerDuty: Incident Priority above.
|Almost the same as PagerDuty: Incident Status above. However, FireHydrant expects
opened instead of
triggered to align with internal terminology.
|Alert Associated Infrastructure
|The associated Service/Functionality in FireHydrant. In order for us to detect this, you must have linked a Service or Functionality to a service in PagerDuty.