Integrating with PagerDuty

Our PagerDuty integration enables various functions to improve the efficiency of your incident management. You can:

  • Automatically create FireHydrant incidents from PagerDuty alerts
  • Notify different channels based on incoming alert content
  • Page out to teams and service in PagerDuty from FireHydrant
  • Automatically pull in on-call responders from PagerDuty's escalation policies to incident channels

Installation

First, you need a PagerDuty API key.

  1. Go to your PagerDuty dashboard and navigate to the page where you can set up API keys. (This will be something like Integrations > API Access Keys ).
Creating an API key in PagerDuty
Creating an API key in PagerDuty
  1. On the API Access page, click Create New API Key and provide a description for the API key.
  2. Click Create Key.
  3. Copy the API Key that PagerDuty generates. Then go to FireHydrant's integrations page.
  4. Click the PagerDuty integration card, then click Setup PagerDuty.
  5. Enter the PagerDuty API key that you copied into the API Token field.
  6. For Domain, we default to showing https://api.pagerduty.com, which is the domain for PagerDuty's US service region. If you are a company based out of the EU and using PagerDuty's EU service region, then you will need to modify that value to https://api.eu.pagerduty.com. For more information, visit PagerDuty's documentation.
  7. Click Authorize Application.

Setup Outgoing Webhook

Note: If you plan to link or import services from PagerDuty, do not configure this outgoing webhook and skip this section. When you link or import PagerDuty Services to FireHydrant services, we automatically create service-scoped hooks in PagerDuty.

After configuring the connection, you will lastly need to configure an outgoing webhook from PagerDuty so FireHydrant can both sync incident states as well as receive webhooks on alerts.

Get the webhook URL after setting up PagerDuty
Get the webhook URL after setting up PagerDuty
  1. In the integration settings page for PagerDuty on FireHydrant, copy the webhook address provided.
  2. In PagerDuty, Head to Integrations > Generic Webhooks and click "+ New Webhook."
  3. In Webhook URL paste in the value you copied from FireHydrant.
  4. For Scope Type choose "Account".
  5. For Event Subscription, you can uncheck all the service.* events, leaving only the incident updates.
  6. Click "Add Webhook".

To test this, go ahead and create an incident in FireHydrant. This will, with default rules in FireHydrant, log an alert processing message.

Webhook logged after PagerDuty alert created
Webhook logged after PagerDuty alert created

Linking PagerDuty Services to FireHydrant Services

After PagerDuty is installed, you also have the capability to link and/or import PagerDuty services into FireHydrant's services. This enables multiple capabilities, including:

  • From Slack, directly paging out to teams or finding who's on call for a particular service
  • Automatically paging owners of a particular service when it's marked impacted in a FireHydrant incident

How to link/import services

  1. Head over to Service Catalog > Services and click the Add services dropdown and Import from third party.
  2. Click the Import button next to PagerDuty.
  3. From here, you can import all of your services, or you can load all of the available services in PagerDuty and selectively pick which ones you'd like to import or link.
Importing/linking services from PagerDuty to FireHydrant
Importing/linking services from PagerDuty to FireHydrant
  1. Tick the checkbox next to services you want to import/link, and then in the dropdown box, you can specify two choices:
    1. Create New - Will create a new entry in the FireHydrant service catalog with the same name as the PagerDuty service. This will also automatically link the new FireHydrant service and the PagerDuty Service.
    2. Link to Existing - If you already have services in FireHydrant, you can choose which one you'd like to link this PagerDuty service to.

When you link services between FireHydrant and PagerDuty, FireHydrant automatically creates Webhooks in PagerDuty for each linked service.

Webhooks created in PagerDuty when linking services
Webhooks created in PagerDuty when linking services

Paging Users and Escalation Policies

When using the /fh page command in Slack, you can send pages to any user or esclation policy in your Pager Duty account. For more information, see Page & Lookup On-Call.

Paging Teams via Services

Via Slack, FireHydrant gives you the capability to directly page the teams that respond to a service through PagerDuty. To make use of this, there are two prerequisites:

  1. You must complete the section/steps above to link PagerDuty services to FireHydrant services - FireHydrant relies on this link to find the correct on-call/escalation schedule for a particular service.
  2. You must fill in the Default User Email field in the integration settings. This is because PagerDuty requires a user as part of each request's header to log who the request is on behalf of.

We recommend using a service account in PagerDuty that you create to represent FireHydrant as a whole.

PagerDuty default user email
PagerDuty default user email

Once you complete the above steps, you should be able to run /fh page service [NAME] within Slack, or pull up a list of services to page with just /fh page service.

For more information, see Page & Lookup On-Call.

Next Steps

Now that you've configured the PagerDuty integration, you can take the next steps to get the full power of FireHydrant + PagerDuty:


PagerDuty Alert Routing

PagerDuty alert routing
PagerDuty alert routing

Once your PagerDuty instance is configured, you can setup Alert Routes to take action on your alerts based on the data included in the alert. You can automatically open new incidents, send alerts to any Slack channel, log an alert in FireHydrant, or simply ignore it. To learn more, read about Alert Routes.

Parameter Mappings

Here is the table of routable parameters on FireHydrant and the corresponding key/value from the inbound PagerDuty webhook(s). The $ refers to the webhook body content as a JSON object.

An explanation of PagerDuty's Webhook content can be found in their docs here.

Parameter Name PagerDuty Webhook Body Notes
PagerDuty: Event Type $.event.event_type The type of event sent by PagerDuty
PagerDuty: Occurred At $.event.occurred_at The datetime of when the event occurred according to PagerDuty
PagerDuty: Alert ID $.event.data.id The ID of the alert in PagerDuty
PagerDuty: Incident ID $.event.data.incident.id The ID of the incident in PagerDuty
PagerDuty: Incident Summary $.event.data.title Short summary of the PagerDuty incident
PagerDuty: Data Type $.event.data.type The type of data, e.g., 'incident' or 'incident_note', present in this message
PagerDuty: Alert Web URL $.event.data.html_url URL to alert page in PagerDuty
PagerDuty: Incident Web URL $.event.data.incident.html_url URL to incident page in PagerDuty
PagerDuty: Incident Priority $.event.data.priority.summary Incident priority in PagerDuty. Must enable priorities in PagerDuty to use this.
PagerDuty: Incident Status $.event.data.status The status of the incident in PagerDuty. Is one of triggered, acknowledged, and resolved.
PagerDuty: Incident Note Content $.event.data.content The content of a note posted to an incident in PagerDuty. Only shows up for incident.annotated events.
PagerDuty: Incident Title $.event.data.title Name of the incident. Essentially the same as Incident Summary.

The following table is a mapping of our overall Alert Routing mapping object - these parameters are common across all Alert Routing/Monitoring integrations.

Parameter Name PagerDuty Webhook Body Notes
Alert Summary $.event.data.title The same as PagerDuty: Incident Summary above.
Alert Description N/A Maps to nothing for PagerDuty. PagerDuty has a Description field in their alerts, but oddly they do not surface it in the webhook, so FireHydrant has no way of accessing it.
Alert Priority $.event.data.priority.summary The same as PagerDuty: Incident Priority above.
Alert Status $.event.data.status Almost the same as PagerDuty: Incident Status above. However, FireHydrant expects opened instead of triggered to align with internal terminology.
Alert Associated Infrastructure N/A The associated Service/Functionality in FireHydrant. In order for us to detect this, you must have linked a Service or Functionality to a service in PagerDuty.

Last updated on 2/9/2024