Runbook recipes
The following are recommended runbook architectures for the categories outlined in Runbook Best Practices.
In the runbook recipes steps are formatted as Step Category > Step Description. Unless otherwise noted, the recommended default step execution rules are: Always execute and Execute automatically. You may wish to modify this according to your business rules.
Note: You will need to have the desired integration configured before you can add an integration-driven runbook step. Runbook conditions are available to paid FireHydrant plans only. If you're on a free plan, you can specify to execute those steps manually or choose not to include them.
Default Runbook
Execution rule: Always attach
Recommended Runbook Steps
- Create a dedicated incident channel.
- Slack > Create Incident Channel
- Create a video conference bridge. [select one]
- Google Meet > Create a Google Meet
- Webex > Create a Webex meeting
- Zoom > Create a Zoom meeting
- Create an incident ticket in your project management ticketing system. [select one]
- Jira Cloud > Create a Jira Cloud Issue
- Jira Server (On Prem) > Create a Jira Server Issue
- Shortcut > Create a Shortcut Issue
- Export the retrospective to a document collaboration platform. [select one]
- Confluence Cloud > Export Retrospective to Confluence Cloud
- Conditions & Scheduling: Automatically execute the step when the Current milestone is Retrospective Started (or Retrospective Completed)
- Google Docs > Export Retrospective to Google Cloud
- Conditions & Scheduling: Automatically execute the step when the Current milestone is Retrospective Started (or Retrospective Completed)
- Archive the Incident Channel.
- Slack > Archive the Incident Channel
- Conditions & Scheduling: Automatically execute the step when the Current milestone is Retrospective Completed.
- Slack > Archive the Incident Channel
- Confluence Cloud > Export Retrospective to Confluence Cloud
SEV-1 (or equivalent) Incident Runbook
Execution rule:Â Â Execute when Current Severity is SEV-1 (or equivalent)Â
Recommended Runbook Steps
-
Remind responders to provide high frequency internal communication updates. For example:
-
Step: Slack > Notify Incident Channel with a Custom Message
- Conditions & Scheduling
- Current milestone is one of: Started, Detected, Acknowledged, Investigating, Identified, Mitigated
- Execute automatically every 30 minutes
- Conditions & Scheduling
-
-
Assign the appropriate incident roles.
- Step: FireHydrant > Assign a Role
-
Publish incident status updates to the internal status page.
- Step: FireHydrant Status Page > Publish to status page Â
- Conditions & Scheduling: Automatically execute the step when the Current Milestone is [insert key milestones] (e.g. Acknowledged, Investigating, Mitigated, etc.)
- Step: FireHydrant Status Page > Publish to status page Â
-
Publish the incident and send updates to the external status page. (e.g. Statuspage.io)
- Step: Statuspage > Creates a Statuspage.io incident
- Step: Statuspage > Updates a Statuspage.io incident
- Conditions & Scheduling: Automatically execute the step when the Current Milestone is [insert key milestones] (e.g. Investigating, Mitigated, etc.)
-
Attach a task list for each critical assigned role.
- Prerequisite: Task lists defined
- Step: FireHydrant > Add Task List
-
Notify other relevant Slack channels with periodic incident status updates.
- Prerequisite:Â The FireHydrant app must be added to the Slack channels for FireHydrant to post your message. Please make sure to
/invite @FireHydrant
to any channels you list in the configuration. - Step: Slack > Notify channel with a custom message
- Add relevant team channels (e.g. #support)
- Conditions & Scheduling
- Current milestone is one of: Started, Detected, Acknowledged, Investigating, Identified, Mitigated
- Execute automatically every 30 minutes
- Prerequisite:Â The FireHydrant app must be added to the Slack channels for FireHydrant to post your message. Please make sure to
SEV-2 (or equivalent) Incident Runbook
Execution rule:Â Â Execute when Current Severity is SEV-2 (or equivalent)Â
Recommended Runbook Steps
- Remind responders to provide medium frequency internal communication updates. For example:
- Step: Slack > Notify Incident Channel with a Custom Message
- Current milestone is one of: Started, Detected, Acknowledged, Investigating, Identified, Mitigated
- Execute automatically every 6_0 minutes_
- Step: Slack > Notify Incident Channel with a Custom Message
- Publish incident status updates to the internal status page.
- Step: FireHydrant Status Page > Publish to status page
- Conditions and Scheduling: Automatically execute the step when the Current Milestone is [insert key milestones] (e.g. Acknowledged, Investigating, Mitigated, etc.)
- Step: FireHydrant Status Page > Publish to status page
- Attach a general task list for the incident.
- Prerequisite: Task list defined
- Step: FireHydrant > Add Task List
- Notify other relevant Slack channels with periodic incident status updates until the incident is resolved.
- Prerequisite:Â The FireHydrant app must be added to the Slack channels for FireHydrant to post your message. Please make sure to
/invite @FireHydrant
to any channels you list in the configuration. - Step: Slack > Notify channel with a custom message
- Add relevant team channels (e.g. #support)
- Condition and scheduling:
- Execute automatically
- Repeat every 60 minutes
- When the Current milestone is one of: Started, Detected, Acknowledged, Investigating, Identified, Mitigated
- Prerequisite:Â The FireHydrant app must be added to the Slack channels for FireHydrant to post your message. Please make sure to
Advanced Runbooks
Service-based Runbook
- Add a team
- Step: FireHydrant > Assign A Team
- Page out to an alerting provider
- Prerequisite: Alerting provider configured
- As appropriate for your provider setup:
- Step: Opsgenie > Create OpsGenie Incident
- Step: PagerDuty > Create PagerDuty Incident
- Step: VictorOps > Create VictorOps Incident
- Notify incident channel of available documentation
- Step: Slack > Notify incident channel with a custom message
- Add the url to your documentation in the message text.
- Step: Slack > Notify incident channel with a custom message
- Tasklist with specific steps to be taken to restore the service based on condition
- Prerequisite: Task list defined
- Step: FireHydrant > Add Task List
- Select the task list.
- Run a custom script
- Step: FireHydrant > Script
- Add your description and custom script to the runbook step configuration.
- Step: FireHydrant > Script
Team-based Runbook
-
Notify a team’s home channel of engagement
- Step: Slack > Notify channel
- Comma Separated Channels: [#team-channel]
- Step: Slack > Notify channel
-
Add a Role
- Step: FireHydrant > Assign a Role
- Add the role and user to the runbook step configuration.
- Step: FireHydrant > Assign a Role
-
Email Manager
- Firehydrant > Send an email notification
- Add the manager's email address, subject, and message to the runbook step configuration.
- Firehydrant > Send an email notification
-
Reminder to team’s home channel for long-running incidents
- Step: Slack > Notify incident channel with a custom message
- Your Message: [Friendly reminder to post an update on the incident status page(s)]
- Action button: Post an update
- Comma Separated Channels: [#team-channel]
- Conditions and Scheduling:
- E** xecute automatically**
- Repeat every 60 minutes
- When the Time since incident opened: greater than 5 hours
- Step: Slack > Notify incident channel with a custom message
-
Tasklist
- Prerequisite: Task list defined
- Step: FireHydrant > Add Task List
- Select the task list.
Customer Facing Issue Runbook
- Let the customer support team know an incident has been opened.
- Slack > Notify channel
- Comma Separated Channels: [#support]
- Slack > Notify channel
- Send reminders to update customer-facing information while the incident is open.
- Slack > Notify incident channel with a custom message
- Your Message: [Friendly reminder to post an update to the external status page.]
- Action button: Post an update
- Condition and scheduling:
- Execute automatically
- Repeat every 60 minutes
- When the Current milestone is one of: Started, Detected, Acknowledged, Investigating, Identified, Mitigated
- Slack > Notify incident channel with a custom message