Search Results

Start for free

Runbook recipes

The following are recommended runbook architectures for the categories outlined in Runbook Best Practices.

In the runbook recipes steps are formatted as Step Category > Step Description.  Unless otherwise noted, the recommended default step execution rules are: Always execute and Execute automatically.  You may wish to modify this according to your business rules.

Note: You will need to have the desired integration configured before you can add an integration-driven runbook step.  Runbook conditions are available to paid FireHydrant plans only. If you're on a free plan, you can specify to execute those steps manually or choose not to include them.

Default Runbook

Execution rule: Always attach

SEV-1 (or equivalent) Incident Runbook

Execution rule:  Execute when Current Severity is SEV-1 (or equivalent) 

  • Remind responders to provide high frequency internal communication updates. For example:

  • Assign the appropriate incident roles.

  • Publish incident status updates to the internal status page.

    • Step: FireHydrant Status Page > Publish to status page  
      • Conditions & Scheduling: Automatically execute the step when the Current Milestone is [insert key milestones] (e.g. Acknowledged, Investigating, Mitigated, etc.)

  • Publish the incident and send updates to the external status page. (e.g. Statuspage.io)

  • Attach a task list for each critical assigned role.

  • Notify other relevant Slack channels with periodic incident status updates.

    • Prerequisite: The FireHydrant app must be added to the Slack channels for FireHydrant to post your message. Please make sure to /invite @FireHydrant to any channels you list in the configuration.
    • Step: Slack > Notify channel with a custom message
      • Add relevant team channels (e.g. #support)
      • Conditions & Scheduling
        • Current milestone is one of: Started, Detected, Acknowledged, Investigating, Identified, Mitigated
        • Execute automatically every 30 minutes

SEV-2 (or equivalent) Incident Runbook

Execution rule:  Execute when Current Severity is SEV-2 (or equivalent) 

  • Remind responders to provide medium frequency internal communication updates. For example:
  • Publish incident status updates to the internal status page.
    • Step: FireHydrant Status Page > Publish to status page
      • Conditions and Scheduling: Automatically execute the step when the Current Milestone is [insert key milestones] (e.g. Acknowledged, Investigating, Mitigated, etc.)
  • Attach a general task list for the incident.
  • Notify other relevant Slack channels with periodic incident status updates until the incident is resolved.
    • Prerequisite: The FireHydrant app must be added to the Slack channels for FireHydrant to post your message. Please make sure to /invite @FireHydrant to any channels you list in the configuration.
    • Step: Slack > Notify channel with a custom message
      • Add relevant team channels (e.g. #support)
      • Condition and scheduling:
        • Execute automatically
        • Repeat every 60 minutes
        • When the Current milestone is one of: Started, Detected, Acknowledged, Investigating, Identified, Mitigated

Advanced Runbooks

Service-based Runbook

  • Add a team
    • Step: FireHydrant > Assign A Team
  • Page out to an alerting provider
    • Prerequisite: Alerting provider configured
    • As appropriate for your provider setup:
      • Step: Opsgenie > Create Opsgenie Incident
      • Step: PagerDuty > Create PagerDuty Incident
      • Step: VictorOps > Create VictorOps Incident
  • Notify incident channel of available documentation
  • Tasklist with specific steps to be taken to restore the service based on condition
  • Run a custom script
    • Step: FireHydrant > Script
      • Add your description and custom script to the runbook step configuration.

Team-based Runbook

  • Notify a team’s home channel of engagement

    • Step: Slack > Notify channel
      • Comma Separated Channels: [#team-channel]

  • Add a Role

    • Step: FireHydrant > Assign a Role
      • Add the role and user to the runbook step configuration.

  • Email Manager

    • Firehydrant > Send an email notification
      • Add the manager's email address, subject, and message to the runbook step configuration.

  • Reminder to team’s home channel for long-running incidents

    • Step: Slack > Notify incident channel with a custom message
      • Your Message: [Friendly reminder to post an update on the incident status page(s)]
      • Action button: Post an update
      • Comma Separated Channels: [#team-channel]
    • Conditions and Scheduling:
      • E** xecute automatically**
      • Repeat every 60 minutes
      • When the Time since incident opened: greater than 5 hours

  • Tasklist

Customer Facing Issue Runbook

  • Let the customer support team know an incident has been opened.
  • Send reminders to update customer-facing information while the incident is open.
    • Slack > Notify incident channel with a custom message
      • Your Message: [Friendly reminder to post an update to the external status page.]
      • Action button: Post an update
      • Condition and scheduling:
        • Execute automatically
        • Repeat every 60 minutes
        • When the Current milestone is one of: Started, Detected, Acknowledged, Investigating, Identified, Mitigated

Last updated on 9/29/2023