LabHub

Blog

Grafana OnCall and Incident Management Automation: From PagerDuty Integration to Runbook Automation

2026-03-08 · 30 min read · #observability#grafana-oncall#incident-management#pagerduty#runbook#on-call

한국어English日本語

Grafana OnCall

Introduction

It is 3 a.m. and an alert tone wakes you up. The warning says the production database has exhausted its connection pool. While you dig through Slack channels looking for the responsible engineer, check who is on call, and try to recall the response procedure, the outage stretches on minute by minute. This is the reality in an organization without incident management automation.

The global incident management market grew sharply through 2025. As microservice architectures and cloud native environments spread, a single failure cascading across dozens of services became an everyday occurrence. According to the Google SRE Workbook, the sustainable number of incidents an on-call engineer can absorb per shift is at most two to three. Beyond that, alert fatigue sets in and response quality degrades sharply.

Grafana Labs released Grafana OnCall as open source to address this problem, and in March 2025 it formally launched Grafana Cloud IRM (Incident Response Management), which merges OnCall and Incident. In this article, we build the entire incident management automation pipeline around Grafana OnCall/IRM with working code, from on-call scheduling through escalation policies, PagerDuty integration, Slack integration, and runbook automation.

Why Incident Management Automation Is Needed

Running incident management by hand makes the following problems recur.

Rising mean time to acknowledge (MTTA): The time spent confirming who is on call, assembling the right people, and locating the response procedure ends up longer than the time spent actually solving the problem. Without automation, an MTTA of 15 to 30 minutes is common.

Escalation failure: Manual escalation depends on human judgement. If an engineer woken in the small hours underestimates the severity, or picks the wrong person to escalate to, the outage drags on.

Knowledge disconnect: When incident response procedures are scattered across a wiki or Confluence, finding the right runbook under pressure is hard. Worse still is a runbook that is out of date.

Burnout: Unfair on-call distribution, excessive alerts, and inefficient escalation, repeated over time, lead to engineer burnout. According to an incident.io survey in 2025, 62% of on-call engineers experience alert fatigue.

An automated incident management system solves all of these problems structurally. When an alert fires it routes automatically to the correct on-call responder, escalates according to a defined policy if there is no response, attaches the relevant runbook automatically, and creates a Slack channel to assemble the response team.

Grafana OnCall Architecture

Grafana OnCall is an on-call management system tightly integrated with the Grafana ecosystem. Since March 2025, OnCall and Incident have been merged into Grafana Cloud IRM on Grafana Cloud, and the open source version (OnCall OSS) has entered maintenance mode. The core concepts and architecture are identical in both, so what follows applies to either version.

┌─────────────────────────────────────────────────────────────────────────┐
│               Incident Management Automation Architecture                  │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                         │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐                   │
│  │ Prometheus   │  │ Grafana      │  │ External       │                   │
│  │ Alertmanager │  │ Alerting     │  │ (Datadog etc.) │                  │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘                   │
│         │                 │                  │                           │
│         └─────────────────┼──────────────────┘                          │
│                           ▼                                             │
│              ┌────────────────────────┐                                  │
│              │   Grafana OnCall/IRM   │                                  │
│              │  ┌──────────────────┐  │                                  │
│              │  │  Integration     │  │  Webhook / API intake            │
│              │  │  Layer           │  │                                  │
│              │  └────────┬─────────┘  │                                  │
│              │           ▼            │                                  │
│              │  ┌──────────────────┐  │                                  │
│              │  │  Route Engine    │  │  Label-based routing              │
│              │  └────────┬─────────┘  │                                  │
│              │           ▼            │                                  │
│              │  ┌──────────────────┐  │                                  │
│              │  │  Escalation      │  │  Runs the escalation chain        │
│              │  │  Engine          │  │                                  │
│              │  └────────┬─────────┘  │                                  │
│              │           ▼            │                                  │
│              │  ┌──────────────────┐  │                                  │
│              │  │  Schedule        │  │  On-call schedule lookup          │
│              │  │  Manager         │  │                                  │
│              │  └──────────────────┘  │                                  │
│              └────────────────────────┘                                  │
│                       │          │                                       │
│          ┌────────────┘          └────────────┐                          │
│          ▼                                    ▼                          │
│  ┌──────────────────┐              ┌──────────────────┐                  │
│  │  Notification     │              │  Outgoing         │                 │
│  │  Channels         │              │  Webhooks         │                 │
│  │                   │              │                   │                 │
│  │  - Slack          │              │  - Run runbook    │                 │
│  │  - MS Teams       │              │  - Create Jira    │                 │
│  │  - Phone Call     │              │  - PagerDuty sync │                 │
│  │  - SMS            │              │  - Auto-heal      │                 │
│  │  - Email          │              │                   │                 │
│  └──────────────────┘              └──────────────────┘                  │
│                                                                         │
└─────────────────────────────────────────────────────────────────────────┘

The core components are as follows.

Installation and Initial Setup

Installing OnCall OSS with Docker Compose

The fastest way to deploy Grafana OnCall OSS to a local or development environment is Docker Compose.

# docker-compose.yml
version: '3.8'

services:
  engine:
    image: grafana/oncall:latest
    restart: always
    ports:
      - '8080:8080'
    command: >
      sh -c "uwsgi --ini uwsgi.ini"
    environment:
      BASE_URL: http://localhost:8080
      SECRET_KEY: ${ONCALL_SECRET_KEY:-my-secret-key-change-in-production}
      RABBITMQ_USERNAME: rabbitmq
      RABBITMQ_PASSWORD: rabbitmq
      RABBITMQ_HOST: rabbitmq
      RABBITMQ_PORT: 5672
      RABBITMQ_DEFAULT_VHOST: /
      MYSQL_DB_NAME: oncall
      MYSQL_USER: root
      MYSQL_PASSWORD: oncall
      MYSQL_HOST: mysql
      MYSQL_PORT: 3306
      REDIS_URI: redis://redis:6379/0
      DJANGO_SETTINGS_MODULE: settings.hobby
      CELERY_WORKER_QUEUE: default,critical,long,slack,telegram,webhook,retry,celery,grafana
      GRAFANA_API_URL: http://grafana:3000
    depends_on:
      mysql:
        condition: service_healthy
      rabbitmq:
        condition: service_healthy
      redis:
        condition: service_healthy

  celery:
    image: grafana/oncall:latest
    restart: always
    command: >
      sh -c "./celery_with_exporter.sh"
    environment:
      BASE_URL: http://localhost:8080
      SECRET_KEY: ${ONCALL_SECRET_KEY:-my-secret-key-change-in-production}
      RABBITMQ_USERNAME: rabbitmq
      RABBITMQ_PASSWORD: rabbitmq
      RABBITMQ_HOST: rabbitmq
      RABBITMQ_PORT: 5672
      MYSQL_DB_NAME: oncall
      MYSQL_USER: root
      MYSQL_PASSWORD: oncall
      MYSQL_HOST: mysql
      MYSQL_PORT: 3306
      REDIS_URI: redis://redis:6379/0
      DJANGO_SETTINGS_MODULE: settings.hobby
      CELERY_WORKER_QUEUE: default,critical,long,slack,telegram,webhook,retry,celery,grafana
    depends_on:
      mysql:
        condition: service_healthy
      rabbitmq:
        condition: service_healthy
      redis:
        condition: service_healthy

  mysql:
    image: mysql:8.0
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: oncall
      MYSQL_DATABASE: oncall
    volumes:
      - oncall-mysql:/var/lib/mysql
    healthcheck:
      test: ['CMD', 'mysqladmin', 'ping', '-h', 'localhost']
      interval: 10s
      timeout: 5s
      retries: 5

  redis:
    image: redis:7-alpine
    restart: always
    healthcheck:
      test: ['CMD', 'redis-cli', 'ping']
      interval: 10s
      timeout: 5s
      retries: 5

  rabbitmq:
    image: rabbitmq:3.12-management-alpine
    restart: always
    environment:
      RABBITMQ_DEFAULT_USER: rabbitmq
      RABBITMQ_DEFAULT_PASS: rabbitmq
    healthcheck:
      test: ['CMD', 'rabbitmq-diagnostics', 'check_running']
      interval: 10s
      timeout: 5s
      retries: 5

  grafana:
    image: grafana/grafana:latest
    restart: always
    ports:
      - '3000:3000'
    environment:
      GF_SECURITY_ADMIN_USER: admin
      GF_SECURITY_ADMIN_PASSWORD: admin
      GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS: grafana-oncall-app
      GF_INSTALL_PLUGINS: grafana-oncall-app
    volumes:
      - grafana-data:/var/lib/grafana

volumes:
  oncall-mysql:
  grafana-data:

Run the initialization steps after installing.

# 1. Start Docker Compose
docker-compose up -d

# 2. Run the DB migrations
docker-compose exec engine python manage.py migrate

# 3. Verify the Grafana OnCall plugin is enabled
# Open http://localhost:3000 in a browser
# Grafana left menu -> Alerts & IRM -> OnCall

# 4. Create an OnCall API token (for Terraform/API integration)
curl -X POST http://localhost:3000/api/plugins/grafana-oncall-app/resources/api/v1/api_token \
  -H "Authorization: Bearer <grafana-admin-api-key>" \
  -H "Content-Type: application/json"

# 5. Health check
curl http://localhost:8080/health/

Kubernetes Deployment with a Helm Chart

In production, deploying to Kubernetes with a Helm chart is recommended.

# Add the Helm repository
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

# Install OnCall (default settings)
helm install oncall grafana/oncall \
  --namespace oncall \
  --create-namespace \
  --set base_url=oncall.example.com \
  --set grafana."grafana\.ini".server.domain=grafana.example.com \
  --set ingress.enabled=true \
  --set ingress.annotations."kubernetes\.io/ingress\.class"=nginx \
  --set celery.workers=4 \
  --set engine.replicaCount=2

On-Call Scheduling

The on-call schedule is the foundation of incident management. A well-designed schedule guarantees a fair distribution of the burden and coverage with no gaps. Grafana OnCall can manage schedules three ways: the web UI, iCal, and the API/Terraform.

Schedule Design Principles

The on-call schedule design principles recommended by the official Grafana documentation are as follows.

  1. Pick a rotation period that matches team size: A weekly rotation suits a team of four to six; a two-day rotation suits a team of eight or more.
  2. Follow-the-sun pattern: For a team spread across three or more time zones, design each region to cover on-call during its business hours only. This model can cut on-call hours per engineer by as much as 67%.
  3. Override mechanism: Use a shift swap for planned absences (vacation, meetings) and an override for urgent ones.
  4. Backup schedule: Always configure a backup schedule in addition to the primary schedule.

Managing Schedules with Terraform

Managing on-call schedules as code makes change history, review, and automation possible.

# terraform/oncall-schedules.tf

terraform {
  required_providers {
    grafana = {
      source  = "grafana/grafana"
      version = ">= 3.0.0"
    }
  }
}

provider "grafana" {
  url                  = var.grafana_url
  auth                 = var.grafana_auth
  oncall_access_token  = var.oncall_access_token
}

# Team data sources
data "grafana_oncall_user" "engineer_a" {
  username = "engineer-a"
}

data "grafana_oncall_user" "engineer_b" {
  username = "engineer-b"
}

data "grafana_oncall_user" "engineer_c" {
  username = "engineer-c"
}

data "grafana_oncall_user" "engineer_d" {
  username = "engineer-d"
}

# Primary on-call schedule - weekly rotation
resource "grafana_oncall_schedule" "primary" {
  name      = "Platform Team - Primary"
  type      = "calendar"
  team_id   = var.team_id
  time_zone = "Asia/Seoul"

  shifts = [
    grafana_oncall_on_call_shift.primary_rotation.id,
  ]
}

resource "grafana_oncall_on_call_shift" "primary_rotation" {
  name       = "Primary Weekly Rotation"
  type       = "rolling_users"
  start      = "2026-03-09T00:00:00"
  duration   = 60 * 60 * 24 * 7  # 7 days (in seconds)
  frequency  = "weekly"
  by_day     = ["MO", "TU", "WE", "TH", "FR", "SA", "SU"]
  time_zone  = "Asia/Seoul"

  rolling_users = [
    [data.grafana_oncall_user.engineer_a.id],
    [data.grafana_oncall_user.engineer_b.id],
    [data.grafana_oncall_user.engineer_c.id],
    [data.grafana_oncall_user.engineer_d.id],
  ]
}

# Backup schedule - senior engineers
resource "grafana_oncall_schedule" "backup" {
  name      = "Platform Team - Backup"
  type      = "calendar"
  team_id   = var.team_id
  time_zone = "Asia/Seoul"

  shifts = [
    grafana_oncall_on_call_shift.backup_rotation.id,
  ]
}

resource "grafana_oncall_on_call_shift" "backup_rotation" {
  name       = "Backup Bi-Weekly Rotation"
  type       = "rolling_users"
  start      = "2026-03-09T00:00:00"
  duration   = 60 * 60 * 24 * 14  # 14 days
  frequency  = "weekly"
  interval   = 2
  time_zone  = "Asia/Seoul"

  rolling_users = [
    [data.grafana_oncall_user.engineer_a.id],
    [data.grafana_oncall_user.engineer_c.id],
  ]
}

Designing Escalation Policies

An escalation policy is the core logic that decides who is notified when an alert fires, in what order, and by what method. A Grafana OnCall escalation chain can combine a variety of actions step by step.

The Basic Escalation Pattern

The basic escalation pattern recommended by the official Grafana documentation is as follows.

  1. Send the default notification to the responder on the on-call schedule
  2. Wait 5 minutes (to allow time to respond)
  3. If there is no response, re-notify through the Important channel
  4. Wait 10 minutes
  5. Escalate to the responder on the backup schedule
  6. Wait 15 minutes
  7. Notify the whole team (last resort)

Building the Escalation Chain with Terraform

# terraform/oncall-escalation.tf

# Escalation chains by severity

# Critical (P1) - fast escalation
resource "grafana_oncall_escalation_chain" "critical" {
  name    = "Critical - P1 Incidents"
  team_id = var.team_id
}

resource "grafana_oncall_escalation" "critical_step_1" {
  escalation_chain_id = grafana_oncall_escalation_chain.critical.id
  type                = "notify_on_call_from_schedule"
  notify_on_call_from_schedule = grafana_oncall_schedule.primary.id
  position            = 0
  important           = true
}

resource "grafana_oncall_escalation" "critical_wait_1" {
  escalation_chain_id = grafana_oncall_escalation_chain.critical.id
  type                = "wait"
  duration            = 300  # 5 minutes
  position            = 1
}

resource "grafana_oncall_escalation" "critical_step_2" {
  escalation_chain_id = grafana_oncall_escalation_chain.critical.id
  type                = "notify_on_call_from_schedule"
  notify_on_call_from_schedule = grafana_oncall_schedule.backup.id
  position            = 2
  important           = true
}

resource "grafana_oncall_escalation" "critical_wait_2" {
  escalation_chain_id = grafana_oncall_escalation_chain.critical.id
  type                = "wait"
  duration            = 300  # 5 minutes
  position            = 3
}

resource "grafana_oncall_escalation" "critical_step_3" {
  escalation_chain_id = grafana_oncall_escalation_chain.critical.id
  type                = "notify_whole_channel"
  position            = 4
}

# Warning (P2) - standard escalation
resource "grafana_oncall_escalation_chain" "warning" {
  name    = "Warning - P2 Incidents"
  team_id = var.team_id
}

resource "grafana_oncall_escalation" "warning_step_1" {
  escalation_chain_id = grafana_oncall_escalation_chain.warning.id
  type                = "notify_on_call_from_schedule"
  notify_on_call_from_schedule = grafana_oncall_schedule.primary.id
  position            = 0
  important           = false
}

resource "grafana_oncall_escalation" "warning_wait_1" {
  escalation_chain_id = grafana_oncall_escalation_chain.warning.id
  type                = "wait"
  duration            = 900  # 15 minutes
  position            = 1
}

resource "grafana_oncall_escalation" "warning_step_2" {
  escalation_chain_id = grafana_oncall_escalation_chain.warning.id
  type                = "notify_on_call_from_schedule"
  notify_on_call_from_schedule = grafana_oncall_schedule.backup.id
  position            = 2
  important           = false
}

# Integration setup - Alertmanager
resource "grafana_oncall_integration" "alertmanager" {
  name = "Prometheus Alertmanager"
  type = "alertmanager"

  default_route {
    escalation_chain_id = grafana_oncall_escalation_chain.warning.id
  }
}

# Routing rules - apply a different escalation chain per severity
resource "grafana_oncall_route" "critical_route" {
  integration_id      = grafana_oncall_integration.alertmanager.id
  escalation_chain_id = grafana_oncall_escalation_chain.critical.id
  routing_regex       = "\"severity\":\"critical\""
  position            = 0
}

resource "grafana_oncall_route" "warning_route" {
  integration_id      = grafana_oncall_integration.alertmanager.id
  escalation_chain_id = grafana_oncall_escalation_chain.warning.id
  routing_regex       = "\"severity\":\"warning\""
  position            = 1
}

Slack/Teams Integration

Grafana OnCall's Slack integration goes beyond simply sending notifications: it provides a two-way interface for managing incidents directly inside Slack. Acknowledge, resolve, and escalate can all be performed from buttons on the Slack message.

Slack App Configuration

To set up the Slack integration on OnCall OSS you have to create an app in the Slack API. The environment must be reachable over HTTPS.

# 1. Create the Slack app
# At https://api.slack.com/apps choose "Create New App" -> "From an app manifest"

# 2. App manifest (YAML format)
# Use the manifest below when creating the Slack app

# slack-app-manifest.yml
display_information:
  name: Grafana OnCall
  description: On-call management and incident response
  background_color: '#1a1a2e'

features:
  bot_user:
    display_name: Grafana OnCall
    always_online: true
  shortcuts:
    - name: Create Incident
      type: message
      callback_id: incident_create
      description: Create a new incident from this message

oauth_config:
  scopes:
    bot:
      - app_mentions:read
      - channels:history
      - channels:read
      - chat:write
      - commands
      - files:write
      - groups:history
      - groups:read
      - im:history
      - im:read
      - im:write
      - reactions:write
      - team:read
      - usergroups:read
      - usergroups:write
      - users:read
      - users:read.email

settings:
  event_subscriptions:
    request_url: https://oncall.example.com/slack/event_api_endpoint/
    bot_events:
      - app_mention
      - message.im
  interactivity:
    is_enabled: true
    request_url: https://oncall.example.com/slack/interactive_api_endpoint/
  org_deploy_enabled: false
  socket_mode_enabled: false

Once the Slack integration is complete, the following information is sent to the Slack channel automatically whenever an alert fires.

Microsoft Teams Integration

An organization on MS Teams can implement a similar integration through an outgoing webhook. Grafana OnCall also ships a dedicated MS Teams integration, and Grafana Cloud IRM supports native Teams integration.

PagerDuty Integration

Some organizations already running PagerDuty migrate to Grafana OnCall, and others run the two systems side by side. Grafana OnCall supports two-way integration with PagerDuty and also provides a migration tool.

PagerDuty Integration from Grafana Alerting

Configuring PagerDuty as a contact point in Grafana Alerting lets you send specific alerts directly to PagerDuty.

# Grafana Alerting - PagerDuty contact point configuration
# grafana/provisioning/alerting/contactpoints.yml

apiVersion: 1
contactPoints:
  - orgId: 1
    name: PagerDuty-Critical
    receivers:
      - uid: pagerduty-critical
        type: pagerduty
        settings:
          integrationKey: '${PAGERDUTY_INTEGRATION_KEY}'
          severity: critical
          class: 'production-incident'
          component: '{{ .CommonLabels.service }}'
          group: '{{ .CommonLabels.alertname }}'
        disableResolveMessage: false

  - orgId: 1
    name: PagerDuty-Warning
    receivers:
      - uid: pagerduty-warning
        type: pagerduty
        settings:
          integrationKey: '${PAGERDUTY_WARNING_KEY}'
          severity: warning
          class: 'production-warning'
          component: '{{ .CommonLabels.service }}'
          group: '{{ .CommonLabels.alertname }}'
        disableResolveMessage: false

# Notification policy - routing by severity
policies:
  - orgId: 1
    receiver: PagerDuty-Warning
    group_by: ['alertname', 'service']
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h
    routes:
      - receiver: PagerDuty-Critical
        matchers:
          - severity = critical
        group_wait: 10s
        group_interval: 1m
        repeat_interval: 1h
        continue: false

Migrating from PagerDuty to Grafana OnCall

The Grafana OnCall team provides a tool that migrates PagerDuty configuration. It converts schedules, escalation policies, and service settings automatically.

# Using the PagerDuty migration tool
# 1. Create a PagerDuty API key (read-only permission)
# 2. Run the migration script

# Export the PagerDuty configuration
pip install pdpyras

# Migration Python script
python3 migrate_pagerduty_to_oncall.py \
  --pagerduty-api-key="${PAGERDUTY_API_KEY}" \
  --oncall-api-url="http://localhost:8080" \
  --oncall-api-token="${ONCALL_API_TOKEN}" \
  --dry-run  # simulate first to confirm

Two-Way Webhook Integration

When running PagerDuty and Grafana OnCall side by side, use outgoing webhooks to set up two-way synchronization.

# webhook_sync.py - PagerDuty <-> Grafana OnCall two-way sync
import os
import json
import hmac
import hashlib
from flask import Flask, request, jsonify
import requests

app = Flask(__name__)

ONCALL_API_URL = os.environ["ONCALL_API_URL"]
ONCALL_API_TOKEN = os.environ["ONCALL_API_TOKEN"]
PAGERDUTY_API_KEY = os.environ["PAGERDUTY_API_KEY"]
WEBHOOK_SECRET = os.environ["WEBHOOK_SECRET"]


def verify_signature(payload: bytes, signature: str) -> bool:
    """Verify the webhook signature."""
    expected = hmac.new(
        WEBHOOK_SECRET.encode(), payload, hashlib.sha256
    ).hexdigest()
    return hmac.compare_digest(expected, signature)


@app.route("/webhook/pagerduty-to-oncall", methods=["POST"])
def pagerduty_to_oncall():
    """Forward a PagerDuty event to Grafana OnCall."""
    payload = request.get_json()

    for message in payload.get("messages", []):
        event = message.get("event", "")
        incident = message.get("incident", {})

        if event == "incident.triggered":
            # Create the alert in OnCall
            oncall_payload = {
                "title": incident.get("title", "PagerDuty Incident"),
                "message": incident.get("description", ""),
                "severity": map_severity(incident.get("urgency", "high")),
                "source_link": incident.get("html_url", ""),
            }

            headers = {
                "Authorization": ONCALL_API_TOKEN,
                "Content-Type": "application/json",
            }

            response = requests.post(
                f"{ONCALL_API_URL}/integrations/v1/webhook/<integration-id>/",
                json=oncall_payload,
                headers=headers,
                timeout=10,
            )
            app.logger.info(
                "Forwarded PagerDuty incident to OnCall: %s", response.status_code
            )

        elif event == "incident.resolved":
            # Resolve the matching alert in OnCall
            resolve_oncall_alert(incident.get("id"))

    return jsonify({"status": "ok"}), 200


@app.route("/webhook/oncall-to-pagerduty", methods=["POST"])
def oncall_to_pagerduty():
    """Forward a Grafana OnCall event to PagerDuty."""
    payload = request.get_json()
    event_type = payload.get("event", {}).get("type", "")
    alert_payload = payload.get("alert_payload", {})

    if event_type == "acknowledge":
        # Acknowledge the matching incident in PagerDuty
        pd_event = {
            "routing_key": os.environ["PAGERDUTY_ROUTING_KEY"],
            "event_action": "acknowledge",
            "dedup_key": alert_payload.get("id", ""),
        }
    elif event_type == "resolve":
        pd_event = {
            "routing_key": os.environ["PAGERDUTY_ROUTING_KEY"],
            "event_action": "resolve",
            "dedup_key": alert_payload.get("id", ""),
        }
    else:
        return jsonify({"status": "ignored"}), 200

    response = requests.post(
        "https://events.pagerduty.com/v2/enqueue",
        json=pd_event,
        timeout=10,
    )
    app.logger.info("Forwarded OnCall event to PagerDuty: %s", response.status_code)
    return jsonify({"status": "ok"}), 200


def map_severity(pd_urgency: str) -> str:
    """Map PagerDuty urgency to OnCall severity."""
    mapping = {"high": "critical", "low": "warning"}
    return mapping.get(pd_urgency, "warning")


def resolve_oncall_alert(pd_incident_id: str):
    """Resolve the OnCall alert by PagerDuty incident ID."""
    headers = {
        "Authorization": ONCALL_API_TOKEN,
        "Content-Type": "application/json",
    }
    # Search for and resolve the alert through the OnCall API
    response = requests.get(
        f"{ONCALL_API_URL}/api/v1/alert_groups/",
        headers=headers,
        params={"search": pd_incident_id},
        timeout=10,
    )
    if response.status_code == 200:
        for alert_group in response.json().get("results", []):
            requests.post(
                f"{ONCALL_API_URL}/api/v1/alert_groups/{alert_group['id']}/resolve/",
                headers=headers,
                timeout=10,
            )


if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000)

Runbook Automation

A runbook is a documented incident response procedure. Documentation alone is not enough, however. Following a runbook by hand in an emergency invites mistakes and takes time. Runbook automation turns a repetitive response procedure into a script so it can be run with one click, or automatically.

Automatic Runbook Execution via Outgoing Webhook

With Grafana OnCall's outgoing webhooks, you can run a runbook script automatically when a specific alert fires.

`

# runbook_executor.py - Runbook auto-execution server
import os
import json
import subprocess
import logging
from datetime import datetime
from flask import Flask, request, jsonify
import requests

app = Flask(__name__)
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# Runbook registry - maps each alert type to an automation script
RUNBOOK_REGISTRY = {
    "HighCPUUsage": {
        "script": "/opt/runbooks/high_cpu_usage.sh",
        "auto_execute": True,
        "severity_threshold": "warning",
        "description": "Automatic response when CPU utilization exceeds the threshold",
        "actions": [
            "Collect CPU usage per process",
            "Identify the top 5 processes",
            "Restart abnormal processes automatically (whitelist based)",
        ],
    },
    "DiskSpaceCritical": {
        "script": "/opt/runbooks/disk_cleanup.sh",
        "auto_execute": True,
        "severity_threshold": "critical",
        "description": "Automatic cleanup when disk space runs low",
        "actions": [
            "Clean up temporary files",
            "Compress and archive old logs",
            "Prune unused Docker images",
        ],
    },
    "DatabaseConnectionPoolExhausted": {
        "script": "/opt/runbooks/db_connection_pool.sh",
        "auto_execute": False,  # manual approval required
        "severity_threshold": "critical",
        "description": "Response procedure when the DB connection pool is exhausted",
        "actions": [
            "Force-close idle connections",
            "Grow the connection pool size dynamically",
            "Identify and kill slow queries",
        ],
    },
    "PodCrashLoopBackOff": {
        "script": "/opt/runbooks/pod_crashloop.sh",
        "auto_execute": True,
        "severity_threshold": "warning",
        "description": "Automatic diagnosis of Pod CrashLoopBackOff",
        "actions": [
            "Collect Pod logs",
            "Analyze previous Pod events",
            "Check resource limits",
            "Decide whether to roll back the recent deployment",
        ],
    },
}

SLACK_WEBHOOK_URL = os.environ.get("SLACK_WEBHOOK_URL", "")
ONCALL_API_URL = os.environ.get("ONCALL_API_URL", "")
ONCALL_API_TOKEN = os.environ.get("ONCALL_API_TOKEN", "")


@app.route("/webhook/runbook", methods=["POST"])
def execute_runbook():
    """Called from the OnCall outgoing webhook - runs the runbook automatically."""
    payload = request.get_json()

    alert_name = extract_alert_name(payload)
    severity = extract_severity(payload)
    alert_id = payload.get("alert_group_id", "unknown")

    logger.info("Received alert: %s (severity: %s, id: %s)", alert_name, severity, alert_id)

    runbook = RUNBOOK_REGISTRY.get(alert_name)
    if not runbook:
        logger.warning("No runbook found for alert: %s", alert_name)
        return jsonify({"status": "no_runbook", "alert": alert_name}), 200

    # Check whether automatic execution is allowed
    if not runbook["auto_execute"]:
        notify_manual_runbook(alert_name, runbook, payload)
        return jsonify({"status": "manual_required", "alert": alert_name}), 200

    # Run the runbook script
    result = run_script(
        runbook["script"],
        env_vars={
            "ALERT_NAME": alert_name,
            "ALERT_ID": alert_id,
            "SEVERITY": severity,
            "PAYLOAD": json.dumps(payload),
        },
    )

    # Report the execution result to Slack
    notify_runbook_result(alert_name, runbook, result, alert_id)

    # Auto-resolve the OnCall alert on success
    if result["returncode"] == 0:
        auto_resolve_alert(alert_id)

    return jsonify({
        "status": "executed",
        "alert": alert_name,
        "success": result["returncode"] == 0,
        "output": result["stdout"][:500],
    }), 200


def extract_alert_name(payload: dict) -> str:
    """Extract the alert name from the payload."""
    alert_payload = payload.get("alert_payload", {})
    labels = alert_payload.get("labels", {})
    return labels.get("alertname", payload.get("title", "Unknown"))


def extract_severity(payload: dict) -> str:
    """Extract the severity from the payload."""
    alert_payload = payload.get("alert_payload", {})
    labels = alert_payload.get("labels", {})
    return labels.get("severity", "unknown")


def run_script(script_path: str, env_vars: dict, timeout: int = 300) -> dict:
    """Run the runbook script and return the result."""
    env = os.environ.copy()
    env.update(env_vars)

    try:
        result = subprocess.run(
            ["/bin/bash", script_path],
            capture_output=True,
            text=True,
            timeout=timeout,
            env=env,
        )
        return {
            "returncode": result.returncode,
            "stdout": result.stdout,
            "stderr": result.stderr,
        }
    except subprocess.TimeoutExpired:
        return {
            "returncode": -1,
            "stdout": "",
            "stderr": f"Script timed out after {timeout}s",
        }
    except Exception as e:
        return {
            "returncode": -1,
            "stdout": "",
            "stderr": str(e),
        }


def notify_runbook_result(alert_name: str, runbook: dict, result: dict, alert_id: str):
    """Report the runbook execution result to Slack."""
    if not SLACK_WEBHOOK_URL:
        return

    status_emoji = "white_check_mark" if result["returncode"] == 0 else "x"
    status_text = "SUCCESS" if result["returncode"] == 0 else "FAILED"

    slack_message = {
        "text": f"Runbook Execution: {status_text}",
        "blocks": [
            {
                "type": "header",
                "text": {
                    "type": "plain_text",
                    "text": f"Runbook: {alert_name} - {status_text}",
                },
            },
            {
                "type": "section",
                "fields": [
                    {"type": "mrkdwn", "text": f"*Alert ID:*\n{alert_id}"},
                    {"type": "mrkdwn", "text": f"*Description:*\n{runbook['description']}"},
                    {"type": "mrkdwn", "text": f"*Timestamp:*\n{datetime.utcnow().isoformat()}"},
                ],
            },
        ],
    }

    if result["stdout"]:
        slack_message["blocks"].append({
            "type": "section",
            "text": {
                "type": "mrkdwn",
                "text": f"*Output:*\n```{result['stdout'][:1000]}```",
            },
        })

    requests.post(SLACK_WEBHOOK_URL, json=slack_message, timeout=10)


def notify_manual_runbook(alert_name: str, runbook: dict, payload: dict):
    """Send Slack guidance for a runbook that must be run manually."""
    if not SLACK_WEBHOOK_URL:
        return

    actions_text = "\n".join(f"  {i+1}. {a}" for i, a in enumerate(runbook["actions"]))
    slack_message = {
        "text": f"Manual Runbook Required: {alert_name}",
        "blocks": [
            {
                "type": "header",
                "text": {
                    "type": "plain_text",
                    "text": f"Manual Runbook: {alert_name}",
                },
            },
            {
                "type": "section",
                "text": {
                    "type": "mrkdwn",
                    "text": (
                        f"*Description:* {runbook['description']}\n\n"
                        f"*Steps:*\n{actions_text}"
                    ),
                },
            },
        ],
    }
    requests.post(SLACK_WEBHOOK_URL, json=slack_message, timeout=10)


def auto_resolve_alert(alert_id: str):
    """Auto-resolve the OnCall alert when the runbook succeeds."""
    if not ONCALL_API_URL or not ONCALL_API_TOKEN:
        return

    headers = {
        "Authorization": ONCALL_API_TOKEN,
        "Content-Type": "application/json",
    }
    try:
        requests.post(
            f"{ONCALL_API_URL}/api/v1/alert_groups/{alert_id}/resolve/",
            headers=headers,
            timeout=10,
        )
        logger.info("Auto-resolved alert: %s", alert_id)
    except Exception as e:
        logger.error("Failed to auto-resolve alert %s: %s", alert_id, e)


if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8000)

Runbook Script Example: Disk Cleanup

#!/bin/bash
# /opt/runbooks/disk_cleanup.sh
# Runbook for automatic cleanup when disk space runs low

set -euo pipefail

LOG_FILE="/var/log/runbook/disk_cleanup_$(date +%Y%m%d_%H%M%S).log"
mkdir -p /var/log/runbook

exec > >(tee -a "$LOG_FILE") 2>&1

echo "=== Disk Cleanup Runbook Started ==="
echo "Timestamp: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
echo "Alert: ${ALERT_NAME:-unknown}"
echo "Severity: ${SEVERITY:-unknown}"
echo ""

# 1. Check current disk usage
echo "--- Step 1: Current Disk Usage ---"
df -h / /var /tmp 2>/dev/null || df -h /
echo ""

# 2. Identify large files
echo "--- Step 2: Top 10 Largest Files in /var ---"
find /var -type f -size +100M -exec ls -lh {} \; 2>/dev/null | sort -k5 -hr | head -10
echo ""

# 3. Clean up temporary files
echo "--- Step 3: Cleaning Temporary Files ---"
TEMP_CLEANED=$(find /tmp -type f -atime +7 -delete -print 2>/dev/null | wc -l)
echo "Removed ${TEMP_CLEANED} temporary files older than 7 days"
echo ""

# 4. Compress old log files
echo "--- Step 4: Compressing Old Log Files ---"
LOG_COMPRESSED=0
for logfile in $(find /var/log -name "*.log" -size +50M -mtime +3 2>/dev/null); do
    gzip "$logfile" && LOG_COMPRESSED=$((LOG_COMPRESSED + 1))
done
echo "Compressed ${LOG_COMPRESSED} log files"
echo ""

# 5. Docker cleanup (when Docker is installed)
if command -v docker &> /dev/null; then
    echo "--- Step 5: Docker Cleanup ---"
    echo "Removing dangling images..."
    docker image prune -f 2>/dev/null || true
    echo "Removing unused volumes..."
    docker volume prune -f 2>/dev/null || true
    echo "Removing stopped containers older than 24h..."
    docker container prune -f --filter "until=24h" 2>/dev/null || true
    echo ""
fi

# 6. systemd journal cleanup
if command -v journalctl &> /dev/null; then
    echo "--- Step 6: Journal Cleanup ---"
    journalctl --vacuum-time=7d 2>/dev/null || true
    echo ""
fi

# 7. Check disk usage after cleanup
echo "--- Step 7: Disk Usage After Cleanup ---"
df -h / /var /tmp 2>/dev/null || df -h /

# 8. Evaluate the result
USAGE_PERCENT=$(df / | tail -1 | awk '{print $5}' | tr -d '%')
if [ "$USAGE_PERCENT" -lt 85 ]; then
    echo ""
    echo "=== Disk Cleanup SUCCESS: Usage is now ${USAGE_PERCENT}% ==="
    exit 0
else
    echo ""
    echo "=== Disk Cleanup PARTIAL: Usage is still ${USAGE_PERCENT}% - Manual intervention needed ==="
    exit 1
fi

Reducing Alert Fatigue

Alert fatigue is what happens when an on-call engineer is exposed to so many alerts that they fall into cognitive overload. Too many alerts means the important ones get missed, response times slow down, and ultimately engineers burn out. The Google SRE Workbook offers at most two to three actionable incidents per shift as the sustainable baseline. If a shift carries eight to ten or more, that is not an on-call problem but an alert design problem.

Strategies for Reducing Alert Fatigue

1. Alert audit

Every month, analyze all alerts from the past 30 days. Any alert an engineer has ignored twice or more without taking action should be reconfigured or removed. An alert that needs no action is not an alert but noise.

2. Severity-tiered notification

Not every alert should go out through the same channel by the same method. Vary the notification method according to severity.

SeverityNotification methodTime restrictionEscalation wait
P0 (Critical)Phone + SMS + Slack24 hours3 min
P1 (High)SMS + Slack24 hours5 min
P2 (Medium)Slack + emailBusiness hours only30 min
P3 (Low)Email + Jira ticketBusiness hours onlyNext business day

3. Alert grouping and deduplication

Bundle multiple alerts arising from the same root cause into one. Make use of Alertmanager's group_by setting, and filter duplicate alerts in Grafana OnCall's routing rules.

4. Auto-resolve

Configure auto-resolve for transient, spiky alerts. For example, if CPU utilization crosses 90% and then falls back below 80% within 5 minutes, resolve the alert automatically.

5. Maintenance windows

For planned deployments, patches, and infrastructure work, set a maintenance window to mute the related alerts temporarily.

6. Periodic review and a feedback loop

Run an alert effectiveness retrospective every quarter. Track metrics such as MTTA, MTTR, alert dismiss rate, and the share of duplicate alerts, and keep improving the alerting policy by folding in feedback from the team.

Alert Quality Metrics Dashboard

The core metrics for measuring and tracking alert fatigue quantitatively are as follows.

Incident Management Tool Comparison

This section compares four incident management tools in wide use today. As of 2025 the market is shifting: Atlassian stopped new sales of OpsGenie (June 2025), and Grafana OnCall OSS entered maintenance mode.

Feature / traitGrafana OnCall/IRMPagerDutyOpsGenie (Atlassian)Splunk On-Call (VictorOps)
Price (50 users)~$11,500/yr (Cloud IRM)~$25,200/yr (Business)~$11,970/yr (Standard)~$24,900/yr (Growth)
Open sourceOSS version (maintenance mode)NoneNoneNone
Grafana integrationNativePluginPluginPlugin
Slack integrationTwo-way (button actions)Two-wayTwo-wayTwo-way
On-call schedulingWeb, iCal, TerraformWeb, APIWeb, APIWeb, API
Escalation policyMulti-step chainMulti-step + round robinMulti-stepMulti-step
Terraform supportOfficial providerCommunity providerLimitedLimited
AI/ML featuresSift (IRM)AIOps (event intelligence)LimitedLimited
Runbook integrationOutgoing WebhookRunbook Automation (PD)LimitedLimited
Mobile appGrafana Cloud appDedicated app (rich)Dedicated appDedicated app
SSO/SAMLGrafana Cloud integrationSupported (Enterprise)Atlassian SSOSupported
SLA99.9% (Cloud)99.9%99.9%99.9%
Learning curveMedium (Grafana experience helps)High (feature rich)LowMedium
Current status (2026)Cloud IRM merge completeMarket leaderNew sales ended (2025.06)Integrating post-Cisco acquisition

Tool Selection Guide

Troubleshooting

Problem 1: Slack Notifications Are Not Delivered

The most common causes in the Slack integration are an expired bot token, insufficient channel permissions, and a mismatched event subscription URL.

# Slack integration diagnostic checklist
# 1. Check the OnCall engine logs
docker-compose logs engine | grep -i slack

# 2. Verify the Slack app event subscription URL
# https://api.slack.com/apps -> select the app -> Event Subscriptions
# Confirm the Request URL is https://oncall.example.com/slack/event_api_endpoint/

# 3. Verify the bot token is valid
curl -X POST https://slack.com/api/auth.test \
  -H "Authorization: Bearer xoxb-your-bot-token" \
  -H "Content-Type: application/json"

# 4. Check channel access permissions
curl -X POST https://slack.com/api/conversations.info \
  -H "Authorization: Bearer xoxb-your-bot-token" \
  -H "Content-Type: application/json" \
  -d '{"channel": "C0XXXXXXX"}'

# 5. Check the user's Slack account link
# Grafana OnCall -> Users -> the user -> confirm the Slack account is linked

Problem 2: Escalation Does Not Fire

The most common cause of escalation failure is that no one is on call on the schedule.

Problem 3: Webhook Failures

When an outgoing webhook fails, runbook automation does not run.

Production Checklist

These are the items to confirm before deploying Grafana OnCall/IRM to production.

Infrastructure

On-Call Schedule

Escalation

Notification Channels

Automation

Monitoring (Meta-Monitoring)

Failure Cases and Recovery

Case 1: Missed Alerts Caused by a Schedule Gap

Situation: A production database failure occurred at 9 p.m. on a Friday. Engineer A's on-call shift had ended at 6 p.m. Friday, and engineer B's shift was configured to start at 9 a.m. Saturday. During the 15-hour schedule gap, the alert found no escalation target and was dropped.

Root cause: The schedule was created considering business hours only, and 24/7 coverage was never verified. No schedule gap detection alert was configured either.

Recovery and prevention:

Case 2: Celery Queue Saturation Caused by an Alert Storm

Situation: A network partition caused hundreds of services to fire alerts simultaneously. The volume exceeded what the Celery workers could process, the queue saturated, and even the genuinely important alerts that followed were delayed.

Root cause: Alertmanager's group_by and group_wait settings were not aggressive enough, and OnCall's routing rules had no duplicate alert filtering. The number of Celery workers was also insufficient.

Recovery and prevention:

Case 3: A False-Positive Runbook Run Caused by Missing Webhook Authentication

Situation: The runbook auto-execution endpoint had no authentication, so a forged webhook request from outside triggered the disk cleanup runbook. Fortunately the cleanup was limited to temporary files and old logs, so there was no data loss, but the security risk was real.

Root cause: HMAC signature verification was never implemented on the outgoing webhook endpoint. The endpoint was exposed to the public internet.

Recovery and prevention:

References

Comments

No comments yet.

Sign in to leave a comment