Prometheus and Alertmanager Alerting Pipeline: From Rule Writing to PagerDuty/Slack Routing

• Introduction
• Writing Prometheus Alerting Rules
  • The structure of alerting rules
  • pending vs firing state transitions
  • Using keep_firing_for
  • Basic infrastructure alert rules
  • SLO-based burn rate alert rules
• Testing Alert Rules
  • Unit tests with promtool
  • Automating alert rule validation in CI/CD
• Advanced Alertmanager Configuration
  • Designing the routing tree
  • Suppressing Downstream Alerts with Inhibition Rules
  • Stopping Alerts During Maintenance with Silences
• PagerDuty and Slack Integration
  • Configuring multiple receivers
• Comparison with Grafana Alerting
  • A hybrid strategy
• Strategies for Preventing Alert Fatigue
  • Causes and symptoms of alert fatigue
  • Improving the signal-to-noise ratio (SNR)
  • Alert priority scheme: P1-P4
  • The weekly alert review process
  • Alerts observability at Cloudflare
• Failure Cases and Recovery Procedures
  • Case 1: alert flood from a misconfigured group_by
  • Case 2: missed incident from a missing PagerDuty route
  • Recovery procedure checklist
• Operational Caveats
  • Alert rule naming conventions
  • Label standardization and governance
  • Regular alert audits
• References

Introduction

Building a monitoring system and putting up dashboards does not by itself complete observability. A dashboard has to be checked by a person, actively, whereas an alert notifies the right person at the right time when the system detects an abnormal state. Real observability starts with metric collection and is completed by the alerting pipeline.

Rob Ewaschuk, formerly of Google SRE, lays out two core principles in "My Philosophy on Alerting". First, alert on the symptom, not the cause. A symptom-based alert such as "the error rate of user requests exceeds 1%" reflects real user impact far more accurately than a cause-based alert such as disk usage at 80%. Second, every alert must trigger immediate action. If the response to an alert is "I can look at it later", that alert has no reason to exist.

This article covers the entire process of building a production alerting pipeline: writing Prometheus alerting rules, designing the Alertmanager routing tree, integrating PagerDuty and Slack, and strategies for preventing alert fatigue.

Writing Prometheus Alerting Rules

The structure of alerting rules

A Prometheus alerting rule is made up of five core fields.

• alert: the unique name of the alert
• expr: the PromQL expression. If the result is not an empty vector, the alert fires
• for: how long the rule stays in the pending state. This filters out transient spikes
• labels: labels attached to the alert. severity, team and the like are used for routing
• annotations: metadata for humans, such as the alert description and a runbook URL

pending vs firing state transitions

An alert cycles through three states. inactive is the default state, in which the condition is not met. When the expr condition is first met the alert enters the pending state, and if the condition keeps being met for the duration given in for, it moves to firing and is sent to Alertmanager. If the condition clears during the for period, it returns to inactive.

Using keep_firing_for

The keep_firing_for field, introduced in Prometheus 2.42, is useful for intermittent metrics. For example, when the metric of a batch job disappears after the job completes the alert resolves immediately; setting keep_firing_for keeps the alert firing for the specified time even after the metric is gone, which buys the on-call engineer time to look at it.

Basic infrastructure alert rules

These are basic alert rules for CPU, memory, and disk. Symptom-based alerts come first, but infrastructure resource exhaustion is still a core cause-level signal that has to be monitored.

groups:
  - name: infrastructure-alerts
    interval: 30s
    rules:
      # CPU usage stays at or above 80% for 5 minutes
      - alert: HighCpuUsage
        expr: |
          100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
          team: infra
        annotations:
          summary: 'High CPU usage'
          description: 'CPU usage on instance {{ $labels.instance }} is {{ $value | printf "%.1f" }}%.'
          runbook_url: 'https://wiki.internal/runbook/high-cpu'

      # Memory usage above 90%
      - alert: HighMemoryUsage
        expr: |
          (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 90
        for: 5m
        labels:
          severity: warning
          team: infra
        annotations:
          summary: 'High memory usage'
          description: 'Memory usage on instance {{ $labels.instance }} is {{ $value | printf "%.1f" }}%.'

      # Disk predicted to fill within 24 hours
      - alert: DiskWillFillIn24Hours
        expr: |
          predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[6h], 24*3600) < 0
        for: 10m
        labels:
          severity: critical
          team: infra
        annotations:
          summary: 'Disk predicted to fill within 24 hours'
          description: 'Mountpoint {{ $labels.mountpoint }} on instance {{ $labels.instance }} is predicted to fill within 24 hours.'
          runbook_url: 'https://wiki.internal/runbook/disk-full'

SLO-based burn rate alert rules

This is the Multi-Window Multi-Burn-Rate alerting approach proposed in the Google SRE Workbook. It detects fast burn and slow burn of the error budget separately, based on the burn rate, and differentiates the urgency of the response accordingly.

groups:
  - name: slo-burn-rate-alerts
    rules:
      # Error rate recording rules (precomputed)
      - record: slo:http_error_rate:ratio_rate5m
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
          /
          sum(rate(http_requests_total[5m])) by (service)

      - record: slo:http_error_rate:ratio_rate30m
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[30m])) by (service)
          /
          sum(rate(http_requests_total[30m])) by (service)

      - record: slo:http_error_rate:ratio_rate1h
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[1h])) by (service)
          /
          sum(rate(http_requests_total[1h])) by (service)

      - record: slo:http_error_rate:ratio_rate6h
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[6h])) by (service)
          /
          sum(rate(http_requests_total[6h])) by (service)

      # SLO 99.9% (error budget 0.1%)
      # Fast burn: 5m/30m windows, burn rate 14.4x → budget exhausted within 1 hour
      - alert: SloHighBurnRate
        expr: |
          slo:http_error_rate:ratio_rate5m{service="api-gateway"} > (14.4 * 0.001)
          and
          slo:http_error_rate:ratio_rate30m{service="api-gateway"} > (14.4 * 0.001)
        for: 2m
        labels:
          severity: critical
          slo: 'availability'
          window: 'fast'
        annotations:
          summary: 'SLO fast burn detected - immediate response required'
          description: 'The api-gateway error budget is shrinking at a rate that exhausts it within 1 hour.'

      # Slow burn: 1h/6h windows, burn rate 6x → budget exhausted within 4 hours
      - alert: SloMediumBurnRate
        expr: |
          slo:http_error_rate:ratio_rate1h{service="api-gateway"} > (6 * 0.001)
          and
          slo:http_error_rate:ratio_rate6h{service="api-gateway"} > (6 * 0.001)
        for: 5m
        labels:
          severity: warning
          slo: 'availability'
          window: 'slow'
        annotations:
          summary: 'SLO slow burn detected - investigation required'
          description: 'The api-gateway error budget is shrinking at a rate that exhausts it within 4 hours.'

The key in the rules above is combining two windows with an AND condition. Looking only at the short window (5m) reacts to transient spikes, and looking only at the long window (30m) makes detection slow. Firing only when both windows meet the condition at once raises accuracy.

Testing Alert Rules

Unit tests with promtool

Alert rules must be tested before they are deployed to production. promtool feeds in synthetic time series data and verifies that an alert fires as expected at a given point in time.

# alert_test.yaml
rule_files:
  - infrastructure_alerts.yaml

evaluation_interval: 1m

tests:
  # Test 1: firing when CPU stays at or above 80% for 5 minutes
  - interval: 1m
    input_series:
      - series: 'node_cpu_seconds_total{mode="idle",instance="node1:9100",cpu="0"}'
        values: '0+0.15x20' # 0.15s idle per minute = 85% usage
      - series: 'node_cpu_seconds_total{mode="idle",instance="node1:9100",cpu="1"}'
        values: '0+0.15x20'
    alert_rule_test:
      # At 4 minutes: should be pending (for: 5m not yet satisfied)
      - eval_time: 4m
        alertname: HighCpuUsage
        exp_alerts: []
      # At 6 minutes: should be firing
      - eval_time: 6m
        alertname: HighCpuUsage
        exp_alerts:
          - exp_labels:
              severity: warning
              team: infra
              instance: 'node1:9100'
            exp_annotations:
              summary: 'High CPU usage'

  # Test 2: no alert when memory is below 90%
  - interval: 1m
    input_series:
      - series: 'node_memory_MemAvailable_bytes{instance="node1:9100"}'
        values: '2147483648x10' # 2GB available
      - series: 'node_memory_MemTotal_bytes{instance="node1:9100"}'
        values: '8589934592x10' # 8GB total = 75% used
    alert_rule_test:
      - eval_time: 10m
        alertname: HighMemoryUsage
        exp_alerts: []

Automating alert rule validation in CI/CD

Alert rule files are managed in Git and validated automatically in the CI pipeline. Syntax errors, PromQL expression errors, and test failures can all be caught before deployment.

# Syntax validation
promtool check rules infrastructure_alerts.yaml
promtool check rules slo_alerts.yaml

# Run unit tests
promtool test rules alert_test.yaml

# Validate the Alertmanager configuration
amtool check-config alertmanager.yaml

# CI pipeline example (GitHub Actions)
# jobs:
#   validate-alerts:
#     steps:
#       - name: Check alert rules syntax
#         run: promtool check rules rules/*.yaml
#       - name: Run alert rule tests
#         run: promtool test rules tests/*.yaml
#       - name: Check Alertmanager config
#         run: amtool check-config alertmanager.yaml

Advanced Alertmanager Configuration

Designing the routing tree

Alertmanager routing works as a tree. The top-level route defines the default receiver for every alert, and child routes branch alerts to the appropriate channel through label matching. Setting continue: true keeps evaluating the next routing rules at the same level even after a match, so a single alert can be delivered to several receivers.

group_by bundles alerts that share the same label values into one group. For example, with group_by: [alertname, cluster], alerts with the same alert name coming from the same cluster are bundled into a single alert message. This way the OOMKilled alerts that fire simultaneously on 100 Pods arrive as one group message rather than 100 individual messages.

# alertmanager.yaml
global:
  resolve_timeout: 5m
  smtp_smarthost: 'smtp.internal:587'
  smtp_from: 'alertmanager@company.com'

route:
  receiver: default-slack
  group_by: [alertname, cluster, namespace]
  group_wait: 30s # wait after the first alert so it can be grouped
  group_interval: 5m # resend interval when a new alert joins an existing group
  repeat_interval: 4h # interval for resending the same alert

  routes:
    # Critical alerts → PagerDuty (immediate page)
    - match:
        severity: critical
      receiver: pagerduty-critical
      group_wait: 10s
      repeat_interval: 1h
      continue: true # also send to Slack at the same time

    # Critical alerts → Slack critical channel (sent in parallel)
    - match:
        severity: critical
      receiver: slack-critical

    # SLO-related alerts → SRE team channel
    - match_re:
        alertname: '^Slo.*'
      receiver: slack-sre-slo
      group_by: [alertname, service]

    # Infrastructure team alerts
    - match:
        team: infra
      receiver: slack-infra
      group_by: [alertname, instance]

    # Backend team alerts
    - match:
        team: backend
      receiver: slack-backend
      group_by: [alertname, service, namespace]

    # Watchdog alert (confirms Alertmanager is working)
    - match:
        alertname: Watchdog
      receiver: 'null'
      repeat_interval: 24h

Suppressing Downstream Alerts with Inhibition Rules

Inhibition rules suppress the related downstream alerts while a particular alert is firing. For example, when a node itself goes down, every Pod alert on that node is unnecessary. When an infrastructure-level failure occurs, suppressing the dozens of downstream alerts it triggers in a chain prevents an alert flood.

# alertmanager.yaml (inhibition section)
inhibit_rules:
  # When a node is down, suppress all downstream alerts for that node
  - source_matchers:
      - alertname = NodeDown
    target_matchers:
      - severity =~ "warning|info"
    equal: [instance]

  # On a cluster-level failure, suppress individual service alerts
  - source_matchers:
      - alertname = ClusterUnreachable
    target_matchers:
      - severity =~ "warning|critical"
    equal: [cluster]

  # While a Critical is firing, suppress the Warning of the same alert
  - source_matchers:
      - severity = critical
    target_matchers:
      - severity = warning
    equal: [alertname, cluster, namespace]

  # On a large-scale failure, suppress SLO alerts (it is an infrastructure-level problem)
  - source_matchers:
      - alertname =~ "NodeDown|ClusterUnreachable"
    target_matchers:
      - alertname =~ "^Slo.*"
    equal: [cluster]

Stopping Alerts During Maintenance with Silences

During planned maintenance the related alerts can be stopped temporarily. A Silence is created from the amtool CLI or the Alertmanager UI. A Silence scopes itself with label matchers and is released automatically once its expiry time passes.

# Create a Silence with amtool (stop alerts for one cluster for 2 hours)
amtool silence add \
  --alertmanager.url=http://alertmanager:9093 \
  --author="sre-team" \
  --comment="Planned maintenance: k8s cluster upgrade" \
  --duration=2h \
  cluster="production-us-east-1"

# List active Silences
amtool silence query --alertmanager.url=http://alertmanager:9093

# Expire a Silence
amtool silence expire --alertmanager.url=http://alertmanager:9093 <silence-id>

PagerDuty and Slack Integration

Configuring multiple receivers

The key is routing alerts to different channels according to severity. Critical alerts page the on-call engineer immediately through PagerDuty, while Warning alerts go to a Slack channel to be investigated during working hours.

# alertmanager.yaml (receivers section)
receivers:
  # Default receiver: Slack general channel
  - name: default-slack
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/T00/B00/XXXXX'
        channel: '#alerts-general'
        send_resolved: true
        title: '{{ if eq .Status "firing" }}FIRING{{ else }}RESOLVED{{ end }} - {{ .CommonLabels.alertname }}'
        text: >-
          *Severity:* {{ .CommonLabels.severity | toUpper }}
          *Cluster:* {{ .CommonLabels.cluster }}
          *Namespace:* {{ .CommonLabels.namespace }}
          {{ range .Alerts }}
          - *{{ .Labels.instance }}*: {{ .Annotations.description }}
          {{ end }}

  # PagerDuty Critical receiver
  - name: pagerduty-critical
    pagerduty_configs:
      - service_key_file: '/etc/alertmanager/secrets/pagerduty-service-key'
        severity: '{{ .CommonLabels.severity }}'
        description: '{{ .CommonAnnotations.summary }}'
        details:
          firing: '{{ .Alerts.Firing | len }}'
          resolved: '{{ .Alerts.Resolved | len }}'
          cluster: '{{ .CommonLabels.cluster }}'
          namespace: '{{ .CommonLabels.namespace }}'
          runbook_url: '{{ .CommonAnnotations.runbook_url }}'

  # Slack Critical channel (sent alongside PagerDuty)
  - name: slack-critical
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/T00/B00/YYYYY'
        channel: '#alerts-critical'
        send_resolved: true
        color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}'
        title: 'CRITICAL: {{ .CommonLabels.alertname }}'
        text: >-
          *Status:* {{ .Status | toUpper }}
          {{ range .Alerts }}
          - {{ .Annotations.description }}
            Runbook: {{ .Annotations.runbook_url }}
          {{ end }}

  # SRE SLO dedicated channel
  - name: slack-sre-slo
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/T00/B00/ZZZZZ'
        channel: '#sre-slo-alerts'
        send_resolved: true
        title: 'SLO Alert: {{ .CommonLabels.alertname }}'
        text: >-
          *Service:* {{ .CommonLabels.service }}
          *Window:* {{ .CommonLabels.window }}
          {{ range .Alerts }}
          - {{ .Annotations.description }}
          {{ end }}

  # Infrastructure team channel
  - name: slack-infra
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/T00/B00/INFRA'
        channel: '#team-infra-alerts'
        send_resolved: true

  # Backend team channel
  - name: slack-backend
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/T00/B00/BACKEND'
        channel: '#team-backend-alerts'
        send_resolved: true

  # Null receiver (absorbs unnecessary alerts such as Watchdog)
  - name: 'null'

When using the PagerDuty Events API v2, routing_key and service_key_file have to be chosen appropriately. service_key_file injects the secret key from a file, which makes it easy to wire up with a Kubernetes Secret. For Slack Incoming Webhooks, create a separate webhook URL per channel so that each receiver delivers to a different channel.

Comparison with Grafana Alerting

Since Grafana 9 introduced Unified Alerting, alert rules can also be managed inside Grafana itself. Compare the characteristics of the two systems and choose what fits the environment.

A hybrid strategy

In practice a hybrid approach works better than picking only one of the two.

• Prometheus Alertmanager: infrastructure metrics, SLO-based alerts, and core alerts that demand high reliability
• Grafana Unified Alerting: log-based alerts (Loki), trace-based alerts (Tempo), business metric alerts (SQL data sources)
• A shared Alertmanager: Grafana alerts can also be routed to an external Alertmanager for unified management

Connecting an external Alertmanager from Grafana lets you use the broad data source support of Grafana and the powerful routing and grouping of Alertmanager at the same time.

Strategies for Preventing Alert Fatigue

Causes and symptoms of alert fatigue

Alert fatigue is the phenomenon where excessive alerts scatter the attention of the on-call engineer until even important alerts end up ignored. According to the PagerDuty 2024 State of Digital Operations report, response quality drops sharply once an on-call engineer receives more than 10 alerts a day on average.

The main causes of alert fatigue are as follows.

• Too many cause-based alerts: alerting separately at CPU 60%, 70%, and 80%
• Unadjusted thresholds: leaving the defaults in place, so normal variation triggers alerts
• No group_by: dozens of individual alerts from a single incident
• Transient conditions that self-heal: alerting every time on an OOM that a restart resolves
• Alerts that ignore business hours: being paged at 3 in the morning by a Warning alert

Improving the signal-to-noise ratio (SNR)

Every alert has to pass the following questions.

1. If this alert arrives, does it require immediate action?
2. Does this alert represent a symptom that affects users?
3. What risk would there be if this alert did not exist?
4. Is there a clear runbook for this alert?

If the answer to even one of them is "no", that alert should be removed or turned into a dashboard panel.

Alert priority scheme: P1-P4

The weekly alert review process

Every week the team reviews the alerts of the past week together.

1. Quantitative analysis: total alert count, distribution by severity, distribution by team, MTTR measurement
2. False positive classification: check whether each alert actually triggered an action
3. Threshold adjustment: adjust the threshold or the for period of alerts that keep producing false positives
4. Rule cleanup: an alert ignored 3 weeks in a row is a removal candidate
5. Documentation: record the adjustments and the reasoning behind them

Alerts observability at Cloudflare

Cloudflare introduced the concept of "Alerts Observability", making the alerts themselves an object of observation. It visualizes alert firing frequency, response time, auto-resolution rate, and escalation rate on a dashboard, and uses that to continuously improve alert quality. Meta-monitoring of the alerting system is the key to managing alert fatigue systematically.

Failure Cases and Recovery Procedures

Case 1: alert flood from a misconfigured group_by

Situation: with group_by set to only [alertname], the DiskWillFillIn24Hours alerts of 100 nodes fired at the same time. Every node alert was bundled into one group and only a single alert was sent, so the individual node information was lost and no response was possible.

Cause: group_by did not include the instance label, so the alerts of every node were merged into one.

Fix: changed it to group_by: [alertname, instance] so that a separate alert group is created per node. Note that including too many labels in group_by splits groups too finely and causes the opposite problem (an alert flood), so the right balance is needed.

Case 2: missed incident from a missing PagerDuty route

Situation: a new microservice was deployed, but its alerts were missing the team: payments label. They matched no routing rule and went only to the default receiver (the Slack general channel), so a payment service outage went undetected for 30 minutes.

Cause: there was no label validation process for alert rules at service deployment time.

Fix: added a label validation step to the CI pipeline so that every alert rule is required to carry the team and severity labels. Routing simulation with amtool was added to the pre-deployment checks.

# Simulate the routing path with amtool
amtool config routes test \
  --config.file=alertmanager.yaml \
  --tree \
  severity=critical team=payments alertname=HighErrorRate

# Expected output: pagerduty-critical → slack-critical
# Without the team=payments label: default-slack (warning raised)

Recovery procedure checklist

These are the step-by-step checks for a failure in the alerting pipeline.

1. Check Alertmanager cluster state: amtool cluster show
2. Check the currently active alerts: the Alertmanager UI or amtool alert query
3. Check Silence state: make sure no unintended Silence is active
4. Check Prometheus alert rule evaluation state: look for errors in the Alerts tab of the Prometheus UI
5. Network connectivity: verify communication from Prometheus to Alertmanager
6. Receiver connectivity: verify PagerDuty API and Slack webhook responses
7. Configuration file integrity: amtool check-config alertmanager.yaml

Operational Caveats

Alert rule naming conventions

Consistent naming is essential for writing routing rules and filtering dashboards. The following convention is recommended.

Use PascalCase, and make sure the alert name alone tells you what is wrong. Meaningless names such as "Alert1" or "Check5" are forbidden.

Label standardization and governance

Define the mandatory labels that every alert rule has to carry.

Label values must be managed as enumerations; allowing free text makes routing rules complex and eventually unmanageable. Validating the label schema with OPA (Open Policy Agent) or in the CI pipeline works well.

Regular alert audits

Run an audit over all alert rules every quarter.

• Active alert count trend: if the alert count keeps rising, rule cleanup is needed
• False positive rate: the share of alerts that auto-resolve within 5 minutes of firing
• Response rate: the share of fired alerts that led to an actual action
• Ownership check: whether every alert rule has an owning team assigned
• Runbook links: whether every Critical/Warning alert has a runbook_url

Based on the audit results, clean up rules, remove unnecessary alerts, and adjust thresholds. Alert rules are a living asset that needs continuous maintenance, just like code.

References

• Alerting Rules | Prometheus
• Alertmanager Configuration | Prometheus
• Alerting Best Practices | Prometheus
• PagerDuty Integration Guide
• Alertmanager Setup Guide | Grafana
• Awesome Prometheus Alerts
• Alerts Observability | Cloudflare