Prometheus PromQL Mastery Guide

• 1. PromQL Fundamentals
  • 4 Data Types
  • Metric Types
• 2. Selectors and Matching
• 3. Core Functions
  • rate() — Per-second rate of change for Counters
  • increase() — Total increase over a time period
  • histogram_quantile() — Percentile calculation
  • irate() — Instantaneous rate of change
• 4. Aggregation Operators
• 5. Practical Query Patterns
  • Error Rate Calculation
  • Resource Utilization
  • Saturation
• 6. Writing Alerting Rules
• 7. Grafana Dashboard Queries
• 8. Quiz
• Quiz

1. PromQL Fundamentals

PromQL (Prometheus Query Language) is the query language used to query and analyze time-series data in Prometheus. Both Grafana dashboards and alerting rules are written in PromQL.

4 Data Types

Metric Types

Counter  — Monotonically increasing value (resets to 0) : http_requests_total
Gauge    — Can go up or down : temperature, memory_usage
Histogram — Distribution measurement (bucket) : http_request_duration_seconds
Summary  — Directly computes quantiles : go_gc_duration_seconds

2. Selectors and Matching

# Basic selector
http_requests_total

# Label matching
http_requests_total{method="GET", status="200"}

# Regex matching
http_requests_total{method=~"GET|POST"}
http_requests_total{status!~"2.."}

# Negation matching
http_requests_total{method!="DELETE"}

# __name__ matching (metric name is also a label)
{__name__=~"http_requests.*"}

3. Core Functions

rate() — Per-second rate of change for Counters

# Requests per second over the last 5 minutes
rate(http_requests_total[5m])

# Requests per second by job
sum(rate(http_requests_total[5m])) by (job)

# Top 5 requests per second by method
topk(5, sum(rate(http_requests_total[5m])) by (method))

Note: rate() should only be used with Counter types. For Gauges, use deriv().

increase() — Total increase over a time period

# Total requests over the last 1 hour
increase(http_requests_total[1h])

# Daily error count
sum(increase(http_requests_total{status=~"5.."}[24h]))

histogram_quantile() — Percentile calculation

# 95th percentile response time
histogram_quantile(0.95,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
)

# 99th percentile by service
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
)

# 50th percentile (median) response time
histogram_quantile(0.5,
  rate(http_request_duration_seconds_bucket[5m])
)

irate() — Instantaneous rate of change

# Instantaneous rate based on the most recent 2 data points
irate(http_requests_total[5m])

rate() computes the average over the entire range, while irate() computes the rate of change between the two most recent data points. For dashboards, irate() is more responsive, while for alerting, rate() is more stable.

4. Aggregation Operators

# sum — Total
sum(rate(http_requests_total[5m])) by (instance)

# avg — Average
avg(node_cpu_seconds_total{mode="idle"}) by (instance)

# max, min
max(container_memory_usage_bytes) by (pod)

# count — Number of time series
count(up == 1) by (job)

# quantile — Quantile
quantile(0.95, rate(http_requests_total[5m]))

# stddev — Standard deviation
stddev(rate(http_requests_total[5m])) by (job)

# without — Aggregate excluding specific labels
sum without (instance)(rate(http_requests_total[5m]))

5. Practical Query Patterns

Error Rate Calculation

# HTTP 5xx error rate (%)
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
* 100

# Availability rate by service
1 - (
  sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
  /
  sum(rate(http_requests_total[5m])) by (service)
)

Resource Utilization

# CPU utilization (%)
100 - (avg by (instance)(
  irate(node_cpu_seconds_total{mode="idle"}[5m])
) * 100)

# Memory utilization (%)
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100

# Disk utilization
(1 - node_filesystem_avail_bytes{mountpoint="/"}
  / node_filesystem_size_bytes{mountpoint="/"}) * 100

Saturation

# Pod CPU throttling
sum(rate(container_cpu_cfs_throttled_seconds_total[5m])) by (pod)
/
sum(rate(container_cpu_cfs_periods_total[5m])) by (pod)

# Queue length
avg_over_time(queue_length[5m])

6. Writing Alerting Rules

# prometheus-rules.yaml
groups:
  - name: application
    rules:
      # High error rate
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
          /
          sum(rate(http_requests_total[5m])) by (service)
          > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: 'High error rate on {{ $labels.service }}'
          description: 'Error rate is {{ $value | humanizePercentage }}'

      # Slow response time
      - alert: SlowResponseTime
        expr: |
          histogram_quantile(0.95,
            sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
          ) > 1.0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: 'P95 latency > 1s on {{ $labels.service }}'

  - name: infrastructure
    rules:
      # High memory usage
      - alert: HighMemoryUsage
        expr: |
          (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) > 0.9
        for: 15m
        labels:
          severity: warning

      # Disk fill prediction
      - alert: DiskWillFillIn24h
        expr: |
          predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 24*3600) < 0
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: 'Disk will fill in 24h on {{ $labels.instance }}'

      # Pod CrashLooping
      - alert: PodCrashLooping
        expr: |
          increase(kube_pod_container_status_restarts_total[1h]) > 5
        for: 10m
        labels:
          severity: critical

7. Grafana Dashboard Queries

# RED Method Dashboard

# Rate (requests/sec)
sum(rate(http_requests_total[5m])) by (service)

# Errors (errors/sec)
sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)

# Duration (latency distribution)
histogram_quantile(0.5, sum(rate(http_request_duration_seconds_bucket[$__rate_interval])) by (le))
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[$__rate_interval])) by (le))
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[$__rate_interval])) by (le))

8. Quiz

Quiz