Monitoring and Alerting¶

Red Hat OpenShift Monitoring Documentation

OpenShift ships with a pre-configured monitoring stack based on Prometheus and Alertmanager. Read-only dashboards are available under Observe → Dashboards in the web console (the platform no longer ships a Grafana UI). The stack is installed automatically — this page covers how to verify it, enable user workload monitoring, and configure alert routing.

Verify the Built-In Monitoring Stack¶

The cluster monitoring operator deploys Prometheus, Alertmanager, and related components into openshift-monitoring:

oc get pods -n openshift-monitoring

All pods should be Running. Key components:

Component	Purpose
`prometheus-k8s-*`	Scrapes metrics from cluster components and stores time-series data
`alertmanager-main-*`	Routes alerts to receivers (email, Slack, PagerDuty, webhooks)
`thanos-querier-*`	Provides a unified query endpoint across Prometheus instances
`kube-state-metrics-*`	Exposes Kubernetes object state as metrics
`node-exporter-*`	Exposes node-level hardware and OS metrics (one per node)

Access Monitoring Dashboards¶

Go to Observe → Dashboards in the web console
Select a dashboard from the dropdown (e.g., Kubernetes / Compute Resources / Cluster)
Dashboards are read-only and automatically populated

To query metrics directly:

Go to Observe → Metrics

Enter a PromQL query, for example:

sum(rate(container_cpu_usage_seconds_total{namespace!=""}[5m])) by (namespace)

View Active Alerts¶

Go to Observe → Alerting in the web console
Review firing and pending alerts
Common alerts to watch during a POC:

Alert	Meaning
`ClusterOperatorDegraded`	A ClusterOperator is not functioning correctly
`KubePodNotReady`	Pods are not reaching Ready state
`NodeNotReady`	A node has left the Ready state
`etcdHighCommitDurations`	etcd is slow — possible disk or network issue
`KubePersistentVolumeFillingUp`	A PVC is running low on space

From the CLI:

oc get prometheusrule -n openshift-monitoring
oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- \
  promtool query instant http://localhost:9090 'ALERTS{alertstate="firing"}'

Enable User Workload Monitoring¶

By default, Prometheus only monitors OpenShift infrastructure components. To monitor application workloads deployed by users, enable user workload monitoring:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    enableUserWorkload: true

oc apply -f cluster-monitoring-config.yaml

Verify the user workload monitoring components start:

oc get pods -n openshift-user-workload-monitoring

You should see prometheus-user-workload-* and thanos-ruler-user-workload-* pods.

Note

User workload monitoring creates a separate Prometheus instance in openshift-user-workload-monitoring. Applications expose metrics via ServiceMonitor or PodMonitor CRs in their own namespaces.

Configure Alertmanager (Alert Routing)¶

The default Alertmanager receives alerts but does not send notifications anywhere. Configure receivers to route alerts to your team.

View the Current Alertmanager Config¶

oc -n openshift-monitoring get secret alertmanager-main \
  -o jsonpath='{.data.alertmanager\.yaml}' | base64 -d

Configure a Receiver¶

Extract the current Alertmanager config, edit it, and replace the secret:

oc -n openshift-monitoring get secret alertmanager-main --template='{{ index .data "alertmanager.yaml" }}' | base64 --decode > alertmanager.yaml
# Edit alertmanager.yaml to add/modify receivers and routes
oc -n openshift-monitoring create secret generic alertmanager-main --from-file=alertmanager.yaml --dry-run=client -o=yaml | oc -n openshift-monitoring replace secret --filename=-

For example, to send critical alerts to a webhook, edit alertmanager.yaml to contain:

global:
  resolve_timeout: 5m
route:
  group_by: ['namespace', 'alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  receiver: default
  routes:
    - matchers:
        - severity = critical
      receiver: webhook-critical
receivers:
  - name: default
  - name: webhook-critical
    webhook_configs:
      - url: '{{ webhook_url }}'
        send_resolved: true

Then apply with the replace workflow shown above.

Other Receiver Types¶

Receiver	Config Key	Use Case
Email	`email_configs`	Send alerts to a team inbox
Slack	`slack_configs`	Post alerts to a Slack channel
PagerDuty	`pagerduty_configs`	Trigger incidents for on-call rotation
Webhook	`webhook_configs`	Forward to any HTTP endpoint

See the Alertmanager configuration documentation for full receiver configuration options.

Configure Persistent Storage for Metrics¶

By default, Prometheus uses emptyDir volumes — metrics are lost on pod restart. For a POC, configure persistent storage:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    enableUserWorkload: true
    prometheusK8s:
      retention: 15d
      volumeClaimTemplate:
        spec:
          storageClassName: {{ storage_class }}
          resources:
            requests:
              storage: 50Gi
    alertmanagerMain:
      volumeClaimTemplate:
        spec:
          storageClassName: {{ storage_class }}
          resources:
            requests:
              storage: 5Gi

oc apply -f cluster-monitoring-config.yaml

Tip

retention: 15d keeps 15 days of metrics. Adjust based on your POC duration and available storage. For a typical POC, 50Gi with 15-day retention is sufficient.

Create a Custom Alert (Optional)¶

Demonstrate custom alerting by creating a PrometheusRule that fires when a node's CPU utilisation stays above 90% for 10 minutes:

Note

Custom alerting rules belong in a user-defined namespace, not in openshift-monitoring. Enable user workload monitoring first (enableUserWorkload: true in the cluster-monitoring-config ConfigMap), then create your PrometheusRule in a user namespace. The user-workload Prometheus instance automatically picks up rules from any namespace with a ServiceMonitor or PrometheusRule.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: poc-example-alerts
  namespace: poc-monitoring
spec:
  groups:
    - name: poc.rules
      rules:
        - alert: HighNodeCPU
          expr: instance:node_cpu_utilisation:rate5m > 0.9
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "Node {{ $labels.instance }} CPU usage above 90%"
            description: "Node {{ $labels.instance }} has had CPU utilisation above 90% for more than 10 minutes."

oc create namespace poc-monitoring
oc apply -f poc-alerts.yaml

Verify the rule is loaded:

oc get prometheusrule poc-example-alerts -n poc-monitoring

Check in the web console under Observe → Alerting → Alerting Rules to see the new rule.

Verify¶

Dashboards: Go to Observe → Dashboards — confirm data is populating
Metrics: Go to Observe → Metrics — run up and verify targets are scraped
Alerts: Go to Observe → Alerting — confirm no unexpected critical alerts
Storage: Check Prometheus PVCs are bound:
```
oc get pvc -n openshift-monitoring
```