Container Failover Test¶

This guide walks through testing container workload failover by deploying a sample application with multiple replicas, simulating a node failure, and verifying the pods are rescheduled to healthy nodes. This assumes Workload Availability is fully configured with the 120-second failover settings.

Prerequisites¶

Workload Availability operators installed (NHC, SNR, Descheduler)
At least 3 worker nodes
A default StorageClass available (for testing with persistent volumes)

Deploy a Sample Application¶

Create a namespace for the test:
```
oc new-project failover-test
```

Deploy a sample application with multiple replicas and a pod anti-affinity rule to spread pods across nodes:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: failover-test-app
  namespace: failover-test
spec:
  replicas: 3
  selector:
    matchLabels:
      app: failover-test
  template:
    metadata:
      labels:
        app: failover-test
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app: failover-test
                topologyKey: kubernetes.io/hostname
      containers:
        - name: app
          image: registry.redhat.io/hi/nginx:latest
          ports:
            - containerPort: 8080
          resources:
            requests:
              memory: "128Mi"
              cpu: "100m"
            limits:
              memory: "256Mi"
              cpu: "200m"

oc apply -f failover-test-app.yaml

Create a Service to front the application:

oc expose deployment failover-test-app --port=8080
oc expose service failover-test-app

Verify the Deployment¶

Confirm all 3 replicas are running and distributed across different nodes:
```
oc get pods -o wide -n failover-test
```
You should see pods on 3 different nodes due to the anti-affinity rule.

Record which pods are on which nodes:

oc get pods -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,IP:.status.podIP -n failover-test

Verify the application is responding through the Service:

ROUTE=$(oc get route failover-test-app -o jsonpath='{.spec.host}')
curl -s http://$ROUTE | head -5

Simulate Node Failure¶

Pick one of the nodes that has a pod running on it:

NODE=$(oc get pods -n failover-test -o jsonpath='{.items[0].spec.nodeName}')
echo "Targeting node: $NODE"

Record the pod on that node:

oc get pods -n failover-test --field-selector spec.nodeName=$NODE

Start a timer and reboot the node to simulate a failure:

date +%T && oc debug node/$NODE -- chroot /host systemctl reboot

Watch the Failover¶

Watch the pods for changes:
```
oc get pods -o wide -n failover-test -w
```
You should see the following sequence: - Pods on the failed node remain in Running state initially - After ~50s: node is marked NotReady - After ~80s: NHC creates a SelfNodeRemediation CR - After ~85s: out-of-service taint is applied, pods are force-deleted - After ~90s: new pod is scheduled on a healthy node and starts
Once the replacement pod shows Running, check the timestamp:
```
date +%T
oc get pods -o wide -n failover-test
```
You should see 3 pods running again, with the replacement on a different node.

Verify the Failover¶

Confirm all replicas are back to Running:
```
oc get deployment failover-test-app -n failover-test
```
The AVAILABLE count should be back to 3.
Verify the Service is still responding without interruption:
```
curl -s http://$ROUTE | head -5
```
Because the Service load balances across all healthy pods, the remaining 2 replicas continued serving traffic during the failover. Once the 3rd pod is rescheduled, full capacity is restored.

Check which nodes are now hosting pods:

oc get pods -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,IP:.status.podIP -n failover-test

Review the remediation events:

oc get selfnoderemediation -A
oc get events -n failover-test --sort-by='.lastTimestamp'

Key Differences from VM Failover¶

Aspect	Container Pods	Virtual Machines
Restart behavior	New pod created by Deployment controller	New VMI created by virt-controller
Downtime	Zero (other replicas keep serving)	~120s (single instance restarts)
IP persistence	New pod gets a new IP (Service abstracts this)	Same IP via CUDN persistent IPAM
Storage	StatefulSet PVCs reattach; Deployment PVCs are fresh	RWX disks reattach to new node
Scaling	Multiple replicas provide built-in HA	Single instance, relies on failover

Why Containers Recover Faster

With multiple replicas behind a Service, container workloads experience zero downtime from the client's perspective during a node failure. The Service immediately stops routing to the unavailable pod, and the remaining replicas handle all traffic. The Deployment controller then creates a replacement pod on a healthy node to restore full capacity — but the application was never actually "down."

Test with a StatefulSet (Persistent Storage)¶

To test failover with persistent volumes, deploy a StatefulSet instead:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: failover-test-stateful
  namespace: failover-test
spec:
  replicas: 3
  serviceName: failover-test-stateful
  selector:
    matchLabels:
      app: failover-test-stateful
  template:
    metadata:
      labels:
        app: failover-test-stateful
    spec:
      containers:
        - name: app
          image: registry.redhat.io/hi/nginx:latest
          ports:
            - containerPort: 8080
          volumeMounts:
            - name: data
              mountPath: /var/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Gi

RWO StatefulSet Failover

StatefulSet pods using RWO volumes depend on the out-of-service taint to force-detach the volume from the failed node. Without Workload Availability (NHC + SNR), Kubernetes would wait for the node lease to expire (6+ minutes) before allowing the volume to attach elsewhere.

Cleanup¶

Delete the test resources:
```
oc delete project failover-test
```
Wait for the rebooted node to come back:
```
oc get nodes -w
```