Hub and Spoke

For fleet management, we recommend starting with a Single Node OpenShift (SNO) installation as the hub cluster, then installing Red Hat Advanced Cluster Management (ACM) to manage your cluster fleet at scale.

Process Overview

1. Complete all prerequisites
2. Set up the installation host
3. Install the SNO hub cluster
4. Install storage on the hub
5. Install Advanced Cluster Management
6. Use ACM to provision bare metal spoke clusters

Why Hub and Spoke?

Benefit	Description
Centralized control	Manage all clusters from a single pane of glass
Consistent policy	Enforce governance and security policies across the fleet
Scalable provisioning	Create new clusters on demand through ACM
Lifecycle management	Upgrade and maintain clusters centrally
Observability	Unified view of cluster health and compliance

---

Install the Hub Cluster (SNO)

A Single Node OpenShift (SNO) cluster runs the control plane and workloads on a single host. It serves as the management hub for the fleet. We use the Assisted Installer to provision this cluster.

Prerequisites

• Complete the prerequisites
• Set up the installation host

Install Using the Assisted Installer

Follow the Assisted Installer guide with the following differences for SNO:

Setting	Full Cluster (6-node)	SNO Hub
Number of control plane nodes	3	1 (Single Node OpenShift)
Number of workers	3	0
Hosts to boot	All 6	1
API VIP / Ingress VIP	Separate VIP addresses	Not required (uses host IP)
Networking -> VIPs	Fill in both	host IP
Installation time	30-45 minutes	20-30 minutes

Note

Since SNO has a single node, the API and Ingress traffic goes directly to that node's IP. You do not need to configure VIPs in the Networking step — the installer will skip that section for SNO.

---

Configure Hub Storage

OpenShift Storage - Persistent Storage using LVMS

After installing the SNO hub cluster, configure storage before installing ACM. These examples are for environments without existing external storage on the hub node.

Install LVM Storage Operator

1. Go to Ecosystem -> Software Catalog -> filter for "LVM Storage" -> click the tile
2. Click Install
3. Leave all the defaults and click Install
4. Wait for the Operator to install

Label the node as a storage node only if you are installing OpenShift Data Foundation (see the optional section below). LVM Storage does not use this label.

Create LVM Storage

1. Go to Ecosystem -> Installed Operators -> click "LVM Storage"
2. Click on the "LVMCluster" tab and then click "Create LVMCluster"

3. Switch to YAML view and update the path for your data disk:

   apiVersion: lvm.topolvm.io/v1alpha1
   kind: LVMCluster
   metadata:
     name: local-storage-lvm-cluster
     namespace: openshift-lvm-storage
   spec:
     storage:
       deviceClasses:
         - name: local-storage
           default: true
           fstype: xfs
           deviceSelector:
             paths:
               - /dev/nvme0n1
           thinPoolConfig:
             name: thin-pool-1
             sizePercent: 90
             overprovisionRatio: 10
             chunkSizeCalculationPolicy: Host
             metadataSizeCalculationPolicy: Host
   

4. Click Create

OPTIONAL — Install OpenShift Data Foundation (Object Storage)

OpenShift Data Foundation Documentation

This is only needed if you are planning to use ODF as part of your OPP subscription. ODF is used here specifically for object storage required by MultiCluster Observability. If your storage provider does not offer object storage, you can deploy MinIO instead. See Using MinIO.

Label the hub node for ODF before creating the StorageCluster:

oc label node {{ node_name }} cluster.ocs.openshift.io/openshift-storage=

1. Go to Ecosystem -> Software Catalog -> filter for "OpenShift Data Foundation" -> click the tile
2. Click Install
3. Leave all the defaults and click Install
4. Wait for the Operator to install
5. Go to Ecosystem -> Installed Operators -> click "OpenShift Data Foundation"
6. Click on the "StorageCluster" tab and then click "Create StorageCluster"

7. Switch to YAML view and paste:

   apiVersion: ocs.openshift.io/v1
   kind: StorageCluster
   metadata:
     name: mcog-storagecluster
     namespace: openshift-storage
   spec:
     arbiter: {}
     encryption:
       keyRotation:
         schedule: '@weekly'
       kms: {}
     externalStorage: {}
     managedResources:
       cephObjectStoreUsers: {}
       cephCluster: {}
       cephBlockPools: {}
       cephNonResilientPools: {}
       cephObjectStores: {}
       cephFilesystems: {}
       cephRBDMirror: {}
       cephToolbox: {}
       cephDashboard: {}
       cephConfig: {}
     multiCloudGateway:
       dbStorageClassName: lvms-local-storage
       reconcileStrategy: standalone
     resourceProfile: balanced
   

8. Click Create

Verify Storage

oc get lvmcluster -n openshift-lvm-storage
oc get storagecluster -n openshift-storage
oc get storageclass
oc get pods -n openshift-storage

Ensure the LVM StorageClass is set as default:

oc get storageclass | grep default

---

Install Advanced Cluster Management

Red Hat ACM Documentation

Red Hat Advanced Cluster Management (ACM) provides multicluster lifecycle management, governance, and observability. Installing ACM also automatically installs the multicluster engine operator.

Note

Only one ACM hub cluster can exist per OpenShift cluster.

Prerequisites

• Hub cluster is installed and storage is configured
• Cluster administrator privileges

Install the Operator via WebUI

1. Go to Ecosystem -> Software Catalog -> filter for "Advanced Cluster Management" -> click the tile
2. Click Install
3. Leave all the defaults and click Install
4. Wait for the Operator to install
5. Go to Ecosystem -> Installed Operators -> click "Advanced Cluster Management for Kubernetes"
6. Click on the "MultiClusterHub" tab and then click "Create MultiClusterHub"
7. Leave all the defaults and click Create

Note

It can take up to 10 minutes for the hub to finish deploying all components.

1. Wait for the status to show Running

Install the Operator via YAML (click to expand)

apiVersion: v1
kind: Namespace
metadata:
  name: open-cluster-management
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: open-cluster-management
  namespace: open-cluster-management
spec:
  targetNamespaces:
    - open-cluster-management
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: acm-operator-subscription
  namespace: open-cluster-management
spec:
  sourceNamespace: openshift-marketplace
  source: redhat-operators
  channel: release-2.17
  installPlanApproval: Automatic
  name: advanced-cluster-management

Verify the ACM Channel

The release-2.17 channel above corresponds to the ACM release stream, not your OCP version. Before applying, verify the default ACM channel: oc get packagemanifest advanced-cluster-management -o jsonpath='{.status.defaultChannel}'

oc apply -f acm-operator.yaml

Wait for the operator to install:

oc get csv -n open-cluster-management -w

The PHASE should show Succeeded.

Create the MultiClusterHub

apiVersion: operator.open-cluster-management.io/v1
kind: MultiClusterHub
metadata:
  name: multiclusterhub
  namespace: open-cluster-management
spec:
  availabilityConfig: Basic

oc apply -f multiclusterhub.yaml

Monitor the status:

oc get mch -n open-cluster-management -w

The status should show Running.

Verify ACM

oc get pods -n open-cluster-management

ACM 2.17 integrates as an OpenShift console plug-in (the Fleet Management perspective) rather than serving a separate multicloud-console route. To access ACM, open the OpenShift web console and switch to the Fleet Management perspective in the left navigation.

Verify the console plug-in is active:

oc get consoleplugin acm -o jsonpath='{.status}'

Enable Bare Metal Provisioning

Enable bare metal provisioning for spoke cluster deployment. On Assisted Installer or bare-metal SNO hubs, the Provisioning CR usually already exists. Check first:

oc get provisioning provisioning-configuration

If it exists, patch it to watch all namespaces:

oc patch provisioning provisioning-configuration \
  --type merge -p '{"spec":{"watchAllNamespaces": true}}'

If it does not exist, create it:

apiVersion: metal3.io/v1alpha1
kind: Provisioning
metadata:
  name: provisioning-configuration
spec:
  provisioningNetwork: "Disabled"
  watchAllNamespaces: true

oc apply -f provisioning.yaml

Create the AgentServiceConfig

The AgentServiceConfig CR enables the Assisted Service (part of multicluster engine) which handles bare metal cluster provisioning. Without it, the assisted-service pods won't deploy and InfraEnv creation will fail.

apiVersion: agent-install.openshift.io/v1beta1
kind: AgentServiceConfig
metadata:
  name: agent
spec:
  databaseStorage:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 10Gi
  filesystemStorage:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 20Gi
  imageStorage:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 50Gi

oc apply -f agentserviceconfig.yaml

Wait for the assisted-service pods to start:

oc get pods -n multicluster-engine -l app=assisted-service

Note

In a connected environment, the Assisted Service automatically pulls the correct RHCOS images for each OpenShift version. In disconnected environments, you must also specify spec.osImages with references to locally mirrored ISO and rootFS images, and spec.mirrorRegistryRef pointing to a ConfigMap with your mirror registry configuration.

Import an Existing Cluster

To import an existing cluster (one not provisioned by ACM) into the hub:

1. Create the ManagedCluster resource on the hub:

   apiVersion: cluster.open-cluster-management.io/v1
   kind: ManagedCluster
   metadata:
     name: {{ managed_cluster_name }}
   spec:
     hubAcceptsClient: true
   

   oc apply -f managed-cluster.yaml
   

2. Wait for ACM to generate the import secret:

   oc get secret -n {{ managed_cluster_name }} | grep import
   

3. Extract the klusterlet CRD and import YAML from the import secret:

   oc get secret {{ managed_cluster_name }}-import -n {{ managed_cluster_name }} \
     -o jsonpath='{.data.crds\.yaml}' | base64 --decode > klusterlet-crd.yaml
   
   oc get secret {{ managed_cluster_name }}-import -n {{ managed_cluster_name }} \
     -o jsonpath='{.data.import\.yaml}' | base64 --decode > import.yaml
   

4. Apply the klusterlet CRD first on the managed cluster, then apply the import YAML:

   oc apply -f klusterlet-crd.yaml --kubeconfig={{ managed_cluster_kubeconfig }}
   oc apply -f import.yaml --kubeconfig={{ managed_cluster_kubeconfig }}
   

   Warning

   The klusterlet CRD must be applied before the import YAML. Applying them out of order will cause errors because the import resources depend on the CRD definitions.

5. Create the KlusterletAddonConfig on the hub to enable ACM add-ons on the managed cluster:

   apiVersion: agent.open-cluster-management.io/v1
   kind: KlusterletAddonConfig
   metadata:
     name: {{ managed_cluster_name }}
     namespace: {{ managed_cluster_name }}
   spec:
     clusterName: {{ managed_cluster_name }}
     clusterNamespace: {{ managed_cluster_name }}
     applicationManager:
       enabled: true
     certPolicyController:
       enabled: true
     policyController:
       enabled: true
     searchCollector:
       enabled: true
   

   oc apply -f klusterlet-addon-config.yaml
   

6. Verify the managed cluster is connected:

   oc get managedcluster {{ managed_cluster_name }}
   

   The HubAcceptedManagedCluster condition should be True and the cluster should show as Available.

---

Provision a Bare Metal Spoke Cluster

Red Hat ACM Documentation - Host Inventory

This section covers using ACM to provision a new bare metal OpenShift cluster. ACM handles the entire lifecycle: booting hosts with a discovery ISO via BMC, registering them as agents, and installing the cluster.

Prerequisites

• ACM is installed and configured on the hub cluster (see above)
• Bare metal provisioning is enabled
• BMC credentials available for all target hosts
• DNS records configured for the spoke cluster (API, Ingress, nodes)

Create InfraEnv

The InfraEnv defines the discovery environment for spoke cluster hosts. ACM uses this to generate a discovery ISO and automatically attach it to hosts via BMC.

1. Create the namespace for the spoke cluster:

   oc create namespace {{ spoke_cluster_name }}
   

2. Create the pull secret in the spoke namespace:

   oc create secret generic pullsecret-{{ spoke_cluster_name }} \
     --from-file=.dockerconfigjson=~/pull-secret.txt \
     --type=kubernetes.io/dockerconfigjson \
     -n {{ spoke_cluster_name }}
   

3. Create the InfraEnv:

   apiVersion: agent-install.openshift.io/v1beta1
   kind: InfraEnv
   metadata:
     name: {{ spoke_cluster_name }}
     namespace: {{ spoke_cluster_name }}
     annotations:
       infraenv.agent-install.openshift.io/enable-ironic-agent: 'true'
     labels:
       agentclusterinstalls.extensions.hive.openshift.io/location: {{ dc_location_name }}
       networkType: static
   spec:
     agentLabels:
       agentclusterinstalls.extensions.hive.openshift.io/location: {{ dc_location_name }}
     cpuArchitecture: x86_64
     imageType: full-iso # (1)!
     ipxeScriptType: DiscoveryImageAlways
     additionalNTPSources:
       - {{ ntp_server_1 }}
       - {{ ntp_server_2 }}
     nmStateConfigLabelSelector:
       matchLabels:
         infraenvs.agent-install.openshift.io: {{ spoke_cluster_name }}
     pullSecretRef:
       name: pullsecret-{{ spoke_cluster_name }}
     sshAuthorizedKey: {{ public_key }}
     # additionalTrustBundle: |
     #   -----BEGIN CERTIFICATE-----
     #   < your proxy or internal CA certificate in PEM format >
     #   -----END CERTIFICATE-----
   

   1. full-iso (~1 GB) includes the RHCOS rootfs so hosts boot without downloading anything over the network — use this for disconnected or bandwidth-constrained environments. minimal-iso (~100 MB) downloads the rootfs at boot time and is the default if omitted.

   NTP

   Use the customer's NTP servers. pool.ntp.org only works if nodes have outbound internet, which most on-prem POCs do not.

   TLS-Intercepting (MITM) Proxy

   If your environment uses a TLS-intercepting proxy, you must uncomment and populate the additionalTrustBundle field with your proxy's root or intermediate CA certificate in PEM format. Without it, the discovery agent cannot pull images from Red Hat registries and will fail silently. See Networking — TLS-Intercepting (MITM) Proxy for how to obtain the certificate.

   oc apply -f infraenv.yaml
   

Adding BareMetalHost to the host inventory

Static IP Configuration

When the discovery ISO boots on a host, the agent inside needs a working network connection to call home to the hub and register. Without DHCP on the data network, the agent has no IP and cannot reach the hub — the host will boot but never appear as an Agent.

NMStateConfig resources provide this static network configuration. The InfraEnv picks up all NMStateConfig resources that match its label selector and bakes them into the discovery ISO. When a host boots, the agent matches the config to the host by MAC address and configures the NIC before phoning home.

Warning

You must create one NMStateConfig for every host that will be registered as a BareMetalHost. If a host does not have a matching NMStateConfig, it will boot the discovery ISO but have no network connectivity and will never register as an Agent.

Create all NMStateConfig resources before creating the BareMetalHosts. Use existing NMStateConfig documentation

Auto-generate from Redfish

Instead of manually collecting MAC addresses from each server, you can query the BMC via Redfish to discover interfaces and generate NMStateConfig YAML automatically. See Redfish Network Discovery.

apiVersion: agent-install.openshift.io/v1beta1
kind: NMStateConfig
metadata:
  name: {{ hostname }}
  namespace: {{ spoke_cluster_name }}
  labels:
    infraenvs.agent-install.openshift.io: {{ spoke_cluster_name }} # must match InfraEnv nmStateConfigLabelSelector
spec:
  interfaces:
    - name: eno1 # Use the actual interface names!
      macAddress: {{ mac_address }}
  config:
    interfaces:
      - name: eno1
        type: ethernet
        state: up
        ipv4:
          enabled: true
          address:
            - ip: {{ host_ip }}
              prefix-length: {{ prefix_length }}
          dhcp: false
    dns-resolver:
      config:
        server:
          - {{ dns_server }}
    routes:
      config:
        - destination: 0.0.0.0/0
          next-hop-address: {{ gateway }}
          next-hop-interface: eno1

Note

The interfaces[].macAddress at the bottom maps the NMState config to the correct physical NIC. This must be the same MAC used in the BareMetalHost bootMACAddress field. The interface name (eno1) must match the actual NIC name on the host.

Finding NIC Names and MAC Addresses

If you are unsure of the NIC names or MAC addresses on your hosts, boot one with the RHCOS Live ISO and run ip link from the console. You don't need to install anything — the live environment gives you all the hardware details you need.

Tip

This is only the discovery stage networking — enough for the agent to phone home and register. The final cluster networking (bonds, VLANs, production static IPs) is configured separately in the cluster install manifests.

Repeat for every host, updating name, host_ip, and macAddress for each. Then apply them all:

oc apply -f {{ hostname }}-nmstate.yaml

Add Host Inventory via BMC

Once the InfraEnv is created, register bare metal hosts. ACM will automatically boot each host with the discovery ISO via Redfish virtual media — no manual ISO download or mounting is required.

BMC Address Formats

The bmc.address field varies by hardware vendor. See Infrastructure — BMC / Out-of-Band Management for the address format table and instructions on discovering the system ID via the Redfish API.

Repeat the following for each host in the spoke cluster.

1. Create the BMC credentials Secret

apiVersion: v1
kind: Secret
metadata:
  name: {{ hostname }}-bmc-secret
  namespace: {{ spoke_cluster_name }}
type: Opaque
stringData:
  username: {{ bmc_username }}
  password: {{ bmc_password }}

oc apply -f {{ hostname }}-bmc-secret.yaml

Tip

Using stringData: lets you supply plain-text values. If you prefer data:, base64-encode first: echo -n 'value' | base64.

2. Create the BareMetalHost

The bmc.address format is vendor-specific — use the correct scheme and system ID for your hardware.

Note

The namespace for the InfraEnv and BareMetalHost must be the same.

Dell iDRACHPE iLOLenovo XCCSupermicro

apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: {{ hostname }}
  namespace: {{ spoke_cluster_name }}
  annotations:
    inspect.metal3.io: disabled
    bmac.agent-install.openshift.io/hostname: {{ hostname }}
  labels:
    infraenvs.agent-install.openshift.io: {{ spoke_cluster_name }} # must match InfraEnv nmStateConfigLabelSelector
spec:
  online: true
  bootMACAddress: {{ boot_mac_address }} # (1)!
  bmc:
    address: idrac-virtualmedia://{{ bmc_ip }}/redfish/v1/Systems/System.Embedded.1
    credentialsName: {{ hostname }}-bmc-secret
    disableCertificateVerification: true
  bootMode: UEFI
  rootDeviceHints:
    deviceName: /dev/sda
  automatedCleaningMode: disabled

1. Must be the MAC of a physical NIC (e.g. eno1), not the BMC/iDRAC management port. For bonded/VLAN setups, use the first physical bond member's MAC.

Dell-Specific Requirements

• iDRAC firmware — virtual media via Redfish needs a reasonably current iDRAC. On iDRAC 9, use 4.40.00.00 or newer; older firmware has flaky or missing virtual-media Redfish support. iDRAC 8 works but is more limited.
• Enterprise/Datacenter license — virtual media requires it. The Express license does not expose the virtual media endpoint.

idrac-virtualmedia vs redfish-virtualmedia

Ironic ships a Dell-optimized driver, idrac-virtualmedia://, which uses the same address format but handles Dell quirks (like boot-mode setting) more reliably. It is supported on OpenShift and is the recommended default for Dell hardware. redfish-virtualmedia:// is not supported on Dell hardware.

rootDeviceHints on Dell

If these are PERC RAID setups, /dev/sda is usually correct. On NVMe or multi-disk boxes, prefer matching by wwn or serialNumber so you don't install to the wrong disk if a reboot reorders device names.

apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: {{ hostname }}
  namespace: {{ spoke_cluster_name }}
  annotations:
    inspect.metal3.io: disabled
    bmac.agent-install.openshift.io/hostname: {{ hostname }}
  labels:
    infraenvs.agent-install.openshift.io: {{ spoke_cluster_name }} # must match InfraEnv nmStateConfigLabelSelector
spec:
  online: true
  bootMACAddress: {{ boot_mac_address }} # (1)!
  bmc:
    address: redfish-virtualmedia://{{ bmc_ip }}/redfish/v1/Systems/1
    credentialsName: {{ hostname }}-bmc-secret
    disableCertificateVerification: true
  bootMode: UEFI
  rootDeviceHints:
    deviceName: /dev/sda
  automatedCleaningMode: disabled

1. Must be the MAC of a physical NIC (e.g. eno1), not the BMC/iDRAC management port. For bonded/VLAN setups, use the first physical bond member's MAC.

apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: {{ hostname }}
  namespace: {{ spoke_cluster_name }}
  annotations:
    inspect.metal3.io: disabled
    bmac.agent-install.openshift.io/hostname: {{ hostname }}
  labels:
    infraenvs.agent-install.openshift.io: {{ spoke_cluster_name }} # must match InfraEnv nmStateConfigLabelSelector
spec:
  online: true
  bootMACAddress: {{ boot_mac_address }} # (1)!
  bmc:
    address: redfish-virtualmedia://{{ bmc_ip }}/redfish/v1/Systems/1
    credentialsName: {{ hostname }}-bmc-secret
    disableCertificateVerification: true
  bootMode: UEFI
  rootDeviceHints:
    deviceName: /dev/sda
  automatedCleaningMode: disabled

1. Must be the MAC of a physical NIC (e.g. eno1), not the BMC/iDRAC management port. For bonded/VLAN setups, use the first physical bond member's MAC.

apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: {{ hostname }}
  namespace: {{ spoke_cluster_name }}
  annotations:
    inspect.metal3.io: disabled
    bmac.agent-install.openshift.io/hostname: {{ hostname }}
  labels:
    infraenvs.agent-install.openshift.io: {{ spoke_cluster_name }} # must match InfraEnv nmStateConfigLabelSelector
spec:
  online: true
  bootMACAddress: {{ boot_mac_address }} # (1)!
  bmc:
    address: redfish-virtualmedia://{{ bmc_ip }}/redfish/v1/Systems/1
    credentialsName: {{ hostname }}-bmc-secret
    disableCertificateVerification: true
  bootMode: UEFI
  rootDeviceHints:
    deviceName: /dev/sda
  automatedCleaningMode: disabled

1. Must be the MAC of a physical NIC (e.g. eno1), not the BMC/iDRAC management port. For bonded/VLAN setups, use the first physical bond member's MAC.

bootMACAddress — Use the Data NIC, Not the BMC

The bootMACAddress must be the MAC of the data/production NIC on the host (e.g. eno1) — the interface that will carry cluster traffic and receive an IP via DHCP or NMStateConfig. Do not use the BMC/iDRAC management port MAC. The BMC has its own dedicated MAC on the management network. Using the wrong MAC is a common cause of hosts never getting an IP after booting the discovery ISO.

Bonded / VLAN environments: If your NMStateConfig uses an LACP bond with a VLAN (e.g. bond0.100), use the MAC of the first physical bond member (e.g. eno1), not the bond or VLAN interface. Bonds and VLANs are derived interfaces that don't exist until the NMStateConfig is applied after the discovery ISO boots — the server only knows its physical NICs at early boot. This same MAC must also appear in the NMStateConfig spec.interfaces mapping so the assisted installer can match the config to the host.

Field Reference

Field	Description
inspect.metal3.io: disabled	Annotation that skips Ironic hardware inspection. Required for ACM/InfraEnv host inventory so hosts register as agents instead of staying in inspecting.
bmac.agent-install.openshift.io/hostname	Sets the hostname on the Agent that registers from this BareMetalHost. Without it, the agent may get a random or DHCP-assigned hostname.
bmc.address	The redfish-virtualmedia:// scheme avoids the provisioning-network requirement. The system ID at the end is vendor-specific (see tabs above).
bootMACAddress	MAC of the data/production NIC (e.g. eno1) — the interface used for cluster traffic. This is not the BMC/iDRAC management port MAC. For bonded/VLAN setups, use the first physical bond member's MAC. Using the wrong MAC is a common cause of hosts never getting an IP after booting the discovery ISO.
disableCertificateVerification	Usually required since BMCs ship with self-signed certs. Remove it if you have installed valid certificates.
bootMode	UEFI (default), legacy, or UEFISecureBoot.
rootDeviceHints	Optional but recommended so Ironic installs to the correct disk. Can also match on model, serialNumber, wwn, minSizeGigabytes, etc.
automatedCleaningMode	Set to disabled for POC to skip the disk-wipe step during provisioning. In production, consider leaving it enabled.

Discovering the System ID

If you're unsure of the system ID for your hardware, query the Redfish API:

curl -sk https://{{ bmc_ip }}/redfish/v1/Systems/ \
  -u {{ bmc_username }}:{{ bmc_password }} | jq '.Members'

See Infrastructure — BMC / Out-of-Band Management for more details.

oc apply -f {{ hostname }}-bmh.yaml

3. Watch for hosts to boot and register as agents

oc get bmh -n {{ spoke_cluster_name }}
oc get agents -n {{ spoke_cluster_name }} -w

With inspect.metal3.io: disabled set, each host transitions through: registering → available (Ironic hardware inspection is skipped). ACM then boots the InfraEnv discovery ISO via virtual media so the host registers as an Agent. Once all hosts show as agents, you can create the cluster.

Warning

Do not omit inspect.metal3.io: disabled on ACM/InfraEnv BareMetalHosts. Without it, the host stays in Ironic inspecting waiting for the IPA ramdisk instead of registering as an assisted-installer agent.

What happens during discovery boot

After the BareMetalHost is available, the Bare Metal Operator attaches the InfraEnv discovery ISO and powers the host on:

1. Power on — The Bare Metal Operator sends a power-on command via the BMC (iDRAC, iLO, XCC, etc.).
2. Hardware POST — The server runs its Power-On Self-Test. Depending on the hardware, this alone can take 5–15 minutes. To reduce this, disable the POST memory test on POC hardware.
3. Virtual media boot — The server boots the discovery ISO mounted by the BMC (not PXE / IPA inspection).
4. Agent startup — The assisted installer agent starts and applies any matching NMStateConfig.
5. Call home — The agent registers with the hub; oc get agents shows the new host.

Watch the BMC console

If you are waiting for the agent to come up, open the server's remote console (iDRAC, iLO, XCC) and watch the screen. Once you see the Linux kernel booting, the agent is only seconds away from starting. Long waits are often still in POST — not a hang.

Approve Agents

Before creating the cluster, verify all agents are discovered and approve them. Agents must be approved before they can be assigned to a cluster.

1. List the agents and confirm all expected hosts appear:

   oc get agents -n {{ spoke_cluster_name }}
   

   You should see one agent per BareMetalHost.

2. Approve each agent:

   oc patch agent {{ agent_name }} -n {{ spoke_cluster_name }} \
     --type merge -p '{"spec":{"approved":true}}'
   

   Or approve all at once:

   for agent in $(oc get agents -n {{ spoke_cluster_name }} -o jsonpath='{.items[*].metadata.name}'); do
     oc patch agent "$agent" -n {{ spoke_cluster_name }} \
       --type merge -p '{"spec":{"approved":true}}'
   done
   

3. Set the role for each agent. You need exactly 3 master agents and the rest as worker:

   oc patch agent {{ agent_name }} -n {{ spoke_cluster_name }} \
     --type merge -p '{"spec":{"role":"master"}}'
   

   Tip

   If you don't set roles manually, the installer will auto-assign them, but it's better to be explicit — especially if your control plane nodes differ from your workers in hardware specs.

Create the Cluster

Once all agents are approved and showing known status, create the cluster resources to trigger installation.

Choosing a ClusterImageSet

The imageSetRef.name must reference a ClusterImageSet that exists on the hub. ACM installs several automatically. List available versions with:

oc get clusterimageset

Pick the version that matches your target OpenShift release (e.g., img4.22.11-x86-64).

1. Create the ClusterDeployment and AgentClusterInstall:

   apiVersion: hive.openshift.io/v1
   kind: ClusterDeployment
   metadata:
     name: {{ spoke_cluster_name }}
     namespace: {{ spoke_cluster_name }}
   spec:
     baseDomain: {{ base_domain }}
     clusterName: {{ spoke_cluster_name }}
     controlPlaneConfig:
       servingCertificates: {}
     installed: false
     clusterInstallRef:
       group: extensions.hive.openshift.io
       kind: AgentClusterInstall
       name: {{ spoke_cluster_name }}
       version: v1beta1
     platform:
       agentBareMetal:
         agentSelector:
           matchLabels:
             infraenvs.agent-install.openshift.io: {{ spoke_cluster_name }}
     pullSecretRef:
       name: pullsecret-{{ spoke_cluster_name }}
   ---
   apiVersion: extensions.hive.openshift.io/v1beta1
   kind: AgentClusterInstall
   metadata:
     name: {{ spoke_cluster_name }}
     namespace: {{ spoke_cluster_name }}
   spec:
     platformType: BareMetal
     clusterDeploymentRef:
       name: {{ spoke_cluster_name }}
     imageSetRef:
       name: {{ cluster_image_set }}
     networking:
       clusterNetwork:
         - cidr: 10.128.0.0/14
           hostPrefix: 23
       serviceNetwork:
         - 172.30.0.0/16
       machineNetwork:
         - cidr: {{ machine_network_cidr }}
     provisionRequirements:
       controlPlaneAgents: 3
       workerAgents: 3
     sshPublicKey: {{ public_key }}
     apiVIPs:
       - {{ api_vip }}
     ingressVIPs:
       - {{ ingress_vip }}
   

   oc apply -f cluster-deployment.yaml
   

2. Verify all agents are bound to the cluster before installation begins:

   oc get agents -n {{ spoke_cluster_name }} \
     -o custom-columns=NAME:.metadata.name,ROLE:.spec.role,APPROVED:.spec.approved,STATUS:.status.debugInfo.state
   

   All agents should show approved: true and status binding or known. If any agent shows insufficient, check its conditions:

   oc get agent {{ agent_name }} -n {{ spoke_cluster_name }} -o jsonpath='{.status.conditions}' | jq .
   

3. Monitor the installation:

   oc get agentclusterinstall {{ spoke_cluster_name }} -n {{ spoke_cluster_name }} -o jsonpath='{.status.conditions}' | jq .
   

   Or watch for completion:

   oc get agentclusterinstall {{ spoke_cluster_name }} -n {{ spoke_cluster_name }} -w
   

   The installation progresses through: requirements-met → preparing-to-install → installing → finalizing → installed. A typical 6-node cluster takes 45–60 minutes.

   Warning

   If the status stalls on installing for more than 90 minutes, check the individual host progress:

   oc get agents -n {{ spoke_cluster_name }} \
     -o custom-columns=NAME:.metadata.name,STAGE:.status.progress.currentStage,PROGRESS:.status.progress.progressInfo
   

4. Once complete, retrieve the kubeconfig and credentials:

   oc get secret {{ spoke_cluster_name }}-admin-kubeconfig -n {{ spoke_cluster_name }} \
     -o jsonpath='{.data.kubeconfig}' | base64 -d > {{ spoke_cluster_name }}-kubeconfig
   

   Retrieve the kubeadmin password:

   oc get secret {{ spoke_cluster_name }}-admin-password -n {{ spoke_cluster_name }} \
     -o jsonpath='{.data.password}' | base64 -d && echo
   

5. Verify the spoke cluster:

   oc --kubeconfig={{ spoke_cluster_name }}-kubeconfig get nodes
   oc --kubeconfig={{ spoke_cluster_name }}-kubeconfig get clusterversion
   oc --kubeconfig={{ spoke_cluster_name }}-kubeconfig get co
   

   All nodes should be Ready, the cluster version should match the target release, and all ClusterOperators should show Available=True.

6. Verify the spoke is managed by ACM:

   oc get managedcluster {{ spoke_cluster_name }}
   

   The cluster should show HubAcceptedManagedCluster=True and ManagedClusterConditionAvailable=True.

Troubleshooting

Symptom	Likely Cause	Fix
BMH stays in registering	BMC unreachable or credentials wrong	Verify curl -sk https://{{ bmc_ip }}/redfish/v1/Systems/ works from the hub; check the Secret
BMH stays in inspecting	Missing inspect.metal3.io: disabled (Ironic IPA inspect path)	Add annotation inspect.metal3.io: disabled on the BareMetalHost; confirm InfraEnv label and online: true
BMH provisioning but host doesn't boot ISO	Virtual media not supported or firmware too old	Check iDRAC/iLO firmware version; confirm Enterprise/Datacenter license (Dell)
Agent shows insufficient	Host doesn't meet minimum requirements (CPU, RAM, disk)	Check oc get agent <name> -o jsonpath='{.status.conditions}'; resolve the flagged validation
Agent shows pending-for-input	Missing network config or role assignment	Approve the agent and assign a role (master or worker)
Install stalls at installing	Host stuck downloading or writing to disk	Check per-host progress with oc get agents custom-columns; look for disk or network errors
Install fails with certificate errors	BMC cert verification failing	Ensure disableCertificateVerification: true is set on the BareMetalHost
Spoke cluster not showing in ACM	ManagedCluster not created or klusterlet not deployed	Check oc get managedcluster; the ClusterDeployment should auto-create it

Deleting BareMetalHosts

Delete a single BMH:

oc delete bmh {{ hostname }} -n {{ spoke_cluster_name }}

Delete all BMHs in a spoke namespace:

oc delete bmh --all -n {{ spoke_cluster_name }}

Tearing Down a Spoke Cluster

If you're tearing down a spoke cluster entirely, delete in this order to avoid orphaned resources:

# 1. Delete the cluster (ClusterDeployment triggers uninstall)
oc delete clusterdeployment {{ spoke_cluster_name }} -n {{ spoke_cluster_name }}

# 2. Wait for agents to unbind, then delete BMHs
oc delete bmh --all -n {{ spoke_cluster_name }}

# 3. Delete the InfraEnv
oc delete infraenv {{ spoke_cluster_name }} -n {{ spoke_cluster_name }}

# 4. Clean up secrets and namespace
oc delete namespace {{ spoke_cluster_name }}

Do Not Remove Finalizers

Do not remove finalizers from BMH objects to force deletion — this can orphan Ironic state and leave the host in a bad state.

BMH Stuck in Deleting

If a BMH sticks in Deleting, it is usually Ironic's cleaning process retrying (up to 3 times, ~5 min each). If it never completes, set automatedCleaningMode: disabled on the BMH and re-delete. Since this guide sets automatedCleaningMode: disabled on all BMHs, you should not hit this issue.