Home / Troubleshooting / MetroCluster

MetroCluster Troubleshooting

Symptoms: metrocluster check reports errors, the mediator is unreachable, a switchover won't complete, or switchback leaves aggregates mirrored-but-degraded.

Scope This guide covers two-node and four-node Fabric-Attached MetroCluster on ONTAP 9.x. For fundamentals (FC vs IP, mediator, switchover/switchback theory) see the MetroCluster reference page; this page is the triage runbook. Always run set -privilege advanced where noted, and open a case with NetApp support before any forced operation — MetroCluster forced switchover can cause data loss if the failed site's NVRAM still holds unmirrored writes.

1. Start with a full health check

Run the built-in validation suite first — it checks cabling, disks, aggregation, configuration quorum, and replication in one pass:

metrocluster check run
metrocluster check show          # summarize results after the run completes
metrocluster operation show      # status of any in-flight operation

Each subsystem reports ok or a list of errors. Fix errors in this order: configuration quorum → mediator → disks/aggregates → replication. Everything else depends on the lower layers being healthy.

2. Mediator problems

The ONTAP Mediator breaks tie races when both sites claim the partner's storage. Symptoms: metrocluster check flags configuration quorum, or switchover says the mediator is unreachable.

metrocluster configuration-settings mediator show
# fields to verify: status = reachable, quorum = true, version matches ONTAP requirement

Unreachable: ping the mediator IP from the cluster management LIF (network ping -lif cluster_mgmt -vserver <admin-svm> -destination <mediator-ip>). Most cases are a firewall change or the mediator VM itself being down — fix the VM before touching ONTAP.
Quorum false: the mediator is reachable but lost quorum — usually after a mediator reinstall. Reconfigure: metrocluster configuration-settings mediator remove then ... mediator add with the same SCSI target credentials.
Version mismatch after ONTAP upgrade: upgrade the mediator package to the version listed for your ONTAP release, then re-run metrocluster check run.

3. Unhealthy disks and root-cause on the fabric

MetroCluster needs both pools of every mirrored aggregate healthy. A single failed disk on one site blocks switchover readiness.

storage disk show -broken
storage aggregate show -aggregate <aggr> -fields mirror-status,state
metrocluster check aggregate show

If many disks go broken at once, suspect the fabric, not the disks. On FC MCC check the FC-VI ports and switches (fcp adapter show, switch port err-dump); on IP MCC check theMetroCluster IP interfaces and ISLs:

metrocluster configuration-settings interface show   # IP MCC: both clusters' MCC LIFs
network port show -node <node> -type cluster        # health of underlying ports
metrocluster interconnect show                       # RCFC/RDAX adapters and link state

4. Switchover fails or hangs

You run metrocluster switchover and it refuses, or hangs in pending. Rank these causes:

Unmirrored writes in NVRAM (forced only): a negotiated switchover is refused because the disaster site's NVRAM contains data that never reached the partner. If the site is truly gone, support will walk you through metrocluster switchover -forced-quorum-site <site> — expect a possible data-loss window for anything written in the final seconds.
Mediator/quorum: re-run section 2 checks; switchover is blocked until quorum is clean.
Aggregates not healthy: metrocluster check aggregate show — any degraded mirror blocks the operation.
Operation already in flight: metrocluster operation show — wait for successful; two overlapping operations queue and look hung.

After a successful (or forced) switchover, verify clients are actually served by the surviving site:

metrocluster show
storage aggregate show -fields is-home, home-name, state

5. Switchback fails or leaves aggregates unmirrored

Switchback requires the original site to be fully re-integrated: hardware replaced, aggregates repaired, and plexes resynced first.

metrocluster node show              # both nodes must be "ready" / not waiting for repair
storage aggregate plex show -fields status, resyncing-info
metrocluster repair                 # re-syncs mirror relationships after the failed site returns
metrocluster switchback

"Waiting for repair" forever: the repair step wasn't run or a plex is still resyncing. Check metrocluster node show fields; run metrocluster repair and let every plex finish before retrying switchback.
Switchback completes but data LIFs stayed home: bring LIFs back with network interface revert per LIF, or the clients still route to the surviving site across the inter-site links.
Config replication drift: if SVM configuration differs between sites after repair, run metrocluster check config-replication show and fix the reported SVM/object before assuming the fabric is healthy again.

6. IP MetroCluster specifics

Check the MCC IP interfaces on both clusters and their gateways: metrocluster configuration-settings interface show.
Jumbo end-to-end: MTU must match on MCC LIFs, switches, and ISLs. Mismatch shows as slow resync and interconnect flaps, not outright failure.
metrocluster interconnect show — look for adapters in a non-up state; bounce the link only after checking the switch-side port.

7. When to stop and call support

Any forced switchover or tiebreak decision — data-loss risk, needs support on the call.
Root aggregate or FC-VI adapter failures on a MetroCluster node.
Plex resync that restarts repeatedly from 0% (usually fabric errors — get a switch-side trace).

Collect before opening the case: metrocluster check show, metrocluster operation show, metrocluster node show, and event log show -message-name *mcc* -time <window> output.

8. Prevention checklist

Schedule metrocluster check run weekly; alert on any non-ok subsystem.
Monitor the mediator VM like production infrastructure — it is the single most common MCC outage amplifier.
Test switchover + switchback quarterly in a maintenance window; a DR design you've never failed over is a design you don't have.
Keep the inter-site links on separate physical paths; a shared ISL is a shared failure domain.