Home / Troubleshooting / Aggregate Offline After Power Loss

Aggregate Offline After Power Loss

The data-center power cycle that scares everyone: after a dirty shutdown the node boots, but aggregates stay offline, volumes show odd errors like "UUID is not valid", or disks come up broken. This is the highest-stakes ONTAP failure mode — here is the safe triage path.

NVRAM write acknowledgement and replay path used during power-loss recovery

1. Why power loss breaks aggregates

ONTAP is designed to survive power loss: NVRAM holds the last writes and replays them at boot, and WAFL's write-anywhere design means the filesystem is never left mid-update (see the WAFL and NVRAM deep dives). What actually goes wrong after a DC power cycle is usually one of these:

Shelves powered on before controllers, or not at all — disks are missing or half-visible at boot, so aggregates that need those disks fail to mount.
Both HA partners booted at once with the cluster interconnect down — each node thinks it's alone, storage failover state is tangled, and aggregates may show as offline or "waiting for giveback".
Disks did not spin back up / shelf firmware is stale — disks come up broken or with stale ownership.
NVRAM battery is low or failed — the node may boot but can't guarantee replay of the last writes; see the NVRAM guide.
Ownership/registry confusion — an aggregate whose disks were briefly unowned can look "foreign" or the volumes on it can report UUID is not valid until the aggregate is back online.

2. Phase 1 — Assess, read-only, before touching anything

The first rule: diagnose before you act. Every command below is read-only. Run them all before considering any recovery step.

# Is the cluster itself healthy and quorate?
cluster show
# Node state: up time, health flag, current date/time (is it still booting?)
system node show -fields node,is-node-healthy,uptime,date
# HA state — is the partner alive? Is the node in takeover?
storage failover show
# What do the aggregates say? STATE column is the key field.
storage aggregate show -aggregate aggr1 -instance
# RAID status per aggregate: degraded? reconstructing? resyncing?
storage aggregate show-status -aggregate aggr1
# Disks: broken, missing, or unowned?
storage disk show -broken
storage disk show -fields disk,state,type,owner,home,container-type

Read the STATE column Aggregates you expect to be online can be online, offline, failed, or degraded. "Offline" after a power cycle is usually just waiting for a clean bring-up; "failed" or disks showing broken is a hardware story. Write down exactly what you see — it is the first thing NetApp support will ask for.

3. Phase 2 — Restore a sane HA/cluster state first

If the node is in a half-takeover state or the partner owns the aggregates, fix that before forcing anything:

# Who owns what? If the partner took over, give back gracefully:
storage failover show -fields node,partner,enabled,takeover-enabled,giveback-enabled
storage failover giveback -ofnode node1
# If a node booted into takeover it should not have taken, reverse it:
storage failover takeover -ofnode node1 -bypass-veto-check
# Re-sync the cluster quorum / node membership if a node rejoined:
cluster show
cluster ring show -node node1 -unitname n1  # advanced: ring state should be "quorum"

Do not flip-flop takeover/giveback repeatedly — each transition is a risk window. If giveback vetoes, read the veto reason (storage failover show-giveback on newer releases) instead of forcing it.

4. Phase 3 — Bring aggregates online the correct way

Once the node is stable and the disks it needs are visible, bring the aggregate up with the documented command:

# If the aggregate is offline and its disks are healthy:
storage aggregate online -aggregate aggr1
# Confirm it mounted and the RAID state is normal:
storage aggregate show -aggregate aggr1 -fields state,percent-used,has-local-root
# Then bring its volumes online:
volume online -vserver vs1 -volume vol1
volume show -aggregate aggr1 -fields volume,state,type

If the aggregate comes online but volumes stay offline, check for a pending replay or a volume-level issue (volume show -instance shows the reason in state/state label). Root-cause it per volume — don't bulk-force.

5. "UUID is not valid" and other scary volume errors

A common post-power-cycle symptom (seen in the field, e.g. r/netapp after a DC power cycle): volume show fails with "UUID is not valid", or volumes vanish from listings. Nine times out of ten this is not data loss — it is the volume registry not seeing the aggregate that hosts the volumes. The fix is almost always upstream: get the aggregate online (above), and the volumes reappear. If they don't:

# Does the volume exist in the registry but its aggregate is offline?
volume show -vserver vs1 -fields volume,aggregate,state -orphaned
# Node-shell view of the filesystem (diagnostic — read-only)
set -privilege diag
aggr status -r
vol status -r
set -privilege admin

If the aggregate is online and volumes are still missing — do not delete anything. This is a registry/ownership situation that needs NetApp support eyes; a volume destroy here is permanent.
If the aggregate itself looks foreign or unowned (e.g. after shelves were physically moved), newer ONTAP has storage aggregate reassign to re-home ownership without data loss — verify exact syntax and prerequisites for your release with support before running.
Recovery point options: once volumes are online, snapshot show lists your recovery points; snapshot restore-file can pull individual files back. See copy-on-write.

6. What NOT to run (critical)

Stop. Do not do these.

No fsck / "WAFL recovery" tools — ONTAP has no user-runnable filesystem repair; anything claiming to "fix" the filesystem outside NetApp support guidance can destroy data.
No storage aggregate destroy / volume destroy on anything that looks broken or missing — an offline aggregate is not a dead aggregate.
No disk re-zeroing or replacing healthy disks (disk zero-spares, storage disk replace) while aggregates are offline — you can pull the rug out from under a resync.
No repeated forced takeovers to "wake up" a partner — each forced transition risks the survivor's data if the other node is actually alive but unreachable.
No shelf power-cycling while disks show amber — you can turn a firmware hiccup into a real disk failure. Assess first.

7. When to open a NetApp support case

Any aggregate stays offline after a clean bring-up attempt, or shows failed.
Disks show as broken/amber, or more than one shelf failed to re-appear.
"UUID is not valid" or missing volumes persist with the aggregate online.
NVRAM battery faults are reported (system node hardware nvram show, battery show).
You have any doubt. This failure mode is exactly what a support case is for — the case number also protects you if recovery goes sideways.

NetApp's recovery tooling (e.g. the diagnostic WAFL checker used by support) is only exercised under their guidance. Your job is safe triage, evidence gathering, and not making it worse.

8. Prevention

Power-on order after an outage: shelves first, wait for disks to spin up, then controllers. Power-off order is the reverse. If your PDU can sequence outlets, encode this.
Clean shutdown procedure for planned work: stop clients/applications, then halt nodes cleanly (system node halt -node node1 -reason "planned maintenance") — never pull power on a live controller.
UPS on controllers and shelves, sized long enough for a scripted clean shutdown; monitor the UPS, not just the storage.
Watch NVRAM battery health — system node hardware nvram show and battery show; replace batteries per lifecycle before they fail (see the NVRAM guide).
Keep shelf firmware current — stale shelf firmware is a classic "disks vanished after power cycle" root cause.
Test your DR power-cycle procedure on a non-production node pair before you need it for real.