ONTAP Troubleshooting Playbook

Five-minute baseline

Run this read-only sweep before following a symptom. It separates cluster health, hardware, capacity, networking, and EMS evidence without changing state.

cluster::> system health alert show
cluster::> system node show -fields health,eligibility,uptime
cluster::> storage failover show
cluster::> storage aggregate show -fields state,size,available,percent-used
cluster::> volume show -fields state,size,available,percent-used
cluster::> network interface show -fields status-admin,status-oper,home-node,curr-node,home-port,curr-port
cluster::> event log show -time >1h
cluster::> node run -node <node> sysconfig -a

sysconfig -a is a nodeshell command; node run invokes it from the clustershell. Treat its hardware inventory as supporting evidence, then use clustershell health and storage commands for current state.

01 · Failed disk or degraded RAID

Symptoms

EMS reports a failed disk, an aggregate shows degraded, reconstruction begins, or a spare is consumed.

Diagnose and isolate

storage disk show -broken
storage disk show -disk <disk> -instance
storage aggregate show-status -aggregate <aggregate>
event log show -time >2h -message-name raid*
node run -node <node> sysconfig -a

1. Confirm the exact disk, owner, shelf/bay, container type, and affected RAID group.
2. Check whether a spare is reconstructing; distinguish one failed drive from several drives disappearing together.
3. If multiple adjacent disks vanished, investigate shelf power/pathing before replacing drives.

Fix

Follow the platform FRU procedure and replace the identified drive; ONTAP normally reconstructs to a suitable spare automatically. Monitor storage aggregate show-status. Do not use storage disk fail merely to silence an alert, and do not remove a reconstructing disk. Escalate if redundancy is exhausted, no compatible spare exists, or identity/path evidence conflicts.

02 · Disk path or shelf fault

Symptoms

Several disks report the same path missing, multipath HA is degraded, a shelf/IOM alerts, or disks flap without media-failure evidence.

Diagnose and isolate

system health alert show
storage shelf show -errors
storage disk show -fields disk,owner,path-lun,physical-size,container-type
event log show -time >2h -severity *
node run -node <node> sysconfig -a

1. Group affected disks by shelf, stack, IOM, and controller path.
2. Check both nodes: a common shelf/path pattern points to cabling, IOM, power, or adapter—not simultaneous media failures.
3. Compare the current topology with the validated cabling plan.

Fix

Use the hardware guide for the exact shelf and platform. Repair one verified path/component at a time during an approved window. Never reseat shared SAS/NVMe shelf components from a generic runbook; collect AutoSupport and involve support when both paths, multiple shelves, or aggregate availability are at risk.

03 · Takeover will not start

Symptoms

Takeover Possible is false, the precheck fails, or a planned takeover aborts.

Diagnose and isolate

storage failover show -instance
storage failover check-takeover -node <node-being-taken-over>
system node show -fields health,eligibility
system health alert show
event log show -time >1h

1. Read the state description: partner state, HA interconnect, NVRAM synchronization, version mismatch, and storage health require different responses.
2. Verify the survivor has performance headroom and client paths can fail over.
3. Correct the named prerequisite, then rerun the check; do not substitute a force option for diagnosis.

Fix

# Only after all prechecks pass
storage failover takeover -ofnode <node>

Stop if logs are unsynchronized, shared storage is unstable, the partner identity is uncertain, or ONTAP warns of data risk. See the HA pair guide; force/immediate options belong in a case-specific NetApp procedure.

04 · Giveback stuck or vetoed

Symptoms

The repaired partner waits for giveback, an aggregate remains on the survivor, or show-giveback names a veto.

Diagnose and isolate

storage failover show
storage failover show-giveback
storage aggregate show -fields home-name,owner-name,state
event log show -time >1h
system health alert show

1. Identify the aggregate or subsystem and retain the full veto text.
2. Verify the destination node is stable and its disks, network ports, and encryption keys are available.
3. Resolve the named condition—such as an operation in progress or failed hardware—then retry normally.

Fix

storage failover giveback -ofnode <node>
storage failover show-giveback

Do not use -override-vetoes true unless the exact veto is documented as safe for the current release and the risk is accepted. Never override missing keys, unknown disks, or unstable hardware.

05 · NFS latency

Symptoms

Mounts remain online but application I/O stalls, client nfsstat shows retransmits, or latency is isolated to one SVM, volume, node, or path.

Diagnose and isolate

statistics catalog instance show -object volume
statistics show -object volume -instance <instance-from-catalog>
qos statistics workload performance show
network interface show -vserver <svm> -fields curr-node,curr-port,status-oper
network port show -fields link,health-status,mtu
event log show -time >30m

1. Compare client latency and retransmits with ONTAP volume/workload latency. Low array latency plus retransmits points toward the network/client path.
2. Check whether latency follows a volume, QoS policy, node, LIF/port, or time-bound job.
3. Separate protocol operations from backend storage and CPU saturation; collect a time-aligned sample.

Fix

Correct the proven bottleneck: failed links/VLAN/MTU, an unintended QoS ceiling, a hot volume, or a competing job. Move a NAS LIF only when its failover target and client routing are valid: network interface migrate -vserver <svm> -lif <lif> -destination-node <node> -destination-port <port>. Do not apply guessed mount tuning cluster-wide.

06 · SMB latency

Symptoms

Slow opens/listing, application pauses, or only SMB clients are affected while storage remains available.

Diagnose and isolate

vserver cifs session show -vserver <svm>
vserver cifs connection show -vserver <svm>
vserver cifs domain discovered-servers show -vserver <svm>
qos statistics workload performance show
statistics show -object volume -instance <instance-from-catalog>
event log show -time >30m

1. Compare ONTAP workload latency with client-observed delay.
2. Check whether delay occurs before authentication, during directory/metadata operations, or during data I/O.
3. Validate DNS, domain-controller reachability, time service, sessions, LIF placement, and network errors.

Fix

Repair the measured dependency: DNS/DC reachability, network path, QoS constraint, or overloaded node/volume. Close sessions or restart the CIFS service only in a planned window; those actions disconnect users. Preserve packet traces and counters if the fault is intermittent.

07 · Volume or Snapshot space exhausted

Symptoms

Clients receive ENOSPC, a volume is nearly full, Snapshot reserve is consumed, or automatic growth stopped.

Diagnose and isolate

volume show -vserver <svm> -volume <volume> -fields size,available,percent-used,state
volume show-space -vserver <svm> -volume <volume>
volume autosize -vserver <svm> -volume <volume>
volume snapshot show -vserver <svm> -volume <volume>
volume snapshot autodelete show -vserver <svm> -volume <volume>
event log show -time >2h

1. Separate active filesystem, Snapshot, reserve, and LUN overwrite consumption.
2. Check aggregate free space and the volume’s autosize maximum before growing.
3. Identify Snapshot owners and replication dependencies before deletion.

Fix

# Example only after capacity and policy review
volume size -vserver <svm> -volume <volume> -new-size <size>

# Delete only an explicitly approved, unneeded Snapshot
volume snapshot delete -vserver <svm> -volume <volume> -snapshot <snapshot>

Do not delete common SnapMirror, backup-owned, or application-consistent snapshots blindly. Configure autosize/autodelete only against an agreed retention and capacity policy.

08 · Aggregate nearly full

Symptoms

Aggregate utilization rises toward the operational threshold, WAFL space events appear, or volumes cannot grow despite thin provisioning.

Diagnose and isolate

storage aggregate show -aggregate <aggregate> -fields size,available,percent-used,state
volume show -aggregate <aggregate> -fields size,available,percent-used,space-guarantee
volume show-footprint -aggregate <aggregate>
storage aggregate show-status -aggregate <aggregate>
event log show -time >4h

1. Rank volume footprints and identify recent growth, guarantees, Snapshot/clone use, and pending operations.
2. Confirm physical capacity, spare policy, RAID state, and FabricPool status where applicable.
3. Forecast time-to-full; do not rely on efficiency savings that have not completed.

Fix

Use a planned combination of deleting verified disposable data/snapshots, moving volumes, adding supported capacity, or adjusting guarantees after risk review. volume move start -vserver <svm> -volume <volume> -destination-aggregate <aggregate> is nondisruptive in supported cases but still needs destination headroom and monitoring.

09 · Node root volume full

Symptoms

EMS reports root-volume space pressure, management operations fail, logs cannot be written, or the node becomes unstable.

Diagnose and isolate

volume show -type RW -is-node-root true -fields vserver,volume,aggregate,size,available,percent-used
volume show-space -vserver <node-svm> -volume <root-volume>
event log show -node <node> -time >4h
system node autosupport history show -node <node>

1. Confirm it is the node root volume—not an SVM root or data volume.
2. Correlate growth with logs, cores, upgrade artifacts, or support collections.
3. Preserve evidence and open a support case when the consumer is not obvious.

Fix

Do not browse into nodeshell and manually remove system files. Follow the release-specific NetApp procedure for the identified consumer; support may direct cleanup or root-volume growth. Generic file deletion can make the node unbootable.

10 · Dedupe or compression savings low

Symptoms

Reported savings fall below expectation, an efficiency job does not run, or post-migration data consumes more physical space.

Diagnose and isolate

volume efficiency show -vserver <svm> -volume <volume> -instance
volume efficiency show -vserver <svm> -volume <volume> -fields state,status,progress,policy
volume show-footprint -vserver <svm> -volume <volume>
event log show -time >4h

1. Verify efficiency is enabled, the last job status, schedule/policy, and whether the workload is eligible.
2. Set expectations by data type: encrypted or already-compressed data and unique blocks deduplicate poorly.
3. Check whether inline versus background features, minimum-change thresholds, capacity pressure, or an interrupted job explains the result.

Fix

volume efficiency on -vserver <svm> -volume <volume>
volume efficiency start -vserver <svm> -volume <volume>

Start a scan only with workload headroom. Options for scanning existing data and feature support vary by ONTAP release, platform, volume type, and licensing; check volume efficiency start ? rather than copying a guessed option.

11 · LIF down or on the wrong port

Symptoms

A data or management IP stops responding, operational status is down, or a LIF remains away from home after failover.

Diagnose and isolate

network interface show -vserver <svm> -lif <lif> -instance
network port show -node <node> -port <port>
network port broadcast-domain show
network interface failover-groups show
event log show -time >1h

1. Check admin/oper status, current versus home node/port, and the hosting port’s link/health.
2. Validate broadcast domain, VLAN, failover group, IPspace, and upstream switch state.
3. For SAN LIFs, remember migration/failover behavior differs from NAS LIFs; follow the protocol guide.

Fix

# Revert only when the home port is healthy and reachable
network interface revert -vserver <svm> -lif <lif>

# Administrative enablement does not repair the underlying port
network interface modify -vserver <svm> -lif <lif> -status-admin up

Do not move a LIF across an unvalidated VLAN or IPspace. Correct physical/switch configuration first if the port is down.

12 · Routing or reachability failure

Symptoms

LIFs are up but clients, DNS, domain controllers, peers, or intercluster endpoints are unreachable.

Diagnose and isolate

network route show -vserver <svm>
network interface show -vserver <svm> -fields address,netmask,curr-node,curr-port,ipspace
network ping -vserver <svm> -destination <ip-address>
network traceroute -vserver <svm> -destination <ip-address>
event log show -time >1h

1. Test the next hop, then the destination, from the correct SVM/IPspace.
2. Check source subnet, netmask, longest-prefix route selection, return route, VLAN, and firewall.
3. If only one node/path fails, compare LIF placement and node-local uplinks.

Fix

# Add only a route approved for this SVM/IPspace
network route create -vserver <svm> -destination <network/prefix> -gateway <gateway-ip>

Before modifying or deleting a route, record current output and confirm the return path. A default route can mask a missing specific route; duplicate or wrong gateways can create asymmetric reachability.

13 · SVM or protocol service down

Symptoms

An SVM is stopped, its data LIFs are unavailable, or NFS/SMB/iSCSI service is not configured or operational.

Diagnose and isolate

vserver show -vserver <svm> -instance
network interface show -vserver <svm>
vserver nfs show -vserver <svm>
vserver cifs show -vserver <svm>
vserver iscsi show -vserver <svm>
event log show -time >1h

1. Determine whether the SVM administrative state, protocol service, LIFs, or only a dependency is down.
2. Check recent administrative changes, licensing/entitlement where applicable, name services, and cluster health.
3. Do not create a duplicate protocol server to repair a stopped one.

Fix

# Start only after confirming the stop was not intentional
vserver start -vserver <svm>

Protocol enable/start syntax and prerequisites differ. Inspect the relevant vserver nfs|cifs|iscsi command help and use the existing configuration. Starting an SVM does not fix broken DNS, AD, export policy, mapping, or LIF routing.

14 · NFS mount fails or access is denied

Symptoms

Mount timeout, “permission denied,” “access denied by server,” or a path is not found.

Diagnose and isolate

vserver nfs show -vserver <svm>
volume show -vserver <svm> -fields volume,state,junction-path
vserver export-policy rule show -vserver <svm> -policyname <policy>
vserver export-policy check-access -vserver <svm> -client-ip <client-ip> -volume <volume> -authentication-method <sys-or-krb> -protocol <nfs3-or-nfs4> -access-type read-write
network interface show -vserver <svm>
event log show -time >30m

1. Prove network reachability to the data LIF and that NFS is enabled.
2. Resolve the exact junction path and effective export policy for the client IP, protocol, auth flavor, and access type.
3. If export access passes, inspect file security style, identity mapping, UNIX permissions, or NFSv4/Kerberos dependencies.

Fix

Correct the smallest mismatched rule or identity dependency. Export-rule creation has many security-sensitive fields; use vserver export-policy rule create ? for the installed release and avoid broad 0.0.0.0/0, anonymous root, or unrestricted superuser access as a shortcut.

15 · SMB authentication or access denied

Symptoms

Logon failure, account lockout, share unavailable, or authentication succeeds but file access is denied.

Diagnose and isolate

vserver cifs show -vserver <svm>
vserver cifs domain discovered-servers show -vserver <svm>
vserver cifs session show -vserver <svm>
vserver cifs share show -vserver <svm> -share-name <share>
vserver cifs share access-control show -vserver <svm> -share <share>
vserver services name-service dns check -vserver <svm>
event log show -time >30m

1. Separate connectivity, authentication, share ACL, and file/NTFS ACL failures.
2. Validate DNS, NTP, domain-controller discovery, machine account health, and the user’s AD status.
3. For access denial, evaluate both share ACL and file ACL, including groups, denies, and inheritance.

Fix

Repair the proven dependency or intended ACL. Do not delete/recreate the CIFS server or computer account as a first response; that can disrupt SPNs and all clients. ACL modifications require an approved identity and permission design.

16 · SnapMirror lag or transfer failure

Symptoms

Relationship health is false, lag grows, scheduled updates fail, or a transfer remains queued/transferring.

Diagnose and isolate

snapmirror show -destination-path <destination> -instance
snapmirror show -fields state,status,healthy,unhealthy-reason,lag-time,last-transfer-error
cluster peer show
vserver peer show
network interface show -role intercluster
network route show
event log show -time >2h

1. Read the last transfer error and separate relationship state from transport, peer, capacity, or common-Snapshot problems.
2. Validate intercluster LIFs and routing on both clusters, then destination capacity and relationship policy/schedule.
3. Confirm no operator removed a required common Snapshot or changed the destination.

Fix

# Retry only after correcting the reported cause
snapmirror update -destination-path <destination>

Do not resync, break, release, or reinitialize casually: those commands can discard newer destination data, change protection state, or require a new baseline. Follow the SnapMirror runbook for the relationship’s exact state.

17 · ONTAP upgrade validation or execution failure

Symptoms

Prechecks block the upgrade, a node pauses during automated update, takeover/giveback fails, or nodes report different versions after interruption.

Diagnose and isolate

cluster image show
cluster image show-update-progress
cluster image validate -version <target-version>
storage failover show
system health alert show
event log show -time >4h

1. Capture the exact validation warning or failed phase; do not restart the workflow repeatedly.
2. Resolve health, HA, disk, firmware, compatibility, image, or application checks named by the validator.
3. Confirm the supported upgrade path, platform support, and host/application interoperability for both current and target releases.

Fix

Correct the blocking condition and rerun validation. Resume or rollback syntax depends on the update state and release; use cluster image update ? plus the exact NetApp upgrade procedure. Do not override warnings or manually force takeover/giveback during the orchestrated upgrade unless NetApp’s procedure explicitly directs it. See the upgrade runbook.

18 · Node CPU saturation or broad latency

Symptoms

Multiple workloads on one node slow simultaneously, CPU stays high, or latency follows LIF/aggregate ownership rather than one protocol.

Diagnose and isolate

statistics show -object processor
qos statistics workload performance show
system node show -fields uptime,health
network interface show -fields curr-node,home-node
volume show -fields node,aggregate
event log show -time >1h

1. Correlate CPU, workload IOPS/throughput/latency, and the incident window.
2. Identify whether load comes from client I/O, efficiency, replication, backup/scanning, or takeover operation.
3. Check whether one node owns a disproportionate set of active LIFs or aggregates and whether the partner is healthy.

Fix

Pause or reschedule the proven noncritical job, correct an unintended QoS/workload placement issue, or rebalance through a planned volume/LIF move. Do not treat a takeover as load balancing. If CPU remains unexplained, collect a time-bounded performance archive/AutoSupport and escalate rather than changing hidden counters.

Escalation bundle

Collect evidence before it rolls out of logs. Avoid including credentials, client data, or secrets in tickets.

system health alert show
system node show -fields health,eligibility,uptime
storage failover show -instance
storage aggregate show-status
volume show -fields state,size,available,percent-used
network interface show -fields status-oper,home-node,curr-node,home-port,curr-port
network route show
event log show -time >4h
system node autosupport invoke -node * -type all -message "CASE-<id>-INCIDENT"

AutoSupport invocation behavior and transport must already be approved and configured. Record cluster serials, ONTAP version, timezone, incident start/end, affected clients, recent changes, and every command run.