NVMe over Fabrics on ONTAP

1. Why NVMe-oF

SCSI transports were designed around comparatively shallow device queues and a command translation layer. NVMe exposes many submission/completion queue pairs, allowing CPU cores to drive I/O with less shared locking. That does not make media or the network disappear: at modest queue depth, controller ownership, fabric congestion, host scheduling, and application sync writes often dominate. Choose NVMe-oF for a measured latency or concurrency requirement—not because its protocol name is newer.

2. ONTAP support reality

• NVMe/FC: available from ONTAP 9.4, using supported FC-NVMe initiators, HBAs, switches, firmware, and host OS combinations.
• NVMe/TCP: available from ONTAP 9.10.1. Initial qualification centered on AFF; current eligibility varies across AFF, ASA, FAS, ONTAP release, adapter, and host stack. Validate the exact design in NetApp IMT.
• NVMe/RDMA: ONTAP does not expose NVMe/RDMA targets. Do not confuse an RDMA-capable Ethernet adapter with a supported ONTAP target transport.
• Mixed protocol: newer ONTAP releases can serve NVMe alongside other protocols on an SVM; early NVMe releases imposed tighter SVM rules. Upgrade guidance and the IMT are authoritative.

3. Namespaces, subsystems, LIFs, and ANA

A namespace is the block device carved inside a volume or qtree. A subsystem is the access-control and discovery boundary: it owns an ONTAP-generated NQN, contains permitted host NQNs, and has mapped namespaces. A namespace maps to one subsystem, while a subsystem can expose multiple namespaces to multiple authorized hosts.

Data LIFs provide target paths. For NVMe/FC, create FC-NVMe LIFs on physical FC target ports and zone each host initiator to target ports across independent fabrics. For NVMe/TCP, create NVMe/TCP data LIFs on dedicated IP networks, normally at least one path per HA node and per fault domain. ONTAP reports path preference with ANA: the host uses optimized paths and retains non-optimized paths for takeover or topology changes.

4. ONTAP CLI provisioning runbook

# Confirm NVMe is allowed, then create/verify the SVM NVMe service
vserver show -vserver svm_nvme -fields allowed-protocols
vserver nvme create -vserver svm_nvme
vserver nvme show -vserver svm_nvme

# Create the namespace at the root of an existing volume
vserver nvme namespace create -vserver svm_nvme \
  -path /vol/vol_nvme_db/ns1 -size 2TB -ostype linux

# Create the host access boundary and authorize its NQN
vserver nvme subsystem create -vserver svm_nvme \
  -subsystem db_hosts -ostype linux
vserver nvme subsystem host add -vserver svm_nvme \
  -subsystem db_hosts -host-nqn nqn.2014-08.org.nvmexpress:uuid:HOST-UUID

# Map the namespace (a namespace can map to one subsystem)
vserver nvme subsystem map add -vserver svm_nvme \
  -subsystem db_hosts -path /vol/vol_nvme_db/ns1

# Parameter names can vary by ONTAP release; verify with tab completion
network interface create -vserver svm_nvme -lif nvme_tcp_a1 \
  -service-policy default-data-nvme-tcp -home-node cluster1-01 \
  -home-port e1a -address 10.40.1.21 -netmask 255.255.255.0
network interface create -vserver svm_nvme -lif nvme_tcp_b1 \
  -service-policy default-data-nvme-tcp -home-node cluster1-02 \
  -home-port e1a -address 10.40.2.22 -netmask 255.255.255.0
network interface show -vserver svm_nvme -data-protocol nvme-tcp
vserver nvme namespace show -vserver svm_nvme -instance
vserver nvme subsystem show -vserver svm_nvme

5. Linux discovery and native multipathing

# Read the host identity and enable native NVMe multipathing
cat /etc/nvme/hostnqn
cat /sys/module/nvme_core/parameters/multipath

# Discover through each fault domain, then connect all advertised paths
nvme discover -t tcp -a 10.40.1.21 -s 4420
nvme connect-all -t tcp -a 10.40.1.21 -s 4420
nvme connect-all -t tcp -a 10.40.2.22 -s 4420

# Validate controllers, namespaces, ANA state, and path count
nvme list
nvme list-subsys
nvme id-ns /dev/nvme0n1
nvme smart-log /dev/nvme0n1

Use the distribution’s supported persistent discovery/autoconnect mechanism rather than a boot-time shell loop. Native NVMe multipath must combine controllers for one namespace; if separate devices appear, fix host multipath configuration before creating a filesystem or volume manager.

6. Performance tuning without folklore

As planning—not a guarantee—well-tuned all-flash NVMe/FC commonly operates in the low hundreds of microseconds for small reads, while NVMe/TCP may add tens to a few hundred microseconds depending on NICs, switches, CPU load, distance, and queueing. End-to-end application latency can be much higher. Establish a baseline at queue depth 1 and under the intended concurrency, then compare FCP/iSCSI and NVMe-oF on identical media and QoS.

7. Troubleshooting matrix

vserver nvme show -vserver svm_nvme
vserver nvme namespace show -vserver svm_nvme -instance
vserver nvme subsystem show -vserver svm_nvme -instance
vserver nvme subsystem host show -vserver svm_nvme
network interface show -vserver svm_nvme -data-protocol nvme-tcp,fc-nvme
event log show -time >1h -severity ERROR