NetApp SolidFire & Element OS: Scale-Out All-Flash Block Storage
SolidFire is NetApp's scale-out all-flash block storage platform, built around per-volume guaranteed QoS and a fully distributed data path. This guide covers the Element OS architecture (nodes, slice services, replicas, fault domains), the QoS model that made SolidFire famous, volumes/accounts/access groups, snapshots and replication, vVols and NetApp HCI, plus a real Element CLI/API operations runbook and a SolidFire-vs-ONTAP decision guide.
Contents
- What SolidFire is
- Scale-out architecture: nodes, slice services, replicas
- Volumes, accounts and access groups
- The QoS model: min, max, burst
- Snapshots, clones and replication
- vSphere, VVols and NetApp HCI
- Management: UI, SSH CLI and the JSON-RPC API
- Day-2 operations runbook
- SolidFire vs ONTAP AFF: which one when
- Lifecycle and further reading
What SolidFire is
NetApp acquired SolidFire in December 2015, and the product line has kept its own operating system (Element OS) and management plane ever since — it was never folded into ONTAP. SolidFire is a block-only, all-flash, scale-out storage platform: every volume is striped across every node in the cluster, all nodes are equal, and each volume carries a hard performance guarantee (minimum IOPS) that the cluster scheduler enforces regardless of what its neighbors are doing.
That guarantee is the product's identity. Where ONTAP QoS caps or floors workloads on a shared AFF pair, SolidFire was designed from day one as multi-tenant storage: service providers and large virtualization estates size the cluster once, then carve volumes with contractual IOPS numbers. The other design pillar is predictable scale-out — adding a node adds capacity and IOPS to every volume in the cluster, because all volumes already span all nodes.
- Element OS — the cluster operating system: distributed data path, scheduler, snapshots, replication, management API.
- Hardware — SolidFire SF-series all-flash nodes (and the storage half of NetApp HCI, which uses the same Element software).
- Protocols — iSCSI and Fibre Channel (Element 11+ added FC support); no NAS, no S3 — block only.
- Integrations — VMware vSphere (including VVols via a VASA provider), OpenStack Cinder, Kubernetes (via the Element CSI driver), and backup products that snapshot or clone through the Element API.
If you administer NetApp estates you will most often meet SolidFire in three places: as dedicated all-flash block for VDI or high-density virtualization, as the storage layer of a NetApp HCI rack, or as the backend behind a managed-service provider's multi-tenant offering. See also the ONTAP vs Pure decision guide and the hardware lineup for where SolidFire sits in NetApp's portfolio.
Scale-out architecture: nodes, slice services, replicas
Element's data path is the opposite of the active/passive HA-pair model in ONTAP HA pairs. A SolidFire cluster is a set of equal nodes (from 4 or so up to triple digits) and every volume lives on all of them:
- Nodes — each node contributes CPU, RAM, NVMe/SAS flash, and network to one pool. There is no "primary" node for a volume; the cluster master (an elected node) handles coordination, not I/O.
- Slice services — each node runs a fixed number (~100) of slice services. A slice service owns a set of 1 MiB slices (the allocation unit) and serves reads/writes for them. Because there are hundreds of slice services spread across all nodes, I/O for any single volume is fanned out across the whole cluster — no hot spots, and node failure only costs you 1/Nth of every volume's throughput.
- Replicas — every slice is replicated (4 copies by default) across nodes; with fault domains configured, copies are forced into separate domains so a whole rack or chassis can fail without data loss. The cluster can also be configured with fewer or more replicas for capacity- or protection-optimized designs.
- Rebalancing — when a node is added or removed, slice services migrate between nodes automatically. Capacity and performance rebalance without a maintenance window, which is the operational payoff of the architecture.
- VIPs — the cluster exposes two virtual IPs on the management network: MVIP (management VIP — Element UI, API, cluster master) and SVIP (storage VIP — the iSCSI/FC target endpoints hosts connect to). Hosts never talk to individual nodes.
Thin provisioning is the only provisioning model: a volume consumes space only as it is written, cluster-wide capacity is shared, and snapshots are space-efficient point-in-time views. This is why capacity math on SolidFire is about the cluster (used vs over-provisioned) rather than per-volume aggregates — see capacity planning for the ONTAP-side comparison.
Volumes, accounts and access groups
Element's access model is deliberately multi-tenant:
- Accounts — a tenant identity with CHAP credentials (initiator secret + target secret). Volumes belong to an account; an account can own many volumes.
- Volumes — thin, block devices sized up to 10 TiB each, created with a QoS profile (below). Volumes can be 512e or 4Kn (the
enable512eflag must match what the host expects). - Volume access groups (VAGs) — the equivalent of an ONTAP igroup: a named list of initiator IQNs/WWNNs plus a list of volumes. An initiator in a VAG sees the VAG's volumes. (An initiator can belong to multiple VAGs; overlapping membership is allowed but is a common source of "why do I see that volume" confusion.)
- CHAP — required for iSCSI (mutual CHAP is supported). Every initiator must present the account's initiator secret; the array presents the target secret.
Access-group hygiene is the day-one discipline: one VAG per host cluster (e.g. one per vSphere cluster), all ESXi hosts' IQNs in it, and only the volumes that cluster should see. See the iSCSI on ONTAP guide for the ONTAP-side equivalent, and SAN multipathing for pathing rules that apply to both.
The QoS model: min, max, burst
Every SolidFire volume carries three QoS numbers, and this triple is the core of the product:
| Setting | Meaning | Planning rule |
|---|---|---|
| minIOPS | Guaranteed floor. The cluster scheduler reserves this across all nodes; under contention, volumes below their min are served before anyone's burst or max. | The sum of all minIOPS must fit inside the cluster's sustainable IOPS envelope. This is the hard capacity number. |
| maxIOPS | Ceiling the volume can sustain indefinitely. | Sum of maxIOPS may oversubscribe the cluster — contention just means not everyone hits max at once. |
| burstIOPS | Short-term ceiling used while burst credits last. | Credits accumulate while the volume runs below its max and are spent during spikes (boot storms, backup windows). |
Two practical consequences. First, the guarantee is enforceable only if you size honestly: if the sum of minIOPS exceeds what the cluster can deliver, the scheduler cannot honor every floor and overall latency rises — Element reports this condition, and the fix is more nodes or lower floors, not tuning. Second, QoS is per-volume, not per-application or per-pool, which is what makes multi-tenant hosting viable: tenant A's boot storm draws down its own burst credits and can never starve tenant B's guaranteed floor.
Element 11.3 and later also support named QoS policies (a reusable min/max/burst triple assigned to many volumes), which keeps service tiers consistent instead of hand-typing numbers per volume. Compare this with ONTAP's QoS policies and adaptive QoS, which are per-workload ceilings/floors on a shared AFF pair — the same idea, a different guarantee model.
Snapshots, clones and replication
- Snapshots — point-in-time, space-efficient views of a volume. Creating a snapshot is instant; space is consumed only as the volume diverges. Common practice is a 5-minute snapshot schedule plus longer-term retention (hourly/daily), exactly the pattern ONTAP admins know from copy-on-write snapshots.
- Group snapshots — a crash-consistent snapshot of a set of volumes at one instant. Use these for multi-volume applications (databases with separate data/log volumes) so a restore brings the set back to a consistent point.
- Clones — writable forks created from a snapshot or volume. Clones share unchanged data with their parent and are the standard mechanism for test/dev refresh, VDI gold-image instant clones, and backup-to-test workflows.
- Replication — asynchronous, per-volume replication between paired clusters (RPO from 5 minutes up to 24 hours, depending on schedule). Cluster pairing uses a one-time pairing key; after pairing, volumes are added to replication pairs and schedules are set per pair. Failover is manual: promote the replica (it becomes writable), and later re-pair/re-sync when the source returns.
Replication and snapshot retention are independent: you can replicate snapshots or only the live volume. For a DR architecture comparison with SnapMirror, MetroCluster and SVM DR, see the DR strategy guide.
vSphere, VVols and NetApp HCI
VVols. SolidFire ships a VASA provider, so vSphere can bind each virtual disk to its own Element volume (storage container = a pool of Element volumes). VVols give per-VM snapshots, clones and QoS through vCenter — the natural fit for the platform, and the reason SolidFire deployments are overwhelmingly VMware shops. Classic (non-VVol) datastores over iSCSI remain fully supported for simpler estates.
NetApp HCI. NetApp HCI combined SolidFire storage nodes with Cisco UCS-based compute nodes in one validated rack: compute and storage scale independently (add compute sleds or storage nodes as needed), and everything is deployed and managed through NetApp Hybrid Cloud Control (the management plane that replaced the original NetApp Deployment Engine), with the Element UI for storage. The compute nodes boot from the SolidFire cluster over iSCSI. If you run across retired HCI gear, see the HCI node repurposing guide — homelabbers have been reflashing those exact nodes for years.
For the ONTAP/Tools side of vSphere management (ONTAP Tools, SnapCenter), see ONTAP Tools for VMware; for the general vSphere-on-NetApp design, see vSphere on NetApp.
Management: UI, SSH CLI and the JSON-RPC API
Element is managed three ways, all backed by the same JSON-RPC API:
- Element Web UI — the management GUI on the mnode (management node) at the MVIP.
- SSH CLI —
ssh admin@<mnode>drops you into a shell where you type API method names directly withkey=valuearguments. - JSON-RPC API —
POST https://<mnode>/json-rpc/2.0with{"method": "...", "params": {...}, "id": n}. Official Python SDK (solidfire-sdk-python) and a PowerShell SDK wrap the same methods.
Basic inventory and capacity:
admin@sf-cluster:~> GetClusterInfo
admin@sf-cluster:~> GetClusterCapacity
admin@sf-cluster:~> ListActiveNodes
admin@sf-cluster:~> ListServices
Tenant and volume lifecycle (SSH CLI syntax):
admin@sf-cluster:~> CreateAccount username=tenant-a \
initiatorSecret=CHAP-secret-1 targetSecret=CHAP-secret-2
admin@sf-cluster:~> CreateVolume name=db01 accountID=7 \
totalSize=10737418240 enable512e=true \
minIOPS=1000 maxIOPS=5000 burstIOPS=10000
admin@sf-cluster:~> ModifyQos volumeID=12 minIOPS=1500 maxIOPS=8000 burstIOPS=16000
admin@sf-cluster:~> ListVolumes
admin@sf-cluster:~> RemoveVolume volumeID=12
The same call over JSON-RPC:
curl -ksu admin:<password> -H 'Content-Type: application/json' \
https://192.0.2.10/json-rpc/2.0 \
-d '{"method":"CreateVolume","params":{
"name":"db01","accountID":7,"totalSize":10737418240,
"enable512e":true,
"qos":{"minIOPS":1000,"maxIOPS":5000,"burstIOPS":10000}},
"id":1}'
Initiator access:
admin@sf-cluster:~> ListInitiators
admin@sf-cluster:~> CreateVolumeAccessGroup name=esxi-cluster-a \
initiators=["iqn.1998-01.com.vmware:esxi01","iqn.1998-01.com.vmware:esxi02"]
admin@sf-cluster:~> AddVolumesToVolumeAccessGroup volumeAccessGroupID=3 volumes=[12,13,14]
Snapshots, clones and replication:
admin@sf-cluster:~> CreateSnapshot volumeID=12 name=db01-hourly
admin@sf-cluster:~> CreateGroupSnapshot volumes=[12,13] name=db01-grp
admin@sf-cluster:~> CloneVolume volumeID=12 name=db01-clone snapshotID=455
admin@sf-cluster:~> ListSnapshots
admin@sf-cluster:~> StartClusterPairing
admin@sf-cluster:~> ListVolumePairings
Performance and session checks:
admin@sf-cluster:~> ListVolumeStats
admin@sf-cluster:~> ListISCSISessions
admin@sf-cluster:~> ListEvents
Method names are stable across recent Element releases, but always confirm against the API reference for your exact Element version before scripting (the API docs ship with the Element UI, and method names are versioned).
Day-2 operations runbook
Health and capacity checks
admin@sf-cluster:~> ListActiveNodes # all nodes Active, no FailedServices
admin@sf-cluster:~> ListEvents # errors/warnings since last review
admin@sf-cluster:~> GetClusterCapacity # used vs overProvisioned vs maxProvisioned
Watch the over-provisioned ratio: thin volumes plus long snapshot retention can push logical usage far past physical flash. Element reports cluster utilization honestly — when used space approaches the physical ceiling, writes slow cluster-wide before they fail, so trend it monthly (see capacity planning).
Node maintenance and replacement
Because every volume spans every node, you do not "move volumes off" a node the way you drain an ONTAP aggregate. Removing a node triggers automatic slice-service migration; Element completes migration before the node is released. The safe sequence is: confirm the cluster has enough spare capacity and replicas, remove the node through the UI/API (which migrates its slice services), verify ListActiveNodes and ListServices are healthy, then power the node down. Replacing a failed node uses the same path — the cluster re-replicates from its remaining copies and rebalances.
Performance triage
ListVolumeStats— find which volumes are hot: sustained IOPS at max, or latency climbing.- Check QoS posture: if many volumes sit at max while others miss their min, the cluster is oversubscribed — raise minIOPS floors only within the cluster's real envelope, or add nodes.
ListISCSISessions— confirm every host has the expected session count and paths (multipathing rules: see SAN multipathing); a missing path shows up as one-sided latency.- Network: jumbo frames on the storage VLAN are the standard expectation; MTU mismatch is the classic "everything is slow on one host" cause.
Common gotchas
- CHAP mismatch — the most common "volume not visible / login failed" cause; the initiator secret must match the account, not the VAG.
- 512e vs 4Kn — created with the wrong
enable512evalue, some OSes refuse to format or perform terribly. - Snapshot sprawl — a 5-minute schedule on hundreds of volumes generates enormous snapshot counts; retention, not schedule, is where space leaks.
- VAG overlap — an initiator in two VAGs sees both volumes sets; "mystery" LUNs are almost always this.
- Min-IOPS debt — adding workloads with guaranteed floors without capacity math is the one mistake that degrades the entire cluster.
SolidFire vs ONTAP AFF: which one when
| Dimension | SolidFire / Element OS | ONTAP AFF / ASA |
|---|---|---|
| Protocols | Block only (iSCSI, FC) | Unified: NFS, SMB, iSCSI, FC, NVMe-oF, S3 object |
| Scale model | Scale-out: add nodes, every volume spans all nodes | Scale-up HA pairs, then clusters; volumes live on aggregates |
| QoS | Per-volume guaranteed minIOPS enforced cluster-wide | QoS policies/adaptive QoS — ceilings and floors on shared pairs |
| Snapshots | Volume + group snapshots, clones | Snapshots, FlexClone, SnapMirror/SnapVault ecosystem |
| Ecosystem | Element API/SDKs, VVols, CSI, OpenStack Cinder | SnapCenter, SnapMirror, FabricPool, BlueXP, ONTAP tools |
| Best fit | Multi-tenant hosting, VDI at scale, guaranteed-IOPS SLAs | Mixed file+block estates, NAS, DR fabrics, cyber-resilience |
In a NetApp-heavy estate the two are complements, not rivals: ASA gives you the ONTAP feature set in a block-focused form, SolidFire gives you enforceable per-tenant IOPS. If the requirement is "a number we can put in a contract," SolidFire's model is the stronger answer; if the requirement is "one platform for NFS, SMB, FC and S3," ONTAP wins. The on-prem vs cloud guide covers the wider platform decision.
Lifecycle and further reading
NetApp has announced lifecycle transitions for NetApp HCI hardware (end-of-sale/end-of-support milestones; exact dates change, so confirm on NetApp's official lifecycle pages before planning around it). SolidFire SF-series systems and Element OS remain supported NetApp products, with Element releases continuing to ship. As with any NetApp platform, validate exact firmware/Element/plugin versions in the NetApp Interoperability Matrix before buying or upgrading.
Related guides: iSCSI on ONTAP · SAN multipathing · ONTAP QoS deep dive · vSphere on NetApp · ONTAP vs Pure · Repurposing NetApp HCI nodes.