ONTAP QoS policy groups: ceilings, floors, and adaptive policy groups

Why QoS exists in ONTAP

A single runaway backup server scanning millions of files can starve an OLTP database sharing the same controller — even though both sit on healthy SSD aggregates. ONTAP solved this class of "noisy neighbor" problem first through System Manager controls and later (ONTAP 9.3 onward) fully at the cluster level with qos commands. Three distinct tools live under one framework:

• Fixed policy groups — a hard upper bound (-max-throughput) on IOPS, throughput (MB/s or GB/s), or both.
• Adaptive policy groups (ONTAP 9.5+) — capacity-proportional limits expressed as IOPS per terabyte, evaluated against consumed space rather than a static number you re-adjust after every resize.
• Floors — a guaranteed minimum throughput share for workloads whose latency must be defended against other tenants on shared media.

All three attach to the same primitive: the workload. Everything else is bookkeeping.

Workloads and policy groups: the model

• A workload is the QoS-visible unit of storage: typically a FlexVol volume, FlexGroup constituents, NVMe namespace, or LUN. QoS measures its IO where it lands on the node.
• A policy group is a named object holding the limits. By default one policy group is enforced individually per attached workload; sharing one group across many volumes then acts as separate budgets, unless you create it as a shared budget (see below) so the limit covers all members combined.
• Scope: classic groups are created against a single SVM; ONTAP 9.7+ supports -cluster-wide true objects that any SVM can reference — handy for uniform desktop or branch-office profiles.
• Default groups: untagged workloads land in the implicit default policies, which set only a very high system ceiling — not an admission ticket, just a safety catch.
• Background housekeeping (SnapMirror transfers, dedupe/scanner activity, tape NDMP) also appears as workloads such as those tagged Workload Type: background; ceilings can be applied to protect foreground traffic during replication windows.

QoS is enforced in software on each node's scheduler ahead of WAFL, so it costs negligible CPU, needs no special hardware, and works uniformly on AFF, ASA, FAS with SSD, and hybrid systems. What it cannot do is make slow media faster — a floor reserves relative priority, not raw speed beyond what the hardware delivers.

Fixed policy groups (max-throughput ceilings)

The bread-and-butter control. Limit by IOPS, bytes per second, or both together; when both are set, whichever threshold hits first throttles.

# Create a ceiling per-SVM: 5,000 IOPS AND 600 MB/s max
cluster::> qos policy-group create -policy-group pg_oltp_cap \
     -vserver svm_sales -max-throughput 5000iops,600MB/s

# Cluster-wide profile usable by every SVM (9.7+)
cluster::> qos policy-group create -policy-group pg_branch \
     -cluster-wide true -max-throughput 1500iops

Unit grammar is consistent everywhere on the platform: iops, b/s, kb/s, mb/s, gb/s. Prefixes behave as k=1000. Inspect what you have, including which workloads currently consume a group:

qos policy-group show
qos policy-group show -policy-group pg_oltp_cap -instance
qos workload show              # every workload + its effective group/limits

Adaptive QoS (IOPS per TB)

Static ceilings age badly: a volume doubles, and yesterday's sensible limit becomes today's outage. Adaptive policy groups (introduced in ONTAP 9.5) express intensity instead — a linear ramp from expected to peak IOPS allocated per terabyte of consumed blocks.

# OLTP profile: expect 2,000 IOPS/TB, allow bursting to 4,000 IOPS/TB,
# never below 750 IOPS no matter how small the volume gets
cluster::> qos adaptive-policy-group create -policy-group apg_oltp \
     -vserver svm_sales -expected-iops 2000/TB -peak-iops 4000/TB \
     -absolute-minimum-iops 750 -block-size 32TB

# Attach while creating/updating a volume
cluster::> vol modify -vserver svm_sales -volume db01 \
     -qos-adaptive-policy-group apg_oltp

• -block-size quantises the math: limits re-evaluate in block-size steps, so with 32TB blocks every 32TB consumed buys another chunk of expected IOPS.
• Used-space gotcha: evaluation follows consumed capacity. Thin-provisioned volumes freshly migrated but mostly free start under-limited until they fill — deliberately pick a smaller block size (or fall back to a fixed group) if you need day-one guarantees.
• -absolute-minimum-iops is the floor beneath the ramp; ONTAP will not cut a workload below it even at near-zero utilization.
• Adaptive groups cannot be edited after creation in early releases — treat them as versioned (create apg_oltp_v2, migrate, delete the old one).

Min-throughput floors

Recent ONTAP releases (9.13–9.16 wave, AFF/ASA platforms — verify availability on your specific version before designing around it) flipped QoS from defensive to contractual: -min-throughput reserves capacity inside the scheduler for designated workloads.

# Guarantee the EHR database at least 12000iops even during restore storms
cluster::> qos policy-group create -policy-group pg_ehr_floor \
     -vserver svm_clinical -min-throughput 12000iops

# Ceiling + floor combined in one contract
cluster::> qos policy-group modify -policy-group pg_ehr_floor \
     -max-throughput 25000iops

Floors only make sense when total reserved bandwidth stays under what the nodes can actually deliver; oversubscribed floors degrade to "everyone fights politely," so sum your reservations and compare them with measured peak service rates before going live. On systems without floor support the flag is rejected at create time — which is your quickest compatibility probe.

Assignment runbook (volumes, LUNs, namespaces)

# Attach / detach / switch groups
vol modify -vserver svm1 -volume vol_homedirs -qos-policy-group pg_files
vol modify -vserver svm1 -volume vol_homedirs -qos-policy-group ""    # detach
vol modify -vserver svm1 -lun lun_oracle -qos-policy-group pg_oltp_cap

# Review mapping and remove an empty group
qos workload show -workload-volume vol_homedirs -instance
qos policy-group delete -policy-group pg_old -vserver svm1

• Ducking out of a group requires setting the attribute to empty string — you cannot simply omit it on modify.
• Group deletion fails while any workload still references it; the error names the offender, which doubles as a handy audit command.
• Moving volumes between groups is non-disruptive; enforcement changes take effect immediately, though per-workload statistics history resets under the new identity.
• FlexGroup constituents inherit assignments individually (each constituent counts as its own workload) — ceilings therefore multiply across constituents; prefer cluster-wide defaults or generous totals there.

Monitoring & counters

# Live IOPS/throughput for assigned workloads
qos statistics volume performance show -sort-by iops
# Latency attribution incl. the QoS wait bucket
qos statistics volume latency show
# Node-level view: system workloads, adaptive utilization, ceilings reached
qos statistics workload resource utilization show

When a ceiling binds, the throttle component lights up in the latency breakdown while disk time stays comfortable — that split is the signature difference between "policy cap" and "media congestion." Correlate with host-side observations before concluding; see our performance troubleshooting QoS section for the step-by-step drill-down and the ONTAP-wide latency anatomy.

Troubleshooting matrix

Production checklist

• Name groups by contract, not hardware: pg_silver_4kiops beats pg_test2 in audits.
• Set ceilings only where contention is real — needless caps add latency without saving capacity.
• Prefer the pair IOPS + MB/s over either alone; they fail differently and complement each other.
• For scaling estates, default new volumes onto adaptive groups and let exceptions get fixed ones.
• Re-validate assignments after storage migrations/reorgs — orphaned stale caps are the top recurring finding.
• Chart qos statistics volume latency show monthly; watch the throttle bucket drift.