FlexCache on ONTAP

What FlexCache is — and what it is not

A FlexCache volume presents the same namespace as its origin volume, but blocks clients read are fetched from the origin only once and then served locally until invalidated. It is populate-on-demand, block-level, and coherent by design.

Key properties worth internalizing:

• Cache size ≠ origin size. A cache can be far smaller than the origin; keep the hottest working set resident. Eviction makes room automatically as new data streams in.
• Invalidation is push-based and prompt. When a client modifies data through the origin (or another cache), the origin tells every cache immediately — you are not betting on a TTL.
• Namespaces join seamlessly. Clients mount the FlexCache junction path exactly like the origin; exports, qtrees, permissions come along.
• Origin stays authoritative for writes. Writes arriving at a cache are forwarded/committed at the origin, then retained locally too — remote write latency still includes the WAN hop.

Where FlexCache shines (and where it backfires)

Good fits:

• Branch offices consuming headquarters media/CAD/VFX assets — big sequential re-reads over a lossy WAN disappear after first pass.
• AI training datasets near GPUs — training re-reads the same samples for many epochs; a cache co-located with the GPU cluster turns epoch 2+ into local reads instead of hammering the origin NAS across campus/WAN. (Different job than KV-cache offload to storage — see the news piece on NVIDIA inference spills.)
• HPC / EDA build & scratch trees — many nodes reading identical toolchains and datasets concurrently.
• Offloading fan-out reads — dozens of readers whose aggregate throughput would saturate one origin filer even on LAN.

Bad fits:

• Write-heavy or streaming-once data (CCTV ingest, logging sinks) — poor re-read ratio means every byte crosses the WAN anyway plus overhead.
• Using it as backup or DR — evicted blocks simply do not exist on the edge anymore.
• Random-read-once patterns where hit rates stay near zero — you pay the invalidation/management tax for nothing.
• Files mutated constantly everywhere — churn triggers constant invalidation traffic; consider syncing differently or redesigning workflow locality.

Prerequisites & design decisions

Checklist

• ONTAP version support: next-generation FlexCache appeared in ONTAP 9.5; NFS origins are the classic case, FlexGroup origins arrived later (9.7-era), and same-cluster caching (no peering needed) is supported in modern releases. SMB-origin specifics vary by release — check your release notes before promising Windows client acceleration.
• Origin and edge clusters must be peered when they differ (cluster-level peer relationship; intercluster LIFs on both sides). Intercluster traffic uses TCP 10000/11104/11105 — mirror your firewall plan accordingly (port reference).
• Aggregates on the edge with enough free space; cache volumes are normal thin-provisioned FlexVols underneath.
• Licensing: FlexCache is licensed separately (on-prem) — confirm entitlement before deploying at scale.

Sizing guidance

• Size to the working set, not the dataset. Ask: how much data will be re-read within an hour/day? That window should fit comfortably in the cache. If cache = 10% of origin but holds 80% of reads, that is a win.
• Too small hurts twice: thrash forces re-fetches AND floods the origin that you were trying to protect.
• Spread across aggregates. Let auto-provisioning distribute chunks; on large edges prefer several smaller caches? No — fewer, bigger caches simplify management. Use -auto-provision-cluster-mode true and let ONTAP place chunks across node pools for bandwidth balance.
• Do not enable inline dedupe/compression on the cache itself expecting savings — hot duplicate blocks mostly exist once already; efficiency features behave differently than on write-heavy vols.

Step-by-step deployment runbook

1 — Verify cluster peering (remote-origin case)

# On BOTH clusters: peer must be healthy, IC LIFs up
cluster peer show
network interface show -role intercluster

# Confirm application authorization exists (or add it)
cluster peer show -instance <peer-cluster-name>

If not yet peered, follow the Cluster & SVM Peering guide: generate passphrases, create peer relationships with IC LIFs on both clusters, verify with cluster ping-cluster.

2 — Create the FlexCache volume (edge side)

# Edge cluster: cache of an origin on the peered cluster
volume flexcache create \
  -vserver            svm_edge \
  -volume             fc_media_cache \
  -aggregate-list     aggr1_node1,aggr1_node2,aggr2_node3,aggr2_node4 \
  -size               2TB \
  -origin-vserver     svm_hq \
  -origin-volume      media_library

# Simpler: let ONTAP spread chunks across a cluster-wide pool
volume flexcache create -vserver svm_edge -volume fc_build_cache \
  -auto-provision-cluster-mode true -size 5TB \
  -origin-vserver svm_hq -origin-volume toolchain_nfs

# Same-cluster cache (edge == origin cluster): no peering needed
volume flexcache create -vserver svm_hq -volume fc_local_hot \
  -auto-provision-cluster-mode true -size 1TB \
  -origin-vserver svm_hq -origin-volume project_files

The command returns quickly (control path); initial population happens lazily as clients read. Namespace, exports, and qtrees appear immediately under the same junction structure as the origin.

3 — Mount and test from a client

showmount -e svm_edge.example.com        # junction paths visible?
mkdir -p /mnt/media && mount -t nfs svm_edge.example.com:/media_library /mnt/media

# First read of a large file = miss (crosses WAN); second read = LAN-speed hit
dd if=/mnt/media/dataset/big.bin of=/dev/null bs=1M count=10000   # slow-ish
dd if=/mnt/media/dataset/big.bin of=/dev/null bs=1M count=10000   # much faster

4 — Verify the cache is doing its job

# Instance-level view: state, origin mapping, per-cache details
volume flexcache show
volume flexcache show-instance

# From the ORIGIN side: which caches consume me?
volume flexcache origin show
volume flexcache origin show-cache-stats -vserver svm_hq -volume media_library

# Hit-rate counters: discover available objects, then watch them
statistics catalog counter show | grep -i flexcache
statistics show -object flexcache -interval 5 -iterations 6

# Latency split (local vs origin-served components)
qos statistics volume latency show -vserver svm_edge -volume fc_media_cache

Healthy signatures: flexcache_reads-family counters climbing faster than remote-fetch counters over time; QoS latency histograms for the cache volume staying well below what WAN RTT would inject.

5 — Manage lifecycle

# Find caches nobody has used lately (run on origin, ONTAP 9.10+)
volume flexcache report-idle-caches -idle-days 30

# Grow a busy cache (online)
volume size -vserver svm_edge -volume fc_media_cache -new-size 4TB

# Decommission cleanly: delete from the CACHE side
volume flexcache delete -vserver svm_edge -volume fc_old_cache
volume flexcache origin show     # origin confirms it dropped out

• Deleting a cache never touches origin data — but it destroys every un-evicted edge copy instantly (that is the point).
• Origin-side deletion attempts fail politely while child caches exist; remove caches first.

REST API equivalents (ONTAP 9.10+)

Automation-friendly sites can do this without SSH. Discover and act via the ONTAP REST API from the edge cluster management IP:

# List existing FlexCache volumes (filter by type)
curl -sku admin https://192.0.2.20/api/storage/volumes?type=flexcache\
  -H "Accept: application/hal+json"

# Create (same fields as the CLI; origin referenced across the peer)
curl -sku admin -X POST https://192.0.2.20/api/storage/volumes \
  -H "Content-Type: application/json" \
  -d '{"name":"fc_ci_cache","svm":{"name":"svm_edge"},
       "aggregates":[{"name":"aggr1_node1"},{"name":"aggr2_node3"}],
       "size":2147483648000,
       "flexcache":{"origin":{"volume":{"name":"toolchain_nfs"},
                                 "svm":{"name":"svm_hq"}}}}'

# Job status returned in the POST response (poll with the job URL/job.uuid)

See the ONTAP REST API automation guide for auth patterns, error bodies, and pagination conventions.

Tuning & operational gotchas

• Invalidation is constant, not periodic. If origin churns writes constantly, messaging overhead scales with change rate — another reason write-heavy origins are anti-patterns.
• Read-ahead helps sequential scans. For media/AI workloads (large sequential re-reads), let default read-ahead do its thing; pathological random small-file trees (email stores, compile outputs written fresh each run) gain little.
• Watch WAN loss for initial fills. Big first-pass files suffer TCP window effects on lossy links between origin IC-LIFs and edge; schedule warms during quiet hours with dd/read loops if predictability matters.
• Eviction means unpopularity. Blocks not recently read get reclaimed first; monitor effective hit ratio rather than assuming yesterday's warm set persists forever.
• Efficiency defaults differ. Do not clone your origin's dedupe/compression policies onto the cache verbatim — cross-check what the release supports on FlexCache volumes before assuming parity.
• Performance floor of misses = origin + WAN. A cache cannot beat physics: p99 on cold data still costs origin queueing + RTT.
• Kubernetes/Trident note: NFS backends can point clients (via Trident static PV) at a cache junction just like any export — but understand eviction/invalidation interplay with RWX workloads doing partial rewrites before committing CI pipelines to it (Trident guide).

Troubleshooting matrix

FAQ

Can I cache an SVM-root or DP volume?

No — origins are regular read-write data volumes. SnapMirror destinations make bad origins precisely because their blocks churn via resync.

Does the client need special software?

No. Plain NFS against the edge junction; nothing kernel-side changes vs mounting the origin, because protocol surface is identical.

What happens when the origin goes down?

For stretches, previously cached (not-yet-invalidated) data may continue serving depending on release/flags, but correctness demands treating origin outage as cache-read-only-at-best. Do not architect around surviving origin loss with FlexCache — that is what MetroCluster/SnapMirror Business Continuity are for (MetroCluster guide, SM-AS guide).

How many caches per origin?

Platform/release-dependent limit — check your release's Maximum Limit Numbers. Plan fan-out explicitly rather than discovering the ceiling mid-rollout.

SnapMirror my cache volume elsewhere?

Nonsensical: cache contents are transient by definition. Mirror the origin instead; edge caches rebuild naturally when relaunched against the mirror (tiering reference covers a different cold-data pattern entirely).

Related pages

• FlexGroup deep-dive — scale-out origins and how caches compose with them
• Cluster & SVM Peering — trust fabric under every remote-origin cache
• FabricPool cloud tiering — capacity play vs FlexCache's performance play
• Network port reference — IC-LIF firewalls for cache↔origin traffic
• REST API automation guide — flexcache endpoints + polling pattern