VCF 9.1 Memory Tiering: Before You Cut the DRAM

Broadcom’s VCF 9.1 Memory Tiering post, published in May 2026, says enterprise DDR5 RDIMM costs have surged since 2023 and that a high-density virtualization node now costs more than twice as much as it did. I already went through the 2026 memory and flash price numbers in my post vSphere Standard Is Coming Back, and Proxmox Just Grew Up a Little, including TrendForce’s DRAM contract prices rising 90 to 95 percent in Q1 and NAND overtaking DRAM in Q2.

With memory prices where they are, this is a good time to explain the VCF feature built to reduce that cost: NVMe Memory Tiering, which uses local NVMe drives as a second memory tier behind DRAM. Customers have noticed. At VMware Explore 2026, Duncan Epping said on the Virtually Speaking podcast that more people were attending Memory Tiering sessions than vSAN sessions, and he put the interest down to the price of memory. My earlier post also showed the catch. The Mixed Use NVMe drives tiering needs got more expensive too, so you have to check any savings against current quotes.

VCF 9.1 made the feature easier to deploy and easier to protect. Software mirroring removed the need for a hardware RAID controller to protect the tier, configuration moved into vSphere Configuration Profiles, and hosts no longer reboot to enable it. Broadcom markets up to 40% lower TCO.

Companies can save money with it, but only in clusters where the numbers hold. The question I care about is when Memory Tiering justifies changing a production cluster design, meaning fewer hosts, less DRAM per host, or a different refresh plan.

I work for Leaseweb as a Senior System Engineer on the VMware Cloud team. The opinions here are mine and do not represent or bind Leaseweb. The three designs below are architecture exercises on paper. Every figure comes either from published Broadcom and VMware material or from illustrative inputs that I label as such, and nothing here is a benchmark or a measured result.

What changed in VCF 9.1 Memory Tiering

Memory Tiering shipped in VCF 9.0. The 9.1 changes that matter for design work:

• In 9.0, tier redundancy required a hardware RAID controller, either Tri-Mode or Intel VROC. 9.1 mirrors the tier in software across two equivalent NVMe devices per host, with no controller.
• Setup moved into a single vSphere Configuration Profiles draft; NVMe partitions are created automatically, and enabling the feature needs maintenance mode but no host reboot.
• Host-level management moved to a new esxcli memtier namespace; the 9.0 commands no longer apply.
• vCenter adds host and cluster summary cards with tier breakdowns and consumed versus active memory, a tiering device list with health status, and per-VM tier bandwidth. vCenter raises an alarm when a tiering device is unhealthy.
• VCF Operations adds a Memory Tiering dashboard and a What-If analysis that models enabling the feature and the resulting savings.
• Low-latency, confidential, and Fault Tolerance VMs can now power on on tiered hosts. They still do not participate in tiering. Nested ESX VMs are excluded by default and can opt in with sched.mem.enableNestedTiering = TRUE, set before power-on, as William Lam shows in his VCF 9.1 lab workarounds post.

Active memory decides the design

Memory Tiering presents DRAM (Tier 0) and NVMe (Tier 1) to VMs as one pool of logical memory. Only VM memory is tiered; kernel memory never is. At the default 1:1 ratio, a host with 1 TB of DRAM gets 1 TB of Tier 1 and 2 TB of logical memory.

Tiering starts when host consumption reaches about 80% of DRAM. From that point, ESX scans VM memory in batches, classifies pages by how recently and how often they are touched, and moves cold pages to NVMe and hot pages back to DRAM. That 80% figure describes when the mechanism activates, and I would not treat it as a sizing target or a recommended host utilization.

The goal of all this movement is to keep each VM’s working set in DRAM. Below the trigger, nothing moves, and Duncan Epping has pointed out that tiering only moves cold pages when there is memory pressure to relieve. Broadcom’s documentation separates it from swap on three points: swap picks pages largely at random, while tiering ages pages by recency and frequency; swap often sits on shared SAN or NAS storage, while the tier is a dedicated local NVMe device; and swap exists to keep VMs correct under extreme overcommit, while tiering is a capacity tier that has to be planned. The scan runs in batches with short pauses, the moves happen at the end of each cycle, and the scan rate follows memory pressure and VM size.

Page size is what makes the classification precise. ESX backs guest memory with 4 KB pages on tiered hosts because a 2 MB page can only move as one unit. Broadcom’s own example is a 2 MB page with 400 of its 512 small pages hot: the 112 cold pages stay in DRAM because the large page cannot be split.

The traffic between the tiers is also a warning sign. In vCenter, NVMe write bandwidth is cold pages moving down, and NVMe read bandwidth is pages being fetched back up because they became active again. A working set that outgrows DRAM shows up as rising read bandwidth, and every one of those reads costs NVMe latency, in the tens to hundreds of microseconds, instead of DRAM latency.

The sizing rule is a different number: Broadcom’s guidance is to keep a host’s active memory at or below 50% of its physical DRAM. It is a performance guideline, which Broadcom’s configuration documentation ties to keeping latency low. Broadcom says most workloads run at 10% to 30% active. Consumed memory is what VMs have allocated. Active memory is what they touch, and only active memory shows whether there is cold memory to move.

Take two clusters with 1 TB of DRAM per host and 800 GB consumed.

On a standard capacity dashboard, they look identical. For tiering, Cluster A has more than 600 GB of cold memory per host, and Cluster B is already over the line before adding any VM.

Measuring this properly takes history. Broadcom’s Memory Tiering Assessment Tool, now MTAT v3.1 on GitHub, reads active and consumed memory from vCenter, shows peak active memory over 24 hours, 7 days, or 30 days, surfaces large-page usage, and models DRAM:NVMe ratios and hardware cost. The historical view needs vCenter statistics at level 2. VCF Operations 9.1 adds the What-If analysis for the same planning step. I would start with MTAT because it reads vCenter history directly and shows the peak, not just the average. The Broadcom tips blog says to capture workloads under real load and in busy periods, so I would use the 30-day window and make sure it includes the month-end or business peak for that cluster. VCF Operations What-If comes after that, when the numbers need to become a business case.

Sizing the NVMe tier

In VCF 9.1 Memory Tiering, the DRAM:NVMe ratio sets how much NVMe capacity ESX adds as Tier 1.. In 9.1, it is set with tier_size_pct in the configuration profile, or esxcli memtier config set --tier-size-pct, as a value from 1 to 400% of DRAM. The default of 100 gives 1:1, and Broadcom recommends it for optimal performance. The sizing table in Broadcom’s documentation shows the default and two lower settings:

NVMe used is the smaller of the ratio value and the partition size. If no partition size is given, ESX uses the full drive, limits the tier to the DRAM size at 1:1, and caps the partition at 4 TB. The 2:1 and 4:1 rows add less NVMe, which leaves more DRAM per tiered page. Values above 100 go the other way, up to 1:4, where the tier is four times the size of DRAM. A higher value lets a host take on more VMs with the same DRAM, and those VMs bring their own active memory, which still has to fit under 50% of DRAM. The 4 TB partition cap also applies, so 1:4 needs 1 TB of DRAM or less.

Broadcom’s guidance is to start at 1:1 and to buy NVMe at least as large as DRAM. The VMware tips blog goes further and recommends larger drives for cell cycling and headroom if the ratio changes later. For a host with 1 TB of DRAM, I would specify the size Broadcom uses in its own example, a 1.6 TB Mixed Use drive, as a mirrored pair. That gives the 1 TB tier with room left over. I would stay at 1:1. Broadcom allows anything from 1 to 400% but recommends the default for performance, and its advice is to leave the ratio alone until measured active memory says otherwise.

The drive class is part of the sizing. Broadcom requires Mixed Use drives at 3 DWPD or higher, Endurance Class D (7,300 TBW or more), Performance Class F or G, installed locally. Page movement produces sustained random writes, and Mixed Use drives typically reserve about 28% of their raw NAND as over-provisioning, compared with about 7% for Read-Intensive drives, giving the controller room for garbage collection without stalling writes. The tier drive cannot be a vSAN cache or capacity disk, so a vSAN ReadyNode may need a separate pair of drives.

For protection, 9.1 mirrors the tier in software across two equivalent NVMe devices per host. In 9.0, the only option was hardware RAID with a Tri-Mode controller or Intel VROC. With mirroring, a consolidation design needs two tier drives per host, and that count belongs in the bill of materials from the start.

What a production design has to survive

A tiered design that works in steady state can still fail in four states: normal operation, workload peak, host failure (N+1), and maintenance.

The 50% active memory guideline applies to the peak and to the N+1 and maintenance states, when the same workload runs on one fewer host. A cluster that passes on averages can fail on peaks.

Tiering removes memory as the first limit, so CPU headroom becomes the next test. If the cluster was memory-bound with idle CPU, consolidation uses that CPU. If CPU was already busy, the bottleneck moves to CPU.

What happens when a tier drive fails

Broadcom documents three cases, and which one applies depends on how the tier is protected and where each VM’s pages sit.

• Mirrored tier, one drive fails. Broadcom describes the 9.1 software mirror as protecting workloads from a single device failure, so the running VMs are covered. The failed drive still has to be replaced. Hot-plug is not supported, so the host goes into maintenance mode, its VMs are evacuated, and the mirror is rebuilt with esxcli memtier config set -d "old-device, new-device" or through the configuration profile.
• Unprotected tier: a VM has pages on the failed drive. The VM keeps running until it touches one of those pages. If that page fault cannot be served, the VM powers off. vSphere HA restart policies can restart it on any other host in the cluster with enough free capacity, and no dedicated failover host is needed.
• VMs that use only DRAM. VMs excluded from tiering, or with nothing on the tier, keep running on the same host.

In both of the first two cases, putting the host into maintenance mode after the failure is manual in 9.1 unless an OEM plug-in such as Dell OpenManage or HPE OneView does it. The practical defense is to see the drive wearing out first. vCenter 9.1 raises an “NVMe Memory Tiering device is not healthy” alarm, and William Lam shows the SMART detail behind it under Monitor, Memory Tiering, or with esxcli system health report get -r vmw.memTierHealth. He treats Available Spare as the metric to watch and plans a replacement when it drops to 10%.

vMotion of a tiered VM takes 1.5 to 2 times longer because pages come back from NVMe first, so host evacuation and vLCM remediation take longer. The destination host starts without the source host’s page heat history; Broadcom says heat transfer is planned for a future release. Quick Boot is not supported on tiered hosts.

A VM excluded from tiering needs a full memory reservation, and that DRAM leaves the tiering pool. On a 1 TB + 1 TB host, two 200 GB reserved VMs move the effective ratio for everything else to about 0.6:1. Broadcom also says not to combine memory overcommit with tiering. Some VMs leave the pool automatically: a passthrough device such as a GPU or NIC on DirectPath I/O pins the VM’s memory in DRAM, and Latency Sensitivity High disables tiering for the VM. Both take DRAM away from the tiered VMs in the same way.

With tiering enabled, ESX backs guest memory with 4 KB pages by default so it can classify hot and cold memory precisely. 2 MB pages can be re-enabled per VM with monitor_control.disable_mmu_largepages = "FALSE", at the cost of cold 4 KB pages stuck inside hot 2 MB pages. VMs configured for 1 GB pages are locked to DRAM and must be sized against DRAM, not logical memory.

Put together, the assessment runs in this order:

• Collect 30 days of active and consumed memory at vCenter statistics level 2, including the business peak.
• Find the peak active memory per host.
• Test that peak against 50% of DRAM with all hosts running and with one host fewer, for N+1 and maintenance.
• Check CPU for the same states with HA admission control.
• List the VMs that stay in DRAM: full reservations, passthrough devices, Latency Sensitivity High, 1 GB pages, and VMs above 512 GB and 64 vCPUs.
• Set the ratio, 1:1 unless the measurements say otherwise, and size the drives: Mixed Use, 3 DWPD or higher, a mirrored pair per host, from the Broadcom Compatibility Guide.

Design 1: VDI and bursty working sets

VDI is on Broadcom’s list of workloads with good tiering results, and the profile fits: many desktops, long idle periods, large consumed memory. The risk is the morning logon storm, when active memory climbs across every desktop at once.

The design, per host (illustrative inputs): 1 TB DRAM, 1 TB mirrored NVMe tier, desktops at 16 GB. Without tiering, 1 TB of DRAM holds at most 64 such desktops before hypervisor overhead and headroom. With tiering, I test 100.

The averages pass, and the peak fails. Two ways to fix it: size for the peak, which, at the same per-desktop behavior, means about 85 desktops per host (roughly 470 GB active at peak, 47% of DRAM), or reduce the peak itself with staggered logons. Broadcom’s capacity planning guidance is to ensure active memory never exceeds 50% of DRAM at any point, so I would size for peak and plan 85 desktops per host.

Once memory stops being the first limit, density becomes a CPU, GPU, and storage question. Omnissa’s Horizon sizing guidance recommends two vCPUs for general users and starts at 8 to 10 desktops per physical core, then asks you to confirm in a pilot that host CPU rarely goes above a safety margin such as 80%. At 8 to 10 desktops per core, 85 desktops need about 9 to 11 cores. Compare that with the host’s core count, then let the pilot against the 80% margin decide. Two VDI-specific checks: GPU desktops and large pages. A desktop with a DirectPath I/O GPU has pinned memory that stays in DRAM, so a GPU pool reduces the DRAM available to everything else on the host. On large pages, Broadcom notes that re-enabling 2 MB pages in a tiered environment reduces both tiering precision and any Transparent Page Sharing savings a VDI design may have counted on.

Tiering can remove memory capacity as the first constraint on a VDI host. Peak active memory then decides the real density.

Design 2: consolidating a memory-bound workload domain

The starting point is Broadcom’s own host example scaled to a 10-host workload domain (illustrative inputs): each host has 1 TB of DRAM, 800 GB consumed, and 200 GB active. Cluster-wide, that is 8 TB consumed and 2 TB active. The candidate is 6-tiered hosts with 1 TB of DRAM and a 1 TB mirrored tier each.

The first pass assumes DRS spreads consumed and active memory evenly, which is optimistic. For the peak, I assume active memory reaches 1.5 times the average; replace that with the peak MTAT or VCF Operations actually reports.

Six hosts pass every state except a host failure during the peak. That leaves three options: accept the risk for that window, move to seven hosts (a failure during peak then lands at 500 GB, 50% of DRAM), or prove with measured data that the peak multiplier is lower than 1.5. I would take seven hosts, because it keeps the design within Broadcom’s 50% guidance even when a host fails during the peak. Broadcom states 50% as guidance; holding it through an N+1 peak is my own production criterion. Measured data can bring it back to six, but only if the measured peak is lower than the 1.5 times I assumed here. Maintenance must also stay outside the peak window, which is a constraint to write into the operations model.

Then the CPU test, assuming identical host specs. If today’s 10 hosts average 25% CPU, 6 hosts run at about 42% and 5 hosts (N+1) at 50%. If they average 45%, 6 hosts run at 75% and 5 hosts at 90%, and memory tiering has only moved the bottleneck to CPU. I found no Broadcom CPU ceiling for this state. vSphere HA admission control can check it: with DRS enabled, setting the “performance degradation VMs tolerate” value below its default of 100% makes vSphere warn when a host failure would cut into running workloads. I would set that value in the design and treat the warning as a failed test.

On cost: memory-bound clusters leave licensed CPU capacity underused. Higher density may allow hardware and core entitlement to be reduced at refresh or renewal, depending on the licensing agreement. Against that, the tiered design adds twelve or fourteen Mixed Use NVMe drives.

What the data can support is 6 or 7 hosts as a defensible candidate design, provided measured peaks and CPU both pass.

Where the model stops working

Tiering needs cold memory to move. A workload whose working set is large and hot by design has little of it, and SAP HANA is the clearest example. VMware’s own Memory Tiering design series lists SAP HANA and other in-memory databases as typical candidates to exclude.

Three documented constraints push these workloads out of the tiered pool:

• VMs with more than 512 GB of memory and 64 vCPUs may see performance degradation on tiered hosts; Broadcom recommends disabling tiering for them and setting a full memory reservation.
• A full reservation removes that DRAM from the tiering pool. On a 1 TB + 1 TB host, reserving 400 GB drops the effective ratio for the remaining VMs to about 0.6:1. A 1 TB HANA VM on the same host would leave no DRAM for tiering at all.
• VMs configured for 1 GB pages are locked to DRAM.

In my Building Resilient SAP Infrastructure on VCF series, the HANA database hosts stay non-tiered. The SAP application servers are a separate decision. I would treat them like any other VM: measure them with MTAT, and exclude any that exceed 512 GB and 64 vCPUs, following Broadcom’s guidance for large VMs. I found no public SAP statement on memory tiering, so the SAP notes for your release need checking before any SAP VM goes onto a tiered host.

The same rule applies beyond SAP: any database sized to keep its working set in memory belongs on DRAM, whatever the vendor.

Day-2 operations

A cluster running VCF 9.1 Memory Tiering needs a few extra checks in normal operation. The host and cluster Memory tiles in vCenter show Tier 0 and Tier 1 usage, and esxcli memtier status get confirm the configuration on a host. For a single VM, the Advanced Memory Performance chart shows NVMe read and write bandwidth. Broadcom recommends monitoring device latency when NVMe read bandwidth exceeds 200 MB/s and considers read latency below 200 microseconds acceptable, depending on the drive.

Changes need more care than before. Activating, deactivating, or reconfiguring tiering, changing the ratio, or replacing a drive all require maintenance mode. Those evacuations take longer than on a DRAM-only host, so vLCM windows need the extra time planned in.

Reservations are the change most likely to catch an operations team out. A full memory reservation added months after the design, for a new database or an application owner’s request, takes DRAM out of the tiering pool for every other VM on that host. The ratio example earlier shows how fast that happens: 400 GB of reservations on a 1 TB + 1 TB host moves the effective ratio to about 0.6:1. I would put new reservations on tiered clusters through change control and rerun the active memory check before approving them.

Conclusion: Would I deploy it?

I would use VCF 9.1 Memory Tiering in production where the measurements support it. The workloads Broadcom lists as a good fit are general virtualization, VDI, dev/test, application servers, and databases with moderate memory activity, and in each case the condition is the same: active memory at or below 50% of DRAM, measured over time.

I would avoid it for SAP HANA and other in-memory databases, latency-sensitive VMs, and passthrough GPU workloads, which stay in DRAM anyway. VMs above 512 GB and 64 vCPUs would default to DRAM unless workload testing gave me a reason to do otherwise. I would also avoid it in clusters whose active memory is already over the line, and in clusters that rely on memory overcommit, which Broadcom says not to combine with tiering.

Before changing a design, I want 30 days of MTAT data at vCenter statistics level 2 that includes the business peak, the 50% test passing in all four states, the CPU check passing through HA admission control, and tier drives from the Broadcom Compatibility Guide.

9.1 closes the gaps that made 9.0 tiering harder to design around: software mirroring without a RAID controller, configuration without a reboot, excluded VM profiles that can now power on, and monitoring in vCenter and VCF Operations. What remains is operational. Maintenance mode after a tier drive failure is still manual unless an OEM plug-in handles it, vMotion does not carry page heat to the destination host, tiered VMs take 1.5 to 2 times longer to migrate, and Quick Boot is not available.

The next decision for any team looking at this is to pull the history. Thirty days of active memory per host shows which clusters sit under the line and which do not, and that result decides whether any of the designs above apply.

Share this article if you think it is worth sharing. If you have any questions or comments, leave them here or contact me on Twitter (yes, for me it’s not X, but still Twitter) or LinkedIn, since I am getting off Twitter.

©2026 ProVirtualzone. All Rights Reserved