Second post in the Veeam Fix Series, where I take real issues I had to fix in customer environments and go through what the problem was and what the fix was. Names, IPs, and job names are dummy. The command output, error messages, and numbers are real.
This one is about two tenants that reported the same problem in the same week. Backups failing, no space. One of them was right. The other one had plenty of space, and the jobs still refused to run. Same symptom, two completely different causes, and two completely different fixes.
The environment
- Veeam Backup & Replication with Veeam Cloud Connect, service provider side
- Scale-out Backup Repository with four extents (E:, G:, H:, I:)
- Multiple tenants, each with their own quota on the cloud repository
- Tenant A: 19.5 TB quota, forward incremental chains going back years
- Tenant B: quota with free space, around ten jobs all starting inside the same 30-minute window
I will share some of the PowerShell commands I used so you can use them when you have a similar issue, or so you can learn what commands to use when you need to fetch data from your VBR.
The symptom
Both customers reported the same thing, but the errors in the console were not the same, and that turned out to matter.
Tenant A failed every run with a metadata write failure, repeated once for each extent in the Scale-out Backup Repository:
Error: Failed to save backup meta to [E:|Backups|TenantA_Backup|SRV008.vbm]
Failed to save backup meta to [G:|Backups|TenantA_Backup|SRV008.vbm]
Failed to save backup meta to [H:|Backups|TenantA_Backup|SRV008.vbm]
Failed to save backup meta to [I:|Backups|TenantA_Backup|SRV008.vbm]
Processing finished with errorsTenant B failed with a quota error, on a quota that still had space in it.
Backup repository quota has been exceededTwo tickets, two customers, one week apart. My first thought was that it was the same problem twice. It was not, and if I had applied the same fix to both, I would have made both worse.
The investigation
Tenant A: the quota really was full
This one took five minutes. The tenant was sitting at 19.5 TB used out of a 19.5 TB quota. Not 95%, not close to the limit. Full.
What confirmed it for me was which file was failing. The job could not write the .vbm metadata file, and it failed on all four extents. The .vbm is a few hundred kilobytes. If that cannot be written anywhere in the repository, there is nothing left to write to.
Veeam was doing exactly what it is supposed to do. The quota system worked. The customer had simply grown past the quota they had.
Where the space had actually gone
Knowing the quota is full is not the same as knowing what filled it. For that, I pull the size of every chain on the tenant, with the restore point count next to it:
Get-VBRBackup | Where-Object { $_.JobName -like "TENANTA_*" } | ForEach-Object { $j=$_.JobName; $rps = Get-VBRRestorePoint -Backup $_; [PSCustomObject]@{ Job=$j; RestorePoints=$rps.Count; GB=[math]::Round((($rps |
ForEach-Object { $_.GetStorage().Stats.BackupSize } | Measure-Object -Sum).Sum/1GB),2) } } | Sort-Object GB -Descending | Format-Table -AutoSizeJob RestorePoints GB
--- ------------- --
TenantA_Job01 496 5824,46
TenantA_Job02 468 5549,31
TenantA_Job03 448 5136,05
TenantA_Job04 420 4865,94
TenantA_Job05 372 4238,70
TenantA_Job06 358 4136,17
TenantA_Job07 20 1629,90
TenantA_Job08 20 1629,39
TenantA_Job09 22 482,68
TenantA_Job10 21 428,70
TenantA_Job11 21 398,17
TenantA_Job12 30 279,19
TenantA_Job13 30 274,91
TenantA_Job14 14 197,72
TenantA_Job15 13 188,40
TenantA_Job16 14 185,10
TenantA_Job17 3 127,61Look at the restore point counts on the top six. Several hundred points each, on jobs with 30-day retention. That is not growth; that is a chain that stopped trimming.
One warning on this, because I got it wrong the first time. There is a ApproxSize property on a restore point, and it is tempting to sum that instead. Do not. ApproxSize is the logical provisioned size that the restore point represents, so adding it up across a chain counts the same data once per point. The first time I did that on a repository of about 26 TB, the total came back near 100 TB:
Backup Points TotalGB
------ ------ -------
FILESRV01 14 42215.30
WindowsInfra 128 28507.50
SecuritySrv 20 12492.90
Finance 45 5973.80
LinuxSrvDC 46 4850.30
WindowsInfraNC 25 3758.20
LinuxInfra 40 1412.00
SQLBackups 8 802.90One VM apparently using 41 TB on a 26 TB repository is obviously not the disk. GetStorage().Stats.BackupSize is the actual backup size and it adds up to something that matches reality.
The restore point count column is as useful as the size. A job with 30-day retention showing well over a hundred restore points is not trimming, and that is usually where the space went.
Tenant B: same complaint, free space on disk
This tenant had room in the quota and free space on the extents, and the jobs still failed. Nothing had changed in the job configuration, and the failures were not random. They clustered around the time when several jobs started together.
That told me the failures were following the schedule and not the data.
How Veeam works out free space on a SOBR
This behavior is documented in Veeam KB2282.
When a job starts and nothing else is running, Veeam reads the actual free space of the extent it has been assigned. It then estimates how big the restore point will be, subtracts that estimate, and stores the result as the extent’s estimated free space. Every job that starts after that, while tasks are still active, works from that stored estimate instead of going back to the disk to look again.
Veeam Support explained the arithmetic on the R&D forums with a simple example. A 10 GB VM reserves 5 GB, which is 50% of the VM size, for its full backup. On a 100 GB extent, Veeam now believes 95 GB is free. It does not refresh that number from the disk while at least one task is still running.
This tenant had around ten jobs starting inside half an hour. Each one subtracted its own worst-case estimate from a number that was already an estimate. The estimated free space dropped towards zero while the disk itself was barely touched. Veeam then refused to start jobs because, by its own accounting, the extent was about to fill up.
This is a safety mechanism, not a bug. It exists to stop concurrent jobs from filling an extent and corrupting a chain. It does not handle a lot of jobs starting at the same time very well.
The fix for Tenant B
KB2282 documents an alternate calculation method. Instead of reading actual free space only when no tasks are active, Veeam recalculates the estimate when a task starts and is assigned to the extent, based on the real free space minus the estimated size of all active tasks. The recalculation will not run more often than once every 15 minutes.
It is enabled with a registry value on the backup server:
Key: HKLM\SOFTWARE\Veeam\Veeam Backup and Replication\
Value Name: SobrForceExtentSpaceUpdate
Value Type: DWORD
Value Data: 10 is the default and keeps the old behavior. 1 enables the periodic recalculation. The Veeam Backup Service has to be restarted for the change to take effect, so I did this in a maintenance window.
Cloud Connect quota is not the same as disk usage
Cloud Connect quota is not measured in bytes written to disk. Veeam tracks tenant quota consumption using the size of the source data before deduplication, which the Cloud Connect limitations page states directly.
The example in the documentation makes it clear. A tenant has a 100 GB quota on a deduplicating appliance. The backup data physically occupies 30 GB. The repository still reports 80 GB used, and the tenant can add only another 20 GB, not another 70 GB. As a service provider, that means over-provisioning the underlying storage to give tenants the quota they were allocated.
There is also a discussion in the Veeam community about repositories on XFS and ReFS, where block cloning saves space that the reported usage does not reflect. That is a forum observation and not official documentation, so I treat it as something to check rather than something to quote at a customer.
How I tell the two apart
I check four things before I touch anything, because the two fixes go in opposite directions. All of these are read-only.
First, load the module. On PowerShell 7, the old Add-PSSnapin does not work, so it is:
Import-Module Veeam.Backup.PowerShellThe warning about unapproved verbs is harmless.
1. Quota used against quota allocated
This is the one that decides everything. The quota and the used space live on the tenant’s Resources property, and both are reported in MB.
Get-VBRCloudTenant -Name "TenantA" | Select-Object -ExpandProperty Resources | Format-List RepositoryFriendlyName, RepositoryQuota, UsedSpaceFor a view across every tenant on the server:
Get-VBRCloudTenant | Select-Object Name, @{Name='QuotaMB';Expression={$_.Resources.RepositoryQuota}}, @{Name='UsedMB';Expression={$_.Resources.UsedSpace}}A tenant with more than one quota returns an array in those columns, so use the first command for anything you want to look at properly. Get-VBRCloudTenant needs a Cloud Connect provider license. Run it on a VBR without one and you get:
Get-VBRCloudTenant: Veeam Cloud Connect service provider license is required.Which usually means the session is connected to the wrong server rather than anything being broken.
2. Which file is failing
Read the failed sessions rather than the summary. If a small .vbm write is failing on every extent, the repository is out of room.
Get-VBRBackupSession | Where-Object { $_.Result -eq "Failed" } | Sort-Object CreationTime -Descending | Select-Object JobName, CreationTime, EndTime, Result3. Real free space on the extents, not the reported usage
This is the point of the whole post. Veeam’s estimate is what refuses the job, so check the disk itself rather than what Veeam reports. List the extents first:
Get-VBRBackupRepository -ScaleOut | Get-VBRRepositoryExtent | Select-Object -ExpandProperty Repository | Select-Object Name, PathThen read the actual volumes on the server hosting them:
Get-Volume | Where-Object { $_.DriveLetter } | Select-Object DriveLetter, FileSystemLabel, @{Name='SizeGB';Expression={[math]::Round($_.Size/1GB,1)}}, @{Name='FreeGB';Expression={[math]::Round($_.SizeRemaining/1GB,1)}}If the disk has room and the tenant is under quota, the estimation is the likely cause.
4. Whether the failures follow the schedule
Estimation failures cluster when jobs overlap. Real exhaustion fails on every run, including the ones that run alone. The CreationTime column from step 2 is what shows this, so sort the failures by time and look at whether they land together.
It is also worth checking whether the KB2282 value is already set on that server before assuming it is not:
Get-ItemProperty -Path "HKLM:\SOFTWARE\Veeam\Veeam Backup and Replication" -Name SobrForceExtentSpaceUpdate -ErrorAction SilentlyContinueNothing returned means the value does not exist and the default behavior applies.
The remediation
For Tenant A, there was no technical fix. The quota was full, and the options were to increase it or to reduce the amount of data being protected. I raised the quota temporarily so backups could run while it was being sorted out, and I told the customer the same day that it was temporary. A temporary increase that nobody follows up on stops being temporary.
In this case, there was also a lot of recoverable space in old chains, incomplete restore points, and more full backups than the retention actually needed. Cleaning that up freed real space, so part of the answer was housekeeping rather than a larger quota.
For Tenant B, I added the registry value and restarted the Veeam Backup Service in a maintenance window. One thing to know before applying it. This is a backup server-wide setting. It changes free space calculation for every Scale-out Backup Repository on that server, not only the tenant I was troubleshooting.
I have not seen enough concurrent windows since to call Tenant B closed. The mechanism is documented, and the change is the one Veeam describes, but it is running and being watched rather than fixed.
An important note on the data
This is a real case, but every name in it is fake. Tenant names, job names, VM names, server names, IPs, and file paths have all been replaced with dummy values. What I did not change is the output itself. The error messages, the restore point counts, the sizes, and the command results are exactly as they came back, because seeing what these commands actually return is the useful part.
Conclusion
Two tickets a week apart, both reported the same way, and I nearly worked them as the same ticket.
The path that separated them was short. Quota used against allocated first, then which file the job was failing to write, then whether the failures lined up with overlapping job starts. A few minutes of checking decides which of two completely different investigations you are in.
What I took from it is that the error text is the weakest evidence in the ticket. Tenant A’s error never mentioned quota at all; it was a metadata write failure. Tenant B’s error said there was not enough disk space when the disk was fine. Both times the numbers told the truth and the message did not.
The other habit I kept is looking at what is actually stored before assuming a tenant has outgrown its quota. Chains that had stopped trimming looked exactly like growth until I pulled the restore point counts.
In any troubleshooting I do, I take notes. I save the commands I ran, the output, and what I learned. I have always kept a file for this, and I still do, because you never know when you will need it again. Many of my blog posts are those notes. I come back here often to check things I fixed in the past, and this time the record is public, and I am using it again.
Share this article if you think it is worth sharing. If you have any questions or comments, leave them here or contact me on Twitter (yes, for me it’s not X, but still Twitter) or LinkedIn, since I am getting off Twitter.
