In this new blog post included in the NSX-T series, I will detail a step by step how to Backup and Restore NSX-T v3.0.

I will focus the Backup and Restore NSX-T article using two Backups examples:

• Backup and Restore NSX-T using NSX-T Appliance Backup options (recommended option).
• Backup and Restore NSX-T using Veeam Backup & Replication

Environment:

• NSX-T Cluster v3.0 with 3x nodes.
• vCenter v6.7
• Veeam Backup & Replication v10
• Backup repository Synology DS1515+

Backup

There are 3 types of backup for the NSX-T nodes and Cluster.

1 – Cluster Backup
In this type of NSX backup is included the state of the Virtual Network.

2 – Node Backup
In this type of NSX backup, we are only backing up the NSX Manager node.

3 – Inventory Backup
In this type of backup is included all the NSX inventory environment, like ESXi hosts (or other Hypervisors) and the Edge Nodes.

When we need to do a Restore, the inventory Backup is the data is used to sync and fix any out of sync or any inconsistencies between any of the Management nodes, Host Transport Nodes, or Edge Nodes. When restoring NSX-T, make sure that all Management Plane in the NSX-T is sync and up to date between all the layers.

Note: Inventory backup is only available in automatic backups.

Backup and Restore NSX-T using NSX-T Appliance Backup options.

Note: Before starting this option, I need to inform you that this one is the recommended method to backup your NSX-T environment. So you should always use this option when you can.

Using the NSX-T appliance backup option, we need o use an SFTP server (TCP port 22). There is no other option when using the NSX-T Appliance option, NSX used STFP to transfer files backup to an SFTP server destination.

There are two methods to backup: Manual and Automated NSX Manager. Both can be used for the NSX-T Manager node or a Cluster Backup.

To accomplish this and have an SFTP server, you can create or use a Linux VM, create an SFTP server, and save your files on it. You can also install an SFTP server in a Windows server. Or you can use your Storage System to do this (if you have this option in your Storage System).

In this case, I will enable SFTP service in my Synology and use it as a destination for the NSX Manager backups.

I will not go through the options to create and use SFTP in Synology, just adding an image where to enable this service.

After you have your SFTP Server up and running, we need to configure our NSX-T to connect to your SFTP server.

By default, NSX Managers are base on their IP addresses and, consequently, also the Backups. If you want to use FQDNs, then your NSX Managers must be set to use FQDNs, then Backup will use FQDN and not IP addresses on the backup nodes records.

In your NSX-T GUI, go to System tab and then select Backup & Restore and Edit SFTP server to configure.

Add SFTP Server

In the Backup Configuration, add your SFTP details like server FQDN/IP, Port (default is always 22), Directory path, and user/pass to connect to the SFTP server.

Note: Be aware if using Windows Server, the maximum length limit for the Directory path is 260 characters.

We will detail the next two options since they are significant for the backups.

1 – Add SSH Fingerprint from SFTP server

This is an example of how to get the 256 (ECDSA) SFTP Server SSH Fingerprint.

#/etc/ssh$ for file in *_key.pub
> do ssh-keygen -lf $file
> done

Note: Depending on your Linux OS, you can use different commands to get the SSH Fingerprint, this is just one of them.

2 – Enter Backup Passphrase

You need to add a Passphare (a normal password) to your backups, and you do remember or saved in a safe place since you cannot restore a backup without this password. If you lose this password, your backups are useless and cannot be restored.

Next, click Save and add your SFTP server to the NSX-T Backup & Restore.

Note: Always double-check the directory path if it is correct. NSX-T doesn’t check if the directory exists and accepts any path that you add. The wrong path will trigger failed backups.

After we add the SFTP server, then NSX-T is ready to start automatic or manual backups.

Click the Start Backup to create a manual backup of your NSX-T Cluster node(will create only a backup of the node and Cluster configuration, not all the nodes in the Cluster).

Just an example this is the screen from previous NSX-T 2.4.x.

Backup Error:

One error that you may encounter when you try to backup your NSX-T is this one: “either sftp server disk is full or bad directory path or check if the directory length is beyond 260 character limit on windows server”.

The route cause can be found in the following:

• • There is available disk space on the SFTP server.
  • The backup directories (i.e., cluster-node-backups, etc.) are created correctly on the SFTP server.
  • The SFTP server used is running on a Windows server.
  • NSX Manager logs (syslog) display message(s) similar to
    “/NSXBackups1/cluster-node-backups’ not found., remoteUri=sftp://192.168.1.252:22/NSXBackups1/cluster-node-backups, errorCode=null, startTime=1589234673911, endTime=1589234675697]; responseBody=null]”

In this case, was the Directory Path was wrong, fix the directory path with the proper name, and will fix this issue. As stated above, the NSX-T backup SFTP configuration will not check if Directory Path is correct or exists.

Configure Schedule Automatic Backups

To schedule automatic backups, click Edit in the Schedule option.

As we can see in the next image, there is daily vs Weekly backups (select the days and hour you want the backups to run) or Intervals setting one backup per hour(s) or minute(s).

Also, you can enable the option “Detect NSX configuration”. This will trigger a backup every time NSX-T as any changes in the database.  Only use this option if it is needed, since NSX-T inventory is backup with the scheduled backup, no need to use this option for that purpose.

Note: You need to select very careful your schedule since this will trigger many backups, mainly Detect NSX configuration option, and it could full your SFTP server.

This is an example of all schedule options enable (1 backup per 1h and also Detect NSX configuration option enabled).

As we can see above, just in 3 days, I have 141 backups. And this is a test environment. With a production environment with more changes, you will reach thousands of backups easily in 1/2 days.

After you configure all the Backup settings, these settings are sync to all nodes, and all nodes will have the same configurations, and will backups can manually start from any  NSX-T Node.

Where are backup files, and what are those files?

Inside of the SFPT Server Directory path, this is the backup folders:

NSX@homestorage:/volume2/NSXBackups$ ls -hl
drwxrwxrwx+ 1 NSX users 138 May 9 01:18 ccp-backups

Folder for previous NSX-T v2.4.x inventory backups. Inside we will see a folder for each Manager Node in the NSX-T Cluster.
File example: ccp-2020-05-08T23_18_38UTC.tar

drwxrwxrwx+ 1 NSX users 552 May 9 03:21 cluster-node-backups

Folder for NSX-T Manager Cluster. Inside is a folder for each Manager node (v2.4x or 3.x) and each backup with the files backups (controller, manager, policy
and node backup files)
Files examples:
  cluster_backup-1eb5cfb4-ba2d-4dca-8d04-f13baf35c9eb-192.168.1.151-nsx-controller.tar
                     cluster_backup-1eb5cfb4-ba2d-4dca-8d04-f13baf35c9eb-192.168.1.151-nsx-manager.tar
                     cluster_backup-1eb5cfb4-ba2d-4dca-8d04-f13baf35c9eb-192.168.1.151-nsx-policy-manager.tar
                     cluster_backup-1eb5cfb4-ba2d-4dca-8d04-f13baf35c9eb-192.168.1.151-phonehome-coordinator.tar
                     node_backup-1eb5cfb4-ba2d-4dca-8d04-f13baf35c9eb-192.168.1.151.tar

drwxrwxrwx+ 1 NSX users 276 May 9 03:25 inventory-summary

Folder for NSX-T Manager Cluster for NSX-T Cluster inventory backups.
Files examples: inventory-2020-05-11T23_52_09UTC.json

With the explanation about NSX-T backups, we finish the section on how to Backup and Restore NSX-T v3.0 using NSX-T appliance options.

Backup and Restore NSX-T using Veeam Backup & Replication.

Important Note: Before we continue with this option is important to reference that VMware or Veeam does not support this process. VMware doesn’t recommend using NSX-T snapshot or image backups, and best practices recommend disabling snapshots in the NSX-T VM.

So why recommend this option for NSX-T Backup & Restore? I have already used a couple of times and did not get any issues with the NSX-T nodes that I needed to restore.

Using Veeam Backup Storage Snapshots, we could bypass the VMware not support issue. Because snapshots are done in the Storage Volumes and not in the VMs. So, if you have available this feature in your Veeam Infrastructure, use it instead of the VM Snapshots backup jobs.

Note: I had a “warning” from Veeam that the above statement was not 100% correct and could lead to misunderstanding. Or it can be done, but the option above is only possible in special occasions and special configurations (like one VM per storage volume / datastore) by using Snapshot Orchestration. That is only available in some Storage systems check HERE with are available.

Why is this backup method not supported? Because while the snapshot vs backup is creating NSX-T inventory can change and those changes will not be taken into account, and when we restore, we could have an inconsistency (since there is no DB truncate here) between NSX-T inventory and the rest of the objects (like vCenter and networks).

But the above statement is the same for the NSX-T Appliance Backup method. Any segments created after the backup was taken are not restored. Example: If you have created a logical switch or segment since the backup, the logical switch or segment will not appear after the restore.

For this option to work is mandatory that NSX-T  Backup needs to be on an NSX-T with a single node and/or in NSX-T Cluster all nodes, and the restore needs to restore all nodes, not just one. Also, NSX-T Nodes should be all synchronized, and no issues should be found in the System Overview.

If you have Edges running in your NSX-T environment you can also backup the Edges. But Edges as easy to re-deploy if needed, and this is not mandatory to backup with Veeam.

So that Veeam can backup properly the NSX-T the backup option, we should enable the option VMware Tools quiescence.

Note: I am still testing this method correctly and do some extra tests for having a consistent NSX-T Veeam Backup.  This option should not be used in a production environment, particularly if have a lot of changes between backups. So please use it carefully and at your own risk.

First, add your NSX-T nodes to the Veeam Backup. In this case, I only add the vCenter Folder where all the NSX-T nodes located. So if there is 1 or more, all will always be backup.

I also added to this backup the Edge if I need to restore.

Next, I select the Incremental option and do a daily backup.

And most important, enable VMware Tools quiescence for this type of backup.

Note: If you enable the VMware tools quiescence option in the job settings, Veeam Backup & Replication will not use Backup from Storage Snapshots to process running Microsoft Windows VMs that have VMware Tools installed. Also by enabling VMware tools quiescence there is always a VM snapshot.

In the next steps, do not change anything.  Do not enable application-aware, since we have enabled VMware Tools quiescence.  Backup is configured, you can click Finish and create the Backup Job.

To finalize this section, a tip for NSX-T Veeam Backups. If you have Veeam Backups Storage snapshots, then the backups of your NSX-T environment will be easy and faster. Since working with Storage Snapshots and not VM Snapshots, the impact on the running NSX-T VMs will be much less.

With Veeam Storage Integration snapshots, we are not using any process that is not recommended by VMware since there are no snapshots on the NSX-T Node VM.

Restore

Restore NSX-T using NSX-T Appliance Restore options

To restore your NSX-T Cluster Node, you need to deploy a new NSX-T VM into the vCenter using the ova file.

You must deploy the new NSX-T Node with the same IP address or FQDN if you are using FQDNs in your NSX-T Cluster (and also in Backups).

Note: Only IP address backups are supported for the Global Manager appliance.

Before you restore an NSX-T backup you need:

• SSH fingerprint from SFTP
• Passphrase of the backup file

After you deploy the new NSX-T, login into GUI and go the System tab and Backup & Restore option and configure the SFTP Server again to connect to SFTP and import all the Backups.

Select your backup and click Restore. You need to select the same Backup Node (in this case, IP, not FQDN) from the same NSX-T Node that you are connected.

After you click Restore, you will get the information about what is mandatory to Restore all your NSX-T Management Cluster.

• • You must Power Off any the NSX-T Nodes of the Cluster you are trying to restore.
  • You need to wait some minutes and then login again to the NSX-T node and resume the Restore.

Confirm that all objects that belong to this NSX-T Cluster are up and running(like vCenter, ESXi hosts, Edges, etc.). Backup needs all the inventory running, if not, the restore may fail.

Restore started.

Next, Restore recognizes that you have NSX-T Cluster or just restoring a standalone NSX-T Node.

You will need to deploy the other NSX-T Nodes after this NSX-T is restored, so that the Restore may continue to restore all inventory and NSX-T environment configuration.

In this case, the 192.168.1.154 is the NSX-T Node that I am restoring.

After a reboot of the NSX-T that was restored, we see only one Node, and we need to deploy the other two(or the number of nodes that you had in your Cluster).

Do not go to the Backup & Restore section in this stage and until you have deployed the NSX-T Nodes and all up and running.

Note: Do not forget that you need to deploy Nodes with the same IP or FDQN that they had before.

After I deploy all the NSX-T Nodes, I return to the Backup & Restore and resume the restore.

The first warning I get is that my ESXi hosts and Edge Node were not accessible. Since this is a nested environment, they were Power off.

So you should Power On all vCenter, ESXi hosts, and Edges.

Then the Restore resumed and finished.

After all, NSX-T Nodes were restored, and all configuration I had my NSX-T environment back and running.

Note: There were some minor issues with some of the Host Transport Zones not being correctly connected, but after putting the NSX Maintenance mode, sync, reboot of the ESXi host, and reconnect, all were green.

This issue could be because I had ESXi host power off when I started the Restore.

After the last steps and fixing some minor issues, the Restore of NSX-T Nodes using NSX-T Appliance options is finished.

Just a couple of tips for maintaining and checking NSX-T Backups.

How to List Available Backups in the SFTP Server.

To list all available backups in the SFTP Server we need to run a script that is located in /var/vmware/nsx/file-store/ the script name is get_backup_timestamps.sh.

Te script can run in the NSX-T Node, but also outside of the NSX-T Node in any Linux Server or even in the Storage System console.

Run the script in NSX-T Node, or copy to another server.

I will run inside the NSX-T Node.

As we can see above, by not entering any number the script will display all backups. If you give it a number of the latest backups, you will have only those.

With this information, you can check witch backup you need and which one you should look for in the Restore option.

You can have more information about this script HERE.

Remove Old Backups

In this is subject I don’t know why VMware doesn’t have this set in the GUI and delete the old Backups automatically.

To remove old backup, you need to run a script that is in /var/vmware/nsx/file-store/ the script name is nsx_backup_cleaner.py.

Running this script, you can remove old backup, or set a retention period. We can set the number of files we want to keep(default is 100) by using -l NumberofFiles and the minimum of days we want to keep a file by using -k NumberofDays.

In my case, I will set 30 days and a minimum of 20 files.

Add this script to your SFTP Server and create a schedule to run it and have a list of backups cleaner.

Note: The script needs to run in the SFTP Server (in my case in Storage Synology).

You can have more information about this script HERE.

Restore NSX-T using Veeam Backup

Note: I will again inform you that VMware does not support this option. That doesn’t mean it doesn’t work, is just not supported.

Also, this solution is only for Virtual NSX-T Nodes VMs. If you are using any NSX-T as a physical server, then you need to use the NSX-T Backup solution. We could also use Veeam to backup a physical NSX-T Node, but I will not present in this blog post.

To restore the NSX-T VM Nodes using Veeam, there are no special requirements to restore this. The only thing is that any Edges should be power off when you are restoring the NSX-T Nodes.

Also, this solution is only for a standalone NSX-T Node, or the full NSX-T Cluster(restoring all Nodes in the Cluster).

Select your NSX-T Backup VMs in Backups – Disks and select Restore entire VM.

Next, select the restore point you want to use and click Next.

In the next steps, you can select if you want to restore in the same ESXi hosts (you should use the same ESXi hosts), same datastore, folder, and name (since I did not delete the old NSX-T VMs, I choose a different name for the restored VMs with a prefix New_).

Click Finish to start the restore.

After restore is finish and VMs are created in the vCenter, they are ready to Power On.

Next, I just Power On the NSX-T Nodes and wait for some minutes so that all could sync, and all is good and ready to be used.

Note: After Power On the NSX-T Nodes, Node 192.168.1.151 was showing degraded, need to restart the manager service using command restart service manager. After a couple of minutes, the node was green.

The last step is to Power On the Edges and wait that they sync with the Hosts Transport Zones and then 100% success.

As we can see above, restoring the NSX-T Nodes with Veeam is very straightforward and very easy. It only took 10/15m to restore all the Nodes and another 5/10m for the Nodes are up and running and ready to be used. This is more than half of the time we spend restoring using NSX-T Backup & Restore option.

With the restore of NSX-T Nodes using Veeam we finish this blog post a step by step how to Backup and Restore NSX-T v3.0.

I hope this article was useful and show you the two different methods that you can use to backup your NSX-T Environment.

More articles about NSX-T:

• How to upgrade NSX-T 2.4.x or 2.5.x to NSX-T 3.0
• NSX-T log partition full and not able to login

Note: Share this article if you think it is worth sharing.

©2020 ProVirtualzone. All Rights Reserve