VMware launched NSX-T 3.0 in the last 7th of April, time to talk about How to upgrade NSX-T 2.4.x or 2.5.x to NSX-T 3.0.

Even it is not very difficult to upgrade the NSX-T environment, we should take into account some rules and processes before and after the upgrade.

Before we start the How to upgrade NSX-T 2.4.x or 2.5.x to NSX-T 3.0, let us check what is new in the NSX-T Data Center 3.0.

What is new in NSX-T 3.0?

There is a lot of changes in this new version; I will only focus on the ones that I think are the most important.

• Cloud-scale Networking: NSX Federation
• Intrinsic Security: Distributed IDS, Micro-Segmentation for Windows Physical Servers, Time-based Firewall Rules, and a feature preview of URL Analysis
• Modern Apps Networking: NSX-T for vSphere with Kubernetes, container networking and security enhancements
• Next-Gen Telco Cloud: L3 EVPN for VM mobility, accelerated data plane performance, NAT64, IPv6 support for containers, E-W service chaining for NFV
• Federation
  NSX-T 3.0 introduces the ability to federate multiple on-premises data centers through a single pane of glass, called Global Manager (GM). GM provides a graphical user interface and an intent-based REST API endpoint. Through the GM, you can configure consistent security policies across multiple locations and stretched networking objects: Tier0 and Tier1 gateways and segments.
  
  
• AAA for NSX-T features in vSphere with Kubernetes
  Users running containerized applications and Kubernetes features in vSphere 7.0 appliance can leverage and troubleshoot a limited number of NSX networking features without additional authentication via vSphere appliance.
• Networking
  • L2 Networking

NSX-T support on VDS 7.0
NSX-T now has the capability to run on the vSphere VDS switch version 7.0. It is recommended that new deployments of NSX and vSphere take advantage of this close integration and start to move toward the use of NSX-T on VDS. The N-VDS NSX-T host switch will be deprecated in a future release.

Going forward, the plan is to converge NSX-T and ESXi host switches. The N-VDS remains the switch on the KVM, NSX-T Edge Nodes, native public cloud NSX agents and for bare metal workloads

• • L3 Networking
    Change of Tier0 Gateway HA mode through UI/API offers the option to change Tier-0 gateway High Availability mode from Active/Active to Active/Standby and vice versa through UI and API.
    VRF Lite support provides multi-tenant data plane isolation through Virtual Routing Forwarding (VRF) in Tier-0 gateway. VRF has its own isolated routing table, uplinks, NAT, and gateway firewall services.
    L3 EVPN support provides a northbound connectivity option to advertise all VRFs on a Tier-0 gateway through MP-BGP EVPN AFI (Route Type 5) to a Provider Edge and maintain the isolation on the dataplane with VXLAN encapsulation by using one VNI per VRF.

• NSX-T 3.0 now supports:
  • Hypervisor:
    • vSphere 7.0 (including VDS 7.0)
  • Bare Metal:
    • RHEL 7.6
    • RHEL 7.7
    • Windows 2016
  • AWS and Azure Gov Clouds
• Distributed Intrusion Detection System (D-IDS)
• NSX Data Center for vSphere to NSX-T Data Center Migration

Migration Coordinator with Maintenance Mode – When you are using vSphere 7.0 and vDS 7.0, the Migration Coordinator will migrate hosts to an existing vDS (version 7.0) instead of migrating to an N-VDS. This minimizes the impact of the migration on the customer environment.

Migration from NSX Data Center for vSphere to NSX-T Data Center using vDS 7.0 – The NSX Migration Coordinator now supports maintenance mode for the final host migration step. This mode allows the migration of virtual machines from a host prior to converting the host from NSX for vSphere to NSX-T. By placing a host into maintenance mode, a virtual machine can be migrated using vMotion to minimize the impact to the data traffic to and from the virtual machine.

• Operations:

  • NSX Alarm Framework and System Alarms/Events
  • Reduction of VIB size in NSX-T
  • Support for both Thin and Thick Disk Mode
  • Increased disk size of NSX Manage
  • Central Appliance Configuration
  • Non-disruptive in-place upgrade

Note: Some of the above features only work with vDS 7.0, this means it is only supported when you have NSX-T 3.0 integrated with vCenter 7.0.

You can check all the new features and changes here in the Release Notes.

Regarding the matrix update not only for NSX-T but also where NSX-T 3.0 is vSphere version supported, we can check in the next images.

For vCenter, NSX-T 3.0 is supported from 6.5U2 to the new 7.0.

For the NSX-T upgrade matrix, it is possible to upgrade directly to 3.0 from NSX-T 2.4.0. If you have a version older than that, you need first to upgrade to 2.4/2.5 and then upgrade to 3.0.

How to upgrade NSX-T 2.4.x or 2.5.x to NSX-T 3.0.

Upgrade environment:

• vCenter 6.7
• 3x ESXi hosts 6.7
• 3x NSX-T nodes 2.4.1 running in a Cluster
• 1x NSX-T Edge Cluster

Note: This upgrade was done in a nested test lab.

Download the Upgrade bundle.

First, download the NSX-T 3.0 upgrade bundle.  Download from your VMware support site.

Select the NSX 3.0 Upgrade bundle.

After you have your upgrade file, you can upload it to your NSX-T Manager controller using GUI. But before we start, we need to do some pre-list checks.

Upgrade NSX-T components process is as follow:

1. Upgrade Cloud components (upgrade or install latest patches)
2. Upgrade vCenter (upgrade or install latest patches)
3. Upgrade the ESXi hosts (upgrade or install latest patches)
4. Upgrade the Upgrade Coordinator
5. Configure ESXi hosts
6. Upgrade the NSX Edge Cluster
7. Upgrade NSX ESXi host
8. Upgrade NSX Management nodes

Upgrade prerequisites tasks:

Make sure all components have the latest updates installed before you start your upgrade process.

1 – Upgrade the Upgrade Coordinator

First, just if your Upgrade Coordinator is running on the node, you are working. Or change to another node.

Login ssh to your NSX node with admin, or root, then su admin and then check the service.

Using get service install-upgrade command, you can check where the Upgrade Coordinator is running. In my case was running in the 10.87.68.155, but I want to run in a different node, so run set-repository-ip will automatically change the Upgrade Coordinator to the node you run this command.

If service is not running, run: start service install-upgrade

If you try to start the upgrade in the wrong NSX-T node, you will get the next warning.

Note: Do not use NSX-T Virtual IP to start any Upgrade process.

2 – Configure ESXi hosts

If the ESXi hosts are part of a fully enabled DRS Cluster, check if you have DRS enabled in your vCenter Cluster.

If your NSX nodes are running in the same ESXi hosts that are used as NSX hosts (in the Host Transport Zones), then be aware that all ESXi hosts during the upgrade will enter Maintenance Mode and all VMs needs to vMotion to another ESXi host during this upgrade.

So always check that you have enough resources in the Cluster for NSX can move vMotion and also other VMs that you may have running in this ESXi hosts.

Note: Any Standalone ESXi hosts or ESX hosts that are part of a disable DRS Cluster, for the safe side, needs to be in maintenance mode.

3 – Configure NSX Nodes

Add a second virtual disk of 100Gb in each NSX-T VM node that you will upgrade to 3.0.

4 – Check your NSX-T Cluster configuration

All NSX-T nodes need to be in sync, and NSX-T Cluster needs to run without any issues.

As we can see in the above image, all nodes are sync and no issues with disks, but still, you should connect to one of the NSX nodes and run command get cluster status.

In the next image, we check the manager with get manager and also the Cluster status. All configuration is stable, and nodes are up, so we go to go to start the upgrade process.

5 – Upload upgrade bundle file

From the NSX-T node that you select to be your Upgrade Coordinator, upload the upgrade bundle file.  In the menu, select the Upgrade tab and upload the file.

After you upload the file NSX upgrade will import and check the file compatibility matrix of the upgrade bundle (supported version). If all is ok, you will get “Upgrade Bundle retrieved successfully,” and you can start your upgrade by clicking in Begin Upgrade.

When the file is uploading, and the NSX-T node is processing the file, you may notice some high utilization of node resources (like CPU). If you are performing this upgrade while the NSX-T infrastructure is in production, be aware of this.

In this case, CPU was 100% utilization for some time until finished the process.

6 – Backup NSX-T Nodes

Last but very important is to backup your NST-T infrastructure before you start the upgrade process.

You can use the Backup & Restore option that exists in the NSX-T node main menu. I will update this post with a new article that I write how to Backup & Restore NSX using this option.

In this case, I Backup the NSX-T VMs with Veeam Backup & Replication using the “application-aware processing” option.

Check HERE my article about how to backup NSX-T Nodes.

Begin Upgrade process

After we have set and configure all the pre-requisites for the upgrade, we now will start the upgrade process, and before we click start, we need to run the pre-checks, so that upgrade process checks that all NSX-T infrastructure is prepared for the upgrade.

Note: Since we already add upload the file, you can ignore the Upload Upgrade Bundle option here.

In my case, I had two errors one in the Edges (one will only be shown in the upgrade Edge step) and one in the Managements.

Errors:

1 – Management hosts.

As we can see in the Management Hosts error message, I forgot to add the second disk of 100Gb in each NSX-T node VM, so the upgrade cannot start if I do not fix this.

You can just add the virtual disk to your NSX node VMs and re-run the pre-checks. It should automatically recognize the disk, but if not, you need to reboot each NSX-T VM.

After you reboot, you can resume the upgrade process without any problem.

Note: The second error in Management Hosts was just warnings about firewall ports and backups. Both should be done before starting the upgrade process.

After Management Hosts errors are fixed, re-run pre-check again, and after error-free, click Next to continue the upgrade process.

Next upgrade Steps are:

1. Upgrade Edges Nodes
2. Upgrade NSX-T ESXi Hosts
3. Upgrade Management NSX-T Nodes

Note: In each section, there is a RESET button, you can click and restart the upgrade process from the beginning.

Click Start in each section to start the upgrade.

Upgrade Edges Nodes

When I click start I get the next error:

The problem with the Edges is that no Edge Node was part of an Edge Cluster.

I quickly went back to options tab Fabric – Nodes – Edge Cluster and create an Edge Cluster.

After fixing the above issue, click Start again. The upgrade process started for Edges Nodes.

Then the upgrade was 100%, and all finish without errors.

Upgrade NSX-T ESXi Hosts

This section is to upgrade the ESXi hosts that have the NSX-T installed.

As stated above, when you start this process will put ESXi hosts in maintenance mode, so plan wisely so that any NSX-T Node VM that may run in the ESXi host can vMotion to another host and also other VMs that you may have running in the ESXi host.

Note: If possible, put all ESXi host that needs to be upgraded in Maintenance Mode before you start the process.

After all the ESXi hosts were upgraded, we can see 100% finished.

Note: I had an issue with one ESXi host that was not able to upgrade, and then the NSX-T was failing. Getting “NSX Installation Failed” and also when trying to uninstall “NSX Uninstalled Failed,” and was not able to use previous or new the version.

It was not an upgrade issue, but a host issue (even was the upgrade that triggers it). It needed a lot of work to fix this issue that would require a different blog post on how to fix this issue. I Will publish that error and the fix and I will update this blog post with a link.

After ESXi hosts are finished, click Next to start the next upgrade step, Management Hosts.

Note: ESXi hosts will reboot after the upgrade.

Upgrade Management NSX-T Nodes

In the section Management Nodes step, click Start to start the upgrade.

Note: Beware that during the process NSX-T, it will reboot each NSX-T after the upgrade. Meaning that the last NSX-T node to be upgraded is the one that has the Upgrade Coordinator.

NSX-T Upgrade will download the images to each NSX-T node and start the upgrade.

After the reboot, the Upgrade Coordinator Node was always saying that needs to finish the upgrade when the process was already completed (in the Upgrade Management Nodes option). I was getting a warning to continue the upgrade when there was nothing to continue.

If I Click continue, will go again to the Management Nodes upgrade step and all Management Nodes upgrade status were failed (sorry I did not create a print screen of this), but if I check in other NSX-T Node I get already the new NSX-T 3.0 GUI and also this:

I reboot the NSX-T Node with the Upgrade Coordinator and try again, but I get the same.  So the upgrade was successful, but somehow the upgrade process was not finished properly in the Upgrade Coordinator.

So I decided to move the Upgrade Coordinator to another NSX-T Node (you can do this as long all NSX-T Nodes in the Cluster status is sync).

How to do this?

Login to another NSX-T Node and run the following commands.

After going back to the new NSX-T Node Upgrade Coordinator and in the Upgrade tab, all was good and green.

After the last fix, all is ok, and all NSX-T shows upgrade to NSX-T 3.0.

I will finish How to upgrade NSX-T 2.4.x or 2.5.x to NSX-T 3.0 blog post with some images of the new NSX-T 3.0 GUI.

The overview is a little different. But more comfortable to read.

Also, now we can deploy a new NSX Intelligence Appliance.

What is the NSX Intelligence Appliance?

“A Distributed Security and Network Analytics Engine for your Data Center

Empower network and application security teams to deliver granular security and segmentation posture, simplify policy compliance analysis, and streamline security operations with VMware NSX Intelligence, a distributed analytics engine built and managed natively within NSX.”

In the Alarms section, we can now see all alarms clearly and can check all the issues in all NSX Infrastructure.

In Monitoring Dashboards, there are also some changes.

Main Dashboard

System Dashboard

And the new GUI I finish this blog post about How to upgrade NSX-T 2.4.x or 2.5.x to NSX-T 3.0 and hope this blog post was useful for your NSX-T Infrastructure upgrade to 3.0.

More articles about NSX-T:

NSX-T log partition full and not able to login

Other articles in the VMware upgrades series:

• How to upgrade vCenter Appliance 6.5 to 6.7 and fix upgrade errors
• How to upgrade vCenter Appliance 6.7 to 7.0 and fix upgrade errors
• How to upgrade vSphere 6.x Clusters to vSphere 7
• How to add extra space to vCenter for the upgrade
• How to upgrade NSX-T 2.4.x or 2.5.x to NSX-T 3.0
• How to upgrade NSX to vSphere 7.0

Note: I will update the links as I write the articles.

Share this article if you think it is worth sharing. If you have any questions or comments, comment here or contact me on Twitter.

©2020 ProVirtualzone. All Rights Reserved