/Critical vulnerability in Apache Log4j apply workaround for vCenter and NSX-T

Critical vulnerability in Apache Log4j apply workaround for vCenter and NSX-T


In the last two days, the IT community has been on the alert because of a severe new security threat using when Apache Log4j, Which almost every software/tool out there uses this open-source. In this Critical vulnerability in Apache Log4j apply workaround for vCenter and NSX-T. We will go through a quick review on VMware impact and how to apply the workaround

First, you can have all the information about this security issue on the Apache Software Foundation page. They provide all the information about this threat and update with new information. So check the page to have all the updates on this security issue.

But lot us focus on VMware products that are using Log4j and how we can apply the workaround.

Until now, VMware did not launch any patch or an update to address this issue, only some workarounds. You may say that is weird, but if you think the problems that they with the last updates launched recently, we understand that they may be more careful before launching any new update.

Note: Only a few products has already a patch to fix this security issue.

This is, of course, a change of policy from VMware since they have separated from Dell. Because before new updates, new versions were mandatory to be launched every 6 months (or quarter), and that is stupid and huge pressure on the Dev teams. That is why the decision to remove all broken versions of vSphere 7 was made days after the VMware / Dell separation. Coincidence? No!

Backup to Apache Log4j security issue, in this blog post, we will focus on the workaround on vCenter and NSX-T. Those are the systems that I needed to do in the last two days.

For the complete VMware product list affected by Apache Log4j, check VMSA-2021-0028.2

A quick list of the impacted VMware products:

  • VMware Horizon
  • VMware vCenter Server
  • VMware HCX
  • VMware NSX-T Data Center
  • VMware Unified Access Gateway
  • VMware WorkspaceOne Access
  • VMware Identity Manager
  • VMware vRealize Operations
  • VMware vRealize Operations Cloud Proxy
  • VMware vRealize Automation
  • VMware vRealize Lifecycle Manager
  • VMware Site Recovery Manager, vSphere Replication
  • VMware Carbon Black Cloud Workload Appliance
  • VMware Carbon Black EDR Server
  • VMware Tanzu GemFire
  • VMware Tanzu Greenplum
  • VMware Tanzu Operations Manager
  • VMware Tanzu Application Service for VMs
  • VMware Tanzu Kubernetes Grid Integrated Edition
  • VMware Tanzu Observability by Wavefront Nozzle
  • Healthwatch for Tanzu Application Service
  • Spring Cloud Services for VMware Tanzu
  • Spring Cloud Gateway for VMware Tanzu
  • Spring Cloud Gateway for Kubernetes
  • API Portal for VMware Tanzu
  • Single Sign-On for VMware Tanzu Application Service
  • App Metrics
  • VMware vCenter Cloud Gateway
  • VMware vRealize Orchestrator
  • VMware Cloud Foundation
  • VMware Workspace ONE Access Connector
  • VMware Horizon DaaS
  • VMware Horizon Cloud Connector
  • VMware NSX Data Center for vSphere
  • VMware AppDefense Appliance
  • VMware Cloud Director Object Storage Extension
  • VMware Telco Cloud Operations
  • VMware vRealize Log Insight
  • VMware Tanzu Scheduler
  • VMware Smart Assurance NCM
  • VMware Smart Assurance SAM [Service Assurance Manager]
  • VMware Integrated OpenStack
  • VMware vRealize Business for Cloud
  • VMware vRealize Network Insight

As we can see above, a lot of products are affected by this issue. You can find the proper workaround for each VMware product in the above link.

What kind of security issue can this threat be on your environment?

A malicious actor with network access to an impacted VMware product may exploit this issue to fully control the target system.

How to apply the workaround on vCenter?

We can do this manually by changing the java wrapper file or using a python script provided by VMware. I will show how to use the script.

First, download the script vmsa-2021-0028-kb87081.py from KB87088

Next, upload the file to your vCenter appliance.

Use a tool like WinSCP to upload the file to your VCSA.

Note: bash needs to be set to root to login using VSCA.

After you upload the phyton script to your VCSA, login, and start the script, all the changes will be automatic. The script will stop vCenter services, apply the changes, and restart the services again.

When it finishes, double-check if all remediation tasks were completed with success.

I have applied this workaround in 6 vCenters(version 6.5, 6.7, and 7.0), and only one did not finish properly. Remediation did complete with success, but services were frozen. So I needed to reboot the vCenter, and it was ok. So I decided to reboot all vCenter after the changes.

Note: For the save side, do a snapshot of the vCenter before applying the script. If anything goes wrong, you can always rollback.

NEW UPDATE: 16/12/2021

###############################

Apache Foundation launched some updates on this security issue and some of the previous mitigation tasks don’t protect 100% our systems. So VMware has launched some extra changes to apply to our vCenter.

Check the new update in: KB87081

Note from VMware: “We believe the instructions in this article to be an effective mitigation for CVE-2021-44228, but in the best interest of our customers we must assume this workaround may not adequately address all attack vectors.”

For vCenter there is still no full script for the new changes, so we need to do it manually. There is one script for Analytics Service only.

You can download the script on the above KB page.

Download the script and follow the same instructions for the previous script.

For the rest of the changes, I have created a script to make these changes.

Note: At the moment I only add changes for vCenter 7.0.2/7.0.3 and 7.0.0. The script will do the changes for those versions, 6.7/6,5 and 6.0, I did not add to the script. If VMware doesn’t provide a script in the next few days, I will update mine and add more versions. The script was created in a couple of hours, so please bear with me if is not the perfect and the cleanest script 🙂

Please use the script carefully and always do a snapshot of your vCenter.

Copy past the script to file .sh. Use vi or another editor to create the file, then change permissions so that you can run the script, for example: chmod 777 script.sh, then run it using sh script.sh.

Check if all services and changes are green.

If you found any issues or errors in the script, please send me a message.

I hope this helps.

###############################

How to apply the workaround on NSX-T?

NEW UPDATE: 16/12/2021 – No updates for NSX-T. The next steps are up to date.

There is no phyton script for this. You need to do it manually. Or if you have many NSX-T do apply, you can create your own to do these tasks 😉

First, before starting any changes, check your NSX-T Cluster nodes to see if all are up and green.

You can check this in your NSX-T Manager GUI or in the console using the command get cluster status.

Login to NSX-T manager using admin user and run the get cluster status.

Critical vulnerability in Apache Log4j apply workaround for vCenter and NSX-T

If all is green and all nodes are UP, start the mitigation workaround tasks.

Note: Exist domain area to run the following commands(needs root access).

Next, create a new folder:

Copy al wrapper config files to the new folder.

Critical vulnerability in Apache Log4j apply workaround for vCenter and NSX-T

Before the change and to check if all is ok and track the changes, I search all the wrapper config files and tested if the new line was already on the files(selected random).

No results, so the new line is not on the files, now we apply the line to all wrapper config files and I will double check if the line is now added to the wrapper config files.

Critical vulnerability in Apache Log4j apply workaround for vCenter and NSX-T

All is good, so I can reboot the node /sbin/reboot

When a node is back, double-check your NSX-T Manager Cluster again, if all is up and running green, start the same process in the next NSX-T node. Do the same process for all NSX-T Nodes in your NSX-T Cluster.

Note: According to VMware, Edges are not affected by the Apache Log4j, so you don’t need to do any changes to your NSX Edges.

For more information about this workaround, check KB87086

I hope this Critical vulnerability in Apache Log4j apply workaround for vCenter and NSX-T blog post helps you to apply the workaround in your VMware environment.

Share this article if you think it is worth sharing. If you have any questions or comments, comment here, or contact me on Twitter.

©2021 ProVirtualzone. All Rights Reserved
By | 2021-12-17T04:18:31+01:00 December 15th, 2021|NSX, VMware, vSphere|0 Comments

About the Author:

I am over 20 years’ experience in the IT industry. Working with Virtualization for more than 10 years (mainly VMware). I am an MCP, VCP6.5-DCV, VMware vSAN Specialist, Veeam Vanguard 2018/2019, vExpert vSAN 2018/2019 and vExpert for the last 4 years. Specialties are Virtualization, Storage, and Virtual Backups. I am working for Elits a Swedish consulting company and allocated to a Swedish multinational networking and telecommunications company as a Teach Lead and acting as a Senior ICT Infrastructure Engineer. I am a blogger and owner of the blog ProVirtualzone.com

Leave a Reply

%d bloggers like this: