Critical vulnerability in Apache Log4j apply workaround for vCenter and NSX-T < ProVirtualzone

In the last two days, the IT community has been on the alert because of a severe new security threat using when Apache Log4j, Which almost every software/tool out there uses this open-source. In this Critical vulnerability in Apache Log4j apply workaround for vCenter and NSX-T. We will go through a quick review on VMware impact and how to apply the workaround

First, you can have all the information about this security issue on the Apache Software Foundation page. They provide all the information about this threat and update with new information. So check the page to have all the updates on this security issue.

But lot us focus on VMware products that are using Log4j and how we can apply the workaround.

Until now, VMware did not launch any patch or an update to address this issue, only some workarounds. You may say that is weird, but if you think the problems that they with the last updates launched recently, we understand that they may be more careful before launching any new update.

Note: Only a few products has already a patch to fix this security issue.

This is, of course, a change of policy from VMware since they have separated from Dell. Because before new updates, new versions were mandatory to be launched every 6 months (or quarter), and that is stupid and huge pressure on the Dev teams. That is why the decision to remove all broken versions of vSphere 7 was made days after the VMware / Dell separation. Coincidence? No!

Backup to Apache Log4j security issue, in this blog post, we will focus on the workaround on vCenter and NSX-T. Those are the systems that I needed to do in the last two days.

For the complete VMware product list affected by Apache Log4j, check VMSA-2021-0028.2

A quick list of the impacted VMware products:

VMware Horizon
VMware vCenter Server
VMware HCX
VMware NSX-T Data Center
VMware Unified Access Gateway
VMware WorkspaceOne Access
VMware Identity Manager
VMware vRealize Operations
VMware vRealize Operations Cloud Proxy
VMware vRealize Automation
VMware vRealize Lifecycle Manager
VMware Site Recovery Manager, vSphere Replication
VMware Carbon Black Cloud Workload Appliance
VMware Carbon Black EDR Server
VMware Tanzu GemFire
VMware Tanzu Greenplum
VMware Tanzu Operations Manager
VMware Tanzu Application Service for VMs
VMware Tanzu Kubernetes Grid Integrated Edition
VMware Tanzu Observability by Wavefront Nozzle
Healthwatch for Tanzu Application Service
Spring Cloud Services for VMware Tanzu
Spring Cloud Gateway for VMware Tanzu
Spring Cloud Gateway for Kubernetes
API Portal for VMware Tanzu
Single Sign-On for VMware Tanzu Application Service
App Metrics
VMware vCenter Cloud Gateway
VMware vRealize Orchestrator
VMware Cloud Foundation
VMware Workspace ONE Access Connector
VMware Horizon DaaS
VMware Horizon Cloud Connector
VMware NSX Data Center for vSphere
VMware AppDefense Appliance
VMware Cloud Director Object Storage Extension
VMware Telco Cloud Operations
VMware vRealize Log Insight
VMware Tanzu Scheduler
VMware Smart Assurance NCM
VMware Smart Assurance SAM [Service Assurance Manager]
VMware Integrated OpenStack
VMware vRealize Business for Cloud
VMware vRealize Network Insight

As we can see above, a lot of products are affected by this issue. You can find the proper workaround for each VMware product in the above link.

What kind of security issue can this threat be on your environment?

A malicious actor with network access to an impacted VMware product may exploit this issue to fully control the target system.

How to apply the workaround on vCenter?

We can do this manually by changing the java wrapper file or using a python script provided by VMware. I will show how to use the script.

First, download the script vmsa-2021-0028-kb87081.py from KB87088

Next, upload the file to your vCenter appliance.

Use a tool like WinSCP to upload the file to your VCSA.

Note: bash needs to be set to root to login using VSCA.

After you upload the phyton script to your VCSA, login, and start the script, all the changes will be automatic. The script will stop vCenter services, apply the changes, and restart the services again.

When it finishes, double-check if all remediation tasks were completed with success.

I have applied this workaround in 6 vCenters(version 6.5, 6.7, and 7.0), and only one did not finish properly. Remediation did complete with success, but services were frozen. So I needed to reboot the vCenter, and it was ok. So I decided to reboot all vCenter after the changes.

Note: For the save side, do a snapshot of the vCenter before applying the script. If anything goes wrong, you can always rollback.

NEW UPDATE: 16/12/2021

###############################

Apache Foundation launched some updates on this security issue and some of the previous mitigation tasks don’t protect 100% our systems. So VMware has launched some extra changes to apply to our vCenter.

Check the new update in: KB87081

Note from VMware: “We believe the instructions in this article to be an effective mitigation for CVE-2021-44228, but in the best interest of our customers we must assume this workaround may not adequately address all attack vectors.”

For vCenter there is still no full script for the new changes, so we need to do it manually. There is one script for Analytics Service only.

You can download the script on the above KB page.

Download the script and follow the same instructions for the previous script.

For the rest of the changes, I have created a script to make these changes.

Note: At the moment I only add changes for vCenter 7.0.2/7.0.3 and 7.0.0. The script will do the changes for those versions, 6.7/6,5 and 6.0, I did not add to the script. If VMware doesn’t provide a script in the next few days, I will update mine and add more versions. The script was created in a couple of hours, so please bear with me if is not the perfect and the cleanest script 🙂

Please use the script carefully and always do a snapshot of your vCenter.


#!/bin/sh
#
##############################################################
# Created by: Luciano Patrão #
# Date: 16-12-2021 #
# Updated: #
# Mitigation changes for vCenter to address CVE-2021-44228 #
##############################################################
clear
vCenter=$(rpm -qa | egrep 'VMware-vpxd-6|VMware-vpxd-7');
Version=${vCenter: +12:5};
if [ $Version = 7.0.2 -o $Version = 7.0.3 ];
then
echo -e '\033[1;94mvCenter 7.02/03'
echo -e '\033[1;32m##### Starting metigation on vCenter '$Version' ######' && echo
#vMON Service
echo -e '\033[0;32mStart changes on vMon Service' && echo
cp -rfp /usr/lib/vmware-vmon/java-wrapper-vmon /usr/lib/vmware-vmon/java-wrapper-vmon.bak
sed -i 's/exec $java_start_bin $jvm_dynargs $security_dynargs $original_args/log4j_arg="-Dlog4j2.formatMsgNoLookups=true"/g' /usr/lib/vmware-vmon/java-wrapper-vmon
echo 'exec $java_start_bin $jvm_dynargs $log4j_arg $security_dynargs $original_args' >> /usr/lib/vmware-vmon/java-wrapper-vmon
echo -e '\033[1;37m....Changing java-wrapper-vmon file permissions' && echo
chown root:cis /usr/lib/vmware-vmon/java-wrapper-vmon
chmod 754 /usr/lib/vmware-vmon/java-wrapper-vmon
echo -e '\033[1;37m....Stopping and starting all services, this can take some time, please be patient' && echo
service-control --stop --all && echo && echo -e '\033[0;32m....Starting Services' && echo && service-control --start --all && echo && echo -e '\033[1;32m#####..... All services are running ......#####'
#sleep 60s
#service-control --start --all
sleep 2s

#Update Manager Service
echo
echo -e '\033[0;32mStart changes on Update Manager Service'
cp -rfp /usr/lib/vmware-updatemgr/bin/jetty/start.ini /usr/lib/vmware-updatemgr/bin/jetty/start.ini.bak
echo '-Dlog4j2.formatMsgNoLookups=true' >> /usr/lib/vmware-updatemgr/bin/jetty/start.ini

sleep 2s
##Analytics Service
# echo -e '\033[0;32mStart changes on Analytics Service'
# cp -rfp /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar.bak
# zip -q -d /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

elif [ $Version = 7.0.0 ]
then
echo -e '\033[1;94mvCenter Version 7.0.0 - Real '$Version'';
echo -e '\033[1;32m##### Starting metigation on vCenter '$Version' ######' && echo
#vMON Service
echo -e '\033[0;32mStart changes on vMon Service'
cp -rfp /usr/lib/vmware-vmon/java-wrapper-vmon /usr/lib/vmware-vmon/java-wrapper-vmon.bak
sed -i 's/exec $java_start_bin $jvm_dynargs "$@"/log4j_arg="-Dlog4j2.formatMsgNoLookups=true"/g' /usr/lib/vmware-vmon/java-wrapper-vmon
echo 'exec $java_start_bin $jvm_dynargs $log4j_arg "$@"' >> /usr/lib/vmware-vmon/java-wrapper-vmon
echo -e '\033[1;37m... changing java-wrapper-vmon file permissions'
chown root:cis /usr/lib/vmware-vmon/java-wrapper-vmon
chmod 754 /usr/lib/vmware-vmon/java-wrapper-vmon
echo -e '\033[1;37mStopping and starting all services, this can take some time'
service-control --stop --all && service-control --start --all && echo -e '\033[0;32mStart changes on Update Manager Service' && echo
fi
###Start Services and validation

echo -e '\033[0;32mRestarting Update Manager Service.........' && echo
service-control --restart vmware-updatemgr
SERVICE=vmware-updatemgr;
sleep 3s
if ps ax | grep -v grep | grep $SERVICE > /dev/null
then
echo -e '\033[0;32m......'$SERVICE' service running, SUCCESS' && echo
else
echo -e '\033[0;31m......'$SERVICE' is not running, ERROR!' && echo
fi

#echo -e '\033[0;32mRestarting Analytics Service.........'
#service-control --restart vmware-analytics
#SERVICE=vmware-analytics;
#sleep 3s
#if ps ax | grep -v grep | grep $SERVICE > /dev/null
#then
# echo -e '\033[0;32m......'$SERVICE' service running, SUCCESS'
#else
# echo -e '\033[0;31m....'$SERVICE' is not running, ERROR!'
#fi

###Start Validation
echo -e '\033[0;32mVerify the Analytics Service changes'
VAR_AS="$(grep -i jndilookup /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar | wc -l)";
if [ "$VAR_AS" = 0 ];
then
echo -e '\033[0;32m.... Metigation was a SUCCESS' && echo
else
echo -e '\033[0;31m.... Metigation has an ERROR!' && echo
fi
#######
echo -e '\033[0;32mVerify the Update Manager change \033[0;37m' && echo
cd /usr/lib/vmware-updatemgr/bin/jetty/
VAR_VUM=$(java -jar start.jar --list-config | grep 'log4j2.formatMsgNoLookups = true (/usr/lib/vmware-updatemgr/bin/jetty/start.ini)')
if [ -z "$VAR_VUM" ]
then
echo -e '\033[0;31m.... Metigation has an ERROR!' && echo && cd /
else
echo -e '\033[0;32m.... Metigation was a SUCCESS' && echo && cd /
fi

100

101

#!/bin/sh

##############################################################

# Created by: Luciano Patrão #

# Date: 16-12-2021 #

# Updated: #

# Mitigation changes for vCenter to address CVE-2021-44228 #

##############################################################

clear

vCenter=$(rpm -qa | egrep 'VMware-vpxd-6|VMware-vpxd-7');

Version=${vCenter: +12:5};

if [ $Version = 7.0.2 -o $Version = 7.0.3 ];

then

echo -e '\033[1;94mvCenter 7.02/03'

echo -e '\033[1;32m##### Starting metigation on vCenter '$Version' ######' && echo

#vMON Service

echo -e '\033[0;32mStart changes on vMon Service' && echo

cp -rfp /usr/lib/vmware-vmon/java-wrapper-vmon /usr/lib/vmware-vmon/java-wrapper-vmon.bak

sed -i 's/exec $java_start_bin $jvm_dynargs $security_dynargs $original_args/log4j_arg="-Dlog4j2.formatMsgNoLookups=true"/g' /usr/lib/vmware-vmon/java-wrapper-vmon

echo 'exec $java_start_bin $jvm_dynargs $log4j_arg $security_dynargs $original_args' >> /usr/lib/vmware-vmon/java-wrapper-vmon

echo -e '\033[1;37m....Changing java-wrapper-vmon file permissions' && echo

chown root:cis /usr/lib/vmware-vmon/java-wrapper-vmon

chmod 754 /usr/lib/vmware-vmon/java-wrapper-vmon

echo -e '\033[1;37m....Stopping and starting all services, this can take some time, please be patient' && echo

service-control --stop --all && echo && echo -e '\033[0;32m....Starting Services' && echo && service-control --start --all && echo && echo -e '\033[1;32m#####..... All services are running ......#####'

#sleep 60s

#service-control --start --all

sleep 2s

#Update Manager Service

echo

echo -e '\033[0;32mStart changes on Update Manager Service'

cp -rfp /usr/lib/vmware-updatemgr/bin/jetty/start.ini /usr/lib/vmware-updatemgr/bin/jetty/start.ini.bak

echo '-Dlog4j2.formatMsgNoLookups=true' >> /usr/lib/vmware-updatemgr/bin/jetty/start.ini

sleep 2s

##Analytics Service

# echo -e '\033[0;32mStart changes on Analytics Service'

# cp -rfp /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar.bak

# zip -q -d /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

elif [ $Version = 7.0.0 ]

then

echo -e '\033[1;94mvCenter Version 7.0.0 - Real '$Version'';

echo -e '\033[1;32m##### Starting metigation on vCenter '$Version' ######' && echo

#vMON Service

echo -e '\033[0;32mStart changes on vMon Service'

cp -rfp /usr/lib/vmware-vmon/java-wrapper-vmon /usr/lib/vmware-vmon/java-wrapper-vmon.bak

sed -i 's/exec $java_start_bin $jvm_dynargs "$@"/log4j_arg="-Dlog4j2.formatMsgNoLookups=true"/g' /usr/lib/vmware-vmon/java-wrapper-vmon

echo 'exec $java_start_bin $jvm_dynargs $log4j_arg "$@"' >> /usr/lib/vmware-vmon/java-wrapper-vmon

echo -e '\033[1;37m... changing java-wrapper-vmon file permissions'

chown root:cis /usr/lib/vmware-vmon/java-wrapper-vmon

chmod 754 /usr/lib/vmware-vmon/java-wrapper-vmon

echo -e '\033[1;37mStopping and starting all services, this can take some time'

service-control --stop --all && service-control --start --all && echo -e '\033[0;32mStart changes on Update Manager Service' && echo

###Start Services and validation

echo -e '\033[0;32mRestarting Update Manager Service.........' && echo

service-control --restart vmware-updatemgr

SERVICE=vmware-updatemgr;

sleep 3s

if ps ax | grep -v grep | grep $SERVICE > /dev/null

then

echo -e '\033[0;32m......'$SERVICE' service running, SUCCESS' && echo

else

echo -e '\033[0;31m......'$SERVICE' is not running, ERROR!' && echo

#echo -e '\033[0;32mRestarting Analytics Service.........'

#service-control --restart vmware-analytics

#SERVICE=vmware-analytics;

#sleep 3s

#if ps ax | grep -v grep | grep $SERVICE > /dev/null

#then

# echo -e '\033[0;32m......'$SERVICE' service running, SUCCESS'

#else

# echo -e '\033[0;31m....'$SERVICE' is not running, ERROR!'

#fi

###Start Validation

echo -e '\033[0;32mVerify the Analytics Service changes'

VAR_AS="$(grep -i jndilookup /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar | wc -l)";

if [ "$VAR_AS" = 0 ];

then

echo -e '\033[0;32m.... Metigation was a SUCCESS' && echo

else

echo -e '\033[0;31m.... Metigation has an ERROR!' && echo

#######

echo -e '\033[0;32mVerify the Update Manager change \033[0;37m' && echo

cd /usr/lib/vmware-updatemgr/bin/jetty/

VAR_VUM=$(java -jar start.jar --list-config | grep 'log4j2.formatMsgNoLookups = true (/usr/lib/vmware-updatemgr/bin/jetty/start.ini)')

if [ -z "$VAR_VUM" ]

then

echo -e '\033[0;31m.... Metigation has an ERROR!' && echo && cd /

else

echo -e '\033[0;32m.... Metigation was a SUCCESS' && echo && cd /

Copy past the script to file .sh. Use vi or another editor to create the file, then change permissions so that you can run the script, for example: chmod 777 script.sh, then run it using sh script.sh.

Check if all services and changes are green.

If you found any issues or errors in the script, please send me a message.

I hope this helps.

###############################

How to apply the workaround on NSX-T?

NEW UPDATE: 16/12/2021 – No updates for NSX-T. The next steps are up to date.

There is no phyton script for this. You need to do it manually. Or if you have many NSX-T do apply, you can create your own to do these tasks 😉

First, before starting any changes, check your NSX-T Cluster nodes to see if all are up and green.

You can check this in your NSX-T Manager GUI or in the console using the command get cluster status.

If all is green and all nodes are UP, start the mitigation workaround tasks.

Note: Exist domain area to run the following commands(needs root access).

Next, create a new folder:


mkdir /root/tanuki-conf

mkdir /root/tanuki-conf

Copy al wrapper config files to the new folder.


cp -p /usr/tanuki/conf/*-wrapper.conf /root/tanuki-conf

cp -p /usr/tanuki/conf/*-wrapper.conf /root/tanuki-conf

Before the change and to check if all is ok and track the changes, I search all the wrapper config files and tested if the new line was already on the files(selected random).


root@nsx-t-domain:~# mkdir /root/tanuki-conf
root@nsx-t-domain:~# cp -p /usr/tanuki/conf/*-wrapper.conf /root/tanuki-conf/
root@nsx-t-domain:~# find /usr/tanuki/conf/ -name '*-wrapper.conf'
/usr/tanuki/conf/intelligence-upgrade-coordinator-tomcat-wrapper.conf
/usr/tanuki/conf/cluster-boot-manager-wrapper.conf
/usr/tanuki/conf/cm-inventory-tomcat-wrapper.conf
/usr/tanuki/conf/search-wrapper.conf
/usr/tanuki/conf/proxy-tomcat-wrapper.conf
/usr/tanuki/conf/corfu-log-replication-server-wrapper.conf
/usr/tanuki/conf/async-replicator-service-wrapper.conf
/usr/tanuki/conf/migration-coordinator-tomcat-wrapper.conf
/usr/tanuki/conf/upgrade-coordinator-tomcat-wrapper.conf
/usr/tanuki/conf/phonehome-coordinator-tomcat-wrapper.conf
/usr/tanuki/conf/corfu-server-wrapper.conf
/usr/tanuki/conf/corfu-nonconfig-server-wrapper.conf
/usr/tanuki/conf/proton-tomcat-wrapper.conf
/usr/tanuki/conf/idps-reporting-service-wrapper.conf
/usr/tanuki/conf/cross-cloud-upgrade-coordinator-tomcat-wrapper.conf
/usr/tanuki/conf/nsx-ccp-wrapper.conf
/usr/tanuki/conf/policy-tomcat-wrapper.conf
root@nsx-t-domain:~#
root@nsx-t-domain:~# cat /usr/tanuki/conf/intelligence-upgrade-coordinator-tomcat-wrapper.conf | grep "wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true"
root@nsx-t-domain:~#

root@nsx-t-domain:~# mkdir /root/tanuki-conf

root@nsx-t-domain:~# cp -p /usr/tanuki/conf/*-wrapper.conf /root/tanuki-conf/

root@nsx-t-domain:~# find /usr/tanuki/conf/ -name '*-wrapper.conf'

/usr/tanuki/conf/intelligence-upgrade-coordinator-tomcat-wrapper.conf

/usr/tanuki/conf/cluster-boot-manager-wrapper.conf

/usr/tanuki/conf/cm-inventory-tomcat-wrapper.conf

/usr/tanuki/conf/search-wrapper.conf

/usr/tanuki/conf/proxy-tomcat-wrapper.conf

/usr/tanuki/conf/corfu-log-replication-server-wrapper.conf

/usr/tanuki/conf/async-replicator-service-wrapper.conf

/usr/tanuki/conf/migration-coordinator-tomcat-wrapper.conf

/usr/tanuki/conf/upgrade-coordinator-tomcat-wrapper.conf

/usr/tanuki/conf/phonehome-coordinator-tomcat-wrapper.conf

/usr/tanuki/conf/corfu-server-wrapper.conf

/usr/tanuki/conf/corfu-nonconfig-server-wrapper.conf

/usr/tanuki/conf/proton-tomcat-wrapper.conf

/usr/tanuki/conf/idps-reporting-service-wrapper.conf

/usr/tanuki/conf/cross-cloud-upgrade-coordinator-tomcat-wrapper.conf

/usr/tanuki/conf/nsx-ccp-wrapper.conf

/usr/tanuki/conf/policy-tomcat-wrapper.conf

root@nsx-t-domain:~#

root@nsx-t-domain:~# cat /usr/tanuki/conf/intelligence-upgrade-coordinator-tomcat-wrapper.conf | grep "wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true"

root@nsx-t-domain:~#

No results, so the new line is not on the files, now we apply the line to all wrapper config files and I will double check if the line is now added to the wrapper config files.


root@nsx-t-domain:~# find /usr/tanuki/conf/ -name '*-wrapper.conf' | xargs -n 1 -I {} sh -c 'echo "wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true" >> {}'
root@nsx-t-domain:~# cat /usr/tanuki/conf/intelligence-upgrade-coordinator-tomcat-wrapper.conf | grep "wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true"
wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
root@nsx-t-domain:~# cat /usr/tanuki/conf/cross-cloud-upgrade-coordinator-tomcat-wrapper.conf | grep "wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true"
wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true
root@nsx-t-domain:~#

root@nsx-t-domain:~# find /usr/tanuki/conf/ -name '*-wrapper.conf' | xargs -n 1 -I {} sh -c 'echo "wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true" >> {}'

root@nsx-t-domain:~# cat /usr/tanuki/conf/intelligence-upgrade-coordinator-tomcat-wrapper.conf | grep "wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true"

wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true

root@nsx-t-domain:~# cat /usr/tanuki/conf/cross-cloud-upgrade-coordinator-tomcat-wrapper.conf | grep "wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true"

wrapper.java.additional.100=-Dlog4j2.formatMsgNoLookups=true

root@nsx-t-domain:~#

All is good, so I can reboot the node /sbin/reboot

When a node is back, double-check your NSX-T Manager Cluster again, if all is up and running green, start the same process in the next NSX-T node. Do the same process for all NSX-T Nodes in your NSX-T Cluster.

Note: According to VMware, Edges are not affected by the Apache Log4j, so you don’t need to do any changes to your NSX Edges.

For more information about this workaround, check KB87086

I hope this Critical vulnerability in Apache Log4j apply workaround for vCenter and NSX-T blog post helps you to apply the workaround in your VMware environment.

Share this article if you think it is worth sharing. If you have any questions or comments, comment here, or contact me on Twitter.

How to apply the workaround on vCenter?

How to apply the workaround on NSX-T?

©2021 ProVirtualzone. All Rights Reserved

About the Author: Luciano Patrao

Leave A Comment Cancel reply

Categories