Chef Products & Log4j Vulnerability

Though Chef doesn’t use log4j, it remains affected by Log4j used by Elastic Search in the Chef Products. Because of this, Chef Backend which uses Elasticsearch 5.6.16 is affected by this Vulnerability which may need mitigation steps as described below after the table, whereas Chef Infra Server and Chef Automate uses Elastic Search 6.x so it doesn’t require any mitigation as Elastic Search has confirmed these versions of Elastic Search are not affected by this Vulnerability.

Steps for Chef Backend 2.2.0 Mitigation : These are the steps that needs to be carried out for Chef Backend mitigation

Step 1: Verify to the Chef Backend cluster and ensure that is healthy by running the below commands

chef-backend-ctl status

Step 2: Output will be similar to this

chef-backend-ctl cluster-status

Step 3: Add the following line to the end of the “/etc/chef-backend/chef-backend.rb” file:

elasticsearch.jvm_opts = [ "-Dlog4j2.formatMsgNoLookups=true" ]  

Then run the following command

sudo chef-backend-ctl reconfigure 

Step 4: Verify the change has been applied by running the following command

ps axww | grep log4j2.formatMsgNoLookups

Step 5: Then again verify whether the cluster nodes are in good condition by running the commands in Step 1 & Step2

elasticsearch.jvm_opts = [ "-Dlog4j2.formatMsgNoLookups=true" ] 







$ sudo chef-backend-ctl status 
Service        Local Status        Time in State   Distributed Node Status 
leaderl        running (pid 2818)  4d 19h 58m 6s   leader: 1; waiting: 0; follower: 2; total: 3 
epmd           running (pid 2641)  4d 19h 58m 23s  status: local-only 
etcd           running (pid 2552)  4d 19h 58m 31s  health: green; healthy nodes: 3/3 
postgresql     running (pid 5264)  4d 19h 47m 2s   leader: 1; offline: 0; syncing: 0; synced: 2 
elasticsearch  running (pid 2700)  4d 19h 58m 18s  state: green; nodes online: 3/3 

System  Local Status                                          Distributed Node Status 
disks   /var/log/chef-backend: OK; /var/opt/chef-backend: OK  health: green; healthy nodes: 3/3 

$ sudo chef-backend-ctl cluster-status 
Name            IP           GUID                              Role      PG        ES          Blocked      Eligible 
ip-10-0-12-194  10.0.12.194  a0db9989fe02e36fe396251c32669427  follower  follower  master      not_blocked  true 
ip-10-0-10-34   10.0.10.34   4549c51f96bada0e02a1fd664645b462  follower  follower  not_master  not_blocked  true 
ip-10-0-8-11    10.0.8.11    076a56710dfe1b611ab74d131a6ffe2d  leader    leader    not_master  not_blocked  true 

### Update the configuration on the followers 

Verify that the cluster node you are currently logged in to is NOT the one that is identified as the 'leader' in the 'Role' column of the 'sudo chef-backend-ctl cluster-status' output. That node will be updated last. It is okay if the follower node is the PG leader or ES master, those will be moved. 

Add the following line to the end of the "/etc/chef-backend/chef-backend.rb" file: 
elasticsearch.jvm_opts = [ "-Dlog4j2.formatMsgNoLookups=true" ] 

Run the following command to apply the configuration change: 
$ sudo chef-backend-ctl reconfigure 

Verify that the change has been applied by running this command and verifying that '-Dlog4j2.formatMsgNoLookups=true' was added to the java process as shown below: 
$ ps axww | grep log4j2.formatMsgNoLookups 
21505 ?        Ssl    0:12 /opt/chef-backend/embedded/open-jre/bin/java -Dlog4j2.formatMsgNoLookups=true -Xmx1292m -Xms1292m -XX:NewSize=80M -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -Djava.io.tmpdir=/var/opt/chef-backend/elasticsearch/tmp -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/opt/chef-backend/embedded/elasticsearch -cp /opt/chef-backend/embedded/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -Epath.conf=/var/opt/chef-backend/elasticsearch/config 

Run the following two commands to re-verify that the cluster status is still healthy: 
$ sudo chef-backend-ctl status 
$ sudo chef-backend-ctl cluster-status 

Repeat these steps on the other follower node 

### Update the configuration on the leader last 

Log in to the node identified as the 'leader' in the 'Role' column of the 'sudo chef-backend-ctl cluster-status' output. 

Verify the cluster is healthy: 
$ sudo chef-backend-ctl status 
$ sudo chef-backend-ctl cluster-status 

Demote the leader, then confirm that another node is now the leader, and the cluster is still healthy 
$ sudo chef-backend-ctl demote 
$ sudo chef-backend-ctl status 
$ sudo chef-backend-ctl cluster-status 

Now repeat all the steps listed in the previous section on this node. 

The mitigation process is now complete.