How to Install ESXi Nested Inside Proxmox VE

Running VMware ESXi nested inside Proxmox VE is perfect for testing vSphere features, learning VMware technologies, or building a homelab. This updated guide for 2026 covers Proxmox VE 8.x and VMware ESXi 8.0 U3 with the correct virtual hardware configuration.

Prerequisites

• Proxmox VE 8.x host with Intel VT-x/AMD-V virtualization support
• ESXi 8.0 U3 ISO (download from VMware Customer Connect)
• Minimum 16GB RAM (32GB+ recommended for running nested VMs)
• SSD storage for better performance
• Sufficient CPU cores (8+ recommended)

Enable Nested Virtualization on Proxmox Host

Check Current Status

First, verify if nested virtualization is enabled:

# For Intel CPUs
cat /sys/module/kvm_intel/parameters/nested

# For AMD CPUs
cat /sys/module/kvm_amd/parameters/nested

If the output is N or 0, nested virtualization is disabled.

Enable Nested Virtualization

For Intel CPUs:

echo "options kvm-intel nested=Y" > /etc/modprobe.d/kvm-intel.conf
echo "options kvm-intel enable_shadow_vmcs=Y" >> /etc/modprobe.d/kvm-intel.conf
echo "options kvm-intel enable_apicv=Y" >> /etc/modprobe.d/kvm-intel.conf
echo "options kvm-intel unrestrict_guest=Y" >> /etc/modprobe.d/kvm-intel.conf

For AMD CPUs:

echo "options kvm-amd nested=1" > /etc/modprobe.d/kvm-amd.conf
echo "options kvm-amd avic=1" >> /etc/modprobe.d/kvm-amd.conf

Reload KVM Module

Stop all VMs and reload the KVM module:

# For Intel
modprobe -r kvm_intel
modprobe -r kvm
modprobe kvm
modprobe kvm_intel

# For AMD
modprobe -r kvm_amd
modprobe -r kvm
modprobe kvm
modprobe kvm_amd

Verify Configuration

# Should return 'Y' or '1'
cat /sys/module/kvm_intel/parameters/nested

# Check CPU flags in VMs
grep -E 'vmx|svm' /proc/cpuinfo

Create the ESXi Virtual Machine

Download ESXi ISO

Download ESXi 8.0 U3 from VMware and upload it to your Proxmox storage:

# Upload via SCP or use the Proxmox web UI
scp VMware-VMvisor-Installer-8.0U3-*.iso root@proxmox:/var/lib/vz/template/iso/

VM Configuration Options

Create a new VM through the Proxmox web UI with these critical settings:

General Settings

System Tab

CPU Tab

Memory Tab

Hard Disk Tab

Network Tab

Advanced: VM Configuration via CLI

For fine-tuned control, create the VM via command line:

qm create 100 --name esxi-nested-01 \
  --memory 16384 \
  --balloon 0 \
  --cpu host \
  --cores 4 \
  --sockets 1 \
  --numa 1 \
  --bios ovmf \
  --machine q35 \
  --scsihw pvscsi \
  --sata0 local:iso/VMware-VMvisor-Installer-8.0U3-*.iso,media=cdrom \
  --sata1 local-lvm:128,ssd=1,discard=on \
  --boot order=sata0;sata1 \
  --net0 virtio,bridge=vmbr0,mtu=1500 \
  --agent enabled=1 \
  --ostype l26

Required VM Configuration Changes

After creating the VM, edit /etc/pve/qemu-server/100.conf and add:

# Enable nested virtualization
args: -cpu host,+vmx

# Or for AMD:
# args: -cpu host,+svm

# Hide KVM hypervisor signature (optional, can help with ESXi detection)
hidden: 1

# CPU flags for better compatibility
cpu: host,flags=+vmx

Install ESXi

Boot the Installer

1. Start the VM and open the console
2. ESXi installer should boot automatically
3. Press Enter to continue through the welcome screen

Installation Steps

1. Accept EULA: Press F11
2. Select Disk: Choose the SATA disk you configured
3. Keyboard Layout: Select your preferred layout (usually US Default)
4. Root Password: Set a strong root password
5. Confirm Installation: Press F11 to begin

Post-Installation

After installation completes:

1. Remove the ISO from the CD-ROM
2. Reboot the VM
3. ESXi will boot and obtain an IP via DHCP
4. Note the IP address from the console

Configure ESXi

Initial Network Configuration

Press F2 to access the Direct Console User Interface (DCUI):

1. Configure Management Network
2. IP Configuration: Set static IP or keep DHCP
3. DNS Configuration: Set hostname and DNS servers
4. Troubleshooting Options: Enable SSH for remote management

Access vSphere Client

From your workstation, navigate to https://<esxi-ip>/ui/ and log in with root credentials.

Install VMware Tools (Optional)

For better integration, you can install VMware Tools in nested VMs, but note that QEMU Guest Agent may work better for Proxmox-managed operations.

Performance Optimization

Proxmox Host Tuning

# Enable IOMMU for Intel (in /etc/default/grub)
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

# Enable IOMMU for AMD (in /etc/default/grub)
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on"

# Update GRUB and reboot
update-grub
reboot

ESXi VM Tuning

Add these to the VM configuration for better performance:

qm set 100 --args '-cpu host,+vmx,+vpid,+vapic,+tsc-adjust'
qm set 100 --machine q35
qm set 100 --cpu cputype=host

Storage Optimization

# For SSD storage, enable discard
qm set 100 --sata1 local-lvm:vm-100-disk-1,ssd=1,discard=on

# Use VirtIO SCSI with discard for better performance
qm set 100 --scsihw virtio-scsi-pci
qm set 100 --scsi0 local-lvm:vm-100-disk-1,ssd=1,discard=on,iothread=1

Troubleshooting

ESXi Installer Doesn’t See Disk

Problem: The ESXi installer reports “No disks found”

Solution:

• Ensure you’re using SATA controller for the disk (not SCSI or VirtIO)
• Set SCSI controller to VMware PVSCSI
• Use q35 machine type

ESXi Boots to Purple Screen

Problem: PSOD (Purple Screen of Death) on boot

Solution:

• Ensure nested virtualization is enabled on the Proxmox host
• Add +vmx or +svm CPU flags to the VM config
• Increase memory allocation (minimum 8GB for ESXi 8.x)

Network Connectivity Issues

Problem: ESXi management network not working

Solution:

• Use vmxnet3 network adapter
• Ensure the Proxmox bridge is properly configured
• Check VLAN settings if using VLANs

Poor Performance

Problem: Nested VMs run slowly

Solution:

• Enable hardware passthrough if available
• Use SSD storage
• Increase CPU and memory allocation
• Enable hugepages on Proxmox host

Running Nested VMs

Once ESXi is installed and configured:

1. Create Port Groups: Set up virtual switches and port groups
2. Upload ISOs: Upload guest OS ISOs to ESXi datastore
3. Create VMs: Create nested VMs through the vSphere Client
4. Install VMware Tools: For better performance in nested VMs

Example: Creating a VM via ESXi CLI

# SSH to ESXi and list datastores
esxcli storage filesystem list

# Create VM registration
vim-cmd solo/registervm /vmfs/volumes/datastore1/vmname/vmname.vmx

Backup and Snapshots

Proxmox Level

You can backup the entire ESXi VM using Proxmox Backup Server or vzdump:

# Create backup
vzdump 100 --storage backup-storage --mode snapshot

# Create snapshot
qm snapshot 100 pre-upgrade

ESXi Level

Use vSphere’s built-in snapshot functionality for nested VMs, but be aware of the performance implications of nested snapshots.

Security Considerations

1. Network Isolation: Place ESXi management on a separate VLAN
2. Firewall Rules: Restrict access to ESXi management interface
3. SSH Access: Disable SSH when not in use
4. Lockdown Mode: Enable ESXi lockdown mode for production
5. Regular Updates: Keep both Proxmox and ESXi updated

Resource Requirements Summary