# VMware vSphere Troubleshooting Basics

> Chapter 15 of the Courseiva VMWARE-VCP-DCV curriculum — https://courseiva.com/learn/vmware-vcp-dcv/vmware-troubleshooting-basics

**Official objective:** 5.3 — Troubleshoot common vSphere issues including hosts, networks, and storage

## Introduction

If you cannot troubleshoot basic vSphere issues, your virtual machines will go offline, your applications will crash, and your boss will blame you. Troubleshooting is not about memorising error codes but about following a logical process to find the root cause of a problem, whether it is a host that will not power on, a virtual machine that cannot connect to the network, or storage that has disappeared. For the VCP-DCV exam, you must understand this process because scenario-based questions will test your ability to diagnose and resolve common faults.

## The Restaurant Kitchen Manager Analogy

A head chef running a busy restaurant kitchen on a Saturday night. The head chef's job is to keep the kitchen running smoothly: making sure every station has ingredients, every oven is at the right temperature, and every order gets out on time. One evening, a waiter comes back saying the grill is smoking. The head chef doesn't panic. They first check if it's just a bit of grease or a real fire. They look at the temperature gauge, check the ventilation hood, and ask the line cook if they changed the settings. The head chef knows that a smoking grill could mean a dirty drip tray, a faulty thermostat, or just a steak that's too fatty. By isolating the symptom (the smoke) and checking the most likely causes in order, they fix it quickly without shutting down the whole kitchen. This is exactly how an IT professional troubleshoots a VMware vSphere environment. When a virtual machine goes offline or a host goes grey in the management tool, they don't guess. They follow a logical process: identify the symptom, isolate the affected component, check the obvious causes first (like network cables or storage paths), and then escalate to more complex possibilities like misconfigured settings or resource exhaustion. The head chef's calm, methodical approach to diagnosing a smoking grill maps precisely to the troubleshooting methodology used in vSphere: gather information, hypothesise, test, and confirm the fix before moving on.

## Core explanation

Troubleshooting in VMware vSphere is the systematic process of identifying, isolating, and resolving problems with virtual infrastructure. At its core, it is a logical method that every IT professional uses when something breaks. The goal is to restore normal operation as quickly as possible while understanding what went wrong to prevent it from happening again.

In a physical data centre, if a server fails, you might walk up to it, see a flashing red light, and swap a hard drive. But in a virtualised environment, the problems are often invisible. A virtual machine (VM) is a software computer that runs on a physical server called an ESXi host. ESXi is a type 1 hypervisor, which means it runs directly on the hardware without needing an operating system underneath. Multiple ESXi hosts are managed by a single vCenter Server, which gives you a central interface to control everything.

When something goes wrong, the first step is to identify the symptom. This could be an alert in the vSphere Client, a user reporting a slow VM, or a host showing as disconnected. You then gather information: check the host's hardware health, look at the VM's resource usage (CPU, memory, disk), and review recent changes. Common causes of issues fall into three categories: host problems, network problems, and storage problems.

Host problems include an ESXi host that won't boot, a host that has lost connectivity to vCenter, or a host that is running out of resources. The first thing to check is the physical hardware: is the server powered on? Are there any hardware error lights? Then check the network: can the host ping its default gateway? If the host is unresponsive, you may need to connect directly to its console via the iLO or iDRAC interface (these are remote management cards built into servers) to see why.

Network problems often manifest as VMs losing connectivity or being unable to communicate with each other. vSphere uses virtual switches, which are software bridges that connect VMs to the physical network. A misconfigured virtual switch port group, a wrong VLAN ID, or a physical network cable that is unplugged can cause issues. The troubleshooting process here is to trace the path: check the VM's virtual network interface, check the port group settings, check the physical uplink from the ESXi host to the physical switch, and then check the physical switch itself.

Storage problems can be the most critical because if storage fails, all VMs on that datastore may go down. A datastore is a storage container that holds VM files. Common issues include a datastore becoming inaccessible, high latency, or running out of space. The first step is to check the storage adapter on the ESXi host, then the SAN fabric (storage area network switches), and finally the storage array itself. vSphere provides tools like the vSphere Client's storage views and the `esxcli` command-line tool to diagnose these issues.

A structured troubleshooting methodology is called the 'scientific method' for IT: define the problem, gather information, hypothesise the cause, test the hypothesis, plan and implement a fix, verify the fix worked, and document the outcome. The VCP-DCV exam expects you to apply this method to scenario questions. For example, if a VM is slow, you should check resource contention (too many VMs on one host), storage latency, or network bottlenecks before assuming it is a hardware fault.

Troubleshooting also involves understanding logs. vSphere generates logs on each ESXi host (called 'hostd' and 'vmkernel' logs) and on the vCenter Server. These logs contain detailed error messages that can point directly to a problem. For instance, a 'All path down' error in the vmkernel log indicates that the host has lost all connections to a particular storage device.

Finally, remember that prevention is better than cure. vSphere has features like HA (High Availability) and DRS (Distributed Resource Scheduler) that help mitigate problems, but they do not replace good troubleshooting skills. Knowing the basics of troubleshooting will help you not only fix issues but also pass the VCP-DCV exam.

## Real-world context

Meet Priya, a junior IT administrator at a mid-sized company running a vSphere environment with 20 ESXi hosts and 300 virtual machines. One Monday morning, her manager calls: 'The finance department's database server is running extremely slowly. Users are complaining they cannot process invoices.' Priya needs to troubleshoot this immediately.

Priya first opens the vSphere Client and finds the VM named 'FIN-DB-01'. She looks at its performance charts. She sees that CPU usage is at 95 per cent and memory usage is at 98 per cent. This is unusual because this VM usually runs at 30 per cent CPU. She checks if any other VMs on the same host are also using high resources. She notices that a new development VM named 'DEV-TEST-07' is using 60 per cent of the host's CPU. This could be the issue: resource contention on the host is starving the database VM.


Now Priya needs to decide what to do. She has several options:
- She could migrate the database VM to another host using vMotion, which moves a running VM to a different host without downtime.
- She could increase the CPU and memory resources allocated to the VM, but that might not help if the host is full.
- She could contact the developer who owns 'DEV-TEST-07' and ask them to stop their resource-intensive test.


Priya chooses to use vMotion to move the database VM to a host that has spare capacity. She right-clicks the VM in the vSphere Client, selects 'Migrate', chooses the target host, and starts the migration. The migration completes in a few minutes, and the database VM's performance returns to normal. She then contacts the developer about the resource-heavy test VM to prevent future issues.

Later that week, another problem arises. A different VM, a web server, loses network connectivity. Users get 'page not found' errors. Priya again opens the vSphere Client and sees that the VM shows 'Disconnected' for its network adapter. She checks the VM's hardware settings and sees that the network adapter is connected to a port group called 'Production-Web'. She checks the port group configuration and realises that it has a VLAN ID of 200, but the physical switch port connected to the ESXi host is configured for VLAN 100. This mismatch means packets from the VM are going to the wrong VLAN and are being dropped. She changes the port group's VLAN ID to 100, and the VM immediately regains connectivity.

Priya's final step in both cases is to document what happened and what she did. She writes a short report: the symptom, the investigation steps, the root cause, and the fix. This documentation helps her team understand the environment better and can be used to train new administrators. In the real world, troubleshooting is not just about fixing the problem; it is about understanding the environment and improving it so similar issues do not recur.

## Exam focus

The VCP-DCV exam (exam code 2V0-21.23) tests troubleshooting in objective 5.3. This section is heavily scenario-based. You will be presented with a description of a problem, such as 'A user reports that a virtual machine cannot connect to the network. The VM has an IP address, but it cannot ping the default gateway.' then you must choose the correct troubleshooting step or the most likely cause from multiple-choice answers.

The exam loves to test your knowledge of the order of operations. A common trap is to jump to a complex solution when a simple one would suffice. For example, if a VM has no network connectivity, the first thing to check is whether the VM's network adapter is connected in the virtual hardware settings, not to reconfigure the virtual switch or restart the host. The correct answer pattern is to always check the simplest and most obvious cause first, working your way up to more complex possibilities.

Key concepts they test repeatedly include:
- The difference between a host problem and a VM problem. If one VM is down but other VMs on the same host are fine, it is likely a VM-specific issue (e.g., guest OS crash, misconfigured network adapter). If all VMs on a host are down, it is likely a host problem (e.g., host failure, storage connectivity loss).
- Storage path failures. They often test situations where storage is 'All Paths Down' (APD). You need to know that APD means the host has lost all connections to the storage device, and the first step is to check the physical cabling, then the SAN switches, then the storage array.
- vMotion failures. They might describe a vMotion that fails with an error. You need to know to check the compatibility of the two hosts (CPU compatibility, network connectivity, and storage accessibility from both hosts).
- vSphere HA errors. If a host goes down and VMs do not restart on other hosts, you need to check the HA configuration, especially the 'Host Isolation Response' setting, or whether the hosts have enough resources to accommodate the restarted VMs.
- Network troubleshooting. They love to test VLAN mismatches, misconfigured port groups, and physical uplink failures. You need to know that a VM can have an IP address but still fail to communicate if the VLAN configuration is wrong.

Exam traps include:
- Reading too much into the scenario. They will include extraneous detail that sounds important but is irrelevant. Focus on the direct symptoms.
- Confusing 'disconnected' state (vCenter cannot communicate with the host) with 'not responding' state (host is powered on but vCenter has lost contact). The troubleshooting steps differ.
- Assuming that restarting the vCenter Server is a good first step. It is almost never the first step for a host or VM problem unless vCenter itself is unresponsive.

To prepare, you should practice applying the troubleshooting methodology to written scenarios. Memorise the common log file names (e.g., 'vmkernel.log' for storage faults). Understand that 'esxcli' commands can diagnose many issues directly from the host command line. Finally, know the difference between proactive tools (like vSphere Health Check) and reactive tools (like log analysis).

## Step by step

1. **Identify the Symptom** — The first step is to clearly define what is wrong. Is a VM slow, unreachable, or completely down? Is a host showing as disconnected? You gather initial information from user reports, alerts in the vSphere Client, or monitoring tools. This step sets the direction for the entire troubleshooting process.
2. **Gather Information** — Collect data about the affected component. Check performance charts, recent events, logs, and configuration settings. For example, if a VM is slow, look at its CPU, memory, and disk usage in the vSphere Client. Also check the host it is running on to see if there is resource contention from other VMs.
3. **Form a Hypothesis** — Based on the information you have gathered, propose a likely root cause. For instance, if a VM is slow and the host it is on is running at 90 per cent CPU, your hypothesis might be that resource contention is the problem. If a VM cannot connect to the network, your hypothesis might be a VLAN mismatch.
4. **Test the Hypothesis** — Perform a test to confirm or rule out your hypothesis. In the resource contention example, you could vMotion the VM to a host with more free CPU and see if performance improves. In the VLAN mismatch example, you could check the port group configuration against the physical switch settings. A successful test gives you confidence in the root cause.
5. **Implement the Fix and Verify** — Apply the corrective action, such as changing a port group VLAN, migrating a VM, or restarting a host service. After the fix, verify that the symptom is resolved. For a VM speed issue, verify by checking performance charts and asking the user if the problem is gone. Then document everything for future reference.

## Comparisons

### Host Problem vs VM Problem

**Host Problem:**
- Affects all VMs on the host simultaneously
- Root cause is on the physical server (e.g., hardware failure, network outage)
- Troubleshoot by checking host hardware, management network, and storage connectivity

**VM Problem:**
- Affects only one or a few VMs while others on the same host remain functional
- Root cause is within the VM or its configuration (e.g., guest OS crash, misconfigured NIC)
- Troubleshoot by checking VM hardware settings, guest OS logs, and application configuration

### Disconnected State vs Not Responding State

**Disconnected State:**
- vCenter cannot communicate with the host, but the host may still be running VMs
- Often caused by a network issue between the host and vCenter
- Fix by checking network connectivity or restarting the host's management agents

**Not Responding State:**
- Host is powered on but not responding to any vCenter requests
- Often caused by a host service crash or resource exhaustion
- Fix by connecting directly to the host console or rebooting the host

### Storage All Paths Down vs Storage High Latency

**Storage All Paths Down:**
- Host has zero paths to storage, so VMs cannot read or write data
- Typically a physical connectivity or configuration issue (cable, switch, HBA)
- Immediate impact: VMs may crash or become unresponsive

**Storage High Latency:**
- Host has paths to storage but data access is slow
- Typically caused by storage array performance issues, network congestion, or too many I/O requests
- Immediate impact: VMs are slow but still functional

### vMotion vs Cold Migration

**vMotion:**
- Migrates a running VM to another host with zero downtime
- Requires shared storage between source and target hosts
- Fails if hosts do not have compatible CPUs or network access

**Cold Migration:**
- Migrates a powered-off VM by moving its files to a new datastore and registering on a new host
- Does not require shared storage; files are copied or moved
- Works between any hosts, regardless of CPU compatibility

## Diagram

_A flowchart showing the six-step troubleshooting methodology used in vSphere: identify symptom, gather information, form hypothesis, test hypothesis, implement and verify fix, and document._

```mermaid
flowchart TD
    A[Identify Symptom] --> B[Gather Information]
    B --> C[Form Hypothesis]
    C --> D[Test Hypothesis]
    D --> E{Is Hypothesis Correct?}
    E -- Yes --> F[Implement Fix]
    F --> G[Verify Fix]
    E -- No --> B
    G --> H[Document Outcome]
```

## Common misconceptions

- **Misconception:** If a virtual machine is slow, the first thing to do is to give it more CPU and memory. **Reality:** You should first identify the bottleneck. It could be storage latency, network congestion, or resource contention from other VMs on the same host. Adding resources to the VM might not help if the host itself is oversubscribed. (This mistake is common because increasing resources seems like a quick fix, but it treats the symptom instead of the root cause. In the exam and real life, the correct first step is to gather performance data and determine which resource is actually constrained.)
- **Misconception:** If an ESXi host shows as disconnected in vCenter, I need to reboot the host. **Reality:** A disconnected host usually means vCenter has lost network connectivity to the host. You should first check the host's network connection, then try to reconnect it from the vSphere Client, then check if the host's management agents are running (using services.sh restart), and only reboot as a last resort. (Rebooting is a common reflex, but it causes unnecessary downtime for VMs running on that host. The exam wants you to follow a logical sequence: check network, check services, then consider a reboot.)
- **Misconception:** Storage 'All Paths Down' means the hard drive inside the storage array failed. **Reality:** 'All Paths Down' means the ESXi host cannot reach the storage at all, but the fault could be on the host (bad HBA), on the storage network (bad cable, switch port), or on the storage array (controller failure). The failure is not necessarily the hard drives themselves. (Beginners often think 'storage' equals 'hard drive', but in a SAN environment, the path includes multiple layers. The exam tests your understanding that APD requires a systematic check of all layers from host to array.)
- **Misconception:** If a VM has an IP address assigned, its network is definitely working. **Reality:** A VM can have an IP address but still be unable to communicate if the VLAN configuration on the virtual switch port group does not match the physical switch, or if the default gateway is unreachable. The IP address only indicates that the VM's network stack is initialised, not that the full network path is functional. (This mistake comes from a common-sense assumption that having an IP means connectivity. In virtual networking, the VLAN and port group configuration can break connectivity even with a valid IP. The exam loves this trap.)

## Key takeaways

- Always troubleshoot by starting with the simplest and most obvious cause before considering complex possibilities.
- If one VM is affected but others on the same host are fine, the problem is likely specific to that VM, not the host.
- If all VMs on a host are affected, the problem is likely with the host itself (resources, storage, or network connectivity).
- Storage 'All Paths Down' requires checking the physical HBA, cables, SAN switches, and storage array controllers in that order.
- A VM can have a valid IP address but still fail to communicate due to a VLAN mismatch on the virtual switch port group.
- Documenting the problem, investigation steps, and fix is as important as the fix itself for preventing future issues.
- The vmkernel log file on an ESXi host is the primary source for diagnosing storage-related faults.
- For vMotion to work, both source and target hosts must have compatible CPUs and access to the same networks and datastores.

## FAQ

**What is the first thing I should check if a virtual machine cannot connect to the network?**

The first thing to check is whether the VM's virtual network adapter is connected in its hardware settings. Go to the VM settings and ensure the 'Connected' checkbox is ticked. If it is, then check the port group configuration and the VLAN settings.

**What does 'All Paths Down' mean in vSphere?**

'All Paths Down' (APD) means an ESXi host has lost all connections to a storage device. It indicates a fault somewhere in the storage path, such as a broken cable, a faulty HBA, a misconfigured SAN switch, or a storage array controller failure.

**How do I check the health of an ESXi host?**

You can check the health of an ESXi host by looking at the 'Hardware Health' tab in the vSphere Client, which shows the status of fans, power supplies, and sensors. You can also use the `esxcli hardware` command from the host's command line.

**What is the difference between a host being 'disconnected' and 'not responding'?**

A 'disconnected' host means vCenter has lost contact with the host, but the host may still be running and VMs may still be functional. A 'not responding' host means the host is powered on but vCenter cannot communicate with it, often due to a network or service issue.

**Can I vMotion a VM to a host with a different CPU?**

vMotion requires CPU compatibility between the source and target hosts. If the CPUs are not from the same family or have different features, vMotion will fail. You can enable Enhanced vMotion Compatibility (EVC) to mask CPU feature differences if the hardware supports it.

**What logs should I check for storage problems in vSphere?**

The primary log to check for storage problems is the `vmkernel.log` file on the ESXi host. It contains entries about SCSI commands, path failures, and storage device events. You can access it via the command line or download it from the vSphere Client.

## Check your understanding

1. **What is the first step in troubleshooting a vSphere issue according to the standard methodology?**

   Answer: The first step is to identify the symptom: clearly defining what is wrong, such as a VM being unresponsive or a host showing as disconnected.

2. **If one virtual machine is slow but other VMs on the same host are performing normally, where should you look for the cause?**

   Answer: You should look at the slow VM specifically, checking its performance charts, guest OS, and application configuration, because the problem is likely VM-specific, not host-wide.

3. **What does 'All Paths Down' (APD) indicate in vSphere storage troubleshooting?**

   Answer: APD indicates that an ESXi host has lost all connections to a storage device, meaning the host cannot access that storage at all.

4. **Can a virtual machine have an IP address but still be unable to communicate on the network? If so, give one reason why.Yes, a VM can have an IP address but still be unable to communicate if the VLAN ID on the virtual switch port group does not match the VLAN configured on the physical switch.**

   Answer: Yes, a VM can have an IP address but still be unable to communicate if the VLAN ID on the virtual switch port group does not match the VLAN configured on the physical switch.

5. **Why should you never restart an ESXi host as the first step when it shows as 'disconnected' in vCenter?**

   Answer: Restarting the host causes unnecessary downtime for all VMs running on it. Instead, you should first check network connectivity and try restarting the management agents.

---

Interactive version with quiz and diagrams: https://courseiva.com/learn/vmware-vcp-dcv/vmware-troubleshooting-basics