Missing VM NIC

The Disappearing Act

A VM went off the network, and actually lost the NIC from within the VM’s hardware.

Pouring through logs, (some thanks to LogInsight, more on that later), I discovered in vmware-xx.log:

2013-11-19T07:33:01.246Z| vcpu-0| Powering off Ethernet0
2013-11-19T07:33:01.246Z| vcpu-0| Hot removal done.

ah ha! This shows Ethernet0 was removed via the “Safely Remove Hardware” icon in the Windows system tray.

The solution is to add a new NIC of the same type.

This is something that happens quite a bit as you can read here: http://blogs.vmware.com/kb/2010/06/nic-is-missing-in-my-virtual-machine.html

A good idea to prevent this would be to disable HotPlug as mentioned in this KB artile: http://kb.vmware.com/kb/1012225

LogInsight was able to show some hints, but didn’t quite nail it. LogInsight showed:
2013-11-19T07:33:01.841Z esxhost.local Vpxa: [FFE6FB90 verbose ‘Default’ opID=WFU-e9559ea6] [VpxaHalVmHostagent] 2746: Config changed ‘config.extraConfig[“ethernet0.pciSlotNumber”].value’

Unfortunately, LogInsight doesn’t index the virtual machine logs (vmware.log). If it did, it would have saved me a lot of time.

William Lam has done a great post on how to get your vmware.log into the ESXi syslog, http://www.virtuallyghetto.com/2013/07/a-hidden-vsphere-51-gem-forwarding.html

Search and Destroy

From here I needed to find out who the culprit was. Unfortunately there’s no central syslog for our Windows servers, so I had to dig through the event logs. The Security logs are pretty noisey, but I was able to see some users were logged around the time of the incident. No one admitted to accidentally ejecting the hardware, and it wouldn’t have mattered if we did identify the individual. Everyone was now aware of it, and knew to be more careful.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s