VMWare — Rolling your own OEM branded image to include missing vibs

Posted by & filed under Virtualization, VMWare.

Problem: A fresh install of HPE branded ESXi 6.5 U1 cannot see the LUNs on the SAN during the installation. The server boots from SAN which means I need to be able to connect to the remote LUNs during installation. There is no local storage. Currently on 5.5u3, it is working fine. The HPE branded 6.5U1 installer does not see the LUNs presented by my SAN. A quick boot into the 5.5 installer confirms it can see the LUNS with no problems ruling out zoning issues, physical issues, etc.

The HPE ESXi 6.5 image seems to be lacking support for the Qlogic BR-815/Qlogic BR-825/Brocade-415/Brocade-825 FC cards which are all mostly the same card. After verifying compatibility of the server, and of the BR-815 FC cards, I determined that the driver simply is not included in the HPE image.

Here are the steps I took to roll my own installer using the HPE branded one as a base using the VMWare Image Builder toolset:

Resources:

  • Customizing installations with Image Builder: https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.install.doc/GUID-48AC6D6A-B936-4585-8720-A1F344E366F9.html
  • Add VIBs to an image profile: pubs.vmware.com/vsphere-51/index.jsp#com…
  • Export image profile to a ISO: pubs.vmware.com/vsphere-51/index.jsp#com…
  • HPE vibs Depot: http://vibsdepot.hpe.com
  • Using vibsdepot with Image Builder: http://vibsdepot.hpe.com/getting_started.html
  • Applying VIBS to a image walkthrough: https://blogs.vmware.com/vsphere/2017/05/apply-latest-vmware-esxi-security-patches-oem-custom-images-visualize-differences.html
  • VMWare Compatibility Guide: https://www.vmware.com/resources/compatibility/search.php
  • HPE VMWare Support and Certification Matrices: http://h17007.www1.hpe.com/us/en/enterprise/servers/supportmatrix/vmware.aspx
  • Info on HPE Custom Images: https://www.hpe.com/us/en/servers/hpe-esxi.html
  • Supported driver firmware versions for I/O devices: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2030818

Basic steps:

  • Identify OEM’s software depot URL, in this case the HPE ESXi 6.5U1 image http://vibsdepot.hpe.com/index-ecli-650.xml
  • Identify where the VIB is available for the driver. In my case, the Brocade BR-815 driver was downloaded via the VMWare compatibility site: https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=io&productid=5346 — Note the VIB is actually inside a zip file inside the zip you download. It will be looking for a index.xml file in the root of the zip.
  • Use the esx-image-creator.ps1 to generate a new image with the newly included software: https://github.com/vmware/PowerCLI-Example-Scripts/blob/master/Scripts/esxi-image-creator.ps1
  • Use Export-EsxImageProfile to generate a ISO for installation.

 

Booting the server with the newly built ISO enables me to see the LUNs so I can complete my boot-from-san installation.

FreeBSD identifying failed disk

Posted by & filed under Hardware.

Using the sas2ircu utility from LSI, we can blink the drive LED to help ID the failed drive correctly. Of course this requires a LSI card. Some LSI cards may need to use the sas3ircu utility instead. There have been some reports from the interwebs that this utility failed to blink the correct drive, but I have not experienced this myself.

As always use the supercomputer between your ears to ensure the physical serial and the serial reported by the system match, etc etc.

Back to the sas2ircu utility in a moment. We need to first acquire the serial number of the failed disk. For a system that is multipath, we can find the actual dev names by running the following to locate a disk in the fail state:

Now we can see da16 is failed. Time to get the serial number of that disk. Or da43. they are the same just multipaths.

Save that serial number for the next step.

Smartctl also outputs other useful information about the drive, statistics, etc. Worth checking out, but not relevant here.

Next, we can display the disks attached to one of those controllers. Be sure to input the correct serial number in the grep command:

Get the enclosure and slot # of the failed drive and turn the led on:

Turn the led off:

NOTE: If you are replacing a disk that is multipath, e.g. you see something like the following when you offline and remove a disk, ensure that the LED above is OFF or GEOM_MULTIPATH will not pickup the new disk as multipath. See the below log for what happens when a disk is inserted with the LED blinking Vs not blinking: