Extending a volume online

Posted by & filed under Server Admin.

I was attempting to resize a GPT partition ( http://natesbox.com/blog/extending-logical-volume-online/ ). Expanded the physical disk from 4TB to 6TB. Found that fdisk reported the disk size to be 6TB, but would not let me use any of the additional sectors:

Disk /dev/sda: 6 TiB, 6597069766656 bytes, 12884901888 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: E3374C46-FDAA-4FA0-AFD2-1C861700B6EA

Device Start End Sectors Size Type
/dev/sda1 2048 4095 2048 1M BIOS boot
/dev/sda2 4096 503807 499712 244M Linux filesystem
/dev/sda3 503808 8589932543 8589428736 4T Linux LVM

Note the disk says it has 12884901888 sectors, but I can’t expand it past the 8589428736’th sector of /dev/sda3

Running the v command in fdisk reports:

Command (m for help): v
MyLBA mismatch with real position at backup header.
1 error detected.

Found this article: serverfault.com/questions/833231/after-h…

This recommends using gdisk to repair the issue as the backup partition table wasn’t moved to the end of the new geometry:

root@mdm-backup-01:~# gdisk /dev/sda
GPT fdisk (gdisk) version 1.0.1

Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present

Found valid GPT with protective MBR; using GPT.

Command (? for help): r

Recovery/transformation command (? for help): v

Problem: The secondary header’s self-pointer indicates that it doesn’t reside
at the end of the disk. If you’ve added a disk to a RAID array, use the ‘e’
option on the experts’ menu to adjust the secondary header’s and partition
table’s locations.

Identified 1 problems!

Recovery/transformation command (? for help): x

Expert command (? for help): e
Relocating backup data structures to the end of the disk

Expert command (? for help): v

No problems found. 4294971325 free sectors (2.0 TiB) available in 2
segments, the largest of which is 4294969311 (2.0 TiB) in size.

Expert command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/sda.
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot or after you
run partprobe(8) or kpartx(8)
The operation has completed successfully.



Proxmox import ova ovf qcow2 disk into VM

Posted by & filed under Virtualization.

  • Create new VM
  • Note the VM ID
  • If you have a .ova or .ovf file, you simply need to do the following to extract the vmdk:
  • tar -xvf *.ova
  • Import the disk into the vmid of the new vm
  • qm importdisk targetvmid disk001.vmdk local -format qcow2

    or

    qm importdisk targetvmid disk001.vmdk local-lvm -format qcow2

    Depending on the storage you have setup.

  • It’s also worth noting that at least on my particular homelab setup, running the qm importdisk command brought the server to it’s knees until the import completes.

Decommissioning a vCenter PSC and VCSA

Posted by & filed under Virtualization, VMWare.

I recently needed to decommission a VCSA and external PSC. Following the VMWare KB 2106736 I proceeded to decomission the servers usign the cmsso utility.

 

Decommission vCenter — connected to the PSC it is registered with and:

root@vcenter-sb-psc [ ~ ]# cmsso-util unregister --node-pnid vcenter-sb.redacted.lan --username administrator@vsphere.local
Password:
2017-11-01T18:20:15.806Z   Running command: ['/usr/lib/vmware-vmafd/bin/dir-cli', 'service', 'list', '--login', 'administrator@vsphere.local']
2017-11-01T18:20:15.863Z   Done running command
Success

Now to connect to the PSC that will be staying online and decommission the other PSC

root@vcenter-psc [ ~ ]# cmsso-util unregister --node-pnid vcenter-sb --username administrator@vsphere.local
Password:
Could not find a host id which maps to vcenter-sb in Component Manager
Failed!!!

I proceeded to use vdcrepadmin to check out the replication partners which is only vcenter-sb-psc.redacted.lan from my PSC that will be staying online:

root@vcenter-psc [ /usr/lib/vmware-vmdir/bin ]# ./vdcrepadmin -f showpartners -h vcenter-psc.redacted.lan -u administrator
password:
ldap://vcenter-sb-psc.redacted.lan

Then I checked the actual servers:

root@vcenter-psc [ /usr/lib/vmware-vmdir/bin ]# ./vdcrepadmin -f showservers -h vcenter-psc.redacted.lan -u administrator
password:
cn=vcenter-psc.redacted.lan,cn=Servers,cn=redactedfl,cn=Sites,cn=Configuration,dc=vsphere,dc=local
cn=vcenter-sb-psc.redacted.lan,cn=Servers,cn=redactedsb,cn=Sites,cn=Configuration,dc=vsphere,dc=local

We see both PSC’s as expected. Finally I removed the PSC that is to be decommissioned:

root@vcenter-psc [ /usr/lib/vmware-vmdir/bin ]# ./vdcleavefed -h vcenter-sb-psc.redacted.lan -u administrator
password:
vdcleavefd offline for server vcenter-sb-psc.redacted.lan
Leave federation cleanup failed. Error[1] - Operations error

Error again. Some googling led me to techbrainblog’s excellent page on using these utilities and also the solutions to some common but cryptic errors. Very useful. The solution to this error in particular is to simply shut down the old PSC. It needs to be offline before the command is ran.

root@vcenter-psc [ /usr/lib/vmware-vmdir/bin ]# ./vdcleavefed -h vcenter-sb-psc.redacted.lan -u administrator
password:
vdcleavefd offline for server vcenter-sb-psc.redacted.lan
 vcenter-sb-psc.tnsc.lan server cleanup performed.

Good to go!

Ref: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2106736

Ref: https://techbrainblog.com/2015/10/02/issues-and-errors-when-decommissioning-the-vcenter-server-or-a-platform-services-controller-vcsa-6-0/

Extending a logical volume online

Posted by & filed under Server Admin, VMWare.

I’ve written about this in past posts. Here is an updated article straight from VMWare: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1006371

 

  1. Power off the virtual machine.
  2. Edit the virtual machine settings and extend the virtual disk size. For more information, see Increasing the size of a virtual disk (1004047).
  3. Power on the virtual machine.
  4. Identify the device name, which is by default /dev/sda, and confirm the new size by running the command:

    # fdisk -l

  5. Create a new primary partition:
    1. Run the command:

      # fdisk /dev/sda (depending the results of the step 4)

    2. Press p to print the partition table to identify the number of partitions. By default, there are 2: sda1 and sda2.
    3. Press n to create a new primary partition.
    4. Press p for primary.
    5. Press 3 for the partition number, depending on the output of the partition table print.
    6. Press Enter two times.
    7. Press t to change the system’s partition ID.
    8. Press 3 to select the newly creation partition.
    9. Type 8e to change the Hex Code of the partition for Linux LVM.
    10. Press w to write the changes to the partition table.
  6. Restart the virtual machine.
  7. Run this command to verify that the changes were saved to the partition table and that the new partition has an 8e type:

    # fdisk -l

  8. Run this command to convert the new partition to a physical volume:

    Note: The number for the sda can change depending on system setup. Use the sda number that was created in step 5.

    # pvcreate /dev/sda3

  9. Run this command to extend the physical volume:

    # vgextend VolGroup00 /dev/sda3

    Note: To determine which volume group to extend, use the command vgdisplay.

  10. Run this command to verify how many physical extents are available to the Volume Group:

    # vgdisplay VolGroup00 | grep “Free”

  11. Run the following command to extend the Logical Volume:

    # lvextend -L+#G /dev/VolGroup00/LogVol00

    Where # is the number of Free space in GB available as per the previous command. Use the full number output from Step 10 including any decimals.

    Note: To determine which logical volume to extend, use the command lvdisplay.

  12. Run the following command to expand the ext3 filesystem online, inside of the Logical Volume:

    # ext2online /dev/VolGroup00/LogVol00

    Notes:

    • Use resize2fs instead of ext2online if it is not a Red Hat virtual machine.
    • By default, Red Hat and CentOS 7 use the XFS file system you can grow the file system by running the xfs_growfs command.
  13. Run the following command to verify that the / filesystem has the new space available:

    # df -h /

VMWare — Rolling your own OEM branded image to include missing vibs

Posted by & filed under Virtualization, VMWare.

Problem: A fresh install of HPE branded ESXi 6.5 U1 cannot see the LUNs on the SAN during the installation. The server boots from SAN which means I need to be able to connect to the remote LUNs during installation. There is no local storage. Currently on 5.5u3, it is working fine. The HPE branded 6.5U1 installer does not see the LUNs presented by my SAN. A quick boot into the 5.5 installer confirms it can see the LUNS with no problems ruling out zoning issues, physical issues, etc.

The HPE ESXi 6.5 image seems to be lacking support for the Qlogic BR-815/Qlogic BR-825/Brocade-415/Brocade-825 FC cards which are all mostly the same card. After verifying compatibility of the server, and of the BR-815 FC cards, I determined that the driver simply is not included in the HPE image.

Here are the steps I took to roll my own installer using the HPE branded one as a base using the VMWare Image Builder toolset:

Resources:

  • Customizing installations with Image Builder: https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.install.doc/GUID-48AC6D6A-B936-4585-8720-A1F344E366F9.html
  • Add VIBs to an image profile: pubs.vmware.com/vsphere-51/index.jsp#com…
  • Export image profile to a ISO: pubs.vmware.com/vsphere-51/index.jsp#com…
  • HPE vibs Depot: http://vibsdepot.hpe.com
  • Using vibsdepot with Image Builder: http://vibsdepot.hpe.com/getting_started.html
  • Applying VIBS to a image walkthrough: https://blogs.vmware.com/vsphere/2017/05/apply-latest-vmware-esxi-security-patches-oem-custom-images-visualize-differences.html
  • VMWare Compatibility Guide: https://www.vmware.com/resources/compatibility/search.php
  • HPE VMWare Support and Certification Matrices: http://h17007.www1.hpe.com/us/en/enterprise/servers/supportmatrix/vmware.aspx
  • Info on HPE Custom Images: https://www.hpe.com/us/en/servers/hpe-esxi.html
  • Supported driver firmware versions for I/O devices: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2030818

Basic steps:

  • Identify OEM’s software depot URL, in this case the HPE ESXi 6.5U1 image http://vibsdepot.hpe.com/index-ecli-650.xml
  • Identify where the VIB is available for the driver. In my case, the Brocade BR-815 driver was downloaded via the VMWare compatibility site: https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=io&productid=5346 — Note the VIB is actually inside a zip file inside the zip you download. It will be looking for a index.xml file in the root of the zip.
  • Use the esx-image-creator.ps1 to generate a new image with the newly included software: https://github.com/vmware/PowerCLI-Example-Scripts/blob/master/Scripts/esxi-image-creator.ps1
  • Use Export-EsxImageProfile to generate a ISO for installation.

 

PowerCLI C:\Users\user> Add-EsxSoftwareDepot http://vibsdepot.hpe.com/index-ecli-650.xml

Depot Url
---------
http://vibsdepot.hpe.com/index-ecli-650.xml


PowerCLI C:\Users\user> Add-EsxSoftwareDepot -DepotUrl C:\Users\user\Downloads\BCD-bfa-3.2.5.0-00000-offline_bundle-2352086.zip

Depot Url
---------
zip:C:\Users\user\Downloads\BCD-bfa-3.2.5.0-00000-offline_bundle-2352086.zip?index.xml


PowerCLI C:\Users\user> Get-EsxSoftwareDepot

Depot Url
---------
http://vibsdepot.hpe.com/index-ecli-650.xml
zip:C:\Users\user\Downloads\BCD-bfa-3.2.5.0-00000-offline_bundle-2352086.zip?index.xml


PowerCLI C:\Users\user> .\esxi-image-creator.ps1 -LeaveCurrentDepotsMounted -NewProfileName ESXi_6.5.0U1_with_HPE_and_Qlogic -Files C:\Users\user\Downloads\VMware-ESXi-6.5.0-Update1-5969303-HPE-650.U1.10.1.3.3-Oct2017-depot.zip -Accepta
nce PartnerSupported

Depot Url
---------
zip:C:\Users\user\Downloads\VMware-ESXi-6.5.0-Update1-5969303-HPE-650.U1.10.1.3.3-Oct2017-depot.zip?index.xml

The following VIBs will not be included in ESXi_6.5.0U1_with_HPE_and_Qlogic:
tools-light

Finished creating ESXi_6.5.0U1_with_HPE_and_Qlogic


PowerCLI C:\Users\user> Export-EsxImageProfile -ExportToIso -ImageProfile "ESXi_6.5.0U1_with_HPE_and_Qlogic" -FilePath C:\Users\user\Downloads\VMWare-ESXi-6.5.0-U1-HPE-Qlogic-Custom-Oct2017.iso

Booting the server with the newly built ISO enables me to see the LUNs so I can complete my boot-from-san installation.

VMWare Update Manager Fails

Posted by & filed under Uncategorized, Virtualization, VMWare.

I received a fairly generic error when running VMWARE Update Manager against some hosts:

No real useful information. The actual log is available on the VCSA 6.5 at: /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server-log4cpp.log

In my case it was as simple as the DNS being set incorrectly on the ESXi hosts due to some networking changes:

[2017-10-10 01:40:04:334 'VciScanTask.ScanTask{44}' 140420998207232 INFO]  [vciTaskBase, 1362] VciTask { id: ScanTask{44}, type: com.vmware.vcIntegrity.ScanTask }: Setting VC task state to: error
[2017-10-10 01:54:19:765 'HostUpdateDepotManager' 140421651932928 ERROR]  [scanHost20, 371] result for host: vm07.redacted.lan (entity: host-188) shows error :
<error errorClass="MetadataDownloadError">
  <errorCode>4</errorCode>
  <errorDesc>Failed to download metadata.</errorDesc>
  <msg>('http://vcenter.redacted.lan:9084/vum/repository/hostupdate/csco/csco-VEM-5.5.0-metadata.zip', '/tmp/tmp6q7F56', '[Errno 4] IOError: &lt;urlopen error [Errno -2] Name or service not known&gt;')</msg>
</error>

Other threads that might be related:

communities.vmware.com/thread/546976

Resetting the VMWare Update Manager Database: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2147284

PowerCLI – Get a VM’s Disks

Posted by & filed under Virtualization, VMWare.

PowerCLI snippets to get a VM’s disks

This command will retrieve the specified VM’s attached disk paths:

Get-HardDisk -VM Guest-VM-Name-Here

But we can also focus on the filename:

PowerCLI C:\> Get-HardDisk -VM VM-core | Select-Object Parent,Name,Filename

Filename
--------
[EMC2_Tier2_Replicated_VMFS1] VM-core/VM-core_16.vmdk
[EMC_Tier2_NotReplicated_VMFS11] VM-core/VM-core_7.vmdk
[EMC2_Tier2_Replicated_VMFS3] VM-core/VM-core_14.vmdk
[EMC2_Tier2_Replicated_VMFS1] VM-core/VM-core_2.vmdk
[EMC2_Tier2_Replicated_VMFS4] VM-core/VM-core_5.vmdk
[EMC2_Tier2_Replicated_VMFS4] VM-core/VM-core_1.vmdk
[EMC_Tier2_Replicated_VMFS2] VM-core/VM-core_17.vmdk
[EMC2_Tier2_Replicated_VMFS1] VM-core/VM-core_1.vmdk
[EMC2_Tier2_Replicated_VMFS1] VM-core/VM-core.vmdk
[EMC2_Tier2_Replicated_VMFS4] VM-core/VM-core_10.vmdk
[EMC2_Tier2_Replicated_VMFS4] VM-core/VM-core_3.vmdk
[EMC2_Tier2_Replicated_VMFS3] VM-core/VM-core.vmdk

We can also see the other columns available:

PowerCLI C:\> Get-HardDisk -VM VM-core | Get-Member


   TypeName: VMware.VimAutomation.ViCore.Impl.V1.VirtualDevice.FlatHardDiskImpl

Name             MemberType Definition
----             ---------- ----------
ConvertToVersion Method     T VersionedObjectInterop.ConvertToVersion[T]()
Equals           Method     bool Equals(System.Object obj)
GetHashCode      Method     int GetHashCode()
GetType          Method     type GetType()
IsConvertableTo  Method     bool VersionedObjectInterop.IsConvertableTo(type type)
LockUpdates      Method     void ExtensionData.LockUpdates()
ToString         Method     string ToString()
UnlockUpdates    Method     void ExtensionData.UnlockUpdates()
CapacityGB       Property   decimal CapacityGB {get;}
CapacityKB       Property   long CapacityKB {get;}
Client           Property   VMware.VimAutomation.ViCore.Interop.V1.VIAutomation Client {get;}
ConnectionState  Property   VMware.VimAutomation.ViCore.Types.V1.VirtualDevice.ConnectInfo ConnectionState {get;}
DiskType         Property   VMware.VimAutomation.ViCore.Types.V1.VirtualDevice.DiskType DiskType {get;}
ExtensionData    Property   System.Object ExtensionData {get;}
Filename         Property   string Filename {get;}
Id               Property   string Id {get;}
Name             Property   string Name {get;}
Parent           Property   VMware.VimAutomation.Sdk.Types.V1.VIObject Parent {get;}
ParentId         Property   string ParentId {get;}
Persistence      Property   VMware.VimAutomation.ViCore.Types.V1.VirtualDevice.PersistencePolicy Persistence {get;}
StorageFormat    Property   VMware.VimAutomation.ViCore.Types.V1.VirtualDevice.VirtualDiskStorageFormat StorageFormat {get;}
Uid              Property   string Uid {get;}

We could also do something like get the Disk paths for all guests:

PowerCLI C:\> Get-VM -Location $datacenter | ForEach-Object {Get-HardDisk -VM $_.Name | Select-Object Parent,Name,Filename}

Parent                                                                          Name                                                                            Filename
------                                                                          ----                                                                            --------
REDACTEDADCSVR01                                                                    Hard disk 1                                                                     [EMC2_Tier1_NotReplicated_VMFS3] REDACTEDADCSVR01/disk0-000001.vmdk
REDACTEDUTLSVR                                                                      Hard disk 1                                                                     [EMC2_Tier1_NotReplicated_VMFS3] REDACTEDUTLSVR/disk0-000001.vmdk
RP-BATCHMASTER2                                                                 Hard disk 1                                                                     [EMC2_Tier1_NotReplicated_VMFS3] RP-BATCHMASTER2/disk0-000001.vmdk
RP-TS02                                                                         Hard disk 1                                                                     [EMC2_Tier1_NotReplicated_VMFS1] REDACTED/disk0-000001.vmdk
REDACTEDFMSVR01                                                                     Hard disk 1                                                                     [EMC2_Tier1_NotReplicated_VMFS3] REDACTEDFMSVR01/disk0-000001.vmdk
REDACTEDSVR01                                                                    Hard disk 1                                                                     [EMC2_Tier1_NotReplicated_VMFS3] REDACTED/disk0-000001.vmdk
REDACTEDSVR01                                                                    Hard disk 2                                                                     [EMC2_Tier1_NotReplicated_VMFS3] REDACTED/disk1-000001.vmdk

 

Changing the IP address of a VMWare PSC via the command line

Posted by & filed under Virtualization, VMWare.

I recently needed to change the IP address of my PSC. Unfortunately it was already inaccessible so I was unable to do it via the standard GUI methods. I SSH’d into the box and had a look but it pretty immediately becomes apparent you can’t just update things the way you would a normal linux box. Enter vami_config_net. I believe this utility is available on any of the VMWare appliances that utilize VAMI/photon but I could be wrong. As you may notice int he article it refers to this being for the vCetner Support Assistant, but it worked just the same for me on my external PSC.

kb.vmware.com/selfservice/microsites/sea…

/opt/vmware/share/vami/vami_config_net

Platform services controller — Migrating to external psc and verifying replication partners

Posted by & filed under Virtualization, VMWare.

In preparation for migrating from vCenter 6.5 w/ embedded PSC, to a external PSC I needed to validate the replication between my new external PSC and the embedded platform services controller. To validate PSC replication partners, the vdcrepadmin utility can be used. For more information see https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2127057

login as: root

VMware vCenter Server Appliance 6.5.0.10000

Type: VMware Platform Services Controller

Using keyboard-interactive authentication.
Password:
Connected to service

    * List APIs: "help api list"
    * List Plugins: "help pi list"
    * Launch BASH: "shell"

Command> shell
Shell access is granted to root
root@vcenter-psc [ ~ ]# cd /usr/lib/vmware-vmdir/bin
root@vcenter-psc [ /usr/lib/vmware-vmdir/bin ]# ./vdcrepadmin -f showservers -h localhost -u Administrator -w Passw\!rd
cn=vcenter.redacted.lan,cn=Servers,cn=redactedfl,cn=Sites,cn=Configuration,dc=vsphere,dc=local
cn=vcenter-psc.redacted.lan,cn=Servers,cn=redactedfl,cn=Sites,cn=Configuration,dc=vsphere,dc=local
root@vcenter-psc [ /usr/lib/vmware-vmdir/bin ]#

Note in the above commands, for the -w parameter, non alpha characters must be escaped with a \ otherwise you may get authentication failures.

I am now able to continue with the external psc migration as detailed here: docs.vmware.com/en/VMware-vSphere/6.5/co…

root@vcenter [ /usr/lib/vmware-vmdir/bin ]# service-control --status --all
Running:
 applmgmt lwsmd pschealth vmafdd vmcad vmdird vmdnsd vmonapi vmware-cis-license vmware-cm vmware-content-library vmware-eam vmware-perfcharts vmware-psc-client vmware-rhttpproxy vmware-sca vmware-sps vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-updatemgr vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-client vsphere-ui
Stopped:
 vmcam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-rbd-watchdog vmware-vcha
root@vcenter [ /usr/lib/vmware-vmdir/bin ]# cmsso-util reconfigure --repoint-psc vcenter-psc.redacted.lan --username administrator --domain-name vsphere.local --passwd Passw0rd!
Validating Provided Configuration ...
Validation Completed Successfully.
Executing reconfiguring steps. This will take few minutes to complete.
Please wait ...
Stopping all the services ...
All services stopped.
Perform update startuptype operation in stop order. startup_type=Disabled, svc_names=[u'vmware-psc-client', u'pschealth', u'vmdnsd', u'vmware-cis-license', u'vmware-stsd', u'vmware-sts-idmd', u'vmcad', u'vmdird'], include_vmonsvcs=False include_coreossvcs=False, include_leafossvcs=False
2017-09-21T16:42:42.742Z   Running command: ['/usr/bin/systemctl', 'mask', u'vmware-psc-client']
2017-09-21T16:42:42.851Z   Done running command
Successfully changed startuptype for service vmware-psc-client
2017-09-21T16:42:42.858Z   Successfully updated starttype: DISABLED for service pschealth
2017-09-21T16:42:42.858Z   Successfully updated pschealth service
Successfully changed startuptype for service pschealth
2017-09-21T16:42:42.866Z   Running command: ['/usr/bin/systemctl', 'mask', u'vmdnsd']
2017-09-21T16:42:42.953Z   Done running command
Successfully changed startuptype for service vmdnsd
2017-09-21T16:42:42.960Z   Successfully updated starttype: DISABLED for service cis-license
2017-09-21T16:42:42.960Z   Successfully updated cis-license service
Successfully changed startuptype for service cis-license
2017-09-21T16:42:42.967Z   Running command: ['/usr/bin/systemctl', 'mask', u'vmware-stsd']
2017-09-21T16:42:43.077Z   Done running command
Successfully changed startuptype for service vmware-stsd
2017-09-21T16:42:43.084Z   Running command: ['/usr/bin/systemctl', 'mask', u'vmware-sts-idmd']
2017-09-21T16:42:43.206Z   Done running command
Successfully changed startuptype for service vmware-sts-idmd
2017-09-21T16:42:43.213Z   Running command: ['/usr/bin/systemctl', 'mask', u'vmcad']
2017-09-21T16:42:43.331Z   Done running command
Successfully changed startuptype for service vmcad
2017-09-21T16:42:43.338Z   Running command: ['/usr/bin/systemctl', 'mask', u'vmdird']
2017-09-21T16:42:43.455Z   Done running command
Successfully changed startuptype for service vmdird
2017-09-21T16:42:43.460Z   Running command: ['/sbin/chkconfig', u'vmware-rhttpproxy']
2017-09-21T16:42:43.486Z   Done running command
2017-09-21T16:42:43.487Z   Running command: ['/sbin/chkconfig', u'vmware-stsd']
2017-09-21T16:42:43.494Z   Done running command
2017-09-21T16:42:43.496Z   Running command: ['/sbin/chkconfig', u'vmware-vpxd']
2017-09-21T16:42:43.503Z   Done running command
2017-09-21T16:42:43.508Z   Running command: ['/sbin/chkconfig', u'vmware-sts-idmd']
2017-09-21T16:42:43.515Z   Done running command
2017-09-21T16:42:43.516Z   Running command: ['/sbin/chkconfig', u'vmware-netdumper']
2017-09-21T16:42:43.531Z   Done running command
2017-09-21T16:42:43.532Z   Running command: ['/sbin/chkconfig', u'vmware-rbd-watchdog']
2017-09-21T16:42:43.541Z   Done running command
2017-09-21T16:42:43.542Z   Running command: ['/sbin/chkconfig', u'vmware-rhttpproxy']
2017-09-21T16:42:43.549Z   Done running command
2017-09-21T16:42:43.550Z   Running command: ['/sbin/chkconfig', u'vmware-stsd']
2017-09-21T16:42:43.556Z   Done running command
2017-09-21T16:42:43.557Z   Running command: ['/sbin/chkconfig', u'vmware-vapi-endpoint']
2017-09-21T16:42:43.563Z   Done running command
2017-09-21T16:42:43.564Z   Running command: ['/sbin/chkconfig', u'vmafdd']
2017-09-21T16:42:43.570Z   Done running command
2017-09-21T16:42:43.571Z   Running command: ['/sbin/chkconfig', u'vmcad']
2017-09-21T16:42:43.579Z   Done running command
2017-09-21T16:42:43.579Z   Running command: ['/sbin/chkconfig', u'vmdird']
2017-09-21T16:42:43.586Z   Done running command
2017-09-21T16:42:43.587Z   Running command: ['/sbin/chkconfig', u'vmware-vpostgres']
2017-09-21T16:42:43.594Z   Done running command
2017-09-21T16:42:43.594Z   Running command: ['/sbin/chkconfig', u'vmware-vpxd']
2017-09-21T16:42:43.602Z   Done running command
2017-09-21T16:42:43.603Z   Running command: ['/sbin/chkconfig', u'vmware-vsm']
2017-09-21T16:42:43.609Z   Done running command
2017-09-21T16:42:43.610Z   Running command: ['/sbin/chkconfig', u'vmware-imagebuilder']
2017-09-21T16:42:43.617Z   Done running command
2017-09-21T16:42:43.618Z   Running command: ['/sbin/chkconfig', u'vmafdd']
2017-09-21T16:42:43.624Z   Done running command
2017-09-21T16:42:43.628Z   Running command: ['/sbin/chkconfig', u'vmware-vpxd']
2017-09-21T16:42:43.635Z   Done running command
2017-09-21T16:42:43.636Z   Running command: ['/sbin/chkconfig', u'vmware-vpxd']
2017-09-21T16:42:43.642Z   Done running command
2017-09-21T16:42:43.647Z   Running command: ['/sbin/chkconfig', u'vmware-vpxd']
2017-09-21T16:42:43.654Z   Done running command
2017-09-21T16:42:43.656Z   Running command: ['/sbin/chkconfig', u'vmdird']
2017-09-21T16:42:43.663Z   Done running command
2017-09-21T16:42:43.664Z   Running command: ['/sbin/chkconfig', u'vmware-vpxd']
2017-09-21T16:42:43.671Z   Done running command
2017-09-21T16:42:43.673Z   Running command: ['/sbin/chkconfig', u'vmware-vpxd']
2017-09-21T16:42:43.680Z   Done running command
2017-09-21T16:42:43.682Z   Running command: ['/sbin/chkconfig', u'vmware-vpxd']
2017-09-21T16:42:43.688Z   Done running command
2017-09-21T16:42:43.690Z   Running command: ['/sbin/chkconfig', u'vmware-rhttpproxy']
2017-09-21T16:42:43.697Z   Done running command
2017-09-21T16:42:43.698Z   Running command: ['/sbin/chkconfig', u'vmafdd']
2017-09-21T16:42:43.704Z   Done running command
2017-09-21T16:42:43.704Z   Running command: ['/sbin/chkconfig', u'vmware-stsd']
2017-09-21T16:42:43.711Z   Done running command
2017-09-21T16:42:43.713Z   Running command: ['/sbin/chkconfig', u'vmware-vpxd']
2017-09-21T16:42:43.719Z   Done running command
2017-09-21T16:42:43.722Z   Running command: ['/sbin/chkconfig', u'vmware-vpostgres']
2017-09-21T16:42:43.729Z   Done running command
2017-09-21T16:42:43.732Z   Running command: ['/sbin/chkconfig', u'vmware-vpxd']
2017-09-21T16:42:43.739Z   Done running command
2017-09-21T16:42:43.741Z   Running command: ['/sbin/chkconfig', u'vmware-vpostgres']
2017-09-21T16:42:43.748Z   Done running command
2017-09-21T16:42:43.750Z   Running command: ['/sbin/chkconfig', u'vmware-rhttpproxy']
2017-09-21T16:42:43.757Z   Done running command
2017-09-21T16:42:43.761Z   Running command: ['/sbin/chkconfig', u'vmware-vpxd']
2017-09-21T16:42:43.768Z   Done running command
2017-09-21T16:42:43.771Z   Running command: ['/sbin/chkconfig', u'vmware-vpxd']
2017-09-21T16:42:43.779Z   Done running command
2017-09-21T16:42:43.780Z   Running command: ['/sbin/chkconfig', u'vmware-vpxd']
2017-09-21T16:42:43.786Z   Done running command
2017-09-21T16:42:43.789Z   Running command: ['/sbin/chkconfig', u'vmware-vpostgres']
2017-09-21T16:42:43.795Z   Done running command
2017-09-21T16:42:43.797Z   Running command: ['/sbin/chkconfig', u'vmafdd']
2017-09-21T16:42:43.803Z   Done running command
2017-09-21T16:42:43.803Z   Running command: ['/sbin/chkconfig', u'vmdird']
2017-09-21T16:42:43.810Z   Done running command
2017-09-21T16:42:43.810Z   Running command: ['/sbin/chkconfig', u'vmcad']
2017-09-21T16:42:43.818Z   Done running command
Starting vmafd service.
Successfully joined the external PSC vcenter-psc.redacted.lan
Cleaning up...
Cleanup completed
Starting all the services ...
Started all the services.
The vCenter Server has been successfully reconfigured and repointed to the external Platform Services Controller vcenter-psc.redacted.lan.
root@vcenter [ /usr/lib/vmware-vmdir/bin ]#

And finally, from the external PSC we can verify replication partners again to see that the embedded PSC has been decommissioned, and the external PSC is the only one listed:

root@vcenter-psc [ /usr/lib/vmware-vmdir/bin ]# ./vdcrepadmin -f showservers -h localhost -u Administrator -w Passw\$ord
cn=vcenter-psc.redacted.lan,cn=Servers,cn=redactedfl,cn=Sites,cn=Configuration,dc=vsphere,dc=local
root@vcenter-psc [ /usr/lib/vmware-vmdir/bin ]#

Recovering from a failed platform services controller installation – vSphere 6.5

Posted by & filed under Virtualization, VMWare.

I have used to below commands to recover from a failed PSC deployment. When trying to redeploy after the failed deployment, I encountered the error:

“Failed to run vdcpromo”

Following the below steps on the current PSC resolved the error and I was then able to successfully restart the PSC deployment.

Also, protip to avoid having to keep redeploying the appliance, take a snapshot right after phase 1 completes. Then you can simply restore the snap and access your vm via the web interface to try again.

login as: root

VMware vCenter Server Appliance 6.5.0.10000

Type: vCenter Server with an embedded Platform Services Controller

Using keyboard-interactive authentication.
Password:
Last login: Wed Sep 20 15:34:18 2017 from 10.110.0.181
Connected to service

    * List APIs: "help api list"
    * List Plugins: "help pi list"
    * Launch BASH: "shell"

Command> shell
Shell access is granted to root
root@vcenter [ ~ ]# cd /usr/lib/vmware-vmdir/bin
root@vcenter [ /usr/lib/vmware-vmdir/bin ]# ./vdcleavefed -h vcenter-psc.redacted.lan -u Administrator
password:
vdcleavefd offline for server vcenter-psc.redacted.lan
 vcenter-psc.redacted.lan server cleanup performed.
root@vcenter [ /usr/lib/vmware-vmdir/bin ]#

 

docs.vmware.com/en/VMware-vSphere/6.5/co…

Additional info: I also ran into this when trying to deploy an additional PSC that had a failed installation, but got a completely different error (see below). Going to Administration -> System Configuration in the flash vSphere web client also displays the failed PSC. Login to the live PSC and use the above commands to cleanup, then restart the new PSC deployment. Refreshing the System Configuration page once the vdcleavefed command was ran confirms the cleanup is complete and the failed install is no longer listed.

The error I received when deploying this PSC was:

Could not connect to VMware Directory Service via LDAP. Verify VMware Directory Service is running on the appropriate system and is reachable from this host.

Removing the failed deployment via vdcleavefed did not resolve the issue.

I decided to test LDAP connectivity to the PSC from the failed PSC deployment. I SSH’d into the box and did the following:

root@localhost [ /usr/lib/vmware-vmdir/bin ]# ./vdcadmintool


==================
Please select:
0. exit
1. Test LDAP connectivity
2. Force start replication cycle
3. Reset account password
4. Set log level and mask
5. Set vmdir state
6. Get vmdir state
7. Get vmdir log level and mask
==================

1
Please enter LDAP server host: vcenter-psc.redacted.lan
Please enter LDAP server port: 389
Please enter LDAP server SSL port: 11712
Please enter LDAP Bind DN: cn=Administrator,cn=Users,dc=vsphere,dc=local
Please enter LDAP Bind UPN: Administrator@vsphere.local
Please enter LDAP Bind password:

ldap://vcenter-psc.redacted.lan:389 (ANONYMOUS) bind succeeded.

++++++++++++++++++++ ldaps://vcenter-psc.redacted.lan:11712 SSL bind failed. (-1)(Can't contact LDAP server)

ldap://vcenter-psc.redacted.lan:389 SRP bind succeeded.

++++++++++++++++++++ ldap://vcenter-psc.redacted.lan:389 GSSAPI bind failed. (9100)(Unknown (extension) error)

Edit: Additional semi-related data

Get machine’s guid

root@vcenter-psc [ /usr/lib/vmware-vmdir/bin ]# /usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost

Get machine’s pnid (machine/host name?)

root@vcenter-psc [ /usr/lib/vmware-vmdir/bin ]# /usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost

Get services in the directory

root@vcenter-psc [ ~ ]# /usr/lib/vmware-vmafd/bin/dir-cli service list

vCenter Server (VCSA) 6.5 – Add root CA cert to Windows Server 2016 AD to enable the default VCSA cert to be trusted

Posted by & filed under Active Directory, Server Admin, Virtualization, VMWare.

The VCSA has it’s own CA built in. It uses that CA to generate certs for all the various services. There are two options available to ensure that the certificate is trusted in the browser:

  1. Generate a CSR for the cert and submit to a CA who can generate the cert.
  2. Use Microsoft Active Directory GPO to push out the VCSA’s root CA cert, thereby allowing the workstations to trust the cert already installed.

I went with the second one because the VCSA is using vcenter.mydomain.lan and is only accessible from inside my network which also means only machines on the domain will be connecting to the web interface. This was very simple to make happen…

On the DC:

To distribute certificates to client computers by using Group Policy

  1. On a domain controller in the forest of the account partner organization, start the Group Policy Management snap-in.
  2. Find an existing Group Policy Object (GPO) or create a new GPO to contain the certificate settings. Ensure that the GPO is associated with the domain, site, or organizational unit (OU) where the appropriate user and computer accounts reside.
  3. Right-click the GPO, and then click Edit.
  4. In the console tree, open Computer Configuration\Policies\Windows Settings\Security Settings\Public Key Policies, right-click Trusted Root Certification Authorities, and then click Import.
  5. On the Welcome to the Certificate Import Wizard page, click Next.
  6. On the File to Import page, type the path to the appropriate certificate files (for example, \\fs1\c$\fs1.cer), and then click Next.
  7. On the Certificate Store page, click Place all certificates in the following store, and then click Next.
  8. On the Completing the Certificate Import Wizard page, verify that the information you provided is accurate, and then click Finish.
  9. Repeat steps 2 through 6 to add additional certificates for each of the federation servers in the farm.

Once the policy is setup, you will need to either wait for machine reboots, or for the GP tp update. As an alternative, you can also run gpupdate /force to cause the update to occur immediately. Once complete, you can verify the cert was installed by running certmgr.msc and inspecting the Trusted Root Certification Authorities tree for the cert. It was my experience that the machine still required a reboot due to the browser still not recognizing the new root CA and therefore still displaying the ugly SSL browser error. After a reboot it was good to go.

Reference: https://docs.microsoft.com/en-us/windows-server/identity/ad-fs/deployment/distribute-certificates-to-client-computers-by-using-group-policy