Extending a logical volume online

Posted by & filed under Server Admin, VMWare.

I’ve written about this in past posts. Here is an updated article straight from VMWare: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1006371

 

  1. Power off the virtual machine.
  2. Edit the virtual machine settings and extend the virtual disk size. For more information, see Increasing the size of a virtual disk (1004047).
  3. Power on the virtual machine.
  4. Identify the device name, which is by default /dev/sda, and confirm the new size by running the command:

    # fdisk -l

  5. Create a new primary partition:
    1. Run the command:

      # fdisk /dev/sda (depending the results of the step 4)

    2. Press p to print the partition table to identify the number of partitions. By default, there are 2: sda1 and sda2.
    3. Press n to create a new primary partition.
    4. Press p for primary.
    5. Press 3 for the partition number, depending on the output of the partition table print.
    6. Press Enter two times.
    7. Press t to change the system’s partition ID.
    8. Press 3 to select the newly creation partition.
    9. Type 8e to change the Hex Code of the partition for Linux LVM.
    10. Press w to write the changes to the partition table.
  6. Restart the virtual machine.
  7. Run this command to verify that the changes were saved to the partition table and that the new partition has an 8e type:

    # fdisk -l

  8. Run this command to convert the new partition to a physical volume:

    Note: The number for the sda can change depending on system setup. Use the sda number that was created in step 5.

    # pvcreate /dev/sda3

  9. Run this command to extend the physical volume:

    # vgextend VolGroup00 /dev/sda3

    Note: To determine which volume group to extend, use the command vgdisplay.

  10. Run this command to verify how many physical extents are available to the Volume Group:

    # vgdisplay VolGroup00 | grep “Free”

  11. Run the following command to extend the Logical Volume:

    # lvextend -L+#G /dev/VolGroup00/LogVol00

    Where # is the number of Free space in GB available as per the previous command. Use the full number output from Step 10 including any decimals.

    Note: To determine which logical volume to extend, use the command lvdisplay.

  12. Run the following command to expand the ext3 filesystem online, inside of the Logical Volume:

    # ext2online /dev/VolGroup00/LogVol00

    Notes:

    • Use resize2fs instead of ext2online if it is not a Red Hat virtual machine.
    • By default, Red Hat and CentOS 7 use the XFS file system you can grow the file system by running the xfs_growfs command.
  13. Run the following command to verify that the / filesystem has the new space available:

    # df -h /

vCenter Server (VCSA) 6.5 – Add root CA cert to Windows Server 2016 AD to enable the default VCSA cert to be trusted

Posted by & filed under Active Directory, Server Admin, Virtualization, VMWare.

The VCSA has it’s own CA built in. It uses that CA to generate certs for all the various services. There are two options available to ensure that the certificate is trusted in the browser:

  1. Generate a CSR for the cert and submit to a CA who can generate the cert.
  2. Use Microsoft Active Directory GPO to push out the VCSA’s root CA cert, thereby allowing the workstations to trust the cert already installed.

I went with the second one because the VCSA is using vcenter.mydomain.lan and is only accessible from inside my network which also means only machines on the domain will be connecting to the web interface. This was very simple to make happen…

On the DC:

To distribute certificates to client computers by using Group Policy

  1. On a domain controller in the forest of the account partner organization, start the Group Policy Management snap-in.
  2. Find an existing Group Policy Object (GPO) or create a new GPO to contain the certificate settings. Ensure that the GPO is associated with the domain, site, or organizational unit (OU) where the appropriate user and computer accounts reside.
  3. Right-click the GPO, and then click Edit.
  4. In the console tree, open Computer Configuration\Policies\Windows Settings\Security Settings\Public Key Policies, right-click Trusted Root Certification Authorities, and then click Import.
  5. On the Welcome to the Certificate Import Wizard page, click Next.
  6. On the File to Import page, type the path to the appropriate certificate files (for example, \\fs1\c$\fs1.cer), and then click Next.
  7. On the Certificate Store page, click Place all certificates in the following store, and then click Next.
  8. On the Completing the Certificate Import Wizard page, verify that the information you provided is accurate, and then click Finish.
  9. Repeat steps 2 through 6 to add additional certificates for each of the federation servers in the farm.

Once the policy is setup, you will need to either wait for machine reboots, or for the GP tp update. As an alternative, you can also run gpupdate /force to cause the update to occur immediately. Once complete, you can verify the cert was installed by running certmgr.msc and inspecting the Trusted Root Certification Authorities tree for the cert. It was my experience that the machine still required a reboot due to the browser still not recognizing the new root CA and therefore still displaying the ugly SSL browser error. After a reboot it was good to go.

Reference: https://docs.microsoft.com/en-us/windows-server/identity/ad-fs/deployment/distribute-certificates-to-client-computers-by-using-group-policy

vCenter 5.5 to 6.5U1 Upgrade – SSL Errors

Posted by & filed under Server Admin, Virtualization, VMWare.

Ran into some issues with the ssl certs on the vCenter server when trying to run the Migration Assistant. Notes on the will follow, but first links to articles on the actual upgrade:

The issues I ran into with the migration assistant complained of the SSL certs not matching. Upon inspecting the certs I found all were issues for domain.lan except for one which was issued to domain.net. I followed the following articles to generate a new vCenter cert and install it:

  • Generate SSL cert using openssl: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2074942
  • Install and activate cert: https://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2061973

As the Appliance Installed reached Stage 2 of the install where it copies the data to the new VCSA, I received the following error (note the yellow warning in the background along with the details in the foreground):

To resolve this error, I followed the following articles:

  • Upgrading to VMware vCenter 6.0 fails with the error: Error attempting Backup PBM Please check Insvc upgrade logs for details (2127574): https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2127574
  • Resetting the VMware vCenter Server 5.x Inventory Service database (2042200): https://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2042200#3

Which essentially had me reset the inventory service’s database due to corruption. I had noticed the vSphere client slow in recent weeks, this could be a side effect.

  • Additional more generic docs for tshooting vCenter upgrades: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2106760

 

VCSA – Joining to AD Domain fails – Error: Enabling Active Directory failed. ERROR_GEN_FAILURE 0x00000001f

Posted by & filed under Active Directory, Server Admin, Virtualization, VMWare.

Attempting to join a freshly deployed VCSA server to a AD domain can be problematic if SMB1 is disabled. In my case it was 5.5 but I believe this issue persists in 6.x. SMB1 was disabled on the DC as it should be as it is broken and insecure. The problem lies in the fact that VCSA doesn’t support SMB2 and this causes the error. The VAMI (web interface) might report something like the following when attempting to join the domain:

Additionally, on the VCSA, /var/log/vmware/vpx/vpxd_cfg.log contains entries like the following:

Of course DNS resolution of the VCSA’s hostname should be validated before continuing, but assuming everything else is in working order, the fix is to enable SMB2 on the VCSA.

Verify SMB2 is disabled (note the Smb2Enabled key is 0:

Enable SMB2:

Restart the lwio service:

Log out of VAMI web interface, log back in and retry joining to the domain.

Tomcat 7 SSL – Swapping a Cert out in the keystore

Posted by & filed under Server Admin.

I have a PKCS12 .pfx export of a cert that I need to import into a Tomcat keystore in order to update an expiring certificate.

 

Need to know a few things beforehand:

  • Tomcat keyfile path
  • Source store password for the pfx file
  • Source alias for the pfx
  • Dest source passwd
  • Dest source alias

In order to get the source alias from the new pfx file:

If you need to get the alias from the existing Tomcat keystore:

Additionally, the above command can be used to verify the certificate, expiry date, etc.

Lastly, if you restart Tomcat and it throws errors like the following in the catalina log, you may need to reset the keystore password:

Reset to the correct password as defined in the servver.xml keyStorePass parameter using the following command. You may need to adjust alias to your needs. You will be prompted for the new password, which should match the previously mentioned keyStorePass parameter.

You can also reset the password for the keystore itself (www.ibm.com/support/knowledgecenter/en/S…):

 

 

EDIT FROM THE FUTURE:

Additional note — when trying to run the import command I was getting the following error:

I ran the following to verify the alias is correct:

Key ID of 2 is displayed correctly here as well as a more verbose output also showed the same:

I then took the same .pfx file and checked it on a linux machine based on a hint from this stackoverflow on binary chars: http://stackoverflow.com/questions/15301005/keytool-cant-find-alias

And lo’ and behold it shows the alias is actually 1!

 

..Back in Windows land:

It accepted alias 1 instead and the cert imported correctly. I love Tomcat -_-

 

 

 

 

Linux – Unable to boot due to missing drive in fstab

Posted by & filed under Linux, Server Admin.

I had a old server I brought up and it was unable to complete it’s boot due to a missing drive in fstab. Editing the fstab in recovery mode is not a option since the filesystem gets flagged as read only.

In order to make the FS writable and therefore be able to successfully edt the fstab, the following command will remount the FS in read/write mode:

 

Windows XP: Recovering The registry using Linux when windows won’t boot

Posted by & filed under Server Admin.

I recently had a Windows XP laptop crash. Windows would not boot to safe mode or anything, and just displayed the following error message:

I could not afford to simply wipe the laptop and reinstall windows as it had some old software that was no longer available.I located the following article which details a procedure to recover from this issue using the MS recovery console and using the System Restore: https://support.microsoft.com/en-us/kb/307545

As this laptop did not have a optical cd-rom, it was a difficult proposition to make a XP bootable USB stick to complete this procedure since I do not have the media handy. Additionally, it seemed like a pain to go thru all the steps when it could be simplified quite a bit with a functioning OS like linux. I decided to attempt to recover using a linux live cd:

  1. Create a bootable USB stick with Ubuntu on it using uNetBootin
  2. Boot to the USB stick.
  3. Make backups of any critical files (just in case)
  4. Backup registry files at C:\windows\system32\config to usb stick:
  5. Access the System Volume Information which should contain restore points for the system. See Part 2 Steps 7 through 10 in above MS article for details, but in a nutshell you want to access C:\System Volume Information. There will be one or more folders inside and their names will be similar to “_restore{D86480E3-73EF-47BC-A0EB-A81BE6EE3ED8}”. Inside these folders, look for RPx folders. There may be more than 1, and x would be a number. Look at the created dates of these folders to identify a fairly recent restore point. For example I found one that was two weeks old in RP47.
  6. Access the snapshot folder to retrieve registry backups. Example:
  7. Inside the snapshot directory, copy the registry files to a temp location, and make a backup of them:
  8. Copy the snapshots to C:\windows\system32\config.
  9. Delete the old crashed registry files:
  10. Rename the backup registry files to replace the ones you just deleted:
  11. Cross your fingers and reboot! If it does not work, and you still receive the same error message, you may need to try a older registry snapshot. Simply follow the above steps to try a different registry snapshot.

Good luck!

Slow DNS Resolution on Ubuntu Linux Server 14.05 LTS

Posted by & filed under Linux, Server Admin.

This all started with WordPress timeouts. I was trying to activate some premium plugins, and the license activation was timing out. I started doing some digging and found they use the WordPress core library WP_http which in turn uses curl to make the request. I wrote my own code to use WP_Http and it failed in the same way with a timeout. I added a timeout parameter to the wp_remote_get() call, and it was able to complete without a timeout. I then used a IP address in place of the domain name and it worked without the need for the timeout parameter.

With that info in hand, I decided it must be on the server. I started doing some tests:

I then did the same test from another server that uses the same DNS servers in resolv.conf:

After much googling, I found a few number of suggested solutions:

  • Disable IPv6
  • Ensure /etc/nsswitch.conf is set correctly (hosts: files dns)

Neither of these worked for me. Finally, I added the following directive into my resolv.conf and it fixed the issue!

Apparently, this is actually somewhat related to ipv6 — from the resolv.conf manpage:

Now, I get good response times when I curl:

Looks like the resolver sends parallel requests, fails to see the IPv6 response, waits 5 sec and sends sequential requests because it thinks the nameserver is broken. By adding the options single-request, glibc makes the requests sequentially be default and does not timeout.

I found some good info and hints on this issue here: https://bbs.archlinux.org/viewtopic.php?id=75770

Lastly, to bring this whole thing full circle, the WprdPress plugins now are able to get out and communicate successfully. Woohoo!

cPanel WHM – Wipe cpHulk Lockouts

Posted by & filed under Firewalls, Security, Server Admin.

cPanel WHM’s cpHulk system manages iptables blocks against IP addresses that fail to authenticate repeatedly. While the settings are fairly lenient and shouldn’t result in legitimate users being blacklisted, occasionally it can happen. The following command will reset the blocklist completely. While this is akin to using a shotgun when a scalpel is required, the blocks are time based and any malicious addresses would get quickly re-blocked.

 

There is a method to remove specific addresses, but I do not have the commands handy at present, and if I remember correctly it entails connecting to the mysql console, running a query to find the IP in the block table and issuing a drop query.

OpenSSL – Extracting a crt and key from a pkcs12 file

Posted by & filed under Linux, Server Admin.

Quick and dirty way to pull out the key and crt from a pkcs12 file:

If you are using this for Apache and need to strip the password out of the certificate so Apache does not ask for it each time it starts:

Scanning for web malware, back doors, spam scripts, etc on Linux based web servers

Posted by & filed under Security, Server Admin.

In the wake of the recent SoakSoak WordPress vulnerability, among others I have began searching for a better way to keep tabs on malicious code that may get uploaded to client’s hosting accounts.

Enter maldet.

Maldet uses a constantly updated database of malware hashes to identify and quarantine (if required) malicious files. Maldet can be set to run automatically via cron, watch newly updates files, and more.

 

 

Exim – Find a list of the most commonly used scripts

Posted by & filed under Server Admin.

I had to deal with a malicious script that was inserted into a website today and was sending out spam. Typically I have a few tools I run, but I couldn’t locate this particular infection. Time to take another angle, Exim logs. The following shows the most used script directories which will help narrow down the suspects substantially.

Exim – Purge Mail Queue

Posted by & filed under Server Admin.

A quick one-liner to purge the Exim mail queue:

 

Recursively finding strings in files

Posted by & filed under Linux, Server Admin.

For example, if you wanted to scan all files in the current directory, and all sub directories for any calls to base64_decode, you could do something like this:

find all files, then execute grep on them, printing out matching lines, filenames and line numbers, finally write output to resultb64.txt

Another twist on this is to filter the filetypes a bit:

Lastly, if we wanted to find and replace (with nothing) a string: