Tuesday 31 May 2016

Error while performing Cert Replacement operation for vCenter 6

While replacing SSL certificates of vCenter 6, Certificate replacement may fail and VMCA rollback the certificates to old SSL certificates.

See the complete process of replacing SSL certificates of vSphere 6 using VMCA.

Replacing SSL Certificates VMware vCenter 6.0 Update 2.

In this Post am documenting common issues which you may encounter while performing SSL certificates replacement.

1. See this KB for list of errors which you may encounter.

https://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=2144086&sliceId=1&docTypeID=DT_KB_1_1&dialogID=103792968&stateId=1%200%20103842854

To avoid issues with vCenter services, make sure you provide unique organization unit name while creating Certificate configuration file.

You can use below Organization Unit name for SSL certificates.

MACHINE_SSL_CERT.cfg   :  Root
machine.cfg                        :  Machine
vsphere-webclient.cfg       :  WebClient
vpxd.cfg                              :  VPXD
vpxd-extension.cfg            :  VPXD-EXT
certool.cfg                          :  IT-VMCA

2. Error while performing Cert Replacement operation, please see /var/log/vmware/vmcad/certificate-manager.log for more information.

You can see below Error message in /var/log/vmware/vmcad/certificate-manager.log

2016-05-28T22:12:54.141Z INFO certificate-manager Running command :- ['/usr/lib/vmware-vmca/bin/certool', '--server=localhost', '--gencert', '--privkey=/storage/certmanager/vpxd.priv', '--cert=/storage/certmanager/vpxd.crt', '--config=/var/tmp/vmware/vpxd.cfg']
2016-05-28T22:12:54.153Z INFO certificate-manager Command output :-
Using config file : /var/tmp/vmware/vpxd.cfg

2016-05-28T22:12:54.153Z ERROR certificate-manager Using config file : /var/tmp/vmware/vpxd.cfg

2016-05-28T22:12:54.153Z ERROR certificate-manager Error while performing Cert Replacement operation, please see /var/log/vmware/vmcad/certificate-manager.log for more information.
2016-05-28T22:12:54.153Z ERROR certificate-manager {
    "resolution": null,
    "detail": [
        {
            "args": [
                "Using config file : /var/tmp/vmware/vpxd.cfg\n"
            ],
            "id": "install.ciscommon.command.errinvoke",
            "localized": "An error occurred while invoking external command : 'Using config file : /var/tmp/vmware/vpxd.cfg\n'",
            "translatable": "An error occurred while invoking external command : '%(0)s'"
        },
        "Error in generating cert for store vpxd"
    ],
    "componentKey": null,
    "problemId": null
}

 

You will receive this error if your SSL certificate configuration file is incorrect and then VMCA will rollback certificates.

Solution

Login to vCSA using SSH with root user.

Go to /var/tmp/vmware directory, Create temporary directory and move all .cfg configuration files to temp directory, later on you can remove these files.

#cd /var/tmp/vmware

#mkdir temp

#mv *.cfg temp

Now run the certificate-manager command once again and start Certificate replacement process.

Follow Replacing SSL Certificates VMware vCenter 6.0 Update 2 to replace SSL certificates.

Thanks…!

Monday 30 May 2016

Replacing SSL Certificates VMware vCenter 6.0 Update 2

SSL certificates are the one of the most important part in VMware vCenter server and VMware ESXi. All traffic between ESXi Host, vCenter and between all vCenter services are encrypted using SSL certificates. Over the time VMware has improved the process to replace SSL certificates for different vCenter components.

If you need to update SSL certificates of vCenter 6.0 then you can follow below KB to replace certificates.

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2111219

However the process to replace SSL certificate of vCenter 6.0 Update 1b is changed and is not well documented, so let’s See How to Replace SSL Certificates of VMware vCenter 6.0 Update 2.

In VMware vCenter 6 VMware has introduced VMware Certificate Authority (VMCA), We are going to make this VMCA as an intermediate CA of our Internal Corporate Microsoft CA server. Then VMCA will take care of replacing SSL certificates of different vCenter services and also it will issue certificates to ESXi Hosts which you add in vCenter.

Important – In vSphere SSL configuration OrgUnit need to be unique for each service or certificate replacement will fail.

In this process, we will be using below Organization Unit for vCenter services.

MACHINE_SSL_CERT.cfg  :  Root
machine.cfg           :  Machine
vsphere-webclient.cfg :  WebClient
vpxd.cfg              :  VPXD
vpxd-extension.cfg    :  VPXD-EXT
certool.cfg           :  IT-VMCA

See this Post for common issues during certificate replacement.

Error while performing Cert Replacement operation for vCenter 6

 

How to make VMCA as Intermediate CA?

1. Login to VMware vCenter Appliance using SSH with root username and password. You can use Putty utility to login to Linux system using SSH.

You need to have ssh enabled on vCenter Appliance.

Follow this document to enable SSH on vCenter server, if it is not enabled already.

http://blog.ukotic.net/2015/06/14/enable-ssh-on-vcenter-server-appliance-6-vcsa/

2. Once you logged in to vCSA using SSH, enable shell access with below commands

>shell.set --enabled True

>shell

image

3. Edit cartool.cfg file

#cd /usr/lib/vmware-vmca/share/config

backup cartool.cfg file before editing, so that you can revert back to original if needed.

#cp certool.cfg certool.cfg.bak

#vi certool.cfg

Press i to enter in edit mode.

Move cursor to specific line and edit details as below, Press Esc key and then :wq to save and close the file.

#
# Template file for a CSR request
#

# Country is needed and has to be 2 characters
Country = US
Name    = vlab-vc01.vprhlabs.com
Organization = your-ognization-name
OrgUnit = IT
State = Missouri
Locality = Raytown
IPAddress = 192.168.10.10   
Email =
vprh@vprhlabs.com
Hostname = vlab-vc02.vprhlabs.com

image

4. Execute certificate-manager command to start SSL Certificate replacement process.

#/usr/lib/vmware-vmca/bin/certificate-manager

image

5. Select option 2 to Replace VMCA Root certificate with Custom Signing Certificate and replace all Certificates and Press Enter.

Follow the prompt and provide your inputs, I have highlighted my inputs. Also to accept the default values [values in name bracket] press Enter

6. Do you wish to generate all certificates using configuration file : Option[Y/N] ? : Y

7. Please provide valid SSO and VC priviledged user credential to perform certificate operations.
Enter username [Administrator@vsphere.local]:
Enter password:****      you cannot see password * characters on Linux, enter correct password.

8. Please configure MACHINE_SSL_CERT.cfg with proper values before proceeding to next step.

Press Enter key to skip optional parameters or use Default value.

  • Enter proper value for 'Country' [Default value : US] :
  • Enter proper value for 'Name' [Default value : vlab-vc01.vprhlabs.com] :
  • Enter proper value for 'Organization' [Default value : vPRHLABS] :
  • Enter proper value for 'OrgUnit' [Default value : ITS] : Root
  • Enter proper value for 'State' [Default value : North Carolina] :
  • Enter proper value for 'Locality' [Default value : Raleigh] :
  • Enter proper value for 'IPAddress' [optional] : 192.168.105.10
  • Enter proper value for 'Email' [Default value : prh@vprhlabs.com] :
  • Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : vlab-vc01.vprhlabs.com

image

Continue the the wizard to create machine.cfg file.

9. Please configure machine.cfg with proper values before proceeding to next step.

Press Enter key to skip optional parameters or use Default value.

  • Enter proper value for 'Country' [Default value : US] :
  • Enter proper value for 'Name' [Default value : vlab-vc01.vprhlabs.com] :
  • Enter proper value for 'Organization' [Default value : vPRHLABS] :
  • Enter proper value for 'OrgUnit' [Default value : ITS] : Machine
  • Enter proper value for 'State' [Default value : North Carolina] :
  • Enter proper value for 'Locality' [Default value : Raleigh] :
  • Enter proper value for 'IPAddress' [optional] : 192.168.105.10
  • Enter proper value for 'Email' [Default value : prh@vprhlabs.com] :
  • Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : vlab-vc01.vprhlabs.com

image

Continue the the wizard, to create vsphere-webclient.cfg file.

10. Please configure vsphere-webclient.cfg with proper values before proceeding to next step.

Press Enter key to skip optional parameters or use Default value.

  • Enter proper value for 'Country' [Default value : US] :
  • Enter proper value for 'Name' [Default value : vlab-vc01.vprhlabs.com] :
  • Enter proper value for 'Organization' [Default value : vPRHLABS] :
  • Enter proper value for 'OrgUnit' [Default value : ITS] : WebClient
  • Enter proper value for 'State' [Default value : North Carolina] :
  • Enter proper value for 'Locality' [Default value : Raleigh] :
  • Enter proper value for 'IPAddress' [optional] : 192.168.105.10
  • Enter proper value for 'Email' [Default value : prh@vprhlabs.com] :
  • Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : vlab-vc01.corp.allscripts.com

image

Please configure vpxd.cfg with proper values before proceeding to next step.

11. Press Enter key to skip optional parameters or use Default value.

  • Enter proper value for 'Country' [Default value : US] :
  • Enter proper value for 'Name' [Default value : vlab-vc01.vprhlabs.com] :
  • Enter proper value for 'Organization' [Default value : vPRHLABS] :
  • Enter proper value for 'OrgUnit' [Default value : ITS] : VPXD
  • Enter proper value for 'State' [Default value : North Carolina] :
  • Enter proper value for 'Locality' [Default value : Raleigh] :
  • Enter proper value for 'IPAddress' [optional] : 192.168.105.10
  • Enter proper value for 'Email' [Default value : prh@vprhlabs.com] :
  • Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : vlab-vc01.vprhlabs.com

image

Continue the wizard to create vpxd-extension.cfg

12. Please configure vpxd-extension.cfg with proper values before proceeding to next step.

Press Enter key to skip optional parameters or use Default value.

  • Enter proper value for 'Country' [Default value : US] :
  • Enter proper value for 'Name' [Default value : vlab-vc01.vprhlabs.com] :
  • Enter proper value for 'Organization' [Default value : vPRHLABS] :
  • Enter proper value for 'OrgUnit' [Default value : ITS] :VPXD-EXT
  • Enter proper value for 'State' [Default value : North Carolina] :
  • Enter proper value for 'Locality' [Default value : Raleigh] :
  • Enter proper value for 'IPAddress' [optional] : 192.168.105.10
  • Enter proper value for 'Email' [Default value : prh@vprhlabs.com] :
  • Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : vlab-vc01.vprhlabs.com

image

13. Continue the wizard and select option 1 to generate Certificate Signing request and key for VMCA root certificate.

  • 1. Generate Certificate Signing Request(s) and Key(s) for VMCA Root Signing certificate
  • 2. Import custom certificate(s) and key(s) to replace existing VMCA Root Signing certificate

Option [1 or 2]: 1

14. Please provide a directory location to write the CSR(s) and PrivateKey(s) to:
Output directory path: /root/

15. Please configure certool.cfg with proper values before proceeding to next step.

  • Press Enter key to skip optional parameters or use Default value.
  • Enter proper value for 'Country' [Default value : US] :
  • Enter proper value for 'Name' [Default value : vlab-vc01.vprhlabs.com] : VMCA
  • Enter proper value for 'Organization' [Default value : vPRHLABS] :
  • Enter proper value for 'OrgUnit' [Default value : ITS] : IT-VMCA
  • Enter proper value for 'State' [Default value : North Carolina] :
  • Enter proper value for 'Locality' [Default value : Raleigh] :
  • Enter proper value for 'IPAddress' [optional] : 192.168.105.10
  • Enter proper value for 'Email' [Default value : prh@vprhlabs.com] :
  • Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : vlab-vc01.vprhlabs.com

image

16. This will generate Certificate Signing request (CSR) and key for VMCA certificate and store at the location specified e.g. /root/ as specified above.


2016-05-30T16:51:51.299Z   Running command: ['/usr/lib/vmware-vmca/bin/certool', '--genkey', '--privkey', '/root/vmca_issued_key.key', '--pubkey', '/tmp/pubkey.pub']
2016-05-30T16:51:51.937Z   Done running command
2016-05-30T16:51:51.937Z   Running command: ['/usr/lib/vmware-vmca/bin/certool', '--gencsr', '--privkey', '/root/vmca_issued_key.key', '--pubkey', '/tmp/pubkey.pub', '--config', '/var/tmp/vmware/certool.cfg', '--csrfile', '/root/vmca_issued_csr.csr']
2016-05-30T16:51:52.343Z   Done running command

CSR generated at: /root/vmca_issued_csr.csr

image

17. Keep the putty session running, don’t close it.

18. Login to vCenter Appliance using WinSCP and download the CSR file on your local system.

18.1. To access vCSA using WinSCP, you need to first set root shell to bash.

18.2. Open one more Putty session and login to vCenter Appliance with root user and password.

>shell.set --enabled True

>shell

#chsh -s /bin/bash root

18.3 Now Download and install WinSCP on your system.

18.4 Launch WinSCP, Enter IP or Hostname of vCenter system, Enter username and password and connect to vCenter.

18.5 Download /root/vmca_issued_csr.csr file from vCSA server to your local system.

19. Submit CSR file to your Certificate Authority to receive SSL certificate.

If you are using Microsoft CA of your organization then first you need to have certificate Template configured which can be used to issue certificates for Intermediate CA or Subordinate CA.

https://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2112016

In My testing setup, am using Internal Microsoft Certificate Authority to issue certificate to VMCA. Make sure you have access to use that certificate template.

Get the CA certificate from Certificate Authority.

19.1 Open Command prompt and Change to the directory where you have downloaded CSR file.

>cd F:\tmp\vlab-vc01

>certreq -submit -config "CAServer\CAServer" -attrib "CertificateTemplate:VMCAIntermediateCA" vmca_issued_csr.csr vlab-vc01-CA.crt

  • VMCAIntermediateCA  is the name of my SSL certificate Template for VMware in Microsoft CA.
  • CAServer – is name of CA server, replace this value with your CA server name.

Alternatively, you can visit the CA URL and request SSL certificate.

19.2 Also you need to download complete Chain of ROOT CA certificates chain.

20. Create a Certificate chain using VMCA certificate, Intermediate CA and root CA certificate.

Open Command prompt and change to directory where you have saved all certificates.

>copy vlab-vc01-VMCA.cer+root64-2.cer+root64-1.cer root_signing_chain.cer

  • root64-2.cer – is first intermediate Certificate of CA
  • root64-1.cer – is root CA certificate.

You can also open certificate files using Notepad++ and copy/paste VMCA certificate, Intermediate and root CA’s certificate to create certificate chain.

Note, while copying certificate, there should not be any empty lines and you should follow below certificate sequence otherwise certificate replacement will not work.

-----BEGIN CERTIFICATE-----

Copy VMCA certificate here

-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----

Copy Intermediate CA certificate here

-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----

Copy Root CA certificate here

-----END CERTIFICATE-----

save this certificate chain file in root_signing_chain.cer file, make sure there are no empty lines or you don’t add any extra characters. 

21. Upload Certificate chain to vCSA server /root/ location using WinSCP.

22. Now, go back to first Putty session where we start the certificate-manager to replace certificate.

23. Press 1 to Import VMCA certificate

  • 1. Continue to importing Custom certificate(s) and key(s) for VMCA Root Signing certificate
  • 2. Exit certificate-manager

Option [1 or 2]: 1

24. Please provide valid custom certificate for Root.
File : /root/root_signing_chain.cer

Please provide valid custom key for Root.
File : /root/vmca_issued_key.key

25. You are going to replace Root Certificate with custom certificate and regenerate all other certificates
Continue operation : Option[Y/N] ? : Y

image

This will import the VMCA certificate chain and then VMCA will replace SSL certificates of vCenter services.

26. Watch the certificate replacement message and make sure there are no errors.

Note, if Certificate manager failed to install new SSL certificates, it will rollback the changes and install old certs and will start services without any issues. Just wait for some time to start all services.

Get site nameCompleted [Replacing Machine SSL Cert...]
vlab-vc-site
Lookup all services
Get service vlab-vc-site:dc733106-31e4-4a48-b7ea-a0fdde9313bc
Update service vlab-vc-site:dc733106-31e4-4a48-b7ea-a0fdde9313bc; spec: /tmp/svcspec_UN174R
Get service vlab-vc-site:b91bc335-1d8d-46b8-8257-879474422483
Update service vlab-vc-site:b91bc335-1d8d-46b8-8257-879474422483; spec: /tmp/svcspec_dA3Ckh
Get service vlab-vc-site:7b9f83ed-c3da-4fd9-bed6-785fcf6a5019
Update service vlab-vc-site:7b9f83ed-c3da-4fd9-bed6-785fcf6a5019; spec: /tmp/svcspec_Wz_2kH
Get service 47c36d0a-bb88-486a-a0f5-eb7f32b8716b
Update service 47c36d0a-bb88-486a-a0f5-eb7f32b8716b; spec: /tmp/svcspec_jQc3Tm
Get service 1d1c2534-cb7e-42f9-99f2-5c08234179ee
Update service 1d1c2534-cb7e-42f9-99f2-5c08234179ee; spec: /tmp/svcspec_3CC3O_
Get service 5c7befd1-cd3b-40a1-b5da-e2ade5812fa8
Update service 5c7befd1-cd3b-40a1-b5da-e2ade5812fa8; spec: /tmp/svcspec_Gy2SCx
Get service b3c094a9-0e70-40fc-a213-b7c0f91208f4
Update service b3c094a9-0e70-40fc-a213-b7c0f91208f4; spec: /tmp/svcspec_tbdzvT
Get service aa81dd46-96f3-4d96-a055-cca9e2230e8e
Update service aa81dd46-96f3-4d96-a055-cca9e2230e8e; spec: /tmp/svcspec_IJPMbR
Get service 61bbcc34-9762-4517-8e57-f13ce25906b8
Update service 61bbcc34-9762-4517-8e57-f13ce25906b8; spec: /tmp/svcspec_uiMUqy
Get service 582aa620-b6d2-4e4f-8e3e-4d7dac838015
Update service 582aa620-b6d2-4e4f-8e3e-4d7dac838015; spec: /tmp/svcspec_mCTfhR
Get service a64b10d5-4a3f-4d5d-b370-a03e17887cc1
Update service a64b10d5-4a3f-4d5d-b370-a03e17887cc1; spec: /tmp/svcspec_bxOJ2K
Get service d3becf3f-c436-4f40-895e-8231fdbbeb38
Update service d3becf3f-c436-4f40-895e-8231fdbbeb38; spec: /tmp/svcspec_fS7RY1
Get service 61bbcc34-9762-4517-8e57-f13ce25906b8_authz
Update service 61bbcc34-9762-4517-8e57-f13ce25906b8_authz; spec: /tmp/svcspec_CbuXPJ
Get service a106c4af-1a97-41ad-9832-0ed06b31c32e
Update service a106c4af-1a97-41ad-9832-0ed06b31c32e; spec: /tmp/svcspec_YJvG5j
Get service ae8a88e1-1566-4fa7-972d-f892575dc108
Update service ae8a88e1-1566-4fa7-972d-f892575dc108; spec: /tmp/svcspec_6LAnr3
Get service 3010064d-717f-45e6-ac0d-4e2ab4063eb4
Update service 3010064d-717f-45e6-ac0d-4e2ab4063eb4; spec: /tmp/svcspec_qbll2S
Get service b3722397-a2d0-49dc-9106-2ddcc5db0e06
Update service b3722397-a2d0-49dc-9106-2ddcc5db0e06; spec: /tmp/svcspec_BgGjx1
Get service bfca6b85-0021-4ee5-84f7-275fcd4fac53
Update service bfca6b85-0021-4ee5-84f7-275fcd4fac53; spec: /tmp/svcspec_WGHmFK
Get service 3a74e01e-613c-4db8-a363-a2af7a63f073
Update service 3a74e01e-613c-4db8-a363-a2af7a63f073; spec: /tmp/svcspec_HxXApQ
Get service b5d47110-e534-46e3-915d-071bcbee5a80
Update service b5d47110-e534-46e3-915d-071bcbee5a80; spec: /tmp/svcspec_yGFJrp
Get service 8a94eb91-4096-48b9-b52d-20e030e9d063
Update service 8a94eb91-4096-48b9-b52d-20e030e9d063; spec: /tmp/svcspec_VlitkR
Get service 61bbcc34-9762-4517-8e57-f13ce25906b8_kv
Update service 61bbcc34-9762-4517-8e57-f13ce25906b8_kv; spec: /tmp/svcspec_Ym7LU3
Get service 72798453-18c0-4558-9bfe-bd2cd281621f
Update service 72798453-18c0-4558-9bfe-bd2cd281621f; spec: /tmp/svcspec_MEXsl8
Get service a64b10d5-4a3f-4d5d-b370-a03e17887cc1_com.vmware.vsan.health
Don't update service a64b10d5-4a3f-4d5d-b370-a03e17887cc1_com.vmware.vsan.health
Get service f32bcd6e-a903-4abe-b86d-09b37f1b6611
Update service f32bcd6e-a903-4abe-b86d-09b37f1b6611; spec: /tmp/svcspec_aT5C2O
Get service 209bfaf2-5e74-469d-bd14-f8bce3c67431
Update service 209bfaf2-5e74-469d-bd14-f8bce3c67431; spec: /tmp/svcspec_6O_TjH
Get service 8d954074-5555-4676-946c-add534bd22f1
Update service 8d954074-5555-4676-946c-add534bd22f1; spec: /tmp/svcspec_YGBFci
Updated 26 service(s)
Status : 100% Completed [All tasks completed successfully]

With this we have successfully configured VMCA as Subordinate CA of internal Microsoft CA and replaced SSL certificates of All vCenter services.

27. Now open the URL of vCenter server to see if it has valid certificate. You can notice the Lock icon of SSL certificate of URL, also see the issued By and Issued to details.

image

28. See the certificate Chain, you should see the Certificates of your Root CA in chain.

image

Now login to vCenter web client and make sure all Services are running fine.

Also you can login to vCenter web client with Administrator@vsphere.local account and go to Certificate Authority to view root Certificate, In Use certs, Issued or revoked certificates.

In my next post, I will share some common errors which we may get while replacing SSL certificates.

 

Thanks…!

Failed to start the virtual machine. The specified device is not a valid physical disk device

In ESXi 6, if you try to Power On Virtual machine you may get below Error.

The specified device is not a valid physical disk device.

An error was received from the ESX host while powering on VM VM-name.
Failed to start the virtual machine.
Module Disk power on failed.
Cannot open the disk '/vmfs/volumes/4f15231a-c162b6a6-0c01-5ef3fcc2c22b/vm-name/vm-name1.vmdk' or one of the snapshot disks it depends on.
The specified device is not a valid physical disk device

image

 

Also if you try to consolidate Virtual machine disk, you will receive below error.

The virtual disk is either corrupted or not a supported format.

Issue

You encounter this issue if you have configured ESXi scratch location at the root location of Local datastore or SAN datastore instead of folder on datastore.

e.g. /vmfs/volumes/datastore-name

Workaround

Take snapshot of Vm and delete it and immediately try to Power On. This will not fix the issue permanently.

Permanent fix

Reconfigure scratch location on ESXi host and point it to folder inside datastore.

e.g. /vmfs/volumes/datastore1-uuid/esxi1-scratch

1. Login to vCenter or ESXi Host.

2. Browse the datastore where you want to configure Scratch Location.

3. Create new folder for specific esxi Host e.g. esxi01-scratch

4. Get the datastore UUID

From ESXI configuration, select the datastore and from the datastore details note down the datastore UUID.

image

5. Select ESXi Host and go to Configuration –> Advance settings.

From Advance settings, select ScratchConfig

6. Enter the Folder path created on datastore to configure scratch location in tab of ScratchConfig.ConfiguredScatchLocation 

e.g.  /vmfs/volumes/506edd0b-9cdc17de-a437-001a64760000/esxi-scratch

image

7. Click on OK.

8. Put ESXi Host in Maintenance mode.

9. Restart ESXi host.

 

Thanks…!

Thursday 26 May 2016

vCenter SQL Agent Jobs on SQL Always ON or Mirrored SQL

 

If you are using Microsoft SQL Always ON or Mirrored SQL then you need to create vCenter SQL Agent jobs on both primary and secondary SQL servers. During installation of vCenter server it only create Agent jobs on the primary SQL Instance.

You need to manually create vCenter SQL Agent Jobs on secondary SQL instance. Refer below KB for manually creating vCenter SQL Agent Jobs.

https://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1004382

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2006097

Once you have Jobs created on both the SQL servers, we need to make sure SQL jobs run only on primary SQL server. Because the database will keep running on primary sql server and if Jobs run secondary instance, they are going to fail.

To avoid failure of vCenter SQL agent Jobs, put the condition in Job scripts to Run a SQL Job only when that SQL server is primary SQL instance.

You need to put this condition in all vCenter SQL agent Jobs and on both the primary and secondary SQL servers.

 

Thanks…!

 

 

 

Cannot mount vmfs volume

If ESXi detect the LUN incorrectly then it do not allow to mount the VMFS volume without deleting exiting VMFS volume and formatting LUN.

In below Image we can see, Keep the existing signature and Assign a new signature Options are grayed out and only Format the disk option is available.

If we select Format the disk option, all the VMs will get deleted from VMFS volume.

 

Solution

We need to mount such VMFS volumes forcefully.

1. Login to ESXi host using local shell or using SSH with root username and password.

2. Find VMFS volume ID of affected LUN.

#esxcfg-volume –l

image

3. Then Mount VMFS volume with UUID.

#esxcfg-volume -M UUID

e.g. #esxcfg-volume –M 5135cc26-5a43cc6d-34c7-0025b5990306

This will forcefully mount existing VMFS volume on ESXi Host and then you can use this volume to run VMs.

 

See this KB for more information - 

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1011387

 

Thanks….!

Wednesday 25 May 2016

vRealize Automation - Migrate vCenter to vCSA

In this post we will see how to migrate Windows based vCenter to vCSA ( vCenter Server Appliance) which is used as backend for vRealize Automation ( formally known as vCAC).

Currently most of the organization are using Windows based vCenter (vCS) as a backend for vRealize Automation.  We will use VMware’s vCS to vCSA convertor to migrate Windows vCenter 5.5 to vCSA 5.5 and then will upgrade vCSA 5.5 to vCSA 6.0 update 2.

Test setup -

  • vCenter Server 5.5 – all vCenter components running on same Windows system
  • vRealize automation 6.2.2.0 – distributed installation of vRA components like app server, iaas, proxy, manager servers….etc. vRealize has its own SSO server.
  • vRealize Orchestrator 6.0.3
  • Make sure your test setup is similar to your production setup and works fine– off course with couple of ESXi Host and VMs. Nested ESXi will also work.

So Before we make any changes in Production, we need to test everything in Test Environment.

As per VMware documents during vCS to vCSA migration it will preserve vCenter's UUID, MoRef and all ESXi Host, Clusters, VM, datastore, Network should be same for vRA to work post migration.

Migration Task –

Before making changes in production setup, take backup of all required servers, DB so that we can revert back to old working state if we run in to any issues.

  • Backup vRA DB
  • Shutdown all servers of vRealize Automation and vRO.
  • Take snapshot of all vRA servers.
  • Backup vCenter SQL DB
  • Take snapshot of vCenter VM.

Migrate Windows based vCenter to vCenter Appliance ( vCS to vCSA Migration ) 

  • Download vCenter to vCenter Application migration (vCS to vCSA ) Fling from VMware site.

https://labs.vmware.com/flings/vcs-to-vcva-converter

VCS Migration Appliance Document v0.9.1

  • Deploy vCS to vCSA migration fling on vCenter or ESXi Host.
  • Deploy vCenter 5.5 Appliance.
  • Using Fling appliance do the migration from vCS to vCSA.
  • Upgrade vCSA 5.5 to vCenter 6.x

Step by Step Process to Upgrade VCSA 5.5 to VCSA 6

  • Make sure vCenter and all vCenter Components are running fine after upgrade.

Start vRealize automation -

  • Start vRealize automation servers and vRO.
  • Test vRealize functionality by deploying VMs, creating blueprint…..etc
  • Test vRO.

if you face issue with vRO then remove vRO from vRA and add again or re-register vRO with vCenter.

 

Thanks…!

 

 

 

 

 

 

Tuesday 24 May 2016

Unable to get signed certificate for host

While trying to add ESXi host in vCenter 6, you may get below error.

Error -

A general system error occurred: Unable to get signed certificate for host: <HOSTNAME>. Error: Failed to connect to the remote host, reason = rpc _s_too_many_rem_connects (0x16c9a046).

Reason -

This error occurs if you have replaced SSL certificates of VMware vCenter Server using VMCA and made VMCA as an Intermediate Certificate Authority.

Solution -

Once you make VMCA as an intermediate Certificate authority, you need to wait for 24 hours to add new ESXi Host in vCenter.

This behavior is changed in VMware vCenter 6.0 Update 2 and later with the advanced setting vpxd.certmgmt.certs.minutesBefore

Login to vCenter server using Administrator credentials, Go to vCenter server settings and update key  vpxd.certmgmt.certs.minutesBefore value to 10.

Workaround -

You can wait for 24 hours to add new Host in vCenter server or Add ESXi Hosts in vCenter before making VMCA as an Intermediate CA

 

Thanks…!

Provide vCenter permissions to Domain Users

This Post is part of VMware vSphere Install, Configure, Manage Training.

Make sure you have added vCenter server in Domain and Added Active Directory as identity source in SSO to proceed with providing permissions. If not Please complete previous task first.

1. Login to vCenter using web client and with Administrator@vsphere.local username/password.

2. Go to Home of web client –> select vCenter Instance –> then Click on Manage Tab.

3. Click on Permission Tab –> Click on + Green icon to provide permission to users.

image_thumb5

4. Select the Role which you want to provide to new Users. In this case I have provided Administrator role. You can also create custom roles and permissions for users.

5. Click on Add Button in Add Permission pop up window

Select domain from which you want to add users.

You can search user or group name from search Box.

Select users/group and click on Add

image_thumb8

6. Click OK to close Select Users/Groups Window.

As per your requirements you check or uncheck Propagate to Children Check box.

In this case I want to propagate permission to all child objects.

7. Click ok to close Add permission process.

8. Now with this, we have provided permission to Windows Users,

Open new browser and open web client to login vCenter server.

Enter your domain Username as domain-name\user-name and enter password to login to web client.

 

Thanks…!

What’s Next - Add ESXi Host in vCenter

Home - VMware vSphere Install, Configure, Manage

Add Active Directory as Identity Source

This Post is part of VMware vSphere: Install, Configure, Manage training.

Before proceeding with this step, make sure you have followed previous post and added vCenter in Active Directory domain.

See – How to Add vCSA in Active Directory domain.

Let’s Add Active directory as Identity Source in vCenter SSO.

1. Login to web client using Administrator@vsphere.local account

2. Go to Administration –> Click Configuration

3. Click on Identity Sources –> Click on Green + Sign to Add Identity Source.

image_thumb6

4. On the Add Identity Source Page, Select Active Directory ( Integrated Windows Authentication )

This will automatically Select Domain name of vCenter

Select Use machine account and Click on OK to complete the process.

image_thumb7 

5. Monitor the task in recent task pane.

if No errors received you are good to proceed.

What’s Next - Provide vCenter permissions to Domain Users

Home - VMware vSphere Install, Configure, Manage

Add vCenter Appliance in Active Directory ( vCSA 6 )

This Post is part of VMware vSphere Install, Configure, Manage Training.

Let’s see how to add vCSA in Active Directory domain.

1. Login to vCenter using web client with Administrator@vsphere.local account.

e.g. https://vlab-vc01/vsphere-client

2. From the Home page Click on Administration

3. Click on System Configuration.

4. Click on vCenter server name

5. Click on Manage –> Click on Active Directory from Settings menu

6. Click on Join, Enter domain name and Domain Admin username/password to add vCenter appliance in Active directory domain. Then Click on OK.

image

Monitor Join Activity directory task in recent task menu.

Note - To avoid issues make sure you have entered correct DNS IP address in vcsa network settings, enter correct domain name, username/password.

7. Once task gets Successfully completed, Go back to System Configuration menu.

8. Select vCenter server and Click the Green+Red Refresh Icon to restart vCenter Appliance.

or you can access VM console from Web Client and restart vCenter VM.

image

Note - if you restart vCenter server then all web client user sessions will get disconnected and you will not be able to manage VMs until vCenter comes online. There will not be any issues with other VMs, all VMs will keep running.

9. If you get Pop window for confirmation, Click Ok.

10. Wait for vCenter VM to complete boot process and start vCenter services.You can login to ESXi Host where vCenter VM was running and access VM console to monitor boot process.

Wait…Once vCenter comes online, if you try to access web client, you may get below error.

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x7f5823752dc0] _serverNamespace = /vsphere-client _isRedirect = false _port = 9090)

It means web client service is not yet initialized . it took me 5 min to get web client page after vCenter was restarted, however if vCenter server comes online,l you can quickly login using native vSphere client.

11. Verify vCSA Joined to Domain - Now login to vCenter using web client with Administrator@vsphere.local account

12. Go to Administration –> Click on System Configuration –> Click on vCenter Instance. 

13. Then from Manage Tab –> Click on Settings –> Click on Active directory

There you can see the domain name to which you have joined your system.

image

if you want to remove vCenter from Domain, you can click on leave button.

What’s Next - Add Active Directory as Identity Source

Home - VMware vSphere Install, Configure, Manage

Wednesday 18 May 2016

Creating Virtual Machine

This Post is part of VMware vSphere: Install, Configure, Manage. Let’s see what is make virtual machine and how to create virtual machine.

What is Virtual machine?

Virtual machine is a set of files created using Hypervisor e.g. ESXi, VMware VMware workstation…..etc. Each virtual machine has VM configuration file, NVRAM BIOS file, Hard Disk file, snapshot file, log files. These files make a Virtual machine and are supported by Physical hardware resource through the Hypervisor layer.

Virtual Machine Files -

File Extension Description
.vmx Virtual machine configuration file.
.nvrm Virtual machine BIOS configuration file
.vmdk Virtual Machine disk file
.log Virtual machine Log file
.vmsn VM snapshot state file
.vmsd Virtual machine metadata snapshots
.vmss Virtual machine suspended state file

Above files make the virtual machine Hardware, ESXi Hypervisor provide the physical resource to Virtual Machine.

See the list of Virtual machine hardware version and supported ESXi versions.

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2007240

While creating Virtual machine, VMware suggest you minimum required configuration for VM based on Guest operating System selection.

See the VMware vSphere 6 Configuration Maximums to know how much maximum CPU, RAM, Disk, NICs are supported by Virtual machine.

How to Create Virtual machine?

1. Login to VMware vCenter using web client. e.g. https://vlab-vc01/vsphere-client User Administrator account.

2. From the Home screen, Click on VM and Templates.

3. Click on Actions –> from the drop down menu select New Virtual Machine –> New Virtual Machine

image

4. Select a Creation type – Create a new Virtual machine then Click on Next

image

5. Select a Name and folder – Enter name for your virtual machine and Folder location.

e.g. I have entered my VM name as CentOSVM

image

6. Select a Compute Resource – Select a ESXi server which will provide Compute resources for VM to run.Click Next to proceed.

image

7. Select Datastore and click on Next

image

8. Select Compatibility level – keep the default ESXi 6.0 and later, then Click Next.

9. Select a Guest OS – ESXi Supports most of the Windows and Linux Operating Systems.

Click Next to proceed.

image

10. Customize hardware – Modify VM settings as needed. Expand each hardware component to see more information. If you need to add additional hardware, Select the new device and add to VM.

image

Click Next once you are ready.

11. Ready to Complete – review the summery information and Click on Finish.

12. Select the newly created virtual machine and right click to get the Action menu and power it on.

image

13. Once you power on the VM, you can connect to VM console and attach Operating system ISO file to install OS in Virtual Machine.

Upload OS ISO file to ESXI datastore then edit VM setting and modify cdrom settings to use ISO file from datastore for OS installation.

To use Remote Console features in vSphere 6, you need to download and install VMRC console.

Connect to VM console and install the Operating System as you do in any Physical server.

Repeat above process to create new VMs.

Also you can create new VM by Cloning existing VM, Deploying VM OVF template. We will these options more in detail in Virtual Machine Management section.

Perform other server related task to make the Virtual machine complete usable system.

From VM Action menu, you can Power On, Power Off VM, Edit VM settings…etc.

What’s Next - Virtual Machine Management

Home - VMware vSphere: Install, Configure, Manage

Add ESXi Host in vCenter

This Post is part of VMware vSphere: Install, Configure, Manage training document..
In previous Post, we have installed VMware vCenter server and also completed the installation of ESXi host. Now let’s see How to Add VMware ESXi host in vCenter server.
Make sure you have network connectivity between vCenter and ESXi server. Create DNS record for ESXI Host. if you are doing this setup in LAB, then you can connect to ESXi with IP address also.
Before Adding ESXi host in vCenter we need to create Virtual Data Center.
1. Open Chrome and Login to VMware vCenter server with Web Client using Administrator@vsphere.local account or any other account which as Administrator permissions on vCenter server.
e.g. https://vlab-vc01/vsphere-client
Ignore the SSL certificate warning message and proceed to login.
2. Click On vCenter Inventory List –> Click on vCenter Servers –> Click on vCenter instance name e.g. vlab-vc01.vprhlabs.com
3. Click on Related Objects –> Click on +DC Icon to Create new Data Center.
In new Datacenter windows give a name to your Data Center and Click on OK.
e.g. I have named my DC as vPRH-DC01

4. Select the Datacenter you created and click +Host icon to add ESXi Host in vCenter.
5. On Add Host page, Enter the name of ESXi Host or IP address and Click Next to proceed.
image
6. On Connection Setting page, Enter ESXi root username and its password, then Click Next
7. Click on Yes to accept ESXi hosts certificate
image
8. On Host Summery, review host details and click Next.
9. Assign License – at this step you can assign License to ESXi host or you can also add license key later. Click Next.
10. Lockdown mode – if enabled you cannot login to esxi host directly and you will have to manage esxi Host and its VMs from vCenter only. select disabled and click Next.
11. Select VM Location as your Datacenter –> Click Next –> Review the Summery and Click Finish.
Watch the task progress in Recent Task windows.



Click on Host tab to see the recently added Host.
Repeat steps 4 to 11, to add all other remaining Host in vCenter.

What’s Next - Creating Virtual Machines
Home - VMware vSphere: Install, Configure, Manage



Introduction to ESXi console Menu options

ESXi console has very limited option to change settings and is meant to use only for basic server management and troubleshooting purpose.

Let’s see what all we can do using ESXi Console. Access ESXi server console to do below steps.

On ESXi console

On ESXi console login windows, we can see the ESXi version and Release Build Version.

Physical server details, configuration.

ESXi IP Address, Host name details.

F2 – you can press F2 to login ESXi and customize ESXi settings

F12 – By pressing F12, you can login to ESXi and restart or shutdown ESXi Host. (make sure you want to do this).

ESXi Console Configuration options

Press F2 to login to ESXi and see the configuration options.

1. Configure password – Set password for ESXi Host.

2. Configure Lockdown Mode – if enabled, users cannot connect to ESXi host directly using Client and ESXi would be managed by vCenter server only. Default option is disabled.

3. Configure Management – configure ESXi IP address, DNS settings.

4. Restart Management Network – Restart ESXi Management network services.

5. Test Management Network – for network troubleshooting you can use this option, to ping to Gateway, DNS servers for ESXi.

6. Network Restore options - Restore ESXi Network settings to default. To the settings as we see after installation.

7. Configure keyboard – configure keyboard layout.

8. Troubleshooting Options – This is another troubleshooting option, you can enable local shell on ESXi, Enable SSH and restart ESXi Host Management Agents using this option. This option is required when ESXi host don’t respond or get disconnected from vCenter. SSH access is required for advance troubleshooting.

9. View System Logs – you can see ESXi System logs, Vmkernel, managements logs, virtual center agent logs, Config and VMware ESXi observation logs using this menu.

10. View Support Information – See ESXi License information and support details.

11. Reset System Configuration – Reset system back to default, erase all ESXi settings.

clip_image002

Next Topic -  Install VMware vCenter Server

 

Home - VMware vSphere: Install, Configure, Manage