Cisco UCS Firmware upgrade Guide

In this post, we will see Step by Step process to Upgrade Cisco Unified Computing System (UCS) firmware. Our most of the UCS systems are used for VMware ESXi servers, so I will provide some VMware steps which need to be taken care while upgrading UCS firmware.

Download Firmware

1. Download latest firmware from Cisco.com.
2. You need to download 3 files from Cisco websites that is Infrastructure bundle, firmware for B Series Blade systems and firmware for C Series for Rack servers.

Download Firmware in UCS

1. Login to UCS system, e.g.  http://ucs01
2. In the left window of UCS Manager Click on Equipment Tab. Click Equipment on Top in the menu list.
3. In right window Click on ‘Firmware Management’ Tab then click on ‘Download Task’ Tab
1. Click on Green + symbol to start firmware download wizard.

4. In this ‘Download Firmware’ Window, Select Location of the Image file as ‘Local File System’
1. Browse the downloaded .bin file location and select firmware image file.
2. Click on OK to start firmware download and monitor the upgrade process.

5. Upload all three file – Infrastructure bundle, B series bundle for Blade and C series bundle for Rack Servers using above process.

In below image I have uploaded latest firmware 2.26c.

I don’t have any Rack UCS server, so I have skipped C series firmware bundle.

Disable HA in VMware vCenter

We need to disable VMware HA, so that during firmware upgrades VMware HA don’t initiate a host fail over. During UCS firmware upgrade each Fabric Interconnects will get restarted one by one. It means Storage HBA and NIC of restarted FI will be unavailable.

Use PowerCLI to Connect vCenter and Disable HA for all Clusters.

Connect-VIServer vlab-vCenter

Get-Cluster * | Set-Cluster -HAEnabled $false -DrsAutomationLevel:Manual -Confirm:$false -RunAsync

ESXi NFS Storage Paths

If you are using NFS based storage for UCS ESXi Hosts, then modify dvswitch/Standard switch NIC teaming to use primary active FI for NFS. And make secondary FI NICs as Standby

If you are not using NFS based storage for UCS ESXi then no need to make any changes in VMware vCenter for NICs.

Also if you are using any physical Windows or Linux systems then you should have NIC teaming or NIC bonding configured in each physical system to avoid network disturbance during Fabric Interconnect firmware upgrade and restart.

Disable Call Home

1. Go back to Cisco UCS manager and disable call home.
2. Click on Admin Tab in left window.
3. Expand Communication Management → Call Home.
4. From Right window, Click on General Tab
1. In Admin Section, for ‘State’ Select radio button next to Off
2. Click on Save Changes.
3. Click on OK.

Check UCS for Faults

Before starting UCS upgrade, make sure there are no major errors in UCS system.

If any, fix these errors before upgrading firmware.

Upgrade UCS Fabric Interconnects

During upgrade process, it will first upgrade secondary fabric Interconnect and all it’s IOM of Chassis.

1. Go back to UCS manager, in the Left pane Click on ‘Equipment’ Tab.
2. Then in the right pane click on Firmware Management tab then click on ‘Firmware Auto Install’.

1. From the General Tab, in Action section click ‘Install Infrastructure Firmware’

3. This will start new window to install firmware, on the ‘Prerequisites’ step review any error or warning message.
1. If there are no major issues, select the check box to ignore all warning messages, then click on Next.

2. On the install infrastructure firmware page, under properties select the latest version from drop down menu, select check box ‘upgrade now’.
3. Then click on Finish

4. This will start the upgrade of UCS manager and firmware of Secondary Fabric Interconnect.
5. Click on FSM tab to watch the progress.

It will kick you out from UCS, wait 3 mins, and then log back into it.

In UCS watch FSM tab for the progress.

This takes TIME. A lot of it. Best thing to do is to just wait…

After about 30 -45 mins it will reboot the secondary FI WITHOUT asking. That’s the tricky part! You’ll see some VMware blips but nothing majors for storage. Notice the 500+ errors, do not be concerned

You can do ping to FI Cluster IP, Primary IP and secondary FI to check Network connectivity and know when Secondary FI comes online.

Errors in vCenter

In VMware vCenter, you can see lot of errors, warning message for UCS blade servers.

As this is due to FI restart, you can ignore these errors until we finish upgrade process.

After waiting another 5-10 mins, the errors will increase, but then after ANOTHER 20-30 mins, they will decrease back down (hopefully)…..

DO NOT GO TO THE NEXT STEP UNTIL ALL OF THE ERRORS DECREASE AND THE PATHS ARE ALL BACK.

This will take a LONG time. Do not go forward… if you do very bad things will happen (ask me how I know….)

You can watch under the IOM of each chassis under the general part of the firmware auto install to see the progress.

Once you are sure Secondary FI and all secondary NIC, HBAs are active for all blades then proceed with next step in upgrade process.

ESXi NFS Storage Paths

Again if you are using NFS storage, then go back to VMware vCenter and now make secondary FI NICs primary for NFS storage under dvswitch or standard switch NIC teaming.  And make primary FI NICs as Standby.  

Upgrade Primary Fabric Interconnect

1. Go back to Cisco UCS
2. Click on Pending Activities from the top.
3. In Pending Activities window, select Fabric Interconnects.
1. Click on Apply
2. On the Reboot Window Click on Yes.
4. This will start the upgrade process of Primary Fabric interconnect.

Now we wait another 5-10 mins because the UCSM will drop out, you’ll have to wait for it to come back then wait another 2 mins and you can click re login.

Do ping to UCS Cluster IP, primary and secondary FI to monitor network connectivity.

Then we wait the super long time of 20-30 mins to have everything settle back down, check all the errors and warnings, make sure all the NICs and paths are connected.

Wait…Wait, do NOT proceed until all errors go away.

Links will stay down, don’t do anything until they come back up

Okay, one last time, just making sure you waited. It’s now been an hour since the start roughly and we are done!!  For the UCSM, FI, and IOM upgrades.

Click on Installed Firmware tab under ‘Firmware management’ to review current firmware version of UCS. It should be new one.

Enable HA in VMware

Once again verify that all NIC and HBA paths are active for ESXi Host. Everything should be good if not fix it.

Now go back into vCenter, enable all host alarm actions, and enable call home in UCS.

>connect-viserver vlab-vc01

>Get-Cluster * | Set-Cluster -HAEnabled $false -DrsAutomationLevel:Manual -Confirm:$false –RunAsync

Enable Call Home in UCS

1. Go back to UCS.
2. Click on Admin Tab in left window.
3. Expand Communication Management → Call Home.
4. From Right window, Click on General Tab
1. In Admin Section, for ‘State’ Select radio button next to On
2. Click on Save Changes.

Server Firmware Upgrade

Warning: Before starting this process, make sure you are using ‘User Ack’ Maintenance Reboot Policy.

If you are using default ‘Immediate’ Reboot policy in Maintenance then your Host will get restarted as soon as you complete below process.  

And am sure, no one want to restart all blades servers all at once. And then Bad things will happen. ☺

Am considering you have ‘user Ack’ Policy for all service profiles which are in use.

So let’s see step by step process to upgrade firmware of UCS Blade servers.

1. Go back to UCS manager.
2. Select Equipment Tab, select Equipment on the top.
3. In the right pane, Click on Firmware management tab, then Click on Firmware Auto Install tab.
4. In the general Tab, in Action section Click on ‘Install Server Firmware’
5. This will launch install Firmware wizard.
1. On the Prerequisites step, review all warning or error messages.
2. Select the checkbox next to ‘Ignore All’ to ignore all warning messages and then click on Next.
3. On the next page, select the B-Series Blade server software and C Series Rack Mount server software if any and Then Click on Next.

4. On the next page select all the ‘Host firmware packages’ to update with new version, then Click on Next.

5. On the next page ‘Host firmware package dependencies’, review all the service profiles to which this change will be applied. Then Click on Next.

6. On the Next page review all ‘Impacted Endpoints Summery’ and Then Click on Install

As I said earlier, if you are using ‘User Ack’ reboot maintenance policy then your blades will wait until you acknowledge the firmware upgrade else all blade servers will restart immediately and start firmware upgrade.

Upgrading Firmware of Host

As we are using VMware ESXi on UCS blades, login to VMware vCenter server with Administrator privileges.

1. Put ESXi Host in Maintenance Mode.
1. Once ESXi Host goes in MM, Power off it.
2. Go back to UCS manager.
3. Click on the ‘Pending Activities’
4. Under service profile Tab, identify your ESXi Host which you have put in Maintenance Mode, select check box under reboot now column for that service profile and then Click on Apply. (See below image)

Make sure you select the same Host/Blade which you need to restart and apply new firmware. As soon as you click on Apply button UCS will restart the blade and start upgrade process.

Couple of times I have restarted Powered on ESXi Host resulting restart of all Virtual servers running it.

Good part was each time I did this with Development VM ESXi hosts, so am saved.  Hopefully it will not happen with production VM ESXi Host. ☺

5. Go to the service profile for which you have applied new firmware.
6. Go to the FSM tab of service profile or blade and monitor the upgrade progress.

Wait for the upgrade process to get finished, you can access blade console and watch the process.

During upgrade blade will restart multiple time.

Once the upgrade get finished, blade will boot Installed operating system correctly.

In my case, I will have powered on ESXi Host in Maintenance Mode and connected in vCenter.

Using above same process you can upgrade all other ESXi/Windows/Linux Host, but make sure downtime is approved for those servers otherwise you will have issues.

Good Luck!

-

vPRH