CISCO HyperFlex Systems Upgrade for VMware ESXi User Guide
- June 13, 2024
- Cisco
Table of Contents
- CISCO HyperFlex Systems Upgrade for VMware ESXi
- Product Information
- Product Usage Instructions
- Preparing for Upgrade
- Guidelines and Limitations
- ssh admin@controller_vm_ip
- hxcli cluster version
- hxcli cluster info|grep -i health Sample output: healthstate : healthy
- ssh admin@controller_vm_ip
- hxcli cluster version
- hxcli cluster info|grep -i health Sample output: healthstate : healthy
- <<<<<—————– Change this
- [<<<<<—————– Change this } ] }, { “id”: “org”, “items”:
- HyperFlex Upgrade Troubleshooting
- References
- Read User Manual Online (PDF format)
- Download This Manual (PDF format)
CISCO HyperFlex Systems Upgrade for VMware ESXi
Product Information
The Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 provides detailed information on upgrading Cisco HyperFlex Systems for VMware ESXi. It was first published on August 22, 2023, and last modified on September 27, 2023.
The manual is provided by Cisco Systems, Inc., headquartered at 170 West Tasman Drive, San Jose, CA 95134-1706, USA. You can find more information about Cisco on their website: http://www.cisco.com. For inquiries, you can contact Cisco at Tel: 408 526-4000 or 800 553-NETS (6387), or fax them at 408 527-0883.
Please note that the specifications and information in them anual are subject to change without notice. While the statements, information, and recommendations are believed to be accurate, they are presented without warranty of any kind. Users are responsible for their application of any products.
The software license and limited warranty for the product can be found in the information packet that shipped with the product. If you are unable to locate it, you should contact your Cisco representative for a copy.
All document files and software from Cisco and its suppliers are provided “as is” with all faults. Cisco and the suppliers disclaim all warranties, expressed or implied, including those of merchantability, fitness for a particular purpose, and non-infringement.
In no event shall Cisco or its suppliers be liable for any indirect, special, consequential, or incidental damages arising out of the use or inability to use this manual, including lost profits or loss or damage to data.
The manual also states that any IP addresses and phone numbers used in the document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and figures included in the document are for illustrative purposes only. Any use of actual IP addresses or phone numbers in the content is unintentional.
All printed copies and duplicate soft copies of the document are considered uncontrolled. It is recommended to refer to the current online version for the latest updates.
Cisco has more than 200 offices worldwide. You can find addresses and phone numbers on the Cisco website: http://www.cisco.com/go/offices.
Cisco and the Cisco logo are trademarks or registered trademarks of Cisco
and/or its affiliates in the U.S. and other countries. For a list of Cisco
trademarks, you can visit this URL:
https://www.cisco.com/c/en/us/about/legal/trademarks.html.
Any third-party trademarks mentioned belong to their respective owners. The
use of the word partner does not imply a partnership relationship between
Cisco and any other company.
Product Usage Instructions
The Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 provides instructions on upgrading Cisco HyperFlex Systems for VMware ESXi. To perform the upgrade, follow the steps outlined in the manual.
It is recommended to review the manual thoroughly before starting the upgrade
process. Make sure to understand the requirements, dependencies, and any
potential risks associated with
the upgrade.
If you encounter any issues or need further assistance during the upgrade, you
can use the provided documentation feedback form available in the right pane
of every online document to provide feedback to Cisco.
In addition, the manual mentions the Cisco Bug Search Tool (BST), which is a
web-based tool that acts as a gateway to the Cisco bug tracking system. BST
provides detailed defect information about Cisco products and software. You
can use BST to search for any known defects or vulnerabilities related to your
products and software.
For more information on using BST or accessing the Cisco bug tracking system, refer to the manual or visit the Cisco website.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5
First Published: 2023-08-22 Last Modified: 2023-09-27
Americas Headquarters
Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA
http://www.cisco.com Tel: 408 526-4000
800 553-NETS (6387) Fax: 408 527-0883
THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE
SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND
RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED
WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL
RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.
THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET
FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE
INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE
SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A
COPY.
The Cisco implementation of TCP header compression is an adaptation of a
program developed by the University of California, Berkeley (UCB) as part of
UCB’s public domain version of the UNIX operating system. All rights reserved.
Copyright © 1981, Regents of the University of California.
NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF
THESE SUPPLIERS ARE PROVIDED “AS IS” WITH ALL FAULTS. CISCO AND THE ABOVE-
NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING,
WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE
AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE
PRACTICE.
IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL,
CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST
PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE
THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE
POSSIBILITY OF SUCH DAMAGES.
Any Internet Protocol (IP) addresses and phone numbers used in this document
are not intended to be actual addresses and phone numbers. Any examples,
command display output, network topology diagrams, and other figures included
in the document are shown for illustrative purposes only. Any use of actual IP
addresses or phone numbers in illustrative content is unintentional and
coincidental.
All printed copies and duplicate soft copies of this document are considered
uncontrolled. See the current online version for the latest version.
Cisco has more than 200 offices worldwide. Addresses and phone numbers are
listed on the Cisco website at www.cisco.com/go/offices.
Cisco and the Cisco logo are trademarks or registered trademarks of Cisco
and/or its affiliates in the U.S. and other countries. To view a list of Cisco
trademarks, go to this URL:
https://www.cisco.com/c/en/us/about/legal/trademarks.html. Third-party
trademarks mentioned are the property of their respective owners. The use of
the word partner does not imply a partnership relationship between Cisco and
any other company. (1721R)
© 2023 Cisco Systems, Inc. All rights reserved.
Communications, Services, Bias-free Language, and Additional Information
· To receive timely, relevant information from Cisco, sign up at Cisco Profile
Manager. · To get the business impact you’re looking for with the technologies
that matter, visit Cisco Services. · To submit a service request, visit Cisco
Support. · To discover and browse secure, validated enterprise-class apps,
products, solutions and services, visit
Cisco Marketplace. · To obtain general networking, training, and certification
titles, visit Cisco Press. · To find warranty information for a specific
product or product family, access Cisco Warranty Finder.
Documentation Feedback To provide feedback about Cisco technical
documentation, use the feedback form available in the right pane of every
online document.
Cisco Bug Search Tool Cisco Bug Search Tool (BST) is a web-based tool that
acts as a gateway to the Cisco bug tracking system that maintains a
comprehensive list of defects and vulnerabilities in Cisco products and
software. BST provides you with detailed defect information about your
products and software.
Bias-Free Language The documentation set for this product strives to use bias-
free language. For purposes of this documentation set, bias-free is defined as
language that does not imply discrimination based on age, disability, gender,
racial identity, ethnic identity, sexual orientation, socioeconomic status,
and intersectionality. Exceptions may be present in the documentation due to
language that is hardcoded in the user interfaces of the product software,
language used based on standards documentation, or language that is used by a
referenced third-party product.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 iii
Communications, Services, Bias-free Language, and Additional Information
Communications, Services, Bias-free Language, and Additional Information
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 iv
1 C H A P T E R
New and Changed Information
· New and Changed Information, on page 1
New and Changed Information
The following table provides an overview of the new features and changes made to this guide for this current release.
Feature
Description
HX Release or Date Added
Where Documented
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi
First release of the 5.5(x) guide
HX 5.5(1a)
n/a
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 1
New and Changed Information
New and Changed Information
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 2
2 C H A P T E R
Overview
· About this Guide, on page 3 · Determining the Correct Upgrade Path for my HX
Deployment, on page 3 · Upgrading from an Unsupported Cisco HyperFlex HX Data
Platform Software Release, on page 4
About this Guide
This document is designed to guide Cisco HyperFlex (HX) users who are
currently running HX Release 3.5(1a) or later and want to upgrade their
environment to a more recent HX release. If your cluster is managed by
Intersight, please refer to Upgrading Cisco HyperFlex Edge Systems with Cisco
Intersight. On-premises HyperFlex upgrade:
· HyperFlex Data Platform Upgrades using HyperFlex Connect · HyperFlex Edge
Upgrades · Upgrades for HyperFlex Stretched Clusters · Split upgrade
procedures
Determining the Correct Upgrade Path for my HX Deployment
Much of this document focuses on upgrading regular HyperFlex deployments using
HyperFlex Connect. Many of the upgrade tasks are the same for Edge and
Stretched Cluster configurations, but there are differences that should be
noted. Use the information below to navigate through this document based on
your specific configuration. 1. Select the Cisco HyperFlex Data Center release
currently running in your environment and follow the upgrade workflow for your
specific upgrade.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 3
Upgrading from an Unsupported Cisco HyperFlex HX Data Platform Software Release
Overview
Cisco HyperFlex Data Center Cisco Upgrade Guide release currently running in your environment
Supporting Upgrade Videos
Cisco HX Release 4.5(2x) or Cisco HyperFlex System Upgrade –
5.0(2x) and later.
Guide for VMware ESXi (this guide).
Continue to the next table to select
your Deployment type.
Cisco HX Release 3.5(2x), 4.0(x), 4.5(1x) or 5.0(1x) 1
Cisco HyperFlex System Upgrade Guide for VMware ESXi (this guide). Continue to the next table to select your Deployment type.
Cisco HX Release 3.5(1x) and earlier
Cisco HyperFlex Systems Upgrade Guide for Unsupported Cisco HX Releases
Cisco Communities Data Center Videos
1 These releases have reached the end of their support, but the Cisco
HyperFlex System Upgrade Guide for VMware ESXi (this guide) provides the
proper upgrade workflow.
2. Verify the minimum supported version of ESXi (vSphere) for your target
upgrade release, then proceed to step 3.
Target Release
Minimum Supported Version of ESXi (vSphere)
3.5(1a) – 4.0(2x)
6.0 U3
4.5(1a) -5.0(2x)
6.5 U3
5.5(1a) and later
7.0 U2
3. Select your deployment type:
Deployment Type
Link to chapter in this guide
Classic HyperFlex Deployment
Prerequisites for Upgrading HyperFlex Software, on page 7
HyperFlex Edge Deployment
HyperFlex Edge Upgrade Overview, on page 33
HyperFlex Edge Upgrade through Intersight Upgrading Cisco HyperFlex Edge Systems with Cisco Intersight
HyperFlex Stretched Clusters
Stretched Cluster Upgrade Overview, on page 41
Upgrading from an Unsupported Cisco HyperFlex HX Data Platform Software
Release
Cisco HyperFlex users who need to upgrade their environment from a Cisco
HyperFlex HX Data Platform software release that is past the last date of
support, to the latest suggested release on the Cisco Software Download site,
should follow the upgrade steps for their current release as defined in the
Cisco HyperFlex
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 4
Overview
Upgrading from an Unsupported Cisco HyperFlex HX Data Platform Software
Release
Systems Upgrade Guide for Unsupported Cisco HX Releases Guide. The steps in
this guide are not applicable for clusters running older software
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 5
Upgrading from an Unsupported Cisco HyperFlex HX Data Platform Software Release
Overview
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 6
3 C H A P T E R
Prerequisites and Guidelines
· Prerequisites for Upgrading HyperFlex Software, on page 7 · Upgrade
Recommendations, on page 8
Prerequisites for Upgrading HyperFlex Software
The following tasks should be performed prior to beginning the upgrade
process:
Important
Using VMware Update Manager (VUM) or VMware Lifecycle Manager (vLCM) for upgrading the ESXi on HyperFlex node is not supported. Using these upgrade methods may delete Cisco custom drivers and cause cluster outages. We recommend using Cisco Intersight or HyperFlex Connect for ESXi upgrades including the security patches from VMware or manually installing patches using the offline zip bundle with ESXCLI commands.
· Ensure Storage I/O Control (SIOC) is completely disabled on each HyperFlex
datastore and the local datastore on each ESXi host in the HyperFlex cluster.
This can be confirmed through the vCenter Web Client:
Datastores ->
Note Please refer to the VMware documentation site for more details and steps
to disable SIOC.
· Clusters running HXDP Release 4.0(2x) or later can upgrade directly to
5.5(1a).
· HXDP Release 5.5(x) supports ESXi version 7.0 U2, 7.0 U3 and 8.0 U1 and
later only. If your current ESXi version is earlier than 7.0 U2, make sure to
perform a combined upgrade of HXDP and ESXi to a target level 7.0 U2 or later.
· If the HXDP is already upgraded to 5.5(x) and an ESXi upgrade is attempted
from 7.0 to 8.0, user needs to upload HXDP bundle and select HXDP along with
the target ESXi 8.0 bundle.
· Migrate all M4 nodes to HyperFlex M5 or higher before attempting to upgrade
to HXDP Release 5.5(x). Attempts to upgrade to HXDP Release 5.5(x) with
HyperFlex M4 nodes in your cluster will fail.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 7
Upgrade Recommendations
Prerequisites and Guidelines
· Review the Cisco HyperFlex Upgrade Guidelines in the Recommended Cisco
HyperFlex HX Data Platform Software Releases – for Cisco HyperFlex HX-Series
Systems.
· Beginning with Cisco HXDP Release 5.0(2a), full feature functionality and
configuration changes require a valid Cisco HyperFlex Software License. HX
Connect users with expired or insufficient licenses at the end of the
evaluation or the grace period after the license compliance date, view a
prominent countdown banner that alerts the user to the license compliance need
and provides a link to the license renewal page until the license expiration
is remedied.
In the event a license passes both the license expiration date and the grace
period countdown, the current configurations will operate as expected with
limited information. Renewing the license allows a user to resume full feature
functionality, and make configuration changes. For details and examples of the
banners, see the License Compliance and Feature Functionality section of the
Cisco HyperFlex Systems Ordering and Licensing Guide.
· vCenter version check: Verify that the vCenter meets the minimum requirement
for the ESXi version being upgraded to. See, VMware Product Interoperability
Matrices to ensure compatibility between vCenter and ESXi.
· Ensure all VM network port groups exist on all nodes in the cluster for
vMotion compatibility.
· Ensure that the management and storage data VLANs are configured on the top-
of-rack network switches to ensure uninterrupted connectivity during planned
fabric failover.
· If using jumbo frames in your environment, ensure jumbo frames are enabled
on the vMotion and data networks on the top of rack switch.
· Verify that the ESXi hosts are not in lockdown mode and SSH service is
enabled and set to start and stop with the host for the duration of the
upgrade. Lockdown mode can be re-enabled after the upgrade is complete along
with disabling SSH service.
· Blade Package and Rack Package versions are not displayed in the Host
Firmware Package: HyperFlex-m5-con and HyperFlex-m6-con for M6 nodes.
· Upgrading the VM compatibility version or hardware version of the Storage
Controller Virtual Machine (SCVM) is not supported and should not be
performed. This action is detrimental to the SCVM and will require a rebuild
of the SCVM if performed.
Upgrade Recommendations
For upgrading supported releases, see the Recommended Cisco HyperFlex HX Data
Platform Software Releases – for Cisco HyperFlex HX-Series Systems.
If you want to upgrade from a release that is no longer supported, see the
Cisco HyperFlex Systems Upgrade Guide for Unsupported Cisco HX Releases.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 8
4 C H A P T E R
Pre-Upgrade Intersight Health Check
· Run the Health Check via Intersight, on page 9 · Test Upgrade Eligibility,
on page 9
Run the Health Check via Intersight
To better ensure a sucessful HyperFlex upgrade, it is essential to verify that
the HyperFlex clusters you intent to upgrade are healthy before starting the
upgrade process. The Health Check in Intersight allows you to view details and
reports about the health of your HypeFlex clusters. To get started, contiued
to the Intersight Help Center “Health Check for HyperFlex Clusters.
Test Upgrade Eligibility
Beginning with Cisco HyperFlex Release 4.0(2a), the Upgrade page displays the
last cluster upgrade eligibility test result and last tested version of UCS
server, HX data platform, and/or ESXi. Before upgrading UCS server firmware,
HyperFlex data platform, and/or ESXi, perform upgrade eligibility test in the
Upgrade page to validate and check the cluster readiness and the
infrastructure compatibility for an upgrade.
Note The Upgrade Eligibility test uses the validations included in the current
running HyperFlex Data Platform version. It does not include newer validations
that are present in the target HX version.
To perform upgrade eligibility test: 1. Select Upgrade > Test Upgrade
Eligibility. 2. Select the UCS Server Firmware check box to test upgrade
eligibility of UCS server firmware.
Enter the Cisco UCS Manager FQDN or IP address, username, and password. In the
Current Version field, click Discover to choose the UCS firmware package
version that need to be validated before upgrade. 3. Select the HX Data
Platform check box to test upgrade eligibility of HyperFlex Data Platform.
Enter the vCenter username and password. Upload the Cisco HyperFlex Data
Platform Upgrade Bundle that need to be validated before upgrade.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 9
Test Upgrade Eligibility
Pre-Upgrade Intersight Health Check
4. Select the ESXi check box to test upgrade eligibility of ESXi. Enter the
vCenter username and password. Upload the Cisco HyperFlex Custom Image Offline
Bundle that need to be validated before upgrade.
5. Click Validate. The progress of the upgrade eligibility test is displayed.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 10
5 C H A P T E R
Preparing for Upgrade
· HyperFlex Upgrade Preparation, on page 11 · Checking Cluster Storage
Capacity, on page 12 · Verify Health of an UCS Fabric Interconnect Cluster in
Cisco UCS Manager, on page 12 · Viewing HyperFlex Cluster Health, on page 13
HyperFlex Upgrade Preparation
Note The following upgrade process only applies to when a user is upgrading
from Cisco HX Release 3.5(2x) or later.
Note If you are upgrading one of the following rather than a HyperFlex
standard cluster: · For HyperFlex Edge clusters, see HyperFlex Edge Upgrade,
on page 33. · For Stretched clusters, see Stretched Cluster Upgrade, on page
41. · For split-upgrade procedures, see HyperFlex Offline Upgrade Workflow ,
on page 49.
The following image describes the general upgrade workflow for a full-stack
HyperFlex standard cluster.
The upgrade requires you to complete the following tasks in this order:
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 11
Checking Cluster Storage Capacity
Preparing for Upgrade
1. Complete all tasks in the Prerequisites section of this guide.
2. Run the Hypercheck Health & Pre-Upgrade tool to on your HyperFlex systems
to ensure its stability and resiliency Hypercheck : Hyperflex Health & Pre-
Upgrade Check Tool
3. Verify that your VMware vCenter version is 7.0 U2, 7.0 U3, or 8.0 or later
and that vCenter and your target ESXi version are compatible. See, VMware
Product Interoperability Matrices on the VMware site. Also, verify the vCenter
version is compatible with the target HXDP version.
Checking Cluster Storage Capacity
Cisco recommends that you check the cluster storage capacity before starting
the upgrade of an existing installation of Cisco HX Data Platform. If the
storage utilization in the cluster is equal to 76% or more (capacity +
overhead), the upgrade validation fails. Refer to the HX Storage Cluster
Overview chapter in the Cisco HyperFlex Data Platform Administration Guide for
background details about checking cluster storage capacity. Perform the
following validations on each HyperFlex node before starting the upgrade.
· Verify that the HyperFlex cluster is healthy and online.
· Verify all HyperFlex cluster nodes are connected to the vCenter and are
online.
· Verify that DRS is enabled and set to fully automated if licensed for DRS.
If DRS is Disabled, manual intervention is required to vMotion the VMs
manually when prompted by the upgrade process.
· Verify vMotion is configured on all the nodes, If vMotion is not configured,
see Verify vMotion Configuration for HX Cluster before starting the upgrade.
· Verify that ESXi Agent Manager (EAM) health is normal.
· Verify the health of the UCSM Fabric Interconnect cluster in Cisco UCS
Manager.
Verify Health of an UCS Fabric Interconnect Cluster in Cisco UCS Manager
Step 1
Step 2 Step 3 Step 4 Step 5
Verify if the high availability status of the fabric interconnects shows that both the fabric interconnects are up and running. See the Cisco UCS Manager System Monitoring Guide for more information. Verify that all servers have been discovered. Verify that the HyperFlex servers have no faults. Verify that vNIC faults are cleared to ensure VMware ESXi vSwitch uplinks are up and operational. Verify that the data path is up and running. See the Cisco UCS Manager Firmware Management Guide for more information.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 12
Preparing for Upgrade
Viewing HyperFlex Cluster Health
Viewing HyperFlex Cluster Health
Using CLI
Log into any controller VM in the storage cluster. Run the command hxcli
cluster info [flags].
address: 192.168.100.82 name: HX-Cluster01 state: online uptime: 0 days 12
hours 16 minutes 44 seconds activeNodes: 5 of 5 compressionSavings:
78.1228617455 deduplicationSavings: 0.0 freeCapacity: 38.1T healingInfo:
inProgress: False resiliencyDetails:
current ensemble size:5 # of ssd failures before cluster shuts down:3 minimum
cache copies remaining:3 minimum data copies available for some user data:3
minimum metadata copies available for cluster metadata:3 # of unavailable
nodes:0 # of nodes failure tolerable for cluster to be available:2 health
state reason:storage cluster is healthy. # of node failures before cluster
shuts down:3 # of node failures before cluster goes into readonly:3 # of hdd
failures tolerable for cluster to be available:2 # of node failures before
cluster goes to enospace warn trying to move the existing data:na # of hdd
failures before cluster shuts down:3 # of hdd failures before cluster goes
into readonly:3 # of ssd failures before cluster goes into readonly:na # of
ssd failures tolerable for cluster to be available:2 resiliencyInfo: messages:
Storage cluster is healthy. state: healthy hddFailuresTolerable: 2
nodeFailuresTolerable: 1 ssdFailuresTolerable: 2 spaceStatus: normal
totalCapacity: 38.5T totalSavings: 78.1228617455 usedCapacity: 373.3G
clusterAccessPolicy: lenient dataReplicationCompliance: compliant
dataReplicationFactor: 3
Sample response that indicates the HyperFlex storage cluster is online and
healthy.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 13
Viewing HyperFlex Cluster Health
Preparing for Upgrade
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 14
6 C H A P T E R
Download Software Bundles
· Downloading Software, on page 15
Downloading Software
For a successful HyperFlex upgrade, the Cisco HyperFlex System component
bundles can be downloaded from the Cisco HyperFlex Download website:
Step 1 Step 2 Step 3
Step 4 Step 5 Step 6 Step 7 Step 8
Navigate to https://www.cisco.com/support and enter HX Data Platform in the
Select a Product search bar. Click on the HyperFlex HX Data Platform downloads
link
Click on the current Suggested Release version.
Click the cart icon for the latest Cisco HyperFlex Data Platform Upgrade
Bundle for upgrading existing HyperFlex clusters from a previous release (.tgz
file).
Note
Read through any Software Advisories to confirm whether any issues may apply to your environment before
proceeding with the download.
Click the cart icon for the corresponding UCS Infrastructure Software Bundle based on the FI model. Click the cart icon for the Software for UCS B-Series and C-Series blade and rack-mounted servers. For upgrading vSphere, click on the cart icon for the latest HX Custom Image for ESXi Offline Bundle for Upgrading from prior ESXi versions. Click on the cart icon at the stop of the screen to confirm the bundles and click Download All. Accept the license agreement and click Ok for each file to save it.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 15
Downloading Software
Download Software Bundles
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 16
7 C H A P T E R
Upgrade UCS Infrastructure Firmware
· Upgrading UCS Infrastructure Firmware Workflow, on page 17 · Guidelines and Limitations, on page 17 · Upgrade UCS Infrastructure Firmware, on page 18
Upgrading UCS Infrastructure Firmware Workflow
Upgrade Type HyperFlex Clusters
Procedure See the workflow below.
HyperFlex Edge Upgrade Clusters HyperFlex Edge Upgrade, on page 33
HyperFlex Stretched Clusters
Stretched Cluster Upgrade, on page 41
Split Upgrade Procedures
HyperFlex Offline Upgrade Workflow , on page 49
Perform the following tasks to upgrade UCS Infra Firmware: · Review Prerequisites for Upgrading HyperFlex Software, on page 7 before beginning upgrade. · Log into the UCSM Fabric Interconnect cluster IP address. · Upload the appropriate Infra, B-Series, and C-Series bundles to the Fabric Interconnect.
Guidelines and Limitations
Consider the following before performing the UCS Infra Firmware upgrade:
· Ensure that the hx-storage-data and vMotion upstream switches are configured
for Jumbo Frames before proceeding forward, otherwise the HyperFlex Cluster
could suffer a network and storage outage during the upgrade window.
· You will lose connectivity to UCS Manager throughout the entire UCS
infrastructure firmware upgrade. This is normal behavior.
· Verify that the Data path is ready. For for more information, see the
Verification that the Data Path is Ready section in the Cisco UCS Manager
Firmware Management Guide..
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 17
Upgrade UCS Infrastructure Firmware
Upgrade UCS Infrastructure Firmware
Upgrade UCS Infrastructure Firmware
Before you begin Download the Infra, B-Series, and C-Series Fabric
Interconnect upgrade bundles. For details, see Downloading Software, on page
15.
Step 1 Step 2 Step 3 Step 4 Step 5 Step 6
Step 7 Step 8
Step 9 Step 10 Step 11
Step 12
Log on to UCS Manager to the Fabric Interconnect cluster IP address using admin privileges. Navigate to Equipment > Firmware Management > Installed Firmware. Expand UCS Manager and confirm the UCS Manager running version. Navigate to Download Tasks > Download Firmware. Browse to your saved Fabric Interconnect bundles and select the Infra A, B-Series, and C-Series bundles you previously saved click Open and OK. Once the files have transferred, click on Firmware Auto Install and then Install Infrastructure Firmware under Actions.
Note
Review carefully all warnings and resolve any issues if required before proceeding.
Once any issues are resolved (if any) select Ignore All and click Next. Select the appropriate Infra Pack from the drop-down and check Upgrade Now and click Finish.
Note
You can click Yes to disregard the warning for the Service Pack not being selected.
Click on the FSM tab to follow the upgrade progress. The upgrade takes some
time to proceed.
Click on the Pending Activities tab at the top and then Fabric Interconnects
to verify that the data path is successfully restored from the Secondary
Fabric Interconnect before you acknowledge the reboot of the Primary Fabric
Interconnect.
Click Reboot Now and then click Yes and OK.
Note
During the course of the upgrade process, you will be logged out of the Fabric Interconnect UI. Log back
into view the upgrade progress.
Once the Upgrade process is completed, view the updated versions on the Installed Firmware tab.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 18
8 C H A P T E R
Upgrade UCS Server Firmware, HX Data Platform and VMware vSphere – Combined
Upgrade
· Upgrade Cisco UCS Firmware, HX Data Platform, and VMware vSphere Workflow,
on page 19 · Guidelines and Limitations, on page 20 · Upgrading HyperFlex Data
Platform Software, VMware ESXi, and Cisco UCS Server Firmware using
HX Connect, on page 21
Upgrade Cisco UCS Firmware, HX Data Platform, and VMware vSphere Workflow
The Cisco HyperFlex “full-stack” upgrade process involves upgrading the
following 3 components: · Cisco HyperFlex Data Platform · VMware vSphere ESXi
· Cisco UCS Server Firmware
Cisco recommends upgrading all these 3 components in a combined upgrade from
HyperFlex Connect. You can choose to upgrade one, two or all three components
in the same upgrade workflow. If you are combining two or more components in a
single upgrade process, follow the procedure below. Otherwise, navigate to
Chapter 9 for individual component upgrade procedures. This section describes
the steps to perform a combined upgrade of HyperFlex Data Platform Software,
VMware ESXi, and UCS Server firmware . In this process, the HyperFlex nodes
will go through an optimized rolling reboot without any workload disruptions,
by use of VMware vMotion.
Note As part of the Server Firmware upgrade operation initiated from HX Connect, some of the UCS policies may be updated to be compatible with the new HXDP version. These changes are applied only to the nodes that are part of the cluster being upgraded. It is highly recommended to use HX Connect to initiate the Server Firmware upgrade to avoid any policy drift.
Upgrade Type HyperFlex Clusters
Procedure See the workflow below.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 19
Guidelines and Limitations
Upgrade UCS Server Firmware, HX Data Platform and VMware vSphere – Combined Upgrade
Upgrade Type HyperFlex Edge Clusters HyperFlex Stretched Clusters Split Upgrade Procedures
Procedure HyperFlex Edge Upgrade, on page 33 Stretched Cluster Upgrade, on page 41 HyperFlex Offline Upgrade Workflow , on page 49
Perform the following tasks to upgrade UCS Firmware and HX Data Platform: · Review Prerequisites for Upgrading HyperFlex Software, on page 7 before beginning upgrade. · Log into HX Connect with admin privileges. · Select the appropriate options from the Upgrade page. · Upload the required files and complete the necessary user inputs.
Guidelines and Limitations
Consider the following before performing the upgrade: · If DRS is Enabled and
set to fully automatic mode, the VMs are automatically vMotioned to other
hosts during the rolling upgrade process.
Note If DRS is Disabled, vMotion the VMs manually to continue the upgrade
process when prompted. For more information, see VMware Documentation for
Migration with vMotion.
· Downgrading ESXi and HXDP is not supported.
· Refer to the Release Notes for software compatibilities of HXDP, UCS
firmware, and VMware ESX. Additionally, ensure vCenter is upgraded to a
compatible version before upgrading ESXi. For details, see the Cisco HyperFlex
Release Notes that correspond with your installation,Recommended Cisco
HyperFlex HX Data Platform Software Releases – for Cisco HyperFlex HX-Series
Systems, and the VMware Product Interoperability Matrix on the VMware site.
· Do not manually upgrade UCS server firmware using the tools available in UCS
Manager. Changes to policies in UCS Manager for HyperFlex servers are
delivered through the orchestrated server firmware upgrade process. Manually
performing firmware updates out of band will result in these important
configuration updates being missing.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 20
Upgrade UCS Server Firmware, HX Data Platform and VMware vSphere – Combined Upgrade Upgrading HyperFlex Data Platform Software, VMware ESXi, and Cisco UCS Server Firmware using HX Connect
Upgrading HyperFlex Data Platform Software, VMware ESXi, and Cisco UCS Server
Firmware using HX Connect
Before you begin · Download the latest Cisco HX Data Platform Upgrade Bundle
for upgrading existing clusters from previous release, from Downloading
Software. · Download the appropriate HX custom ESXi offline upgrade bundle
from https://www.cisco.com/. · Disable snapshot schedule, on the storage
controller VM. SSH to HyperFlex cluster IP and run the command stcli snapshot-
schedule disable snapshot schedule.
Step 1
Step 2 Step 3
Log into HX Connect. a) Enter the administrative username and password. b)
Click Login.
In the Navigation pane, select Upgrade. On the Select Upgrade Type page,
select HX Data Platform, ESXi, and UCS Server Firmware and complete the
following fields:
Field
Essential Information
UCS Manager Connectivity UCS Manager FQDN/IP
Enter the Cisco UCS Manager FQDN or IP address. Example: 10.193.211.120.
User Name
Enter the Cisco UCS Manager
Admin Password
Enter the Cisco UCS Manager
HX Server Firmware
Discover
Click Discover to view the current UCS firmware package version.
M5/M6 Desired Version (Depending on the nodes in the cluster)
Select the appropriate C-Series firmware versions.
Optionally, if you have Compute only B-Series UCS blades in the cluster select
the appropriate B-Series firmware version.
Only C & B bundles uploaded to UCS Manager will be shown in the list. Return
to Upgrading UCS Infrastructure Firmware Workflow, on page 17 if the desired
version is not shown.
Only compatible firmware versions are shown in the dropdown list. If the
desired version is not displayed, confirm the compatibility between HXDP &
Server Firmware in the Cisco HyperFlex Software Requirements and
Recommendations document.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 21
Upgrade UCS Server Firmware, HX Data Platform and VMware vSphere – Combined Upgrade Upgrading HyperFlex Data Platform Software, VMware ESXi, and Cisco UCS Server Firmware using HX Connect
Step 4
Step 5 Step 6 Step 7 Step 8
Note
If the UI doesn’t show your desired UCS server firmware version in dropdown, see HX Connect UCS Server
Firmware Selection Dropdown Doesn’t List the Firmware Version 4.1 or Above, on page 56.
Upload the HyperFlex Data Platform upgrade package (storfs-package).
Field
Essential Information
Drag the HX file here or click to browse
Upload the latest Cisco HyperFlex Data Platform Upgrade Bundle for upgrading
existing clusters with previous release.tgz package file from Download
Software -HyperFlex HX Data Platform.
Sample file name format: storfs-packages-4.5.1a-31601.tgz.
Current version
Displays the current HyperFlex Data Platform version.
Current cluster details
Lists the HyperFlex cluster details like the HyperFlex releaseand cluster upgrade state.
Bundle version
Displays the HyperFlex Data Platform version of the uploaded bundle.
(Optional) Checksum
The MD5 Checksum number is available by hovering over the filename in the
Cisco.com Software Download section.
This is an optional step that helps you verify the integrity of the uploaded
upgrade package bundle
Upload the ESXi offline upgrade bundle. Provide vCenter login credentials. Field User Name Admin Password
Essential Information Enter the vCenter
Click Upgrade to begin the first step of the combined upgrade process.
The Validation Screen on the Upgrade Progress page displays the progress of
the checks performed. Fix validation errors, if any.
Note
At this point, all pre-upgrade checks and validations will run, along with the initial upgrade stage. Within a
few minutes, HX Connect returns and prompts the user to confirm and start the second stage of the upgrade.
The upgrade is not complete until both steps are performed in the UI. The system should never be left in a
state where only the first step of the upgrade is complete.
Note
Do not manually acknowledge servers in UCS Manager. While the servers enter a pending-ack state, the
administrator should not manually intervene. The HyperFlex platform will automatically acknowledge each
server at the correct time.
Note
As of HX 5.0(1b), an upgrade status appears providing the result of the last upgrade along with the versions
that were upgraded (source and target versions for each component selected in the upgrade). You can dismiss
this status only if it is successful. If the last upgrade fails, you will need to fix the issue. This banner is a
reminder to take action to correct the upgrade.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 22
Upgrade UCS Server Firmware, HX Data Platform and VMware vSphere – Combined Upgrade Upgrading HyperFlex Data Platform Software, VMware ESXi, and Cisco UCS Server Firmware using HX Connect
Step 9
The HyperFlex Connect UI refreshes after the first step of the upgrade, and a
banner pops up prompting you to provide the UCS and vCenter credentials and
start the second stage of the upgrade process. Monitor the upgrade page and
confirm that the upgrade is complete.
When upgrade is in progress, you may see an error message, Websocket
connection failed. Automatic refresh disabled. You can either refresh the page
or log out and log back in to clear the error message. You can safely ignore
this error message.
What to do next
Proceed to Confirm That Upgrade Is Complete, on page 31 for post upgrade tasks
once the upgrade is complete. If the upgrade fails, you can re-try the upgrade
or contact Cisco TAC for further assistance. Running a cluster without
remediation after an upgrade failure is not recommended. Care should be taken
to fully complete the upgrade as soon as possible.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 23
Upgrade UCS Server Firmware, HX Data Platform and VMware vSphere – Combined
Upgrade Upgrading HyperFlex Data Platform Software, VMware ESXi, and Cisco UCS
Server Firmware using HX Connect
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 24
9 C H A P T E R
Upgrade UCS Firmware, HX Data Platform and VMware vSphere – Individual
Component Upgrade
· Overview, on page 25 · Cisco HyperFlex Data Platform Upgrade, on page 25 ·
Cisco UCS Server Firmware Upgrade, on page 27 · VMware vSphere/ESXi Upgrade,
on page 28
Overview
Cisco recommends upgrading all these 3 components in a combined full-stack
upgrade from HyperFlex Connect. You can choose to upgrade one, two or all
three components at a time. If you are combining 2 or more components in a
single upgrade process, follow procedures described in the previous chapter.
Otherwise, follow below procedures for individual component upgrade one at a
time.
Cisco HyperFlex Data Platform Upgrade
Before you begin · Download the latest Cisco HX Data Platform Upgrade Bundle
for upgrading existing clusters from previous releases, from Downloading
Software, on page 15. · Disable snapshot schedule, on the storage controller
VM. SSH to the HyperFlex cluster IP and run the command stcli snapshot-
schedule -disable snapshot schedule.
Step 1
Step 2 Step 3
Log into HX Data Platform. a) Enter the administrative username and password.
b) Click Login.
In the Navigation pane, select Upgrade. On the Select Upgrade Type page,
select HX Data Platform and complete the following fields:
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 25
Cisco HyperFlex Data Platform Upgrade
Upgrade UCS Firmware, HX Data Platform and VMware vSphere – Individual Component Upgrade
UI Element Drag the HX file here or click to browse
(Optional) Checksum field
Figure 1: Select Upgrade Type Page
Essential Information
Upload the latest Cisco HyperFlex Data Platform Upgrade Bundle for upgrading
existing clusters with previous release.tgz package file from Download
Software – HyperFlex HX Data Platform.
Sample file name format: storfs-packages-4.5.1a-31601.tgz.
The MD5 Checksum number is available by hovering over the filename in the
Cisco Software Download section.
This is an optional step that helps you verify the integrity of the uploaded
upgrade package bundle.
Step 4
Step 5 Step 6
Enter the vCenter credentials. UI Element User Name field Admin Password field
Essential Information Enter the vCenter
Click Upgrade to begin the cluster upgrade process.
The Validation screen on the Upgrade Progress page displays the progress of
the checks performed. Fix validation errors, if any.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 26
Upgrade UCS Firmware, HX Data Platform and VMware vSphere – Individual Component Upgrade
Cisco UCS Server Firmware Upgrade
Step 7
Note
At this point, all pre-upgrade checks and validations are run, along with the initial upgrade stage. Within a
few minutes, HX Connect returns and prompts you to confirm and start the second stage of the upgrade. The
upgrade is not complete until both steps are performed in the UI. The system should never be left in a state
where only the first step of the upgrade is complete.
The HyperFlex Connect UI refreshes after the first step of the upgrade, and a
banner pops up prompting you to provide the UCS and vCenter credentials and
start the second stage of the upgrade process. Monitor the upgrade page and
confirm that the upgrade is complete.
When upgrade is in progress, you may see an error message Websocket connection
failed. Automatic refresh disabled. You can either refresh the page or log out
and log back in to clear the error message. You can safely ignore this error
message.
What to do next
Proceed to Confirm That Upgrade Is Complete, on page 31 for post upgrade tasks
once the upgrade is complete. If the upgrade fails, you can re-try the upgrade
or contact Cisco TAC for further assistance.
Note Running a cluster without remediation after an upgrade failure is not recommended. Care should be taken to fully complete the upgrade as soon as possible.
Cisco UCS Server Firmware Upgrade
Before you begin · Check and confirm the UCS B-Series and C-Series server
firmware packages have been downloaded to the Fabric Interconnects. · Disable
snapshot schedule, on the storage controller VM. SSH to HyperFlex cluster IP
and run the command stcli snapshot-schedule -disable snapshot schedule.
Step 1
Step 2 Step 3
Log into HX Connect. a) Enter the administrative username and password. b)
Click Login.
In the Navigation pane, select Upgrade. On the Select Upgrade Type page,
select UCS Server Firmware, and complete the following fields:
Field
Essential Information
UCS Manager Connectivity UCS Manager FQDN/IP
Enter the Cisco UCS Manager FQDN or IP address. Example: 10.193.211.120.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 27
VMware vSphere/ESXi Upgrade
Upgrade UCS Firmware, HX Data Platform and VMware vSphere – Individual Component Upgrade
Step 4 Step 5
Field
Essential Information
User Name
Enter the Cisco UCS Manager
Admin Password
Enter the Cisco UCS Manager
HX Server Firmware
Discover
Click Discover to view the current UCS firmware package version.
M5/M6 Desired Version(Depending on the nodes in the cluster)
Select the appropriate C-Series firmware versions.
Optionally, if you have Compute only B-Series UCS blades in the cluster select
the appropriate B-Series firmware version.
Only C & B bundles uploaded to UCS Manager will be shown in the list. Return
to Upgrading UCS Infrastructure Firmware Workflow, on page 17 if the desired
version is not shown.
Only compatible firmware versions are shown in the dropdown list. If the
desired version is not displayed, confirm the compatibility between HXDP &
Server Firmware in the Cisco HyperFlex Software Requirements and
Recommendations document.
Click Upgrade to begin the UCS firmware upgrade process.
The Validation Screen on the Upgrade Progress page displays the progress of
the checks performed. Fix validation errors, if any. Monitor the upgrade page
and confirm that the upgrade is complete.
When upgrade is in progress, you may see an error message, Websocket connection failed. Automatic refresh disabled. You can either refresh the page or log out and log back in to clear the error message. You can safely ignore this error message.
Note
Do not manually acknowledge servers in UCS Manager. While the servers will enter a pending-ack state,
the administrator should not manually intervene. The HyperFlex platform will automatically acknowledge
each server at the correct time.
VMware vSphere/ESXi Upgrade
Important
Using VMware Update Manager (VUM) or VMware Lifecycle Manager (vLCM) for upgrading the ESXi on HyperFlex node is not supported. Using these upgrade methods may delete Cisco custom drivers and cause cluster outages. We recommend using Cisco Intersight or HyperFlex Connect for ESXi upgrades including the security patches from VMware or manually installing patches using the offline zip bundle with ESXCLI commands.
Follow the procedure below to upgrade only VMware ESXi from HyperFlex Connect.
This procedure can be followed for ESXi patch upgrades as well.
Upgrading vSphere requires you to complete the following tasks:
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 28
Upgrade UCS Firmware, HX Data Platform and VMware vSphere – Individual Component Upgrade
VMware vSphere/ESXi Upgrade
· Refer to the release documents for software compatibilities of HXDP, UCS,
and VMware and confirm your vCenter is upgraded before you upgrade. For
details, see the Release Notes for Cisco HX Data Platform for your release,
Cisco HyperFlex Recommended Software Release and Requirements Guide, and
VMware Product Interoperability Matrices.
· Log into HX Connect using admin privileges and navigate to the Upgrade page.
Before you begin
Download the appropriate HX custom ESXi offline upgrade bundle. For details,
see Downloading Software, on page 15. Cisco does not recommend use of non-HX
customized ESXi bundles, although it is supported. Using HX customized bundles
ensures all the latest drivers are updated and compatibility with the
HyperFlex hardware.
Step 1 Step 2 Step 3 Step 4 Step 5
Step 6
Log into HX Connect with admin privileges. Navigate to the System Information tab and confirm the running Hypervisor version. Click on the Upgrade tab and select ESXi. Click in the bundle version window and navigate to your previously saved ESXi offline bundle and click Open. Once the bundle is uploaded, enter your vCenter credentials and click Upgrade.
Note
The upgrade process is non-disruptive and upgrades one server at a time.
Refresh your browser screen to view the upgrade changes on the Dashboard tab.
Note
Click on the System Information tab to confirm that all the nodes are online.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 29
VMware vSphere/ESXi Upgrade
Upgrade UCS Firmware, HX Data Platform and VMware vSphere – Individual Component Upgrade
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 30
1 0 C H A P T E R
Post Upgrade Tasks
· Confirm That Upgrade Is Complete, on page 31 · Enable HyperFlex Software
Encryption, on page 32
Confirm That Upgrade Is Complete
Step 1 Step 2 Step 3 Step 4
Step 5
Step 6
Log into Cisco UCS Manager to ensure that the HX nodes have no pending server
activities. From Servers tab > Servers > Pending Activities tab check for all
server activities.
Confirm that the HX nodes match the expected firmware version. In Cisco UCS
Manager, from Equipment > Firmware Management > Installed Firmware tab, verify
for the correct firmware version.
Log into any controller VM through SSH.
ssh admin@controller_vm_ip
Confirm the HyperFlex Data Platform version.
hxcli cluster version
Cluster version: 5.5(1a) Node hx220-m5-node1 version: 5.5(1a) Node
hx220-m5-node3 version: 5.5(1a) Node hx220-m5-node3 version: 5.5(1a) Node
hx220-m5-node4 version: 5.5(1a)
Verify that the HX storage cluster is online and healthy.
hxcli cluster info|grep -i health Sample output: healthstate : healthy
state: healthy storage cluster is healthy
Verify that the upgrade is complete and is successful.
stcli cluster upgrade-status
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 31
Enable HyperFlex Software Encryption
Post Upgrade Tasks
Step 7
Nodes up to date: [HX-Cluster, HX-Node-1(1.1.1.1), HX-Node-2(1.1.1.2), HX-
Node-3(1.1.1.3)] Cluster upgrade succeeded.
For each browser interface you use, empty the cache and reload the browser
page to refresh the HX Connect content.
Enable HyperFlex Software Encryption
HyperFlex Software Encryption provides file-level end-to-end AES 256-bit
encryption of data at-rest. You can leverage the capability of HyperFlex
Software Encryption to protect the confidentiality of your data against device
theft, such as drives, servers, or entire clusters. Encryption keys are
securely and remotely stored by the Intersight Key Manager, available in both
the Intersight SaaS and the Intersight virtual appliance.
To enable HyperFlex Software Encryption on your cluster, check that you meet
the HX Data Platform and Intersight license requirements, see Cisco HyperFlex
Systems Ordering and Licensing Guide. After confirming license requirements
are met, to enable HyperFlex Software Encryption, you need to download the
encryption package from My Cisco Entitlement, install the package, and then
enable encryption from Intersight. For more information, see HyperFlex
Software Encryption.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 32
1 1 C H A P T E R
HyperFlex Edge Upgrade
· Overview, on page 33 · Cisco HyperFlex Edge Firmware Recommended Versions,
on page 33 · Server Firmware Upgrade Using the Cisco Host Upgrade Utility
Tool, on page 34 · Server Firmware Upgrade Using Cisco Integrated Management
Controller Supervisor, on page 35 · Upgrading Server Firmware on a Cisco UCS
C-Series Server Using the Cisco IMC Supervisor, on page
36 · Upgrading HyperFlex Edge Using HX Connect, on page 37 · Post Upgrade
Tasks for HyperFlex Edge, on page 38
Overview
This section provides information related to upgrading a Cisco HyperFlex Edge
system from the HX Connect. If your cluster was deployed using Cisco
Intersight, please use Intersight to perform the cluster upgrade. For detailed
upgrade prerequisites and instructions about upgrading Edge Clusters using
Intersight.
Important
· Upgrading a HyperFlex Edge system involves upgrading the server firmware,
HyperFlex Data Platform Software, and VMware ESXi.
· You can perform a combined upgrade of HyperFlex Data Platform and VMware
ESXI using HyperFlex Connect or you can choose to perform a split upgrade.
· UCS Server firmware upgrade is not supported from HX Connect. Instead,
perform the UCS server firmware upgrade separately using the Host Upgrade
Utility (HUU) tool or the Integrated Management Controller (IMC) Supervisor.
Cisco HyperFlex Edge Firmware Recommended Versions
· Review the Cisco HyperFlex Release Notes for Cisco HX Data Platform that
corresponds to the release for your planned upgrade.
· For upgrading supported releases, see the Cisco HyperFlex Software
Requirements and Recommendations document.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 33
Server Firmware Upgrade Using the Cisco Host Upgrade Utility Tool
HyperFlex Edge Upgrade
· If you want to upgrade from a release that is no longer supported, see the
Cisco HyperFlex Systems Upgrade Guide for Unsupported Cisco HX Releases.
· For HyperFlex Edge clusters that are upgraded using HX Connect, the
HyperFlex Data Platform upgrade includes upgrade of the embedded storage
firmware in addition to the HyperFlex Data Platform software. This embedded
firmware upgrade includes updates to the SAS passthrough storage controller
and associated drives (housekeeping, cache, and capacity) that run the
distributed storage platform. A manual upgrade of these embedded firmware
storage components via the HUU should not be performed. It is recommended to
complete the server firmware upgrade using HX Connect. If a manual upgrade or
downgrade of these storage components is required, ensure that you do the
following:
· If you are using the Host Update Utility (HUU) to upgrade chassis firmware,
be sure to uncheck the option to upgrade the SAS controller and do not upgrade
drives (except for the boot drive, if desired).
· Avoid using the Upgrade All button in the HUU as this will include the
storage controller by default.
· Firmware management for these devices is handled by HyperFlex Data Platform
automatically and should not be manually changed using other utilities. Only
use the HUU to upgrade these components when deemed necessary for
troubleshooting or as instructed by Cisco TAC.
· For clusters running Cisco IMC version prior to version 4.1(3b), secure boot
must be temporarily disabled to perform server firmware upgrade. Once Cisco
IMC version 4.1(3b) version or later is running on all nodes in the cluster,
secure boot can be enabled for firmware upgrade.
Server Firmware Upgrade Using the Cisco Host Upgrade Utility Tool
The following table summarizes the server firmware upgrade workflow on Cisco HX Servers:
Step Description
Reference
Place a node in HX maintenance mode.
Entering Cisco HyperFlex Maintenance Mode
Note
Upgrade one node at a time, for the
cluster to stay online during upgrade.
Upgrade server firmware using the Host Upgrade See Upgrading the Firmware on a Cisco UCS
Utility tool.
C-Series Server Using the HUU in the Cisco
Host Upgrade Utility User Guide.
Reboot the node back into ESXi. Exit HX
maintenance mode.
Wait until the cluster becomes fully healthy.
Viewing HyperFlex Cluster Health, on page 13
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 34
HyperFlex Edge Upgrade
Server Firmware Upgrade Using Cisco Integrated Management Controller Supervisor
Step Description
Reference
Repeat steps 1-4 on the remaining HX nodes in a
rolling fashion.
Note
Ensure that you check the health state
before entering maintenance mode on
the next host in the cluster.
You can find current and previous releases of the Cisco Host Upgrade Utility User Guide at this location: https://www.cisco.com/c/en/us/support/servers- unified-computing/ucs-c-series-rack-servers/ products-user-guide-list.html.
Server Firmware Upgrade Using Cisco Integrated Management Controller Supervisor
The following table summarizes the server firmware upgrade workflow on Cisco HX Servers:
petSDescription
Reference
. 1Place a node in HX maintenance mode.
Note
Upgrade one node at a time, for the
cluster to stay online during upgrade.
. 2Create a rack group. Add servers to the IMC Supervisor inventory.
. 3Configure auto-discovery profile.
. 4Run inventory on the rack group.
. 5Create firmware profile.
See Managing Server Discovery, Rack Groups, and Rack Accounts in the Cisco IMC
Supervisor Rack-Mount Servers Management Guide.
See Configuring Auto Discovery Profile in the Cisco IMC Supervisor Rack-Mount
Servers Management Guide.
See Collecting Inventory for Rack Accounts or Rack Groups in the Cisco IMC
Supervisor Rack-Mount Servers Management Guide.
Refer to the following tasks in the Cisco IMC Supervisor Rack-Mount Servers
Management Guide:
· Adding Images to a Local Server
· Uploading Images from a Local File System
· Adding Images from a Network Sever
. 6Upgrade firmware using IMC Supervisor, on the node See Upgrading Firmware in the Cisco IMC Supervisor
put in maintenance mode.
Rack-Mount Servers Management Guide.
. 7Reboot the node back into ESXi. Exit HX maintenance mode.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 35
Upgrading Server Firmware on a Cisco UCS C-Series Server Using the Cisco IMC Supervisor
HyperFlex Edge Upgrade
petSDescription
Reference
. 8Wait until the cluster becomes fully healthy.
. 9Repeat step 6 on the remaining HX nodes in a rolling fashion.
Note
Ensure that you check the health state
before entering maintenance mode on
the next host in the cluster.
You can find current and previous releases of the Cisco IMC Supervisor Rack- Mount Servers Management Guide here: https://www.cisco.com/c/en/us/support /servers-unified-computing/ integrated-management-controller-imc-supervisor /products-installation-and-configuration-guides-list.html.
Upgrading Server Firmware on a Cisco UCS C-Series Server Using the Cisco IMC Supervisor
Note Before upgrading Cisco IMC Supervisor and if a firmware profile was already set up, ensure that the Cisco.com credentials and proxy details are configured.
Step 1 Step 2 Step 3 Step 4 Step 5
Step 6
Choose Systems > Firmware Management.
On the Firmware Management page, click Firmware Upgrades.
Click Run Upgrade. A warning message appears, advising you that running the
upgrade on the selected servers will cause the host to reboot into the
firmware update tool. On completion of the firmware update, the servers will
reboot back to the host OS.
Click OK to confirm.
On the Upgrade Firmware screen, complete the following:
Field
Description
Select Profile drop-down list
Choose a profile from the drop-down list.
Platform field Image Version field Image Path field
Click Select and choose the servers from the list. The list displays only those servers whose platforms match the one configured in the selected profile.
Schedule later check box
Check this check box and select an existing schedule to run an upgrade. You can also click the + icon to create a new schedule.
Click Submit.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 36
HyperFlex Edge Upgrade
Upgrading HyperFlex Edge Using HX Connect
Upgrading HyperFlex Edge Using HX Connect
Cisco HyperFlex Edge cluster upgrade process involves upgrading the below 2
components:
· Cisco HyperFlex Data Platform
· VMware vSphere ESXi
You can combine HyperFlex Data Platform and VMware ESXi upgrade in a single
combined upgrade for HyperFlex Edge clusters. Cisco recommends upgrading these
2 components in a combined upgrade from HyperFlex Connect. You can choose to
upgrade one or two of these components at a time.
If you prefer to upgrade individual components one at a time, see VMware
vSphere/ESXi Upgrade, on page 28. The component upgrade process for standard
cluster and HyperFlex Edge cluster are the same.
This section describes the steps to perform a combined upgrade of HyperFlex
Data Platform and VMware vSphere ESXi. In this process, the HyperFlex nodes
will go through an optimized rolling reboot without any workload disruptions
by use of VMware vMotion.
Note HyperFlex Edge clusters deployed via Intersight do not have upgrade capability from HyperFlex Connect. The upgrade is only supported through Intersight.
Step 1
Step 2 Step 3 Step 4
Log into HX Connect. a) Enter the administrative username and password. b)
Click Login.
In the Navigation pane, select Upgrade. On the Select Upgrade Type page,
select HX Data Platform and ESXi and complete the following fields: Upload the
HyperFlex Data Platform upgrade package (storfs-package).
Table 1: Upgrade HX Data Platform
UI Element Drag the HX file here or click to browse
Current version Current cluster details Bundle version
Essential Information
Upload the latest Cisco HyperFlex Data Platform Upgrade Bundle for upgrading
existing clusters with previous release.tgz package file from Download
Software HyperFlex HX Data Platform.
Sample file name format: storfs-packages-5.5.1a-31601.tgz.
Displays the current HyperFlex Data Platform version.
Lists the HyperFlex cluster details like the HyperFlex release and cluster
upgrade state.
Displays the HyperFlex Data Platform version of the uploaded bundle.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 37
Post Upgrade Tasks for HyperFlex Edge
HyperFlex Edge Upgrade
Step 5 Step 6
Step 7 Step 8
Step 9
UI Element (Optional) Checksum field
Essential Information
The MD5 Checksum number is available by hovering over the filename in the
Cisco.com Software Download section.
This is an optional step that helps you verify the integrity of the uploaded
upgrade package bundle
Upload the VMware ESXi custom image offline upgrade bundle. Provide vCenter login credentials:
Essential Information
Essential Information
User Name field
Enter the vCenter
Admin Password field
Enter the vCenter
Click Upgrade to begin the combined upgrade process.
The Validation screen on the Upgrade Progress page displays the progress of
the checks performed. Fix validation errors, if any.
Note
At this point, all pre-upgrade checks and validations are running, along with the initial upgrade stage. Within
a few minutes, HX Connect returns and prompts you to confirm and start the second stage of the upgrade.
The upgrade is not complete until both steps are performed in the UI. The system should never be left in a
state where only the first step of the upgrade is complete.
The HyperFlex Connect UI refreshes after the first step of the upgrade, and a banner pops up prompting you to provide the UCS and vCenter credentials and start the second stage of the upgrade process. Monitor the upgrade page and confirm that the upgrade is complete.
When upgrade is in progress, you may see an error message Websocket connection failed. Automatic refresh disabled. You can either refresh the page or log out and log back in to clear the error message. You can safely ignore this error message.
Note
Proceed to Confirm That Upgrade Is Complete, on page 31 for post upgrade tasks once the upgrade is
complete. If the upgrade fails, you can re-try the upgrade or contact Cisco TAC for further assistance.
Post Upgrade Tasks for HyperFlex Edge
After the upgrade is complete and the HyperFlex Edge cluster has been
upgraded, log out and log back in to HX Connect to see the upgrade changes.
Step 1
Confirm that the HX nodes match the expected firmware version.
Check the firmware version in the IMC supervisor GUI or Cisco IMC UI to verify
for the correct firmware version.
To view the firmware version, in the IMC Supervisor GUI, navigate to the
Systems > Firmware Management tab. See Upgrading Firmware using IMC Supervisor
for more details.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 38
HyperFlex Edge Upgrade
Post Upgrade Tasks for HyperFlex Edge
Step 2 Step 3
Step 4 Step 5
Log into any controller VM through SSH.
ssh admin@controller_vm_ip
Confirm the HyperFlex Data Platform version.
hxcli cluster version
Cluster version: 5.5(1a) Node hx220-m5-node-1 version: 5.5(1a) Node
hx220-m5-node-3 version: 5.5(1a) Node hx220-m5-node-2 version: 5.5(1a) Node
hx220-m5-node-4 version: 5.5(1a)
Verify that the HX storage cluster is online and healthy.
hxcli cluster info|grep -i health Sample output: healthstate : healthy
state: healthy storage cluster is healthy
For each browser interface you use, empty the cache and reload the browser
page to refresh the HX Connect content.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 39
Post Upgrade Tasks for HyperFlex Edge
HyperFlex Edge Upgrade
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 40
1 2 C H A P T E R
Stretched Cluster Upgrade
· Overview, on page 41 · Upgrade Guidelines for Stretched Cluster, on page 42
· Upgrading HyperFlex Stretched Cluster Using HX Connect, on page 42 ·
Upgrading a Witness VM for HXDP Release 5.0(x) and Earlier, on page 44 ·
Manually Upgrading ESXi for Cisco HyperFlex Stretched Cluster, on page 46 ·
Configuring Stretched Cluster for UCS FW Upgrade, on page 47
Overview
This section provides information related to upgrading a Cisco HyperFlex
Stretched Cluster. The procedure for performing a Stretched Cluster upgrade is
similar to the regular HyperFlex cluster upgrade procedure. Cisco HyperFlex
Stretched cluster upgrade process involves upgrading the below 3 components:
· Cisco HyperFlex Data Platform · VMware vSphere ESXi · Cisco UCS Server
Firmware You can combine HyperFlex Data Platform and VMware ESXi upgrade in a
single combined upgrade for HyperFlex Stretched clusters. Cisco recommends
upgrading these 2 components in a combined upgrade from HyperFlex Connect. You
can choose to upgrade one or two of these components at a time. If you prefer
to upgrade individual components one at a time, see Upgrade UCS Firmware, HX
Data Platform and VMware vSphere – Individual Component Upgrade, on page 25.
The component upgrade process for standard cluster and HyperFlex Stretched
cluster are the same. This section describes the steps to perform a combined
upgrade of HyperFlex Data Platform and VMware vSphere ESXi. In this process,
the HyperFlex nodes will go through an optimized rolling reboot without any
workload disruptions by use of VMware vMotion.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 41
Upgrade Guidelines for Stretched Cluster
Stretched Cluster Upgrade
Upgrade Guidelines for Stretched Cluster
Important Do not upgrade HyperFlex Stretch clusters to HXDP 5.0(2b) See
Software Advisory https://www.cisco.com/
c/en/us/td/docs/unified_computing/ucs/sw/SA/sw-advisory-hyperflex-
release-5-0-2b.html
· Upgrade of UCS server firmware is not supported through HX Connect. UCS
firmware upgrade must be done manually using Cisco UCS Manager. See Manage
Firmware through Cisco UCS Manager.
· Upgrade of the HyperFlex Witness node is not required when upgrading
stretched clusters but is strongly recommended. Refer to the Cisco HyperFlex
Software Requirements and Recommendations for the latest witness version
available.
· Health Check — Cisco recommends running this proactive health check on your
HyperFlex cluster prior to upgrade. For more details, see Pre-Upgrade
Intersight Health Check, on page 9
Upgrading HyperFlex Stretched Cluster Using HX Connect
Before you begin · Complete pre-upgrade validation checks. · Download the
latest Cisco HX Data Platform Upgrade Bundle for upgrading existing clusters
from previous releases, from Software Download . · Upgrade Cisco UCS
Infrastructure. · Upgrade UCS server firmware. For more information, see
Manage Firmware through Cisco UCS Manager. · Disable snapshot schedule, on the
storage controller VM. SSH to HyperFlex cluster IP and run the command stcli
snapshot-schedule disable snapshot schedule. · If DRS is Enabled and set to
fully automatic mode, the VMs are automatically migrated to other hosts with
vMotion.
Note If DRS is Disabled, vMotion the VMs manually to continue the upgrade
process when prompted. For more information, see VMware Documentation for
Migration with vMotion.
Step 1
Step 2 Step 3
Log in to HX Connect. a) Enter the administrative username and password. b)
Click Login.
In the Navigation pane, select Upgrade. On the Select Upgrade Type page,
select HX Data Platform and ESXi and complete the following fields:
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 42
Stretched Cluster Upgrade
Upgrading HyperFlex Stretched Cluster Using HX Connect
Step 4
Step 5 Step 6 Step 7 Step 8 Step 9
On the Select Upgrade Type page, select HX Data Platform and complete the following fields:
UI Element
Essential Information
Drag the HX file here or click to browse
Upload the latest Cisco HyperFlex Data Platform Upgrade Bundle for upgrading
existing clusters with previous release.tgz package file from Download
Software – HyperFlex HX Data Platform.
Sample file name format: storfs-packages-4.5.1a-31601.tgz.
Current version
Displays the current HyperFlex Data Platform version.
Current cluster details
Lists the HyperFlex cluster details like the HyperFlex version and Cluster upgrade state.
Bundle version
Displays the HyperFlex Data Platform version of the uploaded bundle.
(Optional) Checksum field
The MD5 Checksum number is available by hovering over the filename in the
Cisco.com Software Download section.
This is an optional step that helps you verify the integrity of the uploaded
upgrade package bundle.
Upload the VMware ESXi custom image offline upgrade bundle. Provide vCenter login credentials:
Essential Information
Essential Information
User Name field
Enter the vCenter
Admin Password field
Enter the vCenter
Click Upgrade to begin the combined upgrade process.
The Validation screen on the Upgrade Progress page displays the progress of
the checks performed. Fix validation errors, if any.
Note
At this point, all pre-upgrade checks and validations are running, along with the initial upgrade stage. Within
a few minutes, HX Connect returns and prompts you to confirm and start the second stage of the upgrade.
The upgrade is not complete until both steps are performed in the UI. The system should never be left in a
state where only the first step of the upgrade is complete.
The HyperFlex Connect UI refreshes after the first step of the upgrade, and a banner pops up prompting you to provide the UCS and vCenter credentials and start the second stage of the upgrade process. Monitor the upgrade page and confirm that the upgrade is complete.
When upgrade is in progress, you may see an error message Websocket connection failed. Automatic refresh disabled. You can either refresh the page or log out and log back in to clear the error message. You can safely ignore this error message.
Note
Perform post upgrade tasks once the upgrade is complete. If the upgrade fails, you can re-try the upgrade or
contact Cisco TAC for further assistance.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 43
Upgrading a Witness VM for HXDP Release 5.0(x) and Earlier
Stretched Cluster Upgrade
Upgrading a Witness VM for HXDP Release 5.0(x) and Earlier
Before you begin
Attention
This workflow is for use with existing Stretched Cluster installs (HXDP Release 5.0(x) and earlier). New Stretched Clusters installed using HXDP 5.5(1a) and later will auto-configure an Invisible Cloud Witness for site arbitration. Invisible Cloud Witness automatically runs the latest version, user maintenance of this component is not required.
· Select the Witness VM version that supports the HXDP version you are
upgrading to.
For supported versions see the HX Data Platform Software Versions for
HyperFlex Witness Node for Stretched Cluster section of the Cisco HyperFlex
Software Requirements and Recommendations
for your respective upgrade. · Upgrade HyperFlex Stretched Cluster.
· The upgraded HyperFlex Stretched Cluster must be in healthy state. To check
the health state of Stretched Cluster after upgrade, run the following
command:
root@StCtlVM:~# hxcli cluster info | grep healthy
Step 1 Step 2 Step 3 Step 4
Step 5
Log into the witness VM using SSH and execute the following command to stop
the service exhibitor.
root@WitnessVM:~# service exhibitor stop
Copy the exhibitor.properties file available in the /usr/share/exhibitor/ path
to a remote machine from where you can retrieve the exhibitor.properties file.
scp root@
Log out from the Witness VM. Power off and rename the Witness VM to WitnessVM.old.
Note
Confirm that the IP address of the old Witness VM is unreachable, using the ping command.
Deploy a new Witness VM and configure the same IP address as the old Witness VM.
Note
If the IP address is not reachable, the Witness OVA deployment may contain stale entries in the
/var/run/network directory. You must manually remove these entries and reboot the VM to have
the assigned IP address become reachable on the network.
To reboot the VM, open the VM console in vCenter/vSphere and execute the following command:
rm -rf /var/run/network/* reboot
Log into the new witness VM using SSH and execute the following command to
stop the service exhibitor.
root@WitnessVM:~# service exhibitor stop
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 44
Stretched Cluster Upgrade
Upgrading a Witness VM for HXDP Release 5.0(x) and Earlier
Step 6 Step 7
Step 8
Step 9
Copy the exhibitor.properties file from the remote machine (copied in Step 2)
to the /usr/share/exhibitor/ path of the new Witness VM.
scp /directory/exhibitor.properties root@
Verify if the following symlinks are preserved in the new Witness VM:
root@Cisco-HX-Witness-Appliance:~# cd /etc/exhibitor/ root@Cisco-HX-Witness-
Appliance:/etc/exhibitor# ls -al total 8 drwxr-xr-x 2 root root 4096 Sep 11
13:00 . drwxr-xr-x 88 root root 4096 Sep 11 12:55 .. lrwxrwxrwx 1 root root 41
Sep 11 13:00 exhibitor.properties lrwxrwxrwx 1 root root 37 Jul 24 16:49
log4j.properties
If the symlinks are not available, execute the following command:
root@Cisco-HX-Witness-Appliance:/etc/exhibitor# ln -s
/usr/share/exhibitor/exhibitor.properties exhibitor.properties
root@Cisco-HX-Witness-Appliance:/etc/exhibitor# ln -s /usr/share/
exhibitor/log4j.properties log4j.properties root@Cisco-HX-Witness-
Appliance:/etc/exhibitor# ls -al total 8 drwxr-xr-x 2 root root 4096 Sep 11
13:00 . drwxr-xr-x 88 root root 4096 Sep 11 12:55 .. lrwxrwxrwx 1 root root 41
Sep 11 13:00 exhibitor.properties -> /usr/share/exhibitor/exhibitor.properties
lrwxrwxrwx 1 root root 37 Jul 24 16:49 log4j.properties
-> /usr/share/exhibitor/log4j.pr
Important Skip this step if your upgrade is an HX Release prior to version
4.5(2a).
Get the cluster UUID value from /etc/springpath/clusteruuid file on any
Controller VM and then edit /usr/share/zookeeper/conf/server.jaas.conf and
replace HX_PLACEHOLDER with Cluster UUID.
Example:
On any Storage Controller VM, get the contents of clusteruuid file -cat
/etc/springpath/clusteruuid 5b4f718033594dad:663273adab09777a
On the Witness VM, replace the HX_PLACEHOLDER value with the cluster UUID
-Before -QuorumServer {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_hyperflex=”HX_PLACEHOLDER”;};QuorumLearner {
org.apache.zookeeper.server.auth.DigestLoginModule required
username=”hyperflex” password=”HX_PLACEHOLDER”;}; After -QuorumServer
{org.apache.zookeeper.server.auth.DigestLoginModule required
user_hyperflex=”5b4f718033594dad:663273adab09777a”;}; QuorumLearner {
org.apache.zookeeper.server.auth.DigestLoginModule required
username=”hyperflex” password=”5b4f718033594dad:663273adab09777a”;};
Note
This step is required for users that are moving to Witness VM Node version 1.1.1 and later, if the Witness
VM being upgraded is a version previous to 1.1.1.
Run the /usr/share/springpath/storfs-misc/setexhibitorconfig.sh command to upgrade the Witness exhibitor configuration.
Note
The setexhibitorconfig.sh automates the process of editing the exhibitor.properties
file, and replaces all of the data IP addresses with the management IP addresses for each corresponding
controller VM.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 45
Manually Upgrading ESXi for Cisco HyperFlex Stretched Cluster
Stretched Cluster Upgrade
Step 10
Note
It is normal for this command to not show any output when upgrading from a Witness VM that is older
than 1.1.1.
Start the service exhibitor by executing the following command:
root@Cisco-HX-Witness-Appliance:~# service exhibitor start exhibitor
start/running, process
Manually Upgrading ESXi for Cisco HyperFlex Stretched Cluster
Step 1 Step 2 Step 3
Step 4
Step 5 Step 6 Step 7
Select one of the hosts and put it in HX maintenance mode using the vSphere
Web Client. After the host enters maintenance mode, complete the following
steps.
Copy files using SCP, start the SSH service in the destination ESXi hosts as
well.
Note
· On HX240, you can use the local SpringpathDS datastore or a mounted HX datastore.
· On HX220, you can use either a mounted HX datastore or create a temporary RAM disk.
scp local_filename user@server:/path/where/file/should/go
Log into ESXi, and execute the following command to query the list of
available image profiles and for profile name verification.
esxcli software sources profile list -d
Example:
[root@localhost:~] esxcli software sources profile list -d /vmfs/volumes /5d3a21da-7f370812-ca58-0025
b5a5a102/HX-ESXi-7.0U3-20328353-Cisco-Custom-7.3.0.10-upgrade-bundle.zip
Name
Vendor Acceptance Level Creation Time
Modification
Time
——————————————- —— —————- ——————-
——————-
HX-ESXi-7.0U3-20328353-Cisco-Custom-7.3.0.10 Cisco PartnerSupported 2019-04-02T00:14:56
2019-04-02T13:38:34
Run the following command to perform the upgrade.
esxcli software profile update -d
Example:
[root@HX-ESXi-01:/vmfs/volumes/1a234567-89bc1234] esxcli software profile
update -d /vmfs/volumes/1a234567-89bc1234/HX-Vmware-ESXi-7.0U3-20328353-Cisco-
Custom-7.3.0.10.zip -p HX-ESXi-7.0U3-20328353-Cisco-Custom-7.3.0.10
After the ESXi host comes up, verify that the host has booted up with the
correct version.
vmware -vl
Exit maintenance mode using the vSphere Web Client. Ensure that the cluster becomes healthy between each ESXi upgrade.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 46
Stretched Cluster Upgrade
Configuring Stretched Cluster for UCS FW Upgrade
Step 8
hxcli cluster info [flags]
Repeat this process for all hosts in the cluster in a sequence.
Note
Make sure that the cluster becomes healthy between each ESXi upgrade.
Configuring Stretched Cluster for UCS FW Upgrade
During upgrade, the following customized UCS policies are validated and
adjusted for HyperFlex:
· HFP (Host Firmware Package) – Host Firmware Packages provide consistent
firmware files for the multiple components of a HyperFlex node. This includes
CIMC, BIOS, HBA and SAS Expander firmware, VIC and other components. Unlike
typical UCS Host Firmware Packages, they also control disk firmware, due to
the criticality of this to HyperFlex Data Platform. Note that Self Encrypting
Drives (SED) firmware is controlled by HyperFlex Data Platform directly and
not UCS Manager policies.
· VNIC Templates – Virtual NIC (VNIC) templates provide consistent
configuration of VNIC’s between UCS fabrics. HyperFlex VNIC Templates are
configured as redundancy pairs to ensure changes to HyperFlex VNIC’s on one
UCS fabric is applied to the other.
· Ethernet Adaptor Policies – Ethernet Adaptor Policies provide performance
related properties for HyperFlex VNIC’s.
· BIOS Policies – BIOS policies control configuration and performance of key
hardware resources on a HyperFlex node, such as CPU and Memory. HyperFlex uses
specific configuration to provide consistent high performance.
· VNIC/VHBA Placement Policies – VNIC/VHBA placement policies determine the
PCI addresses presented to the HyperFlex node for a given VNIC/VHBA. HyperFlex
sets this in a consistent manner so further configuration can proceed
successfully.
Step 1 Step 2 Step 3
Step 4
SSH to any CVM on a site and change directory into /tmp Use the su diag Enter
into the diag shell on a storage controller VM.2
Run the following command: /usr/local/bin/hx.py –upgrade-cluster-config. This
generates a file called “customer_site_config.json” and saves it in the /tmp
directory.
Edit the customer_site_config.json file to change the firmware version and the
org name appropriately. For example:
Example:
{ “id”: “Advanced”, “collapse”: true, “label”: “Advanced”, “groups”: [ { “id”:
“firmware”, “label”: “UCS Firmware”, “items”: [
2 HXDP Release 5.0(2a) and later support the su diag command. For more information, see the Diag User Overview.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 47
Configuring Stretched Cluster for UCS FW Upgrade
Stretched Cluster Upgrade
Step 5
Step 6 Step 7 Step 8
{ “id”: “version”, “label”: “UCS Firmware Version”, “type”: “text”, “description”: “UCS Firmware Version to be used on the HX servers”, “placeholder”: “ex: 3.2(2d)”, “defaultValue”: “3.2(2d)”, “value”: “4.1(1d)”
<<<<<—————– Change this
}, {
“id”: “version-m5”, “label”: “UCS Firmware Version”, “type”: “text”,
“description”: “UCS Firmware Version to be used on the M5 HX servers”,
“placeholder”: “ex: 3.2(2d)”, “defaultValue”: “3.2(2d)”, “value”: “4.1(1i)”
<<<<<—————– Change this } ] }, { “id”: “org”, “items”: [
{ “id”: “name”, “label”: “Hyperflex Org name”, “type”: “text”, “value”:
“Faridabad”, #<<<<<—————– Change this “description”: “The name of the org in
ucsm which is to be used for creation
of all the policies and profiles for this Hyperflex cluster” }
] } ]
Execute the command again and enter the UCSM IP and credentials. For example:
/usr/local/bin/hx.py –upgrade-cluster-config
Example:
[root@SpringpathControllerVP0RX5DWTC:/# /usr/local/bin/hx.py –upgrade-cluster-
config [UCS Manager] [in_progress][ 0.00%][ETA: 0:18:00] Login to UCS API UCS
host name or virtual IP address: 10.42.17.11 Connecting to admin@10.42.17.11…
Password:
Ensure that the command runs without any error. If there is an error, contact Cisco TAC.
Note
Note that this command (hx.py) is being run for the first site FI domain. You need to run the same steps for
the second site FI domain later.
Perform the following steps in vCenter and UCSM: a) Verify that Pending reboot
appears in the pending activities of the UCSM. b) Put one host in maintenance
mode. c) Reboot the server and then wait for the server to come online and
cluster to be online/healthy. d) Perform the same steps for the remaining
nodes.
Repeat Steps 4, 5 and 6 again for the other site.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 48
1 3 C H A P T E R
HyperFlex Offline Upgrade Workflow
· Offline Upgrade Guidelines, on page 49 · Offline Upgrade Process Workflow,
on page 49
Offline Upgrade Guidelines
Important
· Offline upgrade can be performed from HX Connect UI for either combined or split upgrade. Before you proceed, consider following guidelines:
· Cisco recommends performing online upgrade from HX Connect UI for hitless
upgrade experience without any impact operations.
· Offline upgrade requires the cluster to be shutdown.
· Nodes are upgraded with the new version of the Cisco HX Data Platform
software and rebooted one at a time.
· Offline cluster upgrades with nested vCenter is not supported.
Offline Upgrade Process Workflow
Step
Description
Reference
If UCSM (A-bundle) or UCS Server Firmware Software Download
(C-bundle) upgrade is required, download
Cisco UCS Infrastructure A, blade bundle B,
and rack bundle C.
Upgrade Cisco UCS Infrastructure bundle as Upgrade UCS Infrastructure Firmware, on
required.
page 18
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 49
Offline Upgrade Process Workflow
HyperFlex Offline Upgrade Workflow
Step
Description
Reference
Launch the vSphere Web Client and power For more details, see Shut Down and Power
down all user VMs (HyperFlex Controller Off the Cisco HX Storage Cluster.
VMs should stay powered on) residing on HX
servers and all user VMs running on HX
datastores. This includes VMs running on
Compute-only nodes. After the VMs are shut
down, verify the health state of the cluster and
perform a graceful shutdown.
Important
HyperFlex controller VMs (stCtlVMs) must remain powered on.
(Optional) SSH to cluster management IP as Run the command stcli snapshot- schedule
admin user and disable the snapshot schedule. –disable.
Log into HX Connect as admin user and For a combined upgrade see Upgrading
perform either a combined upgrade or an HyperFlex Data Platform Software, VMware
individual component upgrade.
ESXi, and Cisco UCS Server Firmware using
HX Connect, on page 21.
For an individual component upgrade see Overview.
Confirm the upgrade is complete and perform Post Upgrade Tasks, on page 31
the post upgrade tasks .
Start the cluster.
Preparing for HX Storage Cluster Maintenance
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 50
1 4 C H A P T E R
Manual Pre-Upgrade Validations
· Overview, on page 51 · Checking Cluster Storage Capacity, on page 51
Overview
This section describes the manual pre-upgrade validations if you are unable to
run the Hypercheck Health Check described in Pre-Upgrade Intersight Health
Check, on page 9. Running Hypercheck is highly recommended over performing
these checks manually.
Checking Cluster Storage Capacity
Cisco recommends that you check the cluster storage capacity before starting
the upgrade of an existing installation of Cisco HX Data Platform. If the
storage utilization in the cluster is above 70%, the upgrade validation fails.
When storage utilization in the cluster is above 70%, an offline upgrade is
recommended. Refer to the HX Storage Cluster Overview chapter in the Cisco
HyperFlex Data Platform Administration Guide for background details about
checking cluster storage capacity.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 51
Checking Cluster Storage Capacity
Manual Pre-Upgrade Validations
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 52
1 5 C H A P T E R
HyperFlex Upgrade Troubleshooting
· HXDP Release 5.5(1a) Upgrade Error on M4 Servers, on page 53 · VMs Do Not
Migrate During Upgrade, on page 54 · ESXi Host or HyperFlex Controllers In
Lockdown Mode, on page 54 · Connection to HX Connect Lost During Upgrade , on
page 55 · Failed to Upgrade HyperFlex VIBs , on page 55 · HX Connect UCS
Server Firmware Selection Dropdown Doesn’t List the Firmware Version 4.1 or
Above,
on page 56 · Upgrade Fails in the Step – Entering Cluster Node into
Maintenance Mode, on page 56 · Maintenance Mode Not Automatic for Cluster
Containing VMs with vGPUs Configured, on page 57
HXDP Release 5.5(1a) Upgrade Error on M4 Servers
Description Starting with Cisco HyperFlex Release 5.5(1a), M4 servers are not
supported. Attempts to upgrade clusters containing M4 or earlier HX generation
servers to HXDP Release 5.5(1a) or later will fail in the pre-upgrade phase.
The upgrade and activity pages display an error in the Bootstrap Upgrade step.
In some cases, the user is unable to view the error message and is shown a
successful upgrade whenin reality the upgrade has failed. The fallback
mechanism is to display a ClusterUpgradeFailed event along with a banner
stating that the attempted upgrade is disallowed.
Symptom An alert is generated and a banner appears stating that One or more M4
platform nodes was detected in the cluster which is not supported starting
HXDP 5.5(1a). Please follow the graceful node removal procedure to remove
these nodes from the cluster or work with TAC to migrate from these nodes and
retry the upgrade.
Note The message center is populated with the same error message.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 53
VMs Do Not Migrate During Upgrade
HyperFlex Upgrade Troubleshooting
Action Gracefully remove the upsupported nodes and retry the upgrade or
contact TAC for further assistance.
VMs Do Not Migrate During Upgrade
Description Upgrading ESXi cluster fails with error “Node maintenance mode
failed”. This occurs in an online and healthy ESXi cluster where DRS and HA is
enabled.
Action Try the following workarounds in the following order: 1. If HA
admission control policy is enabled and set to slot policy, change it to
cluster resource percentage
to tolerate one host failure and then retry upgrade. 2. Disable HA admission
control policy or disable HA and then retry upgrade. 3. Try powering off a few
VMs to make sure there is enough failover capacity in the cluster to tolerate
at
least one node failure and then retry upgrade.
ESXi Host or HyperFlex Controllers In Lockdown Mode
Description If the ESXi host is in lockdown mode, pre-upgrade validation will
fail with the error message auth cancel.
Action Enable/Disable Lockdown Mode mode on the ESXi host and enable it after
the upgrade is successful. Using HyperFlex Controller VMs 1. Log into HX
Connect. 2. On the Navigation pane, select System Overview. 3. On the System
Overview tab, from the Actions drop-down list, you can enable or disable
access to the
controller VM using SSH as an administrator.
Using ESXi Hosts 1. Log into vSphere Web Client. 2. Browse to the host in the
vSphere Web Client inventory. 3. Click the Manage tab and click Settings. 4.
Under System, select Security Profile. 5. In the Lockdown Mode panel, click
Edit.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 54
HyperFlex Upgrade Troubleshooting
Connection to HX Connect Lost During Upgrade
6. Click Lockdown Mode and set the mode to Disabled.
Connection to HX Connect Lost During Upgrade
Description Connection to HX Connect lost after pre-upgrade step from HX
3.5(2g) to HX 4.0(2a). During the upgrade, if there is an expired certificate
in the upgrade source version, the browser will log user out after pre-upgrade
step. This is accepted secure behavior since the certificate of the server has
changed after pre-upgrade.
Action Refresh the browser and login again.
Failed to Upgrade HyperFlex VIBs
Description HXDP Upgrade to HX 4.5(1a) or above fails with error – “Failed to
upgrade HyperFlex VIBs . Reason: Some(System error)”. The following error logs
appear in the ESXi esxupdate.log file:
2020-12-01T11:59:22Z esxupdate: 333049: root: ERROR:
vmware.esximage.Errors.LiveInstallationError: ([], ‘([], “Error in running rm
/tardisks/scvmclie.v00:\nReturn code: 1\nOutput: rm: can’t remove
‘/tardisks/scvmclie.v00’: Device or resource busy\n\nIt is not safe to
continue. Please reboot the host immediately to discard the unfinished
update.”
Action Follow these steps to kill the process corresponding to getstctlvmlogs
and retry the upgrade. 1. SSH to ESXi with root login. 2. Run the command ps
-c | grep -e cisco -e springpath and note the process ID (PID). For example:
ps -c | grep -e cisco -e springpath 112056 112056 sh /bin/sh
/opt/springpath/support/getstctlvmlogs
3. Kill the process using the command kill -9
kill -9 112056
4. Go back to HX Connect or Intersight and retry the upgrade. If the issue
still persists, please contact Cisco TAC for assistance.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 55
HX Connect UCS Server Firmware Selection Dropdown Doesn’t List the Firmware Version 4.1 or Above
HyperFlex Upgrade Troubleshooting
HX Connect UCS Server Firmware Selection Dropdown Doesn’t List the Firmware
Version 4.1 or Above
Description When you try to perform a combined upgrade from the HX Connect UI,
the dropdown to select UCS server firmware doesn’t show version 4.1 or later.
Action Log into UCS Manager and confirm you have uploaded the UCS B and C
firmware bundles to the Fabric Interconnect. If not, upload them and re-try
the upgrade. If the UCS B and C firmware bundles are already uploaded to the
Fabric Interconnect, apply below workaround to continue with upgrade. 1. From
the HX Connect upgrade page, select HX Data Platform only. 2. Browse and
select the appropriate HXDP upgrade package for your upgrade.3 3. Enter your
vCenter credentials. 4. Click Upgrade. This will bootstrap the management
components. Refresh the UI screen. 5. Once the UI is refreshed, try the
combined upgrade procedure. You should now be able to see the UCS
server firmware version 4.1 or above listed in the dropdown menu.
Upgrade Fails in the Step – Entering Cluster Node into Maintenance Mode
Description Failure at the Entering Cluster Node into Maintenance Mode step is
caused because of a MTU mismatch in the vSwitch and port-groups. If the
cluster has a node that was added at a later point using the node expansion
method, the newly added node may have the MTU set to 9000 while other nodes
are set to MTU 1500.
Note Below remediation is applicable only if your cluster has one or more
nodes that were added as part of a cluster expansion, and they have the MTU
set to 9000 while your original cluster nodes are set at a MTU of 1500. If
this is not the scenario, please contact TAC for further assistance.
Action · Log into vCenter. · Check and confirm the MTU value set on all nodes.
3 The version must be HXDP 4.5 or later.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 56
HyperFlex Upgrade Troubleshooting
Maintenance Mode Not Automatic for Cluster Containing VMs with vGPUs Configured
· If the nodes that were part of the originally built cluster are set at a MTU
of 1500 and some of the other nodes (nodes added later as part of cluster
expansion) have the MTU set to 9000, change the MTU on all such nodes to 1500.
· Retry the upgrade.
Maintenance Mode Not Automatic for Cluster Containing VMs with vGPUs
Configured
Description For clusters that contain VMs with vGPUs configured, entering
maintenance mode does not occur automatically even with DRS fully enabled.
During rolling upgrades, it is necessary to manually handle these VMs to
ensure that each ESXi host can enter maintenance mode and continue with the
upgrade at the appropriate time.
Action You can use one of these methods to proceed forward: 1. Manually
vMotion the vGPU configured VMs to another ESXi host in the cluster. 2.
Temporarily power off the vGPU configured VMs. They can be powered on again
after the ESXi host
reboots and rejoins the cluster
Note This is a limitation of DRS host evacuation and is documented, see “DRS
fails to migrate vGPU enabled VM’s automatically (66813) topic” on the VMware
documentation site.
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 57
Maintenance Mode Not Automatic for Cluster Containing VMs with vGPUs Configured
HyperFlex Upgrade Troubleshooting
Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 5.5 58
References
- Connect Dots
- Cisco UCS M-Series GUI Firmware Management Guide, Release 2.5 - Verifying that the Data Path is Ready [Cisco UCS Manager] - Cisco
- Cisco Press: Source for Cisco Technology, CCNA, CCNP, CCIE Self-Study | Cisco Press
- Data Center and Cloud Videos - Cisco Community
- Cisco Intersight Help Center
- Cisco Intersight Help Center
- Cisco Intersight Help Center
- Cisco: Software, Network, and Cybersecurity Solutions - Cisco
- Cisco Trademarks - Cisco
- Run Hypercheck: Hyperflex Health and Pre-Upgrade Check Tool - Cisco
- Cisco HyperFlex HX Data Platform - Release Notes - Cisco
- Cisco HyperFlex HX Data Platform - End-User Guides - Cisco
- Support - Cisco Support and Downloads – Documentation, Tools, Cases - Cisco
- Cisco Integrated Management Controller (IMC) Supervisor - Configuration Guides - Cisco
- Cisco UCS C-Series Rack Servers - End-User Guides - Cisco
- Support - Bug Search Tool Help - Cisco
- Cisco HyperFlex Data Platform Administration Guide, Release 5.0 - Logging in to HX Data Platform Interfaces [Cisco HyperFlex HX Data Platform] - Cisco
- Cisco HyperFlex Systems Ordering and Licensing Guide - Cisco HyperFlex Software Licensing [Cisco HyperFlex HX Data Platform] - Cisco
- Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 4.5 - Upgrade UCS Infrastructure Firmware [Cisco HyperFlex HX Data Platform] - Cisco
- Cisco HyperFlex Systems Upgrade Guide for Unsupported Cisco HX Releases - Cisco
- Cisco HyperFlex Systems Upgrade Guide for Unsupported Cisco HX Releases - Upgrading Cisco HyperFlex 2.1(1a), 2.1(1b) and 2.1(1c) [Cisco HyperFlex HX Data Platform] - Cisco
- Cisco HyperFlex Software Requirements and Recommendations - Cisco
- Cisco HyperFlex Software Requirements and Recommendations - Cisco HyperFlex Software Recommendations and Guidelines [Cisco HyperFlex HX Data Platform] - Cisco
- Cisco IMC Supervisor Rack-Mount Servers Management Guide, Release 2.0 - Firmware Profiles [Cisco Integrated Management Controller (IMC) Supervisor] - Cisco
- Cisco IMC Supervisor Rack-Mount Servers Management Guide, Release 2.2 - Firmware Profiles [Cisco Integrated Management Controller (IMC) Supervisor] - Cisco
- Cisco IMC Supervisor Rack-Mount Servers Management Guide, Release 2.2 - Firmware Profiles [Cisco Integrated Management Controller (IMC) Supervisor] - Cisco
- Cisco IMC Supervisor Rack-Mount Servers Management Guide, Release 2.2 - Firmware Profiles [Cisco Integrated Management Controller (IMC) Supervisor] - Cisco
- Cisco IMC Supervisor Rack-Mount Servers Management Guide, Release 2.2 - Managing Server Discovery, Rack Groups, and Rack Accounts [Cisco Integrated Management Controller (IMC) Supervisor] - Cisco
- Cisco IMC Supervisor Rack-Mount Servers Management Guide, Release 2.2 - Managing Server Discovery, Rack Groups, and Rack Accounts [Cisco Integrated Management Controller (IMC) Supervisor] - Cisco
- Cisco IMC Supervisor Rack-Mount Servers Management Guide, Release 2.2 - Managing Server Discovery, Rack Groups, and Rack Accounts [Cisco Integrated Management Controller (IMC) Supervisor] - Cisco
- Cisco HyperFlex Software Advisory for HX Release 5.0(2b) - Cisco
- Cisco UCS Manager Firmware Management Guide, Release 4.0 - Manage Firmware through Cisco UCS Manager [Cisco UCS Manager] - Cisco
- Cisco UCS Manager Firmware Management Guide, Release 4.2 - Guidelines and Prerequisites [Cisco UCS Manager] - Cisco
- Cisco UCS Manager System Monitoring Guide, Release 4.2 - Hardware Monitoring [Cisco UCS Manager] - Cisco
- Cisco HyperFlex Data Platform Administration Guide, Release 5.5 - Preparing for HX Storage Cluster Maintenance [Cisco HyperFlex HX Data Platform] - Cisco
Read User Manual Online (PDF format)
Read User Manual Online (PDF format) >>