6300 Series FI Cisco Hyper Flex
User Manual

Supported Versions and System Requirements for Cisco HXDP

Cisco HX Data Platform requires specific software and hardware versions, and networking settings for successful installation.
Table 1: Supported Versions and System Requirements for Cisco HXDP Release 5.5(x)

Confirm the component firmware on the server meets the minimum versions supported.| For more information, see the “FI/Server Firmware
– 5.5(x) Releases” topic in the Cisco Hyper Flex Software Requirements and Recommendations
document.
List of recommended browsers.| For more information, see the “Browser
Recommendations” topic in the Cisco Hyper Flex Software Requirements and Recommendations
document.

Required Hardware Cables

• 6200/6400/6500 series FI: Use at least two 10-Gb Small Form-Factor Pluggable (SFP) cables per server.
  6300 series FI: Use at least two 40-GbE QSFP cables per server.

• Ensure that the Fabric Interconnect console cable (CAB-CONSOLE-RJ45) has an RJ-45 connector on one end and a DB9 connector on the other. This cable is used to connect into the RS-232 console connection on a laptop.

• Ensure that the standard power cords have an IEC C13 connector on the end that plugs into the power supplies. Make sure that the optional jumper power cords have an IEC C13 connector on the end that plugs into the power supplies and an IEC C14 connector on the end that plugs into an IEC C13 outlet receptacle.
  For further details, see the Cisco UCS 6300 Series Fabric Interconnect Hardware Guide.

• The KVM cable provides a connection for the Cisco HX-Series Servers into the system. It has a DB9 serial connector, a VGA connector for a monitor, and dual USB 2.0 ports for a keyboard and mouse. With this cable, you can create a direct connection to the operating system and the BIOS running on the system.

Note
This same KVM cable is used for both UCS rack mount and blade servers.

For further details on cables and ordering information for M series servers, see the respective Cisco Hyper Flex HX-Series Models and Cisco UCS B200 Blade Server Installation and Service Note.

Host Requirements

A Cisco Hyper Flex cluster contains a minimum of three converged Hyper Flex nodes. There is an option of adding compute-only nodes to provide additional compute power if there is no need for extra storage. Each server in a Hyper Flex cluster is also referred as a Hyper Flex node. Make sure that each node has the following settings installed and configured before you deploy the storage cluster.
For further information, refer to the Cisco HX240c/220c Hyper Flex Node Installation Guides.
Ensure that the following host requirements are met.

• Use the same VLAN IDs for all the servers (node or hosts) in the cluster.
• Use the same administrator login credentials for all the ESXi servers across the storage cluster.
• Keep SSH enabled on all ESXi hosts.
• Configure DNS and NTP on all servers.
• Install and configure VMware vSphere.
• VIC and NIC Support: For details, see the Cisco HyperFlex Systems—Networking Topologies document.

Disk Requirements

The disk requirements vary between converged nodes and compute-only nodes. To increase the available CPU and memory capacity, you can expand the existing cluster with compute-only nodes as needed. These compute-only nodes provide no increase to storage performance or storage capacity.
Alternatively, adding converged nodes increase storage performance and storage capacity alongside CPU and memory resources.
Servers with only Solid-State Disks (SSDs) are All-Flash servers. Servers with both SSDs and Hard Disk Drives (HDDs) are hybrid servers.

The following applies to all the disks in a Hyper Flex cluster:

• All the disks in the storage cluster must have the same amount of storage capacity. All the nodes in the storage cluster must have the same number of disks.
• All SSDs must support TRIM and have TRIM enabled.
• All HDDs can be either SATA or SAS type. All SAS disks in the storage cluster must be in a pass-through mode.
• Disk partitions must be removed from SSDs and HDDs. Disks with partitions are ignored and not added to your HX storage cluster.
• Moving operational disks between servers within same cluster or moving them into expansion nodes within the same active cluster is not supported.
• Optionally, you can remove or backup existing data on disks. All existing data on a provided disk is overwritten.

Note
New factory servers are shipped with appropriate disk partition settings. Do not remove disk partitions from new factory servers.

• Only the disks ordered directly from Cisco are supported.
• On servers with Self Encrypting Drives (SED), both the cache and persistent storage (capacity) drives must be SED capable. These servers support Data at Rest Encryption (DARE).
• In the event you see an error about unsupported drives or catalog upgrade, see the Catalog Update.

In addition to the disks listed in the table below, all M5/M6 converged nodes have M.2 SATA SSD with ESXi installed.

Note
Do not mix storage disks type or storage size on a server or across the storage cluster. Mixing storage disk types is not supported.

• When replacing cache or persistent disks, always use the same type and size as the original disk.
• Do not mix any of the persistent drives. Use all HDD or SSD and the same size drives in a server.
• Do not mix hybrid and All-Flash cache drive types. Use the hybrid cache device on hybrid servers and All-Flash cache devices on All-Flash servers.
• Do not mix encrypted and non-encrypted drive types. Use SED hybrid or SED All-Flash drives. On SED servers, both the cache and persistent drives must be SED type.
• All nodes must use same size and quantity of SSDs. Do not mix SSD types.

Please refer to the corresponding server model spec sheet for details of drives capacities and number of drives supported on the different servers.
For information on compatible PIDs when performing an expansion of existing cluster, please refer to the Cisco Hyper Flex Drive Compatibility document.

Compute-Only Nodes
The following table lists the supported compute-only node configurations for compute-only functions. Storage on compute-only nodes is not included in the cache or capacity of storage clusters.

Note
When adding compute nodes to your Hyper Flex cluster, the compute-only service profile template automatically configures it for booting from an SD card. If you are using another form of boot media, update the local disk configuration policy. See the Cisco UCS Manager Server Management Guide for server-related policies.

• Cisco B200 M5/M6
• C240 M5/M6
• C220 M5/M6
• C480 M5
• B480 M5| Choose any method.
Ensure that only one form of boot media is exposed to the server for ESXi installation. Post install, you may add in additional local or remote disks.
USB boot is not supported for HX Compute-only nodes.
Important
• SD Cards in a mirrored configuration with ESXi installed.
• Local drive HDD or SSD.
• SAN boot.
• M.2 SATA SSD Drive.
Note
HW RAID M.2 (UCS-M2-HWRAID and HX-M2-HWRAID) is a supported boot configuration starting with HX Data Platform version 4.5(1a) and later.

Port Requirements

If your network is behind a firewall, in addition to the standard port requirements, VMware recommends ports for VMware ESXi and VMware vCenter.

• CIP-M is for the cluster management IP.
• SCVM is the management IP for the controller VM.
• ESXi is the management IP for the hypervisor.

The comprehensive list of ports required for component communication for the HyperFlex solution is located in Appendix A of the HX Data Platform Security Hardening Guide
Tip
If you do not have standard configurations and need different port settings, refer to Appendix A of the HX Data Platform Security Hardening Guide for customizing your environment.

Hyper Flex External Connections

External Connection| Description| IP Address/ FQDN/Ports/Version| Essential Information
---|---|---|---
Intersight Device
Connector| Supported HX systems are connected to Cisco Intersight through a device connector that is embedded in the management controller of each system.| HTTPS Port Number: 443 1.0.5-2084 or later
(Auto-upgraded by Cisco Intersight)| All device connectors must properly resolve svc.intersight.com and allow outbound-initiated
HTTPS connections on port 443. The current HX Installer supports the use of an HTTP proxy.
The IP addresses of ESXi management must be reachable from Cisco UCS Manager over all the ports that are listed as being needed from installer to ESXi management, to ensure
deployment of ESXi management from Cisco Intersight.
Note
Outbound HTTPS connections on port 443 initiated by ESXi hosts can be blocked by the default ESXi firewall. The ESXi firewall can be temporarily disabled to allow this  connectivity.
To disable the ESXi firewall, use the esxcli network firewall set –enabled=false command and after the installation has completed use the esxcli network firewall set –enabled=false
command to re-enable the firewall.
For more information, see the Network Connectivity Requirements section of the Intersight Help Center.
Auto Support| Auto Support (ASUP) is the alert notification service provided through
HX Data Platform.| SMTP Port Number: 25| Enabling Auto Support is strongly recommended because it provides historical hardware
counters that are valuable in diagnosing future hardware issues, such as a drive failure for a node.

Fabric Interconnect Uplink Provisioning

Prior to setting up the Hyper Flex cluster, plan the upstream bandwidth capacity for optimal network traffic management. This ensures that the flow is in steady state, even if there is a component failure or a partial network outage.
By default, the hx-vm-network vSwitch is configured as active/active. All other vSwitches are configured as active/standby.

Note
All VLANs (including storage-data and vmotion) must be configured upstream. It is imperative that storage-data is configured upstream to prevent fail-back timing issues which require temporary upstream connectivity.

Note
For clusters running Catalyst switches upstream to the FI’s, set the best effort Quality of Service (QOS) MTU to 9216 (located in LAN > LAN Cloud > QoS System Class), otherwise failover will fail.

Note:

1. Dotted lines represent a “standby” link.
2. All “a” vNICs connect to FI-A.
3. All “b” vNICs conect to FI-B.
4. MTU of 9000 is needed for storage-data and vmotion networks.
5. All VLANs by default are tagged on the FI so frames are passed untagged to each vswitch.
6. The vm network port groups are automatically created in 1.8 installer with vlan suffix.

Set the default vSwitch NIC teaming policy and failover policy to yes to ensure that all management, vMotion, and storage traffic are locally forwarded to the fabric interconnects to keep the flow in steady state. When vNIC-a fails, ESXi computes the load balancing and all the virtual ports are repinned to vNIC-b. When vNIC-a comes back online, repinning does apply and virtual ports are rebalanced across vNIC-a and vNIC-b. This reduces the latency and bandwidth utilization upstream of the Cisco UCS fabric interconnects.

Fabric Interconnect Uplink Provisioning

In case one or more server links fail, for instance, if Host 1 loses connectivity to Fabric A while Host 2 loses connectivity to Fabric B, the traffic must go through the upstream switches. Therefore, the uplink network bandwidth usage increases, and you must add more uplinks.

Note
When you have uplinks from a fabric interconnect to two different upstream switches, you encounter a condition called Disjoint Layer 2 (DJL2) on the FI. This is known to happen on the FI on End Host Mode and if the DJL2 is not configured properly.
To deploy the DJL2 properly, refer to the Cisco UCS 6300 Series Fabric Interconnect Hardware Guide—Deploy Layer 2 Disjoint Networks Upstream in End Host Mode white paper.

Network Settings

Important
All IP addresses must be IPv4. Hyper Flex does not support IPv6 addresses.

Note
You cannot use the storage-data VLAN for User VM traffic.

Best Practices

• Must use different subnets and VLANs for each network.
• Directly attach each host to a Cisco UCS fabric interconnect using a 10-Gbps cable.
• Do not use VLAN 1 which is the default VLAN as it can cause networking issues, especially if Disjoint Layer 2 configuration is used.
• Installer sets the VLANs as non-native by default. Ensure to configure the upstream switches to accommodate the non-native VLANs.
• Uplinks from the UCS Fabric Interconnects to all top of rack switch ports need to hahve Port-fast, spanning-tree port type edge trunk, or similar spanning tree configuration that immediately put ports into forwarding mode. Configure spanning tree in edge trunk or portfast edge mode depending on the vendor and model of the switch. This extra configuration ensures that when links flap or change state, they do not transition through unnecessary spanning tree states and incur an extra delay before traffic forwarding begins. Failure to properly configure FI uplinks in portfast edge mode may result in network and cluster outages during failure scenarios and during infrastructure upgrades that leverage the highly available network design native to HyperFlex.

Each ESXi host needs the following networks.

• Management traffic network—From the vCenter, handles the hypervisor (ESXi server) management, and storage cluster management.

• Data traffic network— Handles the hypervisor and storage data traffic and is required to be a unique VLAN per Hyperflex Cluster.

• vMotion network

• VM network
  There are four vSwitches, each carrying a different network.

• vswitch-hx-inband-mgmt—Used for ESXi management, storage controller management, and Replication.
  These two vSwitches are further divided in two port groups with assigned static IP addresses to handle traffic between the storage cluster and the ESXi host.

• vswitch-hx-storage-data—Used for ESXi storage data and HX Data Platform ISCSI. These two vSwitches are further divided in two port groups with assigned static IP addresses to handle traffic between the storage cluster and the ESXi host. These two vSwitches are further divided in two port groups with assigned static IP addresses to handle traffic between the storage cluster and the ESXi host.

• vswitch-hx-vmotion—Used for VM and storage vMotion.
  This vSwitch, has one port group for management, defined through vSphere that connects to all the hosts in the vCenter cluster.
  • vswitch-hx-vm-network—Used for VM data traffic.
  You can add or remove VLANs on the corresponding vNIC templates in Cisco UCS Manager. See Managing VLANs in Cisco UCS Manager and Managing vNIC templates in Cisco UCS Manager for the detailed steps. To create port groups on the vSwitch, refer to Adding Virtual Port Groups to VMware Standard vSwitch.

Note

1. The Cisco HX Data Platform Installer automatically creates the vSwitches.
2. The following services in vSphere must be enabled after the HyperFlex storage cluster is created.

• DRS (Optional, if licensed)
• vMotion
• High Availability

VLAN and vSwitch Requirements

• Provide at least three VLAN IDs.
• All VLANs must be configured on the fabric interconnects during the installation.
• All VLANs (including storage-data and vmotion) must be configured upstream. It is imperative that storage-data is configured upstream to prevent fail-back timing issues which require temporary upstream connectivity.

Must use different subnets and VLANs for each of the following networks.
Note
VLAN ESXi and Hyper Flex Management
Traffic| VLAN Name: (for example “hx-inband-mgmt”)
VLAN ID:
VLAN Hyper Flex Storage Data| VLAN Name: (for example, “hx- storage-data”)
VLAN ID:
VLAN VM vMotion| VLAN Name: (for example,”hx-vmotion”)
VLAN ID:
VLAN VM Network| VLAN VM Network: (for example, “hx-vm- network”). This is required to be a unique VLAN Hyper Flex Cluster.
VLAN ID:

The VLAN tagging with External Switch VLAN Tagging (EST) and vSwitch settings are applied using UCS Manager profiles. The HX Data Platform Installer, simplifies this process.

Note

• Do not use VLAN 1 which is the default VLAN as it can cause networking issues, especially if Disjoint Layer 2 configuration is used. Use a different VLAN other than VLAN 1.
  Installer sets the VLANs as non-native by default. Configure the upstream switches to accommodate the non-native VLANs.

• Inband Management is not supported on VLAN 2 or VLAN 3.

Cisco UCS Requirements

Provide the listed content for the UCS Fabric Interconnect and UCS Manager when prompted.
Cisco UCS Fabric Interconnect Requirements

Ax or Optic).
Fabric Interconnect Cluster IP address| .
FI-A IP Address| .
FI-B IP Address| .
MAC Address Pool| Check 00:00:00 MAC address pool.
IP Blocks| KVM IP pool. A minimum of 4 IP addresses.
Subnet mask| For example, 255.255.0.0.
Default Gateway| For example, 10.193.0.1.

Cisco UCS Manager Requirements

Hypervisor Requirements

Enter the IP address from the range of addresses that are available to the ESXi servers on the storage management network or storage data network through vCenter. Provide static IP addresses for all network addresses.

Note

• Data and Management networks must be on different subnets.
• Data network IP addresses cannot be changed after the storage cluster is created. Contact Cisco TAC for assistance changing Management Network IPs.
• Though, not required by itself, if you are specifying DNS names, enable IP addresses forward and reverse DNS lookup.
• The installer IP address must be reachable from the management subnet used by the hypervisor and the storage controller VMs. The installer appliance must run on the ESXi host or on a VMware workstation that is not a part of the cluster to be installed.

Controller**