CISCO 9300-GX 9000 Series Nexus Switches User Manual

Upgrading or Downgrading the Cisco Nexus 9000
Series NX-OS Software

This chapter describes how to upgrade or downgrade the Cisco NX-OS software. It contains the following sections:

About the Software Image

Each device is shipped with the Cisco NX-OS software preinstalled. The Cisco NX-OS software consists of one NX-OS software image. Only this image is required to load the Cisco NX-OS operating system.
In Cisco NX-OS Release 10.1(1), 10.1(2) there are 32 and 64 bit images.

• The 32-bit Cisco NX-OS image file has the image filename that begins with “nxos” (for example, nxos.10.1.1.bin).
• The 64-bit Cisco NX-OS image file has the image filename that begins with “nxos64” (for example, nxos64.10.1.1.bin).

Note
Beginning with Cisco NX-OS Release 10.1(x), only 9300-GX platforms support 64-bit image.
For 32-bit or 64-bit image support on respective platforms, see the following documents:

• Cisco Nexus 9000 Series NX-OS Release Notes, Release 10.1(2)
• Cisco Nexus 9000 Series NX-OS Release Notes, Release 10.1(1)

The Cisco Nexus 9000 Series switches support disruptive software upgrades and downgrades by default.
Note
Another type of binary file is the software maintenance upgrade (SMU) package file. SMUs contain fixes for specific defects. They are created to respond to immediate issues and do not include new features. SMU package files are available for download from Cisco.com and generally include the ID number of the resolved defect in the filename (for example, n9000-dk10.1.1.CSCab00001.gbin). For more information on SMUs, see the Cisco Nexus 9000 Series NX-OS System Management Configuration Guide.
Note
Cisco also provides electronic programmable logic device (EPLD) image upgrades to enhance hardware functionality or to resolve known hardware issues. The EPLD image upgrades are independent from the Cisco NX-OS software upgrades. For more information on EPLD images and the upgrade process, see the Cisco Nexus 9000 Series FPGA/EPLD Upgrade Release Notes.

About ISSU

An in-service software upgrade (ISSU) allows you to upgrade the device software while the switch continues to forward traffic. ISSU reduces or eliminates the downtime typically caused by software upgrades. You can perform an ISSU, also known as a non-disruptive upgrade, for some switches. (See the ISSU Platform Support, on page 12 for a complete list of supported platforms.)
The default upgrade process is disruptive. Therefore, ISSU needs to be enabled using the command-line interface (CLI), as described in the configuration section of this document. Using the non-disruptive option helps ensure a non- disruptive upgrade. The guest shell is disabled during the ISSU process and it is later reactivated after the upgrade.
Enhanced ISSUs are supported for some Cisco Nexus 9000 Series switches.
The following ISSU scenarios are supported:

• Performing standard ISSU on Top-of-Rack (ToR) switches with a single supervisor
• Performing enhanced ISSU on Top-of-Rack (ToR) switches with a single supervisor

Performing Standard ISSU on Top-of-Rack (ToR) Switches with a Single Supervisor
The ToR Cisco Nexus 9300 platform switches and Cisco Nexus 3100 Series switches are the NX-OS switches with single supervisors. Performing ISSU on the Cisco Nexus 9000 and 3100 Series switches causes the supervisor CPU to reset and to load the new software version. After the CPU loads the updated version of the Cisco NX-OS software, the system restores the control plane to the previous known configuration and the runtime state and it gets in-sync with the data plane, thereby completing the ISSU process.
The data plane traffic is not disrupted during the ISSU process. In other words, the data plane forwards the packets while the control plane is being upgraded, any servers that are connected to the Cisco Nexus 9000 and 3100 Series switches do not see any traffic disruption. The control plane downtime during the ISSU process is approximately less than 120 seconds.
Performing Enhanced ISSU on Top-of-Rack (ToR) Switches with a Single Supervisor
Note
Enhanced ISSU to Cisco NX-OS Release 10.1(x) is not supported as there are kernel updates that cannot take effect without reloading the underlying kernel. The system will prompt the following message:
Host kernel is not compatible with target image. Full ISSU will be performed and control plane will be impacted.
In effect, system will perform non-disruptive ISSU instead of enhanced ISSU.
The Cisco NX-OS software normally runs directly on the hardware. However, configuring enhanced or container-based ISSU on single supervisor ToRs is accomplished by creating virtual instances of the supervisor modules and the line cards. With enhanced ISSU, the software runs inside a separate Linux container (LXC) for the supervisors and the line cards. A third container is created as part of the ISSU procedure, and it is brought up as a standby supervisor.
The virtual instances (or the Linux containers) communicate with each other using an emulated Ethernet connection. In the normal state, only two Linux containers are instantiated: vSup1 (a virtual SUP container in an active role) and vLC (a virtual linecard container). Enhanced ISSU requires 16G memory on the switch.
To enable booting in the enhanced ISSU (LXC) mode, use the [no] boot mode lxc command. This command is executed in the config mode. See the following sample configuration for more information:

Note
When you are enabling enhanced ISSU for the first time, you have to reload the switch first.
During the software upgrade with enhanced ISSU, the supervisor control plane stays up with minimal switchover downtime disruption and the forwarding state of the network is maintained accurately during the upgrade.
The supervisor is upgraded first and the line card is upgraded next.
The data plane traffic is not disrupted during the ISSU process. The control plane downtime is less than 6 seconds.
Note
In-service software downgrades (ISSDs), also known as non-disruptive downgrades, are not supported.
For information on ISSU and high availability, see the Cisco Nexus 9000 Series NX-OS High Availability and Redundancy Guide.

Prerequisites for Upgrading the Cisco NX-OS Software

Upgrading the Cisco NX-OS software has the following prerequisites:

• For ISSU compatibility for all releases, see the Cisco NX-OS ISSU Support Matrix.
• Ensure that everyone who has access to the device or the network is not configuring the device or the network during this time. You cannot configure a device during an upgrade. Use the show configuration session summary command to verify that you have no active configuration sessions.
• Save, commit, or discard any active configuration sessions before upgrading or downgrading the Cisco NX-OS software image on your device. On a device with dual supervisors, the active supervisor module cannot switch over to the standby supervisor module during the Cisco NX-OS software upgrade if you have an active configuration session.
• To transfer NX-OS software images to the Nexus switch through a file transfer protocol (such as TFTP, FTP, SFTP, SCP, etc.), verify that the Nexus switch can connect to the remote file server where the NX-OS software images are stored. If you do not have a router to route traffic between subnets, ensure that the Nexus switch and the remote file server are on the same subnetwork. To verify connectivity to the remote server, transfer a test file using a file transfer protocol of your choice or use the ping command if the remote file server is configured to respond to ICMP Echo Request packets. An example of using the ping command to verify connectivity to a remote file server 192.0.2.100 is shown below:

For more information on configuration sessions, see the Cisco Nexus 9000 Series NX-OS System Management Configuration Guide specific to your release.

Prerequisites for Downgrading the Cisco NX-OS Software

Downgrading the Cisco NX-OS software has the following prerequisites:

• Before you downgrade from a Cisco NX-OS release that supports the Control Plane Policing (CoPP) feature to an earlier Cisco NX-OS release that does not support the CoPP feature, you should verify compatibility using the show incompatibility nxos bootflash:filename command. If an incompatibility exists, disable any features that are incompatible with the downgrade image before downgrading the software.

Cisco NX-OS Software Upgrade Guidelines

Before attempting to upgrade to any software image, follow these guidelines:

• For a device that is running on Cisco Nexus Release 10.1(2), ND-ISSU is not supported if L2 sub-interfaces are configured.

• For ISSU compatibility for all releases, see the ISSU Support Matrix.

• Beginning from Cisco NX-OS Release 10.2(1), Cisco Nexus 9300 and 9500 platform switches support 64-bit image, and non-disruptive upgrade is supported from Cisco NX-OS Release 9.3(9) onwards. Beginning from Cisco NX-OS Release 10.2(1) onwards, Cisco Nexus 9300-FX3 supports non-disruptive upgrade.

• Beginning with Cisco NX-OS Release 10.1(1), during the disruptive upgrade to the 64-bit image or a downgrade from 64-bit to 32-bit image, if feature ITD is enabled, refer to Guidelines and Limitations for ITD in the Cisco Nexus 9000 Series NX-OS Intelligent Traffic Director Configuration Guide, Release 10.1(x), if the upgrade or downgrade proceeds with an ASCII reload.

• When you use install all with no-reload option, the saved configuration cannot be used before you reload the device. Saving configuration in this state can result in incorrect startup configuration once you reload the device with new version of NX-OS.

• When upgrading from Cisco NX-OS Release 9.3(3) to Cisco NX-OS Release 9.3(6) or later, if you do not retain configurations of the TRM enabled VRFs from Cisco NX-OS Release 9.3(3), or if you create new VRFs after the upgrade, the auto-generation of ip multicast multipath s-g-hash next-hop-based CLI, when feature ngmvpn is enabled, will not happen. You must enable the CLI manually for each TRM enabled VRF. For the configuration instructions, see the Cisco Nexus 9000 Series NX-OS VXLAN Configuration Guide, Release 10.1(x).

• When you upgrade a Cisco Nexus 9000 device to Cisco NX-OS Release 10.1(x), if a QSFP port is configured with the manual breakout command and is using a QSA, the configuration of the interface Ethernet 1/50/1 is no longer supported and must be removed. To restore the configuration, you must manually configure the interface Ethernet 1/50 on the device.

• When redistributing static routes, Cisco NX-OS requires the default-information originate command to successfully redistribute the default static route starting in 7.0(3)I7(6).

• To perform an EPLD upgrade after an ISSU upgrade from Cisco NX-OS Release 7.x to Cisco NX-OS Release 9.3(x), before starting the EPLD upgrade, add the copy run start command.

• When upgrading from Cisco NX-OS Release 9.2(4) or earlier releases to Cisco NX-OS Release 9.3(4) or later, running configuration contains extra TCAM configuration lines. You can ignore these extra lines as they do not have an effect on the upgrade and  configuration.

• When performing an ISSU from Cisco NX-OS Release 9.3(1) or 9.3(2) to Cisco NX-OS Release 9.3(3) or later, ensure that the features with user-defined ports, such as , are within the prescribed port range. If the port range is incorrect, follow the syslog message recommendation. For more information about the port range, see Cisco Nexus 9000 Series NX-OS IP SLAs Configuration Guide, Release 10.1(x).

• When upgrading from Cisco NX-OS Release 9.2(2) or earlier releases to Cisco NX-OS Release 10.1(x), you need to make sure that ingress RACL TCAM region is not more than 50% full. Otherwise, the atomic update feature will be enabled after the upgrade and interfaces with RACLs that exceed 50% of TCAM allocation will remain down.

• Beginning with Cisco NX-OS Release 10.1(1), ISSU is supported on FC/FCoE switch mode on Cisco Nexus 93360YC-FX2. For more information about the FC/FCoE switch mode and supported hardware, see Cisco Nexus 9000 Series NX-OS SAN Switching Configuration Guide, Release 10.1(x).

• Beginning with Cisco NX-OS Release 10.1(1), Enhanced ISSU is supported on FC/FCoE switch mode for Cisco Nexus 93180YC-FX and 93360YC-FX2 switches. For more information about the FC/FCoE switch mode and supported hardware, see Cisco Nexus 9000 Series NX-OS SAN Switching Configuration Guide, Release 10.1(x).

• Beginning with Cisco NX-OS Release 10.1(1), Enhanced ISSU is supported on FC/FCoE NPV mode for Cisco Nexus 93180YC-FX and 93360YC-FX2 switches. For more information about the FC/FCoE NPV mode and supported hardware, see Cisco Nexus 9000 Series NX-OS FC-NPV and FCoE NPV Configuration Guide, Release 10.1(x).

• Software image compaction is only supported on Cisco Nexus 9300-series platform switches.

• The compressed image of Cisco Nexus 3000-series is hardware dependent and can only be used on the same device that it got compressed or downloaded from CCO. Do not use the Nexus 3000-series compressed image on Nexus 9000-series

• The following limitation applies to software upgrades from 7.0(3)I5 to 10.1(x) or 9.2(3) to 10.1(x): If you have the same NetFlow configuration in both VLAN and SVI, you must remove the NetFlow flow monitor from the VLAN configuration prior to the upgrade. Once upgraded, reconfigure NetFlow by creating a new flow monitor and adding it to the VLAN configuration. Failure to perform these steps results in error messages and the inability to modify the VLAN NetFlow configuration in the upgraded
  software.

• When upgrading from Cisco NX-OS Releases 7.0(3)I4(8), 7.0(3)I5(3), and 7.0(3)I6(1) to Cisco NX-OS Release 10.1(x) results in a disruptive upgrade. If syncing images to standby SUP failed during the disruptive upgrade from Cisco NX-OS Releases 7.0(3)I4(8), 7.0(3)I5(3,) or 7.0(3)I6(1) to 10.1(x), you should manually copy the image to the standby SUP and perform the disruptive upgrade.

• When upgrading directly to Cisco NX-OS Release 10.1(x) from any release prior to 7.0(x), the upgrade will be disruptive. For a non-disruptive upgrade, an intermediate upgrade to Cisco NX-OS Release 9.x is required. We recommend upgrading to the latest release of Cisco NX-OS Release 9.3(x) as an intermediate hop for the upgrade. For information about the supported upgrade paths, see the ISSU Support Matrix.

• When upgrading from Cisco NX-OS Release 7.0(3)I6(1) or 7.0(3)I7(1) to Cisco NX-OS Release 10.1(x), if the Cisco Nexus 9000 Series switches are running vPC and they are connected to an IOS-based switch via Layer 2 vPC, there is a likelihood that the Layer 2 port channel on the IOS side will become error disabled. The workaround is to disable the spanning-tree etherchannel guard misconfig command on the IOS switch before starting the upgrade process. Once both the Cisco Nexus 9000 Series switches are upgraded, you can re-enable the command.

• If you are upgrading from Cisco NX-OS Release 7.0(3)I5(2) to Cisco NX-OS Release 10.1(x) by using the install all command, BIOS will not be upgraded due to CSCve24965. When the upgrade to Cisco NX-OS Release 10.1(x) is complete, use the install all command again to complete the BIOS upgrade, if applicable.

• An upgrade that is performed via the install all command for Cisco NX-OS Release 7.0(3)I2(2b) to Release 10.1(x) might result in the VLANs being unable to be added to the existing FEX HIF trunk ports. To recover from this, the following steps should be performed after all FEXs have come online and the HIFs are operationally up:
  1. Enter the copy run bootflash:fex_config_restore.cfg command at the prompt.
  2. Enter the copy bootflash:fex_config_restore.cfg running-config echo- commands command at the prompt.

• In Cisco NX-OS Release 7.0(3)I6(1) and earlier, performing an ASCII replay or running the copy file run command on a FEX HIF configuration requires manually reapplying the FEX configuration after the FEX comes back up.

• When upgrading to Cisco NX-OS Release 10.1(x) from 7.0(3)I2(x) or before and running EVPN VXLAN configuration, an intermediate upgrade to 7.0(3)I4(x) or 7.0(3)I5(x) or 7.0(3)I6(x) is required.

• Before enabling the FHS on the interface, we recommend that you carve the ifacl TCAM region on Cisco Nexus 9300 and 9500 platform switches. If you carved the ifacl TCAM region in a previous release, you must reload the system after upgrading to Cisco NX-OS Release 10.1(x). Uploading the system creates the required match qualifiers for the FHS TCAM region, ifacl.

• Before enabling the FHS, we recommend that you carve the ing-redirect TCAM region on Cisco Nexus 9200 and 9300-EX platform switches. If you carved the ing-redirect TCAM region in a previous release, you must reload the system after upgrading to Cisco NX-OS Release 10.1(x). Uploading the system creates the required match qualifiers for the FHS TCAM region, ing-redirect.

• Upgrading from Cisco NX-OS Release 9.3(1), 9.3(2) or 9.3(3) to a higher release, with Embedded Event Manager (EEM) configurations that are saved to the running configuration, may cause a DME error to be presented. The error is in the output of the show consistency-checker dme running-config enhanced command, specifically, the event manager commands. If this error occurs, delete all EEM applet configurations after completing the ISSU, then reapply the EEM configurations.

• For any prior release version upgrading to Cisco NX-OS Release 9.3(5) using ISSU, if the following logging level commands are configured, they are missing in the upgraded version and must be reconfigured:
  • logging level evmc value
  • logging level mvsh value
  • logging level fs-daemon value

• For any prior release version upgrading to Cisco NX-OS Release 9.3(6) using ISSU, if the following logging level commands are configured, they are missing in the upgraded version and must be reconfigured:
  • logging level evmc value
  • logging level mvsh value

• An error occurs when you try to perform an ISSU if you changed the reserved VLAN without entering the copy running-config save-config and reload commands.

• The install all command is the recommended method for software upgrades and downgrades because it performs configuration compatibility checks and BIOS upgrades automatically. In contrast, changing the boot variables and reloading the device bypasses these checks and the BIOS upgrade and therefore it is not recommended.

• Upgrading from Cisco NX-OS Release 7.0(3)I1(2), Release 7.0(3)I1(3), or Release 7.0(3)I1(3a) requires installing a patch for Cisco Nexus 9500 platform switches only. For more information on the upgrade patch, see Patch Upgrade Instructions.

• An ISSU can be performed only from a Cisco NX-OS Release 7.0(3)I4(1) to a later image.

• While performing an ISSU, VRRP and VRRPv3 displays the following messages:
  • If VRRPv3 is enabled:• If VRRP is enabled:

• Guest Shell is disabled during an ISSU and reactivated after the upgrade. Any application running in the Guest Shell is affected.

• If you have ITD probes configured, you must disable the ITD service (using the shutdown command) before upgrading to Cisco NX-OS Release 10.1(x). After the upgrade, enter the feature sla sender command to enable IP SLA for ITD probes and then the no shutdown command to re-enable the ITD service. (If you upgrade without shutting down the service, you can enter the feature sla sender command after the upgrade.)

• Schedule the upgrade when your network is stable and steady.

• Avoid any power interruption, which could corrupt the software image, during the installation procedure.

• On devices with dual supervisor modules, both supervisor modules must have connections on the console ports to maintain connectivity when switchovers occur during a software upgrade. See the Hardware Installation Guide for your specific chassis.

• Perform the installation on the active supervisor module, not the standby supervisor module.

• The install all command is the recommended method for software upgrades because it performs configuration compatibility checks and BIOS upgrades automatically. In contrast, changing the boot variables and reloading the device bypasses these checks and the BIOS upgrade and therefore is not recommended.
  Note
  For Cisco Nexus 9500 platform switches with -R line cards, you must save the configuration and reload the device to upgrade from Cisco NX-OS Release 7.0(3)F3(5) to 9.3(1). To upgrade from Cisco NX-OS Release 9.2(2) or 9.2(3), we recommend that you use the install all command.

• You can detect an incomplete or corrupt NX-OS software image prior to performing an upgrade by verifying the MD5, SHA256 or SHA512 checksum of the software image. To verify the MD5 checksum of the software image, run the show file bootflash:md5sum command and compare the resulting value to the published MD5 checksum for the software image on Cisco’s Software Download website. To verify the SHA512 checksum of the software image, run the show file
  bootflash:sha512sum command and compare the resulting value to the published SHA512 checksum for the software image on Cisco’s Software Download website.

• When upgrading from Cisco Nexus 94xx, 95xx, and 96xx line cards to Cisco Nexus 9732C-EX line cards and their fabric modules, upgrade the Cisco NX-OS software before inserting the line cards and fabric modules. Failure to do so can cause a diagnostic failure on the line card and no TCAM space to be allocated. You must use the write_erase command followed by the reload command.

• If you upgrade from a Cisco NX-OS release that supports the CoPP feature to a Cisco NX-OS release that supports the CoPP feature with additional classes for new protocols, you must either run the setup utility using the setup command or use the copp profile command for the new CoPP classes to be available. For more information on these commands, see the “Configuring Control Plane Policing” chapter in the Cisco Nexus 9000 Series NX-OS Security Configuration Guide, Release 10.1(x) .

• For secure POAP, ensure that DHCP snooping is enabled and set firewall rules to block unintended or malicious DHCP servers. For more information on POAP, see the Cisco Nexus 9000 Series Fundamentals Configuration Guide, Release 10.1(x).

• When you upgrade from an earlier release to a Cisco NX-OS release that supports switch profiles, you have the option to move some of the running-configuration commands to a switch profile. For more information, see the Cisco Nexus 9000 Series NX-OS System Management Configuration Guide, Release 10.1(x).

• By default, the software upgrade process is disruptive.

• OpenFlow and LACP fast timer rate configurations are not supported for ISSU.

• Guest Shell is disabled during an ISSU and reactivated after the upgrade.

• ISSU supports only default hold timers for BGP peers.

• During an ISSU on a Cisco Nexus 9300 Series switch, all First-Hop Redundancy Protocols (FHRPs) will cause the other peer to become active if the node undergoing the ISSU is active.

• Make sure that both vPC peers are in the same mode (regular mode or enhanced mode) before performing a nondisruptive upgrade.
  Note
  vPC peering between an enhanced ISSU mode (boot mode lxc) configured switch and a non-enhanced ISSU mode switch is not supported.

• During an ISSU, the software reload process on the first vPC device locks its vPC peer device by using CFS messaging over the vPC communications channel. Only one device at a time is upgraded. When the first device completes its upgrade, it unlocks its peer device. The second device then performs the upgrade process, locking the first device as it does so. During the upgrade, the two vPC devices temporarily run different releases of Cisco NX-OS; however, the system functions correctly because of its backward compatibility support.

• ISSU is not supported when onePK is enabled. You can run the show feature | include onep command to verify that this feature is disabled before performing an ISSU or enhanced ISSU.
  • In general, ISSUs are supported for the following:
  • From a major release to any associated maintenance release.
  • From the last two maintenance releases to the next two major releases.
  • From an earlier maintenance release to the next two major releases.

Note
For a list of specific releases from which you can perform a disruptive upgrade or a nondisruptive ISSU, see the Cisco Nexus 9000 Series NX-OS Release Notes for your particular release.

• After performing ISSU on Cisco Nexus 9300 platform switches and the Cisco Nexus 3164Q switches, you may see the MTS_OPC_CLISH message on the vPC peers. MTS_OPC_CLISH is the last MTS code that is sent from the back-end component to the VSH to specify the end of the show command output. If the user executes a show command that produces more output and keeps the session on for more than 3 minutes, the following warning message may be displayed on the console. As a workaround, you can set the terminal length as 0 using the terminal length 0 command or the show | no-more option.

There is no functionality impact or traffic loss due to this issue. All the MTS messages are drained once the show command displays the complete output, the user enters CTRL+c, or the session gets closed.

• Occasionally, while the switch is operationally Up and running, the Device not found logs are displayed on the console. This issue is observed because the switch attempts to find an older ASIC version and the error messages for the PCI probe failure are enabled in the code. There is no functionality impact or traffic loss due to this issue.
• ISSU is not supported if EPLD is not at Cisco NX-OS Release 7.0(3)I3(1) or later.
• ISSU supports EPLD image upgrades using install all nxos epld command, during disruptive system (NX-OS) upgrade.
• A simplified NX-OS numbering format is used for platforms that are supported in Cisco NX-OS 10.1(x) releases. In order to support a software upgrade from releases prior to Cisco NX-OS Release 7.0(3)I7(4) that have the old release format, an installer feature supplies an I9(x) label as a suffix to the actual release during the install all operation. This label is printed as part of the image during the install operation from any release prior to Cisco NX-OS Release 7.0(3)I7(4) to 10.1(x), and it can be ignored. See the following example.

• Beginning with Cisco NX-OS Release 9.3(5), standard, nondisruptive ISSU, on switches that are configured with uRPF, is supported on the following:
  • Cisco Nexus 9300-EX platform switches
  • Cisco Nexus 9300-FX/FX2 platform switches
  • Cisco Nexus 9300-GX platform switches

Note
Prior to Cisco NX-OS Release 9.3(5), if any of the above switches were configured with uRPF, standard, nondisruptive ISSU was not supported.

• ISSU is blocked if boot poap enable is configured.
• On performing a non-disruptive ISSU from Cisco NX-OS Release 7.0(3)I6(1) to any higher version, a traffic loss might occur based on the number of VLANs configured. To avoid traffic loss, it is recommended to increase the routing protocol’s graceful restart timer to higher value. The recommended value of the graceful restart timer is 600 seconds. You can further increase or decrease this value based on the scale of the configuration.
• Beginning with Cisco NX-OS Release 10.1(1), Fs_daemon does not support snmpwalk on devices with more than 5000 files. When performing snmpwalk on a device with more than 5000 files, the error resourceUnavailable (This is likely a out-of-memory failure within the agent) is an expected behaviour.
• Beginning with Cisco NX-OS Release 10.1(2), CoPP is supported on N9K-X9624D-R2 and N9K-C9508-FM-R2 platform switches.
• Beginning with Cisco NX-OS Release 10.1(2), RACL is supported on N9K-X9624D-R2 and N9K-C9508-FM-R2 platform switches.
• ISSU is blocked when the delay config is present in track list Boolean/weight.
• If there is a VRF scale, for a non-disruptive ISSU under each VRF, you must configure graceful restart timer to 300 seconds.

ISSU Platform Support

The following tables identify the platforms supporting standard and enhanced ISSU, and the release when the support was introduced.
Note
Enhanced ISSU: Enhanced ISSU to Cisco NX-OS Release 10.1(x) is not supported as there are kernel updates that cannot take effect without reloading the underlying kernel. The system will prompt the following message:
Host kernel is not compatible with target image. Full ISSU will be performed and control plane will be impacted.
In effect, system will perform nondisruptive ISSU instead of enhanced ISSU.
ISSU for Cisco Nexus 9200 Platform Switches

ISSU Type| Release/Supported Platforms| Features Not Supported with Non-disruptive ISSU
---|---|---
Standard| Beginning with Cisco NX-OS Release 7.0(3)I6(1): Cisco Nexus 92300YC
Beginning with Cisco NX-OS Release 9.3(3):
Cisco Nexus 92348GC-X| Both ISSU types are disruptive for Cisco Nexus 9200 platform switches configured with the following features:
•  Segment routing
•  Tetration
Enhanced| Cisco Nexus 92300YC

ISSU for Cisco Nexus 9300 Platform Switches

ISSU Type| Release/Supported Platforms| Features Not Supported with Non-disruptive ISSU
---|---|---
Standard| Beginning with Cisco NX-OS Release 9.3(3): Cisco Nexus 9332C
Cisco Nexus 9364C
Note ISSU on Cisco Nexus 9300 platform switches is supported when the switch is the spanning tree root. You can use the show spanning-tree issu- impact command to verify if the switch meets this criteria.| Both ISSU types are disruptive for Cisco Nexus 9300 platform switches configured with the following features:
•  Dual-homed FEX
•  Segment routing
•  VXLAN
Enhanced| Beginning with Cisco NX-OS Release 9.3(5): Cisco Nexus 9332C
Cisco Nexus 9364C
Note ISSU on Cisco Nexus 9300 platform switches is supported when the switch is the spanning tree root. You can use the show spanning-tree issu- impact command to verify if the switch meets this criteria.

ISSU for Cisco Nexus 9300-EX Platform Switches

ISSU Type| Release/Supported Platforms| Features Not Supported with Non-disruptive ISSU
---|---|---
Standard| Beginning with Cisco NX-OS Release 7.0(3)I6(1): Cisco Nexus 93108TC- EX Cisco Nexus 93180YC-EX| Both ISSU types are disruptive for Cisco Nexus 9300-EX platform switches configured with the following features:
•  Segment routing
•  Tetration
Enhanced| Beginning with Cisco NX-OS Release 7.0(3)I7(3): Cisco Nexus 93108TC- EX Cisco Nexus 93180YC-EX

ISSU for Cisco Nexus 9300-FX Platform Switches

ISSU Type| Release/Supported Platforms| Features Not Supported with Non-disruptive ISSU
---|---|---
Standard| Cisco NX-OS Release 9.3(1) and 9.3(2): None Beginning with Cisco NX- OS Release 9.3(3): Cisco Nexus 9336C-FX2
Cisco Nexus 93240YC-FX2 Cisco Nexus 93240YC-FX2Z Cisco Nexus 9348GC-FXP Cisco Nexus 93108TC-FX
Cisco Nexus 93180YC-FX| Standard ISSU is disruptive for Cisco Nexus 9300-FX platform switches configured with the following features:
•  Segment Routing
•  TRM Feature
ISSU Type| Release/Supported Platforms| Features Not Supported with Non-disruptive ISSU
---|---|---
Enhanced| Cisco NX-OS Release 9.3(1), 9.3(2), and 9.3(3): None
Beginning with Cisco NX-OS Release 9.3(5):
Cisco Nexus 9336C-FX2
Cisco Nexus 93240YC-FX2
Cisco Nexus 93216TC-FX2
Cisco Nexus 93360YC-FX2
Cisco Nexus 93240YC-FX2Z
Cisco Nexus 9348GC-FXP
Cisco Nexus 93108TC-FX
Cisco Nexus 93180YC-FX
Beginning with Cisco NX-OS Release 10.1(1), Enhanced ISSU is supported on the following platforms with FC/FCoE features:
Cisco Nexus 93360YC-FX2
Cisco Nexus 93180YC-FX| Enhanced ISSU is disruptive for Cisco Nexus 9300-FX platform switches configured with the following features:
•  Segment Routing
•  TRM Feature
Note In Cisco NX-OS Releases 9.3(x),
Enhnaced ISSU on Cisco Nexus 93360YC-FX2 and
Cisco Nexus 93180YC-FX with
FC/FCoE features will be disruptive.

ISSU for Cisco Nexus 9300-GX Platform Switches

ISSU Type| Release/Supported Platforms| Features Not Supported with Non-disruptive ISSU
---|---|---
Standard| Beginning with Cisco NX-OS Release 10.1(1): Cisco Nexus 9364C-GX
Cisco Nexus 9316D-GX
Cisco Nexus 93600CD-GX| •  TRM Feature
•  Segment Routing
Enhanced| Beginning with Cisco NX-OS Release 10.1(1): Cisco Nexus 9364C-GX
Cisco Nexus 9316D-GX
Cisco Nexus 93600CD-GX| •  TRM Feature
•  Segment Routing

Cisco NX-OS Software Downgrade Guidelines

Before attempting to downgrade to an earlier software release, follow these guidelines:

• The only supported method of downgrading a Cisco Nexus 9000 Series switch is to utilize the install all command. Changing the boot variables, saving the configuration, and reloading the switch is not a supported method to downgrade the switch.

Disable the Guest Shell if you need to downgrade from Cisco NX-OS Release 9.3(x) to an earlier release.

• Performing an ISSU downgrade from Cisco NX-OS Release 9.3(x) to Release 7.0(3)I4(1) with an FCoE (Fiber Channel over Ethernet) NPV (N-port Virtualization) configuration causes the port channel to crash with a core file:
  [################ ] 38%2016 Apr 18 20:52:35 n93-ns1 %$ VDC-1 %$ %SYSMGR- 2SERVICE_CRASHED:
  Service “port-channel” (PID 14976) hasn’t caught signal 11 (core will be saved)

• ISSU (non-disruptive) downgrade is not supported

• When downgrading from the Cisco NX-OS Release 9.3(x) to earlier releases, any ACL with the statistics per-entry command enabled and applied as RACL needs the statistics per-entry command removed from the running configuration before downgrading. Otherwise, the interfaces on which this ACL is applied as a RACL will be error disabled after the downgrade.

• Prior to downgrading a Cisco Nexus 9500-series switch, with -FX or -FX+EX line cards, from Cisco NX-OS Release 10.1(x) to earlier releases (9.2(x) or 7.x), the TCAM region that applies to NetFlow (ing-netflow) should be carved to zero (0) using the following command:  hardware access-list tcam region ing-netflow 0
  The configuration change is required because the default ing-netflow TCAM region in 9.3(1) and onwards is 512 while the default in 9.2(x) and earlier is

• When downgrading from the Cisco NX-OS Release 10.1(x) to a release prior to 9.3(x), make sure that the ACL TCAM usage for ingress features does exceed the allocated TCAM space in the absence of the label sharing feature. Label sharing is a new feature in Cisco NX-OS Release 9.3(x). Otherwise, interfaces with RACLs that could not fit in the TCAM will be disabled after the downgrade.

• Software downgrades should be performed using the install all command. Changing the boot variables, saving the configuration, and reloading the switch is not a supported method to downgrade the switch.

• The following limitation applies to Cisco Nexus platform switches that support Trust Anchor Module (TAM):
  The TACACS global key cannot be restored when downgrading from Cisco NX-OS Release 9.3(3) and higher to any earlier version. TAM was updated to version-7 in 9.3(3), but earlier NX-OS versions used TAM version-3.

• iCAM must be disabled before downgrading from Release 9.2(x) or Release 9.3(x) → 7.0(3)I7(1). Only Release 9.3(1) → Release 9.2(4) can be performed if iCAM is enabled.

• Beginning with Cisco NX-OS Release 9.3(3), new configuration commands exist for SRAPP (with sub-mode options for MPLS and SRTE). The SRAPP configuration on the switch running release 9.3(3) (or later) will not be present if the switch is downgraded to an earlier release.

• On devices with dual supervisor modules, both supervisor modules must have connections on the console ports to maintain connectivity when switchovers occur during a software downgrade. See the Hardware Installation Guide for your specific chassis.

• Cisco NX-OS automatically installs and enables the guest shell by default. However, if the device is reloaded with a Cisco NX-OS image that does not provide guest shell support, the existing guest shell is automatically removed and a %VMAN-2-INVALID_PACKAGE message is issued. As a best practice, remove the guest shell with the guestshell destroy command before downgrading to an earlier Cisco NX-OS image.

• You must delete the switch profile (if configured) when downgrading from a Cisco NX-OS release that supports switch profiles to a release that does not. For more information, see the Cisco Nexus 9000 Series NX-OS System Management Configuration Guide, Release 10.1(x).

• Software downgrades are disruptive. In-service software downgrades (ISSDs), also known as nondisruptive downgrades, are not supported.

Upgrade Paths

For ISSU compatibility for all release and information about the upgrade paths, see the Cisco NX-OS ISSU Support Matrix.

Upgrade Patch Instructions

On Cisco Nexus 9500 series switches only, a software upgrade from Cisco NX-OS Release 7.0(3)I1(2), 7.0(3)I1(3), or 7.0(3)I1(3a) to any other Cisco NX-OS release requires installing two patches prior to upgrading using the install all command. These patches are available for each respective release and can be downloaded using the links below.
Caution
Failing to follow this procedure could require console access in order to recover the switch after the upgrade.
Note
These patches are only for upgrading. After the upgrade, the patch is automatically removed. If you decide not to upgrade after installing the patches, do not deactivate it. Deactivating the patch may cause a bios_daemon crash.
Cisco NX-OS Release 7.0(3)I1(2) Upgrade Patch
Cisco NX-OS Release 7.0(3)I1(3) Upgrade Patch
Cisco NX-OS Release 7.0(3)I1(3a) Upgrade Patch
To install these patches prior to upgrading using the install all command, follow the instructions shown below. An example is demonstrated below with an NX-OS software patch and upgrade from 7.0(3)I1(2) to 7.0(3)I7(1):

1. Add both patches with the install add bootflash:{patch-file.bin} command.
2. Activate both patches with the install activate {patch-file.bin} command.
3. Commit both patches with the install commit {patch-file.bin} command.
4. Proceed with an NX-OS software upgrade to the desired target release with the install all command.

|
---|---
|
|
|

Configuring Enhanced ISSU

You can enable or disable enhanced (LXC) ISSU.

Note

• Enhanced ISSU to Cisco NX-OS Release 10.1(x) is not supported as there are kernel updates that cannot take effect without reloading the underlying kernel. The system will prompt the following message:
  Host kernel is not compatible with target image. Full ISSU will be performed and control plane will be impacted.
  In effect, system will perform nondisruptive ISSU instead of enhanced ISSU.

Before you begin
Before you enable the LXC mode, ensure that the installed licenses do not include the 27000 string in the license file.
SUMMARY STEPS

1. configure terminal
2.  [no] boot mode lxc
3.  (Optional) show boot mode
4.  copy running-config startup-config
5.  reload

DETAILED STEPS

| Command or Action| Purpose
---|---|---
Step 1| configure terminal Example:
switch# configure terminal switch(config#)| Enters global configuration mode.
Step 2| [ no ] boot mode lxc Example:
switch(config)# boot mode lxc Using LXC boot mode
Example:
switch(config)# no boot mode lxc Using normal native boot mode| Enables or disables enhanced (LXC) ISSU.
Note In order to perform a nondisruptive enhanced ISSU, you must first boot the switch in LXC mode.
Step 3| (Optional) show boot mode Example:
switch(config)# show boot mode LXC boot mode is enabled
Example:
switch(config)# show boot mode LXC boot mode is disabled| Shows whether enhanced (LXC) ISSU is enabled or disabled.
Step 4| copy running-config startup-config Example:
switch(config)# copy running-config startup-config| Saves the running configuration to the startup configuration.
Step 5| reload Example:
switch(config)# reload
This command will reboot the system. (y/n)? [n] Y loader>| Reloads the device. When prompted, press Y to confirm the reboot.

What to do next
Follow the instructions in Upgrading the Cisco NX-OS Software section. Make sure to choose the non-disruptive option if you want to perform an enhanced or regular ISSU.

Upgrading the Cisco NX-OS Software

Use this procedure to upgrade to a Cisco NX-OS 10.1(x) release.
Note
Beginning with Cisco NX-OS Release 10.1(1), the Cisco Nexus -GX series platforms use the 64-bit Cisco NX-OS image file, which has the image filename that begins with “nxos64” (for example, nxos64.10.1.1.bin). The 64-bit software image, which supports software scalability, is available for the Cisco Nexus C9316D-GX, C93600CD-GX, C9364C-GX switches. The non-GX series platforms use the 32-bit Cisco NX-OS image file, which has the image filename that begins with “nxos” (for example, nxos.10.1.1.bin).
Note
For Cisco Nexus 9500 platform switches with -R line cards, you must save the configuration and reload the device to upgrade from Cisco NX-OS Release 7.0(3)F3(5) to 10.1(1). To upgrade from Cisco NX-OS Release 9.2(2) or later, we recommend that you use the install all command.
Note
If an error message appears during the upgrade, the upgrade will fail because of the reason indicated. See the Cisco Nexus 9000 Series NX-OS Troubleshooting Guide, Release 10.1(x) for a list of possible causes and solutions.
Before you begin
Before performing a nondisruptive ISSU to Cisco NX-OS Release 10.1(1), you must configure the BGP graceful restart timer to 180 seconds for Cisco Nexus 3132Q-V platform switches.
SUMMARY STEPS

1. Read the release notes for the software image file for any exceptions to this upgrade procedure. See the Cisco Nexus 9000 Series NX-OS Release Notes.
2. Log in to the device on the console port connection.
3. Ensure that the required space is available for the image file to be copied.
4. If you need more space on the active supervisor module, delete unnecessary files to make space available.
5. Verify that there is space available on the standby supervisor module.
6. If you need more space on the standby supervisor module, delete any unnecessary files to make space available.
7. Log in to Cisco.com, choose the software image file for your device from the following URL, and download it to a file server: http://software.cisco.com/download/navigator.html.
8. Copy the software image to the active supervisor module using a transfer protocol. You can use FTP, TFTP, SCP, or SFTP.
9. You can detect an incomplete or corrupt NX-OS software image prior to performing an upgrade by verifying the MD5, SHA256 or SHA512 checksum of the software image. To verify the MD5 checksum of the software image, run the show file bootflash:md5sum command and compare the resulting value to the published MD5 checksum for the software image on Cisco’s Software Download website. To verify the SHA512 checksum of the software image, run the show file bootflash:sha512sum command and compare the resulting value to the published SHA512 checksum for the software image on Cisco’s Software Download website.
10. Check the impact of upgrading the software before actually performing the upgrade.
11.  Save the running configuration to the startup configuration.
12. If required, upgrade the EPLD image using the install all nxos epld command.
13. Upgrade the Cisco NX-OS software using the install all nxos bootflash:filename [no-reload | non-disruptive | non-interruptive | serial] command.
14. (Optional) Display the entire upgrade process.
15.  (Optional) Log in and verify that the device is running the required software version.
16.  (Optional) If necessary, install any licenses to ensure that the required features are available on the device. See the Cisco NX-OS Licensing Guide.

DETAILED STEPS
Step 1 Read the release notes for the software image file for any exceptions to this upgrade procedure. See the Cisco Nexus 9000 Series NX-OS Release Notes.
Step 2 Log in to the device on the console port connection.
Step 3 Ensure that the required space is available for the image file to be copied.

Step 4
If you need more space on the active supervisor module, delete unnecessary files to make space available.
switch# delete bootflash:nxos.9.2.1.bin
Step 5
Verify that there is space available on the standby supervisor module.

Step 6
If you need more space on the standby supervisor module, delete any unnecessary files to make space available.
switch# delete bootflash://sup-standby/nxos.9.2.1.bin
Step 7
Log in to Cisco.com, choose the software image file for your device from the following URL, and download it to a file server: http://software.cisco.com/download/navigator.html.
Step 8
Copy the software image to the active supervisor module using a transfer protocol. You can use FTP, TFTP, SCP, or SFTP. switch# copy scp://user@scpserver.cisco.com//download/nxos.10.1.1.bin bootflash:nxos.10.1.1.bin
For software images requiring compaction, you must use SCP, HTTP, or HTTPS as the source and bootflash or USB as the destination. The following example uses SCP and bootflash:
The compact keyword compacts the NX-OS image before copying the file to the supervisor module.
Software image compaction is only supported on SCP, HTTP, or HTTPS. If you attempt compaction with any other protocol, the system returns the following error:
Note
Compact option is allowed only with source as scp/http/https and destination as bootflash or usb
Note
Compacted images are not supported with LXC boot mode.
Note
Software image compaction is only supported on Cisco Nexus 9300-series platform switches.
Step 9
You can detect an incomplete or corrupt NX-OS software image prior to performing an upgrade by verifying the MD5, SHA256 or SHA512 checksum of the software image. To verify the MD5 checksum of the software image, run the show file bootflash:md5sum command and compare the resulting value to the published MD5 checksum for the software image on Cisco’s Software Download website. To verify the SHA512 checksum of the software image, run the show file bootflash:sha512sum command and compare the resulting value to the published SHA512 checksum for the software image on Cisco’s Software Download website.

Step 10
Check the impact of upgrading the software before actually performing the upgrade.
switch# # show install all impact nxos bootflash:nxos.10.1.1.bin
During the compatibility check, the following ISSU-related messages may appear in the Reason field:

to upgrade does not support ISSU.
Default upgrade is not hitless| By default, the software upgrade process is disruptive. You must configure the non-disruptive option to perform an ISSU.

Step 11
Save the running configuration to the startup configuration.
switch# copy running-config startup-config
Step 12
If required, upgrade the EPLD image using the install all nxos epld command.
The following is an example output of the install all nxos epld