Mellanox Technologies SX67X0 SwitchX 2 1U Switch Systems User Manual

SX67X0 SwitchX 2 1U Switch Systems
User Manual Mellanox SwitchX®-2
1U Switch Systems
Model: SX67X0
Rev 1.9

NOTE:
THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT (S)”) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER’S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT, INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT , STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT (S) AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Revision History
Table 1 – Revision History Table

•  “Unit Identification LED”
February 2017| 1.7| Updated:
• “Data Interfaces”
•  “Specifications”
• “RJ45 to DB9 Harness Pinout” Added:
• “BBU LED Assignments”
October 2015| 1.5| • Edited “Installation”
August 2015| 1.4| Minor fixes
May 2015| 1.3| Added Hebrew safety warnings
Added BBU safety warnings in all languages
April 2015| 1.2| Added “Telescopic Rail Kit”
January 2015| 1.1| Minor formatting edits
January 2015| 1.0| Initial release of the first edition

About this Manual
This manual describes the installation and basic use of the Mellanox InfiniBand systems.
Intended Audience
This manual is intended for IT managers and system administrators.
References
Table 2 – References

configuration and man- agreement of the MLNX-OS software. See http://www.mellanox.com/page/ _mlnxos.

Conventions
The following icons are used throughout this document to indicate information that is important to the user.
This icon makes recommendations to the user.
This icon indicates information that is helpful to the user.
This icon indicates a situation that can potentially cause damage to hardware or software.
This icon indicates a situation that can potentially cause personal injury.
This icon indicates a situation that can potentially cause personal injury.

Introduction to Mellanox SX67X0 Systems

1.1 Overview
The SX67X0 systems provide the highest performing fabric solution in a 1U form factor by delivering up to 4Tb/s of non-blocking bandwidth with 200ns port-to- port latency.
These systems are the industry’s most cost-effective building blocks for embedded systems and storage with a need for low port density systems. Whether looking at price-to-performance or energy-to-performance, these systems offer superior performance, power and space, reducing capital and operating expenses and providing the best return-on-investment. The systems are an ideal choice for smaller departmental or back-end clustering uses with high-performance needs, such as storage, data base and GPGPU clusters.
Powerful servers combined with high-performance storage and applications that use increasingly complex computations are causing data bandwidth requirements to spiral upward. As servers are deployed with next generation processors, High-Performance Computing (HPC) environments and Enterprise Data Centers (EDC) need every last bit of bandwidth delivered with Mellanox’s FDR InfiniBand systems.
Built with Mellanox’s sixth generation SwitchX®-2 InfiniBand FDR 56Gb/s system device, these standalone systems are an ideal choice for top-of-rack leaf connectivity or for building small to extremely large sized clusters.
These systems enable efficient computing with features such as static routing, adaptive routing, and advanced congestion management. These features ensure the maximum effective fabric bandwidth by eliminating congestion. The managed systems comes with an onboard subnet manager, enabling simple, out-of-the-box fabric bring-up for up to 648 nodes. MLNX-OS® software delivers complete chassis management of firmware, power supplies, fans, ports and other interfaces.
Mellanox’s edge systems can also be coupled with Mellanox’s Unified Fabric Manager (UFM®) software for managing scale-out InfiniBand computing environments. UFM enables data center operators to efficiently provision, monitor and operate the modern data center fabric. UFM boosts application performance and ensures that the fabric is up and running at all times.
InfiniBand systems come as internally or externally managed. Internally managed systems come with a CPU that runs the management software (MLNX-OS®) and management ports which are used to transfer management traffic into the system. Externally managed systems come without the CPU and management ports and are managed using firmware tools.
Mellanox’s InfiniBand to Ethernet gateway, built with Mellanox’s SwitchX®-2 based systems, provides the most cost-effective, high-performance solution for data center unified connectivity solutions. Mellanox’s gateways enable data centers to operate at up to 56Gb/s network speeds while seamlessly connecting to 1, 10 and 40GbE networks with low latency (400ns). Existing LAN infrastructures and management practices can be preserved, easing deployment and providing significant return-on-investment.

1.2 Speed and Switching
Table 3 describes maximum throughput and interface speed per system model.
Table 3 – Speed and Switching Capabilities

System Model| 10GbE SFP+ Interfaces| 40/56GbE QSFP+ Interfaces| Max Throughput
---|---|---|---
SX6710| N/A| 36| 4.032Tb/s
SX6720*| N/A| 36| 4.032Tb/s

*The system can support different interfaces and speed rates using QSFP to SFP adapters or hybrid cables. For further information, see “Breakout Cables and Adapters”.

1.3 Management Interfaces, PSUs and Fans
Table 4 lists the various management interfaces, PSUs and fans per system model.
Table 4 – Management Interfaces, PSUs and Fans

1.4 Features
For a full feature list, please refer to the system’s product brief. Go to http://www.mellanox.com.
In the main menu, click on Products–> InfiniBand/VPI Switch Systems, and select the desired product page.
1.5 Certifications
The list of certifications (such as EMC, Safety and others) per system for different regions of the world is located on the Mellanox website at: http://www.mellanox.com/page/environmental_compliance
1.6 Ordering Information
The following table lists ordering information for the available systems. Please pay attention to the airflow direction when ordering your system. For more details, see “Air Flow”.
Table 5 – Ordering Part Numbers (OPNs)

ports, 2 Power Supplies (AC), x86 dual core, short depth, rear to front air- flow, Rail Kit, RoHS6
MSX6710-FS2F2| SwitchX®-2 based FDR InfiniBand 1U Switch, 36 QSFP+ ports, 2 Power Supplies (AC), x86 dual core, standard depth, rear to front air- flow, Rail Kit, RoHS6
SX6720| MSX6720-FS2F2| SwitchX®-2 based FDR InfiniBand 1U Switch BBU ready,36 QSFP+ ports, 2 Power Supplies (AC), x86 dual core, standard depth, rear to front airflow, Rail Kit, RoHS6

Installation

2.1 Safety Warnings
Prior to the installation, please review the safety warnings as follows:
For Nordic Countries Notices, see Section E.1, “Nordic Countries Notices,” on page 66.
• For Safety Warnings in English, see Section E.2, “Installation Safety Warnings (English),” on page 66.
2.2 System Installation and Initialization
Installation and initialization of the system require attention to the normal mechanical, power, and thermal precautions for rack-mounted equipment.

The rack mounting holes conform to the EIA-310 standard for 19-inch racks. Take precautions to guarantee proper ventilation in order to maintain good airflow at ambient temperature.
Unless otherwise specified, Mellanox products are  designed to work in an environmentally controlled data center with low levels of gaseous and dust (particulate) contamination.
The operation environment should meet severity level G1 as per ISA 71.04 for gaseous contamination and ISO 14644-1 class 8 for cleanliness level

The installation procedure for the system involves the following phases:

1. Follow the safety warnings in Section 2.1.
2. Pay attention to the air flow consideration within the system and rack – refer to “Air Flow” on page 15.
3. Make sure that none of the package contents is missing or damaged – see “Package Contents” on page 16.
4. Mount the system into a rack enclosure – see “Mounting Options” on page 20.
5. Power on the system – refer to “Initial Power On” on page 34.
6. Perform system bring-up – see “System Bring-Up” on page 35.
7. [Optional]: FRU replacements are described in Section 2.10 on page 41.

2.3 Air Flow
Mellanox systems are offered with two air flow patterns:

• Power (rear) side inlet to connector side outlet – marked with blue power supplies/fans FRUs’ handles, as shown in Figure 3.
• Connector (front) side inlet to power side outlet – marked with red power supplies/fans FRUs’ handles, as shown in Figure 4.

All servers and systems in the same rack should be planned with the same airflow direction.
All FRU components need to have the same air flow direction. A mismatch in the air flow will affect the heat dissipation.
Table 6 provides an air flow color legend and respective OPN designation
Table 6 – Air Flow Color Legend

Direction

| OPN Designation| Description
---|---|---
__| Ending with “-R”| Connector side inlet to power side outlet. Red latches are placed on the power inlet side.
__| Ending with “-F”| Power side inlet to connector side outlet. Blue latches are placed on the power inlet side.

2.4 Package Contents
Before installing your new system, unpack it and check against the parts list below that all the parts have been sent. Check the parts for visible damage that may have occurred during shipping.
If anything is damaged or missing, contact your sales representative at support@mellanox.com.
The SX6710/SX6720 package content is as follows:

• 1 – System
• 1 – Rail kit
• 1 – Power cable for each power supply unit – Type C13-C14
• 1 – Harness DB9 to RJ-45
• 1 – Quick Start Guide

2.5 Battery Backup Unit (BBU) Installation for SX6720

A BBU-compatible system is available by ordering an SX6720 system with an appropriate license.
If the BBU is deformed, leaking, corroded or visually damaged, replace it immediately
Do not use a BBU that was not provided by Mellanox Technologies

• The BBU maximum storage temperature is 25°C.
• The WEDU might show a fully charged battery, despite a possible 2-3% deviation. Nevertheless, it is still safe to discharge the battery for up to 5 minutes.
• A BBU that has not been used for six months or longer past ts printed manufacturing date is unusable.
• A BBU that is older than five years, according to the printed manufacturing date, has reached its “end of life” state and may not provide enough energy to handle the platform.
• In case the battery should be replaced, software notifications will appear on a monthly basis, starting three months before EOL time.
Do not insert the battery unit into the switch during its discharge.
A new Battery Back-up Unit (BBU) will be approximately 30% charged when you receive it. In case of a power outage, only a fully charged BBU can supply power of up to 5 minutes to the system.
2.5.1 Installing the Battery
Step 1. Extract the power FRU module from the chassis of the switch by unlocking all latches
Step 2. Insert the BBU into the empty slot in the module by sliding it in and pushing the top end of it down until it snaps into place, as illustrated in the left image of Figure 7.
Step 3. Slide the module back into the chassis.
2.5.2 Extracting the Battery
Step 1. Extract the power FRU module from the chassis of the switch by unlocking all latches.
Step 2. Press both release buttons simultaneously and lift the battery up to remove it, as illustrated in the right image of Figure 7.
Step 3. Slide the module back into the chassis.
Figure 5: Unlocking the Latches and Extracting the PSU

2.6 Mounting Options
By default, the systems are sold with the static rail kit described in Section 2.6.1. For the telescopic rail kit installation instructions, see Section 2.6.2.
2.6.1 Static Rail Kit
At least two people are required to safely mount the system in the rack.

Table 7 – Installation Kit

The following parts are included in the static rail kit ( see Figure 8) :

• 2x Rack mount rails (A)
• 2x Rack mount brackets (B)
• 2x Rack mount blades (C)
• 8x M6 Standard cage nuts¹ ² and 8x M6 Standard pan-head Phillips screws¹ (D)
• 4x Phillips100 DEG F.H TYPE-I ST.ST 6-32 X 1/4 screw with around patch (E)
  ¹ Other threads are available by special order: M5, 10-32, 12-24
  ² G-type cage-nut is available by special order.

To mount the system into the rack:
Before mounting the system to the rack, select the way you wish to place the system. Pay attention to the airflow within the rack cooling, connector and cabling options. While planning how to place the system, consider the two installation options shown in Figure 27, and review the following points:

• Make sure the system air flow is compatible with your installation selection. It is important to keep the airflow within the rack in the same direction.
• Note that the part of the system to which you choose to attach the rails (the front panel direction, as demonstrated in Option 1 or the FRUs direction, as demonstrated in Option 2) will determine the system’s adjustable side. The  system’s part to which the brackets are attached will be adjacent to the cabinet.
• The FRU side is extractable. Mounting the rack brackets inverted to the FRU side (Option 2) will allow you to slide the FRUs, in and out.

Step 1. Attach the left and right rack mount rails (A) to the switch, by gently pushing the switch chassis’ pins through the slider key holes, until locking occurs.
Step 2. Secure the chassis in the rails by screwing 2 flat head Phillips screws (E) in the designated points with a torque of 1.5±0.2 Nm. See Figure 10. Step 3. Attach the left and right rack mount brackets (B) to the switch, by gently pushing the switch chassis’ pins through the slider key holes, until locking occurs. Secure the system in the brackets by screwing the remaining 2 flat head  Phillips screws (E) in the designated points with a torque of 1.5±0.2 Nm. See Figure 11.

Step 4. Install 8 cage nuts in the desired 1U slots of the rack: 4 cage nuts in the non-extractable side (in the top and bottom holes only) and 4 cage nuts in the extractable side.

While your installation partner is supporting the system’s weight, perform the following steps:
Step 5. Mount the system into the rack enclosure, and attach the brackets installed on the system to the rack’s posts. Secure the brackets to the rack’s posts by inserting four M6 screws in the designated cage nuts, as described in Figure 13.  Do not tighten the screws yet.

Step 6. Slide the two blades into the left and right rails, and adjust them to fit your rack’s depth. Use four M6 screws (D) to fix the blades into the rack. Do not tighten the screws yet.

Step 7. Secure the system in the rack by tightening the 8 screws inserted in Step 5 and Step 6 with a torque of 4.5±0.5 Nm.
2.6.1.1 Removing the System from the Rack
 To remove a unit from the rack:
Step 1. Turn off the system and disconnect it from peripherals and from the electrical outlet.
While your installation partner is supporting the system’s weight:
Step 2. Loosen the screws attaching the brackets to the rack. Do not remove them yet.
Step 3. Loosen the screws attaching the blades to the rack, and pull the blades towards you, while your partner is holding the system.
Step 4. Extract the loosened screws from Step 2 and dismount the system from the rack.
Step 5. Remove the rails and brackets from the chassis by unscrewing 8 screws.

2.6.2 Telescopic Rail Kit

The Telescopic rail kit is not included in the system’s package, and can be purchased separately.

There are two installation kit options:

• Standard depth systems should be mounted using the standard rail kit.
• Short depth systems can be mounted using either of the rail kits.

Table 8 – Installation Kit

The following parts are included in the rail kit package (see Figure 15):

• 1x Right inner rail (A)
• 1x Left inner rail (B)
• 2x Outer rails (C)
• 2x Outer rails (D)
• 10x M6 Standard cage nuts¹ ² and 10x M6 Standard pan-head Phillips screws¹ (E)
• 2x Phillips100 DEG F.H TYPE-I ST.ST 6-32 X 1/4 screw with around patch (F)
  ¹ Other threads are available by special order: M5, 10-32, 12-24
  ² G-type cage-nut is available by special order.

The rails must be separated prior to the installation procedure. See Figure 16.

To separate the rails:

1. Extend the rail assembly by pulling the extension outwards (D).
2.  Extract rail A from rail C by pushing it outside from the rear part of the assembly. To allow complete separation of rail A from rail C, press the quick-release latch.

Before mounting the system to the rack, select the way you wish to place the system. Pay attention to the airflow within the rack cooling, connector and cabling options.
While planning how to place the system, review the following points:

• Make sure the system air flow is compatible with your installation selection. It is important to keep the airflow within the rack in the same direction.
• In case there are cables that cannot bend within the rack, or in case more space is needed for cable bending radius, it is possible to recess the connector side or the FRU side by 3” or 4” (7.62 or 10.16cm) by optional placement of the  system’s rails.
• The FRU side is extractable. Mounting the sliding rail inverted to the system will allow you to slide the FRU side of the system, in and out.
  Step 1. Install 10 cage nuts into the desired 1U slots of the rack: 4 cage nuts in the non-extractable side and 6 cage nuts in the extractable side.

Step 2. Mount both of the outer rails (C+D) into the rack (as illustrated in Figure 18), and use 8 standard pan-head screws (E) to fix them to the rack. Do not tighten the screws yet.

Step 3. If cable accommodation is required, route the power cable and/or Eth cable through either of the outer rails.
Step 4. Attach the switch to the left and right inner rails (A+B), by gently pushing the switch chassis’ pins through the slider key holes, until locking occurs.

Step 5. Secure the chassis in the inner rails by screwing the 2 flat head Phillips screws (F) in the designated points with a torque of 1.5±0.2 Nm.

Step 6. Slide the switch into the rack by carefully pushing the inner rails into the outer rails installed on the rack. Step 7. When fully inserted, fix the switch by closing the remaining 2 screws in the middle and tightening the 8 screws inserted in Step 2 with a torque of 4.5±0.5 Nm.
2.6.2.1 Removing the System from the Rack
To remove a unit from the rack:
Step 1. Turn off the system and disconnect it from peripherals and from the electrical outlet.
Step 2. Unscrew the two M6 screws securing the front of the inner rails’ ears to the outer rails and to the rack.
Step 3. Pull the unit out until braking is felt. For safety purposes, the locking mechanism will not allow a complete removal of the unit at this stage. Step 4. Press on the locking spring (appears in red in Figure 23) on both sides simultaneously, and continue pulling the unit towards you until it is fully removed.

2.7 Cable Installation
All cables can be inserted or removed with the unit powered on.
To insert a cable, press the connector into the port receptacle until the connector is firmly seated.
The LED indicator, corresponding to each data port, will light when the physical connection is established. When a logical connection is made, the relevant port LED will turn on.
To remove a cable, disengage the locks and slowly pull the connector away from the port receptacle.
The LED indicator for that port will turn off when the cable is unseated. For full cabling guidelines, ask your Mellanox representative for a copy of Mellanox Cable Management Guidelines and FAQ. For more information about port LEDs, refer to “Port LEDs”.
Do not force the cable into the cage with more than 40 newtons / 9.0 pounds / 4kg force. Greater insertion force may cause damage to the cable or to the cage.

2.7.1 Breakout Cables and Adapters
A 40GbE port can be split to 4 (or less) 10GbE ports, using a Mellanox breakout cable.
When using a port to split a data stream into four 10Gb/s data streams, one of the other switch ports will be disabled and unmapped.
2.7.1.1 Using Breakout Cables
When using this feature, you should log into the MLNX-OS® CLI and configure the individual ports to be ‘split-2’ or ‘split-4’.For further information on Mellanox’s cable, visit http://www.mellanox.com/page/interconnect_overview

2.7.1.2 SX67X0 Splitting Options

Table 9 – Port Splitting Options

Port #| Can be split to 4| Can be split to 2| Port #| Can be split to 4| Can be split to 2
---|---|---|---|---|---
1| No| No| 19| No| No
2| Yes, disables port 1| Yes| 20| Yes, disables port 19| Yes
3| No| No| 21| No| No
4| No| Yes| 22| No| No
5| No| No| 23| No| No
---|---|---|---|---|---
6| Yes, disables port 5| Yes| 24| Yes, disables port 23| Yes
7| No| No| 25| No| No
8| Yes, disables port 7| Yes| 26| Yes, disables port 19| Yes
9| No| No| 27| No| No
10| No| Yes| 28| No| Yes
11| No| No| 29| No| No
12| Yes, disables port 11| Yes| 30| Yes, disables port 23| Yes
13| No| No| 31| No| No
14| Yes, disables port 13| Yes| 32| Yes, disables port 31| Yes
15| No| No| 33| No| No
16| No| No| 34| No| Yes
17| No| No| 35| No| No
18| Yes, disables port 17| Yes| 36| Yes, disables port 35| Yes

2.8 Initial Power On
Each system’s input voltage is specified in the “Specifications” chapter. The power cords should be standard 3-wire AC power cords including a safety ground and rated for 15A or higher.
The system platform will automatically power on when AC power is applied. There is no power system. Check all boards, power supplies, and fan tray modules for proper insertion before plugging in a power cable.
Step 1. Plug in the first power cable.
Step 2. Plug in the second power cable.
Step 3. Wait for the System Status LED to turn green.
It may take up to five minutes to turn on the system. If the System Status LED shows red after five minutes, unplug the system and call your Mellanox representative for assistance.
Step 4. Check the System Status LEDs and confirm that all of the LEDs show status lights consistent with normal operation (initially flashing, and then moving to a steady color) as shown in Figure 27 below. For  more information, refer to “LEDs”.

After inserting a power cable and confirming the green System Status LED light is on, make sure that the Fan Status LED shows green.
If the Fan Status LED is not green, unplug the power connection and check that the fan module is inserted properly and that the mating connector of the fan unit is free of any dirt and/or obstacles. If no obstacles  were found and the problem persists, call your Mellanox representative for assistance.
Figure 28: Two Power Inlets – Electric Caution Notifications
Risk of electric shock and energy hazard. The two power supply units are independent. Disconnect all power supplies to ensure a powered down state inside of the switch platform.
2.9 System Bring-Up
2.9.1 Configuring Network Attributes
The procedures described in this chapter assume that you have already installed and powered on the system according to the instructions in this document. The system comes with a pre-configured DHCP. If you wish to disable it, refer to “Disable Dynamic Host Configuration Protocol (DHCP)”. In case a manual configuration is required, please refer to the instructions in Section 2.9.1.1.
2.9.1.1 Manual Host Configuration
 To perform initial configuration of the system:
Step 1. Connect a host PC to the Console RJ45 ( ) port of the system, using the supplied harness cable (DB9 to RJ45). Make sure to connect to the Console RJ45 port and not to the  (Ethernet) MGT (   ) port.
Step 2. Configure a serial terminal program (for example, HyperTerminal, minicom, or Tera Term) on your host PC with the settings described in Table 10. Once you perform that, you should get the CLI prompt of  the system.
Table 10 – Serial Terminal Program Configuration

Step 3. Login as admin and use admin as password. On the first login, the MLNX-OS configuration wizard will start.
Step 4. To configure network attributes and other initial parameters to the system, follow the configuration wizard as shown in Table 11.
Table 11 – Configuration Wizard Session

Mellanox configuration wizard Do you want to use the wizard for initial configuration? yes| You must perform this configuration the first time you operate the system or after resetting the system. Type ‘y’ and then press