Lenovo ThinkSystem NVIDIA ConnectX-7 NDR InfiniBand OSFP400 Adapters User Guide

ThinkSystem NVIDIA ConnectX-7 NDR InfiniBand
OSFP400 Adapters
Product Guide

ThinkSystem NVIDIA ConnectX-7 NDR InfiniBand OSFP400 Adapters

The ThinkSystem NVIDIA ConnectX-7 NDR InfiniBand OSFP400 Adapters offer 400 Gb/s InfiniBand connectivity for high-performance connectivity when running HPC, cloud, storage and machine learning applications.
The following figure shows the ThinkSystem Nvidia ConnectX-7 NDR OSFP400 1-Port PCIe Gen5 x16 InfiniBand Adapter (the standard heat sink has been removed in this photo).

Figure 1. ThinkSystem Nvidia ConnectX-7 NDR OSFP400 1-Port PCIe Gen5 x16 InfiniBand Adapter

Did you know?
The ConnectX-7 NDR adapters are optimized to deliver accelerated networking for modern cloud, artificial intelligence, and traditional enterprise workloads. ConnectX-7 provides a broad set of software-defined, hardware- accelerated networking, storage, and security capabilities which enable organizations to modernize and secure their IT infrastructures.

Click here to check for update
ThinkSystem NVIDIA ConnectX-7 NDR InfiniBand OSFP400 Adapters

Part number information

The following table shows the part numbers for the adapters.
CTO orders: For configure-to-order builds, these adapters are only available when you select one of the HPC & AI modes in the DCSC configurator. Not available in General Purpose mode of DCSC.

Table 1. Ordering information

Part number| Feature code| NVIDIA equivalent| Description
---|---|---|---
Air-cooled adapter
4XC7A80289| BQ1N| MCX75510AAS- NEAT| ThinkSystem Nvidia ConnectX-7 NDR OSFP400 1-Port PCIe Gen5 x16 InfiniBand Adapter
Water-cooled adapters
4XC7A86671| BKSN| MCX75510AAS- NEAT-DWC| ThinkSystem NVIDIA ConnectX-7 NDR OSFP400 1-Port PCIe Gen5 x16 InfiniBand Adapter DWC
4XC7A86670| BKSP| MCX75510AAS- NEAT-DWC| ThinkSystem NVIDIA ConnectX-7 NDR OSFP400 1-Port PCIe Gen5 x16 InfiniBand Adapter (SharedIO) DWC

Non-DWC adapter part numbers includes the following:

• One NVIDIA adapter with full-height (3U) adapter bracket attached
• Low-profile (2U) adapter bracket
• Documentation

Supported transceivers and cables

The ConnectX-7 NDR adapters have an empty OSFP400 cage for connectivity. The following table lists the supported transceivers.
Table 2. Transceivers for ConnectX-7 NDR adapters

400Gb transceivers
4TC7A81826| BQJT| ThinkSystem NDR OSFP400 IB Multi Mode Solo-Transceiver

The following table lists the supported fiber optic cables and Active Optical Cables.
Table 3. Optical cables

NVIDIA NDR Multi Mode Fibre Cables
4X97A81748| BQJN| Lenovo 3M NVIDIA NDR Multi Mode MPO12 APC Optical Cable
4X97A81749| BQJP| Lenovo 5M NVIDIA NDR Multi Mode MPO12 APC Optical Cable
4X97A81750| BQJQ| Lenovo 7M NVIDIA NDR Multi Mode MPO12 APC Optical Cable
4X97A81751| BQJR| Lenovo 10M NVIDIA NDR Multi Mode MPO12 APC Optical Cable
4X97A81752| BQJS| Lenovo 20M NVIDIA NDR Multi Mode MPO12 APC Optical Cable
4X97A85349| BSN6| Lenovo 30M NVIDIA NDR Multi Mode MPO12 APC Optical Cable

The following table lists the supported direct-attach copper (DAC) cables.
Table 4. Copper cables

NVIDIA NDRx2 OSFP800 to 2x NDR OSFP400 Splitter Copper Cables
4X97A81827| BQJV| Lenovo 1M NVIDIA NDRx2 OSFP800 to 2x NDR OSFP400 Passive Copper Splitter Cable
4X97A81828| BQJW| Lenovo 1.5M NVIDIA NDRx2 OSFP800 to 2x NDR OSFP400 Passive Copper Splitter Cable
4X97A81829| BQJX| Lenovo 2M NVIDIA NDRx2 OSFP800 to 2x NDR OSFP400 Passive Copper Splitter Cable
4X97A81830| BQJY| Lenovo 3M NVIDIA NDRx2 OSFP800 to 2x NDR OSFP400 Passive Copper Splitter Cable

Features

The adapters have the following features:

• Accelerated Networking and Security
  ConnectX-7 provides a broad set of software-defined, hardware-accelerated networking, storage, and security capabilities which enable organizations to modernize and secure their IT infrastructures. Moreover, ConnectX-7 empowers agile and high- performance solutions from edge to core data centers to clouds, all while enhancing network security and reducing the total cost of ownership.

• Accelerate Data-Driven Scientific Computing
  ConnectX-7 provides ultra-low latency, extreme throughput, and innovative NVIDIA In-Network Computing engines to deliver the acceleration, scalability, and feature- rich technology needed for today’s modern scientific computing workloads.

• Accelerate Software- Defined Networking
  NVIDIA ASAP² technology accelerates software-defined networking, delivering line-rate  performance with no CPU penalty.

• Provide Security from Edge to Core
  Hardware engines in ConnectX-7 offload and accelerate security, with in-line encryption/decryption of TLS, IPsec, and MACsec.

• Enhance Storage Performance
  ConnectX-7 enables high- performance and efficient data storage by leveraging RDMA/RoCE, GPUDirect Storage, and hardware-based NVMe-oF offload engines.

• SharedIO (4XC7A86670 only)
  SharedIO (Shared I/O or Multi-Host) is an implementation of Sockets Direct that is offered in Lenovo direct water-cooled (DWC) servers such as the SD650 V3, where there are two server nodes per DWC tray. In this implementation, the ConnectX-7 adapter is installed in a PCIe slot in one node and the adapter is connected via a cable to a PCIe connector on the second node. The result is that the two nodes share the network connection of the adapter with significant savings both in the cost of the adapters but also the cost of switch ports.

Technical specifications

The adapters have the following technical specifications:

• Networking interfaces
  • One OSPF400 cage for 400 Gb/s connectivity
• InfiniBand connectivity
  • Support for InfiniBand NDR
  • InfiniBand Trade Association Spec 1.5 compliant
  • RDMA, send/receive semantics
  • 16 million input/output (IO) channels
  • 256 to 4Kbyte maximum transmission unit (MTU), 2Gbyte messages
• Enhanced InfiniBand Networking
  • Hardware-based reliable transport
  • Extended Reliable Connected (XRC)
  • Dynamically Connected Transport (DCT)
  • GPUDirect RDMA
  • GPUDirect Storage
  • Adaptive routing support
  • Enhanced atomic operations
  • Advanced memory mapping, allowing user mode registration (UMR)
  • On-demand paging (ODP), including registration-free RDMA memory access
  • Enhanced congestion control
  • Burst buffer offload
  • Single root IO virtualization (SR-IOV)
  • Optimized for HPC software libraries including: NVIDIA HPC-X, UCX, UCC, NCCL,
  • OpenMPI, MVAPICH, MPICH, OpenSHMEM, PGAS
  • Collective operations offloads
  • Support for NVIDIA Scalable Hierarchical Aggregation and Reduction Protocol (SHARP)
  • Rendezvous protocol offload
  • In-network on-board memory
• Storage Accelerations
  • Block-level encryption: XTS-AES 256/512-bit key
  • NVMe over Fabrics (NVMe-oF)
  • NVMe over TCP (NVMe/TCP)
  • T10 Data Integrity Field (T10-DIF) signature handover
  • SRP, iSER, NFS over RDMA, SMB Direct
• Management and Control
  • NC-SI, MCTP over SMBus, and MCTP over PCIe
  • PLDM for Monitor and Control DSP0248
  • PLDM for Firmware Update DSP0267
  • PLDM for Redfish Device Enablement DSP0218
  • PLDM for FRU DSP0257
  • SPDM DSP0274
  • Serial Peripheral Interface (SPI) to flash
  • JTAG IEEE 1149.1 and IEEE 1149.6
• Remote Boot
  • Remote boot over InfiniBand
  • Remote boot over iSCSI
  • UEFI
  • PXE
• Cybersecurity
  • Inline hardware IPsec encryption and decryption: AES-GCM 128/256-bit key, IPsec over RoCE
  • Inline hardware TLS encryption and decryption: AES-GCM 128/256-bit key
  • Inline hardware MACsec encryption and decryption: AES-GCM 128/256-bit key
  • Platform security: secure boot with hardware root-of-trust, secure firmware update, flash encryption, and device attestation
• PCI Express Interface
  • PCIe Gen 5.0 x16 host interface
  • Support for PCIe bifurcation
  • SharedIO (NVIDIA Multi-Host) supports connection of up to two hosts (on supported watercooled nodes only)
  • Transaction layer packet (TLP) processing hints (TPH)
  • PCIe switch Downstream Port Containment (DPC)
  • Support for MSI/MSI-X mechanisms
  • Advanced error reporting (AER)
  • Access Control Service (ACS) for peer-to-peer secure communication
  • Process Address Space ID (PASID)
  • Address translation services (ATS)
  • Support for SR-IOV

NVIDIA Unified Fabric Manager

NVIDIA Unified Fabric Manager (UFM) is InfiniBand networking management software that combines enhanced, real-time network telemetry with fabric visibility and control to support scale-out InfiniBand data centers.
The two offerings available from Lenovo are as follows:

• UFM Telemetry for Real-Time Monitoring
  The UFM Telemetry platform provides network validation tools to monitor network performance and conditions, capturing and streaming rich real-time network telemetry information, application workload usage, and system configuration to an on- premises or cloud-based database for further analysis.

• UFM Enterprise for Fabric Visibility and Control
  The UFM Enterprise platform combines the benefits of UFM Telemetry with enhanced network monitoring and management. It performs automated network discovery and provisioning, traffic monitoring, and congestion discovery. It also enables job schedule provisioning and integrates with industry-leading job schedulers and cloud and cluster managers, including Slurm and Platform Load Sharing Facility (LSF).

The following table lists the subscription licenses available from Lenovo.

Table 5. NVIDIA Unified Fabric Manager subscriptions

UFM Telemetry
7S02003HWW| S88D| UFM Telemetry 1-year License and Gold-Support for Lenovo clusters. Per node.
7S02003JWW| S88E| UFM Telemetry 3-year License and Gold-Support for Lenovo clusters. Per node.
7S02003KWW| S88F| UFM Telemetry 5-year License and Gold-Support for Lenovo clusters. Per node.
UFM Enterprise
7S02003LWW| S88G| UFM Enterprise 1-year License and Gold-Support for Lenovo clusters. Per node.
7S02003MWW| S88H| UFM Enterprise 3-year License and Gold-Support for Lenovo clusters. Per node.
7S02003NWW| S88J| UFM Enterprise 5-year License and Gold-Support for Lenovo clusters. Per node.

For more information, see the following web page:https://www.nvidia.com /en-us/networking/infiniband/ufm/

Server support
The following tables list the ThinkSystem servers that are compatible.

Table 6. Server support (Part 1 of 3)

Table 7. Server support (Part 2 of 3)

Table 8. Server support (Part 3 of 3)

Operating system support

The adapters support the operating systems listed in the following tables.

• ThinkSystem Nvidia ConnectX-7 NDR OSFP400 1-Port PCIe Gen5 x16 InfiniBand Adapter, 4XC7A80289
• ThinkSystem NVIDIA ConnectX-7 NDR OSFP400 1-Port PCIe Gen5 x16 InfiniBand Adapter DWC, 4XC7A86671
• ThinkSystem NVIDIA ConnectX-7 NDR OSFP400 1-Port PCIe Gen5 x16 InfiniBand Adapter (SharedIO) DWC, 4XC7A86670

Tip: These tables are automatically generated based on data from Lenovo ServerProven.
Table 9. Operating system support for ThinkSystem Nvidia ConnectX-7 NDR OSFP400 1-port PCIe Gen5 x16 InfiniBand Adapter, 4XC7A80289

Operating systems| SR630 V3| SR635 V3| SR645 V3| SR650 V3| SR655 V3| SR665 V3
---|---|---|---|---|---|---
Microsoft Windows Server 2019| Y| Y| Y| Y| Y| Y
Microsoft Windows Server 2022| Y| Y| Y| Y| Y| Y
Red Hat Enterprise Linux 8.6| Y| Y| Y| Y| Y| Y
Red Hat Enterprise Linux 8.7| Y| Y| Y| Y| Y| Y
Red Hat Enterprise Linux 9.0| Y| Y| Y| Y| Y| Y
Red Hat Enterprise Linux 9.1| Y| Y| Y| Y| Y| Y
SUSE Linux Enterprise Server 15 SP4| Y| Y| Y| Y| Y| Y
Ubuntu 22.04 LTS| Y| Y| Y| Y| Y| Y
VMware vSphere Hypervisor (ESXi) 8.0 U1| Y 1| N| N| N| N| N

1 For limitation, please refer Support TIP 104278
Table 10. Operating system support for ThinkSystem NVIDIA ConnectX-7 NDR OSFP400 1-port PCIe Gen5 x16 InfiniBand Adapter DWC, 4XC7A86671

Table 11. Operating system support for ThinkSystem NVIDIA ConnectX-7 NDR OSFP400 1-port PCIe Gen5 x16 InfiniBand Adapter (SharedIO) DWC, 4XC7A86670

Regulatory approvals
The adapters have the following regulatory approvals:

• Safety: CB / cTUVus / CE
• EMC: CE / FCC / VCCI / ICES / RCM / KC
• RoHS: RoHS Compliant

Operating environment
The adapters have the following operating characteristics:

• Maximum power available through OSFP port: 17W
• Temperature
  • Operational: 0°C to 55°C
  • Non-operational: -40°C to 70°C
• Humidity: 90% relative humidity

Warranty
One year limited warranty. When installed in a Lenovo server, the adapter assumes the server’s base warranty and any warranty upgrades.

Related publications

For more information, refer to these documents:

• Networking Options for ThinkSystem Servers: https://lenovopress.com/lp0765-networking-options-for-thinksystem-servers
• ServerProven compatibility: http://www.lenovo.com/us/en/serverproven
• NVIDIA InfiniBand product page: https://www.nvidia.com/en-us/networking/infiniband-adapters/
• ConnectX-7 user manual: https://docs.nvidia.com/networking/display/ConnectX7VPI/Specifications

Related product families

• Product families related to this document are the following:

InfiniBand & Omni-Path Adapters

Notices
Lenovo may not offer the products, services, or features discussed in this document in all countries. Consult your local Lenovo representative for information on the products and services currently available in your area. Any reference to a Lenovo product,  rogram, or service is not intended to state or imply that only that Lenovo product,
rogram, or service may be used. Any functionally equivalent product, program, or service that does not infringe any Lenovo intellectual property right may be used instead. However, it is the user’s responsibility to evaluate and verify the operation of any other product, program, or service. Lenovo may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: Lenovo (United States), Inc.

Lenovo (United States), Inc.
8001 Development Drive
Morrisville, NC 27560
U.S.A.
Attention: Lenovo Director of Licensing

LENOVO PROVIDES THIS PUBLICATION ”AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some jurisdictions do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you.
This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. Lenovo may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice.

The products described in this document are not intended for use in implantation or other life support applications where malfunction may result in injury or death to persons. The information contained in this document does not affect or change Lenovo product specifications or warranties. Nothing in this document shall operate as an express or implied license or indemnity under the intellectual property rights of Lenovo or third parties. All information contained in this document was obtained in specific environments and is presented as an illustration. The result obtained in other operating environments may vary. Lenovo may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you.
Any references in this publication to non-Lenovo Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this Lenovo product, and use of those Web sites is at your own risk. Any performance data contained herein was determined in a controlled environment. Therefore, the result obtained in other operating environments may vary significantly. Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems. Furthermore, some measurements may have been estimated through extrapolation. Actual results may vary. Users of this document should verify the applicable data for their specific environment.

© Copyright Lenovo 2023. All rights reserved.

This document, LP1692, was created or updated on May 2, 2023.
Send us your comments in one of the following ways:

• Use the online Contact us review form found at: https://lenovopress.lenovo.com/LP1692
• Send your comments in an e-mail to: comments@lenovopress.com

This document is available online at https://lenovopress.lenovo.com/LP1692.

Trademarks

Lenovo and the Lenovo logo are trademarks or registered trademarks of Lenovo in the United States, other countries, or both. A current list of Lenovo trademarks is available on the Web at https://www.lenovo.com/us/en/legal/copytrade/.

The following terms are trademarks of Lenovo in the United States, other countries, or both:
Lenovo®
ServerProven®
ThinkSystem®
The following terms are trademarks of other companies:
Intel® is a trademark of Intel Corporation or its subsidiaries.
Linux® is the trademark of Linus Torvalds in the U.S. and other countries.
Microsoft®, Windows Server®, and Windows® are trademarks of Microsoft Corporation in the United States, other countries, or both.
Other company, product, or service names may be trademarks or service marks of others.

ThinkSystem NVIDIA ConnectX-7 NDR InfiniBand OSFP400 Adapters

References

• ESXi runs into security violation PSOD after a system board replacement or if TPM is in DA lockout mode - Lenovo ThinkSystem - Lenovo Support TW
• DCSC - Data Center Solution Configurator
• Specifications - NVIDIA Docs
• Networking Options for ThinkSystem Servers > Lenovo Press
• InfiniBand & Omni-Path Adapters > Lenovo Press
• Copyright and Trademark Information | Lenovo US | Lenovo US
• Welcome | ServerProven
• NVIDIA InfiniBand Adapters | NVIDIA
• NVIDIA Unified Fabric Manager (UFM) | NVIDIA

Read User Manual Online (PDF format)

Read User Manual Online (PDF format)  >>

Download This Manual (PDF format)

Download this manual  >>