Instruction/ maintenance manual of the product ConnectX-3 VPI Mellanox Technologies
Go to page of 60
www .mellanox.com ConnectX ® -3 VPI Single and Dual QSFP+ Port Adapter Card User Manual P/N: MCX353A-FCBT , MCX353A-TCBT , MCX353A-Q CBT , MCX354A-FCBT , MCX354A-TCBT , MCX354A-QCBT Rev 1.
Mellanox T echnologies 350 Oakmead Parkwa y Suite 100 Sunnyvale, CA 94085 U.S.A. www .mellanox.com T el: (408) 970-3400 Fax: (408) 970-34 03 Mellanox T echnologies, Ltd. PO Box 586 Hermon Building Y okneam 20692 Israel T el: +972-4-909-7200 Fax: +972-4-959-3245 © Copyright 2012.
ConnectX®-3 VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.4 Mellanox Technologies 3 T able of Content s Table of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Rev 1.4 Mellanox Technologies 4 4.2.1 Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 4.2.2 Windows . . . . . . . . . . . . . . . . . . . . . . . . . . .
ConnectX®-3 VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.4 Mellanox Technologies 5 List of T ables Table 1: Revision History Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Table 2: Documents List .
Rev 1.4 Mellanox Technologies 6 List of Figures Figure 1: Card Product Label (Example) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Figure 2: MCX354A-[QFT]CBT Card . . . . . . . . . . . . . . . . . . . . .
ConnectX®-3 VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.4 Mellanox Technologies 7 Revision History This document was printed on Augu st 27, 2012.
Rev 1.4 Mellanox Technologies 8 About this Manual This User Manu al describes Mellanox T echnologies Conn ectX®-3 VPI Single and Dual QSFP+ port PCI Express x8 adapter cards. It provides de tails as to the interfaces of the board, specifica - tions, required software and firmware for oper atin g the board, and relevant documentatio n.
ConnectX®-3 VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.4 Mellanox Technologies 9 Customers who purchased Mellanox M-1 Global Su pport Services, please see your contract for details regarding T echnical Support.
Overview Rev 1.4 Mellanox Technologies 10 1 Overview This document is a User Manual for Mellanox T echnologies VP I adapter base d on the Con - nectX®-3 VPI integrated circuit devic e. The cards described in this manual have the following main features: • V irtual Protocol Interconnect (VPI) • InfiniBand Architecture Specification v1.
ConnectX®-3 VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.4 Mellanox Technologies 11 1.1.3 InfiniBand QDR QDR is a standard InfiniBand data rate, where each lane of a 4X port runs a bit rate of 10Gb/s.
Overview Rev 1.4 Mellanox Technologies 12 1.2 Adapter Cards Cover ed in this Manual Ta b l e 3 lists the single and du al-port QDR adapter cards. Ta b l e 4 lists the single and dual-port FDR10 adapter cards Ta b l e 5 lists the single and dual-port FDR adapter cards.
T able 5 - Single and Dual-port FDR Adapter Cards Orderin g Part N umber (OPN) PCI Express SERDES Speed Data T ransmis- sion Rat e/ # of ports RoHS Adapter IC Part Number Device ID (decimal) MCX353A-FCBT PCIe 3.0 x8 8GT/s 56 Gb/s FDR IB, 40GigE single- port QSFP+ R6 MT27508A1-FCCR-FV MCX354A-FCA T (Legacy product) PCIe 3.
Overview Rev 1.4 Mellanox Technologies 14 1. Installation Instructions Read all installation instruct ions before c onnecting the equipmen t to the power source. 2. Over -temperature This equipment should not be op erated in an area w ith an ambient temperature exceed - ing the maximum recommended: 55°C (131°F).
ConnectX®-3 VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.4 Mellanox Technologies 15 8. Hazardous Radiation Exposure Caution – Use of controls or adjustment or performance of procedures other than those specified herein may result in hazardous radiation expo sure.
Adapter Card Interfaces Rev 1.4 Mellanox Technologies 16 2 Adapter Card Interfaces 2.1 I/O Interfaces Each adapter card includes the following interfaces: • QSFP+ port(s) • PCI Express (PCIe) x8 e.
ConnectX®-3 VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.4 Mellanox Technologies 17 Figure 3: Dual-port Bracke t Figure 4: Single-port Bracket.
Adapter Card Interfaces Rev 1.4 Mellanox Technologies 18 2.1.1 InfiniBand Interface The network ports of the ConnectX®-3 adapter cards are compliant with the InfiniBand Architectur e Specification, Release 1.2.1. InfiniBand traf fic is transmitted through the cards' QSFP+ connectors.
ConnectX®-3 VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.4 Mellanox Technologies 19 2.1.5 LED Assignment There are two I/O LEDs per port in dual-port desi gns and one bicolor LED in single-port designs, green and yello w , located on the I/O panel.
The short bracket has the same port a nd LED footprints as the tall bracket. Adapter Card Interfaces Rev 1.4 Mellanox Technologies 20 2.1.6 I 2 C-comp atible Interface A three-pin header on the adapte r cards is provided as the I 2 C-compatible interface.
ConnectX®-3 VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.4 Mellanox Technologies 21 When a jumper is inserted into the drill holes in the adapter it indicates to the device that the on- board Flash device should not be used fo r boot instructions .
Adapter Card Interfaces Rev 1.4 Mellanox Technologies 22 54 VPD Keyword EC STR Engineering Change Level of the card (rev) 56 Length 0x2 57 Revision RV %STR PCB revision Example: “A1” 59 VPD Keyword SN STR Serial Numbe r 61 Length 0x18 62 SerialNumber SN %STR_SPC “00.
ConnectX®-3 VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.4 Mellanox Technologies 23 1539 Data STR_ZERO R eserved (0 x00) 1792 VPD Keyword RW STR Remaining read/write area 1794 Lengt .
Adapter Card Installation Rev 1.4 Mellanox Technologies 24 3 Adapter Card Inst allation 3.1 Hardware Requirement s A system with a PCI Express x8 slot is required for in stalling the card. 3.2 Inst allation Instructions Read all installation instruc tions befo re connecting t he equipment to the power source.
ConnectX®-3 VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.4 Mellanox Technologies 25 Figure 7: PCI Device 5. In the V alue display box, check the fields VEN a nd DEV (fields are separated by ‘&’).
Adapter Card Installation Rev 1.4 Mellanox Technologies 26 3.4 Cables and Modules Please refer to “Mellanox Products Approved Cable Lists” at http://www .mellanox.com/related- docs/user_manuals/Mella nox_approved_cables.pdf 3.4.1 Cable Inst allation 1.
ConnectX®-3 VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.4 Mellanox Technologies 27 3. Insert the connector into the adapter card. Be careful to insert the conn ector straight into the cage. Do not apply any to rque, up or down, to the connector cage in the adapter card.
Driver Software and Firmware Rev 1.4 Mellanox Technologies 28 4 Driver Sof tware and Firmware 4.1 Driver Sof tware 4.1.1 Linux For Linux, download and install the latest Open Fabrics Enterpr ise Distribution (OFED) software package available via the Mellanox web site at: http://www .
Advertised auto-negotia tion: N Speed: 1000Mb/s Duplex: Full Port: Twisted Pair PHYAD: 0 Transceiver: internal Auto-negotiation: on Supports Wake-on: g Wake-on: d Link detected: yes ConnectX®-3 VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.
Physical state: LinkUp Rate: 56 Base lid: 8 LMC: 0 SM lid: 12 Capability mask : 0x02514868 Port GUID: 0x00 02c903002fefe2 Link layer: Inf iniBand Driver Software and Firmware Rev 1.
ConnectX®-3 VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.4 Mellanox Technologies 31 The “Usage: openibd {start|stop|re start|status}” co mmand to modify this f ile and thereby control the drivers.
Driver Software and Firmware Rev 1.4 Mellanox Technologies 32 Drivers =>InfiniBand & VPI Software/Drivers => Me llanox W inOF VPI => Download. Follow the installation instructions incl uded in the download package (a lso available from the download page).
ConnectX®-3 VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.4 Mellanox Technologies 33 Port configuration is saved in the file: /etc/infi niband/connectx.conf. Th is saved configuration is restored at driver restart only if do ne via "/etc/init.
Driver Software and Firmware Rev 1.4 Mellanox Technologies 34 Step 2. Right-click on the Mellanox ConnectX VPI netw ork adapter and left-clic k Properties. Select the Port Protocol tab from the Properties sheet. Note: The "Port Protocol” tab is display ed only if the NIC is a VPI (IB and ETH).
ConnectX®-3 VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.4 Mellanox Technologies 35 project (see www .ipxe.org). For more information go to http://www .mellanox.com => Products => Software/Drivers => InfiniBa nd & VPI Software/D rivers => FlexBoot => Download.
Driver Software and Firmware Rev 1.4 Mellanox Technologies 36 The number of shared pages by GPUDirect is reported under: 4.4.2.3 NVIDIA Driver Inst allation Install NVIDIA Development Driver for Linux x86_64 , available under: For example: T o make sure that the NVIDIA driver was insta lled successfully: 4.
ConnectX®-3 VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.4 Mellanox Technologies 37 ConnectX®-3 EN with RoCE based network manageme nt is the same as that for any Ethernet and DCB-based network management, eliminating the need for IT managers to learn new technolo - gies.
Driver Software and Firmware Rev 1.4 Mellanox Technologies 38 2. Compare the programmed firmware ve rsion with the latest available. a. Go to Mellanox’ s web site: http://www .mellanox.co m/supportdownlo ader . See Figure 10. b. Enter your card PSID to display the latest firmware.
ConnectX®-3 VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.4 Mellanox Technologies 39 d. Reboot the computer . e. Enter: mst start. f. V erify that the card firmw are was updated successfully .
Rev 1.4 Mellanox Technologies 40 Appendix A: S pecifications A.1 MCX353A-QCBT S pecifications T able 1 1 - MCX353A-QCBT Specifications T able Physical Size: 2.12in. x5.6 in. (53.59mm x 142.25 mm) Connector: QSFP+ 40Gb/s InfiniBand (Copper and optical) Protocol Support InfiniBand: IBT A v1.
ConnectX®-3 VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.4 Mellanox Technologies 41 A.2 MCX354A-QCBT S pecifications T able 12 - MCX354A-QCBT Specifications T able Physical Size: 2.71n. x5.6 in. (68.90mm x 142.25 mm) Connector: QSFP+ 40Gb/s InfiniBand (Copper and optical) Protocol Support InfiniBand: IBT A v1.
Rev 1.4 Mellanox Technologies 42 A.3 MCX353A-TCBT S pecifications T able 13 - MCX353A-TCBT Specifications T able Physical Size: 2.12in. x5.6 in. (53.59mm x 142.25 mm) Connector: QSFP+ 40Gb/s InfiniBand (Copper and optical) Protocol Support InfiniBand: IBT A v1.
ConnectX®-3 VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.4 Mellanox Technologies 43 A.4 MCX354A-TCBT S pecifications T able 14 - MCX354A-TCBT Specifications T able Physical Size: 2.71n. x5.6 in. (68.90mm x 142.25 mm) Connector: QSFP+ 40Gb/s InfiniBand (Copper and optical) Protocol Support InfiniBand: IBT A v1.
Rev 1.4 Mellanox Technologies 44 A.5 MCX353A-FCBT S pecifications T able 15 - MCX353A-FCBT Specifications T able Physical Size: 2.12in. x5.6 in. (53.59mm x 142.25 mm) Connector: QSFP+ InfiniBand (Copper and optical) Protocol Support InfiniBand: IBT A v1.
ConnectX®-3 VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.4 Mellanox Technologies 45 A.6 MCX354A-FCBT S pecifications T able 16 - MCX354A-FCBT Specifications T able Physical Size: 2.71in. x5.6 in. (68.90mm x 142.25 mm) Connector: QSFP+ InfiniBand (Copper and optical) Protocol Support InfiniBand: IBT A v1.
Rev 1.4 Mellanox Technologies 46 A.7 Board Mechanical Drawing and Dimensions All dimensions are i n millimeters. All the mechanical tolerances are +/- 0.1mm Figure 1 1: Mechanical Drawing of the Single-port MCX313353A-B[QFT]CBT 142.24 53.59 40.5 49.09 45.
ConnectX®-3 VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.4 Mellanox Technologies 47 Figure 12: Mechanic al Drawing of the Du al-port MCX3 54A-[QFT]CBT Adapte r Card 142.25 56.15 68.90 45.96 12.75 50.00 64.4 J1 - I 2 C Connector Flash Jump er A.
Rev 1.4 Mellanox Technologies 48 Appendix B: Interface Connectors Pinout B.1 I 2 C-comp atible Connector Pinout Figure 13: Compatible Connecto r Plug and Pinout T able 17 - I2C-compatible Connector Pinou t Connector Pin Number Signal Name 1S P S D A 2 SPSCL 3 GND 4N C 5N C B.
ConnectX®-3 VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.4 Mellanox Technologies 49 Figure 14: PCIe Connector Pinout.
Rev 1.4 Mellanox Technologies 50 B.3 QSFP+ Connector Pinout Figure 15: Connector and Cage V iews.
T able 18 - Connector Pin Number and Name to Signal Name Map Connector Pin Number Connector Pin Name Port A Signal Name 1 GND GND 2 TXN_2 Tx2n 3 TXP_2 Tx2p 4 GND GND 5 TXN_4 Tx4n 6 TXP_4 Tx4p 7 GND GN.
Rev 1.4 Mellanox Technologies 52 Appendix C: Replacing a T all Bracket With a Short Bracket This section provides instructions on how to remove the tall bracket of a standard Me llanox T ech - nologies adapter card and replace it with a short one.
ConnectX®-3 VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.4 Mellanox Technologies 53 C.3 Inst a lling the New Bracket 1. Remove the paper to expose the adhesive on the gasket. 2. Place the gasket for each port onto the new bracke t. Ma ke sure to corr ectly align the gasket with the hole in the bracket.
Rev 1.4 Mellanox Technologies 54 Figure 19: Placing the Br acket on the Card 3. Make sure that the LEDs are a ligned onto the bracket holes. 4. Use a torq ue driver to apply up to 2 lbs-in torque on the screws.
ConnectX®-3 VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.4 Mellanox Technologies 55 Appendix D: A vertissements de sécurité d’inst allation (W arnings in French) 1. Instructions d’installation Lisez toutes les instruction s d’installation avant de brancher le matériel à la source d’alimentation électri que.
Rev 1.4 Mellanox Technologies 56 7. Codes électriques locaux et nationaux Ce matériel doit être installé dans le respect des codes électri ques locaux et nationaux.
ConnectX®-3 VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.4 Mellanox Technologies 57 Appendix E: Sicherheit shinweise (W arnings in German) 1. Installationsanleitungen Lesen Sie alle Installationsanlei tungen, bevor Sie das Gerät an die Stromversor gung anschließen.
This equipment should be installed in compliance with local and national elec - trical codes. Rev 1.4 Mellanox Technologies 58 8. S trahlenkontak Achtung – Nutzun g von S teuerungen oder Einstellung en oder Ausfü hrung von Prozeduren, die hier nicht spezifiziert sind, kann zu gefährlichem Strahlenkon - takt führen.
ConnectX®-3 VPI Single and Dual QSFP+ Port Adapter Card User Manual Rev 1.4 Mellanox Technologies 59 Appendix F: Adverten cias de seguridad p ara la inst alación (W arnings in S p anish) 1. Instrucciones de instalación Antes de conectar el equipo a la fuente de alimentación, leer toda s las instrucciones de instalación.
Rev 1.4 Mellanox Technologies 60 7. Códigos eléctricos locales y nacionales Este equipo se debe i nstalar conforme a los códigos eléctricos locales y nacionales.
An important point after buying a device Mellanox Technologies ConnectX-3 VPI (or even before the purchase) is to read its user manual. We should do this for several simple reasons:
If you have not bought Mellanox Technologies ConnectX-3 VPI yet, this is a good time to familiarize yourself with the basic data on the product. First of all view first pages of the manual, you can find above. You should find there the most important technical data Mellanox Technologies ConnectX-3 VPI - thus you can check whether the hardware meets your expectations. When delving into next pages of the user manual, Mellanox Technologies ConnectX-3 VPI you will learn all the available features of the product, as well as information on its operation. The information that you get Mellanox Technologies ConnectX-3 VPI will certainly help you make a decision on the purchase.
If you already are a holder of Mellanox Technologies ConnectX-3 VPI, but have not read the manual yet, you should do it for the reasons described above. You will learn then if you properly used the available features, and whether you have not made any mistakes, which can shorten the lifetime Mellanox Technologies ConnectX-3 VPI.
However, one of the most important roles played by the user manual is to help in solving problems with Mellanox Technologies ConnectX-3 VPI. Almost always you will find there Troubleshooting, which are the most frequently occurring failures and malfunctions of the device Mellanox Technologies ConnectX-3 VPI along with tips on how to solve them. Even if you fail to solve the problem, the manual will show you a further procedure – contact to the customer service center or the nearest service center
