Instruction/ maintenance manual of the product GEM318P Cisco Systems
Go to page of 74
Americas Headquarters Cisco Systems, In c. 170 West Tasman Drive San Jose, CA 951 34-1706 USA http://www.ci sco.com Tel: 408 526-4000 800 553-NETS (638 7) Fax: 408 527-0883 Cisco SFS Inf iniBand Host Driv er s User Guide f or Linux Re lease 3.
THE SPECIFICATION S AND INFORMAT ION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT T O CHANGE WITH OUT NOTICE. A LL STATEMENTS , INFORMATION, AND RECOMMENDATI ONS IN THI S MANUAL ARE BE LIEVED TO BE A CCURATE BUT ARE PRESENTED WI THOUT WARRANTY OF ANY KIND, EX PRESS OR IMPLIED.
iii Cisco SFS InfiniBand Host Dr ivers User Guide for Linux OL-12309-01 CONTENTS Preface vii Audience vii Organization vii Conventi ons viii Root and Non-root Con ventions in Examples ix Related Docum.
Contents iv Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01 Subinterfaces 3-2 Creating a Subinterface Associated with a Specific IB Partition 3-3 Removing a Subinterface Associated.
Contents v Cisco SFS InfiniBand Host Dr ivers User Guide for Linux OL-12309-01 CHAPTER 7 MVAPICH MPI 7-1 Introduction 7-1 Initial Setup 7-2 Configuring SSH 7-2 Editing Environment Variables 7-5 Settin.
Contents vi Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01.
vii Cisco SFS InfiniBand Host Dr ivers User Guide for Linux OL-12309-01 Preface This preface describes who shoul d read the Cisco SFS Infi niBand Host Drivers User Guide fo r Linux , ho w it is or ganized, and it s document con v entions.
viii Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01 Preface Conventions Conventions This document uses the f ollo wing con ventions: Chapter 8 HCA Utilities and Diag nostics Describes the fundamen tal HCA utilities and diagnostics.
ix Cisco SFS InfiniBand Host Dr ivers User Guide for Linux OL-12309-01 Preface Root and Non-root Conventions in E xamples Notes use the follo wing con vention: Note Means reader take note . Notes contain helpful suggestions or references to material not co vered in the manual.
x Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01 Preface Obtaining Documentation, Obtaining Support, and Security G uidelines.
CH A P T E R 1-1 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 1 About Host Drivers This chapter describes host driv ers and includes the following sections: • Introduction, pag.
1-2 Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01 Chapter 1 Abo ut Host Drivers Architecture Note See the “Root and Non-root Con v entions in Examples” section on page ix for details about the significance of prompt s used in the e xamples in this chapter .
1-3 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 1 About Hos t Drivers Supported Protocols Supported Protocols This section des cribes the suppo rted protocols an d includes the fol lowing topics: • IPoIB • SRP • SDP Protocol here refers to software in the networking layer in kernel space.
1-4 Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01 Chapter 1 Abo ut Host Drivers Supported APIs Supported APIs This section descri bes the supported APIs and includes t he follo wing topics: • MV APICH MPI • uD APL • Intel MPI • HP MPI API refers to software in the network ing layer in user space.
CH A P T E R 2-1 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 2 Installing Host Drivers The chapter includes th e following sections: • Introduction, page 2-1 • Contents of I.
2-2 Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01 Chapter 2 Installing Host Drivers Contents of ISO Image Contents of ISO Image The ISO image contains the follo wing directori es and files: • docs/ This directory contains the related documents.
2-3 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 2 In stalling Host Drivers Uninstalling Host Drivers from an ISO Image topspin-ib-mpi-rhel4-3.2.0-136.x86_64 (MPI libraries, source code, docs, etc) topspin-ib-mod-rhel4-2.
2-4 Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01 Chapter 2 Installing Host Drivers Uninstalling Host Drivers from an ISO Image.
CH A P T E R 3-1 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 3 IP over IB Protocol This chapter describe s IP over IB protocol and includes the following sections : • Introduc.
3-2 Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01 Chapter 3 IP over IB Protocol Manually Configuring IPoI B for Default IB Part ition Manually Configuring IPoI B for Default IB Partition T o manually conf igure IPoIB for the def ault IB partition, perform th e follo wing steps: Step 1 Log in to your Linu x host.
3-3 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 3 IP over IB Protocol Subinterfaces Creating a Subinterface Associated with a Specific IB Partition T o create a subinterface associat ed with a specific IB partition , perform the follo wing steps: Step 1 Create a partition on an IB SFS.
3-4 Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01 Chapter 3 IP over IB Protocol Subinterface s NOARP MTU:1480 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.
3-5 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 3 IP over IB Protocol Verifying IPoIB Functionality TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1024 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) lo Link encap:Local Loopback inet addr:127.
3-6 Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01 Chapter 3 IP over IB Protocol IPoIB Performa nce IPoIB Performance This section describes ho w to verify IPoIB performance by ru nning the Bandwidth test and the Latenc y test. These tests are described in de tail at the fo llowing URL: http://www .
3-7 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 3 IP over IB Protocol IPoIB Performance Step 5 Run the Netperf Latenc y test.
3-8 Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01 Chapter 3 IP over IB Protocol Sample Startup Configuration File Sample Startup Configuration File IP addresses that are con figur ed manually are not pe rsistent across reboots. Y ou must use a conf iguration file to config ure IPoIB when the host boots.
3-9 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 3 IP over IB Protocol IPoIB High Availability Step 3 T ake the i nterfaces of fline.
3-10 Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01 Chapter 3 IP over IB Protocol IPoIB High Availability Step 3 Display the a vai lable interfaces b y entering the ipoibcfg list command.
CH A P T E R 4-1 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 4 SCSI RDMA Protocol This chapter describe s SCSI RDMA protocol and includes the following sections: • Introductio.
4-2 Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01 Chapter 4 SCSI RDMA Protocol Configuring SRP This section contains informati on on how to co nfigu re your IB fabric to connect .
4-3 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 4 SCSI RDMA Protocol Configuring SRP Step 3 Bring up the Fibre Channel gate ways on yo ur SFS, b y performing the follo wing steps: a. Launch Eleme nt Manager . b. Double-click the Fibre Chan nel gate way card that you w ant to bring up.
4-4 Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01 Chapter 4 SCSI RDMA Protocol Configuring SRP Configuring ITLs with Element Manager wh ile Global Policy Restrictions Apply This section descri bes how to conf igure ITLs with Element Manager while global po licy restrictions apply .
4-5 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 4 SCSI RDMA Protocol Configuring SRP Step 9 Click the Next > b utton. The Def ine Ne w SRP Host windo w displays a recommend ed WWNN for the host and recommended WWPNs that represent the host on all exist ing and potential Fibre Channel gatew ay ports.
4-6 Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01 Chapter 4 SCSI RDMA Protocol Configuring SRP Configuring SRP Host This section descri bes how to configu re the SRP host. The SRP host dri ver exposes a Fibre Chan nel target (identified by a WWPN) as a SCS I target to the Linux SCSI mid-layer .
4-7 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 4 SCSI RDMA Protocol Verifying SRP Verifying SRP This section describes h ow t o verify SRP f unctional ity and veri fy S.
4-8 Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01 Chapter 4 SCSI RDMA Protocol Verifying SRP 512000 inodes, 1023996 blocks 51199 blocks (5.
CH A P T E R 5-1 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 5 Sockets Direct Protocol This chapter describes the Sock ets Direct Protocol and in cludes the follo wing sections : • Introduction, page 5-1 • Config uring IPoIB Interf aces, page 5-1 .
5-2 Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01 Chapter 5 Soc kets Direct Protocol Converting Sockets-Based Application Converting Sockets-Based Application This section describes ho w to co n ver t sockets-based applications.
5-3 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 5 Sockets Direct Protoco l Converting Sockets-Based Application Log Statement This section describes the lo g statement. The log di rect iv e allo ws the u ser to specif y which deb ug and error messages are sent and where they are se nt.
5-4 Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01 Chapter 5 Soc kets Direct Protocol SDP Performance The ip_port matches against an IP address, prefix length, and port range.
5-5 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 5 Sockets Direct Protoco l SDP Performance Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs.
5-6 Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01 Chapter 5 Soc kets Direct Protocol Netperf Server with IPoIB and SDP The notable pe rformance v alues in the e xample above are as follo ws: Client CPU utilizat ion is 6.26 percent of client CPU.
5-7 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 5 Sockets Direct Protoco l Netperf Server with IPo IB and SDP The follo wing list describes paramet ers for the netperf command: The notable pe rformance v alues in the e xample above are as follo ws: Throughput is 6.
5-8 Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01 Chapter 5 Soc kets Direct Protocol Netperf Server with IPoIB and SDP.
CH A P T E R 6-1 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 6 uDAPL This chapter describe s uD APL and includes the follo wing sections: • Introduction, page 6-1 • uD APL T.
6-2 Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01 Chapter 6 uDAPL uDAPL Throughput Test Performance The Throughput test measures RDMA WRITE th roughput using uD APL. T o perform a uD APL Throughput test perfor mance, perform the follo wing step s: Step 1 Start the Throu ghput test on the serv er host.
6-3 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 6 uDAPL Step 3 V iew the Throughput test results from the serv er . The follo wing e xample sho ws the Throughput test results: Created an EP with ep_handle = 0x2a95f8a300 queried max_recv_dtos = 256 queried max_request_dtos = 1024 Accept issued.
6-4 Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01 Chapter 6 uDAPL host2$ /usr/local/topspin/bin/lat_client.x ib0 192.168.0.1 200000 1 0 • ib0 is the name of the device. • 192.168.0.1 is the IPoIB address of the server host. • 200000 is the number of RDMAs to p erform for the test.
CH A P T E R 7-1 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 7 MVAPICH MPI The chapter describes MV APICH MPI and i ncludes the follo wing sections: • Introduction, page 7-1 .
7-2 Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01 Chapter 7 MVAPICH MPI Initial Setup Initial Setup This section describes the initial MPI setup. MPI can be used with either IPoIB or Ethernet IP addresses. The dri vers for MPI are aut omatically loaded at boot time if IPoIB or SDP is loaded.
7-3 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 7 MVAPICH MPI Configuring SSH T o conf igure SSH, perform t he follo wing steps: Step 1 Log in to the host th at you want to con figure as the local host, host1.
7-4 Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01 Chapter 7 MVAPICH MPI Configuring SSH Step 5 Change into the .ssh di rectory that you created. The follo wing e xample sho ws ho w to change into t he .ssh directory: host1$ cd .ssh Step 6 Copy the public key that w as just generated to the authorized keys file .
7-5 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 7 MVAPICH MPI Editing Environment Variab les Step 9 Return to host1 and cop y the authorized k eys f ile from Step 6 to the directory that you created in Ste p 8 .
7-6 Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01 Chapter 7 MVAPICH MPI Editing Environment Variables Setting Environment Variables in System-Wide Startup Files This method is used to set a system-wide def ault for which MPI implementation is used.
7-7 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 7 MVAPICH MPI MPI Bandwidth Test Performa nce Editing Environment Variables Manually T ypically , you edit en viro nment va riables manually when it is necessary to run temporarily with a gi v en MPI implementation.
7-8 Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01 Chapter 7 MVAPICH MPI MPI Latency Test Per formance When the test completes successfull y , you see output that is similar to the follo wing: # OSU MPI Bandwidth Test (Version 2.2) # Size Bandwidth (MB/s) 1 3.
7-9 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 7 MVAPICH MPI Intel MPI Benchmarks (IM B) Test Performance • The name of the hostf ile • The latency ex ecutable name.
7-10 Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01 Chapter 7 MVAPICH MPI Intel MPI Benchmar ks (IMB) Test Performance When your installation is no t working prop erly , the IMB test might lead to V API_RETR Y_EXEC errors.
7-11 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 7 MVAPICH MPI Intel MPI Benchmarks (IM B) Test Performance When the test completes su ccessfully , you see output similar to the follo wing: #--------------------------------------------------- # Intel (R) MPI Benchmark Suite V2.
7-12 Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01 Chapter 7 MVAPICH MPI Compiling MPI Programs Compiling MPI Programs This section descri bes how to compile MPI progra ms. Compi ling MPI applications from source code requires adding se ve ral compiler and link er flags.
7-13 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 7 MVAPICH MPI Compiling MPI Programs Step 3 Select the language and compiler of your choice from the sele ction of com piler wrapper s av ailable in Ta b l e 7 - 2 . Step 4 Compile the e xamples as sho wn here: host1$ cd $HOME/mpi-examples host1$ mpicc.
7-14 Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01 Chapter 7 MVAPICH MPI Compiling MPI Programs.
CH A P T E R 8-1 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 8 HCA Utilities and Diagnostics This chapte r describes the HC A utilities and diagnostics and includes t he follo w.
8-2 Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01 Chapter 8 HCA Utilities and Diagnostics hca_self_test Utility Host Driver Initialization ............. PASS Number of HCA Ports Active ............. 2 Port State of Port #0 on HCA #0 .
8-3 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 8 HCA Utilities and Diagnostics tvflash Utility tvflash Utility This section descri bes the tvflash utility and includes .
8-4 Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01 Chapter 8 HCA Utilities and Diagnostics tvflash Utility Upgrading Firmware T o upgrade f irmware on your ho st, perform the follo wing steps: Note Upon installation of the host driv ers, the f irmware is auto matically updated, if required.
8-5 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Chapter 8 HCA Utilities and Diagnostics Diagnostics Diagnostics This section includes diagnosti cs information. A fe w diagnost ic programs are included with the Linux IB host dri vers.
8-6 Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01 Chapter 8 HCA Utilities and Diagnostics Diagnostics.
A-1 Cisco SFS InfiniBand Host Dr ivers User Guide for Linux OL-12309-01 APPENDIX A Acronyms and Abbreviations Ta b l e A - 1 defines t he acronyms and ab bre viations that are u sed in this guide.
A-2 Cisco SFS InfiniBand Host Drivers User Guide for Lin ux OL-12309-01 Appendix A Acronyms an d Abbreviations SSH Secure Shell Protocol TCP T ransmission Cont rol Protocol uD APL User Direct Access P.
IN-1 Cisco SFS InfiniBand Host Dr ivers User Guide for Linux OL-12309-01 INDEX A architecture, HCA supported 1-2 audience vii authenticity messages 7-5 B Bandwidth test default 3-6, 5-7 MPI 7-7 Netperf 5-6 using 3-6 C card type, view 8-3 CLI 4-2 command-line interface.
Index IN-2 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 Gateway, Fibre Cha nnel 4-1 Globally Uni que Identifier. Se e GUID. global policy r estrictions 4-2, 4-4 GNU compiler 7-1 graphical u ser interface .
Index IN-3 Cisco SFS InfiniBand Host Dr ivers User Guide for Linux OL-12309-01 MPI Bandwidth test 7-7 compile programs 7-12 description 1-4, 7-1 Intel Benchmarks test 7-9 Latency test 7-8 MVAPICH 1-4,.
Index IN-4 Cisco SFS InfiniBand Host Drivers User Guide for Linux OL-12309-01 log 5-3 match 5-3 storage area network. See SA N. stream sockets networking 5-1 subinterface create 3-3 description 3-2 remove 3-4 T test Bandwidth 3-6 Bandwidth, default 3-6, 5-7 Bandwidth, MPI 7-7 Bandwidth, with SDP 5-6 IMB 7-9 Intel MPI Benchmarks.
An important point after buying a device Cisco Systems GEM318P (or even before the purchase) is to read its user manual. We should do this for several simple reasons:
If you have not bought Cisco Systems GEM318P yet, this is a good time to familiarize yourself with the basic data on the product. First of all view first pages of the manual, you can find above. You should find there the most important technical data Cisco Systems GEM318P - thus you can check whether the hardware meets your expectations. When delving into next pages of the user manual, Cisco Systems GEM318P you will learn all the available features of the product, as well as information on its operation. The information that you get Cisco Systems GEM318P will certainly help you make a decision on the purchase.
If you already are a holder of Cisco Systems GEM318P, but have not read the manual yet, you should do it for the reasons described above. You will learn then if you properly used the available features, and whether you have not made any mistakes, which can shorten the lifetime Cisco Systems GEM318P.
However, one of the most important roles played by the user manual is to help in solving problems with Cisco Systems GEM318P. Almost always you will find there Troubleshooting, which are the most frequently occurring failures and malfunctions of the device Cisco Systems GEM318P along with tips on how to solve them. Even if you fail to solve the problem, the manual will show you a further procedure – contact to the customer service center or the nearest service center
