Guest

Cisco UCS C-Series Rack-Mount Standalone Server Software

Cisco usNIC Deployment Guide for Cisco UCS C-Series Rack-Mount Standalone Servers, Release 1.5(2)

  • Viewing Options

  • PDF (1.3 MB)
  • EPUB (107.3 KB)
  • Feedback
Overview of Cisco usNIC

Overview of Cisco usNIC

The Cisco user-space NIC (usNIC) feature improves the performance of software applications that run on the Cisco UCS servers in your data center by bypassing the kernel when sending and receiving networking packets. The applications interact directly with the Cisco UCS VIC 1225 adapter, which improves the networking performance of your high-performance computing cluster. To benefit from Cisco usNIC, your applications must use the Message Passing Interface (MPI) instead of sockets or other communication APIs.

Cisco usNIC offers the following benefits for your MPI applications:

  • Provides a low-latency and high-throughput communication transport.
  • Employs the standard and application-independent Ethernet protocol.
  • Takes advantage of low-­latency forwarding, Unified Fabric, and integrated management support in the following Cisco data center platforms:
    • Cisco UCS server
    • Cisco UCS VIC 1225
    • Unified Fabric based on the Cisco Nexus switches
Standard Ethernet applications use user-space socket libraries, which invoke the networking stack in the Linux kernel. The networking stack then uses the Cisco Enic driver to communicate with the Cisco VIC hardware. The following figure shows the contrast between a regular software application and an MPI application that uses usNIC.
Figure 1. Kernel-Based Network Communication versus Cisco usNIC-Based Communication

Configuring Cisco usNIC

Before You Begin
Make sure that the following software and hardware components are installed on the Cisco UCS server:
  • Red Hat Enterprise Linux (RHEL) version 6.4 or later
  • RHEL gcc-c++ and gcc-gfortran RPMs
  • Cisco UCS VIC 1225
Procedure
    Step 1   Configure the Cisco usNIC properties and BIOS settings.

    To configure using the CIMC GUI, see Configuring Cisco usNIC Using the CIMC GUI

    Step 2   Enable the intel_iommu kernel boot command.

    You must enable this setting in the kernel boot command to allow user-space applications to properly communicate with the Cisco usNIC.

    Step 3   Reboot your Cisco UCS server.

    You must reboot your server for the changes to take effect after you configure Cisco usNIC.

    Step 4   Install the Cisco usNIC Linux drivers.

    For more information about installing the drivers, see Installing Linux Drivers for Cisco usNIC.


    What to Do Next

    After you complete configuring Cisco usNIC and installing the Linux drivers, verify that Cisco usNIC is functioning properly. For more information about how to verify the installation, see Verifying the Cisco usNIC Installation.

    Configuring Cisco usNIC Using the CIMC GUI


    Note


    Even though several properties are listed for Cisco usNIC in the usNIC properties dialog box, you must configure only the following properties because the other properties are not currently being used.
    • cq-count
    • rq-count
    • tq-count
    • usnic-count

    Before You Begin

    You must log in to the CIMC GUI with administrator privileges to perform this task.

    Procedure
      Step 1   Log into the CIMC GUI.

      For more information about how to log into CIMC, see the Cisco UCS C-Series Servers Integrated Management Controller GUI Configuration Guide available at this URL: http:/​/​www.cisco.com/​en/​US/​products/​ps10739/​products_​installation_​and_​configuration_​guides_​list.html .

      Step 2   In the Navigation pane, click the Server tab.
      Step 3   On the Server tab, click Inventory.
      Step 4   In the Inventory pane, click the Cisco VIC Adapters tab.
      Step 5   In the Adapter Cards area, select the adapter card.

      If the server is powered on, the resources of the selected adapter card appear in the tabbed menu below the Adapter Cards area.

      Step 6   In the tabbed menu below the Adapter Cards area, click the vNICs tab.
      Step 7   In the Host Ethernet Interfaces area, select a vNIC from the table.
      Step 8   In the Host Ethernet Interfaces area, select a vNIC from the table.
      Note    For each vNIC that you want to configure as a usNIC, select the vNIC entry from the table and specify its properties as explained in steps 9 through step 18.
      Step 9   Click usNIC to open the usNIC Properties dialog box.
      Step 10   In the usNICs property, specify the number of Cisco usNICs that you want to create.

      Each MPI process that is running on the server requires a dedicated usNIC. You might need to create up to 64 usNICs to sustain 64 MPI processes running simultaneously. We recommend that you create at least as many usNICs, per usNIC-enabled vNIC, as the number of physical cores on your server. For example, if you have 8 physical cores on your server, create 8 usNICs.

      Step 11   In the Properties area, update the following fields:
      Field Name Description

      Transmit Queue Count

      The number of transmit queue resources to allocate.

      MPI will use 2 transmit queues per process. Therefore, Cisco recommends that you set this value to 2.

      Receive Queue Count

      The number of receive queue resources to allocate.

      MPI will use 2 receive queues per process. Therefore, Cisco recommends that you set this value to 2.

      Completion Queue Count

      The number of completion queue resources to allocate. In general, the number of completion queue resources you should allocate is equal to the number of transmit queue resources plus the number of receive queue resources.

      Cisco recommends that you set this value to 4.

      Step 12   Click Apply.
      Step 13   In the Navigation pane, click the Server tab.
      Step 14   On the Server tab, click BIOS.
      Step 15   In the Actions area, click Configure BIOS.
      Step 16   In the Configure BIOS Parameters dialog box, click the Advanced tab.
      Step 17   In the Processor Configuration area, set the following properties to Enabled:
      • Intel(R) VT-d
      • Intel(R) VT-d ATS support
      • Intel(R) VT-d Coherency Support
      Step 18   Click Save Changes.

      The changes take effect upon the next server reboot.


      Creating Cisco usNIC Using the CIMC CLI


      Note


      Even though several properties are listed for Cisco usNIC in the usNIC properties dialog box, you must configure only the following properties because the other properties are not currently being used.
      • cq-count
      • rq-count
      • tq-count
      • usnic-count

      Before You Begin

      You must log in to the CIMC CLI with administrator privileges to perform this task.

      Procedure
          Command or Action Purpose
        Step 1 server# scope chassis  

        Enters chassis command mode.

         
        Step 2 server/chassis# scope adapter index 

        Enters the command mode for the adapter card at the PCI slot number specified by index.

        Note   

        Make sure that the server is powered on before you attempt to view or change adapter settings. To view the index of the adapters configured on you server, use the show adapter command.

         
        Step 3 server/chassis/adapter# scope host-eth-if {eth0 | eth1}  

        Enters the command mode for the vNIC. Specify the Ethernet ID based on the number of vNICs that you have configured in your environment. For example, specify eth0 if you configured only one vNIC.

         
        Step 4 server/chassis/adapter/host-eth-if# create usnic-config 0  

        Creates a usNIC config and enters its command mode. Make sure that you always set the index value to 0.

        Note   

        To create a Cisco usNIC for the first time for a given vNIC using the CIMC CLI, you must first create a usnic-config. Subsequently, you only need to scope into the usnic-config and modify the properties for Cisco usNIC. For more information about modifying Cisco usNIC properties, see Modifying a Cisco usNIC using the CIMC CLI.

         
        Step 5 server/chassis/adapter/host-eth-if/usnic-config# set cq-count count  

        Specifies the number of completion queue resources to allocate. We recommend that you set this value to 4.

        The number of completion queues equals the number of transmit queues plus the number of receive queues.

         
        Step 6 server/chassis/adapter/host-eth-if/usnic-config# set rq-count count  

        Specifies the number of receive queue resources to allocate. MPI uses 2 receive queues per process. We recommend that you set this value to 2.

         
        Step 7 server/chassis/adapter/host-eth-if/usnic-config# set tq-count count  

        Specifies the number of transmit queue resources to allocate. MPI uses 2 transmit queues per process. We recommend that you set this value to 2.

         
        Step 8 server/chassis/adapter/host-eth-if/usnic-config# set usnic-count number of usNICs . 

        Specifies the number of Cisco usNICs to create. Each MPI process that is running on the server requires a dedicated Cisco usNIC. Therefore, you might need to create up to 64 Cisco usNICs to sustain 64 MPI processes running simultaneously. We recommend that you create at least as many Cisco usNICs, per Cisco usNIC-enabled vNIC, as the number of physical cores on your server. For example, if you have 8 physical cores on your server, create 8 Cisco usNICs.

         
        Step 9 server/chassis/adapter/host-eth-if /usnic-config# commit  

        Commits the transaction to the system configuration.

        Note   

        The changes take effect when the server is rebooted.

         
        Step 10 server/chassis/adapter/host-eth-if/usnic-config# exit  

        Exits to host Ethernet interface command mode.

         
        Step 11 server/chassis/adapter/host-eth-if# exit  

        Exits to adapter interface command mode.

         
        Step 12 server/chassis/adapter# exit  

        Exits to chassis interface command mode.

         
        Step 13 server/chassis# exit  

        Exits to server interface command mode.

         
        Step 14 server# scope bios  

        Enters Bios command mode.

         
        Step 15 server/bios# scope advanced  

        Enters the advanced settings of BIOS command mode.

         
        Step 16 server/bios/advanced# set IntelVTD Enabled  

        Enables the Intel Virtualization Technology.

         
        Step 17 server/bios/advanced# set ATS Enabled  

        Enables the Intel VT-d Address Translation Services (ATS) support for the processor.

         
        Step 18 server/bios/advanced# set CoherencySupport Enabled  

        Enables Intel VT-d coherency support for the processor.

         
        Step 19 server /bios/advanced# commit  

        Commits the transaction to the system configuration.

        Note   

        The changes take effect when the server is rebooted.

         

        This example shows how to configure Cisco usNIC properties:

        Server # scope chassis
        server /chassis # show adapter
        server /chassis # scope adapter 2
        server /chassis/adapter # scope host-eth-if eth0
        server /chassis/adapter/host-eth-if # create usnic-config 0
        server /chassis/adapter/host-eth-if/usnic-config *# set usnic-count 64
        server /chassis/adapter/host-eth-if/usnic-config *# set cq-count 4
        server /chassis/adapter/host-eth-if/usnic-config *# set rq-count 2
        server /chassis/adapter/host-eth-if/usnic-config *# set tq-count 2
        server /chassis/adapter/host-eth-if/usnic-config *# commit
        Committed settings will take effect upon the next server reset
        server /chassis/adapter/host-eth-if/usnic-config # exit
        server /chassis/adapter/host-eth-if # exit
        server /chassis/adapter # exit
        server /chassis # exit
        server # exit
        server/bios # scope bios
        server/bios/advanced # scope advanced
        server/bios/advanced # set Intel VTD Enabled
        server/bios/advanced # set ATS Enabled
        server/bios/advanced # set CoherencySupport Enabled
        server /chassis/adapter/host-eth-if/usnic-config # commit
        Committed settings will take effect upon the next server reset
        
        

        Modifying a Cisco usNIC using the CIMC CLI


        Note


        Even though several properties are listed for Cisco usNIC in the usNIC properties dialog box, you must configure only the following properties because the other properties are not currently being used.
        • cq-count
        • rq-count
        • tq-count
        • usnic-count

        Before You Begin

        You must log in to the CIMC GUI with administrator privileges to perform this task.

        Procedure
            Command or Action Purpose
          Step 1 server# scope chassis  

          Enters chassis command mode.

           
          Step 2 server/chassis# scope adapter index 

          Enters the command mode for the adapter card at the PCI slot number specified by index.

          Note   

          Make sure that the server is powered on before you attempt to view or change adapter settings. To view the index of the adapters configured on you server, use the show adapter command.

           
          Step 3 server/chassis/adapter# scope host-eth-if {eth0 | eth1}  

          Enters the command mode for the vNIC. Specify the Ethernet ID based on the number of vNICs that you have configured in your environment. For example, specify eth0 if you configured only one vNIC.

           
          Step 4 server/chassis/adapter/host-eth-if# scope usnic-config 0  

          Enters the command mode for the usNIC. Make sure that you always set the index value as 0 to configure a Cisco usNIC.

           
          Step 5 server/chassis/adapter/host-eth-if/usnic-config# set cq-count count  

          The number of completion queue resources to allocate. We recommend that you set this value to 4.

          The number of completion queues equals the number of transmit queues plus the number of receive queues.

           
          Step 6 server/chassis/adapter/host-eth-if/usnic-config# set rq-count count  

          Specifies the number of receive queue resources to allocate. MPI uses two receive queues per process. We recommend that you set this value to 2.

           
          Step 7 server/chassis/adapter/host-eth-if/usnic-config# set tq-count count  

          Specifies the number of transmit queue resources to allocate. MPI uses two transmit queues per process. We recommend that you set this value to 2.

           
          Step 8 server/chassis/adapter/host-eth-if/usnic-config# set usnic-count number of usNICs . 

          Specifies the number of Cisco usNICs to create. Each MPI process running on the server requires a dedicated Cisco usNIC. Therefore, you might need to create up to 64 Cisco usNIC to sustain 64 MPI processes running simultaneously. We recommend that you create at least as many Cisco usNIC, per Cisco usNIC-enabled vNIC, as the number of physical cores on your server. For example, if you have 8 physical cores on your server, create 8 usNICs.

           
          Step 9 server /chassis/adapter/host-eth-if /usnic-config# commit  

          Commits the transaction to the system configuration.

          Note   

          The changes take effect when the server is rebooted.

           
          Step 10 server/chassis/adapter/host-eth-if/usnic-config# exit  

          Exits to host Ethernet interface command mode.

           
          Step 11 server/chassis/adapter/host-eth-if# exit  

          Exits to adapter interface command mode.

           
          Step 12 server/chassis/adapter# exit  

          Exits to chassis interface command mode.

           
          Step 13 server/chassis# exit  

          Exits to server interface command mode.

           
          Step 14 server# scope bios  

          Enters BIOS command mode.

           
          Step 15 server/bios# scope advanced  

          Enters the advanced settings of BIOS command mode.

           
          Step 16 server/bios/advanced# set IntelVTD Enabled  

          Enables the Intel Virtualization Technology.

           
          Step 17 server/bios/advanced# set ATS Enabled  

          Enables the Intel VT-d Address Translation Services (ATS) support for the processor.

           
          Step 18 server/bios/advanced# set CoherencySupport Enabled  

          Enables Intel VT-d coherency support for the processor.

           
          Step 19 server/bios/advanced# commit  

          Commits the transaction to the system configuration.

          Note   

          The changes take effect when the server is rebooted.

           

          This example shows how to configure Cisco usNIC properties:

          server # scope chassis
          server /chassis # show adapter
          server /chassis # scope adapter 2
          server /chassis/adapter # scope host-eth-if eth0
          server /chassis/adapter/host-eth-if # scope usnic-config 0
          server /chassis/adapter/host-eth-if/usnic-config # set usnic-count 64
          server /chassis/adapter/host-eth-if/usnic-config # set cq-count 4
          server /chassis/adapter/host-eth-if/usnic-config # set rq-count 2
          server /chassis/adapter/host-eth-if/usnic-config # set tq-count 2
          server /chassis/adapter/host-eth-if/usnic-config # commit
          Committed settings will take effect upon the next server reset
          server /chassis/adapter/host-eth-if/usnic-config # exit
          server /chassis/adapter/host-eth-if # exit
          server /chassis/adapter # exit
          server /chassis # exit
          server # exit
          server/bios # scope bios
          server/bios/advanced # scope advanced
          server/bios/advanced # set Intel VTD Enabled
          server/bios/advanced # set ATS Enabled
          server/bios/advanced # set CoherencySupport Enabled
          server /chassis/adapter/host-eth-if/usnic-config # commit
          Committed settings will take effect upon the next server reset
          
          

          Deleting Cisco usNIC from a vNIC

          Before You Begin

          You must log in to CIMC CLI with admin privileges to perform this task.

          Procedure
              Command or Action Purpose
            Step 1 server# scope chassis  

            Enters chassis command mode.

             
            Step 2 server/chassis# scope adapter index 

            Enters the command mode for the adapter card at the PCI slot number specified by index.

            Note   

            Make sure that the server is powered on before you attempt to view or change adapter settings. To view the index of the adapters configured on you server, use the show adapter command.

             
            Step 3 server/chassis/adapter# scope host-eth-if {eth0 | eth1}  

            Enters the command mode for the vNIC. Specify the Ethernet ID based on the number of vNICs that you have configured in your environment. For example, specify eth0 if you configured only one vNIC.

             
            Step 4 Server/chassis/adapter/host-eth-if# delete usnic-config 0  

            Deletes the Cisco usNIC configuration for the vNIC.

             
            Step 5 Server/chassis/adapter/host-eth-if# commit 

            Commits the transaction to the system configuration

            Note   

            The changes take effect when the server is rebooted.

             

            This example shows how to delete the Cisco usNIC configuration for a vNIC:

            server # scope chassis
            server/chassis # show adapter
            server/chassis # scope adapter 1
            server/chassis/adapter # scope host-eth-if eth0
            Server/chassis/adapter/host-eth-if # delete usnic-config 0
            server/adapter/host-eth-if/iscsi-boot *# commit
            New host-eth-if settings will take effect upon the next adapter reboot
            
            server /adapter/host-eth-if/usnic-config #
            
            

            Installing Linux Drivers for Cisco usNIC

            The following section lists the contents of the Cisco usNIC build folder that is included in the UCS Drivers ISO bundle. Documentation about known issues and installation instructions (README) for Cisco usNIC are also included in the build folder.


            Note


            To prevent the OS from swapping out the memory that is allocated to the usNIC applications, the installation software increases the locked memory system setting of the OS to Unlimited.


            • kmod-usnic_verbs-1.0.0.77-1.x86_64.rpm—Linux kernel verbs driver for the usNIC feature of the Cisco VIC SR-IOV Ethernet NIC.
            • libusnic_usnic_verbs-1.0.0.77-1.x86_64.rpm—User space library libibverbs plugin for usNIC.
            • kmod-enic-2.1.1.47-rhel6u4.el6.x86_64.rpm—Linux kernel driver for Enic.
            • openmpi-cisco-1.6.5cisco1.0.0.77-1.x86 _64.rpmCisco usNIC Open MPI—Open MPI with the Cisco usNIC BTL MPI transport.
            • usnic_tools-1.0.0.77-1.x86_64.rpm—Utility programs for usNIC.
            • usnic_installer.sh—Scripts for installing the usNIC packages listed in the section.
            • usnic_uninstaller.sh—Script for uninstalling the usNIC packages listed in the section.
            Before You Begin

            Make sure that you have configured the Cisco usNIC properties in the Cisco Integrated Management Controller (CIMC). For more information about how to configure the properties, see Configuring Cisco usNIC.

            You must also make sure that the Cisco UCS server on which you want to install Cisco usNIC has a supported version of the Cisco Enic driver installed. The Cisco Enic driver is the Linux kernel networking driver for the Cisco VIC SR-IOV Ethernet NIC.
            Procedure
              Step 1   #./usnic_installer.sh

              Execute the installer script from the directory where the installation files are located for Cisco usNIC.

              Note   

              You require admin privileges to execute the script at the root (#) prompt.

              Step 2   # chkconfig rdma on

              Verify that the rdma service is started. This service is required for the usnic_verbs kernel module.

              Step 3   Reboot your server for the installation changes to take effect.

              If you do not want to reboot your server, you can manually load the kernel modules. For more information about how to load the modules, see Manually Loading the Kernel Modules for Cisco usNIC.


              Manually Loading the Kernel Modules for Cisco usNIC

              If you do not want to reboot your server, you can manually load the Cisco usNIC kernel modules by using the following steps.

              Procedure
                  Command or Action Purpose
                Step 1 # rmmod enic
                 

                Unloads the existing Enic driver module.

                Note   

                Make sure that you are not logged into the OS using the network, for example, via ssh. Otherwise, your network connection might get permanently disconnected. Alternatively, you can log in to the server using the CIMC KVM to perform this step.

                 
                Step 2 # modprobe enic
                 

                Loads the Enic driver module.

                 
                Step 3 # modprobe usnic_verbs
                 

                Loads the usnic_verbs driver module.

                 

                Uninstalling Linux Drivers for Cisco usNIC

                Procedure
                  Step 1   # /opt/cisco/usnic/bin/usnic_uninstaller.sh

                  Execute the uninstaller script from the bin folder in the directory where the Cisco usNIC installation files are located.

                  Step 2   Reboot your Cisco UCS server.

                  You must reboot the server for the changes to take effect.


                  Verifying the Cisco usNIC Installation

                  After you install the required Linux drivers for Cisco usNIC, perform the following procedure at the Linux prompt to make sure that the installation completed successfully.

                  Procedure
                    Step 1   Search for and list all kernel modules that have usnic_verbs in their names to verify that the kernel module was loaded during the OS driver installation.
                    $ lsmod | grep usnic_verbs

                    The following details are displayed when you enter the $ lsmod | grep usnic_verbs command. The kernel modules listed on your console may differ based on the modules that you have currently loaded in your OS.

                    usnic_verbs            78776  0
                    ib_core                73747  13 ib_ipoib,rdma_ucm,ib_ucm,ib_uverbs,ib_umad,rdma_cm,ib_cm,iw_cm,ib_sa,ib_mad,
                    iw_cxgb4,iw_cxgb3,usnic_verbs
                    enic                   67741  1 usnic_verbs
                    
                    Step 2   View the configuration of Cisco usNIC-enabled NICs.
                    $ ibv_devinfo
                    The following section is a brief example of the results that are displayed when you execute the ibv_devinfo command. The results may differ based on your current installations. When the results are displayed on your console, make sure that the state for each of the listed ports are shown as PORT_ACTIVE. The following example shows two ports ( usnic_1 and usnic_0) that are configured on a Cisco UCS VIC 1225 adapter. If you configured only one usNIC-enables vNIC, you will see a listing for only usnic_0.
                    Note   

                    The ibv_devinfo command displays the value for the transport parameter as iWARP. However, Cisco usNIC does not utilize the iWARP transport.

                    hca_id: usnic_1
                            transport:                      iWARP (1)
                            node_guid:                      0200:00ff:fe00:0000
                            sys_image_guid:                 2657:20ff:fe6f:1100
                            vendor_id:                      0x1137
                            vendor_part_id:                 207
                            hw_ver:                         0x2
                            board_id:                       133
                            phys_port_cnt:                  1
                                    port:   1
                                            state:                  PORT_ACTIVE (4)
                                            max_mtu:                4096 (5)
                                            active_mtu:             4096 (5)
                                            sm_lid:                 0
                                            port_lid:               0
                                            port_lmc:               0x01
                                            link_layer:             Ethernet
                    
                    hca_id: usnic_0
                            transport:                      iWARP (1)
                            node_guid:                      0200:00ff:fe00:0000
                            sys_image_guid:                 2657:20ff:fe6f:1000
                            vendor_id:                      0x1137
                            vendor_part_id:                 207
                            hw_ver:                         0x2
                            board_id:                       133
                            phys_port_cnt:                  1
                                    port:   1
                                            state:                  PORT_ACTIVE (4)
                                            max_mtu:                4096 (5)
                                            active_mtu:             4096 (5)
                                            sm_lid:                 0
                                            port_lid:               0
                                            port_lmc:               0x01
                                            link_layer:             Ethernet
                    
                    Step 3   Verify that the rdma service is started. This service is required for loading the usnic_verbs kernel module.
                    chkconfig - -level 2345 rdma on
                    Step 4   Run the usnic_verbs_check script to view the installed RPMs and their versions.
                    $ /opt/cisco/usnic/bin/usnic_verbs_check
                    

                    If any errors occurred during the OS driver installation, warnings are generated.

                    If the usnic_verbs module failed to load, the following brief example shows the warnings that are generated:
                    $ sudo rmmod usnic_verbs
                    $ /opt/cisco/usnic/bin/usnic_verbs_check
                    enic RPM version 2.1.1.46-2 installed
                    usnic_verbs RPM version 1.0.0.58-1 installed
                    WARNING: usnic_verbs module not loaded
                    libusnic_verbs RPM version 1.0.0.58-1 installed
                    Open MPI RPM version 1.6.5cisco1.0.0.58-1 installed
                    WARNING: No usnic verbs devices found
                    WARNING: No usnic verbs devices found
                    3 warnings
                    
                    Step 5   Verify that the Cisco usNIC network packets are being transmitted correctly between the client and server hosts:
                    1. Determine the name of the Ethernet interface associated with the Cisco usNIC on the server host.
                      <server>$ cat /sys/class/infiniband/usnic_0/config | grep eth
                      usnic_0: 0000:0b:0.0, eth4, fc:99:47:25:42:bd, 64 VFs
                    2. Determine the IP address for the Ethernet interface.
                      <server>$ ip addr show dev eth4 | grep "inet[^6]"
                          inet 10.1.0.1/16 brd 10.1.255.255 scope global eth4
                    3. Run the ibv_ud_pingpong program on the server host.
                      <server>$ ibv_ud_pingpong -g 0 -d usnic_0 -s 200
                      Set the -g flag, which specifies the GID, to 0 for usNIC. For more information about the command line options used with the ibv_ud_pingpong program, see the ibv_ud_pingpong(1) man pages.
                    4. Execute the ibv_ud_pingpong program on the client host by using the IP address that corresponds to the usNIC on the server host.
                      <client>$ ibv_ud_pingpong -g 0 -d usnic_0 -s 200 10.1.0.1
                    The following example shows the results that are displayed when you run the ibv_ud_pingpong program.
                    Server-side:
                    <server>$ ibv_ud_pingpong -g 0 -d usnic_0 -s 200
                      local address:  LID 0x0000, QPN 0x000001, PSN 0x38ffcb: GID fe80::2657:20ff:fe6f:1000
                      remote address: LID 0x0000, QPN 0x000001, PSN 0x3a9e22, GID fe80::b2fa:ebff:fe72:84fd
                      400000 bytes in 0.01 seconds = 637.70 Mbit/sec
                      1000 iters in 0.01 seconds = 5.02 usec/iter
                    
                    
                    Client-side:
                    <client>$ ibv_ud_pingpong -g 0 -d usnic_0 -s 200 10.1.0.1
                      local address:  LID 0x0000, QPN 0x000001, PSN 0x3a9e22: GID fe80::b2fa:eb ff:fe72:84fd
                      remote address: LID 0x0000, QPN 0x000001, PSN 0x38ffcb, GID fe80::2657:20 ff:fe6f:1000
                      400000 bytes in 0.01 seconds = 617.40 Mbit/sec
                      1000 iters in 0.01 seconds = 5.18 usec/iter
                    
                    
                    Step 6   Download, compile, and execute the ring_c test program to validate that the MPI traffic is correctly transmitted between the client and server hosts.

                    You can obtain the ring_c test program from this link: http:/​/​svn.open-mpi.org/​svn/​ompi/​branches/​v1.6/​examples/​ring_​c.c.

                    The following example shows how to use the wget utility to obtain, compile, and execute the ring_c. Alternatively, you can use other methods of obtaining and running the test program.
                    $ wget http://svn.open-mpi.org/svn/ompi/branches/v1.6/examples/ring_c.c
                     --2013-08-06 15:27:33--  http://svn.open-mpi.org/svn/ompi/branches/v1.6/examples/ring_c.c
                     Resolving svn.open-mpi.org... 129.79.13.24 Connecting to 
                     svn.open-mpi.org|129.79.13.24|:80... connected.
                     HTTP request sent, awaiting response... 200 OK
                     Length: 2418 (2.4K) [text/plain]
                     Saving to: “ring_c.c”
                     
                     100%[======================================>] 2,418       --.-K/s   in 0s      
                     
                     Last-modified header invalid -- time-stamp ignored.
                     2013-08-06 15:27:33 (10.7 MB/s) - “ring_c.c” saved [2418/2418]
                     
                     $ mpicc ring_c.c -o ring_c
                     [no output]
                     
                     $ mpiexec --host host1,host2 -n 4 ./ring_c 
                     Process 0 sending 10 to 1, tag 201 (4 processes in ring) 
                     Process 0 sent to 1 
                     Process 0 decremented value: 9 
                     Process 0 decremented value: 8 
                     Process 0 decremented value: 7 
                     Process 0 decremented value: 6 
                     Process 0 decremented value: 5 
                     Process 0 decremented value: 4 
                     Process 0 decremented value: 3 
                     Process 0 decremented value: 2 
                     Process 0 decremented value: 1 
                     Process 0 decremented value: 0 
                     Process 0 exiting 
                     Process 2 exiting 
                     Process 1 exiting 
                     Process 3 exiting ...  
                     
                    
                    

                    If the ibv_ud_pingpong program and the ring_c program executed successfully, you should now be able to run MPI applications over usNIC.