Configuring the Fabric Interconnects

This chapter includes the following sections:

Initial System Setup

The first time that you access a fabric interconnect in a Cisco UCS domain, a setup wizard prompts you for the following information required to configure the system:

  • Installation method (GUI or CLI)

  • Setup mode (restore from full system backup or initial setup)

  • System configuration type (standalone or cluster configuration)

  • System name

  • Admin password

  • Management port IP address and subnet mask

  • Default gateway IP address

  • DNS Server IP address

  • Default domain name

Setup Mode

You can choose to either restore the system configuration from an existing backup file, or manually set up the system by going through the Setup wizard. If you choose to restore the system, the backup file must be reachable from the management network.

System Configuration Type

You can configure a Cisco UCS domain to use a single fabric interconnect in a standalone configuration or to use a redundant pair of fabric interconnects in a cluster configuration.

A cluster configuration provides high availability. If one fabric interconnect becomes unavailable, the other takes over. Only one management port (Mgmt0) connection is required to support a cluster configuration; however, both Mgmt0 ports should be connected to provide link-level redundancy.

In addition, a cluster configuration actively enhances failover recovery time for redundant virtual interface (VIF) connections. When an adapter has an active VIF connection to one fabric interconnect and a standby VIF connection to the second, the learned MAC addresses of the active VIF are replicated but not installed on the second fabric interconnect. If the active VIF fails, the second fabric interconnect installs the replicated MAC addresses and broadcasts them to the network through gratuitous ARP messages, shortening the switchover time.


Note

The cluster configuration provides redundancy only for the management plane. Data redundancy is dependent on the user configuration and might require a third-party tool to support data redundancy.

To use the cluster configuration, you must directly connect the two fabric interconnects together using Ethernet cables between the L1 (L1-to-L1) and L2 (L2-to-L2) high-availability ports, with no other fabric interconnects in between. Also you can connect the fabric interconnects directly through a patch panel to allow the two fabric interconnects to continuously monitor the status of each other and quickly know when one has failed.

Both fabric interconnects in a cluster configuration must go through the initial setup process. You must enable the first fabric interconnect that you set up for a cluster configuration. When you set up the second fabric interconnect, it detects the first fabric interconnect as a peer fabric interconnect in the cluster.

For more information, see to the Cisco UCS 6100 Series Fabric Interconnect Hardware Installation Guide.

Management Port IP Address

In a standalone configuration, you must specify only one IP address and the subnet mask for the single management port on the fabric interconnect.

In a cluster configuration, you must specify the following three IP addresses in the same subnet:

  • Management port IP address for fabric interconnect A

  • Management port IP address for fabric interconnect B

  • Cluster IP address

Performing an Initial System Setup for a Standalone Configuration

Before You Begin

  1. Verify the following physical connections on the fabric interconnect:

    • The console port is physically connected to a computer terminal or console server

    • The management Ethernet port (mgmt0) is connected to an external hub, switch, or router

    For more information, refer to the Cisco UCS Hardware Installation Guide for your fabric interconnect.

  2. Verify that the console port parameters on the computer terminal (or console server) attached to the console port are as follows:

    • 9600 baud

    • 8 data bits

    • No parity

    • 1 stop bit

  3. Collect the following information that you will need to supply during the initial setup:

    • System name.

    • Password for the admin account. Choose a strong password that meets the guidelines for Cisco UCS Manager passwords. This password cannot be blank.

    • Management port IP address and subnet mask.

    • Default gateway IP address.

    • DNS server IP address (optional).

    • Domain name for the system (optional).

Procedure
    Step 1   Connect to the console port.
    Step 2   Power on the fabric interconnect.

    You will see the power on self-test messages as the fabric interconnect boots.

    Step 3   At the installation method prompt, enter gui.
    Step 4   If the system cannot access a DHCP server, you are prompted to enter the following information:

    • IP address for the management port on the fabric interconnect

    • Subnet mask for the management port on the fabric interconnect

    • IP address for the default gateway assigned to the fabric interconnect

    Step 5   Copy the web link from the prompt into a supported web browser and go to the Cisco UCS Manager GUI launch page.
    Step 6   On the Cisco UCS Manager GUI launch page, select Express Setup.
    Step 7   On the Express Setup page, select Initial Setup and click Submit.
    Step 8   In the Cluster and Fabric Setup Area, select the Standalone Mode option.
    Step 9   In the System Setup Area, complete the following fields:

    Field

    Description

    System Name field

    The name assigned to the Cisco UCS domain.

    In a standalone configuration, the system adds "-A" to the system name. In a cluster configuration, the system adds "-A" to the fabric interconnect assigned to fabric A, and "-B" to the fabric interconnect assigned to fabric B.

    Admin Password field

    The password used for the Admin account on the fabric interconnect.

    Choose a strong password that meets the guidelines for Cisco UCS Manager passwords. This password cannot be blank.

    Confirm Admin Password field

    The password used for the Admin account on the fabric interconnect.

    Mgmt IP Address field

    The static IP address for the management port on the fabric interconnect.

    Mgmt IP Netmask field

    The subnet mask for the management port on the fabric interconnect.

    Default Gateway field

    The IP address for the default gateway assigned to the management port on the fabric interconnect.

    DNS Server IP field

    The IP address for the DNS server assigned to the fabric interconnect.

    Domain Name field

    The name of the domain in which the fabric interconnect resides.

    Step 10   Click Submit.

    A page displays the results of your setup operation.

    Initial System Setup for a Cluster Configuration

    Performing an Initial System Setup on the First Fabric Interconnect

    Before You Begin

    1. Verify the following physical connections on the fabric interconnect:

      • A console port on the first fabric interconnect is physically connected to a computer terminal or console server

      • The management Ethernet port (mgmt0) is connected to an external hub, switch, or router

      • The L1 ports on both fabric interconnects are directly connected to each other

      • The L2 ports on both fabric interconnects are directly connected to each other

      For more information, refer to the Cisco UCS Hardware Installation Guide for your fabric interconnect.

    2. Verify that the console port parameters on the computer terminal (or console server) attached to the console port are as follows:

      • 9600 baud

      • 8 data bits

      • No parity

      • 1 stop bit

    3. Collect the following information that you will need to supply during the initial setup:

      • System name.

      • Password for the admin account. Choose a strong password that meets the guidelines for Cisco UCS Manager passwords. This password cannot be blank.

      • Three static IP addresses: two for the management port on both fabric interconnects (one per fabric interconnect) and one for the cluster IP address used by Cisco UCS Manager.

      • Subnet mask for the three static IP addresses.

      • Default gateway IP address.

      • DNS server IP address (optional).

      • Domain name for the system (optional).

    Procedure
      Step 1   Connect to the console port.
      Step 2   Power on the fabric interconnect.

      You will see the power on self-test messages as the fabric interconnect boots.

      Step 3   At the installation method prompt, enter gui.
      Step 4   If the system cannot access a DHCP server, you are prompted to enter the following information:

      • IP address for the management port on the fabric interconnect

      • Subnet mask for the management port on the fabric interconnect

      • IP address for the default gateway assigned to the fabric interconnect

      Step 5   Copy the web link from the prompt into a web browser and go to the Cisco UCS Manager GUI launch page.
      Step 6   On the Cisco UCS Manager GUI launch page, select Express Setup.
      Step 7   On the Express Setup page, select Initial Setup and click Submit.
      Step 8   In the Cluster and Fabric Setup Area:
      1. Click the Enable Clustering option.
      2. For the Fabric Setup option, select Fabric A.
      3. In the Cluster IP Address field, enter the IP address that Cisco UCS Manager will use.
      Step 9   In the System Setup Area, complete the following fields:

      Field

      Description

      System Name field

      The name assigned to the Cisco UCS domain.

      In a standalone configuration, the system adds "-A" to the system name. In a cluster configuration, the system adds "-A" to the fabric interconnect assigned to fabric A, and "-B" to the fabric interconnect assigned to fabric B.

      Admin Password field

      The password used for the Admin account on the fabric interconnect.

      Choose a strong password that meets the guidelines for Cisco UCS Manager passwords. This password cannot be blank.

      Confirm Admin Password field

      The password used for the Admin account on the fabric interconnect.

      Mgmt IP Address field

      The static IP address for the management port on the fabric interconnect.

      Mgmt IP Netmask field

      The subnet mask for the management port on the fabric interconnect.

      Default Gateway field

      The IP address for the default gateway assigned to the management port on the fabric interconnect.

      DNS Server IP field

      The IP address for the DNS server assigned to the fabric interconnect.

      Domain Name field

      The name of the domain in which the fabric interconnect resides.

      Step 10   Click Submit.

      A page displays the results of your setup operation.

      Performing an Initial System Setup on the Second Fabric Interconnect

      Before You Begin

      You must ensure the following:

      • A console port on the second fabric interconnect is physically connected to a computer terminal or console server

      • You know the password for the admin account on the first fabric interconnect that you configured.

      Procedure
        Step 1   Connect to the console port.
        Step 2   Power on the fabric interconnect.

        You will see the power on self-test messages as the fabric interconnect boots.

        Step 3   At the installation method prompt, enter gui.
        Step 4   If the system cannot access a DHCP server, you are prompted to enter the following information:

        • IP address for the management port on the fabric interconnect

        • Subnet mask for the management port on the fabric interconnect

        • IP address for the default gateway assigned to the fabric interconnect

        Step 5   Copy the web link from the prompt into a web browser and go to the Cisco UCS Manager GUI launch page.
        Step 6   On the Cisco UCS Manager GUI launch page, select Express Setup.
        Step 7   On the Express Setup page, select Initial Setup and click Submit.

        The fabric interconnect should detect the configuration information for the first fabric interconnect.

        Step 8   In the Cluster and Fabric Setup Area:
        1. Select the Enable Clustering option.
        2. For the Fabric Setup option, make sure Fabric B is selected.
        Step 9   In the System Setup Area, enter the password for the Admin account into the Admin Password of Master field.
        Step 10   Click Submit.

        A page displays the results of your setup operation.

        Enabling a Standalone Fabric Interconnect for Cluster Configuration

        You can add a second fabric interconnect to an existing Cisco UCS domain that uses a single standalone fabric interconnect. To do this, you must enable the standalone fabric interconnect for cluster operation by configuring it with the virtual IP address of the cluster, and then add the second fabric interconnect to the cluster.

        Procedure
           Command or ActionPurpose
          Step 1UCS-A# connect local-mgmt  

          Enters local management mode.

           
          Step 2UCS-A(local-mgmt) # enable cluster virtual-ip-addr  

          Enables cluster operation on the standalone fabric interconnect with the specified IP address. When you enter this command, you are prompted to confirm that you want to enable cluster operation. Type yes to confirm.

          The IP address must be the virtual IP address for the cluster configuration, not the IP address assigned to the fabric interconnect that you are adding to the cluster.

           

          The following example enables a standalone fabric interconnect with a virtual IP address of 192.168.1.101 for cluster operation: 

          UCS-A# connect local-mgmt
UCS-A(local-mgmt)# enable cluster 192.168.1.101
This command will enable cluster mode on this setup. You cannot change it 
back to stand-alone. Are you sure you want to continue? (yes/no): yes
UCS-A(local-mgmt)#
          What to Do Next

          Add the second fabric interconnect to the cluster.

          Ethernet Switching Mode

          The Ethernet switching mode determines how the fabric interconnect behaves as a switching device between the servers and the network. The fabric interconnect operates in either of the following Ethernet switching modes:

          End-Host Mode

          End-host mode allows the fabric interconnect to act as an end host to the network, representing all server (hosts) connected to it through vNICs. This behavior is achieved by pinning (either dynamically pinned or hard pinned) vNICs to uplink ports, which provides redundancy to the network, and makes the uplink ports appear as server ports to the rest of the fabric. In end-host mode, the fabric interconnect does not run the Spanning Tree Protocol (STP) but it avoids loops by denying uplink ports from forwarding traffic to each other and by denying egress server traffic on more than one uplink port at a time. End-host mode is the default Ethernet switching mode and should be used if either of the following are used upstream:

          • Layer 2 switching for Layer 2 aggregation

          • Virtual Switching System (VSS) aggregation layer


          Note
          When you enable end-host mode, if a vNIC is hard pinned to an uplink port and this uplink port goes down, the system cannot repin the vNIC, and the vNIC remains down.

          Switch Mode

          Switch mode is the traditional Ethernet switching mode. The fabric interconnect runs STP to avoid loops, and broadcast and multicast packets are handled in the traditional way. Switch mode is not the default Ethernet switching mode, and should be used only if the fabric interconnect is directly connected to a router, or if either of the following are used upstream:

          • Layer 3 aggregation

          • VLAN in a box


          Note

          For both Ethernet switching modes, even when vNICs are hard pinned to uplink ports, all server-to-server unicast traffic in the server array is sent only through the fabric interconnect and is never sent through uplink ports. Server-to-server multicast and broadcast traffic is sent through all uplink ports in the same VLAN.

          Configuring Ethernet Switching Mode

          Important:

          When you change the Ethernet switching mode, Cisco UCS Manager logs you out and restarts the fabric interconnect. For a cluster configuration, Cisco UCS Manager restarts both fabric interconnects. The second fabric interconnect can take several minutes to complete the change in Ethernet switching mode and become system ready. The configuration is retained.

          While the fabric interconnects are rebooting, all blade servers will lose all LAN and SAN connectivity, causing a complete outage of all services on the blades. This may cause the operating system to crash.

          Procedure
            Step 1   In the Navigation pane, click the Equipment tab.
            Step 2   On the Equipment tab, expand Equipment > Fabric Interconnects > Fabric_Interconnect_Name.
            Step 3   In the Work pane, click the General tab.
            Step 4   In the Actions area of the General tab, click one of the following links:

            • Set Ethernet Switching Mode

            • Set Ethernet End-Host Mode

            The link for the current mode is dimmed.

            Step 5   In the dialog box, click Yes.

            Cisco UCS Manager restarts the fabric interconnect, logs you out, and disconnects Cisco UCS Manager GUI.

            Fibre Channel Switching Mode

            The Fibre Channel switching mode determines how the fabric interconnect behaves as a switching device between the servers and storage devices. The fabric interconnect operates in either of the following Fibre Channel switching modes:

            End-Host Mode

            End-host mode allows the fabric interconnect to act as an end host to the connected fibre channel networks, representing all server (hosts) connected to it through virtual host bus adapters (vHBAs). This behavior is achieved by pinning (either dynamically pinned or hard pinned) vHBAs to Fibre Channel uplink ports, which makes the Fibre Channel ports appear as server ports (N-ports) to the rest of the fabric. When in end-host mode, the fabric interconnect avoids loops by denying uplink ports from receiving traffic from one another.

            End-host mode is synonymous with NPV mode. This mode is the default Fibre Channel Switching mode.


            Note

            When you enable end-host mode, if a vHBA is hard pinned to a uplink Fibre Channel port and this uplink port goes down, the system cannot repin the vHBA, and the vHBA remains down.

            Switch Mode

            Switch mode is the traditional Fibre Channel switching mode. Switch mode allows the fabric interconnect to connect directly to a storage device. Enabling Fibre Channel switch mode is useful in Pod models where there is no SAN (for example, a single Cisco UCS domain that is connected directly to storage), or where a SAN exists (with an upstream MDS).

            Switch mode is not the default Fibre Channel switching mode.


            Note

            In Fibre Channel switch mode, SAN pin groups are irrelevant. Any existing SAN pin groups are ignored.

            Configuring Fibre Channel Switching Mode

            Important:

            When you change the Fibre Channel switching mode, Cisco UCS Manager logs you out and restarts the fabric interconnect. For a cluster configuration, Cisco UCS Manager restarts both fabric interconnects sequentially. The second fabric interconnect can take several minutes to complete the change in Fibre Channel switching mode and become system ready.


            Note
            When the Fibre Channel switching mode is changed, both UCS fabric interconnects will reload simultaneously. Reloading of fabric interconnects will cause a system-wide downtime for approximately 10-15 minutes.
            Procedure
              Step 1   In the Navigation pane, click the Equipment tab.
              Step 2   On the Equipment tab, expand Equipment > Fabric Interconnects > Fabric_Interconnect_Name.
              Step 3   In the Work pane, click the General tab.
              Step 4   In the Actions area of the General tab, click one of the following links:

              • Set Fibre Channel Switching Mode

              • Set Fibre Channel End-Host Mode

              The link for the current mode is dimmed.

              Step 5   In the dialog box, click Yes.

              Cisco UCS Manager restarts the fabric interconnect, logs you out, and disconnects Cisco UCS Manager GUI.

              Changing the Properties of the Fabric Interconnects


              Note

              To change the subnet for a Cisco UCS domain, you must simultaneously change all subnets, the virtual IP address used to access Cisco UCS Manager, and the IP addresses for all fabric interconnects.

              Procedure
                Step 1   In the Navigation pane, click the Admin tab.
                Step 2   On the Admin tab, click All.
                Step 3   In the Work pane, click the General tab.
                Step 4   In the Actions area, click Management Interfaces to open the Management Interfaces dialog box.
                Step 5   In the Management Interfaces dialog box, modify the values in one or more of the following fields:
                Name Description

                IP Address field

                The IP address assigned to the Cisco UCS Manager GUI.

                Domain Name field

                The domain name assigned to this Cisco UCS domain.

                Enter up to 256 characters. You can use any characters or spaces except ` (accent mark), \ (backslash), ^ (carat), " (double quote), = (equal sign), > (greater than), < (less than), or ' (single quote).

                Name field

                The name assigned to this Cisco UCS domain.

                Enter an alphabetic character followed by up to 29 alphanumeric characters or hyphens (-).

                System Owner field

                The owner of the Cisco UCS domain.

                Enter up to 32 characters. You can use any characters or spaces except ` (accent mark), \ (backslash), ^ (carat), " (double quote), = (equal sign), > (greater than), < (less than), or ' (single quote).

                System Site field

                The site associated with the Cisco UCS domain.

                Enter up to 32 characters. You can use any characters or spaces except ` (accent mark), \ (backslash), ^ (carat), " (double quote), = (equal sign), > (greater than), < (less than), or ' (single quote).

                System Description field

                The description of the Cisco UCS domain.

                Enter up to 256 characters. You can use any characters or spaces except ` (accent mark), \ (backslash), ^ (carat), " (double quote), = (equal sign), > (greater than), < (less than), or ' (single quote).

                Mode field

                How this system is configured for high availability. This can be one of the following:

                • Cluster

                • Stand Alone

                Step 6   To change only the virtual IP address that you use to access Cisco UCS Manager, enter the desired IP address in the IP Address field in the Virtual IP area.
                Step 7   To change only the name assigned to the Cisco UCS domain, enter the desired name in the Name field in the Virtual IP area.
                Step 8   To change the subnet, IP address, and default gateway assigned to the fabric interconnects, update the following fields:
                1. In the Virtual IP area, change the IP address used to access Cisco UCS Manager in the IP Address field.
                2. In the Fabric Interconnect area for each fabric interconnect, update the following fields:
                  Name Description

                  IP Address field

                  The IP address to use when communicating with the fabric interconnect.

                  Subnet Mask field

                  The associated subnet mask.

                  Default Gateway field

                  The associated gateway.

                Step 9   Click OK.
                Step 10   Log out of Cisco UCS Manager GUI and log back in again to see your changes.

                Determining the Leadership Role of a Fabric Interconnect

                Procedure
                  Step 1   In the Navigation pane, click the Equipment tab.
                  Step 2   In the Equipment tab, expand Equipment > Fabric Interconnects.
                  Step 3   Click the fabric interconnect for which you want to identify the role.
                  Step 4   In the Work pane, click the General tab.
                  Step 5   In the General tab, click the down arrows on the High Availability Details bar to expand that area.
                  Step 6   View the Leadership field to determine whether the fabric interconnect is the primary or subordinate.