The first time that you access a fabric interconnect in a
Cisco UCS
instance, a setup wizard prompts you for the following information required to
configure the system:
Installation method (GUI or CLI)
Setup mode (restore from full system backup or initial setup)
System configuration type (standalone or cluster configuration)
You can choose to either restore the system configuration from an
existing backup file, or manually set up the system by going through the Setup
wizard. If you choose to restore the system, the backup file must be reachable
from the management network.
System Configuration Type
You can configure a
Cisco UCS
instance to use a single fabric interconnect in a standalone configuration or
to use a redundant pair of fabric interconnects in a cluster configuration.
A cluster configuration provides high availability. If one fabric
interconnect becomes unavailable, the other takes over.
Only one
management port (Mgmt0) connection is required to support a cluster
configuration; however, both Mgmt0 ports should be connected to provide
link-level redundancy.
In addition, a cluster configuration actively enhances failover recovery time for redundant virtual interface (VIF) connections. When an adapter has an active VIF connection to one fabric interconnect and a standby VIF connection to the second, the learned MAC addresses of the active VIF are replicated but not installed on the second fabric interconnect. If the active VIF fails, the second fabric interconnect installs the replicated MAC addresses and broadcasts them to the network through gratuitous ARP messages, shortening the switchover time.
Note
The cluster configuration provides redundancy only for the management
plane. Data redundancy is dependent on the user configuration and may require a
third-party tool to support data redundancy.
To use the cluster configuration, the two fabric interconnects must be
directly connected together using Ethernet cables between the L1 (L1-to-L1) and
L2 (L2-to-L2) high availability ports, with no other fabric interconnects in
between. This allows the two fabric interconnects to continuously monitor the
status of each other and quickly know when one has failed.
Both fabric interconnects in a cluster configuration must go through the
initial setup process. The first fabric interconnect to be set up must be
enabled for a cluster configuration. Then, when the second fabric interconnect
is set up, it detects the first fabric interconnect as a peer
fabric interconnect in the cluster.
For more information, refer to the
Cisco UCS 6100 Series Fabric Interconnect Hardware
Installation Guide.
Management Port IP Address
In a standalone configuration, you must specify only one IP address and
the subnet mask for the single management port on the fabric interconnect.
In a cluster configuration, you must specify the following three IP
addresses in the same subnet:
Management port IP address for fabric interconnect A
Management port IP address for fabric interconnect B
Cluster IP address
Performing an Initial System Setup for a Standalone
Configuration
Before You Begin
Verify the following physical connections on the fabric
interconnect:
The console port is physically connected to a computer
terminal or console server
The management Ethernet port (mgmt0) is connected to an
external hub, switch, or router
For more information, refer to the
Cisco UCS Hardware Installation
Guide
for your fabric interconnect.
Verify that the console port parameters on the computer terminal
(or console server) attached to the console port are as follows:
9600 baud
8 data bits
No parity
1 stop bit
Collect the following information that you will need to supply
during the initial setup:
System name.
Password for the admin account. Choose a strong password that meets the guidelines for Cisco UCS Manager passwords.
This password cannot be blank.
Management port IP address and subnet mask.
Default gateway IP address.
DNS server IP address (optional).
Domain name for the system (optional).
Procedure
Step 1
Connect to the console port.
Step 2
Power on the fabric interconnect.
You will see the power on self-test messages as the fabric
interconnect boots.
Step 3
When the unconfigured system boots, it prompts you for the setup
method to be used. Enter
console to continue the initial setup using the
console CLI.
Step 4
Enter
setup to continue as an initial system setup.
Step 5
Enter
y to confirm that you want to continue the initial
setup.
Step 6
Enter the password for the admin account.
Step 7
To confirm, re-enter the password for the admin account.
Step 8
Enter
no to continue the initial setup for a standalone
configuration.
Step 9
Enter the system name.
Step 10
Enter the IP address for the management port on the fabric
interconnect.
Step 11
Enter the subnet mask for the management port on the fabric
interconnect.
Step 12
Enter the IP address for the default gateway.
Step 13
Enter
yes if you want to specify the IP address for the
DNS server, or
no if you do not.
Step 14
(Optional)Enter the IP address for the DNS server.
Step 15
Enter
yes if you want to specify the default domain
name, or
no if you do not.
Step 16
(Optional)Enter the default domain name.
Step 17
Review the setup summary and enter
yes to save and apply the settings, or enter
no to go through the Setup wizard again to change
some of the settings.
If you choose to go through the Setup wizard again, it provides the values you previously entered, and the values
appear in brackets. To accept previously entered values, press Enter.
The following example sets up a standalone configuration using the
console:
Enter the installation method (console/gui)? console
Enter the setup mode (restore from backup or initial setup) [restore/setup]? setup
You have chosen to setup a new switch. Continue? (y/n): y
Enter the password for "admin": adminpassword%958
Confirm the password for "admin": adminpassword%958
Do you want to create a new cluster on this switch (select 'no' for standalone setup or if you want this switch to be added to an existing cluster)? (yes/no) [n]: no
Enter the system name: foo
Mgmt0 IPv4 address: 192.168.10.10
Mgmt0 IPv4 netmask: 255.255.255.0
IPv4 address of the default gateway: 192.168.10.1
Configure the DNS Server IPv4 address? (yes/no) [n]: yes
DNS IPv4 address: 20.10.20.10
Configure the default domain name? (yes/no) [n]: yes
Default domain name: domainname.com
Following configurations will be applied:
Switch Fabric=A
System Name=foo
Management IP Address=192.168.10.10
Management IP Netmask=255.255.255.0
Default Gateway=192.168.10.1
DNS Server=20.10.20.10
Domain Name=domainname.com
Apply and save the configuration (select 'no' if you want to re-enter)? (yes/no): yes
Initial System Setup for a Cluster Configuration
Performing an Initial System Setup for the First Fabric
Interconnect
Before You Begin
Verify the following physical connections on the fabric
interconnect:
A console port on the first fabric interconnect is physically
connected to a computer terminal or console server
The management Ethernet port (mgmt0) is connected to an
external hub, switch, or router
The L1 ports on both fabric interconnects are directly
connected to each other
The L2 ports on both fabric interconnects are directly
connected to each other
For more information, refer to the
Cisco UCS Hardware Installation
Guide
for your fabric interconnect.
Verify that the console port parameters on the computer terminal
(or console server) attached to the console port are as follows:
9600 baud
8 data bits
No parity
1 stop bit
Collect the following information that you will need to supply
during the initial setup:
System name.
Password for the admin account. Choose a strong password that meets the guidelines for Cisco UCS Manager passwords.
This password cannot be blank.
Three static IP addresses: two for the management port on both
fabric interconnects (one per fabric interconnect) and one for the cluster IP
address used by
Cisco UCS Manager.
Subnet mask for the three static IP addresses.
Default gateway IP address.
DNS server IP address (optional).
Domain name for the system (optional).
Procedure
Step 1
Connect to the console port.
Step 2
Power on the fabric interconnect.
You will see the power on self-test messages as the fabric
interconnect boots.
Step 3
When the unconfigured system boots, it prompts you for the setup
method to be used. Enter
console to continue the initial setup using the
console CLI.
Step 4
Enter
setup to continue as an initial system setup.
Step 5
Enter
y to confirm that you want to continue the initial
setup.
Step 6
Enter the password for the admin account.
Step 7
To confirm, re-enter the password for the admin account.
Step 8
Enter
yes to continue the initial setup for a cluster
configuration.
Step 9
Enter the fabric interconnect fabric (either
A or
B).
Step 10
Enter the system name.
Step 11
Enter the IP address for the management port on the fabric
interconnect.
Step 12
Enter the subnet mask for the management port on the fabric
interconnect.
Step 13
Enter the IP address for the default gateway.
Step 14
Enter the virtual IP address.
Step 15
Enter
yes if you want to specify the IP address for the
DNS server, or
no if you do not.
Step 16
(Optional)Enter the IP address for the DNS server.
Step 17
Enter
yes if you want to specify the default domain
name, or
no if you do not.
Step 18
(Optional)Enter the default domain name.
Step 19
Review the setup summary and enter
yes to save and apply the settings, or enter
no to go through the Setup wizard again to change
some of the settings.
If you choose to go through the Setup wizard again, it provides the values you previously entered, and the values appear in brackets. To accept previously entered values, press Enter.
The following example sets up the first fabric interconnect for a
cluster configuration using the console:
Enter the installation method (console/gui)? console
Enter the setup mode (restore from backup or initial setup) [restore/setup]? setup
You have chosen to setup a new switch. Continue? (y/n): y
Enter the password for "admin": adminpassword%958
Confirm the password for "admin": adminpassword%958
Do you want to create a new cluster on this switch (select 'no' for standalone setup or if you want this switch to be added to an existing cluster)? (yes/no) [n]: yes
Enter the switch fabric (A/B): A
Enter the system name: foo
Mgmt0 IPv4 address: 192.168.10.10
Mgmt0 IPv4 netmask: 255.255.255.0
IPv4 address of the default gateway: 192.168.10.1
Virtual IPv4 address : 192.168.10.12
Configure the DNS Server IPv4 address? (yes/no) [n]: yes
DNS IPv4 address: 20.10.20.10
Configure the default domain name? (yes/no) [n]: yes
Default domain name: domainname.com
Following configurations will be applied:
Switch Fabric=A
System Name=foo
Management IP Address=192.168.10.10
Management IP Netmask=255.255.255.0
Default Gateway=192.168.10.1
Cluster Enabled=yes
Virtual Ip Address=192.168.10.12
DNS Server=20.10.20.10
Domain Name=domainname.com
Apply and save the configuration (select 'no' if you want to re-enter)? (yes/no): yes
Performing an Initial System Setup for the Second Fabric
Interconnect
Before You Begin
Verify the following physical connections on the fabric
interconnect:
A console port on the second fabric interconnect is physically
connected to a computer terminal or console server
The management Ethernet port (mgmt0) is connected to an
external hub, switch, or router
The L1 ports on both fabric interconnects are directly
connected to each other
The L2 ports on both fabric interconnects are directly
connected to each other
For more information, refer to the
Cisco UCS Hardware Installation
Guide
for your fabric interconnect.
Verify that the console port parameters on the computer terminal
(or console server) attached to the console port are as follows:
9600 baud
8 data bits
No parity
1 stop bit
Collect the following information that you will need to supply
during the initial setup:
Password for the admin account of the peer fabric interconnect. Choose a strong password that meets the guidelines for Cisco UCS Manager passwords.
This password cannot be blank.
Management port IP address in the same subnet as the peer
fabric interconnect.
Procedure
Step 1
Connect to the console port.
Step 2
Power on the fabric interconnect.
You will see the power on self-test messages as the fabric
interconnect boots.
Step 3
When the unconfigured system boots, it prompts you for the setup
method to be used. Enter
console to continue the initial setup using the
console CLI.
Note
The fabric interconnect should detect the peer fabric
interconnect in the cluster. If it does not, check the physical connections
between the L1 and L2 ports, and verify that the peer fabric interconnect has
been enabled for a cluster configuration.
Step 4
Enter
y to add the subordinate fabric interconnect to
the cluster.
Step 5
Enter the admin password of the peer fabric interconnect.
Step 6
Enter the IP address for the management port on the subordinate
fabric interconnect.
Step 7
Review the setup summary and enter
yes to save and apply the settings, or enter
no to go through the Setup wizard again to change
some of the settings.
If you choose to go through the Setup wizard again, it provides the values you previously entered, and the values
appear in brackets. To accept previously entered values, press Enter.
The following example sets up the second fabric interconnect for a
cluster configuration using the console:
Enter the installation method (console/gui)? console
Installer has detected the presence of a peer switch. This switch will be added to the cluster. Continue?[y/n] y
Enter the admin password of the peer switch: adminpassword%958
Mgmt0 IPv4 address: 192.168.10.11
Management Ip Address=192.168.10.11
Apply and save the configuration (select 'no' if you want to re-enter)? (yes/no): yes
Enabling a Standalone Fabric Interconnect for Cluster
Configuration
You can add a second fabric interconnect to an existing
Cisco UCS
instance that uses a single standalone fabric interconnect. To do this, you
must enable the standalone fabric interconnect for cluster operation, and then
add the second fabric interconnect to the cluster.
Procedure
Command or Action
Purpose
Step 1
UCS-A#
connect local-mgmt
Enters local management mode.
Step 2
UCS-A(local-mgmt) #
enable clusterip-addr
Enables cluster operation on the standalone fabric interconnect
with the specified IP address. When you enter this command, you are prompted to
confirm that you want to enable cluster operation. Type yes to confirm.
The following example enables a standalone fabric interconnect with IP
address 192.168.1.101 for cluster operation:
UCS-A# connect local-mgmt
UCS-A(local-mgmt)# enable cluster 192.168.1.101
This command will enable cluster mode on this setup. You cannot change it
back to stand-alone. Are you sure you want to continue? (yes/no): yes
UCS-A(local-mgmt)#
What to Do Next
Add the second fabric interconnect to the cluster.
Changing the System Name
Procedure
Command or Action
Purpose
Step 1
UCS-A # scope system
Enters system mode.
Step 2
UCS-A /system #
set namename
Sets the system name.
Step 3
UCS-A /system #
commit-buffer
Commits the transaction to the system configuration.
The name is updated on both fabric interconnects within about 30 seconds after the transaction is committed.
The following example changes the system name and commits the transaction:
UCS-A# scope system
UCS-A /system* # set name SanJose5
UCS-A /system* # commit-buffer
UCS-A /system #
Changing the Management Subnet of a Cluster
When changing the management subnet in a cluster configuration, you must change the following three IP
addresses simultaneously and you must configure all three in the same subnet:
Management port IP address for fabric interconnect A
Management port IP address for fabric interconnect B
Cluster IP (virtual IP) address
Procedure
Command or Action
Purpose
Step 1
UCS-A#
scope fabric-interconnect a
Enters fabric interconnect mode for fabric A.
Step 2
UCS-A /fabric-interconnect #
set out-of-band ipip-addressnetmasknetmaskgwgateway-ip-address
Sets the IP address, netmask, and gateway IP address of the fabric interconnect.
Step 3
UCS-A /fabric-interconnect #
scope fabric-interconnect b
Enters fabric interconnect mode for fabric B.
Step 4
UCS-A /fabric-interconnect #
set out-of-band ipip-addressnetmasknetmaskgwgateway-ip-address
Sets the IP address, netmask, and gateway IP address of the fabric interconnect.
Step 5
UCS-A /fabric-interconnect # scope system
Enters system mode.
Step 6
UCS-A /system #
set virtual-ipvip-address
Sets the virtual IP address for the cluster.
Step 7
UCS-A /system #
commit-buffer
Commits the transaction to the system configuration.
When you commit the transaction, you are disconnected from the management session. Reconnect at the new management IP address.
This example changes both fabric-interconnect IP addresses, changes the virtual IP address, and commits the transaction, disconnecting the session:
UCS-A# scope fabric-interconnect a
UCS-A /fabric-interconnect # set out-of-band ip 192.0.2.111 netmask 255.255.255.0 gw 192.0.2.1
UCS-A /fabric-interconnect* # scope fabric-interconnect b
UCS-A /fabric-interconnect* # set out-of-band ip 192.0.2.112 netmask 255.255.255.0 gw 192.0.2.1
UCS-A /fabric-interconnect* # scope system
UCS-A /system* # set virtual-ip 192.0.2.113
UCS-A /system* # commit-buffer
Ethernet Switching Mode
The Ethernet switching mode determines how the fabric interconnect
behaves as a switching device between the servers and the network. The fabric
interconnect operates in either of the following Ethernet switching modes:
End-Host
Mode
End-host mode allows the fabric interconnect to act as an end host to
the network, representing all server (hosts) connected to it through vNICs.
This is achieved by pinning (either dynamically pinned or hard pinned) vNICs to
uplink ports, which provides redundancy toward the network, and makes the
uplink ports appear as server ports to the rest of the fabric. When in end-host
mode, the fabric interconnect does not run the Spanning Tree Protocol (STP) and
avoids loops by denying uplink ports from forwarding traffic to each other, and
by denying egress server traffic on more than one uplink port at a time.
End-host mode is the default Ethernet switching mode and should be used if
either of the following are used upstream:
Layer 2 switching for L2 aggregation
Virtual Switching System (VSS) aggregation layer
Note
When end-host mode is enabled, if a vNIC is hard pinned to an
uplink port and this uplink port goes down, the system cannot re-pin the vNIC,
and the vNIC remains down.
Switch Mode
Switch mode is the traditional Ethernet switching mode. The fabric
interconnect runs STP to avoid loops, and broadcast and multicast packets are
handled in the traditional way. Switch mode is not the default Ethernet
switching mode, and should be used only if the fabric interconnect is directly
connected to a router, or if either of the following are used upstream:
Layer 3 aggregation
VLAN in a box
Note
For both Ethernet switching modes, even when vNICs are hard pinned
to uplink ports, all server-to-server unicast traffic in the server array is
sent only through the fabric interconnect and is never sent through uplink
ports. Server-to-server multicast and broadcast traffic is sent through all
uplink ports in the same VLAN.
Configuring Ethernet Switching Mode
Important:
When you change the Ethernet switching mode,
Cisco UCS Manager
logs you out and restarts the fabric interconnect. For a cluster configuration,
Cisco UCS Manager
restarts both fabric interconnects sequentially. The second fabric interconnect can take several minutes to complete the change in Ethernet switching mode and become system ready.
Procedure
Command or Action
Purpose
Step 1
UCS-A#
scope eth-uplink
Enters Ethernet uplink mode.
Step 2
UCS-A /eth-uplink #
set mode {end-host |
switch}
Sets the fabric interconnect to the specified switching mode.
Step 3
UCS-A /eth-uplink #
commit-buffer
Commits the transaction to the system configuration.
Cisco UCS Manager
restarts the fabric interconnect, logs you out, and disconnects
Cisco UCS Manager CLI.
The following example sets the fabric interconnect to end-host mode and commits the transaction:
UCS-A# scope eth-uplink
UCS-A /eth-uplink # set mode end-host
Warning: When committed, this change will cause the switch to reboot
UCS-A /eth-uplink* # commit-buffer
UCS-A /eth-uplink #
Fibre Channel Switching Mode
The Fibre Channel switching mode determines how the fabric interconnect
behaves as a switching device between the servers and storage devices. The fabric
interconnect operates in either of the following Fibre Channel switching modes:
End-Host
Mode
End-host mode allows the fabric interconnect to act as an end host to
the connected fibre channel networks, representing all server (hosts) connected to it through vHBAs.
This is achieved by pinning (either dynamically pinned or hard pinned) vHBAs to Fibre Channel uplink ports, which makes the Fibre Channel ports appear as server ports (N-ports) to the rest of the fabric. When in end-host mode, the fabric interconnect avoids loops by denying uplink ports from receiving traffic from one another.
End-host mode is synonymous with NPV mode. This is the default Fibre Channel Switching mode.
Note
When end-host mode is enabled, if a vHBA is hard pinned to a
uplink Fibre Channel port and this uplink port goes down, the system cannot re-pin the vHBA,
and the vHBA remains down.
Switch Mode
Switch mode is the traditional Fibre Channel switching mode. Switch mode allows the fabric interconnect to connect directly to a storage device. Enabling Fibre Channel switch mode is useful in POD models where there is no SAN (for example, a single Cisco UCS system connected directly to storage), or where a SAN exists (with an upstream MDS).
Note
In Fibre Channel switch mode, SAN pin groups are irrelevant. Any existing SAN pin groups will be ignored.
Switch mode is not the default Fibre Channel switching mode. Enabling Fibre Channel switching mode requires a license.
Configuring Fibre Channel Switching Mode
Important:
When you change the Fibre Channel switching mode,
Cisco UCS Manager
logs you out and restarts the fabric interconnect. For a cluster configuration,
Cisco UCS Manager
restarts both fabric interconnects sequentially. The second fabric interconnect can take several minutes to complete the change in Fibre Channel switching mode and become system ready.
Procedure
Command or Action
Purpose
Step 1
UCS-A#
scope fc-uplink
Enters Fibre Channel uplink mode.
Step 2
UCS-A /fc-uplink #
set mode {end-host |
switch}
Sets the fabric interconnect to the specified switching mode.
Step 3
UCS-A /fc-uplink #
commit-buffer
Commits the transaction to the system configuration.
Cisco UCS Manager
restarts the fabric interconnect, logs you out, and disconnects
Cisco UCS Manager CLI.
The following example sets the fabric interconnect to end-host mode and commits the transaction: