This document provides various symptoms and error messages that might appear when you configure Lights-Out-Management (LOM), and how to troubleshoot them step by step. LOM allows you to use an out-of-band Serial over LAN (SOL) management connection in order to remotely monitor or manage appliances without logging into the web interface of the appliance. You can perform limited tasks, such as view the chassis serial number or monitor such conditions as fan speed and temperature.
Cisco recommends that you have knowledge of FireSIGHT System and LOM.
The information in this document is based on these hardware and software versions:
FireSIGHT Management Center
FirePOWER 7000 Series Appliances, 8000 Series Appliances
Software Version 5.2 or later
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
Unable to Connect to LOM
You might be unable to connect to a FireSIGHT Management Center or FirePOWER Appliance with LOM. The connection requests might fail with these error messages:
Error: Unable to establish IPMI v2 / RMCP+ session Error
Info: cannot activate SOL payload with encryption
The next section describes how to verify a LOM configuration and connections to the LOM interface.
Step 1: Verify and confirm that LOM is enabled and uses a different IP address than the management interface.
Step 2: Verify with the Network team that UDP port 623 is open bidirectionally, and that the routes are configured correctly. Since LOM works over a UDP port, you cannot Telnet to the LOM IP address over port 623. However, an an alternate solution is to test if the device speaks IPMI with the IPMIPING utility. IPMIPING sends two IPMI Get Channel Authentication Capabilities calls via a Get Channel Authentication Capabilities request datagram on UDP port 623 (two requests since it uses UDP and connections are not guaranteed.)
Note: For a more extensive test to confirm if the device listens on UDP port 623, use NMAP scan.
Step 3: Can you ping the IP address of LOM? If not, run this command as root user on the applicable appliance, and verify the settings are correct. For example,
ipmitool lan print
Set in Progress : Set Complete
Auth Type Support : NONE MD5 PASSWORD
Auth Type Enable : Callback : NONE MD5 PASSWORD
: User : NONE MD5 PASSWORD
: Operator : NONE MD5 PASSWORD
: Admin : NONE MD5 PASSWORD
: OEM :
IP Address Source : Static Address
IP Address : 192.0.2.2
Subnet Mask : 255.255.255.0
MAC Address : 00:1e:67:0a:24:32
SNMP Community String : INTEL
IP Header : TTL=0x00 Flags=0x00 Precedence=0x00 TOS=0x00
BMC ARP Control : ARP Responses Enabled, Gratuitous ARP Disabled
Gratituous ARP Intrvl : 0.0 seconds
Default Gateway IP : 192.0.2.1
Default Gateway MAC : 00:00:00:00:00:00
Backup Gateway IP : 0.0.0.0
Backup Gateway MAC : 00:00:00:00:00:00
802.1q VLAN ID : Disabled
802.1q VLAN Priority : 0
RMCP+ Cipher Suites : 1,2,3,6,7,8,11,12,0
Cipher Suite Priv Max : XaaaXXaaaXXaaXX
: X=Cipher Suite Unused
Error: Unable to establish IPMI v2 / RMCP+ session
Note: A connection to the correct IP address, but with the wrong credentials, fails with the previous error immediately. Attempts to connect to LOM at an invalid IP address time out after about 10 seconds and returns this error.
Step 8: Disable the LOM in the GUI, then reboot the appliance. In the appliance's GUI, choose Local > Configuration > Console Configuration. Select VGA, click Save, and click OK in order to reboot.
Afterwards, enable the LOM in the GUI, then reboot the appliance. In the appliance's GUI, choose Local > Configuration > ConsoleConfiguration. Choose Physical Serial Port or LOM, click Save, and click OK to reboot.
Step 9: Shut down the device and complete a power cycle, that is, physically remove the power cable for one minute, plug it back, and then power on. After the appliance powers up fully run this command:
Step 10: Run this command from the appliance in question. This specifically does a cold reset of the bmc:
ipmitool bmc reset cold
Step 11: Run this command from a system on the same local network as the device (that is, does not pass through any intermediate router):
ipmitool -I lanplus -H xxx.xxx.xxx.xxx -U admin power status
arp -an > /var/tmp/arpcache
Send Cisco Technical Support the resulting /var/tmp/arpcache file in order to determine if the BMC responds to an ARP request.
Connection to LOM Interface is Disconnected During Reboot
When you reboot a FireSIGHT Management Center or a FirePOWER Appliance, the connection to the appliance might be lost. The output when rebooting the appliance via the CLI is shown here:
admin@FireSIGHT:~$ sudo shutdown -r now
Broadcast message from root (ttyS0) (Tue Nov 19 19:40:30 Stopping Sourcefire 3D Sensor 7120...nfemsg: Host ID 1 on card 0 endpoint 1 de-registering ...
nfemsg: Host ID 2 on card 0 endpoint 1 de-registering ...
nfemsg: Host ID 27 on card 0 endpoint 1 de-registering ......ok
Stopping Netronome Flow Manager: nfemsg: Fail callback unregistered
Unregistered NFM fail hook handler
nfemsg: Card 0 Endpoint #1 messaging disabled
nfemsg: Module EXIT
WARNING: Deprecanfp nfp.0: [ME] CSR access problem for ME 25
ted config file nfp nfp.0: [vPCI] Removed virtual device 01:00.4
/etc/modprobe.conf, all config files belong into /etc/modprobe.d/. success.
No NMSB present: logging unecessary...[-10G[ OK ]..
Turning off swapfile /Volume/.swaptwo [-10G[ OK ] other currently mounted file systems... Unmounting fuse control filesystem.
The highlighted output Unmounting fuse control filesystem. Un shows that the connection to the appliance is interrupted due to Spanning Tree Protocol (STP) being enabled on the switch where the FireSIGHT System is connected to. Once the managed devices reboots, this error is displayed:
Error sending SOL data; FAIL
SOL session closed by BMC
Note: Before you can connect to an appliance with LOM/SOL, you must disable Spanning Tree Protocol (STP) on any third-party switching equipment connected to the device’s management interface.
A LOM connection of FireSIGHT System is shared with the management port. The link for the management port drops for a very brief time during reboot. Since the link is going down and coming back up, this could trigger a delay in the switch port (typically 30 seconds before it starts passing traffic) due to the listening or learning switch port state caused by having STP configured on the port.