Troubleshoot PoE Imax Errors on Catalyst 3650/3850 Switches

Introduction

This document describes how to troubleshoot Power over Ethernet (PoE) Imax errors on Catalyst 3650/3850 switches.

Background Information

PoE is used by Catalyst 3650/3850 switches in order to provide power to external devices such as Wireless Access Points (APs), IP phones, and so on via the Ethernet cable that attaches them to the switch.

What are Imax Errors

An Imax error occurs when a PoE capable port on the switch draws more power than it negotiated. When an IEEE Powered Device (PD) comes up, it gets classified into a class. Dependent upon what class a device is in, it is allocated a certain amount of Watts by the switch that acts as the Power Source Equipment (PSE). This can be renegotiated later by the device that uses Cisco Discovery Protocol (CDP) or Link Layer Discovery Protocol (LLDP) to request more or less power. This is to allow budgeting of power.

The PD ensures it does not draw more power than it is allocated. The switch controls this by setting an Icutoff value. This is the value that gets set on the controller as the high mark. When a device exceeds the Icutoff value, the switch stops supplying power and logs an Imax error which indicates the attached device exceeded the negotiated wattage.

Comparison to Older Devices

The Catalyst 3650/3850 utilizes a more enhanced PoE controller. Where older devices like the Catalyst 3750 do not support much granularity with regards to setting Icutoff values, the Catalyst 3650 and 3850 do. This often leads to a perception that the Catalyst 3650/3850 experiences issues that the older devices do not. In almost all cases, however, this is just a perception. The older devices have less granularity in policing of the power and allow a PD to draw more power than negotiated. The Catalyst 3650/3850 does police the drawn power more strictly, and as such, Imax errors can occur on Catalyst 3650/3850 where a connection of the same device to an older switch would not show any problem.

Troubleshoot Imax Errors

A determination of how much power a PD really draws in the field is not very easy. When the power controller on the switch detects there is more power being drawn on a port, it shuts the port down and notifies Cisco IOS® of the fact that the PD has exceeded the maximum allocated power. In Cisco IOS, you can see the currently drawn power usage per port with the show power inline <interface> detail command.

3850_4#sh power inline Te 3/0/44 detail
 Interface: Te3/0/44
 Inline Power Mode: auto
 Operational status: on
 Device Detected: yes
 Device Type: Ieee PD
 IEEE Class: 3
 Discovery mechanism used/configured: Ieee and Cisco
 Police: off
 Power Allocated 
 Admin Value: 60.0
 Power drawn from the source: 15.0
 Power available to the device: 15.0
 Actual consumption
 Measured at the port: 6.1
 Maximum Power drawn by the device since powered on: 6.2
 Absent Counter: 0
 Over Current Counter: 0
 Short Current Counter: 0
 Invalid Signature Counter: 0
 Power Denied Counter: 0
 Power Negotiation Used: IEEE 802.3at LLDP
 LLDP Power Negotiation --Sent to PD--      --Rcvd from PD--
   Power Type:          Type 2 PSE           Type 1 PD
   Power Source:        Primary              PSE
   Power Priority:      low                  high
   Requested Power(W):  12.7                 12.7
   Allocated Power(W):  12.7                 12.7
Four-Pair PoE Supported: Yes
Spare Pair Power Enabled: No
Four-Pair PD Architecture: Shared

The measured value shown at the port in this output is measured by the controller. This information is gathered every few seconds and gives some indication about the drawn power. The value shown with Maximum Power drawn appears useful to troubleshoot Imax errors, but unfortunately, that is just a historical display of what the Maximum power drawn by the device has been. If an Imax error occurs, the Power drawn at that time is not reported back to Cisco IOS and cannot be displayed there.

As can be seen in the example, the value allocated to the port is 15W. This is the cutoff value that gets programmed onto the interface. Prior to Cisco bug ID CSCuy7423, the Icutoff value is programmed regularly on a port. Every time a CDP packet is received the value can be reprogrammed. After Cisco bug ID CSCuy74231 (fixed in Cisco IOS XE 3.6.5E and 3.7.5 or later) this programming has been optimized. This reduces the possibility of a failure in reprogramming the Icutoff value which leads to an Imax error.

Programming of Icutoff value can be shown via two commands. Either via the trace where the log can be gathered historically or a debug can be enabled to log a debug message when it occurs. The commands to get this are:

show mgmt-infra trace message platform-mgr-poe  <switch x>
debug platform poe

The show trace command can only be executed if the active switch in the stack is PoE capable. Otherwise, this command is needed in order to first connect to the PoE member switch in the stack to execute it:

session switch <x>
*May 20 00:34:04.445:CDP-PA: Packet received from AP2 on interface TenGigabitEthernet3/0/44
**Entry  found in cache**
*May 20 00:34:04.445: %IOSXE-7-PLATFORM: MEMBER: 3 process platform_mgr: PoE Info: Dequeued POE SPI msg ver 1 if_id 73003723793629284
 num_ports 1 req_id 650 msg_type 20
*May 20 00:34:04.452: %IOSXE-7-PLATFORM: MEMBER: 3 process platform_mgr: PoE Info: E_ILP_SET_CUTOFF if_id 73003723793629284
*May 20 00:34:04.452: %IOSXE-7-PLATFORM: MEMBER: 3 process platform_mgr: PoE Info:port 44 icutoff power 15000
*May 20 00:34:04.452: %IOSXE-7-PLATFORM: MEMBER: 3 process platform_mgr: PoE Info: re_poe_set_icutoff_current port 44 power 15000
*May 20 00:34:04.452: %IOSXE-7-PLATFORM: MEMBER: 3 process platform_mgr: PoE Info: scale factor 22 for power 15000
*May 20 00:34:04.452: %IOSXE-7-PLATFORM: MEMBER: 3 process platform_mgr: PoE Info: POE_SET_CUTOFF_CURRENT_SCALE_FACTOR sent
 for port 44 (e:11)

As mentioned earlier, it is a complex process to diagnose Imax errors. There is not much information logged at the time an Imax error occurs. The controller shuts the port down and the PD would have typically lost all logs in regards to what it was doing at the time it drew more power than allocated. Measurement of the drawn power by a port in the field is not easy, but with static allocated power a determination could be made. By statically allocating more power than would be requested dynamically, it is possible to determine how much more power the PD would draw that would trigger the Icutoff threshold to be exceeded. A static maximum power consumption can be configured on a switch port with the command power inline static max <value>.

3850_4#sh run int te 3/0/44
interface TenGigabitEthernet3/0/44
  power inline static max 20000
end

3850_4#sh power inline te 3/0/44 detail 
Interface: Te3/0/44  
Inline Power Mode: static  
Operational status: on  
Device Detected: yes  
Device Type: Ieee PD
IEEE Class: 3  
Discovery mechanism used/configured: Ieee and Cisco  
Police: off
Power Allocated  Admin Value: 20.0  
Power drawn from the source: 20.0
Power available to the device: 20.0

Power Negotiation

Various IEEE classes have defined levels of power usage. Further negotiation of power is done between the PD and the PSE with either CDP or LLDP. Power negotiation plays an important part when you look at Imax errors. A PD requests how much power can be allocated to it, but it also can ensure that it cannot exceed the requested value.

Class                   PSE           PD 

Class 0/Default    15.4W     12.95W

Class 1                  4.0W       3.84W 

Class 2                  7.0W       6.49W

Class 3                15.4W     12.95W

Class 4                30.0W     25.50W

As per this table, dependent on what class is being detected, the Switch (PSE) allows a certain maximum power to be drawn. It is important to note that the standard also defines the power the PD can be able to consume. The standard allocates for a budget of power to be used by the cabling between the PSE and the PD. This also highlights how important it is to know what type of cables are used when you investigate Imax errors, and to determine in what circumstances they can occur more than in others.

On top of the classification, negotiation of power is completed with the CDP or the LLDP protocol. This allows the switch to allocate more or less power than what the class has set as maximum.

As can be seen in the next example, a PD (Access Point in this case) comes up. Before power negotiation has taken place, it was allocated the default 15.4W that is set for the class.

3850_4#sh cdp neigh te 3/0/44 detail
-------------------------
Device ID: AP2
Entry address(es): 
  IPv6 address: FE80::CEEF:48FF:FEC2:1B9B  (link-local)
Platform: cisco AIR-CAP3501I-E-K9,  Capabilities: Router Trans-Bridge Source-Route-Bridge IGMP 
Interface: TenGigabitEthernet3/0/44,  Port ID (outgoing port): GigabitEthernet0
Holdtime : 163 sec
Version :
Cisco IOS Software, C3500 Software (AP3G1-K9W8-M), Version 15.3(3)JNB3, RELEASE SOFTWARE (fc1)
Technical Support: https://www.cisco.com/c/en/us/support/index.html
Copyright (c) 1986-2016 by Cisco Systems, Inc.
Compiled Tue 05-Jan-16 00:44 by prod_rel_team
advertisement version: 2
Duplex: full
Total cdp entries displayed : 1

3850_4#sh power inline te 3/0/44  
Interface Admin  Oper       Power   Device              Class Max
                            (Watts)
--------- ------ ---------- ------- ------------------- ----- ----
Te3/0/44  auto   on         15.4    AIR-CAP3501I-E-K9   3     60.0 

Now, as soon as power negotiation happened, the switch allocates less power. In the output of the show cdp neig <if> detail command are the various power levels that are requested. While some devices can just have one requirement, there are devices that would request multiple power levels. APs, for example, have the ability to power up or down radios if they would not be granted full power. In this example, the PD requests either 15000 or 14500 mW.

3850_4#sh cdp neigh te 3/0/44 detail
-------------------------
Device ID: AP2
Entry address(es): 
  IP address: 10.1.200.2
  IPv6 address: FE80::CEEF:48FF:FEC2:1B9B  (link-local)
Platform: cisco AIR-CAP3501I-E-K9,  Capabilities: Trans-Bridge Source-Route-Bridge IGMP 
Interface: TenGigabitEthernet3/0/44,  Port ID (outgoing port): GigabitEthernet0
Holdtime : 172 sec
Version :
Cisco IOS Software, C3500 Software (AP3G1-K9W8-M), Version 15.3(3)JNB3, RELEASE SOFTWARE (fc1)
Technical Support: https://www.cisco.com/c/en/us/support/index.html
Copyright (c) 1986-2016 by Cisco Systems, Inc.
Compiled Tue 05-Jan-16 00:44 by prod_rel_team
advertisement version: 2
Duplex: full
Power drawn: 15.000 Watts
Power request id: 15079, Power management id: 2
Power request levels are: 15000 14500 0 0 0 
Management address(es): 
  IP address: 10.1.200.2

3850_4#sh power inline te 3/0/44 detail 
 Interface: Te3/0/44
 Inline Power Mode: auto
 Operational status: on
 Device Detected: yes
 Device Type: cisco AIR-CAP3501I-
 IEEE Class: 3
 Discovery mechanism used/configured: Ieee and Cisco
 Police: off
 Power Allocated 
 Admin Value: 60.0
 Power drawn from the source: 15.0
 Power available to the device: 15.0
 Actual consumption
 Measured at the port: 6.1
 Maximum Power drawn by the device since powered on: 6.2
 Absent Counter: 0
 Over Current Counter: 0
 Short Current Counter: 0
 Invalid Signature Counter: 0
 Power Denied Counter: 0
 Power Negotiation Used: CDP
 LLDP Power Negotiation --Sent to PD--      --Rcvd from PD--
   Power Type:          -                    -
   Power Source:        -                    -
   Power Priority:      -                    -
   Requested Power(W):  -                    -
   Allocated Power(W):  -                    -
Four-Pair PoE Supported: Yes
Spare Pair Power Enabled: No
Four-Pair PD Architecture: Shared

The use of LLDP instead of CDP shows the same results. As the PD gets powered, the device receives full 15.4W as per the class. 

3850_4#sh lldp neighbors te 3/0/44 detail
------------------------------------------------
Local Intf: Te3/0/44
Chassis id: 2c3f.387e.91d0
Port id: Gi0
Port Description: GigabitEthernet0
System Name: AP2.cisco.com
System Description: 
Cisco IOS Software, C3500 Software (AP3G1-K9W8-M), Version 15.3(3)JNB3, RELEASE SOFTWARE (fc1)
Technical Support: https://www.cisco.com/c/en/us/support/index.html
Copyright (c) 1986-2016 by Cisco Systems, Inc.
Compiled Tue 05-Jan-16 00:44 by prod_rel_team
Time remaining: 64 seconds
System Capabilities: B
Enabled Capabilities: B
Management Addresses:
    IP: 10.1.200.2
Auto Negotiation - supported, enabled
Physical media capabilities:
    1000baseT(FD)
    1000baseT(HD)
    100base-TX(FD)
    100base-TX(HD)
    10base-T(FD)
    10base-T(HD)
Media Attachment Unit type: 30
Vlan ID: - not advertised 

Total entries displayed: 1 

3850_4#sh power inline te 3/0/44 detail
 Interface: Te3/0/44
 Inline Power Mode: auto
 Operational status: on
 Device Detected: yes
 Device Type: Ieee PD
 IEEE Class: 3
 Discovery mechanism used/configured: Ieee and Cisco
 Police: off
 Power Allocated 
 Admin Value: 60.0
 Power drawn from the source: 15.4
 Power available to the device: 15.4
 Actual consumption
 Measured at the port: 5.2
 Maximum Power drawn by the device since powered on: 5.3
 Absent Counter: 0
 Over Current Counter: 0
 Short Current Counter: 0
 Invalid Signature Counter: 0
 Power Denied Counter: 0
 Power Negotiation Used: None
 LLDP Power Negotiation --Sent to PD--      --Rcvd from PD--
   Power Type:          -                    -
   Power Source:        -                    -
   Power Priority:      -                    -
   Requested Power(W):  -                    -
   Allocated Power(W):  -                    -
Four-Pair PoE Supported: Yes
Spare Pair Power Enabled: No
Four-Pair PD Architecture: N/A

Once it boots up, allocation gets lowered.

3850_4#sh lldp neighbors te 3/0/44 detail
------------------------------------------------
Local Intf: Te3/0/44
Chassis id: 2c3f.387e.91d0
Port id: Gi0
Port Description: GigabitEthernet0
System Name: AP2.cisco.com 
System Description:
Cisco IOS Software, C3500 Software (AP3G1-K9W8-M), Version 15.3(3)JNB3, RELEASE SOFTWARE (fc1)
Technical Support: https://www.cisco.com/c/en/us/support/index.html
Copyright (c) 1986-2016 by Cisco Systems, Inc.
Compiled Tue 05-Jan-16 00:44 by prod_rel_team
Time remaining: 108 seconds
System Capabilities: B
Enabled Capabilities: B
Management Addresses:
    IP: 10.1.200.2
Auto Negotiation - supported, enabled
Physical media capabilities:
    1000baseT(FD)
    1000baseT(HD)
    100base-TX(FD)
    100base-TX(HD)
    10base-T(FD)
    10base-T(HD)
Media Attachment Unit type: 30
Vlan ID: - not advertised
PoE+ Power-via-MDI TLV:
 Power Pair: Signal
 Power Class: Class 3
 Power Device Type: Type 1 PD
 Power Source: PSE
 Power Priority: high
 Power Requested: 12700 mW
 Power Allocated: 12700 mW
Total entries displayed: 1

3850_4#sh power inline te 3/0/44 detail
 Interface: Te3/0/44
 Inline Power Mode: auto
 Operational status: on
 Device Detected: yes
 Device Type: Ieee PD
 IEEE Class: 3
 Discovery mechanism used/configured: Ieee and Cisco
 Police: off
 Power Allocated 
 Admin Value: 60.0
 Power drawn from the source: 15.0
 Power available to the device: 15.0
 Actual consumption
 Measured at the port: 6.1
 Maximum Power drawn by the device since powered on: 6.2
 Absent Counter: 0
 Over Current Counter: 0
 Short Current Counter: 0
 Invalid Signature Counter: 0
 Power Denied Counter: 0
 Power Negotiation Used: IEEE 802.3at LLDP
 LLDP Power Negotiation --Sent to PD--      --Rcvd from PD--
   Power Type:          Type 2 PSE           Type 1 PD
   Power Source:        Primary              PSE
   Power Priority:      low                  high
   Requested Power(W):  12.7                 12.7
   Allocated Power(W):  12.7                 12.7
Four-Pair PoE Supported: Yes
Spare Pair Power Enabled: No
Four-Pair PD Architecture: Share

Output from the show power inline <interface> detail command shows more information in regards to the negotiation that is being done than what is shown by CDP. There is also another major difference between CDP and LLDP in regards to power negotiation. CDP negotiates the amount of power provided at the port (15W). With LLDP however, you see that the PD does not negotiate the power the port can supply. It requests the amount of power the PD wishes to have. In this case, it is 12.7W. The switch (PSE) has to compensate for the loss in the cabling and allocates 15W to the port. As power negotiation does take place, it is also key to determine what the requested power was at the time of failure. Knowledge of how long the device was up and what events can have taken place at the time of the error can provide more detail around the root cause. For example, an IP phone that comes out of sleep and turns its screen on fully can momentarily draw more power.

Summary

For Imax errors, it is hard to determine the exact cause. In almost all cases, there is found to be an issue with the PD drawing more power, and the PD vendor needs to be engaged in order to investigate why it exceeds the power it has negotiated with the switch.

It is also crucial to investigate the type and length of the cabling as this does change the electrical characteristics and influences the amount of power drawn on the port. It is important as well is to investigate the power negotiation and confirm that the power requested by a device is also the amount of power that gets allocated. In the case of LLDP, additional budget for cabling between PD and PSE is needed. In some cases, with use of statically allocated power, it is possible to work around Imax errors and/or to determine the amount of power the device overdraws on a port. A confirmation that the PD overdraws the amount of power it gets allocated can be achieved only with power measuring and testing devices.

In Cisco IOS XE releases 3.6.5 and 3.7.5 and later, a few improvements have been made around Imax errors:

• The amount of reprogramming of the Icutoff value to the port has been reduced.
• The allowance on the port for overdrawing power has been increased, in some cases this can be enough to prevent an Imax error.
• Some corner case scenarios were resolved where an Imax error can have occurred as a false alarm.