Contents

Introduction

Troubleshooting

Related Information

Introduction

This document describes how to detect the RCM-based UPF (User Plane Function) upgrade failure due to IP pool flush issue.

Troubleshooting

1. From Active Redundancy Configuration Manager (RCM) controller run the command  rcm show-statistics switchover-verbose  and look for the failed UPF switchover details if it has only start_ip_pool_flush field but end_ip_pool_flush is missing.

[RCM-active] rcm# rcm show-statistics switchover-verbose

Thu Jan 18 21:52:11.781 UTC+00:00

message :

{

"stats_verbose": [

{

"status": "Failed",

"started": "Jan 18 05:08:59.442",

"switchoverreason": "Planned Switchover",

"switchoverfailreason": "Old Active moved from PendingStandby to Active due to timeout in receiving Standby state (planned switchover)",

"source_endpoint": "192.168.100.3",

"destination_endpoint": "192.168.100.2",

"start_chkpt_flush": "Jan 18 05:09:01.783",

"end_chkpt_flush_each": {

"1": "Jan 18 05:09:05.776",

"10": "Jan 18 05:09:06.056",

"11": "Jan 18 05:09:05.909",

"12": "Jan 18 05:09:06.189",

"13": "Jan 18 05:09:05.991",

"14": "Jan 18 05:09:05.981",

"15": "Jan 18 05:09:05.531",

"16": "Jan 18 05:09:05.502",

"17": "Jan 18 05:09:05.751",

"18": "Jan 18 05:09:06.116",

"19": "Jan 18 05:09:05.834",

"2": "Jan 18 05:09:05.932",

"20": "Jan 18 05:09:05.889",

"21": "Jan 18 05:09:05.354",

"22": "Jan 18 05:09:05.780",

"3": "Jan 18 05:09:05.671",

"4": "Jan 18 05:09:05.996",

"5": "Jan 18 05:09:05.455",

"6": "Jan 18 05:09:05.839",

"7": "Jan 18 05:09:06.111",

"8": "Jan 18 05:09:06.019",

"9": "Jan 18 05:09:05.996"

},

"end_chkpt_flush": "Jan 18 05:09:06.189",

"start_cfg_push": "Jan 18 05:08:59.495",

"end_cfg_push": "Jan 18 05:09:01.774",

"start_ip_pool_flush": "Jan 18 05:09:01.783"  /* Observe under this line "end_ip_pool_flush" is missing */

},

2. Collect source and destination UPF syslogs covering switchover times +/- 30 minutes.

3. Dig into destination endpoint UPF (192.168.100.2 in output)  syslogs and around the time of UPF switchover and observe vpnmgr6(context n6) reporting several errors, related to IP pool/chunks.

Example log events:

- Jan 18 05:09:01 <UPF> evlogd: [local-60sec1.767] [srp 84220 error] [1/0/9589 <vpnmgr:6> vpnmgr_rcm.c:6235] [context: rcm, contextID: 6] [software internal system syslog] Failure in installing pool routes. Message Bounced.

-  evlogd: [local-60sec12.192] [vpn 5013 error] [1/0/9370 <vpnmgr:3> vpn_ip_pool.c:15699] [context: n6, contextID: 3]  [software internal system syslog] #012Prefix <IPv4 address of the IP pool> is not allocated to this UP: Closest chunk found with id 3145764 start_addr: <IPv4 address of the IP pool> end_addr: <IPv4 address of the IP pool>

Note: Only registered Cisco users can access internal Cisco information and tools.

Related Information

- Cisco Technical Support & Downloads