Troubleshoot of IPsecmgr in Warn or Over State
Available Languages
Introduction
This document describes the IPsecmgr facility in the warning state.
Overview
IPsecmgr is created by the Session Controller. It establishes and manages secure IKEv1, IKEv2, and IPSec data tunnels in Evolved Packet Data Gateway (ePDG).
Prerequisites
Cisco recommends that you have knowledge of these nodes and StarOS facilities:
- ePDG
- IPSecmgr facility
Logs Required
Note: In order to address these types of issues, it is necessary to gather the logs for analysis.
1. Identify the IPsecmgr facilities that are in a warning state and gather heap dumps for two good IPsecmgr instances, along with two bad instances during peak and non-peak hours.
Show task resources
show messenger proclet facility ipsecmgr instance <instance number> heap depth 9
show messenger proclet facility ipsecmgr instance <instance number> system heap depth 9
show messenger proclet facility ipsecmgr instance <instance number> heap
show messenger proclet facility ipsecmgr instance <instance number> system
show messenger proclet facility ipsecmgr instance <instance number> graphs heap
show session subsystem facility ipsecmgr instance <instance number> debug-info verbose
show task resources facility ipsecmgr instance <instance number>
Note: Collect the logs for two functional and two faulty IPsecmgr instances. Every time, replace the instance number with the IPsecmgr instance number.
2. Collect the core files for the same IPsecmgr instances that were collected in the earlier step.
Login to hidden mode
Cli test-commands password <>
Task core facility ipsecmgr instance <instance number>
Note: Collect the core file for the same instance number for which the heap output has been collected.
3. Gather the Show Support Details (SSD) data, Simple Network Management Protocol (SNMP) traps, and system logs (syslogs).
4. Gather the output of the show task resource command multiple times throughout the day.
Note: It is necessary to determine whether the memory is consistently increasing or decreasing when the number of calls is reduced.
5. Determine the services that are currently running on the node.
Show resources
6. Provide the subscriber trend data for each service running on the node, covering a period of two months.
For example, for ePDG-service, the ePDG subscriber trend is required. For the Mme-service, the mme subscriber trend is required.
7. Provide the SSD data for any other node in the network that has identical configurations and services running.
Note: It is necessary to determine whether the issue is with the node or the software version.
8. After the upgrade, the facility IPsecmgr turns into a warning state, and shares multiple Solid State Dr (SSD) data sets from both before and after the upgrade.
Troubleshooting Performed
1. An analysis of the IPsecmgr heap dump must be done. Based on this analysis, it has been identified that these top two functions consume a high amount of memory.
|
Proc |
Aggregate |
% ▴ |
|
ipm_databuf_alloc() |
60900436 |
42.19 |
|
service_recovery_add_dest_handle() |
17302140 |
11.99 |
|
ikev2_encode_alloc_opacket() |
6319608 |
4.38 |
|
demuxmgr_init_smgr_pacing_queue_and_cache() |
6258720 |
4.34 |
|
ipm_sad_ikesa_alloc() |
6053856 |
4.19 |
Note: From the output of the heap analysis, it is evident that the function ipm_databuf_alloc() consumes a significant amount of memory.
2. The ePDG subscriber trend for one month indicates that the traffic trend is normal, as no increase in traffic is observed.
|
Date |
Measurement Level |
Node |
Total User Count |
|
2023-01-01 |
Day Level |
XXX-XX--SGSN-MME-03 |
308580 |
|
2023-01-02 |
Day Level |
XXX-XX--SGSN-MME-03 |
331166 |
|
2023-01-03 |
Day Level |
XXX-XX--SGSN-MME-03 |
332424 |
|
2023-01-04 |
Day Level |
XXX-XX--SGSN-MME-03 |
324741 |
|
2023-01-05 |
Day Level |
XXX-XX--SGSN-MME-03 |
329006 |
|
2023-01-06 |
Day Level |
XXX-XX--SGSN-MME-03 |
326667 |
|
2023-01-07 |
Day Level |
XXX-XX--SGSN-MME-03 |
327323 |
|
2023-01-08 |
Day Level |
XXX-XX--SGSN-MME-03 |
327922 |
|
2023-01-09 |
Day Level |
XXX-XX--SGSN-MME-03 |
331701 |
|
2023-01-10 |
Day Level |
XXX-XX--SGSN-MME-03 |
334258 |
|
2023-01-11 |
Day Level |
XXX-XX--SGSN-MME-03 |
331070 |
|
2023-01-12 |
Day Level |
XXX-XX--SGSN-MME-03 |
327105 |
|
2023-01-13 |
Day Level |
XXX-XX--SGSN-MME-03 |
310919 |
|
2023-01-14 |
Day Level |
XXX-XX--SGSN-MME-03 |
280490 |
|
2023-01-15 |
Day Level |
XXX-XX--SGSN-MME-03 |
299159 |
|
2023-01-16 |
Day Level |
XXX-XX--SGSN-MME-03 |
312329 |
|
2023-01-17 |
Day Level |
XXX-XX--SGSN-MME-03 |
318370 |
|
2023-01-18 |
Day Level |
XXX-XX--SGSN-MME-03 |
316067 |
|
2023-01-19 |
Day Level |
XXX-XX--SGSN-MME-03 |
314299 |
|
2023-01-20 |
Day Level |
XXX-XX--SGSN-MME-03 |
322221 |
|
2023-01-21 |
Day Level |
XXX-XX--SGSN-MME-03 |
317145 |
|
2023-01-22 |
Day Level |
XXX-XX--SGSN-MME-03 |
317951 |
|
2023-01-23 |
Day Level |
XXX-XX--SGSN-MME-03 |
320670 |
|
2023-01-24 |
Day Level |
XXX-XX--SGSN-MME-03 |
320466 |
|
2023-01-25 |
Day Level |
XXX-XX--SGSN-MME-03 |
304693 |
|
2023-01-26 |
Day Level |
XXX-XX--SGSN-MME-03 |
307049 |
|
2023-01-27 |
Day Level |
XXX-XX--SGSN-MME-03 |
304500 |
|
2023-01-28 |
Day Level |
XXX-XX--SGSN-MME-03 |
303082 |
|
2023-01-29 |
Day Level |
XXX-XX--SGSN-MME-03 |
309391 |
|
2023-01-30 |
Day Level |
XXX-XX--SGSN-MME-03 |
314559 |
|
2023-01-31 |
Day Level |
XXX-XX--SGSN-MME-03 |
318688 |
|
2023-02-01 |
Day Level |
XXX-XX--SGSN-MME-03 |
315274 |
|
2023-02-02 |
Day Level |
XXX-XX--SGSN-MME-03 |
320590 |
|
2023-02-03 |
Day Level |
XXX-XX--SGSN-MME-03 |
276683 |
|
2023-02-04 |
Day Level |
XXX-XX--SGSN-MME-03 |
288478 |
|
2023-02-05 |
Day Level |
XXX-XX--SGSN-MME-03 |
298212 |
|
2023-02-06 |
Day Level |
XXX-XX--SGSN-MME-03 |
300505 |
|
2023-02-07 |
Day Level |
XXX-XX--SGSN-MME-03 |
308763 |
|
2023-02-08 |
Day Level |
XXX-XX--SGSN-MME-03 |
303933 |
|
2023-02-09 |
Day Level |
XXX-XX--SGSN-MME-03 |
304191 |
|
2023-02-10 |
Day Level |
XXX-XX--SGSN-MME-03 |
296049 |
|
2023-02-11 |
Day Level |
XXX-XX--SGSN-MME-03 |
304243 |
|
2023-02-12 |
Day Level |
XXX-XX--SGSN-MME-03 |
398876 |
|
2023-02-13 |
Day Level |
XXX-XX--SGSN-MME-03 |
327081 |
|
2023-02-14 |
Day Level |
XXX-XX--SGSN-MME-03 |
318326 |
|
2023-02-15 |
Day Level |
XXX-XX--SGSN-MME-03 |
308936 |
|
2023-02-16 |
Day Level |
XXX-XX--SGSN-MME-03 |
307454 |
|
2023-02-17 |
Day Level |
XXX-XX--SGSN-MME-03 |
299155 |
|
2023-02-18 |
Day Level |
XXX-XX--SGSN-MME-03 |
310671 |
|
2023-02-19 |
Day Level |
XXX-XX--SGSN-MME-03 |
308271 |
|
2023-02-20 |
Day Level |
XXX-XX--SGSN-MME-03 |
313036 |
|
2023-02-21 |
Day Level |
XXX-XX--SGSN-MME-03 |
308364 |
|
2023-02-22 |
Day Level |
XXX-XX--SGSN-MME-03 |
297624 |
|
2023-02-23 |
Day Level |
XXX-XX--SGSN-MME-03 |
307631 |
|
2023-02-24 |
Day Level |
XXX-XX--SGSN-MME-03 |
303706 |
|
2023-02-25 |
Day Level |
XXX-XX--SGSN-MME-03 |
331917 |
|
2023-02-26 |
Day Level |
XXX-XX--SGSN-MME-03 |
304802 |
|
2023-02-27 |
Day Level |
XXX-XX--SGSN-MME-03 |
314204 |
|
2023-02-28 |
Day Level |
XXX-XX--SGSN-MME-03 |
312129 |
|
2023-03-01 |
Day Level |
XXX-XX--SGSN-MME-03 |
318518 |
|
2023-03-02 |
Day Level |
XXX-XX--SGSN-MME-03 |
400818 |
|
2023-03-03 |
Day Level |
XXX-XX--SGSN-MME-03 |
298952 |
|
2023-03-04 |
Day Level |
XXX-XX--SGSN-MME-03 |
286058 |
|
2023-03-05 |
Day Level |
XXX-XX--SGSN-MME-03 |
283346 |
|
2023-03-06 |
Day Level |
XXX-XX--SGSN-MME-03 |
280037 |
|
2023-03-07 |
Day Level |
XXX-XX--SGSN-MME-03 |
275515 |
|
2023-03-08 |
Day Level |
XXX-XX--SGSN-MME-03 |
290489 |
|
2023-03-09 |
Day Level |
XXX-XX--SGSN-MME-03 |
291972 |
|
2023-03-10 |
Day Level |
XXX-XX-SGSN-MME-03 |
289658 |
|
2023-03-11 |
Day Level |
XXX-XX--SGSN-MME-03 |
284311 |
|
2023-03-12 |
Day Level |
XXX-XX--SGSN-MME-03 |
293878 |
|
2023-03-13 |
Day Level |
XXX-XX--SGSN-MME-03 |
298662 |
|
2023-03-14 |
Day Level |
XXX-XX--SGSN-MME-03 |
299430 |
|
2023-03-15 |
Day Level |
XXX-XX--SGSN-MME-03 |
297959 |
|
2023-03-16 |
Day Level |
XXX-XX--SGSN-MME-03 |
299208 |
|
2023-03-17 |
Day Level |
XXX-XX--SGSN-MME-03 |
293534 |
|
2023-03-18 |
Day Level |
XXX-XX--SGSN-MME-03 |
292673 |
|
2023-03-19 |
Day Level |
XXX-XX--SGSN-MME-03 |
296111 |
|
2023-03-20 |
Day Level |
XXX-XX-SGSN-MME-03 |
301181 |
|
2023-03-21 |
Day Level |
XXX-XX--SGSN-MME-03 |
297604 |
|
2023-03-22 |
Day Level |
XXX-XX--SGSN-MME-03 |
309897 |
|
2023-03-23 |
Day Level |
XXX-XX--SGSN-MME-03 |
304351 |
|
2023-03-24 |
Day Level |
XXX-XX--SGSN-MME-03 |
305605 |
|
2023-03-25 |
Day Level |
XXX-XX--SGSN-MME-03 |
295694 |
|
2023-03-26 |
Day Level |
XXX-XX--SGSN-MME-03 |
300188 |
|
2023-03-27 |
Day Level |
XXX-XX--SGSN-MME-03 |
302467 |
|
2023-03-28 |
Day Level |
XXX-XX--SGSN-MME-03 |
303965 |
|
2023-03-29 |
Day Level |
XXX-XX--SGSN-MME-03 |
270432 |
|
2023-03-30 |
Day Level |
XXX-XX--SGSN-MME-03 |
289729 |
|
2023-03-31 |
Day Level |
XXX-XX--SGSN-MME-03 |
249336 |
|
2023-04-01 |
Day Level |
XXX-XX--SGSN-MME-03 |
269109 |
|
2023-04-02 |
Day Level |
XXX-XX--SGSN-MME-03 |
283437 |
|
2023-04-03 |
Day Level |
XXX-XX--SGSN-MME-03 |
298453 |
|
2023-04-04 |
Day Level |
XXX-XX--SGSN-MME-03 |
301649 |
|
2023-04-05 |
Day Level |
XXX-XX--SGSN-MME-03 |
298797 |
|
2023-04-06 |
Day Level |
XXX-XX--SGSN-MME-03 |
306610 |
|
2023-04-07 |
Day Level |
XXX-XX--SGSN-MME-03 |
298449 |
|
2023-04-08 |
Day Level |
XXX-XX--SGSN-MME-03 |
294205 |
|
2023-04-09 |
Day Level |
XXX-XX--SGSN-MME-03 |
306048 |
|
2023-04-10 |
Day Level |
XXX-XX--SGSN-MME-03 |
316718 |
|
2023-04-11 |
Day Level |
XXX-XX--SGSN-MME-03 |
315327 |
|
2023-04-12 |
Day Level |
XXX-XX-SGSN-MME-03 |
313059 |
3. The show task resources from multiple Solid State Drives (SSDs) were collected. Based on the SSD data, it is seen that these IPsecmgr instances are in a warning state.
8/1 ipsecmgr 296 2.64% 50% 193.3M 191.0M 978 1500 1331 4700 - warn
8/2 ipsecmgr 363 2.51% 50% 194.1M 191.0M 854 1500 1304 4700 - warn
9/0 ipsecmgr 231 2.85% 50% 129.0M 191.0M 271 1500 1323 4700 - good
9/1 ipsecmgr 303 2.61% 50% 161.5M 191.0M 1141 1500 1322 4700 - good
9/2 ipsecmgr 381 2.46% 50% 192.9M 191.0M 839 1500 1312 4700 - warn
Note: This output is collected on 12th April at 8:40 PM
This output collected on the 13th April 8 PM.
8/1 ipsecmgr 296 2.61% 50% 193.3M 191.0M 985 1500 1221 4700 - warn
8/2 ipsecmgr 363 2.38% 50% 193.7M 191.0M 828 1500 1222 4700 - warn
9/0 ipsecmgr 231 2.40% 50% 129.0M 191.0M 1116 1500 1237 4700 - good
9/1 ipsecmgr 303 2.43% 50% 161.5M 191.0M 1120 1500 1220 4700 - good
9/2 ipsecmgr 381 2.23% 50% 192.9M 191.0M 829 1500 1228 4700 - warn
No decrease in memory is observed, even when the number of calls is reduced in each of the instances. For example, if you look at the IPsecmgr instance 296, which is in a warning state, the number of calls decreased on April 13th, but the memory did not decrease
4. The show task resources data collected throughout the day also confirms that the memory is not decreasing, even when the number of calls is reduced.
5. The SSD data from before the upgrade shows that there is no increase in memory. This suggests that there can be a problem with the new software.
Based on the analysis, you can suspect a memory leak at the IPsecmgr facility level. Raise a service request (SR) in order to investigate further and address the issue.
Revision History
| Revision | Publish Date | Comments |
|---|---|---|
4.0 |
15-Dec-2023 |
Initial Release |
1.0 |
14-Dec-2023 |
Initial Release |
FeedbackContact Cisco
- Open a Support Case
- (Requires a Cisco Service Contract)