Contents

Introduction

Overview

Prerequisites

Logs Required

Troubleshooting Performed

Introduction

This document describes the IPsecmgr facility in the warning state.

Overview

IPsecmgr is created by the Session Controller. It establishes and manages secure IKEv1, IKEv2, and IPSec data tunnels in Evolved Packet Data Gateway (ePDG).

Prerequisites

Cisco recommends that you have knowledge of these nodes and StarOS facilities:

- ePDG

- IPSecmgr facility

Logs Required

Note: In order to address these types of issues, it is necessary to gather the logs for analysis.

1. Identify the IPsecmgr facilities that are in a warning state and gather heap dumps for two good IPsecmgr instances, along with two bad instances during peak and non-peak hours.

Show task resources

show messenger proclet facility ipsecmgr instance <instance number> heap depth 9

show messenger proclet facility ipsecmgr instance <instance number> system heap depth 9

show messenger proclet facility ipsecmgr instance <instance number> heap

show messenger proclet facility ipsecmgr instance <instance number> system

show messenger proclet facility ipsecmgr instance <instance number> graphs heap

show session subsystem facility ipsecmgr instance <instance number> debug-info verbose

show task resources facility ipsecmgr instance <instance number>

Note: Collect the logs for two functional and two faulty IPsecmgr instances. Every time, replace the instance number with the IPsecmgr instance number.

2. Collect the core files for the same IPsecmgr instances that were collected in the earlier step.

Login to hidden mode

Cli test-commands password <>

Task core facility ipsecmgr instance <instance number>

Note: Collect the core file for the same instance number for which the heap output has been collected.

3. Gather the Show Support Details (SSD) data, Simple Network Management Protocol (SNMP) traps, and system logs (syslogs).

4. Gather the output of the show task resource command multiple times throughout the day.

Note: It is necessary to determine whether the memory is consistently increasing or decreasing when the number of calls is reduced.

5. Determine the services that are currently running on the node.

Show resources

6. Provide the subscriber trend data for each service running on the node, covering a period of two months.

For example, for ePDG-service, the ePDG subscriber trend is required. For the Mme-service, the mme subscriber trend is required.

7. Provide the SSD data for any other node in the network that has identical configurations and services running.

Note: It is necessary to determine whether the issue is with the node or the software version.

8. After the upgrade, the facility IPsecmgr turns into a warning state, and shares multiple Solid State Dr (SSD) data sets from both before and after the upgrade.

Troubleshooting Performed

1. An analysis of the IPsecmgr heap dump must be done. Based on this analysis, it has been identified that these top two functions consume a high amount of memory.

Proc

Aggregate

% ▴

ipm_databuf_alloc()

60900436

42.19

service_recovery_add_dest_handle()

17302140

11.99

ikev2_encode_alloc_opacket()

6319608

4.38

demuxmgr_init_smgr_pacing_queue_and_cache()

6258720

4.34

ipm_sad_ikesa_alloc()

6053856

4.19

Note: From the output of the heap analysis, it is evident that the function ipm_databuf_alloc() consumes a significant amount of memory.

2. The ePDG subscriber trend for one month indicates that the traffic trend is normal, as no increase in traffic is observed.

Date

Measurement Level

Node

Total User Count

2023-01-01

Day Level

XXX-XX--SGSN-MME-03

308580

2023-01-02

Day Level

XXX-XX--SGSN-MME-03

331166

2023-01-03

Day Level

XXX-XX--SGSN-MME-03

332424

2023-01-04

Day Level

XXX-XX--SGSN-MME-03

324741

2023-01-05

Day Level

XXX-XX--SGSN-MME-03

329006

2023-01-06

Day Level

XXX-XX--SGSN-MME-03

326667

2023-01-07

Day Level

XXX-XX--SGSN-MME-03

327323

2023-01-08

Day Level

XXX-XX--SGSN-MME-03

327922

2023-01-09

Day Level

XXX-XX--SGSN-MME-03

331701

2023-01-10

Day Level

XXX-XX--SGSN-MME-03

334258

2023-01-11

Day Level

XXX-XX--SGSN-MME-03

331070

2023-01-12

Day Level

XXX-XX--SGSN-MME-03

327105

2023-01-13

Day Level

XXX-XX--SGSN-MME-03

310919

2023-01-14

Day Level

XXX-XX--SGSN-MME-03

280490

2023-01-15

Day Level

XXX-XX--SGSN-MME-03

299159

2023-01-16

Day Level

XXX-XX--SGSN-MME-03

312329

2023-01-17

Day Level

XXX-XX--SGSN-MME-03

318370

2023-01-18

Day Level

XXX-XX--SGSN-MME-03

316067

2023-01-19

Day Level

XXX-XX--SGSN-MME-03

314299

2023-01-20

Day Level

XXX-XX--SGSN-MME-03

322221

2023-01-21

Day Level

XXX-XX--SGSN-MME-03

317145

2023-01-22

Day Level

XXX-XX--SGSN-MME-03

317951

2023-01-23

Day Level

XXX-XX--SGSN-MME-03

320670

2023-01-24

Day Level

XXX-XX--SGSN-MME-03

320466

2023-01-25

Day Level

XXX-XX--SGSN-MME-03

304693

2023-01-26

Day Level

XXX-XX--SGSN-MME-03

307049

2023-01-27

Day Level

XXX-XX--SGSN-MME-03

304500

2023-01-28

Day Level

XXX-XX--SGSN-MME-03

303082

2023-01-29

Day Level

XXX-XX--SGSN-MME-03

309391

2023-01-30

Day Level

XXX-XX--SGSN-MME-03

314559

2023-01-31

Day Level

XXX-XX--SGSN-MME-03

318688

2023-02-01

Day Level

XXX-XX--SGSN-MME-03

315274

2023-02-02

Day Level

XXX-XX--SGSN-MME-03

320590

2023-02-03

Day Level

XXX-XX--SGSN-MME-03

276683

2023-02-04

Day Level

XXX-XX--SGSN-MME-03

288478

2023-02-05

Day Level

XXX-XX--SGSN-MME-03

298212

2023-02-06

Day Level

XXX-XX--SGSN-MME-03

300505

2023-02-07

Day Level

XXX-XX--SGSN-MME-03

308763

2023-02-08

Day Level

XXX-XX--SGSN-MME-03

303933

2023-02-09

Day Level

XXX-XX--SGSN-MME-03

304191

2023-02-10

Day Level

XXX-XX--SGSN-MME-03

296049

2023-02-11

Day Level

XXX-XX--SGSN-MME-03

304243

2023-02-12

Day Level

XXX-XX--SGSN-MME-03

398876

2023-02-13

Day Level

XXX-XX--SGSN-MME-03

327081

2023-02-14

Day Level

XXX-XX--SGSN-MME-03

318326

2023-02-15

Day Level

XXX-XX--SGSN-MME-03

308936

2023-02-16

Day Level

XXX-XX--SGSN-MME-03

307454

2023-02-17

Day Level

XXX-XX--SGSN-MME-03

299155

2023-02-18

Day Level

XXX-XX--SGSN-MME-03

310671

2023-02-19

Day Level

XXX-XX--SGSN-MME-03

308271

2023-02-20

Day Level

XXX-XX--SGSN-MME-03

313036

2023-02-21

Day Level

XXX-XX--SGSN-MME-03

308364

2023-02-22

Day Level

XXX-XX--SGSN-MME-03

297624

2023-02-23

Day Level

XXX-XX--SGSN-MME-03

307631

2023-02-24

Day Level

XXX-XX--SGSN-MME-03

303706

2023-02-25

Day Level

XXX-XX--SGSN-MME-03

331917

2023-02-26

Day Level

XXX-XX--SGSN-MME-03

304802

2023-02-27

Day Level

XXX-XX--SGSN-MME-03

314204

2023-02-28

Day Level

XXX-XX--SGSN-MME-03

312129

2023-03-01

Day Level

XXX-XX--SGSN-MME-03

318518

2023-03-02

Day Level

XXX-XX--SGSN-MME-03

400818

2023-03-03

Day Level

XXX-XX--SGSN-MME-03

298952

2023-03-04

Day Level

XXX-XX--SGSN-MME-03

286058

2023-03-05

Day Level

XXX-XX--SGSN-MME-03

283346

2023-03-06

Day Level

XXX-XX--SGSN-MME-03

280037

2023-03-07

Day Level

XXX-XX--SGSN-MME-03

275515

2023-03-08

Day Level

XXX-XX--SGSN-MME-03

290489

2023-03-09

Day Level

XXX-XX--SGSN-MME-03

291972

2023-03-10

Day Level

XXX-XX-SGSN-MME-03

289658

2023-03-11

Day Level

XXX-XX--SGSN-MME-03

284311

2023-03-12

Day Level

XXX-XX--SGSN-MME-03

293878

2023-03-13

Day Level

XXX-XX--SGSN-MME-03

298662

2023-03-14

Day Level

XXX-XX--SGSN-MME-03

299430

2023-03-15

Day Level

XXX-XX--SGSN-MME-03

297959

2023-03-16

Day Level

XXX-XX--SGSN-MME-03

299208

2023-03-17

Day Level

XXX-XX--SGSN-MME-03

293534

2023-03-18

Day Level

XXX-XX--SGSN-MME-03

292673

2023-03-19

Day Level

XXX-XX--SGSN-MME-03

296111

2023-03-20

Day Level

XXX-XX-SGSN-MME-03

301181

2023-03-21

Day Level

XXX-XX--SGSN-MME-03

297604

2023-03-22

Day Level

XXX-XX--SGSN-MME-03

309897

2023-03-23

Day Level

XXX-XX--SGSN-MME-03

304351

2023-03-24

Day Level

XXX-XX--SGSN-MME-03

305605

2023-03-25

Day Level

XXX-XX--SGSN-MME-03

295694

2023-03-26

Day Level

XXX-XX--SGSN-MME-03

300188

2023-03-27

Day Level

XXX-XX--SGSN-MME-03

302467

2023-03-28

Day Level

XXX-XX--SGSN-MME-03

303965

2023-03-29

Day Level

XXX-XX--SGSN-MME-03

270432

2023-03-30

Day Level

XXX-XX--SGSN-MME-03

289729

2023-03-31

Day Level

XXX-XX--SGSN-MME-03

249336

2023-04-01

Day Level

XXX-XX--SGSN-MME-03

269109

2023-04-02

Day Level

XXX-XX--SGSN-MME-03

283437

2023-04-03

Day Level

XXX-XX--SGSN-MME-03

298453

2023-04-04

Day Level

XXX-XX--SGSN-MME-03

301649

2023-04-05

Day Level

XXX-XX--SGSN-MME-03

298797

2023-04-06

Day Level

XXX-XX--SGSN-MME-03

306610

2023-04-07

Day Level

XXX-XX--SGSN-MME-03

298449

2023-04-08

Day Level

XXX-XX--SGSN-MME-03

294205

2023-04-09

Day Level

XXX-XX--SGSN-MME-03

306048

2023-04-10

Day Level

XXX-XX--SGSN-MME-03

316718

2023-04-11

Day Level

XXX-XX--SGSN-MME-03

315327

2023-04-12

Day Level

XXX-XX-SGSN-MME-03

313059

3. The show task resources from multiple Solid State Drives (SSDs) were collected. Based on the SSD data, it is seen that these IPsecmgr instances are in a warning state.

8/1 ipsecmgr 296 2.64% 50% 193.3M 191.0M 978 1500 1331 4700 - warn

8/2 ipsecmgr 363 2.51% 50% 194.1M 191.0M 854 1500 1304 4700 - warn

9/0 ipsecmgr 231 2.85% 50% 129.0M 191.0M 271 1500 1323 4700 - good

9/1 ipsecmgr 303 2.61% 50% 161.5M 191.0M 1141 1500 1322 4700 - good

9/2 ipsecmgr 381 2.46% 50% 192.9M 191.0M 839 1500 1312 4700 - warn

Note: This output is collected on 12th April at 8:40 PM

This output collected on the 13th April 8 PM.

8/1 ipsecmgr 296 2.61% 50% 193.3M 191.0M 985 1500 1221 4700 - warn

8/2 ipsecmgr 363 2.38% 50% 193.7M 191.0M 828 1500 1222 4700 - warn

9/0 ipsecmgr 231 2.40% 50% 129.0M 191.0M 1116 1500 1237 4700 - good

9/1 ipsecmgr 303 2.43% 50% 161.5M 191.0M 1120 1500 1220 4700 - good

9/2 ipsecmgr 381 2.23% 50% 192.9M 191.0M 829 1500 1228 4700 - warn

No decrease in memory is observed, even when the number of calls is reduced in each of the instances. For example, if you look at the IPsecmgr instance 296, which is in a warning state, the number of calls decreased on April 13th, but the memory did not decrease

4. The show task resources data collected throughout the day also confirms that the memory is not decreasing, even when the number of calls is reduced.

5. The SSD data from before the upgrade shows that there is no increase in memory. This suggests that there can be a problem with the new software.

Based on the analysis, you can suspect a memory leak at the IPsecmgr facility level. Raise a service request (SR) in order to investigate further and address the issue.