UPF supports subscriber monitoring using CLI configurations to enable monitor subscriber per UE. However, when there are multiple UPFs in the network, it requires the users to enable monitor subscriber for a UE in all the UPFs, or wait for the UE to attach and identify the UPF, which is handling it and enable monitor subscriber on that UPF.

This feature enables monitor subscriber on UPF based on the configurations on SMF, thereby, eliminates the risk of missing captures.

Following are the modifications done in the UPF session manager to support this feature:

• Provide services to the CLI for enabling or disabling the MonSub tracing.

• Control NPUMgr to connect/start/stop/add/delete MonSub sessions.

• Based on the instructions from the CLI, configures panopticon (through NPUMgr) for changes, such as packet size and priority.

• Read the "hex dump module" configurations to manage the storage of generated pcap files.

• If the file copy fails or there are problems with session manager MonSub instantiation, raises the SNMP alarms.

• Capture the control or slowpath packets as a separate slowpath PCAP.

• This feature supports a maximum of four monitor subscriber tracing sessions for a UPF instance. The NPUMgr enforces the tracing limit.

• The MonSub tracing session terminates in the absence of no space on hard disk or no hard disk.

For a multi-PDN call, when you start the MonSub with Multi-trace=OFF, then it traces the only one PDN as a part of that MonSub session. When a new PDN is initiated, then existing PDN tracing stops and new PDN tracing starts. For it to happen, first the new PDN tracing is started and then the existing PDN tracing is stopped. Hence, new PSN and session manager sub-session ID is allocated.

For a multi-PDN call, when you start the MonSub with Multi-trace=ON, it traces the new PDN as part of new fastpath tracing session (that is MonSub session). Hence, after tracing the four PDN, MonSub CLI shows max tracing session reached. Tracing each PDN takes place as a separate MonSub session.

A new mechanism is added to publish the statistics regarding the quality of FASTPATH PCAP capture on MonSub CLI. The new mechanism publishes the statistics whenever it receives the buffer full MEH indication at session manager, throttled at every five seconds. The feature supports a maximum of four buffers for a FASTPATH PCAP corresponding to MonSub session. The feature does not publish the statistics by default, and needs to be enabled through debug CLI on UPF.

• debug uplane monsub-stats disabled

• debug uplane monsub-stats enabled

Packet accepted: 14250000          Packet rejected: 62297
Congestion Short Term: 0                     Congestion Longer Term: 0
Throttled: 0                                 PCAP File Transfer Rate: 9.91 mbps

The PCAP file transfer rate is the rate at which copy Co-Proc writes the PCAP from RAM-FS to HD-RAID.

This feature supports the X-Header capture in slowpath PCAP. When UPF inserts the X-Header for Uplink packet, the UPF captures the packet at entry and exit interfaces. So, the exit packet sent to N6 contains the inserted X-header.

When UPF inserts the X-header for Downlink packet, the UPF captures the packet at entry and exit interfaces. So, the exit packet sent to N3 contains the inserted x-header.

Monitor Subscriber CLI – New Options

The following options with their default value are the additions to existing monitor subscriber command:

UPF Monitor Subscriber CLI

Following are the options:

• W - UP PCAP Trace (ON): This parameter creates PCAP trace for slowpath and fastpath and is only applicable for CUPS.

• U - Mon Display (ON): The non-protocol events (such as statistics, charging information from ECS, etc.) are also captured in slowpath PCAP files and are displayed on U-PLANE monitor console.

• V - PCAP Hexdump (ON): This flag must be set to ON to capture the protocol packets in a text file in hexdump format on U-PLANE.

Note	Currently, UP PCAP Trace flag must be set to ON to capture fastpath and slowpath PCAP files on CUPS.

Monitor Subscriber CLI – New Options

The following options with their default value are the additions to existing monitor subscriber command:

• F - Packet Capture (Full Pkt): Captures all packets from fastpath.

  Using this option, operators can choose between full and partial packet captures. By entering F, the packet capture type can be changed to either full or partial. With partial packet capture, users can enter packet sizes from 1 to 16384 bytes. For example, if input is 20, only the first 20 bytes of fastpath packets will be captured and the remaining packets will be dropped.

  Note	When opening the PCAP file, the summary view displays full length of the packet, but the detailed view shows only the truncated packet.

• / - Priority (0): The value is in the range from "0 – Best Effort" to "7 – Guaranteed"

  • 0 - Best Effort

  • 1 - Low

  • 2 - Med-Low

  • 3 - Medium

  • 4 - Med-High

  • 5 - High

  • 6 - Critical

  • 7 - Guaranteed

  Caution

  It is strongly recommended not to change the default value. It can adversely affect the system performance.

• N - MEH Header (OFF): The MEH header is stripped from the IP packet, when this option is configured

Show Monitor Subscriber Sessions

Following is the new CLI to show the ongoing MonSub session.

You can trigger the CLI show monitor subscriber fastpath session all from UPF.

• SessId: This is the local session id for MonSub session on UP session manager.

• CallID: Call id on Userplane.

• PSN: This is panopticon sequence number. There is a maximum of four MonSub fastpath tracing sessions on one UP with PSN ranging from 0-3.

• Start time: Time at which MonSub tracing session starts.

• Interface Type: This is to identify the call type, for which MonSub fastpath tracing session is started, whether it is Sxa, Sxb or Sxab.

Disconnect Monitor Subscriber Sessions

Following is the new CLI to disconnect the ongoing MonSub session. You can trigger the CLI from UPF.

monitor subscriber fastpath disconnect sessmgr-instance <UP SMGR Instance ID> session-id <Local Monitor subscriber Session ID at SMGR instance level>

If the MonSub session disconnect is successful, the following message dispalys on console.

Session Disconnected Successfully

If the MonSub session disconnect fails, the following message dispalys on console.

Monitor Subscriber session does not exist

Note	Only security administrator can execute the monitor disconnect CLI.

Hexdump module must be configured to provide operators the provision to configure Files names and Poll timers. The Hexdump module is one of the modules such as—EDR, UDR, and so on, that are part of the CDRMOD functionality. Configure the hexdump in a non-local context such as the ECS context. The local context does not support Hexdump modules.

For more information on Hexdump module and its configuration, refer to the Configuring the Hexdump Module for MonSub in UPF section.

Following section discusses the naming conventions for PCAP files:

Note	Only monitor-subscriber-file-name and rotation options are used in naming PCAP files.

Slowpath File Name Convention

The slowpath file names appear in the following format:

curr_slowpath_{SMGR Mon Sub Session Id}_{monsub_file_name_option_val}_{Timestamp}_{RotationCount}.pcap

or

slowpath_{SMGR Mon Sub Session Id}_{monsub_file_name_option_val}_{Timestamp}_{RotationCount}.pcap

File with ‘curr_’ prefix is the file, that is currently being written to, which is still not closed. When files are to be rotated (depending on the file rotation parameters), file without the ‘curr_’ prefix are copied to hard disk.

The SMGR MonSub Session Id – This is the session Id for MonSub session created on Uplane SMGR instance ID, which created this PCAP. This Id is local to SMGR instance, so there could be two SLOWPATH pcap captured with same ID.

When files are to be copied to hard disk, the monsub_file_name_option_val is replaced by:

• IMSI value if monitor-subscriber-file-name is set to "imsi".

• Call ID value if monitor-subscriber-file-name is set to "call-id"

• Username value if monitor-subscriber-file-name is set to ‘username’

Timestamp is in the following format "MMDDYYYYHHMMSS", where:

• MM - Month, DD - Date and YYYY - Year.

• HH -Hour, MM - Minutes and SS - Seconds.

RotationCount is a 9-digit value that is incremented every time an old file is rotated, and a new file is generated.

Rotation of slowpath files is determined by following option in hexdump-module file configuration:

rotation { num-records number | time seconds| volume bytes }

• num-records: num-records specifies the number of packets after which, a new file is generated and ‘RotationCount’ in the filename is incremented. The range of number is between 100 to 10240, and the default value is 1024.

• time: time specifies the time to wait in seconds before a new file is generated and ‘RotationCount’ in the filename is incremented. seconds must be an integer from 30 through 86400. The default value is 3600.

• volume: volume specifies the number of bytes after which a new file is generated and ‘RotationCount’ in the filename is incremented. bytes must be an integer from 51200 through 62914560. The default value is 102400.

Note	The tarriff-time parameter under rotation is ignored as it is not suitable for PCAP file capture.

The following are examples of the file naming conventions for slowpath PCAP files:

• For the ‘imsi’ option where IMSI is ‘112233445566778’, slowpath files are named as:

  slowpath_S0_112233445566778_07152019050907_000000000.pcap

• For ‘call_id’ option where Call Id is ‘01317b22’, slowpath files are named as:

  slowpath_S0_01317b22_07152019050907_000000000.pcap

Note	The parameter tarrif-time is not applicable for PCAP file capture.

Fastpath File Name Convention

The fastpath file names appear in the following format:

vpp_{S}_{B}_{encap}_{monsub_file_name_option}_{Timestamp}_{FileCount}.pcap

• S is replaced by either ‘S1’, ‘S2’,'S3’, or 'S4’.

• B is replaced by either ‘B0’,‘B1’,'B2', or 'B3' depending on the bundle generated by Panopticon.

• monsub_file_name_option is replaced by:

  • IMSI value if monitor-subscriber-file-name is set to "imsi".

  • Call ID value if monitor-subscriber-file-name is set to "call-id"

  • Username value if monitor-subscriber-file-name is set to ‘username’

Timestamp is in the following format "MMDDYYYYHHMMSS", where:

• MM - Month, DD - Date and YYYY - Year.

• HH -Hour, MM - Minutes and SS - Seconds.

RotationCount is a 9-digit value that is incremented every time an old file is rotated, and a new file is generated.

Fast path "FileCount" is not the same as the slowpath "RotationCount" parameters and hence ‘hexdump-module file rotation’ parameters are ignored while naming fastpath files.

In Phases 1 of the feature, fastpath generated file names are like ‘vpp_S1_B0_ip.pcap’ or ‘vpp_S1_B1_ip.pcap’, they are renamed to following when being copied over to non-volatile storage:

• vpp_S1_B0_ip_01317b22_07152019050907_000000000.pcap

• vpp_S1_B1_ip_01317b22_07152019050908_000000001.pcap

• vpp_S1_B0_ip_01317b22_07152019050908_000000002.pcap

In MonSub phase 3, a PCAP “bundle” is replaced with a single PCAP file that uses Ethernet encapsulation.

In Phase 3, each fastpath session file is captured in the Ethernet PCAP file that is ‘vpp_S0_B0_eth.pcap’ and they are renamed to following when being copied to a non-volatile storage:

vpp_S0_B0_eth_01317b22_07152019050907_000000000.pcap

For ‘callid’ option where Call Id is ‘12345678ef’:

• slowpath_S0_12345678ef _07152019050907_000000000.pcap

• vpp_S1_B0_eth_12345678ef _07152019050907_000000000.pcap

For ‘username’ option where username is ‘9890098900’:

• slowpath_S0_07152019050907_000000000_9890098900.pcap

• vpp_S1_B0_eth_07152019050907_000000000_9890098900.pcap

Fastpath PCAP files are written to the /records/pcap directory in same card and CPU complex where the session manager owns the subscriber session resides.

/records directory is mapped to the "tmpfs" filesystem that is mapped to RAM. In this state, the files are suffixed with a ".pending" extension. For example:

Note	The files size at this stage is not the actual file size when it is written to a persistent storage.

Once the fastpath tracing mechanism has written the files, they are converted to ‘.pcap’ files and renamed as given below. Additionally, there is a file that ends with a ".done" extension:

-rw-rw-r--    1 root     root       8689188 Oct 16 22:06 vpp_S0_B0_eth.pcap

After the PCAP files are written by fastpath tracing mechanism, the Co-Proc functionality instantiates and copies the files to a hard disk or a persistent storage.

The aforementioned file location process for Fastpath is also applicable to Slowpath.

The target file location in all cases is: /hd_raid/records/hexdump, except for the case in the hexdump module configuration where use-harddisk is enabled and the directory option under the hexdump file is to a custom value. For example, if the directory option is set to a value "abc" then the target location for the PCAP file will be: /hd_raid/records/hexdump/abc/.

In this feature implementation, a predefined location is set for PCAP files.

• To make sure that /records/pcap directory is not populated when issues are encountered with the use of use-harddisk and hexdump module configurations.

• For regular cleanup from /hd_raid/records/hexdump directory.

File Transfer to External Location

Once the files have been copied to the hard disk, they can be copied over to an external server using the command: transfer-mode option under the hexdump command in the hexdump-module configuration.

Apart from transfer-mode , other relevant options under hexdump can be used for external file transfer. Operators can use these commands to avoid excessive storage during fastpath processing.

Some of the known limitations of the monitor subscriber feature are:

• This feature does not support SR/ICSR.

• Max of 4 mon sub sessions can be enabled per UPF.

• Monitoring for a given IMSI should not be enabled from both the SMF and the UP console.

• A combo call consists of 2 mon sub-sessions since it involves monitoring 2 PDNs.

• Incoming uplink packets (with GTPU) are not captured in VPP ingress in this phase.

• Restarting a trace immediately after quitting may result in fastpath files in /records/pcap the directory to be overwritten. Therefore, restart the session after a few seconds.

• When a MonSub trace stops, the tear down process may take a few seconds, so it is recommended to wait. It takes a maximum of (5 seconds, hexdump poll timer value in sec) before toggling the MonSub trace to start. Else, the operators may observe MAX TRACING SESSIONS REACHED momentarily.

• Monitor Subscriber fastpath sessions show CLI does not display the sessions that are being stopped. Hence, there is a transient period, where the new MonSub sessions can be rejected due to maximum sessions reached. However, show CLI shows fewer sessions. Therefore, wait for some time before starting a new MonSub trace session.

• Changing fastpath configuration options is only possible when UP Pcap Trace is set to OFF.

• When MT=ON in the Multi-PDN, then once MT=OFF, new PDN tracing is not started due to MAX TRACING REACHED, and then all other tracing is STOPPED. It happens because the first new PDN tracing starts and then all previous PDNs were STOPPED for MT=OFF case.

• Do not launch the same UE MonSub sessions from different CLIs.

• In slowpath PCAP, the egress DL packets do not show the GTP-U header, because the functionality to add GTP-U is available in fastpath. Hence, the ingress and egress DL packets show the duplicates, unless there are some packet modifications like HTTP X-headers applied over the ingress packets.

• Toggling C and D options does not impact the PCAP captures in UPF.

• For Multi-PDN, the fastpath file names do not use the Call ID. As by definition, the multi-PDN case has more than one call ID and hence a higher-level configuration such as IMSI is more suitable for naming the files.

• Only the named options that are explicitly mentioned in this document are supported from the hexdump-module file configuration.

• Number of streams that can be traced in fastpath is limited to 5000. A stream is defined as a TCP or UDP flow. It includes source IP address, destination IP address, source port, and destination port, transport protocol such as, TCP or UDP.

• Fastpath packets cannot be streamed to an external server. They are stored on the hard-disk and transferred either manually or by using transfer-mode options.

• The UP PCAP trace must be set to ON to capture fastpath and slowpath PCAP files.

• MonSub CLI option ‘<SPACE> Pause’ is only to pause console events. There is no impact on other tracing events (slowpath PCAP, fastpath PCAP, and protocol packets tracing in a text file in hexdump format) with this option.

• The UP trace PCAP file does not contain the initial PFCP Sx Request/Response, due to race condition.

• The ICMP Packets and first packets of TCP and UDP streams flow through both slowpath and fastpath. Default values of GTPU (option 26) and User L3 (Option 19) are set to OFF. As a result, these packets are not captured in slowpath captures. If Option 26 is set to ON, then these packets are captured in slowpath PCAP captures. Option 19 has no effect on slowpath PCAP capture.

• Data Events flag must be set to ON, to capture fastpath and slowpath PCAP files.

• The MonSub tracing is not supported for option Next-SAEGW Call on UP.

• The MonSub tracing is not supported for option Next call by APN for Pure-S call type.

• On an ASR-5500 setup with the default value of poll-timer, all the packets may not be captured due to a known issue. To avoid rejection for a large number of packets, change the poll-timer value to the lowest possible (10 ms).

• If context replacement occurs (if the same subscriber reattaches without a detach), then the slowpath captures for the new call remain in the old slowpath files.

Use this configuration to set the frequency of PCAP file capture check.

configure 
   context context_name 
      hexdump-module 
         hexdump monitor-subscriber-poll-timeout poll_timer_value  
         end 

NOTES:

• hexdump monitor-subscriber-poll-timeout : This option specifies how frequently the check for newly captured PCAP files in the volatile storage must be done before they are copied to persistent storage.

• poll_timer_value : Specifies the poll timer value in milliseconds. It must be an integer in the range of 10 ms to 60 seconds. Default: 30 seconds.

  Note	The timer should not be configured with a value less than 5 seconds.

• This option is only applicable when MonSub is enabled for the products that have fastpath functionality - PGW, SAEGW on ASR-5500 and VPC-SI.

Use the following configuration to specify the file name of the PCAP file which contains IMSI, Call ID, or Username.

configure 
   context context_name 
      hexdump-module 
         file rotation { num-records number | tariff-time minute minutes hour hours | time seconds| volume bytes | monitor-subscriber-file-name { imsi | username | call-id }  
         end 

NOTES:

• monitor-subscriber-file-name { imsi | username | call-id } : This option specifies if the name of the captured PCAP files will contain IMSI, Call Id or Username. This option is only applicable on products that have fastpath functionality (PGW, SAEGW on ASR 5500 and VPC-SI) AND only when Monitor Subscriber functionality is enabled. Default: IMSI.

• rotation { num-records number | tariff-time minute minutes hour hours | time seconds | volume bytes } : Specifies when to close a hexdump file and create a new one.

  • num-records number : Specifies the maximum number of records that should be added to a hexdump file. When the number of records in the file reaches this value, the file is complete.

    number must be an integer from 100 through 10240. Default: 1024

  • tariff-time minute minutes hour hours : Specifies to close the current hexdump file and create a new one based on the tariff time (in minutes and hours).

    minutes must be an integer from 0 through 59.

    hours must be an integer from 0 through 23.

  • time seconds : Specifies the period of time to wait (in seconds) before closing the current hexdump file and creating a new one.

    seconds must be an integer from 30 through 86400. Default: 3600

    Important

    Set the rotation time to 30 seconds.

  • volume bytes : Specifies the maximum size of the hexdump file (in bytes) before closing it and creating a new one.

    bytes must be an integer from 51200 through 62914560. Note that a higher setting may improve the compression ratio when the compression keyword is set to gzip. Default: 102400

The following SNMP trap(s) are added in support of the Monitor Subscriber feature:

• MonSubProcessInitFailure: This trap is triggered when MonSub handler process has failed for a particular process and service.