Following are the modifications that are done in the UPF SessMgr to support this feature.

• Provide services to the CLI for enabling or disabling the MonSub tracing.

• Control NPUMgr to connect/start/stop/add/delete streams or TEP bearers and disconnect.

• The SessMgr maintains the PSN from the NPUMgr (as part of CONNECT API) and sub session ID, which is SessMgr (local to SMGR instance) specific. The SessMgr sends all requests with PSN and sub session ID to NPUMgr for a monitor subscriber tracing session.

• Based on the instructions from the CLI, configures panopticon (through NPUMgr) for changes, such as packet size and priority.

• Read the "hex dump module" configurations and store them locally. Pass the relevant parameters (such as filename) to Session Manager Co-Proc.

• Instantiate SessMgr Co-Proc and then instruct it to copy panopticon generated PCAP files to hard disk. Also handle the termination of SessMgr Co-Proc when MonSub session is over.

• Handle file copy message from SessMgr Co-Proc and inform panopticon about the copied bundle.

• If the file copy fails or there are problems with SessMgr Co-Proc instantiation, raises the SNMP alarms.

• Handle the buffer full indications from panopticon and copy the PCAP from the RAM disk to the configured destination directory.

• Capture the control or slowpath packets. Pass them to SessMgr Co-Proc to publish it as a separate PCAP.

• This feature supports a maximum of four monitor subscriber tracing sessions for a UPF instance. The NPUMgr enforces the tracing limit.

• The MonSub tracing session terminates in the absence of no space on hard disk or no hard disk.

• There are C-Proc (file copy and logging) per UPF SessMgr instance, when monitor subscriber tracing is initiated for that SessMgr instance.

• The MonSub session tear down takes time depending on the final poll timer and disconnect responses from Co-Proc/NPUMgr.

For a multi-PDN call, when you start the MonSub with Multi-trace=OFF, then it traces the only one PDN as a part of that MonSub session. When new PDN is initiated, then existing PDN tracing stops and new PDN tracing starts. For this, first the new PDN tracing is started and then existing PDN tracing is stopped and hence new PSN and SessMgr sub-session ID is allocated.

For a multi-PDN call, when you start the MonSub with Multi-trace=ON, then it traces the new PDN as part of new fastpath tracing session (that is MonSub session). Hence, after tracing the four PDN, MonSub CLI shows max tracing session reached. Tracing of the each PDN takes place as a separate MonSub session.

A new mechanism is added to publish the statistics regarding the quality of FASTPATH PCAP capture on MonSub CLI. The new mechanism publishes the statistics whenever it receives the buffer full MEH indication at SessMgr, throttled at every five seconds. The feature supports a maximum of four buffers for a FASTPATH PCAP corresponding to MonSub session. The feature does not publish the statistics by default, and needs to be enabled through debug CLI on UPF.

• debug uplane monsub-stats disabled

• debug uplane monsub-stats enabled

Packet accepted: 14250000          Packet rejected: 62297
Congestion Short Term: 0                     Congestion Longer Term: 0
Throttled: 0                                 PCAP File Transfer Rate: 9.91 mbps

The PCAP file transfer rate is the rate at which copy Co-Proc writes the PCAP from RAM-FS to HD-RAID.

This feature supports the X-Header capture in slowpath PCAP. When UPF inserts the X-Header for Uplink packet, the UPF captures the packet at entry and exit interfaces. So, the exit packet sent to N6 contains the inserted X-header.

When UPF inserts the X-header for Downlink packet, the UPF captures the packet at entry and exit interfaces. So, the exit packet sent to N3 contains the inserted x-header.

Monitor Subscriber CLI – New Options

The following options with their default value are added to existing monitor subscriber command.

UPF Monitor Subscriber CLI

Following are the options:

• W - UP PCAP Trace (ON ): This parameter is used to create PCAP trace for slowpath and fastpath and is only applicable for CUPS.

• U - Mon Display (ON): The non-protocol events (such as statistics and charging information from ECS and so on) are also captured in slowpath PCAP files and are displayed on U-PLANE monitor console.

• V - PCAP Hexdump (ON): This flag must be set to ON to capture the protocol packets in a text file in hexdump format on U-PLANE.

Note	Currently, UP PCAP Trace flag must be set to ON to capture fastpath and slowpath PCAP files on CUPS.

Monitor Subscriber CLI – New Options

The following options with their default value are added to existing monitor subscriber command.

• F - Packet Capture (Full Pkt): Captures all packets from fastpath.

  Using this option, operators can choose between full and partial packet captures. By entering F, the packet capture type can be changed to either full or partial. With partial packet capture, users can enter packet sizes from 1 to 16384 bytes. For example, if input is given as 20, only the first 20 bytes of fastpath packets will be captured and the remaining packets will be dropped.

  Note	When opening the PCAP file, the summary view will display full length of the packet, but the detailed view will show only the truncated packet.

• / - Priority (0): The value is in the range from "0 – Best Effort" to "7 – Guaranteed"

  • 0 - Best Effort

  • 1 - Low

  • 2 - Med-Low

  • 3 - Medium

  • 4 - Med-High

  • 5 - High

  • 6 - Critical

  • 7 - Guaranteed

  Caution

  It is strongly recommended to not change the default value. It can adversely affect the system performance.

• N - MEH Header (OFF) : The MEH header is stripped from the IP packet if this option is configured

Show Monitor Subscriber Sessions

Following is the new CLI to show the ongoing MonSub session.

You can trigger the CLI show monitor subscriber fastpath session all from UPF.

• SessId: This is the local session id for MonSub session on UP Sessmgr.

• CallID: Call id on Userplane.

• PSN: This is panopticon sequence no. There is a maximum of four MonSub fastpath tracing sessions on one UP with PSN ranging from 0-3.

• Start time: Time at which MonSub tracing session starts.

• Interface Type: This is to identify the call type for which MonSub fastpath tracing session was started, whether it is Sxa, Sxb or Sxab.

Disconnect Monitor Subscriber Sessions

Following is the new CLI to disconnect the ongoing MonSub session. You can trigger the CLI from UPF.

monitor subscriber fastpath disconnect sessmgr-instance <UP SMGR Instance ID> session-id <Local Monitor subscriber Session ID at SMGR instance level>

If the MonSub session disconnect is successful, the following message dispalys on console.

Session Disconnected Successfully

If the MonSub session disconnect fails, the following message dispalys on console.

Monitor Subscriber session does not exist

Note	Only security administrator can execute the monitor disconnect CLI.

Hexdump module must be configured to provide operators the provision to configure Files names and Poll timers. The Hexdump module is one of the modules such as—EDR, UDR and so on, that are part of the CDRMOD functionality. It is recommended to configure the hexdump in a non-local context such as the ECS context. Hexdump modules are not supported in local context.

For more information on Hexdump module and its configuration, refer to the Packet Capture (PCAP) Trace chapter in the ASR5500 System Administration Guide

The naming conventions for PCAP files are discussed in the following sections

Note	Only monitor-subscriber-file-name and rotation options are used in naming PCAP files.

Slowpath File Name Convention

The slow path file names appear in the following format:

curr_slowpath_{SMGR Mon Sub Session Id}_{monsub_file_name_option_val}_{Timestamp}_{RotationCount}.pcap

or

slowpath_{SMGR Mon Sub Session Id}_{monsub_file_name_option_val}_{Timestamp}_{RotationCount}.pcap

File with ‘curr_’ prefix is the file, that is currently being written to, that is still not closed. When files are to be rotated (depending on the file rotation parameters), file without the ‘curr_’ prefix are copied to hard disk.

The SMGR MonSub Session Id – This is the session Id for MonSub session created on Uplane SMGR instance ID, which created this PCAP. This Id is local to SMGR instance, so there could be two SLOWPATH pcap captured with same ID.

When files are to be copied to hard disk, The monsub_file_name_option_val is replaced by:

• IMSI value if monitor-subscriber-file-name is set to "imsi".

• Call ID value if monitor-subscriber-file-name is set to "call-id"

• Username value if monitor-subscriber-file-name is set to ‘username’

Timestamp is in the following format "MMDDYYYYHHMMSS", where:

• MM - Month, DD - Date and YYYY - Year.

• HH -Hour, MM - Minutes and SS - Seconds.

RotationCount is a 9-digit value that is incremented every time an old file is rotated, and a new file is generated.

Rotation of slowpath files is determined by following option in hexdump-module file configuration:

rotation { num-records number | time seconds| volume bytes }

• num-records: num-records specifies the number of packets after which a new file is generated and ‘RotationCount’ in the filename is incremented. The range of number is between 100 to 10240, and the default value is 1024.

• time: time specifies the time to wait in seconds before a new file is generated and ‘RotationCount’ in the filename is incremented. seconds must be an integer from 30 through 86400. The default value is 3600.

• volume: volume specifies the number of bytes after which a new file is generated and ‘RotationCount’ in the filename is incremented. bytes must be an integer from 51200 through 62914560. The default value is 102400.

Note	The tarriff-time parameter under rotation is ignored as it is not suitable for PCAP file capture.

The following are examples of the file naming conventions for slowpath PCAP files.

• For the ‘imsi’ option where IMSI is ‘112233445566778’, slowpath files are named as:

  slowpath_S0_112233445566778_07152019050907_000000000.pcap

• For ‘call_id’ option where Call Id is ‘01317b22’, slowpath files are named as:

  slowpath_S0_01317b22_07152019050907_000000000.pcap

Note	The parameter tarrif-time is not applicable for PCAP file capture.

Fastpath File Name Convention

The fastpath file names appear in the following format:

vpp_{S}_{B}_{encap}_{monsub_file_name_option}_{Timestamp}_{FileCount}.pcap

• S is replaced by either ‘S1’, ‘S2’,'S3’, or 'S4’.

• B is replaced by either ‘B0’,‘B1’,'B2', or 'B3' depending on the bundle generated by Panopticon.

• monsub_file_name_option is replaced by:

  • IMSI value if monitor-subscriber-file-name is set to "imsi".

  • Call ID value if monitor-subscriber-file-name is set to "call-id"

  • Username value if monitor-subscriber-file-name is set to ‘username’

Timestamp is in the following format "MMDDYYYYHHMMSS", where:

• MM - Month, DD - Date and YYYY - Year.

• HH -Hour, MM - Minutes and SS - Seconds.

RotationCount is a 9-digit value that is incremented every time an old file is rotated, and a new file is generated.

Fast path "FileCount" is not the same as the slowpath "RotationCount" parameters and hence ‘hexdump-module file rotation’ parameters are ignored while naming fastpath files.

In Phases 1 of the feature, fastpath generated file names are like ‘vpp_S1_B0_ip.pcap’ or ‘vpp_S1_B1_ip.pcap’, they are renamed to following when being copied over to non-volatile storage:

• vpp_S1_B0_ip_01317b22_07152019050907_000000000.pcap

• vpp_S1_B1_ip_01317b22_07152019050908_000000001.pcap

• vpp_S1_B0_ip_01317b22_07152019050908_000000002.pcap

In MonSub phase 3, a PCAP “bundle” is replaced with a single PCAP file that uses Ethernet encapsulation.

In Phase 3, each fastpath session file is captured in the Ethernet PCAP file that is ‘vpp_S0_B0_eth.pcap’ and they are renamed to following when being copied to a non-volatile storage:

vpp_S0_B0_eth_01317b22_07152019050907_000000000.pcap

For ‘callid’ option where Call Id is ‘12345678ef’:

• slowpath_S0_12345678ef _07152019050907_000000000.pcap

• vpp_S1_B0_eth_12345678ef _07152019050907_000000000.pcap

For ‘username’ option where username is ‘9890098900’:

• slowpath_S0_07152019050907_000000000_9890098900.pcap

• vpp_S1_B0_eth_07152019050907_000000000_9890098900.pcap

Fastpath PCAP files are written to the /records/pcap directory in same card and CPU complex where the SMGR owns the subscriber session resides.

/records directory is mapped to the "tmpfs" filesystem that is mapped to RAM. In this state, the files are suffixed with a ".pending" extension. For example:

Note	The files size at this stage is not the actual file size when it is written to a persistent storage.

Once the fastpath tracing mechanism has written the files, they are converted to ‘.pcap’ files and renamed as given below. Additionally, there is a file that ends with a ".done" extension:

-rw-rw-r--    1 root     root       8689188 Oct 16 22:06 vpp_S0_B0_eth.pcap

After the PCAP files are written by fastpath tracing mechanism, the Co-Proc functionality instantiates and copies the files to a hard disk or a persistent storage.

The above file location process for Fastpath is also applicable to Slowpath.

The target file location in all cases is: /hd_raid/records/hexdump, except for the case in the hexdump module configuration where use-harddisk is enabled and the directory option under the hexdump file is to a custom value. For example, if the directory option is set to a value "abc" then the target location for the PCAP file will be: /hd_raid/records/hexdump/abc/.

In this feature implementation, a predefined location is set for PCAP files.

• To make sure that /records/pcap directory is not populated when issues are encountered with the use of use-harddisk and hexdump module configurations.

• For regular cleanup from /hd_raid/records/hexdump directory.

File Transfer to External Location

Once the files have been copied to the hard disk, they can be copied over to an external server using the command: transfer-mode option under the hexdump command in the hexdump-module configuration.

Apart from transfer-mode , other relevant options under hexdump can be used for external file transfer. Operators can use these commands to avoid excessive storage during fastpath processing.

Following are the Limitations:

• SR/ICSR is not supported for this feature.

• Restarting trace immediately after quitting may result in fastpath files in /records/pcap directory to be overwritten. We recommend you to restart the session after a brief moment (a few seconds).

• When MonSub trace is stopped, the tear down process can take a few seconds, so it is recommended to wait for few seconds. A maximum of (5 sec, hexdump poll timer value in sec) before toggling the MonSub trace to start, else operators may observe MAX TRACING SESSIONS REACHED momentarily.

• Show monitor subscriber fastpath sessions CLI does not display the MonSub sessions that are being stopped. Hence there is a transient period where new MonSub sessions can be rejected due to max sessions reached, whereas show CLI shows fewer sessions. We recommend you to wait for some time before starting a new MonSub trace session.

• Changing fastpath configuration options is only possible when UP Pcap Trace is set to OFF.

• When MT=ON in the Multi-PDN, then once MT=OFF, new PDN tracing is not started due to MAX TRACING REACHED, and then all other tracing is STOPPED. This is because the first new PDN tracing is started and then all previous PDNs were STOPPED for MT=OFF case.

• We recommend not to launch the same UE MonSub sessions from different CLIs.

• In slowpath PCAP, the egress DL packets do not show the GTPU-U header because the functionality to add GTP-U is with fastpath. So, ingress and egress DL packets shows up the duplicates, unless there is some packet modification like HTTP X-headers applied over the ingress packets.

• Toggling C and D options does not impact the PCAP captures in UPF.

• For Multi-PDN, the fastpath filenames do not use the Call ID, because, by definition the multi-PDN case has more than one call ID and hence a higher-level configuration such as IMSI is more suitable for naming the files.

• Only the named options that are explicitly mentioned in this document are supported from hexdump-module file configuration.

• Number of streams that can be traced in fastpath is limited to 5000. Stream is defined as a TCP or UDP flow which is made up of {source IP address, destination IP address, source port, and destination port, transport protocol such as TCP or UDP).

• Fastpath packets cannot be streamed to an external server. They are stored on the hard-disk and transferred (either manually) or by using transfer-mode options.

• The UP PCAP trace must be set to ON to capture fastpath and slowpath PCAP files.

• MonSub CLI option ‘<SPACE> Pause’ is only to pause console events. There is no impact on other tracing events (slowpath PCAP, fastpath PCAP and protocol packets tracing in a text file in hexdump format) with this option.

• The UP trace PCAP file does not contain the initial PFCP Sx Request/Response, due to race condition.

• The ICMP Packets and first packets of TCP and UDP streams flow through both slowpath and fastpath. Default values of GTPU (option 26) and User L3 (Option 19) are set to OFF. As a result, these packets are not captured in slowpath captures. If Option 26 is set to ON, then these packets are captured in slowpath PCAP captures. As mentioned in previous point, option 19 has no effect on slowpath PCAP capture.

• Data Events flag must be set to ON to capture fastpath and slowpath PCAP files.

• The Mon sub tracing is not supported for option Next-SAEGW Call on UP.

• The Mon sub tracing is not supported for option Next call by APN for Pure-S call type.

• On ASR-5500 setup with the default value of poll-timer, all the packets may not be captured due to a known issue. To avoid large number of packets to be rejected, we recommend you to change the poll-timer value to the lowest possible (10 ms).

• If context replacement occurs (if the same subscriber reattaches without a detach), then the slowpath captures for the new call remain in the old slowpath files.

Use this configuration to set the frequency of PCAP file capture check.

configure 
   context context_name 
      hexdump-module 
         hexdump monitor-subscriber-poll-timeout poll_timer_value  
         end 

NOTES:

• hexdump monitor-subscriber-poll-timeout : This option specifies how frequently the check for newly captured PCAP files in the volatile storage must be done before they are copied to persistent storage.

• poll_timer_value : Specifies the poll timer value in milliseconds. It must be an integer in the range of 10 ms to 60 seconds. Default: 30 seconds.

  Note	It is strongly recommended to not configure the timer with a value less than 5 seconds.

• This option is only applicable when MonSub is enabled for the products that have fastpath functionality - PGW, SAEGW on ASR-5500 and VPC-SI.

Use the following configuration to specify the file name of the PCAP file which contains IMSI, Call ID or Username.

configure 
   context context_name 
      hexdump-module 
         file rotation { num-records number | tariff-time minute minutes hour hours | time seconds| volume bytes | monitor-subscriber-file-name { imsi | username | call-id }  
         end 

NOTES:

• monitor-subscriber-file-name { imsi | username | call-id } : This option specifies if the name of the captured PCAP files will contain IMSI, Call Id or Username. This option is only applicable on products that have fastpath functionality (PGW, SAEGW on ASR 5500 and VPC-SI) AND only when Monitor Subscriber functionality is enabled. Default: IMSI.

• rotation { num-records number | tariff-time minute minutes hour hours | time seconds | volume bytes } : Specifies when to close a hexdump file and create a new one.

  • num-records number : Specifies the maximum number of records that should be added to a hexdump file. When the number of records in the file reaches this value, the file is complete.

    number must be an integer from 100 through 10240. Default: 1024

  • tariff-time minute minutes hour hours : Specifies to close the current hexdump file and create a new one based on the tariff time (in minutes and hours).

    minutes must be an integer from 0 through 59.

    hours must be an integer from 0 through 23.

  • time seconds : Specifies the period of time to wait (in seconds) before closing the current hexdump file and creating a new one.

    seconds must be an integer from 30 through 86400. Default: 3600

    Important

    It is recommended to set the rotation time to 30 seconds.

  • volume bytes : Specifies the maximum size of the hexdump file (in bytes) before closing it and creating a new one.

    bytes must be an integer from 51200 through 62914560. Note that a higher setting may improve the compression ratio when the compression keyword is set to gzip. Default: 102400

The following SNMP trap(s) are added in support of the Monitor Subscriber feature:

• MonSubProcessInitFailure: This trap is triggered when MonSub handler process has failed for a particular process and service.