Common Execution Environment

Overview

The Common Execution Environment (CEE) is a software solution developed for monitoring mobile and cable applications that are deployed on the Subscriber Microservices Infrastructure (SMI). The CEE captures information (key metrics) from the applications in a centralized way for engineers to debug and troubleshoot.

The CEE is the common set of tools that are installed for all the applications. It comes equipped with a dedicated Ops Center, which provides the user interface (Command Line Interface) and APIs for managing the monitoring tools. There is only one CEE available for each cluster.

The CEE includes the following components:

  • CEE Ops Center - The CEE Ops Center allows users to configure and install the CEE. The CEE Ops Center contains the following components:

    • Metrics Collection - It includes functions such as reporting from Prometheus, alerting and Bulk statistics and so on.

    • Metrics Visualization - The metrics are displayed to the end users through a Grafana dashboard. The dashboard displays the key metrics such as CPU usage, memory, and disk input and output (I/O) utilization of each application deployed on the SMI. Use cases include:

      • Import custom Grafana dashboard from a GIT repository.

      For more information, refer Grafana section.

    • Bulk Statistics - Configures application specific statistics, which are collected through Gather TAC feature. The Bulk Statistics are automatically generated based on the user requirements at repeated intervals. Use cases include:

      • Generate query for current PDU per 4G session.

      • Generate query for current PDU per 4G IPv6 session.

      • View bulk statistics.

      For more information, refer to the Configuring Bulk Statistics section.

    • Metrics Global Query - Thanos - a set of software components for metric system - provides the ability to perform global query across multiple clusters. Use cases include:

      • In cable environment with multiple Kubernetes clusters, where instances of Prometheus collect metrics specific to cluster, global Prometheus instance (set up as a part of an application Ops Center) is used as focal point to gather data and respond to query for metrics from all Prometheus pods.

      For more information, refer Cluster Monitoring section.

    • Alerting - Enables you to monitor applications, containers or nodes by setting up alert rules. The CEE uses the Prometheus Alert Manager for generating alerts. Use cases include:

      • Monitor the success rate of SMF session creation by configuring Prometheus alert rule to report if session creation is less than threshold.

      • Configure Prometheus alert rule to report if pod has restarted.

      • Alerts addon: If snmp-trapper is configured, alert is also sent as SNMP Trap to the receiving agent.

      • View active alerts.

      • View alerts history.

      For more information, refer to the Configuring Alerts section.

    • Log Monitoring - The Kubetail utility in the CEE Ops Center allows end users to monitor the logs of an application in real time.

    • Log Forwarding - The Log Forwarding function collects and forwards all the logs to any of the third-party applications present in the customer infrastructure. Use cases include:

      • Configure log forwarding to an external Splunk server.

      • Configure log forwarding to an external Fluent-D or Fluent-Bit instance, where logs can be streamed to supporting application such as ElasticSearch.

      For more information, refer Log Forwarding section.

    • Gather TAC - The Gather TAC function is used for creating log files at specified intervals of time. The logs are collected based on the pods that are deployed on the Kubernetes cluster. Use cases include:

      • When a Network Function (NF) exhibits some issues, the log collection can be configured to include data and statistics for the system and pods in a specific namespace within the last few hours.

    For more information, refer Gather TAC section.

CEE Installation

This section describes the procedures involved in installing the CEE using the Ops Center.

Prerequisites

The prerequisites for installing the CEE are:

  1. Installing the SMI Cluster Manager.

  2. Storing the CEE and associated product tarballs in the local repository.

  3. Applying the necessary cluster configuration for bringing the Kubernetes Cluster on the target nodes.

Requirements

All the versions of CEE.

Components Used

The following components are used for installing the CEE:

  1. The SMI Cluster Manager.

  2. The SMI CEE.

Installing CEE

Smart Software Licensing

Smart Licensing is a cloud-based approach to licensing that simplifies the purchase, deployment, and management of Cisco software assets. Entitlements are purchased through your Cisco account via Cisco Commerce Workspace (CCW) and immediately deposited into your Virtual Account for usage. This eliminates the need to install license files on every device. Products that are smart enabled communicate directly to Cisco to report consumption. A single location is available to customers to manage Cisco software licenses — the Cisco Software Central (CSC). License ownership and consumption are readily available to help make better purchase decision based on consumption or business need. See https://www.cisco.com/c/en/us/buy/smart-accounts/software-licensing.html for more information about Cisco Smart Licensing.

Comparison Between Legacy Licensing and Smart Licensing

Cisco employs two types of license models - Legacy Licensing and Smart Software Licensing. Legacy Licensing consists of software activation by installing Product Activation Keys (PAK) on to the Cisco product. A Product Activation Key is a purchasable item, ordered in the same manner as other Cisco equipment and used to obtain license files for feature set on Cisco Products. Smart Software Licensing is a cloud based licensing of the end-to-end platform through the use of a few tools that authorize and deliver license reporting. Smart Software Licensing functionality incorporated into CEE complete the product registration and authorization.

Cisco Software Central

Cisco Software Central (CSC) enables the management of software licenses and Smart Account from a single portal. The interface allows you to activate your product, manage entitlements, and renew and upgrade software. A functioning Smart Account is required to complete the registration process. To access the Cisco Software Central, see https://software.cisco.com.

Smart Accounts/Virtual Accounts

A Smart Account provides a single location for all Smart-enabled products and entitlements. It helps speed procurement, deployment, and maintenance of Cisco Software. When creating a Smart Account, you must have the authority to represent the requesting organization. After submitting, the request goes through a brief approval process.

A Virtual Account exists as a sub-account withing the Smart Account. Virtual Accounts are a customer-defined structure based on organizational layout, business function, geography or any defined hierarchy. They are created and maintained by the Smart Account administrator.

See https://software.cisco.com to learn about, set up, or manage Smart Accounts.

Request a Cisco Smart Account

A Cisco Smart Account is an account where all products enabled for Smart Licensing are deposited. A Cisco Smart Account allows you to manage and activate your licenses to devices, monitor license use, and track Cisco license purchases. Through transparent access, you have a real-time view into your Smart Licensing products. IT administrators can manage licenses and account users within your organization's Smart Account through the Software Central.

Procedure


Step 1

In a browser window, enter the following URL:

https://software.cisco.com

Step 2

Log in using your credentials, and then click Request a Smart Account in the Administration area.

The Smart Account Request window is displayed.

Step 3

Under Create Account, select one of the following options:

  • Yes, I have authority to represent my company and want to create the Smart Account – If you select this option, you agree to authorization to create and manage product and service entitlements, users, and roles on behalf of your organization.

  • No, the person specified below will create the account – If you select this option, you must enter the email address of the person who will create the Smart Account.

Step 4

Under Account Information:

  1. Click Edit beside Account Domain Identifier.

  2. In the Edit Account Identifier dialog box, enter the domain, and click OK. By default, the domain is based on the email address of the person creating the account and must belong to the company that will own this account.

  3. Enter the Account Name (typically, the company name).

Step 5

Click Continue.

The Smart Account request will be in pending status until it has been approved by the Account Domain Identifier. After approval, you will receive an email confirmation with instructions for completing the setup process.

CEE Smart Licensing

At present, the Smart Licensing feature supports application entitlement for online and offline licensing for CEE. The application usage is unrestricted during all stages of licensing including Out of Compliance (OOC) and expired stages.


Note

A 90 day evaluation period is granted for all licenses in use. Currently, the functionality and operation of the CEE is unrestricted even after the end of the evaluation period.


Software Tags and Entitlement Tags

Tags for the following software and entitlements have been created to identify, report, and enforce licenses.

Software Tags

Software tags uniquely identify each licenseable software product or product suite on a device. The following software tags exist for the CEE.

Product Type / Description Software Tag

Ultra Cloud Core - Subscriber Microservices Infrastructure (SMI), Basic

regid.2020-04.com.cisco.SMI,1.0_d679f8dd-6cf2-4fe1-9dfc-450650c50301

Ultra Cloud Core - Subscriber Microservices Infrastructure (SMI), Plus

regid.2020-04.com.cisco.SMI,1.0_d679f8dd-6cf2-4fe1-9dfc-450650c50301

Entitlement Tags

The following entitlement tags identify licenses in use:

Product Type / Description Entitlement Tag

Ultra Cloud Core - Subscriber Microservices Infrastructure (SMI), Basic

regid.2020-04.com.cisco.SMI_BASE,1.0_d1771c37-9daf-4b4a-b809-e0c77de37545

Ultra Cloud Core - Subscriber Microservices Infrastructure (SMI), Plus

regid.2020-04.com.cisco.SMI_PLUS,1.0_77b9f9cf-1349-4f75-a106-04efb6bba944


Note

The license information are retained during software upgrades and rollback.


Configuring Smart Licensing

You can configure Smart Licensing after a new CEE deployment.

Users with Access to CSC

This section describes the procedure involved in configuring Smart Licensing for users with access to CSC portal from their internal environment.

Setting Up the Product and Entitlement in CSC

Before you begin, you need to setup your product and entitlement in the CSC. To setup your product and entitlement:

  1. Log in to your CSC account.

  2. Click Add Product and enter the following details.

    • Product name – Specify the name of the deployed product. For example, CEE.

    • Primary PM CEC ID – Specify the primary Project Manager's CEC ID for the deployed product.

    • Dev Manager CEC ID – Specify the Development Manager's CEC ID for the deployed product.

    • Description (Optional) – Specify a brief description of the deployed product.

    • Product Type – Specify the product type.

    • Software ID Tag – Specify the software ID Tag provided by the Cisco Account's team.

  3. Click Create.

  4. Select your product from the Product/Entitlement Setup grid.

  5. Click Entitlement drop-down and select Create New Entitlement.

  6. Select New Entitlement in Add Entitlement and enter the following details.

    • Entitlement Name – Specify the license entitlement name. For example, SMI_BASE.

    • Description (Optional) – Specify a breif description about the license entitlement.

    • Entitlement Tag – Specify the entitlement tag provided by the Cisco Account's team.

    • Entitlement Type – Specify the type of license entitlement.

    • Vendor String – Specify the vendor name.

  7. Click Entitlement Allocation.

  8. Click Add Entitlement Allocation.

  9. In New License Allocation, provide the following details:

    • Product – Select your product from the drop-down list.

    • Entitlement – Select your entitlement from the drop-down list.

  10. Click Continue.

  11. In New License Allocation window, provide the following details:

    • Quantity – Specify the number of licenses.

    • License Type – Specify the type of license.

    • Expiring Date – Specify the date of expiry for the license purchased.

  12. Click Create.

  13. Verify the status of Smart Licensing using the following command.

    show license all 

    Example:

    CEE# show license all
    
    Smart Licensing Status
    =======================
    Smart Licensing is ENABLED
    
    Registration:
      Status: UNREGISTERED
      Export-Controlled Functionality: Not Allowed
    
    License Authorization:
      Status: EVAL MODE
      Evaluation Period Remaining: 83 days, 0 hr, 15 min, 8 sec
      Last Communication Attempt: NONE
    
    License Conversion: 
     Automatic Conversion Enabled: true
     Status: NOT STARTED
    
    Utility:
      Status: DISABLED
    
    Transport:
      Type: CALLHOME
    
    Evaluation Period:
      Evaluation Mode: In Use
      Evaluation Period Remaining: 83 days, 0 hr, 15 min, 8 sec
    
    License Usage
    =============
    License Authorization Status: EVALUATION MODE
      Evaluation Period Remaining: 83 days, 0 hr, 15 min, 8 sec
    
    UCC SMI BASE (SMI_BASE)
      Description: Ultra Cloud Core - Subscriber Microservices Infrastructure (SMI), Base Minimum
      Count: 1
      Version: 1.0
      Status: EVAL MODE
      Export status: RESTRICTED_NOTALLOWED
      Feature Name: <empty>
      Feature Description: <empty>
    
    Product Information
    ===================
    UDI: PID:SMI,SN:6GKJ2OA-NMUWA7Y
    
    Agent Version
    =============
    Smart Agent for Licensing: 3.0.13
    
Registering Smart Licensing

You need to register the product entitled to the license with CSC. To register, you need to generate a ID token from CSC.

  1. Log in to your CSC account.

  2. Click General > New Token and enter the following details:

    • Description – Specify a brief description about the ID token.

    • Expires After – Specify the number of days for the token to expire.

    • Max. Number Users – Specify the maximum number users.

  3. Click Create Token.

  4. Select new ID token in Product Instance Registration Token.

  5. Click Actions > Copy.

  6. Log in to CEE Ops Center CLI and paste the ID token using the following configuration:

    license smart register idtoken 

    Example:

    CEE# license smart register
    Value for 'idtoken' (<string>): MTI2Y2FlNTAtOThkMi00YTAxLWE4M2QtOTNhNzNjNjY4ZmFiLTE2MTc4N
    Tky%0AMTA5MDh8ck1jUHNwc3k1ZC9nWFFCSnVEcUp4QU1jTFoxOGxDTU5kQ3lpa25E%0Ab04wST0%3D%0A
    CEE# 
    
  7. Verify the status of Smart Licensing using the following command.

    show license all 

    Example:

    CEE# show license all 
    
    Smart Licensing Status
    =======================
    Smart Licensing is ENABLED
    
    Registration:
      Status: REGISTERED
      Smart Account: Cisco Systems, Inc.
      Virtual Account: CEE-SMF
      Export-Controlled Functionality: Allowed
      Initial Registration: SUCCEEDED on Apr 15 05:45:07 2020 GMT
      Last Renewal Attempt: SUCCEEDED on Apr 15 05:45:07 2020 GMT
      Next Renewal Attempt: Oct 12 05:45:07 2020 GMT
      Registration Expires: Apr 15 05:40:31 2021 GMT
    
    License Authorization:
      Status: AUTHORIZED on Apr 15 05:45:12 2020 GMT
      Last Communication Attempt: SUCCEEDED on Apr 15 05:45:12 2020 GMT
      Next Communication Attempt: May 15 05:45:12 2020 GMT
      Communication Deadline: Jul 14 05:40:40 2020 GMT
    
    License Conversion: 
     Automatic Conversion Enabled: true
     Status: NOT STARTED
    
    Utility:
      Status: DISABLED
    
    Transport:
      Type: CALLHOME
    
    Evaluation Period:
      Evaluation Mode: Not In Use
      Evaluation Period Remaining: 83 days, 0 hr, 10 min, 43 sec
    
    License Usage
    =============
    License Authorization Status: AUTHORIZED as of Apr 15 05:45:12 2020 GMT
    
    UCC SMI BASE (SMI_BASE)
      Description: Ultra Cloud Core - Subscriber Microservices Infrastructure (SMI), Base Minimum
      Count: 1
      Version: 1.0
      Status: AUTHORIZED
      Export status: RESTRICTED_ALLOWED
      Feature Name: <empty>
      Feature Description: <empty>
    
    Product Information
    ===================
    UDI: PID:SMI,SN:6GKJ2OA-NMUWA7Y
    
    Agent Version
    =============
    Smart Agent for Licensing: 3.0.13
    

    NOTES:

    • license smart register – Registers Smart Licensing with CSC.

    • idtoken – Specifies the ID token generated from CSC.

Deregistering Smart Licensing

You can deregister the registered product from Smart Licensing if required.

  1. Log in to CEE Ops Center CLI and use the following configuration:

    license smart deregister 

    Example:

    CEE# license smart deregister
    CEE# 
    
  2. Verify the status of Smart Licensing using the following command.

    show license all 

    Example:

    CEE# show license all
    
    Smart Licensing Status
    =======================
    Smart Licensing is ENABLED
    
    Registration:
      Status: UNREGISTERED
      Export-Controlled Functionality: Not Allowed
    
    License Authorization:
      Status: EVAL MODE
      Evaluation Period Remaining: 83 days, 0 hr, 10 min, 43 sec
      Last Communication Attempt: NONE
    
    License Conversion: 
     Automatic Conversion Enabled: true
     Status: NOT STARTED
    
    Utility:
      Status: DISABLED
    
    Transport:
      Type: CALLHOME
    
    Evaluation Period:
      Evaluation Mode: In Use
      Evaluation Period Remaining: 83 days, 0 hr, 10 min, 43 sec
    
    License Usage
    =============
    License Authorization Status: EVALUATION MODE
      Evaluation Period Remaining: 83 days, 0 hr, 10 min, 43 sec
    
    UCC SMI BASE (SMI_BASE)
      Description: Ultra Cloud Core - Subscriber Microservices Infrastructure (SMI), Base Minimum
      Count: 1
      Version: 1.0
      Status: EVAL MODE
      Export status: RESTRICTED_NOTALLOWED
      Feature Name: <empty>
      Feature Description: <empty>
    
    Product Information
    ===================
    UDI: PID:SMI,SN:6GKJ2OA-NMUWA7Y
    
    Agent Version
    =============
    Smart Agent for Licensing: 3.0.13
    
    CEE#
    

    NOTES:

    • license smart deregister – Deregisters Smart Licensing from CSC.

Users without Access to CSC

The Smart License Reservation feature – Perpetual Reservation – is reserved for customers without access to CSC from their internal environments. With this feature, Cisco allows customers to reserve licenses from their virtual account and tie them to their devices Unique Device Identifier (UDI). This enables customers to use their devices with reserved licenses in a disconnected mode.

The subsequent sections describe the procedure involved in reserving Smart License for users without access to CSC from their internal environment.

Enabling Smart License Reservation

You can enable Smart License reservation through CEE Ops Center CLI.

  1. Log in to CEE Ops Center CLI and use the following configuration:

    configure terminal 
      license smart reservation 
      commit 
      exit 

    Notes:

    • license smart reservation – Enables license reservation.

Generating Smart License Reservation Request Code

You can generate the Smart License reservation request code through CEE Ops Center CLI.

  1. Log in to CEE Ops Center CLI and using the following configuration to enable the reservation:

    configure terminal 
      license smart reservation  
      commit 
      exit 
  2. Use the following configuration to request a reservation code:

    license smart reservation request 

    Example:

    CEE# license smart reservation request
    reservation-request-code CJ-ZCEE:6GKJ2OA-NMUWA7Y-Ai75GxtBs-3B
    CEE# 
    Message from confd-api-manager at 2020-04-15 05:51:37...
    Global license change NotifyReservationInProgress reason code Success - Successful.
    CEE# 
    

    NOTES:

    • license smart reservation – Enables license reservation request code.

    • license smart reservation request – Generates the license reservation request code.


      Important

      You need to copy the generated license request code from the CEE Ops Center CLI.


Generating an Authorization Code from CSC

You can generate an authorization code from CSC using the license reservation request code.

  1. Log in to your CSC account.

  2. Click License Reservation.

  3. Enter the Request Code: Paste the license reservation request code copied from the CEE Ops Center CLI in the Reservation Request Code text-box.

  4. Select the Licenses: Click Reserve a Specific License radio-button and select UCC SMI BASE.


    Note

    In the Reserve text-box enter the value 1.


  5. Review your selection.

  6. Click Generate Authorization Code.

  7. Download the response file: The authorization code is generated and displayed on-screen. Click Download as File to download the authorization code.

  8. Click Close.

Reserving Smart Licensing

You can reserve Smart License for the deployed product using the authorization code generated in CSC.

  1. Log in to CEE Ops Center CLI and use the following configuration:

    license smart reservation install 
    authorization_code 

    Example:

    CEE# license smart reservation install
    Value for 'key' (<string>): <specificPLR><authorizationCode><flag>A</flag><version>C</version>
    <piid>35757dc6-2bdf-4fa1-ba7e-4190f5b6ea22</piid><timestamp>1586929992297</timestamp>
    <entitlements><entitlement><tag>regid.2020-04.com.cisco.SMI_BASE,1.0_60b1da6f-3832-4687-90c9-8879dc815a27</tag>
    <count>1</count><startDate>2020-Apr-08 UTC</startDate><endDate>2020-Oct-05 UTC</endDate>
    <licenseType>TERM</licenseType><displayName>UCC SMI BASE</displayName>
    <tagDescription>Ultra Cloud Core - Subscriber Microservices Infrastructure (SMI), Base Minimum</tagDescription>
    <subscriptionID></subscriptionID></entitlement></entitlements></authorizationCode>
    <signature>MEYCIQC/9v5LpgFoEk2l4omIgjjk83g5WXjzs09kQnsO8D0jRgIhAMh+D6DRuYmqh1TlfJoZxNte0fPKw6fHEY5CEF3+kPQj</signature>
    <udi>P:SMI,S:6GKJ2OA-NMUWA7Y</udi></specificPLR>
    CEE# 
    
  2. Verify the status of smart licensing using the following command.

    show license all 

    Example:

    CEE# show license all
    
    Smart Licensing Status
    =======================
    Smart Licensing is ENABLED
    License Reservation is ENABLED
    
    Registration:
      Status: REGISTERED - SPECIFIC LICENSE RESERVATION 
      Export-Controlled Functionality: Allowed
      Initial Registration: SUCCEEDED on Wed Apr 15 05:53:31 GMT 2020
      Last Renewal Attempt: None
    
    License Authorization:
      Status: AUTHORIZED - RESERVED on Wed Apr 15 05:53:31 GMT 2020 
    
    Utility:
      Status: DISABLED
    
    Transport:
      Type: CALLHOME
    
    Evaluation Period:
      Evaluation Mode: Not In Use
      Evaluation Period Remaining: 83 days, 0 hr, 5 min, 15 sec
    
    License Usage
    =============
    License Authorization Status: 
      Status: AUTHORIZED - RESERVED  on Wed Apr 15 05:53:31 GMT 2020
      Last Communication Attempt: SUCCEEDED on Apr 15 05:53:31 2020 GMT
      Next Communication Attempt: NONE
      Communication Deadline: NONE
    
    UCC SMI BASE (SMI_BASE)
      Description: Ultra Cloud Core - Subscriber Microservices Infrastructure (SMI), Base Minimum
      Count: 1
      Version: 1.0
      Status: AUTHORIZED 
      Export status: NOT RESTRICTED
      Feature Name: <empty>
      Feature Description: <empty>
      Reservation:
        Reservation Status: SPECIFIC INSTALLED
        Total Reserved Count: 1
        Term expiration: 2020-Oct-05 GMT
    
    Product Information
    ===================
    UDI: PID:SMI,SN:6GKJ2OA-NMUWA7Y
    
    Agent Version
    =============
    Smart Agent for Licensing: 3.0.13
    

    NOTES:

    • license smart reservation install authorization_code – Installs a Smart License Authorization code.

Returning the Reserved License

You can return the reserved license to CSC if required. Use the following procedures to return the reserved license:

  1. When the license reservation authorization code is installed in the CEE Ops Center.

    1. Log in to the CEE Ops Center CLI and use the following configuration:

      license smart reservation return 

      Example:

      CEE# license smart reservation return 
      reservation-return-code CJ6m3k-RAvu6b-hMNmwf-mrdcko-NoSwKL-tF7orz-9aNtEu-yVjGAm-D6j
      CEE# 
      
    2. Copy the license reservation return code generated in CEE Ops Center CLI.

    3. Log in to your CSC account.

    4. Select your product instance from the list.

    5. Click Actions > Remove.

    6. Paste the license reservation return code in Return Code text-box.

    NOTES:

    • license smart reservation return – Returns a reserved Smart License.

  2. When the license reservation authorization code is not installed in the CEE Ops Center.

    1. Log in to the CEE Ops Center CLI and use the following configuration to generate the return code.

      license smart reservation return 
      authorization_code 

      Important

      Paste the license reservation authorization code generated in CSC to generate the return code.


    2. Log in to your CSC account.

    3. Select your product instance from the list.

    4. Click Actions > Remove.

    5. Paste the license reservation return code in Return Code text-box.

  3. Verify the status of smart licensing using the following command.

    show license all 
    Example:
    CEE# show license all
    
    Smart Licensing Status
    =======================
    Smart Licensing is ENABLED
    License Reservation is ENABLED
    
    Registration:
      Status: UNREGISTERED
      Export-Controlled Functionality: Not Allowed
    
    License Authorization:
      Status: EVAL MODE
      Evaluation Period Remaining: 83 days, 0 hr, 5 min, 15 sec
      Last Communication Attempt: SUCCEEDED on Apr 15 05:53:31 2020 GMT
      Next Communication Attempt: NONE
      Communication Deadline: NONE
    
    License Conversion: 
     Automatic Conversion Enabled: true
     Status: NOT STARTED
    
    Utility:
      Status: DISABLED
    
    Transport:
      Type: CALLHOME
    
    Evaluation Period:
      Evaluation Mode: In Use
      Evaluation Period Remaining: 83 days, 0 hr, 5 min, 15 sec
    
    License Usage
    =============
    License Authorization Status: EVALUATION MODE
      Evaluation Period Remaining: 83 days, 0 hr, 5 min, 15 sec
    
    UCC 5G SMI BASE (SMI_BASE)
      Description: Ultra Cloud Core - Subscriber Microservices Infrastructure (SMI), Base Minimum
      Count: 1
      Version: 1.0
      Status: EVAL MODE
      Export status: RESTRICTED_NOTALLOWED
      Feature Name: <empty>
      Feature Description: <empty>
    
    Product Information
    ===================
    UDI: PID:CEE,SN:6GKJ2OA-NMUWA7Y
    
    Agent Version
    =============
    Smart Agent for Licensing: 3.0.13
    
    CEE#
    

Monitoring and Troubleshooting Smart Licensing

You can use the following show commands to display information about Smart Licensing in the CEE Ops Center.

show licesne [all | UDI | displaylevel | reservation | smart | status | summary | tech-support | usage] 

NOTES:

  • all – Displays an overview of Smart Licensing information that includes license status and, usage, product information and Smart Agent version.

  • UDI – Displays Unique Device Identifiers (UDI) details.

  • displaylevel – Depth to display information.

  • reservation – Displays Smart Licensing reservation information.

  • smart – Displays Smart Licensing information.

  • status – Displays the overall status of Smart Licensing.

  • summary – Displays a summary of Smart Licensing.

  • tech-support – Displays Smart Licensing debugging information.

  • usage – Displays the license usage information for all the entitlements that are currently in use.

Accessing CEE Ops Center

You can access the CEE Ops Center CLI through the ingress URL. For example:


https://cli.cee-global-ops-center.<ip_address>.<customer_specific_domain_name> 

Upgrading CEE

This section describes the procedure involved in upgrading the CEE Ops Center and CEE products.

Upgrading CEE Ops Center

To upgrade the CEE Ops Center, use the following configurations:

  1. Download the tarball from the repository through the SMI Cluster Manager CLI.

    software-packages download url repo_url user username password password accept-self-signed-certificate 
    software-packages list 

    NOTES:

    • software-packages download url repo_url - Specifies the software package tarball to download from the repository. repo_url specifies the URL of the repository where the software package resides. For example, http://ipv4address:8080/packages/cee-version.tar

    • user username - (Optional) Specifies the user name. It is an optional parameter.

    • password password accept-self-signed-certificate - (Optional) Specifies the password. accept-self-signed-certificate accepts self signed certificates.

  2. Use the following configuration to modify the CEE Ops Center to point it to the new tarball.

    
    configure 
         cluster cluster_name 
             ops-centers app_name instance_name 
             repository repo_url 
             initial-boot-parameters auto-deploy true 
             exit 
         commit 
    

    NOTES:

    • cluster cluster_name - Specifies the name of the cluster. For example, aio.

    • ops-centers app_name instance_name - Specifies the installation of the Ops Center. app_name is the name of the application. For example, cee.The instance_name is the name of the instance. For example, global.

    • repository repo_url - Specifies the product chart repository URL.

    • initial-boot-parameters auto-deploy true – Deploys the product chart automatically.

    • commit - Commits the configuration changes.

  3. Run the cluster synchronization to upgrade the CEE Ops Center.


    Note

    Ensure that you enable auto deploy for the CEE products that are being updated.
    clusters cluster_name actions sync run 
    

    NOTES:

    • clusters cluster_name actions sync run - Synchronizes the committed changes to the cluster.

  4. Verify whether the helm charts have been updated through the CEE Ops Center.

    show helm charts 
    

    A sample output is shown below:

    CHART INSTANCE STATUS VERSION REVISION RELEASE NAMESPACE 
    ------------------------------------------------------------------------------------------------------------------------------------------- 
    pv-manager * DEPLOYED 0.1.0-master-0003-190828151827-38fee28 1 cee-global-pv-manager cee-global 
    product-documentation * DEPLOYED 0.5.1-master-0027-190806220948-8d68a0e 1 cee-global-product-documentation cee-global 
    smi-show-tac * DEPLOYED 0.1.0-master-0088-190913221302-4ca7229 1 cee-global-smi-show-tac cee-global 
    storage-provisioner * DEPLOYED 0.1.0-master-0075-190906225356-7c816b6 1 cee-global-storage-provisioner cee-global 
    cnat-monitoring * DEPLOYED 0.5.1-master-0018-190913224841-02a0e12 1 cee-global-cnat-monitoring cee-global 
    

    NOTES:

    • show helm charts - Displays the helm charts.

  5. Verify the status of the system.

    show system status 
    

    A sample output is shown below:

    system status deployed true 
    system status percent-ready 91.3 

    NOTES:

    • show helm status - Displays the status of the system.

Upgrading CEE Products

To upgrade the CEE products, use the following configurations:

  1. Access the CEE Ops Center through the ingress URL.

    
    https://cli.cee-global-ops-center.<ipv4_address>.<customer_specific_domain_name> 
    

    NOTES:

    • customer_specific_domain_name - Specifies the name of the domain specific to the customer.

  2. Use the following configuration to update the CEE products chart URL.

    configure 
                helm default-repository repo_name 
                helm repository repo_name 
                url cee_product_chart_url 
                username username 
                password password 
                exit 
             commit 
    

    NOTES:

    • customer_specific_domain_name - Specifies the name of the domain specific to the customer.

    • helm default-repository repo_name - Specifies the default helm repository name.

    • helm repository repo_name - Specifies the name of the helm repository to update.

    • url cee_product_chart_url - Specifies the product chart URL. For example, http://charts.<ipv4address>.<domain_name>/cee-2019-09-13/

    • username username - Specifies the user name.

    • passwordpassword - Specifies the password.

    • commit - Commits the configuration changes.

Configuring CEE

The subsequent sections provide more information about the CEE configuration procedures.

Configuring Alerts

When an anomaly is detected, the system generates a notification called an alert. Based on the statistics pegged by the system, alerts are fired. You can configure an expression to fire an alert when the expression becomes true.

The CEE uses the Prometheus Alert Manager for alerting operations. The CEE YANG model - either through CLI or API - allows users to view the active alerts, silenced alerts and alert history. A predefined set of alerting rules are added whenever the application is installed or updated. Also, the applications can call the alert API directly to add or clear alerts. The Prometheus Alert Manager API (v2) is the standard API used.

The Prometheus Alerts Manger includes the following options:

  • Defining Alert Rules – This option defines the Alert Manager on what to alert. You can define the alerts using the Prometheus Query Language (PromQL).

  • Defining Alert Routing – This option defines the Prometheus Alert Manager on what to do with the received alerts. At present, the SNMP Trapper is supported as the outbound alerting. Also, the CEE provides an Alert Logger for storing the generated alerts.

Requirements

For configuring alerts:

  1. The CEE product must be installed and running.

  2. The CEE Ops Center must be accessible.

Configuring Alert Rules

Use the following configuration to configure the alert rules.


configure 
   alerts rules group alert_group_name 
   interval-seconds seconds 
   rule rule_name 
      expression promql_expression 
      duration duration 
      severity severity_level 
      type alert_type 
      annotation annotation_name 
      value annotation_value 
      exit 
exit 

NOTES:

  • alerts rules – Specifies the Prometheus alerting rules.

  • interval-seconds seconds – Specifies the evaluation interval of the rule group in seconds.

  • group alert_group_name – Specifies the Prometheus alerting rule group. One alert group can have multiple list of rules. alert_group_name is the name of the alert group. The alert-group-name must be a string in the range of 0 through 64 characters.

  • rule rule_name – Specifies the alerting rule definition. rule_name is the name of the rule.

  • expression promql_expression – Specifies the PromQL alerting rule expression. promql_expression is the alert rule query expressed in PromQL syntax. The promql_expression must be a string.

  • duration duration – Specifies the duration of a true condition before it is considered true. duration is the time interval before the alert is fired.

  • severity severity_level – Specifies the relative level of urgency for the operator's attention. severity_level is the severity level of the alert. The severity levels are: critical, major, minor and warning.

  • type alert_type – Specifies the type of the alert. alert_type is the user-defined alert types. For example, Communications Alarm, Environmental Alarm, Equipment Alarm, Indeterminate Integrity Violation, Operational Violation, Physical Violation, Processing Error Alarm, Quality of Service Alarm, Security Service, Mechanism Violation, or Time Domain Violation.

  • annotation annotation_name – Specifies the annotation to attach to the alerts. annotation_name is the name of the annotation.

  • value annotation_value – Specifies the annotation value. annotation_value is the value of the annotation.

The following example monitors the success rate of SMF session creation by configuring Prometheus alert rule to report if session creation is less than threshold.

Example:
cee# configure terminal
  alerts rules group SMFProcStatus
  interval-seconds 300
  rule PDNSessCreate
  expression "sum(increase(smf_service_stats{app_name=\"SMF\",procedure_type=\"pdn_sess_create\",status=\"success\"}[5m]))  / sum(increase(smf_service_stats{app_name=\"SMF\",procedure_type=\"pdn_sess_create\",status=\"attempted\"}[5m])) < 0.95"
  severity   major
  type       "Communications Alarm"
  annotation summary
  value "This alert is fired when the success percentage of pdn_sess_create procedure is lesser threshold”
  exit

In the following example, a alert is sent as SNMP Trap to receiving agent when a snmp-trapper is configured.

Example:
cee# configure terminal
  snmp-trapper enable true v2c-target 172.16.181.41 community public port 161
  exit

The following example configures an alert, which is fired when the percentage of UDM responses is less than the specified threshold limit.

Example:

cee# configure terminal
		alerts rules group SMFUDMchk_incr
		interval-seconds 300
		rule SMFUDMchk_incr
		expression "sum(increase(smf_restep_http_msg_total{nf_type=\"udm\", message_direction=\"outbound\", response_status=~\"2..\"}[3m])) /  sum(increase(smf_restep_http_msg_total{nf_type=\"udm\", message_direction=\"outbound\"}[3m])) < 0.95"
		severity major
		type "Communications Alarm"
		annotation summary
		value "This alert is fired when the percentage of UDM responses is less than threshold“
		exit
	exit
exit

You can view the configured alert using the show running-config alerts command.

Example:

The following example displays the alerts configured in the running configuration:

cee# show running-config alerts
               interval-seconds 300
               rule SMFUDMchk_incr
               expression "sum(increase(smf_restep_http_msg_total{nf_type=\"udm\", message_direction=\"outbound\", response_status=~\"2..\"}[3m])) /  sum(increase(smf_restep_http_msg_total{nf_type=\"udm\", message_direction=\"outbound\"}[3m])) < 0.95"
               severity major
               type "Communications Alarm"
               annotation summary
               value "This alert is fired when the percentage of UDM responses is less than threshold“              
	   exit
      exit
exit

Configuring Alerts for ETCD Nodes

The ETCD runs on separate nodes in a multi-node environment as opposed to a container within Kubernetes environment. A Node-Exporter runs on each of the ETCD nodes to obtain host level metrics. Also, the CEE Prometheus starts scrapping the metrics automatically after deployment.

You can create alerting rules based on the ETCD Node-Exporter metrics. To configure alerting rules based on ETCD Node-Exporter metrics, use the ETCD Node IP as the instance label instead of the Pod name in the expression.


Important

The Node-Exporter on ETCD is not running as a Kuberenets Pod.


The following examples configure alerting rules based on ETCD Node-Exporter metrics.

Example:

The following expression configures alerts based on the availability of host memory (less than 30%):

((node_memory_MemAvailable_bytes{{{instance="<ETCD-Node-IP>:9100"}}} / node_memory_MemTotal_bytes{{{instance="<ETCD-Node-IP>:9100"}}}) < 30

The following expression configures alerts based on the average CPU usage for five minutes. (greater than 70%):

sum(avg without (cpu)(irate(node_cpu_seconds_total{instance="<ETCD-Node-IP>:9100",mode!="idle"}[5m]))) * 100 > 70

Helm Deployment Alert Rule

The CEE Ops Center comes equipped with a built-in alert rule - helm_deploy_failure - to indicate the failure status of helm chart deployment. This alert rule comes by default as a Prometheus alerting rule during CEE deployment.

The following is an alert rule definition for helm_deploy_failure alert in Prometheus:

- alert: helm-deploy-failure
    annotations:
      type: Processing Error Alarm
      description: 'Helm chart {{$labels.chart}}/{{$labels.namespace}} deployment failed'
      summary: 'Helm chart failed to deploy for 5 minutes'
    expr: |
      helm_chart_deploy_success < 1
    labels:
      severity: critical
    for: 5m

The following example shows an alert generated when helm chart deployment fails.

alerts active helm-deploy-failure 3edde79a3f86
state active
severity critical
type "Processing Error Alarm"
startsAt 2020-04-17T17:55:57.084Z
source tfchan-dev
labels [ "chart: smi-show-tac" "chartVersion: 0.1.0-helmfail-0108-200310183805-6888120" "component: ops-center" "exported_release: cee-smi-show-tac" "instance: 192.168.190.28:8082" "job: kubernetes-pods" "namespace: cee" "pod: ops-center-cee-ops-center-5ccddd5d9f-6rffw" "pod_template_hash: 5ccddd5d9f" "release: cee-ops-center" ]
annotations [ "description: Helm chart smi-show-tac/cee deployment failed" "summary: Helm chart failed to deploy for 5 minutes" ]

Note

If SNMP Trapper is configured, this alert goes to the external SNMP receiver as an SNMP trap. For instance, when there is already a conflict of resources, the Helm deployment fails.


Viewing Alert Logger

The Alert Logger stores all the generated alerts by default. You can view the stored alerts using the following show command.

show alert history [ filtering ]

You can narrow down the result using the following filtering options:

  • annotations – Specifies the annotations of the alert.

  • endsAt – Specifies the end time of the alert.

  • labels – Specifies the additional labels of the alert.

  • severity – Specifies the severity of the alert.

  • source – Specifies the source of the alert.

  • startsAt – Specifies the start time of the alert.

  • type – Specifies the type of the alert.

The following examples displays the history of the alerts configured in the system:

Example:
cee# show alerts history
alerts history k8s-job-failed e29cae925397
 severity    warning
 type        "Processing Error Alarm"
 startsAt    2020-01-07T01:56:49.094375165Z
 endsAt      ""
 source      user-master-01
 labels      [ "component: kube-state-metrics" "exported_namespace: default" "instance: 10.0.2.4:8080" "job: kubernetes-pods" "job_name: istio-security-post-install-1.2.0" "namespace: cee" "pod: kube-state-metrics-5bb954789f-7gmq9" "pod_template_hash: 1665103459" "release: cee-cnat-monitoring" ]
 annotations [ "summary: Job cee/istio-security-post-install-1.2.0 failed to complete." ]

You can view the active and silenced alerts with the show alerts active and show alerts active commands

Example:
cee# show alerts history
alerts active SMFUDMchk_incr ac2a970ab621
state active
severity major
type "Communications Alarm"
startsAt 2019-11-15T08:26:48.283Z
source System
annotations [ "summary:This alert is fired when the percentage of UDM responses is less than threshold.” ]

The following examples displays the active alerts. The alerts remain active as long as the evaluated expression is true.

Example:
cee# show alerts active
alerts active k8s-job-failed 5dac1f81a95d
 state       active
 severity    warning
 type        "Processing Error Alarm"
 startsAt    2020-01-07T01:56:49.094Z
 source      user-master-01
 labels      [ "component: kube-state-metrics" "exported_namespace: default" "instance: 10.0.2.4:8080" "job: kubernetes-pods" "job_name: istio-security-post-install-1.2.0" "namespace: cee" "pod: kube-state-metrics-5bb954789f-7gmq9" "pod_template_hash: 1665103459" "release: cee-cnat-monitoring" ]
 annotations [ "summary: Job cee/istio-security-post-install-1.2.0 failed to complete." ]
Example:
cee# show alerts active
alerts active SMFUDMchk_incr ac2a970ab621
state active
severity major
type "Communications Alarm"
startsAt 2019-11-15T08:26:48.283Z
source System
annotations [ "summary:This alert is fired when the percentage of UDM responses is less than threshold.” ]

Enabling SNMP Traps

Use the following configuration to enable the SNMP Traps.


configure 
   snmp-trapper enable true 
   snmp-trapper { v2c-target target | v3-target target | v3-engine-id source_engine_id }  
      community [ community_string ] 
      port [ port ] 
      exit 
   snmp-trapper source-ip-routes [ vip_options ] 
   exit 

NOTES:

  • snmp-trapper enable true – Enables the snmp-trapper parameters

  • v2c-target|v3-target [ target ] – Specifies the list of SNMP v2c and v3 trap receivers.

  • community [ community_string ] – Specifies the SNMP Trap receiver community.

  • v3-engine-id source_engine_id – Specifies the source engine ID for the v3 traps. source_engine_id must be an hexagonal string. For instance, 80004f.

  • port [ port ] – Specifies the SNMP Trap receiver port. port must be an integer in the range of 0 through 65535. The default value is 162.

  • source-ip-routes [ vip_options ] – Enables binding to source IP for SNMP routing. vip specifies the virtual IP (VIP) address. The different options for virtual IP addresses include:

    • default-external-vip – Specifies the default external VIP for source IP routing.

    • internal-vip – Specifies the internal VIP for source IP routing.

    • source-external-vips -Specifies the external VIP per namespace.

Disabling SNMP Traps

Use the following configuration to disable SNMP Traps.


configure 
   no snmp-trapper enable 
   exit 

NOTES:

  • no snmp-trapper enable - Disables SNMP Traps.

Configuring Bulk Statistics

Bulk statistics provide a mechanism to view the summary of the CEE metrics. You can configure bulk statistics to pull the CEE metrics periodically. Also, you can download the metrics in Comma-Separated Value (CSV) format through Secure File Transfer Protocol (SFTP).

Use the following configuration to configure bulk statistics in CEE Ops Center.


configure 
   bulk-stats enable [ true ] 
   bulk-stats external-ip [ ipv4_address ] 
   bulk-stats external-port [ port ] 
   bulk-stats interval-minutes [ interval ] 
   bulk-stats pod-query [ pod_query ] default-value value  
   bulk-stats prune-interval-days [ prune_interval ] 
   bulk-stats query [ query ] default-value value 
   bulk-stats query [ query ] default-namespace namespace 
   bulk-stats user [ user ] 
   bulk-stats vnf-name [ vnf ] 
   bulk-stats global-default-value [ default_value ] 
   bulk-stats global-default-namespace [ default_namespace ] 
   bulk-stats vnf-alias [ vnf_alias ] 
   exit 

NOTES:

  • bulk-stats enable [ true ] – Enables the bulk statistics.

  • global-default-value [ default_value ] – Specifies the default value used in bulk-stats, if pod-query or query fails to return any value.

  • global-default-namespace [ default_namespace ] – Configure this value when the global-default-value is configured. For each namespace in the list, one entry is added to bulk-stats with query returned value or configured default value, if query fails to return any value.

  • external-ip [ ipv4_address ] – Specifies the external IP for downloading the bulk statistics over SFTP.

  • external-port [ port ] – Specifies the external port for downloading the bulk statistics over SFTP.

  • interval-minutes [ interval ] – Specifies the time interval (in minutes) to create the bulk statistics.

  • pod-query [ pod_query ] default-value value – Specifies the query to execute for retrieving the bulk statistics data. ] default-value value is the default value used in bulk-stats, if the configured pod-query fails to return any value. value will override the global-default-value

  • prune-interval-days [ prune_interval ] – Prunes the interval (in days) to remove the bulk statistics.

  • query [ query ] default-value value – Specifies the query to execute for retrieving the bulk statistics. default-value value is the default value used in bulk-stats, if the configured query fails to return any value. value will override the global-default-value

  • query [ query ] default-namespace namespace – Specifies the query to execute for retrieving the bulk statistics.

    • default-namespace namespace – Overrides the global-default-namespace. You can configure this value when the default-value is configured. For each namespace in the list, one entry is added to bulk-stats with query returned value or configured default value, if query fails to return any value.

  • user [ user ] – Specifies the user authorized to download the bulk statistics files.

  • vnf-name [ vnf ] – Specifies the VNF name (namespace) to add in the bulk statistics CSV file.

  • vnf-alias [ vnf_alias ] – Specifies the VNF alias for a specified namespace.

The following example generates query for current PDU per 4G session.

Example:

cee# configure terminal
  bulk-stats enable true
  bulk-stats user admin
  bulk-stats external-ip 172.16.181.41
  bulk-stats external-port 2222
  bulk-stats vnf-name lbucs009
  bulk-stats query 4G_current_pdu_sessions
  expression "sum(smf_up_session_counters{app_name=\"SMF\",rat_type=\"EUTRA\"})"
  label      4G_current_pdu_sessions
  exit

The following example generates query for current PDU per 4G IPv6 session.

Example:

cee# configure terminal
  bulk-stats enable true
  bulk-stats user admin
  bulk-stats external-ip 172.16.181.41
  bulk-stats external-port 2222
  bulk-stats vnf-name lbucs009
  bulk-stats query 4G_current_pdu_sessions_IPv6
  expression "sum(smf_up_session_counters{app_name=\"SMF\",rat_type=\"EUTRA\",pdu_type=\"ipv6\"})"
 label      4G_current_pdu_sessions_IPv6
 exit

Retrieving Bulk Statistics

The CEE stores the Bulk statistics in the CSV format. You can download the stored files from the external host. You must configure the external IP and port to download the bulk statistics.

Use the following command from the external host, which is accessible to the CEE cluster, to download or retrieve the bulk statistics in CSV format.


scp -P [external-port] [user]@[external-ip]:/data/[bulk | rate]/[filename].csv [local-folder] 

Also, you can use any of the Graphical User Interface (GUI) SFTP tool to browse and download the CSV files. A SFTP tool displaying the directory where bulk statistics are stored in CSV format is shown below:

Figure 1. Bulk Statistics - GUI

The following example displays the various parameters in bulk statistics.

Example:
cee# show bulk-stats
NAMESPACE   KEY                      LABEL               VALUE 
---------------------------------------------------------------
*        active-alerts               watchdog            1.0   
*        active-alerts               k8s-job-completion  1.0   
*        active-a                    k8s-job-failed      1.0   
cee      configuration-change-total  System              12.0  
cee      cpu-core-count              user-master-01    24.0  
*        cpu-idle                    user-master-01    85.659
*        cpu-iowait                  user-master-01    0.083 
*        cpu-softirq                 user-master-01    0.162 
*        cpu-steal                   user-master-01    0.0   
*        cpu-system                  user-master-01    4.726 
*        cpu-user                    user-master-01    8.024 
cee      daemonset-ready-percent     core-retriever      100.0 
cee      daemonset-ready-percent     kube-proxy          100.0 
cee      daemonset-ready-percent     logs-retriever      100.0 
cee      daemonset-ready-percent     node-exporter       100.0 
cee      daemonset-ready-percent     path-provisioner    100.0 
cee      daemonset-ready-percent     weave-net           100.0 
cee      deployment-ready-percent    alert-logger        100.0 
cee      deployment-ready-percent    api-cee-ops-center  100.0 
cee      deployment-ready-percent    cdl-ep-session-c1   100.0 
cee      deployment-ready-percent    cee-product-
                                        documentation    100.0 
cee      deployment-ready-percent    documentation       100.0 

Grafana

Grafana is an open source data visualization tool used for displaying application metrics in interactive dashboards.

Accessing Grafana

You can access Grafana login page through Ingress using any of the standard web browsers. For instance, using Google Chrome navigate to the Grafana login page with the following Ingress URL:

https://grafana.<ipv4_address>.<customer_specific_domain_name>

NOTES:

  • customer_specific_domain_name - Specifies the customer's domain name.

Figure 2. Grafana – Login Page

Important

Authentication to Grafana happens through the CEE Ops Center since Grafana is associated with it. If the CEE Ops Center is configured with Lightweight Directory Access Protocol (LDAP), Grafana authenticates through LDAP.

Using Dashboards

Grafana home page lists the dashboards bundled with CEE. The dashboards provide an overall status of the system.

To view the dashboards, perform the following:

  1. Navigate to Grafana Login page using any standard web browser.

  2. Login to Grafana to view the home page.

    Figure 3. Grafana – Home Page
  3. Click icon on the left pane.

  4. Click Dashboards.

    1. By default, you can view the following dashboards.

      1. Host Summary – This dashboard provides a overview of CPU, Memory, Disk I/O Utilization, Filesystem Fullness and Filesystem Fill UP time. You can choose the machine from Machine drop-down list to view the host summary of individual machines available in the clusters.

      2. Host Details – This dashboard provides a detailed view on each of the following categories:

        - Basic CPU/ Mem / Disk Gauge

        - Basic CPU/ Mem / Disk Info

        - Basic CPU / Mem Graph

        - Basic Net / Disk Info

        - CPU Memory Net Disk

        - Memory Detail Meminfo

        - Memory Detail Vmstat

        - Memory Detail Vmstat Counters

        - System Detail

        - Disk Detail

        - Filesystem Detail

        - Network Traffic Detail

        - Network Sockstat

        - Network Netstat

        - Network Netstat TCP

        - Network Netstat TCP Linux MIPs

        - Network Netstat UDP

        - Network Netstat ICMP

        - Node Exporter

You can create new Dashboards or Data Sources in Grafana through settings ( icon) tab.

Figure 4. Settings Tab

Important

The bundled dashboards are provisioned statically and cannot be modified. However, you can copy or clone the dashboards before saving the changes.

The following example imports custom Grafana dashboard from a GIT repository.

Example:
cee# configure terminal
  grafana dashboards sample
  git-url https://wwwin-github.cisco.com/mobile-cnat-sample/sample-dashboards.git
  exit

User Management in Grafana

You can create and manage the local users and user groups through the CEE Ops Center as described in the Provisioning Local Users section.

Provisioning Local Users

A new YANG model is introduced in SMI to support user management in compliance with Cisco Secure Development Life-cycle (CSDL) requirements.

Note

This new YANG model is applicable to SMI Cluster Manager and all other product Ops Centers.


User Management

This chapter describes how to create and manage local users using the Ops Center CLI (for both the products and SMI Cluster Manager Ops Center).


Important

Users with administrator privileges can add, modify, and delete other users and groups. All the other users only have privileges to change their own password.


Adding a User

To add a new user, use the following configurations:

configure
  smiuser add-user username username password password
  exit 

Note

  • smiuser add-user - Adds a new local user.

  • username username - Specifies the name of the user.

    username must be alphanumeric string.
  • password password - Specifies the password. password must be:

    • Minimum 8 characters in length.

    • Contain at least one lowercase character.

    • Contain at least one uppercase character.

    • Contain at least one numeric character.

    • Contain at least one special character, which includes the following

      • [ '~', '@', '#', '%', '^', '&', '*', '(', ')', '_', '+', '`', '-', '=', '{', '}', '[', ']', ':', '"', ';', '\'', '|', '\\', '<', '>', '?', ',', '.', '/', '$']


The following example adds a new user called 'user1' and assigns the password for the new user.

cee# configure terminal
  smiuser add-user username user1 password Cisco@123
message User added

The following example adds a new user called 'user2' and assigns the password for the new user.

cee# configure terminal
  smiuser add-user username user2 password Cisco@12345
message User added

In the following example, when an existing user name (user2) is added as a new user, the Ops Center displays an error message.

cee# configure terminal
  smiuser add-user username user2 password Cisco@12345
message User already exists

Deleting a User

To delete a user, use the following configuration:

configure
  smiuser delete-user username username
  exit 

Note

  • smiuser delete-user - Deletes a local user.

  • username username - Specifies the name of the user.

    username must be alphanumeric string.

The following example deletes a user called 'user2'.

cee# configure terminal
  smiuser delete-user username user2
message User deleted

In the following example, when a non-existing user is deleted, the Ops Center displays an error message.

cee# configure terminal
  smiuser delete-user username user2
message User does not exist

Modifying the Password

To modify the password (for self), use the following configuration:
configure
  smiuser change-self-password current_password current password new_password new_password 
  confirm_password new_password password_expire_days number_of_days
  exit
 

Note

  • smiuser change-password - Modifies the password for an user.

  • current_password current_password - Specifies the current password for an user.

  • new_password new_password - Assign a new password for the user. For information on password policy, see Adding a User section.

  • confirm_password new_password - Enter the newly assigned password one more time.

  • password_expire_days number_of_days - (Optional) Specifies the expiry date of the password. The default value is 180 days.


The following example updates the password for the current user.

cee# configure terminal
  smiuser change-self-password current_password Cisco@123 new_password Cisco@345 
  confirm_password Cisco@345 password_expire_days 180
message Password updated successfully

The following example updates the password for the user called 'user1' without assigning the password expiry date.

cee# configure terminal
  smiuser change-self-password current_password Cisco@123 new_password Cisco@345 confirm_password Cisco@345
message Password updated successfully
Reset the Administrator Password

You can reset the administrator password if you have access to the K8s Cluster through kubectl command-line utility.

To reset the administrator password:

  1. Enter the Ops Center Pod's EXEC mode.

  2. Use the following command to reset the administrator password.

    kubectl exec -it <pod_name> -n <pod_namespace> /usr/local/bin/reset-admin 
  3. Enter the new password when prompted.

NOTES:

  • kubectl exec -it - Executes a command inside a container. -it passes the standard input stream to the container or TTY.

  • <pod_name> -n - Specifies the name of the Pod. -n specifies the namespace scope for this CLI request.

  • <pod_namespace> - Specifies the namespace of the Pod.

  • /usr/local/bin/reset-admin - Resets the administrator password.

Modifying the Password for Other Users

You can modify the password for other users using the following configuration:

configure
  smiuser change-password username username current_password current_password new_password new_password 
  confirm_password new_password password_expire_days number_of_days
  exit
 

Note

  • smiuser change-password - Modifies the password for an user.

  • username username - Specifies the name of the user.

    username must be alphanumeric string.
  • current_password current_password - Specifies the current password for an user.

  • new_password new_password - Assign a new password for the user. For information on password policy, see Adding a User section.

  • confirm_password new_password - Enter the newly assigned password one more time.

  • password_expire_days number_of_days - (Optional) Specifies the expiry date of the password. The default value is 180 days.


The following example updates the password for the user called 'user1'.

cee# configure terminal
  smiuser change-password username user1 current_password Cisco@123 new_password Cisco@345 confirm_password Cisco@345 password_expire_days 180
message Password updated successfully

The following example updates the password for the user called 'user1' without assigning the optional password expiry date.

cee# configure terminal
  smiuser change-password username user1 current_password Cisco@123 new_password Cisco@345 confirm_password Cisco@345
message Password updated successfully

The following example updates the password for the user called 'user1' without assigning the password expiry date.

cee# configure terminal
  smiuser change-password username user1 current_password Cisco@123 new_password Cisco@345 confirm_password Cisco@345
message Password updated successfully

The following example updates the password for the user called 'user1' with an existing password.

cee# configure terminal
  smiuser change-password username user1 current_password Cisco@345 new_password Cisco@345 confirm_password Cisco@345
message Password has been already used

The following example updates the password for the user called 'user1' with different values for new password and confirm password parameters.

cee# configure terminal
  smiuser change-password username user1 current_password Cisco@345 new_password Cisco@345 confirm_password Cisco@567
message Passwords do not match

Updating the Password Length

To update the length of the password, use the following configuration:

configure
smiuser update-password-length length number_of_characters 
 exit 

Note

  • smiuser update-password-length - Updates the length of the password.

  • length number_of_characters - Specifies the length of the password. number_of_characters must be a numeric value.


The following example updates the minimum length of the password to 10 characters.

cee# configure terminal
  smiuser update-password-length length 10
message Password updated successfully

Group Management

This chapter describes how to create and manage user groups using the Ops Center CLI (of both the products and SMI Cluster Manager).

Adding a User Group

To add a user group, use the following configuration:

configure
smiuser add-group groupname group_name 
 exit 

Note

  • smiuser add-group - Adds a new user group.

  • groupname group_name - Specifies the name of the user group. group_name must be a alphanumeric value.


The following example adds a new user group called 'group1'.

cee# configure terminal
  smiuser add-group groupname group1
message Group added

In the following example, when a user group that already exists is added, the Ops Center displays an error message.

cee# configure terminal
  smiuser add-group groupname group1
message Group already exists

Deleting a User Group

To delete a user group, use the following configuration:
configure
smiuser delete-group groupname group_name 
 exit 

Note

  • smiuser delete-group - Deletes a user group.

  • groupname group_name - Specifies the name of the user group. group_name must be a alphanumeric value.


The following example deletes a new user group called 'group2'.

cee# configure terminal
  smiuser delete-group groupname group2
message Group deleted

In the following example, when a user group that does not exist is deleted, the Ops Center displays an error message.

cee# configure terminal
  smiuser delete-group groupname group2
message Group does not exist

Assigning an User to an User Group

To assign an user to an user group, use the following configuration:

configure
  smiuser assign-user-group username username group group_name 
   exit 

Note

  • smiuser assign-user-group - Assigns an user to a user group.

  • username username - Specifies the name of the user. username must be alphanumeric value.

  • groupname group_name - Specifies the name of the user group. group_name must be a alphanumeric value.


The following example assigns an user called 'user1' to a group called 'group1'.

cee# configure terminal
  smiuser assign-user-group username user1 group group1
message User assigned to group successfully

The following example assigns a non-existing user to an existing group.

cee# configure terminal
  smiuser assign-user-group username user20 group group1
message User does not exist

The following example assigns a non-existing group to an existing user.

cee# configure terminal
  smiuser assign-user-group username user1 group group10
message Group does not exist

Unassigning a User from a User Group

To unassign a user from a user group, use the following configuration:

configure
  smiuser unassign-user-group username username group group_name 
   exit 

Note

  • smiuser unassign-user-group - Removes an user from a user group.

  • username username - Specifies the name of the user. username must be alphanumeric value.

  • groupname group_name - Specifies the name of the user group. group_name must be a alphanumeric value.


The following example removes an user from a group.

cee# configure terminal
  smiuser unassign-user-group username user1 group group1
message User un-assigned from group successfully

The following example removes a non-existing user from a group.

cee# configure terminal
  smiuser unassign-user-group username user10 group group1
message User is not a member of this group

The following example removes an user from an non-existing group.

cee# configure terminal
  smiuser unassign-user-group username user1 group group10
message Group does not exist

Log Forwarding

Log Forwarding allows you to forward the log entries (including the host and container level log entries) stored in the JournalD to the external collectors. Currently, it supports the external Fluent-x and Splunk as endpoint. For Fluent-x, it uses the Forward protocol for Splunk it uses HTTPS.

Figure 5. Log Forwarding Architecture

Prerequisites

You can enable Log Forwarding in CEE to forward log entries to the external collectors. You must ensure that the K8s cluster is installed in the CEE before enabling Log Forwarding.

Requirements

The following are the requirements for enabling Log Forwarding:

Fluent-x

  1. The target endpoint must be a Fluentbit or FluentD instance or cluster with the Forward protocol input plugin enabled.

  2. The endpoint must be hosted within the Kubernetes clusters or a remote system with network reachability.

Enabling Log Forwarding

Enabling Log Forwarding on Fluent-x

Use the following configuration to enable Log Forwarding on Fluent-x.

configure 
logging fluent host fluentbit/fluentd_endpoint_fqdn/ipv4_address port endpoint_port 

NOTES:

  • logging fluent – Specifies the Fluent forwarding parameters.

  • host fluentbit/fluentd_endpoint_fqdn/ipv4_address – Specifies the Fluentbit or Fluentd instance host information.

  • port endpoint_port – Specifies the Fluentbit or Fluentd instance port information.

The log forwarding to an external Fluent-D or Fluent-Bit instance, where logs can be streamed to supporting application such as ElasticSearch.

Example:
cee# configure terminal
  logging fluent host 172.16.181.41 port 8001
  exit

Enabling Log Forwarding on StarOS

Use the following configuration to enable Log Forwarding from StarOS.

configure  
  logging 
    syslog cee_ops_center_listener_ip_address 
    facility local5  
    msg-format rfc5424  
    commit 
    

NOTES:

  • syslog - Specifies the syslog messages.

  • cee_ops_center_listener_ip_address - Specifies the CEE Ops Center Listener IP address.

  • facility local5 - Specifies the syslog facility values.

  • msg-format rfc5424 - Specifies the syslog message format.

Configuring CEE Ops Center as a Listener

You can configure the CEE Ops Center to listen to the logs from StarOS.

Use the following configuration to configure CEE Ops Center as listener:

configure  
  logging 
    listener enable 
    external-ip cee_ops_center_listener_ip_address 
    commit 
  

NOTES:

  • listener enable - Enables the CEE Ops Center to listen to the logs from StarOS.

  • external-ipcee_ops_center_listener_ip_address - Specifies the CEE Ops Center Listener IP address.

Viewing the Logs in Loki

You must enable the Loki (Grafana) to view all the logs the CEE Ops Center was listening.

Use the following configuration to enable the Loki:

configure  
  logging 
    loki enable 
    retention-period retention_period_in_hours 
    commit 
  

NOTES:

  • loki enable - Enables Loki to view to the logs.

  • retention-periodretention_period_in_hours - Specifies the retention period of the logs in hours.

Verifying Log Forwarding

You can verify the external collectors for the log entries received. The logs are specific to the external collector. For example, you can use Kibana to verify the entries in ELK stack.

Troubleshooting

This section provides information on the common issues encountered while enabling log forwarding.

To resolve the issues related to Log Forwarding, verify if:

  • The configured endpoint IP/FQDN and port number are correct.

  • The external endpoint is reachable from the all Kubernetes nodes (both master and worker).

  • The Forward protocol plugin is enabled at the endpoint.

  • The logs are generated from any of the nodes.

  • The external endpoint is configured to dump the logs into a file for verifying the incoming entries.

Log Rate Limiting

This section describes the basic principle used in rate limiting log messages in SMI Logging functionality.

Rate Limiting Log Messages

The SMI uses the systemd-journald service—a Linux system service for collecting and storing log data—for storing Kubernetes system and pods level log messages to files on the disk. You can configure Rate Limiting to reduce the number of messages logged. Also, Rate Limiting discards some log messages while limiting others. You can apply Rate Limiting to all the messages in the system based on the service so that logs from the services do not interfere with each others limit.

You can configure Rate Limit by defining the RateLimitIntervalSec and RateLimitBurst parameter in /etc/systemd/journald.conf file. If the messages exceed the specified value defined in the RateLimitBurst parameter within the specified interval defined in the RateLimitIntervalSec parameter, the log messages are dropped until the interval period is over.

In the following example, the log messages are dropped, if it exceeds 10000 messages within an interval of one second.
RateLimitIntervalSec=1s 
RateLimitBurst=10000 

The disk usage reserved for journal log affects the RateLimitBurst parameter. The value defined in the RateLimitBurst parameter is multiplied by a factor based on the disk usage reserved for the journal logs. More messages are dropped within interval when less disk space is available.

You can run the following command to find out if the log messages are dropped:

sudo systemctl status systemd-journald 

Note

Using this command, you can verify the number of suppressed messages as well.


The following example shows the number of suppressed messages from the docker.service:

Sep 02 21:09:58 tb15-ultram-cnat-cnat-core-protocol-data1 systemd-journald[3791]: 
Suppressed 12229 messages from docker.service

Gather TAC

Gather TAC is the primary mechanism through which the application debug files are extracted from a cluster. Whenever a debug package is required, a user can trigger the Gather TAC through the CLI or API. The user can specify a start and end time to download the index files (for that specific time period) of all the artifacts from the system. The user can extract the following data using Gather TAC:

  • A tar ball of system and K8s pod logs.

  • A tar ball of all bulk statistics produced within the specified time period.

  • A tar ball of the current configuration, last 100 commits, and all audit information.

  • A tar ball of the Prometheus data covering the time period.

  • A list of all core files covering the time period

Debugging Data in CEE

Using the Gather TAC function, you can collect logs from the coredump, Kubernetes, Kernel, Kubelet, and container logs. The collected files are compressed and stored in an internal Apache server.

Debugging Data

  • The following commands are used for requesting the TAC debug information.

    
    tac-debug-pkg create {from start_time | to end_time} {logs-filter namespace namespace | pod_name pod_name} 
    {cores-filter { process process_name }} {{ cfg | cores  | logs | metrics | stats } {false | true}} 
    tac-debug-pkg delete tac-id tac_id 
    tac-debug-pkg merge tac-id tac_id 
    tac-debug-pkg status 
    
  • Access the Apache server through the Ops Center.

    
    URL: https://show-tac-manager.smi-show-tac.{IP address}.<domain_name> 
    
  • A directory is created based on the tac-id: /tac/[tac-id]/

  • A manifest file is created for each of the tac-debug-pkg to the store metadata. A sample manifest.json file is shown below:

    
    { 
           tac-id:"1554868784", 
           from:"2019-04-08_00:00:00", 
           to:"2019-04-10_00:00:00", 
           cores:[{node:"node-01", 
                   file:"/cores/node-01/core.test.0.2f4afbe0dc494e879d3f42429fed1c38.20130.1554770483000000.xz"}, 
                  {node:"node-01", 
                   file:"/cores/node-01/core.test.0.2f4afbe0dc494e879d3f42429fed1c38.18448.1554770577000000.xz"}], 
           config:[{node:"node-01", 
                    file:"/tac/1554868784/config/10.0.1.96_configuration.tar.gz.base64"}], 
           stats:[{node:"node-01", 
                   file:"/tac/1554868784/stats/Stats_2019-04-08_00-00-00_2019-04-10_00-00-00.tar.gz"}], 
           logs:[{node:"node-01", 
                  file:"/tac/1554868784/logs/Logs_2019-04-10_04-00-17.tar.gz"}], 
           metrics:[{node:"node-01", 
                  file:"/tac/1554868784/metrics/Metrics_2019-04-08_00-00-00_2019-04-10_00-00-00.tar.gz"}]} 
        } 
    

Important

Authentication to the Apache server is enabled by default.
  • The following services collect and retrieve logs, data chunks, and bulk statistics.

    • Core retrieving service - This service retrieves the list of coredump based on the time duration. The systemd coredump service compresses the core files. The configuration parameters in the core files determines the name of the core file. Due to the core file large size, they are not copied on the disk. You can access it through the proxy from its original location.

      
      file location: 
         ./cores/{node name}/core.xxx... 
         Sample file location on server: 
         cores/node-1/core.test.0.99775297099c489ea08052d533206b66.10213.1554504010000000.xz 
      
    • Logs retrieving service - This service collects Kernel, System, Containers level logs using JournalD service. In return, the sender receives a tar file which contains logs files based on the time duration. The files are created with following naming convention:

      
      ./tmp/logs/{random string}/{namespace}/{pod}/{container.log} 
      

A sample file (Tar) format with the timestamp embedded in the file name is shown below:


./tmp/logs/{random string/Logs_{yyyy-mm-dd_hh-mm-ss}.tar.gz 
  • Prometheus data retrieving service - This service retrieves snapshot of data chunks saved by the Prometheus service. You can specify the time duration for saving the snapshots. A sample file and directory structure for the data snapshot is shown below:

    
    directory: data/snapshots/20190405T175611Z-7ee562389bd9ab66/01D7N0QVNBRXF5MRVFQB5MQQCW 
       files: 
       ./chunk/0001 
       ./index 
       ./meta.json 
       ./tombstones 
    
  • Bulk statistics retrieving service - This service retrieves statistics saved by the Prometheus service. You can specify the time duration for saving the statistics. A tar file is stored onto the Apache server for review. A sample file location on the server is shown below:

    
       tac/0123456789/stats/Stats_2019-4-04_00-00-00_2019-04-04_18-00-00.tar.gz 
    

The following example collect logs for pods in cdl-global namespace for CDL application.

Example:
cee# tac-debug-pkg create from 2019-12-18_00:00:00 to 2019-12-18_20:00:00 logs-filter { namespace cdl-global }
response : Tue Dec 18 18:40:55 UTC 2019 tac-debug pkg ID : 157660805

Log Monitoring

For real time monitoring of application logs, the CEE Ops Center uses the Kubetail utility. The Kubetail utility allows:

  • Tailing multiple pods in a single stream.

  • Tailing all containers within the Pods.

  • Using regular expression to match or find Pod names.

  • Color coding the output of each pod.