Contents

Introduction

Alerts in CEE

How to troubleshoot

Introduction

This document describes how to troubleshoot Alerts generated on Common Execution Environment (CEE) with Grafana dashboard.

Alerts in CEE

Alert rules can be configured on CEE Ops Center, here is an example:

alerts rules group Pod

interval-seconds 300

rule Memory_Major

expression "(go_memstats_heap_inuse_bytes{pod=~\"rest-ep.*|smf-service.*|gtpc-ep.*|protocol.*|udp-proxy.*|cache-pod.*\"} /16000000000) >= 0.5"

duration 15m

severity major

type "Processing Error Alarm"

annotation summary

value "\"POD {{ $labels.pod }} in Namespace: {{ $labels.namespace }} has reached 50% of utilization\""

exit

exit

exit

The expression relies on PromQL, in the example it monitors go_memstats_heap_inuse_bytes(= memory usage) for each pods specified. And it calculates the memory utilization with the total memory as 16GB, and if the utilization exceeds 75% generates an alert. Generated alerts can be seen by show alerts history or show alerts active CLI.

[unknown] cee# show alerts active summary | include Memory_Major

Memory_Major 68e812264ed6 major 10-28T02:23:44 worker1 POD cache-pod-0 in Namespace: smf-data has reached 50% of utilization

Memory_Major 627af1cdd01c major 10-28T02:23:44 worker1 POD cache-pod-1 in Namespace: smf-data has reached 50% of utilization

Memory_Major 394d713e294b major 10-28T02:23:44 worker1 POD gtpc-ep-n0-0 in Namespace: smf-data has reached 50% of utilization

Memory_Major bd95b1a35ef5 major 10-28T02:23:44 worker1 POD smf-rest-ep-n0-0 in Namespace: smf-data has reached 50% of utilization

Memory_Major 57254fd42f1a major 10-28T02:23:44 worker1 POD smf-udp-proxy-0 in Namespace: smf-data has reached 50% of utilization

Memory_Major 56135a34c635 major 10-28T02:23:44 worker1 POD smf-service-n0-0 in Namespace: smf-data has reached 50% of utilization

Further explanation about Alerts can be found at this document.

Application-based Alerts

https://www.cisco.com/c/en/us/td/docs/wireless/ucc/smf/b_SMF/b_SMF_chapter_0110101.html

How to troubleshoot

The CLIs don't provide actual measured value, or trending data. The best way to troubleshoot it further is to use Grafana dashboard. As described, Alerts are defined by PromQL, hence same syntax can be applied to create graphs on Grafana.

Taking the rule as an exmaple, this syntax can be used to create a graph.

(go_memstats_heap_inuse_bytes{pod=~"rest-ep.*|smf-service.*|gtpc-ep.*|protocol.*|udp-proxy.*|cache-pod.*"}/16000000000)*100

Notes:

1. Remove "\" in the syntax which is used as escape sequence

2. Multiply by 100 to make it as percentage representation