Prometheus Integration adds support for exporting metrics from both opflex-agent and opflex-server that can be collected by in-cluster or external Prometheus server instance. Following are the goals of this integration:

• Large clusters are a black box; goal is to improve debugging and troubleshooting of operational and network issues

• Collect as many control (OpFlex) and data-path (Open vSwitch) statistics as possible

• Expose these statistics as Prometheus metrics to maintain time series per metric

• Scrape these metrics from the cluster nodes and endpoints

• Use Grafana to display dashboard of these metrics

Prometheus is an open-source systems monitoring and alerting toolkit which can act as data source for grafana, a frontend visualization for the exported metrics. Unlike many other stats collectors, Prometheus prefers collecting metrics via a pull model from each of the exporters.

Opflex-agent and opflex-server have been integrated with prometheus-cpp client exporter library. By default opflex-agent exports metrics on port 9612 and opflex-server exports metrics on port 9632.

To perform automatic service discovery of these ports in k8s, the following annotations and container ports have been configured inside "aci-containers-host" pod:

name: aci-containers-host
        network-plugin: aci-containers
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
        prometheus.io/scrape: true
        prometheus.io/port: 9612
…
        - name: opflex-server
          ports:
            - containerPort: 8009
            - name: metrics
              containerPort: 9632

Relabeling is a powerful tool used by the Prometheus server to dynamically rewrite the label set of a target before it gets scraped. Multiple relabeling steps can be configured per scrape configuration. They are applied to the label set of each target in order of their appearance in the configuration file.

For more information, see:

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config

You can use in-cluster prometheus server "relabel configuration" to scrape metrics from opflex-agent and opflex-server below:

• Relabel_config for pulling metrics from opflex-agent: "prometheus.io/scrape = true" and "prometheus.io/port" to scrape from port 9612

• Relabel_config for pulling metrics from opflex-server: "prometheus.io/scrape = true", "containerPortName = metrics" and "containerPort" to scrape from 9632

Relabel_config is part of scrape_config. For more information, see:

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config

Note	To pull metrics from opflex-agent and opflex-server, we recommend you have two relabel_configs and both under different scrape_jobs.

The OpFlex Prometheus Integration provides several benefits:

• Provides visibility of control (OpFlex) and data-path (Open vSwitch) statistics from compute nodes.

• Time series per metric aids in debugging and troubleshooting of operational and network issues.

• Prometheus metrics can be scraped by Prometheus server, Grafana can be used to create dashboards and alerting based out of these metrics.

• The exported metrics helps in answering many operations questions:

  • How many OpFlex proxy connections are present?

  • Is the OpFlex connection stable?

  • Are there any drops in Open vSwitch tables?

  • Are there policy drops in a particular VRF?

  • What services are used by a deployment?

  • Which clients consume a service?

  • Top N users of a service

  • Top N services used by a client

  • How much traffic is ingressing or egressing a service across all nodes?

  • Is load-balancing working reasonably for a service?

  • Cost analysis for local vs remote region service-pods

• Useful label annotations for every metric. For example, the endpoint gets annotated with the pod name and namespace to uniquely identify the endpoint within k8s. Another example is a contract, which gets annotated with a source EPG, destination EPG along with tenant and classifier information. This helps in correlating the configs present in Cisco ACI with the statistics exposed in compute together.

Be aware of the following issues when configuring the OpFlex Prometheus Integration:

• NodePort service stats get accounted under ClusterIp service stats in on-premises fabrics.

• Useful metrics are exported as part of this release. Disclaimer: https://github.com/noironetworks/opflex/blob/judith/docs/prometheus.md#disclaimer

• Data for various features is pulled and exported to Prometheus at different timer intervals. In a steady state, the data will become eventually consistent. However, if traffic is flowing continuously, there could be slight discrepancy when comparing counters of various exported metrics. For example, the POD and Service counts are pulled at different intervals of time. While checking these at a particular instant of time can show variations, once in a steady state the counters will be eventually consistent.

• In general, there should be a federated Prometheus server to avoid hogging CPU and memory of a cluster. The in-cluster Prometheus server must have a very low retention time.

• Default Port Allocations

  9612 and 9632 are being used by other exporters. The expectation is that customers won't be using other exporters that listen on the same ports. Just to be future proof, ports 9894 and 9895 have been reserved for opflex-agent and opflex-server respectively.

You must complete the following tasks before you configure the OpFlex Prometheus Integration:

• You must have basic working knowledge of the following:

  • Provisioning Kubernetes or OpenStack with the relative ACI plug-ins

    • Cisco ACI CNI Plug-in for Kubernetes or OpenShift

  • OpFlex protocol communication between:

    • compute (opflex-agent) and leaf (opflex-proxy) for on-prem Kubernetes and OpenStack solutions

    • compute (opflex-agent) and opflex-server for overlay CNI solutions

  • A generic understanding of how CNI plug-ins are working